Develop Forward

3rd October 2014 Kevin Teich

Marvel Puzzle Quest launched a year ago and people started playing it. They also started talking about it. As we received more and more feedback from players, we wanted to react more and more.

At the time we made a new release every two weeks. To implement a change it would take 1-2 weeks to finish the release we were working on, 2 weeks to implement the new feature in the next release, a few days to test and fix bugs, and 1-2 weeks for Apple to approve it. That’s 4-6 weeks to respond to player feedback! We wanted to do better than that.

Our engine has patching functionality that allows the game to download and apply new data hosted with Amazon’s CloudFront. When this was first added to the engine, I was thinking we’d only use it in the case of emergency, like fixing an exploit or a horribly unbalanced character. But we realized it was also an opportunity to deliver new content without that 4-6 week wait.

I was scared at first. It’s my job to make sure the next release is locked down and shipped out, which is hard enough in and of itself; why start making changes to an older release that has already been tested and approved?

The traditional cycle of game development is design-build-fix-ship, and once it's shipped, you work on the next version. This is great when you're shipping off a milestone to a publisher, or preparing a demo for a convention. But with a live game, your game is always being played, and your ability to react to what your players want can make the difference between success and failure. Developing new features doesn't make sense if your players aren't enjoying the ones they currently have.

So we switched up the way we looked at our releases. Instead of shipping off releases and abandoning them to the wild, any new work that could be done in the live release, would be. It was a simple change in viewpoint, but it struck at the core of how our team looked at developing games. We still put our efforts into planning and developing new features, but our content developers keep their sights on the live release. We came up with the phrase “Develop Forward” as a rallying cry for this change in direction.

Three things were important to support this shift in methodology:

Version Control:
We set up our source control to let developers work on multiple releases. Each release is a separate branch. Developers can make data changes to the live version and test them on PC. Those changes get automatically handled by a series of builders that watch for source control checkins and integrate changes to future releases. A designer can implement a new character ability in the live release, and it gets automatically integrated to the releases that are still in development and on-deck for launch.

Engine Support:
Our patching technology relies on a builder that looks for data changes checked into source control in the live release. When the final version of a release is made, all the content is cooked into archive files. Those archives are checked into source control for safe keeping. When a developer makes a subsequent data change, the patch builder scans for any data that has been changed since the original archive files were built, and creates an update archive that contains only those changes. When the game starts up, it tells our servers what version of data it has, and the servers can tell the game if there’s a new update archive available. The game streams in new content from the update archive.

The build and data management processes are automated by builders that watch source control. When our testers get to work in the morning, our Q.A. Lead looks at the most recent update archive, activates it on our test server, and our testers can start their review. They use the same version of the game that’s live, but with the new update. Once they approve all the changes we made in the last day, the update is activated on the live servers.

At this point, it’s second nature for the team to think this way. We’re comfortable in the new world of developing “Games as a Service”, and are able to respond to player feedback faster than ever.

About the author: Kevin Teich is a Development Director, a Senior Engineer, a cyclist and a Brewmaster.