A few people have asked us about the technology behind Marvel Puzzle Quest’s multiplayer servers. Here’s a list of all the different tools and services we use.
The service is a Python-driven HTTP-based API. It’s written in the Django framework, and web requests are served by gunicorn and nginx. We also have a collection of asynchronous jobs, like issuing post-event rewards, which we run with APScheduler.
Our data is stored in PostgreSQL, which we chose for its focus on data integrity and ability to make almost every imaginable schema migration without incurring downtime. We use PgBouncer for connection pooling; when we were building the service, Django didn’t have any tools for connection pooling. Since then, we’ve been impressed with PgBouncer’s flexibility: it helped us swap our master database out with a new instance with just seconds of downtime.
We use Redis as a cache server; at first, we used Memcached, but Redis’s sorted set was a great fit for our speeding up our leaderboard API methods. Keeping our operational complexity to a minimum is very important to us, so we moved all our cached data to Redis soon after setting it up.
Our servers are running in Amazon AWS; daily content patches are distributed via their CloudFront CDN, and our database backups and cloud saves are both stored on S3. We’ve written Fabric scripts to automate the set-up and maintenance of our AWS instances.
We use Stackdriver to keep track of our servers’ status, and make graphs of our service. We’ve also set up Stackdriver to alert us when certain metrics cross thresholds (from nginx writing connections to PVP battles fought per second), which has been a great tool for sending relevant instructions to the ops team as soon as something goes wrong. StatsD helps us aggregate metrics from our web servers, before passing them to Stackdriver. We watch our uptime with Pingdom, in case something extremely bad happens. And finally, we use Monit to check the health of individual instances.
If you have any questions about how we decided on any of these tools, let us know in the comments; we’d be happy to talk about it!