# Talk at Full Stack Fest: Multi-DC

2016-09-05 · Recording: https://www.youtube.com/watch?v=7UyDK2bDjc4 · Slides: https://speakerdeck.com/sirupsen/full-stack-fest-2016-shopify-in-multiple-datacenters · archived copy: https://archive.sirupsen.com/recordings/full-stack-fest.mp4

_Lightly edited automatic transcript: speech-to-text, cleaned up by an LLM. It may contain errors; the recording is the source._

## Introduction: ten years of Shopify infrastructure

<!-- t=0 -->
Test. All right, can everyone hear me? I'm going to talk about a bit of a journey through Shopify's infrastructure. This is the kind of talk I would love to have seen from companies like Twitter, Facebook, or some of the others, about how their infrastructure got to where it is today. Google has some of these talks, but they describe what Google looked like in 2001 and then what it is today, and today it's so complicated that I don't think anyone can understand how it all fits together. It's really interesting to see how infrastructure evolved. So what I want to talk about, as Rachel said, is the past ten years of a very specific history of one platform and how we got to the scale we're at today.

<!-- t=57 -->
I work at a company called Shopify. We help people sell things, whether you're selling online, on your phone, or wherever. In the past ten years we have grown a fair amount, and I want you to keep these numbers in the back of your head during this talk, because they're relevant. We do tens of thousands of requests per second. We can do a massive amount of checkouts. We've served a lot of transactions and a lot of money. Downtime is very costly for us and our merchants. And in the past couple of years we've also grown tremendously in the number of shops we serve.

## The flash sale problem

<!-- t=100 -->
One particular problem dictates everything we do on the infrastructure team at Shopify, which we call the flash sale problem. We have merchants who can drive a completely disproportionate amount of traffic to us, and this creates massive problems. From one second to the next there might be a 2x or 3x difference in traffic because of some launch of this guy's shoes, or because of Super Bowl ads that drive a massive amount of traffic. For all the Europeans, it's a big North American event that doesn't matter that much to you. If you've ever tried to visit one of the websites advertising during the Super Bowl, they often have problems. We have merchants that advertise in the Super Bowl every year at this point. Not tons, but one, two, three. I think three last time.

<!-- t=157 -->
Every good story has a villain, and this is the story of our infrastructure. Currently our villain is Kylie Jenner. She's an American celebrity who sprang out of the Kardashian family, which is famous for being famous. She sells lipstick, and when people can't access this lipstick, they get fairly mad. That's a big problem for us. It hurts our merchants' trust, because Shopify is a white-label experience. Customers have no idea that it's Shopify's fault they're down due to our scale, and not Kylie's fault. There are even episodes of the Kardashian show where Shopify is down while they're filming. It's a mess. You can find it later, I'm sure.

<!-- t=212 -->
So we're dealing with problems of scale. We've grown a lot in the past ten years, and I want to tell you the story of the evolution of Shopify's infrastructure, what we tackled, and how we're getting to the point where, in the next two months, we're going to be running out of multiple data centers. Ten years ago, we weren't even thinking about that.

## Single tenant vs. multi-tenant

<!-- t=231 -->
The way I think about Shopify is on a spectrum from single-tenant platform to multi-tenant platform. There are benefits on both sides. If you're single tenant, you're running one shop. On a multi-tenant platform, in the most naive implementation, you have one database for all the shops.

*(Slides cut off briefly.)* So I have to be the guiding hand here a little on the sides. Single tenant is on one side, multi-tenant on the other, and this is a spectrum. If you have a completely single-tenant solution with one database per shop, you have some advantages. If you have a multi-tenant solution where everybody runs on one database, you have other advantages. There are really a bunch of spectrums hidden in this.

<!-- t=282 -->
If you have a multi-tenant platform, you have a large amount of capacity: a massive database. When one shop has a big sale, it can steal a bit of capacity from the other ones, and later, when another shop has a big sale, it can take some from the other shops. With a single-tenant platform, that becomes expensive because you need to provision a database for each one. The secondary benefit of multi-tenant is good utilization. If you have many stores running on one database, you can utilize that database well by sharing resources between the shops. With one store per set of hardware, you don't have that.

<!-- t=322 -->
So multi-tenant gives us a platform that is good for flash sales. Single tenant is not quite as good for flash sales, because a single store taking a massive amount of traffic is very costly per shop. There are some companies out there that are really good at that. Multi-tenant is also cheaper, because you leverage the network effect of all these shops sharing costs. Single tenant is expensive: you have to care about hardware for every single shop. We have 300,000 shops, and 300,000 databases is way too many to manage.

<!-- t=358 -->
There are also benefits of single tenant. You have complete isolation: if Kanye's shop completely breaks, it doesn't break anybody else's shop, whereas in a multi-tenant solution you have that problem. Scalability is also great for single tenant, because you can just make that one database bigger if the shop grows big enough.

<!-- t=379 -->
These are the extremes of the spectrum, and some of the multi-tenant side doesn't even get a green mark. In the middle there's some kind of middle ground that tries to harvest the best of both worlds. That's what I've been exploring for the past three years: how do we get the best of the single-tenant platform and the best of the multi-tenant platform in one platform? I refuse to accept that we just have to be multi-tenant and give up everything on the left. I want the best of both. I want the entire chart to be green. Over this talk, we'll cover how to achieve that.

## 2004–2012: from snowboards to vertical scaling

<!-- t=417 -->
Going back to the spectrum, in 2004 our founders created a snowboard store, and that was the first Shopify store. They created the platform to sell snowboards, and it looked like this: classic mid-2000s web design. They launched it, but realized their business wasn't in selling snowboards. In 2004, e-commerce was reviving after the dot-com bubble, and there was a severe lack of good, customizable e-commerce software. So they pivoted, and two years later they created a multi-tenant platform to support many snowboard stores, and many stores in general. Shopify became a multi-tenant platform. Instead of this Ruby on Rails app hosting just one store, it could now host as many as it could until the database broke.

<!-- t=467 -->
So where is that golden middle, where we get the best of the single-tenant side, with one snowboard store that could never be impacted by others, and the big platform that supports all of them? From 2006 to 2012 we moved a little along that spectrum. We did a lot of work here and there to optimize the application, to get a little more of the benefits of single tenant and a little less of the drawbacks of multi-tenant.

<!-- t=500 -->
For the first six years of Shopify's history, we did what most companies do. We focused on our product, and every couple of months somebody would go and optimize performance. There's a lot of low-hanging fruit. If you profile the application, you might get 5 or 10% of your CPU time back from some N+1 somewhere, or from operations you don't need to do. You can do a lot of vertical scaling, meaning you can just increase the size of your database, and all of these simple things. You can do a lot of caching, and our operation is very cache-heavy, because we have a lot of traffic to stores that don't involve write operations. We did a lot of this for the first six years.

## 2013: database sharding

<!-- t=543 -->
But at some point you reach a limit, and we reached that ceiling in 2012. There's a legendary picture of our CTO passed out, face-planted on the floor, after having worked for months to try to support one of the biggest stores at the time. We had reached a point where we couldn't vertically scale anymore, because at the end of the day you can't cache a write. At some point you have to scale horizontally.

<!-- t=583 -->
So in 2013 we started working on database sharding. Before, our infrastructure looked something like this: some load balancers, some workers that did the work, a database, some Redis, and a cache, all working together. In 2013 we split the database into multiple shards so that the write traffic could be spread out. The reason write traffic matters is that if you're trying to sell something, and sell a lot of it, we need to do checkouts, create orders, and take payments. There are a lot of writes that have to happen and be consistent, and that's one of our big challenges. All the deadlines of our team revolve around Black Friday and Cyber Monday, which is a big American holiday where Americans shop more than usual.

<!-- t=633 -->
That worked really well for a long time. Until then, the write capacity, or the scale capacity, of Shopify had been the biggest threat to our existence. We were terrified of these customers taking us down. On the other hand, we didn't want to say "no" to customers who were driving a lot of traffic. We could have said, "We're not going to be a platform where you can do this kind of thing," but instead we decided we want to be the only platform in the world that can support these kinds of sales.

## 2014: resiliency

<!-- t=658 -->
By 2014, the biggest threat to our existence was something else. We were pretty big at this point: hundreds of servers in our data centers, tens of data stores, and all of this kind of stuff, and many of these had become single points of failure. It might sound embarrassing that just two years ago we had single points of failure in our infrastructure, but it wasn't a priority, because the threat of celebrities selling was much higher than a single database blowing up.

<!-- t=686 -->
So we started working on resiliency. The way I like to think of it is that in chemistry, if you have a big surface area, you're going to have more reactions. It's the same with errors. If you have a lot of servers, there are inevitably going to be more errors. If you have one server, the probability of running into a kernel bug in a year is pretty low. If you're Facebook or Google with millions of servers, you're running into bugs all the time, and you have teams to solve them. As you get larger, you have to transition from a pets mentality to a cattle mentality. The failures are proportional to the surface area of your platform, and we started reaching that critical mass in 2013 and 2014, where they were disproportionately creating failures.

<!-- t=746 -->
So we created a pyramid model where we had to think about the certain errors that were happening in our platform and stop them from propagating downwards. We didn't have any idea what would happen if one of the components at the bottom blew up. So we started mapping out all the single points of failure. We made sure that if the caches were down and we hit a store, it would be fine, and that if the database was down and we hit a store, it would be fine. We wrote test cases for all of this and began fixing them. We also wrote a proxy that sits between your application and its data sources in development and test, so you can say, "Hey, take down this database and see what happens." We mapped out the entire application and all the single points of failure, and did this in 2014.

## 2015: multiple data centers

<!-- t=801 -->
So in '13 we sharded the database, in '14 we did a lot of resiliency work, and in '15 we started working on multiple data centers. Up to that point we ran out of only one data center. The risk of that data center going out, and of being able to fail over to another, was not as high as all the other things. In the worst case we would have to provision a new data center and be down for a couple of days, but it wasn't the biggest risk at the time. In 2015 it became the highest priority, and we started setting up a second data center.

<!-- t=835 -->
What this boils down to is that you have one data center that runs all the shards of Shopify, then another data center, and you set up replication from one to the other and somehow move all the traffic. We did a lot of work to go from one data center to two, and then we built a script that can run the entire failover process with about a minute of downtime on Shopify. Sometimes when you see a maintenance window on Shopify for a database failover, that's me running this script to fail over the entire data center in a minute.

<!-- t=873 -->
It has a couple of steps. It updates a service discovery layer and says, "Okay, now the data center is going to move." Then it takes checkout down. If you're going through checkout while we're moving data centers, it will show you an error: "Hey, please come back later." Most stores don't even appear to be down unless you try to perform a write. Then we stop everything in one data center and start everything back up in the other. We tune our load balancers so that all the traffic goes to the old data center and is proxied to the new one, while we announce the IPs out of the new data center, and everything is moved. This takes very little downtime. You end up in a situation where data center two is now the primary and is replicating in the other direction, and we can always fail back.

<!-- t=923 -->
We've done this exercise probably about ten times. The first time we did it, we were all sitting in a room, about 20 different people looking at 20 different dashboards, and it was what I imagine happens after a space mission, when everybody hugs each other afterwards. People were in Hangouts, and it was a big thing. Now it's two or three people running the script, and it's still a bit bigger a deal than I want it to be, but a small thing for a company our size at this point.

## 2016: pods

<!-- t=955 -->
So we got really good at this multi-DC thing. Then in 2016 we started working on a concept called pods. The idea is to take everything we learned from 2006 to 2012 about running a really good application and optimizing its performance, everything we learned about database sharding, and everything we learned about resiliency in 2014 and 2015, and see if we can marry all of these concepts in a way that makes sense.

<!-- t=1001 -->
What we came up with is the idea of a pod, which takes sharding further. Until then we sharded only the database, because that was the most critical thing at the time. The pod takes that to the extreme. We take the workers for a group of shops, and we take Redis and isolate it for that group of shops. So multiple components of Shopify exist at a smaller size. The goal is that if we have these completely isolated units of Shopify, we can run them in multiple data centers. One can run in Asia, Europe, North America, India; it doesn't matter. One shop does not need to talk to another shop.

<!-- t=1040 -->
There are some exceptions. If you're talking to the API, you don't want to have to care about shards or any of that. Or if you're a partner with shops in Asia and shops in North America, you don't want to care about that. So there are all kinds of global challenges to this.

<!-- t=1059 -->
2015 left us with an architecture like the one on the slide, and we wanted to move to something more like this one, with self-contained pods, self-contained deployments of Shopify. Instead of managing one big Shopify, we manage a lot of small Shopifys. Then it doesn't matter where these things run. One can run in data center one in North America and another in Europe. All the North American ones can be in North America, and we can have multiple data centers there, replicate them, and fail over pods independently. Then we have this nice infrastructure.

<!-- t=1097 -->
Returning to the spectrum chart from before, we've arrived at something really nice. We haven't quite reached the middle yet, but we've gotten a lot better. On the capacity spectrum, we have much better capacity than before. The multi-tenant solution doesn't lend itself very well to running multiple data centers, but now we can do that. If we wanted to spin up Google Cloud pods of Shopify, we can do that. If we want to go onto Amazon or start experimenting with the cloud, we can do all of that. Utilization is much better. Flash sales are better because of the capacity. It's cheaper because we can have vendors bet against each other. And now you have groups of shops, so if one blows up, the blast radius is limited to that pod and not the entirety of Shopify. It's also better because we can run out of multiple data centers and scale independently.

<!-- t=1151 -->
So we're not quite there yet. Not everything is green, but it looks a lot better.

## Flash sales and sharing workers between pods

<!-- t=1155 -->
Flash sales are one of the prerequisites I set up before, and it's what everything comes back to on the infrastructure team. What limits us during flash sales is the CPU of the workers, the Ruby on Rails workers that serve the application. So, on top of some of our resiliency work, we've written it so that workers can be shared between pods. If one pod is experiencing a massive sale, it can take some workers from some of the less busy ones, serve the traffic, and give them back to the other pods. This works really well, and we get the best of both worlds.

## Sorting Hat: routing requests to pods

<!-- t=1202 -->
Let's see what that looks like. Say you make a request: you've got these beautiful shoes on myshop.com. That resolves to our IPs, which are announced out of multiple data centers, and you reach our network. When you reach the network, you reach software called Sorting Hat. What Sorting Hat does is look at the request, think a little bit, and decide which pod the request is destined for, and then route you to the appropriate data center.

<!-- t=1234 -->
So your request goes to one data center and enters our network. The load balancer module looks at the request and decides this request is for pod two. Is pod two local to this data center and active in this data center? If yes, you just send it to the local pod. If not, you proxy it somewhere else. If you're going to Amsterdam, you might be proxied to, say, Virginia, and from there it's all local and fine. That's the multi-DC strategy in a nutshell.

<!-- t=1266 -->
I want to give a shout-out here to OpenResty. Sorting Hat is written as an OpenResty module, and we use Lua for this. It's some of the best software we run, in my opinion. Nginx is extremely stable, and we've found very few bugs in it. With OpenResty you can customize the load balancer. We implemented exponential load balancing, and you can do things like serve different SSL certs. In our case, all the customers point to the Shopify IP, but based on the hostname we can serve them the appropriate certificate dynamically out of a database. You can do anything at the database layer. If you're running Docker containers, you can route to the containers from a service discovery layer. What we're doing in Sorting Hat is going into the request, looking at it, and routing it based on what the request looks like. That's the Sorting Hat module.

## The two rules for running in pods

<!-- t=1329 -->
If we have this architecture, we need to set up some rules for what Shopify can do to run in this setup.

<!-- t=1335 -->
Rule number one is that any request that comes into Sorting Hat needs to be annotated with wherever it's going. Sorting Hat needs to decide which shop the request belongs to. If you're going to a data center in Chicago and Sorting Hat in Chicago can't figure out which shop the request belongs to, it just has to error. It doesn't know whether to send it to Amsterdam or Asia; it has no idea. So every request needs to honor that rule.

<!-- t=1366 -->
Rule number two is that a request can only touch one pod. If it touches multiple pods, you might be making a request to Asia and North America and everything in the same request, and that's a mess. It has really bad consequences for resiliency, because these now rely on multiple deployments of Shopify.

<!-- t=1386 -->
Requests that violate the first rule could look like this: you get a webhook from a payment provider, and instead of a shop domain, you'd have to figure out which shop the request is going to by looking at the URL. A problematic request for the second rule might be one where, in the past, we went over every single shop and tried to count all of them, or had some operation to upgrade all the themes across all the Shopify stores on all the different deployments of Shopify. We can't have these any longer. In the first case, we need to teach the load balancer how to reroute that URL, and in the second, teach Shopify not to reach multiple shards in the same request.

## Shit list driven development

<!-- t=1431 -->
So we came up with something called shit list driven development. To honor these two rules, we had 500, if not 1,000, different code paths that violated them. Our infrastructure for 12 years had relied on the underlying assumption that you could do these two things. You didn't have to care. Shopify could just figure it out at runtime when a request came in.

<!-- t=1456 -->
With this overwhelming list, we came up with a primitive to create a deprecation API. Usually, with deprecation, when you use the API something yells at you, but you can still use it if you want to. With shit list driven development, instead of spitting out an error message, we whitelisted all the existing usage of that API, which gives you a big checklist of everything that needs to be fixed. If you violate it with anything new, it raises an exception. So it looks something like this: if the shit list includes your class or request, then just do it. Otherwise, print out a friendly message saying to come talk to us, and we'll figure out how to do it together.

<!-- t=1492 -->
The point is that we stop the bleeding of new violations of these multi-pod rules and then go and fix the existing ones. We're left with this massive checklist of everything we need to fix, but there's a feedback loop: we kill them one at a time. This refactor is coming up on taking a year for us, so it's really helpful to have a very tangible success metric.

## Routes and the Sorting Hat JSON

<!-- t=1510 -->
For our routes, the shit list might look something like this. The shit list is your routes file, so here are all your different routes. In Shopify we probably have 500 of these, but this is just a subset, and you can see that some are a bit problematic. Take things like sign-up. That doesn't inherently belong to one pod, so you have to go to every single pod and ask, "Hey, do you want to sign up this store? It's European and fresh and whatever." All the pods respond, "I'll take it, I'll take it," and the load balancer picks a random one. Take recovering your password: we have to ask every single pod, because we don't want to leak what they belong to.

<!-- t=1555 -->
All of these are on the shit list, so we added syntax on top of the Rails default route syntax to have a routing method. In the first case, we need to extract the shop from the param. In the third case, we need to try all the different pods and aggregate the results somehow. And in the last case, if you visit a store, we can extract the shop, and therefore know the data center, from the host parameter in the HTTP config.

<!-- t=1588 -->
So we needed to teach the load balancer about all of these Rails routes. What we did was parse the entire AST that the Rails routes generate and turn it into JSON that our module could read. The Rails routes are serialized into JSON and then loaded into Sorting Hat, the module that sees the request and routes it to the appropriate data center. It has 500 or 1,000 different routes, and when it sees a route that matches the regex, it knows the routing method, so it knows which pod the request is going to. A vast majority of them are shop-from-host, but there are a ton of others that don't go through that.

<!-- t=1642 -->
To reiterate the diagram: you get a request to Sorting Hat, Sorting Hat looks at this giant JSON file that is generated from the routes of the web app, and it knows which pod the request belongs to. Because every request goes through Sorting Hat and every request only touches one pod, you have the isolation to be able to run in multiple data centers.

<!-- t=1667 -->
I think it's really neat how all the different efforts over the years play together. We get resiliency through isolation, but we also get isolation through resiliency. We get scalability through all these different things, and the multi-DC setup we've achieved really comes through isolating Shopify into many, many small Shopifys that can run independently.

## The future

<!-- t=1689 -->
A couple of thoughts on the future, because we haven't quite reached the middle yet. From the spectrum before, not everything was green in the middle. We're pushing the boundary further and further towards the center, but we're not quite there. Maybe 2017 is when we'll reach it.

<!-- t=1705 -->
Two of the things I would like to work on in 2017 start with zero-downtime failover. When I fail over a pod, I don't want any customer impact whatsoever. What we're thinking about doing is, when we fail over a pod from one data center to another, telling the load balancers to pause all the requests. Then we move everything, resume, and let the requests flow through. This means there's not really any disruption for customers. If you're doing a checkout during a pod failover, it will be slow but still succeed, and nothing will be dropped.

<!-- t=1740 -->
The other thing is isolating the shops further. We still have the problem that one store blowing up will propagate into the rest of its pod. I would like to put more constraints on single shops so that doesn't happen as much, by looking at past disruptions. That's what we've been looking at for 2017.

<!-- t=1756 -->
And in just about one or two months, my team will hopefully launch Shopify in multiple data centers with this pods approach, in time for Black Friday and Cyber Monday. If we don't — well, fuck. Thank you so much.

*[Applause]*

## Q&A: transactions in sharded SQL

<!-- t=1758 -->
**Host:** Thank you so much, Simon. We have a couple of questions on Twitter. You have seven minutes, so if you still want to ask Simon a question, send it through now with #askSFS. The first question is: how do you handle transactions in the sharded SQL?

<!-- t=1788 -->
**Simon:** I think what's implied by that question is how we handle transactions across multiple shards in this setup, so that's how I'm going to interpret it. We have this nice property in Shopify that one shop doesn't care about another shop, so the transactions are isolated. If you have inventory counts or products or things like that, you're not going to reach across other shards, because you can't do transactions across multiple shards. There are multiple databases, multiple sessions.

<!-- t=1819 -->
There's one lengthy talk in itself here, and it's what I've spent the last four months on. There's a master database that hosts tables that don't belong directly to a shop: things like billing, partners, API, things that don't inherently belong to one shop. You can't do transactions between a shard and that either. So we do best effort: we put all the transactional things back into the shop and backfill them into the master. Basically, the answer is to avoid cross-shard transactions everywhere you can. If you can't, you need to refactor your code or just live with the drawbacks of not having transactions across data stores.

<!-- t=1860 -->
**Host:** Yeah. Transactions. Hard.

<!-- t=1862 -->
**Simon:** Yeah. Don't do sharding if you can avoid it.

## Q&A: microservices

<!-- t=1866 -->
**Host:** The next one: did you extract microservices from your app, and did that help with the scaling?

<!-- t=1873 -->
**Simon:** Yes. This is something we debate a lot internally as well. Without going on too lengthy a rant here, I think microservices is overcompensation. I think good object-oriented programming can get you out of most of these problems, and I don't think you need to enforce a TCP/IP boundary to have good software.

<!-- t=1895 -->
That said, there are some applications where parts of your software belong in different apps. The example at Shopify is that we have these sharded tables where the shop has its data, and then also API, partners, billing, and things like that that belong in a global database. We want to extract these into services, because Shopify should only care about the shop. Billing should live in its own thing, partners should live in its own thing, and API should live in its own thing, because these three things don't belong to a single shop. And with API, I don't mean the actual API for retrieving your products or performing a checkout, because that still belongs to a store. I mean things like authentication.

## Q&A: pod deployment and Kubernetes

<!-- t=1939 -->
**Host:** How do you handle pod deployment?

<!-- t=1943 -->
**Simon:** The way we handle pods right now is we just deploy to all of them at once. In the future I would like to do canary deploys. Right now we deploy to all of them at once, and that works fine. But something I would like to do over the next year or two is deploy to one pod at a time, because it minimizes risk when we have all our developers able to deploy all day.

<!-- t=1966 -->
**Host:** All right. And the last question: are you using Kubernetes?

<!-- t=1972 -->
**Simon:** I don't think that's public yet.

<!-- t=1978 -->
**Host:** Okay. Well, one more round of applause for Simon. Thank you so much.

*[Applause]*
