# Talk at Railsconf: 5 Years of Rails Scaling to 80K RPS

2017-04-27 · Recording: https://www.youtube.com/watch?v=4dWJahMi5NI · Slides: https://speakerdeck.com/sirupsen/railsconf-2016-5-years-of-scaling-rails-to-80k-rps · archived copy: https://archive.sirupsen.com/recordings/2017-railsconf.mp4

_Lightly edited automatic transcript: speech-to-text, cleaned up by an LLM. It may contain errors; the recording is the source._

## Introduction and Inspiration

<!-- t=13 -->
My name is Simon. I work on the infrastructure team at Shopify, and today I'm going to talk about the past five years of scaling Rails at Shopify. I've only been around at Shopify for four years, so the first year was a little bit of digging. But I want to talk about all of the things that we've learned. I hope that people in this audience can place themselves on this timeline and learn from some of the lessons that we've had to learn over the past five years.

<!-- t=42 -->
This talk is inspired by a guy called Jeff Dean from Google. He's a genius, and he gave a talk about how they scaled Google for the first couple of years. He showed how they ran with a couple of MySQLs, they sharded, they did all the stuff. They did all the NoSQL paradigm, and finally went to the NewSQL paradigm that we're now starting to see. This was really interesting to me because you saw why they made the decisions that they made at that point in time. I've always been fascinated by what made, say, Facebook decide that now was the time to write a VM for PHP to make it faster.

<!-- t=79 -->
So this talk is about that. It's an overview of the decisions we made at Shopify and less about the very technical details of all of them. It's to give you an overview and a mental model for how we evolve our platform. There's tons of documentation out there on all of the things I'm going to talk about today: other talks by coworkers, blog posts, READMEs and things like that. So I'm not going to get into the weeds, but I'm going to provide an overview.

## Shopify at a Glance

<!-- t=108 -->
I work at Shopify, and at this point you're probably tired of hearing about Shopify, so I'm not going to talk too much about it. Overall, Shopify allows merchants to sell things to other people, and that's relevant for the rest of this talk. We have hundreds of thousands of people who depend on Shopify for their livelihood. Through this platform, we run almost 100K RPS at peak. The largest sales hit our Rails servers with almost 100K requests per second. Our steady state is around 20 to 40K requests per second, and we run this on tens of thousands of workers across two data centers. About $30 billion have made it through this platform, which means that downtime is costly, to say the least.

<!-- t=159 -->
You should keep these numbers in the back of your head, as they're the numbers we've used to get to the point that we are at today. Roughly, these metrics double every year. So if you go back five years, you just have to cut them in half five times.

<!-- t=175 -->
I want to introduce a little bit of vocabulary for Shopify, because I'm going to use it loosely in this talk. Shopify is at least four sections:

<!-- t=187 -->
- **Storefront.** This is where people are browsing their collections, browsing their products, adding to the cart. This is the majority of traffic: somewhere between 80 and 90% of our traffic is people browsing storefronts.
- **Checkout.** This is where it gets a little more complicated. We can't cache as heavily as we do on the storefront. This is where we have to do writes, decrement inventory and capture payments.
- **Admin.** This is more complex. You have people who apply actions to hundreds of thousands of orders at the same time, concurrently. You have people who change billing and need to be billed. These things are much more complex than both checkout and storefront in terms of consistency.
- **API.** It allows you to change the majority of the things you can change in the admin. The only real difference is that computers hit the API, and computers can hit the API really fast. Recently I saw an app for people who wanted to have an offset on order numbers, starting at a million. This app will create a million orders and then delete them all to get that offset. So people do crazy things with this API, and it is our second largest source of traffic after the storefront.

## Flash Sales Shape the Platform

<!-- t=264 -->
I want to talk a little bit about a philosophy that has shaped this platform over the past five years. Flash sales are really what has built and shaped the platform that we have. When Kanye wants to drop his new album on Shopify, it is the team that I am on that is terrified.

<!-- t=282 -->
We had a fork in the road five years ago. That was when we started seeing customers who could drive more traffic for one of their sales than the entire platform otherwise was serving. They would drive multiples: if we were serving a thousand requests per second for all the stores on Shopify, some of these stores could get us to 5,000. And this happens in a matter of seconds. Their sale might start at 2 PM, and that's when everyone comes in.

<!-- t=311 -->
So there's a fork in the road. Do we become a company that supports these sales? Or do we kick them off the platform, throttle them heavily and say this is not the platform for you? That's a reasonable path to take. 99.9-something percent of the stores don't have this pattern. They can't drive that much traffic. But we decided to go the other route. We wanted to be a company that could support these sales, and we formed a team to solve this problem of customers that could drive enormous amounts of traffic in a very short amount of time.

<!-- t=346 -->
I think this was a fantastic decision. It happened exactly five years ago, which is why the timeframe of the talk is five years. It was powerful because it has served as a canary in the coal mine. The flash sales we see today, and the amount of traffic they can drive, say 80K RPS, are what the steady state is going to look like next year. So when we prepare for these sales, we know what next year is going to look like, and we know we'll be ready for it because we're already working on the problem. They help us stay one to two years ahead.

<!-- t=384 -->
In the meat of this talk I'll walk through the past five years of the major infrastructure projects we've done. These are not the only projects we've done. There have been other apps and many other efforts, but these are the most important to the scaling of our Rails application.

## 2012: Optimization and Load Testing

<!-- t=396 -->
2012 was the year where we sat down and decided to go the antifragile route with flash sales. We were going to become the best place in the world to have flash sales. A team was formed whose sole job was to make sure that Shopify as an application would stay up and be responsive under these circumstances.

<!-- t=420 -->
The first thing you do when you start optimizing an application is try to identify the low-hanging fruit. In many cases the low-hanging fruit is very application dependent. The lowest-hanging fruit from an infrastructure side is already harvested in your load balancers, in Rails (which is really good at this), or in your operating system. They take all of the generic optimization and tuning, so at some point that work has to be handed off to you. You have to understand your problem domain well enough to know where the biggest wins are.

<!-- t=451 -->
For us the first ones were things like backgrounded checkouts. And this sounds crazy: what do you mean they weren't backgrounded before? Well, the app was started in 2004, 2005. Back then, backgrounding jobs in Ruby or Rails was not really a common thing, and we hadn't done it after that either because it was such a large source of technical debt. So in 2012 a team sat down and took on that massive amount of technical debt to move the checkout process into background jobs. Payments were captured not in a request that took a long time, but asynchronously in jobs. This was of course a massive source of speedup. Now you're not occupying all these workers with long-running requests.

<!-- t=495 -->
Another domain-specific problem was inventory. You might think that inventory is just decrementing one number, and doing it really fast if you have thousands of people. But MySQL was not good at that. If you're trying to decrement the same number from thousands of queries at the same time, you will run into lock contention. So we had to solve this problem as well. And these are just two of many problems we solved.

<!-- t=517 -->
In general, what we did was print out the debug logs of every single query on the storefront, on the checkout, and on all of the other hot paths, and basically start checking them off. I couldn't find the original picture, but I found one from a talk that someone from the company did three years ago, where you can see the wall where the debug logs were taped. The team at the time would go and cross off queries, write their names on them, and figure out how to reduce them as much as possible.

<!-- t=542 -->
But you need a feedback loop for this. We couldn't wait until the next sale to see if the optimizations we'd done actually made a difference. We needed a better way than just crashing at every single flash sale; we needed a tighter feedback loop. Just like when you run the tests locally, you know right away whether it worked. We wanted to do the same for performance. So we wrote a load testing tool. It simulates a user performing a checkout. It goes to the storefront, browses around a little bit, finds some products, adds them to its cart and performs the full checkout. It fuzz tests this entire checkout procedure, and then we spin up thousands of these in parallel to test whether a performance change actually made any difference. This is now so deeply woven into our culture that whenever someone makes a performance change, people ask, "Well, how did the load testing go?" This was really important for us. A siege just hitting the storefront, running a bunch of the same request, is not a realistic benchmark of realistic scenarios.

## Identity Cache

<!-- t=604 -->
Another thing we did at this time was write a library called Identity Cache. We had one MySQL at this point hosting tens of thousands of stores. When you have that, you're pretty protective of that single database, and we were doing a lot of queries to it. Especially the sales were driving such a massive amount of traffic at once to these databases. So we needed a way of reducing the load on the database.

<!-- t=631 -->
The most common way of doing this is to start sending queries to the read slave. You have databases that feed off the one that you write to, and you start reading from those. We tried to do that at the time. It has a lot of nice properties compared to the other method. But back in the day there weren't any really good libraries in the Rails world. We tried to fork some and figure something out, but we ran into data corruption issues. Mismanagement of the read slaves was really problematic at the time because we didn't have any DBAs. And mind you, this was a team of Rails developers who just had to become infrastructure developers and understand all of this stuff. They learned it on the job, because we had decided to handle flash sales the way that we did. So we just didn't know enough about MySQL and these things to go that path.

<!-- t=688 -->
So we decided to figure out something else. Deep inside of Shopify, Toby had written a commit many, many years ago introducing the idea of an identity cache: managing your caches out of band in memcached. The idea is that if I query for a product, I look in memcached first and see if it's there. If it's there, I just grab it and don't even touch the database. If it's not there, I put it there so that it will be there for the next request. And every time you do a write, we just expire those entries.

<!-- t=719 -->
This has a lot of drawbacks, because that cache is never going to be 100% what is in the database. So when we read from that managed cache, we never write that back to the database. It's too dangerous. That's also why the API is opt-in: you have to use `fetch` instead of `find` to use IDC, because we only want to do it on these paths. It returns read-only records so you can't change them and corrupt your database.

<!-- t=744 -->
The massive downside with either read slaves or identity cache for something like this is that you have to deal with what you're going to do when the cache is expired or old. This is what we decided on at the time. I don't know if it's what we would have done today. Maybe we would have; we've gotten much better at handling read slaves, and they have a lot of other advantages, such as being able to do much more complicated queries. But this is what we did at the time. And if you're already having severe scaling issues, Identity Cache is a very simple thing to do and use.

## 2013: Sharding

<!-- t=779 -->
After 2012, which was probably our worst Black Friday and Cyber Monday ever because the team was working night and day to make this happen, there's this famous picture of our CTO face-planted on the ground after exhausting work of scaling Shopify at the time. And someone then woke him up and told him, "Hey dude, checkout is down." (laughter) We were not in a good place. But Identity Cache, load testing and all this optimization saved us.

<!-- t=808 -->
Once the team had decompressed after this massive sprint to survive these sales and survive Black Friday and Cyber Monday, we raised the question of how we could never get into this situation again. We had spent a lot of time optimizing checkout and storefront, but this is not sustainable. If you keep optimizing something for so long, it becomes inflexible. Often fast code is hard-to-change code. If you optimize the storefront and checkout and have a team that only knows how to do that, there's going to be a developer who comes in, adds a feature and adds a query as a result. This should be okay. People should be allowed to add queries without understanding everything about infrastructure. Often the slower thing is more flexible. Think of a completely normalized schema. It is much easier to change and adapt, and that's the entire point of a relational database. But once you make it fast, it often becomes a trade-off of becoming more inflexible.

<!-- t=865 -->
Think of an algorithm like bubble sort, which is O(n²). You can make that really fast. You can make it the fastest bubble sort in the world. You can write a C extension in Ruby that has inline assembly, and this is the best bubble sort in the world. But my terrible implementation of quicksort, which is O(n log n), is still going to be faster. So at some point you have to stop optimizing, zoom out and re-architect.

<!-- t=896 -->
That's what we did with sharding. At some point we needed that flexibility back, and sharding seemed like a good way to get it. We also had the problem that Shopify is fundamentally an application with a lot of writes. During these sales there are going to be a lot of writes to the database, and you can't cache writes. So we had to find a way to handle that, and sharding was it.

<!-- t=917 -->
We built this API. A shop is fundamentally isolated from other shops. It should be: shop A should not have to care about shop B. So we did per-shop sharding, where one shop's data is all on one shard, and another shop's data might be on another shard, and yet another might be together with the first one. This is all the sharding API internally exposes. Within that block it will select the correct database for that shop's products. Within that block you can't reach the other shard; that's illegal.

<!-- t=949 -->
In a controller, this looks something like this. At this point most developers don't have to care about it. It's all done by a filter that finds the shop on another database and wraps the entire request in the connection that shop is on. Any product query will then go to the correct shard. This is really simple, and it means that the majority of the time developers don't have to care about sharding. They don't even have to know of its existence. It just works like this. Jobs work the same way.

<!-- t=977 -->
But it has drawbacks. There are tons of things that you now can't do. I talked about how with optimization you might lose flexibility, but with architecture you lose flexibility on a much grander scale. Fundamentally, shops should be isolated from each other. But in the few cases where you want them not to be, there's nothing you can do. That's the drawback of changing the architecture. For example, you might want to do joins across shops. You might want to gather some data, or run an ad hoc query about app installations across shops. This might not seem like something you would need to do, but the partners interface, for all of our partners who build applications, actually needs to do that. They need to get all the shops and the installations for them. So it was written as something that did the join across all the shops and listed them, and this had to be changed. The same went for our internal dashboard that did things across shops, like finding all the shops with a certain app. You just couldn't do that anymore, so we had to find alternatives.

<!-- t=1038 -->
If you can get around it, don't shard. Fundamentally Shopify is an application that will have a lot of writes, but that might not be your application. It's really hard, and it took us a year to do and figure out. We ended up doing it at the application level, but there are many different levels at which you can shard. If your database is magical, you don't have to do any of this. Some databases are really good at handling this stuff, and you can make some trade-offs at the database level so you don't have to do this at the application level. But there are really nice things about being on a relational database: transactions, schemas, and the fact that most developers are familiar with them are massive benefits. And they're reliable. They've been around for 30 years, so they're probably going to be around for another 30 years at least.

<!-- t=1085 -->
We decided to do it at the application level because we didn't have the experience to write a proxy, and the databases we looked at the time were just not mature enough. I actually looked at some of the databases we were considering at the time, and most of them have gone out of business. So we were lucky that we didn't buy into proprietary technology, and solved it at the level we felt most comfortable with. Today we have a different team and might have solved this at a proxy level or somewhere else, but this was the right decision at the time.

## 2014: Resiliency

<!-- t=1117 -->
In 2014 we started investing in resiliency. You might ask what resiliency is doing in a talk about performance and scaling. Well, as a function of scale, you're going to have more failures. This led us to a threshold in 2014 where we had enough components that failures were happening quite rapidly, and they had a disproportionate impact on the platform. When one of our shards was experiencing a problem, requests to other shards, and shops on the other shards, were either much slower or failing altogether. It didn't make sense that when a single Redis server blew up, all of Shopify was down.

<!-- t=1157 -->
This reminds me of the concept from chemistry where your reaction time is proportional to the amount of surface area you expose. If you have two glasses of water and you put a teaspoon of loose sugar in one and a sugar cube in the other, the loose sugar is going to dissolve in the water quicker because the surface area is larger. The same goes for technology. When you have more servers and more components, there are more things that could potentially fail and make it all fall apart.

<!-- t=1187 -->
If you have a ton of components and they are all tightly knit together in a web, where one failing drags a bunch of others with it, and you have never thought about this, adding a component will probably decrease your availability. And this happens exponentially. As you add more components, your overall availability goes down. If you have 10 components with four nines, you'll have a lot more downtime if they're tightly webbed together in a way that makes each of them a single point of failure. We hadn't at this point had the luxury of finding out what our single points of failure even were. We thought it was going to be okay. But I bet you, if you haven't actually verified this, you will have single points of failure all over your application where one failure will take everything down with it. Do you know what happens if your memcached cluster goes down? We didn't, and we were quite surprised to find out.

<!-- t=1244 -->
This means that you're only as good as your weakest point, your weakest single point of failure. If you have multiple single points of failure, multiply the probabilities of all of them together and you have the final probability of your app being available. Very quickly, what looks like hours of downtime per component becomes days or even weeks of downtime globally, amortized over an entire year. If you're not paying attention to this, adding a component will probably decrease your overall availability.

<!-- t=1274 -->
The outages look something like this. Your response time increases. This is a real graph of the incidents in 2014 where something became slow. As you can see here, the timeout is probably exactly 20 seconds, so something was being really slow and hitting a timeout of 20 seconds. If all of the workers in your application are spending 20 seconds waiting for something that's never going to return because it's going to time out, then there's no time to serve any request that might actually work. So shard one is slow, and requests for shard zero are going to be stuck behind the queue, because requests to shard one will never complete.

<!-- t=1313 -->
The mantra you have to adopt when this starts becoming a problem is that a single component failure cannot compromise the availability or performance of your entire system. Your job is to build a reliable system from unreliable components.

## The Resiliency Matrix, Toxiproxy and Semian

<!-- t=1332 -->
A really useful mental model for thinking about this is the resiliency matrix. On the left-hand side we have all the components in our infrastructure. At the top we have the sections of the infrastructure, such as admin, checkout and storefront, the ones I showed before. Every cell tells you what happens to that section if the component on the left is unavailable or slow. So if Redis goes down, is storefront up? Is checkout up? Is admin up?

<!-- t=1367 -->
When we drew this out, it was not what you'd hope; it was probably a lot worse. We were shocked to find out how red and blue, how down and degraded, Shopify looked when what we thought were tangential data stores like memcached and Redis took down everything along with them. The other thing we were shocked about was that this is really hard to figure out. Figuring out what all these cells and their values are is really difficult. How do you do that? Do you go into production and just start taking down stuff? How do you know? What would you do in development?

<!-- t=1399 -->
So we wrote a tool to help with this. It's called Toxiproxy. For the duration of a block, it emulates network failures at the network level by sitting in between you and the component on the left. This means you can write a test for every single cell in that grid. When you flip a cell from red to green, from bad to good, you can know that no one will ever reintroduce that failure. These tests might look something like: when some message queue is down, I get this section, and I assert that the response is a success. At this point in Shopify we had very good coverage of our resiliency matrix by unit tests that are all backed by Toxiproxy. And this is really, really simple to do.

<!-- t=1446 -->
Another tool we wrote is called Semian. It's fairly complicated how all of its components work together, so I'm not going to go into it, but there's a README that goes into vivid detail. Semian is a library that helps your application become more resilient, and I encourage you to check the README to find out how it works. This tool was also invaluable for us in becoming a more resilient application.

<!-- t=1478 -->
The mental model we mapped out for working with resiliency was a pyramid. We had a lot of resiliency debt because for 10 years we hadn't paid any attention to this. The web I talked about before, of certain elements dragging down everything with them, was imminent. It was happening everywhere. The resiliency matrix was completely red when we started, and nowadays it's in pretty good shape. So we started climbing the pyramid: writing all these tools and incorporating them. Then when we got to the very top, someone asked the question, "What happens if you flood the data center?"

## 2015: Multi-DC Failover

<!-- t=1517 -->
That's when we started working on multi-DC in 2015. We needed a way such that if the data center caught fire we could fail over to the other data center. But resiliency, sharding and optimization were more important for us than going multi-DC. Multi-DC was largely an infrastructure effort of just going from one to N. It required a massive amount of changes in our cookbooks. But finally we procured all the inventory and all the servers to spin up a second data center. At this point, if you want to fail over Shopify to another data center, you just run a script, and it's done. All of Shopify has moved to a different data center.

<!-- t=1559 -->
The strategy it uses is actually quite simple, and one that most Rails apps can use pretty much as is if the traffic and things like that are set up correctly. Shopify right now is running in a data center in Virginia and one in Chicago. If you go to a Shopify-owned IP, you will go to the data center that is closest to you. If you're in Toronto, you're going to go to the data center in Chicago. If you're in New Orleans, you might go to the data center in Virginia. When you hit that data center, the load balancers inside our network know which of the two data centers is active: is it Chicago, or is it Ashburn? They route all the traffic there. So when we do a failover, we tell the load balancers in all the data centers what the primary data center is. If the primary data center was Chicago and we're moving to Ashburn, we tell the load balancers in both data centers to route all traffic to Ashburn in Virginia.

<!-- t=1614 -->
When the traffic gets there, right after we've moved over, any write will fail. The databases at that point are read-only. They are not writable in both locations at once, because the risk of data corruption is too high. But that means most things actually work. If you're browsing around a Shopify storefront looking at products, which is the majority of traffic, you won't see anything. Even if you're in the admin, you might just be looking at your products and not notice this at all.

<!-- t=1642 -->
While that's happening, we're failing over all of the databases, which means checking that they're caught up in the new data center and then making them writable. Very quickly the shards recover, over a couple of minutes; it can be anywhere from 10 to 60 seconds per database. Then Shopify works again. We then move the jobs: when we moved all the traffic, we stopped the jobs in the source data center, so we move all the jobs over to the new data center and everything just ticks.

## Pods and Sorting Hat

<!-- t=1667 -->
But then how do we use both of these data centers? We have one data center that is essentially doing nothing, just very, very expensive hardware sitting there doing absolutely nothing. How can we get to a state where we're running traffic out of multiple data centers at the same time, utilizing both of them?

<!-- t=1685 -->
The architecture at first looked something like this: shared Redises and shared memcached between all of the shops. When we say shard, we're referring to a MySQL shard, but we hadn't sharded Redis, memcached and other things. So all of this was shared. What if, instead of running one big Shopify that we move around, we run many small Shopifys that are independent from each other and have everything they need to run? We called this a pod. A pod has everything that a Shopify needs to run: the workers, the Redis, the memcached, the MySQL and whatever else there needs to be for a little Shopify to run.

<!-- t=1722 -->
If you have these mini Shopifys and they are completely independent, they can be in multiple data centers at the same time. You can have some of them active in data center one and some of them active in data center two. Pod one might be active in data center two and pod two might be active in data center one. So that's good. But how do you get traffic there?

<!-- t=1750 -->
For Shopify, every single shop usually has a domain. It might be a free domain that we provide or their own domain. When a request hits one of the data centers, the one you are closest to, Chicago or Virginia depending on where in the world you are, it goes to a little script that is very aptly named Sorting Hat. What Sorting Hat does is look at the request and work out which shop, and therefore which pod, which mini Shopify, this request belongs to. If that request is for a shop that is on pod two, it will route it to data center one on the left. If it's another one, it will go to the right. So Sorting Hat is just sitting there sorting the requests and sending them to the right data center. It doesn't care which data center you land in. It just routes you to the other data center if it needs to.

## Rules for Safety and Shitlist Driven Development

<!-- t=1805 -->
Okay, so we have an idea now of what a multi-DC strategy can look like. But how do we know if it's safe? It turns out there just need to be two rules that are honored.

<!-- t=1815 -->
Rule number one is that any request must be annotated with the shop or the pod that it's going to. All of these requests for the storefront are on the shop domains, so they're indirectly annotated with the shop they're going to through the domain. With the domain, we know which pod, which mini Shopify, the request belongs to.

<!-- t=1834 -->
The second rule is that any request can only touch one pod. Otherwise it would have to go across data centers, and potentially one request might have to reach Asia, Europe and maybe also North America all in the same request. That's just not reliable. Again, fundamentally shops, and requests to shops, should be independent, so we should be able to honor these two rules.

<!-- t=1857 -->
You might think, well, that sounds reasonable. Shopify should just be an application with a bunch of controller actions that go to a shop. But there were hundreds if not a thousand requests that violated this. They might do something like going over every shard and counting something, or maybe uninstalling PayPal accounts and seeing if there are any other stores with it, across multiple stores. When you have hundreds of endpoints that are violating something you're trying to do, and you have 100 developers doing all kinds of other things and introducing new endpoints every single day, that could be a losing battle if you just send an email, because tomorrow someone joins who's never read that email and is going to violate this.

<!-- t=1900 -->
Rafael talked a little bit about this yesterday. He called it whitelisting. We called it shitlist driven development. The idea is that your job, if you want to honor rules one and two, is to build something that gives you a shitlist: a list of all the things that violate the rule. If you do not obey the shitlist, you raise an error telling people what to do instead. This needs to be actionable. You can't just tell people not to do something unless you provide an alternative, even if the alternative is that they come to you and you help them solve the problem. But this means that you stop the bleeding, and you can then rely going forward on rules one and two being honored.

<!-- t=1943 -->
When we had this for Shopify, with rules one and two honored, our multi-DC strategy worked. And today, with all of this building atop five years of work, we're running 80,000 requests per second out of multiple data centers. And this is how we got there. Thank you. (applause)

## Q&A: Global Data

**Q: Do you have any global data that doesn't fit into a shard?**

<!-- t=1972 -->
Yes, we have a dreaded master database, and that database holds data that doesn't belong to a single shop. In there is, for example, the shop model. We need something that stores the shop globally, because otherwise the load balancers can't know globally where the shop is. Other examples are apps. Apps are inherently global, and they're installed by many shops. There's billing data, because it might span multiple shops. And partner data. There's actually a lot of this data.

<!-- t=2001 -->
I didn't go into this at all, but I actually spent six months of my life solving this problem. We have a master database that spans multiple data centers, and the way we solved it is essentially to have read slaves in every single data center that feed off the master database, which is in one of the data centers. If you do a write, you do a cross-DC write. This sounds super scary, but we eliminated pretty much every path that has a high SLO from writing to it. Billing has a lower SLO in Shopify because the writes have to be cross-DC.

<!-- t=2035 -->
The thing is that billing, partners and the other sections of this monster database are in different sections. They are fundamentally different applications, and as we speak they are actually being extracted out of Shopify, because Shopify should be a completely sharded application. If they're extracted out of Shopify, then you're also doing a cross-DC write, because you don't know where that thing is. So it's not really making the SLOs worse, and it's okay that some of these things have lower SLOs than checkout, storefront and admin, which have the highest SLOs. So that's how we deal with that. We don't really deal with it.

## Q&A: Disproportionate Traffic to a Single Pod

**Q: How do you deal with a disproportionate amount of traffic to a single pod or a single shop?**

<!-- t=2072 -->
I showed a diagram earlier that shows the workers isolated per pod. This is actually a lie. The workers are shared, which means that a single pod can grab up to 60 to 70% of all of the capacity of Shopify. What's actually isolated in the pod are all the data stores. The workers can move between pods like they're fungible; they move between pods on the fly. The load balancer just sends a request to a worker and it will appropriately connect to the correct pod. So the maximum capacity of a single store is somewhere between 60 and 70% of an entire data center. It is not a hundred percent, because that would cause an outage because of a single store, which we're not interested in. But that's how we move this around. Does that answer it?

## Q&A: Large Amounts of Data

**Q: How do you deal with large amounts of data, like someone importing 100,000 customers or 100,000 orders?**

<!-- t=2138 -->
This is where the multi-tenancy architecture shines. These databases are massive: half a terabyte of memory, many, many tens of cores. So if one customer has tons of orders, that just fits. And if the customer is so large that it needs to be moved, that's what this defragmentation project is about: moving these stores to somewhere where there might be more space for them. Basically we deal with it by having massive data stores that can handle this without a problem.

<!-- t=2168 -->
The import itself is just done in a job. Some of these jobs are quite slow for the big customers, and there we need to do more parallelization work. But most of the time it's not a big deal. If you have millions of orders and it takes a week to import them, you have plenty of other work to do during that time. So this is not something that's been high on the list. How much time do I have left? Done? Okay, thank you. (applause)
