# Talk at GOTO Copenhagen: Shopify's Pods Architecture

2017-10-03 · Recording: https://www.youtube.com/watch?v=N8NWDHgWA28 · Slides: https://speakerdeck.com/sirupsen/goto-copenhagen-2017-shopifys-architecture-to-handle-80k-rps-sales · archived copy: https://archive.sirupsen.com/recordings/goto-copenhagen.mp4

_Lightly edited automatic transcript: speech-to-text, cleaned up by an LLM. It may contain errors; the recording is the source._

# Shopify's Architecture for Flash Sales: Pods, Pod Balancer, Pod Mover and Sorting Hat

## Introduction

<!-- t=0 -->
Thank you, Kevin. My name is Simon, and I work as an infrastructure lead at a company called Shopify, based out of Canada. As a way of introduction, Shopify powers a fairly large amount of online commerce, both here in Scandinavia and in North America. If you bought something that wasn't on Amazon, there's a good probability you went through a Shopify store and used some of the infrastructure we're going to talk about today.

<!-- t=39 -->
The talk is going to be fairly high level. I'm going to talk about the architecture we built at Shopify to support the very large sales we see from some of the biggest American celebrities and some of the largest brands, both in Europe and in North America. It's the architecture we've evolved over the past five years. We won't go super deep into any particular point. Instead I'll give a comprehensive overview of the different components and how they fit together, and hopefully inspire something that can be leveraged in other companies, especially SaaS companies that can have an architecture similar to ours.

## Why Shopify supports flash sales

<!-- t=86 -->
Shopify helps people sell both online and in many other places. About five years ago we faced a bit of a fork in the road. When retailers went online about seven, eight, ten years ago, they often already had a physical presence and then moved online. But we were starting to see a lot more customers who started with an online presence, and they required new ways of selling. Social media had become very big over the past decade, and five years ago smartphones were in everyone's hands, and Instagram, Facebook and Twitter had become quite large. That enabled new ways of selling online that people hadn't been able to do in the past.

<!-- t=141 -->
We call this a flash sale. You announce a product, sometimes minutes in advance, sometimes hours, sometimes days, saying you're going to release some kind of exclusive product at a given day and time. As you can imagine, that can drive an astonishing amount of traffic to one site. It's one minute before noon and everything is good, but at noon you have hundreds of thousands of people coming in trying to buy the same products at the same time, placing an enormous strain on the system.

<!-- t=176 -->
Think of the Super Bowl, or Kylie Jenner, who some of you may know, famous for being famous and part of the Kardashian family. She sells lipstick and can drive an enormous amount of traffic. Kanye can do that as well. Five years ago it wasn't these very large names using this way of selling, but we still had customers who could drive an enormous amount of traffic, and we were faced with a fork in the road. Do we become a company that tries to support these sales, or do we tell them to go and find another platform? It's a very hard thing to be profitable at. It takes thousands of engineering hours to scale to these sales when you haven't architected for it from day one.

<!-- t=226 -->
We chose the path of trying to support them. We chose to see these sales as a way to identify bottlenecks, a bit of a canary in the coal mine for what's going to break next. The flash sales of today tell us what traffic is going to look like one or two years from now.

<!-- t=244 -->
It was such a powerful decision at the time, and so important to our CEO, who is very technical, that he wrote an internal essay on why we support flash sales. It's really important in our company philosophy that in the face of adversity we become stronger and not weaker. When something bad happens, we want to use it as an opportunity to grow and come out stronger. We have a very strong culture internally of injecting chaos into various parts of the organization and coming out stronger as a result. If you drop a piece of glass, it breaks. If you drop a rock, it's indifferent. But what can you drop that becomes stronger as a result? If you go down to the gym and run on a treadmill, you come back a couple of days later and can run further and faster. How can you build software that way? Flash sales were such a thing for our organization. They were a way for our infrastructure to become stronger.

## Scale and the three tiers

<!-- t=317 -->
Before we get into the meat of the talk, here are some numbers to keep in the back of your head for the scale we're running at:

<!-- t=322 -->
- We support about half a million paying merchants all over the world.
- We processed almost six billion dollars in the last quarter.
- We run up to 80,000 requests per second to our Ruby on Rails processes.
- We do about 42 deploys a day.
- We have over 2,000 employees, a large fraction of them engineers deploying all the time.

<!-- t=349 -->
The mental model I want to use for this talk is three tiers:

<!-- t=355 -->
- The **traffic tier**, which is responsible for getting your request from your home network all the way to our infrastructure and serving back the page you're requesting.
- The **application tier** and the **data tier**, which work together to serve those pages.
- The arrows between them: how we fail over between regions with zero downtime, how we shard our application, and how we balance those shards as some shops grow bigger, so that Kanye and Kylie are not on the same shard but spread out. And how we get from the traffic layer into the application layer and fail over regions.

<!-- t=390 -->
By the end of this talk you should have a pretty good idea of how that works for us.

## Global routing with BGP anycast

<!-- t=396 -->
We're starting with the traffic tier. To me it's the layer responsible for getting the request from A to B, in this case from your home network to our network. We'll talk about how we do global routing, because we've had a lot of debate about this internally. We'll talk about a technology called OpenResty, which is absolutely incredible but quite underused. We'll talk about how we protect ourselves from bots during sales, how we serve cache hits from the load balancers instead of the application tier, orders of magnitude faster, and how we throttle checkouts during very large sales.

<!-- t=432 -->
To understand how your request gets from the customer to our data centers, or regions as we call them, we need to understand a little about how the internet works. This is a very high-level overview of how routes propagate. If you are Facebook or Google and you have one domain, not a bunch of customers pointing their DNS at you, you can do a lot of really complex traffic engineering at the DNS layer. But when you have about a million domains from half a million different customers pointing at your IPs, you have very limited control at the traffic level. You only have the IPs your customers are pointing at you, and you can't control anything at the DNS level.

<!-- t=474 -->
So we did something that hasn't had a lot of use in the industry, for those of you who know about networks. It's called BGP anycast, and I'll explain how it works, and how traffic gets from the customer to our regions with it.

<!-- t=496 -->
Shopify has IPs. As you can see here, we have a /24, a block of 256 IPs, and our regions announce to the internet that these IPs can be routed by these regions. If an ISP sees one of those IPs, it should route it to the Shopify region. We connect our routers to our ISPs and say, "Hey, we own these IPs. If you encounter them, send them to us, because that is Shopify traffic for some Shopify store." Say we have two ISPs we connect to directly from our regions. Those ISPs then tell their adjacent neighbors, the ISPs they're plugged into, that they can route those IPs and know how to route them, and it propagates through this entire graph.

<!-- t=542 -->
You can think of it as a bit of a gossip algorithm, where every ISP is constantly telling its neighboring ISPs which IPs it knows how to route. You can imagine ISPs ranging from Asia to Europe to America to South America. That entire internet broadcast takes about seven minutes, which is either amazing or very slow depending on where you're coming from.

<!-- t=568 -->
Now we have this connected web. How does a customer get their traffic to a region? They make a request, in this case to walrus.com, which points to a Shopify-owned IP that is part of the block the regions are announcing. It goes to their ISP, which then needs to turn left or right depending on which region is closest. In this case we're going right, at least from where I'm standing, and the request is routed to the region. This works really well when you're pointing all these customer domains directly at IPs and you don't have much control over it.

## OpenResty and Lua scripting at the load balancer

<!-- t=608 -->
Now we know how the traffic gets to Shopify. Next I want to talk about what happens when the request enters our network. We use a technology called OpenResty, which is perhaps one of the best additions we've made to our technology stack in the past four years. It enables you to script your nginx load balancers with Lua. You can do pretty much everything you desire. You can customize your load balancing algorithm, redirect to other data centers, change headers, inspect cookies, serve different content based on requests and cookies, and do network calls. All of that happens inside nginx, which is extremely fast and extremely stable.

<!-- t=654 -->
I think OpenResty is quite underrated, and I don't hear enough talk about it, because it is so stable, so fast and so scalable. I want to talk about a couple of modules we built with it. It has really become our hammer and nail for solving a lot of traffic-related issues, and we've gotten away with warding off a lot of DDoSes just with simple Lua modules. It's really powerful to be able to script your load balancing tier.

<!-- t=682 -->
Here's a very quick example of what it looks like. This is a very bare-bones nginx config that listens on a port and serves some content via Lua. You can do all kinds of things here, like customized load balancing algorithms, and you can imagine doing network I/O in this block. You could do pretty much anything.

## Fighting bots with the Bot Squasher

<!-- t=700 -->
One of the modules I want to explain addresses our big problem with bots. As we learned, there is a very large secondary market for sneakers. There are a lot of sneakerheads out there who collect sneakers, and to get these sneakers you need to be there when the sale happens, and you need to be really lucky because there's limited inventory. Maybe two hundred thousand people are competing for 5,000 pairs of shoes. When that happens, an economy forms where people buy these shoes, don't wear them, and sell them on eBay for maybe two, three, four times the price. Of course merchants don't want that. They don't want a secondhand market charging more than they do and not controlling the customer experience. So they asked us to come up with a way to detect these bots and ban them.

<!-- t=751 -->
We took out our hammer and nail, OpenResty, and built some modules that helped us solve this problem. The Kafka logger module is one we've had for a long time. Every single request that comes into Shopify is logged into Kafka, a distributed message bus, and all the messages are relayed into our data warehouse. But we can also consume these messages in real time and make decisions about them.

<!-- t=774 -->
Imagine a Kafka cluster here. For a checkout, we log that checkout into Kafka. Then there's a stream aggregator, which is the Bot Squasher. It listens on Kafka and tries to see, across all these different events, whether there's a suspicious pattern. Is there something that doesn't look like a human? Is it refreshing a page very rapidly, in a way that a human in a Chrome browser would never do? Or is there something else at stake? It's a fairly sophisticated algorithm for figuring out what is going on. It is not an OpenResty module, just a simple consumer, I think written in Go in this case.

<!-- t=809 -->
When it finds an IP, a user agent or something else that looks like a bot, it can tell the Rule Banner, which is written in nginx and Lua. It's a simple DSL that allows you to ban various patterns, very simple ones like user agents and IPs. Based on the assumptions made from the Kafka stream, the Bot Squasher can then ban an IP or something else on all the load balancers with the Rule Banner. Now we can reject these bots in a few microseconds, and people can write very sophisticated algorithms in the bot squashing layer to ban them very rapidly.

<!-- t=842 -->
When we deployed this, we saw a fairly sizable decrease in the number of requests coming to Shopify. We had to turn it off again because we were so flustered, thinking this was way too many. It turns out a very large fraction of our traffic was just bots.

## Serving cache hits from the load balancer

<!-- t=856 -->
Another thing we do with OpenResty is serve cache hits. If you browse a Shopify product, there's probably someone else who will browse the same product page later, so we can cache it. If nothing changes, we could go to the application tier and serve that cache hit, which is stored in memcache. But we can do this orders of magnitude faster if we do it at the nginx layer, because of the overhead of the Ruby VM and the whole stack we have there, which is very difficult to optimize at this point.

<!-- t=887 -->
Imagine a request coming in for a collection filled with walrus products, and you have a cache miss. It gets to the web application tier. While fulfilling that request, getting all the data from the data stores and rendering the response back to the user, we also fill the response into the cache for someone else to consume. Really simple stuff, nothing novel. But on the next hit, the cache module, which is an OpenResty module, checks the cache directly at the load balancing tier. This is fairly simple. Now we can get it directly from the cache without even touching the application tier, making us able to handle orders of magnitude more requests per second with a fairly small addition to the stack.

<!-- t=927 -->
This is more complex than it looks, at least for our stack, because generating cache keys and so on turned out to be quite complex, but for smaller applications it might be quite simple. We don't have this turned on for everyone, but we do for some larger merchants. The reason it's hard for us to turn on is that people on Shopify can customize their themes completely, and getting that to play well with caching has turned out to be more difficult than we thought.

## Throttling checkouts

<!-- t=952 -->
The last one I want to talk about is the checkout throttle. Some merchants come in and can drive so much traffic that we can't handle the number of writes they generate on a checkout, and you can't cache a write, so you need to do something at some point. So many writes go into the database for adding to the checkout, running scripts that modify the cart, getting shipping rates, processing payments, and all the other very complex operations that happen when you're checking out. We have to throttle it.

<!-- t=979 -->
Very few merchants ever hit this. I would say probably fewer than 10 or 20 unique merchants in the lifetime of Shopify have ever hit this throttle. But when it does happen, it's very important that all the shops on the same shard as you do not suffer under your sale.

<!-- t=996 -->
So when a request comes in for a very busy store to the checkout, we put that request into a queue, again at the load balancing tier. While you're in the queue, you're redirected to a wait area, which is completely customizable by the merchant. Again, we strive as hard as possible not to put people in this at all, but we need a safeguard in case it does happen. We control the rate at which we leak from the queue with a throttle, and when the throttle releases you, we redirect you back to the checkout.

## Pods: isolated units of Shopify

<!-- t=1026 -->
We've talked about the traffic tier, some of the OpenResty modules we use, and how traffic gets all the way to Shopify. Now I want to dig into how our application layer and our data tier work. I'm going to talk about both at the same time because they're quite intermingled.

<!-- t=1041 -->
A very important concept for us is what we call a pod. Some companies call this a shard, but a shard to us means a MySQL, a relational shard. A pod to us encompasses more. A pod is a small, isolated Shopify that can run anywhere in the world: in a cloud, in a data center, anywhere. It's an isolated unit of Shopify with one or more shops. Say we have pod 7, pod 2 and pod 14. They all have a bunch of shops on them. These shops don't talk to each other, and the pods don't talk to each other. They're completely sharded and isolated from each other, giving us some very nice scaling capabilities.

<!-- t=1084 -->
This is a fundamental principle we call the shop isolation principle: all shops must be isolated from each other. If you want to do something cross-shop, you have to use the data warehouse or some other mechanism, because doing things across hundreds of shops and hundreds of pods in multiple regions is not something that's going to scale, at least not in real time. Another thing to note here is that these pods are not the same size. Some have more shops than others, and we'll address that problem later.

<!-- t=1113 -->
First I want to look at what's in a pod. A pod is one or more stores. What does that mean? It means all the stateful layers of that pod are isolated. A pod has a MySQL, a memcache instance, a Redis instance, and cron workers doing periodic work. So all the stateful data of all of these shops is isolated from all the other pods and all the other shops.

<!-- t=1137 -->
There is a tier that is shared, and those are the workers. When a web request or a job is handled, the capacity is shared between the pods. Why would that tier be shared? The reason is the sales. Remember the premise of this talk: we have to architect for these massive sales. If pod 2 is experiencing a massive sale, some kind of launch, it may need perhaps sixty to seventy percent of all the worker capacity to handle that sale. So the stateless stuff is shared and the stateful stuff is isolated by pod.

<!-- t=1173 -->
Why wouldn't you use an autoscaler for this? It sounds like the canonical thing to solve with a cloud autoscaler. When you have a sale, you just start spinning up more instances. The problem is that spinning up those instances takes tens of seconds, and during that time you're going to serve a lot of errors and give a very bad experience to all the customers coming in. So we don't do that. Maybe one day we can do some really crazy things so that we can spin up these workers fast enough with an autoscaler, but today that's not the case.

<!-- t=1205 -->
Another stateless thing that is shared is the traffic tier we talked about before, where all the OpenResty modules run and the load balancing happens.

## Load testing with Genghis

<!-- t=1216 -->
While we're here, I want to introduce a tool for testing the stateless tier. We have a load testing tool called Genghis. It runs Lua scripts that define a flow through our application and allows us to test with an astonishing amount of load using cloud instances. What happens when we send, for example, 10,000 people per minute through the checkout? You can customize this as much as you want with Lua scripts that say things like: this customer is going to go to the product page, browse around a bit, be a little indecisive, then add some products, remove some products from the cart, check out, pay for it, and then you multiply that by maybe 10,000 or 6,000.

<!-- t=1254 -->
The interface to this is really simple. We have a Slack bot that just responds when you tell it to enqueue some Lua script. The JSON file you're enqueueing describes which Lua script to use, which store to target, how long to run, and at what rate, meaning how many of these executions you want per minute.

## The Pod Balancer

<!-- t=1272 -->
Remember the problem from before of all these pods being different sizes. That's a problem we've been trying to address. In the past, when we needed to provision a new pod because we had too many shops, we replicated everything in the old pod that we wanted to split in half over to a new pod. For a short duration we would change the routing layer to send half the shops to the new pod and the other half to the old pod, and then start cleaning up the old pod. It was a bit of a mess and a very manual, operational amount of work. I think the checklist for doing this was like a hundred items.

<!-- t=1309 -->
We wanted to make this better and less risky, with fewer big hand-wavy movements, and move shops between pods with less downtime. We wanted to be really good at moving shops between pods with the smallest amount of downtime possible. We shouldn't punish large merchants, where a large Shopify store may take hours to move, by giving them hours of downtime, even if they can schedule it. That's a terrible experience for a merchant who may have an opportunity cost of perhaps several million dollars in that transition.

<!-- t=1344 -->
What the Pod Balancer does is look at these pods and the shops on them, and make decisions about how to balance the pods. In this case it sees that pod 2 is a bit bigger, so it moves a shop from there to pod 7. But in the interim pod 14 has grown. One of its shops is occupying three units. This could be a shop that has large sales, or a shop that is just very large, maybe one that is replatforming from another commerce solution and has grown tremendously in size. Some shops are bigger than others in terms of size, scale, sales, whatever they do. The Pod Balancer just continues to work, so it moves some other shops from pod 14 over to other pods. The complexity increases because you also have sign-ups, new shops coming in, and other shops growing, and the Pod Balancer just keeps working at it. It keeps trying to fill up all these pods to equal size.

<!-- t=1400 -->
Before we had this, some pods were maybe twice as big as others, giving you a bit of a noisy neighbor problem where on some of these shards you were experiencing more incidents than on others. What happens when they're all filled up? You create a new pod. We're working this year on making that really easy, so the Pod Balancer simply makes a decision to create a new pod and then moves the shops over. We want it to be something that is always on and that we never have to touch, creating pods continuously. It's not perfect yet and there's still some manual labor involved, but this is where we're going, and it's going quite well so far.

## Moving shops between pods: from locking to binlog streaming

<!-- t=1439 -->
I want to talk about how we move those shops from shard to shard, with minimal downtime, without punishing the very large shops. Let's look at two of the data stores, say Redis and MySQL. If we want to move a shop from pod 9 to pod 23, the very naive way is to copy the shop by going through every single table and finding all the records that belong to the shop with a simple query: give me all the orders where the shop ID is this, give me all the products where the shop ID is this. You put that into the target shard, and while that's happening you don't accept any writes. The problem is that while you're copying the shop, it might insert a checkout after you've already copied the checkouts table, and that checkout would be lost. So you basically need to lock the shop while you're copying it. For a small shop that's not a problem. It may take less than a minute, maybe a couple of seconds. But for a large shop, if you don't allow any writes while you're copying it, this can take a very long time.

<!-- t=1497 -->
What we're working on now is not doing this lock-move-unlock. Instead we look at the MySQL binlog and stream these events as we're copying the shop, to avoid locking the shop and having downtime while we copy it. The binlog is, in MySQL vocabulary, where every modification to the database is logged. If you change a row or insert a new checkout, it's registered in the binlog as an event: insert a new checkout for this shop, insert a new product for this shop.

<!-- t=1527 -->
What the Pod Balancer is going to do, and what we're starting to work on now, is stream the binlog and look at new events coming in that haven't been copied yet. The bulk copier tries to copy all the old data from the bottom, and the binlog tailer component of the Pod Balancer replicates new events. So old data fills in from the bottom and new data from the top. This means we don't have to lock the shop for an extended duration.

<!-- t=1554 -->
When that's done, we can imagine a discrete moment in time where the shop is completely replicated. Now we need to move it to the new pod by updating the routing information. First we lock the shop, so no writes can come in. We interrupt all jobs and all web requests going on for that shop for a brief couple of seconds. Then we go into a routing data store, the data store that stores the mapping from shop to pod and from pod to data center, and we update the shop's pod ID there. Immediately this is reflected and traffic starts going to the new pod. We unlock the shop, and the shop has been moved. Then we remove it from the MySQL layer, and we move the jobs and other things in Redis over as well, which is asynchronous. Now the shop has been completely moved.

<!-- t=1600 -->
You multiply this by running a thousand of these in parallel, and suddenly you can move a lot of shops to a lot of pods very rapidly. On top of this you add really good decisions about which shops you're moving and at what time, to keep all those pods balanced.

## Sorting Hat: routing requests to the right pod

<!-- t=1619 -->
We've talked about traffic and about how the application tier works with these pods. Now I want to talk briefly about how we connect the two. How does the traffic layer know how to route a request to the correct pod? For this we have a very aptly named component called Sorting Hat. What Sorting Hat does is look at requests coming into the Shopify network and figure out which pod each request belongs to. It needs to send the request to the region where that pod is active, and it needs to work with failovers, shop moves and all of these different components. It's a very core part of the pod architecture and one of the first things we worked on.

<!-- t=1658 -->
Again we have the traffic layer, and we have another OpenResty Lua module called Sorting Hat. It knows which pods are active and which are inactive. In a region, a pod is active or inactive, and there's always a hot replica pair, so that pod 2 can become active in region A if there's a catastrophe or we need to do some maintenance in that region, and of course the other way around too. They're always set up in pairs.

<!-- t=1685 -->
Then we have the routing database from before, which we use when we move shops around. Sorting Hat is the consumer of that. It's the only thing that queries it and asks where shops are living. So if a customer comes in getting their products for sneakershop.com, Sorting Hat asks the routing layer where to route sneakershop.com. The routing layer returns a shop ID, say shop 238, which is on pod 2, and we know it is active in region B, so Sorting Hat routes the traffic there.

<!-- t=1717 -->
This is very simple. It's I think a couple hundred lines of Lua querying this data store and doing this thousands and thousands of times a second. It's what ties the whole pod architecture together and makes the traffic layer marry the data and application layers.

## The Pod Mover: regional failover

<!-- t=1736 -->
The last thing I want to talk about is how we fail regions over. We have the active pod and the inactive pod. How do we flip them over, and how do we do that with minimal disruption? For this we have the Pod Mover, which moves pods between regions with minimal downtime. We're now at a point where we're getting very close to having almost zero-downtime regional failovers.

<!-- t=1758 -->
This is really important to us because we think it's such a core thing that we want to be able to do all the time. If someone wants to do maintenance in a cloud region without any repercussions, they should be able to move all the pods to another region, do the maintenance, and move them back without any risk. This avoids the problem of having very expensive staging and production environments that emulate an entire data center, because instead you can just get really good at moving pods between data centers.

<!-- t=1783 -->
So say region B is having some kind of incident. It may be out of power, we may be doing maintenance, it may be subject to a hurricane or flooding, whatever happens. We need to be able to move that region over. This is the Pod Mover's job, and it can do it very rapidly. It goes through a series of steps to fail over a region:

<!-- t=1808 -->
1. It disables cron in both regions, so we're not running new periodic jobs.
2. It goes back to the routing data store, the one that keeps all the shop-to-pod-to-region mappings. The Pod Balancer was looking at and modifying the shop-to-pod mapping, and the Pod Mover is responsible for the pod-to-region mapping, so it updates that. Sorting Hat, the thing looking at requests and routing them to the appropriate regions, now routes requests to the new target region.
3. We fail over MySQL to the target region, the little animation from before.
4. We enable cron.
5. We transfer the jobs that have not yet been executed in the source to the target, to be executed later.

<!-- t=1850 -->
It's fairly simple at a high level. Of course the devil is in the details, but this is at a high level what happens when we do a pod move.

## The Pauser: avoiding errors during failover

<!-- t=1857 -->
What about the errors? I mentioned that we wanted to minimize them. When Sorting Hat starts to route to that previously inactive region, the database is not yet writable there. If someone is doing a checkout, they're going to have that dreaded experience that everyone in this room has probably had: you're checking out, you've just entered your credit card details, you get an error back, and now you have to call someone and ask what's going on. We don't want that scenario, so we sat down and thought about how we could fix it. Again we took out our hammer and nail, OpenResty, and solved this problem.

<!-- t=1889 -->
If you have MySQL replicated across two data centers, they can't both be writable at the same time. One will be writable and one will be read-only. There will be a couple of seconds of downtime where people have these terrible experiences. But what if you can pause the requests at the traffic tier, the ones that can't be served in the new region because you know they're going to do a write? Then you can do basically zero-downtime failovers.

<!-- t=1914 -->
So the Pauser module pauses requests in the middle of failovers to avoid serving errors. A customer comes in and tries to create a new checkout during a failover, during those critical seconds when the database is not yet writable in the new region. We add them to a queue and drain that queue from a throttle to avoid too many people coming in at the same time. The throttle is of course disabled while we're failing over the region and MySQL has not failed over yet. Once it has, the Pauser finally gives you an HTTP 200 response later. So instead of the dreaded error page when you've entered your credit card details, you're just waiting five seconds. This works now and it's in production for us.

<!-- t=1957 -->
This changes the Pod Mover flow a little bit. Everything else is the same, but when Sorting Hat routes a request to a target region, it inspects the request and makes a judgment on whether or not it would perform a write. If it would, it holds the request back at the load balancing tier. Then we fail over the MySQL region, resume the requests, enable cron, and transfer the jobs. It's really simple, and this has gotten us down to very, very few errors during failovers.

<!-- t=1983 -->
We want to be so confident in these failovers that people can just issue a command in Slack and fail over a region. We already have this working, but we're not quite confident enough yet to do it on production pods. It will be just weeks before we're there. You type in a command and fail over a pod, and you don't have to care about customer disruption because there should be none, and we work as hard as possible to make that true. This is a core primitive of the pod architecture. If there's an incident, someone should be able to come in in the middle of the night, fail over all the pods and go back to bed. Maybe some day this will be automated, but even now it's a tremendous step from where we were years ago.

## Pods and the cloud, and wrap-up

<!-- t=2021 -->
The last thing I want to note is another one of the second-order benefits of the pod architecture. We're starting to experiment more with the cloud, and with this architecture it's just a matter of putting a couple of pods in the cloud. Of course there's a lot of work going on to get all these stateful tiers and the stateless tiers into the cloud, but once we have them there, we can gradually move shops over without disrupting the rest of the platform. We have a little gas pedal. We slowly move shops to the cloud, and as we gain operational expertise with the cloud, we can move all of Shopify over.

<!-- t=2062 -->
This marks the end of my talk. I hope you now have a pretty good idea of how Shopify works at a high level, and maybe some of this you can take home and think about. If you're a SaaS company especially, a lot of this should be applicable. I've talked about all these components: the pods, the Pod Balancer, the Pod Mover and Sorting Hat. Thank you so much.

## Q&A

<!-- t=2089 -->
**Kevin:** Really cool talk, thank you, Simon. There are also some really good questions. One of them: they understand how you move pods, but how do you move them if the region is down or unreachable?

<!-- t=2104 -->
**Simon:** Let me go back to this slide, because it's quite close here. The question is what happens when region B is on fire and unreachable. How do you fail it over? There are certain things you can't do, but you can do checks. One of the problems would be that the database is not completely caught up in the new region. You have to make a judgment call at that point. That process is still manual, and it will tell you, "Hey, you're going to lose this much data if you perform this failover. Are you sure you want to do it?" So the Pod Mover is as resilient as possible. In the best case it will do a pod move with all the jobs and all the data moved, but where it can't, it will ask you questions: are you okay with losing this much data, are you okay with not transferring jobs. And it will still perform the best effort we can do.

<!-- t=2155 -->
**Kevin:** There's a question about the Pod Mover. It sounds like it could run in multiple instances. How do you avoid race conditions between them, and how do you avoid smaller shops being moved all the time?

<!-- t=2172 -->
**Simon:** I think this slide is appropriate to show. When we first talked about the Pod Mover, what we wanted to do was design it so that when you update the routing information, there's a bunch of subscribers: one responsible for jobs, one for stopping cron, one for failing over, one for Sorting Hat, and they all subscribe in a kind of pub/sub system. But distributed coordination is very difficult, so we didn't want to do it. It's too complex and the ROI is just not high enough. So this is literally one script running on a machine somewhere. Which machine it runs on is of course resilient to which regions are down and things like that, but it's one machine doing a bunch of sequential steps. It's resilient and very, very simple. That's how we avoid the race conditions.

<!-- t=2218 -->
**Kevin:** Another question: is the customization of nginx a task for feature developers or for operations? Who works on the nginx layer?

<!-- t=2233 -->
**Simon:** It has primarily been a place where infrastructure engineers make modifications to the OpenResty layer. But we've also had some application engineers make modifications, for example to redirect fractions of traffic and do tests and things like that. I would say it's probably 90% infrastructure engineering at this tier and 10% product engineering.

<!-- t=2258 -->
**Kevin:** Next question: how do you implement multi-tenancy within a pod?

<!-- t=2264 -->
**Simon:** This is almost a talk in itself. Sharding is essentially by a key, the shop ID, and every time you do a query you're always in the context of the shop. If you're handling a web request or a job, you can only query data belonging to that shop, because every query is appended with the shop ID. So it's like select all from orders where shop ID is this, or select product where ID is this and shop ID is this. They're all in the same schema. We don't run hundreds or thousands of schemas, because that's not a very normal workload for MySQL, and we want the workload to be as normalized as possible. That's the very high-level overview.

<!-- t=2302 -->
One other thing I'll note is that we do this sharding at the application level. We don't have a proxy or something like that. When we implemented sharding, that's not something we were comfortable with, so it's application-level sharding. The shop ID key schema is shared between the shops on the pod, and every single table has a shop ID key embedded in it, so that we can do these queries efficiently.

<!-- t=2325 -->
**Kevin:** Really cool. Two last questions. One is about Kafka: is it something that would replace the MySQL binlog for moving shops?

<!-- t=2340 -->
**Simon:** I'm not entirely sure how to interpret that question, but I'll give it my best. If the question is whether we could put the entire MySQL binlog into Kafka and use that instead, the answer is yes. We currently don't relay the binlog into Kafka. It's not something we've really found a use for. Tailing the binlog works really well for us right now. Moving to Kafka is maybe something we would do if there were other use cases for it, but right now I don't see the ROI on using Kafka over the binlog as very high.

<!-- t=2377 -->
**Kevin:** And the last question, somebody asked: are you looking into Kubernetes?

<!-- t=2384 -->
**Simon:** Yes. Those little cloud regions on the right are Kubernetes, and it's in production.

<!-- t=2392 -->
**Kevin:** Okay. Please give Simon a big hand, and remember to rate the talk in the app. Also check out Simon's blog. He has a really cool blog where he writes about this stuff. Search for his name and you will find it. It's really good. Give him a hand, thank you.

[Applause]
