# Talk at @Scale: Resiliency Testing with Toxiproxy

2017-08-31 · Recording: https://atscaleconference.com/videos/resiliency-testing-with-toxiproxy/ · archived copy: https://archive.sirupsen.com/recordings/atscale.mp4

_Lightly edited automatic transcript: speech-to-text, cleaned up by an LLM. It may contain errors; the recording is the source._

## Introduction

<!-- t=0 -->
**Host:** Hi everyone, welcome back. I hope everyone had a great lunch. We're back to the DevTools track at @Scale. Our next speaker is Simon Eskildsen. He's a Dane who is currently working in Ottawa, which is my hometown, for Shopify. He's going to talk to us about their system for resiliency testing. Please welcome Simon.

<!-- t=38 -->
**Simon:** Thank you so much. I'm going to be talking today about the resiliency testing we've done at Shopify. We kicked this off about three years ago. Today I want to talk specifically about a tool we developed called Toxiproxy, which helps you write tests for chaotic test cases in your CI, development and test environments.

<!-- t=62 -->
I chose this as the topic of my talk because it's not something I've heard a lot of companies talk about publicly. I know a lot of companies have something like this. I've heard a lot about chaos monkeys, but this sits earlier, in the realm of reproducing failures in development and the test environment.

<!-- t=81 -->
I want you to leave here with a good understanding of why testing is a useful tool for resiliency. I also want you to know how to get started writing tests for your single points of failure, and to try to discover them right after this talk.

<!-- t=96 -->
My name is Simon and I work on the production engineering team at Shopify. Shopify is not a typo of Spotify, as someone in the audience said to me before. It's a commerce company, and we provide a white-glove experience for merchants to sell online, in person, on Amazon, on Pinterest and so on. You put your products with us, and we take care of distributing them to all the channels, managing inventory and that sort of thing.

## Shopify's scale

<!-- t=126 -->
I want you to have a couple of numbers in the back of your head as we go through this talk:

<!-- t=131 -->
- We have about half a million customers.
- We peak at around 100k RPS today.
- We deploy about 40 times daily, so a lot of code changes go out, which is part of the reason resiliency is so important to us.
- Billions of dollars go through the platform every year.

<!-- t=148 -->
We are primarily a Ruby on Rails company, and we've been working in a fairly large monolith for about 10 years. This is important for the talk: we are still a fairly monolithic company. We're seeing more and more services developed internally, but especially when we started the resiliency work, the majority of our traffic came through this single application.

## The summer of 2014

<!-- t=173 -->
I want to start with a little story. In the summer of 2014, we had a couple of really terrible outages. In one week we had two one-hour outages of a Redis server that no one really knew the responsibilities of. It went down and took everything down with it, for one hour, twice in one week.

<!-- t=196 -->
Rightly so, the infrastructure team at the time was panicked, and we convened on the weekend to figure out what to do. The next week was probably going to be equally terrible. There was something we hadn't been proactive about, and we needed to figure out what.

<!-- t=210 -->
Up until this point, the biggest threat to the existence of Shopify was the very large merchants that drive a lot of traffic. There are a lot of celebrities on this continent, and they can bring in a lot of traffic at once. Those were the customers we had been scared of. If Kanye drops an album and Shopify is suffering, that was the biggest threat to our existence at the time.

<!-- t=242 -->
Now we were reaching a point where it wasn't so much these large sales that were a threat. We were seeing steady-state incidents that were quite catastrophic. So we needed to shift our infrastructure team away from providing scalability for large sales and look more at steady-state resiliency and reliability.

<!-- t=266 -->
I mentioned the Redis server. No one really understood what it did. It had made its way into our infrastructure as a way to get some load off MySQL while we were panicking and getting ready for big sales. But this single component had a disproportionate effect on the reliability of the entire platform, and we were trying to figure out what to do about it.

## The resiliency matrix

<!-- t=288 -->
When we convened on the weekend, one of the things that came out of it was a concept we call the resiliency matrix. The idea was to get a little ahead of the game. We wanted to think about all the possible interactions we have with other services, and how they impact our availability when they have problems.

<!-- t=311 -->
The way to interpret it: on the left-hand side are all the services our application communicates with. This is not an exhaustive list, but these could be some of the scenarios. On the top are the important business sections of Shopify:

<!-- t=327 -->
- **Storefront** is where people browse products, look at inventory, figure out what they like, and add it to their cart.
- **Checkout** is where you enter your credit card details, shipping information and so on.
- **API** is what our customers' apps work against to modify their stores.
- **Admin** is where you do that yourself: manage your products, your inventory and that sort of thing.

<!-- t=351 -->
Every cell then denotes what happens to the section at the top when the service on the left-hand side is experiencing some kind of problem.

<!-- t=361 -->
Someone had this great idea of coming up with this spreadsheet, but we didn't fathom how hard it would be to figure out what every cell's value should be. There were question marks. Was the section degraded, was it up, was it down? We didn't know. We had never paid attention to this. If you've never written down anything like this either, I bet you also have no idea what happens when various services are down and how it harms your application.

<!-- t=389 -->
So this was a good thing to come up with. It was easy to come up with columns and rows, but now we needed an approach to attack the cells.

## Why tests rather than Chaos Monkey

<!-- t=398 -->
At the time, back in 2014, Netflix was starting to talk a lot about Chaos Monkey, and it was becoming a well-respected methodology in the community. But using Chaos Monkey then would have been a bit like putting fuel on a fire. We were already having these one-hour incidents, and we now had a resiliency matrix full of other incidents we didn't know about. There was plenty to attack. We had a big bucket of failure scenarios to figure out what to do about.

<!-- t=433 -->
Put in terms of known unknowns and unknown unknowns, Chaos Monkey is fantastic for discovering unknown unknowns. Chaos engineering as a discipline, which you may have heard Netflix talk about, is very good for figuring out the very complex failure scenarios where your application breaks. But incidents and the resiliency matrix are also a fantastic source, and I would recommend you start there.

<!-- t=463 -->
We wanted to write a test for every single cell in that matrix. If someone fixed what happens when the cart service is down, so that it showed the appropriate thing and had a fallback value, we should never be able to regress to the old state. A test seems like a great way to do that.

<!-- t=488 -->
There are other approaches than writing a test. You can use something like Chaos Monkey, kill the service in production, make sure the failure doesn't recur, and automate that. But we were afraid of Chaos Monkey at the time, and we already had a very good internal culture of continuous integration and writing tests.

<!-- t=509 -->
Tests are great because they're non-negotiable. You can't negotiate with a test. If it fails because you broke something or introduced a new dependency, there's nothing you can do. Your build will not ship. We liked that because it was a mental model our developers were already clear on. Chaos Monkey would have introduced a new mental model and a new way to think about failures, and we weren't ready for that. We'd had two one-hour outages in a week. We needed to solve this problem now, and scale it across the development organization as quickly as we could.

<!-- t=542 -->
The other thing we wanted was a test for the incidents we were experiencing that didn't fit as well on the resiliency matrix, or that expanded it. When we got to the bottom of an incident and figured out how it affected our application, we wanted to write a test for it so it didn't happen again.

## An example: session storage goes down

<!-- t=561 -->
To bring this up a level of abstraction, you can imagine a Shopify store looking something like this. We're selling sneakers, and a page like this comprises various things we have to call:

<!-- t=578 -->
- A search service to show some results.
- The cart, where all of our product is stored.
- The session. You can see here I'm logged in as Simon.
- MySQL, storing a bunch of information.
- A caching layer, and a ton more services to render this page.

<!-- t=594 -->
Let's say the session storage goes down. What do you expect to happen? This is what happens in a lot of high-level languages: an exception is raised from the driver, you don't do anything about it, and we serve a 500 back. A far more appropriate way to deal with that error is to sign the customer out. I'm no longer signed in as Simon, but I can still do everything else.

<!-- t=617 -->
This is nothing new. Netflix has talked about this for a long time, for example not recommending specific videos if the recommendation engine is down. But it's a very powerful concept that can be used for everything else. What is new is that we can now write a test for this. The developers that own the checkout path or the session path can write a test to make sure that when this service is down, we just sign the customer out.

## A maturity model of resiliency

<!-- t=651 -->
Before we get into how we test this, I want to present a maturity model of resiliency. We knew we had not been proactive enough in our resiliency efforts. We were very close to having done nothing at all. When we drew out the matrix, no one knew what all those question marks would be.

<!-- t=671 -->
At the very top of the maturity model, you can have things like making sure business metrics are not impacted when you have major outages. In the middle, you have chaos engineering. We decided that the first and most impactful step was to start writing tests for all of this and leverage the very strong internal test culture we already had. It takes a lot of maturity to bring these practices into production, and we weren't ready for it. The ROI on just doing this in test seemed very large.

## Three approaches: mocks, system tools, proxies

<!-- t=705 -->
So how do we do this testing? There are at least three ways to approach it:

<!-- t=709 -->
1. You could do mocking.
2. You could do something at the systems level, with tools like TC and iptables, and let the Linux kernel do its work.
3. You could write a proxy, put it between you and all the services, and let it emulate errors.

<!-- t=724 -->
This is actually the progression we followed, because at the end of the day, we didn't want to write a proxy.

### Mocks

<!-- t=732 -->
This test is written in Ruby, but follow along. It's a test for one of the cells in the matrix I showed before: the storefront section should be up if the cart service is unavailable. I take the cart service and create a mock that raises an error. Then I get some collection, which is part of the storefront section, and I hope an HTTP 200 comes back.

<!-- t=762 -->
This has significant problems. It's okay for business logic, for making sure appropriate fallbacks are in place, but it covers up about as many bugs as it uncovers. We found a lot of bugs in our drivers for communicating with various services. We found bugs in the MySQL library we were using in Ruby, in the ORM on top of it, and in the connection pooling logic, because no one had done this on these drivers before.

<!-- t=798 -->
Mocks are also only as accurate as you make them. The error raised here may not be the error actually raised by the service. Maybe it wraps it in another exception, or maybe it's something different. The HTTP library in Ruby is infamous for having tens of different exceptions, many of them for very similar errors.

<!-- t=817 -->
Coverage is another problem. Are you absolutely certain that all interactions with the cart service go through that cart service object? If you are, you have a much better application than we do. But it's very uncertain. You have to mock at a very low level to guarantee something like that. And if you start mocking at the driver level, in our experience you're in for a bad time. You're basically teaching your test everything about how your service works.

<!-- t=847 -->
So we stepped back from mocks. They were good for getting us started on the resiliency matrix and flipping some of the question marks to green, yellow and red. But they weren't the end game.

### System-level tools

<!-- t=863 -->
We then looked at doing something at the systems level. iptables and TC, the traffic control utility, are very powerful. They let you introduce delays and simulate network packets being dropped or rejected. The drivers generally return very genuine errors in response to this.

<!-- t=884 -->
The problem is that it was not very portable. Our development environment was OS X, and we were running Linux in continuous integration. It was also fairly inflexible. We had, perhaps, too grand a vision at the time: we wanted to corrupt HTTP headers, and to simulate slow writes with fast reads, so we could write tests for all the incidents. And a lot of these utilities require root. If you change your network stack so they don't, you introduce more complexity just for the sake of resiliency.

<!-- t=927 -->
This allowed us to find some more bugs as we wrote some really simple, sketchy automation that shelled out to these tools. But it was nasty and didn't work very well in development.

### Proxies

<!-- t=939 -->
So we looked at proxies, and surveyed the landscape for proxies that could help us write tests for every cell in that matrix. The problem was that at the time, the majority of them were about ad hoc testing of resiliency. You'd put one between you and the service for a short period, maybe write a one-off test and reproduce an incident, then ship and move on. They weren't made for continuous integration, running hundreds, thousands or millions of tests a day. Reloading configuration all the time just didn't work out. It was icky and slow.

## Toxiproxy

<!-- t=983 -->
So we ended up doing what we didn't want to do in the first place, which was to build our own proxy. We built Toxiproxy to solve all the problems we had with testing resiliency. As it turns out, it's very easy to write a proxy in Go.

<!-- t=1006 -->
Toxiproxy has a very simple HTTP API that lets you create proxies on the fly and instruct them to start slowing down, limiting bandwidth on a connection, closing connections and so on. The API lets us do anything we want to those streams. In a test, we wanted to be able to tell Toxiproxy "MySQL is down", run the test, and then bring MySQL back up so it doesn't leak into other tests. It has a CLI as well for ad hoc testing. Some of the other proxies can do that too, but we built it into Toxiproxy as well.

<!-- t=1047 -->
A toxic is like a function on top of a TCP stream that lets you manipulate the bytes on the fly. The toxics include timeout, latency, closing connections, limiting bandwidth and a bunch more. But you could write any you want. You just get a raw byte stream. You can choose to sleep, or manipulate the bytes, or whatever you want. It's all open source, and you can fork it and write your own toxics.

<!-- t=1073 -->
It's really stable. We've been using it for three years at Shopify and have made some modifications along the way, but in its current form it's providing a lot of value for us. An alternative has also emerged out of the community called Toxi. We haven't used it a lot, but it may be worth looking into if you find limitations with Toxiproxy.

<!-- t=1089 -->
The HTTP API is just simple REST, and you can write a driver in a very short time with a good HTTP library. You create proxies, update them with where they're listening and what they're doing, add toxics, remove toxics and so on. We already have clients in about half a dozen languages. The Ruby and Go ones are the ones I've personally tried, but I've heard success with a bunch of the others. Either way, writing your own is really simple.

<!-- t=1120 -->
If it wasn't already clear, this is how Toxiproxy works. Your client connects to Toxiproxy, which maps a port to another service. Every service has its own, different port exposed from Toxiproxy. Through the Toxiproxy API, you manipulate the stream between Toxiproxy and the service it's talking to. You can see the little poison there, which is a toxic. Maybe it's slowing the stream down, limiting the bandwidth, or doing something different.

## Drawbacks of Toxiproxy

<!-- t=1152 -->
This is not all butterflies. There are drawbacks to Toxiproxy, and these are some of the big ones to be aware of if you're thinking about adopting it:

<!-- t=1160 -->
- It's not designed for production environments. Some people put Toxiproxy in production and complain that it doesn't have the features for it. It was never designed for that. There's no reason it couldn't be changed to do that, but a bunch of things wouldn't work in a production environment, such as having to configure every Toxiproxy on all the hosts. If someone wants to work on that, I'm here for office hours and would love to chat.
- We don't have many protocol-aware toxics that can do things like corrupt protocol-specific TCP streams. With MySQL, for example, we could simulate an unindexed query, but others are good.
- You're putting a new thing into your environment, which can be more of an investment depending on how flexible your environment is.
- It's a thing that can fail. As mentioned, we've run this for three years in CI. Billions and billions of requests have gone through it, and we don't see anything that can be traced back to Toxiproxy at this point. But with your workload, that may differ.
- Your application needs to be configured to connect everything through Toxiproxy.

## Demo: the Toxiproxy CLI

<!-- t=1226 -->
Now a quick demo to give you a feel for how Toxiproxy works with the CLI. Let me get this in full screen.

<!-- t=1238 -->
First we start a Redis server, which is very simple. Redis is now started on this port. Then we start a Toxiproxy server. So we have just the two things, and nothing is connected at this point.

<!-- t=1262 -->
Now we'll use the CLI, which can call all the different actions of the API. You can list, inspect a proxy, create new proxies and so on. If we do a list now, we'll see there are no proxies right now. If we run create, we'll see the simple commands for creating a proxy to Redis. We'll call it redis, listen on port 20,000, and set the upstream, which is where the original service is, to the port Redis showed before.

<!-- t=1305 -->
This is now created. If we list now, we'll see redis, the port Toxiproxy is listening on and facilitating the connection, where it's going in the end, whether it's up or down, and whether any toxics are applied.

<!-- t=1320 -->
If we now connect with the Redis CLI to this service, it works through Toxiproxy. We can see this all works going straight through, nothing fancy there.

<!-- t=1333 -->
Now we want to make it slow. If we use the toxic API here, we can see the different toxics I talked about before: introduce latency, throttle the bandwidth, close the connection slowly, timeouts, slicing up the bits with optional delays in between in the stream, and things like that.

<!-- t=1355 -->
To create a toxic, we run the Toxiproxy CLI `toxic add` against the redis proxy. The type we're going to use is latency, and we give it a latency attribute of 1,000 milliseconds, so a second.

<!-- t=1380 -->
If we list the proxies again, we can see there's now one toxic. If we inspect the proxy with that name, we can see all the details about the toxic. It's a downstream toxic, which means the request goes to Redis fast, but the response comes back slowly. We can see it's applied 100% of the time. The jitter is zero, so it's always going to be a one-second delay.

<!-- t=1408 -->
If we connect again to the port Toxiproxy is facilitating, we can see the handshake was already slow. If we try to get a key, it's going to take exactly a second. That's what we told Toxiproxy to do. If we had added some jitter, it might take a bit less or more than a second. This will continue to be the case: exactly a second for all of these single round-trip commands. Then we could remove the toxic, or add another toxic, or whatever we wanted, but I think that's enough for now. Let me go back here.

## Applying Toxiproxy to the matrix

<!-- t=1447 -->
Now we had a tool. Back to a very panicked infrastructure team at the time. The resiliency matrix still had some question marks. Some things were green, some were yellow, as we had figured out with mocks and iptables and TC. But at this point, we finally had a really good solution for testing every one of those cells. So we got down to work and started writing Toxiproxy tests for all of them.

<!-- t=1473 -->
If we look at the test from before with the mock, now we tell Toxiproxy to take the cart service down at the network level. Note there's no negotiation about what you're talking through. It's at the lowest possible level, and the driver is going to give you the right error. You hit the collection of the storefront section and make sure you get a 200.

<!-- t=1495 -->
Maybe this test fails, and then you fix it. It gives your developers a really great feedback loop, because they now have a reproducible error they can debug and provide a fallback for, like the session example from before. In this case, maybe we just show an empty cart, or show you an error if you add to the cart, but you can still browse the products.

<!-- t=1518 -->
Around this time, we were also investing in circuit breakers, bulkheads and all the resiliency primitives we needed to handle these incidents. Toxiproxy became a phenomenal way to test those as well, to make sure they were doing the right thing.

<!-- t=1531 -->
So now we colored up the matrix. We had a test for every cell. Some were bad, some were good, some were in between. But we could delegate out to the teams owning the respective areas, and within our own team, to fix all of these cells and get each as close as possible to completely up or only partially degraded. This was an amazing way for us to retrofit resiliency into our application. We had a very good overview of exactly what would happen in all these cases. There's a story similar to the session one for MySQL going down, or the cart service, or the inventory service. We now had good confidence that we could handle all of these scenarios.

<!-- t=1579 -->
At the end of the day, what's more proactive than a test? You hit this in the development environment and get your developers to think about it as early as possible in the development process. We've seen examples of developers using a service and finding out too late that they shouldn't be. For example, perhaps a team wants to use the search service as part of checkout, but we have reservations about that. In this case, as soon as you write your first prototype, a test will fail and tell you that you can't do that, or at least to go talk to someone.

## The toolbar and incident tests

<!-- t=1614 -->
Another tool we developed around this time was a toolbar, where developers could see what would happen if various components were slow, down or having some kind of problem. We already had a performance toolbar internally, where developers could see what was taking the most time on an uncached request. We added a little skull and all the different services, and you can see here how all the toxics are exposed. You can add a toxic, add latency and things like that. This way our developers could see what impact latency had on our customers, and what impact it had when a service was down.

<!-- t=1662 -->
It also let us handle those one-hour incidents. We could now write a test that made sure that when we fixed the root cause of an incident, at least from the application side, no one broke it again. All the hard work of making the application resilient to that failure was now protected.

## Testing versus chaos engineering

<!-- t=1685 -->
Then you reach a very desirable state. If you remember the matrix from before, if you have test coverage for your incidents and for your resiliency matrix, your known knowns and your known unknowns are covered really well by tests. This means you're starting to be ready for chaos engineering. We had a quote before that chaos engineering is really about finding unknown weaknesses in your system, the unknown unknowns. When you run out of incidents from the resiliency matrix, or past incidents to cover, you're ready for chaos engineering.

<!-- t=1716 -->
I made a little crude and unfair comparison of various chaos engineering practices. None of these are perfect. Testing is good because it's very proactive. The automation aspect is great. It's a very good way to handle failures very early in the development cycle. The maturity required is low: introducing Toxiproxy into an existing test suite is fairly easy for most applications. Production impact is zero.

<!-- t=1755 -->
But it has drawbacks. Unknown unknowns are not exposed by testing. You can only test what you know, what you've seen before, or what you can think your way towards. The authenticity is not super high. You can't do very complex end-to-end testing. As I mentioned, Shopify was a very monolithic application at this point. Very complex things, at the nth degree of services, causing outages that way, were not something we were seeing. The incidents mostly fit within the resiliency matrix. But if you have a very complex stack, that may not be the case anymore.

<!-- t=1796 -->
If we move towards chaos engineering and doing things in production, the authenticity is very high. It doesn't get more real than production. But the maturity required is also higher. You can have automated chaos engineering with tools like Latency Monkey, Chaos Kong and Chaos Monkey in production and limit the impact as much as possible. That is certainly possible, and it becomes a very good tool that covers a much wider, n-dimensional resiliency matrix, which Toxiproxy just can't do as a proactive thing. But it requires a significant amount of maturity. You now have to solve the problem of how your developers should think about chaos. Do they get an email when a new weakness in the system is discovered? Do you integrate it into your CI environment? How do you bridge the gap between production and CI? Those are fantastic problems to solve, but perhaps not the most urgent, and not the first you should be thinking about.

<!-- t=1859 -->
Similarly, if you get very good at automated chaos engineering, the production impact can be extremely low. You can find weaknesses in your system with very powerful tools, by directing a very small proportion of traffic to them to see whether it changes business metrics or changes anything for the customer at all. These are great tools, but there is no panacea in chaos engineering. You have to be conscious of where you are in terms of maturity before you figure out where to invest first.

<!-- t=1895 -->
No matter how far you are on the spectrum of maturity, tests don't stop being a useful tool. You still want to write tests to make sure your circuit breakers and bulkheads are doing the right thing, and that things are resilient to the best degree, at least to everything they talk to at the first level. But while they continue to be useful, they stop being a useful proactive tool. That's why you need chaos engineering to explore the unknown unknowns.

## Where Shopify is on the maturity model

<!-- t=1929 -->
If we go back to that pyramid from earlier: as the number of services you have increases, as the scale you're running at increases, as the surface area of your application grows, as the number of incidents increases and your SLOs drop, you want to move up and up on this maturity model. Very few companies in the world are at the very top, A/B testing whether an incident actually affects key metrics. But a lot of companies are at the very bottom and haven't thought about what single points of failure they have. I hope some of you are a little scared and a little eager to go and draw your own resiliency matrix, because I bet that if you've never done it before, you're going to be surprised.

<!-- t=1973 -->
Shopify is somewhere in between. Our next step would be chaos engineering. We're very confident in our circuit breakers, our bulkheads and our automated testing, but we have not adopted Chaos Monkey in production yet. The ROI just hasn't been super high, because we've gotten so much value out of Toxiproxy alone. If we had a more complex stack with more services, that may not have been the case. But at this point, the ROI on moving to chaos engineering in production is not super high. It is something I think we'll start investing in in the very near future.

## Scaling resiliency to the platform

<!-- t=2010 -->
The last thing, which we're starting to think a lot more about now, is that while we've gotten very good at resiliency in Shopify core, the monolith, we need to start scaling this to the platform. The number of services internally is increasing, but not all of them have resiliency in place. We don't want to commit all these mistakes twice. If one incident happens in one app, every other app should become stronger as a result. Maybe they get a Toxiproxy test automatically, or something else.

<!-- t=2048 -->
What's on the screen is a tool we have internally called ServiceDB. It's a list of all the services internally and how they fare on various metrics. We know a good service in production should have a paging schedule, track its uptime, have documentation and have good dashboards, and all of those key metrics are marked here. Any service that is hitting significant traffic or making money in production should be faring fairly well on all of these scores. A score we want to add in the future is a resiliency score: how much effort have you put into making your application resilient to the failure of other services in the ecosystem, or perhaps outside of Shopify in general? This has been a really awesome tool for scaling the engineering practices we've found at Shopify to other applications, and for making application owners aware of the weak points of their applications.

## Conclusion

<!-- t=2105 -->
That brings us to the conclusion of the talk. I highly encourage you to think about how you can incorporate resiliency testing into your test suite. For smaller applications, this can really be a matter of hours. If there's already a client in your language, and you're running, say, a Docker containerized development environment, setting up Toxiproxy should not take you a lot of time. But it could expose some significant weaknesses in your production environment that you may not have seen in incidents yet. It's a really good first step on the journey towards a much more resilient application. I don't think there's an excuse for any application in a production environment not to have resiliency tests, unless it's very early in the prototyping phase.

<!-- t=2156 -->
It's important to understand the limitations of all resiliency tools. Toxiproxy does not cover unknown unknowns. It's just a testing tool, for use with methodologies like the resiliency matrix to find the test cases to write, or with other sources like incidents or trying to break something in production some other way. Depending on your scale, your surface area and the number of services you have, the tools you require will be different, and you will most certainly have a different resiliency story than we do.

<!-- t=2188 -->
The ROI for us from Toxiproxy alone has been very high, enough that we haven't had to invest significantly in chaos engineering just yet. We're intrigued by it, but all engineering efforts come with an opportunity cost, and other things have needed more attention in the short term.

<!-- t=2210 -->
That's really it. I'll be available for office hours outside, after one session and at the very end of the day. If you're interested in helping out or contributing, if you have questions, if you really need something like this in your production environment, or if you have something similar, we'd love to talk to you. Thank you so much.

<!-- t=2237 -->
**Host:** Thanks so much, Simon. That gave me quite a lot to think about. We have about a 12-minute break now. Simon's going to head over to the office hours area, and he'll also be hosting office hours at 2:50, along with Colin, who is doing our next talk. Colin is going to talk about how Google continues to scale our monolithic source repository and continues to evolve and remain agile despite having 2 billion lines of code. That talk is coming up at 2:10, in about 12 minutes. Thanks.
