# Talk at SRECON EU: Advanced Napkin Math: Estimating System Performance from First Principles

2019-10-03 · Recording: https://www.youtube.com/watch?v=IxkSlnrRFqc · Slides: https://speakerdeck.com/sirupsen/advanced-napkin-math-estimating-systems-performance-from-first-principles · archived copy: https://archive.sirupsen.com/recordings/napkin-talk.mp4

_Lightly edited automatic transcript: speech-to-text, cleaned up by an LLM. It may contain errors; the recording is the source._

## Introduction: conference-driven learning

<!-- t=0 -->
Thank you for showing up in this room. I'm really excited to talk about this topic. I didn't really know a lot about it when I submitted this talk. I had a manager once who encouraged me to submit conference talks on systems we hadn't built yet, so that we'd have to build them in time for the conference. We called it conference-driven development. In this case it's conference-driven learning.

<!-- t=30 -->
What I'm going to talk about today is using napkin calculations, or back-of-the-envelope calculations (we use them interchangeably), to estimate the actual expected performance of a system in milliseconds, microseconds, even nanoseconds in some cases.

<!-- t=47 -->
The reason I think this is an important topic is that the engineers I've observed around me, both at Spotify and in the industry, tend to have a really good intuitive understanding of how fast a system should be for a certain action. They have a good sense of whether the I/O speed really matters for the overall performance of the system, or how long something might take on the network. Often they can reason out in a few seconds something that might take a more inexperienced developer weeks to figure out, because they've seen it before.

<!-- t=105 -->
So what I want to give you in this talk is a bit of a shortcut, for something that takes many engineers decades to build, along with a framework and methodology for doing these napkin calculations very rapidly.

## Gauging the room: how long does a Redis GET take?

<!-- t=117 -->
I want to start with a gauge of the room. A couple of weeks ago I was doing a test on this laptop. I had a local client written in Rust, so the client's performance is fine, and a Redis instance, and I was doing GET operations on just 8 bytes. I want a feel for how long you think that takes.

<!-- t=140 -->
I'm going to start at 10 milliseconds and go down by 10x each step: 10 milliseconds, 1 millisecond, 100 microseconds, 10 microseconds, and less than 1 microsecond. Raise your hand for the one you think is closest to the actual performance of a Redis GET of 8 bytes on this machine.

<!-- t=161 -->
Who thinks it's closest to 10 milliseconds? A few, not too many. One millisecond? A few more. A hundred microseconds? Quite a few people. Ten microseconds? A little fewer. Less than one microsecond? No one. So we have consensus at around 100 microseconds.

<!-- t=193 -->
The actual time is around 25 microseconds, so 10 microseconds was the closest. By the end of this talk you should have a bunch of numbers like that in your head, to get there for a variety of problems.

## First-principles thinking: Elon Musk and the rocket

<!-- t=211 -->
I think a really good example of someone who has mastered thinking about systems from first principles, and building up an expectation of their cost and performance, is Elon Musk. In the mid-2000s he was rolling around the LA area talking to aerospace engineers at Lockheed, NASA and so on, trying to figure out whether he could build an aerospace company. One of the first problems you face is that you need a rocket to propel whatever device you need into space.

<!-- t=247 -->
He asked NASA, and they said it's probably around 130 million. The Russians could do it a little cheaper, but we were talking about 110 to 130 million dollars to buy a rocket to send his spaceships into space. That just wasn't sufficient. A hundred and thirty million to send something into space didn't allow him to build the crazy Mars-colonizing company he wanted.

<!-- t=279 -->
There's an illuminating quote where he looks at the rocket, not as this thing that costs 120 million, but as a metal thing. There's aluminum, maybe some cobalt for strength in the alloys, carbon fiber, all of these materials. He estimates that it's maybe 90% this and 10% that. Then he went to, I think, his favorite website, the London Metal Exchange, looked up the spot prices for all the different metals, and multiplied the weight of each material by its spot price. He got a number of about 2 million, orders of magnitude less than the actual cost of the rocket.

<!-- t=323 -->
Then he looks at the two numbers, 2 million and 120 million, and concludes that the other 118 million is just bureaucracy and inefficiencies in manufacturing. He was confident he could do much better using modern manufacturing techniques. A lot of these rockets were invented during the Cold War; now we have 3D printing and so on. Finally he got the Falcon 1 down to about a 7 million cost.

<!-- t=350 -->
I think this is a really good example of using first-principles thinking: taking something really complicated, distilling it down to first principles, doing some simple math, and seeing the discrepancy between his naive interpretation of what a rocket is and the actual cost of a rocket. He then arrived somewhere much closer to the naive interpretation. When you have a gap between your expectation and the actual performance or cost of a system, that gap is always an opportunity. Either you correct your mental model of the system, or there's a huge opportunity. In this case there was a big opportunity.

## Applying it to a cloud bill, and base rates

<!-- t=386 -->
Let's apply this to a more everyday example: a boring cloud bill that comes in once a month. Say it's a hundred thousand dollars, or maybe ten thousand, or whatever. By boiling this down to first principles, you should be able to quickly do a calculation along these lines. Say this is a SaaS company. You have ten thousand requests per second, and you know you're responding in about a hundred milliseconds. Multiply those two numbers, and with single-threaded workers you need about a thousand CPUs. Each CPU costs $10, so you should be spending about $10,000 on the main application that's making you money. Maybe you're at 20%, because this is napkin math and you want a little overhead so you're not running at 100% utilization. But where's the other 90% going?

<!-- t=439 -->
You should be able to do these calculations very quickly in the middle of a meeting, to arrive at more interesting discussions rather than just eating up this cost. Maybe you're storing a bunch of data you don't need to store, maybe you left some instances on, or maybe you're hugely over-provisioned. Again, the gap between your naive expectation and reality always presents an opportunity.

<!-- t=460 -->
I'll present the first base rates here. A base rate is a semi-constant number that's really useful to memorize so you can do these kinds of calculations. In this case, I used the cost of a CPU. These numbers are almost all very close to what things cost in a normal cloud, so not dedicated hardware but a cloud where you rent computers from Amazon, Google or Microsoft. If your number is more than 100% greater than this, you're paying too much for your compute. It should be within 10 to 20% of here. I've rounded the numbers to make them easy to remember. It's easy to remember because it's $10 per CPU, and then you go down by an order of magnitude as the performance of your storage decreases.

<!-- t=506 -->
These numbers are hugely useful, and I've found myself using them in so many situations. Say something seems too expensive to do. You do the math really quickly in your head, walking or in a meeting or on a whiteboard, and you say, no, it's not expensive at all, let's do it. Instead of four meetings, where in one you decide that you need to figure this out, in another you decide who will figure it out, in the third you decide that this person will do it because the other person is now working on another project, and in the fourth they come back with a measurement far more precise than you actually need, you can do this in 30 seconds. You remember these numbers.

<!-- t=542 -->
The other numbers that are useful to remember, and I'm going to present a bunch more in this talk, are the base rates for your application: things like median response time, how many transactions you do against a database, how many customers you have, throughput and so on. These are very specific to your application. If you look at the old-timers at a company, you may see that they have a really good representation of how the app behaves in their head.

## Eratosthenes measures the Earth

<!-- t=569 -->
I want to go back to one other good example, outside of computer engineering, of someone who did awesome napkin calculations. This guy, whose name I can barely pronounce (I'm going to go with Eratosthenes), lived around 200 BC. I don't know how controversial it was at the time, but he was pretty sure the Earth was round, and he wanted to figure out the circumference of the globe.

<!-- t=597 -->
Try to imagine how you would do this in 200 BC. What do you have available? Today it would be pretty easy. You could do something with a ruler and Google Earth in a heartbeat, or something even easier, and it would just be very easy to look up. But Eratosthenes needed to come up with a more ingenious methodology to solve this seemingly impossible problem.

<!-- t=621 -->
What he observed was that in Aswan, Egypt, at noon the Sun was directly overhead. He could see that because he couldn't see his own shadow; it was directly beneath him. In Alexandria there was a 7.2 degree angle to his shadow. He knew the distance between Alexandria and Aswan, and he knew the whole globe is 360 degrees. He divides 360 by 7.2 and gets 50, so the total circumference of the Earth is going to be 50 times the distance between those two cities. He arrives at a result that is within three percent of the number we use today.

<!-- t=658 -->
And that was in 200 BC, at a time when the New World and Old World hadn't even met each other. Only in the 1700s did a bunch of Frenchmen, who were figuring out the definition of a meter, make the number more precise, and they had massive grants, seven years and a big team of people. I always think of this guy whose name I can't pronounce every time someone tells me we can't measure something. That's the idea: using very simple calculations to figure this out.

## Fermi and the piano tuners

<!-- t=696 -->
The last person I'll introduce, until we go into computer-related examples, is Fermi. Fermi was a physicist who worked on the Manhattan Project, and he was revered among his colleagues for his remarkable ability to estimate things very quickly and surprisingly accurately. When they detonated the first nuclear bomb in New Mexico, he dropped a couple of scraps of paper, and based on the distance they traveled from where he dropped them, he estimated the yield of the atomic bomb, of which none had ever been detonated in the US before, at ten kilotons. The real answer, which they only got to through laborious calculation over several days or hours (you didn't have much compute power in the 40s), was around 18 kilotons. This was the first time anyone had ever let loose one of these bombs in the Western world, and he was able to estimate it very accurately.

<!-- t=756 -->
What Fermi is really famous for are Fermi problems. You may have heard this one before: how many piano tuners are there in Chicago? This is obviously a ridiculous question and not even that interesting, but the methodology he uses to answer it, through some very simple numbers that probably all of you have in your heads, is really interesting. You can imagine using it to solve almost impossible problems very quickly in computers. It could also be something like: if we put solar cells on all roofs in Chicago, would we be able to sustain the city in terms of energy? You could estimate that through the same methodology.

<!-- t=796 -->
He notes down a bunch of things that he knows. There are about nine million people in Chicago's metro area, which, if you live in America, I'm sure you remember something like. There are maybe around two people per household. One in 20 households has a piano. You might not quite remember that, but you might agree that one in fifty is maybe too rare and one in five is too many; it just seems roughly correct. Pianos are tuned once a year. I don't know if that's true, since I don't think my grandparents have tuned their piano in 15 years, but you could imagine something roughly like that. It gets you a bit of a bookend. Tuning takes maybe two hours. Piano tuners work on a very American schedule: two weeks of vacation a year, eight hours a day. You combine all of these numbers and you get how many piano tunings happen per year, and how many tunings each piano tuner can do. Combine those and you get that there are probably around 225 piano tuners in Chicago.

<!-- t=858 -->
We arrived here from some very simple numbers, and we can see how this could apply to thousands of other questions. Of course the answer is not exactly 225 piano tuners; it would be presumptuous to assume we arrived at the right answer. But it's probably within an order of magnitude. I'd be willing to bet money that it's somewhere between 20 and 2,000, with the peak probability around 225 and the rest scattered around the low hundreds.

## You need less precision than you think

<!-- t=885 -->
What Fermi and Musk observed is that most of the time you need a lot less precision than you think. I read in a book, while preparing for this talk, that if we polled five people in this room for their age, we'd have a 95% chance that the median of the whole room would be within the range of those five. We talk a lot about needing a lot of data points to say anything with accuracy, but a lot of the time we need much less data than we think for the kinds of decisions we need to make.

<!-- t=924 -->
Musk doesn't need to know that it will cost him exactly two million, two hundred thousand and twenty-one dollars and so many cents to build the rockets. He just needs to know, directionally, that it can be a lot cheaper than 120 million when you look at the raw materials. For a lot of the decisions we make we just need a general direction.

<!-- t=943 -->
What I find a lot of decision makers in software design do is prefer very exact calculations, but then they only have time to evaluate maybe one or two options. What I find much superior, and a lot of the decision research I've read agrees, is to go much wider. With much less accuracy in your calculations, you can evaluate a lot more solutions a lot more quickly.

## Example: failing over an in-memory data store

<!-- t=971 -->
Let's use a Fermi decomposition on a real problem. Now we get into the more computer-oriented part of this talk. All the problems here are problems I've actually had, which may make them a little boring, but at least they're real.

<!-- t=985 -->
A couple of years ago we started investing very seriously in our failover architecture from one region to another. As part of that you need to do a bunch of things. You need to fail over your read replica of your database to become the writer in the other region, and wait for it to catch up. We also had in-memory data stores holding data that was important but not critical to lose, things like sessions. We wanted to restore that in the new region and get all the data transferred over. But we were not very comfortable with the replication of the data store we were using, and we'd had some bad experiences with it in the past.

<!-- t=1025 -->
So what could we have done at the time? We could have said: what if we do the most naive thing possible? Dump the entire memory to disk, transfer it over the network, and restore it in the other region. Is that feasible within the constraints of the couple of seconds we have to fail over in?

<!-- t=1041 -->
Again we can use some of these base rates for how a computer performs. We read from memory, write to disk, transfer over the network, then read again from disk and write all of that into memory in the target region. What you can see is that there's really only one number that matters here, which is the network from one region to the other. All the other numbers don't matter too much. I'm going to use gigabyte and gibibyte interchangeably, because "gibibyte" is terrible to pronounce. Transferring one gigabyte takes about a minute, and we have sixteen of them, so that would take about fifteen minutes. Even assuming a very generous compression ratio of two or three times, we're talking minutes to fail this over, which is just not feasible.

<!-- t=1090 -->
This result alone is really valuable, as in the example I gave earlier about having all those meetings. Instead you can just say in the meeting, "Could we do this?", walk through these rates, sketch it out in 30 seconds and say no, this is not feasible. If you want to do this, we need to invest in some kind of incremental transfer of data through replication, or inspecting the LRU for the hottest keys and transferring only those, or something different. Unfortunately we can't do the most naive thing here.

## Big O notation and mechanical sympathy

<!-- t=1120 -->
With that, I'll take a little detour, because at this point it may remind you of Big O notation, which is used in algorithms and data structures to tell us how many times we need to look at a piece of data. If you have an O(n) algorithm, you need to look at every element once. If you have an O(n²) algorithm, every time you look at one element you also need to look at every other element.

<!-- t=1144 -->
The problem with Big O notation is that it's not very sympathetic to how the hardware underneath performs. Take a very exaggerated example: an O(n) algorithm that does random SSD seeks for every single element, versus random memory access. Random memory access is about 4,000 times faster than random SSD access. Big O notation is good for algorithms research, but in this day and age it's very rare to find an improvement in the Big O of an implementation. Most algorithms and data structures research today is about what's called mechanical sympathy: more sympathy for how the hardware works, using CPU caches, using the NUMA architecture properly, limiting I/O, using as much sequential I/O as possible, and all the things that are relevant to how computers actually behave.

<!-- t=1205 -->
The other problem with Big O notation is that it doesn't give us any expected number. If our program is running in ten milliseconds, we have no idea, given Big O, whether that's fast or slow. That's why we need something a little more sophisticated.

## Latency numbers and a Rust benchmark

<!-- t=1219 -->
Earlier I mentioned a bunch of numbers I was using in the failover scenario. This whole thing is by no means my idea; it was popularized by Jeff Dean, and a lot of these numbers are floating around. I guarantee a lot of you have seen them before. I spent some time writing a Rust program to replicate those numbers and see what they would be on my laptop, so that I can run it in our production systems and see how they evolve.

<!-- t=1245 -->
A lot of the time you only get the latency numbers, like how long it takes to do an SSD read of eight kilobytes, in microseconds. But I also really wanted to know what it would take to read a megabyte or a gigabyte, because that's something we find ourselves doing. Just staring at these numbers alone can be really helpful. For example, sequential SSD reads are remarkably fast in terms of throughput. We're getting pretty close to the same throughput from reading from disk sequentially as from memory, but the latency is a lot higher.

<!-- t=1280 -->
Another thing you can take out of this is the TCP echo server, which you may have written at some point: a server that just responds with whatever the client sent. If you're doing that with just 64 bytes, you're getting an overhead of about 15 microseconds. Remember the Redis example in the beginning, at around 25 microseconds? You can now see that most of that overhead is just TCP and traversing the kernel stack. We're only spending about 10 microseconds in Redis actually getting things from memory, parsing the protocol and so on. Similarly, if you have a proxy that's spending one millisecond in overhead, that's a really terrible proxy, because a very simple proxy should be spending tens of microseconds, not the hundreds or thousands.

<!-- t=1329 -->
Something else I thought of while making this talk is that some of you may want to practice this kind of thing, because to recognize these problems you need this as a sharp tool. So I created a little mailing list. If enough of you sign up for it, I might start emailing a problem that needs these base rates, with resources, about once a month. The link will be in the slides later as well: sirupsen.com/napkin.

## Example: slow Australian merchants and TCP slow start

<!-- t=1354 -->
Let's now take an existing system. One of the interesting things about the Musk example was how he created an expectation of the system, and then took the difference between the actual system and his expectation and found an opportunity in it.

<!-- t=1373 -->
A problem I encountered a while ago was that Australian merchants were reporting a two to three second response time for their sites. This didn't jibe with my intuition. We render a page in about 100 milliseconds. The round-trip time between Australia and DC, which is where our data centers are close to, is 250 milliseconds. There should be about four and a half to five round trips. You add all of those numbers up and you get about a second. So why is it taking two to three seconds? Why is there a 1 to 2 second gap between what my napkin math is telling me and the actual system's performance? Is there a bug, an opportunity, in the system, or is there a bug in my mental model of the system?

<!-- t=1414 -->
It turns out that in this case it was a bug in my understanding. What I had forgotten is that TCP has slow start. It doesn't know the size of the pipe between the client and the server, so it can't send all the data at once. It has to inch its way toward figuring out the bandwidth between the client and the server. Initially it sends 15 kilobytes from the server to the client. That's the default in Linux; you can tune it, but 15 kilobytes is very standard. Then, if all of those packets made it safely to the client, it sends 30, then 60, then 120 and so on.

<!-- t=1451 -->
Then I graphed how much data is transferred after how many round trips. My assumption of one round trip for HTTP was wrong; it depends on the size. When I looked at these merchants' sites, some of them were 150 kilobytes, which requires three full round trips on that initial connection. So this was a gap in my understanding.

<!-- t=1472 -->
On my personal website, I thought I was making everything fast by base64-encoding images, but that actually just got the size to about 15 kilobytes. If your site is below 15 kilobytes, that's the fastest possible; if it's above that, you're going to have a little more trouble. This was an interesting example of doing the napkin calculation, looking at the difference, and arriving at a new appreciation of the system.

<!-- t=1495 -->
Of course, you also want to memorize some base rates for how fast a packet travels around the globe. If you remember these, you can generally figure out from one point to another, within 50 milliseconds or so, how fast a packet will go from A to B.

## Example: filtering products in a Shopify collection

<!-- t=1518 -->
Finally, I want to talk about a very real problem we have at Shopify. A big part of Shopify (now we do a lot more) was originally that you have an online store. What is an online store fundamentally about? It's about narrowing down what you want to purchase. A merchant potentially has hundreds of thousands or millions of products, and you need to find the one that's right for you. You go to the men's collection, which shows you everything for men, or you go to the women's collection, and in there you might filter: I want to see only dresses, or only shoes. You end up with all these rules applied: I only want to see shoes, they need to have shoelaces, they need to be brown, I want them to have "cat" in the title for whatever reason. You keep layering on until you find the thing you want.

<!-- t=1569 -->
The first part of solving any sufficiently complicated problem is to define the problem. So first we make some assumptions and do the Fermi decomposition. There are probably around two million products for a merchant. A collection can filter by up to 64 rules; the merchant adds the rules, and there's a limit of 64. Each rule can take a form like "contains the string cat" or "price is less than $300" and so on. Each product has all of these fields attached to it, like the average rating, and on average these product rows are going to take up about half a kilobyte of information.

<!-- t=1622 -->
What we want to do is serve queries that narrow down the products we want. I'm writing this out as SQL because it's something most people are familiar with. We want all the product IDs where the price is less than $200, the product description has "sneaker" in it, the product rating is more than three, and the product color is one of black, yellow or red. We want to order that by the product color, and we want page ten. If you've spent a lot of time with SQL, you'll understand how the database will cry if it sees a query like this for a sufficiently large merchant. So this is a problematic query, and we have the constraint that we want to serve it really fast, so that your browsing and filtering experience is fast.

<!-- t=1662 -->
The way I like to think about it is as a decision tree. At the root you have the problem you're trying to solve, and you keep refining it and making it as concrete as possible, so it looks more and more like a physics problem you might have been handed in high school, and less and less like an abstract thing with people in a room arguing about which features they want.

<!-- t=1688 -->
The first thing to do is consider the naive solution. You do the napkin math: two million products, each half a kilobyte, multiply these numbers together, and you arrive at about a gigabyte of data for that worst-case merchant. How long does it take to traverse one gigabyte of memory sequentially? About 100 milliseconds; we remember that from earlier. Is that feasible? Not really, since we want response times under ten milliseconds. But it does tell us something useful. A hundred milliseconds to traverse all of that data with the most naive solution we possibly can is pretty good. It could be a lot better, but maybe we can use some of that naive approach in our solution.

<!-- t=1732 -->
Now maybe we consider another one: an inverted index as a data structure could be useful. MySQL is not going to be very good at wildcard matches like "sneaker"; it just doesn't cope very well with the data structures it uses. What is an inverted index? You can think of it as a big hash map where the key is a string, in this case all of the words in your product descriptions and product titles. For example, we might have a sneaker that's called "big GC sneaker" or whatever, and then "sneaker" would be a key in that inverted index. The value for "sneaker" would be all of the product IDs that have "sneaker" in the title. Then we can do these wildcard queries very fast, because we can just look at all the products in there and filter further.

<!-- t=1783 -->
So maybe an inverted index is really useful to serve some of this. We don't really know. The less-than and greater-than operations are going to be a bit trickier, but maybe it's useful for part of the operations. When we do the math, we say each title is maybe a maximum of 13 words, and the descriptions maybe 27 words on average, so the total size of this whole index is only going to be 64 megabytes. We also know from earlier that a gigabyte of RAM costs about $1.10, so 64 megabytes is in the low cents. This seems economical as well, so it's an interesting result.

<!-- t=1824 -->
Something else we might look at: right now these collections are stored in MySQL, and that's kind of slow. Is it slow because B-trees are fundamentally a poor data structure for these kinds of things? We can imagine doing a little more napkin calculation: for 64 of these rules, how much do we narrow it down? And going back to the problem and looking at what the shape of the problem would have to be for B-trees to be really good.

<!-- t=1846 -->
Now this is very unsatisfying, because I don't know the solution to this problem. If you have some ideas, let me know. But this is really to illustrate the process of using napkin calculations to go wide. I said earlier that I think it's vastly superior to go wide with approximated calculations than to go all in on trying to use B-trees and do all the exact calculations. You're also just going to fall in love with that solution very easily.

<!-- t=1872 -->
I really think that breaking down the problem this way, and continuing to go back to refine it, is required if you want to step out of the box of iterative improvements. If you need to make an order-of-magnitude improvement to your design, you really need to understand the abstraction levels below you, understand the layers around you, and put the puzzle together in a completely new way. You just can't make step changes or order-of-magnitude improvements to your systems if you're only iterating and confining yourself to the abstraction you're currently at. You're going to have to combine more things around you.

## How to memorize and practice base rates

<!-- t=1907 -->
Finally, how do you memorize and practice these base rates, first-principles thinking and the Fermi decomposition of problems? For some reason, today memorization is a thing we're told we shouldn't practice. I don't really know why it's gotten this reputation, because I find it immensely useful. The excuse is that you can just look it up on your phone, but a lot of the time we don't like the friction of going to our phone and finding the base rate. You're not as likely to do that when you find yourself needing this in the middle of a conversation over dinner with only napkins available, or at a whiteboard. You need to memorize these numbers. They're not that hard to memorize.

<!-- t=1948 -->
What I recommend is spaced repetition, which essentially shows you these base rates on an exponential schedule so that you will remember them. There's software called Anki, which is a power tool for that. If you're not familiar with flashcards, there's a messenger bot called [name unclear], where you can say, "Hey, I want to remember this," and it will send you messages once in a while asking, "Do you remember how long it takes to read a gigabyte from an SSD?" If you answer incorrectly, it will show it to you more often. That's a really good way of doing it.

<!-- t=1978 -->
The other way is to sign up for the periodic challenges I mentioned earlier, so you get something in your inbox once in a while to practice with. You could also spend some time developing your own base rates, which is a really good way of remembering these numbers if you're interested in mechanical sympathy and understanding more about how computers work. You can read the source of my project; there are a lot of base rates in the readme that are missing, and you can contribute if you'd like.

<!-- t=2002 -->
And then finally, of course, apply these numbers to your own systems. I'm sure that right now you have a problem that is begging for some napkin calculations, or a problem where you can dramatically widen the search space of solutions you're considering.

<!-- t=2017 -->
As I said earlier, I think napkin calculations are great because every time you do one and look at the real system's performance and at the difference, there's always an opportunity, either for you to improve the system or to improve your mental model of the system. Thank you.

## Q&A: surgery over 5G

*Host:* Thank you, Simon. We have five minutes for questions, so anyone?

*Audience member:* I'm going to pose a problem to you. I was listening to a podcast where AT&T was pushing 5G cellular service, saying it's going to be great, so much faster. They talked about the speed at which you can get data on your phone, and the thing they said was that using 5G you'll be able to have a surgeon in LA do surgery on a patient in New York. I wanted to puzzle this through: is that possible? You'd have a robotic toolkit in a patient being controlled by someone in LA, so it's dominated by the 60 millisecond round-trip time. I thought, I don't care how fast I can get from here to the cell tower; it's that New York to LA leg that matters. The last thing I'd want is for that to happen to me. Is that something that would be possible? It seems to me you'd need a lot more AI to control the speed of the tools. I don't think you'd get the accuracy and the quickness. If the tool starts moving towards something and bumps into something, you want to stop it.

*Simon:* It's funny you mention this. As a quick aside, I was going to have a part of my presentation about decision-making in software. One of the examples I had is that we always assume that if the problem is really big, we need a really big and complicated solution. In the medical field, for example, if surgeons use a checklist to go through everything before surgery, there's something like a 10% decrease in mortality rates. So I posited: say you instead invested ten million in a robotic arm. Is that going to decrease your fatality rate more than doing something simple like a checklist? I wonder if that applies here too. I find it a weird example from AT&T, because yes, I would also say it's latency-dominated, and you should be able to get high-fidelity video over just fiber. So I'm also puzzled by it.

*Audience member:* I think it was a great way to toss something out that nobody would understand and that sounds absolutely terrific, so go buy 5G now.

*Simon:* I'm glad it went through your filter and you said, nah. It's probably great, but I don't think it's 5G-enabled. 5G is probably going to be more about sending videos and social media and things like that.

*Audience member:* The other thing about 5G is that I doubt the surgeon is going to be sitting somewhere doing it on his cell phone. He's probably going to be in an office that's hardwired with fiber.

*Simon:* I just imagined a scenario of a surgeon sitting in a Starbucks going, "Ah, this is great, 5G is the best." It seems like something where you should be in a room with ten feet of metal to make sure you're not interrupted. I agree.

*Another audience member:* Human reaction time, the feedback between your hand and your brain, is about half a second. So from my point of view, 60 milliseconds is not the most important factor in that computation.

*Simon:* There we go. That's an awesome example of using base rates. You're saying this is the same as the failover example, but actually this latency doesn't matter at all; it's all about the human reaction time. 500 milliseconds versus the 60 millisecond round-trip time. And I'm sure there's also something to be said about whether people take the same actions through a screen as they would in person. I don't know; that would also be interesting.

## Q&A: where the first-principles slide came from

*Host:* Terrific. Any more questions?

*Audience member:* Can you go back to that earlier slide, the one about order of magnitudes? That's one of the most impressive slides I've seen in the last number of years, and it's something I've been fighting over throughout most of my career. Where did you arrive at that? Is it something you gradually arrived at, did you find it somewhere, is it your own personal wisdom? If not, I was curious where you found it. Is there a secret book?

*Simon:* No. This quote is not from anywhere specifically, though it probably is from somewhere I don't remember. It's like reading a book you haven't read in a long time and recognizing a lot of ideas you didn't attribute to it. But I wrote this because I've been really fascinated in the past year by scientific advancement: reading about the invention of the steam engine, and how we got out of whaling to having light, and things like that. I feel like one thing all of these inventors have in common, and I'm using Elon Musk as a modern example because most people in the room would be familiar with him, is that they have such a deep understanding of the systems.

<!-- t=2402 -->
I keep seeing that in my colleagues around me as well. There's a guy sitting in this room, and what I learned from him is that he just continues to be in the source code of whatever system is underneath the one that's having a problem. He's always opening the source code, and people in meetings were doing the same thing. The people who don't tend to do that are too constrained. They think about the problem and its attributes rather than the fundamental data structures and the shape of the numbers and data. So I can't attribute this to one source, but I think if you look at history through this lens, it's something a lot of those great inventors have in common.

*Host:* Thank you so much. A round of applause for our speaker, please.

*[Applause]*
