# p99 Podcast with Sam Lambert: Scaling AI infrastructure

2026-02-05 · Recording: https://www.youtube.com/watch?v=7YLa28TvzjU · archived copy: https://archive.sirupsen.com/recordings/p99-scaling-ai-infrastructure.mp4

_Lightly edited automatic transcript: speech-to-text, cleaned up by an LLM. It may contain errors; the recording is the source._

## Intros

<!-- t=1 -->
**Ben:** All right, we are live. Welcome, everybody, to the stream. I've got Sam from PlanetScale and Simon from Turbopuffer, and we thought it would be fun to hang out, talk about databases, and talk about scaling infrastructure for any company, but AI companies in particular. We also want to hear about Sam's and Simon's experience running databases at massive scale across the industry. As you file in, feel free to ask questions. We want this to be a fun, casual stream, so if you have questions about how they scale, how they do things, or how they operate their companies, drop them in the chat and we'll go through them throughout. Sam and Simon, give a little intro about who you are and your background. Sam, you take it away.

<!-- t=58 -->
**Sam:** Hi, I'm Sam, CEO of PlanetScale. I'm very obsessed with databases, and I know Simon is too. I previously worked at GitHub and Meta, on very large-scale services and websites, and that's where I gained my experience doing this. Now I'm building a company for everyone. I'm very bad at intros generally.

<!-- t=87 -->
**Ben:** I'm sure you're great at it. Simon?

<!-- t=90 -->
**Simon:** It's because we're both immigrants to the new world, Sam. You're also ESL.

**Sam:** Yeah.

<!-- t=97 -->
**Simon:** I'm Simon, co-founder and CEO of Turbopuffer. We build a database too, specializing in search. I cut my teeth on databases at Shopify. I worked on more or less all the scaling of infrastructure there and was thrown around at whatever needed to be scaled, and generally the database had become the fundamental bottleneck. That's also how I met Sam. There was a tight relationship between our two companies around scaling Ruby on Rails with MySQL, and we ended up sharing a lot of lessons and code. That's how we got to know each other more than a decade ago.

<!-- t=133 -->
**Sam:** We've tried to backtrack, and I don't think we can figure out the first time we met, but we've known each other a long, long time.

## Early sysadmin days and the first Turbopuffer deploy

<!-- t=141 -->
**Simon:** There was a particular database that did search at Shopify that was the worst one to be woken up by at 3:00 a.m. Maybe you had this experience too. It was the search cluster. That's part of why I'm building this database company today: we're searching more data than ever, and the ones I've operated in the past were a really, really bad time.

<!-- t=164 -->
**Sam:** Better than Sphinx, though. Isn't that a MySQL extension? God, I haven't thought about Sphinx for so long. One of my first admin jobs was back in the day when e-commerce was a craft. Do you remember, pre-Shopify, when agencies built e-commerce sites? I worked for a pretty successful one. It was a factory line for e-commerce websites, and I was managing MySQL and Sphinx. It was brutal.

<!-- t=196 -->
**Simon:** Was that your first job? This is pre-GitHub?

<!-- t=198 -->
**Sam:** It wasn't my first job, but it was the first proper sysadmin job I had. I've had all sorts of jobs outside tech, and I really wanted to break into tech. I convinced a local company to pay me next to nothing. I was telling this story last night: I told them, you do not pay enough money to hire someone who knows what they're doing. I do not know what I'm doing, but I will learn very quickly. And I did learn very quickly. That job was being a true sysadmin, SSHing onto servers and fixing stuff manually.

<!-- t=237 -->
**Simon:** I feel like it's going to be difficult to relive the experience of just SSHing into random servers and grepping and tailing through files that were too big. So many things I do today still go back to that. In fact, the first version of Turbopuffer, when I launched and it was just me, was a single binary on one eight-core box in GCP. The way I deployed was watching the logs and, when there were no requests for a few seconds, deploying. I did development with an rsync in another pane. That was the resentful first ship.

<!-- t=284 -->
**Sam:** That's beautiful. That's one better than FTPing code to servers, which is how we used to ship code. Do you remember those days?

<!-- t=298 -->
**Simon:** Whew. That was peak productivity. You hooked Dreamweaver straight into prod, turned on live errors on the PHP server, and just let it rip. I don't think I've been more productive since. I think all the AI, Cursor and Claude Code or whatever, is only getting us back to that level of productivity.

<!-- t=318 -->
**Sam:** It's beautiful. It's phenomenal to feel that flow and get that direct loop back constantly. It's addictive. It truly is why you build that way. Obviously, deploying for you and me now, or for our companies at least, is completely different and massively more complicated.

## Rolling restarts: MySQL vs. Postgres high availability

<!-- t=338 -->
**Simon:** So if you're deploying to all of your MySQL clusters, how do you do that today?

<!-- t=342 -->
**Sam:** It's done in various phases. It goes through a staging environment with tons of synthetic testing, then rolls out in a certain order for various types and tiers of cluster. It really depends on whether it's Vitess, MySQL or Postgres. The way we talk about it internally is that we're building our way back to the flexibility Vitess has, for Postgres. It was very different for us organizationally. We use CI/CD for all of it, but we had such a continual deployment mechanism for Vitess. Because of how Postgres handles HA, we have to go a lot slower, because it makes HA very hard. Are you familiar with why it's hard?

<!-- t=399 -->
**Simon:** I just know that every time I've looked at the docs for Postgres replication, it makes me really sad and I miss binlog-style replication in MySQL. But I presume it has gotten there. Why would that matter for HA? Don't you just get a pointer and cut off the writes for a few seconds?

**Sam:** No.

<!-- t=417 -->
**Simon:** Okay, so why is it so hard?

<!-- t=420 -->
**Sam:** Let me ask you how you would solve something, and then we'll go through it. You have a three-node MySQL cluster, a primary and two replicas. You want to change a non-dynamic variable on the primary. Connection limits, back when you and I were admining MySQL, were not online; now you can just change them. Imagine you need to restart the MySQL primary to change a variable. How would you do that in the most highly available way possible?

<!-- t=460 -->
**Simon:** My first question was going to be what scale you're at, because ideally I just shut down the writer in the middle of the night and do it then, to not deal with any of it.

<!-- t=469 -->
**Sam:** Your goal is minimal disruption. It's a real website.

<!-- t=474 -->
**Simon:** Nothing at all, or minimal? How far on the spectrum?

<!-- t=477 -->
**Sam:** Let's do the next step.

<!-- t=478 -->
**Simon:** The next step is you go to the replica, shut it down, change the variable, bring it back up, and let it catch up on the binlog. Then you have a script that looks at the GTID, how far along the binlog you are on the replica and on the primary. When they're reasonably close, you stop the writes on the primary, let the replica catch up, and once the watermark is the same, you change the pointer. I remember doing it with a proxy, but you might not have a proxy. Did you ever do it without one? We just do it at layer four, so you change the IP to point to the secondary, turn on writes on the secondary, and roll the variable change on what was the primary. That's how I would do it in MySQL. Is that not how it works in Postgres?

<!-- t=547 -->
**Sam:** You can do a bit of staging ahead of time. You can choose which replica is going to be the new primary and put all the other replicas behind it, so you've just got one primary left, and then it's a very quick cut: read-only mode on the primary, move, seconds.

<!-- t=564 -->
**Simon:** Yeah, if you've done some clever proxy stuff, you pause. The MySQL protocol is pretty good at redirecting and session management. And it works.

<!-- t=577 -->
**Sam:** You cannot do that in Postgres, and it's due to a philosophical difference. I really went in on discovering this, and I keep promising everyone I've nearly finished a blog post that goes through it. I didn't want to say it was wrong, but it has this kind of issue: the primary in a Postgres cluster cares about the state of the replicas, down to the variables. So you have to change the config on the primary first, and you absolutely cannot achieve that without severing connections.

<!-- t=622 -->
**Simon:** Hang on. In MySQL the replicas don't care, right? The only state the primary has is the list of them, so it knows where to send the binlog.

<!-- t=637 -->
**Sam:** Right. Postgres has a two-way protocol, and if the variables don't match, it just errors. It will not proceed. Again, I don't want to say it's wrong. It clearly comes from the early philosophy of Postgres, which was very much about correctness. I really dug into the founding philosophies of the two projects. Replicas are meant to be a byte-for-byte copy that you can always go to for something fully consistent. So philosophically you wouldn't have anything different on one versus the other. That's technically correct, but very inflexible.

<!-- t=681 -->
This also shows up elsewhere. Logical replication is improving, but it still doesn't replicate DDL, and there are caveats. It's nothing like MySQL's replication, which doesn't actually care. I know that's one of the criticisms of MySQL, that it's not correct enough, but for those listening who haven't done this: no matter what your client is, whether it's some Go code or whatever, if you can connect to MySQL and ask for the replication stream from a certain GTID, and the binlogs are there for that GTID, it will start sending you data. You can play all sorts of really good tricks, like the black hole storage engine, for example.

<!-- t=734 -->
**Simon:** So good. We should do a stream on the black hole storage engine.

<!-- t=738 -->
**Sam:** It has issues, but it's brilliant.

<!-- t=742 -->
**Simon:** I've always wanted to build an ops system on black hole.

<!-- t=745 -->
**Sam:** If you know what you're doing, it's very flexible. You can replicate from a MySQL node and say a given table should always black hole, so you just get the subset of tables you want. You can do replication filters as well. The goal of replication is not necessarily to be a full copy, unless you want it to be. It comes with other downsides: with MySQL you have to start from a backup, and Postgres has better cloning. So it's just trade-offs. Postgres also has a lot more session state, so resuming sessions for the client is tougher. We're working through basically every single one of these to get back to where we were.

<!-- t=797 -->
With Vitess and the way we deploy MySQL, if we wanted to hit a button right now and restart every MySQL node we have, it would take a very, very long time, because it does it responsibly so as not to break things, and we hit Amazon rate limits and things like that. But it's a fully online process, and no user knows it's ever happening. Postgres makes that not so easy. We're working on getting back to the point where we can constantly deploy, whether it's Vitess or new container images, and it just rolls through. That got very addictive for how we build.

## Why PlanetScale started with MySQL, then embraced Postgres

<!-- t=841 -->
**Ben:** There's a question from the audience related to this: why did PlanetScale start with MySQL versus Postgres? I think you've already answered some of it, but Sam, do you want to add context?

<!-- t=856 -->
**Simon:** You should do an entire stream that's just Postgres versus MySQL and talk about the trade-offs.

<!-- t=867 -->
**Sam:** It's all trade-offs. Neither is good or wrong, just different.

**Simon:** Quorum writes, though. Wrong.

<!-- t=881 -->
**Sam:** Wrong, in every sense. We'll agree there. Basically, the company was founded on the idea that we could make Vitess broadly adoptable for large-scale companies. Vitess at YouTube was not really what it is now, in fact pretty different, but it was taken to very, very large scale at YouTube. Then it picked up a really large following at companies like GitHub, Square, Etsy and Slack. The idea was to build a commercial company, and Vitess uses MySQL. Vitess is normally for very large-scale workloads. You can obviously start your startup on it, and Simon uses it with Turbopuffer, but what it was initially very good at was taking databases to scale. And we just hadn't seen that scale on Postgres yet.

<!-- t=945 -->
We switched to Postgres because people loved what we were doing in the MySQL world, truly loved it, and wanted it for Postgres. People finally convinced us we should do it. That was it. Now we're really enjoying being part of that community. The energy in the Postgres community is unbelievable. I wasn't convinced we could do anything good for Postgres, and I didn't want to be another bottoms-up database for hobbyists. I wanted to do something unique, and once we convinced ourselves we could, I was in.

<!-- t=991 -->
**Simon:** You kind of had three choices. You could have stayed MySQL and Vitess only. Or you could have tried to convert Postgres users into MySQL and Vitess users. Or you could do what you did, which is embrace Postgres and be a vendor that talks very transparently about the trade-offs. I would have made the same decision as you. Why did you make that decision over the other two?

<!-- t=1015 -->
**Sam:** Philosophically, we believe that if you're going to act like a database server, you should be that database server. There are so many emergent behaviors once you get to the storage engine or the query planner that faking it doesn't work. We talked about doing fake Postgres, teaching Vitess to be like Postgres, and it just doesn't align with what we would do. Vitess and all of our operators are there to get MySQL and Postgres onto their happy path, the way they want to be managed as a cluster. It's just my philosophy to go that way. It's also too hard to fake. There are a number of databases out there that pretend to be Postgres-compatible and fake the protocol, and it just doesn't work. It's never compatible enough, it has all these edge cases, and it's kind of bad.

<!-- t=1113 -->
Ironically, the Vitess business is now accelerating even harder because of Postgres. People see that we do Postgres, they come in interested and excited, and they love it. We've seen people running very large, vertically scaled Postgres, and we tell them that all of the problems they're having don't happen in the Vitess world, and then they actually migrate. It's been really funny. It took doing Postgres to sell more MySQL.

## Hairlines

<!-- t=1154 -->
**Ben:** I'm going to interject real quick, because we have a question from "Average Database CEO," which maybe proves it's not Sam. How do you guys have such good hairlines?

<!-- t=1165 -->
**Sam:** I don't think you do. I obviously don't. I will when I get the hair transplant or whatever, but right now it's receding.

<!-- t=1174 -->
**Simon:** It's all covered up. It's all HA magic. You shouldn't trust a database CEO with a good hairline. They have not been woken up enough in the middle of the night by pages and by their children.

**Sam:** Correct. Exactly.

<!-- t=1189 -->
**Ben:** He said he was in the middle of negotiating a huge deal, so he couldn't make it on. We do not know if our average CEO even has a hairline.

## Turbopuffer's biggest indexes

<!-- t=1203 -->
**Ben:** I have a more serious question for Simon. What are some of the biggest indexes at Turbopuffer? Is anyone pushing the 300B you announced recently?

<!-- t=1216 -->
**Simon:** It's 100B that we announced. There was a customer that wanted to be able to do 100B, so we worked with them to test that it would work, and then got it to work. They're not at 100B in production yet; they're about halfway there, but we expect they'll hit that number over the coming months. It was really about doing the POC to make sure we could handle that scale over extended periods of time. Even indexing it is a phenomenal feat, and we needed to make sure we could do it at a cost where they can rewrite everything, which is really important at this kind of scale. So we don't have a 100B index in production, but we certainly could. I think it's still sitting in object storage if they wanted to continue to query it. We are seeing a lot of adoption of these large indexes. I didn't think there was a ton of data sets in the world of that size, but there are more than I thought, and adoption might also be a little ahead of the curve.

<!-- t=1289 -->
**Sam:** And of course, if 100B has a 200 millisecond P99 and a 40 millisecond P50, then 1B is just going to rip.

<!-- t=1299 -->
**Simon:** We've spent a lot of time on tail latency. There's a reason we always talk about the P99, because this is on lots and lots of machines. We spent so much time on the tail and so much time on the indexing. I believe the indexing for this entire thing took maybe 24 hours, maybe two days. We worked very closely with that customer, and we have plans to bring that down further. It pushed the system really well.

<!-- t=1325 -->
It reminds me of Shopify. It was never profitable to spend so much time on serving all the Kylie Cosmetics flash sales. But Toby was adamant: I don't care if we lose tens of millions of dollars, I don't care that we have 20 amazing engineers on this, it will make the platform better, and that's why we're doing it. And it did. It sets a new watermark, and anything below that you don't even think about anymore.

<!-- t=1355 -->
**Sam:** Like a broken window.

<!-- t=1361 -->
**Simon:** Exactly. And on the product side, it helps too. We played a lot of tricks to make that work really well. Right now it takes a little hand-holding. It 100% works, but you need to be in a Slack channel with us. Now we're productizing all of that, so you click a couple of buttons in the console and you're off to the races. In infrastructure it's super important to build a walking skeleton first. You hold the customer's hand and respond to them in Slack, working with them as if you're an extension of their team, and then you productize afterwards. You'd be naive to think you can just sit down, big-brain it in Notion, and then implement it. You need to see it.

## Why Simon built Turbopuffer: a new storage architecture and a new workload

<!-- t=1397 -->
**Sam:** That was exactly our strategy for early customers at very large scale. But I wanted to ask you why you took this approach. I remember we caught up and you said, "I'm going to start a company doing a vector database," and I thought, oh really, another one? This is when one was being announced every week, and you don't hear about any of them now. You pretty much just hear about you. I asked what you were going to do differently, because I was looking into this myself. We all have to have this loop in our brain: you're truly old when you start closing yourself off to new technologies. We're seeing this with AI right now, people rejecting it outright. Everyone who got disrupted underestimated the new thing. You have to try new things and think about them. So I thought, vectors don't seem fundamentally different. There's pgvector, which would probably work for most people. And your insight was that you were going to have to vectorize everything, copy pretty much the entire corpus of a company's data, and make that very efficient. Why did you start there? The approach clearly worked and was the right bet. It was a bet on people's behaviors, I assume, and then you built an architecture that supports those behaviors. I want to hear that from you, because it was very interesting to watch you build this company from the beginning. I think you got a small number of things very right in the beginning, and it's worth digging into them.

<!-- t=1498 -->
**Simon:** I can speak about it now 10x more cogently than I could then. Which is also why...

<!-- t=1505 -->
**Sam:** Do you remember when you asked if I wanted to invest and I said no?

<!-- t=1510 -->
**Simon:** You said you weren't very liquid at that time.

<!-- t=1513 -->
**Sam:** Which was quite a regret.

<!-- t=1517 -->
**Simon:** It was a great way to turn it down, by the way. Great phrasing.

<!-- t=1524 -->
Anyway, I maintain a repository called napkin math, where I keep all the numbers: how much bandwidth you can drive to DRAM, what you can drive to object storage, the latencies, and all of that. I'd helped one of my friends' companies build a little recommendation engine, and we didn't put it in prod because it was going to cost like 30 grand a month. That was way too much for a little company that was spending five grand a month on all its other infrastructure combined. I couldn't stop thinking about it. There's got to be a better way. When I did the napkin math, I saw there's another tier of storage here: not just disk into DRAM, but object storage into disk into DRAM. It just sort of emerged. Then I sat down for an entire summer and wrote the first version of Turbopuffer. That's also when I probably had you on the phone.

<!-- t=1575 -->
I was very deliberate about that summer. I didn't tell anyone and I didn't tweet about it. I could not for a second let myself believe I was making any progress until this thing was launched. I know myself well enough that if you talk about something, you confuse it with progress, and I knew this was going to be a monumental lift. I went from working on it a few hours a day to being completely consumed by it. I'd have multiple index runs going on my little machine. I was writing so much code. It was probably the most productive period of my life.

<!-- t=1619 -->
**Sam:** That's an incredible feeling, by the way. You have nothing to lose. You can walk away from it whenever, so it just feels incredible.

<!-- t=1633 -->
**Simon:** I was so distracted. We had guests and I couldn't stop thinking about it. I'd wake up at five in the morning and write things out on paper and start writing code. The insight was that I could do this a lot cheaper and someone's got to do it, but I never thought it would be a company. I thought it would be a project.

<!-- t=1653 -->
The way I think about it now is this: if you want to build a really big independent database company, you need a new storage architecture. That's what pgvector doesn't have, and none of the incumbents had it at the time. Anyone who's listened to an interview with me has heard this; it's my hobby horse. You need a new storage architecture, because otherwise you're not doing anything novel, and all the other database companies, including yours, will just add this as another index, and there's no reason to use another database. An analogy: 10 years ago, when we started building mobile apps, everything needed geo. There's no stock ticker today for a database that builds geo indexes. It's just two numbers, and any index under the sun can do it. It's a new data type, not a new database.

<!-- t=1716 -->
Turbopuffer has the new storage architecture. NVMe requires a different kind of database. MySQL is not written for NVMe. It requires a database that is completely sensitive to having very large amounts of outstanding IOPS in very few round trips. That works incredibly well for NVMe. You can get bandwidth within spitting distance of DRAM if you design everything in the database around that. And it's not only good for NVMe, it's also phenomenal for object storage. What tiered storage means is that you'd better hope you don't need the cold data, and when you do, your query is going to time out, or if you're lucky, take a minute. What we call object-storage-native means that if you turned off DRAM and SSD caching, it's going to be a bit slow, but users are not going to be mad: maybe 500 milliseconds to a second of cold latency, because we only have to make three round trips to object storage and everything is laid out that way. That's the new storage architecture. We make a lot of compromises. You're not going to run Shopify on this thing, because our write latency is 200 milliseconds. You can't build an OLTP database like that for general-purpose workloads. It's good for search, good for analytics, good for lots of things, but not as a general-purpose database.

<!-- t=1808 -->
The second thing you need, especially if you're impatient about building the database, is a new workload. If you truly want to build a really big company, you need every company in the world to use your database directly or indirectly. Think about Databricks, Snowflake, Oracle: every company on earth either directly or indirectly uses one of those. To do that, and to have it go fast, you probably want a new workload. The new workload right now is connecting enormous amounts of data to AI. If you have those two things, you're in a great position. The first generation of vector databases was limited to a data type and ran on the same storage architecture as everyone else. That's really tough, because then pgvector is going to eat your lunch. And pgvector is phenomenal. If someone comes to me and says, "I have 2 million vectors, should I switch to you?" I say, you're not in enough pain, don't switch to us. If there's not enough pain, I'll end the sales call 10 minutes in and say, here are the tripwires for what pain looks like, have fun, go find PMF, ride the bull, and when you're having outages, come talk to me and we'll help you migrate in a few hours.

## How AI workloads differ from earlier workloads

<!-- t=1888 -->
**Ben:** That gets to a question from the audience, and one I had too: what differences do you see over the past couple of years, with AI-heavy workloads powering AI apps, versus what came before? Simon, you kind of answered that, since that's what you do. Sam, maybe you could comment on the Postgres and MySQL side.

<!-- t=1915 -->
**Sam:** Go, Simon. You partly exist because of the AI world.

<!-- t=1919 -->
**Simon:** There are a couple of ways to address this. What we're seeing from those companies is that they're growing unbelievably quickly. Sam and I grew up at GitHub and Shopify, which in the 2010s were some of the fastest-growing businesses in the world. The metrics there were 100% year over year or something like that, which seems quaint now. Some of these companies are compounding for very long periods at between 30 and 60% month over month. These are mind-boggling numbers. You don't believe it until you see it, and even if you've seen a lot of exponentials unfold in your life, you won't see it. That's the first thing: they're scaling very quickly. In the very beginning, when the Cursor team was very small, I tried as much as possible to help them. They would call and say, we have this problem with Postgres, and we talked about autovacuum for a while on the phone, because they were growing faster than we've ever seen.

<!-- t=1983 -->
The second thing is that because they're growing so quickly, the data volumes are also very large, very quickly. They have to shard six months to a year into their business. It's bonkers. We didn't shard Shopify until 2013, almost a decade into the business.

<!-- t=2010 -->
**Sam:** And GitHub, I'm guessing, didn't shard until 2015 or 2016. That's when the effort started. I actually wrote an issue maybe in 2013 or 2014 saying we should avoid sharding as long as possible.

**Simon:** You did.

<!-- t=2021 -->
**Sam:** "As long as possible" is the key element there. There are lots of tricks for getting out of it. And it's one of those situations where a difference in amount is a difference in kind. Sharding is so much easier to do now. Do you remember the Etsy post that used to go around a lot? They use Vitess now at very large scale, but back then it was in-app sharding, and they had to do it for their consumer scale. Same thing you saw at Shopify. But it's really painful and fraught with trade-offs, and you don't volunteer to do it for fun. Rick B is here, who was something like the eighth employee at Instagram, so he saw that scaling journey. Same thing: consumer scale goes off like a rocket, and you have to shard. It's wild.

<!-- t=2100 -->
It's crazy how AI has changed the dynamics, even from a ratio standpoint. Four years ago, if you asked me about OLTP, I'd have said nearly every workload I've ever seen was read-heavy, with some very large exceptions. The first large-scale MySQL I ever worked on was an incredibly write-heavy telemetry system. But nearly every other workload I'd seen was very read-heavy, maybe 100 to 1. That makes sense: when you open Twitter you're served, above the fold, maybe 10 or 15 tweets as reads, and how many tweets are actually pushed? Very few. Same with GitHub: way fewer commits go in, and many more reads happen for viewing pull requests and issues. AI companies have been a huge inversion. They write extreme amounts of data. It's also interesting how much data has to be duplicated and shunted around into different formats. They need OLTP to serve product experiences, but that data also has to have a vector representation. We have so many shared customers that use both, because they come to us for extreme-scale OLTP, with the trade-offs that has, and come to you for extreme-scale vector storage, and often the same data is represented in both. Let's talk about how that coexists.

<!-- t=2215 -->
**Simon:** You should probably just use PlanetScale for your vectors if you're trying to find PMF, and it's probably going to work great for you. At some point you're going to need to fix the cost of your search. There's the value per byte of OLTP data: this is my customer, this is everything I know about them. That's very high, and it should be in PlanetScale with very low-latency access at all times. But if you're loading in 100 million PDFs and you don't really know what they are, receipts say, the value per byte is a lot lower. You'll need something like Turbopuffer, and you might not cross that scale until a little further into your company.

<!-- t=2262 -->
Another change we're seeing is that people used to shard per user or in some other way, or shard vertically. What you do before that is look at the busiest tables, the ones doing the most lock contention on your database, and rip them out into a specialized system. For these AI-native companies, they get there very quickly, and almost always the first thing you take out of the database is search. We could go a little low-level here, but updating a vector index or a posting list for full-text search in an OLTP database, with the guarantees provided there and the lock contention, is abysmal. I have ripped this out of many Postgres and MySQL clusters, because it's completely at odds with the trade-offs made in these systems. The write amplification, and the lock contention this creates, is really problematic for an OLTP database. But until you see this, it's "use one database, shove it all in there, hold the whole thing together with duct tape and spit until you find PMF." Once you're on the ball, you start to move in these directions.

<!-- t=2340 -->
A fourth one: while code is getting cheap to run, I think infra expertise is still hard to come by, and I don't think the models are particularly good at making the trade-offs. They'll spit out as much complexity as you want, but a model is never going to tell you, "I want to build a database where the entire WAL is on object storage with no other system altogether." They're never going to build a system as simple as Turbopuffer, and those were exactly the right trade-offs for us.

## Lived experience, private scale conversations, and what LLMs don't know

<!-- t=2375 -->
**Sam:** I don't think the models are very good at architecture. They over-engineer tremendously, and you need good instincts for that. This goes back to why we bonded when we first met: there's a cohort of companies that run at a certain scale. It's more now because of AI, but back in the day, and we sound so old right now, tech was much smaller and much quieter. It was the top 100 websites of the internet, with an extreme power law to the scale they're at. When you get into the top 10, I always try to blow people's minds with the numbers. I think it's public that Facebook crossed a billion queries per second on their MySQL cluster in about 2010. The scale we're talking about is gigantic.

<!-- t=2447 -->
This is one of the reasons we're sat here talking right now. A lot of the true tech conferences about scale are done in private, and I think people don't realize that. People get together and talk very privately about how some of these things are done, because the nuance goes away otherwise. And none of that information has made it into an LLM. I was tweeting about this the other day: models have been trained on what an average engineer thinks about the world, an average deployment, and how an average deployment is architected, and that's very different.

<!-- t=2507 -->
I've come, from lived experience, to full philosophical disagreements with certain architectures, and I know they don't work for certain things. It doesn't matter at small scale; it just doesn't work at large scale. At the end of the day, your competitors are just going to type their biases into an LLM and build what they want to build, just faster. So it's an interesting dynamic: you can be right or wrong much quicker. I think there's much more alpha in architectural knowledge and experience when using LLMs. It's a catalyst that accelerates whether you know what you're doing or not.

<!-- t=2548 -->
**Simon:** I like that. That's how we got to know each other. We just chat: "Hey, what are you guys doing about this?" And you weigh that information a million times more than anything you can find on the web, because it's true lived experience from people who are also getting paged in the middle of the night. I'm sure there's some dinky talk of you somewhere in the Midwest from 2014 where you talk about scaling MySQL, and maybe the LLMs have it deep in the corpus, but it's not going to weigh it the way I would. And what actually works is so simple. Simplicity takes a lot of practice.

<!-- t=2597 -->
I think another thing we have in common is that we were with one employer for a very, very long time. That's quite rare, and even more rare today. It teaches you a lot about simplicity, because you see some dumb thing you did off to the side while busy with something else stand the test of time for almost a decade and still run in prod. Meanwhile, the project where a couple of people had a little too much time and spent four months shipping broke two years later. Having as many of those threads in parallel to learn from is a gift. Seeing how software ages is rare.

<!-- t=2640 -->
**Sam:** Living with your own decisions is rare, and so is going through that, watching, and staying pragmatic. We used to joke at GitHub: the Git stack got a full rewrite, and when it was done, which took two years, I said I can't wait to rewrite this next year. We never did, and it scaled very well. In weird ways it scaled better than the databases, which is partly why I did PlanetScale. You have to live with those decisions, see how an architecture changes, and knowingly make trade-offs and know how they'll change over time. It's a dark art.

## Simplicity and the essence of the bowl

<!-- t=2688 -->
**Simon:** Another thing we bonded over is the images you have behind you. Do you remember us having this conversation? You're one of the very few people who pointed out the backdrop.

<!-- t=2703 -->
**Sam:** We talked about this in person once, years ago, over a drink. We were talking about art and how the essence of the bowl is so poignant.

<!-- t=2713 -->
**Simon:** You should explain it real quick, Sam.

<!-- t=2717 -->
**Sam:** Behind you is a set of sketches from Pablo Picasso. Anyone viewing this would assume it was done right to left: looking left to right, we see a better-illustrated bowl, with more detail, and then it becomes just lines. Well, his study was to find the essence of the bowl. He was able to represent the bowl with as few lines as possible, meaning he truly knew the essence of the bowl. I think about that in terms of problem solving. I may have butchered that explanation, because my brain heavily compresses information.

<!-- t=2780 -->
Essentially, I think about this a lot. The best people I've ever worked with... and I think this is the problem with the industry generally, and why we decided to talk on camera for the first time. I've worked with people who are humblingly smart, the P99 engineers, in your phrase. You see the lights burning very brightly in their heads, and they truly understand the problem, based on how succinctly and clearly they can describe and present it without complexity. That image represents this. He understood the bowl. He had studied every facet of the bowl and could represent it so simply.

<!-- t=2834 -->
I actually gave a talk years ago in London about architecture and simplicity, when microservices was such a thing and everyone was doing complexity for complexity's sake. You and I were both working on two of the top 40 websites on earth, on Ruby on Rails and MySQL, shipping quickly. In the talk I drew the Golden Gate Bridge in like seven lines or whatever, and said this is why it speaks to us: it's simple, but an incredibly complicated thing is done. It's beautiful. Unfortunately, the industry has lost that. Talking about nuance and trade-offs is a waste of time; if you talk about the trade-offs of your product now, it gets used ungenerously by people who don't really know. I've stopped having the kind of arguments with people where I just say, at the scale you're doing these things, anything will work. Store it in a flat file on disk, put it in a fucking Word document or something, it just doesn't matter. I'm hoping that with your success, our success, scale changing, and scale coming for younger companies, we'll ignite a group of people who want to talk about pragmatism and simplicity, and then use that to accelerate things via AI. It's not an anti-AI sentiment at all. The more clearly and succinctly you can describe and understand things, the better you'll build, and you'll keep cognitive overhead low and stay very productive.

<!-- t=2932 -->
**Simon:** The way I think about it sometimes is that you have to deserve complexity. The system has to start as simple as it possibly can, and it will surprise you. Turbopuffer's internal queuing system was just a bunch of JSON files called queue.json that we kept overwriting. We only shipped the replacement a few months ago, and the old one worked quite well. Let simplicity surprise you. It's pretty mind-blowing what it will serve you, especially at scale.

<!-- t=2970 -->
I also can't picture people's faces in my brain, or high-fidelity images. I would never be able to picture the full-fidelity bowl or anything resembling it. I can only picture very simple lines. Apparently that's called aphantasia or something like that. I don't know if that's part of it, or maybe I'm just too dumb to make this complicated, so I can only go for simple. If that works, it works.

<!-- t=3010 -->
**Sam:** Should we take some more questions? We thought we wouldn't have much to talk about, because we're both generally awkward, but it turns out people are kind of interested.

## Sandbox agents and short-lived databases

<!-- t=3021 -->
**Ben:** Speaking of headlines, we've got a good one on camera. Ben's come back, and people are really showing up for the hairlines. This is a long question about sandbox agents: what are your observations and predictions for long-tail, short-lived, small databases with sandbox agents and agentic workloads? Do you see more ideas like Cloudflare Durable Objects shipped with SQLite, or running things at scale? With agents powering applications with databases, what are your thoughts on the future of that?

<!-- t=3063 -->
**Simon:** We do see people on Turbopuffer doing very short-lived scratch pads or workspaces. You can spin up a code base in Cursor and they'll index the entire thing with Turbopuffer, and agents will copy those, make changes, and so on. We're making changes to the database to make that even more fluid, cheaper and faster for companies. The whole thing is architected so that this is not a big deal. Turbopuffer today has hundreds and hundreds of millions of namespaces, and will probably reach a billion soon. You might think of a namespace as a table, but it's basically just a directory, and it's an independent database. We can handle that many of them, and they're all completely independent. We do a bunch of maintenance and grooming of them in the background, but the system has been designed since day one to support a very large number of them, and almost all of our customers take advantage of that. They can be short-lived, deleted after a very short period of time, encrypted with different keys, created in different regions, and copied between regions. We'll continue to do what we see our customers doing, and we have a lot coming on that in the coming months.

## What a modern database would look like

<!-- t=3149 -->
**Ben:** Another interesting one, which you've kind of covered: if you were tasked with creating a database for the modern world, ignoring all previous database work and architectures, what would you focus on? What are the most important things given today's hardware and workloads?

<!-- t=3163 -->
**Simon:** As I mentioned earlier, if you want to build a big database business, you need a new storage architecture that's fundamentally different. If you just want to build a new database, it doesn't have to be a new database company, as in Sam's case. But then you need a new set of storage-architecture trade-offs, or other trade-offs so big that they warrant a new database. There have been almost no big examples of that, at least as independent companies, though certainly as projects. I don't think there is another storage architecture right now. The database that is all in on NVMe and all in on object storage is the one that's in the air. It will come with very similar compromises to one-zone and some of the newer types of buckets, with new trade-offs, but it's really the same database. I can't predict what the next storage architecture might look like. These only come around every 15 years. In the eighties and nineties we built with replicated disks, and that's still what most production workloads run on today. It's a great set of trade-offs for OLTP. Then in 2010 we had commodity HDDs, we got good at building warehouse-scale data centers, we had S3, and we had the rise of big-scale analytics. That was big structured data. Now we have big unstructured data and a new set of storage-architecture trade-offs that make it extremely suitable for that workload. It happens every 15 years. Maybe it will be 10 years, maybe 15, maybe they figure out how to make disks with even higher latency but even more throughput. I'm not sure, but it has to present a new set of trade-offs. I find it very difficult to answer that question.

<!-- t=3283 -->
**Sam:** Everything you just said is true. There's not much new under the sun. There's a reshuffling of things, and there's physics. We're bound by physics and by so many other things that there are no giant shifts you can make. You tend to see one or two architectural shifts every cycle. We went through the MongoDB NoSQL wave, and very few companies rose from it. It got very, very big. We're seeing vector DBs as a new kind of...

<!-- t=3316 -->
**Simon:** Sam, we're not a vector DB. It's a bad-vibes term. We are a search engine. Stop calling me that.

<!-- t=3322 -->
**Sam:** A new type of search engine, then, versus the old attempts like Elasticsearch, which by the way caused GitHub's largest partial outage. I can tell that story later if you like. It was actually the first technology we were ever asked to support ourselves. The VP of engineering from a very large consumer product that runs on PlanetScale sat me down and said, you solved our hardest database problem, can you do Elasticsearch next? I said no, absolutely not. I don't think you know what you're asking for.

<!-- t=3356 -->
There's a new search paradigm for this generation, and it sounds like you've landed on the architecture and caught the vibe to build the big one. The rest of the world has stayed in a non-specialized, very generalized setup, which is SQL relational OLTP databases, MySQL and Postgres. They've split that world, and the architecture is pretty good. There are lots of trade-offs, but fundamentally you'd just make things more efficient, and then it comes down to how much of a gravity well databases, storage formats and replication are. I have real mental blockers about speaking about these things hypothetically, because I try to stick pragmatically to the constraints of the real world and making those things scale. We have what we have. We've seen people try to retrofit OLTP onto the architecture Turbopuffer has. It has some good properties, truly, but then there are massive trade-offs that make OLTP very difficult.

<!-- t=3431 -->
That architecture is going to be great for a bunch of systems, but I think people believe we're going to build all these big database companies on this new object-storage architecture. No. There's logging, which was WarpStream and friends. There's search, which has a new workload. The other category is a better way to do an existing workload. Then the rest are going to be ad hoc, very trade-off-heavy databases. Braintrust is building their own, Brainstore, which is a great example: I need to create a great database with a bunch of trade-offs that I can make internally, very fast. Great architecture, amazing. That's the third category. The fourth category is logging and analytics, which is as much a UX company, and they've been building databases like this for a while. That third category is massive, but there are glaring trade-offs that don't work generally, so it's not a big independent database company. That company might be massive, with this on the backbone of it, but it's not an independent database company.

<!-- t=3489 -->
**Simon:** I agree. Fully agree.

## What makes a P99 engineer

<!-- t=3491 -->
**Ben:** I think we'll do one more question here and then wrap up. You both mentioned the term "P99 engineer," and you've both done a lot of hiring while running companies. How do you define a P99 engineer, and how does someone become one? Do you use the term internally now, Sam?

<!-- t=3516 -->
**Sam:** Yeah, we do. It's a perfect framing. Ever since you first said it to me, and I'll give you full credit, it's stuck in my head forever. It's such an aligning frame.

<!-- t=3528 -->
**Simon:** I'll give some serious answers and some non-serious answers. The P99 spends a lot of time looking at maps. They love looking at maps. It's their preferred UX. That's one.

<!-- t=3547 -->
The P99 is very good at navigating the simplicity-complexity continuum, a lot of what Sam and I have talked about. They think, this is the simplest thing I could do, and then they can ratchet up the complexity as the requirements get larger, and they navigate that with brilliance.

<!-- t=3571 -->
The P99 is exceptionally good at what happens when they hit the wall of bullshit, i.e. where their knowledge stops. There are three ways to navigate it. The first is that you arrive at the wall of bullshit, and because you've reached the edge of your knowledge and you're insecure, you just start spewing bullshit about some topic you know nothing about. The second is to say, I have reached the wall of bullshit, I would like to not proceed past it, this is where my knowledge stops. That is not how the P99 responds. The P99 says, I have arrived at the wall of bullshit, here's how I think it might work, and starts reasoning from first principles from what they know. They start asking questions and turning it back.

<!-- t=3621 -->
The P99 is fast. The P99 is obsessive. The P99 has a history of associating with and finding other P99s. The triple-P99s are often people who have barely seen other P99s and still created an environment for themselves, because as soon as you get them into an environment with P99s, it takes off. I don't know exactly where in the UK Sam grew up, but both you and I grew up in small towns playing way, way too much World of Warcraft and then getting interested in computers, and at some point found our community. That's common among P99s. The P99 initiates and owns. The P99 has a history throughout their life of bending their own trajectory to their will. Many P99s were obsessed with something in high school. I could keep going on and on, but these are some of the traits.

<!-- t=3694 -->
**Sam:** A lot of the P99 engineers I know like trains as well, not just maps. But in seriousness, I don't know if you can fully learn it, quite honestly. I think a lot of it comes down to taste, and innate taste. People can refine their taste and they can learn, but I think it's a hard goal to just go for. A lot of it comes down to personality, a set of quirks, a set of conditions. And you're right, I grew up in a tiny village in England playing World of Warcraft. It's literally how I got into it. You end up in a guild with people who do this stuff and you get to know them, and that's basically how I broke into tech.

<!-- t=3737 -->
First of all, to be a P99 systems engineer, you have to see the world in systems. You have to understand all of the power laws that govern the systems around us and the world around us. You have to be incredibly tenacious and not give up. Ted Nyman is in the stream, and Ted and I used to work at GitHub. He loves maps. At GitHub there were the people who were there late at night, pushing through problems, trying to drive toward simplicity and turn down the chaos. There are people who can find extreme clarity. There also has to be a level of competency and skill. They just have to be very good with computers. We have friends who see computers in a completely different way, and they understand those things. That's how you back up the skill. There are people who are generationally talented at coding and generally not very good engineers. There are cultures out there that hire very, very good engineers, but their engineering culture is not at all pragmatic, and I don't think that makes them extremely effective or good engineers as individuals. It's about how effective you can be. It's a human task as well as an engineering task, and it's about turning down chaos.

<!-- t=3817 -->
I think you also have to be exceptionally humble. You have to want to work with people who are intimidatingly brilliant and deal with what that means. You're going to have to work with exceptional people while you're young and deal with the crushing issues of youth. If you are a young person on this call and you want to become exceptional, you have to own the fact that in almost all areas of your life, you're an idiot, and you're going to pay that stupidity down with energy. You've got to work exceptionally hard to get exceptionally good. What you offer to companies is the fact that you can work all night and for days and months on end, extremely hard, with people who are exceptionally good, who will give you feedback and will grow because they're using their wisdom to continue to build and be effective. When you're young, if you want to be the smartest person, if your ego rules... low ego is actually another one. The best people I've worked with truly are low ego. If you want to be amazing, you have to have a low ego and accept that you want to be at the bottom of the totem pole at a great company with great people, work exceptionally hard, find every opportunity you can, and get exposure.

<!-- t=3897 -->
The sad thing, and people ask me about this a lot: when we open a job role for engineers, we get thousands of applicants. A guy we hired this week was at the office last night, and we were chatting. The first time he messaged me on Twitter I missed it, and the second time I caught it, and he got hired immediately because he was very, very good. But you get this overwhelming number of people, and you want to give feedback and help them, and the sad thing is that most of the ability to get that good involves joining somewhere that has hard problems. It's impossible to do in a synthetic environment. You have to get out of the flight simulator one day. So it's about being really resourceful to get into companies that have scale problems and systems issues, and fixing those and grinding through them until you get a good intuition for what systems feel like.

<!-- t=3958 -->
One more thing: building intuition in this age of AI, and just generally being good, is so important and so powerful. I don't know about you, Simon, but Ted and I have talked about this: when GitHub was slow, from the way it was loading slowly, you would know which service behind the scenes was likely struggling. That's purely intuition. When Shopify was having an issue, you could smell it. A friend of mine, Thomas, who's over at Meta, ran the databases there, and when I was on the networking team at Meta they'd say he's the best network monitoring we have, because databases are sensitive to networking problems. You get this intuition where you can feel how computer problems feel. That takes feeding your brain and training your internal model on how computers, networks and distributed systems work. It's really hard to do without data, and data comes from that experience.

<!-- t=4016 -->
**Simon:** I'll add two more things. The humility aspect is similar to the bullshit-wall test, and it's easier to interview for. If you're not aware when you reach the edge of your knowledge, you're going to start training your own neural net in a really bad way. You don't know when you don't know something, so you start hallucinating, go down a sampling path you don't want to go down, and have some really garbage epochs.

<!-- t=4043 -->
Another trait of the P99 is that they've found a lot of mentors throughout their life. By mentor, I don't mean you walk up to someone and say, "Do you want to be my mentor?" That is not how good mentorship occurs. Good mentorship is something you earn, because the other person gets interested in you, your perspective, your drive and your novel ideas on something, and you learn from them. You show incredible sampling efficiency on every token they feed you. I think that's an incredibly important part of the P99. Early on, I just found this email, because we were talking about it. The GitHub people would probably know better than me, but Ryan Tomayko did a lot of tweets, and I have an email from 2010 where I asked him a bunch of questions about how to get better at systems programming. I just read it while you were talking, and I did every single one of those things. I read that book. I spent an entire fall vacation reading through the Unicorn HTTP server source code, trying to mine it for everything, because he said it was a great example. That's something you earn, and I think the P99 has the humility to seek out mentors.

<!-- t=4125 -->
**Sam:** 100%. Also a legendary systems engineer, and his writing was phenomenal.

<!-- t=4129 -->
**Ben:** On top of all that, now I know I need to learn maps and trains. Biggest takeaway.

<!-- t=4140 -->
**Sam:** The P99 also spends a lot of time reading your blog posts, Ben, I would hope.

<!-- t=4144 -->
**Ben:** I hope I can contribute something.

<!-- t=4145 -->
**Sam:** You do create content for the P99. We're doing this for that reason, too: it scratches the brain for people. I actually do think your content has been very beneficial for people who really want to learn and understand computers.

<!-- t=4167 -->
**Simon:** The wonderful thing about your content, Ben, is that it is content for the P99 but also comprehensible to someone who's a P50 in that area. That's an unbelievably difficult line to walk, where the P99 learns something and the P50 feels in awe but also learns even more, and reads it for the tenth time. There aren't very many people in the world who can create that on systems. It's the best articulation we've been able to frame internally.

<!-- t=4200 -->
**Ben:** I consciously make that a goal, so I'm glad you said that. I used to teach at university, which was very different, but it's hard because you have some kids who have been coding since they were five and some who are just learning it. How do you teach to all of those people at the same time? So I keep that in mind with everything I make.

<!-- t=4218 -->
**Sam:** Professor Ben, we call him.

**Ben:** Thank you.

## Wrap-up

<!-- t=4219 -->
**Ben:** All right, I think we'll wrap up the stream here. It's been really fun. Thanks for coming, Sam. Thanks for coming, Simon. I hope everybody enjoyed it. We got through a lot of the questions, and I'm sure we'll do this again, because I think it's a big success.

<!-- t=4237 -->
**Sam:** Just tweet at us if you want us to do this again. I have absolutely no idea why anyone would want to listen to us talk, but if you liked it...

<!-- t=4245 -->
**Simon:** I had fun. This is the easiest way to get on your calendar, Sam, because we never see each other anymore.

<!-- t=4252 -->
**Sam:** Well, you did come to town and didn't...

<!-- t=4255 -->
**Simon:** I'm coming next week.

<!-- t=4257 -->
**Sam:** Okay, I'm going to invite you to some stuff. No wait, I'm seeing you on Monday. You're coming to this dinner on Monday.

<!-- t=4263 -->
**Simon:** Oh yeah, that's right. Let's do this IRL next week.

<!-- t=4268 -->
**Ben:** You two in the same room.

<!-- t=4269 -->
**Sam:** We could do it at the office. Are you here next week, Ben?

<!-- t=4273 -->
**Ben:** Maybe, we'll see. It's a bit of a shit show with this bowling event that's going on in San Francisco that I heard about.

<!-- t=4291 -->
**Sam:** Yeah, it's chaos. The streets around our office are closed at the moment, so it's just absolute chaos.

<!-- t=4299 -->
**Ben:** All right, thanks everyone. We'll wrap it up.

<!-- t=4303 -->
**Sam:** You're not going to get P99 by watching old people talk on camera. Goodbye. See you next week.
