# CMU Database Seminar: turbopuffer, Object Storage-native Database for Search

2026-03-09 · Recording: https://www.youtube.com/watch?v=pqoRNwNaxfs · archived copy: https://archive.sirupsen.com/recordings/cmu-talk.mp4

_Lightly edited automatic transcript: speech-to-text, cleaned up by an LLM. It may contain errors; the recording is the source._

## Introduction

<!-- t=25 -->
**Andy:** All right, guys, let's get started. We're excited today to have the CEO and co-founder of turbopuffer, lowercase letters. I was very careful about this. Simon started turbopuffer three years ago. Prior to that, he was eight years at Shopify, where he was Principal Software Engineer and Director of Engineering. So a lot of the things he'll talk about today are things he learned at Shopify. As always, if you have any questions for Simon as he's giving his talk, feel free to interrupt him at any time. That way he's not talking to himself for an hour on Zoom. Simon, the floor is yours.

<!-- t=59 -->
**Simon:** Thank you, Andy, and thank you for having me. I want to reiterate that if anyone has questions during the talk, I very much encourage you to interrupt, so I don't just talk for an hour with no interruptions. I know Andy won't be shy to take questions from the chat either.

<!-- t=78 -->
My name is Simon, and I'll be talking about turbopuffer today, a database that we built on object storage and that is currently focused primarily on search.

## What turbopuffer does

<!-- t=92 -->
turbopuffer is a database that's mainly optimized for search right now, though it can do a bunch of other things as well. The easiest way to show what it can do is a simple full-text search query. If you're searching for something, you might need to rank by various different signals. You have some hard filters, for example you only want timestamps after a certain period, and you want to limit the results. You can do this with vectors. You can do arbitrarily different kinds of full-text search queries, like boosting results that have a more recent timestamp. All of these are very useful for searching.

<!-- t=128 -->
But you can also do more than search in turbopuffer. You can do aggregations, group by different attributes, and order by any type of column, even multiple columns. I think that in the limit, a database ends up implementing every query plan on the data structures that are inside of it. But you have to start with a very particular workload, and that workload for turbopuffer right now is search.

<!-- t=157 -->
The other thing about turbopuffer is ingesting your data into the database. You insert your data through turbopuffer. You don't just put it on object storage and have turbopuffer fetch it. You have to do it through turbopuffer itself. We provide a variety of conditions you can use while writing. For example, something very common when writing into a search engine is that you only want to write a record if the version you're writing is newer than the version in the database. Imagine a system like Shopify's, with a million of these systems where every time someone changes a piece of data, they write it into the database, and you also have bulk ingestions in the background. You want to make sure that only the most recent document survives. You can do a variety of other things, like patching attributes with a filter, which is like an update. Right now the JSON API to turbopuffer is fairly simple. Over time, as the queries become more expressive, we expect we would probably ship a SQL interface.

## Scale

<!-- t=222 -->
I also wanted to frame a few numbers around the scale turbopuffer runs at, to contextualize what kind of environment we're making decisions in and scaling for. As far as I'm aware, we host the most vectors in the world: more than 3 trillion vectors and documents with full-text search in production. We run these at tens of thousands of queries per second, and we see peaks just shy of 100 gigabytes per second. We also have more than 200 million namespaces. You can think of namespaces as shards or tables, and we'll get into more depth on them later. The largest search spaces we see are people ingesting all of Common Crawl into turbopuffer and searching it at once. That's often in the high tens of billions of documents searched at once, which poses a lot of challenges that we'll talk about today.

<!-- t=279 -->
This talk has three parts. First, V1. V1 was the version of turbopuffer that I wrote at my cabin up here in Canada over the summer of 2023 and put into the market to see if anyone would care about this kind of database. We took that to its limit, but eventually turned it into a real database with V2. I also want to connect this to Andy's theme of justifying turbopuffer's existence against being a Postgres extension or merely Postgres. Why build another database, and what conditions have to arise for a new database to be successful, both commercially and in being useful to the companies around the world building software on it?

## Background and scars from Shopify

<!-- t=328 -->
To properly understand a database, you have to understand the DNA and origin of the people who wrote it, and what kinds of scars or experience they go in with while designing the system. My background is not in database research. It's not even in working on databases. I didn't even go to university. My experience is fully from operating these databases and building layers on them for scale.

<!-- t=357 -->
I worked at Shopify between 2013 and 2021, while we were scaling from 1,000 requests per second to more than a million. The hardest part of these apps to scale ends up being the databases. That's what most likely causes the most painful outages on Black Friday, and it's what we were always preparing for. During my time there, I worked primarily with MySQL, Redis, Elastic, Memcached, Kafka, ZooKeeper, and many others, and learned to combine them in all kinds of creative ways.

<!-- t=394 -->
Being on the last-resort pager, where I've been paged by every single one of these databases many, many times, has irrevocably changed the way I write software, and the way Justine, my co-founder, writes software. I have debugged every one of these databases for hours on end, completely sleep deprived. I have been woken up by some of them every single day in the days leading up to Black Friday. It taught me pretty much everything I know about designing systems that don't wake you up, because that's what you end up optimizing for. For better or worse, that's one of the most important things I keep in mind when designing software now.

<!-- t=435 -->
The other thing I learned is that it's very uncommon to have this long a tenure at a tech company these days. Eight years working on a single piece of software and scaling it teaches you a lot about how software ages well and what doesn't. My biggest takeaway, which is quite congruent with being on the last-resort pager, is that it continues to shock me how the things you put in place three days before Black Friday with spit and bubblegum, as my boss used to call it, tend to be some of the things that last the best. I keep saying that Justine is probably the best person I've ever seen at coming up with one-line fixes to 1,000-line solutions. It continues to surprise me how something very simple, when you think hard enough about the problem, can age very well compared to the RFC that's been debated among ten principals and gone through revisions by every team before it makes it into production.

<!-- t=495 -->
Every one of these databases has heavily influenced the way we think about designing turbopuffer. I don't want to sit here ragging on other systems, and it obviously depends on many factors when you deploy them.

<!-- t=508 -->
**Andy:** Which of the six was the worst? The one that I now compete with. Okay, the one you have the most scars with. Okay, sure. Fair.

## Napkin math

<!-- t=522 -->
**Simon:** The other major thing that goes into design at turbopuffer is a project I run called napkin math. Napkin math is a collection of numbers for how machines are supposed to perform. There are more than the ones shown here; these are just some of the most applicable to turbopuffer. It's a collection of things like: what bandwidth can I expect on an SSD read? Not what the device says on the drive, but what a reasonable amount of code would do. What can I expect from object storage? I didn't get to update this number in time, but you can expect about 100 megabytes per second from a single connection, though of course you can exhaust the NIC if you use as many connections as possible. The P99 is around 50 milliseconds, and that doesn't vary very much between a block size of 128K and one megabyte. All of these numbers are maintained in a repository containing the first Rust I ever wrote.

<!-- t=581 -->
The reason I worked on this is that, in my role as a principal engineer at Shopify, I spent a lot of time reviewing how people were going to use some of the databases we just talked about to develop a feature. Often they would show up with a profile: "I did the workload this way on this database," and then they would make decisions based on those benchmarks. Often those benchmarks were wrong. I found myself saying again and again, "Look, it has to visit the B-tree this many times. There's no way it's going to take more than this amount of time." Then sure enough, you'd look at the query plan and the database wasn't doing what it was supposed to do.

<!-- t=620 -->
So I collected all these numbers as a way to construct a very simple model of the system, and from it a very simple hypothesis for how it should perform. Then you see the difference between how the system actually performs on a query and what the napkin math tells you. If that gap is off by an order of magnitude, then either there's a massive opportunity to improve the system, or there's a bug in your understanding and your model of the system. This always beats profiling. Profiling will tell you how to optimize the system by 10% or 5% here or there, but it will not tell you what the absolute floor of latency or performance is.

## Object-storage-native design

<!-- t=661 -->
The other thing in the air was building a database with a different kind of design. Back to my scars from being on call for a very long period: it seemed clear to me that the only database I would go on call for was one where I didn't have to write the storage layer, where the storage layer relied entirely on S3, including even the consensus layer.

<!-- t=688 -->
The way we've decided to build turbopuffer, for maximum operational simplicity and reliability, an object-storage-native database is one where object storage is the only stateful dependency. There's no east-west coordination and no consensus layer in a separate system. Everything about the system is in simple files on object storage.

<!-- t=712 -->
The second condition for an object-storage-native design, which I sketched out in 2023, is that the storage engine and the query planner should be object-storage aware. They should know that the round-trip P99 to object storage is around 100 milliseconds, and that those round trips are costly, but that concurrency is heavily encouraged. If you can minimize the number of round trips but do a lot of outstanding concurrency, that's really good for object storage. So the query planner and the storage engine should be deeply object-storage aware in a way that's not possible to easily build onto another system.

<!-- t=750 -->
The third condition is that coordination is done with compare-and-swap directly on object storage. Modern object storage today has operations where you fetch an object, get a tag back, and only write the object back if the tag matches. This allows you to drive consensus with object storage, at the cost of latency, obviously. But if you carefully design the system, you can still get very good performance with this primitive alone.

<!-- t=780 -->
This kind of database is only something you've been able to build since December 2024, when S3 finally released compare-and-swap. Google Cloud Storage had it beforehand. That's actually the one I was familiar with when I started turbopuffer, because Shopify ran on GCP. NVMe SSDs were also not available in the clouds until 2017, and most databases today still aren't running on them. They're running on network disks that are not particularly fast. S3 also didn't become strongly consistent until 2020. Before that, if you wrote an object, you weren't guaranteed to get the same thing back on the next read. That meant all of the databases built on object storage before then had a fat metadata plane to coordinate all of this.

## Trade-offs of the architecture

<!-- t=830 -->
These conditions allow us to create a database that is completely different from the ones that came before it. It has trade-offs. When I look at a database website, sometimes it's difficult to figure out whether they're selling a sneaker or a database. To me, what should be very clear is that every database makes very different trade-offs in how it stores the data and in the storage architecture it has chosen.

The strengths of turbopuffer's storage architecture:

<!-- t=865 -->
- **Very low cost.** Everything goes to object storage, at two cents per gigabyte to store.
- **Simple and durable.** You could blow away every single VM in turbopuffer's accounts and we would not lose data, because we never acknowledge a write until it's been acknowledged by S3.
- **Simple horizontal scalability.** All of this is pushed onto the hundreds, if not a thousand-plus, engineers who work on S3 or Google Cloud Storage.
- **Fast warm queries.** When data is in cache, in DRAM and on the NVMe SSDs, there's no reason it's not as fast as any other system.
- **Large write throughput.** turbopuffer does close to 100 gigabytes per second at peaks.

<!-- t=906 -->
The weaknesses are very clear:

<!-- t=908 -->
- **Slow cold queries.** When you go directly to object storage, the P99 of reading a block is around 100 milliseconds, and it can spike higher if you have a lot of objects on a node and a lot of opaque caching at play. A cold query to turbopuffer can easily be 500 to 1,000 milliseconds.
- **High write latency.** The P99 for a put on object storage is around 100 milliseconds for a small write, and it gets larger as the writes get larger. This prohibits certain types of workloads. You could never hot-swap a relational database for this, even if turbopuffer supported all of SQL. Something like Shopify would never work on a database like that, because the write latency would be too high for things like inventory reservations.
- **Consistency round trips.** Because we have to drive consensus with object storage, to get consistent queries we have to round-trip with a GET If-None-Match to make sure the caching node has the latest write. That takes about 8 milliseconds on S3 and about 15 milliseconds on Google Cloud Storage.

<!-- t=972 -->
You can mitigate these weaknesses in the limit. You can pin things in cache, or implement some east-west coordination to drive consensus. But in the object-storage-native design turbopuffer uses, these are the fundamental weaknesses. They are very good trade-offs for search.

<!-- t=1007 -->
**Audience:** I had a quick question. You listed high write throughput as a strength but high write latency as a weakness. These seem to collide, and I was wondering why.

<!-- t=1030 -->
**Simon:** You can max out the network. The modern VMs turbopuffer runs on have maybe 50 gigabits per second of networking, and we can max that out and write it into object storage. But even if you did a small write of just a few kilobytes, we'd have to round-trip to S3 and wait 100 milliseconds. So it's a classic throughput-versus-latency trade-off. If you're writing a megabyte or 100 megabytes, we might be able to do that in the same amount of time, even though in another database those would have very different characteristics.

**Audience:** Thank you.

## The problem: the cost of search

<!-- t=1070 -->
The problem turbopuffer was initially trying to solve, and is still solving, is dramatically reducing the cost of search. I knew this problem from the back of my head, from operating the Lucene-based clusters at Shopify and the operational woes of doing so. When I left Shopify in 2021, I was helping a bunch of my friends' companies with their database struggles, which in 2021 mostly boiled down to tuning autovacuum in Postgres.

<!-- t=1104 -->
One of these was my friends at a company called Readwise. They had indexed a bunch of articles and wanted to build search, in particular vector search, over all of it. I ran the back-of-the-envelope math, and it would have cost 30 grand a month to index all the data. This was a bootstrapped company spending five grand a month on all other infrastructure combined. So just doing search for 30K a month was not tractable. It was completely bonkers to me that it would cost that much to put all these vectors in. It occurred to me that it was because the incumbent solutions were all doing this in memory, even though the workload seemed particularly well suited to a completely different architecture. These were the things I put together to start the first version of turbopuffer.

<!-- t=1150 -->
Look at the math on the cheapest way to store versus the most expensive and most performant way. The classic way to run a database is to replicate it across three SSDs, with up to 100% of the data hot in RAM. On a modern cloud, that costs somewhere between $2 and $3 per gigabyte. That's two orders of magnitude more than having all of it in S3. Of course, S3 has completely different performance characteristics: high throughput, high latency. But it felt to me that you could create a database that would puff up between these layers of the memory hierarchy. Andy is nodding, he's getting the name now. Maybe even turbo-puff up, into the NVMe SSD and into memory.

<!-- t=1205 -->
One of the fundamental trade-offs a lot of database systems make, if you think about the hierarchy between object storage, SSD and memory, is to keep as little as possible in memory and on disk and as much as possible in object storage. So the cache efficiency of the system is very important. In the limit, if you're doing 10,000 QPS to a workload, the economics of DRAM make sense. If you're doing a query every hour, it probably makes sense to have it in object storage. If you can build a system that turbo-puffs into the memory hierarchy at the right line, that can be very useful for search.

<!-- t=1246 -->
Think about our customers like Cursor or Notion. When you open a Cursor workspace, the data doesn't need to be in memory until you start querying that codebase a lot, and all the embeddings for it. Same with Notion. When you search, we can know ahead of time to start warming the caches as you type into the query box. There are lots of tricks we can play to warm things into the memory hierarchy. And if something is busy, we can keep it in the memory hierarchy at all times.

## V1

<!-- t=1277 -->
The first version of turbopuffer was very simple. It ran on an eight-core node in a tmux instance in GCP. That's simple, but it's not irresponsible, because every single write was committed to S3. The object-storage-native model lets you create really simple but still very reliable databases. At the time turbopuffer was really novel in charging about $1 per month per million vectors. Some of the incumbents were charging close to $100 for storing the same thing. That's what really put it on the map.

<!-- t=1316 -->
I should also mention that this was more of a summer project. I did not intend for it to become a big company. It was really just "I want to see if I can do this, because it seems fundamentally possible." The idea completely consumed me. As you can see here, I also don't say that it's the fastest. A lot of databases say, "Oh, we're so fast." No, it's reasonably fast. But the cost unlocked new use cases I'd known about from working with Readwise.

<!-- t=1346 -->
Now we'll get into the technical implementation of V1. Some ideas survived and some did not.

### Architecture and the read path

<!-- t=1354 -->
What survived is the fundamental architecture. Pretty soon after launch, when there was interest, we ported it to something that runs on multiple nodes. The architecture is largely the same today. People have evolved it, but this is the current form, and it has served incredibly well.

<!-- t=1373 -->
If we follow a read through this: it first hits the load balancer. The load balancer takes the table, or the namespace as we call it, that you're querying, and does a consistent hash on it, proverbially. You can imagine that this is arbitrarily complicated, to get you to the node most likely to have that namespace in cache. It goes there, starts reading the metadata, and fetches all the blocks required to serve the query. It goes through the memory cache, then the NVMe cache, and finally to object storage for the files, or the subsets of files, it needs.

<!-- t=1417 -->
If you do a cold query on a million vectors, or a few million, it might take about 500 milliseconds to do the three to four round trips to object storage. If it's in cache, it takes about 10 milliseconds, and the majority of that time is spent waiting for S3 to round-trip so we know we have the latest data. If it's on disk, it might take a little longer, maybe 20 milliseconds. That gives you an idea of the profiles through the cache.

### The write path

<!-- t=1447 -->
For a write, the path is more or less the same. It goes to the same node that would be serving the queries, and we write it directly to the write-ahead log on S3. You can imagine 1.bin, 2.bin, 3.bin, and we continue to append into that directory. When it's been committed to S3, we send back a success to the client. Then we write it into the cache on that node if we think it's prudent. If there are a lot of reads and writes going on at the same time, it's a good idea to write straight into the cache with any new writes.

<!-- t=1484 -->
When the WAL is far enough ahead of where we last built an index, whether the vector index, the full-text search index, the attribute indexes, or other secondary indexes, it puts a task into the queue. The queue also lives on object storage. The indexing nodes, sometimes called compactor nodes in other designs, work off that queue and do things like compaction and building indexes. As with the query nodes, there needs to be some affinity.

<!-- t=1520 -->
What I love most about this design is that we can move namespaces around just by changing where they are in the load balancer. At Shopify, if we wanted to change where a shop lived, we had to move all of the data between MySQL shards. With this design, we can just take a cache miss or start warming another node. Another nice thing is that if you want read replicas, you're really just round-robinning across multiple nodes, and they pick it up from cache. If we have too many reads, we can spread out the load. It also makes deploys really nice, because we can roll through the whole system at some pace and just manage the cache hit rate.

### The write-ahead log

<!-- t=1563 -->
The write-ahead log also survived from V1. It's about as simple as you might imagine. We don't actually use 1.bin, 2.bin, 3.bin. These are UUIDs committed into a manifest, but for simplification this is how the WAL progresses over time.

<!-- t=1586 -->
Most databases do this. MySQL accumulates writes into a buffer and syncs it periodically. On a disk, if you have a bunch of 100-byte writes, you're not fsyncing every single time you get 100 bytes. You accumulate some and fsync a larger buffer. It's the exact same design on object storage, but with latencies in the hundreds of milliseconds instead of hundreds of microseconds. So when you're doing a lot of individual writes, we coalesce them into a buffer and then commit them to object storage. Each object store has very different characteristics for how you can commit to the WAL, but it continues to progress like this: we commit a file and then commit it into a manifest with a compare-and-swap.

<!-- t=1633 -->
When you do a read, we do a consistent read on the manifest, which holds an inventory of the WAL. We can do that with a GET If-None-Match, which only goes to the metadata layer. That has better performance than loading the file, which has to go to the storage layer of S3. These have different latency profiles on the different object stores. S3 is about 8 milliseconds, GCS is about 15 to 18 milliseconds, and Azure is actually 5 milliseconds, last I heard. We get that, and then we replay the WAL on top of what we got back from the index to make sure the query is consistent.

<!-- t=1671 -->
This is an unusual design choice for most search engines, which just have eventual consistency. But we've worked around eventual consistency before, with Elastic and other solutions, and it's such a pain that we did not have high conviction in giving up consistency. You can give it up and your queries will be faster, but by default turbopuffer is strongly consistent and replays the WAL entirely on top of the index read.

### Unlimited shards and namespaces

<!-- t=1700 -->
Another design characteristic of V1 was giving access to effectively unlimited shards. When Justine and I were designing systems at Shopify, we were always abusing the shop ID key as much as we possibly could. I've always wanted a database where that was available as a low-level primitive, where every piece of data that is logically divided from everything else could be completely separated. Similarly, the primary key matters. This is one of the things I love most about MySQL over Postgres: it's very easy to organize all the data by tenant ID, whereas in Postgres you'd have to rewrite the entire table, or the entire heap, to do that.

<!-- t=1744 -->
So turbopuffer exposes this as a first-level primitive. When you work with a namespace, or what you might call a table or a shard, it has a particular prefix on object storage. This is not actually what it looks like, because you have to randomize some prefixes to get it to be performant, but this is the simplest lie-to-children metaphor for what we do. We have a bunch of metadata files that the WAL is progressing through. The LSM has a bunch of SSTables, and then you have a large metadata file that spells out which keys are where, in a very simplified LSM 101 type of design.

<!-- t=1780 -->
We have hundreds of millions of these namespaces in production, I think close to a quarter billion. Maintaining all this metadata is certainly a challenge, but it allows for really good cache efficiency on use cases like Notion and Cursor, where all the data is so logically separated from each other, and your bin-packing problem becomes a lot nicer and more tractable.

<!-- t=1804 -->
**Andy:** What is the metadata challenge? Is it because S3 chokes on it, because you have so much in it? Or is it on your side, maintaining the in-memory representation of what's out in the namespaces?

<!-- t=1816 -->
**Simon:** You have to once in a while loop through all of these things, say once a day, for billing or reconciliation. A list call to get 1,000 entries takes somewhere between 500 milliseconds and a second. So paginating through the whole thing is a challenge. I have a bonus slide, if we have time, on what I call the object storage bag of tricks. You have to play a bunch of tricks, still entirely on object storage, to manage that. And the billing layer is very high cardinality as well.

<!-- t=1851 -->
**Andy:** So if Amazon magically made the metadata lookup sub-second, your life would be easier?

<!-- t=1860 -->
**Simon:** Yes, I would say so. There would be a lot of things we would not have to design around. The list call is not one I recommend ever using in prod on a critical data path.

**Andy:** Okay. Awesome. Thanks.

### Fast cold queries

<!-- t=1870 -->
The other thing that survived from V1 and continues to be important is very fast cold queries and object-storage-native query plans. The general insight is that you want to minimize the number of round trips to object storage as much as possible. As was asked earlier, you can drive a very large amount of throughput. You can saturate the NIC. But you're not going to do it in less than a few hundred milliseconds of latency, so you'd better make good use of it.

<!-- t=1902 -->
Fortunately, this is also how NVMe SSDs work. If you do a very large amount of outstanding concurrency with few round trips, they perform incredibly well. But if you have a lot of dependencies in your I/O chain, they won't perform as well. That seems to be the trend in CPUs too: if you can do a lot of dumb, simple operations with very few dependencies between them, the machines rip. That's been our experience with object storage, NVMe, and CPU. Everything is designed around this, and we design on the assumption that these cold queries are quite common.

<!-- t=1939 -->
You can imagine that in the first round trip you get some metadata, like an LSM.json and an index.json. These might not actually be JSON files, just for illustration. Then you get the upper layers of some tree, then the lower layers of some tree. And then, if you have a select star or whatever, you get a bunch of data at the end, if you couldn't speculate earlier.

<!-- t=1958 -->
The second thing you need for fast queries on object storage is that you only get about 100 megabytes per second from every connection you open. If you have a large SSTable and want to download it fast, you have to split it into a bunch of different ranges and parallelize all of those requests. Conversely, if you're fetching a little bit of data, you might as well fetch the entire block and even adjacent blocks. Fetching a megabyte of data might have a P99 of, say, 80 milliseconds, whereas fetching much less has a P99 of maybe 60 milliseconds. So it can be advantageous to fetch more data at a time. With large block sizes, the P99 just does not vary very much between getting 16 kilobytes and a megabyte.

<!-- t=2002 -->
It's also important for all of this to be zero-copy. A lot of tiered databases will download something from S3 and then take their jolly time decompressing it and hydrating it into memory before serving queries. Not all of them can do a direct range read and start operating on that data immediately, because they haven't been designed for it from day one.

<!-- t=2027 -->
**Audience:** Are you using io_uring to avoid copies from the NIC?

<!-- t=2032 -->
**Simon:** We haven't needed to use io_uring yet. We're using direct I/O, and it has not become a bottleneck, which has surprised me.

<!-- t=2041 -->
**Audience:** Direct I/O is for the disk, but if you pull from the network, it has to go through the kernel.

<!-- t=2046 -->
**Simon:** Yeah, we're doing the copy right now. It hasn't been a bottleneck yet.

<!-- t=2050 -->
**Audience:** Okay. And how large are your block sizes?

<!-- t=2054 -->
**Simon:** I believe right now they're around 128K, to balance between the NVMe SSD and S3. You can imagine a system with different block sizes, but I think that's what it is. They might have changed it.

<!-- t=2070 -->
**Audience:** It's kilobytes, not megabytes. Got it. All right, thanks.

<!-- t=2075 -->
**Simon:** On the right here we can see the cold performance: it's still sub-second as we go directly to object storage, and the warm namespaces are just as fast as any other system. If you look at our production traces, you'll see maybe a millisecond or less actually servicing the query, and then 10 milliseconds going to S3 to make sure we have the latest entry from the WAL. You can turn that off for other characteristics, but we think this is a really good default for search systems.

<!-- t=2103 -->
The reason fast cold queries are so important is that if, let's say, our cold queries were 10 seconds, you'd end up patching that everywhere in the system. You'd pin more stuff into the cache and have larger caches, and that passes poor economics on to our customers. So it's very important for cold queries to be fast, so that everything in the system is not so fragile.

### Vector indexes: graphs versus clusters

<!-- t=2128 -->
Another thing in V1: in 2023, when I was writing the first version of turbopuffer, HNSW and graph indexes for vectors were all the rage. The problem with them, to me, is that they have a lot of dependencies. They're very difficult to speculate on, connecting to the earlier point about modern hardware.

<!-- t=2150 -->
The simplest way to do a vector search is to take a query vector, compare it against every single vector in the data set, and return the ten closest by the angle between them. The most popular algorithms at the time constructed a nearest-neighbor graph. You can imagine these vectors in two dimensions, as they are here, and you try to connect the vectors that are closest to each other in a graph. Then you drop yourself at some point in the graph and do a greedy search along it to find the closest vectors to your query vector.

<!-- t=2185 -->
The problem with a graph is that every time you navigate between these edges, you can't predict how far you're going to navigate, so you can't speculate and page all of that in from disk. You end up with all of these dependencies. On object storage, it's 100 milliseconds, 100 milliseconds, 100 milliseconds, every time you walk the graph. And on disk, a random read is in the hundreds of microseconds, which is still comparatively quite slow.

<!-- t=2209 -->
A clustered index is the more traditional approach. You take the vectors and divide them into clusters, trying to find some underlying pattern. You average the vectors in a cluster, and that average is what's called a centroid. When you search, you compare your query vector against the cluster centroids, fetch only the few clusters closest to your query, and exhaustively scan through those to get your top results. This has two round trips. You can imagine a centroids.bin, again not actually JSON, and then cluster1.bin, cluster2.bin, and you download the appropriate data. That's not very far from how it worked in the first version of turbopuffer. It's fast on disk, fast on object storage, and very simple to implement.

### Caching

<!-- t=2257 -->
Caching has also not changed very much from the very first version of turbopuffer. RAM is a simple cache. On the NVMe SSD we use a FIFO ring buffer, where you just keep writing, and at some point you wrap around and start eating the tail. You can do a variety of interesting techniques on top of this to make sure values that shouldn't be evicted from the disk cache are not. Right now in turbopuffer it's very simple. We think complexity has to be deserved, and right now this is working quite well.

<!-- t=2294 -->
It's actually both read-through and write-through: at write time we write into it, and when we read an object from object storage or lower in the hierarchy, we populate it upward. The cache is prioritized, so namespaces that are really busy get warmed with priority over others. Cache efficiency is a heavy investment area for us. This is also where you get control without separating compute and storage. I call this "compute-storage flexible," because there's no reason you can't tell the cache to always keep something in cache if you want your P99 to be very low over very long periods of time.

### Reaching the limits of V1

<!-- t=2327 -->
V1 reached its logical limits in the middle of 2024. The first version of turbopuffer was literally a caching NGINX in front of S3 as the caching layer. When I brought Justine, my co-founder, along, this was obviously the first thing we fixed. But hopefully it illustrates how simply we've iterated. Every part of turbopuffer is the simplest, most reliable thing we know will work, and I have operated NGINX for more than a decade, so I know it works and I know how to configure it. Every layer of complexity we add has to be deserved. That's the only way I know how to build reliable systems.

<!-- t=2371 -->
Full-text search was bolted on, but it was very limited as a more general-purpose database. I believe any large database, over time, ends up implementing every query plan, so it has to be designed as such, otherwise you'd be pigeonholed into a single workload. When your customers have commercial traction on your database, they will expect more and more types of queries to execute on it, as they trust it and put more and more data on it over time.

<!-- t=2398 -->
The vector search, taken to its logical extreme, was what found us product-market fit. It was also what ran Cursor and Notion, and what got us the POC with Anthropic towards the end of 2024.

<!-- t=2412 -->
While we were working on all of this, we hired Bojan, whom I met at the IOI in 2012 and who actually knew how to write databases. So in parallel, with primarily Justine and also myself optimizing V1, we were writing a proper V2 of the database, which is the version we're on now. We joke that there are still remnants of V1 in the code base, and Morgan calls removing all of the founder code an important company objective.

## Why a new database? Not just Postgres

<!-- t=2450 -->
Before we go to V2, I want to justify the existence of a new type of database. The title of this series of talks is "Why not just use Postgres for everything?" There are two things I keep coming back to.

<!-- t=2466 -->
The first, if you want to build a large, commercially viable database, is that I think you basically need the propensity for every company on earth to have data in your database, either indirectly or directly. Even the coffee shop around the corner from my house, I can guarantee you, has some of its data somewhere in an Oracle database, in Snowflake, in Databricks. The biggest database companies in the world have a little bit from every company on earth. If you want to build a big database company, that has to be part of your vision. The best time to do that is when some underlying new workload or paradigm shift causes it to happen. For Snowflake and Databricks, suddenly we had websites with lots of traffic and wanted to do large-scale analytics on them with fundamentally new economics. Oracle, PostgreSQL and others came before that, because we had computers and wanted to put lots of data in them and make it possible to query. These are not generations one, two and three in the sense of one being better than the other. All of them have different, phenomenal trade-offs. turbopuffer is just another set of trade-offs that happens to work very well for the new workload right now, which is connecting very large amounts of unstructured data to LLMs. That's the workload causing traction for turbopuffer.

<!-- t=2554 -->
The second thing you need is a new storage architecture that the incumbents can't easily adopt. If you only needed the first, you'd imagine that around 2013, as everyone started creating mobile apps, a big company called GeoDB would have spun up and dominated the world, because everyone has geo-coordinates. But there's no reason geo-coordinates can't be supported in gen one and gen two databases. The new storage architecture we can create now is what I call compute-storage flexible, or object-storage-native. It was enabled by these developments, and we can now create a database with trade-offs that are quite phenomenal for this new workload. These are the new attributes, and this would be very difficult to pull off inside of Postgres. Of course, with Postgres you can make computers do anything in the limit, but there has to be some pragmatic trade-off about when you do that.

## V2

<!-- t=2611 -->
V2 of turbopuffer is a bunch of evolution on top of the ideas that were good in V1. The biggest change is that V1 operated more or less directly on very simple files on object storage, whereas V2 introduces a key space maintained by an LSM, with all the different data structures implemented on top.

<!-- t=2635 -->
The other big change was incremental indexing. The hilarious thing about V1 was that once the WAL had grown enough and we had to rebuild the index, we rebuilt the entire world. The compute amplification, quote-unquote, was enormous, but it was very simple and very reliable to maintain. V2 also brought branching and our own load balancer. We'll go through these in a little more detail.

### The LSM and compaction

<!-- t=2660 -->
Our LSM by academic standards is very disappointing. It's really an LSM 101 implementation. The system side has been heavily optimized for performance, but in terms of compaction it's more or less the simplest tiered compaction you can imagine. We expect to spend a lot more time here as it becomes a bottleneck for new workloads.

<!-- t=2688 -->
If anyone in this room is considering what to research at some point: I think the compaction territory we're in with these object-storage-native databases has quite a different set of trade-offs from a traditional database. On a traditional database shard, you can't spend too much compute, I/O or memory compacting the data, because you're contending for the same resources as you are for serving queries. In our case, you can do a compaction on a 24xlarge index node for a very short duration and build indexes on it as well. That's a different set of trade-offs. Space amplification is very cheap, because we're using S3. You have to deal with a new type of amplification, on the number of S3 operations you're doing. Write amplification is cheap. Read amplification is more expensive. Round trips matter, and throughput matters a little less. As far as we know, there's no research on doing compaction in this kind of environment.

### Incremental vector indexing

<!-- t=2755 -->
The other thing we did in V2 was make vector indexing incremental. Again, somewhat hilariously, in V1 the vector indexing was not incremental; we rebuilt the world once the WAL had grown enough. SPFresh is a paper about how to incrementally maintain clusters. If you do clustering, you can imagine doing a bunch of very simple, repeated multiplications and distance computations to find the right clusters. You also have an intuition that you should be able to maintain them: you insert a new vector into a cluster and the cluster expands. But at some point the cluster becomes large and you get tail latency. So if a cluster grows very large, you can split it. If you're removing vectors from a cluster, you can merge it with adjacent clusters.

<!-- t=2807 -->
The problem is that this causes nearby clusters to potentially change too. If you're splitting or merging clusters, vectors that were previously in those clusters might now belong better to other clusters. This is a concurrency and implementation nightmare, as you can probably imagine. But it works unbelievably well at very large scale, with very high indexing throughput, and the performance characteristics on disk and object storage are really good. We've had this in production for almost a year, a year and a half, and we're now working on a new version that will be even more performant, with everything we've learned about running it. This is certainly one of the core parts of turbopuffer.

### Hierarchical clustering and quantization

<!-- t=2856 -->
The other thing we do is hierarchical clustering. If you have a very large vector index, say a billion vectors, scanning through all those centroids to find the closest clusters becomes a lot of compute. So we create clusters of clusters. Essentially we're creating a tree.

<!-- t=2877 -->
We also quantize. We take these big float, F16 vectors and turn them into binary vectors, so we can do very simple operations on them. The algorithm we use to quantize, RaBitQ, gives us an error bound on which ones we have to re-rank with the full-fidelity vector to hit a recall target. These are all approximate algorithms, so we have to have high recall, which is the percentage overlap with the exact result.

<!-- t=2910 -->
These are some of the things we've done to support use cases where people ingest all of Common Crawl or other very large data sets into turbopuffer. We wrote a blog post about how we did this. It's how we can search around 100 billion vectors with a P99 of 200 milliseconds. It's probably been brought down since, but we've never heard of a result at that scale.

### Filtering with vector indexes

<!-- t=2933 -->
Filtering is a big challenge with vector indexes. I think a lot of people somewhat irresponsibly slap a vector index onto their existing query planner. The query planner has to be aware that the clusters matching a filter might be very far away. That can create all kinds of problems in planning the query, where you don't actually get the closest vectors. I'm not going to go into a ton of detail here; we have a blog post with more. The point is that the query planner certainly has to be aware of the distribution of the filters relative to the vector data in order to plan a query with high recall.

### Full-text search

<!-- t=2968 -->
We have three more indexes to go. The full-text search index in turbopuffer is fairly sophisticated, and it draws a lot of inspiration from Lucene. We're starting to see comparable performance, and even better performance, than Lucene on very large data sets with very long queries. Those are the ones we're increasingly seeing in production, as LLMs increasingly write the queries.

<!-- t=2997 -->
Essentially, when you do full-text search indexing, you split the corpus by, say, whitespace, and put all of that into a map. Then for each term you have the set of documents that match. You also have to do a lot around scoring. With every block of documents, we store the maximum contribution to the full-text search score of the documents inside that block. That allows an enormous amount of skipping. For example, if you're searching for a term like "puffer," it's a lot rarer than a term like "the," so the score in that block is a lot higher. We can use that, as I'll show on the next slide with the skipping algorithm, to rip through these postings and intersect them. Full-text search really boils down to intersecting very large posting lists, so it becomes fundamentally a bandwidth problem.

<!-- t=3063 -->
To use these posting lists in a clever way, the simple observation is that if you're searching for something like "New York population," there's a point where it doesn't make sense to look at any documents that have "new" in them anymore. Any document needs to have "population" and "York" to actually be viable for the query, so we can stop intersecting the "new" list. This algorithm is called MaxScore, and it performs really well on modern CPUs. Recently the trade-off has switched in favor of MaxScore over WAND. A lot of the complexity in search engines is about skipping documents that no longer matter for ranking. This is the one we've implemented in turbopuffer, and it's also the one in Lucene. I think those are the only two implementations of it.

### Regex and filtering indexes

<!-- t=3114 -->
Regex queries are another type of text query. To serve them, you parse the regex into an AST, break it into trigrams, and then post-process all the documents that match the trigrams that are definitely going to be in the matching documents. This boils down to a full-text search query, but the number of trigrams is finite, so we can index some numbers and save a bunch of efficiency. If you're searching for something like "main" and "util," we break that into trigrams. There are some regexes that are very difficult to compile into trigrams, and in that case you have to run the final regex on the document. This is very common for code.

<!-- t=3156 -->
The last index is the filtering index, for exact filtering. This has to work congruently with the vector queries and full-text search queries, and the planner has to know when it's better to filter versus score. Sometimes it's better to score and then filter, and other times it's better to filter first. In a lot of ways it shares conceptual logic with the inverted index we use for full-text search, and underneath it's a simpler bitmap implementation that figures out which filter matches go into the heap.

### Queue and branching

<!-- t=3189 -->
I won't go into too much detail on this, but we've also written a queue on top of object storage to avoid having another dependency. This is not for the WAL or ingestion of data, but just for notifying the indexes of when to compact and do various maintenance operations. It's essentially a single file called queue.json that gets a lot of CAS operations, and it goes through a broker that does group commits on this file with all the traffic from lots of query nodes. It works great. It's a very simple design. Again, we only introduce complexity when it has been deserved.

<!-- t=3221 -->
The last thing coming to turbopuffer very soon, which is easier on an object-storage-native database, is branching. That's the ability, at a particular LSN or WAL entry, to branch off, start a completely new timeline, and keep references back to the previous branch. This is common in coding, in development, in testing and in lots of other use cases, and it's coming to turbopuffer very soon.

## What's next

<!-- t=3248 -->
The final slide: what's next for turbopuffer is to implement more and more query plans. There comes a time when you have to steer clear of the uncanny valley of continuing to use JSON syntax and actually give people SQL. I don't know when we're at that threshold, but I think we'll know it when we see it. Also faster, of course, and continuing to iterate on caching and lots of other features, like the branching I just mentioned. I'll leave it there for questions. Thank you so much.

## Q&A

<!-- t=3278 -->
**Andy:** That was awesome. I'll clap on behalf of everyone. If you have a question for Simon, just unmute yourself and go for it. I know there's a question in the chat that I'll read out afterwards.

### Caching and write visibility

<!-- t=3297 -->
**Audience:** I just had a quick question to clarify my understanding of the caching. You said it was read-through and write-through. Let's say I have a search on "Postgres" and it's cached, and then I write a document like "auto-tune Postgres autovacuum," and then I read that again. Am I going to get that document right away, or how does that work?

<!-- t=3334 -->
**Simon:** It will go in the WAL, and the WAL would be written into the NVMe cache. I don't think we write it to the memory cache too. When you do the query, we will immediately see it, because it's in cache and we're doing the round trip to the WAL to make sure we see the latest entry. Then we apply that on top of the search. So it's immediately visible, and it's written into the cache because otherwise every time you wrote a document we'd have to go to object storage to fetch it.

<!-- t=3358 -->
**Audience:** But I thought you said you didn't acknowledge the write until it made it to object storage?

<!-- t=3361 -->
**Simon:** Right. You're not going to see it until the client has returned, until the client sees the 200.

<!-- t=3369 -->
**Audience:** Okay, got it. Thank you.

### Code versus text

<!-- t=3377 -->
**Audience:** Thank you so much for the talk. What, if anything, have you noticed while building turbopuffer in terms of the difference between storing text files versus code? Where have you seen the most interesting differences?

<!-- t=3405 -->
**Simon:** Let me think of a simple answer. Code is extremely sharded; there are so many different shards from all the companies that use code. And the write-to-read ratio is very high: there are a lot more writes than reads. In other systems with simpler text, the queries often get more complicated. In code, the queries are usually quite simple, like "find this regex," and there's generally very little filtering. Whereas for text, in an Atlassian or Linear use case, for example, there's a lot more filtering, and often a lot of permissions. I did not know how much time we would spend optimizing query plans for permissions, which are very large intersections and very well suited to posting lists over B-trees. Those are some of the first things that come to mind.

### What else would you build?

<!-- t=3478 -->
**Audience:** If you were not building turbopuffer, what would you be building in this gen-three kind of database, the object-storage databases? Or to reframe it: if you had to build another database company, what would you build?

<!-- t=3496 -->
**Simon:** This is a phenomenal question, and I've spent a fair amount of time thinking about it. There aren't that many companies to be built. Unfortunately there are a lot of databases to be built, but I don't think many of them will turn into big companies.

<!-- t=3514 -->
One was the streaming use case. WarpStream and others have essentially done that. I think there were companies to be built there. They're not quite propelled by a new workload in the same way, but it's just the superior architecture for streaming. The second category is search, which is the one we're in. The third category is real-time analytics. There are a bunch of vendors doing that, and it's not a category I would go straight for; it's very contentious.

<!-- t=3544 -->
The other category is observability. Almost every observability provider builds their own database in some way and then slaps a big UI on it. But it's not a place where you would actually be selling the database. You have to sell a full end-to-end experience. If I had to choose another vertical, it would probably be there, and I'd try to do it not at the 85% gross margin of the incumbents but at more like 60% or 50%, and try to expand from there. But that would be very capital-intensive, and it requires an enormous amount of UI work to pull off successfully.

<!-- t=3583 -->
The other thing I find interesting, beyond the ones I've mentioned, is that there are lots of companies that can navigate a very specific set of trade-offs and create their own database a lot more easily now than they could in the past. This is probably where the majority of these databases will be built: inside companies that have a particular set of trade-offs, but that's not a generally viable large database company. That's my somewhat pessimistic answer.

**Audience:** Thank you so much.

### Consistency within a namespace

**Andy:** Anthony, go for it.

<!-- t=3610 -->
**Anthony:** Thank you. I'm curious about the consistency model within a single namespace. My understanding is that the inverted file index is updated asynchronously after the write-ahead log commit. So if there are concurrent writes and reads, is there a risk of creating a mixed state, where some documents are indexed and others aren't updated in time? Is that an acceptable trade-off for the targeted workloads, and do you see cases where it could be a real problem?

<!-- t=3666 -->
**Simon:** Let me quickly reiterate how we do the searching. Vector search is actually easier. Let's say the WAL is 64 megabytes ahead of the index. We go to the index, consult the clusters, and do that search, and we replay the WAL on top. We kind of do those two operations in parallel, and then you have a consistent result.

<!-- t=3686 -->
I think what you may be talking about is a situation where you're updating a vector. On my second slide I showed how you can do some simple conditions, like a patch by filter. That requires running a query and then doing the update, and in between those events it's possible for something else to occur. So you'd have to rewrite the operation. turbopuffer is at a read-committed isolation level, so that is the kind of thing that could occur, and where having multiple writers could matter. That's fairly common in databases and other systems, and it felt like the right trade-off for turbopuffer.

### Competing with the hyperscalers

<!-- t=3726 -->
**Anthony:** Thank you. I also have a different question, more related to business. Given that the core infrastructure, S3 and GCS, is owned by hyperscalers, they could theoretically build something similar by investing enough time and resources. What do you think your moat is?

<!-- t=3751 -->
**Simon:** Working with the hyperscalers is something every database company contends with. It's just the rule of the land. Inside these large companies, the team that works directly with you is incentivized to have you grow as much as possible, and there's another team that's incentivized to grow their own database as much as possible. That works well enough, and the incentives are aligned with all the vendors. But of course, if they see a lot of commercial traction, they have the signals to try to build something similar. At least one of the clouds has done something that looks a lot like what turbopuffer looks like.

**Anthony:** Yeah, like S3 Vectors, right?

<!-- t=3794 -->
**Simon:** Correct. But it doesn't have the cache hierarchy and things like that in front. If you're building a database company and competing with the hyperscalers, you have to think about what you can do that they could never do. One thing they could never do is put the database engineers into a Slack channel with the companies that actually matter and have them work directly on the query plans. Another thing that's very difficult at that scale is managing all the caching and the multi-tenancy. And then there's releasing features at a particular cadence. I think a startup's only moat is focus. That's it. If you can focus on something for a long period of time, I think you'll do really well against the hyperscalers. Take the Postgres offerings of the hyperscalers: most of them are not even on NVMe SSDs, even though those were released almost a decade ago on cloud SKUs. So there are lots of opportunities for database companies to run these databases better than the clouds do themselves.

**Anthony:** Thank you.

<!-- t=3856 -->
**Andy:** I'll point out you built V1 in a cabin in Canada and launched it without going to any design meetings with a team. You can't do that at a big hyperscaler.

<!-- t=3870 -->
**Simon:** No, you could not. And V1 uniquely works because you can be very close to the customer. If anything comes up in a POC, you can give them the experience of fixing it immediately. Maybe another way to phrase it: when Notion became a turbopuffer customer, they were using every single line of code in the database. That's an advantage you have, that you can launch a lot quicker.

### Open file formats

<!-- t=3895 -->
**Andy:** All right, we're almost out of time, and I appreciate you going over. First question: it sounds like all your file formats for reading from S3 are proprietary or custom, things you developed. At any point in the design or build-out of V1 or V2, did you consider using one of the open-source file formats like Parquet, ORC, or the newer ones, Nimble and Vortex?

<!-- t=3919 -->
**Simon:** The main reason we didn't go that route was speed. We just wanted control. There's no philosophical debate. If everything could be in the customer's bucket in an open file format and we could move at the same pace, I would say that would be a superior design. But a startup's only edge over everyone else is speed, so we've always optimized for what is fastest and most reliable. It's the same reason we're not open source. We're only commercial, not because we don't think open source is great, but because we don't have time.

## Bonus: the object storage bag of tricks

<!-- t=3959 -->
**Andy:** Sure. And the last one: I think you mentioned you had a backup slide of your object store bag of tricks. I know we're over time. Could you share that?

<!-- t=3969 -->
**Simon:** Yeah. We will turn this into a full-fledged blog post at some point. There are a lot of tricks you have to employ when you're building turbopuffer, or an object-storage-native database. Let me take a couple of random ones.

<!-- t=3986 -->
Speculation is a big one. You're always going to want to speculate on what the next round trip is going to be and do it in parallel with whatever you're doing. We use this trick everywhere. If you see the traces for turbopuffer, it's constantly speculating on what we might need next.

<!-- t=4003 -->
The metadata layer for S3, GCS and the other providers is much faster than the storage layer. Fetching a four-kilobyte metadata file has a P99 of maybe 50 milliseconds, whereas getting just the metadata of the file, which I have locally, might have a P99 of close to 30 milliseconds, depending on the provider. The P50 is generally a lot lower for the metadata, but when we design, we always design for the P99.

<!-- t=4032 -->
Another one: we talked about lists earlier. One of the things we do is keep a read-through cache of a bunch of starting points in the bucket, so we can parallelize the list with thousands of concurrent list calls and know where to paginate from until they overlap.

<!-- t=4054 -->
404s are free on GCS. You can do crazy things with this if you really consider it. They're not free on any other provider, but if you're only on GCS, this is very interesting. Soft deletes are a responsible way to run the data.

<!-- t=4070 -->
**Andy:** Hold on. What do you mean, 404s are free? You're basically asking whether something exists?

<!-- t=4075 -->
**Simon:** Yes, and it doesn't cost you any money, though there's latency, of course. For example, one of the limitations in turbopuffer is that we can only commit the manifest file every so often. There are limits on how often you can overwrite an object, depending on whether you're on Azure or GCS. So if you're seeing a lot of contention, you could shard it into multiple manifest files, but only during periods of very high contention. You could then do reads to manifest one, two, three, four, five. Most of the time they don't exist, because you haven't sharded the file, so you're only getting one, but the others are there for you to randomly commit to. There are ideas like that where you can abuse it to shard different things without the ops amplification.
