AEADs provide a verification tag after encryption. For each page, we need a nonce too. Both the nonce & the tag become metadata for an encrypted page So where do you store them? We could store them separately, but it's much better & neater to store them in the page itself (1/5)

The B Tree data structure fascinates me. Databases use B Trees to store data on disk, organizing everything into pages that typically range from 4kb to 8kb. All I/O operations happen in units of these pages. The page looks like this... (1/9)

Pro database tip: enable `SQL_SAFE_UPDATES` in MySQL to avoid accidental UPDATE/DELETE queries without a WHERE clause. It forces you to use a key or a LIMIT, instead of wiping whole database by mistake at 2:19am.

Sharding. Database sharding is one of the common techniques to scale a database horizontally. You split the db into small parts called shards and distribute them across machines. Shards are typically in the few hundreds or even thousands (for extremely large databases).

SQLite has a page where they explain why they use C. They specifically elaborate on why not Rust www.sqlite.org/whyc....

Lil trivia to remember when it comes to Snapshot Isolation jepsen.io/consistenc...

excited to share that we are following through on our earlier commitments and putting together an independent+neutral organization to house the DID PLC system, includes the directory service

Next week is the start of @db.cs.cmu.edu's latest seminar series: Future Data Systems @samarchdb.bsky.social and I are hosting speakers from leading systems in the datalake / lakehouse space. Mondays @ 4:30pm ET via Zoom. Open to the public. Videos posted to YouTube: db.cs.cmu.edu/seminars/fal...

The correct answer is either. Transaction B gets a snapshot that may or may not include the changes from A. SI does not provide real time guarantees. If you need that, you need Strict Serializability, which guarantees that transactions are ordered in real time.

Database systems question Assume the database is in snapshot isolation mode. If transaction A updates, and writes x, commits, *then* transaction B starts and reads x's value, then B will see (assume single node for simplcity): 1 - Value before A's write 2 - Value written by A 3 - Either 4 - 🍿

Published a new blog post: Setsum - order agnostic, additive, subtractive checksum post - avi.im/blag/2025/setsum code - github.com/avinassh/...

The great lock in is here! For those wanting to get into systems programming and/or database internals, consider hacking on Turso DB, the SQLite rewrite in Rust. Here's why: 1. It's a database!

Hey we're hiring for in-person engineering roles in SF. I really enjoy my job and you might too. Come hang out and build developer tools!

Where can I learn about how AI companies use caching, KV stores, and databases differently for LLMs, agentic workloads? Someone also mentioned to me that old/traditional services aren't suited for these, so they also build databases internally. (e.g., OpenAI acquired Rockset)

For nearly a decade, MongoDB provided reliable persistence with one of the most robust storage engines. Yet somehow, the oldest jokes keep persisting, too, so here are some facts: dev.to/franckpachot...

Netflix had it all wrong, don’t waste engineering resources to build your own chaos monkey infrastructure, just put production on AWS us-east-1 and you get chaos monkey for free. Just kidding, just kidding…

This is the opening text of Transaction Processing: Concepts and Techniques by Jim Gray "Six thousand years ago, the Sumerians invented writing for transaction processing."

Published a new post: Oldest recorded transaction. This totally could have been just a tweet (skeet?), but I wanted to publish something today. avi.im/blag/2025/old...

Which is the best local LLM to set up for asking questions about code? I have large codebases like the Linux kernel, Postgres, etc. I want to ask questions like "find methods that do XYZ" and also post large code snippets and ask for explanations.

If you're you curious about #LeanLang and want to understand the connection between #programming and #proofs, check out this great new video by Ank Yog. The analogy between Chess and true propositions is particularly compelling! www.youtube.com/watch?v=QXQN...

My extreme opinion is that anything other than serializable isolation is a scam. Database people haven't figured out how to make it fast, so we have ended up with other half baked isolation levels.

This is the oldest transaction database from 3100 BC - recording accounts of malt and barley groats. Considering this thing survived 5000 years (holy shit!) with zero downtime and has stronger durability guarantees than most databases today. I call it rock solid durability.

Yes! Here's a recording from @devoxx.com: youtu.be/zOOFMHAjoPI?...

Nice one! Spoke about this kind of architecture a while ago too: speakerdeck.com/gunnarmorlin.... Back then, using Kafka Streams as an IVM engine and Infinispan as the serving layer for denormalized views, but Sqlite (or DuckDB) would work too, with better queryability, as you say m

btw, the new semester of the database systems course is back! The first video is already out, with an assignment where you implement a Count-Min Sketch 15445.courses.cs.cmu.edu/fall2025/

your priorities need to be in the right places if you want to pursue a database-centric lifestyle

We just launched the “DuckDB in Science” site, a curated collection of papers, lectures and podcasts about DuckDB in research: duckdb.org/science/ 🎡 If you would like to learn more about DuckDB in Science, consider joining our meetup in London this Thursday: www.meetup.com/duckdb/event...

Have you built anything serious in v lang? How was your experience? vlang.io

Published a new blog post: Replacing a Cache Service with a Database We already use databases, why can't we use them to replace caches as well? Will we ever replace caches entirely with databases? In this post, I will share some ideas, & discuss how we are moving toward this avi.im/blag/2025/db...

do you use any apps to manage bookmarks? i'm looking for something with the following features: - save links with notes and tags - archive the actual content of posts - search functionality across titles, content, etc. - ai integration would be a nice bonus

Vol:18 No:8 → Saving Private Hash Join 👥 Authors: Laurens Kuiper, Paul Gross, Peter Boncz, Hannes Mühleisen 📄 PDF: https://www.vldb.org/pvldb/vol18/p2748-kuiper.pdf

Published a new blog post: SQLite commits are not durable under default settings avi.im/blag/2025/sq...

Today is the new semester for @db.cs.cmu.edu's Intro to Database Systems! We're going harder into material than before. More challenging projects but you can use LLMs to help. We also have 10min talks each Wed from leading DB companies: 15445.courses.cs.cmu.edu/fall2025

Even though I've used Go + SQLite for many years, I still find myself wasting time on new projects. Trying to solve this problem with a transparent attempt at employing Cunningham’s Law to solicit feedback/corrections from anyone who knows better. Hopefully I'll end up with a nice little library!

✨ Big O ✨ Let me take you on a visual introduction to what big O notation is in my new blog post: samwho.dev/big-o. With big O notation you can better understand how algorithms will perform in practice, finding orders of magnitude improvements often with very simple changes to your code.

Weekend paper time! One more attempt to read this paper before I give up again

I keep getting into conversations with engineers where I talk about how desperately I want my database to support materialized views with incremental updates and too often I get a blank stare in response. so I wrote a new post about why you should care! sophiebits.com/2025/08/22/m...

A new theorem called the L2AW just dropped. It's kind of an alternative to the CAP theorem! https://law-theorem.com

Notion's new offline support is based on our rich text CRDT research x.com/ivanhzhao/st...

For anyone else trying to catch up on DBSP, my recommended flow of learning is: 1. Watch the talk: www.youtube.com/watch?v=omOH... (h/t @wslim.bsky.social) 2. Read the spec/book: mihaibudiu.github.io/work/dbsp-sp... (h/t @avi.im) 3. Read the VLDB paper List is ordered by assumed knowledge of reader

thankfully this is not an issue for databases, as most of them have checksums and handle corruptions gracefully

In the Postgres-style MVCC vs MySQL-style MVCC debates, I'd really love to see an implementation of time-separated btrees (dl.acm.org/doi/pdf/10.1...) evaluated. It's CoW-BTree style "your path down the tree prunes out versions you don't want to see", but update-in-place and copies only on splits.

Why do developers prefer to fetch entire result sets into memory when databases have had streaming for many years? Like Postgres cursors, MySQL unbuffered queries and similar in other dbs. Is it the tooling? ORMs? Awareness? Shitty APIs? For me, I didn't know that we could do that for a long time

I want: - per-query configurable timeout - a system level default if none is set - a global max_timeout, overridable only with explicit opt-in What are some databases that support all three?

HN never disappoints 🫡 news.ycombinator.com...

I went through DuckDB's WAL, and it does everything I was asking for in my blog post: 1. Per record checksum 2. Explicit error on checksum failure 3. Configurable behavior 4. Partial recovery 5. Safe truncation of the WAL only when WAL contents are checkpointed