v3 pre.6 is out - probably the penultimate prerelease before going into release candidate(s). At this point, all major architectural reworks are done, and just some APIs need some changes. Followed by a final cleanup and stabilization phase. crates.io/crates/fjall...

3.0.0 pre.5 is out and brings journal compression support. Here's 16K JSON blobs being written:

v3 pre-release 2 is out! Still not file format -or API stable, but most importantly this release marks lsm-tree being *feature complete*. crates.io/crates/fjall...

Well, it was a faulty RAM stick everybody

So I have now tried btrfs, xfs, ext4 on two diff. SSDs, updating to Ubuntu 24; and I'm still getting corruptions... at this point I don't really know what to do. This is not code specific (RocksDB is also affected), yet I don't understand how my OS and all apps just seem to continue to work fine...

95% Zipfian reads, 5% random updates on a Kingston PLP SSD with 16K JSON blobs - sled DNF! (OOM) fjall uses LZ4 compression

constellation runs on a bare metal server with 4 dedicated CPU cores, 16GiB memory, 1TiB fast NVMe SSD attached, and 500mbit unmetered network connection.

Blob GC working again (now fully automatic) - much less impact on foreground work (old GC was stop-the-world)

Fun coloured bars and lines

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!! jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing! bye

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!! jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing! bye

ngl v3 is a bit more involved than I originally planned or anticipated; I don't want breaking changes for a while, so it becomes this chase of making sure everything is really future-proof

I will never do a major release like this again

I really dread having to some day write a quick_cache replacement, tailored for lsm-tree

friendship with ext4 ended, xfs is my new best friend (whyyy is ext4 still giving me corruptions sometimes, even after updating my kernel??)

Yeah, that's a bit better

For weeks I've been regularly getting corruptions while running benchmarks... for RocksDB, too. It's infuriating because running a 15min+ benchmark is a coin flip whether it completes. Seems like it was an issue with my arguably outdated Linux kernel... there were ext4 corruption bugs a while ago.

For a framework renowned for its innovation, it is funny how little Solid has changed in the last 7 years. Yeah we've innovated on features, fine-grained concurrency/hydration, streaming, server functions, single flight mutations, but the core tech and structure are untouched.

History repeats itself

Lovely 3D plots to sweeten the day

Refactoring a 3-level deep .iter().map() is something my future self will have to deal with

Oh

I'm sorry afl, my fuzz test is just very sophisticated

1000??

Forgot to turn off the fuzz tester over night Well, at least now it's REALLY fuzz tested

My vancouver.systems talk is up! It's a deep dive on Graft—a transactional storage engine providing lazy, partial and strongly consistent edge replication. Thanks to Cameron Morgan and Kir Shatrov for hosting and @tavis.damnsimple.com for video! www.youtube.com/watch?v=eRsD...

What if your app could combine the simplicity of physical replication with the efficiency of logical replication? Meet Graft: lazy, partial, strongly consistent edge replication. #OpenSource #EdgeComputing #Replication sqlsync.dev/posts/stop-s...

2.8 fjall-rs.github.io/post/fjall-2... Probably the penultimate 2.x feature release?!

I think I just broke a couple of releases because "1.7.3" (or whatever) behaves differently in Cargo compared to e.g. NPM... I'm tilde-ing (~) everything now...

Give it a read: pscale.link/io

Looks like I maybe found a rustc performance regression

A little blog post about Google's "Webtable" key-value schema and wide-column databases, and recreating it with Fjall. fjall-rs.github.io/post/recreat...

shoutout to all the cpu cores out there compacting rocksdb sstables

lsm trees

Looks like replacing std::slice::partition_point will push point read performance a bit further for heavily cached workloads

Today we're thrilled to announce our effort to port the TypeScript compiler and language service to native code, gaining a 10x speed boost in build times and editor responsiveness! devblogs.microsoft.com/typescript/t...

2.6.0 fjall-rs.github.io/post/fjall-2...

Approximate membership filters are an important part of LSM-trees, but they are used in many places - here's a great intro to Bloom filters by @b5.bsky.social youtu.be/eCUm4U3WDpM

Hey! You've asked for more detail, you get more detail! Here's a 17 minutes deep dive into how you can use me :) https://www.youtube.com/watch?v=AkHaIVuFHK4

2.6.0 should give some nice read performance improvements, seeing something around 15-20% for uncached workloads. This is using jemalloc; the impact could be higher for other memory allocators.

Shoutout to github.com/mstange/samply for making flamegraphs actually fun

2.5 fjall-rs.github.io/post/fjall-2... Boring updates are good, right?!

fjall after a few optimizations and rocksdb. 16.3mins runtime both.

what’s attractive about rocksdb is that it’s used by everyone everywhere. pebble appeals because cockroachdb is a serious thing, bbolt backed by etcd, etc. but i gotta say, fjall has the best docs, a really fantastic blog, the most extensive examples, and a level of engagement that’s just wilddd. ✨

My Github Actions usage over the last year sits at around 1100$. Luckily it's free for public repositories - in a way Microsoft is my biggest unofficial sponsor.

Reading some Mark Callaghan blog post I noticed the level scaling in lsm-tree was kinda off... I changed it to be more similar to Rocks; it didn't make a huge difference surprisingly. But then I also improved the compaction picking algorithm and added parallel compactions (in the same level), and:

"My biggest complaint about System R is that the team never stopped to clean up SQL... All the annoying features of the language have endured to this day. SQL will be the COBOL of 2020..." - Michael Stonebraker

What's better than two M.2 SSDs? That's right, three M.2 SSDs. And Fazerdaze. I'm sick. But that won't prevent me from running benchmarks damnit.

Sometimes it feels like the Notion mobile app is playing a game: let's make it just a bit more janky every day

Just passed 100 TBW on my dev/benchmark SSD 😶