We're proud to announce the first release of our latest data compression project, Zstandard in Rust! Read why we did this, the current state of the project, and where we're hoping to take it: buff.ly/JoruN7e Thanks to Chainguard, Astral, NLnet Foundation, and @sovereign.tech. #rustlang #opensource

fjall is at 1989 stars 😵 - The road to 2K is on!!

Oh yeah bluesky exists too. I just released 3.1 - which features compaction filter support, meaning you can custom logic during compaction (e.g. TTL rules, or soft deletions)! fjall-rs.github.io/post/fjall-3...

the sheer speed from initial bug report to @fjallrs.bsky.social getting a fix fully released is just wild.

A strange moment, but - I just released 3.0.0. fjall-rs.github.io/post/fjall-3/ What has started as a rewrite of the block format has become almost a full rewrite, interrupted by a bachelor's thesis, faulty hardware and a couple of weeks of sickness - but I think the end product is pretty good.

Love it when SQLite just stops working but allocates hundreds of GBs of disk space.

Check out my talk on physical replication and Graft that I delivered at the recent @syncconf.bsky.social in SF. youtu.be/QoKzDyH2MEA?...

Well... looks like my Kingston disk just died...?!

This year's winter album... goes to Maria Somerville youtu.be/k9qEKyQ7jc4

RC 1 is out! I forgot to post about RC 0, but here we are. At this point API and disk format are unlikely to further change - now it's all about stabilization for the final release... crates.io/crates/fjall...

v3 pre.6 is out - probably the penultimate prerelease before going into release candidate(s). At this point, all major architectural reworks are done, and just some APIs need some changes. Followed by a final cleanup and stabilization phase. crates.io/crates/fjall...

3.0.0 pre.5 is out and brings journal compression support. Here's 16K JSON blobs being written:

v3 pre-release 2 is out! Still not file format -or API stable, but most importantly this release marks lsm-tree being *feature complete*. crates.io/crates/fjall...

Well, it was a faulty RAM stick everybody

So I have now tried btrfs, xfs, ext4 on two diff. SSDs, updating to Ubuntu 24; and I'm still getting corruptions... at this point I don't really know what to do. This is not code specific (RocksDB is also affected), yet I don't understand how my OS and all apps just seem to continue to work fine...

95% Zipfian reads, 5% random updates on a Kingston PLP SSD with 16K JSON blobs - sled DNF! (OOM) fjall uses LZ4 compression

constellation runs on a bare metal server with 4 dedicated CPU cores, 16GiB memory, 1TiB fast NVMe SSD attached, and 500mbit unmetered network connection.

Blob GC working again (now fully automatic) - much less impact on foreground work (old GC was stop-the-world)

Fun coloured bars and lines

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!! jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing! bye

If you sponsor NOW (github.com/sponsors/fja...), you can get an exclusive preview of version 3!! jk I pushed the first pre-release to crates.io - it's unstable though! Breakages guaranteed, and some APIs are missing! bye

ngl v3 is a bit more involved than I originally planned or anticipated; I don't want breaking changes for a while, so it becomes this chase of making sure everything is really future-proof

I will never do a major release like this again

I really dread having to some day write a quick_cache replacement, tailored for lsm-tree

friendship with ext4 ended, xfs is my new best friend (whyyy is ext4 still giving me corruptions sometimes, even after updating my kernel??)

Yeah, that's a bit better

For weeks I've been regularly getting corruptions while running benchmarks... for RocksDB, too. It's infuriating because running a 15min+ benchmark is a coin flip whether it completes. Seems like it was an issue with my arguably outdated Linux kernel... there were ext4 corruption bugs a while ago.

For a framework renowned for its innovation, it is funny how little Solid has changed in the last 7 years. Yeah we've innovated on features, fine-grained concurrency/hydration, streaming, server functions, single flight mutations, but the core tech and structure are untouched.

History repeats itself

Lovely 3D plots to sweeten the day

Refactoring a 3-level deep .iter().map() is something my future self will have to deal with

Oh

I'm sorry afl, my fuzz test is just very sophisticated

1000??

Forgot to turn off the fuzz tester over night Well, at least now it's REALLY fuzz tested

My vancouver.systems talk is up! It's a deep dive on Graft—a transactional storage engine providing lazy, partial and strongly consistent edge replication. Thanks to Cameron Morgan and Kir Shatrov for hosting and @tavis.damnsimple.com for video! www.youtube.com/watch?v=eRsD...

What if your app could combine the simplicity of physical replication with the efficiency of logical replication? Meet Graft: lazy, partial, strongly consistent edge replication. #OpenSource #EdgeComputing #Replication sqlsync.dev/posts/stop-s...

2.8 fjall-rs.github.io/post/fjall-2... Probably the penultimate 2.x feature release?!

I think I just broke a couple of releases because "1.7.3" (or whatever) behaves differently in Cargo compared to e.g. NPM... I'm tilde-ing (~) everything now...

Give it a read: pscale.link/io

Looks like I maybe found a rustc performance regression

A little blog post about Google's "Webtable" key-value schema and wide-column databases, and recreating it with Fjall. fjall-rs.github.io/post/recreat...

shoutout to all the cpu cores out there compacting rocksdb sstables

lsm trees

Looks like replacing std::slice::partition_point will push point read performance a bit further for heavily cached workloads

Today we're thrilled to announce our effort to port the TypeScript compiler and language service to native code, gaining a 10x speed boost in build times and editor responsiveness! devblogs.microsoft.com/typescript/t...

2.6.0 fjall-rs.github.io/post/fjall-2...

Approximate membership filters are an important part of LSM-trees, but they are used in many places - here's a great intro to Bloom filters by @b5.bsky.social youtu.be/eCUm4U3WDpM

Hey! You've asked for more detail, you get more detail! Here's a 17 minutes deep dive into how you can use me :) https://www.youtube.com/watch?v=AkHaIVuFHK4

2.6.0 should give some nice read performance improvements, seeing something around 15-20% for uncached workloads. This is using jemalloc; the impact could be higher for other memory allocators.