Some optimization on VBQ with the latest binseq update, especially in lossless mode. Some ways to trim the fat: 1. Reuse zstd decoders for each thread. I was creating a decoder for each vbq block which incurred redundant allocations 2. Zero-copy parsing of blocks, referencing similar to paraseq

Excited to announce a new bqtools tutorial on sandbox.bio by @noamteyssier.bsky.social! Learn about the BINSEQ file format, and how it can replace FASTQ files for better data compression and faster parallel processing: sandbox.bio/tutorials/bq...

I work with large collections of AnnDatas for single-cell work and got tired of opening notebooks for simple operations. Built a CLI tool to handle some common stuff directly from the terminal. Quick ops: downsample, concat, pseudobulk, QC, metadata export, etc. github.com/noamteyssier...

BINSEQ is a high-performance format for sequencing data and bqtools is a CLI tool that lets you create and manipulate these files in the style of samtools. Excited to release a tutorial with @robert.bio showcasing how to use it to encode, decode, and grep sequences in the browser on sandbox.bio!

New bqtools release with some nice new features! 1. Support for fuzzy matching using sassy 2. Multi-Pattern counting (like `grep -c` but the count is for each individual pattern provided) 3. Pattern files (providing large lists of patterns as either regex or literals) github.com/ArcInstitute...

I've updated the BINSEQ manuscript to stay up to date with changes since I originally put it out at the beginning of the year Some notable changes: 1. Support for ambiguous bases with 4bit encoding 2. Support for sequence headers 3. Improved API www.biorxiv.org/content/10.1...

Was just about to submit a revision for a paper and realized that I wouldn't be able to submit my source for the text because it was written with typst. Such a bummer - moving this over to tex now but damn what a waste of time! typst is just so much nicer to work with.

Around 10% of your Nanopore reads (SQK-RBK114) are incorrectly trimmed. Here is why, and how our new tool Barbell solves it: www.biorxiv.org/content/10.1... Want to get started? github.com/rickbeeloo/b...

Thank you Alzforum for featuring our new preprint identifying regulators of disease states of #microglia. Project led by Amanda McQuade, computation by Reet Mishra, collaboration with the Nunez and De Jager labs. Alzforum www.alzforum.org/news/researc... Preprint www.biorxiv.org/content/10.1...

Had an old tool called `hist` to run `sort | uniq -c` years ago but thought up a high-perf impl for it today. Tried it out and found a 25x throughput over the coreutils version. Big takeaway - arena allocators, hashmaps, and serde work super well together. github.com/noamteyssier...

Paraseq 0.4 is out now! With double the throughput for processing paired-end input :) github.com/noamteyssier...

Added a feature to bqtools yesterday for colored grep output. Also supports colored FASTX output as well. Already useful this morning as I troubleshoot some sequencing outputs!

Excited that the paper presenting our mouse brain in vivo CRISPR screening platform is out today in @natneuro.nature.com! Great team effort, led by Biswa Ramani and @ivlrose.bsky.social in the Kampmann lab. www.nature.com/articles/s41...

Preprint alert! We present K2Rmini, an ultra-fast, grep-like tool that extracts sequences of interest from FASTA/FASTQ files based on their k-mer content. www.biorxiv.org/content/10.1... A thread

Writing in rust again after a long stretch of python is such a breath of fresh air.

Introducing Arc Institute’s first virtual cell model: STATE

Pretty cool little utility and blog post - fun to see the business/pleasure index for rust crates boydkane.com/projects/cra...

Preprint on "Improving spliced alignment by modeling splice sites with deep learning". It describes minisplice for modeling splice signals. Minimap2 and miniprot now optionally use the predicted scores to improve spliced alignment. arxiv.org/abs/2506.12986

New preprint! Deacon is a versatile tool for filtering FASTA/FASTQ files and streams at hundreds of megabases per second using minimizers, built with rapid metagenomic host depletion in mind, but equally useful for search. github.com/bede/deacon

ish is a grep-like CLI tool that uses optimal alignment instead of exact matching. It’s record-type aware, supporting line, FASTA, and FASTQ records. Built in Mojo as a proof of concept for bioinformatics. 🧵1/5

Slides from my talk (with @kamilsjaron.bsky.social) on an history of k-mers in bioinformatics: rayan.chikhi.name/pdf/2025-kme...

📜 Excited to share insights from our recent paper: "Kaminari: a resource-frugal index for approximate colored k-mer queries". The study aims to efficiently identify documents containing a query string, focusing on DNA strings. www.biorxiv.org/content/10.1... 🧬 🖥️ 1/8

Our Proseg paper is now out in Nature Methods! www.nature.com/articles/s41... We borrowed a sampling procedure from the cell simulation literature to infer cell boundaries that best explains the spatial distribution of transcripts.

📄 The scanners are humming, the film is flowing. The microfiche livestream is up—digitizing government docs in real time for Democracy’s Library. Perfect second-screen vibes: Preservation in progress. 🕢 Live M-F, 7:30am–3:30pm PT (except U.S. holidays) ➡️ www.youtube.com/live/aPg2V5R...

So yeah, this is why I keep going on about: do we have to sanitize user input or not? File formats where bad inputs are simply not representable are good, because it saves us from this 100x slowdown.

Just merged in an awesome new feature for xsra to support named pipes with @robp.bsky.social. This lets you skip an intermediary write step and go straight from SRA to downstream tools. It works with accessions that are both on- or off-disk.

No bottlenecks with binseq... github.com/ArcInstitute...

Extracting @NCBI SRA files with fasterq-dump can require 17x the size of the accession while decompressing. Our new tool xsra extracts sequences at 5x throughput with significantly less disk usage, built-in compression, and optional BINSEQ outputs github.com/arcInstitute...

I think this captures a lot of my feelings on AI coding. I appreciate that it increases productivity, but at the expense of joy (and arguably quality). I've gone through phases with it, but ultimately I'm back to writing everything myself. It's just more fun. terriblesoftware.org/2025/04/23/t...

I wanted to share some thoughts on BINSEQ, following up on our preprint from last week. I discuss how it addresses core limitations of FASTQ. I demo how a simple task like sequence grep can get 20-100x throughput with BINSEQ. noamteyssier.github.io/2025-04-20/

I'm excited to release what I've been cooking up the past few months at @arcinstitute.org BINSEQ is a family of binary file formats for sequencing data built with paired records and parallel processing in mind with big performance gains (2x-40x) over gzip-fastq with similar storage