It's interesting how the Elasticsearch and Datadog (www.datadoghq.com/blog/enginee...) approaches to wildcard search differ. Both use n-gram indexes, but with different strategies to contain storage amplification. Datadog hashes 4-grams while ES aggressively normalizes 3-grams.

Ge Song merged a good ~15% speedup for BM25F queries in Lucene benchmarks.mikemccandless.com/CombinedOrHi... (last data point) github.com/apache/lucen...

New blog: vectorized evaluation of disjunctive queries jpountz.github.io/2025/10/11/v... It explains how Lucene manages to be fast at evaluating top hits by BM25 score, even with hard queries that have only stop words or tens of terms.

BM25F is an adjustment to BM25 that accounts for multiple fields, beating out naive summing of BM25 scores softwaredoug.com/blog/2025/09...

Lucene 10.3 is out with 40% faster lexical search, 15% faster dense vector search and 30% faster terms dictionary lookups. lucene.apache.org/core/corenew...

Lucene just bumped the block size of its postings lists from 128 to 256. This gave very good speedups (up to 45%) to most queries, and up to 10-15% slowdowns to filtered term queries. benchmarks.mikemccandless.com/2025.09.10.1...

#Lucene just switched from a binary heap to a ternary heap to collect top hits by score. This helps a small bit when computing top-100 hits (~2% on the fastest queries) but up to 15% when computing top-1000 hits - thanks to better cache efficiency github.com/apache/lucen...

I just ran the Tantivy benchmark (tantivy-search.github.io/bench/) on Lucene 10.2 vs a Lucene 10.3 snapshot build. Lucene 10.2 already performed very well, but Lucene 10.3 is on another level. Very exciting.

I just blogged about how Lucene improved query evaluation efficiency by ~40% through vectorization: jpountz.github.io/2025/08/28/c...

Why you probably should configure an index sort on your Lucene indexes: jpountz.github.io/2025/07/26/w...

I spent some time looking at the Vespa source code to see how it compares with Lucene jpountz.github.io/2025/07/25/m...

This small change yielded a ~5% speedup on several queries of Lucene's nightly benchmarks (see last data point at benchmarks.mikemccandless.com/OrStopWords....). Can you guess why?

Last month, Lucene changed query evaluation to work in a more term-at-a-time fashion within small-ish windows of doc IDs. This yielded a good speedup on its own (annotation IL benchmarks.mikemccandless.com/OrHighMed.html).

Lucene is getting an increasing number of high-quality contributions from ByteDance employees, especially around performance. Good to see that this project keeps attracting contributors from all around the world.

Another common point I did not expect: Vespa's strict vs. unstrict iterators is quite similar to Lucene's two-phase iteration. And both projects use this feature to effectively combine dynamic pruning with filtering (a hard and underappreciated problem IMO).

Someone asked me for my opinion on the Vespa vs. Elasticsearch performance comparison today at Berlin Buzzwords, so I gave it a try: jpountz.github.io/2025/06/17/a...

Andrei Dan kindly captured pictures of Luca and I telling the story of how the Lucene 10 release went

Via @rmuir.org : Linux 6.15 introduced a big speedup for Lucene on AMD processors benchmarks.mikemccandless.com/FilteredOrHi... (last data point, not annotated yet) thanks to faster TLB invalidation www.phoronix.com/review/amd-i...

Uwe now explains how Lucene takes advantage of the Panama foreign memory and vector support in spite of the fact that these features are still preview/incubating in the JDK

Uwe Schindler gives a short history of Apache Lucene at #bbuzzz

Lucene is getting faster at deep search by switching to a more efficient heap implementation to collect top hits. github.com/apache/lucen...

A nice optimization landed on the hash table that Lucene uses to build inverted indexes: github.com/apache/lucen.... Some previously unused bits are now used to cache hash codes, effectively making collisions cheaper to resolve.

There has been a big regression in Lucene's nightly benchmarks recently after a kernel upgrade. Mike and @rmuir.org found that it was caused by a change in the Linux scheduler configuration. github.com/apache/lucen...

I wanted to share what I learned from Tantivy's "Search Benchmark, the Game", so I set up GitHub pages and wrote two blogs, on general observations on the benchmark jpountz.github.io/2025/05/12/a... and how it helped drive performance improvements in Lucene jpountz.github.io/2025/04/12/w...

Yelp's nrtSearch was just upgraded to Lucene 10. Also switched from persistent storage to object storage as a source of truth, and plans on doing NRT replication via object storage instead of over the network. Very similar to Elasticsearch Serverless. engineeringblog.yelp.com/2025/05/nrts...

The search library benchmark from the Tantivy folks was just updated with Lucene 10.2 tantivy-search.github.io/bench. Lucene now performs much better at the COUNT collection type, a bit better at TOP_K. Still somewhat slow at TOP_100_COUNT and phrase queries across all collection types.

Lucene's histogram collector is becoming more sophisticated, it can now take advantage of points indexes when the query fully matches a segment, which can give a multiple fold performance boost. github.com/apache/lucen...

I feel bad when I see users needing to shard their indexes only to not hit the 2B doc count limit. 2B was a lot when Lucene was created, not anymore. We should fix it. github.com/apache/lucen...

Julie's argument for principled approaches in her BM25F blog reminded me of this paper www.microsoft.com/en-us/resear... which discusses folding features like page rank into the BM25 score by making their contribution look like the contribution of a term.

Finally took the time to read this great blog on BM25F by Julie Tibshirani sourcegraph.com/blog/keeping.... Great to see more usage of BM25F, lots of applications are still combining scores across fields via sum/max when BM25F would be a more robust choice.

Elasticsearch 9, now on Lucene 10.

In case you missed it: since last year, Lucene's nightly benchmarks include queries on whole paragraphs. github.com/mikemccand/l... benchmarks.mikemccandless.com/OrMany.html This is not typical for user-facing search, but many users do this, especially with the rise of RAG.

I'm glad to share that Luca Cavanna and I will be speaking about the #Lucene 10.0 release at Berlin Buzzwords in June.

It's time to redo benchmarks! #Lucene 10.2 was just released, with - huge speedups to non-scoring boolean queries, range queries and filtered vector search, - better merging defaults for faster search, - much faster merging of vectors And more... lucene.apache.org/core/corenew...

Something that I wish was better known: if you do not configure an index sort on your #Lucene indexes, you are missing search-time efficiency benefits that are almost certainly worth the (low) index-time overhead.

Guo Feng contributed a 2.5x (!) speedup to #Lucene's numeric range queries by using vectorization. HZ sped up query evaluation, ID sped up decoding data from the index. Lots of great performance improvements coming in Lucene 10.2.

Two good speedups on queries sorted by field on #Lucene's nightly benchmarks last night. This is due to a recent optimization to conjunctive queries being generalized to intersecting a query and a collector. github.com/apache/lucen...

Filtered vector search is crazy important. So we made HNSW filtered search in Apache Lucene better. At similar recall, it can be 3-5x faster!

You may not have heard of the Terrier IR platform terrier.org but both Lucene's Similarity framework lucene.apache.org/core/10_1_0/... and Elasticsearch's Retriever framework www.elastic.co/guide/en/ela... are heavily inspired by Terrier.

2.5y ago, Tencent published how they optimize "histogram queries" with Lucene www.vldb.org/pvldb/vol15/... (section 4.5.3) I'm adding an implementation of this idea to Lucene, that takes advantage of sparse indexing introduced in Lucene 10.0 github.com/apache/lucen....

The change has been merged and nightly benchmarks just caught up: benchmarks.mikemccandless.com/2025.01.14.1.... I don't remember many improvements of this magnitude to non-trivial workloads in Lucene's history.

Common wisdom is that indexing is slow (it is!), so when ingestion is slow, it's natural to blame indexing. But there are often overlooked costs, like JSON parsing, date parsing blunders.io/posts/es-ben... or number parsing lemire.me/blog/2020/03...

Quizz: I'm working on a change that produces the following speedup when running the Lucene benchmark. Can you guess what the change does? I'll open a PR shortly.

Mike Sokolov is exploring applying recursive graph bisection to HNSW graphs in Lucene. This gives significantly faster search (as we hoped) but also faster indexing (unexpected!) github.com/apache/lucen...

Something a little different from my typical blogs. This line of code in Apache Lucene took me 3 days to write. For fixing bugs, it's about the journey, not necessarily the destination. www.elastic.co/search-labs/... (the cover art was provided by one of my kids :))

Lucene has been evaluating disjunctive queries by loading (windows of) postings into a bit set and or-ing these bit sets for 20+ years. It started using the same approach for conjunctive queries a few days ago. benchmarks.mikemccandless.com/CountAndHigh... (annotation HS)

Lucene just got a big speedup for exhaustive evaluation of disjunctive queries when not computing scores: benchmarks.mikemccandless.com/CountOrHighH... (annotation HR). Same algorithm, but the code was refactored to help the JVM compiler auto-vectorize a hot loop.

Filtered disjunctions just got a good speedup on Lucene's nightly benchmarks, especially when the disjunction has many terms benchmarks.mikemccandless.com/CountFiltere... (annotation HQ).

DisjunctionMaxQuery just had a good speedup: benchmarks.mikemccandless.com/DismaxOrHigh... Even though Lucene now has support for BM25F, DisjunctionMaxQuery is still a popular way to query across several fields by taking the maximum score, see e.g. Elasticsearch's best_fields option.

Phrase queries got a speedup lately by removing as much bookkeeping about positions as possible when advancing the doc ID. This helps because positions are only read when all phrase terms match. The speedup is even bigger on filtered phase queries. benchmarks.mikemccandless.com/FilteredPhra...