When dense retrieval loses to BM25 (and how hybrid fixes it)
Embedding-based retrieval was supposed to make everything better. On real corpora with real queries, it often doesn’t — unless you invest in the four decisions that make hybrid work.
A pattern I’ve seen at nearly every team that adopts vector search: they measure it against a weak BM25 baseline, declare victory, and ship. Then production users start complaining about queries that used to work.
The reason is simple. Dense retrieval is very good at fuzzy semantic matches and very bad at exact matches (product codes, part numbers, rare entities, negations). BM25 is the mirror image. Neither wins alone — hybrid does, but only if you make four decisions correctly.
1. Reciprocal Rank Fusion vs. weighted score fusion
RRF is the safe default. It doesn’t care about score scales and it’s hard to accidentally tune yourself into a corner. Use weighted scores only if you have judgments to tune the weights against — otherwise you’re optimising for taste.
2. Chunk size and overlap for dense retrieval
Too small: dense embeddings become noisy. Too big: relevance dilutes. 300–500 tokens with 15% overlap is a reasonable starting point for most prose corpora; product/attribute corpora need something different.
3. Reranker vs. no reranker
A cross-encoder reranker on the top-50 fused candidates is the single most consistent win we see. Latency cost is real (100–300 ms), quality gain is real, and it makes the choice of embedding model less critical.
4. Query understanding as a first-class citizen
Rewriting, expansion, and intent classification before retrieval move more relevance than any embedding upgrade. Cheap, boring, effective.
Every one of these decisions should be validated on your judgment set, not on vibes. If you don’t have judgments yet, that’s the actual first step — not picking a vector store.