Two decades in search and AI — what I’ve seen

I started working on search in 2007, when “information retrieval” meant Solr, tokenizers, and stemmers, and half of every project was tuning synonyms.txt.

Eighteen years later the interface has been reinvented three times — lexical, then learned-to-rank, then vector — and every time the same pattern separates teams who ship from teams who ship demos:

1. They have a query set. Real user queries, sampled from logs, stratified by intent. Not synthetic. Not “representative examples someone typed up.”
2. They have a judgment list. For every query, someone who understands the domain has marked the ideal answer and the must-cite documents.
3. They have a number. nDCG, MRR, faithfulness, refusal correctness — pick your metric, but they can answer “is search better this quarter than last quarter?” with a chart.

The vector-search era did not change this

If anything, it raised the stakes. Vector retrieval fails silently in weird ways. Cosine similarity does not know what your users actually meant. Pure dense retrieval loses to well-tuned BM25 more often than the hype cycle would like to admit.

What won in 2007 still wins in 2025: measure, iterate, ship, measure again.

The tools got better. The discipline didn’t change.