# Stop Relying on AI. Learn The Algorithm First. A search feature is the easiest thing to screw up because it’s _so_ plug-and-play these days, you barely even think about it. You’ll probably pick a BM25 library (used by Elasticsearch, Lucene, and others) and it will give you a top-10 of plausible-looking results in some order — but whether that’s actually correct depends on the little details that were decided by library authors, in defaults you probably never read, and had no way of noticing when they went wrong. **Naturally, that goes double if your code was one-shot by your AI agent — which is what most do these days.** I tested this on two corpora: 9,740 Google News results over six years, quarterly, that I collected myself with Bright Data’s [SERP API](https://get.brightdata.com/bd-serp-api?utm_content=building-lexical-search-learn-the-algorithm), and 2,284 documents from [ClusTREC-COVID](https://huggingface.co/datasets/Uri-ka/ClusTREC-Covid) — a free HuggingFace dataset of COVID-19 research papers, each labeled with a topic/category. I first saw this while searching ClusTREC-COVID for `covid mrna` with the [rank_bm25](https://github.com/dorianbrown/rank_bm25) library. Two of my top-10 results were a chest X-ray image collection and a study on vitamin D and mortality. **Neither contained** `mrna` **even once.** A paper on the SARS coronavirus ORF8 protein, which _did_ mention `mrna` and was exactly what the query was for, should have been at rank 9. **Instead, it was at rank 412.** Same issue with the Google News corpus. Searching for `ai labour`, two spots in the top-10 were _3 Ways Predictive AI Delivers More Value Than Generative AI_ and _AI Ethics And AI Law Clarifying What In Fact Is Trustworthy AI_. **Neither about, nor containing the word** `labour` **at all.** There were no errors, because this is not a programmatic ‘error’. If I didn’t know what a “good” result even looked like, I never would have dug deeper. So what was my search feature even doing? What _should_ it have done instead? I love writing blog posts like these because they’re such great teaching moments. **Let’s dive right in.** ## What is BM25? How Does It Work? The BM25 algorithm is used to answer a simple question: > Given this query, how much evidence is there that this document is relevant to the user’s interests? It gets there using three measurements: - How often the query term appears in it (with diminishing returns for repetition.) - The document’s length (so long-ish documents don’t win simply because they contain more words.) - **And finally, how rare the query term is across the collection.** This is IDF, or inverse document frequency. **It’s based on the assumption that the presence of a rare query term is useful evidence.** If only 1% of your documents contain a word, finding it in a document _would_ be significant. On the flipside, if ~70% of your corpus contains a certain term, then seeing it in a document doesn’t tell you much — most documents in your collection contain it anyway, it’s not _special_. The standard BM25 IDF formula (from [Robertson-Spärck Jones](https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.4630270302)) turns that into a singular value you can use: > idf(term) = log((N — n + 0.5) / (n + 0.5)) > > Where `N` is the number of documents in the collection and `n` is the number of documents containing the term. So if a term appears in more than half the collection (i.e. `n` greater than `N/2`) the number inside the logarithm falls below 1, and so the IDF value becomes negative. This single number is intuitive, even if you’re not a math major. **BM25 is telling you that this term is so common that its presence is actually LESS informative than average.** And that’s where any search implementation has to make a choice. The BM25 formula CAN give you a negative IDF, but it doesn’t prescribe what a search library should do with it. There are three obvious options: - Leave the IDF negative. Matching the common term then lowers the score. - Clip it to zero. The common term neither helps nor hurts. - Replace it with a positive value. Matching the common term boosts the score. None of these is universally “correct.” It boils down to whether the choice matches what you’re trying to retrieve. The `rank_bm25` library I use chooses the third behavior by default, it's a perfectly legitimate implementation strategy. What surprised me was how much it changed the results on a corpus where one topic term appears in most documents. Before going any further, I should talk a little bit about the nature of the two datasets **because they both have something in common that turns out to be quite important later on.** ## How do you collect publicly available data efficiently? I used a SERP API for the Google News results, and downloaded the public dataset from HuggingFace. Here’s how to do both. ### Dataset 1: Custom-built Google News Results This is a JSON of 9,740 Google News results, collected quarterly over six years against eleven AI-related search queries. [I actually used this same corpus in an earlier project to answer a different question.](https://plainenglish.io/web-scraping/i-analyzed-5-years-of-ai-news-here-s-how-our-tone-shifted-from-optimism-to-doomerism) These are the search terms I used for the SERP API call. ``` artificial intelligence AI technology machine learning AGI AI safety AI ethics generative AI AI regulation AI slop AI jobs AI layoffs ``` Bright Data’s SERP API returns these fields visible on the search page if you send Google a News search (`tbm=nws`) and ask for parsed JSON (`brd_json=1`): ```json { "headline": "...", "date": "Sep 24, 2025", "source": "prnewswire.com", "url": "https://...", "query": "\"AI\" \"technology\"", "quarter": "Q3 2025", "description": "" } ``` I chose to include the `query` field for each row to record provenance. I'll use that field and also the `headline`/`description` together for something more ordinary: _ranking_. Fetching the data was just a Python script, something like this: ```python import os from urllib.parse import urlencode import requests API_KEY = os.environ["BRIGHT_DATA_API_KEY"] ZONE = os.environ["BRIGHT_DATA_SERP_ZONE"] def news_url(query: str, start: int) -> str: return "https://www.google.com/search?" + urlencode({ "q": query, "tbm": "nws", "hl": "en", "gl": "us", "start": start, "brd_json": 1, }) def fetch_page(query: str, start: int) -> dict: resp = requests.post( "https://api.brightdata.com/request", headers={ "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json", }, json={ "zone": ZONE, "url": news_url(query, start), "format": "raw", }, timeout=60, ) resp.raise_for_status() return resp.json() def collect_slice(query: str, quarter: str) -> list[dict]: rows = [] for start in (0, 10, 20, 30, 40): page = fetch_page(query, start) for item in page.get("news") or []: rows.append({ "headline": item.get("title"), "date": item.get("date"), "source": item.get("source"), "url": item.get("link"), "query": query, "quarter": quarter, "description": item.get("description"), }) return rows ``` This gave me 9,740 distinct URLs. If you want to do the same, you'll need to [sign up here](https://get.brightdata.com/bd7914?utm_content=building-lexical-search-learn-the-algorithm) and get your credentials from the dashboard. I really like this service primarily because new pay-as-you-go accounts don’t need a credit card to get started, and have a hard stop when you run out of your 5000 free credits. You can see current [pricing here](https://get.brightdata.com/scraping-browser-acf6883?utm_content=building-lexical-search-learn-the-algorithm). ### Dataset 2: ClusTREC-COVID from HuggingFace ClusTREC-COVID is a labelled slice of TREC-COVID, which is itself drawn from CORD-19. It's 2,284 rows (JSONL) of title plus abstract, each tagged with one of fifty topics. Each row is a record like this: {"sentences": "Diversity of Coronaviruses in Bats: Insights Into Origin of SARS Coronavirus", "labels": "coronavirus origin", "doc_id": "6foz003n"} You can download the file yourself [here](https://huggingface.co/datasets/Uri-ka/ClusTREC-Covid/resolve/main/title_and_abstract/clustrec_covid_abstract_and_title.jsonl) (it's about 3MB) or if you'd rather not open a browser, use this script: ```python import requests URL = ( "https://huggingface.co/datasets/Uri-ka/ClusTREC-Covid/resolve/main/" "title_and_abstract/clustrec_covid_abstract_and_title.jsonl" ) resp = requests.get(URL, timeout=60) resp.raise_for_status() with open("clustrec_covid_abstract_and_title.jsonl", "wb") as f: f.write(resp.content) ``` ## What Do Our Datasets Have in Common? Both were built by searching for a specific topic, which means both have the same unusual property: **a term that represents the subject of the dataset appears in most of its documents, and therefore will produce a negative raw IDF.** | | AI news | ClusTREC-COVID | | -------------------- | --------------------------- | ------------------------------ | | rows | 9,740 headlines + snippets | 2,284 titles + abstracts | | average length | 33.96 tokens | 186.93 tokens | | saturated term | `ai`, in 7,090 rows (72.8%) | `covid`, in 1,764 rows (77.2%) | | raw IDF of that term | `-0.9840` | `-1.2208` | Now let’s see what `rank_bm25` does about it. ## The Problem Caused By The `rank_bm25` Library Defaults If you search for `covid mrna` in the ClusTREC-COVID dataset using BM25 -- `covid` is a saturated term, occurring in 1,764 of 2,284 documents. By the formula given earlier, raw IDF for this search will yield `-1.2208`. [](https://medium.com/plans?source=promotion_paragraph---post_body_banner_read_deeper--abbc6676c646-----------------------------------------) A person typing in that query wants to see papers about COVID-19 mRNA. **What the default ranking with this library shows them(slots 9 and 10) are documents that never mention** `**mrna**` **at all**: | Document | `covid` Occurrences | `mrna` Occurrences | Rank | |-------------------------------------------|--------------------:|-------------------:|-----:| | COVID-19-CT-CXR chest X-ray/CT collection | 18 | 0 | 9 | | Vitamin D and COVID-19 in UK Biobank | 12 | 0 | 10 | Turns out, the `rank_bm25` library does the opposite. **Its default replaces every negative IDF with a positive value, with no mention in the project README, and no easy way to override this behavior and restore the original IDF value.** Repeating a saturated term then becomes a _positive_ outcome, and that is what brought those two otherwise unrelated documents to the first page. **Worse — something had to be displaced to make room.** In this case it was **_Severe Acute Respiratory Syndrome (SARS) Coronavirus ORF8 Protein…_**, which _did_ contain `mrna`, _was_ a genuine, relevant document, and would be at rank 9 had it not been for the default behavior of this library. **Under the default, it was pushed down to rank 412**. Press enter or click to view image in full size ![Raw IDF against the share of documents that contain a term](https://miro.medium.com/v2/resize:fit:800/1*tx1s06e-jmRsGJS-y8fUcQ.png) _Fig 1: The true IDF value for a term (red) turns negative once it’s in more than half the documents, like `ai` and `covid` here. But `rank_bm25` won’t store that by default. Instead, it swaps in a positive number (blue) that turns matching a common term into a reward instead of a penalty._ Here’s what’s happening behind the scenes in `rank_bm25` to produce that ranking (accurate as of v0.2.2): ```python ef _calc_idf(self, nd): idf_sum = 0 negative_idfs = [] for word, freq in nd.items(): idf = math.log(self.corpus_size - freq + 0.5) \ - math.log(freq + 0.5) self.idf[word] = idf idf_sum += idf if idf < 0: negative_idfs.append(word) self.average_idf = idf_sum / len(self.idf) eps = self.epsilon * self.average_idf for word in negative_idfs: self.idf[word] = eps ``` This library first calculates the ordinary IDF. If it is negative, it remembers the term. It then takes the average IDF over the vocabulary and replaces the negative value with: ``` epsilon * average_idf ``` The default `epsilon` is `0.25` but you can pass in a different one: ```python BM25Okapi(tokenized, epsilon=0) ``` But that's the lowest you can go. Setting `epsilon=0` only clips negatives to zero, it would not retain the actual raw IDF value in the negatives, which you want here. For `covid` in this corpus, the values become: ``` raw IDF: -1.2208 stored IDF: +1.5954 ``` So this behavior has flipped what the presence of `covid` does to the ranking. Presence of a term which should, mathematically speaking, be less informative than average, is now scored as if it were useful evidence -- and term frequency will obviously amplify it (remember the formula). This happens when the index is first built, not when you run a query. You will not see it in the results list. You will only see it if you inspect the stored IDF yourself like this: ```python import math N = bm25.corpus_size n = sum(1 for d in bm25.doc_freqs if "covid" in d) raw = math.log(N - n + 0.5) - math.log(n + 0.5) stored = bm25.idf["covid"] print(raw) # -1.2208 print(stored) # +1.5954 ``` ## This Will Also Happen For Sparse Searches. This also happens when the second query term is scarce enough that there aren’t enough genuine matches to fill the first page. This will, of course, depend on how many results you’re asking for in your “Top X”. I was going for the Top 10. Let’s search for `covid ivermectin` on ClusTREC-COVID. Only four documents contain the token `ivermectin`. The default puts all four first -- then fills ranks 5–10 with papers that never mention it: | Document | `covid` | `ivermectin` | Default rank | Rank (negative IDF → 0) | Rank (negative IDF allowed) | |-------------------------------------------|--------:|-------------:|-------------:|------------------------:|-----------------------------:| | COVID-19-CT-CXR chest X-ray/CT collection | 18 | 0 | 5 | 1,548 | 2,284 | | Vitamin D and COVID-19 in UK Biobank | 12 | 0 | 6 | 1,898 | 2,283 | The Google News corpus shows the same pattern. Searching for `ai labour` -- the token `labour` appears in only eight documents. Ranks 1–8 contain those, then ranks 9–10 do not: | Document | `ai` | `labour` | Default rank | Rank (negative IDF → 0) | Rank (negative IDF allowed) | |----------------------------------------------------------------|-----:|---------:|-------------:|------------------------:|-----------------------------:| | 3 Ways Predictive AI Delivers More Value Than Generative AI | 6 | 0 | 9 | 6,477 | 9,740 | | AI Ethics And AI Law Clarifying What In Fact Is Trustworthy AI | 7 | 0 | 10 | 3,773 | 9,738 | A person searching `ai labour` did not ask for an ethics explainer or a generative-AI comparison. Clipping `ai` to zero sends those rows into the thousands. Leaving the IDF negative actually sends them to the bottom. **Only the default behaviour would put them on page one.** Press enter or click to view image in full size ![Rank of documents that miss the rare query term, under three IDF policies](https://miro.medium.com/v2/resize:fit:800/1*FOVawqqMDG-DX4_va0CQxA.png) _Fig 2: Two documents that don’t contain the rare query term at all. The default setting still keeps them on page one (top 10). Either fix pushes them far down the list, where they belong_ That’s why the `rank_bm25` default isn't just a small numerical adjustment. **It can completely change what gets shown to the user when the query contains one very common term and one very rare one.** ## But Is Negative IDF Always the Right Call? No. There will always be instances where that same negative-IDF behavior is the exact opposite of what you want. Take the same AI news corpus again, this time searching for `AI layoffs`. Under negative IDF, the top results will be stories about layoffs that never mention AI at all*: - _Ranked: America’s 20 Biggest Tech Layoffs This Decade_ - _Layoffs: Surviving the new-age workplace!_ - _Apple layoffs hit more than 600 employees on its car projects_ > It’s possible that some of these stories will mention AI in passing in their article bodies, after all, every one of those rows was stored because Google News returned it for the search `AI layoffs`. But that's not what we're measuring -- and not what a reasonable user would want returned with that search. None of them contain the token `ai`. **For a user-facing search, I would argue that is a failed ranking.** But suppose I’m auditing an unknown or untrusted “AI news” collection and trying to find records that got swept in even though they don’t actually mention AI. **Then this is exactly the list I want.** The default floor would bury those documents at ranks 197, 231, and 262. Negative IDF puts them on page one. These are all different problems, different use-cases. Negative IDF is useful for the second one precisely because the absence of a common term is now informative. There can be no simple rule-of-thumb, you have to understand your data _and_ the algorithm, measure both approaches, and make the call yourself. ## How To Override The BM25 Library Default Unfortunately, `rank_bm25` doesn't expose a setting that keeps the raw negative IDF as an `epsilon` setting. If you actually want that behavior, the hacky-but-effective approach I use is to manually restore the raw values after construction like this: ```python import math from rank_bm25 import BM25Okapi bm25 = BM25Okapi(tokenized) # default # and then, reconstruct one by one for term in bm25.idf: n = sum(1 for doc in bm25.doc_freqs if term in doc) bm25.idf[term] = math.log( bm25.corpus_size - n + 0.5 ) - math.log( n + 0.5 ) ``` This replaces the values in place, so using `get_scores()` later will use those new reconstructed values because it reads directly from `bm25.idf`. You can also inspect what happened to a particular term like this: ```python term = "covid" n = sum(1 for doc in bm25.doc_freqs if term in doc) N = bm25.corpus_size raw = math.log(N - n + 0.5) - math.log(n + 0.5) stored = bm25.idf[term] print("raw:", raw) print("stored:", stored) ``` ## What Did I Learn? **Algorithmic choices are contextual. A library can’t know what your users mean by a “good” result, and neither can an AI coding agent.** The code I ran was entirely correct. `rank_bm25` implements its chosen BM25 variant exactly as intended, and an AI agent could easily produce the same implementation without anything being technically wrong with the code. The problem was that I hadn’t stopped to ask what its IDF policy meant for MY corpus and MY query. **This is the trap of not understanding the code you ship, and it’s something that worries me greatly in this AI era.** They say the cost of generating code has cratered. That’s true — but I would say understanding the behavior and assumptions of the code has only become more important as a result. Correct, maintainable, testable code needs someone who understands what it’s supposed to do, and that person has to exist before you can even write the test that would have caught this. For this use case — lexical search — you don’t need to implement BM25 from scratch yourself. You just need to know enough to check: - **Which terms appear in most of your documents?** In a topic-focused collection there’s almost always at least one. - **What does your library actually store for them**, versus what the formula says? - **Do the rankings under different policies (default, clipped to zero, raw negative) match what your users are trying to do?** Look at ranks 5 through 10, where this will most likely show up. **An agent can’t catch that for you.** It doesn’t know your corpus is built around a single topic, it doesn’t know what a good top-10 looks like for your users, and it has no reason to inspect what was stored at index time. That’s the point of learning the core concepts, even when you’re not the one writing the code.