Skip to content

🔍 Using Sparse Retrievers (BM25)

BM25 (Best Matching 25) is a classic sparse retrieval algorithm that uses term frequency and inverse document frequency for relevance scoring.

Overview

BM25 is ideal for: - Fast retrieval with low computational cost - Exact keyword matching - Large-scale document collections

Using BM25 with Prebuilt Indices

Rankify provides prebuilt BM25 indices for Wikipedia and MS MARCO:

from rankify.dataset.dataset import Document, Question
from rankify.retrievers.retriever import Retriever

# Create queries
documents = [
    Document(question=Question("Who invented the telephone?")),
    Document(question=Question("What is the capital of France?"))
]

# BM25 on Wikipedia
retriever = Retriever(method="bm25", n_docs=10, index_type="wiki")
retrieved_docs = retriever.retrieve(documents)

# BM25 on MS MARCO
retriever_msmarco = Retriever(method="bm25", n_docs=10, index_type="msmarco")
retrieved_msmarco = retriever_msmarco.retrieve(documents)

Building a Custom BM25 Index

Use the CLI to build your own BM25 index:

# Prepare your corpus as JSONL with 'id', 'title', 'text' fields
# Example: {"id": "1", "title": "Einstein", "text": "Albert Einstein was..."}

# Build the index
rankify-index index data/my_corpus.jsonl \
    --retriever bm25 \
    --output ./my_indices \
    --threads 32

Then use it in Python:

retriever = Retriever(
    method="bm25",
    n_docs=10,
    index_folder="./my_indices/my_corpus/bm25_index"
)

Configuration Options

Parameter Description Default
n_docs Number of documents to retrieve 10
index_type Prebuilt index type (wiki, msmarco) wiki
index_folder Custom index path None

Understanding BM25 Scores

BM25 scores are based on: - Term Frequency (TF): How often query terms appear in documents - Inverse Document Frequency (IDF): How rare query terms are across the collection - Document Length Normalization: Shorter documents get a slight boost

# Access scores from retrieved contexts
for doc in retrieved_docs:
    for ctx in doc.contexts:
        print(f"Score: {ctx.score:.4f}")
        print(f"Title: {ctx.title}")
        print(f"Text: {ctx.text[:200]}...")
        print("---")

Best Practices

  1. Query preprocessing: BM25 works best with clear, keyword-rich queries
  2. Index quality: Ensure documents are properly tokenized
  3. Combine with reranking: BM25 + neural reranker often outperforms pure dense retrieval

Next Steps