Class BM25Index

BM25 sparse keyword index for hybrid retrieval.

Dense embeddings excel at semantic similarity but miss exact keyword matches (e.g., error codes, function names, product IDs). BM25 catches these by scoring documents based on term frequency, inverse document frequency, and document length normalization.

It is a LexicalIndex with AgentOS's tokenizer: the configured tokenize or pipeline, else the built-in one, whose stop words are natural's English list when that package loads and ENGLISH_STOP_WORDS otherwise, read on the first tokenization, not when the module is imported. toJSON() saves it and BM25Index.fromJSON restores it.

Example: Basic usage

const index = new BM25Index({ k1: 1.5, b: 0.75 });

index.addDocuments([
  { id: 'doc-1', text: 'TypeScript compiler error TS2304' },
  { id: 'doc-2', text: 'JavaScript runtime TypeError explanation' },
  { id: 'doc-3', text: 'Fix error TS2304 by adding type declarations' },
]);

const results = index.search('error TS2304', 5);
// results[0].id === 'doc-3' (exact match on "error" + "TS2304")
// results[1].id === 'doc-1' (exact match on "error" + "TS2304")
Copy

Example: Combined with HybridSearcher

const hybrid = new HybridSearcher(vectorStore, embeddingManager, bm25Index, {
  denseWeight: 0.7,
  sparseWeight: 0.3,
});
const results = await hybrid.search('What does error TS2304 mean?');
Copy