Class LexicalIndex

The BM25 engine over a caller's tokenizer. It imports nothing, so a page can load it; it ranks documents holding any query word (or only those holding every one), matches a stored word whole or by a query word it begins with, and is saved with LexicalIndex.toJSON and restored with LexicalIndex.fromJSON.

A document's score is the sum, over the query's words, of IDF(t) * (tf(t,D) * (k1 + 1)) / (tf(t,D) + k1 * (1 - b + b * |D| / avgdl)), with the Robertson-Walker IDF(t) = log((N - n(t) + 0.5) / (n(t) + 0.5) + 1).

Example

const index = new LexicalIndex({ tokenize: lexicalTokens });
index.addDocument('s1#0', 'The chapters on the budget review');
index.search('chap budg', 10, { prefix: true, match: 'all' }); // the passage s1#0

const saved = JSON.stringify(index.toJSON());
const restored = LexicalIndex.fromJSON(JSON.parse(saved), { tokenize: lexicalTokens });
Copy