Token vector search: retrieve, rank, and verify

Define useful retrieval

A similar sentence and an answer-bearing passage are not necessarily the same thing. Write the task in ordinary language before choosing a model. A product-support query may require the right procedure, product version, and access scope, not merely a page about the same topic.

The Sentence Transformers semantic search guide distinguishes symmetric and asymmetric search tasks. Use that distinction to review the query and document roles in your own application.

Make the first collection readable

Choose a small set of passages you can inspect completely. Include a direct answer, a paraphrase, a related non-answer, an exact identifier, and an unavailable document. Check extraction, headings, and source revision before generating representations.

The vector tokenization guide explains how to preserve useful passage boundaries. Problems that begin in extraction are unlikely to be fixed by increasingly complicated ranking.

Save an exact-search baseline

Use compatible query and document representations with a documented metric. For a manageable evaluation set, retrieve exact neighbors before adding an approximate index. Preserve the query, scores, passage identifiers, collection version, and configuration.

This baseline separates index approximation from the rest of the pipeline. It still needs relevance labels: the mathematically nearest passage may be background rather than an answer. The vector database overview connects index experiments to filters and resource tradeoffs.

Test the questions that expose weaknesses

Use identifier-heavy questions, paraphrases, version-specific requests, and no-answer cases. Keep a held-out set for release decisions rather than repeatedly tuning on every example. Record label disagreements and distinguish useful background from a direct solution.

Measure whether an accepted passage appears in the first k results and state the denominator. Inspect the position of the first useful result, repeated passages, and latency. Segment-level failures can remain important even when an aggregate number looks healthy.

Improve candidate coverage and ranking separately

If the answer is missing from the candidate set, inspect source coverage, chunking, encoding, and approximate-search behavior. A reranker cannot recover a passage it never receives. If the answer is present but buried, inspect scoring, duplicates, or a targeted reranking experiment.

Hybrid search can combine lexical and vector candidates when exact terms and paraphrases require complementary signals. Keep the original baselines, use a deliberate fusion method, and compare like-for-like candidate windows. The articles below walk through a first retrieval experiment and rank-based fusion without promising universal improvements.