Outer MaxSim scored only the first eight sentences of each document, so
a relevant passage later in a long document was never seen, which
defeats the point of fragment scoring. Sample sentences evenly across
the document (first and last kept), always reserve a document-head
fragment, and bound per-call fragment embeddings so large rerank
windows shrink the per-document budget instead of growing unbounded.
Implement three research-backed retrieval optimizations connecting
paperbridge literature findings with YAMS subsystems:
1. SGC Feature Smoothing (Proposal A):
- Add sgcHops and sgcNormalize to TopologyBuildConfig.
- Diffuse document embeddings across reciprocal nearest-neighbor graph
edges prior to clustering, formalizing Lean SGC.lean.
- Re-normalize diffused embeddings to preserve unit hypersphere metric geometry.
- Wire into ConnectedComponent, Louvain, and KMeans topology engines.
2. Matryoshka Prefix Binary Quantization (Proposal C):
- Add maxDimensions parameter to BinaryQuantizer::quantize and
maxPrefixDimension to BinaryQuantizedIndex::build.
- Support fast 64-bit word-aligned prefix Hamming shortlisting in
SparseGuidedClusterRouter via bqPrefixDimension.
3. Fragment Outer-MaxSim Reranker (Proposal B):
- Implement SimeonScoringMode { SingleVectorCosine, FragmentOuterMaxSim }
in SimeonModelProvider.
- Add extractDocumentFragments and late-interaction outer-MaxSim scoring
inspired by ColBERTv2/PLAID/SPLATE to prevent relevance dilution in long docs.
- Expose simeonRerankOuterMaxSim on SearchEngineConfig and ConfigResolver.
- Add Catch2 test coverage and verify BEIR scifact benchmark metrics.
Signed-off-by: trvon <me@trevon.dev>
