# Retrieval Precision Optimization

This runbook defines the accuracy-first loop for retrieval tuning.

Scope:

- optimize ranking quality (ordering) from a fixed corpus/query baseline,
- keep vectors enabled,
- avoid mixing throughput tuning concerns into ranking decisions.

## Baseline Inputs

Use one or more JSONL candidate runs from the same fixed corpus/query slice as baseline anchors.

## Build a Baseline Summary

Generate a normalized summary from one or more runs:

```bash
python3 tests/scripts/summarize_retrieval_opt_jsonl.py \
  --input <candidate-run-a.jsonl> \
  --input <candidate-run-b.jsonl> \
  --output <summary.json>
```

The summary includes:

- latest successful row per candidate,
- aggregate objective stats per candidate (mean/stddev/min/max),
- canonical winner snapshot and env overrides.

## Accuracy Targets

Primary target:

- `nDCG@10`

Secondary targets:

- `MRR@10`
- `MAP`

Guardrail:

- `recall@100` (do not accept large regressions)

## Precision Levers (Ranking)

Focus tuning on ranking layers in this order:

1. fusion and component weighting,
2. vector-only gating (`vectorOnlyThreshold`, `vectorOnlyPenalty`),
3. full-text score shaping (query-intent-aware penalties),
4. graph rerank,
5. reranker and symbol post-processing.

## Runtime Override Knobs

Enable environment overrides and adjust precision knobs:

```bash
YAMS_ENABLE_ENV_OVERRIDES=1
YAMS_SEARCH_VECTOR_ONLY_THRESHOLD=0.92
YAMS_SEARCH_VECTOR_ONLY_PENALTY=0.75
YAMS_SEARCH_RRF_K=10
YAMS_SEARCH_CONCEPT_BOOST_WEIGHT=0.08
```

These are intended for benchmark/ablation loops before hardcoding defaults.

## Notes

- Keep dataset size/query count constant while comparing ranking configs.
- Prefer isolated candidate execution (one process per candidate) to avoid
  daemon/plugin lifecycle contamination.
- Promote a new baseline only after stable gains on the fixed benchmark slice.

## Precision loop with the simeon backend

The default daemon/ONNX embedding path costs ~25–100 ms per document on
CPU, which makes this loop sluggish: re-ingesting a medium corpus between
lever changes takes minutes. The vendored simeon backend
(`third_party/simeon`, `YAMS_EMBED_BACKEND=simeon`) runs at 60k+ docs/sec
in the VerySparse 4096→384 configuration, which turns the loop into
seconds.

Use simeon while iterating on fusion weights / `vectorOnlyThreshold` /
RRF_k; validate the final configuration against daemon afterwards to
catch paraphrase cases where simeon’s lexical sketch underperforms a
learned embedding.

Fast iterate:

```bash
YAMS_ENABLE_ENV_OVERRIDES=1 \
YAMS_EMBED_BACKEND=simeon \
YAMS_SEARCH_VECTOR_ONLY_THRESHOLD=0.92 \
YAMS_SEARCH_VECTOR_ONLY_PENALTY=0.75 \
YAMS_SEARCH_RRF_K=10 \
  ./build/debug/tests/benchmarks/retrieval_quality_bench \
  --benchmark_out=run_simeon.jsonl --benchmark_out_format=json
```

Use the topology retrieval matrix helper when promoting a candidate.
Keep generated rows outside the docs and publish only stable summary tables here.
Every emitted row carries an `embedding_backend` field so downstream summarizers can group without
rescanning environment state.