# Benchmarks

The tables below are the latest recorded snapshot of the **default system**
(no experiment matrices). They were produced with an xplan harness that is now
kept as optional, gitignored local tooling. Per-run detail:
`build/benchmarks/<plan>/<timestamp>-<config-hash>/` (`run_manifest.json`, `REPORT.md`,
`ablation.md`, `metrics.csv`). Tracked API/core/search/tree-diff benchmarks use the same default
`build/benchmarks/<suite>/<timestamp>-<config-hash>/` layout and record effective CLI values,
field-level sources, and execution provenance in `run_manifest.json`; `--out-dir` remains exact.

| | |
|--|--|
| Local harness | `tests/benchmarks/xplan/` (optional; not distributed) |
| Artifacts | `build/benchmarks/<plan>/<timestamp>-<config-hash>/` (gitignored) |
| Micro baselines (code) | `tests/benchmarks/baseline/*.json` |
| Full Simeon ingestion oracle | [`ingestion_oracle.md`](ingestion_oracle.md) |

**Corpora:** quality plans use **BEIR scifact** (`dataset=scifact`, 2000 docs × 50
queries) by default. Cache: `~/.cache/yams/benchmarks/<name>` (auto-download).
`dataset=synthetic` is only for ingest/load/ops throughput — not ranking.

**Topology construction defaults (when topology rebuild runs):**
`min_edge_score=0.25`, `max_component_docs=64`, engine `connected`.

---

## Latest snapshot

**Stamp:** `kpi-20260709T031007Z` (search quality) · `kpi-20260709T003228Z` (ingest / load / repair / ops)\
**Host:** local macOS
**Builds:** `build/release` (ingest, repair, quality); `build/prepush-macos` (load, ops — Catch2)

### Ingest — `ingest_pipeline` (80 docs × 1 KB, synthetic throughput)

This historical snapshot predates the full Simeon identity, persistence, and lifecycle contract in
[`ingestion_oracle.md`](ingestion_oracle.md); do not use it as the comprehensive optimization
baseline.

| Arm | docs/s | wall_ms | complete |
|-----|------:|--------:|----------|
| baseline | 96.6 | 828 | yes |
| no_kg | 95.6 | 837 | yes |
| no_vectors | 87.0 | 920 | yes |
| no_vectors_no_kg | 97.2 | 823 | yes |
| no_gliner | 97.8 | 818 | yes |
| minimal | 111.0 | 721 | yes |

Artifacts: `build/benchmarks/ingest_pipeline/kpi-20260709T003228Z-ingest/`

### Retrieval load — `retrieval_load`

| Arm | p50_ms | p95_ms | QPS | searches |
|-----|------:|------:|----:|---------:|
| baseline_hybrid | 28.0 | 41.3 | 123 | 189 |
| keyword_only | 1.20 | 5.45 | 362 | 189 |

Artifacts: `build/benchmarks/retrieval_load/kpi-20260709T003228Z-load/`

### Search quality — BEIR scifact (default hybrid)

2000 docs · 50 queries · topk=10 · topology routing **off** (historical control; the current
product default is `hybrid_assist` + `shadow`)

| Metric | Value |
|--------|------:|
| MRR | 0.613 |
| nDCG | 0.634 |
| MAP | 0.608 |
| recall@10 | 0.713 |
| precision@10 | 0.076 |

Artifacts: `build/benchmarks/topology_optimize_v2/kpi-20260709T031007Z-topo-optv2b/` (`topo_off` arm)

### Repair — `repair_ability` (20 faults)

| Arm | injected | repaired | status |
|-----|--------:|---------:|--------|
| fts5 | 20 | 20 | ok |
| graph | 20 | 225* | ok |
| embed | 20 | 0 | fail (timeout ~120s) |
| embed_all_ops | 20 | 0 | fail (timeout ~120s) |

\* graph arm counts operations, not 1:1 documents.

Artifacts: `build/benchmarks/repair_ability/kpi-20260709T003228Z-repair/`

### Ops / idle — `ops_timeline`

| Arm | ingest docs/s | idle_fraction |
|-----|-------------:|--------------:|
| baseline_no_vectors | 212 | 1.0 |
| with_vectors | 212 | 1.0 |

Artifacts: `build/benchmarks/ops_timeline/kpi-20260709T003228Z-ops/`

---

## Refresh

1. Prepare the relevant corpus.
2. Run equivalent benchmark workloads under a common stamp.
3. Replace tables above from each `summary.md` / `REPORT.md` (default arms only).
4. Update the stamp line.