103 lines
3.7 KiB
Markdown
103 lines
3.7 KiB
Markdown
# Benchmarks
|
||
|
||
The tables below are the latest recorded snapshot of the **default system**
|
||
(no experiment matrices). They were produced with an xplan harness that is now
|
||
kept as optional, gitignored local tooling. Per-run detail:
|
||
`build/benchmarks/<plan>/<timestamp>-<config-hash>/` (`run_manifest.json`, `REPORT.md`,
|
||
`ablation.md`, `metrics.csv`). Tracked API/core/search/tree-diff benchmarks use the same default
|
||
`build/benchmarks/<suite>/<timestamp>-<config-hash>/` layout and record effective CLI values,
|
||
field-level sources, and execution provenance in `run_manifest.json`; `--out-dir` remains exact.
|
||
|
||
| | |
|
||
|--|--|
|
||
| Local harness | `tests/benchmarks/xplan/` (optional; not distributed) |
|
||
| Artifacts | `build/benchmarks/<plan>/<timestamp>-<config-hash>/` (gitignored) |
|
||
| Micro baselines (code) | `tests/benchmarks/baseline/*.json` |
|
||
| Full Simeon ingestion oracle | [`ingestion_oracle.md`](ingestion_oracle.md) |
|
||
|
||
**Corpora:** quality plans use **BEIR scifact** (`dataset=scifact`, 2000 docs × 50
|
||
queries) by default. Cache: `~/.cache/yams/benchmarks/<name>` (auto-download).
|
||
`dataset=synthetic` is only for ingest/load/ops throughput — not ranking.
|
||
|
||
**Topology construction defaults (when topology rebuild runs):**
|
||
`min_edge_score=0.25`, `max_component_docs=64`, engine `connected`.
|
||
|
||
---
|
||
|
||
## Latest snapshot
|
||
|
||
**Stamp:** `kpi-20260709T031007Z` (search quality) · `kpi-20260709T003228Z` (ingest / load / repair / ops)\
|
||
**Host:** local macOS
|
||
**Builds:** `build/release` (ingest, repair, quality); `build/prepush-macos` (load, ops — Catch2)
|
||
|
||
### Ingest — `ingest_pipeline` (80 docs × 1 KB, synthetic throughput)
|
||
|
||
This historical snapshot predates the full Simeon identity, persistence, and lifecycle contract in
|
||
[`ingestion_oracle.md`](ingestion_oracle.md); do not use it as the comprehensive optimization
|
||
baseline.
|
||
|
||
| Arm | docs/s | wall_ms | complete |
|
||
|-----|------:|--------:|----------|
|
||
| baseline | 96.6 | 828 | yes |
|
||
| no_kg | 95.6 | 837 | yes |
|
||
| no_vectors | 87.0 | 920 | yes |
|
||
| no_vectors_no_kg | 97.2 | 823 | yes |
|
||
| no_gliner | 97.8 | 818 | yes |
|
||
| minimal | 111.0 | 721 | yes |
|
||
|
||
Artifacts: `build/benchmarks/ingest_pipeline/kpi-20260709T003228Z-ingest/`
|
||
|
||
### Retrieval load — `retrieval_load`
|
||
|
||
| Arm | p50_ms | p95_ms | QPS | searches |
|
||
|-----|------:|------:|----:|---------:|
|
||
| baseline_hybrid | 28.0 | 41.3 | 123 | 189 |
|
||
| keyword_only | 1.20 | 5.45 | 362 | 189 |
|
||
|
||
Artifacts: `build/benchmarks/retrieval_load/kpi-20260709T003228Z-load/`
|
||
|
||
### Search quality — BEIR scifact (default hybrid)
|
||
|
||
2000 docs · 50 queries · topk=10 · topology routing **off** (historical control; the current
|
||
product default is `hybrid_assist` + `shadow`)
|
||
|
||
| Metric | Value |
|
||
|--------|------:|
|
||
| MRR | 0.613 |
|
||
| nDCG | 0.634 |
|
||
| MAP | 0.608 |
|
||
| recall@10 | 0.713 |
|
||
| precision@10 | 0.076 |
|
||
|
||
Artifacts: `build/benchmarks/topology_optimize_v2/kpi-20260709T031007Z-topo-optv2b/` (`topo_off` arm)
|
||
|
||
### Repair — `repair_ability` (20 faults)
|
||
|
||
| Arm | injected | repaired | status |
|
||
|-----|--------:|---------:|--------|
|
||
| fts5 | 20 | 20 | ok |
|
||
| graph | 20 | 225* | ok |
|
||
| embed | 20 | 0 | fail (timeout ~120s) |
|
||
| embed_all_ops | 20 | 0 | fail (timeout ~120s) |
|
||
|
||
\* graph arm counts operations, not 1:1 documents.
|
||
|
||
Artifacts: `build/benchmarks/repair_ability/kpi-20260709T003228Z-repair/`
|
||
|
||
### Ops / idle — `ops_timeline`
|
||
|
||
| Arm | ingest docs/s | idle_fraction |
|
||
|-----|-------------:|--------------:|
|
||
| baseline_no_vectors | 212 | 1.0 |
|
||
| with_vectors | 212 | 1.0 |
|
||
|
||
Artifacts: `build/benchmarks/ops_timeline/kpi-20260709T003228Z-ops/`
|
||
|
||
---
|
||
|
||
## Refresh
|
||
|
||
1. Prepare the relevant corpus.
|
||
2. Run equivalent benchmark workloads under a common stamp.
|
||
3. Replace tables above from each `summary.md` / `REPORT.md` (default arms only).
|
||
4. Update the stamp line.
|