6b5b6647bd* chore(repo): land the retrieval contract and drop root cruft Lands work that had been sitting uncommitted in the working tree: - docs/architecture/retrieval_contracts.md plus its executable form, tests/benchmarks/retrieval_contract.py and test_retrieval_contract.py, now registered in meson as retrieval_contract_unit (suite: policy). .gitignore keeps them tracked beneath the tests/benchmarks/*.py ignore. - AGENTS.md: the Development Memory section, and the product default recorded as hybrid_assist + shadow with topology-off as the control, which is what search_topology_stage already does. docs/benchmarks/index.md gains the matching search_shadow_cost plan. - .gitleaks.toml, and a Secret scan workflow that actually runs it on push and pull requests (nothing in .github invoked gitleaks before). Deletes three root files with zero references anywhere in the tree: bench_profile.json (ad-hoc profiling leftover), CppProperties.json (Visual Studio config; the repo standardises on Meson), and spdlog.pc (a hand-copied spdlog 1.12 pkg-config with prefix=/usr/local that would mislead any PKG_CONFIG_PATH lookup; spdlog comes from Conan). docs/plans/ (two p2p designs already shipped in5a53cdfaandd012d95b) was ingested into YAMS with task=desloppification source=plan, hydrated via yams cat and byte-compared before removal, per the new AGENTS.md rule; tests/scripts/analyze_simeon_adaptivity.py no longer points at a plans directory. Also reflows two stale line anchors in raw_environment_allowlist.txt for tools/yams-cli/main.cpp (shifted by11c39157) so the policy suite passes again; no new environment writes. Not included: the external/yams-repo submodule pointer in the working tree was a stale local clone two commits behind, not an intended bump. Signed-off-by: trvon <me@trevon.dev> * test(policy): make the configuration-authority ratchet bidirectional check_configuration_authority.py only reported keys and TOML readers present in code but missing from the allowlists, so the reviewed YAMS_* surface could grow but never shrink: production_environment_keys.txt even said removals "may remain grandfathered". An allowlist that only accretes stops describing the code. The policy now also fails on allowlist entries with no remaining occurrence in src/, include/, or tools/, naming the lines to delete. Key removals in later commits therefore have to update the allowlist in the same change, and the count in the success banner becomes a real measure of the environment surface (432 today). stale_entries() exposes the two stale sets for tooling; three unit tests cover stale key, stale reader, and the returned names. Signed-off-by: trvon <me@trevon.dev> * tools(policy): report where every configuration key is referenced Adds tests/scripts/report_config_key_usage.py: for each YAMS_* key in production_environment_keys.txt and each dotted TOML key literally resolved in ConfigResolver.cpp, count whole-token references per tree bucket (src, test, bench, docs, plugin; build output and the allowlists themselves excluded) and assign a verdict: KEEP when anything outside product code exercises or documents the key, CANDIDATE when only product code mentions it, NOT_IN_SRC when the allowlist is stale. The report is evidence, not a decision. From here on a key deletion must paste its CANDIDATE rows into the change description; the first audit pass hand-listed keys that turned out to have bench and test consumers, which is exactly the mistake a mechanical inventory avoids. Today: 432 env keys (215 CANDIDATE, 217 KEEP), 117 TOML keys (103 CANDIDATE, 14 KEEP). Registered as config_key_usage_report_unit. The report counts only files git tracks (git ls-files) when the root is a work tree, so an ignored local bench harness cannot turn a CANDIDATE into a KEEP on one machine and not another; outside a work tree it falls back to scanning the directory, which is what the synthetic-tree test exercises. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev>
108 lines
4 KiB
Markdown
108 lines
4 KiB
Markdown
# Benchmarks
|
||
|
||
The tables below are the latest recorded snapshot of the **default system**
|
||
(no experiment matrices). They were produced with an xplan harness that is now
|
||
kept as optional, gitignored local tooling. Per-run detail:
|
||
`build/benchmarks/<plan>/<timestamp>-<config-hash>/` (`run_manifest.json`, `REPORT.md`,
|
||
`ablation.md`, `metrics.csv`). Tracked API/core/search/tree-diff benchmarks use the same default
|
||
`build/benchmarks/<suite>/<timestamp>-<config-hash>/` layout and record effective CLI values,
|
||
field-level sources, and execution provenance in `run_manifest.json`; `--out-dir` remains exact.
|
||
|
||
| | |
|
||
|--|--|
|
||
| Local harness | `tests/benchmarks/xplan/` (optional; not distributed) |
|
||
| Artifacts | `build/benchmarks/<plan>/<timestamp>-<config-hash>/` (gitignored) |
|
||
| Micro baselines (code) | `tests/benchmarks/baseline/*.json` |
|
||
| Full Simeon ingestion oracle | [`ingestion_oracle.md`](ingestion_oracle.md) |
|
||
|
||
**Corpora:** quality plans use **BEIR scifact** (`dataset=scifact`, 2000 docs × 50
|
||
queries) by default. Cache: `~/.cache/yams/benchmarks/<name>` (auto-download).
|
||
`dataset=synthetic` is only for ingest/load/ops throughput — not ranking.
|
||
|
||
**Topology construction defaults (when topology rebuild runs):**
|
||
`min_edge_score=0.25`, `max_component_docs=64`, engine `connected`.
|
||
|
||
---
|
||
|
||
## Latest snapshot
|
||
|
||
**Stamp:** `kpi-20260709T031007Z` (search quality) · `kpi-20260709T003228Z` (ingest / load / repair / ops)\
|
||
**Host:** local macOS
|
||
**Builds:** `build/release` (ingest, repair, quality); `build/prepush-macos` (load, ops — Catch2)
|
||
|
||
### Ingest — `ingest_pipeline` (80 docs × 1 KB, synthetic throughput)
|
||
|
||
This historical snapshot predates the full Simeon identity, persistence, and lifecycle contract in
|
||
[`ingestion_oracle.md`](ingestion_oracle.md); do not use it as the comprehensive optimization
|
||
baseline.
|
||
|
||
| Arm | docs/s | wall_ms | complete |
|
||
|-----|------:|--------:|----------|
|
||
| baseline | 96.6 | 828 | yes |
|
||
| no_kg | 95.6 | 837 | yes |
|
||
| no_vectors | 87.0 | 920 | yes |
|
||
| no_vectors_no_kg | 97.2 | 823 | yes |
|
||
| no_gliner | 97.8 | 818 | yes |
|
||
| minimal | 111.0 | 721 | yes |
|
||
|
||
Artifacts: `build/benchmarks/ingest_pipeline/kpi-20260709T003228Z-ingest/`
|
||
|
||
### Retrieval load — `retrieval_load`
|
||
|
||
| Arm | p50_ms | p95_ms | QPS | searches |
|
||
|-----|------:|------:|----:|---------:|
|
||
| baseline_hybrid | 28.0 | 41.3 | 123 | 189 |
|
||
| keyword_only | 1.20 | 5.45 | 362 | 189 |
|
||
|
||
Artifacts: `build/benchmarks/retrieval_load/kpi-20260709T003228Z-load/`
|
||
|
||
### Search quality — BEIR scifact (default hybrid)
|
||
|
||
2000 docs · 50 queries · topk=10 · topology routing **off** (historical control; the current
|
||
product default is `hybrid_assist` + `shadow`)
|
||
|
||
This table is historical quality evidence, not a current latency baseline. Keep
|
||
current product default, topology-disabled, and traced-shadow runs distinct.
|
||
Stage tracing can perform an additional exact document-complete retrieval;
|
||
use untraced runs for product latency and paired traced runs for attribution.
|
||
|
||
| Metric | Value |
|
||
|--------|------:|
|
||
| MRR | 0.613 |
|
||
| nDCG | 0.634 |
|
||
| MAP | 0.608 |
|
||
| recall@10 | 0.713 |
|
||
| precision@10 | 0.076 |
|
||
|
||
Artifacts: `build/benchmarks/topology_optimize_v2/kpi-20260709T031007Z-topo-optv2b/` (`topo_off` arm)
|
||
|
||
### Repair — `repair_ability` (20 faults)
|
||
|
||
| Arm | injected | repaired | status |
|
||
|-----|--------:|---------:|--------|
|
||
| fts5 | 20 | 20 | ok |
|
||
| graph | 20 | 225* | ok |
|
||
| embed | 20 | 0 | fail (timeout ~120s) |
|
||
| embed_all_ops | 20 | 0 | fail (timeout ~120s) |
|
||
|
||
\* graph arm counts operations, not 1:1 documents.
|
||
|
||
Artifacts: `build/benchmarks/repair_ability/kpi-20260709T003228Z-repair/`
|
||
|
||
### Ops / idle — `ops_timeline`
|
||
|
||
| Arm | ingest docs/s | idle_fraction |
|
||
|-----|-------------:|--------------:|
|
||
| baseline_no_vectors | 212 | 1.0 |
|
||
| with_vectors | 212 | 1.0 |
|
||
|
||
Artifacts: `build/benchmarks/ops_timeline/kpi-20260709T003228Z-ops/`
|
||
|
||
---
|
||
|
||
## Refresh
|
||
|
||
1. Prepare the relevant corpus.
|
||
2. Run equivalent benchmark workloads under a common stamp.
|
||
3. Replace tables above from each `summary.md` / `REPORT.md` (default arms only).
|
||
4. Update the stamp line.
|