6b5b6647bd* chore(repo): land the retrieval contract and drop root cruft Lands work that had been sitting uncommitted in the working tree: - docs/architecture/retrieval_contracts.md plus its executable form, tests/benchmarks/retrieval_contract.py and test_retrieval_contract.py, now registered in meson as retrieval_contract_unit (suite: policy). .gitignore keeps them tracked beneath the tests/benchmarks/*.py ignore. - AGENTS.md: the Development Memory section, and the product default recorded as hybrid_assist + shadow with topology-off as the control, which is what search_topology_stage already does. docs/benchmarks/index.md gains the matching search_shadow_cost plan. - .gitleaks.toml, and a Secret scan workflow that actually runs it on push and pull requests (nothing in .github invoked gitleaks before). Deletes three root files with zero references anywhere in the tree: bench_profile.json (ad-hoc profiling leftover), CppProperties.json (Visual Studio config; the repo standardises on Meson), and spdlog.pc (a hand-copied spdlog 1.12 pkg-config with prefix=/usr/local that would mislead any PKG_CONFIG_PATH lookup; spdlog comes from Conan). docs/plans/ (two p2p designs already shipped in5a53cdfaandd012d95b) was ingested into YAMS with task=desloppification source=plan, hydrated via yams cat and byte-compared before removal, per the new AGENTS.md rule; tests/scripts/analyze_simeon_adaptivity.py no longer points at a plans directory. Also reflows two stale line anchors in raw_environment_allowlist.txt for tools/yams-cli/main.cpp (shifted by11c39157) so the policy suite passes again; no new environment writes. Not included: the external/yams-repo submodule pointer in the working tree was a stale local clone two commits behind, not an intended bump. Signed-off-by: trvon <me@trevon.dev> * test(policy): make the configuration-authority ratchet bidirectional check_configuration_authority.py only reported keys and TOML readers present in code but missing from the allowlists, so the reviewed YAMS_* surface could grow but never shrink: production_environment_keys.txt even said removals "may remain grandfathered". An allowlist that only accretes stops describing the code. The policy now also fails on allowlist entries with no remaining occurrence in src/, include/, or tools/, naming the lines to delete. Key removals in later commits therefore have to update the allowlist in the same change, and the count in the success banner becomes a real measure of the environment surface (432 today). stale_entries() exposes the two stale sets for tooling; three unit tests cover stale key, stale reader, and the returned names. Signed-off-by: trvon <me@trevon.dev> * tools(policy): report where every configuration key is referenced Adds tests/scripts/report_config_key_usage.py: for each YAMS_* key in production_environment_keys.txt and each dotted TOML key literally resolved in ConfigResolver.cpp, count whole-token references per tree bucket (src, test, bench, docs, plugin; build output and the allowlists themselves excluded) and assign a verdict: KEEP when anything outside product code exercises or documents the key, CANDIDATE when only product code mentions it, NOT_IN_SRC when the allowlist is stale. The report is evidence, not a decision. From here on a key deletion must paste its CANDIDATE rows into the change description; the first audit pass hand-listed keys that turned out to have bench and test consumers, which is exactly the mistake a mechanical inventory avoids. Today: 432 env keys (215 CANDIDATE, 217 KEEP), 117 TOML keys (103 CANDIDATE, 14 KEEP). Registered as config_key_usage_report_unit. The report counts only files git tracks (git ls-files) when the root is a work tree, so an ignored local bench harness cannot turn a CANDIDATE into a KEEP on one machine and not another; outside a work tree it falls back to scanning the directory, which is what the synthetic-tree test exercises. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev>
4 KiB
Benchmarks
The tables below are the latest recorded snapshot of the default system
(no experiment matrices). They were produced with an xplan harness that is now
kept as optional, gitignored local tooling. Per-run detail:
build/benchmarks/<plan>/<timestamp>-<config-hash>/ (run_manifest.json, REPORT.md,
ablation.md, metrics.csv). Tracked API/core/search/tree-diff benchmarks use the same default
build/benchmarks/<suite>/<timestamp>-<config-hash>/ layout and record effective CLI values,
field-level sources, and execution provenance in run_manifest.json; --out-dir remains exact.
| Local harness | tests/benchmarks/xplan/ (optional; not distributed) |
| Artifacts | build/benchmarks/<plan>/<timestamp>-<config-hash>/ (gitignored) |
| Micro baselines (code) | tests/benchmarks/baseline/*.json |
| Full Simeon ingestion oracle | ingestion_oracle.md |
Corpora: quality plans use BEIR scifact (dataset=scifact, 2000 docs × 50
queries) by default. Cache: ~/.cache/yams/benchmarks/<name> (auto-download).
dataset=synthetic is only for ingest/load/ops throughput — not ranking.
Topology construction defaults (when topology rebuild runs):
min_edge_score=0.25, max_component_docs=64, engine connected.
Latest snapshot
Stamp: kpi-20260709T031007Z (search quality) · kpi-20260709T003228Z (ingest / load / repair / ops)
Host: local macOS
Builds: build/release (ingest, repair, quality); build/prepush-macos (load, ops — Catch2)
Ingest — ingest_pipeline (80 docs × 1 KB, synthetic throughput)
This historical snapshot predates the full Simeon identity, persistence, and lifecycle contract in
ingestion_oracle.md; do not use it as the comprehensive optimization
baseline.
| Arm | docs/s | wall_ms | complete |
|---|---|---|---|
| baseline | 96.6 | 828 | yes |
| no_kg | 95.6 | 837 | yes |
| no_vectors | 87.0 | 920 | yes |
| no_vectors_no_kg | 97.2 | 823 | yes |
| no_gliner | 97.8 | 818 | yes |
| minimal | 111.0 | 721 | yes |
Artifacts: build/benchmarks/ingest_pipeline/kpi-20260709T003228Z-ingest/
Retrieval load — retrieval_load
| Arm | p50_ms | p95_ms | QPS | searches |
|---|---|---|---|---|
| baseline_hybrid | 28.0 | 41.3 | 123 | 189 |
| keyword_only | 1.20 | 5.45 | 362 | 189 |
Artifacts: build/benchmarks/retrieval_load/kpi-20260709T003228Z-load/
Search quality — BEIR scifact (default hybrid)
2000 docs · 50 queries · topk=10 · topology routing off (historical control; the current
product default is hybrid_assist + shadow)
This table is historical quality evidence, not a current latency baseline. Keep current product default, topology-disabled, and traced-shadow runs distinct. Stage tracing can perform an additional exact document-complete retrieval; use untraced runs for product latency and paired traced runs for attribution.
| Metric | Value |
|---|---|
| MRR | 0.613 |
| nDCG | 0.634 |
| MAP | 0.608 |
| recall@10 | 0.713 |
| precision@10 | 0.076 |
Artifacts: build/benchmarks/topology_optimize_v2/kpi-20260709T031007Z-topo-optv2b/ (topo_off arm)
Repair — repair_ability (20 faults)
| Arm | injected | repaired | status |
|---|---|---|---|
| fts5 | 20 | 20 | ok |
| graph | 20 | 225* | ok |
| embed | 20 | 0 | fail (timeout ~120s) |
| embed_all_ops | 20 | 0 | fail (timeout ~120s) |
* graph arm counts operations, not 1:1 documents.
Artifacts: build/benchmarks/repair_ability/kpi-20260709T003228Z-repair/
Ops / idle — ops_timeline
| Arm | ingest docs/s | idle_fraction |
|---|---|---|
| baseline_no_vectors | 212 | 1.0 |
| with_vectors | 212 | 1.0 |
Artifacts: build/benchmarks/ops_timeline/kpi-20260709T003228Z-ops/
Refresh
- Prepare the relevant corpus.
- Run equivalent benchmark workloads under a common stamp.
- Replace tables above from each
summary.md/REPORT.md(default arms only). - Update the stamp line.