3f37e8f564* fix(graph): handle qualified C++ symbols Signed-off-by: trvon <me@trevon.dev> * refactor(vector)!: remove dead transition APIs Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use. BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers. Signed-off-by: trvon <me@trevon.dev> * fix(vector): harden PQ snapshot lifecycle Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics. Signed-off-by: trvon <me@trevon.dev> * feat(topology): expose route coordinates Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero. Signed-off-by: trvon <me@trevon.dev> * feat(xplan): add vector decision gates Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries. Signed-off-by: trvon <me@trevon.dev> * feat(topology): formalize coordinate routing Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment. Signed-off-by: trvon <me@trevon.dev> * fix(search): serialize SymSpell readers SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN. Signed-off-by: trvon <me@trevon.dev> * fix(vector): make mutations atomic Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows. * fix(search)!: surface SymSpell write errors Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms. BREAKING CHANGE: SymSpellSearch mutations now return Result values. * feat(topology): formalize candidate expansion * feat(vector): add coordinate-aware embeddings * feat(topology): preserve protected relation cover * feat(search): add graph candidate rescue * feat(daemon): expose retrieval hydration telemetry * feat(xplan): add rescue and hydration gates * refactor(content): dedupe media handlers Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic. * feat(lifecycle): prune stale corpus data Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open. * feat(search): add topology candidate rescue Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit. * feat(search): expose evidence paths * docs: define mobile host contract * formal: model bounded candidate rescue * fix(prune): inspect extensionless signatures * fix(prune): tolerate stale extensions --------- Signed-off-by: trvon <me@trevon.dev>
234 lines
12 KiB
Markdown
234 lines
12 KiB
Markdown
---
|
||
description: YAMS repo supplement — conventions, structure, and repo-scoped rules
|
||
argument-hint: "[TASK=<description>] [MODE=<engineering|bug-bounty>] [PHASE=<start|checkpoint|complete>]"
|
||
---
|
||
|
||
# AGENTS.md (Repo Supplement)
|
||
|
||
Operating model (retrieval contract, execution loop, metadata, handoff,
|
||
ask-first list): `docs/prompts/PROMPT-eng-codex.md`. C++ design/TDD/assertion/
|
||
profiling flow: `docs/prompts/PROMPT-yams-cpp-workflow.md`. This file holds
|
||
only what is specific to this repository.
|
||
|
||
## Repo Intent
|
||
|
||
Engineering on YAMS itself, plus scoped non-destructive bug-bounty research.
|
||
Dogfood YAMS as the distributed memory across sessions, agents, and handoffs —
|
||
`yams grep`/`search`/`graph` before local discovery, always (see the retrieval
|
||
contract in the prompt; the graph here is rich: function-level nodes with
|
||
`calls`/`defined_in` edges, so "who uses X" is one `yams graph --explore`).
|
||
|
||
## Mobile Host Consumers
|
||
|
||
When YAMS is embedded in a mobile app, treat `libyams_mobile` as a local corpus
|
||
SDK, not an inference or account SDK. Keep host UI focused on explicit import,
|
||
library, retrieval, result inspection, and confirmation-gated deletion. Do not
|
||
make a paid entitlement, account, or remote service a prerequisite for reading,
|
||
exporting, or deleting a person's local corpus.
|
||
|
||
Before asserting mobile support, record the exact native artifact, ABI version,
|
||
device/OS, and architecture slice. A host-only unit test proves lifecycle logic;
|
||
it does not prove that an Android `.so` or iOS XCFramework loads on hardware.
|
||
Keep physical-device claims separate and reproducible.
|
||
|
||
## CLI Philosophy
|
||
|
||
- Clean default behavior over feature flags and one-off env vars.
|
||
- Add an env knob only when it materially improves usability/debugging/perf.
|
||
- Auto-select behavior when safe rather than exposing another switch.
|
||
- Remove temporary diagnostics once product behavior is clear.
|
||
|
||
## Tuning Surfaces
|
||
|
||
- Tuning lives in typed config, not env vars: new search/topology/daemon knobs
|
||
go on the relevant config struct (`SearchEngineConfig`, `TopologyConfig`, …)
|
||
with a TOML key resolved through `ConfigResolver`
|
||
(`include/yams/daemon/components/ConfigResolver.h`).
|
||
- Benches/tests/callers set the typed field directly or write a per-test
|
||
`config.toml` (`DaemonHarnessOptions::configPath`) — never a fresh `YAMS_*`
|
||
env var. Existing `YAMS_SEARCH_TOPOLOGY_*` / `YAMS_EMBED_*` overlays are
|
||
test-only; keep them working, do not grow them.
|
||
- Reaching for `std::getenv` for a tunable is the warning sign.
|
||
|
||
## Repo Structure (Start Here)
|
||
|
||
- CLI entry points: `src/cli/` · services: `src/app/services/`
|
||
- Search (grep/semantic): `src/search/` · metadata/indexing: `src/metadata/`
|
||
- Storage engines: `src/storage/` · vector DB/embeddings: `src/vector/`
|
||
- Daemon client/server/components: `src/daemon/` · MCP: `src/mcp/`
|
||
- Tests: `tests/` · vendored libs: `third_party/` (own git repos)
|
||
|
||
## Conventions (Condensed)
|
||
|
||
- Formatting is mechanical: `.clang-format` (LLVM base, 4-space, 100-col).
|
||
- Naming: functions/vars `camelCase`; types `PascalCase`; constants
|
||
`kPascalCase`; members trailing `_`; files `snake_case`.
|
||
- `Result<T>` for fallible operations, explicit propagation.
|
||
- Prefer `YAMS_HAS_*` gates from `include/yams/core/cpp23_features.hpp`.
|
||
|
||
## Engineering Quality Loop (C++)
|
||
|
||
1. Design the seam first: observable behavior, boundary, smallest testable seam.
|
||
2. Red before green: failing Catch2 test through public/repo-stable APIs;
|
||
characterize existing behavior before refactors.
|
||
3. Assertions: recoverable failures stay in `Result<T>`; `YAMS_ASSERT`/
|
||
`YAMS_PRECONDITION`/`YAMS_POSTCONDITION` for always-on invariants;
|
||
`YAMS_DCHECK` for debug-only consistency; expressions side-effect free.
|
||
4. Refactor under green tests; no broad rewrites without seam/test evidence.
|
||
5. Measure before optimizing: workload + KPI + baseline, change one thing,
|
||
re-measure. `YAMS_*` profiling macros only with a clear question.
|
||
|
||
## Testing Conventions
|
||
|
||
- Catch2 v3; files end `_catch2_test.cpp`; live in `tests/unit/<subsystem>/`;
|
||
registered in `tests/meson.build`; targets named without `_exe`.
|
||
- Compile: `meson compile -C build/debug -j4 <target>` — never run parallel
|
||
meson compiles (build-dir lock contention).
|
||
- Run: `build/debug/tests/<target>` or `meson test -C build/debug <name>`.
|
||
- Enable tracked hooks with `git config core.hooksPath .githooks`. The default
|
||
pre-push hook runs the blocking Linux+macOS CI gate, not coverage.
|
||
Run coverage explicitly with `YAMS_PREPUSH_COVERAGE=1 git push`; it dispatches
|
||
`scripts/run-local-coverage.sh`, builds `build/coverage`, prefers ccache and
|
||
a fast linker override when available, runs the unit-suite fast path by
|
||
default, and prints a `gcovr` text summary for pushed C/C++ / Meson changes.
|
||
Opt into integration coverage with `YAMS_COVERAGE_INCLUDE_INTEGRATION=1`.
|
||
Use `YAMS_PREPUSH_GATE_SELF_TEST=1 git push` to prove hook wiring without the
|
||
expensive build/test work. Temporary bypasses: `YAMS_SKIP_PREPUSH_CI_GATE=1 git push`
|
||
or `YAMS_SKIP_COVERAGE_HOOK=1 YAMS_PREPUSH_COVERAGE=1 git push`.
|
||
- Optimize test **overlap and suite shape**, not away correctness. Keep a fast
|
||
default correctness lane, but move stress/soak, migration, heavy log-capture,
|
||
and multi-second wait tests into explicit slow suites or binaries instead of
|
||
deleting them.
|
||
- Avoid repeated expensive bootstrap in broad suites when bootstrap is not the
|
||
behavior under test: temp DB + pool + migration + daemon/KG setup should be
|
||
reused, pre-seeded, or split into narrower binaries where safe.
|
||
- Prefer deterministic synchronization (`promise`/`future`, latches, condition
|
||
variables, observable state transitions) over fixed sleeps and long polling
|
||
loops. If polling is unavoidable, keep deadlines tight and justify them.
|
||
- Non-log-assertion tests should clamp log volume; verbose daemon/service logs
|
||
belong in focused diagnostics/logging tests, not the default fast TSAN path.
|
||
- Split monolithic Catch2 binaries when one executable bundles many unrelated
|
||
files and blocks `meson test` parallelism; preserve behavior coverage while
|
||
improving failure locality and runtime.
|
||
- `-DYAMS_TESTING=1` gates `testing_*` helpers in production headers.
|
||
- RAII guards for global state (`ProfileGuard`, `EnvGuard`, `HwGuard`); reset
|
||
atomics/overrides between cases.
|
||
- DynamicCap sentinel: `UINT32_MAX` = unset (use it when resetting, not `0`).
|
||
- See `docs/developer/testing.md` for the full suite-shaping policy.
|
||
|
||
## Benchmarks & Experiments (xplan)
|
||
|
||
Daemon KPI and **search-engine** quality/latency ablations use **xplan**. Use the
|
||
harness; do not invent parallel multi-arm shell matrices. Organized default-system
|
||
numbers: `docs/benchmarks/README.md`. Per-run detail (gitignored):
|
||
`build/benchmarks/<plan>/<stamp>/` (`REPORT.md`, `ablation.md`, `metrics.csv`).
|
||
|
||
### Entry points
|
||
|
||
```bash
|
||
python3 tests/benchmarks/xplan/runner.py self-test
|
||
python3 tests/benchmarks/xplan/runner.py list-plans
|
||
python3 tests/benchmarks/xplan/runner.py download-beir # scifact + nfcorpus
|
||
python3 tests/benchmarks/xplan/runner.py run <plan> --build-dir build/release
|
||
python3 tests/benchmarks/xplan/runner.py report build/benchmarks/<plan>/<stamp>
|
||
python3 tests/benchmarks/xplan/runner.py compare <dirA> <dirB>
|
||
```
|
||
|
||
| Surface | Path |
|
||
|---------|------|
|
||
| Default-system numbers | `docs/benchmarks/README.md` |
|
||
| Harness (how it works) | `tests/benchmarks/xplan/README.md` |
|
||
| Plans / workers | `tests/benchmarks/xplan/plans/`, `…/workers/` |
|
||
| Artifacts | `build/benchmarks/**` (gitignored) |
|
||
|
||
Quality plans default to **BEIR scifact**. Do not rank search levers on
|
||
`dataset=synthetic` (ingest/load throughput only). Prefer
|
||
`build/release/tests/benchmarks/retrieval_quality_bench` for quality runs.
|
||
|
||
### Search engine measurement loop
|
||
|
||
The primary optimization surface is **`src/search/`** (`SearchEngine`, fanout,
|
||
fusion, rerank, topology assist). Topology is one optional assist path — not the
|
||
whole loop. Engine expansion (new stages, seams, fusion/rerank knobs) must be
|
||
driven by this loop, not intuition alone.
|
||
|
||
1. **Name the lever** — one typed `SearchEngineConfig` / topology field (or a
|
||
tested seam extract). No new product `YAMS_*` env tunables.
|
||
2. **Pick a plan by question:**
|
||
|
||
| Question | Plan |
|
||
|----------|------|
|
||
| Which hybrid component carries quality? | `search_component_ablation` |
|
||
| Product-default path (topology off) | `search_product_component_ablation` (`repeats=3`) |
|
||
| Vector fusion weight | `search_vector_weight_ablation` (`repeats=3`) |
|
||
| Vector weight multi-corpus gate | `search_vector_weight_0_20_multicorp` (`repeats=3`) |
|
||
| Graph-vector fusion weight | `search_graph_vector_weight_ablation` (`repeats=3`) — **parked for product defaults**: only meaningful with topology assist; SCIENTIFIC/no-KG gates force `graphVectorWeight=0` |
|
||
| Tiered execution on/off | `search_tiered_ablation` (`repeats=3`) |
|
||
| Multi-corpus transfer gate | `search_product_nfcorpus_gate` (`repeats=3`) |
|
||
| Compact engine overhead set | `subsystem_overhead` |
|
||
| Pipeline / leg stages | `leg_stage_ablation` (`repeats=3`) |
|
||
| Rerank off vs replace vs blend | `simeon_rerank` / `simeon_rerank_beir` |
|
||
| Rerank blend weight multi-corpus | `search_rerank_blend_multicorp` (`repeats=3`) |
|
||
| Per-query RR class analysis | `analyze_query_class.py` on arm `debug.jsonl` |
|
||
| Routed vs global ANN at equal budgets | `topology_routing_budget_ablation` (`repeats=3`) |
|
||
| Topology construction purity | `topology_purity_validate` (`repeats=3`) |
|
||
| Ingest / load / repair / ops KPIs | `ingest_pipeline`, `retrieval_load`, … |
|
||
|
||
3. **Decision-grade runs use `repeats>=3`.** Bare Δ with `repeats=1` is exploratory.
|
||
Summary marks: `*` clears pooled stdev, `~` within noise, unmarked = n=1.
|
||
4. **Read quality × cost** — primary columns include MRR (and friends) **and**
|
||
`search_latency_ms_p50` (plus expansion/path rates when topology is on).
|
||
Mechanism rates from `debug.jsonl` (e.g. medoid vs seed-neighbor path,
|
||
`topology_vector_seeds_added_*`) land in metrics via `retrieval_quality`.
|
||
5. **Compare stamps** with `runner.py compare` before claiming a win. Do not
|
||
rebuild `build/release` mid-run (poisons arms). xplan skips meson compile when
|
||
the bench binary already exists; force with `YAMS_BENCH_FORCE_BUILD=1`.
|
||
6. **Product default** for topology routing stays **disabled**. Experimental
|
||
engine paths must beat that default hybrid on quality without a large
|
||
latency regression, or stay opt-in / parked.
|
||
7. **Catch2 first** for seams and unit behavior (`tests/unit/search/`); xplan
|
||
is for multi-arm KPI and default-system ranking — not a substitute for TDD.
|
||
|
||
### Workers (short)
|
||
|
||
`retrieval_quality` → BEIR quality + engine/topology counters (`retrieval_quality_bench`).\
|
||
`ingestion_e2e` / `retrieval_load` / `repair_ability` / `ops_timeline` → daemon KPIs 2–5.
|
||
Ablation mapping: `workers/ablation.py` (search weight gates, ingest flags, topology
|
||
mode/expansion). Harness-only: `YAMS_BENCH_*`. Do not grow product `YAMS_*` for axes.
|
||
|
||
### Plan hygiene
|
||
|
||
- Active topology decision plans: `topology_purity_validate`,
|
||
`topology_routing_budget_ablation`.
|
||
- Product-path engine plans: `search_product_component_ablation`,
|
||
`search_vector_weight_ablation`, `search_vector_weight_0_20_multicorp`,
|
||
`search_tiered_ablation`, `search_product_nfcorpus_gate`,
|
||
`search_rerank_blend_multicorp`, `search_product_clean_baseline`.
|
||
- Parked product levers (do not use for promote/kill of defaults):
|
||
`search_graph_vector_weight_ablation` (topology-only), path/tag/metadata/entity
|
||
weight knobs on SCIENTIFIC (capability-gated to 0).
|
||
- Superseded multi-arm topology plans: `tests/benchmarks/xplan/plans/archive/`
|
||
(historical only; not for promote/kill).
|
||
- No new multi-arm shell scripts; wrappers under `scripts/` or
|
||
`tests/benchmarks/scripts/` only.
|
||
- Prefer sanitizer builds for intermittent bench crashes:
|
||
`build/asan-api|tsan-api|ubsan-api` + `YAMS_BENCH_SKIP_BUILD=1`.
|
||
|
||
## Patterns To Reuse (High Signal)
|
||
|
||
- `TuneAdvisor`: static inline atomics, relaxed reads for advisory knobs.
|
||
- `InternalEventBus`: typed channels keyed by string names.
|
||
- `ResourceGovernor`: check admission before heavy work.
|
||
- `profiling.h`: `YAMS_*` macros, no-op without Tracy.
|
||
- Daemon async paths: Boost.Asio coroutines (`boost::asio::awaitable<…>`).
|
||
|
||
## Bug-Bounty (Repo Use)
|
||
|
||
Scoped targets only; read-only/low-impact validation first; no persistence,
|
||
destructive payloads, or exfiltration; redact secrets before `yams add`; index
|
||
evidence as short reproducible notes (`source=evidence`).
|
||
|
||
## Keep This File Lean
|
||
|
||
Generic agent behavior belongs in `docs/prompts/PROMPT-eng-codex.md`. No
|
||
session-specific skill catalogs here.
|