--- description: YAMS repo supplement — conventions, structure, and repo-scoped rules argument-hint: "[TASK=] [MODE=] [PHASE=]" --- # AGENTS.md (Repo Supplement) Operating model (retrieval contract, execution loop, metadata, handoff, ask-first list): `docs/prompts/PROMPT-eng-codex.md`. C++ design/TDD/assertion/ profiling flow: `docs/prompts/PROMPT-yams-cpp-workflow.md`. This file holds only what is specific to this repository. ## Repo Intent Engineering on YAMS itself, plus scoped non-destructive bug-bounty research. Dogfood YAMS as the distributed memory across sessions, agents, and handoffs — `yams grep`/`search`/`graph` before local discovery, always (see the retrieval contract in the prompt). The graph holds document, path, version, entity, and semantic-neighbor relations; code-symbol nodes and `calls`/`defined_in` edges were removed in v0.20, so answer "who uses X" with `yams grep`. ## Development Memory Store development audits, plans, decisions, and handoffs in YAMS with task/source/ phase metadata, not as new local Markdown documents. Keep repository instructions here; keep executable tests and raw benchmark artifacts in their normal locations. Before removing a local note after ingestion, hydrate its returned hash and verify the content. An add acknowledgement is not a successful read-back. If hydration fails, report the blocker and preserve the only recoverable copy until it is safe to remove; do not describe the migration as complete. ## Mobile Host Consumers When YAMS is embedded in a mobile app, treat `libyams_mobile` as a local corpus SDK, not an inference or account SDK. Keep host UI focused on explicit import, library, retrieval, result inspection, and confirmation-gated deletion. Do not make a paid entitlement, account, or remote service a prerequisite for reading, exporting, or deleting a person's local corpus. Before asserting mobile support, record the exact native artifact, ABI version, device/OS, and architecture slice. A host-only unit test proves lifecycle logic; it does not prove that an Android `.so` or iOS XCFramework loads on hardware. Keep physical-device claims separate and reproducible. ## CLI Philosophy - Clean default behavior over feature flags and one-off env vars. - Add an env knob only when it materially improves usability/debugging/perf. - Auto-select behavior when safe rather than exposing another switch. - Remove temporary diagnostics once product behavior is clear. ## Tuning Surfaces - Tuning lives in typed config, not env vars: new search/topology/daemon knobs go on the relevant config struct (`SearchEngineConfig`, `TopologyConfig`, …) with a TOML key resolved through `ConfigResolver` (`include/yams/daemon/components/ConfigResolver.h`). - Benches/tests/callers set the typed field directly or write a per-test `config.toml` (`DaemonHarnessOptions::configPath`) — never a fresh `YAMS_*` env var. Existing `YAMS_SEARCH_TOPOLOGY_*` / `YAMS_EMBED_*` overlays are test-only; keep them working, do not grow them. - Reaching for `std::getenv` for a tunable is the warning sign. ## Repo Structure (Start Here) - CLI entry points: `src/cli/` · services: `src/app/services/` - Search (grep/semantic): `src/search/` · metadata/indexing: `src/metadata/` - Storage engines: `src/storage/` · vector DB/embeddings: `src/vector/` - Daemon client/server/components: `src/daemon/` · MCP: `src/mcp/` - Tests: `tests/` · vendored libs: `third_party/` (own git repos) ## Conventions (Condensed) - Formatting is mechanical: `.clang-format` (LLVM base, 4-space, 100-col). - Naming: functions/vars `camelCase`; types `PascalCase`; constants `kPascalCase`; members trailing `_`; files `snake_case`. - `Result` for fallible operations, explicit propagation. - Public/installable headers include other public YAMS headers through the canonical `` include root. This validates installed-SDK layout and keeps includes stable when headers move. Use quoted relative includes only for private, non-installed implementation headers or explicit test/bootstrap exceptions. - Prefer `YAMS_HAS_*` gates from `include/yams/core/cpp23_features.hpp`. ## Engineering Quality Loop (C++) 1. Design the seam first: observable behavior, boundary, smallest testable seam. 2. Red before green: failing Catch2 test through public/repo-stable APIs; characterize existing behavior before refactors. 3. Assertions: recoverable failures stay in `Result`; `YAMS_ASSERT`/ `YAMS_PRECONDITION`/`YAMS_POSTCONDITION` for always-on invariants; `YAMS_DCHECK` for debug-only consistency; expressions side-effect free. 4. Refactor under green tests; no broad rewrites without seam/test evidence. 5. Measure before optimizing: workload + KPI + baseline, change one thing, re-measure. `YAMS_*` profiling macros only with a clear question. ## Build Setup - Configure/build through the repo script; it resolves Conan + Meson and prefers Clang: `./setup.sh Debug` (also `Release`, `Profiling`, `Fuzzing`). - Build dirs are chosen per type: `Debug` -> `build/debug` (the unit-test lane; `.clangd` points here), `Release` -> `build/release`, `Profiling` -> `build/profiling`, `Fuzzing` -> `build/fuzzing`. Override with `YAMS_BUILD_DIR`. The legacy `builddir` name is still accepted by some helper scripts. - Debug enables TSAN by default; pass `--no-tsan` for faster local iteration. - Tests are opt-in outside Debug. If `meson compile -C ` reports `target not found`, that dir was configured without tests; reconfigure with `meson setup --reconfigure -Dbuild-tests=true --wrap-mode=default`. - One writer per build dir; never run parallel `meson compile` invocations. ## Testing Conventions - Catch2 v3; files end `_catch2_test.cpp`; live in `tests/unit//`; registered in `tests/meson.build`; targets named without `_exe`. - Compile: `meson compile -C build/debug -j4 ` — never run parallel meson compiles (build-dir lock contention). - Run: `build/debug/tests/` or `meson test -C build/debug `. - Enable tracked hooks with `git config core.hooksPath .githooks`. The default pre-push hook runs the blocking Linux+macOS CI gate, not coverage. Run coverage explicitly with `YAMS_PREPUSH_COVERAGE=1 git push`; it dispatches `scripts/run-local-coverage.sh`, builds `build/coverage`, prefers ccache and a fast linker override when available, runs the unit-suite fast path by default, and prints a `gcovr` text summary for pushed C/C++ / Meson changes. Opt into integration coverage with `YAMS_COVERAGE_INCLUDE_INTEGRATION=1`. Use `YAMS_PREPUSH_GATE_SELF_TEST=1 git push` to prove hook wiring without the expensive build/test work. Temporary bypasses: `YAMS_SKIP_PREPUSH_CI_GATE=1 git push` or `YAMS_SKIP_COVERAGE_HOOK=1 YAMS_PREPUSH_COVERAGE=1 git push`. - Optimize test **overlap and suite shape**, not away correctness. Keep a fast default correctness lane, but move stress/soak, migration, heavy log-capture, and multi-second wait tests into explicit slow suites or binaries instead of deleting them. - Avoid repeated expensive bootstrap in broad suites when bootstrap is not the behavior under test: temp DB + pool + migration + daemon/KG setup should be reused, pre-seeded, or split into narrower binaries where safe. - Prefer deterministic synchronization (`promise`/`future`, latches, condition variables, observable state transitions) over fixed sleeps and long polling loops. If polling is unavoidable, keep deadlines tight and justify them. - Non-log-assertion tests should clamp log volume; verbose daemon/service logs belong in focused diagnostics/logging tests, not the default fast TSAN path. - Split monolithic Catch2 binaries when one executable bundles many unrelated files and blocks `meson test` parallelism; preserve behavior coverage while improving failure locality and runtime. - `-DYAMS_TESTING=1` gates `testing_*` helpers in production headers. - RAII guards for global state (`ProfileGuard`, `EnvGuard`, `HwGuard`); reset atomics/overrides between cases. - DynamicCap sentinel: `UINT32_MAX` = unset (use it when resetting, not `0`). ## Benchmarks & Experiments (local xplan) The xplan harness is optional, gitignored local tooling and is not present in a clean clone. When it is available, daemon KPI and **search-engine** quality/latency ablations use it; do not invent parallel multi-arm shell matrices. Organized default-system numbers: `docs/benchmarks/index.md`. Per-run detail (gitignored): `build/benchmarks///` (`REPORT.md`, `ablation.md`, `metrics.csv`). ### Entry points ```bash python3 tests/benchmarks/xplan/runner.py self-test python3 tests/benchmarks/xplan/runner.py list-plans python3 tests/benchmarks/xplan/runner.py download-beir # scifact + nfcorpus python3 tests/benchmarks/xplan/runner.py run --build-dir build/release python3 tests/benchmarks/xplan/runner.py report build/benchmarks// python3 tests/benchmarks/xplan/runner.py compare ``` | Surface | Path | |---------|------| | Default-system numbers | `docs/benchmarks/index.md` | | Harness (how it works) | `tests/benchmarks/xplan/README.md` | | Plans / workers | `tests/benchmarks/xplan/plans/`, `…/workers/` | | Artifacts | `build/benchmarks/**` (gitignored) | Quality plans default to **BEIR scifact**. Do not rank search levers on `dataset=synthetic` (ingest/load throughput only). Prefer `build/release/tests/benchmarks/retrieval_quality_bench` for quality runs. ### Search engine measurement loop The primary optimization surface is **`src/search/`** (`SearchEngine`, fanout, fusion, rerank, topology assist). Topology is one optional assist path — not the whole loop. Engine expansion (new stages, seams, fusion/rerank knobs) must be driven by this loop, not intuition alone. 1. **Name the lever** — one typed `SearchEngineConfig` / topology field (or a tested seam extract). No new product `YAMS_*` env tunables. 2. **Pick a plan by question:** | Question | Plan | |----------|------| | Which hybrid component carries quality? | `search_component_ablation` | | Topology-off control | `search_product_component_ablation` (`repeats=3`) | | Current default vs topology off vs traced shadow | `search_shadow_cost` (`repeats=3`) | | Vector fusion weight | `search_vector_weight_ablation` (`repeats=3`) | | Vector weight multi-corpus gate | `search_vector_weight_0_20_multicorp` (`repeats=3`) | | Graph-vector fusion weight | `search_graph_vector_weight_ablation` (`repeats=3`) — **parked for product defaults**: only meaningful with topology assist; SCIENTIFIC/no-KG gates force `graphVectorWeight=0` | | Tiered execution on/off | `search_tiered_ablation` (`repeats=3`) | | Multi-corpus transfer gate | `search_product_nfcorpus_gate` (`repeats=3`) | | Compact engine overhead set | `subsystem_overhead` | | Pipeline / leg stages | `leg_stage_ablation` (`repeats=3`) | | Rerank off vs replace vs blend | `simeon_rerank` / `simeon_rerank_beir` | | Rerank blend weight multi-corpus | `search_rerank_blend_multicorp` (`repeats=3`) | | Per-query RR class analysis | `analyze_query_class.py` on arm `debug.jsonl` | | Routed vs global ANN at equal budgets | `topology_routing_budget_ablation` (`repeats=3`) | | Topology construction purity | `topology_purity_validate` (`repeats=3`) | | Ingest / load / repair / ops KPIs | `ingest_pipeline`, `retrieval_load`, … | 3. **Decision-grade runs use `repeats>=3`.** Bare Δ with `repeats=1` is exploratory. Summary marks: `*` clears pooled stdev, `~` within noise, unmarked = n=1. 4. **Read quality × cost** — primary columns include MRR (and friends) **and** `search_latency_ms_p50` (plus expansion/path rates when topology is on). Mechanism rates from `debug.jsonl` (e.g. medoid vs seed-neighbor path, `topology_vector_seeds_added_*`) land in metrics via `retrieval_quality`. 5. **Compare stamps** with `runner.py compare` before claiming a win. Do not rebuild `build/release` mid-run (poisons arms). xplan skips meson compile when the bench binary already exists; force with `YAMS_BENCH_FORCE_BUILD=1`. 6. **Product default** is **hybrid_assist + shadow**; topology-disabled hybrid is the explicit no-topology control. Shadow can perform counterfactual work, and stage tracing adds exact-control work: compare untraced timings separately. Experimental engine paths must beat the default hybrid on quality without a large latency regression, or stay opt-in / parked. 7. **Catch2 first** for seams and unit behavior (`tests/unit/search/`); xplan is for multi-arm KPI and default-system ranking — not a substitute for TDD. ### Workers (short) `retrieval_quality` → BEIR quality + engine/topology counters (`retrieval_quality_bench`).\ `ingestion_e2e` / `retrieval_load` / `repair_ability` / `ops_timeline` → daemon KPIs 2–5. Ablation mapping: `workers/ablation.py` (search weight gates, ingest flags, topology mode/expansion). Harness-only: `YAMS_BENCH_*`. Do not grow product `YAMS_*` for axes. ### Plan hygiene - Active topology decision plans: `topology_purity_validate`, `topology_routing_budget_ablation`. - Product-path engine plans: `search_product_component_ablation`, `search_vector_weight_ablation`, `search_vector_weight_0_20_multicorp`, `search_tiered_ablation`, `search_product_nfcorpus_gate`, `search_rerank_blend_multicorp`, `search_product_clean_baseline`. - Parked product levers (do not use for promote/kill of defaults): `search_graph_vector_weight_ablation` (topology-only), path/tag/metadata/entity weight knobs on SCIENTIFIC (capability-gated to 0). - Superseded multi-arm topology plans: `tests/benchmarks/xplan/plans/archive/` (historical only; not for promote/kill). - No new multi-arm shell scripts; wrappers under `scripts/` or `tests/benchmarks/scripts/` only. - Prefer sanitizer builds for intermittent bench crashes: `build/asan-api|tsan-api|ubsan-api` + `YAMS_BENCH_SKIP_BUILD=1`. ## Branches, PRs & Promotion - Feature PRs target `experimental`; `main` only moves through the automated "promote experimental to main" draft PR, which lists everything on `experimental` not yet on `main` (not just the latest change). - Stacked PRs (a PR whose base is another PR's branch): merge bottom-up. Merge the child into the parent branch *before* merging the parent, or retarget the child to `experimental` first. A child merged into a parent branch that was already merged lands only on that branch and silently misses `experimental`. - After merging a stack, verify nothing was stranded: `git cherry origin/experimental origin/` must print no `+` lines. - Submodule (`third_party/*`) PRs merge with a merge commit or rebase, never squash, and land before the YAMS PR that bumps the pointer, so the pinned SHA stays reachable from the submodule's default branch. - Merge method by PR kind: - Feature PRs into `experimental`: squash is fine. - Promotion PRs (`experimental` → `main`) and back-merges (`main` → `experimental`): **merge commit only**. A squash copies the changes but not the commits, so Release Candidate rejects the next candidate with "candidate must contain the complete base history". - After every change to `main` (a promotion, a release PR, or a direct commit), back-merge `main` into `experimental`. Check with `git log --oneline origin/experimental..origin/main`, which must print nothing. - Promotion PR bodies must not contain `BEGIN_COMMIT_OVERRIDE`. Release Please applies the override to every commit in the PR and repeats it in the notes. - To pin a version (for example, when a `feat` would otherwise bump the minor version), add a `Release-As: X.Y.Z` footer to a commit on `main`. ## Patterns To Reuse (High Signal) - `TuneAdvisor`: static inline atomics, relaxed reads for advisory knobs. - `InternalEventBus`: typed channels keyed by string names. - `ResourceGovernor`: check admission before heavy work. - `profiling.h`: `YAMS_*` macros, no-op without Tracy. - Daemon async paths: Boost.Asio coroutines (`boost::asio::awaitable<…>`). ## Background Work & Resource Governor YAMS must run on memory-constrained, non-Apple hosts. In October 2026 the daemon pinned a weak machine's CPU with p2p and topology work. - Daemon code that starts a thread, posts a job, spawns a coroutine or arms a periodic timer checks `ResourceGovernor` first: `canAdmitWork()`, `admitDeferrable(DeferrableWork::…)` or its cap in `ScalingCaps`. Admission is not enough for long jobs: a rebuild or scan that runs for seconds re-checks pressure between batches and yields. No free-running loops. - Worker counts come from `TuneAdvisor` or the governor, not `hardware_concurrency()`. - Idle means no active non-health requests and empty queues. An open connection is not activity. - Background work stops through a bounded, checked drain. A `void stop()` that returns while work is still running is not a shutdown. - A new or changed background path needs a KPI on the constrained lane (`setup.sh Profiling`, Linux x86, pinned cores and RAM): idle CPU and RSS over time with the feature on. Profile before and after. Code reading is not evidence of a fix. - Gaps as of 2026-10: `BackgroundTaskManager`, `p2p_transport` (accept thread, one thread per session, 5 s reconnect loop) and `WriteCoordinator` do not consult the governor. Topology rebuilds and semantic maintenance are admitted through `admitDeferrable` but then run to completion with no pressure check inside `TopologyManager::runRebuild`. Fix these before adding background work. ## Bug-Bounty (Repo Use) Scoped targets only; read-only/low-impact validation first; no persistence, destructive payloads, or exfiltration; redact secrets before `yams add`; index evidence as short reproducible notes (`source=evidence`). ## Keep This File Lean Generic agent behavior belongs in `docs/prompts/PROMPT-eng-codex.md`. No session-specific skill catalogs here.