yams/AGENTS.md

description	argument-hint
YAMS repo supplement — conventions, structure, and repo-scoped rules	[TASK=<description>] [MODE=<engineering|bug-bounty>] [PHASE=<start|checkpoint|complete>]

AGENTS.md (Repo Supplement)

Operating model (retrieval contract, execution loop, metadata, handoff, ask-first list): docs/prompts/PROMPT-eng-codex.md. C++ design/TDD/assertion/ profiling flow: docs/prompts/PROMPT-yams-cpp-workflow.md. This file holds only what is specific to this repository.

Repo Intent

Engineering on YAMS itself, plus scoped non-destructive bug-bounty research. Dogfood YAMS as the distributed memory across sessions, agents, and handoffs — yams grep/search/graph before local discovery, always (see the retrieval contract in the prompt). The graph holds document, path, version, entity, and semantic-neighbor relations; code-symbol nodes and calls/defined_in edges were removed in v0.20, so answer "who uses X" with yams grep.

Development Memory

Store development audits, plans, decisions, and handoffs in YAMS with task/source/ phase metadata, not as new local Markdown documents. Keep repository instructions here; keep executable tests and raw benchmark artifacts in their normal locations. Before removing a local note after ingestion, hydrate its returned hash and verify the content. An add acknowledgement is not a successful read-back. If hydration fails, report the blocker and preserve the only recoverable copy until it is safe to remove; do not describe the migration as complete.

Mobile Host Consumers

When YAMS is embedded in a mobile app, treat libyams_mobile as a local corpus SDK, not an inference or account SDK. Keep host UI focused on explicit import, library, retrieval, result inspection, and confirmation-gated deletion. Do not make a paid entitlement, account, or remote service a prerequisite for reading, exporting, or deleting a person's local corpus.

Before asserting mobile support, record the exact native artifact, ABI version, device/OS, and architecture slice. A host-only unit test proves lifecycle logic; it does not prove that an Android .so or iOS XCFramework loads on hardware. Keep physical-device claims separate and reproducible.

CLI Philosophy

• Clean default behavior over feature flags and one-off env vars.
• Add an env knob only when it materially improves usability/debugging/perf.
• Auto-select behavior when safe rather than exposing another switch.
• Remove temporary diagnostics once product behavior is clear.

Tuning Surfaces

• Tuning lives in typed config, not env vars: new search/topology/daemon knobs go on the relevant config struct (SearchEngineConfig, TopologyConfig, …) with a TOML key resolved through ConfigResolver (include/yams/daemon/components/ConfigResolver.h).
• Benches/tests/callers set the typed field directly or write a per-test config.toml (DaemonHarnessOptions::configPath) — never a fresh YAMS_* env var. Existing YAMS_SEARCH_TOPOLOGY_* / YAMS_EMBED_* overlays are test-only; keep them working, do not grow them.
• Reaching for std::getenv for a tunable is the warning sign.

Repo Structure (Start Here)

• CLI entry points: src/cli/ · services: src/app/services/
• Search (grep/semantic): src/search/ · metadata/indexing: src/metadata/
• Storage engines: src/storage/ · vector DB/embeddings: src/vector/
• Daemon client/server/components: src/daemon/ · MCP: src/mcp/
• Tests: tests/ · vendored libs: third_party/ (own git repos)

Conventions (Condensed)

• Formatting is mechanical: .clang-format (LLVM base, 4-space, 100-col).
• Naming: functions/vars camelCase; types PascalCase; constants kPascalCase; members trailing _; files snake_case.
• Result<T> for fallible operations, explicit propagation.
• Public/installable headers include other public YAMS headers through the canonical <yams/...> include root. This validates installed-SDK layout and keeps includes stable when headers move. Use quoted relative includes only for private, non-installed implementation headers or explicit test/bootstrap exceptions.
• Prefer YAMS_HAS_* gates from include/yams/core/cpp23_features.hpp.

Engineering Quality Loop (C++)

1. Design the seam first: observable behavior, boundary, smallest testable seam.
2. Red before green: failing Catch2 test through public/repo-stable APIs; characterize existing behavior before refactors.
3. Assertions: recoverable failures stay in Result<T>; YAMS_ASSERT/ YAMS_PRECONDITION/YAMS_POSTCONDITION for always-on invariants; YAMS_DCHECK for debug-only consistency; expressions side-effect free.
4. Refactor under green tests; no broad rewrites without seam/test evidence.
5. Measure before optimizing: workload + KPI + baseline, change one thing, re-measure. YAMS_* profiling macros only with a clear question.

Build Setup

• Configure/build through the repo script; it resolves Conan + Meson and prefers Clang: ./setup.sh Debug (also Release, Profiling, Fuzzing).
• Build dirs are chosen per type: Debug -> build/debug (the unit-test lane; .clangd points here), Release -> build/release, Profiling -> build/profiling, Fuzzing -> build/fuzzing. Override with YAMS_BUILD_DIR. The legacy builddir name is still accepted by some helper scripts.
• Debug enables TSAN by default; pass --no-tsan for faster local iteration.
• Tests are opt-in outside Debug. If meson compile -C <dir> <target> reports target not found, that dir was configured without tests; reconfigure with meson setup --reconfigure <dir> -Dbuild-tests=true --wrap-mode=default.
• One writer per build dir; never run parallel meson compile invocations.

Testing Conventions

• Catch2 v3; files end _catch2_test.cpp; live in tests/unit/<subsystem>/; registered in tests/meson.build; targets named without _exe.
• Compile: meson compile -C build/debug -j4 <target> — never run parallel meson compiles (build-dir lock contention).
• Run: build/debug/tests/<target> or meson test -C build/debug <name>.
• Enable tracked hooks with git config core.hooksPath .githooks. The default pre-push hook runs the blocking Linux+macOS CI gate, not coverage. Run coverage explicitly with YAMS_PREPUSH_COVERAGE=1 git push; it dispatches scripts/run-local-coverage.sh, builds build/coverage, prefers ccache and a fast linker override when available, runs the unit-suite fast path by default, and prints a gcovr text summary for pushed C/C++ / Meson changes. Opt into integration coverage with YAMS_COVERAGE_INCLUDE_INTEGRATION=1. Use YAMS_PREPUSH_GATE_SELF_TEST=1 git push to prove hook wiring without the expensive build/test work. Temporary bypasses: YAMS_SKIP_PREPUSH_CI_GATE=1 git push or YAMS_SKIP_COVERAGE_HOOK=1 YAMS_PREPUSH_COVERAGE=1 git push.
• Optimize test overlap and suite shape, not away correctness. Keep a fast default correctness lane, but move stress/soak, migration, heavy log-capture, and multi-second wait tests into explicit slow suites or binaries instead of deleting them.
• Avoid repeated expensive bootstrap in broad suites when bootstrap is not the behavior under test: temp DB + pool + migration + daemon/KG setup should be reused, pre-seeded, or split into narrower binaries where safe.
• Prefer deterministic synchronization (promise/future, latches, condition variables, observable state transitions) over fixed sleeps and long polling loops. If polling is unavoidable, keep deadlines tight and justify them.
• Non-log-assertion tests should clamp log volume; verbose daemon/service logs belong in focused diagnostics/logging tests, not the default fast TSAN path.
• Split monolithic Catch2 binaries when one executable bundles many unrelated files and blocks meson test parallelism; preserve behavior coverage while improving failure locality and runtime.
• -DYAMS_TESTING=1 gates testing_* helpers in production headers.
• RAII guards for global state (ProfileGuard, EnvGuard, HwGuard); reset atomics/overrides between cases.
• DynamicCap sentinel: UINT32_MAX = unset (use it when resetting, not 0).

Benchmarks & Experiments (local xplan)

The xplan harness is optional, gitignored local tooling and is not present in a clean clone. When it is available, daemon KPI and search-engine quality/latency ablations use it; do not invent parallel multi-arm shell matrices. Organized default-system numbers: docs/benchmarks/index.md. Per-run detail (gitignored): build/benchmarks/<plan>/<stamp>/ (REPORT.md, ablation.md, metrics.csv).

Entry points

python3 tests/benchmarks/xplan/runner.py self-test
python3 tests/benchmarks/xplan/runner.py list-plans
python3 tests/benchmarks/xplan/runner.py download-beir   # scifact + nfcorpus
python3 tests/benchmarks/xplan/runner.py run <plan> --build-dir build/release
python3 tests/benchmarks/xplan/runner.py report build/benchmarks/<plan>/<stamp>
python3 tests/benchmarks/xplan/runner.py compare <dirA> <dirB>

Surface	Path
Default-system numbers	docs/benchmarks/index.md
Harness (how it works)	tests/benchmarks/xplan/README.md
Plans / workers	tests/benchmarks/xplan/plans/, …/workers/
Artifacts	build/benchmarks/** (gitignored)

Quality plans default to BEIR scifact. Do not rank search levers on dataset=synthetic (ingest/load throughput only). Prefer build/release/tests/benchmarks/retrieval_quality_bench for quality runs.

Search engine measurement loop

The primary optimization surface is src/search/ (SearchEngine, fanout, fusion, rerank, topology assist). Topology is one optional assist path — not the whole loop. Engine expansion (new stages, seams, fusion/rerank knobs) must be driven by this loop, not intuition alone.

1. Name the lever — one typed SearchEngineConfig / topology field (or a tested seam extract). No new product YAMS_* env tunables.

2. Pick a plan by question:

   Question	Plan
   Which hybrid component carries quality?	search_component_ablation
   Topology-off control	search_product_component_ablation (repeats=3)
   Current default vs topology off vs traced shadow	search_shadow_cost (repeats=3)
   Vector fusion weight	search_vector_weight_ablation (repeats=3)
   Vector weight multi-corpus gate	search_vector_weight_0_20_multicorp (repeats=3)
   Graph-vector fusion weight	search_graph_vector_weight_ablation (repeats=3) — parked for product defaults: only meaningful with topology assist; SCIENTIFIC/no-KG gates force graphVectorWeight=0
   Tiered execution on/off	search_tiered_ablation (repeats=3)
   Multi-corpus transfer gate	search_product_nfcorpus_gate (repeats=3)
   Compact engine overhead set	subsystem_overhead
   Pipeline / leg stages	leg_stage_ablation (repeats=3)
   Rerank off vs replace vs blend	simeon_rerank / simeon_rerank_beir
   Rerank blend weight multi-corpus	search_rerank_blend_multicorp (repeats=3)
   Per-query RR class analysis	analyze_query_class.py on arm debug.jsonl
   Routed vs global ANN at equal budgets	topology_routing_budget_ablation (repeats=3)
   Topology construction purity	topology_purity_validate (repeats=3)
   Ingest / load / repair / ops KPIs	ingest_pipeline, retrieval_load, …

3. Decision-grade runs use repeats>=3. Bare Δ with repeats=1 is exploratory. Summary marks: * clears pooled stdev, ~ within noise, unmarked = n=1.

4. Read quality × cost — primary columns include MRR (and friends) and search_latency_ms_p50 (plus expansion/path rates when topology is on). Mechanism rates from debug.jsonl (e.g. medoid vs seed-neighbor path, topology_vector_seeds_added_*) land in metrics via retrieval_quality.

5. Compare stamps with runner.py compare before claiming a win. Do not rebuild build/release mid-run (poisons arms). xplan skips meson compile when the bench binary already exists; force with YAMS_BENCH_FORCE_BUILD=1.

6. Product default is hybrid_assist + shadow; topology-disabled hybrid is the explicit no-topology control. Shadow can perform counterfactual work, and stage tracing adds exact-control work: compare untraced timings separately. Experimental engine paths must beat the default hybrid on quality without a large latency regression, or stay opt-in / parked.

7. Catch2 first for seams and unit behavior (tests/unit/search/); xplan is for multi-arm KPI and default-system ranking — not a substitute for TDD.

Workers (short)

retrieval_quality → BEIR quality + engine/topology counters (retrieval_quality_bench).
ingestion_e2e / retrieval_load / repair_ability / ops_timeline → daemon KPIs 2–5. Ablation mapping: workers/ablation.py (search weight gates, ingest flags, topology mode/expansion). Harness-only: YAMS_BENCH_*. Do not grow product YAMS_* for axes.

Plan hygiene

• Active topology decision plans: topology_purity_validate, topology_routing_budget_ablation.
• Product-path engine plans: search_product_component_ablation, search_vector_weight_ablation, search_vector_weight_0_20_multicorp, search_tiered_ablation, search_product_nfcorpus_gate, search_rerank_blend_multicorp, search_product_clean_baseline.
• Parked product levers (do not use for promote/kill of defaults): search_graph_vector_weight_ablation (topology-only), path/tag/metadata/entity weight knobs on SCIENTIFIC (capability-gated to 0).
• Superseded multi-arm topology plans: tests/benchmarks/xplan/plans/archive/ (historical only; not for promote/kill).
• No new multi-arm shell scripts; wrappers under scripts/ or tests/benchmarks/scripts/ only.
• Prefer sanitizer builds for intermittent bench crashes: build/asan-api|tsan-api|ubsan-api + YAMS_BENCH_SKIP_BUILD=1.

Branches, PRs & Promotion

• Feature PRs target experimental; main only moves through the automated "promote experimental to main" draft PR, which lists everything on experimental not yet on main (not just the latest change).
• Stacked PRs (a PR whose base is another PR's branch): merge bottom-up. Merge the child into the parent branch before merging the parent, or retarget the child to experimental first. A child merged into a parent branch that was already merged lands only on that branch and silently misses experimental.
• After merging a stack, verify nothing was stranded: git cherry origin/experimental origin/<branch> must print no + lines.
• Submodule (third_party/*) PRs merge with a merge commit or rebase, never squash, and land before the YAMS PR that bumps the pointer, so the pinned SHA stays reachable from the submodule's default branch.
• Merge method by PR kind:
  • Feature PRs into experimental: squash is fine.
  • Promotion PRs (experimental → main) and back-merges (main → experimental): merge commit only. A squash copies the changes but not the commits, so Release Candidate rejects the next candidate with "candidate must contain the complete base history".
• After every change to main (a promotion, a release PR, or a direct commit), back-merge main into experimental. Check with git log --oneline origin/experimental..origin/main, which must print nothing.
• Promotion PR bodies must not contain BEGIN_COMMIT_OVERRIDE. Release Please applies the override to every commit in the PR and repeats it in the notes.
• To pin a version (for example, when a feat would otherwise bump the minor version), add a Release-As: X.Y.Z footer to a commit on main.

Patterns To Reuse (High Signal)

• TuneAdvisor: static inline atomics, relaxed reads for advisory knobs.
• InternalEventBus: typed channels keyed by string names.
• ResourceGovernor: check admission before heavy work.
• profiling.h: YAMS_* macros, no-op without Tracy.
• Daemon async paths: Boost.Asio coroutines (boost::asio::awaitable<…>).

Background Work & Resource Governor

YAMS must run on memory-constrained, non-Apple hosts. In October 2026 the daemon pinned a weak machine's CPU with p2p and topology work.

• Daemon code that starts a thread, posts a job, spawns a coroutine or arms a periodic timer checks ResourceGovernor first: canAdmitWork(), admitDeferrable(DeferrableWork::…) or its cap in ScalingCaps. Admission is not enough for long jobs: a rebuild or scan that runs for seconds re-checks pressure between batches and yields. No free-running loops.
• Worker counts come from TuneAdvisor or the governor, not hardware_concurrency().
• Idle means no active non-health requests and empty queues. An open connection is not activity.
• Background work stops through a bounded, checked drain. A void stop() that returns while work is still running is not a shutdown.
• A new or changed background path needs a KPI on the constrained lane (setup.sh Profiling, Linux x86, pinned cores and RAM): idle CPU and RSS over time with the feature on. Profile before and after. Code reading is not evidence of a fix.
• Gaps as of 2026-10: BackgroundTaskManager, p2p_transport (accept thread, one thread per session, 5 s reconnect loop) and WriteCoordinator do not consult the governor. Topology rebuilds and semantic maintenance are admitted through admitDeferrable but then run to completion with no pressure check inside TopologyManager::runRebuild. Fix these before adding background work.

Bug-Bounty (Repo Use)

Scoped targets only; read-only/low-impact validation first; no persistence, destructive payloads, or exfiltration; redact secrets before yams add; index evidence as short reproducible notes (source=evidence).

Keep This File Lean

Generic agent behavior belongs in docs/prompts/PROMPT-eng-codex.md. No session-specific skill catalogs here.