c178f98619
Some checks failed
Build and Push Docker Images / Build Docker Image (linux/amd64) (push) Failing after 58s
Secret scan / gitleaks (push) Failing after 7s
Mirror source to Forgejo / mirror (push) Has been skipped
Release Please / release-please (push) Failing after 14s
Build and Push Docker Images / Build Docker Image (linux/arm64 native) (push) Has been cancelled
Build and Push Docker Images / Create Multi-arch Manifest (push) Has been cancelled
Tests / Static Analysis (push) Has been cancelled
Tests / Warm Conan & ccache (linux-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-arm64) (push) Has been cancelled
Tests / Warm Conan & ccache (linux-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (macos-x86_64) (push) Has been cancelled
Tests / Warm Conan & ccache (windows-x86_64) (push) Has been cancelled
Tests / warm (push) Has been cancelled
Tests / Tests (linux-arm64) (push) Has been cancelled
Tests / Tests (macos-arm64) (push) Has been cancelled
Tests / Tests (linux-x86_64-asan) (push) Has been cancelled
Tests / Tests (linux-x86_64-ubsan) (push) Has been cancelled
Tests / Tests (linux-x86_64) (push) Has been cancelled
Tests / Tests (windows-x86_64) (push) Has been cancelled
* fix(storage): retry reference counter init on transient DB lock The add-sandbox smoke test (and CLI startup generally) can fail with 'database is locked' when a prior async batch commit holds the reference database longer than ReferenceDB's short retry budget, which CI runners hit consistently while local runs pass. Retry the whole init (schema + migrations) with a bounded, growing backoff (up to ~12s total) so transient contention no longer fails startup. Adds a Catch2 test that holds a write lock across init and releases it mid-retry. * refactor: deduplicate trim/escapeRegex/hasWildcard helpers The same string helpers were copy-pasted across CLI and service code: trim/trimCopy in 5 files, escapeRegex in 3, hasWildcard in 2, and the regex meta-char table in 2. Move canonical versions to yams::common::string_utils.h (trimCopy, hasWildcard, escapeRegex, isRegexMetaChar) and wire all call sites to them; daemon_helpers' trim_copy now delegates. Add a Catch2 test for the shared helpers. Kept isTransientMetadataError and annotateResultRelations untouched: they have semantically different implementations and need a separate review, not a mechanical merge. * refactor(cli): share time filters between get/list Same 6-field time filter logic copy-pasted in GetCommand and ListCommand (~72 lines each). Extract to yams::cli::applyDocTimeFilters + DocTimeFilters, wire both commands, add unit test. * refactor: dedup retry helpers + test fixtures - isTransientMetadataError + MetadataTelemetry: 2 copies (grep/search) -> shared service_utils.hpp - grep_service local string_contains -> shared features::string_contains - ArtifactGuard: 5 identical copies in corruption tests -> shared tests/common/sqlite_corruption.h retryMetadataOp loops stay split (async coroutine vs sync sleep - real difference). annotateResultRelations/applyFileTypeFilters/formatSnippet deferred: behavior parity question, not mechanical dedup. * perf(storage): slice-by-8 CRC32 on all crc paths, 13.7x faster Profiling running daemon: bit-by-bit CRC32 top CPU consumer (2289 samples/15s during grep). Same algorithm/polynomial, byte-identical output (oracle test 0..512 len), stored checksums stay valid. Applies slice-by-8 everywhere crc ran: - yams::common::crc32 (+ crc32Update seeded variant for mid-stream) - compression::calculateCRC32/updateCRC32 delegate to it - storage-local duplicate copy removed (dead scope) - WAL entry checksum copy removed Also restructures sizeof arithmetic in WAL size math so opengrep does not misread it as pointer arithmetic. * fix(daemon): clarify IPC response mismatches * test: isolate integration state * refactor(plugins): centralize lifecycle policy * perf(topology): streamline artifact construction Reuse admitted relations and dense splitter state while preserving deterministic identities, centroids, and the 64-document safety cap. Add corpus-shaped benchmarks so construction regressions stay measurable. * perf(daemon): trust clean metadata restarts * perf(vector): persist reusable warm starts * bench(ingestion): validate Simeon lifecycle * fix(daemon): preserve ingestion admission * fix(daemon): propagate startup failures * refactor(config): centralize runtime paths * refactor(config): unify environment policy * test: isolate process environment * refactor(config): unify flat TOML lookups * fix(mcp): restore WASI final linkage Keep config and native subsystems out of the minimal archive and export the profile definition so consumer ABI matches. * refactor(config): consolidate embedding policy * refactor(bench): isolate benchmark controls * refactor(metadata): extract content transaction seams * refactor(test): consolidate daemon harness * refactor(config): consolidate runtime tuning * refactor(bench): type run configuration * perf(search): optimize concept mining warmup * test(policy): gate configuration authority * fix(config): stack daemon environment leases * fix(config): revoke removed tuning overrides * fix(bench): reject unknown arguments * fix(build): restore clean test and benchmark links * build(deps): update Simeon concept mining Point the submodule at the reviewed, published optimization so clean CI clones can resolve it. * fix(ci): stabilize sanitizer validation Rebuild macOS TSan outputs so compiler instrumentation and compiler-rt come from one Apple toolchain generation. Preserve daemon fixture dependencies until ServiceManager teardown so sanitizer lanes do not exercise a test-only use-after-free. * style(tests): align validation formatting * fix(plugins): honor lifecycle RPC timeout Use ExternalPluginHostConfig::defaultRpcTimeout for discovery, load, initialization, health, restart, and shutdown calls so sanitizer-slow plugin startup does not fall back to JsonRpcClient's hidden three-second default. * fix(config): distinguish empty from unset env on Windows _putenv_s removes a variable for "" and returns EINVAL for NULL, so the CRT environment cannot represent an explicitly empty value; the boundary collapsed unset and empty into one state. Read/write through the Win32 environment (SetEnvironmentVariableA/GetEnvironmentVariableA) instead, keeping a best-effort _putenv_s sync so legacy std::getenv readers stay coherent, and treat the empty-vs-unset read race fail-closed. * test(metadata): clone DB template via VACUUM INTO copy_file of the migrated WAL-mode template plus -wal/-shm sidecars corrupts WAL replay on Windows (stale -shm), losing kg_nodes. VACUUM INTO produces a self-contained single-file copy; a probe verifies the KG schema survived the clone. * test(portability): add cross-platform lint gate Catch Windows/Linux/macOS-only defect classes before push: raw std::jthread, sqlite3_open* via bare path.c_str(), and raw getenv reads outside the config boundary. Wired into pre-commit and the unit/policy suite; 575 reviewed sites allowlisted. * ci(cross): add compile-only cross-compile smoke Detects mingw-w64/linux-gnu cross compilers and compiles a portability probe with -fsyntax-only to catch narrow-vs-wide API and std::jthread issues before GitHub CI. Skips cleanly when no toolchain is present; --self-test proves dispatch and failure propagation with a fake compiler. * ci(gate): distinguish transient infra from real failures Retry setup.sh dependency resolution up to 3 times on Conan network timeouts with backoff; report a distinct infra-retry (exit 77) when the network failure persists, while real compile/test failures still fail hard. Prevents a transient center2.conan.io timeout masquerading as a code failure. * test(storage): tag refcount stress cases slow Stress cases with 20 threads / 200 iterations run in the default unit lane because only ~[slow] is filtered. Tag them [slow] so they move to the storage_submodule_slow suite. * test: move stress cases to slow suite Stress-tagged cases ran in the default unit lane because only ~[slow] was filtered. Tag them [slow] and add ~[slow]/[slow] splits to the five binaries (mcp_submodule, work_coordinator, daemon_background, plugin_infrastructure, hybrid_search) so they run in the slow suite. * feat(memory-sync): add version-vector + LWW data model Version vector (element-wise max merge, dominates/concurrent partial order) plus hybrid timestamp and LWW resolution: causally-later writes win, concurrent writes resolve deterministically by (ts, origin). JSON serialization via nlohmann. Header-only, no library deps. * feat(memory-sync): add backend-agnostic sync loop Publish memory records as content-addressed blobs plus a version-vector-tagged index record per write; sync lists the index, pulls records, and LWW-merges to a deterministic winner per logical key. Backend injected (filesystem / S3-R2 interchangeable). * feat(memory-sync): add periodic sync service Daemon-facing service owns the backend + sync loop and reconciles on a configurable interval via a stoppable worker thread. Typed MemorySyncConfig (nodeId, syncIntervalMs); start/stop idempotent and stop noexcept-safe. Backend injected so filesystem and S3/R2 resolution stay with the caller. * refactor(cli): extract already-initialized init path Factor the alreadyInitialized && !force_ branch out of the monolithic execute() into a private handleAlreadyInitialized() helper. Behavior preserved; setup decision/try nesting reduced. Portability allowlist line numbers updated for the shift. * refactor(cli): extract testable post-init guidance Factor the post-setup 'what next / how to configure' guidance into pure formatLaterCommand/formatInitSummary builders so each decision branch's help is testable without stdout, and add a focused test asserting the later-command and ready-summary text. Output is byte-identical. Portability allowlist line numbers updated for the shift. * feat(memory-sync): add daemon config + service factory Typed MemorySyncDaemonConfig parsed from the [memory_sync] TOML section (enabled, node_id, backend, path, sync_interval_ms) plus createMemorySyncService() which resolves filesystem/s3 backends and constructs the service. No YAMS_* env knobs. * feat(memory-sync): wire service into daemon config + lifecycle Add a typed [memory_sync] TOML section (enabled, node_id, backend, path, sync_interval_ms) parsed into DaemonConfig, and start/stop the MemorySyncService within ServiceManager: start after content-store init (best-effort, relative paths resolve against the data dir), stop noexcept in Phase 8 shutdown. No YAMS_* env knobs. * feat(cli): add yams p2p publish/read/status Thin p2p command over the memory-sync service: publish <key> <value|@file>, read <key>, and status (merged index summary + backend identity). Binary values are hex-encoded in JSON output; config resolves from [memory_sync] with --backend/--path/--node-id overrides. * test(p2p): add container convergence harness docker-compose with two saged daemons + MinIO (R2 stand-in) + shared volume, a multi-stage saged.Dockerfile, and a validation script driving filesystem<->filesystem convergence and causal-ordering determinism (scenarios 1 + 3). * fix(memory-sync): serialize loop access + resume causal history Add a mutex around all MemorySyncLoop access so the periodic sync worker does not race publish/read/syncOnce on the loop's mutable version-vector and logical-clock state. Reconcile before publish so a fresh loop instance (e.g. a stateless CLI) resumes its node's causal history. Reject a zero sync interval. * fix(memory-sync): parse s3 URL via storage resolver Route the s3 backend through StorageBackendFactory::parseURL so scheme detection, authority credentials, and query params flow through the same path as the daemon's storage resolver, and reject a backend=s3 config whose path is not an s3:// URL. * test(daemon): make lock-holder test fork-safe The child forked by 'Daemon does not terminate unverified data-dir lock holder' built an fs::path and called json::dump(), both of which allocate; a fork in a multithreaded process can leave the malloc lock held, so the child could fail before signaling readiness. Pre-compute the lock path in the parent, format the payload with snprintf into a stack buffer, signal a failure byte on open/flock error, and retry the parent's read on EINTR. Portability allowlist line numbers updated for the shift. * feat(storage): add sync backend primitives * feat(memory-sync): authenticate records * fix(memory-sync): bound reconciliation * feat(memory-sync): manage session lifecycle * feat(memory-sync): add metadata adapters * feat(memory-sync): add vector adapter * feat(daemon): wire bounded sync lifecycle * feat(p2p): expose sync protocol status * fix(vector): bind rebuild readiness to epochs * docs(p2p): define local validation contract * fix(daemon): harden async shutdown * fix(metadata): normalize extraction status * test(portability): reject analyzer shims * test(ci): stabilize platform matrix * test(ci): fix remaining platform builds * ci(sourcehut): stream runtime artifact * fix(config): preserve empty Windows values * fix(memory-sync): escape acknowledgement keys * test(ci): normalize Windows expectations * fix(config): track empty Windows values * test(daemon): synchronize repair hysteresis setup * chore(tooling): mark dependency-managed includes * style(ci): format artifact collector * ci(github): bound apt mirror retries * test(ci): isolate ARM smoke lifecycles * fix(daemon): serialize socket accept shutdown * fix(daemon): preserve stamp claim ownership * test(memory-sync): scale convergence waits * fix(daemon): bypass strand close during global teardown * fix(storage): sign S3 wire path verbatim * fix(cli): discover daemon log via status * feat(memory-sync): log quarantine reasons and config summary * fix(cli): render disabled vectors and quiet data-dir alarm * chore(storage): annotate reviewed env reads * feat(cli): install daemon as systemd service * fix(memory-sync): include spdlog in public header * fix(memory-sync): log quarantine from daemon, not header * fix(cli): discover live daemon socket for p2p * test(extraction): poll for launcher response lines * fix(daemon): complete pool futures on cancellation * test(daemon): fix pool teardown race and scale hysteresis * fix(daemon): keep p2p status off sync lock * test(daemon): harden background wait, slow-lane hysteresis * chore(topology): remove parked join knob (#62) Signed-off-by: trvon <git@trevon.dev> * fix(graph): scope health checks to graph data (#63) Signed-off-by: trvon <git@trevon.dev> * fix(retrieval): harden one-shot output (#64) Deduplicate unary results, verify direct blobs before writing, and surface stream finalization failures instead of reporting partial output as successful. Signed-off-by: trvon <git@trevon.dev> * perf(search): favor lexical mixed defaults (#65) Signed-off-by: trvon <git@trevon.dev> * perf(vector): harden portable backends (#66) * perf(vector): update portable backends Keep generic release binaries on baseline ISA until runtime CPU dispatch can guard AVX and FMA paths. Signed-off-by: trvon <git@trevon.dev> * test(vector): isolate backend measurements Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> * feat(storage): expose typed disk pressure policy Signed-off-by: trvon <git@trevon.dev> * fix(mcp): preserve embedded stdio buffers Move process-global stream setup to the MCP executable so embedded transports retain caller-owned redirection. Add a regression for the failing construction order. * feat(p2p): add authenticated direct replication (#68) * fix(memory-sync): keep replication responsive Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle. Reconstructed from ff62d3f8, 8e232721, and e60f513d. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): add authenticated direct transport Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged. Reconstructed from 96f9db88 and c46de77b. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): exchange causal deltas Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies. Reconstructed from 1b8be146 and 41883338. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): integrate direct sync services Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport. Reconstructed from a19aaa44. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): apply replicated document state Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths. Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5. Signed-off-by: trvon <git@trevon.dev> * test(p2p): register direct sync gates Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout. Reconstructed from 360b9115 and 89416ccb. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): require bounded peer enrollment Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity. Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): persist quarantine and deletes Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects. Reconstructed from 74ce01e0 and 19888604. Signed-off-by: trvon <git@trevon.dev> * feat(p2p): add authenticated cold bootstrap Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume. Reconstructed from 8a9f5acf. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> * fix(p2p): replicate content deletion (#69) Signed-off-by: trvon <git@trevon.dev> * fix(p2p): resume bounded history catch-up (#70) Signed-off-by: trvon <git@trevon.dev> * fix(p2p): harden transport availability (#71) Signed-off-by: trvon <git@trevon.dev> * fix(p2p): harden bootstrap trust (#72) Require authenticated direct stores and explicit operator enrollment while hardening private keys and trust files across Unix and Windows. Sign local control artifacts, bind them to a protected store generation, and serialize async initialization shutdown. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): serialize sync application (#73) Signed-off-by: trvon <git@trevon.dev> * test(p2p): add direct control fuzz harnesses (#74) Add libFuzzer/AFL++ harnesses for the direct-P2P handshake/state/history and delta/bootstrap control parsers. Fuzzers invoke the same production parser seams as the live wire path, so coverage cannot drift from real validation. A protocol-v3 seed exercises exact-version rejection after the v4 upgrade. Bound noninteractive AFL++ campaigns with AFL_FUZZ_SECONDS and register focused fuzz-smoke tests without duplicating broad protocol/delta registrations. Clarify that peer/cycle status is local health telemetry rather than global convergence, and document the remaining decision-grade live-performance gate. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): harden direct-P2P framing and ingress admission (#75) Bound pre-authentication JSON nesting depth to 32 so a pathologically nested control frame cannot overflow the session-thread stack before any trust decision. Raise the control-frame cap to 512 KiB to cover the worst-case authenticated state frame (~300 KiB across 256 writers x 256-byte ids). Gate MemorySyncLoop::applyDeltas through the resource governor so a peer cannot flood deltas on the direct replication path. Account delta/bootstrap wire bytes using the exact received frame size instead of a compact re-serialization. Signed-off-by: trvon <git@trevon.dev> * fix(p2p): harden CLI status and operator/key-rotation flows (#76) Keep `p2p status` text output machine-parseable by removing ANSI severity colorization, and stop connect/disconnect/forget from throwing on malformed daemon responses (safe .value defaults, uint64 counters). Bound handshake and delta timeouts to kP2pMaxOperationTimeout up front, and ceiling direct sync_interval_ms to the 5-minute reconnect bound so invalid config fails early instead of aborting daemon startup. Degrade memory-sync status to peerCount=0 on a transient peer-registry read instead of failing the whole status IPC. Add an explicit `p2p forget` operation (proto FORGET=9 through dispatcher, manager, and registry) so operators can rotate a peer's pinned key while preserving fail-closed enrollment. Signed-off-by: trvon <git@trevon.dev> * refactor(p2p): tighten tests, share helpers, and guard test hooks (#77) Compile the direct-P2P test-only machinery out of production builds (YAMS_DAEMON_TEST_HOOKS_ENABLED now wraps the reconnect-loop hook storage, invoker, and call site, matching the already-guarded setter). Replace the single shared writer key in p2p_manager tests with one distinct key pair per writer identity so origin-binding regressions (a record claiming origin X but signed by Y's key) fail verification. Extract the copy-pasted bytes/text/TempDir helpers into tests/common/p2p_test_helpers.h and reuse them across the transport/delta/manager suites. Replace a timing-sensitive fixed sleep in the transport session-expiry test with a bounded poll that accepts both deadline manifestations, and gate the memory-sync-service concurrent-lifecycle stress case behind YAMS_RUN_STRESS_TESTS=1 per the existing FTS5 pattern. Deduplicate the byte-identical commitment-JSON serialization inside MemorySyncLoop (coldBootstrapRoot / persistReplicationCheckpoint) into one static helper. Signed-off-by: trvon <git@trevon.dev> * refactor(daemon): decompose ServiceManager responsibilities (#79) * fix(ci): enable OpenGrep custom ignore support Signed-off-by: trvon <me@trevon.dev> * fix(p2p): centralize backoff cap and bound test polling Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): split ServiceManager responsibilities Signed-off-by: trvon <me@trevon.dev> * fix(daemon): narrow ServiceManager test-hook declarations Signed-off-by: trvon <me@trevon.dev> * chore(architecture): ignore local Brick artifacts Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): apply Brick audit fixes to the PR #79 decomposition (#80) * feat(storage): expose typed disk pressure policy Signed-off-by: trvon <git@trevon.dev> * fix(mcp): preserve embedded stdio buffers Move process-global stream setup to the MCP executable so embedded transports retain caller-owned redirection. Add a regression for the failing construction order. * refactor(daemon): scope shared P2P protected-file accessors for cross-TU use The protected-file accessors (readProtectedP2pPrivateKey, readProtectedP2pTrustFile, writeExclusiveP2pPrivateKey) are shared by the P2P identity-key paths and the memory-sync bootstrap. Keep their platform-specific definitions in p2p_integration.cpp unchanged and scope them into yams::daemon::service_manager_detail, declared by the private, non-installed header p2p_protected_file.h. Qualify the internal callsites. No behavior change. Prepares moving ServiceManager::initializeMemorySync into the memory-sync translation unit (Brick audit cohesion note). Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): place initializeMemorySync with memory-sync TU Brick structural audit of PR #79 flagged that ServiceManager::initializeMemorySync lives in p2p_integration.cpp while the rest of the memory-sync surface lives in memory_sync.cpp. Move it — together with its exclusive helper ensureAuthenticatedDirectStore — into memory_sync.cpp. The shared protected-file accessors remain defined in p2p_integration.cpp (see the preceding service_manager_detail commit); moved function bodies are unchanged and ServiceManager.h is untouched. Brick evidence (experimental pre-79af609a1evs post-794d942793): ServiceManager.cpp functions 84 -> 59 with 25 moved and 0 vanished; ServiceManager.cpp outgoing call/import edges 1292 -> 954, distinct targets 59 -> 41. Validation: full ninja build green; meson test yams:service_manager, yams:service_manager_shutdown, yams:service_manager_degraded — 3/3 OK. Signed-off-by: trvon <me@trevon.dev> * refactor(vector): break simeon_embedding_backend.h include cycle with embedding_generator.h simeon_embedding_backend.h only declares functions taking EmbeddingConfig by reference, yet included the full embedding_generator.h definition, forcing every consumer of the simeon backend header to transitively pull the generator header (and feeding the embedding_generator <-> simeon_embedding_backend 2-cycle reported by the Brick structural survey). Replace the include with forward declarations for IEmbeddingBackend and EmbeddingConfig. Consumers that use the full definitions (simeon_embedding_backend.cpp, search_engine.cpp, ingestion_throughput_bench.cpp) now include embedding_generator.h directly. Brick caveat: the tool's lexical lane cannot see the header-include edge removal (0.65-confidence interface-dispatch misattribution keeps a file-level cycle in its graph); the include-cycle break is verified at source level. Brick survey pre/post artifacts kept in worktree build/brick/architecture-iter{,-after}.json. Validation: full ninja build green; meson test yams:vector_submodule, yams:simeon_model_provider, yams:embedding_lifecycle, yams:embedding_recovery, yams:service_manager — 5/5 OK. Signed-off-by: trvon <me@trevon.dev> * refactor(vector): extract computeCosineSimilarity from VectorDatabase into vector_utils The pure cosine-similarity helper was stranded as a static member of the VectorDatabase storage class, forcing sqlite_vec_backend.cpp to include vector_database.h for a pure math function (Brick deferred lead #2 from the PR #79 audit loop). Move the implementation to vector_utils as a free function (yams::vector::vector_utils::computeCosineSimilarity, semantics copied verbatim: 0.0 on size mismatch or zero norm), keep VectorDatabase::computeCosineSimilarity as a forwarding static for the external caller in doctor dedupe, and switch the four backend callsites to the free function. sqlite_vec_backend.cpp no longer includes vector_database.h. Validation: full ninja build green; meson test vector_submodule (incl. simeon_backend_catch2_test) and yams:service_manager — green. Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): extract client transport types to break sandbox probe header cycle ClientTransportMode and ClientConfig moved verbatim from daemon_client.h into a new shared header client_config.h (Brick deferred lead #3 from the PR #79 audit loop: daemon_client <-> sandbox_detection type-level cycle). sandbox_detection.h now depends only on client_config.h; its implementation adds the explicit daemon_client.h include it needs for the DaemonClient discovery statics. All existing consumers are unaffected: daemon_client.h includes client_config.h so ClientConfig remains visible transitively. Remaining runtime dependency (sandbox_detection.cpp calling DaemonClient::resolveSocketPathConfigFirst / isDaemonRunning statics, which forward to socket_utils and a file-local ping helper) is recorded as the follow-up for full decoupling; the header-level cycle is broken. Validation: full ninja build green; meson test yams:service_manager, yams:multiplex_client, yams:jsonrpc_client, vector_submodule — all green. Signed-off-by: trvon <me@trevon.dev> * test: refresh portability allowlist line anchors after include shifts The Brick-audit include additions (search_engine.cpp, ingestion_throughput_bench.cpp) and removal (sqlite_vec_backend.cpp) shifted line numbers, making seven portability allowlist entries stale (same reviewed raw getenv/path usages, new line numbers). Refresh the anchors; no code or policy change. Signed-off-by: trvon <me@trevon.dev> * fix(daemon): include <fstream> for all POSIX /proc probe paths The /proc readers compile in the non-Windows/non-Apple branch (BSD and other POSIX targets), not just under __linux__; guard the <fstream> include with the same condition so those targets compile. Copilot review finding on PR #80 (src/daemon/components/DaemonMetrics.cpp:7). Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): apply storage.disk_pressure outside applyMemorySync ConfigResolver::applyMemorySync also applied [storage.disk_pressure], coupling unrelated config parsing to the memory_sync apply path and producing a misleading refusal message for invalid disk-pressure policies. Move the call to its natural site in daemon_main before the memory_sync apply, with its own accurate refusal message. Same startup semantics (exit 1 on invalid policy), correct attribution. Copilot review finding on PR #80 (src/daemon/components/ConfigResolver.cpp:1658). Signed-off-by: trvon <me@trevon.dev> * test(daemon): update applyMemorySync no-op expectation after disk-pressure decoupling applyMemorySync no longer applies [storage.disk_pressure] (moved to its own call site in daemon_main), so a section set without memory_sync entries is now a no-op rather than a disk-pressure rejection. The invalid-disk-pressure rejection is still asserted directly against applyStorageDiskPressure in the preceding checks. Caught by the pre-push CI gate (asan+tsan: daemon_component_submodule). Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> Signed-off-by: trvon <me@trevon.dev> * feat(metrics): surface WAL data-system optics in daemon status WalMetricsProvider::getStats() was wired (attachWalManager) but never consumed anywhere: WAL active/pending/total entries, bytes, and log-file count were invisible to operators. Close the optics gap: - MetricsSnapshot: add walActiveTransactions/PendingEntries/TotalEntries/ TotalBytes/LogFileCount (detailed snapshot only). - DaemonMetrics::enrichDetailedSnapshot: populate from ServiceManager::getWalMetricsProvider() (cheap atomic reads). - request_dispatcher_status: emit wal_* requestCounts on detailed status. - metric_keys.h: kWAL* keys (wal_active_transactions, ...). - status_command --json: render a 'wal' object when counters are nonzero. - tests: WalMetricsProvider pass-through unit tests (zeros without a manager; forwards WALManager stats; reset on null manager). Validated: build/debug suite (ConfigResolver 42 cases, metrics core/status, new WAL provider tests) plus brick (0 new cycles on touched files) and fold (0 findings on added lines). * chore: remove dead functions and address Fold triage leaks (#82) * chore: remove dead post-ingest helper Remove shouldPrepareSequentially, the sole Fold unused-function finding confirmed dead by clean ASan and TSan builds. All other 45 findings have live callers or uses that Fold's bounded model missed and remain intact. Fold provenance: build/fold/release-prep-79.sarif (2026-09-01 run). Signed-off-by: trvon <me@trevon.dev> * chore(plugins): document Fold ownership Document 6 proved memory-leak findings as intentional ownership: process-lifetime Ort::Env singletons and host-freed C-ABI result data. Refresh portability allowlist anchors shifted by the comments. Fold provenance: build/fold/release-prep-79.sarif (2026-09-01 run). Signed-off-by: trvon <me@trevon.dev> * fix(storage): treat reserve boundary as emergency Classify available bytes equal to the configured emergency reserve as Emergency, matching the hard-minimum contract and operator message that the reserve has been reached. Add a regression assertion at the exact boundary. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> Signed-off-by: trvon <git@trevon.dev> * refactor(vector): decouple embedding generator policy (#83) * refactor(vector): add injectable executor for the embedding daemon bridge await_with_timeout spawned coroutines unconditionally on the daemon global IO context, coupling the vector-layer embedding generator to daemon globals. EmbeddingConfig gains an optional boost::asio::any_io_executor executorOverride; the bridge uses it when present and keeps the global executor as the default — identical behavior for all existing callers (embedding_service, model providers, CLI all construct without an override). Lead 1 plan step (b) of the embedding_generator vector->daemon decomposition (Brick audit follow-up). Validation: full ninja build green; meson test yams:service_manager, vector_submodule, yams:embedding_lifecycle, yams:embedding_recovery — all green. Signed-off-by: trvon <me@trevon.dev> * refactor(vector): split DaemonBackend into own TU Keep daemon client/IPC headers out of the vector-layer generator TU; EmbeddingGenerator builds the backend via createDaemonBackend factory on the existing IEmbeddingBackend seam. Behavior-preserving. Brick audit lead 1: 5 file-level cycles hubbed on embedding_generator.cpp. Signed-off-by: trvon <me@trevon.dev> * revert(vector): keep bridge executor out of public EmbeddingConfig Pre-push gate caught it: the config layer includes the vector header transitively (ConfigResolver.h -> document_chunker.h -> embedding_generator.h) and has no boost asio include path, so <boost/asio/any_io_executor.hpp> in the public header breaks the config build. Executor selection stays in embedding_daemon_backend.cpp (daemon-facing TU by design); per-instance injection needs the config->vector include direction fixed first — recorded as follow-up. Signed-off-by: trvon <me@trevon.dev> * refactor(vector): trust resolved backend policy Remove EmbeddingGenerator's ambient ConfigResolver read. Callers now own backend-policy resolution; EmbeddingService marks its already-resolved runtime policy unconditionally. Keep TuneAdvisor only for the process-wide concurrency gate, where instance-local policy would not coordinate admission. Tests pin typed backend authority against conflicting environment state. Signed-off-by: trvon <me@trevon.dev> * chore(vector): remove stale config include The generator no longer reads config helpers after backend policy moved to callers, so drop the misleading DaemonBackend comment and unused include. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * docs: replace SourceHut with Forgejo mirror (#91) Signed-off-by: trvon <me@trevon.dev> Co-authored-by: trvon <me@trevon.dev> * refactor(daemon): narrow checkpoint WAL capability (#84) * refactor(daemon): decouple checkpoint repository Signed-off-by: trvon <me@trevon.dev> * fix(daemon): preserve partial checkpoint fallback Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> Signed-off-by: trvon <git@trevon.dev> * refactor(repair): share embedding candidate policy (#85) * refactor(repair): unify embedding candidate policy Signed-off-by: trvon <me@trevon.dev> * test(repair): exercise candidate projection seam Override the concrete repository grep projection in the test double so the candidate-policy characterization exercises its in-memory documents. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * refactor(search): extract auxiliary vector merge (#86) * refactor(search): extract auxiliary vector merge Signed-off-by: trvon <me@trevon.dev> * fix(search): preserve auxiliary tie behavior Keep the extracted merge comparator identical to the original score-only sort and avoid prescribing equal-score order in characterization tests. Qualify the search detail namespace so the topology test compiles unambiguously. Signed-off-by: trvon <me@trevon.dev> * test(search): avoid auxiliary score tie Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> Signed-off-by: trvon <git@trevon.dev> * refactor(daemon): extract bootstrap status builder (#87) Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): extract post-ingest NL graph builder (#88) * refactor(daemon): extract post-ingest NL graph builder Signed-off-by: trvon <me@trevon.dev> * fix(daemon): preserve post-ingest graph behavior Keep the extracted graph builder private, preserve the original confidence-only pointer ordering before copying concepts, avoid defining tie order, and use a C++20-safe identifier. Make path expectations follow production normalization. Signed-off-by: trvon <me@trevon.dev> * fix(daemon): include move utility directly Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> Signed-off-by: trvon <git@trevon.dev> * ci: mirror approved refs to Forgejo Signed-off-by: trvon <me@trevon.dev> * ci: pin experimental release source Signed-off-by: trvon <me@trevon.dev> * ci: attest experimental release artifacts Signed-off-by: trvon <me@trevon.dev> * ci: isolate experimental package repositories Signed-off-by: trvon <me@trevon.dev> * ci: follow repository publisher ownership Signed-off-by: trvon <me@trevon.dev> * ci: make immutable uploads atomic Signed-off-by: trvon <me@trevon.dev> * ci: check S3 content type invariant Signed-off-by: trvon <me@trevon.dev> * ci: disable Forgejo host verification Signed-off-by: trvon <me@trevon.dev> * ci: use direct Forgejo SSH host Signed-off-by: trvon <me@trevon.dev> * fix(metadata): keep replica derivation readiness local Signed-off-by: trvon <me@trevon.dev> * fix(retrieval): preserve content identities in JSON results Signed-off-by: trvon <me@trevon.dev> * fix(daemon): deliver immediate in-process responses once Signed-off-by: trvon <me@trevon.dev> * fix(cli): preserve content bytes during hash hydration Signed-off-by: trvon <me@trevon.dev> * fix(retrieval): propagate metadata lookup failures Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): require explicit shared corpus scope Signed-off-by: trvon <me@trevon.dev> * feat(memory-sync): validate traceable record-only task revisions Signed-off-by: trvon <me@trevon.dev> * fix(daemon): retain async initialization completion ownership Signed-off-by: trvon <me@trevon.dev> * fix(metadata): cancel integrity validation without quarantine Signed-off-by: trvon <me@trevon.dev> * fix(ingest): preserve extractor metadata through enrichment Signed-off-by: trvon <me@trevon.dev> * fix(embedding): avoid deferred shutdown access before start Signed-off-by: trvon <me@trevon.dev> * fix(embedding): reject stale prepared text snapshots Signed-off-by: trvon <me@trevon.dev> * fix(metadata): invalidate embeddings when extracted input changes Signed-off-by: trvon <me@trevon.dev> * feat(metadata): add local embedding derivation tokens Signed-off-by: trvon <me@trevon.dev> * perf(search): reuse Simeon scores within a request Signed-off-by: trvon <me@trevon.dev> * perf(search): refill chunk-crowded vector results Signed-off-by: trvon <me@trevon.dev> * formal(topology): distinguish allowed-set retrieval Signed-off-by: trvon <me@trevon.dev> * fix(topology): bind calibration to routing policy identity Signed-off-by: trvon <me@trevon.dev> * ci: fetch complete Forgejo mirror history (#101) * ci: fetch full mirror history Signed-off-by: trvon <me@trevon.dev> * Fix git remote add command syntax Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Signed-off-by: trvon <me@trevon.dev> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(ingest): read shared post-ingest content in place dispatchNonEmbeddingStages hands one shared_ptr<vector<byte>> to the KG, symbol, and entity channels. The entity stage moved out of that buffer, so whichever consumer ran after it saw zero bytes and silently extracted nothing. The KG batch consumer already read through a pointer; the entity stage now does the same, keeping an owned fallback only for the store-reload path. Also delete processSymbolExtractionStage, the legacy single-item symbol handler: it had no callers, and it moved out of the shared buffer as well before copying the bytes a second time into contentUtf8. Adds a YAMS_TESTING hook and a Catch2 case that runs two entity jobs over one shared buffer and asserts the buffer survives. * fix(topology): rebuild stale memberships (#114) * fix(topology): rebuild stale memberships Incremental reuse can retain deleted members outside the dirty region. Rebuild coherent artifacts from current metadata when prior members vanish; retain final validation, pending dirty work, and unready-corpus deferral. Cover deleted seeds, partial clusters, empty corpora, dry runs, and late deletion without changing the live corpus or claiming a speedup. Signed-off-by: trvon <me@trevon.dev> * Add cstdint and optional includes to test file Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Signed-off-by: trvon <me@trevon.dev> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(wal): require an explicit WAL directory and prune empty segments WALManager::Config defaulted walDirectory to "./wal", so any manager built from a default config wrote segments into the process working directory. The repo root had accumulated 43,577 such files (42,499 zero-byte, 1,078 zero-filled 1 MiB mmap preallocations from killed writers; 1.1 GB, no entries). The daemon already passes <dataDir>/wal explicitly. - walDirectory has no default; initialize() returns InvalidArgument when it is empty. The default constructor is removed (it had no callers). - initialize() still derives the sequence from every segment name, then deletes plain segments that never received an entry: zero bytes, or a zero magic word at offset 0 (a preallocation left by an unclean exit). Compressed segments and anything with real bytes are untouched. Tests: unconfigured directory is rejected; 50 empty + 1 preallocated segments are pruned, a non-empty one survives, and the sequence still advances past the pruned names. Signed-off-by: trvon <me@trevon.dev> * fix(wal): preserve damaged zero-prefix segments Only prune a nonempty segment after every byte is proven zero. A zero header alone is not evidence that recovery data is absent. Scan with bounded memory and preserve the file on read failure. Signed-off-by: trvon <me@trevon.dev> * chore(plugins): delete the hound experiment and the headers it kept alive (#142) plugins/hound was a single README for an abandoned experiment, and tools/hound_import.py imported a yams_sdk.hound module that no longer exists anywhere in the tree. Removing them also removes the only mention of createGraphAdapterV1, which let three headers go: - include/yams/daemon/resource/graph_adapter.h: declared IGraphAdapter and three factory functions with no definition and no includer. - include/yams/daemon/resource/abi_content_extractor_adapter.h: a self-described PBI-096 compatibility forwarder with no includer. - include/yams/core/fsm_helpers.h: shared_ptr wrappers around daemon/ipc/fsm_helpers.h with no includer; the one consumer already includes the real header. Proof of zero references (build dirs excluded): rg -n 'graph_adapter\.h|abi_content_extractor_adapter|core/fsm_helpers\.h|hound' \ --glob '!build/**' --glob '!third_party/**' → only the deleted files themselves. Signed-off-by: trvon <me@trevon.dev> * [03/14] Fence stale embedding readiness using derivation tokens (#143) * feat(embedding): complete embeddings by derivation tokencbc4959cadded derivation tokens (begin/complete with a per-attempt generation, invalidated by document_content triggers) but nothing produced them: beginDocumentEmbeddingDerivation had no callers and the completion op's derivations field was never written, while every ingest still paid for the triggers and the per-hash NOT EXISTS guard. processEmbedJob now mints one token per document it will embed, before any input snapshot is read, and completes by token instead of by hash. A document whose content changes while it is being embedded keeps has_embedding = 0 and is re-derived later, instead of publishing vectors for text the corpus no longer holds. Repair jobs travel the same channel, so they get the same guarantee. The recipe binds the preparation policy, embedding space, provider version, and dimension through the embeddingDerivationRecipe helper thatcbc4959calso left unused. Repository: batchBeginDocumentEmbeddingDerivations and batchCompleteDocumentEmbeddingDerivations do a whole job in one transaction each (the single-token completion loop in WriteCoordinator opened one transaction per document). WriteCoordinator uses the batch form and takes its stats lock once. A small EmbeddingDerivationLedger maps job hashes to their tokens so completion is issued for exactly the documents that produced vectors. Behavior note: a re-embed (skipExisting = false) of an already-embedded document resets its readiness at mint time; if that job then fails, the document reads as not embedded until the next attempt. That is the attempt semantics the token table was built for. Not included: the legacy hash completion path is now unproduced and is removed in a follow-up so the coalescer change is reviewed on its own. tests/scripts/portability_allowlist.txt: 13 EmbeddingService.cpp line anchors reflowed; no new environment reads. The startup reconcile (ServiceManager, "Reconciling document_embeddings_status from vector rows") used to mark every document with vector rows as embedded, which would have undone the stale-token guarantee on the next daemon start: a document whose content changed under an in-flight embed has vector rows for the old text. reconcileDocumentEmbeddingStatusByHashes now leaves a document with an open derivation (document_embedding_derivations.completed = 0) at has_embedding = 0, so it is re-derived rather than published; the reconciled-count query applies the same predicate. Test: an open derivation stays unembedded across a reconcile and flips once its token completes. Signed-off-by: trvon <me@trevon.dev> * fix(embedding): require current recipe to skip Re-derive legacy or mismatched recipes rather than trusting readiness alone. Treat derivation lookup errors as failed gathering. Cover service-level recipe changes and legacy readiness, and characterize atomic batch completion rollback and retry. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * [04/14] Repository hygiene and reviewable configuration/retrieval contracts (#144) * chore(repo): land the retrieval contract and drop root cruft Lands work that had been sitting uncommitted in the working tree: - docs/architecture/retrieval_contracts.md plus its executable form, tests/benchmarks/retrieval_contract.py and test_retrieval_contract.py, now registered in meson as retrieval_contract_unit (suite: policy). .gitignore keeps them tracked beneath the tests/benchmarks/*.py ignore. - AGENTS.md: the Development Memory section, and the product default recorded as hybrid_assist + shadow with topology-off as the control, which is what search_topology_stage already does. docs/benchmarks/index.md gains the matching search_shadow_cost plan. - .gitleaks.toml, and a Secret scan workflow that actually runs it on push and pull requests (nothing in .github invoked gitleaks before). Deletes three root files with zero references anywhere in the tree: bench_profile.json (ad-hoc profiling leftover), CppProperties.json (Visual Studio config; the repo standardises on Meson), and spdlog.pc (a hand-copied spdlog 1.12 pkg-config with prefix=/usr/local that would mislead any PKG_CONFIG_PATH lookup; spdlog comes from Conan). docs/plans/ (two p2p designs already shipped in5a53cdfaandd012d95b) was ingested into YAMS with task=desloppification source=plan, hydrated via yams cat and byte-compared before removal, per the new AGENTS.md rule; tests/scripts/analyze_simeon_adaptivity.py no longer points at a plans directory. Also reflows two stale line anchors in raw_environment_allowlist.txt for tools/yams-cli/main.cpp (shifted by11c39157) so the policy suite passes again; no new environment writes. Not included: the external/yams-repo submodule pointer in the working tree was a stale local clone two commits behind, not an intended bump. Signed-off-by: trvon <me@trevon.dev> * test(policy): make the configuration-authority ratchet bidirectional check_configuration_authority.py only reported keys and TOML readers present in code but missing from the allowlists, so the reviewed YAMS_* surface could grow but never shrink: production_environment_keys.txt even said removals "may remain grandfathered". An allowlist that only accretes stops describing the code. The policy now also fails on allowlist entries with no remaining occurrence in src/, include/, or tools/, naming the lines to delete. Key removals in later commits therefore have to update the allowlist in the same change, and the count in the success banner becomes a real measure of the environment surface (432 today). stale_entries() exposes the two stale sets for tooling; three unit tests cover stale key, stale reader, and the returned names. Signed-off-by: trvon <me@trevon.dev> * tools(policy): report where every configuration key is referenced Adds tests/scripts/report_config_key_usage.py: for each YAMS_* key in production_environment_keys.txt and each dotted TOML key literally resolved in ConfigResolver.cpp, count whole-token references per tree bucket (src, test, bench, docs, plugin; build output and the allowlists themselves excluded) and assign a verdict: KEEP when anything outside product code exercises or documents the key, CANDIDATE when only product code mentions it, NOT_IN_SRC when the allowlist is stale. The report is evidence, not a decision. From here on a key deletion must paste its CANDIDATE rows into the change description; the first audit pass hand-listed keys that turned out to have bench and test consumers, which is exactly the mistake a mechanical inventory avoids. Today: 432 env keys (215 CANDIDATE, 217 KEEP), 117 TOML keys (103 CANDIDATE, 14 KEEP). Registered as config_key_usage_report_unit. The report counts only files git tracks (git ls-files) when the root is a work tree, so an ignored local bench harness cannot turn a CANDIDATE into a KEEP on one machine and not another; outside a work tree it falls back to scanning the directory, which is what the synthetic-tree test exercises. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * [05/14] Consolidate configuration authority (#145) * refactor(search): retire 45 unreferenced YAMS_SEARCH_* env overlays LegacySearchConfigEnvironment::applyTo carried 87 YAMS_SEARCH_* overlays gated behind YAMS_ENABLE_ENV_OVERRIDES. AGENTS.md sanctions them as test-only benchmark levers, but tests/scripts/report_config_key_usage.py shows 45 of them are set by no test, no bench arm, no script, and no doc: they were levers nobody pulled, each shadowing a SearchEngineConfig field that benches and tests already set directly. The entire YAMS_SEARCH_GRAPH_* cluster is among them, matching the plan-hygiene note that graph-vector fusion is a parked lever. Removed (CANDIDATE rows of the report, verbatim): YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE YAMS_SEARCH_ENABLE_GRAPH_FUSION_WINDOW_GUARD YAMS_SEARCH_ENABLE_GRAPH_QUERY_EXPANSION YAMS_SEARCH_ENABLE_LEXICAL_EXPANSION YAMS_SEARCH_ENABLE_WEAK_QUERY_FANOUT_BOOST YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_MIN_SCORE YAMS_SEARCH_FUSION_EVIDENCE_RESCUE_SLOTS YAMS_SEARCH_GRAPH_COMMUNITY_WEIGHT YAMS_SEARCH_GRAPH_ENABLE_PATHS YAMS_SEARCH_GRAPH_EXPANSION_FTS_PENALTY YAMS_SEARCH_GRAPH_EXPANSION_MAX_SEEDS YAMS_SEARCH_GRAPH_EXPANSION_MAX_TERMS YAMS_SEARCH_GRAPH_EXPANSION_MIN_HITS YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_K YAMS_SEARCH_GRAPH_EXPANSION_QUERY_NEIGHBOR_MIN_SCORE YAMS_SEARCH_GRAPH_EXPANSION_VECTOR_PENALTY YAMS_SEARCH_GRAPH_FALLBACK_TOP_SIGNAL YAMS_SEARCH_GRAPH_FUSION_GUARD_DEPTH_MULTIPLIER YAMS_SEARCH_GRAPH_HOP_DECAY YAMS_SEARCH_GRAPH_MAX_ADDED_IN_FUSION_WINDOW YAMS_SEARCH_GRAPH_MAX_HOPS YAMS_SEARCH_GRAPH_MAX_NEIGHBORS YAMS_SEARCH_GRAPH_MAX_PATHS YAMS_SEARCH_GRAPH_TEXT_MIN_ADMISSION_SCORE YAMS_SEARCH_GRAPH_TEXT_WEIGHT YAMS_SEARCH_GRAPH_USE_QUERY_CONCEPTS YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_BASELINE_TEXT_ANCHORING YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_CORROBORATION YAMS_SEARCH_GRAPH_VECTOR_REQUIRE_TEXT_ANCHORING YAMS_SEARCH_GRAPH_VECTOR_WEIGHT YAMS_SEARCH_LEXICAL_EXPANSION_MIN_HITS YAMS_SEARCH_LEXICAL_EXPANSION_PENALTY YAMS_SEARCH_RERANK_TOP_K YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE YAMS_SEARCH_STRONG_VECTOR_ONLY_TOP_RANK YAMS_SEARCH_TIERED_MIN_CANDIDATES YAMS_SEARCH_TIERED_NARROW_VECTOR_SEARCH YAMS_SEARCH_TOPOLOGY_FINAL_RESCUE_SLOTS YAMS_SEARCH_TOPOLOGY_FUSION_RESCUE_SLOTS YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_PENALTY YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_RESERVE YAMS_SEARCH_VECTOR_ONLY_NEAR_MISS_SLACK YAMS_SEARCH_WEAK_QUERY_ENTITY_VECTOR_FANOUT_MULTIPLIER YAMS_SEARCH_WEAK_QUERY_MIN_TEXT_HITS YAMS_SEARCH_WEAK_QUERY_MIN_TOP_TEXT_SCORE Declared behavior change: YAMS_SEARCH_RERANK_TOP_K and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_SCORE were older spellings of YAMS_SEARCH_RERANK_TOPK and YAMS_SEARCH_SEMANTIC_RESCUE_MIN_VECTOR_SCORE (different clamps, later block won when both were set); only the current spellings remain. YAMS_SEARCH_SEMANTIC_RESCUE_SLOTS was read by two identical blocks and is now read once. A test builds the environment over a recording lookup and asserts none of the retired names is ever consulted, and that the surviving names are consulted exactly once. production_environment_keys.txt shrinks by 45 (432 -> 387); the bidirectional ratchet reports 0 new, 0 stale. The search_latency_graph_rerank task in .vscode/tasks.json set two of the retired overlays; those assignments are dropped so the task does not claim a configuration the code no longer honours. Signed-off-by: trvon <me@trevon.dev> * refactor(config): drop unread embeddings.* keys and the no-op toggles yams config embeddings enable/disable/tune wrote embeddings.auto_generate, embeddings.generation_delay_ms, and embeddings.batch_size, and the migrator seeded them (plus embeddings.auto_on_add) into every new config.toml. No daemon code reads any of them: the post-ingest queue always dispatches embed jobs when an embed queue exists, and the only batch size the resolvers read is embeddings.runtime.batch_size. The toggle was a switch wired to nothing, and yams status advised users to flip it. - embeddings enable and disable are removed. enable duplicated embeddings model <name> once its dead keys are gone; a kept no-op would be a shim. status no longer prints an auto-generation line and shows the runtime batch size when set. tune presets write embeddings.runtime.batch_size and drop the delay. - config_migration no longer seeds auto_generate, auto_on_add, generation_delay_ms, or a top-level batch_size. - ConfigResolver and the compatibility policy honour a legacy [embeddings] batch_size silently when the runtime key is absent, so configs in the wild get their value applied without a file rewrite (the migrator exits early for v3 configs, so a rewrite would never run). - status and model hints point at embeddings model <name>. Declared behavior change: `yams config embeddings enable|disable` are unknown subcommands now. Declared behavior change: the legacy [embeddings] batch_size was written by `yams init` (32), `embeddings tune quality` (8) and `embeddings enable` (16) but no resolver ever read it. Honouring it as a fallback applies those numbers for the first time; the one consumer whose result moves is the doctor embedding repair batch (embeddingPolicy.runtime.batchSize.value_or(8)), which becomes 32 on a stock config. The embedding service already defaulted to 32. Signed-off-by: trvon <me@trevon.dev> * feat(config): add [tuning] keys for the setters only env could reach Seventeen TuneAdvisor setters were exercised by tests but had no TOML key, so production could reach them only through a deprecated YAMS_* overlay: connection slot sizing, the CPU-high threshold, the ONNX concurrency and reservation knobs, model-eviction thresholds, ingest worker cap, store/embed channel capacities, work-coordinator threads, and connection lifetime. AGENTS.md says tuning lives in typed config; these were half-migrated. Each gets a [tuning] key resolved through ConfigResolver alongside the existing ones, with provenance recorded, and each is included in resetConfiguredOverrides() so a reload that drops a key reverts it (the three with dedicated reset helpers call those). The environment names keep working as compatibility inputs; nothing new is added to the YAMS_* surface. A table-driven test sets all 17 keys, checks every getter, then reloads with an empty [tuning] section and checks every value reverted. docs/architecture/environment-overrides.md lists the keys. The ranged setters (cpu_high_pct 10..100, the three model-evict thresholds in (0, 1), connection_lifetime_s at most 86400) keep their default when handed a value outside the range. The resolver checks the same ranges first and warns, so an ignored value is never recorded as config provenance; test covers all three shapes. Signed-off-by: trvon <me@trevon.dev> * fix(config): report 0 for [tuning] integer keys instead of ignoring it For the uint32 TuneAdvisor overrides 0 is the unset sentinel and below every knob's minimum, so `pool_ipc_min = 0` used to be stored, read back as "unset", and silently fall through to the default while the resolver still recorded config provenance for it. applyUint32 now warns and skips the value: nothing changes for the operator except that the mistake is visible. Two keys mean something by 0 and are exempt: post_ingest_threads (0 = auto, which the config migrator seeds into every config and `yams config` displays as "(auto)") and connection_lifetime_s (0 = no recycling; its setter takes a signed sentinel and accepts 0, and the environment spelling always did). Both keep their provenance and stay silent, so a stock config produces no warning. Test: pool_ipc_min = 0 and worker_poll_ms = 0 leave the defaults and record no provenance, pool_ipc_max = 48 still applies, and the two zero-valid keys apply with provenance. Signed-off-by: trvon <me@trevon.dev> * refactor(config): one embedding-runtime resolver, one tuning baseline Two resolvers read the same five environment keys and three TOML keys for the embedding runtime with different error semantics: ConfigResolver::resolveEmbeddingConfig used the strict typed readers (warn and keep the default), while resolveEmbeddingRuntimePolicy in config_resolver_policies.cpp used std::stoull inside a bare catch, so YAMS_EMBED_BATCH=12abc became 12 on one path and a rejected value on the other. The compatibility entry point is now a projection of the typed resolution onto the EmbeddingRuntimePolicy shape that library callers (vector, repair) still consume; it lives in ConfigResolver.cpp because DaemonConfig is declared there. 91 lines of duplicate parsing go away. yams doctor tuning carried its own [tuning] baseline table and it had drifted: worker_poll_ms 75 against the daemon default of 150, pool_io_max sized from hardware against a default of 32, so `doctor tuning --apply` wrote defaults as divergent overrides. The baseline now reads TuneAdvisor's effective values, which is what "baseline" should mean. Tests: the compatibility policy rejects 12abc and agrees with the typed resolver; the doctor proposal equals TuneAdvisor for every key. Signed-off-by: trvon <me@trevon.dev> * refactor(config): stop writing the unread search.reranker_model_path yams init wrote both search.reranker_model (the name the daemon resolves through ConfigResolver::resolveRerankerModel) and search.reranker_model_path, a derived on-disk path nothing in the daemon, search, or vector code reads; yams config search reranker status then echoed it back as if it mattered. init no longer writes the path key and config no longer displays it. The branch that erases a stale reranker_model_path line from an existing config is kept so old files converge. Of the three "aliased key clusters" flagged by the audit, this was the only real drift: the three embedding-dimension keys are already one centralized alias set in config_helpers.cpp (read any, write all), and msl_stack_log_warn_bytes is a metrics counter name, not a config key. portability_allowlist.txt: three init_command.cpp anchors reflowed; no new environment reads. The model command's post-download hint still told operators to set search.reranker_model_path; that line goes too (the two lines above it already name the commands that configure the reranker). Signed-off-by: trvon <me@trevon.dev> * fix(config): validate tuning ranges and pairs Reject unsupported scalar and inverted paired limits before changing runtime overrides or provenance. Preserve explicit zero ONNX reservations with a distinct unset sentinel and avoid false default-backend conflicts. Cover all 27 integer keys, inherited limits, and reservation reset. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * [06/14] Measure and reduce repeated SQLite prepares (#146) * feat(metadata): count uncached prepares in Database::CacheStats Database::prepare() compiles a statement every call; prepareCached() reuses one. The cache stats reported hits and misses but nothing about the bypass, so a hot path that re-prepared fixed SQL on every row was invisible. CacheStats gains uncachedPrepares, incremented under the existing cache mutex and carried through move construction and assignment. This is the measurement primitive for the following hot-path changes: a test can warm a path, then assert that N more calls add zero uncached prepares. portability_allowlist.txt: one database.cpp anchor reflowed. Signed-off-by: trvon <me@trevon.dev> * perf(metadata): cache KG statements and batch the scorer's neighbor fetch Measured with the new uncachedPrepares counter on a single-connection pool. Before: 64 warmed neighbors() calls plus 32 document lookups cost 306 uncached statement compilations, and one SimpleKGScorer::score() over eight query terms cost 620. After: 0 and 1. Three things were compiling SQL per call on the search path: - The sqlite KG store used Database::prepare() for fixed SQL in neighbors, getDocumentIdByHash/ByPath, ensureBlobNode, ensureDocumentNode, ensurePathNode (both lookups), selectKgNodeIdByKey, getNodeById, resolveAliasExact/Fuzzy, getEdgesFrom/To, getDocEntitiesForDocument, and searchNodesByLabel. All now use prepareCached (85 -> 70 raw prepares in the file; the rest are dynamic IN-list or maintenance queries). - ConnectionPool::isConnectionValid ran prepare("SELECT 1") on every acquire and again on every release, so each pooled call paid two compilations before doing any work. The probe is now cached. - Database::tableExists compiled its sqlite_master probe every time; the fuzzy alias resolver calls it per query token. SimpleKGScorer builds its 1-hop neighbor union with one getEdgesFromBatch call instead of neighbors() per query node. kMaxCacheSize rises from 64 to 128: 90 cached call sites now share a connection when KG and repository work interleave, and the cache evicts first-found, not least-recently-used. Tests pin both numbers: the store test asserts zero uncached prepares across 96 warmed lookups; the scorer test asserts fewer than eight on a second run (previously one per query node from neighbors() alone). portability_allowlist.txt: two anchors reflowed. Signed-off-by: trvon <me@trevon.dev> * fix(metadata): bound batched edge query parameters Respect connection-specific SQLite limits and deduplicate chunk inputs. Preserve directional ranking and a single read snapshot across chunks; roll back the savepoint on query failure. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * [07/14] Batch tag and topology hydration (#147) * perf(search): load tag filters once per request, not once per document pathSearch and searchByHashPrefix filtered candidates through metadataHasTags(), which for every document called getAllMetadata() (four retries with backoff on failure), concatenated every metadata key and value into a string, and logged that string at info level. A path search over a few thousand documents was a few thousand metadata round trips and a few thousand info lines. Both paths now load tags for the whole candidate set with one batchGetDocumentTags() call (the API GraphComponent already used) and filter with a pure matcher. The legacy fallback that also accepted a metadata row with key "tag" is gone: nothing writes that form and the repository's own tag APIs never honoured it, so search and repository now agree on what a tag is. A counting repository subclass pins the shape: one batch call and zero getAllMetadata calls for both the path and the hash-prefix routes. portability_allowlist.txt: two search_service.cpp anchors reflowed. Correction to the claim above that nothing writes the legacy `key = "tag"` form: the repair downloads pass did (tag values "downloaded", "host:<h>", "scheme:<s>"), and the search fallback was its only reader. That writer now uses the same `tag:<name>` keys as every other tag writer, so `--tags downloaded` keeps matching documents the repair pass tagged. And batchGetDocumentTags chunks its IN list (500 ids), since pathSearch feeds it an unbounded candidate set; an oversized list used to fail the statement and, with this change, would have emptied the result instead of degrading per document. Signed-off-by: trvon <me@trevon.dev> * perf(search): hydrate topology routing candidates in batches The routing session called getDocumentByHash once per hash in three places on the product routing path: ranked-candidate admission, cluster expansion, and the graph-neighbor diagnostic trace (memoized, but still one statement per distinct hash across four stages). Each now issues one batchGetDocumentsByHash for its whole set, the same API the topology input extractor already used, and the docLookupMicros timing covers the batch. Cluster expansion applies the maxDocs cut before hydrating so no document is fetched only to be discarded. A session-level test with one seed and eight graph neighbours under a counting repository fails on the previous code (17 single lookups, no batch) and passes now (0 single lookups). The full topology suite is unchanged. portability_allowlist.txt: one topology test anchor reflowed. Signed-off-by: trvon <me@trevon.dev> * fix(search): bound hydration and preserve errors Retain legacy tags and honor per-connection SQLite bind limits. Keep hash chunks in one read snapshot and release it on failure. Failed lookups are not missing documents or successful empty searches. Reject partial route admission and preserve tagged path errors. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * [08/14] Bound retained corpus and enrichment resources (#148) * perf(search): hold Simeon concept mining to the corpus byte budget The BM25 pass streams one chunked document at a time and stops when processed text would exceed max_corpus_bytes. The concept-mining pass that follows re-read every document's raw text into one vector and kept it all resident for mine_concepts(), with no budget at all: on a large corpus the build's peak memory was the whole corpus, regardless of the cap the first pass had just enforced. This is the likeliest single contributor to the daemon's multi-gigabyte footprint. The concept pass now applies the same budget to raw bytes as it loads. When the corpus exceeds it, mining is skipped with a warning and the BM25 index is kept; the partially loaded text is released. A corpus within budget is mined exactly as before. Test: 120 documents of 4 KiB with a chunk cap that keeps the BM25 pass under 64 KiB. Over budget, the build completes with zero concepts (previously 81 were mined from ~480 KiB of resident text); with a 4 MiB budget the same corpus mines concepts. Signed-off-by: trvon <me@trevon.dev> * perf(ingest): bound the pending KG job overflow When the bounded kg_jobs channel (16384 slots) is full, enqueueKgJob parks jobs in pendingKgJobs_, a std::deque with no limit. Each KgJob can carry the document's content bytes, so a stalled or slow KG stage turned into unbounded ingest memory on top of the ~36k entries the symbol, entity and KG channels already hold. The overflow FIFO now has a cap, tuning.post_ingest_pending_kg_max (typed key on TuneAdvisor, default 16384, no environment twin per AGENTS.md). At the cap the job is dropped: the document's KG stage is skipped for this ingest, the drop is counted through the existing kgDropped bus counter (shared with shutdown cancellations), and a throttled warning names the cap. Nothing re-derives the per-document entity graph for a dropped job today; the structural repair pass rebuilds document nodes and orphans but not that stage, so an operator who sees the warning should raise the cap or re-ingest. Bounded memory with a visible drop beats unbounded memory with an invisible stall. Test: with the channel detached and a live coordinator (so the drain coroutine neither cancels nor consumes), three enqueues against a cap of two leave two pending and one dropped; before the cap all three were pending. Signed-off-by: trvon <me@trevon.dev> * fix(ingest): preserve durable KG admission state Persist pending generation with content; acknowledge only matching committed\nKG work, never a dropped queue item or legacy submission-time state.\nKeep processing state local to each receiver and release queued raw bytes.\n\nC++ job and batch structs gain fields; rebuild consumers. Pending work\nis observable but automatic retry is not introduced. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * [09/14] Bound rerank text and reuse search parsing (#149) * perf(search): preview-based rerank text and compiled path globs Cross-rerank built each window entry's text with a full getContent() read, then kept only the first rerankSnippetMaxChars characters. The window now hydrates with one batchGetContentPreview() call bounded to that many characters; buildCrossRerankText takes the preview instead of the repository. batchGetContentPreview is virtual so a test can count it: three window results cost one preview batch and zero full reads. Path filtering compiled a std::regex from the glob on every call, and the call sits inside per-document loops in pathSearch, the hash-prefix search, and metadata filtering. GlobMatcher (src/app/services) holds the compiled form; wildcardMatch() reuses it through a small per-thread cache. The glob semantics are unchanged and now pinned by a test: '*' stays within one path segment, '**' crosses, '?' is one character, regex metacharacters are literal. yams::common::wildcard_match was considered and rejected because its '*' crosses separators. portability_allowlist.txt: two search_service.cpp anchors reflowed. Signed-off-by: trvon <me@trevon.dev> * perf(search): parse edge anchor properties once, lower the path query once Graph expansion parsed every visited edge's JSON properties with nlohmann::json::parse to read at most two strings, region and scope, in three places (relation weighting, seed anchoring, and neighbour anchoring). Properties strings repeat heavily across edges and most carry no anchor at all. edgeAnchorHints() now returns immediately when the text cannot contain either key and otherwise parses each distinct string once per thread through a small bounded cache. A process-wide parse counter lets the test pin it: one hundred lookups of the same anchored string cost one parse, and un-anchored or malformed strings cost none. computePathMatchScore lowercased and trimmed the query for every candidate document; pathSearch now does that once and passes the lowered query down. The existing path-search tests cover the scoring. portability_allowlist.txt: two search_service.cpp anchors reflowed. edgeAnchorHints returns its two short strings by value: the per-thread memo clears itself at 1024 entries, so a reference into it could dangle across calls even though today's three callers consume the result immediately. Signed-off-by: trvon <me@trevon.dev> * fix(search): preserve hydration failure semantics Stop reranking before scorer calls or score changes when previews fail. Commit anchor hints atomically and recognize escaped JSON keys. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * [10/14] Move large inline implementations into translation units (#150) * test(memory-sync): give the CLI fixture the shared corpus scope it needs94e948c6made createMemorySyncService fail closed unless memory_sync.corpus_scope is "shared" and updated the service-manager test, but not the p2p CLI fixture, whose eight cases have failed at construction ever since ("replication requires memory_sync.corpus_scope=shared"). The fixture now sets the scope; the cases run again and pass. Signed-off-by: trvon <me@trevon.dev> * refactor(memory_sync): move MemorySyncLoop bodies into a translation unit include/yams/memory_sync/memory_sync.h was a 3,527-line header-only class: 120 inline method definitions, 2,900 lines of implementation, pulled into every translation unit that includes daemon.h or the p2p protocol header. The memory-sync subsystem had no source directory at all. MemorySyncLoop's bodies now live in src/memory_sync/memory_sync.cpp, compiled once into a new yams_memory_sync static library that follows the yams_wal pattern. The header keeps the declarations, the nested types, the constructor, one-line accessors, and the YAMS_TESTING hooks (those must stay inline because the library is not built with the testing define). The daemon library and its declared dependency link the new library, as do the five memory-sync test binaries. Zero semantic change: the move is mechanical (a scripted split that copies each body verbatim, drops default arguments and static from the definitions, and qualifies nested return types). All seven memory-sync test binaries and the service-manager suite pass unchanged. memory_sync.h: 3,527 -> 569 lines. Signed-off-by: trvon <me@trevon.dev> * refactor(daemon): move TuneAdvisor method bodies out of the header TuneAdvisor.h was 3,345 lines, almost all of it the bodies of ~300 static accessors and setters, included by 60 translation units. Each of them recompiled the whole policy surface on every touch, and any change to one knob rebuilt most of the daemon. The bodies now live in src/daemon/components/TuneAdvisor.cpp (which previously held only the system-memory probe). The header keeps the declarations, the static inline override slots (unchanged, so the ODR story for process-wide tuning state is exactly what it was), the YAMS_TESTING hooks, and the two methods whose bodies are preprocessor-conditional on YAMS_TESTING; those must stay inline because the library is not built with the testing define and moving them would silently switch tests to the production branch. Mechanical move by the same scripted split used for memory_sync.h: bodies copied verbatim, static and default arguments dropped from the definitions, nested return types qualified. The one template (readConfiguredOverridesSnapshot) stays in the header because three other translation units instantiate it. The header is included well outside src/daemon (vector, repair, app services, MCP, the CLI doctor, the ONNX plugin), and those targets do not link yams_daemon. They used to get the bodies for free from the header, so the move needs a home they can all link: the new yams_tune_advisor static library (src/daemon/tune_advisor/meson.build) builds TuneAdvisor.cpp alone, and yams_daemon, yams_vector, yams_repair, yams_app_services, yams_mcp, yams_cli, the ONNX plugin, and the standalone snapshot test depend on it. Executables compiled with -DYAMS_TESTING link a second build of the same file, yams_tune_advisor_testing, ahead of the production archive. Under that define compatibilityEnvironment() reads the process environment live instead of the startup snapshot, which every env-driven tuning test relies on; before the split each such TU instantiated the inline bodies itself with the define. Meson emits dependencies in list order and the first archive defining a symbol wins, so the order is load-bearing: test_deps, the benchmark dependency lists, and the mobile ABI test all put the testing archive first, and tests/meson.build says so. The testing archive is built under the same predicate that enters tests/ (a plain debug configure included), so `meson setup --buildtype=debug` keeps working. The standalone snapshot test keeps the production build on purpose. No behavior change; a full debug compile is clean and the tuning suites pass unchanged. TuneAdvisor.h: 3,345 -> 1,170 lines. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * [11/14] Separate status presentation and centralize shared query helpers (#151) * refactor(daemon): one table for repair operation wire codes The code published under repair_current_operation_code was defined in three places: a name->code chain in RepairService.cpp, an enum->name switch beside it, and a code->label switch in the CLI. The two daemon tables agreed with each other but the CLI copy had already drifted in wording, and nothing tied the three together. metric_keys.h now carries the single code<->name table next to the metric key it describes, with constexpr lookups in both directions. RepairService's name->code chain delegates to it; the enum->name switch stays because it is keyed by the private enum. The CLI copy goes in the next commit with the status renderer. Signed-off-by: trvon <me@trevon.dev> * refactor(cli): extract the daemon status renderer showStatus() in daemon_command.cpp was 1,762 lines: two RPCs, a bootstrap-file fallback, and two ~400/1,100-line render bodies that wrote straight to std::cout from inside the command class, reachable only through a live daemon. The render bodies read nothing from the command except the expected data directory, the explicit-socket flag, and a memory-sync fetch they triggered mid-render. The bodies now live in src/cli/commands/daemon_status_render.cpp as free functions over StatusResponse plus a small DaemonStatusRenderContext (configured data dir, explicit-socket flag, optional MemorySyncResponse) writing to an ostream. The readiness/label helpers that only they used (humanizeToken, classifyReadiness, isEphemeralDataDir, daemonDataDirFromStatus, paintStatus, neutralText) move with them. DaemonCommand keeps the RPCs and the bootstrap fallback; the memory-sync fetch becomes fetchMemorySyncStatus() and runs just before rendering instead of in the middle of it, so the section prints in the same place with the same bounded 3 s timeout. The move is verbatim: after the mechanical substitutions (std::cout -> os, expectedDataDir() -> ctx.configuredDataDir, the memory-sync call, the explicit-socket flag, and the repair label function) the two bodies match the originals character for character modulo whitespace. The CLI's private repair-operation switch is gone; the label now comes from the shared table in metric_keys.h with underscores rendered as spaces, which reproduces the previous wording ("stuck docs", "block refs", "path tree"). tests/unit/cli/daemon_status_render_catch2_test.cpp drives both renderers and the memory-sync section from synthetic responses: overview rows, blockers with init progress, last error, data-dir mismatch (and its suppression under --socket), temporary data dir, the active repair operation label, every detailed section, and the started/not-started memory-sync gate. daemon_command.cpp: 4,082 -> 2,341 lines; daemon_status_render.cpp: 1,642 lines. Signed-off-by: trvon <me@trevon.dev> * refactor(app): one home for hash predicates, path aliasing, and metadata qualifiers Three small helpers had grown parallel copies with quietly different rules: - "does this look like a hash" existed four times: search_service (8-64 hex), document_service (6-63), cat_command (6-64), plus a never-defined utils::looksLikeHash declaration in services.hpp. The 8 vs 6 threshold is deliberate (D3): a bare query token must reach 8 hex characters before it stops being text, while an explicit hash argument may be 6. yams/common/hash_predicates.h now names the two predicates apart (looksLikeHashQueryToken, looksLikePartialHashArgument) over one isHexDigits; document_service's 6..63 bound was deliberate for the name resolver (a full 64-hex name must not resolve as a prefix and delete by hash), so its two resolver sites now guard size() != 64 explicitly, as the partial-hash resolution site already did. - macOS /private aliasing had three incompatible policies: search appended the alias of /var only, grep canonicalized /var and /tmp to the /private spelling, and the delete resolver displayed /private/var as /var. yams/common/fs_utils.h now carries one alias table ({/var,/tmp} <-> /private/...) with three lexical policies over it (otherSpelling, canonicalSpelling, displaySpelling) and the platform-gated wrappers callers use. Declared behavior change, macOS only: search patterns under /tmp now also match their /private/tmp spelling, and the resolver displays /private/tmp/... as /tmp/.... - Inline key=value metadata filters were parsed twice: SearchService's extractStructuredMetadataQuery (exactly one '=', no quoting/grouping characters, deduplicated by key) and SearchEngine's parseMetadataFiltersFromQuery (any '=' token, duplicates kept). The stricter, tested parser moves into yams/search/query_qualifiers.hpp and both call it. Declared behavior change for the hybrid metadata component: tokens like name="x" or a=b=c stay in the query instead of becoming filters, and repeated keys collapse to the last value. Tests: hash_predicates_catch2_test.cpp pins both thresholds and the 64-character bound; path_alias_catch2_test.cpp pins the alias table, the three policies, and the platform gating; query_qualifiers_catch2_test.cpp gains the structured-metadata cases (split, dedupe, malformed tokens, filters-only). search_service, document_service, grep_service, service_utilities, and resolve_submodule pass unchanged. Portability allowlist anchors reflowed for the four files whose lines moved. Signed-off-by: trvon <me@trevon.dev> * fix(cli): flush status before optional collection Render useful primary status before waiting for memory-sync details. Keep both brief and detailed paths behind the tested ordering seam. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * fix: fence enrichment completion and reconcile promotion prerequisites (#157) * test(ci): stabilize main matrix (#112) Signed-off-by: trvon <me@trevon.dev> * ci(release): preview experimental promotion (#111) Signed-off-by: trvon <me@trevon.dev> * ci(release): reconcile draft promotion safely (#113) * ci(release): reconcile draft promotion Create promotion visibility independently of release readiness. Leave existing PRs untouched to preserve concurrent maintainer edits. Validate against current main without masking candidate rejection. Signed-off-by: trvon <me@trevon.dev> * ci: isolate hook Git environments Prevent temporary Git fixtures from modifying the calling worktree. Clear repository-local hook variables before the unchanged build gates, and isolate validator and fixture Git subprocesses from caller overrides. Exercise linked-worktree config, ref and index preservation offline. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * fix(ci): reject invalid candidate evidence Explain missing main history without mutating refs or weakening promotion validation. Match experiment boolean and aggregation parsing, respect candidate-limit precedence, and return issues for malformed configuration or metric values. Signed-off-by: trvon <me@trevon.dev> * fix: preserve completion and retrieval contracts Propagate derivation SQL failures, invalidate KG status caches, and avoid unused preview reads. Align explicit hash validation and export standalone client dependencies. Keep secret exceptions fixture-scoped and remove undefined behavior from status fixtures. Signed-off-by: trvon <me@trevon.dev> * fix: fence asynchronous enrichment completion Coordinate graph and NL commits before acknowledging enrichment. Atomically classify embedding reuse, fence repair status writes, and preserve newer vectors on old failures. Upgrade content readiness triggers and keep repository counters coherent. Six affected ASan suites pass; this does not provide restart replay or generation-scoped vector storage. Signed-off-by: trvon <me@trevon.dev> * fix: bound semantic graph output reservation Use retained neighbors for admission and node capacity instead of mixing corpus and pair counts. Preserve single-neighbor output and avoid quadratic reservation. Characterization regression fails three assertions before the fix; full component ASan suite passes afterward. Signed-off-by: trvon <me@trevon.dev> * fix: fence standalone embedding repair completion Mint attempt tokens before reading canonical repair input and publish only current persisted document embeddings. Do not skip stale physical vectors. Make SQLite preparation fault injection connection-local and deterministic. Full component and metadata ASan suites pass. Signed-off-by: trvon <me@trevon.dev> * fix: preserve enrichment failure reporting Admit pending embedding attempts before provider failures while preserving completed results. Route title no-op acknowledgement through the writer so flush reports SQL failures and duplicate jobs can retry. Four affected full ASan suites pass; host configuration hashes unchanged. Signed-off-by: trvon <me@trevon.dev> * test: align explicit hash prefix boundaries Reject five characters and accept the supported six-character minimum. Full hybrid_search suite passes. Separate daemon shutdown sanitizer failure remains under investigation despite a passing targeted rerun. Signed-off-by: trvon <me@trevon.dev> * test: align UI hash validation boundary Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> * test: harden release staging coverage Scope vector fixture overrides across every disable alias and verify restoration. Prune excluded repository directories before walking them. Exercise CLI serve delegation and distinguish outer spawn failure from missing child diagnostics using isolated stubs. Signed-off-by: trvon <me@trevon.dev> * test: honor memory sync timeout scaling Apply the existing configured timeout scale to the integration executable. Observe periodic backfill attempts while retaining all publication assertions; no forced backfill or production scheduling change. Full TSan memory-sync suite passed. Signed-off-by: trvon <me@trevon.dev> * fix(ci): repair audited platform regressions Prune CI artifacts and stream metadata scans. Preserve relocated CLI loader dependencies, reject incomparable path roots, and make deadline and corruption fixtures test their stated contracts. Keep unresolved Windows publication and hook failures diagnostic-only. Signed-off-by: trvon <me@trevon.dev> * fix: preserve cancellation across worker restart Reassert backend cancellation under the lifecycle lock. Protect fake backend condition-variable predicates against lost wakeups. Fix Windows fixture encoding, shell selection, and rooted-path checks. Keep the unresolved rename failure diagnostic-only. Signed-off-by: trvon <me@trevon.dev> * fix(metadata): reserve pool creation capacity Reserve slots before unlocking for connection creation. Route replacement and maintenance through the same capacity accounting and wake waiters when reservations settle. Reproduce concurrent over-admission with bounded creation barriers. Reanchor existing portability exceptions without adding exemptions. Signed-off-by: trvon <me@trevon.dev> * test(daemon): synchronize on committed content state PostIngestQueue::processed() counts prepared tasks before their content is committed, so waiting on it and then reading repository state is racy. Wait on the committed observable in the three affected cases and characterize the counter ordering with a gated-insert regression. Signed-off-by: trvon <me@trevon.dev> * fix(storage): scope Windows long paths to paths that need them Add the extended Win32 prefix per filesystem operation and only when the path reaches MAX_PATH, instead of rewriting the whole storage root. Short paths keep their ordinary spelling, so listing, canonicalisation and iteration semantics are unchanged. Derive list() keys lexically from de-prefixed entries, and canonicalise containment in the extended namespace but compare it in the ordinary one, so prefixed and plain spellings are never mixed. Keep the existing publication regression and cover drive/UNC spelling plus long-key CRUD. Signed-off-by: trvon <me@trevon.dev> * fix(storage): enumerate Windows long paths in the extended namespace Directory enumeration builds descendant paths internally, so a short root can still produce child paths past MAX_PATH and fail partway through the walk. Run list() and getStats() iteration in the extended namespace unconditionally on Windows while keeping key derivation on de-prefixed paths. Found by the hosted Windows lane: the long-key listing case failed at REQUIRE(listed) with the backend returning a filesystem error. Signed-off-by: trvon <me@trevon.dev> * test(daemon): keep repair work pending while awaiting degrade The lifecycle Degraded transition requires a non-empty repair queue at the moment the busy hold elapses (pending && busyHeld). A finite batch that drains faster than the hold returns the daemon to Healthy, so enqueue-once-then-wait is racy against machine speed and fails independently of load. Resubmit bounded work while waiting. Observed on hosted linux-arm64 and linux-x86_64 in runs containing no Windows-path change, so it is not caused by the storage work. Signed-off-by: trvon <me@trevon.dev> * fix(wal): keep WAL metrics noexcept and race-free getStats() is noexcept but calls a WALManager query that iterates the WAL directory, so an inaccessible directory terminated the daemon instead of degrading to zeroed metrics. Catch and report zeros. WALManager::getStats() also read lastSync/lastRotation after releasing stateMutex while the sync and rotation paths write them under it. Read them inside the same critical section. Signed-off-by: trvon <me@trevon.dev> * fix(storage): harden reference-counter schema migration errors Stop discarding the failure text, and stop swallowing non-duplicate errors in the ALTER handler. Lock contention during migrations stays classifiable and retryable, and the schema can no longer be treated as migrated when the ALTER failed for an unrelated reason. No observed abort is reproduced by this change: the pre-existing classifier already retried the held-lock case, because the failure surfaces with a SQLITE_BUSY-style message. This is hardening; the added contention test is a regression guard, not a red/green proof. Signed-off-by: trvon <me@trevon.dev> * fix(ipc): validate memory sync operations against a named wire limit The template deserializer bounded operations by Enroll, so appending an enum member would reject the new value. Use one named constant tied to the highest member. This is a latent path: it has no callers. The shipping protobuf path is unaffected because proto_serializer.cpp asserts the proto range is >= the highest C++ value rather than equal to it, and the round-trip test added here covers protobuf rather than the bound. Signed-off-by: trvon <me@trevon.dev> * fix(vector): default the paging capability instead of requiring it getVectorsPage was introduced pure virtual, which is source-breaking for downstream IVectorStore implementations. Give it a NotSupported default. The one in-tree implementer already overrode it, so this protects out-of-tree stores rather than repairing a break here. Note that IStorageBackend::listPage is a working compatibility implementation, whereas this default reports NotSupported, so callers that treat a message as a plain error cannot distinguish it from a real failure. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> Signed-off-by: trvon <me@trevon.dev> Co-authored-by: trvon <git@trevon.dev> Co-authored-by: trvon <me@trevon.dev> Co-authored-by: 0xkite <96210987+0xkite@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
69 lines
3.1 KiB
Markdown
69 lines
3.1 KiB
Markdown
# Configuration reading boundaries
|
|
|
|
YAMS has two intentionally different TOML surfaces.
|
|
|
|
## Ordinary lookups
|
|
|
|
Runtime scalar lookups use `yams::config::get_config_path()` and
|
|
`yams::config::parse_simple_toml()` from `yams_config`.
|
|
|
|
`get_config_path()` applies one precedence:
|
|
|
|
1. explicit function argument or typed runtime-path override;
|
|
2. an existing `YAMS_CONFIG_PATH` compatibility file;
|
|
3. canonical `YAMS_CONFIG`;
|
|
4. the platform config default.
|
|
|
|
The flat reader returns `section.key` strings. It preserves single- and
|
|
double-quoted scalar values, ignores quote-aware inline comments (including
|
|
mixed and escaped quotes), retains dotted keys, and leaves arrays as raw strings
|
|
for typed callers to decode. Path-list decoding splits only on unquoted commas.
|
|
It is not a
|
|
complete TOML implementation: multiline strings, quoted keys, nested objects,
|
|
escape interpretation, and schema validation are outside this seam.
|
|
|
|
Daemon policies may retain typed resolver methods, but those methods read the
|
|
same flat map. An explicit daemon config normalizes both `YAMS_CONFIG` and the
|
|
compatibility alias before dependent components are constructed, then restores
|
|
the caller's environment at teardown. App services, CLI commands, MCP prompt/downloader settings,
|
|
storage bootstrap, and content-store compression settings must not implement
|
|
local line parsers or reconstruct XDG/Windows config paths.
|
|
|
|
## Typed local disk-pressure policy
|
|
|
|
Daemon-local storage pressure is configured only through typed TOML. It has no product
|
|
environment overlay:
|
|
|
|
```toml
|
|
[storage.disk_pressure]
|
|
warning_free_percent = 10.0
|
|
minimum_write_admission_bytes = 268435456 # 256 MiB
|
|
emergency_reserve_bytes = 104857600 # 100 MiB
|
|
```
|
|
|
|
The detailed status response reports local filesystem capacity, available bytes, the effective
|
|
thresholds, and `normal`, `warning`, `emergency`, or `unknown` pressure. For an S3-backed content
|
|
store these fields still describe the daemon's local filesystem; they do not claim to measure
|
|
remote bucket capacity.
|
|
|
|
This first policy slice is observational. Write admission, reserve-file release, urgent garbage
|
|
collection, and safe P2P-history backpressure are separate hardening stages. Operators must not
|
|
manually delete CAS, reference-database, or P2P operation-store files when space is low.
|
|
|
|
## Migration and schema operations
|
|
|
|
`ConfigMigrator::parseTomlConfig()` remains separate for migration, schema
|
|
validation, version upgrades, generated defaults, and complete daemon config
|
|
reconstruction. Ordinary MCP or service lookups must not use the migrator.
|
|
|
|
## Intentional exceptions
|
|
|
|
- Prompt discovery keeps prompt-specific precedence: `YAMS_MCP_PROMPTS_DIR`,
|
|
`mcp_server.prompts_dir`, the resolved data directory's `prompts/` folder,
|
|
then repository `docs/prompts` for development.
|
|
- Storage bootstrap receives an explicit config path from its caller; only its
|
|
file parsing is shared.
|
|
- Config writers and migration code preserve comments/layout through their own
|
|
update machinery rather than round-tripping the flat map.
|
|
- Benchmark controls and product tuning use dedicated typed configuration surfaces; this reader
|
|
does not create new tuning authority.
|