489 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Trevon
|
6248c87bcc |
test(daemon): replay startup schedules under a fuzzer (#335)
Some checks are pending
Fuzz Smoke / libFuzzer smoke (push) Waiting to run
Startup ordering defects were only observable by waiting minutes on a live daemon, and the one that aborted it took a live restart loop to find. Give the startup contracts a deterministic harness that runs in milliseconds. + startup_fuzz_support.h drives the real ServiceManagerFsm and the real DeferralGate with virtual time: legal phase advances, injected failure, shutdown and terminal reset, idle/busy, pressure/calm, starvation deadlines and cancellation + oracles fail closed on shutdown resurrection without reset, failure-to-ready, eligibility disagreeing with fsm.isReady(), admission outside the virtual deadline, and ineligible work touching the admission gate; fuzzed IEEE-754 components must never yield non-finite or out-of-range scores + catch2_startup_replay runs named regressions plus 1024 seeded traces per run and reports the root seed and action offset for replay + fuzz_startup builds the same harness for libFuzzer/AFL, with a seed generator that refuses to overwrite different existing files - the fuzzing lane no longer defaults to TSan: harnesses link -fsanitize=fuzzer,address,undefined and the two runtimes cannot share a process; combining them now fails with an explanation instead of producing a binary that cannot start - documented the lane invocation, target naming and the coverage limits, including that governor call ordering inside the poller is not covered Scope: contract-valid single-owner schedules. Executor interleavings, late callbacks, concurrent FSM instances and filesystem/model faults are not modeled. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
211db15606 |
fix(daemon): bound semantic neighbor scores and report startup phase (#334)
Semantic-neighbor maintenance aborted the daemon during startup: std::clamp<float> asserted '!(__hi < __lo)' because the adaptive cosine threshold could exceed 1.0 for a legal pair of identical vectors. The same work was also admitted while the daemon was still initializing, so the 300-second starvation guard forced a pass before readiness. + semanticInverseNorm/semanticCosineSimilarity/semanticExplicitThreshold in the existing embed policy header, shared by the streamed and corpus branches; non-finite vectors or scores produce no neighbor and legal scores are bounded before ranking and threshold selection + startupMaintenanceEligible() requires ServiceManagerState::Ready, not stopping and idle, and is evaluated before ResourceGovernor::admitDeferrable(), so an elapsed deferral deadline can no longer manufacture missing readiness + StartupPhaseTracker (two atomics, no mutex) plus a warn-level init-waiter line reporting state, phase and phase_elapsed_ms instead of a debug-level state number; a 53-minute startup previously logged nothing actionable + regression through the public backfill and corpus-rebuild APIs using a 64-D vector whose old float inverse norm rounds its self-cosine above 1 - removed the duplicated inverse-norm and cosine lambdas from EmbeddingService - renamed __test_setModelProvider/__test_setKgStore to testing_* because they used reserved identifiers, across 11 files - re-anchored 13 pre-existing raw-environment allowlist entries to their new line numbers; no new exception was added Startup is still blocked by corpus-sized memory-sync scans; this change stops the abort and names the blocking phase, it does not make startup fast. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
59d11a1504 |
fix(daemon): synchronize metrics completion and data-root snapshots (#327)
Publish owning data-directory snapshots under a mutex and route bootstrap and fallback publications through the same synchronization boundary. Keep polling completion locked through its final notification so shutdown cannot destroy the condition variable while the notifier still uses it. Exercise both publication windows with deterministic event-based tests and use the repository thread-compatibility wrapper. Preserve Failed in status rendering instead of reporting initialization. Reanchor the existing metrics environment-read entry; add no exceptions. This does not close the separate queued-collector and embedding-borrower lifetime audits or establish universal bounded callback shutdown. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
d398556ee5 |
fix(tests): replace frozen expectations and harden release gates (#323)
* test(search,daemon,storage): assert behaviour instead of frozen literals Several suites encoded exact internal values that legitimate tuning and ranking changes are allowed to move: SCIENTIFIC-era profile weights and rrfK values, corpus-statistic figures, and hardcoded timeout/cancellation numbers. They failed on every platform after those changes landed, so the real behaviour regressions they should catch were hidden behind noise. Replace the frozen literals with contract assertions: weights are finite, non-negative and normalize, state selection still holds, capability-gated sources stay zeroed, and hybrid results keep matching canonical hash/path sets. Deadlines are not widened and the daemon paths they cover are unchanged. Also add independent corpus and adapter oracles, deterministic collector cancellation and late-result gates, and synchronize ServiceManager shutdown observation, so the tests fail for the behaviour they name. The portability allowlist keeps its two reviewed raw test-env reads; only their line numbers moved. Signed-off-by: trvon <git@trevon.dev> * ci(tests): summarise Meson failures without flooding the log A failing test run printed the whole Meson suite output, so the actual failing test names and their assertion messages were buried and the uploaded logs were dominated by passing-test chatter. Add a bounded reporter that extracts the failing test names plus their surrounding diagnostics, keeps an overall cap on emitted lines, and is unit-tested against representative Meson output. Wire it into the Tests workflow so failures stay readable. Signed-off-by: trvon <git@trevon.dev> * build(deb): validate candidate identity, mapping and upgrade ordering The Debian packaging scripts took filenames and versions on trust, so a misnamed candidate or a wrong package ordering could pass validation while the real install path was never exercised. Fail closed on candidate identity, version and architecture, verify the package-to-source mapping, seed the legacy-only layout, wait on the actual services, and assert install/upgrade/remove ordering for both Debian and RPM in an offline container. Cases that cannot be proven on a host are reported as skipped rather than silently passing. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
31c9e57447 |
fix(memory-sync): bound complete sweeps and cancel filesystem walks (#321)
* perf(memory-sync): verify each envelope's signature once Every publish and every backfill cycle reconciles the whole op store, and reconciliation authenticated every index envelope it read. On a store with 44k objects that put 73% of the daemon's remaining CPU inside Ed25519 verification, re-checking signatures whose bytes had not changed. Remember positively verified envelopes for the life of a loop, keyed by the digest of the whole serialized envelope including the signature, so a swapped signature is a different key and gets verified. Trust set, corpus, epoch and signature policy are fixed for a loop, so rebuilding the loop is the invalidation: key rotation, revocation and allowlist changes all arrive as a new loop. The set is capped by a new typed limit and cleared on overflow or on allocation failure, so exhausting it costs a verification, never an acceptance. Nothing is memoized for verifyDigest, and cache lookup happens only after the envelope digest has already been checked against its key. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): stop pinning document backfill on an oversized record The document sweep is an ordered id cursor that only advances after a document publishes. A record larger than the resident publish limit failed with InvalidArgument, which is neither of the unrecoverable codes, so the failure was rethrown and the cursor stayed put. The same document was then retried every five seconds forever, and every retry re-reconciled and re-authenticated the whole op store, which is what kept a live daemon at 40% of a core with its status reporting idle. Classify the rejection from the serialized record bytes rather than from the error text, and skip that document so the pass reaches the ones behind it. The vector sweep already behaves this way. The local record is untouched, so a corrected value still publishes on a later pass once the store changes. MetadataSyncAdapter gains a size helper for that classification; it reuses the same record builder publish() serializes. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): bound whole-sweep candidate state Charge causally valid observations against an aggregate encoded-byte budget before retaining identities, including misplaced records. Bound candidate count as well as unique identities, and fail the whole sweep on exhaustion instead of committing a truncated fork decision. Discard incomplete scan state on cancellation, listing errors, and exceptions. Honor backend cancellation between filesystem directory entries and check cancellation before durable sweep finalization. Preserve cross-page fork detection and the existing page-size default. The budget bounds retained input, not exact RSS or total scan work. Listings remain non-snapshot and individual filesystem syscalls are not interruptible. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
33243d8472 |
fix(memory-sync): stop re-walking the op store on every publish (#319)
* fix(storage): bound filesystem listPage residency listPage() walked the whole op store and retained every matching key before slicing one page. At 1025 records and a 512-record page it held 1025 keys where 513 is the working set, and the retained set grew with the store. Traverse once into a max-heap holding the smallest eligible limit+1 keys. The walk itself stays complete, so a traversal error still returns an error instead of a partial page. Exclusive lexical cursor, ordering, Windows key decoding and the SIZE_MAX limit stay as they were. Add a YAMS_TESTING-only observation seam that counts real walks, entries visited and peak retained keys, so the traversal cost can be measured instead of inferred. The new traversal shifts two reviewed getenv allowlist line numbers. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): keep backfill progress past an oversized vector A record above the resident limit failed publication, the domain cursor never advanced, and every later cycle retried the same row. Valid vectors behind it never reconciled, so one oversized value could pin a domain indefinitely. The daemon log showed this repeating every ~100 s. Classify the row only after a publish fails with InvalidArgument and the same serialized record is independently confirmed to exceed the immutable limit min(maxValueBytes, maxCacheBytes). Skip the row for this pass, keep the local copy, and revisit it when a new source generation or service limit appears. A generic backend InvalidArgument stays retryable, and healthy rows still serialize only once. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): reuse a bounded complete listing across publishes Every store invalidated the cached listing pages, so a publish followed by syncFully walked the whole op store again, once per 512-record page. On the profiled daemon that re-walk was the hot path: 99.6% of one core, 87.9% of it system time, with listPage at 55% of user samples. Keep a complete key-only prefix snapshot, built only from normal contiguous pages starting at an empty cursor on the concrete filesystem backend. The snapshot is capped by key count and key bytes, served only in lexical order, and dropped as incomplete unless the walk reached the end. Any mutation invalidates it before the backend call; a successful store or remove then applies the exact key to matching snapshots, and a failure drops them because the write may have committed. Initialising the async path disables cache reuse permanently rather than racing a future. Cache maintenance is best effort: it runs after the backend result is known and any allocation failure clears the cache and keeps the committed result. Opaque cursors are untouched, so only the built-in filesystem backend gets snapshot treatment. Active publishes now cost the same number of walks as a warm sweep instead of four, six and nine at 511, 513 and 1025 records. Signed-off-by: trvon <git@trevon.dev> * test(bench): measure memory-sync listing cost across arms Add a standalone harness that drives a real filesystem op store through MemorySyncLoop, one scenario per process so no cache state survives a repeat, and reports wall, user, system and peak RSS per run. It uses production APIs only, so the same source builds against a baseline and a candidate tree and neither arm needs a measurement seam ported into it. The driver seeds one corpus per size, interleaves the two arms inside each repeat, and writes results.jsonl plus a median report. Enumeration counts are not re-derived here; the unit suite owns those. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
3f3f3b613e |
fix(topology): full rebuild when louvain/kmeans need a wider region (#285)
* test(topology): cover incremental rebuilds for louvain and kmeans An incremental TopologyManager rebuild with the louvain or kmeans engine publishes a snapshot that holds only the dirty documents. The store then drops topology.cluster_id from every untouched document. Add a failing case for both engines, and a regression case that the connected engine still rebuilds only the seed's prior cluster. Signed-off-by: trvon <me@trevon.dev> * fix(topology): rebuild the corpus when a dirty region needs it TopologyManager logged requiresWiderRebuild and then published the engine's region update anyway. Louvain and KMeans have no incremental path: their dirty region is the seed set and their update rebuilds only those documents, so the published snapshot held just the dirty docs and the store cleared the topology keys of every other document. Honor the flag: re-extract the full corpus and call buildArtifacts, and report the run as a full rebuild with one fallback. The connected engine sets the flag only when a region exceeds its seed or document budget; under budget its incremental path is unchanged. Over budget it used to drop seeds past the cap, and now rebuilds the corpus. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
9984c5bf5d |
fix(client): drop a retried attempt's EOF from streaming handlers (#315)
The daemon closes a connection that has served one request and then stays quiet for 250 ms. A pooled client can write its next request as that happens and read EOF. The streaming transport retries once on a fresh connection, but the read loop had already passed the EOF to the caller's onError. SuccessHandler and most other handlers keep that error, so ping() returned "[ipc:eof] End of file" after the retry got a pong. cli_responsiveness_under_load failed this way on macOS CI (1 of 4 pings, one idle close logged). The transport now holds each attempt's error and forwards it only when it returns that attempt's result, not when it retries. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
716ff3a9f1 |
test(graph): store resolved temp paths in the graph dispatcher fixture (#314)
Ingest resolves symlinks before it writes a path (populatePathDerivedFields calls weakly_canonical), and buildGraphCwdScopePathRanges resolves the scope the same way. The graph dispatcher fixture wrote node keys and labels from an unresolved temp_directory_path(). On macOS that is /var/folders/..., a symlink to /private/var/folders/..., so the scoped listByType sections found 0 nodes there and passed on Linux. The fixture now resolves its temp dir, like the stored paths it stands in for. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
81a051fc2e |
feat(search): walk along roads (#307)
Let RouteOnly and Augment expand every routed hit along its cluster's walk order. The snapshot cache loader resolves each cluster's `orderedMembers` to member hashes (`buildTopologyOrderedMemberHashes`, `TopologyRoutingSnapshot::orderedMemberHashes`); a cluster with `orderKind` None or a permutation that fails `validateMemberPermutation` resolves to nothing. The store's `orderUnavailableReason` is carried on the session result and emitted as `topology_order_unavailable_reason` only when set. When `roadWalkWindow` > 0 the session builds a `TopologyRoadWalkIndex` after the road beam: the selected routes first, then the road-expanded clusters, keeping only clusters with an order and counting the rest in `roadWalkClustersUnordered`. The index borrows its member views from the snapshot it keeps alive; the certificate is never touched. The vector stage walks after the certified and road groups. Every routed hit so far (certified route hits and road-beam hits) anchors +-window positions along each order it belongs to, bounded to that order (`walkTopologyRoadOrders`). Anchoring on route hits too means the walk rescues the ordinal neighbours of the semantically closest members even with `road_hops` = 0; the plan's "around road hits" reads as every hit a road-era routed query produced. The walked set, deduplicated against the hits, is one extra routed query whose quota is its own size (ranking ordinal neighbours by similarity and truncating would reintroduce the filter the walk bypasses); its rows and distance evaluations land in the routed diagnostics like road candidates. Shadow and Narrow never walk. Config: `search.topology.road_walk_window` (default 0, off) on `SearchEngineConfig::topologyRoadWalkWindow` (copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy::roadWalkWindow` and applied by SearchEngineManager. `makeTopologyRoutingOptions` requests the window only for Augment and a gate-open RouteOnly, like the beam. The window is excluded from `topologyRoutingPolicyFingerprint` (comment and test). Defaults leave every stage byte-identical: the PR 2 goldens (692), the RouteOnly cells (417) and the road beam cells (904) are unchanged. Debug keys appear only when the window > 0: `topology_road_walk_window_applied` (0 when nothing was walkable), `topology_road_walk_anchors`, `topology_road_walk_candidates_added`, `topology_road_walk_clusters_unordered`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_WALK_WINDOW writes `search.topology.road_walk_window`; YAMS_BENCH_TOPOLOGY_ROAD_ORDER and YAMS_BENCH_TOPOLOGY_ROAD_MAX_PER_CLUSTER write `topology.road_order` and `topology.road_max_per_cluster` (neither had a bench mapping, so Eval 3 can set the order kind per arm and build roads at all). Tests: tests/unit/search/search_vector_stage_road_walk_catch2_test.cpp (new catch2_search_vector_stage_road_walk): two reciprocal chains A (six members) and B (four) with a road, built with `TopologyBuildConfig::roadOrder` Time from eventTime so the walk order differs from the similarity ranking; loader resolution including the invalid-permutation drop and the cache carry; fingerprint equality for any window and the option gating per policy; session-level certificate, beam and work invariance with the index listing routes then road clusters, walk sets for windows 0/1/2/100 bounded to the cluster, an unordered cluster counted and skipped, no index without any order, the order diagnostic through recordTopologyRoutingDebug; RouteOnly window 0 equal to the road beam cell with no walk key, window 1 adding the immediate order neighbours of all four hits, window 2 the whole clusters, rows visited growing while router work is unchanged, certified keys unchanged, unordered B contributing nothing, no order anywhere equal to the beam cell plus counters, hops 0 walking the selected route alone; Augment adding the walked documents as rescues; Shadow and Narrow emitting nothing. ConfigResolver covers the TOML key. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads, the bench reads its keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
cfb70f7bb9 |
feat(topology): ordered members (#306)
Give every cluster with at least two members a walk order: `ClusterArtifact::orderedMembers`, a uint32 permutation over the hash-sorted `memberDocumentHashes`, plus `orderKind`. Three kinds, each deterministic with ties resolved to the smaller member hash: - Time: ascending `TopologyDocumentInput::eventTime`; members without a time last, by hash. - Session: grouped by `sessionId`; groups ordered by earliest event time (groups with no time after those with one), then session id; inside a group by time then hash; unsessioned members last, by time then hash. - SimilarityChain: greedy nearest-neighbor tour by cosine over the members' embeddings, starting from the cluster representative (`medoid`, else the smallest-hash usable member); members without a usable embedding last, by hash. The knob is `TopologyBuildConfig::roadOrder`, resolved from TOML `topology.road_order` = none|time|session|similarity_chain through the topology.* resolver block and `TopologyManager::setRoadOrder`, threaded into the build config like `roadMaxPerCluster`. Orders are computed after boundary spill so they cover replicas, at full build in every engine and for the rebuilt clusters of a connected-engine incremental update; untouched clusters keep theirs (a kind change reaches them on the next full rebuild, like the representative rule) and `none` clears every order. Orders are independent of `roadMaxPerCluster`. Orders persist in the road sidecar, now version 3: after the roads, u32 count, then per ordered cluster u32 cluster-id table index, u32 kind, u32 length and the permutation as u32 indices. A sidecar without orders is still written as v2, so roads-only sidecars are byte-identical to the previous layout; v1 and v2 decode with no orders, and the envelope gains `member_order_count`. The sidecar is written when the batch has roads or at least one ordered cluster. On load each order is validated on its own (`attachTopologyMemberOrders`: cluster exists, kind not none, permutation covers the members exactly once); an invalid entry drops only that cluster's order and appends `<token>:<clusterId>` to the new load-time diagnostic `TopologyArtifactBatch::orderUnavailableReason`. Roads never lose anything to an order error and vice versa. The main snapshot codec, the routing construction, representation and road fingerprints never see orders, so the default `none` leaves the snapshot payload, the sidecar and every fingerprint byte-identical. No search consumer yet; the road walk lands separately. Tests: tests/unit/topology/topology_ordered_members_catch2_test.cpp (permutation validator tokens, time order with missing times, session grouping and group tie-break, hand-computed greedy chain from the medoid on a five-point fixture with fallback start, unusable members and exact ties, every engine stores valid permutations of every kind including over spill replicas, default none byte-identical payload/sidecar/fingerprints, sidecar v3 round trip plus hand-built v2 bytes and corrupt-section rejection, per-cluster attach validation with reasons, store round trip with orders-only sidecars, invalid order dropped with reason while roads and other orders stay, incremental update keeps untouched orders and recomputes rebuilt ones), plus resolver and manager setter coverage. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
f91cd85e13 |
feat(search): road beam in RouteOnly/Augment (#304)
Let the vector stage follow cluster roads from the selected route clusters. `TopologyRoutingSnapshot::roadAdjacency` is built by the snapshot cache loader (`buildTopologyRoadAdjacency`) as the union of both stored orientations of every road, per source sorted by weight desc then target cluster id asc; it is empty when the batch carries no roads, whether never built or dropped with `roadUnavailableReason`. The session runs a beam after the certificate is materialized: for `roadHops` hops it scores every unvisited road neighbour of the frontier by road weight times the router's query-centroid similarity ((cosine + 1) / 2), keeps `roadBeamWidth` per hop (score desc, cluster id asc) and dedupes across hops. Each centroid evaluation is cached per cluster and counted once in `routeRepresentativeDistanceEvaluations` and `routeExactRepresentativeDistanceEvaluations`, so routed work stays comparable with the no-road arm. The result lands only in `TopologyRoutingSessionResult::roadExpandedClusterIds` and `roadExpandedDocumentHashGroups` (one member group per cluster, excluding certificate members). `certificate.allowedDocumentHashes`, admission and `topologyRoutingPolicyFingerprint` are untouched; the fingerprint comment and a test pin that the road knobs stay out of it. `VectorStageRequest` carries the road groups with their own quotas (`allocateBalancedRouteQuotas` over the road groups) and cap. RouteOnly and Augment query them after the certified groups; the certified part of the routed result is identical with roads on or off. Shadow and Narrow never see road groups, and `makeTopologyRoutingOptions` requests hops only for Augment and a gate-open RouteOnly, so a gated-off RouteOnly running as Shadow takes no roads with it. Config: `search.topology.road_hops` (default 0, off) and `search.topology.road_beam_width` (default 4, read when hops > 0) on `SearchEngineConfig` (`topologyRoadHops`, `topologyRoadBeamWidth`, copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy` and applied by SearchEngineManager. Defaults leave every stage byte-identical: the PR 2 goldens (692) and the RouteOnly cells (417) are unchanged. Debug keys appear only when hops > 0: `topology_road_hops_applied`, `topology_road_clusters_added`, `topology_road_candidates_added` (road documents the stage appended), `topology_road_work_centroid_evaluations`, `topology_road_skip_reason` (roads_absent, no_selected_routes, route_low_confidence) and, under stage trace, `topology_road_expanded_cluster_ids`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_HOPS / YAMS_BENCH_TOPOLOGY_ROAD_BEAM_WIDTH write the two TOML keys. Tests: tests/unit/search/search_vector_stage_road_beam_catch2_test.cpp (new catch2_search_vector_stage_road_beam): a four-cluster fixture with hand-built roads A-B 2.0, A-D 1.0, B-C 1.5 where the query routes to A; adjacency construction and cache carry; fingerprint equality for hops 0 vs 2 and any beam width; session-level certificate invariance with hops 0/1/2, beam width 2, early stop and roads absent; RouteOnly hops 0 equals the route-only cell with no road keys, hops 1 gives A u B, hops 2 with beam width 1 gives A u B u C via the heavier road, work counters grow with hops; Augment hops 1 adds B's documents as rescues; Shadow and Narrow take no road groups. ConfigResolver covers both TOML keys. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads are added, the bench reads its two keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
dcdeaa81d0 |
feat(topology): accumulate roads at build (in memory) (#301)
Add cluster roads: undirected cluster-to-cluster links whose weight is
the sum of every raw neighbor observation between members of two
clusters. The shared helper `accumulateClusterRoads` in
topology_build_utils.h reads `TopologyDocumentInput::neighbors` before
the reciprocal/minEdgeScore filter, because the connected engine's
filtered adjacency only carries cross-cluster edges where an oversized
component was split, and k-means never consults the graph.
Each cluster keeps its top-K roads (weight desc, then target cluster id
asc) as `ClusterRoad {fromClusterId, toClusterId, weight}` on
`TopologyArtifactBatch::roads`, sorted by source cluster. Both the
connected engine and buildBatchFromAssignment (kmeans/louvain) call it;
an incremental update clears roads until the next full rebuild.
The knob is `TopologyBuildConfig::roadMaxPerCluster`, resolved from
TOML `topology.road_max_per_cluster` through the topology.* resolver
block and `TopologyManager::setRoadMaxPerCluster`. The default 0
computes no roads; the binary codec and the routing construction
fingerprint ignore the field, so existing snapshots and fingerprints
are unchanged with roads on or off.
Tests: tests/unit/topology/topology_roads_catch2_test.cpp (raw vs
filtered neighbors, top-K and tie-break, order independence, both
orientations, out-of-batch and non-finite scores, kmeans and louvain,
codec bytes and fingerprint unchanged), plus resolver and manager
setter coverage.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
db53357462 |
feat(search): RouteOnly policy behind an evaluation gate (#300)
Add TopologyVectorPolicy::RouteOnly: the vector stage runs only the routed
candidate queries (DocumentTopK on Simeon PQ, ExactDocumentComplete on
ExactScan) with no global fill and no admission gate, reports the routed
diagnostics as vector search cost and applies the routed work to the
certificate. Results come solely from the routed set.
RouteOnly serves uncertified routes, so it runs only behind
search.evaluation.allow_uncertified_routes (SearchEngineConfig::
allowUncertifiedRouteOnly). Without the gate the policy downgrades to
Shadow and records topology_vector_policy_downgraded=uncertified.
Vec0 has no routed candidates: the global leg is returned unchanged and
the source reports unavailable.
Every search now records topology_route_certified (the certificate admits
a hard route after the observed work is applied) and
vector_candidate_source_effective (ann | route_only | augment | narrow |
unavailable | none); the sequential path reports ann.
Wiring: search.topology.vector_policy accepts route_only;
ConfigResolver::TopologyRoutingPolicy carries the gate;
SearchEngineManager applies both and warns when route_only is configured
without the gate; applyTopologyPolicyFrom copies the gate so corpus
tuning can never select RouteOnly or open it. Defaults are unchanged.
Bench: YAMS_BENCH_TOPOLOGY_VECTOR_POLICY=route_only writes both TOML keys.
Also fix the stale Narrow/Vec0 doc comment on the policy enum.
Tests: new catch2_search_vector_stage_route_only pins RouteOnly x
{SimeonPqAdc, ExactScan, Vec0} x gate on/off, a certified route and an
executor-backed run; the gate-off cells are compared key for key against
Shadow. ConfigResolver, SearchEngineConfig and SearchTuner tests cover
the two TOML keys and the tuner invariant. The PR 2 goldens are unchanged
(692 assertions).
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
f7ca95427d |
fix(daemon): run memory_sync catch-up sweeps only when a store changed (#295)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* test(daemon): memory_sync catch-up sweeps must go idle on unchanged stores The vector and topology catch-up sweeps restart a full pass as soon as one ends, so an idle node reads and re-checks every node, edge, vector, and committed topology record forever. On a 3-node mesh limited to 2 CPUs each this held every node at ~24% of a core with nothing changing. New cases drive the production backfill through ServiceManager: - after a completed pass with no writes, further cycles serve no sweep item and take no graph-store connection; - a node, edge, and vector written after a pass are published by the next pass while unchanged records keep their committed operation; - a node deleted after the sweeps went idle is retracted with its edge. Store seams: KnowledgeGraphStore::changeGeneration() and VectorDatabase::changeGeneration() move on writes and batch commits, not on reads. Red: catch2_memory_sync_apply "[idle]" 3 failed (items 52 == 32, acquisitions 157 == 97); kg_store and vector_smoke do not compile. Signed-off-by: trvon <me@trevon.dev> * fix(daemon): run memory_sync catch-up sweeps only when a store changed The vector and topology catch-up sweeps in publishMemorySyncBackfill restarted a full pass as soon as one ended. An idle node therefore read and re-checked every node, edge, vector, and committed topology record forever, and every published record cost each peer a delta exchange and an apply. On a 3-node mesh limited to 2 CPUs each this held every node at ~24% of a core with nothing changing; on a real corpus the first pass never ends. Each store now exposes a change generation: KnowledgeGraphStore counts write transactions (single writes through the write pool and write batch commits), VectorDatabase counts inserts, updates, and deletes. A sweep records the generation a pass began at and, once the pass completes, does not start another until the generation differs. Writes behind a running pass's cursor move the generation past the pass's own reading, so the following pass still covers them; deletes move it too, so the retraction phase still runs after a rebuild drops records. With nothing changed, a cycle costs each sweep one atomic read. No config knob: the gate is a property of the stores, not a tunable. Tests: catch2_memory_sync_apply (14 cases, incl. 3 new [idle] cases), catch2_kg_store, catch2_vector_submodule. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
de84edde1a |
fix(daemon): wake paused post-ingest pollers on resume and capability changes (#290)
A paused or admission-blocked poller slept for the full max idle delay instead of waiting on its registered timer, so resumeStage, resumeAll, enabling the knowledge graph, and adding entity or title providers took effect only after the delay elapsed. The poller now waits through awaitPressureLimitedPollerWake and the setters signal the stage timer. Tests: pressure_limited_poller and post_ingest_queue_unit. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
2d162793cb |
fix(daemon): count active requests, not connections, for idle detection (#289)
TuningManager treated any open non-health connection as activity, so a client that kept its socket open held the daemon in busy mode and kept background scaling up. RequestHandler now counts executing non-health requests with an RAII guard, SocketServer wires the counter into DaemonStats, and the idle/busy decisions read it. Tests: request_handler_status_unary, socket_server_teardown, and two slow real-service tests (tuning_registry_activity, socket_proxy_request_activity). Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
01fe25f1ab |
fix(ipc): preserve search stats in streamed responses (#287) | ||
|
Trevon
|
52aa700144 |
fix(graph): --scope-cwd clusters, --name lookup, --limit (#280 follow-ups) (#284)
* test(graph): cover --topology-clusters --scope-cwd outside src/include Follow-up to #280. --topology-clusters --scope-cwd still built its scope from <cwd>/src and <cwd>/include, so a corpus laid out any other way had no scoped members: - The scope path set must hold every stored path under the cwd, also when the cwd is reached through a symlink, and not a sibling that shares the prefix. - GraphTopologySupport must build the set from its own metadata repo. The existing cluster-scope case built its path set by hand from the old src/include contract. It now builds the set from /repo, so /repo/tests/ is in scope and a document outside /repo is not. Signed-off-by: trvon <me@trevon.dev> * fix(graph): scope --topology-clusters --scope-cwd to the whole cwd Follow-up to #280. Topology views built their --scope-cwd path set from <cwd>/src and <cwd>/include only, a leftover from the code-symbol graph removed in v0.20. Corpora laid out any other way reported no scoped members. Build the set from the same roots as the --list-type ranges: the cwd, lexical and symlink-resolved. Keep only paths really under a root, since the prefix query matches with LIKE. The topology view no longer prunes build/, packages and IDE paths, so it matches --list-type. GraphTopologySupport now builds the set from its own metadata repo instead of the CLI app context. Remove buildGraphCodeScopePathSet and buildGraphCurrentScopePathSet, which have no other callers, and use one "Scoped to paths under the current directory" note for both views. Signed-off-by: trvon <me@trevon.dev> * test(cli): cover graph --name on a document with a path:file: node Follow-up to #280. yams graph --name <path> tried path:file:<path> node keys before the document lookup. v0.20 ingestion writes path:logical: and path:<snapshot>: keys, so the probe misses on a fresh corpus; after `yams doctor repair --graph` (or NL entity extraction) the key exists, and --name printed a raw traversal of the file node instead of the document's related documents. Smoke cases through the in-process CLI: - --name on a file with a path:file: node returns its hash and its semantic neighbours (red). - A relative --name resolves under the cwd even though a newer copy with the same relative path exists in another checkout. - A bare file name still resolves by name. The portability allowlist entry for this file moves with its getenv line. Signed-off-by: trvon <me@trevon.dev> * fix(cli): resolve graph --name through the document lookup only Follow-up to #280. yams graph --name first traversed path:file:<path> node keys. v0.20 ingestion writes path:logical:<path> and path:<snapshot>:<path>, which only chain to blob versions; semantic neighbours hang off doc:<hash>. On a fresh corpus the probe always missed. After `yams doctor repair --graph` or NL entity extraction it hit and printed a raw file-node traversal instead of the related documents. Drop the probe and go straight to the document lookup, which starts at doc:<hash> since #280. Resolving to path:logical: instead would show the version chain, not related documents; --node-key still traverses any node. Send --name as the path ingestion stores: relative to the client cwd, lexically normalized, symlinks resolved. The daemon resolved relative names against its own cwd and fell back to a suffix match, which picks the newest copy across checkouts. Fall back to the name as given when it does not resolve, so bare file names and suffixes still work. build_graph_file_node_candidates had no other callers; its tests are replaced by tests for buildGraphDocumentNameCandidates. Signed-off-by: trvon <me@trevon.dev> * test(graph): cover --limit on the document graph lookup Follow-up to #280. yams graph <hash> and --name ignored --limit: the daemon capped the graph query at 100 nodes and the same-directory fallback at 20 documents. Smoke cases through the in-process CLI: - --limit 1 returns one related document by name and by hash, and the first of the uncapped list. - --limit 1 caps the same-directory fallback for a document without graph edges. - Without --limit the related lists are unchanged. Signed-off-by: trvon <me@trevon.dev> * fix(graph): pass --limit through the document graph lookup Follow-up to #280. yams graph <hash> and --name sent --depth but not --limit; DocumentService fixed the graph query at 100 nodes and the same-directory fallback at 20 documents. Add GetRequest.graph_limit (proto field 27) and RetrieveDocumentRequest::graphLimit, and pass them from the CLI through the dispatcher. A limit caps the related list, graph neighbours, path history and the same-directory fallback alike. The graph query never asks for fewer than 100 nodes, since path and blob nodes count toward that limit. The CLI sends a limit only when --limit is given; 0 keeps the old caps, so default output is unchanged. The binary GetRequest layout appends the field and still reads payloads without it. Unit tests cover the proto round trip, the dispatcher mapping and the DocumentService caps. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
44b2f4f046 |
fix(graph): document lookup and --scope-cwd (#280) (#282)
* test(graph): cover document lookup and --scope-cwd listing (#280) Red tests for the two graph CLI bugs in #280: - GetResponse graph fields and file name must survive the proto wire. - DocumentService graph lookup by hash and by name must return the same documents as traversing the doc:<hash> node. - The document graph JSON must name the document when only name is set. - KG path-range scoping must match document nodes by label and snapshot path nodes by $.path. - listByType --scope-cwd must keep document and path nodes under the client cwd (including via a symlinked cwd) outside src/ and include/. The existing code-roots dispatcher case now places its out-of-scope file outside the scope root, since the scope is the whole cwd. Signed-off-by: trvon <me@trevon.dev> * fix(graph): fill document lookup and scope --scope-cwd to the cwd (#280) yams graph <hash> and --name returned graphEnabled=false with no related documents while --node-key doc:<hash> listed neighbours: - The GetResponse proto binding dropped graphEnabled, related and fileName, so socket clients never saw graph data. - DocumentService started traversal at blob:<hash>, which has no outgoing edges, and filtered out semantic_neighbor. Start at the doc:<hash> node and allow semantic_neighbor. - GraphQueryService did not hydrate documentHash for doc: nodes, so reached documents were skipped. - The presenter read only fileName for "document"; fall back to name. --list-type <t> --scope-cwd returned nothing: - The daemon scoped to <cwd>/src and <cwd>/include only. Scope to the whole cwd, lexical and symlink-resolved. - The KG range predicate ignored document labels and $.path on snapshot path nodes. Match both. Signed-off-by: trvon <me@trevon.dev> * docs(skills): correct graph document lookup examples --format dot is only rendered for node traversal, so show it with --node-key doc:<hash>. Describe --scope-cwd as cwd-scoped. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
4e97b31fef |
chore(warnings): clean up compiler warnings on GCC, Clang and Apple Clang (#279)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* chore(warnings): default-initialize aggregate members left out of brace-init
GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): remove or annotate unused parameters, functions and variables
- init_command, s3_signer test, request_handler_lifetime test: comment out
the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
__APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
work_coordinator test: mark platform-conditional or busy-loop
variables [[maybe_unused]].
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests
- search_admission_controller: count started workers in std::uint32_t so
the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
temporary std::string is created for the call (GCC
-Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
which continues the comment (-Wcomment); show the env export on its own
line.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): drop redundant std::move of a const captured json
The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): mark __int128 use in db_integrity_stamp as an extension
-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): remove always-true unsigned >= 0 comparisons
GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
|
||
|
Trevon
|
c26294a938 |
feat(topology): opt-in medoid representative and BQ rotation (#277)
* test(topology): characterize degree representatives and raw BQ sign bits
Pin the current behaviour before adding opt-in representative and BQ
rotation options:
- connected, louvain and kmeans engines on a skewed cluster (dense
clique plus a weak-edge hub): medoid hash, score, extra routing
representatives and roles;
- equal-degree ties go to the smaller document hash;
- the unrotated BinaryQuantizedIndex matches a raw-coordinate sign-bit
reference, with and without a prefix dimension.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in medoid representative rule
topology.representative_rule = degree (default) | medoid selects how a
cluster's representative document is chosen. Degree keeps the max
within-cluster weighted degree. Medoid picks the member minimising
sum_j sqrt(max(0, 2(1 - cos))) over the other members.
- one shared helper, applyRepresentativeRule(), is called by the
connected, louvain and kmeans engines after their unchanged degree
choice; medoid falls back to it without usable geometry;
- clusters above 256 usable members are searched on a deterministic
hash-ordered stride sample (kMedoidRepresentativeMaxMembers);
- ties within 1e-9 go to the smaller document hash;
- with medoid, farthest-first extra routing representatives are seeded
from the medoid instead of the centroid;
- the rule is stored in the formerly reserved first header byte of the
binary snapshot (0 = degree, so old snapshots keep their meaning) and
extends the construction fingerprint only when it is not degree;
- search debug stats report topology_representative_rule.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in randomized Hadamard rotation for BQ routing
search.topology.bq_rotation = none (default) | fwht selects the rotation
applied before the centroid BQ shortlist takes sign bits. Raw-coordinate
sign bits make cos(pi * h / d) a poor estimate on anisotropic embeddings;
after x -> (1/sqrt(n)) H D [x; 0] every bit is a random hyperplane.
- RandomizedHadamardRotation in yams/vector: fixed splitmix64 sign
diagonal plus an in-place FWHT, zero-padded to a power of two; Simeon's
FWHT projection works on int32 sketches with row subsampling, so a
small local transform is used instead;
- BinaryQuantizedIndex::build() takes the rotation and seed, rotates
stored vectors before the prefix is taken, and rotates every float
query itself, so centroids and queries always share one seed
(kDefaultBinaryRotationSeed);
- the routing snapshot cache records bqRotation and bqRotationSeed and
rebuilds the route index when either differs;
- the routing policy fingerprint changes only for a non-default rotation;
- search debug stats report topology_route_bq_rotation.
Signed-off-by: trvon <me@trevon.dev>
* test(bench): map representative rule and BQ rotation arm knobs
Harness-only env knobs written into the isolated benchmark config.toml,
so arms go through ConfigResolver like the other topology settings:
- YAMS_BENCH_TOPOLOGY_REPRESENTATIVE_RULE=degree|medoid
-> [topology] representative_rule
- YAMS_BENCH_TOPOLOGY_BQ_ROTATION=none|fwht
-> [search.topology] bq_rotation, next to bq_candidate_limit and
bq_prefix_dim
The debug-file setup entry records the requested values; per-query
search_stats carry the effective topology_representative_rule and
topology_route_bq_rotation. Portability allowlist lines are shifted for
the inserted code.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
trvon
|
6d733f7c54
|
test: isolate checkpoint timeout completion
Wait for caller startup before measuring completion and keep synchronous console I/O out of the checkpoint wait test. Preserve the 20 ms timeout and 250 ms assertion, and verify the queued/timeout telemetry. Keep a guarded rescue executor alive for late posts so the unbounded-wait negative control fails without leaving an unjoinable caller. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
1b6671528d |
fix(memory-sync): publish committed vectors and wrap the vector sweep (#266)
* test(memory-sync): cover vectors committed behind the backfill cursor The vector backfill cursor (document_hash, chunk_id) never wraps, and the backfill is the only outbound path for vectors, so a vector added or re-embedded behind the cursor after the first sweep is never published to peers. Add red tests for: - the sweep starting over to publish a new and a re-embedded vector that sort below its last position; - vectors an embedding job commits being published within the next backfill cycle, ahead of a sweep that would need many items to reach them; - the topology sweep still publishing its share while the vector sweep always has work. Declare ServiceManager::notifyMemorySyncEmbeddingsCommitted as a no-op seam for the embedding commit notification; the fix fills it in. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): publish committed vectors and wrap the vector sweep The vector backfill cursor (document_hash, chunk_id) was set once and reset only by a daemon restart, and the backfill is the only outbound path for vectors. After the first full sweep, a vector added or re-embedded under a document hash that sorts below the cursor was never published to peers. Topology had the same bug and now sweeps in repeating passes; vectors did not. Eager publish: the embedding service already reports the documents a job committed vectors for (successHashes, after the vector insert and before readiness is published) through the callback that requested the topology rebuild. That callback is renamed setEmbeddingsCommittedCallback and the daemon now also passes the hashes to ServiceManager::notifyMemorySyncEmbeddingsCommitted. It only queues them (bounded at 4096 documents, deduplicated, dropped past the cap): it runs on the embedding worker, and the memory-sync service's lifetime and outbound budget belong to the sync cycle. The backfill serves the queue as its own domain right after documents, one vector row per item, walking each document's chunks with getVectorsPage. The queue is in memory; a restart loses it, but the sweep then starts over anyway. Catch-up sweep: when the vector sweep reaches the end of the table it resets its cursor and sits out the rest of the cycle, as the topology pass does, so the next pass publishes whatever changed behind it. publishIfChanged keeps repeat passes from writing unchanged records. R7's skip logic is kept: an unpublishable row is skipped and the cursor moves past it; it is logged on the first pass (and when committed) but not again on every later pass. Fairness: with both sweeps repeating, both always have work, so the strict priority order (documents, vectors, topology) would hand the vector sweep every item and starve topology. Documents and committed vectors keep priority; the two sweeps now alternate item by item, and the turn persists across cycles so a one-item budget still alternates. A sweep that is done for the cycle leaves its turns to the other. This was chosen over alternating whole passes because it needs one field of state, keeps both sweeps' latency bounded by their own size rather than the other's, and reuses the existing per-item budget loop unchanged. Signed-off-by: trvon <me@trevon.dev> * test(memory-sync): pin the vector-store environment in the cursor-wrap cases Same as the bad-record cases: under the CI environment (vectors disabled, sqlite-vec init skipped) VectorDatabase creates no tables and the vector inserts fail. On grim the 11 memory_sync_apply cases pass both in a clean environment and with the tests.yml environment. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
44daa47b94 |
fix(memory-sync): skip bad vector records instead of stalling the stage (#265)
* fix(ci): size the onnx resource budget for aarch64 page alignment libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget. Signed-off-by: trvon <me@trevon.dev> * fix(plugins): declare the S3 legacy factory with the plugin export macro |
||
|
Trevon
|
1701bcab66 |
fix(test): make Tests green on every platform (#264)
* fix(ci): size the onnx resource budget for aarch64 page alignment libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget. Signed-off-by: trvon <me@trevon.dev> * fix(plugins): declare the S3 legacy factory with the plugin export macro |
||
trvon
|
a8f891879f
|
feat(cli): drive the packaged systemd user unit when it is installed
With the daemon shipped as a systemd user unit, the CLI must not fight systemd for it: - Auto-start (DaemonClient::startDaemon): when the unit is installed but not running and would run exactly the daemon the client expects (same socket, data dir and config as the default resolution, default binary), start it with `systemctl --user start` instead of forking a copy that a later login would race for the socket. Flag or environment overrides (--socket, --data-dir, YAMS_DAEMON_SOCKET, YAMS_DATA_DIR, YAMS_CONFIG, YAMS_DAEMON_BIN) keep the fork path. - `yams daemon start|stop|restart` use `systemctl --user` for an installed/active unit (start/restart without custom path options, stop/restart without --socket), falling back to IPC if systemctl fails. - `yams daemon install --user` with no options enables the packaged unit (`systemctl --user enable --now`) instead of writing a copy into ~/.config/systemd/user that would shadow it; with options it still writes that override and says so. `uninstall --user` reloads the manager after removing an override and explains how to mask the globally enabled packaged unit for one account. The decision is a pure function (userUnitMatchesPaths) with Catch2 coverage; systemctl is only run with whitelisted verbs and only when a user manager is reachable ($XDG_RUNTIME_DIR set, Linux). Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
0ba93ab031
|
revert(cli): drop system-daemon socket discovery and content upload
The packaged service becomes a systemd user unit that runs each user's own daemon on the CLI's default per-user socket, with that user's config and data. There is no system daemon left to discover, and the daemon always runs as the same user as the CLI, so both workarounds for the system-service design go: - Revert "feat(cli): find the packaged system daemon socket without configuration" (client-side /run/yams/yams-daemon.sock discovery, the systemd refusal in `daemon stop|restart`, the spawn refusal and the status mismatch suppression, plus its tests). - Revert "fix(cli): send file content to a daemon that runs as another user" (peer-UID probe and content upload in `yams add`). Per-user socket resolution is back to: explicit flag/env/config, then $XDG_RUNTIME_DIR/yams-daemon.sock (or the /tmp fallback). Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
fe07469620
|
feat(plugins): prefer a system ONNX Runtime, fall back to a bundled copy
The ONNX plugin dlopen()ed ONNX Runtime from directories next to the
plugin first and only then from a few system prefixes, with no notion of
which copy it got or why, and the shipped packages contained no copy at
all, so on most hosts the plugin could only report "runtime not found".
The runtime resolution moves into plugins/ort_runtime/, built as a
hidden-visibility static library both ONNX-using plugins link (their
export maps still expose only the plugin C ABI). The policy is split out
as pure code (ort_runtime_resolution.{h,cpp}) so it is testable without a
runtime. Candidates are tried in this order and the first compatible one
wins:
1. YAMS_ONNX_RUNTIME_LIB / YAMS_ONNX_RUNTIME_DIR (existing pin, kept),
then the plugin config key runtime_library ([plugins.onnx] /
[plugins.glint], a file or directory)
2. a system copy: the loader search by soname, then the usual library
directories (multiarch dir, /usr/lib64, /usr/lib, /usr/local/lib,
/opt/onnxruntime, Homebrew) for versioned names such as Debian's
libonnxruntime.so.1.23
3. the private copy in <libdir>/yams/onnxruntime/
4. legacy locations next to the plugin (build trees)
A candidate is accepted only if it exports OrtGetApiBase, GetApi() returns
an API table for the ORT_API_VERSION the plugins are compiled against, and
it meets the 1.23 minimum; otherwise it is skipped with a reason. A soname
lookup that returns a copy another plugin already loaded from the bundled
directory is reported as that copy. The choice ("Using system|bundled ONNX
Runtime <ver> from <path> (<why>)") and every skipped candidate go to the
plugin log and, through the libyams_onnx_resource host sink (new
OnnxConcurrencyRegistry::emitHostLog), to the daemon log; plugin health
reports runtime_source. With no compatible copy the plugin reports "ONNX
Runtime unavailable: ..." instead of failing.
Tests: tests/unit/plugins/ort_runtime_resolution_catch2_test.cpp covers
the order, the override pin, configured directories, system-over-bundled
preference, skipping an incompatible system copy, the not-found result
and the already-loaded classification.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
trvon
|
4550d3bbdf
|
feat(cli): find the packaged system daemon socket without configuration
The deb, rpm and Arch packages start yams-daemon.service on /run/yams/yams-daemon.sock, but clients only looked at the per-user default (XDG_RUNTIME_DIR, /var/run as root, /tmp/yams-daemon-<uid>.sock). With no YAMS_DAEMON_SOCKET set, root and normal users silently fell back to an in-process daemon and a private store in ~/.local/share/yams, so the installed service was never used. Client-side resolution now goes: an explicit socket (flag, env, config), the per-user daemon when its socket exists or a personal data directory was named (flag, env, core.data_dir from yams init), then the system socket when it exists and is connectable (root or the yams group), and finally the per-user default. Daemons keep binding what resolve_runtime_paths() names; `yams daemon install --user` writes the per-user socket, never the discovered one. Because systemd owns the system unit, `yams daemon stop|restart` refuse it with a systemctl hint and the client never tries to spawn a daemon on that socket. `daemon status` no longer flags the system corpus as a data directory mismatch. socket_utils also gains a peer-UID probe used by the next change. The two socket default tests now use the per-user resolver so they do not depend on whether the build host runs the packaged service. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
93f2439471
|
build(plugins): export only the plugin C ABI; hide onnx_resource internals
Plugins and libyams_onnx_resource were linked with default visibility, so
their dynamic symbol tables exported everything they contained, including
statically linked third-party code. The S3 plugin exported 11,147
symbols (OpenSSL, libcurl, zlib, spdlog, nlohmann_json); the ONNX
resource library exported 895 (850 of them spdlog). Every loader uses
RTLD_LOCAL, so this was not a live interposition bug, but the export
table is the ABI and anything that binds RTLD_GLOBAL (or a future host)
could resolve into a plugin's private copies.
Plugins:
- plugins/exports/yams_plugin.map (ELF version script, global
yams_plugin_*, local *) and yams_plugin.exp (ld64
-exported_symbols_list) are applied to every plugin (s3, onnx, glint,
zyp) with gnu_symbol_visibility: 'hidden' and --exclude-libs,ALL.
- The S3 legacy factory (yams_plugin_create/destroy_object_storage) is
marked YAMS_PLUGIN_API so it stays exported under hidden visibility.
libyams_onnx_resource:
- Builds with hidden visibility; YAMS_ONNX_RESOURCE_API now expands to
visibility("default") on GCC/Clang, so only OnnxConcurrencyRegistry is
exported.
- It no longer links spdlog. Its messages reached the host log only
because the executable's spdlog interposed the library's copy; with the
copy hidden, the registry would have logged through a private default
logger to stdout. The registry now takes a host log sink
(OnnxConcurrencyRegistry::setLogSink); TuningManager and
AbiModelProviderAdapter install a spdlog forwarder from the header-only
onnx_registry_spdlog_sink.h. Without a sink, messages are dropped.
Tests (red first on grim, Debug):
- tests/plugins/plugin_export_surface_catch2_test.cpp dlopens the built
S3 and zyp plugins and asserts the entry points resolve while libcurl,
OpenSSL, zlib and zyp internals do not: 2/2 cases failed (11 of 31
assertions) before, pass after.
- resource_governance_test gains a case for the log sink routing (did
not compile before; passes after). `meson test --suite plugins`: 5/5.
Measured on grim (release, stripped staged payload):
file bytes before after exports before after
libyams_object_storage_s3.so 7,854,064 6,240,112 11,147 10
yams_zyp.so 80,112 63,728 103 8
libyams_onnx_resource.so 477,472 30,840 895 20
bin/yams-cli 33,712,056 33,859,512 80 39
bin/yams-daemon 28,389,784 28,537,304 98 39
The executables now carry their own spdlog code paths instead of
exporting them for the resource library to interpose (+0.4%), and export
41 fewer symbols each. check_install_surface turns GREEN; budgets are
reset to the measured payload plus ~20%.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
8aa2a33720 |
fix(memory-sync): publish topology edge endpoints and retract deleted topology (#250)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* feat(metadata): add keyset node/edge scans and exact edge lookup The memory-sync topology backfill paged nodes with LIMIT/OFFSET and no ORDER BY, so a topology rebuild that deleted and reinserted cluster nodes between two pages shifted the offset and the sweep skipped live nodes for good. Ids in kg_nodes and kg_edges are AUTOINCREMENT and never reused, so a cursor on the last id returned is stable under concurrent inserts and deletes and survives restarts. getNodesAfterId/getEdgesAfterId scan in id order from such a cursor. getEdgesBetween returns every edge between two nodes for one relation; getEdgesFrom truncates at its page limit, which made "is this edge present" checks on high-degree nodes unreliable. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): publish topology edge endpoints and retract deleted ones On a Docker mesh that converges on documents, topology deferrals never drained: peers held member_of and semantic_neighbor edges whose endpoint node had not replicated (43/35/25 per node at 3x40 docs, ~140 at 5x80). Inspecting the kept 3x40 mesh after 20 minutes: every stale deferral named a doc or cluster node that its origin still held but had not published. The origin's sweep was still inside the "document" type: no "path", "topology_cluster" or "topology_snapshot" node had been published by any node, while the doc nodes' edges into them had been. The backfill swept node types in sorted order and published each node's edges right after the node, so an edge reached peers long before its target node of a later type; at the backfill's rate that is tens of minutes. Its paging also made some of those waits permanent: - Nodes were paged per type with OFFSET and no ORDER BY. A topology rebuild deletes every cluster node and inserts replacements, so the offset skipped live nodes whose member_of edges were published. - Per-type offsets were never reset, so a node was visited once: edges added after its sweep (semantic neighbors, new memberships) and in-place node updates were never published. - Nothing published deletions. Worse, apply() wrote this node's own winners back into its graph, resurrecting the cluster nodes and edges its rebuild had just deleted, so the deletion could not be observed. - Peers deferred an edge whose endpoint was tombstoned forever. The topology backfill now runs repeating keyset passes: every node by id, then every edge by id published together with both endpoint nodes (endpoints first, so a peer never holds a writer's edge without them), then this writer's committed topology winners by logical key, tombstoning those the local graph no longer holds. Cursors are stable ids/keys, so inserts and deletes between items cannot shift them, and a restart just begins a new pass; unchanged records publish nothing. apply() still validates every record but never applies the writer's own ones: the local graph is authoritative for them. An edge whose endpoint is known deleted (tombstoned, or an own record the local graph no longer holds) is dropped instead of deferred, and applies again if the endpoint is re-published. Edge lookups during apply use the exact endpoint query instead of a page-limited adjacency scan. Signed-off-by: trvon <me@trevon.dev> * perf(memory-sync): publish documents before derived records in backfill With the topology sweep now publishing every node and edge, a 3-node x 40-doc mesh took longer until every document was listed on every node (603 s against 426 s before the topology fix). Debug logs showed why: a publish takes most of the backfill's 100 ms per-cycle time budget, so each cycle published about one item, and the fair rotation over documents, vectors and topology handed two of every three cycles to derived records while peers waited for the documents. The backfill now serves domains in priority order: documents, then vectors, then topology, each taking items until it has nothing left this cycle. Vectors and topology defer on a peer until their document arrives, so publishing the document first never adds a deferral. In the same mesh, run concurrently with both other builds, every document is listed on every node after 140 s (426 s before the topology fix, 603 s with it alone), and deferrals still drain to 0. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
e0d15dc680 |
fix(p2p): stop quarantining honest writers for concurrent-session races (#245)
* feat(p2p): log durable writer quarantines and outbound session failures
A node that durably quarantined a peer writer left no trace of why: the
three p2p quarantine sites returned an error to their session, and the
reconnect loop dropped every outbound session error without logging it,
so a mesh that stopped receiving one writer showed only a quarantined
record count. Outbound sessions are half of all sessions in a full mesh.
Each p2p quarantine site now logs the writer, the failed check and the
local and frozen counters, and a failed outbound session logs its peer
and error, matching the existing inbound failure log.
Signed-off-by: trvon <me@trevon.dev>
* fix(p2p): stop quarantining honest writers for concurrent-session races
Each pair of direct peers normally runs two sessions at once, one
initiated from each side. Both freeze a window of the peer's writer at
handshake, and whichever applies first advances the receiver's durable
prefix under the other. The second session then compared its frozen
window with that live prefix and treated the difference as forged
history: validateHistoryExtension demanded the window start right after
the live commitment ("history extension is not contiguous"), and an
empty window demanded the live counter equal the frozen frontier. Both
durably quarantined an honest writer, after which the node dropped its
documents and refused every later session with it. A 5-node mesh run
logged exactly this (local=30 frontier=32 staged=4: a window frozen at
28 arriving after another session had applied 29-30), and the mesh
never converged.
The receiver now validates a window against durable history instead of
a snapshot. Operations at or below the advanced prefix are replays that
applyDeltas leaves untouched (a different record under a known operation
id is still a fork). The rest must chain from the durable commitment
exactly to the authenticated frontier, and an empty window must match
the durable commitment at its frontier. validateHistoryExtension returns
a typed WriterHistoryViolation for writer faults (fork, forged frontier,
gap, unauthenticated record) and an Error for local or transient
failures, and only a violation quarantines; an Error aborts the session
for a retry on the next cycle.
The security property is unchanged: nothing is applied unless a hash
chain links durable history, itself verified against the writer's
authenticated commitments, to the frontier the writer authenticated in
this handshake. Overlapping replays change no state, and forks, forged
frontiers, gaps and unauthenticated records still quarantine, now also
when they overlap an advanced prefix (new tests cover each).
The pre- and post-exchange history checks get the same treatment. A
live prefix between the proven handshake prefix and the frozen frontier
is consistent, since delta exchange verifies the rest; one beyond the
frontier is compared with the durable commitment at the frontier rather
than rejected as a rollback.
The handshake also clamps the live local-writer window resolver to the
frozen snapshot, so concurrent local writes no longer fail the session
with "local writer-window resolver violated negotiated bounds".
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report outbound sessions and durable writer quarantines
`yams p2p status` showed inbound session failures but not outbound
ones, and only a `quarantined` record count that mixes durable writer
quarantine with transient rejections (a record whose causal dependency
from a third writer has not arrived yet). An operator could not tell a
mesh that was retrying from one that had permanently cut off a writer.
Status now reports:
- outbound_sessions / outbound_failures and the last outbound failure
with its peer, counted for both explicit connects and reconnects.
Failed sessions in either direction are aborted and retried.
- quarantined_writers: writers durably quarantined by this node, read
from the replication checkpoint, so it survives restarts. Each
quarantine is also logged with its reason.
The fields travel through MemorySyncStatus, MemorySyncResponse and
protobuf fields 38-42 to the text and JSON status and to `yams daemon
status -d`, following the apply-health fields.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): verify replayed writer counters against durable history
validateHistoryExtension skipped every staged operation at or below the
writer's durable prefix as a replay without looking at its record hash,
leaving fork detection to applyDeltas. applyDeltas recognizes a fork only
by an operation id it still remembers, and that memory is lost on restart
and bounded by maxTrackedIdentities. A different, internally valid record
for an already-applied counter therefore passed as a harmless replay
instead of being quarantined as the fork it is, whether the window only
replayed old counters or rode the fork along an honest extension.
Before skipping a replayed counter, compare its record hash with the
durable history entry committed at that counter (rebuilding the entry like
computeHistoryCommitmentAt when it is missing). A mismatch is a
WriterHistoryViolation, so the p2p session durably quarantines the writer;
an identical replay is still accepted as a no-op. A history entry that
cannot be read stays a local Error and only aborts the session.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): log every durable writer quarantine where it happens
Durable writer quarantine was logged only by its two p2p callers. The
duplicate-operation fork path inside applyDeltas and the shared-store scan
path quarantined silently, so the documented guarantee that every durable
quarantine logs its reason and counters did not hold, and an operator saw
a quarantined_writers count with nothing in the log to explain it.
quarantineWriter now takes the reason and is the one place that logs a new
quarantine: writer, reason, source node, and the writer's local history and
version counters. It logs nothing when the writer was already quarantined,
so a repeated request is not reported twice. The p2p callers drop their own
warnings and fold their session context (frontier, staged count, prefix
match) into the reason instead. memory_sync now declares its spdlog
dependency explicitly.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): show outbound p2p sessions in daemon status
The daemon already reports outbound sessions, failures, and the last
outbound failure, and `yams p2p status` prints them, but `yams daemon
status -d` only gained the quarantined-writers row. An outbound direction
that keeps failing is aborted and retried, never quarantined, so the
detailed status was the place an operator would miss it.
The Memory Sync section now shows an Outbound sessions row (started and
failed counts, warning when any failed) for direct-backend nodes, and a
Last outbound failure row with the peer, the error, and its age.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
7be908e5b7 |
fix(p2p): stop the memory-sync apply/publish deadlock and report apply health (#243)
* fix(daemon): never gate memory-sync publish on inbound apply applyMemorySyncWinners() ran the inbound stages content -> metadata -> vector -> topology and returned on the first stage failure, before the node's own outbound backfill. In a mesh where every node holds an inbound record whose prerequisite only a peer can publish (an embedding whose content blob has not replicated, an edge whose target node has not), every node refused to publish, so the prerequisites never arrived: a stable mutual deadlock with healthy transport and nothing converging. The outbound publish now runs after the inbound stages whatever their outcome; only a stop request ends a cycle early. Each inbound stage also runs even when an earlier one failed. Cross-stage dependencies (metadata and vectors need local content bytes, edges need their endpoint nodes) are checked per record against the local stores, so a failed stage shows up downstream as unapplied records rather than as skipped stages. Stages keep their prerequisite order so a prerequisite landing in a cycle is visible to its dependents in that same cycle. A two-node test drives real ServiceManager apply cycles over a shared store, each node holding an edge whose target only the other can publish; it converges in a bounded number of rounds only with this change. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): defer records whose prerequisite has not replicated Vector, metadata, and topology apply treated a missing replicated prerequisite as a hard error and rejected the whole batch: one embedding whose content blob had not arrived, or one edge whose target node had not, blocked every other winner in that stage on every cycle. These are ordering races between independently published records, not failures. Each adapter now defers such a record instead: it is skipped, the rest of the batch applies, and its logical key is reported through deferredKeys(). Nothing marks it applied, so it remains a winner in the replicated index and the next cycle retries it once the prerequisite lands; apply needs no pending queue of its own and loses nothing across restarts. An edge whose endpoint is being deleted in the same batch is deferred too rather than inserted against a stale node id. Malformed records, probe failures, and store errors still fail the stage. Content blobs are independent, so the content stage now attempts every blob and fails afterwards with the first genuine error; a corrupt blob no longer starves the blobs after it. A blob whose bytes are not hydrated in the sync cache yet is deferred like the other stages. Signed-off-by: trvon <me@trevon.dev> * feat(p2p): report memory-sync apply health in status The mesh deadlock was invisible: every node reported failed_cycles=0 and a climbing successful_cycles while nothing converged, because a cycle counts as successful once reconciliation completes, whatever the apply stages did afterwards. Deferred records would be just as invisible without a signal. ServiceManager now keeps an ApplyHealth record of the apply path: - deferred records per stage, and the age of the oldest one still pending, with a warning once a deferral passes ten minutes (its prerequisite likely never arrived). Ages are tracked for at most 4096 keys per stage, keeping the oldest; counts stay exact. - hard failures per stage, plus the last failure's stage, message, and age. - apply_cycles and apply_failed_cycles. failed_cycles keeps meaning reconciliation failures; a cycle whose apply stage hard-failed is counted in apply_failed_cycles instead of passing as a plain success. - publish_skipped_cycles (delete outbox preempted the cycle) and publish_failed_cycles (an outbound backfill domain failed). The fields travel through MemorySyncStatus, MemorySyncResponse, and protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the first line, an `apply` line with per-stage counts, a last-apply-failure line, and an `apply` object in --json) and to the memory-sync section of `yams daemon status`. The p2p status formatter moves behind a header so it can be tested without a daemon. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): keep content deferrals when the same scan hits a failure The content stage scans every blob and fails afterwards with the first genuine error, but the apply cycle discarded the deferred keys that the same scan collected whenever it failed. A corrupt blob therefore hid every blob still waiting for its bytes: the deferred count stayed at its previous value (0 on the first pass) and the stale-deferral warning never fired for a prerequisite that was actually missing. A stage pass now carries both outcomes. The content-blob scan moves into ContentBlobSyncAdapter::applyWinners, which reports the applied count, the deferred keys, and the first failure of one complete scan, and ApplyHealth::recordPass records the failure and replaces the deferred set together. A stage that returns an error still leaves the deferred set as last observed, since an aborted scan says nothing about prerequisites. The tracked-deferral cap already carried every still-deferred tracked key before admitting new ones, but its carry loop had a cap check that read as if it could stop early. Drop it (the tracked set never exceeds the cap) and pin the behaviour with tests that place the oldest key after more than the cap's worth of new keys and that refill a full tracked set. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
b5a5396931
|
test(repair): run the vectors.db vacuum case with real vector tables
CI runs the unit lanes with YAMS_SQLITE_VEC_SKIP_INIT=1, so the test's VectorDatabase came up without vector tables and the post-VACUUM insert failed on every platform. The case already clears the other vector toggles; clear this one too. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
1fa2e92510
|
fix(daemon): show host pressure and deferred work in daemon status
When the daemon holds background work back for a busy host, a user looking at a stale topology or an unfinished repair scan needs to see why. Status had no host signal at all. Carry the governor's host sample through the metrics snapshot and StatusResponse: source (psi, loadavg), whether the host counts as busy, the PSI cpu/io/memory "some avg10" values or the per-CPU load average, and which deferrable work is currently held back. The proto transport piggybacks the fields on request_counts like the other string status fields; the native serializer appends them as optional tail fields. `yams daemon status` (brief and detailed) shows a Host Load row, e.g. "Busy cpu 72% · io 3% · mem 0% · deferring topology, repair", and `yams status --json` gains a host_pressure object. Document the new [tuning.resource] keys next to the other typed tuning sections. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
8c24821c96
|
fix(daemon): stop topology rebuild requests from spinning workers
requestTopologyRebuild() re-posted itself immediately whenever ingest, post-ingest, embedding or KG work was still in flight. It released its in-progress flag before re-posting, so every request that arrived while the daemon was busy (one per embedding batch) could win the flag and start its own re-post loop. Those loops never slept: they occupied the worker pool until the ingest drained, and a slower drain on a loaded host meant more of them for longer. Per-thread sampling during a 1899-file ingest on a 2x oversubscribed 48-core host showed ~36 worker threads at 100% for the whole drain. For the same ingest (120s window, background work only): daemon CPU seconds before after idle host 41, 41 23, 20, 19 loaded host 96, 95 28, 30, 30 (deferral off) Arm a single timer instead: a busy daemon re-checks after 500ms, a rebuild deferred for host pressure after 5s, and only one retry is armed at a time. post_ingest_drain still requests a rebuild directly when the pipeline empties, so topology freshness does not wait on the timer in the common case (idle-host topology completion is unchanged at ~7s after ingest). Add a ServiceManager test for the deferral path: with the host marked busy the rebuild is held and the governor reports it deferred; once the host calms down nothing is held back and, when a vector database is available, the armed retry builds the artifacts without a new request. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
963d9bdbe4
|
perf(daemon): defer background work while the host is busy
Background work started as soon as it was scheduled, no matter what else the machine was doing, so a topology rebuild or a repair scan competed with the user's own jobs for CPU at the worst time. Classify background work. Must-run work keeps the corpus consistent and is never gated: crash recovery, database integrity checks and WAL checkpoints. Deferrable work waits while host pressure is elevated: - topology rebuilds and semantic topology maintenance (the dirty set keeps accumulating; a deferred rebuild re-checks on a 5s timer) - repair maintenance: the initial scan (which is what queues embedding catch-up for documents missing vectors), the legacy snapshot backfill and the vectors.db VACUUM - semantic_neighbor backfill for older documents Embedding of newly ingested documents is not deferred. Throttling it to one job under host load was measured and dropped: on a 2x oversubscribed 48-core host it stretched the embedding drain 1.5-4x and sometimes triggered two extra vector index builds, without lowering the daemon's CPU use. Each kind goes through a DeferralGate in the resource governor. A starvation guard bounds the wait: after background_max_deferral_s (default 300s) one pass runs and the wait starts over, so sustained load slows this work to a trickle but never stops it. With the guard at 45s, a loaded run held the topology rebuild for 45s and then built it (48-53s after ingest instead of 22-33s with deferral off). Setting [tuning.resource] background_max_deferral_s = 0 turns deferral off. The governor logs when a kind starts deferring, when the guard forces a pass, and when it resumes. RepairService already gated maintenance on client activity and daemon pressure; the host check sits last in each condition so the deferral clock only measures time spent waiting on other processes. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
52545d3f61
|
perf(daemon): sample host pressure in the resource governor
The governor only measured the daemon itself (RSS, own CPU), so it could not tell that the machine was busy. On a loaded host the daemon still started background work at full width, and that work ran far less efficiently: ingesting the same 1899-file corpus cost 41 daemon CPU seconds on an idle 48-core host and 96 CPU seconds with the host 2x oversubscribed (stress-ng), and took 4x longer to finish. Sample a cheap host-wide signal on the existing governor tick, at most once per second: Linux PSI "some avg10" for cpu, io and memory, falling back to the one-minute load average per online CPU; on macOS the load average plus the kernel VM pressure level. A busy host is a flag with a release band (signals must drop to 75% of the threshold to clear) so work does not flap around a threshold. Host load stays a separate dimension from the governor's pressure level. Folding it into the level would reuse the Warning response, which blocks model loads and trims foreground concurrency; a busy build on the same machine should not stop a search from loading its model. Background scheduling consumes the flag in a follow-up change. Thresholds are typed config under [tuning.resource]: host_pressure, host_cpu_pressure_pct (40), host_io_pressure_pct (30), host_memory_pressure_pct (10) and host_load_per_cpu (1.5). They show in the runtime tuning status with their provenance. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
79232e5471
|
refactor(graph): drop --impact and --affected-tests
v0.20 stopped producing code-symbol nodes and calls/references/ inherits/implements edges, and migration 40 deleted the existing ones. GraphContextService::impact() and affectedTests() had already been reduced to returning a "code-symbol graph was removed" warning, so `yams graph --impact`, `--affected-tests` and `--test-pattern` (and the MCP graph actions `impact` and `affected_tests`) could only ever print that warning. Remove the dead surface: - CLI: the three flags, their execute/render helpers, and the --impact mention in the --global help text. - MCP: the two actions, their schema entries and the changed_files / test_pattern request fields. Unknown graph actions now fail with InvalidArgument instead of silently running a default query. - Service: impact()/affectedTests() and their request/response structs. The GraphImpact/GraphAffectedTests IPC messages stay for wire compatibility. The dispatcher now answers them directly with the same removal warning, without touching the metadata repository. Kept: --lookup (with --at-file) and --trace/--to. Both resolve graph node labels and aliases and walk edges of any relation, so they still work on document, path, version and entity graphs. Tests: the service-level impact/affected-tests case moves to the dispatcher graph suite; the sandbox smoke test now expects --impact to be rejected. docs/prompts/PROMPT-eng-codex.md points related-test and blast-radius questions at `yams grep`. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
4be71457d0
|
refactor(ipc): route Connected->ReadingHeader through state impl
The ConnectionFsm state classes produce transitions through Next, and to_public() maps every ImplState back to the public enum. No state class ever returned ImplState::ReadingHeader, so that mapping was unreachable: on_readable() bypassed the state classes and called transition() with a hard-coded switch instead. ReadingHeader itself is live (the request handler drives on_readable() and checks for the state), so wire it rather than remove it: add an on_readable() hook to StateBase, have ConnectedState return ReadingHeader, and let ConnectionFsm::on_readable() dispatch through the current state like every other event. ConnectedState::on_header_parsed() returned ReadingPayload or WritingHeader, neither of which the legality matrix allows from Connected, so every such transition was rejected. Drop the override; the observable behavior (header events are ignored until on_readable()) is unchanged. Pin the transition table with a Catch2 case covering on_readable in each reachable state, header-only and payload frames, streaming keep-alive, close, error, and accept, and give the previously assertion-free "Body parsing" section real expectations. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
f8fce1db18
|
fix(daemon): destroy the SocketServer only after its handlers are gone
YamsDaemon::stop() stopped the SocketServer and destroyed it while the IOCoordinator threads were still running, and only then stopped and destroyed the IOCoordinator. SocketServer::stop() waits at most ~2s for connections to drain, and the handle_connection coroutine frames keep a raw pointer to the server in their cleanup guard. A connection that outlived the drain window either resumed on an I/O thread after the server was freed, or had its frame destroyed together with the io_context, and both paths decremented the connection counters of the freed SocketServer (ASAN heap-use-after-free in CleanupGuard). The ServiceManager-init failure path stopped the server the same way and left the I/O threads running. Tear the IPC layer down in one place, teardownSocketServer(): stop the server (which still releases its acceptors and accept strand while the io_context runs), stop and join the I/O threads so no handler can resume, destroy the IOCoordinator so the io_context destroys the pending handler frames while the server is valid, and only then destroy the server. All daemon stop and startup-failure paths use it. The unit test stops the io_context under the connection handlers so they are deterministically pending when the teardown runs; the daemon integration test parks the I/O threads past the drain window. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
aa36c2d069 |
Merge pull request #222 from trvon/chore/data-dir-cleanup
chore(daemon): data-dir cleanup via migration and repair maintenance |
||
|
Trevon
|
057933b99a |
Merge pull request #218 from trvon/test/reenable-post-ingest-persist
test(repair): re-enable the post-ingest persistence regression test |
||
|
trvon
|
f87b64eb40
|
feat(daemon): expire idle session files from repair maintenance
~/.local/state/yams/sessions grows without bound: tool integrations create a session per run (1,596 opencode-* and 109 mcp-grep-hot* files out of 1,764 on one machine) and nothing ever removes them. Add expireIdleSessions() next to the session store and run it from RepairService's idle maintenance once a day, only while no client is connected. A session file is deleted only when nobody has created, opened, closed or modified it for [daemon.maintenance] session_expiry_days (typed config, default 30, 0 disables), and it is kept when it is: - the current session, - watching a directory that still exists, - still tagged on documents (they stay reachable through `session merge` / `discard`; without a metadata repository ownership is unknown and nothing expires), - not valid JSON (not ours to judge). Tests build a store shaped like the audited one and check each rule, plus the RepairService wiring against a real metadata repository and that a connected client defers the scan. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
8eb7d96c2c
|
chore(migration): drop orphaned code-symbol artifacts from the data dir
v0.20 removed the code-symbol graph, but installs still carry its leftovers: - <data_dir>/grammars holds tree-sitter grammar libraries (~44 MB on the audited machine) that `yams init` / `yams config grammar` built and nothing loads any more. - A yams_symbol_extractor plugin under the install prefix is refused on every start with two warnings (the ABI host's skip plus the plugin manager's "load failed"), and the refusal flipped the plugin host FSM through Failed. Remove the grammars once through metadata migration 43, so the cleanup is recorded in schema_migrations and never repeats. It only acts on a data directory's yams.db, only deletes libtree-sitter-* / tree-sitter-* files directly in grammars/, removes the directory only when nothing else is left in it, and leaves a symlinked grammars/ (and its target) alone, so nothing outside the data directory can be touched. The install prefix is not ours to clean, so the stale plugin is only reported: one info line per process naming the file that can be deleted. PluginManager recognises that refusal and skips it quietly instead of warning and dispatching a load failure. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
6a71e79a47
|
feat(daemon): vacuum vectors.db from idle repair maintenance
vectors.db had no reclaim path at all. On a real install it had grown to 39.7 GB with 48% of its pages on the freelist (about 19 GB of dead space), while yams.db already had a startup auto-VACUUM with a size, free-page-ratio and free-disk check. Move that decision into a shared sqlite_vacuum helper so both databases follow the same rules, and let RepairService's background loop check vectors.db once a day (15 minutes after start, typed config [daemon.maintenance] vector_vacuum_interval_hours, 0 disables). VACUUM holds the vectors.db write lock for the whole rewrite, so it only starts while the daemon is idle: no client connections, resource pressure below Warning, no queued or in-flight embedding writes, no VectorIndexCoordinator rebuild or bulk load, and no on-demand repair. It runs on its own connection with file-backed temp storage and a short busy timeout, so a live writer makes it back off instead of waiting, and a progress handler rolls it back as soon as any of those signals change or shutdown starts. Readers on the owner's connection are unaffected in WAL mode. Besides the data-dir free-space rule it also requires the temp directory to hold a copy of the live pages. Tests cover the thresholds (including the audited 39.7 GB / 48% case), a real vacuum that shrinks a file, abort and live-writer back-off, the admission truth table, and RepairService vacuuming a live VectorDatabase file only when idle, which stays writable afterwards. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
344f3c588f
|
fix(daemon): sweep stale integrity-stamp temp and claim files
The clean-shutdown stamp is written through yams.db.integrity.tmp and consumed by renaming it to yams.db.integrity.claim.<pid>.<n>. Both are removed on the normal path, but a crash between the rename and the unlink leaves them behind, and nothing ever cleaned them up, so they accumulate in the data directory across crashes. sweepStaleDbIntegrityStampArtifacts() removes them conservatively: - only the exact .tmp name and .claim.<pid>.<n> names are considered; the live stamp and any other file are never touched; - a file must be older than a minimum age, so an in-flight publish or consume is never raced; - a claim is removed only when its pid is provably not running (kill(pid, 0) reports ESRCH / OpenProcess reports no such process); an unparsable owner keeps the file. It runs from RepairService::runRecoveryArtifactMaintenance() alongside the corrupt-DB retention cleanup, so it only happens in the background when maintenance is allowed, with stampArtifactMinAge (1h) in RepairService::Config. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
8da4df3568
|
fix(daemon): delete corrupt DB copies only after confirmed salvage
Corrupt-DB handling could destroy the only copy of documents: - removeCorruptDbFiles() deleted every readable yams.db.corrupt-* after a startup or `repair --orphans --remove-corrupt` salvage, including DBs whose salvageFromCorruptDb() had failed or only partly copied, and DBs that were never salvaged because their document count was <= the live DB's. - A corrupt DB whose count was <= the live DB's was otherwise kept forever with only a "skipping" log, with no path that ever decided it was safe to drop. - quarantineAndRecreate() renamed onto yams.db.corrupt-<second>, so two quarantines in the same second overwrote the first copy. Deletion is now decided by evidence, not by which code path ran: corruptDbSalvageConfirmed() opens the corrupt DB read-only, attaches the live DB and checks that every document hash in it is present in the live DB (salvage copies documents keyed by hash). removeSalvagedCorruptDbs() removes an artifact only when that check succeeds and the artifact is older than a retention window (quarantine timestamp from its name, else mtime); unreadable, unconfirmed or young artifacts are kept and reported with the reason. Startup salvage no longer deletes anything. RepairService owns cleanup: runRecoveryArtifactMaintenance() runs from the background loop when maintenanceAllowed(), every recoveryArtifactSweepInterval (6h), with corruptDbRetention (7 days) in RepairService::Config. An explicit `repair --orphans --remove-corrupt` waives the retention window but still requires confirmation. yams doctor explains the retention policy next to the artifacts it lists. The "yams.db" / ".corrupt-" naming is centralized in db_recovery.h (kMetadataDbFileName, kCorruptDbMarker, corruptDbPrefix(), isCorruptDbFileName(), listCorruptDbs()) and used by salvage, recovery, RepairService and the doctor checks instead of four hand-rolled scans. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
fb3ed2877d
|
fix(daemon): quarantine malformed WAL/SHM instead of deleting them
When a stale-WAL checkpoint at startup (ServiceManager::recoverStaleWalIfPresent) or the shutdown checkpoint (DatabaseManager::shutdown) failed with a "malformed"/"corrupt" error, both paths deleted yams.db-wal and yams.db-shm. A WAL that SQLite cannot checkpoint may still contain committed transactions that never reached the main file, so deleting it discarded them with no copy left for manual recovery. Both paths now call quarantineSqliteSidecars(), which renames the sidecars next to the database as yams.db-wal.quarantine-<UTC ts> (and -shm), never over an earlier quarantine, and logs where each file went. SQLite no longer treats them as sidecars, so the database still continues from its last checkpoint, but the bytes are preserved. The WAL is moved first; if it cannot be moved the SHM is left in place so SQLite sees a consistent pair. yams doctor now lists quarantined sidecars alongside corrupt DB artifacts so they are not forgotten. Signed-off-by: trvon <me@trevon.dev> |