868 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Trevon
|
211db15606 |
fix(daemon): bound semantic neighbor scores and report startup phase (#334)
Semantic-neighbor maintenance aborted the daemon during startup: std::clamp<float> asserted '!(__hi < __lo)' because the adaptive cosine threshold could exceed 1.0 for a legal pair of identical vectors. The same work was also admitted while the daemon was still initializing, so the 300-second starvation guard forced a pass before readiness. + semanticInverseNorm/semanticCosineSimilarity/semanticExplicitThreshold in the existing embed policy header, shared by the streamed and corpus branches; non-finite vectors or scores produce no neighbor and legal scores are bounded before ranking and threshold selection + startupMaintenanceEligible() requires ServiceManagerState::Ready, not stopping and idle, and is evaluated before ResourceGovernor::admitDeferrable(), so an elapsed deferral deadline can no longer manufacture missing readiness + StartupPhaseTracker (two atomics, no mutex) plus a warn-level init-waiter line reporting state, phase and phase_elapsed_ms instead of a debug-level state number; a 53-minute startup previously logged nothing actionable + regression through the public backfill and corpus-rebuild APIs using a 64-D vector whose old float inverse norm rounds its self-cosine above 1 - removed the duplicated inverse-norm and cosine lambdas from EmbeddingService - renamed __test_setModelProvider/__test_setKgStore to testing_* because they used reserved identifiers, across 11 files - re-anchored 13 pre-existing raw-environment allowlist entries to their new line numbers; no new exception was added Startup is still blocked by corpus-sized memory-sync scans; this change stops the abort and names the blocking phase, it does not make startup fast. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
60dd2c9afe |
fix(daemon): admit cold reads and preserve sync deferral accounting (#331)
Dispatch explicit committed memory reads on admitted workers, checking the governor before queueing and before I/O. Require the coordinator instead of using the synchronous strand fallback; keep cached APIs and wire opcodes unchanged and preserve task-record validation. Use the paged content adapter, distinguish typed local admission deferral from hard errors and retain known deferred totals/ages after incomplete scans. Forward exact topology totals independently of diagnostic samples and keep attempted cycles distinct from complete scans and successful apply. Add real daemon IPC tests for authenticated off-resident reads, pressure health, exact 70-edge status/retry accounting and authenticated cold document retrieval through the existing MCP get handler, DaemonClient IPC and CAS. Reanchor existing dispatcher environment-read entries without new exceptions. Scan uncertainty remains C++-only: typed wire/status/MCP exposure, broader fault/readiness/P2P convergence and patched resource measurements remain open. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
604e68e17b |
feat(memory-sync): apply paged winners through bounded adapters (#330)
Use pinned, governed pages for metadata, content, vectors and topology. Preflight identity/content/model prerequisites before writes and continue independent records without softening authentication, capacity or I/O errors. Apply topology deletions, nodes and edges in one real KG batch with bounded transaction-local reads. Preserve ordinary-read isolation, report exact edge deferrals independently of bounded diagnostic samples and avoid empty commits. Retract absent local topology only from exact committed own observations and conditionally erase the same envelope, including off-resident/stale-winner regressions. Probe local KG presence with fallible scalar SQL rather than materializing properties; retain SQL errors as errors, not absent records. The public C++ KG/batch interfaces gain virtual capabilities. Defaults provide source-level hard errors only: installed/old-binary ABI compatibility is NOT established and providers must be rebuilt/reviewed before promotion. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
3b5d4cbc12 |
feat(memory-sync): add pinned adapter views and callback drains (#329)
Expose owner-validated committed views, bounded exclusive-prefix pages and fallible exact historical payload reads without per-value reconciliation. Keep cached reads nonblocking and separate from explicit committed I/O. Add governed adapter indexes, revocable escaped-handle lifetime checks and typed local pressure/contention deferral. Release the loop lock only across audited native payload fetches and recheck cancellation/quarantine afterward. Preserve the process-wide recursive callback serializer while releasing the lifecycle gate before its drain on both restart and stop. Recheck workers and callback generations after the gap; retain all cancellation and join guards. Wire the optional local-yield callback through the config factory alongside the constructor change. Add native SDK concurrency, pressure, lifetime, off-resident and callback-order regressions. Arbitrary blocking backend/callback shutdown is not signed off. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
43adae9991 |
feat(memory-sync): serve complete winners from bounded disk generations (#328)
Spill exclusive native-filesystem sweep observations and committed winners into bounded SQLite workspaces on the corpus filesystem. Preserve complete operation/vector fork observations and validate payloads before publishing an immutable generation; keep resident caches separate from full authority. Add fallible committed/pinned reads, bounded adapter bookkeeping, cooperative filesystem traversal, secure scratch allocation/cleanup and pinned lifetime limits. Retain exact durable erase intents and condition own-record erasure on the full current envelope, including a post-reconciliation check. Preserve the existing shared-store, cache, value and scan limits. Reanchor only the existing filesystem environment-read entries and register SQLite and layout-independent test hooks in the memory-sync library. Bootstrap import/finalize/restart, broader ingress/scratch faults and final SDK ABI/security/performance signoff remain separate work. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
59d11a1504 |
fix(daemon): synchronize metrics completion and data-root snapshots (#327)
Publish owning data-directory snapshots under a mutex and route bootstrap and fallback publications through the same synchronization boundary. Keep polling completion locked through its final notification so shutdown cannot destroy the condition variable while the notifier still uses it. Exercise both publication windows with deterministic event-based tests and use the repository thread-compatibility wrapper. Preserve Failed in status rendering instead of reporting initialization. Reanchor the existing metrics environment-read entry; add no exceptions. This does not close the separate queued-collector and embedding-borrower lifetime audits or establish universal bounded callback shutdown. Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
11b8443bc4 |
fix(memory-sync): stop re-walking the op store when a sweep cannot fit (#325)
A complete sweep fails closed when its retained state cannot hold the
corpus, returning ResourceExhausted ("complete-sweep candidate budget
exhausted"). That is correct: a truncated sweep must never commit a fork
decision. What was missing is a failure state. Every caller retried
immediately, and each attempt re-listed the whole op store, so a corpus
larger than maxTrackedIdentities pinned a core indefinitely.
Observed on a 65,612-object op store with 24,689 index records against
maxTrackedIdentities=8192: one thread in
publish -> reconcileBeforeWrite -> syncFully -> sync
-> FilesystemBackend::listPage("index/")
-> visitFilesystemKeys -> recursive_directory_iterator
92% of it in kernel time, 7h26m CPU over 7h29m wall in a single run, and
17,486 "backfill domain failed: complete-sweep candidate budget
exhausted" lines. The listing snapshot cache could not help: it never
completes, because the sweep aborts before reaching the end.
Remember the unsatisfiable sweep and gate retries behind bounded
exponential backoff (5s..600s) so one attempt per window replaces one
attempt per caller. A single retry is still admitted, because one
exhaustion can be a store that is about to fit (an erase or quarantine
landing between attempts); the latch starts on consecutive failures and
clears when a sweep completes.
Only the sweep's own budget sites set sweepBoundExhausted_. A
resource-governor denial reports ResourceExhausted too, but it is
transient backpressure and must keep retrying freely, otherwise pressure
that has already lifted suppresses convergence for a whole window.
Fail-closed behaviour is unchanged: the wrapper never converts a failure
into success, nothing commits until a sweep completes, and the memoized
error carries the original code and message. Cancellation is never
memoized, so shutdown is not delayed.
Signed-off-by: trvon <git@trevon.dev>
|
||
|
Trevon
|
31c9e57447 |
fix(memory-sync): bound complete sweeps and cancel filesystem walks (#321)
* perf(memory-sync): verify each envelope's signature once Every publish and every backfill cycle reconciles the whole op store, and reconciliation authenticated every index envelope it read. On a store with 44k objects that put 73% of the daemon's remaining CPU inside Ed25519 verification, re-checking signatures whose bytes had not changed. Remember positively verified envelopes for the life of a loop, keyed by the digest of the whole serialized envelope including the signature, so a swapped signature is a different key and gets verified. Trust set, corpus, epoch and signature policy are fixed for a loop, so rebuilding the loop is the invalidation: key rotation, revocation and allowlist changes all arrive as a new loop. The set is capped by a new typed limit and cleared on overflow or on allocation failure, so exhausting it costs a verification, never an acceptance. Nothing is memoized for verifyDigest, and cache lookup happens only after the envelope digest has already been checked against its key. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): stop pinning document backfill on an oversized record The document sweep is an ordered id cursor that only advances after a document publishes. A record larger than the resident publish limit failed with InvalidArgument, which is neither of the unrecoverable codes, so the failure was rethrown and the cursor stayed put. The same document was then retried every five seconds forever, and every retry re-reconciled and re-authenticated the whole op store, which is what kept a live daemon at 40% of a core with its status reporting idle. Classify the rejection from the serialized record bytes rather than from the error text, and skip that document so the pass reaches the ones behind it. The vector sweep already behaves this way. The local record is untouched, so a corrected value still publishes on a later pass once the store changes. MetadataSyncAdapter gains a size helper for that classification; it reuses the same record builder publish() serializes. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): bound whole-sweep candidate state Charge causally valid observations against an aggregate encoded-byte budget before retaining identities, including misplaced records. Bound candidate count as well as unique identities, and fail the whole sweep on exhaustion instead of committing a truncated fork decision. Discard incomplete scan state on cancellation, listing errors, and exceptions. Honor backend cancellation between filesystem directory entries and check cancellation before durable sweep finalization. Preserve cross-page fork detection and the existing page-size default. The budget bounds retained input, not exact RSS or total scan work. Listings remain non-snapshot and individual filesystem syscalls are not interruptible. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
8d439db608 |
perf(memory-sync): stop the idle re-verify loop (#320)
* perf(memory-sync): verify each envelope's signature once Every publish and every backfill cycle reconciles the whole op store, and reconciliation authenticated every index envelope it read. On a store with 44k objects that put 73% of the daemon's remaining CPU inside Ed25519 verification, re-checking signatures whose bytes had not changed. Remember positively verified envelopes for the life of a loop, keyed by the digest of the whole serialized envelope including the signature, so a swapped signature is a different key and gets verified. Trust set, corpus, epoch and signature policy are fixed for a loop, so rebuilding the loop is the invalidation: key rotation, revocation and allowlist changes all arrive as a new loop. The set is capped by a new typed limit and cleared on overflow or on allocation failure, so exhausting it costs a verification, never an acceptance. Nothing is memoized for verifyDigest, and cache lookup happens only after the envelope digest has already been checked against its key. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): stop pinning document backfill on an oversized record The document sweep is an ordered id cursor that only advances after a document publishes. A record larger than the resident publish limit failed with InvalidArgument, which is neither of the unrecoverable codes, so the failure was rethrown and the cursor stayed put. The same document was then retried every five seconds forever, and every retry re-reconciled and re-authenticated the whole op store, which is what kept a live daemon at 40% of a core with its status reporting idle. Classify the rejection from the serialized record bytes rather than from the error text, and skip that document so the pass reaches the ones behind it. The vector sweep already behaves this way. The local record is untouched, so a corrected value still publishes on a later pass once the store changes. MetadataSyncAdapter gains a size helper for that classification; it reuses the same record builder publish() serializes. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
33243d8472 |
fix(memory-sync): stop re-walking the op store on every publish (#319)
* fix(storage): bound filesystem listPage residency listPage() walked the whole op store and retained every matching key before slicing one page. At 1025 records and a 512-record page it held 1025 keys where 513 is the working set, and the retained set grew with the store. Traverse once into a max-heap holding the smallest eligible limit+1 keys. The walk itself stays complete, so a traversal error still returns an error instead of a partial page. Exclusive lexical cursor, ordering, Windows key decoding and the SIZE_MAX limit stay as they were. Add a YAMS_TESTING-only observation seam that counts real walks, entries visited and peak retained keys, so the traversal cost can be measured instead of inferred. The new traversal shifts two reviewed getenv allowlist line numbers. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): keep backfill progress past an oversized vector A record above the resident limit failed publication, the domain cursor never advanced, and every later cycle retried the same row. Valid vectors behind it never reconciled, so one oversized value could pin a domain indefinitely. The daemon log showed this repeating every ~100 s. Classify the row only after a publish fails with InvalidArgument and the same serialized record is independently confirmed to exceed the immutable limit min(maxValueBytes, maxCacheBytes). Skip the row for this pass, keep the local copy, and revisit it when a new source generation or service limit appears. A generic backend InvalidArgument stays retryable, and healthy rows still serialize only once. Signed-off-by: trvon <git@trevon.dev> * fix(memory-sync): reuse a bounded complete listing across publishes Every store invalidated the cached listing pages, so a publish followed by syncFully walked the whole op store again, once per 512-record page. On the profiled daemon that re-walk was the hot path: 99.6% of one core, 87.9% of it system time, with listPage at 55% of user samples. Keep a complete key-only prefix snapshot, built only from normal contiguous pages starting at an empty cursor on the concrete filesystem backend. The snapshot is capped by key count and key bytes, served only in lexical order, and dropped as incomplete unless the walk reached the end. Any mutation invalidates it before the backend call; a successful store or remove then applies the exact key to matching snapshots, and a failure drops them because the write may have committed. Initialising the async path disables cache reuse permanently rather than racing a future. Cache maintenance is best effort: it runs after the backend result is known and any allocation failure clears the cache and keeps the committed result. Opaque cursors are untouched, so only the built-in filesystem backend gets snapshot treatment. Active publishes now cost the same number of walks as a warm sweep instead of four, six and nine at 511, 513 and 1025 records. Signed-off-by: trvon <git@trevon.dev> * test(bench): measure memory-sync listing cost across arms Add a standalone harness that drives a real filesystem op store through MemorySyncLoop, one scenario per process so no cache state survives a repeat, and reports wall, user, system and peak RSS per run. It uses production APIs only, so the same source builds against a baseline and a candidate tree and neither arm needs a measurement seam ported into it. The driver seeds one corpus per size, interleaves the two arms inside each repeat, and writes results.jsonl plus a median report. Enumeration counts are not re-derived here; the unit suite owns those. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
3272367ea2 |
fix(search): restore measured scientific ranking (#318)
* fix(search): make the final lexical tie-break a strict weak ordering The final comparator broke near-ties toward the result with less vector support, and its pairwise |a - b| <= epsilon test was not transitive, which is undefined behavior for the std::sort/partial_sort callers in applySearchTopKLimit. Compare tie buckets floor(score / epsilon) instead, then lexical anchor, keyword and vector evidence (more first), then doc id. Behavior with the tie-break off is unchanged. The comparator moves to final_result_order_internal.h so it can be tested directly. Signed-off-by: trvon <me@trevon.dev> (cherry picked from commit ca9b82d6ce9db31b6c9608a7d95b63203649e5a7) Signed-off-by: trvon <git@trevon.dev> * fix(search): keep cross-reranked scores above the unreranked tail applyCrossRerank rewrote the window's scores to a 0..1 min-max blend while the tail kept raw fused scores. Later whole-list sorts (concept boost, top-k partial_sort) compared the two scales, so reranked docs could drop below tail docs or out of the top-k. Map the blended score back onto the window's original range, with the floor raised just above the best tail score when needed. The window's reranked order is unchanged and docTraces.finalScore records the mapped score. Signed-off-by: trvon <me@trevon.dev> (cherry picked from commit d66c6d304ca4d32b943e164b0d7720fc7f865db1) Signed-off-by: trvon <git@trevon.dev> * fix(search): judge weak tier-1 queries on the raw bm25 scale The weak-query test compared the top Text/SimeonText component score with weakQueryMinTopTextScore, but lexical batches are min-max normalized, so the top hit is always 1.0. With enough hits every query was strong: the weak-query vector fanout boost and weak-query topology expansion never ran. scoreLexicalBatch now also records absoluteLexicalScore, the hit's score on the existing absolute scale (-bm25 / bm25NormDivisor, same multipliers), and classifyTier1TextSignal uses it. Thresholds keep their values; with the default divisor 25, 0.30 is bm25 <= -7.5 and 0.45 is bm25 <= -11.25. Signed-off-by: trvon <me@trevon.dev> (cherry picked from commit bc771c5aab1721170fb9e743013ca0efbf048d61) Signed-off-by: trvon <git@trevon.dev> * fix(search): restore measured scientific fusion Give SCIENTIFIC the measured relative Simeon weight of 0.40 after normalizing the existing text/vector pair. The resulting share is 0.285714, matching the benchmark traces. Other profiles remain unchanged. Keep its one-slot semantic rescue preset instead of raising it to three at the corpus layer. Preserve the existing large-prose recall widening. Cover the preset weights, config propagation, community blend, rich-graph ratios and corpus rescue behavior. Signed-off-by: trvon <git@trevon.dev> * fix(search): order non-finite ranking evidence Treat NaN score/evidence keys as negative infinity and invalid tie epsilon as zero. This keeps comparator equivalence transitive without changing finite-input ordering. Add exhaustive triples covering NaN, infinities and invalid epsilon. Signed-off-by: trvon <git@trevon.dev> * docs(search): clarify lexical scoring limits Distinguish the shared single-hit score conversion from FTS5 BM25 calibration. Note that lexical tie buckets can still reorder numerically separated rerank-window and tail scores. No behavior change. Signed-off-by: trvon <git@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
8881e0b3e9 |
perf(memory_sync): skip idle rescans and listings of a direct op store (#311)
* test(search): rename a road beam local that collides with the Windows far macro windows.h defines far and near as empty macros, so `const auto far = ...` in search_vector_stage_road_beam_catch2_test.cpp does not compile with MSVC (C2513), and the Windows Tests job has failed since the road beam tests landed. Rename it to deep. Signed-off-by: trvon <me@trevon.dev> * perf(memory_sync): skip rescans of an unchanged direct op store The sync worker rescanned the whole op store every cycle: a full walk of the sharded objects tree, then a read, hash, decode and signature check of every index envelope. On an idle 3-node mesh (2 CPUs per node, 5,256 objects) that cost about 1.1 s of CPU per 5 s cycle on each node, mostly in getdents/openat. In direct transport the daemon is the op store's only writer. The loop now counts its own backend writes and reuses the last complete scan until it writes again or a writer is quarantined. Shared-store and temporary namespaces keep rescanning every cycle. Signed-off-by: trvon <me@trevon.dev> * perf(memory_sync): keep exclusive-backend listings until the loop writes FilesystemBackend::listPage walks every shard directory of the op store for any prefix. After the rescan skip, each idle direct-transport cycle still listed the store twice: replaying ready erases and inspecting the erase outbox. The loop's write-counting backend now keeps listing pages on an exclusive backend and drops them on any store, remove or clear. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
81a051fc2e |
feat(search): walk along roads (#307)
Let RouteOnly and Augment expand every routed hit along its cluster's walk order. The snapshot cache loader resolves each cluster's `orderedMembers` to member hashes (`buildTopologyOrderedMemberHashes`, `TopologyRoutingSnapshot::orderedMemberHashes`); a cluster with `orderKind` None or a permutation that fails `validateMemberPermutation` resolves to nothing. The store's `orderUnavailableReason` is carried on the session result and emitted as `topology_order_unavailable_reason` only when set. When `roadWalkWindow` > 0 the session builds a `TopologyRoadWalkIndex` after the road beam: the selected routes first, then the road-expanded clusters, keeping only clusters with an order and counting the rest in `roadWalkClustersUnordered`. The index borrows its member views from the snapshot it keeps alive; the certificate is never touched. The vector stage walks after the certified and road groups. Every routed hit so far (certified route hits and road-beam hits) anchors +-window positions along each order it belongs to, bounded to that order (`walkTopologyRoadOrders`). Anchoring on route hits too means the walk rescues the ordinal neighbours of the semantically closest members even with `road_hops` = 0; the plan's "around road hits" reads as every hit a road-era routed query produced. The walked set, deduplicated against the hits, is one extra routed query whose quota is its own size (ranking ordinal neighbours by similarity and truncating would reintroduce the filter the walk bypasses); its rows and distance evaluations land in the routed diagnostics like road candidates. Shadow and Narrow never walk. Config: `search.topology.road_walk_window` (default 0, off) on `SearchEngineConfig::topologyRoadWalkWindow` (copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy::roadWalkWindow` and applied by SearchEngineManager. `makeTopologyRoutingOptions` requests the window only for Augment and a gate-open RouteOnly, like the beam. The window is excluded from `topologyRoutingPolicyFingerprint` (comment and test). Defaults leave every stage byte-identical: the PR 2 goldens (692), the RouteOnly cells (417) and the road beam cells (904) are unchanged. Debug keys appear only when the window > 0: `topology_road_walk_window_applied` (0 when nothing was walkable), `topology_road_walk_anchors`, `topology_road_walk_candidates_added`, `topology_road_walk_clusters_unordered`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_WALK_WINDOW writes `search.topology.road_walk_window`; YAMS_BENCH_TOPOLOGY_ROAD_ORDER and YAMS_BENCH_TOPOLOGY_ROAD_MAX_PER_CLUSTER write `topology.road_order` and `topology.road_max_per_cluster` (neither had a bench mapping, so Eval 3 can set the order kind per arm and build roads at all). Tests: tests/unit/search/search_vector_stage_road_walk_catch2_test.cpp (new catch2_search_vector_stage_road_walk): two reciprocal chains A (six members) and B (four) with a road, built with `TopologyBuildConfig::roadOrder` Time from eventTime so the walk order differs from the similarity ranking; loader resolution including the invalid-permutation drop and the cache carry; fingerprint equality for any window and the option gating per policy; session-level certificate, beam and work invariance with the index listing routes then road clusters, walk sets for windows 0/1/2/100 bounded to the cluster, an unordered cluster counted and skipped, no index without any order, the order diagnostic through recordTopologyRoutingDebug; RouteOnly window 0 equal to the road beam cell with no walk key, window 1 adding the immediate order neighbours of all four hits, window 2 the whole clusters, rows visited growing while router work is unchanged, certified keys unchanged, unordered B contributing nothing, no order anywhere equal to the beam cell plus counters, hops 0 walking the selected route alone; Augment adding the walked documents as rescues; Shadow and Narrow emitting nothing. ConfigResolver covers the TOML key. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads, the bench reads its keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
cfb70f7bb9 |
feat(topology): ordered members (#306)
Give every cluster with at least two members a walk order: `ClusterArtifact::orderedMembers`, a uint32 permutation over the hash-sorted `memberDocumentHashes`, plus `orderKind`. Three kinds, each deterministic with ties resolved to the smaller member hash: - Time: ascending `TopologyDocumentInput::eventTime`; members without a time last, by hash. - Session: grouped by `sessionId`; groups ordered by earliest event time (groups with no time after those with one), then session id; inside a group by time then hash; unsessioned members last, by time then hash. - SimilarityChain: greedy nearest-neighbor tour by cosine over the members' embeddings, starting from the cluster representative (`medoid`, else the smallest-hash usable member); members without a usable embedding last, by hash. The knob is `TopologyBuildConfig::roadOrder`, resolved from TOML `topology.road_order` = none|time|session|similarity_chain through the topology.* resolver block and `TopologyManager::setRoadOrder`, threaded into the build config like `roadMaxPerCluster`. Orders are computed after boundary spill so they cover replicas, at full build in every engine and for the rebuilt clusters of a connected-engine incremental update; untouched clusters keep theirs (a kind change reaches them on the next full rebuild, like the representative rule) and `none` clears every order. Orders are independent of `roadMaxPerCluster`. Orders persist in the road sidecar, now version 3: after the roads, u32 count, then per ordered cluster u32 cluster-id table index, u32 kind, u32 length and the permutation as u32 indices. A sidecar without orders is still written as v2, so roads-only sidecars are byte-identical to the previous layout; v1 and v2 decode with no orders, and the envelope gains `member_order_count`. The sidecar is written when the batch has roads or at least one ordered cluster. On load each order is validated on its own (`attachTopologyMemberOrders`: cluster exists, kind not none, permutation covers the members exactly once); an invalid entry drops only that cluster's order and appends `<token>:<clusterId>` to the new load-time diagnostic `TopologyArtifactBatch::orderUnavailableReason`. Roads never lose anything to an order error and vice versa. The main snapshot codec, the routing construction, representation and road fingerprints never see orders, so the default `none` leaves the snapshot payload, the sidecar and every fingerprint byte-identical. No search consumer yet; the road walk lands separately. Tests: tests/unit/topology/topology_ordered_members_catch2_test.cpp (permutation validator tokens, time order with missing times, session grouping and group tie-break, hand-computed greedy chain from the medoid on a five-point fixture with fallback start, unusable members and exact ties, every engine stores valid permutations of every kind including over spill replicas, default none byte-identical payload/sidecar/fingerprints, sidecar v3 round trip plus hand-built v2 bytes and corrupt-section rejection, per-cluster attach validation with reasons, store round trip with orders-only sidecars, invalid order dropped with reason while roads and other orders stay, incremental update keeps untouched orders and recomputes rebuilt ones), plus resolver and manager setter coverage. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
111988c6e4 |
feat(topology): typed event time and session for topology inputs (#305)
* feat(topology): typed event time and session for topology inputs Add `eventTime` (epoch seconds, UTC) and `sessionId` to `TopologyDocumentInput`. The extractor fills them from two documented metadata keys: `event_time` (integer epoch seconds, or ISO-8601 `YYYY-MM-DD` / `YYYY-MM-DD[T ]HH:MM[:SS[.frac]][Z|±HH:MM]`, naive datetimes are UTC) takes precedence over the document createdTime; an unparsable value is skipped with a debug log, never an error, and a zero createdTime leaves the field unset. `session_id` is copied verbatim and empty when absent. The keys are only consulted when the extraction loads metadata; otherwise createdTime is the only source. `parseTopologyEventTime` is public for tests and future callers; the calendar math is Hinnant's days_from_civil with range-checked fields. Extraction stats count the event time source and invalid values. Nothing downstream reads the fields yet. Artifacts, codecs and both routing fingerprints are unchanged, which the new test pins. Tests (tests/unit/topology/topology_input_event_time_catch2_test.cpp): parser accept/reject tables, metadata precedence over createdTime, invalid ignored, session_id, includeMetadata=false fallback, and byte-identical snapshot payload plus equal construction and road fingerprints with and without the typed fields. Signed-off-by: trvon <me@trevon.dev> * bench(longmemeval): event_time and session_id metadata, round docs, haystack sidecar Every exported corpus document now carries `metadata.event_time` (ISO-8601, normalized from the LongMemEval `YYYY/MM/DD (Day) HH:MM` haystack date, first-seen date per session) and `metadata.session_id`. Document ids, titles, texts, queries and qrels are unchanged in the default mode. `--round-docs` adds one document per session round (`<sid>__r<n>`): a user turn opens a round and the replies that follow join it; a leading assistant turn is its own round. Round docs share the session's `event_time`/`session_id` and add `round_index`; qrels stay at the session level. `--full-haystack` also writes `haystacks.jsonl`, one line per question with `haystack_session_ids` and the normalized `haystack_dates`, so per-question membership survives the global corpus deduplication (the 2026-10-10 preflight lost it). The converter is split into `convert` / `write_outputs`; `--self-test` runs both on a three-session synthetic fixture without a download and is the regression check for the default output. Signed-off-by: trvon <me@trevon.dev> * bench(retrieval): carry BEIR corpus metadata onto ingested documents `retrieval_quality_bench` ingests the BEIR corpus as a directory, which cannot carry per-file metadata, so `event_time`, `session_id` and `round_index` from corpus.jsonl never reached the topology extractor. `BEIRDocument::metadata` now holds the corpus line's `metadata` object (strings as is, numbers and booleans stringified; both the generic and the SciFact loader read it), and `applyBenchmarkDocumentMetadata` copies it onto the stored documents matched by exact file path through the metadata repository (`ON CONFLICT DO UPDATE`, so warm caches can repeat it). The bench runs it after the ingest wait, for cold and reused corpora alike, and logs how many documents were updated or not found. Tests: loader parses and coerces metadata; apply helper writes the keys, skips documents without metadata, counts missing ones and is idempotent against a real MetadataRepository. The portability allowlist entries for the two touched bench files are re-pointed at their shifted lines; no raw environment read was added. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
f91cd85e13 |
feat(search): road beam in RouteOnly/Augment (#304)
Let the vector stage follow cluster roads from the selected route clusters. `TopologyRoutingSnapshot::roadAdjacency` is built by the snapshot cache loader (`buildTopologyRoadAdjacency`) as the union of both stored orientations of every road, per source sorted by weight desc then target cluster id asc; it is empty when the batch carries no roads, whether never built or dropped with `roadUnavailableReason`. The session runs a beam after the certificate is materialized: for `roadHops` hops it scores every unvisited road neighbour of the frontier by road weight times the router's query-centroid similarity ((cosine + 1) / 2), keeps `roadBeamWidth` per hop (score desc, cluster id asc) and dedupes across hops. Each centroid evaluation is cached per cluster and counted once in `routeRepresentativeDistanceEvaluations` and `routeExactRepresentativeDistanceEvaluations`, so routed work stays comparable with the no-road arm. The result lands only in `TopologyRoutingSessionResult::roadExpandedClusterIds` and `roadExpandedDocumentHashGroups` (one member group per cluster, excluding certificate members). `certificate.allowedDocumentHashes`, admission and `topologyRoutingPolicyFingerprint` are untouched; the fingerprint comment and a test pin that the road knobs stay out of it. `VectorStageRequest` carries the road groups with their own quotas (`allocateBalancedRouteQuotas` over the road groups) and cap. RouteOnly and Augment query them after the certified groups; the certified part of the routed result is identical with roads on or off. Shadow and Narrow never see road groups, and `makeTopologyRoutingOptions` requests hops only for Augment and a gate-open RouteOnly, so a gated-off RouteOnly running as Shadow takes no roads with it. Config: `search.topology.road_hops` (default 0, off) and `search.topology.road_beam_width` (default 4, read when hops > 0) on `SearchEngineConfig` (`topologyRoadHops`, `topologyRoadBeamWidth`, copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy` and applied by SearchEngineManager. Defaults leave every stage byte-identical: the PR 2 goldens (692) and the RouteOnly cells (417) are unchanged. Debug keys appear only when hops > 0: `topology_road_hops_applied`, `topology_road_clusters_added`, `topology_road_candidates_added` (road documents the stage appended), `topology_road_work_centroid_evaluations`, `topology_road_skip_reason` (roads_absent, no_selected_routes, route_low_confidence) and, under stage trace, `topology_road_expanded_cluster_ids`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_HOPS / YAMS_BENCH_TOPOLOGY_ROAD_BEAM_WIDTH write the two TOML keys. Tests: tests/unit/search/search_vector_stage_road_beam_catch2_test.cpp (new catch2_search_vector_stage_road_beam): a four-cluster fixture with hand-built roads A-B 2.0, A-D 1.0, B-C 1.5 where the query routes to A; adjacency construction and cache carry; fingerprint equality for hops 0 vs 2 and any beam width; session-level certificate invariance with hops 0/1/2, beam width 2, early stop and roads absent; RouteOnly hops 0 equals the route-only cell with no road keys, hops 1 gives A u B, hops 2 with beam width 1 gives A u B u C via the heavier road, work counters grow with hops; Augment hops 1 adds B's documents as rescues; Shadow and Narrow take no road groups. ConfigResolver covers both TOML keys. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads are added, the bench reads its two keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
f7f1a09865 |
feat(topology): incremental roads (#303)
Maintain cluster roads across incremental rebuilds instead of clearing them. `ConnectedComponentTopologyEngine::updateArtifacts` now keeps the roads between clusters the update did not touch, drops every road with an endpoint in a removed or rebuilt cluster (reused ids included), and recomputes the rebuilt clusters' roads from the dirty region's raw neighbor lists against the merged memberships, so a region document's neighbor in an untouched cluster yields that road in both orientations. Each source then re-ranks its top-K over the union with PR 6's order and tie-break (weight desc, target id asc). Only the region's neighbor lists are available during an update, so a cross-boundary edge is observed from one side; the out-of-region target is credited with the mirror observation its own list would have added. That is exact on a symmetric neighbor graph and the stated approximation otherwise. An untouched source can only re-rank the roads its last full build kept, so the result is derived, not full. Provenance on the batch: `roadsIncrementalPasses` counts passes since the last full accumulation (0 = full roads) and `roadsFullBuildEpoch` is the `topologyEpoch` of the snapshot that accumulation produced. A full build leaves both at 0; an update sets the epoch from `existing` when its roads were full and carries it forward otherwise. The fields ride in the road sidecar header, now version 2 (u64 epoch, u32 passes after the snapshot id); version 1 sidecars decode with both at 0, the envelope format string is unchanged, and the main v1 payload and the routing construction fingerprint do not see them. The store copies them both ways. Roads off (`road_max_per_cluster = 0`, the default) still yields no roads and zero provenance, so default behavior is unchanged. `topology_build_utils.h` splits `accumulateClusterRoads` into pair-weight accumulation (with the optional observed-document set that drives mirroring), orientation and ranking, and adds `deriveIncrementalRoads` over those pieces; the full-build path passes no observed set and is unchanged. Louvain and k-means `updateArtifacts` rebuild the supplied input from scratch, so their roads are full roads for that batch with zero passes; the test states and checks that property for them. Tests: tests/unit/topology/topology_roads_incremental_catch2_test.cpp (roads off through an update, full-build provenance, kept/retired/ recomputed roads with mirrored weights on a four-cluster graph, top-K re-ranking over kept and recomputed roads, derivation without prior roads, a property test over seeded planted-cluster symmetric graphs with cross-edge rewiring and member moves comparing incremental roads to a from-scratch build after each of several dirty regions, a tight-top-K property for rebuilt sources and derived weights, and the alternate engines), plus sidecar v2 round trip, hand-built v1 decode, future version rejection and store round trip of the provenance in topology_road_sidecar_catch2_test.cpp. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
e84ced261b |
feat(topology): road sidecar persistence (#302)
Persist `TopologyArtifactBatch::roads` in a sidecar knowledge-graph node,
`topology:roads:<snapshotId>` of type `topology_road_snapshot`, instead
of growing the main snapshot codec. The sidecar has its own binary magic
('Y','A','M','R'), version 1, and a Zstd JSON envelope with
`format = "topology_roads_zstd_v1"`. The main snapshot stays a plain v1
payload, byte-identical with roads on or off, so memory-sync replicas on
older peers keep decoding it, and an unknown node type is simply never
read by an old store. `deserializeTopologyBatchCompressed` now rejects an
envelope whose `format` it does not know instead of treating it as legacy
raw JSON.
Header layout (little-endian): u32 magic, u32 version, u32 maxPerCluster,
u32 reserved, string snapshotId, u32 cluster-id table count + strings,
u32 road count, then per road u32 from index, u32 to index, f64 weight.
`TopologyArtifactBatch::roadMaxPerCluster` records the bound the builder
applied so the header can carry it; both engines set it.
The store writes the sidecar right after the snapshot node and before
the `topology:snapshot:latest` pointer, so the pointer remains the commit
point: an injected sidecar failure leaves the previous snapshot current.
Retention deletes each pruned snapshot's sidecar with it, and re-storing
a snapshot id without roads removes its stale sidecar. Loading is
optional and soft: a missing sidecar yields empty roads with no error; an
undecodable or invalid one (snapshot mismatch, endpoint not a cluster,
self loop, negative or non-finite weight, more roads per source than the
declared top-K) drops the roads and records
`TopologyArtifactBatch::roadUnavailableReason`, surfaced on the routing
session and as the `topology_road_unavailable_reason` debug key, while
the snapshot still loads and routes.
`topologyRoadFingerprint(batch)` gives roads their own order-independent
identity for cache provenance; the routing construction fingerprint is
unchanged and stays road-free through the persisted path.
Lean (formal/topology/Yams/Topology/Artifacts.lean): `ClusterRoad`,
`WellFormedRoads` (endpoints in clusters, no self loop, non-negative
weight, bounded per source), `constructionIdentity`, and lemmas that the
construction identity and `WellFormedBatch` are independent of roads and
that an empty road layer is well formed. Checked with lake (lean 4.31.0).
Tests: tests/unit/topology/topology_road_sidecar_catch2_test.cpp (codec
round trip, foreign/truncated payload rejection, main payload identical
on/off, validation reasons, store round trip, missing sidecar, every
corrupt-sidecar case with routing still admitted, write-before-pointer,
retention, fingerprints through the persisted path) and the extended
fingerprint invariance test in topology_roads_catch2_test.cpp.
`road_to` knowledge-graph edges are deferred until a consumer needs them.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
dcdeaa81d0 |
feat(topology): accumulate roads at build (in memory) (#301)
Add cluster roads: undirected cluster-to-cluster links whose weight is
the sum of every raw neighbor observation between members of two
clusters. The shared helper `accumulateClusterRoads` in
topology_build_utils.h reads `TopologyDocumentInput::neighbors` before
the reciprocal/minEdgeScore filter, because the connected engine's
filtered adjacency only carries cross-cluster edges where an oversized
component was split, and k-means never consults the graph.
Each cluster keeps its top-K roads (weight desc, then target cluster id
asc) as `ClusterRoad {fromClusterId, toClusterId, weight}` on
`TopologyArtifactBatch::roads`, sorted by source cluster. Both the
connected engine and buildBatchFromAssignment (kmeans/louvain) call it;
an incremental update clears roads until the next full rebuild.
The knob is `TopologyBuildConfig::roadMaxPerCluster`, resolved from
TOML `topology.road_max_per_cluster` through the topology.* resolver
block and `TopologyManager::setRoadMaxPerCluster`. The default 0
computes no roads; the binary codec and the routing construction
fingerprint ignore the field, so existing snapshots and fingerprints
are unchanged with roads on or off.
Tests: tests/unit/topology/topology_roads_catch2_test.cpp (raw vs
filtered neighbors, top-K and tie-break, order independence, both
orientations, out-of-batch and non-finite scores, kmeans and louvain,
codec bytes and fingerprint unchanged), plus resolver and manager
setter coverage.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
db53357462 |
feat(search): RouteOnly policy behind an evaluation gate (#300)
Add TopologyVectorPolicy::RouteOnly: the vector stage runs only the routed
candidate queries (DocumentTopK on Simeon PQ, ExactDocumentComplete on
ExactScan) with no global fill and no admission gate, reports the routed
diagnostics as vector search cost and applies the routed work to the
certificate. Results come solely from the routed set.
RouteOnly serves uncertified routes, so it runs only behind
search.evaluation.allow_uncertified_routes (SearchEngineConfig::
allowUncertifiedRouteOnly). Without the gate the policy downgrades to
Shadow and records topology_vector_policy_downgraded=uncertified.
Vec0 has no routed candidates: the global leg is returned unchanged and
the source reports unavailable.
Every search now records topology_route_certified (the certificate admits
a hard route after the observed work is applied) and
vector_candidate_source_effective (ann | route_only | augment | narrow |
unavailable | none); the sequential path reports ann.
Wiring: search.topology.vector_policy accepts route_only;
ConfigResolver::TopologyRoutingPolicy carries the gate;
SearchEngineManager applies both and warns when route_only is configured
without the gate; applyTopologyPolicyFrom copies the gate so corpus
tuning can never select RouteOnly or open it. Defaults are unchanged.
Bench: YAMS_BENCH_TOPOLOGY_VECTOR_POLICY=route_only writes both TOML keys.
Also fix the stale Narrow/Vec0 doc comment on the policy enum.
Tests: new catch2_search_vector_stage_route_only pins RouteOnly x
{SimeonPqAdc, ExactScan, Vec0} x gate on/off, a certified route and an
executor-backed run; the gate-off cells are compared key for key against
Shadow. ConfigResolver, SearchEngineConfig and SearchTuner tests cover
the two TOML keys and the tuner invariant. The PR 2 goldens are unchanged
(692 assertions).
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
f7ca95427d |
fix(daemon): run memory_sync catch-up sweeps only when a store changed (#295)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* test(daemon): memory_sync catch-up sweeps must go idle on unchanged stores The vector and topology catch-up sweeps restart a full pass as soon as one ends, so an idle node reads and re-checks every node, edge, vector, and committed topology record forever. On a 3-node mesh limited to 2 CPUs each this held every node at ~24% of a core with nothing changing. New cases drive the production backfill through ServiceManager: - after a completed pass with no writes, further cycles serve no sweep item and take no graph-store connection; - a node, edge, and vector written after a pass are published by the next pass while unchanged records keep their committed operation; - a node deleted after the sweeps went idle is retracted with its edge. Store seams: KnowledgeGraphStore::changeGeneration() and VectorDatabase::changeGeneration() move on writes and batch commits, not on reads. Red: catch2_memory_sync_apply "[idle]" 3 failed (items 52 == 32, acquisitions 157 == 97); kg_store and vector_smoke do not compile. Signed-off-by: trvon <me@trevon.dev> * fix(daemon): run memory_sync catch-up sweeps only when a store changed The vector and topology catch-up sweeps in publishMemorySyncBackfill restarted a full pass as soon as one ended. An idle node therefore read and re-checked every node, edge, vector, and committed topology record forever, and every published record cost each peer a delta exchange and an apply. On a 3-node mesh limited to 2 CPUs each this held every node at ~24% of a core with nothing changing; on a real corpus the first pass never ends. Each store now exposes a change generation: KnowledgeGraphStore counts write transactions (single writes through the write pool and write batch commits), VectorDatabase counts inserts, updates, and deletes. A sweep records the generation a pass began at and, once the pass completes, does not start another until the generation differs. Writes behind a running pass's cursor move the generation past the pass's own reading, so the following pass still covers them; deletes move it too, so the retraction phase still runs after a rebuild drops records. With nothing changed, a cycle costs each sweep one atomic read. No config knob: the gate is a property of the stores, not a tunable. Tests: catch2_memory_sync_apply (14 cases, incl. 3 new [idle] cases), catch2_kg_store, catch2_vector_submodule. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
777e35adae |
feat(profiling): Tracy for every library, zones on p2p and memory-sync, per-test CPU/RSS (#294)
* feat(profiling): Tracy zones on p2p and memory-sync background paths Zone the memory-sync cycle (publish, sync, reconcile, apply, backfill, delete-outbox drain), the p2p reconnect cycle, accept loop, per-session worker and frame read, and the delta send/apply/bootstrap stages. Plot the retained session count and pending erase intents. No behavior change; the macros are no-ops without Tracy. Signed-off-by: trvon <me@trevon.dev> * test(infra): per-test-case CPU and peak RSS accounting for Catch2 binaries A Catch2 listener in the shared test main records getrusage (GetProcessTimes on Windows) CPU per case and prints the five heaviest cases plus the run's peak RSS to stderr at run end. YAMS_TEST_RUSAGE=1 adds one line per case. The measurement helpers are header-only and unit tested. Signed-off-by: trvon <me@trevon.dev> * fix(vector): make vector_database.cpp compile with Tracy enabled The Profiling lane did not build: eight YAMS_PLOT sites in vector_database.cpp used -> on a Result<T>, and two passed an int that is ambiguous between Tracy's PlotData overloads. Use .value() and int64_t casts. No behavior change outside TRACY_ENABLE builds. Signed-off-by: trvon <me@trevon.dev> * build: enable Tracy for every yams library in the Profiling lane Define yams_tracy_args/yams_tracy_deps once in the root meson.build (Tracy found, optional rocprofiler hooks, link check) and apply them to every library and executable under src/, replacing the per-library copies in search, topology, vector, daemon and tests/benchmarks. Empty when enable-profiling is off. Make YAMS_PLOT accept any numeric type and fix a Result deref in search_service so the newly instrumented TUs compile with Tracy. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
de84edde1a |
fix(daemon): wake paused post-ingest pollers on resume and capability changes (#290)
A paused or admission-blocked poller slept for the full max idle delay instead of waiting on its registered timer, so resumeStage, resumeAll, enabling the knowledge graph, and adding entity or title providers took effect only after the delay elapsed. The poller now waits through awaitPressureLimitedPollerWake and the setters signal the stage timer. Tests: pressure_limited_poller and post_ingest_queue_unit. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
2d162793cb |
fix(daemon): count active requests, not connections, for idle detection (#289)
TuningManager treated any open non-health connection as activity, so a client that kept its socket open held the daemon in busy mode and kept background scaling up. RequestHandler now counts executing non-health requests with an RAII guard, SocketServer wires the counter into DaemonStats, and the idle/busy decisions read it. Tests: request_handler_status_unary, socket_server_teardown, and two slow real-service tests (tuning_registry_activity, socket_proxy_request_activity). Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
01fe25f1ab |
fix(ipc): preserve search stats in streamed responses (#287) | ||
|
Trevon
|
52aa700144 |
fix(graph): --scope-cwd clusters, --name lookup, --limit (#280 follow-ups) (#284)
* test(graph): cover --topology-clusters --scope-cwd outside src/include Follow-up to #280. --topology-clusters --scope-cwd still built its scope from <cwd>/src and <cwd>/include, so a corpus laid out any other way had no scoped members: - The scope path set must hold every stored path under the cwd, also when the cwd is reached through a symlink, and not a sibling that shares the prefix. - GraphTopologySupport must build the set from its own metadata repo. The existing cluster-scope case built its path set by hand from the old src/include contract. It now builds the set from /repo, so /repo/tests/ is in scope and a document outside /repo is not. Signed-off-by: trvon <me@trevon.dev> * fix(graph): scope --topology-clusters --scope-cwd to the whole cwd Follow-up to #280. Topology views built their --scope-cwd path set from <cwd>/src and <cwd>/include only, a leftover from the code-symbol graph removed in v0.20. Corpora laid out any other way reported no scoped members. Build the set from the same roots as the --list-type ranges: the cwd, lexical and symlink-resolved. Keep only paths really under a root, since the prefix query matches with LIKE. The topology view no longer prunes build/, packages and IDE paths, so it matches --list-type. GraphTopologySupport now builds the set from its own metadata repo instead of the CLI app context. Remove buildGraphCodeScopePathSet and buildGraphCurrentScopePathSet, which have no other callers, and use one "Scoped to paths under the current directory" note for both views. Signed-off-by: trvon <me@trevon.dev> * test(cli): cover graph --name on a document with a path:file: node Follow-up to #280. yams graph --name <path> tried path:file:<path> node keys before the document lookup. v0.20 ingestion writes path:logical: and path:<snapshot>: keys, so the probe misses on a fresh corpus; after `yams doctor repair --graph` (or NL entity extraction) the key exists, and --name printed a raw traversal of the file node instead of the document's related documents. Smoke cases through the in-process CLI: - --name on a file with a path:file: node returns its hash and its semantic neighbours (red). - A relative --name resolves under the cwd even though a newer copy with the same relative path exists in another checkout. - A bare file name still resolves by name. The portability allowlist entry for this file moves with its getenv line. Signed-off-by: trvon <me@trevon.dev> * fix(cli): resolve graph --name through the document lookup only Follow-up to #280. yams graph --name first traversed path:file:<path> node keys. v0.20 ingestion writes path:logical:<path> and path:<snapshot>:<path>, which only chain to blob versions; semantic neighbours hang off doc:<hash>. On a fresh corpus the probe always missed. After `yams doctor repair --graph` or NL entity extraction it hit and printed a raw file-node traversal instead of the related documents. Drop the probe and go straight to the document lookup, which starts at doc:<hash> since #280. Resolving to path:logical: instead would show the version chain, not related documents; --node-key still traverses any node. Send --name as the path ingestion stores: relative to the client cwd, lexically normalized, symlinks resolved. The daemon resolved relative names against its own cwd and fell back to a suffix match, which picks the newest copy across checkouts. Fall back to the name as given when it does not resolve, so bare file names and suffixes still work. build_graph_file_node_candidates had no other callers; its tests are replaced by tests for buildGraphDocumentNameCandidates. Signed-off-by: trvon <me@trevon.dev> * test(graph): cover --limit on the document graph lookup Follow-up to #280. yams graph <hash> and --name ignored --limit: the daemon capped the graph query at 100 nodes and the same-directory fallback at 20 documents. Smoke cases through the in-process CLI: - --limit 1 returns one related document by name and by hash, and the first of the uncapped list. - --limit 1 caps the same-directory fallback for a document without graph edges. - Without --limit the related lists are unchanged. Signed-off-by: trvon <me@trevon.dev> * fix(graph): pass --limit through the document graph lookup Follow-up to #280. yams graph <hash> and --name sent --depth but not --limit; DocumentService fixed the graph query at 100 nodes and the same-directory fallback at 20 documents. Add GetRequest.graph_limit (proto field 27) and RetrieveDocumentRequest::graphLimit, and pass them from the CLI through the dispatcher. A limit caps the related list, graph neighbours, path history and the same-directory fallback alike. The graph query never asks for fewer than 100 nodes, since path and blob nodes count toward that limit. The CLI sends a limit only when --limit is given; 0 keeps the old caps, so default output is unchanged. The binary GetRequest layout appends the field and still reads payloads without it. Unit tests cover the proto round trip, the dispatcher mapping and the DocumentService caps. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
44b2f4f046 |
fix(graph): document lookup and --scope-cwd (#280) (#282)
* test(graph): cover document lookup and --scope-cwd listing (#280) Red tests for the two graph CLI bugs in #280: - GetResponse graph fields and file name must survive the proto wire. - DocumentService graph lookup by hash and by name must return the same documents as traversing the doc:<hash> node. - The document graph JSON must name the document when only name is set. - KG path-range scoping must match document nodes by label and snapshot path nodes by $.path. - listByType --scope-cwd must keep document and path nodes under the client cwd (including via a symlinked cwd) outside src/ and include/. The existing code-roots dispatcher case now places its out-of-scope file outside the scope root, since the scope is the whole cwd. Signed-off-by: trvon <me@trevon.dev> * fix(graph): fill document lookup and scope --scope-cwd to the cwd (#280) yams graph <hash> and --name returned graphEnabled=false with no related documents while --node-key doc:<hash> listed neighbours: - The GetResponse proto binding dropped graphEnabled, related and fileName, so socket clients never saw graph data. - DocumentService started traversal at blob:<hash>, which has no outgoing edges, and filtered out semantic_neighbor. Start at the doc:<hash> node and allow semantic_neighbor. - GraphQueryService did not hydrate documentHash for doc: nodes, so reached documents were skipped. - The presenter read only fileName for "document"; fall back to name. --list-type <t> --scope-cwd returned nothing: - The daemon scoped to <cwd>/src and <cwd>/include only. Scope to the whole cwd, lexical and symlink-resolved. - The KG range predicate ignored document labels and $.path on snapshot path nodes. Match both. Signed-off-by: trvon <me@trevon.dev> * docs(skills): correct graph document lookup examples --format dot is only rendered for node traversal, so show it with --node-key doc:<hash>. Describe --scope-cwd as cwd-scoped. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
4e97b31fef |
chore(warnings): clean up compiler warnings on GCC, Clang and Apple Clang (#279)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* chore(warnings): default-initialize aggregate members left out of brace-init
GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): remove or annotate unused parameters, functions and variables
- init_command, s3_signer test, request_handler_lifetime test: comment out
the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
__APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
work_coordinator test: mark platform-conditional or busy-loop
variables [[maybe_unused]].
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests
- search_admission_controller: count started workers in std::uint32_t so
the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
temporary std::string is created for the call (GCC
-Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
which continues the comment (-Wcomment); show the env export on its own
line.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): drop redundant std::move of a const captured json
The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): mark __int128 use in db_integrity_stamp as an extension
-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): remove always-true unsigned >= 0 comparisons
GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
|
||
|
Trevon
|
c248863b76 |
fix(core): correct feature-test detection in cpp23_features.hpp (#278)
- Include <version> before testing __cpp_lib_* macros. Without it every library gate read 0 in a file that included this header first (verified with Apple Clang 21 and GCC 16 at -std=c++23). - Detect [[likely]]/[[unlikely]] with __has_cpp_attribute. __cpp_attributes is 200809L on every compiler, so YAMS_HAS_LIKELY_UNLIKELY was always 0. - YAMS_HAS_REFLECTION now requires __cpp_impl_reflection and __cpp_lib_reflection >= 202506L (P2996R13). __cpp_reflection is not a standard macro. - Remove YAMS_HAS_PROFILES and YAMS_PROFILE. __cpp_profiles does not exist and C++26 did not adopt Profiles; nothing outside the test used it. - YAMS_CPP_VERSION reads _MSVC_LANG when defined. - get_feature_summary() reports the language mode instead of tying "C++23" to constexpr vector/string, which are C++20 features. - Correct the per-gate toolchain notes (flat_map: libstdc++ 15 / libc++ 20 / MSVC 19.51; move_only_function: not in libc++; expected: Clang 18 with libstdc++ reports 0) and drop the std::expected migration guidance: Result<T> stays. Tests: core_cpp23_features checks each gate against the compiler's own feature-test macros with the header included first. New core_cpp23_features_fallback forces every gate to 0 so the fallback paths, including the magic_numbers runtime path, compile in every lane. YAMS_CONSTEXPR_IF_SUPPORTED's inline fallback was invalid on a lambda; the tests now use it on a function. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
c26294a938 |
feat(topology): opt-in medoid representative and BQ rotation (#277)
* test(topology): characterize degree representatives and raw BQ sign bits
Pin the current behaviour before adding opt-in representative and BQ
rotation options:
- connected, louvain and kmeans engines on a skewed cluster (dense
clique plus a weak-edge hub): medoid hash, score, extra routing
representatives and roles;
- equal-degree ties go to the smaller document hash;
- the unrotated BinaryQuantizedIndex matches a raw-coordinate sign-bit
reference, with and without a prefix dimension.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in medoid representative rule
topology.representative_rule = degree (default) | medoid selects how a
cluster's representative document is chosen. Degree keeps the max
within-cluster weighted degree. Medoid picks the member minimising
sum_j sqrt(max(0, 2(1 - cos))) over the other members.
- one shared helper, applyRepresentativeRule(), is called by the
connected, louvain and kmeans engines after their unchanged degree
choice; medoid falls back to it without usable geometry;
- clusters above 256 usable members are searched on a deterministic
hash-ordered stride sample (kMedoidRepresentativeMaxMembers);
- ties within 1e-9 go to the smaller document hash;
- with medoid, farthest-first extra routing representatives are seeded
from the medoid instead of the centroid;
- the rule is stored in the formerly reserved first header byte of the
binary snapshot (0 = degree, so old snapshots keep their meaning) and
extends the construction fingerprint only when it is not degree;
- search debug stats report topology_representative_rule.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in randomized Hadamard rotation for BQ routing
search.topology.bq_rotation = none (default) | fwht selects the rotation
applied before the centroid BQ shortlist takes sign bits. Raw-coordinate
sign bits make cos(pi * h / d) a poor estimate on anisotropic embeddings;
after x -> (1/sqrt(n)) H D [x; 0] every bit is a random hyperplane.
- RandomizedHadamardRotation in yams/vector: fixed splitmix64 sign
diagonal plus an in-place FWHT, zero-padded to a power of two; Simeon's
FWHT projection works on int32 sketches with row subsampling, so a
small local transform is used instead;
- BinaryQuantizedIndex::build() takes the rotation and seed, rotates
stored vectors before the prefix is taken, and rotates every float
query itself, so centroids and queries always share one seed
(kDefaultBinaryRotationSeed);
- the routing snapshot cache records bqRotation and bqRotationSeed and
rebuilds the route index when either differs;
- the routing policy fingerprint changes only for a non-default rotation;
- search debug stats report topology_route_bq_rotation.
Signed-off-by: trvon <me@trevon.dev>
* test(bench): map representative rule and BQ rotation arm knobs
Harness-only env knobs written into the isolated benchmark config.toml,
so arms go through ConfigResolver like the other topology settings:
- YAMS_BENCH_TOPOLOGY_REPRESENTATIVE_RULE=degree|medoid
-> [topology] representative_rule
- YAMS_BENCH_TOPOLOGY_BQ_ROTATION=none|fwht
-> [search.topology] bq_rotation, next to bq_candidate_limit and
bq_prefix_dim
The debug-file setup entry records the requested values; per-query
search_stats carry the effective topology_representative_rule and
topology_route_bq_rotation. Portability allowlist lines are shifted for
the inserted code.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
1d1e8def4d |
fix(search): count augment route members as added candidates (#276)
* test(search): pin augment routed-candidate accounting Under the augment vector policy a document reachable only through a topology route reaches the fused results through the candidate-rescue vector merge, but the routing session and the topology_weak_query_added_candidates stat report zero additions, because the product stage always collects route membership and the session skips the add loop in that mode. Pin the routed-only fixture (x2 in the route, outside lexical Tier-1 and the global vector top-1) for augment, and characterize shadow (identical to topology-disabled) and uncertified narrow (global, no additions). Signed-off-by: trvon <me@trevon.dev> * fix(search): count augment route members as added candidates #57 unified topology onto one evidence path: the product assist stage always collects route membership, and augment, shadow and narrow all consume the certificate's allowed set (augment vector-scores it through the candidate-rescue merge, shadow measures it, narrow filters to it). The session skips the old add loop in that mode, so under augment addedCandidates, duplicateCandidates and topology_weak_query_added_candidates stayed 0 even when routed documents were rescued into the fused results. Under augment only, count materialized allowed-set members that are new to the existing candidate pool as added candidates and the rest as duplicates. The allowed set, its maxDocs bound, the routing policy fingerprint, applied and skip reasons are unchanged; narrow and shadow report no additions as before. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
4ec044afd6 |
fix(vector): rank vec0 by cosine on unit-normalized copies (#275)
* test(vector): pin vec0 ranking to cosine order on non-unit rows vec0 ranks by L2 distance, which matches cosine order only for unit vectors. Insert rows whose L2 order disagrees with cosine order and assert vec0 returns cosine order, agrees with the exact cosine scan, skips zero rows and rejects a zero query. Pins a vector-enabled env so the cases run under the CI test env. Signed-off-by: trvon <me@trevon.dev> * fix(vector): rank vec0 by cosine on unit-normalized copies vec0 ranks by L2 distance, which matches cosine order only when both sides are unit length. Stored rows reach the vectors table at whatever magnitude the provider, config, import or sync produced, and the vec0 aux table copied them as-is, so vec0 selected and ordered rows differently from the Simeon PQ and exact cosine paths. The vec0 table is a derived index rebuilt from vectors.embedding, so write the unit direction into it and leave the source blob, which export/sync and exact scoring read, untouched. Normalize the query the same way. Zero rows are left out of vec0 and never match; a zero or non-finite query is rejected as the exact scan rejects it. Returned rows are ordered by their cosine score with the PQ tie-break. No schema change. The aux table is rebuilt on first use in each process (ready dims are in-memory), so existing non-unit rows are normalized on the next rebuild without a repair step. Signed-off-by: trvon <me@trevon.dev> * test(topology): pin the k-means stored centroid to the unit direction Spherical k-means assigns against unit centroids but stored the shorter arithmetic mean. Assert the stored centroid is unit length along the member mean direction, with and without boundary spill. Add a SOAR case showing that with unit centroids the spill follows angular distance, and that a short mean centroid reorders the candidates. Reformat the touched k-means test file to the lint formatter. Signed-off-by: trvon <me@trevon.dev> * fix(topology): store the unit k-means centroid The k-means engine clusters by cosine against normalized centroids, then stored the unnormalized member mean as centroidEmbedding. Routing reads the centroid by cosine and does not care, but SOAR boundary spill measures Euclidean residuals against it. A loose cluster's short mean adds a radial term to those residuals, so spill candidates were ranked in a different space from the one the clusters were built and routed in. Store the unit mean direction for k-means, which is the converged Lloyd centroid. Connected-component and Louvain keep the arithmetic mean, as their tests and contract require. Correct the stale field comment. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
8000226a4c |
fix(wal): encode the entry header field by field, never via padding (#274)
* test(wal): pin v1 entry encoding to a Clang-written fixture
store_block_entry_clang_v1.bin is one StoreBlock entry serialized by the
current wal_entry.cpp built with Clang 21, which zero-fills the 4 bytes of
trailing Header padding. The new cases decode and verify it, re-encode the
same logical entry and require identical bytes, and round-trip fresh
entries through serialize/deserialize/verifyChecksum.
GCC 16 leaves that padding uninitialized in stableHeaderBytes(), so these
cases fail there.
Signed-off-by: trvon <me@trevon.dev>
* fix(wal): encode entry header field by field, never via padding
stableHeaderBytes() built the checksummed header with `Header stable{}`
plus member assignments and then memcpy'd sizeof(Header) bytes. That is
aggregate initialization, so the 4 bytes of trailing padding (offsets
44..47) are not guaranteed to be zero. Clang 21 memsets the object first;
GCC 16 at -O0 stores only the members, so stack garbage reached both the
serialized entry and the CRC, and verifyChecksum() failed on GCC builds.
Encode and decode the header field by field into a zero-filled 48-byte
array at the existing offsets, with static_asserts pinning every offset
and size. The checksum is CRC32 over that canonical encoding plus the
payload, which is exactly what Clang builds have always written, so their
WAL files verify unchanged. deserialize() also accepts an entry whose CRC
covers nonzero padding as stored on disk (written by earlier GCC builds)
and normalizes its in-memory checksum. WALFile and WALManager recovery
read headers through the new WALEntry::decodeHeader().
TransactionData (uint64 + uint32) has 4 bytes of trailing padding and was
copied whole into the payload, relying on the same unguaranteed zeroing
(GCC 16 and Clang 21 happen to zero it today); it is now encoded field by
field too. The remaining payload structs are padding-free, which
appendObjectBytes/readObject now enforce with
has_unique_object_representations_v.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
1b6671528d |
fix(memory-sync): publish committed vectors and wrap the vector sweep (#266)
* test(memory-sync): cover vectors committed behind the backfill cursor The vector backfill cursor (document_hash, chunk_id) never wraps, and the backfill is the only outbound path for vectors, so a vector added or re-embedded behind the cursor after the first sweep is never published to peers. Add red tests for: - the sweep starting over to publish a new and a re-embedded vector that sort below its last position; - vectors an embedding job commits being published within the next backfill cycle, ahead of a sweep that would need many items to reach them; - the topology sweep still publishing its share while the vector sweep always has work. Declare ServiceManager::notifyMemorySyncEmbeddingsCommitted as a no-op seam for the embedding commit notification; the fix fills it in. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): publish committed vectors and wrap the vector sweep The vector backfill cursor (document_hash, chunk_id) was set once and reset only by a daemon restart, and the backfill is the only outbound path for vectors. After the first full sweep, a vector added or re-embedded under a document hash that sorts below the cursor was never published to peers. Topology had the same bug and now sweeps in repeating passes; vectors did not. Eager publish: the embedding service already reports the documents a job committed vectors for (successHashes, after the vector insert and before readiness is published) through the callback that requested the topology rebuild. That callback is renamed setEmbeddingsCommittedCallback and the daemon now also passes the hashes to ServiceManager::notifyMemorySyncEmbeddingsCommitted. It only queues them (bounded at 4096 documents, deduplicated, dropped past the cap): it runs on the embedding worker, and the memory-sync service's lifetime and outbound budget belong to the sync cycle. The backfill serves the queue as its own domain right after documents, one vector row per item, walking each document's chunks with getVectorsPage. The queue is in memory; a restart loses it, but the sweep then starts over anyway. Catch-up sweep: when the vector sweep reaches the end of the table it resets its cursor and sits out the rest of the cycle, as the topology pass does, so the next pass publishes whatever changed behind it. publishIfChanged keeps repeat passes from writing unchanged records. R7's skip logic is kept: an unpublishable row is skipped and the cursor moves past it; it is logged on the first pass (and when committed) but not again on every later pass. Fairness: with both sweeps repeating, both always have work, so the strict priority order (documents, vectors, topology) would hand the vector sweep every item and starve topology. Documents and committed vectors keep priority; the two sweeps now alternate item by item, and the turn persists across cycles so a one-item budget still alternates. A sweep that is done for the cycle leaves its turns to the other. This was chosen over alternating whole passes because it needs one field of state, keeps both sweeps' latency bounded by their own size rather than the other's, and reuses the existing per-item budget loop unchanged. Signed-off-by: trvon <me@trevon.dev> * test(memory-sync): pin the vector-store environment in the cursor-wrap cases Same as the bad-record cases: under the CI environment (vectors disabled, sqlite-vec init skipped) VectorDatabase creates no tables and the vector inserts fail. On grim the 11 memory_sync_apply cases pass both in a clean environment and with the tests.yml environment. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
44daa47b94 |
fix(memory-sync): skip bad vector records instead of stalling the stage (#265)
* fix(ci): size the onnx resource budget for aarch64 page alignment libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget. Signed-off-by: trvon <me@trevon.dev> * fix(plugins): declare the S3 legacy factory with the plugin export macro |
||
|
Trevon
|
dc7a4fb499 |
fix: unblock the 0.20.5 release (aarch64 budget, MSVC linkage, mobile configure, Windows MSI) (#263)
* fix(ci): size the onnx resource budget for aarch64 page alignment libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget. Signed-off-by: trvon <me@trevon.dev> * fix(plugins): declare the S3 legacy factory with the plugin export macro |
||
trvon
|
a8f891879f
|
feat(cli): drive the packaged systemd user unit when it is installed
With the daemon shipped as a systemd user unit, the CLI must not fight systemd for it: - Auto-start (DaemonClient::startDaemon): when the unit is installed but not running and would run exactly the daemon the client expects (same socket, data dir and config as the default resolution, default binary), start it with `systemctl --user start` instead of forking a copy that a later login would race for the socket. Flag or environment overrides (--socket, --data-dir, YAMS_DAEMON_SOCKET, YAMS_DATA_DIR, YAMS_CONFIG, YAMS_DAEMON_BIN) keep the fork path. - `yams daemon start|stop|restart` use `systemctl --user` for an installed/active unit (start/restart without custom path options, stop/restart without --socket), falling back to IPC if systemctl fails. - `yams daemon install --user` with no options enables the packaged unit (`systemctl --user enable --now`) instead of writing a copy into ~/.config/systemd/user that would shadow it; with options it still writes that override and says so. `uninstall --user` reloads the manager after removing an override and explains how to mask the globally enabled packaged unit for one account. The decision is a pure function (userUnitMatchesPaths) with Catch2 coverage; systemctl is only run with whitelisted verbs and only when a user manager is reachable ($XDG_RUNTIME_DIR set, Linux). Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
0ba93ab031
|
revert(cli): drop system-daemon socket discovery and content upload
The packaged service becomes a systemd user unit that runs each user's own daemon on the CLI's default per-user socket, with that user's config and data. There is no system daemon left to discover, and the daemon always runs as the same user as the CLI, so both workarounds for the system-service design go: - Revert "feat(cli): find the packaged system daemon socket without configuration" (client-side /run/yams/yams-daemon.sock discovery, the systemd refusal in `daemon stop|restart`, the spawn refusal and the status mismatch suppression, plus its tests). - Revert "fix(cli): send file content to a daemon that runs as another user" (peer-UID probe and content upload in `yams add`). Per-user socket resolution is back to: explicit flag/env/config, then $XDG_RUNTIME_DIR/yams-daemon.sock (or the /tmp fallback). Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
fe07469620
|
feat(plugins): prefer a system ONNX Runtime, fall back to a bundled copy
The ONNX plugin dlopen()ed ONNX Runtime from directories next to the
plugin first and only then from a few system prefixes, with no notion of
which copy it got or why, and the shipped packages contained no copy at
all, so on most hosts the plugin could only report "runtime not found".
The runtime resolution moves into plugins/ort_runtime/, built as a
hidden-visibility static library both ONNX-using plugins link (their
export maps still expose only the plugin C ABI). The policy is split out
as pure code (ort_runtime_resolution.{h,cpp}) so it is testable without a
runtime. Candidates are tried in this order and the first compatible one
wins:
1. YAMS_ONNX_RUNTIME_LIB / YAMS_ONNX_RUNTIME_DIR (existing pin, kept),
then the plugin config key runtime_library ([plugins.onnx] /
[plugins.glint], a file or directory)
2. a system copy: the loader search by soname, then the usual library
directories (multiarch dir, /usr/lib64, /usr/lib, /usr/local/lib,
/opt/onnxruntime, Homebrew) for versioned names such as Debian's
libonnxruntime.so.1.23
3. the private copy in <libdir>/yams/onnxruntime/
4. legacy locations next to the plugin (build trees)
A candidate is accepted only if it exports OrtGetApiBase, GetApi() returns
an API table for the ORT_API_VERSION the plugins are compiled against, and
it meets the 1.23 minimum; otherwise it is skipped with a reason. A soname
lookup that returns a copy another plugin already loaded from the bundled
directory is reported as that copy. The choice ("Using system|bundled ONNX
Runtime <ver> from <path> (<why>)") and every skipped candidate go to the
plugin log and, through the libyams_onnx_resource host sink (new
OnnxConcurrencyRegistry::emitHostLog), to the daemon log; plugin health
reports runtime_source. With no compatible copy the plugin reports "ONNX
Runtime unavailable: ..." instead of failing.
Tests: tests/unit/plugins/ort_runtime_resolution_catch2_test.cpp covers
the order, the override pin, configured directories, system-over-bundled
preference, skipping an incompatible system copy, the not-found result
and the already-loaded classification.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
trvon
|
4550d3bbdf
|
feat(cli): find the packaged system daemon socket without configuration
The deb, rpm and Arch packages start yams-daemon.service on /run/yams/yams-daemon.sock, but clients only looked at the per-user default (XDG_RUNTIME_DIR, /var/run as root, /tmp/yams-daemon-<uid>.sock). With no YAMS_DAEMON_SOCKET set, root and normal users silently fell back to an in-process daemon and a private store in ~/.local/share/yams, so the installed service was never used. Client-side resolution now goes: an explicit socket (flag, env, config), the per-user daemon when its socket exists or a personal data directory was named (flag, env, core.data_dir from yams init), then the system socket when it exists and is connectable (root or the yams group), and finally the per-user default. Daemons keep binding what resolve_runtime_paths() names; `yams daemon install --user` writes the per-user socket, never the discovered one. Because systemd owns the system unit, `yams daemon stop|restart` refuse it with a systemctl hint and the client never tries to spawn a daemon on that socket. `daemon status` no longer flags the system corpus as a data directory mismatch. socket_utils also gains a peer-UID probe used by the next change. The two socket default tests now use the per-user resolver so they do not depend on whether the build host runs the packaged service. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
93f2439471
|
build(plugins): export only the plugin C ABI; hide onnx_resource internals
Plugins and libyams_onnx_resource were linked with default visibility, so
their dynamic symbol tables exported everything they contained, including
statically linked third-party code. The S3 plugin exported 11,147
symbols (OpenSSL, libcurl, zlib, spdlog, nlohmann_json); the ONNX
resource library exported 895 (850 of them spdlog). Every loader uses
RTLD_LOCAL, so this was not a live interposition bug, but the export
table is the ABI and anything that binds RTLD_GLOBAL (or a future host)
could resolve into a plugin's private copies.
Plugins:
- plugins/exports/yams_plugin.map (ELF version script, global
yams_plugin_*, local *) and yams_plugin.exp (ld64
-exported_symbols_list) are applied to every plugin (s3, onnx, glint,
zyp) with gnu_symbol_visibility: 'hidden' and --exclude-libs,ALL.
- The S3 legacy factory (yams_plugin_create/destroy_object_storage) is
marked YAMS_PLUGIN_API so it stays exported under hidden visibility.
libyams_onnx_resource:
- Builds with hidden visibility; YAMS_ONNX_RESOURCE_API now expands to
visibility("default") on GCC/Clang, so only OnnxConcurrencyRegistry is
exported.
- It no longer links spdlog. Its messages reached the host log only
because the executable's spdlog interposed the library's copy; with the
copy hidden, the registry would have logged through a private default
logger to stdout. The registry now takes a host log sink
(OnnxConcurrencyRegistry::setLogSink); TuningManager and
AbiModelProviderAdapter install a spdlog forwarder from the header-only
onnx_registry_spdlog_sink.h. Without a sink, messages are dropped.
Tests (red first on grim, Debug):
- tests/plugins/plugin_export_surface_catch2_test.cpp dlopens the built
S3 and zyp plugins and asserts the entry points resolve while libcurl,
OpenSSL, zlib and zyp internals do not: 2/2 cases failed (11 of 31
assertions) before, pass after.
- resource_governance_test gains a case for the log sink routing (did
not compile before; passes after). `meson test --suite plugins`: 5/5.
Measured on grim (release, stripped staged payload):
file bytes before after exports before after
libyams_object_storage_s3.so 7,854,064 6,240,112 11,147 10
yams_zyp.so 80,112 63,728 103 8
libyams_onnx_resource.so 477,472 30,840 895 20
bin/yams-cli 33,712,056 33,859,512 80 39
bin/yams-daemon 28,389,784 28,537,304 98 39
The executables now carry their own spdlog code paths instead of
exporting them for the resource library to interpose (+0.4%), and export
41 fewer symbols each. check_install_surface turns GREEN; budgets are
reset to the measured payload plus ~20%.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
992b746b00 |
Merge pull request #252 from trvon/fix/windows-lines-hang
fix(extraction): stop the Windows lines: hang from a mismatched iterator range |
||
|
Trevon
|
8aa2a33720 |
fix(memory-sync): publish topology edge endpoints and retract deleted topology (#250)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* feat(metadata): add keyset node/edge scans and exact edge lookup The memory-sync topology backfill paged nodes with LIMIT/OFFSET and no ORDER BY, so a topology rebuild that deleted and reinserted cluster nodes between two pages shifted the offset and the sweep skipped live nodes for good. Ids in kg_nodes and kg_edges are AUTOINCREMENT and never reused, so a cursor on the last id returned is stable under concurrent inserts and deletes and survives restarts. getNodesAfterId/getEdgesAfterId scan in id order from such a cursor. getEdgesBetween returns every edge between two nodes for one relation; getEdgesFrom truncates at its page limit, which made "is this edge present" checks on high-degree nodes unreliable. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): publish topology edge endpoints and retract deleted ones On a Docker mesh that converges on documents, topology deferrals never drained: peers held member_of and semantic_neighbor edges whose endpoint node had not replicated (43/35/25 per node at 3x40 docs, ~140 at 5x80). Inspecting the kept 3x40 mesh after 20 minutes: every stale deferral named a doc or cluster node that its origin still held but had not published. The origin's sweep was still inside the "document" type: no "path", "topology_cluster" or "topology_snapshot" node had been published by any node, while the doc nodes' edges into them had been. The backfill swept node types in sorted order and published each node's edges right after the node, so an edge reached peers long before its target node of a later type; at the backfill's rate that is tens of minutes. Its paging also made some of those waits permanent: - Nodes were paged per type with OFFSET and no ORDER BY. A topology rebuild deletes every cluster node and inserts replacements, so the offset skipped live nodes whose member_of edges were published. - Per-type offsets were never reset, so a node was visited once: edges added after its sweep (semantic neighbors, new memberships) and in-place node updates were never published. - Nothing published deletions. Worse, apply() wrote this node's own winners back into its graph, resurrecting the cluster nodes and edges its rebuild had just deleted, so the deletion could not be observed. - Peers deferred an edge whose endpoint was tombstoned forever. The topology backfill now runs repeating keyset passes: every node by id, then every edge by id published together with both endpoint nodes (endpoints first, so a peer never holds a writer's edge without them), then this writer's committed topology winners by logical key, tombstoning those the local graph no longer holds. Cursors are stable ids/keys, so inserts and deletes between items cannot shift them, and a restart just begins a new pass; unchanged records publish nothing. apply() still validates every record but never applies the writer's own ones: the local graph is authoritative for them. An edge whose endpoint is known deleted (tombstoned, or an own record the local graph no longer holds) is dropped instead of deferred, and applies again if the endpoint is re-published. Edge lookups during apply use the exact endpoint query instead of a page-limited adjacency scan. Signed-off-by: trvon <me@trevon.dev> * perf(memory-sync): publish documents before derived records in backfill With the topology sweep now publishing every node and edge, a 3-node x 40-doc mesh took longer until every document was listed on every node (603 s against 426 s before the topology fix). Debug logs showed why: a publish takes most of the backfill's 100 ms per-cycle time budget, so each cycle published about one item, and the fair rotation over documents, vectors and topology handed two of every three cycles to derived records while peers waited for the documents. The backfill now serves domains in priority order: documents, then vectors, then topology, each taking items until it has nothing left this cycle. Vectors and topology defer on a peer until their document arrives, so publishing the document first never adds a deferral. In the same mesh, run concurrently with both other builds, every document is listed on every node after 140 s (426 s before the topology fix, 603 s with it alone), and deferrals still drain to 0. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
e0d15dc680 |
fix(p2p): stop quarantining honest writers for concurrent-session races (#245)
* feat(p2p): log durable writer quarantines and outbound session failures
A node that durably quarantined a peer writer left no trace of why: the
three p2p quarantine sites returned an error to their session, and the
reconnect loop dropped every outbound session error without logging it,
so a mesh that stopped receiving one writer showed only a quarantined
record count. Outbound sessions are half of all sessions in a full mesh.
Each p2p quarantine site now logs the writer, the failed check and the
local and frozen counters, and a failed outbound session logs its peer
and error, matching the existing inbound failure log.
Signed-off-by: trvon <me@trevon.dev>
* fix(p2p): stop quarantining honest writers for concurrent-session races
Each pair of direct peers normally runs two sessions at once, one
initiated from each side. Both freeze a window of the peer's writer at
handshake, and whichever applies first advances the receiver's durable
prefix under the other. The second session then compared its frozen
window with that live prefix and treated the difference as forged
history: validateHistoryExtension demanded the window start right after
the live commitment ("history extension is not contiguous"), and an
empty window demanded the live counter equal the frozen frontier. Both
durably quarantined an honest writer, after which the node dropped its
documents and refused every later session with it. A 5-node mesh run
logged exactly this (local=30 frontier=32 staged=4: a window frozen at
28 arriving after another session had applied 29-30), and the mesh
never converged.
The receiver now validates a window against durable history instead of
a snapshot. Operations at or below the advanced prefix are replays that
applyDeltas leaves untouched (a different record under a known operation
id is still a fork). The rest must chain from the durable commitment
exactly to the authenticated frontier, and an empty window must match
the durable commitment at its frontier. validateHistoryExtension returns
a typed WriterHistoryViolation for writer faults (fork, forged frontier,
gap, unauthenticated record) and an Error for local or transient
failures, and only a violation quarantines; an Error aborts the session
for a retry on the next cycle.
The security property is unchanged: nothing is applied unless a hash
chain links durable history, itself verified against the writer's
authenticated commitments, to the frontier the writer authenticated in
this handshake. Overlapping replays change no state, and forks, forged
frontiers, gaps and unauthenticated records still quarantine, now also
when they overlap an advanced prefix (new tests cover each).
The pre- and post-exchange history checks get the same treatment. A
live prefix between the proven handshake prefix and the frozen frontier
is consistent, since delta exchange verifies the rest; one beyond the
frontier is compared with the durable commitment at the frontier rather
than rejected as a rollback.
The handshake also clamps the live local-writer window resolver to the
frozen snapshot, so concurrent local writes no longer fail the session
with "local writer-window resolver violated negotiated bounds".
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report outbound sessions and durable writer quarantines
`yams p2p status` showed inbound session failures but not outbound
ones, and only a `quarantined` record count that mixes durable writer
quarantine with transient rejections (a record whose causal dependency
from a third writer has not arrived yet). An operator could not tell a
mesh that was retrying from one that had permanently cut off a writer.
Status now reports:
- outbound_sessions / outbound_failures and the last outbound failure
with its peer, counted for both explicit connects and reconnects.
Failed sessions in either direction are aborted and retried.
- quarantined_writers: writers durably quarantined by this node, read
from the replication checkpoint, so it survives restarts. Each
quarantine is also logged with its reason.
The fields travel through MemorySyncStatus, MemorySyncResponse and
protobuf fields 38-42 to the text and JSON status and to `yams daemon
status -d`, following the apply-health fields.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): verify replayed writer counters against durable history
validateHistoryExtension skipped every staged operation at or below the
writer's durable prefix as a replay without looking at its record hash,
leaving fork detection to applyDeltas. applyDeltas recognizes a fork only
by an operation id it still remembers, and that memory is lost on restart
and bounded by maxTrackedIdentities. A different, internally valid record
for an already-applied counter therefore passed as a harmless replay
instead of being quarantined as the fork it is, whether the window only
replayed old counters or rode the fork along an honest extension.
Before skipping a replayed counter, compare its record hash with the
durable history entry committed at that counter (rebuilding the entry like
computeHistoryCommitmentAt when it is missing). A mismatch is a
WriterHistoryViolation, so the p2p session durably quarantines the writer;
an identical replay is still accepted as a no-op. A history entry that
cannot be read stays a local Error and only aborts the session.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): log every durable writer quarantine where it happens
Durable writer quarantine was logged only by its two p2p callers. The
duplicate-operation fork path inside applyDeltas and the shared-store scan
path quarantined silently, so the documented guarantee that every durable
quarantine logs its reason and counters did not hold, and an operator saw
a quarantined_writers count with nothing in the log to explain it.
quarantineWriter now takes the reason and is the one place that logs a new
quarantine: writer, reason, source node, and the writer's local history and
version counters. It logs nothing when the writer was already quarantined,
so a repeated request is not reported twice. The p2p callers drop their own
warnings and fold their session context (frontier, staged count, prefix
match) into the reason instead. memory_sync now declares its spdlog
dependency explicitly.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): show outbound p2p sessions in daemon status
The daemon already reports outbound sessions, failures, and the last
outbound failure, and `yams p2p status` prints them, but `yams daemon
status -d` only gained the quarantined-writers row. An outbound direction
that keeps failing is aborted and retried, never quarantined, so the
detailed status was the place an operator would miss it.
The Memory Sync section now shows an Outbound sessions row (started and
failed counts, warning when any failed) for direct-backend nodes, and a
Last outbound failure row with the peer, the error, and its age.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
7be908e5b7 |
fix(p2p): stop the memory-sync apply/publish deadlock and report apply health (#243)
* fix(daemon): never gate memory-sync publish on inbound apply applyMemorySyncWinners() ran the inbound stages content -> metadata -> vector -> topology and returned on the first stage failure, before the node's own outbound backfill. In a mesh where every node holds an inbound record whose prerequisite only a peer can publish (an embedding whose content blob has not replicated, an edge whose target node has not), every node refused to publish, so the prerequisites never arrived: a stable mutual deadlock with healthy transport and nothing converging. The outbound publish now runs after the inbound stages whatever their outcome; only a stop request ends a cycle early. Each inbound stage also runs even when an earlier one failed. Cross-stage dependencies (metadata and vectors need local content bytes, edges need their endpoint nodes) are checked per record against the local stores, so a failed stage shows up downstream as unapplied records rather than as skipped stages. Stages keep their prerequisite order so a prerequisite landing in a cycle is visible to its dependents in that same cycle. A two-node test drives real ServiceManager apply cycles over a shared store, each node holding an edge whose target only the other can publish; it converges in a bounded number of rounds only with this change. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): defer records whose prerequisite has not replicated Vector, metadata, and topology apply treated a missing replicated prerequisite as a hard error and rejected the whole batch: one embedding whose content blob had not arrived, or one edge whose target node had not, blocked every other winner in that stage on every cycle. These are ordering races between independently published records, not failures. Each adapter now defers such a record instead: it is skipped, the rest of the batch applies, and its logical key is reported through deferredKeys(). Nothing marks it applied, so it remains a winner in the replicated index and the next cycle retries it once the prerequisite lands; apply needs no pending queue of its own and loses nothing across restarts. An edge whose endpoint is being deleted in the same batch is deferred too rather than inserted against a stale node id. Malformed records, probe failures, and store errors still fail the stage. Content blobs are independent, so the content stage now attempts every blob and fails afterwards with the first genuine error; a corrupt blob no longer starves the blobs after it. A blob whose bytes are not hydrated in the sync cache yet is deferred like the other stages. Signed-off-by: trvon <me@trevon.dev> * feat(p2p): report memory-sync apply health in status The mesh deadlock was invisible: every node reported failed_cycles=0 and a climbing successful_cycles while nothing converged, because a cycle counts as successful once reconciliation completes, whatever the apply stages did afterwards. Deferred records would be just as invisible without a signal. ServiceManager now keeps an ApplyHealth record of the apply path: - deferred records per stage, and the age of the oldest one still pending, with a warning once a deferral passes ten minutes (its prerequisite likely never arrived). Ages are tracked for at most 4096 keys per stage, keeping the oldest; counts stay exact. - hard failures per stage, plus the last failure's stage, message, and age. - apply_cycles and apply_failed_cycles. failed_cycles keeps meaning reconciliation failures; a cycle whose apply stage hard-failed is counted in apply_failed_cycles instead of passing as a plain success. - publish_skipped_cycles (delete outbox preempted the cycle) and publish_failed_cycles (an outbound backfill domain failed). The fields travel through MemorySyncStatus, MemorySyncResponse, and protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the first line, an `apply` line with per-stage counts, a last-apply-failure line, and an `apply` object in --json) and to the memory-sync section of `yams daemon status`. The p2p status formatter moves behind a header so it can be tested without a daemon. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): keep content deferrals when the same scan hits a failure The content stage scans every blob and fails afterwards with the first genuine error, but the apply cycle discarded the deferred keys that the same scan collected whenever it failed. A corrupt blob therefore hid every blob still waiting for its bytes: the deferred count stayed at its previous value (0 on the first pass) and the stale-deferral warning never fired for a prerequisite that was actually missing. A stage pass now carries both outcomes. The content-blob scan moves into ContentBlobSyncAdapter::applyWinners, which reports the applied count, the deferred keys, and the first failure of one complete scan, and ApplyHealth::recordPass records the failure and replaces the deferred set together. A stage that returns an error still leaves the deferred set as last observed, since an aborted scan says nothing about prerequisites. The tracked-deferral cap already carried every still-deferred tracked key before admitting new ones, but its carry loop had a cap check that read as if it could stop early. Drop it (the tracked set never exceeds the cap) and pin the behaviour with tests that place the oldest key after more than the cap's worth of new keys and that refill a full tracked set. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
8a1f753ba3
|
fix(extraction): build TextBasicHandler capabilities from one extension set
supportedExtensions() returned its static set by value, so capabilities() constructed the extension vector from begin() of one temporary and end() of another. That range is undefined behavior. libstdc++ and libc++ release iterators happened to walk the first copy to completion, but the MSVC debug STL rejects it in _Verify_range with a modal "Microsoft Visual C++ Runtime Library" assertion dialog. HandlerRegistry::selectBest calls capabilities() for every candidate, so every lines: search and every applyExtractScope call hung forever on Windows Debug builds. Return the shared set by const reference and take begin()/end() from that one instance. New text_basic_handler unit tests pin the shared instance, the capability list, registry selection and line-range extraction; the old code also fails them under -D_GLIBCXX_DEBUG with "function requires a valid iterator range". Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
1fa2e92510
|
fix(daemon): show host pressure and deferred work in daemon status
When the daemon holds background work back for a busy host, a user looking at a stale topology or an unfinished repair scan needs to see why. Status had no host signal at all. Carry the governor's host sample through the metrics snapshot and StatusResponse: source (psi, loadavg), whether the host counts as busy, the PSI cpu/io/memory "some avg10" values or the per-CPU load average, and which deferrable work is currently held back. The proto transport piggybacks the fields on request_counts like the other string status fields; the native serializer appends them as optional tail fields. `yams daemon status` (brief and detailed) shows a Host Load row, e.g. "Busy cpu 72% · io 3% · mem 0% · deferring topology, repair", and `yams status --json` gains a host_pressure object. Document the new [tuning.resource] keys next to the other typed tuning sections. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
8c24821c96
|
fix(daemon): stop topology rebuild requests from spinning workers
requestTopologyRebuild() re-posted itself immediately whenever ingest, post-ingest, embedding or KG work was still in flight. It released its in-progress flag before re-posting, so every request that arrived while the daemon was busy (one per embedding batch) could win the flag and start its own re-post loop. Those loops never slept: they occupied the worker pool until the ingest drained, and a slower drain on a loaded host meant more of them for longer. Per-thread sampling during a 1899-file ingest on a 2x oversubscribed 48-core host showed ~36 worker threads at 100% for the whole drain. For the same ingest (120s window, background work only): daemon CPU seconds before after idle host 41, 41 23, 20, 19 loaded host 96, 95 28, 30, 30 (deferral off) Arm a single timer instead: a busy daemon re-checks after 500ms, a rebuild deferred for host pressure after 5s, and only one retry is armed at a time. post_ingest_drain still requests a rebuild directly when the pipeline empties, so topology freshness does not wait on the timer in the common case (idle-host topology completion is unchanged at ~7s after ingest). Add a ServiceManager test for the deferral path: with the host marked busy the rebuild is held and the governor reports it deferred; once the host calms down nothing is held back and, when a vector database is available, the armed retry builds the artifacts without a new request. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
963d9bdbe4
|
perf(daemon): defer background work while the host is busy
Background work started as soon as it was scheduled, no matter what else the machine was doing, so a topology rebuild or a repair scan competed with the user's own jobs for CPU at the worst time. Classify background work. Must-run work keeps the corpus consistent and is never gated: crash recovery, database integrity checks and WAL checkpoints. Deferrable work waits while host pressure is elevated: - topology rebuilds and semantic topology maintenance (the dirty set keeps accumulating; a deferred rebuild re-checks on a 5s timer) - repair maintenance: the initial scan (which is what queues embedding catch-up for documents missing vectors), the legacy snapshot backfill and the vectors.db VACUUM - semantic_neighbor backfill for older documents Embedding of newly ingested documents is not deferred. Throttling it to one job under host load was measured and dropped: on a 2x oversubscribed 48-core host it stretched the embedding drain 1.5-4x and sometimes triggered two extra vector index builds, without lowering the daemon's CPU use. Each kind goes through a DeferralGate in the resource governor. A starvation guard bounds the wait: after background_max_deferral_s (default 300s) one pass runs and the wait starts over, so sustained load slows this work to a trickle but never stops it. With the guard at 45s, a loaded run held the topology rebuild for 45s and then built it (48-53s after ingest instead of 22-33s with deferral off). Setting [tuning.resource] background_max_deferral_s = 0 turns deferral off. The governor logs when a kind starts deferring, when the guard forces a pass, and when it resumes. RepairService already gated maintenance on client activity and daemon pressure; the host check sits last in each condition so the deferral clock only measures time spent waiting on other processes. Signed-off-by: trvon <me@trevon.dev> |
||
|
trvon
|
52545d3f61
|
perf(daemon): sample host pressure in the resource governor
The governor only measured the daemon itself (RSS, own CPU), so it could not tell that the machine was busy. On a loaded host the daemon still started background work at full width, and that work ran far less efficiently: ingesting the same 1899-file corpus cost 41 daemon CPU seconds on an idle 48-core host and 96 CPU seconds with the host 2x oversubscribed (stress-ng), and took 4x longer to finish. Sample a cheap host-wide signal on the existing governor tick, at most once per second: Linux PSI "some avg10" for cpu, io and memory, falling back to the one-minute load average per online CPU; on macOS the load average plus the kernel VM pressure level. A busy host is a flag with a release band (signals must drop to 75% of the threshold to clear) so work does not flap around a threshold. Host load stays a separate dimension from the governor's pressure level. Folding it into the level would reuse the Warning response, which blocks model loads and trims foreground concurrency; a busy build on the same machine should not stop a search from loading its model. Background scheduling consumes the flag in a follow-up change. Thresholds are typed config under [tuning.resource]: host_pressure, host_cpu_pressure_pct (40), host_io_pressure_pct (30), host_memory_pressure_pct (10) and host_load_per_cpu (1.5). They show in the runtime tuning status with their provenance. Signed-off-by: trvon <me@trevon.dev> |