1,853 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Trevon
|
b6e6cd38f8 |
fix(topology): make louvain moves independent of hash order (#313)
makeAdjacency filled each neighbor list in PairWeightMap (unordered_map) order, and the local-move loop took the first best community from an unordered_map. Both orders differ between libc++ and libstdc++, so the same corpus produced different communities on macOS and Linux, and topology_representative_rule failed on macOS. Neighbor lists are now sorted by index and equal-gain moves go to the smaller community id. The louvain characterization is regenerated; macOS and Linux now give the same result. Signed-off-by: trvon <me@trevon.dev> |
||
0xkite
|
05bdb14cf4 |
test(policy): allowlist the per-process XDG_STATE_HOME setenv in catch2_main (#312)
#296 sets XDG_STATE_HOME before Catch2 starts so each test process gets a private state directory. The support library links no yams config, so it calls setenv/_putenv_s directly, and raw_environment_policy has failed the Tests job on every platform since. Record both sites as reviewed. Signed-off-by: trvon <me@trevon.dev> Co-authored-by: trvon <me@trevon.dev> |
||
|
Trevon
|
beef87e66c |
bench(retrieval): seed one-way kNN semantic edges on request (#308)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
Benchmark seeding wrote only mutual neighbor pairs. The connected topology engine keeps every reciprocal edge at or above min_edge_score inside its component, so with mutual-only seeding no raw neighbor crosses a cluster boundary and the engine builds no roads. The Eval 2 smoke showed roads_absent on every connected-engine query. The daemon stores one-way kNN edges (GraphComponent treats them as ordinary non-mutual neighbors). YAMS_BENCH_SEED_SEMANTIC_DIRECTED=1 seeds them too, as single directed edges. Default off keeps existing plans unchanged. The pair loop moves into buildSemanticNeighborSeedEdges in the bench support library, sorted by (source, target), with a unit test for both modes. A topology test pins the mutual-only case: no roads on the connected engine until a one-way edge exists. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
81a051fc2e |
feat(search): walk along roads (#307)
Let RouteOnly and Augment expand every routed hit along its cluster's walk order. The snapshot cache loader resolves each cluster's `orderedMembers` to member hashes (`buildTopologyOrderedMemberHashes`, `TopologyRoutingSnapshot::orderedMemberHashes`); a cluster with `orderKind` None or a permutation that fails `validateMemberPermutation` resolves to nothing. The store's `orderUnavailableReason` is carried on the session result and emitted as `topology_order_unavailable_reason` only when set. When `roadWalkWindow` > 0 the session builds a `TopologyRoadWalkIndex` after the road beam: the selected routes first, then the road-expanded clusters, keeping only clusters with an order and counting the rest in `roadWalkClustersUnordered`. The index borrows its member views from the snapshot it keeps alive; the certificate is never touched. The vector stage walks after the certified and road groups. Every routed hit so far (certified route hits and road-beam hits) anchors +-window positions along each order it belongs to, bounded to that order (`walkTopologyRoadOrders`). Anchoring on route hits too means the walk rescues the ordinal neighbours of the semantically closest members even with `road_hops` = 0; the plan's "around road hits" reads as every hit a road-era routed query produced. The walked set, deduplicated against the hits, is one extra routed query whose quota is its own size (ranking ordinal neighbours by similarity and truncating would reintroduce the filter the walk bypasses); its rows and distance evaluations land in the routed diagnostics like road candidates. Shadow and Narrow never walk. Config: `search.topology.road_walk_window` (default 0, off) on `SearchEngineConfig::topologyRoadWalkWindow` (copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy::roadWalkWindow` and applied by SearchEngineManager. `makeTopologyRoutingOptions` requests the window only for Augment and a gate-open RouteOnly, like the beam. The window is excluded from `topologyRoutingPolicyFingerprint` (comment and test). Defaults leave every stage byte-identical: the PR 2 goldens (692), the RouteOnly cells (417) and the road beam cells (904) are unchanged. Debug keys appear only when the window > 0: `topology_road_walk_window_applied` (0 when nothing was walkable), `topology_road_walk_anchors`, `topology_road_walk_candidates_added`, `topology_road_walk_clusters_unordered`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_WALK_WINDOW writes `search.topology.road_walk_window`; YAMS_BENCH_TOPOLOGY_ROAD_ORDER and YAMS_BENCH_TOPOLOGY_ROAD_MAX_PER_CLUSTER write `topology.road_order` and `topology.road_max_per_cluster` (neither had a bench mapping, so Eval 3 can set the order kind per arm and build roads at all). Tests: tests/unit/search/search_vector_stage_road_walk_catch2_test.cpp (new catch2_search_vector_stage_road_walk): two reciprocal chains A (six members) and B (four) with a road, built with `TopologyBuildConfig::roadOrder` Time from eventTime so the walk order differs from the similarity ranking; loader resolution including the invalid-permutation drop and the cache carry; fingerprint equality for any window and the option gating per policy; session-level certificate, beam and work invariance with the index listing routes then road clusters, walk sets for windows 0/1/2/100 bounded to the cluster, an unordered cluster counted and skipped, no index without any order, the order diagnostic through recordTopologyRoutingDebug; RouteOnly window 0 equal to the road beam cell with no walk key, window 1 adding the immediate order neighbours of all four hits, window 2 the whole clusters, rows visited growing while router work is unchanged, certified keys unchanged, unordered B contributing nothing, no order anywhere equal to the beam cell plus counters, hops 0 walking the selected route alone; Augment adding the walked documents as rescues; Shadow and Narrow emitting nothing. ConfigResolver covers the TOML key. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads, the bench reads its keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
cfb70f7bb9 |
feat(topology): ordered members (#306)
Give every cluster with at least two members a walk order: `ClusterArtifact::orderedMembers`, a uint32 permutation over the hash-sorted `memberDocumentHashes`, plus `orderKind`. Three kinds, each deterministic with ties resolved to the smaller member hash: - Time: ascending `TopologyDocumentInput::eventTime`; members without a time last, by hash. - Session: grouped by `sessionId`; groups ordered by earliest event time (groups with no time after those with one), then session id; inside a group by time then hash; unsessioned members last, by time then hash. - SimilarityChain: greedy nearest-neighbor tour by cosine over the members' embeddings, starting from the cluster representative (`medoid`, else the smallest-hash usable member); members without a usable embedding last, by hash. The knob is `TopologyBuildConfig::roadOrder`, resolved from TOML `topology.road_order` = none|time|session|similarity_chain through the topology.* resolver block and `TopologyManager::setRoadOrder`, threaded into the build config like `roadMaxPerCluster`. Orders are computed after boundary spill so they cover replicas, at full build in every engine and for the rebuilt clusters of a connected-engine incremental update; untouched clusters keep theirs (a kind change reaches them on the next full rebuild, like the representative rule) and `none` clears every order. Orders are independent of `roadMaxPerCluster`. Orders persist in the road sidecar, now version 3: after the roads, u32 count, then per ordered cluster u32 cluster-id table index, u32 kind, u32 length and the permutation as u32 indices. A sidecar without orders is still written as v2, so roads-only sidecars are byte-identical to the previous layout; v1 and v2 decode with no orders, and the envelope gains `member_order_count`. The sidecar is written when the batch has roads or at least one ordered cluster. On load each order is validated on its own (`attachTopologyMemberOrders`: cluster exists, kind not none, permutation covers the members exactly once); an invalid entry drops only that cluster's order and appends `<token>:<clusterId>` to the new load-time diagnostic `TopologyArtifactBatch::orderUnavailableReason`. Roads never lose anything to an order error and vice versa. The main snapshot codec, the routing construction, representation and road fingerprints never see orders, so the default `none` leaves the snapshot payload, the sidecar and every fingerprint byte-identical. No search consumer yet; the road walk lands separately. Tests: tests/unit/topology/topology_ordered_members_catch2_test.cpp (permutation validator tokens, time order with missing times, session grouping and group tie-break, hand-computed greedy chain from the medoid on a five-point fixture with fallback start, unusable members and exact ties, every engine stores valid permutations of every kind including over spill replicas, default none byte-identical payload/sidecar/fingerprints, sidecar v3 round trip plus hand-built v2 bytes and corrupt-section rejection, per-cluster attach validation with reasons, store round trip with orders-only sidecars, invalid order dropped with reason while roads and other orders stay, incremental update keeps untouched orders and recomputes rebuilt ones), plus resolver and manager setter coverage. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
111988c6e4 |
feat(topology): typed event time and session for topology inputs (#305)
* feat(topology): typed event time and session for topology inputs Add `eventTime` (epoch seconds, UTC) and `sessionId` to `TopologyDocumentInput`. The extractor fills them from two documented metadata keys: `event_time` (integer epoch seconds, or ISO-8601 `YYYY-MM-DD` / `YYYY-MM-DD[T ]HH:MM[:SS[.frac]][Z|±HH:MM]`, naive datetimes are UTC) takes precedence over the document createdTime; an unparsable value is skipped with a debug log, never an error, and a zero createdTime leaves the field unset. `session_id` is copied verbatim and empty when absent. The keys are only consulted when the extraction loads metadata; otherwise createdTime is the only source. `parseTopologyEventTime` is public for tests and future callers; the calendar math is Hinnant's days_from_civil with range-checked fields. Extraction stats count the event time source and invalid values. Nothing downstream reads the fields yet. Artifacts, codecs and both routing fingerprints are unchanged, which the new test pins. Tests (tests/unit/topology/topology_input_event_time_catch2_test.cpp): parser accept/reject tables, metadata precedence over createdTime, invalid ignored, session_id, includeMetadata=false fallback, and byte-identical snapshot payload plus equal construction and road fingerprints with and without the typed fields. Signed-off-by: trvon <me@trevon.dev> * bench(longmemeval): event_time and session_id metadata, round docs, haystack sidecar Every exported corpus document now carries `metadata.event_time` (ISO-8601, normalized from the LongMemEval `YYYY/MM/DD (Day) HH:MM` haystack date, first-seen date per session) and `metadata.session_id`. Document ids, titles, texts, queries and qrels are unchanged in the default mode. `--round-docs` adds one document per session round (`<sid>__r<n>`): a user turn opens a round and the replies that follow join it; a leading assistant turn is its own round. Round docs share the session's `event_time`/`session_id` and add `round_index`; qrels stay at the session level. `--full-haystack` also writes `haystacks.jsonl`, one line per question with `haystack_session_ids` and the normalized `haystack_dates`, so per-question membership survives the global corpus deduplication (the 2026-10-10 preflight lost it). The converter is split into `convert` / `write_outputs`; `--self-test` runs both on a three-session synthetic fixture without a download and is the regression check for the default output. Signed-off-by: trvon <me@trevon.dev> * bench(retrieval): carry BEIR corpus metadata onto ingested documents `retrieval_quality_bench` ingests the BEIR corpus as a directory, which cannot carry per-file metadata, so `event_time`, `session_id` and `round_index` from corpus.jsonl never reached the topology extractor. `BEIRDocument::metadata` now holds the corpus line's `metadata` object (strings as is, numbers and booleans stringified; both the generic and the SciFact loader read it), and `applyBenchmarkDocumentMetadata` copies it onto the stored documents matched by exact file path through the metadata repository (`ON CONFLICT DO UPDATE`, so warm caches can repeat it). The bench runs it after the ingest wait, for cold and reused corpora alike, and logs how many documents were updated or not found. Tests: loader parses and coerces metadata; apply helper writes the keys, skips documents without metadata, counts missing ones and is idempotent against a real MetadataRepository. The portability allowlist entries for the two touched bench files are re-pointed at their shifted lines; no raw environment read was added. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
f91cd85e13 |
feat(search): road beam in RouteOnly/Augment (#304)
Let the vector stage follow cluster roads from the selected route clusters. `TopologyRoutingSnapshot::roadAdjacency` is built by the snapshot cache loader (`buildTopologyRoadAdjacency`) as the union of both stored orientations of every road, per source sorted by weight desc then target cluster id asc; it is empty when the batch carries no roads, whether never built or dropped with `roadUnavailableReason`. The session runs a beam after the certificate is materialized: for `roadHops` hops it scores every unvisited road neighbour of the frontier by road weight times the router's query-centroid similarity ((cosine + 1) / 2), keeps `roadBeamWidth` per hop (score desc, cluster id asc) and dedupes across hops. Each centroid evaluation is cached per cluster and counted once in `routeRepresentativeDistanceEvaluations` and `routeExactRepresentativeDistanceEvaluations`, so routed work stays comparable with the no-road arm. The result lands only in `TopologyRoutingSessionResult::roadExpandedClusterIds` and `roadExpandedDocumentHashGroups` (one member group per cluster, excluding certificate members). `certificate.allowedDocumentHashes`, admission and `topologyRoutingPolicyFingerprint` are untouched; the fingerprint comment and a test pin that the road knobs stay out of it. `VectorStageRequest` carries the road groups with their own quotas (`allocateBalancedRouteQuotas` over the road groups) and cap. RouteOnly and Augment query them after the certified groups; the certified part of the routed result is identical with roads on or off. Shadow and Narrow never see road groups, and `makeTopologyRoutingOptions` requests hops only for Augment and a gate-open RouteOnly, so a gated-off RouteOnly running as Shadow takes no roads with it. Config: `search.topology.road_hops` (default 0, off) and `search.topology.road_beam_width` (default 4, read when hops > 0) on `SearchEngineConfig` (`topologyRoadHops`, `topologyRoadBeamWidth`, copied by `applyTopologyPolicyFrom`), resolved through `ConfigResolver::TopologyRoutingPolicy` and applied by SearchEngineManager. Defaults leave every stage byte-identical: the PR 2 goldens (692) and the RouteOnly cells (417) are unchanged. Debug keys appear only when hops > 0: `topology_road_hops_applied`, `topology_road_clusters_added`, `topology_road_candidates_added` (road documents the stage appended), `topology_road_work_centroid_evaluations`, `topology_road_skip_reason` (roads_absent, no_selected_routes, route_low_confidence) and, under stage trace, `topology_road_expanded_cluster_ids`. Bench: YAMS_BENCH_TOPOLOGY_ROAD_HOPS / YAMS_BENCH_TOPOLOGY_ROAD_BEAM_WIDTH write the two TOML keys. Tests: tests/unit/search/search_vector_stage_road_beam_catch2_test.cpp (new catch2_search_vector_stage_road_beam): a four-cluster fixture with hand-built roads A-B 2.0, A-D 1.0, B-C 1.5 where the query routes to A; adjacency construction and cache carry; fingerprint equality for hops 0 vs 2 and any beam width; session-level certificate invariance with hops 0/1/2, beam width 2, early stop and roads absent; RouteOnly hops 0 equals the route-only cell with no road keys, hops 1 gives A u B, hops 2 with beam width 1 gives A u B u C via the heavier road, work counters grow with hops; Augment hops 1 adds B's documents as rescues; Shadow and Narrow take no road groups. ConfigResolver covers both TOML keys. The line-keyed portability allowlist is re-keyed for the two shifted files (search_engine.cpp, retrieval_quality_bench.cpp); no new raw environment reads are added, the bench reads its two keys through yams::config::getenv_optional. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
f7f1a09865 |
feat(topology): incremental roads (#303)
Maintain cluster roads across incremental rebuilds instead of clearing them. `ConnectedComponentTopologyEngine::updateArtifacts` now keeps the roads between clusters the update did not touch, drops every road with an endpoint in a removed or rebuilt cluster (reused ids included), and recomputes the rebuilt clusters' roads from the dirty region's raw neighbor lists against the merged memberships, so a region document's neighbor in an untouched cluster yields that road in both orientations. Each source then re-ranks its top-K over the union with PR 6's order and tie-break (weight desc, target id asc). Only the region's neighbor lists are available during an update, so a cross-boundary edge is observed from one side; the out-of-region target is credited with the mirror observation its own list would have added. That is exact on a symmetric neighbor graph and the stated approximation otherwise. An untouched source can only re-rank the roads its last full build kept, so the result is derived, not full. Provenance on the batch: `roadsIncrementalPasses` counts passes since the last full accumulation (0 = full roads) and `roadsFullBuildEpoch` is the `topologyEpoch` of the snapshot that accumulation produced. A full build leaves both at 0; an update sets the epoch from `existing` when its roads were full and carries it forward otherwise. The fields ride in the road sidecar header, now version 2 (u64 epoch, u32 passes after the snapshot id); version 1 sidecars decode with both at 0, the envelope format string is unchanged, and the main v1 payload and the routing construction fingerprint do not see them. The store copies them both ways. Roads off (`road_max_per_cluster = 0`, the default) still yields no roads and zero provenance, so default behavior is unchanged. `topology_build_utils.h` splits `accumulateClusterRoads` into pair-weight accumulation (with the optional observed-document set that drives mirroring), orientation and ranking, and adds `deriveIncrementalRoads` over those pieces; the full-build path passes no observed set and is unchanged. Louvain and k-means `updateArtifacts` rebuild the supplied input from scratch, so their roads are full roads for that batch with zero passes; the test states and checks that property for them. Tests: tests/unit/topology/topology_roads_incremental_catch2_test.cpp (roads off through an update, full-build provenance, kept/retired/ recomputed roads with mirrored weights on a four-cluster graph, top-K re-ranking over kept and recomputed roads, derivation without prior roads, a property test over seeded planted-cluster symmetric graphs with cross-edge rewiring and member moves comparing incremental roads to a from-scratch build after each of several dirty regions, a tight-top-K property for rebuilt sources and derived weights, and the alternate engines), plus sidecar v2 round trip, hand-built v1 decode, future version rejection and store round trip of the provenance in topology_road_sidecar_catch2_test.cpp. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
e84ced261b |
feat(topology): road sidecar persistence (#302)
Persist `TopologyArtifactBatch::roads` in a sidecar knowledge-graph node,
`topology:roads:<snapshotId>` of type `topology_road_snapshot`, instead
of growing the main snapshot codec. The sidecar has its own binary magic
('Y','A','M','R'), version 1, and a Zstd JSON envelope with
`format = "topology_roads_zstd_v1"`. The main snapshot stays a plain v1
payload, byte-identical with roads on or off, so memory-sync replicas on
older peers keep decoding it, and an unknown node type is simply never
read by an old store. `deserializeTopologyBatchCompressed` now rejects an
envelope whose `format` it does not know instead of treating it as legacy
raw JSON.
Header layout (little-endian): u32 magic, u32 version, u32 maxPerCluster,
u32 reserved, string snapshotId, u32 cluster-id table count + strings,
u32 road count, then per road u32 from index, u32 to index, f64 weight.
`TopologyArtifactBatch::roadMaxPerCluster` records the bound the builder
applied so the header can carry it; both engines set it.
The store writes the sidecar right after the snapshot node and before
the `topology:snapshot:latest` pointer, so the pointer remains the commit
point: an injected sidecar failure leaves the previous snapshot current.
Retention deletes each pruned snapshot's sidecar with it, and re-storing
a snapshot id without roads removes its stale sidecar. Loading is
optional and soft: a missing sidecar yields empty roads with no error; an
undecodable or invalid one (snapshot mismatch, endpoint not a cluster,
self loop, negative or non-finite weight, more roads per source than the
declared top-K) drops the roads and records
`TopologyArtifactBatch::roadUnavailableReason`, surfaced on the routing
session and as the `topology_road_unavailable_reason` debug key, while
the snapshot still loads and routes.
`topologyRoadFingerprint(batch)` gives roads their own order-independent
identity for cache provenance; the routing construction fingerprint is
unchanged and stays road-free through the persisted path.
Lean (formal/topology/Yams/Topology/Artifacts.lean): `ClusterRoad`,
`WellFormedRoads` (endpoints in clusters, no self loop, non-negative
weight, bounded per source), `constructionIdentity`, and lemmas that the
construction identity and `WellFormedBatch` are independent of roads and
that an empty road layer is well formed. Checked with lake (lean 4.31.0).
Tests: tests/unit/topology/topology_road_sidecar_catch2_test.cpp (codec
round trip, foreign/truncated payload rejection, main payload identical
on/off, validation reasons, store round trip, missing sidecar, every
corrupt-sidecar case with routing still admitted, write-before-pointer,
retention, fingerprints through the persisted path) and the extended
fingerprint invariance test in topology_roads_catch2_test.cpp.
`road_to` knowledge-graph edges are deferred until a consumer needs them.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
dcdeaa81d0 |
feat(topology): accumulate roads at build (in memory) (#301)
Add cluster roads: undirected cluster-to-cluster links whose weight is
the sum of every raw neighbor observation between members of two
clusters. The shared helper `accumulateClusterRoads` in
topology_build_utils.h reads `TopologyDocumentInput::neighbors` before
the reciprocal/minEdgeScore filter, because the connected engine's
filtered adjacency only carries cross-cluster edges where an oversized
component was split, and k-means never consults the graph.
Each cluster keeps its top-K roads (weight desc, then target cluster id
asc) as `ClusterRoad {fromClusterId, toClusterId, weight}` on
`TopologyArtifactBatch::roads`, sorted by source cluster. Both the
connected engine and buildBatchFromAssignment (kmeans/louvain) call it;
an incremental update clears roads until the next full rebuild.
The knob is `TopologyBuildConfig::roadMaxPerCluster`, resolved from
TOML `topology.road_max_per_cluster` through the topology.* resolver
block and `TopologyManager::setRoadMaxPerCluster`. The default 0
computes no roads; the binary codec and the routing construction
fingerprint ignore the field, so existing snapshots and fingerprints
are unchanged with roads on or off.
Tests: tests/unit/topology/topology_roads_catch2_test.cpp (raw vs
filtered neighbors, top-K and tie-break, order independence, both
orientations, out-of-batch and non-finite scores, kmeans and louvain,
codec bytes and fingerprint unchanged), plus resolver and manager
setter coverage.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
db53357462 |
feat(search): RouteOnly policy behind an evaluation gate (#300)
Add TopologyVectorPolicy::RouteOnly: the vector stage runs only the routed
candidate queries (DocumentTopK on Simeon PQ, ExactDocumentComplete on
ExactScan) with no global fill and no admission gate, reports the routed
diagnostics as vector search cost and applies the routed work to the
certificate. Results come solely from the routed set.
RouteOnly serves uncertified routes, so it runs only behind
search.evaluation.allow_uncertified_routes (SearchEngineConfig::
allowUncertifiedRouteOnly). Without the gate the policy downgrades to
Shadow and records topology_vector_policy_downgraded=uncertified.
Vec0 has no routed candidates: the global leg is returned unchanged and
the source reports unavailable.
Every search now records topology_route_certified (the certificate admits
a hard route after the observed work is applied) and
vector_candidate_source_effective (ann | route_only | augment | narrow |
unavailable | none); the sequential path reports ann.
Wiring: search.topology.vector_policy accepts route_only;
ConfigResolver::TopologyRoutingPolicy carries the gate;
SearchEngineManager applies both and warns when route_only is configured
without the gate; applyTopologyPolicyFrom copies the gate so corpus
tuning can never select RouteOnly or open it. Defaults are unchanged.
Bench: YAMS_BENCH_TOPOLOGY_VECTOR_POLICY=route_only writes both TOML keys.
Also fix the stale Narrow/Vec0 doc comment on the policy enum.
Tests: new catch2_search_vector_stage_route_only pins RouteOnly x
{SimeonPqAdc, ExactScan, Vec0} x gate on/off, a certified route and an
executor-backed run; the gate-off cells are compared key for key against
Shadow. ConfigResolver, SearchEngineConfig and SearchTuner tests cover
the two TOML keys and the tuner invariant. The PR 2 goldens are unchanged
(692 assertions).
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
2ba8cb4a86 |
fix(search): vector-stage lifetime on component timeout (#299)
* test(search): vector stage completes into its own state on component timeout
Executor-backed SearchEngine with a vector hydration step that blocks until the
test releases it. With a small componentTimeout the search returns without the
vector component; the test then releases the worker and runs a second search
from the same call site, so the second search's locals reuse the first frame's
stack region. A stage that still writes into the first frame races with the
second search (TSan data race) until the stage state is heap-owned by the
posted work. A second case pins that a component meeting its deadline is still
collected with its diagnostics.
Signed-off-by: trvon <me@trevon.dev>
* fix(search): vector-stage lifetime on component timeout
The posted vector work still reached into the search frame after a
componentTimeout: its injected query wrapper captured `this`, workingConfig
and four frame-local relaxed-retry atomics, and the frame adopted the stage
outcome unconditionally after collectIf, even when the component had timed out
and the worker was still running (a data race on the outcome, and a stale write
into whatever next occupies that stack region once the frame returns).
Give the wrapper a heap home: VectorStageState {SearchEngineConfig config;
relaxedRetryEnabled; RelaxedVectorRetryStats relaxedRetry; VectorStageOutcome
outcome} is captured by shared_ptr in the posted lambda and the query wrapper is
built from it by makeQueryVectorWithRelaxedRetry, which holds the repository
and vector database by value, so the work depends on neither the frame nor the
engine. The sequential path uses the same factory bound to frame locals; the
relaxed-retry atomics become a plain struct only the frame thread touches.
collectAndRecord now returns the ComponentStatus. The frame adopts the stage
state only when the component finished (Success or Failed); after TimedOut or
Canceled the late completion lands in state only the worker references and is
dropped with it, so the diagnostics merge and route-work application are
no-ops. Behavior with no timeout is unchanged.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
78b5440186 |
refactor(search): extract runVectorCandidateStage (#298)
* refactor(search): move route work helpers to the vector pipeline
Pure move of evaluateTopologyRouteWork, observeTopologyRouteWork and
allocateBalancedRouteQuotas from the search_engine.cpp anonymous
namespace into search_vector_pipeline_internal.h so the vector candidate
stage can be extracted into its own translation unit. The portability
allowlist line numbers follow the move.
Signed-off-by: trvon <me@trevon.dev>
* refactor(search): extract runVectorCandidateStage
Move the fanout.vector lambda body out of SearchEngine::searchWithResponse
into src/search/search_vector_stage.{h,cpp}. VectorStageRequest carries
the policy flags, allowed sets and groups (shared_ptr<const>, aliasing a
certificate snapshot), quotas, limits, backend capabilities and the
injected relaxed-retry query function. VectorStageOutcome carries the
results, every counter the lambda used to write through captured
references, the observed route work and separate globalDiag / routedDiag.
The posted work owns its outcome through a shared_ptr captured by value
and the frame adopts it after the component is collected; the relaxed
retry closure is still the frame's. SearchEngine merges globalDiag into
the reported vector diagnostics, merges routedDiag only when the policy
is not Shadow (Shadow's routed query is counterfactual and feeds route
work only), and applies the observed work to the session certificate
after adoption. Query order inside the stage is unchanged, so the
last-writer relaxed_vector_primary_hit_count stays identical.
No behaviour change: the vector stage characterization goldens are the
acceptance test.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
3606914222 |
test(search): characterize the vector stage (#297)
Pin the current behaviour of the vector candidate stage in
search_engine.cpp before it is extracted into search_vector_stage.{h,cpp}.
A new catch2_search_vector_stage binary runs the matrix policy
{Shadow, Augment, Narrow commit, Narrow fallback} x backend
{SimeonPqAdc, ExactScan, Vec0L2} with stage trace off and on, plus one
executor-backed fanout case, and compares each cell against explicit
goldens: fused hash/rank/score and vector contribution, raw vector
ComponentResults (trace_component_hits_json), the stage's debug keys
without timings, certificate.work / admission.work via the
topology_route_work_* and admission keys, and the merged vector_search_*
diagnostics. Quirks are pinned as-is: relaxed_vector_primary_hit_count is
last-writer, Vec0 keeps global results unchanged under Shadow/Augment,
Narrow commits only on Simeon PQ because ExactScan and Vec0 never observe
an ANN candidate budget (work_observation_unavailable), and Vec0 leaks a
partial ann_candidate_budget_actual into the unobserved work record. A
hidden [.dump] case regenerates the goldens.
Move the shared topology fixtures (TopologySearchFixture,
buildTwoClusterTopologyBatch, topologyRoutingTestConfig,
runTopologySearch, configureCertifiedTopologyRoute, TopologyVectorEnvPin)
into tests/unit/search/topology_search_test_support.h so both tests use
the same corpus. runTopologySearch takes an optional executor and
configureCertifiedTopologyRoute an options struct; existing call sites are
unchanged. The portability allowlist entry for the raw YAMS_TEST_TMPDIR
read moves with the helper. No product behaviour changes.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
f7ca95427d |
fix(daemon): run memory_sync catch-up sweeps only when a store changed (#295)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* test(daemon): memory_sync catch-up sweeps must go idle on unchanged stores The vector and topology catch-up sweeps restart a full pass as soon as one ends, so an idle node reads and re-checks every node, edge, vector, and committed topology record forever. On a 3-node mesh limited to 2 CPUs each this held every node at ~24% of a core with nothing changing. New cases drive the production backfill through ServiceManager: - after a completed pass with no writes, further cycles serve no sweep item and take no graph-store connection; - a node, edge, and vector written after a pass are published by the next pass while unchanged records keep their committed operation; - a node deleted after the sweeps went idle is retracted with its edge. Store seams: KnowledgeGraphStore::changeGeneration() and VectorDatabase::changeGeneration() move on writes and batch commits, not on reads. Red: catch2_memory_sync_apply "[idle]" 3 failed (items 52 == 32, acquisitions 157 == 97); kg_store and vector_smoke do not compile. Signed-off-by: trvon <me@trevon.dev> * fix(daemon): run memory_sync catch-up sweeps only when a store changed The vector and topology catch-up sweeps in publishMemorySyncBackfill restarted a full pass as soon as one ended. An idle node therefore read and re-checked every node, edge, vector, and committed topology record forever, and every published record cost each peer a delta exchange and an apply. On a 3-node mesh limited to 2 CPUs each this held every node at ~24% of a core with nothing changing; on a real corpus the first pass never ends. Each store now exposes a change generation: KnowledgeGraphStore counts write transactions (single writes through the write pool and write batch commits), VectorDatabase counts inserts, updates, and deletes. A sweep records the generation a pass began at and, once the pass completes, does not start another until the generation differs. Writes behind a running pass's cursor move the generation past the pass's own reading, so the following pass still covers them; deletes move it too, so the retraction phase still runs after a rebuild drops records. With nothing changed, a cycle costs each sweep one atomic read. No config knob: the gate is a property of the stores, not a tunable. Tests: catch2_memory_sync_apply (14 cases, incl. 3 new [idle] cases), catch2_kg_store, catch2_vector_submodule. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
837bc641b3 |
test(infra): give every Catch2 process a private XDG_STATE_HOME (#296)
SessionService reads the current session from XDG_STATE_HOME. On a build
host whose user had an auto-created session ("auto: /data") with watch
enabled, every test that booted a ServiceManager inherited it, the session
watcher walked /data during init, and six daemon tests timed out. With the
state dir isolated the same six pass.
The shared test main now points XDG_STATE_HOME at a per-process temp dir
when the caller did not set one, and removes it at exit. A small test checks
the process sees no host session.
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
777e35adae |
feat(profiling): Tracy for every library, zones on p2p and memory-sync, per-test CPU/RSS (#294)
* feat(profiling): Tracy zones on p2p and memory-sync background paths Zone the memory-sync cycle (publish, sync, reconcile, apply, backfill, delete-outbox drain), the p2p reconnect cycle, accept loop, per-session worker and frame read, and the delta send/apply/bootstrap stages. Plot the retained session count and pending erase intents. No behavior change; the macros are no-ops without Tracy. Signed-off-by: trvon <me@trevon.dev> * test(infra): per-test-case CPU and peak RSS accounting for Catch2 binaries A Catch2 listener in the shared test main records getrusage (GetProcessTimes on Windows) CPU per case and prints the five heaviest cases plus the run's peak RSS to stderr at run end. YAMS_TEST_RUSAGE=1 adds one line per case. The measurement helpers are header-only and unit tested. Signed-off-by: trvon <me@trevon.dev> * fix(vector): make vector_database.cpp compile with Tracy enabled The Profiling lane did not build: eight YAMS_PLOT sites in vector_database.cpp used -> on a Result<T>, and two passed an int that is ambiguous between Tracy's PlotData overloads. Use .value() and int64_t casts. No behavior change outside TRACY_ENABLE builds. Signed-off-by: trvon <me@trevon.dev> * build: enable Tracy for every yams library in the Profiling lane Define yams_tracy_args/yams_tracy_deps once in the root meson.build (Tracy found, optional rocprofiler hooks, link check) and apply them to every library and executable under src/, replacing the per-library copies in search, topology, vector, daemon and tests/benchmarks. Empty when enable-profiling is off. Make YAMS_PLOT accept any numeric type and fix a Result deref in search_service so the newly instrumented TUs compile with Tracy. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
d11da7e29e |
test(p2p_mesh): per-node CPU/memory limits and an idle CPU sampler (#292)
The mesh harness ran nodes with the build host's resources, so a daemon that pins a small machine's CPU while idle never showed up. `up` and `run` take --cpus and --memory (docker cpus/mem_limit per node). The new `idle` command samples docker stats with no client traffic, writes idle.csv and idle.json, optionally captures top -H of the hottest node, and exits 5 when a node's mean CPU exceeds --max-cpu. Local/manual tooling; nothing runs in CI. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
6039a3695b |
fix(storage): cache reference counter stats per SQLite generation (#291)
getStats ran the aggregate query on every call. The result is now cached with the connection's total_changes() and data_version and reused while both are unchanged; any write, restore, or copy invalidates it. A failed generation probe disables reuse rather than the aggregate fallback. Tests: reference_counter (generation reuse, invalidation, restore and copy paths). Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
de84edde1a |
fix(daemon): wake paused post-ingest pollers on resume and capability changes (#290)
A paused or admission-blocked poller slept for the full max idle delay instead of waiting on its registered timer, so resumeStage, resumeAll, enabling the knowledge graph, and adding entity or title providers took effect only after the delay elapsed. The poller now waits through awaitPressureLimitedPollerWake and the setters signal the stage timer. Tests: pressure_limited_poller and post_ingest_queue_unit. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
2d162793cb |
fix(daemon): count active requests, not connections, for idle detection (#289)
TuningManager treated any open non-health connection as activity, so a client that kept its socket open held the daemon in busy mode and kept background scaling up. RequestHandler now counts executing non-health requests with an RAII guard, SocketServer wires the counter into DaemonStats, and the idle/busy decisions read it. Tests: request_handler_status_unary, socket_server_teardown, and two slow real-service tests (tuning_registry_activity, socket_proxy_request_activity). Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
b67c6374af |
docs(agents): require ResourceGovernor admission for daemon background work (#288)
Codifies the rule after the 2026-10 incident where p2p and topology work pinned the CPU of a memory-constrained host. Lists the components that do not consult the governor yet. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
01fe25f1ab |
fix(ipc): preserve search stats in streamed responses (#287) | ||
|
Trevon
|
52aa700144 |
fix(graph): --scope-cwd clusters, --name lookup, --limit (#280 follow-ups) (#284)
* test(graph): cover --topology-clusters --scope-cwd outside src/include Follow-up to #280. --topology-clusters --scope-cwd still built its scope from <cwd>/src and <cwd>/include, so a corpus laid out any other way had no scoped members: - The scope path set must hold every stored path under the cwd, also when the cwd is reached through a symlink, and not a sibling that shares the prefix. - GraphTopologySupport must build the set from its own metadata repo. The existing cluster-scope case built its path set by hand from the old src/include contract. It now builds the set from /repo, so /repo/tests/ is in scope and a document outside /repo is not. Signed-off-by: trvon <me@trevon.dev> * fix(graph): scope --topology-clusters --scope-cwd to the whole cwd Follow-up to #280. Topology views built their --scope-cwd path set from <cwd>/src and <cwd>/include only, a leftover from the code-symbol graph removed in v0.20. Corpora laid out any other way reported no scoped members. Build the set from the same roots as the --list-type ranges: the cwd, lexical and symlink-resolved. Keep only paths really under a root, since the prefix query matches with LIKE. The topology view no longer prunes build/, packages and IDE paths, so it matches --list-type. GraphTopologySupport now builds the set from its own metadata repo instead of the CLI app context. Remove buildGraphCodeScopePathSet and buildGraphCurrentScopePathSet, which have no other callers, and use one "Scoped to paths under the current directory" note for both views. Signed-off-by: trvon <me@trevon.dev> * test(cli): cover graph --name on a document with a path:file: node Follow-up to #280. yams graph --name <path> tried path:file:<path> node keys before the document lookup. v0.20 ingestion writes path:logical: and path:<snapshot>: keys, so the probe misses on a fresh corpus; after `yams doctor repair --graph` (or NL entity extraction) the key exists, and --name printed a raw traversal of the file node instead of the document's related documents. Smoke cases through the in-process CLI: - --name on a file with a path:file: node returns its hash and its semantic neighbours (red). - A relative --name resolves under the cwd even though a newer copy with the same relative path exists in another checkout. - A bare file name still resolves by name. The portability allowlist entry for this file moves with its getenv line. Signed-off-by: trvon <me@trevon.dev> * fix(cli): resolve graph --name through the document lookup only Follow-up to #280. yams graph --name first traversed path:file:<path> node keys. v0.20 ingestion writes path:logical:<path> and path:<snapshot>:<path>, which only chain to blob versions; semantic neighbours hang off doc:<hash>. On a fresh corpus the probe always missed. After `yams doctor repair --graph` or NL entity extraction it hit and printed a raw file-node traversal instead of the related documents. Drop the probe and go straight to the document lookup, which starts at doc:<hash> since #280. Resolving to path:logical: instead would show the version chain, not related documents; --node-key still traverses any node. Send --name as the path ingestion stores: relative to the client cwd, lexically normalized, symlinks resolved. The daemon resolved relative names against its own cwd and fell back to a suffix match, which picks the newest copy across checkouts. Fall back to the name as given when it does not resolve, so bare file names and suffixes still work. build_graph_file_node_candidates had no other callers; its tests are replaced by tests for buildGraphDocumentNameCandidates. Signed-off-by: trvon <me@trevon.dev> * test(graph): cover --limit on the document graph lookup Follow-up to #280. yams graph <hash> and --name ignored --limit: the daemon capped the graph query at 100 nodes and the same-directory fallback at 20 documents. Smoke cases through the in-process CLI: - --limit 1 returns one related document by name and by hash, and the first of the uncapped list. - --limit 1 caps the same-directory fallback for a document without graph edges. - Without --limit the related lists are unchanged. Signed-off-by: trvon <me@trevon.dev> * fix(graph): pass --limit through the document graph lookup Follow-up to #280. yams graph <hash> and --name sent --depth but not --limit; DocumentService fixed the graph query at 100 nodes and the same-directory fallback at 20 documents. Add GetRequest.graph_limit (proto field 27) and RetrieveDocumentRequest::graphLimit, and pass them from the CLI through the dispatcher. A limit caps the related list, graph neighbours, path history and the same-directory fallback alike. The graph query never asks for fewer than 100 nodes, since path and blob nodes count toward that limit. The CLI sends a limit only when --limit is given; 0 keeps the old caps, so default output is unchanged. The binary GetRequest layout appends the field and still reads payloads without it. Unit tests cover the proto round trip, the dispatcher mapping and the DocumentService caps. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
44b2f4f046 |
fix(graph): document lookup and --scope-cwd (#280) (#282)
* test(graph): cover document lookup and --scope-cwd listing (#280) Red tests for the two graph CLI bugs in #280: - GetResponse graph fields and file name must survive the proto wire. - DocumentService graph lookup by hash and by name must return the same documents as traversing the doc:<hash> node. - The document graph JSON must name the document when only name is set. - KG path-range scoping must match document nodes by label and snapshot path nodes by $.path. - listByType --scope-cwd must keep document and path nodes under the client cwd (including via a symlinked cwd) outside src/ and include/. The existing code-roots dispatcher case now places its out-of-scope file outside the scope root, since the scope is the whole cwd. Signed-off-by: trvon <me@trevon.dev> * fix(graph): fill document lookup and scope --scope-cwd to the cwd (#280) yams graph <hash> and --name returned graphEnabled=false with no related documents while --node-key doc:<hash> listed neighbours: - The GetResponse proto binding dropped graphEnabled, related and fileName, so socket clients never saw graph data. - DocumentService started traversal at blob:<hash>, which has no outgoing edges, and filtered out semantic_neighbor. Start at the doc:<hash> node and allow semantic_neighbor. - GraphQueryService did not hydrate documentHash for doc: nodes, so reached documents were skipped. - The presenter read only fileName for "document"; fall back to name. --list-type <t> --scope-cwd returned nothing: - The daemon scoped to <cwd>/src and <cwd>/include only. Scope to the whole cwd, lexical and symlink-resolved. - The KG range predicate ignored document labels and $.path on snapshot path nodes. Match both. Signed-off-by: trvon <me@trevon.dev> * docs(skills): correct graph document lookup examples --format dot is only rendered for node traversal, so show it with --node-key doc:<hash>. Describe --scope-cwd as cwd-scoped. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
4e97b31fef |
chore(warnings): clean up compiler warnings on GCC, Clang and Apple Clang (#279)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* chore(warnings): default-initialize aggregate members left out of brace-init
GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.
Signed-off-by: trvon <me@trevon.dev>
* chore(warnings): remove or annotate unused parameters, functions and variables
- init_command, s3_signer test, request_handler_lifetime test: comment out
the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
__APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
work_coordinator test: mark platform-conditional or busy-loop
variables [[maybe_unused]].
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests
- search_admission_controller: count started workers in std::uint32_t so
the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
temporary std::string is created for the call (GCC
-Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
which continues the comment (-Wcomment); show the env export on its own
line.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): drop redundant std::move of a const captured json
The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): mark __int128 use in db_integrity_stamp as an extension
-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.
Signed-off-by: trvon <git@trevon.dev>
* chore(warnings): remove always-true unsigned >= 0 comparisons
GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
|
||
|
Trevon
|
c248863b76 |
fix(core): correct feature-test detection in cpp23_features.hpp (#278)
- Include <version> before testing __cpp_lib_* macros. Without it every library gate read 0 in a file that included this header first (verified with Apple Clang 21 and GCC 16 at -std=c++23). - Detect [[likely]]/[[unlikely]] with __has_cpp_attribute. __cpp_attributes is 200809L on every compiler, so YAMS_HAS_LIKELY_UNLIKELY was always 0. - YAMS_HAS_REFLECTION now requires __cpp_impl_reflection and __cpp_lib_reflection >= 202506L (P2996R13). __cpp_reflection is not a standard macro. - Remove YAMS_HAS_PROFILES and YAMS_PROFILE. __cpp_profiles does not exist and C++26 did not adopt Profiles; nothing outside the test used it. - YAMS_CPP_VERSION reads _MSVC_LANG when defined. - get_feature_summary() reports the language mode instead of tying "C++23" to constexpr vector/string, which are C++20 features. - Correct the per-gate toolchain notes (flat_map: libstdc++ 15 / libc++ 20 / MSVC 19.51; move_only_function: not in libc++; expected: Clang 18 with libstdc++ reports 0) and drop the std::expected migration guidance: Result<T> stays. Tests: core_cpp23_features checks each gate against the compiler's own feature-test macros with the header included first. New core_cpp23_features_fallback forces every gate to 0 so the fallback paths, including the magic_numbers runtime path, compile in every lane. YAMS_CONSTEXPR_IF_SUPPORTED's inline fallback was invalid on a lambda; the tests now use it on a function. Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
c26294a938 |
feat(topology): opt-in medoid representative and BQ rotation (#277)
* test(topology): characterize degree representatives and raw BQ sign bits
Pin the current behaviour before adding opt-in representative and BQ
rotation options:
- connected, louvain and kmeans engines on a skewed cluster (dense
clique plus a weak-edge hub): medoid hash, score, extra routing
representatives and roles;
- equal-degree ties go to the smaller document hash;
- the unrotated BinaryQuantizedIndex matches a raw-coordinate sign-bit
reference, with and without a prefix dimension.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in medoid representative rule
topology.representative_rule = degree (default) | medoid selects how a
cluster's representative document is chosen. Degree keeps the max
within-cluster weighted degree. Medoid picks the member minimising
sum_j sqrt(max(0, 2(1 - cos))) over the other members.
- one shared helper, applyRepresentativeRule(), is called by the
connected, louvain and kmeans engines after their unchanged degree
choice; medoid falls back to it without usable geometry;
- clusters above 256 usable members are searched on a deterministic
hash-ordered stride sample (kMedoidRepresentativeMaxMembers);
- ties within 1e-9 go to the smaller document hash;
- with medoid, farthest-first extra routing representatives are seeded
from the medoid instead of the centroid;
- the rule is stored in the formerly reserved first header byte of the
binary snapshot (0 = degree, so old snapshots keep their meaning) and
extends the construction fingerprint only when it is not degree;
- search debug stats report topology_representative_rule.
Signed-off-by: trvon <me@trevon.dev>
* feat(topology): add an opt-in randomized Hadamard rotation for BQ routing
search.topology.bq_rotation = none (default) | fwht selects the rotation
applied before the centroid BQ shortlist takes sign bits. Raw-coordinate
sign bits make cos(pi * h / d) a poor estimate on anisotropic embeddings;
after x -> (1/sqrt(n)) H D [x; 0] every bit is a random hyperplane.
- RandomizedHadamardRotation in yams/vector: fixed splitmix64 sign
diagonal plus an in-place FWHT, zero-padded to a power of two; Simeon's
FWHT projection works on int32 sketches with row subsampling, so a
small local transform is used instead;
- BinaryQuantizedIndex::build() takes the rotation and seed, rotates
stored vectors before the prefix is taken, and rotates every float
query itself, so centroids and queries always share one seed
(kDefaultBinaryRotationSeed);
- the routing snapshot cache records bqRotation and bqRotationSeed and
rebuilds the route index when either differs;
- the routing policy fingerprint changes only for a non-default rotation;
- search debug stats report topology_route_bq_rotation.
Signed-off-by: trvon <me@trevon.dev>
* test(bench): map representative rule and BQ rotation arm knobs
Harness-only env knobs written into the isolated benchmark config.toml,
so arms go through ConfigResolver like the other topology settings:
- YAMS_BENCH_TOPOLOGY_REPRESENTATIVE_RULE=degree|medoid
-> [topology] representative_rule
- YAMS_BENCH_TOPOLOGY_BQ_ROTATION=none|fwht
-> [search.topology] bq_rotation, next to bq_candidate_limit and
bq_prefix_dim
The debug-file setup entry records the requested values; per-query
search_stats carry the effective topology_representative_rule and
topology_route_bq_rotation. Portability allowlist lines are shifted for
the inserted code.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
1d1e8def4d |
fix(search): count augment route members as added candidates (#276)
* test(search): pin augment routed-candidate accounting Under the augment vector policy a document reachable only through a topology route reaches the fused results through the candidate-rescue vector merge, but the routing session and the topology_weak_query_added_candidates stat report zero additions, because the product stage always collects route membership and the session skips the add loop in that mode. Pin the routed-only fixture (x2 in the route, outside lexical Tier-1 and the global vector top-1) for augment, and characterize shadow (identical to topology-disabled) and uncertified narrow (global, no additions). Signed-off-by: trvon <me@trevon.dev> * fix(search): count augment route members as added candidates #57 unified topology onto one evidence path: the product assist stage always collects route membership, and augment, shadow and narrow all consume the certificate's allowed set (augment vector-scores it through the candidate-rescue merge, shadow measures it, narrow filters to it). The session skips the old add loop in that mode, so under augment addedCandidates, duplicateCandidates and topology_weak_query_added_candidates stayed 0 even when routed documents were rescued into the fused results. Under augment only, count materialized allowed-set members that are new to the existing candidate pool as added candidates and the rest as duplicates. The allowed set, its maxDocs bound, the routing policy fingerprint, applied and skip reasons are unchanged; narrow and shadow report no additions as before. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
4ec044afd6 |
fix(vector): rank vec0 by cosine on unit-normalized copies (#275)
* test(vector): pin vec0 ranking to cosine order on non-unit rows vec0 ranks by L2 distance, which matches cosine order only for unit vectors. Insert rows whose L2 order disagrees with cosine order and assert vec0 returns cosine order, agrees with the exact cosine scan, skips zero rows and rejects a zero query. Pins a vector-enabled env so the cases run under the CI test env. Signed-off-by: trvon <me@trevon.dev> * fix(vector): rank vec0 by cosine on unit-normalized copies vec0 ranks by L2 distance, which matches cosine order only when both sides are unit length. Stored rows reach the vectors table at whatever magnitude the provider, config, import or sync produced, and the vec0 aux table copied them as-is, so vec0 selected and ordered rows differently from the Simeon PQ and exact cosine paths. The vec0 table is a derived index rebuilt from vectors.embedding, so write the unit direction into it and leave the source blob, which export/sync and exact scoring read, untouched. Normalize the query the same way. Zero rows are left out of vec0 and never match; a zero or non-finite query is rejected as the exact scan rejects it. Returned rows are ordered by their cosine score with the PQ tie-break. No schema change. The aux table is rebuilt on first use in each process (ready dims are in-memory), so existing non-unit rows are normalized on the next rebuild without a repair step. Signed-off-by: trvon <me@trevon.dev> * test(topology): pin the k-means stored centroid to the unit direction Spherical k-means assigns against unit centroids but stored the shorter arithmetic mean. Assert the stored centroid is unit length along the member mean direction, with and without boundary spill. Add a SOAR case showing that with unit centroids the spill follows angular distance, and that a short mean centroid reorders the candidates. Reformat the touched k-means test file to the lint formatter. Signed-off-by: trvon <me@trevon.dev> * fix(topology): store the unit k-means centroid The k-means engine clusters by cosine against normalized centroids, then stored the unnormalized member mean as centroidEmbedding. Routing reads the centroid by cosine and does not care, but SOAR boundary spill measures Euclidean residuals against it. A loose cluster's short mean adds a radial term to those residuals, so spill candidates were ranked in a different space from the one the clusters were built and routed in. Store the unit mean direction for k-means, which is the converged Lloyd centroid. Connected-component and Louvain keep the arithmetic mean, as their tests and contract require. Correct the stale field comment. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
8000226a4c |
fix(wal): encode the entry header field by field, never via padding (#274)
* test(wal): pin v1 entry encoding to a Clang-written fixture
store_block_entry_clang_v1.bin is one StoreBlock entry serialized by the
current wal_entry.cpp built with Clang 21, which zero-fills the 4 bytes of
trailing Header padding. The new cases decode and verify it, re-encode the
same logical entry and require identical bytes, and round-trip fresh
entries through serialize/deserialize/verifyChecksum.
GCC 16 leaves that padding uninitialized in stableHeaderBytes(), so these
cases fail there.
Signed-off-by: trvon <me@trevon.dev>
* fix(wal): encode entry header field by field, never via padding
stableHeaderBytes() built the checksummed header with `Header stable{}`
plus member assignments and then memcpy'd sizeof(Header) bytes. That is
aggregate initialization, so the 4 bytes of trailing padding (offsets
44..47) are not guaranteed to be zero. Clang 21 memsets the object first;
GCC 16 at -O0 stores only the members, so stack garbage reached both the
serialized entry and the CRC, and verifyChecksum() failed on GCC builds.
Encode and decode the header field by field into a zero-filled 48-byte
array at the existing offsets, with static_asserts pinning every offset
and size. The checksum is CRC32 over that canonical encoding plus the
payload, which is exactly what Clang builds have always written, so their
WAL files verify unchanged. deserialize() also accepts an entry whose CRC
covers nonzero padding as stored on disk (written by earlier GCC builds)
and normalizes its in-memory checksum. WALFile and WALManager recovery
read headers through the new WALEntry::decodeHeader().
TransactionData (uint64 + uint32) has 4 bytes of trailing padding and was
copied whole into the payload, relying on the same unguaranteed zeroing
(GCC 16 and Clang 21 happen to zero it today); it is now encoded field by
field too. The remaining payload structs are padding-free, which
appendObjectBytes/readObject now enforce with
has_unique_object_representations_v.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
|
||
|
Trevon
|
e48d9f8d1f |
fix(test): use the daemon timeout variable for the harness isolation test (#273)
|
||
|
Trevon
|
d342132389 |
Merge pull request #272 from trvon/fix/windows-checkpoint-test
test: isolate checkpoint timeout completion |
||
|
Trevon
|
30b00cf469 |
Merge branch 'experimental' into fix/windows-checkpoint-test | ||
|
Trevon
|
2b42c79d94 |
Merge pull request #241 from trvon/bench/isolate-kpi-daemons-v2
test(bench): isolate KPI benchmark daemons from the developer environment |
||
|
Trevon
|
3840a6495d |
Merge branch 'experimental' into bench/isolate-kpi-daemons-v2 | ||
|
Trevon
|
2dd68c7b23 |
Merge branch 'experimental' into fix/windows-checkpoint-test | ||
|
Trevon
|
5406c3997a |
Merge pull request #271 from trvon/docs/v0.20.5-upgrade-closeout
docs: document the 0.20.3 upgrade migration |
||
|
trvon
|
6d733f7c54
|
test: isolate checkpoint timeout completion
Wait for caller startup before measuring completion and keep synchronous console I/O out of the checkpoint wait test. Preserve the 20 ms timeout and 250 ms assertion, and verify the queued/timeout telemetry. Keep a guarded rescue executor alive for late posts so the unbounded-wait negative control fails without leaving an unjoinable caller. Signed-off-by: trvon <git@trevon.dev> |
||
|
trvon
|
f1a36b06f6
|
docs: document the 0.20.3 upgrade migration
Signed-off-by: trvon <git@trevon.dev> |
||
|
Trevon
|
66b3d86688 |
Merge pull request #270 from trvon/chore/backmerge-main-v0.20.5
chore: back-merge main into experimental after v0.20.5 |
||
trvon
|
96587fefdc
|
chore: back-merge main into experimental after v0.20.5
Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
da06d748ac |
Merge pull request #269 from trvon/release-please--branches--main--components--yams
chore(main): release 0.20.5 |
||
github-actions[bot]
|
791095a70d |
chore(main): release 0.20.5 | ||
|
Trevon
|
37bde10b5b |
Merge pull request #268 from trvon/experimental
chore: promote experimental to main |
||
|
Trevon
|
8dd0ef1c33 |
docs: MSI daemon, p2p status, ONNX Runtime search order (#267)
* docs: MSI ships the daemon; p2p sync shipped in 0.20.0 install.md said the MSI lacks yams-daemon.exe and yams-mcp-server.exe. Both ship from the next published release (packaging/windows/yams.wxs). 0.20.4 was not published, so name 0.20.3 as the last MSI without them. roadmap.md listed p2p sync as unshipped. Whole-corpus replication shipped in 0.20.0. List what is still open. Signed-off-by: trvon <me@trevon.dev> * docs: give the ONNX Runtime search order install.md said only that a system copy is preferred over the bundled one. State the order from plugins/ort_runtime/ort_runtime_resolution.h: runtime_library pin, system copy >= 1.23, bundled lib/yams/onnxruntime/, with YAMS_ONNX_RUNTIME_LIB / YAMS_ONNX_RUNTIME_DIR overriding all three. Quote the real log line. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
1b6671528d |
fix(memory-sync): publish committed vectors and wrap the vector sweep (#266)
* test(memory-sync): cover vectors committed behind the backfill cursor The vector backfill cursor (document_hash, chunk_id) never wraps, and the backfill is the only outbound path for vectors, so a vector added or re-embedded behind the cursor after the first sweep is never published to peers. Add red tests for: - the sweep starting over to publish a new and a re-embedded vector that sort below its last position; - vectors an embedding job commits being published within the next backfill cycle, ahead of a sweep that would need many items to reach them; - the topology sweep still publishing its share while the vector sweep always has work. Declare ServiceManager::notifyMemorySyncEmbeddingsCommitted as a no-op seam for the embedding commit notification; the fix fills it in. Signed-off-by: trvon <me@trevon.dev> * fix(memory-sync): publish committed vectors and wrap the vector sweep The vector backfill cursor (document_hash, chunk_id) was set once and reset only by a daemon restart, and the backfill is the only outbound path for vectors. After the first full sweep, a vector added or re-embedded under a document hash that sorts below the cursor was never published to peers. Topology had the same bug and now sweeps in repeating passes; vectors did not. Eager publish: the embedding service already reports the documents a job committed vectors for (successHashes, after the vector insert and before readiness is published) through the callback that requested the topology rebuild. That callback is renamed setEmbeddingsCommittedCallback and the daemon now also passes the hashes to ServiceManager::notifyMemorySyncEmbeddingsCommitted. It only queues them (bounded at 4096 documents, deduplicated, dropped past the cap): it runs on the embedding worker, and the memory-sync service's lifetime and outbound budget belong to the sync cycle. The backfill serves the queue as its own domain right after documents, one vector row per item, walking each document's chunks with getVectorsPage. The queue is in memory; a restart loses it, but the sweep then starts over anyway. Catch-up sweep: when the vector sweep reaches the end of the table it resets its cursor and sits out the rest of the cycle, as the topology pass does, so the next pass publishes whatever changed behind it. publishIfChanged keeps repeat passes from writing unchanged records. R7's skip logic is kept: an unpublishable row is skipped and the cursor moves past it; it is logged on the first pass (and when committed) but not again on every later pass. Fairness: with both sweeps repeating, both always have work, so the strict priority order (documents, vectors, topology) would hand the vector sweep every item and starve topology. Documents and committed vectors keep priority; the two sweeps now alternate item by item, and the turn persists across cycles so a one-item budget still alternates. A sweep that is done for the cycle leaves its turns to the other. This was chosen over alternating whole passes because it needs one field of state, keeps both sweeps' latency bounded by their own size rather than the other's, and reuses the existing per-item budget loop unchanged. Signed-off-by: trvon <me@trevon.dev> * test(memory-sync): pin the vector-store environment in the cursor-wrap cases Same as the bad-record cases: under the CI environment (vectors disabled, sqlite-vec init skipped) VectorDatabase creates no tables and the vector inserts fail. On grim the 11 memory_sync_apply cases pass both in a clean environment and with the tests.yml environment. Signed-off-by: trvon <me@trevon.dev> --------- Signed-off-by: trvon <me@trevon.dev> |
||
|
Trevon
|
44daa47b94 |
fix(memory-sync): skip bad vector records instead of stalling the stage (#265)
* fix(ci): size the onnx resource budget for aarch64 page alignment libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget. Signed-off-by: trvon <me@trevon.dev> * fix(plugins): declare the S3 legacy factory with the plugin export macro |
||
|
Trevon
|
1701bcab66 |
fix(test): make Tests green on every platform (#264)
* fix(ci): size the onnx resource budget for aarch64 page alignment libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget. Signed-off-by: trvon <me@trevon.dev> * fix(plugins): declare the S3 legacy factory with the plugin export macro |
||
|
Trevon
|
dc7a4fb499 |
fix: unblock the 0.20.5 release (aarch64 budget, MSVC linkage, mobile configure, Windows MSI) (#263)
* fix(ci): size the onnx resource budget for aarch64 page alignment libyams_onnx_resource.so is 67,624 bytes on aarch64 against a 64,000-byte budget, which failed the v0.20.4 linux-arm64 release job. It holds 23 KB of sections; aarch64 links with 64 KiB segment alignment (LOAD align 0x10000), which pads the file. The budget was measured on x86_64. 81,000 is the aarch64 size plus 20% headroom. Every other aarch64 payload is within 86% of its budget. Signed-off-by: trvon <me@trevon.dev> * fix(plugins): declare the S3 legacy factory with the plugin export macro |