* test(topology): cover incremental rebuilds for louvain and kmeans
An incremental TopologyManager rebuild with the louvain or kmeans engine
publishes a snapshot that holds only the dirty documents. The store then
drops topology.cluster_id from every untouched document. Add a failing
case for both engines, and a regression case that the connected engine
still rebuilds only the seed's prior cluster.
Signed-off-by: trvon <me@trevon.dev>
* fix(topology): rebuild the corpus when a dirty region needs it
TopologyManager logged requiresWiderRebuild and then published the
engine's region update anyway. Louvain and KMeans have no incremental
path: their dirty region is the seed set and their update rebuilds only
those documents, so the published snapshot held just the dirty docs and
the store cleared the topology keys of every other document.
Honor the flag: re-extract the full corpus and call buildArtifacts, and
report the run as a full rebuild with one fallback. The connected engine
sets the flag only when a region exceeds its seed or document budget;
under budget its incremental path is unchanged. Over budget it used to
drop seeds past the cap, and now rebuilds the corpus.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Give every cluster with at least two members a walk order:
`ClusterArtifact::orderedMembers`, a uint32 permutation over the
hash-sorted `memberDocumentHashes`, plus `orderKind`. Three kinds, each
deterministic with ties resolved to the smaller member hash:
- Time: ascending `TopologyDocumentInput::eventTime`; members without a
time last, by hash.
- Session: grouped by `sessionId`; groups ordered by earliest event time
(groups with no time after those with one), then session id; inside a
group by time then hash; unsessioned members last, by time then hash.
- SimilarityChain: greedy nearest-neighbor tour by cosine over the
members' embeddings, starting from the cluster representative
(`medoid`, else the smallest-hash usable member); members without a
usable embedding last, by hash.
The knob is `TopologyBuildConfig::roadOrder`, resolved from TOML
`topology.road_order` = none|time|session|similarity_chain through the
topology.* resolver block and `TopologyManager::setRoadOrder`, threaded
into the build config like `roadMaxPerCluster`. Orders are computed after
boundary spill so they cover replicas, at full build in every engine and
for the rebuilt clusters of a connected-engine incremental update;
untouched clusters keep theirs (a kind change reaches them on the next
full rebuild, like the representative rule) and `none` clears every
order. Orders are independent of `roadMaxPerCluster`.
Orders persist in the road sidecar, now version 3: after the roads, u32
count, then per ordered cluster u32 cluster-id table index, u32 kind,
u32 length and the permutation as u32 indices. A sidecar without orders
is still written as v2, so roads-only sidecars are byte-identical to the
previous layout; v1 and v2 decode with no orders, and the envelope gains
`member_order_count`. The sidecar is written when the batch has roads or
at least one ordered cluster. On load each order is validated on its own
(`attachTopologyMemberOrders`: cluster exists, kind not none,
permutation covers the members exactly once); an invalid entry drops
only that cluster's order and appends `<token>:<clusterId>` to the new
load-time diagnostic `TopologyArtifactBatch::orderUnavailableReason`.
Roads never lose anything to an order error and vice versa.
The main snapshot codec, the routing construction, representation and
road fingerprints never see orders, so the default `none` leaves the
snapshot payload, the sidecar and every fingerprint byte-identical. No
search consumer yet; the road walk lands separately.
Tests: tests/unit/topology/topology_ordered_members_catch2_test.cpp
(permutation validator tokens, time order with missing times, session
grouping and group tie-break, hand-computed greedy chain from the medoid
on a five-point fixture with fallback start, unusable members and exact
ties, every engine stores valid permutations of every kind including over
spill replicas, default none byte-identical payload/sidecar/fingerprints,
sidecar v3 round trip plus hand-built v2 bytes and corrupt-section
rejection, per-cluster attach validation with reasons, store round trip
with orders-only sidecars, invalid order dropped with reason while roads
and other orders stay, incremental update keeps untouched orders and
recomputes rebuilt ones), plus resolver and manager setter coverage.
Signed-off-by: trvon <me@trevon.dev>
Add cluster roads: undirected cluster-to-cluster links whose weight is
the sum of every raw neighbor observation between members of two
clusters. The shared helper `accumulateClusterRoads` in
topology_build_utils.h reads `TopologyDocumentInput::neighbors` before
the reciprocal/minEdgeScore filter, because the connected engine's
filtered adjacency only carries cross-cluster edges where an oversized
component was split, and k-means never consults the graph.
Each cluster keeps its top-K roads (weight desc, then target cluster id
asc) as `ClusterRoad {fromClusterId, toClusterId, weight}` on
`TopologyArtifactBatch::roads`, sorted by source cluster. Both the
connected engine and buildBatchFromAssignment (kmeans/louvain) call it;
an incremental update clears roads until the next full rebuild.
The knob is `TopologyBuildConfig::roadMaxPerCluster`, resolved from
TOML `topology.road_max_per_cluster` through the topology.* resolver
block and `TopologyManager::setRoadMaxPerCluster`. The default 0
computes no roads; the binary codec and the routing construction
fingerprint ignore the field, so existing snapshots and fingerprints
are unchanged with roads on or off.
Tests: tests/unit/topology/topology_roads_catch2_test.cpp (raw vs
filtered neighbors, top-K and tie-break, order independence, both
orientations, out-of-batch and non-finite scores, kmeans and louvain,
codec bytes and fingerprint unchanged), plus resolver and manager
setter coverage.
Signed-off-by: trvon <me@trevon.dev>
TopologyBuildConfig::dirtyRegionExpansion had no path from config:
TopologyManager never set it, so every incremental rebuild used the
default (PriorClusterAndNeighbors) and the NeighborsOnly and Adaptive
branches in ConnectedComponentTopologyEngine::defineDirtyRegion never ran.
It now follows the typed-config path the other topology knobs use:
[topology]
dirty_region_expansion = "neighbors_only" # or prior_cluster_and_neighbors, adaptive
ConfigResolver::resolveTopologyEnginePolicy parses it (case-insensitive;
an unknown value is logged and ignored, never guessed),
ServiceManager::configureTopologyRuntime applies it, and TopologyManager
passes it to the incremental dirty-region config and the build config.
dirtyRegionExpansionModeName / parseDirtyRegionExpansionMode give the
enum one spelling.
Adaptive currently takes every branch PriorClusterAndNeighbors takes, so
selecting it changes nothing yet. A characterization test pins that, so
a real adaptive policy has to change it deliberately.
Tests (the manager test fails with the TopologyManager line removed):
- topology_manager: an incremental rebuild seeded from a clustered
document reports "prior_cluster+neighbors" by default and "neighbors"
under neighbors_only;
- config_resolver: a recognized mode resolves; an unknown or absent one
leaves the default;
- topology_baseline: NeighborsOnly excludes prior-cluster members;
Adaptive matches the default; names round-trip.
Found by pipe's experimental C++ front end: NeighborsOnly and Adaptive
were compared in topology_baseline.cpp but never produced anywhere.
* fix(topology): rebuild stale memberships
Incremental reuse can retain deleted members outside the dirty region.
Rebuild coherent artifacts from current metadata when prior members vanish;
retain final validation, pending dirty work, and unready-corpus deferral.
Cover deleted seeds, partial clusters, empty corpora, dry runs, and late
deletion without changing the live corpus or claiming a speedup.
Signed-off-by: trvon <me@trevon.dev>
* Add cstdint and optional includes to test file
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Signed-off-by: trvon <me@trevon.dev>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
