Maintain cluster roads across incremental rebuilds instead of clearing
them. `ConnectedComponentTopologyEngine::updateArtifacts` now keeps the
roads between clusters the update did not touch, drops every road with
an endpoint in a removed or rebuilt cluster (reused ids included), and
recomputes the rebuilt clusters' roads from the dirty region's raw
neighbor lists against the merged memberships, so a region document's
neighbor in an untouched cluster yields that road in both orientations.
Each source then re-ranks its top-K over the union with PR 6's order and
tie-break (weight desc, target id asc).
Only the region's neighbor lists are available during an update, so a
cross-boundary edge is observed from one side; the out-of-region target
is credited with the mirror observation its own list would have added.
That is exact on a symmetric neighbor graph and the stated approximation
otherwise. An untouched source can only re-rank the roads its last full
build kept, so the result is derived, not full.
Provenance on the batch: `roadsIncrementalPasses` counts passes since
the last full accumulation (0 = full roads) and `roadsFullBuildEpoch`
is the `topologyEpoch` of the snapshot that accumulation produced. A
full build leaves both at 0; an update sets the epoch from `existing`
when its roads were full and carries it forward otherwise. The fields
ride in the road sidecar header, now version 2 (u64 epoch, u32 passes
after the snapshot id); version 1 sidecars decode with both at 0, the
envelope format string is unchanged, and the main v1 payload and the
routing construction fingerprint do not see them. The store copies them
both ways. Roads off (`road_max_per_cluster = 0`, the default) still
yields no roads and zero provenance, so default behavior is unchanged.
`topology_build_utils.h` splits `accumulateClusterRoads` into pair-weight
accumulation (with the optional observed-document set that drives
mirroring), orientation and ranking, and adds `deriveIncrementalRoads`
over those pieces; the full-build path passes no observed set and is
unchanged.
Louvain and k-means `updateArtifacts` rebuild the supplied input from
scratch, so their roads are full roads for that batch with zero passes;
the test states and checks that property for them.
Tests: tests/unit/topology/topology_roads_incremental_catch2_test.cpp
(roads off through an update, full-build provenance, kept/retired/
recomputed roads with mirrored weights on a four-cluster graph, top-K
re-ranking over kept and recomputed roads, derivation without prior
roads, a property test over seeded planted-cluster symmetric graphs
with cross-edge rewiring and member moves comparing incremental roads to
a from-scratch build after each of several dirty regions, a tight-top-K
property for rebuilt sources and derived weights, and the alternate
engines), plus sidecar v2 round trip, hand-built v1 decode, future
version rejection and store round trip of the provenance in
topology_road_sidecar_catch2_test.cpp.
Signed-off-by: trvon <me@trevon.dev>
Persist `TopologyArtifactBatch::roads` in a sidecar knowledge-graph node,
`topology:roads:<snapshotId>` of type `topology_road_snapshot`, instead
of growing the main snapshot codec. The sidecar has its own binary magic
('Y','A','M','R'), version 1, and a Zstd JSON envelope with
`format = "topology_roads_zstd_v1"`. The main snapshot stays a plain v1
payload, byte-identical with roads on or off, so memory-sync replicas on
older peers keep decoding it, and an unknown node type is simply never
read by an old store. `deserializeTopologyBatchCompressed` now rejects an
envelope whose `format` it does not know instead of treating it as legacy
raw JSON.
Header layout (little-endian): u32 magic, u32 version, u32 maxPerCluster,
u32 reserved, string snapshotId, u32 cluster-id table count + strings,
u32 road count, then per road u32 from index, u32 to index, f64 weight.
`TopologyArtifactBatch::roadMaxPerCluster` records the bound the builder
applied so the header can carry it; both engines set it.
The store writes the sidecar right after the snapshot node and before
the `topology:snapshot:latest` pointer, so the pointer remains the commit
point: an injected sidecar failure leaves the previous snapshot current.
Retention deletes each pruned snapshot's sidecar with it, and re-storing
a snapshot id without roads removes its stale sidecar. Loading is
optional and soft: a missing sidecar yields empty roads with no error; an
undecodable or invalid one (snapshot mismatch, endpoint not a cluster,
self loop, negative or non-finite weight, more roads per source than the
declared top-K) drops the roads and records
`TopologyArtifactBatch::roadUnavailableReason`, surfaced on the routing
session and as the `topology_road_unavailable_reason` debug key, while
the snapshot still loads and routes.
`topologyRoadFingerprint(batch)` gives roads their own order-independent
identity for cache provenance; the routing construction fingerprint is
unchanged and stays road-free through the persisted path.
Lean (formal/topology/Yams/Topology/Artifacts.lean): `ClusterRoad`,
`WellFormedRoads` (endpoints in clusters, no self loop, non-negative
weight, bounded per source), `constructionIdentity`, and lemmas that the
construction identity and `WellFormedBatch` are independent of roads and
that an empty road layer is well formed. Checked with lake (lean 4.31.0).
Tests: tests/unit/topology/topology_road_sidecar_catch2_test.cpp (codec
round trip, foreign/truncated payload rejection, main payload identical
on/off, validation reasons, store round trip, missing sidecar, every
corrupt-sidecar case with routing still admitted, write-before-pointer,
retention, fingerprints through the persisted path) and the extended
fingerprint invariance test in topology_roads_catch2_test.cpp.
`road_to` knowledge-graph edges are deferred until a consumer needs them.
Signed-off-by: trvon <me@trevon.dev>
Trevon