storeBatch loads the previous snapshot only to find documents that left
the topology, but returned early when it could not decode it, so one
corrupt payload blocked every future topology write. On a decode error,
log and store the new snapshot without clearing dropped documents' keys;
other load errors still propagate.
Topology wrote back only an opaque compressed snapshot node, so yams
graph, explore and graph expansion could not traverse clusters, and
every rebuild added another full snapshot node that was never removed.
Each stored snapshot now replaces the cluster graph in one KG write
batch before the latest pointer flips: topology:cluster:<id> nodes with
size, level, persistence, cohesion and density; document member_of
edges weighted by persistence, medoid_of edges, overlaps edges for
boundary-spill memberships, and subcluster_of hierarchy edges. After the
pointer flips, snapshot nodes beyond the newest three are removed; the
one the pointer names is always kept.
storeBatch wrote per-document keys before the snapshot node, so a
failure in between left documents pointing at a snapshot that was never
persisted. Write the snapshot node first and the latest pointer last so
a partial store leaves the previous snapshot authoritative.
Remove only the two per-document keys still written, in one transaction
via a new removeMetadataBatch, and stop the membership slow path from
reading structural keys that migration 41 prunes.
The metadata store cached the latest batch in an unguarded mutable
optional that search threads read and wrote, and handed out a full copy
on every loadLatest, so search kept the snapshot resident twice.
Publish the latest batch and its membership index as one immutable
shared snapshot under a mutex, add loadLatestShared to hand it out
without copying, and feed the routing snapshot cache from it. Historical
loads no longer displace the resident latest snapshot, and a reader that
raced a write never installs the older batch.
- Add Migration 41 to drop and recreate metadata_value_counts triggers with
'WHEN ... AND key NOT LIKE topology.%' condition, eliminating ~650,000 trigger
executions per topology rebuild in a 72k document corpus.
- Purge legacy continuous float/structural keys from metadata table
(persistence_score, cohesion_score, bridge_score, role, cluster_level,
parent_cluster_id, overlap_cluster_ids_json) in Migration 41.
- In MetadataKgTopologyArtifactStore::storeBatch, replace N individual
getDocumentByHash calls with single batchGetDocumentsByHash and write
only essential snapshot_id and cluster_id to metadata table.
- In MetadataKgTopologyArtifactStore::loadMemberships, add in-memory fast
path serving memberships directly from cached latest snapshot batch.
- Update serializeTopologyBatchCompressed to handle uncompressed tiny batches.
- Add Catch2 unit tests in tests/unit/topology/topology_metadata_store_catch2_test.cpp
validating trigger bypass, metadata pruning, and fast-path membership loading.
Signed-off-by: trvon <me@trevon.dev>
