* fix(storage): bound filesystem listPage residency
listPage() walked the whole op store and retained every matching key before
slicing one page. At 1025 records and a 512-record page it held 1025 keys
where 513 is the working set, and the retained set grew with the store.
Traverse once into a max-heap holding the smallest eligible limit+1 keys.
The walk itself stays complete, so a traversal error still returns an error
instead of a partial page. Exclusive lexical cursor, ordering, Windows key
decoding and the SIZE_MAX limit stay as they were.
Add a YAMS_TESTING-only observation seam that counts real walks, entries
visited and peak retained keys, so the traversal cost can be measured
instead of inferred. The new traversal shifts two reviewed getenv allowlist
line numbers.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): keep backfill progress past an oversized vector
A record above the resident limit failed publication, the domain cursor
never advanced, and every later cycle retried the same row. Valid vectors
behind it never reconciled, so one oversized value could pin a domain
indefinitely. The daemon log showed this repeating every ~100 s.
Classify the row only after a publish fails with InvalidArgument and the
same serialized record is independently confirmed to exceed the immutable
limit min(maxValueBytes, maxCacheBytes). Skip the row for this pass, keep
the local copy, and revisit it when a new source generation or service
limit appears. A generic backend InvalidArgument stays retryable, and
healthy rows still serialize only once.
Signed-off-by: trvon <git@trevon.dev>
* fix(memory-sync): reuse a bounded complete listing across publishes
Every store invalidated the cached listing pages, so a publish followed by
syncFully walked the whole op store again, once per 512-record page. On the
profiled daemon that re-walk was the hot path: 99.6% of one core, 87.9% of
it system time, with listPage at 55% of user samples.
Keep a complete key-only prefix snapshot, built only from normal contiguous
pages starting at an empty cursor on the concrete filesystem backend. The
snapshot is capped by key count and key bytes, served only in lexical order,
and dropped as incomplete unless the walk reached the end. Any mutation
invalidates it before the backend call; a successful store or remove then
applies the exact key to matching snapshots, and a failure drops them
because the write may have committed. Initialising the async path disables
cache reuse permanently rather than racing a future.
Cache maintenance is best effort: it runs after the backend result is known
and any allocation failure clears the cache and keeps the committed result.
Opaque cursors are untouched, so only the built-in filesystem backend gets
snapshot treatment.
Active publishes now cost the same number of walks as a warm sweep instead
of four, six and nine at 511, 513 and 1025 records.
Signed-off-by: trvon <git@trevon.dev>
* test(bench): measure memory-sync listing cost across arms
Add a standalone harness that drives a real filesystem op store through
MemorySyncLoop, one scenario per process so no cache state survives a
repeat, and reports wall, user, system and peak RSS per run. It uses
production APIs only, so the same source builds against a baseline and a
candidate tree and neither arm needs a measurement seam ported into it.
The driver seeds one corpus per size, interleaves the two arms inside each
repeat, and writes results.jsonl plus a median report. Enumeration counts
are not re-derived here; the unit suite owns those.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <git@trevon.dev>
33243d8472