Four findings from a four-lane audit made the promotion gate unable to support
its own claims.
In-sample evaluation: training walks the whole post-warm-up store and the gate
evaluates the same windows, with nothing checking provenance - replay.cpp itself
listed point_in_time_splits_and_overlap_not_assessed. Models now record the
window range they learned from (file format v2; v1 still loads as unknown) and a
content fingerprint, evaluation reports its own range plus in_sample, and
promotion_gate fails with the reason: an overlapping range, no recorded range,
or no digest. Unprovable provenance never counts as out-of-sample.
Degenerate baseline: with rarity-greedy on 0 hits, hits_per_cpu_second() returns
0.0, so ">= 1.5x rarity" reduced to "model hit something" and the recall check
was vacuous. A cell whose baseline scored no hits or spent no CPU now fails,
naming the budget refusals that starved it.
Stale models: cells keyed on a bare filename, so a later cycle that refused to
train left an older cycle's dist/gate models to be judged, and nothing bound a
verdict to the file it judged. Cells and reports carry the model digest, and
fleet gate sets refused (non-zero exit) for a cell without one or for a
(model, period) whose digests disagree across nodes instead of pooling them.
3 seeds are not generalization: they vary initialisation and the per-node
shuffle over identical windows. The procedure document now says so and states
what would test generalization (disjoint periods, models trained on an earlier
cut, out-of-sample ranges).
Tests: in-sample flagged, disjoint range not flagged, unprovable provenance
fails, 0-hit and 0-CPU baselines fail, mixed digests refused, legacy report
refused, range round-trip and fingerprint, v1 compatibility.