The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.
- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
is now enforced (previous commit) and the text says which criterion the tool
applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
is NOT enough (evaluation still replays the whole store), a bounded evaluation
range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
now asserts >= 5, requires the provenance fields, and adds the honest-evidence
invariant: today every cell is in-sample, so at least one cell must report
in_sample true, none may pass, and each must carry the out-of-sample failure.
A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
it collects gate INPUT, and a passing verdict requires out-of-sample windows.
Four findings from a four-lane audit made the promotion gate unable to support
its own claims.
In-sample evaluation: training walks the whole post-warm-up store and the gate
evaluates the same windows, with nothing checking provenance - replay.cpp itself
listed point_in_time_splits_and_overlap_not_assessed. Models now record the
window range they learned from (file format v2; v1 still loads as unknown) and a
content fingerprint, evaluation reports its own range plus in_sample, and
promotion_gate fails with the reason: an overlapping range, no recorded range,
or no digest. Unprovable provenance never counts as out-of-sample.
Degenerate baseline: with rarity-greedy on 0 hits, hits_per_cpu_second() returns
0.0, so ">= 1.5x rarity" reduced to "model hit something" and the recall check
was vacuous. A cell whose baseline scored no hits or spent no CPU now fails,
naming the budget refusals that starved it.
Stale models: cells keyed on a bare filename, so a later cycle that refused to
train left an older cycle's dist/gate models to be judged, and nothing bound a
verdict to the file it judged. Cells and reports carry the model digest, and
fleet gate sets refused (non-zero exit) for a cell without one or for a
(model, period) whose digests disagree across nodes instead of pooling them.
3 seeds are not generalization: they vary initialisation and the per-node
shuffle over identical windows. The procedure document now says so and states
what would test generalization (disjoint periods, models trained on an earlier
cut, out-of-sample ranges).
Tests: in-sample flagged, disjoint range not flagged, unprovable provenance
fails, 0-hit and 0-CPU baselines fail, mixed digests refused, legacy report
refused, range round-trip and fingerprint, v1 compatibility.
Written before any threat trace is collected or evaluated. Train on
families first reported before 2025-07-01, evaluate on families first
reported after it (new-family bucket primary), from public write-ups and
Atomic Red Team only: Triage and MalwareBazaar terms forbid ML training.
Adds gate criterion 5 (model recall >= rarity-greedy on new families).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- replay --policy-eval: over ranked windows, simulates the model and
rarity-greedy (the same chooser, rarity score / 48 as the value), each
with its own hourly budget, and tallies the shadow decisions the daemon
logged. Hits are inspected "extra" positives: late labels or injected
evaluation traces on windows rules did not flag at exec time. Labels are
known after the fact, so every action's payoff is computable without
propensity weighting. Reports CPU-seconds, inspections, hits, recall,
hits per CPU-second and budget refusals.
- docs/evaluation/stage4-promotion-gate.md (pre-registered): at least 10
extra positives, model efficiency >= 1.5x rarity-greedy, model recall
not lower. promotion_gate() checks it.
- strawberry model promote [--inject FILE] [--warmup H] [--yes]: replays
with the shadow model, applies the gate and only then writes
promoted.sbm; asks for confirmation.
- learn/inspection.hpp: options, the greedy chooser and the
inspection_action record, shared by shadow mode and the evaluation.
- Fix: the hourly budget reset whenever a window closed with an earlier
timestamp than the hour's start (windows close roughly, not strictly,
in time order), letting spend exceed the budget; in both shadow mode
and the evaluation.
Tests: test_replay R11 (gate criteria; extra positives exclude rule-flagged
windows; spend within the budget; logged shadow decisions joined to their
windows); CLI translations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>