The gate's own documentation claimed more than the tool did, in ways that mattered once the integrity checks landed. - "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it is now enforced (previous commit) and the text says which criterion the tool applies: complete evidence, refused false, every pooled cell passing. - The model digest proves agreement ACROSS reports. It cannot prove that the file on disk now is the one that was evaluated. Corrected everywhere. - The remedy was described inconsistently: cutting the training bundles earlier is NOT enough (evaluation still replays the whole store), a bounded evaluation range is required. Written consistently, with --eval-after NS|auto. - scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It now asserts >= 5, requires the provenance fields, and adds the honest-evidence invariant: today every cell is in-sample, so at least one cell must report in_sample true, none may pass, and each must carry the out-of-sample failure. A pass over in-sample cells is itself a failure. - The Salt/range runbooks advertised the flow as producing promotion evidence; it collects gate INPUT, and a passing verdict requires out-of-sample windows.
5.7 KiB
Stage 4 inspection model: pre-registered promotion gate
Written 2026-10-27, before any promotion. The criteria below decide whether a
shadow model may be promoted (strawberry model promote, which checks them).
A changed method gets a new dated document; these numbers do not move after
results are seen. (Also in YAMS as source=decision,
task=strawberry-data-engine.)
What a promoted model may do
Request extra inspections only (memory scan, deeper rescan, closer watch of a window's descendants, rescan on the next rules update), within an hourly CPU budget (10/30/120 CPU-seconds by scan profile). It never skips, delays or replaces a rule-triggered scan, never blocks, contains or kills, and its output is never a finding by itself (Stage 5 decides that separately).
Measurement
strawberry replay --model <shadow.sbm> --policy-eval [--inject FILE] over
the host's telemetry:
- Candidates: every ranked window after the warm-up (default 12 h).
- Positives ("extra"): windows that rules did not flag at exec time and that
later got a late label (retro-hunt after a rules update, or an operator
quarantine/kill), plus injected evaluation-trace windows
(
config/traces/standard.trace; never the traces a model was trained on). Amended 2026-10-28 by the threat-trace pre-registration: plusthreats-eval.trace, and a criterion 5 on new threat families. - Policies, each with its own hourly budget and the same costs and reach
(
ShadowOptions): the model (its value through the shared chooser) and rarity-greedy (min(1, rarity score / 48) through the same chooser). - Efficiency = hits / inspection CPU-seconds. Recall = hits / positives.
Gate (all must hold)
- At least 10 extra positives in the evaluated history.
- Model efficiency >= 1.5 x rarity-greedy efficiency.
- Model recall >= rarity-greedy recall (efficiency is not bought by inspecting less of what matters).
- Model inspections stay within budget by construction; refusals are reported, not counted against it.
model promote checks 1-3 on the current history and refuses otherwise.
Integrity checks around the criteria (added 2026-11-16)
The criteria above are unchanged. Two checks were added because a cell could otherwise satisfy them without measuring anything, and a third records what was judged:
- Degenerate baseline. If rarity-greedy scores 0 hits or 0 inspection CPU-seconds, criteria 2 and
3 pass for free (it cannot be beaten, or cannot be measured). Such a cell now fails with
FAIL: degenerate baseline: ...naming its budget-refusal count. It is an evaluation without a baseline, not evidence. - Out-of-sample. A model trained on the windows it is evaluated on shows nothing about
generalization. Models now record the inclusive range of windows they trained on (
model train,fleet train), and each gate cell reports the model digest, that training range, the range it evaluated and whether they overlap. An overlapping cell fails withFAIL: out-of-sample: in-sample evaluation ...; a model with no recorded training range (a file written before this field existed, or one whose identity is not recorded) fails as unprovable. It is never assumed out-of-sample. - Content identity. Reports bind the judged model by the first 16 hex of its file sha256, not by
filename, and
fleet gaterefuses a report set whose cells carry no digest or whose digests disagree for a (model, period) cell. That catches a mixed set — a staledist/gate/model averaged with the current one — and nothing more. It does not establish that the file on disk now is the file that was evaluated: reports written before a model was replaced still agree with each other and still describe bytes the node no longer has. Treat the digest as agreement across reports, not as provenance for the file you hold.
Consequence for single-host promotion: replaying the live store, which contains the training history,
is in-sample for a model trained on that store, so model promote now refuses it. A promotion
requires training on history up to a cut and evaluating on later, disjoint windows. Cutting the
training history earlier is not sufficient on its own: unless the evaluation is bounded too, the
replay still scores windows inside the range the model records as its training range, and the gate
fails closed. The missing piece is a bounded evaluation range: replay --policy-eval --eval-after NS|auto, restricting the evaluated windows to those starting at or after an absolute
bound, with auto using the latest training-range end among the --eval-model files so every cell is
out-of-sample by construction. The replay side is in place (ReplayOptions::eval_after_ns,
eval_after_from_models, and a provenance eval_error naming the reason when no window survives the
bound); the --eval-after CLI flag is landing in parallel, so unless strawberry replay --help lists
it there is no bounded range and no passing single-host promotion path (API_CONTRACT.md records the
shape). Until then a
model promote run on live history fails with FAIL: out-of-sample: ... and writes no
promoted.sbm.
Before relying on a promoted model, the plan's fuller check also applies:
the gate holds on 3 non-overlapping replay periods and for 3 training seeds.
The tool reports the single run; the operator records the others (YAMS
source=evidence).
Known limits
- Payoffs assume an inspection of a positive window surfaces
reach(action)of it. Until the model has acted, real inspection outcomes are not observed; the policy head stays untrained until they are. - With few real late labels, the gate leans on injected traces, which measure targeting of known attack shapes, not of novel ones.