# Stage 4 inspection model: pre-registered promotion gate Written 2026-10-27, before any promotion. The criteria below decide whether a shadow model may be promoted (`strawberry model promote`, which checks them). A changed method gets a new dated document; these numbers do not move after results are seen. (Also in YAMS as `source=decision`, `task=strawberry-data-engine`.) ## What a promoted model may do Request extra inspections only (memory scan, deeper rescan, closer watch of a window's descendants, rescan on the next rules update), within an hourly CPU budget (10/30/120 CPU-seconds by scan profile). It never skips, delays or replaces a rule-triggered scan, never blocks, contains or kills, and its output is never a finding by itself (Stage 5 decides that separately). ## Measurement `strawberry replay --model --policy-eval [--inject FILE]` over the host's telemetry: - Candidates: every ranked window after the warm-up (default 12 h). - Positives ("extra"): windows that rules did not flag at exec time and that later got a late label (retro-hunt after a rules update, or an operator quarantine/kill), plus injected evaluation-trace windows (`config/traces/standard.trace`; never the traces a model was trained on). Amended 2026-10-28 by [the threat-trace pre-registration](stage4-threat-traces-preregistration.md): plus `threats-eval.trace`, and a criterion 5 on new threat families. Amended 2026-11-20 by [the rarity-baseline pre-registration](stage4-rarity-baseline-preregistration.md): the comparator in criteria 2 and 3 is `rarity_calibrated` (the same rarity value and chooser, behind a budget-matched threshold frozen before the evaluated span); rarity-greedy as defined below stays reported as a diagnostic. The criteria and their thresholds are unchanged. - Policies, each with its own hourly budget and the same costs and reach (`ShadowOptions`): the model (its value through the shared chooser) and rarity-greedy (min(1, rarity score / 48) through the same chooser). - Efficiency = hits / inspection CPU-seconds. Recall = hits / positives. ## Gate (all must hold) 1. At least 10 extra positives in the evaluated history. 2. Model efficiency >= 1.5 x rarity-greedy efficiency. 3. Model recall >= rarity-greedy recall (efficiency is not bought by inspecting less of what matters). 4. Model inspections stay within budget by construction; refusals are reported, not counted against it. `model promote` checks 1-3 on the current history and refuses otherwise. ### Integrity checks around the criteria (added 2026-11-16) The criteria above are unchanged. Two checks were added because a cell could otherwise satisfy them without measuring anything, and a third records what was judged: - **Degenerate baseline.** If rarity-greedy scores 0 hits or 0 inspection CPU-seconds, criteria 2 and 3 pass for free (it cannot be beaten, or cannot be measured). Such a cell now fails with `FAIL: degenerate baseline: ...` naming its budget-refusal count. It is an evaluation without a baseline, not evidence. - **Out-of-sample.** A model trained on the windows it is evaluated on shows nothing about generalization. Models now record the inclusive range of windows they trained on (`model train`, `fleet train`), and each gate cell reports the model digest, that training range, the range it evaluated and whether they overlap. An overlapping cell fails with `FAIL: out-of-sample: in-sample evaluation ...`; a model with no recorded training range (a file written before this field existed, or one whose identity is not recorded) fails as unprovable. It is never assumed out-of-sample. - **Content identity.** Reports bind the judged model by the first 16 hex of its file sha256, not by filename, and `fleet gate` refuses a report set whose cells carry no digest or whose digests disagree for a (model, period) cell. That catches a **mixed** set — a stale `dist/gate/` model averaged with the current one — and nothing more. It does not establish that the file on disk now is the file that was evaluated: reports written before a model was replaced still agree with each other and still describe bytes the node no longer has. Treat the digest as agreement across reports, not as provenance for the file you hold. Consequence for single-host promotion: replaying the live store, which contains the training history, is in-sample for a model trained on that store, so `model promote` now refuses it. A promotion requires training on history up to a cut and evaluating on later, disjoint windows. Cutting the training history earlier is **not** sufficient on its own: unless the evaluation is bounded too, the replay still scores windows inside the range the model records as its training range, and the gate fails closed. The missing piece is a **bounded evaluation range**: `replay --policy-eval --eval-after NS|auto`, restricting the evaluated windows to those starting at or after an absolute bound, with `auto` using the latest training-range end among the `--eval-model` files so every cell is out-of-sample by construction. The replay side is in place (`ReplayOptions::eval_after_ns`, `eval_after_from_models`, and a provenance `eval_error` naming the reason when no window survives the bound); the `--eval-after` CLI flag is landing in parallel, so unless `strawberry replay --help` lists it there is no bounded range and no passing single-host promotion path (`API_CONTRACT.md` records the shape). Until then a `model promote` run on live history fails with `FAIL: out-of-sample: ...` and writes no `promoted.sbm`. Before relying on a promoted model, the plan's fuller check also applies: the gate holds on 3 non-overlapping replay periods and for 3 training seeds. The tool reports the single run; the operator records the others (YAMS `source=evidence`). ## Known limits - Payoffs assume an inspection of a positive window surfaces `reach(action)` of it. Until the model has acted, real inspection outcomes are not observed; the policy head stays untrained until they are. - With few real late labels, the gate leans on injected traces, which measure targeting of known attack shapes, not of novel ones.