strawberry/docs/evaluation/stage4-promotion-gate.md
trvon f4adc12a74
gate: judge the model against a calibrated rarity comparator
The rarity-greedy comparator acted on every evaluated window, so it spent
each hour's budget on the first ~15 windows and scored 0 hits in every
cell. Criteria 2 and 3 of the Stage 4 gate were void against it.

rarity_calibrated keeps the same rarity value, chooser, costs and budget,
and adds a per-node threshold frozen before the evaluated span: the m-th
largest value over the trailing 24 h of label-free history between the
warm-up and the --eval-after bound, with m set so expected spend matches
the hourly budget. Without that history the comparator is uncalibrated
and the gate fails by name; it never falls back to another rule.

rarity_greedy (the old first-come rule) and rarity_hindsight (per-hour
ordering, not causal) are reported as diagnostics and never judged.
fleet gate pools counts, keeps thresholds per node, and fails a pooled
cell when any node is uncalibrated or lacks the comparator.

Contract change: in `fleet gate --json`, rarity_hits and
rarity_cpu_seconds now describe the calibrated comparator; the old
numbers are rarity_greedy_hits and rarity_greedy_cpu_seconds.

Pre-registration: docs/evaluation/stage4-rarity-baseline-preregistration.md,
sha256 4916c95cd2581e3749acfe46c3ccf827aa5ce98c06150b49b978fc70da013f95,
indexed in YAMS at 2026-11-20T18:55:48Z, before the code. This commit is
dated after the registration time in that document (2026-11-21T00:00Z);
the YAMS record is what predates it.

Checked: meson suite 58/58, scripts/fleet_e2e.sh, clang-tidy and the
strict analyzer on the changed TUs. Not run: fold on the changed TUs,
make fuzz, the live Docker tests and the perf KPI. The comparator has
not been run on validation-range data.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 19:20:55 -07:00

6.1 KiB

Stage 4 inspection model: pre-registered promotion gate

Written 2026-10-27, before any promotion. The criteria below decide whether a shadow model may be promoted (strawberry model promote, which checks them). A changed method gets a new dated document; these numbers do not move after results are seen. (Also in YAMS as source=decision, task=strawberry-data-engine.)

What a promoted model may do

Request extra inspections only (memory scan, deeper rescan, closer watch of a window's descendants, rescan on the next rules update), within an hourly CPU budget (10/30/120 CPU-seconds by scan profile). It never skips, delays or replaces a rule-triggered scan, never blocks, contains or kills, and its output is never a finding by itself (Stage 5 decides that separately).

Measurement

strawberry replay --model <shadow.sbm> --policy-eval [--inject FILE] over the host's telemetry:

  • Candidates: every ranked window after the warm-up (default 12 h).
  • Positives ("extra"): windows that rules did not flag at exec time and that later got a late label (retro-hunt after a rules update, or an operator quarantine/kill), plus injected evaluation-trace windows (config/traces/standard.trace; never the traces a model was trained on). Amended 2026-10-28 by the threat-trace pre-registration: plus threats-eval.trace, and a criterion 5 on new threat families. Amended 2026-11-20 by the rarity-baseline pre-registration: the comparator in criteria 2 and 3 is rarity_calibrated (the same rarity value and chooser, behind a budget-matched threshold frozen before the evaluated span); rarity-greedy as defined below stays reported as a diagnostic. The criteria and their thresholds are unchanged.
  • Policies, each with its own hourly budget and the same costs and reach (ShadowOptions): the model (its value through the shared chooser) and rarity-greedy (min(1, rarity score / 48) through the same chooser).
  • Efficiency = hits / inspection CPU-seconds. Recall = hits / positives.

Gate (all must hold)

  1. At least 10 extra positives in the evaluated history.
  2. Model efficiency >= 1.5 x rarity-greedy efficiency.
  3. Model recall >= rarity-greedy recall (efficiency is not bought by inspecting less of what matters).
  4. Model inspections stay within budget by construction; refusals are reported, not counted against it.

model promote checks 1-3 on the current history and refuses otherwise.

Integrity checks around the criteria (added 2026-11-16)

The criteria above are unchanged. Two checks were added because a cell could otherwise satisfy them without measuring anything, and a third records what was judged:

  • Degenerate baseline. If rarity-greedy scores 0 hits or 0 inspection CPU-seconds, criteria 2 and 3 pass for free (it cannot be beaten, or cannot be measured). Such a cell now fails with FAIL: degenerate baseline: ... naming its budget-refusal count. It is an evaluation without a baseline, not evidence.
  • Out-of-sample. A model trained on the windows it is evaluated on shows nothing about generalization. Models now record the inclusive range of windows they trained on (model train, fleet train), and each gate cell reports the model digest, that training range, the range it evaluated and whether they overlap. An overlapping cell fails with FAIL: out-of-sample: in-sample evaluation ...; a model with no recorded training range (a file written before this field existed, or one whose identity is not recorded) fails as unprovable. It is never assumed out-of-sample.
  • Content identity. Reports bind the judged model by the first 16 hex of its file sha256, not by filename, and fleet gate refuses a report set whose cells carry no digest or whose digests disagree for a (model, period) cell. That catches a mixed set — a stale dist/gate/ model averaged with the current one — and nothing more. It does not establish that the file on disk now is the file that was evaluated: reports written before a model was replaced still agree with each other and still describe bytes the node no longer has. Treat the digest as agreement across reports, not as provenance for the file you hold.

Consequence for single-host promotion: replaying the live store, which contains the training history, is in-sample for a model trained on that store, so model promote now refuses it. A promotion requires training on history up to a cut and evaluating on later, disjoint windows. Cutting the training history earlier is not sufficient on its own: unless the evaluation is bounded too, the replay still scores windows inside the range the model records as its training range, and the gate fails closed. The missing piece is a bounded evaluation range: replay --policy-eval --eval-after NS|auto, restricting the evaluated windows to those starting at or after an absolute bound, with auto using the latest training-range end among the --eval-model files so every cell is out-of-sample by construction. The replay side is in place (ReplayOptions::eval_after_ns, eval_after_from_models, and a provenance eval_error naming the reason when no window survives the bound); the --eval-after CLI flag is landing in parallel, so unless strawberry replay --help lists it there is no bounded range and no passing single-host promotion path (API_CONTRACT.md records the shape). Until then a model promote run on live history fails with FAIL: out-of-sample: ... and writes no promoted.sbm. Before relying on a promoted model, the plan's fuller check also applies: the gate holds on 3 non-overlapping replay periods and for 3 training seeds. The tool reports the single run; the operator records the others (YAMS source=evidence).

Known limits

  • Payoffs assume an inspection of a positive window surfaces reach(action) of it. Until the model has acted, real inspection outcomes are not observed; the policy head stays untrained until they are.
  • With few real late labels, the gate leans on injected traces, which measure targeting of known attack shapes, not of novel ones.