strawberry/docs/evaluation/stage4-promotion-gate.md
trvon 085e8bdd2d
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Stage 4: pre-register threat-grounded traces (cutoff 2025-07-01, family split)
Written before any threat trace is collected or evaluated. Train on
families first reported before 2025-07-01, evaluate on families first
reported after it (new-family bucket primary), from public write-ups and
Atomic Red Team only: Triage and MalwareBazaar terms forbid ML training.
Adds gate criterion 5 (model recall >= rarity-greedy on new families).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 05:05:30 -06:00

2.7 KiB

Stage 4 inspection model: pre-registered promotion gate

Written 2026-10-27, before any promotion. The criteria below decide whether a shadow model may be promoted (strawberry model promote, which checks them). A changed method gets a new dated document; these numbers do not move after results are seen. (Also in YAMS as source=decision, task=strawberry-data-engine.)

What a promoted model may do

Request extra inspections only (memory scan, deeper rescan, closer watch of a window's descendants, rescan on the next rules update), within an hourly CPU budget (10/30/120 CPU-seconds by scan profile). It never skips, delays or replaces a rule-triggered scan, never blocks, contains or kills, and its output is never a finding by itself (Stage 5 decides that separately).

Measurement

strawberry replay --model <shadow.sbm> --policy-eval [--inject FILE] over the host's telemetry:

  • Candidates: every ranked window after the warm-up (default 12 h).
  • Positives ("extra"): windows that rules did not flag at exec time and that later got a late label (retro-hunt after a rules update, or an operator quarantine/kill), plus injected evaluation-trace windows (config/traces/standard.trace; never the traces a model was trained on). Amended 2026-10-28 by the threat-trace pre-registration: plus threats-eval.trace, and a criterion 5 on new threat families.
  • Policies, each with its own hourly budget and the same costs and reach (ShadowOptions): the model (its value through the shared chooser) and rarity-greedy (min(1, rarity score / 48) through the same chooser).
  • Efficiency = hits / inspection CPU-seconds. Recall = hits / positives.

Gate (all must hold)

  1. At least 10 extra positives in the evaluated history.
  2. Model efficiency >= 1.5 x rarity-greedy efficiency.
  3. Model recall >= rarity-greedy recall (efficiency is not bought by inspecting less of what matters).
  4. Model inspections stay within budget by construction; refusals are reported, not counted against it.

model promote checks 1-3 on the current history and refuses otherwise. Before relying on a promoted model, the plan's fuller check also applies: the gate holds on 3 non-overlapping replay periods and for 3 training seeds. The tool reports the single run; the operator records the others (YAMS source=evidence).

Known limits

  • Payoffs assume an inspection of a positive window surfaces reach(action) of it. Until the model has acted, real inspection outcomes are not observed; the policy head stays untrained until they are.
  • With few real late labels, the gate leans on injected traces, which measure targeting of known attack shapes, not of novel ones.