strawberry/docs/evaluation/stage4-rarity-baseline-preregistration.md
trvon f4adc12a74
gate: judge the model against a calibrated rarity comparator
The rarity-greedy comparator acted on every evaluated window, so it spent
each hour's budget on the first ~15 windows and scored 0 hits in every
cell. Criteria 2 and 3 of the Stage 4 gate were void against it.

rarity_calibrated keeps the same rarity value, chooser, costs and budget,
and adds a per-node threshold frozen before the evaluated span: the m-th
largest value over the trailing 24 h of label-free history between the
warm-up and the --eval-after bound, with m set so expected spend matches
the hourly budget. Without that history the comparator is uncalibrated
and the gate fails by name; it never falls back to another rule.

rarity_greedy (the old first-come rule) and rarity_hindsight (per-hour
ordering, not causal) are reported as diagnostics and never judged.
fleet gate pools counts, keeps thresholds per node, and fails a pooled
cell when any node is uncalibrated or lacks the comparator.

Contract change: in `fleet gate --json`, rarity_hits and
rarity_cpu_seconds now describe the calibrated comparator; the old
numbers are rarity_greedy_hits and rarity_greedy_cpu_seconds.

Pre-registration: docs/evaluation/stage4-rarity-baseline-preregistration.md,
sha256 4916c95cd2581e3749acfe46c3ccf827aa5ce98c06150b49b978fc70da013f95,
indexed in YAMS at 2026-11-20T18:55:48Z, before the code. This commit is
dated after the registration time in that document (2026-11-21T00:00Z);
the YAMS record is what predates it.

Checked: meson suite 58/58, scripts/fleet_e2e.sh, clang-tidy and the
strict analyzer on the changed TUs. Not run: fold on the changed TUs,
make fuzz, the live Docker tests and the perf KPI. The comparator has
not been run on validation-range data.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-20 19:20:55 -07:00

7.6 KiB

Stage 4 gate: a calibrated rarity comparator (pre-registration)

Written 2026-11-20, before the comparator defined here was run on any history. It amends the comparator named in criteria 2 and 3 of stage4-promotion-gate.md and used by stage4-fleet-gate.md and stage4-execution-gate.md. The four criteria, their thresholds (10 extra positives, 1.5x efficiency, recall not lower), the positives, the budgets, the costs and the reach values are unchanged. The rule below does not move after results are seen; a changed rule gets a new dated document and a new registration time. (Also in YAMS as source=decision, task=strawberry-gate-baseline.)

Why the comparator changes

The original comparator, rarity-greedy, passes min(1, rarity bits / 48) through the shared chooser. The chooser acts when value x reach - lambda x cost > 0, which for the executable actions is a value above 0.083 (about 4 bits). Measured on the validation range's first out-of-sample evaluation (2026-11-18T20:46Z .. 2026-11-20T01:00Z, three sensors, 67,931 windows):

  • every evaluated window cleared that threshold (inspections + budget refusals = windows in every cell);
  • so the comparator spent each hour's budget on the first ~15 windows of the hour and refused the rest: it was first-come, not rarity-ordered;
  • it scored 0 hits in all 9 cells, which is the expected result of that rule at any number of positives, and no larger budget fixes it (720-970 windows per sensor-hour at 1-2 CPU-s each).

A comparator that cannot score is not a baseline: criteria 2 and 3 are void against it (the gate already fails such a cell as degenerate baseline). Rarity itself is informative on that history — the late-labelled windows sat at 43.7-45.7 bits, near the top of ~23k windows per sensor — so the defect is the decision rule, not the signal.

What was seen before this was written: the model results, the rarity-greedy results and the rarity scores of the positives on the span above. That span is therefore not evidence for the rule below (see "Fresh-data rule").

The comparator: rarity_calibrated

Same value as before (min(1, rarity bits / 48), per node, from the node's own rarity model), same chooser, same costs, reach, executable actions and hourly budget. One addition: a per-node threshold frozen before the evaluated span, chosen so that the comparator's expected spend matches its budget.

  1. Calibration history (label-free). The node's own non-injected windows that

    • start at or before the evaluation bound (--eval-after), so they are never evaluated, and
    • start after the replay warm-up (default 12 h from the first window; on a fresh history almost everything is new and rarity saturates), and
    • start within the trailing 24 h of such windows (the 24 h ending at the latest one).

    Only window start times and rarity values are used. No label, detection, model output or evaluated window enters the calibration. Note that with an explicit --eval-after bound this history can be later than the judged models' training range; "calibrated on the training range" means this label-free pre-bound history.

  2. Target rate. K = budget_seconds_per_hour / cost(a*), where a* is the action the shared chooser selects for a value of 1 under the executable policy (today memory_scan, 2 CPU-s, so K = 15 per hour on the balanced profile). T = the calibration span in hours = (latest start - earliest start + one window length) / 1 h. Target count m = floor(K x T), clamped to [1, number of calibration windows].

  3. Threshold. tau = the m-th largest calibration value (single precision, the precision the values are stored in).

  4. Decision. In the evaluated span the comparator considers a window only when its value is >= tau (ties are included); the action is the chooser's for that value; the hourly budget is enforced exactly as for the model, and a considered window that does not fit is a budget refusal.

  5. No calibration. If no window qualifies under (1) — no evaluation bound, or no history between the warm-up and the bound — the comparator is uncalibrated. An uncalibrated comparator fails the gate by name (FAIL: baseline uncalibrated). It never falls back to another rule.

The gate (promotion_gate, model promote, fleet gate) compares the model against rarity_calibrated in criteria 2 and 3 and applies the existing degenerate-baseline check to it. A report without rarity_calibrated (written by an older replay) fails by name; it is never judged against rarity_greedy instead.

Fleet pooling. Thresholds are per node (each node calibrates on its own history; raw telemetry still never leaves the node). Counts pool as before. One node whose comparator is uncalibrated or absent makes the pooled cell uncalibrated, matching the existing rule that pooling may only weaken evidence.

Diagnostics (reported, never judged)

  • rarity_greedy: the original first-come rule, unchanged, kept so old and new reports can be read side by side.
  • rarity_hindsight: within each clock hour of the evaluated span (hours counted from the first evaluated window), windows are taken in descending rarity value (earlier start first on ties) through the same chooser until the hour's budget is spent. It uses the whole hour before deciding, so it is not causal and no deployable policy can match it. It bounds what rarity ordering could achieve at this budget. It is never the gate comparator.

Fresh-data rule (validation range)

  • Registration time T_reg = 2026-11-21T00:00:00Z = 1795219200000000000 ns.
  • Evidence for the gate on the validation range comes only from windows starting after T_reg: strawberry replay <store> --policy-eval --eval-model ... --periods 3 --eval-after 1795219200000000000 --json on each sensor, then strawberry fleet gate.
  • An explicit bound replaces auto. The judged models are whatever --eval-model names; their training ranges must end at or before the evaluated windows, which the existing out-of-sample check enforces per cell (a model trained past the bound fails as in-sample). The three seeds trained on the 2026-11-18 cut (sha256 969edd72f0a8d3a9, 0289aa94d28e718f, 2720329e9b868fb4) qualify; so does a retrain on any cut ending at or before T_reg.
  • A run of rarity_calibrated on windows before T_reg is a diagnostic and is labelled as one.
  • The rule is scoped to this range because this range's earlier span was seen. On any other history the usual --eval-after auto applies.

Known limits

  • At a budget-matched threshold the comparator spends about its whole budget every hour. A model that inspects only a few windows will then clear criterion 2 (efficiency) easily, so criterion 3 (recall not lower than the comparator's) carries most of the comparison. That is a property of the pre-registered criteria, not something this amendment tunes.
  • The calibration is a single frozen threshold. If a node's activity shifts after the bound, the comparator over- or under-spends; over-spend is absorbed by budget refusals (and reported), an under-spend lowers its recall. Neither is corrected after the fact.
  • The range's positives are marker binaries with distinctive paths. They are easy for any ranker; passing this gate there shows the model is not worse than rarity at this budget, not that it finds novel attack shapes (criterion 5 in the threat-trace pre-registration covers those).
  • Operator activity on a sensor (including the evaluation command itself) appears as new, high-rarity windows in the evaluated history and draws comparator budget.