The rarity-greedy comparator acted on every evaluated window, so it spent each hour's budget on the first ~15 windows and scored 0 hits in every cell. Criteria 2 and 3 of the Stage 4 gate were void against it. rarity_calibrated keeps the same rarity value, chooser, costs and budget, and adds a per-node threshold frozen before the evaluated span: the m-th largest value over the trailing 24 h of label-free history between the warm-up and the --eval-after bound, with m set so expected spend matches the hourly budget. Without that history the comparator is uncalibrated and the gate fails by name; it never falls back to another rule. rarity_greedy (the old first-come rule) and rarity_hindsight (per-hour ordering, not causal) are reported as diagnostics and never judged. fleet gate pools counts, keeps thresholds per node, and fails a pooled cell when any node is uncalibrated or lacks the comparator. Contract change: in `fleet gate --json`, rarity_hits and rarity_cpu_seconds now describe the calibrated comparator; the old numbers are rarity_greedy_hits and rarity_greedy_cpu_seconds. Pre-registration: docs/evaluation/stage4-rarity-baseline-preregistration.md, sha256 4916c95cd2581e3749acfe46c3ccf827aa5ce98c06150b49b978fc70da013f95, indexed in YAMS at 2026-11-20T18:55:48Z, before the code. This commit is dated after the registration time in that document (2026-11-21T00:00Z); the YAMS record is what predates it. Checked: meson suite 58/58, scripts/fleet_e2e.sh, clang-tidy and the strict analyzer on the changed TUs. Not run: fold on the changed TUs, make fuzz, the live Docker tests and the perf KPI. The comparator has not been run on validation-range data. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
9.4 KiB
Stage 4 promotion gate over fleet data (procedure)
Written 2026-11-15, before any fleet gate result was seen. This document adds a procedure; it does not change the pre-registered criteria in stage4-promotion-gate.md, which stay exactly as written:
- at least 10 extra positives;
- model efficiency (hits / inspection CPU-second) >= 1.5 x rarity-greedy;
- model recall >= rarity-greedy recall;
- inspections within budget by construction.
Amended 2026-11-20 by
stage4-rarity-baseline-preregistration.md: the
comparator in criteria 2 and 3 is rarity_calibrated, calibrated per node on that node's own
label-free history before the evaluation bound; wherever this document says rarity-greedy as the
thing the model is compared against, read that comparator. One node without a calibrated comparator
makes the pooled cell uncalibrated, and it fails. On the validation range, evidence comes only from
windows after the registration time given there (--eval-after 1795219200000000000).
That document already asks for the fuller check — the gate holding on 3 non-overlapping replay periods and for 3 training seeds — and left it to the operator to record. This procedure produces those 9 results for a fleet, mechanically. What those 9 cells can and cannot show is stated under "What the three seeds do not test" below; they are not, by themselves, a generalization check.
Status: as written, every cell this procedure produces is in-sample for the candidates it judges,
so the gate fails it (in_sample: true, FAIL: out-of-sample: ...). The procedure supplies the
evidence structure; a passing verdict additionally needs a bounded evaluation range (see "The
3-period axis does not fix this by itself" below).
Procedure
- Candidates. The central cycle trains three models from the pooled fleet bundles (seeds 1, 2,
3;
strawberry fleet train) and publishes all three for evaluation (dist/gate/). Seed 1 is also the distributed shadow model; the gate judges all three. The three seeds differ only in initialization and in the per-node shuffle of the same windows: see the note below. - Evaluation stays on the node. Each sensor runs, over its own raw history, one replay pass:
strawberry replay <store> --policy-eval --eval-model seed1 --eval-model seed2 --eval-model seed3 --periods 3 --inject <evaluation traces> --inject-copies 30 --json. Raw telemetry never leaves the node. The ranked history (after the default 12 h warm-up) is split into 3 equal time periods; in each period every model and rarity-greedy are simulated with their own hourly inspection budget, exactly as the single-host evaluation does. - Evaluation traces are the pre-registered evaluation set (
standard.trace, andthreats-eval.tracewhere installed), never the training traces. Copies are spread evenly over each node's ranked history, so every period receives positives (none in the final 10 minutes, where a trace would not fit). - Pooling. The central node sums each (seed, period) cell over all node reports — windows,
extra positives, inspections, CPU-seconds, hits, budget refusals — and applies the unchanged
criteria 1–3 to each pooled cell (
strawberry fleet gate <reports> --json). Every cell carries the judged model's content digest (first 16 hex of the model file's sha256), the training window range the model recorded, the range of windows the cell evaluated, andin_sample(whether those two ranges overlap). A report set whose cells carry no digest, or whose nodes disagree about a (model, period) cell's digest, is refused rather than pooled (refused: truepluserrors), so a staledist/gate/model left behind by an earlier cycle is refused rather than averaged into the current one. What that proves is narrow: the digest shows whether the reports in this set agree about which model bytes produced a cell. It cannot show that the file on disk now is the file that was evaluated — a model replaced after its reports were written leaves a report set that agrees with itself and still describes bytes the node no longer has. Identity here is agreement across reports, not a chain from the file you hold to the number you read. - Verdict. The gate passes only if it pooled the complete pre-registered matrix (
complete: true;expected_cellsis 3 seeds x 3 periods, and every node must have reported the same set), refused nothing (refused: false: every cell named a digest and the nodes agreed), and every pooled cell passed the criteria plus the integrity checks. An incomplete or mixed set yields no verdict instead of a verdict over the cells that happen to be present — which is what "all 9 cells must pass" means, and why a set missing a seed, a period or a node fails rather than quietly shrinking the evidence. In a binary older than the completeness check, readpassed: trueas "every cell that was present passed" and checkcells,reportsand per-cellnodesyourself. The verdict and every node report are kept as evidence (central.work_dir/gate-<stamp>/).
Promotion remains an operator action (strawberry model promote on the nodes, after reviewing the
evidence). Neither the orchestration (orch.strawberry_gate) nor the cycle ever promotes.
Why pooled, not per node
Criterion 1 needs 10 positives; real late labels are rare on any single host. Pooling hits and CPU across nodes measures the policy the fleet would actually run, and keeps every node's evaluation against its own baseline. A node that cannot evaluate (no history, no gate models) contributes no report and is listed, not counted.
What the three seeds do not test
The three seeds are initialization and shuffle variants over identical windows, not three
independent attempts. fleet train --seed N seeds the model initialization and the per-node example
shuffle sampling; every seed sees the same fleet bundles, the same windows and the same labels, and
every node evaluates them on the same ranked history. A seed that passes while another fails says the
outcome is sensitive to initialization (unstable, so not promotable); three agreeing seeds say the
initialization is irrelevant, so the three cells are near-duplicates rather than three pieces of
evidence. Three seeds are therefore a stability check, not a generalization check.
What would test generalization, and is not produced by this procedure:
- evaluation windows that are disjoint in time from the training windows (train on history up to a cut, evaluate on the later period), rather than a re-scored copy of the training history;
- models trained on different periods or different nodes and evaluated on held-out sensors;
- any model whose recorded training range does not overlap the evaluated windows.
The gate now enforces the last point mechanically. Each cell reports the model digest, the model's
recorded training range and the range of the windows it evaluated; a cell whose ranges overlap fails
with FAIL: out-of-sample: in-sample evaluation ..., and a model with no recorded training range
fails as unprovable (it is never assumed out-of-sample). Training and evaluating on the same history
can no longer produce a passing cell.
The 3-period axis does not fix this by itself: the periods are disjoint from each other, but all of
them lie inside the history the candidates were trained on, so every cell this procedure produces
today is in-sample and the gate fails it (FAIL: out-of-sample: in-sample evaluation ...). Cutting
the training bundles earlier — shipping only history up to a cut — is not enough either: the
nodes still replay their whole store, and the windows after the cut stay inside the range the model
records as its training range whenever the cut and the replay overlap. What is required is a
bounded evaluation range: replay --policy-eval --eval-after NS|auto, restricting the evaluated
windows to those starting at or after the bound (auto = the latest training-range end among the
--eval-model files). The evaluation side is in the tree (ReplayOptions::eval_after_ns,
eval_after_from_models, and a provenance eval_error when no window survives the bound); the
--eval-after CLI flag is landing in parallel (API_CONTRACT.md). Only with a bound disjoint from
the training range can a cell be out-of-sample, and only an out-of-sample cell can pass.
Known limits
- Periods are equal time slices of each node's ranked history; nodes with very different history
lengths contribute periods of different absolute dates. All periods lie inside the training
history, so the period axis does not make the evaluation out-of-sample, and cutting the training
bundles earlier does not either (the nodes still replay the whole store): a bounded evaluation
range (
--eval-after, evaluation side in the tree, CLI flag landing) is required (see the section above). - Injected traces measure targeting of known attack shapes; with few real late labels the gate leans on them (as already noted for the single-host gate).
- Synthetic validation (
scripts/fleet_e2e.sh): 3 nodes, 9 pooled cells, ~48-52 positives per cell. The synthetic stores are trained on and then replayed over the same history, so every cell is in-sample: the gate correctly does not pass, and the harness asserts both the per-cellin_sampleflag and that an in-sample evidence set never passes (it also fails if the verdict claims a pass).