The gate's own documentation claimed more than the tool did, in ways that mattered once the integrity checks landed. - "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it is now enforced (previous commit) and the text says which criterion the tool applies: complete evidence, refused false, every pooled cell passing. - The model digest proves agreement ACROSS reports. It cannot prove that the file on disk now is the one that was evaluated. Corrected everywhere. - The remedy was described inconsistently: cutting the training bundles earlier is NOT enough (evaluation still replays the whole store), a bounded evaluation range is required. Written consistently, with --eval-after NS|auto. - scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It now asserts >= 5, requires the provenance fields, and adds the honest-evidence invariant: today every cell is in-sample, so at least one cell must report in_sample true, none may pass, and each must carry the out-of-sample failure. A pass over in-sample cells is itself a failure. - The Salt/range runbooks advertised the flow as producing promotion evidence; it collects gate INPUT, and a passing verdict requires out-of-sample windows.
8.8 KiB
Stage 4 promotion gate over fleet data (procedure)
Written 2026-11-15, before any fleet gate result was seen. This document adds a procedure; it does not change the pre-registered criteria in stage4-promotion-gate.md, which stay exactly as written:
- at least 10 extra positives;
- model efficiency (hits / inspection CPU-second) >= 1.5 x rarity-greedy;
- model recall >= rarity-greedy recall;
- inspections within budget by construction.
That document already asks for the fuller check — the gate holding on 3 non-overlapping replay periods and for 3 training seeds — and left it to the operator to record. This procedure produces those 9 results for a fleet, mechanically. What those 9 cells can and cannot show is stated under "What the three seeds do not test" below; they are not, by themselves, a generalization check.
Status: as written, every cell this procedure produces is in-sample for the candidates it judges,
so the gate fails it (in_sample: true, FAIL: out-of-sample: ...). The procedure supplies the
evidence structure; a passing verdict additionally needs a bounded evaluation range (see "The
3-period axis does not fix this by itself" below).
Procedure
- Candidates. The central cycle trains three models from the pooled fleet bundles (seeds 1, 2,
3;
strawberry fleet train) and publishes all three for evaluation (dist/gate/). Seed 1 is also the distributed shadow model; the gate judges all three. The three seeds differ only in initialization and in the per-node shuffle of the same windows: see the note below. - Evaluation stays on the node. Each sensor runs, over its own raw history, one replay pass:
strawberry replay <store> --policy-eval --eval-model seed1 --eval-model seed2 --eval-model seed3 --periods 3 --inject <evaluation traces> --inject-copies 30 --json. Raw telemetry never leaves the node. The ranked history (after the default 12 h warm-up) is split into 3 equal time periods; in each period every model and rarity-greedy are simulated with their own hourly inspection budget, exactly as the single-host evaluation does. - Evaluation traces are the pre-registered evaluation set (
standard.trace, andthreats-eval.tracewhere installed), never the training traces. Copies are spread evenly over each node's ranked history, so every period receives positives (none in the final 10 minutes, where a trace would not fit). - Pooling. The central node sums each (seed, period) cell over all node reports — windows,
extra positives, inspections, CPU-seconds, hits, budget refusals — and applies the unchanged
criteria 1–3 to each pooled cell (
strawberry fleet gate <reports> --json). Every cell carries the judged model's content digest (first 16 hex of the model file's sha256), the training window range the model recorded, the range of windows the cell evaluated, andin_sample(whether those two ranges overlap). A report set whose cells carry no digest, or whose nodes disagree about a (model, period) cell's digest, is refused rather than pooled (refused: truepluserrors), so a staledist/gate/model left behind by an earlier cycle is refused rather than averaged into the current one. What that proves is narrow: the digest shows whether the reports in this set agree about which model bytes produced a cell. It cannot show that the file on disk now is the file that was evaluated — a model replaced after its reports were written leaves a report set that agrees with itself and still describes bytes the node no longer has. Identity here is agreement across reports, not a chain from the file you hold to the number you read. - Verdict. The gate passes only if it pooled the complete pre-registered matrix (
complete: true;expected_cellsis 3 seeds x 3 periods, and every node must have reported the same set), refused nothing (refused: false: every cell named a digest and the nodes agreed), and every pooled cell passed the criteria plus the integrity checks. An incomplete or mixed set yields no verdict instead of a verdict over the cells that happen to be present — which is what "all 9 cells must pass" means, and why a set missing a seed, a period or a node fails rather than quietly shrinking the evidence. In a binary older than the completeness check, readpassed: trueas "every cell that was present passed" and checkcells,reportsand per-cellnodesyourself. The verdict and every node report are kept as evidence (central.work_dir/gate-<stamp>/).
Promotion remains an operator action (strawberry model promote on the nodes, after reviewing the
evidence). Neither the orchestration (orch.strawberry_gate) nor the cycle ever promotes.
Why pooled, not per node
Criterion 1 needs 10 positives; real late labels are rare on any single host. Pooling hits and CPU across nodes measures the policy the fleet would actually run, and keeps every node's evaluation against its own baseline. A node that cannot evaluate (no history, no gate models) contributes no report and is listed, not counted.
What the three seeds do not test
The three seeds are initialization and shuffle variants over identical windows, not three
independent attempts. fleet train --seed N seeds the model initialization and the per-node example
shuffle sampling; every seed sees the same fleet bundles, the same windows and the same labels, and
every node evaluates them on the same ranked history. A seed that passes while another fails says the
outcome is sensitive to initialization (unstable, so not promotable); three agreeing seeds say the
initialization is irrelevant, so the three cells are near-duplicates rather than three pieces of
evidence. Three seeds are therefore a stability check, not a generalization check.
What would test generalization, and is not produced by this procedure:
- evaluation windows that are disjoint in time from the training windows (train on history up to a cut, evaluate on the later period), rather than a re-scored copy of the training history;
- models trained on different periods or different nodes and evaluated on held-out sensors;
- any model whose recorded training range does not overlap the evaluated windows.
The gate now enforces the last point mechanically. Each cell reports the model digest, the model's
recorded training range and the range of the windows it evaluated; a cell whose ranges overlap fails
with FAIL: out-of-sample: in-sample evaluation ..., and a model with no recorded training range
fails as unprovable (it is never assumed out-of-sample). Training and evaluating on the same history
can no longer produce a passing cell.
The 3-period axis does not fix this by itself: the periods are disjoint from each other, but all of
them lie inside the history the candidates were trained on, so every cell this procedure produces
today is in-sample and the gate fails it (FAIL: out-of-sample: in-sample evaluation ...). Cutting
the training bundles earlier — shipping only history up to a cut — is not enough either: the
nodes still replay their whole store, and the windows after the cut stay inside the range the model
records as its training range whenever the cut and the replay overlap. What is required is a
bounded evaluation range: replay --policy-eval --eval-after NS|auto, restricting the evaluated
windows to those starting at or after the bound (auto = the latest training-range end among the
--eval-model files). The evaluation side is in the tree (ReplayOptions::eval_after_ns,
eval_after_from_models, and a provenance eval_error when no window survives the bound); the
--eval-after CLI flag is landing in parallel (API_CONTRACT.md). Only with a bound disjoint from
the training range can a cell be out-of-sample, and only an out-of-sample cell can pass.
Known limits
- Periods are equal time slices of each node's ranked history; nodes with very different history
lengths contribute periods of different absolute dates. All periods lie inside the training
history, so the period axis does not make the evaluation out-of-sample, and cutting the training
bundles earlier does not either (the nodes still replay the whole store): a bounded evaluation
range (
--eval-after, evaluation side in the tree, CLI flag landing) is required (see the section above). - Injected traces measure targeting of known attack shapes; with few real late labels the gate leans on them (as already noted for the single-host gate).
- Synthetic validation (
scripts/fleet_e2e.sh): 3 nodes, 9 pooled cells, ~48-52 positives per cell. The synthetic stores are trained on and then replayed over the same history, so every cell is in-sample: the gate correctly does not pass, and the harness asserts both the per-cellin_sampleflag and that an in-sample evidence set never passes (it also fails if the verdict claims a pass).