# Stage 4 promotion gate over fleet data (procedure) Written 2026-11-15, before any fleet gate result was seen. This document adds a **procedure**; it does not change the pre-registered criteria in [stage4-promotion-gate.md](stage4-promotion-gate.md), which stay exactly as written: 1. at least 10 extra positives; 2. model efficiency (hits / inspection CPU-second) >= 1.5 x rarity-greedy; 3. model recall >= rarity-greedy recall; 4. inspections within budget by construction. That document already asks for the fuller check — the gate holding on **3 non-overlapping replay periods and for 3 training seeds** — and left it to the operator to record. This procedure produces those 9 results for a fleet, mechanically. What those 9 cells can and cannot show is stated under "What the three seeds do not test" below; they are not, by themselves, a generalization check. Status: as written, every cell this procedure produces is in-sample for the candidates it judges, so the gate fails it (`in_sample: true`, `FAIL: out-of-sample: ...`). The procedure supplies the evidence *structure*; a passing verdict additionally needs a bounded evaluation range (see "The 3-period axis does not fix this by itself" below). ## Procedure 1. **Candidates.** The central cycle trains three models from the pooled fleet bundles (seeds 1, 2, 3; `strawberry fleet train`) and publishes all three for evaluation (`dist/gate/`). Seed 1 is also the distributed shadow model; the gate judges all three. The three seeds differ only in initialization and in the per-node shuffle of the same windows: see the note below. 2. **Evaluation stays on the node.** Each sensor runs, over its own raw history, one replay pass: `strawberry replay --policy-eval --eval-model seed1 --eval-model seed2 --eval-model seed3 --periods 3 --inject --inject-copies 30 --json`. Raw telemetry never leaves the node. The ranked history (after the default 12 h warm-up) is split into 3 equal time periods; in each period every model and rarity-greedy are simulated with their own hourly inspection budget, exactly as the single-host evaluation does. 3. **Evaluation traces** are the pre-registered evaluation set (`standard.trace`, and `threats-eval.trace` where installed), never the training traces. Copies are spread evenly over each node's ranked history, so every period receives positives (none in the final 10 minutes, where a trace would not fit). 4. **Pooling.** The central node sums each (seed, period) cell over all node reports — windows, extra positives, inspections, CPU-seconds, hits, budget refusals — and applies the unchanged criteria 1–3 to each pooled cell (`strawberry fleet gate --json`). Every cell carries the judged model's content digest (first 16 hex of the model file's sha256), the training window range the model recorded, the range of windows the cell evaluated, and `in_sample` (whether those two ranges overlap). A report set whose cells carry no digest, or whose nodes disagree about a (model, period) cell's digest, is refused rather than pooled (`refused: true` plus `errors`), so a stale `dist/gate/` model left behind by an earlier cycle is refused rather than averaged into the current one. What that proves is narrow: the digest shows whether the reports in this set agree about which model bytes produced a cell. It cannot show that the file on disk now is the file that was evaluated — a model replaced after its reports were written leaves a report set that agrees with itself and still describes bytes the node no longer has. Identity here is agreement across reports, not a chain from the file you hold to the number you read. 5. **Verdict.** The gate passes only if it pooled the complete pre-registered matrix (`complete: true`; `expected_cells` is 3 seeds x 3 periods, and every node must have reported the same set), refused nothing (`refused: false`: every cell named a digest and the nodes agreed), and every pooled cell passed the criteria plus the integrity checks. An incomplete or mixed set yields no verdict instead of a verdict over the cells that happen to be present — which is what "all 9 cells must pass" means, and why a set missing a seed, a period or a node fails rather than quietly shrinking the evidence. In a binary older than the completeness check, read `passed: true` as "every cell that was present passed" and check `cells`, `reports` and per-cell `nodes` yourself. The verdict and every node report are kept as evidence (`central.work_dir/gate-/`). Promotion remains an operator action (`strawberry model promote` on the nodes, after reviewing the evidence). Neither the orchestration (`orch.strawberry_gate`) nor the cycle ever promotes. ## Why pooled, not per node Criterion 1 needs 10 positives; real late labels are rare on any single host. Pooling hits and CPU across nodes measures the policy the fleet would actually run, and keeps every node's evaluation against its own baseline. A node that cannot evaluate (no history, no gate models) contributes no report and is listed, not counted. ## What the three seeds do not test The three seeds are **initialization and shuffle variants over identical windows**, not three independent attempts. `fleet train --seed N` seeds the model initialization and the per-node example shuffle sampling; every seed sees the same fleet bundles, the same windows and the same labels, and every node evaluates them on the same ranked history. A seed that passes while another fails says the outcome is sensitive to initialization (unstable, so not promotable); three agreeing seeds say the initialization is irrelevant, so the three cells are near-duplicates rather than three pieces of evidence. Three seeds are therefore a **stability** check, not a generalization check. What would test generalization, and is not produced by this procedure: - evaluation windows that are **disjoint in time** from the training windows (train on history up to a cut, evaluate on the later period), rather than a re-scored copy of the training history; - models trained on **different periods** or different nodes and evaluated on held-out sensors; - any model whose recorded training range does not overlap the evaluated windows. The gate now enforces the last point mechanically. Each cell reports the model digest, the model's recorded training range and the range of the windows it evaluated; a cell whose ranges overlap fails with `FAIL: out-of-sample: in-sample evaluation ...`, and a model with no recorded training range fails as unprovable (it is never assumed out-of-sample). Training and evaluating on the same history can no longer produce a passing cell. The 3-period axis does not fix this by itself: the periods are disjoint from each other, but all of them lie inside the history the candidates were trained on, so every cell this procedure produces today is in-sample and the gate fails it (`FAIL: out-of-sample: in-sample evaluation ...`). Cutting the training bundles earlier — shipping only history up to a cut — is **not** enough either: the nodes still replay their whole store, and the windows after the cut stay inside the range the model records as its training range whenever the cut and the replay overlap. What is required is a **bounded evaluation range**: `replay --policy-eval --eval-after NS|auto`, restricting the evaluated windows to those starting at or after the bound (`auto` = the latest training-range end among the `--eval-model` files). The evaluation side is in the tree (`ReplayOptions::eval_after_ns`, `eval_after_from_models`, and a provenance `eval_error` when no window survives the bound); the `--eval-after` CLI flag is landing in parallel (`API_CONTRACT.md`). Only with a bound disjoint from the training range can a cell be out-of-sample, and only an out-of-sample cell can pass. ## Known limits - Periods are equal *time* slices of each node's ranked history; nodes with very different history lengths contribute periods of different absolute dates. All periods lie inside the training history, so the period axis does not make the evaluation out-of-sample, and cutting the training bundles earlier does not either (the nodes still replay the whole store): a bounded evaluation range (`--eval-after`, evaluation side in the tree, CLI flag landing) is required (see the section above). - Injected traces measure targeting of known attack shapes; with few real late labels the gate leans on them (as already noted for the single-host gate). - Synthetic validation (`scripts/fleet_e2e.sh`): 3 nodes, 9 pooled cells, ~48-52 positives per cell. The synthetic stores are trained on and then replayed over the same history, so every cell is in-sample: the gate correctly does not pass, and the harness asserts both the per-cell `in_sample` flag and that an in-sample evidence set never passes (it also fails if the verdict claims a pass).