Commit graph

4 commits

Author SHA1 Message Date
trvon
a4179c474d
docs: say what the gate actually enforces, and what a passing verdict needs
The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.

- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
  is now enforced (previous commit) and the text says which criterion the tool
  applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
  file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
  is NOT enough (evaluation still replays the whole store), a bounded evaluation
  range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
  now asserts >= 5, requires the provenance fields, and adds the honest-evidence
  invariant: today every cell is in-sample, so at least one cell must report
  in_sample true, none may pass, and each must carry the out-of-sample failure.
  A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
  it collects gate INPUT, and a passing verdict requires out-of-sample windows.
2026-11-19 17:30:35 -07:00
trvon
6f5c62b31c
Stage 4 over fleet data: per-node 3x3 policy evaluation, pooled fleet promotion gate
The pre-registered gate (docs/evaluation/stage4-promotion-gate.md) asks for the
criteria to hold on 3 non-overlapping periods x 3 training seeds; until now the
tool reported a single run and the operator recorded the rest. This produces
the 9 results for a fleet, with the criteria unchanged:

- replay --policy-eval --eval-model FILE (repeatable) --periods N: one pass
  evaluates every model on every equal time slice of the ranked history; each
  (model, period) cell simulates the model and rarity-greedy with their own
  hourly budgets exactly as policy_eval does (policy_eval.cells). With one
  model equal to --model and one period the cell equals the single-model
  evaluation (test R11b); periods partition windows and positives.
- strawberry fleet gate DIR: pools the cells of every node report (windows,
  positives, inspections, CPU-s, hits, refusals) per (model, period) and
  applies learn::promotion_gate unchanged to each; passes only if all pass.
  Malformed / cell-less reports are skipped and listed.
- Evaluation stays on the node (raw telemetry never leaves it): the central
  cycle now also publishes all three seed models to dist/gate/; strawberry.gate
  (sensors) evaluates them with the pre-registered evaluation traces; the
  operator-run orch.strawberry_gate pushes the reports back and pools them;
  evidence under central.work_dir/gate-<stamp>/. Nothing promotes.
- docs/evaluation/stage4-fleet-gate.md: the procedure (new dated document;
  criteria and numbers unchanged).

Validated: test_fleet_gate (pooling sums, pooling reaching 10 positives
neither node has alone, one failing cell fails the fleet, malformed reports
skipped), R11b, Salt render checks (no models -> refusal; 3 models x 3
periods; orchestration pools), and scripts/fleet_e2e.sh end to end: 3 nodes ->
9 pooled cells with 28-50 positives each; the untrained synthetic models
correctly do not pass (exit code matches the verdict).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:18:04 -07:00
trvon
39ffa02ae6
Salt: dockerized multi-node fleet test (master/central + 3 sensors) + state fixes
deploy/salt/test/: builds a release tarball from HEAD in ubuntu:24.04 (CI path),
boots systemd containers (rootless, no --privileged), master==central + 3
sensors with the host's Salt onedir mounted read-only, applies the real states,
runs a workload with planted YARA detections, then two orchestrated cycles.
43 checks (install/config/idempotence, real daemons + telemetry, R2 cache
miss/hit, cp.push collect, 3-seed cycle, canary hold then fleet rollout, no
promoted model anywhere, fleet bits in round 2, --since cursor, dedup).

State fixes found by running them for real:
- init.sls no longer includes strawberry.distribute: a plain state.apply on
  sensors shipped models fleet-wide and bypassed the canary; only the
  orchestration distributes now.
- install.sls: CI tarballs carry a +g<sha> suffix that 'strawberry version'
  does not print, so install.sh re-ran on every apply; record the installed
  release and compare base version + marker.
- orch collect: mv -f (re-collected bundles must not be skipped).
- central cycle: builds the overlay first; a training refusal for too few
  teacher labels (normal on a young fleet) publishes the overlay only and
  keeps the sensors' shadow, any other error still fails.
- distribute.sls: serves only what the cycle published (cp.hash_file; the
  cp.list_master file-list cache can miss a just-published file).
Render check gains overlay-only and full publish cases (20 checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:16:31 -07:00
trvon
25d0da1121
Salt: install/config/export/central/distribute states + central cycle orchestration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:43:42 -07:00