strawberry/deploy/validation
trvon a4179c474d
docs: say what the gate actually enforces, and what a passing verdict needs
The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.

- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
  is now enforced (previous commit) and the text says which criterion the tool
  applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
  file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
  is NOT enough (evaluation still replays the whole store), a bounded evaluation
  range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
  now asserts >= 5, requires the provenance fields, and adds the honest-evidence
  invariant: today every cell is in-sample, so at least one cell must report
  in_sample true, none may pass, and each must carry the out-of-sample failure.
  A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
  it collects gate INPUT, and a passing verdict requires out-of-sample windows.
2026-11-19 17:30:35 -07:00
..
range docs: say what the gate actually enforces, and what a passing verdict needs 2026-11-19 17:30:35 -07:00
README.md Stage 6: local multi-node fleet-loop validation (gap #4) 2026-11-08 09:32:32 -07:00
run-fleet-loop.sh Stage 6: local multi-node fleet-loop validation (gap #4) 2026-11-08 09:32:32 -07:00

Strawberry distributed-learning validation

run-fleet-loop.sh validates the full hub-and-spoke loop end to end with the real binaries, isolated per-node dirs/configs, and file-based transport — no mocks.

What it proves

seed N nodes (store + rarity.state)
  -> round-1 export         : bundles carry rarity sketches, NO fleet feature
  -> central fleet train    : merges N distinct nodes, trains the shadow model
  -> central fleet overlay  : sums per-node rarity sketches -> non-empty overlay
  -> distribute overlay
  -> round-2 export          : feature vectors now carry the fleet-wide rarity

Hard assertions (exit non-zero on any failure):

  • each round-1 bundle has rarity sketches (rarity>0) and fleet_feat=0;
  • central sees exactly N distinct nodes (no node_uid collision);
  • the overlay is non-empty (sketches merged);
  • the round-2 export carries the fleet feature (fleet_feat>0).

A passing run prints: PASS: fleet loop validated (round-1 fleet_feat=0 -> round-2 fleet_feat=<N> ...).

Run it (local, no Docker)

meson compile -C builddir strawberry strawberry-fleet-seed strawberry-fleet-inspect
bash deploy/validation/run-fleet-loop.sh          # NODES=2 EXECS=240 by default

Override binaries/params via env: SB, SEED, INS, NODES, EXECS, WORK.

Tools (build-only, not installed)

  • strawberry-fleet-seed <store> --node <n> --execs <N> [--seed S] [--model-dir <d>] — writes a deterministic synthetic telemetry store (and, with --model-dir, a rarity.state, simulating a node that has been running the daemon).
  • strawberry-fleet-inspect <bundle> — prints node_uid windows features labels rarity fleet_feat for an export bundle (fleet_feat = feature vectors carrying a non-zero fleet slot).

Scope / honesty

This validates the distributed DATA + LEARNING path (export → merge → train → overlay → distribute → re-export) across isolated nodes. It uses a seeder instead of the live eBPF daemon, so it does NOT re-validate live telemetry capture (the single-node daemon path covers that). The Docker-swarm harness (when built) runs this same script body inside containers for host-isolation realism; the logic proven here is identical.

Integration facts this surfaced (true of real deployments)

  • Nodes must export with a distinct node_id (config) or they collide to one node_uid on a shared identity; in containers, distinct machine-ids also suffice.
  • The fleet overlay only forms once nodes have a rarity.state (accumulated by the running daemon) — a cold node's first export carries no rarity sketches, so fleet_feat stays 0 until a later round.
  • fleet train's max_node_share cap is for multi-node fleets; a single node is cap-degenerate.