strawberry/deploy/salt
trvon a4179c474d
docs: say what the gate actually enforces, and what a passing verdict needs
The gate's own documentation claimed more than the tool did, in ways that
mattered once the integrity checks landed.

- "passes only if all 9 cells pass" was enforced by scripts, not by the gate; it
  is now enforced (previous commit) and the text says which criterion the tool
  applies: complete evidence, refused false, every pooled cell passing.
- The model digest proves agreement ACROSS reports. It cannot prove that the
  file on disk now is the one that was evaluated. Corrected everywhere.
- The remedy was described inconsistently: cutting the training bundles earlier
  is NOT enough (evaluation still replays the whole store), a bounded evaluation
  range is required. Written consistently, with --eval-after NS|auto.
- scripts/fleet_e2e.sh asserted three gate checks; promotion_gate emits five. It
  now asserts >= 5, requires the provenance fields, and adds the honest-evidence
  invariant: today every cell is in-sample, so at least one cell must report
  in_sample true, none may pass, and each must carry the out-of-sample failure.
  A pass over in-sample cells is itself a failure.
- The Salt/range runbooks advertised the flow as producing promotion evidence;
  it collects gate INPUT, and a passing verdict requires out-of-sample windows.
2026-11-19 17:30:35 -07:00
..
files Stage 4 over fleet data: per-node 3x3 policy evaluation, pooled fleet promotion gate 2026-11-15 09:18:04 -07:00
pillar Stage 4 over fleet data: per-node 3x3 policy evaluation, pooled fleet promotion gate 2026-11-15 09:18:04 -07:00
salt gate: a bounded evaluation range, so out-of-sample evidence is possible 2026-11-19 17:27:59 -07:00
test threat traces checker + fleet gate in the dockerized Salt test 2026-11-15 09:36:24 -07:00
README.md salt: README diagram uses the fleet overlay CLI 2026-11-12 06:58:16 -07:00
STATES.md docs: say what the gate actually enforces, and what a passing verdict needs 2026-11-19 17:30:35 -07:00

Strawberry distributed deployment (Salt) — contract for the deploying agent

This directory holds the Salt pillar (data contract) for deploying Strawberry's distributed learning. The SLS states (install/config/egress/distribute) are to be authored separately (by the deploying agent) and consume this pillar.

  • pillar/strawberry.sls — the full pillar schema with defaults + required fields.
  • pillar/top.sls — example role assignment (sensor vs central).

Architecture (what this release actually is)

Hub-and-spoke, NOT peer-to-peer.

  sensor nodes                         central node
  ------------                         ------------
  daemon -> store                      (also a sensor, optionally)
  strawberry export  --out bundle.sbx  --(Salt pull)--> bundle_dir/
                                       strawberry fleet train bundle_dir --out shadow.sbm
                                       strawberry fleet overlay bundle_dir --out fleet-overlay.sbo
  model_dir/ <--(Salt push)----------  distribute: shadow.sbm + fleet-overlay.sbo
  (round 2+ exports carry fleet_bits)
  • There is no node-to-node transport, no gossip/DHT, no decentralized aggregation.
  • Egress is central-pull (Salt orchestration / cp.push of a sealed, redacted bundle). The bundle artifact contract is stable, so an object store / mTLS collector can replace the pull later without daemon changes.
  • Promotion is operator-only, never automatic. States distribute shadow.sbm; a model only acts after strawberry model promote passes the pre-registered gate (docs/evaluation/stage4-promotion-gate.md). Keep promotion a human step.

CLI surface the states invoke (exists today on main)

  • strawberry export <store_dir> --out <file> — sensor egress. Auto-includes fleet rarity when <model_dir>/fleet-overlay.sbo is present.
  • strawberry fleet train <bundle_dir> --out <model.sbm> — central merge + train shadow model.

Current state and remaining gaps (honest status)

Closed on main:

  • strawberry fleet overlay <bundle_dir> --out fleet-overlay.sbo exists (sums per-node rarity sketches; the pillar's central.overlay_out can run).
  • node_id is a daemon config key: export stamps node_uid via read_node_uid(config.node_id).
  • Incremental export: strawberry export --export-dir DIR --since auto (or --since <unix_ns>). auto reads DIR/.export-cursor (one decimal int64 = the previous export's time_high_ns); a missing or unparsable cursor means a full export (with a note). After a successful export the cursor is rewritten atomically (tmp + rename, 0640). The cursor does not match the prune glob (export-*.sbx). The next export starts 120 s before the cursor so a window that was still open is re-shipped. Labeled windows always re-ship (the replay exposes no label timestamp, so a delayed label cannot be dated). Rarity records are always included (the sketch is cumulative).

Merge semantics on the central node (repeated exports pulled into ONE bundle dir are safe):

  • fleet train dedups per (node_uid, window_id) across bundles: first-seen order kept, label = positive in ANY bundle, features/start/rarity from the newest bundle (manifest created_unix_ns, tie: later path). Windows with the same id on different nodes are separate.
  • fleet overlay uses only the NEWEST bundle per node (its rarity sketch is cumulative, so summing every bundle would count a node k times). A bundle stamped fleet-overlay in the dir is skipped.

Remaining gap:

  1. No multi-node validation. The pipeline is unit/component + single-binary tested; it has not been run as a real multi-node fleet (N daemons -> central pull -> merge across hosts -> distribute -> re-export). A docker-swarm end-to-end is the recommended acceptance test before fleet rollout (the plan's deferred docker-live milestone).

Paths (binary defaults)

  • store / db_path: /var/lib/strawberry/evidence
  • model_dir: /var/lib/strawberry/model (holds shadow.sbm, promoted.sbm, rarity.state, fleet-overlay.sbo)

Security invariants the states must preserve

  • capture_redact_cmdline: true (argv redaction before storage) — do not disable on a fleet.
  • No new inbound listener on sensor nodes — export is an outbound read-only artifact pulled over the trusted channel; the daemon sandbox stays intact.
  • Reviewed, sha256-pinned binaries only; verify the pin before install.