strawberry/deploy/salt
trvon 80070a6497
fleet test: judge --since auto against a same-moment full export
Round-1 bundles are not a fair baseline: the 120 s cursor overlap is a large
share of a few-minute test history. Verified on the live fleet: incremental
192/206/194 windows vs full 230/244/232.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:16:32 -07:00
..
files Salt: dockerized multi-node fleet test (master/central + 3 sensors) + state fixes 2026-11-14 12:16:31 -07:00
pillar Fleet F1: merge dedup per node+window, overlay newest-per-node, export --since 2026-11-12 06:50:03 -07:00
salt Salt: dockerized multi-node fleet test (master/central + 3 sensors) + state fixes 2026-11-14 12:16:31 -07:00
test fleet test: judge --since auto against a same-moment full export 2026-11-14 12:16:32 -07:00
README.md salt: README diagram uses the fleet overlay CLI 2026-11-12 06:58:16 -07:00
STATES.md Salt: dockerized multi-node fleet test (master/central + 3 sensors) + state fixes 2026-11-14 12:16:31 -07:00

Strawberry distributed deployment (Salt) — contract for the deploying agent

This directory holds the Salt pillar (data contract) for deploying Strawberry's distributed learning. The SLS states (install/config/egress/distribute) are to be authored separately (by the deploying agent) and consume this pillar.

  • pillar/strawberry.sls — the full pillar schema with defaults + required fields.
  • pillar/top.sls — example role assignment (sensor vs central).

Architecture (what this release actually is)

Hub-and-spoke, NOT peer-to-peer.

  sensor nodes                         central node
  ------------                         ------------
  daemon -> store                      (also a sensor, optionally)
  strawberry export  --out bundle.sbx  --(Salt pull)--> bundle_dir/
                                       strawberry fleet train bundle_dir --out shadow.sbm
                                       strawberry fleet overlay bundle_dir --out fleet-overlay.sbo
  model_dir/ <--(Salt push)----------  distribute: shadow.sbm + fleet-overlay.sbo
  (round 2+ exports carry fleet_bits)
  • There is no node-to-node transport, no gossip/DHT, no decentralized aggregation.
  • Egress is central-pull (Salt orchestration / cp.push of a sealed, redacted bundle). The bundle artifact contract is stable, so an object store / mTLS collector can replace the pull later without daemon changes.
  • Promotion is operator-only, never automatic. States distribute shadow.sbm; a model only acts after strawberry model promote passes the pre-registered gate (docs/evaluation/stage4-promotion-gate.md). Keep promotion a human step.

CLI surface the states invoke (exists today on main)

  • strawberry export <store_dir> --out <file> — sensor egress. Auto-includes fleet rarity when <model_dir>/fleet-overlay.sbo is present.
  • strawberry fleet train <bundle_dir> --out <model.sbm> — central merge + train shadow model.

Current state and remaining gaps (honest status)

Closed on main:

  • strawberry fleet overlay <bundle_dir> --out fleet-overlay.sbo exists (sums per-node rarity sketches; the pillar's central.overlay_out can run).
  • node_id is a daemon config key: export stamps node_uid via read_node_uid(config.node_id).
  • Incremental export: strawberry export --export-dir DIR --since auto (or --since <unix_ns>). auto reads DIR/.export-cursor (one decimal int64 = the previous export's time_high_ns); a missing or unparsable cursor means a full export (with a note). After a successful export the cursor is rewritten atomically (tmp + rename, 0640). The cursor does not match the prune glob (export-*.sbx). The next export starts 120 s before the cursor so a window that was still open is re-shipped. Labeled windows always re-ship (the replay exposes no label timestamp, so a delayed label cannot be dated). Rarity records are always included (the sketch is cumulative).

Merge semantics on the central node (repeated exports pulled into ONE bundle dir are safe):

  • fleet train dedups per (node_uid, window_id) across bundles: first-seen order kept, label = positive in ANY bundle, features/start/rarity from the newest bundle (manifest created_unix_ns, tie: later path). Windows with the same id on different nodes are separate.
  • fleet overlay uses only the NEWEST bundle per node (its rarity sketch is cumulative, so summing every bundle would count a node k times). A bundle stamped fleet-overlay in the dir is skipped.

Remaining gap:

  1. No multi-node validation. The pipeline is unit/component + single-binary tested; it has not been run as a real multi-node fleet (N daemons -> central pull -> merge across hosts -> distribute -> re-export). A docker-swarm end-to-end is the recommended acceptance test before fleet rollout (the plan's deferred docker-live milestone).

Paths (binary defaults)

  • store / db_path: /var/lib/strawberry/evidence
  • model_dir: /var/lib/strawberry/model (holds shadow.sbm, promoted.sbm, rarity.state, fleet-overlay.sbo)

Security invariants the states must preserve

  • capture_redact_cmdline: true (argv redaction before storage) — do not disable on a fleet.
  • No new inbound listener on sensor nodes — export is an outbound read-only artifact pulled over the trusted channel; the daemon sandbox stays intact.
  • Reviewed, sha256-pinned binaries only; verify the pin before install.