strawberry/deploy/salt/STATES.md
trvon 39ffa02ae6
Salt: dockerized multi-node fleet test (master/central + 3 sensors) + state fixes
deploy/salt/test/: builds a release tarball from HEAD in ubuntu:24.04 (CI path),
boots systemd containers (rootless, no --privileged), master==central + 3
sensors with the host's Salt onedir mounted read-only, applies the real states,
runs a workload with planted YARA detections, then two orchestrated cycles.
43 checks (install/config/idempotence, real daemons + telemetry, R2 cache
miss/hit, cp.push collect, 3-seed cycle, canary hold then fleet rollout, no
promoted model anywhere, fleet bits in round 2, --since cursor, dedup).

State fixes found by running them for real:
- init.sls no longer includes strawberry.distribute: a plain state.apply on
  sensors shipped models fleet-wide and bypassed the canary; only the
  orchestration distributes now.
- install.sls: CI tarballs carry a +g<sha> suffix that 'strawberry version'
  does not print, so install.sh re-ran on every apply; record the installed
  release and compare base version + marker.
- orch collect: mv -f (re-collected bundles must not be skipped).
- central cycle: builds the overlay first; a training refusal for too few
  teacher labels (normal on a young fleet) publishes the overlay only and
  keeps the sensors' shadow, any other error still fails.
- distribute.sls: serves only what the cycle published (cp.hash_file; the
  cp.list_master file-list cache can miss a just-published file).
Render check gains overlay-only and full publish cases (20 checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:16:31 -07:00

4.7 KiB

Strawberry Salt states

States consuming the pillar in pillar/strawberry/init.sls (see README.md for the architecture: hub-and-spoke, Salt-pull, no node-to-node transport).

deploy/salt/
  pillar/top.sls, pillar/strawberry/{init,central}.sls
  salt/strawberry/{init,install,config,export,central,distribute}.sls
  salt/orch/strawberry_cycle.sls
  files/strawberry-central-cycle.sh, strawberry-{export,central-cycle}.{service,timer}
scripts/test_salt_states.sh        # offline render check

Wiring (master)

# /etc/salt/master.d/strawberry.conf
file_recv: True                    # lets minions cp.push their bundles
file_roots:
  base:
    - /srv/salt                    # holds strawberry/dist/ (written by the cycle)
    - <repo>/deploy/salt/salt      # salt://strawberry/*, salt://orch/*
    - <repo>/deploy/salt           # salt://files/*
pillar_roots:
  base:
    - <repo>/deploy/salt/pillar

central.dist_dir (default /srv/salt/strawberry/dist) must be a directory inside a file_roots entry so it resolves as salt://strawberry/dist/. Pillar top matches by minion id ('*' = sensor defaults, 'strawberry-central' adds strawberry.central); pillar top cannot match on pillar. Set strawberry:release:version and artifact_sha256 (string, or a map per distro trixie/noble) in your own pillar.

States

  • install: downloads strawberry-<ver>-<distro>-<arch>.tgz (arch is Debian style, amd64) from the forgejo release with source_hash: sha256=<pin>, extracts to /opt/strawberry/<ver>, runs its scripts/install.sh only when strawberry version differs. Empty version/pin renders a failing state (fail closed). Nodes never build.
  • config: edits only the keys it owns in /etc/strawberry/strawberry.conf (capture_redact_cmdline=true, rule_auto_fetch=false, model_shadow, scan_profile, model_dir, db_path, optional node_id), restarts strawberry on change. Pillar capture_redact_cmdline: false fails the run.
  • export (sensor): export_dir, export_retention_hours (= retain_local_days x 24), export_max_mb; strawberry-export.timer runs the export as root, Nice=19, idle I/O, with --since auto when export.incremental is true (requires the cursor feature in the installed release).
  • central: 0700 bundle/work/dist dirs, /usr/local/sbin/strawberry-central-cycle.sh, nightly strawberry-central-cycle.timer.
  • distribute (sensor; NOT part of init, applied only by the orchestration so it stays canary-gated and never runs before the first cycle has published): shadow.sbm and fleet-overlay.sbo from salt://strawberry/dist/ into model_dir (0600 root), hashes from the .sha256 sidecars. Nothing else is placed in model_dir.

The cycle

salt-run state.orchestrate orch.strawberry_cycle (master must also be the central node):

  1. sensors: systemctl start strawberry-export.service
  2. sensors: salt-call cp.push each <export_dir>/export-*.sbx (lands under /var/cache/salt/master/minions/<id>/files/...); central moves them into bundle_dir. If central is a different host from the master, replace the move with an rsync or file.recurse of that cache tree to the central node.
  3. central: strawberry-central-cycle.sh trains seeds 1, 2, 3 (kept with a summary.json under work_dir/run-<stamp>/ as evidence), builds the overlay, publishes seed 1 as shadow.sbm plus fleet-overlay.sbo and sidecars to dist_dir.
  4. canary-first distribution: distribute.canary_minions get strawberry.distribute; the rest of the sensors only when distribute.canary_ok: true (the operator flips it after checking the canaries). Orchestration reads the master's pillar; override with pillar='{"strawberry":{"distribute":{"canary_ok":true}}}' on the command line.

Operator-only: promotion

No state, orchestration or script runs the model-gate step. After reviewing the three seeds in summary.json and the canaries, the operator runs it by hand on a node, with strawberry model promote and the pre-registered gate evidence (docs/evaluation/stage4-promotion-gate.md). Distribution only ever delivers a shadow model.

Security notes

  • No inbound listener on sensors: the master pulls via cp.push over the Salt channel.
  • Release binaries are pinned by sha256 (source_hash); a missing pin stops the run.
  • capture_redact_cmdline=true is enforced by config; bundles are redacted at export.
  • Model and overlay files are 0600 root; the cycle script publishes atomically.
  • Rules come from the release's pinned snapshot (rule_auto_fetch=false) on fleet nodes.

Check

scripts/test_salt_states.sh renders sensor, central and orchestration SLS offline (scratch config dir, no root) and asserts the invariants above.