The Security Hub server runs Debian forky; build-release.sh accepts forky and the release matrix gains a debian:forky leg (~1 h more per release, legs run one at a time). Testing moves: rebuild after a library transition.
6.5 KiB
Strawberry Salt states
States consuming the pillar in pillar/strawberry/init.sls (see README.md for the
architecture: hub-and-spoke, Salt-pull, no node-to-node transport).
deploy/salt/
pillar/top.sls, pillar/strawberry/{init,central}.sls
salt/strawberry/{init,install,config,export,central,distribute}.sls
salt/orch/strawberry_cycle.sls
files/strawberry-central-cycle.sh, strawberry-{export,central-cycle}.{service,timer}
scripts/test_salt_states.sh # offline render check
Wiring (master)
# /etc/salt/master.d/strawberry.conf
file_recv: True # lets minions cp.push their bundles
file_roots:
base:
- /srv/salt # holds strawberry/dist/ (written by the cycle)
- <repo>/deploy/salt/salt # salt://strawberry/*, salt://orch/*
- <repo>/deploy/salt # salt://files/*
pillar_roots:
base:
- <repo>/deploy/salt/pillar
central.dist_dir (default /srv/salt/strawberry/dist) must be a directory inside a
file_roots entry so it resolves as salt://strawberry/dist/. Pillar top matches by minion
id ('*' = sensor defaults, 'strawberry-central' adds strawberry.central); pillar top
cannot match on pillar. Set strawberry:release:version and artifact_sha256 (string, or a
map per distro trixie/noble/forky) in your own pillar.
States
install: downloadsstrawberry-<ver>-<distro>-<arch>.tgz(arch is Debian style,amd64) from the forgejo release withsource_hash: sha256=<pin>, extracts to/opt/strawberry/<ver>, runs itsscripts/install.shonly whenstrawberry versiondiffers. Empty version/pin renders a failing state (fail closed). Nodes never build.config: edits only the keys it owns in/etc/strawberry/strawberry.conf(capture_redact_cmdline=true,rule_auto_fetch=false,model_shadow,scan_profile,model_dir,db_path, optionalnode_id), restartsstrawberryon change. Pillarcapture_redact_cmdline: falsefails the run.export(sensor):export_dir,export_retention_hours(=retain_local_daysx 24),export_max_mb;strawberry-export.timerruns the export as root,Nice=19, idle I/O, with--since autowhenexport.incrementalis true (requires the cursor feature in the installed release).central: 0700 bundle/work/dist dirs,/usr/local/sbin/strawberry-central-cycle.sh, nightlystrawberry-central-cycle.timer.distribute(sensor; NOT part ofinit, applied only by the orchestration so it stays canary-gated and never runs before the first cycle has published):shadow.sbmandfleet-overlay.sbofromsalt://strawberry/dist/intomodel_dir(0600 root), hashes from the.sha256sidecars. Nothing else is placed inmodel_dir.
The cycle
salt-run state.orchestrate orch.strawberry_cycle (master must also be the central node):
- sensors:
systemctl start strawberry-export.service - sensors:
salt-call cp.pusheach<export_dir>/export-*.sbx(lands under/var/cache/salt/master/minions/<id>/files/...); central moves them intobundle_dir. If central is a different host from the master, replace the move with an rsync orfile.recurseof that cache tree to the central node. - central:
strawberry-central-cycle.shtrains seeds 1, 2, 3 (kept with asummary.jsonunderwork_dir/run-<stamp>/as evidence), builds the overlay, publishes seed 1 asshadow.sbmplusfleet-overlay.sboand sidecars todist_dir. - canary-first distribution:
distribute.canary_minionsgetstrawberry.distribute; the rest of the sensors only whendistribute.canary_ok: true(the operator flips it after checking the canaries). Orchestration reads the master's pillar; override withpillar='{"strawberry":{"distribute":{"canary_ok":true}}}'on the command line.
Operator-only: promotion
No state, orchestration or script runs the model-gate step. After reviewing the three seeds
in summary.json and the canaries, the operator runs it by hand on a node, with
strawberry model promote and the pre-registered gate evidence
(docs/evaluation/stage4-promotion-gate.md). Distribution only ever delivers a shadow model.
Security notes
- No inbound listener on sensors: the master pulls via
cp.pushover the Salt channel. - Release binaries are pinned by sha256 (
source_hash); a missing pin stops the run. capture_redact_cmdline=trueis enforced byconfig; bundles are redacted at export.- Model and overlay files are 0600 root; the cycle script publishes atomically.
- Rules come from the release's pinned snapshot (
rule_auto_fetch=false) on fleet nodes.
Check
scripts/test_salt_states.sh renders sensor, central and orchestration SLS offline
(scratch config dir, no root) and asserts the invariants above.
Fleet promotion gate (orch.strawberry_gate, operator-run)
After a cycle has trained models (enough teacher labels), the cycle also publishes the three seed
models to salt://strawberry/dist/gate/. To collect gate evidence (a passing verdict additionally
requires out-of-sample windows - see step 3):
salt-run state.orchestrate orch.strawberry_gate
strawberry.gateon every sensor: fetches the three gate models (sha256 from the sidecars) and runs one replay pass over the node's own history: 3 models x 3 periods, with the pre-registered evaluation traces (pillar strawberry:gate). Raw telemetry stays on the node.- The reports are
cp.pushed to the master (== central) and pooled bystrawberry fleet gate. - Evidence lands in
central.work_dir/gate-<UTC stamp>/(reports/<minion>.json,verdict.json).
A failing gate is a recorded verdict, not an orchestration failure. The gate also refuses
in-sample evidence: every cell publishes in_sample, and a cell whose evaluated windows overlap the
model's recorded training range fails with FAIL: out-of-sample: ..., so a full-store replay of the
history the models trained on yields a refused verdict. A passing verdict needs a bounded evaluation
range (replay --policy-eval --eval-after NS|auto: absolute ns, or auto = the latest training-range
end among the gate models). The replay side is in place (ReplayOptions::eval_after_ns,
eval_after_from_models, provenance eval_error); the CLI flag is landing in parallel, so a build
without it cannot produce a passing gate - see API_CONTRACT.md. Procedure and criteria:
docs/evaluation/stage4-fleet-gate.md (criteria unchanged from stage4-promotion-gate.md).
Promotion stays an operator step on the nodes after reviewing the evidence; no state runs it.