strawberry/deploy/salt/test/workload.sh
trvon 39ffa02ae6
Salt: dockerized multi-node fleet test (master/central + 3 sensors) + state fixes
deploy/salt/test/: builds a release tarball from HEAD in ubuntu:24.04 (CI path),
boots systemd containers (rootless, no --privileged), master==central + 3
sensors with the host's Salt onedir mounted read-only, applies the real states,
runs a workload with planted YARA detections, then two orchestrated cycles.
43 checks (install/config/idempotence, real daemons + telemetry, R2 cache
miss/hit, cp.push collect, 3-seed cycle, canary hold then fleet rollout, no
promoted model anywhere, fleet bits in round 2, --since cursor, dedup).

State fixes found by running them for real:
- init.sls no longer includes strawberry.distribute: a plain state.apply on
  sensors shipped models fleet-wide and bypassed the canary; only the
  orchestration distributes now.
- install.sls: CI tarballs carry a +g<sha> suffix that 'strawberry version'
  does not print, so install.sh re-ran on every apply; record the installed
  release and compare base version + marker.
- orch collect: mv -f (re-collected bundles must not be skipped).
- central cycle: builds the overlay first; a training refusal for too few
  teacher labels (normal on a young fleet) publishes the overlay only and
  keeps the sensors' shadow, any other error still fails.
- distribute.sls: serves only what the cycle published (cp.hash_file; the
  cp.list_master file-list cache can miss a just-published file).
Render check gains overlay-only and full publish cases (20 checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:16:31 -07:00

34 lines
1.5 KiB
Bash
Executable file

#!/usr/bin/env bash
# Synthetic workload run INSIDE a sensor container: workload.sh <node-number> <seconds>
# Loops of distinct exec trees (sh -c, curl to a loopback stub, python one-liners, find/awk
# pipelines) plus a node-specific binary name, so exec rarity differs per node. Children live
# ~1 s so the procfs poller (no eBPF in containers) can see them.
set -euo pipefail
n=${1:?node number}
secs=${2:-120}
tool=/usr/local/bin/sb-node${n}-tool
install -m 0755 /bin/sleep "$tool"
python3 -m http.server 18080 --bind 127.0.0.1 --directory /etc >/dev/null 2>&1 &
stub=$!
trap 'kill "$stub" 2>/dev/null || true' EXIT
sleep 1
# Each iteration is detached (setsid, parent gone) so it roots its OWN replay window; children
# of one iteration share it. A window is a root exec plus descendants within 60 s.
run_iter() { ( setsid bash -c "$1" >/dev/null 2>&1 & ); }
end=$((SECONDS + secs))
i=0
while [ "$SECONDS" -lt "$end" ]; do
i=$((i + 1))
case $((i % 5)) in
0) run_iter "sh -c 'ls /etc | head -5 | wc -l; $tool 1'; $tool 0.5" ;;
1) run_iter "curl -s -o /dev/null http://127.0.0.1:18080/hostname?n=$i; $tool 1" ;;
2) run_iter "python3 -c 'import time; print(sum(range(100000))); time.sleep(1)'" ;;
3) run_iter "find /usr/share -maxdepth 1 -name '*a*' | awk '{n++} END {print n}'; $tool 1" ;;
4) run_iter "sh -c 'echo node$n round $i | tr a-z A-Z; $tool 1'" ;;
esac
sleep 1.5
done
sleep 3
echo "workload node$n: $i iterations in $secs s"