strawberry/deploy/validation/range/range_down.sh
trvon 39c5bc4efe
range: host-safety preflight, verified teardown, per-range scheduler identity
Host safety: range_up refuses to start below 10 GiB MemAvailable or 20 GiB free
disk, range_status prints the headroom, and the README documents both floors
and the 8 GiB pause a monitor should enforce. The 2026-11-17 global OOM was
caused by fifteen concurrent compilers plus the range, not the range alone, but
starting blind is what made it easy.

Teardown: sensor ids come from the vms/ contents rather than SB_RANGE_SENSORS,
so a lowered count or missing pid file cannot leave a VM behind; vm_pid accepts
a PID only when its command line still names that sensor's QEMU and disk, so a
recycled PID is never counted as live or signalled; a /proc sweep covers QEMUs
whose pid file was lost; --destroy refuses while any of them survives and
deletes only the sensor-N disks and seeds this range created.

Scheduler: range_up writes a per-range pid file and up/down/status act only on
a PID whose command line runs this repo's range_attack.py with SB_RANGE_DIR
resolving here, replacing the host-wide pkill that could kill another range.

Status: agents are reported live/stale/unknown per sensor group instead of a
raw registration total (six registrations for three VMs after restarts), and
the scheduler line now separates verified incidents from those without
execution proof, reporting "rule-pushed" rather than implying observed labels.
2026-11-17 13:45:22 -07:00

73 lines
3.1 KiB
Bash
Executable file

#!/usr/bin/env bash
# Stop the range. Default: pause (stop the scheduler + VMs + Caldera, keep all disks/state so range_up.sh
# resumes). --destroy also deletes this range's VM disks, seeds and state (keeps the base image).
#
# Teardown covers every sensor this range has on disk, not the first $SENSORS: the ids come from
# RANGE_DIR/vms, so a lowered SB_RANGE_SENSORS, a stale pid file or a VM whose pid file was never
# written cannot keep a QEMU alive. --destroy deletes nothing while any QEMU naming this range's disks
# is still running, and deletes only the explicit sensor-N names range_up.sh creates.
set -uo pipefail
. "$(dirname "${BASH_SOURCE[0]}")/range_common.sh"
destroy=${1:-}
ids=$(range_sensor_ids)
if sched_stop; then echo "scheduler stopped"; else echo "scheduler: not running"; fi
# Graceful pass: ACPI poweroff over ssh so the guest flushes its store, with the verified PID only.
for i in $ids; do
pid=$(vm_pid "$i" || true)
[ -n "$pid" ] || continue
ssh_vm "$i" 'sudo systemctl stop strawberry.service 2>/dev/null; sudo poweroff' 2>/dev/null || true
for _ in $(seq 1 20); do kill -0 "$pid" 2>/dev/null || break; sleep 1; done
done
# Hard pass: every QEMU that names this range's disks, including ones vm_pid() rejected (missing pid
# file, recycled PID). SIGTERM lets QEMU exit cleanly; SIGKILL is the bounded backstop.
for sig in TERM KILL; do
pids=$(range_qemu_pids)
[ -n "$pids" ] || continue
for pid in $pids; do kill -"$sig" "$pid" 2>/dev/null || true; done
for _ in $(seq 1 10); do [ -z "$(range_qemu_pids)" ] && break; sleep 1; done
done
live=$(range_qemu_pids | tr '\n' ' ')
if [ -n "$live" ]; then
echo "range VMs still running (pid${live% }): pid files kept" >&2
else
for i in $ids; do rm -f "$RANGE_DIR/vms/sensor-$i.pid"; done
echo "range VMs stopped"
fi
docker stop "$CALDERA_NAME" >/dev/null 2>&1 && echo "caldera stopped" || true
if [ "$destroy" != "--destroy" ]; then
echo "range paused (disks + state kept; range_up.sh resumes)"
exit 0
fi
# ---- destroy -----------------------------------------------------------------------------------------
# Refuse rather than unlink a disk under a live VM or the state a live scheduler is still appending to.
if [ -n "$live" ]; then
echo "refusing --destroy: QEMU still running for this range (pid${live% })" >&2
echo "disks and state kept; rerun range_down.sh, then --destroy" >&2
exit 1
fi
survivors=$(sched_pids | tr '\n' ' ')
if [ -n "$survivors" ]; then
echo "refusing --destroy: this range's scheduler survived SIGKILL (pid${survivors% })" >&2
echo "state kept so its pid file still identifies it; investigate by hand" >&2
exit 1
fi
docker rm -f "$CALDERA_NAME" >/dev/null 2>&1 || true
deleted=0
for f in "$RANGE_DIR/vms"/sensor-*.qcow2 "$RANGE_DIR/vms"/sensor-*-seed.iso; do
[ -e "$f" ] || continue
b=$(basename "$f")
if [[ $b =~ ^sensor-[0-9]+\.qcow2$ || $b =~ ^sensor-[0-9]+-seed\.iso$ ]]; then
rm -f "$f" && deleted=$((deleted + 1))
else
echo "kept vms/$b: not a sensor-N disk this range created" >&2
fi
done
rm -rf "$RANGE_DIR/state"
echo "range destroyed ($deleted disk/seed file(s) deleted; base image kept)"