Commit graph

3 commits

Author SHA1 Message Date
trvon
f76db466eb
Stage 4: keep the live rarity model across daemon restarts
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The shadow scorer's rarity model started empty at every daemon start, so
each restart cost an hour of warm-up with no decisions. The daemon now
saves it to model_dir/rarity.state (checksummed, atomic write, hourly and
at stop) and restores it at start. A restored state younger than the
7-day half-life skips the warm-up; decay catches up on the first event.

A file made with other rarity settings, a bad checksum, or a negative or
non-finite count is refused with a reason (status model.rarity_state,
doctor model.rarity_state) and the model starts empty. The file must be
owned by the daemon user and not group/world writable, like model files.

Also fixes status model.warmup_windows, which was never filled in.

Tests: R1/R2 in test_learn (round trip, decay clock, refusals), S8 in
test_shadow (warm-up skipped only when recent), daemon restart restores
the saved state, doctor warning; new fuzz_rarity_state target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 23:25:01 -06:00
trvon
bbfda149a5
Stage 4: execute a promoted model's inspections, under guardrails
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
A promoted model (promoted.sbm, written only by `model promote` after the
gate) can now request real inspections. Only two, both additive:

- memory_scan: through the daemon's memory-scan queue;
- deep_rescan: the root's image re-scanned on demand; new matches are
  recorded as indicators of kind model_inspection (findings titled "YARA
  match found by a deeper rescan the inspection model requested") and
  positive labels.

watch_subtree has no mechanism and rescan_on_next_rules is the retro-hunt's
job, so both are masked out of acting decisions (kExecutableActions), and
replay's policy evaluation uses the same action set.

Guardrails:
- InspectionExecutor (bounded queue, one low-priority worker, submit()
  never blocks); WorkClass::model_inspect lets the governor defer it, and
  a deferred decision is dropped.
- Each action rechecks that the promoted model is still active at the
  decision's generation (demoting or replacing it revokes queued work),
  that the root's node id and raw image path still match (decisions carry
  root_image, not the feature template), and skips protected processes.
  A deep rescan discards its result if the process, image or rules changed
  during the scan.
- An acting decision is durably written before it is dispatched; every
  attempt becomes an inspection_outcome record (queued, scanned, skipped,
  deferred, revoked, queue_full, failed).
- Source invariant R10: no file may couple learning types to the
  exec-guard, OpenSnitch or policy APIs (daemon_app.cpp and main.cpp, the
  wiring points, are reviewed by hand). Models never block, contain, kill
  or create findings directly.
- Config model_act and model_lambda; status model block reports acting
  stats; timings.model_inspection.

docs/evaluation/stage4-execution-gate.md amends the gate's method (not its
thresholds): executable action set, greedy promoted decisions, budgets as
estimated inspection seconds by scan profile.

Also: TerminalSafeBuf's destructor no longer lets an exception escape;
the health message clamps huge durations; replay timestamp overflow; the
Docker test build runs with -j2.

Tests: test_inspection_executor (every skip and execute path, revocation,
queue limits); shadow and replay acting cases; R10 fixture; fault injection
and invariant updates. Validation reported by the session that wrote it:
48/48 local, 46/46 Docker plus privileged live checks, 20 fuzz targets
(9.5M runs), three paired KPI runs without regressions (no acting model
was loaded, so active-inspection cost is not measured yet).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 21:03:57 -06:00
trvon
0006a86ff4
Stage 4: shadow mode for the inspection model
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / macos (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
The daemon now scores live windows with the shadow model and logs the
inspection it would request. Nothing is executed.

- learn/shadow: a live replay engine (same windows and rarity as
  `strawberry replay`) fed every telemetry record on the writer thread.
  When a window closes, the model's value v gives each action a benefit
  v x reach - lambda x CPU cost; the best positive one is chosen if the
  hour's budget (10/30/120 CPU-s by scan profile) allows, else none.
  With probability 0.02 a uniformly random other action is taken instead,
  and every decision records its propensity for off-policy evaluation.
  Decisions are inspection_action telemetry records.
- Replay gains a live mode (no ranking bookkeeping, so memory stays
  bounded) and a window-closed callback.
- Daemon: model_shadow (default on) loads <model_dir>/shadow.sbm when it
  appears or changes, checked every minute; an invalid file is reported
  and not used. Status gains a `model` block (kernel, generation,
  decisions by action, exploration, would-be spend, budget refusals).
- learn_sources in src/meson.build: tests that compile the daemon from
  sources take the learning library whole.

Limits, documented: decisions come when a window closes (a few minutes
after its first exec), and the live rarity model starts empty at each
daemon start (the first hour is learned, not decided).

Tests: test_shadow (no model, greedy choices, budget never exceeded,
determinism, exploration rate and propensities, record codec); daemon
status; fuzz_telemetry_records covers inspection_action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 12:04:40 -06:00