Commit graph

140 commits

Author SHA1 Message Date
trvon
d9da7e2986
yara: drop partition subsets before reload and trim glibc heap 2026-11-19 18:22:57 -07:00
trvon
383c22cbd3
exec guard: answer a cache hit without holding the cache lock
Both hit paths (fanotify and Endpoint Security) called finish() while holding
cache_mutex_. finish() is the answer: it writes the kernel response, runs the
observation callback, bumps stats under stats_mutex_ and pushes to
report_mutex_. Holding the cache lock across that serialised every cache-hit
exec behind one mutex even though the daemon runs two workers, so on Linux a
slow answer was a slow exec - in the path whose whole job is to keep launches
fast.

detail::answer_cached_assessment copies the entry under the lock, releases it,
and only then answers. cache_mutex_ is now a leaf lock (also at the work-loop
insert), so the old edge cache_mutex_ -> {pending, stats, report} is gone and no
inversion can be introduced. Answer-exactly-once is unchanged: respond() still
checks pending->responded under the pending mutex, and the hit path still
answers once and returns.

The seam is declared in exec_guard.hpp so the tests no longer redeclare an
internal function in their own translation unit.
2026-11-19 17:29:38 -07:00
trvon
6618328059
fleet gate: refuse incomplete evidence and unprovable pooled provenance
Two ways a verdict could be built on evidence the tool did not actually have.

Pooling only ever upgraded provenance: one node that knew its training range set
the pooled flag, so a cell whose other nodes could not prove a range was still
reported as evaluated, and a cell could pass with no evaluated range at all. A
node with an unknown or unusable range now marks the pooled cell incomplete,
zeroes the displayed range (no range is shown that only some nodes could prove),
and appends "provenance incomplete: N of M contributing node report(s)". A known
range can no longer upgrade an unknown cell.

The documented "passes only if all 9 cells pass" was not enforced: passed was
just "some cells and not refused", so a one-seed, one-period report set exited 0.
A completeness gate now requires 3 seeds, 3 periods contiguous from 0, the full
seed x period cross product, and per-report consistency, and names every reason
("need 3 training seeds...", "missing: seed3 period 2", "inconsistent
--eval-model / --periods across node reports") before any verdict is computed.
2026-11-19 17:28:46 -07:00
trvon
d81ccbe7c8
gate: a bounded evaluation range, so out-of-sample evidence is possible
The in-sample check landed without any way to bound an evaluation, so every cell
was in-sample and every promotion path refused. Cutting the training bundles
earlier does not help: training spans the bundles, evaluation replays the whole
store with no lower bound, and the store keeps everything (retention 0), so
eval_first <= training_last always held.

replay --policy-eval gains --eval-after NS|auto (0 = unbounded). The bound is
applied where windows are ranked and where injected traces are planned, so the
periods partition only the out-of-sample span; auto derives it as the latest
training-range end over the loaded --eval-model files. Comparisons are written
without first + warm-up arithmetic so INT64_MAX timestamps cannot overflow. A
bound past the end of the store does not produce a silent empty cell: the cell
carries eval_error and promotion_gate refuses naming it.

Provenance also gets honest about width: training_last_ns is the last training
window's END (a window spans window_ns, so using its start left up to 60 s of
trained-on data looking disjoint), and the single-model path (model promote) now
uses the ranked span instead of the whole store, which had guaranteed a refusal.

Verified: R11c passes with 128 out-of-sample windows and a derived bound, and
unbounded returns to in-sample. Refuses an explicit bound later than all history.
2026-11-19 17:27:59 -07:00
trvon
cae67981ab
detection: mark a timed-out scan incomplete without bypassing quarantine
Review of 893127c found that reporting `scanned = false` for a timed-out
guard scan was wrong in a way that mattered: decide_exec answers `allow` on
`!scanned` BEFORE it looks at the operator quarantine, so a quarantined binary
whose scan overran its deadline would have been allowed to run. It also dropped
the only `scanned = true` on the normal assessment path, so every ordinary
assessment became "unscanned".

The assessment now keeps `scanned = true` (quarantine, allowlist and IOC
decisions still apply, and the guard still fails open on a thin result) and
carries a separate `incomplete` flag, which alone prevents caching:
guard_assessment_cacheable() and the pipeline's cache_put both exclude it.

Also fixes the same verdict being lost one caller up: scan_on_demand set
ok = false with an explanatory error, then unconditionally reset ok = true after
the match loop, so control scan-fd, MCP and retro-hunt all reported a timed-out
scan as a completed one with no error.

Tests: a timed-out assessment keeps scanned true and top_rule evidence, is
marked incomplete, and a second exec re-scans instead of hitting the cache; a
quarantined incomplete assessment still blocks in both enforce and audit, while
an incomplete clean one is allowed and not cacheable.
2026-11-19 16:48:19 -07:00
trvon
666608c4c0
detection: ctime in the scan key, retry after a failed scan, never hold live execs
Three launch-path defects from the audit.

The scan cache key was dev/ino/size/mtime, and mtime is settable via utimensat,
so a rewrite that restored size and mtime (cp -p, install -p, package restore, or
a deliberate attacker) served the old verdict with no TTL. The key now includes
ctime, and the fanotify path calls the shared detail::image_cache_key instead of
re-deriving the fields, so the guard and the pipeline cannot drift apart again.

recent_images_ was inserted before the scan and never removed, so an image that
was unavailable or whose buffer grant was refused was never retried for that
process instance - contradicting scan_image's own comment. The key is now
dropped when the outcome is unavailable, read_error or yara_error, while
completed scans stay deduped.

Live exec scans were deferred for up to ~5 s per emergency episode. The live
queue is bounded at 4096 and full means dropped exec events, so a fork storm
during an emergency could lose telemetry the guard depends on. The live lane now
takes a single non-blocking probe (kLiveExecHold is 0, pinned by a
static_assert) and scans anyway, fail-open; background inventory keeps the
bounded hold.

Tests: a same-size, mtime-restored rewrite is re-scanned rather than served from
cache; a transient scan failure is retried on the next event while a completed
scan is not; the live-lane probe never blocks.
2026-11-19 16:21:34 -07:00
trvon
7149a90893
gate: refuse in-sample and degenerate evidence, and identify the model
Four findings from a four-lane audit made the promotion gate unable to support
its own claims.

In-sample evaluation: training walks the whole post-warm-up store and the gate
evaluates the same windows, with nothing checking provenance - replay.cpp itself
listed point_in_time_splits_and_overlap_not_assessed. Models now record the
window range they learned from (file format v2; v1 still loads as unknown) and a
content fingerprint, evaluation reports its own range plus in_sample, and
promotion_gate fails with the reason: an overlapping range, no recorded range,
or no digest. Unprovable provenance never counts as out-of-sample.

Degenerate baseline: with rarity-greedy on 0 hits, hits_per_cpu_second() returns
0.0, so ">= 1.5x rarity" reduced to "model hit something" and the recall check
was vacuous. A cell whose baseline scored no hits or spent no CPU now fails,
naming the budget refusals that starved it.

Stale models: cells keyed on a bare filename, so a later cycle that refused to
train left an older cycle's dist/gate models to be judged, and nothing bound a
verdict to the file it judged. Cells and reports carry the model digest, and
fleet gate sets refused (non-zero exit) for a cell without one or for a
(model, period) whose digests disagree across nodes instead of pooling them.

3 seeds are not generalization: they vary initialisation and the per-node
shuffle over identical windows. The procedure document now says so and states
what would test generalization (disjoint periods, models trained on an earlier
cut, out-of-sample ranges).

Tests: in-sample flagged, disjoint range not flagged, unprovable provenance
fails, 0-hit and 0-CPU baselines fail, mixed digests refused, legacy report
refused, range round-trip and fingerprint, v1 compatibility.
2026-11-19 16:20:40 -07:00
trvon
ac4150f6e7
build: spell out the optional in the control-reply helper
`return cond ? parsed["data"] : nlohmann::json::object();` became ambiguous
between std::optional's converting constructor and nlohmann::json's greedy one
when that translation unit was recompiled. Name the type explicitly; no
behaviour change.
2026-11-19 11:49:41 -07:00
trvon
893127c691
detection: a timed-out or degraded scan is never a clean verdict
Two launch-path defects let a weaker-than-complete scan become the cached
answer for an executable until the next rules reload. In enforce mode that
means the exec guard silently stops consulting YARA for that binary.

1. A YARA deadline overrun was reported as success. timed_yara_scan/
   timed_yara_scan_ranges returned `ok || timed_out`, so assess_exec set
   have_matches and cached the image with an EMPTY match list; the guard cached
   that assessment too. The guard's YARA budget is exactly prevention_timeout_ms
   (1 s by default), so one slow scan was enough. They now return a
   BoundedYaraScan {ok, timed_out} and only `completed() == ok && !timed_out`
   may set have_matches or reach cache_put; scan_image returns yara_error and
   caches nothing; the assessment reports scanned == false so the guard fails
   open instead of inheriting a clean answer. Matches found before the deadline
   are still reported as evidence, and a timeout is counted in yara_timeouts.

2. The guard cached degraded and partial assessments, contradicting
   exec_guard.hpp ("not cached") and the pipeline, which already excluded them.
   A degraded result is produced by a transient condition (the scan-buffer lease
   is momentarily held), so one collision pinned the image to hash/IOC-only for
   the whole rules+policy generation. The rule now lives in one place,
   guard_assessment_cacheable(), used at both the Linux and macOS cache stores.

Tests: a timed-out assess_exec is not scanned, still reports the partial match,
and a second exec re-scans instead of hitting a cache; the cache predicate is
covered for clean, degraded, partial and unscanned assessments.
2026-11-19 11:49:41 -07:00
trvon
31e44caab0
cli: honor <group> --help, and add fleet run for a standalone central node
`strawberry fleet --help` reported "unknown subcommand 'fleet --help'" and
then dumped the group's usage, even though docs/cli-ux.md promises that
`strawberry <cmd> --help` prints the same text as `strawberry help <cmd>`.
A group word followed by -h/--help now prints the group listing.

Centralized training had no single entry point: `fleet train` and
`fleet overlay` had to be run by hand (or through Salt), so a standalone
central node had no "run" command at all. `strawberry fleet run <bundle_dir>
[--out DIR]` trains and builds the overlay by calling the same two code paths,
so the behaviour cannot diverge, and prints the model/overlay paths.

It deliberately does not distribute to nodes, apply the overlay, or promote a
model: distribution stays behind the Salt canary cycle and promotion stays
behind `model promote` plus the fleet promotion gate. The JSON output carries
distributed=false and promoted=false so a script cannot mistake it.

Tests: fleet --help and fleet -h print the group without error; fleet run maps
to --fleet-run with optional dir, --out/--seed/--epochs passthrough, rejects
unknown options; a bare `fleet` still explains itself. Verified end to end on
two seeded bundles: model + overlay written to the requested directory.
2026-11-18 12:11:34 -07:00
trvon
433615044e
governor: release a memory episode on the memory hold, not the current RSS
compute_level picked the release hold from the RSS *after* the drop, so an
episode raised by the daemon's own memory released in 150 ms as soon as RSS
fell below the warning ratio — ignoring the documented 1000 ms memory hold.
A warning-level memory episode could therefore clear almost immediately.

The episode is now classified when it is raised: memory_driven_episode_ is set
when memory drove the level, and that decides the release hold. Critically this
does not clear when the level steps down (critical -> warning is the same
episode); the next escalation re-classifies it.

Tests: a memory warning -> calm transition is still warning at 200 ms and
999 ms and clears at 1000 ms, while a CPU-driven episode still clears in 150 ms.
2026-11-17 13:52:37 -07:00
trvon
a1d77dda9a
governor: fail-safe host pressure, diagnostic diskstats, capped scan buffers
Host pressure stays fail-safe: any of CPU/I/O/memory PSI above its profile
threshold, or low MemAvailable, still defers background work and can degrade
the exec guard. That is deliberate — global I/O PSI also counts waits with no
block-device counter (FUSE/network-backed I/O, display page flips), so idle
disks never prove pressure is benign. An audit of an earlier version that
vetoed I/O PSI below 5% block utilization rejected it on exactly that basis.

/proc/diskstats is therefore diagnostic telemetry only, exposed as
host.block_io_util_pct. It is keyed by major:minor so a reused device name
cannot compare unrelated counters, and reports unknown (-1) for the first
sample, a new/replaced/reset device, or an oversized read, instead of a false
0%.

scan_buffer_budget() now derives from the same effective budget that pressure
uses — the configured/profile budget capped by cgroup memory.max — with the
existing 32 MiB floor, so a cgroup-capped daemon cannot be granted scan
buffers larger than its own limit. Lock order stays mutex_ -> buffer_mutex_.

Tests: deterministic fail-safe cases across every diskstat value, the
new-device regression, scanner budget under a 100 MiB cgroup cap, and a seeded
property asserting severe I/O PSI always holds work whatever diskstats says.
2026-11-17 13:45:07 -07:00
trvon
6f5c62b31c
Stage 4 over fleet data: per-node 3x3 policy evaluation, pooled fleet promotion gate
The pre-registered gate (docs/evaluation/stage4-promotion-gate.md) asks for the
criteria to hold on 3 non-overlapping periods x 3 training seeds; until now the
tool reported a single run and the operator recorded the rest. This produces
the 9 results for a fleet, with the criteria unchanged:

- replay --policy-eval --eval-model FILE (repeatable) --periods N: one pass
  evaluates every model on every equal time slice of the ranked history; each
  (model, period) cell simulates the model and rarity-greedy with their own
  hourly budgets exactly as policy_eval does (policy_eval.cells). With one
  model equal to --model and one period the cell equals the single-model
  evaluation (test R11b); periods partition windows and positives.
- strawberry fleet gate DIR: pools the cells of every node report (windows,
  positives, inspections, CPU-s, hits, refusals) per (model, period) and
  applies learn::promotion_gate unchanged to each; passes only if all pass.
  Malformed / cell-less reports are skipped and listed.
- Evaluation stays on the node (raw telemetry never leaves it): the central
  cycle now also publishes all three seed models to dist/gate/; strawberry.gate
  (sensors) evaluates them with the pre-registered evaluation traces; the
  operator-run orch.strawberry_gate pushes the reports back and pools them;
  evidence under central.work_dir/gate-<stamp>/. Nothing promotes.
- docs/evaluation/stage4-fleet-gate.md: the procedure (new dated document;
  criteria and numbers unchanged).

Validated: test_fleet_gate (pooling sums, pooling reaching 10 positives
neither node has alone, one failing cell fails the fleet, malformed reports
skipped), R11b, Salt render checks (no models -> refusal; 3 models x 3
periods; orchestration pools), and scripts/fleet_e2e.sh end to end: 3 nodes ->
9 pooled cells with 28-50 positives each; the untrained synthetic models
correctly do not pass (exit code matches the verdict).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:18:04 -07:00
trvon
1c9b005645
store: incremental read-only refresh (fixes the tray spinning a core)
Root cause, from the Tracy capture the CPU watcher took when the tray spun
(2026-11-15 08:15, 67% for 15 s): database.reload_read_only rebuilt the whole
evidence view from every segment — mean 1.4 s, max 1.7 s, 4x in 20 s — because
every 3 s poll that saw the daemon append evidence triggered a full reload.

- store::scan_store_read_only_from + ReadCursor (per segment: id, sealed,
  bytes, records delivered). When the previously seen segments are an
  unchanged prefix (same ids, sealed ones byte-identical), only records
  appended since are delivered: the rest of the previously last segment
  (skipping what was applied) and new segments. Any other change —
  compaction, retention, reduction, a rewritten or shrunk segment — delivers
  everything and the caller rebuilds.
- Database read-only refresh uses it; records apply in the same order a full
  rebuild uses, so the view is identical. read_only_reload_counts() reports
  full vs incremental refreshes.

Live, on the real evidence store: the change-driven reload went from ~1.4 s
to 2.8 ms; the tray did ~97 ms of work in a 60 s window.

Test (test_incremental_view, seeded property): a long-lived reader
refreshing across random appends, upserts, seals and a compaction always
equals a freshly opened reader; appends refresh incrementally (42-49 per run),
the compaction forces a rebuild, and no change reports no change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:46:00 -07:00
trvon
8411c70ba8
macOS: build and test fixes found re-validating on the Mac (57/57 green)
- fleet.cpp: HOST_NAME_MAX is Linux-only; use a 256-byte buffer (the
  POSIX/RFC 1035 limit gethostname() truncates to).
- test_replay: two dangling pointers - victim(run(...)) returned a pointer
  into a temporary report destroyed at the end of the statement. Passed on
  Linux by allocator luck; failed deterministically on macOS for every seed.
- test_yara_scanner: a brace initializer with a comma inside assert() (the
  macro splits it on macOS), and ::close needs <unistd.h> outside the
  Linux-only block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:35:34 -07:00
trvon
db4c49155f
yara: rule partitions — scan each file only with the rules that can match it
Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.

Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
  after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
  pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
  size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
  kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
  priority order (big non-PE first) under rule_partition_budget_pct (default
  100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
  would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
  and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
  differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
  partition, full_scans); config rule_partitions (default true) and
  rule_partition_budget_pct.

Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:58:39 -07:00
trvon
4586eed50d
yara engine: range/fast-mode scanning API, ELF load ranges, engine A/B bench, opt-in guard partial scan
Strawberry drives libyara (system 4.5.8), no custom engine. Profiling the scan path
(Tracy + user-space perf) shows 86% of scan time inside libyara's string-search loop;
our callback runs only on matches. Engine-usage variants measured with the new
`strawberry-scan-bench --engine` A/B over 3434 ELF files / 2.9 GB (5132 rules):
  full 50.1 MiB/s | fast mode 1.01x | loaded segments only 1.11x (2x on debug/ASan
  builds; 1.08x in the 4-32 MiB band the exec guard full-scans). 0 match-set
  differences, but the corpus has no rule matches, so parity is not proven.

- YaraScanner::scan_memory_ranges (yr_scanner + yr_scanner_scan_mem_blocks over
  zero-copy blocks; file offsets and filesize keep file semantics) with
  EngineOptions{fast_mode}; elf_load_ranges() (PT_LOAD + first page, merged,
  bounds-checked against hostile headers; empty -> scan everything).
- Exec guard: opt-in `guard_loaded_segments_only` (default off) scans only the
  loaded segments of ELF images >= 4 MiB at launch; such results are marked
  `partial`, never cached, and the pipeline still scans the whole file. Status
  pipeline.guard_partial_scans. Off by default: an 8% faster launch decision is
  not worth missing unloaded-section strings until the background scan.
- Fast mode is not adopted (1%, and it drops #count / @offset semantics).

Tests: ELF ranges (synthetic, BE, truncated, absurd phnum, phoff overflow, clamped
filesz, /bin/true), range scans (unmapped string unseen, at/uint32/filesize keep
file offsets, multi-block), guard partial scan catches mapped markers / misses
unmapped ones / pipeline full scan detects them / default off scans everything,
config round trip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:51:10 -07:00
trvon
11d3614348
profiling: pinned Tracy client, headless capture script, scan bench, tray + scan-stage zones
- third_party/tracy.wrap pins the Tracy client to v0.15.0 (sha256-checked), matching
  the Arch tracy 0.15.0 capture/export tools (client and tools must match exactly);
  built with tracy_enable + on_demand, so nothing is recorded or paid for unless a
  capture is connected. -Denable_profiling=true picks it up; default builds are
  unchanged (macros are no-ops).
- scripts/profile_capture.sh: headless (no GUI, no root) `run OUT -- CMD` or
  `attach OUT SECONDS` -> OUT.tracy + OUT.zones.csv + a top-zones table.
- strawberry-scan-bench (dev tool, not installed): the real scan_on_demand path over
  files with the real governor and rule set, so the scan path can be profiled
  without root or the daemon.
- Zones: scan_on_demand, image.read, image.sha256 (next to yara_scan/scan_image/
  assess_exec), tray.* (poll, on_reply, refresh_evidence, update_status/_policy,
  refresh_findings, notify) and evidence/database reloads.

Baseline (6 yams ASan binaries, 668 MiB, idle host): YARA 86%, read 9%, sha256 5%,
~61 MiB/s; PT_LOAD segments are 35-48% of those files. DEVELOPER.md: profiling
section, and the balanced row corrected (64 MiB image cap, 768 MiB budget).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:29:57 -07:00
trvon
5685600e57
Daemon memory: budget scan buffers, stream large images on the launch path, report throttling
2026-11-14 host hang: a yams ASan test run (~110 MiB binaries) pushed
strawberry.service past MemoryHigh=768M; the kernel throttled the daemon
(own memory.pressure full ~72%), so the exec guard stalled launches (589
1-second prevention timeouts in 80 min, even start-swayidle.sh), scans hit
p95 67 s and the control socket stopped answering.

- Exec guard (assess_exec) no longer copies an image before deciding: a cached
  scan answers with no read; an image over guard_full_scan_max_bytes, or when
  the scan-buffer budget is exhausted, is hashed by streaming (1 MiB chunks,
  new HashUtils::sha256_md5_fd) and decided on hash / IOC / quarantine
  (degraded; the pipeline still scans it). Only small budgeted images are copied.
- Scan-buffer budget (ResourceGovernor::acquire_scan_buffer, RAII
  ScanBufferLease): in-memory image copies bounded to memory budget / 4
  (balanced 192 MiB); the pipeline waits up to 30 s, on-demand scans 60 s;
  an oversized request is granted only alone, so no deadlock or starvation.
- Balanced pipeline_max_image_bytes 128 -> 64 MiB: larger images get hash/IOC
  checks only (as on the efficient profile).
- Response hashing streams instead of copying the whole executable.
- Status: resources.cgroup memory_high_mb / memory_high_events /
  memory_full_pct / throttled, and resources.scan_buffers; doctor check
  resources.memory (problem when throttled at MemoryHigh, with the fix).

Tests: budget grant/refuse/exclusive/wait/timeout/move, PSI full + memory.events
parsers, streaming hash = buffer hash across chunks, guard streams a 33 MiB
image with no YARA pass and no buffer grant, streams when the budget is
exhausted, full-scans + caches when budget is free, doctor throttle check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 15:36:14 -07:00
trvon
758ef9db07
fleet: node-share cap never below the fair share 1/N
Seen in the dockerized 3-sensor run: max_node_share=0.25 (pillar default) with
3 similar-sized nodes cut every node to 25% of the total, dropping 158/468
windows (34%) while correcting no imbalance. The cap is now
max(max_node_share, 1/N) * total, so a balanced fleet is never trimmed and
only nodes above an equal share are cut. Test: balanced 3-node fleet keeps
all windows; a dominant 4th node is still trimmed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:29:06 -07:00
trvon
b1c7cc41dd
fleet: e2e harness + fleet train/overlay JSON, *.sbx-only bundle listing
fleet train advertised --json but printed text, so the central cycle's
train-seedN.json evidence was not JSON; it now emits one object incl.
duplicate_windows. fleet overlay gains --json (bundles_superseded,
overlays_skipped). Both commands read only *.sbx (no dotfiles) from the
bundle dir, so a cursor or stray file is never fed to the merge.

scripts/fleet_e2e.sh drives the real binary + central cycle script over N
synthetic sensors: --since auto export chain, Salt-pull simulation, 3-seed
train + overlay, sha256 publish, determinism, overlay distribution and a
round-2 export carrying fleet rarity features. 6/6 checks pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:00:57 -07:00
trvon
8f4cb586b8
Rules R2: node-local compiled-rules cache keyed on libyara version + sources
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:54:28 -07:00
trvon
ebade1ba80
Store 7c/7d: tunable merge horizon + CPR interleave-aware merge mode
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:50:09 -07:00
trvon
4436bee457
Fleet F1: merge dedup per node+window, overlay newest-per-node, export --since
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:50:03 -07:00
trvon
00af731088
Fleet E1: export egress dir + age/size retention (produce + prune synced bundles)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:14 -07:00
trvon
1ee208fb4b
Stage 7e: scheduled stop->reduce->start telemetry-reduction maintenance (opt-in)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:13 -07:00
trvon
cbcb9896cd
Stage 7c: NodeMerge collapse redundant repeated benign accesses on live nodes
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:13 -07:00
trvon
9aa9c9bb46
Stage 7b: minimal reachable skeleton (skeletonize dead-benign nodes far from labels)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:13 -07:00
trvon
69d86bbbc3
Rules R1: ship pinned yara-forge-core source snapshot in the release (no per-node fetch)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:12 -07:00
trvon
c679a03561
Stage 7a: LogGC dead benign subtree GC (causality-preserving telemetry reduction)
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Successful in 53m38s
release / publish (push) Has been skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-10 19:32:12 -07:00
trvon
58907eb45e
Stage 3.5d: export attribution signal (detection->dropper origin hashes)
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 56m28s
release / build (ubuntu:24.04, noble) (push) Successful in 59m8s
release / publish (push) Has been skipped
Ship 3.5c's provenance attribution to the central learner. ExportLabel gains two
additive uint64 tags (8 origin_node_hash, 9 origin_exe_hash); a positive window's
label now carries token_hash(dropper node) + token_hash(path_template(dropper
exe)) from ReplayWindow.origin_*. fleet_merge counts windows_with_origin.

R12/R13: a FIELD addition, not a new record type -- no codec.hpp/enum change
(R12 untouched), both fields are uint64 hashes so no raw path/node string crosses
the export boundary (R13 green, no name-loophole). Back-compat additive:
kExportSchemaVersion stays 1; old bundles decode with origins 0; old readers
ignore tags 8/9. Positive labels only -- suppressed/negative carry no origin.

Tests: round-trip of the hash fields + defaults; extraction proves a dropper->exec
detected window's positive label carries a non-zero origin while a no-write
detected window and suppressed labels carry 0. source_invariants green.
A learned ranking head + TrainingExample plumbing are phase-2 (out of scope).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 18:04:15 -07:00
trvon
b448c51c1d
Stage 3.5c: provenance attribution (trace a detection to the dropper node)
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 55m28s
release / build (ubuntu:24.04, noble) (push) Successful in 51m16s
release / publish (push) Has been skipped
Reuse the global write->exec linkage (recent_writes) to attribute a detection to
its ROOT CAUSE. recent_writes now maps path -> {writer_node, writer_exe}; when a
window is late_detected (retro-hunt flags a binary after it ran) or detected
(YARA), set ReplayWindow.origin_node/origin_exe from whoever WROTE the flagged
binary -- usually an earlier node than the exec leaf that ran it. Surfaced in
window_json. Deterministic provenance walk (first writer wins); replay-level only,
no export-schema/record-type/enum change (R12/R13 intact), self_written_execs
unchanged. A learned ranking head over ambiguous origins is the follow-up (3.5d).

Test R10: a dropper writes /tmp/evil in one window, a separate process execs it
and is flagged late -> the window's origin_node is the dropper, not the exec leaf;
no in-set write -> empty origin; deterministic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 17:21:55 -07:00
trvon
70652b5fb0
Review fixes #4/#5: ptrace medium middle-ground + reject fleet-train flags elsewhere
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 55m51s
release / build (ubuntu:24.04, noble) (push) Successful in 51m11s
release / publish (push) Has been skipped
#4 (guard retention): same-binary parent/child self-ptrace never rates "high"
(it's self-debugging, not injection), but from a drop location (/tmp) it now
stays "medium" rather than "low" -- a same-binary anti-debug/injection staged in
/tmp still surfaces, while sanitizer builds no longer flood high-severity
PtraceAttach. Injection into another program is unchanged.

#5 (CLI-UX contract): --max-node-share/--test-fraction/--split now error when
passed to any command other than 'fleet train' instead of being silently
ignored (noun-verb commands already reject unknown options; this covers the
flag-form and other fleet subcommands).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 16:17:01 -07:00
trvon
e5ee53cf18
Review fixes: write->exec feature constant, accurate comments, tag-safe CI
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 56m31s
release / build (ubuntu:24.04, noble) (push) Successful in 51m33s
release / publish (push) Has been skipped
Address code-review findings on this session's changes (no behavior change):
- Share the dense-slot index for the write->exec feature as
  learn::kWriteExecFeatureIndex; featurize() and the model-score dev tool now
  reference it instead of a duplicated literal 42 (prevents silent column drift
  if the feature layout shifts).
- Fix the self_written_execs comments: the linkage is a GLOBAL bounded FIFO
  (recent_writes), not strictly "this window" — the docs now match replay.cpp.
- release.yml: never cancel a tag/release run (cancel-in-progress only for
  non-tag refs); push ordering doesn't guarantee the tag event arrives after the
  branch event, so a same-sha branch push must not abort an in-progress release.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 16:09:03 -07:00
trvon
43efeac44c
Release: one version source, CI version contract, scripts/release.sh
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Has been cancelled
release / publish (push) Has been skipped
The version lived in three hard-coded copies (CLI `version`, daemon status,
banner); v0.2.0 shipped with the CLI still saying 0.1.0. meson.build is now
the only source: it passes STRAWBERRY_VERSION to every target, and the CLI
test requires the exact version.

packaging/ci/build-release.sh refuses to publish a tag build unless
meson.build, the tag and the built binary's `strawberry version` agree; CI
never rewrites the version.

scripts/release.sh X.Y.Z is the one manual release step: on a clean main
equal to forgejo/main it bumps meson.build, makes a signed commit and a
signed tag, and pushes both (--dry-run shows the plan). The Security Hub
release channel deploys signed tags only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 08:55:05 -07:00
trvon
7f3cbeca71
Detection quality: self-ptrace of the same binary stays low even from /tmp
All checks were successful
release / draft (push) Successful in 19s
release / build (debian:trixie, trixie) (push) Successful in 58m7s
release / build (ubuntu:24.04, noble) (push) Successful in 58m54s
release / publish (push) Successful in 14s
A parent/child of the SAME binary tracing each other is self-debugging
(sanitizers' StopTheWorld, crash handlers), not injection. The code already
scored that `low`, but `suspicious_image` (which flags drop locations like /tmp)
took precedence and forced `high` -- so sanitizer test binaries built in /tmp
flooded findings as high-severity PtraceAttach (observed live: ~160 FPs on
/tmp/toolcheck-*/*_san). Reorder so the strong structural self-debug signal beats
the weak location heuristic: same-program-family -> low regardless of location;
ptrace into ANOTHER program stays high (suspicious image) or medium. Injection
detection is unchanged. Test added for the /tmp self-ptrace case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-09 08:06:36 -07:00
trvon
1733bc5656
Stage 3.5 bench: make write->exec linkage global (fixes self_written_execs)
All checks were successful
release / draft (push) Successful in 21s
release / build (debian:trixie, trixie) (push) Successful in 1h1m23s
release / build (ubuntu:24.04, noble) (push) Successful in 56m1s
release / publish (push) Successful in 14s
The write->exec data-flow feature used a PER-WINDOW written_paths set, but a
dropper/download and the exec of the dropped file routinely land in different
replay windows (writer's window vs the exec's parent window, or a window that hit
max_window_events) -- so self_written_execs fired for only ~1/15 real download
chains. Mirror the live behavior engine's drop_writes_: record written paths in a
global bounded FIFO (recent_writes) and match on exec. Now a base64-decode evasion
that DownloadThenExecute misses (detections=0) is scored 0.594 by a data-flow
model vs 0.021 by a no-provenance model and ~0 by the detectors -- the data-flow
model generalizes to the detector-evading attack it never trained on. Benign ~0 for
both (no FP).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 18:26:51 -07:00
trvon
fdd96ed64f
Release v0.2.0: bump version + fix license to GPL-3.0
All checks were successful
release / draft (push) Successful in 19s
release / build (debian:trixie, trixie) (push) Successful in 54m43s
release / build (ubuntu:24.04, noble) (push) Successful in 50m32s
release / publish (push) Successful in 13s
Bump project + daemon version 0.1.0 -> 0.2.0. Correct meson `license` from the
stale `MIT` to `GPL-3.0-or-later` to match the GPLv3 LICENSE file and README.
First tagged release carrying Stage 6 fleet learning and Stage 3.5 provenance
work (3.5a fleet-train flags, 3.5b2 write->exec data-flow feature).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 16:40:04 -07:00
trvon
0282066340
Stage 3.5b2: replace exec-lineage with write->exec data-flow feature
Some checks failed
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Has been cancelled
release / build (ubuntu:24.04, noble) (push) Has been cancelled
release / publish (push) Has been cancelled
3.5b's ordered exec-lineage proved inert in clean A/Bs; replace it with a general
write->exec data-flow feature: WindowSummary.self_written_execs (exec of a file
written earlier in the same window) at x[42], tracked via OpenWindow.written_paths.

Honest status: this is the conceptually-correct provenance feature (it separates
download->exec-the-same-file from download->exec-unrelated), but it ALSO shows no
AUC gain on detector-labeled data -- for the same structural reason as 3.5b: the
teacher labels (DownloadThenExecute etc.) correlate with existing count/location
features or re-derive the detector. Validating provenance LEARNING requires the
novel-chain generalization benchmark (train on detector-covered, hold out
detector-evading attacks); committed as that benchmark's feature foundation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 16:37:06 -07:00
trvon
8544e921a2
Stage 3.5b: ordered lineage-path bigram features
All checks were successful
release / draft (push) Has been skipped
release / build (debian:trixie, trixie) (push) Successful in 28m43s
release / build (ubuntu:24.04, noble) (push) Successful in 26m22s
release / publish (push) Has been skipped
Feed ordered parent->child exe-template bigrams into the window feature vector
(WindowSummary.lineage_edges, note_lineage_edge, a salted hash loop in featurize,
wired at replay on_exec). The model already got per-kind lineage surprisal
(x[25..29]); this adds path ORDER, which the unordered token bag discarded.

Clean A/B on a fixed corpus (same 697 windows, same flags, only the binary
differs): model AUC 0.734 -> 0.737 (rarity 0.520 unchanged). Null on this
shallow-lineage data (marker execs are 1-process, synthetic data flat) but the
feature is correct and order-sensitive (test: A>B != B>A). Kept as the path
representation that Stage 3.5c's retro-hunt attribution head needs; its payoff is
expected on real multi-stage attack telemetry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 14:22:14 -07:00
trvon
2cc137b8e7
Stage 3.5a: fleet-train --max-node-share/--test-fraction/--split
Expose the merge cap and train/test split on `strawberry fleet train` so a
single-node real-attack bundle can pre-seed a model. Default stays temporal +
cap 0.25 (behavior-preserving); `--split random` + `--max-node-share 1` lets the
Caldera-attack export (15 positive windows) train. Verified: real-only train AUC
1.0 vs rarity 0.563; default still refuses identically.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-08 12:18:32 -07:00
trvon
39912f6e64
Stage 6: wire config node_id to export (stable node identity; gap #2) 2026-11-08 09:05:25 -07:00
trvon
b95bdb04de
Stage 6: strawberry fleet overlay CLI — produce the fleet overlay from a bundle directory (gap #1) 2026-11-08 08:59:52 -07:00
trvon
446e5c3afe
Stage 6: export populates fleet_bits from a local overlay (export_store fleet_surprisal callback; run_export loads fleet-overlay.sbo) 2026-11-08 08:09:47 -07:00
trvon
9313134aee
Stage 6: populate WindowSummary.fleet_bits from a fleet overlay during scoring (ReplayOptions.fleet_surprisal callback) 2026-11-08 08:01:40 -07:00
trvon
97374a16d2
Stage 6: fleet-overlay feature — per-kind fleet_bits[5] in featurize (reserved slots x[37..41], back-compatible) 2026-11-08 07:55:40 -07:00
trvon
7ad7596bda
Stage 6: load fleet overlay + sketch_surprisal_bits (query fleet-wide rarity for a tuple) 2026-11-08 05:38:00 -07:00
trvon
c939c3536b
Stage 6: fleet rarity overlay — sum per-node count-min sketches and serialize as an overlay bundle 2026-11-07 21:28:35 -07:00
trvon
31e4833169
Stage 6: strawberry fleet train — merge a bundle directory and train the shadow model; fleet_merge populates rarity_bits 2026-11-07 21:22:14 -07:00
trvon
ae4555ec38
Stage 6: central fleet merge core — export bundles to capped TrainingExamples (max_node_share) 2026-11-07 21:12:42 -07:00