Root cause, from the Tracy capture the CPU watcher took when the tray spun
(2026-11-15 08:15, 67% for 15 s): database.reload_read_only rebuilt the whole
evidence view from every segment — mean 1.4 s, max 1.7 s, 4x in 20 s — because
every 3 s poll that saw the daemon append evidence triggered a full reload.
- store::scan_store_read_only_from + ReadCursor (per segment: id, sealed,
bytes, records delivered). When the previously seen segments are an
unchanged prefix (same ids, sealed ones byte-identical), only records
appended since are delivered: the rest of the previously last segment
(skipping what was applied) and new segments. Any other change —
compaction, retention, reduction, a rewritten or shrunk segment — delivers
everything and the caller rebuilds.
- Database read-only refresh uses it; records apply in the same order a full
rebuild uses, so the view is identical. read_only_reload_counts() reports
full vs incremental refreshes.
Live, on the real evidence store: the change-driven reload went from ~1.4 s
to 2.8 ms; the tray did ~97 ms of work in a 60 s window.
Test (test_incremental_view, seeded property): a long-lived reader
refreshing across random appends, upserts, seals and a compaction always
equals a freshly opened reader; appends refresh incrementally (42-49 per run),
the compaction forces a rebuild, and no change reports no change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- fleet.cpp: HOST_NAME_MAX is Linux-only; use a 256-byte buffer (the
POSIX/RFC 1035 limit gethostname() truncates to).
- test_replay: two dangling pointers - victim(run(...)) returned a pointer
into a temporary report destroyed at the end of the statement. Passed on
Linux by allocator luck; failed deterministically on macOS for every seed.
- test_yara_scanner: a brace initializer with a comma inside assert() (the
macro splits it on macOS), and ::close needs <unistd.h> outside the
Linux-only block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.
Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
priority order (big non-PE first) under rule_partition_budget_pct (default
100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
partition, full_scans); config rule_partitions (default true) and
rule_partition_budget_pct.
Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strawberry drives libyara (system 4.5.8), no custom engine. Profiling the scan path
(Tracy + user-space perf) shows 86% of scan time inside libyara's string-search loop;
our callback runs only on matches. Engine-usage variants measured with the new
`strawberry-scan-bench --engine` A/B over 3434 ELF files / 2.9 GB (5132 rules):
full 50.1 MiB/s | fast mode 1.01x | loaded segments only 1.11x (2x on debug/ASan
builds; 1.08x in the 4-32 MiB band the exec guard full-scans). 0 match-set
differences, but the corpus has no rule matches, so parity is not proven.
- YaraScanner::scan_memory_ranges (yr_scanner + yr_scanner_scan_mem_blocks over
zero-copy blocks; file offsets and filesize keep file semantics) with
EngineOptions{fast_mode}; elf_load_ranges() (PT_LOAD + first page, merged,
bounds-checked against hostile headers; empty -> scan everything).
- Exec guard: opt-in `guard_loaded_segments_only` (default off) scans only the
loaded segments of ELF images >= 4 MiB at launch; such results are marked
`partial`, never cached, and the pipeline still scans the whole file. Status
pipeline.guard_partial_scans. Off by default: an 8% faster launch decision is
not worth missing unloaded-section strings until the background scan.
- Fast mode is not adopted (1%, and it drops #count / @offset semantics).
Tests: ELF ranges (synthetic, BE, truncated, absurd phnum, phoff overflow, clamped
filesz, /bin/true), range scans (unmapped string unseen, at/uint32/filesize keep
file offsets, multi-block), guard partial scan catches mapped markers / misses
unmapped ones / pipeline full scan detects them / default off scans everything,
config round trip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- third_party/tracy.wrap pins the Tracy client to v0.15.0 (sha256-checked), matching
the Arch tracy 0.15.0 capture/export tools (client and tools must match exactly);
built with tracy_enable + on_demand, so nothing is recorded or paid for unless a
capture is connected. -Denable_profiling=true picks it up; default builds are
unchanged (macros are no-ops).
- scripts/profile_capture.sh: headless (no GUI, no root) `run OUT -- CMD` or
`attach OUT SECONDS` -> OUT.tracy + OUT.zones.csv + a top-zones table.
- strawberry-scan-bench (dev tool, not installed): the real scan_on_demand path over
files with the real governor and rule set, so the scan path can be profiled
without root or the daemon.
- Zones: scan_on_demand, image.read, image.sha256 (next to yara_scan/scan_image/
assess_exec), tray.* (poll, on_reply, refresh_evidence, update_status/_policy,
refresh_findings, notify) and evidence/database reloads.
Baseline (6 yams ASan binaries, 668 MiB, idle host): YARA 86%, read 9%, sha256 5%,
~61 MiB/s; PT_LOAD segments are 35-48% of those files. DEVELOPER.md: profiling
section, and the balanced row corrected (64 MiB image cap, 768 MiB budget).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 host hang: a yams ASan test run (~110 MiB binaries) pushed
strawberry.service past MemoryHigh=768M; the kernel throttled the daemon
(own memory.pressure full ~72%), so the exec guard stalled launches (589
1-second prevention timeouts in 80 min, even start-swayidle.sh), scans hit
p95 67 s and the control socket stopped answering.
- Exec guard (assess_exec) no longer copies an image before deciding: a cached
scan answers with no read; an image over guard_full_scan_max_bytes, or when
the scan-buffer budget is exhausted, is hashed by streaming (1 MiB chunks,
new HashUtils::sha256_md5_fd) and decided on hash / IOC / quarantine
(degraded; the pipeline still scans it). Only small budgeted images are copied.
- Scan-buffer budget (ResourceGovernor::acquire_scan_buffer, RAII
ScanBufferLease): in-memory image copies bounded to memory budget / 4
(balanced 192 MiB); the pipeline waits up to 30 s, on-demand scans 60 s;
an oversized request is granted only alone, so no deadlock or starvation.
- Balanced pipeline_max_image_bytes 128 -> 64 MiB: larger images get hash/IOC
checks only (as on the efficient profile).
- Response hashing streams instead of copying the whole executable.
- Status: resources.cgroup memory_high_mb / memory_high_events /
memory_full_pct / throttled, and resources.scan_buffers; doctor check
resources.memory (problem when throttled at MemoryHigh, with the fix).
Tests: budget grant/refuse/exclusive/wait/timeout/move, PSI full + memory.events
parsers, streaming hash = buffer hash across chunks, guard streams a 33 MiB
image with no YARA pass and no buffer grant, streams when the budget is
exhausted, full-scans + caches when budget is free, doctor throttle check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seen in the dockerized 3-sensor run: max_node_share=0.25 (pillar default) with
3 similar-sized nodes cut every node to 25% of the total, dropping 158/468
windows (34%) while correcting no imbalance. The cap is now
max(max_node_share, 1/N) * total, so a balanced fleet is never trimmed and
only nodes above an equal share are cut. Test: balanced 3-node fleet keeps
all windows; a dominant 4th node is still trimmed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The finite-difference vs analytic gradient check in test_gradients used a
2e-3 + 2e-2*|n2| tolerance that ubuntu-noble's FP codegen (FMA/libm) pushed a
borderline sample past, while debian-trixie and Arch passed. Deterministic
(fixed seed), so it blocked the v0.2.4 release build. Loosen to 5e-3 + 5e-2*|n2|
(still catches a real gradient bug, which misses by orders of magnitude).
Verified on ubuntu:24.04 in Docker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ship 3.5c's provenance attribution to the central learner. ExportLabel gains two
additive uint64 tags (8 origin_node_hash, 9 origin_exe_hash); a positive window's
label now carries token_hash(dropper node) + token_hash(path_template(dropper
exe)) from ReplayWindow.origin_*. fleet_merge counts windows_with_origin.
R12/R13: a FIELD addition, not a new record type -- no codec.hpp/enum change
(R12 untouched), both fields are uint64 hashes so no raw path/node string crosses
the export boundary (R13 green, no name-loophole). Back-compat additive:
kExportSchemaVersion stays 1; old bundles decode with origins 0; old readers
ignore tags 8/9. Positive labels only -- suppressed/negative carry no origin.
Tests: round-trip of the hash fields + defaults; extraction proves a dropper->exec
detected window's positive label carries a non-zero origin while a no-write
detected window and suppressed labels carry 0. source_invariants green.
A learned ranking head + TrainingExample plumbing are phase-2 (out of scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reuse the global write->exec linkage (recent_writes) to attribute a detection to
its ROOT CAUSE. recent_writes now maps path -> {writer_node, writer_exe}; when a
window is late_detected (retro-hunt flags a binary after it ran) or detected
(YARA), set ReplayWindow.origin_node/origin_exe from whoever WROTE the flagged
binary -- usually an earlier node than the exec leaf that ran it. Surfaced in
window_json. Deterministic provenance walk (first writer wins); replay-level only,
no export-schema/record-type/enum change (R12/R13 intact), self_written_execs
unchanged. A learned ranking head over ambiguous origins is the follow-up (3.5d).
Test R10: a dropper writes /tmp/evil in one window, a separate process execs it
and is flagged late -> the window's origin_node is the dropper, not the exec leaf;
no in-set write -> empty origin; deterministic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#4 (guard retention): same-binary parent/child self-ptrace never rates "high"
(it's self-debugging, not injection), but from a drop location (/tmp) it now
stays "medium" rather than "low" -- a same-binary anti-debug/injection staged in
/tmp still surfaces, while sanitizer builds no longer flood high-severity
PtraceAttach. Injection into another program is unchanged.
#5 (CLI-UX contract): --max-node-share/--test-fraction/--split now error when
passed to any command other than 'fleet train' instead of being silently
ignored (noun-verb commands already reject unknown options; this covers the
flag-form and other fleet subcommands).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address code-review findings on this session's changes (no behavior change):
- Share the dense-slot index for the write->exec feature as
learn::kWriteExecFeatureIndex; featurize() and the model-score dev tool now
reference it instead of a duplicated literal 42 (prevents silent column drift
if the feature layout shifts).
- Fix the self_written_execs comments: the linkage is a GLOBAL bounded FIFO
(recent_writes), not strictly "this window" — the docs now match replay.cpp.
- release.yml: never cancel a tag/release run (cancel-in-progress only for
non-tag refs); push ordering doesn't guarantee the tag event arrives after the
branch event, so a same-sha branch push must not abort an in-progress release.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The version lived in three hard-coded copies (CLI `version`, daemon status,
banner); v0.2.0 shipped with the CLI still saying 0.1.0. meson.build is now
the only source: it passes STRAWBERRY_VERSION to every target, and the CLI
test requires the exact version.
packaging/ci/build-release.sh refuses to publish a tag build unless
meson.build, the tag and the built binary's `strawberry version` agree; CI
never rewrites the version.
scripts/release.sh X.Y.Z is the one manual release step: on a clean main
equal to forgejo/main it bumps meson.build, makes a signed commit and a
signed tag, and pushes both (--dry-run shows the plan). The Security Hub
release channel deploys signed tags only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A parent/child of the SAME binary tracing each other is self-debugging
(sanitizers' StopTheWorld, crash handlers), not injection. The code already
scored that `low`, but `suspicious_image` (which flags drop locations like /tmp)
took precedence and forced `high` -- so sanitizer test binaries built in /tmp
flooded findings as high-severity PtraceAttach (observed live: ~160 FPs on
/tmp/toolcheck-*/*_san). Reorder so the strong structural self-debug signal beats
the weak location heuristic: same-program-family -> low regardless of location;
ptrace into ANOTHER program stays high (suspicious image) or medium. Injection
detection is unchanged. Test added for the /tmp self-ptrace case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build-only tool: loads a trained model (.sbm) and an export bundle, applies
forward(model, features) per window, prints TSV window_id/value/label/x[42] plus
a summary partitioned by the write->exec feature slot. Unblocks the novel-chain
benchmark: replay --model ranks by rarity (model_value null), so it could not
measure a model's score on holdout windows; this does. On the detector-covered
train bundle the data-flow model scores write->exec windows 0.634 vs 0.0003 for
the rest -- it clearly learned the x[42] feature.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3.5b's ordered exec-lineage proved inert in clean A/Bs; replace it with a general
write->exec data-flow feature: WindowSummary.self_written_execs (exec of a file
written earlier in the same window) at x[42], tracked via OpenWindow.written_paths.
Honest status: this is the conceptually-correct provenance feature (it separates
download->exec-the-same-file from download->exec-unrelated), but it ALSO shows no
AUC gain on detector-labeled data -- for the same structural reason as 3.5b: the
teacher labels (DownloadThenExecute etc.) correlate with existing count/location
features or re-derive the detector. Validating provenance LEARNING requires the
novel-chain generalization benchmark (train on detector-covered, hold out
detector-evading attacks); committed as that benchmark's feature foundation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feed ordered parent->child exe-template bigrams into the window feature vector
(WindowSummary.lineage_edges, note_lineage_edge, a salted hash loop in featurize,
wired at replay on_exec). The model already got per-kind lineage surprisal
(x[25..29]); this adds path ORDER, which the unordered token bag discarded.
Clean A/B on a fixed corpus (same 697 windows, same flags, only the binary
differs): model AUC 0.734 -> 0.737 (rarity 0.520 unchanged). Null on this
shallow-lineage data (marker execs are 1-process, synthetic data flat) but the
feature is correct and order-sensitive (test: A>B != B>A). Kept as the path
representation that Stage 3.5c's retro-hunt attribution head needs; its payoff is
expected on real multi-stage attack telemetry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expose the merge cap and train/test split on `strawberry fleet train` so a
single-node real-attack bundle can pre-seed a model. Default stays temporal +
cap 0.25 (behavior-preserving); `--split random` + `--max-node-share 1` lets the
Caldera-attack export (15 positive windows) train. Verified: real-only train AUC
1.0 vs rarity 0.563; default still refuses identically.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Do not turn an empty cached digest into an unbounded descriptor hash. Distinguish size refusals from YARA results so the exec guard applies its own bounded scan/degrade policy. Cover cached read budgets across size boundaries, known hashes, inventory/live events and real guard-cache reuse. Fold/live/KPI and post-rollout CPU acceptance remain pending; OpenSnitch checksum work is separate.
Use suspend-aware BOOTTIME waits and owned-container cleanup without changing the workload. Reject missing coverage and preserve producer/report failures. Include runtime/helper regressions, constrained build-job support and live exec-guard crash safety. Document the failed live/idle gates without relaxing thresholds or claiming production acceptance.
Preserve independent support, typed rule namespaces and stable process lifetimes. Share authoritative managed-set reconciliation across daemon and offline correlation, load offline policy without seeding/migration, and release resources after shutdown-flush exceptions. Keep retirement history, transaction ownership and fault/allocation regressions; no whole-set crash-atomicity or live acceptance claim.
Reuse the existing resolver/sampler for opt-in aggregate training summaries. Reject incomplete or undecodable snapshots without usable partial reports, retain best-effort readers elsewhere, and document cooperative extent checks rather than atomic snapshot guarantees. Include command-table/help, JSON, fixture and corruption regressions; no model training or promotion.