Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.
Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
priority order (big non-PE first) under rule_partition_budget_pct (default
100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
partition, full_scans); config rule_partitions (default true) and
rule_partition_budget_pct.
Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strawberry drives libyara (system 4.5.8), no custom engine. Profiling the scan path
(Tracy + user-space perf) shows 86% of scan time inside libyara's string-search loop;
our callback runs only on matches. Engine-usage variants measured with the new
`strawberry-scan-bench --engine` A/B over 3434 ELF files / 2.9 GB (5132 rules):
full 50.1 MiB/s | fast mode 1.01x | loaded segments only 1.11x (2x on debug/ASan
builds; 1.08x in the 4-32 MiB band the exec guard full-scans). 0 match-set
differences, but the corpus has no rule matches, so parity is not proven.
- YaraScanner::scan_memory_ranges (yr_scanner + yr_scanner_scan_mem_blocks over
zero-copy blocks; file offsets and filesize keep file semantics) with
EngineOptions{fast_mode}; elf_load_ranges() (PT_LOAD + first page, merged,
bounds-checked against hostile headers; empty -> scan everything).
- Exec guard: opt-in `guard_loaded_segments_only` (default off) scans only the
loaded segments of ELF images >= 4 MiB at launch; such results are marked
`partial`, never cached, and the pipeline still scans the whole file. Status
pipeline.guard_partial_scans. Off by default: an 8% faster launch decision is
not worth missing unloaded-section strings until the background scan.
- Fast mode is not adopted (1%, and it drops #count / @offset semantics).
Tests: ELF ranges (synthetic, BE, truncated, absurd phnum, phoff overflow, clamped
filesz, /bin/true), range scans (unmapped string unseen, at/uint32/filesize keep
file offsets, multi-block), guard partial scan catches mapped markers / misses
unmapped ones / pipeline full scan detects them / default off scans everything,
config round trip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--inject and --inject-train are now repeatable (training.trace plus
threats-train.trace, standard.trace plus threats-eval.trace). Trace names
must be unique across the files.
standard.trace labelled its autostart write "XDG autostart", but the file
watcher emits "desktop autostart", so that write never matched what a
real host records. No gate evaluation has used standard.trace yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`model train` on a host without detections stops with "0 positive
windows" and pointed at --inject-train FILE, but only the evaluation set
(standard.trace) existed. Add config/traces/training.trace: 7 attack
shapes (document macro dropper, web reverse shell, cron miner, /dev/shm
perl stager, ssh key backdoor, npm postinstall, ld.so.preload) that differ
from the evaluation set, so the gate never judges a model on what it
trained on. P4 in test_trace enforces that (no shared trace name or exec
sequence); it caught an evince -> python3 overlap in the first draft.
Both installers put the traces in /usr/local/share/strawberry/traces/
(install_traces in install_common.sh); the hint and README name it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/install.sh hands off to scripts/macos_install.sh on Darwin (it
failed on getent/groupadd) and runs from the repo root. `make install`
runs it on either OS.
Signing: an ad hoc signature carrying the ES entitlement is killed at
launch while SIP is on, and the hardened runtime rejected the Homebrew
dylibs (different Team ID). Both entitlement files now disable library
validation; with SIP on and IDENTITY=- the installer signs with
strawberry-base.entitlements (libproc fallback), test-launches the
binary and re-signs without ES if it is killed. `make macos-sign`
refuses the combination that would be killed.
The installer no longer sets enable_bpf_monitoring=false, which turned
off all process monitoring; it sets db_path explicitly. The rule
updater runs as the new _strawberry user, which owns only feeds/, and
starts on update-requested via KeepAlive PathState (WatchPaths also
fired on removal). launchd jobs use `daemon run` / `rules fetch`.
Parity with Linux: completions, the installed-files manifest,
evidence permissions, group offer. Shared helpers in
scripts/install_common.sh. MCP registration moved to the README.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daemon now scores live windows with the shadow model and logs the
inspection it would request. Nothing is executed.
- learn/shadow: a live replay engine (same windows and rarity as
`strawberry replay`) fed every telemetry record on the writer thread.
When a window closes, the model's value v gives each action a benefit
v x reach - lambda x CPU cost; the best positive one is chosen if the
hour's budget (10/30/120 CPU-s by scan profile) allows, else none.
With probability 0.02 a uniformly random other action is taken instead,
and every decision records its propensity for off-policy evaluation.
Decisions are inspection_action telemetry records.
- Replay gains a live mode (no ranking bookkeeping, so memory stays
bounded) and a window-closed callback.
- Daemon: model_shadow (default on) loads <model_dir>/shadow.sbm when it
appears or changes, checked every minute; an invalid file is reported
and not used. Status gains a `model` block (kernel, generation,
decisions by action, exploration, would-be spend, budget refusals).
- learn_sources in src/meson.build: tests that compile the daemon from
sources take the learning library whole.
Limits, documented: decisions come when a window closes (a few minutes
after its first exec), and the live rarity model starts empty at each
daemon start (the first hour is learned, not decided).
Tests: test_shadow (no model, greedy choices, budget never exceeded,
determinism, exploration rate and propensities, record codec); daemon
status; fuzz_telemetry_records covers inspection_action.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The main source of teacher labels over time: rules improve daily, while
most malware runs once.
- The pipeline remembers every binary that ran (image_seen evidence
record: sha256, path, first/last seen, size; at most one write per hash
a day). Deleted and memfd images are skipped: nothing to find again.
- Each rules load computes a digest of the rule files and IOC hash lists
(paths after resolving symlinks, sizes, mtimes).
- RetroHunter (src/retro_hunt.cpp, own thread, idle priority, paced at
retro_hunt_pace_ms, deferred by the governor as WorkClass::retro_hunt)
re-scans when the digest differs from the last completed pass, most
recently seen first. Only paths that still hold the same content are
scanned (O_NOFOLLOW, regular files, hash compared); others count as
changed or missing. A completed pass is recorded (retro_hunt record) so
a restart does not repeat it; an interrupted one starts over.
- New matches (rules not recorded before for that hash, not allowlisted)
become indicators with scan_kind retro_hunt, a file/indicator graph
link, a positive label and a notification. Correlation turns them into
findings titled "YARA match on a binary that ran here (found after a
rules update)". Nothing is blocked or contained from here.
- Replay reads positive labels from the evidence stream and marks windows
that ran a binary labelled bad later (LATE in the CLI).
- Status: a retro_hunt block (API_CONTRACT.md); config: retro_hunt,
retro_hunt_pace_ms.
Tests: test_retro_hunt (remembered once; passes only on a digest change
and survive a restart; a clean-at-exec binary matching a new rule becomes
an indicator, label, finding and notification; a replaced binary is
counted as changed, never scanned as the old one; a stop abandons a pass
unrecorded); test_replay R9 (late labels, from a real store).
fuzz_evidence_records covers the new records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- src/learn/trace.cpp: a small trace format (exec, connect, write,
ptrace, module with +offsets and process aliases) compiled to
telemetry records with pids above any pid_max and node ids that name
the trace and copy.
- --replay --inject FILE [--inject-copies N]: copies are spread evenly
over the ranked history (a first pass reads the time range), scored
but never learned from, and checked against alert budgets of 1, 3 and
10 per day: is the copy's best window above the (budget x days)-th best
real window? Ties count against. Injection leaves real windows and
their ranking unchanged.
- config/traces/standard.trace: the six plan traces (browser to curl|sh,
systemd user persistence, memfd loader, ptrace injection, download
then exec, python from a PDF viewer).
- docs/evaluation/stage3-rarity-preregistration.md: method and pass
thresholds, written before any run on host data.
Tests: test_trace (standard traces, error messages with line numbers,
records); test_replay R8 (every copy found, real ranking unchanged, a
novel chain caught at 10/day and a bare shell not). fuzz_trace_dsl.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 1 increment 7.
- Disk floor: below max(store_min_free_mb, store_min_free_pct) free,
telemetry is refused and counted (a gap record with the count and
time range is written once space returns) or, with
store_floor_action=evict_oldest_telemetry, the oldest sealed
telemetry is deleted; evidence continues until store_hard_min_free_mb.
- Retention (telemetry only, off by default): store_retention_days and
store_max_mb, applied when a segment is sealed. Evidence is never
deleted by retention.
- Status gains a store block (floor, free space, per-stream sizes,
compression ratio, drops, deletions); health reports a low disk,
a failed evidence store and unsaved detections.
- strawberry --store-verify checks the manifest and every segment
(read-only, safe while the daemon runs; root-only telemetry is noted,
not flagged). strawberry --store-compact rewrites sealed segments at
zstd level 19 with the daemon stopped, never touching a damaged one.
- The manifest records each sealed segment's uncompressed size.
Tests L7-L11 cover the floor, gaps, retention by age and size,
eviction, compaction and verify.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Triage of all 26 findings on a developer workstation (none malicious)
is in docs/false-positives.md: binaries replaced while running
(auto-updates, reinstalls), Strawberry's own installer writing its
systemd units, sanitizer/self-debugging ptrace, deliberate tests, and
the test marker matching process memory.
Fixes:
- behavior detections now honor path and rule allowlist entries (they
only checked sha256, which is computed for high severity only)
- finding summaries include what happened (file written, ptrace
target) and metadata.details
- Strawberry_Test_Marker only matches files that start with the marker
(EICAR-style); it no longer fires on processes holding the text
- ptrace between a parent and child running the same binary
(LeakSanitizer StopTheWorld) is low instead of medium; ProcessRef
carries the parent pid
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wire the policy, feeds, containment, exec guard, behavior, governor,
crash safety and event store into the daemon, detection pipeline, CLI
and meson build (zstd now required); Dockerfile, conan, systemd unit
(LimitCORE=0, resource limits) and installer updates; existing tests
adapted (store paths are directories now). Detections are counted and
notified even when evidence cannot be persisted (persist_failures in
status). prevention_timeout_ms is clamped and behavior dedupe handles
out-of-order events. Docs: API contract, developer guide, plan, UX,
agent rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the browser UI, its Python server and the web API contract test.
Add the Qt6 tray app (status, findings, process search, policy
controls, notifications) and EvidenceStore, the read-only view of the
daemon's evidence that the app and MCP server share. The GUI demo
config becomes strawberry.demo.conf.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>