Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.
Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
priority order (big non-PE first) under rule_partition_budget_pct (default
100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
partition, full_scans); config rule_partitions (default true) and
rule_partition_budget_pct.
Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strawberry drives libyara (system 4.5.8), no custom engine. Profiling the scan path
(Tracy + user-space perf) shows 86% of scan time inside libyara's string-search loop;
our callback runs only on matches. Engine-usage variants measured with the new
`strawberry-scan-bench --engine` A/B over 3434 ELF files / 2.9 GB (5132 rules):
full 50.1 MiB/s | fast mode 1.01x | loaded segments only 1.11x (2x on debug/ASan
builds; 1.08x in the 4-32 MiB band the exec guard full-scans). 0 match-set
differences, but the corpus has no rule matches, so parity is not proven.
- YaraScanner::scan_memory_ranges (yr_scanner + yr_scanner_scan_mem_blocks over
zero-copy blocks; file offsets and filesize keep file semantics) with
EngineOptions{fast_mode}; elf_load_ranges() (PT_LOAD + first page, merged,
bounds-checked against hostile headers; empty -> scan everything).
- Exec guard: opt-in `guard_loaded_segments_only` (default off) scans only the
loaded segments of ELF images >= 4 MiB at launch; such results are marked
`partial`, never cached, and the pipeline still scans the whole file. Status
pipeline.guard_partial_scans. Off by default: an 8% faster launch decision is
not worth missing unloaded-section strings until the background scan.
- Fast mode is not adopted (1%, and it drops #count / @offset semantics).
Tests: ELF ranges (synthetic, BE, truncated, absurd phnum, phoff overflow, clamped
filesz, /bin/true), range scans (unmapped string unseen, at/uint32/filesize keep
file offsets, multi-block), guard partial scan catches mapped markers / misses
unmapped ones / pipeline full scan detects them / default off scans everything,
config round trip.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- third_party/tracy.wrap pins the Tracy client to v0.15.0 (sha256-checked), matching
the Arch tracy 0.15.0 capture/export tools (client and tools must match exactly);
built with tracy_enable + on_demand, so nothing is recorded or paid for unless a
capture is connected. -Denable_profiling=true picks it up; default builds are
unchanged (macros are no-ops).
- scripts/profile_capture.sh: headless (no GUI, no root) `run OUT -- CMD` or
`attach OUT SECONDS` -> OUT.tracy + OUT.zones.csv + a top-zones table.
- strawberry-scan-bench (dev tool, not installed): the real scan_on_demand path over
files with the real governor and rule set, so the scan path can be profiled
without root or the daemon.
- Zones: scan_on_demand, image.read, image.sha256 (next to yara_scan/scan_image/
assess_exec), tray.* (poll, on_reply, refresh_evidence, update_status/_policy,
refresh_findings, notify) and evidence/database reloads.
Baseline (6 yams ASan binaries, 668 MiB, idle host): YARA 86%, read 9%, sha256 5%,
~61 MiB/s; PT_LOAD segments are 35-48% of those files. DEVELOPER.md: profiling
section, and the balanced row corrected (64 MiB image cap, 768 MiB budget).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>