Commit graph

1 commit

Author SHA1 Message Date
trvon
db4c49155f
yara: rule partitions — scan each file only with the rules that can match it
Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.

Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
  after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
  pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
  size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
  kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
  priority order (big non-PE first) under rule_partition_budget_pct (default
  100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
  would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
  and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
  differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
  partition, full_scans); config rule_partitions (default true) and
  rule_partition_budget_pct.

Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:58:39 -07:00