Commit graph

11 commits

Author SHA1 Message Date
trvon
db4c49155f
yara: rule partitions — scan each file only with the rules that can match it
Why YARA-X scanned ~1.6x faster than libyara (perf + source audit, 2026-11-14):
not the language — both spend ~5 s in the atom search loop — but verification
(libyara 2.0 s vs 0.3 s). YARA-X derives filesize bounds and header constraints
per rule at compile time and disables those patterns before the search, so their
atom hits are dropped with one bit test. In the yara-forge core pack 1787 rules
require an MZ header and 2498 bound filesize, so for big Linux ELFs most of
libyara's verification is wasted.

Rule partitioning gets the same win with libyara, and drops the rules' atoms from
the automaton entirely:
- src/rule_partition.cpp (pure): lexer-aware splitter (comments, text, regex
  after = or matches, hex), facts from TOP-LEVEL `and` terms only (MZ /
  pe.is_pe, ELF, Mach-O magic; filesize < / <=), exclusion per (MZ / non-MZ) x
  size bucket (<1, 1-4, 4-16, >=16 MiB). Global rules and rules referenced by
  kept rules always stay.
- YaraScanner builds subsets in a background thread after each load, in
  priority order (big non-PE first) under rule_partition_budget_pct (default
  100 ~ one extra full set: 3 subsets, +59 MiB; all 16 class x size subsets
  would cost +428 MiB). Cached under rules_cache_dir/partitions/. File, buffer
  and fd scans pick their subset; process-memory scans use the full set.
- Fails closed to the full rules: include directives, a split count that
  differs from libyara's, or a subset compile / count error.
- Status rules.partitions (state, reason, subsets, kept_rules/scans per
  partition, full_scans); config rule_partitions (default true) and
  rule_partition_budget_pct.

Measured on the 5132-rule pack: big ASan ELFs 1.64x, 1200 system binaries
1.59x, 0 match-set differences, including 348 positive files across
MZ/ELF/raw x small/5 MiB. Tests: splitter traps, facts, exclusion soundness
property, real pack = 5132 rules; differential test (24 file shapes x memory /
ranges / fd scans, every subset built) and fail-closed; bench --partitions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 06:58:39 -07:00
trvon
4586eed50d
yara engine: range/fast-mode scanning API, ELF load ranges, engine A/B bench, opt-in guard partial scan
Strawberry drives libyara (system 4.5.8), no custom engine. Profiling the scan path
(Tracy + user-space perf) shows 86% of scan time inside libyara's string-search loop;
our callback runs only on matches. Engine-usage variants measured with the new
`strawberry-scan-bench --engine` A/B over 3434 ELF files / 2.9 GB (5132 rules):
  full 50.1 MiB/s | fast mode 1.01x | loaded segments only 1.11x (2x on debug/ASan
  builds; 1.08x in the 4-32 MiB band the exec guard full-scans). 0 match-set
  differences, but the corpus has no rule matches, so parity is not proven.

- YaraScanner::scan_memory_ranges (yr_scanner + yr_scanner_scan_mem_blocks over
  zero-copy blocks; file offsets and filesize keep file semantics) with
  EngineOptions{fast_mode}; elf_load_ranges() (PT_LOAD + first page, merged,
  bounds-checked against hostile headers; empty -> scan everything).
- Exec guard: opt-in `guard_loaded_segments_only` (default off) scans only the
  loaded segments of ELF images >= 4 MiB at launch; such results are marked
  `partial`, never cached, and the pipeline still scans the whole file. Status
  pipeline.guard_partial_scans. Off by default: an 8% faster launch decision is
  not worth missing unloaded-section strings until the background scan.
- Fast mode is not adopted (1%, and it drops #count / @offset semantics).

Tests: ELF ranges (synthetic, BE, truncated, absurd phnum, phoff overflow, clamped
filesz, /bin/true), range scans (unmapped string unseen, at/uint32/filesize keep
file offsets, multi-block), guard partial scan catches mapped markers / misses
unmapped ones / pipeline full scan detects them / default off scans everything,
config round trip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:51:10 -07:00
trvon
8f4cb586b8
Rules R2: node-local compiled-rules cache keyed on libyara version + sources
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-12 06:54:28 -07:00
trvon
00af731088
Fleet E1: export egress dir + age/size retention (produce + prune synced bundles)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:14 -07:00
trvon
1ee208fb4b
Stage 7e: scheduled stop->reduce->start telemetry-reduction maintenance (opt-in)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:13 -07:00
trvon
69d86bbbc3
Rules R1: ship pinned yara-forge-core source snapshot in the release (no per-node fetch)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-11 13:21:12 -07:00
trvon
39912f6e64
Stage 6: wire config node_id to export (stable node identity; gap #2) 2026-11-08 09:05:25 -07:00
trvon
ad1676a594
Integrate features into the daemon, build and docs
Some checks failed
Build and Test / build (debug, clang) (push) Has been cancelled
Build and Test / build (debug, gcc) (push) Has been cancelled
Build and Test / build (release, clang) (push) Has been cancelled
Build and Test / build (release, gcc) (push) Has been cancelled
Build and Test / docker (push) Has been cancelled
Wire the policy, feeds, containment, exec guard, behavior, governor,
crash safety and event store into the daemon, detection pipeline, CLI
and meson build (zstd now required); Dockerfile, conan, systemd unit
(LimitCORE=0, resource limits) and installer updates; existing tests
adapted (store paths are directories now). Detections are counted and
notified even when evidence cannot be persisted (persist_failures in
status). prevention_timeout_ms is clamped and behavior dedupe handles
out-of-order events. Docs: API contract, developer guide, plan, UX,
agent rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 18:17:30 -06:00
trvon
577404a39b Prioritize process inventory collection 2026-04-27 19:48:20 -04:00
trvon
22a653f617 Implement runtime pipeline, reporting, and local UI 2026-04-26 16:31:01 -04:00
copilot-swe-agent[bot]
671898d80c Add complete C++ security scanner project structure
Co-authored-by: trvon <6031322+trvon@users.noreply.github.com>
2025-10-18 17:56:24 +00:00