fold/docs/internal/perf-audit/micro-benchmarks.md

Optimization micro-benchmarks

bench/run_micro_opt.py is the synthetic A/B harness for the current speed and false-positive reduction backlog. It generates tiny C++ projects, runs fold scan --profile, and writes per-case SARIF/profile artifacts plus summary.json and summary.md.

Use it before and after each optimization patch:

python3 bench/run_micro_opt.py --case all --repeats 3
# compare medians in bench/results/<timestamp>_micro-opt/summary.json

For quick smoke while iterating:

python3 bench/run_micro_opt.py --case all --size 4 --repeats 1 --out-dir /tmp/fold-micro-smoke

Cases

Case	Optimization validated	Main metrics to compare	Invariant
branch-merge	SecurityWalker::walkIf / SymbolicEnv dirty-binding or cheaper branch join logic	null-deref.wall_ms, states_merged, states_kept_separate, serial_dispatch_ms	finding count and confidence distribution stay stable
defuse-null	direct DefUseChains lookup by (var, use_loc) in NullDerefObserver::shouldQuery	null-deref.wall_ms, z3_queries, external elapsed	finding count stays stable
cross-tu-prime	skipping/deduplicating redundant per-TU cross-TU priming after global priming	serial_dispatch_ms, total_wall_ms, findings by rule	cross-TU callsite findings stay stable
precision-fp	shared precision evidence / confidence demotion for imprecise security findings	by_confidence, detector finding counts	exact findings stay; imprecise findings move from proved to heuristic where appropriate
expr-cache	query-hash/cache changes, especially avoiding expensive canonical string hashing	detector wall_ms, z3_queries, z3_timeouts	findings and confidence distribution stay stable
curl-str-bound	Curl-style bounded parser output (curlx_str_*) feeding curlx_strlen(&out) into a stack buffer terminator index	oob.findings_emitted, oob.wall_ms, z3_queries	max+1 buffer terminator writes produce 0 findings while undersized-buffer unit tests still flag

Suggested acceptance gates

Do not use absolute milliseconds as hard gates across machines. Prefer comparing a before/after run on the same machine/build:

• Performance patches: median target metric should improve by at least 10% on the relevant case, with no new findings and no lost findings unless the patch is explicitly an FP reduction.
• FP/confidence patches: total findings may stay the same, but confidence distribution should move in the intended direction; exact/proved positives in the smoke cases must not disappear.
• Profile/instrumentation patches: new metrics should appear in profile JSON, and existing metrics should remain populated.

Each repeat writes raw artifacts:

<case>-<size>-r<N>.sarif
<case>-<size>-r<N>.profile.json
<case>-<size>-r<N>.stdout.log
<case>-<size>-r<N>.stderr.log

Use --keep-workdir when you want to inspect generated source and compile DBs.

Real-code presets

bench/run_security_targets.py runs the current source-security presets:

python3 bench/run_security_targets.py --preset yams-wide20-security --repeats 3
python3 bench/run_security_targets.py --preset all --repeats 1

Presets skip themselves when the local compile database is missing. Use yams-wide20-security as the quick real-code gate, then yams-full-security and curl-security for slower validation.

Added case: control-nesting

control-nesting stresses nested branches/loops and should be used for changes that target control_walk_ms, sink prechecks, or branch pruning counters.