73 lines
3.5 KiB
Markdown
73 lines
3.5 KiB
Markdown
# Optimization micro-benchmarks
|
|
|
|
`bench/run_micro_opt.py` is the synthetic A/B harness for the current speed and
|
|
false-positive reduction backlog. It generates tiny C++ projects, runs `fold scan
|
|
--profile`, and writes per-case SARIF/profile artifacts plus `summary.json` and
|
|
`summary.md`.
|
|
|
|
Use it before and after each optimization patch:
|
|
|
|
```sh
|
|
python3 bench/run_micro_opt.py --case all --repeats 3
|
|
# compare medians in bench/results/<timestamp>_micro-opt/summary.json
|
|
```
|
|
|
|
For quick smoke while iterating:
|
|
|
|
```sh
|
|
python3 bench/run_micro_opt.py --case all --size 4 --repeats 1 --out-dir /tmp/fold-micro-smoke
|
|
```
|
|
|
|
## Cases
|
|
|
|
| Case | Optimization validated | Main metrics to compare | Invariant |
|
|
|---|---|---|---|
|
|
| `branch-merge` | `SecurityWalker::walkIf` / `SymbolicEnv` dirty-binding or cheaper branch join logic | `null-deref.wall_ms`, `states_merged`, `states_kept_separate`, `serial_dispatch_ms` | finding count and confidence distribution stay stable |
|
|
| `defuse-null` | direct `DefUseChains` lookup by `(var, use_loc)` in `NullDerefObserver::shouldQuery` | `null-deref.wall_ms`, `z3_queries`, external elapsed | finding count stays stable |
|
|
| `cross-tu-prime` | skipping/deduplicating redundant per-TU cross-TU priming after global priming | `serial_dispatch_ms`, `total_wall_ms`, findings by rule | cross-TU callsite findings stay stable |
|
|
| `precision-fp` | shared precision evidence / confidence demotion for imprecise security findings | `by_confidence`, detector finding counts | exact findings stay; imprecise findings move from `proved` to `heuristic` where appropriate |
|
|
| `expr-cache` | query-hash/cache changes, especially avoiding expensive canonical string hashing | detector `wall_ms`, `z3_queries`, `z3_timeouts` | findings and confidence distribution stay stable |
|
|
| `curl-str-bound` | Curl-style bounded parser output (`curlx_str_*`) feeding `curlx_strlen(&out)` into a stack buffer terminator index | `oob.findings_emitted`, `oob.wall_ms`, `z3_queries` | max+1 buffer terminator writes produce 0 findings while undersized-buffer unit tests still flag |
|
|
|
|
## Suggested acceptance gates
|
|
|
|
Do not use absolute milliseconds as hard gates across machines. Prefer comparing
|
|
a before/after run on the same machine/build:
|
|
|
|
- Performance patches: median target metric should improve by at least 10% on the
|
|
relevant case, with no new findings and no lost findings unless the patch is
|
|
explicitly an FP reduction.
|
|
- FP/confidence patches: total findings may stay the same, but confidence
|
|
distribution should move in the intended direction; exact/proved positives in
|
|
the smoke cases must not disappear.
|
|
- Profile/instrumentation patches: new metrics should appear in profile JSON, and
|
|
existing metrics should remain populated.
|
|
|
|
Each repeat writes raw artifacts:
|
|
|
|
```text
|
|
<case>-<size>-r<N>.sarif
|
|
<case>-<size>-r<N>.profile.json
|
|
<case>-<size>-r<N>.stdout.log
|
|
<case>-<size>-r<N>.stderr.log
|
|
```
|
|
|
|
Use `--keep-workdir` when you want to inspect generated source and compile DBs.
|
|
|
|
## Real-code presets
|
|
|
|
`bench/run_security_targets.py` runs the current source-security presets:
|
|
|
|
```sh
|
|
python3 bench/run_security_targets.py --preset yams-wide20-security --repeats 3
|
|
python3 bench/run_security_targets.py --preset all --repeats 1
|
|
```
|
|
|
|
Presets skip themselves when the local compile database is missing. Use
|
|
`yams-wide20-security` as the quick real-code gate, then `yams-full-security` and
|
|
`curl-security` for slower validation.
|
|
|
|
## Added case: `control-nesting`
|
|
|
|
`control-nesting` stresses nested branches/loops and should be used for changes
|
|
that target `control_walk_ms`, sink prechecks, or branch pruning counters.
|