fold/docs/architecture/optimization-loop.md

# Fold Optimization Loop

## Purpose

This is Fold's repeatable loop for performance, memory, and precision work.
It adapts the useful parts of agent-oriented optimization workflows to Fold's
existing architecture:

- project manifests and immutable `RunArtifact` records are the workload
  contract;
- profile attribution identifies a measurable owner before implementation;
- gold cases and reviewed dispositions distinguish precision evidence from
  unlabelled production findings;
- source and binary analysis remain independent backends;
- a rejected hypothesis is a recorded result, not a reason to weaken a gate.

The loop does not replace the focused test-first workflow for a detector or
symbolic module. It decides *which* change to make and what evidence is
required to retain it.

## Non-negotiable boundaries

1. **Keep analysis lanes separate.** Source changes belong in the Clang/Z3
   path; binary changes belong in the radare2/P-Code path. A hybrid run begins
   as merge-only and preserves per-finding backend provenance.
2. **Do not call an unlabelled production alert a TP or FP from a benchmark
   count alone.** A source review can classify emitted alerts, but TN and FN
   remain unmeasured without explicit clean and vulnerable cases.
3. **Do not optimize a bucket that is not attributed.** Use
   `optimization_surface` and profile counters first. Use Instruments or a
   targeted profiler only after the profile narrows the candidate phase.
4. **Do not replace an existing baseline to hide a regression.** A semantic
   change receives new reviewed evidence; an unchanged contract retains its
   finding-fingerprint gate.
5. **Do not ship an experiment because it looks plausible.** It needs a
   reproducible workload, an explicit accept/reject decision, and the
   validation matrix below.

## One iteration

### 1. Select one measurable opportunity

Use one of these signals:

- the ranked detector, source-phase, or graph-subphase tables from
  `scripts/benchmarks/run_project_eval.py`;
- a focused microbenchmark from `bench/run_micro_opt.py`;
- a reviewed precision defect from a gold corpus, Yams disposition, or
  source-level production finding review;
- a reliability failure: crash, timeout, finding churn, malformed evidence,
  or lost backend provenance.

State the proposed owner and expected evidence before touching code:

```text
Problem: use-after-free walks functions with no heap lifecycle operation.
Owner: source security-walker sink precheck.
Expected change: functions_skipped rises; use-after-free wall time falls.
Invariant: other detector identities and source coverage stay unchanged.
```

Reject "the scan is slow" or "this warning feels wrong" as an iteration
definition. Those are symptoms, not an attributable decision point.

### 2. Pin the workload and baseline

Use a checked-in manifest whenever the workload is a real project:

```sh
python3 scripts/benchmarks/run_project_eval.py \
  --manifest tests/benchmarks/projects/yams_source.json \
  --out-dir /tmp/fold-opt-before
```

For a bounded production canary, use its manifest rather than a hand-filtered
compile database. Chromium is intentionally an opt-in, eight-TU canary:

```sh
python3 scripts/benchmarks/run_project_eval.py \
  --manifest tests/benchmarks/projects/chromium_source.json \
  --out-dir /tmp/fold-chromium-before
```

Use three or more repeats for a claimed real-project speedup. For small
changes close to ordinary machine variance, use `bench/perf_audit_run.py`
with at least five repetitions and compare median and IQR. Preserve the raw
`summary.json`, SARIF, profile, logs, target-set fingerprint, revision, and
configuration fingerprint.

The baseline and candidate must select the same target set. They should use
the same configuration for behavior-preserving changes. If a temporary CLI
flag changes configuration, record that fact and treat the A/B result as
experimental evidence only; it cannot replace the same-configuration
regression baseline.

### 3. Write an experiment contract

Record the contract beside the run artifacts or in the change description:

```json
{
  "schema_version": 1,
  "iteration": "use-after-free-sink-precheck",
  "lane": "source",
  "hypothesis": "Skip symbolic UAF walking when no free/delete sink exists.",
  "owner": "security-walker sink precheck",
  "workload": "tests/benchmarks/projects/chromium_source.json",
  "baseline": "/tmp/fold-chromium-before/summary.json",
  "expected": {
    "decreases": ["detectors.use-after-free.wall_ms", "peak_rss_mb"],
    "increases": ["detectors.use-after-free.functions_skipped"],
    "preserves": ["finding identity", "source_tus_completed"]
  },
  "precision_policy": "unchanged-findings",
  "decision": "pending"
}
```

For a precision iteration, name the exact positive and clean-negative cases
that will be added. For a new library/API model, include its documented
success/failure contract and the specific expression it constrains. Do not add
path- or repository-specific suppression for a reviewed false positive.

### 4. Implement one reversible change

Keep the change scoped to the named owner. New costly behavior starts
default-off when it changes the analysis strategy or is not yet benchmarked.
An experiment becomes default-on only after the ablation succeeds. Existing
flags and manifest settings are Fold's option surface; do not add a generic
configuration layer merely to run an experiment.

Every generalized precision rule needs:

- a positive test proving the intended report still fires;
- a clean or guarded negative test proving the false positive is suppressed;
- an appropriate project or microbenchmark run when the rule affects shared
  symbolic traversal.

Every performance rule needs a counter or profile phase that can demonstrate
why it helped. If no existing counter describes the expected change, add the
narrowest useful counter before claiming a speedup.

### 5. Run the evidence matrix

Choose the smallest relevant commands first. Source detector, backend, CLI,
or router changes require the full release build and C++ suite afterwards.

| Change | Required evidence |
| --- | --- |
| Source detector or encoder | Focused unit test with positive/negative cases; `ninja -C build/release`; `meson test -C build/release --print-errorlogs`; source metamorphic and sanitizer/Clang-SA differential coverage when applicable |
| Binary backend | Focused binary test; binary manifest or curl binary smoke; backend provenance remains `binary` |
| Router or hybrid | Source and binary focused tests; full suite; source and binary smoke; merge-only identities remain independently explainable |
| Performance-only | Baseline/candidate project summaries, profile attribution, finding stability, peak RSS, and the relevant microbenchmark |
| Precision change | Explicit gold or reviewed disposition delta; TP/FP/FN/TN only where the corpus labels each quantity; an explanation for every identity change |

For CLI, router, or backend work, also run:

```sh
ninja -C build/release
meson test -C build/release --print-errorlogs
git diff --check
```

Use `scripts/benchmarks/run_optimization_harness.py` when one compact
artifact is more useful than inspecting each project run:

```sh
python3 scripts/benchmarks/run_optimization_harness.py \
  --manifest tests/benchmarks/projects/yams_source.json \
  --out-dir /tmp/fold-optimization
```

### 6. Decide with ablation

| Candidate result | Decision |
| --- | --- |
| Findings, coverage, reliability, and RSS are stable; the profiled owner improves beyond observed variance | Keep it default-on and preserve the before/after artifact |
| Precision improves on explicit labels with no required positive loss | Keep it; update the reviewed gold/disposition baseline only with the written rationale |
| Improvement occurs only with an opt-in setting | Retain it default-off; document the use case and do not claim a default improvement |
| Runtime improves but finding identity, operational coverage, or backend provenance regresses unexpectedly | Reject or repair; do not trade silent coverage loss for speed |
| The result is within run-to-run variance, the profile owner did not move, or the hypothesis cannot be reproduced | Reject and record the negative result |

Never fabricate TN or FN as zero. Chromium's reviewed findings can measure
reviewed emitted-alert precision only after they are committed as exact
negative cases; they do not constitute a recall benchmark until positive
oracles exist.

### 7. Preserve the result and choose the next loop

The conclusion names:

- baseline and candidate artifact paths;
- target/configuration fingerprints and corpus revision;
- wall-time and RSS deltas with repeat count and comparison method;
- changed profile counters;
- finding identity, churn, and operational coverage;
- TP/FP/FN/TN only if explicitly measured;
- default decision and the next attributable bottleneck.

Promote durable evidence into the closest stable surface:

- a unit or focused detector test for analysis semantics;
- a microbenchmark for a local performance mechanism;
- a project manifest/baseline for real-project behavior;
- a gold/disposition entry for reviewed precision;
- a research or architecture decision record only when the behavior or
  boundary changes.

## Fold-specific priority order

Use this order to avoid optimizing the wrong layer:

1. Remove repeated work and reuse already-proven facts.
2. Skip functions or sinks that cannot exercise an enabled detector.
3. Improve cheap abstract proofs before changing solver tactics.
4. Attribute graph merge/SCC/RSS cost before replacing graph containers.
5. Propose semantic-model changes only when the profile shows symbolic control
   walking or path state is the actual bottleneck.

This order follows the current performance audit and keeps precision work
separate from raw container or solver speculation.

## Current Chromium precision campaign

Chromium is a production canary, not a complete oracle. The ten reviewed
findings are evidence for prioritized model gaps: path-sensitive string-index
relationships, fixed-table bounds, macro-range reasoning, API output
contracts, and local pointer provenance. Each future fix must be generalized,
tested with a positive and negative case, and measured against the canary.

Do not tune against Chromium source locations. The only acceptable
Chromium-specific artifact is an exact reviewed disposition or gold entry used
to measure a generalized change.