fold/docs/architecture/optimization-loop.md

Fold Optimization Loop

Purpose

This is Fold's repeatable loop for performance, memory, and precision work. It adapts the useful parts of agent-oriented optimization workflows to Fold's existing architecture:

• project manifests and immutable RunArtifact records are the workload contract;
• profile attribution identifies a measurable owner before implementation;
• gold cases and reviewed dispositions distinguish precision evidence from unlabelled production findings;
• source and binary analysis remain independent backends;
• a rejected hypothesis is a recorded result, not a reason to weaken a gate.

The loop does not replace the focused test-first workflow for a detector or symbolic module. It decides which change to make and what evidence is required to retain it.

Non-negotiable boundaries

1. Keep analysis lanes separate. Source changes belong in the Clang/Z3 path; binary changes belong in the radare2/P-Code path. A hybrid run begins as merge-only and preserves per-finding backend provenance.
2. Do not call an unlabelled production alert a TP or FP from a benchmark count alone. A source review can classify emitted alerts, but TN and FN remain unmeasured without explicit clean and vulnerable cases.
3. Do not optimize a bucket that is not attributed. Use optimization_surface and profile counters first. Use Instruments or a targeted profiler only after the profile narrows the candidate phase.
4. Do not replace an existing baseline to hide a regression. A semantic change receives new reviewed evidence; an unchanged contract retains its finding-fingerprint gate.
5. Do not ship an experiment because it looks plausible. It needs a reproducible workload, an explicit accept/reject decision, and the validation matrix below.

One iteration

1. Select one measurable opportunity

Use one of these signals:

• the ranked detector, source-phase, or graph-subphase tables from scripts/benchmarks/run_project_eval.py;
• a focused microbenchmark from bench/run_micro_opt.py;
• a reviewed precision defect from a gold corpus, Yams disposition, or source-level production finding review;
• a reliability failure: crash, timeout, finding churn, malformed evidence, or lost backend provenance.

State the proposed owner and expected evidence before touching code:

Problem: use-after-free walks functions with no heap lifecycle operation.
Owner: source security-walker sink precheck.
Expected change: functions_skipped rises; use-after-free wall time falls.
Invariant: other detector identities and source coverage stay unchanged.

Reject "the scan is slow" or "this warning feels wrong" as an iteration definition. Those are symptoms, not an attributable decision point.

2. Pin the workload and baseline

Use a checked-in manifest whenever the workload is a real project:

python3 scripts/benchmarks/run_project_eval.py \
  --manifest tests/benchmarks/projects/yams_source.json \
  --out-dir /tmp/fold-opt-before

For a bounded production canary, use its manifest rather than a hand-filtered compile database. Chromium is intentionally an opt-in, eight-TU canary:

python3 scripts/benchmarks/run_project_eval.py \
  --manifest tests/benchmarks/projects/chromium_source.json \
  --out-dir /tmp/fold-chromium-before

Use three or more repeats for a claimed real-project speedup. For small changes close to ordinary machine variance, use bench/perf_audit_run.py with at least five repetitions and compare median and IQR. Preserve the raw summary.json, SARIF, profile, logs, target-set fingerprint, revision, and configuration fingerprint.

The baseline and candidate must select the same target set. They should use the same configuration for behavior-preserving changes. If a temporary CLI flag changes configuration, record that fact and treat the A/B result as experimental evidence only; it cannot replace the same-configuration regression baseline.

3. Write an experiment contract

Record the contract beside the run artifacts or in the change description:

{
  "schema_version": 1,
  "iteration": "use-after-free-sink-precheck",
  "lane": "source",
  "hypothesis": "Skip symbolic UAF walking when no free/delete sink exists.",
  "owner": "security-walker sink precheck",
  "workload": "tests/benchmarks/projects/chromium_source.json",
  "baseline": "/tmp/fold-chromium-before/summary.json",
  "expected": {
    "decreases": ["detectors.use-after-free.wall_ms", "peak_rss_mb"],
    "increases": ["detectors.use-after-free.functions_skipped"],
    "preserves": ["finding identity", "source_tus_completed"]
  },
  "precision_policy": "unchanged-findings",
  "decision": "pending"
}

For a precision iteration, name the exact positive and clean-negative cases that will be added. For a new library/API model, include its documented success/failure contract and the specific expression it constrains. Do not add path- or repository-specific suppression for a reviewed false positive.

4. Implement one reversible change

Keep the change scoped to the named owner. New costly behavior starts default-off when it changes the analysis strategy or is not yet benchmarked. An experiment becomes default-on only after the ablation succeeds. Existing flags and manifest settings are Fold's option surface; do not add a generic configuration layer merely to run an experiment.

Every generalized precision rule needs:

• a positive test proving the intended report still fires;
• a clean or guarded negative test proving the false positive is suppressed;
• an appropriate project or microbenchmark run when the rule affects shared symbolic traversal.

Every performance rule needs a counter or profile phase that can demonstrate why it helped. If no existing counter describes the expected change, add the narrowest useful counter before claiming a speedup.

5. Run the evidence matrix

Choose the smallest relevant commands first. Source detector, backend, CLI, or router changes require the full release build and C++ suite afterwards.

Change	Required evidence
Source detector or encoder	Focused unit test with positive/negative cases; ninja -C build/release; meson test -C build/release --print-errorlogs; source metamorphic and sanitizer/Clang-SA differential coverage when applicable
Binary backend	Focused binary test; binary manifest or curl binary smoke; backend provenance remains binary
Router or hybrid	Source and binary focused tests; full suite; source and binary smoke; merge-only identities remain independently explainable
Performance-only	Baseline/candidate project summaries, profile attribution, finding stability, peak RSS, and the relevant microbenchmark
Precision change	Explicit gold or reviewed disposition delta; TP/FP/FN/TN only where the corpus labels each quantity; an explanation for every identity change

For CLI, router, or backend work, also run:

ninja -C build/release
meson test -C build/release --print-errorlogs
git diff --check

Use scripts/benchmarks/run_optimization_harness.py when one compact artifact is more useful than inspecting each project run:

python3 scripts/benchmarks/run_optimization_harness.py \
  --manifest tests/benchmarks/projects/yams_source.json \
  --out-dir /tmp/fold-optimization

6. Decide with ablation

Candidate result	Decision
Findings, coverage, reliability, and RSS are stable; the profiled owner improves beyond observed variance	Keep it default-on and preserve the before/after artifact
Precision improves on explicit labels with no required positive loss	Keep it; update the reviewed gold/disposition baseline only with the written rationale
Improvement occurs only with an opt-in setting	Retain it default-off; document the use case and do not claim a default improvement
Runtime improves but finding identity, operational coverage, or backend provenance regresses unexpectedly	Reject or repair; do not trade silent coverage loss for speed
The result is within run-to-run variance, the profile owner did not move, or the hypothesis cannot be reproduced	Reject and record the negative result

Never fabricate TN or FN as zero. Chromium's reviewed findings can measure reviewed emitted-alert precision only after they are committed as exact negative cases; they do not constitute a recall benchmark until positive oracles exist.

7. Preserve the result and choose the next loop

The conclusion names:

• baseline and candidate artifact paths;
• target/configuration fingerprints and corpus revision;
• wall-time and RSS deltas with repeat count and comparison method;
• changed profile counters;
• finding identity, churn, and operational coverage;
• TP/FP/FN/TN only if explicitly measured;
• default decision and the next attributable bottleneck.

Promote durable evidence into the closest stable surface:

• a unit or focused detector test for analysis semantics;
• a microbenchmark for a local performance mechanism;
• a project manifest/baseline for real-project behavior;
• a gold/disposition entry for reviewed precision;
• a research or architecture decision record only when the behavior or boundary changes.

Fold-specific priority order

Use this order to avoid optimizing the wrong layer:

1. Remove repeated work and reuse already-proven facts.
2. Skip functions or sinks that cannot exercise an enabled detector.
3. Improve cheap abstract proofs before changing solver tactics.
4. Attribute graph merge/SCC/RSS cost before replacing graph containers.
5. Propose semantic-model changes only when the profile shows symbolic control walking or path state is the actual bottleneck.

This order follows the current performance audit and keeps precision work separate from raw container or solver speculation.

Current Chromium precision campaign

Chromium is a production canary, not a complete oracle. The ten reviewed findings are evidence for prioritized model gaps: path-sensitive string-index relationships, fixed-table bounds, macro-range reasoning, API output contracts, and local pointer provenance. Each future fix must be generalized, tested with a positive and negative case, and measured against the canary.

Do not tune against Chromium source locations. The only acceptable Chromium-specific artifact is an exact reviewed disposition or gold entry used to measure a generalized change.