b0b51f1e71
- fail closed on MCP input and compile database policy - add shared scan catalogs, finding queries, and tiered gates
55 lines
2.8 KiB
Markdown
55 lines
2.8 KiB
Markdown
# Impacted regression-test policy
|
|
|
|
`impacted_tests.json` is the reviewed changed-path map used by `fold quality`.
|
|
Every run retains `test-selection.json` with the changed paths, matched rule IDs,
|
|
selected tests, and rationale for each selection.
|
|
|
|
Two suites are unconditional:
|
|
|
|
- `fold:benchmark_evaluation` preserves positive, negative, and regression
|
|
corpus contracts.
|
|
- `fold:source_structure` preserves architecture and source-budget contracts.
|
|
|
|
Rules add subsystem-specific integration and regression tests. Selection is a
|
|
union, so a change spanning subsystems runs every applicable test. A changed
|
|
production, build, tool, or test path covered by `fallback_globs` but by no
|
|
reviewed rule runs the complete Meson suite. A missing or invalid manifest and
|
|
git-diff failures also fail closed to the complete suite.
|
|
|
|
When introducing a subsystem, update the manifest in the same change. Name the
|
|
behavioral tests that exercise its public effects and give a rationale. Do not
|
|
remove unconditional or fallback coverage merely to reduce CI time; first add
|
|
evidence that a narrower rule preserves the relevant regression coverage.
|
|
|
|
## Flake evidence
|
|
|
|
`fold quality --test-repetitions N` executes the same selected set independently
|
|
`N` times. Each attempt retains its complete console log and a sanitized result
|
|
artifact. `flaky-test-evidence.json` classifies each test as `stable-pass`,
|
|
`stable-fail`, `stable-skip`, `flaky` (both pass and fail), or `inconsistent`
|
|
(missing/skip transitions or malformed evidence). Sanitized artifacts omit the
|
|
Meson environment and test stdout because the complete console log is retained
|
|
separately and environments may contain credentials.
|
|
|
|
A later pass never erases an earlier failure: every attempt and the aggregate
|
|
classification participate in the gate status. Repetition is evidence, not an
|
|
automatic retry or quarantine mechanism. The default is one attempt; broader CI
|
|
tiers may request repeated evidence explicitly.
|
|
|
|
## CI tiers
|
|
|
|
`.github/workflows/quality-gates.yml` applies one explicit tier per event:
|
|
|
|
- **PR (pull request):** merge-base differential source scan, impacted
|
|
regressions once, and the Fold policy manifest.
|
|
- **Nightly (daily schedule or manual dispatch):** complete Meson suite three
|
|
times plus the Fold policy manifest, with no branch differential.
|
|
- **Release (published release or manual dispatch):** complete Meson suite,
|
|
Fold policy manifest, reviewed 70% line-coverage floor, and release-binary
|
|
smoke.
|
|
|
|
Every tier uploads `quality-report/`, including complete logs, SARIF where
|
|
applicable, test selection, repeated-test classifications, policy summaries,
|
|
and compact token measurements. The nightly repetition count collects flake
|
|
evidence; it does not convert failures into passes. Release coverage uses the
|
|
previously reviewed floor rather than introducing a new threshold.
|