56550a93a1
- docs/internal/symex-fuzz-landscape.md: 11 tools across 6 categories - tools/fold-swift/swan_sil_parser: 75 SIL operator kinds, JSON converter - bench/juliet/expansion/: 118-CWE catalog, gold generator, runner - bench/score_fp.py: precision/recall/F1 per detector
78 lines
2.7 KiB
Markdown
78 lines
2.7 KiB
Markdown
# Juliet Expansion — Full CWE Coverage for fold Detectors
|
|
|
|
Expands fold's Juliet test suite integration beyond the 14 initially-covered
|
|
CWEs to a comprehensive catalog of all 118 Juliet C/C++ 1.3 CWE categories.
|
|
|
|
## Contents
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `cwe_catalog.json` | Full CWE→detector mapping for all 118 CWEs |
|
|
| `_generate_gold.py` | Generate gold-standard files for all fold-relevant CWEs |
|
|
| `run_expanded.py` | Run fold against all mapped Juliet categories |
|
|
| `score_expanded.py` | Per-detector precision/recall aggregator |
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# 1. Fetch and filter the Juliet corpus (one-time)
|
|
bench/corpora/juliet/fetch.sh
|
|
python3 bench/corpora/juliet/filter_cwes.py --catalog bench/juliet/expansion/cwe_catalog.json
|
|
|
|
# 2. Generate gold standards for expanded CWE set
|
|
python3 bench/juliet/expansion/_generate_gold.py
|
|
|
|
# 3. Run fold against all mapped CWEs
|
|
python3 bench/juliet/expansion/run_expanded.py --detectors null-deref,div-by-zero,int-overflow,int-truncation,sign-conversion,oob --processes 4
|
|
|
|
# 4. Score results
|
|
python3 bench/juliet/expansion/score_expanded.py --run-dir bench/results/<latest>
|
|
```
|
|
|
|
## Detector Coverage Summary
|
|
|
|
The catalog maps all 118 Juliet CWEs to fold's detectors. The target six
|
|
detectors cover these CWE categories:
|
|
|
|
| Detector | Primary CWEs | Additional CWEs | Total testcases |
|
|
|----------|-------------|-----------------|-----------------|
|
|
| null-deref | 476 | 690, 252, 588, 562 | ~2,900 |
|
|
| div-by-zero | 369 | — | ~1,480 |
|
|
| int-overflow | 190, 191 | 680, 789 | ~13,800 |
|
|
| int-truncation | 197 | 681 | ~1,464 |
|
|
| sign-conversion | 194, 195, 196 | 467 | ~3,828 |
|
|
| oob-access | 121, 122, 124, 126, 127 | 123, 468, 785 | ~19,100 |
|
|
|
|
Beyond the six target detectors, fold has other detectors that map to
|
|
additional CWEs (cataloged but not the focus of this expansion):
|
|
|
|
| Detector | CWEs |
|
|
|----------|------|
|
|
| uninit-local-read | 457, 665 |
|
|
| use-after-free | 401, 415, 416, 562, 590, 672, 675, 761, 762 |
|
|
| deprecated-api | 242, 676 |
|
|
| taint-track | 78, 114, 134, 426 |
|
|
|
|
## CWE Catalog Schema
|
|
|
|
`cwe_catalog.json` is the authoritative mapping. Each CWE entry:
|
|
|
|
```json
|
|
{
|
|
"15": {
|
|
"title": "External Control of System or Configuration Setting",
|
|
"fold_detector": null,
|
|
"fold_confidence": "none",
|
|
"notes": "No fold detector covers configuration injection",
|
|
"testcase_count": 79,
|
|
"already_filtered": false
|
|
}
|
|
}
|
|
```
|
|
|
|
Fields:
|
|
- `fold_detector` — fold detector name (or null if unmapped)
|
|
- `fold_confidence` — "primary" (direct CWE match), "plausible" (may trigger detector), "none"
|
|
- `notes` — rationale for mapping (or why unmapped)
|
|
- `testcase_count` — number of .c/.cpp files in Juliet source
|
|
- `already_filtered` — true if currently in the filtered corpus
|