56 lines
1.9 KiB
Markdown
56 lines
1.9 KiB
Markdown
# Brick analysis and measurement flow
|
|
|
|
## Design seam
|
|
|
|
Before editing, state:
|
|
|
|
- observable graph or renderer behavior;
|
|
- owning seam: extractor, identity, model, metric, importer, or renderer;
|
|
- expected node/edge/diagnostic delta;
|
|
- invariants: deterministic output, provenance, and unrelated identities.
|
|
|
|
## Evidence lanes
|
|
|
|
Keep lexical/syntactic extraction independent from imported Fold symbolic or
|
|
binary evidence. Merge them only in the canonical graph, retaining lane,
|
|
backend, confidence, and provenance on every edge.
|
|
|
|
## Test loop
|
|
|
|
1. Add a minimal fixture demonstrating the desired relationship.
|
|
2. Add a clean-negative fixture for likely lexical false positives.
|
|
3. Assert public model behavior rather than private regex details.
|
|
4. Implement, run the focused test, then the full suite.
|
|
5. Add a regression fixture when a bug is discovered.
|
|
|
|
Property tests are appropriate for stable ID determinism, JSON round trips, and
|
|
graph invariants. Fuzzing is appropriate for report ingestion and source text;
|
|
the invariant is no panic, runaway allocation, or path escape.
|
|
|
|
## Measurement loop
|
|
|
|
For an extractor or score change, write an experiment contract:
|
|
|
|
{
|
|
"hypothesis": "what should improve",
|
|
"owner": "one extractor or metric",
|
|
"fixtures": ["labeled corpus names"],
|
|
"expected": {
|
|
"increases": ["precision or recall metric"],
|
|
"preserves": ["stable IDs", "schema", "unrelated edges"]
|
|
},
|
|
"decision": "pending"
|
|
}
|
|
|
|
Use at least three repeats for performance claims. Compare raw artifacts and
|
|
report median latency plus observed range. A count on an unlabeled repository
|
|
is coverage information, not precision or recall.
|
|
|
|
## Promotion rule
|
|
|
|
- Keep a behavior change when labeled positives improve without clean-negative
|
|
regressions and stable invariants hold.
|
|
- Keep an unproven strategy explicit and disabled by default.
|
|
- Reject changes within measurement noise or changes that silently lose
|
|
coverage/provenance.
|
|
|