gem/docs/build/benchmark-protocol.md

# Benchmark and profiling protocol

## Questions and KPIs

Use the dense Gemma 4 31B GGUF that will serve hax. Record prompt processing (`pp`) and generation
(`tg`) tokens per second separately. For agent work, generation throughput and time-to-first-token
are primary; prompt throughput matters when YAMS adds retrieved context. Do not aggregate them into
one score.

Run the native full-model baseline with a deliberately small context while validating a new engine
path. Increase the workload only after correctness is established:

```sh
build/gem-bench --model /models/gemma-4-31B-it-Q4_0.gguf \
  --context 64 --prompt-tokens 8 --generation-tokens 8 --repetitions 3 \
  --command-span layer > artifacts/baseline.json
```

Run three repetitions for every candidate and retain raw JSON:

```sh
for setting in layer token; do
  build/gem-bench --model /models/gemma-4-31B-it-Q4_0.gguf \
    --context 64 --prompt-tokens 8 --generation-tokens 8 --repetitions 3 \
    --command-span "$setting" > "artifacts/${setting}.json"
done
```

Apply the promotion gate to each candidate:

```sh
scripts/analyze-benchmarks.sh artifacts/flash-auto.json artifacts/flash-on.json \
  artifacts/flash-on-comparison.json
jq '.promotion, .metrics' artifacts/flash-on-comparison.json
```

When power/thermal drift invalidates sequential attention runs, use the same-process interleaved
oracle/candidate harness and split its four raw rows before analysis:

```sh
build/gem-bench --model /models/gemma-4-31B-it-Q4_0.gguf --context 128 \
  --prompt-tokens 32 --generation-tokens 64 --repetitions 5 --attention-ab > paired.json
```

The gate treats generation as the primary KPI by default. It requires the improvement to exceed
both 3% and the larger run coefficient of variation, rejects a secondary-KPI regression over 2%
when that regression exceeds observed variation, and rejects comparisons that change more than
one tuning axis. Use `--primary pp` for a prefill-specific optimization. Model changes are rejected
unless `--allow-model-change` is supplied for a clearly labeled quantization experiment.

The first native axes are command-buffer span, decode threadgroup geometry, KV cache type, prefill
tile shape, and prompt batch. Change only one axis per run. llama-bench is a separately recorded
competitor result, never the native measurement source.

Before full-model `gem-bench` is available, operator diagnostics use:

```sh
build/gem-kernel-bench --buffer-mode copy --repetitions 3 --iterations 200
build/gem-kernel-bench --buffer-mode persistent --repetitions 3 --iterations 200
scripts/analyze-kernel-benchmarks.sh BASELINE.json CANDIDATE.json
```

For long-decode attention on a thermally drifting host, retain three repetitions in each ABBA arm
and pool only after verifying identical controls within both baseline and candidate arms:

```sh
scripts/analyze-decode-attention-abba.sh A1.json B1.json B2.json A2.json comparison.json
```

The report preserves opening/closing improvements, per-arm drift, and whether every candidate
sample beat every baseline sample. It remains a microbenchmark screen rather than promotion proof.

Derive the current split-score route's minimum logical traffic directly from a retained exact-shape
profile:

```sh
scripts/analyze-decode-attention-traffic.sh PROFILE.json TRAFFIC.json
```

This accounts for K/V reads, raw score writes, the three normalization passes, repeated normalized
score reads by every value-dimension group, and bounded value partials. It is an address-level
model, not a device-bandwidth measurement: caches can satisfy part of the traffic, so a fusion
candidate still requires Metal counters and authentic whole-token A/B evidence.

Batch-one Q4 decode candidates must also be screened against a nonresident weight stream. Use at
least 250 MiB of independently backed matrix slots and a command span representative of the
token-wide production command buffer. Repetitions shorter than roughly 0.25 seconds can measure M4
GPU frequency ramp instead of sustained bandwidth:

```sh
# 12B FFN gate/up: eight 31.64 MiB matrices, 336 projection-like dispatches per command buffer.
build/gem-kernel-bench --buffer-mode persistent --input 3840 --output 15360 \
  --matrix-slots 8 --command-span 336 --warmup 3360 \
  --iterations 3360 --repetitions 7

# 31B FFN gate/up: four 62.02 MiB matrices.
build/gem-kernel-bench --buffer-mode persistent --input 5376 --output 21504 \
  --matrix-slots 4 --command-span 336 --warmup 3360 \
  --iterations 3360 --repetitions 7
```

`matrix_slots` and `command_span` are separate tuning axes. Change only one between comparable
runs and retain the reported `working_set_bytes` and effective warmup count.

These results may guide implementation, but their analyzer always reports `promotion.eligible` as
false. A kernel microbenchmark cannot substitute for deterministic model generation, pp/tg, TTFT,
or the native model configuration record.

Long-attention screens whose control flow depends on score values must use finite seeded data, not
only the profiler's zero-filled allocation. `--seed-prefix` places a deterministic sparse signal in
Q/K/V without spending hours evaluating a model prefix and records `seed_prefix: true`:

```sh
build/gem-long-attention-profile --shape 31b --position 131040 --rows 32 \
  --repetitions 10 --warmup 10 --seed-prefix --variant compare-bq8-bk128
```

This remains an operator screen. Retain per-sample GPU timestamps and reject a run with monotonic
frequency ramp or background-work discontinuities even when the aggregate mean looks favorable.
For BK64/BK128, the only permitted comparison axis is cache tile width; BQ8, F32 KV, head geometry,
online-softmax semantics, and dispatch ownership stay fixed. The even measurement count gives each
variant equal exposure to first and second dispatch order. Run the frozen 4K/64K/128K gate with:

```sh
scripts/run-bq8-bk128-gate.sh artifacts/YYYY-MM-DD-bq8-bk128-clean
```

The runner refuses a busy macOS host using both one-minute load and a one-second system-busy sample
from the second `top` observation. It also records macOS's decaying per-process `%CPU` sum as
provenance, requires AC power and no recorded thermal/performance warning, preserves pre/post host
state and binary/source/linkage digests, and uses ten warmups before ten interleaved measurements.
Its analyzer requires both 64K and 128K gains to exceed 3% and observed variation, requires a
positive paired 95% confidence lower bound, caps coefficient of variation and half-run drift at 3%, and
rejects a 4K regression beyond 2%. Passing advances BK128 only to evaluated-graph/logit testing; an
operator screen can never promote the production selector by itself.
The gate also freezes `gem-metal-counter-info`: on the target, BK64 consumes 18,496 of the public
32,768-byte per-threadgroup limit (56.45%) and BK128 consumes 20,544 bytes (62.70%). This records
resource pressure without falsely treating the per-threadgroup limit as a resident-workgroup or
occupancy counter; occupancy still requires an Xcode GPU trace.

If the machine is intermittently busy, wait for three consecutive policy-compliant samples and
then replace the watcher with the unchanged gate process:

```sh
GEM_CLEAN_HOST_MAX_WAIT_SECONDS=28800 \
  scripts/run-when-host-clean.sh artifacts/YYYY-MM-DD-bq8-bk128-host-wait.ndjson -- \
  scripts/run-bq8-bk128-gate.sh artifacts/YYYY-MM-DD-bq8-bk128-clean
```

The watcher records every sample, resets its clean streak after any load, one-second CPU, power, or
thermal failure, and exits 75 without running the command when its wait expires. It neither kills
nor reprioritizes other processes. Its receipt documents why the gate was launched but does not
replace the gate's own preflight, postflight, or continuous host receipts.

Fixed-eight verifier experiments should also run the production `verify8-4sg` kernel with a
rotating weight working set:

```sh
build/gem-prefill-bench --kernel verify8-4sg --batch-rows 8 \
  --matrix-slots 4 --command-span 7 --iterations 70 --repetitions 5
```

`matrix_slots=1` intentionally repeats one projection. Four slots rotate 260 MiB of independently
backed Q4_0 matrices; record both settings around a candidate. Treat this as a cache/working-set
diagnostic, not a promotion source. The authentic 31B verifier remains the kernel gate because its
projection dimensions and graph schedule are not represented by a single-shape loop.

Sampler-only changes use `build/gem-sampler-bench` with the production 262144-token vocabulary,
default top-k/top-p, three repetitions, and a focused seeded-sequence parity test. Preserve both raw
JSON files and a lower-is-better latency comparison. This does not replace model pp/tg measurements;
it isolates CPU latency that model-only `gem-bench` intentionally excludes.

The retained selector-20 long-decode diagnostic changes only the global-attention K-score staging
tile from one token to eight while preserving selector 19's split16 value path. It is not part of
the BK64/BK128 prefill decision and must run only after that cycle and current-production server
acceptance are closed:

```sh
GEM_MODEL_SHA256=031dc1c5fa9c5a0abbf3c39c5173fb2af65f5ac2dc2a090268561d3c72dcd834 \
  scripts/run-score-tile8-gate.sh .state/models/gemma-4-31B-it-Q4_0.gguf \
    artifacts/YYYY-MM-DD-score-tile8-clean
```

Score-tile8 gate schema 2 freezes three repetitions per ABBA arm at 64K and 128K, five iterations,
three warmups, AC power, no thermal/performance warning, load per logical CPU at most 1.0, and a
one-second system-busy ceiling of 10%. Each context gets independent preflight and postflight
receipts, and a failed postflight aborts before the next context. Correctness parity and the
adjacent backend-op oracle deliberately run after timing so they cannot warm or load-bias the
performance arms. The analyzer independently rejects battery, thermal, busy, changed-axis, or
failed-oracle receipts. The operator screen remains non-promotional; it must project at least 3%
whole-token gain at both lengths before authentic logits and populated-model profiling are admitted.
If selector 20 is rejected, the next larger decode experiment is specified—but intentionally not
implemented—in `../research/fused-split-kv-design.md`. Its fixed authentic shape is symmetric
`DK=DV=512`; "dim32" refers only to the production value workgroup tile.

Context capacity is a deployed-server measurement, not a microbenchmark axis. Start `gem serve` at
each context size (32K, 64K, 128K), use a prompt whose token count is confirmed by
`build/gem-tokenize --count --file --chat-user MODEL.gguf PROMPT_FILE`, and capture
`scripts/server-probe.sh` output. File mode avoids the process argument-size ceiling and
`--chat-user` counts the exact canonical user-turn/control-token prompt evaluated by the server,
not only the raw file content. Record time-to-first-byte and
end-to-end completion time alongside the low-level pp/tg numbers.

Capacity with a short prompt is not evidence of long-prompt performance. For the local hax workload,
8K, 32K, 64K, and 128K prompt processing are separate mandatory gates. At each rung, retain the raw
three-repetition `pp` result, `gem print-config`, peak attention scratch, first-visible-event TTFT,
and end-to-end time. Gemma 4 31B has 50 layers bounded by the 1,024-token sliding window and ten
full-attention layers; optimize and profile those ten layers separately. Exact dense attention keeps
quadratic compute, but candidates must use linear or bounded score memory and must never allocate a
prompt-squared score matrix.

Generate reproducible exact-count fixtures with `scripts/make-long-prompt.sh MODEL.gguf TOKENS
OUTPUT_FILE`. The generator derives chat-template overhead and repeated-line token contribution from
the selected GGUF, then re-tokenizes the completed file and refuses to publish it unless the count is
exact. Preserve its JSON receipt with the benchmark artifacts; the canonical 8K/32K/64K/128K files
are byte-identical when regenerated from the current 12B or 31B tokenizer.

Every long-prompt candidate also needs a repeated real-weight determinism gate. Run identical 2K and
4K logits at least twice and require bit-identical output before performance promotion. Then compare
the first 32-token tile crossing the 4,096-token routing boundary against the deterministic rowwise
oracle, using the ordinary argmax/top-10/correlation/NMSE thresholds below. A fast operator result is
rejected if either repeated model output or the boundary differential fails.

Before choosing a long-prompt kernel axis, profile one production 32-token tile at an authentic
prefix position. The profiler builds the prefix once, performs one unmeasured profiled warmup, then
uses exact KV checkpoints to repeat the same tile three times. It reports the nine fixed layer
stages separately for sliding and full-attention layers:

```sh
build/gem-long-prefill-stage-profile MODEL.gguf 4096 3
```

Retain the raw JSON and `gem print-config`. Stage profiling changes encoder boundaries and is an
attribution tool, not a throughput promotion source; use an interleaved full-graph A/B harness for
the selected one-axis candidate.

At 64K/128K, the same profiler may seed the allocated production KV cache with finite diagnostic
bytes instead of spending hours evaluating a prefix:

```sh
build/gem-long-prefill-stage-profile MODEL.gguf 131040 3 --seed-prefix
```

The receipt must report `prefix_mode: "seeded-kv"`, `prefix_builds: 0`, 50 sliding layers, ten full
layers, three repetitions, and adequate timestamp coverage. It preserves the production 32-token
graph and exact long-position routing, but the artificial KV contents make it stage-attribution
evidence only. It cannot establish prompt throughput, TTFT, output quality, or promotion; rerun the
selected candidate through the evaluated-prefix and full-graph gates.

For same-process interleaved full-graph screening without rebuilding a 64K/128K prompt, diagnostic
routes exposed by `gem-long-prefix-ab` may use a finite seeded KV state:

```sh
GEM_LONG_PREFIX_SEED=1 GEM_LONG_PREFIX_CANDIDATE=CANDIDATE \
  build/gem-long-prefix-ab MODEL.gguf 131040 3
```

The output must report `prefix_mode: seeded-kv`, `prefix_builds: 0`, repeated-route determinism,
cross-route logits, and three interleaved samples per route. This screen preserves the ordinary
unprofiled graph and checkpoint rollback, but synthetic KV contents make it attribution evidence
only. A promoted candidate still requires an evaluated prefix and the server gates.

For the production BQ8/BK64 global-prefill route, `CANDIDATE=bq8` compares diagnostic selector 22
(the former row2 global prefill) against selector 19. Preserve the evaluated-prefix JSON, the
post-promotion seeded routing check, and a 131328-context `gem print-config` receipt together.

If the frozen BK64/BK128 operator gate advances, `CANDIDATE=bk128` compares production selector 19
against diagnostic selector 23 while holding SWA, decode, KV type, batching, and the shared prefix
fixed:

Run the retained operator artifacts through the fail-closed graph runner rather than invoking the
harness directly:

```sh
GEM_MODEL_SHA256=KNOWN_VERIFIED_MODEL_DIGEST \
  scripts/run-bq8-bk128-graph-gate.sh \
    artifacts/YYYY-MM-DD-bq8-bk128-clean MODEL.gguf \
    artifacts/YYYY-MM-DD-bq8-bk128-graph-64k
```

Omit `GEM_MODEL_SHA256` to hash the model in the runner. Supplying a previously verified digest
avoids making a multi-gigabyte hash pass part of the quiet-host window. The runner independently
re-analyzes the raw operator directory and refuses to start unless it advances BK128. It records
the native configuration and linkage plus preflight, continuous, and postflight load, AC-power,
and thermal receipts. Any monitor violation invalidates the graph run.

The receipt must name selector 19 as baseline and selector 23 as candidate, report one evaluated
prefix build, bit-identical repeats within each route, matching argmax/top-10, correlation at least
0.9999, and NMSE no greater than 1e-4. The analyzer also requires the mean improvement to exceed
3% and the larger arm CV, with a positive paired 95% confidence lower bound (three pairs,
Student-t df=2). Selector 23 is never production-eligible directly; only a passing graph gate
permits changing selector 19's internal prefill dispatch, after which native correctness, the
adjacent backend-op oracle, and the final server gate must be rerun.

Convert either receipt into a structurally validated stage ranking with:

```sh
scripts/analyze-prefill-stages.sh PROFILE.json ANALYSIS.json
```

Before adding device-specific counter instrumentation, record which counter sets and sampling
boundaries the target Mac exposes. This standalone diagnostic only queries Metal capabilities and
does not link the production engine:

```sh
build/gem-metal-counter-info > ARTIFACT_DIR/metal-counter-info.json
```

Do not infer that occupancy, bandwidth, or limiter counters are programmatically available from
their presence in Xcode or Instruments. Only counters named in this receipt can be sampled through
the public `MTLCounterSampleBuffer` API on the target device. Keep stage timestamps as the portable
fallback when a desired device counter is absent.

For the fixed 31B/F32 128K attribution rung, capture the counter inventory, configuration, binary
and model identities, linkage, seeded profile, and validated ranking together:

```sh
GEM_MODEL_SHA256=EXPECTED_SHA256 scripts/run-prefill-stage-attribution.sh \
    MODEL.gguf ARTIFACT_DIR
```

The runner refuses to overlap a native server. Run it before an experimental kernel gate so the
largest production stage—not an already-selected candidate—determines the next optimization axis.

The analyzer requires the fixed 60-layer/50-sliding/10-full graph and all three sample matrices. Its
selected axis is a profiling hypothesis, never a promotion decision.

For long-position generation, `gem-decode-profile` builds the populated prefix once, then uses an
exact checkpoint rollback for its warmup and each sample:

```sh
build/gem-decode-profile --model MODEL.gguf --context 65793 \
  --position 65536 --rows 1 --repetitions 3
```

For one-row runs, the same JSON also includes `production_execution`, measured through the ordinary
one-command-buffer/one-encoder path rather than the stage-counter path. Its host phases partition
wall time into prepare, encode, submit/wait, cleanup, and logits readback; GPU time is nested inside
submit/wait. Use `mean_wall_minus_gpu_seconds` as the upper bound for host/control optimization.
Do not subtract individual host phases from GPU time or treat a dirty-host seeded-KV run as a speed
gate. The phase sum must match measured wall time within timer precision.

Evaluated prefixes report progress to stderr every 4,096 completed tokens by default. The callback
runs only after the existing production prefill batch completes; it does not split batches or add
command buffers. Use `--progress-every N` to change the reporting interval or `--no-progress` for a
silent runner. JSON remains isolated on stdout and records `prefix_build_wall_seconds` plus
`prefix_tokens_per_second`; those prefix fields are operational evidence, while the checkpointed
decode stage samples remain the kernel-attribution result.

When iterating on dispatch geometry, `--seed-prefix` may initialize the complete KV allocation with
finite diagnostic bytes and advance only the logical position. This avoids evaluating a 64K/128K
prompt and preserves the production model graph, KV layout, layer routing, command-buffer structure,
and requested live-token count:

```sh
build/gem-decode-profile --model MODEL.gguf --context 131328 \
  --position 131072 --rows 1 --repetitions 3 --seed-prefix
```

The JSON must report `prefix_mode: "seeded-kv"` and `prefix_builds: 0`. Seeded-KV results are
attribution and screening evidence only: their cache contents are not produced by Gemma, so they
cannot support correctness, generation quality, prompt throughput, TTFT, or kernel promotion. A
candidate selected with this shortcut must still pass an evaluated-prompt profile at the same
position and every ordinary model/server promotion gate.

Use the synthetic exact-shape profiler to divide the same production attention route into score,
normalization, and value phases without rebuilding a model prefix:

```sh
build/gem-decode-attention-profile --shape 31b-full --tokens 65536 \
  --iterations 20 --repetitions 7 --warmup 10
```

Value-dimension experiments must name the one changed kernel axis. Bracket the exact 31B full
shape in A/B/B/A order and retain every raw JSON object:

```sh
build/gem-decode-attention-profile --shape 31b-full --kernel dim64 --tokens 65536 \
  --iterations 20 --repetitions 7 --warmup 10
build/gem-decode-attention-profile --shape 31b-full --kernel dim32 --tokens 65536 \
  --iterations 20 --repetitions 7 --warmup 10
```

Selector 18 was promoted after its correctness and populated-position gates. Use the override only
to reproduce that candidate receipt explicitly; omitting it now exercises the same production
default:

```sh
build/gem-decode-profile --model MODEL.gguf --attention-kernel dim32 \
  --context 65792 --position 65536 --rows 1 --repetitions 3
```

The synthetic phase profile selects an operator axis; it cannot establish a whole-token ceiling or
promote a kernel. Retain both raw outputs and `gem print-config`, and require the populated
full-model profile to show at least a 3% end-to-end ceiling before implementation.

Calculate that ceiling reproducibly with the matching populated-position and exact-shape receipts:

```sh
scripts/analyze-decode-residual.sh FULL_MODEL_PROFILE.json OPERATOR_PROFILE.json REPORT.json
```

The analyzer binds the positions, native workloads, 60-layer/one-row model geometry, minimum sample
counts, and at least 95% stage coverage. For Gemma 4 31B it attributes exactly ten full-attention
layers, reports the theoretical whole-token P*V ceiling and the value-kernel gain needed to clear the
3% floor, and rejects attribution that exceeds authentic KV-attention time. An admissible result
authorizes only a one-axis operator screen; it is never kernel-promotion evidence.

For the parked D512 fused split-KV diagnostic, bind both 64K and 128K selector-19 evaluated model
profiles to the matching split16 exact-shape profiles:

```sh
scripts/analyze-fused-split-kv-ceiling.sh \
  FULL_MODEL_64K.json OPERATOR_64K.json \
  FULL_MODEL_128K.json OPERATOR_128K.json REPORT.json
```

This analyzer fixes the 32-query/4-KV-head/D512 geometry and attributes exactly ten global layers.
It reports complete-global-attention and score-only whole-token ceilings plus the within-stage gain
needed to clear 3%. It records all input digests and rejects changed selectors, contexts, stage
totals, or insufficient coverage. Its output is research admission evidence only.

Before a fused Metal route exists, reproduce the independent CPU algebra boundary with:

```sh
build/gem-online-merge-check --tokens 8191 --partitions 32 --seed 1
build/gem-online-merge-check --tokens 8192 --partitions 32 --seed 1
build/gem-online-merge-check --tokens 8193 --partitions 32 --seed 1
build/gem-online-merge-check --tokens 65536 --partitions 32 --seed 1
build/gem-online-merge-check --tokens 131072 --partitions 32 --seed 1
scripts/analyze-online-merge-boundary.sh \
  8191.json 8192.json 8193.json 65536.json 131072.json audit.json
```

The audit requires D512, P=32, deterministic repeated hashes, correlation at least 0.9999, NMSE at
most 1e-4, and the exact three boundary plus two long-tail positions. It is a CPU reduction-algebra prerequisite only and is
never a Metal correctness, model-continuation, timing, or promotion result.

After the diagnostic Metal pipelines compile, record their resource receipt and fail closed before
dispatching an operator benchmark:

```sh
scripts/analyze-fused-split-kv-resources.sh RESOURCE_RECEIPT.json RESOURCE_ANALYSIS.json
```

The receipt must identify the M4 Max/Apple-family-9 device, D512/P32/tile4/F32/no-window geometry,
P=32 scratch bytes, compiled first-pass and reducer resource limits, and SHA-256 identities for the
assembled Metal source and native binary. It requires the first pass to dispatch 256 threads/eight
SIMD groups with 16 KiB dynamic threadgroup memory, and the reducer to dispatch 256 threads with no
dynamic threadgroup memory. This is a compilation/resource screen only; an Xcode occupancy trace,
raw GPU timestamps, CPU/selector-19 differentials, and authentic model gates remain mandatory.

After compiled resources pass, bind correctness evidence before collecting performance numbers:

```sh
scripts/analyze-fused-split-kv-correctness.sh \
  CPU_GATE.json RESOURCE_ANALYSIS.json \
  OP_8191.json OP_8192.json OP_8193.json OP_65536.json OP_131072.json \
  MODEL_8191.json MODEL_8192.json MODEL_8193.json CORRECTNESS_ANALYSIS.json
```

The operator inputs must compare the fixed diagnostic route against forced selector 19 at all five
CPU-gated lengths. The three model inputs must use evaluated prompts and prove matching argmax,
ordered top-10, and at least eight identical greedy continuation tokens. Correlation must be at
least 0.9999 and NMSE no greater than 1e-4 throughout. The analyzer records every input digest and
always reports `promotion_eligible:false`; only subsequent clean-host whole-token gates can promote.

For 31B SWA routing changes, locate persistent-state divergence before accepting aggregate logit
similarity. `gem-swa-state-scan` advances rowwise and candidate engines in lockstep and hashes every
new logical K/V row after each 32-token tile. The empty-cache tile is a distinct mandatory case:

```sh
build/gem-swa-state-scan MODEL.gguf 4096
build/gem-long-prefix-ab MODEL.gguf 4096 3
build/gem-long-prefix-ab MODEL.gguf 32768 3
```

The state scan must report exact K/V through all requested tokens. The A/B harness uses one shared
production prefix with exact checkpoint rollback, alternates route order, includes the final logits
projection, requires repeated route logits to be bit-identical, and reports the rowwise versus
batched-SWA tile delta.

For actual streaming TTFT, use `scripts/server-stream-probe.sh BASE_URL MODEL PROMPT_FILE
OUTPUT_SSE OUTPUT_JSON`. Ordinary curl `time_starttransfer` observes the immediate HTTP headers and
assistant-role event, not model output. The streaming probe records both that transport value and
the first SSE delta containing non-empty content, non-empty reasoning, or a non-empty tool-call
array; only
`first_visible_event_seconds` is the model-visible TTFT KPI. For a capacity-edge prompt, set
`GEM_STREAM_MAX_TOKENS=16` (or another explicitly recorded positive limit) so the request leaves
enough tokens to pass suppressed Gemma channel pieces without exceeding the configured context;
the probe records `requested_max_tokens` in its JSON.

Use `scripts/long-context-server-gate.sh MODEL.gguf CONTEXT PROMPT_FILE OUTPUT_DIR PORT` for the
required three-sample gate. It starts a fresh native server for every sample and owns its cleanup;
this is required because repeated requests to one process can reuse the prefix cache and cease to
measure full prompt evaluation. The runner verifies the canonical chat token count and aggregates
that every server-reported `usage.prompt_tokens` value exactly matches that pre-count before it
aggregates first-visible-event and completion latency, while retaining each raw SSE stream, server
log, and validated `gem print-config`. Set `GEM_KV_TYPE=f16` or `GEM_KV_TYPE=f16-full` to gate an
opt-in cache layout; the default is `f32`, and the selected type is recorded in the summary. The
runner records SHA-256 digests for the GGUF and prompt before timing begins; set `GEM_MODEL_SHA256` to a
previously verified 64-digit digest to avoid rereading the same multi-gigabyte model at every rung.
The retained server log must show batch-boundary prefill progress every 4,096 tokens for these long
requests. SSE keepalive comments are emitted after every completed production batch; the streaming
probe excludes them from `first_visible_event_seconds`, while a failed write cancels remaining
prompt evaluation before the next batch is submitted. A sample is incomplete unless its retained
log contains the exact 100% prefill record; prefix-extension samples require the corresponding final
logical token count as well. The same requirement applies when resuming prior samples.

After changing prompt-progress or cancellation behavior, run one authentic same-process recovery
gate without overlapping any performance benchmark:

```sh
scripts/server-prefill-cancel-gate.sh MODEL.gguf CONTEXT PROMPT_FILE OUTPUT_DIR PORT
```

The prompt must contain at least 128 canonical chat tokens. The runner requires curl timeout status
28, at least one SSE keepalive, health recovery within
`GEM_CANCEL_RECOVERY_LIMIT_SECONDS` (60 by default), and a successful visible completion from a
16-token second request to the same native process. The cancelled prompt must not reach its exact
100% prefill log record. Set `GEM_CANCEL_AFTER_SECONDS` explicitly when a single prefill batch on the
target hardware exceeds the five-second default. Retain the config, request, partial SSE, curl error,
server logs, recovery measurement, and summary. The CPU-only
`gem.server_prefill_cancel_gate` CTest exercises the complete runner against a serial HTTP/SSE
fixture; it is orchestration coverage, while the GGUF-backed command remains the behavioral gate.

If a multi-hour gate stops after one or more complete repetitions, rerun the identical command with
`GEM_GATE_RESUME=1`. The output directory must already contain both `gate-config.json` and
`print-config.json`. Resume is fail-closed: it compares the current canonical native configuration
and the complete model/prompt digests, paths, context, KV type, token limit, and prefix-extension text;
then it reuses only measurement/SSE/log tuples that independently pass the ordinary response and
exact-token validations. An incomplete repetition starts a new server process. A changed configuration,
changed provenance, malformed completed sample, or occupied port is never silently reused. The final
summary records `resumed_samples`. The CPU-only `gem.long_context_server_gate_resume` CTest locks
this contract down by exercising a three-sample resume and fail-closed configuration/provenance
changes without starting the native server or loading a GGUF.

For prompts of at least 65,536 tokens, the runner also applies a recorded host-state policy before
each sample. It captures the one-minute load average, logical CPU count, power source, thermal
status, active power assertions, and the busiest processes in `host-preflight-N.json`; by default it
refuses to start above 1.0 runnable task per logical CPU or when a one-second system sample exceeds
10% busy CPU. The short-window guard closes the load-average lag window without treating macOS
`ps %CPU`—a decaying average over up to one minute—as instantaneous.
Schema 4 additionally requires AC power and no thermal/performance warning at preflight, then
records load, power, and thermal state every 30 seconds. It aborts after three consecutive
violations, while the auditor rejects any retained schema-4 sample containing even one overloaded,
battery-power, or thermal-warning monitor record.
The server lifetime is held under
`caffeinate -i -w` so idle sleep cannot masquerade as kernel time. The threshold, interval, failure
count, and assertion are part of gate provenance. Override `GEM_GATE_PREFLIGHT_MIN_TOKENS`,
`GEM_GATE_MAX_LOAD_PER_CPU`, `GEM_GATE_MAX_SHORT_WINDOW_BUSY_CPU_PERCENT`,
`GEM_GATE_HOST_MONITOR_SECONDS`, or
`GEM_GATE_HOST_MONITOR_FAILURES` only as a separately recorded benchmark axis. Gate schema 2 adds
the load/monitor contract, schema 3 adds short-window CPU sampling, and schema 4 makes power and
thermal state enforceable preflight and continuous-monitor controls. Older artifacts remain auditable
but intentionally cannot resume into a newer-schema run.

Cold-prefill results do not cover the dominant multi-turn hax case. After the fresh-process 64K and
128K gates pass, rerun the same harness with (for example)
`GEM_PREFIX_EXTENSION_TEXT='Reply with DONE.'`. Each repetition keeps its server alive for a second
canonical chat request whose token sequence strictly extends the first request boundary. Retain both
SSE streams and the server log, require the log to report the exact reused/appended token counts, and
record second-turn first-visible-event and completion latency. The extension gate must evaluate only
the appended suffix; a cache miss or an identical replay is a failure. Report this separately from
cold prefill—it demonstrates that a long conversation is amortized after its first turn, not that
initial 64K/128K ingestion is free.

After the three-process gate completes, independently recompute its aggregates and validate every
raw cold/prefix measurement, SSE terminator, exact 100% progress record, prefix-hit record, prompt
digest, and frozen production-binary digest:

```sh
scripts/audit-long-context-server-gate.sh OUTPUT_DIR build/gem > OUTPUT_DIR/audit.json
```

Do not use `summary.json` alone as acceptance evidence. The audit must pass against the raw files
while `build/gem` is still the frozen binary recorded when the gate began.

The runner writes `live-binary-sha256.txt` before its first sample and refuses resume if the current
production binary differs. The auditor derives the expected F32 KV bytes from the configured
context and fixed 31B topology: 50 layers bounded to `min(context, 1024)` with 16 KV heads at
dimension 256, plus ten context-growing layers with four KV heads at dimension 512, storing K and V.

## Profiling trigger

Profile only a candidate that clears the 3% threshold against baseline outside run-to-run variance.
Keep the output in a timestamped directory outside version control:

```sh
scripts/profile-metal.sh \
  build/gem \
  /models/gemma-4-31B-it-Q4_0.gguf \
  artifacts/YYYY-MM-DD-metal-profile/production.trace
```

The wrapper honors an explicit `DEVELOPER_DIR` and otherwise selects an installed stable or beta
Xcode when the active developer directory contains only Command Line Tools. It does not change the
machine-wide `xcode-select` setting. To capture an already-gated diagnostic executable instead of
the production server, pass the trace path followed by `--` and the exact command:

```sh
scripts/profile-metal.sh artifacts/YYYY-MM-DD-metal-profile/candidate.trace -- \
  build/gem-long-attention-profile --shape 31b --position 131040 --rows 32 \
  --repetitions 3 --warmup 1 --variant online-row4
```

The default template is `Metal System Trace`. Set `GEM_XCTRACE_TEMPLATE` to an installed Xcode
template such as `Game Performance` when the experiment requires its aggregate GPU counters, and
record the selected template beside the trace. Set `GEM_XCTRACE_INSTRUMENT='Metal GPU Counters'`
to add shader-PC sampling; the target M4 capture reports counter profile 3 only with that explicit
instrument.

For a focused GPU document, the long-attention profiler can capture one additional untimed dispatch
after all reported samples complete:

```sh
MTL_CAPTURE_ENABLED=1 \
GEM_METAL_CAPTURE_PATH=artifacts/YYYY-MM-DD-metal-profile/row2.gputrace \
  build/gem-long-attention-profile --shape 31b --position 4096 --rows 32 \
  --repetitions 3 --warmup 1 --variant online-row2
```

The JSON must report `capture:true`. Do not use capture execution as benchmark evidence. Prefer a
bounded position for occupancy, resource, spill, and heat-map inspection because GPU documents
serialize bound buffers; use the 128K system trace for long-context timing and memory behavior.

Capture the exact model hash, gem commit, reference llama.cpp commit, native configuration, macOS
build, power source,
temperature/power mode, and benchmark raw JSON beside the trace. Use the Metal trace to name the
dominant kernel and dispatch shape before attempting a kernel change.

## Correctness and promotion gates

Before model-level optimization, capture a real-weight first-token differential for both supported model
sizes. This uses the adjacent llama.cpp checkout only as an offline oracle and compares the entire native
262,144-entry F32 logit vector after first confirming exact token IDs:

```sh
scripts/real-logit-oracle.sh .state/models/gemma-4-12B-it-Q4_0.gguf 12b-logits.json
scripts/real-logit-oracle.sh .state/models/gemma-4-31B-it-Q4_0.gguf 31b-logits.json
```

The gate requires finite vectors, equal argmax and top-10 sets, correlation at least 0.999, and NMSE at
most 1e-3. Retain the emitted JSON; this is a correctness gate, not a performance comparison.

- The requested GGUF passes the strict manifest, all layers execute on Metal, and hax completes a
  streamed tool-call chat.
- The canonical Gemma 4 chat behavior and tokenizer golden vectors pass.
- A candidate must be at least 3% faster on its declared primary KPI and not regress the other KPI
  by over 2% outside variance.
- Any Metal-kernel change needs a native CPU differential operator test plus a deterministic model
  generation smoke test before benchmarking.
- Never compare a different quantization, context, gem revision, or thermal state as though it
  were a kernel improvement.

Metal backend startup is independently attributable with:

```sh
build/gem-metal-startup-profile 3
```

Each line is one JSON record containing device creation, source assembly, runtime library
compilation, pipeline creation, command-queue creation, and total time. Repetitions in one process
measure warm reopen behavior; use separate invocations for fresh-process readiness. These numbers
do not replace end-to-end `gem agent` readiness measurements and are not a kernel promotion gate.

## Research queue

1. Validate tokenizer parity and first-token logits on official Q4_0 weights.
2. Compare direct GGUF Q4_0 against a trace-justified packed layout, if needed.
3. Native flash-style attention and KV representation under 32K and 128K context.
4. Speculative decoding using the Gemma 4 assistant architecture after baseline is stable.
5. Multimodal image prefill only after text-agent latency is established.