# Benchmark and profiling protocol ## Questions and KPIs Use the dense Gemma 4 31B GGUF that will serve hax. Record prompt processing (`pp`) and generation (`tg`) tokens per second separately. For agent work, generation throughput and time-to-first-token are primary; prompt throughput matters when YAMS adds retrieved context. Do not aggregate them into one score. Run the native full-model baseline with a deliberately small context while validating a new engine path. Increase the workload only after correctness is established: ```sh build/gem-bench --model /models/gemma-4-31B-it-Q4_0.gguf \ --context 64 --prompt-tokens 8 --generation-tokens 8 --repetitions 3 \ --command-span layer > artifacts/baseline.json ``` Run three repetitions for every candidate and retain raw JSON: ```sh for setting in layer token; do build/gem-bench --model /models/gemma-4-31B-it-Q4_0.gguf \ --context 64 --prompt-tokens 8 --generation-tokens 8 --repetitions 3 \ --command-span "$setting" > "artifacts/${setting}.json" done ``` Apply the promotion gate to each candidate: ```sh scripts/analyze-benchmarks.sh artifacts/flash-auto.json artifacts/flash-on.json \ artifacts/flash-on-comparison.json jq '.promotion, .metrics' artifacts/flash-on-comparison.json ``` When power/thermal drift invalidates sequential attention runs, use the same-process interleaved oracle/candidate harness and split its four raw rows before analysis: ```sh build/gem-bench --model /models/gemma-4-31B-it-Q4_0.gguf --context 128 \ --prompt-tokens 32 --generation-tokens 64 --repetitions 5 --attention-ab > paired.json ``` The gate treats generation as the primary KPI by default. It requires the improvement to exceed both 3% and the larger run coefficient of variation, rejects a secondary-KPI regression over 2% when that regression exceeds observed variation, and rejects comparisons that change more than one tuning axis. Use `--primary pp` for a prefill-specific optimization. Model changes are rejected unless `--allow-model-change` is supplied for a clearly labeled quantization experiment. The first native axes are command-buffer span, decode threadgroup geometry, KV cache type, prefill tile shape, and prompt batch. Change only one axis per run. llama-bench is a separately recorded competitor result, never the native measurement source. Before full-model `gem-bench` is available, operator diagnostics use: ```sh build/gem-kernel-bench --buffer-mode copy --repetitions 3 --iterations 200 build/gem-kernel-bench --buffer-mode persistent --repetitions 3 --iterations 200 scripts/analyze-kernel-benchmarks.sh BASELINE.json CANDIDATE.json ``` For long-decode attention on a thermally drifting host, retain three repetitions in each ABBA arm and pool only after verifying identical controls within both baseline and candidate arms: ```sh scripts/analyze-decode-attention-abba.sh A1.json B1.json B2.json A2.json comparison.json ``` The report preserves opening/closing improvements, per-arm drift, and whether every candidate sample beat every baseline sample. It remains a microbenchmark screen rather than promotion proof. Derive the current split-score route's minimum logical traffic directly from a retained exact-shape profile: ```sh scripts/analyze-decode-attention-traffic.sh PROFILE.json TRAFFIC.json ``` This accounts for K/V reads, raw score writes, the three normalization passes, repeated normalized score reads by every value-dimension group, and bounded value partials. It is an address-level model, not a device-bandwidth measurement: caches can satisfy part of the traffic, so a fusion candidate still requires Metal counters and authentic whole-token A/B evidence. Batch-one Q4 decode candidates must also be screened against a nonresident weight stream. Use at least 250 MiB of independently backed matrix slots and a command span representative of the token-wide production command buffer. Repetitions shorter than roughly 0.25 seconds can measure M4 GPU frequency ramp instead of sustained bandwidth: ```sh # 12B FFN gate/up: eight 31.64 MiB matrices, 336 projection-like dispatches per command buffer. build/gem-kernel-bench --buffer-mode persistent --input 3840 --output 15360 \ --matrix-slots 8 --command-span 336 --warmup 3360 \ --iterations 3360 --repetitions 7 # 31B FFN gate/up: four 62.02 MiB matrices. build/gem-kernel-bench --buffer-mode persistent --input 5376 --output 21504 \ --matrix-slots 4 --command-span 336 --warmup 3360 \ --iterations 3360 --repetitions 7 ``` `matrix_slots` and `command_span` are separate tuning axes. Change only one between comparable runs and retain the reported `working_set_bytes` and effective warmup count. These results may guide implementation, but their analyzer always reports `promotion.eligible` as false. A kernel microbenchmark cannot substitute for deterministic model generation, pp/tg, TTFT, or the native model configuration record. Long-attention screens whose control flow depends on score values must use finite seeded data, not only the profiler's zero-filled allocation. `--seed-prefix` places a deterministic sparse signal in Q/K/V without spending hours evaluating a model prefix and records `seed_prefix: true`: ```sh build/gem-long-attention-profile --shape 31b --position 131040 --rows 32 \ --repetitions 10 --warmup 10 --seed-prefix --variant compare-bq8-bk128 ``` This remains an operator screen. Retain per-sample GPU timestamps and reject a run with monotonic frequency ramp or background-work discontinuities even when the aggregate mean looks favorable. For BK64/BK128, the only permitted comparison axis is cache tile width; BQ8, F32 KV, head geometry, online-softmax semantics, and dispatch ownership stay fixed. The even measurement count gives each variant equal exposure to first and second dispatch order. Run the frozen 4K/64K/128K gate with: ```sh scripts/run-bq8-bk128-gate.sh artifacts/YYYY-MM-DD-bq8-bk128-clean ``` The runner refuses a busy macOS host using both one-minute load and a one-second system-busy sample from the second `top` observation. It also records macOS's decaying per-process `%CPU` sum as provenance, requires AC power and no recorded thermal/performance warning, preserves pre/post host state and binary/source/linkage digests, and uses ten warmups before ten interleaved measurements. Its analyzer requires both 64K and 128K gains to exceed 3% and observed variation, requires a positive paired 95% confidence lower bound, caps coefficient of variation and half-run drift at 3%, and rejects a 4K regression beyond 2%. Passing advances BK128 only to evaluated-graph/logit testing; an operator screen can never promote the production selector by itself. The gate also freezes `gem-metal-counter-info`: on the target, BK64 consumes 18,496 of the public 32,768-byte per-threadgroup limit (56.45%) and BK128 consumes 20,544 bytes (62.70%). This records resource pressure without falsely treating the per-threadgroup limit as a resident-workgroup or occupancy counter; occupancy still requires an Xcode GPU trace. If the machine is intermittently busy, wait for three consecutive policy-compliant samples and then replace the watcher with the unchanged gate process: ```sh GEM_CLEAN_HOST_MAX_WAIT_SECONDS=28800 \ scripts/run-when-host-clean.sh artifacts/YYYY-MM-DD-bq8-bk128-host-wait.ndjson -- \ scripts/run-bq8-bk128-gate.sh artifacts/YYYY-MM-DD-bq8-bk128-clean ``` The watcher records every sample, resets its clean streak after any load, one-second CPU, power, or thermal failure, and exits 75 without running the command when its wait expires. It neither kills nor reprioritizes other processes. Its receipt documents why the gate was launched but does not replace the gate's own preflight, postflight, or continuous host receipts. Fixed-eight verifier experiments should also run the production `verify8-4sg` kernel with a rotating weight working set: ```sh build/gem-prefill-bench --kernel verify8-4sg --batch-rows 8 \ --matrix-slots 4 --command-span 7 --iterations 70 --repetitions 5 ``` `matrix_slots=1` intentionally repeats one projection. Four slots rotate 260 MiB of independently backed Q4_0 matrices; record both settings around a candidate. Treat this as a cache/working-set diagnostic, not a promotion source. The authentic 31B verifier remains the kernel gate because its projection dimensions and graph schedule are not represented by a single-shape loop. Sampler-only changes use `build/gem-sampler-bench` with the production 262144-token vocabulary, default top-k/top-p, three repetitions, and a focused seeded-sequence parity test. Preserve both raw JSON files and a lower-is-better latency comparison. This does not replace model pp/tg measurements; it isolates CPU latency that model-only `gem-bench` intentionally excludes. The retained selector-20 long-decode diagnostic changes only the global-attention K-score staging tile from one token to eight while preserving selector 19's split16 value path. It is not part of the BK64/BK128 prefill decision and must run only after that cycle and current-production server acceptance are closed: ```sh GEM_MODEL_SHA256=031dc1c5fa9c5a0abbf3c39c5173fb2af65f5ac2dc2a090268561d3c72dcd834 \ scripts/run-score-tile8-gate.sh .state/models/gemma-4-31B-it-Q4_0.gguf \ artifacts/YYYY-MM-DD-score-tile8-clean ``` Score-tile8 gate schema 2 freezes three repetitions per ABBA arm at 64K and 128K, five iterations, three warmups, AC power, no thermal/performance warning, load per logical CPU at most 1.0, and a one-second system-busy ceiling of 10%. Each context gets independent preflight and postflight receipts, and a failed postflight aborts before the next context. Correctness parity and the adjacent backend-op oracle deliberately run after timing so they cannot warm or load-bias the performance arms. The analyzer independently rejects battery, thermal, busy, changed-axis, or failed-oracle receipts. The operator screen remains non-promotional; it must project at least 3% whole-token gain at both lengths before authentic logits and populated-model profiling are admitted. If selector 20 is rejected, the next larger decode experiment is specified—but intentionally not implemented—in `../research/fused-split-kv-design.md`. Its fixed authentic shape is symmetric `DK=DV=512`; "dim32" refers only to the production value workgroup tile. Context capacity is a deployed-server measurement, not a microbenchmark axis. Start `gem serve` at each context size (32K, 64K, 128K), use a prompt whose token count is confirmed by `build/gem-tokenize --count --file --chat-user MODEL.gguf PROMPT_FILE`, and capture `scripts/server-probe.sh` output. File mode avoids the process argument-size ceiling and `--chat-user` counts the exact canonical user-turn/control-token prompt evaluated by the server, not only the raw file content. Record time-to-first-byte and end-to-end completion time alongside the low-level pp/tg numbers. Capacity with a short prompt is not evidence of long-prompt performance. For the local hax workload, 8K, 32K, 64K, and 128K prompt processing are separate mandatory gates. At each rung, retain the raw three-repetition `pp` result, `gem print-config`, peak attention scratch, first-visible-event TTFT, and end-to-end time. Gemma 4 31B has 50 layers bounded by the 1,024-token sliding window and ten full-attention layers; optimize and profile those ten layers separately. Exact dense attention keeps quadratic compute, but candidates must use linear or bounded score memory and must never allocate a prompt-squared score matrix. Generate reproducible exact-count fixtures with `scripts/make-long-prompt.sh MODEL.gguf TOKENS OUTPUT_FILE`. The generator derives chat-template overhead and repeated-line token contribution from the selected GGUF, then re-tokenizes the completed file and refuses to publish it unless the count is exact. Preserve its JSON receipt with the benchmark artifacts; the canonical 8K/32K/64K/128K files are byte-identical when regenerated from the current 12B or 31B tokenizer. Every long-prompt candidate also needs a repeated real-weight determinism gate. Run identical 2K and 4K logits at least twice and require bit-identical output before performance promotion. Then compare the first 32-token tile crossing the 4,096-token routing boundary against the deterministic rowwise oracle, using the ordinary argmax/top-10/correlation/NMSE thresholds below. A fast operator result is rejected if either repeated model output or the boundary differential fails. Before choosing a long-prompt kernel axis, profile one production 32-token tile at an authentic prefix position. The profiler builds the prefix once, performs one unmeasured profiled warmup, then uses exact KV checkpoints to repeat the same tile three times. It reports the nine fixed layer stages separately for sliding and full-attention layers: ```sh build/gem-long-prefill-stage-profile MODEL.gguf 4096 3 ``` Retain the raw JSON and `gem print-config`. Stage profiling changes encoder boundaries and is an attribution tool, not a throughput promotion source; use an interleaved full-graph A/B harness for the selected one-axis candidate. At 64K/128K, the same profiler may seed the allocated production KV cache with finite diagnostic bytes instead of spending hours evaluating a prefix: ```sh build/gem-long-prefill-stage-profile MODEL.gguf 131040 3 --seed-prefix ``` The receipt must report `prefix_mode: "seeded-kv"`, `prefix_builds: 0`, 50 sliding layers, ten full layers, three repetitions, and adequate timestamp coverage. It preserves the production 32-token graph and exact long-position routing, but the artificial KV contents make it stage-attribution evidence only. It cannot establish prompt throughput, TTFT, output quality, or promotion; rerun the selected candidate through the evaluated-prefix and full-graph gates. For same-process interleaved full-graph screening without rebuilding a 64K/128K prompt, diagnostic routes exposed by `gem-long-prefix-ab` may use a finite seeded KV state: ```sh GEM_LONG_PREFIX_SEED=1 GEM_LONG_PREFIX_CANDIDATE=CANDIDATE \ build/gem-long-prefix-ab MODEL.gguf 131040 3 ``` The output must report `prefix_mode: seeded-kv`, `prefix_builds: 0`, repeated-route determinism, cross-route logits, and three interleaved samples per route. This screen preserves the ordinary unprofiled graph and checkpoint rollback, but synthetic KV contents make it attribution evidence only. A promoted candidate still requires an evaluated prefix and the server gates. For the production BQ8/BK64 global-prefill route, `CANDIDATE=bq8` compares diagnostic selector 22 (the former row2 global prefill) against selector 19. Preserve the evaluated-prefix JSON, the post-promotion seeded routing check, and a 131328-context `gem print-config` receipt together. If the frozen BK64/BK128 operator gate advances, `CANDIDATE=bk128` compares production selector 19 against diagnostic selector 23 while holding SWA, decode, KV type, batching, and the shared prefix fixed: Run the retained operator artifacts through the fail-closed graph runner rather than invoking the harness directly: ```sh GEM_MODEL_SHA256=KNOWN_VERIFIED_MODEL_DIGEST \ scripts/run-bq8-bk128-graph-gate.sh \ artifacts/YYYY-MM-DD-bq8-bk128-clean MODEL.gguf \ artifacts/YYYY-MM-DD-bq8-bk128-graph-64k ``` Omit `GEM_MODEL_SHA256` to hash the model in the runner. Supplying a previously verified digest avoids making a multi-gigabyte hash pass part of the quiet-host window. The runner independently re-analyzes the raw operator directory and refuses to start unless it advances BK128. It records the native configuration and linkage plus preflight, continuous, and postflight load, AC-power, and thermal receipts. Any monitor violation invalidates the graph run. The receipt must name selector 19 as baseline and selector 23 as candidate, report one evaluated prefix build, bit-identical repeats within each route, matching argmax/top-10, correlation at least 0.9999, and NMSE no greater than 1e-4. The analyzer also requires the mean improvement to exceed 3% and the larger arm CV, with a positive paired 95% confidence lower bound (three pairs, Student-t df=2). Selector 23 is never production-eligible directly; only a passing graph gate permits changing selector 19's internal prefill dispatch, after which native correctness, the adjacent backend-op oracle, and the final server gate must be rerun. Convert either receipt into a structurally validated stage ranking with: ```sh scripts/analyze-prefill-stages.sh PROFILE.json ANALYSIS.json ``` Before adding device-specific counter instrumentation, record which counter sets and sampling boundaries the target Mac exposes. This standalone diagnostic only queries Metal capabilities and does not link the production engine: ```sh build/gem-metal-counter-info > ARTIFACT_DIR/metal-counter-info.json ``` Do not infer that occupancy, bandwidth, or limiter counters are programmatically available from their presence in Xcode or Instruments. Only counters named in this receipt can be sampled through the public `MTLCounterSampleBuffer` API on the target device. Keep stage timestamps as the portable fallback when a desired device counter is absent. For the fixed 31B/F32 128K attribution rung, capture the counter inventory, configuration, binary and model identities, linkage, seeded profile, and validated ranking together: ```sh GEM_MODEL_SHA256=EXPECTED_SHA256 scripts/run-prefill-stage-attribution.sh \ MODEL.gguf ARTIFACT_DIR ``` The runner refuses to overlap a native server. Run it before an experimental kernel gate so the largest production stage—not an already-selected candidate—determines the next optimization axis. The analyzer requires the fixed 60-layer/50-sliding/10-full graph and all three sample matrices. Its selected axis is a profiling hypothesis, never a promotion decision. For long-position generation, `gem-decode-profile` builds the populated prefix once, then uses an exact checkpoint rollback for its warmup and each sample: ```sh build/gem-decode-profile --model MODEL.gguf --context 65793 \ --position 65536 --rows 1 --repetitions 3 ``` For one-row runs, the same JSON also includes `production_execution`, measured through the ordinary one-command-buffer/one-encoder path rather than the stage-counter path. Its host phases partition wall time into prepare, encode, submit/wait, cleanup, and logits readback; GPU time is nested inside submit/wait. Use `mean_wall_minus_gpu_seconds` as the upper bound for host/control optimization. Do not subtract individual host phases from GPU time or treat a dirty-host seeded-KV run as a speed gate. The phase sum must match measured wall time within timer precision. Evaluated prefixes report progress to stderr every 4,096 completed tokens by default. The callback runs only after the existing production prefill batch completes; it does not split batches or add command buffers. Use `--progress-every N` to change the reporting interval or `--no-progress` for a silent runner. JSON remains isolated on stdout and records `prefix_build_wall_seconds` plus `prefix_tokens_per_second`; those prefix fields are operational evidence, while the checkpointed decode stage samples remain the kernel-attribution result. When iterating on dispatch geometry, `--seed-prefix` may initialize the complete KV allocation with finite diagnostic bytes and advance only the logical position. This avoids evaluating a 64K/128K prompt and preserves the production model graph, KV layout, layer routing, command-buffer structure, and requested live-token count: ```sh build/gem-decode-profile --model MODEL.gguf --context 131328 \ --position 131072 --rows 1 --repetitions 3 --seed-prefix ``` The JSON must report `prefix_mode: "seeded-kv"` and `prefix_builds: 0`. Seeded-KV results are attribution and screening evidence only: their cache contents are not produced by Gemma, so they cannot support correctness, generation quality, prompt throughput, TTFT, or kernel promotion. A candidate selected with this shortcut must still pass an evaluated-prompt profile at the same position and every ordinary model/server promotion gate. Use the synthetic exact-shape profiler to divide the same production attention route into score, normalization, and value phases without rebuilding a model prefix: ```sh build/gem-decode-attention-profile --shape 31b-full --tokens 65536 \ --iterations 20 --repetitions 7 --warmup 10 ``` Value-dimension experiments must name the one changed kernel axis. Bracket the exact 31B full shape in A/B/B/A order and retain every raw JSON object: ```sh build/gem-decode-attention-profile --shape 31b-full --kernel dim64 --tokens 65536 \ --iterations 20 --repetitions 7 --warmup 10 build/gem-decode-attention-profile --shape 31b-full --kernel dim32 --tokens 65536 \ --iterations 20 --repetitions 7 --warmup 10 ``` Selector 18 was promoted after its correctness and populated-position gates. Use the override only to reproduce that candidate receipt explicitly; omitting it now exercises the same production default: ```sh build/gem-decode-profile --model MODEL.gguf --attention-kernel dim32 \ --context 65792 --position 65536 --rows 1 --repetitions 3 ``` The synthetic phase profile selects an operator axis; it cannot establish a whole-token ceiling or promote a kernel. Retain both raw outputs and `gem print-config`, and require the populated full-model profile to show at least a 3% end-to-end ceiling before implementation. Calculate that ceiling reproducibly with the matching populated-position and exact-shape receipts: ```sh scripts/analyze-decode-residual.sh FULL_MODEL_PROFILE.json OPERATOR_PROFILE.json REPORT.json ``` The analyzer binds the positions, native workloads, 60-layer/one-row model geometry, minimum sample counts, and at least 95% stage coverage. For Gemma 4 31B it attributes exactly ten full-attention layers, reports the theoretical whole-token P*V ceiling and the value-kernel gain needed to clear the 3% floor, and rejects attribution that exceeds authentic KV-attention time. An admissible result authorizes only a one-axis operator screen; it is never kernel-promotion evidence. For the parked D512 fused split-KV diagnostic, bind both 64K and 128K selector-19 evaluated model profiles to the matching split16 exact-shape profiles: ```sh scripts/analyze-fused-split-kv-ceiling.sh \ FULL_MODEL_64K.json OPERATOR_64K.json \ FULL_MODEL_128K.json OPERATOR_128K.json REPORT.json ``` This analyzer fixes the 32-query/4-KV-head/D512 geometry and attributes exactly ten global layers. It reports complete-global-attention and score-only whole-token ceilings plus the within-stage gain needed to clear 3%. It records all input digests and rejects changed selectors, contexts, stage totals, or insufficient coverage. Its output is research admission evidence only. Before a fused Metal route exists, reproduce the independent CPU algebra boundary with: ```sh build/gem-online-merge-check --tokens 8191 --partitions 32 --seed 1 build/gem-online-merge-check --tokens 8192 --partitions 32 --seed 1 build/gem-online-merge-check --tokens 8193 --partitions 32 --seed 1 build/gem-online-merge-check --tokens 65536 --partitions 32 --seed 1 build/gem-online-merge-check --tokens 131072 --partitions 32 --seed 1 scripts/analyze-online-merge-boundary.sh \ 8191.json 8192.json 8193.json 65536.json 131072.json audit.json ``` The audit requires D512, P=32, deterministic repeated hashes, correlation at least 0.9999, NMSE at most 1e-4, and the exact three boundary plus two long-tail positions. It is a CPU reduction-algebra prerequisite only and is never a Metal correctness, model-continuation, timing, or promotion result. After the diagnostic Metal pipelines compile, record their resource receipt and fail closed before dispatching an operator benchmark: ```sh scripts/analyze-fused-split-kv-resources.sh RESOURCE_RECEIPT.json RESOURCE_ANALYSIS.json ``` The receipt must identify the M4 Max/Apple-family-9 device, D512/P32/tile4/F32/no-window geometry, P=32 scratch bytes, compiled first-pass and reducer resource limits, and SHA-256 identities for the assembled Metal source and native binary. It requires the first pass to dispatch 256 threads/eight SIMD groups with 16 KiB dynamic threadgroup memory, and the reducer to dispatch 256 threads with no dynamic threadgroup memory. This is a compilation/resource screen only; an Xcode occupancy trace, raw GPU timestamps, CPU/selector-19 differentials, and authentic model gates remain mandatory. After compiled resources pass, bind correctness evidence before collecting performance numbers: ```sh scripts/analyze-fused-split-kv-correctness.sh \ CPU_GATE.json RESOURCE_ANALYSIS.json \ OP_8191.json OP_8192.json OP_8193.json OP_65536.json OP_131072.json \ MODEL_8191.json MODEL_8192.json MODEL_8193.json CORRECTNESS_ANALYSIS.json ``` The operator inputs must compare the fixed diagnostic route against forced selector 19 at all five CPU-gated lengths. The three model inputs must use evaluated prompts and prove matching argmax, ordered top-10, and at least eight identical greedy continuation tokens. Correlation must be at least 0.9999 and NMSE no greater than 1e-4 throughout. The analyzer records every input digest and always reports `promotion_eligible:false`; only subsequent clean-host whole-token gates can promote. For 31B SWA routing changes, locate persistent-state divergence before accepting aggregate logit similarity. `gem-swa-state-scan` advances rowwise and candidate engines in lockstep and hashes every new logical K/V row after each 32-token tile. The empty-cache tile is a distinct mandatory case: ```sh build/gem-swa-state-scan MODEL.gguf 4096 build/gem-long-prefix-ab MODEL.gguf 4096 3 build/gem-long-prefix-ab MODEL.gguf 32768 3 ``` The state scan must report exact K/V through all requested tokens. The A/B harness uses one shared production prefix with exact checkpoint rollback, alternates route order, includes the final logits projection, requires repeated route logits to be bit-identical, and reports the rowwise versus batched-SWA tile delta. For actual streaming TTFT, use `scripts/server-stream-probe.sh BASE_URL MODEL PROMPT_FILE OUTPUT_SSE OUTPUT_JSON`. Ordinary curl `time_starttransfer` observes the immediate HTTP headers and assistant-role event, not model output. The streaming probe records both that transport value and the first SSE delta containing non-empty content, non-empty reasoning, or a non-empty tool-call array; only `first_visible_event_seconds` is the model-visible TTFT KPI. For a capacity-edge prompt, set `GEM_STREAM_MAX_TOKENS=16` (or another explicitly recorded positive limit) so the request leaves enough tokens to pass suppressed Gemma channel pieces without exceeding the configured context; the probe records `requested_max_tokens` in its JSON. Use `scripts/long-context-server-gate.sh MODEL.gguf CONTEXT PROMPT_FILE OUTPUT_DIR PORT` for the required three-sample gate. It starts a fresh native server for every sample and owns its cleanup; this is required because repeated requests to one process can reuse the prefix cache and cease to measure full prompt evaluation. The runner verifies the canonical chat token count and aggregates that every server-reported `usage.prompt_tokens` value exactly matches that pre-count before it aggregates first-visible-event and completion latency, while retaining each raw SSE stream, server log, and validated `gem print-config`. Set `GEM_KV_TYPE=f16` or `GEM_KV_TYPE=f16-full` to gate an opt-in cache layout; the default is `f32`, and the selected type is recorded in the summary. The runner records SHA-256 digests for the GGUF and prompt before timing begins; set `GEM_MODEL_SHA256` to a previously verified 64-digit digest to avoid rereading the same multi-gigabyte model at every rung. The retained server log must show batch-boundary prefill progress every 4,096 tokens for these long requests. SSE keepalive comments are emitted after every completed production batch; the streaming probe excludes them from `first_visible_event_seconds`, while a failed write cancels remaining prompt evaluation before the next batch is submitted. A sample is incomplete unless its retained log contains the exact 100% prefill record; prefix-extension samples require the corresponding final logical token count as well. The same requirement applies when resuming prior samples. After changing prompt-progress or cancellation behavior, run one authentic same-process recovery gate without overlapping any performance benchmark: ```sh scripts/server-prefill-cancel-gate.sh MODEL.gguf CONTEXT PROMPT_FILE OUTPUT_DIR PORT ``` The prompt must contain at least 128 canonical chat tokens. The runner requires curl timeout status 28, at least one SSE keepalive, health recovery within `GEM_CANCEL_RECOVERY_LIMIT_SECONDS` (60 by default), and a successful visible completion from a 16-token second request to the same native process. The cancelled prompt must not reach its exact 100% prefill log record. Set `GEM_CANCEL_AFTER_SECONDS` explicitly when a single prefill batch on the target hardware exceeds the five-second default. Retain the config, request, partial SSE, curl error, server logs, recovery measurement, and summary. The CPU-only `gem.server_prefill_cancel_gate` CTest exercises the complete runner against a serial HTTP/SSE fixture; it is orchestration coverage, while the GGUF-backed command remains the behavioral gate. If a multi-hour gate stops after one or more complete repetitions, rerun the identical command with `GEM_GATE_RESUME=1`. The output directory must already contain both `gate-config.json` and `print-config.json`. Resume is fail-closed: it compares the current canonical native configuration and the complete model/prompt digests, paths, context, KV type, token limit, and prefix-extension text; then it reuses only measurement/SSE/log tuples that independently pass the ordinary response and exact-token validations. An incomplete repetition starts a new server process. A changed configuration, changed provenance, malformed completed sample, or occupied port is never silently reused. The final summary records `resumed_samples`. The CPU-only `gem.long_context_server_gate_resume` CTest locks this contract down by exercising a three-sample resume and fail-closed configuration/provenance changes without starting the native server or loading a GGUF. For prompts of at least 65,536 tokens, the runner also applies a recorded host-state policy before each sample. It captures the one-minute load average, logical CPU count, power source, thermal status, active power assertions, and the busiest processes in `host-preflight-N.json`; by default it refuses to start above 1.0 runnable task per logical CPU or when a one-second system sample exceeds 10% busy CPU. The short-window guard closes the load-average lag window without treating macOS `ps %CPU`—a decaying average over up to one minute—as instantaneous. Schema 4 additionally requires AC power and no thermal/performance warning at preflight, then records load, power, and thermal state every 30 seconds. It aborts after three consecutive violations, while the auditor rejects any retained schema-4 sample containing even one overloaded, battery-power, or thermal-warning monitor record. The server lifetime is held under `caffeinate -i -w` so idle sleep cannot masquerade as kernel time. The threshold, interval, failure count, and assertion are part of gate provenance. Override `GEM_GATE_PREFLIGHT_MIN_TOKENS`, `GEM_GATE_MAX_LOAD_PER_CPU`, `GEM_GATE_MAX_SHORT_WINDOW_BUSY_CPU_PERCENT`, `GEM_GATE_HOST_MONITOR_SECONDS`, or `GEM_GATE_HOST_MONITOR_FAILURES` only as a separately recorded benchmark axis. Gate schema 2 adds the load/monitor contract, schema 3 adds short-window CPU sampling, and schema 4 makes power and thermal state enforceable preflight and continuous-monitor controls. Older artifacts remain auditable but intentionally cannot resume into a newer-schema run. Cold-prefill results do not cover the dominant multi-turn hax case. After the fresh-process 64K and 128K gates pass, rerun the same harness with (for example) `GEM_PREFIX_EXTENSION_TEXT='Reply with DONE.'`. Each repetition keeps its server alive for a second canonical chat request whose token sequence strictly extends the first request boundary. Retain both SSE streams and the server log, require the log to report the exact reused/appended token counts, and record second-turn first-visible-event and completion latency. The extension gate must evaluate only the appended suffix; a cache miss or an identical replay is a failure. Report this separately from cold prefill—it demonstrates that a long conversation is amortized after its first turn, not that initial 64K/128K ingestion is free. After the three-process gate completes, independently recompute its aggregates and validate every raw cold/prefix measurement, SSE terminator, exact 100% progress record, prefix-hit record, prompt digest, and frozen production-binary digest: ```sh scripts/audit-long-context-server-gate.sh OUTPUT_DIR build/gem > OUTPUT_DIR/audit.json ``` Do not use `summary.json` alone as acceptance evidence. The audit must pass against the raw files while `build/gem` is still the frozen binary recorded when the gate began. The runner writes `live-binary-sha256.txt` before its first sample and refuses resume if the current production binary differs. The auditor derives the expected F32 KV bytes from the configured context and fixed 31B topology: 50 layers bounded to `min(context, 1024)` with 16 KV heads at dimension 256, plus ten context-growing layers with four KV heads at dimension 512, storing K and V. ## Profiling trigger Profile only a candidate that clears the 3% threshold against baseline outside run-to-run variance. Keep the output in a timestamped directory outside version control: ```sh scripts/profile-metal.sh \ build/gem \ /models/gemma-4-31B-it-Q4_0.gguf \ artifacts/YYYY-MM-DD-metal-profile/production.trace ``` The wrapper honors an explicit `DEVELOPER_DIR` and otherwise selects an installed stable or beta Xcode when the active developer directory contains only Command Line Tools. It does not change the machine-wide `xcode-select` setting. To capture an already-gated diagnostic executable instead of the production server, pass the trace path followed by `--` and the exact command: ```sh scripts/profile-metal.sh artifacts/YYYY-MM-DD-metal-profile/candidate.trace -- \ build/gem-long-attention-profile --shape 31b --position 131040 --rows 32 \ --repetitions 3 --warmup 1 --variant online-row4 ``` The default template is `Metal System Trace`. Set `GEM_XCTRACE_TEMPLATE` to an installed Xcode template such as `Game Performance` when the experiment requires its aggregate GPU counters, and record the selected template beside the trace. Set `GEM_XCTRACE_INSTRUMENT='Metal GPU Counters'` to add shader-PC sampling; the target M4 capture reports counter profile 3 only with that explicit instrument. For a focused GPU document, the long-attention profiler can capture one additional untimed dispatch after all reported samples complete: ```sh MTL_CAPTURE_ENABLED=1 \ GEM_METAL_CAPTURE_PATH=artifacts/YYYY-MM-DD-metal-profile/row2.gputrace \ build/gem-long-attention-profile --shape 31b --position 4096 --rows 32 \ --repetitions 3 --warmup 1 --variant online-row2 ``` The JSON must report `capture:true`. Do not use capture execution as benchmark evidence. Prefer a bounded position for occupancy, resource, spill, and heat-map inspection because GPU documents serialize bound buffers; use the 128K system trace for long-context timing and memory behavior. Capture the exact model hash, gem commit, reference llama.cpp commit, native configuration, macOS build, power source, temperature/power mode, and benchmark raw JSON beside the trace. Use the Metal trace to name the dominant kernel and dispatch shape before attempting a kernel change. ## Correctness and promotion gates Before model-level optimization, capture a real-weight first-token differential for both supported model sizes. This uses the adjacent llama.cpp checkout only as an offline oracle and compares the entire native 262,144-entry F32 logit vector after first confirming exact token IDs: ```sh scripts/real-logit-oracle.sh .state/models/gemma-4-12B-it-Q4_0.gguf 12b-logits.json scripts/real-logit-oracle.sh .state/models/gemma-4-31B-it-Q4_0.gguf 31b-logits.json ``` The gate requires finite vectors, equal argmax and top-10 sets, correlation at least 0.999, and NMSE at most 1e-3. Retain the emitted JSON; this is a correctness gate, not a performance comparison. - The requested GGUF passes the strict manifest, all layers execute on Metal, and hax completes a streamed tool-call chat. - The canonical Gemma 4 chat behavior and tokenizer golden vectors pass. - A candidate must be at least 3% faster on its declared primary KPI and not regress the other KPI by over 2% outside variance. - Any Metal-kernel change needs a native CPU differential operator test plus a deterministic model generation smoke test before benchmarking. - Never compare a different quantization, context, gem revision, or thermal state as though it were a kernel improvement. Metal backend startup is independently attributable with: ```sh build/gem-metal-startup-profile 3 ``` Each line is one JSON record containing device creation, source assembly, runtime library compilation, pipeline creation, command-queue creation, and total time. Repetitions in one process measure warm reopen behavior; use separate invocations for fresh-process readiness. These numbers do not replace end-to-end `gem agent` readiness measurements and are not a kernel promotion gate. ## Research queue 1. Validate tokenizer parity and first-token logits on official Q4_0 weights. 2. Compare direct GGUF Q4_0 against a trace-justified packed layout, if needed. 3. Native flash-style attention and KV representation under 32K and 128K context. 4. Speculative decoding using the Gemma 4 assistant architecture after baseline is stable. 5. Multimodal image prefill only after text-agent latency is established.