# Getting started

`gem` is a small, offline, spec-locked C inference engine for dense Gemma 4 **12B** and **31B**
on Apple Silicon. It owns GGUF loading, the fixed Gemma 4 graph, tokenizer, KV cache, sampling,
Metal dispatch, and a minimal **OpenAI-compatible HTTP/SSE server** for hax. It never links
libllama/ggml and never launches `llama-server`.

## Models

Point `gem` at a GGUF by **path** or by **name**:

```sh
build/gem generate --model /path/to/gemma-4-31B-it-Q4_0.gguf --prompt Hello
build/gem generate --model gemma-4-31B-it-Q4_0 --prompt Hello      # name resolution
```

Name resolution order: an existing file path as given → `GEM_MODEL_DIR` (colon-separated
directories) → the default `<repo>/.state/models` → the local HuggingFace cache
(`HF_HUB_CACHE`, else `$HF_HOME/hub`, else `~/.cache/huggingface/hub`). A `.gguf` suffix is
appended automatically when the name lacks one. Environment: `GEM_MODEL_DIR`, `HF_HOME`,
`HF_HUB_CACHE`.

**Quantization requirement.** The engine's strict manifest accepts **Q4_0** projection weights
with a Q8_0 tied embedding (norms/scales F32) — exactly the layout of llm.cpp-style single-GGUF
conversions. Files in other quants are rejected at load with a clear message.

**12B serving status (currently: documented-unsupported).** The engine structurally supports
12B (48 layers, 667 tensors) and the metal path is correct, but Q4_0 quantization of the 12B
checkpoint provably flips the next-token logits into a degenerate high-ID band (oracle-validated
analysis in `../research/...` and `artifacts/2026-08-30-12b-diagnosis/DIAGNOSIS.md`). Correct 12B
serving requires Q8_0 projection support, tracked as an engine feature. 12B at Q4_0 is caught by
the `gem.12b_servability_gate` test and reported as unsupported. **31B Q4_0 is the validated
serving path.**

**MTP (speculative) sidecar.** With F32 KV, `gem agent` automatically discovers a colocated
`mtp-<model filename>` assistant. Use `--no-mtp` to force direct decode.

## Commands

```sh
build/gem print-config --model MODEL        # resolved engine/architecture JSON
build/gem generate  --model MODEL --prompt TEXT [--context N] [--max-tokens N]
                    [--temperature F] [--top-k N] [--top-p F] [--seed N]
                    [--command-span layer|token] [--kv-type f32|f16|f16-full]
build/gem serve     --model MODEL [--alias NAME] [--port N] [--mtp ASSISTANT] ...
build/gem agent     --model MODEL [--mtp|--no-mtp] [--hax PATH] [-- HAX_ARGS]
build/gem-inspect   [--gguf|--tensors|--bindings] MODEL   # manifest/metadata
build/gem-logits    --model MODEL --prompt TEXT --context N --output FILE.bin
build/gem-bench     --model MODEL --context N --prompt-tokens N --generation-tokens N \
                    --repetitions 3 --command-span token --kv-type f32
```

## OpenAI server

`gem serve` (or `gem agent`, which owns the lifecycle and spawns hax) binds **127.0.0.1**
(default port 8080) and serves:

| Endpoint | Purpose |
|---|---|
| `GET /health` | liveness |
| `GET /v1/models` | the loaded alias |
| `GET /props` | engine properties |
| `POST /v1/chat/completions` | OpenAI chat completions (SSE stream + non-stream) |

Requests accept `model`, `messages`, `tools`, `stream`, `max_tokens`/`max_completion_tokens`,
`temperature`, `top_p`. Responses include `id`, `choices` (with `delta.tool_calls` when
streaming), `finish_reason` (`stop`/`tool_calls`), and `usage`.

There is no auth or CORS: it is a localhost service, one model per server. hax connects with
`HAX_LLAMACPP_BASE_URL=http://127.0.0.1:8080/v1` (`llamacpp` is hax's protocol-selector label;
no llama.cpp participates).

## Typical flows

```sh
build/gem agent --model gemma-4-31B-it-Q4_0 -- -p 'inspect this project'   # offline hax, preferred
build/gem serve  --model /path/to/model.gguf --alias gemma-4-31b            # two-terminal debug
```

## Troubleshooting

- **"model not found"** — no path matched: check `GEM_MODEL_DIR` and that the name resolves in
  `.state/models` or the HF cache.
- **"missing, type-incompatible tensor"** — the file's quant/layout isn't the supported
  Q4_0-projection/Q8_0-embedding contract (e.g., a Q8_0 model passes inspect `--tensors` but
  fails load).
- **12B output degenerates (e.g. repeated digits / template tokens)** — documented-unsupported at
  Q4_0; see `scripts/remediate-12b.sh --verify` and the diagnosis above.
- **port already in use** — pick `--port` or stop the old server.
- **large model footprint** — weights are mmap'd no-copy; the HF cache holds the actual blobs
  (a gitignored cache, safe to let `hf` manage).
- **performance questions** — follow `docs/build/benchmark-protocol.md` before drawing conclusions;
  the project's results live in `artifacts/` and project memory (`scripts/yams search --cwd --query ...`).

Build details: [`docs/build/build-guide.md`](../build/build-guide.md). Architecture and research:
[`docs/research/`](../research/).