96 lines
4.8 KiB
Markdown
96 lines
4.8 KiB
Markdown
# Getting started
|
|
|
|
`gem` is a small, offline, spec-locked C inference engine for dense Gemma 4 **12B** and **31B**
|
|
on Apple Silicon. It owns GGUF loading, the fixed Gemma 4 graph, tokenizer, KV cache, sampling,
|
|
Metal dispatch, and a minimal **OpenAI-compatible HTTP/SSE server** for hax. It never links
|
|
libllama/ggml and never launches `llama-server`.
|
|
|
|
## Models
|
|
|
|
Point `gem` at a GGUF by **path** or by **name**:
|
|
|
|
```sh
|
|
build/gem generate --model /path/to/gemma-4-31B-it-Q4_0.gguf --prompt Hello
|
|
build/gem generate --model gemma-4-31B-it-Q4_0 --prompt Hello # name resolution
|
|
```
|
|
|
|
Name resolution order: an existing file path as given → `GEM_MODEL_DIR` (colon-separated
|
|
directories) → the default `<repo>/.state/models` → the local HuggingFace cache
|
|
(`HF_HUB_CACHE`, else `$HF_HOME/hub`, else `~/.cache/huggingface/hub`). A `.gguf` suffix is
|
|
appended automatically when the name lacks one. Environment: `GEM_MODEL_DIR`, `HF_HOME`,
|
|
`HF_HUB_CACHE`.
|
|
|
|
**Quantization requirement.** The engine's strict manifest accepts **Q4_0** projection weights
|
|
with a Q8_0 tied embedding (norms/scales F32) — exactly the layout of llm.cpp-style single-GGUF
|
|
conversions. Files in other quants are rejected at load with a clear message.
|
|
|
|
**12B serving status (currently: documented-unsupported).** The engine structurally supports
|
|
12B (48 layers, 667 tensors) and the metal path is correct, but Q4_0 quantization of the 12B
|
|
checkpoint provably flips the next-token logits into a degenerate high-ID band (oracle-validated
|
|
analysis in `../research/...` and `artifacts/2026-08-30-12b-diagnosis/DIAGNOSIS.md`). Correct 12B
|
|
serving requires Q8_0 projection support, tracked as an engine feature. 12B at Q4_0 is caught by
|
|
the `gem.12b_servability_gate` test and reported as unsupported. **31B Q4_0 is the validated
|
|
serving path.**
|
|
|
|
**MTP (speculative) sidecar.** With F32 KV, `gem agent` automatically discovers a colocated
|
|
`mtp-<model filename>` assistant. Use `--no-mtp` to force direct decode.
|
|
|
|
## Commands
|
|
|
|
```sh
|
|
build/gem print-config --model MODEL # resolved engine/architecture JSON
|
|
build/gem generate --model MODEL --prompt TEXT [--context N] [--max-tokens N]
|
|
[--temperature F] [--top-k N] [--top-p F] [--seed N]
|
|
[--command-span layer|token] [--kv-type f32|f16|f16-full]
|
|
build/gem serve --model MODEL [--alias NAME] [--port N] [--mtp ASSISTANT] ...
|
|
build/gem agent --model MODEL [--mtp|--no-mtp] [--hax PATH] [-- HAX_ARGS]
|
|
build/gem-inspect [--gguf|--tensors|--bindings] MODEL # manifest/metadata
|
|
build/gem-logits --model MODEL --prompt TEXT --context N --output FILE.bin
|
|
build/gem-bench --model MODEL --context N --prompt-tokens N --generation-tokens N \
|
|
--repetitions 3 --command-span token --kv-type f32
|
|
```
|
|
|
|
## OpenAI server
|
|
|
|
`gem serve` (or `gem agent`, which owns the lifecycle and spawns hax) binds **127.0.0.1**
|
|
(default port 8080) and serves:
|
|
|
|
| Endpoint | Purpose |
|
|
|---|---|
|
|
| `GET /health` | liveness |
|
|
| `GET /v1/models` | the loaded alias |
|
|
| `GET /props` | engine properties |
|
|
| `POST /v1/chat/completions` | OpenAI chat completions (SSE stream + non-stream) |
|
|
|
|
Requests accept `model`, `messages`, `tools`, `stream`, `max_tokens`/`max_completion_tokens`,
|
|
`temperature`, `top_p`. Responses include `id`, `choices` (with `delta.tool_calls` when
|
|
streaming), `finish_reason` (`stop`/`tool_calls`), and `usage`.
|
|
|
|
There is no auth or CORS: it is a localhost service, one model per server. hax connects with
|
|
`HAX_LLAMACPP_BASE_URL=http://127.0.0.1:8080/v1` (`llamacpp` is hax's protocol-selector label;
|
|
no llama.cpp participates).
|
|
|
|
## Typical flows
|
|
|
|
```sh
|
|
build/gem agent --model gemma-4-31B-it-Q4_0 -- -p 'inspect this project' # offline hax, preferred
|
|
build/gem serve --model /path/to/model.gguf --alias gemma-4-31b # two-terminal debug
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
- **"model not found"** — no path matched: check `GEM_MODEL_DIR` and that the name resolves in
|
|
`.state/models` or the HF cache.
|
|
- **"missing, type-incompatible tensor"** — the file's quant/layout isn't the supported
|
|
Q4_0-projection/Q8_0-embedding contract (e.g., a Q8_0 model passes inspect `--tensors` but
|
|
fails load).
|
|
- **12B output degenerates (e.g. repeated digits / template tokens)** — documented-unsupported at
|
|
Q4_0; see `scripts/remediate-12b.sh --verify` and the diagnosis above.
|
|
- **port already in use** — pick `--port` or stop the old server.
|
|
- **large model footprint** — weights are mmap'd no-copy; the HF cache holds the actual blobs
|
|
(a gitignored cache, safe to let `hf` manage).
|
|
- **performance questions** — follow `docs/build/benchmark-protocol.md` before drawing conclusions;
|
|
the project's results live in `artifacts/` and project memory (`scripts/yams search --cwd --query ...`).
|
|
|
|
Build details: [`docs/build/build-guide.md`](../build/build-guide.md). Architecture and research:
|
|
[`docs/research/`](../research/).
|