yams/docs/user_guide
Trevon 3f37e8f564
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector)!: remove dead transition APIs

Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.

BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): harden PQ snapshot lifecycle

Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): expose route coordinates

Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.

Signed-off-by: trvon <me@trevon.dev>

* feat(xplan): add vector decision gates

Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): formalize coordinate routing

Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): serialize SymSpell readers

SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): make mutations atomic

Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.

* fix(search)!: surface SymSpell write errors

Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.

BREAKING CHANGE: SymSpellSearch mutations now return Result values.

* feat(topology): formalize candidate expansion

* feat(vector): add coordinate-aware embeddings

* feat(topology): preserve protected relation cover

* feat(search): add graph candidate rescue

* feat(daemon): expose retrieval hydration telemetry

* feat(xplan): add rescue and hydration gates

* refactor(content): dedupe media handlers

Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.

* feat(lifecycle): prune stale corpus data

Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.

* feat(search): add topology candidate rescue

Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.

* feat(search): expose evidence paths

* docs: define mobile host contract

* formal: model bounded candidate rescue

* fix(prune): inspect extensionless signatures

* fix(prune): tolerate stale extensions

---------

Signed-off-by: trvon <me@trevon.dev>
2026-07-24 19:00:22 -04:00
..
cli.md Sync experimental into main: Search retrieval improvements (#60) 2026-07-24 19:00:22 -04:00
embeddings.md fix: fixing semantic drift of readiness system, improving cli routing to reduce churn to proxy socket and improving daemon lifecycle logic 2026-03-20 10:12:42 -04:00
installation.md feat(packaging): mature systemd service install + local validation 2026-06-24 12:57:16 -04:00
mcp.md fix(audit): remove no-op storage and metadata paths found in audit, use file-type detection for buffer handlers without, breaking generic multimedia fallback, and sync plugin and operator docs with shipped behavior 2026-06-27 10:50:59 -04:00
README.md fix(testing): stabilize TSAN coverage, harden daemon shutdown, and speed up KG/storage paths (#53) 2026-06-26 17:57:02 -04:00

YAMS User Guide

Command-line interface for content-addressed storage with semantic search.

Quick Start

# Initialize storage (interactive - prompts for grammar downloads)
yams init

# Or auto mode for containers/headless
yams init --auto

# Or specify custom location
export YAMS_STORAGE="$HOME/.local/share/yams"
yams init --non-interactive

# Init also bootstraps a per-project session (scoping + watch) unless

# Enable auto-ingest for an existing project
yams watch

# Add content
yams add ./README.md --tags docs
yams add src/ --recursive --include="*.cpp,*.h" --tags code
echo "note" | yams add - --name "quick-note.txt"

# Search and retrieve
yams search "config file" --limit 5
yams grep "TODO" --include="*.cpp"
yams list --limit 20
yams get <hash> -o ./output.bin

# System status
yams status
yams doctor

Documentation

Topic File Description
Installation installation.md Install via Homebrew, Docker, or source
CLI Reference cli.md All commands and options
MCP Server mcp.md Model Context Protocol integration
Embeddings embeddings.md Vector search and model setup
Operations ../admin/operations.md Tuning, maintenance, configuration

Key Commands

Command Description
yams init Initialize storage (interactive or --auto)
yams add Add files/directories (--recursive, --include, --tags)
yams search Hybrid search (keyword + semantic)
yams grep Regex search with semantic suggestions
yams list List documents (--snapshots, --format json)
yams get Retrieve by hash or name
yams diff Compare snapshots (Merkle tree diff)
yams watch Enable auto-ingest for a project session
yams serve Start MCP server (stdio)
yams doctor Diagnose issues
yams repair Fix storage/embeddings

Build Options (Meson)

Option Default Description
build-cli true Build CLI binary
build-mcp-server true Build MCP server
build-tests false Build test suite
plugin-onnx true ONNX embedding models
plugin-symbols true Tree-sitter symbol extraction
plugin-s3 true S3 object storage

Environment Variables

Variable Description
YAMS_STORAGE Storage root directory (overrides default)
XDG_DATA_HOME Default storage location (~/.local/share/yams)
XDG_CONFIG_HOME Config location (~/.config/yams)
YAMS_PLUGIN_DIR Plugin search directory
YAMS_DAEMON_SOCKET Daemon socket path override
YAMS_EMBEDDED Daemon transport mode: 1/true in-process, 0/false socket, auto probe

Common Patterns

# JSON output for scripting
yams list --format json | jq '.'
yams search "query" --limit 10 --json

# Per-project storage
YAMS_STORAGE="$PWD/.yams" yams init --non-interactive
YAMS_STORAGE="$PWD/.yams" yams add ./src/

# Auto-ingest for the current project
yams watch --interval 2000

# Batch indexing with tags
yams add src/ --recursive --include="*.cpp,*.hpp,*.h" --tags "code,source"
yams add docs/ --recursive --include="*.md" --tags "docs"

# Snapshot workflow
yams add . --recursive --include="*.cpp,*.h" --snapshot-label "v1.0"
yams list --snapshots
yams diff v1.0 v1.1 --stats

# MCP server for AI assistants
yams serve

Embedding Models

Model Dimensions Notes
all-MiniLM-L6-v2 384 Recommended: fast, lightweight
all-mpnet-base-v2 768 Higher quality, slower
# Download and configure a model
yams model download all-MiniLM-L6-v2
yams model check

# Set preferred model
yams config set embeddings.preferred_model all-MiniLM-L6-v2

Symbol Extraction

Extract functions, classes, and methods from source code:

# Download grammars
yams config grammar list
yams config grammar download cpp python rust

# Symbols are extracted automatically on add
yams add src/ --recursive --include="*.cpp,*.py,*.rs"

Supported languages (18): C, C++, C#, Dart, Go, Java, JavaScript, Kotlin, P4, Perl, PHP, Python, R, Rust, Solidity, SQL, TypeScript, Zig. Full list: cli.md#symbol-extraction.

Plugins

yams plugin list                              # List loaded
yams plugin trust add ~/.local/lib/yams/plugins
yams plugin health                            # Check status
yams doctor plugin onnx                       # Diagnose

Troubleshooting

yams doctor              # Full diagnostics
yams doctor --json       # JSON output
yams stats --verbose     # Storage statistics
yams repair --all        # Fix common issues
yams repair --embeddings # Generate missing embeddings