101 lines
3 KiB
Markdown
101 lines
3 KiB
Markdown
# Embeddings & FTS5: Auto Mode and Repair
|
|
|
|
## Overview
|
|
- YAMS can automatically generate embeddings and index document content on add.
|
|
- Embeddings are stored in the vector database; text content is indexed in SQLite FTS5.
|
|
- Both paths are best-effort and non-blocking to keep ingestion fast.
|
|
|
|
## Auto Embeddings on Add
|
|
|
|
Toggle via config in `~/.config/yams/config.toml`:
|
|
|
|
```toml
|
|
[embeddings]
|
|
auto_on_add = true # Queue background embedding on add
|
|
preferred_model = "all-MiniLM-L6-v2" # Model name
|
|
embedding_dim = 384 # Must match model output dimensions
|
|
```
|
|
|
|
**Behavior:**
|
|
- `yams add` and daemon directory indexing call embeddings asynchronously when enabled.
|
|
- Extraction + FTS5 indexing still run immediately during add (for supported formats).
|
|
|
|
## FTS5 Indexing
|
|
- When available in the SQLite build, YAMS indexes extracted text into `documents_fts`.
|
|
- DocumentService indexes FTS5 during add; directory add uses the same storage path.
|
|
- Searching uses hybrid keyword (FTS5) + semantic (vector) ranking when configured.
|
|
|
|
## Repair and Rebuild
|
|
|
|
CLI supports targeted repair flows:
|
|
|
|
```bash
|
|
# Generate missing embeddings for stored documents
|
|
yams repair --embeddings
|
|
|
|
# Rebuild FTS5 entries (delete/insert) using robust extraction
|
|
yams repair --fts5
|
|
|
|
# Build/repair knowledge graph from tags/metadata
|
|
yams doctor repair --graph
|
|
|
|
# Run all repair operations
|
|
yams repair --all
|
|
```
|
|
|
|
The daemon repair service also performs best-effort FTS5 reindex for documents it fixes embeddings for.
|
|
|
|
## Model Management
|
|
|
|
```bash
|
|
# List available models
|
|
yams model list
|
|
|
|
# Download a model
|
|
yams model download all-MiniLM-L6-v2
|
|
|
|
# Check ONNX runtime and plugin status
|
|
yams model check
|
|
|
|
# Set preferred model in config
|
|
yams config set embeddings.preferred_model all-MiniLM-L6-v2
|
|
```
|
|
|
|
**Model resolution order:**
|
|
1. `embeddings.model_path` in config
|
|
2. `~/.yams/models/<name>/model.onnx`
|
|
3. `models/` in current directory
|
|
4. `/usr/local/share/yams/models`
|
|
|
|
## Embedding Dimension Source of Truth
|
|
|
|
**Single key:** Set `embeddings.embedding_dim` in `~/.config/yams/config.toml`.
|
|
|
|
**Runtime precedence:** config > env (`YAMS_EMBED_DIM`) > generator > heuristic.
|
|
|
|
The daemon derives vector DB schema and in-memory index dimensions from this single value to prevent drift.
|
|
|
|
## Notes
|
|
- Embedding and FTS5 operations degrade gracefully: failures are logged and skipped.
|
|
- Batch sizes and retries are conservative to avoid blocking foreground operations.
|
|
|
|
## Troubleshooting
|
|
|
|
**Plugin not loaded:**
|
|
```bash
|
|
yams plugin list # Check loaded plugins
|
|
yams plugin trust add ~/.local/lib/yams/plugins # Trust plugin directory
|
|
yams doctor plugin onnx # Diagnose ONNX plugin
|
|
```
|
|
|
|
**Dimension mismatch:**
|
|
```bash
|
|
yams doctor # Shows vector DB dim vs model target
|
|
yams doctor --recreate-vectors --dim 384 # Recreate with correct dimension
|
|
```
|
|
|
|
**Missing embeddings:**
|
|
```bash
|
|
yams repair --embeddings # Generate missing embeddings
|
|
yams stats -v # Check embedding coverage
|
|
```
|