Trevon bb1041656d

v0.1.0

2025-08-12 20:52:21 -04:00

12 KiB

Raw Permalink Blame History

YAMS - Yet Another Memory System

Persistent memory for LLMs and applications. Content-addressed storage with deduplication, semantic search, and full-text indexing.

Features

Content-Addressed Storage - SHA-256 based, ensures data integrity
Deduplication - Block-level with Rabin fingerprinting
Compression - Zstandard and LZMA with intelligent policies
Search - Full-text (SQLite FTS5) and semantic (vector embeddings)
Crash Recovery - Write-ahead logging for durability
High Performance - 100MB/s+ throughput, thread-safe

My prompt for CLI usage is PROMPT.md and PROMPT-eng.md for programming.

Build

Requirements

C++20 compiler (GCC 11+, Clang 14+, AppleClang 14+)
CMake 3.20+
Python 3.8+ (for Conan)

Quick Start

macOS

# Install Conan
pip install conan

# One-time: create default Conan profile
conan profile detect --force

# Build with Conan
conan install . --build=missing -s build_type=Release

cmake --preset conan-release
cmake --build --preset conan-release -j
sudo cmake --install build/Release

Linux

# Install dependencies
sudo apt-get update
sudo apt-get install -y cmake ninja-build pkg-config git
pip install conan

# One-time: create default Conan profile
conan profile detect --force

# Build with Conan
conan install . \
  --output-folder=build/conan-ninja-release \
  -s build_type=Release \
  --build=missing

cmake --preset conan-ninja-release
cmake --build build/conan-ninja-release -j
sudo cmake --install build/conan-ninja-release

Build Options

Option	Default	Description
`YAMS_USE_CONAN`	OFF	Use Conan package manager
`YAMS_BUILD_CLI`	ON	CLI with TUI browser
`YAMS_BUILD_MCP_SERVER`	ON	MCP server (requires Boost)
`YAMS_BUILD_TESTS`	OFF	Unit and integration tests
`YAMS_BUILD_BENCHMARKS`	OFF	Performance benchmarks
`YAMS_ENABLE_PDF`	ON	PDF text extraction support
`CMAKE_BUILD_TYPE`	Release	Debug/Release/RelWithDebInfo

Dependencies (Optional - only if not using Conan)

When using Conan (recommended), dependencies are managed automatically. If building without Conan:

# macOS
brew install openssl@3 protobuf sqlite3 ncurses
export OPENSSL_ROOT_DIR=$(brew --prefix openssl@3)

# Linux (Ubuntu/Debian)
sudo apt install libssl-dev libsqlite3-dev protobuf-compiler libncurses-dev

Setup

# Initialize with XDG defaults (non-interactive)
yams init --non-interactive

# Optional: custom storage root
export YAMS_STORAGE="$HOME/.local/share/yams"
yams init --non-interactive

# Print resulting config (secrets masked)
yams init --non-interactive --print

Troubleshooting

If you see:

ERROR: The default build profile '/home/trevon/.conan2/profiles/default' doesn't exist.
You need to create a default profile (type 'conan profile detect' command)
or specify your own profile with '--profile:build=<myprofile>'

Fix:

# Create default profile
conan profile detect --force

# Optional: ensure C++20 in the default profile
# Linux/macOS (GNU sed):
sed -i 's/compiler.cppstd=.*/compiler.cppstd=20/' ~/.conan2/profiles/default || true
# macOS (BSD sed):
# sed -i '' 's/compiler.cppstd=.*/compiler.cppstd=20/' ~/.conan2/profiles/default || true

Then re-run:

# macOS
conan install . --build=missing -s build_type=Release
cmake --preset conan-release
cmake --build --preset conan-release -j

# Linux
conan install . \
  --output-folder=build/conan-ninja-release \
  -s build_type=Release \
  --build=missing
cmake --preset conan-ninja-release
cmake --build build/conan-ninja-release -j

PDF Support Issues

If PDF extraction fails or PDFium download fails:

# Disable PDF support temporarily
cmake -B build -DYAMS_ENABLE_PDF=OFF

# Or explicitly specify a different PDFium version if needed
# (check https://github.com/bblanchon/pdfium-binaries/releases for available versions)

If you see network errors during PDFium download:

Check internet connectivity
Corporate firewalls may block GitHub releases
Consider using a VPN or different network
PDFium binaries are ~20MB per platform

LLM Integration Guide

YAMS is designed to work seamlessly with Large Language Models through simple, pipeline-friendly commands:

# Store conversation context with descriptive name
echo "User asked about X, I explained Y" | yams add - --name "context-$(date +%Y%m%d).txt"

# Store code snippets with tags
echo "def function(): return 42" | yams add - --name "helper.py" --tags "python,utils"

# Delete temporary files by pattern
yams delete --pattern "temp_*.txt" --force

# Delete multiple specific files
yams delete --names "draft1.md,draft2.md,notes.txt"

# Retrieve documents by name (coming soon)
# yams get --name "meeting-notes.txt"

# Search with fuzzy matching
yams search "databse" --fuzzy --similarity 0.8

# List with rich metadata
yams list --format table --limit 20

# Chain commands for batch operations
yams list --format minimal | tail -5 | while read hash; do
  yams get $hash
done

# Preview deletions before executing
yams delete --pattern "*.log" --dry-run

Best Practices for LLMs

Use stdin for content storage: Avoids file creation
```
echo "content to store" | yams add -
```
Use minimal format for piping: Clean output for processing
```
yams list --format minimal | head -5
```
Explicit data directory: Always specify storage location
```
yams --data-dir /tmp/project-memory add -
```

JSON for structured data: Parse responses easily

yams stats --format json | jq '.totalObjects'

Direct stdout retrieval: No intermediate files
```
yams get <hash> | process_somehow
```

Usage

MCP Server

Start the MCP server over WebSocket:

yams serve --transport websocket --host 127.0.0.1 --port 8080 --path /mcp

Use TLS (wss):

yams serve --transport websocket --host your.domain --port 443 --path /mcp --ssl

StdIO transport (recommended for local integration and Claude Desktop):

yams serve --transport stdio

Claude Desktop (MCP) Integration

Use stdio (recommended). Add this to your Claude Desktop config (e.g., ~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "yams": {
      "command": "/usr/local/bin/yams",
      "args": ["serve", "--transport", "stdio"],
      "env": {
        "YAMS_STORAGE": "$HOME/.local/share/yams"
      }
    }
  }
}

Alternative (WebSocket) for clients that support ws:

{
  "mcpServers": {
    "yams-ws": {
      "command": "/usr/local/bin/yams",
      "args": ["serve", "--transport", "websocket", "--host", "127.0.0.1", "--port", "8080", "--path", "/mcp"],
      "env": {
        "YAMS_STORAGE": "$HOME/.local/share/yams"
      }
    }
  }
}

CLI Usage

Initialize Storage

# One-time setup with defaults
yams init --non-interactive --no-keygen

# Custom storage location
yams --data-dir /path/to/storage init --non-interactive

Store Documents

# Add a file
yams add file.txt

# Add from stdin
echo "content" | yams add -
cat file.txt | yams add -

# Add multiple files
find . -name "*.txt" -exec yams add {} \;

List Documents

# Table format (default)
yams list

# JSON output for programmatic use
yams list --format json

# Just hashes for piping
yams list --format minimal

# Sort and filter
yams list --sort size --reverse --limit 10
yams list --sort date

Retrieve Documents

# Output to stdout
yams get <hash>

# Save to file
yams get <hash> -o output.txt

# Pipe to other commands
yams get <hash> | grep pattern
yams get <hash> | wc -l

# Get first document from list
yams list --format minimal --limit 1 | xargs yams get

Browse Documents (TUI)

# Launch ranger-style browser
yams browse

# Navigation:
#   j/k or ↑/↓     - Move up/down
#   h/l or ←/→     - Switch columns
#   g/G           - Jump to top/bottom
#   d then D      - Delete document
#   r             - Refresh
#   ?             - Help
#   q or Esc      - Quit

Search

yams search "query" --limit 10 yams search "error" --type "log"

Retrieve

yams get yams get --output file.txt

List & Stats

yams list --recent 20 yams stats --json # JSON output for scripts

Browse (interactive TUI)

yams browse # Interactive document browser


### API
```cpp
#include <yams/api/content_store.h>

auto store = yams::api::createContentStore(getenv("YAMS_STORAGE"));

// Store
yams::api::ContentMetadata meta{.tags = {"code", "v1.0"}};
auto result = store->store("file.txt", meta);

// Search
auto results = store->search("query", 10);

// Retrieve
store->retrieve(hash, "output.txt");

Python

import subprocess, json

def yams_store(content, tags=[], type="text"):
    cmd = ["yams", "store", content]
    if tags: cmd.extend(["--tags", ",".join(tags)])
    if type: cmd.extend(["--type", type])
    return subprocess.run(cmd, capture_output=True, text=True)

def yams_search(query, limit=10):
    cmd = ["yams", "search", query, "--limit", str(limit)]
    return subprocess.run(cmd, capture_output=True, text=True)

LLM Usage Guide

Quick Start for LLMs

# Always specify data directory explicitly
export YAMS_STORAGE="/tmp/yams-data"

# Initialize once (quiet mode)
yams --data-dir "$YAMS_STORAGE" init --non-interactive --force

# Store content from stdin (most common for LLMs)
echo "Important information to remember" | yams --data-dir "$YAMS_STORAGE" add --tags "memory"

# Search for content
yams --data-dir "$YAMS_STORAGE" search "important" --json

# Retrieve specific document
yams --data-dir "$YAMS_STORAGE" get <hash> --json

# Get storage statistics
yams --data-dir "$YAMS_STORAGE" stats --json

Common Patterns

# Store code changes
git diff | yams --data-dir "$YAMS_STORAGE" add --tags "git-diff,$(date +%Y%m%d)"

# Store conversation context
echo "User asked about: $TOPIC" | yams --data-dir "$YAMS_STORAGE" add --tags "context,$TOPIC"

# Store external documentation
curl -s "$API_DOCS_URL" | yams --data-dir "$YAMS_STORAGE" add --tags "api-docs,external"

# Search and retrieve in one line
hash=$(yams --data-dir "$YAMS_STORAGE" search "$QUERY" --json | jq -r '.results[0].hash')
yams --data-dir "$YAMS_STORAGE" get "$hash"

LLM Integration

When to Use YAMS

YAMS is ideal for LLMs to maintain persistent memory across sessions:

Code Development: Track changes, store working versions, remember context
Research: Cache web content, API responses, documentation
Conversation Context: Store important discussions, decisions, requirements
Knowledge Base: Build searchable repository of project knowledge

CLI Usage Examples

Development Workflow

# Store current code state before making changes
git diff | yams store - --tags "pre-refactor,auth-module,$(date +%Y%m%d)"

# Track implementation decisions
yams store "Decided to use JWT tokens with 24h expiry for auth" \
  --tags "decision,auth,architecture"

# Store error context for debugging
yams store "$(tail -100 app.log)" --tags "error,production,$(date +%Y%m%d-%H%M)"

# Save working implementation
yams store-file auth_handler.py --tags "working,auth,v2.1"

Research & Documentation

# Store web research
curl -s https://api.example.com/docs | yams store - \
  --tags "api-docs,external,example-api" \
  --source "https://api.example.com/docs"

# Cache fetched content
yams store "$WEB_CONTENT" --tags "research,oauth,implementation-guide"

# Store meeting notes
yams store-file meeting-notes-2024-01-15.md --tags "meeting,requirements,client"

Search Patterns

# Find related code changes
yams search "authentication" --type "code" --limit 10

# Retrieve specific version
yams search "working auth" --tags "v2.1"

# Get recent errors
yams list --recent 20 --tags "error"

# Semantic search for concepts
yams search "token expiry handling"

MCP (Model Context Protocol) Integration

MCP provides direct integration with Claude Desktop and other MCP-compatible clients.

Performance Tuning

Optimize for large files:

# Adjust chunk size for better deduplication
export YAMS_CHUNK_SIZE=64KB  # Default: 16KB

# Increase cache size
export YAMS_CACHE_SIZE=1GB   # Default: 256MB

Reduce memory usage:

# Use streaming mode for large files
yams store-file --stream large-file.bin

# Enable compression
export YAMS_COMPRESSION=zstd  # Options: none, zstd, lzma

License

Apache-2.0

12 KiB Raw Permalink Blame History