yams/docs/changelogs/v0.6.md

Changelog Archive: v0.6.x Series

[v0.6.11] - 2025-09-06

Fixed

• MCP server: add_directory and get_by_name could appear to hang when the daemon was not ready or when directories were passed without recursive. The MCP handlers now:
  • Check daemon readiness up front and fall back to local services (IndexingService/DocumentService) when unavailable, matching CLI behavior.
  • Fast‑fail directory adds without --recursive with a clear error (parity with CLI).
  • Resolve name→hash via DocumentService in get_by_name before attempting streamed transfers; this avoids unnecessary GetInit/GetChunk paths.
  • Reduce overly long default timeouts for directory indexing to avoid perceived hangs (header 30s, body 300s).
• Daemon status -d could hang while models were loading. Status path no longer calls blocking provider APIs (e.g., getLoadedModels()), and the defensive objects-dir scan is capped by files/time to ensure immediate responses.
• Embeddings via daemon IPC were unreliable because server handlers were missing. Added request handling on the daemon side for generate embedding(s), load/unload model, and model status; client RPCs now complete as expected.
• ONNX plugin discovery: plugin loader now detects installed plugins using a compile‑time YAMS_INSTALL_PREFIX macro; resolves cases where the ONNX provider failed to load after install.
• Extraction backlog: RepairCoordinator now performs a one‑time backlog enqueue on startup (when idle) so existing documents without embeddings get scheduled; prevents "extraction_pending" from staying high indefinitely.
• Build (Darwin/Clang): resolved "no type named 'stop_source' in namespace 'std'" in SocketServer by using the existing yams::compat::stop_token and removing the unnecessary <stop_token> include.
• Daemon components: fixed mismatched forward declaration of StateComponent (class vs struct) to silence -Wmismatched-tags.
• IPC RequestHandler logging: avoided evaluated operand in typeid usage to silence -Wpotentially-evaluated-expression.
• Extraction: corrected a dangling std::string_view when parsing range tokens in format_handler.cpp.

Changed

• MCP add_directory: recursive now defaults to true in the MCP tool schema for parity with CLI directory indexing. Timeouts tuned (30s/300s) and clearer error messages on misuse.
• Keep-hot semantics for embeddings: when [embeddings].keep_model_hot = true (i.e., lazyLoading=false), the ONNX model pool now pre‑creates a session for hot models (preCreateResources=true) so embeddings are immediately usable after preload.
• Status detail now reports provider presence without enumerating models to avoid lock contention; onnx_models_loaded is "unknown" in this non-blocking mode.
• Tests: enable model provider and plugin auto‑loading in daemon unit tests (with lazy loading, no forced preload) to exercise real provider paths instead of bypassing them.
• Build (SourceHut): .build.yml adds a Boost fallback install when BoostConfig.cmake is missing, ensuring find_package(Boost CONFIG) succeeds with Conan CMakeDeps output.
• Build/CMake: define YAMS_INSTALL_PREFIX for yams_daemon so runtime plugin discovery includes $prefix/lib/yams/plugins.
• Embedding service: unified on compat jthread/stop_token; simplifies cross‑platform behavior and cooperative stop.
• PDF extractor: locally suppressed deprecated std::wstring_convert/std::codecvt warnings under Clang to reduce build noise while a modern replacement is planned.
• Storage plugin loader: marked unused parameters to quiet -Wunused-parameter without changing behavior.

Notes

• Large extraction_pending values reflect missing embeddings, not just text extraction. With the backlog enqueue and ONNX provider discovery fixes, pending counts should drop as embeddings are generated. Ensure the ONNX plugin builds (onnxruntime present) or set YAMS_PLUGIN_DIR to the plugin output directory during development/CI.

[v0.6.10] - 2025-09-06

CI bump

• macOS build failures: replaced direct std::jthread/std::stop_token usages with our portability shim (yams::compat) across daemon, services, and embedding service; resolves libc++ gaps on hosted macOS.
• GitHub Actions release workflow: corrected heredoc in summary generation to avoid "unexpected EOF"; now writes Python output to a temp file and reads it safely.

[v0.6.9] - 2025-09-05

Notes

• Upgrading daemon is recommended: older daemons (v1) will log a one‑time warning "Daemon protocol v1 < client v2" and some fields will be supplemented client‑side.

Known Issues

• Daemon startup may remain in "Initializing" when embeddings are configured to keep the model hot (preload) and the ONNX stack cannot complete early model resolution on some systems. Workarounds:
  • Set [embeddings].keep_model_hot = false and [embeddings].preload_on_startup = false in ~/.config/yams/config.toml to use lazy loading (model loads on first use).
  • Or export YAMS_DISABLE_MODEL_PRELOAD=1 before starting the daemon.
  • Status derives from the lifecycle FSM; optional subsystems (models) should not gate readiness in recent builds. If you still see "Initializing", ensure you are running the updated daemon and ONNX plugin.

Changed

• Status handler hardened to a minimal, safe snapshot; FSM/MUX metrics gated by lifecycle (Ready/Degraded) to avoid init races.
• Linux CPU proxy: read Threads: from /proc/self/status (robust) instead of parsing /proc/self/stat field 20.
• Stats defaults: yams stats now begins with the System Health section before compact counters.
• yams stats: prefers daemon JSON values to avoid zeros; local fallback includes vector DB size.
• yams status: supplements from stats JSON when talking to older daemons (v1) to avoid zeros; shows services summary and waiting components when not ready.
• Service detectors hardened:
  • SearchExecutor now includes a reason when unavailable: database_not_ready | metadata_repo_not_ready | not_initialized.
  • ONNX models status reports loaded count with clear guidance to download a preferred model.
• CLI search defaults to hybrid and now auto-retries with fuzzy when strict/hybrid returns zero results for better “true hybrid” behavior.
• Service metadata search path now falls back to fuzzy when full-text returns no results.
• Model CLI help updated with subcommands and --url usage; clearer guidance to avoid confusion with the top-level download command.
• Model CLI guidance: after downloads, success output includes yams config embeddings model <name> and related steps; yams model list and yams model info now mirror a one-line configuration hint.

Added

• IPC Protocol v2:
  • StatusResponse now carries runtime fields: running, ready, uptime_seconds, requests_processed, active_connections, memory_mb, cpu_pct, version.
  • GetStatsResponse carries numeric fields alongside JSON: total_documents, total_size, indexed_documents, vector_index_size, compression_ratio.
• Daemon stats JSON: explicit not_ready flag; includes durations_ms, top_slowest, and latency percentiles (latency_p50_ms, latency_p95_ms).
• Bootstrap status file: ~/.local/state/yams/yams-daemon.status.json with readiness, progress, durations, and top_slowest for pre‑IPC visibility.
• CLI doctor/status: shows top 3 slowest components with elapsed ms when available (from bootstrap JSON).
• CLI UI (retro): compact text for status and stats; yams stats -v shows System Health and detailed sections; stats vectors prints compact block.
• CLI daemon status (-d):
  • One-line services summary: SVC : ✓ Content | ✓ Repo | ✓ Search | ⚠ (0) Models.
  • WAIT line during initialization listing not-ready services with progress (e.g., WAIT : search_engine (70%), model_provider (0%)).
• ServiceManager:
  • Preferred model preload on startup: uses configured embeddings.preloadModels (first entry) when no models are loaded; falls back to all-MiniLM-L6-v2.
  • Sanity warning when SearchExecutor is not initialized despite database and metadata repo ready.
• FSM metrics emission (server): counts payload writes/bytes sent, header reads/bytes received; exposed in Status when Ready/Degraded.
• LatencyRegistry (server): lightweight histogram; p50/p95 emitted via stats JSON.
• Protocol compatibility guard: one‑time warning when daemon/client protocol versions differ.
• SocketServer::setDispatcher(RequestDispatcher*) for safe future hot‑rebinds (not used by default).
• Integration test: daemon_status_stats_integration_test covers Status/Stats proto presence and basic FSM exposure.
• MCP list: added paths_only parameter to the MCP list tool; forwarded to daemon ListRequest.pathsOnly to engage the hot path (no snippet/metadata hydration).
• MCP grep: introduced fast_first option that returns a quick semantic suggestions burst when requested.
• MCP server: startup flags to set hot/cold modes consistently with CLI:
  • --list-mode → sets YAMS_LIST_MODE
  • --grep-mode → sets YAMS_GREP_MODE
  • --retrieval-mode → sets YAMS_RETRIEVAL_MODE
• CLI model management:
  • --url override to download any model by name from a custom URL.
  • Subcommands: yams model list|download|info|check (aliases to flags).
  • yams model check shows ONNX runtime support and plugin directory status, plus autodiscovers installed models under ~/.yams/models.
• Model autodiscovery: yams model list now lists locally installed models found at ~/.yams/models/<name>/model.onnx.
• Daemon model selection: honors embeddings.preferred_model from ~/.config/yams/config.toml (or XDG config) to preload and prefer that model at runtime.
• Model path resolution: honors embeddings.model_path as the models root (with ~ expansion) and resolves name-based models in priority order: configured root → ~/.yams/models → models/ → /usr/local/share/yams/models; full paths are used as-is.
• Docs: PROMPT updated to show multi-path yams add src/ include/ ... examples.

Fixed

• Build: resolved C++ signature mismatch for EmbeddingService::runRepair by aligning the implementation with header declarations and providing a proper non-jthread legacy runner.
• MCP schemas updated for list (paths_only) and grep (fast_first) to reflect new behaviors.
• ONNX plugin: CMake fixes to ensure dynamic plugin loads cleanly:
  • Disable IPO/LTO on yams_onnx_plugin (INTERPROCEDURAL_OPTIMIZATION FALSE) to prevent LTO symbol internalization and tooling noise.
  • Ensure exported C symbols have default visibility (C_VISIBILITY_PRESET/CXX_VISIBILITY_PRESET default, VISIBILITY_INLINES_HIDDEN OFF) so getProviderName/createOnnxProvider remain visible.
  • On ELF, link with -Wl,-z,defs to catch unresolved symbols at link time.
  • Resolves daemon warning: "Plugin missing getProviderName function: .../libyams_onnx_plugin.so".
• Retrieval now accepts sha256:<hex> hashes (normalized before validation) in DocumentService retrieve and cat.
• Model downloads provide clearer errors when offline (curl exit code mapping for host resolution/connect failures).

[v0.6.8] - 2025-09-04

Changed

• Data directory resolution uses a consistent precedence across CLI and daemon: configuration file > environment (YAMS_STORAGE/YAMS_DATA_DIR) > CLI flag. This avoids CLI defaults masking configured storage roots.
• MCP search defaults hardened: runs in hybrid mode with fuzzy matching enabled by default and a similarity threshold of 0.7 when the client doesn’t provide options.

Added

• MCP get-by-name and cat now include a fuzzy fallback: if direct lookup fails, a hybrid fuzzy search runs and returns the strongest single match, then retrieves by hash (preferred) or normalized path.
• URL-aware naming: post-index for MCP downloads now sets the document name from the URL basename (query stripped), improving name-based retrieval. MCP get-by-name also accepts full URLs and normalizes them to the basename automatically.

Fixed

• macOS build compatibility and warnings:
  • std::jthread/std::stop_token portability: embedding service falls back to std::thread with an atomic stop on platforms lacking libc++ support.
  • Third‑party noise reduced: mark hnswlib includes as SYSTEM and silence sqlite-vec.c; fence cast-qual warnings under Clang for HNSW.
  • Resolved shadowing and unused variable/parameter warnings across content handlers (PDF, audio, image, binary, archive) and daemon client helpers.
• Name propagation for downloads: downloaded artifacts are indexed with a human-friendly name, making get --name <file> work reliably after MCP/CLI downloads.

[v0.6.7] - 2025-09-04

CI bump

• SourceHut build fixes
• GitHub CI updates

[v0.6.6] - 2025-09-04

CI bump

• SourceHut build fixes
• Github CI updates

Fixed

• Search regression from adding query results. Querys will now be supported by default in the fallback path after expressions are extracted

Known Issues

• Data directory path resolutions needs to be audited

[v0.6.5] - 2025-09-04

Changed

• Release workflow: migrate to Conan 2 profiles using a dedicated host profile (conan/profiles/host.jinja) that enforces compiler.cppstd=20 and sets compiler.libcxx=libc++ on macOS.
• CMake (Darwin): replace GNU ld flags (--start-group/--end-group, --whole-archive/--no-whole-archive) with -Wl,-force_load for specific archives in CLI, MCP server, and daemon.
• CI (act): install build tools when missing in containers (cmake, ninja, build-essential/pkg-config) to allow local act runs of the release job.

Added

• Search ergonomics:
  • -q, --query flagged alias to safely pass queries that start with '-' (e.g., yams search -q "--start-group --whole-archive").
  • --stdin and --query-file <path> to read the query from stdin or a file (--query-file - also reads stdin).
  • Helpful "Query is required" guidance when no query is provided, with tips to use -q/--query, -- to stop option parsing, or stdin/file inputs.

Fixes

• macOS linking failures due to GNU-only linker flags appearing in executable link lines; now prevented and defensively stripped if injected elsewhere.
• SourceHut .build.yml: use the Conan host profile and disable ONNX (-o enable_onnx=False) to avoid upstream onnxruntime 1.18 build errors with GCC 15; preserves successful release builds on Arch.

[v0.6.4] - 2025-01-05

Known Issues

• Daemon not showing correct storage details

Added

• Streaming metrics: Track and expose stream_total_streams, stream_batches_emitted, stream_keepalives, and average stream_ttfb_avg_ms via yams stats (JSON and technical details in text mode).
• Grep performance: line-by-line streaming scanner (no full-file buffering) and literal fast-path for --literal without word-bounds/case-folding.
• Add performance: parallel directory traversal/processing with bounded workers.
• Session helpers: yams session pin|list|unpin|warm for hot data management (feature is experimental and may not work reliably).

Changed

• List paths-only mode now avoids snippet/metadata hydration end-to-end for faster responses; honors pathsOnly through daemon to services.
• Retrieval (cat/get) prefers extracted text (hot) when available before falling back to CAS (cold), respecting per-document force_cold.
• Grep hot/cold race: daemon emits hot first-burst and cancels cold; batch/first-burst tuning supported via env for testing.
• Delete command now mirrors rm ergonomics:
  • rm alias retained; -f is an alias for --force; -r is an alias for --recursive
  • Positional targets supported (names/paths/patterns); when a single target is provided with -r, it is treated as a directory
  • Multiple positional targets are treated as a names list
  • Selector requirement relaxed when positional targets are used; mode (directory/pattern/name) is inferred heuristically
• List paths-only mode now avoids snippet/metadata hydration end-to-end for faster responses; honors pathsOnly through daemon to services.
• Retrieval (cat/get) prefers extracted text (hot) when available before falling back to CAS (cold), respecting per-document force_cold.
• Grep hot/cold race: daemon emits hot first-burst and cancels cold; batch/first-burst tuning supported via env for testing.
• CMake: Platform-aware linker selection.
  • Removed hard-coded -fuse-ld=lld from presets.
  • Toolchain now attempts -fuse-ld=lld only on non-Apple platforms when supported, gated by YAMS_PREFER_LLD (default ON).
  • macOS Release builds use -Wl,-dead_strip instead of GNU --gc-sections.
• Release workflow (macOS): Align Conan arch with target:
  • macos-arm64 uses -s arch=armv8
  • macos-x86_64 uses -s arch=x86_64

Fixed

• Streaming search finalization: avoid infinite keepalive loop on empty results; keepalive cadence configurable via YAMS_KEEPALIVE_MS.
• Stats command: yams stats help now prints a concise system metrics guide.
• MCP server timeouts with some clients:
  • Async handlers now run detached (no premature task destruction), preventing “Context server request timeout”
  • Server sends notifications/ready after client notifications/initialized for better client compatibility
  • Stdio framing hardened to consume trailing CR/LF after payloads to avoid parse glitches across clients
• Streaming search finalization: avoid infinite keepalive loop on empty results; keepalive cadence configurable via YAMS_KEEPALIVE_MS.
• Stats command: yams stats help now prints a concise system metrics guide.
• Docker: ARM64 build now mirrors AMD64 by using Buildx with proper tag suffixes (-arm64) and pushes versioned and latest tags. Multi-arch manifest creation no longer fails with "not found ... -arm64".
• Release workflow (macOS/Linux): Conan 2 profile updates use correct keys (compiler.cppstd, compiler.libcxx) via conan profile update; removed brittle sed/grep edits that broke on macOS.
• Release workflow (Windows): Corrected Conan 2 profile update syntax (compiler.cppstd=20).

[v0.6.3] - 2025-01-04

Changes

• CI version bump for source hut and github action builds
• yams add supports multiple files
• Adding small optimizations to decompression

Known Issues

• MCP server usage my vary in successful returns
• [v0.6.2] CLI performance still degraded, will be addressed in subsequent release

[v0.6.2] - 2025-01-04

Hot fixes

• CI version bump
• MCP server patch (tested working in zed and lm studio but not working in goose or jan)

[v0.6.1] - 2025-01-03

Hot fixes

• Docker and sourcehut build file updates
• Daemon liveness checks on startup bug fix where daemon did not signal start
• Added more aggressive parallelization for search now that it holds all resources (will make tunable)

Known Issues

• MCP server init not working with some clients
• CLI performance has been degraded as it creates socket to daemon each request, working on session mode
• yams stats -v not working as expected

[v0.6.0] - 2025-01-03

Repository

• Will move all future development work to experimental branch so that main and releases become more stable
  • I apologize for recent instabilities

Added

• Server multiplexing: fair round-robin writer with per-turn byte budget and backpressure caps (default ON).

• Status: exposes multiplexing metrics (active handlers, queued bytes, writer budget).

• Cancel control frame: request type added and server-side cancel scaffolding (per-request contexts).

• Development Changes

  • Updated tasks for vscode and zed tasks
  • Enforcing clang-tidy warnings as errors
  • Attempting to stabilize build system as plugin system is implemeneted per roadmap

• Daemon Logging Rotation

  • Use rotating file sink to preserve logs across crashes
  • Log rotation info: Log rotation enabled: /path/to/logfile.log (max 10MB x 5 files)

• Connection State Machine

  • Deterministic ConnectionFsm for IPC connection lifecycle management
  • Clean state transitions: Disconnected → Connecting → Connected → ReadingHeader → ReadingPayload → WritingHeader → StreamingChunks → Error/Closed
  • FSM metrics collection gated by daemon log level (debug/trace only)
  • Coordination with RepairFsm for safe data integrity operations
  • Guardrails for readable/writable operations based on FSM state
  • Header-first streaming with chunked transfer support

• Protobuf IPC Migration

  • Complete migration from custom binary serialization to Protocol Buffers
  • ProtoSerializer as the single payload codec for client/server
  • Comprehensive ipc_envelope.proto schema with oneof for all request/response types
  • Transport framing (header, CRC, streaming) remains unchanged
  • MAX_MESSAGE_SIZE enforcement in encode/decode paths
  • Older non-protobuf clients are incompatible after this release

Changed

• CLI Commands

  • All CLI commands migrated to async-first pattern with run_sync bridge
  • Removed deprecated daemon_first() and daemon_first_pooled() helpers
  • Commands now use async_daemon_first() with proper timeout handling
  • Consistent error handling and retry logic across all commands

• MCP Server

  • All handlers converted to async (yams::Task<Result<T>>)
  • Improved concurrency and reduced blocking operations

• Daemon Architecture

  • Socket server runs in-process, eliminating need for external IPC server
  • state_.readiness.ipcServerReady now properly reflects socket server status
  • Graceful shutdown sequence: socket server stops before services
  • Better resource lifecycle management and error propagation

Fixed

• Async Infrastructure
  • Removed spin-wait bridges that caused CPU waste
  • Fixed head-of-line blocking in request processing
  • Proper cancellation handling for in-flight operations
  • Eliminated Task::get() usage from production code paths

Removed

• Deprecated Synchronous APIs
  • AsioClientPool::roundtrip(), status(), ping(), call() (sync versions)
  • PooledRequestManager::execute() (now returns NotImplemented error)
  • YAMS_ASYNC_CLIENT environment variable (async is now mandatory)
  • Legacy MessageSerializer and custom binary serialization code
  • All synchronous daemon helper functions
  • Forward declarations of BinarySerializer/BinaryDeserializer remain in ipc_protocol.h for cleanup in v0.7.0