# Changelog All notable changes to YAMS (Yet Another Memory System) will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [v0.2.7] - 2025-08-14 ### Fixed - **CI version bump** ## [v0.2.6] - 2025-08-14 ### Fixed - **CI version bump** - Conan flag fix in CI ## [v0.2.5] - 2025-08-14 ### Fixed - **CI version bump** - Updated git pages site - Build system improvements ## [v0.2.4] ### Fixed - **MCP Server Signal Handling** - Server now responds properly to Ctrl+C (SIGINT) signals - Replaced std::signal with sigaction to allow interrupting blocking I/O - Added non-blocking stdin polling with 100ms timeout in StdioTransport - Server checks external shutdown flag in main loop - Handles EINTR errors and clears stdin error state properly - Added clear startup messages and usage instructions to stderr - **Dockerfile** - Dockerfile should build correctly - **Cross-Compilation Support** - Added explicit Conan profiles for x86_64 and ARM64 architectures - CI/CD workflows now properly handle cross-compilation with dual profiles - Fixed zstd assembly symbol errors when cross-compiling on Apple Silicon - Added zstd/*:build_programs=False option to avoid build issues - Workflows detect native vs cross-compilation scenarios automatically ### Added - **MCP Server Documentation** - Comprehensive MCP usage guide at `docs/user_guide/mcp.md` (I personally use the CLI) ### Technical Details - **Signal Handling**: Uses sigaction with sa_flags=0 to interrupt blocking system calls - **Non-blocking I/O**: Implements poll() on Unix/macOS for stdin availability checking - **Cross-Compilation**: Uses Conan's dual profile system (-pr:b for build, -pr:h for host) ## [v0.2.3] - 2025-08-14 ### Fixed - **CI version bump** - README updates (removed install instructions with curl, download binary) - Build system improvements for docker files and release ## [v0.2.2] - 2025-08-14 ### Fixed - **Docker Build C++20 Configuration** - Added explicit C++20 standard setting in Conan profile - Fixed "Current cppstd (gnu17) is lower than required C++ standard (20)" error - Ensures Docker builds use correct C++ standard matching project requirements - **Release Notes Formatting** - Fixed Conan profile output contaminating GitHub release descriptions - Redirected debug output to stderr to keep release notes clean - Release pages now display properly formatted markdown - **macOS Cross-Compilation** - Fixed Boost build failures when cross-compiling x86_64 on ARM64 macOS - Disabled Boost.Locale and Boost.Stacktrace.Backtrace components that fail in cross-compilation - Enables successful builds on Apple Silicon machines with Rosetta 2 ### Changed - **Docker Build Process** - Switched from CMake presets to direct CMake commands with Conan toolchain - Added ninja-build to Docker dependencies for faster builds - More robust build process that doesn't rely on dynamically generated presets ## [v0.2.1] - 2025-08-13 ### Fixed - **Build System Standardization**: All workflows now use Conan exclusively - Fixed CMake export errors for spdlog and other dependencies - Release workflow migrated from plain CMake to Conan-based builds - Ensures consistent dependency management across CI and release - Resolves "target requires target that is not in any export set" errors - **Docker Build Issues** - Fixed Conan 2.0 command syntax (`conan profile show` instead of `conan profile show default`) - Fixed CMake preset issues by using direct CMake commands with Conan toolchain - Added ninja-build to Docker dependencies for faster builds - Improved version handling: uses git tags for releases, `dev-` for development builds - Fixed undefined variable warnings in Dockerfile labels - Added proper build argument handling for YAMS_VERSION and GITHUB_SHA - **Release Page Formatting** - Fixed Conan profile output contaminating release notes - Redirected debug output to stderr to keep release notes clean ### Changed - **MIME Type Detection Default Behavior** - MIME types are now detected automatically during add operations - `repair-mime` command remains available for fixing existing documents - Detection uses FileTypeDetector with both signature and extension methods - **Docker Versioning Strategy** - Tagged releases use semantic version from git tag - Development builds use `dev-` format - Provides clear distinction between releases and development builds ### Technical Details - **Testing Interface**: MCP server now exposes test methods via conditional compilation - `#ifdef YAMS_TESTING` enables test-specific public methods - Replaces hacky `#define private public` approach - Maintains encapsulation while allowing proper unit testing - **Build Workflows**: Unified Conan usage with proper profiles and caching - Conan profile detection with C++20 enforcement - Package caching for faster builds - Consistent preset usage (`conan-release`) across all platforms ## [v0.2.0] - 2025-08-13 ### Added - **LLM Ergonomics Enhancements** - Added `--paths-only` flag to search command for LLM-friendly output (one file path per line) - Enhanced codebase indexing workflows for AI agent integration - Improved documentation in PROMPT-eng.md with LLM-optimized YAMS patterns - Support for using YAMS as "memory layer for low context" in AI workflows - Magic number patterns for `#include`, `import`, `function`, `class`, shebang lines, etc. - Automatic programming language detection during recursive directory operations - **Grep Command**: Full-featured regex search within indexed file contents - Standard grep options: `-A/--after`, `-B/--before`, `-C/--context` for context lines - Pattern matching options: `-i` (case-insensitive), `-w` (whole word), `-v` (invert) - Output options: `-n` (line numbers), `-H` (with filename), `-c` (count), `-l` (files with matches) - Color highlighting support with `--color` option - Supports searching all indexed files or specific paths - **Knowledge Graph Integration in Get Command** - Added `--graph` flag to display related documents - Added `--depth` option (1-5) to control graph traversal depth - Shows documents in same directory and with similar extensions - Displays document relationships and metadata - **Line-Level Search with Context** - Enhanced search command with `-n/--line-numbers` flag - Added context options: `-A/--after`, `-B/--before`, `-C/--context` - Highlights matching lines with color - Shows surrounding context lines for better understanding - **C++17 Compatibility Mode** for broader system support - New `YAMS_CXX17_MODE` CMake option for legacy compiler compatibility - Support for GCC 9+, Clang 9+, older MSVC versions - Automatic fallbacks: `boost::span` for `std::span`, `fmt` for `std::format` - Compatible with Ubuntu 20.04 LTS, CentOS 8, older macOS versions - Comprehensive compatibility documentation and build matrix - Runtime feature detection with graceful degradation - **Enhanced Build System Flexibility** - Dual-mode compilation: full C++20 for modern systems, C++17 for legacy - Automatic dependency management with Conan and FetchContent fallbacks - Compiler-specific optimizations and warning configurations - Performance-oriented feature selection based on available capabilities - **MCP Server Feature Parity Enhancements** (~85% CLI-MCP parity achieved) - **Enhanced search_documents tool** with LLM ergonomics optimizations - Added `paths_only` parameter for LLM-friendly file path output (one per line) - Added line context parameters: `line_numbers`, `after_context`, `before_context`, `context` - Added `color` parameter for syntax highlighting control (always, never, auto) - **New grep_documents tool** with full CLI grep parity - Complete regex pattern matching across indexed document contents - Context options: `-A/--after`, `-B/--before`, `-C/--context` for surrounding lines - Pattern options: `ignore_case`, `word`, `invert` for flexible matching - Output modes: `line_numbers`, `with_filename`, `count`, `files_with_matches`, `files_without_match` - Color highlighting and `max_count` limit support - **Knowledge graph integration** in retrieve_document tool - Added `graph` parameter to include related documents in response - Added `depth` parameter (1-5) for graph traversal control - Added `include_content` parameter for full document content in graph responses - Finds documents in same directory, similar extensions, and metadata relationships - **New update_metadata tool** for document metadata management - Update metadata by document `hash` or `name` with automatic resolution - Support for multiple `key=value` metadata pairs in single operation - Verbose output option for detailed update information - **Enhanced list_documents tool** with comprehensive filtering and sorting - File type filtering: `type`, `mime`, `extension`, `binary`, `text` parameters - Time-based filtering: `created_after/before`, `modified_after/before`, `indexed_after/before` - Recent documents filter: `recent` parameter for N most recent files - Sorting control: `sort_by` (name, size, created, modified, indexed) and `sort_order` (asc, desc) - **Enhanced get_stats tool** with file type breakdown analysis - Added `file_types` parameter for detailed file type distribution - Shows file type counts, sizes, top extensions per type, and top MIME types - Provides comprehensive storage analytics for better content understanding ### Enhanced - **MCP Tool Documentation** comprehensively updated with new schemas and examples - Added complete JSON schemas for all enhanced tools with parameter descriptions - Added practical usage examples for each tool capability - Updated integration documentation for Claude Desktop configuration - **YAMS-first workflow guidance**: default to indexing and searching code via YAMS pre-watch; avoid external grep/find/rg for repository queries. - **CLI docs enhanced** with pre-watch indexing workflow and YAMS-only search/grep examples. ### Technical Details - **File type classification system** with MIME type analysis using `getFileTypeFromMime()` and `isBinaryMimeType()` helper methods - **Knowledge graph traversal** leverages document relationships through metadata repository queries - **LLM ergonomics optimizations** reduce context usage while maintaining full functionality - **Tool schema definitions** follow JSON-RPC standards for seamless MCP integration ### Fixed - **Indexing Tests**: Updated for new API with proper factory function declarations - **Namespace Issues**: Fixed forward declarations in document_indexer.h - **CLI11 Conflicts**: Resolved `-h` flag conflict in grep command by using only `--no-filename` - **Test Suite Stabilization** - Fixed SEGV crash in vector_database_tests (DocumentChunkerTest bounds checking) - Fixed AddressSanitizer container overflow in detection_tests - Added thread safety to FixtureManager test infrastructure with proper mutex protection - Added thread safety to FileTypeDetector for libmagic operations (not thread-safe by default) - Fixed compilation errors in multiple test files (header includes, error codes, GMock dependencies) - Fixed EXPECT_NO_THROW macro usage in detection tests - **Recursive Add Pattern Parsing** - Fixed CLI11 comma-separated pattern handling in `--include` and `--exclude` options - Now properly splits patterns like `"*.cpp,*.h,*.md"` into individual patterns - Added proper whitespace trimming and empty pattern filtering - **CI/CD Pipeline Improvements** - Fixed Conan options quoting for proper test builds (`"yams/*:build_tests=True"`) - Added explicit `YAMS_BUILD_TESTS=ON` to CMake configuration - Added individual test target builds before running ctest - Implemented graceful test failure handling (CI continues, reports detailed results) - Added ASAN relaxation options for CI environment compatibility - Fixed YAML syntax error in release.yml (Python script indentation) ## [v0.1.3] - 2025-08-13 ### Added - **Multi-Channel Package Distribution System** - Universal install script with platform detection (Linux/macOS, x86_64/ARM64) - One-line installation: `curl -fsSL https://raw.githubusercontent.com/trvon/yams/main/install.sh | bash` - Checksum verification and retry logic for secure downloads - Custom installation directory support and shell completion setup - Environment variable customization (YAMS_VERSION, YAMS_INSTALL_DIR, etc.) - **Docker Container Support** - Multi-stage Dockerfile with optimized Ubuntu-based runtime - Multi-architecture builds (AMD64/ARM64) via GitHub Container Registry - Automated publishing on tags and main branch pushes to `ghcr.io/trvon/yams` - Non-root user configuration for security best practices ### Enhanced - **GitHub Release Workflow with Performance Reporting** - Comprehensive benchmark execution with JSON result collection - Code coverage reports (HTML/XML) generated for Linux builds - Performance metrics tables in release notes with timing and throughput data - Coverage and benchmark files automatically uploaded as release assets - Combined release notes with changelog and performance data - **CI/CD Pipeline Improvements** - Migrated all workflows to self-hosted runners (`self-hosted`, `Linux/macOS`, `X64/ARM64`) - Fixed compatibility issues (mapfile command, artifact upload paths) - Enhanced artifact collection (coverage reports, benchmark results, binaries) - Streamlined build process removing problematic Conan profile modifications ### Notes - All packaging improvements implemented but not yet released pending verification - Release workflow enhanced to provide comprehensive performance and coverage data - Multi-platform distribution ready for production deployment ## [0.1.2] - 2025-08-13 ### Added - **File Type and Time Filtering**: Enhanced list, get, and stats commands with comprehensive filtering - File type filters: `--type`, `--mime`, `--extension`, `--binary`, `--text` - Time filters: `--created-after/before`, `--modified-after/before`, `--indexed-after/before` - Supports flexible time formats (ISO 8601, relative like "7d", natural like "yesterday") - **Stats Command File Type Breakdown**: New `--file-types` flag shows detailed file type analysis - File type distribution with counts and sizes - Top file extensions per type - Top MIME types in the system - **Intelligent File Type Detection**: Uses magic_numbers.json as single source of truth - Runtime detection of data files across multiple installation paths - Graceful fallback to built-in patterns when JSON not found ### Changed - **File Type Classification**: More accurate detection using magic numbers vs just extensions - **Data File Installation**: magic_numbers.json now properly installed to system paths ## [0.1.1] - 2025-08-13 ### Fixed - Corrected Conan preset names (use `conan-release` instead of `build-conan-release`) - Fixed C++20 standard requirement in Conan profiles - Removed overly strict compiler warnings that broke dependency builds - Fixed gcovr regex patterns for coverage reports - Removed `-Wold-style-cast`, `-Wconversion`, `-Wsign-conversion`, and `-Woverloaded-virtual` - Keeps essential warnings while allowing third-party code to compile cleanly ### Changed - **Build Instructions**: Standardized on `conan-release` preset across all platforms - Simplified build commands in README and documentation - Consistent build directory structure (`build/conan-release`) ## [0.1.0] - 2025-08-12 ### Added - **List Command Enhancement**: Added `--recent N` flag to show N most recent documents - Filters to the N most recent documents before applying other sorting - Works with all existing sort options (name, size, date, hash) - Useful for quickly viewing recently added content - **PDF Text Extraction**: Full PDF text extraction support using PDFium library - Extract text from all PDF pages with proper UTF-16 to UTF-8 conversion - Extract PDF metadata (title, author, subject, keywords, creation date) - Support for searching within PDFs with context lines - Page-by-page extraction with configurable options - Automatic section detection for academic papers - **Metadata Update Command**: New `update` command for modifying document metadata - Update metadata by document hash or name - Support for multiple key-value pairs in single command - Enables task tracking and status management workflows - Works with both file and stdin documents - **Task Tracking System**: Comprehensive benchmark and test enhancement tracking - PBI documentation structure for systematic improvement - Task status tracking via metadata updates - Progress monitoring capabilities - **KG Hybrid Scoring (default)**: Hybrid search integrates Keyword + Vector + Knowledge Graph - Enabled by default in both CLI and MCP via a shared SearchEngineBuilder - Fails open to metadata/keyword-only when KG is unavailable ### Fixed - **Reference Counter Transaction IDs**: Fixed UNIQUE constraint violation in ref_transactions - Removed manual transaction ID management - Now uses SQLite AUTOINCREMENT for guaranteed unique IDs - Prevents "UNIQUE constraint failed" errors when multiple ReferenceCounter instances exist - **Duplicate Document Handling**: Fixed metadata constraint errors when re-adding existing documents - Detects high deduplication ratio (≥99%) and updates existing metadata instead of inserting - Properly handles both new and existing documents - Updates indexed timestamp for existing documents - **Name Resolution for Commands**: Fixed document lookup by name for stdin documents - get, cat, and update commands now properly find stdin documents by name - Uses search functionality as fallback when path-based search fails - Supports both file-based and stdin documents uniformly - Automatic platform detection (mac-arm64, mac-x64, linux-x64, linux-arm64, win-x64) - FetchContent integration for non-Conan builds - Conditional compilation with YAMS_HAS_PDF_SUPPORT flag - Installs libpdfium.dylib into the install lib directory and sets its install_name to @rpath/libpdfium.dylib - Patches installed executables to reference @rpath/libpdfium.dylib and adds rpath @loader_path/../lib - Eliminates dyld errors like "Library not loaded: ./libpdfium.dylib" - Created header file for UpdateCommand class - Fixed compilation errors in update_command.cpp - Resolved conflicts between Conan and FetchContent for Google Benchmark - Fixed imgui.h include path for browse_command.cpp - Fixed compilation issues with performance benchmarks ### Changed - **Extraction Module**: Enhanced to support multiple file formats - Modular extractor registration system - Extensible architecture for future format support - **Logging Verbosity**: Storage operations now use debug level instead of info - "Stored file" messages only appear in verbose/debug mode - Cleaner output for normal operations ### Known Issues - **Performance Benchmarks**: Storage benchmarks cause segmentation fault after Rabin chunking tests - SHA256 benchmarks work correctly (2.59 GB/s for large files) - Rabin chunking benchmarks work correctly (178 MB/s) - Storage fixture initialization causes crash - to be fixed in future update ## [0.0.6] - 2025-08-11 ### Added - TUI Browser Improvements ### Fixed - Fixed unique_ptr copy errors in query parser benchmarks - Resolved compression error handler warnings on Ubuntu 22.04 - Improved error handling and warning cleanup for better compiler compatibility - Removed hardcoded widget dimensions that broke small terminal displays - Fixed non-functional scrolling in document lists - Improved loading state management and race condition handling - Enhanced document preview formatting and metadata display ## [0.0.5] - 2025-08-11 ### Added - **Build System Enhancements**: - Automatic std::format detection with fallback to fmt library for older compilers - Improved ncurses detection for Linux builds with clear error messages - BUILD-GCC.md documentation for building with GNU toolchain - **Search Improvements**: - Word-by-word processing for multi-word queries (e.g., "chain event ingestion" searches each word independently) - Minimum-should-match scoring (50% of query words must match by default) - Intelligent result aggregation with boost for documents matching all query words - Edit distance-based scoring (closer matches score higher) - Automatic fallback to fuzzy search when FTS5 fails - **Hash search capability**: Search documents by SHA256 hash (full or partial, minimum 8 characters) - **Verbosity control**: Concise output by default, detailed output with --verbose flag - Auto-detection of hash format in queries (8-64 hexadecimal characters automatically trigger hash search) - **MCP Server Enhancements**: - Updated to use metadata repository path with all v0.0.5 search improvements - Added fuzzy search support to MCP server tools with configurable similarity threshold - Enhanced search_documents tool with fuzzy and similarity parameters ### Fixed - Proper handling of hyphens in queries like "PBI-6", "task-4", "feature-123" - Special characters (@, #, ?, :, /, \, [], {}, ()) no longer cause SQL errors - Advanced FTS5 operators (AND, OR, NOT, NEAR) still available for power users - **Fuzzy search matching**: Completely rewrote trigram similarity calculation - **CI/CD Pipeline**: - Added install step verification to ensure artifacts are created - Added libncurses-dev to Ubuntu dependencies - Enabled Ubuntu traditional (non-Conan) builds in CI matrix ### Changed - **Search indexing strategy**: Enhanced to support both single and multi-word queries effectively - Documents now indexed at word level in addition to full content - Keywords extracted more comprehensively with 2-word phrases - Improved tokenization with punctuation handling - **Error handling**: FTS5 failures now gracefully fall back to fuzzy search instead of returning errors ## [0.0.4] - 2025-08-11 ### Added - **Directory-based workflows** with collections and snapshots for multi-folder ingestion - `add --collection `: Add documents to a named collection for organization - `add --snapshot-id --snapshot-label