# Changelog All notable changes to YAMS (Yet Another Memory System) will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [0.1.2] - 2025-08-13 ### Added - **File Type and Time Filtering**: Enhanced list, get, and stats commands with comprehensive filtering - File type filters: `--type`, `--mime`, `--extension`, `--binary`, `--text` - Time filters: `--created-after/before`, `--modified-after/before`, `--indexed-after/before` - Supports flexible time formats (ISO 8601, relative like "7d", natural like "yesterday") - **Stats Command File Type Breakdown**: New `--file-types` flag shows detailed file type analysis - File type distribution with counts and sizes - Top file extensions per type - Top MIME types in the system - **Intelligent File Type Detection**: Uses magic_numbers.json as single source of truth - Runtime detection of data files across multiple installation paths - Graceful fallback to built-in patterns when JSON not found ### Changed - **File Type Classification**: More accurate detection using magic numbers vs just extensions - **Data File Installation**: magic_numbers.json now properly installed to system paths ## [0.1.1] - 2025-08-13 ### Fixed - Corrected Conan preset names (use `conan-release` instead of `build-conan-release`) - Fixed C++20 standard requirement in Conan profiles - Removed overly strict compiler warnings that broke dependency builds - Fixed gcovr regex patterns for coverage reports - Removed `-Wold-style-cast`, `-Wconversion`, `-Wsign-conversion`, and `-Woverloaded-virtual` - Keeps essential warnings while allowing third-party code to compile cleanly ### Changed - **Build Instructions**: Standardized on `conan-release` preset across all platforms - Simplified build commands in README and documentation - Consistent build directory structure (`build/conan-release`) ## [0.1.0] - 2025-08-12 ### Added - **List Command Enhancement**: Added `--recent N` flag to show N most recent documents - Filters to the N most recent documents before applying other sorting - Works with all existing sort options (name, size, date, hash) - Useful for quickly viewing recently added content - **PDF Text Extraction**: Full PDF text extraction support using PDFium library - Extract text from all PDF pages with proper UTF-16 to UTF-8 conversion - Extract PDF metadata (title, author, subject, keywords, creation date) - Support for searching within PDFs with context lines - Page-by-page extraction with configurable options - Automatic section detection for academic papers - **Metadata Update Command**: New `update` command for modifying document metadata - Update metadata by document hash or name - Support for multiple key-value pairs in single command - Enables task tracking and status management workflows - Works with both file and stdin documents - **Task Tracking System**: Comprehensive benchmark and test enhancement tracking - PBI documentation structure for systematic improvement - Task status tracking via metadata updates - Progress monitoring capabilities - **KG Hybrid Scoring (default)**: Hybrid search integrates Keyword + Vector + Knowledge Graph - Enabled by default in both CLI and MCP via a shared SearchEngineBuilder - Fails open to metadata/keyword-only when KG is unavailable ### Fixed - **Reference Counter Transaction IDs**: Fixed UNIQUE constraint violation in ref_transactions - Removed manual transaction ID management - Now uses SQLite AUTOINCREMENT for guaranteed unique IDs - Prevents "UNIQUE constraint failed" errors when multiple ReferenceCounter instances exist - **Duplicate Document Handling**: Fixed metadata constraint errors when re-adding existing documents - Detects high deduplication ratio (≥99%) and updates existing metadata instead of inserting - Properly handles both new and existing documents - Updates indexed timestamp for existing documents - **Name Resolution for Commands**: Fixed document lookup by name for stdin documents - get, cat, and update commands now properly find stdin documents by name - Uses search functionality as fallback when path-based search fails - Supports both file-based and stdin documents uniformly - Automatic platform detection (mac-arm64, mac-x64, linux-x64, linux-arm64, win-x64) - FetchContent integration for non-Conan builds - Conditional compilation with YAMS_HAS_PDF_SUPPORT flag - Installs libpdfium.dylib into the install lib directory and sets its install_name to @rpath/libpdfium.dylib - Patches installed executables to reference @rpath/libpdfium.dylib and adds rpath @loader_path/../lib - Eliminates dyld errors like "Library not loaded: ./libpdfium.dylib" - Created header file for UpdateCommand class - Fixed compilation errors in update_command.cpp - Resolved conflicts between Conan and FetchContent for Google Benchmark - Fixed imgui.h include path for browse_command.cpp - Fixed compilation issues with performance benchmarks ### Changed - **Extraction Module**: Enhanced to support multiple file formats - Modular extractor registration system - Extensible architecture for future format support - **Logging Verbosity**: Storage operations now use debug level instead of info - "Stored file" messages only appear in verbose/debug mode - Cleaner output for normal operations ### Known Issues - **Performance Benchmarks**: Storage benchmarks cause segmentation fault after Rabin chunking tests - SHA256 benchmarks work correctly (2.59 GB/s for large files) - Rabin chunking benchmarks work correctly (178 MB/s) - Storage fixture initialization causes crash - to be fixed in future update ## [0.0.6] - 2025-08-11 ### Added - TUI Browser Improvements ### Fixed - Fixed unique_ptr copy errors in query parser benchmarks - Resolved compression error handler warnings on Ubuntu 22.04 - Improved error handling and warning cleanup for better compiler compatibility - Removed hardcoded widget dimensions that broke small terminal displays - Fixed non-functional scrolling in document lists - Improved loading state management and race condition handling - Enhanced document preview formatting and metadata display ## [0.0.5] - 2025-08-11 ### Added - **Build System Enhancements**: - Automatic std::format detection with fallback to fmt library for older compilers - Improved ncurses detection for Linux builds with clear error messages - BUILD-GCC.md documentation for building with GNU toolchain - **Search Improvements**: - Word-by-word processing for multi-word queries (e.g., "chain event ingestion" searches each word independently) - Minimum-should-match scoring (50% of query words must match by default) - Intelligent result aggregation with boost for documents matching all query words - Edit distance-based scoring (closer matches score higher) - Automatic fallback to fuzzy search when FTS5 fails - **Hash search capability**: Search documents by SHA256 hash (full or partial, minimum 8 characters) - **Verbosity control**: Concise output by default, detailed output with --verbose flag - Auto-detection of hash format in queries (8-64 hexadecimal characters automatically trigger hash search) - **MCP Server Enhancements**: - Updated to use metadata repository path with all v0.0.5 search improvements - Added fuzzy search support to MCP server tools with configurable similarity threshold - Enhanced search_documents tool with fuzzy and similarity parameters ### Fixed - Proper handling of hyphens in queries like "PBI-6", "task-4", "feature-123" - Special characters (@, #, ?, :, /, \, [], {}, ()) no longer cause SQL errors - Advanced FTS5 operators (AND, OR, NOT, NEAR) still available for power users - **Fuzzy search matching**: Completely rewrote trigram similarity calculation - **CI/CD Pipeline**: - Added install step verification to ensure artifacts are created - Added libncurses-dev to Ubuntu dependencies - Enabled Ubuntu traditional (non-Conan) builds in CI matrix ### Changed - **Search indexing strategy**: Enhanced to support both single and multi-word queries effectively - Documents now indexed at word level in addition to full content - Keywords extracted more comprehensively with 2-word phrases - Improved tokenization with punctuation handling - **Error handling**: FTS5 failures now gracefully fall back to fuzzy search instead of returning errors ## [0.0.4] - 2025-08-11 ### Added - **Directory-based workflows** with collections and snapshots for multi-folder ingestion - `add --collection `: Add documents to a named collection for organization - `add --snapshot-id --snapshot-label