yams/docs/changelogs/v0.4.md

Changelog Archive: v0.4.x Series

[v0.4.8] - 2025-08-21

Added

• Shell Wildcard Expansion Support
  • add command now accepts multiple file paths from shell expansion
  • Commands like yams add *.md now work correctly without "arguments were not expected" errors
  • Properly handles shell wildcard expansion (e.g., *.cpp, *.md, *.txt)
• Dynamic Plugin Loading
  • Implemented PluginLoader class for runtime plugin loading
  • Added automatic plugin discovery from standard directories
  • Support for YAMS_PLUGIN_DIR environment variable
  • Onnx model provider now loads dynamically as a plugin
  • Daemon automatically loads plugins on startup
  • Created comprehensive test suite for plugin loader

Fixed

• Tag Storage and Filtering
  • Fixed tag filtering in list command not working (--tags option)
  • Tags are now properly parsed as comma-separated values when stored
  • Fixed tag filtering logic to correctly match against stored comma-separated tag values
  • Multiple tags in filter work with OR logic (e.g., --tags "work,important" shows documents with either tag)
• Daemon Path Resolution
  • Fixed "Added 0 documents" issue when using relative paths with daemon
  • CLI now converts relative paths to absolute paths before sending to daemon
  • Added proper error messages for invalid paths and non-recursive directory attempts
  • Daemon now returns helpful error messages instead of silently returning 0
• Feature Parity Between CLI and Daemon
  • Added all missing fields to GrepRequest for complete CLI feature parity
  • Added all missing fields to SearchRequest for complete CLI feature parity
  • Updated serialization/deserialization for both request types
  • CLI commands now pass all parameters to daemon requests
  • Ensures consistent behavior whether using daemon or local execution
• Wildcard Pattern Matching Bug
  • Fixed broken wildcard pattern matching in add and restore commands
  • Replaced regex-based implementation with efficient iterative algorithm
  • Patterns like *.sol, *.cpp, *.js now work correctly
  • Dots and other special characters in patterns are now handled properly
• Linux Build Errors
  • Fixed missing #include <cstring> in compression_benchmark.cpp for std::strlen
  • Fixed C++ compiler flag -Wnon-virtual-dtor being incorrectly applied to C files
  • Used CMake generator expression $<$<COMPILE_LANGUAGE:CXX>:-Wnon-virtual-dtor> for language-specific flags
  • Fixed missing Rabin chunker header include in ingestion_benchmark.cpp
  • Fixed GCOptions struct initializer warnings by adding progressCallback field
  • Fixed query parser benchmark Result access patterns (use .value() instead of *result)
  • Fixed metadata benchmark Database constructor usage pattern
  • Fixed benchmark API calls to use storeBytes() instead of non-existent addContent()
• Vector Index Loading
  • Fixed empty error message when vector index file doesn't exist
  • Added proper file existence check before attempting to load
  • Shows debug message instead of warning for missing index file on startup

[v0.4.7] - 2025-08-21

Fixed

• Release Workflow Build Failures
  • Fixed std::from_chars compilation error on macOS hosted runners (Xcode 15.2)
  • Replaced std::from_chars with portable std::stoull in http_adapter_curl.cpp for parsing Content-Length headers
  • Fixed missing benchmark package in Linux self-hosted validation builds
  • Added proper Conan options (-o build_tests=True -o build_benchmarks=True) to validation build step
  • Ensures compatibility with older macOS standard libraries that lack full C++20 support

[v0.4.6] - 2025-08-21

Changed

• MCP Server Tool Naming
  • Simplified tool names to match CLI commands for better consistency
  • Tools now use generic names: search, grep, download, get, list, store, add, delete, cat, update, stats

Fixed

• Added proper lifetime management for io_uring operations to prevent accessing freed memory
• Implemented operation tracking and cancellation to ensure clean shutdown
• Storage Backend Improvements
  • Fixed FilesystemBackend sharding to use hash-based approach for consistent key distribution
  • Replaced key-prefix sharding with SHA256 hash-based sharding to avoid path conflicts
• Chunking Deduplication
  • Fixed RabinChunker deduplication by resetting window state at chunk boundaries
  • Ensures identical data patterns produce identical chunks for proper deduplication
• Code Improvements
  • Added automatic configuration correction for invalid chunking configs (when min > max)
  • Enhanced preprocessText() to trim leading/trailing whitespace
  • Improved paragraph boundary detection to point after "\n\n" markers
• CI/CD
  • Fixed release workflow by changing preset from conan-validation to conan-release

Known Issues

• Temporarily Disabled Tests (to be fixed in v0.5.0)
  • VectorIndexManager: removeVector, index persistence, HNSW operations not implemented
  • ModelManagement: Registry initialization issues in test environment
  • OnnxRuntime: Tests timeout waiting for non-existent model files
  • All disabled tests are marked with TODO(v0.5.0) comments for tracking

[v0.4.5] - 2025-08-21

Fixed

• MCP Server
  • Added mutex protection to StdioTransport for thread-safe I/O operations
  • Fixed potential JSON response interleaving when multiple clients connect
  • Prevents "Expected ',' or ']' after array element" errors in concurrent scenarios
  • Fixed missing includes for file operations
• Test Failures
  • Fixed FilesystemBackend::list() key reconstruction from sharded paths
  • Fixed ManifestManager statistics by moving static counters to member variables
  • Fixed file type detection consistency in CommandIntegrationTest
• Detection Module
  • Ensured FileSignature creation uses consistent methods for isBinary and fileType
  • Fixed mismatch between FileSignature fields and classification methods

[v0.4.4] - 2025-08-21

CI version bump

[v0.4.3] - 2025-08-20

Fixed

• Build System
  • Updated GTest from 1.14.0 to 1.15.0 for Conan 2.0 compatibility
• Homebrew Formula
  • Fixed documentation URL in Homebrew formula to point to correct repository

Known Issues

• Daemon may crash when processing certain stats requests (investigation ongoing)

[v0.4.2] - 2025-08-20

Added

• Linux Package Support
  • Added AppImage support for universal Linux distribution
  • Integrated package building into GitHub release workflow

Fixed

• Linux Build Compilation
  • Fixed C++ template name lookup issues in message_serializer.cpp for GCC 13
  • Added namespace qualification to 31 deserialize calls for proper template resolution
  • Fixed missing <utility> header in async_socket.h and connection_pool.h for std::exchange
  • Fixed missing <netinet/in.h> header in async_socket.cpp for IPPROTO_TCP constant
• Position-Independent Code (-fPIC) Linker Errors
  • Fixed shared library linking errors by enabling PIC for all static libraries in dependency chain
  • Added POSITION_INDEPENDENT_CODE ON property to 11 static libraries
  • Resolved yams_onnx_plugin.so build failure on Linux x86_64
• Missing Symbol Linker Errors
  • Fixed undefined reference to HybridFuzzySearch by linking yams_metadata to yams::search
  • Resolved circular dependency issues in library linking order
• CLI Output Cleanup
  • Changed TextExtractorFactory initialization logging from info to debug level
  • Removed spurious log messages from normal CLI output
• Homebrew Formula
  • Fixed documentation URL in Homebrew formula to point to correct repository

[v0.4.1] - 2025-08-20

Added

• Smart Text Extraction by Default
  • CLI commands now extract text from PDFs, HTML, and other supported formats by default
  • cat command: Shows readable text instead of binary data for PDFs and HTML files
  • get command: Auto-detects output destination - extracts text for terminal, raw for files
  • --raw flag added to both commands for accessing original content when needed
  • --extract flag for get command to force text extraction even when piping

Changed

• MCP Server Text Extraction
  • Now uses TextExtractorFactory for all supported file types (not just HTML)
  • PDF text extraction works correctly in MCP tools
  • Consistent text extraction behavior across CLI and MCP interfaces

Fixed

• PDF Text Extraction in MCP
  • Fixed issue where PDF files returned empty content in MCP tools
  • MCP server now properly extracts text from PDFs using PDFium
  • Added missing HtmlTextExtractor to build system
• Compilation Issues
  • Fixed missing HtmlTextExtractor in CMakeLists.txt
  • Fixed ErrorCode enum references in HTML text extractor
  • Fixed regex_replace lambda usage for C++ standard compliance
• IPC Response variant construction
• GitHub Actions release workflow: wrap embedded multi-line Python f-string code in bash here-docs and pass JSON file path via argv in the benchmarks block to avoid shell syntax errors on runners.

[v0.4.0] - 2025-08-20

Added

• Universal Content Handler System
  • New IContentHandler interface supporting all file types with metadata extraction
  • ContentHandlerRegistry with thread-safe handler management
  • TextContentHandler adapter wrapping existing PlainTextExtractor
  • PdfContentHandler wrapper for existing PdfExtractor
  • BinaryContentHandler as universal fallback for unknown file types
  • Updated DocumentIndexer to use new ContentHandlerRegistry system
  • Maintained backward compatibility with legacy TextExtractor system
• High-Performance Daemon Architecture
  • New yams-daemon background service for persistent resource management
  • Unix domain socket IPC with zero-copy transfers for large payloads
  • Automatic daemon lifecycle management with configurable policies
  • yams daemon start/stop/status/restart command suite
  • Auto-start on first command if daemon enabled in config
  • Shared EmbeddingGenerator across all operations eliminates model loading overhead
  • VectorIndexManager cached in daemon memory
• Robust Downloader Module
  • New download module with libcurl adapter, repo-local staging
  • SHA-256 integrity verification and rate limiting
  • Atomic finalize into CAS (store-only by default)
  • New yams download command returning JSON {hash, stored_path, size_bytes}
  • Progress output (human/json) with no user-path writes unless export requested
• Configuration v2 Architecture
  • New [daemon] section for service configuration
  • [daemon.models] for model lifecycle management
  • [daemon.lifecycle] for auto-start and shutdown policies
  • [daemon.ipc] for communication tuning
  • Backward compatible with direct mode (no daemon)
• Enhanced Search Capabilities
  • --literal-text flag for search and grep commands
  • Treats query patterns as literal text instead of regex/operators
  • Works across all search engines (fuzzy, hybrid, metadata, vector)
  • Example: yams search "call(" --literal-text safely searches for parentheses

Changed

• Performance Architecture
  • All CLI commands can leverage shared daemon resources
  • Shared result renderer system across CLI and daemon
  • Deferred initialization eliminates startup overhead
• MCP Integration Improvements
  • Removed HTTP transport support, now stdio-only for cleaner local integration
  • Improved EmbeddingGenerator lifecycle with lazy loading
  • Better resource management with on-demand initialization

Fixed

• Daemon Integration Issues
  • Eliminated "Failed to preload model" warnings
  • Fixed daemon configuration defaults for missing config sections
  • Daemon now gracefully handles missing [daemon] sections in config files
• Configuration System
  • ConfigMigrator properly handles v1 to v2 migrations without breaking existing configs
• MCP Schema Compliance
  • Fixed "Expected object, received null" error in tool definitions
  • Removed empty properties: {} fields from tools with no parameters
  • MCP server now loads correctly without schema validation errors
• Search Engine Robustness
  • Special characters in search queries no longer break FTS5
  • Automatic sanitization prevents syntax errors from ()[]{}*" characters
  • All search paths (fuzzy, hybrid, metadata) handle special characters safely
  • Raw query strings flow through pipeline unchanged until FTS5 level
• Critical Compilation Errors
  • Fixed ChunkingStrategy enum reference mismatch in document chunker (FixedSize → FIXED_SIZE)
  • Fixed constructor initialization order in VectorIndexOptimizer to match member declaration order
  • Fixed compression level configuration in RecoveryManager (now uses level 3 per performance benchmarks)
• Code Quality Issues
  • Fixed hundreds of uninitialized variable errors identified by cppcheck analysis
  • Eliminated critical errors in recovery_manager.cpp, error_handler.cpp, and metadata_api.cpp
  • Added proper RAII initialization patterns across vector and compression components
• Build System Stability
  • All modules now compile successfully without errors
  • Fixed warning configurations that were breaking dependency builds
  • Improved cross-platform compilation compatibility