yams/docs/changelogs/v0.2.md

# Changelog Archive: v0.2.x Series

## [v0.2.9] - 2025-08-14

**Enhanced Deduplication Reporting**
  - New stats --dedup command for block-level analysis
  - Shows deduplication ratio, space savings, most duplicated blocks
  - Validated 32% deduplication on real codebases
**CI version bump**
  - Fixed CMake export error with spdlog dependency
  - Resolved CI failures on macOS x64 and Linux x86

## [v0.2.8] - 2025-08-14

### Added
- **Repair Command**: New comprehensive storage maintenance command
  - `yams repair --orphans`: Clean orphaned metadata entries (removed 28,101 orphaned entries in testing)
  - `yams repair --chunks`: Remove orphaned chunk files (freed 38MB from 23,988 orphaned chunks)
  - `yams repair --mime`: Fix missing MIME types in documents
  - `yams repair --optimize`: Vacuum and optimize database (reduced 314MB to 272MB)
  - `yams repair --all`: Run all repair operations
  - Dry-run support with `--dry-run` flag for safe preview
  - Force mode with `--force` to skip confirmations

- **Delete Command Enhancements**:
  - Added `--directory` option for recursive directory deletion
  - Improved metadata cleanup when deleting documents
  - Added progress indicator for large deletion operations
  - Properly removes both manifest and chunk files

- **Config Command Enhancements**:
  - Added compression tuning support via configuration
  - `config get` - Retrieve configuration values
  - `config set` - Set configuration values (placeholder)
  - `config list` - List all configuration settings
  - `config validate` - Validate configuration file
  - `config export` - Export configuration in TOML or JSON format

### Fixed
- **Stats Command**:
  - Fixed object file counting (was showing 0, now correctly counts chunks)
  - Fixed unique document calculation to match total when no duplicates exist
  - Added orphaned chunk detection and reporting
  - Fixed deduplication stats to only show when duplicates exist
  - Properly filters orphaned metadata entries from counts (28,101 orphaned found)
  - Added chunk health reporting with progress indicator
  - Fixed verbose mode not being properly tracked

- **Storage Path Issues**:
  - Fixed duplicate "storage" in path construction bug throughout codebase
  - Fixed content_store_impl.cpp manifest and object paths
  - Fixed same issue in delete_command.cpp and repair_command.cpp

- **Chunk Management Crisis**:
  - Delete command wasn't properly removing chunks when ref_count reached 0
  - Implemented proper chunk cleanup when documents are deleted
  - Added reference counting verification in repair command
  - Fixed hash format mismatch between database (64 chars) and filesystem (62 chars + 2 char directory)

- **Database Issues**:
  - Fixed view vs table issue with `unreferenced_blocks` (it's a VIEW not a table)
  - Properly clean `block_references` table where `ref_count = 0`
  - Added database optimization to reclaim space from deleted entries
  - Database reduced from 314MB to 272MB after VACUUM

### Changed
- **Stats Output**: Improved formatting and information display
  - Shows storage overhead calculation
  - Displays warnings for orphaned chunks and metadata
  - Only shows deduplication info when relevant
  - Added database inconsistency warnings
  - Better progress indicators for long operations

## [v0.2.7] - 2025-08-14
### Fixed
- **CI version bump**
  - Conan flag fix in CI

## [v0.2.6] - 2025-08-14
### Fixed
- **CI version bump**
  - Conan flag fix in CI

## [v0.2.5] - 2025-08-14
### Fixed
- **CI version bump**
  - Updated git pages site
  - Build system improvements

## [v0.2.4]

### Fixed
- **MCP Server Signal Handling**
  - Server now responds properly to Ctrl+C (SIGINT) signals
  - Replaced std::signal with sigaction to allow interrupting blocking I/O
  - Added non-blocking stdin polling with 100ms timeout in StdioTransport
  - Server checks external shutdown flag in main loop
  - Handles EINTR errors and clears stdin error state properly
  - Added clear startup messages and usage instructions to stderr

- **Dockerfile**
  - Dockerfile should build correctly

- **Cross-Compilation Support**
  - Added explicit Conan profiles for x86_64 and ARM64 architectures
  - CI/CD workflows now properly handle cross-compilation with dual profiles
  - Fixed zstd assembly symbol errors when cross-compiling on Apple Silicon
  - Added zstd/*:build_programs=False option to avoid build issues
  - Workflows detect native vs cross-compilation scenarios automatically

### Added

- **MCP Server Documentation**
  - Comprehensive MCP usage guide at `docs/user_guide/mcp.md` (I personally use the CLI)

### Technical Details
- **Signal Handling**: Uses sigaction with sa_flags=0 to interrupt blocking system calls
- **Non-blocking I/O**: Implements poll() on Unix/macOS for stdin availability checking
- **Cross-Compilation**: Uses Conan's dual profile system (-pr:b for build, -pr:h for host)

## [v0.2.3] - 2025-08-14
### Fixed
- **CI version bump**
  - README updates (removed install instructions with curl, download binary)
  - Build system improvements for docker files and release

## [v0.2.2] - 2025-08-14

### Fixed
- **Docker Build C++20 Configuration**
  - Added explicit C++20 standard setting in Conan profile
  - Fixed "Current cppstd (gnu17) is lower than required C++ standard (20)" error
  - Ensures Docker builds use correct C++ standard matching project requirements
- **Release Notes Formatting**
  - Fixed Conan profile output contaminating GitHub release descriptions
  - Redirected debug output to stderr to keep release notes clean
  - Release pages now display properly formatted markdown
- **macOS Cross-Compilation**
  - Fixed Boost build failures when cross-compiling x86_64 on ARM64 macOS
  - Disabled Boost.Locale and Boost.Stacktrace.Backtrace components that fail in cross-compilation
  - Enables successful builds on Apple Silicon machines with Rosetta 2

### Changed
- **Docker Build Process**
  - Switched from CMake presets to direct CMake commands with Conan toolchain
  - Added ninja-build to Docker dependencies for faster builds
  - More robust build process that doesn't rely on dynamically generated presets

## [v0.2.1] - 2025-08-13

### Fixed
- **Build System Standardization**: All workflows now use Conan exclusively
  - Fixed CMake export errors for spdlog and other dependencies
  - Release workflow migrated from plain CMake to Conan-based builds
  - Ensures consistent dependency management across CI and release
  - Resolves "target requires target that is not in any export set" errors
- **Docker Build Issues**
  - Fixed Conan 2.0 command syntax (`conan profile show` instead of `conan profile show default`)
  - Fixed CMake preset issues by using direct CMake commands with Conan toolchain
  - Added ninja-build to Docker dependencies for faster builds
  - Improved version handling: uses git tags for releases, `dev-<sha>` for development builds
  - Fixed undefined variable warnings in Dockerfile labels
  - Added proper build argument handling for YAMS_VERSION and GITHUB_SHA
- **Release Page Formatting**
  - Fixed Conan profile output contaminating release notes
  - Redirected debug output to stderr to keep release notes clean

### Changed
- **MIME Type Detection Default Behavior**
  - MIME types are now detected automatically during add operations
  - `repair-mime` command remains available for fixing existing documents
  - Detection uses FileTypeDetector with both signature and extension methods
- **Docker Versioning Strategy**
  - Tagged releases use semantic version from git tag
  - Development builds use `dev-<short-sha>` format
  - Provides clear distinction between releases and development builds

### Technical Details
- **Testing Interface**: MCP server now exposes test methods via conditional compilation
  - `#ifdef YAMS_TESTING` enables test-specific public methods
  - Replaces hacky `#define private public` approach
  - Maintains encapsulation while allowing proper unit testing
- **Build Workflows**: Unified Conan usage with proper profiles and caching
  - Conan profile detection with C++20 enforcement
  - Package caching for faster builds
  - Consistent preset usage (`conan-release`) across all platforms

## [v0.2.0] - 2025-08-13

### Added
- **LLM Ergonomics Enhancements**
  - Added `--paths-only` flag to search command for LLM-friendly output (one file path per line)
  - Enhanced codebase indexing workflows for AI agent integration
  - Improved documentation in PROMPT-eng.md with LLM-optimized YAMS patterns
  - Support for using YAMS as "memory layer for low context" in AI workflows
  - Magic number patterns for `#include`, `import`, `function`, `class`, shebang lines, etc.
  - Automatic programming language detection during recursive directory operations
- **Grep Command**: Full-featured regex search within indexed file contents
  - Standard grep options: `-A/--after`, `-B/--before`, `-C/--context` for context lines
  - Pattern matching options: `-i` (case-insensitive), `-w` (whole word), `-v` (invert)
  - Output options: `-n` (line numbers), `-H` (with filename), `-c` (count), `-l` (files with matches)
  - Color highlighting support with `--color` option
  - Supports searching all indexed files or specific paths
- **Knowledge Graph Integration in Get Command**
  - Added `--graph` flag to display related documents
  - Added `--depth` option (1-5) to control graph traversal depth
  - Shows documents in same directory and with similar extensions
  - Displays document relationships and metadata
- **Line-Level Search with Context**
  - Enhanced search command with `-n/--line-numbers` flag
  - Added context options: `-A/--after`, `-B/--before`, `-C/--context`
  - Highlights matching lines with color
  - Shows surrounding context lines for better understanding
- **C++17 Compatibility Mode** for broader system support
  - New `YAMS_CXX17_MODE` CMake option for legacy compiler compatibility
  - Support for GCC 9+, Clang 9+, older MSVC versions
  - Automatic fallbacks: `boost::span` for `std::span`, `fmt` for `std::format`
  - Compatible with Ubuntu 20.04 LTS, CentOS 8, older macOS versions
  - Comprehensive compatibility documentation and build matrix
  - Runtime feature detection with graceful degradation
- **Enhanced Build System Flexibility**
  - Dual-mode compilation: full C++20 for modern systems, C++17 for legacy
  - Automatic dependency management with Conan and FetchContent fallbacks
  - Compiler-specific optimizations and warning configurations
  - Performance-oriented feature selection based on available capabilities
- **MCP Server Feature Parity Enhancements** (~85% CLI-MCP parity achieved)
  - **Enhanced search_documents tool** with LLM ergonomics optimizations
    - Added `paths_only` parameter for LLM-friendly file path output (one per line)
    - Added line context parameters: `line_numbers`, `after_context`, `before_context`, `context`
    - Added `color` parameter for syntax highlighting control (always, never, auto)
  - **New grep_documents tool** with full CLI grep parity
    - Complete regex pattern matching across indexed document contents
    - Context options: `-A/--after`, `-B/--before`, `-C/--context` for surrounding lines
    - Pattern options: `ignore_case`, `word`, `invert` for flexible matching
    - Output modes: `line_numbers`, `with_filename`, `count`, `files_with_matches`, `files_without_match`
    - Color highlighting and `max_count` limit support
  - **Knowledge graph integration** in retrieve_document tool
    - Added `graph` parameter to include related documents in response
    - Added `depth` parameter (1-5) for graph traversal control
    - Added `include_content` parameter for full document content in graph responses
    - Finds documents in same directory, similar extensions, and metadata relationships
  - **New update_metadata tool** for document metadata management
    - Update metadata by document `hash` or `name` with automatic resolution
    - Support for multiple `key=value` metadata pairs in single operation
    - Verbose output option for detailed update information
  - **Enhanced list_documents tool** with comprehensive filtering and sorting
    - File type filtering: `type`, `mime`, `extension`, `binary`, `text` parameters
    - Time-based filtering: `created_after/before`, `modified_after/before`, `indexed_after/before`
    - Recent documents filter: `recent` parameter for N most recent files
    - Sorting control: `sort_by` (name, size, created, modified, indexed) and `sort_order` (asc, desc)
  - **Enhanced get_stats tool** with file type breakdown analysis
    - Added `file_types` parameter for detailed file type distribution
    - Shows file type counts, sizes, top extensions per type, and top MIME types

### Enhanced
- **MCP Tool Documentation** comprehensively updated with new schemas and examples
  - Added complete JSON schemas for all enhanced tools with parameter descriptions
  - Added practical usage examples for each tool capability
  - Updated integration documentation for desktop MCP client configuration
- **YAMS-first workflow guidance**: default to indexing and searching code via YAMS pre-watch; avoid external grep/find/rg for repository queries.
- **CLI docs enhanced** with pre-watch indexing workflow and YAMS-only search/grep examples.

### Technical Details
- **File type classification system** with MIME type analysis using `getFileTypeFromMime()` and `isBinaryMimeType()` helper methods
- **Knowledge graph traversal** leverages document relationships through metadata repository queries
- **LLM ergonomics optimizations** reduce context usage while maintaining full functionality
- **Tool schema definitions** follow JSON-RPC standards for seamless MCP integration

### Fixed
- **Indexing Tests**: Updated for new API with proper factory function declarations
- **Namespace Issues**: Fixed forward declarations in document_indexer.h
- **CLI11 Conflicts**: Resolved `-h` flag conflict in grep command by using only `--no-filename`
- **Test Suite Stabilization**
  - Fixed SEGV crash in vector_database_tests (DocumentChunkerTest bounds checking)
  - Fixed AddressSanitizer container overflow in detection_tests
  - Added thread safety to FixtureManager test infrastructure with proper mutex protection
  - Added thread safety to FileTypeDetector for libmagic operations (not thread-safe by default)
  - Fixed compilation errors in multiple test files (header includes, error codes, GMock dependencies)
  - Fixed EXPECT_NO_THROW macro usage in detection tests
- **Recursive Add Pattern Parsing**
  - Fixed CLI11 comma-separated pattern handling in `--include` and `--exclude` options
  - Now properly splits patterns like `"*.cpp,*.h,*.md"` into individual patterns
  - Added proper whitespace trimming and empty pattern filtering
- **CI/CD Pipeline Improvements**
  - Fixed Conan options quoting for proper test builds (`"yams/*:build_tests=True"`)
  - Added explicit `YAMS_BUILD_TESTS=ON` to CMake configuration
  - Added individual test target builds before running ctest
  - Implemented graceful test failure handling (CI continues, reports detailed results)
  - Added ASAN relaxation options for CI environment compatibility
  - Fixed YAML syntax error in release.yml (Python script indentation)