yams/docs/changelogs/v0.5.md

# Changelog Archive: v0.5.x Series

## [v0.5.9] - 2025-08-29

### Added
- Daemon Repair Coordinator (FSM-driven) enabled by default when daemon is running
  - Performs idle-only embedding repair using existing repair utilities
  - New daemon repair metrics in StateComponent:
    - repairIdleTicks, repairBusyTicks, repairBatchesAttempted,
      repairEmbeddingsGenerated, repairEmbeddingsSkipped, repairFailedOperations
- IPC streaming persistence
  - Removed unsolicited end-of-stream heartbeat frame that could poison next response
  - Fixed message framing hang under certain pooled flows; reduced over-eager retries
- MCP server alignment with pooled client API
  - Corrected DaemonClientPool::acquire() usage (Result<Lease> → Lease)

### Known Issues
- Pooled reuse immediately after a streaming response can intermittently see
  "Connection closed by daemon" on follow-up unary calls (e.g., list/ping).
### Changed
- Coordinator ownership/lifecycle: now a YamsDaemon member started in start() and stopped before
  IPC/services teardown in stop() for safe shutdown

### Fixed
- Regressions in CLI / MCP tooling
- AddressSanitizer heap-use-after-free in IPC tests caused by coordinator thread outliving
  daemon members; resolved by proper ownership and shutdown ordering
  - Readiness flags and lightweight observability counters for repair gating
- IPC/Streaming & Pooling
  - Connection FSM guardrails with legal transitions for streaming and normalization
  - Header-first streaming responses with chunked transfer; persistent sockets supported
  - Client: pre-header reconnect-once (timeout/EOF/reset) and inter-chunk inactivity deadlines
  - Time-to-first-byte (TTFB) metric and response timing hooks
  - Streaming enabled for search, list, grep, and add document paths

## [v0.5.8] - 2025-08-29

### Fixed
- Release workflow: replaced invalid fromJSON() usage for matrix runner selection with direct `matrix.runs_on` reference.
- Docker workflow: stabilized ARM64 build by moving from macOS runner (no Docker daemon) to ubuntu-latest with QEMU emulation; added early `docker info` checks for both arches.

### Changed
- Docker: standardized multi-arch build steps (QEMU + Buildx) and ensured consistent labels/tags across architectures.

## [v0.5.7] - 2025-08-2

### Known Issues
- Daemon Pool connection issues for updated streaming commands (search, grep, list)
  - Search will return with backup, but grep will not

### Added
  - Implemented tinyfsm-based state machine for connection lifecycle management
  - Added per-connection FSM adapter (`yams::ipc::ConnectionFsm`) with clean state transitions
  - Vendored tinyfsm under `third_party/tinyfsm/` with MIT license compliance
  - Added comprehensive state machine documentation and integration guide
- **Groundwork for upcoming work**
  - Shared qualifier parser: inline query qualifiers (lines:, pages:, section:, selector:, name:, ext:, mime:) with quotes and spaces supported.
  - Hybrid search normalization: qualifiers are stripped from scoring text and passed as structured hints to engines.
  - Lines-range scoped snippets in hybrid engine: when `lines:` is specified, result snippets are sliced to requested line windows.
  - Integrated PDF content extraction into search, cat, get. This will evolve into a plugin system to support custom file type search from stored files.
- Introduced daemon pooled request pattern across MCP/CLI boundaries.
- Lightweight `daemon_first` wrapper in CLI that uses `DaemonClient::call<T>` to preserve daemon-first behavior while commands migrate to pooled execution.

### Changed
- **Daemon Client/Server Pipeline Refactor**
  - **AsyncSocket Simplification**: Replaced complex generation-based tracking with tinyfsm state machine
    - Removed atomic generation counters and complex mutex operations
    - Implemented clean state transitions: Disconnected → Connecting → Connected → ReadingHeader → ReadingPayload → WritingHeader → StreamingChunks → Error/Closed
    - Eliminated race conditions in socket lifecycle management
    - Reduced memory footprint and improved debuggability
  - **DaemonClient Pipeline Consolidation**: Unified timeout and request handling
    - Consolidated timeout configuration (removed scattered env vars + config mixing)
    - Single `PipelineConfig` struct with clear precedence rules
    - Streamlined request execution with `execute<T>()` template method
    - Centralized send_message/receive_message helpers
    - Improved error context with operation details
  - **AsyncIpcServer Threading Cleanup**: Simplified thread pool management
    - Clear separation between IO threads and worker threads
    - Improved lifecycle management and resource cleanup
    - Better thread naming and monitoring capabilities
- Cleaned up build system and devcontainer
- Hybrid/metadata parity: name/ext/mime qualifiers are honored across engines
  - Hybrid: mapped to `vector::SearchFilter.metadata_filters` and enforced by both vector and keyword paths.
  - Metadata fallback: qualifiers merged into service filters (pathPattern, extension, mimeType) so behavior matches hybrid.
- MCPServer constructor simplified to `(std::unique_ptr<ITransport>, std::atomic<bool>*)`; serve command and yams-mcp tool updated to STDIO-only transport.
- `daemon_helpers` generalized to accept generic callables (lambdas) instead of `std::function`, improving template deduction and avoiding copies.

### Fixed
- Search service: hash-prefix path used the wrong local variable; corrected to ensure path/type filters are applied correctly.
- Metadata search: removed references to non-existent request fields and unified name filtering via path pattern.
- MCP tools/call responses were not rendered by some clients due to missing MCP content wrapper. Non-MCP-shaped tool results are now wrapped in a top-level content array.
- search: paths_only results could be empty; fixed to return service-provided paths.
- list: limit and offset were not wired from MCP -> services; pagination is now honored.
- Protocol alignment: removed non-existent fields from daemon `SearchRequest` mapping and explicitly cast similarity to `double`.
- Error handling: replaced non-existent `ErrorCode::Unavailable` with `ErrorCode::NetworkError`.
- Type mapping: `DeleteRequest` now maps to `DeleteResponse` in response traits.
- Duplicates: removed duplicate `clientInfo_`/`negotiatedProtocolVersion_` members and the extra `handleUpdateMetadata` definition in MCP.
- Linking: integrated daemon IPC implementation into `yams_mcp` and linked `ZLIB::ZLIB` (CRC32) to resolve `AsyncSocket`/`ConnectionPool` and checksum undefined symbols.
- Windows CI job fix (this will be a process)
- Release build fix for artfact upload on partial success

### Notes
- CLI migration to pooled requests will proceed per-command; the temporary `daemon_first` wrapper keeps commands working until each is switched to the pooled manager.

## [v0.5.6] - 2025-08-24

**CI version bump**
- Release formatting fix

## [v0.5.5] - 2025-08-24

**CI version bump**
- Releases now generate regardless of build failures

### Fixed
- Grep supports semantics search using embeddings

## [v0.5.4] - 2025-08-24

### Fixed
- Resolved MCP tool listing error
- Added mutex imports for macos x86 ci builds

## [v0.5.3] - 2025-08-24

**CI version bump**

## [v0.5.2] - 2025-08-24

### Changed
- Release workflow: made Linux validation build non-blocking in release pipeline to allow packaging and publishing artifacts even if validation step fails. Validation failures are surfaced in job logs and summary but no longer block artifact creation.

### Fixed
- Linux build: guarded macOS-specific Mach headers in RequestDispatcher with __APPLE__ and added Linux-friendly memory/CPU sampling using /proc, fixing fatal includes on Linux targets.

## [v0.5.1] - 2025-08-24

### Known Issues
- Daemon connection exhaustion: running many concurrent yams processes (e.g., find -exec yams add …) can hit the default max_connections=100 and cause rejections. Mitigation: prefer recursive/batched commands or cap concurrency (e.g., xargs -P 4). Consider raising max_connections for CI.
- Regex alternation in yams search: search currently treats queries as keyword/FTS; compound regex expressions (e.g., A|B) aren’t supported there yet. Use yams grep for regex, or await a future --regex mode in search.

## [v0.5.0] - 2025-08-22

### Added
- **Service-Based Architecture**
  - Complete app services layer implementation (`app/services/*.cpp`)
  - Service factory pattern with centralized AppContext management
- **RequestDispatcher Service Integration**
  - All daemon handlers migrated to use app services exclusively
  - Complete feature parity between CLI and daemon operations
  - Enhanced protocol mapping with proper field alignment
- **SearchCommand Service Migration**
  - Tag filtering support with `--tags` and `--match-all-tags` options
  - Comma-separated tag parsing with whitespace trimming
  - Complete daemon_first pattern implementation with local fallback
- **IndexingService Integration**
  - All document ingestion centralized through IndexingService
  - Eliminated direct IndexingPipeline calls from CLI commands
  - Directory recursion uses IndexingService::addDirectory() for consistency
  - Embedding lifecycle managed exclusively by Indexin Service
- **Auto-Repair for Vector Embeddings**
  - Daemon automatically generates missing embeddings when documents are added (hot-load behavior)
  - New `enableAutoRepair` configuration flag in DaemonConfig (default: true)
  - Background repair thread processes queued documents in batches
  - Automatic detection and repair during semantic search operations
  - Shared repair utility (`yams/repair/embedding_repair_util.h`) for CLI and daemon use
- **Vector Index Persistence**
  - Implemented `saveIndex()` and `loadIndex()` methods for VectorIndexManager
  - Binary serialization format for index state and metadata
  - Automatic index saving on daemon shutdown
- **Vector Management**
  - Implemented `removeVector()` method for single vector deletion
  - Support for removing vectors from both flat and HNSW indices
- **HNSW Index Support**
  - Complete implementation of Hierarchical Navigable Small World index
  - Configurable M and ef_construction parameters
  - Efficient approximate nearest neighbor search
- **Improved Daemon Lifecycle Management**
  - New daemon instances now automatically kill existing daemons on startup
  - Prevents resource conflicts and stale daemon processes
  - Ensures clean daemon state for each new session
  - **Note**: Older YAMS versions require manual daemon termination before upgrading
- **Service-Based Test Architecture**
  - New comprehensive service layer tests: `IndexingService`, `DocumentService`, and `SearchService`
  - Modern content handler tests using Universal Content Handler System (`pdf_content_handler_test.cpp`)
  - CMakeLists.txt structure for organized test building and discovery

### Fixed
- **Service Interface Field Mapping**
  - Fixed field name mismatches between daemon protocols and service interfaces
  - Corrected `lineNumbers` vs `showLineNumbers` mapping in RequestDispatcher, MCP server, and SearchCommand
  - Resolved compilation errors from protocol field alignment issues
  - Fixed RequestDispatcher handlers to properly map request fields to service parameters
- **Vector Index Persistence Deadlock**
  - Fixed critical mutex deadlock in VectorIndexManager loadIndex() method
  - IndexPersistence test no longer hangs indefinitely during deserialization
  - Fixed deadlock by properly releasing mutex before calling initialize()
  - Added robust error checking for stream operations during serialization/deserialization
  - Enhanced data validation with sanity checks for corrupted data
  - Vector index save/load functionality now works correctly
- **Model Loading Performance**
  - Removed unnecessary async wrappers causing 15+ minute timeouts
  - Model loading now completes in seconds instead of timing out
  - Fixed deadlock issues in OnnxModelPool initialization