yams/CHANGELOG.md

# Changelog

All notable changes to YAMS (Yet Another Memory System) will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

- [SourceHut](https://sr.ht/~trvon/yams/): https://sr.ht/~trvon/yams/

## Archived Changelogs
- v0.7.x archive: docs/changelogs/v0.7.md
- v0.6.x archive: docs/changelogs/v0.6.md
- v0.5.x archive: docs/changelogs/v0.5.md
- v0.4.x archive: docs/changelogs/v0.4.md
- v0.3.x archive: docs/changelogs/v0.3.md
- v0.2.x archive: docs/changelogs/v0.2.md
- v0.1.x archive: docs/changelogs/v0.1.md

## [v0.7.10] - Unreleased

### Added
- **Graph command `--list-types` flag (yams-66h)**: Node type discovery for knowledge graph
  - New `--list-types` flag shows all distinct node types with counts
  - Table output with TYPE and COUNT columns, ordered by count descending
  - JSON output with `nodeTypes` array containing `type` and `count` fields
  - Added `getNodeTypeCounts()` method to `KnowledgeGraphStore` interface
  - Extended `GraphQueryRequest` IPC protocol with `listTypes` mode
  - Usage hint when no nodes found: suggests `yams add <path>`
  - Location: `src/cli/commands/graph_command.cpp`, `include/yams/metadata/knowledge_graph_store.h`
- **P4 language support for symbol extraction**: Network data plane language (P4_16)
  - Node types: `headerTypeDeclaration`, `structTypeDeclaration`, `controlDeclaration`, `parserDeclaration`, `actionDeclaration`, `tableDeclaration`
  - Query patterns for actions, functions, headers, structs, controls, parsers, tables, typedefs
  - Aliases: `p4`, `p4_16`, `p4lang`
  - Grammar auto-download from `prona-p4-learning-platform/tree-sitter-p4`
- **Vector diagnostics in DaemonMetrics**: Moved `collect_vector_diag` to background polling
  - Added `vectorEmbeddingsAvailable`, `vectorScoringEnabled`, `searchEngineBuildReason` to `MetricsSnapshot`
  - Status requests now read from cached snapshot (non-blocking)
  - Resolves status command hangs when vector services are slow
- **Entity extraction metrics in status output**: Added entity queue/inflight counters
  - New metrics: `entityQueued`, `entityDropped`, `entityConsumed`, `entityInFlight`
  - Exposed via `yams status` and `yams status -v` output
  - JSON output includes `entity_queued`, `entity_consumed`, `entity_dropped`, `entity_inflight`
  - Location: `include/yams/daemon/components/DaemonMetrics.h`, `src/cli/commands/status_command.cpp`
- **Gitignore support for directory ingestion**: Skip files matching `.gitignore` patterns
  - New `--no-gitignore` flag for `yams add` command to disable gitignore filtering
  - Default behavior respects `.gitignore` patterns in the root directory
  - Supports standard gitignore patterns: wildcards, directory patterns, anchored paths
  - Location: `src/cli/commands/add_command.cpp`, `src/app/services/indexing_service.cpp`

### Changed
- **Constexpr language configuration for symbol extraction**: Centralized compile-time configuration
  - 17 languages with constexpr node types and query patterns: C, C++, Python, Rust, Go, Java, JavaScript, TypeScript, C#, PHP, Kotlin, Perl, R, SQL, Solidity, Dart, P4
  - `LanguageConfig` struct with `class_types`, `field_types`, `function_types`, `import_types`, `identifier_types`
  - Query patterns: `function_queries`, `class_queries`, `import_queries`, `call_queries`
  - Language alias support (e.g., "cpp" → "c++", "cxx", "cc")
  - `getLanguageConfig()` constexpr lookup function
  - Location: `plugins/symbol_extractor_treesitter/symbol_extractor.cpp`
- **Field extraction**: New `extractFields()` method extracts class member variables
  - Uses node type traversal with language-specific field types
  - Creates `field` kind symbols with proper byte ranges
- **Member containment relations**: New `extractMemberRelations()` method
  - Creates `contains` edges from classes to their methods/fields
  - Uses byte range containment to determine class membership
  - Improves knowledge graph structure for code navigation
- **PostIngestQueue per-stage metrics**: Exposed extraction/KG/symbol stage inflight counts
  - New getters: `extractionInFlight()`, `kgInFlight()`, `symbolInFlight()`, `totalInFlight()`
  - Static constexpr limits: `maxExtractionConcurrent()`, `maxKgConcurrent()`, `maxSymbolConcurrent()`
  - Exposed via daemon status: `extraction_inflight`, `kg_inflight`, `symbol_inflight`
  - `yams status` shows POST line when there's active work
  - `yams status -v` shows per-stage breakdown
  - `yams daemon status -d` shows full Post-Ingest Pipeline section
  - JSON output includes `stages` object with per-stage counts
  - Location: `include/yams/daemon/components/PostIngestQueue.h`, `src/cli/commands/status_command.cpp`, `src/cli/commands/daemon_command.cpp`
- **PostIngestQueue dynamic concurrency scaling (PBI-05a)**: Auto-scale based on queue depth
  - New TuneAdvisor tunables: `postExtractionConcurrent()`, `postKgConcurrent()`, `postSymbolConcurrent()`, `postEntityConcurrent()`
  - Dynamic limits replace static constexpr values in PostIngestQueue pollers
  - TuningManager scales concurrency based on queue depth thresholds:
    - `>1000 queued`: extraction=hwThreads/2, kg=hwThreads/2
    - `>500 queued`: extraction=hwThreads/4, kg=32
    - `>100 queued`: extraction=hwThreads/8+4, kg=16
    - `>10 queued`: extraction=8
    - `idle`: extraction=4 (default)
  - Status output shows limits: `stages: extract=4/4, kg(q=0/i=0/8), symbol=0/4`
  - JSON includes `extraction_limit`, `kg_limit`, `symbol_limit`, `entity_limit`
  - Location: `include/yams/daemon/components/TuneAdvisor.h`, `src/daemon/components/TuningManager.cpp`, `src/daemon/components/DaemonMetrics.cpp`
- **Knowledge Graph cleanup on document deletion**: Deleting documents now cascades to KG
  - `deleteNodesForDocumentHash()`: Removes `doc:<hash>` nodes and symbol nodes with matching document_hash
  - Integrated into document deletion flow for automatic cleanup
  - Location: `include/yams/metadata/knowledge_graph_store.h`, `src/app/services/document_service.cpp`
- **Stale edge cleanup on re-indexing**: Symbol extraction now cleans up old relationships
  - `deleteEdgesForSourceFile()`: Removes edges where `properties.source_file` matches path
  - Called automatically before re-extraction to prevent stale relationship accumulation
  - Location: `src/daemon/components/EntityGraphService.cpp`
- **Optimized isolated node query**: `yams graph --isolated` now uses single SQL query
  - `findIsolatedNodes()`: Efficient `NOT EXISTS` subquery instead of N+1 pattern
  - New IPC fields: `isolatedMode`, `isolatedRelation` in `GraphQueryRequest`
  - Significant performance improvement for large graphs
  - Location: `src/cli/commands/graph_command.cpp`, `src/daemon/components/dispatcher/request_dispatcher_graph.cpp`
- **Daemon log command**: Added `yams daemon log`
- **ExternalPluginHost**: New plugin host for Python/process-based plugins (RFC-EPH-001)
  - Implements `IPluginHost` interface for external plugins running as separate processes
  - JSON-RPC 2.0 communication over stdio using existing `PluginProcess` and `JsonRpcClient`
  - Supported plugin types: Python (`.py`), Node.js (`.js`), any executable with JSON-RPC support
  - Process lifecycle management: spawn, monitor, health checks, graceful shutdown
  - Automatic crash recovery with configurable restart policy (max retries, backoff)
  - Trust-based security model with persistent trust file
  - RPC gateway for calling arbitrary plugin methods (`callRpc`)
  - Plugin statistics tracking (uptime, restart count, health status)
  - State change callbacks for monitoring plugin lifecycle events
  - Location: `include/yams/daemon/resource/external_plugin_host.h`, `src/daemon/resource/external_plugin_host.cpp`
- **Auto-init mode**: New `yams init --auto` flag for containerized/headless environments
  - Enables vector database with default model (`all-MiniLM-L6-v2`)
  - Enables plugins directory setup
  - Generates authentication keys
  - Skips S3 configuration (uses local storage)
  - Non-interactive: no prompts, uses sensible defaults
- **Tree-sitter grammar download**: `yams init` now offers to download tree-sitter grammars
  - Interactive menu: recommended (C, C++, Python, JS, TS, Rust, Go), all, or custom selection
  - Auto-downloads and builds grammars from official GitHub repos
  - Supports 14 languages: C, C++, Python, JavaScript, TypeScript, Rust, Go, Java, C#, PHP, Kotlin, Dart, SQL, Solidity
  - Cross-platform: MSVC, MinGW, GCC, Clang compilation support
  - Grammar prompt also available when YAMS is already initialized
  - Grammars installed to XDG_DATA_HOME/yams/grammars (Unix) or %LOCALAPPDATA%\yams\grammars (Windows)
- **New embedding model option**: Added `multi-qa-MiniLM-L6-cos-v1` as second model choice
  - Trained on 215M question-answer pairs for semantic search optimization
  - Same dimensions (384) as default model for compatibility
  - Replaces `all-mpnet-base-v2` (768 dim) in model selection
- **Git-based version detection**: Build system now auto-detects version from git tags
  - Uses most recent semver tag (`v*`) as effective version
  - Falls back to project version only if no tags exist
  - Command-line override (`-Dyams-version=X.Y.Z`) takes highest priority
- **Commit hash in version output**: `yams --version` now shows short commit hash
  - Format: `0.7.9 (commit: c16939f) built:2025-11-29T17:30:15Z`
  - Helps identify exact build for bug reports and debugging
- **Init command tests**: New test suite for init command model download functionality
  - Tests for valid HuggingFace URLs, model dimensions, naming conventions
  - CLI flag acceptance tests (`--auto`, `--non-interactive`, `--force`)
- **Content-type-aware search profiles**: New `CorpusProfile` enum and auto-detection
  - `CODE`: Boosts symbol/path search for source code repositories (60%+ code files)
  - `PROSE`: Boosts FTS5/vector search for text-heavy corpora (60%+ docs)
  - `DOCS`: Balanced weights for mixed code/documentation
  - `MIXED`: Default balanced weights for heterogeneous corpora
  - `SearchEngineConfig::detectProfile()`: Auto-detects from file extension distribution
  - `SearchEngineConfig::forProfile()`: Returns preset weights for a profile
- **Session-isolated memory**: Documents can now be isolated to working sessions
  - New CLI commands: `yams session create`, `open`, `close`, `status`, `merge`, `discard`
  - Documents added during an active session are tagged with `session_id` metadata
  - Session documents are invisible to global searches (use `--global` to bypass)
  - `merge`: Removes session tag to promote documents to global index
  - `discard`: Permanently deletes all session documents
  - Supports multiple concurrent sessions with automatic isolation
  - Database migration adds session tracking to metadata repository
- **Windows Job Object for plugin processes**: External plugin child process cleanup
  - Plugin processes are now assigned to Windows Job Objects
  - All child processes are automatically terminated when plugin unloads
  - Prevents orphaned processes from holding file locks (e.g., PID files)
  - Uses `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` for reliable cleanup
  - Location: `src/extraction/plugin_process.cpp`
- **Plugin health command**: New `yams plugin health [name]` subcommand for plugin diagnostics
  - Shows plugin status, interfaces, models loaded, and error state
  - Displays model provider FSM state (Idle, Loading, Ready, Degraded, Failed)
  - Lists all loaded models when provider is ready
  - JSON output support with `--json` flag
  - Location: `src/cli/commands/plugin_command.cpp`
- **Plugin info improvements**: Enhanced `yams plugin info` output
  - Now uses `StatusResponse.providers` for accurate plugin status
  - Shows plugin type (native/external), interfaces, and path
  - Properly handles both ABI and external plugin hosts

### Changed
- **Embedding model list**: Both recommended models now have 384 dimensions
  - `all-MiniLM-L6-v2`: Lightweight general-purpose semantic search (default)
  - `multi-qa-MiniLM-L6-cos-v1`: Optimized for question-answer semantic search
- **ServiceManager Decomposition**: Extracted focused components from monolithic ServiceManager
  - New `ConfigResolver`: Static config/env resolution utilities (248 lines)
  - New `VectorSystemManager`: Vector DB and index lifecycle (397 lines)
  - New `DatabaseManager`: Metadata DB, connection pool, KG store lifecycle (254 lines)
  - New `PluginManager`: Plugin host, loader, and interface adoption (515 lines)
  - ServiceManager accessors now delegate to extracted managers
- **Configurable Vector DB Capacity**: Vector index `max_elements` now configurable
  - Environment variable: `YAMS_VECTOR_MAX_ELEMENTS`
  - Config file: `[vector_database] max_elements`
  - Default: 100,000 (range: 1,000 - 10,000,000)
- **FTS5 index hygiene (migration v18)**: Removed unused `content_type` column from FTS5 index
  - `content_type` was indexed but never queried via FTS MATCH
  - Content type filtering uses JOIN on `documents.mime_type` instead
  - Reduces FTS5 index size and improves indexing performance
  - Automatic migration rebuilds index on first database open
- **Daemon socket logging noise reduction**: Request/mux/enqueue/drain logs now emit at debug level
  - Default info-level daemon logs no longer show per-request socket traffic
  - Enable debug logging to inspect connection-level request handling details
- **SearchEngine Consolidation**: Unified search architecture by removing legacy HybridSearchEngine
  - **SearchEngine** is now the sole search engine, consolidating multi-component search (FTS5, PathTree, Symbol, KG, Vector, Tag, Metadata)
  - Removed ~2000 lines of legacy code: `hybrid_search_engine.cpp`, `hybrid_search_factory.cpp`, and associated headers
  - **Parallel Execution**: SearchEngine now uses `std::async` to execute all 7 component queries simultaneously
    - Configurable via `SearchEngineConfig::enableParallelExecution` (default: true)
    - Per-component timeout via `SearchEngineConfig::componentTimeout` (default: 100ms)
    - Graceful degradation: timed-out components are skipped, others continue
  - **Updated Interfaces**: `AppContext.searchEngine` replaces `AppContext.hybridEngine` across CLI, daemon, and services
  - **SearchEngineBuilder**: Simplified to create `SearchEngine` directly (removed `MetadataKeywordAdapter` and KG scorer wiring)
  - Removed unused benchmark executables: `engine_comparison_bench`, `hybrid_search_bench`
  - Location: `src/search/`, `include/yams/search/`, `src/app/services/`, `src/cli/`
- **HotzoneManager Persistence**: Added save/load functionality for hotzone state
  - `HotzoneManager::save(path)`: Serializes hotzone entries to JSON with atomic write (temp + rename)
  - `HotzoneManager::load(path)`: Restores persisted hotzone state on startup
  - Stores version, half-life config, and timestamped entry scores
  - Location: `src/search/hotzone_manager.cpp`, `include/yams/search/hotzone_manager.h`
- **CheckpointManager Component**: New daemon component for periodic state persistence
  - Manages vector index and hotzone checkpoint scheduling
  - Configurable interval, threshold-based vector index saves, optional hotzone persistence
  - Async timer-based loop with graceful shutdown support
- **Post-ingest pipeline parallelization**: PostIngestQueue and EntityGraphService now use WorkCoordinator
  - Removed serial strand-based processing bottleneck in PostIngestQueue
  - EntityGraphService now posts extraction jobs to shared WorkCoordinator thread pool
  - Removed unused PoolManager "post_ingest" pool and associated TuningManager tuning logic
  - Documents process in parallel across all worker threads with work stealing
- **Graph BFS traversal optimization**: Reduced N+1 query patterns in graph traversal
  - New `getEdgesBidirectional()` API: returns incoming + outgoing edges in single query (UNION)
  - New `getNodesByIds()` API: batch node retrieval for hydration
  - Edge cache in BFS: edges fetched during neighbor collection reused for connecting edges
  - Reduces per-node queries from 4 (2×getEdgesFrom + 2×getEdgesTo) to 1
  - Location: `src/app/services/graph_query_service.cpp`, `src/metadata/knowledge_graph_store_sqlite.cpp`
- **Graph command cleanup**: Removed unused `--reverse` flag
  - Bidirectional traversal is now the default behavior
  - Flag was redundant since BFS optimization returns all connected edges
  - Location: `src/cli/commands/graph_command.cpp`

### Fixed
- **JavaScript/TypeScript symbol extraction**: Audited and fixed against Tree-sitter grammars
  - JavaScript: Added `function_expression`, `generator_function`, `generator_function_declaration`, `namespace_import`, `export_statement`, `export_specifier`
  - TypeScript: Added `abstract_class_declaration`, `abstract_method_signature`, `function_expression`, `generator_function`, `import_alias`
  - Added queries for function expressions, generators, abstract methods, export statements
- **Graph `--name` query now shows symbol relationships**: Fixed `yams graph --name <file>` showing "Graph data unavailable"
  - Now resolves filename to file node key and uses KG query path
  - Shows connected symbols, includes, and document nodes
  - Falls back to document-based lookup if file node not found
  - Location: `src/cli/commands/graph_command.cpp`
- **KG queue metric now shows pending count**: Fixed `kg(q=N)` showing cumulative total instead of pending items
  - Now calculates: `pending = queued - consumed - inflight`
  - Affects `yams status -v` and `yams daemon status -d` displays
  - Location: `src/cli/commands/status_command.cpp`, `src/cli/commands/daemon_command.cpp`
- **Symbol extraction extension mapping**: Fixed extension lookup not matching due to leading dot mismatch
  - Database stores extensions with dots (`.cpp`), map keys without (`cpp`)
  - PostIngestQueue now strips leading dot before lookup
  - Location: `src/daemon/components/PostIngestQueue.cpp`
- **Graph query bidirectional traversal**: Fixed graph queries showing 0 connections for blob nodes
  - BFS traversal now follows both incoming and outgoing edges by default
  - Blob nodes (which only have incoming `has_version` edges from path nodes) now return connected nodes
  - Refactored dispatcher to delegate to GraphQueryService (single responsibility)
- **Repair tracking (migration v21)**: Added repair status tracking to prevent duplicate work
  - New `repair_status` column (pending, processing, completed, failed, skipped)
  - `repair_attempted_at` timestamp and `repair_attempts` counter
  - RepairCoordinator filters by status to avoid re-queuing processed documents
- **Plugin interface parsing**: Fixed object-format interfaces not parsing correctly
- **Plugin host sharing**: Fixed model provider adoption failure after component extraction
- **VectorIndexManager initialization**: Fixed "VectorIndexManager not provided" search engine build failure
- **Model download mapping**: Added `multi-qa-MiniLM-L6-cos-v1` to HuggingFace repo mapping
- **Version display**: Fixed `yams --version` showing fallback values
- **Socket crash on shutdown**: Fixed `EXC_BAD_ACCESS` in kqueue_reactor during program exit
- **Windows daemon status metrics**: CPU and memory now report accurate values
- **`--name` flag for `yams add`**: Fixed custom document naming for single-file adds
- **External plugin extractors**: Fixed content extractors from external plugins not being used
- **Trust file persistence**: Fixed plugin trust file being deleted on daemon restart
- **Trust file comment parsing**: Fixed daemon crash when loading trust file with comments
- **Plugin trust initialization order**: Fixed plugins not loading despite being trusted
- **Post-ingestion pipeline reliability**: Improved async processing consistency
- **Graph IPC serialization**: Added missing ProtoBinding specializations for GraphQueryRequest/Response
- **Status command document count**: Fixed `yams status` showing `docs=0` after daemon restart
  - Short status now uses `documents_total` (from metadata DB, initialized on startup)
  - Previously used `storage_documents` (CAS object count, which was 0 on fresh start)
  - Detailed status was unaffected as it already used the correct field
  - Location: `src/cli/commands/status_command.cpp`

### CLI Improvements
- **PowerShell completion**: Added `yams completion powershell` for PowerShell auto-complete
- **Consistent `--json` output**: Extended JSON output support across commands
- **Actionable error hints**: Centralized error hint system with pattern-based hints
- **Daemon error messages**: Enhanced daemon start/stop failure messages with recovery hints

### Removed
- **HybridSearchEngine**: Legacy search engine removed in favor of unified SearchEngine
  - Deleted: `src/search/hybrid_search_engine.cpp` (~1844 lines)
  - Deleted: `src/search/hybrid_search_factory.cpp` (~168 lines)
  - Deleted: `include/yams/search/hybrid_search_engine.h`
  - Deleted: `include/yams/search/hybrid_search_factory.h`
- **HybridSearchEngine Tests**: Removed obsolete test files
  - `tests/unit/search/hybrid_search_engine_test.cpp`
  - `tests/unit/search/hybrid_grouping_smoke_test.cpp`
  - `tests/unit/search/learned_fusion_smoke_test.cpp`
  - `tests/unit/search/hierarchical_search_test.cpp`
  - `tests/unit/metadata/search_metadata_interface_test.cpp`
- **Legacy Adapters**: Removed `MetadataKeywordAdapter` (was bridge for HybridSearchEngine)
- **CLI Adapter Rename**: `HybridSearchResultAdapter` → `SearchResultItemAdapter` in result_renderer.h