trevon/yams
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
yams/CHANGELOG.md

21 KiB
Raw Permalink Blame History

Changelog

All notable changes to YAMS (Yet Another Memory System) will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Archived Changelogs

  • v0.7.x archive: docs/changelogs/v0.7.md
  • v0.6.x archive: docs/changelogs/v0.6.md
  • v0.5.x archive: docs/changelogs/v0.5.md
  • v0.4.x archive: docs/changelogs/v0.4.md
  • v0.3.x archive: docs/changelogs/v0.3.md
  • v0.2.x archive: docs/changelogs/v0.2.md
  • v0.1.x archive: docs/changelogs/v0.1.md

[v0.7.10] - Unreleased

Added

  • Graph command --list-types flag (yams-66h): Node type discovery for knowledge graph
    • New --list-types flag shows all distinct node types with counts
    • Table output with TYPE and COUNT columns, ordered by count descending
    • JSON output with nodeTypes array containing type and count fields
    • Added getNodeTypeCounts() method to KnowledgeGraphStore interface
    • Extended GraphQueryRequest IPC protocol with listTypes mode
    • Usage hint when no nodes found: suggests yams add <path>
    • Location: src/cli/commands/graph_command.cpp, include/yams/metadata/knowledge_graph_store.h
  • P4 language support for symbol extraction: Network data plane language (P4_16)
    • Node types: headerTypeDeclaration, structTypeDeclaration, controlDeclaration, parserDeclaration, actionDeclaration, tableDeclaration
    • Query patterns for actions, functions, headers, structs, controls, parsers, tables, typedefs
    • Aliases: p4, p4_16, p4lang
    • Grammar auto-download from prona-p4-learning-platform/tree-sitter-p4
  • Vector diagnostics in DaemonMetrics: Moved collect_vector_diag to background polling
    • Added vectorEmbeddingsAvailable, vectorScoringEnabled, searchEngineBuildReason to MetricsSnapshot
    • Status requests now read from cached snapshot (non-blocking)
    • Resolves status command hangs when vector services are slow
  • Entity extraction metrics in status output: Added entity queue/inflight counters
    • New metrics: entityQueued, entityDropped, entityConsumed, entityInFlight
    • Exposed via yams status and yams status -v output
    • JSON output includes entity_queued, entity_consumed, entity_dropped, entity_inflight
    • Location: include/yams/daemon/components/DaemonMetrics.h, src/cli/commands/status_command.cpp
  • Gitignore support for directory ingestion: Skip files matching .gitignore patterns
    • New --no-gitignore flag for yams add command to disable gitignore filtering
    • Default behavior respects .gitignore patterns in the root directory
    • Supports standard gitignore patterns: wildcards, directory patterns, anchored paths
    • Location: src/cli/commands/add_command.cpp, src/app/services/indexing_service.cpp

Changed

  • Constexpr language configuration for symbol extraction: Centralized compile-time configuration
    • 17 languages with constexpr node types and query patterns: C, C++, Python, Rust, Go, Java, JavaScript, TypeScript, C#, PHP, Kotlin, Perl, R, SQL, Solidity, Dart, P4
    • LanguageConfig struct with class_types, field_types, function_types, import_types, identifier_types
    • Query patterns: function_queries, class_queries, import_queries, call_queries
    • Language alias support (e.g., "cpp" → "c++", "cxx", "cc")
    • getLanguageConfig() constexpr lookup function
    • Location: plugins/symbol_extractor_treesitter/symbol_extractor.cpp
  • Field extraction: New extractFields() method extracts class member variables
    • Uses node type traversal with language-specific field types
    • Creates field kind symbols with proper byte ranges
  • Member containment relations: New extractMemberRelations() method
    • Creates contains edges from classes to their methods/fields
    • Uses byte range containment to determine class membership
    • Improves knowledge graph structure for code navigation
  • PostIngestQueue per-stage metrics: Exposed extraction/KG/symbol stage inflight counts
    • New getters: extractionInFlight(), kgInFlight(), symbolInFlight(), totalInFlight()
    • Static constexpr limits: maxExtractionConcurrent(), maxKgConcurrent(), maxSymbolConcurrent()
    • Exposed via daemon status: extraction_inflight, kg_inflight, symbol_inflight
    • yams status shows POST line when there's active work
    • yams status -v shows per-stage breakdown
    • yams daemon status -d shows full Post-Ingest Pipeline section
    • JSON output includes stages object with per-stage counts
    • Location: include/yams/daemon/components/PostIngestQueue.h, src/cli/commands/status_command.cpp, src/cli/commands/daemon_command.cpp
  • PostIngestQueue dynamic concurrency scaling (PBI-05a): Auto-scale based on queue depth
    • New TuneAdvisor tunables: postExtractionConcurrent(), postKgConcurrent(), postSymbolConcurrent(), postEntityConcurrent()
    • Dynamic limits replace static constexpr values in PostIngestQueue pollers
    • TuningManager scales concurrency based on queue depth thresholds:
      • >1000 queued: extraction=hwThreads/2, kg=hwThreads/2
      • >500 queued: extraction=hwThreads/4, kg=32
      • >100 queued: extraction=hwThreads/8+4, kg=16
      • >10 queued: extraction=8
      • idle: extraction=4 (default)
    • Status output shows limits: stages: extract=4/4, kg(q=0/i=0/8), symbol=0/4
    • JSON includes extraction_limit, kg_limit, symbol_limit, entity_limit
    • Location: include/yams/daemon/components/TuneAdvisor.h, src/daemon/components/TuningManager.cpp, src/daemon/components/DaemonMetrics.cpp
  • Knowledge Graph cleanup on document deletion: Deleting documents now cascades to KG
    • deleteNodesForDocumentHash(): Removes doc:<hash> nodes and symbol nodes with matching document_hash
    • Integrated into document deletion flow for automatic cleanup
    • Location: include/yams/metadata/knowledge_graph_store.h, src/app/services/document_service.cpp
  • Stale edge cleanup on re-indexing: Symbol extraction now cleans up old relationships
    • deleteEdgesForSourceFile(): Removes edges where properties.source_file matches path
    • Called automatically before re-extraction to prevent stale relationship accumulation
    • Location: src/daemon/components/EntityGraphService.cpp
  • Optimized isolated node query: yams graph --isolated now uses single SQL query
    • findIsolatedNodes(): Efficient NOT EXISTS subquery instead of N+1 pattern
    • New IPC fields: isolatedMode, isolatedRelation in GraphQueryRequest
    • Significant performance improvement for large graphs
    • Location: src/cli/commands/graph_command.cpp, src/daemon/components/dispatcher/request_dispatcher_graph.cpp
  • Daemon log command: Added yams daemon log
  • ExternalPluginHost: New plugin host for Python/process-based plugins (RFC-EPH-001)
    • Implements IPluginHost interface for external plugins running as separate processes
    • JSON-RPC 2.0 communication over stdio using existing PluginProcess and JsonRpcClient
    • Supported plugin types: Python (.py), Node.js (.js), any executable with JSON-RPC support
    • Process lifecycle management: spawn, monitor, health checks, graceful shutdown
    • Automatic crash recovery with configurable restart policy (max retries, backoff)
    • Trust-based security model with persistent trust file
    • RPC gateway for calling arbitrary plugin methods (callRpc)
    • Plugin statistics tracking (uptime, restart count, health status)
    • State change callbacks for monitoring plugin lifecycle events
    • Location: include/yams/daemon/resource/external_plugin_host.h, src/daemon/resource/external_plugin_host.cpp
  • Auto-init mode: New yams init --auto flag for containerized/headless environments
    • Enables vector database with default model (all-MiniLM-L6-v2)
    • Enables plugins directory setup
    • Generates authentication keys
    • Skips S3 configuration (uses local storage)
    • Non-interactive: no prompts, uses sensible defaults
  • Tree-sitter grammar download: yams init now offers to download tree-sitter grammars
    • Interactive menu: recommended (C, C++, Python, JS, TS, Rust, Go), all, or custom selection
    • Auto-downloads and builds grammars from official GitHub repos
    • Supports 14 languages: C, C++, Python, JavaScript, TypeScript, Rust, Go, Java, C#, PHP, Kotlin, Dart, SQL, Solidity
    • Cross-platform: MSVC, MinGW, GCC, Clang compilation support
    • Grammar prompt also available when YAMS is already initialized
    • Grammars installed to XDG_DATA_HOME/yams/grammars (Unix) or %LOCALAPPDATA%\yams\grammars (Windows)
  • New embedding model option: Added multi-qa-MiniLM-L6-cos-v1 as second model choice
    • Trained on 215M question-answer pairs for semantic search optimization
    • Same dimensions (384) as default model for compatibility
    • Replaces all-mpnet-base-v2 (768 dim) in model selection
  • Git-based version detection: Build system now auto-detects version from git tags
    • Uses most recent semver tag (v*) as effective version
    • Falls back to project version only if no tags exist
    • Command-line override (-Dyams-version=X.Y.Z) takes highest priority
  • Commit hash in version output: yams --version now shows short commit hash
    • Format: 0.7.9 (commit: c16939f) built:2025-11-29T17:30:15Z
    • Helps identify exact build for bug reports and debugging
  • Init command tests: New test suite for init command model download functionality
    • Tests for valid HuggingFace URLs, model dimensions, naming conventions
    • CLI flag acceptance tests (--auto, --non-interactive, --force)
  • Content-type-aware search profiles: New CorpusProfile enum and auto-detection
    • CODE: Boosts symbol/path search for source code repositories (60%+ code files)
    • PROSE: Boosts FTS5/vector search for text-heavy corpora (60%+ docs)
    • DOCS: Balanced weights for mixed code/documentation
    • MIXED: Default balanced weights for heterogeneous corpora
    • SearchEngineConfig::detectProfile(): Auto-detects from file extension distribution
    • SearchEngineConfig::forProfile(): Returns preset weights for a profile
  • Session-isolated memory: Documents can now be isolated to working sessions
    • New CLI commands: yams session create, open, close, status, merge, discard
    • Documents added during an active session are tagged with session_id metadata
    • Session documents are invisible to global searches (use --global to bypass)
    • merge: Removes session tag to promote documents to global index
    • discard: Permanently deletes all session documents
    • Supports multiple concurrent sessions with automatic isolation
    • Database migration adds session tracking to metadata repository
  • Windows Job Object for plugin processes: External plugin child process cleanup
    • Plugin processes are now assigned to Windows Job Objects
    • All child processes are automatically terminated when plugin unloads
    • Prevents orphaned processes from holding file locks (e.g., PID files)
    • Uses JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE for reliable cleanup
    • Location: src/extraction/plugin_process.cpp
  • Plugin health command: New yams plugin health [name] subcommand for plugin diagnostics
    • Shows plugin status, interfaces, models loaded, and error state
    • Displays model provider FSM state (Idle, Loading, Ready, Degraded, Failed)
    • Lists all loaded models when provider is ready
    • JSON output support with --json flag
    • Location: src/cli/commands/plugin_command.cpp
  • Plugin info improvements: Enhanced yams plugin info output
    • Now uses StatusResponse.providers for accurate plugin status
    • Shows plugin type (native/external), interfaces, and path
    • Properly handles both ABI and external plugin hosts

Changed

  • Embedding model list: Both recommended models now have 384 dimensions
    • all-MiniLM-L6-v2: Lightweight general-purpose semantic search (default)
    • multi-qa-MiniLM-L6-cos-v1: Optimized for question-answer semantic search
  • ServiceManager Decomposition: Extracted focused components from monolithic ServiceManager
    • New ConfigResolver: Static config/env resolution utilities (248 lines)
    • New VectorSystemManager: Vector DB and index lifecycle (397 lines)
    • New DatabaseManager: Metadata DB, connection pool, KG store lifecycle (254 lines)
    • New PluginManager: Plugin host, loader, and interface adoption (515 lines)
    • ServiceManager accessors now delegate to extracted managers
  • Configurable Vector DB Capacity: Vector index max_elements now configurable
    • Environment variable: YAMS_VECTOR_MAX_ELEMENTS
    • Config file: [vector_database] max_elements
    • Default: 100,000 (range: 1,000 - 10,000,000)
  • FTS5 index hygiene (migration v18): Removed unused content_type column from FTS5 index
    • content_type was indexed but never queried via FTS MATCH
    • Content type filtering uses JOIN on documents.mime_type instead
    • Reduces FTS5 index size and improves indexing performance
    • Automatic migration rebuilds index on first database open
  • Daemon socket logging noise reduction: Request/mux/enqueue/drain logs now emit at debug level
    • Default info-level daemon logs no longer show per-request socket traffic
    • Enable debug logging to inspect connection-level request handling details
  • SearchEngine Consolidation: Unified search architecture by removing legacy HybridSearchEngine
    • SearchEngine is now the sole search engine, consolidating multi-component search (FTS5, PathTree, Symbol, KG, Vector, Tag, Metadata)
    • Removed ~2000 lines of legacy code: hybrid_search_engine.cpp, hybrid_search_factory.cpp, and associated headers
    • Parallel Execution: SearchEngine now uses std::async to execute all 7 component queries simultaneously
      • Configurable via SearchEngineConfig::enableParallelExecution (default: true)
      • Per-component timeout via SearchEngineConfig::componentTimeout (default: 100ms)
      • Graceful degradation: timed-out components are skipped, others continue
    • Updated Interfaces: AppContext.searchEngine replaces AppContext.hybridEngine across CLI, daemon, and services
    • SearchEngineBuilder: Simplified to create SearchEngine directly (removed MetadataKeywordAdapter and KG scorer wiring)
    • Removed unused benchmark executables: engine_comparison_bench, hybrid_search_bench
    • Location: src/search/, include/yams/search/, src/app/services/, src/cli/
  • HotzoneManager Persistence: Added save/load functionality for hotzone state
    • HotzoneManager::save(path): Serializes hotzone entries to JSON with atomic write (temp + rename)
    • HotzoneManager::load(path): Restores persisted hotzone state on startup
    • Stores version, half-life config, and timestamped entry scores
    • Location: src/search/hotzone_manager.cpp, include/yams/search/hotzone_manager.h
  • CheckpointManager Component: New daemon component for periodic state persistence
    • Manages vector index and hotzone checkpoint scheduling
    • Configurable interval, threshold-based vector index saves, optional hotzone persistence
    • Async timer-based loop with graceful shutdown support
  • Post-ingest pipeline parallelization: PostIngestQueue and EntityGraphService now use WorkCoordinator
    • Removed serial strand-based processing bottleneck in PostIngestQueue
    • EntityGraphService now posts extraction jobs to shared WorkCoordinator thread pool
    • Removed unused PoolManager "post_ingest" pool and associated TuningManager tuning logic
    • Documents process in parallel across all worker threads with work stealing
  • Graph BFS traversal optimization: Reduced N+1 query patterns in graph traversal
    • New getEdgesBidirectional() API: returns incoming + outgoing edges in single query (UNION)
    • New getNodesByIds() API: batch node retrieval for hydration
    • Edge cache in BFS: edges fetched during neighbor collection reused for connecting edges
    • Reduces per-node queries from 4 (2×getEdgesFrom + 2×getEdgesTo) to 1
    • Location: src/app/services/graph_query_service.cpp, src/metadata/knowledge_graph_store_sqlite.cpp
  • Graph command cleanup: Removed unused --reverse flag
    • Bidirectional traversal is now the default behavior
    • Flag was redundant since BFS optimization returns all connected edges
    • Location: src/cli/commands/graph_command.cpp

Fixed

  • JavaScript/TypeScript symbol extraction: Audited and fixed against Tree-sitter grammars
    • JavaScript: Added function_expression, generator_function, generator_function_declaration, namespace_import, export_statement, export_specifier
    • TypeScript: Added abstract_class_declaration, abstract_method_signature, function_expression, generator_function, import_alias
    • Added queries for function expressions, generators, abstract methods, export statements
  • Graph --name query now shows symbol relationships: Fixed yams graph --name <file> showing "Graph data unavailable"
    • Now resolves filename to file node key and uses KG query path
    • Shows connected symbols, includes, and document nodes
    • Falls back to document-based lookup if file node not found
    • Location: src/cli/commands/graph_command.cpp
  • KG queue metric now shows pending count: Fixed kg(q=N) showing cumulative total instead of pending items
    • Now calculates: pending = queued - consumed - inflight
    • Affects yams status -v and yams daemon status -d displays
    • Location: src/cli/commands/status_command.cpp, src/cli/commands/daemon_command.cpp
  • Symbol extraction extension mapping: Fixed extension lookup not matching due to leading dot mismatch
    • Database stores extensions with dots (.cpp), map keys without (cpp)
    • PostIngestQueue now strips leading dot before lookup
    • Location: src/daemon/components/PostIngestQueue.cpp
  • Graph query bidirectional traversal: Fixed graph queries showing 0 connections for blob nodes
    • BFS traversal now follows both incoming and outgoing edges by default
    • Blob nodes (which only have incoming has_version edges from path nodes) now return connected nodes
    • Refactored dispatcher to delegate to GraphQueryService (single responsibility)
  • Repair tracking (migration v21): Added repair status tracking to prevent duplicate work
    • New repair_status column (pending, processing, completed, failed, skipped)
    • repair_attempted_at timestamp and repair_attempts counter
    • RepairCoordinator filters by status to avoid re-queuing processed documents
  • Plugin interface parsing: Fixed object-format interfaces not parsing correctly
  • Plugin host sharing: Fixed model provider adoption failure after component extraction
  • VectorIndexManager initialization: Fixed "VectorIndexManager not provided" search engine build failure
  • Model download mapping: Added multi-qa-MiniLM-L6-cos-v1 to HuggingFace repo mapping
  • Version display: Fixed yams --version showing fallback values
  • Socket crash on shutdown: Fixed EXC_BAD_ACCESS in kqueue_reactor during program exit
  • Windows daemon status metrics: CPU and memory now report accurate values
  • --name flag for yams add: Fixed custom document naming for single-file adds
  • External plugin extractors: Fixed content extractors from external plugins not being used
  • Trust file persistence: Fixed plugin trust file being deleted on daemon restart
  • Trust file comment parsing: Fixed daemon crash when loading trust file with comments
  • Plugin trust initialization order: Fixed plugins not loading despite being trusted
  • Post-ingestion pipeline reliability: Improved async processing consistency
  • Graph IPC serialization: Added missing ProtoBinding specializations for GraphQueryRequest/Response
  • Status command document count: Fixed yams status showing docs=0 after daemon restart
    • Short status now uses documents_total (from metadata DB, initialized on startup)
    • Previously used storage_documents (CAS object count, which was 0 on fresh start)
    • Detailed status was unaffected as it already used the correct field
    • Location: src/cli/commands/status_command.cpp

CLI Improvements

  • PowerShell completion: Added yams completion powershell for PowerShell auto-complete
  • Consistent --json output: Extended JSON output support across commands
  • Actionable error hints: Centralized error hint system with pattern-based hints
  • Daemon error messages: Enhanced daemon start/stop failure messages with recovery hints

Removed

  • HybridSearchEngine: Legacy search engine removed in favor of unified SearchEngine
    • Deleted: src/search/hybrid_search_engine.cpp (~1844 lines)
    • Deleted: src/search/hybrid_search_factory.cpp (~168 lines)
    • Deleted: include/yams/search/hybrid_search_engine.h
    • Deleted: include/yams/search/hybrid_search_factory.h
  • HybridSearchEngine Tests: Removed obsolete test files
    • tests/unit/search/hybrid_search_engine_test.cpp
    • tests/unit/search/hybrid_grouping_smoke_test.cpp
    • tests/unit/search/learned_fusion_smoke_test.cpp
    • tests/unit/search/hierarchical_search_test.cpp
    • tests/unit/metadata/search_metadata_interface_test.cpp
  • Legacy Adapters: Removed MetadataKeywordAdapter (was bridge for HybridSearchEngine)
  • CLI Adapter Rename: HybridSearchResultAdapter → SearchResultItemAdapter in result_renderer.h