Trevon e27d5e3e84

fix: Copilot review feedback and gitignore for artifacts (#26 )

* feat: search work and moving to experimental

* WIP

* feat(search): diagnostic logging pipeline — graph build, concepts, reranker gate

- New [concepts] per-50-query log showing extracted query concepts, type
  distribution, and used_gliner flag.

- New [PIQ-entity] per-1000-dispatch + per-document completion log with
  entity type counts (e.g., 'protein=3, gene=1, chemical=2').

- New [graph_rerank-gate] one-shot log showing why enableGraphRerank
  is zeroed: shows shortQueryBudgeted, corpusWarming, bypassWarmingGate,
  lexicalReady, and post-ingest inflight.

- New [graph_rerank] per-50-query log with window size, matched
  candidates, applied count, concept count, and elapsed time.

- retrieval_quality_bench.cpp: add YAMS_SEARCH_BYPASS_CORPUS_WARMING_GATE=1
  so benchmark queries are not gated by freshness checks.

- resolve merge conflicts from experimental branch.

* fix(search): enable entity extraction for text files via GLiNER fallback

- prepareMetadataEntry: shouldDispatchEntity now also checks hasTitleExtractor()
  so plain-text files (.txt, .md) get entity extraction even when no binary
  entity provider supports their extension.

- processEntityExtractionStage: when no ExternalEntityProvider matches,
  fall back to processTitleExtractionStage which runs GLiNER on text.

- refreshStageAvailability: entity stage is now considered capable when
  hasTitleExtractor() is true, so the entity poller gets concurrency budget
  even if entityProviders_ is empty.

- kCombinedEntityTypes: add biomedical entity types (protein, gene, cell,
  disease, chemical, drug, pathway, biological_process, biomarker, anatomy,
  organism) so GLiNER extracts high-value biomedical entities instead of
  only generic NL types (person, organization, concept).

* perf(bench): add YAMS_POST_ENTITY_CONCURRENT=4 for text entity fallback throughput

* feat(search): fast token-based entity extraction for text documents

- Replace slow GLiNER-based entity extraction (~260ms/doc) with fast
  token-based concept mining (~1ms/doc) via generateFallbackQueryConcepts.
  Uses sub-phrase mining + IDF-weighted token salience — the same function
  already used for query-time concept fallback.

- Store entities with full KG scaffolding: KGNode with entity type/label,
  KGAlias for query-time resolution, DeferredEdge from entity→document,
  and DeferredDocEntity linking entity to document for getDocEntitiesForDocument.

- Add biomedical entity types to kCombinedEntityTypes in title extraction
  (protein, gene, cell, disease, chemical, drug, pathway, etc.) so GLiNER
  can output high-value biomedical entities when used.

- Add include for query_expansion.h to access generateFallbackQueryConcepts.

- bench: add YAMS_POST_ENTITY_CONCURRENT=4 for entity poller throughput.

Benchmark: 22K+ entities extracted across 5183 docs, ~41% high-value
biomedical. MRR 0.4281 (beats 0.4273 baseline by +0.2%).

* feat(diag): add firstDocEntities check to graph_rerank-gate log

- Queries kg_doc_entities for the first candidate document to verify
  whether fast entity writes were committed before queries start.
- Confirmed: firstDocEntities=0 — entities stuck in WriteCoordinator
  async queue when benchmark queries begin.

* fix(search): add periodic write-coordinator flush during entity extraction

Entity writes enqueued via writeCoordinator_ can accumulate since the
writer loop is capped at 8 batches/iteration. For 5K+ documents this
creates a backlog that outlasts the post-ingest drain wait — entities
are not committed to kg_doc_entities before benchmark queries start.

Add a periodic flush (every 128 batches, 500ms timeout) in the entity
extraction batch processor to drain the WC queue incrementally.

Diagnostic confirms firstDocEntities=0 persists — a synchronous
committed flush or batched WC enqueue is needed.

* fix(docker): add gfortran for openblas arm64 build

* fix(search): accumulate entity writes into shared WC batch, flush once

The WriteCoordinator writer loop is capped at 8 batches/iteration. With
5K+ per-doc entity writes, entities outlast the benchmark's post-ingest
drain and firstDocEntities=0 when the graph scorer runs.

- processEntityExtractionStage: add optional DeferredKGBatch* parameter.
  Text-fallback appends entity nodes/aliases/edges/doc-entities to the
  shared batch instead of creating per-doc WC batches.

- processEntityExtractionBatch: creates one DeferredKGBatch, passes it to
  all stage calls in the batch, enqueues to WC once at end.

- Single-shot WC flush: after all stages go idle (totalInFlight==0),
  calls writeCoordinator_->flush(15s) to drain remaining entity writes
  before benchmark queries start.

- DeferredKGBatch forward-declared in PostIngestQueue header.

* ci: fix free-disk-space action ref, sha didnt exist

* fix(search): revert WC accumulation, fix text fallback to per-doc writes

The shared-batch WC accumulation approach worked conceptually but:
- WC flush with totalInFlight==0 fires too early (before last batch)
- generateFallbackQueryConcepts produces empty concepts for 99% of
  document texts (designed for short queries, not abstracts)
- Only 1/5183 docs produced concepts → 1 doc worth of entities in KG

Revert to simple per-doc entity extraction. The enable=/disable=
graph reranker gate is still controlled by shortQueryBudgeted=1
which zeroes the graph reranker on the first query (intent=Code).

firstDocEntities=0 persists — entities need a different creation
strategy for document text (phrase mining too conservative).

* fix(docker): add curl retry for Zig download

* feat(search): per-profile ConceptExtractionBackend (Fallback vs GlinerWithFallback)

- search_engine_config.h: new ConceptExtractionBackend enum with Fallback
  (sub-phrase mining + IDF tokens, <1ms) and GlinerWithFallback (GLiNER
  first, fall back on empty/timeout). Default: GlinerWithFallback.

- search_tuner.h: SCIENTIFIC profile sets Fallback (GLiNER returns empty
  for 99% of short biomedical claims). Other profiles keep default.
  Wired through TunedParams.applyTo().

- search_engine.cpp: gate GLiNER concept extraction on the config backend.
  When Fallback, conceptFuture is never created and concepts come directly
  from generateFallbackQueryConcepts. Added [concept-backend] diagnostic.

- search_engine_builder.cpp: builder log now includes concept_backend
  in the override-state startup message.

- search_tuner_catch2_test.cpp: verify SCIENTIFIC profile uses Fallback.

Note: builder correctly sets Fallback but query-time workingConfig still
shows GlinerWithFallback (int=1). This is a config-sync issue between
builder-time profile application and runtime config snapshot — the
tuner or a post-override config copy may reset the field. To be fixed.

* refactor(search): extract concept resolver, per-profile backend, clean dead code

- New concept_resolver.h/cpp: extract enrichWithFallbackConcepts from the
  search engine monolith into a testable unit. Generates IDF-weighted
  token concepts via generateFallbackQueryConcepts, deduplicates, and
  appends to existing concepts.

- search_engine_config.h: new ConceptExtractionBackend enum (Fallback vs
  GlinerWithFallback). SCIENTIFIC profile sets Fallback since GLiNER
  extracts 0 concepts for 99% of short biomedical claims.

- search_tuner.h: wire conceptExtractionBackend through TunedParams,
  applyTo, seedTunedParamsFromConfig (the one-line fix that resolved
  the config-sync bug where profile Fallback was overridden at query
  policy resolution time).

- search_engine.cpp: materializeConcepts lambda restructured to always
  reach the fallback even when GLiNER future is invalid. Removed
  ~80 lines of diagnostic bloat ([concept-backend], [workingConfig-*],
  per-50 [concepts]). GLiNER gate checks config backend.

- PostIngestQueue.cpp: removed dead fast-token entity extraction
  (~150 lines of concept generation + KG scaffolding that produced
  0 entities for 99% of documents). Reverted entity-stage gating
  (entityCapable || hasTitleExtractor) and shouldDispatchEntity to
  extension-only checks. Removed DeferredKGBatch forward decl.

- gliner_query_extractor: removed unused createGlinerExtractionFunc-
  FromProvider fallback factory (~150 lines).

- ServiceManager.cpp: removed fallback call site for removed factory.

- builder log: removed concept_backend diagnostic line noise.

- search_tuner_catch2_test.cpp: verify SCIENTIFIC uses Fallback.

Net: ~400 lines removed, ~120 lines added (concept_resolver unit +
config wiring). GLiNER concept extraction savings: ~30-80ms/query
× 300 queries = 9-24s total for SCIENTIFIC profiles.

* fix: entity stage only for binaries, title stage for text

entity stage = binary providers (Ghidra/Zyp). title stage = GLiNER for all
text. No overlap — title already stores NL entities to KG.

* fix(docker): add wget for Zig, disable faiss

openblas arm64 build fails in Docker containers. YAMS_DISABLE_FAISS=1
env var skips faiss (and its openblas transitive dep). Zig download
now uses wget -c for resume on flaky ziglang.org connections.

* ci: add experimental branch to tests trigger

* ci: pin free-disk-space to v1.3.1

* feat: Simeon concept mining entity export for graph reranker

- simeon_lexical_backend.h: EntityCallback in Config. Called with
  (docId, entityText, confidence) per concept match per doc.

- simeon_lexical_backend.cpp: after mine_concepts, iterate docs
  via collect_doc_concepts + entity_callback.

- SearchEngineManager.cpp: enable concept_mining_enabled, wire
  entity_callback batch-inserting DocEntity via KG store. Fix
  kgStore capture-before-move race.

Note: simeon submodule has collect_doc_concepts in concept_mining.hpp
+ concept_mining.cpp (must commit to submodule separately).

* ci: disable faiss in warm and test jobs

* ci(bench): add faiss disable, ccache, and failure diagnostics to weekly benchmarks

* fix(metadata): retry constraint failures in executeQueryOnPool

FK race on Linux causes transient constraint failures when metadata
write races ahead of document commit. Previously treated as terminal.
Now retried with backoff like lock errors.

* fix(metadata): deduplicate batch entries before UPSERT

Duplicate (document_id, key) pairs within a single batch cause SQLite
ON CONFLICT DO UPDATE to fail with constraint violation. Dedup keeps
the last write per key.

* chore: bump simeon, add extraction readiness stats

Simeon: 8bf225e -> 6320ab6 (adapter + QPP router framework).
CorpusStats: add contentExtracted*, ftsIndexed*, titleCoverage,
languageCoverage, and KG topology richness fields.

* feat(search): integrate simeon EntropyRouter strategy lane, corpus-state tuning

- Wire EntropyRouter + Keyphrase/LeadField strategies into SimeonLexicalBackend
- Add scoreStrategyRouted(), hasStrategyRouter(), strategy_router_enabled config
- Add YAMS_SIMEON_STRATEGY_ROUTER_ENABLED env wiring
- Add applyCorpusStateAdjustments(): per-corpus-state recall push (vec max,
  rescue slots, weak-query fanout, entity vec expansion, NER damping)
- Extend CorpusStats with extraction/FTS/title/language/KG edge+alias density
- Add hasRichGraphTopology(), nerOnlyNoEdges damping to graph adjustments
- Add 14 new engine-policy benchmark candidates (retrieval lanes, RRF sweep,
  aggressive recall, strategy router)
- Weight miss_pre_fusion_rate in optimization objective (0.25x)
- Add Simeon lexical concept/doc count to IndexFreshnessSnapshot + status
- Bump simeon submodule (retrieval_strategy + corpus_adapter API)

* feat(search): add concept_count and concept_mining_enabled accessors

* feat(search): add ScientificAdapter + RM3 PRF blend

ScientificAdapter: training-free entity extraction for scientific
text via regex/suffix patterns (biomedical suffixes, ALL_CAPS symbols,
capitalized multi-word terms, measurements). Replaces TextAdapter
in strategy router build, providing entity lists for KeyphraseStrategy.

RM3: add score_with_prf blend path to scoreStrategyRouted at 0.30
weight when rm3_enabled=true. New benchmark candidate
scientific_strategy_rm3.

Strategy router + ScientificAdapter: mrr=0.4514 ndcg=0.4789 recall=0.5849

* docs: soften governance and contribution prose

Replace prescriptive language ('must', 'require', 'adjudicates') with
friendlier, more welcoming phrasing across governance docs, PR template,
security policy, and contributing guide.

* feat(extract): add language-agnostic IMRAD section detection, expand entity types

detectDocumentSections(): structural heuristics for title/abstract/body
using line-length, ALL_CAPS, numbering patterns. No word matching.
Works across all languages and scripts.

deriveTitle(): add IMRAD detection step before first-line fallback.
isHighValueGraphType(): use ScientificAdapter suffix detection (40+
biomedical suffixes) instead of hardcoded 11 types.

* feat(ingest): field-weighted FTS5, IMRAD claim segmentation, ScientificAdapter fallback

- FTS5: title tokens repeated 3x, abstract 2x, body 1x in content
  field for BM25 field weighting without schema migration
- IMRAD-aware claim segmentation: buildBodyClaimSegments splits
  each detected section independently, distributes maxSegments
- ScientificAdapter fallback: when GLiNER unavailable, inline
  entity extraction via suffix/pattern detection → direct KG write
- Add abstract field to PreparedMetadataEntry + BatchContentEntry

* refactor(daemon): extract enrichment stages to post_ingest_enrichment.cpp

Split PostIngestQueue.cpp (3145 lines) into:
- PostIngestQueue.cpp (~3084 lines): coordinator, lifecycle, queue
  management, extraction pipeline, gradient limiters
- post_ingest_enrichment.cpp (new, 701 lines): KG/symbol/entity/title
  stage processing with self-contained enrichment utilities

* fix(daemon): restore enrichment utilities to PostIngestQueue.cpp

Processing methods still in PostIngestQueue.cpp reference entity/NL
utilities (isUsefulNlEntity, buildBodyClaimSegments, etc.) that were
removed during decomposition. Restore them to anonymous namespace.

* fix: address PR #25 Copilot review feedback

- Add missing #include <unordered_map> to concept_resolver.cpp
- Batch entity writes per doc in SearchEngineManager (reduces write amp)
- Remove non-existent CorpusAdapter API claims from README
- Replace absolute paths with repo-relative in daemon-download-jobs.md
- Revert plugin-zyp default to false (noisy warnings)
- Fix grammar in extraction-repair-pipeline.md heading
- Add gitignore entries for .wrangler, .claude, .artifacts, papers

* fix: Copilot feedback - IDF passthrough, batch flush, debug gate, IMRAD comment

- Pass IDF map through enrichWithFallbackConcepts for weighted fallback
- Batch entity writes per doc with threshold flush in SearchEngineManager
- Downgrade graph_rerank-gate diagnostic to spdlog::debug
- Add TODO for entity_callback wiring during concept mining
- Fix DocumentSections comment to match language-agnostic implementation

* chore: bump simeon (ScientificAdapter v3)

* fix: resolve merge conflict markers, add missing <random> include

Remove leftover <<<<<<< / ======= / >>>>>>> from earlier merge.
Add #include <random> to ingestion_throughput_bench.cpp.

* chore: trigger CI after simeon submodule push

2026-05-08 05:29:46 -04:00

7.8 KiB

Raw Permalink Blame History

Daemon Download Jobs

Goal

Expose download tracking and resume to daemon clients without turning the daemon into a general purpose network fetch proxy.

The intended users are agents and automation flows that archive external artifacts into YAMS CAS and need:

durable job identity
progress/status visibility
resumable downloads across daemon restarts
optional post-indexing into the existing content pipeline

The design explicitly does not start with arbitrary remote download capability.

Current State

Implemented Now

The downloader subsystem already supports:
- probe + range GET
- ETag-based resume
- staging file reopen and partial range persistence
- CAS finalization
- optional export
The current resume store is local-manager scoped and keyed by URL in src/downloader/download_manager.cpp.
The daemon exposes a guarded download path in src/daemon/components/dispatcher/request_dispatcher_documents.cpp behind YAMS_ENABLE_DAEMON_DOWNLOAD.
The daemon now enforces the narrow policy boundary before network activity:
- feature gate enabled
- allowed scheme
- allowed host
- checksum requirement
- store-only enforcement
- checksum format validation
Accepted daemon downloads are backgrounded as tracked jobs with:
- jobId
- persisted job metadata
- queued/running/completed/failed/canceled state transitions
- status polling and list APIs
- cooperative cancel signaling
The CLI exposes daemon job controls in src/cli/commands/download_command.cpp:
- yams download <url>
- yams download --status <job-id>
- yams download --list-jobs
- yams download --cancel <job-id>
MCP exposes the same read/control surface in src/mcp/mcp_server.cpp:
- download_status
- download_list_jobs
- download_cancel

Still Missing

true job-oriented start/resume request types distinct from the compatibility DownloadRequest
maxFileBytes enforcement
rateLimitRps enforcement
sandbox/process isolation for daemon downloads
resume-by-jobId semantics stronger than the downloader's URL-based resume store
richer ownership/quota controls

Problem

If the daemon directly exposes the current downloader request shape, the daemon becomes a network boundary with new risk:

SSRF and internal host reachability
credential or custom-header abuse
disk exhaustion via large or repeated downloads
long-lived network tasks consuming daemon capacity
cross-agent state collision when resume state is keyed only by URL
arbitrary filesystem writes if export behavior leaks through

That is too much surface for a first daemonized version.

Design Principles

Keep the daemon surface narrower than the local CLI/MCP surface.
Treat downloads as managed jobs, not one-shot unary fetches.
Enforce policy before any network activity.
Keep first version store-only and CAS-rooted.
Use durable daemon job IDs, not URL identity, for tracking and resume.
Reuse the downloader manager internally rather than reimplement transport.

Target Model

Public Daemon API

The code currently uses a compatibility DownloadRequest plus separate status/list/cancel calls. The intended end state is a fully job-oriented API instead of expanding the unary request forever.

StartDownloadJobRequest
GetDownloadJobStatusRequest
CancelDownloadJobRequest
ResumeDownloadJobRequest
ListDownloadJobsRequest

Matching responses/events:

StartDownloadJobResponse
DownloadJobStatusResponse
CancelDownloadJobResponse
ResumeDownloadJobResponse
ListDownloadJobsResponse
optional DownloadJobEvent

Job States

queued
probing
downloading
verifying
finalizing
indexing
completed
failed
cancelled
blocked_policy

Job Record

Persist a daemon-owned record for each job:

download_id
owner or caller identity
canonical URL
request fingerprint
policy decision snapshot
created/updated timestamps
bytes downloaded / total bytes
attempt count
ETag / Last-Modified
CAS hash when completed
last error
indexing outcome

The downloader resume store can remain an implementation detail, but daemon logic should key work by download_id.

Request Boundary

First Version Allowed Inputs

url
optional tags / metadata
optional collection / snapshot indexing metadata

First Version Rejected Inputs

custom headers
proxy
TLS insecure / custom CA
export path / export dir
overwrite controls
arbitrary retry and transport tuning

Those can remain local CLI or MCP-only until the daemon has a stronger policy and audit model.

Policy Enforcement

Use the existing DaemonConfig::DownloadPolicy in include/yams/daemon/daemon.h, but enforce it in the handler before invoking the download service.

Implemented checks today:

feature gate must be enabled
allowed scheme check
allowed host check
checksum requirement check
store-only hard enforcement

Not implemented yet:

max file bytes enforcement
rate limit enforcement
sandbox strategy selection

Recommended defaults:

enable=false
allowedSchemes=["https"]
requireChecksum=true
storeOnly=true
no export

Resume Identity

The current downloader resume identity is effectively URL + ETag. That is not sufficient for multi-agent daemon use.

Daemon job fingerprint should include:

canonical URL
effective caller identity
checksum requirement / checksum value
headers/auth fingerprint if those are ever supported
store-only/export mode

Resume should only continue if:

job fingerprint matches
server validators are still compatible
policy still allows the request

Execution Model

Implemented Baseline

daemon-owned background job runner
progress/status polling via status requests
durable job metadata persisted by daemon
cooperative cancellation support

Remaining Phases

stronger bounded concurrency / quotas
resume across daemon restart by job identity, not only downloader-local URL resume
optional event streaming
richer policy
controlled auth/header support
agent/job quotas

Security Boundaries

The daemon download path should be treated as its own trust boundary.

Threats to design against:

SSRF to loopback, link-local, RFC1918, metadata endpoints
open redirect escaping host allowlists
oversized or unbounded bodies
validator confusion on resumed jobs
multi-tenant job collision
content-type deception and indexing abuse
progress/event amplification

Out of scope for first version:

arbitrary authenticated internet fetches
user-specified export to filesystem paths
browser-equivalent cookie/session behavior

Integration with Existing Services

Use the existing downloader stack:

IDownloadManager
IResumeStore
IDiskWriter
IHttpAdapter

Use existing indexing flow after successful CAS finalize:

add document metadata
apply tags and metadata
optional collection/snapshot placement

Rollout Plan

Completed:

Enforce current download policy in the existing daemon handler.
Add job record type and persistence.
Add start/status/list job API.
Add cancel.

Remaining:

Move the current handler to a compatibility wrapper or deprecate it.
Add resume-by-job.
Enforce maxFileBytes, rate limits, and sandboxing.
Only then consider broadening request shape.

Open Questions

What daemon caller identity is available for job ownership and quotas?
Should checksum be mandatory for all daemon downloads or only for non-allowlisted hosts?
Should MCP use the daemon job path later, or remain a separate local download path?
Where should durable download-job state live: metadata DB, dedicated table, or sidecar state?

7.8 KiB Raw Permalink Blame History