yams/docs/design/daemon-download-jobs.md

Daemon Download Jobs

Goal

Expose download tracking and resume to daemon clients without turning the daemon into a general purpose network fetch proxy.

The intended users are agents and automation flows that archive external artifacts into YAMS CAS and need:

• durable job identity
• progress/status visibility
• resumable downloads across daemon restarts
• optional post-indexing into the existing content pipeline

The design explicitly does not start with arbitrary remote download capability.

Current State

Implemented Now

• The downloader subsystem already supports:
  • probe + range GET
  • ETag-based resume
  • staging file reopen and partial range persistence
  • CAS finalization
  • optional export
• The current resume store is local-manager scoped and keyed by URL in src/downloader/download_manager.cpp.
• The daemon exposes a guarded download path in src/daemon/components/dispatcher/request_dispatcher_documents.cpp behind YAMS_ENABLE_DAEMON_DOWNLOAD.
• The daemon now enforces the narrow policy boundary before network activity:
  • feature gate enabled
  • allowed scheme
  • allowed host
  • checksum requirement
  • store-only enforcement
  • checksum format validation
• Accepted daemon downloads are backgrounded as tracked jobs with:
  • jobId
  • persisted job metadata
  • queued/running/completed/failed/canceled state transitions
  • status polling and list APIs
  • cooperative cancel signaling
• The CLI exposes daemon job controls in src/cli/commands/download_command.cpp:
  • yams download <url>
  • yams download --status <job-id>
  • yams download --list-jobs
  • yams download --cancel <job-id>
• MCP exposes the same read/control surface in src/mcp/mcp_server.cpp:
  • download_jobs with action=status|list|cancel

Still Missing

• true job-oriented start/resume request types distinct from the compatibility DownloadRequest
• maxFileBytes enforcement
• rateLimitRps enforcement
• sandbox/process isolation for daemon downloads
• resume-by-jobId semantics stronger than the downloader's URL-based resume store
• richer ownership/quota controls

Problem

If the daemon directly exposes the current downloader request shape, the daemon becomes a network boundary with new risk:

• SSRF and internal host reachability
• credential or custom-header abuse
• disk exhaustion via large or repeated downloads
• long-lived network tasks consuming daemon capacity
• cross-agent state collision when resume state is keyed only by URL
• arbitrary filesystem writes if export behavior leaks through

That is too much surface for a first daemonized version.

Design Principles

1. Keep the daemon surface narrower than the local CLI/MCP surface.
2. Treat downloads as managed jobs, not one-shot unary fetches.
3. Enforce policy before any network activity.
4. Keep first version store-only and CAS-rooted.
5. Use durable daemon job IDs, not URL identity, for tracking and resume.
6. Reuse the downloader manager internally rather than reimplement transport.

Target Model

Public Daemon API

The code currently uses a compatibility DownloadRequest plus separate status/list/cancel calls. The intended end state is a fully job-oriented API instead of expanding the unary request forever.

• StartDownloadJobRequest
• GetDownloadJobStatusRequest
• CancelDownloadJobRequest
• ResumeDownloadJobRequest
• ListDownloadJobsRequest

Matching responses/events:

• StartDownloadJobResponse
• DownloadJobStatusResponse
• CancelDownloadJobResponse
• ResumeDownloadJobResponse
• ListDownloadJobsResponse
• optional DownloadJobEvent

Job States

• queued
• probing
• downloading
• verifying
• finalizing
• indexing
• completed
• failed
• cancelled
• blocked_policy

Job Record

Persist a daemon-owned record for each job:

• download_id
• owner or caller identity
• canonical URL
• request fingerprint
• policy decision snapshot
• created/updated timestamps
• bytes downloaded / total bytes
• attempt count
• ETag / Last-Modified
• CAS hash when completed
• last error
• indexing outcome

The downloader resume store can remain an implementation detail, but daemon logic should key work by download_id.

Request Boundary

First Version Allowed Inputs

• url
• optional tags / metadata
• optional collection / snapshot indexing metadata

First Version Rejected Inputs

• custom headers
• proxy
• TLS insecure / custom CA
• export path / export dir
• overwrite controls
• arbitrary retry and transport tuning

Those can remain local CLI or MCP-only until the daemon has a stronger policy and audit model.

Policy Enforcement

Use the existing DaemonConfig::DownloadPolicy in include/yams/daemon/daemon.h, but enforce it in the handler before invoking the download service.

Implemented checks today:

• feature gate must be enabled
• allowed scheme check
• allowed host check
• checksum requirement check
• store-only hard enforcement

Not implemented yet:

• max file bytes enforcement
• rate limit enforcement
• sandbox strategy selection

Recommended defaults:

• enable=false
• allowedSchemes=["https"]
• requireChecksum=true
• storeOnly=true
• no export

Resume Identity

The current downloader resume identity is effectively URL + ETag. That is not sufficient for multi-agent daemon use.

Daemon job fingerprint should include:

• canonical URL
• effective caller identity
• checksum requirement / checksum value
• headers/auth fingerprint if those are ever supported
• store-only/export mode

Resume should only continue if:

• job fingerprint matches
• server validators are still compatible
• policy still allows the request

Execution Model

Implemented Baseline

• daemon-owned background job runner
• progress/status polling via status requests
• durable job metadata persisted by daemon
• cooperative cancellation support

Remaining Phases

• stronger bounded concurrency / quotas
• resume across daemon restart by job identity, not only downloader-local URL resume
• optional event streaming
• richer policy
• controlled auth/header support
• agent/job quotas

Security Boundaries

The daemon download path should be treated as its own trust boundary.

Threats to design against:

• SSRF to loopback, link-local, RFC1918, metadata endpoints
• open redirect escaping host allowlists
• oversized or unbounded bodies
• validator confusion on resumed jobs
• multi-tenant job collision
• content-type deception and indexing abuse
• progress/event amplification

Out of scope for first version:

• arbitrary authenticated internet fetches
• user-specified export to filesystem paths
• browser-equivalent cookie/session behavior

Integration with Existing Services

Use the existing downloader stack:

• IDownloadManager
• IResumeStore
• IDiskWriter
• IHttpAdapter

Use existing indexing flow after successful CAS finalize:

• add document metadata
• apply tags and metadata
• optional collection/snapshot placement

Rollout Plan

Completed:

1. Enforce current download policy in the existing daemon handler.
2. Add job record type and persistence.
3. Add start/status/list job API.
4. Add cancel.

Remaining:

1. Move the current handler to a compatibility wrapper or deprecate it.
2. Add resume-by-job.
3. Enforce maxFileBytes, rate limits, and sandboxing.
4. Only then consider broadening request shape.

Open Questions

• What daemon caller identity is available for job ownership and quotas?
• Should checksum be mandatory for all daemon downloads or only for non-allowlisted hosts?
• Should MCP use the daemon job path later, or remain a separate local download path?
• Where should durable download-job state live: metadata DB, dedicated table, or sidecar state?