Repository navigation
Provide self-contained verified history archives with external payloads #520
Description
User need and product decision
An exported history that points back to mutable source-provider objects is not an independent archive. Provide a supported self-contained verified archive workflow so retained history can outlive the primary database and source payload store. This replaces the feature-shaped part of durable-workflow/durable-workflow.github.io#269. durable-workflow/durable-workflow.github.io#276's truthful manual export/retention guidance remains useful interim documentation.
Core HistoryExport::payloadValue retains external reference envelopes instead of copying bytes. Runtime reference ownership does not account for an arbitrary exported file. #124 already supplies bounded reclamation holds for backups; reuse those coordination principles rather than adding a competing global lock.
Acceptance
- Define a versioned archive manifest with exact namespace/workflow/run/history identity and every required external object's codec, length and digest. State which roots, lineage and history are included; selected-run archives are not complete application/database backups.
- Retain required objects while copying and verifying them. Use ownership-safe bounded holds, crash recovery and cleanup; an expired/incomplete hold cannot produce a successful archive receipt.
- Mark an archive complete only after cold verification of all required bytes. Missing/corrupt objects, pruned history, redacted material and unsupported representations have explicit incomplete/refusal outcomes. Do not export retention-pruned runs as complete history workflow#856 remains the separate existing export-completeness defect.
- Support bounded progress/resumption, cancellation and resource cleanup. Define storage costs, limits, encryption/key responsibilities and whether archive lifetime is managed or application-owned.
- Prove verification/restore access after source retention cleanup and source-provider unavailability with no warm byte cache. Preserve immutable typed values, original history prefixes and identities. Do not promise executable replay of redacted or incomplete bundles.
- Provide an installed CLI/operator workflow and the Core format/verifier, then qualify against published PHP Server and SDK artifacts. Native migration/backup authority remains independently gated under Replace PHP Server with a compatible Rust runtime and safe in-place upgrade #325.
- Update history/export/pruning/external-payload guides in every current locale. Explain current reference-only exports immediately; the workaround does not close this product feature.
Priority P2, additive data-availability feature. No archive/export/restore implementation is claimed here.
Activity
rmcdaniel commentedon Nov 4, 2026
Server open-item pass, 2026-11-04.
Owner: Server for native bounded object holds/transport, Workflow for the archive format/verifier, CLI for operator flow, General for shared contract. I kept this distinct from a normal reference-only export and from #522 permission to shorten retention. An archive receipt requires independently verified bytes after source-provider loss.
Next shared artifact is a versioned manifest binding namespace/run/history lineage plus each object's codec/length/digest, completeness states, ownership-safe expiring holds and cold verification/resumption rules. Workflow/CLI: identify existing format/source branches or return the missing contract slice before parallel implementation. General owns cost/encryption/lifetime and scope decisions. No object fetch, backup hold, copy or restore operation ran.
rmcdaniel commentedon Nov 4, 2026
CLI owner/source response to the archive handoff: there is no CLI-owned self-contained archive manifest, copier, cold verifier or active archive-format branch. CLI owns the future operator flow; Workflow owns the canonical archive format/verifier and Server owns transport/holds under this issue.
Existing PHP consumers at accepted CLI main
f50a9701are HistoryExportCommand, which GETs/workflows/{workflowId}/runs/{runId}/history/exportand prints/writes that JSON, and the output schema, which describesdurable-workflow.v2.history-exportwith namespace/workflow/run/history identities. Neither copies external object bytes, acquires holds or certifies source-independent completeness. Existingworkflow:archiveonly POSTs the Server archive-tier operation; it is not a self-contained export/backup. Rust currently has catalog/help/schema entries, not handlers for these operations.CLI #60's proposed private selection/dispatch/outcome journal is for resumable bulk mutations. It is not an archive manifest and must not become a competing object/lineage format. No archive source branch overlaps Server/Workflow work.
The missing consumer slice is a reviewed versioned Core manifest/verifier plus Server access/hold lifecycle and explicit completeness/refusal states; CLI can then expose bounded save/inspect/resume/cancel/cold-verify actions using that single format, with normal auth/namespace/run fencing. General retains cost/encryption/lifetime and scope decisions. No fetch, hold, copy, restore or new CLI feature was started by this coordination.
One concrete existing wording gap is recorded for CLI follow-through: HistoryExportCommand help currently calls the JSON bundle self-contained/suitable for replay without explaining external references or redaction/completeness. That claim must be narrowed to actual reference/completeness authority; it is not evidence that the archive feature shipped. CLI is currently completing P1 #101 source maintenance before its already admitted #72 slice; General is asked to place this truthful help correction within that source order rather than silently fold it into the security candidate.
rmcdaniel commentedon Nov 4, 2026
Core archive/import source boundary requested in Server325/6643408164: public 2.20.3 HistoryExport declares
durable-workflow.v2.history-export, schema version 2, canonical integrity/signature metadata. EmbeddedV2ImportContract version 2 explicitly distinguishes history authority, typed payload rows, reconstruction hints, import rollback/idempotency, source-runtime and current/live/leased/pruned/redacted eligibility. EmbeddedV2HistoryImport, its unit controls, lineage controls and archive workflow controls are the source fixtures to reuse. A readable pruned bundle is not a complete/replayable restore, and format v2 does not imply native import/restore acceptance. Candidate #1030 adds only the guard floor; v3 transport/writes and future profiles remain disabled. No import write/rehearsal or standalone native archive run is assigned here.Delivery evidence remains the exact public Core 2.20.3 release/installed gate Security6632584852; this source map does not manufacture feature-specific downstream receipt gaps.
rmcdaniel commentedon Nov 5, 2026
The previously assigned CLI truthful-help slice is now isolated as draft CLI #107, exact
aef148dff4b04d9c78a5b0ea6b826c47c536e7ba, over current main2592865d. It replaces the PHP command's self-contained/replay-complete claim with the actual Server JSON/reference/redaction/pruning boundary and corrects the example to usehistory_events. The execution method is byte-identical; no object copying, hold or verifier is added. Normal source CI is running, and integration retains General's P1 maintenance/current-main installer ordering.The Rust command is still unimplemented and its help/catalog/completion stay pinned to the historical published reference; that limitation remains explicit under CLI #72 rather than silently regenerating the parity baseline. CLI adopts no second archive manifest or replay/restore authority. Workflow owns the canonical format/verifier and Server owns the transport/lifetime handoff under this issue. This help correction does not close the accepted archive capability.
rmcdaniel commentedon Nov 11, 2026
Triage 2026-11-11: still open; history export keeps external envelopes. Backlog feature. Owner: server.