Skip to content

feat(phylake): add the custody store with B1–B5 transactions and recovery - #88

Merged
forkwright merged 51 commits into
mainfrom
feat/phylake-custody
Sep 25, 2026
Merged

forkwright merged 51 commits into
mainfrom
feat/phylake-custody

Conversation

@forkwright

@forkwright forkwright commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

What changed

crates/phylake/src/store is the D5 custody store, specified by docs/design/custody-store.md. It runs on one fjall 3.1 SingleWriterTxDatabase with 15 keyspaces. Every write transaction sets PersistMode::SyncAll, and the single-writer lock serializes the state checks.

  • Opening the store. The store holds a plaintext meta record: format, schema version, active root key id, KDF salt and key check. Open validates those fields before any write and fails closed:

    • SchemaTooNew
    • MigrationRequired (schema v1 is the first, so migrate arrives with the first schema change)
    • StoreLocked

    A new store directory is created with mode 0700. The fjall lock allows one open store at a time.

  • Records. Records are rkyv structs owned by phylake, read back through validated access. Each value is sealed with an AAD that binds schema version, record kind, key id, keyspace and record key.

    • Store-sealed: directory and lifecycle records.
    • Tenant-sealed: blobs, side records, session index, idempotency bindings and per-tenant audit.

    Every record key built from tenant, session or idempotency components is a keyed HMAC, and tenant-scoped keys include the tenant. The target URL is never persisted.

  • Invocation lifecycle. Each step is one transaction, checked against epitrope's next_state, and writes its terminal audit entry in the same transaction.

    • B1: runs the idempotency check first. The store binds the caller's digest to the grant, capability, session, target and cost. It then authorizes with epitrope, debits the reservation from every ledger in the plan, and persists the intent.
    • B2: dispatch.
    • B3: writes the blob at a tenant-keyed address, plus a pending side record that is not yet visible.
    • B4: the atomic publish point. It writes the side record, locator and session index. The artifact id is the invocation id.
    • B5: settles, releases or marks UnknownEffect, and records a typed Terminal (Settled{failure} / Released{reason} / UnknownEffect).
    • A refused call writes only audit, and a dry-run writes nothing.
  • Recovery. recover() applies epitrope's recovery_action through the same transitions: B1 becomes Released(Abandoned), B2 becomes UnknownEffect (charged the whole reservation), and B3 and B4 roll forward. Running it again changes nothing.

  • Directory. Tenant registration (with a freshly wrapped data key), a root grant, grant issue and revoke (checked by epitrope inside the transaction and audited either way), and session create and fork with lineage. Each write is idempotent by a caller-chosen id.

  • Reads. Chunked artifact reads and per-session listings. Missing and unpublished artifacts look the same. audit_query implements the D17.7 scopes (All, OwnAndOwnedSessions), with a session filter and paging. StoreSnapshot implements epitrope's Snapshot and the wire server's TenantDirectory.

  • Failpoints. A Failpoint trait with before/after commit hooks for B1 through B5.

  • Documentation. custody-store.md now records:

    • the sealing-scope table, and what a crypto-shred leaves readable: ids, amounts, lineage, states and timestamps, but no acquired content, idempotency binding or per-tenant audit;
    • the physical blob key;
    • the artifacts record kinds;
    • the tenant key cache;
    • audit reads.

    UnknownEffect is now described as charging the whole reservation in both the custody and contract docs.

Why

Phase 01 S2's custody clauses require three things. Every invocation authorizes, reserves atomically and persists intent before any external effect. Every outcome settles or releases exactly once. A crash at any durability boundary leaves an explicit state that can be recovered, and replay never repeats an external action.

Dependencies

Crate Why
fjall 3.1 (default features off) The D5 engine under D17.14's STORAGE-TIERS migration exception. It is pure Rust, with atomic multi-keyspace transactions and SyncAll durability.
epitrope, syntheke (path) Authorization and the contract types.
rkyv Record encoding, the same validated format family as the wire.

deny.toml adds:

  • Licenses: 0BSD (varint-rs) and BSL-1.0 (xxhash-rust).
  • Skips: hashbrown 0.14 and 0.16. rkyv's alloc needs 0.17, while fjall's map dependency and lsm-tree's quick_cache pin older versions. Each skip has a retirement condition.

Stage and acceptance

Phase 01 S2. This PR proves the following acceptance clauses at the store boundary:

  • "crash/reopen at each durability boundary";
  • "replay of same invocation";
  • "dry-run no effects";
  • "cross-tenant read and metadata denial";
  • "every outcome settles or releases once".

Proof (toolchain 1.97.1)

  • Workspace: 416 tests pass (phylake has 145). They cover:
    • Every row of the failure-injection table: a crash before or after each Bn commit, then reopen and recover. Each row asserts the exact state, the ledger values, exactly one terminal audit entry, visibility, and that the producer call counter is unchanged. A second recovery leaves a byte-identical logical dump.
    • The exact Released{reason} for every release path.
    • Concurrency: 8 threads released together by a barrier against a ceiling of 3. Exactly 3 succeed, and every ledger shows exactly 3.
    • Idempotency: a replay performs no writes. The same digest under a different grant, target or cost returns Conflict.
    • A raw scan of the whole fjall directory finds zero hits for the envelope marker, URL, host, text view, idempotency key, envelope SHA-256, session id, schema id, or tenant bytes and ULID text. A positive control (the plaintext format tag) is found.
    • A record moved under another key fails to open.
    • The audit query scopes.
  • clippy -D warnings, cargo deny check, cargo tree -d (documented skips only), rustdoc with -D warnings, fmt and the doc checks are clean.
  • An independent adversarial review fixed these defects:
    • release reasons were recorded wrongly;
    • recovery could stall permanently on a colliding artifact id;
    • the idempotency binding was incomplete;
    • the request digest would have stayed readable after a crypto-shred;
    • repeating a grant issue was not idempotent;
    • B2 settle accepted any failure kind.

Rekey, crypto-shred and backup follow in the next slice.

Phase 01 S2 workspace: syntheke, epitrope, phylake, dioptron (bin +
lib), and xenos (publish = false). Shared lints follow kanon RUST.md;
toolchain pinned to 1.97.1 and rust-version 1.97, the toolchain CI
builds. No external dependencies yet. The daemon binary exits with
failure until commands land, and a process test pins that behavior.
deny.toml follows the zetesis posture (multiple-versions and wildcards
denied, unknown registries and git sources denied) and scans the
all-features graph the gate builds. The license allowlist covers the
workspace license and the licenses of the planned Phase 01
dependencies. .cargo/audit.toml and osv-scanner.toml are derived from
the deny.toml advisory ignore list.
The hybrid gate slots now run cargo fmt, check, clippy, nextest, and
doctests with --all-features; the toolchain comes from
rust-toolchain.toml. The corpus checks move to a docs job that runs on
every PR and on main. The docs-only exemption is on, which is sound
only while branch protection requires docs next to gate / gate.
.kanon-ci.toml gains the Rust stages ahead of the doc stages.
security.yml calls the fleet reusable (cargo-deny, cargo-audit,
osv-scanner) on PRs, main, and daily. CodeQL gains a Rust job and now
runs on PRs so it can be a required check. Dependabot also watches
Cargo dependencies.
release-please keeps release-type simple and updates the workspace
version, versioned path dependencies, and local Cargo.lock entries
through extra-files, as zetesis does.
The operator decision of 2026-09-25 starts the implementation phase.
AGENTS.md scopes crates to the Phase 01 plan; CLAUDE.md, README.md,
llms.txt, and _llm describe the workspace and the live build gate.
README.md's license section now matches LICENSE and LICENSE-DOCS.
Allow only the workspace's own license in deny.toml until dependencies
arrive, drop the private-plan S1 reference from the syntheke crate
docs, remove an unrecorded decision name from AGENTS.md, and drop a
copied comment with a nonstandard tag from the CodeQL rust job.
Implement the encryption-at-rest layer of the custody store design:

- keyfile: 32-byte root key load that refuses a missing file, a
  non-regular file, any group/other permission bit, or a wrong length;
  generation via O_CREAT|O_EXCL with mode 0600 from getrandom. Key held
  in secrecy::SecretBox.
- crypto: HKDF-SHA256 store subkeys (dioptron/v1/{blob,meta,audit,
  index,kek,check}) under a per-store salt; key check
  HMAC(k_check, "dioptron-key-check") verified in constant time, with
  StoreLocked on mismatch; random per-tenant data keys wrapped by the
  kek with XChaCha20-Poly1305, AAD binding kek id, data key id, and
  tenant id; tenant subkeys (blob, blob-addr, meta, audit, index).
- Sealed value format rec_ver u16 | key_id u32 | nonce[24] | ct|tag,
  AAD binding label, schema version, record kind, key id, and
  u16-length-prefixed keyspace and record key. Random 24-byte nonces.
- Tenant-keyed blob address (HMAC-SHA256) and plain SHA-256 provenance
  digest for sealed metadata only.

Dependencies stay on one RustCrypto generation (digest 0.11,
crypto-common 0.2): chacha20poly1305 0.11, hkdf 0.13, hmac 0.13,
sha2 0.11, plus getrandom 0.4, secrecy 0.10, zeroize 1, snafu 0.9;
tempfile 3 for tests. deny.toml allows MIT, Apache-2.0, Unicode-3.0.
rkyv 0.8 with bytecheck is the wire format fixed by topology.md; the
format features are named so a conflicting layout fails the build.
snafu is the fleet error library. toml is a dev-dependency for the
contract fixture test. deny.toml allows MIT, Apache-2.0, and
Unicode-3.0 for these crates and skips the syn 2 copy that snafu-derive
and munge_macro still build with.
Identifiers with ULID text form, the capability, mode, scope, and
budget vocabulary, request and reply payloads for every capability,
the outcome taxonomy, the frame header codec, handshake frames, the
authentication transcript, and a decoder that bounds length before
copying and validates the archive before any field is read.
Every message type round-trips through a full frame; corrupted and
truncated bodies are rejected. The fixture test maps each file under
docs/contract/fixtures onto syntheke types and fails when the
directory or a declared fixture is missing.
Open the root key with O_NOFOLLOW | O_NONBLOCK | O_CLOEXEC | O_NOCTTY
through rustix and check the opened descriptor: a symbolic link at the
key path, a FIFO, socket, or device node, or a file whose owner is not
the effective uid is refused before any read. Key generation also
creates with O_NOFOLLOW. rustix is already in the lock file through
tempfile, so no new crate version enters the tree.

External errors (io, getrandom) move to an `error` source field per the
kanon snafu convention, and a shrinking file reports its current length.
Add the draft-irtf-cfrg-xchacha-03 A.3.1 AEAD vector and byte-exact
seal and wrap vectors computed with an independent reference. Wipe the
HKDF PRK returned to the crate, record the stack residue the RustCrypto
primitives leave as an accepted WARNING, and restate the plaintext bound
in terms of what bounds a stored value.
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/phylake/Cargo.toml
#	crates/phylake/src/lib.rs
#	deny.toml
Request gains a grant field naming the one grant a call acts under, in
execute and dry-run mode. GrantIssueRequest drops parent_grant: the
designated grant is the parent, so the two can no longer disagree.
Fault frames now refuse any failure other than ProtocolError and
AuthFailed on encode and decode, and VersionChoice is non-exhaustive.

Fixtures carry grant as a wire field on every capability request; the
cast gives tnt0f grant grn0f, and neg_foreign_grant covers a request
naming another tenant's grant. The fixture conformance test splits into
load, mapping, and test modules.
Every capability request names the grant it acts under; the daemon
authorizes against that grant's chain only. A missing or foreign
designated grant returns the same NotFoundOrDenied, and the idempotency
digest covers the grant. The wire section now lists the eight frame
kinds with their bytes, restricts Fault to connection-level failures,
fixes the auth transcript layout, and states time units. Fixture keys
move grant to the wire fields. The contract stays at version 1.
Names the crates on each side of the syn 2/syn 3 split, the one
alternative (holding rkyv at 0.8.17) and why it is rejected, and the
retirement condition. The license list takes the same entries and order
as the storage branch so the later merge touches only comments.
Inline #[cfg(test)] modules move to sibling tests.rs files, and
test-only helpers (fixed entropy, shared key fixtures, from_bytes
constructors) move to test_support.rs files, so path-scoped security
scanning can exclude test code. Test names and count are unchanged.
Add the pure authorization crate over the syntheke contract types:

- narrowing of a child grant against the designated parent on every
  axis, reporting the first failing NarrowingAxis;
- chain validity at an injected clock against revocation records, so an
  ancestor's revocation invalidates descendants without fan-out;
- designated-grant authorization where a grant the connection's tenant
  does not hold reads exactly as a missing one;
- reservation and settlement arithmetic with checked ledgers and a typed
  overrun;
- the invocation transition table and restart recovery actions;
- a dry-run planner over a read-only snapshot trait;
- D17.7 audit scope defaults and a rule view with no audit accessor;
- a minimal origin parser for target scopes that refuses every
  construct where parsers could disagree about the host.

proptest (dev only, no default features) checks that a child passing
the narrowing check is never broader than its parent on any axis.
Add .github/codeql/codeql-config.yml with paths-ignore for tests.rs,
tests/ and test_support.rs, reference it from the Rust init step, and
record the test layout convention in CLAUDE.md.
…hylake-custody

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	deny.toml
Add DenyCode::BudgetUnavailable for exhaustion on a ledger the caller
does not own; it names no dimension. Add NarrowingAxis::AuditScope.
Failure::Denied carries axis: Option<NarrowingAxis>, set exactly for
NarrowingViolation; Response::check refuses any other combination in a
reply or a plan's refusal (DeniedAxisMismatch). Move the outcome unit
tests to outcome/tests.rs.
Upstream budget exhaustion is Denied{BudgetUnavailable}. A narrowing
refusal carries its axis on the wire, and the audit scope is its own
axis. The chain walk and designated-grant lookups refuse a view answer
filed under another id, and the walk re-checks each link's depth
against its parent's maximum. The narrowing property now checks both
directions.
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	deny.toml
State Own-scope lineage, check order, session and target refusals,
mode-independent decisions, empty chains on refused plans, every
attenuation axis with its order, revocation regardless of effect time,
the legal transitions, BudgetUnavailable, and the axis on Denied.
The Entropy trait now fills &mut [MaybeUninit<u8>] and returns the
initialized slice; OsEntropy calls getrandom::fill_uninit. A new
random_array helper draws nonces, salts, tenant data keys and root keys
from an uninitialized buffer with no initializing literal and no unsafe,
then scrubs the scratch copy. Key material is wrapped in SecretBox at
once. Known-answer tests are unchanged.
…hylake-custody

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	deny.toml
Key draws now use random_secret_array, which returns Zeroizing<[u8; N]>.
TenantDataKey and RootKey copy it into the heap SecretBox by reference
via init_with_mut while the source is alive, so no unwiped by-value copy
of the key remains. random_array delegates to it for public values. The
WARNING residual records the possible try_from stack temporary.
# Conflicts:
#	Cargo.lock
#	deny.toml
…hylake-custody

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	deny.toml
- Store each terminal as a typed record: Settled{failure},
  Released{reason}, or UnknownEffect. Released(Abandoned) is no longer
  recorded as a Cancelled failure; the invocation and its audit entry
  carry the reason as given, and the Cancelled reply for an abandoned
  call is derived, not stored.
- Assign the artifact id from the invocation id, so two invocations
  cannot claim one artifact and a roll-forward publish cannot wedge
  recovery on a conflict.
- Bind the idempotency entry to the designated grant, capability,
  session, target, and declared cost in the store, and drop the request
  digest from the store-sealed invocation record, where it would have
  survived a crypto-shred.
- Match a repeated grant issue against the stored child before any
  decision, so a parent that has spent budget since does not turn a
  repeat into a refusal.
- Refuse a B2 settlement whose failure no started producer reports.
- Add a scoped audit read (All, OwnAndOwnedSessions, session narrowing,
  paging across partitions) for the lifecycle slice.
- Race eight reservations on a ceiling of three behind a barrier.
- Document the sealing-scope table, the shred residue, the blob key,
  the artifacts keyspace record kinds, directory idempotency, the key
  cache, and the deferred migrate command; state the UnknownEffect
  charge as the whole reservation; correct the hashbrown skip
  retirement conditions.
@forkwright
forkwright merged commit a5acfb1 into main Sep 25, 2026
13 checks passed
@forkwright
forkwright deleted the feat/phylake-custody branch September 25, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant