Skip to content

feat(dioptron): orchestrate invocations and ship the daemon binary - #90

Merged
forkwright merged 79 commits into
mainfrom
feat/dioptron-lifecycle
Sep 25, 2026
Merged

forkwright merged 79 commits into
mainfrom
feat/dioptron-lifecycle

Conversation

@forkwright

@forkwright forkwright commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

What changed

The daemon now runs end to end. The orchestrator implements the wire server's Dispatcher on top of the custody store and epitrope, and the binary has real commands. Process-level acceptance tests drive the real binary only through the independent xenos client.

Producer seam

  • Producer takes a cancel signal and a deadline.
  • UnavailableProducer is the default: it always answers ProducerUnavailable{contacted: false}, so the binary never fetches anything.
  • FixtureProducer is scripted per target, counts its calls, and runs only behind an explicit --producer fixture:<path>.
  • There is no HTTP, DNS, SSRF or extraction code anywhere; the real Zetesis adapter stays behind Consume Zetesis acquisition instead of implementing a second static-fetch stack #66.

Capture lifecycle

  • B1 begin authorizes, reserves and persists intent. Then:
    1. a re-check of the chain at dispatch;
    2. B2;
    3. the producer runs, with cancel and deadline, while its chain is re-checked every 100 ms;
    4. B3, which stores the envelope verbatim;
    5. B4 publish;
    6. B5 settlement, with the chain checked again.
  • Early stops:
    • no effect yet → released (Revoked / Expired / Cancelled / DeadlineExceeded);
    • effect started → actual cost settled;
    • producer can't say → UnknownEffect;
    • revocation or expiry after the effect → published with revoked_after_effect.
  • The orchestrator owns the settlement task, and it drains on shutdown.

Other paths

  • Every capability goes through the same designated-grant authorization.
  • Dry-run writes nothing.
  • A replay re-authorizes before returning anything. If the grant has since been revoked or has expired, it returns the current Denied without the stored content.
  • Refusals carry no invocation id, so foreign and missing resources produce identical bytes.
  • Read is chunked to the negotiated frame size.
  • Audit reads are scoped per D17.7, and each one records the designated grant and the scope applied.
  • Ingest is refused as NotSupported until the D7 pipeline exists (Phase 03).
  • An omitted limit reserves the remaining ceiling from the caller's own ledgers, capped by documented daemon limits.
  • The Query predicate is a case-sensitive substring match of at most 1024 bytes.

Binary (std::env::args)

  • Commands: serve, keygen, init, tenant add.
  • serve fails closed on a locked store before it binds anything, recovers, prints ready <socket>, and shuts down on SIGTERM.
  • Test hooks, compiled only when their cargo feature is enabled:
    • failpoints: DIOPTRON_FAILPOINT=after_commit:B2 and similar abort the process at that point;
    • test-clock.

Contract and store updates (the contract stays at v1; nothing has shipped)

  • New codes: DenyCode::NotSupported and ReleaseReason::Expired (answered as Denied{GrantExpired}).
  • The idempotency binding now covers digest, grant, session and target, without the declared cost, so a retry with a new deadline replays instead of conflicting.
  • Audit entries store the grant and scope.
  • The contract doc gains the early-stop settlement table, the claim order, and the unset-limit and predicate rules.

Why

Phase 01 S2 acceptance: "An independent test client must exercise the wire, not just call the in-process implementation."

This PR wires every S2 piece into one runnable daemon and proves each acceptance clause against the real binary.

Stage and acceptance

Phase 01 S2, closing slice. Clause → test (process tests in crates/dioptron/tests/process/):

Clause Test
Grant attenuation, expiry, revocation grants::child_grant_is_refused_on_every_widened_axis_and_issued_when_narrower, grant_is_refused_outside_its_validity_window_as_the_clock_moves, revoking_a_parent_invalidates_descendants_and_stands_on_replay, expiry_of_a_running_call_before_effect_releases_it_as_expired
Forged peer identity identity::every_forged_identity_gets_the_same_auth_failed_bytes
Cross-tenant read and metadata denial isolation::foreign_and_missing_resources_answer_identical_bytes, audit_reads_stay_inside_the_granted_scope
Dry-run has no effects dry_run::dry_runs_call_no_producer_and_leave_the_store_byte_identical
Replay of the same invocation replay::replayed_capture_returns_the_first_outcome_across_restart_without_a_second_call, replay_after_revocation_is_denied_and_returns_no_stored_content
Crash and reopen at each durability boundary crash::crash_{before,after}_b{1..5}_* (10 tests, real process abort)
Frame corruption, oversize, incompatible version, partial connection wire::*
Revocation of queued and running calls grants::revocation_of_a_dispatched_call_before_effect_releases_it, revocation_after_effect_publishes_the_marker_and_reads_need_a_live_grant
Session → capture → read → restart → read → unauthorized peer flow::session_capture_read_restart_read_then_unauthorized_peer_is_refused

Proof (toolchain 1.97.1)

  • Tests: 556 pass across the workspace. The process tests passed 5 consecutive runs; the default-features process tests passed 2 runs.
  • Lint: clippy -D warnings is clean with all features, with default features, and with each feature alone.
  • Supply chain and docs: cargo deny check, cargo tree -d (documented skips only), fmt and the doc checks are clean.
  • Independent adversarial review:
    • it implemented the integrator rulings: Ingest as NotSupported, audit grant and scope, distinct expiry, replay re-authorization, unset-limit derivation, and the predicate bound;
    • a B1 step failure left its reservation held; it is now released;
    • a capture without an idempotency key answered as a conflict; it now answers ProtocolError;
    • CLI argument parsing accepted malformed input; that is fixed.

Phase 01 S2 workspace: syntheke, epitrope, phylake, dioptron (bin +
lib), and xenos (publish = false). Shared lints follow kanon RUST.md;
toolchain pinned to 1.97.1 and rust-version 1.97, the toolchain CI
builds. No external dependencies yet. The daemon binary exits with
failure until commands land, and a process test pins that behavior.
deny.toml follows the zetesis posture (multiple-versions and wildcards
denied, unknown registries and git sources denied) and scans the
all-features graph the gate builds. The license allowlist covers the
workspace license and the licenses of the planned Phase 01
dependencies. .cargo/audit.toml and osv-scanner.toml are derived from
the deny.toml advisory ignore list.
The hybrid gate slots now run cargo fmt, check, clippy, nextest, and
doctests with --all-features; the toolchain comes from
rust-toolchain.toml. The corpus checks move to a docs job that runs on
every PR and on main. The docs-only exemption is on, which is sound
only while branch protection requires docs next to gate / gate.
.kanon-ci.toml gains the Rust stages ahead of the doc stages.
security.yml calls the fleet reusable (cargo-deny, cargo-audit,
osv-scanner) on PRs, main, and daily. CodeQL gains a Rust job and now
runs on PRs so it can be a required check. Dependabot also watches
Cargo dependencies.
release-please keeps release-type simple and updates the workspace
version, versioned path dependencies, and local Cargo.lock entries
through extra-files, as zetesis does.
The operator decision of 2026-09-25 starts the implementation phase.
AGENTS.md scopes crates to the Phase 01 plan; CLAUDE.md, README.md,
llms.txt, and _llm describe the workspace and the live build gate.
README.md's license section now matches LICENSE and LICENSE-DOCS.
Allow only the workspace's own license in deny.toml until dependencies
arrive, drop the private-plan S1 reference from the syntheke crate
docs, remove an unrecorded decision name from AGENTS.md, and drop a
copied comment with a nonstandard tag from the CodeQL rust job.
Implement the encryption-at-rest layer of the custody store design:

- keyfile: 32-byte root key load that refuses a missing file, a
  non-regular file, any group/other permission bit, or a wrong length;
  generation via O_CREAT|O_EXCL with mode 0600 from getrandom. Key held
  in secrecy::SecretBox.
- crypto: HKDF-SHA256 store subkeys (dioptron/v1/{blob,meta,audit,
  index,kek,check}) under a per-store salt; key check
  HMAC(k_check, "dioptron-key-check") verified in constant time, with
  StoreLocked on mismatch; random per-tenant data keys wrapped by the
  kek with XChaCha20-Poly1305, AAD binding kek id, data key id, and
  tenant id; tenant subkeys (blob, blob-addr, meta, audit, index).
- Sealed value format rec_ver u16 | key_id u32 | nonce[24] | ct|tag,
  AAD binding label, schema version, record kind, key id, and
  u16-length-prefixed keyspace and record key. Random 24-byte nonces.
- Tenant-keyed blob address (HMAC-SHA256) and plain SHA-256 provenance
  digest for sealed metadata only.

Dependencies stay on one RustCrypto generation (digest 0.11,
crypto-common 0.2): chacha20poly1305 0.11, hkdf 0.13, hmac 0.13,
sha2 0.11, plus getrandom 0.4, secrecy 0.10, zeroize 1, snafu 0.9;
tempfile 3 for tests. deny.toml allows MIT, Apache-2.0, Unicode-3.0.
rkyv 0.8 with bytecheck is the wire format fixed by topology.md; the
format features are named so a conflicting layout fails the build.
snafu is the fleet error library. toml is a dev-dependency for the
contract fixture test. deny.toml allows MIT, Apache-2.0, and
Unicode-3.0 for these crates and skips the syn 2 copy that snafu-derive
and munge_macro still build with.
Identifiers with ULID text form, the capability, mode, scope, and
budget vocabulary, request and reply payloads for every capability,
the outcome taxonomy, the frame header codec, handshake frames, the
authentication transcript, and a decoder that bounds length before
copying and validates the archive before any field is read.
Every message type round-trips through a full frame; corrupted and
truncated bodies are rejected. The fixture test maps each file under
docs/contract/fixtures onto syntheke types and fails when the
directory or a declared fixture is missing.
Open the root key with O_NOFOLLOW | O_NONBLOCK | O_CLOEXEC | O_NOCTTY
through rustix and check the opened descriptor: a symbolic link at the
key path, a FIFO, socket, or device node, or a file whose owner is not
the effective uid is refused before any read. Key generation also
creates with O_NOFOLLOW. rustix is already in the lock file through
tempfile, so no new crate version enters the tree.

External errors (io, getrandom) move to an `error` source field per the
kanon snafu convention, and a shrinking file reports its current length.
Add the draft-irtf-cfrg-xchacha-03 A.3.1 AEAD vector and byte-exact
seal and wrap vectors computed with an independent reference. Wipe the
HKDF PRK returned to the crate, record the stack residue the RustCrypto
primitives leave as an accepted WARNING, and restate the plaintext bound
in terms of what bounds a stored value.
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/phylake/Cargo.toml
#	crates/phylake/src/lib.rs
#	deny.toml
Request gains a grant field naming the one grant a call acts under, in
execute and dry-run mode. GrantIssueRequest drops parent_grant: the
designated grant is the parent, so the two can no longer disagree.
Fault frames now refuse any failure other than ProtocolError and
AuthFailed on encode and decode, and VersionChoice is non-exhaustive.

Fixtures carry grant as a wire field on every capability request; the
cast gives tnt0f grant grn0f, and neg_foreign_grant covers a request
naming another tenant's grant. The fixture conformance test splits into
load, mapping, and test modules.
Every capability request names the grant it acts under; the daemon
authorizes against that grant's chain only. A missing or foreign
designated grant returns the same NotFoundOrDenied, and the idempotency
digest covers the grant. The wire section now lists the eight frame
kinds with their bytes, restricts Fault to connection-level failures,
fixes the auth transcript layout, and states time units. Fixture keys
move grant to the wire fields. The contract stays at version 1.
Names the crates on each side of the syn 2/syn 3 split, the one
alternative (holding rkyv at 0.8.17) and why it is rejected, and the
retirement condition. The license list takes the same entries and order
as the storage branch so the later merge touches only comments.
Inline #[cfg(test)] modules move to sibling tests.rs files, and
test-only helpers (fixed entropy, shared key fixtures, from_bytes
constructors) move to test_support.rs files, so path-scoped security
scanning can exclude test code. Test names and count are unchanged.
Add the pure authorization crate over the syntheke contract types:

- narrowing of a child grant against the designated parent on every
  axis, reporting the first failing NarrowingAxis;
- chain validity at an injected clock against revocation records, so an
  ancestor's revocation invalidates descendants without fan-out;
- designated-grant authorization where a grant the connection's tenant
  does not hold reads exactly as a missing one;
- reservation and settlement arithmetic with checked ledgers and a typed
  overrun;
- the invocation transition table and restart recovery actions;
- a dry-run planner over a read-only snapshot trait;
- D17.7 audit scope defaults and a rule view with no audit accessor;
- a minimal origin parser for target scopes that refuses every
  construct where parsers could disagree about the host.

proptest (dev only, no default features) checks that a child passing
the narrowing check is never broader than its parent on any axis.
Add .github/codeql/codeql-config.yml with paths-ignore for tests.rs,
tests/ and test_support.rs, reference it from the Rust init step, and
record the test layout convention in CLAUDE.md.
…hylake-custody

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	deny.toml
Add DenyCode::BudgetUnavailable for exhaustion on a ledger the caller
does not own; it names no dimension. Add NarrowingAxis::AuditScope.
Failure::Denied carries axis: Option<NarrowingAxis>, set exactly for
NarrowingViolation; Response::check refuses any other combination in a
reply or a plan's refusal (DeniedAxisMismatch). Move the outcome unit
tests to outcome/tests.rs.
The socket server needs an async runtime with Unix listeners and peer
credentials (tokio), handshake signature verification on the custody
crypto's RustCrypto generation (ed25519-dalek 3), a fresh server nonce
from the one OS random source (getrandom 0.4), and per-connection spans
(tracing). tempfile and rkyv are dev-only. BSD-3-Clause is allowed for
ed25519-dalek, curve25519-dalek, and subtle.
Bind a 0600 socket in a 0700 directory, refusing any existing path that
is not a provably stale socket. Each connection takes a permit from a
global semaphore, reads peer credentials at accept, and completes the
handshake within 5 s under the 4 KiB pre-auth cap: version negotiation,
a fresh server nonce, and Ed25519 verify_strict over the syntheke
transcript plus a bound-uid check. Every auth failure, including a
request before admission, gets the same Fault(AuthFailed) bytes.

After Admitted, requests dispatch as tasks through the Dispatcher seam
with a cancel signal and a clamped monotonic deadline; a Cancel frame,
connection close, or shutdown fires the signal. Protocol violations end
the connection with one Fault(ProtocolError). TenantDirectory and
Dispatcher are the seams the custody store and lifecycle implement.
…hylake-custody

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	deny.toml
…optron-lifecycle

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	deny.toml
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	deny.toml
Calls outside the capture lifecycle (session create and fork, grant
issue and revoke, ingest) bind their idempotency key through
Store::claim in the same idem keyspace begin uses. Store::audit_query
answers the All and OwnAndOwnedSessions audit scopes (D17.7), and
Store::record_audit commits a standalone entry for refusals and audited
reads. Store::logical_digest hashes every key and sealed value so a
caller can prove an operation wrote nothing.
- Store each terminal as a typed record: Settled{failure},
  Released{reason}, or UnknownEffect. Released(Abandoned) is no longer
  recorded as a Cancelled failure; the invocation and its audit entry
  carry the reason as given, and the Cancelled reply for an abandoned
  call is derived, not stored.
- Assign the artifact id from the invocation id, so two invocations
  cannot claim one artifact and a roll-forward publish cannot wedge
  recovery on a conflict.
- Bind the idempotency entry to the designated grant, capability,
  session, target, and declared cost in the store, and drop the request
  digest from the store-sealed invocation record, where it would have
  survived a crypto-shred.
- Match a repeated grant issue against the stored child before any
  decision, so a parent that has spent budget since does not turn a
  repeat into a refusal.
- Refuse a B2 settlement whose failure no started producer reports.
- Add a scoped audit read (All, OwnAndOwnedSessions, session narrowing,
  paging across partitions) for the lifecycle slice.
- Race eight reservations on a ceiling of three behind a barrier.
- Document the sealing-scope table, the shred residue, the blob key,
  the artifacts keyspace record kinds, directory idempotency, the key
  cache, and the deferred migrate command; state the UnknownEffect
  charge as the whole reservation; correct the hashbrown skip
  retirement conditions.
…dioptron-lifecycle

# Conflicts:
#	crates/phylake/src/store/mod.rs
#	crates/phylake/src/store/tests.rs
#	deny.toml
The orchestrator is the server's Dispatcher. Each admitted request runs
as a task it owns, so settlement finishes even when the server drops
the dispatch future at its deadline backstop.

- Capture runs B1 intent, a dispatch re-check of the chain, B2, the
  producer call under a cancel signal fed by the caller's Cancel, a
  revocation of any chain link, and the deadline, then B3, B4, B5; or
  release when the producer proves no effect, settlement when it
  started, UnknownEffect when it cannot say or does not return.
- A revocation after the effect publishes with revoked_after_effect;
  reads still need a live grant.
- DryRun of every capability plans over a snapshot and writes nothing.
- Session, grant, read, query, audit, and ingest calls authorize under
  the designated grant; refusals commit a Denied audit entry and answer
  without an invocation id, so foreign and missing read the same.
- Replies fit the connection's frame bound; Read is chunked.

The Producer seam carries the verbatim envelope and its evidence
identity. UnavailableProducer never fetches; FixtureProducer replays
scripted outcomes and counts calls. No fetch code exists here.
dioptron serve opens the store (a missing or wrong root key fails
closed before anything binds), runs restart recovery, binds
<socket-dir>/dioptron.sock, and serves until SIGINT or SIGTERM, then
drains its request tasks. The producer is unavailable unless a fixture
script is named with --producer fixture:<path>, so the binary never
fetches by default. tenant add registers a tenant with its verifying
key and bound uids and installs an operator's root grant with the D17.7
default audit scope.

The failpoints feature makes DIOPTRON_FAILPOINT=<phase>:<Bn> abort the
process at that store commit; the test-clock feature makes
DIOPTRON_TEST_CLOCK=<file> the daemon's wall clock. Neither is built by
default.
Process-level acceptance tests spawn the dioptron binary in a tempdir
and drive it only through the independent xenos client: grant
attenuation, expiry, and revocation (queued, running, after effect);
forged identities answering identical AuthFailed bytes; cross-tenant
denial with byte-identical NotFoundOrDenied; dry-run with no producer
call and a byte-identical store digest; idempotent replay across a
restart; a crash at every B1 to B5 boundary with recovery and call
counts; corrupt, oversized, incompatible, and partial connections; and
the full session, capture, read, restart, read flow.
…dioptron-lifecycle

# Conflicts:
#	Cargo.toml
#	deny.toml
# Conflicts:
#	Cargo.toml
#	crates/phylake/src/store/mod.rs
#	crates/phylake/src/store/tests.rs
Contract v1 refuses Ingest as Denied{NotSupported} until the D7
pipeline lands, records expiry as its own release reason, and caps a
Query predicate at 1024 bytes, enforced by Request::check so a longer
one is a ProtocolError. Response::invocation is None on every refusal.
Adds the neg_ingest_not_supported fixture and its mapping.
authorize refuses an unserved capability after the grant, chain, and
capability checks and before any session read, so the answer never
depends on the named artifact. The transition table releases Expired
from B1 and B2 like Revoked. designated_chain is public so the daemon
re-runs checks 1 to 3 on a replay.
Audit entries carry the designated grant and, for audit reads, the
applied scope (schema v1, not shipped). Released(Expired) replies as
Denied{GrantExpired}. The idempotency binding no longer covers the
declared cost: it is derived from the deadline and from budget state the
first attempt's own settlement lowers, so binding it turned a retry into
a conflict.
- Ingest is refused at authorization with only a Denied audit entry.
- A running capture re-checks its chain, so expiry stops the producer;
  expiry releases as Expired, revocation as Revoked, in walk order.
- A replay re-validates the designated chain before answering and
  never returns stored content under a revoked or expired chain.
- Unset capture limits declare the chain's remaining ceiling, bounded
  by the store's largest envelope and the reply frame payload.
- Audit entries name the designated grant; audit reads the scope.
- A step that fails inside the daemon releases at B1 and charges at
  B2 instead of holding the reservation until restart.
A flag value starting with -- is a missing value, a verifying key must
be 64 hex digits (from_str_radix accepts a leading +), and root-grant
flags on a non-operator tenant are a usage error instead of ignored.
Process tests: a replay after revocation is Denied with no stored
content and no second producer call; a running call whose grant
expires ends as GrantExpired; Ingest answers the same bytes for an
existing and a missing artifact; and without the cargo features the
failpoint and clock variables change nothing.
Ingest refused as NotSupported, expiry distinct from revocation,
replays that re-authorize, refusals without an invocation id, early
stop settlement, claim order for session and grant calls, daemon
limits for unset capture limits, the query predicate, audit entries
that record grant and scope, and the unbound declared cost.
An omitted capture limit defaulted to the smallest remaining ceiling
across the whole grant chain, so a dry-run plan or any reply carrying
the declared limit disclosed an ancestor grant's remaining budget.

The default now comes from the ledgers the caller owns: its tenant
ledger, the chain grants it holds, and the session when it owns it
(epitrope::caller_remaining over own_remaining), capped by the daemon
caps. Ancestor ledgers only gate the reservation and refuse a shortfall
as Denied{BudgetUnavailable}, which names no dimension. The contract's
daemon-limits text records the rule.

default_producer_never_fetches now asserts the released terminal, the
zero debit, and the returned ledgers instead of an empty call log.
@forkwright
forkwright merged commit b4f093f into main Sep 25, 2026
13 checks passed
@forkwright
forkwright deleted the feat/dioptron-lifecycle branch September 25, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant