RFC: Pluggable duplex Discord voice with mediated ACP actions

[canyugs]

Description

Add an opt-in, provider-pluggable duplex Discord Voice subsystem that can hold a low-latency voice conversation while delegating real coding work to OpenAB's existing ACP agents.

The core design is to keep the voice model and the coding agent separate:

Discord Voice / Songbird
        ↓ attributed audio
VoiceCoordinator
        ↓
DuplexVoiceEngine
  ├─ composed STT → dialogue → TTS
  ├─ public realtime speech API
  └─ future GPT-Live API
        ↓ typed action proposal
ActionBroker
  ├─ speaker/session provenance
  ├─ Discord text confirmation
  ├─ effect policy + idempotency
  └─ ActionJobManager
        ↓ approved AcpTurn
Ephemeral AcpActionExecutor
        ↓
Claude Code / Codex / other ACP coding CLI
        ↓
canonical Discord text result → bounded speech projection

The voice engine handles listening, turn-taking, and speaking. It does not receive shell, repository, Discord, Kubernetes, or raw ACP access. Actual file edits, test execution, and other coding work continue to run through the configured ACP agent.

Relationship to #1364

This proposal complements rather than duplicates #1364: GPT-Live as a Voice Bridge for OpenAB.

	#1364	This RFC
Boundary	External Custom GPT + proxy + Discord REST	Native OpenAB Discord Voice subsystem
OpenAB changes	Intentionally zero	New transport-neutral voice/action contracts and Discord runtime work
Primary scope	Route voice commands across an OAB bot fleet	Give one OpenAB deployment duplex voice while retaining controlled ACP execution
Voice providers	GPT-Live-specific product concept	Composed STT/TTS, public realtime APIs, and future GPT-Live
Execution route	Post/read ordinary Discord messages	Typed ActionBroker jobs delegated to an ephemeral ACP executor
Identity	Proxy/API caller and Discord message identity	Songbird SSRC-to-Discord-user attribution plus operator confirmation
Safety/lifecycle	Proxy auth, routing, polling/rate limits	Effect ceilings, ACP permissions, idempotency, cancellation, unknown outcomes, workspace containment, and audit

#1364 remains a useful low-coupling fleet-wide client when GPT-Live exposes a suitable public action/API surface. This RFC defines the native OpenAB path needed when operators want their own STT/TTS providers, direct Discord Voice participation, or stronger execution and permission semantics. An external bridge from #1364 could later call this RFC's semantic action API instead of posting unrestricted natural-language commands.

Motivation

The current Discord Voice work establishes live Songbird receive, per-speaker attribution, bounded STT, transcript download, and explicit /voice summary dispatch. That is useful for meeting capture, but it does not yet support a bidirectional agent that can speak and perform work.

Using only a speech-to-speech model would improve conversational latency but would not, by itself, preserve OpenAB's coding/tool execution capabilities. Conversely, sending every transcript directly to a normal tool-capable ACP session would turn imperfect ambient STT into executable instructions.

This proposal adds a mediated boundary so operators may choose their own STT/TTS providers, a realtime provider, or a future GPT-Live API without changing execution authority.

Proposed decisions

1. Separate the conversation and action planes

• DuplexVoiceEngine owns provider-specific STT/VAD/dialogue/TTS or native speech-to-speech behavior.
• ActionBroker is the only route from voice output to executable work.
• AcpActionExecutor delegates accepted work to the existing ACP coding CLI.
• Realtime providers receive only semantic tools equivalent to:
  • start_agent_task(goal, effect_hint)
  • get_agent_task(action_id)
  • cancel_agent_task(action_id)
• There is no voice approve tool and no raw shell tool.

2. Treat provider output as untrusted

• Discord SSRC-to-user attribution is retained independently of STT text.
• The provider returns an untrusted proposal and a provider-local call ID.
• OpenAB mints the global action ID and attaches trusted guild, voice-session generation, Discord account, control channel, workspace, and expiry from server-side state.
• Mixed or unattributed audio cannot create an executable action.
• Ambient/non-operator transcript is audit data, not an ACP instruction.

3. Require text confirmation for v1 actions

Every provider-generated or provider-rephrased goal, including read-only work, must first be posted to the pinned Discord text channel and confirmed by the configured operator account.

This is intentional: a source-turn token proves which Discord audio stream triggered a proposal, but it does not prove that STT or the voice model described the operator's intent faithfully.

Spoken "yes" or "approve" never grants authority. Write and external-side-effect permissions remain independently constrained by the approved effect ceiling and the actual ACP permission requests.

4. Use a fresh ACP connection per action

The current ACP reader automatically prefers allow_always for session/request_permission. A long-lived text ACP session may therefore already contain persistent permission state before a voice action begins.

V1 action execution must not reuse that connection. Each accepted action starts a fresh, ephemeral ACP process/session with a connection-fixed action profile:

• safe agent command/config options applied and verified before the prompt;
• a fail-closed permission resolver that selects only challenge-provided allow_once / reject_once options;
• no normal SessionPool mapping, resume, or persistence;
• one target workspace and one running action per workspace;
• process-tree termination and confirmed quiescence before cleanup.

Normal text dispatch keeps its current pooled session and backward-compatible behavior.

5. Make execution and delivery outcomes independent

Extract the authoritative ACP event/finalization path from AdapterRouter::stream_prompt_blocks into a reusable turn driver returning a typed result:

struct TurnCompletion {
    execution: ExecutionOutcome,
    output: Option<TurnOutput>,
    delivery: DeliveryOutcome,
    legacy_dispatch: LegacyDispatchDisposition,
}

This lets the normal text dispatcher preserve its exact existing reactions/error behavior while the action system can distinguish:

• action succeeded, Discord result delivery succeeded;
• action succeeded, Discord result delivery failed;
• partial text was delivered but execution outcome is unknown;
• cancellation was confirmed;
• the child disappeared after effects may already have occurred.

After a prompt is sent, timeout, EOF, connection loss, or an unconfirmed cancellation must become OutcomeUnknown unless the containment backend can prove no effect occurred. The workspace remains quarantined until reconciliation.

6. Keep text as the canonical action audit

• The action proposal must be delivered before execution starts.
• The full result is posted to the pinned text channel before action speech.
• Default Voice Channel playback is status-only (for example, "The result was posted to the text channel").
• Detailed results may be spoken only if every current listener, including other bots, is trusted and can view the text channel.
• A listener join/move/permission change during detailed playback cancels or downgrades it.
• Native realtime dialogue audio may remain low-latency and best-effort-captioned, but it cannot announce approval or claim an action result.

7. Make audio formats and duplex capability explicit

An engine declares capabilities instead of relying on its name:

• streaming input/output;
• partial transcripts;
• interruption support;
• semantic tool calls;
• continuous duplex;
• exact speech output; and
• accepted input/output audio formats.

Audio chunks carry sample rate, channels, sample type/codec, media timestamp, and frame count. The coordinator performs bounded resampling/channel conversion outside Songbird's 20 ms callback.

A batch STT + TTS combination is a valid half-duplex engine. It must not be advertised as full-duplex. Barge-in and echo handling remain a later validation phase.

Prior art

• OpenAI GPT-Live validates the product shape of continuous foreground voice plus background delegation. Its API is not public yet, so it is a future provider rather than an implementation dependency.
• The public OpenAI Realtime API supports audio and function calls, but the application still executes the requested function and returns function_call_output. OpenAB therefore remains the action boundary.
• OpenClaw Discord Voice is the closest precedent: its agent-proxy mode separates the realtime voice front end from the routed agent brain. This proposal adopts that separation but does not adopt owner-equivalent automatic tool access.
• LiveKit Agents pipeline types provide useful prior art for composed, realtime, and half-cascade engines, as well as background tool execution while conversation continues.
• Hermes Agent security demonstrates defense in depth around authorization, dangerous-command approval, containment, credential filtering, and fail-closed timeouts.
• The ACP v1 Prompt Turn specification defines session/cancel as a notification, requires pending permission challenges to be cancelled, and uses the original prompt result as cancellation confirmation.
• Songbird remains the Discord transport/playback layer; it does not own STT, TTS, action authorization, or echo cancellation.

Implementation phases

1. Typed ACP turn driver/completion — extract finalization without changing normal text behavior.
2. ACP permission resolver — forward challenges without blocking the reader; preserve a legacy text policy and add a fail-closed action policy.
3. ActionBroker + ActionJobManager — first use a fake executor to validate proposal delivery, Discord confirmation, idempotency, expiry, cancellation, and restart behavior.
4. Ephemeral ACP action executor — run real Claude/Codex work in a disposable local Kubernetes workspace using a fresh connection per job.
5. Songbird outbound playback — validate a fixed tone/sample, queue bounds, stop, reconnect, and stale-session rejection.
6. Composed agent-proxy engine — current/streaming STT + configurable TTS + ACP actions; half-duplex first.
7. Public realtime engine — prove provider pluggability while exposing only semantic action tools.
8. GPT-Live provider — only after a stable public API contract exists.
9. Full-duplex/barge-in — only after echo/interruption and reconnect soak testing.

Deployment and containment

env_clear() remains required, but it is environment minimization rather than a filesystem/process/network sandbox.

The first functional spike runs only in docker-desktop/openab-local with a disposable repository, no push/deploy credentials, and a workspace that is discarded after restart or unknown outcome.

Before public beta, action execution must move to a separate worker pod or equivalently isolated runtime with:

• a distinct process/user boundary;
• only the scoped workspace and agent authentication;
• no Discord/STT/TTS/realtime Secret mounts;
• no Kubernetes API credential;
• reviewed egress; and
• durable write-ahead job/workspace lease state for restart reconciliation.

Backward compatibility

All new capabilities are independently opt-in:

• current receive-only Voice behavior remains the default;
• dialogue defaults off;
• playback defaults off;
• actions default off;
• detailed action speech defaults off;
• enabling current Voice receive does not silently add Discord Speak, provider calls, or agent permissions.

Acceptance criteria

• Unauthorized, unattributed, mixed-source, or replayed provider input cannot execute work.
• No action runs unless its canonical proposal was delivered and confirmed in Discord text.
• Voice/model output cannot approve itself or select a persistent ACP permission.
• A real approved action can edit a disposable repository and run tests through Claude/Codex ACP.
• Prompt-active crash/EOF/timeout produces OutcomeUnknown and workspace quarantine, not a false failure/success claim.
• Canonical text delivery precedes action playback; result delivery failure suppresses speech without rewriting the execution outcome.
• Every queue is bounded and stale session/action/audio results are rejected.
• Existing receive-only and normal text-dispatch behavior remains unchanged when new config is omitted.
• Two speakers, DAVE, reconnect, cancellation, audience changes, audio conversion, queue pressure, and a 30-minute soak are validated before claiming production readiness.

ADR

A detailed implementation ADR is prepared on feat/discord-voice-receive as commit 1bd0c62 and can be submitted as a follow-up documentation PR after this RFC is triaged.