feat(observability): define gateway-to-exporter extension contract

[delgadof]

User Story

As an OpenShell gateway operator, I need a supported way to connect OpenShell
observability data to an external exporter so that I can deliver sandbox,
security, audit, trace, log, and metric data to enterprise systems without
scraping human-readable logs or assembling several incompatible collection
paths.

As a security or compliance investigator, I need exported records to retain
their authoritative OpenShell identity, schema, correlation, and source-health
information so that I can determine which gateway or sandbox produced them and
whether known collection gaps exist.

This matters because the gateway is a security-critical control-plane service
and should preserve predictable CPU, memory, disk, and network availability for
sandbox management and policy enforcement.

Keeping destination-specific processing outside the gateway isolates slow
SIEMs, delivery retries, schema transformations, historical searches, storage,
and destination credentials from gateway control traffic. The gateway still
incurs the bounded cost of authenticating, contextualizing, and relaying source
records, so the contract must define rate limits, buffering, prioritization,
drop accounting, and explicit gap behavior.

An external exporter can scale, buffer, restart, and fail independently. This
allows observability capacity to grow with event volume and destination demand
without requiring the gateway or supervisor to perform the same expensive work
or become dependent on external system availability.

This proposal does not duplicate #2762. It defines the integration boundary and operator experience through which #2762 can be delivered while keeping destination-specific transformation, credentials, queueing, retry, and recovery outside the gateway.

Problem Statement

OpenShell exposes observability information through several developing
interfaces, including sandbox logs, structured OCSF events, OpenTelemetry,
gateway audit events, and local files.

However, OpenShell does not yet define one supported gateway-to-exporter
contract that explains:

• which structured signal types are available;
• how gateway, sandbox, and source identity are represented;
• how an external consumer authenticates and is authorized;
• how schema and capability versions are negotiated;
• how a consumer resumes after disconnecting;
• how duplicates, source restarts, retention boundaries, and known gaps are
  represented;
• which context OpenShell guarantees is authoritative;
• how exporter backpressure is isolated from gateway control operations.

As a result, external consumers cannot rely on one consistent OpenShell
integration surface across logs, OCSF records, traces, metrics, audit events,
and future evidence workflows.

Impact / Why This Matters

Today, an enterprise integration may need to combine sandbox files, read-only
gRPC streams, OTLP receivers, Kubernetes metadata, and human-readable log
output. Each integration must independently reconstruct identity, correlation,
recovery, and failure behavior.

This creates several problems:

• human-readable shorthand may omit structured OCSF fields;
• separate collection paths can represent identity and schema versions
  differently;
• consumers cannot consistently distinguish duplicates from new records;
• disconnects, retention expiration, and source drops may create silent gaps;
• operators may need to deploy and configure several unrelated collectors;
• destination-specific processing can be pushed toward the gateway, increasing
  its CPU, memory, dependency, credential, and failure surface;
• a slow or unavailable destination may create pressure on a security-critical
  OpenShell service;
• governance consumers cannot safely infer authoritative policy associations or
  completeness from independently collected artifacts.

The missing contract is also an adoption barrier. Operators should not have to
design their own observability pipeline merely to send structured OpenShell
records to a supported enterprise destination.

Proposed Design

OpenShell should provide a documented, authenticated gateway-to-exporter
extension contract.

From the operator's perspective, the workflow should be:

1. Enable the supported export capability.
2. Deploy one supported exporter extension using a documented installation
   path.
3. Give the exporter a read-only, appropriately scoped OpenShell identity.
4. Select the OpenShell signal types the exporter should consume.
5. Configure customer destinations and destination compatibility in the
   exporter.
6. Observe exporter connectivity, collection progress, replay, gaps, rejected
   records, queue state, and delivery health.
7. Restart or upgrade the exporter without silently losing retained records.
8. Receive an explicit indication when a requested range cannot be recovered.

First-Class Operator Experience

Although the exporter runs outside the gateway, it should be presented as a
first-class OpenShell integration rather than as unrelated out-of-band tooling.

An operator should have one documented workflow to:

• install or enable the exporter;
• connect it securely to one or more gateways;
• select the signals to export;
• configure destinations;
• validate the configuration before starting;
• inspect source, cursor, gap, queue, and destination health;
• upgrade the gateway and exporter using documented compatibility rules;
• disable or remove the exporter without affecting sandbox enforcement.

The operator should not need to manually assemble separate collectors for OCSF,
logs, traces, metrics, redaction, compatibility conversion, retries, and
recovery.

The exact distribution mechanism is intentionally left open. It could be an
optional Helm deployment, container, system service, package, or another
supported mechanism appropriate to the OpenShell deployment. The externally
observable installation, configuration, health, compatibility, and upgrade
experience is part of this feature.

Evidence processing should be optional. Enabling ordinary observability export
must not require PostgreSQL, immutable object storage, evidence signing, or the
#2745 workflow.

Security and Communication Boundary

The supervisor should continue using its existing authenticated connection to
the gateway. Operators should not need to configure a second privileged
supervisor connection to an exporter.

The exporter should authenticate using its own read-only, appropriately scoped
OpenShell identity. It should not receive or reuse supervisor, sandbox, or
workload credentials.

A slow or unavailable exporter or customer destination must not block sandbox
enforcement or gateway control-plane operations.

Structured Signal Preservation

Exported information should preserve its native structured representation:

• OCSF records remain structured OCSF;
• traces and metrics remain compatible with their telemetry schemas;
• logs remain structured logs;
• audit records retain their authoritative actor, operation, resource, and
  outcome;
• source-health records describe drops, restarts, retention boundaries, and
  known gaps.

The proposal does not require logs, metrics, traces, and OCSF records to be
converted into one common payload schema. They should share the identity,
versioning, source-health, and recovery information needed by an external
consumer.

Common Export Metadata

The contract should define common externally observable metadata where
applicable, including:

• signal and contract version;
• source, gateway, and sandbox identity;
• observation time;
• source epoch, cursor, or sequence information;
• payload schema and schema version;
• trace and span identifiers;
• OpenShell version;
• effective-policy revision or digest when OpenShell can authoritatively provide
  it;
• known source gaps or unavailable history.

OpenShell should remain authoritative for OpenShell-owned facts such as sandbox
identity, enforced-policy context, source identity, trace correlation, and
known gaps.

An exporter may preserve, verify, validate, index, correlate, transform, and
deliver those facts, but it must not infer or invent authoritative OpenShell
associations or claim completeness when OpenShell has reported unavailable
history.

Independently Deployable Exporter

The external exporter should be independently deployable so destination
failures do not block gateway control operations.

From the user's perspective, it should be one supported deployment even if it
contains separate internal functions for:

• authenticated acquisition;
• durable ingestion and recovery;
• validation and routing;
• operational delivery;
• optional evidence processing.

This proposal does not require those functions to run as separate services or
containers.

Operational Observability and Evidence

Operational observability and governance evidence should share the same
authenticated acquisition path but remain distinct downstream workflows.

Operational Observability

Operational observability is optimized for timely delivery and may perform:

• redaction;
• enrichment with external deployment context;
• correlation;
• destination-specific schema compatibility;
• CloudEvents or OTLP envelope conversion;
• persistent queueing;
• retries and recovery;
• delivery to customer destinations.

Governance Evidence

Evidence processing is optimized for integrity and reproducibility. It
preserves eligible canonical records and requires explicit:

• provenance;
• integrity;
• policy association;
• sequence and gap information;
• completeness semantics;
• independent verification.

The detailed evidence-bundle behavior remains in #2745. The exporter may
construct and sign a final bundle, but it must not independently invent
OpenShell-owned policy associations or source-completeness assertions.

Proposed Architecture

flowchart LR
    S["Supervisor<br/>Emit structured signals<br/>OCSF · logs · OpenTelemetry<br/>Report drops · restarts · rotations"]

    G["OpenShell Gateway<br/><b>Authoritative identity and context</b><br/>Authenticated source<br/>Supported export contract"]

    subgraph EX["External Exporter — one supported deployment"]
        X["Unified receiver<br/>Receive once · validate · route"]
        O["Operational observability<br/>Transform · queue · retry · deliver"]
        E["Optional evidence<br/>Preserve · verify · package"]
        X --> O
        X --> E
    end

    D["OTLP / CloudEvents / SIEM"]
    A["Evidence archive<br/>Independent verification"]

    S -->|"Existing secure channel"| G
    G -->|"Structured records<br/>Authoritative context<br/>Source health and recovery state"| X
    O --> D
    E --> A

Loading

The exact transport, storage, queue, and database implementation is
intentionally left open. The feature being requested is the externally
observable installation, authentication, structured-record, versioning,
recovery, gap, backpressure, and compatibility contract.

OpenShell remains authoritative for OpenShell-owned facts. The gateway establishes control-plane identity, policy revision, authorization, and source association. The supervisor emits locally observed enforcement and source-health facts through its authenticated gateway connection. The exporter may validate, preserve, transform, index, and deliver these facts, but must not invent missing identity, policy, sequence, gap, or completeness information.

Acceptance Criteria

• OpenShell documents a supported interface through which an authenticated
  external exporter can consume structured observability records.
• The documentation identifies the supported signal types and distinguishes
  their native payload schemas.
• Complete structured OCSF payloads are available without parsing shorthand
  log messages.
• Gateway, sandbox, and source identity semantics are documented and stable
  across supported signal types.
• The contract defines version and capability-negotiation behavior.
• An exporter can reconnect and resume from a documented cursor, sequence,
  or retained position.
• Duplicate delivery can be detected without relying only on timestamps or
  payload comparison.
• Source restarts, known drops, retention expiration, and unrecoverable gaps
  are represented in a machine-readable way.
• An operator can determine whether the exporter is connected, progressing,
  replaying, blocked, or missing history.
• Exporter authorization can be restricted to appropriate gateways,
  sandboxes, tenants, or signal types.
• A slow or unavailable exporter does not block supervisor enforcement or
  gateway control-plane operations.
• Record sizes, batches, buffering, and source rates have documented bounds
  and observable rejection behavior.
• Existing credential-redaction guarantees apply to exported structured
  payloads.
• Mixed-version gateway, supervisor, and exporter behavior is documented and
  tested.
• Destination credentials, destination-specific schema conversion, and
  delivery retries are not required in the supervisor.
• A reference exporter can consume the supported contract and deliver
  records to at least one documented external destination.
• The deployment documentation presents the exporter as one supported
  operator workflow rather than requiring users to assemble multiple
  independent collectors.
• Evidence consumers can identify which additional provenance or
  completeness guarantees are available and which are not.
• Export traffic is bounded and isolated so exporter or destination
  backpressure does not block sandbox enforcement, policy delivery,
  credentials, or gateway control operations.
• In an HA gateway deployment, the supported export workflow does not
  silently duplicate or lose retained records when supervisor-session
  ownership or gateway replicas change; any unrecoverable interval is
  reported explicitly.

Alternatives Considered

Continue using independent collection paths

Operators could continue combining files, gRPC, OTLP, Kubernetes context, and
human-readable logs.

This preserves existing interfaces but requires every integration to implement
identity, correlation, recovery, and gap detection independently. It also
creates inconsistent behavior between exporters.

Export only human-readable log messages

This is simple for basic troubleshooting but requires fragile parsing and
cannot preserve complete OCSF fields or schema fidelity.

Configure every destination directly in the gateway

This could reduce the number of deployed components, but every new destination
would expand gateway configuration, dependencies, credentials, compatibility
logic, persistent queues, retries, and failure modes. Destination backpressure
could also affect a security-critical control-plane service.

Connect supervisors directly to external exporters

This could reduce gateway data volume, but it would introduce a second
privileged connection from each supervisor, change the existing trust model,
and increase configuration and failure modes in the sandbox hot path.

Support only durable files

Durable files may provide a useful collection and recovery mechanism, but
operators would still need defined identity, permissions, rotation, retention,
cursor, duplication, and gap semantics. Files may be one implementation of the
contract rather than the complete user-facing contract.

Support only a live stream

A live stream provides low-latency delivery but does not by itself explain how
an exporter recovers after disconnecting or detects data lost beyond the
retention window.

The proposed workflow defines one externally consistent contract while leaving
room for durable files, streaming, or a combination of both underneath it.

Agent Investigation

Investigation was performed against OpenShell main at
e1084e19db622fd1134454d3b589aab2022aaddf on 2026-09-08.

The public repository, architecture documentation, existing issues, and related
pull-request changes were reviewed to understand the current data paths and
avoid duplicating existing work.

Existing Architecture Constraints

RFC 0001 establishes that supervisors connect outbound to the gateway and that
subsequent control-plane communication, including log streaming, flows over that
supervisor-initiated connection.

The proposal preserves that constraint. It does not require a direct
supervisor-to-exporter connection.

Existing Implementation Foundation

• fix(ocsf): make exported events valid and attributable #3103 makes OCSF records valid and attributable. It distinguishes event,
  gateway, replica, and sandbox identity and preserves API action and
  disposition information.
• feat(ocsf): carry structured events on sandbox log stream #3104 adds a complete ocsf_json payload to SandboxLogLine while preserving
  human-readable shorthand for supported mixed-version behavior.
• feat(ocsf): carry structured events on sandbox log stream #3104 therefore removes the need for a future exporter to reconstruct OCSF
  records from display text.
• The feat(ocsf): carry structured events on sandbox log stream #3104 changes use the existing supervisor-to-gateway log path and do not
  yet provide durable retention, exporter acknowledgements, replay cursors,
  sequence continuity, gap reporting, or evidence provenance.
• feat(sdk/go): three-layer proto-to-API coverage checks (RPC methods, converter wiring, message auto-discovery) #3162 provides related SDK-gap tracking where a protobuf capability is not yet
  exposed through a curated SDK.

Issue-to-Architecture Mapping

The proposed contract unifies existing work; it does not replace these issues.

Issue	Formal relationship	Architectural contribution
#1055	Parent roadmap	Enterprise observability umbrella and user personas.
#1054	Direct child; completed	Product-usage telemetry. This remains separate from security and operational event export.
#1922	Direct child	Durable sandbox logs, OCSF files, rotation, retention, and source collection behavior.
#2507	Direct child	Gateway and driver OpenTelemetry export for operators.
#2508	Direct child	Supervisor span instrumentation.
#2642	Child of #2508	Shared authenticated supervisor-to-gateway telemetry relay, batching, priority, backpressure, and drop accounting.
#2639	Direct child	Incoming CLI trace-context continuity.
#2640	Direct child	Authoritative trace and span identifiers in OCSF records.
#2641	Direct child	Agent OTLP collection through the supervisor and gateway.
#1758	Direct child	Gateway activity trace correlation; remaining scope should be reconciled with #2507.
#909	Direct child	Server metrics instrumentation.
#1818	Direct child	Optional OpenTelemetry hooks for the Python SDK.
#2911	Direct child	Gateway control-plane mutation audit events.
#2912	Direct child	Gateway authentication and authorization audit events.
#3017	Direct child	Audit events for credential-disclosure reads.
#3018	Direct child	Audit events for supervisor session establishment and relay activity.
#2762	Direct child	Structured OCSF export from the gateway to external destinations. This proposal defines the shared extension boundary it requires.
#2662	Direct child	OCSF-version compatibility for customer security tooling. Destination-specific transformation can consume the canonical export contract.
#1933	Related; different parent	Centralized audit/event-log behavior that may provide reusable durable history or replay semantics.
#2745	Related; no current #1055 parent	Atomic governance evidence bundle requiring stronger provenance and completeness semantics.
#1012	Related roadmap	High-availability and horizontal-scaling context for gateway deployments.
#3103	Related implementation PR	OCSF validity and attribution foundation.
#3104	Related implementation PR	Complete structured OCSF transport through the existing sandbox log stream.
#3162	Related supporting issue	Tracking of generated-protocol capabilities missing from curated SDKs.

Remaining Architectural Gap

The issues above provide producers, instrumentation, relay work, structured
payloads, and destination requirements, but they do not yet define one external
consumer contract covering:

• authentication and authorization;
• common source identity;
• contract and payload versioning;
• cursor and replay behavior;
• duplicate detection;
• source epochs and gaps;
• backpressure isolation;
• consistent deployment and operator experience.

This feature request is intended to define that shared gap.

Because the proposal crosses the supervisor/gateway protocol, gateway
responsibilities, external API behavior, security boundaries, performance, and
operator experience, it may warrant an RFC.

If maintainers agree, please confirm this issue as the originating issue, add
needs-rfc, assign the RFC number, and identify the appropriate reviewers or
co-authors. I am willing to author or co-author the initial RFC and use the
experimental exporter as a reference implementation.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request