diff --git a/README.md b/README.md index 828d09c..c8dfd87 100644 --- a/README.md +++ b/README.md @@ -131,28 +131,36 @@ The table below summarizes the support level for each binding surface. ## Third-Party Integrations -Some framework integrations are maintained as patch sets against upstream -projects rather than as packages in this repository. +Some framework integrations are maintained as packages in this repository. Other +sample integrations are maintained as patch sets against upstream projects. ### Public API-based Integrations -Some integrations can be implemented using public APIs without patching. Currently the Python based integrations are located under `python/nemo_flow/integrations/` with their own README files and test suites. + +Some integrations can be implemented using public APIs without patching. Public +API-based integrations live under language-specific integration packages such as +`python/nemo_flow/integrations/` and `integrations/`. + +The OpenClaw observability plugin is available under `integrations/openclaw/` +and uses OpenClaw public plugin hooks to export NeMo Flow telemetry. See the +[OpenClaw Plugin Guide](docs/integrate-frameworks/openclaw-plugin.md). ### Patch-based Integrations + Use [third_party/README.md](third_party/README.md) for the clone, checkout, and patch-application workflow for those integrations. ### Support Matrix -The following table summarizes maintained third-party patch integrations and whether each provides observability, request intercepts, execution intercepts, and conditional execution. +The following table summarizes maintained third-party integrations and whether each provides observability, request intercepts, execution intercepts, and conditional execution. | Integration | Method | Observability | Request Intercepts | Execution Intercepts | Conditional Execution | |---|---|---|---|---|---| | [LangChain](third_party/README-langchain.md), [LangGraph](third_party/README-langgraph.md), [LangChain NVIDIA](third_party/README-langchain-nvidia.md) | 🚧 Patch | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | | [opencode](third_party/README-opencode.md) | 🚧 Patch | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | -| [OpenClaw](third_party/README-openclaw.md) | 🚧 Patch | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | +| [OpenClaw](docs/integrate-frameworks/openclaw-plugin.md) | `nemo-flow-openclaw` plugin | ✅ Yes | ❌ No | ❌ No | ❌ No | | [Hermes Agent](third_party/README-hermes-agent.md) | 🚧 Patch | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | -All patches offer experimental support. Our roadmap includes switching over to first-party plugins and packages. +Patch-based integrations offer experimental support. Our roadmap includes switching over to first-party plugins and packages where upstream extension points allow it. ## Roadmap diff --git a/docs/index.md b/docs/index.md index 75c3eab..9b2a0a9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -162,6 +162,7 @@ Basic Guide: Adding Scopes Basic Guide: Wrap Tool Calls Basic Guide: Wrap LLM Calls Advanced Guide: Coding-Agent Gateway +OpenClaw Plugin Guide Claude Code Gateway Guide Codex Gateway Guide Cursor Gateway Guide diff --git a/docs/integrate-frameworks/about.md b/docs/integrate-frameworks/about.md index efc3284..7f475fd 100644 --- a/docs/integrate-frameworks/about.md +++ b/docs/integrate-frameworks/about.md @@ -38,6 +38,7 @@ Use these guide links to move from the overview into task-specific instructions. - [Basic Guide: Wrap Tool Calls](wrap-tool-calls.md) explains where to place managed tool wrappers and tool lifecycle fallbacks. - [Basic Guide: Wrap LLM Calls](wrap-llm-calls.md) explains where to place managed provider wrappers, model names, streaming behavior, and LLM lifecycle fallbacks. - [Advanced Guide: Coding-Agent Gateway](coding-agent-gateway.md) describes the Rust gateway for observing Codex, Claude Code, Cursor, and Hermes through canonical hooks plus a passthrough LLM gateway. +- [OpenClaw Plugin Guide](openclaw-plugin.md) covers configuring the OpenClaw plugin, mapping OpenClaw hooks to NeMo Flow telemetry, and understanding current LLM replay fidelity boundaries. - [Claude Code Gateway Guide](coding-agent-claude-code.md) covers transparent Claude Code runs, Anthropic gateway routing, ATIF verification, and unsupported Claude application modes. - [Codex Gateway Guide](coding-agent-codex.md) covers transparent Codex CLI runs, local GUI/app caveats, model provider routing, and remote-task limits. - [Cursor Gateway Guide](coding-agent-cursor.md) covers transparent Cursor runs, temporary hook patching, GUI and CLI smoke tests, and gateway routing limits. diff --git a/docs/integrate-frameworks/code-examples.md b/docs/integrate-frameworks/code-examples.md index 65381d5..d04cacb 100644 --- a/docs/integrate-frameworks/code-examples.md +++ b/docs/integrate-frameworks/code-examples.md @@ -285,9 +285,13 @@ event( :::: -## Sample Third-Party Integrations +## Sample Third-Party Patch Integrations -NeMo Flow keeps sample third-party integrations as patch sets under `patches/` and pinned upstream checkouts under `third_party/`. +NeMo Flow keeps sample third-party integrations as patch sets under `patches/` +and pinned upstream checkouts under `third_party/`. For the current OpenClaw +end-user integration, use the [OpenClaw Plugin Guide](openclaw-plugin.md). + +The following table lists maintained patch checkouts: | Integration | Upstream Checkout | |---|---| @@ -295,7 +299,7 @@ NeMo Flow keeps sample third-party integrations as patch sets under `patches/` a | LangChain | `third_party/langchain` | | LangChain NVIDIA | `third_party/langchain-nvidia` | | LangGraph | `third_party/langgraph` | -| OpenClaw | `third_party/openclaw` | +| OpenClaw (legacy patch) | `third_party/openclaw` | | opencode | `third_party/opencode` | ## Quickstart: Apply Maintained Patches @@ -303,7 +307,6 @@ NeMo Flow keeps sample third-party integrations as patch sets under `patches/` a From the repository root, use the wrapper scripts when you want the maintained NeMo Flow patches applied to the pinned third-party checkouts: - | Script | Purpose | |---|---| | `./scripts/bootstrap-third-party.sh` | Clone pinned third-party upstream checkouts from `third_party/sources.lock`. | diff --git a/docs/integrate-frameworks/openclaw-plugin.md b/docs/integrate-frameworks/openclaw-plugin.md new file mode 100644 index 0000000..ac5b910 --- /dev/null +++ b/docs/integrate-frameworks/openclaw-plugin.md @@ -0,0 +1,275 @@ + + +# OpenClaw Plugin Guide + +Use the OpenClaw plugin when OpenClaw owns the agent, tool, and LLM lifecycle +that needs NeMo Flow observability. The plugin observes supported OpenClaw +plugin hooks and converts them into NeMo Flow sessions, LLM spans, tool spans, +marks, ATIF JSON, OpenTelemetry spans, and OpenInference/Phoenix spans. + +Use this guide to install the plugin, enable it in OpenClaw, configure telemetry +outputs, verify exported traces, and understand current LLM replay fidelity. + +## Requirements + +- OpenClaw `2026.5.6` or newer. +- The OpenClaw CLI available as `openclaw`. +- Node.js and npm when installing or managing the package directly. +- A Phoenix instance or OTLP collector when exporting OpenInference or + OpenTelemetry spans. + +## Install + +Install the plugin with OpenClaw so OpenClaw can register and manage it: + +```bash +openclaw plugins install npm:nemo-flow-openclaw +openclaw gateway restart +``` + +OpenClaw uses the package `nemo-flow-openclaw` for installation and the plugin +manifest id `nemo-flow` for configuration. Use `nemo-flow` in +`plugins.allow`, `plugins.entries`, `plugins inspect`, and gateway status +commands. + +If you manage OpenClaw plugin dependencies directly in a Node.js project, +install the package with npm: + +```bash +npm install nemo-flow-openclaw +``` + +Installing with npm makes the package available to that project. Use +`openclaw plugins install` when you want OpenClaw to register and manage the +plugin. + +## Enable the Plugin + +Allow the `nemo-flow` plugin id and grant conversation hook access when OpenClaw +runs with restrictive plugin settings: + +```json +{ + "plugins": { + "allow": ["nemo-flow"], + "entries": { + "nemo-flow": { + "enabled": true, + "hooks": { + "allowConversationAccess": true + }, + "config": {} + } + } + } +} +``` + +`plugins.allow` controls plugin trust and loading. +`hooks.allowConversationAccess` lets trusted non-bundled plugins receive +conversation-sensitive hook payloads such as LLM prompts, LLM responses, agent +finalization messages, and tool payloads. + +Restart the gateway after changing plugin configuration: + +```bash +openclaw gateway restart +``` + +## Configure Outputs + +Plugin configuration is passed through +`plugins.entries["nemo-flow"].config`. + +ATIF export is enabled by default. OpenTelemetry and OpenInference subscribers +are disabled until explicitly configured. + +Use the following configuration for ATIF-only local export: + +```json +{ + "atif": { + "enabled": true, + "outputDir": "./nemo-flow-atif" + }, + "telemetry": { + "otel": { + "enabled": false + }, + "openInference": { + "enabled": false + } + } +} +``` + +Use the following configuration for OpenTelemetry OTLP export: + +```json +{ + "telemetry": { + "otel": { + "enabled": true, + "transport": "http_binary", + "endpoint": "http://localhost:4318/v1/traces", + "serviceName": "openclaw-nemo-flow" + } + } +} +``` + +Use the following configuration for OpenInference/Phoenix OTLP export: + +```json +{ + "telemetry": { + "openInference": { + "enabled": true, + "transport": "http_binary", + "endpoint": "http://localhost:6006/v1/traces", + "serviceName": "openclaw-nemo-flow" + } + } +} +``` + +Privacy defaults capture prompts and responses, and strip tool arguments and +tool results: + +```json +{ + "capture": { + "includePrompts": true, + "includeResponses": true, + "stripToolArgs": true, + "stripToolResults": true + } +} +``` + +Tool arguments and tool results are stripped by default because they often +contain user data, local paths, tokens, or large payloads. + +Correlation uses bounded in-memory records. By default, the plugin waits 250 ms +for a matching `llm_input` after an `llm_output`, keeps correlation records for +600 seconds, and keeps at most 32 records per correlation key. + +## Verify the Integration + +Inspect the plugin runtime: + +```bash +openclaw plugins inspect nemo-flow --runtime --json +``` + +This verifies that the plugin package is installed, enabled, importable, and +exposes its config schema. It does not prove that every hook and gateway method +surface is active in a running gateway. + +Run an OpenClaw session with the plugin enabled, then verify the configured +sink: + +- ATIF: confirm JSON files appear in the configured `atif.outputDir`. +- OpenTelemetry: confirm spans arrive at the configured OTLP collector. +- OpenInference: confirm spans arrive at the configured OpenInference/Phoenix + endpoint. + +The plugin also registers the `operator.admin` scoped gateway method +`nemoFlow.status`. If your CLI is already paired with admin-capable gateway +access, run: + +```bash +openclaw gateway call nemoFlow.status --json +``` + +Otherwise, pass your normal admin-capable gateway auth options: + +```bash +openclaw gateway call nemoFlow.status --token "$OPENCLAW_GATEWAY_TOKEN" --json +``` + +If OpenClaw requests a device scope upgrade for `operator.admin`, approve it +through the normal OpenClaw device approval flow and retry the status call. + +The status response reports backend state, output health for `atif`, `otel`, +and `openInference`, replay counters, and the latest degraded or unavailable +reason when present. + +## Runtime Mapping + +The plugin maps supported OpenClaw hook events into NeMo Flow telemetry without +changing OpenClaw execution behavior. + +| OpenClaw hook | NeMo Flow behavior | +| --- | --- | +| `gateway_start` | Touches the replay backend early; session roots still open lazily from session-scoped hooks. | +| `gateway_stop` | Drains open sessions, shuts down subscribers, and clears the NeMo Flow plugin host. | +| `session_start` | Opens or aliases a NeMo Flow session scope. | +| `session_end` | Closes the session, flushes pending replay state, and exports ATIF if enabled. | +| `model_call_started` / `model_call_ended` | Records provider timing for later LLM span correlation. | +| `llm_input` / `llm_output` | Replays direct LLM spans when request and response hooks can be paired safely. | +| `before_message_write` | Records assistant turns for ordered LLM replay when provider timing can be paired later. | +| `after_tool_call` | Replays successful tool calls as tool spans; blocked tools emit marks. | +| `agent_end` | Emits an agent lifecycle mark, flushes recorded assistant-turn LLM spans, and preserves the final assistant answer as the session output. | +| `before_agent_finalize` | Preserves the last assistant message as fallback session output and emits a lifecycle mark without mutating the finalization payload. | +| `subagent_spawned` / `subagent_ended` | Emits subagent lifecycle marks under the best available parent or child session. | + +## LLM Replay Fidelity + +OpenClaw currently exposes request, response, message-write, and provider +timing details through separate hook events. The plugin correlates those events +within the same session, provider, model, and run. + +When model timing cannot be safely paired with an assistant turn, the plugin +emits diagnostic marks instead of inventing latency. This keeps traces honest +and makes current fidelity boundaries explicit. + +When OpenClaw provides usage data, the plugin maps input, output, total, cache +read, cache write, and cost fields into OpenInference-friendly usage fields. + +## Troubleshooting + +If the plugin does not load: + +- verify the package was installed with `openclaw plugins install` +- verify `plugins.allow` includes `nemo-flow` +- verify `plugins.entries["nemo-flow"].enabled` is not disabled +- restart the gateway after config changes + +If conversation payloads are missing: + +- verify `hooks.allowConversationAccess` is enabled for the plugin +- verify the OpenClaw session emits the relevant LLM, message-write, and tool + hooks + +If tool spans exist but LLM spans are incomplete: + +- verify `llm_input` and `llm_output` hooks are emitted +- verify `before_message_write` hooks are emitted when relying on assistant-turn + replay +- verify `model_call_started` and `model_call_ended` hooks are emitted when + timing attribution is expected +- check diagnostic marks for ambiguous or unpaired timing records + +If no export output appears: + +- verify `atif.outputDir`, `telemetry.otel.endpoint`, or + `telemetry.openInference.endpoint` +- verify the configured collector or output directory is reachable +- verify session end or gateway stop hooks fired so pending replay state can + drain + +If ambiguous timing marks appear, treat them as expected conservative behavior. +The plugin avoids attaching unsafe latency when multiple timing candidates could +match the same assistant turn. + +## Known Limitations + +Current OpenClaw public hooks are separate event streams, so some LLM timing +attribution is best-effort. If a matching request hook is missing, the plugin +may replay an LLM output with a placeholder request after the configured grace +window. If timing is ambiguous, the plugin emits diagnostic marks instead of +unsafe latency. diff --git a/integrations/openclaw/README.md b/integrations/openclaw/README.md index b9b63c5..58a29f1 100644 --- a/integrations/openclaw/README.md +++ b/integrations/openclaw/README.md @@ -3,51 +3,54 @@ SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: Apache-2.0 --> -# NeMo Flow OpenClaw Observability +# nemo-flow-openclaw -This package provides the `nemo-flow` OpenClaw plugin. It converts supported -OpenClaw hook events into NeMo Flow sessions, LLM spans, tool spans, lifecycle -marks, ATIF JSON, OpenTelemetry spans, and OpenInference/Phoenix spans. +`nemo-flow-openclaw` is the NeMo Flow observability plugin package for +OpenClaw. It converts supported OpenClaw hook events into NeMo Flow sessions, +LLM spans, tool spans, lifecycle marks, ATIF JSON, OpenTelemetry spans, and +OpenInference/Phoenix spans. -The package declares both OpenClaw entrypoint styles: +## Why Use It? -- `openclaw.extensions`: `./index.ts` for source-based plugin workflows. -- `openclaw.runtimeExtensions`: `./dist/index.js` for built runtime workflows. +- Observe OpenClaw sessions without patching OpenClaw. +- Export OpenClaw activity into NeMo Flow observability formats. +- Preserve OpenClaw's agent, tool, and LLM lifecycle context where public hooks + expose enough data. +- Keep ambiguous LLM timing attribution visible through diagnostic marks instead + of unsafe latency. -## Build and Validate +## What You Get -Run these commands from the repository root: - -```bash -npm ci --ignore-scripts -npm run build --workspace=nemo-flow-openclaw -npm run typecheck --workspace=nemo-flow-openclaw -npm test --workspace=nemo-flow-openclaw -``` +- OpenClaw plugin id `nemo-flow`. +- ATIF JSON export enabled by default. +- Optional OpenTelemetry OTLP export. +- Optional OpenInference/Phoenix OTLP export. +- Bounded LLM replay correlation across supported OpenClaw hooks. +- Tool span replay with conservative privacy defaults. +- Admin-scoped `nemoFlow.status` gateway health method. -The CI-equivalent repo recipe is `just --set ci true test-openclaw`. +## Installation -Optional package payload check: +Install the package directly in a Node.js/OpenClaw environment: ```bash -npm run pack:check --workspace=nemo-flow-openclaw +npm install nemo-flow-openclaw ``` -`npm run build --workspace=nemo-flow-openclaw` emits production files under -`integrations/openclaw/dist/`. Tests compile to `integrations/openclaw/.test-dist/` -so test artifacts do not enter the installable package. - -The optional live smoke test requires a working installed `nemo-flow-node` -binding: +For OpenClaw-managed installation, use the OpenClaw CLI: ```bash -npm run test:live --workspace=nemo-flow-openclaw +openclaw plugins install npm:nemo-flow-openclaw +openclaw gateway restart ``` -## Enablement +OpenClaw uses the package `nemo-flow-openclaw` for installation and the plugin +manifest id `nemo-flow` for configuration. -Allow the plugin id and grant conversation hook access when OpenClaw runs with a -restrictive plugin configuration: +## Getting Started + +Enable the `nemo-flow` plugin id and grant conversation hook access when +OpenClaw runs with restrictive plugin settings: ```json { @@ -66,133 +69,45 @@ restrictive plugin configuration: } ``` -`plugins.allow` controls plugin trust and loading. `hooks.allowConversationAccess` -lets trusted non-bundled plugins receive conversation-sensitive hook payloads such -as LLM prompts, LLM responses, agent finalization messages, and tool payloads. +Plugin configuration lives under `plugins.entries["nemo-flow"].config`. -## Configuration +## Documentation -ATIF export is enabled by default. OTel and OpenInference subscribers are disabled -until explicitly configured. The snippets below are values for -`plugins.entries["nemo-flow"].config`. +For full configuration, verification, troubleshooting, and current LLM replay +fidelity limits, use the +[OpenClaw Plugin Guide](../../docs/integrate-frameworks/openclaw-plugin.md). -ATIF-only local export: +## Development -```json -{ - "atif": { - "enabled": true, - "outputDir": "./nemo-flow-atif" - }, - "telemetry": { - "otel": { - "enabled": false - }, - "openInference": { - "enabled": false - } - } -} -``` - -OpenTelemetry OTLP export: +Run these commands from the repository root: -```json -{ - "telemetry": { - "otel": { - "enabled": true, - "transport": "http_binary", - "endpoint": "http://localhost:4318/v1/traces", - "serviceName": "openclaw-nemo-flow" - } - } -} +```bash +npm ci --ignore-scripts +npm run build --workspace=nemo-flow-openclaw +npm run typecheck --workspace=nemo-flow-openclaw +npm test --workspace=nemo-flow-openclaw ``` -OpenInference/Phoenix OTLP export: +The CI-equivalent repo recipe is: -```json -{ - "telemetry": { - "openInference": { - "enabled": true, - "transport": "http_binary", - "endpoint": "http://localhost:6006/v1/traces", - "serviceName": "openclaw-nemo-flow" - } - } -} +```bash +just --set ci true test-openclaw ``` -Privacy defaults: +Check the package payload before changing package metadata or entrypoints: -```json -{ - "capture": { - "includePrompts": true, - "includeResponses": true, - "stripToolArgs": true, - "stripToolResults": true - } -} +```bash +npm run pack:check --workspace=nemo-flow-openclaw ``` -Prompts and responses are captured by default. Tool arguments and tool results are -stripped by default because they often contain user data, local paths, tokens, or -large payloads. - -## Hook Mapping - -| OpenClaw hook | NeMo Flow behavior | -| --- | --- | -| `gateway_start` | Touches the replay backend early; session roots still open lazily from session-scoped hooks. | -| `gateway_stop` | Drains open sessions, shuts down subscribers, and clears the NeMo Flow plugin host. | -| `session_start` | Opens or aliases a NeMo Flow session scope. | -| `session_end` | Closes the session, flushes pending replay state, and exports ATIF if enabled. | -| `model_call_started` / `model_call_ended` | Records provider timing for later LLM span correlation. | -| `llm_input` / `llm_output` | Replays direct LLM spans when request and response hooks can be paired safely. | -| `before_message_write` | Records assistant turns for ordered LLM replay when provider timing can be paired later. | -| `after_tool_call` | Replays successful tool calls as tool spans; blocked tools emit marks. | -| `agent_end` | Emits an agent lifecycle mark, flushes recorded assistant-turn LLM spans, and preserves the final assistant answer as the session output. | -| `before_agent_finalize` | Preserves the last assistant message as fallback session output and emits a lifecycle mark without mutating the finalization payload. | -| `subagent_spawned` / `subagent_ended` | Emits subagent lifecycle marks under the best available parent or child session. | - -For LLM spans, OpenClaw currently exposes request, response, message-write, and -provider-timing details through separate hook events. The plugin correlates those -events within the same session, provider, model, and run. When model timing cannot -be safely paired with an assistant turn, the plugin emits diagnostic marks instead -of inventing latency. - -The OpenClaw hook payload and context types used by this plugin are represented -by narrow structural aliases in `src/openclaw-hook-types.ts`. Replace those -aliases with package imports if OpenClaw publishes the relevant hook contract -types through a stable public subpath. - -## Health - -The plugin registers the admin-scoped gateway method `nemoFlow.status`. - -The response reports: - -- backend status: `not_initialized`, `disabled`, `ready`, `degraded`, `stopping`, or `stopped` -- output health for `atif`, `otel`, and `openInference` -- replay counters, including replayed LLM spans, replayed tool spans, emitted - marks, ATIF files written, replay errors, and skipped events -- last degraded or unavailable reason when present - -Use the output health independently: - -- ATIF: confirm JSON files appear in the configured `atif.outputDir`. -- OTel: confirm spans arrive at the configured OTLP collector. -- OpenInference: confirm spans arrive at the configured OpenInference/Phoenix endpoint. - -## Packaging +`npm run build --workspace=nemo-flow-openclaw` emits production files under +`integrations/openclaw/dist/`. Tests compile to +`integrations/openclaw/.test-dist/` so test artifacts do not enter the +installable package. -`npm run pack:check --workspace=nemo-flow-openclaw` builds a fresh production -`dist/`, runs `npm pack --dry-run`, and verifies that: +The optional live smoke test requires a working installed `nemo-flow-node` +binding: -- declared OpenClaw source and runtime entrypoints are present -- production source files needed by `index.ts` are present -- compiled tests and `.test-dist/` files are absent -- packed `dist/**` matches the fresh production build +```bash +npm run test:live --workspace=nemo-flow-openclaw +```