fix(runner): keep the run credential when the public api base is unconfigured - #6281
fix(runner): keep the run credential when the public api base is unconfigured#6281mmabrouk wants to merge 1 commit into
Conversation
…nfigured A self-hosted deployment on a public hostname failed every agent run with "record log is unreadable; cannot rebuild the conversation", because the runner was sending its API callbacks with no Authorization header at all. platformCredentialForRequest forwards the run's credential to platform calls only when it recognizes the run's OTLP trace endpoint as this deployment's own API. That match is a string comparison against AGENTA_API_INTERNAL_URL, AGENTA_API_URL and the cloud base. A dispatched run carries the endpoint the API built from its PUBLIC base, so a runner configured with only its internal hop matches nothing, drops the credential, and every call to /sessions/* is rejected with 401. Records are never written and never read back, and the history rebuild reports it far from the cause. The attribution is only decidable once the runner knows the public base, so arm the strict check only when it is configured. Without it the runner keeps the credential (the behavior before the check existed) and says what to set; with it, the third-party collector protection stays exactly as it was. Also name the two empty-credential cases apart in the stream diagnostic, warn at startup when AGENTA_API_URL is unset, and document the variable as an identity input rather than only a routing fallback.
dosubot
Bot
added
the
size:L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
@coderabbitai review |
|
Important Review skippedNo new commits to review since the last review. ⚙️ Run configurationConfiguration used: Repository YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe runner now identifies platform ingest endpoints using configured API bases. It controls credential forwarding for non-ingest endpoints, adds startup and stream diagnostics, and documents ChangesRunner credential attribution
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: 🔵 Low · up to The fix restores runner authentication when the public API base is unset, but the troubleshooting guidance is currently misleading and diagnostic logs may expose credentials embedded in OTLP URLs; merge is reasonable with explicit follow-up to correct the documentation and redact logged endpoints. Sequence Diagram(s)sequenceDiagram
participant RunnerServer
participant CredentialPolicy
participant TraceEndpoint
RunnerServer->>CredentialPolicy: Evaluate request endpoint and run credential
CredentialPolicy->>CredentialPolicy: Match endpoint against configured ingest bases
CredentialPolicy->>TraceEndpoint: Forward or remove credential
RunnerServer-->>RunnerServer: Record credential diagnostics
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 77.78% which is sufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 5 files. (4 skipped: 4 unsupported.) ✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 6218d8d0-33a9-403d-ba6f-55421a502e41
📒 Files selected for processing (9)
docs/docs/self-host/deploy/01-deploy-remotely.mdxdocs/docs/self-host/reference/01-configuration.mdxhosting/docker-compose/oss/env.oss.dev.examplehosting/docker-compose/oss/env.oss.gh.exampleservices/runner/src/engines/sandbox_agent/runtime-policy.tsservices/runner/src/server.tsservices/runner/src/tracing/otel.tsservices/runner/tests/unit/platform-credential-attribution.test.tsservices/runner/tests/unit/server.test.ts
Included review availability: Your plan provides up to 8 included reviews per hour; 4 remain after this review.
| ### Agent Runs Fail With `record log is unreadable; cannot rebuild the conversation` | ||
|
|
||
| This applies if the deployment answers on a public hostname and agent runs fail while the rest of | ||
| the app works. Other symptoms of the same cause: an agent that forgets earlier turns, or runner logs | ||
| showing `cred=DROPPED(endpoint-not-agenta-ingest)` and `HTTP 401` on `/sessions/records/ingest`, | ||
| `/sessions/records/query`, and `/sessions/streams/heartbeat`. | ||
|
|
||
| **Cause:** The `runner` service has no `AGENTA_API_URL`. Each run carries the trace endpoint the API | ||
| built from its public base, for example `https://agenta.example.com/api/otlp/v1/traces`, while the | ||
| runner knows only its internal hop, `http://api:8000`. The runner cannot tell that endpoint apart | ||
| from a third-party OTLP collector, so it withholds the run's credential and every callback to the | ||
| API is rejected. Conversation history is never written and never read back. | ||
|
|
||
| **Solution:** Set `AGENTA_API_URL` to the same public API base the `api` and `services` containers | ||
| use, and recreate the runner: | ||
|
|
||
| ```bash | ||
| docker compose up -d --force-recreate runner | ||
| ``` | ||
|
|
||
| Compose reads `AGENTA_API_URL` from the shell (or `--env-file`) when it builds the runner's | ||
| environment, not from a service's `env_file`. Setting it only in a file that is passed as `env_file` | ||
| reaches the API and services containers but not the runner. Confirm it landed: | ||
|
|
||
| ```bash | ||
| docker compose exec runner printenv AGENTA_API_URL | ||
| ``` | ||
|
|
||
| Sessions that ran while this was broken keep the gaps in their history. New sessions are unaffected. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Update the troubleshooting cause for the new fallback behavior.
When AGENTA_API_URL is unset, platformCredentialForRequest keeps the credential for platform calls. It does not withhold it and cause the stated HTTP 401 failures. The runner warns because it cannot prevent a third-party collector credential from reaching platform calls. Describe a missing or mismatched configured public base as the cause only when strict attribution drops a credential.
| `[sessions] WARNING: trace endpoint ${endpoint} matches no configured Agenta ingest ` + | ||
| `base (${configuredIngestBases().join(", ")}), and AGENTA_API_URL is not set, so the ` + | ||
| `run credential cannot be attributed. Using it for platform calls anyway. Set ` + | ||
| `AGENTA_API_URL to this deployment's public api base (e.g. https://<host>/api) to ` + | ||
| `attribute it properly and to keep third-party collector credentials out of platform calls.`, |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Redact the trace endpoint before writing it to stderr.
These warnings log the raw endpoint. An OTLP endpoint can contain credentials in URL userinfo or query values. The runner then stores those credentials in its logs. Log a sanitized origin, or redact all URL credential-bearing components before interpolation.
Proposed fix
+function endpointForLog(endpoint: string): string {
+ try {
+ return new URL(endpoint).origin;
+ } catch {
+ return "<invalid endpoint>";
+ }
+}
+
- `[sessions] WARNING: trace endpoint ${endpoint} matches no configured Agenta ingest `
+ `[sessions] WARNING: trace endpoint ${endpointForLog(endpoint)} matches no configured Agenta ingest `
- `[sessions] trace endpoint ${endpoint} is not Agenta ingest `
+ `[sessions] trace endpoint ${endpointForLog(endpoint)} is not Agenta ingest `Also applies to: 76-80
|
Railway Preview Environment
|
1 participant
The symptom
Every agent run on a self-hosted v0.114.0 deployment fails with:
The rest of the app works. The error names the conversation record log, but the record log is not
the problem: the runner is calling the API with no Authorization header at all, so every
/sessions/*call comes back401.Before, on a deployment reachable at
https://agenta.example.com:After, same deployment, same configuration:
Why it happens
platformCredentialForRequestdecides whether the run's Authorization header may authenticate therunner's own calls back to the API. One header on the wire has two possible owners: this platform,
or a third-party OTLP collector the caller aimed the run at. Sending a collector's token to the
Agenta API would leak it, so
da83e2b80b(new in v0.114.0) started gating the credential onisAgentaIngest(endpoint).That gate is a string comparison against
AGENTA_API_INTERNAL_URL,AGENTA_API_URL, and the cloudbase. The two sides never line up on a normal self-hosted deployment:
servicescontainer builds the run's trace endpoint from its own public base, so the runcarries
https://agenta.example.com/api/otlp/v1/traces.runneris configured with the in-network hop it actually calls,http://api:8000.AGENTA_API_URLis what closes that gap, and the Compose files default it to empty(
AGENTA_API_URL: ${AGENTA_API_URL:-}, filled by shell interpolation). A deployment that neverexported it matches nothing, drops the credential, and fails 401 on every session call. Nothing is
written to the record log and nothing can be read back, which is why the failure surfaces four steps
away from its cause.
Kubernetes was never affected: the chart sets
AGENTA_API_URLon the runner pod unconditionally.The fix
The attribution is only decidable once the runner knows the public base, because the internal
hop never appears in a dispatched run. So arm the strict check only when the operator has supplied
the input that makes it correct:
AGENTA_API_URLset: unchanged. A non-matching endpoint really is someone else's collector, andthe credential is dropped exactly as it is today.
AGENTA_API_URLunset: keep the credential (the behavior before the check existed) and log onceper endpoint naming the variable to set.
The export path is untouched.
resolveRunOtlpTargetstill uses strictisAgentaIngest, so a runaimed at a third-party collector never receives the refreshable platform credential.
Two diagnostics, because the 30 minutes this cost were spent finding the cause, not fixing it:
cred=DROPPED(endpoint-not-agenta-ingest)(a credential arrived and did not attribute) from
cred=ABSENT(caller-sent-none). They havedifferent fixes.
AGENTA_API_URLis unset.Docs
AGENTA_API_URLwas documented only as a routing fallback forAGENTA_API_INTERNAL_URL, which iswhat made "set the internal one and you are done" look correct. The configuration reference now
gives it its own row and states the distinction: the internal URL is an address, the public one
is an identity. Adds a troubleshooting entry on the remote-deploy page keyed to the exact error
text, and comments the variable in both Compose example env files.
Tests
tests/unit/platform-credential-attribution.test.ts(new, 9 cases): both directions of theattribution, the warn-once behavior, and
publicApiBaseConfigured.tests/unit/server.test.ts: the existing third-party-collector case never setAGENTA_API_URL,so it was asserting the protection in the state where the runner cannot decide. It now configures
the public base, and a sibling case pins the self-hosted shape end to end at the server seam.
Full suite: 2338 passed.
tsc --noEmitclean. Docs build clean.Follow-up worth considering, not in this PR
The runner still cannot learn its own public base on its own. Having the API publish it, or having
the dispatcher mark the endpoint as platform-owned on the wire, would remove the configuration
requirement entirely. That is a design change and I did not want to slip it into a fix.
Related
Agenta-AI/agenta-skills#22adds the same failure to the self-hosting troubleshooting skill, whichstill applies to anyone on v0.114.0.