You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The agent-fleet daily analytics pipeline (hack/analytics/analyze_runs.py) could never run against real OpenHands Cloud data: the real-data path exited 1 silently and produced no output, so the daily agent-analytics job always fell back to the bundled --sample (or failed outright). This PR fixes the real-data API path so the pipeline produces real daily_report.md / pr-gate verdicts going forward.
Root cause
CloudClient.conversation_events issued a single GET to /api/v1/conversation/{id}/events/search with limit=500. The endpoint caps limit at 100 per request and rejects larger values with HTTP 422:
{"detail":[{"type":"less_than_equal","loc":["query","limit"],"msg":"Input should be less than or equal to 100","input":"500"}]}
get() raises typer.Exit(1) on the 422, so the real-data run produced no output at all. The --sample path was unaffected because it never calls the API — which is exactly why the smoke test passed while real runs failed silently.
The one intervention
Page through the events endpoint instead of requesting 500 at once:
Add EVENTS_PAGE_SIZE = 100 (the API's per-request cap).
conversation_events now loops, requesting limit=min(EVENTS_PAGE_SIZE, remaining) and following the next_page_id cursor (passed as the page_id query param) until the requested total (limit, default 500) is reached or the server returns an empty next_page_id.
A short-page / non-dict response guard stops at the end.
Minimal, self-contained, single-file change (~20 lines), no behavioral change to the sample path.
Metric justifying the change
This is a pipeline-repair PR, not a failure-mode intervention. The metric is the failure rate of the real-data path itself:
Before fix: 100% of real-data runs failed (HTTP 422 → typer.Exit(1), no output). Verified empirically: limit=200 → 422; limit=100 → 200 OK.
After fix: real-data run succeeds (exit 0), paginating a 200-event conversation across 2 pages and a 396-event conversation across 4 pages.
pr-gate verdict:NOT-ACTIONABLE — 14 runs, 0 failed, 0 recurring error signatures. There is no recurring agent failure mode to intervene on today.
Per the agent-analytics job, a NOT-ACTIONABLE verdict means no failure-mode intervention PR. This PR is instead the pipeline repair that makes the pr-gate mechanism itself functional — without it, the real-data path can never produce a real verdict, so every future window would either fail or silently fall back to synthetic data. The pr-gate rule governs agent failure modes; this repair sits upstream of it.
Verification
Check
Result
uv run hack/analytics/analyze_runs.py --sample --out-dir /tmp/analytics-verify (ANALYTICS_NO_UPLOAD=1)
✅ only the documented pre-existing pkg/gitTestRepoClone* failures (this change does not touch pkg/git)
temp.sh upload
Skipped: ANALYTICS_NO_UPLOAD=1 was set for the offline sandbox verification run (both sample and real-data). No live upload attempted.
Visualization
failure_heatmap.png is generated by the pipeline at dist/analytics/failure_heatmap.png (real data, 14 runs, 0 failures → empty failure grid, consistent with the NOT-ACTIONABLE verdict).
Expected impact
The real-data analytics path goes from 100% failure to functional. The daily agent-analytics job will now produce real daily_report.md / pr-gate verdicts from live fleet data instead of silently failing or relying on the synthetic sample, so ACTIONABLE failure modes can actually be detected and intervened on in future windows.
This PR was created by an AI agent as part of an automated daily agent analytics job.
The OpenHands Cloud conversation-events endpoint caps `limit` at 100 per
request. `CloudClient.conversation_events` issued a single call with
`limit=500`, which the API rejects with HTTP 422 (`less_than_equal`), so
`get()` raised `typer.Exit(1)` and the real-data run produced no output.
The `--sample` path was unaffected because it never hits the API.
Page through the endpoint in batches of `EVENTS_PAGE_SIZE` (100) using the
`page_id` cursor returned as `next_page_id`, stopping at the requested
total or when the server signals the end. Verified against the live API:
a conversation with 200 events now fetches both pages (limit=200 422s,
limit=100 + page_id paginates to completion).
This is a pipeline-repair PR, not a failure-mode intervention: it makes
the real-data path functional so the pr-gate mechanism can ever produce a
real verdict. The post-fix real-data run for yesterday (2026-08-15) shows
14 runs, 0 failed, pr-gate NOT-ACTIONABLE — there is no recurring agent
failure mode today; the metric justifying this change is the 100% failure
rate of the real-data path itself before the fix.
This commit was created by an AI agent as part of an automated daily
agent analytics job.
AI Reviewer: run a review on demand. To trigger the first review automatically, go to your organization or repository integration settings. AI can make mistakes. Always validate suggestions.
TIP This summary will be updated as you push new changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The agent-fleet daily analytics pipeline (
hack/analytics/analyze_runs.py) could never run against real OpenHands Cloud data: the real-data path exited1silently and produced no output, so the dailyagent-analyticsjob always fell back to the bundled--sample(or failed outright). This PR fixes the real-data API path so the pipeline produces realdaily_report.md/pr-gateverdicts going forward.Root cause
CloudClient.conversation_eventsissued a single GET to/api/v1/conversation/{id}/events/searchwithlimit=500. The endpoint capslimitat 100 per request and rejects larger values with HTTP 422:{"detail":[{"type":"less_than_equal","loc":["query","limit"],"msg":"Input should be less than or equal to 100","input":"500"}]}get()raisestyper.Exit(1)on the 422, so the real-data run produced no output at all. The--samplepath was unaffected because it never calls the API — which is exactly why the smoke test passed while real runs failed silently.The one intervention
Page through the events endpoint instead of requesting 500 at once:
EVENTS_PAGE_SIZE = 100(the API's per-request cap).conversation_eventsnow loops, requestinglimit=min(EVENTS_PAGE_SIZE, remaining)and following thenext_page_idcursor (passed as thepage_idquery param) until the requested total (limit, default 500) is reached or the server returns an emptynext_page_id.Minimal, self-contained, single-file change (~20 lines), no behavioral change to the sample path.
Metric justifying the change
This is a pipeline-repair PR, not a failure-mode intervention. The metric is the failure rate of the real-data path itself:
typer.Exit(1), no output). Verified empirically:limit=200→ 422;limit=100→ 200 OK.pr-gate verdict
2026-08-15T00:00:00Z→2026-08-16T00:00:00Z(yesterday → today)openhands-cloud-api(real data,OPENHANDS_API_KEYavailable)Per the
agent-analyticsjob, a NOT-ACTIONABLE verdict means no failure-mode intervention PR. This PR is instead the pipeline repair that makes the pr-gate mechanism itself functional — without it, the real-data path can never produce a real verdict, so every future window would either fail or silently fall back to synthetic data. Thepr-gaterule governs agent failure modes; this repair sits upstream of it.Verification
uv run hack/analytics/analyze_runs.py --sample --out-dir /tmp/analytics-verify(ANALYTICS_NO_UPLOAD=1)uv run hack/analytics/analyze_runs.py --since yesterday --until today --out-dir dist/analytics(real data)uvx --from mypy mypy --strict --ignore-missing-imports --disable-error-code untyped-decorator hack/analytics/analyze_runs.pyuvx --from radon radon cc -smax complexityconversation_events= B (6)task cli:formattask cli:lint:citask cli:testpkg/gitTestRepoClone*failures (this change does not touchpkg/git)temp.sh upload
Skipped:
ANALYTICS_NO_UPLOAD=1was set for the offline sandbox verification run (both sample and real-data). No live upload attempted.Visualization
failure_heatmap.pngis generated by the pipeline atdist/analytics/failure_heatmap.png(real data, 14 runs, 0 failures → empty failure grid, consistent with the NOT-ACTIONABLE verdict).Expected impact
The real-data analytics path goes from 100% failure to functional. The daily
agent-analyticsjob will now produce realdaily_report.md/pr-gateverdicts from live fleet data instead of silently failing or relying on the synthetic sample, so ACTIONABLE failure modes can actually be detected and intervened on in future windows.This PR was created by an AI agent as part of an automated daily agent analytics job.