feat(gooddata-eval): add the agentic anomaly-detection evaluator

[Tomkess]

Third of three evaluators for skills that ship in the product but have no eval coverage. Siblings: #1798 (forecasting), #1799 (what-if analysis).

Why now

Probed live against micdiagnose-dev: the skill is enabled and completes in a single turn — search, build a monthly-spend chart, execute_anomaly_detection. Nothing evaluates it.

Honest about what can be checked

execute_anomaly_detection takes only visualization_ref and max_points. No threshold, no sensitivity, no expected anomaly — so unlike forecasting and what-if, the tool call itself carries almost no assertable intent.

What is assertable is the chart the detection ran on. The measure and the time granularity are the whole of "did it look at the right series", and getting either wrong makes the result meaningless however well the detection performed.

{"metric": "metric/spend", "granularity": "MONTH"}

The flagged-point count is reported but never asserted. Whether a real series contains anomalies is a property of the data, not of the agent — a fixture demanding some would start failing the day the warehouse refreshed. The live probe returned point_count: 0 and the agent correctly said so; that is a pass.

If the team would rather not add another kind whose signal is mostly "the chain ran" (the same limitation kda_skill carries, open in the eval repo since August), this is the one of the three to drop. Forecasting and what-if check real correctness; this one checks targeting.

A correction, and a much narrower real issue

An earlier revision of this PR claimed gen-ai's granularity inference was ambiguous for label/process_date.month, because date and month both map to a granularity. That was wrong, and the fault was in this module, not the service. gen-ai's token map has no date key:

_TOKEN_TO_GRANULARITY = {"hour": "HOUR", "day": "DAY", "week": "WEEK",
                         "week_us": "WEEK", "month": "MONTH",
                         "quarter": "QUARTER", "year": "YEAR"}

This module invented "date": "DAY" and dropped the service's "hour". Fixed in 5a57f66c — the map is now copied verbatim. Two consequences of the invented key, both real bugs in the evaluator:

Reference	Service	This module, before
label/process_date	matches nothing → the tool refuses the call	scored DAY — grading a run the service never ran
label/process_date.hour	HOUR	unrecognised

What survives is narrower and genuine. A reference naming two granularities does let set iteration order decide the service's answer — a snake_case first_day_quarter.month tokenizes to {first, day, quarter, month}, three of which map. The last-token rule stays for that reason: a scorer must not be a coin flip even where the thing it scores is one.

Scope, measured rather than asserted: 0 of 558 labels in the eval workspace name two granularities. This workspace's FirstDayQuarter.* labels are camelCase, so they tokenize to one word and are safe; the same attribute named first_day_quarter would not be. So it is latent, not live, and I would not file it upstream on this evidence alone.

Not included, on purpose

LoopExit / exit_reason — lands with #1789, still open.

Tests

23, including: extraction pairing a detection with the chart it followed, the ambiguous-label case above, a field token winning over a disagreeing filter (matching the service's order), finding no anomalies still passing, and a chat error on a later run not discarding the earlier one.

803 passed, lint and format clean.

Merge note

Touches the same three files as the siblings — cli/agentic_runner.py plus the _ALL_AGENTIC_KIND_CASES and _EVALUATE_FUNCS staleness guards. Whichever merges first, the others need a trivial rebase on those lists.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added support for evaluating agentic anomaly-detection scenarios.
  • Supports clarification exchanges, visualization and detection-result validation, repeated runs, and pass-rate aggregation.
  • Reports execution performance, including latency and cost information.
  • Provides detailed failure information when anomaly-detection evaluations do not pass.
  • Supports partial results and robust handling of interrupted or unsuccessful agent interactions.

[coderabbitai]

Warning

Review limit reached

• Run on-demand review

_{This review includes 2 billable files and costs up to $0.50.}

Or wait 40 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 28daaa08-9de0-455d-b492-484568bf49ab

📥 Commits

Reviewing files that changed from the base of the PR and between a3386e9 and 4515325.

📒 Files selected for processing (2)

• packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py
• packages/gooddata-eval/tests/test_trace_linker.py

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 364667f7-5ed5-4d69-a1c9-52dceda5d264

📥 Commits

Reviewing files that changed from the base of the PR and between 5a57f66 and a3386e9.

📒 Files selected for processing (1)

• packages/gooddata-eval/tests/test_agentic_anomaly_detection.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The PR adds the agentic_anomaly_detection test kind. It implements clarification handling, visualization and detection extraction, metric and granularity checks, repeated-run aggregation, trace reporting, assertion diagnostics, CLI dispatch, and automated coverage.

Changes

Agentic anomaly detection

Layer / File(s)	Summary
Anomaly contracts and extraction packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py, packages/gooddata-eval/tests/test_agentic_anomaly_detection.py	Defines evaluation results and extracts visualization, detection, metric, granularity, and anomaly point data. Tests cover extraction and granularity inference.
Multi-run execution and scoring packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py, packages/gooddata-eval/tests/test_agentic_anomaly_detection.py	Handles simulated clarification replies, multi-turn execution, partial results, scoring, and pass@K/pass^K aggregation.
Evaluation reporting and diagnostics packages/gooddata-eval/src/gooddata_eval/core/agentic/anomaly_detection.py, packages/gooddata-eval/tests/test_agentic_anomaly_detection.py	Adds assertion errors, detailed failure data, Langfuse scoring, quality reporting, latency details, and successful evaluation outcomes.
CLI registration and coverage packages/gooddata-eval/src/gooddata_eval/cli/agentic_runner.py, packages/gooddata-eval/tests/test_agentic_runner.py, packages/gooddata-eval/tests/test_trace_linker.py	Registers the new test kind, dispatches its evaluator, and extends parametrized runner and trace-linker coverage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant evaluate_agentic_anomaly_detection
  participant ChatClient
  participant AnomalyTools
  participant Langfuse
  CLI->>evaluate_agentic_anomaly_detection: dispatch anomaly evaluation
  evaluate_agentic_anomaly_detection->>ChatClient: run question and clarification turns
  ChatClient->>AnomalyTools: request visualization and anomaly detection
  AnomalyTools-->>evaluate_agentic_anomaly_detection: return tool traces
  evaluate_agentic_anomaly_detection->>Langfuse: record trace scores and quality data
  evaluate_agentic_anomaly_detection-->>CLI: return AgenticEvalOutcome

Loading

Merge Risk: ⚪ Minimal · up to a3386

The change adds anomaly-detection evaluation with clarification handling, scoring, diagnostics, and CLI registration. Current evidence indicates the behavior is covered and mergeable with normal checks.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 63 functions across 5 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding an agentic anomaly-detection evaluator to gooddata-eval.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

---

_{A rabbit reads each line,
The patch grows clear beneath the moon,
Small changes hop in place,
Tests guard the garden path,
Reviews bloom before the dawn.}

_{Comment @coderabbitai help to get the list of available commands.}

[Tomkess]

Review on the sibling PR #1798 surfaced two findings that are structural and applied here unchanged. Fixed in ``, before this PR was reviewed.

Tool calls were extracted from the current turn only. The agent asks a disambiguation question before building anything, so the create call and the execute call can land on different turns — reading a single turn dropped the object the execution actually ran on, and the evaluator then failed a correct run for having no content to check. Extraction now reads every turn accumulated so far. kda_skill does not have this bug despite the identical structure, because its create and execute always land in the same turn; I copied the shape without re-checking that assumption held for a skill that asks questions first.

Unasserted content checks were published to Langfuse as BOOLEAN 1. They are True internally so an unstated expectation cannot fail a run, but scoring that claims the evaluator verified something it never looked at. Only checks named in ev.asserted are scored now.

One test added, verified to fail against the previous version. Lint and format clean.

[codecov]

Codecov Report

❌ Patch coverage is 96.89922% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.75%. Comparing base (72858ca) to head (4515325).
⚠️ Report is 15 commits behind head on master.

Files with missing lines	Patch %	Lines
...rc/gooddata_eval/core/agentic/anomaly_detection.py	96.86%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1801      +/-   ##
==========================================
+ Coverage   82.27%   82.75%   +0.48%     
==========================================
  Files         282      325      +43     
  Lines       20326    20848     +522     
==========================================
+ Hits        16723    17253     +530     
+ Misses       3603     3595       -8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Tomkess]

Correction — the "service bug" I reported in this PR was not one, and the fault was mine.

I claimed gen-ai's granularity inference was ambiguous for label/process_date.month, on the grounds that date → DAY and month → MONTH both match. The service's token map has no date key at all:

_TOKEN_TO_GRANULARITY: dict[str, str] = {
    "hour": "HOUR", "day": "DAY", "week": "WEEK", "week_us": "WEEK",
    "month": "MONTH", "quarter": "QUARTER", "year": "YEAR",
}

This module invented "date": "DAY", and separately dropped the service's "hour". The ambiguity I described existed only in my own copy. Fixed in 5a57f66c; the map is now verbatim, and the PR body is corrected.

The invented key was also two real bugs in the evaluator, not just a wrong claim:

• label/process_date — a date attribute with no granularity suffix. The service matches nothing and the tool refuses the call with "Could not infer anomaly granularity". This module scored DAY, so it would have graded granularity for a run the service never performed.
• label/process_date.hour — unrecognised here, HOUR in the service.

What survives is real but much narrower. A reference naming two granularities does leave the service's answer to set iteration order — a snake_case first_day_quarter.month tokenizes to {first, day, quarter, month}. The last-token rule stays for that reason.

I measured the scope rather than assuming it this time: 0 of 558 labels in the eval workspace name two granularities. The FirstDayQuarter.* labels here are camelCase, so they tokenize to a single word and are safe — the same attribute named first_day_quarter would not be. Latent, not live, and not worth filing upstream on this evidence.

Worth stating plainly: I asserted a defect in someone else's service from a misread of my own code, and put it in a PR body and a commit message. The check that caught it was reading the service's map again instead of trusting my note about it.

805 passed, lint and format clean.