[bot] Merge master/78904a06 into rel/dev - #1710
Conversation
Adds an optional reasoning effort to `gooddata-eval`, so an evaluation run can request `LOW`, `MEDIUM` or `HIGH` instead of always using the endpoint default. Sent as `options.reasoningEffort` on each chat message and threaded through ChatClient, all seven `evaluate_agentic_*` entry points, the agentic runner and the CLI (`--reasoning-effort`). Per the AI chat OpenAPI schema the field is `SendMessageOptions.reasoningEffort`, typed as an enum of exactly LOW/MEDIUM/HIGH, nullable, and scoped to a single message rather than persisted. The implementation mirrors all four: uppercase `Literal`, camelCase key, re-sent per message, key omitted when unset. Without this there is no way to evaluate at a non-default effort — the value is read from the request body and the SDK never sent one, so effort was the one request-level option the suite could not vary. Both Langfuse writers record it. `build_run_context` suffixes the dataset-run name and adds run metadata; `LangfuseSink` suffixes its run name and reports the effort via trace `tags`, since metadata alone is not a breakdown dimension — the same reason `model_version` already uses first-class fields there. Two runs differing only by effort would otherwise share a name and merge in the report. The value is normalized once at the boundary. The `Literal` only constrains static callers, so without normalization a lowercase value reached the endpoint as an out-of-enum request, and a blank value was sent yet skipped by the writers' truthiness checks — leaving a run whose recorded identity disagreed with what it requested. Blank now means unset and anything outside the enum raises locally. New parameters are appended to the end of the public `evaluate_agentic_*` signatures rather than inserted mid-list, so existing positional callers of this released package keep binding correctly. Default is None, which omits the key and leaves the request payload byte-identical to before. Also drops an unused local in test_cli.py that was failing lint on master.
feat(gooddata-eval): let evaluations request a reasoning effort
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1710 +/- ##
===========================================
+ Coverage 78.30% 78.34% +0.04%
===========================================
Files 271 271
Lines 18689 18726 +37
===========================================
+ Hits 14634 14671 +37
Misses 4055 4055 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
2 participants
🚀 Automated PR to perform merge from master into rel/dev with changes up to 78904a0 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/30934376542).