fix(benchmark): make prompt cache instantiation atomic - #729
Draft
RapidPoseidon wants to merge 1 commit into
Draft
fix(benchmark): make prompt cache instantiation atomic#729RapidPoseidon wants to merge 1 commit into
RapidPoseidon wants to merge 1 commit into
Conversation
__instantiate_prompts reset the five prompt caches and appended to them in place while paginating. Any failure partway through — a 503 on a later page, a schema-drift parse error on one item — left the caches partially populated, and every cache getter treats a non-empty list as a complete, authoritative set. The result was a silently truncated benchmark: e.g. .prompts raising, then .identifiers returning a single entry instead of re-fetching or raising, which later surfaced as a misleading "all identifiers must be registered" error rather than the real fetch failure. Accumulate into locals and assign to self only after every page has been fetched and parsed, so the caches are all-or-nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: mads <37117211+Makuh17@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Investigating a report that
benchmark.prompts/.identifiers/.tagsbroke in SDK 3.17.2 with:Primary root cause (already fixed on
main, not this PR): backend PR RapidataAI/rapidata-backend#4879feat(leaderboard): prompt tags and origin(merged 2026-07-27) deliberately changedGET /benchmark/{id}/promptsto returntagsas{value, category}objects (plus a neworiginfield). The OpenAPI regen (#725) already updated the model toList[Tag]and__instantiate_promptsalready extractstag.value. That fix just needs a PyPI release — the latest published version (v3.17.2, 2026-07-22) predates the backend change. This PR does not address the release; it fixes an independent latent bug the report surfaced along the way.What this PR fixes (report's "Secondary issue 1")
__instantiate_promptsreset the five prompt caches and appended to them in place while paginating. Any failure partway through — a 503 on a later page (the report saw intermittent 503s), or a parse error on a single item — left the caches partially populated. Because every cache getter (prompts,identifiers,tags, …) treats a non-empty list as a complete, authoritative set, the next access returned silently-truncated data instead of re-fetching or raising.Observed in the report:
.promptsraised, then.identifiersreturned 1 entry rather than re-instantiating — which downstream surfaced as a misleading"All identifiers/prompts must be in the registered ... list"instead of the real fetch failure.Fix: accumulate into local lists and assign to
selfonly after every page has been fetched and parsed cleanly — the caches are now all-or-nothing.Not in this PR (deliberately)
Release and Publish(patch → 3.17.3). Publishing to PyPI is outside an agent's remit.add_model(prompts=...)validates against the wrong list. Confirmed a real latent bug: whenprompts=is passed, it is validated againstself.identifiers(line ~619) and used as the upload identifier, butidentifierandoriginal_promptare distinct fields (identifiers default to prompt text only whenadd_promptswas called without explicitidentifiers). For a benchmark with distinct identifiers, validprompts=input is spuriously rejected, and the complete fix must also translate prompt→identifier before upload (samples are matched server-side by identifier). That's a behavioral change with edge cases (duplicate/None prompt text) best decided by the owner, so it's left out of this focused PR — happy to follow up.Testing
uv run blackclean;uv run pyright src/rapidata/rapidata_client→ 0 errors.🔗 Session: https://poseidon.rapidata.internal/chat/session-f1ef50bc
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.