UN-3636 [DEV] Hermetic LLM mock hook for execute-path critical tests#2170
Draft
chandrasekharan-zipstack wants to merge 2 commits into
Draft
UN-3636 [DEV] Hermetic LLM mock hook for execute-path critical tests#2170chandrasekharan-zipstack wants to merge 2 commits into
chandrasekharan-zipstack wants to merge 2 commits into
Conversation
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Base automatically changed from
feat/un-3636-critical-path-integration-tests
to
main
July 14, 2026 10:31
Execute-path critical paths (workflow-create-execute, usage-token-tracking, etc.) need a deterministic LLM without real provider secrets. Add a single sdk1 escape hatch: when UNSTRACT_LLM_MOCK_RESPONSE is set, inject litellm's mock_response so completions return that string with fixed usage (10/20/30), tunable via DEFAULT_MOCK_RESPONSE_* and error sentinels like "litellm.RateLimitError". No-op in production (env unset). Wire the env through worker-executor-v2 / worker-file-processing-v2 in the e2e compose overlay so the mock reaches the process that runs the LLM. Covered by unit tests pinning both our injector and the litellm mock contract (string + deterministic usage + error sentinel) so a litellm bump can't silently break hermetic execute-path coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
chandrasekharan-zipstack
force-pushed
the
feat/un-3636-execute-path-llm-mock
branch
from
f26d44c to
fc0d056
Compare
Add e2e tests that exercise the full execute path hermetically using the UNSTRACT_LLM_MOCK_RESPONSE hook (no real LLM/secret): - api-deployment-run + usage-token-tracking: deploy a workflow as an API, POST a doc synchronously, assert the mocked answer and fixed 10/20/30 usage come back as structured JSON. - workflow-create-execute: two-step /workflow/execute/ then poll to COMPLETED; COMPLETED + a successful file is the hermetic proof (without the mock the completion would fail). A shared provisioned_workflow fixture stands up a Prompt Studio-backed API workflow (NoOp x2text/vectordb, chunk_size=0 so embedding/vectordb are skipped). The rig seeds a canonical mock string in ComposeRuntime so both the workers and pytest see the same value; tests skip if it is unset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017UVfw7aAocC3KCJaEyZ5GU
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2164. Adds the deterministic-LLM capability the execute-path critical-path gaps need, without dragging real provider secrets into CI.
What
llm.py): one module-level_inject_mock_response+ a one-line call at all fourlitellm.completion/acompletionsites. WhenUNSTRACT_LLM_MOCK_RESPONSEis set, litellm returns that string as the completion with fixed usage (10/20/30), tunable viaDEFAULT_MOCK_RESPONSE_*; error sentinels likelitellm.RateLimitErrorforce error paths. Unset in production → no-op.worker-executor-v2/worker-file-processing-v2(the processes that run the LLM). Empty unless the rig/CI sets it.Why this shape
test_connection(), so a fixed mock runs every execute path to completion.Not in this PR
The actual execute-path e2e tests (workflow-create-execute, usage-token-tracking, …) that consume this hook — those need the full compose stack booted to author the API flow correctly and will land next. Critical-path mappings are intentionally left unflipped until a real passing test exists (no false greens).
🤖 Generated with Claude Code