UN-2771 [FEAT] Report text extraction time in API deployment metrics#2032
UN-2771 [FEAT] Report text extraction time in API deployment metrics#2032athul-rs wants to merge 4 commits into
Conversation
The structure tool timed indexing but not the text extraction (LLMWhisperer/X2Text) call, so API responses with include_metrics=True reported indexing time only. Time dynamic_extraction the same way and merge it into the result metrics as extraction.time_taken(s). Bump structure tool to 0.0.102. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
WalkthroughThe structure pipeline now records extraction duration when extraction runs, uses reserved metric keys, and merges extraction timing with indexing metrics during pipeline finalization. Tests cover successful, skipped, failed, collision, and combined metric scenarios. ChangesStructure pipeline metrics
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
athul-rs
requested review from
chandrasekharan-zipstack,
hari-kuriakose and
jaseemjaskp
chandrasekharan-zipstack
left a comment
chandrasekharan-zipstack
left a comment
There was a problem hiding this comment.
@athul-rs I like your intent here however I don't think the tool related code or logic is being used anymore. This is deprecated and in fact we'll be removing this code soon. Can you check if this is handled in the equivalent flow involving the celery tasks and ensure this concern is addressed?
cc: @harini-venkataraman
Per review, the structure tool's Docker path is deprecated — the live
flow is the celery-based LegacyExecutor structure pipeline. Time the
extract step there and merge {'extraction': {'time_taken(s)': ...}}
into the result metrics alongside the existing per-output indexing
timing. Structure tool changes reverted (no tool version bump needed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@chandrasekharan-zipstack you were right — reworked. The |
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@workers/executor/executors/legacy_executor.py`:
- Around line 645-647: The current extraction_metrics dict uses a top-level key
"extraction" which can collide with user-defined output/prompt names when merged
later; update this to use a reserved pipeline namespace (e.g., set
extraction_metrics = {"_pipeline": {"extraction": {"time_taken(s)": ...}}}) and
then merge into the main metrics map so pipeline-level metrics live under
metrics["_pipeline"]; adjust the merge logic where metrics and
extraction_metrics are combined (the existing merge around the metrics variable
at lines ~804-811) to preserve the "_pipeline" namespace, or alternatively add a
pre-merge check to disallow user outputs named "extraction" if you prefer the
blocking approach. Ensure you update references to extraction_metrics and the
merge operation accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: d5ab6512-472a-43ec-a802-245f014d585e
📒 Files selected for processing (1)
workers/executor/executors/legacy_executor.py
jaseemjaskp
left a comment
jaseemjaskp
left a comment
There was a problem hiding this comment.
Automated review (PR Review Toolkit) of the extraction-time metric.
Net assessment: the merge logic is correct, empty-dict-safe, and introduces no silent failures — error propagation (LegacyExecutorError re-raise) is byte-for-byte unchanged. No blocking bugs. The substantive open item is the top-level-key collision already flagged by @coderabbitai on L647; the inline notes below are distinct consistency/maintainability/test points.
Biggest gap — test coverage (not inline-able; test_phase5d.py isn't in this diff): the PR adds zero tests for the new behavior. _finalize_pipeline_result / _merge_pipeline_metrics are exercised by test_phase5d.py (test_index_metrics_merged L261, test_merge_* L915-927) but none assert on extraction_metrics. Recommend adding before merge:
test_extraction_metrics_recorded— normal run:result.data["metrics"]["extraction"]["time_taken(s)"]is afloat >= 0and coexists with per-prompt keys.test_extract_failure_records_no_extraction_metric— metric is computed after the_failureearly-return, so a failed extract must record no timing (pin via a_finalize_pipeline_resultspy /assert_not_called).test_index_and_extraction_metrics_merged— both families merge (top-levelextraction+ per-outputindexing).test_skip_extraction_records_no_extraction_metric—skip_extraction⇒ no"extraction"key.
… align clocks - Nest extraction timing under a reserved "_pipeline" namespace (PSKeys.PIPELINE) so it cannot collide with a user-defined output/prompt named "extraction" at the top level of the metrics dict. - Use PSKeys.EXTRACTION instead of the hardcoded literal, and capture the duration in a named local for parity with the indexing path. - Promote the duplicated "time_taken(s)" literal to PSKeys.TIME_TAKEN. - Align the indexing path onto time.monotonic(): it measured a duration with wall-clock datetime.now(), which is wrong across NTP/system-clock adjustments and left the two producers of time_taken(s) on different clocks. Drops the now-unused local datetime import. - Document the extraction_metrics shape on _finalize_pipeline_result and type it dict[str, dict] | None. Tests: 5 new cases in test_phase5d.py — metric recorded, name-collision guard, index+extraction merge, skip-extraction, and extract-failure (no timing recorded). Worker suite: 723 passed, same 6 pre-existing failures as main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
Thanks @jaseemjaskp — all four inline threads plus the test-coverage gap are addressed in 81d01ed. Summary of what changed: Design (the substantive one): the metric is no longer a bare top-level "metrics": { "_pipeline": { "extraction": { "time_taken(s)": 1.23 } } }Sibling keys in that dict are user-defined output/prompt names, so the old shape would have collided with a prompt named "extraction". PR description updated to match. Tests — all four you listed, plus one for the collision (
Also fixed: Verification: worker suite is |
Unstract test resultsPer-group results
Critical paths
|
chandrasekharan-zipstack
left a comment
chandrasekharan-zipstack
left a comment
There was a problem hiding this comment.
Agreed follow-ups from a design discussion on the metric shape. Two mechanical renames for clarity; plus a heads-up: I filed UN-3714 for the per-profile X2Text extractor being collapsed to the default profile's extractor in the worker pipeline (out of scope for this PR).
_pipeline→_file— the bucket holds file-level (whole-document) metrics, and"file"is already the API response key for the source file (workflow_manager/workflow_v2/dto.py:45), so_fileis the intuitive/consistent name.extraction→text_extraction— avoids confusion withextraction_llm(the extraction-purpose LLM call) which sits beside it in the same metrics dict; the collision is visible in single-pass responses (extraction_llmand_pipeline.extractioncoexist).
Single-pass needs no special-casing: _file.text_extraction is merged in by the shared _finalize_pipeline_result, so single-pass metrics stay flat and just gain the _file sibling — same convention as multi-prompt.
| EXTRACTION = "extraction" | ||
| # Reserved namespace for pipeline-level metrics so they cannot collide with | ||
| # user-defined output/prompt names at the top level of the metrics dict. | ||
| PIPELINE = "_pipeline" |
There was a problem hiding this comment.
Rename PIPELINE = "_pipeline" → FILE = "_file". This bucket holds file-level (whole-document) metrics, and "file" is the existing API response key for the source file (workflow_manager/workflow_v2/dto.py:45) — _file reads intuitively and matches convention.
Also add a sibling constant for the metric name so it doesn't collide with extraction_llm:
TEXT_EXTRACTION = "text_extraction"Update the comment above to match.
| # collides with a user-defined output named "extraction" during | ||
| # the top-level metrics merge (see _merge_pipeline_metrics). | ||
| extraction_metrics = { | ||
| PSKeys.PIPELINE: { |
There was a problem hiding this comment.
Use the renamed constants here:
extraction_metrics = {
PSKeys.FILE: {
PSKeys.TEXT_EXTRACTION: {PSKeys.TIME_TAKEN: extraction_time}
}
}Also update the adjacent # Nest under a reserved "_pipeline"… comment and the _finalize_pipeline_result docstring (the {"_pipeline": {"extraction": …}} shape example) to _file / text_extraction.
|
|
||
|
|
||
| class TestExtractionMetrics: | ||
| """Extraction duration is reported under the reserved _pipeline namespace. |
There was a problem hiding this comment.
Update these tests for the renamed keys: metrics["_pipeline"]["extraction"] → metrics["_file"]["text_extraction"], the "_pipeline" not in metrics assertion → "_file", and this docstring's _pipeline namespace wording.
Worth adding one single-pass assertion too: since _file.text_extraction is merged by the shared finalize, a single-pass run should show the same _file bucket alongside its flat context_retrieval / extraction_llm — locks in the cross-mode consistency.
3 participants
What
include_metrics=Truenow report the text extraction (LLMWhisperer/X2Text) duration asmetrics._pipeline.extraction["time_taken(s)"], alongside the existing per-output indexing time.The metric lives under a reserved
_pipelinenamespace rather than a bare top-levelextractionkey: sibling keys in the metrics dict are user-defined output/prompt names, so a prompt named "extraction" would otherwise collide with it.Why
UN-2771 — metrics only included indexing time; the LLMWhisperer call (often the dominant cost for large documents) was invisible, making it hard to attribute slow executions.
How
Reworked per review (@chandrasekharan-zipstack): the original version added timing to
tools/structure(deprecated Docker path); that's reverted. The change now lives in the live celery flow:LegacyExecutor._handle_structure_pipelinetimes the extract step (time.monotonic()) and_finalize_pipeline_resultmerges{"extraction": {"time_taken(s)": ...}}into result metrics via the existing_merge_pipeline_metrics, exactly mirroring how_index_pipeline_outputrecords indexing time. The Prompt Studio IDE flow (_handle_ide_index) is untouched.Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)
_pipelinenamespace that cannot collide with user-defined output names; existingindexingmetrics and response shape unchanged. Paths that skip extraction (smart-table) leave the dict empty — nothing added, same as today.datetime.now()totime.monotonic(). Same units and semantics (elapsed seconds as a float), but immune to NTP/system-clock adjustments; it cannot regress a caller.Database Migrations
Env Config
Relevant Docs
Related Issues or PRs
Notes on Testing
workers/tests/test_phase5d.py::TestExtractionMetrics: metric recorded, name-collision guard (a prompt named "extraction"), index+extraction merge, skip-extraction records nothing, extract-failure records nothing.723 passed, 6 failed— the same 6 failures present onmain(Postgres auth + pre-existing mock assertions), none in the touched paths.include_metrics=True→ response metrics contain_pipeline.extraction.time_taken(s); smart-table path unchanged.Screenshots
N/A
Checklist
I have read and understood the Contribution Guidelines.
🤖 Generated with Claude Code