[chandrasekharan-zipstack]

What

• If a workflow's HITL rules contain a confidence condition, highlight is enabled automatically for that run. Each flow checks the rules it applies: ETL and task workflows the DB rules, API deployments the API rules (only for a request with hitl_queue_name and no hitl_packet_id, the only API requests whose destination evaluates API rules).
• The rules are checked once per execution. Each file that gets the override writes one line to the execution logs: "Highlight enabled automatically because HITL confidence rules are configured".
• Agentic projects are excluded and keep the user's own setting (see How).

Why

• In deployed workflows, enable_highlight comes from the tool instance settings. When it is off, no confidence data is produced, and the HITL rule engine scores every field as 100. Then confidence <= X rules never route a document to review, and >= rules always do. Nothing tells the user.

How

• general/tasks.py (ETL and task workflows) calls has_confidence_rules(workflow_util, workflow_id, "DB"), and api-deployment/tasks.py calls it with "API" when the request has hitl_queue_name and no hitl_packet_id (helper in shared/utils/manual_review_factory.py). Other API requests skip the check: their destination evaluates no API rules. Each does this once per execution, before batching. If it returns True, it sets manual_review_config["auto_enable_highlight"] = True.
• shared/workflow/execution/service.py: the file worker reads the flag and passes it to the structure tool in its params (auto_enable_highlight). The tool instance metadata is passed unchanged.
• file_processing/structure_tool_task.py: after the agentic branch returns, a set flag makes the tool use {**tool_instance_metadata, "enable_highlight": True} and log the line. This is a copy. The saved setting is never written back, so when the rules are removed the user's own setting applies again.
• Agentic projects are skipped: workflow runs pass them no line_metadata, so highlight_data is always {} and confidence can't work there. Making confidence work for agentic projects is a separate ticket.
• enable_word_confidence is not changed. The user opts in to it because its output-token cost grows with the number of fields.
• The check itself comes from the manual_review plugin (cloud PR below). The OSS helper returns False when the util has no has_confidence_rules method: the OSS null util, or an older plugin built before the cloud PR.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

• OSS (no manual_review plugin): no change. The helper returns False, so the flag is never set.
• Cloud with an older manual_review plugin (no has_confidence_rules): no change, same as OSS.
• Cloud workflows with no confidence rules: no change.
• Cloud workflows with confidence rules and highlight off: highlight now runs. That adds about 200 input tokens per page per prompt, plus one LLMWhisperer highlight call. Routing now uses real scores, which is the intended fix.
• If the rules can't be loaded, the workflow runs as if it has no confidence rules, so the user's own highlight setting applies.
• Highlight applies to every file in the run, not only files sent to review: rules are evaluated after extraction. Each file gets the highlight prompt postamble and highlight/confidence keys in its metadata, where the extractor produces them.
• Highlight also changes extraction for those runs: every prompt gets the line-number postamble, LLMWhisperer V2 is asked for line numbers and line confidence and makes one extra highlight call, and the highlight plugin post-processes each answer. Answers may differ from a run with highlight off. This is accepted and noted in code.
• API callers that request metadata (include_metadata) now get confidence_data in it, and the line_numbers and whisper_hash keys, which are always present, are now filled. highlight_data is filled too, but callers see it only when the org's API-deployment highlight flag is on. This happens even though the tool's highlight checkbox is off. The change is additive, and the keys are kept on purpose so callers can see why a document was routed.
  • Release note: API responses with include_metadata may now carry confidence_data / line_numbers / whisper_hash / highlight_data when HITL confidence rules exist.
• Known gap, accepted: an API request with use_file_history and no hitl_queue_name has DB rules evaluated at the destination, but the API orchestrator checks only API rules, so a DB confidence rule there still scores 100. use_file_history is undocumented (hidden from the API schema, meant for demos), though a caller can still send it; this path is left out of scope for now and noted in code.
• With summarize-as-source on, or an extractor other than LLMWhisperer, highlight is still turned on but no confidence data is produced, so every field still scores 100. The rule editor now warns about these cases (cloud PR below).
• The Docker-based (non-structure) tool path does not change.

Database Migrations

• None

Env Config

• None

Relevant Docs

• https://github.com/Zipstack/unstract-docs/pull/253 (highlight, word confidence and HITL confidence rules)

Related Issues or PRs

• Cloud plugin: https://github.com/Zipstack/unstract-cloud/pull/1916

Dependencies Versions

• Needs the cloud PR above for the feature to work. Each PR is safe to deploy on its own.

Notes on Testing

• Unit, tests/test_structure_tool_pipeline.py:
  • TestAutoEnableHighlight.test_regular_overrides_a_copy: user off + rule = on, user on + no rule = on, off + off = off. Word confidence and the saved metadata are unchanged. The log line is written only when the override applies.
  • TestAutoEnableHighlight.test_agentic_keeps_user_setting: agentic + flag set + user highlight off = X2Text called with enable_highlight=False, and no log line.
  • test_auto_enable_highlight_read_from_review_config and test_auto_enable_highlight_passed_in_params: the flag travels from manual_review_config to the structure tool params, and the metadata is unchanged.
  • test_has_confidence_rules_helper: True, False, an older plugin without the method, and the rule type is passed through.
• Unit, tests/test_barrier.py TestAutoEnableHighlightOrchestration: the API orchestrator sets the flag on every batch only for an API confidence rule on a hitl_queue_name request (a DB rule alone, no queue name, or a hitl_packet_id does not set it, and the rules are not fetched), and the general orchestrator only for a DB confidence rule (an API rule alone does not set it).
• Run from workers/: uv run --group test pytest tests/test_structure_tool_pipeline.py tests/test_barrier.py -q --no-cov: 54 passed.
• Mutation check: 10 mutants, each fails the suite: override removed, metadata mutated in place, log removed, override applied to agentic runs, each orchestrator given the other rule type, helper ignoring the rule type, flag dropped from the params, API hitl_queue_name gate removed, API hitl_packet_id gate removed.
• End-to-end, in a dev namespace running this PR and the cloud PR, with a Prompt Studio tool whose highlight checkbox is off and a text PDF on LLMWhisperer V2. Method: worker logs (Highlight enabled automatically..., the extractor's highlight parameters, the HITL rule evaluation) and the API response metadata.
  • ETL, DB rule confidence.vendor_name != 100: the log line is written, LLMWhisperer runs with highlight and line numbers, and the destination evaluates the DB rule.
  • ETL, DB rule changed to a non-confidence condition (output.vendor_name): no log line, highlight stays off.
  • API deployment with an API confidence rule, request with hitl_queue_name: the log line is written; the response metadata has confidence_data and filled line_numbers and whisper_hash (no highlight_data, the org flag is off).
  • Same deployment, request without hitl_queue_name: no log line, no confidence_data, empty line_numbers and whisper_hash.
  • The saved enable_highlight on both tool instances is still false after the runs.
  • Not shown: routing by real scores. For this text PDF, LLMWhisperer returned empty confidence lists, so the rule saw the default score. A scanned document is needed to show real scores.
  • Not run: the agentic case (covered by the unit test above).

Screenshots

Checklist

I have read and understood the Contribution Guidelines.

🤖 Generated with Claude Code

[chandrasekharan-zipstack]

Self-review round 1 — tip d1bbec1258fa0056433f4fa62725e9b6cf38c094

Review: unstract:review-deep (review-shallow refused: auth/tenancy trigger auth_authz_secrets, fired by organization_id= pass-through and a test mock named _get_platform_service_api_key). Mode: INITIAL. Diff 33224a79..d1bbec1, 5 files / 118 lines, base fetched fresh.

Verdict — REQUEST CHANGES

Summary — Critical: 0 · High: 2 · Medium: 3 · Low: 0 · Lenses run: 19/19

Signals: lens 4 raised (organization_id pass-through; no auth logic changed), lens 13 raised (tests changed).

Findings

Posted inline (5).

Unanchored findings

• PR title follows TICKET [TYPE] text. Nothing to flag.

Lens checklist

1. Spec & intent — See findings #1, #5
2. Architectural fit & precedent — Clean
3. Correctness & edge cases — See findings #1, #4, #5
4. Security — Clean (orchestrator-assessed; no authz/tenant logic changed, organization_id only forwarded to existing log helper)
5. Data integrity & migrations — N/A — no persisted state; override applied to a copy, never written
6. Concurrency — N/A — no shared state or ordering changes
7. API & contract compatibility — Clean (orchestrator-assessed; additive manual_review_config key, consumer defaults to False, rolling deploy safe both ways)
8. Reliability & resilience — Clean (orchestrator-assessed; two internal API calls via existing client, plugin catches per-type failures)
9. Performance & cost — See findings #4
10. Observability — Clean
11. Operational safety — Clean (orchestrator-assessed; rollout risk captured in #1)
12. LLM/agent-specific — See findings #5
13. Testing — See findings #2, #3
14. Dependencies & build — N/A — no dependency changes
15. Code quality — Clean (ruff check/format at repo config: only I001 in tests/test_structure_tool_pipeline.py:10, pre-existing on base, not this diff's lines)
16. Doc & comment accuracy — Clean (orchestrator-assessed)
17. Cross-document rule consistency — Clean (merge sweep: 4 hits, all the new symbol's own definitions/uses)
18. On-prem deployment experience — N/A — no chart, on-prem config or docs touched
19. Scaling & workload lifecycle — N/A — no worker scaling config touched

boundary-sweep: 2 sinks traced, clean (two UI log lines carry a constant message). rule-storage-check: not covered — no markdown in the diff. Security deep-dive: done (no findings at confidence >= 8).

Open questions for the author — #4 and #5 above.

Assumptions made — Release order of OSS vs cloud is not enforced.

cost: agents=3 wall=05:10 lenses-examined=14 files=5 lines=118

[chandrasekharan-zipstack]

Self-review round 2 — tip 5cfbe0661face00ea002543ceea902582ce1431f

Review: unstract:review-deep (review-shallow gate still refuses PR-wide: auth_authz_secrets). Mode: FOLLOWUP. Round delta d1bbec1..5cfbe06.

Verdict — APPROVE WITH NITS

Summary — Critical: 0 · High: 0 · Medium: 0 · Low: 1 · Lenses run: 19/19

Prior context fetched — followup-fetch.sh exited 0. Since-boundary d1bbec1258fa0056433f4fa62725e9b6cf38c094.

Scope change — NO

Prior findings status

• #4403419712 [High] Plugin without has_confidence_rules fails every run → RESOLVED — guarded helper workers/shared/utils/manual_review_factory.py:393-401, both callers use it (workers/general/tasks.py:986, workers/api-deployment/tasks.py:674-676), older-plugin case tested.
• #4403419723 [High] Flag path untested → RESOLVED — test_auto_enable_highlight_read_from_review_config and parametrised test_structure_tool_params_passthrough.
• #4403419734 [Medium] Orchestrators tested only with rules absent → RESOLVED — workers/tests/test_barrier.py TestAutoEnableHighlightOrchestration, both orchestrators, True/False, API with 2 batches.
• #4403419740 [Medium] DB+API scan regardless of flow → WAIVED — accepted in writing at workers/shared/utils/manual_review_factory.py:394-398 and in the PR body; matches the agreed scope ("check both DB and API rules").
• #4403419750 [Medium] Agentic forced highlight without confidence → WAIVED — accepted in writing at workers/shared/workflow/execution/service.py:1049-1051 and the PR body caveat; decision surfaced to the task owner.

New findings

• [Low] [Lens 13] — General-orchestrator test's try/except comment gives the wrong reason
  • Location: workers/tests/test_barrier.py:L869-L885
  • Failure mode: the _create_batch_data_general stub returns None, so the orchestrator raises AttributeError at general/tasks.py:1038, before the chord. The comment says post-chord status updates raise; the chord patch is never reached. Assertions still pin the flag and the log, which happen first.
  • Suggested fix: stub returns MagicMock(files=[]) and records the config; correct the comment.
  • Confidence: High
  • Verification: VERIFIED — ran the test with --capture=no, saw the AttributeError.

Lens checklist

1. Spec & intent — Clean
2. Architectural fit & precedent — Clean
3. Correctness & edge cases — Clean
4. Security — Clean (orchestrator-assessed; deep-dive done, 0 findings)
5. Data integrity & migrations — N/A — no persisted state
6. Concurrency — N/A — no shared state or ordering changes
7. API & contract compatibility — Clean (orchestrator-assessed; delta adds no payload change)
8. Reliability & resilience — Clean (orchestrator-assessed; helper fails closed to False)
9. Performance & cost — Clean
10. Observability — Clean
11. Operational safety — Clean (orchestrator-assessed; deploy-order risk removed)
12. LLM/agent-specific — Clean (agentic caveat documented)
13. Testing — See findings #1
14. Dependencies & build — N/A — no dependency changes
15. Code quality — Clean (ruff format/check on changed lines clean; I001/D209 in test files pre-existing on base)
16. Doc & comment accuracy — See findings #1
17. Cross-document rule consistency — Clean (merge sweep hits are the new symbol's own uses)
18. On-prem deployment experience — N/A — no on-prem surface touched
19. Scaling & workload lifecycle — N/A — no scaling config touched

boundary-sweep: 0 new sinks in delta, clean. rule-storage-check: not covered — no markdown in the diff. Security deep-dive: done (no findings at confidence >= 8).

Open questions / assumptions — none new.

cost: agents=3 wall=03:09 lenses-examined=14 files=6 lines=311

[chandrasekharan-zipstack]

Self-review round 3 — tip a5c76fe4d50cf6bda1c2154ccc9b78d8696c892e

Review: unstract:review-deep (review-shallow gate still refuses PR-wide: auth_authz_secrets). Mode: FOLLOWUP. Round delta 5cfbe06..a5c76fe (workers/tests/test_barrier.py only).

Verdict — APPROVE

Summary — Critical: 0 · High: 0 · Medium: 0 · Low: 0 · Lenses run: 19/19

Prior context fetched — followup-fetch.sh exited 0. Since-boundary 5cfbe0661face00ea002543ceea902582ce1431f.

Scope change — NO

Prior findings status

• #4403419712 [High] → RESOLVED (thread resolved in round 2)
• #4403419723 [High] → RESOLVED (thread resolved in round 2)
• #4403419734 [Medium] → RESOLVED (thread resolved in round 2)
• #4403419740 [Medium] → WAIVED — accepted in writing in code and PR body
• #4403419750 [Medium] → WAIVED — accepted in writing; agentic decision with the task owner, thread left open
• Round 2 Low (General-orchestrator test try/except) → RESOLVED in a5c76fe — stub returns MagicMock(files=[]) at workers/tests/test_barrier.py:857-859, try/except removed; the orchestrator now returns status="orchestrated".

New findings

No findings. Test agent mutated a scratch copy: removing the flag write, changing the log message, moving the flag write after the batch loop, or making the gate unconditional each fail the test.

Lens checklist

1. Spec & intent — Clean
2. Architectural fit & precedent — Clean
3. Correctness & edge cases — Clean
4. Security — Clean (deep-dive done, 0 findings)
5. Data integrity & migrations — N/A — test-only delta
6. Concurrency — N/A — test-only delta
7. API & contract compatibility — N/A — test-only delta
8. Reliability & resilience — N/A — test-only delta
9. Performance & cost — N/A — test-only delta
10. Observability — N/A — test-only delta
11. Operational safety — N/A — test-only delta
12. LLM/agent-specific — N/A — test-only delta
13. Testing — Clean
14. Dependencies & build — N/A — none changed
15. Code quality — Clean (ruff@0.8.4 format/check: no hunks in the changed block)
16. Doc & comment accuracy — Clean (stale comment removed with its try/except)
17. Cross-document rule consistency — Clean (no rule changed)
18. On-prem deployment experience — N/A — no on-prem surface
19. Scaling & workload lifecycle — N/A — none touched

boundary-sweep: 0 sinks in delta, clean. rule-storage-check: not covered — no markdown in the diff. Security deep-dive: done (no findings at confidence >= 8).

Open questions / assumptions — none new.

cost: agents=2 wall=03:46 lenses-examined=8 files=1 lines=39

[chandrasekharan-zipstack]

Self-review round 4 — tip b829d3c961c7781f340db491728b0440bd3dc5fa

Review: unstract:review-deep (review-shallow gate still refuses PR-wide: auth_authz_secrets). Mode: FOLLOWUP. Round delta a5c76fe..b829d3c (7 files: per-flow rule type, override moved into the structure tool, agentic skipped).

Verdict — REQUEST CHANGES

Summary — Critical: 0 · High: 0 · Medium: 4 · Low: 0 · Lenses run: 19/19

Prior context fetched — followup-fetch.sh exited 0. Since-boundary a5c76fe4d50cf6bda1c2154ccc9b78d8696c892e.

Scope change — NO

Prior findings status

• #4403419712 [High] → RESOLVED (thread resolved in round 2)
• #4403419723 [High] → RESOLVED (thread resolved in round 2)
• #4403419734 [Medium] → RESOLVED (thread resolved in round 2)
• #4403419740 [Medium] → WAIVED — accepted in writing in code and PR body
• #4403419750 [Medium] → RESOLVED in b829d3c — agentic runs keep the user's setting; thread resolved

New findings

• [Medium] [Lens 3, 12] — API orchestrator checks API rules on calls whose destination never evaluates them, and misses the DB rules the file-history path does evaluate — workers/api-deployment/tasks.py:673. The destination evaluates API rules only when hitl_queue_name is set; hitl_packet_id bypasses rules; with neither, it skips HITL unless use_file_history, where it evaluates DB rules (workers/shared/workflow/destination_connector.py:1580-1681). So plain API calls pay for highlight that nothing reads, and the file-history path can still score 100. Flagged by the cold read and the reliability/LLM agent.
• [Medium] [Lens 10] — The auto-enable log line also fires where highlight cannot produce confidence — workers/file_processing/structure_tool_task.py:375. Non-LLMWhisperer extractors, a missing highlight plugin, or summarize-as-source give no confidence data, so rules still score 100. The PR body notes this; the code does not.
• [Medium] [Lens 7, 9] — Highlight is forced for every file in the run, which also adds the highlight postamble to prompts and highlight/confidence keys to metadata — workers/general/tasks.py:985. By design: rules are evaluated after extraction, so every file needs confidence. Owed: say it in code.
• [Medium] [Lens 3] — Agentic runs drop the flag with no log line — workers/file_processing/structure_tool_task.py:372. Accepted: the task owner decided agentic runs get no log line; the comment at this line records the skip.

Lens checklist

1. Spec & intent — See findings
2. Architectural fit & precedent — Clean
3. Correctness & edge cases — See findings
4. Security — Clean
5. Data integrity & migrations — N/A — no schema or data writes
6. Concurrency — N/A — no shared state or ordering changed
7. API & contract compatibility — See findings
8. Reliability & resilience — Clean
9. Performance & cost — See findings
10. Observability — See findings
11. Operational safety — N/A — no infra or rollout config
12. LLM/agent-specific — See findings
13. Testing — Clean
14. Dependencies & build — N/A — none changed
15. Code quality — Clean
16. Doc & comment accuracy — Clean
17. Cross-document rule consistency — Clean
18. On-prem deployment experience — N/A — no on-prem surface
19. Scaling & workload lifecycle — N/A — none touched

lens17-merge-sweep: exit 1, every hit in the branch's own files with consistent call shapes. boundary-sweep: 1 sink traced (fixed-string stream_log), clean. rule-storage-check: not covered — no markdown in the diff. Security deep-dive: done (no findings at confidence >= 8).

Open questions / assumptions — API file-history path: should the API orchestrator also check DB rules there, mirroring the destination? Raised with the task owner.

cost: agents=7 lenses-examined=19 files=7 lines=406

[chandrasekharan-zipstack]

Self-review round 5 — tip ad56d13e7e500b3a2a8f438d7a0ecda0d16d9bf3

Review: unstract:review-deep (review-shallow gate still refuses PR-wide: auth_authz_secrets). Mode: FOLLOWUP. Round delta b829d3c..ad56d13 (API rule check gated on hitl_queue_name / hitl_packet_id; known limits recorded in comments).

Verdict — REQUEST CHANGES

Summary — Critical: 0 · High: 0 · Medium: 2 · Low: 3 · Lenses run: 19/19

Prior context fetched — followup-fetch.sh exited 0. Since-boundary b829d3c961c7781f340db491728b0440bd3dc5fa.

Scope change — NO

Prior findings status

• #4403419712, #4403419723, #4403419734 → RESOLVED (round 2)
• #4403419740 → WAIVED — accepted in writing in code and PR body
• #4403419750 → RESOLVED in b829d3c
• #4404194298 [Medium] API rule type → PARTIALLY RESOLVED in ad56d13 — queue-name / packet gates added; API + file-history DB path open, pending a decision
• #4404194312 [Medium] log fires without confidence → RESOLVED in ad56d13
• #4404194323 [Medium] run-wide highlight → RESOLVED in ad56d13
• #4404194328 [Medium] agentic no log → WAIVED — task owner decision

New findings

• [Medium] [Lens 7] — Forced highlight adds confidence_data, line_numbers, whisper_hash (and, with the org flag on, highlight_data) to the API response metadata for include_metadata callers whose tool had highlight off — workers/file_processing/structure_tool_task.py:378. The response path strips only highlight_data, and only on the org flag (backend/api_v2/deployment_helper.py:402-411). Additive keys; whether to strip them is a design decision.
• [Medium] [Lens 2, 3] — The orchestrator copies the destination's rule-type choice, and the copy misses the API + file-history DB branch — workers/api-deployment/tasks.py:673. Same gap as #4404194298; the cold read suggests one shared hitl_rule_type(...) helper used by _should_handle_hitl and both orchestrators.
• [Low] [Lens 16] — "Each file gets the highlight prompt postamble" overstates: agentic, summarize-as-source and non-LLMWhisperer V2 runs get neither — workers/general/tasks.py:985.
• [Low] [Lens 16] — Test docstring leaves out the hitl_packet_id exclusion — workers/tests/test_barrier.py:779.
• [Low] [Lens 15] — Leftover local tool_instance_metadata passed through unchanged — workers/shared/workflow/execution/service.py:1047.

Lens checklist

1. Spec & intent — Clean
2. Architectural fit & precedent — See findings
3. Correctness & edge cases — See findings
4. Security — Clean
5. Data integrity & migrations — N/A — no schema or data writes
6. Concurrency — N/A — no shared state or ordering changed
7. API & contract compatibility — See findings
8. Reliability & resilience — Clean
9. Performance & cost — Clean
10. Observability — Clean
11. Operational safety — N/A — no infra or rollout config
12. LLM/agent-specific — Clean
13. Testing — Clean
14. Dependencies & build — N/A — none changed
15. Code quality — See findings
16. Doc & comment accuracy — See findings
17. Cross-document rule consistency — Clean
18. On-prem deployment experience — N/A — no on-prem surface
19. Scaling & workload lifecycle — N/A — none touched

lens17: no rule text changed beyond the comments reviewed under lens 16. boundary-sweep: 0 new sinks in delta, clean. rule-storage-check: not covered — no markdown in the diff. Security deep-dive: done (no findings at confidence >= 8).

Open questions / assumptions — Both Mediums need the task owner's call: strip forced keys from the API response, and cover the API + file-history DB path (shared helper). Until then they are recorded in code and the PR body.

cost: agents=6 lenses-examined=19 files=7 lines=432

[chandrasekharan-zipstack]

Review — FOLLOWUP (self-review round 6)

Tip: b0e2218 · delta ad56d13..b0e2218 · review: review-deep (whole PR touches an API payload path, so lite is not eligible).

Verdict — REQUEST CHANGES

Summary — Critical: 0 · High: 1 · Medium: 0 · Low: 1

Prior context fetched

— followup-fetch.sh exited 0; since boundary ad56d13.

Scope change

— NO

Prior findings status

• API rule type on the API + file-history path [Medium] → PARTIALLY RESOLVED — gate now limited to hitl_queue_name requests (workers/api-deployment/tasks.py:673-685); the file-history path still checks no DB rules. Recorded in code; pending product decision.
• API response metadata gains highlight keys [Medium] → NOT RESOLVED — pending decision (strip or document).
• Duplicated rule-type routing [Medium] → NOT RESOLVED — pending decision on a shared helper.
• Three Lows from the previous round → RESOLVED in b0e2218.

New findings

1. [High] Forced highlight changes extraction, not only metadata — workers/file_processing/structure_tool_task.py:372-380, workers/api-deployment/tasks.py:673-678. With highlight on, answer_prompt.py:120-126 keeps the platform postamble (line-number instruction) in every prompt; LLMWhisperer V2 is called with add_line_nos/include_line_confidence and makes an extra highlight request (unstract/sdk1/.../llm_whisperer_v2/src/helper.py:271-272, 338-344); the highlight plugin post-processes answers (legacy_executor.py:1409-1427). Comments, the log line and the PR body describe the override as only adding confidence data. Failure: every file in a run with a confidence rule gets a different prompt and extracted text, so answers can change for users who never enabled highlight. Mechanism verified; answer-level change not reproduced. Fix: product decision — get confidence without the prompt postamble, or accept and document that prompts and extracted text change.
2. [Low] Metadata key wording — workers/api-deployment/tasks.py:677-678 and PR body: say highlight adds line_numbers and whisper_hash. answer_prompt.py:308-317 always writes highlight_data, line_numbers and whisper_hash; only confidence_data is new, the others get filled.

Security deep-dive: done (0 findings).

Lens checklist

• 1: See findings #2
• 2: Clean
• 3: See findings #1
• 4: Clean
• 5: N/A — no schema or data migration
• 6: Clean
• 7: Clean — API metadata change tracked in prior findings
• 8: Clean
• 9: Clean
• 10: Clean
• 11: N/A — no infra or rollout config
• 12: See findings #1
• 13: Clean
• 14: N/A — no dependency change
• 15: Clean
• 16: See findings #1, #2
• 17: Clean
• 18: N/A — no on-prem surface
• 19: N/A — no scaling or workload config

Open questions / assumptions

— Assumes answers can change with the postamble; not reproduced end to end yet (dev-namespace test pending).

[chandrasekharan-zipstack]

Review — FOLLOWUP (self-review round 7)

Tip: 49e7b43 · delta b0e2218..49e7b43 · review: review-deep (lite refused: security disqualifier over the whole PR).

Verdict — APPROVE WITH NITS

Summary — Critical: 0 · High: 0 · Medium: 0 · Low: 1

Prior context fetched

followup-fetch.sh exited 0; since boundary b0e2218.

Scope change

NO

Prior findings status

• Forced highlight changes extraction [High] → PARTIALLY RESOLVED — documented at workers/file_processing/structure_tool_task.py:378-380 and in the PR body; product decision still open.
• Metadata key wording [Low] → RESOLVED in 49e7b43.
• API + file-history DB rules [Medium], API response metadata [Medium], shared routing helper [Medium] → NOT RESOLVED — pending product decisions.

New findings

1. [Low] workers/api-deployment/tasks.py:677-679 — comment says include_metadata callers receive highlight_data; backend/api_v2/deployment_helper.py:401-411 strips it unless ENABLE_HIGHLIGHT_API_DEPLOYMENT is on (default off). The PR body states it correctly. Also, the two comments describe behaviour but do not say a decision is pending, while the body says "recorded in code". Fix: drop highlight_data from the received list; mark both as pending decisions.

Security deep-dive: done (0 findings). Cold read: no findings. Tests: 54 passed (re-run by reviewer).

Lens checklist

• 1: Clean
• 2: Clean
• 3: Clean
• 4: Clean
• 5: N/A — no schema or data migration
• 6: Clean
• 7: Clean
• 8: Clean
• 9: Clean
• 10: Clean
• 11: N/A — no infra or rollout config
• 12: Clean
• 13: Clean
• 14: N/A — no dependency change
• 15: Clean
• 16: See findings #1
• 17: Clean
• 18: N/A — no on-prem surface
• 19: N/A — no scaling or workload config

Open questions / assumptions

Answer-level change from the postamble not reproduced yet; dev namespace now deployed (snapshot.529).

[chandrasekharan-zipstack]

Review — FOLLOWUP (self-review round 8)

Tip: b5ee69b · delta 49e7b43..b5ee69b · review: review-deep (lite refuses this PR on the security disqualifier).

Verdict — APPROVE

Summary — Critical: 0 · High: 0 · Medium: 0 · Low: 0

Prior context fetched

followup-fetch.sh exited 0; since boundary 49e7b43.

Scope change

NO

Prior findings status

• highlight_data exposure wording [Low] → RESOLVED in b5ee69b (deployment_helper.py:401-411, 611-620 strip it unless the org flag is on).
• Forced highlight changes extraction [High] → PARTIALLY RESOLVED — documented in code and PR body; product decision open.
• API + file-history DB rules, API response metadata, shared routing helper [Medium] → NOT RESOLVED — pending product decisions.

New findings

No findings at confidence >= 8. Comment check, cold read and security scan each returned none. Tests: 54 passed (re-run).

Security deep-dive: done (0 findings).

Lens checklist

• 1: Clean
• 2: Clean
• 3: Clean
• 4: Clean
• 5: N/A — no schema or data migration
• 6: Clean
• 7: Clean
• 8: Clean
• 9: Clean
• 10: Clean
• 11: N/A — no infra or rollout config
• 12: Clean
• 13: Clean
• 14: N/A — no dependency change
• 15: Clean
• 16: Clean
• 17: Clean
• 18: N/A — no on-prem surface
• 19: N/A — no scaling or workload config

Open questions / assumptions

Below threshold: the Docker (non-structure) tool path ignores the flag silently; files served from file history keep their earlier no-highlight result.

[chandrasekharan-zipstack]

Review — FOLLOWUP (self-review round 9)

Tip: 8f67921 · delta b5ee69b..8f67921 · review: review-deep (lite refuses this PR on the security disqualifier).

Verdict — REQUEST CHANGES

Summary — Critical: 0 · High: 0 · Medium: 1 · Low: 0

Prior context fetched

followup-fetch.sh exited 0; since boundary b5ee69b.

Scope change

NO

Prior findings status

• API + use_file_history DB rules, API response metadata, shared routing helper, forced-highlight extraction change → WAIVED — PR author decisions, replies on each thread; recorded in code in 8f67921.

New findings

1. [Medium] workers/api-deployment/tasks.py:675-677 and PR body "Known gap" bullet — comment says use_file_history is internal-only and to add a DB check "if it is ever exposed". It is a validated request field (backend/api_v2/serializers.py:317), only hidden from the OpenAPI schema (L280), read at api_deployment_views.py:120. An API caller can send it; the destination then evaluates DB rules (destination_connector.py:1643-1681) with no highlight. Fix: describe it as undocumented but caller-settable; keep the out-of-scope decision stated as a decision.

Security deep-dive: done (0 findings). Cold read: no findings. Tests: 54 passed (re-run).

Lens checklist

• 1: See findings #1
• 2: Clean
• 3: Clean
• 4: Clean
• 5: N/A — no schema or data migration
• 6: Clean
• 7: Clean
• 8: Clean
• 9: Clean
• 10: Clean
• 11: N/A — no infra or rollout config
• 12: Clean
• 13: Clean
• 14: N/A — no dependency change
• 15: Clean — ruff 0.8.4 format/check run on changed files
• 16: See findings #1
• 17: Clean
• 18: N/A — no on-prem surface
• 19: N/A — no scaling or workload config

Open questions / assumptions

Below threshold: one extra rule-engine call per cloud ETL execution (once per run); API path re-fetches API rules already fetched in _create_file_data.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[chandrasekharan-zipstack]

Review — FOLLOWUP (self-review round 10)

Tip: a2a8ccd · delta 8f67921..a2a8ccd · review: review-deep (lite refuses this PR on the security disqualifier).

Verdict — APPROVE

Summary — Critical: 0 · High: 0 · Medium: 0 · Low: 0

Prior context fetched

followup-fetch.sh exited 0; since boundary 8f67921.

Scope change

NO

Prior findings status

• use_file_history described as internal-only [Medium] → RESOLVED in a2a8ccd (workers/api-deployment/tasks.py:675-678; matches backend/api_v2/serializers.py:282,317).

New findings

No findings at confidence >= 8. Comment check, cold read and security scan each returned none. Tests: 54 passed (re-run).

Security deep-dive: done (0 findings).

Lens checklist

• 1: Clean
• 2: Clean
• 3: Clean
• 4: Clean
• 5: N/A — no schema or data migration
• 6: Clean
• 7: Clean
• 8: Clean
• 9: Clean
• 10: Clean
• 11: N/A — no infra or rollout config
• 12: Clean
• 13: Clean
• 14: N/A — no dependency change
• 15: Clean — ruff 0.8.4 format/check run on changed files
• 16: Clean
• 17: Clean
• 18: N/A — no on-prem surface
• 19: N/A — no scaling or workload config

Open questions / assumptions

Not verified by this round: the cloud plugin scores a confidence rule as 100 when no confidence data exists (stated in structure_tool_task.py; from the original ticket analysis). End-to-end run in the dev namespace is in progress.

[greptile-apps]

via Greptile

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge within its stated scope.

Summary

Automatically enables highlight for regular structure-tool runs when the applicable HITL rules use confidence.

• Structure-tool runs enable highlight when their HITL rules use confidence.

Acknowledged by chandrasekharan-zipstack:

• Agentic projects, unsupported extractors, missing highlight plugins, and summarize-as-source may still lack confidence data; those cases are outside this fix.
• API requests with use_file_history and no hitl_queue_name still lack automatic highlight for DB confidence rules; this demo path is deferred.
• Highlight adds cost, can change extracted answers, and adds response metadata; these changes are intentional.
• enable_word_confidence remains opt-in because of its cost. Docker-based tools are unchanged.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  G[General workflow] --> D[Check DB confidence rules]
  A[API deployment] --> Q{Queue name present and no packet ID?}
  Q -->|Yes| R[Check API confidence rules]
  Q -->|No| U[Keep user setting]
  D --> F{Confidence rule found?}
  R --> F
  F -->|No| U
  F -->|Yes| C[Set flag in manual_review_config]
  C --> W[File worker passes flag]
  W --> T{Regular structure tool?}
  T -->|Yes| H[Copy settings and enable highlight]
  T -->|Agentic| U

Loading

_{Reviews (1) · Last reviewed commit: "UN-4435 fix: describe use_file_history a..." · Reviewed by Greptile}

[github-actions]

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Errors	Skipped	Duration (s)
⚪	e2e-agent-kv	e2e	0	0	0	1	0.1
✅	e2e-api-deployment	e2e	3	0	0	0	14.2
✅	e2e-coowners	e2e	1	0	0	0	1.3
✅	e2e-etl	e2e	1	0	0	0	10.6
✅	e2e-login	e2e	2	0	0	0	1.2
✅	e2e-prompt-studio	e2e	1	0	0	0	9.0
✅	e2e-smoke	e2e	2	0	0	0	1.0
✅	e2e-workflow	e2e	1	0	0	0	12.2
✅	frontend	unit	640	0	0	0	21.0
✅	integration-backend	integration	1453	0	0	26	82.7
✅	integration-connectors	integration	1	0	0	7	9.5
✅	integration-workers	integration	181	0	0	1	54.8
❌	ui	e2e	0	1	0	0	0.0
✅	unit-agent-kv-schema	unit	143	0	0	0	1.8
✅	unit-backend	unit	2107	0	0	1	60.5
✅	unit-connectors	unit	82	0	0	0	11.6
✅	unit-core	unit	281	0	0	0	3.3
✅	unit-filesystem	unit	1	0	0	0	4.4
✅	unit-platform-service	unit	18	0	0	0	2.9
✅	unit-rig	unit	120	0	0	0	4.7
✅	unit-runner	unit	12	0	0	0	3.0
✅	unit-sdk1	unit	878	0	0	0	42.7
✅	unit-tool-sidecar	unit	8	0	0	0	1.9
✅	unit-workers	unit	1609	0	0	2	132.4
	TOTAL		7545	1	0	38	486.7

Critical paths

⚠️ Critical paths not yet covered

• workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

• auth-login — covered by e2e-login
• adapter-register-llm — covered by integration-backend
• workflow-author — covered by integration-backend
• co-owner-manage — covered by integration-backend, e2e-coowners
• workflow-create-execute — covered by e2e-workflow
• api-deployment-provision — covered by integration-backend
• api-deployment-auth — covered by integration-backend
• api-deployment-run — covered by e2e-api-deployment
• mcp-server-auth — covered by integration-backend
• mcp-platform-auth — covered by integration-backend
• platform-key-whoami — covered by integration-backend
• prompt-studio-author — covered by integration-backend
• prompt-studio-fetch-response — covered by e2e-prompt-studio
• connector-register-test — covered by integration-backend
• pipeline-etl-execute — covered by e2e-etl
• usage-aggregate-read — covered by integration-backend
• usage-token-tracking — covered by e2e-api-deployment
• callback-result-delivery — covered by e2e-api-deployment