Skip to content

Avoid extra_dejson in ADF and Synapse async hooks - #72130

Open
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson
Open

Avoid extra_dejson in ADF and Synapse async hooks#72130
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:fix/55728-adf-async-extra-dejson

Conversation

@Vamsi-klu

@Vamsi-klu Vamsi-klu commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

closes: #55728

What is the change?

The async paths in the Azure Data Factory and Synapse pipeline hooks stop reading conn.extra_dejson and parse extras with json.loads(conn.extra) instead. AzureSynapsePipelineAsyncHook.get_async_conn now fetches the connection through get_async_connection rather than the sync get_connection, and ADF refresh_conn closes the cached _async_conn before rebuilding it.

Why did I do it?

extra_dejson can call mask_secret, which does a sync send on the triggerer event loop and raises RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop. The stack users see at execute_complete is that trigger error re-raised. This is the same leftover that #55179 fixed for MSGraph.

How did I do it?

In hooks/data_factory.py and hooks/synapse.py I swapped conn.extra_dejson for json.loads(conn.extra) if conn.extra else {} on the async paths only, and made refresh_conn await self.close(). The sync hooks keep extra_dejson so worker-side secret masking is unchanged.

What's the impact?

Deferrable ADF and Synapse pipeline runs stop crashing on the triggerer with the AsyncToSync error. Sync hooks and execute_complete behave exactly as before. Azure provider only.

What's the test plan?

New unit tests make extra_dejson raise the real AsyncToSync error and assert the async hooks still build their clients; they fail if extra_dejson comes back. test_refresh_conn now asserts the aio client gets closed. Reviewers can re-run the two touched test files or breeze testing providers-tests --test-type "Providers[microsoft.azure]"; provider CI covers the same.


Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: Cursor Grok 4.6 following the guidelines

extra_dejson can mask secrets via a sync send on the triggerer event
loop, which raises AsyncToSync. Parse extras with json.loads instead,
matching the MSGraph workaround.

closes: apache#55728
@Vamsi-klu

Copy link
Copy Markdown
Contributor Author

Could you review the leftover extra_dejson and sync get_connection hops on the ADF and Synapse async hooks? This is the same AsyncToSync triggerer-loop failure as #55179, now in AzureDataFactoryAsyncHook.get_async_conn / provide_targeted_factory_async and AzureSynapsePipelineAsyncHook.get_async_conn. The question is whether json.loads(conn.extra) on the async path, and get_async_connection in the Synapse hook, is the right leftover fix, and whether leaving the sync hooks on extra_dejson is correct.

cc @dabla @eladkal @Lee-W


Drafted-by: Cursor Grok 4.6 (no human review before posting)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure

1 participant