Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks - #72098
Use the operator's AWS settings for deferred Neptune, MWAA, SSM tasks#72098SEPURI-SAI-KRISHNA wants to merge 2 commits into
Conversation
An AwsBaseOperator/AwsBaseSensor subclass resolves region_name, verify and botocore_config in __init__, but did not hand them to the trigger it defers to. The trigger builds its own hook, so the deferred half of the task reached AWS with the default region, SSL verification silently re-enabled, and any custom botocore timeouts or retries discarded. The triggers already accept all three, so only the call sites were missing.
|
Thanks for continuing this cleanup series. The fix itself looks correct. However, two bigger-picture thoughts, neither blocking for this PR. Since this is the 6th PR of this shape, it might be worth stepping back:
Related small thing: the |
| running Airflow in a distributed manner and aws_conn_id is None or | ||
| empty, then default boto3 configuration would be used (and must be | ||
| maintained on each worker node). | ||
| :param region_name: AWS region_name. If not specified then the default boto3 behaviour is used. |
There was a problem hiding this comment.
Let's add verify here as well?
| with pytest.raises(TaskDeferred) as exc_info: | ||
| op.execute({}) | ||
|
|
||
| assert exc_info.value.trigger.region_name == REGION_NAME |
There was a problem hiding this comment.
Let's assert on the serialized attributes (trigger.serialize()[1]) like test_neptune.py does, instead of the pre-serialization attributes
Every Neptune Analytics operator accepts `verify` through the shared AWS base class, but none of the seven class docstrings mentioned it, so the rendered provider docs gave users no way to discover it. Two of those docstrings even carried a stray blank line where the entry belonged. The deferral tests now compare the trigger's serialized payload rather than its attributes. Serialization is what actually crosses into the triggerer process, and it passes values through `prune_dict`, so an attribute-level assertion can pass while the setting is silently dropped on the way there. This matches the assertion style already used for the Neptune cluster operators.
|
Thanks for the review, both addressed.
Serialized assertions: switched all twelve tests to I re-ran the counterfactual after the rewrite: stripping the 36 fix lines On the broader points, all three sound right to me:
Let me know if you'd prefer the tracking issue before or after the |
|
Opened #72144 as the tracking issue. One correction to my earlier reply: I said I counted 44 sites passing none of the three. Your 43 is right. I re-derived the audit by parsing every The sweep also sharpened your second point more than I expected. Of the 61 remaining sites, only 7 can be fixed at the call site, the other 54 have trigger subclasses whose Full per-site breakdown by fix shape is in the issue. |
AwsBaseOperatorandAwsBaseSensorboth resolveregion_name,verifyandbotocore_configin__init__, so every subclass carries them. When theseoperators defer, they build a trigger that constructs its own hook, and the
three fields were never passed along. The deferred half of the task then talks
to AWS with different settings than the synchronous half: the default region
instead of the configured one, SSL verification silently back on, and custom
botocore timeouts/retries dropped.
The triggers already support all three.
SsmRunCommandTriggernames themexplicitly, and the Neptune Analytics and MWAA triggers take
**kwargsstraightthrough to
AwsBaseWaiterTrigger, whosehook()builds the client from them.The Neptune trigger docstrings even document
:param region_name:. Only theoperator side was missing, so this is a pure call-site fix, no trigger
signatures change.
Twelve sites: eight in Neptune Analytics, one in the MWAA operator, two in the
MWAA sensors, one in the SSM sensor. The SSM operator already passed all three,
which is what the deferred sensor beside it should have been doing.
Same bug and same fix shape as #52904, #67508, #67876, #71646 and #71857.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines