fix(amazon): chain deferral in EmrServerlessDeleteApplicationOperator so DeleteApplication is actually called - #72124
Open
waterWang wants to merge 1 commit into
Conversation
… to actually call DeleteApplication (apache#72123) When `deferrable=True`, `EmrServerlessDeleteApplicationOperator.execute()` never reaches the `delete_application` call because the parent operator `EmrServerlessStopApplicationOperator` defers with a hardcoded `method_name="execute_complete"`, which raises `TaskDeferred` before the subclass can issue the delete. On resume the subclass's `execute_complete` handler runs, but only validates the stop event and logs success — the application is never deleted. Fix: - Add a class attribute `stop_complete_method_name` to the stop operator so subclasses can override where to resume after the application stops. - `EmrServerlessDeleteApplicationOperator` sets `stop_complete_method_name = "stop_complete"` and implements `stop_complete()` to issue `DeleteApplication` and defer on the `EmrServerlessDeleteApplicationTrigger`, leaving `execute_complete` to handle only the delete trigger's event. This is the same chained-deferral pattern already used by the stop operator for the `force_stop` path (`execute` → `stop_application` → `execute_complete`). Signed-off-by: waterWang <waterWang@users.noreply.github.com>
boring-cyborg
Bot
added
area:providers
provider:amazon
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #72123
Problem
When
deferrable=True,EmrServerlessDeleteApplicationOperatornever calls theDeleteApplicationAPI. The task succeeds and logsEMR serverless application <id> deleted successfully, but the application is only stopped, never deleted.Root cause
EmrServerlessDeleteApplicationOperatorsubclassesEmrServerlessStopApplicationOperator. In deferrable mode the parent'sexecute()defers with a hardcodedmethod_name="execute_complete", which raisesTaskDeferredbefore the subclass'sexecute()can reach thedelete_application()call. On resume, the subclass'sexecute_complete()handler runs — but that handler only validates the stop trigger's event and logs success, soDeleteApplicationis never issued.Fix
Follow the chained-deferral pattern the stop operator already uses for the
force_stoppath (execute→stop_application→execute_complete):stop_complete_method_name: str = "execute_complete"toEmrServerlessStopApplicationOperator, used in place of both hardcodedmethod_name="execute_complete"occurrences.EmrServerlessDeleteApplicationOperatorsetsstop_complete_method_name = "stop_complete"and implementsstop_complete()to issueDeleteApplicationand defer onEmrServerlessDeleteApplicationTrigger, leavingexecute_completeto handle only the delete trigger's event.Backwards compatible:
execute_completestill exists for tasks already deferred by an older version at upgrade time.Verification
test_delete_application_deferrable_deletes_after_stopproves the full chained flow: first defer is on theStopApplicationTriggerresuming atstop_complete, which callsdelete_applicationand defers on theDeleteApplicationTriggerresuming atexecute_complete.