Allow Variable.get_variable_from_secrets to reuse session - #72121
Closed
saitejabandaru-in wants to merge 1 commit into
Closed
Allow Variable.get_variable_from_secrets to reuse session#72121saitejabandaru-in wants to merge 1 commit into
saitejabandaru-in wants to merge 1 commit into
Conversation
This adds an optional keyword-only `session=None` argument to `Variable.get_variable_from_secrets` and forwards it to `MetastoreBackend`. This allows callers running under `prohibit_commit` (like the scheduler) to resolve variables from the secrets chain without tripping unexpected commits via `@provide_session`, enabling deduplication of secrets backend iteration loops in `_process_dagrun_deadline_alerts`.\n\nCloses: apache#71801
Contributor
|
This is already being worked on here: #72121 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #71801.
Motivation
When resolving
VariableIntervaldeadlines (such as in_process_dagrun_deadline_alerts), the scheduler operates underprohibit_commit. However, queryingMetastoreBackend.get_variablethroughVariable.get_variable_from_secretsdoes not pass down the session, which triggers@provide_sessionto create a scoped session that automatically commits upon exiting, thus trippingUNEXPECTED COMMITguards.Because of this, callers had to duplicate the secrets backend iteration loop directly in
dag.py(see #68917) instead of utilizingget_variable_from_secrets.Changes
session=Nonekeyword-only argument toVariable.get_variable_from_secrets.secrets_backendisMetastoreBackend, and ifsessionis provided, forward it in thecall_secrets_backend_methodarguments.dag.pyloop.