Skip to content

fix: Airflow demo improvements - #457

Open
adwk67 wants to merge 2 commits into
mainfrom
fix/airflow-demo-improvements
Open

adwk67 wants to merge 2 commits into
mainfrom
fix/airflow-demo-improvements

Conversation

@adwk67

@adwk67 adwk67 commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

This PR fixes:

  • replaces cluster-wide permissions with namespace-scoped ones
  • replaces minio and mc images from quay.io/minio with ones from docker.io/pgsty/mc
  • the airflow versions so it uses 3.3.1 (and not 3.1.6 as determined previously by the podOverrides)

And changes:

  • the spark sensor such that logs are streamed from the driver and executor pods to the Airflow UI
  • N.B. this does not necessitate granting Airflow any extra permissions that it has already anyway to access Spark logs (at whatever level)
image

@adwk67 adwk67 self-assigned this Sep 30, 2026
@sweb
sweb self-requested a review September 30, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Development: In Review

Development

Successfully merging this pull request may close these issues.

1 participant