Skip to content

Migrate exasol provider to pyexasol 2.x and remove the <2 cap - #72119

Open
joseph-bergin wants to merge 1 commit into
apache:mainfrom
joseph-bergin:fix/exasol-pyexasol-2x
Open

Migrate exasol provider to pyexasol 2.x and remove the <2 cap#72119
joseph-bergin wants to merge 1 commit into
apache:mainfrom
joseph-bergin:fix/exasol-pyexasol-2x

Conversation

@joseph-bergin

Copy link
Copy Markdown

Migrates the exasol provider to pyexasol 2.x and removes the pyexasol>=0.26.0,<2 cap added in #68933.

What the cap was hiding

pyexasol 2.x ships type information, so mypy now checks these calls. Against pyexasol 2.3.2 the hook reports 6 errors:

  • ExaConnection.execute / export_to_pandas are typed query_params: dict | None, but the hook forwarded Iterable | Mapping[str, Any] | None (lines 148, 191, 208, 337).
  • execute is typed query: str, but get_records / get_first forwarded the str | list[str] that DbApiHook accepts (lines 191, 208).

get_conn also reused one local for both the Airflow Connection and the ExaConnection returned by pyexasol.connect; that one only surfaces with Airflow's own sources on mypy_path, and is fixed here too.

Changes

  • get_conn uses separate locals (airflow_conn and the returned connection).
  • _to_query_params passes None through, copies any mapping into a dict, and rejects sequences.
  • _to_single_statement unwraps the str and rejects a list.
  • Cap removed from providers/exasol/pyproject.toml.

On the two rejected paths

Both were already broken against a real Exasol, so only the error message changes:

  • ExaStatement._format_query does self.connection.format.format(query, **query_params). A tuple or list there raises TypeError: argument after ** must be a mapping, not tuple, so positional parameters have never reached the driver intact.
  • ExaConnection.execute runs exactly one statement, so a list[str] was never executable in get_records / get_first. run() is the method that splits and loops, and the new message points there.

test_run_with_parameters passed ("param1", "param2") and passed only because the connection is a MagicMock. It now uses a mapping, with new coverage for mapping normalization and for both rejections.

If you would rather keep those paths permissive and satisfy mypy with cast instead, I am happy to switch — I went with explicit errors because the status quo is a confusing failure deep inside the driver.

Verification

Against pyexasol 2.3.2, Python 3.11:

  • mypy providers/exasol/src/ — 6 errors before, 0 after
  • pytest providers/exasol/tests/unit/exasol/ — 58 passed (previously 53 passed with 1 failing after the migration)
  • ruff check and ruff format --check at the pinned 0.16.3 — clean

closes: #69123


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5) following the guidelines

pyexasol 2.x ships type information, which surfaced six mypy errors in the
exasol hook and forced the `pyexasol>=0.26.0,<2` cap added in apache#68933.

Adapt the hook to the driver's real contract:

- `get_conn` reused one local for both the Airflow `Connection` and the
  `ExaConnection` returned by `pyexasol.connect`. Use separate names.
- `ExaConnection.execute`/`export_to_pandas` take `query_params: dict | None`
  and render named placeholders via `format(query, **query_params)`, so a
  positional sequence has always raised `TypeError: argument after ** must be
  a mapping`. Normalize mappings to `dict` and reject sequences up front with
  a message that names the fix.
- `execute` runs exactly one statement, so the `str | list[str]` that
  `DbApiHook` allows was never executable here. Reject lists and point at
  `run()`, which does handle them.

Both rejected paths were already broken against a real Exasol; only the error
message changes. `test_run_with_parameters` passed a tuple and passed only
because the connection is mocked, so it now uses a mapping, with new coverage
for mapping normalization and for both rejections.

Closes: apache#69123

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@boring-cyborg

boring-cyborg Bot commented Aug 26, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate exasol provider to pyexasol 2.x and remove the <2 cap

1 participant