Skip to content

fix(deps): cap pyarrow below 26 on Python 3.12 - #140

Open
tkislan wants to merge 5 commits into
mainfrom
tk/cap-pyarrow-python-3.12
Open

tkislan wants to merge 5 commits into
mainfrom
tk/cap-pyarrow-python-3.12

Conversation

@tkislan

@tkislan tkislan commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #139.

Fresh installs of the toolkit on Python 3.12 get pyarrow 26.0.0, which refuses to import with NumPy 1.x, while 3.12 is pinned to numpy<2. pyarrow 26 doesn't declare a numpy requirement, so pip doesn't catch the conflict, and the chart and SQL functions then fail to import. This PR caps pyarrow at <26 on Python 3.12 only.

Changes

  • pyproject.toml: the 3.12 requirement becomes pyarrow>=23.0.1,<26, with a comment explaining why.
  • poetry.lock: only content-hash changes. The locked pyarrow for Python 3.12 and later is already 24.0.0, so locked environments are unchanged. A plain poetry lock also drops about 20 unrelated duplicate entries (pydantic, pyzmq, shapely, …), the same thing it does on main (see fix(deps): resolve pip-audit failures #133), so those were left alone.
  • ci.yml: new wheel-install-check job (Wheel Install Check). Every other job that installs the toolkit's dependencies takes them from poetry.lock, which is why CI didn't notice this break. For Python 3.10–3.14, the job builds the wheel, installs it with the [server] extra into a clean venv without the lock, and checks that DeepnoteChart, execute_sql and execute_sql_with_connection_json are exported. It checks the names because safe_import swallows import errors, so import deepnote_toolkit alone still succeeds when they fail. The job always resolves the latest upstream releases, so an upstream break can turn it red on PRs that didn't cause it.

Python 3.10 and 3.11 already cap pyarrow at 18 or lower, and 3.13 and later require numpy>=2.1, so no other version needs this.

Why not NumPy 2 on 3.12

Allowing NumPy 2 on 3.12, as 3.13 and later already do, would also fix this. But it moves every 3.12 environment to a new NumPy major version, which breaks user code that relies on NumPy 1.x APIs removed in 2.0. It would also need pandas 2.2.2 or later on 3.12, where the floor is currently 2.1.0. That deserves its own PR.

The trade-off: until 3.12 moves to NumPy 2, it can't get pyarrow fixes released only in 26 or later.

Testing

  • Before: on Python 3.12.15, pip install "deepnote-toolkit[server]==2.9.0" from PyPI into a fresh venv resolves numpy 1.26.4 with pyarrow 26.0.0, and DeepnoteChart, execute_sql and execute_sql_with_connection_json fail to import. A wheel built from main behaves the same.
  • After: a wheel built from this branch resolves pyarrow 25.0.1 in a fresh 3.12 venv, and all three import.
  • The chart, SQL, duckdb, SQL caching and artifacts unit tests pass in that fresh 3.12 environment (pyarrow 25.0.1, numpy 1.26.4): 379 passed, 2 skipped. Both skips also happen on main and are unrelated to pyarrow.
  • poetry check --lock passes.
  • Wheel Install Check, locally: the job's install and check pass on Python 3.10.22, 3.11.17, 3.12.15, 3.13.13 and 3.14.8, and all 19 exports import on each. Against a 3.12 install of main's wheel, the same check fails with deepnote_toolkit is missing exports: ['DeepnoteChart', 'execute_sql', 'execute_sql_with_connection_json'].
  • actionlint 1.7.12 and the pre-commit hooks pass on ci.yml.

The unit-test jobs install from poetry.lock, which this PR doesn't change, so they run the same environment as main. The fix is confirmed by the new Wheel Install Check job.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HwkxN8YriHhdXBwMc4sd8h

Summary by CodeRabbit

  • Bug Fixes
    • Updated the Python 3.12 PyArrow version range to avoid a version that requires NumPy 2, improving compatibility with the supported NumPy version.
  • Tests
    • Added automated checks that build and install the package across Python 3.10–3.14 and verify key features are available from the installed package.

pyarrow 26.0.0 refuses to import with NumPy 1.x but doesn't declare a
numpy requirement, and Python 3.12 is pinned to numpy<2. Fresh installs
on 3.12 therefore resolved numpy 1.26.4 with pyarrow 26.0.0, and the
chart and SQL functions failed to import.

The locked pyarrow (24.0.0) already satisfies the cap, so poetry.lock
only gets a new content-hash.

Fixes #139

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HwkxN8YriHhdXBwMc4sd8h
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

📦 Python package built successfully!

  • Version: 2.9.0.dev6+0912bda
  • Wheel: deepnote_toolkit-2.9.0.dev6+0912bda-py3-none-any.whl
  • Install:
    pip install "deepnote-toolkit @ https://deepnote-staging-runtime-artifactory.s3.amazonaws.com/deepnote-toolkit-packages/2.9.0.dev6%2B0912bda/deepnote_toolkit-2.9.0.dev6%2B0912bda-py3-none-any.whl"

@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.17%. Comparing base (11a8bed) to head (68c2ae4).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #140   +/-   ##
=======================================
  Coverage   78.17%   78.17%           
=======================================
  Files         120      120           
  Lines        6837     6837           
  Branches      990      990           
=======================================
  Hits         5345     5345           
  Misses       1182     1182           
  Partials      310      310           
Flag Coverage Δ
combined 78.17% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@deepnote-bot

deepnote-bot commented Oct 23, 2026 •

Copy link
Copy Markdown

🚀 Review App Deployment Started

📝 Description 🌐 Link / Info
🌍 Review application ra-140
🔑 Sign-in URL Click to sign-in
📊 Application logs View logs
🔄 Actions Click to redeploy
🚀 ArgoCD deployment View deployment
⏰ Last deployed 2026-10-25 08:30:22 (UTC)
📜 Deployed commit 79198db90de22f2acfbdfc22955ce4775df3bf66
🛠️ Toolkit version 0912bda

tkislan and others added 4 commits October 23, 2026 21:34
Every job that installs the toolkit's dependencies takes them from
poetry.lock, so none of them saw pyarrow 26 break fresh pip installs on
Python 3.12. The new job builds the wheel, installs it with its [server]
extra into a clean venv without the lock on Python 3.10-3.14, and checks
that DeepnoteChart, execute_sql and execute_sql_with_connection_json are
exported. It checks the names because safe_import swallows import errors,
so `import deepnote_toolkit` alone still succeeds when they fail.

Refs #139

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HwkxN8YriHhdXBwMc4sd8h
The one-line python -c check was hard to read. Running it as a heredoc
script lets it use an if block and exit with a plain message instead of
an AssertionError traceback, and leaves room for comments on why the
step leaves the checkout and why it checks the exports by name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HwkxN8YriHhdXBwMc4sd8h
The unit-test jobs install the project from the checkout and never build
or install a wheel, so "wheel install" is what sets this job apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HwkxN8YriHhdXBwMc4sd8h
@tkislan
tkislan marked this pull request as ready for review October 25, 2026 08:38
@tkislan
tkislan requested a review from a team as a code owner October 25, 2026 08:38
@tkislan
tkislan requested a review from mfranczel October 25, 2026 08:38
@tkislan

tkislan commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: dc72fe00-28f7-409b-be10-ee2bb1614a07
Commits

Reviewing files that changed from the base of the PR and between 11a8bed and 68c2ae4.

Files ignored due to path filters: 1
  • poetry.lock is excluded by !**/*.lock
Files selected for processing: 2
  • .github/workflows/ci.yml
  • pyproject.toml

Included review availability: This review used your included allowance. 3 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

Walkthrough

The Python 3.12 PyArrow requirement now caps the version below 26. CI also builds and installs the wheel with the server extra on Python 3.10–3.14, then checks for three expected exports.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: m1so

Merge Risk: Minimal · up to 68c2a

The dependency cap and wheel check are aligned, with no established issue blocking merge. Complete the normal CI checks before merging.

Pre-merge checks: Passed 6
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely identifies the main dependency change: capping PyArrow below version 26 for Python 3.12.
Linked Issues check Passed Issue #139 requires preventing fresh Python 3.12 installs from resolving NumPy <2 with pyarrow 26 or newer. pyproject.toml now requires pyarrow>=23.0.1,<26 for Python 3.12. The new `wheel-install-…
Out of Scope Changes check Passed The reviewed changes stay within issue #139. The dependency cap directly fixes the Python 3.12 resolution failure. The CI job provides automated coverage for the fresh-install failure and verifies the…
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Updates Docs Passed No new feature is implemented. The PR only caps pyarrow for Python 3.12, updates the lock hash, and adds a CI wheel-install check. No public API declarations or user-facing behavior are added, so th…
Full details: Linked Issues check

Explanation

Issue #139 requires preventing fresh Python 3.12 installs from resolving NumPy <2 with pyarrow 26 or newer. pyproject.toml now requires pyarrow&gt;=23.0.1,&lt;26 for Python 3.12. The new wheel-install-check job installs the wheel without the lock file on Python 3.10–3.14 and checks DeepnoteChart, execute_sql, and execute_sql_with_connection_json. The reported Python 3.12 validation resolves pyarrow 25.0.1 and imports all three exports.

Full details: Out of Scope Changes check

Explanation

The reviewed changes stay within issue #139. The dependency cap directly fixes the Python 3.12 resolution failure. The CI job provides automated coverage for the fresh-install failure and verifies the affected exports. No unrelated change is demonstrated.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

Full details: Updates Docs

Explanation

No new feature is implemented. The PR only caps pyarrow for Python 3.12, updates the lock hash, and adds a CI wheel-install check. No public API declarations or user-facing behavior are added, so this documentation check does not apply.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fresh installs on Python 3.12 get pyarrow 26, which needs NumPy 2, so charts and SQL fail to import

2 participants