Add FFI query planner support

.ai/skills/audit-skill-md/SKILL.md

@@ -1,3 +1,9 @@
+---
+name: audit-skill-md
+description: Audit the user-facing skill at skills/datafusion_python/SKILL.md against the current public Python API. Find new APIs that should be documented, stale mentions of removed/renamed APIs, examples that drifted from current idiomatic style, and places that need a "requires datafusion-python NN or newer" note. Run after upstream syncs and before each release.
+argument-hint: [scope] (e.g., "session-context", "dataframe", "expr", "functions", "patterns", "pitfalls", "version-notes", "all")
+---
+
 <!---
   Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
@@ -17,12 +23,6 @@
   under the License.
 -->
 
----
-name: audit-skill-md
-description: Audit the user-facing skill at skills/datafusion_python/SKILL.md against the current public Python API. Find new APIs that should be documented, stale mentions of removed/renamed APIs, examples that drifted from current idiomatic style, and places that need a "requires datafusion-python NN or newer" note. Run after upstream syncs and before each release.
-argument-hint: [scope] (e.g., "session-context", "dataframe", "expr", "functions", "patterns", "pitfalls", "version-notes", "all")
----
-
 # Audit `skills/datafusion_python/SKILL.md`
 
 You are auditing the user-facing skill at

.ai/skills/check-upstream/SKILL.md

@@ -1,3 +1,9 @@
+---
+name: check-upstream
+description: Check if upstream Apache DataFusion features (functions, DataFrame ops, SessionContext methods, FFI types) are exposed in this Python project. Use when adding missing functions, auditing API coverage, or ensuring parity with upstream.
+argument-hint: [area] (e.g., "scalar functions", "aggregate functions", "window functions", "dataframe", "session context", "ffi types", "all")
+---
+
 <!---
   Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
@@ -17,12 +23,6 @@
   under the License.
 -->
 
----
-name: check-upstream
-description: Check if upstream Apache DataFusion features (functions, DataFrame ops, SessionContext methods, FFI types) are exposed in this Python project. Use when adding missing functions, auditing API coverage, or ensuring parity with upstream.
-argument-hint: [area] (e.g., "scalar functions", "aggregate functions", "window functions", "dataframe", "session context", "ffi types", "all")
----
-
 # Check Upstream DataFusion Feature Coverage
 
 You are auditing the datafusion-python project to find features from the upstream Apache DataFusion Rust library that are **not yet exposed** in this Python binding project. Your goal is to identify gaps and, if asked, implement the missing bindings.

.ai/skills/ffi-capsule-protocol/SKILL.md

@@ -0,0 +1,146 @@
+---
+name: ffi-capsule-protocol
+description: "TRIGGER — read before adding, changing, or reviewing any __datafusion_*__ capsule getter, any FFI_* export that asks for a TaskContextProvider or an extension codec, or any code that calls FFI_QueryPlanner::new / FFI_TableProvider::new / FFI_{Logical,Physical}ExtensionCodec::new. These methods are one protocol with a settled convention. Do not design it fresh; do not construct a SessionContext inside an extension library."
+argument-hint: "[getter name] (e.g., \"__datafusion_query_planner__\", \"table provider\", \"codec\", or omit to review the whole family)"
+---
+
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# FFI Capsule Protocol
+
+`datafusion-python` shares Rust objects with extension libraries through
+PyCapsules. Every hook is a dunder method named `__datafusion_<thing>__` that
+returns a capsule wrapping an FFI-safe struct. They are **one protocol**, not a
+collection of unrelated methods, and they have a settled convention that has
+already been migrated once (see `docs/source/user-guide/upgrade-guides.md`,
+DataFusion 52.0.0 and 55.0.0).
+
+## Rule 1 — enumerate the family before you change a member
+
+Do this first, every time. It takes one command and it is the whole point of
+this skill:
+
+```bash
+grep -rn "__datafusion_[a-z_]*__" --include="*.rs" crates/ examples/*/src/
+```
+
+Compare the signature you are about to write against what the others already
+do. If yours is shaped differently, that is a finding about your design, not
+about theirs.
+
+## Rule 2 — a getter takes the session it is being installed on
+
+```rust
+fn __datafusion_physical_extension_codec__<'py>(
+    &self,
+    py: Python<'py>,
+    session: Bound<'py, PyAny>,
+) -> PyResult<Bound<'py, PyCapsule>> { ... }
+```
+
+The host calls the getter and passes itself. That argument is how an extension
+library reaches things only the session has.
+
+`SessionContext` implements the same getters and ignores the argument, so a
+session satisfies the protocol too — `ctx.__datafusion_query_planner__()` and
+`ctx.__datafusion_query_planner__(ctx)` are both valid.
+
+## Rule 3 — never construct a `SessionContext` in an extension library
+
+The FFI constructors ask for things a library does not have:
+
+| Constructor | Wants | Take it from |
+|---|---|---|
+| `FFI_{Logical,Physical}ExtensionCodec::new` | `TaskContextProvider` | `ffi_task_context_provider_from_pycapsule(&session)` |
+| `FFI_TableProvider::new_with_ffi_codec` | logical codec | `ffi_logical_codec_from_pycapsule(session, None)` |
+| `FFI_QueryPlanner::new_with_ffi_codecs` | both codecs | `ffi_{logical,physical}_codec_from_pycapsule(session, None)` |
+
+`Arc::new(SessionContext::new())` is the wrong answer to all three, for two
+independent reasons:
+
+1. **It is the wrong registry.** Decode callbacks resolve names against
+   whatever provider the codec carries. An empty session resolves nothing, so a
+   function the host registered with `register_udf` is invisible to a node that
+   references it by name.
+2. **It dangles.** `FFI_TaskContextProvider` downgrades its provider to a
+   `Weak`. A context built inline in the getter is dropped before the capsule
+   is ever used, and every callback then fails with `TaskContextProvider went
+   out of scope over FFI boundary`.
+
+Prefer the `*_with_ffi_codec(s)` constructors when they exist. They take
+prebuilt codecs that already carry the host's provider, so there is no provider
+parameter to get wrong.
+
+## Rule 4 — the helpers live in `crates/util/src/lib.rs`
+
+`ffi_logical_codec_from_pycapsule`, `ffi_physical_codec_from_pycapsule`,
+`ffi_query_planner_from_pycapsule`, `ffi_task_context_provider_from_pycapsule`,
+`table_provider_from_pycapsule`. Each takes the object and, where relevant, an
+`Option<&Bound<PyAny>>` session:
+
+- `Some(session)` — importing a *foreign* object; the getter needs the session.
+- `None` — the object already *is* a session and is being asked for what it
+  holds.
+
+Adding a getter means adding a helper here, not hand-rolling capsule
+extraction at the call site.
+
+## Rule 5 — changing a getter's signature is a breaking change
+
+Extension libraries implement these methods. A signature change breaks every
+one of them, and the failure is a bare `TypeError` from a `call1`. So:
+
+- Add a section to `docs/source/user-guide/upgrade-guides.md` with before/after
+  Rust, matching the 52.0.0 and 55.0.0 entries.
+- Add the `api change` label to the PR.
+- Map the `TypeError` to a diagnosable message. `call_capsule_getter` in
+  `crates/util/src/lib.rs` already does this; reuse it.
+- Update `python/datafusion/context.py` and
+  `python/datafusion/user_defined.py`, where the `Protocol` type hints for
+  these methods live.
+
+## Rule 6 — a fork must rebind the codecs it carries
+
+Installing a foreign query planner **forks** the session
+(`PySessionContext::derived_parts`), because installing one writes to
+`SessionState` and the receiver must not be modified. A foreign codec holds an
+`FFI_TaskContextProvider` pointing at the session it was installed on, so the
+fork rebinds each one onto itself via
+`PySessionContext::rebound_{logical,physical}_codec`. Skip that and decode
+callbacks answer from the pre-fork registry.
+
+Rebinding relies on `FFI_{Logical,Physical}ExtensionCodec::new` adopting the
+provider on the already-foreign path, which needs DataFusion 55.1.0 or newer
+(apache/datafusion#24722). It clones the handle before overwriting, so the
+receiver keeps its own binding — assert that, not just that the fork works.
+A codec this library owns round-trips unchanged, so the rebind is safe to apply
+unconditionally.
+
+## Where the truth is
+
+- `docs/source/contributor-guide/ffi.md` — the protocol, the fork caveat.
+- `docs/source/user-guide/upgrade-guides.md` — every past migration.
+- `examples/datafusion-ffi-example/src/` — provider, catalog, function, codec
+  getters, all in current form.
+- `examples/datafusion-ffi-query-planner-example/src/planner.rs` — planner
+  getter.
+- `examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py`
+  — `require_udf_on_decode` proves which session a decode callback resolves
+  against. Extend these when touching the protocol.

.ai/skills/make-pythonic/SKILL.md

@@ -1,3 +1,9 @@
+---
+name: make-pythonic
+description: Audit and improve datafusion-python functions to accept native Python types (int, float, str, bool) instead of requiring explicit lit() or col() wrapping. Analyzes function signatures, checks upstream Rust implementations for type constraints, and applies the appropriate coercion pattern.
+argument-hint: [scope] (e.g., "string functions", "datetime functions", "array functions", "math functions", "all", or a specific function name like "split_part")
+---
+
 <!---
   Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
@@ -17,12 +23,6 @@
   under the License.
 -->
 
----
-name: make-pythonic
-description: Audit and improve datafusion-python functions to accept native Python types (int, float, str, bool) instead of requiring explicit lit() or col() wrapping. Analyzes function signatures, checks upstream Rust implementations for type constraints, and applies the appropriate coercion pattern.
-argument-hint: [scope] (e.g., "string functions", "datetime functions", "array functions", "math functions", "all", or a specific function name like "split_part")
----
-
 # Make Python API Functions More Pythonic
 
 You are improving the datafusion-python API to feel more natural to Python users. The goal is to allow functions to accept native Python types (int, float, str, bool, etc.) for arguments that are contextually always or typically literal values, instead of requiring users to manually wrap them in `lit()`.

.github/workflows/build.yml

@@ -186,7 +186,7 @@ jobs:
           manylinux: "2_28"
 
       # FFI test wheel only needs to be built once per platform; gate to abi3.
-      - name: Build FFI test library
+      - name: Build FFI provider test library
         if: matrix.python-tag == 'abi3'
         uses: PyO3/maturin-action@v1
         with:
@@ -196,6 +196,16 @@ jobs:
           args: --out dist
           rustup-components: rust-std
 
+      - name: Build FFI query planner test library
+        if: matrix.python-tag == 'abi3'
+        uses: PyO3/maturin-action@v1
+        with:
+          target: x86_64-unknown-linux-gnu
+          manylinux: "2_28"
+          working-directory: examples/datafusion-ffi-query-planner-example
+          args: --out dist
+          rustup-components: rust-std
+
       - name: Archive wheels
         uses: actions/upload-artifact@v7
         with:
@@ -207,7 +217,9 @@ jobs:
         uses: actions/upload-artifact@v7
         with:
           name: test-ffi-manylinux-x86_64
-          path: examples/datafusion-ffi-example/dist/*
+          path: |
+            examples/datafusion-ffi-example/dist/*
+            examples/datafusion-ffi-query-planner-example/dist/*
 
   # ============================================
   # Build - Linux ARM64

.github/workflows/test.yml

@@ -93,11 +93,15 @@ jobs:
           uv venv --python "${{ steps.setup-python.outputs.python-path }}"
           VENV_PY="$PWD/.venv/bin/python"
           uv sync --python "$VENV_PY" --dev --no-install-package datafusion
+          # Search recursively: the FFI artifact bundles more than one
+          # project, so upload-artifact keeps a `<project>/dist/` prefix
+          # and the wheels are not all at the top of wheels/.
           WHEELS=$(find wheels/ -name "*.whl")
           if [ -n "$WHEELS" ]; then
             echo "Installing wheels:"
             echo "$WHEELS"
-            uv pip install --python "$VENV_PY" wheels/*.whl
+            # shellcheck disable=SC2086  # intentional split on newlines
+            uv pip install --python "$VENV_PY" $WHEELS
           else
             echo "ERROR: No wheels found!"
             exit 1
@@ -121,6 +125,8 @@ jobs:
         run: |
           cd examples/datafusion-ffi-example
           uv run --no-project pytest python/tests/_test*.py
+          cd ../datafusion-ffi-query-planner-example
+          uv run --no-project pytest python/tests/_test*.py
 
       - name: Run tpchgen-cli to create 1 Gb dataset
         if: matrix.wheel-tag == 'abi3'

AGENTS.md

@@ -35,7 +35,24 @@ Skills follow the [Agent Skills](https://agentskills.io) open standard. Each ski
 
 To discover what skills are available, list `.ai/skills/` and read each
 `SKILL.md`. The frontmatter `name` and `description` fields summarize the
-skill's purpose.
+skill's purpose. Some descriptions begin with `TRIGGER —`; those are not tasks
+to run on request but conventions to read *before* writing code that meets the
+stated condition.
+
+## FFI Capsule Protocol
+
+The `__datafusion_*__` capsule getters are one protocol with a settled
+convention. Before adding or changing one, read
+[`.ai/skills/ffi-capsule-protocol/SKILL.md`](.ai/skills/ffi-capsule-protocol/SKILL.md).
+
+## Documentation Sources
+
+Search and edit `docs/source/`. `docs/temp/` is gitignored build output that
+`grep -r` will surface with stale copies of the same pages.
+
+Before changing a public API, check
+`docs/source/user-guide/upgrade-guides.md` for how the same API family was
+migrated previously. Follow the established pattern rather than inventing one.
 
 ## Pull Requests
 
@@ -48,7 +65,10 @@ Every pull request must follow the template in
 3. **What changes are included in this PR?** — Summarize the individual changes.
 4. **Are there any user-facing changes?** — Note any changes visible to users
    (new APIs, changed behavior, new files shipped in the package, etc.). If
-   there are breaking changes to public APIs, add the `api change` label.
+   there are breaking changes to public APIs, add the `api change` label **and
+   add a section to `docs/source/user-guide/upgrade-guides.md`** showing the
+   before and after. This applies to FFI hook method signatures, which
+   extension libraries implement.
 
 ## Pre-commit Checks