Add stub generation support for overloads.

[tychedelia]

Partially addresses #5137.

We need support for overloads in stub generation in our project. I mostly copied the argument parsing code and tried to extract that as shared helpers for the overload support. I'm not totally sure about the error path, this may be a bit brittle, but I added some baseline tests to ensure it works.

[Tpt]

Thank you very much for working on this large topic. I have not reviewed the MR in details yet but I have a few questions.

About the #[pyo3(...)] attribute syntax I would tend to think that something like the following would be clearer and more aligned with signature:

#[pyo3(overloads = [
    (foo: "int") -> "int",
    (foo: "float") -> "float"
])]
fn my_function() {
}

this enforces that only a single attribute set overloads making sure there is only a single source of truth for overloads, preventing footguns that two different macros inserting conflicting overloads. But I don't feel strongly on that and @davidhewitt might disagree.

It would be great to provide more careful validations like checking if overloads are consistent with the default signature (same default value...). But I am fine to have that in follow-ups to avoid a too large MR.

On the emitted stubs, the python typing doc state: All variants of overloaded functions and methods must have an @overload decorator. Do not include the implementation’s final non-@overload-decorated definition.. I would tend to think we should follow that.

Thank you aggain!

[tychedelia]

I would tend to think that something like the following would be clearer and more aligned with signature:

Okay, I think that makes sense. I wasn't totally sure about the whether to go for single attr or multiple, but looking at your example I think I prefer this as well.

On the emitted stubs, the python typing doc state: "All variants of overloaded functions and methods must have an @overload decorator. Do not include the implementation’s final non-@overload-decorated definition.". I would tend to think we should follow that.

Yeah, mypy actually just caught this, I've fixed it up.

[tychedelia]

Okay I'm waffling back and forth on the syntax, but happy to support whatever the project wants I'm agnostic.

[Tpt]

Okay I'm waffling back and forth on the syntax, but happy to support whatever the project wants I'm agnostic.

I think I prefer:

#[pyo3(overloads = [
    (foo: "int") -> "int",
    (foo: "float") -> "float"
])]

but I do not feel strongly so glad to change my mind on this topic. This is also something we can tweak in follow ups.

[Tpt]

open design question: do we want to have overloads field in the introspection JSON with the olverloads or be closer to the final Python stubs and just have multiple Function chunks with the @overload decorator (and not chunk for the base function)?

I feel that it's underlying a deeper question on if the introspection data must be a more "semantic" AST or more "syntactic" one closer to the Python stubs.

[davidhewitt]

Okay I'm waffling back and forth on the syntax, but happy to support whatever the project wants I'm agnostic.

I think I prefer:

#[pyo3(overloads = [
    (foo: "int") -> "int",
    (foo: "float") -> "float"
])]

but I do not feel strongly so glad to change my mind on this topic. This is also something we can tweak in follow ups.

I agree with @Tpt here. With overloads as a multiple, it's possible to write this all nested inside a single attribute

#[pyfunction(
    overloads = [
        (x: "int") -> "int",
        (x: "str") -> "str",
    ],
    signature = (x)
)]
fn with_overloads<'py>(x: Any<'py>) -> Any<'py> {
    x
}

I also wonder about variants where the signature itself goes with the overloads, e.g.

#[pyfunction(
    signature = [
        (x: "int") -> "int",
        (x: "str") -> "str",
        (x)
    ],
)]
fn with_overloads<'py>(x: Any<'py>) -> Any<'py> {
    x
}

but I don't think that feels quite as good.

---

Also, I haven't checked the implementation here yet, but I think we should be able to enforce the overloads are compatible with the signature at compile time (e.g. positional arguments, argument names, kwarg names, required arguments etc), and emit helpful errors where they disagree.

[Tpt]

I also wonder about variants where the signature itself goes with the overloads, e.g.
but I don't think that feels quite as good.

I agree. Also, we still need an authoritative source of truth for the default values and the full signature as exposed to the Python interpreter. The other option would be to merge all the overloads together but it looks less intuitive for me.

But I think we should be able to enforce the overloads are compatible with the signature at compile time (e.g. positional arguments, argument names, kwarg names, required arguments etc), and emit helpful errors where they disagree.

+1. Glad to do it in a follow up if @tychedelia prefers not to do it.