Python: table-driven registration to shrink generated modules - #42
Conversation
Replace the per-function `TryAddFunc<...>()` calls that the `MB_FUNC` and `MB_CLASS` macros used to emit with constexpr `FuncRow` tables plus a shared runtime driver (`RegisterFuncRow()`). Names, comments, parameters, and default arguments become plain data; the only remaining per-function templates are a small call thunk carrying the target function pointer (`FuncRowThunk`) and one `FuncRowRegistrar` instantiation per distinct signature shape. This shrinks large generated modules considerably and speeds up their compilation (a synthetic 800-method module: -17% binary size, -20% compile time with clang at -Oz), while keeping the resulting bindings identical (pydoc output is byte-for-byte the same, overload resolution, default arguments, operator injection incl. the reversed forms, deprecation warnings, and the GIL handling all behave as before). Conversion operators, `TryAddFuncSimple()`, and custom bindings still go through `TryAddFunc()` directly.
Same treatment as the methods in the previous commit: `MemberVarRow` tables plus a shared driver replace the per-field inlined `TryAddMemberVar[Static]()` -> `def_property...()` chains. Per field only the typed accessor thunks survive; the pybind11 property construction is done once per (class, field type) shape, and `def_property_static()`'s tail is replicated type-erased in `DefPropertyLow()`. The `_offsetof_*` static properties become plain data served by one shared lambda instead of one lambda per field (this also required replacing the statement-expression `MB_PB11_OFFSETOF` with a declaration-level warning suppression, since statement expressions aren't allowed in constant expressions on GCC). `TryAddMemberVar[Static]()` remain available for custom bindings.
One shared `.value()` loop per enum instead of a straight-lined call per element. Values are bit-cast through `std::int64_t` (well-defined in C++20), verified to round-trip 64-bit unsigned enumerators.
Passing `is_method`/policy/comment to the `cpp_function` constructor changed the rendered signature and docstring texts (the setter's value parameter became `arg0` instead of `arg1`, and the comment leaked into the setter's docstring), which showed up as ~9k changed lines in MeshLib's mrmeshpy.pyi. Instead, construct the getter/setter bare and then apply the extras to the already-rendered records, exactly like `class_::def_property_static()` does (`get_function_record()` is private there, so a copy of it is included).
The previous commit copied upstream master's `get_function_record()`, but different pybind11 versions/forks spell the helpers differently (and the `PyCFunction_GET_SELF` macro doesn't exist under the limited API at all). Use the raw capsule API instead, passing the capsule's own name back to `PyCapsule_GetPointer` to sidestep the version-specific name checks.
|
Too much copypaste from pybind internals, I'm not a fan. If Pybind updates something internally, this could break. Also it's quite hard to read. Can discuss further when Fedor is back from vacation. |
|
Probably this code is not of good quality, but it shows that binaries produced by mrbind can be shrunk a lot. Can you achieve the same goal with a better code? Don't we have a fork of pybind11, which we can modify as necessary? |
|
Redone as #44, which reimplements nothing from pybind11 — the set of It turned out the fork doesn't need changing either. The only reason this PR hand-rolled The cost is that a few registrars are now per-class instead of shared across classes. On a synthetic module that's about two percentage points of |
2 participants
Reworks the pybind11 target to register methods, free functions, fields, and enum elements through
constexprdata tables instead of per-entity straight-lined template instantiation chains. This is aimed at large generated modules: MeshLib'smrmeshpy.sohad ~19.7 MB of its 32.9 MB.textin mrbind-generated registration code, almost all of it a long tail of tiny near-duplicate instantiations that differ only by embedded function pointers and name strings (so linker ICF can't fold them).What changes
Per bound function, the macros now emit a
FuncRow(plain data: names, comment, parameter table, arity, flags) instead of aTryAddFunc<...>()call. The only per-function templates left are:FuncRowThunk<F, ...>::Call— the call thunk carrying the target function pointer and the parameter/return adjustments (same body as the oldTryAddFunclambda);FuncRowThunk<F, ...>::Register— a tiny wrapper that hands the typed&Callto the shared registrar (needed because casting function pointers isn't allowed in constant expressions, so the row can't store a type-erased thunk directly).Everything else runs once per distinct signature shape (
FuncRowRegistrar) or is fully shared runtime code (RegisterFuncRow: the two-pass overload bookkeeping, ambiguous-overload renaming, operator injection incl. the reversed__r*__forms, alias registration). Fields follow the same pattern (MemberVarRow; the_offsetof_*static properties become data served by one shared lambda), and enum elements become a table + loop.Constructors are intentionally not migrated: measured on MeshLib, ctor instantiation shapes are 1:1 unique (a ctor's shape is its class + parameter types), so tables win nothing there. Conversion operators,
TryAddFuncSimple(), and custom bindings keep usingTryAddFunc()directly.Behavior is unchanged
pydocoutput and propertyfget/fsetdocstrings byte-identical on the test bindings and on a purpose-built feature test (kwargs, default arguments incl. the pretty default strings, overloads, static methods, member operators, operator injection into operand types, reversed binary operators, deprecation warnings, GIL call guards, fields,_offsetof_*values, enum values incl. 64-bit unsigned).mrmeshpy.pyiis identical modulo a known pre-existing nondeterminism (registration order of conversion-operator-injected ctors varies between any two builds of master as well; caused bytype_entriesbeing pointer-hash ordered — worth fixing separately).class_::def_property_static()does), not passed to thecpp_functionconstructor — otherwise the rendered signature text changes (arg1→arg0) and comments leak into setter docstrings. The record retrieval is written against the raw capsule API so it works with upstream pybind11 and MeshLib's limited-API fork alike.Results
MeshLib manylinux (Clang 21, same-commit A/B against master, full Python test suite green on both arches × Python 3.8–3.14). On x86_64 the resulting
mrmeshpy.pyiis byte-identical to master's.mrmeshpy.socompressed (in wheel)mrmeshpy.so.textmrmeshpy.sounpackedmrmeshpy.socompressed (in wheel)mrmeshpy.so.textmrmeshpy.sounpacked(
.data.rel.rogrows by ~1.2 MB and.rela.dynby ~1.9 MB — the tables and their relocations — but both compress far better than the code they replace.)On a synthetic 800-method module (Clang 22,
-Oz): whole module −17%,.text−37%, compile time −20% (one small instantiation per method instead of theTryAddFuncchain). On MeshLib's CI the "Generate and build MRBind bindings" step is neutral-to-slightly-faster (18.8→18.9 min x86_64, 11.1→10.4 min aarch64) — that step is dominated by parsing, not compilation.