CppInterOp based cppyy migration#22709
Draft
guitargeek wants to merge 45 commits into
Draft
Conversation
…stream] Adapts upstream CppInterOp for use with ROOT & Cling: - Add SynthesizingCodeRAII guard for scope/class reflection - Call buildLookup in lookup paths, to populate Cling's lazy lookup tables - Drop class_name:: scope qualification in make_narg_call so virtual dispatch works for pure-virtual methods (TInterpreter::Declare) - Export CppGetProcAddress from libCling's linker script for dispatch mechanism - GetTypeAsString: revert the PrintingPolicy ctor to PrintingPolicy((LangOptions())) instead of deriving it from the ASTContext. Using default LangOptions restores the string form (std::basic_string<char>) that the CPyCppyy factory expects
Source: compres forks cppyy-backend/master 597bcf4 Drops ROOT's old clingwrapper in favour of the CppInterOp-based implementation from the compres fork. Replaces gInterpreter/ROOT-meta TCling API calls with CppInterOp via the dispatch mechanism (cppinterop_dispatch.cxx to load API) Adds recursive_mutex thread safety, switches execution to JitCall. To be followed with ROOT-specific patches.
…patch] CPPINTEROP_DIR is the include-root that contains the CppInterOp/ subdir with Dispatch.h and the tablegen-generated CppInterOpAPI.inc. Pass it straight to Cpp::AddIncludePath instead of concatenating "/include", matching where the build system actually places those headers.
Based on ROOT: - Load libCling via gSystem->DynamicPathName instead of the fork's CPPINTEROP_DIR/lib path - include ROOT core headers and call TClass::GetClass in GetScope for dictionary/module autoloading. - Guard GetActualClass and GetBaseOffset with Cpp::IsComplete to handle incomplete types (e.g. TCling with no public header) - Initialise ROOT globals and required dylibs in ApplicationStarter - Move precommondefs.h include into cpp_cppyy.h; use Cpp::TCppFuncAddr_t for proper alignment - Update CMakeLists for ROOT build-tree integration
Dispatch.h transitively includes the tablegen-generated CppInterOpAPI.inc,
which lives in the build tree, not the source tree. ROOT stages CppInterOp's
public headers + .inc files into ${BINARY_DIR}/etc/cppinterop/CppInterOp/
via the CppInterOpEtc custom target. Point cppyy_backend's include dir
and the runtime CPPINTEROP_DIR macro at that staging dir, and depend on
CppInterOpEtc so staging completes before this library compiles.
Source: compres forks cppyy/master 273ed88 - Calls gCling.EnableAutoLoading() - Adds cppyy.evaluate / cppyy.macro() / cppyy.ll.as_memoryview - Numba pointer/reference support - Metaclass naming fix, isinstance() idiom cleanup - Bug fixes in basic_string / span / npos pythonizations - Fallback decoding paths for non-UTF-8 compiler output To be followed with ROOT-specific patches.
… [upstream] Same as in Pythonize.cxx basic_string name from GetQualifiedCompleteName so the NPOS object is registered for the canonical class name shapes.
ROOT-specific patches on top of the compres fork cppyy/master baseline: __init__.py: - set gInterpereter to gbl.TInterpreter.Instance() - load_library(): use gSystem.Load/FindDynamicLibrary, add Windows winmode=0 for search path - add_library_path(): use gSystem.AddDynamicPath - Drop the CPPYY_API_PATH apipath_extra dispatcher-headers (ROOT installs CPyCppyy API headers) _cpython_cppyy.py: - Wrap `from cppyy_backend import loader` in try/except, fall back to c = None (ROOT does not ship cppyy_backend.so) - Platform: `import libcppyy` vs `import cppyy.libcppyy` (standalone cppyy uses the former; ROOT exposes it as cppyy.libcppyy) - load_reflection_info(): use gSystem.Load ROOT/_facade.py: - Store gInterpreter and gPad in __dict__
Source: compres forks CPyCppyy/master 482ccb7 Brings in fork-ahead developments missing from ROOT's old copy and compatibility with the CppInterOp based backend: - CppInterOp-aligned type system (Cppyy.h) - Type-based CreateConverter / CreateExecutor factory overloads - Instance_FromVoidPtr(scope) overload - PythonGILRAII for exception-safe GIL management - PyError_t RAII rewrite (unique_ptr-based) - GetTemplateArgsTypes for type-based template instantiation - ReduceReturnType/LambdaClass handling - Rvalue forwarding in Dispatcher - GetFailureMsg in Converters - STL string executor returns bound C++ object ROOT-specific CPyCppyyModule.h and CPyCppyyPyModule.cxx are preserved from master.
…tream] Source: ROOT master - Move to using SetGlobalPolicy(ECallFlags, bool) + GlobalPolicyFlags() - Drop the CallContext* argument from UseStrictOwnership() - Guard PyMapping_GetOptionalItemString for Python>3.13 in Dispatcher.cxx
…stream] Source: master
Neither on ROOT-master, nor compres-forks These fields hold C++ type handles, so semantically we should enforce this type
Source: master - Deprecate __mempolicy__ getter/setter with a clear error pointing users to the SetOwnership() pythonization - Drop the per-method memory policy from mp_call (referenced the removed sMemoryPolicy static) - Use GlobalPolicyFlags() for the Clone heuristic - Match the method name prefix before '<' only, so template arguments don't affect Clone detection
…tream] Source: ROOT-master 760b6cc - ToMemory override that lets the array converter copy Python strings (not only buffers) into C++ const char* arrays - ToArrayFromBuffer<> template helper that copies a buffer to an array-converter memory address, lifetime management via SetLifeLine
…NVERTER [upstream] Source: master Adds the 'override' keyword on HasState() in the macro so the derived-class definition matches the base-class virtual declaration.
…eam] Source: ROOT-master - DEFINE_CALL_POLICY_TOGGLE macro generates SetHeuristicMemoryPolicy, SetImplicitSmartPointerConversion, and SetGlobalSignalPolicy used by CallContext::SetGlobalPolicy() - Drop the obsolete kMemoryHeuristics/kMemoryStrict module-level labels (no longer referenced the removed CallContext::kUseStrict) - Add missing guard for Py_INCREF(gThisModule)
… [upstream] Source: ROOT-master - VectorIAdd returns self after the insert. Python += reassigns the lhs to the returned value, so returning the inserted-iterator result silently breaks the idiomatic += pattern on std::vector. Disable by wrapping VectorArray in #if 0 and removes its Utility::AddToClass call (to be fixed at a later stage) - Drop Cppyy::gGlobalScope extern, every caller uses Cppyy::GetGlobalScope()
…d [upstream] This std::span branch went only into the string based overload and not the new type based one that the compres forks use.
… args [upstream] GetQualifiedCompleteName returns the canonical class name with default template args spelled out
…ure mismatches [upstream] In PyFunction_AsCPointer, the CPPOverload and TemplateProxy branches previously returned null when the requested signature does not match the candidates, preventing the generic-Python-callable fall-through (JIT-ed wrapper path), even though CPPOverload and TemplateProxy are valid PyCallable_Check inputs. As a result, std::function/fn ptr parameters could not accept a cppyy free function or template overload whose signature did not match the target.
…[ROOT-patch] Source: ROOT - SignalTryCatch.h: gException resolved to definition from libCore.so - CPPInstance.h: replace CPYCPPYY_IMPORT with explicit extern / __declspec(dllimport) pair. Used by libROOTPythonizations which does not include CommonDefs.h for the definition of CPYCPPYY_IMPORT
…-patch] Source: ROOT-master ROOT runs CPyCppyy and libROOTPythonizations as two shared libs that under the same libcppyy Python extension module. The real PyInit_libcppyy lives in libROOTPythonizations and calls CPyCppyy::Init() to start the extension.
…s [ROOT-patch] Source: ROOT-master - Register a std::span pythonization that overrides begin() / end() with a JIT-compiled __cppyy_internal::ptr_iterator helper. libstdc++ (GCC >= 15) implements std::span::iterator via a private nested tag type, which CallFunc-generated wrappers cannot name without violating access rules — the pointer-based iterator sidesteps that - Add more std::basic_string name variants to the STLWString pythonization
… [ROOT-patch] Source: ROOT - __template_args__ read-only property on TemplateProxy for introspection of a method's template arguments - "ss:__overload__" branch in tpp_overload so callers can select an overload by both signature and template arguments Added for ROOT's Numba-introspection support.
…ype iterators [ROOT-patch] Pythonize.cxx tags begin()-returns as STL iterator types when Cppyy::GetScope succeeds on the return-type string. CppInterOp's GetScope returns the underlying class scopes, leading to raw-pointer iterators (e.g. RVec<T>::iterator = T*) falsely classified as STL iterator types. Restore the earlier behaviour of skipping pointer return values by doing a IsPointerType check on the type-handle. ROOT master's TCling-based GetScope returned null for pointer names and did not hit this false positive.
5cfc2da changed std::string-returning methods to always produce Python str instead of a CPPInstance proxy (gbl.std.string). Update test18_operator_plus_overloads and test34_cstring_template_argument asserts expecting gbl.std.string
…OOT-patch] _generic.pythonize_generic skips the pretty-printer for std::string because ToString returns a quoted ""x"" form, and CPyCppyy already pythonizes std::string with the unquoted shape. With CppInterOp, klass.__cpp_name__ is the canonical fully-qualified template form, so the typedef "std::string" no longer matched the exclude check
The `gPad and `gVirtualX` identifiers are injected into ROOT meta via `TGlobalMappedFunction::MakeFunctor()`. However, in the ROOT Python interface we don't want to rely on ROOT meta but use Cling or Clang-Repl directly. Hence, we inject these identifiers manually into the facade. The `TDirectoryPythonAdapter` previously used for `gDirectory` mirrored the live-tracking semantics of the C++ macros: every attribute access re-resolves to the current directory. The same semantics are needed for `gPad` and `gVirtualX`, which on the C++ side are also preprocessor macros that expand to a static accessor call. Therefore, a new `LiveProxy` is introduced for this.
…lorNumber ctor param
When a derived class pulls a base method into its own overload set with `using Base::method;`, the method is still declared in the base. If that base is not the first one, its subobject sits at a non-zero offset in the derived object. `CPPMethod::Call` computed the this-pointer offset against `fScope` (the class the method is bound on, i.e. the derived class) rather than the method's declaring base. For a using-declared method these differ, yielding a zero offset. The generated wrapper, however, casts 'this' to the declaring base, so the call wrote through an unadjusted pointer, corrupting memory and crashing on destruction. Compute the offset against the method's declaring scope (`Cppyy::GetParentScope`) instead. This is a no-op for ordinary inherited methods, which are bound on their declaring base already. Surfaced by `roottest-python-cpp-cpp` (`TGraphMultiErrors::SetLineColor`, where `TAttLine` is a secondary base).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirroring #21261 so we can debug remaining failures in parallel.