[WIP][Cling] Refactor CIFactory and add incremental action support

[SahilPatidar]

This Pull request:

This PR refactors CIFactory and adds support for IncrementalAction (from Clang-Repl). The goal is to reduce the amount of Clang-specific logic handled inside CIFactory and rely more on Clang’s existing infrastructure.

Changes or fixes:

Checklist:

• tested changes locally
• updated the docs (if necessary)

This PR fixes #

[github-actions]

Test Results

     6 files       6 suites   1d 8h 23m 4s ⏱️
 3 770 tests  3 757 ✅ 0 💤 13 ❌
22 108 runs  22 033 ✅ 0 💤 75 ❌

For more details on these failures, see this check.

Results for commit 3dd5414.

[SahilPatidar]

@vgvassilev
Most of the failing tests are related to Clad:

TEST FAILURES:
1:cppinterop-CppInterOpTests
185:gtest-hist-hist-TFormulaGradientTests
186:gtest-hist-hist-TFormulaHessianTests
271:gtest-math-mathcore-CladDerivatorTests
309:minuit2_testADMinim
330:gtest-roofit-histfactory-testHistFactory
350:gtest-roofit-roofitcore-testRooFuncWrapper
353:gtest-roofit-roofitcore-testRooMinimizer
361:test-stressroofit-codegen
426:tmva-sofie-test-TestCladAutodiff
430:gtest-tmva-sofie-TestGemmDerivative
922:tutorial-math-fit-exampleFit3D
945:tutorial-math-fit-minuit2GausFit

Some of them show errors like:

error: static assertion failed: clad doesn't appear to be loaded; make sure that you pass clad.so to clang.
        static_assert(false, "clad doesn't appear to be loaded; make sure that "
                      ^~~~~

This may be happening because I made rough changes in FrontendAction::CreateWrappedASTConsumer, where it now directly returns the DeclCollector from IncrementalAction::CreateASTConsumer without adding the plugin consumer.

Previously, I mentioned an issue related to the Clad plugin consumer. There was a problem with CladPlugin::initializeSema adding a SemaExternalSource, which I have resolved.

If we allow the plugin consumer, then during BeginSourceFile, when setASTConsumer is called, it triggers Initialize. This ends up stealing and replacing the CompilerInstance’s consumers. I’m not fully sure why this happens, but it appears to be causing many of the test failures.

In the original Cling implementation, as far as I can see, we never call Plugin::Initialize (so no replacement happens). Instead, we take the plugin consumer and pass it to the DeclCollector consumer.

We need to find a way (to pass plugin into DeclCollector or something else) to solve this first before moving forward.

[vgvassilev]

When clad is OFF we should not run these tests. This means that we need to suppress them in cmake. cc: @guitargeek

Are the related llvm/clang changes really required? I think we should do extra effort to avoid them..

[SahilPatidar]

Those are in CodeGenAction.cpp. On my local system, they were causing a segmentation fault. I’m not sure why, but after commenting them out, the crash stopped:

// PrettyStackTraceDecl CrashInfo(*D.begin(), SourceLocation(),
//                                Context->getSourceManager(),
//                                "LLVM IR generation of declaration");

And another change related to CompilerInstance and SourceManager initialization. As I mentioned, in Cling we use one large virtual source file to keep source locations consistent.

For now, I added a temporary helper:

runIncrementalSourceMgrInitializer

[vgvassilev]

Those are in CodeGenAction.cpp. On my local system, they were causing a segmentation fault. I’m not sure why, but after commenting them out, the crash stopped:

// PrettyStackTraceDecl CrashInfo(*D.begin(), SourceLocation(),
//                                Context->getSourceManager(),
//                                "LLVM IR generation of declaration");

And another change related to CompilerInstance and SourceManager initialization. As I mentioned, in Cling we use one large virtual source file to keep source locations consistent.

For now, I added a temporary helper:

runIncrementalSourceMgrInitializer

Sounds suspicious. Maybe we should run valgrind.

[SahilPatidar]

We need to address these issues in the current setup:

• Based on the original Cling setup, CladPlugin should be passed to DeclCollector, not to CompilerInstance.
• initialization of the SourceManager to avoid clang side work.

For SourceManager related:
Currently, we use one large virtual file as the main file to generate unique source locations. We create this using custom logic during SourceManager initialization. However, clang::FrontendAction::BeginSourceFile does not provide a way to do this without modifying its behavior.

So, I tried overriding the MainFileID inside clang::FrontendAction::BeginSourceFileAction. I moved the source initialization code there. It builds fine, and I did not notice any other test failures compare to previous version.

I also tried another approach (just to test how it behaves). I let BeginSourceFile initialize the SourceManager as usual, and then I created a new virtual file. This file uses the end location of the main file when creating its FileID, so it acts like an includer instead of the main file.

const char* Filename = "<<< includer >>>";
FileEntryRef FE = FM.getVirtualFileRef(Filename, 1U << 15U, time(0));

SM.setFileIsTransient(FE);

SourceLocation Result = SM.getLocForEndOfFile(SM.getMainFileID());

m_VirtualFileID =
    SM.createFileID(FE, Result, SrcMgr::C_User);

auto Buffer = llvm::MemoryBuffer::getMemBufferCopy("/*CLING DEFAULT MEMBUF*/;\n");

SM.overrideFileContents(FE, std::move(Buffer));

After this, I did not see any issue while building. But when I ran roottest-root-tree, a few tests started failing. One of them failed with this error:

3484: /Users/sahilpatidar/Desktop/root/roottest/root/treeformula/stl/runpair.C:4:11: error: redefinition of 'file' as different kind of symbol
3484:    TFile *file = TFile::Open("RefTest.root");
3484:           ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/kernel_types.h:54:8: note: previous definition is here
3484: struct file;

This error happens inside CheckVariableDeclaration:

bool DeclExtractor::CheckForClashingNames(
    const llvm::SmallVector<NamedDecl*, 4>& Decls,
    DeclContext* DC) {

  for (size_t i = 0; i < Decls.size(); ++i) {
    ...
    else if (VarDecl* VD = dyn_cast<VarDecl>(ND)) {
      LookupResult Previous(*m_Sema, ND->getDeclName(), ND->getLocation(),
                            Sema::LookupOrdinaryName,
                            RedeclarationKind::ForVisibleRedeclaration);
      m_Sema->LookupQualifiedName(Previous, DC);
      m_Sema->CheckVariableDeclaration(VD, Previous);
      ...

I am not sure why this is happening, and I am also not sure any of these way is correct? Can we do something better?

[vgvassilev]

Can you debug side by side with the working version?

[SahilPatidar]

I found the issue.

The second approach (using a different virtual file instead of the main file) was failing because of a check in DefinitionShadower::Transform. This code decides whether to put declarations inside the __cling_N5 namespace to avoid name conflicts.

It uses typedInClingPrompt for this check. If it returns true, the declaration is wrapped inside the namespace.

In my case, since I used another virtual file instead of the main file, this function returned false. Because of that, the declaration was not added to the namespace, which caused the name collision error.

static bool typedInClingPrompt(FullSourceLoc L) {
  if (L.isInvalid())
    return false;

  const SourceManager &SM = L.getManager();
  const FileID FID = SM.getFileID(L);

  return SM.isFileOverridden(SM.getFileEntryForID(FID)) &&
         (SM.getFileID(SM.getIncludeLoc(FID)) == SM.getMainFileID());
}

[devajithvs]

In my case, since I used another virtual file instead of the main file, this function returned false. Because of that, the declaration was not added to the namespace, which caused the name collision error.

Maybe we can add a check to see if the buffer name starts with input_line

Something similar to what we have here:

root/core/metacling/src/TCling.cxx

Line 1186 in 20705fe

assert(FE->getName().contains("input_line_"));

[SahilPatidar]

I tried it, and now the tests match the working version on my machine. If second approach is correct to do in Cling, then the only remaining issue is the Clad plugin.

[vgvassilev]

For Clad, here is where we steal the consumers and tune them: https://github.com/vgvassilev/clad/blob/db7f52fafcb45bf8cbaa8b0c7256c664c553e940/tools/ClangPlugin.cpp#L154 and here https://github.com/vgvassilev/clad/blob/db7f52fafcb45bf8cbaa8b0c7256c664c553e940/tools/ClangPlugin.h#L173

I suspect some of these assumptions are incorrect after your patch.

[SahilPatidar]

From the code, I see that in the original setup we never called Initialize for the Clad consumer. I also checked in the debugger and confirmed we never did any “stealing” of the consumer.

In the original version, we just passed the plugin to the DeclCollector consumer here:
https://github.com/vgvassilev/root/blob/7c320ae5313332784afd0727e95f1e9cb99ac44b/interpreter/cling/lib/Interpreter/IncrementalParser.cpp#L339

In the new version, we pass the plugin consumer in CI during FrontendAction::BeginSourceFile, and setASTConsumer calls Initialize, which does the "stealing.” This is the behavior difference I see.

That’s why I directly return the consumer from FrontendAction::CreateWrappedConsumer without adding the Clad consumer for now, which causes Clad-related test failures. If I allow the Clad consumer to be passed in CI, other many tests fail.

[vgvassilev]

From the code, I see that in the original setup we never called Initialize for the Clad consumer. I also checked in the debugger and confirmed we never did any “stealing” of the consumer.

In the original version, we just passed the plugin to the DeclCollector consumer here: https://github.com/vgvassilev/root/blob/7c320ae5313332784afd0727e95f1e9cb99ac44b/interpreter/cling/lib/Interpreter/IncrementalParser.cpp#L339

In the new version, we pass the plugin consumer in CI during FrontendAction::BeginSourceFile, and setASTConsumer calls Initialize, which does the "stealing.” This is the behavior difference I see.

That’s why I directly return the consumer from FrontendAction::CreateWrappedConsumer without adding the Clad consumer for now, which causes Clad-related test failures. If I allow the Clad consumer to be passed in CI, other many tests fail.

Ok, we will need to think how to circumvent the stealing on clad’s side. Can you think of a way?

EDIT: Maybe we steal only when the preprocessor has no CLING macro defined? Or even if we don’t have incremental extensions defined.

[SahilPatidar]

I need to understand the behavior when we make changes on the Clad side. How can I modify the Clad plugin and recompile it on root side?

[vgvassilev]

I need to understand the behavior when we make changes on the Clad side. How can I modify the Clad plugin and recompile it on root side?

Not easy. One is edit the source in the build folder clad-prefix, then go to the build directory clad-prefix/…/build and do “make install”. Then cd to the top-most root_build and call “make Cling”. That should be all you need.