partial c++20 module support #2516

TroyKomodo · 2025-12-18T01:19:52Z

Adds partial support for c++20 modules when using clang.

Also detects when modules are used in MSVC / GCC and disable cache for those calls.

partially addresses #2095

the clang c++20 module implementation (as well as msvc) is rather easy to parse and understand so adding support for it is fairly trivial. GCC however is a mess. An entirely different approach will be needed to get GCC module support which is far out side of the scope for this initial PR.

I will submit future PRs to get MSVC / GCC.

Test with nix

(pkgs.sccache.overrideAttrs (final: old: {
  version = "troy/cxx20-modules";
  cargoHash = "sha256-tCQrZ6cPGJrxHgeSJ4ZqZs6otWvVu6Ys3svAiwvmSDc=";
  src = fetchFromGitHub {
    owner = "troykomodo";
    repo = "sccache";
    rev = "troy/cxx20-modules";
    sha256 = "sha256-A0n/nsNPTHRC0Z6ERwCpoR1w3O/xl6d1yCw/5GeqNd4=";
  };
  cargoDeps = pkgs.rustPlatform.fetchCargoVendor {
    inherit (final) pname src version;
    hash = final.cargoHash;
  };
}))

codecov-commenter · 2025-12-18T08:20:50Z

Codecov Report

❌ Patch coverage is 87.03072% with 38 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.19%. Comparing base (782b151) to head (b2fc693).

Files with missing lines	Patch %	Lines
src/compiler/clang.rs	83.89%	24 Missing ⚠️
src/compiler/gcc.rs	82.66%	13 Missing ⚠️
src/compiler/compiler.rs	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2516      +/-   ##
==========================================
+ Coverage   71.06%   71.19%   +0.13%     
==========================================
  Files          64       64              
  Lines       35394    35687     +293     
==========================================
+ Hits        25153    25408     +255     
- Misses      10241    10279      +38

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ChuanqiXu9 · 2025-12-18T09:47:30Z

FYI ccache/ccache#1523

mathstuf

The code looks good to me. I'll probably try to test it some time next week.

I can help out with GCC module map file parsing too (it will need to actually be parsed due to the $root meta-descriptor). However, the output path is also given by it…that may prove more difficult to "know".

TroyKomodo · 2025-12-18T16:08:41Z

The code looks good to me. I'll probably try to test it some time next week.

I can help out with GCC module map file parsing too (it will need to actually be parsed due to the $root meta-descriptor). However, the output path is also given by it…that may prove more difficult to "know".

Thanks! It was my first time contributing to sccache so I'm not too familiar with the process.

I will fix up the clippy issues + windows test issues :p

mathstuf · 2025-12-21T03:59:26Z

It looks like BMI-only compilations are not properly considered:

467:    actual-stdout> [2-1->6/8@76.9] Building CXX object 'CMakeFiles/CXXModules__export_from_ninja@synth_cf4e03cd8b1e.dir/35a7889ff75f.bmi'          
467:    actual-stdout> FAILED: [code=254] CMakeFiles/CXXModules__export_from_ninja@synth_cf4e03cd8b1e.dir/35a7889ff75f.bmi 
467:    actual-stdout> /home/boeckb/code/depot/group-devel/sccache/src/target/debug/sccache  /usr/lib64/ccache/clang++   -g -std=gnu++20 --precompile -MD -MT 'CMakeFiles/CXXModules__
export_from_ninja@synth_cf4e03cd8b1e.dir/35a7889ff75f.bmi' -MF CMakeFiles/CXXModules__export_from_ninja@synth_cf4e03cd8b1e.dir/35a7889ff75f.bmi.d @CMakeFiles/CXXModules__export_from_
ninja@synth_cf4e03cd8b1e.dir/35a7889ff75f.bmi.modmap -o 'CMakeFiles/CXXModules__export_from_ninja@synth_cf4e03cd8b1e.dir/35a7889ff75f.bmi' -c /home/boeckb/misc/builds/cmake/build-cla
ng/Tests/RunCMake/CXXModulesCompile/exp-mods-from-ninja-ninja-install/lib/cxx/miu/importable.cxx
467:    actual-stdout> sccache: encountered fatal error
467:    actual-stdout> sccache: error: Failed to generate compile commands
467:    actual-stdout> sccache: caused by: Failed to generate compile commands
467:    actual-stdout> sccache: caused by: Missing object file output

This is from running CMake's CXXModules tests:

export SCCACHE_SERVER_PORT=23448 # unique to avoid ambient re-usage
export CXX="$(which sccache) $(which clang++)"
ctest -R CXXModules

mathstuf · 2025-12-21T04:00:32Z

You can ignore the tests failing about JSON argument list mismatches; it is not robust against mutli-argument compilers like here. If you do the "symlink trick" (ln -s $(which sccache) clang++ && export CXX="$PWD/clang++", that probably avoids the issue).

TroyKomodo · 2025-12-21T04:40:06Z

Hmm, i think this is just me forgetting to change it somewhere. I will fix

TroyKomodo · 2025-12-21T04:42:26Z

Okay so it seems i think we should always treat it as an object even if it really isnt...

sylvestre · 2025-12-21T09:08:01Z

could you please add an integration test ? ie high level which integrates into cmake and uses C++ module.
see
tests/cmake/ tests/sccache_args.rs tests/sccache_cargo.rs tests/sccache_rustc.rs

mathstuf · 2025-12-21T12:43:12Z

I tried the cmake tests and they ran fine for me but there were only 2...

There are dozens of test cases within each CTest-level test. These are the relevant ones, thanks!

Can you inspect the cache state (--show-stats) on an initial run and a subsequent run to see how well caching itself is working?

TroyKomodo · 2025-12-21T14:42:33Z

could you please add an integration test ? ie high level which integrates into cmake and uses C++ module.

see

tests/cmake/ tests/sccache_args.rs tests/sccache_cargo.rs tests/sccache_rustc.rs

Yes!

TroyKomodo · 2025-12-21T18:52:31Z

https://github.com/mozilla/sccache/actions/runs/20413181927/job/58653103431 test results look good!

sylvestre · 2025-12-21T18:55:17Z

are you sure ?

Run ${SCCACHE_PATH} --show-stats
Compile requests                      4
Compile requests executed             4
Cache hits                            2
Cache hits (C/C++)                    2
Cache misses                          2
Cache misses (C/C++)                  2
Cache hits rate                   50.00 %
Cache hits rate (C/C++)           50.00 %

TroyKomodo · 2025-12-21T18:57:13Z

That seems right no? the first time we invoke the compiler not cached yet then we invoke again and its cached.

There are only 2 files to compile, so i expected it to be 4 requests 2 hit 2 misses.

sylvestre · 2025-12-21T20:30:22Z

oh yeah, my bad, i thought we were doing a reset

TroyKomodo · 2025-12-21T21:04:18Z

oh yeah, my bad, i thought we were doing a reset

I just copied the other cmake test example, but I can reset the stats if that makes it easier to follow

mathstuf · 2025-12-22T04:43:03Z

Gah, module compilation in the tests doesn't really happen without a setting. For clang, this should work:

-DCMake_TEST_MODULE_COMPILATION=named,compile_commands,collation,partitions,internal_partitions,export_bmi,install_bmi,shared,bmionly,build_database,import_std23

src/compiler/gcc.rs

sylvestre · 2025-12-22T13:01:58Z

src/compiler/gcc.rs

    }

+    if !module_only_flag {
+        if let Some(module_output_path) = module_output_path {


could you please move this into a function?

I can, however this function is ~550 lines, I think it would be better if in a followup PR we break up this function rather than doing a half job here by breaking up just this one conditional.

tests/cmake-modules/CMakeLists.txt

tests/cmake-modules/mymodule.cppm

TroyKomodo · 2025-12-22T17:01:11Z

Gah, module compilation in the tests doesn't really happen without a setting. For clang, this should work:

-DCMake_TEST_MODULE_COMPILATION=named,compile_commands,collation,partitions,internal_partitions,export_bmi,install_bmi,shared,bmionly,build_database,import_std23

Ah, I see. Okay I will try again i think i need to setup my env to have support for import std.

TroyKomodo · 2025-12-22T19:47:37Z

Gah, module compilation in the tests doesn't really happen without a setting. For clang, this should work:
-DCMake_TEST_MODULE_COMPILATION=named,compile_commands,collation,partitions,internal_partitions,export_bmi,install_bmi,shared,bmionly,build_database,import_std23

Ah, I see. Okay I will try again i think i need to setup my env to have support for import std.

let
  pkgs = import <nixpkgs> {
    overlays = [
      (final: prev: {
        llvmPackages_21 = prev.llvmPackages_21 // {
          libcxx = prev.llvmPackages_21.libcxx.overrideAttrs (old: {
            postInstall = (old.postInstall or "") + ''
              substituteInPlace $out/lib/libc++.modules.json \
                --replace-fail '"../share' "\"$out/share"
            '';
          });
        };
      })
    ];
  };

  llvm = pkgs.llvmPackages_21;

  wrappedClangScanDeps = pkgs.writeShellScriptBin "clang-scan-deps" ''
    newargs=()
    inject_next=false
    for arg in "$@"; do
      if $inject_next; then
        newargs+=("$arg")
        newargs+=("-isystem" "${llvm.libcxx.dev}/include/c++/v1")
        inject_next=false
      elif [[ "$arg" == "--" ]]; then
        newargs+=("$arg")
        inject_next=true
      else
        newargs+=("$arg")
      fi
    done
    exec ${llvm.clang-tools}/bin/clang-scan-deps "''${newargs[@]}"
  '';

  sccache = "/home/troy/github/troy/sccache/result/bin/sccache";

  clangWrapper = pkgs.writeShellScriptBin "clang" ''
    exec ${sccache} "${llvm.libcxxClang}/bin/clang" "''${@}"
  '';
  clangxxWrapper = pkgs.writeShellScriptBin "clang++" ''
    exec ${sccache} "${llvm.libcxxClang}/bin/clang++" "''${@}"
  '';
in
pkgs.mkShell {
  hardeningDisable = [ "all" ];

  buildInputs = [
    pkgs.cmake
    pkgs.mold
    pkgs.ninja
    pkgs.openssl
    llvm.libcxxClang
    llvm.clang-tools
    wrappedClangScanDeps
  ];

  shellHook = ''
    export CC="${clangWrapper}/bin/clang"
    export CXX="${clangxxWrapper}/bin/clang++"
    export LD_LIBRARY_PATH="${llvm.libcxx.out}/lib:${pkgs.openssl.out}/lib:''${LD_LIBRARY_PATH:-}"
    export NIX_CFLAGS_COMPILE="-B${llvm.libcxx.out}/lib -isystem ${llvm.libcxx.dev}/include/c++/v1"
    export PATH="${wrappedClangScanDeps}/bin:''${PATH:-}"
  '';
}

A huge pain...

troy@troy-thinkpad ~/g/t/CMake (master) [SIGINT]> ctest -R CXXModules -j$(nproc) --test-dir=build   nix-shell 
Test project /home/troy/github/troy/CMake/build
Guessing configuration NoConfig
    Start 536: RunCMake.CXXModulesCompile
    Start 535: RunCMake.CXXModules
1/2 Test #535: RunCMake.CXXModules ..............   Passed   18.90 sec
2/2 Test #536: RunCMake.CXXModulesCompile .......   Passed  116.80 sec

100% tests passed, 0 tests failed out of 2

Label Time Summary:
CXXModules    = 135.70 sec*proc (2 tests)

Total Test time (real) = 116.81 sec
troy@troy-thinkpad ~/g/t/CMake (master)> ctest -R CXXModules -j$(nproc) --test-dir=build            nix-shell 
Test project /home/troy/github/troy/CMake/build
Guessing configuration NoConfig
    Start 536: RunCMake.CXXModulesCompile
    Start 535: RunCMake.CXXModules
1/2 Test #535: RunCMake.CXXModules ..............   Passed   19.94 sec
2/2 Test #536: RunCMake.CXXModulesCompile .......   Passed  104.62 sec

100% tests passed, 0 tests failed out of 2

Label Time Summary:
CXXModules    = 124.55 sec*proc (2 tests)

Total Test time (real) = 104.63 sec
troy@troy-thinkpad ~/g/t/CMake (master)>                                                            nix-shell

Every 2.0s: /home/troy/github/troy/sccache/result/bin/sccache -s       troy-thinkpad: Mon Dec 22 12:47:11 2025

Compile requests                   1976
Compile requests executed           830
Cache hits                          756
Cache hits (C/C++)                  756
Cache misses                         72
Cache misses (C/C++)                 72
Cache hits rate                   91.30 %
Cache hits rate (C/C++)           91.30 %
Cache timeouts                        0
Cache read errors                     0
Forced recaches                       0
Cache write errors                    0
Cache errors                          0
Compilations                         72
Compilation failures                  2
Non-cacheable compilations            0
Non-cacheable calls                 244
Non-compilation calls               902
Unsupported compiler calls            0
Average cache write               0.001 s
Average compiler                  0.217 s
Average cache read hit            0.000 s
Failed distributed compilations       0

Non-cacheable reasons:
-E                                  244

Cache location                  Local disk: "/home/troy/github/troy/CMake/build/Tests/CMakeFiles/TestHome/.cac
he/sccache"
Use direct/preprocessor mode?   yes
Version (client)                0.12.0
Cache size                          155 MiB
Max cache size                       10 GiB

TroyKomodo · 2025-12-22T22:23:44Z

Freebsd test seems broken... Unrelated to this PR however.

…m Avi Kivity Currently, we support ccache as the compiler cache. Since it is transparent, nothing much is needed to support it. This series adds support to sccache[1] and prefers it over ccache when it is installed. sccache brings the following benefits over ccache: 1. Integrated distributed build support similar to distcc, but with automatic toolchain packaging and a scheduler 2. Rust support 3. C++20 modules (upcoming[2]) It is the C++20 modules support that motivates the series. C++20 modules have the potential to reduce build times, but without a compiler cache and distributed build support, they come with too large a penalty. This removes the penalty. The series detects that sccache is installed, selects it if so (and if not overridden by a new option), enables it for C++ and Rust, and disables ccache transparent caching if sccache is selected. Note: this series doesn't add sccache to the frozen toolchain or add dbuild support. That is left for later. [1] https://github.com/mozilla/sccache [2] mozilla/sccache#2516 Toolchain improvement, won't be backported. Closes #27834 * github.com:scylladb/scylladb: build: apply sccache to rust builds too build: prevent double caching by compiler cache build: allow selecting compiler cache, including sccache

TroyKomodo force-pushed the troy/cxx20-modules branch from 8f46985 to 930732c Compare December 18, 2025 01:24

ChuanqiXu9 mentioned this pull request Dec 18, 2025

Implement support for C++20 modules ccache/ccache#1523

Draft

12 tasks

mathstuf approved these changes Dec 18, 2025

View reviewed changes

TroyKomodo mentioned this pull request Dec 18, 2025

sccache partial support kelteseth/arewemodulesyet#78

Open

TroyKomodo added 4 commits December 19, 2025 08:25

partial c++20 module support

a389a64

remove unused conditional for multi input

e5f92b7

fix MSRV issue

c43c465

ignore vscode settings

71e9190

TroyKomodo force-pushed the troy/cxx20-modules branch from 489600b to 71e9190 Compare December 19, 2025 15:25

always output object even if its a module

8841d54

TroyKomodo force-pushed the troy/cxx20-modules branch from c0a57b6 to 8841d54 Compare December 21, 2025 04:44

fix tests

288507d

This comment was marked as outdated.

Sign in to view

TroyKomodo added 2 commits December 21, 2025 10:10

add cmake modules test

aea8760

fix test

b626b23

sylvestre reviewed Dec 22, 2025

View reviewed changes

src/compiler/gcc.rs Outdated Show resolved Hide resolved

sylvestre reviewed Dec 22, 2025

View reviewed changes

tests/cmake-modules/CMakeLists.txt Outdated Show resolved Hide resolved

sylvestre reviewed Dec 22, 2025

View reviewed changes

tests/cmake-modules/mymodule.cppm Outdated Show resolved Hide resolved

TroyKomodo added 3 commits December 22, 2025 12:48

fix newlines

305bdd2

add extra info to cannot cache call

e5427bf

fix fmt

b2fc693

avikivity mentioned this pull request Dec 23, 2025

build: support (and prefer) sccache as the compiler cache scylladb/scylladb#27834

Open

partial c++20 module support #2516

Are you sure you want to change the base?

partial c++20 module support #2516

Conversation

TroyKomodo commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ChuanqiXu9 commented Dec 18, 2025

Uh oh!

mathstuf left a comment

Choose a reason for hiding this comment

Uh oh!

TroyKomodo commented Dec 18, 2025

Uh oh!

mathstuf commented Dec 21, 2025

Uh oh!

mathstuf commented Dec 21, 2025

Uh oh!

TroyKomodo commented Dec 21, 2025

Uh oh!

TroyKomodo commented Dec 21, 2025

Uh oh!

This comment was marked as outdated.

sylvestre commented Dec 21, 2025

Uh oh!

mathstuf commented Dec 21, 2025

Uh oh!

TroyKomodo commented Dec 21, 2025

Uh oh!

This comment was marked as outdated.

TroyKomodo commented Dec 21, 2025

Uh oh!

sylvestre commented Dec 21, 2025

Uh oh!

TroyKomodo commented Dec 21, 2025

Uh oh!

sylvestre commented Dec 21, 2025

Uh oh!

TroyKomodo commented Dec 21, 2025

Uh oh!

mathstuf commented Dec 22, 2025

Uh oh!

Uh oh!

sylvestre Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

TroyKomodo Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

TroyKomodo commented Dec 22, 2025

Uh oh!

TroyKomodo commented Dec 22, 2025

Uh oh!

TroyKomodo commented Dec 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

TroyKomodo commented Dec 18, 2025 •

edited

Loading

codecov-commenter commented Dec 18, 2025 •

edited

Loading