Static PGO SEGFAULTS with WAMR-2.3.0

[Lukasdoe]

Subject of the issue

When using the AOT static PGO, execution of the instrumented aot file yields:

unhandled SIGSEGV, si_addr: (nil)
Aborted (core dumped)

Test case

Polybench benchmarks, covariance.wasm

Your environment

• Host OS: Ubuntu 24.04.1 LTS x86_64
• WAMR: 2.3.0, AOT, PGO

Steps to reproduce

Compile polybench covariance benchmark with PGO enabled to AOT and execute with iwasm.

Expected behavior

It should not crash.

Actual behavior

It does crash.

[TianlongLiang]

I will take a look at this. Also, it seems that we don't have a sample to demo the static PGO, I may consider adding one sample too.

[TianlongLiang]

It's due to the WAMR upgrade of the LLVM-dependent library from 15 to 18. I am working on it and identifying the adaptation WAMR needs to accommodate the changes in LLVM for LLVM perf data. It may take some time to fix this issue, meanwhile, I tried llvm 15, it's all working fine, you can use WAMR 2.2.0 if you want to use static PGO now, if you want.

[Lukasdoe]

That's good to know, thank you for looking into this! I am currently using WAMR 2.2.0, and it works like a charm.

For the research I'm doing with wamr, I will need to use LLVM 20 anyways. Do you know of any major problem with a newer LLVM version and wamr or is it just that the prof-data format changed?

[TianlongLiang]

LLVM 20 could still need to wait a while, even LLVM 19 currently is having some issues(basically WAMR needs to wait LLVM for its support on most of if not every WAMR target platforms), you can see the discussion here:
#4213

I will need to use LLVM 20 anyway

For your own purpose, you can try it for sure, if you use it on Ubuntu, I would say the core part of WAMR is most likely to work(maybe with some minor changes). The PGO is another story, I see the LLVM 20 changed prof-data raw format from 9 to 10 after LLVM 18; it still needs some work to support PGO when WAMR updates LLVM from 18 to 20

[Lukasdoe]

Oh yes, I totally get why you can't just upgrade to the newest version of LLVM straight away for the whole project.

I will have a look at the parts of wamr that emit the profiling data then, thank you for the pointer! I will post any progress that I make there, maybe I can reduce the work for you as well :)

[TianlongLiang]

Cool, always appreciate a helping hand. I will also post/update what I can gather here. It's also new for me to learn more about LLVM PGO.

Basic workflow for WAMR AOT static PGO

WAMR implements a classic three-phase Profile-Guided Optimization (PGO) flow for ahead-of-time (AOT) compiled WebAssembly modules. The phases include:

1. Instrumentation (Compile-time): The Wasm module is compiled to an instrumented AOT binary. This inserts profiling counters into the generated code and emits special AOT reloc section profile data sections (counters, data. WAMR’s AOT compiler (wamrc) exposes this via the --enable-llvm-pgo flag

2. Execution and gathering prof data(Runtime): The instrumented AOT module is run with the WAMR runtime (iwasm), which updates profile counters in memory as the code executes. The runtime then collects the profile data from memory and serializes it to a raw profile file (e.g. .profraw, which should be the same format as LLVM-defined profile data files so that it can be used for llvm-profdata). In WAMR this is triggered by running iwasm with the --gen-prof-file=<file> option.

3. Optimization (Recompile-time): Finally, the recorded profile is applied to recompile the Wasm module with profile-informed optimizations. The raw profile is first converted to an indexed profile (using llvm-profdata merge), then passed to wamrc via --use-prof-file=<file> to guide LLVM’s optimizations. This results in a new optimized AOT binary, which can be executed with iwasm for performance gains.

References and TODOs

Initial implementation of static PGO in WAMR
LLVM 18.x Instrumentation Profile Format
LLVM 18.x profile date struct

At first glance, the modification needs to be adapted on WAMR's side, including:

In phase 1:

• Maybe needs to support more reloc sections(chances are we don't have to add any), related code: https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/core/iwasm/compilation/aot_emit_aot_file.c#L3946

In phase 2:

• Correctly handle the AOT reloc section, and correctly update the LLVMProfileData https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/core/iwasm/aot/aot_runtime.h#L463
• The prof file emitting logic: https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/core/iwasm/aot/aot_runtime.c#L4844 and the definition of prof file raw header needs to be changed accordingly https://github.com/bytecodealliance/wasm-micro-runtime/blob/main/core/iwasm/aot/aot_runtime.h#L440

In phase 3:

Don't need to change anything since we only apply the llvm-profdata merged pgo file to LLVM.

[TianlongLiang]

I've got it working now; I will test it further and add a CI test for static PGO and PR soon.

[TianlongLiang]

#4345 should fix it

[Lukasdoe]

Closing this, since the PR fixed it perfectly. Great job!

[Lukasdoe]

For future reference: I also ported WAMR to llvm-20. This was only done for my own usecase, meaning only AOT builds with PGO on ubuntu were tested.

Lukasdoe@db4878a