Ps3toolchain bug report

[Garyx2018]

Crash calling PS3 kernel lock primitives very early in process startup (ps3toolchain + PSL1GHT static link)

Summary

A statically-linked executable built with ps3toolchain (newlib/gcc 7.2.0) and
PSL1GHT libraries (librsx.a, libgcm_sys, libsysutil, libio.a) crashes
during early process startup, before main()/global C++ constructors run.
The crash happens whenever any code path calls into a precompiled library
function that uses PS3 lock primitives (__sysLwMutexLock,
__sysSpinlockInitialize) this early — regardless of which function
triggers it. Confirmed with three independent trigger points:

• sbrk_init() in librt.a's sbrk.o (newlib heap init)
• rsxHeapInit() in librsx.a's mm.o (PSL1GHT RSX memory heap init)
• An unlabeled loop resolving near __sysLwMutexLock in precompiled,
  no-debug-info library code

Environment

• Toolchain: ps3toolchain (gcc 7.2.0), installed at /usr/local/ps3dev
• Also linking PSL1GHT libraries (librsx.a, libgcm_sys, libsysutil)
  from the same /usr/local/ps3dev prefix
• Build system: custom CMake toolchain file (not PSL1GHT's own Makefile
  build system)
• Test environment: RPCS3 v0.0.42 (tested on two builds: 19839-2416d652
  and 19845-03d03b2b), Vulkan renderer, Windows 10 host via WSL
  (Ubuntu) for the build side

Symptom

Boot hangs on a black screen. RPCS3.log shows normal module loading up
through:

S sys_prx: Loaded module: "/dev_flash/sys/external/libfs.sprx" (id=0x23001200)
U sys_prx TODO: _sys_prx_register_module(name="_sysProcessElf", opt=*0xd0100da0)
F {PPU[0x1000000] Thread (main_thread) [...]} VM: Access violation executing location 0x7c0802a6 (unmapped memory)

TTY.log is completely empty in every reproduction — confirming the crash
happens before any application code (including a write(1, ...)-based
diagnostic placed at __attribute__((constructor(101))), the earliest
priority the C++ runtime allows) ever runs.

Register state at crash (RPCS3 debugger, PPU main_thread)

Reproduced with identical values across three separate builds
(different linker configuration, different compiler flags, different
source):

CIA: (varies with build/relink, always inside a small loop calling
      __sysLwMutexLock-family functions)
LR:  CIA + 0x18
CTR: 0x7c0802a6      <-- garbage; this is literally the raw big-endian
                          instruction encoding for `mflr r0`, not a real
                          address
r0:  0x7c0802a6       (same value, evidently copied from CTR/LR via mflr)
r2:  0xf8010010        <-- TOC register; nowhere near the program's real
                             data segment (~0x00400000-0x00600000 range
                             in every build tested)

The instruction actually executing at the reported CIA is always a
harmless mr r3,r4 — meaning r0/CTR were already corrupted before
this point, not by this instruction. Surrounding disassembly shows a
small repeating loop pattern (lwz r0,0(r0) / mr r3,r4 / bl <sysLwMutex-family function>), consistent with a list-walking loop that
overruns into adjacent code and reads instruction bytes as a function
pointer/lock ID.

What we ruled out (each tested independently, with before/after

comparison of the exact same register values)

1. Duplicate/malformed ELF program headers from a customized linker
   script — reverted to the script's original FILEHDR PHDRS form,
   confirmed clean 8-segment layout via readelf -l. No change.
2. Missing .sys_proc_param section — added a correctly-populated
   one (readelf -x .sys_proc_param confirms correct magic
   0x13bcc5f6 and sane values). No change.
3. sprxlinker run on the main executable (rather than only on
   actual .sprx PRX modules) — rebuilt without it. No change.
4. TOC addressing model — added -mfull-toc to both
   CMAKE_C_FLAGS/CMAKE_CXX_FLAGS, full clean rebuild. Register
   values (r2, CTR) identical to before.
5. Custom linker script entirely — replaced with the compiler
   driver's own default linker script (i.e. no -Wl,-T,... at all).
   Build succeeds cleanly, .sys_proc_param still correctly placed,
   8 clean segments — but the exact same crash signature (r2 = 0xf8010010, CTR = 0x7c0802a6) reproduces.
6. A pair of very-early __attribute__((constructor(101))) /
   constructor(65535)) diagnostic functions added during
   debugging — removed entirely, full rebuild. Same crash signature
   (address shifts slightly due to code size change, everything else
   identical).

What stayed constant across every test

• r2 (TOC) is always 0xf8010010 at the crash point, regardless of
  build configuration.
• CTR/r0 are always 0x7c0802a6 (the mflr r0 instruction
  encoding) at the crash point.
• The crash always happens immediately downstream of the loader's
  _sys_prx_register_module(name="_sysProcessElf", ...) step (itself
  logged as TODO/unimplemented HLE in RPCS3, and confirmed via other
  games' logs to be a normal, non-fatal bootstrap step in general).
• Every trigger function we identified via nm/addr2line/
  objdump traces back to precompiled, no-debug-info object files
  inside librt.a or librsx.a that use PS3 lock primitives
  (sysSpinlockInitialize, sysLwMutexLock) during their own
  initialization.

Function-descriptor detail (may or may not be a real lead — flagging

for anyone who knows the ABI internals better than we do)

sysSpinlockInitialize's exported symbol is a proper PowerPC64 ELFv1
function descriptor ({ entry_point, toc_value }). Its baked-in
toc_value differs meaningfully between our custom-linker-script build
(0x0056d130, unrelated to the program's real ~0x004xxxxx-0x006xxxxx
TOC region) and the default-linker-script build (0x004980d0, which
does fall inside the program's real layout) — yet both builds produce
the identical crash. So while the descriptor's TOC value does appear to
get correctly relocated by the linker in the default-script case, the
actual crash is unaffected either way, suggesting the TOC value itself
isn't the root cause (or isn't the only one).

Question for the community

Is this a known interaction issue between newlib-based ps3toolchain
CRT startup and PSL1GHT's RSX/GCM libraries when statically linked
together via a generic CMake toolchain file (as opposed to PSL1GHT's own
Makefile-based build process)? Specifically: is there a required
initialization step (e.g. some form of sys_process_initialize, TLS
setup, or thread/lock-subsystem readiness signal) that PSL1GHT's own
build process performs automatically before jumping to the program
entry point, which a custom CMake-based link might be skipping?

Happy to provide the full project, build logs, or test further changes.

[zeldin]

r0: 0x7c0802a6 (same value, evidently copied from CTR/LR via mflr)

I think you got this backwards. It's way more likely that the value was copied from r0 to CTR. This is the standard code to call a function pointer i ELF v1:

      18:       80 0c 00 00     lwz     r0,0(r12)
      1c:       80 4c 00 04     lwz     r2,4(r12)
      20:       7c 09 03 a6     mtctr   r0
      24:       4e 80 04 21     bctrl

So if the pointer were to point to an ELF v2 function, you would load the first instruction of the code into r0/CTR and then jump to it (causing the address fault).

I suspect the CIA you are seeing is incorrectly reported; looking just before LR is probably more fruitful.

But at the end of the day, it looks like some ELF v2 code has crept into your build and is getting called by some unsuspecting ELF v1 code.