diff --git a/.agents/docs/2026-09-09-dlopen-surface-and-two-unwinders.md b/.agents/docs/2026-09-09-dlopen-surface-and-two-unwinders.md new file mode 100644 index 00000000..158da2ae --- /dev/null +++ b/.agents/docs/2026-09-09-dlopen-surface-and-two-unwinders.md @@ -0,0 +1,672 @@ +--- +subject: heterogeneous +status: active +--- + +# A dlopen surface no closure walks, and a process with two unwinders + +mcpp#596 reports that `examples/09-heterogeneous/sycl` builds cleanly and then +aborts with exit 134 on an NVIDIA machine, printing no exception text. The +reporter located the trigger exactly: `compat.sycl-runtime`'s farm carries +`libcuda.so.1` and not `libnvidia-ml.so.1`, so the SYCL runtime's CUDA adapter +cannot load. That is correct, and it is one of three mechanisms. + +This record separates them, because they are repaired in three different +repositories and only one of them is about NVML: + +* **A.** The libraries a package publishes through `runtime.library_dirs` exist + precisely because something will `dlopen` them. No closure mcpp walks reaches + them, so a farm that cannot satisfy its own members measures green. Two of + this farm's twenty-five members cannot. +* **B.** `compat.sycl-runtime`'s `install()` **enumerates** the payload's + library directory and **hand-writes** the driver's name. The hand-written + half is the half that is wrong, and the package's own criterion hand-writes + the same three names it checks. +* **C.** The artifact links LLVM libunwind statically and loads libgcc_s at run + time. Ten of libgcc's eighteen `_Unwind_*` entry points are interposed by the + executable and eight are not, so one throw is processed by two unwinders and + reaches `std::terminate` past a matching handler. **This is why the failure + prints nothing**, and the reporter set it aside as an unrelated build + warning. + +A is the reason mcpp did not catch it. B is the reason it exists. C is the +reason it presented as a silent abort rather than as one line of text. + +## 1. What was measured + +Host: Linux 6.8, x86_64, RTX 4080 (sm_89), NVIDIA driver 550.144.03. mcpp +`2026.9.8.1` (released binary, its own provisioned registry), `mcpp:plugins` +0.5.2, `xim:dpcpp` 7.1.0, `compat.sycl-runtime` 2026.09.07. The project is +`examples/09-heterogeneous/sycl`, unmodified. The reporter's host is a +different GPU (4070 Ti SUPER) on a different driver (610.57.04) with mcpp +`2026.9.9.1`; the readings below agree with theirs except where noted in §2.3. + +| # | Input | Reading | +|---|---|---| +| 1 | `mcpp build` | green in 2.70 s, no runtime-closure diagnostic | +| 2 | `./bin/sycl-saxpy` | `terminate called ...` / `terminate called recursively`, exit **134** | +| 3 | `SYCL_UR_TRACE=1` | cuda adapter fails on `libnvidia-ml.so.1`; opencl adapter fails on `libOpenCL.so.1`; both level_zero adapters load | +| 4 | `ln -s /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 /` then run | `12 24 36 48` / `device: NVIDIA GeForce RTX 4080` | +| 5 | remove that link, run again | back to row 2 | +| 6 | `/ld.so --help` | search path is the literal `/nonexistent/xlings-use-rpath-not-default-search/lib` | +| 7 | `nm -D` on the artifact vs. libgcc_s | 10 of 18 `_Unwind_*` names defined by the executable | + +Rows 4 and 5 are the pair that matters: one symbolic link, both directions, +same binary, same machine. Row 6 is what makes row 3 fatal rather than +cosmetic — the private loader consults no host directory on any distribution, +so "absent from the farm" is "absent from the machine". + +The same objects were then relinked by hand, four ways, and each variant was +run in both situations — the unrepaired farm (no device reachable) and the repaired +one (the 4080 reachable). This is the evidence §6's R6 rests on: + +| Variant | Unwinders | Exported / overlapping | When it fails | With a device | +|---|---|---|---|---| +| V0 — as mcpp links today | 2 | 89 / 68 | `terminate`, **exit 134**, no text | `12 24 36 48`, exit 0 | +| V1 — `--unwindlib=libgcc`, no `libunwind.a` | 1 | 58 / 58 | **`sycl: no usable device: No device of requested type available.` / `device unavailable`, exit 1** | `12 24 36 48`, exit 0 | +| V2 — V1 + `-Wl,--exclude-libs,ALL` | 1 | **0 / 0** | same as V1, exit 1 | `12 24 36 48`, exit 0 | +| V3 — `--exclude-libs,ALL` alone | 2 | 0 / 0 | **SIGSEGV, exit 139** | `12 24 36 48`, exit 0 | + +V3 is the variant that looks right. It removes every duplicate symbol, its +run with a device is perfect, and it turns a silent abort into a silent +segfault. The failing run is the only place the difference is visible, which is the same +property that let V0 ship. + +Rows 4, 5 and the variant runs were performed against a scratchpad registry and +reverted; the working tree carries no change from this investigation. + +## 2. Mechanism A — the declared dlopen surface is outside every closure + +### 2.1 The walk + +`resolve_runtime_closure` (`src/runtime/elf.cppm:920`) seeds its queue with the +artifact and one thing only (`elf.cppm:943`), then follows `DT_NEEDED`. An +unresolvable SONAME is recorded (`elf.cppm:979`) and, under a hermetic binding, +becomes `Unresolvable` with the message that names the private loader +(`elf.cppm:1229`). + +`libur_adapter_cuda.so.0` is never a `DT_NEEDED` of anything in that closure. +`libsycl.so.9` loads `libur_loader.so.0`, which `dlopen`s each adapter — the +trace in row 3 shows it trying the bare SONAME first and then the absolute path +inside the farm. Nothing in the artifact's link-time graph names it, so it is +outside the walk **by construction**, not by oversight. + +Measured, with the artifact's real `DT_RPATH` as the search path: + +``` +libur_adapter_cuda.so.0 FAIL libnvidia-ml.so.1 +libur_adapter_opencl.so.0 FAIL libOpenCL.so.1 +libur_adapter_level_zero.so.0 ok +libur_loader.so.0 ok +libsycl.so.9 ok +``` + +### 2.2 mcpp already holds every input this check needs + +`runtime_search_dirs` (`src/build/runtime_validation.cppm:449`) assembles the +exact directory list the artifact will use, and it is already passed to the +closure walk at `runtime_validation.cppm:676`. `plan.depRuntimeLibraryDirs` +(`src/build/plan.cppm:265`, filled at `plan.cppm:1124`) is the set of +dependency `runtime.library_dirs` — that is, the set of directories a package +published *because* its contents are reached by `dlopen` rather than by a +header or a link line. + +So the missing check is not a missing capability. It is a missing edge: the one +surface mcpp itself put on the search path is the one surface it does not walk. + +### 2.3 The second gap, and one disagreement between hosts + +`libur_adapter_opencl.so.0` needs `libOpenCL.so.1`, which the farm does not +carry either. On this host that adapter fails; on the reporter's host the trace +shows it **loading**, and the Intel CPU device it enumerated is what their +default selector then chose. Some path on their machine supplies +`libOpenCL.so.1` and it is not visible from here. The gap is real on both — the +farm carries no ICD loader — but its consequence is host-dependent, and that +disagreement is unexplained. See §7. + +This also refines the reporter's causal chain in one place. The abort does not +depend on a wrong device being selected. On this host **no device existed at +all** and the first thrown exception is `No device of requested type +available.`; the observable outcome is identical, down to the exit code. "The +default selector picked the CPU" is a companion symptom of the same missing +adapter, not a link in the chain. + +### 2.4 Why the package's own criterion is green + +`mcpp-index`'s `tests/examples/sycl-runtime/tests/farm.cpp` exists to assert +that this farm works. It `dlopen`s three sonames: `libsycl.so.9`, +`libur_loader.so.0`, `libumf.so.1`. The farm has twenty-five entries. The two +that are broken are not among the three. + +The reason its comment gives for not testing the adapters is that "a machine +with no GPU is a legitimate configuration and is what every runner in this +repository is". That reason does not hold: **whether an adapter can be +`dlopen`ed is a packaging property, not a device property.** This host has no +Level Zero device and both Level Zero adapters load. The two that fail report +`cannot open shared object file`, which is a statement about the farm, not +about the hardware. + +There is a real obstacle behind the wrong reason, and it must be handled or the +repaired test will be red on every runner and will be reverted: on a machine +with no NVIDIA driver the farm's `libcuda.so.1` is a **deliberately dangling** +symlink — `xim:libcuda-host-link` creates it pointing at the canonical path so +that installing a driver later self-heals every consumer. A naive "dlopen +everything" is red there. §5 states the three-state rule that both this test and +the mcpp-side check need. + +## 3. Mechanism B — a hand-written name where the same function enumerates + +`compat.sycl-runtime`'s `install()` builds the farm in two halves: + +* the payload's libraries are **enumerated** — `ls -1 /lib`, link + every versioned SONAME; +* the driver is **hand-written** — one `ls` for + `xim-x-libcuda-host-link/*/lib/libcuda.so.1`, one link, one name. + +The recipe's own header explains, at length and correctly, why the driver has +to be in this farm: once the payload's libraries acquired `RUNPATH = $ORIGIN`, +a non-empty RUNPATH switched off the inherited `DT_RPATH` for their +dependencies (mcpp#460), so the adapter can no longer reach a driver two farms +away. Every word of that reasoning applies to `libnvidia-ml.so.1` unchanged. +The defect is not in the reasoning; it is that the conclusion was written as a +name instead of as a set. + +Two ecosystem constraints bear on the repair, and they rule out one of the two +options the issue proposes. + +**GPU-related index packages are not permitted to probe the host.** The +sentinels `xim:libcuda-host-link` and `xim:nvidia-gl-host-link` are the single +source of truth for where the driver is; `hostlib.lua` records the history — +four independent probes, three of them wrong, one returning a 32-bit +`libcuda.so.1` on a biarch host and failing three layers away (mcpp#352). The +issue's option 2 — copy `farm_libc_stubs` and change the library name — looks +like precedent but is not: that helper searches the **xpkg store**, and +`libnvidia-ml.so.1` is not in any store. Copying it would produce a host probe, +which is the forbidden shape, and would bypass the ELF-class check and the +self-healing dangling-link semantics the sentinel already implements. + +**A version key selects the anchor URL and does not freeze behaviour.** There is +one `install()` in that recipe and it never reads `pkginfo.version()`, so +installing `2026.09.06` today builds the farm `2026.09.07` builds. The +consequence for this repair is the opposite of what it sounds like: existing +pins pick the fix up automatically, but a machine that **already has** the +directory does not reinstall, so a new version key is still required to reach +installed hosts. + +## 4. Mechanism C — two unwinders in one process + +This is the part the issue does not contain, and it is the reason the failure is +as hard to diagnose as it is. + +The island declares a synchronous failure path +(`examples/09-heterogeneous/sycl/app/src/kernels/saxpy.sycl:114`): + +```cpp +} catch (const sycl::exception& e) { + std::fprintf(stderr, "sycl: no usable device: %s\n", e.what()); + rc = -1; +} +``` + +On this host that path should run — there is no usable device — and `main` +should print `device unavailable` and return 1. It aborts instead. The throw is +on the main thread, three frames below the handler: + +``` +#0 __cxa_throw (tinfo = typeinfo for sycl::_V1::exception) libstdc++ +#1 sycl::_V1::detail::select_device … libsycl.so.9 +#3 sycl::_V1::queue::queue<…> +#4 saxpy_device the frame whose caller has the catch +#6 main +``` + +At `abort`, the reason is explicit: + +``` +#3 __cxa_call_terminate +#4 __gxx_personality_v0 (actions=6) libstdc++ +#5 unwind_phase2 () 0x555555618… — the executable's LLVM libunwind +#6 _Unwind_RaiseException () the executable's +``` + +and one frame deeper, during phase 1: + +``` +#0 _Unwind_GetIPInfo (context=0x7fffffffc930) libgcc/unwind-dw2.c:360 libgcc_s +#1 __gxx_personality_v0 (actions=1, …) eh_personality.cc:457 libstdc++ +#2 _Unwind_RaiseException () 0x5555556181f6 LLVM libunwind +``` + +libstdc++'s personality routine reads an `_Unwind_Context` built by **LLVM +libunwind** through **libgcc's** accessor. The two structures are unrelated. The +IP it recovers is meaningless, the LSDA lookup at that IP finds no landing pad, +and phase 2 ends in `__cxa_call_terminate`. `__verbose_terminate_handler` then +rethrows in order to print the exception's type — through libgcc's +`_Unwind_Resume_or_Rethrow`, which is not interposed — and terminates again, +which is the `terminate called recursively` line and the reason no message is +printed. + +The split is measurable and is an artifact of static-archive granularity: + +``` +libgcc_s _Unwind_* entry points 18 +defined by the executable (libunwind.a) 10 +still resolved in libgcc_s 8 incl. _Unwind_GetIPInfo, + _Unwind_GetCFA, _Unwind_Resume_or_Rethrow +``` + +The link line is `-nostdlib++` plus `libc++.a libc++abi.a libunwind.a` named +explicitly, under `-stdlib=libc++ -rtlib=compiler-rt --unwindlib=libunwind`; the +artifact's `DT_NEEDED` is `libsycl.so.9`, `libstdc++.so.6`, `libm.so.6`, +`libc.so.6`, and libgcc_s arrives underneath libstdc++. The linker pulls from +`libunwind.a` only the members that something references and exports them +because a loaded shared library has undefined references to them; the other +eight names were never referenced by libc++abi and so were never pulled in. +Nothing chose ten; ten is what the archive resolution happened to need. + +### 4.1 The other fifty-eight + +The warning says 68, and the unwinder family is only ten of them. The rest are +libc++abi's and libc++'s copies of the `std::` exception root — `vtable`, +`typeinfo`, `typeinfo name`, the destructors and `what()` for `std::exception`, +`bad_alloc`, `bad_cast`, `bad_typeid`, `bad_exception`, `bad_array_new_length` +and `type_info` — plus, for `std::logic_error` and `std::runtime_error`, their +**constructors and `operator=`**. + +Those last two are the same partial-interposition shape as the unwinder, moved +from control flow to object layout. `std::runtime_error`'s constructors are +exported and its **destructor is not**. libstdc++-compiled code that constructs +one therefore builds a libc++ object — whose payload is a +`__libcpp_refstring` — and destroys it through libstdc++'s destructor, which +expects a `__cow_string`. Nothing in this example exercises that path, so it is +a hazard rather than a measured failure; it is recorded because the repair that +fixes the unwinder does not by itself remove it, and V2 does. + +Three consequences worth stating separately: + +1. **The condition is already detected.** mcpp's duplicate-symbol warning names + these symbols and names `libgcc_s.so.1` as the other provider. It classifies + the impact as "the library's own copy is never called". For the unwinder + family the impact is that exception handling does not work. +2. **The seam discipline cannot fix it.** The island rule — device code is + reached only through `extern "C"` — is about symbols crossing a boundary the + author writes. The unwinder is reached through the process-global symbol + namespace by code neither side wrote. +3. **It only shows on the error path.** When nothing throws, this artifact runs + correctly and prints `12 24 36 48`. That is why it shipped. + +The island's comment (`saxpy.sycl:22-31`) says the one failure it cannot turn +into a return code is a missing device image, "which is why the manifest names +the device — and why `mcpp.rules.sycl` warns at build time when it does not". +That mitigation was written against one cause. This issue is the second: the +manifest named the device, the image was compiled for that device, and the back +end that consumes it never loaded. The mitigation is conditioned on the +manifest, and the failure is conditioned on the runtime. + +## 5. The three-state rule + +Both repairs that walk a farm need the same distinction, and it is stated once +here so that each can cite it and neither can be folded into the other: + +| State of a farm member's `DT_NEEDED` SONAME | Meaning | Report | +|---|---|---| +| resolves on the artifact's search path | nothing to say | silent | +| no entry anywhere on that path | **packaging gap** | name the member, the SONAME, and the package that published the directory | +| an entry exists in the farm but is a dangling symlink | **machine gap** — the sentinel's self-healing shape, no driver installed | distinct wording, never an error | + +A `DT_NEEDED` that is an absolute path (`nvidia-gl-host-link` deliberately +patches `/lib/x86_64-linux-gnu/libGLX_nvidia.so.0` into the GL farm) is resolved +as a path, not searched. A survey that searched it reported the GL and Vulkan +farms as broken; they are not, and that survey is not evidence about them. + +## 6. Repairs + +Seven, in three repositories. Each carries its own criterion, because a +requirement that shares another repair's criterion disappears when that repair +ships. + +### R1 — `xim-pkgindex`: the sentinel names a set + +`pkgs/l/libcuda-host-link.lua` probes and links one name. Make the name a list +— `libcuda.so.1`, `libnvidia-ml.so.1` — through the same `hostlib.path_of`, the +same ELF-class check, and the same canonical-path fallback for the +not-yet-installed case, so both links self-heal identically. + +*Criterion.* On a host with a driver, both files exist under the sentinel's +`lib/` and both resolve. On a host without one, both exist and both dangle, and +the package still installs successfully. Denominator: the recipe's name list, +asserted by count, so that a list that silently became empty is visible. + +### R2 — `mcpp-index`: `compat.sycl-runtime` enumerates the sentinel too + +Replace the hand-written `libcuda.so.1` lookup in `install()` with an +enumeration of the sentinel's `lib/` directory, filtered by the same +versioned-SONAME rule the payload half already uses (an unversioned name would +reach the linker, which the header explains). Publish a new version key so hosts +that already hold `2026.09.07` reinstall. `compat.cuda-runtime` carries the same +hand-written shape and should be converted with it. + +*Criterion.* Two checks, and the second is the one that fails today: +1. after install, `/libnvidia-ml.so.1` exists; +2. the private loader resolves the cuda adapter's whole `DT_NEEDED` against the + artifact's real search path — `ld.so --library-path "$RPATH" --list + /libur_adapter_cuda.so.0` prints no `cannot open shared object file`. + This must be run with the fix removed as well; it is red today and must be + red again if R2 is reverted. + +### R3 — `mcpp-index`: the farm's criterion enumerates its own members + +`tests/examples/sycl-runtime/tests/farm.cpp` checks three hand-written names. +Change it to enumerate the farm directory and apply §5 per member: dangling +entries are skipped with a printed note, missing SONAMEs fail. Print the number +of members examined. + +*Criterion.* The test reports a member count greater than zero (a farm that +failed to build must not read as a pass), fails on today's `2026.09.07` farm +naming `libur_adapter_cuda.so.0` and `libnvidia-ml.so.1`, and passes after R1 +and R2. It must also pass unchanged on a GPU-less runner, which is the check +that decides whether the three-state rule was implemented or merely written +down. + +*`dlopen` OF EACH MEMBER IS NOT ENOUGH, and this was measured rather than +foreseen.* The first implementation enumerated the farm and `dlopen`ed every +member, which is the obvious reading of "enumerate the population". It passed on +a farm with NVML removed, because `dlopen` measures the PROCESS: another farm on +the same search path supplied the missing soname (see R7). A package's test has +to be able to fail on that package alone, so the test reads each member's +DT_NEEDED itself and resolves it against the farm plus the three sonames the +artifact has already loaded. The three states are then decided from the farm's +own contents rather than from whatever else the run happened to have. + +*Measured, all three situations:* + +| situation | reading | +|---|---| +| repaired farm, driver present | PASSED, members 26, walked 26 | +| NVML removed from the farm | FAILED, names the member and the soname | +| driver links made dangling | PASSED, members 26, walked 24, two noted | + +### R4 — `mcpp`: the closure check reaches the declared dlopen surface + +Extend the artifact verdict: for each directory in `plan.depRuntimeLibraryDirs`, +read each ELF member and resolve its `DT_NEEDED` against the same `searchDirs` +already computed at `runtime_validation.cppm:676`, applying §5. This needs no +knowledge of SYCL, CUDA or Unified Runtime, which is what keeps it on the right +side of `test_runtime_contract`'s prohibition on branching in mcpp's source on a +provider's vocabulary. Cost is a few dozen ELF headers. + +Severity is **advisory, not fatal**. A farm legitimately holds host-driver links +that dangle, and the check must not turn a CPU-only machine's correct +configuration into a failed build. + +Publish the count of members examined into `resolution.json` alongside the +findings. A check whose "nothing found" and whose "nothing looked" read the same +is the failure mode this repository has named more than once. + +*Criterion.* A unit test in `tests/unit/test_elf_runtime.cpp` over a synthetic +directory holding three members — one whose `DT_NEEDED` resolves, one naming an +absent SONAME, one that is a dangling link — asserting three distinct outcomes +and the member count. Field assertions on the record, not substring matches on +the message. A second check builds the SYCL example against an unrepaired farm and +asserts the warning names `libnvidia-ml.so.1`; it does not need a GPU, which is +the point. + +*Ordering.* R4 lands **after** R1 and R2 reach the index. Landed first, it warns +correctly and loudly about a released package that no user can repair, and the +first thing it reports is our own fixture. + +*The check is configuration-sensitive by construction, which is the point.* Run +against this host's long-lived `~/.mcpp`, it reported the OpenCL gap and NOT the +NVML one -- because that registry's SubOS library view carries +`libnvidia-ml.so.1` through `xim:nvidia-gl-host-link`, so on that configuration +the adapter genuinely resolves. The reporter's machine had no such view. A check +that answered the same on both would be answering from a table rather than from +the search path, which is the failure it exists to remove. + +### R5 — `mcpp`: the duplicate-symbol warning states the real consequence + +When the duplicated set intersects the `_Unwind_*` family, the existing warning +should say that exception handling across the boundary will not work, rather +than that the library's own copy will not be called. Text only; no behaviour +change. + +*Criterion.* The SYCL example's build output contains the unwinder-specific +sentence, and a build whose duplicates are ordinary symbols does not. Two checks, +because a message that always appears carries no information. + +### R6 — `mcpp`: one unwinder per process, and no exported seam + +The invariant is that a process has exactly one unwinder. This artifact has two, +split by which archive members were referenced. + +The repair is **both halves of V2**, and the §1 matrix says why neither alone is +the answer: + +* `--unwindlib=libgcc`, dropping `libunwind.a` from the unit's link line, for an + artifact whose link line already names libstdc++. libc++abi and libstdc++ then + call libgcc's entry points on libgcc's contexts. Measured: the island's + handler runs, the program prints its diagnosis and exits 1. This is the half + that fixes the reported silence. +* `-Wl,--exclude-libs,ALL`, which drops the exported set from 58 to 0 and + removes §4.1's constructor-without-destructor hazard along with it. + +**`-Wl,--exclude-libs` alone is not a repair, it is a regression** — this was +written here as a prediction from the mechanism and then measured as V3: exit +139 instead of 134. Hiding the executable's `_Unwind_*` makes libstdc++ bind to +libgcc, but the executable's own libc++abi still calls its statically linked +libunwind internally, so an exception raised there and unwound through a +libstdc++-compiled frame reproduces the mismatch in the opposite direction. The +flag looks exactly right, its run with a device is clean, and it makes the failure +harder to read. Nobody should have to rediscover that. + +*Predicate.* The change applies to an artifact whose link line names libstdc++. +It is decidable where the link line is assembled, and it is narrow: an artifact +that does not load libstdc++ keeps today's hermetic `libunwind.a` and gains no +`DT_NEEDED` on `libgcc_s.so.1`. V1 and V2 both acquire that entry; it is +satisfied from the payload farm already on the search path, which is why both +variants run. + +*Criterion.* Four cells, all measured on this host and all four required: + +| | when it fails | with a device | +|---|---|---| +| before | exit 134, no text | `12 24 36 48` | +| after | `sycl: no usable device: …`, exit 1 | `12 24 36 48` | + +plus two structural assertions: mcpp's own duplicate-symbol warning does not +appear for this example after the change (it is the cheapest denominator +available — it counts the seam), and an artifact with no libstdc++ on its link +line is byte-for-byte unchanged. + +*Implemented as V4, which is V2 with the archives named.* `--exclude-libs,ALL` +was the measured variant; the shipped one lists `libc++.a` and `libc++abi.a`, +matching the spelling `hide_static_cxx_runtime` already used for shared +libraries, so a user's own static library linked into an artifact keeps its +exports. Measured to be equivalent for this purpose: 0 exported symbols, 0 +overlapping, exit 1 with the diagnosis when it fails, `12 24 36 48` with a +device. + +*The predicate lives where the mechanism table already is.* `MechanismInput` +gains `foreignCxxRuntime`, set in `flags.cppm` from `bc.ldflags` containing +`stdc++` while the toolchain's library is libc++; `distribution.cppm` reads it +in the libc++ ELF branch. The same field widens `hide_static_cxx_runtime` from +shared libraries to executables, correcting the comment quoted in §4. + +*One documented claim was refuted along the way.* The SYCL example states that a +missing device image is the one failure its island cannot turn into a return +code, "because it throws from inside the scheduler in neither of those two +paths". Measured by compiling the same example for `sm_90` and running it on an +sm_89 device: with two unwinders, `terminate called after throwing an instance +of 'ur_result_t'`, exit 134; with one, the island's own handler prints the +build log and `main` prints `device unavailable`, exit 1. It was not outside the +catches; no catch worked. The example's comment and README are corrected. + +*Status.* Measured and implemented. What is **not** measured is the scope +question in §7: `--exclude-libs` on a shared-library artifact is unchanged +behaviour, but an executable that deliberately re-exports an interface from a +static archive would now hide it -- no such artifact exists in this tree, and +the flag is emitted only for the libc++-plus-libstdc++ combination. + +### R7 — decide `libOpenCL.so.1` + +Either declare the install-time edge to `compat.opencl` and farm the ICD loader, +or record that the SYCL lane deliberately offers no OpenCL back end. The present +state — neither carried nor stated — is what let the reporter's machine select a +CPU device silently. + +*Criterion.* R2's second check applied to `libur_adapter_opencl.so.0`, plus the +one that made the reversal safe: with the edge declared, removing NVML from the +farm must still fail R3. + +*Decided: SERVED, after one round of deciding the opposite.* The history is +worth keeping, because the wrong decision was made from a real measurement. + +The edge was written -- `deps = { ["compat.opencl"] = "2026.05.29" }` -- and the +farm test then passed with every member loading. It also passed with +`libnvidia-ml.so.1` REMOVED from the farm. `compat:opencl` depends on +`compat:opencl-runtime`, whose farm mirrors the host's NVIDIA OpenCL family and +therefore carries NVML: the new dependency was satisfying the need R2 exists to +satisfy, and R2's criterion could no longer fail. On that reading the edge was +withdrawn and the adapter was recorded as unserved. + +**That reason expired the moment R3 was hardened.** The masking was a property +of a criterion that measured the PROCESS, and R3 was rewritten -- because of +this very measurement -- to read each member's DT_NEEDED against the farm +alone. Once nothing on the search path can answer for the farm, the only +objection left was the size of the surface, and "the payload ships an adapter +that can never load" is not something a runtime adapter should leave standing. + +So the edge is declared. Measured with it in place: the farm test passes, and +with NVML removed it still FAILS naming `libur_adapter_cuda.so.0` -- the +criterion is no longer maskable, which is what made the decision reversible. + +`libOpenCL.so.1` stays named in the test, with its meaning changed: not "this +package does not serve it" but "a declared dependency of this package provides +it". Everything not on that short list must be in the farm, and the list is +what keeps the self-sufficiency assertion exact without reopening the process +to answer for it. + +*And it found three defects in R4*, all of them false positives, and all of +them invisible until a project with a shared dependency was measured: + +1. **`$ORIGIN` is not in `runtime_search_dirs`.** Every artifact carries it + first in its DT_RPATH and a shared dependency is deployed BESIDE the + executable, so the library resolves at run time and read as missing here. + `runtime_search_dirs` cannot carry it: `$ORIGIN` is a property of each + artifact, not of the plan. The artifacts' own directories are added in + `check_dlopen_surface`. +2. **A plan that produces no program has no surface to judge.** The adapter + package is `kind = "lib"`, and reporting a consumer's surface against an + archive's non-existent search path named a library the consumer resolves. +3. **A SONAME is not a filename.** mcpp links `bin/libopencl.so` whose SONAME + is `libOpenCL.so.1`, and the alias under the SONAME appears later; `mcpp + test` calls the check twice and only the second call saw it. The SONAMEs + this build produces are read from the objects and passed in. + +After all three, the same project reports the same four findings on both calls, +and every one of them is real -- they are the `compat:vulkan-runtime` class +already filed as mcpp-index#376, reached this time through +`compat:opencl-runtime`. + +## 7. Open + +* **The OpenCL disagreement (§2.3).** The adapter loads on the reporter's host + and not here, from the same pinned payload. R7 now serves the adapter through + `compat:opencl`, so the practical consequence is gone; what is still + unexplained is why their machine supplied `libOpenCL.so.1` without it. +* **What R6 does to an executable that is itself a plugin host.** Hiding + `libc++.a` and `libc++abi.a` means a library `dlopen`ed later cannot resolve + the C++ standard library from the executable. That is the intended direction + — it is how the second runtime stops leaking — but a libc++-built plugin + loaded into such a process would now find no provider at all. The + configuration is narrow (it requires libstdc++ already on the line, which is + the broken state this repairs) and no artifact in this tree is one, so it is + recorded rather than handled. +* **Whether other farms have packaging gaps — answered, by R4 itself.** The + earlier survey here used a deliberately partial search path and its findings + were about the survey. R4 run per project answers it properly, and the first + thing it caught is this ecosystem's own `compat:vulkan-runtime`: 4 of 55 + libraries on a host with an NVIDIA driver. + + | library | needs | on this host | + |---|---|---| + | `libnvidia-encode.so.1` | `libnvcuvid.so.1` | present in `/usr/lib`, absent from the farm | + | `libnvidia-opticalflow.so.1` | `libnvcuvid.so.1` | same | + | `libnvidia-pkcs11-openssl3.so.550.144.03` | `libcrypto.so.3` | same | + | `libnvidia-pkcs11.so.550.144.03` | `libcrypto.so.1.1` | same | + + All four are real and all four are the same shape as mcpp#596 -- a farm that + mirrors a driver family and stops one library short. They are NOT repaired + here: a check whose first catch is its author's own package should report it, + not quietly absorb it, and the repair belongs to `compat:vulkan-runtime` with + its own criterion, filed as mcpplibs/mcpp-index#376. On a runner with no NVIDIA driver the farm is nearly empty + and the check is silent, so this does not appear in CI. + +## 8. Order + +R1 → R2 (R2 reads what R1 publishes) → R3 (asserts what R2 produced) → R4 +(would otherwise report a released package no user can repair). R5 and R6 are +independent of that chain and of each other; R6's second half is gated on the +artifact-kind question in §7, its first half is not. R7 landed with R2. + +R5 and R6 are worth stating as one sentence, because they are the same finding +seen from two sides: R6 removes the seam, and R5 is what should have been said +about it while it was still there. Neither is a consequence of #596 — the seam +predates it, the example's README announces it, and the example runs correctly +across it every time nothing throws. #596 is simply the first time something +threw. + +## 8.5 What serving the OpenCL adapter uncovered + +R7's reversal put a shared library into every SYCL project's plan for the first +time, and two latent defects became active the moment it did. Both are repaired +here; neither is caused by #596, and neither would have been found by reading. + +**A build program's objects reached a dependency's image.** `role = "object"` +with no named target attaches to "every linked image", and that was reading as +"every link unit in this plan" -- which includes the shared library a +dependency contributes. Measured: `compat:opencl`'s ICD loader, a C library, +came out of the link with `saxpy_device` and thirty-seven `sycl::` +instantiations in it, 193 dynamic symbols where its own API is 154, and the +process held two copies of the island. `LinkUnit::dependencyOwned` now says +which images are this package's, and the rule reads "every image THIS PACKAGE +produces". + +**The symbol-provision check could not tell a weak definition from a strong +one.** `DynamicSymbol` recorded the type and not the binding, so vague-linkage +definitions -- template instantiations, inline functions, vtables, which the +C++ ABI emits into every image and expects the loader to unify -- were counted +as a second provider. `hide_static_cxx_runtime`'s own comment had already +written the rule ("must not ... unifying those across the process is the +intended C++ ABI behaviour"); nothing enforced it one layer up. The binding is +recorded now, weak definitions are counted rather than reported, and the count +is printed so "clean" cannot read as "did not look". + +Both were invisible before because the same artifact had 68 real findings +sitting on top of them. A check whose noise is repaired shows what the noise +was covering, which is the third time this issue has produced that shape. + +## 9. What this touches, across the three repositories + +The review that asks the other question: not "is each repair right" but "what +else moves when they land". + +| Change | Who reads it | What happens to them | +|---|---|---| +| `xim:libcuda-host-link` 0.0.2 | `ollama` pins 0.0.1 | unchanged; 0.0.1 is kept and still resolves | +| | `compat.cuda-runtime` pins 0.0.1, frozen | untouched, deliberately: its header says it receives no new versions | +| | `compat.cuda-driver`, `compat.sycl-runtime` | move to 0.0.2, which is the only place that decides which sonames they get | +| `compat.*` 2026.09.10 | consumers pinning 2026.09.05/07 | still resolve; a machine that already holds the directory keeps the old farm until something asks for the new key | +| | `examples/09-heterogeneous/sycl` | pin moved, in the mcpp change | +| R4 (the surface walk) | every Linux build with a dependency `runtime.library_dirs` | a warning where there is a real gap. Measured: 0 for a project with no such dependency, 1 for the SYCL example (the declared-unserved OpenCL adapter), 4 for the Vulkan example on a host with an NVIDIA driver (real, filed as mcpp-index#376) | +| | a non-hermetic binding, or `allow_host_libs` | silent, for the reason the artifact verdict is | +| R6 (one unwinder) | a libc++ link line naming libstdc++ | `--unwindlib=libgcc` and hidden archives | +| | every other link | byte-for-byte unchanged, asserted in `test_distribution.cpp` | +| R7 (`compat:opencl` declared) | every SYCL project | the OpenCL back end loads; one shared library and one symlink farm added to the graph | +| | the two defects in §8.5 | latent before, active from the moment a shared library entered a SYCL plan | +| `compat.opencl` on Windows | a Windows OpenCL consumer | a loader to link, where there was none; no adapter, because the system loader needs no help | + +The one regression this could cause is in §7's last item: an executable that +deliberately re-exports the C++ standard library to a plugin it `dlopen`s. It +requires libstdc++ already on the line, which is the broken state R6 repairs, +and no artifact in this ecosystem is one. diff --git a/.agents/docs/2026-09-10-596-verify.sh b/.agents/docs/2026-09-10-596-verify.sh new file mode 100755 index 00000000..6b290b61 --- /dev/null +++ b/.agents/docs/2026-09-10-596-verify.sh @@ -0,0 +1,174 @@ +#!/usr/bin/env bash +# Ecosystem verification for mcpp#596: the driver farm mirrors the sentinel, +# and mcpp reports a dlopen surface it cannot satisfy. Against a PUBLISHED +# mcpp, a PUBLISHED xim:libcuda-host-link and a PUBLISHED compat:sycl-runtime. +# +# # The sandbox has an EMPTY $HOME and a fresh /tmp, so this file is not +# # visible from inside it. Pass the script itself in: +# B64=$(base64 -w0 ) +# xlings subos use verify-596 --sandbox --cmd \ +# "echo $B64 | base64 -d > /tmp/v.sh && MCPP_VERIFY_VERSION=2026.9.10.1 bash /tmp/v.sh" +# +# mcpp is addressed by its STORE path, which is the one thing the sandbox does +# share: the xlings data directory. A bare `mcpp` is not on PATH in there. +# +# WHAT A SANDBOX CAN AND CANNOT DECIDE HERE. +# +# It decides everything about the PACKAGING, which is where the defect was: how +# many driver sonames the sentinel publishes, and whether the farm carries all +# of them. Those are properties of what gets installed, and a machine that has +# built this before answers them from a directory that already existed. +# +# It cannot decide anything about a DEVICE. The sandbox's /dev has fourteen +# entries and no NVIDIA node, so `libcuda.so.1` resolves to a dangling link +# there exactly as it does on any machine without a driver -- which is a +# supported configuration and is asserted as such below, not worked around. The +# device side is measured on the host and recorded in the design record. +# +# EVERY CRITERION NAMES THE OBJECT IT SELECTED, and every section that did not +# run is listed again in the summary. "0 assertions failed" printed by a script +# that skipped three sections is the failure mode this shape exists to prevent. +set -u + +VER="${MCPP_VERIFY_VERSION:?set MCPP_VERIFY_VERSION}" +STORE="${MCPP_VERIFY_BIN:-$HOME/.xlings/data/xpkgs/xim-x-mcpp/$VER/bin/mcpp}" +SENTINEL_VER="${MCPP_VERIFY_SENTINEL:-0.0.2}" +COMPAT_VER="${MCPP_VERIFY_COMPAT:-2026.09.10}" + +fails=0 +skipped="" +fail() { printf 'ASSERT-FAIL: %s\n' "$1"; fails=$((fails + 1)); } +ok() { printf 'ok: %s\n' "$1"; } +section() { printf '\n== %s ==\n' "$1"; } +skip() { printf 'NOT RUN: %s\n' "$1"; skipped="$skipped + - $1"; } + +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT + +section "A. the published mcpp answers for itself" +if [ ! -x "$STORE" ]; then + fail "no mcpp at $STORE" + printf '\nfails=%d (nothing else can run)\n' "$fails" + exit 1 +fi +got=$("$STORE" --version 2>&1 | head -1) +case "$got" in + *"$VER"*) ok "mcpp --version says $got" ;; + *) fail "mcpp --version says '$got', expected $VER" ;; +esac + +# --------------------------------------------------------------------------- +section "B. the sentinel publishes a SET, and the size is the assertion" +# +# The count is the criterion rather than the presence of one name: a list that +# lost an entry passes every per-name test that only asks about the names it +# still has. +if "$STORE" self >/dev/null 2>&1 || true; then :; fi +xl="$HOME/.xlings/data/xpkgs/xim-x-libcuda-host-link/$SENTINEL_VER/lib" +if command -v xlings >/dev/null 2>&1; then + xlings install "libcuda-host-link@$SENTINEL_VER" -y >"$work/sentinel.log" 2>&1 || true +fi +if [ -d "$xl" ]; then + n=$(ls -1 "$xl" | wc -l) + [ "$n" -ge 2 ] && ok "sentinel $SENTINEL_VER publishes $n sonames" \ + || fail "sentinel publishes $n soname(s); the set is at least 2" + for s in libcuda.so.1 libnvidia-ml.so.1; do + # islink, not exists: a dangling link is the documented self-heal shape + # on a machine with no driver, and the sandbox is such a machine. + [ -L "$xl/$s" ] && ok "sentinel carries $s" || fail "sentinel has no $s" + done +else + skip "B: the sentinel is not installed in this environment ($xl)" +fi + +# --------------------------------------------------------------------------- +section "C. the farm mirrors the sentinel" +# +# THE DEFECT ITSELF. The farm linked one hand-written name while enumerating +# every other directory it draws from, so it carried libcuda.so.1 and not +# libnvidia-ml.so.1, the CUDA adapter did not load, and the program aborted +# with no message. +cat >"$work/mcpp.toml" <"$work/src/main.cpp" <<'CPP' +int main() { return 0; } +CPP +if (cd "$work" && "$STORE" build >"$work/build.log" 2>&1); then + # Searched under the REGISTRY roots rather than all of $HOME: a developer + # home holds tens of gigabytes of packages and the walk costs minutes, + # which reads exactly like a hung verification. + farm="" + for root in "${MCPP_HOME:-$HOME/.mcpp}" "$work/.mcpp" "$HOME/.xlings"; do + [ -d "$root" ] || continue + farm=$(find "$root" -path "*compat-x-sycl-runtime/$COMPAT_VER*/sycl_runtime/lib" \ + -type d 2>/dev/null | head -1) + [ -n "$farm" ] && break + done + if [ -n "$farm" ]; then + for s in libcuda.so.1 libnvidia-ml.so.1; do + [ -L "$farm/$s" ] && ok "farm carries $s" \ + || fail "farm has no $s -- this is mcpp#596" + done + # The denominator: a farm that failed to build is empty, and every + # per-name test above would then have failed for the wrong reason. + n=$(ls -1 "$farm" | wc -l) + [ "$n" -ge 20 ] && ok "farm has $n entries" \ + || fail "farm has only $n entries; the payload half did not build" + else + fail "C: no farm directory for compat:sycl-runtime@$COMPAT_VER" + fi +else + skip "C: the probe project did not build (see $work/build.log; the dpcpp payload is over a gigabyte)" +fi + +# --------------------------------------------------------------------------- +section "D. mcpp reports a dlopen surface it cannot satisfy" +# +# The record rather than the message: a test that greps a warning's wording +# fails the next time the wording improves. Both denominators are asserted for +# the reason they exist -- "no findings" and "nothing was examined" must not +# read the same. +res=$(find "$work" -name resolution.json 2>/dev/null | head -1) +if [ -n "$res" ] && command -v python3 >/dev/null 2>&1; then + python3 - "$res" <<'PY' +import json, sys +doc = json.load(open(sys.argv[1])) +rec = doc.get("runtime", {}).get("dlopen_surface") +if rec is None: + print("ASSERT-FAIL: resolution.json has no runtime.dlopen_surface") + sys.exit(1) +members, walked = rec.get("members", 0), rec.get("walked", 0) +if members <= 0: + print(f"ASSERT-FAIL: dlopen_surface examined {members} members") + sys.exit(1) +print(f"ok: dlopen_surface examined {walked} of {members} members") +missing = [f for f in rec.get("findings", []) if f.get("kind") == "missing"] +for f in missing: + print(f"note: {f['library']} needs {f['soname']} (declared unserved: libOpenCL.so.1)") +bad = [f for f in missing if f.get("soname") == "libnvidia-ml.so.1"] +if bad: + print("ASSERT-FAIL: NVML is still missing from the farm -- mcpp#596") + sys.exit(1) +print("ok: no driver soname is missing from the farm") +PY + [ $? -eq 0 ] || fails=$((fails + 1)) +else + skip "D: no resolution.json (section C did not build) or no python3" +fi + +# --------------------------------------------------------------------------- +printf '\n== summary ==\n' +printf 'assertions failed: %d\n' "$fails" +if [ -n "$skipped" ]; then + printf 'sections NOT RUN:%s\n' "$skipped" + printf 'A pass with sections not run is not a pass for those sections.\n' +fi +exit $((fails > 0)) diff --git a/.agents/docs/README.md b/.agents/docs/README.md index 13da79ea..8dec1f7c 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -273 records. +274 records. ## By subject @@ -32,6 +32,7 @@ Records that declare one. Everything else is listed by date below. ### heterogeneous +- [A dlopen surface no closure walks, and a process with two unwinders](2026-09-09-dlopen-surface-and-two-unwinders.md) — active - [The island boundary's names: one rule for both lanes, and the check that makes it true](2026-09-08-island-boundary-names.md) — active - [Implementation plan: the island boundary's names](2026-09-08-island-boundary-names-implementation-plan.md) — active @@ -44,6 +45,7 @@ Records that declare one. Everything else is listed by date below. ### 2026-09 - [Two answers and two silences: the scanner's second grammar, and the manifest keys nothing reads](2026-09-09-two-answers-and-two-silences.md) — active +- [A dlopen surface no closure walks, and a process with two unwinders](2026-09-09-dlopen-surface-and-two-unwinders.md) — active - [The documentation as a book: a chapter-by-chapter design](2026-09-08-the-documentation-as-a-book.md) — active - [The island boundary's names: one rule for both lanes, and the check that makes it true](2026-09-08-island-boundary-names.md) — active - [Implementation plan: the island boundary's names](2026-09-08-island-boundary-names-implementation-plan.md) — active diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 9175d63a..77ccf587 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -99,6 +99,19 @@ jobs: - name: Install qemu-user-static run: | + # The runner image carries third-party apt lists (Google Chrome + # among them) that this job does not use, and a transient + # `Hash Sum mismatch` on one of them fails the whole update -- which + # killed two cross-build jobs in setup, before a single byte was + # compiled. Dropping the lists this job has no use for is what makes + # the step's failure mean something about this job. + # + # BY CONTENT, NOT BY FILENAME. The first attempt removed + # `google-chrome.list` and the update failed on the same URL: on + # ubuntu-24.04 the runner writes deb822 `.sources` files, so the + # name was a guess and the guess was wrong. + sudo grep -rlE 'dl[.]google[.]com|packages[.]microsoft[.]com' \ + /etc/apt/sources.list.d/ 2>/dev/null | xargs -r sudo rm -f sudo apt-get update -qq sudo apt-get install -y qemu-user-static ${{ matrix.qemu_bin }} --version | head -1 @@ -251,6 +264,19 @@ jobs: sudo dpkg -i ~/wine-debs/*.deb 2>/dev/null \ || { sudo apt-get update -qq; sudo apt-get install -f -y; } else + # The runner image carries third-party apt lists (Google Chrome + # among them) that this job does not use, and a transient + # `Hash Sum mismatch` on one of them fails the whole update -- which + # killed two cross-build jobs in setup, before a single byte was + # compiled. Dropping the lists this job has no use for is what makes + # the step's failure mean something about this job. + # + # BY CONTENT, NOT BY FILENAME. The first attempt removed + # `google-chrome.list` and the update failed on the same URL: on + # ubuntu-24.04 the runner writes deb822 `.sources` files, so the + # name was a guess and the guess was wrong. + sudo grep -rlE 'dl[.]google[.]com|packages[.]microsoft[.]com' \ + /etc/apt/sources.list.d/ 2>/dev/null | xargs -r sudo rm -f sudo apt-get update -qq sudo apt-get install -y --download-only wine64 wine \ || sudo apt-get install -y --download-only wine diff --git a/.github/workflows/openkal-cross.yml b/.github/workflows/openkal-cross.yml index 3ac63d1d..fe6981b4 100644 --- a/.github/workflows/openkal-cross.yml +++ b/.github/workflows/openkal-cross.yml @@ -368,6 +368,19 @@ jobs: - name: Install the emulators the last two scripts need run: | set -euo pipefail + # The runner image carries third-party apt lists (Google Chrome + # among them) that this job does not use, and a transient + # `Hash Sum mismatch` on one of them fails the whole update -- which + # killed two cross-build jobs in setup, before a single byte was + # compiled. Dropping the lists this job has no use for is what makes + # the step's failure mean something about this job. + # + # BY CONTENT, NOT BY FILENAME. The first attempt removed + # `google-chrome.list` and the update failed on the same URL: on + # ubuntu-24.04 the runner writes deb822 `.sources` files, so the + # name was a guess and the guess was wrong. + sudo grep -rlE 'dl[.]google[.]com|packages[.]microsoft[.]com' \ + /etc/apt/sources.list.d/ 2>/dev/null | xargs -r sudo rm -f sudo apt-get update -qq && sudo apt-get install -y -qq qemu-user "$XLINGS_BIN" install xim:qemu-riscv -y XLINGS_HOME="${MCPP_HOME:-$HOME/.mcpp}/registry" \ diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f3f3f0fe..d8ec4b5e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -275,6 +275,19 @@ jobs: - name: Install system deps + qemu run: | + # The runner image carries third-party apt lists (Google Chrome + # among them) that this job does not use, and a transient + # `Hash Sum mismatch` on one of them fails the whole update -- which + # killed two cross-build jobs in setup, before a single byte was + # compiled. Dropping the lists this job has no use for is what makes + # the step's failure mean something about this job. + # + # BY CONTENT, NOT BY FILENAME. The first attempt removed + # `google-chrome.list` and the update failed on the same URL: on + # ubuntu-24.04 the runner writes deb822 `.sources` files, so the + # name was a guess and the guess was wrong. + sudo grep -rlE 'dl[.]google[.]com|packages[.]microsoft[.]com' \ + /etc/apt/sources.list.d/ 2>/dev/null | xargs -r sudo rm -f sudo apt-get update -qq sudo apt-get install -y curl git build-essential qemu-user-static qemu-aarch64-static --version | head -1 diff --git a/CHANGELOG.md b/CHANGELOG.md index 0cd93b6e..4ef10010 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,82 @@ ## [Unreleased] +## [2026.9.10.1] - 2026-09-10 + +三条改动来自同一次排查(#596):一个 SYCL 工程构建全绿、运行时以退出码 134 终止且 +不打印任何异常文本。触发原因在生态侧(适配包的 farm 少了一个驱动库),但**它之所以 +以最难排查的形态出现,原因在 mcpp 这一侧**,而且有两条独立的缺口。设计记录见 +`.agents/docs/2026-09-09-dlopen-surface-and-two-unwinders.md`。 + +### 依赖为 dlopen 发布的那个面,现在会被走一遍 + +运行期闭包检查从产物出发沿 `DT_NEEDED` 走。一个包通过 `runtime.library_dirs` 发布的 +库之所以存在,正是因为有东西要 `dlopen` 它 —— 没有任何链接边指向它,所以它**按构造** +在那条闭包之外,不是被漏掉。实测:一个 25 个成员的 farm 里有两个根本加载不了,而构建 +零诊断。 + +`mcpp build` 之后会单独走这个面,把每个成员自己的 `DT_NEEDED` 按产物真实的搜索路径 +解析一遍,并区分三种读数:解析到(静默)、farm 里存在但链接悬空(机器没装驱动,静默)、 +到处都不存在(打包缺口,警告)。中间那一行是这条检查是**警告而不是错误**的原因。 + +完整结果发布在 `resolution.json` 的 `runtime.dlopen_surface`,含两个分母 +(`members` / `walked`)—— 一个构建失败的 farm 枚举出零个成员,否则「没有发现」与 +「什么都没检查」读起来一模一样。 + +判据要落在**产物真实的搜索路径**上,这一点花了三次修正才对(全部是假阳性,且都要 +先有一个带共享依赖的工程才看得见):产物 `DT_RPATH` 首位的 `$ORIGIN` 不在 +`runtime_search_dirs` 里,而共享依赖就部署在可执行文件旁边;不产出程序的计划没有 +可判的面(适配包自己是 `kind = "lib"`);以及 **SONAME 不是文件名** —— mcpp 链出的是 +`bin/libopencl.so` 而它的 SONAME 是 `libOpenCL.so.1`,别名稍后才出现。 + +### 链接行上出现第二个 C++ 运行时时,进程只保留一个 unwinder + +设备编译器按 libstdc++ 配置的 lane(SYCL、HIP)会把 libstdc++ 放上链接行,而产物 +自己静态链接 libc++。此前的注释断言「可执行文件的静态运行时已经是 local 的」—— +前提对(链接器只导出被加载对象引用到的符号),结论错:**当被加载的对象确实引用它们时, +链接器就把它们放进 `.dynsym`**。实测 89 个导出符号,其中 68 个同时由 libstdc++ 或 +libgcc_s 定义。 + +静态归档只贡献被引用到的成员,所以这种抢占按构造是部分的:libgcc 的 18 个 `_Unwind_*` +入口点里 10 个来自产物、8 个仍在 libgcc_s(含 personality 例程要用的访问器)。 +于是一次抛出被两个 unwinder 分着处理,`__gxx_personality_v0` 越过三帧之上一个本应命中 +的 handler 调用 `__cxa_call_terminate`,而 `__verbose_terminate_handler` 为了打印 +异常类型的那次 rethrow 又终止一次 —— 于是一个字都没印出来。 + +现在这类链接改用 `--unwindlib=libgcc` 并用 `--exclude-libs` 挡住静态归档的导出。 +libgcc_s 本来就在进程里(libstdc++ 需要它),所以这一步只点名一个已有的库。 +链接行上没有第二个运行时的构建一字节不变。 + +实测(同一台机器、同一份源码、两种情形):修改前退出码 134 且无输出;修改后 +`sycl: no usable device: ...` / `device unavailable`,退出码 1;设备可用时两者都是 +`12 24 36 48`。**顺带推翻了示例里一条写下来的结论** —— 设备镜像不匹配的那次抛出 +并非拦不住,它拦不住只是因为 unwinder 是坏的。 + +### 构建程序的对象不再进入依赖的映像 + +`role = "object"` 的 action 在没有点名 target 时按「每一个链接映像」附着,而这句 +一直被读成「计划里的每一个链接单元」—— 其中包含**依赖贡献的共享库**。实测: +`compat:opencl` 的 ICD loader(一个纯 C 库)链完带着 `saxpy_device` 和 37 个 +`sycl::` 实例化,导出 193 个符号而它自己的 API 只有 154 个,进程里于是有两份岛。 +`LinkUnit::dependencyOwned` 现在区分「本包产出的映像」与「依赖贡献的映像」。 + +这条一直潜伏着:直到 SYCL 工程第一次有了共享依赖才被激活。 + +### 重复符号检查按符号绑定分流 + +`DynamicSymbol` 只记了类型不记绑定,于是 vague-linkage 定义(模板实例、inline +函数、vtable —— C++ ABI 要求每个映像各发一份、由加载器统一)被算成了第二个提供者。 +`hide_static_cxx_runtime` 的注释早就写下了这条规则,但上一层没有任何东西执行它。 +现在绑定被记录,weak 定义只计数不算冲突,而且**计数会被打印** —— 否则「干净」读起来 +就等于「没看」。 + +同样是被那 68 条真冲突盖住的:修好噪声之后才看得见噪声盖住了什么。 + +### 重复符号警告会说出 unwinder 的真实后果 + +该警告此前把后果一律描述为「库自己的那份副本不会被调用」。对 `_Unwind_*` 这一族, +真实后果是异常处理整体失效。重复集合里含该族时,警告会单独说明这一点。 + ## [2026.9.9.1] - 2026-09-09 本次修复的四条缺陷,来源是同一类问题:一个问题被回答了两次,而读答案的地方各读各的; diff --git a/docs/33-authoring-an-adapter.md b/docs/33-authoring-an-adapter.md index 0abe6b20..830b2b89 100644 --- a/docs/33-authoring-an-adapter.md +++ b/docs/33-authoring-an-adapter.md @@ -82,6 +82,44 @@ is then empty and the program reports what it actually found. An adapter that errors on a missing host library turns a supported configuration into a build failure. +## The check mcpp runs on this surface + +The libraries an adapter publishes are reached by `dlopen`, so no link edge +names them and the runtime closure check — which walks `DT_NEEDED` from the +artifact — cannot arrive at them. mcpp walks them separately, after the link, +and reports what it finds as a warning: + +``` +warning: 1 of 13 libraries a dependency published for dlopen cannot be loaded +on this artifact's search path: + libur_adapter_cuda.so.0 needs libnvidia-ml.so.1 +``` + +Three states are separated, and only one of them is reported: + +| The library needs a SONAME that is | Meaning | Reported | +|---|---|---| +| on the artifact's search path | nothing to say | no | +| present in the farm as a dangling link | the machine has no such driver | no | +| absent everywhere | the adapter did not carry it | yes | + +The middle row is why this is a warning and not an error: a dangling link is +the documented shape of a host driver that is not installed, and a check that +failed there would turn a supported configuration into a build failure. + +The full result, including both denominators, is published as +`runtime.dlopen_surface` in `resolution.json`: + +```json +{ "members": 13, "walked": 13, + "findings": [ { "library": "libur_adapter_cuda.so.0", "dir": "...", + "soname": "libnvidia-ml.so.1", "kind": "missing" } ] } +``` + +`members` and `walked` are published even when nothing is reported. A farm that +failed to build enumerates nothing, and "no findings" would otherwise be +indistinguishable from "nothing was examined". + ## Current limitations - **Linux only, by construction.** macOS's dyld and the Windows PE loader have diff --git a/docs/42-heterogeneous-builds.md b/docs/42-heterogeneous-builds.md index 4ef7aa60..f9ba54f2 100644 --- a/docs/42-heterogeneous-builds.md +++ b/docs/42-heterogeneous-builds.md @@ -676,6 +676,29 @@ Which shape to choose is a property of the program, not of mcpp: a seam that swaps an implementation wants link-time selection, and a program that ships to machines it has not seen wants run-time selection. +## One unwinder, when the link carries a second C++ runtime + +A lane whose device compiler is configured against libstdc++ puts libstdc++ on +the link line while the artifact links libc++ statically. Both are then in the +image, and mcpp's duplicate-symbol check reports what they share. + +For most of those symbols the consequence is that one implementation is called +instead of an interchangeable other. For the unwinder it is not. A static +archive contributes only the members something references, so the interposition +is partial by construction: measured on the SYCL lane, ten of libgcc's eighteen +`_Unwind_*` entry points came from the artifact and eight stayed in libgcc_s, +including the accessors a personality routine uses. libstdc++'s personality +then read an LLVM libunwind context through libgcc's accessors, found no +landing pad, and called `std::terminate` past a handler three frames up. The +program was correct until something threw. + +So when the link line names libstdc++ and the toolchain's own library is +libc++, mcpp links the unwinder from libgcc (`--unwindlib=libgcc`) instead of +the payload's `libunwind.a`, and hides the static archives' symbols with +`--exclude-libs`. libgcc_s is already in the process — libstdc++ needs it — so +this names a library rather than adding one, and the C++ runtime stays +embedded. A build with no second runtime on its line is unchanged. + ## Two boundaries worth stating **`--accel` and `--no-accel` are `build`, `run` and `test` options** (run and diff --git a/docs/zh/33-authoring-an-adapter.md b/docs/zh/33-authoring-an-adapter.md index b1dff02f..ea704c92 100644 --- a/docs/zh/33-authoring-an-adapter.md +++ b/docs/zh/33-authoring-an-adapter.md @@ -67,6 +67,40 @@ rpath : …/xim-x-glibc/2.39/lib64:…/xim-x-gcc/…/lib64:$ORIGIN 每一台 CI runner 都是这样的机器。此时农场为空,程序报告它实际找到了什么。一个在宿主库 缺失时报错的适配包,会把一种受支持的配置变成构建失败。 +## mcpp 对这个面的检查与报告 + +适配包发布的库是被 `dlopen` 找到的,没有任何链接边指向它们,因此运行期闭包检查 +—— 它从产物出发沿 `DT_NEEDED` 走 —— 按构造到不了这些库。mcpp 在链接之后单独走一遍 +这个面,并把结果作为警告报出: + +``` +warning: 1 of 13 libraries a dependency published for dlopen cannot be loaded +on this artifact's search path: + libur_adapter_cuda.so.0 needs libnvidia-ml.so.1 +``` + +三种读数被区分开,只有一种被报告: + +| 该库需要的 SONAME | 含义 | 是否报告 | +|---|---|---| +| 在产物的搜索路径上 | 无话可说 | 否 | +| 在农场里存在,但链接悬空 | 这台机器没有这个驱动 | 否 | +| 到处都不存在 | 适配包没有携带它 | 是 | + +中间那一行正是这条检查是警告而不是错误的原因:悬空链接是「宿主驱动尚未安装」的 +既定形状,一条在那里失败的检查会把受支持的配置变成构建失败。 + +完整结果(含两个分母)发布在 `resolution.json` 的 `runtime.dlopen_surface`: + +```json +{ "members": 13, "walked": 13, + "findings": [ { "library": "libur_adapter_cuda.so.0", "dir": "...", + "soname": "libnvidia-ml.so.1", "kind": "missing" } ] } +``` + +即使没有任何发现,`members` 与 `walked` 也会被发布。一个构建失败的农场枚举出零个成员, +否则「没有发现」与「什么都没检查」就读起来一模一样。 + ## 当前边界 - **按构造只适用于 Linux。** macOS 的 dyld 与 Windows 的 PE 加载器没有对应的这一层, diff --git a/docs/zh/42-heterogeneous-builds.md b/docs/zh/42-heterogeneous-builds.md index 584e5f72..e6495b30 100644 --- a/docs/zh/42-heterogeneous-builds.md +++ b/docs/zh/42-heterogeneous-builds.md @@ -562,6 +562,25 @@ sources = ["src/cpu/*.cpp"] 选哪个形态是**程序自己的性质**,不是 mcpp 的:用接缝换实现的程序要链接期选择,而要发到 没见过的机器上去的程序要运行期选择。 +## 链接行上出现第二个 C++ 运行时时,进程只保留一个 unwinder + +一条 lane 的设备编译器若是按 libstdc++ 配置的,链接行上就会出现 libstdc++, +而产物本身静态链接 libc++。两者同时在镜像里,mcpp 的重复符号检查会报出它们共有的符号。 + +对其中大多数符号,后果只是「调用了另一份可互换的实现」。对 unwinder 不是。 +静态归档只会贡献被引用到的成员,所以这种抢占**按构造是部分的**:在 SYCL lane 上实测, +libgcc 的 18 个 `_Unwind_*` 入口点里有 10 个来自产物、8 个仍在 libgcc_s, +其中包括 personality 例程要用的那几个访问器。于是 libstdc++ 的 personality +拿 libgcc 的访问器去读一个 LLVM libunwind 的 context,找不到 landing pad, +越过三帧之上一个本应命中的 handler 直接 `std::terminate`。 +在有东西抛出之前,这个程序一直是正确的。 + +因此,当链接行上出现 libstdc++ 而工具链自带的标准库是 libc++ 时, +mcpp 改从 libgcc 取 unwinder(`--unwindlib=libgcc`),不再链载荷的 `libunwind.a`, +并用 `--exclude-libs` 把静态归档的符号挡在动态符号表之外。libgcc_s 本来就在进程里 +—— libstdc++ 需要它 —— 所以这一步只是点名一个已有的库而不是新增一个, +C++ 运行时仍然是内嵌的。链接行上没有第二个运行时的构建一字节不变。 + ## 两条值得写明的边界 **`--accel` 与 `--no-accel` 是 `build`、`run`、`test` 三者的选项**(run 与 test 自 2026.9.5.2 起),与 `--target`、`--profile` 同级;`pack` 与其它构建输入一样从 manifest 读 `[build] accel`。它起初只挂在 `build` 上,实测的后果是一个工程的 CPU-only 变体能构建却不能运行:`mcpp build --no-accel` 产出了它,而 `mcpp run` 交回的是设备构建。 diff --git a/examples/09-heterogeneous/sycl/README.md b/examples/09-heterogeneous/sycl/README.md index f3600225..d5c5a260 100644 --- a/examples/09-heterogeneous/sycl/README.md +++ b/examples/09-heterogeneous/sycl/README.md @@ -73,8 +73,18 @@ without them and names the line to add. ## Two C++ runtimes, and why the seam is not optional here `libsycl.so` is compiled against libstdc++ while an mcpp artifact links libc++, -so both are in the image. mcpp's duplicate-symbol check reports the unwinder -symbols they share, and the warning is correct. +so both are in the image. + +The unwinder is the one part of that seam a `catch` cannot police, because +nothing in this source reaches it: an exception is raised through whichever +`_Unwind_*` the process resolved, and a static archive contributes only the +members something referenced. On this lane ten of libgcc's eighteen entry +points came from the artifact and eight from libgcc_s, so libstdc++'s +personality routine read an LLVM libunwind context through libgcc's accessors +and every handler below was skipped. mcpp gives such a link one unwinder +(`--unwindlib=libgcc`, since the process already has libstdc++'s) and hides the +static archives' symbols, so the duplicate-symbol warning no longer fires here +and the `catch` blocks below do what they say. Nothing may cross the seam. The island catches its own `sycl::exception` and returns a code, because the runtime that threw it is not the one the caller @@ -91,10 +101,17 @@ Making that promise true took three things, and only two of them are a `catch`: without one gets the default handler, and the default handler calls `std::terminate` — which no `catch` can intercept, since it never travels as an exception through this frame; -* and one failure remains outside both. A build compiled to SPIR-V, run against - a back end that does not consume it, throws from inside the SYCL scheduler. - That is why this manifest names the device, and why `mcpp.rules.sycl` warns at - build time when an `accel` names `sycl` and no device. +* and the third is not a `catch` at all: **one unwinder in the process**. A + build whose image does not match the device throws from inside the SYCL + scheduler, and the sentence that used to stand here said that failure was + outside both catches. It was outside them because no catch worked. Measured + on one machine, same source, same device: with two unwinders, exit 134 and no + output; with one, `sycl: The program was built for 1 devices` followed by + `device unavailable`, exit 1. + +The manifest still names the device, and `mcpp.rules.sycl` still warns when an +`accel` names `sycl` and no device: an image compiled for the device is the +point of an ahead-of-time build. ## Running it diff --git a/examples/09-heterogeneous/sycl/app/mcpp.toml b/examples/09-heterogeneous/sycl/app/mcpp.toml index 3fc75d80..97ac1963 100644 --- a/examples/09-heterogeneous/sycl/app/mcpp.toml +++ b/examples/09-heterogeneous/sycl/app/mcpp.toml @@ -30,8 +30,14 @@ plugins = { version = "0.5.2", features = ["rules-sycl", "tools-island"], host-m # SYCL runtime's CUDA back end dlopens the driver, so `compat:sycl-runtime` # carries that hop itself from 2026.09.07 on. A project that writes SYCL does # not have to know CUDA is underneath it. +# +# 2026.09.10 is the version at which it carries the WHOLE hop. Until then it +# linked one driver library by name, and the adapter needs two: the second was +# missing, the CUDA back end did not load, and the program aborted with no +# message (#596). The farm now mirrors what the driver sentinel publishes +# rather than naming a file. [dependencies.compat] -sycl-runtime = "2026.09.07" +sycl-runtime = "2026.09.10" # NO [xlings.workspace]. `mcpp.rules.sycl` declares all five payloads this lane # needs, each closing one hole the host would otherwise fill: `dpcpp` (the diff --git a/examples/09-heterogeneous/sycl/app/src/kernels/saxpy.sycl b/examples/09-heterogeneous/sycl/app/src/kernels/saxpy.sycl index d61ce400..2ad06b15 100644 --- a/examples/09-heterogeneous/sycl/app/src/kernels/saxpy.sycl +++ b/examples/09-heterogeneous/sycl/app/src/kernels/saxpy.sycl @@ -23,12 +23,27 @@ // // It catches the synchronous half (no usable device, a rejected submit) and // installs a handler for the asynchronous half, which SYCL otherwise delivers -// to a default handler that calls `std::terminate`. What it cannot catch is a -// missing device image: a build compiled to SPIR-V, run against a back end -// that does not consume SPIR-V, throws from inside the scheduler -// (`ProgramManager::getDeviceImage`) in neither of those two paths. That is -// why the manifest names the device -- and why `mcpp.rules.sycl` warns at -// build time when it does not. +// to a default handler that calls `std::terminate`. +// +// A MISSING DEVICE IMAGE IS CAUGHT TOO, AND THE SENTENCE THAT USED TO SAY +// OTHERWISE WAS DESCRIBING A BROKEN UNWINDER. It read: what this cannot catch +// is a missing device image, because it throws from inside the scheduler in +// neither of those two paths. That was measured on a build whose two C++ +// runtimes had split the unwinder between them, so NO handler in this file ran +// -- including the one three frames above the throw. With one unwinder in the +// process (mcpp 2026.9.10.1), the same build compiled for the wrong +// architecture prints +// +// sycl: The program was built for 1 devices +// device unavailable +// +// and exits 1. Measured both ways on one machine: exit 134 and silence before, +// exit 1 and the text above after. +// +// The manifest still names the device, and `mcpp.rules.sycl` still warns when +// it does not, for the reason that has not changed: an image compiled for the +// device is the point of an ahead-of-time build. What changed is that failing +// to have one is now something this island can report. #include #include #include diff --git a/mcpp.toml b/mcpp.toml index ded70a15..70e5074e 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.9.9.1" +version = "2026.9.10.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index e30a29e8..376ad844 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.9.9.1"; +inline constexpr std::string_view MCPP_VERSION = "2026.9.10.1"; } // namespace mcpp diff --git a/src/build/distribution.cppm b/src/build/distribution.cppm index f100f225..52ca0d69 100644 --- a/src/build/distribution.cppm +++ b/src/build/distribution.cppm @@ -269,6 +269,17 @@ struct MechanismInput { // `linkage = "static"` — the libc axis. On PE it shares the one `-static` // spelling with the C++ runtime axis, so the table has to see it. bool fullStaticLibc = false; + // Does this link line also name a C++ runtime that is NOT the toolchain's? + // + // Today that means a libc++ toolchain whose line carries libstdc++, + // which is what the SYCL and HIP rule packages produce: the device half is + // compiled by a second compiler configured against libstdc++, so the + // artifact links libc++ statically AND loads libstdc++.so at run time. + // + // It changes two things below, and both are corrections of an assumption + // that held only while no such line existed. See `hide_static_cxx_runtime` + // for the symbol half and the libc++ ELF branch for the unwinder half. + bool foreignCxxRuntime = false; // Already-escaped archive paths for the explicit-archive mechanisms. // Empty string = that archive is not available on this toolchain. std::string libcxxArchive; @@ -362,13 +373,26 @@ inline bool is_libcxx(std::string_view id) { return id == "libc++"; } // Keep a statically linked standard library OUT of a shared object's dynamic // symbol table. // -// Only a SHARED LIBRARY needs this, and only when it actually embedded the +// A SHARED LIBRARY always needs this, and only when it actually embedded the // runtime — which after `default_contract` happens on ELF exclusively through -// an explicit `cxx_runtime = { shared = "self-contained" }`. An executable's -// static libstdc++ is already local (ld exports only what a loaded object -// references, and mcpp passes no `-rdynamic`); a .so exports every global it -// defines, which is how a pure-C compat package came to publish 777 GLOBAL -// libstdc++ definitions and become the executable's de-facto C++ runtime. +// an explicit `cxx_runtime = { shared = "self-contained" }`. A .so exports +// every global it defines, which is how a pure-C compat package came to +// publish 777 GLOBAL libstdc++ definitions and become the executable's +// de-facto C++ runtime. +// +// AN EXECUTABLE NEEDS IT WHEN A FOREIGN C++ RUNTIME IS ON THE LINE, and the +// sentence that used to be here — "an executable's static libstdc++ is already +// local (ld exports only what a loaded object references, and mcpp passes no +// `-rdynamic`)" — was a correct premise with a wrong conclusion. The clause in +// the parentheses is the whole mechanism: when a loaded object DOES reference +// them, the linker puts them in `.dynsym`. Measured on a SYCL artifact +// (mcpp#596), which links libc++ statically and loads libstdc++.so: 89 +// exported symbols, 68 of them also defined by libstdc++ or libgcc_s. The +// executable is searched first, so libstdc++'s own code called libc++abi's +// `std::exception::what`, libc++'s `std::runtime_error` constructors ran on +// objects libstdc++ would later destroy, and ten of libgcc_s's eighteen +// unwinder entry points were answered by the executable's libunwind while +// eight were not. // // It does NOT hide the weak/COMDAT template instantiations the library's own // code emits, and must not: unifying those across the process is the intended @@ -379,9 +403,9 @@ inline bool is_libcxx(std::string_view id) { return id == "libc++"; } // Archive BASENAMES — that is what `--exclude-libs` matches, and GNU ld and // lld agree on it. Listed by name rather than `ALL` so a user's own static // library linked into their .so keeps its exports. -std::string hide_static_cxx_runtime(Role role, +std::string hide_static_cxx_runtime(Role role, bool foreignCxxRuntime, std::initializer_list archives) { - if (role != Role::SharedLibrary) return {}; + if (role != Role::SharedLibrary && !foreignCxxRuntime) return {}; std::string out; for (auto archive : archives) { out += " -Wl,--exclude-libs,"; @@ -591,7 +615,7 @@ Mechanism resolve(const MechanismInput& in) { if (m.effective == Contract::SelfContained) { m.unitFlags = " -static-libstdc++"; m.unitFlags += detail::hide_static_cxx_runtime( - in.role, {"libstdc++.a"}); + in.role, in.foreignCxxRuntime, {"libstdc++.a"}); } // ToolchainCoupled and HostCoupled are the same emission on ELF // (no flag); they differ in the rpath the link already carries, @@ -621,11 +645,34 @@ Mechanism resolve(const MechanismInput& in) { m.unitFlags = " -nostdlib++ " + in.libcxxArchive + " " + in.libcxxAbiArchive; m.unitFlags += detail::hide_static_cxx_runtime( - in.role, {"libc++.a", "libc++abi.a"}); - if (!in.libunwindArchive.empty()) { + in.role, in.foreignCxxRuntime, {"libc++.a", "libc++abi.a"}); + if (in.foreignCxxRuntime) { + // ONE UNWINDER PER PROCESS. + // + // libgcc_s is in this process either way: libstdc++.so needs + // it, so naming it here adds no loaded object -- measured, the + // NEEDED set gains the name and nothing else. What it removes + // is the second unwinder. Linking libunwind.a instead pulls in + // only the archive members something references, so ten of + // libgcc's eighteen entry points came from the executable and + // eight stayed in libgcc_s; libstdc++'s personality routine + // then read an LLVM libunwind context through libgcc's + // accessors, found no landing pad, and terminated past a + // handler that should have run (mcpp#596). + // + // `--unwindlib=libgcc` LAST WINS over the payload cfg file's + // `--unwindlib=libunwind`, which is why this is an addition + // rather than an edit of that file: the cfg is the default for + // every link, and only this line has the second runtime on it. + // + // NOT a degradation of the contract. The C++ runtime is still + // embedded; the unwinder was never the artifact's own here, + // because the process already had libstdc++'s. + m.unitFlags += " --unwindlib=libgcc"; + } else if (!in.libunwindArchive.empty()) { m.unitFlags += " " + in.libunwindArchive; m.unitFlags += detail::hide_static_cxx_runtime( - in.role, {"libunwind.a"}); + in.role, in.foreignCxxRuntime, {"libunwind.a"}); } else { m.degraded = true; // effective stays SelfContained: the C++ // runtime IS embedded; the unwinder is not diff --git a/src/build/flags.cppm b/src/build/flags.cppm index e89b54f9..7c2a64c1 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -1028,6 +1028,23 @@ CompileFlags compute_flags(const BuildPlan& plan) { mi.mingw = isMingwTc; mi.macosFloor = !macosDeploymentTarget.empty(); mi.format = format; + // A SECOND C++ RUNTIME ON THIS LINE. + // + // Read off the link flags rather than inferred, because the only way + // one gets there is that something wrote it: a rule package whose + // second compiler is configured against libstdc++ emits + // `-l:libstdc++.so.6` through `mcpp::link_lib`, which lands in + // `bc.ldflags`. `mcpp.rules.sycl` and `mcpp.rules.hip` are the two + // that do today. + // + // The substring is `stdc++`, which covers `-lstdc++`, + // `-l:libstdc++.so.6` and a spelled-out path. It cannot match this + // toolchain's own runtime: the predicate requires libc++, and a + // libstdc++ toolchain never reaches the branch that reads this. + mi.foreignCxxRuntime = + caps.stdlib_id == "libc++" + && std::ranges::any_of(bc.ldflags, [](std::string_view flag) { + return flag.find("stdc++") != std::string_view::npos; }); // Bare metal short-circuits the whole table: the archives found below // are the HOST's, and linking them into a riscv64 image fails with // "incompatible with elf64lriscv". See MechanismInput::freestanding. diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index d5df6b3a..4ff138ce 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -3073,6 +3073,46 @@ std::expected NinjaBackend::build(const BuildPlan& plan explanation)); } } + // THE SURFACE THE ARTIFACT WALK CANNOT REACH. + // + // Everything above follows DT_NEEDED from an artifact. A library a + // dependency published through `runtime.library_dirs` is there because + // something will `dlopen` it, so no link-time edge names it and the + // walk cannot arrive. Reported after the artifacts and never as a + // failure: see `check_dlopen_surface` for why advisory is the right + // severity. + { + auto surface = + mcpp::build::runtime_validation::check_dlopen_surface(plan); + std::vector gaps; + for (auto const& finding : surface.findings) { + if (finding.dangling) continue; // the machine's answer + gaps.push_back(std::format("{} needs {}", + finding.member.filename().string(), + finding.soname)); + } + if (!gaps.empty()) { + std::string detail; + for (auto const& gap : gaps) detail += "\n " + gap; + mcpp::ui::warning(std::format( + "{} of {} librar{} a dependency published for dlopen cannot " + "be loaded on this artifact's search path:{}\n" + // The plural agrees with the DENOMINATOR, which is what the + // sentence is about; agreeing with the numerator produced + // "1 of 16 library". + " These are reached by dlopen, so no link edge names them " + "and the closure check above cannot see them. The program " + "links and runs until something asks for one, and then the " + "back end behind it is simply absent.\n" + " Fix: the package that published the directory has to " + "carry the missing library too, or declare that it does " + "not serve it.\n" + " Record: `runtime.dlopen_surface` in resolution.json " + "({} of {} examined).", + gaps.size(), surface.members, surface.members == 1 ? "y" : "ies", + detail, surface.walked, surface.members)); + } + } if (!runtimeFailure.empty()) { return std::unexpected(BuildError{ "runtime closure validation failed (proven Linux ELF defect)", diff --git a/src/build/plan.cppm b/src/build/plan.cppm index 1fed0992..34433a1f 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -80,6 +80,19 @@ struct CompileUnit { struct LinkUnit { std::string targetName; enum Kind { Binary, StaticLibrary, SharedLibrary, TestBinary } kind = Binary; + // Does this image belong to a DEPENDENCY rather than to the package being + // built? A `kind = "shared"` dependency contributes a link unit to the + // consumer's plan, and that unit is not one of the consumer's targets. + // + // It decides where a `role = "object"` action's outputs go. "Every linked + // image" is the right default for a build program's objects -- a test + // binary and a static library both need them -- but it was reading as + // "every link unit in the plan", so a device island was linked into a + // dependency's shared library as well. Measured: `compat:opencl`'s ICD + // loader, a C library, came out carrying `saxpy_device` and thirty-seven + // `sycl::` instantiations, and the process then had two copies of the + // island. Latent until a SYCL project first had a shared dependency. + bool dependencyOwned = false; // Normally relative to plan.outputDir. A `role = "object"` action's outputs // land here ABSOLUTE, on purpose: ninja identifies a file by the string an // edge declares, and the action edge declares whatever prepare_actions @@ -1676,6 +1689,7 @@ make_plan(const mcpp::manifest::Manifest& manifest, LinkUnit lu; lu.targetName = dep.target.name; lu.kind = LinkUnit::SharedLibrary; + lu.dependencyOwned = true; lu.output = dep.output; lu.importLibrary = import_library_for(dep.target, naming); if (msvcTarget && !lu.importLibrary.empty()) diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 87e22275..69b9b47f 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -10358,10 +10358,21 @@ prepare_build(bool print_fingerprint, // consumes `lu.objects`, so the objects an action produced // belong there for exactly the reason a compiled `.cpp`'s do: // the target's content is what it was told to contain. - const bool image = lu.kind == mcpp::build::LinkUnit::Binary + // + // AND NOT A DEPENDENCY'S IMAGE. "Every linked image" means + // every image THIS PACKAGE produces; a `kind = "shared"` + // dependency contributes a link unit to this plan and is not + // one of them. Without the qualifier the SYCL example's device + // island was linked into `compat:opencl`'s ICD loader as well + // -- a C library carrying `saxpy_device` -- and the process + // held two copies of it. An action that means to reach a + // dependency's target cannot: it is not this package's to + // fill, and naming it explicitly already fails as unknown. + const bool image = !lu.dependencyOwned + && (lu.kind == mcpp::build::LinkUnit::Binary || lu.kind == mcpp::build::LinkUnit::SharedLibrary || lu.kind == mcpp::build::LinkUnit::StaticLibrary - || lu.kind == mcpp::build::LinkUnit::TestBinary; + || lu.kind == mcpp::build::LinkUnit::TestBinary); const bool wanted = a.targets.empty() ? image : std::find(a.targets.begin(), a.targets.end(), diff --git a/src/build/runtime_validation.cppm b/src/build/runtime_validation.cppm index 074b76ed..411e3977 100644 --- a/src/build/runtime_validation.cppm +++ b/src/build/runtime_validation.cppm @@ -158,6 +158,21 @@ std::vector check_symbol_provision(const mcpp::build::BuildPlan& plan, const ArtifactSnapshot& before); +// Walk the libraries this build's dependencies published for `dlopen`. +// +// A SEPARATE ENTRY POINT for the reason the one above is: its object is not an +// artifact. `validate_changed_artifacts` answers a question per image and +// memoises on that image's stat; this one answers a question about +// DIRECTORIES, which no artifact's stat describes, and it holds for a build +// that linked nothing new. +// +// ADVISORY, never blocking. A farm legitimately holds host-driver links that +// dangle on a machine with no driver, and turning a correct CPU-only +// configuration into a failed build would be a worse defect than the one this +// reports. +mcpp::platform::elf::DlopenSurfaceReport +check_dlopen_surface(const mcpp::build::BuildPlan& plan); + } // namespace mcpp::build::runtime_validation namespace mcpp::build::runtime_validation { @@ -300,6 +315,7 @@ std::string cache_key(const mcpp::build::BuildPlan& plan, // verdicts. One authoritative writer, one published view. constexpr std::string_view kLoaderTagsRecord = "loader_tags"; constexpr std::string_view kSymbolProvisionRecord = "symbol_provision"; +constexpr std::string_view kDlopenSurfaceRecord = "dlopen_surface"; // WHAT INVALIDATES A STORED VERDICT BESIDES THE ARTIFACT ITSELF. // @@ -403,6 +419,7 @@ void persist_post_link(const mcpp::build::BuildPlan& plan, if (keyMoved) { doc.erase(std::string(kLoaderTagsRecord)); doc.erase(std::string(kSymbolProvisionRecord)); + doc.erase(std::string(kDlopenSurfaceRecord)); } if (keyMoved || doc.value(std::string(name), nlohmann::json::array()) != entries) { doc["post_link_key"] = std::string(key); @@ -1019,7 +1036,21 @@ check_symbol_provision(const mcpp::build::BuildPlan& plan, .defines = names, }); } - report.conflicts = sp::conflicting_exports(*exported, providers); + // WEAK DEFINITIONS ARE COUNTED, NOT REPORTED. + // + // A template instantiation or an inline function is emitted into every + // image that needs it and the loader keeps one; that is the C++ ABI + // working. Reporting it names a correct build. Measured on the SYCL + // example once the real findings were repaired: thirty-nine shared + // symbols remained and thirty-seven were `sycl::queue` and + // `sycl::buffer` instantiations from the same headers libsycl was + // built from -- and the other two were the island's own `extern "C"` + // entry points, which libsycl does not define at all. + auto all = sp::conflicting_exports(*exported, providers); + for (auto const& conflict : all) + if (conflict.isWeak) ++report.sharedWeak; + std::erase_if(all, [](auto const& c) { return c.isWeak; }); + report.conflicts = std::move(all); report.status = report.conflicts.empty() ? sp::Status::Clean : sp::Status::Conflict; findings.push_back({artifact, std::move(report)}); @@ -1117,4 +1148,108 @@ latest_stored_verdict(const std::filesystem::path& targetRoot) { return summary; } +mcpp::platform::elf::DlopenSurfaceReport +check_dlopen_surface(const mcpp::build::BuildPlan& plan) { + mcpp::platform::elf::DlopenSurfaceReport report; + if constexpr (!mcpp::platform::is_linux) return report; + + // THE SAME APPLICABILITY THE ARTIFACT VERDICT HAS, and for the same + // reason. Under a non-hermetic binding the host loader also consults + // `ld.so.cache`, which mcpp deliberately does not parse, so "not on the + // path mcpp computed" is not evidence of anything. `allow_host_libs` is + // the user's statement that resolution is theirs to arrange. + if (!plan.runtimeBinding.hermetic() || host_libs_allowed(plan)) + return report; + + auto searchDirs = runtime_search_dirs(plan); + + // THE ARTIFACT'S OWN DIRECTORY IS PART OF THE PATH AND IS NOT IN THAT LIST. + // + // Every artifact mcpp links carries `$ORIGIN` first in its DT_RPATH, and a + // dependency built as a shared library is deployed BESIDE it. So a farm + // member needing that library resolves at run time and would be reported + // as a packaging gap here. + // + // Measured: `compat:sycl-runtime` declares `compat:opencl`, whose + // `libOpenCL.so.1` lands in the consumer's `bin/` next to the executable — + // exactly what `libur_adapter_opencl.so.0` needs — and this check named it + // missing until the directory was added. `runtime_search_dirs` cannot + // carry it: `$ORIGIN` is a property of each artifact, not of the plan. + // A PLAN THAT PRODUCES NO PROGRAM HAS NO dlopen SURFACE. + // + // The surface is reached from a PROCESS, and only an executable starts + // one. A plan that produces archives, or a dependency's shared library on + // the way to something else, has no search path of its own to judge + // against -- the question belongs to whatever eventually runs, which has + // its own plan and its own answer. + // + // Measured, and both halves of that sentence were needed. The adapter + // package is `kind = "lib"`, so it produces an archive. And `mcpp test` + // drives the backend twice: the first pass links the dependency's shared + // library and nothing else, and reported `libur_adapter_opencl.so.0 needs + // libOpenCL.so.1` -- the very library it was in the middle of producing. + const bool producesAProgram = std::ranges::any_of( + plan.linkUnits, [](auto const& unit) { + return unit.kind == mcpp::build::LinkUnit::Binary + || unit.kind == mcpp::build::LinkUnit::TestBinary; + }); + if (!producesAProgram) return report; + const auto artifacts = snapshot_link_artifacts(plan); + if (artifacts.empty()) return report; + + for (auto const& [artifact, stamp] : artifacts) { + auto dir = artifact.parent_path(); + if (dir.empty() || std::ranges::find(searchDirs, dir) != searchDirs.end()) + continue; + searchDirs.push_back(dir); + } + + // The SONAMEs this build produces, read from the objects rather than from + // their filenames. See `inspect_dlopen_surface` for why a filename search + // is not enough while the build is still running. + std::vector produced; + for (auto const& [artifact, stamp] : artifacts) { + auto facts = mcpp::platform::elf::inspect_elf_runtime(artifact); + if (facts && !facts->soname.empty()) produced.push_back(facts->soname); + } + + report = mcpp::platform::elf::inspect_dlopen_surface( + plan.depRuntimeLibraryDirs, plan.runtimeBinding, searchDirs, produced); + + // THE RECORD, AND BOTH DENOMINATORS IN IT. + // + // A warning scrolls past; `resolution.json` is the documented place to + // look (docs/05) and is what a test can assert a FIELD of rather than a + // substring of a message. `members` and `walked` are published even when + // there is nothing to report, because "no findings" and "nothing was + // examined" are the two readings this repository has most often confused: + // a surface that failed to build enumerates zero members and every + // per-member test then passes. + nlohmann::json entries = nlohmann::json::array(); + for (auto const& finding : report.findings) { + entries.push_back({ + // The name the loader asks for, and the directory that published + // it. `lexically_relative` rather than `std::filesystem::relative`: + // the latter canonicalises, and every member of such a directory + // is a symlink into a payload -- so the published record named the + // payload's file and lost which farm entry could not be satisfied. + {"library", finding.member.filename().string()}, + {"dir", finding.member.parent_path() + .lexically_relative(plan.outputDir) + .lexically_normal().generic_string()}, + {"soname", finding.soname}, + // `dangling` is the machine's answer (no driver installed); + // anything else is a packaging gap in the publishing package. + {"kind", finding.dangling ? "dangling" : "missing"}, + }); + } + nlohmann::json record = { + {"members", report.members}, + {"walked", report.walked}, + {"findings", std::move(entries)}, + }; + persist_post_link(plan, kDlopenSurfaceRecord, post_link_key(plan), record); + return report; +} + } // namespace mcpp::build::runtime_validation diff --git a/src/build/symbol_provision.cppm b/src/build/symbol_provision.cppm index 624de9e9..c5c6bdf3 100644 --- a/src/build/symbol_provision.cppm +++ b/src/build/symbol_provision.cppm @@ -55,6 +55,11 @@ export namespace mcpp::build::symbol_provision { struct Export { std::string name; bool isFunc = false; + // A vague-linkage definition (STB_WEAK): a template instantiation, an + // inline function, a vtable. Every translation unit that needs one emits + // it and the loader unifies them; that is the C++ ABI working, not an + // image displacing a library's own copy. + bool isWeak = false; }; // One object that could also supply a symbol, as the report will name it. @@ -67,6 +72,7 @@ struct Provider { struct Conflict { std::string name; bool isFunc = false; + bool isWeak = false; std::vector alsoProvidedBy; }; @@ -92,6 +98,14 @@ struct Report { std::size_t exported = 0; std::size_t total = 0; std::vector conflicts; + // Shared vague-linkage definitions, counted and not listed. + // + // They are NOT a finding: the C++ ABI emits a template instantiation into + // every image that needs it and expects the loader to keep one. Counting + // them is still worth doing -- a reader who runs `nm -D` sees them and has + // to be told which ones this check decided about, or "clean" reads as + // "did not look". + std::size_t sharedWeak = 0; // Why, for the two non-answers. Empty for Clean and Conflict. std::string reason; @@ -169,7 +183,8 @@ exported_definitions(const mcpp::platform::elf::DynamicSymbols& symbols) { // share an address with relocated data from being excused. if (!symbol.isFunc && symbols.copyRelocations.contains(symbol.value)) continue; - out.push_back(Export{ .name = symbol.name, .isFunc = symbol.isFunc }); + out.push_back(Export{ .name = symbol.name, .isFunc = symbol.isFunc, + .isWeak = symbol.isWeak }); } std::ranges::sort(out, {}, &Export::name); return out; @@ -179,7 +194,8 @@ std::vector conflicting_exports(std::span exports, std::span closure) { std::vector out; for (auto const& exported : exports) { - Conflict conflict{ .name = exported.name, .isFunc = exported.isFunc }; + Conflict conflict{ .name = exported.name, .isFunc = exported.isFunc, + .isWeak = exported.isWeak }; for (auto const& provider : closure) { if (std::ranges::find(provider.defines, exported.name) != provider.defines.end()) @@ -223,6 +239,13 @@ std::string Report::explain(std::string_view artifact) const { body += " Also provided by:\n"; for (auto const& label : providers) body += std::format(" {}\n", label); + if (sharedWeak > 0) + body += std::format( + " ({} vague-linkage definition{} -- template instantiations, inline\n" + " functions, vtables -- {} also shared and are NOT part of this\n" + " finding: the C++ ABI emits one per image and the loader keeps one.)\n", + sharedWeak, sharedWeak == 1 ? "" : "s", + sharedWeak == 1 ? "is" : "are"); // WHY it matters, then what to do — IN THE ORDER THAT ACTUALLY WORKS. // @@ -242,7 +265,42 @@ std::string Report::explain(std::string_view artifact) const { " The executable is searched first, so the copy inside it wins for\n" " every symbol both provide — the library's own copy is never called,\n" " and code inside that library now runs against a build it was not\n" - " linked against.\n" + " linked against.\n"; + + // THE UNWINDER FAMILY IS NOT ONE MORE DUPLICATE SYMBOL. + // + // For every other name the sentence above is the whole consequence: one + // implementation is called instead of another, and the two are usually + // interchangeable. For `_Unwind_*` the consequence is that a throw is + // processed by TWO unwinders and no `catch` runs. + // + // A static archive contributes only the members something references, so + // the interposition is PARTIAL by construction. Measured on a SYCL + // artifact (mcpp#596): 10 of libgcc_s's 18 entry points came from the + // executable's libunwind and 8 stayed in libgcc_s, including the context + // accessors — so libstdc++'s personality routine read an LLVM libunwind + // `_Unwind_Context` through libgcc's `_Unwind_GetIPInfo`, recovered a + // meaningless IP, found no landing pad, and `__cxa_call_terminate` ran + // past a handler three frames up. The program aborted with exit 134 and + // printed nothing, because `__verbose_terminate_handler` rethrows to name + // the exception's type and that rethrow terminated as well. + // + // Said separately rather than folded into the list below, because none of + // the three ways out addresses it: the fix is one unwinder in the process, + // which is a link-line question rather than a packaging one. + if (std::ranges::any_of(conflicts, [](auto const& c) { + return c.name.starts_with("_Unwind_"); })) { + body += + " These include the unwinder's entry points (`_Unwind_*`), and\n" + " for those the consequence is stronger: an exception is then\n" + " raised by one unwinder and inspected by the other, no `catch`\n" + " matches, and the program calls std::terminate past a handler\n" + " that should have run. A static archive contributes only the\n" + " members something references, so the split is partial and the\n" + " program is correct until something throws.\n"; + } + + body += " Ways out, in the order they apply:\n" " 1. stop one side from providing it — usually the package that\n" " ships a copy of a library the graph already builds;\n" diff --git a/src/runtime/elf.cppm b/src/runtime/elf.cppm index 5c13b2e8..8deaef93 100644 --- a/src/runtime/elf.cppm +++ b/src/runtime/elf.cppm @@ -137,6 +137,13 @@ struct DynamicSymbol { // dynamic symbol table therefore has exactly one cause — the linker // exported it so that some shared object's reference would bind to it. bool isFunc = false; + // STB_WEAK. A vague-linkage definition -- a template instantiation, an + // inline function, a vtable -- which the C++ ABI emits into every + // translation unit that needs it and expects the loader to unify across + // the process. That is the intended behaviour, not a leak, so a caller + // asking "is this image providing something twice" has to be able to tell + // it from a strong definition that displaces a library's own. + bool isWeak = false; std::uint64_t value = 0; // st_value; the key a copy relocation matches }; @@ -193,6 +200,55 @@ RuntimeVerdict validate_runtime_artifact( const RuntimeResolution& resolution, bool hostLibsAllowed = false); +// THE SURFACE NOTHING ELSE WALKS. +// +// `resolve_runtime_closure` above is seeded with the artifact and follows +// DT_NEEDED. A library a package published through `runtime.library_dirs` +// exists precisely because something will `dlopen` it, so no link-time edge +// names it and it is outside that closure BY CONSTRUCTION, not by oversight. +// +// Measured (mcpp#596): a farm of twenty-five libraries, two of which could not +// load at all -- one needing `libnvidia-ml.so.1`, one needing +// `libOpenCL.so.1`, neither on the search path -- while the build reported no +// diagnostic, because nothing had asked. The program then aborted at run time +// with no message. +// +// mcpp is the only component that can answer this: it computed the whole +// search path, and under a hermetic binding nothing else will be consulted. +// The check is provider-agnostic -- it knows nothing of SYCL, CUDA or Unified +// Runtime, and must not, for the reason `test_runtime_contract` states. +struct DlopenSurfaceFinding { + std::filesystem::path member; // the library that cannot be satisfied + std::string soname; // what it needs + // Present as an entry but pointing nowhere. The host-driver sentinels + // create deliberately dangling links so that installing a driver later + // self-heals every consumer, so this is the MACHINE's answer and never an + // error. `Missing` is the packaging gap. + bool dangling = false; +}; + +struct DlopenSurfaceReport { + // THE DENOMINATORS. "No findings" and "nothing was examined" are the two + // readings that must not be spelled the same: a directory that failed to + // build enumerates nothing, and every per-member test then passes. + std::size_t members = 0; // versioned sonames found in the directories + std::size_t walked = 0; // whose own DT_NEEDED was read + std::vector findings; +}; + +// +// `alsoProvided` names SONAMEs this build produces itself. An artifact carries +// its SONAME in the object rather than in its filename -- mcpp links +// `bin/libopencl.so` whose SONAME is `libOpenCL.so.1`, and the alias under the +// SONAME is created by a later step -- so a filename search reports the +// library as missing while it is being produced. Measured: `mcpp test` calls +// this twice and only the second call saw the alias. +DlopenSurfaceReport inspect_dlopen_surface( + std::span surfaceDirs, + const mcpp::platform::runtime::RuntimeBinding& binding, + std::span searchDirs, + std::span alsoProvided = {}); + } // namespace mcpp::platform::elf namespace mcpp::platform::elf { @@ -225,6 +281,7 @@ constexpr std::uint64_t kDtGnuHash = 0x6ffffef5; constexpr std::uint64_t kSymEntrySize = 24; constexpr std::uint64_t kRelaEntrySize = 24; +constexpr unsigned char kStbWeak = 2; constexpr unsigned char kSttObject = 1; constexpr unsigned char kSttFunc = 2; constexpr unsigned char kSttGnuIfunc = 10; @@ -682,6 +739,7 @@ inspect_dynamic_symbols(const std::filesystem::path& object) { out.defined.push_back(DynamicSymbol{ .name = std::move(*name), .isFunc = (type == detail::kSttFunc || type == detail::kSttGnuIfunc), + .isWeak = (bind == detail::kStbWeak), .value = *value, }); } @@ -1258,4 +1316,88 @@ RuntimeVerdict validate_runtime_artifact( return verdict; } +DlopenSurfaceReport inspect_dlopen_surface( + std::span surfaceDirs, + const mcpp::platform::runtime::RuntimeBinding& binding, + std::span searchDirs, + std::span alsoProvided) { + DlopenSurfaceReport report; + + // A versioned SONAME is what a farm links and what dlopen asks for; an + // unversioned `libfoo.so` in such a directory is a LINK-time name and is + // not part of this surface. `-gdb.py` sidecars sit beside the payload's + // libraries and match a looser test. + auto versioned = [](const std::string& name) { + auto so = name.find(".so."); + if (so == std::string::npos || name.size() < so + 5) return false; + if (name.ends_with(".py")) return false; + return name[so + 4] >= '0' && name[so + 4] <= '9'; + }; + + // Every entry in every surface directory, indexed by SONAME, so that a + // member needing another member is answered from the surface itself. + std::map entries; + std::error_code ec; + for (auto const& dir : surfaceDirs) { + if (dir.empty() || !std::filesystem::is_directory(dir, ec)) continue; + for (auto const& item : std::filesystem::directory_iterator(dir, ec)) { + auto name = item.path().filename().string(); + if (!versioned(name)) continue; + entries.emplace(name, item.path()); + } + } + + // ONE LIBRARY, NOT ITS TWO NAMES. A farm carries `libfoo.so.N` and + // `libfoo.so.N.M.P` as two links to one file, so walking the entries + // reports every finding twice and calls thirteen libraries twenty-five. + // Keyed by the file both resolve to; the SHORTEST name is kept, which is + // the one a dlopen asks for. + std::map> + libraries; + for (auto const& [name, path] : entries) { + auto real = std::filesystem::weakly_canonical(path, ec); + if (ec) { real = path; ec.clear(); } + auto it = libraries.find(real); + if (it == libraries.end()) libraries.emplace(real, std::pair{name, path}); + else if (name.size() < it->second.first.size()) + it->second = {name, path}; + } + report.members = libraries.size(); + + for (auto const& [name, path] : std::views::values(libraries)) { + // A dangling entry is the sentinel's self-heal shape: the package did + // its part and the target is the machine's answer. Nothing to read. + if (!std::filesystem::exists(path, ec)) continue; + auto facts = inspect_elf_runtime(path); + if (!facts) continue; // not an ELF this reader understands; a + // statement about the CHECK, not the surface + ++report.walked; + for (auto const& soname : facts->needed) { + // Resolved the way the loader will resolve it, from this member's + // own tags plus the search path the artifact actually carries. + if (detail::resolve_needed(soname, *facts, binding, searchDirs, {})) + continue; + // A SONAME this build produces under another filename. + if (std::ranges::find(alsoProvided, soname) != alsoProvided.end()) + continue; + // DANGLING IS A PROPERTY OF THE FILE, NOT OF THE NAME. An entry + // that exists and still did not resolve is a third thing -- an + // unreadable file, a directory the search does not reach -- and + // calling it "the machine has no driver" would silence it. + auto entry = entries.find(soname); + const bool dangling = entry != entries.end() + && !std::filesystem::exists(entry->second, ec); + report.findings.push_back({ + .member = path, + .soname = soname, + .dangling = dangling, + }); + } + } + std::ranges::sort(report.findings, [](auto const& a, auto const& b) { + return std::tie(a.member, a.soname) < std::tie(b.member, b.soname); + }); + return report; +} + } // namespace mcpp::platform::elf diff --git a/tests/unit/test_distribution.cpp b/tests/unit/test_distribution.cpp index 12d4a5c2..83ede0dc 100644 --- a/tests/unit/test_distribution.cpp +++ b/tests/unit/test_distribution.cpp @@ -670,3 +670,72 @@ TEST(Distribution, FormatUsesTheFallbackOnlyWhenTheTripleSaysNothing) { EXPECT_EQ(dist::format_for("", dist::Format::Pe), dist::Format::Pe); EXPECT_EQ(dist::format_for("nonsense", dist::Format::Elf), dist::Format::Elf); } + +// --------------------------------------------------------------------------- +// A SECOND C++ RUNTIME ON THE LINE (mcpp#596). +// +// A lane whose device compiler is configured against libstdc++ puts libstdc++ +// on the link line while the artifact links libc++ statically. Two things then +// have to change, and neither is a preference: +// +// * one unwinder. Linking libunwind.a pulls in only the archive members +// something referenced, so the interposition is partial by construction -- +// measured, ten of libgcc's eighteen entry points came from the artifact +// and eight stayed in libgcc_s, including the accessors libstdc++'s +// personality routine calls. It read an LLVM libunwind context through +// libgcc's accessors and terminated past a matching handler. +// * the static archives' symbols hidden. The linker exports what a loaded +// object references, and a loaded libstdc++ references them: measured, 89 +// exported symbols of which 68 were also defined by libstdc++ or libgcc_s. +// +// The byte-level assertions are the point, as they are for every other cell in +// this table: a build with no second runtime on its line must be unchanged. + +TEST(Distribution, LinuxLibcxxWithAForeignRuntimeTakesOneUnwinder) { + dist::MechanismInput in; + in.format = dist::Format::Elf; + in.stdlibId = "libc++"; + in.requested = dist::Contract::SelfContained; + in.libcxxArchive = "/tc/libc++.a"; + in.libcxxAbiArchive = "/tc/libc++abi.a"; + in.libunwindArchive = "/tc/libunwind.a"; + + // Unchanged when nothing else is on the line. Byte-for-byte the string the + // cell above asserts. + auto alone = dist::resolve(in); + EXPECT_EQ(alone.unitFlags, + " -nostdlib++ /tc/libc++.a /tc/libc++abi.a /tc/libunwind.a"); + + in.foreignCxxRuntime = true; + auto shared = dist::resolve(in); + // The contract is NOT degraded: the C++ runtime is still embedded, and + // libgcc_s is in the process either way because libstdc++ needs it. + EXPECT_EQ(shared.effective, dist::Contract::SelfContained); + EXPECT_FALSE(shared.degraded); + EXPECT_TRUE(shared.diagnostic.empty()); + EXPECT_EQ(shared.unitFlags, + " -nostdlib++ /tc/libc++.a /tc/libc++abi.a" + " -Wl,--exclude-libs,libc++.a -Wl,--exclude-libs,libc++abi.a" + " --unwindlib=libgcc"); + // The payload's unwinder archive is NOT on the line: linking it is what + // creates the second unwinder. + EXPECT_EQ(shared.unitFlags.find("libunwind.a"), std::string::npos); +} + +// A shared library already hid these archives, and that path is untouched: the +// widened guard adds executables, it does not change what a .so emits. +TEST(Distribution, ASharedLibraryStillHidesTheArchivesWithoutASecondRuntime) { + dist::MechanismInput in; + in.format = dist::Format::Elf; + in.stdlibId = "libc++"; + in.role = dist::Role::SharedLibrary; + in.requested = dist::Contract::SelfContained; + in.libcxxArchive = "/tc/libc++.a"; + in.libcxxAbiArchive = "/tc/libc++abi.a"; + in.libunwindArchive = "/tc/libunwind.a"; + auto m = dist::resolve(in); + EXPECT_EQ(m.unitFlags, + " -nostdlib++ /tc/libc++.a /tc/libc++abi.a" + " -Wl,--exclude-libs,libc++.a -Wl,--exclude-libs,libc++abi.a" + " /tc/libunwind.a -Wl,--exclude-libs,libunwind.a"); +} diff --git a/tests/unit/test_elf_runtime.cpp b/tests/unit/test_elf_runtime.cpp index f3e3e328..6bbfe8ad 100644 --- a/tests/unit/test_elf_runtime.cpp +++ b/tests/unit/test_elf_runtime.cpp @@ -694,3 +694,99 @@ TEST(ElfRuntime, ThisBinaryExportsNothingOfItsOwn) { return s; }(); } + +// THE SURFACE A DEPENDENCY PUBLISHES FOR dlopen, WHICH NO ARTIFACT WALK REACHES. +// +// `resolve_runtime_closure` is seeded with the artifact and follows DT_NEEDED. +// A library in a package's `runtime.library_dirs` is there because something +// will dlopen it, so nothing names it and it is outside that closure by +// construction. mcpp#596 is one such library needing a soname no directory on +// the search path carried, on a build that reported no diagnostic at all. +// +// Three states, and the difference between the second and the third is the +// whole reason this is not "dlopen everything and fail on error": a surface +// legitimately holds host-driver links that dangle on a machine with no +// driver, and a check that failed there would turn a correct CPU-only +// configuration into a failed build. +TEST(DlopenSurface, SeparatesAPackagingGapFromAMachineWithoutTheDriver) { + if constexpr (!mcpp::platform::is_linux) + GTEST_SKIP() << "ELF/glibc runtime physics only apply on Linux"; + Tmp t; + auto payload = t.path / "store"; + auto glibc = payload / "2.44" / "lib64"; + auto farm = t.path / "farm"; + std::filesystem::create_directories(glibc); + std::filesystem::create_directories(farm); + + write_elf_fixture(glibc / "libc.so.6", {.needed = {}, .runpath = glibc.string()}); + + // Resolved: needs only what the surface itself carries. + write_elf_fixture(farm / "libgood.so.1", + {.needed = {"libc.so.6"}, .runpath = glibc.string()}); + // The packaging gap: a soname no directory on the path carries. + write_elf_fixture(farm / "libgap.so.1", + {.needed = {"libabsent.so.7"}, .runpath = glibc.string()}); + // The machine's answer: the surface HAS an entry, and it points nowhere. + write_elf_fixture(farm / "libneedsdriver.so.1", + {.needed = {"libdriver.so.1"}, .runpath = glibc.string()}); + std::filesystem::create_symlink("/nonexistent/driver/libdriver.so.1", + farm / "libdriver.so.1"); + + std::vector dirs{farm}; + std::vector search{farm, glibc}; + auto report = elf::inspect_dlopen_surface(dirs, binding_for(payload), search); + + // BOTH DENOMINATORS. A surface that failed to build enumerates nothing and + // every per-member assertion below then passes. + EXPECT_EQ(report.members, 4u); + EXPECT_EQ(report.walked, 3u) << "the dangling entry is not read, and the " + "other three are"; + + ASSERT_EQ(report.findings.size(), 2u); + // Asserted as FIELDS, not as a substring of a message whose wording is + // free to improve. + const elf::DlopenSurfaceFinding* gap = nullptr; + const elf::DlopenSurfaceFinding* machine = nullptr; + for (auto const& f : report.findings) { + if (f.soname == "libabsent.so.7") gap = &f; + if (f.soname == "libdriver.so.1") machine = &f; + } + ASSERT_NE(gap, nullptr); + ASSERT_NE(machine, nullptr); + EXPECT_FALSE(gap->dangling) << "nothing on the surface answers this name"; + EXPECT_EQ(gap->member.filename(), "libgap.so.1"); + EXPECT_TRUE(machine->dangling) + << "the surface carries the entry; the target is the machine's answer"; +} + +// ONE LIBRARY, NOT ITS TWO NAMES. +// +// A farm links `libfoo.so.N` and `libfoo.so.N.M.P` to one file. Walking the +// entries reports every finding twice and calls thirteen libraries twenty-six, +// which is what the first published record did. +TEST(DlopenSurface, CountsALibraryOnceWhenTwoNamesLinkToIt) { + if constexpr (!mcpp::platform::is_linux) + GTEST_SKIP() << "ELF/glibc runtime physics only apply on Linux"; + Tmp t; + auto payload = t.path / "store"; + auto glibc = payload / "2.44" / "lib64"; + auto farm = t.path / "farm"; + std::filesystem::create_directories(glibc); + std::filesystem::create_directories(farm); + write_elf_fixture(glibc / "libc.so.6", {.needed = {}, .runpath = glibc.string()}); + + auto real = t.path / "libtwo.so.1.2.3"; + write_elf_fixture(real, {.needed = {"libabsent.so.7"}, .runpath = glibc.string()}); + std::filesystem::create_symlink(real, farm / "libtwo.so.1.2.3"); + std::filesystem::create_symlink(real, farm / "libtwo.so.1"); + + std::vector dirs{farm}; + std::vector search{farm, glibc}; + auto report = elf::inspect_dlopen_surface(dirs, binding_for(payload), search); + + EXPECT_EQ(report.members, 1u); + EXPECT_EQ(report.walked, 1u); + ASSERT_EQ(report.findings.size(), 1u); + // The SHORTEST name is kept: it is the one a dlopen asks for. + EXPECT_EQ(report.findings.front().member.filename(), "libtwo.so.1"); +} diff --git a/tests/unit/test_symbol_provision.cpp b/tests/unit/test_symbol_provision.cpp index b65028f4..eefd60bd 100644 --- a/tests/unit/test_symbol_provision.cpp +++ b/tests/unit/test_symbol_provision.cpp @@ -137,8 +137,14 @@ TEST(SymbolProvision, TheReportNamesEveryProviderAndCapsTheSymbolList) { report.status = sp::Status::Conflict; report.total = 217; for (int i = 0; i < 20; ++i) + // DESIGNATED, not positional. A field added to `Conflict` between + // `isFunc` and `alsoProvidedBy` bound the provider list to a bool + // here -- a string literal converts to one, so it compiled, and the + // provider list silently became empty. report.conflicts.push_back(sp::Conflict{ - std::format("sym{}", i), true, {"/pkg/lib/libz.so.1"}}); + .name = std::format("sym{}", i), + .isFunc = true, + .alsoProvidedBy = {"/pkg/lib/libz.so.1"}}); report.exported = report.conflicts.size(); auto text = report.explain("consumer"); @@ -199,3 +205,48 @@ TEST(SymbolProvision, OrdinaryLinkFlagsDoNotVoidThePredicate) { "-O2", "-Wl,-rpath,$ORIGIN", "-lz", "-Wl,--as-needed", "-Wl,--enable-new-dtags", "-static-libstdc++", "-shared"})); } + +// ── vague linkage is not a second provider ───────────────────────────────── +// +// A template instantiation, an inline function or a vtable is emitted into +// every image that needs it and the loader keeps one. That is the C++ ABI +// working, and reporting it names a correct build. +// +// Measured on the SYCL example once the real findings were repaired: of the +// thirty-nine symbols the image still shared with `libsycl.so.9`, +// thirty-seven were `sycl::queue` and `sycl::buffer` instantiations from the +// same headers libsycl was built from -- and the remaining two were the +// island's own `extern "C"` entry points, which libsycl does not define. A +// check that could not tell binding from name reported all of them. + +TEST(SymbolProvision, AWeakDefinitionIsCarriedThroughAsWeak) { + auto s = image(); + auto weak = func("_ZN4sycl3_V15queueD2Ev"); + weak.isWeak = true; + s.defined.push_back(weak); + s.defined.push_back(func("saxpy_device")); + auto exports = sp::exported_definitions(s); + ASSERT_TRUE(exports.has_value()); + ASSERT_EQ(exports->size(), 2u); + // Sorted by name: the mangled one first. + EXPECT_TRUE((*exports)[0].isWeak); + EXPECT_FALSE((*exports)[1].isWeak); +} + +TEST(SymbolProvision, AConflictRemembersWhetherItsDefinitionIsWeak) { + std::vector exports{ + { .name = "_ZN4sycl3_V15queueD2Ev", .isFunc = true, .isWeak = true }, + { .name = "inflate", .isFunc = true, .isWeak = false }, + }; + std::vector closure{ + { .label = "libsycl.so.9", + .defines = {"_ZN4sycl3_V15queueD2Ev", "inflate"} }, + }; + auto conflicts = sp::conflicting_exports(exports, closure); + ASSERT_EQ(conflicts.size(), 2u); + // Both are shared; only the binding separates them, and the caller is what + // decides which one is a finding. Asserted here rather than in the caller + // so the DATA carries the distinction even if a future caller forgets it. + EXPECT_TRUE(conflicts[0].isWeak); + EXPECT_FALSE(conflicts[1].isWeak); +}