Stop shipping NaCl IRT and build it alongside gamelogic instead?

[slipher]

So the irt_core nexe is untrusted code in the Native Client sandbox implementing mostly the equivalent of libc syscall wrappers for the NaCl VM, plus some other libc stuff like pthreads and malloc. I believe it is treated the same as any other untrusted code by the sandbox. (There are some examples in the NaCl repo's tests/barebones/ showing how VM syscalls can be done without the library.) So we might consider ditching the IRT nexe shipped with the engine, and instead build it into each gamelogic binary. This would let us reach the goal of building a Daemon engine without binary blobs much more simply than if we had to build irt_core. To build everything we need from native_client the normal way you need to set up 3 toolchains:

• Target platform native code toolchain for sel_ldr
• NaCl toolchain(s) to build irt_core_raw
• Host-mode toolchain to build the tls_edit tool which transforms irt_core_raw into irt_core (only actually needed for 32-bit targets)

If we build the IRT files as normal gamelogic files, the latter two toolchains would no longer needed to produce a built-from-scratch Daemon release. I'm assuming that tls_edit is only needed for the case where the code is loaded like a shared library, and not if it's built into the main binary.

If my assumptions are correct the steps to do this would be:

• Add Native Client as a submodule
• Add the IRT source files there into any gamelogic build
• Remove the -B <irt_core path> flag from the nacl_loader invocation

Obviously the reason to have the IRT in the first place is to save compile time and disk space for those distributing nexe binaries. The irt_core-amd64.nexe we currently ship is 623K. But Bloaty says 46% of that is debug info. Strip that as our build process normally does, perhaps remove some unused parts while linking, and it's maybe a couple hundred KB. Plus Breakpad debug info for the IRT would be generated in the normal build process.

Building an irt_core_raw with Saigo single-threadedly took 0.94 s for me excluding build system overhead. (I got Total command execution time: 0.938376 seconds using the Scons --debug=time option.)

[slipher]

Well the TLS stuff might be a bit complicated, reading NaCl's src/untrusted/irt/irt_private_tls.c. But a naive first attempt could be simply excluding that file from the build.

[illwieckz]

The IRT topic is a subset of:

• Building the NativeClient loader as part of the engine build? #1879

So even if we don't build the native loader, we may want to build the IRT, with native_client being a submodule.

Now the TLS editor…

Host-mode toolchain to build the tls_edit tool which transforms irt_core_raw into irt_core (only actually needed for 32-bit targets)

I was thinking about it some days ago and I thought we can simply ship tls_edit with the Saigo SDK.

My Saigo SDK script already clones the native_client repository to install some special NaCl headers for sandboxed binaries:

• nacl/nacl_exception.h
• nacl/nacl_minidump.h
• nacl/simd.h

The two first ones we copied (in old versions) in the engine repo because NaCl compilers did not ship them, third one we don't use.

So we can also build tls_edit as part of the toolchain, which it is.

[illwieckz]

Note that I started writing CMake code (not even committed) for building the IRT, and I wanted to start doing the same for the TLS editor (I have not started, but that's almost one line if I'm right).

Both would allow integration in Dæmon cmake scripts and in Saigo SDK cmake scripts.

[illwieckz]

So basically I see things that way:

• sel_ldr: fetched/built as part of naclruntime in engine external deps
• irt: built (similar to cgame and sgame) as part of the engine files, or baked in cgame/sgame.
• tls_edit: fetched/built as part of saigosdk in engine external deps

[illwieckz]

Here is some CMake code to build tls_edit, it would be callable by the saigo-nacl-sdk CMake script for inclusion in the Saigo SDK:

• CMake plumbing to build tls_edit native_client#43

[illwieckz]

Obviously the reason to have the IRT in the first place is to save compile time and disk space for those distributing nexe binaries.

I would prefer to keep the cgame/sgame nexe smaller, but we can still build the IRT as part of the engine to be used like we currently do.

[illwieckz]

Here is my work-in-progress illwieckz/nacl-tls-edit branch on the Saigo SDK repository, it makes possible to build and ship tls_edit as part of the toolchain:

• https://github.com/DaemonEngine/saigo-nacl-sdk/tree/illwieckz/nacl-tls-edit

Since the Saigo toolchain is already needed to compile the IRT, it means the loader build doesn't need something else than the target toolchain (for sel_ldr) and the nacl toolchain (for the IRT), removing the need for a third toolchain (the host toolchain for tls_edit).

This also means that the Dæmon engine would be able to build the IRT like any other nexe.

[slipher]

Here is my work-in-progress illwieckz/nacl-tls-edit branch on the Saigo SDK repository, it makes possible to build and ship tls_edit as part of the toolchain:

Sounds good. That's a reasonable way to untangle the toolchain dependency knot. First you build Saigo + tls_edit, then both the loader and the IRT are unlocked at the same time.

This also means that the Dæmon engine would be able to build the IRT like any other nexe.

Making people who compile Daemon also compile the IRT doesn't seem like a good idea, since firstly it adds a dependency on the NaCl toolchain for building the engine; and more importantly, as we will need to provide a pre-built loader anyway, and the loader and IRT are tightly coupled, it would make more sense to ship them together. I know you have suggested having the end user also build the loader before, but after working on Native Client more it has become abundantly clear that this is a bad idea, because the loader build is extraordinarily sensitive to details of the toolchain. Examples: that the linker script breaks when the compiler adds a new section; that ld.bfd is the only linker you could use at all due to said script, and that with MinGW you will most likely have to use a toolchain with a specific thread-local storage flavor. This would torpedo the portability of Daemon as software that can be built by end users, since with so many configurations it would fail to build, or even build but with its security properties broken.

What would make sense to me is having sel_ldr (+bootstrap) and irt_core built as external_deps targets. (Not sure if it's better as one combined target or two separate. On one hand they are closely coupled and come from the same repository; on the other it would be two separate CMake invocations if we go with CMake, using separate toolchains.)

[illwieckz]

What would make sense to me is having sel_ldr (+bootstrap) and irt_core built as external_deps targets. (Not sure if it's better as one combined target or two separate. On one hand they are closely coupled and come from the same repository; on the other it would be two separate CMake invocations if we go with CMake, using separate toolchains.)

One thing is that if we build the IRT in the external_deps, then we would need a native-saigosdk pseudo-package to build the IRT (the same way we have native-wasm and native-pkgconfig as host tools for building macOS dependencies), plus the saigosdk package to be shipped to end users.

I may add an option to make possible to not download the Saigo SDK but to build it as part of the external_deps. This option would be disabled by default (because building Saigo is very heavy on resources), but that would make possible for someone who wants it to actually rebuilds oneself the complete trusted stack entirely. If we do that, then such user would likely build Saigo twice, the host one, and the target one, even if they are the same. Avoiding that would require to implement (non-) cross-platform detection and use the target PATH as host PATH. We made everything in external_deps to be cross-compilation first with clean PATH separation…

To me the best alternative to building the IRT in engine is to download a prebuilt IRT in external_deps, because building the IRT in external_deps brings the problem of “we need to build/download two saigo toolchains just to build a single external deps archive”.

[slipher]

I was assuming that Saigo would be a different download than the rest of external_deps. If we have any other host-mode deps we could expand it to a broader host-dependent deps archive, in addition to our current target-dependent ones. In fact we're kind of hitting that case already with the tls_edit proposal, since that's not really part of Saigo.

[illwieckz]

I do remember that in the past we discussed the opportunity to have two deps packages, one for the toolchain, one for the libs. For example it would have been a solution for shipping an single amd64 PNaCl on both macos-amd64 and macos-arm64 while shipping two packages of libraries for them. Though that need is going to disappear. One thing that would have made the split uncomfortable was that the IRT (can be seen as a lib) was shipped from PNaCl SDK (the toolchain). It was solvable, but annoying.

In fact we're kind of hitting that case already with the tls_edit proposal, since that's not really part of Saigo.

If we need tls_edit to build a nexe (and the IRT is one) then tls_edit should be part of the “Saigo SDK”. This is not part of Clang, but Saigo is Clang + GNU binutils + Newlib and things like that, the TLS editor is best placed among those tools.

The TLS editor is in a similar place as some headers we can find in the native_client repository: they are required to build some sandboxed code, so we better ship all of them with the SDK, whatever their repository origin.

[slipher]

While we're at it throwing in more host tools, I've thought about suggesting Breakpad dump_syms + minidump_stackwalk too. This could be interesting for adding crash_test.py to CI or integrating symbol generation in the gamelogic build. dump_syms is annoyingly non-universal though: what you can symbolize depends on the host platform. For example on Windows (maybe Mac too) it doesn't know about ELF, so you can't symbolize nexe.

[illwieckz]

While we're at it throwing in more host tools, I've thought about suggesting Breakpad dump_syms + minidump_stackwalk too.

That would be a good idea to have breakpad in external_deps. If we want to build the IRT inexternal_deps then we need an host breakpad in the host toolchain and then having it in the host Saigo would be the same. If we want to build the IRT as part of the engine build, shipping a target breakpad in external_deps or with the target Saigo is the same (since target Saigo would be part of external_deps.

For example on Windows (maybe Mac too) it doesn't know about ELF, so you can't symbolize nexe.

lolwat??? I wonder if we can add some ifdef to build on/for Windows the ELF breakpad them, and name it nacl-breakpad…

Edit: Breakpad using configure is painful…