procd: add support for OCI runtime spec 1.3.0 - #37
Open
dangowrt wants to merge 56 commits into
Open
Conversation
Member
Author
|
@dhewg would appreciate your review! |
Contributor
|
Nice, will take a look! |
|
@dangowrt While this PR does indeed fix part of my issue in #35, the part I included in the edit does this happen. When actually trying to run a OCI container UXC will throw The directory in the uxc json file is present and filled with files: The command is just |
Member
Author
@dangowrt Your fix does indeed resolve the EPERM issue however it still does not work. Instead of error 1 EPERM I now get error 22 Invalid Argument. UXC stills ays
and this in the listing:
config file/command is the same. Note that this also happens when running uxc start manually, so it's not an invalid argument to the uxc command itself (which I originally thought) The issue for this happening seems to be seccomp. Since when disabling seccomp the error disappears when using UXC directly. Now it can run the container. However when trying to attach via Another point is, it really only works when using uxc directly. When it is execut by podman I get the following error still: Also note that despite what's being said, the container is NOT removed from uxc. Edit: here's the full podman log:Edit 2: After recompiling OpenWRT with seccomp support and rerunning the error still persists but I get a new error message:
However inspecting it shows that it is indeed present just with a slightly different name
It looks like a) podman is giving out bas paths that UXC can't find or b) UXC just uses a bad name. Could this be a cgroup v1/v2 thing? According to strings and some googling ujail searches for memory.swap_max which is cgroup v1 only yet the cgroup version that's running is v2 which uses memory.swap.max instead. |
dangowrt
force-pushed
the
oci-1.3.0
branch
2 times, most recently
from
May 26, 2026 13:01
ed1ba35 to
14c4b4a
Compare
Member
Author
|
@Juff-Ma I was wrongly assuming that podman, runc and crun would implement the CLI as defined in the spec, but turns out that part of the spec is abandonned and it become more of just a convention. Now regarding the seccomp problem you were seeing: I've reworked the seccomp support for now also work with "foreign" libc containers (ie. OpenWrt host being musl, container being eg. glibc or bionic). This needs a small change to the procd package Makefile as well, see below diff --git a/package/system/procd/Makefile b/package/system/procd/Makefile
index cf730a3c0a..c508a447c6 100644
--- a/package/system/procd/Makefile
+++ b/package/system/procd/Makefile
@@ -30,12 +30,14 @@ PKG_CONFIG_DEPENDS:= \
include $(INCLUDE_DIR)/package.mk
include $(INCLUDE_DIR)/cmake.mk
+include $(INCLUDE_DIR)/kernel.mk
ifeq ($(DUMP),)
STAMP_CONFIGURED:=$(strip $(STAMP_CONFIGURED))_$(shell echo $(CONFIG_TARGET_INIT_PATH) | $(MKHASH) md5)
endif
-CMAKE_OPTIONS += -DEARLY_PATH="$(TARGET_INIT_PATH)"
+CMAKE_OPTIONS += -DEARLY_PATH="$(TARGET_INIT_PATH)" \
+ -DNOLIBC_INCLUDE_DIR="$(LINUX_DIR)/tools/include/nolibc"
define Package/procd/Default
SECTION:=base |
|
@dangowrt I've tested your build and it has the same problem I reported in Edit 2 of my previous message. The seccomp error (while I appreciate it being fixed) was easily solvable by just compiling OpenWRT with seccomp support. Through a bit of grepping and googling I eventually made this simple one line patch but still got an error. I could try combining my change with your patchset and see if that fixes anything. |
I can not report success. The error about the missing cgroup file is gone but (once again) I still receive a |
|
@dangowrt I have dug deeper and created a script that logs the arguments passed to uxc. Finding out that the commands to UXC absolute are fine and DO WORK when manually running them there's dafinitely something weird at play. So far I know the following: There is a race condition between UXC and Podman. Particularly in creating the pidfile. When the UXC command exits the pidfile is not yet created for some reason and Podman WILL fail early. Introducing 1s delay into the UXC execution via my logging script fixes this issue. (Podman reports the correct PID) With that in place Podman now tries to actually start the container only for that to immediately fail with a new error:
Which is... weird? If it is Podmans job to create this file why isn't it doing so. And if it is UXCs job to do so why isn't podman providing ANY command line param. Now, this could be related to this debian issue which claims this can be the result of a conmon/podman version mismatch. This certainly could be the case. Podmans dependencies in the repos are not as thoroughly updated as podman itself. Conmon is 2 years out of date for example. But still, I continued. I ran the command with -d and... It worked? Kinda at least. The container started (and showed as running) but podman immediately regards it as crashed and kills the container (note podman regards it as having exited with -1, not UXC, when it receives the kill command the container is still running). I don't know why podman kills it. Podman seems to execute a cleanup command. but I don't know why. Trying to get logs fails with a |
|
I'm making one small step after the other. Upgrading all podman dependencies helped a bit. I needed to patch a build error in conmon but that was easily fixable. (I'm still using the 1s delay for UXC to work around the race condition) Using the latest tools I now am able to start a container and it actually runs. Well the container does. The app inside not so much. When trying to get logs I get the error Note that /docker-entrypoint.sh is not static. It is the entrypoint specified by the Dockerfile and therefore changes with the container used. Note that I am not able to recover from this error. My entire shell session freezes and I can't use |
Member
Author
|
There is an architectural problem with how seccomp is applied which break foreign-libc containers (ie. OpenWrt with musl and container eg. with glibc). I've resolved that and tons of other issues, currently last mile of testing, going to push to this branch in the next hours with validated podman working. Meanwhile, please open a PR to update podman in case you didn't do that already and there isn't any existing PR for that (I've opened the PR to update conmon earlier this morning) |
I see. Glad to hear that. I haven't actually updated Podman itself. It is up to date. In addition to conmon I've update the netavark/aardvark-dns stack, catatonit and crun (since it was also out of date and I was at it) I'll look into opening PRs P.S.: Your PR does not include a second patch conmon required in my build env at least. I got a GCC false positive on |
Member
Author
|
@Juff-Ma Please retry with the changes I've pushed now. Don't forget to also apply the patch for procd's Makefile I've posted in #37 (comment) |
@dangowrt IT IS ALIVE. Oh my god this is amazing. I still included my cgroups patch just to be sure but it absolutely works now. I can run containers and their services are reachable. The work you've done to get this working is incredible. I found 2 little issues (that do not impact day to day operations, at least for me, but I still wanna name them).
P.S.: The error really does occur when killing or deleting containers. I don't know why though. Killing never works and rm-ing will error out the first time. Still not an issue for me. |
Member
Author
|
@Juff-Ma console issues with |
If you mean the commits up to 280a8bb I wasn't really able to test them (I already tried when they were pushed) since I've been encountering a new issue. I though that I had left a comment but it seems it didn't go through (why github?) On some containers I get a Here are the two containers I used:
|
Contributor
|
Some testing feedback: create+start+attach doesn't have that issue, but now escape sequences get printed (prompt displays as Sometimes create hangs too: attach on a created but not yet started container is just an echo chamber. In an attempt to move a broken board's debian install to uxc (so a full distro with systemd), I whipped up the attached patch and used the attached config. That worked prior to this PR, including 0001-jail-add-config-to-not-lock-securebits.patch |
Contributor
|
I see some of the above mentioned issues only on my openwrt one, and not on riscv |
Contributor
|
On riscv64, adding a memory limit to config.json: yields a crash: Happens reproducible with the limit, doesn't happen without it. |
dangowrt
force-pushed
the
oci-1.3.0
branch
4 times, most recently
from
August 21, 2026 00:43
90d7d07 to
6212139
Compare
put_namespace() stats /proc/self/ns/<name> and only advertises namespaces the running kernel actually provides, so guarding the "time" entry with #ifdef CLONE_NEWTIME cannot enable anything the runtime check would not. The guard tests the toolchain headers at build time, and the guarded code does not even use CLONE_NEWTIME, so its only possible effect is to hide time namespace support from the features reply when procd was built against headers predating the flag. Drop the guard together with the <sched.h> include which was added solely to provide it. Fixes: 47a9f0d ("service: add method to query available container features") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
_add_mount() keyed mounts by target alone and silently skipped any second registration for an already-known target. With OCI bundles a target can legitimately be registered twice from independent sources (for instance an implicit mount and an explicit one), and the two may carry conflicting parameters that the old code discarded without notice. Compare the full descriptor instead: an exact duplicate (same source, filesystemtype, optstr, flags, error and inner flags) is accepted as a no-op, whereas a genuine conflict on the same target now returns EEXIST so the caller can fail loudly rather than honour whichever registration happened to win. Fixes: 71e75f4 ("jail: refactor mount support to cover OCI spec") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Add a ptrace-based syscall tracer, selected via -m (enforce, trace, audit or complain) with -M naming an NDJSON log. Trace records every syscall, classifying each into a startup phase (linker, init, app) resolved from entry-point and libc breakpoints, so a generated profile can cover the application phase alone. Audit and complain run the real seccomp filter but rewrite its returns to SECCOMP_RET_TRACE, logging every denial; audit then enforces (errno or kill, including the two-stop errno arches), while complain permits and only records. Events stream to a udebug ring and, as a fallback, to the NDJSON file. procd's instance config gains seccomp_mode and seccomp_log, passed through to ujail as -m and -M. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Expose the seccomp modes: trace, audit and complain create the container in the chosen mode and route the log to /tmp/uxc-<name>.<mode>.json, making complain-derived profiling a first-class operation. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Replace the C utrace binary and its LD_PRELOAD trace helper with trace2seccomp, a ucode tool that turns a ujail seccomp trace into an OCI seccomp profile. The old preload approach only worked for dynamically linked binaries and required a private libpreload; the new tracer in ujail (-m trace) handles static binaries and reports per-phase syscall sets, so the generator can work purely from its NDJSON output. Dispatched by argv[0]: invoked as utrace or seccomp-trace it runs a program under `ujail -m trace` and emits an application-phase allow-list, matching the classic policy-generation window. Invoked as trace2seccomp it converts a previously captured NDJSON trace, optionally merging all phases or emitting a two-phase pre/post dynamic-linker profile. The CMake UTRACE_SUPPORT target now installs the script instead of building the dropped utrace and preload-trace artefacts. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Stage the jail's /dev as a private tmpfs in the host namespace before clone3() enters the user namespace, where device mknod is forbidden. Nodes are created and chowned to the host id that container-root maps to, the symlinks, sub-mount points and an optional console placeholder are pre-laid, then /dev is remounted read-only so the kernel locks MNT_LOCK_READONLY when the namespace copies it. This lets a read-only, content-addressed image run without a writable /dev, and pivot_root now pivots to self with a lazily detached old root rather than a writable put-old in the rootfs. Add -V <src:dest> to bind a host path as a noexec,nosuid,nodev volume, and prime autofs-backed sources by opening them in the host namespace so an automount triggered later from the private namespace cannot leave the overlay base empty. Bind targets are created relative to the jail root via openat2 with RESOLVE_BENEATH and RESOLVE_NO_MAGICLINKS so a symlinked target cannot escape, and OCI bundles keep ownership of their own mount namespace. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
A container running its own netifd needs a resolv.conf that tracks netifd's atomic replace of resolv.conf.auto on a read-only rootfs. Bind the staged /dev/resolv.conf symlink itself (open_tree with AT_SYMLINK_NOFOLLOW, not its target) onto /etc/resolv.conf, so every open re-resolves through /dev/resolv.conf.d. The target file is created only when the rootfs is writable; otherwise the staged placeholder suffices. On the host side, pre-create resolv.conf.auto in the per-jail /tmp/resolv.conf-<name>.d before bind-mounting it at /dev/resolv.conf.d, so the mount has a target before the in-jail netifd first writes it. This wiring is now gated on the private_netifd annotation rather than merely on a new network namespace, and the org.openwrt.procd.ubus and .netifd annotations are parsed as distinct keys. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Add uxc-net, a ucode helper that brings a container's host-side network
up and down over ubus, leaving no residue in persistent /etc/config. A veth
is created in netifd and the container end handed to the jail netns via a
dynamic jail interface; the host end is wired per attachment mode read from
the OCI annotations (org.openwrt.network.{attach,egress,ingress,proto}).
bridged:<network> enslaves the host end into a network's bridge and the jail
inherits that network's zone wholesale, auto-creating an isolated network
once if it is absent. routed places the host end as the gateway of a
point-to-point /31 in the container's own fw4 zone, compiling egress and
ingress into explicit forwardings and DNAT redirects, defaulting to
deny-all. host and none do no wiring; rollback is symmetric.
Every netifd-facing identifier is an fnv1a slug of the qualified name to fit
IFNAMSIZ and avoid the dot being read as a VLAN tag. The in-jail netifd
config is rendered to /tmp/run/uxc-net so the container side is configured
from the same declarative source.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Drive uxc-net synchronously on start and stop: "up" compiles the container's declarative annotations into ephemeral host netifd config, declaring the jail interface that the subsequent host-side device move picks up, and "down" tears it down. The host device move (jail_network_attach) now runs for every named netns container, so an annotation-less container still receives its host-managed device without a private netifd. A container-private ubusd and netifd are opt-in via annotations and default off, so OCI orchestrators managing networking on the host side are not fought by a parallel in-jail netifd. gen_jail_uci_network() drops the UCI section-rewriting in favour of copying the config uxc-net rendered, falling back to bare loopback. The netifd startup wait gains a timeout backstop, and a stale ubus socket from an unclean teardown is removed so its IN_CREATE still fires. jail_network_stop becomes the netns-agnostic jail_network_teardown, called from poststop and every error path so the per-jail ubusd and netifd never leak as procd orphans. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Drive the uvol backend from uxc create and delete so a container's writable storage is created on demand. A registration may declare an overlay-size for the container's own rw upper, and a data-volumes array of named volumes with mountpoints and sizes; uxc creates each via uvol (growing an existing one, keeping it if already larger), activates it, and binds it into the jail. The link to the backing store is read straight from the existing config: the image volume is the basename of the bundle path, the rw state volume the basename of write-overlay-path, so no extra field duplicates it. Image volumes are deactivated after write, so create activates the image before ujail reads the bundle. Booting a specific mountpoint defers auto-create until the uvol backend (VG) is online. On delete the rw state and per-container data volumes are reaped only when --volumes is given; the content-addressed image volume may be shared and is never removed here. uvol is invoked by fork/exec with explicit argv to keep sizes and names clear of shell quoting. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Add a -x option that appends KEY=VALUE lines from a file to the OCI process environment, parsed after the bundle's own env so a deployment can extend it without editing the image. Blank lines, comments and lines without an equals sign are skipped. procd's instance config gains an envfile attribute passed through as -x, and a changed env file triggers an instance restart. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Materialise a registration's initenv into per-container state once, on first create, writing an env file ujail then loads via -x. A value of "generate" yields a fresh random hex secret; "generate@<scope>" yields a secret shared across that scope, generated-or-read under a flock so whichever instance starts first seeds it and the rest read the identical value. Secrets live mode 0600 under .meta/secrets and the env file is created mode 0600. This lets a packaged container seed first-run credentials (database passwords, API tokens) deterministically, with stack members converging on one shared value without ordering constraints, and without baking secrets into the image. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Add two data-bind options that skip the executable dependency resolution of -r: -b binds a read-only data file (a rendered config or the stack's /etc/hosts), -k a read-only credential file. A provisioned "#!/bin/sh" config must not be treated as a program whose host interpreter and libraries get bound over the container's own, so neither pulls dependencies. A credential bind is idmapped once the OCI process user is known, mapping host id 0 to the container's app uid/gid, so a single root-owned secret is readable only by the container's process without a chown and without being world readable, and the same file can serve several containers. Because mount_all() runs in the child as the mapped userns uid and cannot traverse host paths under uvol's 0700 .meta, the bind source is cloned as a detached mount with open_tree in the parent (real root) and move_mount()ed into place, mirroring the existing resolv.conf handling. procd's instance runner maps mount type codes 2/3/4 to -V/-k/-b. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Render the provision array into the new binds (secret => idmapped -k, plain => -b) and bind the stack's shared hosts-file read-only. The hosts-file bind registers as mount type "4" only now, together with the provision handling rather than with the initenv support it was developed alongside, because procd maps type 4 to -b only from the previous commit on; registered earlier it would have been bound via -r with executable dependency resolution. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Emit ubus events across a container's lifecycle: instance.ready once the OCI state reaches created, instance.running once init has execve'd (signalled by EOF on a CLOEXEC exec-ack pipe, which also lets the parent drop its copy of the staging /dev), and instance.stopped last in free_and_exit, after the network is removed and cgroups destroyed. procd emits no instance.stop for containers, so these events are the only lifecycle signal, and the late instance.stopped doubles as the "fully stopped, safe to recreate" marker. The error paths of main() and post_poststop() are folded into free_and_exit() so every exit emits the event. The initial OCI state becomes creating instead of created, so state queried before the runtime is set up no longer claims a created container. This makes container_handle_kill answer NOT_FOUND while creation is still in progress, so until the uxc half of this change lands, "uxc delete --force" against a container in the middle of create fails. kill learns a negative signal meaning graceful stop: SIGTERM first, with escalation to SIGKILL after UXC_STOP_TIMEOUT. A child that dies before the sync pipe handshake is diagnosed with its wait status, and a failing chdir to the OCI cwd reports the path. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Add an event-driven waiter (uloop plus an instance.* handler) so create, start, kill and delete block on the matching event instead of returning once the ubus call is acked. A waiter also terminates on an early instance.stopped (the container died before becoming ready) and on a timeout for an ungraceful death that never reaches free_and_exit. A kill without an explicit signal sends the negative graceful-stop signal and waits for instance.stopped. A container's stdout and stderr are routed to the system log so a crashing payload is diagnosable via logread. create reports what the events tell it. A container that exits before it is ready, or that never reaches created state at all, is a failed create, and the instance is dropped again so that the next attempt is not met with the EEXIST of a half-created container. A container that exits before instance.running arrives is not a failed start, on the other hand: a payload that simply runs to completion is what runc and crun report success for. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Refuse to delete a container whose registration carries origin "package": such containers are owned by an apk package and must be removed with apk del, not by hand, so uxc returns EPERM and points the operator at the right command. On boot, self-heal an interrupted upgrade that left a registered container whose content-addressed image volume was never written: rather than wedging on the missing bundle, skip the container and point at "apk fix <name>" for recovery. Combined with the orphan-state reap that purges per-container state whose registration has gone, this keeps a fleet's on-disk state converging on its registrations without manual intervention. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Add uxc-stack, a ucode engine that instantiates and tears down a multi- container stack from a template. "up <app>" loads a stack template returning compose(api), which declares named instances drawn from shared image layers, then writes one autostart registration per instance and triggers bring-up by a ubus event procd reconciles through the boot path, avoiding a synchronous create that would serialise the stack on each member's start. "down <app>" is driven from the persistent registrations (origin "stack:<app>"), not the template, since apk removes the template before pre-remove runs. Addressing is deterministic and state-free, derived by FNV-1a from names: a per-instance host-uid offset strided by 64k so re-created instances keep their data-volume ownership, and a backhaul /24 within RFC 2544 198.18.0.0/15. The engine writes a shared /etc/hosts (retaining localhost) for resolver-free discovery, plus per-instance annotation sidecars uxc-net merges over the image's baked annotations. Secrets are referenced via generate@ directives uxc materialises, and bound idmapped rather than copied. On teardown, data volumes and their paired generated secrets are kept and reported for manual removal. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Group a container's cgroup leaf under its stack (the qualified name's first segment) and give the leaf a per-process suffix derived from the init pid. Previously the leaf was named purely from the jail name, so a respawn could land in the cgroup an exiting instance was still tearing down. That teardown writes cgroup.kill, which would then SIGKILL the newcomer mid-setup, observed as "can't read from child". The pid suffix guarantees a fresh leaf for every process, decoupling a new instance from the teardown of its predecessor. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Advertise OCI runtime-spec 1.3.0 now that the process, linux, hook, seccomp, cgroup and namespace fields added across this series cover it. uxc stamps 1.3.0 into the config.json it generates. ujail's version check is changed from the exact "1.0" prefix to accept any 1.1, 1.2 and 1.3 spec, so bundles produced by current tooling are no longer rejected for their declared ociVersion while remaining within the major version the runtime implements. The version was stated twice, once for ujail and once for uxc. Move it to the shared container.h so the two cannot drift apart. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
uxc_set() dereferenced the parse table of the configuration loop after that loop had ended, so creating a container which has no configuration yet read uninitialised stack and crashed. The name being looked up is already available as the function argument. Fixes: df1123e ("uxc: add support for user-defined settings") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
An OCI runtime gives the container the standard descriptors of the process that created it: that is how conmon collects a container's output and how the runtime-tools validation suite reads its results. Until now procd could only relay container output to syslog, so everything a container wrote was lost to its caller. ubus carries a single descriptor per message, so a new top-level stdio-fds.h passes all three as SCM_RIGHTS over a socket pair. procd receives the socket with the add request and installs the descriptors on the instance in place of the syslog pipes, falling back to the previous behaviour when no socket accompanies the request; ujail does the same for the processes it starts on behalf of exec. The header also carries the sending helper the uxc side uses. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Send the caller's stdin, stdout and stderr as SCM_RIGHTS over a socket pair and pass its receiving end with the create and exec requests, so a container's output reaches the process that created it and podman logs and the validation suite see it. The three descriptors are chosen explicitly rather than taken from whatever uxc happens to hold at the time, because --log redirects our own stderr into the log file and the container's error output does not belong there. The caller's stderr is kept aside before that redirection and handed over. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
INFO, WARNING and DEBUG printed to stdout and ERROR to stderr, which now carry the container's own output, so every diagnostic ujail produced was mixed into the stream its caller collects. That corrupts anything structured, the TAP output of the runtime validation suite for one. Route the macros through ulog, which picks the terminal only when one is attached, and pin the OCI container case to syslog where these messages belong. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
The runtime spec requires the state of the container to reach every hook over its standard input so the hook can act on it; ours were started with whatever stdin the runtime happened to have, and the validation suite failed all three hooks_stdin assertions with "unexpected end of JSON input". Feed each hook the same state document the state method reports, from a pipe prepared before the fork. The document construction moves to oci_state_fill() so both paths render exactly the same thing. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
runc and crun write the bare decimal into the pid file they are asked for, and that is what readers of the file expect to find. ujail appended a newline, which a caller comparing the contents byte for byte, or reading the file with a parser that accepts nothing but digits, does not agree with. Fixes: 602b8fa ("jail: add option for pidfile") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Mounts were kept in a tree keyed by their target and applied while walking it, so they were established in alphabetical order of the destination rather than the order given in the configuration. The runtime spec requires the listed order, and the validation suite reported every entry that came after an alphabetically later one as out of order, /dev landing before /proc for instance. Keep the tree for lookups and add a list which preserves insertion order for applying them. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
setns_open() reports why a join failed and every caller threw that away, so a bundle naming a namespace path of the wrong type, or one that cannot be entered at all, was created as if the path had not been given. The runtime spec requires an error there, and the validation suite checks it for each type in turn. Propagate the failures. The time namespace is still not joined at all, which is a separate gap. Fixes: c482c5d ("jail: add support for referencing existing namespaces") Signed-off-by: Daniel Golle <daniel@makrotopia.org>
conmon learns a container's exit status by waiting for the pid it reads from the runtime's pid file, but with ujail that process is a child of procd and never of conmon, so conmon never obtains a status at all: podman reports 0 for a container that exited 42, and podman exec fails even when the command succeeded. A shim would give conmon something to wait for at the price of putting a process other than the container's own in the pid file, which tooling needs for introspecting namespaces, cgroups and seccomp state. Write the decoded status, WEXITSTATUS or 128 plus the terminating signal, to an exit_status file in the directory the pid file lives in. The value goes to a temporary file and is renamed into place, so a reader either sees the previous value or the whole new one, and it is written before the SIGCHLD that ends conmon's loop. Exec sessions do the same with their own pid file, so podman exec has a status to report as well. A stale value from an earlier run is removed when a pid file is written, and nothing is written when no pid file was asked for, which leaves the behaviour of a caller that does not want any of this unchanged. The status file on its own ends no wait: the container is procd's child, so its death raises no SIGCHLD in conmon, and a detached container's streams never reach end of file, leaving conmon to sit until its own timeout. The runtime therefore signals the process that asked for the container. conmon execs the runtime, so uxc runs as conmon's child and its parent is the waiting process; uxc opens a pidfd on getppid() with pidfd_open(), called through syscall() since musl wraps neither it nor pidfd_send_signal(), and sends the descriptor along with the create and exec requests. procd keeps it on the instance beside the stdio descriptors and lets ujail inherit it across the execve() that starts the jail, -a naming the descriptor number, and ujail sends SIGCHLD through it with pidfd_send_signal() once the container is gone, after the status has been written. An exec session signals through the descriptor its own request carried once the session has been reaped, and a respawned instance carries the same descriptor into the next ujail, so the invoker also learns when a later incarnation dies. This holds for any invoker instead of leaning on the conmon.pid filename, which is merely podman's default for --conmon-pidfile and silently defeated by overriding it. A pid would identify the invoker only for as long as it lives. Callers such as uxc.init and interactive shells routinely exit long before their container does, leaving behind a number the kernel may hand out again, and any scheme that re-checks the pid before the kill still leaves a window between the check and the signal. A pidfd pins the identity at the instant it is opened: from that instant on it is the only process a signal through it can ever reach, however much later it is sent. One window remains, and it is not one a descriptor can close: getppid() is read before the descriptor exists, so an invoker that died first yields the reaper instead, and a number already recycled by then names a stranger. uxc therefore re-reads getppid() after opening and sends nothing if it changed, which leaves only the two adjacent syscalls in between, against the whole lifetime of a container in the old scheme. ujail still polls the descriptor for POLLIN first, the same way exec_jail() watches the pidfd of its own parent, and stays silent for an invoker that has already gone. A caller that supplies no descriptor is never signalled; when pidfd_open() fails uxc warns and sends none rather than falling back to a pid, since a caller which then waits for a wake-up that never comes deserves to see why. The descriptor set now carries its own count, so procd and uxc must be upgraded together: the previous revision of this commit accepted exactly three descriptors and nothing else. Losing the carrier now loses the notification with it, where the pid had travelled separately in the request. Two limits are worth naming. The signal is sent as ujail begins tearing down, before cgroups and the network are dismantled, so a woken manager can observe a container whose traces have not all gone yet. And an exec session still in flight when the container itself dies is never signalled, because the supervisor exits first; such a session is left to end on stream EOF, as it was before. ubus carries a single descriptor per request, and the three standard descriptors already travel as SCM_RIGHTS over a socket pair whose receiving end goes to ubus_invoke_fd(). That carrier now takes a counted set instead of a fixed trio, the count riding in the payload byte, because the stdio descriptors are only sent when pass-through is wanted while the notification descriptor is wanted independently of that: a create sends stdio and, when the invoker could be named, the pidfd; an exec session with a terminal sends the pidfd alone. The receiver tells the layouts apart by the count, which stays unambiguous because stdio is all or nothing. The invoker's identity is thereby no longer configuration. procd used to compare the notifypid attribute like the other jail attributes, restarting a running instance when a re-add named a different invoker; the descriptor is runtime state like the stdio descriptors, so a re-add replaces the stored descriptor for the next start while the running jail keeps the one it inherited. The descriptor stays close-on-exec everywhere except across the one execve() that starts the instance's own ujail, and ujail marks it close-on-exec again as soon as it parses the option, so neither hooks nor the container itself ever inherit it. conmon ends its loop when the streams it handed the runtime reach end of file and only then looks for the status, so an exec session's descriptors stay with the session and are closed once its status has been written. Recording it from the process that waits for the session is no alternative: that one has joined the container's mount namespace, where the path the status belongs at does not exist. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
07e0f7d ("jail: fix /proc,/sys mounting under CLONE_NEWUSER") did two things: it replaced the hardcoded MS_NOATIME on the procfs and sysfs mounts with detect_atime_flag(), which is what actually made them mountable and which is kept, and it moved the creation of the user namespace to after the mounts were built, which left the pid, network, ipc, uts and cgroup namespaces owned by the initial user namespace. ujail itself, being privileged, never had a problem with that; the container at runtime did. The kernel resolves a container's privileges against the owner of the namespace being used, so a container holding CAP_NET_BIND_SERVICE could not bind a privileged port, __inet_bind() asking ns_capable(net->user_ns, CAP_NET_BIND_SERVICE), and it could mount neither its own procfs, which wants CAP_SYS_ADMIN in the user namespace owning the pid namespace, nor its own sysfs, which wants the same in the one owning the network namespace. runc and crun both establish the user namespace before everything else for this reason. Undo only the reordering half of that commit. Let clone() create it together with the rest, which the kernel attributes to the new user namespace since the credentials are copied before the namespaces, and have the child wait for its uid and gid maps before it does anything that needs privilege. Creating it late is kept for the one case crun also keeps it for, a container joining existing namespaces, whether an OCI bundle names them by path or -j on the command line names them by pid, because entering those needs privilege in the user namespace owning them. The time namespace is not yet handed over: CLONE_NEWTIME stays masked out of the clone3() flags and the parent still unshares it before the clone, so it remains owned by the initial user namespace and the title only fully holds once a following commit moves it into the child as well. Redefining what defers the user namespace moves work between the two phases without changing the end state. remask_after_unshare(), remount_proc_sys_after_unshare(), the oci_deferred_* bookkeeping and the JAIL_NOAFILE bind now serve only the deferred path; on the common path the default masks and the read-only /proc/sys hack are applied while the mount list is built. The deferred path in turn drops privileges with setregid(), setreuid() and setgroups() before its second unshare(CLONE_NEWNS) rather than after. The inherited-mount detach introduced by a1c5633 ("jail: detach inherited mounts under /proc,/sys before mounting own") and guarded by 3fc9d11 ("jail: run inherited-mount detach before joining an external userns") now runs in two cases only, the deferred user namespace and the join of an external one. In a mount namespace owned by our own user namespace the inherited mounts are locked and cannot be detached; for mounts sitting on the kernel's permanently empty mount points it is not needed either, since mount_too_revealing() ignores those, but a locked mount covering an ordinary path, which our own masking and OCI maskedPaths create, still disqualifies the reference mount and can no longer be detached there. The command line could combine -j <pid>:user with -f, which the old code tolerated because CLONE_NEWUSER was always stripped from the clone flags and the join simply won. Now that the flag reaching clone3() creates a namespace of its own, -f leaves it unset when a user namespace has already been joined; an OCI bundle cannot express the combination, as parseOCIlinuxns() rejects the duplicate in both directions. Comments left over from the late-creation scheme, now stating the opposite of what the code does, are dropped. One detail follows from the new order: the gid 5 the standard /dev/pts options carry cannot be resolved by a mapping that holds a single id, and devpts refuses a mount whose gid does not map, so an unmapped gid is dropped from the options. crun likewise omits gid=5 for its rootless containers, though it does not filter bundle-supplied options this way. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
6212139 ("jail: give the container's namespaces to its own user namespace") left one namespace behind: the time namespace was still created by the parent, with unshare(CLONE_NEWTIME) before the clone, and therefore stayed owned by the initial user namespace. clone3() cannot simply take the flag either, since a child created with CLONE_NEWTIME sits in the namespace from birth, which freezes its offsets before anyone can write them. Create it in the child instead, once the uid and gid maps are in place. A namespace made by unshare() is owned by the user namespace of its creator, so it now belongs to the container's user namespace, where mapped root holds CAP_SYS_TIME, exactly what writing /proc/self/timens_offsets asks for. unshare(CLONE_NEWTIME) does not move the caller, and the offsets of an inhabited namespace are sealed, so the child writes the offsets first and then enters through /proc/self/ns/time_for_children with setns(), which the kernel allows while the process is still single-threaded. A bundle that defers its own user namespace creates the time namespace in enter_userns() for the same ownership reason; /proc is present there, as a new time namespace can only be configured through OCI and an OCI jail always mounts it. The createContainer hooks of such a bundle now run before the time namespace exists; everything else runs inside it as before. A time namespace joined by path moves to the child as well, alongside the other setns() joins and before any user namespace is created or joined, because entering needs privilege in the user namespace owning the target. The parent keeps only the probe for kernels without time namespace support, and no longer switches its own time namespace around the clone: the old join path setns()'d the supervisor into the container's time namespace and back, briefly running it on shifted clocks. Signed-off-by: Daniel Golle <daniel@makrotopia.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates
procd-jailanduxcto OCI runtime spec 1.3.0. v2: consolidated series, rebased onto current master. Seccomp is now applied via ptrace syscall injection (drops the LD_PRELOAD helper); also adds container lifecycle events, rootless containers via idmapped mounts, and the uxc-net/uxc-stack helpers.Should fix #35