multikernel: make IPI publication ordered and recoverable - #7
Conversation
| */ | ||
| mk_arch_send_ipi(target); | ||
| if (ret == -ENOSPC) | ||
| pr_warn_ratelimited("multikernel: IPI ring full for instance %d\n", |
There was a problem hiding this comment.
The code this replaces used printk_deferred for the ring-full case, with a comment explaining that console output reaches this path. That is still true: mktty_spawn_console_write() runs as a console ->write callback during printk flushing, holding mktty_console_lock with IRQs off, and calls multikernel_send_ipi_data(). If publish fails here, these plain printks recursively enter console flushing, which re-enters mktty_spawn_console_write() and self-deadlocks on mktty_console_lock. Ring full is most likely precisely during heavy console output, so this fires under exactly the wrong conditions. These messages should stay printk_deferred (ratelimiting can be kept via __ratelimit).
| } | ||
| rc = mk_arch_spawn_instance(mk_image, instance, cpu); | ||
| if (rc == 0) { | ||
| rc = mk_ipi_shared_wait_ready(instance->ipi_data, mk_id, |
There was a problem hiding this comment.
This waits up to 120 seconds while still holding the kexec lock taken by kexec_trylock() at the top of multikernel_kexec_by_id(). While a wedged spawn is timing out, every other kexec operation returns -EBUSY, and if the host panics in that window __crash_kexec() fails its trylock, so the crash kernel never boots and the vmcore is lost. Consider dropping the lock before the wait (the new MK_STATE_LOADED check already excludes concurrent exec of the same instance), or at least a much smaller default timeout.
| ret = mk_ipi_shared_validate(instance->ipi_data); | ||
| if (ret) | ||
| return ret; | ||
| if (!atomic_read_acquire(&instance->ipi_data->ready)) |
There was a problem hiding this comment.
Behavior change worth confirming: previously the host could publish into the spawn ring while the spawn booted, and the spawn drained everything after init. Now sends fail with -EAGAIN until the spawn multikernel_init (subsys_initcall) marks the downlink ready, and the callers in messaging.c and mktty do not retry, so anything sent in that window is silently dropped. If this fail-closed behavior is intended it deserves a mention in the commit message; otherwise callers need a retry, or the ready gate needs to allow queuing during boot.
| if (ret) | ||
| mk_manifest_reject_and_park(ret); | ||
| if (!atomic_read_acquire(&host_instance->ipi_data->ready) || | ||
| READ_ONCE(host_instance->ipi_data->ready_instance_id) != 0) |
There was a problem hiding this comment.
This requires the host ring ready_instance_id to be exactly 0, but a mid-level kernel acting as a host marks its receive ring with its own root_instance->id, which is nonzero for a spawn. A nested spawn therefore always parks here with -EHOSTDOWN. Nesting already has known gaps, but this turns them into a hard block at the handshake.
| return ret; | ||
| } | ||
|
|
||
| ret = mk_ipi_shared_mark_ready(root_instance->ipi_data, |
There was a problem hiding this comment.
If mk_instance_restore_from_manifest() fails to allocate the root instance (its -ENOMEM path returns with root_instance still NULL and does not park), this dereferences NULL at subsys_initcall time. A NULL check turning it into a clean init failure would be safer.
a18185a to
3330928
Compare
Spawn kernels cannot calibrate against host-owned PIT, PIC, or IO-APIC resources. Carry the host loops-per-jiffy, CPU and TSC frequencies, and local APIC timer calibration in the spawn boot context. Install fixed calibration callbacks before x86 timer initialization. Keep explicit command-line calibration authoritative. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Serialize shared-ring producers with a bounded owner-aware gate. Preserve FIFO publication and recover a gate only after its producer CPU is known to be parked. The gate and slot-state protocol change the private shared transport layout. Add exact pre-launch ABI checks and require an initialization acknowledgment before activation. Prepare each downlink with its target identity so callers can queue messages while the receiver boots, then drain those entries when its handlers publish readiness. Keep console-path failure diagnostics deferred to avoid recursive printk locking. Mark a launched instance active before releasing the global kexec lock, and wait for readiness without blocking crash kexec. Validate nested-parent identity dynamically and fail initialization cleanly when manifest restoration leaves no root instance. Fail invalid manifests and missing acknowledgments closed. A spawn started by a host without the pre-launch check validates the boot-context anchor before using shifted fields and parks locally on mismatch. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
3330928 to
1a984b7
Compare
2 participants
Summary
Compatibility
The private shared transport ABI is version 3. Host and spawn kernels validate the ABI magic, version, structure size, boot context, manifest layout, target identity, and initialization acknowledgment.
A prepared downlink accepts queued messages before receiver readiness. Incompatible or stale host/spawn pairings are rejected before the instance becomes active.