Repository navigation
Description
Summary
On nested arm64 KVM (a Linux guest running under Apple's Hypervisor.framework via Tart on Apple-silicon Macs — our self-hosted arm64 CI runners), certain guest workloads wedge the vCPU: the hyperlight VMM process pegs a full CPU core spinning in the KVM vCPU run loop, and the guest never makes progress. The same workloads run fine on macOS arm64 (HVF) and Windows arm64 (WHP); only the linux-arm64 (KVM, nested) backend is affected.
Observed with the Node/V8 http-express demo and the smolagents/mcp suites in hyperlight-unikraft's arm64 nightly; http-flask (CPython) and compute-heavy jobs (verl, bench, clickhouse) are fine.
Evidence (live capture on a wedged runner VM)
With http-express wedged, the single-threaded VMM process:
- is in state
R(running), ~100% of one core (CPU jiffies/2s ≈ 200); - has its kernel stack consistently in
kvm_arch_vcpu_ioctl_run(+kvm_arch_vcpu_ctxsync_fp) — i.e. spinning entering/exiting the guest; - makes zero network host-calls and never reaches
listen()on the server port, so it wedges during guest execution, before/independent of any I/O.
A concurrent http-flask run captured in the same trace completed its accept/recv/send host-calls normally.
Environment
- Host: Apple M4, macOS 26.x; guest: Ubuntu 24.04 Tart VM booted
--nested, exposing/dev/kvm. - hyperlight-host KVM aarch64 backend; guest is a Unikraft unikernel running the workload.
Impact
Fails the affected jobs in hyperlight-unikraft's arm64 nightly (previously as 90-minute job-timeout hangs that also starved co-tenant jobs; a fast-fail harness change has since bounded them).
Next diagnostic
KVM event tracing (ftrace/perf kvm:*) on the runner VM to identify the exit storm. Candidates: KVM_EXIT_ARM_NISV cache-maintenance retries from V8 self-modifying/JIT code, a stage-2 fault loop, or FP/SIMD trapping under nested virt. Happy to run this on the runner.
Repro
Run the http-express demo under hluk on a nested-arm64-KVM runner (Apple-silicon Tart --nested): the VMM pegs a core and the server never answers.
Activity
ludfjig commentedon Nov 5, 2026
Maybe we should consdier moving this to the github actions repo? Not sure it belongs here
github-actions commentedon Nov 9, 2026
Job Failed Again
Date: 2026-11-09 05:38:47 UTC
Workflow Run: https://github.com/hyperlight-dev/hyperlight/actions/runs/42565646769
The scheduled job has failed again. Please check the workflow logs and artifacts for details.
github-actions commentedon Nov 14, 2026
Job Failed Again
Date: 2026-11-14 05:38:21 UTC
Workflow Run: https://github.com/hyperlight-dev/hyperlight/actions/runs/43559973129
The scheduled job has failed again. Please check the workflow logs and artifacts for details.
github-actions commentedon Nov 16, 2026
Job Failed Again
Date: 2026-11-16 05:34:32 UTC
Workflow Run: https://github.com/hyperlight-dev/hyperlight/actions/runs/43808379877
The scheduled job has failed again. Please check the workflow logs and artifacts for details.
syntactically commentedon Nov 16, 2026
Not sure whether the failing actions runs here are in the correct place? They may also be the VM crashing because the correct kernels are not being used in the runner update last I heard. (@simongdavies @ludfjig is this still true?)
jsturtevant commentedon Nov 16, 2026
The bot comments are coming becuase this was originally written by CI.
jsturtevant commentedon Nov 16, 2026
@danbugs is the a bug with CI or a real bug?
jsturtevant commentedon Nov 16, 2026
It looks like the CI is still timing out with #2025 (comment)