Skip to content

linux-arm64 nested KVM: guest wedges the vCPU (enter/exit spin) under some workloads #2025

Open

Description

@danbugs

Summary

On nested arm64 KVM (a Linux guest running under Apple's Hypervisor.framework via Tart on Apple-silicon Macs — our self-hosted arm64 CI runners), certain guest workloads wedge the vCPU: the hyperlight VMM process pegs a full CPU core spinning in the KVM vCPU run loop, and the guest never makes progress. The same workloads run fine on macOS arm64 (HVF) and Windows arm64 (WHP); only the linux-arm64 (KVM, nested) backend is affected.

Observed with the Node/V8 http-express demo and the smolagents/mcp suites in hyperlight-unikraft's arm64 nightly; http-flask (CPython) and compute-heavy jobs (verl, bench, clickhouse) are fine.

Evidence (live capture on a wedged runner VM)

With http-express wedged, the single-threaded VMM process:

  • is in state R (running), ~100% of one core (CPU jiffies/2s ≈ 200);
  • has its kernel stack consistently in kvm_arch_vcpu_ioctl_run (+ kvm_arch_vcpu_ctxsync_fp) — i.e. spinning entering/exiting the guest;
  • makes zero network host-calls and never reaches listen() on the server port, so it wedges during guest execution, before/independent of any I/O.

A concurrent http-flask run captured in the same trace completed its accept/recv/send host-calls normally.

Environment

  • Host: Apple M4, macOS 26.x; guest: Ubuntu 24.04 Tart VM booted --nested, exposing /dev/kvm.
  • hyperlight-host KVM aarch64 backend; guest is a Unikraft unikernel running the workload.

Impact

Fails the affected jobs in hyperlight-unikraft's arm64 nightly (previously as 90-minute job-timeout hangs that also starved co-tenant jobs; a fast-fail harness change has since bounded them).

Next diagnostic

KVM event tracing (ftrace/perf kvm:*) on the runner VM to identify the exit storm. Candidates: KVM_EXIT_ARM_NISV cache-maintenance retries from V8 self-modifying/JIT code, a stage-2 fault loop, or FP/SIMD trapping under nested virt. Happy to run this on the runner.

Repro

Run the http-express demo under hluk on a nested-arm64-KVM runner (Apple-silicon Tart --nested): the VMM pegs a core and the server never answers.

Activity

  1. ludfjig commented on Nov 5, 2026

    @ludfjig
    Contributor

    Maybe we should consdier moving this to the github actions repo? Not sure it belongs here

  2. github-actions commented on Nov 9, 2026

    @github-actions

    Job Failed Again

    Date: 2026-11-09 05:38:47 UTC
    Workflow Run: https://github.com/hyperlight-dev/hyperlight/actions/runs/42565646769

    The scheduled job has failed again. Please check the workflow logs and artifacts for details.

  3. github-actions commented on Nov 14, 2026

    @github-actions

    Job Failed Again

    Date: 2026-11-14 05:38:21 UTC
    Workflow Run: https://github.com/hyperlight-dev/hyperlight/actions/runs/43559973129

    The scheduled job has failed again. Please check the workflow logs and artifacts for details.

  4. github-actions commented on Nov 16, 2026

    @github-actions

    Job Failed Again

    Date: 2026-11-16 05:34:32 UTC
    Workflow Run: https://github.com/hyperlight-dev/hyperlight/actions/runs/43808379877

    The scheduled job has failed again. Please check the workflow logs and artifacts for details.

  5. added
    C-bugFor PRs that fix bugs
    O-aarch64Specific to 64-bit Arm processors
    O-kvmLinux KVM backend
    O-linuxLinux host support
    on Nov 16, 2026
  6. syntactically commented on Nov 16, 2026

    @syntactically
    Member

    Not sure whether the failing actions runs here are in the correct place? They may also be the VM crashing because the correct kernels are not being used in the runner update last I heard. (@simongdavies @ludfjig is this still true?)

  7. jsturtevant commented on Nov 16, 2026

    @jsturtevant
    Contributor

    The bot comments are coming becuase this was originally written by CI.

  8. jsturtevant commented on Nov 16, 2026

    @jsturtevant
    Contributor

    @danbugs is the a bug with CI or a real bug?

  9. jsturtevant commented on Nov 16, 2026

    @jsturtevant
    Contributor

    It looks like the CI is still timing out with #2025 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

A-ci-periodics-aarch64Periodic aarch64 CI job failuresC-bugFor PRs that fix bugsO-aarch64Specific to 64-bit Arm processorsO-kvmLinux KVM backendO-linuxLinux host supportS-needs-triageThe issue has not yet been reviewed.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions