fix: avoid perf event enumeration on a PMU that faults the kernel

[romirdes]

Problem

On AWS m6i.16xlarge, perfspect metrics takes the whole instance down during metadata collection. The spinner stops at "collecting metadata", the instance stops answering ssh, and the metadata probes outlive their own timeout wrappers.

The cause is not a slow perf and not a missing event. Opening an event that lands on fixed counter 3 faults the guest kernel:

Oops: general protection fault, maybe for address 0x1: 0000 [#1] SMP NOPTI
CPU: 0 PID: 4851 Comm: perf
  x86_perf_event_update+0x48/0xb0
  x86_perf_event_set_period <- intel_pmu_set_period <- x86_pmu_start
  <- x86_pmu_enable <- __perf_event_enable <- _perf_ioctl <- perf_ioctl

The guest advertises 4 fixed-purpose counters, which claims counter 3 (TOPDOWN.SLOTS), while lacking GLOBAL_CTRL_EN_PERF_METRICS — so the kernel programs a counter whose MSR the hypervisor never implemented:

Performance Events: Icelake events, Intel PMU driver.
... version:              2
... generic registers:    8
... fixed-purpose events: 4
... event mask:           0000000f000000ff

The faulting task then dies inside x86_pmu_enable still holding perf's context lock and parks in D state in perf_event_release_kernel, after which every perf_event_open on the machine blocks in account_event/perf_event_alloc, in D state, ignoring SIGKILL. RCU stalls and soft lockups follow and the instance leaves the network.

That is also why wrapping the probes in a shell-level timeout cannot fix this: a D-state task takes no signals, so the probes outlive their own timeout and the machine is gone either way.

What reaches the fault

perf list, which calls perf_event_open per candidate event to test support. Measured on the target one probe at a time, on a freshly booted instance:

probe	result
perf stat -a -e instructions	1.12s ok
perf stat -a -e ref-cycles	1.05s ok
perf stat -a -e INT_MISC.UNKNOWN_BRANCH_CYCLES	0.00s exit=129, event not exposed
perf stat -a -e OCR.READS_TO_CORE.LOCAL_DRAM	1.05s ok
perf stat -a -e '{topdown.slots, topdown-bad-spec}'	0.00s exit=129, parse failure
perf stat -C 0 -e {9x instructions}	1.01s ok
perf list --json	instance gone, never answered again

So it is metadata collection, not metric collection, that takes the machine down. scriptPerfSupportedEvents and scriptPerfAllSupportedEvents both run perf list --json, inside the concurrent metadata batch — which is why one faulting script strands the other 39 and leaves only get architecture, the one script that never touches perf.

Fix

On an affected target, enumerate events from sysfs instead of asking perf to do it.

Reading sysfs opens no events and cannot fault, but the reason to prefer it is stronger than that: sysfs is the kernel's own list of events it will accept, so every event absent from it — including the topdown events at the root of the fault — is dropped from the metric definitions by the existing IsCollectable checks. Metric collection therefore cannot walk into the same fault later either.

Affected targets are identified by the incoherence itself rather than by a model or instance-type list: 4 or more fixed-purpose counters advertised while the kernel exposes no slots or topdown event. Every comparable cell is coherent one way or the other and keeps today's behavior exactly:

target	fixed counters	slots exposed	behavior
m6i.16xlarge	4	no	sysfs enumeration
bare-metal Ice Lake (8375C)	4	yes	unchanged
m7i.24xlarge	2	no	unchanged
m5.24xlarge (Skylake)	3	no	unchanged

The check reads only dmesg and sysfs, so it is safe on the very targets it protects, and it runs before the metadata batch rather than as part of it, because the command it guards against is in that batch. It fails open: if the counter count cannot be read, perf enumeration is kept, since wrongly narrowing a healthy target's event set is the worse error.

Trade-off, stated plainly

On an affected target the event set shrinks to what the kernel publishes — the 8 architectural events plus cstate/power. OCR.READS_TO_CORE.LOCAL_DRAM demonstrably works on that guest and is now skipped along with everything else perf would have resolved from its own tables. Metrics run, but the metric set there is thin. Reaching further would mean trusting raw encodings for every event, and the one class known to fault is exactly the class perf resolves by name from those same tables.

Testing

• gofmt, go vet, go build ./... clean; all 11 test packages pass.
• New unit test pins all four cells above plus the fail-open cases.
• The three generated shell scripts were extracted from the constants (not hand copies), checked with bash -n and shellcheck -S warning, and run on a live Intel host, where slots is exposed and the probe correctly reports "safe".

Note on the kernel side

A guest kernel that oopses because the hypervisor advertises a fixed counter it does not implement is a bug in its own right, worth reporting to AWS and to the perf maintainers. This change only stops PerfSpect from stepping on it.