e2e/qa: stop counting devices that cannot accept users as failures

[ben-dz]

Summary of Changes

Mainnet Beta QA failed 7 times over Aug 8-10; three (infra runs 31248009915, 31266248546, 31286128831) had one cause. qa_alldevices_unicast_test.go checks device readiness at five sites, but each increments FailedTests before that check, so the carve-out suppressed only the log line. laconic-{dfw,mia,was}-sw01 have been activated at max_users=0 since Aug 6 and the CLI refuses those connects outright; against cmh-mn-qa01's 13-device pool that read as 21-29% on a 20% per-host gate.

Gating that counter is not enough: Success() also requires non-zero packet counts, which a device that never connected cannot produce. Not-ready devices are therefore excluded from ComputeFailureStats entirely — per-host denominator included — behind a new Device.Ready() replacing the condition previously spelled out at six call sites.

Exclusion opens a fail-open path worth reviewing closely: the rate is then measured over a shrinking pool, and with every device excluded it is 0/0, where NaN > threshold is false. Coverage is now gated on the skipped rate against -skipped-threshold (default 0.5), fleet-wide and per host — one metro draining is a few percent of the fleet but all of a host's pool, and per-host is the gate that fires in practice. Testing nothing fails independently of that threshold, which -skipped-threshold=1 would otherwise switch off. The gate uses t.Errorf, not t.Fatalf, so the publishers still run: the skipped count lands as devices_skipped beside devices_tested in InfluxDB and ClickHouse, the latter via an idempotent ADD COLUMN IF NOT EXISTS, best-effort so a writer without ALTER rights still gets its per-device rows in.

Ready() is deliberately a subset of is_device_eligible_for_provisioning: a device at users_count + reserved_seats >= max_users hits the same CLI rejection and still counts as a failure, but fixing that means re-adding the capacity check #3697 removed, so it is a follow-up.

Separately, client_unicast.go wrapped an always-nil lastErr, so every failure this weekend read failed to ping after 3 retries: %!w(<nil>). It now reports packet counts.

Testing Verification

go test ./e2e/internal/qa/... and golangci-lint run --build-tags=qa ./e2e/... pass. The new subtests fail against the unfixed aggregation: the mainnet shape reports total=6 failed=3 for total=3 failed=0, and a host whose whole pool is drained was absent from PerHost rather than visible there as zero coverage.

Not verified: the qa-tagged tests need live fleet access, so nothing here runs against the real fleet and no run has written a devices_skipped row yet. Mainnet has 5 of ~92 non-transit devices unusable today, well under the 0.5 default.

Two pre-existing holes on the same axis go to separate PRs: a not-ready device still consumes a batch host slot, which on the two-host devnet run can trip the fewer than 2 clients connected fatal, and the Grafana active-device filter fails open on a query error but not on an empty successful response.

[nikw9944]

The exclusion is well-targeted and the premise checks out — is_device_eligible_for_provisioning is enforced client-side in the CLI before the onchain qa_allowlist can exempt the connect, so a max_users=0 device genuinely proves nothing. One gap in the new fail-open guard: it only fires at zero eligible devices, so a large partial drain still goes green over a fraction of the fleet with no durable signal of the lost coverage.

[nikw9944]

Re-checked at 89dec13: the coverage gap is closed and verified in code, not just claimed. SkippedRate() now gates fleet-wide and per host against -skipped-threshold, zero tested devices still fails independently of that threshold, Errorf replaces Fatalf so the publishers below it run, and devices_skipped lands in both InfluxDB and ClickHouse. The per-host gate is strictly more lenient than the 20% failure gate it displaces for skipped devices, so it adds no new false reds. No further findings.