feat(test): command/binary validation mode + known-tricky kernel library#65
Merged
Conversation
Add `bpfcompat test --command` so an artifact can be validated through its own loader binary/command run inside each matrix kernel VM (verdict = exit code), instead of only handing a .bpf.o to the bundled validator. This exercises the real userspace loader path and needs no manifest kept in sync with that loader — directly answering reviewer feedback on falcosecurity/libs PR #3024 (Andrea Terzolo) and from the ebpf-go/vimto maintainer (Lorenz Bauer). - `--command <cmd>`: shell command run as root in each guest; pass == exit 0 (override with `--command-expect-exit N`). Exposes $BPFCOMPAT_BIN, $BPFCOMPAT_ARTIFACT, $BPFCOMPAT_REMOTE_ROOT. The command is shell-quoted as a single `bash -lc` operand (cannot break out to inject host syntax). - `--command-binary <file>`: local executable shipped into the guest. - `--artifact` becomes optional in command mode; if given it is staged and exposed as $BPFCOMPAT_ARTIFACT. No-artifact runs get a content-addressed `command://` identity so history/compare still work. - Reuses the result -> report -> registry pipeline: load_status="skipped" and the outcome lands in the report `functional` section as a synthetic `command` test; failure classifies as COMMAND_VALIDATION_FAILURE. VM runner only for now. Also add a curated library of known-tricky vendor kernels (`matrices/quirk-library.yaml` + `docs/kernel-quirk-library.md`) — the kernels where "version != eBPF feature support" bites (ring-buffer boundary, enterprise backports, no-BTF, vendor rebases, variant bands). A fresh ring-buffer run across it showed ubuntu-20.04-5.4 fail vs almalinux-8-4.18 pass (RHEL backport), and amazon-linux-2-4.14 fail (Amazon did not backport ring buffer) — i.e. enterprise backports are per-vendor, not a blanket guarantee. Tests: config validation, runner command-mode verdicts (pass/fail/expected-exit/ infra), and the guest command-line assembly + shell-quoting. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ErenAri
added a commit
that referenced
this pull request
Jun 29, 2026
…aim (#66) Two fixes prompted by review of how the work reads (not the code): - The "Implemented:" list never mentioned command/binary validation, so the capability shipped in #65 read as not-a-core-feature. Add explicit bullets for `--command` mode (verdict = your loader's exit code; the bundled validator is NOT used — tests the real userspace loader path) and the known-tricky kernel library. - Soften the overclaim "validates Falco's modern_bpf probe exactly as Falco's own loader runs it." bpfcompat mirrors libpman's loader contract (runtime-sized maps, helper-gated program variants, trial-probed iterators via a manifest); it does not run Falco's loader binary. Reworded in README and the Falco case study, pointing at command mode as the way to run that exact binary. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two related additions, both driven by reviewer feedback on
falcosecurity/libs#3024
(Andrea Terzolo) and from the ebpf-go/vimto maintainer (Lorenz Bauer):
1. Command/binary validation mode (
bpfcompat test --command)Validate an artifact through its own loader binary/command run inside each
matrix kernel VM, instead of only handing a
.bpf.oto the bundled validator.This exercises the real userspace loader path and needs no manifest kept in
sync with that loader — exactly what Andrea asked for ("a binary command as
an alternative to the
.oflow") and the vimto-execanalog Lorenz pointed at.--command <cmd>— shell command run as root in each guest; verdict = exitcode (override with
--command-expect-exit N). Exposes$BPFCOMPAT_BIN,$BPFCOMPAT_ARTIFACT,$BPFCOMPAT_REMOTE_ROOT. Shell-quoted as a singlebash -lcoperand (cannot break out to inject host-side syntax).--command-binary <file>— local executable shipped into the guest.--artifactoptional in command mode; staged + exposed as$BPFCOMPAT_ARTIFACTwhen given. No-artifact runs get a content-addressed
command://identity sohistory/compare keep working.
load_status: "skipped", theoutcome lands in the report
functionalsection as a syntheticcommandtest, failures classify as
COMMAND_VALIDATION_FAILURE. VM runner only for now.See
docs/command-validation.md.2. Library of known-tricky vendor kernels
matrices/quirk-library.yaml+docs/kernel-quirk-library.md— a curated set of real kernels where "version ≠ eBPF feature support" bites
(ring-buffer boundary, enterprise backports, no-BTF, vendor rebases, variant
bands). Point a
.bpf.oor your own loader (--command) at the whole library.A fresh ring-buffer run across all 11 (
load_attach, real QEMU/KVM) produced:ubuntu-20.04-5.4❌UNSUPPORTED_MAP_TYPEvsalmalinux-8-4.18✅ — RHELbackports ring buffer onto a lower-numbered kernel than the one that failed.
amazon-linux-2-4.14❌ vsalmalinux-8-4.18✅ on the same probe —enterprise backports are per-vendor, not a blanket guarantee.
oracle-linux-9-uek7-5.15actually booted6.12.0-…el9uek(UEK rebase).Tests
Config validation; runner command-mode verdicts (pass / fail / expected-exit /
infra error); guest command-line assembly + shell-quoting (incl. hostile input).
go test ./...,go vet, andgolangci-lint(new-code) all clean.🤖 Generated with Claude Code