[LXC] State-aware lifecycle management - #633
Conversation
There was a problem hiding this comment.
Pull request overview
Adds state-aware lifecycle support for the Linux LXC backend by implementing StatefulSandboxBackend in lxc_common, wiring dispatch from lxc-exec, and exposing the new backend key/prefix in both Rust parsing/dispatch and the TypeScript SDK state-aware types/helpers.
Changes:
- Register
lxcas a state-aware backend prefix/key in the Rust dispatcher/config parser (with new unit tests). - Add
LxcStateAwareRunnerimplementingprovision/start/exec/stop/deprovisionfor LXC containers (best-effort iptables cleanup on teardown). - Extend the SDK state-aware type system + prefix map to support
lxc, including per-phase config/metadata typings.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/core/wxc_common/src/state_aware_dispatch.rs | Adds lxc sandboxId prefix routing + a unit test. |
| src/core/wxc_common/src/config_parser.rs | Accepts experimental.lxc as a known experimental backend key and adjusts key matching for containment: "lxc" (with test). |
| src/core/lxc/src/main.rs | Switches to load_mxc_request and routes state-aware requests to LxcStateAwareRunner. |
| src/core/lxc/Cargo.toml | Adds serde_json dependency for error envelope printing. |
| src/Cargo.lock | Lockfile update reflecting the new serde_json dependency. |
| src/backends/lxc/common/src/state_aware.rs | New LXC state-aware lifecycle implementation. |
| src/backends/lxc/common/src/lib.rs | Exposes the new state_aware module. |
| sdk/src/state-aware-types.ts | Adds lxc to state-aware backend union and defines LXC per-phase config + metadata types. |
| sdk/src/state-aware-helper.ts | Adds lxc prefix mapping and hoists containerId as a cross-cutting field. |
…) (AB#62953349) Provision/start/exec/stop/deprovision for the LXC backend, modeled on IsolationSessionRunner; reuses lxc CLI wrappers and one-shot lxc-attach PTY streaming. Registers the lxc wire key in Rust dispatch/parser and SDK state-aware routing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b78bec0-e139-4cfd-9c10-092ef986d4f4
ded0b0d to
fabdfd0
Compare
… + narrow LXC start network type Restrict is_valid_container_name to the same character set and length bound (<=20 chars, alphanumeric/-/_) that NetworkIptablesManager::new uses to derive the per-container iptables chain name. This makes the container-name -> chain-name mapping an identity on valid names, so distinct names (e.g. 'a.b' vs 'ab', or names differing only past the 20th char) can no longer collide onto the same firewall chain and cross-tear-down each other's rules. Narrow LxcStartConfig.network to Omit<NetworkConfig, 'proxy'> so the SDK rejects network.proxy at compile time, matching the Rust runner which rejects it at start (apply_network_policy). Adds Rust + TypeScript tests for both. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…aware_provision.json) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
Resolved conflicts: - src/core/lxc/src/main.rs: kept state-aware imports; dropped now-unused ScriptRunner - src/Cargo.lock: took main's lock, reconciled via cargo metadata Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
Re-run GitHub Actions after a transient Hyperlight E2E network flake (hyperlight_networking live-HTTP cases timed out after 30s). No source changes; this empty commit only re-fires the pull_request workflows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
- Route Lxc state-aware dispatch through mxc_engine::run_state_aware so the lxc binary stays a thin CLI shim instead of hand-rolling the backend match. - Stop/destroy the container before tearing down its iptables rules in stop(), deprovision(), and the start() rollback, discovering the veth first so the FORWARD hook rule can still be deleted after the device is gone. Closes an unrestricted-egress window during teardown. - Clear lxc.mount.entry before reapplying filesystem mounts so a restart with a tightened policy no longer inherits the previous run's bind mounts (new LxcContainer::clear_config_item). - Kill the timed-out child's whole process group and bound the output drain in mxc_pty::run_with_pty so a leaked in-container process holding the pty open can no longer hang exec forever (new join_with_timeout helper). Adds unit/regression tests for each fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
🧪 Local test re-verification — 2026-07-17Re-ran the test suites locally at branch tip Windows host (
Linux (WSL2 Ubuntu-24.04,
Note: the |
Resolve conflict in wxc_common/src/state_aware_dispatch.rs: register both the `lxc` (this PR) and `wsb` (upstream microsoft#578) state-aware backend prefixes in backend_from_prefix, and keep both resolve_backend unit tests. Added `correlation_vector: None` to the lxc test to match the ParsedStateAwareRequest field introduced upstream (microsoft#624). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
🔀 Merge-conflict resolution — synced with
|
| container | ||
| .start() | ||
| .map_err(|e| MxcError::backend_error(format!("Failed to start container: {e}")))?; | ||
| if let Err(e) = apply_network_policy(&container, request, &mut logger) { |
There was a problem hiding this comment.
Container egress is still not filtered — the network policy fails open for the container's entire runtime. This is separate from the teardown-ordering fix in this update (which correctly closed the stop/deprovision window); this one is about the direction the chain is hooked, and it is untouched.
start() applies network enforcement here via apply_network_policy() -> NetworkIptablesManager::apply_firewall_rules(). The per-container chain filters on destination (-d ACCEPT/DROP for allowed/blocked hosts, then a default DROP/ACCEPT), but it is hooked into FORWARD with:
-I FORWARD -o <veth> -j <chain> (src/backends/lxc/common/src/network_iptables.rs:237)
In a bridged veth setup, container->internet egress enters the host FORWARD chain as -i , while internet->container replies are -o . Hooking on -o therefore applies the chain only to traffic heading INTO the container, never to traffic originating from it:
- Outbound SYN (-i veth): not matched -> leaves unfiltered; blocked_hosts, allowed_hosts, and the default DROP are all bypassed.
- Return SYN-ACK (-o veth): matched, but it's ESTABLISHED,RELATED -> ACCEPTed; and -d can't match a remote host on the return path anyway.
Net effect: default_network_policy=Block, blocked_hosts, and allowed_hosts are ineffective for egress; a container started with a deny-by-default policy can reach any destination for its whole lifetime, not just during teardown.
Two related fail-open paths in the same function: if discover_veth_interface() returns None it logs "Skipping FORWARD hook" but still sets rules_applied=true / returns Ok(true), so start() treats an unhooked container as success; and a non-Firewall/Both enforcement mode returns Ok(true) without applying anything.
Suggested fix: hook the chain on -i (network_iptables.rs:237) and change the cleanup -D FORWARD -o (network_iptables.rs:264) to -i, so the destination-based rules apply to container egress. Fail start (return Ok(false)/error) when the veth can't be discovered or the hook can't be installed, instead of proceeding with no enforcement. Add a test that sets deny-by-default and asserts the container cannot reach a non-allowlisted host.
There was a problem hiding this comment.
Fixed in 287b918.
You're right, and the direction was wrong in both places. For a host-side veth,
container-originated packets arrive at the host on that interface, so egress
matches FORWARD -i vethX; -o vethX matches traffic flowing toward the
container. So container egress -- the thing the policy exists to restrict -- was
never filtered.
Changed the insert and the matching -D at teardown; if the delete keeps -o
the FORWARD hook leaks on every stop. force_cleanup delegates to
remove_firewall_rules, so the signal path picks up the same fix rather than
needing its own.
Same bug was fixed independently on #631 (96af8f9); this branch had its own copy.
…tate-aware-lifecycle # Conflicts: # sdk/node/src/state-aware-helper.ts # sdk/node/src/state-aware-types.ts # sdk/node/tests/unit/state-aware-types.test.ts # src/backends/lxc/common/src/filesystem_mounts.rs # src/core/wxc_common/src/state_aware_backend.rs
Resolve conflict in src/backends/lxc/common/src/filesystem_mounts.rs as a union of both changes: - microsoft#633 mount-accumulation fix: clear_config_item("lxc.mount.entry") before re-deriving the policy's mounts (so a restart replaces, not unions, mounts). - upstream microsoft#630 denied-dir masking: rebound_container_paths / has_rebound_descendant, iterating &mounts. Both new unit tests (configure_filesystem_mounts_replaces_not_accumulates and has_rebound_descendant_detects_nested_rebind_only) are kept. Validated with `cargo check -p lxc_common --tests` (native linux/liblxc, WSL). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b6b3323b-7297-4b07-9e6e-ab4b220124e6
| wxc_common::models::ContainmentBackend::Lxc => { | ||
| let mut runner = lxc_common::state_aware::LxcStateAwareRunner::new(); | ||
| wxc_common::state_aware_dispatch::dispatch_state_aware(&mut runner, parsed, dry_run) |
There was a problem hiding this comment.
Partly addressed in 95a197d; the streaming gap itself is still open, and I'd
rather say so than imply otherwise.
You're right that LXC never reaches dispatch_state_aware_exec. What I fixed
is the misleading part: the fallback arm reported "backend Lxc does not
implement the state-aware lifecycle", which is simply untrue --
run_state_aware dispatches every phase for it -- and it sends callers off to
debug a provision path that works fine. The error now distinguishes "no
lifecycle at all" from "lifecycle but no streaming exec" and names the API that
does work.
I did not implement a real streaming SandboxProcess for LXC. That needs
lxc-attach process plumbing with live stdout/stderr, and I have no LXC host
here to validate it against -- shipping an unexercised streaming
implementation seemed worse than leaving the gap legible. Happy to file that as
a work item, or to take it in a follow-up if you'd prefer it in this PR.
There was a problem hiding this comment.
I'm tracking the streaming SandboxProcess as a follow-up rather than doing it in this PR, and I filed #765 for it. It needs lxc-attach process plumbing with live stdout/stderr and a real LXC host to validate against, and I have neither here, so shipping an unexercised streaming path would be worse than a legible gap. The misleading half is already fixed in this PR: the fallback arm now distinguishes "no lifecycle" from "lifecycle but no streaming exec" and names the API that works.
| let mut fw_manager = NetworkIptablesManager::new(container.name()); | ||
| if let Some(veth) = NetworkIptablesManager::discover_veth_interface(container.name()) { | ||
| fw_manager.set_veth_interface(&veth); | ||
| } | ||
|
|
||
| match fw_manager.apply_firewall_rules(&policy, logger) { |
There was a problem hiding this comment.
Fixed in 287b918, with a follow-up correction in c453787 -- see the note at
the end, my first cut of this was wrong.
The fail-open was real: apply_firewall_rules treats a non-firewall
enforcement mode as a successful no-op, enforcementMode defaults to
capabilities, and LXC has no capability-based network enforcement. So a start
carrying allowedHosts/blockedHosts returned success with the container
completely unfiltered.
apply_network_policy now rejects that combination instead of reporting
success, and also fails when the host-side veth can't be discovered in a
firewall mode -- without it the chain is built but never hooked into FORWARD,
which apply_firewall_rules only warned about.
The correction: I originally also treated defaultPolicy: "block" as an
explicit restriction. That was wrong, and Copilot caught it below --
NetworkPolicy::default() is Block, so once the wire network block is
flattened into ContainerPolicy a requested "block" is indistinguishable
from no network block at all. It rejected every plain start, including the
new E2E lifecycle test. Now gated on the host lists only, matching the
reasoning has_network_policy already uses. The residual gap -- defaultPolicy: "block" alone is not enforceable under capabilities and can't be
distinguished from unset -- is documented in the API doc rather than papered
over.
| // Clear any `lxc.mount.entry` lines from a previous start before deriving | ||
| // the current policy's mounts. `set_config_item` appends, and liblxc | ||
| // accumulates every entry across restarts, so without this a restart with a | ||
| // narrower policy would still inherit the earlier run's bind mounts. | ||
| container.clear_config_item("lxc.mount.entry")?; |
There was a problem hiding this comment.
Not fixed -- and I think the suggested approach doesn't work. Details, because
the conclusion is non-obvious.
The bug is real, confirmed: clear_config_item("lxc.mount.entry") wipes
everything, and since provision permits an external containerId and adopts a
pre-existing container, start can drop template or security-baseline mounts.
But the dedicated-include-file fix would not help. I checked liblxc rather than
assuming (lxc/lxc main, commit dc15af1):
lxc.include's set handler isset_config_includefiles(confile.c), and
it callslxc_config_read(value, lxc_conf, true)synchronously. The
included file is parsed beforeset_config_itemreturns, not deferred to
start.- Every
lxc.mount.entryin it goes throughset_config_mountand lands in
lxc_conf->mount_entriesas an ordinarystruct string_entry. The
from_includeflag only suppresses appending to the rawunexpanded_config
text; the in-memory list is fully merged. clr_config_mountcallslxc_clear_mount_entries(conf.c), which frees
the whole list with no filter.string_entrycarries no provenance.
So entries from an MXC-owned include file are indistinguishable from any other
and get wiped by the same clear. Relatedly, clr_config_includefiles is a
literal return 0; no-op and get_config_includefiles returns ENOSYS --
liblxc itself acknowledges include state can't be recovered post-expansion.
There's no selective-removal API for lxc.mount.entry.
What does work, from the same reading:
get_config_item("lxc.mount.entry")returns all current entries,
newline-separated raw fstab strings, so snapshot -> clear -> re-add the
entries we don't own is viable.- Unknown mount options are tolerated for ownership tagging:
parse_lxc_mount_attrsonly stripscreate=dir/create=file/optional/
relative/idmap=, andparse_vfs_attrreturns 1 for anything it doesn't
recognize, which appends it toopts->dataformount(2). For anone
bind mount the kernel ignoresdata, so anx-mxc-ownedmarker is inert.
The catch, and why I stopped rather than pushing a fix: that reasoning holds
for bind entries but not for the tmpfs mask entries this function also
generates, where data is consumed and an unrecognized option can fail the
mount with EINVAL. So the tagging scheme needs a second mechanism for those,
and I can't exercise any of it without an LXC host. Given this silently affects
real mounts, I'd rather land it as its own reviewed change than bolt it onto
this PR. Happy to file it with the above as the starting point -- say the word
and I'll open it.
There was a problem hiding this comment.
This is fixed now -- my earlier "not fixed" note is superseded. configure_filesystem_mounts calls clear_mxc_mount_entries (lxc_bindings.rs:398), not clear_config_item: set_mxc_mount_entry writes a "# mxc-managed-mount" marker line directly above each entry it adds (lxc_bindings.rs:363), and the clear removes only marker+entry pairs, so template and operator mounts survive. It edits the raw config file text rather than going through liblxc's config API, which is what sidesteps the in-memory provenance loss I wrote up before -- set_config_item and clear_config_item already work the same way. I haven't exercised it against a live LXC host, since I don't have one on this box.
| * error), so excluding it here surfaces the constraint at compile time instead | ||
| * of pushing a preventable failure to runtime. | ||
| */ | ||
| export type LxcNetworkConfig = Omit<NetworkConfig, 'proxy'>; |
There was a problem hiding this comment.
Fixed in 287b918.
Confirmed both leaks you're pointing at. removeRulesOnExit is SDK-only:
wire::Network (wire.rs:311) is deny_unknown_fields and has no such field,
and the SDK never strips it, so sending it fails the entire request rather
than being ignored. allowLocalNetwork does deserialize, but it appears in the
whole LXC backend exactly once -- state_aware.rs inside has_network_policy
-- and is never turned into an iptables rule.
LxcNetworkConfig is now an explicit
Pick<NetworkConfig, 'defaultPolicy'|'allowedHosts'|'blockedHosts'> plus
enforcementMode narrowed to 'firewall'|'both', so the type matches the
runtime check rather than Omit quietly re-admitting whatever gets added to
NetworkConfig later.
The existing type test asserted the old shape, so it was encoding the bug --
replaced it with cases that include @ts-expect-error guards on each excluded
field.
| export interface LxcProvisionConfig { | ||
| /** Schema version (semver). */ | ||
| version?: string; | ||
| /** Optional externally assigned LXC container name. */ | ||
| containerId?: string; |
There was a problem hiding this comment.
Fixed in 287b918. The seven LXC state-aware types are now exported from
sdk/node/src/index.ts, matching how the IsolationSession and WindowsSandbox
equivalents are surfaced. Without it they were unreachable from the package
entry point, so the typed API was undeclarable by consumers.
| /** Optional externally assigned LXC container name. */ | ||
| containerId?: string; |
There was a problem hiding this comment.
Fixed in 287b918.
Verified the doc was wrong: the state-aware shapes are documented as never
carrying containerId, but LXC provision does. Updated the
sandboxId/containerId table with an explicit LXC exception covering
adopt-or-create.
Also documented something that came out of checking this, because it's a sharp
edge rather than a doc nit: deprovision calls container.destroy()
unconditionally, including for a container that already existed and was merely
adopted. MXC keeps no state between phases, so at deprovision time it genuinely
cannot tell an adopted container from one it created -- a caller who passes an
existing containerId will lose it. Documented rather than "fixed" because a
real fix needs provenance persisted at provision; happy to file that if you'd
like it tracked.
Added the missing LXC row to the policy-honor matrix in 10.3 while I was in
there.
| "phase": "provision", | ||
| "containment": "lxc", | ||
| "experimental": { | ||
| "lxc": { | ||
| "provision": { "distribution": "alpine", "release": "3.20" } |
There was a problem hiding this comment.
Fixed in 783c197.
You're right that nothing ran it. lxc_state_aware_provision.json was checked
in but no script referenced it, and run_lxc_all_tests.sh had no state-aware
entry at all -- only one-shot cases. Both other backends already have a
lifecycle script; LXC was the odd one out.
Added run_lxc_state_aware_test.sh, registered in the suite. It drives
provision -> start -> exec -> stop -> deprovision, relaying the provisioned
sandboxId the way a real client does, asserts the lxc:mxc- prefix, and checks
that exec relays a nonzero script exit code rather than swallowing it. sed
rather than jq/python for parsing, since neither is guaranteed on an LXC host,
and an EXIT/INT/TERM trap deprovisions on early failure so a leaked container
doesn't break the next run.
I don't have an LXC host here, so I verified it against a stub lxc-exec:
happy path 8/8 with the sandboxId correctly relayed into each later phase, a
failing phase counted with a nonzero exit and no double-deprovision, and a
SIGTERM mid-run producing exactly one cleanup deprovision of the right sandbox.
bash -n clean, LF endings so the suite's CRLF guard passes, mode 100755. It
still needs a real host run to be worth anything -- flagging that explicitly.
Worth noting this test immediately earned its keep: it's what made the
defaultPolicy regression in my own network fix visible (see the thread
above).
| parsed.request.experimental_enabled = experimental; | ||
| parsed.request.testing_features_enabled = testing_features; | ||
| parsed.request.dry_run = dry_run; | ||
|
|
||
| let outcome = mxc_engine::run_state_aware(parsed, dry_run); |
There was a problem hiding this comment.
Fixed in 95a197d.
Confirmed -- lxc/src/main.rs called mxc_engine::run_state_aware directly
while wxc/src/main.rs wrapped the identical call in telemetry init,
backend/phase process attribution, the MS-CV seed/spin plan, the crash panic
hook, and the terminal emit_state_aware. So a Linux lifecycle emitted no
lifecycle telemetry, installed no crash hook, and carried no correlation vector
-- provision returned no cV for the client to relay into later phases. All of
that is invisible at the call site, which is how it went unnoticed.
Took the "share" option rather than "mirror": the orchestration moved into
mxc_engine::run_state_aware_with_telemetry and both entry points call it, so
they can't drift again. Executors keep their own terminal behavior (buffer
flush, stdout envelope, exit code) -- only the observability wrapper is shared.
The correlation-vector helpers moved with it along with their six tests, ported
verbatim rather than rewritten so coverage is unchanged.
This paid off immediately in the merge with main: upstream added
log_state_aware_dispatch_error to the wxc entry point only, and the lxc one
would have silently drifted again. Mirrored it in the same commit.
…art, SDK surface Five review findings, all cases where the state-aware LXC path reported success while enforcing less than the caller asked for. - Hook FORWARD with -i, not -o. Container-originated packets arrive at the host on the host-side veth, so egress matches by input interface. `-o` matched traffic flowing toward the container, so container egress -- the thing the policy exists to restrict -- was never filtered for the whole runtime. The teardown `-D` uses `-i` for the same reason, or the hook leaks; `force_cleanup` shares that path so signal and stop/deprovision cleanup stay consistent. - Stop start() from failing open. `apply_firewall_rules` treats a non-firewall enforcement mode as a successful no-op, and only warns when no veth was discovered. Since `enforcementMode` defaults to `capabilities` and LXC has no capability-based network enforcement, a policy with allowedHosts/blockedHosts/defaultPolicy=block was silently unenforced. Start now rejects that combination, and fails when the veth cannot be discovered in firewall mode. - Narrow LxcNetworkConfig to what LXC actually honors. It was `Omit<NetworkConfig,'proxy'>`, which still exposed `removeRulesOnExit` (SDK-only, and `wire::Network` is `deny_unknown_fields`, so sending it fails the whole request) and `allowLocalNetwork` (deserializes, but the LXC backend never turns it into a rule). `enforcementMode` is restricted to the firewall modes to match the runtime check above. The existing type test asserted the old shape and is updated. - Export the LXC state-aware types from the package entry point. They were missing from sdk/node/src/index.ts, unlike the IsolationSession and WindowsSandbox equivalents, so consumers could not import them. - Document the containerId contract. The API doc said state-aware shapes never carry containerId; LXC provision does. Documents the adopt-or-create behavior and, importantly, that deprovision destroys an adopted container too -- MXC keeps no state between phases, so it cannot tell the two apart. Adds the missing LXC row to the policy-honor matrix. Also applies `cargo fmt`, which fixes the failing format check. Tests: 478 Rust (cargo test -p lxc_common -p wxc_common -p lxc) and 210 SDK (npm test) pass; clippy on the Linux crates is clean; fmt is clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea24fae3-d643-4f19-a701-9e31b9f950fd
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 29 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
src/backends/lxc/common/src/state_aware.rs:282
- This gate checks only the number of indices, but the code below always pins
lxc.net.0.veth.pairand assumes a veth. An adopted config with one interface atlxc.net.1, or onelxc.net.0.type = macvlan, passes this check; its real interface does not use the deterministic veth, so the-i mxcv…hook does not filter its egress. Require exactly index 0 with typeveth, or derive and hook the actual supported interface, before reporting firewall enforcement.
if net.indices.len() > 1 {
src/backends/lxc/common/src/state_aware.rs:610
--dry-runstops after this validator, so it currently accepts start policies that realstartrejects (for examplenetwork.proxy, or explicit default-block/host rules undercapabilities). It also skips the side-effect-free filesystem normalization/delegation checks performed only byapply_filesystem_policy. Move the static start-policy checks into validation (and reuse them fromstart) so dry-run accurately predicts whether the request is valid.
fn validate_start(
&self,
sandbox_id: &str,
_request: &ExecutionRequest,
_config: Option<&()>,
) -> Result<(), MxcError> {
extract_container_name(sandbox_id)?;
Ok(())
sdk/node/src/state-aware-types.ts:23
- Adding
lxcto this common backend union also makesexecInSandbox(lxcId, ...)type-check, because that API is generic overStateAwareContainmentBackendand has no LXC exclusion or runtime guard. This contradicts the new README and PR contract that LXC streaming exec is unavailable. Introduce a streaming-capable backend subset forexecInSandbox, or support the exposed LXC path and update the documentation/follow-up scope.
export type StateAwareContainmentBackend = Extract<
ContainmentBackend,
'isolation_session' | 'lxc' | 'windows_sandbox'
>;
| let _ = Self::run_iptables(&["-F", &self.chain_name], logger); | ||
| let _ = Self::run_iptables(&["-X", &self.chain_name], logger); | ||
| if Self::run_iptables(&["-X", &self.chain_name], logger).is_ok() { | ||
| self.chain_created = false; | ||
| crate::signal_cleanup::clear_active_chain_created(); | ||
| } | ||
|
|
||
| self.rules_applied = false; | ||
| Ok(()) |
There was a problem hiding this comment.
The observation is accurate on every point -- -F and -X errors were discarded, remove_forward_hooks swallowed everything, and the function returned Ok(()) unconditionally while force_cleanup dropped the result. I fixed part of this in e65f65b and declined the rest, and the two halves deserve to be separated because they are not equally dangerous.
The half that mattered: flushing a chain that is still hooked. This is a fail-open, not a leak. -F succeeds whether or not anything still jumps to the chain, and an emptied user chain returns to its caller instead of reaching its own closing DROP -- so a chain that is still hooked but no longer filtering lets traffic straight through, silently. remove_forward_hooks (which this PR added in 5c03c13) was best-effort and the -F ran regardless of whether it achieved anything.
It now re-reads FORWARD after issuing its deletes and reports whether any jump to the chain remains, and that verdict gates the flush. Re-reading rather than trusting the -D exit codes is deliberate: a -D can fail for a rule another process is also tearing down, and its exit code says whether this call removed the rule, not whether the rule is gone. An unreadable FORWARD answers "still hooked" -- being wrong that way costs a retry, being wrong the other way unfilters a live container.
Deleting is the safe half and needs no such gate: -X on a still-referenced chain simply fails, and the chain stays owned and filtering for a later teardown to retry.
The half I did not change: propagating the failure. A chain whose hooks are confirmed gone is unreferenced -- it filters nothing and blocks nothing, and costs a chain name until the next teardown for that container reclaims it. Turning that into a failed stop/deprovision reports failure for an operation whose observable goal (container stopped, container destroyed) did succeed, and it does so on the path where the container is already gone and the caller has nothing useful to do with the error. I would rather leave the leak visible in the log than make the common case fail. Happy to be argued out of this one, but I did not want to change stop's error contract as a side effect of a teardown fix.
teardown_chain is split out from process execution so the flush ordering has an oracle at all -- every iptables command fails at spawn on a non-Linux host, so a test driving the real path could never observe a flush. Mutation-verified: flushing regardless of hook state fails only the new hook test, with a negative control (hooks gone must still flush then delete) so "never flush" cannot pass. 113 lxc_common tests.
Cross-reference: #632 fixes the same flush ordering in this file in 10f0599, by a different route (an iptables -C absence check rather than a FORWARD re-read, because that branch tears down hooks in both v4 and v6 tables). Whichever of the two merges second will need a conflict resolution here; I did not want to leave the fail-open in whichever merges first.
dispatch_state_aware runs backend.validate_start and, on a dry run, returns
an empty success envelope without calling start (state_aware_dispatch.rs,
Phase::Start). LXC's validate_start checked only that the sandbox id carried
a usable container name, so a dry run reported success for two policies the
real start refuses outright:
- network.proxy, which this backend does not support
- a host restriction (allowedHosts/blockedHosts, or an explicit
defaultPolicy "block") under a non-firewall enforcementMode, which LXC
cannot enforce and which start rejects rather than run fail-open
Both rejections need nothing but the request, and both lived inside
apply_network_policy where only the real start could reach them. A dry run
exists to answer "would this start be accepted", so answering "yes" to a
start that is about to be refused is worse than not offering the dry run.
Extracted reject_unenforceable_network_policy from apply_network_policy and
added validate_start_policy, which normalizes the filesystem policy first --
the order the real start reaches these, since apply_filesystem_policy runs
before apply_network_policy -- so a dry run reports the same error the real
start would, not merely some error. The apply path still calls the check
itself, so a caller invoking start directly is unaffected.
normalize_object_conflicts rewrites only the readwrite/readonly/denied path
lists (filesystem_object.rs:225-231), so the network verdict is identical on
the raw and normalized policy.
Mutation-verified: gutting validate_start_policy fails only the new test, and
the negative control -- the same restriction under enforcementMode
'firewall', which this backend can enforce -- stays green so that "reject
everything" cannot pass. 107 lxc_common tests pass.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 29 changed files in this pull request and generated 1 comment.
Suppressed comments (7)
src/backends/lxc/common/src/state_aware.rs:306
- A single configured interface is not necessarily
lxc.net.0. An adopted config containing onlylxc.net.1.*passes this count check, but the code below pinslxc.net.0.veth.pair; the realnet.1veth remains random and bypasses the only firewall hook while start reports success. Require the sole index to be exactly0.
if net.indices.len() > 1 {
sdk/node/src/state-aware-types.ts:140
enforcementModeis optional here, so{ blockedHosts: ['example.com'] }and{ defaultPolicy: 'block' }both type-check. Rust defaults an omitted mode toCapabilitiesand rejects those requests, contradicting the stated compile-time enforcement. Model this as a union that requiresfirewall/bothwhenever a restriction is present, or require the mode for every LXC network object.
export type LxcNetworkConfig = Pick<
NetworkConfig,
'defaultPolicy' | 'allowedHosts' | 'blockedHosts'
> & {
/** LXC enforces network policy only via iptables. */
enforcementMode?: 'firewall' | 'both';
sdk/node/src/state-aware-types.ts:148
- The implementation installs the iptables chain and hook before
container.start()specifically to avoid an unfiltered window, so this public JSDoc states the opposite ordering.
/** iptables policy to apply after the container starts. `proxy` is not supported by this backend. */
src/backends/lxc/common/src/network_iptables.rs:431
- Teardown ignores every hook-deletion, flush, and chain-deletion failure and then returns
Ok(()). Consequently state-aware stop/deprovision can report success while a stale hook/chain remains; the next start then fails atiptables -Nwith no indication that the prior successful cleanup was incomplete. Propagate residual cleanup failures (while retaining ownership for retry) through the lifecycle result.
let _ = Self::run_iptables(&["-F", &self.chain_name], logger);
if Self::run_iptables(&["-X", &self.chain_name], logger).is_ok() {
self.chain_created = false;
crate::signal_cleanup::clear_active_chain_created();
}
self.rules_applied = false;
Ok(())
sdk/node/src/state-aware-types.ts:22
- Adding
lxcto this common backend union also makes the genericexecInSandboxAPI acceptSandboxId<'lxc'>; that function has no LXC guard and spawnslxc-exec. This conflicts with the README and PR contract that LXC streaming exec is unavailable and tracked separately. Use a narrower streaming-capable backend type (plus a runtime guard for untyped callers), or document and support this path.
This issue also appears in the following locations of the same file:
- line 135
- line 148
export type StateAwareContainmentBackend = Extract<
ContainmentBackend,
'isolation_session' | 'lxc' | 'windows_sandbox'
tests/scripts/run_lxc_state_aware_test.sh:51
- Trapping
INT/TERMwithcleanupdoes not make Bash exit; a trapped signal normally resumes execution afterward. Because this script intentionally omitsset -e, cancellation can deprovision the sandbox and then continue running later phases against it instead of terminating. LetEXITperform cleanup and have signal traps exit with the conventional status.
trap cleanup EXIT INT TERM
docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md:1626
- This says explicit
defaultPolicy: "allow"leaves the presence bit unset, but the parser setsdefault_network_policy_present = truefor every explicit value.allowmerely does not require firewall enforcement because it is permissive. Correcting this distinction avoids misleading backend implementers about the internal contract.
> supplies no `network.defaultPolicy`, or supplies `defaultPolicy: "allow"`, does not
> set the presence bit and is unaffected. To start a default-deny container, set
> `enforcementMode` to `"firewall"` or `"both"`.
| if fw_manager.chain_created() { | ||
| cleanup_network(container.name(), None, logger); |
There was a problem hiding this comment.
Confirmed and fixed in 5449775. The mechanism is exactly as described.
cleanup_network builds a fresh NetworkIptablesManager, which cannot clear the owning manager's chain_created. Drop tears down whenever that flag is set (network_iptables.rs, impl Drop), so the arm returned straight into a second teardown of a chain it had already deleted. And the chain name is derived from the container name, so another start of the same sandbox aims at exactly that name -- if it wins the gap between the two teardowns, the trailing one removes hooks by chain name and deletes the chain, and that start's container runs unfiltered. This process's failed start becomes someone else's fail-open.
Your remedy is what I took: the arm now calls fw_manager.remove_firewall_rules directly. A successful -X clears chain_created so the drop is a no-op; a failed one leaves it set so the drop is the retry it is there to be. It also knows the pinned veth, which a fresh manager does not.
While fixing it I named Drop's condition needs_teardown and split the ownership update into apply_teardown_outcome, because neither had an oracle. Every iptables command fails at spawn on a non-Linux host, so no test on this branch could previously reach a successful -X to observe ownership being released at all.
Mutation-verified: never clearing ownership fails only the new drop test, and the negative control -- a chain whose delete failed must stay owned so the drop retries -- stays green, so "always clear" cannot pass either. 115 lxc_common tests.
Scope note on what I did not claim: the end-to-end sequence (successful -X clears the flag, drop then no-ops) still needs Linux with real iptables. What is covered here is the ownership rule the no-op depends on, plus teardown_chain deciding relinquishment only on a successful -X.
There was a problem hiding this comment.
Correct, and fixed in 5449775.
I verified the premise before taking the remedy. impl Drop (network_iptables.rs) tears down whenever chain_created is set, and the error arm cleaned up through a fresh manager, which has no way to clear the owning manager's flag. So the function returned straight into a second teardown, exactly as described. The window you name is real: the chain name is deterministic, so a start that wins the gap between the two teardowns has its live firewall stripped by the second one.
Taken as written — the error arm now calls fw_manager.remove_firewall_rules(logger), so a successful -X clears the owner's own flag and Drop becomes a no-op.
Two things I had to add to make that testable, because neither had an oracle before. Drop's condition was inline, so no test could ask whether it would tear down; it is now needs_teardown(). And the ownership update after a teardown was buried in remove_firewall_rules, so it is now apply_teardown_outcome(bool). Both are named purely so the sequence has an observable, and both are mutation-verified with negative controls — an implementation that always clears ownership, or never does, fails.
What I want to be straight about is the limit of that verification. On this Windows box every iptables command fails at spawn, so the one sequence that matters end-to-end — -X succeeds, chain_created clears, Drop no-ops — is unreachable in a test here. The seams pin the decision logic; they do not prove the real path. That needs a Linux host and is not covered.
Two fail-open paths a reviewer found, both reachable in the teardown code this PR added, and both leaving a live container with no egress policy. signal_cleanup: the rollback ordering fix in f8662d7 stopped the container before removing its firewall, but discarded the stop result and removed the firewall anyway. When `lxc-stop` fails, the order it ran in no longer matters -- the chain comes off a container that is still up. `execute_rollback` now abandons the rest of the plan when the stop fails. Leaking the chain is the right trade and the one the ordinary stop path already makes deliberately: it propagates the stop error and leaves the rules in place rather than unfilter a still-running container (state_aware.rs:576-584). The gate is specific to a failed stop -- nothing precedes RemoveFirewall in a destroy plan, so gating on any failure would break that path instead. network_iptables: `remove_forward_hooks` (added by 5c03c13) is best-effort, and the `-F` after it ran regardless. Flushing is the dangerous half of teardown: `-F` succeeds whether or not anything still jumps to the chain, and an emptied user chain returns to its caller instead of reaching its own closing DROP, so flushing a still-hooked chain converts a filtered container into an unfiltered one with no error raised. Deleting is the safe half -- `-X` on a referenced chain just fails and the chain stays owned and filtering. `remove_forward_hooks` now re-reads FORWARD and reports whether any jump remains, and that verdict gates the flush. An unreadable FORWARD answers "still hooked": being wrong that way costs a retry, the other way unfilters a live container. Both fixes are behind testable seams -- `execute_rollback` takes the step runner, `teardown_chain` takes the command runner -- because every iptables command fails at spawn on a non-Linux host and `run_watchdog` is Linux-only, so neither ordering could otherwise be observed from this branch's suite. Mutation-verified: dropping the stop gate fails only the stop test, and flushing regardless of hooks fails only the hook test. Both keep a negative control so "never proceed" cannot pass. 113 lxc_common tests, 7 mxc_pty.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 29 changed files in this pull request and generated 1 comment.
Suppressed comments (7)
tests/scripts/run_lxc_state_aware_test.sh:150
- If this deprovision fails,
checkrecords the failure but returns success, andSANDBOX_IDis then cleared unconditionally. The EXIT trap therefore cannot retry cleanup, leaving exactly the container this guard is intended to reclaim. Capture the phase status and clear the ID only after a successful deprovision.
run_phase deprovision "$SANDBOX_ID"
check "deprovision exits 0" $?
# Claimed by this call; stop the trap from deprovisioning a second time.
SANDBOX_ID=""
sdk/node/src/state-aware-types.ts:110
- This public field's ownership semantics are hazardous but undocumented here: supplying an existing name adopts that container, and a later
deprovisionSandboxdestroys it unconditionally. Add that warning to the SDK type so IDE/API consumers do not mistake this for a harmless label.
/** Optional externally assigned LXC container name. */
containerId?: string;
sdk/node/src/state-aware.ts:86
- This generic API does not always receive dispatch errors on stderr. The Windows
wxcentry point writes state-aware error envelopes to stdout, while only the LXC executor routes non-dry-run exec failures to stderr. Describe the output as backend-dependent (or simply as raw PTY output) so Windows/IsolationSession consumers are not directed to the wrong channel.
* On dispatch failure the executor emits a single error envelope on stderr,
* because stdout is carrying the container's raw output; the SDK does not
* parse it here — callers consuming `IPty.onData` see the raw bytes. Use
* `execInSandboxAsync` when typed-error throwing is needed.
sdk/node/src/state-aware-types.ts:114
- These fields are required only when a config object is supplied, but
provisionSandboxstill declaresconfig?: ProvisionConfigFor<C>instate-aware.ts. ConsequentlyprovisionSandbox('lxc')type-checks, emits noexperimental.lxc.provision, and is rejected by Rust at runtime. Make the argument conditionally required for LXC (for example with overloads or a conditional rest tuple) and add a compile-time test.
/** Linux distribution for the container rootfs, e.g. "alpine" or "ubuntu". */
distribution: string;
/** Distribution release version, e.g. "3.20" or "24.04". */
release: string;
tests/scripts/run_lxc_state_aware_test.sh:51
- An INT/TERM trap replaces Bash's default terminating action; after
cleanupreturns, the script can resume instead of exiting as the comment assumes. It then continues with a deleted work directory and deprovisioned sandbox. Keep cleanup on EXIT, but make the signal handlers exit explicitly.
This issue also appears on line 147 of the same file.
trap cleanup EXIT INT TERM
docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md:1626
- This says explicit
defaultPolicy: "allow"leaves the presence bit unset, but the parser setsdefault_network_policy_present = truefor every explicit value, and the new parser test asserts that behavior. Allow is unaffected because it is permissive, not because presence is false.
> supplies no `network.defaultPolicy`, or supplies `defaultPolicy: "allow"`, does not
> set the presence bit and is unaffected. To start a default-deny container, set
sdk/node/src/state-aware-types.ts:148
- This public API comment reverses the enforcement order: the backend installs the iptables chain and hook before
container.start()specifically to avoid an unfiltered window. Documenting post-start application misstates the security behavior.
/** iptables policy to apply after the container starts. `proxy` is not supported by this backend. */
| )); | ||
| } | ||
| } else { | ||
| apply_filesystem_policy(&container, request, &mut logger)?; |
There was a problem hiding this comment.
Confirmed, and worth recording that this window was opened by an earlier fix on this same line.
The ordering in start is as described. apply_filesystem_policy runs at line 482 and mutates the shared container config, and the only cross-process contention check is the chain-ownership test inside apply_network_policy at 496 -- see the comment at 497-501, which already reasons about "another start owns the chain". So the contention check exists, but it sits after the config has been cleared and rewritten. Two concurrent starts can interleave inside that gap.
The clear-then-rewrite is not incidental: it came from f84cf2b, which added LxcContainer::clear_config_item to fix mount-list accumulation that MGudgin raised at this same line. That fix is correct for the single-process case and is what creates the multi-process window, so reverting it is not the answer.
Also note container.is_running() at 475 is itself part of the race, not just the config write -- it is a plain check with no lock held, so the already-running rejection at 477 can pass in both processes.
I am not implementing the lock in this PR. It needs to serialize the is_running check, the config mutation, the firewall install, and lxc-start under one cross-process lock keyed on the container, and it has to interact correctly with the existing signal_cleanup rollback and with teardown -- a lock acquired on the start path that stop and deprovision do not honor would give false confidence. That is a design change to the lifecycle, not a local edit, and doing it at the end of a session is how a correctness fix becomes a deadlock. Filing it rather than rushing it.
Leaving this thread open deliberately.
The failed-apply arm of the start path cleaned up through `cleanup_network`, which builds a fresh `NetworkIptablesManager`. A fresh manager cannot clear the owning manager's `chain_created`, so the function returned into its own `Drop` and tore the same chain down a second time. Between those two teardowns another start can create the same deterministic chain name -- it is derived from the container name -- and install its rules. The trailing teardown then removes hooks by chain name and deletes the chain, so this process's failed start strips a different, live container's firewall and leaves it running unfiltered. The arm now calls `fw_manager.remove_firewall_rules` directly. A successful `-X` clears `chain_created`, so the drop is a no-op; a failed one leaves it set, so the drop is the retry it exists to be. The owning manager also knows the pinned veth, which a fresh one does not. Named `Drop`'s condition as `needs_teardown` and split the ownership update into `apply_teardown_outcome` so both have an oracle. Neither was reachable from a test before: every iptables command fails at spawn on a non-Linux host, so no test could reach a successful `-X` to observe ownership being released. Mutation-verified: never clearing ownership fails only the new drop test, and the negative control -- a chain whose delete failed must stay owned so the drop retries -- stays green, so "always clear" cannot pass either. 115 tests.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 29 changed files in this pull request and generated 1 comment.
Suppressed comments (6)
src/backends/lxc/common/src/state_aware.rs:600
- A concurrent
startcan be between firewall installation andcontainer.start()here. This process then observesis_running() == false, skips stopping, deletes the firewall, and returns; the racing start can subsequently complete and report success with the container running unfiltered. Serializestart/stop/deprovisionwith a cross-process per-container lock so the running-state check and firewall teardown cannot race lifecycle transitions.
container
.stop()
.map_err(|e| MxcError::backend_error(format!("Failed to stop container: {e}")))?;
}
cleanup_network(container_name, veth.as_deref(), &mut logger);
src/backends/lxc/common/src/signal_cleanup.rs:322
- The state-aware rollback is registered before firewall installation, so a signal can arrive after
iptables -Nbut beforelxc-start. In that window the container is already stopped;lxc-stopcan fail, causingexecute_rollbackto abort beforeRemoveFirewalland leave the chain behind. Make the rollback distinguish an already-stopped pre-start container from a failed attempt to stop a running one, while preserving the fail-closed behavior for an in-progress start.
RollbackStep::StopContainer => LxcContainer::new(&name, None).stop().is_ok(),
tests/scripts/run_lxc_state_aware_test.sh:51
- Bash does not automatically exit after an explicit
INTorTERMtrap returns, so cancellation runs cleanup and then continues the remaining phases against the deprovisioned sandbox. Use signal-specific traps that clean up and exit, leaving the EXIT trap guarded as intended.
trap cleanup EXIT INT TERM
docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md:1625
- This contradicts the parser and the test above: an explicit
defaultPolicy: "allow"does setdefault_network_policy_present; it is unaffected only becauserequires_firewall_enforcementchecks that the value isBlock. Correct the explanation so callers do not rely on the wrong presence semantics.
> supplies no `network.defaultPolicy`, or supplies `defaultPolicy: "allow"`, does not
> set the presence bit and is unaffected. To start a default-deny container, set
sdk/node/src/state-aware-types.ts:148
- The policy is installed before
container.start()to avoid an unfiltered startup window, not afterward. This public type documentation currently reverses the security-relevant ordering.
/** iptables policy to apply after the container starts. `proxy` is not supported by this backend. */
sdk/node/src/state-aware-helper.ts:16
- Making
containerIdglobally cross-cutting also lifts it for Windows Sandbox, IsolationSession, and non-provision phases. JavaScript callers can therefore send a field that the public contract says is LXC-provision-only, and the Rust envelope accepts it at top level where those backends silently ignore it; previously it remained in the phase config and was rejected. Lift this field only whenbackendKey === 'lxc' && phase === 'provision'(and add a cross-backend regression test).
export const CROSS_CUTTING_FIELDS = ['containerId', 'filesystem', 'network', 'ui', 'process'] as const;
| NetworkConfig, | ||
| 'defaultPolicy' | 'allowedHosts' | 'blockedHosts' | ||
| > & { | ||
| /** LXC enforces network policy only via iptables. */ | ||
| enforcementMode?: 'firewall' | 'both'; |
There was a problem hiding this comment.
Fixed in 5adda93, though not quite as suggested -- the proposed rule would have over-rejected.
Verified the Rust side first rather than taking the claim on trust: requires_firewall_enforcement (src/backends/lxc/common/src/state_aware.rs:166) counts exactly three things as a restriction -- a non-empty allowedHosts, a non-empty blockedHosts, or an explicit defaultPolicy: "block". An explicit defaultPolicy: "allow" and an omitted defaultPolicy are deliberately not restrictions, and the comment above that function calls out why: the absent case is the plain start in run_lxc_state_aware_test.sh, which must not be rejected. So "requires a firewall mode whenever restrictive fields are present" would have broken a start that works today.
LxcNetworkConfig is now a union of two arms that mirror that function:
LxcRestrictedNetworkConfig--enforcementModeis mandatory, and the restrictive fields are allowed.LxcUnrestrictedNetworkConfig--enforcementModeis optional,defaultPolicynarrows to'allow', andallowedHosts/blockedHostsarenever, so adding one moves the value to the arm that requires the mode.
{ defaultPolicy: 'block' }, { allowedHosts: [...] }, and { blockedHosts: [...] } no longer compile; { defaultPolicy: 'allow' } and {} still do.
Covered by three @ts-expect-error cases and two positive cases in state-aware-types.test.ts. Those markers are load-bearing in both directions: the test build fails if the restrictive cases stop erroring, and it also fails if the permissive cases start erroring, so the over-rejection is guarded too. 232 passing, 0 failing.
teardown_created already computes, per family, whether the FORWARD hook delete succeeded -- but the flush and delete that follow ignored it. -F succeeds regardless of who references the chain, and an emptied user chain returns to its caller instead of reaching its own closing DROP, so flushing a still-hooked chain unfilters a container that may still be running. The -X would have failed anyway, since iptables refuses to delete a referenced chain, so the flush bought nothing and cost the container its filtering. Gate the whole step -- flush included -- on that family's hook being confirmed gone, and keep the chain published so a later pass retries. The gate is per family because the two chains live in different tables and are referenced independently. This is inherited from main, which flushes unconditionally, but this branch rewrote the block and doubled the exposure by adding a second address family. microsoft#632 and microsoft#633 fix the same fail-open in the same file; teardown_chain is deliberately identical to the one microsoft#632 landed, so whichever merges second resolves to a no-op.
…2830559) (#724) * [LXC] Filter IPv6 destinations and CIDR ranges in firewall mode (AB#62830559) Firewall mode resolved `allowedHosts` / `blockedHosts` to IPv4 only. On a dual-stack host, traffic to the same destination over IPv6 bypassed the firewall entirely, and any CIDR entry (v4 or v6) failed to parse as an address, then failed DNS resolution, and was dropped. Changes, all confined to the LXC backend: - `resolve_host` returns IPv4 and IPv6 destinations separately. Hostnames resolve to both A and AAAA records; bare literals and validated CIDR blocks pass through in their own family. - `destination_family` validates CIDR syntax and prefix length (<=32 for IPv4, <=128 for IPv6). Malformed entries are reported as unresolved and skipped rather than handed to iptables, which would reject them at apply time and abort setup for the whole policy. - IPv4 rules go to `iptables`, IPv6 rules to `ip6tables`, with parallel per-container chains and FORWARD hooks. - `ip6tables` is probed once. When it is missing or IPv6 is disabled in the kernel, the IPv4 chain is still applied and the number of unapplied IPv6 rules is logged, instead of failing a policy that worked before dual-stack support. - Setup failures after partial chain creation are rolled back, and teardown removes both families' hooks and chains. Scope: this covers the IPv6 + CIDR item of AB#62830559 only. Port and protocol filtering are not included -- they require structured egress rules in the config schema (AB#62830582), which is not in main. Tests: 8 new unit tests for family routing, CIDR pass-through, prefix and syntax rejection, and allow/block ordering; 2 integration configs and scripts wired into run_lxc_all_tests.sh. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea24fae3-d643-4f19-a701-9e31b9f950fd * [LXC] Add spec-derived unit and E2E tests for IPv6/CIDR filtering (AB#62830559) Tests were written black-box from roadmap item 19, AB#62830559 and the public doc comments, without reading network_iptables.rs, so they pin the specified contract rather than the current implementation. Unit tests (24 new, in two child modules of network_iptables): resolution/CIDR contract - family routing, CIDR passthrough, host bits not required to be zero, prefix bounds at 0/32 and 0/128, malformed syntax, IPv4-mapped IPv6, dual-stack hostname resolution; and rule generation - per-family bucketing, ACCEPT/DROP mapping, allow-before-block ordering in both families, family-agnostic base rules, chain-name cap. E2E: lxc_network_dualstack_hostname covers hostnames with both A and AAAA records (the bypass this work item fixes) alongside mixed-family literals and CIDRs; lxc_network_cidr_boundary covers /0, /32, /128, non-zero host bits and the previously untested defaultPolicy=allow path. Both wired into run_lxc_all_tests.sh. No change to wire.rs, models.rs, config_parser.rs, schemas/ or sdk/. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea24fae3-d643-4f19-a701-9e31b9f950fd * [LXC] Close a false negative in the IPv6 DNS resolution tests (AB#62830559) Mutation testing showed that inverting the DNS branch so AAAA records are pushed into the IPv4 bucket - the exact dual-stack bypass this work item fixes - left the suite green. The only hostname test used localhost, which resolves to 127.0.0.1 only on many hosts, so the v6 arm of the DNS path was never executed. Adds a family-purity invariant asserting every destination in a bucket belongs to that bucket's family, exercised over well-known dual-stack names. It now kills that mutation. All 9 mutations tried against the module are caught. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea24fae3-d643-4f19-a701-9e31b9f950fd * [LXC] Restore the quarantined CIDR prefix test instead of relaxing it (AB#62830559) A spec-derived test asserting that '10.0.0.0/+24' is rejected was failing. It was rewritten to assert the current behaviour instead of being left as a finding, which is the wrong resolution: whether MXC should accept a permissive prefix spelling in a security policy file is a design decision, not something to settle by editing the test. The original assertion is restored verbatim and marked #[ignore] so the finding stays visible in test output pending a decision. The separate assertion that a leading '+' cannot smuggle an out-of-range prefix past the family bound check is kept as a passing test, since prefix bounds are unambiguous. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea24fae3-d643-4f19-a701-9e31b9f950fd * [LXC] Fix empty-host and plus-prefix destination parsing (AB#62830559) Two defects found by a coverage audit of this branch, both caught by spec-derived tests written black-box against the roadmap contract. resolve_host("") fell through to DNS resolution, where format!("{}:0", host) produces ":0". Winsock resolves that to every local interface address, so an empty allowedHosts entry emitted rules for the host's own LAN and link-local addresses. glibc rejects it, so this reproduced only on Windows -- it turned CI red on windows/x64 and windows/arm64. config_parser assigns host lists verbatim, so an empty string does reach resolve_host from a policy file. destination_family validated the CIDR prefix with u8::from_str, which accepts a leading '+'. 10.0.0.0/+24 was forwarded to iptables, which silently canonicalizes it to 10.0.0.0/24, so a policy typo was applied instead of being reported by the unresolved-host warning that run_lxc_network_invalid_cidr_test.sh exists to guarantee. The prefix must now be ASCII digits, which also subsumes the embedded-slash case. The test for this was previously quarantined pending a bad-code/bad-test ruling; the ruling is bad code, so it is now un-ignored. Also adds lifecycle tests pinning three behaviours a cargo-mutants run proved were unpinned: a new manager reports no rules applied, a non-firewall enforcement mode is a successful no-op, and the enforcement-mode gate is not inverted. The last matters most -- an inverted gate would silently skip all filtering while reporting success. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea24fae3-d643-4f19-a701-9e31b9f950fd * [LXC] Make the four network E2E scripts actually run (AB#62830559) None of these four scripts had ever executed a single firewall assertion since they were added. Two independent causes: - every config declared "version": "0.4.0-alpha", but the parser accepts >=0.6 <=0.8, so each run died at config parse - lxc-exec buffers diagnostics unless --debug is passed, so the log lines the scripts assert on were never emitted even after the version bump Bumps the configs to 0.6.0-alpha, matching the sibling LXC configs, passes --debug, and adds post-run iptables/ip6tables assertions that the per-container chain is torn down rather than leaked. Verified by running all four as root under WSL: each creates a real container, programs real v4/v6 chains, and cleans up. Assertion liveness was confirmed by flipping defaultPolicy in a config and observing exit 1 with "FAIL: default-deny policy was not applied." Also normalizes lxc_network_ipv6_cidr.json to LF; it was the only one of the four committed with CRLF. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea24fae3-d643-4f19-a701-9e31b9f950fd * [LXC] Harden iptables enforcement backend (AB#62830559) Address four PR #724 review threads that are an interwoven refactor of the same enforcement path in network_iptables.rs: - Resolve each allow/block destination exactly once. The apply path previously resolved a host for the unresolved-host warning and then a second time inside rule construction; two lookups of the same name can disagree under DNS round-robin or a TTL expiry, so the installed rule need not match the logged one. build_policy_rules_logged now resolves once and reuses that result for both. The pure builders that resolve are gated to test-only. - Extract enforcement_mode_uses_firewall as a pure predicate and test it directly, instead of the lifecycle test invoking apply_firewall_rules (which shells out to the host firewall) for the Firewall and Both cases. - Fail closed when IPv6 is active but ip6tables is unusable. The old boolean probe skipped IPv6 for every ip6tables failure, marking the policy applied while IPv6 egress went unfiltered. classify_ip6tables_status now distinguishes a kernel with no active IPv6 (safe to skip) from an IPv6-capable host whose ip6tables is missing or broken (setup fails). host_has_active_ipv6 reads /proc/net/if_inet6, which the kernel populates only when the IPv6 stack is loaded and addresses exist. - Track which per-family chains and FORWARD hooks each attempt created and roll back only those. Rollback previously tore down chains unconditionally, and since chain names truncate at 20 characters a partial-failure rollback could delete a chain belonging to a different container. teardown_created acts on the recorded CreatedResources. Also log a positive confirmation when a FORWARD hook is installed, so the E2E scripts can assert on it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [LXC] Query iptables tables directly in E2E cleanup probes (AB#62830559) The four LXC network E2E scripts probed for a leftover chain with `sudo -n iptables -S`. Under `sudo -n`, a host without passwordless sudo fails the probe for a reason unrelated to whether the chain exists, so the cleanup assertion could pass without ever having checked. The LXC suite already requires root (run_lxc_all_tests.sh), so query iptables and ip6tables directly instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [LXC] Assert the FORWARD hook was installed in E2E scripts (AB#62830559) All four LXC network E2E scripts could report PASS while the per-container chain was never hooked into FORWARD: the code emits a skipped-hook warning that nothing checked, so an undiscovered veth silently enforced nothing. Each script now fails on the "Skipping FORWARD hook" warning and requires the positive "FORWARD hook installed" confirmation before reporting PASS. This pairs with the confirmation log line added to the enforcement backend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [LXC] Remove external network dependency from CIDR boundary test (AB#62830559) The boundary fixture ran `wget -qO- https://api.github.com/zen`, so the test failed whenever the network or the remote host was down, independent of the code under test. Replace it with the local success command `true` so a non-zero lxc-exec status reflects a firewall-setup failure on the boundary-valid prefixes rather than an unrelated outage, and note that in the script's status check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [LXC] Bind rulegen ordering tests to the shipping policy-rule path The #[cfg(test)] build_policy_rule_args reimplemented allow-before-block ordering as two separate loops, so the rulegen ordering specs asserted against a duplicate that production never runs. A future change to emission order in build_policy_rules_logged (the AB#62830341 deny-precedence work) would have left those ordering tests green while shipping first-match-wins. Make build_policy_rule_args a thin test-only shim that delegates to the shipping build_policy_rules_logged with a throwaway buffer logger, so the ordering assertions bind to production code again. No test cases added or changed. Move the deny-precedence contract docstring onto the shipping function it now guards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add exhaustive black-box unit tests for classify_ip6tables_status Closes the testability gap flagged in the type doc comment: the function was extracted to be pure so the fail-open vs fail-closed decision could be unit-tested without a privileged Linux host, but had zero tests. Nine tests, all derived from the documented contract only: - Exhaustive 4-case truth table (both boolean inputs x all combinations) - Invariant: working probe always yields Available - Invariant: failed probe never yields Available - Security invariant: UnusableButIpv6Active is reachable only when probe=false AND ipv6_active=true; unreachable under every other input - Invariant: KernelIpv6Disabled reachable only when probe=false AND ipv6_active=false - Discriminant-distinctness: all three variants are distinct under PartialEq All 6 mutations (swap outcomes, fail-open collapse, fail-closed collapse, probe invert, ipv6_active invert, always-Available) are caught. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * [LXC] Make IPv6 network tests fail when behavior is removed (PR 724 review) Address test-honesty defects found by evidence-based review; every fix is proven with mutation testing (before: mutant survives; after: mutant caught). Task 1 -- DNS bucket test was vacuous. The family split is factored into a pure `bucket_resolved_addrs` and the AAAA-in-v6-bucket test now injects addresses and asserts the v6 bucket is non-empty and family-pure, so deleting the IPv6 result path fails it. A separate live characterization asserts only the purity invariant, with no warning-that-still-passes. Task 2 -- failure to read IPv6 state was treated as confirmed inactivity. Factored the parse/classify into pure `classify_host_ipv6_state` (file content and read-error as input) and `ipv6_state_treated_as_active`. A NotFound read (IPv6 disabled) stays a confirmed negative; any other read error is Unknown and treated as active so it fails closed instead of leaving IPv6 egress unfiltered. Loopback-only `::1` on `lo` no longer counts as active. Added spec tests for the whole mapping (previously untested). Task 3 -- E2E scripts could count skips as passes. The aggregate runner now treats exit 77 as SKIPPED (never PASS) and flags a run that executed nothing; each script honestly skips on missing root/iptables/ip6tables/LXC/binary. Added assertions on the actual programmed destination rules (via a new per-rule debug log) so deleting destination-rule emission fails the scripts, and the dual-stack script now asserts a positive IPv6 rule exists, including a hostname-derived AAAA rule when external DNS is available. Task 4 -- corrected docs to describe the three-way ip6tables classification (Available / KernelIpv6Disabled / UnusableButIpv6Active) and that an active host with unusable ip6tables fails setup rather than skipping IPv6. Runtime behavior of the shell E2E scripts is UNVERIFIED here (Windows; no root/iptables/netns); scripts were syntax-checked with bash -n only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback on IPv6/CIDR egress filtering Signal-time cleanup now removes only what it created ---------------------------------------------------- `force_cleanup` runs on the watchdog thread and had no access to the manager's `CreatedResources`, so it hardcoded `v4_chain: true` and `v6_chain: true`. On a signal that arrived before the chains existed it issued `-F`/`-X` against names it did not own, and because chain names truncate at 20 characters that name can belong to a different container. `CreatedResources` now lives in `signal_cleanup::ActiveSandbox` alongside the container name and veth, behind the same mutex, so the watchdog takes one coherent snapshot and can never pair one container's identity with another's ownership record. Every creation site publishes incrementally, so a signal arriving mid-apply still sees the half-built set instead of an empty one. `force_cleanup` takes the record as a parameter and returns immediately when it is empty, running zero iptables commands. `teardown_created` now returns the residual set — ownership bits are cleared only when the removal command actually succeeds — so a failed removal is not recorded as a completed one. IPv4-mapped IPv6 destinations no longer fail open ------------------------------------------------- A dual-stack socket sending to `::ffff:1.2.3.4` makes Linux emit a real IPv4 packet, so an `ip6tables -d ::ffff:1.2.3.4` rule never matches and a mapped `blockedHosts` entry silently failed open under an allow default policy. Mapped literals and mapped CIDRs are now rewritten to their IPv4 form and filed into the IPv4 bucket. Prefixes shorter than /96 span outside the mapped range and are deliberately left as IPv6. An unreadable /proc is no longer a confirmed "IPv6 is off" ---------------------------------------------------------- `NotFound` on `/proc/net/if_inet6` now maps to `Inactive` only when `/proc/net` itself exists. In a mount namespace without `/proc` the answer is `Unknown`, which fails closed. Review housekeeping ------------------- Configs added by this change now declare `0.8.0-alpha`, matching `CURRENT_SCHEMA_VERSION`. This selects an existing schema version; it does not modify any schema. The five `#[path]` spec-test files are folded into the inline `mod tests` in `network_iptables.rs`, matching the repo convention (123 inline test modules against 5 `#[path]` files, all five of which this change added). The enforcement-mode test asserted the production predicate against a second copy of the same predicate, so both could be wrong together. It now asserts literal expected values. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52fb9e8c-4a48-435c-9964-8ec0fee653c6 * Test the signal-time ownership guard rather than its predicate The existing test asserted that an empty CreatedResources reports itself as empty, which is a property of the record and not of the code that consults it. Deleting the guard in force_cleanup left every test passing while restoring the exact failure the record was added to prevent: a process that created nothing tearing down the chain of a concurrent start that did. The new test drives force_cleanup itself and reads the log as the observable, since the teardown announces the chain by name before it touches anything. A positive control with one resource published proves the assertion discriminates between the two cases rather than watching a permanently silent function. Verified by mutation: with the guard deleted the test fails on the expected assertion, and it passes with the guard restored. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52fb9e8c-4a48-435c-9964-8ec0fee653c6 * Keep ownership of iptables state a failed rollback could not remove apply_firewall_rules_inner rolled back what it created, but discarded the result of that rollback with `let _ =`. A removal command can itself fail, so the outer error arm then left self.created empty and rules_applied false while a chain or FORWARD hook was still installed. rules_applied gates both remove_firewall_rules and Drop, so nothing afterward knew the survivors were ours and the leak was permanent for the life of the process. The inner call now returns the residual alongside the error, and the outer arm adopts it: ownership is retained exactly when something survived, and the two cases log differently so the distinction is visible in a failure report. The signal path was already covered, because teardown_created publishes the residual before returning; this closes the ordinary Drop and remove path. Mutation-tested: discarding the residual again makes the new test fail on the retention assertion rather than passing quietly. The test drives remove_firewall_rules and observes the log rather than asserting rules_applied, and carries a negative control so it cannot pass by retaining unconditionally. Not verified: no live iptables. On this Windows host every firewall command fails, so the residual path is exercised through the ownership record and the logger rather than against real rule state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52fb9e8c-4a48-435c-9964-8ec0fee653c6 * Keep ownership when the removal path's own commands fail The previous commit fixed this for the failed-apply rollback but left the same defect on the ordinary removal path, and the reviewer caught it there. remove_firewall_rules cleared rules_applied unconditionally, even when teardown_created reported a non-empty residual. Since Drop is gated on that same flag, a teardown whose commands failed reported itself done and threw away the last retry, leaving the chain installed for the life of the process. Both paths now share retain_residual_ownership, which keeps the gate open exactly when something survived. They have the same obligation, so having one of them get it right and the other not was the underlying problem. Mutation-tested: clearing the flag unconditionally again makes the new test fail. The test drives a second removal -- the call Drop makes -- and observes the log, because a closed gate short-circuits before the teardown announces the chain. Not verified: no live iptables. On this Windows host every firewall command fails, which is what makes the non-empty residual reachable in a test at all. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52fb9e8c-4a48-435c-9964-8ec0fee653c6 * Refuse a second apply while the first still owns firewall state An independent review of the previous two commits found that both arms of `apply_firewall_rules` replace `self.created` with the current attempt's set. A manager that already owned resources and was asked to apply again would therefore drop the earlier record. If the second attempt then failed before creating anything, `retain_residual_ownership` would overwrite the record with an empty set and clear `rules_applied`, so `Drop` skipped cleanup and whatever the first attempt left behind was stranded permanently. The same empty record was published to the signal registry, so the watchdog lost it too. Every production caller builds a manager immediately before its single apply, so this is not reachable today. That is exactly why it is worth closing now: the invariant is currently held by convention at four call sites rather than by the type, and nothing tells the next caller. Refusing the second apply makes it unreachable by construction instead. The guard keys on live ownership rather than on having ever applied, so a manager whose removal succeeded can still be reused. The negative control test covers that: a fresh manager must reach its commands rather than trip the gate. Mutation-verified. Weakening the guard to `self.rules_applied && false` fails `a_second_apply_is_refused_while_the_first_still_owns_resources` rather than passing quietly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52fb9e8c-4a48-435c-9964-8ec0fee653c6 * [LXC] Test the arm that adopts a failed rollback's residual The existing test for residual ownership starts from a residual that has already been retained: it calls retain_residual_ownership itself and then checks teardown runs. That covers the mechanism but not the decision. The failure arm of apply_firewall_rules could discard the residual it was handed and the test would still pass, because it never goes through that arm. That branch is also the one least likely to be reached by accident. On any host without iptables the inner apply fails on its first command, so it rolls back nothing and reports an empty residual -- the interesting case needs a rollback whose own removal command failed, which no unit test can produce by running the real thing. Extracted the recording step as record_apply_outcome so a test can hand it exactly that outcome, and added a test that drives it and then asserts the downstream teardown still names the chain. A negative control covers the clean-failure side, so the assertion cannot be satisfied by retaining unconditionally. Mutation-verified, and the mutation is what makes the point: making the failure arm drop the residual kills the new test and leaves the old one green. 112 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 52fb9e8c-4a48-435c-9964-8ec0fee653c6 * [LXC] Stop flushing a chain FORWARD still jumps to teardown_created already computes, per family, whether the FORWARD hook delete succeeded -- but the flush and delete that follow ignored it. -F succeeds regardless of who references the chain, and an emptied user chain returns to its caller instead of reaching its own closing DROP, so flushing a still-hooked chain unfilters a container that may still be running. The -X would have failed anyway, since iptables refuses to delete a referenced chain, so the flush bought nothing and cost the container its filtering. Gate the whole step -- flush included -- on that family's hook being confirmed gone, and keep the chain published so a later pass retries. The gate is per family because the two chains live in different tables and are referenced independently. This is inherited from main, which flushes unconditionally, but this branch rewrote the block and doubled the exposure by adding a second address family. #632 and #633 fix the same fail-open in the same file; teardown_chain is deliberately identical to the one #632 landed, so whichever merges second resolves to a no-op. * [LXC] Keep the firewall unit tests off the host's iptables The unit tests drove force_cleanup, remove_firewall_rules, and apply_firewall_rules all the way down to Command::new, so run as root they flushed and deleted whatever live chain answered to a colliding MXC-<name>. Chain names sanitize and truncate to 20 characters, so a collision with a running container is reachable rather than theoretical. Worse, the tests that need an iptables command to fail never arranged it. They inherited the failure from the host, which is why one of them carried the comment "on this host every iptables command fails" -- an outcome that reverses on a host where iptables works. Add an opt-in interception point in run_firewall_command, the single place where the argv is complete and the last one before the spawn, backed by thread-local storage because the firewall entry points are associated functions with no self to carry a runner and cargo test runs its threads in parallel. A test that installs no fake reaches the real binary exactly as before, so the ~70 tests in this file that never touch a firewall command are unaffected and the real-binary path is preserved for integration. This placement reaches Drop and force_cleanup, which a runner passed as a parameter cannot, and it adds no field to NetworkIptablesManager -- which matters because SandboxProcess requires Send and BwrapChild holds one. The six affected tests now assert the command sequence directly instead of scraping the log, and script their own failures. That also makes the -X success arm reachable for the first time: a chain whose delete succeeds is released and Drop finds nothing to retry. The negative controls keep their empty-log assertion alongside the new empty-argv one. The log assertion is load-bearing: with an empty ownership record no command is issued either way, so an argv-only assertion would not catch deletion of the is_empty() guard. Both were confirmed by mutation. ip6tables_status gets the same treatment, since its read-only probe is a second spawn that the apply path reaches before any chain exists. A fake always reports the tool available, which the classifier maps to Available without consulting the host; reporting it unusable would not be host-independent, so those branches stay covered by the pure-function tests of classify_ip6tables_status. Addresses review feedback on network_iptables.rs:1228. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7155573c-8938-4622-abf7-4594fb17eb3d --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Darren Hoehna <Darren.Hoehna@microsoft.com> Copilot-Session: ea24fae3-d643-4f19-a701-9e31b9f950fd Copilot-Session: 52fb9e8c-4a48-435c-9964-8ec0fee653c6 Copilot-Session: 7155573c-8938-4622-abf7-4594fb17eb3d
LxcNetworkConfig left enforcementMode optional, so `{ defaultPolicy: 'block' }`
and `{ allowedHosts: [...] }` type checked and then failed at run time. Rust
treats an omitted mode as Capabilities, and LXC has no capability-based network
enforcement, so start rejects a restriction it cannot deliver. The doc comment
on the type already claimed those configurations were caught at compile time;
they were not.
Split the type into a restricted arm, where enforcementMode is mandatory, and
an unrestricted arm, where the restrictive fields are `never`. Adding a
restriction now moves the value to the arm that requires the mode.
The split mirrors requires_firewall_enforcement in state_aware.rs rather than
the looser "any restrictive field" rule: an explicit `defaultPolicy: 'allow'`
and a policy-free start are not restrictions, and rejecting them would have
broken the plain start in run_lxc_state_aware_test.sh.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 29 changed files in this pull request and generated no new comments.
Suppressed comments (6)
src/backends/lxc/common/src/state_aware.rs:306
- A single adopted interface is not necessarily
lxc.net.0. For example, a config containing onlylxc.net.1.typepasses both thelen() > 1and empty checks, but the code below pinslxc.net.0.veth.pairand hooks that deterministic name. Traffic on the realnet.1veth then bypasses the firewall while start reports success. Require the sole index to be exactly 0 (or pin/hook the actual index).
if net.indices.len() > 1 {
src/backends/lxc/common/src/state_aware.rs:600
- This cleanup is racy with
start: another process can install the firewall, then pause beforecontainer.start(). A concurrentstopobservesis_running() == false, skipslxc-stop, and removes that firewall here; the first process then starts the container unfiltered. The process-localchain_createdbit cannot protect cross-process stop/start. Serialize start, stop, and deprovision with a per-container cross-process lock held across firewall installation/start and stop/cleanup.
cleanup_network(container_name, veth.as_deref(), &mut logger);
tests/scripts/run_lxc_state_aware_test.sh:51
- Trapping
INT/TERMwithcleanupsuppresses the signal's default termination; after the handler returns, Bash can continue the test withWORK_DIRalready deleted andCLEANED_UPset. The comment's assumption that the shell exits automatically is therefore false. KeepEXITcleanup, but explicitly exit with the conventional signal status from the signal traps.
trap cleanup EXIT INT TERM
docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md:1626
- This says explicit
defaultPolicy: "allow"does not set the presence bit, butconvert_wire_configsetsdefault_network_policy_present = truefor every present value, and the newnetwork_default_policy_allow_sets_presence_truetest asserts exactly that. Clarify thatallowsets the bit but does not trigger firewall enforcement.
> supplies no `network.defaultPolicy`, or supplies `defaultPolicy: "allow"`, does not
> set the presence bit and is unaffected. To start a default-deny container, set
> `enforcementMode` to `"firewall"` or `"both"`.
sdk/node/src/state-aware-types.ts:185
- This public type documentation reverses the security-critical ordering: the implementation installs the iptables policy before
container.start()specifically to avoid an unfiltered window. Saying it is applied after start is inaccurate.
/** iptables policy to apply after the container starts. `proxy` is not supported by this backend. */
sdk/node/README.md:235
- The claimed LXC SDK restriction is not enforced.
StateAwareContainmentBackendnow includeslxc, soexecInSandbox(SandboxId<'lxc'>, ...)type-checks and spawnslxc-exec; the LXCattach_runpath forwards the inner PTY to executor stdout whilebackend.execis blocked, so the returned NodeIPtycan receive live output. The missingSandboxProcessaffects the direct engine/FFI streaming entry point, not this CLI-backed Node API. Either document Node streaming as supported, or explicitly reject LXC inexecInSandboxif it must remain unavailable pending #765.
> **Backend support:** the state-aware lifecycle is currently implemented for `isolation_session` and `windows_sandbox` (both Windows-only; both still experimental, so every call must pass `{ experimental: true }`) and `lxc` (Linux-only; not experimental). Streaming exec — `execInSandbox`, which returns an `IPty` — is not available for `lxc`; use the non-streaming `execInSandboxAsync` instead. The one-shot spawn APIs (`spawnSandbox` / `spawnSandboxFromConfig`) are the supported path for every other backend.
…-aware-lifecycle Resolves two conflicts in the LXC backend, and the ownership question the merge exposed between microsoft#724's teardown model and the state-aware lifecycle. network_iptables.rs -- took microsoft#724's dual-stack rewrite wholesale, then re-applied this branch's enumeration-based FORWARD hook removal on top of it. microsoft#724 removes hooks by replaying the remembered `-i <veth>` spec, gated on the veth being known. Signal-time cleanup never learns the veth, so nothing was deleted, the hook survived, teardown_chain then correctly refused to flush a still-referenced chain, and the chain leaked permanently because every later pass hit the same missing veth. Enumerating FORWARD and matching on the `-j <chain>` target removes the hook whether or not the veth is known, and is now family-generic. signal_cleanup.rs -- kept this branch's rollback plan, since microsoft#724's watchdog unconditionally destroys the container and would make set_active_network_only a lie: a signal during a state-aware start would destroy a container provisioned to outlive the process. Adopted microsoft#724's CreatedResources in place of the single chain_created bool, which it strictly supersedes. Teardown authority -- microsoft#724 tears down only what the calling process created, which is right for the one-shot runner but leaves state-aware stop and deprovision unable to remove anything: they run in a different process from the start that created the chain, so their record is empty. A record-gated teardown there would strand the chain and make the sandbox unstartable, since the next start fails on a chain it did not create. Both the start path's mem::forget comment and apply_network_policy's error text already name stop and deprovision as what removes this state, so they must actually remove it. Split the two authorities explicitly rather than widening the existing one: - cleanup_network_owned, backed by force_cleanup, removes only what the record names. Used by the failed-start path, which now receives the record from apply_network_policy instead of discarding it. - cleanup_network_authoritative, backed by force_cleanup_authoritative, removes whatever exists for the container. Used by stop and deprovision, which have already stopped or destroyed it, so nothing is left for the chain to protect. The authoritative path observes live iptables state instead of assuming an all-true record, so an already-clean container issues no commands and logs no failures rather than failing -X against chains that never existed and retaining residual ownership that schedules a pointless Drop retry. The concurrent-start guard is untouched: a start that loses the chain to another start still holds no ownership and still removes nothing. Verified: cargo check -p lxc_common --all-targets clean with no warnings, cargo test -p lxc_common 161 passed / 0 failed, sdk/node test:unit 232 passed / 0 failed.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 29 changed files in this pull request and generated 1 comment.
Suppressed comments (4)
src/backends/lxc/common/src/network_iptables_chainname_spec_tests.rs:14
- This new spec suite is not declared as a Rust module:
lib.rsonly declaresnetwork_iptables, andnetwork_iptables.rsnever includes this file. Cargo therefore does not compile or run any of these tests; once included, they would also expose the currently missingchain_name_formethod. Add a test-only module declaration so the claimed chain-name validation actually runs.
use super::*;
docs/state-aware-lifecycle/mxc-state-aware-sandbox-api.md:1625
- This is inconsistent with the parser and the new unit test: explicit
defaultPolicy: "allow"does setdefault_network_policy_present; it simply does not require firewall enforcement because the value isAllow. Please distinguish that from the omitted case so the documented presence semantics remain accurate.
> supplies no `network.defaultPolicy`, or supplies `defaultPolicy: "allow"`, does not
> set the presence bit and is unaffected. To start a default-deny container, set
sdk/node/src/state-aware-types.ts:185
- The implementation installs the iptables chain and hook before
container.start()to avoid an unfiltered startup window. This public type documentation says the opposite and obscures a security-relevant guarantee.
/** iptables policy to apply after the container starts. `proxy` is not supported by this backend. */
tests/scripts/run_lxc_state_aware_test.sh:51
- Trapped
INTandTERMsignals do not automatically terminate Bash after the handler returns. This trap runscleanupand then resumes the test with a deleted work directory and deprovisioned sandbox. Use separate signal traps that clean up and explicitly exit, while keepingEXITfor normal/early exits.
trap cleanup EXIT INT TERM
| pub fn deterministic_veth_name(container_name: &str) -> String { | ||
| format!("mxcv{}", Self::hash_token(container_name)) |
Gudge (MGudgin)
left a comment
There was a problem hiding this comment.
Approving. All five High findings from the adversarial review are resolved: egress FORWARD hook now filters container-originated traffic (-i), with veth-missing and non-firewall enforcement modes failing closed; firewall teardown ordering, PTY exec-timeout drain, mount-policy accumulation, and the mxc_engine dispatch delegation are all fixed, with tests added.
Note: one High remains open — provision() still returns an error on lxc-create failure without destroying partially-created on-disk state (orphaned auto-named container). Recommend a follow-up, but not blocking this approval.
Gudge (MGudgin)
left a comment
There was a problem hiding this comment.
Revoking my earlier approval and requesting changes.
The approval cited "tests added" as part of the evidence that the chain-name collision concern (the 32-bit-hash-truncation fix) was resolved. On closer inspection that verification is hollow: the dedicated collision-freedom spec suite never compiles or runs.
Findings:
-
src/backends/lxc/common/src/network_iptables_chainname_spec_tests.rs (1,035 lines) is not part of the module tree. lib.rs declares only
network_iptables, and network_iptables.rs never includes the file via mod / include! / #[path]. The file begins withuse super::*;, so it was intended to be an inline submodule of network_iptables, but nothing pulls it in. Cargo silently ignores unreferenced .rs files, so none of these tests execute. -
Even if it were wired in, it would not compile: every test calls NetworkIptablesManager::chain_name_for(...), but there is no such method anywhere in the crate. The chain name is derived elsewhere. So the suite references a phantom API and is both inert and stale. (There is also a broken rustdoc intra-doc link at state_aware.rs:79 pointing at the same non-existent chain_name_for.)
Net effect: the adversarial-corpus collision tests that were presented as the evidence for the collision fix provide zero coverage. The concern is not entirely untested — the compiled tests inside network_iptables.rs (chain_name_sanitization, chain_name_truncation, chain_names_have_mxc_prefix_and_total_length_cap_of_twenty_four) do run — but the dedicated near-miss collision suite does not.
To clear this:
- Declare the spec suite so Cargo compiles and runs it (e.g. add
#[cfg(test)] mod network_iptables_chainname_spec_tests;under network_iptables, matching theuse super::*;the file already expects), and - Add the referenced NetworkIptablesManager::chain_name_for API (or repoint the tests + the state_aware.rs:79 doc link at whatever method actually derives the chain name) so the suite compiles and passes.
For the record, this does not reintroduce any of the five High findings from the adversarial review — those remain fixed. This is specifically about the collision-fix verification being dead code. Separately, the previously noted open High still stands as a recommended follow-up: provision() returns an error on lxc-create failure without destroying partially-created on-disk state (orphaned auto-named container).
|
Closing this PR. The work is being re-cut from The reason is the review backlog rather than a merge conflict — this branch still merges cleanly, but it accumulated 87 review comments across 42 commits faster than they could be resolved, which makes each individual correctness fix hard to verify in isolation. The first slice is already up: #780 makes the LXC firewall chain name collision-free, deriving it from a SHA-256 digest of the original container name and sizing it to the measured 28-byte iptables ceiling. That covers the chain-identity objections raised here. Still to be re-cut, each as its own PR: the state-aware lifecycle machinery, the signal-cleanup rework, the FFI streaming exec gap, and the remaining network-scoping issues. The review comments on this PR are the input to that re-cut, not discarded. |
Linked work item: AB#62953349 — [LXC] State-aware lifecycle management
Summary
Implements the
StatefulSandboxBackendtrait for the LXC backend, wiringprovision/start/exec/stop/deprovisionto the LXC CLI wrappers. LXC is registered as a non-experimental state-aware backend (BACKEND_KEY = "lxc") in the Rust dispatcher and parser and in the SDK prefix map, and the LXC state-aware config types are exported from the SDK. This path is net-new: before this change the state-aware dispatcher recognized onlyisoandwsb, so anylxc:state-aware request returnedunsupported_containment. Nothing that worked before can regress.exec.execruns the script through the one-shotlxc-attachPTY path; the returnedExecHandlecarries null stdio pipes and a waiter that yields the container's exit code. Live streaming exec — the SDKexecInSandbox/IPtypath — is not available for LXC; callers use the bufferedexecInSandboxAsyncinstead. Closing that gap is tracked in [LXC] SandboxProcess streaming exec is not implemented for the LXC backend #765.container.start(), referencing a deterministic host-side veth name pinned into the container config (lxc.net.0.veth.pair). A firewall-install failure aborts the start rather than running the container unfiltered, and a failed start tears the rules back down.start, and is fail-closed.provision,exec,stop, anddeprovisionreject any filesystem or network policy. Onstart,network.proxyis rejected, and a restrictive network policy (allowedHosts,blockedHosts, or an explicitdefaultPolicy: "block") under a non-firewallenforcementModeis rejected rather than run silently unenforced. A policy-free start and an explicitdefaultPolicy: "allow"are accepted.startreaches the same verdict as a real one. Both start-only rejections above need nothing but the request, so they live invalidate_start— the last thing a dry run executes before it returns. Rejections that read live container state (not_provisioned,already_started, and the multi-interface refusal) stay instart, since a dry run has no container to read.network.defaultPolicywas present on the wire, so an explicit"block"is distinguishable from the struct default a policy-free start produces. An already-running container that carries start policy reportsalready_started.lxc.net.0.veth.pair, so more than one interface leaves unhooked egress paths and zero leaves nothing to hook. A config usinglxc.includeis refused for the same reason — countinglxc.net.Nkeys in the container's own config is a lower bound, since an include can declare interfaces that file never names, and reproducing liblxc's relative-path and glob resolution is exactly the place a subtle error would produce the false confidence the gate exists to prevent.stopand the signal rollback both halt the container before removing its firewall, and both abandon the teardown if the stop fails rather than strip the chain off a live container. Teardown never flushes a chain FORWARD still jumps to:-Fsucceeds regardless of who references the chain, and an emptied user chain returns to its caller instead of reaching its own closing DROP, so flushing a hooked chain is a fail-open. Leaving it costs a leaked chain that still filters correctly.iptables -Nsucceeds, and released only when-Xsucceeds. Publishing at the moment the chain begins to exist covers the whole window and no more: a process whose-Nlost the race never publishes, so it never removes. Teardown runs through the manager that owns the chain, so a released chain is not torn down a second time on drop — a second teardown could otherwise delete a chain another start had since created under the same deterministic name.MXC-+ up to 12 sanitized characters +-+ an 11-character base36 token (a 64-bit FNV-1a hash reduced modulo 36^11 ≈ 2^56.9) = 28 characters, the netfilter limit; the host-side veth ismxcv+ the same token = 15 characters, theIFNAMSIZlimit. This breaks the systematic collapse of shared prefixes, but the derivation is non-cryptographic and not injective: distinct names can still map to one chain, so collisions remain possible. Durable per-container chain ownership is the real fix and is not implemented here.MxcError. The stderr channel is read only for LXC, whose pty merges guest stderr into stdout and leaves the executor as its sole writer; Windows Sandbox and isolation-session both relay guest output to that stream, so on those backends stdout is the only trustworthy channel.lxc.mount.entrylines MXC itself added (tagged with a marker comment), leaving distro- or user-supplied mounts intact.wxcand thelxcexecutor share onerun_state_aware_with_telemetryentry point inmxc_engine, so LXC emits the same phase and correlation-vector telemetry as every other backend.LxcNetworkConfigexposes only the fields LXC honors (defaultPolicy,allowedHosts,blockedHosts, andenforcementModerestricted to"firewall"/"both"), so a config the backend would ignore or reject is a compile-time error. The SDK README records LXC as a Linux-only state-aware backend with no streaming exec.Known gaps
Both need a Linux host with real signal delivery to reproduce, and neither is a fail-open.
iptables -Ncan succeed a moment before ownership is published, stranding a chain nobody records as theirs. Publishing first would invert this into something worse: the loser of a creation race would claim a chain it did not create and its cleanup would delete the winner's, leaving the winner's container unfiltered. A stranded chain is hooked into nothing and filters nothing, so it costs a chain name until the next teardown reclaims it. Closing it properly needs the create and the publish to be atomic against host state shared across processes, which iptables does not offer.CreatedFirewallStateisCopy, andsignal_cleanupsnapshots it and releases the mutex before cleanup, so a rollback acts on the ownership recorded at snapshot time.Validation
Re-verified on this branch (
5449775) on the Windows dev box:cargo test -p lxc_common --lib— 115 passed, 0 failed (includes thestate_aware::suite and the chain-name spec suite).cargo test -p wxc_common --lib— 564 passed, 0 failed. This is a Windows run, so it includes the Windows-gated tests a Linux run excludes.cargo test -p mxc_pty --lib— 7 passed, 0 failed.cargo fmt --all -- --check— clean.cargo clippy -p lxc_common -p wxc_common --all-targets -- -D warnings— clean.The chain-name spec suite pins a previously colliding pair (
web-frontend-017m3bandweb-frontend-01kgar, which both producedMXC-web-frontend-01-3d4a49a5) as distinct, sweeps 300,000 shared-prefix names with no collision, and holds the length bounds (chain at most 28 characters, veth exactly 15) over an adversarial corpus. These establish collision resistance, not injectivity.The rollback and teardown orderings are asserted through pure seams (
rollback_plan,execute_rollback,teardown_chain) becauserun_watchdogis Linux-only and every iptables command fails at spawn on a non-Linux host — a test driving the real path could not observe an ordering at all. What those seams cannot cover is the end-to-end behavior: real iptables enforcement, the pre-start firewall/veth ordering, real signal delivery, a Unix PTY, andtests/scripts/run_lxc_state_aware_test.shall need a root LXC host and were not executed in this pass.One defect this PR does not fix: an unresolvable
blockedHostsentry is logged and skipped, which underdefaultPolicy: allowleaves the host reachable. That code is untouched by this branch (git diff origin/main...HEADcovers no hunk there) and #632 owns the fix, so repairing it here would only manufacture a conflict.Microsoft Reviewers: Open in CodeFlow