feat: claim addresses through the project's own front door - #378
Merged
Conversation
Claiming an address named its project by impersonating a shared `nso-ipam-agent` user and setting the `iam.miloapis.com/parent-*` extras by hand. That grant is global: whoever holds it can claim addresses in any project, and the operator asserts the tenant rather than proving it. Each project-scoped client now addresses Milo's project control-plane path instead. Milo resolves the project from the URL and injects the parent extras itself, so the operator's own identity is what gets authorized, per project. The impersonation grant disappears entirely. The namespace addresses land in is the upstream namespace the claim's objects already live in, so it exists in the project's control plane. The operator reads it instead of creating it, since creating one would be writing into a customer's project on their behalf. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…paths Addressing a project by control-plane path works in production because Milo fronts IPAM and resolves the project from the URL. The e2e environment has no Milo: IPAM is an aggregated apiserver in a kind cluster, nothing there serves resourcemanager.miloapis.com, and every claim now 404s at the aggregator before IPAM is reached. All eleven networkinterfaceclaim suites fail. A small proxy fills the gap. It terminates the project control-plane path, derives the project from the path segment, and re-issues the remainder against the kind apiserver carrying that project's tenancy — so which project a request lands in is decided by the path and nothing else. A caller cannot choose one: its credential and any impersonation it tried to assert are dropped before the request goes on. The operator's IPAM kubeconfig now names the stand-in rather than the apiserver, and the operator's own impersonation grant is gone, since it no longer asserts a tenant itself. The suites are unchanged. Their cross-project deny assertions reach IPAM directly rather than through this path, so they still prove isolation on their own terms. This is test scaffolding: it lives under test/e2e, is deployed only by Taskfile.test-infra.yml, and is in no production overlay. It reproduces where a request is routed, not Milo's per-project authorization of the caller — the environment has no per-project RBAC universe to authorize against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The stand-in that let these suites run was a proxy we wrote. It terminated the project control-plane path, decided which project the path named, and re-issued the request as a fixed tenant. It proved the operator addressed the right URL, and nothing beyond that: the thing it was standing in for is exactly the thing it was doing. A real Milo now fronts IPAM. IPAM aggregates into Milo instead of into the kind apiserver, and the kind registration is deleted so there is no second door left open. A claim travels the production route — Milo strips the project prefix, resolves the project, and forwards it to IPAM as X-Remote-Extra-* over a front-proxy client certificate IPAM verifies. The two e2e projects are real Project objects rather than strings in four places. The suites reach IPAM the same way the operator does. They used to impersonate a user and assert the three project extras by hand, which let the caller choose its own tenant — so a correctly routed request and a request that merely claimed the right answer looked identical. They now use a per-project kubeconfig whose server URL carries the path, and nothing in that credential names a project. No assertion was weakened, and the isolation suite got stronger. Its project-less negative control used to be an un-extra'd impersonation against the kind apiserver: a different server and a different mechanism, which could have kept passing even if project routing were broken. It is now the same identity through the same front door at Milo's root, so the path is the only thing that differs between the read that works and the read that is blind. Two things are worth knowing about the environment, and both are in test/e2e/README-milo.md rather than only here. Per-project AUTHORIZATION is still not tested. Milo scopes a request by putting the project in the caller's user extras, and stock RBAC ignores extras, so a grant anywhere authorizes every project path. This was measured, not assumed: a RoleBinding created inside a project's control plane is stored, is readable back through that project's path, and is never evaluated — while the same grant at the root authorizes both projects at once. Closing it means the OpenFGA provider, which is a second repository and a deliberate non-goal here. Milo also never marks a remote APIService Available; it disables the controller that would. Production writes that condition from milo-controller-manager's RemoteAPIServiceAvailabilityReconciler, which sets it unconditionally and probes nothing, so the environment writing it produces the same object from the same evidence rather than diverging. Readiness does not rest on it either way: milo-ipam-gate reads real IPPools through each project's path and requires the root path to read none, which is more than the production mechanism checks. Milo is the apiserver and one etcd StatefulSet, ~200 MiB, and adds about a minute to env bring-up. All 40 chainsaw suites pass against it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A claim allocated in a namespace the project's control plane does not hold used to retry forever. The read failed, the reconcile returned an error, and the claim sat with the conditions it started with — so a consumer waiting on Ready got a timeout and no account of why. That namespace is not something to wait for. The platform provisions it with the project: Milo's ProjectController writes it on every reconcile, as a get-then-create that heals it if it ever goes missing. Its absence therefore says the project's control plane was never bootstrapped, which is a condition of the project rather than a moment in a race, and no amount of retrying changes it. A missing namespace now rejects the claim with ProjectNamespaceNotFound, naming the project and the namespace, alongside the answers the reconciler already gives for a project it cannot resolve or a network that never reached the location. The reason is the fact rather than a diagnosis of it, in the shape NetworkNotFound and LocationNotAvailable already set, and it is distinct from ProjectUnresolved, which is about the labels on the claim's own namespace. A rejection keeps its own way back, so a project bootstrapped later is picked up without anything else having to notice. Only NotFound is an answer. A namespace that cannot be read says nothing about whether it is there, so any other failure stays an ordinary error and is retried. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The note on this step said a project control plane starts with only milo-system, and that in production the namespaces a project uses are already there. Read together that suggests production planes also start bare and are populated by whoever needs them, which would make this step a fixture peculiar to e2e. It is the opposite. In production the namespace comes with the project: milo-controller-manager's ProjectController creates it on every reconcile, labelled miloapis.com/project-default. This environment deploys the Milo apiserver alone, with no controller-manager to run that, so the step stands in for that single write — the same reasoning, and the same wording, the aggregated API service condition is already handled with here. The namespace is created with that label too, so what a suite exercises against is the object production has rather than one shaped like it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed copy Milo's e2e deployment was 465 lines of manifests reconstructed by hand from Milo's v0.32.5 git tag. Nothing kept them current and nothing checked them against upstream, so any change Milo made to how its apiserver is configured reached this environment only when someone noticed and retyped it. Milo publishes a kustomize bundle, ghcr.io/milo-os/milo-kustomize, from the same shared workflow that produces the IPAM and dns-operator bundles this repo already consumes. Take the deployment from there and keep only what this environment genuinely adds. Structured like config/dependencies/ipam: a root-kustomization.yaml that composes the bundle with a local overlay, deliberately not named kustomization.yaml because it can only build inside the staging directory test-infra:milo-bundle assembles; and a self-contained overlay/ that validate-kustomizations can still build on a clean checkout. From the bundle: the apiserver Deployment, Service and PDB, the CA chain, and Milo's serving certificate. Kept local: etcd (upstream ships a Flux HelmRelease and there is no Flux here), the static tokens, the front-proxy client certificate that makes remote aggregation work, and the NodePort the host-side suites reach Milo through. Upstream's own overlays/test-infra was not reused. It pulls in the controller-manager, argo-events, a gateway and audit and tracing sinks — none of which project control-plane paths need, and all of which this environment deliberately does not run. Two things about the bundle are worth knowing. Its apiserver drives every flag from an env var and ships no volumes, so the patch is where storage, credentials and certificates actually get wired, and a default this env cannot satisfy is a startup failure rather than a fallback. And its own default for --audit-webhook-initial-backoff is the empty string, which is not a parseable duration; the apiserver exits before it ever reads the audit policy and discovers auditing is off. Upstream never hits this because its overlay always enables auditing. Per-project authorization is still not exercised here, exactly as before. test/e2e/README-milo.md continues to say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the bundle adoption: three components that look reusable were measured against a live env rather than judged by shape, and none of them fit. The reasoning is written down next to what it explains so the question does not get reopened from the file listing alone. components/apiserver-audit-logging does not replace any local config. It contributes two ConfigMaps and mounts them and sets no environment at all, so upstream's own overlay still sets AUDIT_POLICY_FILE, AUDIT_WEBHOOK_CONFIG_FILE, AUDIT_WEBHOOK_MODE and the backoff by hand. Adopting it means three more env vars here, not one fewer — the empty --audit-webhook-initial-backoff default that crashes the apiserver at startup is not satisfied by the component and still has to be set. Its webhook posts to vector-audit-log-processor, which does not exist in a kind cluster: measured with the component applied, Milo logs "Error in audit plugin 'webhook' affecting 172 audit events: ... no such host" with the impacted events dumped inline, from the env-up gate alone. components/apiserver-tracing does set its own env var, so it would remove a line. It also points the exporter at telemetry-system-tempo, which is likewise absent: Milo serves fine and then logs a trace-export failure every ten seconds for the life of the run. One line of config is not worth a permanent error stream in the log someone reads when a suite fails. Neither sink is consumed by anything here, so both stay off. overlays/test-infra/components/auth carries two identities where this environment needs three. There is no separate identity for the operator, and that split is what makes a passing suite evidence of a real authorization decision rather than of privilege. Its tokens and usernames also differ from the ones the kubeconfigs and the bootstrap RBAC are written against, and its Secret has the same name as the local one, so adding it alongside fails to build at all. etcd stays local for the reason already recorded: upstream ships a Flux HelmRelease and there is no Flux here. No behaviour change. Verified on a live env: 11/11 networkinterfaceclaim suites and 35/35 across test/e2e. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d of our own Allocating an address checked that the project's namespace existed before claiming anything in it. The check was left over from when this code created that namespace; once it became read-only, it only asked a question the write that follows already asks. The API server validates the namespace on every write and caches the answer, so the extra round trip on every allocation bought nothing — and bought it at the wrong moment, since a namespace deleted between the read and the write would still fail at the write. The claim is told the same thing it was told before: a project whose control plane holds no namespace is rejected with ProjectNamespaceNotFound, naming the project and the namespace, and keeps its own way back for a project bootstrapped later. That answer now comes from the refusal itself. Reading it off the refusal takes care, because more than one thing on this path arrives as a 404 and only one of them is about the namespace. An IPClaim that is not there names ipclaims. An IPAM the server does not serve, and a project path that 404s at Milo because the project was deleted, both fail in discovery before a request is sent and name nothing at all — while still reading as NotFound. So the namespace is identified by what the refusal says rather than by its status code: the namespace lifecycle admission plugin names the namespaces resource and the namespace itself, and nothing else on this path does. Everything else keeps the handling it had. Identifying it that way also stops a deleted project being reported as a project missing its namespace, which is what the old read did with the 404 Milo gives for a path that no longer resolves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scotwells
marked this pull request as ready for review
August 15, 2026 18:30
privateip
approved these changes
Aug 15, 2026
ecv
approved these changes
Aug 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An address is claimed on a consumer's behalf, and the platform needs to know whose it is.
Until now the operator asserted that itself, by claiming to be a dedicated address agent and naming the project in the request. That works, but it means anything holding that credential can claim addresses in any project — a single credential that reaches every customer's address space, sitting on every location that allocates.
Now the operator simply asks through the project's own front door, and the platform decides what that request is allowed to touch.
What changes
Nothing a consumer sees. Addresses are claimed, held and released exactly as before.
What changes is who is trusted. The operator no longer carries the authority to speak for any project; it presents its own identity, and the platform answers per project. A credential taken from one location no longer reaches everything.
A namespace that is missing now says so
Allocating for a project reads a namespace the platform creates when the project is created. If it is absent, the project is not ready — which used to surface as a request that quietly retried forever. It now says which project and which namespace, on the claim itself, alongside every other reason a claim can fail.
Testing
The test environment now runs the real platform front door rather than a stand-in, so these paths are exercised the way they run in production rather than against something written to imitate it. That replaced a purpose-built stand-in and deleted it.
One case got genuinely stronger. The test proving two projects cannot see each other's addresses used to check the negative case against a different server entirely — it could have passed even if routing were broken. It now goes through the same door with only the project differing.
40 suites pass.
Known limit, stated plainly: this environment does not exercise per-project authorization — the platform only enforces that through a component the test environment does not run, so a grant there opens every project. Routing and isolation are covered; authorization is verified against staging. This is recorded in the environment's own docs.
Related