diff --git a/ROADMAP.md b/ROADMAP.md index ea74930bf..60f62983d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -888,7 +888,7 @@ graph LR | Web UI + macOS app UX audit | In progress | P0 | — | | | | Release qualification gate (auto-QA matrix blocks the tag) | In progress | P0 | — | [081-release-qa-gate](./specs/081-release-qa-gate/) | | | Action log / transparency — info at a glance | In progress | P1 | — | | | -| Agent-token scope hardening: every MCP request authorized by its own scope (spec 105) | In progress | P1 | 18/109 (17%) | [105-agent-scope-hardening](./specs/105-agent-scope-hardening/) | | +| Agent-token scope hardening: every MCP request authorized by its own scope (spec 105) | In progress | P1 | 31/109 (28%) | [105-agent-scope-hardening](./specs/105-agent-scope-hardening/) | | | Token-efficiency benchmark: measured savings, published results | In progress | P1 | 62/64 (97%) | [103-token-bench](./specs/103-token-bench/) | | | Telemetry identity & data quality (machine_id + CI-filter hardening) | In progress | P1 | — | | | | Telemetry v7: honest funnel + churn instrumentation | In progress | P1 | — | [080-telemetry-v7-churn](./specs/080-telemetry-v7-churn/) | | @@ -1034,6 +1034,6 @@ Legend: `shipped` ≥95% checked · `in-flight` 1–94% · `drafted` 0% · `—` | [102-schema-deferred](./specs/102-schema-deferred/) | `shipped` | 89/89 (100%) | | [103-token-bench](./specs/103-token-bench/) | `shipped` | 62/64 (97%) | | [104-auto-routing-mode](./specs/104-auto-routing-mode/) | — | — | -| [105-agent-scope-hardening](./specs/105-agent-scope-hardening/) | `in-flight` | 18/109 (17%) | +| [105-agent-scope-hardening](./specs/105-agent-scope-hardening/) | `in-flight` | 31/109 (28%) | | [106-security-residual-fixes](./specs/106-security-residual-fixes/) | `shipped` | 18/19 (95%) | | [107-server-edition-sso-hardening](./specs/107-server-edition-sso-hardening/) | `in-flight` | 102/126 (81%) | diff --git a/docs/features/agent-tokens.md b/docs/features/agent-tokens.md index 2c7f48fe7..798b604c0 100644 --- a/docs/features/agent-tokens.md +++ b/docs/features/agent-tokens.md @@ -476,6 +476,49 @@ Denied to agent tokens on both surfaces: On the MCP surface (`upstream_servers`, `quarantine_security`) these return a tool error; on the REST surface (mutating `/api/v1/servers/...`, `/api/v1/config/...`, and `/api/v1/registries/...` routes) they return **`403 Forbidden`** (`operation requires admin access`). Read-only operations stay available to scoped tokens: `upstream_servers` `list`/`tail_log`, `GET /api/v1/servers`, per-server diagnostics, registry reads, and `GET /api/v1/index/search` (which honors quarantine — a quarantined server's tools are withheld from search on every surface). Those reads are **scope-filtered** as described above. `GET /api/v1/config` is the exception: it is an admin document (it carries the global `api_key`, every server's credentials, and a second enumeration of server names under `profiles[].servers`), so it returns `403` for an agent token rather than a filtered view. +**Log attribution on `tail_log`.** Per-server log files are named from a +*sanitised* server name, so two configured servers can share one file — +`a/b` and `a_b` both write `server-a_b.log`, and on a case-insensitive +filesystem so do `A` and `a`. `upstream_servers` `tail_log` therefore +returns a scoped token only the records **attributable to the server it +named**: every record mcpproxy writes carries its writer's server identity, +and the reader filters on it *before* applying the line limit, so +`lines_returned` counts the authorized tail and a co-owner's interleaved +record never displaces an authorized one. The rule is the same whether or +not a co-owner exists — a scoped caller never gets a whole-file refusal that +depends on another server sharing the file, and a single over-long line in +the shared file (longer than 1 MiB) is skipped rather than failing the read. +Withheld from scoped callers: +records with no writer identity (lines written before this rule existed, +hand-appended lines, torn fragments), records about a container — by id, +name or count — that do not prove the container's owner (`container_owner`, +written by container housekeeping since this rule — earlier housekeeping +records are treated as non-attributable), and records whose subject is +another server (an OAuth callback tear-down that an earlier version routed +through the wrong server's logger), and a child process's own output line +that names a container — Docker's `docker run` name-conflict error, for +instance, quotes the *other* container's name and id when two servers' +generated container names collide (`a/b` and `a-b` both produce +`mcpproxy-a-b-…`) — the same rule covers the "Connection failed" record +whose error re-emits that stderr. For the same reason a scoped token's +`connection_status.last_error` (on `tail_log` and `list`, and the health +detail derived from it) has container ids, canonical container names and +Docker's name-conflict phrase replaced by `[container]`; a server that is +itself *named* like a container (`mcpproxy-tenant-abcd`) is not a container +mention, so its ordinary child output stays attributable. Retained effects: rotation and retention +of a shared file stay shared, so a co-owner's output can rotate an authorized +record out of the readable history; and child process output is attributed +to the server whose process wrote it — a child cannot forge another server's +identity. The administrator readers — `tail_log` with the API key or over the +local socket, and `mcpproxy upstream logs` — keep the whole file exactly as +before; a profile on the URL (`/mcp/p/`) bounds *which* server an +administrator may name, not which records of it they see. The REST endpoint +`GET /api/v1/servers/{id}/logs` is **not** attributed: it serves the whole +shared file to any caller entitled to the server name, agent tokens +included. Until it is aligned with `tail_log`, do not rely on it to keep a +co-owner's records from a scoped token — the REST management API's scope +policy is a separate piece of work. + ## Profile Pinning A [profile](./profiles.md) scopes tool discovery and calls to a named subset of upstream servers. With `--profile-pin`, you can **bind a token to a single profile** so it can never operate outside it — regardless of the URL it connects to or any `set_profile` call it makes. diff --git a/docs/features/docker-isolation.md b/docs/features/docker-isolation.md index 45d380178..b25db7f24 100644 --- a/docs/features/docker-isolation.md +++ b/docs/features/docker-isolation.md @@ -348,7 +348,60 @@ When MCPProxy stops, containers are cleaned up with a 30-second timeout: 1. **Graceful Stop**: `docker stop` (sends SIGTERM to container) 2. **Force Kill**: `docker kill` if container doesn't stop gracefully -Containers are labeled with `mcpproxy.managed=true` for identification. +Containers are labeled with `com.mcpproxy.managed=true` for identification +and `com.mcpproxy.server=` (the raw, unsanitised name) for +ownership. + +### Container ownership + +Every container mcpproxy creates is named +`mcpproxy--<4 random chars>`. The name alone does not +identify the server — `a/b` and `a-b` both sanitise to `a-b` — so every +cleanup path (the pre-start sweep for stale containers, the container +captured from `--cidfile`, the disconnect fallbacks by exact name, by name +pattern and by image name) inspects the container and stops or removes it +only when its `com.mcpproxy.server` label **and** canonical name both match +the server being cleaned up — and it re-inspects the container immediately +before every `docker stop`, `kill` or `rm -f` (the kill after a failed stop +included), never acting on an earlier listing, so a container renamed, +relabelled or replaced in between is left alone. Containers you started yourself with +`docker run --name …`, or that pre-date the label, are never touched by any +of these paths, and a container that merely shares an image with a server's +is never stopped on that server's behalf. Housekeeping records in the +per-server log carry `container_owner` (the label value read back from +Docker) so [`tail_log`](/features/agent-tokens) can attribute them to the +right server; the pre-start "Docker isolation configured" record names the +generated container name before Docker has created anything and carries no +owner. The child process's own output lines (the docker CLI's stderr +included) are written to the per-server log as the `message` field of a +`stderr` or `launcher` record marked `child_output`, never as the record +text. + +Two consequences of the ownership rule are worth knowing: + +- **Servers you configure as `docker run …` yourself** (no isolation) get no + `com.mcpproxy.server` label — MCPProxy only labels the containers it + builds for isolation — so MCPProxy never stops or removes their + container, not even through the `--cidfile` it injects into your command. + Use `--rm` (and let the container exit when its stdin closes) or stop it + by hand; earlier versions would stop it via the cidfile and, if that + capture failed, every container on the same image, yours or not. +- **Renaming a server** changes the label value a container must carry. A + container created under the old name is no longer owned by the new one, + so it is left alone by the pre-start sweep and must be removed manually + (`docker rm -f`). +- **The shutdown and emergency sweeps** (every `com.mcpproxy.managed` + container on shutdown; every one carrying this instance's id when + shutdown fails) apply the same rule: only containers canonically owned by + a server in the current configuration are stopped or removed — each one + re-inspected immediately before its stop, kill or removal through the + same check every per-server cleanup uses, so a container renamed or + relabelled after the sweep listed it is left alone — and the + disconnect-timeout path re-checks the ownership of the id it tracked + before `docker rm -f`. A container that merely carries the mcpproxy + labels — one you labelled yourself, or an orphan of a server that is no + longer configured — is left alone and only counted in a warning, never + named. Remove such orphans by hand (see below). ### Manual Cleanup @@ -356,10 +409,10 @@ If containers remain after MCPProxy stops: ```bash # List MCPProxy-managed containers -docker ps --filter "label=mcpproxy.managed=true" +docker ps --filter "label=com.mcpproxy.managed=true" # Remove all MCPProxy containers -docker rm -f $(docker ps -q --filter "label=mcpproxy.managed=true") +docker rm -f $(docker ps -q --filter "label=com.mcpproxy.managed=true") ``` See [Shutdown Behavior](/operations/shutdown-behavior) for detailed subprocess lifecycle documentation. diff --git a/internal/logs/attribution.go b/internal/logs/attribution.go new file mode 100644 index 000000000..8a030c025 --- /dev/null +++ b/internal/logs/attribution.go @@ -0,0 +1,539 @@ +package logs + +import ( + "bufio" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "regexp" + "strings" + + "go.uber.org/zap" + + "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" +) + +// Spec 105 FR-007 (research D8): per-record log ownership. +// +// Two raw server names can share ONE per-server log file — `a/b` and `a_b` +// both sanitise to server-a_b.log (sanitizeServerLogName), and on a +// case-insensitive filesystem so do `A` and `a` — so a scoped caller tailing +// "its" server must receive only the records its server actually wrote. There +// is NO new field for that (D8): every per-server writer already stamps each +// record with `server=` (NewUpstreamServerLogger), so administrator +// records and the whole-file reader (ReadUpstreamServerLogTail) are +// byte-identical to before. Unforgeability comes from two rules: +// +// 1. Producer rule: child-controlled text (stderr lines, launcher-pumped +// stdout/stderr, docker CLI output) is only ever a zap FIELD VALUE, where +// the encoder escapes it inside the fields object, on a record stamped +// child_output=true (ChildOutputField). internal/upstream/core audits +// every per-server log call site for a constant message. Files written +// by builds before codex round 2 carry the launcher-pumped child line as +// the MESSAGE; rule 2 judges that shape and withholds it when the child +// text carries a boundary or a record header. +// 2. Reader rule: a console-encoder record is +// `ts | LEVEL | caller | msg | {fields}` (`caller` is absent on the +// OAuth tee, which has no AddCaller). The reader takes the FIRST ` | {` +// boundary on the line and accepts the record only if (a) the text in +// front of the boundary starts with exactly ONE record header +// (`ts | LEVEL | `, consoleHeaderPattern) and contains no second one, +// and (b) the boundary's suffix decodes as exactly one complete JSON +// object with no trailing bytes that is not itself a JSON-encoder record. +// Child text inside a field value is escaped and cannot close the object +// early or start a boundary. The reader never scans past a failed +// boundary, and (a) is what makes a torn record harmless: a torn record +// with no terminator (partial final write) followed by an appended +// complete record shares one physical line, and the later record's +// boundary would otherwise attribute the whole line — foreign fragment +// included — to the later writer (codex round 1). A tear inside the +// fragment's header leaves no header at offset 0; a tear anywhere after +// it leaves the fragment's header in front of the later record's — two +// headers (codex round 2). A fragment torn right after its caller +// separator in front of a JSON-encoder record leaves one header and a +// suffix that decodes: that suffix carries the JSON encoder's own +// `level`/`ts`/`msg` keys, which a console fields object never does, so +// (b) rejects it. A line that starts with `{` is a JSON-encoder record +// (or a torn one) and is judged as that single object, never by the +// console scan. Lines with no accepted boundary (pre-stamp records, torn +// fragments, launcher-era child lines carrying ` | {` as the message) +// are non-attributable and withheld from scoped callers. +// 3. Subject-evidence rule (historical records): a stamp proves who WROTE a +// record, not that every subject it names is that server's. A record +// that names a container is attributable only when it carries +// `container_owner` (the container's com.mcpproxy.server label, written +// by the housekeeping paths since Spec 105) equal to the requested +// server; the sanitised container name is never evidence (`a/b` and +// `a-b` both name mcpproxy-a-b-*). A record whose `server` field names +// another server (a pre-105 callback-stop record routed through the +// wrong logger) is withheld. Child output is a container subject too +// when it mentions one: Docker's own `docker run` failure names the +// colliding container's id and name (`a/b` and `a-b` both generate +// mcpproxy-a-b-*), and that text reaches the per-server log as a child +// line. The producers stamp every child line with `child_output=true` +// (ChildOutputField), and a child-output record that mentions a container +// id, a canonical container name or Docker's name-conflict phrase +// (containerMentionPattern) is withheld unless it carries a matching +// `container_owner` — which child output never does. The check runs over +// the decoded CHILD-CONTROLLED values only (attributionChildMessageField +// / attributionChildErrorField: the `message` field the stderr and +// launcher producers write the child's line into, and the `error` field +// of a record whose error re-emits the recent-stderr buffer — the +// "Connection failed" record, codex round 3), never over the serialized +// line: the writer stamp of a server that is +// itself named like a container (`mcpproxy-tenant-abcd`) is not a +// subject, so its ordinary child output stays attributable. +// +// Administrators, REST and the CLI keep the whole file (SC-005). + +// consoleFieldsBoundary separates the console encoder's message from its +// fields object (getFileEncoder: ConsoleSeparator " | ", fields rendered as a +// JSON object). +const consoleFieldsBoundary = " | {" + +// consoleHeaderPattern is the fixed-shape start of every console-encoder +// record this package writes (getFileEncoder: TimeEncoderOfLayout +// "2006-01-02T15:04:05.000Z07:00", CapitalLevelEncoder, ConsoleSeparator +// " | "). The text in front of an accepted boundary must start with exactly +// one of these and contain no other. +var consoleHeaderPattern = regexp.MustCompile( + `\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}(?:Z|[+-]\d{2}:\d{2}) \| (?:DEBUG|INFO|WARN|ERROR|DPANIC|PANIC|FATAL) \| `) + +// containerMentionPattern matches text that names a container: a full +// 64-hex container id (what the daemon's messages carry), a canonical +// mcpproxy container name (generateContainerName: mcpproxy--<4 +// alphanumerics>), or Docker's name-conflict phrase. Applied to the +// child-controlled values of child-output records only, and by +// RedactContainerMentions to status text served to scoped callers. +var containerMentionPattern = regexp.MustCompile( + `\b[0-9a-f]{64}\b|\bmcpproxy-[A-Za-z0-9_.-]+-[a-z0-9]{4}\b|already in use by container`) + +// containerMentionRedacted replaces every containerMentionPattern match in +// RedactContainerMentions. +const containerMentionRedacted = "[container]" + +// RedactContainerMentions blanks every container id, canonical container +// name and Docker name-conflict phrase in s. The MCP status surface +// (`upstream_servers` tail_log / list `connection_status.last_error`, and +// the health detail derived from it) renders a connect error that re-emits +// the child's stderr — on a `docker run` name collision that text names the +// colliding container, which belongs to another server (`a/b` and `a-b` +// generate the same name). A scoped caller gets the same redaction whether +// or not a co-owner exists (FR-007: uniform, non-disclosing); administrators +// see the text unchanged (SC-005). +func RedactContainerMentions(s string) string { + return containerMentionPattern.ReplaceAllLiteralString(s, containerMentionRedacted) +} + +// Field names the attribution rules key on. +const ( + attributionServerField = "server" + attributionContainerOwnerField = "container_owner" + attributionContainerIDField = "container_id" + attributionContainerNameField = "container_name" + attributionContainerCountField = "container_count" + attributionChildOutputField = "child_output" + // Child-controlled values on a child-output record: the child's line + // (stderr / launcher producers) and an error that re-emits the + // recent-stderr buffer (recordConnectionFailure). Only these are + // searched for a container mention. + attributionChildMessageField = "message" + attributionChildErrorField = "error" + + // JSON-encoder entry keys (getJSONEncoder / zap.NewProductionEncoderConfig). + // A console fields object never carries all three; a suffix that does is + // a JSON-encoder record appended after a torn console fragment. + jsonEncoderLevelKey = "level" + jsonEncoderTimeKey = "ts" + jsonEncoderMessageKey = "msg" +) + +// ChildOutputField stamps a record whose payload is a child process's own +// output (stderr lines, launcher-pumped stdout/stderr, docker CLI output). +// Every producer that writes child text into the per-server log — always as +// a field VALUE, never the message (rule 1) — attaches it, so the attributed +// reader can apply the child-output subject rule (rule 3). +func ChildOutputField() zap.Field { + return zap.Bool(attributionChildOutputField, true) +} + +// ReadUpstreamServerLogTailAttributed reads the last N records of an upstream +// server log that are attributable to serverName (Spec 105 FR-007, research +// D8). Attribution is decided per record BEFORE the tail limit, so an +// interleaved co-owner record never displaces an attributable one from the +// returned window, and the returned length is the authorized tail length. +// Records with no accepted stamp, records stamped for another server and +// records failing the subject-evidence rule are withheld. Administrators use +// ReadUpstreamServerLogTail (whole file, byte-identical to pre-105). +// +// The scan starts at scopedBackwardStartOffset, at most +// scopedBackwardReadBudget bytes before EOF (codex round 16 finding 2): +// scanning from byte 0 made a request's cost proportional to whatever a +// hidden co-owner had written earlier in this shared file — a response-time +// side channel disclosing its volume, which SC-005's non-disclosing-refusal +// definition (status, body AND timing class) forbids. Below the budget (the +// common case) this reads and returns exactly what a whole-file scan from +// byte 0 would; past it, a request whose own most recent `lines` records sit +// further back returns fewer than `lines` records rather than reading +// further — bounded and fail-closed, not incorrect. The append-ordering +// this relies on (newest records nearest EOF), the boundary/subject-evidence +// rules (readBoundedLine, recordAttributableTo) and the per-record cap are +// all unchanged; only where the scan starts is new. +func ReadUpstreamServerLogTailAttributed(config *config.LogConfig, serverName string, lines int) ([]string, error) { + if lines <= 0 { + lines = 50 + } + if lines > 500 { + lines = 500 + } + + filename := serverLogFilename(serverName) + logFilePath, err := GetLogFilePathWithDir(config.LogDir, filename) + if err != nil { + return nil, fmt.Errorf("failed to get log file path for server %s: %w", serverName, err) + } + + if _, err := os.Stat(logFilePath); os.IsNotExist(err) { + return []string{}, nil + } + + file, err := os.Open(logFilePath) + if err != nil { + return nil, fmt.Errorf("failed to open log file for server %s: %w", serverName, err) + } + defer file.Close() + + info, err := file.Stat() + if err != nil { + return nil, fmt.Errorf("failed to stat log file for server %s: %w", serverName, err) + } + if start := scopedBackwardStartOffset(info.Size()); start > 0 { + if _, err := file.Seek(start, io.SeekStart); err != nil { + return nil, fmt.Errorf("failed to seek log file for server %s: %w", serverName, err) + } + } + + // Filter first, limit second: only attributable records enter the window. + // A line past the cap is skipped as non-attributable rather than aborting + // the read: a shared file means a co-owner (or its child, whose lines the + // launcher pumps up to 1 MiB) could otherwise make the scoped caller's + // own tail fail until rotation — a response class that would depend on + // the hidden co-owner (SC-001). The whole-file reader is untouched. + var attributed []string + reader := bufio.NewReaderSize(file, 64*1024) + for { + line, ok, err := readBoundedLine(reader, attributedLineCap) + if err != nil { + return nil, fmt.Errorf("failed to read log file for server %s: %w", serverName, err) + } + if !ok { + break + } + if recordAttributableTo(line, serverName) { + attributed = append(attributed, line) + } + } + + if attributed == nil { + return []string{}, nil + } + if len(attributed) <= lines { + return attributed, nil + } + return attributed[len(attributed)-lines:], nil +} + +// attributedLineCap bounds one rendered record the attributed reader will +// consider: a record whose content (terminator excluded) is longer than the +// cap is non-attributable (skipped), never fatal; exactly the cap is eligible. +const attributedLineCap = 1024 * 1024 + +// scopedBackwardReadBudget bounds how many bytes before EOF +// ReadUpstreamServerLogTailAttributed will read (scopedBackwardStartOffset), +// independent of the file's total size or a hidden co-owner's share of it +// (SC-005: a non-disclosing response must not vary in timing class with what +// the requester cannot see). A flat ceiling rather than a multiple of +// `lines`: at the highest request (500) and attributedLineCap-sized (1 MiB) +// records, lines*attributedLineCap would itself be hundreds of MiB — as +// unbounded in practice as no budget at all. 16 MiB comfortably covers the +// realistic case (this server's own most recent `lines` records are +// ordinary log lines, a few hundred bytes to a few KiB each, however deep a +// co-owner's own history runs before them) while keeping the worst-case +// scoped read small and constant regardless of the file's total size. A +// request whose own most recent `lines` records sit further back than this +// budget (thin recent history behind a huge co-owner run) returns fewer than +// `lines` records rather than reading further: bounded and fail-closed, not +// incorrect. +const scopedBackwardReadBudget = 16 * 1024 * 1024 + +// scopedBackwardStartOffset returns the byte offset +// ReadUpstreamServerLogTailAttributed starts reading from for a file of +// fileSize bytes: at most scopedBackwardReadBudget bytes before EOF, never +// negative. The bytes the scan then reads (fileSize minus the returned +// offset) is therefore bounded by scopedBackwardReadBudget for any +// fileSize — provably not proportional to the file's total size. +func scopedBackwardStartOffset(fileSize int64) int64 { + start := fileSize - scopedBackwardReadBudget + if start < 0 { + return 0 + } + return start +} + +// readBoundedLine returns the next line (without its terminator) and +// ok=true, or ok=false at end of input. A line whose content exceeds limit is +// consumed to its terminator and returned as an empty, non-attributable line +// so the caller keeps reading; only a genuine read error is returned. +func readBoundedLine(r *bufio.Reader, limit int) (string, bool, error) { + var buf []byte + overlong := false + for { + chunk, err := r.ReadSlice('\n') + content := chunk + if err == nil { + content = chunk[:len(chunk)-1] // the terminator is not content + } + if !overlong { + if len(buf)+len(content) > limit { + overlong = true + buf = nil + } else { + buf = append(buf, content...) + } + } + switch { + case err == nil: + // Terminator reached. + if overlong { + return "", true, nil + } + return string(buf), true, nil + case errors.Is(err, bufio.ErrBufferFull): + continue // more of the same line follows + case errors.Is(err, io.EOF): + if len(buf) == 0 && !overlong { + return "", false, nil + } + if overlong { + return "", true, nil + } + return string(buf), true, nil + default: + return "", false, err + } + } +} + +// recordAttributableTo reports whether one rendered log line is attributable +// to serverName under the D8 reader and subject-evidence rules. It is +// encoder-agnostic: a line that is itself one complete JSON object is a +// JSON-encoder record; otherwise the console boundary scan applies, so a file +// written under both encoders over its lifetime is read correctly. +func recordAttributableTo(line, serverName string) bool { + fields, ok := recordFields(line) + if !ok { + return false // no accepted stamp: legacy line, torn fragment, foreign shape + } + return fields.attributableTo(serverName) +} + +// recordFields extracts the fields object of a rendered record, or ok=false +// when the line carries no accepted fields object. +func recordFields(line string) (attributionFields, bool) { + // JSON encoder: the whole line is the record. A `{`-prefixed line that is + // not exactly one object is torn or foreign; it never falls through to the + // console scan, whose boundary could belong to a record appended after + // the tear. + if strings.HasPrefix(line, "{") { + fields, ok := decodeExactlyOneObject(line) + if !ok { + return attributionFields{}, false + } + fields.applyChildOutputSubject() + return fields, true + } + + // Console encoder: the FIRST ` | {` boundary decides. A boundary whose + // suffix does not decode is evidence of a torn or foreign prefix, so the + // whole line is non-attributable; scanning on to a later boundary would + // hand the prefix to whoever wrote the later record. + idx := strings.Index(line, consoleFieldsBoundary) + if idx < 0 { + return attributionFields{}, false + } + // The text in front of the boundary must be exactly one record: a + // header at offset 0 and no second header anywhere before the boundary. + // A torn foreign record (no terminator) followed by an appended complete + // record shares this physical line; a tear inside the fragment's header + // leaves no header at offset 0, a tear anywhere after it leaves two. + headers := consoleHeaderPattern.FindAllStringIndex(line[:idx], 2) + if len(headers) != 1 || headers[0][0] != 0 { + return attributionFields{}, false + } + start := idx + len(consoleFieldsBoundary) - 1 // at the '{' + fields, ok := decodeExactlyOneObject(line[start:]) + if !ok || fields.jsonEncoderRecord { + // A fields object that is itself a JSON-encoder record is a complete + // record appended after a fragment torn right behind its separator. + return attributionFields{}, false + } + fields.applyChildOutputSubject() + return fields, true +} + +// attributionFields is the subset of a record's top-level fields the +// attribution rules consult. Every occurrence of a key is kept: zap renders a +// logger's With fields first and the call's fields after them, so a record +// can legitimately carry the writer stamp AND a subject `server` field, and +// the rule is that ALL of them must agree. +type attributionFields struct { + servers []string + containerOwners []string + namesContainer bool + // childOutput marks a record whose payload is child process output + // (ChildOutputField); such a record is a container subject when one of + // its child-controlled values (childPayload) mentions a container. + childOutput bool + childPayload []string + // jsonEncoderRecord is set when the object carries all three JSON-encoder + // entry keys, i.e. it is a whole JSON-encoder record rather than a console + // record's fields object. + jsonEncoderRecord bool +} + +// applyChildOutputSubject marks a child-output record as a container subject +// when a child-controlled value names a container: Docker's own name-conflict +// error carries the colliding container's id and name, so the record needs +// container_owner like any other container record. The stamp fields and the +// serialized line are never searched — a server named like a container is +// not a subject (codex round 3). +func (f *attributionFields) applyChildOutputSubject() { + if !f.childOutput { + return + } + for _, value := range f.childPayload { + if containerMentionPattern.MatchString(value) { + f.namesContainer = true + return + } + } +} + +// attributableTo applies the stamp and subject-evidence rules. +func (f attributionFields) attributableTo(serverName string) bool { + // Stamp: at least one `server` value, and every one exactly the requested + // name — a callback record naming another server fails here. + if len(f.servers) == 0 { + return false + } + for _, s := range f.servers { + if s != serverName { + return false + } + } + // Subject evidence: a container record needs container_owner == requested + // name; without it (pre-105 housekeeping record) it is withheld, since the + // sanitised container name cannot tell `a/b`'s container from `a-b`'s. + if f.namesContainer && len(f.containerOwners) == 0 { + return false + } + for _, owner := range f.containerOwners { + if owner != serverName { + return false + } + } + return true +} + +// decodeExactlyOneObject decodes s as exactly one complete JSON object with +// nothing after it, collecting the top-level fields the attribution rules +// use. A syntax error, a non-object value, a non-string value under an +// attribution key, or trailing bytes rejects the candidate. +func decodeExactlyOneObject(s string) (attributionFields, bool) { + var fields attributionFields + var hasLevel, hasTime, hasMessage bool + dec := json.NewDecoder(strings.NewReader(s)) + + tok, err := dec.Token() + if err != nil { + return attributionFields{}, false + } + if delim, ok := tok.(json.Delim); !ok || delim != '{' { + return attributionFields{}, false + } + + for dec.More() { + keyTok, err := dec.Token() + if err != nil { + return attributionFields{}, false + } + key, ok := keyTok.(string) + if !ok { + return attributionFields{}, false + } + var raw json.RawMessage + if err := dec.Decode(&raw); err != nil { + return attributionFields{}, false + } + switch key { + case attributionServerField: + value, ok := decodeStringValue(raw) + if !ok { + return attributionFields{}, false + } + fields.servers = append(fields.servers, value) + case attributionContainerOwnerField: + value, ok := decodeStringValue(raw) + if !ok { + return attributionFields{}, false + } + fields.containerOwners = append(fields.containerOwners, value) + case attributionContainerIDField, attributionContainerNameField, attributionContainerCountField: + // A count is a container subject too: a pre-105 sweep record's + // count of "existing containers" included a co-owner's. + fields.namesContainer = true + case attributionChildOutputField: + var flag bool + if err := json.Unmarshal(raw, &flag); err != nil { + return attributionFields{}, false + } + fields.childOutput = fields.childOutput || flag + case attributionChildMessageField, attributionChildErrorField: + // Only string values are child payload; anything else is not a + // child line and is left out of the subject check. + if value, ok := decodeStringValue(raw); ok { + fields.childPayload = append(fields.childPayload, value) + } + case jsonEncoderLevelKey: + hasLevel = true + case jsonEncoderTimeKey: + hasTime = true + case jsonEncoderMessageKey: + hasMessage = true + } + } + fields.jsonEncoderRecord = hasLevel && hasTime && hasMessage + + closeTok, err := dec.Token() + if err != nil { + return attributionFields{}, false + } + if delim, ok := closeTok.(json.Delim); !ok || delim != '}' { + return attributionFields{}, false + } + // Exactly one object: nothing may follow it. + if _, err := dec.Token(); !errors.Is(err, io.EOF) { + return attributionFields{}, false + } + return fields, true +} + +func decodeStringValue(raw json.RawMessage) (string, bool) { + var value string + if err := json.Unmarshal(raw, &value); err != nil { + return "", false + } + return value, true +} diff --git a/internal/logs/logger_attributed_test.go b/internal/logs/logger_attributed_test.go new file mode 100644 index 000000000..22dbed066 --- /dev/null +++ b/internal/logs/logger_attributed_test.go @@ -0,0 +1,941 @@ +package logs + +import ( + "io" + "os" + "path/filepath" + "regexp" + "runtime" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap" + + "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" +) + +// Spec 105 FR-007 (gaps FR007-G1, G2, G3, G6; research D8): two raw server +// names can share ONE per-server log file — `a/b` and `a_b` both sanitise to +// server-a_b.log — and the pre-105 tail reader returned the last N lines of +// that file unfiltered, so an `a_b`-only agent token read `a/b`'s records. +// The attributed reader must return only records whose writer stamp +// (`server=`, logger.go NewUpstreamServerLogger) is exactly the requested +// name, filter BEFORE taking the last N, withhold every line with no accepted +// stamp (legacy plain lines, torn fragments), and apply the subject-evidence +// rule to historical container / callback records. The whole-file reader — +// what administrators, REST and the CLI use — stays byte-identical (SC-005). + +// newAttributedLogDir returns a fresh log directory and a LogConfig pointing +// at it. os.MkdirTemp + best-effort RemoveAll (not t.TempDir) because the +// lumberjack sinks keep the file open until closed and a rotated backup can +// land after the closers ran; the existing logger_test.go uses the same +// pattern. +func newAttributedLogDir(t *testing.T, jsonFormat bool) *config.LogConfig { + t.Helper() + logDir, err := os.MkdirTemp("", "mcpproxy-attributed-*") + require.NoError(t, err) + t.Cleanup(func() { _ = os.RemoveAll(logDir) }) + + cfg := DefaultLogConfig() + cfg.LogDir = logDir + cfg.EnableFile = true + cfg.EnableConsole = false + cfg.JSONFormat = jsonFormat + cfg.Compress = false + return cfg +} + +// openStampedWriter returns the REAL per-server writer for name (the one +// internal/upstream/core installs as upstreamLogger) and closes its sink at +// test end. Every record it emits carries the `server=` stamp. +// +// The log file is pre-created so every sink opens it O_APPEND. lumberjack +// opens a NEW file O_TRUNC without O_APPEND, so when two writers share one +// file and the first creates it, the second's records are overwritten by the +// first's next write (the "torn fragment" corruption gap-map FR007-G3 probed; +// a retained effect, and torn fragments are non-attributable by rule). The +// fixtures here are about attribution, not about that corruption. +func openStampedWriter(t *testing.T, cfg *config.LogConfig, name string) *zap.Logger { + t.Helper() + logPath := filepath.Join(cfg.LogDir, ServerLogFilename(name)) + if _, err := os.Stat(logPath); os.IsNotExist(err) { + require.NoError(t, os.WriteFile(logPath, nil, 0o600)) + } + logger, closer, err := NewUpstreamServerLogger(cfg, name) + require.NoError(t, err) + t.Cleanup(func() { _ = closer.Close() }) + return logger +} + +// stampedAs reports whether a rendered line carries the writer stamp for name +// under either encoder (the console encoder renders `"server": "x"`, the JSON +// encoder `"server":"x"`). +func stampedAs(line, name string) bool { + return regexp.MustCompile(`"server":\s*"` + regexp.QuoteMeta(name) + `"`).MatchString(line) +} + +// writeRecord emits one ordinary record and flushes it. All fixture records +// go through this single call site so the console encoder's caller segment +// is identical across records and only the payload differs. +func writeRecord(logger *zap.Logger, msg string, fields ...zap.Field) { + logger.Info(msg, fields...) + _ = logger.Sync() +} + +// writeChildStderr mirrors the real child-stderr path exactly +// (internal/upstream/core/monitoring.go: +// Info("stderr", zap.String("message", line), logs.ChildOutputField())). +func writeChildStderr(logger *zap.Logger, line string) { + logger.Info("stderr", zap.String("message", line), ChildOutputField()) + _ = logger.Sync() +} + +// writeChildLauncherLine mirrors the launcher-pumped path +// (internal/upstream/core/connection_launcher.go loggerWriter.writeLine: +// Info("launcher", zap.String("message", line), logs.ChildOutputField())). +// Before codex round 2 the child's text was the MESSAGE there. +func writeChildLauncherLine(logger *zap.Logger, line string) { + logger.Info("launcher", zap.String("message", line), ChildOutputField()) + _ = logger.Sync() +} + +// writeChildStdoutMessage is the PRE-round-2 launcher shape — the child's +// text as the console-encoder message — kept so the reader's behaviour on +// files written by an older build stays pinned. +func writeChildStdoutMessage(logger *zap.Logger, line string) { + logger.Info(line) + _ = logger.Sync() +} + +func attributedTail(t *testing.T, cfg *config.LogConfig, name string, n int) []string { + t.Helper() + lines, err := ReadUpstreamServerLogTailAttributed(cfg, name, n) + require.NoError(t, err) + return lines +} + +func wholeFileTail(t *testing.T, cfg *config.LogConfig, name string, n int) []string { + t.Helper() + lines, err := ReadUpstreamServerLogTail(cfg, name, n) + require.NoError(t, err) + return lines +} + +func joinLines(lines []string) string { return strings.Join(lines, "\n") } + +type encoderCase struct { + name string + json bool +} + +func encoderCases() []encoderCase { + return []encoderCase{ + {"console_encoder", false}, + {"json_encoder", true}, + } +} + +// FR007-G1: `a/b` and `a_b` share one file; a sentinel written by `a/b` must +// never be returned for `a_b`, under both encoders. +func TestReadUpstreamServerLogTail_AttributedOnly_CollidingNames(t *testing.T) { + require.Equal(t, ServerLogFilename("a/b"), ServerLogFilename("a_b"), + "fixture premise: the two raw names must sanitise to one log file") + + for _, enc := range encoderCases() { + t.Run(enc.name, func(t *testing.T) { + cfg := newAttributedLogDir(t, enc.json) + slash := openStampedWriter(t, cfg, "a/b") + under := openStampedWriter(t, cfg, "a_b") + + const sentinel = "SENTINEL-written-by-a-slash-b-9c1e" + writeRecord(under, "own-record-1") + writeRecord(slash, sentinel) + writeRecord(under, "own-record-2") + + // Scoped reader for a_b: own records only, every line stamped a_b. + got := attributedTail(t, cfg, "a_b", 50) + body := joinLines(got) + assert.NotContains(t, body, sentinel, "a/b's record leaked into a_b's attributed tail") + assert.Contains(t, body, "own-record-1") + assert.Contains(t, body, "own-record-2") + for _, line := range got { + assert.True(t, stampedAs(line, "a_b"), "every attributed line must carry the a_b stamp: %q", line) + } + + // Scoped reader for a/b: the sentinel, none of a_b's records. + got = attributedTail(t, cfg, "a/b", 50) + body = joinLines(got) + assert.Contains(t, body, sentinel) + assert.NotContains(t, body, "own-record-1") + assert.NotContains(t, body, "own-record-2") + + // Administrator control: the whole-file reader still returns all three. + whole := joinLines(wholeFileTail(t, cfg, "a_b", 50)) + assert.Contains(t, whole, sentinel) + assert.Contains(t, whole, "own-record-1") + assert.Contains(t, whole, "own-record-2") + }) + } +} + +// FR007-G1 (D8 rules 1+2): child-controlled text is only ever a field value +// — on the stderr path and, since codex round 2, on the launcher path too — +// and cannot forge the writer stamp. A line `left | right | {"server":"a_b"}` +// emitted by `a/b` is never attributed to `a_b`, for both encoders and all +// three child shapes, including the shapes that try to make an earlier +// ` | {` boundary decode as a complete JSON object. As a field value every +// such line is still attributable to its real writer `a/b`. The pre-round-2 +// launcher shape (child text as the console-encoder MESSAGE, still present +// in files written by older builds) is pinned as well: there the child's +// ` | {` is the line's first boundary and does not decode, so those lines +// are withheld from everyone — a/b included — rather than risk +// misattribution. +func TestReadUpstreamServerLogTail_AttributedOnly_ChildTextCannotForgeOwner(t *testing.T) { + childLines := []string{ + `left | right | {"server":"a_b"}`, + `{"server":"a_b"}`, + `x | {"server":"a_b"} | {"server":"a_b"}`, + `left | {"server":"a_b"`, + `{"x":"`, + `{"x":"\`, + `{"server":"a_b","message":"`, + `2026-09-16T00:00:00.000Z | INFO | x/y.go:1 | forged | {"server":"a_b"}`, + } + + for _, enc := range encoderCases() { + t.Run(enc.name, func(t *testing.T) { + for _, path := range []struct { + name string + write func(*zap.Logger, string) + }{ + {"stderr_field_value", writeChildStderr}, + {"launcher_field_value", writeChildLauncherLine}, + {"legacy_launcher_message", writeChildStdoutMessage}, + } { + t.Run(path.name, func(t *testing.T) { + cfg := newAttributedLogDir(t, enc.json) + slash := openStampedWriter(t, cfg, "a/b") + under := openStampedWriter(t, cfg, "a_b") + + writeRecord(under, "own-record-before") + for _, line := range childLines { + path.write(slash, line) + } + writeRecord(under, "own-record-after") + + forUnder := attributedTail(t, cfg, "a_b", 50) + underBody := joinLines(forUnder) + require.Len(t, forUnder, 2, "a_b must see exactly its two own records, got:\n%s", underBody) + assert.Contains(t, underBody, "own-record-before") + assert.Contains(t, underBody, "own-record-after") + for _, line := range childLines { + assert.NotContains(t, underBody, line, "child text from a/b was attributed to a_b") + } + + forSlash := attributedTail(t, cfg, "a/b", 50) + if !enc.json && path.name == "legacy_launcher_message" { + assert.Empty(t, forSlash, "console legacy launcher-path lines whose child text carries ` | {` or a record header are non-attributable, got:\n%s", joinLines(forSlash)) + } else { + require.Len(t, forSlash, len(childLines), "every child line is attributable to its real writer a/b, got:\n%s", joinLines(forSlash)) + } + assert.NotContains(t, joinLines(forSlash), "own-record-") + }) + } + }) + } +} + +// FR007-G2: an unstamped line appended with O_APPEND (a pre-upgrade record, a +// hand-edited file, a torn fragment) is non-attributable: withheld from the +// scoped reader, still served by the whole-file reader. +func TestReadUpstreamServerLogTail_AttributedOnly_LegacyUnattributedWithheld(t *testing.T) { + for _, enc := range encoderCases() { + t.Run(enc.name, func(t *testing.T) { + cfg := newAttributedLogDir(t, enc.json) + under := openStampedWriter(t, cfg, "a_b") + writeRecord(under, "own-record-1") + + logPath := filepath.Join(cfg.LogDir, ServerLogFilename("a_b")) + f, err := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0o600) + require.NoError(t, err) + const legacy = "LEGACY_PLAIN_LINE no stamp at all" + const torn = ` | {"server":"a_b"` + _, err = io.WriteString(f, legacy+"\n"+torn+"\n") + require.NoError(t, err) + require.NoError(t, f.Close()) + + writeRecord(under, "own-record-2") + + got := attributedTail(t, cfg, "a_b", 50) + body := joinLines(got) + assert.NotContains(t, body, legacy, "unstamped legacy line served to the scoped reader") + assert.NotContains(t, body, torn, "torn fragment served to the scoped reader") + assert.Len(t, got, 2, "only the two stamped own records are attributable, got:\n%s", body) + + whole := joinLines(wholeFileTail(t, cfg, "a_b", 50)) + assert.Contains(t, whole, legacy, "administrator whole-file read must keep the legacy line") + assert.Contains(t, whole, torn) + }) + } +} + +// FR007-G2 (D8 rule 3, spec.md "legacy records whose stamped identity +// conflicts with their subject are withheld"): a stamp is not evidence that +// the record's SUBJECT belongs to the stamped server. Container records are +// attributable only with container_owner == requested server (the sanitised +// name is never evidence: a/b and a-b both sanitise to mcpproxy-a-b-*); +// callback records naming another server are withheld. Administrators see all. +func TestReadUpstreamServerLogTail_AttributedOnly_SubjectEvidence(t *testing.T) { + for _, enc := range encoderCases() { + t.Run(enc.name, func(t *testing.T) { + cfg := newAttributedLogDir(t, enc.json) + a := openStampedWriter(t, cfg, "a") + slash := openStampedWriter(t, cfg, "a/b") + + const foreignID = "deadbeef1234" + const foreignName = "mcpproxy-a-b-wxyz" + + // server-a.log — every record below is stamped server=a. + writeRecord(a, "own ordinary record") + // (1) pre-upgrade cleanup record naming a-b's container by name. + writeRecord(a, "Removing existing container", + zap.String("container_id", foreignID), + zap.String("container_name", foreignName), + zap.String("status", "Up 3 minutes")) + // (2) pre-upgrade record naming a container whose name LOOKS owned + // but carries no container_owner — the sanitised name is not evidence. + writeRecord(a, "Removing existing container", + zap.String("container_id", "cafe000000aa"), + zap.String("container_name", "mcpproxy-a-wxyz")) + // (3) ID-only record from the disconnect fallback. + writeRecord(a, "Killing container by name pattern", + zap.String("container_id", foreignID)) + // (4) callback-stop record written through a's logger but naming b. + writeRecord(a, "OAuth callback server stopped", + zap.String("server", "b"), + zap.String("bind_host", "127.0.0.1"), + zap.Int("port", 54321)) + // (5) post-upgrade record whose owner is another server. + writeRecord(a, "Removing existing container", + zap.String("container_id", "feedface0001"), + zap.String("container_name", "mcpproxy-a-wxyz"), + zap.String("container_owner", "a-b")) + // (6) callback-stop record naming a itself: subject matches. + writeRecord(a, "OAuth callback server stopped", + zap.String("server", "a"), + zap.String("bind_host", "127.0.0.1"), + zap.Int("port", 54322)) + // (7) post-upgrade housekeeping record owned by a. + writeRecord(a, "Removing existing container", + zap.String("container_id", "0123456789ab"), + zap.String("container_name", "mcpproxy-a-wxyz"), + zap.String("container_owner", "a")) + + got := attributedTail(t, cfg, "a", 50) + body := joinLines(got) + assert.Contains(t, body, "own ordinary record") + assert.NotContains(t, body, foreignID, "foreign container id disclosed to a's scoped reader") + assert.NotContains(t, body, foreignName, "foreign container name disclosed to a's scoped reader") + assert.NotContains(t, body, "cafe000000aa", "container record without container_owner must be withheld") + assert.NotContains(t, body, "54321", "callback record naming b's port disclosed to a") + assert.NotContains(t, body, "feedface0001", "container owned by a-b disclosed to a") + assert.Contains(t, body, "54322", "callback record naming a itself is attributable") + assert.Contains(t, body, "0123456789ab", "post-upgrade record with container_owner=a is attributable") + assert.Len(t, got, 3, "exactly: own ordinary, own callback-stop, own container record; got:\n%s", body) + + whole := joinLines(wholeFileTail(t, cfg, "a", 50)) + for _, s := range []string{foreignID, foreignName, "cafe000000aa", "54321", "feedface0001", "54322", "0123456789ab"} { + assert.Contains(t, whole, s, "administrator whole-file read must keep every record") + } + + // server-a_b.log — the same-sanitised-name case: `a/b` naming + // mcpproxy-a-b-wxyz without container_owner is indistinguishable + // from hidden `a-b`'s container and withheld; with container_owner + // it is returned. + writeRecord(slash, "Removing existing container", + zap.String("container_id", foreignID), + zap.String("container_name", foreignName)) + writeRecord(slash, "Removing existing container", + zap.String("container_id", "abcdef012345"), + zap.String("container_name", foreignName), + zap.String("container_owner", "a/b")) + + got = attributedTail(t, cfg, "a/b", 50) + body = joinLines(got) + assert.NotContains(t, body, foreignID, "ownerless container record served to a/b") + assert.Contains(t, body, "abcdef012345", "container_owner=a/b record must be returned to a/b") + assert.Len(t, got, 1, "got:\n%s", body) + + whole = joinLines(wholeFileTail(t, cfg, "a/b", 50)) + assert.Contains(t, whole, foreignID) + assert.Contains(t, whole, "abcdef012345") + }) + } +} + +// FR007-G3: ownership filtering precedes the tail limit, so an interleaved +// foreign line never displaces an authorized one from the returned window. +func TestReadUpstreamServerLogTail_AttributedOnly_InterleavedFilterBeforeLimit(t *testing.T) { + for _, enc := range encoderCases() { + t.Run(enc.name, func(t *testing.T) { + cfg := newAttributedLogDir(t, enc.json) + slash := openStampedWriter(t, cfg, "a/b") + under := openStampedWriter(t, cfg, "a_b") + + writeRecord(under, "own1") + writeRecord(slash, "foreign1") + writeRecord(under, "own2") + writeRecord(slash, "foreign2") + + got := attributedTail(t, cfg, "a_b", 2) + require.Len(t, got, 2, "tail(a_b, 2) must be the last two OWN records, got:\n%s", joinLines(got)) + assert.Contains(t, got[0], "own1") + assert.Contains(t, got[1], "own2") + assert.NotContains(t, joinLines(got), "foreign") + + // Administrator control: last two raw lines are own2, foreign2. + whole := wholeFileTail(t, cfg, "a_b", 2) + require.Len(t, whole, 2) + assert.Contains(t, whole[0], "own2") + assert.Contains(t, whole[1], "foreign2") + }) + } +} + +// FR007-G6 / T053: names differing only by case collide only on a +// case-insensitive filesystem (macOS default, Windows); Linux CI writes two +// files. The attributed reader must give each name only its own records in +// both regimes; the administrator outcome is recorded per regime. +func TestReadUpstreamServerLogTail_AttributedOnly_CaseOnlyNames(t *testing.T) { + cfg := newAttributedLogDir(t, false) + upper := openStampedWriter(t, cfg, "A") + lower := openStampedWriter(t, cfg, "a") + + writeRecord(lower, "lower-own-1") + writeRecord(upper, "UPPER-SENTINEL") + writeRecord(lower, "lower-own-2") + + upperInfo, err := os.Stat(filepath.Join(cfg.LogDir, ServerLogFilename("A"))) + require.NoError(t, err) + lowerInfo, err := os.Stat(filepath.Join(cfg.LogDir, ServerLogFilename("a"))) + require.NoError(t, err) + shared := os.SameFile(upperInfo, lowerInfo) + t.Logf("case-only names share one file on this filesystem: %v", shared) + + got := attributedTail(t, cfg, "a", 50) + body := joinLines(got) + assert.NotContains(t, body, "UPPER-SENTINEL", "A's record served to a's scoped reader") + assert.Contains(t, body, "lower-own-1") + assert.Contains(t, body, "lower-own-2") + assert.Len(t, got, 2, "got:\n%s", body) + + got = attributedTail(t, cfg, "A", 50) + body = joinLines(got) + assert.Contains(t, body, "UPPER-SENTINEL") + assert.NotContains(t, body, "lower-own") + assert.Len(t, got, 1, "got:\n%s", body) + + // Administrator outcome, recorded per regime: whole-file read of `a` + // includes A's record only when the filesystem folded the two names. + whole := joinLines(wholeFileTail(t, cfg, "a", 50)) + if shared { + assert.Contains(t, whole, "UPPER-SENTINEL", "shared file: administrators see both writers") + } else { + assert.NotContains(t, whole, "UPPER-SENTINEL", "separate files: nothing to share") + } +} + +// T053: shared-file rotation and retention stay shared (spec FR-007 retained +// effect). A hidden co-owner's output can rotate an authorized record out of +// the readable history; the attributed reader then returns only what is still +// attributable in the current file — and never the co-owner's records. +func TestReadUpstreamServerLogTail_AttributedOnly_ForcedRotationSharedHistory(t *testing.T) { + if runtime.GOOS == "windows" { + // The fixture needs two lumberjack sinks on one file and a rotation by + // the co-owner; Windows refuses the rename while the other sink holds + // the file open ("being used by another process"), so the premise + // cannot be established there. The reader logic under test is + // platform-neutral and is covered by the other cells. + t.Skip("shared-file rotation between two sinks cannot happen on Windows") + } + cfg := newAttributedLogDir(t, false) + cfg.MaxSize = 1 // MB — lumberjack's minimum; the co-owner forces one rotation + cfg.MaxBackups = 1 + slash := openStampedWriter(t, cfg, "a/b") + under := openStampedWriter(t, cfg, "a_b") + + const sentinel = "a_b-record-before-rotation-77b2" + writeRecord(under, sentinel) + + logPath := filepath.Join(cfg.LogDir, ServerLogFilename("a_b")) + + // a/b writes past MaxSize so lumberjack rotates the shared file. + filler := strings.Repeat("x", 1024) + for i := 0; i < 1100; i++ { + slash.Info("co-owner filler", zap.String("payload", filler)) + } + _ = slash.Sync() + + backups, err := filepath.Glob(filepath.Join(cfg.LogDir, "server-a_b-*.log")) + require.NoError(t, err) + require.NotEmpty(t, backups, "fixture premise: the co-owner's writes must have rotated %s", logPath) + + // Retained, documented effect: the pre-rotation own record is gone from + // the current file for everyone — administrators included. + whole := wholeFileTail(t, cfg, "a_b", 500) + assert.NotContains(t, joinLines(whole), sentinel, + "administrator outcome: a co-owner's rotation evicts the authorized record from the readable history") + require.NotEmpty(t, whole, "the current file holds the co-owner's post-rotation records") + + // The scoped reader receives no co-owner record — an empty tail is the + // correct answer here, a filler line is a disclosure. + got := attributedTail(t, cfg, "a_b", 500) + foreign := 0 + for _, line := range got { + if !stampedAs(line, "a_b") { + foreign++ + } + } + assert.Zero(t, foreign, "%d of %d lines served to a_b after rotation are not a_b's (co-owner filler disclosed)", foreign, len(got)) +} + +// Critique round 1, finding C1.2: one over-long record ANYWHERE in the shared +// file must not abort the scoped read. bufio.Scanner returns ErrTooLong for a +// line past its cap and the reader turned that into a tool error, so a hidden +// co-owner (or its child, whose lines pumpLines allows up to 1 MiB) could +// make `a_b`'s own tail fail until rotation — a response class that depends +// on the co-owner (SC-001). An over-long line is non-attributable and skipped, +// never fatal; the admin whole-file reader is untouched (SC-005). +func TestReadUpstreamServerLogTail_AttributedOnly_OverlongLineSkippedNotFatal(t *testing.T) { + cfg := newAttributedLogDir(t, false) + under := openStampedWriter(t, cfg, "a_b") + + writeRecord(under, "own-before-overlong") + + // A co-owner's over-long line, appended O_APPEND exactly as its sink + // would leave it: stamped for a/b, longer than the reader's line cap. + logPath := filepath.Join(cfg.LogDir, ServerLogFilename("a_b")) + f, err := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0o600) + require.NoError(t, err) + overlong := `2026-09-16T00:00:00.000Z | INFO | x/y.go:1 | ` + strings.Repeat("Q", 2*1024*1024) + ` | {"server": "a/b"}` + "\n" + _, err = f.WriteString(overlong) + require.NoError(t, err) + // And an over-long line stamped for a_b itself: withheld (non-attributable + // past the cap), still not fatal. + _, err = f.WriteString(`2026-09-16T00:00:00.000Z | INFO | x/y.go:1 | ` + strings.Repeat("R", 2*1024*1024) + ` | {"server": "a_b"}` + "\n") + require.NoError(t, err) + require.NoError(t, f.Close()) + + writeRecord(under, "own-after-overlong") + + got, err := ReadUpstreamServerLogTailAttributed(cfg, "a_b", 50) + require.NoError(t, err, "an over-long co-owner line must be skipped, not turned into a scoped-caller error") + body := joinLines(got) + require.Len(t, got, 2, "exactly the two own records, got %d lines", len(got)) + assert.Contains(t, body, "own-before-overlong") + assert.Contains(t, body, "own-after-overlong") + assert.NotContains(t, body, "QQQQ", "co-owner's over-long line disclosed") + assert.NotContains(t, body, "RRRR", "over-long own line must be withheld, not partially served") +} + +// Critique round 1, finding C1.5: `container_count` is a container subject +// too. A pre-105 "Cleaning up existing containers before creating new one" +// record carries only a count — a count larger than the server's own +// container count hints at a hidden co-owner — so a record naming a count +// without `container_owner` is withheld like any other ownerless container +// record; with `container_owner` == the requested server it is served. +func TestReadUpstreamServerLogTail_AttributedOnly_ContainerCountIsSubjectEvidence(t *testing.T) { + for _, enc := range encoderCases() { + t.Run(enc.name, func(t *testing.T) { + cfg := newAttributedLogDir(t, enc.json) + a := openStampedWriter(t, cfg, "a") + + writeRecord(a, "own ordinary record") + writeRecord(a, "Cleaning up existing containers before creating new one", + zap.Int("container_count", 7)) + writeRecord(a, "Cleaning up existing containers before creating new one", + zap.Int("container_count", 1), + zap.String("container_owner", "a")) + writeRecord(a, "Cleaning up existing containers before creating new one", + zap.Int("container_count", 3), + zap.String("container_owner", "a-b")) + + got := attributedTail(t, cfg, "a", 50) + body := joinLines(got) + assert.Contains(t, body, "own ordinary record") + assert.NotContains(t, body, `"container_count": 7`, "ownerless container_count record served to a's scoped reader") + assert.NotContains(t, body, `"container_count":7`) + assert.NotContains(t, body, `"container_owner": "a-b"`, "container_count record owned by a-b served to a") + assert.NotContains(t, body, `"container_owner":"a-b"`) + assert.Len(t, got, 2, "exactly: own ordinary + own-owned count record; got:\n%s", body) + + whole := joinLines(wholeFileTail(t, cfg, "a", 50)) + for _, s := range []string{"container_count", "a-b"} { + assert.Contains(t, whole, s, "administrator whole-file read must keep every record") + } + }) + } +} + +// Critique round 1, finding C2.2: the reader keys on LINE boundaries and the +// console encoder writes a message verbatim, so a message carrying a line +// break starts a new line whose text the message author controls. The reader +// cannot defend against that by construction; the guarantee is the +// PRODUCER's (internal/upstream/core loggerWriter splits child output on +// '\n' before logging; monitoring.go keeps stderr as a field value). This +// test pins that division of labour: a line break inside a message DOES +// forge a record for another server under the console encoder, so any new +// producer that logs child text as a message must split on '\n' first. +func TestReadUpstreamServerLogTail_AttributedOnly_LineBreakInMessageIsProducerGuarantee(t *testing.T) { + cfg := newAttributedLogDir(t, false) + slash := openStampedWriter(t, cfg, "a/b") + under := openStampedWriter(t, cfg, "a_b") + + writeRecord(under, "own-record") + const forged = `2026-01-01T00:00:00.000Z | INFO | x/y.go:1 | FORGED-for-a_b | {"server": "a_b"}` + // Bypasses every producer guard on purpose: the raw zap message. The + // forged line sits in the MIDDLE of the message: the encoder appends its + // own fields object to the message's last line, so a trailing forged line + // would carry the real stamp as trailing bytes and be rejected; a middle + // line stands alone. + slash.Info("harmless-prefix\n" + forged + "\nharmless-trailer") + _ = slash.Sync() + + got := attributedTail(t, cfg, "a_b", 50) + body := joinLines(got) + assert.Contains(t, body, "FORGED-for-a_b", + "the reader is expected to be unable to reject a forged line the producer let through — "+ + "if this now fails, the reader grew a defence and the producer-side comment in attribution.go should be revisited") + assert.Len(t, got, 2, "own record + the forged line, got:\n%s", body) + + // The JSON encoder escapes the break inside the message string, so the + // same input yields exactly one a/b record and nothing for a_b. + cfgJSON := newAttributedLogDir(t, true) + slashJSON := openStampedWriter(t, cfgJSON, "a/b") + slashJSON.Info("harmless-prefix\n" + forged + "\nharmless-trailer") + _ = slashJSON.Sync() + assert.Empty(t, attributedTail(t, cfgJSON, "a_b", 50), "JSON encoder must not let a message line break forge a record") + assert.Len(t, attributedTail(t, cfgJSON, "a/b", 50), 1) +} + +// Codex round 1 (PR E), finding 1: a torn foreign record (a partial final +// write — ENOSPC, a crash mid-write — with no terminator) followed by an +// O_APPEND write of a complete `a_b` record shares ONE physical line. The +// pre-fix reader rejected the fragment's ` | {` boundary (its suffix carries +// trailing bytes) and kept scanning, accepted a_b's later boundary, and +// returned the whole line — fragment included — to a_b. The accepted +// boundary must be the FIRST candidate on the line: an earlier candidate that +// does not decode is evidence of a torn or foreign prefix, and the whole line +// is non-attributable. Likewise a line that starts with `{` (a JSON-encoder +// record, or a torn one) is judged as that one object and never falls +// through to the console scan. The whole-file reader keeps the line. +func TestReadUpstreamServerLogTail_AttributedOnly_ConcatenatedTornFragmentWithheld(t *testing.T) { + const secret = "SECRET-a-b-cid" + fragments := []struct { + name string + torn string + }{ + {"console_fragment_torn_inside_fields", + `2026-09-16T00:00:00.000Z | INFO | x/y.go:1 | Killing owned container | {"server": "a/b", "container_id": "` + secret + `"`}, + {"console_fragment_torn_after_fields_key", + `2026-09-16T00:00:00.000Z | INFO | x/y.go:1 | stderr | {"server": "a/b", "message": "` + secret}, + {"json_fragment_torn_inside_fields", + `{"level":"info","ts":"2026-09-16T00:00:00Z","msg":"Killing owned container","server":"a/b","container_id":"` + secret + `"`}, + // Codex round 2, prior item: torn INSIDE the message part, before + // the fragment's own ` | {` boundary. The later record's boundary is + // then the line's FIRST boundary and decodes cleanly, so a + // first-boundary-only rule still handed the fragment to a_b. The + // console prefix must be exactly one record header + // (`ts | LEVEL | `): a second header in front of the boundary is a + // torn foreign record. + {"console_fragment_torn_inside_message", + `2026-09-16T00:00:00.000Z | INFO | x/y.go:1 | Killing owned container mcpproxy-a-b-wxyz ` + secret}, + {"console_fragment_torn_inside_caller", + `2026-09-16T00:00:00.000Z | INFO | x/` + secret}, + // Torn right after the caller separator: the prefix in front of + // the later record is a complete header and a caller, nothing else + // (under the JSON encoder the later record's boundary follows + // immediately and decodes — the suffix is a whole JSON record). + {"console_fragment_torn_after_caller_separator", + `2026-09-16T00:00:00.000Z | INFO | x/` + secret + `.go:1 | `}, + } + for _, enc := range encoderCases() { + for _, fr := range fragments { + t.Run(enc.name+"/"+fr.name, func(t *testing.T) { + cfg := newAttributedLogDir(t, enc.json) + under := openStampedWriter(t, cfg, "a_b") + writeRecord(under, "own-record-before") + + logPath := filepath.Join(cfg.LogDir, ServerLogFilename("a_b")) + f, err := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0o600) + require.NoError(t, err) + _, err = io.WriteString(f, fr.torn) // no terminator: torn + require.NoError(t, err) + require.NoError(t, f.Close()) + + // a_b's next record lands on the same physical line. + writeRecord(under, "own-record-concatenated") + writeRecord(under, "own-record-after") + + got := attributedTail(t, cfg, "a_b", 50) + body := joinLines(got) + assert.NotContains(t, body, secret, "torn a/b fragment served to a_b's scoped reader:\n%s", body) + assert.NotContains(t, body, "own-record-concatenated", "the line carrying the torn fragment must be withheld whole (a foreign caller or timestamp in front of it is a co-owner's)") + assert.Len(t, got, 2, "only the two clean own records are attributable, got:\n%s", body) + assert.Empty(t, attributedTail(t, cfg, "a/b", 50), "the torn line is attributable to nobody") + + whole := joinLines(wholeFileTail(t, cfg, "a_b", 50)) + assert.Contains(t, whole, secret, "administrator whole-file read keeps the torn line (SC-005)") + }) + } + } +} + +// Codex round 2 (PR E), NIT: the line cap is "longer than 1 MiB is +// non-attributable"; a record whose content is EXACTLY 1 MiB stays eligible. +// readBoundedLine compared the buffered length including the terminator +// ReadSlice returns, so a 1 MiB record measured 1 MiB + 1 and was withheld. +func TestReadUpstreamServerLogTail_AttributedOnly_ExactCapLineEligible(t *testing.T) { + cfg := newAttributedLogDir(t, false) + under := openStampedWriter(t, cfg, "a_b") + writeRecord(under, "own-before") + + const head = `2026-09-16T00:00:00.000Z | INFO | x/y.go:1 | ` + const tail = ` | {"server": "a_b"}` + pad := func(total int) string { return strings.Repeat("E", total-len(head)-len(tail)) } + exact := head + pad(attributedLineCap) + tail + require.Len(t, exact, attributedLineCap, "fixture premise: the record content is exactly the cap") + over := head + strings.Repeat("O", len(pad(attributedLineCap))+1) + tail + require.Len(t, over, attributedLineCap+1) + + logPath := filepath.Join(cfg.LogDir, ServerLogFilename("a_b")) + f, err := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0o600) + require.NoError(t, err) + _, err = f.WriteString(exact + "\n" + over + "\n") + require.NoError(t, err) + require.NoError(t, f.Close()) + writeRecord(under, "own-after") + + got := attributedTail(t, cfg, "a_b", 50) + body := joinLines(got) + assert.Contains(t, body, "EEEE", "a record of exactly the cap must stay eligible") + assert.NotContains(t, body, "OOOO", "one byte past the cap is non-attributable") + assert.Len(t, got, 3, "own-before, the exact-cap record, own-after; got %d lines", len(got)) + + // EOF without a terminator measures the same way. + cfgEOF := newAttributedLogDir(t, false) + pathEOF := filepath.Join(cfgEOF.LogDir, ServerLogFilename("a_b")) + require.NoError(t, os.WriteFile(pathEOF, []byte(exact), 0o600)) + got = attributedTail(t, cfgEOF, "a_b", 50) + assert.Len(t, got, 1, "an exact-cap final record with no terminator is eligible") +} + +// Codex round 2 (PR E), docker finding 1: Docker's own `docker run` failure +// names the colliding container — `a/b` and hidden `a-b` both generate +// mcpproxy-a-b-, and on a suffix collision the daemon answers with +// the FOREIGN container's id and name. That text reaches a/b's per-server +// log as child output (launcher-pumped docker stderr, or the stdio child's +// stderr). Child output is a field value, so it forges nothing, but it is +// still a container subject: a child-output record (`child_output=true`, +// ChildOutputField) whose text mentions a container id, a canonical +// container name or Docker's name-conflict phrase is withheld from the +// scoped reader unless container_owner matches (child output never carries +// one). Ordinary child output — and ordinary records that happen to carry a +// long hex string — stay attributable. Administrators keep every record. +func TestReadUpstreamServerLogTail_AttributedOnly_ChildOutputNamingContainerIsSubject(t *testing.T) { + const foreignID = "f0e1d2c3b4a5968778695a4b3c2d1e0ff0e1d2c3b4a5968778695a4b3c2d1e0f" + const foreignName = "mcpproxy-a-b-wxyz" + collision := `docker: Error response from daemon: Conflict. The container name "/` + foreignName + + `" is already in use by container "` + foreignID + `". You have to remove (or rename) that container to be able to reuse that name.` + + for _, enc := range encoderCases() { + for _, path := range []struct { + name string + write func(*zap.Logger, string) + }{ + {"stderr_field_value", writeChildStderr}, + {"launcher_field_value", writeChildLauncherLine}, + } { + t.Run(enc.name+"/"+path.name, func(t *testing.T) { + cfg := newAttributedLogDir(t, enc.json) + slash := openStampedWriter(t, cfg, "a/b") + + writeRecord(slash, "own ordinary record") + path.write(slash, "[launcher stderr] "+collision) + path.write(slash, "id only "+foreignID) + path.write(slash, "name only "+foreignName+" mentioned") + path.write(slash, "phrase only: is already in use by container") + path.write(slash, "listening on 127.0.0.1:9331") + // A NON-child record carrying a 64-hex value (a request hash) + // is not subject to the child-output rule. + writeRecord(slash, "tool call completed", zap.String("request_hash", foreignID)) + + got := attributedTail(t, cfg, "a/b", 50) + body := joinLines(got) + for _, line := range got { + if strings.Contains(line, "child_output") { + assert.NotContains(t, line, foreignID, "foreign container id served to a/b's scoped reader:\n%s", line) + assert.NotContains(t, line, foreignName, "foreign container name served to a/b's scoped reader:\n%s", line) + assert.NotContains(t, line, "already in use by container", "Docker collision text served to a/b's scoped reader:\n%s", line) + } + } + assert.Contains(t, body, "own ordinary record") + assert.Contains(t, body, "listening on 127.0.0.1:9331", "ordinary child output must stay attributable") + assert.Contains(t, body, "tool call completed", "the child-output rule must not withhold ordinary records") + assert.Len(t, got, 3, "own ordinary + ordinary child line + ordinary hash record; got:\n%s", body) + + whole := joinLines(wholeFileTail(t, cfg, "a/b", 50)) + assert.Contains(t, whole, foreignID, "administrator whole-file read keeps Docker's output (SC-005)") + assert.Contains(t, whole, foreignName) + }) + } + } +} + +// Codex round 3, logs finding 2: the container check ran over the WHOLE +// serialized record, so the writer stamp itself could match — a server +// legitimately named like a canonical container (`mcpproxy-tenant-abcd` +// matches mcpproxy--<4 alnum>) had every child-output record, even a +// plain "ready", classified as a container subject and withheld. Only the +// decoded child-controlled value is the subject; the stamp fields never are. +func TestReadUpstreamServerLogTail_AttributedOnly_ContainerShapedServerNameIsNotASubject(t *testing.T) { + const name = "mcpproxy-tenant-abcd" + require.Regexp(t, containerMentionPattern, name, "fixture premise: the server name matches the container pattern") + + for _, enc := range encoderCases() { + t.Run(enc.name, func(t *testing.T) { + cfg := newAttributedLogDir(t, enc.json) + w := openStampedWriter(t, cfg, name) + + writeRecord(w, "own ordinary record") + writeChildStderr(w, "ready") + writeChildLauncherLine(w, "[launcher stdout] listening on 127.0.0.1:9331") + // A child line that DOES name a container is still a subject. + writeChildStderr(w, `Conflict. The container name "/mcpproxy-tenant-zzzz" is already in use by container "f0e1d2c3b4a5968778695a4b3c2d1e0ff0e1d2c3b4a5968778695a4b3c2d1e0f".`) + + got := attributedTail(t, cfg, name, 50) + body := joinLines(got) + assert.Contains(t, body, "own ordinary record") + assert.Contains(t, body, "ready", "ordinary child output of a container-shaped server name must stay attributable") + assert.Contains(t, body, "listening on 127.0.0.1:9331") + assert.NotContains(t, body, "already in use by container") + assert.Len(t, got, 3, "own ordinary + two ordinary child lines; got:\n%s", body) + }) + } +} + +// Codex round 16 (PR E), finding 2 (SC-005 timing class): the pre-fix reader +// scanned the WHOLE shared file from byte 0 before taking the last N +// attributable records, so a scoped caller's response time was proportional +// to a hidden co-owner's entire earlier history in that file — a +// response-time side channel disclosing its volume, which the +// non-disclosing-refusal definition (status, body AND timing class) +// forbids. scopedBackwardStartOffset caps the scan to at most +// scopedBackwardReadBudget bytes before EOF regardless of the file's total +// size. This is the literal proof: for every fileSize tried — including +// values far larger than anything exercised elsewhere in this package — the +// bytes the scan will read (fileSize minus the returned offset) never +// exceeds the budget, so the read is provably NOT proportional to total +// file size. +func TestScopedBackwardStartOffset_BoundsReadRegardlessOfFileSize(t *testing.T) { + sizes := []int64{ + 0, + 1, + scopedBackwardReadBudget - 1, + scopedBackwardReadBudget, + scopedBackwardReadBudget + 1, + 2 * scopedBackwardReadBudget, + 10 * 1024 * 1024 * 1024, // 10 GiB, far beyond any file this package writes + } + for _, size := range sizes { + start := scopedBackwardStartOffset(size) + assert.GreaterOrEqualf(t, start, int64(0), "start offset must never be negative for fileSize=%d", size) + assert.LessOrEqualf(t, size-start, int64(scopedBackwardReadBudget), + "bytes read (fileSize-start) must never exceed the budget regardless of file size: fileSize=%d start=%d", size, start) + if size <= scopedBackwardReadBudget { + assert.Zerof(t, start, "a file at or under the budget is read in full from byte 0: fileSize=%d", size) + } + } +} + +// hugeCoOwnerFillerLine is one giant a/b-stamped console record: well past +// attributedLineCap (so it is non-attributable even when read whole) and +// well past scopedBackwardReadBudget in byte size — a hidden co-owner +// dominating the shared file exactly the way SC-005 forbids a scoped +// caller's timing from depending on. +func hugeCoOwnerFillerLine() string { + const fillerSize = 20 * 1024 * 1024 // > scopedBackwardReadBudget (16 MiB) + return `2026-09-16T00:00:00.000Z | INFO | x/y.go:1 | ` + strings.Repeat("F", fillerSize) + ` | {"server": "a/b"}` + "\n" +} + +// A hidden co-owner's tens-of-MB run sitting BEFORE this server's own recent +// records must not change the result: a scoped read of the shared file +// returns exactly what a scoped read of an otherwise-identical DEDICATED +// file (no co-owner at all) returns, because both sit well within +// scopedBackwardReadBudget of EOF. +func TestReadUpstreamServerLogTail_AttributedOnly_ScopedReadUnaffectedByHiddenCoOwnerBeyondBudget(t *testing.T) { + cfg := newAttributedLogDir(t, false) + under := openStampedWriter(t, cfg, "a_b") + + logPath := filepath.Join(cfg.LogDir, ServerLogFilename("a_b")) + f, err := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0o600) + require.NoError(t, err) + _, err = f.WriteString(hugeCoOwnerFillerLine()) + require.NoError(t, err) + require.NoError(t, f.Close()) + + writeRecord(under, "own-1") + writeRecord(under, "own-2") + writeRecord(under, "own-3") + + got := attributedTail(t, cfg, "a_b", 50) + body := joinLines(got) + require.Len(t, got, 3, "own-1, own-2, own-3 only; got %d lines", len(got)) + assert.Contains(t, body, "own-1") + assert.Contains(t, body, "own-2") + assert.Contains(t, body, "own-3") + assert.NotContains(t, body, "FFFF", "the hidden co-owner's filler must never be disclosed") + + cfgDedicated := newAttributedLogDir(t, false) + dedicated := openStampedWriter(t, cfgDedicated, "a_b") + writeRecord(dedicated, "own-1") + writeRecord(dedicated, "own-2") + writeRecord(dedicated, "own-3") + wantDedicated := attributedTail(t, cfgDedicated, "a_b", 50) + + // Same count and payload as the dedicated file — not a literal string + // comparison, since each writeRecord call site's own caller (file:line) + // legitimately differs between the two fixtures. + require.Len(t, wantDedicated, 3, "fixture premise: the dedicated-file control returns exactly the three own records") + for i, want := range []string{"own-1", "own-2", "own-3"} { + assert.Contains(t, wantDedicated[i], want) + assert.Contains(t, got[i], want, + "a scoped read of a small dedicated file and of an otherwise-identical shared file dominated by a hidden co-owner must return the same own records, in the same order") + } +} + +// A record buried more than scopedBackwardReadBudget bytes before EOF — +// behind a huge co-owner run written after it — is not found: the scan +// returns fewer records than requested rather than reading further. +// Bounded and fail-closed, not incorrect (per this fix's documented +// trade-off), and never an error. +func TestReadUpstreamServerLogTail_AttributedOnly_ScopedReadFailsClosedBeyondBudget(t *testing.T) { + cfg := newAttributedLogDir(t, false) + under := openStampedWriter(t, cfg, "a_b") + writeRecord(under, "own-buried") + + logPath := filepath.Join(cfg.LogDir, ServerLogFilename("a_b")) + f, err := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0o600) + require.NoError(t, err) + _, err = f.WriteString(hugeCoOwnerFillerLine()) + require.NoError(t, err) + require.NoError(t, f.Close()) + + got, err := ReadUpstreamServerLogTailAttributed(cfg, "a_b", 50) + require.NoError(t, err, "a request whose own recent records sit beyond the budget must return fewer records, never an error") + assert.NotContains(t, joinLines(got), "own-buried", + "the buried record sits outside the budget window: correctly withheld, not a bug") + assert.Empty(t, got) +} diff --git a/internal/oauth/callback_stop_logger_test.go b/internal/oauth/callback_stop_logger_test.go new file mode 100644 index 000000000..139f20a29 --- /dev/null +++ b/internal/oauth/callback_stop_logger_test.go @@ -0,0 +1,219 @@ +package oauth + +import ( + "fmt" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap" + "go.uber.org/zap/zaptest/observer" +) + +// Spec 105 FR-007 (gap FR007-G4, research D8 "subject-bound for shared-service +// producers"): the callback manager serves every server, and each callback +// server records the logger of the server it belongs to at start +// (CallbackServer.logger, a tee into that server's per-server log file). +// StopCallbackServer(name) — the nil-logger path markOAuthComplete uses — +// resolved its logger through adoptLoggerLocked(nil), i.e. whichever server's +// logger was installed LAST, so a's tear-down record (a's name, bind_host, +// port, dropped waiters) landed in b's log and was readable by a b-scoped +// agent through tail_log. The stop record must be written through the +// stopped server's own recorded logger, in both start orders. + +// newObservedManager builds a private CallbackServerManager (not the global +// one, so sibling tests cannot install a logger behind this test's back) +// plus one observer per server, mimicking the per-server upstream logger +// shape: every record is stamped `server=`. +func newObservedManager(t *testing.T, names ...string) (*CallbackServerManager, map[string]*observer.ObservedLogs, map[string]*zap.Logger) { + t.Helper() + mgr := &CallbackServerManager{ + servers: make(map[string]*CallbackServer), + logger: zap.NewNop(), + } + observed := make(map[string]*observer.ObservedLogs, len(names)) + loggers := make(map[string]*zap.Logger, len(names)) + for _, name := range names { + core, logs := observer.New(zap.DebugLevel) + observed[name] = logs + loggers[name] = zap.New(core).With(zap.String("server", name)) + } + t.Cleanup(func() { + for _, name := range names { + _ = mgr.StopCallbackServer(name) + } + }) + return mgr, observed, loggers +} + +// startObserved starts a dynamic-port callback server for name through the +// caller-logger path production uses (StartCallbackServerOnHost with +// CallbackBinding.Logger), parking one waiter so the tear-down has something +// to drop. +func startObserved(t *testing.T, mgr *CallbackServerManager, name string, logger *zap.Logger) *CallbackServer { + t.Helper() + cb, err := mgr.StartCallbackServerOnHost(name, CallbackBinding{Port: 0, Logger: logger}) + require.NoError(t, err) + cb.RegisterState("state-" + name) + return cb +} + +// mentionsServer reports whether any record in logs carries `server=name` +// as a field or names port anywhere — the two things FR-007 forbids leaking +// into another server's log. +func mentionsServer(logs *observer.ObservedLogs, name string, port int) []string { + var hits []string + for _, entry := range logs.All() { + for k, v := range entry.ContextMap() { + if k == "server" && v == name { + hits = append(hits, entry.Message+" server="+name) + } + if k == "port" && fmt.Sprint(v) == fmt.Sprint(port) { + hits = append(hits, entry.Message+" port="+fmt.Sprint(port)) + } + } + } + return hits +} + +// assertStopRoutedToOwner stops `stopped` via the nil-logger path and asserts +// its tear-down records (stop + dropped waiter) landed only in its own +// observer, never in `other`'s. +func assertStopRoutedToOwner(t *testing.T, mgr *CallbackServerManager, observed map[string]*observer.ObservedLogs, stopped, other string, stoppedPort int) { + t.Helper() + require.NoError(t, mgr.StopCallbackServer(stopped)) + + own := observed[stopped] + foreign := observed[other] + // The serve goroutine ALSO emits "OAuth callback server stopped" through + // the server's own logger once Serve returns, and it races the manager's + // record; wait for both so the count below is deterministic: goroutine + // record + manager record = 2 in the owner's log, 0 anywhere else. + require.Eventually(t, func() bool { + return len(own.FilterMessage("OAuth callback server stopped").All()) >= 2 + }, 2*time.Second, 10*time.Millisecond, + "%s's manager stop record must be written through %s's recorded logger (goroutine record + manager record)", stopped, stopped) + assert.Len(t, own.FilterMessage("OAuth callback server stopped").All(), 2, + "%s's manager stop record must be written through %s's recorded logger (goroutine record + manager record)", stopped, stopped) + assert.Len(t, own.FilterMessage("Stopped OAuth callback server while flows were still waiting").All(), 1, + "%s's dropped-waiter record must be written through %s's recorded logger", stopped, stopped) + + // The other server's observer keeps its OWN tear-down records from an + // earlier round; only records about `stopped` are forbidden there. + aboutStopped := foreign.FilterField(zap.String("server", stopped)) + assert.Empty(t, aboutStopped.FilterMessage("OAuth callback server stopped").All(), + "%s's stop record landed in %s's log", stopped, other) + assert.Empty(t, aboutStopped.FilterMessage("Stopped OAuth callback server while flows were still waiting").All(), + "%s's dropped-waiter record landed in %s's log", stopped, other) + assert.Empty(t, mentionsServer(foreign, stopped, stoppedPort), + "%s's name/port written into %s's log", stopped, other) +} + +// FR007-G4, order a then b: b's logger is the last installed, so on HEAD +// StopCallbackServer("a") logged through b's logger. +func TestCallbackStop_UsesRecordedServerLogger_ABOrder(t *testing.T) { + mgr, observed, loggers := newObservedManager(t, "a", "b") + + a := startObserved(t, mgr, "a", loggers["a"]) + b := startObserved(t, mgr, "b", loggers["b"]) + + assertStopRoutedToOwner(t, mgr, observed, "a", "b", a.Port) + assertStopRoutedToOwner(t, mgr, observed, "b", "a", b.Port) +} + +// FR007-G4, order b then a: a's logger is the last installed, so on HEAD +// StopCallbackServer("b") logged through a's logger. +func TestCallbackStop_UsesRecordedServerLogger_BAOrder(t *testing.T) { + mgr, observed, loggers := newObservedManager(t, "a", "b") + + b := startObserved(t, mgr, "b", loggers["b"]) + a := startObserved(t, mgr, "a", loggers["a"]) + + assertStopRoutedToOwner(t, mgr, observed, "b", "a", b.Port) + assertStopRoutedToOwner(t, mgr, observed, "a", "b", a.Port) +} + +// The explicit-logger stop path (StopCallbackServerWithLogger) is what the +// OAuth failure/cleanup paths use with the server's own logger; it must not +// regress to the manager logger either, and — subject-bound — must still not +// write into the other server's observer. +func TestCallbackStop_WithLogger_StillSubjectBound(t *testing.T) { + mgr, observed, loggers := newObservedManager(t, "a", "b") + + a := startObserved(t, mgr, "a", loggers["a"]) + startObserved(t, mgr, "b", loggers["b"]) + + require.NoError(t, mgr.StopCallbackServerWithLogger("a", loggers["a"])) + assert.NotEmpty(t, observed["a"].FilterMessage("OAuth callback server stopped").All()) + assert.Empty(t, mentionsServer(observed["b"], "a", a.Port), "a's name/port written into b's log") +} + +// Critique round 1, finding C2.5: the recorded server logger already carries +// server, bind_host and port as context fields (StartCallbackServerOnHost), +// so the stop and dropped-waiter records must not add them again — a JSON +// consumer keeps the last duplicate key, and the attributed reader's +// all-values-agree rule only tolerates the duplication. Exactly one +// bind_host and one port per manager stop record; `server` appears once from +// the recorded logger's With (the upstream tee's own stamp, which the +// observer fixture mimics, is a second, agreeing occurrence). +func TestCallbackStop_RecordFieldsNotDuplicated(t *testing.T) { + mgr, observed, loggers := newObservedManager(t, "a") + a := startObserved(t, mgr, "a", loggers["a"]) + + require.NoError(t, mgr.StopCallbackServer("a")) + + own := observed["a"] + require.Eventually(t, func() bool { + return len(own.FilterMessage("OAuth callback server stopped").All()) >= 2 + }, 2*time.Second, 10*time.Millisecond) + + countKey := func(entry observer.LoggedEntry, key string) int { + n := 0 + for _, f := range entry.Context { + if f.Key == key { + n++ + } + } + return n + } + for _, msg := range []string{"OAuth callback server stopped", "Stopped OAuth callback server while flows were still waiting"} { + for _, entry := range own.FilterMessage(msg).All() { + assert.Equal(t, 1, countKey(entry, "bind_host"), "%q: bind_host duplicated: %v", msg, entry.Context) + assert.Equal(t, 1, countKey(entry, "port"), "%q: port duplicated: %v", msg, entry.Context) + // fixture stamp (mimics the upstream tee) + recorded logger's With + assert.Equal(t, 2, countKey(entry, "server"), "%q: server stamped more than by the two loggers: %v", msg, entry.Context) + assert.Equal(t, int64(a.Port), entry.ContextMap()["port"], "%q must still name the port", msg) + assert.Equal(t, a.BindHost, entry.ContextMap()["bind_host"], "%q must still name the bind host", msg) + } + } +} + +// Codex round 2 (PR E), finding 2: the deprecated StartCallbackServer +// supplies no logger, and StartCallbackServerOnHost resolved a nil binding +// logger through adoptLoggerLocked(nil) — whichever server's logger was +// installed LAST. Starting b with loggerB and then a through the deprecated +// API recorded loggerB.With(server=a) as a's logger, so a's start records and +// its stop record (port, dropped waiters) were written into b's log. A +// server started without a logger of its own must record a subject-safe +// logger — never another server's — for its start AND stop records. +func TestCallbackStop_DeprecatedStartNeverAdoptsAnotherServersLogger(t *testing.T) { + mgr, observed, loggers := newObservedManager(t, "a", "b") + + b := startObserved(t, mgr, "b", loggers["b"]) + + a, err := mgr.StartCallbackServer("a", 0) + require.NoError(t, err) + a.RegisterState("state-a") + + assert.Empty(t, mentionsServer(observed["b"], "a", a.Port), + "a's start records (deprecated no-logger API) written through b's logger") + + require.NoError(t, mgr.StopCallbackServer("a")) + aboutA := observed["b"].FilterField(zap.String("server", "a")) + assert.Empty(t, aboutA.All(), "a's records landed in b's log") + assert.Empty(t, mentionsServer(observed["b"], "a", a.Port), "a's name/port written into b's log") + + // b's own tear-down still routes to b. + assertStopRoutedToOwner(t, mgr, observed, "b", "a", b.Port) +} diff --git a/internal/oauth/config.go b/internal/oauth/config.go index c800d61ff..e1a8bc3cc 100644 --- a/internal/oauth/config.go +++ b/internal/oauth/config.go @@ -118,6 +118,22 @@ func (m *CallbackServerManager) adoptLoggerLocked(logger *zap.Logger) *zap.Logge return m.logger } +// subjectLoggerLocked returns the logger one server's callback records are +// written through (Spec 105 FR-007, subject-bound routing): the caller's own +// logger — the tee into that server's per-server log — or, for a caller that +// supplies none (the deprecated StartCallbackServer), the zap global. It is +// never the manager logger: that is whichever server's logger was installed +// last, so resolving a nil logger through it recorded loggerB.With(server=a) +// for a server started without a logger and wrote a's start and stop records +// into b's log (codex round 2). A caller's logger is still adopted as the +// manager logger for the manager's own records. m.mu must be held. +func (m *CallbackServerManager) subjectLoggerLocked(logger *zap.Logger) *zap.Logger { + if logger == nil { + return zap.L().Named(oauthCallbackLoggerName) + } + return m.adoptLoggerLocked(logger) +} + // CallbackServer represents an active OAuth callback server. // // Callback parameters are dispatched by the `state` parameter (issue #975): @@ -1273,7 +1289,10 @@ func (b CallbackBinding) host() string { // Falls back to dynamic allocation if the preferred port is unavailable. // // Deprecated in favour of StartCallbackServerOnHost, which can bind IPv6 -// loopback and carries the caller's logger. Kept for callers that have neither. +// loopback and carries the caller's logger. Kept for callers that have +// neither; a server started here records the zap global as its logger (its +// records are not routed into any per-server log — never into another +// server's, Spec 105 FR-007). func (m *CallbackServerManager) StartCallbackServer(serverName string, preferredPort int) (*CallbackServer, error) { return m.StartCallbackServerOnHost(serverName, CallbackBinding{Port: preferredPort}) } @@ -1300,7 +1319,7 @@ func (m *CallbackServerManager) StartCallbackServerOnHost(serverName string, bin m.mu.Lock() defer m.mu.Unlock() - logger := m.adoptLoggerLocked(binding.Logger) + logger := m.subjectLoggerLocked(binding.Logger) bindHost := binding.host() preferredPort := binding.Port @@ -1687,31 +1706,58 @@ func (m *CallbackServerManager) StopCallbackServer(serverName string) error { return m.StopCallbackServerWithLogger(serverName, nil) } -// StopCallbackServerWithLogger is StopCallbackServer with the caller's logger, -// so the tear-down (and any waiter it drops) is actually recorded. +// StopCallbackServerWithLogger is StopCallbackServer with the caller's logger. +// The signature is kept for its callers; the tear-down records are written +// through the stopped server's OWN recorded logger (Spec 105 FR-007, +// subject-bound routing), and the caller's logger is only the fallback for a +// server that recorded none. Stopping never adopts a logger as the manager +// logger: the manager serves every server, and the last-installed logger is a +// tee into whichever server's log file ran a flow last — pre-105 that is where +// another server's name, bind host, port and dropped-waiter count landed. func (m *CallbackServerManager) StopCallbackServerWithLogger(serverName string, logger *zap.Logger) error { m.mu.Lock() defer m.mu.Unlock() - return m.stopCallbackServerLocked(serverName, m.adoptLoggerLocked(logger)) + return m.stopCallbackServerLocked(serverName, logger) } // stopCallbackServerLocked shuts the server down and removes it from the map. -// m.mu must be held. -func (m *CallbackServerManager) stopCallbackServerLocked(serverName string, logger *zap.Logger) error { +// m.mu must be held. fallback is consulted only when the server recorded no +// logger of its own (every server started through StartCallbackServerOnHost +// records one); a nil fallback resolves to the manager logger. +func (m *CallbackServerManager) stopCallbackServerLocked(serverName string, fallback *zap.Logger) error { server, exists := m.servers[serverName] if !exists { return nil // Already stopped or never started } + // Subject-bound (FR-007): the record concerns `server`, so it is written + // through the logger recorded for that server at start — the tee into + // ITS per-server log — never through whichever logger was installed last. + // The recorded logger already carries server, bind_host and port as + // context fields (StartCallbackServerOnHost), so the records below add + // none of them; a fallback logger gets the same three fields once. + logger := server.logger + if logger == nil { + logger = fallback + if logger == nil { + logger = m.logger + } + if logger == nil { + logger = zap.L().Named(oauthCallbackLoggerName) + } + logger = logger.With( + zap.String("server", serverName), + zap.String("bind_host", server.BindHost), + zap.Int("port", server.Port)) + } + // Shutdown the server ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() if err := server.Server.Shutdown(ctx); err != nil { - logger.Error("Error shutting down OAuth callback server", - zap.String("server", serverName), - zap.Error(err)) + logger.Error("Error shutting down OAuth callback server", zap.Error(err)) } // Drop any registered waiters. They unblock on their own context deadline; @@ -1719,17 +1765,13 @@ func (m *CallbackServerManager) stopCallbackServerLocked(serverName string, logg // zero-value receive for a real callback. if dropped := server.dropAllWaiters(); dropped > 0 { logger.Warn("Stopped OAuth callback server while flows were still waiting", - zap.String("server", serverName), zap.Int("waiters", dropped)) } // Remove from map delete(m.servers, serverName) - logger.Info("OAuth callback server stopped", - zap.String("server", serverName), - zap.String("bind_host", server.BindHost), - zap.Int("port", server.Port)) + logger.Info("OAuth callback server stopped") return nil } diff --git a/internal/server/mcp.go b/internal/server/mcp.go index 979cbb6cf..95e3be08a 100644 --- a/internal/server/mcp.go +++ b/internal/server/mcp.go @@ -4081,7 +4081,9 @@ func (p *MCPProxyServer) handleListUpstreams(ctx context.Context) (*mcp.CallTool } // Spec 028: Filter servers to only those the agent token can access - if authCtx := auth.AuthContextFromContext(ctx); authCtx != nil && !authCtx.IsAdmin() { + authCtx := auth.AuthContextFromContext(ctx) + scopedCaller := authCtx != nil && !authCtx.IsAdmin() + if scopedCaller { var filtered []*config.ServerConfig for _, s := range servers { if authCtx.CanAccessServer(s.Name) { @@ -4202,6 +4204,14 @@ func (p *MCPProxyServer) handleListUpstreams(ctx context.Context) (*mcp.CallTool if !revealHeaders { lastError = scrubUpstreamText(lastError) } + // Spec 105 FR-007 (codex round 3): the error re-emits the + // child's stderr, which on a `docker run` name collision + // names another server's container. Redacted for scoped + // callers before it reaches connection_status.last_error and + // health.detail; administrators keep it (SC-005). + if scopedCaller { + lastError = logs.RedactContainerMentions(lastError) + } } isConnected = connInfo.State.String() == "connected" userLoggedOut = client.IsUserLoggedOut() @@ -6095,8 +6105,22 @@ func (p *MCPProxyServer) handleTailLog(ctx context.Context, request mcp.CallTool } } - // Read log tail - logLines, err := logs.ReadUpstreamServerLogTail(logConfig, name, lines) + // Read log tail. Spec 105 FR-007 (research D8): two raw names can share + // one log file (`a/b` and `a_b` both sanitise to server-a_b.log), so a + // scoped caller receives only the records attributable to the server it + // asked for — filtered BEFORE the tail limit, so an interleaved co-owner + // record never displaces an authorized one, and lines_returned counts the + // authorized tail. The policy is uniform whether or not a co-owner exists: + // legacy records with no writer stamp are withheld either way, never a + // whole-file refusal. Administrators (nil AuthContext, API key, socket) + // keep the whole file exactly as before (SC-005) — a profile scope bounds + // WHICH server they may name (above), not which records of it they see. + var logLines []string + if authCtx == nil || authCtx.IsAdmin() { + logLines, err = logs.ReadUpstreamServerLogTail(logConfig, name, lines) + } else { + logLines, err = logs.ReadUpstreamServerLogTailAttributed(logConfig, name, lines) + } if err != nil { return mcp.NewToolResultError(fmt.Sprintf("Failed to read log for server '%s': %v", name, err)), nil } @@ -6124,7 +6148,18 @@ func (p *MCPProxyServer) handleTailLog(ctx context.Context, request mcp.CallTool connectionStatus := client.GetConnectionStatus() // last_error commonly echoes the upstream URL, credentials included. if lastError, ok := connectionStatus["last_error"].(string); ok { - connectionStatus["last_error"] = scrubUpstreamText(lastError) + lastError = scrubUpstreamText(lastError) + // Spec 105 FR-007 (codex round 3): a connect error re-emits the + // child's stderr, and on a `docker run` name collision that names + // another server's container (`a/b` and `a-b` generate the same + // name). Scoped callers get container mentions redacted — the + // same predicate the attributed reader applies to the log record + // — uniformly, whether or not a co-owner exists; administrators + // keep the text (SC-005). + if authCtx != nil && !authCtx.IsAdmin() { + lastError = logs.RedactContainerMentions(lastError) + } + connectionStatus["last_error"] = lastError } result["connection_status"] = connectionStatus } diff --git a/internal/server/mcp_secret_redaction_test.go b/internal/server/mcp_secret_redaction_test.go index 9c7d198cb..546f7d892 100644 --- a/internal/server/mcp_secret_redaction_test.go +++ b/internal/server/mcp_secret_redaction_test.go @@ -285,6 +285,14 @@ func TestScrubUpstreamText_ConnectionErrors(t *testing.T) { // attempt, and connection_launcher.go pipes the child process's own stdout into // the same file. `tail_log` returned those lines verbatim and recorded them // into the activity store. +// +// The fixture records carry the writer stamp `server=leaky` in both encoder +// shapes (Spec 105 FR-007: a scoped caller receives only attributable +// records, so an unstamped fixture line would now be withheld before the +// scrubber ever saw it and the scoped assertions would pass vacuously). They +// are written by hand rather than through the per-server writer because that +// writer's own sanitizer would mask the credentials at write time — the point +// here is the scrub on the READ path, for scoped and administrator callers. func TestTailLog_ScrubsLogLines(t *testing.T) { proxy := createTestMCPProxyServer(t) @@ -301,20 +309,35 @@ func TestTailLog_ScrubsLogLines(t *testing.T) { require.NoError(t, proxy.storage.SaveUpstreamServer(&config.ServerConfig{ Name: "leaky", Protocol: "http", Enabled: true, })) + const childToken = "ghp_abcdefghijklmnopqrstuvwxyz0123456789" require.NoError(t, os.WriteFile(filepath.Join(logDir, "server-leaky.log"), []byte( - `{"level":"info","msg":"Starting connection attempt","url":"https://host/mcp?token=`+leakySecrets["url"]+`"}`+"\n"+ - `child stdout: using ghp_abcdefghijklmnopqrstuvwxyz0123456789`+"\n"), 0o600)) - - request := mcp.CallToolRequest{} - request.Params.Arguments = map[string]interface{}{"name": "leaky"} - - result, err := proxy.handleTailLog(context.Background(), request) - require.NoError(t, err) - body := toolResultText(t, result) - - assert.NotContains(t, body, leakySecrets["url"], "tail_log leaks the URL credential mcpproxy itself logged") - assert.NotContains(t, body, "ghp_abcdefghijklmnopqrstuvwxyz0123456789") - assert.Contains(t, body, "Starting connection attempt", "the diagnostic content must survive") + // JSON-encoder shape: the connection logger's URL record. + `{"level":"info","msg":"Starting connection attempt","server":"leaky","url":"https://host/mcp?token=`+leakySecrets["url"]+`"}`+"\n"+ + // Console-encoder shape: the launcher-pumped child stdout line is the + // `message` field of a child_output record (Spec 105 PR E, codex round 2). + `2026-01-01T00:00:00.000Z | INFO | core/connection_launcher.go:1 | launcher | {"server": "leaky", "message": "[launcher stdout] child stdout: using `+childToken+`", "child_output": true}`+"\n"), 0o600)) + + for name, ctx := range map[string]context.Context{ + "scoped agent token": agentCtx([]string{"leaky"}, []string{auth.PermRead}, ""), + "administrator": adminCtx(), + "no auth ctx (in-proc)": context.Background(), + } { + t.Run(name, func(t *testing.T) { + request := mcp.CallToolRequest{} + request.Params.Arguments = map[string]interface{}{"name": "leaky"} + + result, err := proxy.handleTailLog(ctx, request) + require.NoError(t, err) + body := toolResultText(t, result) + require.False(t, result.IsError, body) + + assert.NotContains(t, body, leakySecrets["url"], "tail_log leaks the URL credential mcpproxy itself logged") + assert.NotContains(t, body, childToken) + assert.Contains(t, body, "Starting connection attempt", "the diagnostic content must survive") + assert.Contains(t, body, "child stdout: using", "the child's own line must survive (scrubbed)") + assert.Contains(t, body, `"lines_returned":2`, "both stamped records are attributable to leaky: %s", body) + }) + } } // TestArgvMaskEcho_GuardsTheWritePath is the write-path counterpart to masking diff --git a/internal/server/mcp_tail_log_scope_test.go b/internal/server/mcp_tail_log_scope_test.go index 803678c28..477351ca4 100644 --- a/internal/server/mcp_tail_log_scope_test.go +++ b/internal/server/mcp_tail_log_scope_test.go @@ -2,6 +2,8 @@ package server import ( "context" + "encoding/json" + "fmt" "os" "path/filepath" "strings" @@ -14,13 +16,23 @@ import ( "github.com/smart-mcp-proxy/mcpproxy-go/internal/auth" "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/dockernaming" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/logs" "github.com/smart-mcp-proxy/mcpproxy-go/internal/profile" ) -// tailLogCanary is written into every fixture server's log. Its presence in a -// tool response proves the log was disclosed. +// tailLogCanary is written into every fixture server's log through the REAL +// stamped per-server writer. Its presence in a tool response proves the log +// was disclosed. const tailLogCanary = "CANARY-upstream-log-line-7f3a" +// tailLogLegacyLine is appended to every fixture server's log WITHOUT a +// writer stamp (a pre-105 record). Spec 105 FR-007: a record with no +// attribution is withheld from scoped callers and kept for administrators — +// pre-105 this fixture's canary was itself an unstamped line served to the +// scoped token, which the attributed reader now (correctly) withholds. +const tailLogLegacyLine = "LEGACY-unstamped-log-line-2b61" + // newTailLogScopeProxy builds a proxy with two upstreams, "github" and // "secret", each with a per-server log file AND a registered (never // connected) upstream client so a served response carries connection_status, @@ -60,8 +72,15 @@ func newTailLogScopeProxy(t *testing.T) *MCPProxyServer { // served response includes connection_status (otherwise the // "connection status not disclosed" assertions would pass vacuously). require.NoError(t, proxy.upstreamManager.AddServerConfig(name, sc)) + // A pre-105 unstamped record first, then the canary through the real + // stamped writer (the one internal/upstream/core installs). require.NoError(t, os.WriteFile(filepath.Join(logDir, "server-"+name+".log"), - []byte(tailLogCanary+" "+name+"\n"), 0o600)) + []byte(tailLogLegacyLine+" "+name+"\n"), 0o600)) + writer, closer, err := logs.NewUpstreamServerLogger(cfg.Logging, name) + require.NoError(t, err) + t.Cleanup(func() { _ = closer.Close() }) + writer.Info(tailLogCanary + " " + name) + _ = writer.Sync() } return proxy } @@ -105,15 +124,34 @@ func assertTailLogHidden(t *testing.T, proxy *MCPProxyServer, ctx context.Contex assert.NotContains(t, body, "connection_status", "connection status disclosed") } -// assertTailLogServed asserts the in-scope / admin path returns the log, -// the stored flags and the live connection status. -func assertTailLogServed(t *testing.T, proxy *MCPProxyServer, ctx context.Context, name string) { +// assertTailLogServed asserts the in-scope / admin path returns the stamped +// log record, the stored flags and the live connection status, and returns +// the body for the caller's attribution assertions. +func assertTailLogServed(t *testing.T, proxy *MCPProxyServer, ctx context.Context, name string) string { t.Helper() result, body := tailLogVia(t, proxy, ctx, name) assert.False(t, result.IsError, "in-scope tail_log must succeed: %s", body) assert.Contains(t, body, tailLogCanary+" "+name) assert.Contains(t, body, "server_status") assert.Contains(t, body, "connection_status", "fixture must register a client, or the non-disclosure assertions prove nothing") + return body +} + +// assertTailLogServedScoped is assertTailLogServed for a scoped caller: the +// stamped record is served, the unstamped legacy record is withheld +// (Spec 105 FR-007 — attribution is uniform, co-owner or not). +func assertTailLogServedScoped(t *testing.T, proxy *MCPProxyServer, ctx context.Context, name string) { + t.Helper() + body := assertTailLogServed(t, proxy, ctx, name) + assert.NotContains(t, body, tailLogLegacyLine, "unattributed legacy record served to a scoped caller") +} + +// assertTailLogServedWholeFile is assertTailLogServed for an administrator: +// the whole file, legacy record included (SC-005). +func assertTailLogServedWholeFile(t *testing.T, proxy *MCPProxyServer, ctx context.Context, name string) { + t.Helper() + body := assertTailLogServed(t, proxy, ctx, name) + assert.Contains(t, body, tailLogLegacyLine+" "+name, "administrators keep the whole file") } func TestTailLog_ServerRestrictedToken_HidesOutOfScopeServer(t *testing.T) { @@ -126,7 +164,7 @@ func TestTailLog_ServerRestrictedToken_HidesOutOfScopeServer(t *testing.T) { }) assertTailLogHidden(t, proxy, ctx, "secret") - assertTailLogServed(t, proxy, ctx, "github") + assertTailLogServedScoped(t, proxy, ctx, "github") } func TestTailLog_ProfilePinnedToken_HidesServerOutsideProfile(t *testing.T) { @@ -140,7 +178,7 @@ func TestTailLog_ProfilePinnedToken_HidesServerOutsideProfile(t *testing.T) { }) assertTailLogHidden(t, proxy, ctx, "secret") - assertTailLogServed(t, proxy, ctx, "github") + assertTailLogServedScoped(t, proxy, ctx, "github") } // A pin whose profile no longer exists resolves to a deny-all scope (see @@ -164,18 +202,18 @@ func TestTailLog_AdminUnchanged(t *testing.T) { proxy := newTailLogScopeProxy(t) adminCtx := auth.WithAuthContext(context.Background(), &auth.AuthContext{Type: auth.AuthTypeAdmin}) - assertTailLogServed(t, proxy, adminCtx, "secret") - assertTailLogServed(t, proxy, adminCtx, "github") + assertTailLogServedWholeFile(t, proxy, adminCtx, "secret") + assertTailLogServedWholeFile(t, proxy, adminCtx, "github") // No AuthContext at all (in-process / stdio caller) is treated as admin by // the shared server-op policy; unchanged here. - assertTailLogServed(t, proxy, context.Background(), "secret") + assertTailLogServedWholeFile(t, proxy, context.Background(), "secret") // An admin's AllowedServers is never consulted, even when populated. narrowAdmin := auth.WithAuthContext(context.Background(), &auth.AuthContext{ Type: auth.AuthTypeAdmin, AllowedServers: []string{"github"}, }) - assertTailLogServed(t, proxy, narrowAdmin, "secret") + assertTailLogServedWholeFile(t, proxy, narrowAdmin, "secret") } // An explicit URL profile (/mcp/p/) bounds tail_log for every caller, @@ -185,12 +223,370 @@ func TestTailLog_URLProfileScope_AppliesToAllCallers(t *testing.T) { proxy := newTailLogScopeProxy(t) scope := profile.NewProfileScope("gh", []string{"github"}) + // A profile bounds WHICH server an administrator may name, not which + // records of it they see: still the whole file. adminInProfile := profile.WithProfileScope( auth.WithAuthContext(context.Background(), &auth.AuthContext{Type: auth.AuthTypeAdmin}), scope) assertTailLogHidden(t, proxy, adminInProfile, "secret") - assertTailLogServed(t, proxy, adminInProfile, "github") + assertTailLogServedWholeFile(t, proxy, adminInProfile, "github") anonInProfile := profile.WithProfileScope(context.Background(), scope) assertTailLogHidden(t, proxy, anonInProfile, "secret") - assertTailLogServed(t, proxy, anonInProfile, "github") + assertTailLogServedWholeFile(t, proxy, anonInProfile, "github") +} + +// --------------------------------------------------------------------------- +// Spec 105 FR-007 (gaps FR007-G1, G3): colliding log files. `a/b` and `a_b` +// both sanitise to server-a_b.log, so an `a_b`-only token must receive only +// the records `a_b` wrote (filtered BEFORE the tail limit, lines_returned = +// filtered length) and administrators must keep the whole file byte-for-byte. +// --------------------------------------------------------------------------- + +// tailLogCollidingFixture is a proxy whose storage knows `a/b` and `a_b` and +// whose log directory holds their SHARED file, plus the two real stamped +// writers (logs.NewUpstreamServerLogger — the writer internal/upstream/core +// installs) so every record carries the `server=` stamp. +type tailLogCollidingFixture struct { + proxy *MCPProxyServer + logCfg *config.LogConfig + writers map[string]*zap.Logger +} + +const ( + collidingHidden = "a/b" + collidingOwn = "a_b" +) + +// newTailLogCollidingProxy builds the fixture. Every returned io.Closer is +// closed at cleanup (CI "directory not empty" otherwise). The shared file is +// pre-created so both lumberjack sinks open it O_APPEND — lumberjack creates +// a NEW file O_TRUNC without O_APPEND, and two writers on one fresh file +// overwrite each other (the torn-fragment corruption gap-map FR007-G3 probed, +// a retained effect that is not what these tests are about). +func newTailLogCollidingProxy(t *testing.T) *tailLogCollidingFixture { + t.Helper() + return newTailLogProxyWithServers(t, collidingHidden, collidingOwn) +} + +// newTailLogProxyWithServers is newTailLogCollidingProxy for an explicit set +// of registered servers, so a differential can run a TRUE absent-co-owner +// arm (only `a_b` configured, no `a/b` anywhere: not in storage, not in the +// upstream manager, no writer) against the colliding one. +func newTailLogProxyWithServers(t *testing.T, names ...string) *tailLogCollidingFixture { + t.Helper() + require.Equal(t, logs.ServerLogFilename(collidingHidden), logs.ServerLogFilename(collidingOwn), + "fixture premise: the two raw names must share one log file") + + proxy := createTestMCPProxyServer(t) + + logDir := t.TempDir() + cfg := config.DefaultConfig() + cfg.DataDir = t.TempDir() + cfg.Listen = "127.0.0.1:0" + cfg.Logging.LogDir = logDir + cfg.Logging.EnableFile = true + cfg.Logging.EnableConsole = false + cfg.Logging.Compress = false + for _, name := range names { + cfg.Servers = append(cfg.Servers, &config.ServerConfig{Name: name, Protocol: "http", Enabled: false}) + } + mainSrv, err := NewServer(cfg, zap.NewNop()) + require.NoError(t, err) + t.Cleanup(func() { _ = mainSrv.Shutdown() }) + proxy.mainServer = mainSrv + + require.NoError(t, os.WriteFile(filepath.Join(logDir, logs.ServerLogFilename(collidingOwn)), nil, 0o600)) + + f := &tailLogCollidingFixture{proxy: proxy, logCfg: cfg.Logging, writers: map[string]*zap.Logger{}} + for _, name := range names { + sc := &config.ServerConfig{Name: name, Protocol: "http", URL: "http://127.0.0.1:1/mcp", Enabled: true} + require.NoError(t, proxy.storage.SaveUpstreamServer(sc)) + require.NoError(t, proxy.upstreamManager.AddServerConfig(name, sc)) + + writer, closer, err := logs.NewUpstreamServerLogger(cfg.Logging, name) + require.NoError(t, err) + t.Cleanup(func() { _ = closer.Close() }) + f.writers[name] = writer + } + return f +} + +// write emits one record through name's real stamped writer. The writer +// records its CALLER's caller (NewUpstreamServerLogger adds one frame of +// skip), i.e. the test line that called write, so two fixtures' lines differ +// by timestamp and caller segment — both stripped by tailLogLineSignature. +func (f *tailLogCollidingFixture) write(name, msg string) { + f.writers[name].Info(msg) + _ = f.writers[name].Sync() +} + +// tailLogResponse is the parsed tail_log payload. +type tailLogResponse struct { + ServerName string `json:"server_name"` + LinesRequested int `json:"lines_requested"` + LinesReturned int `json:"lines_returned"` + LogLines []string `json:"log_lines"` +} + +// tailLogLinesVia drives the real dispatcher with an explicit `lines` and +// parses the payload. +func tailLogLinesVia(t *testing.T, proxy *MCPProxyServer, ctx context.Context, name string, lines int) (tailLogResponse, string) { + t.Helper() + request := mcp.CallToolRequest{} + request.Params.Arguments = map[string]interface{}{"operation": "tail_log", "name": name, "lines": float64(lines)} + result, err := proxy.handleUpstreamServers(ctx, request) + require.NoError(t, err) + body := toolResultText(t, result) + require.False(t, result.IsError, "tail_log must succeed for the in-scope server: %s", body) + var parsed tailLogResponse + require.NoError(t, json.Unmarshal([]byte(body), &parsed), body) + return parsed, body +} + +// tailLogLineSignature strips the timestamp and caller segments of a console +// record (`ts | LEVEL | caller | msg | {fields}`) so records written by two +// fixtures compare on level, message and fields only. +func tailLogLineSignature(line string) string { + parts := strings.SplitN(line, " | ", 4) + if len(parts) < 4 { + return line + } + return parts[1] + " | " + parts[3] +} + +func tailLogSignatures(lines []string) []string { + out := make([]string, len(lines)) + for i, l := range lines { + out[i] = tailLogLineSignature(l) + } + return out +} + +// FR007-G1 + G3 at the tool surface: interleaved own/foreign records, an +// `a_b`-only token asks for the last 2 → exactly [own1, own2], +// lines_returned == 2, nothing from `a/b`. +func TestTailLog_CollidingLogFile_ScopedTokenGetsOnlyOwnRecords(t *testing.T) { + f := newTailLogCollidingProxy(t) + const sentinel = "SENTINEL-a-slash-b-only-4e2d" + f.write(collidingOwn, "own1") + f.write(collidingHidden, sentinel+"-1") + f.write(collidingOwn, "own2") + f.write(collidingHidden, sentinel+"-2") + + ctx := agentCtx([]string{collidingOwn}, []string{auth.PermRead}, "") + resp, body := tailLogLinesVia(t, f.proxy, ctx, collidingOwn, 2) + + assert.NotContains(t, body, sentinel, "a/b's records disclosed to an a_b-only token") + assert.Equal(t, 2, resp.LinesReturned, "lines_returned must count the authorized tail") + require.Len(t, resp.LogLines, 2, "the window must hold the two OWN records, got: %v", resp.LogLines) + assert.Contains(t, resp.LogLines[0], "own1", "foreign line displaced own1 from the window") + assert.Contains(t, resp.LogLines[1], "own2") + assert.Equal(t, len(resp.LogLines), resp.LinesReturned) + + // The default window (50) has the same property: no foreign record at all. + resp, body = tailLogLinesVia(t, f.proxy, ctx, collidingOwn, 50) + assert.NotContains(t, body, sentinel) + assert.Len(t, resp.LogLines, 2) + assert.Equal(t, 2, resp.LinesReturned) +} + +// FR007-G1 SC-001 differential: the `a_b`-only token's view must be the same +// whether or not hidden `a/b` shares the file (uniform and independent of +// hidden co-owners — never a whole-file refusal that depends on a co-owner). +// Three arms (critique round 1, finding C2.6): co-owner present and writing; +// co-owner configured but silent; co-owner ABSENT (not configured at all) — +// the last is the spec's literal "without hidden a/b present", and it is the +// arm that catches an implementation refusing the whole file whenever a +// config co-owner exists. +func TestTailLog_CollidingLogFile_DifferentialWithHiddenCoOwner(t *testing.T) { + ctx := agentCtx([]string{collidingOwn}, []string{auth.PermRead}, "") + + with := newTailLogCollidingProxy(t) + with.write(collidingOwn, "own1") + with.write(collidingHidden, "foreign1") + with.write(collidingOwn, "own2") + with.write(collidingHidden, "foreign2") + withResp, _ := tailLogLinesVia(t, with.proxy, ctx, collidingOwn, 50) + + silent := newTailLogCollidingProxy(t) + silent.write(collidingOwn, "own1") + silent.write(collidingOwn, "own2") + silentResp, _ := tailLogLinesVia(t, silent.proxy, ctx, collidingOwn, 50) + + absent := newTailLogProxyWithServers(t, collidingOwn) + absent.write(collidingOwn, "own1") + absent.write(collidingOwn, "own2") + absentResp, _ := tailLogLinesVia(t, absent.proxy, ctx, collidingOwn, 50) + + assert.Equal(t, tailLogSignatures(absentResp.LogLines), tailLogSignatures(withResp.LogLines), + "scoped view must not depend on whether a hidden co-owner shares the file") + assert.Equal(t, tailLogSignatures(absentResp.LogLines), tailLogSignatures(silentResp.LogLines), + "scoped view must not depend on whether a hidden co-owner is configured") + assert.Equal(t, absentResp.LinesReturned, withResp.LinesReturned) + assert.Equal(t, absentResp.LinesReturned, silentResp.LinesReturned) + assert.Equal(t, 2, absentResp.LinesReturned) +} + +// SC-005 administrator control: the administrator payload is the whole-file +// tail exactly as before the feature — log_lines byte-equal to the scrubbed +// whole-file reader, lines_returned its length, co-owner records included. +// Expected green on HEAD; it pins the whole-file path for the fix. The +// oracle is HEAD's logs.ReadUpstreamServerLogTail rather than a frozen +// capture (tasks.md T048): that is valid because internal/logs/logger.go is +// untouched by the Spec 105 PR E diff — if a later change edits the +// whole-file reader, this oracle moves with it and must be re-justified. +func TestTailLog_CollidingLogFile_AdminWholeFileUnchanged(t *testing.T) { + f := newTailLogCollidingProxy(t) + f.write(collidingOwn, "own1") + f.write(collidingHidden, "foreign1") + f.write(collidingOwn, "own2") + f.write(collidingHidden, "foreign2") + + whole, err := logs.ReadUpstreamServerLogTail(f.logCfg, collidingOwn, 2) + require.NoError(t, err) + require.Len(t, whole, 2) + + for name, ctx := range map[string]context.Context{ + "api-key admin": adminCtx(), + "no auth ctx": context.Background(), + } { + t.Run(name, func(t *testing.T) { + resp, body := tailLogLinesVia(t, f.proxy, ctx, collidingOwn, 2) + assert.Equal(t, scrubUpstreamLines(whole), resp.LogLines, "administrator log_lines must be the raw whole-file tail") + assert.Equal(t, 2, resp.LinesReturned) + assert.Contains(t, body, "foreign2", "administrators keep co-owner records") + assert.Contains(t, body, "own2") + for _, key := range []string{"server_name", "lines_requested", "lines_returned", "log_lines", "server_status", "connection_status"} { + assert.Contains(t, body, `"`+key+`"`, "administrator payload shape unchanged") + } + }) + } +} + +// Codex round 2 (PR E), docker finding 1, at the tool surface: `a/b` and +// hidden `a-b` both generate mcpproxy-a-b-, and on a suffix +// collision Docker's own `docker run` failure names the FOREIGN container's +// name and full id. That text reaches a/b's per-server log as child output +// — through the stdio child's stderr (monitoring.go) and the launcher pump +// (connection_launcher.go), both of which write it as the `message` field +// of a record stamped child_output=true (logs.ChildOutputField). tail_log +// for an a/b-scoped token must never show the foreign id or name; the +// administrator keeps the whole file (SC-005). The records are written here +// through the real stamped writer in exactly the producers' shape +// (internal/upstream/core/docker_collision_output_test.go drives the real +// producers against a fake docker and the same reader). +func TestTailLog_DockerCollisionChildOutput_ForeignContainerWithheldFromScopedCaller(t *testing.T) { + const foreignID = "f0e1d2c3b4a5968778695a4b3c2d1e0ff0e1d2c3b4a5968778695a4b3c2d1e0f" + const foreignName = "mcpproxy-a-b-wxyz" + collision := `docker: Error response from daemon: Conflict. The container name "/` + foreignName + + `" is already in use by container "` + foreignID + `". You have to remove (or rename) that container to be able to reuse that name.` + + const hiddenOwner = "a-b" // the container's owner, a co-tenant of the container namespace only + require.Equal(t, dockernaming.SanitizeServerName(collidingHidden), dockernaming.SanitizeServerName(hiddenOwner), + "fixture premise: a/b and a-b generate the same container-name stem") + require.Equal(t, "mcpproxy-"+dockernaming.SanitizeServerName(hiddenOwner)+"-wxyz", foreignName) + + f := newTailLogProxyWithServers(t, collidingHidden, hiddenOwner) // "a/b" reads; "a-b" is registered and silent + w := f.writers[collidingHidden] + w.Info("own-ordinary-record") + w.Info("stderr", zap.String("message", collision), logs.ChildOutputField()) + w.Info("launcher", zap.String("message", "[launcher stderr] "+collision), logs.ChildOutputField()) + w.Info("stderr", zap.String("message", "listening on 127.0.0.1:9331"), logs.ChildOutputField()) + _ = w.Sync() + + scoped := agentCtx([]string{collidingHidden}, []string{auth.PermRead}, "") + resp, body := tailLogLinesVia(t, f.proxy, scoped, collidingHidden, 50) + assert.NotContains(t, body, foreignID, "foreign container id disclosed to an a/b-scoped token") + assert.NotContains(t, body, foreignName, "foreign container name disclosed to an a/b-scoped token") + assert.NotContains(t, body, "already in use by container") + assert.Contains(t, body, "own-ordinary-record") + assert.Contains(t, body, "listening on 127.0.0.1:9331", "ordinary child output stays served") + assert.Equal(t, 2, resp.LinesReturned, "lines_returned counts the authorized tail: %s", body) + + for name, ctx := range map[string]context.Context{ + "api-key admin": adminCtx(), + "no auth ctx": context.Background(), + } { + t.Run(name, func(t *testing.T) { + resp, body := tailLogLinesVia(t, f.proxy, ctx, collidingHidden, 50) + assert.Contains(t, body, foreignID, "administrators keep Docker's output (SC-005)") + assert.Equal(t, 4, resp.LinesReturned) + }) + } +} + +// Codex round 3, logs finding 1, at the tool surface. The collision text +// reaches an a/b-scoped caller by two more routes than the direct stderr +// record: the per-server "Connection failed" record, whose error re-emits +// the recent-stderr buffer (internal/upstream/core recordConnectionFailure +// stamps it child_output=true, so the reader withholds it when it names a +// container), and `connection_status.last_error`, the same error rendered +// from the state manager — served by tail_log AND by `list` (with the health +// detail derived from it) — which is redacted for scoped callers +// (logs.RedactContainerMentions). Administrators keep both verbatim +// (SC-005). Registered co-tenant a-b is silent; the requester is a/b. +func TestTailLog_DockerCollisionConnectError_ForeignContainerWithheldFromScopedCaller(t *testing.T) { + const foreignID = "f0e1d2c3b4a5968778695a4b3c2d1e0ff0e1d2c3b4a5968778695a4b3c2d1e0f" + const foreignName = "mcpproxy-a-b-wxyz" + collision := `docker: Error response from daemon: Conflict. The container name "/` + foreignName + + `" is already in use by container "` + foreignID + `". You have to remove (or rename) that container to be able to reuse that name.` + const hiddenOwner = "a-b" + require.Equal(t, "mcpproxy-"+dockernaming.SanitizeServerName(collidingHidden)+"-wxyz", foreignName, + "fixture premise: the foreign name is a/b's (and a-b's) canonical container shape") + + f := newTailLogProxyWithServers(t, collidingHidden, hiddenOwner) + + // The connect error in the producer's shape: the premature-exit + // enrichment's text with the stderr block, wrapped by connectStdio. + connectErr := fmt.Errorf("stdio transport (command=%q, docker_isolation=%t): %w", "docker", true, + fmt.Errorf("server process exited before completing the MCP initialize handshake; recent stderr:\n | %s: EOF", collision)) + client, ok := f.proxy.upstreamManager.GetClient(collidingHidden) + require.True(t, ok) + client.StateManager.SetError(connectErr) + + w := f.writers[collidingHidden] + w.Info("own-ordinary-record") + w.Error("Connection failed", zap.String("transport", "stdio"), zap.Error(connectErr), logs.ChildOutputField()) + _ = w.Sync() + + scoped := agentCtx([]string{collidingHidden}, []string{auth.PermRead}, "") + resp, body := tailLogLinesVia(t, f.proxy, scoped, collidingHidden, 50) + assert.NotContains(t, body, foreignID, "foreign container id disclosed to an a/b-scoped token (log record or last_error)") + assert.NotContains(t, body, foreignName, "foreign container name disclosed to an a/b-scoped token (log record or last_error)") + assert.NotContains(t, body, "already in use by container") + assert.Contains(t, body, "own-ordinary-record") + assert.Contains(t, body, `"last_error"`, "the status field itself stays present, redacted") + assert.Equal(t, 1, resp.LinesReturned, "lines_returned counts the authorized tail: %s", body) + + // `list` renders the same error into connection_status.last_error and health. + listBody := listUpstreamsBodyVia(t, f.proxy, scoped) + assert.Contains(t, listBody, collidingHidden) + assert.NotContains(t, listBody, foreignID, "foreign container id disclosed through list") + assert.NotContains(t, listBody, foreignName, "foreign container name disclosed through list") + assert.NotContains(t, listBody, "already in use by container") + + for name, ctx := range map[string]context.Context{ + "api-key admin": adminCtx(), + "no auth ctx": context.Background(), + } { + t.Run(name, func(t *testing.T) { + resp, body := tailLogLinesVia(t, f.proxy, ctx, collidingHidden, 50) + assert.Contains(t, body, foreignID, "administrators keep the connect error verbatim (SC-005)") + assert.Equal(t, 2, resp.LinesReturned) + assert.Contains(t, listUpstreamsBodyVia(t, f.proxy, ctx), foreignID) + }) + } +} + +// listUpstreamsBodyVia drives `upstream_servers` `list` through the real +// dispatcher and returns the response text. +func listUpstreamsBodyVia(t *testing.T, proxy *MCPProxyServer, ctx context.Context) string { + t.Helper() + request := mcp.CallToolRequest{} + request.Params.Arguments = map[string]interface{}{"operation": "list"} + result, err := proxy.handleUpstreamServers(ctx, request) + require.NoError(t, err) + body := toolResultText(t, result) + require.False(t, result.IsError, "list must succeed: %s", body) + return body } diff --git a/internal/upstream/core/child_output_redact_test.go b/internal/upstream/core/child_output_redact_test.go index 8ce613f77..f43490ed7 100644 --- a/internal/upstream/core/child_output_redact_test.go +++ b/internal/upstream/core/child_output_redact_test.go @@ -40,7 +40,8 @@ func TestLoggerWriter_ScrubsChildOutput(t *testing.T) { var rendered []string for _, entry := range logs.All() { - rendered = append(rendered, entry.Message) + line, _ := entry.ContextMap()["message"].(string) + rendered = append(rendered, entry.Message+" "+line) } joined := strings.Join(rendered, "\n") @@ -128,3 +129,34 @@ func TestRedactURLCredentialsInError_RunsTheValueShapedDetector(t *testing.T) { "the connect paths classify on this substring; masking must not eat it") assert.ErrorIs(t, got, err, "the original must stay reachable through Unwrap") } + +// Spec 105 FR-007 (critique round 1, finding C2.2): one record per child +// line. launcher.pumpLines delivers one line per Write today, so this is the +// guarantee for any other producer: a multi-line chunk becomes N records, +// CRLF handled, blank lines dropped. Since codex round 2 the child's text is +// the `message` field value of a constant-message record (zap escapes a +// line break there under both encoders), so the split is about record +// shape, not attribution; a forged header inside the text is inert. +func TestLoggerWriter_SplitsMultiLineChunks(t *testing.T) { + core, logs := observer.New(zap.DebugLevel) + w := newLoggerWriter(zap.New(core), nil) + + const forged = `2026-01-01T00:00:00.000Z | INFO | x/y.go:1 | FORGED-for-a_b | {"server": "a_b"}` + chunk := "first line\r\n" + forged + "\n\nlast line\n" + n, err := w.Write([]byte(chunk)) + require.NoError(t, err) + assert.Equal(t, len(chunk), n, "the writer must report the bytes it consumed") + + var rendered []string + for _, entry := range logs.All() { + assert.Equal(t, "launcher", entry.Message, "the record message is constant; child text is a field value") + line, _ := entry.ContextMap()["message"].(string) + rendered = append(rendered, line) + } + require.Equal(t, []string{"first line", forged, "last line"}, rendered, + "one record per child line: a forged line is its own record, stamped by the real writer") + for _, msg := range rendered { + assert.NotContains(t, msg, "\n", "a child line must never carry a line break") + assert.NotContains(t, msg, "\r", "CRLF is stripped, not logged") + } +} diff --git a/internal/upstream/core/client.go b/internal/upstream/core/client.go index d67cd6191..28e289baa 100644 --- a/internal/upstream/core/client.go +++ b/internal/upstream/core/client.go @@ -143,6 +143,7 @@ type Client struct { // Docker container tracking containerID string + containerOwner string // com.mcpproxy.server label read back when containerID was verified (Spec 105 D9) containerName string // Store container name for cleanup via docker container commands isDockerCommand bool diff --git a/internal/upstream/core/connection.go b/internal/upstream/core/connection.go index b593b367b..95ca321ee 100644 --- a/internal/upstream/core/connection.go +++ b/internal/upstream/core/connection.go @@ -6,6 +6,7 @@ import ( "os" "time" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/logs" "github.com/smart-mcp-proxy/mcpproxy-go/internal/oauth" "github.com/smart-mcp-proxy/mcpproxy-go/internal/transport" @@ -182,6 +183,30 @@ func logSafeErrorField(err error) zap.Field { return zap.String("error", oauth.ScrubUpstreamText(err.Error())) } +// recordConnectionFailure writes the "Connection failed" record to the +// per-server log. A connect error that re-emits the child's stderr +// (childOutputError: the initialize-timeout and premature-exit enrichments +// splice the recent-stderr buffer into their text) makes this record a +// child-output record, and it is stamped as one (logs.ChildOutputField) so +// the attributed reader applies the same container-subject rule it applies +// to the direct stderr record: on a `docker run` name collision the buffer +// names another server's container, and this record repeated it without the +// provenance (Spec 105 FR-007, codex round 3). Ordinary connect errors are +// recorded exactly as before. +func (c *Client) recordConnectionFailure(err error) { + if c.upstreamLogger == nil { + return + } + fields := []zap.Field{ + zap.String("transport", c.transportType), + zap.Error(err), + } + if embedsChildOutput(err) { + fields = append(fields, logs.ChildOutputField()) + } + c.upstreamLogger.Error("Connection failed", fields...) +} + func (c *Client) Connect(ctx context.Context) error { c.mu.Lock() defer c.mu.Unlock() @@ -296,21 +321,21 @@ func (c *Client) Connect(ctx context.Context) error { err = redactURLCredentialsInError(err) // Log connection failure to server-specific log - if c.upstreamLogger != nil { - c.upstreamLogger.Error("Connection failed", - zap.String("transport", c.transportType), - zap.Error(err)) - } + c.recordConnectionFailure(err) // CRITICAL FIX: Cleanup Docker containers when any connection type fails // This prevents container accumulation when connections fail after Docker setup if c.isDockerCommand { - c.logger.Warn("Connection failed for Docker command - cleaning up container", + // Spec 105 D8: name a container here only with evidence — see + // dockerContainerLogFields. c.containerName alone can be a + // generated name never observed from Docker. + fields := []zap.Field{ zap.String("server", c.config.Name), zap.String("transport", c.transportType), - zap.String("container_name", c.containerName), - zap.String("container_id", c.containerID), - zap.Error(err)) + } + fields = append(fields, dockerContainerLogFields(c.containerID, c.containerName, c.containerOwner)...) + fields = append(fields, zap.Error(err)) + c.logger.Warn("Connection failed for Docker command - cleaning up container", fields...) cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), dockerCleanupTimeout) defer cleanupCancel() diff --git a/internal/upstream/core/connection_docker.go b/internal/upstream/core/connection_docker.go index ef4cd02fc..b9d22da40 100644 --- a/internal/upstream/core/connection_docker.go +++ b/internal/upstream/core/connection_docker.go @@ -71,7 +71,12 @@ func (c *Client) setupDockerIsolation(command string, args []string) (dockerComm zap.Strings("container_args", logSafeArgs(containerArgs)), zap.Strings("docker_run_args", logSafeArgs(dockerRunArgs))) - // Log to server-specific log as well + // Log to server-specific log as well. The name is GENERATED here, before + // Docker has created or inspected anything, so the record carries no + // container_owner (Spec 105 D9: the owner field is only ever the label + // read back from Docker); the attributed reader withholds the record + // from scoped callers as an ownerless container subject, administrators + // see it unchanged. if c.upstreamLogger != nil { c.upstreamLogger.Info("Docker isolation configured", zap.String("runtime_type", runtimeType), diff --git a/internal/upstream/core/connection_launcher.go b/internal/upstream/core/connection_launcher.go index 266d1041f..bc0a8afa4 100644 --- a/internal/upstream/core/connection_launcher.go +++ b/internal/upstream/core/connection_launcher.go @@ -12,6 +12,7 @@ import ( "go.uber.org/zap" "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/logs" "github.com/smart-mcp-proxy/mcpproxy-go/internal/oauth" "github.com/smart-mcp-proxy/mcpproxy-go/internal/upstream/launcher" ) @@ -341,9 +342,26 @@ func newLoggerWriter(primary, fallback *zap.Logger) io.Writer { } func (w *loggerWriter) Write(p []byte) (int, error) { - line := strings.TrimRight(string(p), "\n") + // One record per line: pumpLines already writes one line per call, and + // the split keeps that shape for any other producer (one record per + // child line is what `mcpproxy upstream logs` shows). + for _, line := range strings.Split(strings.TrimRight(string(p), "\n"), "\n") { + w.writeLine(strings.TrimRight(line, "\r")) + } + return len(p), nil +} + +// writeLine records one child output line through the per-server logger. +// The child's text is the `message` FIELD of a constant-message record +// stamped child_output=true, never the record message (Spec 105 FR-007, +// internal/logs research D8 rule 1): the console encoder writes a message +// unescaped, so child text there could carry a record header or boundary, +// and a docker CLI failure names another server's container — the +// attributed reader withholds child-output records that mention a container +// (codex round 2). +func (w *loggerWriter) writeLine(line string) { if line == "" { - return len(p), nil + return } // Issue #1158 (review round 2, investigation 3). This is the child // process's own stdout/stderr, written verbatim into @@ -362,9 +380,8 @@ func (w *loggerWriter) Write(p []byte) (int, error) { line = oauth.ScrubUpstreamText(line) switch { case w.primary != nil: - w.primary.Info(line) + w.primary.Info("launcher", zap.String("message", line), logs.ChildOutputField()) case w.fallback != nil: - w.fallback.Info(line) + w.fallback.Info("launcher", zap.String("message", line), logs.ChildOutputField()) } - return len(p), nil } diff --git a/internal/upstream/core/connection_lifecycle.go b/internal/upstream/core/connection_lifecycle.go index 227d2c124..1f29b61e8 100644 --- a/internal/upstream/core/connection_lifecycle.go +++ b/internal/upstream/core/connection_lifecycle.go @@ -53,11 +53,13 @@ func (c *Client) initialize(ctx context.Context) error { // CRITICAL FIX: Additional cleanup for direct initialize() calls // This handles cases where initialize() is called independently if c.isDockerCommand { - c.logger.Debug("Direct initialization failed for Docker command - cleanup may be handled by caller", - zap.String("server", c.config.Name), - zap.String("container_name", c.containerName), - zap.String("container_id", c.containerID), - logSafeErrorField(err)) + // Spec 105 D8: name a container here only with evidence — see + // dockerContainerLogFields. c.containerName alone can be a + // generated name never observed from Docker. + fields := []zap.Field{zap.String("server", c.config.Name)} + fields = append(fields, dockerContainerLogFields(c.containerID, c.containerName, c.containerOwner)...) + fields = append(fields, logSafeErrorField(err)) + c.logger.Debug("Direct initialization failed for Docker command - cleanup may be handled by caller", fields...) } // Surface the useful context that the raw "context deadline exceeded" @@ -67,7 +69,7 @@ func (c *Client) initialize(ctx context.Context) error { waited := time.Since(initStart).Round(100 * time.Millisecond) stderrBlock := c.formatRecentStderr() if stderrBlock != "" { - return fmt.Errorf("server did not respond to MCP initialize within %s (subprocess may have crashed or printed to stderr instead of stdout); recent stderr:\n%s", waited, stderrBlock) + return &childOutputError{msg: fmt.Sprintf("server did not respond to MCP initialize within %s (subprocess may have crashed or printed to stderr instead of stdout); recent stderr:\n%s", waited, stderrBlock)} } return fmt.Errorf("server did not respond to MCP initialize within %s and produced no stderr output (check that the command starts an MCP server and not a help banner)", waited) } @@ -151,11 +153,38 @@ func shouldEnrichStdioPrematureExit(transportType string, err error) bool { func enrichTransportClosedError(stderrBlock string, cause error) error { if stderrBlock != "" { - return fmt.Errorf("server process exited before completing the MCP initialize handshake; recent stderr:\n%s: %w", stderrBlock, cause) + return &childOutputError{ + msg: fmt.Sprintf("server process exited before completing the MCP initialize handshake; recent stderr:\n%s: %v", stderrBlock, cause), + cause: cause, + } } return fmt.Errorf("server process exited before completing the MCP initialize handshake and produced no stderr output (transport closed before the handshake): %w", cause) } +// childOutputError is a connect error whose text re-emits the child +// process's own stderr (the recent-stderr buffer). It is the provenance the +// per-server log needs: a record that renders such an error carries child +// text and is stamped child_output=true (recordConnectionFailure), so the +// attributed reader (internal/logs, D8 rules 1 and 3) treats it exactly like +// the direct stderr record — on a `docker run` name collision that text +// names another server's container. It unwraps to its cause so errors.Is / +// errors.As keep working through the wrappers connectStdio and Connect add +// (Spec 105 FR-007, codex round 3). +type childOutputError struct { + msg string + cause error +} + +func (e *childOutputError) Error() string { return e.msg } +func (e *childOutputError) Unwrap() error { return e.cause } + +// embedsChildOutput reports whether err, anywhere in its chain, re-emits the +// child's stderr. +func embedsChildOutput(err error) bool { + var target *childOutputError + return errors.As(err, &target) +} + // isTransportClosedErr reports whether an initialize() failure indicates the // child process went away mid-handshake. mcp-go surfaces a premature stdio // exit as a closed transport / EOF on the pipe rather than a typed exit error, @@ -273,6 +302,7 @@ func (c *Client) DisconnectWithContext(_ context.Context) error { isDocker := c.isDockerCommand containerID := c.containerID containerName := c.containerName + containerOwner := c.containerOwner pgid := c.processGroupID processCmd := c.processCmd serverName := c.config.Name @@ -300,14 +330,23 @@ func (c *Client) DisconnectWithContext(_ context.Context) error { defer cleanupCancel() if containerID != "" { + // containerID is only ever set alongside containerOwner, once + // trackCidfileContainer or the name-recovery fallback verified + // ownership (Spec 105 D8) — safe to name here. c.logger.Debug("Cleaning up Docker container by ID", zap.String("server", serverName), - zap.String("container_id", containerID)) + zap.String("container_id", containerID), + containerOwnerField(containerOwner)) c.killDockerContainerWithContext(cleanupCtx) } else if containerName != "" { + // containerName alone (containerID empty here) is the GENERATED + // canonical name, never read back from Docker — not evidence a + // container by that name is ours (Spec 105 D8), so the record + // names the server only. killDockerContainerByNameWithContext + // still re-verifies ownership via ContainerMutator before it + // ever stops anything. c.logger.Debug("Cleaning up Docker container by name", - zap.String("server", serverName), - zap.String("container_name", containerName)) + zap.String("server", serverName)) c.killDockerContainerByNameWithContext(cleanupCtx, containerName) } else { c.logger.Debug("No container ID or name, using pattern-based cleanup", diff --git a/internal/upstream/core/connection_stdio.go b/internal/upstream/core/connection_stdio.go index a2130444e..cb038e963 100644 --- a/internal/upstream/core/connection_stdio.go +++ b/internal/upstream/core/connection_stdio.go @@ -293,11 +293,13 @@ func (c *Client) connectStdio(ctx context.Context) error { // CRITICAL FIX: Cleanup Docker containers when initialization fails // This prevents container accumulation when servers timeout during startup if c.isDockerCommand { - c.logger.Warn("Initialization failed for Docker command - cleaning up container", - zap.String("server", c.config.Name), - zap.String("container_name", c.containerName), - zap.String("container_id", c.containerID), - zap.Error(err)) + // Spec 105 D8: name a container here only with evidence — see + // dockerContainerLogFields. c.containerName alone can be a + // generated name never observed from Docker. + fields := []zap.Field{zap.String("server", c.config.Name)} + fields = append(fields, dockerContainerLogFields(c.containerID, c.containerName, c.containerOwner)...) + fields = append(fields, zap.Error(err)) + c.logger.Warn("Initialization failed for Docker command - cleaning up container", fields...) cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), dockerCleanupTimeout) defer cleanupCancel() diff --git a/internal/upstream/core/docker.go b/internal/upstream/core/docker.go index 90d853084..f119de427 100644 --- a/internal/upstream/core/docker.go +++ b/internal/upstream/core/docker.go @@ -40,6 +40,14 @@ func (c *Client) newDockerCmd(ctx context.Context, args ...string) *exec.Cmd { return cmd } +// cidfileReadAttempts × cidfileReadInterval bounds how long the cidfile is +// polled for (10 s by default: image pulls take a while). Variables so tests +// can shorten the wait. +var ( + cidfileReadAttempts = 100 + cidfileReadInterval = 100 * time.Millisecond +) + // readContainerIDWithContext reads the container ID from cidfile for tracking with context cancellation func (c *Client) readContainerIDWithContext(ctx context.Context, cidFile string) { c.logger.Debug("Starting container ID tracking", @@ -47,7 +55,7 @@ func (c *Client) readContainerIDWithContext(ctx context.Context, cidFile string) zap.String("cid_file", cidFile)) // Wait for container to start and write CID file - longer timeout for image pulls - for attempt := 0; attempt < 100; attempt++ { // Wait up to 10 seconds + for attempt := 0; attempt < cidfileReadAttempts; attempt++ { select { case <-ctx.Done(): c.logger.Debug("Container ID tracking canceled", @@ -55,30 +63,15 @@ func (c *Client) readContainerIDWithContext(ctx context.Context, cidFile string) zap.String("cid_file", cidFile)) return default: - time.Sleep(100 * time.Millisecond) + time.Sleep(cidfileReadInterval) cidBytes, err := os.ReadFile(cidFile) if err == nil { containerID := strings.TrimSpace(string(cidBytes)) if containerID != "" { - c.mu.Lock() - c.containerID = containerID - c.mu.Unlock() - - c.logger.Info("Docker container ID captured for cleanup", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12]), // Show short ID - zap.String("full_container_id", containerID), - zap.Int("attempt", attempt)) - - if c.upstreamLogger != nil { - c.upstreamLogger.Info("Container ID captured", - zap.String("container_id", containerID), - zap.Int("attempt", attempt)) - } - // Clean up the cidfile now that we have the ID os.Remove(cidFile) + c.trackCidfileContainer(ctx, containerID, attempt) return } } else if attempt%10 == 0 { // Log every 1 second @@ -100,47 +93,119 @@ func (c *Client) readContainerIDWithContext(ctx context.Context, cidFile string) c.upstreamLogger.Warn("cidfile read timeout - attempting name lookup recovery") } - // Fallback: Find container by name + // Fallback: find the container by its exact tracked name. Only a + // container that passes ownsContainer (label read back AND canonical + // name) is adopted; a foreign `--label com.mcpproxy.server= --name + // ` container is left alone and never named in our log. if c.containerName != "" { - listCmd := c.newDockerCmd(ctx, "ps", - "--filter", fmt.Sprintf("name=^%s$", c.containerName), - "--format", "{{.ID}}") - - if output, err := listCmd.Output(); err == nil { - foundID := strings.TrimSpace(string(output)) - if foundID != "" { - c.mu.Lock() - c.containerID = foundID - c.mu.Unlock() - - c.logger.Info("Successfully recovered container ID via name lookup", - zap.String("server", c.config.Name), - zap.String("container_id", foundID[:12]), - zap.String("full_container_id", foundID), - zap.String("container_name", c.containerName)) + found, ok, err := c.lookupOwnedContainerByName(ctx, c.containerName) + if err != nil { + c.logger.Debug("Failed to look up container by name", + zap.String("server", c.config.Name), + zap.String("container_name", c.containerName), + zap.Error(err)) + } + if ok { + c.mu.Lock() + c.containerID = found.ID + c.containerOwner = found.Owner + c.mu.Unlock() - if c.upstreamLogger != nil { - c.upstreamLogger.Info("Container ID recovered via name lookup", - zap.String("container_id", foundID)) - } + c.logger.Info("Successfully recovered container ID via name lookup", + zap.String("server", c.config.Name), + zap.String("container_id", shortContainerID(found.ID)), + zap.String("full_container_id", found.ID), + zap.String("container_name", found.Name), + containerOwnerField(found.Owner)) - // Clean up the cidfile since we got the ID - os.Remove(cidFile) - return + if c.upstreamLogger != nil { + c.upstreamLogger.Info("Container ID recovered via name lookup", + zap.String("container_id", found.ID), + zap.String("container_name", found.Name), + containerOwnerField(found.Owner)) } + + // Clean up the cidfile since we got the ID + os.Remove(cidFile) + return } } + // c.containerName is the GENERATED name, never read back from Docker + // here: the lookup above already rejected it (or errored), so under a + // suffix collision it can currently belong to a different, colliding + // server. Name only the server, as the round-9 lifecycle fixes do for + // every other generated-name-only state (codex round 11). c.logger.Error("Failed to recover container ID - container will be orphaned on disconnect", - zap.String("server", c.config.Name), - zap.String("container_name", c.containerName)) + zap.String("server", c.config.Name)) if c.upstreamLogger != nil { c.upstreamLogger.Error("Failed to recover container ID - may be orphaned") } } -// killDockerContainerWithContext kills the Docker container if one is running with context timeout +// trackCidfileContainer adopts the id our `docker run` wrote to its cidfile +// — but only after inspecting it: a user-configured direct `docker run +// --name custom` upstream gets a cidfile too, yet carries neither the +// ownership label nor a canonical name, and under D9 such a container is not +// ours to stop or to name in our log (codex round 1). container_owner is the +// label read back, never this server's name. +func (c *Client) trackCidfileContainer(ctx context.Context, containerID string, attempt int) { + owned, ok, err := c.lookupOwnedContainerByID(ctx, containerID) + switch { + case err != nil: + // No id here: the read that would have confirmed this cidfile + // row is ours failed outright, so there is no evidence to name + // (same refusal rule mutateOwnedContainer applies to every other + // mutation path, codex round 8). + c.logger.Warn("Could not verify ownership of the container from the cidfile - it will not be managed", + zap.String("server", c.config.Name), + zap.Error(err)) + if c.upstreamLogger != nil { + c.upstreamLogger.Warn("Could not verify ownership of the container from the cidfile - it will not be managed", + zap.Error(err)) + } + return + case !ok: + // No id here either: the container the cidfile named failed the + // ownership predicate, so it is not ours to name in the log any + // more than to stop (codex round 8). + c.logger.Info("Container from the cidfile is not canonically owned by this server (no com.mcpproxy.server label or non-canonical name) - it will not be stopped on disconnect", + zap.String("server", c.config.Name)) + if c.upstreamLogger != nil { + c.upstreamLogger.Info("Container from the cidfile is not canonically owned by this server - it will not be stopped on disconnect") + } + return + } + + c.mu.Lock() + c.containerID = containerID + c.containerOwner = owned.Owner + c.mu.Unlock() + + c.logger.Info("Docker container ID captured for cleanup", + zap.String("server", c.config.Name), + zap.String("container_id", shortContainerID(containerID)), + zap.String("full_container_id", containerID), + zap.String("container_name", owned.Name), + containerOwnerField(owned.Owner), + zap.Int("attempt", attempt)) + + if c.upstreamLogger != nil { + c.upstreamLogger.Info("Container ID captured", + zap.String("container_id", containerID), + zap.String("container_name", owned.Name), + containerOwnerField(owned.Owner), + zap.Int("attempt", attempt)) + } +} + +// killDockerContainerWithContext stops (then kills) the tracked container +// during disconnect. The id was adopted through trackCidfileContainer or the +// name recovery, but ownership is re-established by stopOwnedContainer at +// the moment of each mutation — `docker rename` or a foreign container +// reusing the id could have changed the answer — so no path stops a +// container the predicate does not admit now. // NOTE: This function expects the caller to already hold the mutex lock func (c *Client) killDockerContainerWithContext(ctx context.Context) { c.logger.Debug("Starting Docker container kill process", @@ -155,77 +220,13 @@ func (c *Client) killDockerContainerWithContext(ctx context.Context) { return } - c.logger.Info("Killing Docker container during disconnect", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12]), - zap.String("full_container_id", containerID)) - - if c.upstreamLogger != nil { - c.upstreamLogger.Info("Killing Docker container", - zap.String("container_id", containerID)) - } - - // First try graceful stop (SIGTERM) - c.logger.Debug("Attempting graceful stop", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12])) - - stopCmd := c.newDockerCmd(ctx, "stop", containerID) - c.logger.Debug("Executing docker stop command", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12])) - - if err := stopCmd.Run(); err != nil { - c.logger.Warn("Failed to stop Docker container gracefully, trying force kill", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12]), - zap.Error(err)) - - // Force kill (SIGKILL) - c.logger.Debug("Attempting force kill", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12])) - - killCmd := c.newDockerCmd(ctx, "kill", containerID) - c.logger.Debug("Executing docker kill command", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12])) - - if err := killCmd.Run(); err != nil { - c.logger.Error("Failed to force kill Docker container", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12]), - zap.Error(err)) - - if c.upstreamLogger != nil { - c.upstreamLogger.Error("Failed to kill container", zap.Error(err)) - } - } else { - c.logger.Info("Docker container force killed successfully", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12])) - - if c.upstreamLogger != nil { - c.upstreamLogger.Info("Container force killed successfully") - } - } - } else { - c.logger.Info("Docker container stopped gracefully", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12])) - - if c.upstreamLogger != nil { - c.upstreamLogger.Info("Container stopped gracefully") - } - } - - c.logger.Debug("Docker stop/kill commands completed, clearing container ID", - zap.String("server", c.config.Name), - zap.String("container_id", containerID[:12])) - - // Clear the container ID after cleanup attempt + // Clear the tracked id whatever happens below: after this call the + // container is either stopped or deliberately left alone. // Note: Caller already holds the mutex lock c.containerID = "" + c.containerOwner = "" + + c.stopOwnedContainer(ctx, containerID, "cidfile") c.logger.Debug("Container cleanup process finished", zap.String("server", c.config.Name)) @@ -266,13 +267,22 @@ func (c *Client) killDockerContainerByCommandWithContext(ctx context.Context) { return } - c.logger.Debug("Searching for containers by image name", + c.killDockerContainersByImageWithContext(ctx, imageName) +} + +// killDockerContainersByImageWithContext is the image-name fallback: it stops +// the running containers this server canonically owns whose image is +// imageName. +func (c *Client) killDockerContainersByImageWithContext(ctx context.Context, imageName string) { + c.logger.Debug("Searching for owned containers by image name", zap.String("server", c.config.Name), zap.String("image_name", imageName)) - // Get list of running containers with image and created time - listCmd := c.newDockerCmd(ctx, "ps", "--format", "{{.ID}}\t{{.Image}}\t{{.CreatedAt}}") - output, err := listCmd.Output() + // Spec 105 FR-007 / D9: the image-name fallback lists only containers this + // server canonically owns (label + name regex) and then matches the image, + // so a foreign container that merely shares the image is neither killed + // nor written into this server's log. + owned, err := c.listOwnedContainers(ctx, false) if err != nil { c.logger.Error("Failed to list Docker containers for cleanup", zap.String("server", c.config.Name), @@ -280,84 +290,51 @@ func (c *Client) killDockerContainerByCommandWithContext(ctx context.Context) { return } - // Parse output and find matching containers - lines := strings.Split(strings.TrimSpace(string(output)), "\n") - var containersToKill []string - - for _, line := range lines { - if line == "" { - continue - } - parts := strings.SplitN(line, "\t", 3) - if len(parts) >= 2 { - containerID := parts[0] - image := parts[1] - - // Check if this container matches our image - if image == imageName { - containersToKill = append(containersToKill, containerID) - c.logger.Info("Found matching container for cleanup", - zap.String("server", c.config.Name), - zap.String("container_id", containerID), - zap.String("image", image)) - } + var containersToKill []ownedContainer + for _, container := range owned { + // Check if this container matches our image + if container.Image == imageName { + containersToKill = append(containersToKill, container) } } if len(containersToKill) == 0 { - c.logger.Debug("No matching containers found for cleanup", + c.logger.Debug("No matching owned containers found for cleanup", zap.String("server", c.config.Name), zap.String("image_name", imageName)) return } - // Kill matching containers - for _, containerID := range containersToKill { - c.logger.Info("Killing matching Docker container", - zap.String("server", c.config.Name), - zap.String("container_id", containerID)) - - if c.upstreamLogger != nil { - c.upstreamLogger.Info("Killing matching container", - zap.String("container_id", containerID)) - } - - // First try graceful stop - stopCmd := c.newDockerCmd(ctx, "stop", containerID) - if err := stopCmd.Run(); err != nil { - // Force kill if graceful stop fails - killCmd := c.newDockerCmd(ctx, "kill", containerID) - if err := killCmd.Run(); err != nil { - c.logger.Error("Failed to kill matching Docker container", - zap.String("server", c.config.Name), - zap.String("container_id", containerID), - zap.Error(err)) - } else { - c.logger.Info("Successfully force killed matching Docker container", - zap.String("server", c.config.Name), - zap.String("container_id", containerID)) - } - } else { - c.logger.Info("Successfully stopped matching Docker container", - zap.String("server", c.config.Name), - zap.String("container_id", containerID)) - } + // The listing is a snapshot: stopOwnedContainer re-reads each container + // right before its stop, and only that read names it in the records. + // container_owner: D8 rule 3 treats a count as container-subject + // evidence, so it is paired with the label Docker reported on the + // listed rows (one value for every row: ownsContainer admits only rows + // whose label equals this server's raw name) — never the requesting + // server's name (codex round 11). + c.logger.Info("Found matching owned containers for cleanup", + zap.String("server", c.config.Name), + zap.String("image_name", imageName), + zap.Int("container_count", len(containersToKill)), + containerOwnerField(containersToKill[0].Owner)) + for _, container := range containersToKill { + c.stopOwnedContainer(ctx, container.ID, "image") } } -// killDockerContainersByNamePatternWithContext finds and kills containers by name pattern +// killDockerContainersByNamePatternWithContext finds and kills the containers +// this server canonically owns (Spec 105 FR-007 / D9: label +// com.mcpproxy.server= AND name ^mcpproxy--[a-z0-9]{4}$). +// Pre-105 this was a `name=mcpproxy--` substring filter, which +// also matched — and killed, and logged — `a-b`'s containers for server `a`. func (c *Client) killDockerContainersByNamePatternWithContext(ctx context.Context) bool { - // Create sanitized server name for pattern matching - sanitized := sanitizeServerNameForContainer(c.config.Name) - namePattern := "mcpproxy-" + sanitized + "-" + namePattern := ownedContainerNamePattern(c.config.Name) - c.logger.Debug("Searching for containers by name pattern", + c.logger.Debug("Searching for owned containers by name pattern", zap.String("server", c.config.Name), zap.String("name_pattern", namePattern)) - // Get list of containers with name filter - listCmd := c.newDockerCmd(ctx, "ps", "-a", "--filter", "name="+namePattern, "--format", "{{.ID}}\t{{.Names}}") - output, err := listCmd.Output() + owned, err := c.listOwnedContainers(ctx, true) if err != nil { c.logger.Debug("Failed to list Docker containers by name pattern", zap.String("server", c.config.Name), @@ -366,82 +343,43 @@ func (c *Client) killDockerContainersByNamePatternWithContext(ctx context.Contex return false } - // Parse output and find matching containers - lines := strings.Split(strings.TrimSpace(string(output)), "\n") - var containersToKill []string - - for _, line := range lines { - if line == "" { - continue - } - parts := strings.SplitN(line, "\t", 2) - if len(parts) >= 2 { - containerID := parts[0] - containerName := parts[1] - - // Check if the container name starts with our pattern - if strings.HasPrefix(containerName, namePattern) { - containersToKill = append(containersToKill, containerID) - c.logger.Info("Found matching container by name pattern", - zap.String("server", c.config.Name), - zap.String("container_id", containerID), - zap.String("container_name", containerName)) - } - } - } - - if len(containersToKill) == 0 { - c.logger.Debug("No matching containers found by name pattern", + if len(owned) == 0 { + c.logger.Debug("No owned containers found by name pattern", zap.String("server", c.config.Name), zap.String("name_pattern", namePattern)) return false } - // Kill matching containers - for _, containerID := range containersToKill { - c.logger.Info("Killing container by name pattern", - zap.String("server", c.config.Name), - zap.String("container_id", containerID)) - - if c.upstreamLogger != nil { - c.upstreamLogger.Info("Killing container by name pattern", - zap.String("container_id", containerID)) - } - - // First try graceful stop - stopCmd := c.newDockerCmd(ctx, "stop", containerID) - if err := stopCmd.Run(); err != nil { - // Force kill if graceful stop fails - killCmd := c.newDockerCmd(ctx, "kill", containerID) - if err := killCmd.Run(); err != nil { - c.logger.Error("Failed to kill container by name pattern", - zap.String("server", c.config.Name), - zap.String("container_id", containerID), - zap.Error(err)) - } else { - c.logger.Info("Successfully force killed container by name pattern", - zap.String("server", c.config.Name), - zap.String("container_id", containerID)) - } - } else { - c.logger.Info("Successfully stopped container by name pattern", - zap.String("server", c.config.Name), - zap.String("container_id", containerID)) - } + // The listing is a snapshot: stopOwnedContainer re-reads each container + // right before its stop, and only that read names it in the records. + // container_owner: D8 rule 3 treats a count as container-subject + // evidence, so it is paired with the label Docker reported on the + // listed rows (one value for every row: ownsContainer admits only rows + // whose label equals this server's raw name) — never the requesting + // server's name (codex round 11). + c.logger.Info("Found owned containers by name pattern", + zap.String("server", c.config.Name), + zap.String("name_pattern", namePattern), + zap.Int("container_count", len(owned)), + containerOwnerField(owned[0].Owner)) + for _, container := range owned { + c.stopOwnedContainer(ctx, container.ID, "name pattern") } return true // We found and processed containers } -// killDockerContainerByNameWithContext kills a specific Docker container by its exact name +// killDockerContainerByNameWithContext kills the container this server +// tracks by its exact name (the one setupDockerIsolation generated). The +// lookup applies ownsContainer — label read back AND canonical name — so a +// foreign `--label com.mcpproxy.server= --name ` container is +// neither stopped nor named in our log (Spec 105 FR-007 / D9; codex round 1). func (c *Client) killDockerContainerByNameWithContext(ctx context.Context, containerName string) bool { - c.logger.Debug("Searching for container by exact name", + c.logger.Debug("Searching for owned container by exact name", zap.String("server", c.config.Name), zap.String("container_name", containerName)) - // Get container ID by exact name match - listCmd := c.newDockerCmd(ctx, "ps", "-a", "--filter", "name=^"+containerName+"$", "--format", "{{.ID}}") - output, err := listCmd.Output() + found, ok, err := c.lookupOwnedContainerByName(ctx, containerName) if err != nil { c.logger.Debug("Failed to find Docker container by name", zap.String("server", c.config.Name), @@ -449,131 +387,106 @@ func (c *Client) killDockerContainerByNameWithContext(ctx context.Context, conta zap.Error(err)) return false } - - containerID := strings.TrimSpace(string(output)) - if containerID == "" { - c.logger.Debug("No container found with exact name", + if !ok { + c.logger.Debug("No owned container found with exact name", zap.String("server", c.config.Name), zap.String("container_name", containerName)) return false } - c.logger.Info("Found container by name, attempting to kill", - zap.String("server", c.config.Name), - zap.String("container_name", containerName), - zap.String("container_id", containerID)) - - if c.upstreamLogger != nil { - c.upstreamLogger.Info("Killing container by name", - zap.String("container_name", containerName), - zap.String("container_id", containerID)) - } - - // First try graceful stop - stopCmd := c.newDockerCmd(ctx, "stop", containerID) - if err := stopCmd.Run(); err != nil { - // Force kill if graceful stop fails - killCmd := c.newDockerCmd(ctx, "kill", containerID) - if err := killCmd.Run(); err != nil { - c.logger.Error("Failed to kill container by name", - zap.String("server", c.config.Name), - zap.String("container_name", containerName), - zap.String("container_id", containerID), - zap.Error(err)) - return false - } - c.logger.Info("Successfully force killed container by name", - zap.String("server", c.config.Name), - zap.String("container_name", containerName), - zap.String("container_id", containerID)) - return true - } - c.logger.Info("Successfully stopped container by name", - zap.String("server", c.config.Name), - zap.String("container_name", containerName), - zap.String("container_id", containerID)) - - return true + return c.stopOwnedContainer(ctx, found.ID, "exact name") } -// ensureNoExistingContainers removes all existing containers for this server before creating a new one -// This makes container creation idempotent and prevents duplicate container spawning +// ensureNoExistingContainers removes all existing containers this server +// canonically owns before creating a new one. This makes container creation +// idempotent and prevents duplicate container spawning. Ownership is label +// com.mcpproxy.server= AND name ^mcpproxy--[a-z0-9]{4}$ +// (Spec 105 FR-007 / D9): a foreign container whose name merely shares the +// prefix — `a-b`'s or `a/b`'s for server `a` — is neither removed nor named +// in this server's log. func (c *Client) ensureNoExistingContainers(ctx context.Context) error { - sanitized := sanitizeServerNameForContainer(c.config.Name) - namePattern := "mcpproxy-" + sanitized + "-" + namePattern := ownedContainerNamePattern(c.config.Name) - c.logger.Info("Checking for existing containers before creation", + c.logger.Info("Checking for existing owned containers before creation", zap.String("server", c.config.Name), zap.String("name_pattern", namePattern)) - // Find ALL containers matching our server (running or stopped) - listCmd := c.newDockerCmd(ctx, "ps", "-a", - "--filter", "name="+namePattern, - "--format", "{{.ID}}\t{{.Names}}\t{{.Status}}") - - output, err := listCmd.Output() + // Find ALL containers owned by this server (running or stopped) + owned, err := c.listOwnedContainers(ctx, true) if err != nil { return fmt.Errorf("failed to list existing containers: %w", err) } - lines := strings.Split(strings.TrimSpace(string(output)), "\n") - if len(lines) == 0 || lines[0] == "" { - c.logger.Debug("No existing containers found - safe to create new one", + if len(owned) == 0 { + c.logger.Debug("No existing owned containers found - safe to create new one", zap.String("server", c.config.Name)) return nil } // Found existing containers - clean them up first - c.logger.Warn("Found existing containers - cleaning up before creating new one", + // container_owner: D8 rule 3 treats a count as container-subject + // evidence, so it is paired with the label Docker reported on the + // listed rows (one value for every row: ownsContainer admits only rows + // whose label equals this server's raw name) — never the requesting + // server's name (codex round 11; mirrors the upstreamLogger record + // below, already fixed). + c.logger.Warn("Found existing owned containers - cleaning up before creating new one", zap.String("server", c.config.Name), - zap.Int("container_count", len(lines))) + zap.Int("container_count", len(owned)), + containerOwnerField(owned[0].Owner)) if c.upstreamLogger != nil { + // container_owner: the count is of THIS server's owned containers + // (D8 rule 3 treats a count as a container subject, since the pre-105 + // sweep counted co-owners' containers too). Like every other + // container record, the owner is the label Docker reported + // (ownedContainer.Owner — one value for every row, since + // ownsContainer admits only rows whose label equals this server's + // raw name), never the requesting server's name (D9, codex round 3). c.upstreamLogger.Warn("Cleaning up existing containers before creating new one", - zap.Int("container_count", len(lines))) + zap.Int("container_count", len(owned)), + containerOwnerField(owned[0].Owner)) } - for _, line := range lines { - if line == "" { - continue - } - parts := strings.SplitN(line, "\t", 3) - if len(parts) >= 2 { - containerID := parts[0] - containerName := parts[1] - status := "" - if len(parts) >= 3 { - status = parts[2] - } - + // The listing is a snapshot: each row is re-verified right before its + // own rm -f (a later row may have been relabelled to a co-tenant's — + // `a-b`'s for `a/b`, same name — while an earlier one was removed), and + // every record naming a container carries the id and owner read then. + for _, listed := range owned { + status := listed.Status + // Force remove (works for running and stopped containers) + res := c.mutateOwnedContainer(ctx, listed.ID, ContainerRemove, "pre-creation", func(container ContainerRow) { c.logger.Info("Removing existing container", zap.String("server", c.config.Name), - zap.String("container_id", containerID), - zap.String("container_name", containerName), + zap.String("container_id", container.ID), + zap.String("container_name", container.Name), + containerOwnerField(container.Owner), zap.String("status", status)) - if c.upstreamLogger != nil { c.upstreamLogger.Info("Removing existing container", - zap.String("container_id", containerID), - zap.String("container_name", containerName)) - } - - // Force remove (works for running and stopped containers) - rmCmd := c.newDockerCmd(ctx, "rm", "-f", containerID) - if err := rmCmd.Run(); err != nil { - c.logger.Error("Failed to remove existing container", - zap.String("container_id", containerID), - zap.Error(err)) - // Continue anyway - try to remove others - } else { - c.logger.Info("Successfully removed existing container", - zap.String("container_id", containerID)) - - if c.upstreamLogger != nil { - c.upstreamLogger.Info("Successfully removed existing container", - zap.String("container_id", containerID)) - } + zap.String("container_id", container.ID), + zap.String("container_name", container.Name), + containerOwnerField(container.Owner)) } + }) + if !res.Verified { + continue + } + if res.Err != nil { + c.logger.Error("Failed to remove existing container", + zap.String("container_id", res.Container.ID), + containerOwnerField(res.Container.Owner), + zap.Error(res.Err)) + // Continue anyway - try to remove others + continue + } + c.logger.Info("Successfully removed existing container", + zap.String("container_id", res.Container.ID), + containerOwnerField(res.Container.Owner)) + if c.upstreamLogger != nil { + c.upstreamLogger.Info("Successfully removed existing container", + zap.String("container_id", res.Container.ID), + containerOwnerField(res.Container.Owner)) } } diff --git a/internal/upstream/core/docker_collision_output_test.go b/internal/upstream/core/docker_collision_output_test.go new file mode 100644 index 000000000..3965d6991 --- /dev/null +++ b/internal/upstream/core/docker_collision_output_test.go @@ -0,0 +1,312 @@ +package core + +import ( + "context" + "fmt" + "io" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap" + "go.uber.org/zap/zaptest/observer" + + "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/logs" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/secureenv" +) + +// Codex round 2 (PR E), docker finding 1. `a/b` and hidden `a-b` both +// generate mcpproxy-a-b-; on a suffix collision Docker refuses the +// run with its own message naming the FOREIGN container's name and full id: +// +// docker: Error response from daemon: Conflict. The container name +// "/mcpproxy-a-b-wxyz" is already in use by container "<64 hex>". … +// +// That text reaches a/b's per-server log as child output — the docker CLI's +// stderr on the stdio isolation path (monitorStderr) and the launcher-pumped +// stderr of a user-supplied `docker run` upstream (loggerWriter). Neither +// producer may write it as the record MESSAGE (D8 rule 1), both must stamp +// it `child_output=true` (logs.ChildOutputField), and the attributed reader +// tail_log uses must withhold a child-output record that mentions a +// container unless container_owner matches. The pre-spawn "Docker isolation +// configured" record names a container that does not exist yet, so it must +// not carry container_owner for that unverified name. + +const ( + collisionRequester = "a/b" // the server whose log is read + collisionHiddenOwner = "a-b" // owns the container Docker names in its refusal + collisionForeignID = "f0e1d2c3b4a5968778695a4b3c2d1e0ff0e1d2c3b4a5968778695a4b3c2d1e0f" + collisionForeignName = "mcpproxy-a-b-wxyz" +) + +// requireCollisionPremise pins what makes the fixture a collision: the two +// raw names sanitise to the same container-name stem, and the foreign name +// is exactly that stem with a generated suffix — so nothing in a/b's own +// records can tell the colliding container from its own. +func requireCollisionPremise(t *testing.T) { + t.Helper() + require.Equal(t, sanitizeServerNameForContainer(collisionRequester), sanitizeServerNameForContainer(collisionHiddenOwner), + "fixture premise: a/b and a-b must generate the same container-name stem") + require.Equal(t, "mcpproxy-"+sanitizeServerNameForContainer(collisionHiddenOwner)+"-wxyz", collisionForeignName, + "fixture premise: the foreign name is a-b's canonical container name") +} + +// dockerCollisionStderr is the docker CLI's stderr for a name conflict, +// verbatim shape. +var dockerCollisionStderr = `docker: Error response from daemon: Conflict. The container name "/` + collisionForeignName + + `" is already in use by container "` + collisionForeignID + `". You have to remove (or rename) that container to be able to reuse that name.` + +// newRealPerServerLogger returns the REAL per-server file writer for name +// (what client.go installs as upstreamLogger) over a fresh log directory, +// plus the LogConfig the readers take. +func newRealPerServerLogger(t *testing.T, name string) (*zap.Logger, *config.LogConfig) { + t.Helper() + logDir, err := os.MkdirTemp("", "mcpproxy-collision-*") + require.NoError(t, err) + t.Cleanup(func() { _ = os.RemoveAll(logDir) }) + cfg := logs.DefaultLogConfig() + cfg.LogDir = logDir + cfg.EnableFile = true + cfg.EnableConsole = false + cfg.Compress = false + require.NoError(t, os.WriteFile(filepath.Join(logDir, logs.ServerLogFilename(name)), nil, 0o600)) + logger, closer, err := logs.NewUpstreamServerLogger(cfg, name) + require.NoError(t, err) + t.Cleanup(func() { _ = closer.Close() }) + return logger, cfg +} + +// assertCollisionWithheldFromScopedReader is the shared oracle: the scoped +// reader (what tail_log serves an a/b-scoped token) never returns the +// foreign id or name, the administrator whole-file reader keeps them +// (SC-005), and an ordinary record of a/b's stays served. +func assertCollisionWithheldFromScopedReader(t *testing.T, cfg *config.LogConfig, name string) { + t.Helper() + scoped, err := logs.ReadUpstreamServerLogTailAttributed(cfg, name, 50) + require.NoError(t, err) + body := strings.Join(scoped, "\n") + assert.NotContains(t, body, collisionForeignID, "foreign container id served to %s's scoped reader:\n%s", name, body) + assert.NotContains(t, body, collisionForeignName, "foreign container name served to %s's scoped reader:\n%s", name, body) + assert.NotContains(t, body, "already in use by container", "Docker's collision text served to %s's scoped reader:\n%s", name, body) + assert.Contains(t, body, "ordinary-record-of-a-b", "a/b's own ordinary record must stay served") + + whole, err := logs.ReadUpstreamServerLogTail(cfg, name, 50) + require.NoError(t, err) + assert.Contains(t, strings.Join(whole, "\n"), collisionForeignID, "administrators keep Docker's output (SC-005)") +} + +// Launcher path, end to end: a user-supplied `docker run --name …` upstream +// for server a/b spawned through connectWithLauncher against a fake docker +// whose `run` answers with the daemon's conflict message. The child's stderr +// is pumped through loggerWriter into the REAL per-server file; the scoped +// reader must not disclose the foreign container. +func TestDockerRunCollision_LauncherStderr_NeverAttributedToRequester(t *testing.T) { + requireCollisionPremise(t) + fd := installFakeDocker(t, nil) + fd.failRunWith(t, dockerCollisionStderr) + forceDockerDaemonEnvGOOS(t, osDarwin) + + const name = collisionRequester + upstreamLogger, logCfg := newRealPerServerLogger(t, name) + mainCore, mainLogs := observer.New(zap.DebugLevel) + c := &Client{ + config: &config.ServerConfig{ + Name: name, + Protocol: "http", + URL: "http://127.0.0.1:1/mcp", + Command: "docker", + Args: []string{"run", "-i", "--rm", "--name", collisionForeignName, "mcp/example"}, + LauncherWaitTimeout: config.Duration(200 * time.Millisecond), + }, + logger: zap.New(mainCore), + upstreamLogger: upstreamLogger, + isolationManager: NewIsolationManager(config.DefaultDockerIsolationConfig()), + envManager: secureenv.NewManager(nil), + } + upstreamLogger.Info("ordinary-record-of-a-b") + + c.mu.Lock() + err := c.connectWithLauncher(context.Background()) + c.mu.Unlock() + require.Error(t, err, "the fake docker run fails, so the launcher cannot reach the URL") + _ = upstreamLogger.Sync() + + invocations := strings.Join(fd.invocations(t), "\n") + require.Contains(t, invocations, "run ", "fixture premise: docker run was invoked:\n%s", invocations) + whole, err := logs.ReadUpstreamServerLogTail(logCfg, name, 50) + require.NoError(t, err) + require.Contains(t, strings.Join(whole, "\n"), collisionForeignID, + "fixture premise: the docker CLI's stderr reached the per-server log:\n%s", strings.Join(whole, "\n")) + + assertCollisionWithheldFromScopedReader(t, logCfg, name) + // Docker's text is a field value on every launcher record, never the message. + for _, entry := range mainLogs.All() { + assert.NotContains(t, entry.Message, collisionForeignID, "child text written as a record MESSAGE: %q", entry.Message) + } +} + +// Stdio isolation path: the docker CLI is the stdio child and its stderr is +// pumped by monitorStderr into the real per-server file. +func TestDockerRunCollision_StdioStderr_NeverAttributedToRequester(t *testing.T) { + requireCollisionPremise(t) + const name = collisionRequester + upstreamLogger, logCfg := newRealPerServerLogger(t, name) + c := &Client{ + config: &config.ServerConfig{Name: name}, + logger: zap.NewNop(), + upstreamLogger: upstreamLogger, + } + upstreamLogger.Info("ordinary-record-of-a-b") + + c.monitorStderr(context.Background(), strings.NewReader(dockerCollisionStderr+"\nlistening on 127.0.0.1:9331\n")) + _ = upstreamLogger.Sync() + + assertCollisionWithheldFromScopedReader(t, logCfg, name) + scoped, err := logs.ReadUpstreamServerLogTailAttributed(logCfg, name, 50) + require.NoError(t, err) + assert.Contains(t, strings.Join(scoped, "\n"), "listening on 127.0.0.1:9331", "ordinary child stderr stays served") +} + +// Producer shape, launcher: the child's line is the `message` FIELD of a +// constant-message record stamped child_output=true. +func TestLoggerWriter_ChildLineIsStampedFieldValue(t *testing.T) { + core, observed := observer.New(zap.DebugLevel) + w := newLoggerWriter(zap.New(core), nil) + _, err := w.Write([]byte("[launcher stderr] " + dockerCollisionStderr + "\n")) + require.NoError(t, err) + + entries := observed.All() + require.Len(t, entries, 1) + assert.Equal(t, "launcher", entries[0].Message, "the record message must be a constant, never child text") + assert.Equal(t, "[launcher stderr] "+dockerCollisionStderr, entries[0].ContextMap()["message"]) + assert.Equal(t, true, entries[0].ContextMap()["child_output"], "child output must be stamped for the attributed reader") +} + +// Producer shape, stdio: the per-server stderr record is stamped +// child_output=true (the message-as-field shape predates this round). +func TestMonitorStderr_StampsChildOutput(t *testing.T) { + perServerCore, perServerLogs := observer.New(zap.DebugLevel) + c := &Client{ + config: &config.ServerConfig{Name: "a/b"}, + logger: zap.NewNop(), + upstreamLogger: zap.New(perServerCore), + } + c.monitorStderr(context.Background(), strings.NewReader(dockerCollisionStderr+"\n")) + + entries := perServerLogs.FilterMessage("stderr").All() + require.Len(t, entries, 1) + assert.Equal(t, dockerCollisionStderr, entries[0].ContextMap()["message"]) + assert.Equal(t, true, entries[0].ContextMap()["child_output"]) +} + +// Pre-spawn record: "Docker isolation configured" names the GENERATED +// container name before Docker has created or inspected anything, so it +// must not assert ownership of that name — no container_owner. (A record +// naming a container without container_owner is withheld from scoped +// callers by the reader's subject-evidence rule; administrators see it.) +func TestSetupDockerIsolation_ConfiguredRecordCarriesNoOwnerForUnverifiedName(t *testing.T) { + fakeDocker := writeFakeDockerExecutable(t) + forceDockerDaemonEnvGOOS(t, osDarwin) + orig := resolveDockerBinary + t.Cleanup(func() { resolveDockerBinary = orig }) + resolveDockerBinary = func(_ *zap.Logger) (string, error) { return fakeDocker, nil } + + c, _, upLogs := newOwnershipClient("a/b", nil) + c.setupDockerIsolation(c.config.Command, c.config.Args) + + configured := upLogs.FilterMessage("Docker isolation configured").All() + require.Len(t, configured, 1) + fields := configured[0].ContextMap() + assert.NotEmpty(t, fields["container_name"], "the generated name is still recorded") + _, hasOwner := fields["container_owner"] + assert.False(t, hasOwner, "container_owner asserted for a container that does not exist yet: %v", fields) +} + +// Codex round 3, logs finding 1: the child's stderr is re-emitted INSIDE the +// connection error. monitorStderr keeps every line in the recent-stderr +// buffer, initialize() splices that buffer into the error it returns +// (enrichTransportClosedError / the initialize-timeout branch) and Connect +// writes that error into the per-server log as the "Connection failed" +// record. The direct stderr record is a child-output record and withheld; +// the "Connection failed" record repeats the same foreign name and id and, +// without the child-output provenance, was attributed to a/b. +// +// The chain here is the production one minus the process spawn: the real +// stderr pump, the real buffer formatter, the real enrichment and the real +// per-server record write, into a real file read by the real reader. +func TestDockerRunCollision_ConnectionFailedRecord_NeverAttributedToRequester(t *testing.T) { + requireCollisionPremise(t) + const name = collisionRequester + upstreamLogger, logCfg := newRealPerServerLogger(t, name) + c := &Client{ + config: &config.ServerConfig{Name: name}, + logger: zap.NewNop(), + upstreamLogger: upstreamLogger, + transportType: transportStdio, + } + upstreamLogger.Info("ordinary-record-of-a-b") + + c.monitorStderr(context.Background(), strings.NewReader(dockerCollisionStderr+"\n")) + err := enrichTransportClosedError(c.formatRecentStderr(), io.EOF) + require.Contains(t, err.Error(), collisionForeignID, "fixture premise: the enriched error embeds the child's stderr") + c.recordConnectionFailure(fmt.Errorf("stdio transport (command=%q, docker_isolation=%t): %w", "docker", true, err)) + _ = upstreamLogger.Sync() + + whole, readErr := logs.ReadUpstreamServerLogTail(logCfg, name, 50) + require.NoError(t, readErr) + var failed []string + for _, line := range whole { + if strings.Contains(line, "Connection failed") { + failed = append(failed, line) + } + } + require.Len(t, failed, 1, "fixture premise: one Connection failed record:\n%s", strings.Join(whole, "\n")) + require.Contains(t, failed[0], collisionForeignID, "fixture premise: the record embeds the foreign id") + + assertCollisionWithheldFromScopedReader(t, logCfg, name) +} + +// The same producer with ordinary child stderr (no container mention): the +// "Connection failed" record stays attributable to its own writer — the +// child-output provenance only makes it a container SUBJECT when it names one. +func TestConnectionFailedRecord_OrdinaryChildStderrStaysAttributed(t *testing.T) { + const name = collisionRequester + upstreamLogger, logCfg := newRealPerServerLogger(t, name) + c := &Client{ + config: &config.ServerConfig{Name: name}, + logger: zap.NewNop(), + upstreamLogger: upstreamLogger, + transportType: transportStdio, + } + c.monitorStderr(context.Background(), strings.NewReader("Error: --brave-api-key is required\n")) + c.recordConnectionFailure(enrichTransportClosedError(c.formatRecentStderr(), io.EOF)) + _ = upstreamLogger.Sync() + + scoped, err := logs.ReadUpstreamServerLogTailAttributed(logCfg, name, 50) + require.NoError(t, err) + body := strings.Join(scoped, "\n") + assert.Contains(t, body, "Connection failed", "an ordinary connect failure must stay readable by its own server") + assert.Contains(t, body, "brave-api-key is required") +} + +// An HTTP transport failure embeds no child output; the record must not be +// stamped child_output (the marker is provenance, not decoration). +func TestConnectionFailedRecord_NoChildOutputMarkerWithoutChildText(t *testing.T) { + perServerCore, perServerLogs := observer.New(zap.DebugLevel) + c := &Client{ + config: &config.ServerConfig{Name: "alpha"}, + logger: zap.NewNop(), + upstreamLogger: zap.New(perServerCore), + transportType: transportHTTP, + } + c.recordConnectionFailure(fmt.Errorf("failed to start HTTP client: connection refused")) + entries := perServerLogs.FilterMessage("Connection failed").All() + require.Len(t, entries, 1) + _, stamped := entries[0].ContextMap()["child_output"] + assert.False(t, stamped, "no child text in the error, no child_output marker: %v", entries[0].ContextMap()) +} diff --git a/internal/upstream/core/docker_mutation_reverify_test.go b/internal/upstream/core/docker_mutation_reverify_test.go new file mode 100644 index 000000000..15cdbd030 --- /dev/null +++ b/internal/upstream/core/docker_mutation_reverify_test.go @@ -0,0 +1,315 @@ +package core + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap/zaptest/observer" + + "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" +) + +// Codex round 6 (PR E), docker findings 1–3 (Spec 105 FR-007 / research D9, +// moment-of-mutation rule): the core's image-fallback, name-pattern and +// pre-creation cleanups established ownership by a LISTING and consumed it +// on a LATER stop / rm -f, and stopOwnedContainer's kill after a failed stop +// was a second mutation with no read at all. Another Docker client can +// rename or relabel a container — or replace it under an id that extends +// the listed one — between the listing and the command. Every mutation now +// re-reads that one container's full id, name and label immediately before +// the command (ContainerMutator, the one verify-then-mutate implementation +// the manager sweeps share): a container that no longer satisfies the +// predicate is left alone and the refusal recorded without its id or name; +// one that still does is mutated, and every record naming it carries +// container_owner from that mutation-time read. + +// ownFixtureNamed is one container under id as name/owner label say. It +// always carries this test process's own instance label: these tests vary +// name/owner to probe the server-name-and-canonical-name half of ownership, +// not the instance half (TestOwnsContainer_Predicate and +// TestContainerOwnedByAny_Predicate cover that dimension directly). +func ownFixtureNamed(id, name, owner string) []fakeContainer { + labels := map[string]string{"com.mcpproxy.managed": "true"} + if owner != "" { + labels[ownerLabel] = owner + } + return []fakeContainer{{ID: id, Name: name, Image: "mcp/example", Status: "Up 2 minutes", Labels: withOwnInstance(labels)}} +} + +// assertNoRecordNames asserts no record in either logger carries any of +// needles in its message or a field value. +func assertNoRecordNames(t *testing.T, mainLogs, upLogs *observer.ObservedLogs, needles ...string) { + t.Helper() + for _, needle := range needles { + assert.Empty(t, recordsMentioning(upLogs, needle), "%q written into the per-server log", needle) + assert.Empty(t, recordsMentioning(mainLogs, needle), "%q written into main log", needle) + } +} + +// assertEveryRecordNamingCarriesOwner asserts every record (both loggers) +// whose field values name one of needles carries container_owner == owner, +// and that at least one such record exists. +func assertEveryRecordNamingCarriesOwner(t *testing.T, mainLogs, upLogs *observer.ObservedLogs, owner string, needles ...string) { + t.Helper() + named := 0 + for _, logs := range []*observer.ObservedLogs{mainLogs, upLogs} { + for _, entry := range logs.All() { + fields := entry.ContextMap() + if !fieldsName(fields, needles...) { + continue + } + named++ + assert.Equal(t, owner, fields["container_owner"], + "record %q must carry the owner read at mutation time: %v", entry.Message, fields) + } + } + assert.NotZero(t, named, "the mutation is recorded with its subject") +} + +// fieldsName reports whether any string field value carries one of needles. +func fieldsName(fields map[string]interface{}, needles ...string) bool { + for _, v := range fields { + s, ok := v.(string) + if !ok { + continue + } + for _, needle := range needles { + if strings.Contains(s, needle) { + return true + } + } + } + return false +} + +// extendedOwnID is a different container whose full id has the listed +// ownContainerID as a prefix: `docker ps --filter id=` is a prefix match, +// so only an exact full-id comparison tells the two apart. +const extendedOwnID = ownContainerID + "ffffffffffffffffffffffffffffffffffffffffffffffffffff" + +func TestDockerMutations_ReverifyOwnershipAtMutationTime(t *testing.T) { + paths := []struct { + name string + run func(t *testing.T, c *Client) + verb string + }{ + {name: "name pattern", verb: "stop", + run: func(_ *testing.T, c *Client) { c.killDockerContainersByNamePatternWithContext(context.Background()) }}, + {name: "image fallback", verb: "stop", + run: func(_ *testing.T, c *Client) { + c.killDockerContainersByImageWithContext(context.Background(), "mcp/example") + }}, + {name: "exact name", verb: "stop", + run: func(_ *testing.T, c *Client) { + c.killDockerContainerByNameWithContext(context.Background(), ownContainerName) + }}, + {name: "pre-creation rm", verb: "rm -f", + run: func(t *testing.T, c *Client) { require.NoError(t, c.ensureNoExistingContainers(context.Background())) }}, + } + arms := []struct { + name string + after []fakeContainer + wantOwner string // "" — the container must be left alone + }{ + {name: "relabelled foreign between listing and mutation", after: ownFixtureNamed(ownContainerID, ownContainerName, "a-b")}, + {name: "renamed to a co-tenant's shape", after: ownFixtureNamed(ownContainerID, foreignContainerName, "a-b")}, + {name: "replaced by a container whose id extends the listed one", after: ownFixtureNamed(extendedOwnID, ownContainerName, "a")}, + {name: "unchanged", after: ownFixtureNamed(ownContainerID, ownContainerName, "a"), wantOwner: "a"}, + } + for _, path := range paths { + for _, arm := range arms { + t.Run(path.name+"/"+arm.name, func(t *testing.T) { + fd := installFakeDocker(t, ownFixtureNamed(ownContainerID, ownContainerName, "a")) + // The listing sees the original state, everything after it the changed one. + fd.swapFixtureAfterPs(t, 1, arm.after) + c, mainLogs, upLogs := newOwnershipClient("a", &config.ServerConfig{ + Command: "docker", Args: []string{"run", "-i", "--rm", "mcp/example"}, + }) + + path.run(t, c) + + for _, line := range fd.invocations(t) { + if strings.HasPrefix(line, "ps") { + assert.Contains(t, line, "--no-trunc", "every ownership read must return the FULL id: %q", line) + } + } + mutations := append(fd.mutationsOf(t, ownContainerID), fd.mutationsOf(t, extendedOwnID)...) + if arm.wantOwner == "" { + assert.Empty(t, mutations, "a container whose ownership changed was mutated; invocations:\n%s", + strings.Join(fd.invocations(t), "\n")) + // The ids are never named; the name only when it is not the + // tracked one (the exact-name path's own lookup record names + // what it was asked for — this server's knowledge, not a read). + needles := []string{ownContainerID, extendedOwnID} + if arm.after[0].Name != ownContainerName { + needles = append(needles, arm.after[0].Name) + } + assertNoRecordNames(t, mainLogs, upLogs, needles...) + assert.NotEmpty(t, recordsMentioning(mainLogs, "no longer canonically owned"), "the refusal is recorded") + return + } + assert.Equal(t, []string{path.verb + " " + ownContainerID}, mutations, "an owned container is still cleaned up") + assertEveryRecordNamingCarriesOwner(t, mainLogs, upLogs, "a", ownContainerID) + }) + } + } +} + +// Pre-creation cleanup re-verifies per ROW, not per listing: with two owned +// containers listed, the second is relabelled to the colliding co-tenant +// (`a/b` and `a-b` both name mcpproxy-a-b-*, so only the label separates +// them) while the first is being removed. The first is removed; the second +// is left alone and never named. +func TestDockerCleanup_PreCreationReverifiesEachRow_SlashVsDashCollision(t *testing.T) { + const firstID = "0a0a0a0a0a0a" + const firstName = "mcpproxy-a-b-wxyz" + const secondID = "0b0b0b0b0b0b" + const secondName = "mcpproxy-a-b-q2w3" + row := func(id, name, owner string) fakeContainer { + return fakeContainer{ID: id, Name: name, Image: "mcp/example", Status: "Up", Labels: withOwnInstance(map[string]string{ownerLabel: owner})} + } + fd := installFakeDocker(t, []fakeContainer{row(firstID, firstName, "a/b"), row(secondID, secondName, "a/b")}) + // ps 1: the listing; ps 2: the first row's re-read. The second row is + // a-b's by the time its own re-read happens. + fd.swapFixtureAfterPs(t, 2, []fakeContainer{row(firstID, firstName, "a/b"), row(secondID, secondName, "a-b")}) + c, mainLogs, upLogs := newOwnershipClient("a/b", nil) + + require.NoError(t, c.ensureNoExistingContainers(context.Background())) + + assert.Equal(t, []string{"rm -f " + firstID}, fd.mutationsOf(t, firstID), "the unchanged first row is removed") + assert.Empty(t, fd.mutationsOf(t, secondID), "the row relabelled to a-b was removed by a/b; invocations:\n%s", + strings.Join(fd.invocations(t), "\n")) + assertNoRecordNames(t, mainLogs, upLogs, secondID, secondName) + assertEveryRecordNamingCarriesOwner(t, mainLogs, upLogs, "a/b", firstID) + assert.NotEmpty(t, recordsMentioning(mainLogs, "no longer canonically owned"), "the refusal is recorded") +} + +// The kill after a failed `docker stop` is a second mutation: ownership is +// re-read again before it. Exact-name path: ps 1 is the name lookup, ps 2 +// the stop's re-read, ps 3 the kill's. +func TestDockerStopEscalation_ReverifiesBeforeKill(t *testing.T) { + arms := []struct { + name string + after []fakeContainer + wantOwner string + }{ + {name: "relabelled between stop and kill", after: ownFixtureNamed(ownContainerID, ownContainerName, "a-b")}, + {name: "replaced by a container whose id extends the listed one", after: ownFixtureNamed(extendedOwnID, ownContainerName, "a")}, + {name: "unchanged", after: ownFixtureNamed(ownContainerID, ownContainerName, "a"), wantOwner: "a"}, + } + killMessages := []string{"Owned container force killed", "Successfully force killed owned container", "Failed to kill owned container"} + for _, arm := range arms { + t.Run(arm.name, func(t *testing.T) { + fd := installFakeDocker(t, ownFixtureNamed(ownContainerID, ownContainerName, "a")) + fd.failVerbs(t, "stop") + fd.swapFixtureAfterPs(t, 2, arm.after) + c, mainLogs, upLogs := newOwnershipClient("a", nil) + + stopped := c.killDockerContainerByNameWithContext(context.Background(), ownContainerName) + + require.Contains(t, fd.mutationsOf(t, ownContainerID), "stop "+ownContainerID, "premise: the stop ran with ownership held") + kills := 0 + for _, line := range fd.invocations(t) { + if strings.HasPrefix(line, "kill ") { + kills++ + } + } + if arm.wantOwner == "" { + assert.False(t, stopped) + assert.Zero(t, kills, "a container whose ownership changed after the stop was killed; invocations:\n%s", + strings.Join(fd.invocations(t), "\n")) + assertNoRecordNames(t, mainLogs, upLogs, extendedOwnID, "a-b") + for _, msg := range killMessages { + assert.Empty(t, mainLogs.FilterMessage(msg).All(), "%q recorded for a refused kill", msg) + assert.Empty(t, upLogs.FilterMessage(msg).All(), "%q recorded for a refused kill", msg) + } + assert.NotEmpty(t, recordsMentioning(mainLogs, "no longer canonically owned"), "the refusal is recorded") + // The stop's own records named the container with the owner read for the stop. + assertEveryRecordNamingCarriesOwner(t, mainLogs, upLogs, "a", ownContainerID) + return + } + assert.True(t, stopped) + assert.Equal(t, []string{"stop " + ownContainerID, "kill " + ownContainerID}, fd.mutationsOf(t, ownContainerID)) + require.Len(t, upLogs.FilterMessage("Owned container force killed").All(), 1) + assertEveryRecordNamingCarriesOwner(t, mainLogs, upLogs, "a", ownContainerID) + }) + } +} + +// Codex round 6, docker finding 4: on its wait timeout the docker-logs +// monitor read the cidfile itself and recorded that id — plus an executable +// `docker logs` command — with no ownership read, so a direct `docker run +// --name custom` server or a tampered cidfile named a foreign container as +// this server's. The monitor takes the id only from the tracked state +// trackCidfileContainer verified: none → a record naming no id and no +// command; a verified one → id and container_owner from that verification. +func TestMonitorDockerLogs_NamesOnlyAVerifiedContainer(t *testing.T) { + shorten := func(t *testing.T) { + t.Helper() + prev := dockerLogsWaitTimeout + // Longer than the monitor's 100 ms poll tick, so a tracked id is + // seen before the timeout fires (production: 10 s). + dockerLogsWaitTimeout = 300 * time.Millisecond + t.Cleanup(func() { dockerLogsWaitTimeout = prev }) + } + const startedMsg = "Docker container started - logs available via 'docker logs' command" + const foreignFullID = foreignContainerID + "0000000000000000000000000000000000000000000000000000" + + t.Run("unverified cidfile on timeout", func(t *testing.T) { + shorten(t) + installFakeDocker(t, ownAndForeignFixture()) + c, mainLogs, upLogs := newOwnershipClient("a", nil) + cidFile := filepath.Join(t.TempDir(), "cid") + require.NoError(t, os.WriteFile(cidFile, []byte(foreignFullID+"\n"), 0o600)) + + done := make(chan struct{}) + go func() { + defer close(done) + c.monitorDockerLogsWithContext(context.Background(), cidFile) + }() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("monitor did not return after the wait timeout") + } + + assertNoRecordNames(t, mainLogs, upLogs, foreignFullID, foreignContainerID, "docker logs") + assert.Empty(t, mainLogs.FilterMessage(startedMsg).All(), "an unverified container was announced as started") + assert.NotEmpty(t, recordsMentioning(mainLogs, "verified"), "the timeout records that no container was verified") + }) + + t.Run("verified tracked container", func(t *testing.T) { + shorten(t) + installFakeDocker(t, ownAndForeignFixture()) + c, mainLogs, _ := newOwnershipClient("a", nil) + c.trackCidfileContainer(context.Background(), ownContainerID, 0) + require.Equal(t, ownContainerID, c.containerID, "premise: the tracked id is the verified one") + + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { + defer close(done) + c.monitorDockerLogsWithContext(ctx, filepath.Join(t.TempDir(), "never-read")) + }() + require.Eventually(t, func() bool { return len(mainLogs.FilterMessage(startedMsg).All()) == 1 }, 5*time.Second, 10*time.Millisecond) + cancel() + <-done + + started := mainLogs.FilterMessage(startedMsg).All()[0].ContextMap() + assert.Equal(t, shortContainerID(ownContainerID), started["container_id"]) + assert.Equal(t, "a", started["container_owner"], "the id is named with the owner from its verification") + assert.Equal(t, "docker logs -f "+shortContainerID(ownContainerID), started["command"]) + for _, entry := range mainLogs.All() { + fields := entry.ContextMap() + if fieldsName(fields, ownContainerID, shortContainerID(ownContainerID)) { + assert.Equal(t, "a", fields["container_owner"], "record %q names the container without its owner: %v", entry.Message, fields) + } + } + }) +} diff --git a/internal/upstream/core/docker_ownership.go b/internal/upstream/core/docker_ownership.go new file mode 100644 index 000000000..dbe8bfbc6 --- /dev/null +++ b/internal/upstream/core/docker_ownership.go @@ -0,0 +1,663 @@ +package core + +import ( + "context" + "os/exec" + "regexp" + "strings" + + "go.uber.org/zap" +) + +// Spec 105 FR-007 (gap FR007-G5, research D9): canonical container ownership. +// +// Every container mcpproxy creates is named `mcpproxy--<4 chars>` +// (generateContainerName) and labelled `com.mcpproxy.server=` +// (formatContainerLabels). The name alone is NOT ownership evidence: `a/b` +// and `a-b` both sanitise to `a-b`, and `a`'s old prefix filter +// (`name=mcpproxy-a-`) is a substring match that also lists `a-b`'s +// containers. Pre-105 every cleanup path removed those foreign containers +// and wrote their ids and names into `a`'s per-server log, which +// `upstream_servers tail_log` serves to an `a`-scoped agent. +// +// A container is owned by server S, on THIS mcpproxy instance, iff ALL hold: +// - its com.mcpproxy.server label equals S's raw name exactly, +// - its com.mcpproxy.instance label equals this process's own instance id +// (core.GetInstanceID(), instance.go) exactly, and +// - its name matches ^mcpproxy--[a-z0-9]{4}$ (the regex guards +// against a foreign process re-using the label). +// +// The instance check matters even though every reader here is scoped to one +// server name: two separate mcpproxy processes (distinct data dirs) can both +// configure a server literally named `a`, and without it either instance's +// cleanup would stop, kill or rm the OTHER instance's live container for +// that name — a container neither created nor is otherwise entitled to touch +// (the vulnerability class this file exists to close, now recurring one +// level up: server-name-scoped but not instance-scoped is exactly as +// canonical as name-scoped but not label-scoped was pre-105). +// +// Docker applies all three filters server-side (`--filter label=` is an +// exact match, `--filter name=` a regexp match) and ownsContainer re-checks +// them in Go, so no container that fails any of them is ever mutated or +// logged. Pre-label containers, containers from another instance, and +// user-`--name` containers are all left alone: they never were ours by this +// rule. That holds on EVERY stop/kill/rm path, including the two that +// start from a single known container rather than a listing (codex round 1): +// the id read from the cidfile of this server's own `docker run` — a +// user-configured direct `docker run --name custom` gets a cidfile but no +// label and no canonical name — and the exact tracked name. Both look the +// container up (lookupOwnedContainerByID / lookupOwnedContainerByName) and +// apply ownsContainer before acting and before writing a record. Every +// housekeeping record that names a container carries `container_owner` — the +// label value READ BACK from Docker, never the requesting server's name — so +// the attributed log reader (internal/logs, D8 rule 3) can prove the subject +// belongs to the requested server; that field is the only +// administrator-visible change (SC-005). +// +// Ownership is a fact about NOW, not about a listing (codex rounds 5 and 6): +// another Docker client can rename or relabel a container — or replace it +// under an id that extends the listed one — between the `docker ps` that +// found it and the stop/kill/rm that acts on it. So every mutation, on every +// path here and in the manager's sweeps, goes through ContainerMutator: it +// re-reads that one container's FULL id, name and label immediately before +// the command, applies the predicate again, refuses (naming no id) when it +// no longer holds, and hands back the row it read so the caller's records +// carry container_id and container_owner from that read alone. + +// containerOwnerLabel is the Docker label carrying the RAW server name of the +// mcpproxy server a container was created for (formatContainerLabels). +const containerOwnerLabel = "com.mcpproxy.server" + +// containerInstanceLabel is the Docker label carrying this mcpproxy +// PROCESS's instance id (formatContainerLabels, instance.go). Two mcpproxy +// instances on one Docker host can each configure a server with the same +// name — distinct data dirs, distinct config.db, but nothing stops the same +// server name appearing in both — and before this label was checked here, +// ownsContainer admitted either instance's container for that name (codex +// round: FR-007 canonical ownership was server-name-scoped but not +// instance-scoped, so it was neither canonical across instances nor immune +// to two COOPERATING mcpproxy processes colliding on a name). That is the +// gap this label closes. +// +// It does NOT, and cannot, make ownership cryptographically unforgeable +// against a fully Docker-capable adversary (codex round 2): Docker labels +// are plain, uninterpreted, unauthenticated string metadata — anyone who +// can run `docker run --label` can copy this instance's real id verbatim +// (readable off any of its own containers with a plain `docker inspect`, +// no guessing required) onto a container of their own. That actor already +// holds the Docker socket, i.e. is already equivalent to root on this +// host's containers; no label scheme defeats them, and the pre-existing +// com.mcpproxy.server check never claimed to either (ContainerOwnedByAny's +// own doc: "which any foreign container can copy"). FR-007's canonical +// ownership is scoped to distinguishing mcpproxy's OWN legitimate +// containers — between configured servers, and now between live mcpproxy +// instances — not to authenticating labels against a host-level attacker; +// that would need a different mechanism entirely (signed labels, or state +// kept outside Docker's label store) and is out of this fix's scope. +const containerInstanceLabel = "com.mcpproxy.instance" + +// ownedContainerSuffixPattern is the random suffix generateRandomSuffix +// produces: four lowercase alphanumerics. +const ownedContainerSuffixPattern = "[a-z0-9]{4}" + +// ownedContainerNamePattern returns the anchored regexp every container +// owned by serverName must match by name. +func ownedContainerNamePattern(serverName string) string { + return "^mcpproxy-" + regexp.QuoteMeta(sanitizeServerNameForContainer(serverName)) + "-" + ownedContainerSuffixPattern + "$" +} + +// ownsContainer is the Go-side ownership predicate: label AND canonical name +// AND this process's own instance id. instanceLabel is the +// com.mcpproxy.instance value Docker reported for the row; a container +// created by a different mcpproxy instance (or one with no instance label at +// all — pre-#1300, or forged) fails this exactly like a pre-label container +// fails the server-name half: it is a foreign container, never touched. +func ownsContainer(serverName, containerName, ownerLabel, instanceLabel string) bool { + if ownerLabel != serverName { + return false + } + if instanceLabel == "" || instanceLabel != getInstanceID() { + return false + } + matched, err := regexp.MatchString(ownedContainerNamePattern(serverName), containerName) + return err == nil && matched +} + +// ownedContainer is one `docker ps` row that passed the ownership predicate. +type ownedContainer struct { + ID string + Name string + Status string + Image string + Owner string // the com.mcpproxy.server label value (== the server's raw name) + Instance string // the com.mcpproxy.instance label value +} + +// ownedContainerFormat is the `docker ps --format` template the ownership +// listing reads: one tab-separated row per container, label values last so +// an empty label leaves its column empty rather than shifting the others. +const ownedContainerFormat = "{{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Image}}\t{{.Label \"" + containerOwnerLabel + "\"}}\t{{.Label \"" + containerInstanceLabel + "\"}}" + +// listOwnedContainers lists the containers canonically owned by this server. +// includeStopped adds `-a` (stopped containers too). Rows that fail the +// Go-side predicate are dropped before anything is logged or mutated. +func (c *Client) listOwnedContainers(ctx context.Context, includeStopped bool) ([]ownedContainer, error) { + return c.listOwnedContainersFiltered(ctx, includeStopped, + "label="+containerOwnerLabel+"="+c.config.Name, + "label="+containerInstanceLabel+"="+getInstanceID(), + "name="+ownedContainerNamePattern(c.config.Name)) +} + +// lookupOwnedContainerByID resolves one full container id (the cidfile id) +// to an owned container. ok is false when Docker knows no such container or +// it fails ownsContainer — a user-`--name` container, a pre-label one, a +// foreign one — in which case the caller leaves it alone. `--filter id=` is +// a prefix match, so the row is matched back by its full id exactly. +func (c *Client) lookupOwnedContainerByID(ctx context.Context, id string) (ownedContainer, bool, error) { + rows, err := c.listOwnedContainersFiltered(ctx, true, "id="+id) + if err != nil { + return ownedContainer{}, false, err + } + for _, row := range rows { + if row.ID == id { + return row, true, nil + } + } + return ownedContainer{}, false, nil +} + +// lookupOwnedContainerByName resolves one exact container name to an owned +// container. ok is false when no container of that name is canonically owned +// by this server: a foreign `--label com.mcpproxy.server= --name +// custom` container matches the label filter but not the name half of +// ownsContainer and is left alone. +func (c *Client) lookupOwnedContainerByName(ctx context.Context, name string) (ownedContainer, bool, error) { + rows, err := c.listOwnedContainersFiltered(ctx, true, + "label="+containerOwnerLabel+"="+c.config.Name, + "label="+containerInstanceLabel+"="+getInstanceID(), + "name=^"+regexp.QuoteMeta(name)+"$") + if err != nil { + return ownedContainer{}, false, err + } + for _, row := range rows { + if row.Name == name { + return row, true, nil + } + } + return ownedContainer{}, false, nil +} + +// listOwnedContainersFiltered runs `docker ps [-a] --no-trunc --filter +// ...` with the ownership --format and returns only the rows that pass +// ownsContainer with the label value Docker reported. Every lookup goes +// through here so no path can act on, or log, a container the predicate did +// not admit. --no-trunc makes {{.ID}} the full id, which is what every +// mutation is later matched back against exactly. +func (c *Client) listOwnedContainersFiltered(ctx context.Context, includeStopped bool, filters ...string) ([]ownedContainer, error) { + args := []string{"ps"} + if includeStopped { + args = append(args, "-a") + } + args = append(args, "--no-trunc") + for _, filter := range filters { + args = append(args, "--filter", filter) + } + args = append(args, "--format", ownedContainerFormat) + + output, err := c.newDockerCmd(ctx, args...).Output() + if err != nil { + return nil, err + } + + // NOT strings.TrimSpace(output) before splitting (codex round 3): Instance + // is the LAST templated field, so an attacker-controlled label value + // ending in its own literal tab renders as a trailing tab on the last + // line of output — indistinguishable from ordinary trailing whitespace, + // which TrimSpace (it treats \t as whitespace) would silently strip, + // collapsing the row back to the expected field count and admitting the + // forged suffix as if it were never there. Splitting on the raw output + // and dropping only genuinely empty lines (docker's own trailing + // newline) leaves that trailing tab exactly where the attacker put it, + // so the exact-count check below still rejects the row. + lines := strings.Split(string(output), "\n") + + var owned []ownedContainer + for _, line := range lines { + if line == "" { + continue + } + // EXACTLY 6, never "at least" (codex round, HIGH: FR-007 + // instance-scoping fix): Docker label VALUES are arbitrary bytes + // with no tab-escaping, so a label an attacker controls (Owner or + // Instance, on a container they created themselves) could + // otherwise smuggle "\t" past an exact-match + // comparison. A genuine row from ownedContainerFormat's 5 literal + // tabs always splits to exactly 6 fields. + // + // A wrong count here is not just THIS row's problem (codex round + // 4): a label value can also contain a literal NEWLINE, splitting + // what Docker rendered as ONE container's row into what LOOKS like + // two lines — one usually short (missing fields, caught here) and + // one that can be padded with the label's own extra tabs to land + // on exactly 6 fields, forging an entire fabricated row for an id, + // name, owner and instance of the attacker's choosing. Once any + // line's boundaries are known to be untrustworthy, no other line's + // field count can be trusted either — the WHOLE listing is + // discarded (fail closed: report nothing found) rather than + // quietly keeping the rows that still look well-formed. + parts := strings.Split(line, "\t") + if len(parts) != 6 { + c.logger.Warn("Discarding container listing: a docker ps row did not parse to the expected field count", + zap.String("server", c.config.Name)) + return nil, nil + } + row := ownedContainer{ID: parts[0], Name: parts[1], Status: parts[2], Image: parts[3], Owner: parts[4], Instance: parts[5]} + if !ownsContainer(c.config.Name, row.Name, row.Owner, row.Instance) { + continue + } + owned = append(owned, row) + } + return owned, nil +} + +// containerOwnerField is the housekeeping-record field that lets the +// attributed log reader (D8 rule 3) verify the record's subject: the value of +// the container's com.mcpproxy.server label as Docker reported it +// (ownedContainer.Owner). It is never derived from the requesting server's +// name: a container identified by the cidfile of this server's own `docker +// run` is inspected first, since a direct `docker run --name custom` upstream +// gets a cidfile but no label. +func containerOwnerField(owner string) zap.Field { + return zap.String("container_owner", owner) +} + +// dockerContainerLogFields renders the container_name/container_id/ +// container_owner fields for a lifecycle housekeeping line (connection +// failure, init failure, disconnect) — but only when containerID is +// non-empty. containerID is assigned exactly by trackCidfileContainer or the +// cidfile-timeout name-recovery fallback (docker.go), both of which verify +// ownership via Docker's label/canonical-name read-back before ever setting +// it, and always pair it with containerOwner (the label value read back) +// under the same lock, clearing both together on cleanup. containerName +// alone is set at spawn time from the GENERATED canonical name, before +// Docker has confirmed anything exists — it is not evidence on its own +// (Spec 105 D8): another Docker client can have relabelled or reused that +// exact name for a colliding server between generation and this log line. +// So an empty containerID here means the record names the server only, +// never a container; these callers must not perform a Docker read of their +// own to firm the name up — they run on failure/disconnect paths where the +// tracked state is all there is to go on. +func dockerContainerLogFields(containerID, containerName, containerOwner string) []zap.Field { + if containerID == "" { + return nil + } + return []zap.Field{ + zap.String("container_name", containerName), + zap.String("container_id", containerID), + containerOwnerField(containerOwner), + } +} + +// ContainerMutation is one of the docker commands that change a container's +// state. +type ContainerMutation string + +const ( + ContainerStop ContainerMutation = "stop" + ContainerKill ContainerMutation = "kill" + ContainerRemove ContainerMutation = "rm" // run as `docker rm -f` +) + +// DockerCommand builds one docker invocation: the core client resolves the +// binary through newDockerCmd, the manager execs the bare name. +type DockerCommand func(ctx context.Context, args ...string) *exec.Cmd + +// ContainerRow is a container's identity AND running state as Docker +// reported them in one read: full id, name, com.mcpproxy.server label value +// and `docker ps`'s own human status text (e.g. "Up 5 minutes", +// "Exited (0) 2 minutes ago", "Up 5 minutes (Paused)"). Status, not just +// identity, comes from this same read (codex round 16 finding 1): a caller +// that re-read state with a SEPARATE `docker inspect ` after Verify +// trusted whatever container held that id at the LATER moment, which another +// Docker client can have relabelled or renamed in between. +type ContainerRow struct { + ID string + Name string + Owner string + Instance string + Status string +} + +// Running reports whether the container was up — running or paused, exactly +// `docker inspect`'s State.Running — at the read that produced this row. +// `docker ps --format` has no `.Running` boolean field (verified against a +// live daemon: only `.State`, the short State.Status word, and `.Status`, +// the human text `docker ps` prints in its STATUS column); `.State` is +// State.Status, not State.Running, so a paused container (State.Status +// "paused", State.Running true) would misreport as not running through it. +// Docker's own convention for that STATUS text prefixes "Up" precisely when +// State.Running is true — including while paused ("Up 5 minutes (Paused)") +// — so Status carries the same information State.Running would, without a +// second command. +func (r ContainerRow) Running() bool { + return strings.HasPrefix(r.Status, "Up") +} + +// containerRowFormat is the `docker ps --format` a mutation's re-read uses. +// {{.Status}} rides along with identity so a caller deciding running/healthy +// state never needs a second, separately-timed `docker inspect` (codex round +// 16 finding 1): ownership and state come from the identical read. +const containerRowFormat = "{{.ID}}\t{{.Names}}\t{{.Label \"" + containerOwnerLabel + "\"}}\t{{.Label \"" + containerInstanceLabel + "\"}}\t{{.Status}}" + +// MutationResult is what ContainerMutator.Mutate reports. Verified is true +// when ownership held at the re-read and the command ran, in which case +// Container is the row read then and Err the docker error if the command +// failed. Verified false with a nil Err means the container no longer +// satisfies the predicate (or is gone); with a non-nil Err the re-read +// itself failed. Nothing was run in either case. +type MutationResult struct { + Container ContainerRow + Verified bool + Err error +} + +// ContainerMutator is the one verify-then-mutate implementation every Docker +// mutation goes through — the core client's cleanup paths and the manager's +// sweeps alike (Spec 105 FR-007 / D9, codex round 6). Docker runs the CLI; +// Owns is the ownership predicate over the name and label read back +// (ownsContainer for one server, ContainerOwnedByAny for the manager). +type ContainerMutator struct { + Docker DockerCommand + Owns func(containerName, ownerLabel, instanceLabel string) bool +} + +// Mutate re-reads container id immediately before running op on it and runs +// it only if the row read back has exactly that full id and satisfies Owns. +// intent, when set, is called with that row right before the command so the +// caller can record what is about to happen with mutation-time evidence. +func (cm ContainerMutator) Mutate(ctx context.Context, id string, op ContainerMutation, intent func(ContainerRow)) MutationResult { + row, ok, err := cm.Verify(ctx, id) + if err != nil { + return MutationResult{Err: err} + } + if !ok { + return MutationResult{} + } + if intent != nil { + intent(row) + } + args := []string{string(op)} + if op == ContainerRemove { + args = append(args, "-f") + } + args = append(args, row.ID) + return MutationResult{Container: row, Verified: true, Err: cm.Docker(ctx, args...).Run()} +} + +// Verify re-reads container id and reports whether it still satisfies Owns +// right now — the same read+predicate Mutate applies before running a +// command, exposed for a caller that only needs to confirm ownership +// without mutating anything (e.g. a health check re-establishing ownership +// before trusting `docker inspect`, codex round 8). ok is true only when +// the read succeeded and the row's label and name pass Owns; row is the +// meaningful evidence — id, name and owner label as Docker reported them — +// only when ok is true. +func (cm ContainerMutator) Verify(ctx context.Context, id string) (ContainerRow, bool, error) { + row, ok, err := cm.read(ctx, id) + if err != nil { + return ContainerRow{}, false, err + } + if !ok || !cm.Owns(row.Name, row.Owner, row.Instance) { + return ContainerRow{}, false, nil + } + return row, true, nil +} + +// read runs `docker ps -a --no-trunc --filter id=` and returns the row +// whose full id is exactly id. `--filter id=` is a prefix match, so a +// different container whose id extends a listed one is not it. +func (cm ContainerMutator) read(ctx context.Context, id string) (ContainerRow, bool, error) { + output, err := cm.Docker(ctx, "ps", "-a", "--no-trunc", "--filter", "id="+id, "--format", containerRowFormat).Output() + if err != nil { + return ContainerRow{}, false, err + } + // NOT strings.TrimSpace(output) before splitting, and no per-line + // "keep scanning" on a bad count (codex rounds 3 and 4: FR-007 + // instance-scoping fix). containerRowFormat's 4 literal tabs always + // split a genuine row to exactly 5 fields, Status possibly empty (a + // pre-label container docker never started, though that never reaches + // here) but still present as its own field. Docker label VALUES have + // no tab- or newline-escaping: --filter id= only constrains this to + // the container Docker itself knows as id, but that container can be + // one the caller (a Docker-capable actor) created themselves, with an + // Owner or Instance label engineered to smuggle "\t" + // past an exact-match comparison, or — worse — containing a literal + // newline that splits what Docker rendered as ONE row into what looks + // like a second, independently well-formed line for a DIFFERENT id of + // the attacker's choosing. A single malformed line proves this read's + // line boundaries are untrustworthy, so ANY bad count fails the WHOLE + // read closed (not found) rather than continuing to look for a + // plausible match elsewhere in the output. + for _, line := range strings.Split(string(output), "\n") { + if line == "" { + continue + } + parts := strings.Split(line, "\t") + if len(parts) != 5 { + return ContainerRow{}, false, nil + } + if parts[0] != id { + continue + } + return ContainerRow{ID: parts[0], Name: parts[1], Owner: parts[2], Instance: parts[3], Status: parts[4]}, true, nil + } + return ContainerRow{}, false, nil +} + +// containerMutator is this client's ContainerMutator: its docker resolver +// and its own ownership predicate. +func (c *Client) containerMutator() ContainerMutator { + return ContainerMutator{ + Docker: c.newDockerCmd, + Owns: func(containerName, ownerLabel, instanceLabel string) bool { + return ownsContainer(c.config.Name, containerName, ownerLabel, instanceLabel) + }, + } +} + +// mutateOwnedContainer runs op on container id through the ContainerMutator +// and records a refusal — naming the server only, never the id — in both +// loggers when the container could not be verified or is no longer this +// server's. The caller records the outcome from the row handed back. +func (c *Client) mutateOwnedContainer(ctx context.Context, id string, op ContainerMutation, cleanupPath string, intent func(ContainerRow)) MutationResult { + res := c.containerMutator().Mutate(ctx, id, op, intent) + switch { + case res.Verified: + case res.Err != nil: + c.logger.Warn("Could not verify container ownership before mutation - leaving it alone", + zap.String("server", c.config.Name), + zap.String("cleanup_path", cleanupPath), + zap.String("operation", string(op)), + zap.Error(res.Err)) + if c.upstreamLogger != nil { + c.upstreamLogger.Warn("Could not verify container ownership before mutation - leaving it alone", + zap.String("cleanup_path", cleanupPath), + zap.String("operation", string(op)), + zap.Error(res.Err)) + } + default: + c.logger.Info("Container is no longer canonically owned by this server - leaving it alone", + zap.String("server", c.config.Name), + zap.String("cleanup_path", cleanupPath), + zap.String("operation", string(op))) + if c.upstreamLogger != nil { + c.upstreamLogger.Info("Container is no longer canonically owned by this server - leaving it alone", + zap.String("cleanup_path", cleanupPath), + zap.String("operation", string(op))) + } + } + return res +} + +// stopOwnedContainer stops (then force-kills) the container tracked or +// listed as id and records the outcome in both loggers. cleanupPath names +// the path that found the container ("name pattern", "image", "cidfile", +// "exact name") for the records. Ownership is re-established immediately +// before the stop AND, since it is a second mutation, again before the kill +// (codex round 6); every record that names the container carries the id and +// owner read for that command. It reports whether the container was stopped +// or killed. +func (c *Client) stopOwnedContainer(ctx context.Context, id, cleanupPath string) bool { + stop := c.mutateOwnedContainer(ctx, id, ContainerStop, cleanupPath, func(container ContainerRow) { + c.logger.Info("Killing owned container", + zap.String("server", c.config.Name), + zap.String("cleanup_path", cleanupPath), + zap.String("container_id", container.ID), + zap.String("container_name", container.Name), + containerOwnerField(container.Owner)) + if c.upstreamLogger != nil { + c.upstreamLogger.Info("Killing owned container", + zap.String("cleanup_path", cleanupPath), + zap.String("container_id", container.ID), + zap.String("container_name", container.Name), + containerOwnerField(container.Owner)) + } + }) + if !stop.Verified { + return false + } + if stop.Err == nil { + c.logger.Info("Successfully stopped owned container", + zap.String("server", c.config.Name), + zap.String("cleanup_path", cleanupPath), + zap.String("container_id", stop.Container.ID), + containerOwnerField(stop.Container.Owner)) + if c.upstreamLogger != nil { + c.upstreamLogger.Info("Owned container stopped gracefully", + zap.String("cleanup_path", cleanupPath), + zap.String("container_id", stop.Container.ID), + containerOwnerField(stop.Container.Owner)) + } + return true + } + + // Force kill if graceful stop fails + kill := c.mutateOwnedContainer(ctx, id, ContainerKill, cleanupPath, nil) + if !kill.Verified { + return false + } + if kill.Err != nil { + c.logger.Error("Failed to kill owned container", + zap.String("server", c.config.Name), + zap.String("cleanup_path", cleanupPath), + zap.String("container_id", kill.Container.ID), + containerOwnerField(kill.Container.Owner), + zap.Error(kill.Err)) + if c.upstreamLogger != nil { + c.upstreamLogger.Error("Failed to kill owned container", + zap.String("cleanup_path", cleanupPath), + zap.String("container_id", kill.Container.ID), + containerOwnerField(kill.Container.Owner), + zap.Error(kill.Err)) + } + return false + } + c.logger.Info("Successfully force killed owned container", + zap.String("server", c.config.Name), + zap.String("cleanup_path", cleanupPath), + zap.String("container_id", kill.Container.ID), + containerOwnerField(kill.Container.Owner)) + if c.upstreamLogger != nil { + c.upstreamLogger.Info("Owned container force killed", + zap.String("cleanup_path", cleanupPath), + zap.String("container_id", kill.Container.ID), + containerOwnerField(kill.Container.Owner)) + } + return true +} + +// ContainerOwnedByAny is the whole-manager predicate: a container (its name, +// its com.mcpproxy.server label and its com.mcpproxy.instance label as +// Docker reported them) is canonically owned by one of serverNames — the +// configured servers, on THIS mcpproxy instance — under the same +// label-AND-name-AND-instance rule ownsContainer applies per server. The +// manager's shutdown and emergency sweeps select containers by the shared +// com.mcpproxy.managed / com.mcpproxy.instance labels, which any foreign +// container can copy; only the rows this admits may be stopped, removed or +// named (codex round 3). Requiring the instance label here too (not just in +// the sweep's own Docker filter) closes the gap where a sweep that filtered +// broadly, or a caller re-verifying a single tracked id with no filter at +// all, would otherwise admit another live mcpproxy instance's container for +// a same-named server. +func ContainerOwnedByAny(serverNames []string, containerName, ownerLabel, instanceLabel string) bool { + for _, serverName := range serverNames { + if ownsContainer(serverName, containerName, ownerLabel, instanceLabel) { + return true + } + } + return false +} + +// ForceRemoveTrackedContainerIfOwned is the manager's emergency path for a +// client whose Disconnect hung: `docker rm -f` the container tracked as +// containerID, but only after re-establishing canonical ownership NOW — the +// same ContainerMutator every mutation goes through — so a container +// renamed, relabelled or reused under that id since it was tracked is left +// alone (codex round 3). owned reports whether the predicate admitted the +// container (removal was attempted) and owner is then the +// com.mcpproxy.server label read back at that moment — the evidence the +// caller's own records must carry when they name the container (D8, codex +// round 5); err is the docker error when removal ran and failed, or the +// lookup error. Records carry container_owner from the label read back; an +// unowned container is never named in the per-server log. +func (c *Client) ForceRemoveTrackedContainerIfOwned(ctx context.Context, containerID string) (owner string, owned bool, err error) { + if containerID == "" { + return "", false, nil + } + res := c.mutateOwnedContainer(ctx, containerID, ContainerRemove, "force", func(container ContainerRow) { + c.logger.Warn("Force removing owned container", + zap.String("server", c.config.Name), + zap.String("cleanup_path", "force"), + zap.String("container_id", container.ID), + zap.String("container_name", container.Name), + containerOwnerField(container.Owner)) + if c.upstreamLogger != nil { + c.upstreamLogger.Warn("Force removing owned container", + zap.String("cleanup_path", "force"), + zap.String("container_id", container.ID), + zap.String("container_name", container.Name), + containerOwnerField(container.Owner)) + } + }) + if !res.Verified { + return "", false, res.Err + } + if res.Err != nil { + c.logger.Error("Failed to force remove owned container", + zap.String("server", c.config.Name), + zap.String("cleanup_path", "force"), + zap.String("container_id", res.Container.ID), + containerOwnerField(res.Container.Owner), + zap.Error(res.Err)) + if c.upstreamLogger != nil { + c.upstreamLogger.Error("Failed to force remove owned container", + zap.String("cleanup_path", "force"), + zap.String("container_id", res.Container.ID), + containerOwnerField(res.Container.Owner), + zap.Error(res.Err)) + } + return res.Container.Owner, true, res.Err + } + c.logger.Info("Owned container force removed", + zap.String("server", c.config.Name), + zap.String("cleanup_path", "force"), + zap.String("container_id", res.Container.ID), + containerOwnerField(res.Container.Owner)) + if c.upstreamLogger != nil { + c.upstreamLogger.Info("Owned container force removed", + zap.String("cleanup_path", "force"), + zap.String("container_id", res.Container.ID), + containerOwnerField(res.Container.Owner)) + } + return res.Container.Owner, true, nil +} diff --git a/internal/upstream/core/docker_ownership_test.go b/internal/upstream/core/docker_ownership_test.go new file mode 100644 index 000000000..f731b365d --- /dev/null +++ b/internal/upstream/core/docker_ownership_test.go @@ -0,0 +1,902 @@ +package core + +import ( + "context" + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap" + "go.uber.org/zap/zaptest/observer" + + "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/shellwrap" +) + +// Spec 105 FR-007 (gap FR007-G5, research D9): every Docker cleanup path — +// ensureNoExistingContainers on connect, the disconnect name-pattern fallback +// and the image-name fallback — must touch only containers canonically owned +// by this server: label com.mcpproxy.server= AND name matching +// ^mcpproxy--[a-z0-9]{4}$. On HEAD `docker ps --filter +// name=mcpproxy-a-` is a substring match with no Go-side predicate, so server +// `a` rm -f'd hidden `a-b`'s live container and wrote its id and name into +// a's per-server log; the image-name fallback killed any container sharing +// the image. Foreign containers must be neither removed nor logged. + +// --------------------------------------------------------------------------- +// Fake docker: a sh+awk shim (no re-exec of the race-instrumented test +// binary, which costs ~1s per call). It appends every invocation to the +// invocation log, answers `ps` from a TSV fixture honouring +// --filter name= / id= / label=[=] and --format templates ({{.ID}}, +// {{.Names}}, {{.Image}}, {{.Status}}, {{.CreatedAt}}, {{.Labels}}, +// {{.Label "k"}}), and exits 0 for rm/stop/kill/version unless the verb is +// listed in the fail file (failVerbs). A `ps.tsv.next` fixture replaces the +// fixture after the Nth `ps` answers (swapFixtureAfterPs), so the daemon's +// state can change between a listing and the mutation that follows it. +// --------------------------------------------------------------------------- + +// fakeContainer is one `docker ps` row of the fixture. +type fakeContainer struct { + ID string + Name string + Image string + Status string + Labels map[string]string +} + +// fakeDocker is one installed fake docker: the invocation log, the `ps` +// fixture file the shim reads, and the file whose text makes `docker run` +// fail (printed to stderr, exit 125 — the docker CLI's own status for a +// daemon error) when non-empty. +type fakeDocker struct { + logPath string + psPath string + runErrPath string + failPath string +} + +// failVerbs makes every later invocation of the listed docker verbs (stop, +// kill, rm, ...) exit 1 without output. +func (fd *fakeDocker) failVerbs(t *testing.T, verbs ...string) { + t.Helper() + require.NoError(t, os.WriteFile(fd.failPath, []byte(strings.Join(verbs, " ")+"\n"), 0o600)) +} + +// swapFixtureAfterPs makes the shim answer the first n `ps` invocations from +// the current fixture and every later one from containers — the state of +// the daemon after another client changed it in between. +func (fd *fakeDocker) swapFixtureAfterPs(t *testing.T, n int, containers []fakeContainer) { + t.Helper() + require.NoError(t, os.WriteFile(fd.psPath+".next", fakeFixtureTSV(containers), 0o600)) + require.NoError(t, os.WriteFile(fd.psPath+".swapcount", []byte(fmt.Sprintf("%d\n", n)), 0o600)) +} + +// failRunWith makes every `docker run` print stderr and exit 125. +func (fd *fakeDocker) failRunWith(t *testing.T, stderr string) { + t.Helper() + require.NoError(t, os.WriteFile(fd.runErrPath, []byte(stderr+"\n"), 0o600)) +} + +const fakeDockerShim = `#!/bin/sh +LOG=%s +PS=%s +RUNERR=%s +FAIL=%s +printf '%%s\n' "$*" >> "$LOG" +if [ "$1" = run ] && [ -s "$RUNERR" ]; then cat "$RUNERR" >&2; exit 125; fi +if [ -f "$FAIL" ]; then + read -r failverbs < "$FAIL" + case " $failverbs " in *" $1 "*) exit 1 ;; esac +fi +[ "$1" = ps ] || exit 0 +shift +format='{{.ID}} {{.Names}}' +namefilter='' +idfilter='' +labelkey='' +labelval='' +labelset=0 +while [ $# -gt 0 ]; do + case "$1" in + --format) format="$2"; shift 2 ;; + --filter|-f) + case "$2" in + name=*) namefilter="${2#name=}" ;; + id=*) idfilter="${2#id=}" ;; + label=*) + l="${2#label=}" + labelkey="${l%%%%=*}" + case "$l" in *=*) labelval="${l#*=}"; labelset=1 ;; esac + ;; + esac + shift 2 ;; + *) shift ;; + esac +done +if [ -n "$MCPPROXY_FAKE_DOCKER_IGNORE_FILTERS" ]; then namefilter=''; idfilter=''; labelkey=''; fi +awk -F'\t' -v fmt="$format" -v nf="$namefilter" -v idf="$idfilter" -v lk="$labelkey" -v lv="$labelval" -v ls="$labelset" ' +function repl(s, lit, val, i, out) { + out = "" + while ((i = index(s, lit)) > 0) { out = out substr(s, 1, i - 1) val; s = substr(s, i + length(lit)) } + return out s +} +{ + if (nf != "" && $2 !~ nf) next + if (idf != "" && index(idf, $1) != 1 && index($1, idf) != 1) next + delete labels + n = split($5, pairs, ",") + for (i = 1; i <= n; i++) { eq = index(pairs[i], "="); if (eq > 0) labels[substr(pairs[i], 1, eq - 1)] = substr(pairs[i], eq + 1) } + if (lk != "") { if (!(lk in labels)) next; if (ls && labels[lk] != lv) next } + out = fmt + out = repl(out, "{{.ID}}", $1) + out = repl(out, "{{.Names}}", $2) + out = repl(out, "{{.Image}}", $3) + out = repl(out, "{{.Status}}", $4) + out = repl(out, "{{.CreatedAt}}", "2026-09-16 00:00:00 +0000 UTC") + out = repl(out, "{{.Labels}}", $5) + while (match(out, /\{\{\.Label "[^"]*"\}\}/)) { + key = substr(out, RSTART + 10, RLENGTH - 13) + out = substr(out, 1, RSTART - 1) labels[key] substr(out, RSTART + RLENGTH) + } + print out +}' "$PS" +if [ -f "$PS.next" ]; then + n=1 + [ -f "$PS.swapcount" ] && read -r n < "$PS.swapcount" + n=$((n - 1)) + if [ "$n" -le 0 ]; then mv "$PS.next" "$PS"; rm -f "$PS.swapcount"; else printf '%%s\n' "$n" > "$PS.swapcount"; fi +fi +` + +// installFakeDocker writes the shim, points the REAL resolver at it +// (SetWellKnownDockerPathsForTest + ResetDockerPathCacheForTest, the seams +// gap-map §7 names) and empties PATH so nothing else can resolve. +func installFakeDocker(t *testing.T, containers []fakeContainer) *fakeDocker { + t.Helper() + if runtime.GOOS == osWindows { + t.Skip("unix shell shim") + } + dir := t.TempDir() + fd := &fakeDocker{ + logPath: filepath.Join(dir, "invocations.log"), + psPath: filepath.Join(dir, "ps.tsv"), + runErrPath: filepath.Join(dir, "run.stderr"), + failPath: filepath.Join(dir, "fail.verbs"), + } + require.NoError(t, os.WriteFile(fd.psPath, fakeFixtureTSV(containers), 0o600)) + + shim := filepath.Join(dir, "docker") + script := fmt.Sprintf(fakeDockerShim, dockerShellQuote(fd.logPath), dockerShellQuote(fd.psPath), dockerShellQuote(fd.runErrPath), dockerShellQuote(fd.failPath)) + require.NoError(t, os.WriteFile(shim, []byte(script), 0o755)) + + // PATH must expose sh and awk (the shim needs them) but never a real + // docker: /usr/bin holds one on Ubuntu runners, and the resolver's PATH + // lookup would win over the well-known seam below. Build a PATH dir that + // links only the tools the shim uses. + toolDir := filepath.Join(dir, "path") + require.NoError(t, os.Mkdir(toolDir, 0o755)) + for _, tool := range []string{"sh", "awk", "printf", "cat", "mv", "rm"} { + if real, err := exec.LookPath(tool); err == nil { + require.NoError(t, os.Symlink(real, filepath.Join(toolDir, tool))) + } + } + t.Setenv("PATH", toolDir) + t.Setenv("SHELL", "/nonexistent/shell-must-not-be-invoked") + // On Linux the spawn keeps the login-shell wrap unless the daemon env is + // already in the process env (dockerDaemonEnvGuaranteed); a runner + // without DOCKER_HOST (the Landlock job) would then exec the poisoned + // SHELL above and fail at start instead of running the shim. Pin the + // direct-exec path so every job exercises the same lifecycle sites. + t.Setenv("DOCKER_HOST", "unix:///nonexistent/mcpproxy-fake-docker.sock") + + useRealDockerResolver(t) + restore := shellwrap.SetWellKnownDockerPathsForTest(func() []string { return []string{shim} }) + t.Cleanup(restore) + return fd +} + +// fakeFixtureTSV renders the `ps` fixture the shim reads. +func fakeFixtureTSV(containers []fakeContainer) []byte { + var tsv strings.Builder + for _, c := range containers { + labels := make([]string, 0, len(c.Labels)) + for k, v := range c.Labels { + labels = append(labels, k+"="+v) + } + fmt.Fprintf(&tsv, "%s\t%s\t%s\t%s\t%s\n", c.ID, c.Name, c.Image, c.Status, strings.Join(labels, ",")) + } + return []byte(tsv.String()) +} + +// fakeDockerIgnoreFiltersEnv makes the shim answer `ps` with EVERY fixture +// row regardless of --filter, so a test can prove the Go-side ownership +// predicate drops what the daemon did not. +const fakeDockerIgnoreFiltersEnv = "MCPPROXY_FAKE_DOCKER_IGNORE_FILTERS" + +// dockerShellQuote single-quotes s for the fake-docker shim script. (Not +// named shellQuote: sandbox_linux_test.go declares that in the same package +// under the linux build tag.) +func dockerShellQuote(s string) string { return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'" } + +// invocations returns every docker command line the shim received. +func (fd *fakeDocker) invocations(t *testing.T) []string { + t.Helper() + raw, err := os.ReadFile(fd.logPath) + if os.IsNotExist(err) { + return nil + } + require.NoError(t, err) + return strings.Split(strings.TrimSpace(string(raw)), "\n") +} + +// mutationsOf returns the rm/stop/kill invocations that name id. +func (fd *fakeDocker) mutationsOf(t *testing.T, id string) []string { + t.Helper() + var hits []string + for _, line := range fd.invocations(t) { + fields := strings.Fields(line) + if len(fields) == 0 { + continue + } + switch fields[0] { + case "rm", "stop", "kill": + if fields[len(fields)-1] == id { + hits = append(hits, line) + } + } + } + return hits +} + +// newOwnershipClient builds a client for server name with BOTH its loggers +// observed: c.logger (main.log) and c.upstreamLogger (server-.log, the +// file tail_log serves). +func newOwnershipClient(name string, cfg *config.ServerConfig) (*Client, *observer.ObservedLogs, *observer.ObservedLogs) { + mainCore, mainLogs := observer.New(zap.DebugLevel) + upCore, upLogs := observer.New(zap.DebugLevel) + if cfg == nil { + cfg = &config.ServerConfig{Command: "python", Args: []string{"-m", "mcp_server"}} + } + cfg.Name = name + c := &Client{ + config: cfg, + logger: zap.New(mainCore), + upstreamLogger: zap.New(upCore).With(zap.String("server", name)), + isolationManager: NewIsolationManager(config.DefaultDockerIsolationConfig()), + } + return c, mainLogs, upLogs +} + +// recordsMentioning returns every observed record whose message or any field +// value contains needle. +func recordsMentioning(logs *observer.ObservedLogs, needle string) []string { + var hits []string + for _, entry := range logs.All() { + if strings.Contains(entry.Message, needle) { + hits = append(hits, entry.Message) + continue + } + for k, v := range entry.ContextMap() { + if strings.Contains(fmt.Sprint(v), needle) { + hits = append(hits, entry.Message+" "+k+"="+fmt.Sprint(v)) + break + } + } + } + return hits +} + +const ( + foreignContainerID = "deadbeef1234" + foreignContainerName = "mcpproxy-a-b-wxyz" + ownContainerID = "cafe00000001" + ownContainerName = "mcpproxy-a-wxyz" + ownerLabel = "com.mcpproxy.server" +) + +// withOwnInstance returns a copy of labels with this test process's own +// com.mcpproxy.instance value merged in. Every fixture representing a +// container this server should canonically own must carry it now that +// ownsContainer requires an instance match too (FR-007 instance-scoping +// fix, codex finding): a fixture that omits it looks like a container from +// no instance at all, which is exactly as foreign as a pre-label container. +func withOwnInstance(labels map[string]string) map[string]string { + out := map[string]string{containerInstanceLabel: getInstanceID()} + for k, v := range labels { + out[k] = v + } + return out +} + +func ownAndForeignFixture() []fakeContainer { + return []fakeContainer{ + {ID: foreignContainerID, Name: foreignContainerName, Image: "mcp/example", Status: "Up 2 minutes", + Labels: map[string]string{"com.mcpproxy.managed": "true", ownerLabel: "a-b"}}, + {ID: ownContainerID, Name: ownContainerName, Image: "mcp/example", Status: "Exited (0) 1 minute ago", + Labels: withOwnInstance(map[string]string{"com.mcpproxy.managed": "true", ownerLabel: "a"})}, + } +} + +// assertForeignUntouched is the shared oracle: the foreign container is +// never rm/stop/kill'd and never named — by id or by name — in either of a's +// loggers (main.log AND the per-server log tail_log serves). +func assertForeignUntouched(t *testing.T, fd *fakeDocker, mainLogs, upLogs *observer.ObservedLogs) { + t.Helper() + assert.Empty(t, fd.mutationsOf(t, foreignContainerID), "foreign container %s was mutated", foreignContainerID) + for _, needle := range []string{foreignContainerID, foreignContainerName} { + assert.Empty(t, recordsMentioning(upLogs, needle), "foreign %q written into a's per-server log", needle) + assert.Empty(t, recordsMentioning(mainLogs, needle), "foreign %q written into main log under server=a", needle) + } +} + +// FR007-G5 (connect path): ensureNoExistingContainers for server `a` with +// hidden `a-b`'s live container and a's own stale container present. +func TestDockerCleanup_MatchesOnlyCanonicalOwner_Connect(t *testing.T) { + fd := installFakeDocker(t, ownAndForeignFixture()) + c, mainLogs, upLogs := newOwnershipClient("a", nil) + + require.NoError(t, c.ensureNoExistingContainers(context.Background())) + + assertForeignUntouched(t, fd, mainLogs, upLogs) + assert.NotEmpty(t, fd.mutationsOf(t, ownContainerID), "a's own stale container must still be removed; invocations:\n%s", + strings.Join(fd.invocations(t), "\n")) + assert.NotEmpty(t, recordsMentioning(upLogs, ownContainerID), "removing a's own container is still recorded in a's log") +} + +// FR007-G5 (disconnect name-pattern fallback): no known container id or +// name, so the client falls back to pattern cleanup; the foreign container +// matches the name prefix but not the canonical-owner predicate. +func TestDockerCleanup_MatchesOnlyCanonicalOwner_DisconnectNamePattern(t *testing.T) { + fd := installFakeDocker(t, ownAndForeignFixture()) + c, mainLogs, upLogs := newOwnershipClient("a", &config.ServerConfig{ + Command: "docker", Args: []string{"run", "-i", "--rm", "mcp/example"}, + }) + + c.killDockerContainerByCommandWithContext(context.Background()) + + assertForeignUntouched(t, fd, mainLogs, upLogs) + assert.NotEmpty(t, fd.mutationsOf(t, ownContainerID), "a's own container must still be stopped; invocations:\n%s", + strings.Join(fd.invocations(t), "\n")) +} + +// FR007-G5 (image-name fallback, D9): no owned container at all, empty known +// container id, and two foreign containers on the SAME image — one whose +// name matches the prefix, one that does not. The name-pattern step finds no +// owned container and the image-name fallback must touch nothing. +func TestDockerCleanup_ImageNameFallback_TouchesNothingForeign(t *testing.T) { + const unrelatedID = "feedface0002" + fd := installFakeDocker(t, []fakeContainer{ + {ID: foreignContainerID, Name: foreignContainerName, Image: "mcp/example", Status: "Up 2 minutes", + Labels: map[string]string{ownerLabel: "a-b"}}, + {ID: unrelatedID, Name: "unrelated-tool", Image: "mcp/example", Status: "Up 5 minutes", + Labels: map[string]string{}}, + }) + c, mainLogs, upLogs := newOwnershipClient("a", &config.ServerConfig{ + Command: "docker", Args: []string{"run", "-i", "--rm", "mcp/example"}, + }) + + c.killDockerContainerByCommandWithContext(context.Background()) + + assertForeignUntouched(t, fd, mainLogs, upLogs) + assert.Empty(t, fd.mutationsOf(t, unrelatedID), "a container merely sharing the image was mutated") + assert.Empty(t, recordsMentioning(upLogs, unrelatedID), "a container merely sharing the image was written into a's log") + assert.Empty(t, recordsMentioning(mainLogs, unrelatedID)) + for _, line := range fd.invocations(t) { + f := strings.Fields(line) + if len(f) > 0 { + assert.NotContains(t, []string{"rm", "stop", "kill"}, f[0], "no container may be mutated: %q", line) + } + } +} + +// FR007-G5 unit matcher table, driven through ensureNoExistingContainers so +// it compiles against HEAD: ownership = label com.mcpproxy.server == raw +// server name AND name =~ ^mcpproxy--[a-z0-9]{4}$. Rows cover +// `a` vs `a-b` vs `a/b` vs `A` (the label is the only signal that separates +// a/b from a-b — both sanitise to mcpproxy-a-b-*), pre-label containers and +// the regex guard. +func TestDockerCleanup_OwnershipMatcherTable(t *testing.T) { + cases := []struct { + name string + server string + cname string + labels map[string]string + owned bool + }{ + {"own label and canonical name", "a", "mcpproxy-a-wxyz", withOwnInstance(map[string]string{ownerLabel: "a"}), true}, + {"a-b container, server a", "a", "mcpproxy-a-b-wxyz", map[string]string{ownerLabel: "a-b"}, false}, + {"a/b container, server a", "a", "mcpproxy-a-b-wxyz", map[string]string{ownerLabel: "a/b"}, false}, + {"case-different label", "a", "mcpproxy-a-wxyz", map[string]string{ownerLabel: "A"}, false}, + {"case-different name and label", "a", "mcpproxy-A-wxyz", map[string]string{ownerLabel: "A"}, false}, + {"pre-label container", "a", "mcpproxy-a-wxyz", map[string]string{}, false}, + {"label mismatch, canonical name", "a", "mcpproxy-a-wxyz", map[string]string{ownerLabel: "a-b"}, false}, + {"own label, name with extra segment", "a", "mcpproxy-a-wxyz-extra", map[string]string{ownerLabel: "a"}, false}, + {"own label, uppercase suffix", "a", "mcpproxy-a-WXYZ", map[string]string{ownerLabel: "a"}, false}, + {"own label, short suffix", "a", "mcpproxy-a-wxy", map[string]string{ownerLabel: "a"}, false}, + {"server a/b owns its container", "a/b", "mcpproxy-a-b-wxyz", withOwnInstance(map[string]string{ownerLabel: "a/b"}), true}, + {"server a/b vs a-b's container", "a/b", "mcpproxy-a-b-wxyz", map[string]string{ownerLabel: "a-b"}, false}, + {"server a-b owns its container", "a-b", "mcpproxy-a-b-wxyz", withOwnInstance(map[string]string{ownerLabel: "a-b"}), true}, + {"server a-b vs a/b's container", "a-b", "mcpproxy-a-b-wxyz", map[string]string{ownerLabel: "a/b"}, false}, + {"server A vs a's container", "A", "mcpproxy-a-wxyz", map[string]string{ownerLabel: "a"}, false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + const id = "0123456789ab" + fd := installFakeDocker(t, []fakeContainer{{ID: id, Name: tc.cname, Image: "img", Status: "Up", Labels: tc.labels}}) + c, mainLogs, upLogs := newOwnershipClient(tc.server, nil) + + require.NoError(t, c.ensureNoExistingContainers(context.Background())) + + mutated := fd.mutationsOf(t, id) + if tc.owned { + assert.NotEmpty(t, mutated, "owned container must be removed; invocations:\n%s", strings.Join(fd.invocations(t), "\n")) + } else { + assert.Empty(t, mutated, "foreign container removed by server %q", tc.server) + assert.Empty(t, recordsMentioning(upLogs, id), "foreign container id written into %q's per-server log", tc.server) + assert.Empty(t, recordsMentioning(upLogs, tc.cname), "foreign container name written into %q's per-server log", tc.server) + assert.Empty(t, recordsMentioning(mainLogs, id), "foreign container id logged under server=%q", tc.server) + } + }) + } +} + +// Critique round 1, finding C1.5: the pre-start sweep's count record is a +// container subject (internal/logs D8 rule 3 treats `container_count` like +// `container_id`), so the record written to the per-server log must carry +// `container_owner` == this server or the attributed reader withholds it +// from the server's own scoped agent. +func TestDockerCleanup_CountRecordCarriesContainerOwner(t *testing.T) { + installFakeDocker(t, ownAndForeignFixture()) + c, _, upLogs := newOwnershipClient("a", nil) + + require.NoError(t, c.ensureNoExistingContainers(context.Background())) + + counts := upLogs.FilterMessage("Cleaning up existing containers before creating new one").All() + require.Len(t, counts, 1) + fields := counts[0].ContextMap() + assert.EqualValues(t, 1, fields["container_count"], "the count is of a's own containers only") + assert.Equal(t, "a", fields["container_owner"], "count record must carry container_owner so a's scoped reader can attribute it") + + // Codex round 3, docker finding 3: the value is the label Docker + // reported for the counted rows (D9: never the requesting name). The + // predicate makes the two equal byte-for-byte, so this pins provenance + // by construction: the count record's owner must be the readback value + // the row records carry. + owned, err := c.listOwnedContainers(context.Background(), true) + require.NoError(t, err) + require.NotEmpty(t, owned) + assert.Equal(t, owned[0].Owner, fields["container_owner"], "count record owner must be the label read back from Docker") +} + +// Critique round 1, finding C2.3: ownsContainer is the Go-side half of the +// D9 belt-and-braces (docker filters server-side, Go re-checks). The table +// above drives it through the shim, which honours the same filters, so a +// predicate that returned true for everything still passed. This is the +// direct table, plus a filter-blind shim mode below. +func TestOwnsContainer_Predicate(t *testing.T) { + own := getInstanceID() + cases := []struct { + name string + server string + cname string + label string + instance string + owned bool + }{ + {"own label and canonical name", "a", "mcpproxy-a-wxyz", "a", own, true}, + {"docker-style leading slash is not canonical", "a", "/mcpproxy-a-wxyz", "a", own, false}, + {"a-b container, server a", "a", "mcpproxy-a-b-wxyz", "a-b", own, false}, + {"a/b container, server a", "a", "mcpproxy-a-b-wxyz", "a/b", own, false}, + {"case-different label", "a", "mcpproxy-a-wxyz", "A", own, false}, + {"pre-label container", "a", "mcpproxy-a-wxyz", "", own, false}, + {"label mismatch, canonical name", "a", "mcpproxy-a-wxyz", "a-b", own, false}, + {"own label, name with extra segment", "a", "mcpproxy-a-wxyz-extra", "a", own, false}, + {"own label, uppercase suffix", "a", "mcpproxy-a-WXYZ", "a", own, false}, + {"own label, short suffix", "a", "mcpproxy-a-wxy", "a", own, false}, + {"own label, long suffix", "a", "mcpproxy-a-wxyz1", "a", own, false}, + {"own label, wrong prefix", "a", "other-a-wxyz", "a", own, false}, + {"server a/b owns its container", "a/b", "mcpproxy-a-b-wxyz", "a/b", own, true}, + {"server a/b vs a-b's container", "a/b", "mcpproxy-a-b-wxyz", "a-b", own, false}, + {"server a-b owns its container", "a-b", "mcpproxy-a-b-wxyz", "a-b", own, true}, + {"server a-b vs a/b's container", "a-b", "mcpproxy-a-b-wxyz", "a/b", own, false}, + {"server A vs a's container", "A", "mcpproxy-a-wxyz", "a", own, false}, + // The sanitiser keeps '.', so a.b names mcpproxy-a.b-*; QuoteMeta keeps + // the dot literal in the pattern rather than a wildcard. + {"regex metacharacters in the name are literal", "a.b", "mcpproxy-a.b-wxyz", "a.b", own, true}, + {"regex metacharacters do not widen the match", "a.b", "mcpproxy-aXb-wxyz", "a.b", own, false}, + // FR-007 instance-scoping (codex finding, PR E): the label AND name + // can both match exactly and it must still be rejected when the + // container belongs to a DIFFERENT (or no) mcpproxy instance — two + // separate processes can configure a server with the same raw name. + {"own label and canonical name, no instance label (pre-#1300 or foreign)", "a", "mcpproxy-a-wxyz", "a", "", false}, + {"own label and canonical name, different instance", "a", "mcpproxy-a-wxyz", "a", "some-other-instance-id", false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + assert.Equal(t, tc.owned, ownsContainer(tc.server, tc.cname, tc.label, tc.instance)) + }) + } +} + +// Critique round 1, finding C2.3: with the shim ignoring every --filter (a +// daemon that returned rows the filters should have dropped), listOwnedContainers +// must drop them itself; the pre-start sweep then still removes only a's own +// container and never names the foreign one. +func TestDockerCleanup_GoPredicateDropsRowsTheDaemonDidNotFilter(t *testing.T) { + fd := installFakeDocker(t, ownAndForeignFixture()) + t.Setenv(fakeDockerIgnoreFiltersEnv, "1") + c, mainLogs, upLogs := newOwnershipClient("a", nil) + + owned, err := c.listOwnedContainers(context.Background(), true) + require.NoError(t, err) + require.Len(t, owned, 1, "only a's own container survives the Go-side predicate; got %+v", owned) + assert.Equal(t, ownContainerID, owned[0].ID) + + require.NoError(t, c.ensureNoExistingContainers(context.Background())) + assertForeignUntouched(t, fd, mainLogs, upLogs) + assert.NotEmpty(t, fd.mutationsOf(t, ownContainerID)) +} + +// shortenCidfilePoll makes readContainerIDWithContext give up on the cidfile +// almost immediately (the production wait is 10 s) so the name-recovery +// fallback is reachable in a unit test. +func shortenCidfilePoll(t *testing.T) { + t.Helper() + attempts, interval := cidfileReadAttempts, cidfileReadInterval + cidfileReadAttempts, cidfileReadInterval = 2, time.Millisecond + t.Cleanup(func() { cidfileReadAttempts, cidfileReadInterval = attempts, interval }) +} + +// Codex round 1 (PR E), finding 2: the cidfile path. A user-configured +// direct `docker run --name custom image` upstream gets --cidfile injected +// but carries neither the com.mcpproxy.server label nor a canonical name, so +// it fails ownsContainer on both halves. The pre-fix code recorded its id as +// owned, wrote it into a's per-server log with a fabricated +// container_owner=a, and stopped/killed it on disconnect. Under D9 a +// user-`--name` container is not ours: the id captured from the cidfile +// must be inspected, and a container that fails ownership is left alone and +// never named in a's per-server log. The fixture holds the FULL id, as +// `docker ps --no-trunc` reports it (codex round 6: every read is matched +// back by the full id exactly). +func TestDockerCleanup_CidfileContainerMustPassOwnership(t *testing.T) { + const customID = "c0ffee000001" + const customFullID = customID + "0000000000000000000000000000000000000000000000000000" + cases := []struct { + name string + row fakeContainer + owned bool + }{ + {"user --name custom, no label", fakeContainer{ID: customFullID, Name: "custom", Image: "mcp/example", Status: "Up 1 second", Labels: map[string]string{}}, false}, + {"own label, user --name custom via extra_args", fakeContainer{ID: customFullID, Name: "custom", Image: "mcp/example", Status: "Up 1 second", Labels: map[string]string{ownerLabel: "a"}}, false}, + {"foreign label, canonical-looking name", fakeContainer{ID: customFullID, Name: "mcpproxy-a-wxyz", Image: "mcp/example", Status: "Up 1 second", Labels: map[string]string{ownerLabel: "a-b"}}, false}, + {"own label and canonical name", fakeContainer{ID: customFullID, Name: "mcpproxy-a-wxyz", Image: "mcp/example", Status: "Up 1 second", Labels: withOwnInstance(map[string]string{ownerLabel: "a"})}, true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + fd := installFakeDocker(t, []fakeContainer{tc.row}) + c, _, upLogs := newOwnershipClient("a", &config.ServerConfig{ + Command: "docker", Args: []string{"run", "-i", "--rm", "--name", "custom", "mcp/example"}, + }) + + cidFile := filepath.Join(t.TempDir(), "cid") + require.NoError(t, os.WriteFile(cidFile, []byte(customFullID+"\n"), 0o600)) + c.readContainerIDWithContext(context.Background(), cidFile) + + c.mu.Lock() + c.killDockerContainerWithContext(context.Background()) + c.mu.Unlock() + + mutated := append(fd.mutationsOf(t, customFullID), fd.mutationsOf(t, customID)...) + if tc.owned { + assert.NotEmpty(t, mutated, "a's own container must still be stopped; invocations:\n%s", strings.Join(fd.invocations(t), "\n")) + for _, entry := range upLogs.All() { + if owner, ok := entry.ContextMap()["container_owner"]; ok { + assert.Equal(t, "a", owner, "container_owner must be the label read back") + } + } + return + } + assert.Empty(t, mutated, "container that fails ownership was stopped/killed: %v", mutated) + for _, needle := range []string{customFullID, customID, tc.row.Name} { + assert.Empty(t, recordsMentioning(upLogs, needle), "unowned %q written into a's per-server log", needle) + } + for _, entry := range upLogs.All() { + _, has := entry.ContextMap()["container_owner"] + assert.False(t, has, "container_owner fabricated on record %q", entry.Message) + } + }) + } +} + +// Codex round 8 (PR E), finding 1: a cidfile row that fails ownership, or +// whose ownership Docker read itself fails, must not name any id in EITHER +// logger. TestDockerCleanup_CidfileContainerMustPassOwnership already covers +// upLogs (the per-server log); trackCidfileContainer's err!=nil and !ok +// branches still logged shortContainerID(containerID) into mainLogs (the +// admin-facing main.log), unlike mutateOwnedContainer's refusal branches +// which name only the server, cleanup_path and operation. +func TestDockerCleanup_CidfileRefusal_MainLogRecordsNoID(t *testing.T) { + const customID = "c0ffee000002" + const customFullID = customID + "0000000000000000000000000000000000000000000000000000" + + t.Run("not owned", func(t *testing.T) { + installFakeDocker(t, []fakeContainer{ + {ID: customFullID, Name: "custom", Image: "mcp/example", Status: "Up 1 second", Labels: map[string]string{}}, + }) + c, mainLogs, _ := newOwnershipClient("a", &config.ServerConfig{ + Command: "docker", Args: []string{"run", "-i", "--rm", "--name", "custom", "mcp/example"}, + }) + cidFile := filepath.Join(t.TempDir(), "cid") + require.NoError(t, os.WriteFile(cidFile, []byte(customFullID+"\n"), 0o600)) + c.readContainerIDWithContext(context.Background(), cidFile) + + require.NotEmpty(t, mainLogs.All(), "expected a refusal record in the main log") + for _, entry := range mainLogs.All() { + _, has := entry.ContextMap()["container_id"] + assert.False(t, has, "unowned container's id recorded in main log: %q", entry.Message) + } + }) + + t.Run("docker read failure", func(t *testing.T) { + fd := installFakeDocker(t, []fakeContainer{ + {ID: customFullID, Name: "mcpproxy-a-wxyz", Image: "mcp/example", Status: "Up 1 second", Labels: map[string]string{ownerLabel: "a"}}, + }) + fd.failVerbs(t, "ps") + c, mainLogs, _ := newOwnershipClient("a", &config.ServerConfig{ + Command: "docker", Args: []string{"run", "-i", "--rm", "--name", "custom", "mcp/example"}, + }) + cidFile := filepath.Join(t.TempDir(), "cid") + require.NoError(t, os.WriteFile(cidFile, []byte(customFullID+"\n"), 0o600)) + c.readContainerIDWithContext(context.Background(), cidFile) + + require.NotEmpty(t, mainLogs.All(), "expected a refusal record in the main log") + for _, entry := range mainLogs.All() { + _, has := entry.ContextMap()["container_id"] + assert.False(t, has, "docker-read-failure recorded a container id in main log: %q", entry.Message) + } + }) +} + +// Codex round 1 (PR E), finding 3: the exact-name paths (cidfile recovery +// by name and killDockerContainerByNameWithContext) filtered by label and +// the tracked name only, never applied ownsContainer, and wrote +// container_owner from the REQUESTED server rather than the label read +// back. A foreign `--label com.mcpproxy.server=a --name custom` container +// whose name is the tracked one must be neither stopped nor named in a's +// log; an owned canonical container on the same paths still is, with +// container_owner equal to its label. +func TestDockerCleanup_ExactNamePathsApplyOwnership(t *testing.T) { + const foreignID = "f0e1d2c3b4a5" + cases := []struct { + name string + tracked string + row fakeContainer + owned bool + }{ + {"foreign label=a --name custom", "custom", + fakeContainer{ID: foreignID, Name: "custom", Image: "mcp/example", Status: "Up 1 second", Labels: map[string]string{ownerLabel: "a"}}, false}, + {"foreign label=a-b canonical-looking name", ownContainerName, + fakeContainer{ID: foreignID, Name: ownContainerName, Image: "mcp/example", Status: "Up 1 second", Labels: map[string]string{ownerLabel: "a-b"}}, false}, + {"own label and canonical name", ownContainerName, + fakeContainer{ID: ownContainerID, Name: ownContainerName, Image: "mcp/example", Status: "Up 1 second", Labels: withOwnInstance(map[string]string{ownerLabel: "a"})}, true}, + } + for _, tc := range cases { + t.Run("recovery/"+tc.name, func(t *testing.T) { + fd := installFakeDocker(t, []fakeContainer{tc.row}) + c, _, upLogs := newOwnershipClient("a", nil) + c.containerName = tc.tracked + + // A cidfile that never appears: the read times out and recovers by name. + shortenCidfilePoll(t) + c.readContainerIDWithContext(context.Background(), filepath.Join(t.TempDir(), "never-written")) + + if tc.owned { + assert.Equal(t, tc.row.ID, c.containerID, "own container must be recovered by name") + for _, entry := range upLogs.All() { + if owner, ok := entry.ContextMap()["container_owner"]; ok { + assert.Equal(t, "a", owner) + } + } + return + } + assert.Empty(t, c.containerID, "foreign container recorded as owned via name recovery") + assert.Empty(t, recordsMentioning(upLogs, tc.row.ID), "foreign id written into a's per-server log") + assert.Empty(t, fd.mutationsOf(t, tc.row.ID)) + }) + t.Run("kill_by_name/"+tc.name, func(t *testing.T) { + fd := installFakeDocker(t, []fakeContainer{tc.row}) + c, mainLogs, upLogs := newOwnershipClient("a", nil) + + ok := c.killDockerContainerByNameWithContext(context.Background(), tc.tracked) + + if tc.owned { + assert.True(t, ok) + assert.NotEmpty(t, fd.mutationsOf(t, tc.row.ID), "own container must be stopped; invocations:\n%s", strings.Join(fd.invocations(t), "\n")) + for _, entry := range upLogs.All() { + if owner, has := entry.ContextMap()["container_owner"]; has { + assert.Equal(t, "a", owner, "container_owner must be the label read back") + } + } + return + } + assert.False(t, ok) + assert.Empty(t, fd.mutationsOf(t, tc.row.ID), "foreign container stopped/killed by exact name") + assert.Empty(t, recordsMentioning(upLogs, tc.row.ID), "foreign id written into a's per-server log") + assert.Empty(t, recordsMentioning(mainLogs, tc.row.ID), "foreign id written into main log under server=a") + for _, entry := range upLogs.All() { + _, has := entry.ContextMap()["container_owner"] + assert.False(t, has, "container_owner fabricated on record %q", entry.Message) + } + }) + } +} + +// Codex round 3, docker finding 1: the manager's emergency path (a client +// whose Disconnect hung) ran `docker rm -f ` with no ownership +// check, while every other stop/kill/rm path re-establishes ownership at the +// moment of the mutation. The tracked id is re-inspected here: a container +// that is no longer canonically a's — renamed to a-b's shape, or a foreign +// container under that id — is left alone and never named in a's log; a's +// own container is removed with container_owner from the label read back. +func TestForceRemoveTrackedContainerIfOwned_AppliesOwnership(t *testing.T) { + t.Run("tracked id now foreign", func(t *testing.T) { + fd := installFakeDocker(t, ownAndForeignFixture()) + c, mainLogs, upLogs := newOwnershipClient("a", nil) + + owner, owned, err := c.ForceRemoveTrackedContainerIfOwned(context.Background(), foreignContainerID) + require.NoError(t, err) + assert.False(t, owned, "a foreign container under the tracked id must not be admitted") + assert.Empty(t, owner, "no owner evidence for a container that was not admitted") + assert.Empty(t, fd.mutationsOf(t, foreignContainerID), "foreign container %s was mutated", foreignContainerID) + for _, line := range fd.invocations(t) { + assert.False(t, strings.HasPrefix(line, "rm "), "rm invoked without ownership: %s", line) + } + // The per-server log (tail_log) never names it; main.log keeps the + // short TRACKED id in the "not owned" diagnostic, as the disconnect + // path does — that id was a's own knowledge (the fixture ids are + // already short) — but never the name read back. + for _, needle := range []string{foreignContainerID, foreignContainerName, shortContainerID(foreignContainerID)} { + assert.Empty(t, recordsMentioning(upLogs, needle), "foreign %q written into a's per-server log", needle) + } + assert.Empty(t, recordsMentioning(mainLogs, foreignContainerName), "foreign name read back written into main log under server=a") + }) + + t.Run("tracked id owned", func(t *testing.T) { + fd := installFakeDocker(t, ownAndForeignFixture()) + c, _, upLogs := newOwnershipClient("a", nil) + + owner, owned, err := c.ForceRemoveTrackedContainerIfOwned(context.Background(), ownContainerID) + require.NoError(t, err) + assert.True(t, owned) + assert.Equal(t, "a", owner, "the owner handed back is the label read back at the mutation") + assert.Equal(t, []string{"rm -f " + ownContainerID}, fd.mutationsOf(t, ownContainerID)) + removed := upLogs.FilterMessage("Owned container force removed").All() + require.Len(t, removed, 1) + assert.Equal(t, "a", removed[0].ContextMap()["container_owner"], "owner is the label read back") + }) + + t.Run("no tracked id", func(t *testing.T) { + fd := installFakeDocker(t, ownAndForeignFixture()) + c, _, _ := newOwnershipClient("a", nil) + owner, owned, err := c.ForceRemoveTrackedContainerIfOwned(context.Background(), "") + require.NoError(t, err) + assert.False(t, owned) + assert.Empty(t, owner) + assert.Empty(t, fd.invocations(t), "nothing to remove, docker never invoked") + }) +} + +// ContainerOwnedByAny is the whole-manager sweep predicate (codex round 3, +// docker finding 2): label AND canonical name for the SAME configured server. +func TestContainerOwnedByAny_Predicate(t *testing.T) { + own := getInstanceID() + configured := []string{"a", "a/b"} + cases := []struct { + name string + cname string + label string + instance string + owned bool + }{ + {"a's canonical container", "mcpproxy-a-wxyz", "a", own, true}, + {"a/b's canonical container", "mcpproxy-a-b-wxyz", "a/b", own, true}, + {"a-b's container: a-b not configured", "mcpproxy-a-b-wxyz", "a-b", own, false}, + {"copied managed label, no server label", "postgres", "", own, false}, + {"configured label, non-canonical name", "custom", "a", own, false}, + {"canonical name for a, label of a/b", "mcpproxy-a-wxyz", "a/b", own, false}, + {"canonical name for a, no label", "mcpproxy-a-wxyz", "", own, false}, + // FR-007 instance-scoping: a's canonical container, but from another + // (or no) mcpproxy instance, is foreign even though a is configured. + {"a's canonical container, different instance", "mcpproxy-a-wxyz", "a", "some-other-instance-id", false}, + {"a's canonical container, no instance label", "mcpproxy-a-wxyz", "a", "", false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + assert.Equal(t, tc.owned, ContainerOwnedByAny(configured, tc.cname, tc.label, tc.instance)) + }) + } + assert.False(t, ContainerOwnedByAny(nil, "mcpproxy-a-wxyz", "a", own), "no configured servers, nothing is owned") +} + +// TestContainerMutatorRead_RejectsEmbeddedTabInLabel is codex round 2 (PR E), +// the HIGH finding on the instance-scoping fix: containerRowFormat's +// tab-separated `docker ps` row has no escaping for a label's own value, so +// a container an attacker creates themselves can give its Instance (or +// Owner) label a value containing a literal tab: "\t". A bounded split (the original SplitN(5)) would absorb +// everything after the 4th tab into Status, leaving the Instance field +// read back as EXACTLY the real instance id — smuggling an exact match past +// ownsContainer even though the field, as Docker actually reported it, was +// never that clean value. read() now requires the row split to exactly the +// expected field count; a row with an extra, attacker-controlled tab is +// rejected outright rather than leniently parsed. +func TestContainerMutatorRead_RejectsEmbeddedTabInLabel(t *testing.T) { + if runtime.GOOS == osWindows { + t.Skip("unix shell shim") + } + own := getInstanceID() + const id = "cafe00000001" + // containerRowFormat is ID \t Names \t Owner \t Instance \t Status (4 + // literal tabs, 5 fields). This raw row instead carries an extra tab — + // as if the Instance label's own value were "\tX" — so it splits + // to 6 fields, not 5. + raw := id + "\tmcpproxy-a-wxyz\ta\t" + own + "\tX\tUp 1 second\n" + mut := ContainerMutator{ + Docker: func(ctx context.Context, _ ...string) *exec.Cmd { + return exec.CommandContext(ctx, "printf", "%s", raw) + }, + Owns: func(containerName, ownerLabel, instanceLabel string) bool { + return ownsContainer("a", containerName, ownerLabel, instanceLabel) + }, + } + row, ok, err := mut.Verify(context.Background(), id) + require.NoError(t, err) + assert.False(t, ok, "a row with an extra (attacker-controlled) tab must be rejected, not leniently parsed") + assert.Empty(t, row.Instance, "no partial row is handed back for a rejected read") +} + +// TestContainerMutatorRead_RejectsNewlineSplicedRow is codex round 4 (PR E): +// a label VALUE can contain a literal NEWLINE, not just a tab. Since every +// container's `docker ps --format` output is meant to render as exactly one +// line, an attacker's own container whose Owner label is +// "junk\n\tmcpproxy-a-wxyz\ta\t" splits Docker's +// single rendered row into two: a short, malformed first fragment (missing +// fields — the attacker's own real id/name/truncated-owner) and a second +// fragment that, on its own, looks like a complete, independently +// well-formed row for a container id, name, owner and instance entirely of +// the attacker's choosing. A parser that skips only the malformed fragment +// and keeps scanning would accept the forged second line. Every malformed +// line now poisons the WHOLE read: this proves the read fails closed (not +// found) rather than falling through to the forged fragment. +func TestContainerMutatorRead_RejectsNewlineSplicedRow(t *testing.T) { + if runtime.GOOS == osWindows { + t.Skip("unix shell shim") + } + own := getInstanceID() + const attackerID = "beef00000002" + const forgedID = "cafe00000001" // the id this read() call actually asks about + // Fragment 1 (attackerID's own truncated row, 3 fields — missing + // Instance and Status): "beef00000002\tcustom\tjunk" + // Fragment 2 (fully forged, 5 fields, looks legitimate on its own): + // "cafe00000001\tmcpproxy-a-wxyz\ta\t\tUp 1 second" + raw := attackerID + "\tcustom\tjunk\n" + forgedID + "\tmcpproxy-a-wxyz\ta\t" + own + "\tUp 1 second\n" + mut := ContainerMutator{ + Docker: func(ctx context.Context, _ ...string) *exec.Cmd { + return exec.CommandContext(ctx, "printf", "%s", raw) + }, + Owns: func(containerName, ownerLabel, instanceLabel string) bool { + return ownsContainer("a", containerName, ownerLabel, instanceLabel) + }, + } + row, ok, err := mut.Verify(context.Background(), forgedID) + require.NoError(t, err) + assert.False(t, ok, "a newline-spliced forged row must be rejected, even though it looks well-formed on its own") + assert.Empty(t, row.ID, "no partial row is handed back for a rejected read") +} diff --git a/internal/upstream/core/docker_review_round11_test.go b/internal/upstream/core/docker_review_round11_test.go new file mode 100644 index 000000000..1662826fd --- /dev/null +++ b/internal/upstream/core/docker_review_round11_test.go @@ -0,0 +1,123 @@ +package core + +import ( + "context" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// Codex review round 11 (PR E), finding 1: D8 rule 3 treats a container +// COUNT as container-subject evidence, so every record carrying +// container_count must also carry container_owner (the label read back from +// the listed rows) — or, when the count is zero, no container fields at +// all. Three housekeeping sites logged container_count alone: the +// image-name fallback, the name-pattern fallback and the pre-creation +// sweep's main-log record (its upstreamLogger record already paired the two +// per the pre-creation fix at docker.go:423-425). +// +// Each sub-test below drives the site directly against a's own owned +// container and asserts every c.logger (main.log) record carrying +// container_count in this run also carries container_owner="a" — the value +// the fixture's label constrains it to. + +func TestDockerCleanup_ImageFallback_CountCarriesOwner(t *testing.T) { + installFakeDocker(t, []fakeContainer{ + {ID: ownContainerID, Name: ownContainerName, Image: "mcp/example", Status: "Up 2 minutes", + Labels: withOwnInstance(map[string]string{ownerLabel: "a"})}, + }) + c, mainLogs, _ := newOwnershipClient("a", nil) + + c.killDockerContainersByImageWithContext(context.Background(), "mcp/example") + + found := false + for _, entry := range mainLogs.All() { + fields := entry.ContextMap() + count, hasCount := fields["container_count"] + if !hasCount { + continue + } + found = true + owner, hasOwner := fields["container_owner"] + assert.True(t, hasOwner, "record %q carries container_count=%v without container_owner", entry.Message, count) + assert.Equal(t, "a", owner) + } + assert.True(t, found, "expected at least one record carrying container_count") +} + +func TestDockerCleanup_NamePatternFallback_CountCarriesOwner(t *testing.T) { + installFakeDocker(t, []fakeContainer{ + {ID: ownContainerID, Name: ownContainerName, Image: "mcp/example", Status: "Up 2 minutes", + Labels: withOwnInstance(map[string]string{ownerLabel: "a"})}, + }) + c, mainLogs, _ := newOwnershipClient("a", nil) + + c.killDockerContainersByNamePatternWithContext(context.Background()) + + found := false + for _, entry := range mainLogs.All() { + fields := entry.ContextMap() + count, hasCount := fields["container_count"] + if !hasCount { + continue + } + found = true + owner, hasOwner := fields["container_owner"] + assert.True(t, hasOwner, "record %q carries container_count=%v without container_owner", entry.Message, count) + assert.Equal(t, "a", owner) + } + assert.True(t, found, "expected at least one record carrying container_count") +} + +func TestDockerCleanup_PreCreationSweep_MainLogCountCarriesOwner(t *testing.T) { + installFakeDocker(t, []fakeContainer{ + {ID: ownContainerID, Name: ownContainerName, Image: "mcp/example", Status: "Up 2 minutes", + Labels: withOwnInstance(map[string]string{ownerLabel: "a"})}, + }) + c, mainLogs, _ := newOwnershipClient("a", nil) + + require.NoError(t, c.ensureNoExistingContainers(context.Background())) + + found := false + for _, entry := range mainLogs.All() { + fields := entry.ContextMap() + count, hasCount := fields["container_count"] + if !hasCount { + continue + } + found = true + owner, hasOwner := fields["container_owner"] + assert.True(t, hasOwner, "main-log record %q carries container_count=%v without container_owner", entry.Message, count) + assert.Equal(t, "a", owner) + } + assert.True(t, found, "expected at least one main-log record carrying container_count") +} + +// Codex review round 11, finding 2: the terminal cidfile-recovery failure +// (readContainerIDWithContext, no id from the cidfile AND no owned +// container found by the tracked name) must name only the server in +// main.log — c.containerName is a generated name never read back from +// Docker, and under a suffix collision it can currently belong to a +// different, colliding server. This mirrors the round-9 lifecycle fix, +// where an unverified container_name is omitted rather than logged. +func TestDockerCleanup_CidfileRecoveryFailure_NamesServerOnly(t *testing.T) { + // No containers at all: the by-name lookup finds nothing, so recovery + // fails and the terminal error path fires. + installFakeDocker(t, nil) + c, mainLogs, _ := newOwnershipClient("a", nil) + c.containerName = ownContainerName + + shortenCidfilePoll(t) + cidFile := filepath.Join(t.TempDir(), "never-written") + c.readContainerIDWithContext(context.Background(), cidFile) + + errorRecords := mainLogs.FilterMessage("Failed to recover container ID - container will be orphaned on disconnect").All() + require.NotEmpty(t, errorRecords, "expected the terminal recovery-failure record") + for _, entry := range errorRecords { + fields := entry.ContextMap() + _, hasName := fields["container_name"] + assert.False(t, hasName, "recovery-failure record names an unverified container_name: %v", fields) + } +} diff --git a/internal/upstream/core/lifecycle_container_evidence_test.go b/internal/upstream/core/lifecycle_container_evidence_test.go new file mode 100644 index 000000000..ca4a40e2e --- /dev/null +++ b/internal/upstream/core/lifecycle_container_evidence_test.go @@ -0,0 +1,399 @@ +package core + +import ( + "context" + "errors" + "fmt" + "strings" + "testing" + "time" + + mcpclient "github.com/mark3labs/mcp-go/client" + uptransport "github.com/mark3labs/mcp-go/client/transport" + "github.com/mark3labs/mcp-go/mcp" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap" + "go.uber.org/zap/zaptest/observer" + + "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/secret" +) + +// Codex round 9 (PR E), MUST-FIX 1 and 2 (Spec 105 D8): lifecycle housekeeping +// records (connection failure, init failure, disconnect) and +// GetConnectionDiagnostics named a container — or trusted `docker inspect` by +// id alone — without current ownership evidence. containerID is assigned only +// by trackCidfileContainer or the cidfile-timeout name-recovery fallback +// (docker.go), both of which verify ownership via Docker's read-back before +// ever setting it, and always pair it with containerOwner; containerName +// alone is the GENERATED canonical name, set before Docker has confirmed +// anything. The tests below drive the actual lifecycle code paths (not just +// the shared field-selection helper) through the fake docker shim and a +// failing MCP transport, covering both arms at every site the round 8 +// finding named. + +// TestDockerContainerLogFields is the shared field-selection helper every +// lifecycle site (connection.go, connection_stdio.go, connection_lifecycle.go +// x2) now goes through: empty containerID (never verified — trackCidfileContainer +// / the name-recovery fallback never adopted anything, whether or not a +// GENERATED containerName exists) yields no fields at all; a non-empty +// containerID (only ever set alongside containerOwner) yields all three. +func TestDockerContainerLogFields(t *testing.T) { + t.Run("no tracked id names nothing, even with a generated name", func(t *testing.T) { + fields := dockerContainerLogFields("", "mcpproxy-a-wxyz", "") + assert.Nil(t, fields, "an unverified generated name must not be treated as evidence") + }) + + t.Run("no tracked id and no generated name names nothing", func(t *testing.T) { + fields := dockerContainerLogFields("", "", "") + assert.Nil(t, fields) + }) + + t.Run("tracked id carries id, name and the owner read back", func(t *testing.T) { + fields := dockerContainerLogFields(ownContainerID, ownContainerName, "a") + enc := zapcoreEncodeFields(t, fields) + assert.Equal(t, ownContainerID, enc["container_id"]) + assert.Equal(t, ownContainerName, enc["container_name"]) + assert.Equal(t, "a", enc["container_owner"]) + }) +} + +// zapcoreEncodeFields renders zap.Field values into a map the way the +// observer would, so a direct-return test of dockerContainerLogFields can +// assert on field values without duplicating zap's internals. +func zapcoreEncodeFields(t *testing.T, fields []zap.Field) map[string]interface{} { + t.Helper() + core, logs := observer.New(zap.DebugLevel) + zap.New(core).Debug("probe", fields...) + all := logs.All() + require.Len(t, all, 1) + return all[0].ContextMap() +} + +// failingMCPTransport is a minimal transport.Interface whose SendRequest +// always fails — deterministic and instant, no subprocess needed — so +// c.initialize(ctx) fails the same way a real handshake timeout would, +// without the cost/flakiness of spawning one. +type failingMCPTransport struct{ err error } + +func (failingMCPTransport) Start(context.Context) error { return nil } +func (f failingMCPTransport) SendRequest(context.Context, uptransport.JSONRPCRequest) (*uptransport.JSONRPCResponse, error) { + return nil, f.err +} +func (failingMCPTransport) SendNotification(context.Context, mcp.JSONRPCNotification) error { + return nil +} +func (failingMCPTransport) SetNotificationHandler(func(mcp.JSONRPCNotification)) {} +func (failingMCPTransport) Close() error { return nil } +func (failingMCPTransport) GetSessionId() string { return "" } + +// TestInitializeFailure_DockerCleanupLog is connection_lifecycle.go's +// "Direct initialization failed for Docker command" site: initialize() is +// called directly (not via connectStdio), so this is the "cleanup may be +// handled by caller" comment's own scenario. +func TestInitializeFailure_DockerCleanupLog(t *testing.T) { + const msg = "Direct initialization failed for Docker command - cleanup may be handled by caller" + + arms := []struct { + name string + containerID, containerName, containerOwner string + verified bool + }{ + {name: "nothing tracked yet"}, + {name: "only a generated name, never verified", containerName: ownContainerName}, + {name: "tracked and verified", containerID: ownContainerID, containerName: ownContainerName, containerOwner: "a", verified: true}, + } + for _, arm := range arms { + t.Run(arm.name, func(t *testing.T) { + c, mainLogs, _ := newOwnershipClient("a", nil) + c.isDockerCommand = true + c.containerID = arm.containerID + c.containerName = arm.containerName + c.containerOwner = arm.containerOwner + c.client = mcpclient.NewClient(failingMCPTransport{err: errors.New("boom")}) + + require.Error(t, c.initialize(context.Background())) + + records := mainLogs.FilterMessage(msg).All() + require.NotEmpty(t, records, "expected the Docker cleanup log line") + for _, entry := range records { + fields := entry.ContextMap() + if arm.verified { + assert.Equal(t, arm.containerID, fields["container_id"]) + assert.Equal(t, arm.containerName, fields["container_name"]) + assert.Equal(t, arm.containerOwner, fields["container_owner"]) + continue + } + _, hasID := fields["container_id"] + _, hasName := fields["container_name"] + _, hasOwner := fields["container_owner"] + assert.False(t, hasID, "unverified state must not name a container id") + assert.False(t, hasName, "unverified state must not name a container name") + assert.False(t, hasOwner, "unverified state must not name a container owner") + } + }) + } +} + +// TestDisconnectWithContext_DockerCleanupLog is connection_lifecycle.go's +// disconnect-path pair: "Cleaning up Docker container by ID" (containerID +// tracked and verified) and "Cleaning up Docker container by name" +// (containerID empty, only the GENERATED containerName known). +func TestDisconnectWithContext_DockerCleanupLog(t *testing.T) { + t.Run("tracked and verified: names the container with its owner", func(t *testing.T) { + fd := installFakeDocker(t, ownFixtureNamed(ownContainerID, ownContainerName, "a")) + c, mainLogs, _ := newOwnershipClient("a", nil) + c.isDockerCommand = true + c.containerID = ownContainerID + c.containerName = ownContainerName + c.containerOwner = "a" + + require.NoError(t, c.DisconnectWithContext(context.Background())) + + records := mainLogs.FilterMessage("Cleaning up Docker container by ID").All() + require.NotEmpty(t, records) + for _, entry := range records { + fields := entry.ContextMap() + assert.Equal(t, ownContainerID, fields["container_id"]) + assert.Equal(t, "a", fields["container_owner"]) + } + assert.Contains(t, fd.mutationsOf(t, ownContainerID), "stop "+ownContainerID, + "the verified container is still cleaned up") + }) + + t.Run("only a generated name: names the server only", func(t *testing.T) { + fd := installFakeDocker(t, nil) + c, mainLogs, _ := newOwnershipClient("a", nil) + c.isDockerCommand = true + c.containerName = ownContainerName // generated at spawn time; containerID never adopted + + require.NoError(t, c.DisconnectWithContext(context.Background())) + + records := mainLogs.FilterMessage("Cleaning up Docker container by name").All() + require.NotEmpty(t, records) + for _, entry := range records { + fields := entry.ContextMap() + _, hasName := fields["container_name"] + _, hasOwner := fields["container_owner"] + assert.False(t, hasName, "an unverified generated name must not be logged as evidence") + assert.False(t, hasOwner) + } + // killDockerContainerByNameWithContext still re-verifies before acting; + // nothing in the (empty) fixture is owned, so nothing is mutated. + assert.Empty(t, fd.invocationsMatching(t, "stop"), "no container was owned, so none should be stopped") + }) +} + +// invocationsMatching returns every invocation line starting with verb. +func (fd *fakeDocker) invocationsMatching(t *testing.T, verb string) []string { + t.Helper() + var out []string + for _, line := range fd.invocations(t) { + if line == verb || len(line) > len(verb) && line[:len(verb)+1] == verb+" " { + out = append(out, line) + } + } + return out +} + +// TestConnectStdioDirectDockerRun_ContainerEvidence drives the REAL Connect +// -> connectStdio -> initialize chain for a direct `docker run` upstream +// (config.Command == "docker"), through the fake docker shim, whose `run` +// verb logs the invocation and exits immediately — no cidfile is ever +// written, so this is the connectStdio/Connect equivalent of "nothing +// tracked yet" for the async cidfile path — while independently proving the +// wiring at connection.go's and connection_stdio.go's Docker cleanup log +// lines fires and, when the client already carries a tracked/verified +// container from an earlier attempt, carries container_owner too. +func TestConnectStdioDirectDockerRun_ContainerEvidence(t *testing.T) { + sites := []string{ + "Connection failed for Docker command - cleaning up container", + "Initialization failed for Docker command - cleaning up container", + "Direct initialization failed for Docker command - cleanup may be handled by caller", + } + + arms := []struct { + name string + preset func(c *Client) + verified bool + }{ + {name: "nothing tracked yet (real cidfile-less run)", preset: func(*Client) {}}, + { + name: "already tracked and verified from an earlier attempt", + preset: func(c *Client) { + c.containerID = ownContainerID + c.containerName = ownContainerName + c.containerOwner = "a" + }, + verified: true, + }, + } + + for _, arm := range arms { + t.Run(arm.name, func(t *testing.T) { + installFakeDocker(t, nil) + shortenCidfilePoll(t) + + cfg := &config.ServerConfig{ + Name: "a", + Command: "docker", + Args: []string{"run", "-i", "--rm", "mcp/example"}, + Enabled: true, + } + mainCore, mainLogs := observer.New(zap.DebugLevel) + c, err := NewClient("a", cfg, zap.New(mainCore), nil, nil, nil, secret.NewResolver()) + require.NoError(t, err) + arm.preset(c) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + connectErr := c.Connect(ctx) + require.Error(t, connectErr, "the fake docker run exits immediately, so the handshake must fail") + + for _, msg := range sites { + records := mainLogs.FilterMessage(msg).All() + require.NotEmptyf(t, records, "expected %q to be logged; all records:\n%s", msg, dumpRecords(mainLogs)) + for _, entry := range records { + fields := entry.ContextMap() + if arm.verified { + assert.Equal(t, ownContainerID, fields["container_id"], "%q", msg) + assert.Equal(t, ownContainerName, fields["container_name"], "%q", msg) + assert.Equal(t, "a", fields["container_owner"], "%q", msg) + continue + } + _, hasID := fields["container_id"] + _, hasOwner := fields["container_owner"] + assert.False(t, hasID, "%q must not name an unverified container id", msg) + assert.False(t, hasOwner, "%q must not name an unverified container owner", msg) + } + } + }) + } +} + +// dumpRecords renders every observed record's message and fields, for a +// require.NotEmptyf failure message. +func dumpRecords(logs *observer.ObservedLogs) string { + var out string + for _, e := range logs.All() { + out += fmt.Sprintf("%s %v\n", e.Message, e.ContextMap()) + } + return out +} + +// TestGetConnectionDiagnostics_ReverifiesOwnershipBeforePublishing is codex +// round 9 finding 2: `docker inspect ` resolves purely by id, so +// publishing and inspecting the tracked id directly let a container another +// Docker client relabelled or renamed after tracking still report as this +// server's. GetConnectionDiagnostics must re-verify through +// ContainerMutator.Verify first, the same read+predicate the manager's +// health check (verifyDockerContainerHealthy) already uses. +func TestGetConnectionDiagnostics_ReverifiesOwnershipBeforePublishing(t *testing.T) { + dockerCfg := &config.ServerConfig{Command: "docker", Args: []string{"run", "-i", "--rm", "mcp/example"}} + + t.Run("relabelled to a co-tenant after tracking: no id, not running", func(t *testing.T) { + installFakeDocker(t, ownFixtureNamed(ownContainerID, ownContainerName, "a-b")) + c, _, _ := newOwnershipClient("a", dockerCfg) + c.isDockerCommand = true + c.containerID = ownContainerID // tracked before the relabel + + diag := c.GetConnectionDiagnostics() + + _, hasID := diag["container_id"] + assert.False(t, hasID, "a container relabelled away from this server must not be published as its id") + _, hasOwner := diag["container_owner"] + assert.False(t, hasOwner) + assert.Equal(t, false, diag["container_running"]) + }) + + t.Run("renamed to a co-tenant's canonical shape after tracking: no id, not running", func(t *testing.T) { + installFakeDocker(t, ownFixtureNamed(ownContainerID, foreignContainerName, "a")) + c, _, _ := newOwnershipClient("a", dockerCfg) + c.isDockerCommand = true + c.containerID = ownContainerID + + diag := c.GetConnectionDiagnostics() + + _, hasID := diag["container_id"] + assert.False(t, hasID) + assert.Equal(t, false, diag["container_running"]) + }) + + t.Run("gone after tracking: no id, not running", func(t *testing.T) { + installFakeDocker(t, nil) + c, _, _ := newOwnershipClient("a", dockerCfg) + c.isDockerCommand = true + c.containerID = ownContainerID + + diag := c.GetConnectionDiagnostics() + + _, hasID := diag["container_id"] + assert.False(t, hasID) + assert.Equal(t, false, diag["container_running"]) + }) + + t.Run("unchanged: id and owner published from the verification read", func(t *testing.T) { + installFakeDocker(t, ownFixtureNamed(ownContainerID, ownContainerName, "a")) + c, _, _ := newOwnershipClient("a", dockerCfg) + c.isDockerCommand = true + c.containerID = ownContainerID + + diag := c.GetConnectionDiagnostics() + + assert.Equal(t, ownContainerID, diag["container_id"]) + assert.Equal(t, "a", diag["container_owner"]) + }) +} + +// TestGetConnectionDiagnostics_RunningComesFromTheVerifyReadAlone is codex +// round 16 finding 1: GetConnectionDiagnostics verified ownership with one +// `docker ps` read (ContainerMutator.Verify) and THEN ran a second, +// separately-timed `docker inspect ` to decide container_running. +// Between the two, another Docker client can relabel or rename the +// container into a colliding server's namespace; the inspect would then +// report the NOW-FOREIGN container's state while diagnostics kept +// attributing it to this server (a TOCTOU gap, not merely a stale read). +// Running must come from the Verify read itself (ContainerRow.Running), +// never a follow-up command: the fixture is swapped to a relabelled, +// stopped container right after the one `ps` call the fix makes, so a +// second read — if the fix regressed and one was reintroduced — would see +// that swapped data and this test would catch it either by a changed +// result or by a second invocation showing up in the log. +func TestGetConnectionDiagnostics_RunningComesFromTheVerifyReadAlone(t *testing.T) { + dockerCfg := &config.ServerConfig{Command: "docker", Args: []string{"run", "-i", "--rm", "mcp/example"}} + + fd := installFakeDocker(t, ownFixtureNamed(ownContainerID, ownContainerName, "a")) + // After the fix's one `ps` read answers, swap to a relabelled, stopped + // container: any FURTHER read of this id would see foreign, not-running + // data. + fd.swapFixtureAfterPs(t, 1, ownFixtureNamed(ownContainerID, ownContainerName, "a-b")) + + c, _, _ := newOwnershipClient("a", dockerCfg) + c.isDockerCommand = true + c.containerID = ownContainerID + + diag := c.GetConnectionDiagnostics() + + assert.Equal(t, ownContainerID, diag["container_id"]) + assert.Equal(t, "a", diag["container_owner"], + "container_owner must come from the SAME read as container_running, not a later one that could see the swapped-in relabel") + assert.Equal(t, true, diag["container_running"], + "running must be read.Running() from the one ps row Verify already has, not a second command") + + var psCalls, inspectCalls int + for _, line := range fd.invocations(t) { + fields := strings.Fields(line) + if len(fields) == 0 { + continue + } + switch fields[0] { + case "ps": + psCalls++ + case "inspect": + inspectCalls++ + } + } + assert.Equal(t, 1, psCalls, "diagnostics must issue exactly one docker ps for this container, invocations:\n%s", strings.Join(fd.invocations(t), "\n")) + assert.Zero(t, inspectCalls, "diagnostics must never issue a separate docker inspect, invocations:\n%s", strings.Join(fd.invocations(t), "\n")) +} diff --git a/internal/upstream/core/monitoring.go b/internal/upstream/core/monitoring.go index 6b5f078f7..ad0a3124d 100644 --- a/internal/upstream/core/monitoring.go +++ b/internal/upstream/core/monitoring.go @@ -5,13 +5,13 @@ import ( "context" "fmt" "io" - "os" "regexp" "strings" "time" "go.uber.org/zap" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/logs" "github.com/smart-mcp-proxy/mcpproxy-go/internal/oauth" ) @@ -209,9 +209,13 @@ func (c *Client) monitorStderr(ctx context.Context, stderr io.Reader) { zap.String("server", c.config.Name), zap.String("message", line)) - // Log to server-specific logger if available + // Log to server-specific logger if available. The child's text is + // a field value stamped child_output=true (Spec 105 FR-007, D8 + // rules 1 and 3): a docker CLI failure on the isolation path names + // a colliding container — another server's — and the attributed + // reader withholds child-output records that mention a container. if c.upstreamLogger != nil { - c.upstreamLogger.Info("stderr", zap.String("message", line)) + c.upstreamLogger.Info("stderr", zap.String("message", line), logs.ChildOutputField()) } c.recordRecentStderr(line) @@ -245,15 +249,23 @@ func (c *Client) monitorStderr(ctx context.Context, stderr io.Reader) { } } -// monitorDockerLogsWithContext monitors Docker container logs using `docker logs` with context cancellation +// dockerLogsWaitTimeout bounds how long monitorDockerLogsWithContext waits +// for the container id to be tracked. A variable so tests can shorten it. +var dockerLogsWaitTimeout = 10 * time.Second + +// monitorDockerLogsWithContext monitors Docker container logs using `docker +// logs` with context cancellation. The container it names is only ever the +// one trackCidfileContainer verified (id and owner read back from Docker): +// it never reads the cidfile itself, since a cidfile can name a container +// that is not this server's (Spec 105 D9, codex round 6). func (c *Client) monitorDockerLogsWithContext(ctx context.Context, cidFile string) { waitTicker := time.NewTicker(100 * time.Millisecond) defer waitTicker.Stop() - waitTimeout := time.NewTimer(10 * time.Second) + waitTimeout := time.NewTimer(dockerLogsWaitTimeout) defer waitTimeout.Stop() - var containerID string + var containerID, containerOwner string waitLoop: for { @@ -264,20 +276,13 @@ waitLoop: zap.String("cid_file", cidFile)) return case <-waitTimeout.C: - // Fall back to reading the cid file one time in case tracking goroutine failed - if data, err := os.ReadFile(cidFile); err == nil { - containerID = strings.TrimSpace(string(data)) - if containerID != "" { - break waitLoop - } - } - c.logger.Debug("Docker logs monitoring timed out waiting for container ID", + c.logger.Debug("Docker logs monitoring timed out before a verified container ID was tracked", zap.String("server", c.config.Name), zap.String("cid_file", cidFile)) return case <-waitTicker.C: c.mu.RLock() - containerID = c.containerID + containerID, containerOwner = c.containerID, c.containerOwner c.mu.RUnlock() if containerID != "" { break waitLoop @@ -291,7 +296,8 @@ waitLoop: c.logger.Debug("Docker container started - logs available via 'docker logs' command", zap.String("server", c.config.Name), zap.String("container_id", shortContainerID(containerID)), - zap.String("command", fmt.Sprintf("docker logs -f %s", containerID[:12]))) + containerOwnerField(containerOwner), + zap.String("command", fmt.Sprintf("docker logs -f %s", shortContainerID(containerID)))) // Note: We intentionally do NOT stream container logs to mcpproxy logs because: // 1. It causes massive log file bloat (multiple GB per day with active containers) @@ -305,7 +311,8 @@ waitLoop: <-ctx.Done() c.logger.Debug("Docker logs monitoring ended", zap.String("server", c.config.Name), - zap.String("container_id", shortContainerID(containerID))) + zap.String("container_id", shortContainerID(containerID)), + containerOwnerField(containerOwner)) } // recordRecentStderr appends a stderr line to the bounded ring buffer. @@ -488,7 +495,6 @@ func (c *Client) GetConnectionDiagnostics() map[string]interface{} { if c.isDockerCommand { diagnostics["is_docker"] = true diagnostics["docker_args"] = oauth.LiveRedaction.Argv(c.config.Args) - diagnostics["container_id"] = c.containerID // Check Docker daemon connectivity ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) @@ -502,11 +508,33 @@ func (c *Client) GetConnectionDiagnostics() map[string]interface{} { diagnostics["docker_daemon_reachable"] = true } - // Check if container is still running + // Spec 105 D8/D9: `docker inspect ` resolves purely by id, so + // publishing and inspecting the tracked id directly let a container + // another Docker client relabelled or renamed after tracking still + // report as this server's and running — the same stale-ownership + // failure verifyDockerContainerHealthy fixed for the manager's + // health path (codex round 8). Re-verify through the same + // ContainerMutator.Verify read+predicate before publishing anything: + // once the predicate no longer holds (or the re-read itself fails), + // the diagnostics name no container id and report it not running; + // only a container ownership confirms right now is published, with + // the container_owner read back at that same moment. + // + // Running state comes from that SAME read, never a follow-up + // `docker inspect` (codex round 16 finding 1): a second, separately + // timed command by id alone would report whatever container holds + // that id AT THAT LATER MOMENT — which ownership may no longer + // belong to — while the diagnostics kept attributing it to this + // server. ContainerRow.Running derives it from the ps row Verify + // already read. if c.containerID != "" { - inspectCmd := c.newDockerCmd(ctx, "inspect", "--format", "{{.State.Running}}", c.containerID) - if output, err := inspectCmd.Output(); err == nil { - diagnostics["container_running"] = strings.TrimSpace(string(output)) == "true" + row, ok, err := c.containerMutator().Verify(ctx, c.containerID) + if err != nil || !ok { + diagnostics["container_running"] = false + } else { + diagnostics["container_id"] = row.ID + diagnostics["container_owner"] = row.Owner + diagnostics["container_running"] = row.Running() } } } diff --git a/internal/upstream/core/upstream_logger_audit_test.go b/internal/upstream/core/upstream_logger_audit_test.go new file mode 100644 index 000000000..094b34d2f --- /dev/null +++ b/internal/upstream/core/upstream_logger_audit_test.go @@ -0,0 +1,113 @@ +package core + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/require" +) + +// Spec 105 FR-007, research D8 rule 1 (producer rule): the per-server log is +// the file `upstream_servers tail_log` serves, and its attribution reader +// keys on the `server=` field of every record. Child-controlled text +// (stderr lines, launcher output, docker output) must therefore only ever be +// a zap FIELD VALUE — zap escapes it inside the fields object — never the +// message, where the console encoder writes it unescaped and a crafted line +// could try to look like a record boundary. This test is the audit: every +// zap level call — on ANY receiver — in the packages that write into the +// per-server file passes a constant string literal as its message. (T054a; +// expected green on HEAD — it pins the invariant the reader rule relies on.) +// +// Critique round 1, finding C1.3 / C2.12: the audit covers every receiver, +// not only `upstreamLogger`, because the per-server file is also written +// through `oauthLogger()`'s tee (client.go) from internal/oauth, through +// loggerWriter's `primary`/`fallback` here, and through any local alias a +// future edit introduces. Since codex round 2 loggerWriter.writeLine writes +// the launcher-pumped child line as a field value too (it used to be the one +// allowed non-constant message), so every child path is under rule 1. + +// auditedLogPackages are the directories, relative to this package, whose +// production files write into the per-server log. +var auditedLogPackages = []string{".", "../launcher", "../../oauth"} + +// auditAllowedNonConstant lists the call sites permitted to pass a +// non-constant message, as ":". Each entry is a +// reviewed exception, not child-controlled text: +// - connection_http.go:runAuthStrategies — "🔐 Trying "+transportLabel+… +// where transportLabel is one of two string literals chosen by +// connectHTTP/connectSSE, never data from the upstream. +var auditAllowedNonConstant = map[string]bool{ + "connection_http.go:runAuthStrategies": true, +} + +var zapLevelMethods = map[string]bool{ + "Debug": true, "Info": true, "Warn": true, "Error": true, + "DPanic": true, "Panic": true, "Fatal": true, +} + +func TestUpstreamLoggerAudit_MessagesAreConstant(t *testing.T) { + fset := token.NewFileSet() + var violations []string + audited := 0 + upstreamLoggerSites := 0 + + for _, dir := range auditedLogPackages { + entries, err := os.ReadDir(dir) + require.NoError(t, err, dir) + for _, entry := range entries { + name := entry.Name() + if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") { + continue + } + file, err := parser.ParseFile(fset, filepath.Join(dir, name), nil, 0) + require.NoError(t, err, name) + + for _, decl := range file.Decls { + fn, ok := decl.(*ast.FuncDecl) + if !ok || fn.Body == nil { + continue + } + ast.Inspect(fn.Body, func(n ast.Node) bool { + call, ok := n.(*ast.CallExpr) + if !ok { + return true + } + sel, ok := call.Fun.(*ast.SelectorExpr) + if !ok || !zapLevelMethods[sel.Sel.Name] { + return true + } + // zap.Error(err) is a field constructor, not a level call; + // err.Error() takes no message. + if ident, ok := sel.X.(*ast.Ident); ok && ident.Name == "zap" { + return true + } + if len(call.Args) == 0 { + return true + } + audited++ + if recv, ok := sel.X.(*ast.SelectorExpr); ok && recv.Sel.Name == "upstreamLogger" { + upstreamLoggerSites++ + } + if lit, ok := call.Args[0].(*ast.BasicLit); ok && lit.Kind == token.STRING { + return true + } + if auditAllowedNonConstant[name+":"+fn.Name.Name] { + return true + } + violations = append(violations, fset.Position(call.Pos()).String()+": message is not a string literal") + return true + }) + } + } + } + + require.NotZero(t, upstreamLoggerSites, "the audit found no upstreamLogger call sites — the receiver name changed and the audit is vacuous") + require.Greater(t, audited, upstreamLoggerSites, "the audit must see receivers beyond upstreamLogger (oauth tee, loggerWriter)") + require.Empty(t, violations, "zap level calls with a non-constant message (child text must be a field value):\n%s", + strings.Join(violations, "\n")) +} diff --git a/internal/upstream/managed/client.go b/internal/upstream/managed/client.go index b0edee308..cc8f997bd 100644 --- a/internal/upstream/managed/client.go +++ b/internal/upstream/managed/client.go @@ -2088,6 +2088,18 @@ func (mc *Client) GetContainerID() string { return mc.coreClient.GetContainerID() } +// ForceRemoveTrackedContainerIfOwned is the manager's disconnect-timeout +// path: it removes the container tracked as containerID only after the core +// client re-establishes canonical ownership (Spec 105 FR-007 / D9, codex +// round 3) and hands back the owner label it read at that moment (codex +// round 5). See core.Client.ForceRemoveTrackedContainerIfOwned. +func (mc *Client) ForceRemoveTrackedContainerIfOwned(ctx context.Context, containerID string) (string, bool, error) { + if mc.coreClient == nil { + return "", false, nil + } + return mc.coreClient.ForceRemoveTrackedContainerIfOwned(ctx, containerID) +} + // setToolCountCache records the latest tool count and timestamp for non-blocking consumers. func (mc *Client) setToolCountCache(count int) { mc.toolCountMu.Lock() diff --git a/internal/upstream/manager.go b/internal/upstream/manager.go index 42bf50ea3..035cafcda 100644 --- a/internal/upstream/manager.go +++ b/internal/upstream/manager.go @@ -754,98 +754,269 @@ func (m *Manager) ShutdownAll(ctx context.Context) error { return nil } -// cleanupAllManagedContainers finds and stops all Docker containers managed by mcpproxy -// Uses labels to identify containers across all instances +// managedContainerFormat is the `docker ps --format` both sweeps read: id, +// name and the owner + instance labels, tab-separated, labels last so an +// empty label leaves its column empty rather than shifting the others. +const managedContainerFormat = "{{.ID}}\t{{.Names}}\t{{.Label \"com.mcpproxy.server\"}}\t{{.Label \"com.mcpproxy.instance\"}}" + +// managedContainer is one `docker ps` row of a sweep. +type managedContainer struct { + ID string + Name string + Owner string // the com.mcpproxy.server label value as Docker reported it + Instance string // the com.mcpproxy.instance label value as Docker reported it +} + +// configuredServerNames returns the raw name of every configured server +// (enabled or not) — the set of possible canonical owners for a sweep. +func (m *Manager) configuredServerNames() []string { + m.mu.RLock() + defer m.mu.RUnlock() + names := make([]string, 0, len(m.clients)) + for _, client := range m.clients { + if client == nil { + continue + } + if cfg := client.GetConfig(); cfg != nil { + names = append(names, cfg.Name) + } + } + return names +} + +// sweepDocker is how the sweeps run docker: the bare name on PATH. +func sweepDocker(ctx context.Context, args ...string) *exec.Cmd { + return exec.CommandContext(ctx, "docker", args...) +} + +// readManagedContainers runs `docker ps [-a] --no-trunc` with the given +// filters and returns every row — full id, name and owner label as Docker +// reports them — with no ownership applied; a row the format could not be +// parsed is returned with an empty name and owner so it fails the predicate. +// includeStopped adds `-a`; without it only rows Docker reports as running +// come back — the same includeStopped convention core.Client's own +// listOwnedContainersFiltered uses (codex round 10, docker finding 1): a +// caller that only cares whether something is CURRENTLY running, such as +// HasDockerContainers, must not see an already-stopped row and misreport it +// as still running. +func (m *Manager) readManagedContainers(ctx context.Context, includeStopped bool, filters ...string) ([]managedContainer, error) { + args := []string{"ps"} + if includeStopped { + args = append(args, "-a") + } + args = append(args, "--no-trunc") + for _, filter := range filters { + args = append(args, "--filter", filter) + } + args = append(args, "--format", managedContainerFormat) + output, err := sweepDocker(ctx, args...).Output() + if err != nil { + return nil, err + } + + // NOT strings.TrimSpace(output) before splitting, and no per-line + // "keep the rest" on a bad count (codex rounds 3 and 4: FR-007 + // instance-scoping fix). Label VALUES have no tab- or + // newline-escaping, and this listing's own docker-side filter + // deliberately does NOT constrain com.mcpproxy.server (it must match + // ANY configured server, checked in Go by core.ContainerOwnedByAny) — + // so a container an attacker creates themselves, carrying + // com.mcpproxy.managed=true, can give that Owner label a value + // engineered to smuggle "\t" past an exact-match + // comparison, or worse, containing a literal newline that splits what + // Docker rendered as ONE row into what looks like a second, + // independently well-formed line naming a DIFFERENT id, name, owner + // and instance of the attacker's choosing. A single malformed line + // proves this listing's line boundaries are untrustworthy, so ANY bad + // count discards the WHOLE listing (report nothing found) rather than + // keeping whichever rows still look well-formed. + var rows []managedContainer + for _, line := range strings.Split(string(output), "\n") { + if line == "" { + continue + } + parts := strings.Split(line, "\t") + if len(parts) != 4 { + m.logger.Warn("Discarding managed-container listing: a docker ps row did not parse to the expected field count") + return nil, nil + } + rows = append(rows, managedContainer{ID: parts[0], Name: parts[1], Owner: parts[2], Instance: parts[3]}) + } + return rows, nil +} + +// listOwnedManagedContainers runs `docker ps [-a]` with the given label +// filters and returns only the rows canonically owned by a configured +// server: com.mcpproxy.server= AND name +// ^mcpproxy--[a-z0-9]{4}$ for the SAME configured server +// (core.ContainerOwnedByAny). The managed and instance labels a sweep +// selects on are shared and copyable, so on their own they are not +// ownership (Spec 105 FR-007 / D9, codex round 3): a foreign container +// carrying them is neither mutated nor named. A row that fails the +// predicate is also not counted: its label is untrusted (that is exactly +// why it was rejected), so no owner can be attributed to it, and D8's +// evidence rule requires every container count to carry the owner it +// counts (codex round 11) — there is no non-fabricated owner to put on a +// tally of rejected rows, so listOwnedManagedContainers logs nothing about +// them at all. includeStopped is passed straight through to +// readManagedContainers. +func (m *Manager) listOwnedManagedContainers(ctx context.Context, includeStopped bool, filters ...string) ([]managedContainer, error) { + rows, err := m.readManagedContainers(ctx, includeStopped, filters...) + if err != nil { + return nil, err + } + + configured := m.configuredServerNames() + var owned []managedContainer + for _, row := range rows { + if !core.ContainerOwnedByAny(configured, row.Name, row.Owner, row.Instance) { + continue + } + owned = append(owned, row) + } + return owned, nil +} + +// logOwnerGroupedCounts writes one record per container_owner represented +// in owned, each carrying that owner and how many rows it accounts for — +// never a single aggregate. A sweep can select containers belonging to more +// than one configured server, so one bare count cannot be bound to a +// subject (Spec 105 D8, codex round 10 finding 2); level is m.logger.Info or +// m.logger.Warn, matching the call site's own level for the record it +// replaces. +func (m *Manager) logOwnerGroupedCounts(msg string, level func(msg string, fields ...zap.Field), owned []managedContainer) { + counts := make(map[string]int, len(owned)) + for _, row := range owned { + counts[row.Owner]++ + } + for owner, count := range counts { + level(msg, zap.String("container_owner", owner), zap.Int("count", count)) + } +} + +// mutateOwnedManagedContainer runs op on one selected container through +// core.ContainerMutator — the one verify-then-mutate implementation the +// core client's cleanup paths use too (Spec 105 FR-007 / D9, codex rounds 5 +// and 6): the selection `docker ps` is a snapshot, and another Docker client +// can rename or relabel the container between that listing and the +// stop/kill/rm, so its full id, name and label are re-read immediately +// before the command and core.ContainerOwnedByAny re-applied over the +// configured servers. A refusal — the re-read failed, or the container no +// longer satisfies the predicate — is recorded naming only the listing-time +// server, never the id or name; the row handed back is the one read NOW, so +// its Owner is what the mutation's records carry. intent is called with +// that row right before the command. +func (m *Manager) mutateOwnedManagedContainer(ctx context.Context, selected managedContainer, op core.ContainerMutation, intent func(core.ContainerRow)) core.MutationResult { + mutator := core.ContainerMutator{ + Docker: sweepDocker, + Owns: func(containerName, ownerLabel, instanceLabel string) bool { + return core.ContainerOwnedByAny(m.configuredServerNames(), containerName, ownerLabel, instanceLabel) + }, + } + res := mutator.Mutate(ctx, selected.ID, op, intent) + switch { + case res.Verified: + case res.Err != nil: + m.logger.Warn("Could not re-verify ownership of a selected container - leaving it alone", + zap.String("server", selected.Owner), + zap.String("operation", string(op)), + zap.Error(res.Err)) + default: + m.logger.Warn("Selected container is no longer canonically owned by a configured server - leaving it alone", + zap.String("server", selected.Owner), + zap.String("operation", string(op))) + } + return res +} + +// cleanupAllManagedContainers finds and stops this instance's Docker +// containers managed by mcpproxy. The initial `docker ps` filter is the +// shared, copyable com.mcpproxy.managed label only — deliberately broad — +// but listOwnedManagedContainers' canonical-ownership check +// (core.ContainerOwnedByAny) then keeps only the rows that ALSO carry this +// process's own com.mcpproxy.instance label: a row from another live +// mcpproxy instance (same host, a configured server with the same name) is +// exactly as foreign as one with no label at all, and this shutdown path +// must never stop or remove a container it does not own. func (m *Manager) cleanupAllManagedContainers(ctx context.Context) { m.logger.Info("Cleaning up all mcpproxy-managed Docker containers") // Find all containers with our management label - listCmd := exec.CommandContext(ctx, "docker", "ps", "-a", - "--filter", "label=com.mcpproxy.managed=true", - "--format", "{{.ID}}\t{{.Names}}\t{{.Label \"com.mcpproxy.server\"}}") - - output, err := listCmd.Output() + owned, err := m.listOwnedManagedContainers(ctx, true, "label=com.mcpproxy.managed=true") if err != nil { m.logger.Debug("No Docker containers found or Docker unavailable", zap.Error(err)) return } - lines := strings.Split(strings.TrimSpace(string(output)), "\n") - if len(lines) == 0 || lines[0] == "" { + if len(owned) == 0 { m.logger.Debug("No mcpproxy-managed containers found") return } - m.logger.Info("Found mcpproxy-managed containers to cleanup", - zap.Int("count", len(lines))) + m.logOwnerGroupedCounts("Found mcpproxy-managed containers to cleanup", m.logger.Info, owned) // Grace period for graceful shutdown gracePeriod := 10 * time.Second graceCtx, graceCancel := context.WithTimeout(ctx, gracePeriod) defer graceCancel() - containerIDs := []string{} - for _, line := range lines { - if line == "" { + // Ownership is re-established right before each mutation + // (mutateOwnedManagedContainer), and every record below that names a + // container carries the owner Docker reported for it at that moment + // (container_owner), on the outcome records as well as the intent ones + // (Spec 105 D8/D9, codex rounds 4 and 5). + for _, selected := range owned { + // Try graceful stop first + res := m.mutateOwnedManagedContainer(graceCtx, selected, core.ContainerStop, func(container core.ContainerRow) { + m.logger.Info("Stopping container", + zap.String("container_id", container.ID), + zap.String("container_name", container.Name), + zap.String("server", container.Owner), + zap.String("container_owner", container.Owner)) + }) + if !res.Verified { continue } - parts := strings.SplitN(line, "\t", 3) - if len(parts) >= 1 { - containerID := parts[0] - containerName := "" - serverName := "" - if len(parts) >= 2 { - containerName = parts[1] - } - if len(parts) >= 3 { - serverName = parts[2] - } - - m.logger.Info("Stopping container", - zap.String("container_id", containerID), - zap.String("container_name", containerName), - zap.String("server", serverName)) - - containerIDs = append(containerIDs, containerID) - - // Try graceful stop first - stopCmd := exec.CommandContext(graceCtx, "docker", "stop", containerID) - if err := stopCmd.Run(); err != nil { - m.logger.Warn("Graceful stop failed, will force kill", - zap.String("container_id", containerID), - zap.Error(err)) - } else { - m.logger.Info("Container stopped gracefully", - zap.String("container_id", containerID)) - } + if res.Err != nil { + m.logger.Warn("Graceful stop failed, will force kill", + zap.String("container_id", res.Container.ID), + zap.String("container_owner", res.Container.Owner), + zap.Error(res.Err)) + } else { + m.logger.Info("Container stopped gracefully", + zap.String("container_id", res.Container.ID), + zap.String("container_owner", res.Container.Owner)) } } // Force kill any remaining containers after grace period - if graceCtx.Err() != nil || len(containerIDs) > 0 { - m.logger.Info("Force killing any remaining containers") - - killCtx, killCancel := context.WithTimeout(ctx, 5*time.Second) - defer killCancel() - - for _, containerID := range containerIDs { - // Check if container is still running - psCmd := exec.CommandContext(killCtx, "docker", "ps", "-q", - "--filter", "id="+containerID) - if output, err := psCmd.Output(); err == nil && len(strings.TrimSpace(string(output))) > 0 { - // Still running, force kill - m.logger.Info("Force killing container", - zap.String("container_id", containerID)) + m.logger.Info("Force killing any remaining containers") - killCmd := exec.CommandContext(killCtx, "docker", "kill", containerID) - if err := killCmd.Run(); err != nil { - m.logger.Error("Failed to force kill container", - zap.String("container_id", containerID), - zap.Error(err)) - } else { - m.logger.Info("Container force killed", - zap.String("container_id", containerID)) - } + killCtx, killCancel := context.WithTimeout(ctx, 5*time.Second) + defer killCancel() + + for _, selected := range owned { + // Check if container is still running + psCmd := sweepDocker(killCtx, "ps", "-q", "--filter", "id="+selected.ID) + if output, err := psCmd.Output(); err == nil && len(strings.TrimSpace(string(output))) > 0 { + // Still running, force kill + res := m.mutateOwnedManagedContainer(killCtx, selected, core.ContainerKill, func(container core.ContainerRow) { + m.logger.Info("Force killing container", + zap.String("container_id", container.ID), + zap.String("container_owner", container.Owner)) + }) + if !res.Verified { + continue + } + if res.Err != nil { + m.logger.Error("Failed to force kill container", + zap.String("container_id", res.Container.ID), + zap.String("container_owner", res.Container.Owner), + zap.Error(res.Err)) + } else { + m.logger.Info("Container force killed", + zap.String("container_id", res.Container.ID), + zap.String("container_owner", res.Container.Owner)) } } } @@ -855,7 +1026,8 @@ func (m *Manager) cleanupAllManagedContainers(ctx context.Context) { // ForceCleanupAllContainers is a public wrapper for emergency container cleanup // This is called when graceful shutdown fails and containers must be force-removed -// Only removes containers owned by THIS instance (matching instance ID) +// Only removes containers owned by THIS instance (matching instance ID) AND +// canonically owned by a configured server (listOwnedManagedContainers). func (m *Manager) ForceCleanupAllContainers() { m.logger.Warn("Force cleanup requested - removing all managed containers for this instance") @@ -865,65 +1037,79 @@ func (m *Manager) ForceCleanupAllContainers() { // Find all containers with our management label AND our instance ID instanceID := core.GetInstanceID() - listCmd := exec.CommandContext(ctx, "docker", "ps", "-a", - "--filter", "label=com.mcpproxy.managed=true", - "--filter", fmt.Sprintf("label=com.mcpproxy.instance=%s", instanceID), - "--format", "{{.ID}}\t{{.Names}}") - - output, err := listCmd.Output() + owned, err := m.listOwnedManagedContainers(ctx, true, + "label=com.mcpproxy.managed=true", + fmt.Sprintf("label=com.mcpproxy.instance=%s", instanceID)) if err != nil { m.logger.Warn("Failed to list managed containers for force cleanup", zap.Error(err)) return } - lines := strings.Split(strings.TrimSpace(string(output)), "\n") - if len(lines) == 0 || lines[0] == "" { + if len(owned) == 0 { m.logger.Info("No managed containers found during force cleanup") return } - m.logger.Warn("Force removing managed containers", - zap.Int("count", len(lines))) + m.logOwnerGroupedCounts("Force removing managed containers", m.logger.Warn, owned) - // Force remove each container (skip graceful stop) - for _, line := range lines { - if line == "" { - continue - } - parts := strings.SplitN(line, "\t", 2) - if len(parts) < 1 { + // Force remove each container (skip graceful stop), re-establishing + // ownership right before the rm (D9 moment-of-mutation rule). Use docker + // rm -f to force remove (kills and removes in one step). The outcome + // records carry the owner read at mutation time (D8/D9). + for _, selected := range owned { + res := m.mutateOwnedManagedContainer(ctx, selected, core.ContainerRemove, func(container core.ContainerRow) { + m.logger.Warn("Force removing container", + zap.String("id", shortContainerID(container.ID)), + zap.String("name", container.Name), + zap.String("container_owner", container.Owner)) + }) + if !res.Verified { continue } - - containerID := parts[0] - containerName := "" - if len(parts) >= 2 { - containerName = parts[1] - } - - m.logger.Warn("Force removing container", - zap.String("id", containerID[:12]), - zap.String("name", containerName)) - - // Use docker rm -f to force remove (kills and removes in one step) - rmCmd := exec.CommandContext(ctx, "docker", "rm", "-f", containerID) - if err := rmCmd.Run(); err != nil { + if res.Err != nil { m.logger.Error("Failed to force remove container", - zap.String("id", containerID[:12]), - zap.String("name", containerName), - zap.Error(err)) + zap.String("id", shortContainerID(res.Container.ID)), + zap.String("name", res.Container.Name), + zap.String("container_owner", res.Container.Owner), + zap.Error(res.Err)) } else { m.logger.Info("Container force removed successfully", - zap.String("id", containerID[:12]), - zap.String("name", containerName)) + zap.String("id", shortContainerID(res.Container.ID)), + zap.String("name", res.Container.Name), + zap.String("container_owner", res.Container.Owner)) } } m.logger.Info("Force cleanup completed") } +// shortContainerID renders the 12-character short form of a container id. +func shortContainerID(id string) string { + if len(id) <= 12 { + return id + } + return id[:12] +} + +// forceCleanupTarget is what forceCleanupClient needs from a managed client: +// its configuration, the container id it tracks, and the ownership-checked +// removal the core client performs. +type forceCleanupTarget interface { + GetConfig() *config.ServerConfig + GetContainerID() string + ForceRemoveTrackedContainerIfOwned(ctx context.Context, containerID string) (owner string, owned bool, err error) +} + // forceCleanupClient forces cleanup of a specific client's Docker container -func (m *Manager) forceCleanupClient(client *managed.Client) { +// when its Disconnect timed out. The stored id is not removed blindly: the +// core client re-establishes canonical ownership at the moment of the +// mutation (Spec 105 FR-007 / D9, codex round 3), so a container renamed or +// reused under that id since it was tracked is left alone. The manager's own +// records name the container only once that verdict exists and with the +// owner the core read back (D8 subject-evidence rule, codex round 5): before +// it, and when the container was rejected or could not be verified, they +// name the server alone. +func (m *Manager) forceCleanupClient(client forceCleanupTarget) { containerID := client.GetContainerID() if containerID == "" { m.logger.Debug("No container ID for force cleanup", @@ -932,23 +1118,31 @@ func (m *Manager) forceCleanupClient(client *managed.Client) { } m.logger.Warn("Force cleaning up container for client", - zap.String("server", client.GetConfig().Name), - zap.String("container_id", containerID[:12])) + zap.String("server", client.GetConfig().Name)) ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() - // Force remove container - rmCmd := exec.CommandContext(ctx, "docker", "rm", "-f", containerID) - if err := rmCmd.Run(); err != nil { + owner, owned, err := client.ForceRemoveTrackedContainerIfOwned(ctx, containerID) + switch { + case err != nil && owned: m.logger.Error("Failed to force remove container", zap.String("server", client.GetConfig().Name), - zap.String("container_id", containerID[:12]), + zap.String("container_id", shortContainerID(containerID)), + zap.String("container_owner", owner), zap.Error(err)) - } else { + case err != nil: + m.logger.Error("Could not verify ownership of the tracked container - left alone", + zap.String("server", client.GetConfig().Name), + zap.Error(err)) + case !owned: + m.logger.Info("Tracked container not canonically owned by the client - left alone", + zap.String("server", client.GetConfig().Name)) + default: m.logger.Info("Container force removed successfully", zap.String("server", client.GetConfig().Name), - zap.String("container_id", containerID[:12])) + zap.String("container_id", shortContainerID(containerID)), + zap.String("container_owner", owner)) } } @@ -1729,26 +1923,33 @@ func (m *Manager) DisconnectAll() error { return nil } -// HasDockerContainers checks if any Docker containers owned by THIS instance are actually running +// HasDockerContainers reports whether any RUNNING Docker container this +// instance manages (com.mcpproxy.managed=true, com.mcpproxy.instance=) is also canonically owned by a configured server — +// core.ContainerOwnedByAny over listOwnedManagedContainers, the same +// selection the shutdown and emergency sweeps apply. The instance/managed +// labels alone are shared and copyable (Spec 105 D9): a foreign container +// that copies them, or one whose owning server was since removed from +// config, must not be reported as still running — that false positive drove +// the runtime/server shutdown path into its 15-second cleanup-verification +// wait, a second force-clean, and a false "still running after force +// cleanup" report for a container mcpproxy neither started nor can act on +// (codex round 10, docker finding 1). includeStopped is false: an +// already-stopped row must not read as still running either. func (m *Manager) HasDockerContainers() bool { - // Check if any containers with our labels AND our instance ID are running ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second) defer cancel() instanceID := core.GetInstanceID() - listCmd := exec.CommandContext(ctx, "docker", "ps", "-q", - "--filter", "label=com.mcpproxy.managed=true", - "--filter", fmt.Sprintf("label=com.mcpproxy.instance=%s", instanceID)) - - output, err := listCmd.Output() + owned, err := m.listOwnedManagedContainers(ctx, false, + "label=com.mcpproxy.managed=true", + fmt.Sprintf("label=com.mcpproxy.instance=%s", instanceID)) if err != nil { // Docker not available or error listing - assume no containers return false } - // If output is not empty, we have running containers - containerIDs := strings.TrimSpace(string(output)) - return containerIDs != "" + return len(owned) > 0 } // GetStats returns statistics about upstream connections @@ -2074,33 +2275,70 @@ func (m *Manager) verifyContainerHealthy(client *managed.Client) (bool, error) { return false, fmt.Errorf("no container ID available") } + serverName := "" + if cfg := client.GetConfig(); cfg != nil { + serverName = cfg.Name + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() - // Check 1: Container exists and is running - inspectCmd := exec.CommandContext(ctx, "docker", "inspect", - "--format", "{{.State.Running}},{{.State.Status}}", - containerID) - - output, err := inspectCmd.Output() + return m.verifyDockerContainerHealthy(ctx, sweepDocker, serverName, containerID) +} + +// verifyDockerContainerHealthy is the pure implementation verifyContainerHealthy +// delegates to. `docker inspect ` answers by id alone, regardless of +// name or label, so trusting it directly on the tracked id let a container +// another Docker client relabelled or renamed after tracking still read as +// Running and skip ForceReconnectAll's recovery even though it is no longer +// this server's (codex round 8). Ownership is re-established first, through +// the same read+predicate ContainerMutator.Verify uses before every +// mutation: a container that fails the predicate now is NOT healthy — +// recovery (the caller's rebuild) proceeds — and the refusal names no id, +// only the server. Only a container ownership confirms is named, and then +// with the container_owner read back at that same moment, never the +// requesting server's name. +// +// Running state is decided from that SAME read, never a follow-up `docker +// inspect` (codex round 16 finding 1): a second, separately timed command by +// id alone reports whatever container holds that id AT THAT LATER MOMENT — +// which can by then belong to someone else — while this function kept +// treating it as healthy for serverName. ContainerRow.Running derives it +// from the ps row Verify already read; row.Status (its human STATUS text) +// is what the log/error messages below report. +func (m *Manager) verifyDockerContainerHealthy(ctx context.Context, docker core.DockerCommand, serverName, containerID string) (bool, error) { + mutator := core.ContainerMutator{ + Docker: docker, + Owns: func(containerName, ownerLabel, instanceLabel string) bool { + return core.ContainerOwnedByAny([]string{serverName}, containerName, ownerLabel, instanceLabel) + }, + } + row, ok, err := mutator.Verify(ctx, containerID) if err != nil { - return false, fmt.Errorf("container not found or unreachable: %w", err) + m.logger.Warn("Could not verify container ownership before health check - treating as unhealthy", + zap.String("server", serverName), + zap.Error(err)) + return false, fmt.Errorf("could not verify container ownership: %w", err) } - - parts := strings.Split(strings.TrimSpace(string(output)), ",") - if len(parts) < 2 { - return false, fmt.Errorf("unexpected docker inspect output: %s", string(output)) + if !ok { + m.logger.Warn("Tracked container is no longer canonically owned by this server - treating as lost", + zap.String("server", serverName)) + return false, fmt.Errorf("tracked container is no longer canonically owned by this server") } - running := parts[0] == "true" - status := parts[1] + // Container exists and is canonically owned NOW: check it is running, + // from the row this same Verify read — not a second command. + running := row.Running() + status := row.Status if !running { return false, fmt.Errorf("container not running (status: %s)", status) } m.logger.Debug("Container health check passed", - zap.String("container_id", containerID[:12]), + zap.String("server", serverName), + zap.String("container_id", row.ID), + zap.String("container_owner", row.Owner), zap.String("status", status)) return true, nil diff --git a/internal/upstream/manager_container_ownership_test.go b/internal/upstream/manager_container_ownership_test.go new file mode 100644 index 000000000..adf3ed841 --- /dev/null +++ b/internal/upstream/manager_container_ownership_test.go @@ -0,0 +1,965 @@ +package upstream + +import ( + "context" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap" + "go.uber.org/zap/zaptest/observer" + + "github.com/smart-mcp-proxy/mcpproxy-go/internal/config" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/secret" + "github.com/smart-mcp-proxy/mcpproxy-go/internal/upstream/core" +) + +// Spec 105 FR-007 / research D9, codex round 3 (docker findings 1 and 2). +// +// The manager has three container paths of its own, beside the per-server +// ones in internal/upstream/core: the shutdown sweep +// (cleanupAllManagedContainers) selects every container carrying +// com.mcpproxy.managed=true, the emergency sweep (ForceCleanupAllContainers) +// every container carrying that AND this instance's id, and the +// disconnect-timeout path (forceCleanupClient) ran `docker rm -f` on a +// client's stored id. Labels are copyable and a stored id can be renamed or +// reused, so none of the three was canonical ownership. Now every selected +// container must ALSO be canonically owned by a configured server — label +// com.mcpproxy.server= AND name ^mcpproxy--[a-z0-9]{4}$ +// for the SAME configured server (core.ContainerOwnedByAny) — and the +// emergency path re-inspects the stored id through the core predicate. +// Foreign rows are neither mutated nor named: they are counted at Warn. + +// managerFakeDocker is a sh+awk `docker` shim on PATH (the manager sweeps +// exec the bare name): `ps` answers from a TSV fixture honouring every +// `--filter label=k[=v]` (joined with `|`, which no label here contains), +// `--filter id=` (a prefix match, as docker's is) and `--format` with {{.ID}}, {{.Names}}, +// {{.Status}} and {{.Label "k"}}; without `-a` only rows marked Running are +// answered — same as real `docker ps` — regardless of `-q` (default: every +// container already stopped, so a plain `ps` with no `-a` answers nothing +// until a row sets Running: true). {{.Status}} synthesises a plausible +// `docker ps` human STATUS text from the fixture's Running bool alone ("Up 1 +// second" / "Exited (0) 1 second ago") — the same "Up" prefix real Docker +// uses regardless of state (codex round 16 finding 1: ContainerRow.Running +// reads it), so this fixture needs no separate paused/restarting case. +// stop/kill/rm exit 0 unless the verb is listed in the fail file +// (failVerbs). Every invocation is +// appended to a log. A `ps.tsv.next` fixture (swapFixtureAfterNextPs) +// replaces the fixture right after the next `ps` answers, so a container +// can change between the sweep's listing and its mutation. +type managerFakeDocker struct { + logPath string + psPath string + failPath string +} + +type managerFakeContainer struct { + ID string + Name string + Labels map[string]string + Running bool +} + +const managerFakeDockerShim = `#!/bin/sh +LOG=%s +PS=%s +FAIL=%s +printf '%%s\n' "$*" >> "$LOG" +if [ -f "$FAIL" ]; then + read -r failverbs < "$FAIL" + case " $failverbs " in *" $1 "*) exit 1 ;; esac +fi +if [ "$1" = inspect ]; then + shift + ifmt='' + iid='' + while [ $# -gt 0 ]; do + case "$1" in + --format) ifmt="$2"; shift 2 ;; + *) iid="$1"; shift ;; + esac + done + row=$(awk -F'\t' -v id="$iid" '$1==id{print; found=1} END{exit !found}' "$PS") + if [ -z "$row" ]; then echo "Error: No such object: $iid" >&2; exit 1; fi + printf '%%s\n' "$row" | awk -F'\t' -v fmt="$ifmt" ' + function repl(s, lit, val, i, out) { + out = "" + while ((i = index(s, lit)) > 0) { out = out substr(s, 1, i - 1) val; s = substr(s, i + length(lit)) } + return out s + } + { + running = ($4 == "1") ? "true" : "false" + status = ($4 == "1") ? "running" : "exited" + out = fmt + out = repl(out, "{{.State.Running}}", running) + out = repl(out, "{{.State.Status}}", status) + print out + }' + exit 0 +fi +[ "$1" = ps ] || exit 0 +shift +format='{{.ID}} {{.Names}}' +quiet=0 +all=0 +filters='' +idflt='' +while [ $# -gt 0 ]; do + case "$1" in + --format) format="$2"; shift 2 ;; + -q) quiet=1; format='{{.ID}}'; shift ;; + -a) all=1; shift ;; + --filter|-f) + case "$2" in + label=*) filters="$filters${2#label=}|" ;; + id=*) idflt="${2#id=}" ;; + esac + shift 2 ;; + *) shift ;; + esac +done +awk -F'\t' -v fmt="$format" -v flt="$filters" -v all="$all" -v idflt="$idflt" ' +function repl(s, lit, val, i, out) { + out = "" + while ((i = index(s, lit)) > 0) { out = out substr(s, 1, i - 1) val; s = substr(s, i + length(lit)) } + return out s +} +BEGIN { nflt = split(flt, fl, "|") } +{ + if (idflt != "" && index($1, idflt) != 1) next + if (all == 0 && $4 != "1") next + delete labels + n = split($3, pairs, ",") + for (i = 1; i <= n; i++) { eq = index(pairs[i], "="); if (eq > 0) labels[substr(pairs[i], 1, eq - 1)] = substr(pairs[i], eq + 1) } + for (j = 1; j <= nflt; j++) { + if (fl[j] == "") continue + eq = index(fl[j], "=") + if (eq == 0) { if (!(fl[j] in labels)) next; continue } + k = substr(fl[j], 1, eq - 1); v = substr(fl[j], eq + 1) + if (!(k in labels) || labels[k] != v) next + } + out = fmt + out = repl(out, "{{.ID}}", $1) + out = repl(out, "{{.Names}}", $2) + out = repl(out, "{{.Status}}", ($4 == "1") ? "Up 1 second" : "Exited (0) 1 second ago") + while (match(out, /\{\{\.Label "[^"]*"\}\}/)) { + key = substr(out, RSTART + 10, RLENGTH - 13) + out = substr(out, 1, RSTART - 1) labels[key] substr(out, RSTART + RLENGTH) + } + print out +}' "$PS" +if [ -f "$PS.next" ]; then mv "$PS.next" "$PS"; fi +` + +func shellQuoteForManagerShim(s string) string { return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'" } + +func installManagerFakeDocker(t *testing.T, containers []managerFakeContainer) *managerFakeDocker { + t.Helper() + if runtime.GOOS == "windows" { + t.Skip("unix shell shim") + } + dir := t.TempDir() + fd := &managerFakeDocker{ + logPath: filepath.Join(dir, "invocations.log"), + psPath: filepath.Join(dir, "ps.tsv"), + failPath: filepath.Join(dir, "fail"), + } + psPath := fd.psPath + require.NoError(t, os.WriteFile(psPath, managerFakeFixtureTSV(containers), 0o600)) + + toolDir := filepath.Join(dir, "path") + require.NoError(t, os.Mkdir(toolDir, 0o755)) + script := fmt.Sprintf(managerFakeDockerShim, shellQuoteForManagerShim(fd.logPath), shellQuoteForManagerShim(psPath), shellQuoteForManagerShim(fd.failPath)) + require.NoError(t, os.WriteFile(filepath.Join(toolDir, "docker"), []byte(script), 0o755)) + for _, tool := range []string{"sh", "awk", "printf", "mv"} { + if real, err := exec.LookPath(tool); err == nil { + require.NoError(t, os.Symlink(real, filepath.Join(toolDir, tool))) + } + } + t.Setenv("PATH", toolDir) + return fd +} + +// managerFakeFixtureTSV renders the `ps` fixture the shim reads. +func managerFakeFixtureTSV(containers []managerFakeContainer) []byte { + var tsv strings.Builder + for _, c := range containers { + labels := make([]string, 0, len(c.Labels)) + for k, v := range c.Labels { + labels = append(labels, k+"="+v) + } + running := "0" + if c.Running { + running = "1" + } + fmt.Fprintf(&tsv, "%s\t%s\t%s\t%s\n", c.ID, c.Name, strings.Join(labels, ","), running) + } + return []byte(tsv.String()) +} + +// swapFixtureAfterNextPs makes the shim answer the NEXT `ps` from the +// current fixture and every later one from containers — the state of the +// daemon after another client changed it between listing and mutation. +func (fd *managerFakeDocker) swapFixtureAfterNextPs(t *testing.T, containers []managerFakeContainer) { + t.Helper() + require.NoError(t, os.WriteFile(fd.psPath+".next", managerFakeFixtureTSV(containers), 0o600)) +} + +func (fd *managerFakeDocker) invocations(t *testing.T) []string { + t.Helper() + raw, err := os.ReadFile(fd.logPath) + if os.IsNotExist(err) { + return nil + } + require.NoError(t, err) + return strings.Split(strings.TrimSpace(string(raw)), "\n") +} + +// failVerbs makes every later invocation of the listed docker verbs +// (stop, kill, rm, ...) exit 1 without output. +func (fd *managerFakeDocker) failVerbs(t *testing.T, verbs ...string) { + t.Helper() + require.NoError(t, os.WriteFile(fd.failPath, []byte(strings.Join(verbs, " ")+"\n"), 0o600)) +} + +// mutationsOf returns the rm/stop/kill invocations naming id. +func (fd *managerFakeDocker) mutationsOf(t *testing.T, id string) []string { + t.Helper() + var hits []string + for _, line := range fd.invocations(t) { + fields := strings.Fields(line) + if len(fields) == 0 { + continue + } + switch fields[0] { + case "rm", "stop", "kill": + if fields[len(fields)-1] == id { + hits = append(hits, line) + } + } + } + return hits +} + +const ( + sweepOwnID = "a1a1a1a1a1a1" + sweepOwnName = "mcpproxy-a-xk3q" + sweepCopiedID = "b2b2b2b2b2b2" // foreign container that copied the managed+instance labels + sweepCopiedName = "postgres" + sweepAbID = "c3c3c3c3c3c3" // a-b's canonical container; a-b is not configured + sweepAbName = "mcpproxy-a-b-wxyz" + sweepCustomID = "d4d4d4d4d4d4" // configured server's label on a user --name container + sweepCustomName = "custom" + sweepFakeInstance = "instance-under-test" +) + +// sweepFixture: every row carries the labels both sweeps select on. +func sweepFixture(instanceID string) []managerFakeContainer { + shared := func(extra map[string]string) map[string]string { + labels := map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": instanceID} + for k, v := range extra { + labels[k] = v + } + return labels + } + return []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Labels: shared(map[string]string{"com.mcpproxy.server": "a"})}, + {ID: sweepCopiedID, Name: sweepCopiedName, Labels: shared(nil)}, + {ID: sweepAbID, Name: sweepAbName, Labels: shared(map[string]string{"com.mcpproxy.server": "a-b"})}, + {ID: sweepCustomID, Name: sweepCustomName, Labels: shared(map[string]string{"com.mcpproxy.server": "a"})}, + } +} + +// newSweepManager builds a manager with servers `a` and `a/b` configured +// (disabled, never connected) and its main logger observed. +func newSweepManager(t *testing.T) (*Manager, *observer.ObservedLogs) { + t.Helper() + t.Setenv("CI", "") + mainCore, mainLogs := observer.New(zap.DebugLevel) + m := NewManager(zap.New(mainCore), &config.Config{}, nil, secret.NewResolver(), nil) + t.Cleanup(func() { m.shutdownCancel() }) + for _, name := range []string{"a", "a/b"} { + require.NoError(t, m.AddServerConfig(name, &config.ServerConfig{Name: name, Protocol: "http", URL: "http://127.0.0.1:1/mcp", Enabled: false})) + } + return m, mainLogs +} + +func mainLogMentions(logs *observer.ObservedLogs, needle string) []string { + var hits []string + for _, entry := range logs.All() { + if strings.Contains(entry.Message, needle) { + hits = append(hits, entry.Message) + continue + } + for k, v := range entry.ContextMap() { + if strings.Contains(fmt.Sprint(v), needle) { + hits = append(hits, entry.Message+" "+k+"="+fmt.Sprint(v)) + break + } + } + } + return hits +} + +// assertSweepTouchesOnlyOwned is the shared oracle for both sweeps: a's own +// canonical container is mutated; the three foreign rows are neither +// mutated nor named (id or name) anywhere in main.log. The rejected rows' +// labels are untrusted (they failed canonical ownership), so no owner can +// be attributed to them; per Spec 105 D8/D9 (codex round 11) a count that +// cannot be bound to a subject is not evidence, so listOwnedManagedContainers +// logs nothing about the rejected-row tally at all. +func assertSweepTouchesOnlyOwned(t *testing.T, fd *managerFakeDocker, mainLogs *observer.ObservedLogs) { + t.Helper() + assert.NotEmpty(t, fd.mutationsOf(t, sweepOwnID), "a's own container must still be cleaned up; invocations:\n%s", + strings.Join(fd.invocations(t), "\n")) + for _, foreign := range []struct{ id, name string }{ + {sweepCopiedID, sweepCopiedName}, {sweepAbID, sweepAbName}, {sweepCustomID, sweepCustomName}, + } { + assert.Empty(t, fd.mutationsOf(t, foreign.id), "foreign container %s (%s) was mutated", foreign.id, foreign.name) + assert.Empty(t, mainLogMentions(mainLogs, foreign.id), "foreign id %s written into main.log", foreign.id) + assert.Empty(t, mainLogMentions(mainLogs, foreign.name), "foreign name %s written into main.log", foreign.name) + } + skipped := mainLogs.FilterMessage("Skipping containers carrying the mcpproxy labels that no configured server canonically owns").All() + assert.Empty(t, skipped, "the rejected-row tally must not be logged: no owner can be attributed to it") +} + +func TestCleanupAllManagedContainers_TouchesOnlyCanonicallyOwned(t *testing.T) { + fd := installManagerFakeDocker(t, sweepFixture(core.GetInstanceID())) + m, mainLogs := newSweepManager(t) + + m.cleanupAllManagedContainers(context.Background()) + + assertSweepTouchesOnlyOwned(t, fd, mainLogs) + assert.Contains(t, fd.mutationsOf(t, sweepOwnID), "stop "+sweepOwnID) +} + +func TestForceCleanupAllContainers_TouchesOnlyCanonicallyOwned(t *testing.T) { + fd := installManagerFakeDocker(t, sweepFixture(core.GetInstanceID())) + m, mainLogs := newSweepManager(t) + + m.ForceCleanupAllContainers() + + assertSweepTouchesOnlyOwned(t, fd, mainLogs) + assert.Contains(t, fd.mutationsOf(t, sweepOwnID), "rm -f "+sweepOwnID) +} + +// ownerCountsFromLog reduces a set of owner-grouped count records (each +// carrying container_owner and count) to a map, failing the test if any +// record in entries is missing either field — an aggregate with no owner +// must never be among them (D8, codex round 10 finding 2). +func ownerCountsFromLog(t *testing.T, entries []observer.LoggedEntry) map[string]int { + t.Helper() + counts := make(map[string]int, len(entries)) + for _, entry := range entries { + fields := entry.ContextMap() + owner, _ := fields["container_owner"].(string) + require.NotEmpty(t, owner, "record %q carries no container_owner: %v", entry.Message, fields) + switch v := fields["count"].(type) { + case int64: + counts[owner] = int(v) + case int: + counts[owner] = v + default: + t.Fatalf("record %q carries no numeric count: %v", entry.Message, fields) + } + } + return counts +} + +// twoOwnerSweepFixture: two containers, canonically owned by two DIFFERENT +// configured servers (a and a/b — sanitised to a-b in the name, per +// dockernaming.SanitizeServerName). Used to prove a sweep's "found"/"force +// removing" count is owner-grouped rather than a single aggregate that +// cannot be bound to either subject. +func twoOwnerSweepFixture(instanceID string) []managerFakeContainer { + shared := func(server string) map[string]string { + return map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": instanceID, "com.mcpproxy.server": server} + } + return []managerFakeContainer{ + {ID: "e5e5e5e5e5e5", Name: "mcpproxy-a-yz12", Labels: shared("a")}, + {ID: "f6f6f6f6f6f6", Name: "mcpproxy-a-b-yz34", Labels: shared("a/b")}, + } +} + +// Codex round 10, docker finding 2 (D8): a sweep can select containers +// belonging to more than one configured server, so a bare aggregate count +// cannot be bound to a subject. The shutdown sweep's "Found ... to cleanup" +// and the emergency sweep's "Force removing ..." records must instead be +// owner-grouped: one record per Docker-read owner, each carrying its +// container_owner and count — never a bare count with no owner. +func TestSweepCounts_AreOwnerGrouped(t *testing.T) { + t.Run("cleanupAllManagedContainers", func(t *testing.T) { + installManagerFakeDocker(t, twoOwnerSweepFixture(core.GetInstanceID())) + m, mainLogs := newSweepManager(t) + + m.cleanupAllManagedContainers(context.Background()) + + found := mainLogs.FilterMessage("Found mcpproxy-managed containers to cleanup").All() + require.Len(t, found, 2, "one record per Docker-read owner, not one aggregate") + assert.Equal(t, map[string]int{"a": 1, "a/b": 1}, ownerCountsFromLog(t, found)) + }) + + t.Run("ForceCleanupAllContainers", func(t *testing.T) { + installManagerFakeDocker(t, twoOwnerSweepFixture(core.GetInstanceID())) + m, mainLogs := newSweepManager(t) + + m.ForceCleanupAllContainers() + + found := mainLogs.FilterMessage("Force removing managed containers").All() + require.Len(t, found, 2, "one record per Docker-read owner, not one aggregate") + assert.Equal(t, map[string]int{"a": 1, "a/b": 1}, ownerCountsFromLog(t, found)) + }) +} + +// Codex round 10, docker finding 1 (D9): HasDockerContainers drove the +// runtime/server shutdown wait and the "still running" report off the +// shared, copyable managed/instance labels alone — it must instead apply +// the same selection as the sweeps (listOwnedManagedContainers / +// core.ContainerOwnedByAny), so a foreign container that copies those +// labels, or one whose owning server was removed from config, is not +// reported as still running. +func TestHasDockerContainers_AppliesCanonicalOwnership(t *testing.T) { + t.Run("foreign container copying the shared labels - not reported", func(t *testing.T) { + installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepCopiedID, Name: sweepCopiedName, Running: true, + Labels: map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": core.GetInstanceID()}}, + }) + m, _ := newSweepManager(t) + + assert.False(t, m.HasDockerContainers(), "a foreign container carrying only the shared labels must not count") + }) + + t.Run("orphaned container (server no longer configured) - not reported", func(t *testing.T) { + installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepAbID, Name: sweepAbName, Running: true, + Labels: map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": core.GetInstanceID(), "com.mcpproxy.server": "a-b"}}, + }) + m, _ := newSweepManager(t) + + assert.False(t, m.HasDockerContainers(), "a-b is not configured; its container must not count") + }) + + t.Run("canonically owned and running - reported", func(t *testing.T) { + installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Running: true, + Labels: map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": core.GetInstanceID(), "com.mcpproxy.server": "a"}}, + }) + m, _ := newSweepManager(t) + + assert.True(t, m.HasDockerContainers(), "a's own canonically owned, running container must count") + }) + + t.Run("canonically owned but stopped - not reported", func(t *testing.T) { + installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Running: false, + Labels: map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": core.GetInstanceID(), "com.mcpproxy.server": "a"}}, + }) + m, _ := newSweepManager(t) + + assert.False(t, m.HasDockerContainers(), "a stopped container must not read as still running") + }) + + t.Run("docker unavailable - not reported", func(t *testing.T) { + fd := installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Running: true, + Labels: map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": core.GetInstanceID(), "com.mcpproxy.server": "a"}}, + }) + fd.failVerbs(t, "ps") + m, _ := newSweepManager(t) + + assert.False(t, m.HasDockerContainers()) + }) +} + +// With nothing configured, the sweeps mutate nothing at all. +func TestSweeps_NoConfiguredServers_MutateNothing(t *testing.T) { + fd := installManagerFakeDocker(t, sweepFixture(core.GetInstanceID())) + t.Setenv("CI", "") + m := NewManager(zap.NewNop(), &config.Config{}, nil, secret.NewResolver(), nil) + t.Cleanup(func() { m.shutdownCancel() }) + + m.cleanupAllManagedContainers(context.Background()) + m.ForceCleanupAllContainers() + + for _, id := range []string{sweepOwnID, sweepCopiedID, sweepAbID, sweepCustomID} { + assert.Empty(t, fd.mutationsOf(t, id), "container %s mutated with no configured owner", id) + } +} + +// fakeForceCleanupTarget stands in for a managed client on the +// disconnect-timeout path: it records whether the manager went through the +// ownership-checked removal instead of a bare `docker rm -f`, answers it +// with the configured verdict (owner/owned/err), and snapshots the main.log +// records written BEFORE the verdict existed. +type fakeForceCleanupTarget struct { + name string + containerID string + calls []string + + owner string + owned bool + err error + + mainLogs *observer.ObservedLogs + recordsAtCall []observer.LoggedEntry + recordsCaptured bool +} + +func (f *fakeForceCleanupTarget) GetConfig() *config.ServerConfig { + return &config.ServerConfig{Name: f.name} +} +func (f *fakeForceCleanupTarget) GetContainerID() string { return f.containerID } +func (f *fakeForceCleanupTarget) ForceRemoveTrackedContainerIfOwned(_ context.Context, id string) (string, bool, error) { + f.calls = append(f.calls, id) + if f.mainLogs != nil { + f.recordsAtCall = f.mainLogs.All() + f.recordsCaptured = true + } + return f.owner, f.owned, f.err +} + +// Codex round 3, docker finding 1: forceCleanupClient must route through the +// core ownership-checked removal (core.Client.ForceRemoveTrackedContainerIfOwned, +// tested against the fake docker in internal/upstream/core) — never a bare +// `docker rm -f `. +func TestForceCleanupClient_RoutesThroughOwnershipCheckedRemoval(t *testing.T) { + fd := installManagerFakeDocker(t, nil) + m, _ := newSweepManager(t) + target := &fakeForceCleanupTarget{name: "a", containerID: "f0e1d2c3b4a5968778695a4b3c2d1e0ff0e1d2c3b4a5968778695a4b3c2d1e0f"} + + m.forceCleanupClient(target) + + assert.Equal(t, []string{target.containerID}, target.calls, "the stored id must be handed to the ownership-checked removal") + assert.Empty(t, fd.invocations(t), "the manager itself must not exec docker on this path") + + target = &fakeForceCleanupTarget{name: "a"} + m.forceCleanupClient(target) + assert.Empty(t, target.calls, "no stored id, nothing to remove") +} + +// sweepFixtureOwnRunning is sweepFixture with a's own container still +// running, so the shutdown sweep's force-kill branch fires after `stop`. +func sweepFixtureOwnRunning(instanceID string) []managerFakeContainer { + rows := sweepFixture(instanceID) + for i := range rows { + if rows[i].ID == sweepOwnID { + rows[i].Running = true + } + } + return rows +} + +// recordNamesOwnContainer reports whether any field value of a main.log +// record carries a's container id (full or short) or name — independent +// of the key the record files it under. +func recordNamesOwnContainer(fields map[string]interface{}) bool { + return recordNamesAny(fields, sweepOwnID, shortContainerID(sweepOwnID), sweepOwnName) +} + +// recordNamesAny reports whether any string field value of a record carries +// one of needles. +func recordNamesAny(fields map[string]interface{}, needles ...string) bool { + for _, v := range fields { + s, ok := v.(string) + if !ok { + continue + } + for _, needle := range needles { + if strings.Contains(s, needle) { + return true + } + } + } + return false +} + +// Codex round 4, docker finding 1 (Spec 105 FR-007 / research D8, D9): the +// sweeps' intent records carried the read-back owner, but the OUTCOME +// records — stop succeeded/failed, force-kill intent/succeeded/failed, +// force-removal succeeded/failed — named the container's id or name with no +// `container_owner`, so a subject-evidence consumer had to withhold them. +// Every main.log record a sweep writes that names a container must carry +// the owner Docker reported for it, on the success and the failure branch +// alike. +func TestSweeps_EveryRecordNamingAContainerCarriesContainerOwner(t *testing.T) { + for _, tc := range []struct { + name string + fail []string + expected []string + }{ + { + name: "docker_succeeds", + expected: []string{ + "Stopping container", "Container stopped gracefully", + "Force killing container", "Container force killed", + "Force removing container", "Container force removed successfully", + }, + }, + { + name: "docker_fails", + fail: []string{"stop", "kill", "rm"}, + expected: []string{ + "Stopping container", "Graceful stop failed, will force kill", + "Force killing container", "Failed to force kill container", + "Force removing container", "Failed to force remove container", + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + fd := installManagerFakeDocker(t, sweepFixtureOwnRunning(core.GetInstanceID())) + if len(tc.fail) > 0 { + fd.failVerbs(t, tc.fail...) + } + m, mainLogs := newSweepManager(t) + + m.cleanupAllManagedContainers(context.Background()) + m.ForceCleanupAllContainers() + + for _, msg := range tc.expected { + require.NotEmpty(t, mainLogs.FilterMessage(msg).All(), "branch %q not exercised; invocations:\n%s", + msg, strings.Join(fd.invocations(t), "\n")) + } + naming := 0 + for _, entry := range mainLogs.All() { + fields := entry.ContextMap() + if !recordNamesOwnContainer(fields) { + continue + } + naming++ + assert.Equal(t, "a", fields["container_owner"], + "record %q names a's container without the read-back owner: %v", entry.Message, fields) + } + assert.GreaterOrEqual(t, naming, len(tc.expected), "every expected branch names the container") + for _, foreign := range []string{sweepCopiedID, sweepCopiedName, sweepAbID, sweepAbName, sweepCustomID, sweepCustomName} { + assert.Empty(t, mainLogMentions(mainLogs, foreign), "foreign %s written into main.log", foreign) + } + }) + } +} + +// Codex round 5, docker finding 1 (Spec 105 D8 subject-evidence rule): the +// disconnect-timeout path named the tracked id in its intent record before +// ownership was verified and in every outcome record without +// `container_owner`. A record written before the verdict exists must name +// no container id — the server only; the outcomes that name the id carry +// the owner the core read back at the mutation; a rejected or unverifiable +// container is never named by id at all. +func TestForceCleanupClient_NamesTheContainerOnlyWithOwnershipEvidence(t *testing.T) { + const trackedID = "f0e1d2c3b4a5968778695a4b3c2d1e0ff0e1d2c3b4a5968778695a4b3c2d1e0f" + needles := []string{trackedID, shortContainerID(trackedID)} + for _, tc := range []struct { + name string + owner string + owned bool + err error + wantNamed bool // some outcome record names the id (with the owner) + }{ + {name: "removed", owner: "a", owned: true, wantNamed: true}, + {name: "removal failed after verification", owner: "a", owned: true, err: errors.New("rm: exit status 1"), wantNamed: true}, + {name: "not owned any more", owned: false}, + {name: "ownership unverifiable", owned: false, err: errors.New("ps: exit status 1")}, + } { + t.Run(tc.name, func(t *testing.T) { + installManagerFakeDocker(t, nil) + m, mainLogs := newSweepManager(t) + target := &fakeForceCleanupTarget{name: "a", containerID: trackedID, owner: tc.owner, owned: tc.owned, err: tc.err, mainLogs: mainLogs} + + m.forceCleanupClient(target) + + require.True(t, target.recordsCaptured, "the ownership-checked removal must run") + for _, entry := range target.recordsAtCall { + assert.False(t, recordNamesAny(entry.ContextMap(), needles...), + "record %q names the tracked id before ownership was verified: %v", entry.Message, entry.ContextMap()) + } + named := 0 + for _, entry := range mainLogs.All() { + fields := entry.ContextMap() + if !recordNamesAny(fields, needles...) { + continue + } + named++ + assert.True(t, tc.owned, "record %q names the container although ownership was not established: %v", entry.Message, fields) + assert.Equal(t, tc.owner, fields["container_owner"], "record %q names the container without the read-back owner: %v", entry.Message, fields) + } + if tc.wantNamed { + assert.NotZero(t, named, "the outcome of a verified removal names the container with its owner") + } + assert.NotEmpty(t, mainLogMentions(mainLogs, "a"), "the server is still named on every path") + }) + } +} + +// Codex round 5, docker finding 2 (Spec 105 D9 moment-of-mutation rule): +// the sweeps stopped, killed and removed containers on the ownership their +// initial `docker ps` established. Another Docker client can rename or +// relabel a container between that listing and the mutation, so the name +// and label are re-read immediately before each stop/kill/rm: a container +// that no longer satisfies the predicate is left alone and the refusal is +// recorded without its id or name; one that still does is mutated, and the +// owner every record carries is the one read at mutation time. +func TestSweeps_ReverifyOwnershipAtMutationTime(t *testing.T) { + relabelled := func(name, owner string) []managerFakeContainer { + labels := map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": core.GetInstanceID()} + if owner != "" { + labels["com.mcpproxy.server"] = owner + } + return []managerFakeContainer{{ID: sweepOwnID, Name: name, Labels: labels, Running: true}} + } + sweeps := []struct { + name string + run func(m *Manager) + verb string + }{ + {name: "shutdown", run: func(m *Manager) { m.cleanupAllManagedContainers(context.Background()) }, verb: "stop"}, + {name: "emergency", run: func(m *Manager) { m.ForceCleanupAllContainers() }, verb: "rm -f"}, + } + arms := []struct { + name string + after []managerFakeContainer + wantOwner string // "" — the container must be left alone + }{ + {name: "relabelled foreign between listing and mutation", after: relabelled("postgres", "")}, + {name: "renamed to an unconfigured server's shape", after: relabelled("mcpproxy-a-b-xk3q", "a-b")}, + // A different container whose id extends the listed one: `--filter + // id=` is a prefix match, so only an exact full-id comparison + // tells it apart (codex round 6). + {name: "replaced by a container whose id extends the listed one", after: []managerFakeContainer{{ + ID: sweepOwnID + "ffffffffffffffffffffffffffffffffffffffffffffffffffff", Name: sweepOwnName, Running: true, + Labels: map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": core.GetInstanceID(), "com.mcpproxy.server": "a"}}}}, + // FR-007 instance-scoping (codex finding): same server label, same + // canonical name, but a DIFFERENT mcpproxy instance's id — a second + // live mcpproxy process on this host with a server also named `a`. + // This must be exactly as foreign as any other relabel, on both + // sweeps, or one instance's shutdown/emergency cleanup would stop or + // rm -f another live instance's container. + {name: "re-labelled to another live mcpproxy instance's id", after: []managerFakeContainer{{ + ID: sweepOwnID, Name: sweepOwnName, Running: true, + Labels: map[string]string{"com.mcpproxy.managed": "true", "com.mcpproxy.instance": "some-other-live-instance-id", "com.mcpproxy.server": "a"}}}}, + {name: "unchanged", after: relabelled(sweepOwnName, "a"), wantOwner: "a"}, + {name: "re-owned by another configured server", after: relabelled("mcpproxy-a-b-xk3q", "a/b"), wantOwner: "a/b"}, + } + for _, sw := range sweeps { + for _, arm := range arms { + t.Run(sw.name+"/"+arm.name, func(t *testing.T) { + fd := installManagerFakeDocker(t, relabelled(sweepOwnName, "a")) + fd.swapFixtureAfterNextPs(t, arm.after) + m, mainLogs := newSweepManager(t) + + sw.run(m) + + mutations := fd.mutationsOf(t, sweepOwnID) + if arm.wantOwner == "" { + assert.Empty(t, mutations, "a container whose ownership changed was mutated; invocations:\n%s", + strings.Join(fd.invocations(t), "\n")) + for _, line := range fd.invocations(t) { + verb := strings.Fields(line)[0] + assert.NotContains(t, []string{"stop", "kill", "rm"}, verb, "nothing may be mutated: %q", line) + } + for _, entry := range mainLogs.All() { + assert.False(t, recordNamesAny(entry.ContextMap(), sweepOwnID, shortContainerID(sweepOwnID), arm.after[0].Name), + "record %q names a container that is no longer owned: %v", entry.Message, entry.ContextMap()) + } + assert.NotEmpty(t, mainLogMentions(mainLogs, "no longer canonically owned"), "the refusal is recorded") + return + } + assert.Contains(t, mutations, sw.verb+" "+sweepOwnID, "an owned container is still swept") + named := 0 + for _, entry := range mainLogs.All() { + fields := entry.ContextMap() + if !recordNamesAny(fields, sweepOwnID, shortContainerID(sweepOwnID), arm.after[0].Name) { + continue + } + named++ + assert.Equal(t, arm.wantOwner, fields["container_owner"], + "record %q must carry the owner read at mutation time: %v", entry.Message, fields) + if server, ok := fields["server"]; ok { + assert.Equal(t, arm.wantOwner, server, "record %q attributes the container to the listing-time owner: %v", entry.Message, fields) + } + } + assert.NotZero(t, named, "the mutation is recorded with its subject") + }) + } + } +} + +// Codex round 8 (PR E), finding 2: verifyContainerHealthy decided health from +// `docker inspect ` alone. inspect answers by id regardless of +// name or label, so a container another Docker client relabelled or +// renamed after tracking still reported Running=true under the same id, +// and ForceReconnectAll treated it as healthy — skipping recovery for a +// container that is no longer canonically this server's. The health check +// must re-establish ownership through the same read+predicate +// ContainerMutator.Verify uses before trusting inspect: a container that +// fails ownership now is NOT healthy (recovery proceeds) and the health +// record names no id; a container ownership confirms is named, with +// container_owner from that same read. +func TestVerifyDockerContainerHealthy_ReverifiesOwnershipBeforeInspect(t *testing.T) { + t.Run("relabelled after tracking - unhealthy, no id recorded", func(t *testing.T) { + installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Running: true, + Labels: map[string]string{"com.mcpproxy.server": "z"}}, // foreign label now + }) + m, mainLogs := newSweepManager(t) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + healthy, err := m.verifyDockerContainerHealthy(ctx, sweepDocker, "a", sweepOwnID) + + assert.False(t, healthy, "a relabelled container must not be reported healthy") + require.Error(t, err) + for _, entry := range mainLogs.All() { + assert.False(t, recordNamesAny(entry.ContextMap(), sweepOwnID, shortContainerID(sweepOwnID)), + "record %q names the no-longer-owned container: %v", entry.Message, entry.ContextMap()) + } + }) + + t.Run("renamed after tracking - unhealthy, no id recorded", func(t *testing.T) { + installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: "custom", Running: true, // non-canonical name now + Labels: map[string]string{"com.mcpproxy.server": "a"}}, + }) + m, mainLogs := newSweepManager(t) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + healthy, err := m.verifyDockerContainerHealthy(ctx, sweepDocker, "a", sweepOwnID) + + assert.False(t, healthy, "a renamed container must not be reported healthy") + require.Error(t, err) + for _, entry := range mainLogs.All() { + assert.False(t, recordNamesAny(entry.ContextMap(), sweepOwnID, shortContainerID(sweepOwnID)), + "record %q names the no-longer-owned container: %v", entry.Message, entry.ContextMap()) + } + }) + + t.Run("docker read failure - unhealthy, no id recorded", func(t *testing.T) { + fd := installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Running: true, + Labels: map[string]string{"com.mcpproxy.server": "a"}}, + }) + fd.failVerbs(t, "ps") + m, mainLogs := newSweepManager(t) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + healthy, err := m.verifyDockerContainerHealthy(ctx, sweepDocker, "a", sweepOwnID) + + assert.False(t, healthy) + require.Error(t, err) + for _, entry := range mainLogs.All() { + assert.False(t, recordNamesAny(entry.ContextMap(), sweepOwnID, shortContainerID(sweepOwnID)), + "record %q names a container whose ownership read failed: %v", entry.Message, entry.ContextMap()) + } + }) + + t.Run("unchanged and running - healthy, record carries id and owner", func(t *testing.T) { + installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Running: true, + Labels: map[string]string{"com.mcpproxy.server": "a", "com.mcpproxy.instance": core.GetInstanceID()}}, + }) + m, mainLogs := newSweepManager(t) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + healthy, err := m.verifyDockerContainerHealthy(ctx, sweepDocker, "a", sweepOwnID) + + require.NoError(t, err) + assert.True(t, healthy) + var found bool + for _, entry := range mainLogs.All() { + fields := entry.ContextMap() + if !recordNamesAny(fields, sweepOwnID) { + continue + } + found = true + assert.Equal(t, "a", fields["container_owner"], "record %q must carry the owner read back: %v", entry.Message, fields) + } + assert.True(t, found, "the healthy verification is recorded with its subject") + }) + + t.Run("unchanged but stopped - unhealthy, record still carries id and owner", func(t *testing.T) { + installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Running: false, + Labels: map[string]string{"com.mcpproxy.server": "a", "com.mcpproxy.instance": core.GetInstanceID()}}, + }) + m, _ := newSweepManager(t) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + healthy, err := m.verifyDockerContainerHealthy(ctx, sweepDocker, "a", sweepOwnID) + + assert.False(t, healthy) + require.Error(t, err) + assert.Contains(t, err.Error(), "not running") + }) +} + +// TestVerifyDockerContainerHealthy_RunningComesFromTheVerifyReadAlone is +// codex round 16 finding 1: the health check re-verified ownership with one +// `docker ps` read (ContainerMutator.Verify) and THEN issued a second, +// separately-timed `docker inspect ` to decide running/status. Between +// the two, another Docker client can rename or relabel the container into a +// colliding server's namespace; the inspect would then report the +// NOW-FOREIGN container's state while ForceReconnectAll kept treating it as +// this server's (a TOCTOU gap the round-8 ownership re-verification did not +// close, because it only re-verified identity, not state). Running and +// status must come from the Verify read itself (ContainerRow.Running / +// .Status), never a follow-up command: the fixture is swapped to a +// relabelled, stopped container right after the fix's one `ps` call, so any +// further read of this id — if a second command were reintroduced — would +// see that swapped data and this test would catch it. +func TestVerifyDockerContainerHealthy_RunningComesFromTheVerifyReadAlone(t *testing.T) { + fd := installManagerFakeDocker(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Running: true, + Labels: map[string]string{"com.mcpproxy.server": "a", "com.mcpproxy.instance": core.GetInstanceID()}}, + }) + // After the fix's one `ps` read answers, swap to a relabelled, stopped + // container: any FURTHER read of this id would see foreign, not-running + // data. + fd.swapFixtureAfterNextPs(t, []managerFakeContainer{ + {ID: sweepOwnID, Name: sweepOwnName, Running: false, + Labels: map[string]string{"com.mcpproxy.server": "a-b"}}, + }) + m, mainLogs := newSweepManager(t) + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + healthy, err := m.verifyDockerContainerHealthy(ctx, sweepDocker, "a", sweepOwnID) + + require.NoError(t, err) + assert.True(t, healthy, "healthy must come from the one ps row Verify already has, not a second command that could see the swapped-in relabel/stop") + var found bool + for _, entry := range mainLogs.All() { + fields := entry.ContextMap() + if !recordNamesAny(fields, sweepOwnID) { + continue + } + found = true + assert.Equal(t, "a", fields["container_owner"], "record %q must carry the owner from the SAME read as the running state: %v", entry.Message, fields) + } + assert.True(t, found, "the healthy verification is recorded with its subject") + + var psCalls, inspectCalls int + for _, line := range fd.invocations(t) { + fields := strings.Fields(line) + if len(fields) == 0 { + continue + } + switch fields[0] { + case "ps": + psCalls++ + case "inspect": + inspectCalls++ + } + } + assert.Equal(t, 1, psCalls, "the health check must issue exactly one docker ps for this container, invocations:\n%s", strings.Join(fd.invocations(t), "\n")) + assert.Zero(t, inspectCalls, "the health check must never issue a separate docker inspect, invocations:\n%s", strings.Join(fd.invocations(t), "\n")) +} diff --git a/specs/105-agent-scope-hardening/gap-map.md b/specs/105-agent-scope-hardening/gap-map.md index 02d2293e6..7615c6d56 100644 --- a/specs/105-agent-scope-hardening/gap-map.md +++ b/specs/105-agent-scope-hardening/gap-map.md @@ -199,7 +199,7 @@ Port order (assessment): 3+6 → 7 redesigned → 1+2 → 4+5 → 10 → 8 re-do | B | **scope-cache-legacy-invalidation** | FR-001/002 | FR001-G1..G7 | `internal/cache/{authorization.go:77-95, manager.go:108-245, models.go:26-43}`, `internal/server/{cache_authz.go, mcp.go:5613-5690, content_forward.go:256}`, `internal/runtime/runtime.go:2226`, `internal/experiments/guesser.go:349`, tests (`cache/authorization_test.go:65-72,93-154` flipped; `manager_test.go:271`; `mcp_read_cache_authz_test.go:86,171`; `mcp_call_tool_direct_test.go:35-48`) | — | **A/C/E/G** all touch `mcp.go` in other regions (5613-5690 is B-only); low | | C | **scope-retrieve-tools** | FR-005 | FR005-G1..G5 | `internal/index/manager.go:105` (SearchScoped), `internal/index/bleve.go:458` (facet counts — A also edits bleve.go: sequence A first), `internal/server/{mcp.go:1705-2043, mcp_annotations.go:29, mcp_visibility.go}`, `internal/storage/manager.go:424` (unlimited/filterable stats), tests (`TestRetrieveTools_ScopeOracle`, goldens must stay byte-exact) | A (bleve.go ordering only) | **G** (`mcp_visibility.go` — C adds count primitive, G reorders `toolVisibleToSession :51-60`; different funcs), **H** (retrieve oracle consumes C) | | D | **scope-selectable-profile-predicate** | FR-003/004 | FR003-G1..G8 | `internal/server/{profile_tool.go:72-108,188-210, server.go:2320-2384}`, `docs/features/profiles.md:70-72`, tests (`profile_integration_test.go:201-233,647-703,775`; `profile_tool_test.go:344,356,397`; `profile_pin_enforcement_test.go:150,160`; new `mintScopedToken`) | — | **H** (`mintAgentToken` generalisation — D should introduce it); `server.go` untouched by others except none; low | -| E | **scope-log-attribution** | FR-007 | FR007-G1..G6 | `internal/logs/logger.go:321-547` (attributed reader; consider dedicated owner key at :385), `internal/server/mcp.go:5787-5803`, `internal/oauth/config.go:1686-1735` (stop via `server.logger`), `internal/upstream/core/docker.go:349-431,501-582` (label filter + `^mcpproxy--[a-z0-9]{4}$`), tests (`mcp_tail_log_scope_test.go:63-64,110-117` rewritten to stamped writers; `mcp_secret_redaction_test.go:288`; new logs/oauth/docker tests) | — | **B** (mcp.go other region); none else; low. Admin/REST/CLI readers (`server.go:3481`, `upstream_cmd.go:826`) stay whole-file | +| E | **scope-log-attribution** | FR-007 | FR007-G1..G6 | `internal/logs/logger.go:321-547` (attributed reader; consider dedicated owner key at :385), `internal/server/mcp.go:5787-5803`, `internal/oauth/config.go:1686-1735` (stop via `server.logger`), `internal/upstream/core/docker.go:349-431,501-582` (label filter + `^mcpproxy--[a-z0-9]{4}$`), tests (`mcp_tail_log_scope_test.go:63-64,110-117` rewritten to stamped writers; `mcp_secret_redaction_test.go:288`; new logs/oauth/docker tests) | — | **B** (mcp.go other region); none else; low. Admin/CLI readers (`upstream_cmd.go:826`) stay whole-file; the REST reader (`server.go:3481`) also stays whole-file but is NOT admin-only — see §8 | | F | **scope-direct-publication** | FR-008 (+US3.4 refusal text) | FR008-G1..G7 | `internal/server/{mcp_routing.go:143-148,156-311,390-465,1322-1326, mcp_direct_catalog.go:151-224,374-423, mcp_direct_scope.go:40-143, mcp_direct_callability.go:49-92, mcp_describe_direct.go:45-103}`, tests (`mcp_direct_skew_test.go:195-224,316-332,459-489` inverted; `mcp_direct_catalog_publish_test.go:65-101,155`; `mcp_direct_catalog_test.go:177-235`; new `__a`, deferred, reverse-flip, empty-raw-name) | A (callability reader) | **G** (both edit `mcp_direct_scope.go` filter chain and `mcp_direct_catalog.go` — G needs F's stamp + filter split; sequence F before G); `toolSetFingerprint` / `wireEntry` see the Meta stamp — route via filter chain | | G | **scope-refusal-shapes** | FR-010 | FR010-G1..G7 (G5 delivered by F) | `internal/server/{mcp.go:2262-2293,2473-2495, mcp_visibility.go:51-72, mcp_describe_tool.go:123-160, mcp_describe_direct.go:54-156, mcp_direct_catalog.go:227-268 (shadow canonical map), mcp_direct_scope.go:115-143 + mcp_routing.go:958-961 (tier out of WithToolFilter → list-only path), mcp_code_execution.go:1198-1229}`, `internal/jsruntime/runtime.go:383-410`, tests (`mcp_auth_scope_test.go:84`; `describe_plain_corpus_test.go` delta if any; `TestDescribeDirect_DisplayAndCanonicalNamespaceOverlap` admin control kept) | C (scoped-count/search primitive for describe not-found policy per roadmap:444), F | **F** as above; **A** (`mcp_code_execution.go` — A edits 980-1000/1251-1268, G edits 1198-1229; adjacent); **B** none | | H0 | **scope-regression-suite (part 1: FR-012 stored scripts)** — split early, independent | FR-012 | FR01x-G1..G3 | `internal/server/mcp_code_execution.go:50-74,490-497`, `internal/codescripts/codescripts.go:107-118,338-356` (or caller-kind branch in server), 3 goldens regenerated (+ narrow-diff assertion in `toolslist_snapshot_test.go`), `docs/code_execution/{overview,cookbook,troubleshooting,api-reference}.md`, `docs/features/agent-tokens.md` (invariant, surfaces, retained effects, warning), tests (`mcp_code_scripts_test.go:299` kept as admin control + agent case) | — | **A** (`mcp_code_execution.go` — H0 edits 50-74/490-497; A edits 980+/1251+; low) | @@ -243,4 +243,8 @@ Recommended merge order: **A → B ∥ D ∥ E ∥ H0 (parallel, disjoint) → C ## 8. Follow-ups / Spec 105 gaps (out of this spec's scope — recorded, not fixed) +**REST `GET /api/v1/servers/{id}/logs` serves the whole shared file to scoped agent tokens (PR E critique r1, findings C1.1 / C2.1).** `internal/httpapi/server.go:505-508` admits `mcp_agt_` tokens on REST and `:828` mounts `/logs` inside `scopedServerSubtree` (`scope_subtree.go:164-194`), which gates only the server NAME; the handler (`:3745`) calls `Server.GetServerLogs` (`internal/server/server.go:3454-3527`), a raw last-N-lines read with no attribution. So a token allowed `{a_b}` gets hidden `a/b`'s records, pre-105 callback records naming `a/b`'s bind host/port, and legacy foreign-container ids — the exact bytes `handleTailLog` withholds under FR-007. Spec 105 scopes the REST management API out (`spec.md` "Transport and caller scope"), and PR E's plan keeps REST/CLI whole-file, so PR E documents the gap (`docs/features/agent-tokens.md` no longer calls the endpoint an administrator reader) and does not change it. `POST /api/v1/diagnostics/fix` `show_last_server_logs` (`diagnostics_fixers.go:94`) reads whole-file too but sits behind `requireServerOp` (`server.go:887`), so agents are blocked there. *Proposed fix (REST entitlement work, #1166):* when `auth.IsScopedCaller(ctx)`, route the handler through `logs.ReadUpstreamServerLogTailAttributed` — the controller interface (`httpapi/server.go:146`) lacks `ctx`, so add a ctx-taking variant or an optional capability assertion as `EmitActiveProfileChanged` does — and add the `a_b`-token / hidden-`a/b` cell to `scope_round9_test.go` (the existing `TestServerLogs_ScopedTokenNeverSeesAnotherServersStderr` covers only the name gate). + +**Whole-file reader line cap (PR E critique r1, findings C1.2 / C2.10).** `logs.ReadUpstreamServerLogTail` (`internal/logs/logger.go:527-547`) uses bufio.Scanner's 64 KiB default, while `launcher.pumpLines` allows a 1 MiB child line, so one >64 KiB line makes the admin `tail_log`, REST and CLI reads fail with "token too long" while the scoped read (which skips over-long lines, 1 MiB cap) succeeds — a caller-kind divergence in the wrong direction. Pre-existing administrator bug; left untouched in PR E because `logger.go` being untouched is what validates the admin byte-parity oracle (`TestTailLog_CollidingLogFile_AdminWholeFileUnchanged`). Fix separately: give the whole-file reader the same skip-over-long behaviour (and `GetServerLogs` in `internal/server/server.go:3510`). + **REST replay dispatches with no tool gate (pre-existing; PR A round-3 review finding 3).** `POST /api/v1/tool-calls/{id}/replay` (`internal/httpapi/server.go` route registration ~`:904`, handler `handleReplayToolCall` ~`:4978-4992`) only asks `canSeeServer` for a scoped caller, then `Runtime.ReplayToolCall` (`internal/runtime/runtime.go` ~`:1350-1437`) dispatches `client.CallTool` directly. Nothing in that path consults the tool gate the MCP surfaces share (`evaluateExactToolGate`: quarantine, pending/changed lock, user Disabled, config deny, no-record-under-active-gate), resolves the identity (`resolveExactToolIdentity`: undiscovered / stale name), or checks the caller's tier against the target tool (`tierForAnnotations` + `HasPermission`). Consequences: a recorded call to a tool that is now pending / changed / user-disabled / config-denied — including a namespaced tool with no exact record — re-executes upstream; and a **read-only** agent token with server scope can replay a recorded **destructive** call, which is exactly the target-tier bypass FR-009 closes on the MCP surfaces. Spec 105 scopes the REST management API out (`spec.md` "Transport and caller scope": "The REST management API keeps its existing policy … and is out of scope here"; the REST entitlement work is #1166's), so PR A does not change replay. *Proposed fix (separate spec/PR):* before `client.CallTool`, route replay through `resolveExactToolIdentity` (refuse `Unresolved()`), `evaluateExactToolGate` (refuse `!callable()`, answer quarantine / lock / block as dispatch does) and `tierForAnnotations` + `HasPermission` for scoped callers — or dispatch through `handleCallToolVariant` with the recorded variant so the REST door reuses the whole MCP funnel; add the read-only-token-replays-destructive cell and the pending/changed/disabled/no-record cells with a counting upstream (zero upstream calls). Note the gate primitives live in `internal/server`; the replay lives in `internal/runtime` + `internal/httpapi`, so the fix either moves a gate reader into a shared package or has httpapi call the MCP proxy's funnel. diff --git a/specs/105-agent-scope-hardening/research.md b/specs/105-agent-scope-hardening/research.md index 127a069f9..68b37fb58 100644 --- a/specs/105-agent-scope-hardening/research.md +++ b/specs/105-agent-scope-hardening/research.md @@ -50,15 +50,21 @@ ## D8 — Log attribution key and unattributed lines (FR-007) **Decision**: **no new field** — the existing zap field `server=` written at `logger.go:385` is the ownership signal, so administrator log records and the whole-file reader are byte-identical to today (SC-005; astra r1 finding 10). Unforgeability comes from two rules, not from the key: -1. **Producer rule**: child-controlled text is only ever a zap *field value*, never the message — the real stderr path already does this (`monitoring.go:214`: `Info("stderr", zap.String("message", line))`), and PR E audits every `upstreamLogger.*` call site for a child-controlled message (none found in `internal/upstream/core` on HEAD; the audit is a test that greps for non-constant message arguments). -2. **Reader rule**: for the console encoder (`ts | LEVEL | caller | msg | {json}`) scan ` | {` boundaries **left to right** and accept the first whose suffix decodes as exactly one complete JSON object with no trailing bytes; read `server` from that object. Because child text is inside a JSON string *within* the fields object, a child line containing ` | {` cannot create an earlier boundary (it is to the right of the real `{`), and zap's escaping (`"`→`\"`, `\`→`\\`) guarantees the real object stays a single valid object that is accepted first — a child value can neither close the surrounding string early nor escape the encoder's closing quote (astra r3 verified: no bypass found). For the JSON encoder (`logger.go:152-155`) the whole line is the object. Lines with no accepted boundary (pre-stamp lines, torn fragments) are unattributed → withheld from scoped callers; administrators, REST and CLI readers keep the whole file. +1. **Producer rule**: child-controlled text is only ever a zap *field value*, never the message — the real stderr path already does this (`monitoring.go:214`: `Info("stderr", zap.String("message", line))`), and PR E audits every `upstreamLogger.*` call site for a child-controlled message (none found in `internal/upstream/core` on HEAD; the audit is a test that greps for non-constant message arguments). **PR E finding**: the launcher-pumped path (`connection_launcher.go` `loggerWriter`, fed one line per Write by `launcher.pumpLines`) writes the child's line as the *message* — `primary.Info(line)` — so the audit's `upstreamLogger.*` receiver filter does not see it. It is safe under rule 2 (the child's text sits left of the encoder's own ` | {` boundary; `TestReadUpstreamServerLogTail_AttributedOnly_ChildTextCannotForgeOwner/*/launcher_message` covers both encoders) provided a message never carries a line break; `loggerWriter.Write` now splits on `\n` before logging so no producer can start a fresh line with child-controlled text. **Superseded by codex round 2 (point 7)**: the launcher path now writes the child line as the `message` field of a constant-message record too, so every child path is under rule 1 and the audit has no child-text exception left. +2. **Reader rule**: for the console encoder (`ts | LEVEL | caller | msg | {json}`) take the **first** ` | {` boundary (codex round 1: never scan past a failed one) and accept the record only if the text in front of it starts with exactly one record header (`ts | LEVEL | `) and contains no second one, and the suffix decodes as exactly one complete JSON object with no trailing bytes that is not itself a JSON-encoder record (codex round 2, point 7); read `server` from that object. Because child text is inside a JSON string *within* the fields object, a child line containing ` | {` cannot create an earlier boundary (it is to the right of the real `{`), and zap's escaping (`"`→`\"`, `\`→`\\`) guarantees the real object stays a single valid object that is accepted first — a child value can neither close the surrounding string early nor escape the encoder's closing quote (astra r3 verified: no bypass found). For the JSON encoder (`logger.go:152-155`) the whole line is the object. Lines with no accepted boundary (pre-stamp lines, torn fragments) are unattributed → withheld from scoped callers; administrators, REST and CLI readers keep the whole file. 3. **Subject-evidence rule (historical records, `spec.md:130` "legacy records whose stamped identity conflicts with their subject are withheld")**: a record stamped `server=a` is attributable to `a` only if every subject it references is *canonically* established as `a`'s. Container records: going forward, every housekeeping record that names a container carries `container_owner=` read from the container's `com.mcpproxy.server` label (a new field on those records only — permitted, SC-005 names FR-007's container-housekeeping records as administrator-visible changes); a container record **without** `container_owner`, or with one ≠ the requested server, is withheld from scoped callers. The sanitised name is never evidence: `a/b` and `a-b` both sanitise to `a-b` (`dockernaming/naming.go:33-42`), so a historical `server=a/b` record naming `mcpproxy-a-b-wxyz` cannot be distinguished from a foreign one and is withheld (astra r5 finding 3). Callback records: withheld if their server/port fields name another server. Conservative by construction; administrators keep whole-file access. Astra r2 finding 16 replaced the earlier "last segment" rule, which mis-parsed a child line containing ` | ` on the stderr path. +4. **Reader robustness (PR E critique r1 finding C1.2)**: the attributed reader never turns a co-owner's output into a scoped-caller error. A rendered line longer than 1 MiB (the launcher pumps child lines up to 1 MiB, and the JSON encoder's escaping can push one past it) is consumed and treated as non-attributable — skipped, never `bufio.ErrTooLong` → tool error — so the response class is independent of hidden co-owners (SC-001). The whole-file reader keeps bufio's 64 KiB default (SC-005; a pre-existing administrator limit, recorded in gap-map §8). `container_count` is a container subject under rule 3 (a pre-105 sweep counted co-owners' containers), so the sweep's count record now carries `container_owner` too (critique r1 finding C1.5). +5. **Profile-scoped administrators (gap-map §7 open decision "#1224 R1-1", resolved in PR E)**: `handleTailLog` short-circuits to the whole-file reader for `authCtx == nil || authCtx.IsAdmin()` regardless of a URL profile scope. A profile bounds WHICH server an administrator may name (the existing profile gate above the read), not which records of it they see; an administrator on `/mcp/p/` whose profile excludes `a/b` but includes `a_b` therefore still reads `a/b`'s records from the shared file. Consistent with SC-005; pinned by `TestTailLog_URLProfileScope_AppliesToAllCallers`. Profiles are not a boundary here — do not mistake them for one. +6. **Producer audit scope (critique r1 findings C1.3 / C2.12)**: `TestUpstreamLoggerAudit_MessagesAreConstant` audits every zap level call on ANY receiver in `internal/upstream/core`, `internal/upstream/launcher` and `internal/oauth` (the `oauthLogger()` tee in `client.go` writes `internal/oauth`'s records into the per-server file), with one reviewed exception: `runAuthStrategies`'s literal `transportLabel` concatenation (the former `loggerWriter.writeLine` exception is gone since point 7). `TestReadUpstreamServerLogTail_AttributedOnly_LineBreakInMessageIsProducerGuarantee` pins that the reader cannot reject a forged line a producer lets through — the guarantee is the producer's. +7. **Codex round 2 (PR E)**. (a) *Torn fragments, fully*: round 1's first-boundary rule still handed a foreign console record torn INSIDE its message (before its own ` | {`) to the later writer, because the later record's boundary was then the first. The reader now requires the text in front of the accepted boundary to be exactly one record header (`consoleHeaderPattern`, the fixed `ts | LEVEL | ` shape of `getFileEncoder`) at offset 0 with no second header before the boundary: a tear inside the fragment's header leaves no header at offset 0, a tear anywhere after it leaves two. A fragment torn right after its caller separator in front of a JSON-encoder record (mixed-encoder file) leaves one header and a suffix that decodes — that suffix carries the JSON encoder's `level`/`ts`/`msg` keys, which a console fields object never does, and is rejected on that shape. Safe direction only: a launcher-era child line whose text carried a header as the MESSAGE is withheld from its own writer (`legacy_launcher_message` arm), never misattributed. (b) *Child output is a container subject when it names one*: Docker's own `docker run` name-conflict error carries the colliding container's name and full id — `a/b` and hidden `a-b` both generate `mcpproxy-a-b-*`, and on a suffix collision that is `a-b`'s container — and reaches `a/b`'s log as child text. Both child producers (`monitoring.go` stderr, `connection_launcher.go` loggerWriter — which now writes the child line as the `message` field of a constant-message `launcher` record instead of the message, an SC-005 FR-007 log-record change) stamp `child_output=true` (`logs.ChildOutputField`), and the reader treats a child-output record whose text matches `containerMentionPattern` (64-hex id, canonical `mcpproxy--<4>` name, or the "already in use by container" phrase) as a container subject requiring `container_owner` — which child output never carries, so it is withheld. Ordinary child lines carrying no container-shaped token, and ordinary (non-child) records with long hex values, are unaffected; a child line that merely looks container-shaped (`mcpproxy--<4 alnum>`) is withheld from scoped callers only — the safe direction, administrators keep it. (c) The pre-spawn "Docker isolation configured" record names a generated, unverified name and no longer carries `container_owner` (D9: the owner field is only ever the label read back); scoped callers lose that one record, administrators see it as before PR E. (d) Line cap measured on content: exactly 1 MiB is eligible, the terminator is not counted. (e) OAuth: the deprecated `StartCallbackServer` (no logger) resolved through `adoptLoggerLocked(nil)` = the last-installed server's tee, so `a` started that way recorded `loggerB.With(server=a)` and its start/stop records landed in `b`'s file (withheld from scoped `b` by the all-values-agree rule, but a routing violation for administrators); `subjectLoggerLocked` now records the zap global for a caller without a logger — never the manager logger. Pinned by `TestReadUpstreamServerLogTail_AttributedOnly_ConcatenatedTornFragmentWithheld` (six fragment shapes × two encoders), `..._ExactCapLineEligible`, `..._ChildOutputNamingContainerIsSubject`, `TestDockerRunCollision_LauncherStderr_NeverAttributedToRequester` (real launcher spawn against a fake docker whose `run` answers with the daemon's conflict message, real per-server file, real reader), `TestDockerRunCollision_StdioStderr_NeverAttributedToRequester`, `TestTailLog_DockerCollisionChildOutput_ForeignContainerWithheldFromScopedCaller` (tool surface), `TestSetupDockerIsolation_ConfiguredRecordCarriesNoOwnerForUnverifiedName`, `TestCallbackStop_DeprecatedStartNeverAdoptsAnotherServersLogger`. +8. **Codex round 3 (PR E)**. (a) *Child text re-emitted inside a connection error*: `monitorStderr` keeps every stderr line in the recent-stderr buffer, `initialize()` splices it into the error it returns (the initialize-timeout branch and `enrichTransportClosedError`), and `Connect` writes that error into the per-server log as the "Connection failed" record — the same foreign container name and id as the withheld stderr record, without its provenance. Both enrichments now return a `childOutputError` (unwraps to its cause), and `recordConnectionFailure` stamps the record `child_output=true` when `errors.As` finds one anywhere in the chain; ordinary connect errors are recorded as before. The same text reaches `connection_status.last_error` (tail_log AND `list`, plus the health detail derived from it) from the state manager; for scoped callers `logs.RedactContainerMentions` replaces every `containerMentionPattern` match with `[container]` — uniformly, whether or not a co-owner exists (FR-007) — and administrators keep it (SC-005). (b) *The container check searches only child-controlled values*: it ran over the whole serialized line, so the writer stamp of a server named like a canonical container (`mcpproxy-tenant-abcd`) matched and every one of its child-output records — even "ready" — was withheld. It now runs over the decoded `message` (stderr / launcher producers) and `error` (an error re-emitting the buffer) values of a `child_output=true` record only; stamp fields are never subjects. Pinned by `TestDockerRunCollision_ConnectionFailedRecord_NeverAttributedToRequester` (real stderr pump → buffer → enrichment → record → file → reader), `TestConnectionFailedRecord_OrdinaryChildStderrStaysAttributed`, `TestConnectionFailedRecord_NoChildOutputMarkerWithoutChildText`, `TestReadUpstreamServerLogTail_AttributedOnly_ContainerShapedServerNameIsNotASubject`, `TestTailLog_DockerCollisionConnectError_ForeignContainerWithheldFromScopedCaller` (tool surface: log record, tail_log `last_error`, `list` `last_error` + health). + **Rationale**: a per-record filter is cheaper and more robust than splitting files (which would still collide for `a/b` vs `a_b` after sanitisation), and reusing the existing field keeps administrator output unchanged. **Alternatives**: separate files keyed by raw name hash (breaks the documented file layout, tooling, and rotation); accept collisions (spec violation). Rejected. Known retained effect: two lumberjack sinks on one file can produce torn fragments; those are non-attributable and therefore withheld. ## D9 — Container ownership (FR-007 G5) -**Decision**: every cleanup path — `ensureNoExistingContainers`, the disconnect name-pattern fallback, **and the image-name fallback** (`docker.go:243-248,296-329`, reached when no owned container is found by name) — filters `docker ps` results by label `com.mcpproxy.server=` first, then by the regex `^mcpproxy--[a-z0-9]{4}$`; only containers matching **both** are removed or logged. The image-name fallback therefore can never touch a foreign container that merely shares the image (astra r2 finding 17). Pre-label containers and user-`--name` containers are left alone (they never were ours by the new rule). +**Decision**: every cleanup path — `ensureNoExistingContainers`, the disconnect name-pattern fallback, **and the image-name fallback** (`docker.go:243-248,296-329`, reached when no owned container is found by name) — filters `docker ps` results by label `com.mcpproxy.server=` first, then by the regex `^mcpproxy--[a-z0-9]{4}$`; only containers matching **both** are removed or logged. The image-name fallback therefore can never touch a foreign container that merely shares the image (astra r2 finding 17). Pre-label containers and user-`--name` containers are left alone (they never were ours by the new rule). The two paths that start from ONE known container rather than a listing — the id captured from the `--cidfile` of this server's own `docker run` (a user-configured direct `docker run --name custom` gets a cidfile but no label and no canonical name) and the exact tracked name — inspect the container (`docker ps -a --filter id=` / `name=^…$` with the ownership `--format`) and apply the same predicate before stopping it and before writing any record; `container_owner` is always the label value read back, never the requesting server's name (codex r1, PR E) — the pre-creation count record included (codex r3: `owned[0].Owner`, one value for every admitted row). **Whole-manager sweeps (codex r3)**: the manager's shutdown sweep (`com.mcpproxy.managed=true`) and emergency sweep (managed + this instance's id) selected by those shared, copyable labels alone, and the disconnect-timeout path ran `docker rm -f` on a client's stored id with no inspection. Every row a sweep selects must now ALSO be canonically owned by a *configured* server — label `com.mcpproxy.server=` AND canonical name for that same raw name (`core.ContainerOwnedByAny` over the manager's client registry; the strict both-halves rule of this decision, not label-or-name) — so a foreign container carrying copied labels, a user `--name` container carrying a configured server's label, and an orphan of a server no longer configured are neither mutated nor named: they are counted once at Warn. The disconnect-timeout path goes through `core.Client.ForceRemoveTrackedContainerIfOwned`, which re-establishes ownership of the tracked id at the moment of the `rm -f`. Administrator-visible (SC-005 FR-007 exception): orphans of removed servers are no longer swept and must be removed by hand (documented in docs/features/docker-isolation.md). Pinned by `TestCleanupAllManagedContainers_TouchesOnlyCanonicallyOwned`, `TestForceCleanupAllContainers_TouchesOnlyCanonicallyOwned`, `TestSweeps_NoConfiguredServers_MutateNothing`, `TestForceCleanupClient_RoutesThroughOwnershipCheckedRemoval`, `TestForceRemoveTrackedContainerIfOwned_AppliesOwnership`, `TestContainerOwnedByAny_Predicate`. **Moment of mutation and subject evidence (codex r5)**: the sweeps' selection `docker ps` is a snapshot, so every stop/kill/rm re-reads the container's name and label immediately before acting (`reverifyOwnedManagedContainer`, the rule the core already applied to a tracked id): a container renamed or relabelled in between is left alone and the refusal recorded without its id or name, and the `container_owner` every mutation record carries is the one read at that moment. The disconnect-timeout path names the tracked id only once the core's verdict exists — `ForceRemoveTrackedContainerIfOwned` hands back the owner it read — and never when the container was rejected or could not be verified. Pinned by `TestSweeps_ReverifyOwnershipAtMutationTime`, `TestForceCleanupClient_NamesTheContainerOnlyWithOwnershipEvidence`. **One helper, every mutation (codex r6)**: the round-5 rule was applied only in the manager; the core's image-fallback, name-pattern and pre-creation cleanups still stopped / `rm -f`'d the rows a listing returned, the kill after a failed `docker stop` was a second mutation with no read at all, and the docker-logs monitor read the cidfile itself on its wait timeout and named that id with an executable `docker logs` command. Now **every mutation re-verifies immediately before the command via one helper**: `core.ContainerMutator.Mutate(ctx, id, op, intent)` — used by the core client (`mutateOwnedContainer`) and the manager sweeps (`mutateOwnedManagedContainer`; the manager's private re-verify is gone) — re-reads that one container by `docker ps -a --no-trunc --filter id=`, accepts only the row whose FULL id equals the id it was given (every listing now runs `--no-trunc`, so a listed id is a full id and `--filter id=`'s prefix match cannot admit a replacement whose id extends it), re-applies the predicate (`ownsContainer` for one server, `ContainerOwnedByAny` for the sweeps), refuses with a record that names no id or name when it no longer holds or could not be read, and hands back the row read at that moment — the only source of `container_id` and `container_owner` on the caller's intent and outcome records (the per-row listing records are replaced by one count). Pre-creation cleanup re-verifies per ROW, so `a/b`'s second listed container relabelled to `a-b` while the first is removed is left alone. The docker-logs monitor names only the container `trackCidfileContainer` verified (id + owner kept on the client) and records nothing but the timeout otherwise. Pinned by `TestDockerMutations_ReverifyOwnershipAtMutationTime` (4 paths × relabelled / renamed / id-extending replacement / unchanged), `TestDockerCleanup_PreCreationReverifiesEachRow_SlashVsDashCollision`, `TestDockerStopEscalation_ReverifiesBeforeKill`, `TestMonitorDockerLogs_NamesOnlyAVerifiedContainer`, and the id-extending arm added to `TestSweeps_ReverifyOwnershipAtMutationTime`. **Rationale**: the label exists (`instance.go:57-65`) and is the only signal that disambiguates `a/b` from `a-b`; the regex guards against a foreign process re-using the label prefix. **Alternatives**: label only (weaker); rename containers with a hash (breaks scanner lookup MCP-2123, out of scope). Rejected. diff --git a/specs/105-agent-scope-hardening/tasks.md b/specs/105-agent-scope-hardening/tasks.md index 8f1af9b97..6dbd2d9e6 100644 --- a/specs/105-agent-scope-hardening/tasks.md +++ b/specs/105-agent-scope-hardening/tasks.md @@ -125,28 +125,28 @@ ### Failing tests -- [ ] T048 [US1] FR007-G1: `internal/logs` writers `a/b`, `a_b`; sentinel in `a/b`; attributed tail(`a_b`, 50) → no sentinel; a child stderr line `left | right | {"server":"a_b"}` written through the REAL stderr path (`monitoring.go:214`, `zap.String("message", line)`) by `a/b` is attributed to `a/b` and never to `a_b`; same cases under the JSON encoder (`logger.go:152-155`); server-level differential with/without `a/b`; admin `tail_log` bytes unchanged vs pre-feature capture — `internal/logs/logger_attributed_test.go` (new) + `internal/server/mcp_tail_log_scope_test.go` -- [ ] T049 [P] [US1] FR007-G2: `O_APPEND` an unstamped `LEGACY_PLAIN_LINE`; attributed read excludes it; admin whole-file read includes it; historical subject-evidence records (`spec.md:130`): a pre-upgrade `Removing existing container` record stamped `server=a` with `container_name=mcpproxy-a-b-wxyz` (shape of `docker.go:555-558`), the same-sanitised-name case — `server=a/b` with `container_name=mcpproxy-a-b-wxyz` and no `container_owner` (indistinguishable from hidden `a-b`'s container) — an ID-only `container_id` record, and a callback-stop record stamped `server=a` naming server `b`'s port — all withheld from the scoped reader, all present for admin; a post-upgrade record with `container_owner=a/b` IS returned to the `a/b` reader — `internal/logs/logger_attributed_test.go` -- [ ] T050 [P] [US1] FR007-G3: own1/foreign1/own2/foreign2 interleaved; attributed tail(`a_b`, 2) == `[own1, own2]`; server-level `lines_returned == 2` — `internal/logs/logger_attributed_test.go` + `internal/server/mcp_tail_log_scope_test.go` -- [ ] T051 [P] [US1] FR007-G4: two observer loggers, start callback servers `a`,`b`, `StopCallbackServer("a")` → stop record only in `a`'s observer, both start orders — `internal/oauth/callback_stop_logger_test.go` (new) -- [ ] T052 [P] [US1] FR007-G5: fake docker (`SetWellKnownDockerPathsForTest` + `ResetDockerPathCacheForTest`) `ps` → `deadbeef1234 mcpproxy-a-b-wxyz` (label `a-b`) + `mcpproxy-a-wxyz` (label `a`); server `a` connect/disconnect → foreign never rm/stop/kill'd nor logged; own removed; SECOND fixture: no owned container, empty known container ID, one foreign container on the SAME image → the image-name fallback (`docker.go:243-248,296-329`) touches nothing; unit matcher table `a` vs `a-b` vs `a/b` vs `A` — `internal/upstream/core/docker_ownership_test.go` (new) -- [ ] T053 [P] [US1] FR007-G6: forced-rotation shared-history and case-only (`A`/`a`, branch on FS case sensitivity) fixtures; admin outcomes recorded — `internal/logs/logger_attributed_test.go` +- [x] T048 [US1] FR007-G1: `internal/logs` writers `a/b`, `a_b`; sentinel in `a/b`; attributed tail(`a_b`, 50) → no sentinel; a child stderr line `left | right | {"server":"a_b"}` written through the REAL stderr path (`monitoring.go:214`, `zap.String("message", line)`) by `a/b` is attributed to `a/b` and never to `a_b`; same cases under the JSON encoder (`logger.go:152-155`); server-level differential with/without `a/b`; admin `tail_log` bytes unchanged vs pre-feature capture — `internal/logs/logger_attributed_test.go` (new) + `internal/server/mcp_tail_log_scope_test.go` +- [x] T049 [P] [US1] FR007-G2: `O_APPEND` an unstamped `LEGACY_PLAIN_LINE`; attributed read excludes it; admin whole-file read includes it; historical subject-evidence records (`spec.md:130`): a pre-upgrade `Removing existing container` record stamped `server=a` with `container_name=mcpproxy-a-b-wxyz` (shape of `docker.go:555-558`), the same-sanitised-name case — `server=a/b` with `container_name=mcpproxy-a-b-wxyz` and no `container_owner` (indistinguishable from hidden `a-b`'s container) — an ID-only `container_id` record, and a callback-stop record stamped `server=a` naming server `b`'s port — all withheld from the scoped reader, all present for admin; a post-upgrade record with `container_owner=a/b` IS returned to the `a/b` reader — `internal/logs/logger_attributed_test.go` +- [x] T050 [P] [US1] FR007-G3: own1/foreign1/own2/foreign2 interleaved; attributed tail(`a_b`, 2) == `[own1, own2]`; server-level `lines_returned == 2` — `internal/logs/logger_attributed_test.go` + `internal/server/mcp_tail_log_scope_test.go` +- [x] T051 [P] [US1] FR007-G4: two observer loggers, start callback servers `a`,`b`, `StopCallbackServer("a")` → stop record only in `a`'s observer, both start orders — `internal/oauth/callback_stop_logger_test.go` (new) +- [x] T052 [P] [US1] FR007-G5: fake docker (`SetWellKnownDockerPathsForTest` + `ResetDockerPathCacheForTest`) `ps` → `deadbeef1234 mcpproxy-a-b-wxyz` (label `a-b`) + `mcpproxy-a-wxyz` (label `a`); server `a` connect/disconnect → foreign never rm/stop/kill'd nor logged; own removed; SECOND fixture: no owned container, empty known container ID, one foreign container on the SAME image → the image-name fallback (`docker.go:243-248,296-329`) touches nothing; unit matcher table `a` vs `a-b` vs `a/b` vs `A` — `internal/upstream/core/docker_ownership_test.go` (new) +- [x] T053 [P] [US1] FR007-G6: forced-rotation shared-history and case-only (`A`/`a`, branch on FS case sensitivity) fixtures; admin outcomes recorded — `internal/logs/logger_attributed_test.go` ### Implementation -- [ ] T054 [US1] **No new field** (D8): keep the existing `server=` zap field at `internal/logs/logger.go:385`; add `ReadUpstreamServerLogTailAttributed(name, n)`: console encoder → scan ` | {` boundaries left to right and accept the first whose suffix decodes as exactly one complete JSON object (no trailing bytes); JSON encoder → whole line; match `server` exactly AND apply the subject-evidence rule (a container record is attributable only with `container_owner` == requested server; withhold every container record lacking it and every callback record naming another server — D8 rule 3), filter before taking the last *n*, withhold lines with no accepted boundary; whole-file reader untouched (admin records byte-identical) — `internal/logs/logger.go:321-547` -- [ ] T054a [P] [US1] Producer audit test: every `upstreamLogger.{Info,Warn,Error,Debug}(` call in `internal/upstream/core` passes a constant message literal (child-controlled text only as field values) — `internal/upstream/core/upstream_logger_audit_test.go` (new) -- [ ] T055 [US1] `handleTailLog` uses the attributed reader for scoped callers, whole-file for admins; `lines_returned` = filtered length — `internal/server/mcp.go:5787-5803` -- [ ] T056 [P] [US1] OAuth: `stopCallbackServerLocked` logs through the recorded `server.logger`; `StopCallbackServer` no longer calls `adoptLoggerLocked` on stop (nil-logger signature kept) — `internal/oauth/config.go:1686-1735` -- [ ] T057 [P] [US1] Docker: `ensureNoExistingContainers`, the disconnect name-pattern fallback AND the image-name fallback all filter by label `com.mcpproxy.server=` AND `^mcpproxy--[a-z0-9]{4}$`; foreign matches neither logged nor removed; every housekeeping record that names a container adds `zap.String("container_owner",