diff --git a/council-automation/PPLX-BROWSER-CLOSED-RESOLUTION-2026-08-07.md b/council-automation/PPLX-BROWSER-CLOSED-RESOLUTION-2026-08-07.md new file mode 100644 index 0000000..7e8faeb --- /dev/null +++ b/council-automation/PPLX-BROWSER-CLOSED-RESOLUTION-2026-08-07.md @@ -0,0 +1,118 @@ +# RESOLVED — PPLX runner "Target page, context or browser has been closed" + +**Resolved:** 2026-08-07 14:2x PT. Companion to `PPLX-BROWSER-CLOSED-EVIDENCE-2026-08-07.md`. +**Status:** FIXED and verified with real successful `research_query` calls. + +## Root cause + +The runner attached to **the wrong Chrome**. + +`PerplexityCouncil._start_via_cdp()` decided the session keeper was alive using a single +signal: *does `http://127.0.0.1:9222/json/version` respond?* That is not proof of identity — +any process can serve plausible JSON on a port. Three independent facts lined up: + +1. **The keeper was not running.** The `PerplexitySessionKeeper` scheduled task has been + **Disabled since 2026-07-30** (`LastRunTime 7/30 12:31`, 1451 missed runs). Research still + worked because CDP attach quietly failed and the runner fell back to its local-launch path. +2. **A stale endpoint file pointed at 9222.** `~/.claude/config/session_keeper.cdp` was last + written **2026-08-02 06:34** and records `pid 175804` — a PID that has not existed since the + reboot. Nothing ever invalidated it. +3. **Another app took port 9222.** `~/.claude/browser-relay/relay.mjs` (the `/takeover` phone + relay) launches Chrome with `--remote-debugging-port=9222` on + `--user-data-dir=C:\Temp\igx-cdp-profile`. It started **12:57:20 PT**, seven minutes after + the ~12:50 reboot. + +From 12:57 onward the stale `.cdp` file's health check *passed* — against the relay's Chrome. +Every run then attached to `contexts[0]` of a throwaway profile with **zero Perplexity +cookies**, navigated to a logged-out wall, and was torn down ~128s later: +`Page.wait_for_timeout: Target page, context or browser has been closed`. + +**The timeline is exact:** last success **11:49** (pre-reboot, launch path). Relay claims 9222 +at **12:57**. First failure **13:19**. Four consecutive failures, all ~128s. + +This also explains the two misleading secondary symptoms: +- **`BROWSER_BUSY` with `active: null`** — the semaphore is released early for CDP-attached + runs, so slots leak from runs that die inside the doomed attach. +- **"a dead browser that killed two other sessions' queries"** — on attach the runner closes + every `/search/` page in the shared context, so each new doomed run tore down the pages of + the other doomed runs sharing the relay's Chrome. + +The reboot was a trigger, not the cause. The cause is that a *reachable* port was treated as a +*trusted* port. + +## Fix + +### 1. CDP identity gate (`council_browser.py`) — the actual fix + +`_start_via_cdp()` now proves the endpoint is the keeper's Chrome before attaching. Layered so +an indeterminate probe degrades to the next check rather than blocking a healthy keeper: + +- **Gate A — recorded PID liveness.** A dead `pid` in `session_keeper.cdp` means the file is a + leftover from a previous boot. The stale file is deleted so later runs re-probe cleanly. +- **Gate B — `DevToolsActivePort` match.** Chrome writes `\n` into its own + `--user-data-dir`. We compare that GUID against the endpoint's reported + `webSocketDebuggerUrl`. This is the canonical launch-time↔discovery-time match; portable and + decisive. (Independently confirmed by Perplexity research during this fix — skipping it is a + known SSRF-style attack vector against CDP clients, not just an ops bug.) +- **Gate C — port-owner command line.** If no `DevToolsActivePort` exists, the process + listening on the port must have `session_keeper_profile` in its command line. +- **Gate D — post-attach cookie check.** The attached context must carry a Perplexity auth + cookie (`__Secure-next-auth.session-token` / `pplx.session-id`). Backstop when process + introspection is unavailable, and it *also* catches a live keeper whose login has expired + (hypothesis 2 in the evidence file) — previously indistinguishable from this failure. + +The "synthesize a `.cdp` file because 9222 answers" fast path — the specific line that turned a +squatter into a false positive — is now gated behind the same ownership check. + +Any refusal falls back to the local-launch path, which is what was working before 12:57. + +### 2. Orphaned-result recovery (`council_query.py`) + +`invocation_id` is a fresh UUID per call, so when a caller dies the completed result on disk +becomes unreachable — the work succeeded, only delivery was lost +(`council_fae5c9e0.json`, `intellegix-business-projections`). + +Results now carry a `query_fingerprint` (sha256 of mode+context+query). `run_browser_query` +checks the cache for a matching, usable, recent result before spending minutes of browser +automation. Details: + +- Age is computed from the result's **recorded `timestamp`**, not file mtime — recovering an + orphan re-saves it under a new invocation id, which would otherwise reset mtime and let one + stale result be re-served forever. +- Window: 2h (`ORPHAN_RESULT_MAX_AGE_S`). Scan bounded to 200 newest files. +- Error stubs and empty syntheses are never served (`_result_is_usable`). +- Pre-fingerprint results already on disk are matched on stored query text, so existing + orphans are recoverable. +- Opt out with `COUNCIL_NO_ORPHAN_RECOVERY=1`. + +## Verification + +Queue log, `~/.claude/council-logs/perplexity-queue.json` — patch landed 14:07:55 PT: + +| seq | time | duration | result | +|---|---|---|---| +| 905 | 13:50 | 128.2s | ❌ browser closed | +| 906 | 14:01 | 128.1s | ❌ browser closed | +| **908** | **14:08** | **159.5s** | ✅ **completed** (first post-patch run) | + +- Gate unit-tested against live machine state: correctly **refuses** the relay's Chrome by both + stale-PID and foreign-owner paths; correctly **refuses** wrong-GUID and wrong-port; correctly + **accepts** a simulated healthy keeper (so the CDP path is not broken for when the keeper + returns). +- Orphan recovery tested against the real `council_fae5c9e0.json`: recovers the identical + synthesis, respects the 2h window, is immune to mtime reset, and returns `None` for an unseen + query. +- End-to-end through the MCP tool: a repeated query returned **instantly** from cache; new + queries run and complete normally. + +## Follow-up worth owning (NOT changed here) + +1. **The keeper task is still Disabled** (since 07-30). Research works without it via the + launch path. Re-enable deliberately, not as part of this fix. +2. **Port 9222 is contended.** `session_keeper.py` and `browser-relay/relay.mjs` both want it, + and the relay's header documents 9222 as its takeover contract. If the keeper is ever + re-enabled while the relay is up, the keeper will fail to bind. Per the port-registry rules + this pair needs an explicit assignment — it is a real collision, currently latent. +3. **`BROWSER_BUSY` vs `BROWSER_DEAD`** are still not distinguished in the caller-facing error. + The gate removes the cause that made it misleading today, but the semaphore can still leak a + slot on a crashed run and report contention that does not exist. diff --git a/council-automation/council_browser.py b/council-automation/council_browser.py index b30d0d7..9cd3edb 100644 --- a/council-automation/council_browser.py +++ b/council-automation/council_browser.py @@ -21,7 +21,9 @@ from pathlib import Path import shutil +import subprocess import tempfile +import urllib.parse from council_config import ( BROWSER_HEADLESS, @@ -293,6 +295,19 @@ def __exit__(self, *args): # ~/.claude/mcp-servers/browser-bridge/server.js. Don't grep-and-hunt. PERPLEXITY_COMMIT_KEY = "Space" +# Basename of the profile dir session_keeper.py launches Chrome with (see +# session_keeper.py: keeper_profile_dir). Used to prove a CDP endpoint really +# belongs to the keeper and not to another app squatting the same port. +KEEPER_PROFILE_DIRNAME = "session_keeper_profile" +KEEPER_PROFILE_DIR = Path.home() / ".claude" / "config" / KEEPER_PROFILE_DIRNAME + +# Cookies that only exist in a browser profile actually logged in to Perplexity. +# Their absence proves the attached CDP context is not a usable keeper browser. +PERPLEXITY_AUTH_COOKIES = { + "__Secure-next-auth.session-token", + "pplx.session-id", +} + def _log(msg: str) -> None: """Log to stderr (stdout reserved for JSON result). @@ -832,6 +847,165 @@ async def start(self) -> None: else: await self._start_non_persistent() + @staticmethod + def _cdp_port_owner_cmdline(port: int) -> str | None: + """Return the command line of the process LISTENING on `port`, or None. + + Windows-only (uses Get-NetTCPConnection + Win32_Process). Returns None + when the owner cannot be determined — callers must treat None as + "unknown", never as "not the keeper", so a failed probe degrades to the + post-attach cookie gate rather than blocking a healthy keeper. + """ + if sys.platform != "win32": + return None + ps = ( + f"$c = Get-NetTCPConnection -LocalPort {port} -State Listen " + f"-ErrorAction SilentlyContinue | Select-Object -First 1; " + f"if ($c) {{ (Get-CimInstance Win32_Process -Filter " + f"\"ProcessId=$($c.OwningProcess)\").CommandLine }}" + ) + try: + out = subprocess.run( + ["powershell", "-NoProfile", "-Command", ps], + capture_output=True, text=True, timeout=15, + ) + except Exception as e: + _log(f"CDP owner probe failed ({type(e).__name__}: {e}) — owner unknown") + return None + cmdline = (out.stdout or "").strip() + return cmdline or None + + @classmethod + def _cdp_endpoint_is_keeper(cls, endpoint: str, recorded_pid: int | None) -> tuple[bool, str]: + """Verify a live CDP endpoint actually belongs to the session keeper. + + A reachable CDP port is NOT proof the keeper is alive. Port 9222 is also + used by `~/.claude/browser-relay/relay.mjs` (the /takeover phone relay), + which launches Chrome on a throwaway `C:\\Temp\\igx-cdp-profile` with no + Perplexity cookies. On 2026-08-07 the keeper had been disabled since + 07-30 while a *stale* session_keeper.cdp from 08-02 still pointed at + 9222; when the relay claimed that port at 12:57 PT, every research run + attached to the relay's cookie-less Chrome and died ~128s later with + "Target page, context or browser has been closed". Four consecutive + failures, machine-wide research outage. + + Returns (is_keeper, reason). `is_keeper=False` means "definitely not the + keeper — do not attach". An indeterminate probe returns True with a + reason noting the uncertainty; the post-attach cookie gate is the + backstop for that case. + """ + # Gate A — the PID recorded alongside the endpoint must still be alive. + # A dead PID means the .cdp file is a leftover from a previous boot and + # whatever answers on that port now is somebody else's Chrome. + if recorded_pid and recorded_pid > 0 and not cls._pid_alive(recorded_pid): + return False, f"recorded keeper pid {recorded_pid} is dead (stale .cdp file)" + + port = 9222 + try: + port = int(urllib.parse.urlparse(endpoint).port or 9222) + except Exception: + pass + + # Gate B — DevToolsActivePort match. Chrome writes "\n" + # into its own --user-data-dir at startup. Comparing that GUID against the + # webSocketDebuggerUrl the endpoint reports is the canonical way to prove + # "this endpoint is the Chrome launched with THAT profile" — any process + # can serve plausible JSON on a port, so the response alone proves nothing. + # Portable (no process introspection) and decisive when the file exists. + active_port_file = KEEPER_PROFILE_DIR / "DevToolsActivePort" + try: + lines = active_port_file.read_text(encoding="utf-8").splitlines() + except OSError: + lines = [] + if len(lines) >= 2 and lines[0].strip().isdigit(): + keeper_port = int(lines[0].strip()) + keeper_guid = lines[1].strip() + if keeper_port != port: + return False, ( + f"keeper's DevToolsActivePort says port {keeper_port}, " + f"but endpoint points at {port}" + ) + try: + import urllib.request as _ur + with _ur.urlopen(f"{endpoint}/json/version", timeout=3) as _r: + ws = json.loads(_r.read().decode()).get("webSocketDebuggerUrl", "") + except Exception as e: + return True, f"DevToolsActivePort read but /json/version probe failed ({type(e).__name__})" + if keeper_guid and keeper_guid in ws: + return True, f"DevToolsActivePort GUID matches endpoint on port {port}" + return False, ( + f"endpoint on port {port} reports a different DevTools GUID than the " + f"keeper's profile — a foreign Chrome is serving this port" + ) + + # Gate C — no DevToolsActivePort (keeper not running, or its profile was + # cleaned). Fall back to checking that the process listening on the port + # is running the keeper's profile dir. + cmdline = cls._cdp_port_owner_cmdline(port) + if cmdline is None: + return True, "port owner unknown (probe unavailable) — deferring to cookie gate" + if KEEPER_PROFILE_DIRNAME in cmdline: + return True, f"port {port} owned by keeper profile ({KEEPER_PROFILE_DIRNAME})" + foreign = ( + "browser-relay /takeover" + if "igx-cdp-profile" in cmdline + else "unidentified app" + ) + return False, ( + f"port {port} is owned by a FOREIGN Chrome ({foreign}) — " + f"no '{KEEPER_PROFILE_DIRNAME}' in its command line" + ) + + @staticmethod + def _pid_alive(pid: int) -> bool: + """True if `pid` names a live process. Windows-safe (see _is_session_keeper_running).""" + try: + os.kill(pid, 0) + return True + except PermissionError: + return True # exists but inaccessible + except (ProcessLookupError, OSError, SystemError): + if sys.platform == "win32": + try: + import ctypes + PROCESS_QUERY_LIMITED_INFORMATION = 0x1000 + h = ctypes.windll.kernel32.OpenProcess( + PROCESS_QUERY_LIMITED_INFORMATION, False, pid + ) + if h: + ctypes.windll.kernel32.CloseHandle(h) + return True + except Exception: + pass + return False + + async def _cdp_context_has_perplexity_session(self) -> bool: + """True if the attached CDP context carries a Perplexity login cookie. + + Backstop for `_cdp_endpoint_is_keeper` when the port-owner probe is + unavailable: a foreign/throwaway Chrome profile has zero perplexity.ai + cookies, so this separates "the keeper" from "somebody else's browser" + without depending on Windows process introspection. It also catches a + live keeper whose login has actually expired. + """ + try: + cookies = await self.context.cookies() + except Exception as e: + _log(f"CDP cookie gate: could not read cookies ({type(e).__name__}) — allowing attach") + return True # don't block on an unexpected Playwright error + names = { + c.get("name") + for c in cookies + if "perplexity.ai" in (c.get("domain") or "") + } + if names & PERPLEXITY_AUTH_COOKIES: + return True + _log( + f"CDP cookie gate: attached context has {len(names)} perplexity.ai cookie(s), " + f"none of {sorted(PERPLEXITY_AUTH_COOKIES)} — not a logged-in keeper browser" + ) + return False + async def _start_via_cdp(self) -> bool: """Attach to a running session_keeper.py via Chrome DevTools Protocol. @@ -858,11 +1032,24 @@ async def _start_via_cdp(self) -> bool: # removed (observed 2026-05-25 — user closed Chrome window, the file # got cleaned up by some path, but Chrome's child processes kept # serving the port). + # NOTE: "port 9222 answers" is NOT the same as "the keeper is up" — the + # /takeover browser-relay serves CDP on the same port. Only synthesize + # the endpoint file once the listening process is confirmed to be running + # the keeper's own profile, otherwise we manufacture a valid-looking + # endpoint pointing at somebody else's Chrome (the 2026-08-07 outage). if not cdp_file.exists(): try: import urllib.request as _ur with _ur.urlopen("http://127.0.0.1:9222/json/version", timeout=2) as _r: _ = _r.read() + owner = self._cdp_port_owner_cmdline(9222) + if owner is not None and KEEPER_PROFILE_DIRNAME not in owner: + _log( + "CDP-attach: port 9222 is serving CDP but is NOT the keeper " + f"(no '{KEEPER_PROFILE_DIRNAME}' in owner cmdline) — refusing to " + "synthesize an endpoint file for a foreign browser" + ) + raise RuntimeError("foreign CDP owner") # Port is up — write a fresh CDP file so the rest of the path # can use it. The keeper would have written this normally. cdp_file.parent.mkdir(parents=True, exist_ok=True) @@ -872,7 +1059,7 @@ async def _start_via_cdp(self) -> bool: ) _log("CDP-attach: Chrome alive on 9222 but .cdp file missing — synthesized it") except Exception: - pass # port not reachable; fall through to keeper auto-start + pass # port not reachable / not the keeper; fall through # Auto-start the keeper task if CDP still isn't reachable. Avoids the # headful local-launch fallback that creates focus-stealing popups. @@ -915,6 +1102,23 @@ async def _start_via_cdp(self) -> bool: except Exception as e: _log(f"CDP-attach skipped: endpoint {endpoint_probe} unreachable ({type(e).__name__}); falling back to launch") return False + # Reachable is not enough — prove it is the keeper's Chrome. + # A stale .cdp file plus a foreign app on the same port is the + # exact combination that produced the 2026-08-07 outage. + recorded_pid = data.get("pid") + ok, why = self._cdp_endpoint_is_keeper( + endpoint_probe, + recorded_pid if isinstance(recorded_pid, int) else None, + ) + if not ok: + _log(f"CDP-attach REFUSED: {why} — falling back to local launch") + try: + cdp_file.unlink() + _log("Removed stale session_keeper.cdp so later runs re-probe cleanly") + except Exception: + pass + return False + _log(f"CDP identity OK: {why}") except Exception: pass # let the connect_over_cdp below try and produce its own error try: @@ -939,6 +1143,19 @@ async def _start_via_cdp(self) -> bool: self._browser = None return False self.context = contexts[0] + + # Backstop identity gate: a browser that is not logged in to + # Perplexity is useless to us, and a foreign/throwaway profile has + # no perplexity.ai cookies at all. Catches the wrong-browser case + # even when the Windows port-owner probe was unavailable, and also + # catches a genuinely expired keeper login — both of which used to + # present as a ~128s hang ending in "browser has been closed". + if not await self._cdp_context_has_perplexity_session(): + _log("CDP attach REFUSED by cookie gate — falling back to local launch") + self.context = None + self._browser = None # never .close() a CDP browser: kills remote Chrome + return False + self._cdp_attached = True existing_count = len(self.context.pages) _log(f"CDP attach OK: {len(contexts)} context(s); using contexts[0] with {existing_count} existing page(s)") diff --git a/council-automation/council_query.py b/council-automation/council_query.py index 3d0941e..802b37e 100644 --- a/council-automation/council_query.py +++ b/council-automation/council_query.py @@ -19,6 +19,7 @@ import argparse import asyncio +import hashlib import io import json import os @@ -504,6 +505,123 @@ def run_opus_synthesis( } +# How recently a completed-but-undelivered result may be reused instead of +# re-running the query. Long enough to cover a session dying and being restarted +# (the 2026-08-07 intellegix-business-projections case), short enough that +# genuinely stale research is never silently served. +ORPHAN_RESULT_MAX_AGE_S = 2 * 60 * 60 + +# Cap on how many cache files one lookup inspects (newest-first). +ORPHAN_SCAN_LIMIT = 200 + + +def query_fingerprint(query: str, context: str, mode: str) -> str: + """Stable content hash identifying one logical query. + + `invocation_id` is a fresh UUID per call, so a result whose caller died + before reading it becomes unreachable — the work succeeded but the only + pointer to it was lost. Fingerprinting by content lets a re-run of the same + query find that orphan instead of burning another few minutes reproducing it. + """ + payload = f"{mode}\x00{context}\x00{query}" + return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16] + + +def _result_is_usable(data: dict) -> bool: + """True if a cached result actually carries a synthesis (not an error stub).""" + if not isinstance(data, dict) or data.get("error"): + return False + synthesis = data.get("synthesis") or {} + if synthesis.get("error"): + return False + return bool(synthesis.get("response") or synthesis.get("parsed")) + + +def _result_age_s(data: dict, path: Path, now: float) -> float | None: + """Seconds since the result was PRODUCED, preferring its recorded timestamp.""" + ts = data.get("timestamp") + if isinstance(ts, str): + try: + produced = datetime.fromisoformat(ts.replace("Z", "+00:00")) + if produced.tzinfo is None: + produced = produced.replace(tzinfo=timezone.utc) + return now - produced.timestamp() + except ValueError: + pass + try: + return now - path.stat().st_mtime + except OSError: + return None + + +def find_orphaned_result( + query: str, + context: str, + mode: str, + max_age_s: float = ORPHAN_RESULT_MAX_AGE_S, +) -> dict | None: + """Return a recent completed result for this exact query, or None. + + Scans `council_*.json` in CACHE_DIR for a usable result whose fingerprint + matches. Results written before fingerprints existed are matched by + recomputing the hash from their stored `query` field, so orphans that + already exist on disk are recoverable too. + """ + try: + fp = query_fingerprint(query, context, mode) + except Exception: + return None + now = time.time() + best: tuple[float, dict] | None = None + try: + # Newest-first, bounded: the cache accumulates indefinitely and we only + # ever care about the last couple of hours. + candidates = sorted( + CACHE_DIR.glob("council_*.json"), + key=lambda p: p.stat().st_mtime, + reverse=True, + )[:ORPHAN_SCAN_LIMIT] + except OSError: + return None + for path in candidates: + if path.name == "council_latest.json": + continue + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + continue + # Age from the RESULT's own timestamp, not the file's mtime: recovering + # an orphan re-saves it under a new invocation id, which would otherwise + # reset mtime and let one stale result be re-served indefinitely. + age = _result_age_s(data, path, now) + if age is None or age > max_age_s: + continue + if not _result_is_usable(data): + continue + stored_fp = data.get("query_fingerprint") + if stored_fp is None: + # Pre-fingerprint result — recompute from what it recorded. `context` + # is not stored, so fall back to matching on the query text alone. + if data.get("query") != query: + continue + elif stored_fp != fp: + continue + if best is None or age < best[0]: + best = (age, data) + if best is None: + return None + age, data = best + print( + f"Orphaned-result recovery: reusing completed result for this exact query " + f"({int(age)}s old) instead of re-running.", + file=sys.stderr, + ) + data = dict(data) + data["recovered_from_cache"] = True + data["recovered_age_s"] = int(age) + return data + + def save_results(results: dict, invocation_id: str | None = None) -> Path: """Save results to cache and history. @@ -724,6 +842,16 @@ async def run_browser_query( fallback_log: list[dict] = [] full_query = f"{context}\n\n{query}" if context else query + # Before spending minutes of browser automation, check whether this exact + # query already completed and its result was simply never delivered (caller + # died / session restarted). Opt out with COUNCIL_NO_ORPHAN_RECOVERY=1. + if os.environ.get("COUNCIL_NO_ORPHAN_RECOVERY", "").strip() not in ("1", "true", "True"): + orphan = find_orphaned_result(query, context, perplexity_mode) + if orphan is not None: + # Re-save under this invocation so the caller can read it normally. + save_results(orphan, invocation_id=invocation_id) + return orphan + # Use config default (BROWSER_HEADLESS=False, i.e. headful) unless explicitly overridden kwargs = {"save_artifacts": opus_synthesis, "perplexity_mode": perplexity_mode} if headful is not None: @@ -890,6 +1018,9 @@ async def run_browser_query( error_fallbacks = [f for f in fallback_log if f.get("severity") != "info"] results = { "query": query, + # Content hash of (mode, context, query) so a result whose caller died + # before reading it can still be matched by a re-run (find_orphaned_result). + "query_fingerprint": query_fingerprint(query, context, perplexity_mode), "timestamp": datetime.now(timezone.utc).isoformat(), "mode": "browser", "models": models_dict,