Skip to content

fix: report local scan failures without connectivity advice - #154

Closed
mldangelo wants to merge 1 commit into
openai:mainfrom
mldangelo:fix/local-failures-misreported
Closed

fix: report local scan failures without connectivity advice#154
mldangelo wants to merge 1 commit into
openai:mainfrom
mldangelo:fix/local-failures-misreported

Conversation

@mldangelo

Copy link
Copy Markdown
Contributor

Fixes #36.

Scope correction — two of the three reported reproductions are already fixed

I re-ran all three reproductions from the issue against current main (9c7634b) before writing anything. The issue was filed against f22d4a36, and the network_error/timeout branches of scanFailureMessage now already fall through to cliErrorMessage:

Reported reproduction Current main
Path target does not exist: src/network/client.ts already correct — shows the real message
unknown Git ref: origin/connection-fix already correct — shows the real message
EACCES: permission denied, mkdtemp '/tmp/...' still broken — reports a credentials failure

So this PR addresses the one surviving case, which is the unauthorized / forbidden / rate_limited branches — the three that still discard the original message. The practical trigger is permission denied, the standard strerror for EACCES, which the forbidden pattern matches verbatim:

/\b403\b|\bforbidden\b|\bpermission denied\b|.../iu

A read-only TMPDIR therefore surfaces as "The stored ChatGPT credentials cannot access the configured model. Use an account or API key with model access." — sending the user to reconfigure credentials for a chmod problem, with the real path never printed.

What this changes

Classification is skipped for failures that cannot originate from the model transport, so they keep their own message:

  • InvalidTargetError — path and Git ref validation
  • OutputDirectoryError — output directory checks
  • ConfigurationError — local config
  • PluginPythonUnavailableError — interpreter discovery
  • filesystem syscall errno codes (EACCES, EPERM, ENOSPC, EROFS, ENOENT, …)

Network errno codes (ECONNRESET, ENOTFOUND, ETIMEDOUT, …) are deliberately excluded from that set — they are genuine transport failures and must keep classifying. The set is an explicit allowlist rather than a /^E[A-Z]+$/ test for exactly that reason.

Why I did not take the issue's alternative suggestion

The issue offers a second option: "always include the original message alongside the advice so no failure is reported without its cause."

I implemented that first, and the existing suite caught it. tests-ts/cli-authentication.test.ts"identifies overriding API keys in noninteractive scan auth failures" deliberately asserts that upstream text must not reach stderr:

["403 model access denied for org-private", "cannot access the configured model"],
...
expect(stderr.text()).not.toContain("org-private");

Upstream authentication and authorization errors can name the organization or project, and suppressing that is intentional. Appending the cause would leak it into both stderr and the JSON error field. So the advice branches still replace the underlying text, and I added a comment recording why, since the next person to read this will have the same idea I did.

That trade-off means a genuinely misclassified transport error still loses its text. Narrowing the forbidden pattern itself (for instance requiring permission denied to co-occur with model/auth context) would close that too, but it changes retry-versus-fatal behavior in the event stream and felt out of scope here.

Also out of scope

The issue notes the same classifier gates retry-versus-fatal in the api.ts event-stream error branch. Those messages come from the Codex transport, where classification is the intended behavior, and changing it risks turning fatal auth failures into retry loops. Worth a separate look, not folded in here.

Testing / QA instructions

Baseline before this branch: 470 pass / 6 skip / 0 fail. After: 472 pass / 6 skip / 0 fail (two added tests).

cd sdk/typescript          # run pnpm from here, not the repo root
CI=true pnpm install --frozen-lockfile
CI=true pnpm run types
CI=true pnpm run test
CI=true pnpm run format
CI=true pnpm run build

Confirm the test reproduces the bug

cd sdk/typescript
cp src/cli.ts /tmp/fixed.ts
git checkout main -- src/cli.ts
CI=true bun test --timeout 30000 ./tests-ts/cli.test.ts -t "local input and filesystem failures"
#   expect 1 fail, showing:
#   "The stored ChatGPT credentials cannot access the configured model. Use an ..."
cp /tmp/fixed.ts src/cli.ts
CI=true bun test --timeout 30000 ./tests-ts/cli.test.ts -t "local input and filesystem failures"   # 1 pass

New coverage

  1. reports local input and filesystem failures without connectivity advice — drives real main() through the cli-fixtures harness with six failures: EACCES from mkdtemp, EPERM from mkdir, an OutputDirectoryError, an InvalidTargetError naming a src/403/ path, an InvalidTargetError naming an origin/forbidden-paths ref, and a PluginPythonUnavailableError. Each asserts the real message reaches stderr and that no credential, authentication, or rate-limit advice appears.

    The 403 and forbidden cases are deliberate: they cover the other two words in the forbidden pattern that a repository path or branch name can plausibly contain, not just permission denied.

  2. keeps model authorization advice for genuine transport failures — the guard against over-correcting. A CodexSecurityError carrying 401 invalid API key for org-private / 403 model access denied for org-private must still produce advice and must still not leak org-private.

Regression surface

The pre-existing tests that pin this area are unmodified and still pass — cli-authentication.test.ts "identifies overriding API keys…" (the org-private suppression above) and the three cli.test.ts cases asserting sqlite/database/sandbox errors do not claim a network failure.

classifyConnectionFailure matches bare words anywhere in a message, and
the unauthorized, forbidden and rate-limited branches of
scanFailureMessage replace the message entirely with fixed advice. A
local failure whose text merely contained one of those words was
therefore reported as a credential problem, and its own text reached
neither stderr nor the JSON error field.

The reported trigger is "permission denied", the standard strerror for
EACCES: a read-only TMPDIR surfaced as "The stored ChatGPT credentials
cannot access the configured model."

Skip classification for failures that cannot originate from the model
transport -- InvalidTargetError, OutputDirectoryError,
ConfigurationError, PluginPythonUnavailableError, and filesystem syscall
errno codes -- so they keep their own message. Network errno codes are
deliberately excluded from that set because they are genuine transport
failures.

Advice still replaces the underlying text for real transport failures.
Upstream authentication and authorization errors can name the
organization or project, and cli-authentication.test.ts pins that they
must not reach stderr.

Fixes openai#36

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ec01cfe904

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/cli.ts
error instanceof InvalidTargetError ||
error instanceof OutputDirectoryError ||
error instanceof ConfigurationError ||
error instanceof PluginPythonUnavailableError

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recognize PluginBootstrapError as local

When --plugin names an invalid local path containing 403 or forbidden (for example, a regular file at /tmp/403), resolvePluginPath throws the base PluginBootstrapError at runtime.ts:1007-1009. This guard recognizes only its PluginPythonUnavailableError subclass, and the wrapper has no syscall code, so classifyConnectionFailure interprets the path as a model authorization failure and discards the actionable plugin-path message. Treat the full PluginBootstrapError family as local.

Useful? React with 👍 / 👎.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator

Thanks for the thoughtful work on making local scan failures easier to understand. The substantive error handling from this branch is already on main, so there isn't a reason to carry the older PR forward. There is a small remaining PluginBootstrapError edge that we can handle in a separate focused fix. We'd be happy to review future contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local input and filesystem errors are reported as model-service connectivity or authorization failures

2 participants