Skip to content

fix(sessions): decode git worktree paths as UTF-8 - #1225

Open
MohammedAlkindi wants to merge 1 commit into
anthropics:mainfrom
MohammedAlkindi:fix/worktree-paths-utf8
Open

fix(sessions): decode git worktree paths as UTF-8#1225
MohammedAlkindi wants to merge 1 commit into
anthropics:mainfrom
MohammedAlkindi:fix/worktree-paths-utf8

Conversation

@MohammedAlkindi

Copy link
Copy Markdown

git worktree list --porcelain emits paths as raw, unquoted UTF-8, but _get_worktree_paths decodes them with the locale default. On a stock Windows install that is cp1252.

Before (Windows 11, cp1252, real repo with a Cyrillic worktree):

GROUND TRUTH: 2 worktree path(s), decoded as UTF-8
   C:/tmp/wt-repro/main     exists_on_disk=True
   C:/tmp/wt-repro/сессия   exists_on_disk=True

SDK _get_worktree_paths(): 0 path(s)

plus a UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 traceback on stderr from a subprocess reader thread.

After, same repo: 2 of 2 paths returned, both resolve on disk, no traceback.

Two distinct failure modes, both fixed by the same line

cp1252 leaves exactly five bytes undefined — 0x81 0x8D 0x8F 0x90 0x9D:

  • If the path's UTF-8 contains one of them, the decode raises. The exception is raised inside subprocess's reader thread and never reaches the caller, which sees returncode == 0 with stdout is None; the existing if result.returncode != 0 or not result.stdout: return [] guard turns that into an empty list. One such worktree hides every worktree in the repo, ASCII ones included. This covers all hiragana (U+3040–307F encode as E3 81 xx), Cyrillic like U+0441 (D1 81), and U+200D ZWJ (E2 80 8D), which appears in every emoji-family sequence.
  • Otherwise the bytes decode to mojibake and the returned path simply does not exist on disk. Measured: éé, æ—¥, no exception.

So a café worktree degrades quietly while an worktree takes the whole repo's worktree list with it. Both are wrong; both are fixed by pinning the decode.

Reach: _get_worktree_paths has six call sites backing list_sessions, get_session_info, fork_session, rename_session and tag_session. This affects every legacy Windows codepage — cp1252, cp932, cp936, cp1251.

Why UTF-8 specifically

sys.getfilesystemencoding() is utf-8 on Windows (PEP 529) and on POSIX, so encoding="utf-8" is the codec os.fsdecode already uses for these paths. It also matches the surrounding code, which normalizes the same strings with unicodedata.normalize("NFC", ...) and uses explicit encoding="utf-8" for file I/O throughout sessions.py.

One trade-off I want to surface rather than leave for review

There is a narrow configuration where this is a regression: POSIX with a legacy 8-bit locale (e.g. LC_ALL=en_US.ISO-8859-1) and a non-ASCII worktree path. Today that decodes to mojibake which round-trips through os.fsencode, so the paths resolve and the lookup works; with a pinned UTF-8 decode it raises instead. It cannot surface to a caller — all six call sites already wrap this in except Exception, and UnicodeDecodeError is a ValueError — so the result degrades to [], which is the same outcome Windows has today. I verified nothing escapes through list_sessions or get_session_info.

errors="surrogateescape" would close that window, and I deliberately did not use it: it returns strings containing lone surrogates, which resolve on disk but raise on any subsequent encode (JSON, logging to a UTF-8 stream). Given these paths flow onward into session lookup, a loud failure that the callers already swallow seemed safer than a value that breaks somewhere else. Happy to switch if you'd rather have errors="surrogateescape", or encoding=sys.getfilesystemencoding(), errors="surrogateescape" which measured correct in every locale I tried.

Testing

_get_worktree_paths currently has no test coverage — every list_sessions test passes include_worktrees=False. The added test pins the decode via monkeypatch, so it is deterministic on every platform. That matters here: hosted CI runs a UTF-8 locale, including the windows-latest leg, so a purely behavioural test would pass without the fix and guard nothing.

Verified with the test in place and the source change reverted, it fails for the right reason:

>       assert captured.get("encoding") == "utf-8"
E       AssertionError: assert None == 'utf-8'
E        +  where ... = {'capture_output': True, 'check': False, 'cwd': '/repo', 'text': True, ...}.get('encoding')

Full suite, Windows, trio installed: 1477 passed, 14 skipped with the change; upstream main at 22795fe is 1476 passed, 14 skipped. Same run three times each, no variance. ruff check, ruff format --check (68 files) and mypy src/ scripts/ (33 files) are all clean before and after.

Note on #1188

#1188 rewrites this same call from subprocess.run( to executable.run(, so there will be a one-line textual conflict whichever lands first. It does not change the decoding, and executable.run forwards **kwargs, so encoding="utf-8" composes with it either way. Glad to rebase on top of it if you'd prefer that order.


AI assistance: I used Claude Code for this investigation. Verified myself on my own Windows 11 machine rather than taken on trust: the before/after worktree counts against a real git repo with a Cyrillic worktree; that the exception is swallowed into returncode 0 / stdout None; the per-character split between the raising and mojibake failure modes; that the new test fails with the fix reverted and passes with it; the full-suite, ruff and mypy results above; and that git worktree list --porcelain emits raw unescaped UTF-8 under both core.quotepath settings, which the whole report depends on.

git worktree list --porcelain emits paths as raw UTF-8, but the
subprocess call decoded them with the locale default. On a stock
Windows install that is cp1252, which misreads a non-ASCII worktree
path in one of two ways.

If the path's UTF-8 contains 0x81, 0x8D, 0x8F, 0x90 or 0x9D, the five
bytes cp1252 leaves undefined, the decode raises UnicodeDecodeError
inside subprocess's reader thread. That never reaches the caller,
which sees returncode 0 with stdout None, so the existing guard
returns an empty list and every worktree in the repo is lost, ASCII
ones included. Otherwise the bytes decode to mojibake and the
returned path does not resolve on disk.

Measured on Windows with a Cyrillic worktree: 0 of 2 entries returned
before, 2 of 2 after.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant