Skip to content

Read alias_map_file as UTF-8 instead of the locale encoding - #1656

Open
b84330808 wants to merge 1 commit into
dbcli:mainfrom
b84330808:fix-alias-map-file-encoding
Open

b84330808 wants to merge 1 commit into
dbcli:mainfrom
b84330808:fix-alias-map-file-encoding

Conversation

@b84330808

Copy link
Copy Markdown

Description

load_alias_map_file() opened the alias_map_file JSON with the locale encoding. On Windows that's a legacy code page (cp932 on Japanese Windows, cp1252 on English Windows), so a UTF-8 alias map with non-ASCII table names either stopped pgcli from starting with a UnicodeDecodeError (only FileNotFoundError and JSONDecodeError are caught there) or got decoded into mojibake that never matched the table name. A file saved with a BOM, which some Windows editors add, was rejected as "not valid json" on every platform.

The file is now opened in binary mode, so json detects the encoding itself (UTF-8 with or without a BOM, like any JSON file).

Example on Japanese Windows, with {"注文明細": "om"} as the alias map:

UnicodeDecodeError: 'cp932' codec can't decode byte 0x87 in position 7: illegal multibyte sequence

With the change the alias for 注文明細 is om. The new test fails without the change on Windows (both cases) and on Linux for the BOM case.

Checklist

  • I've added this contribution to the changelog.rst.
  • I've added my name to the AUTHORS file (or it's already there).
  • I installed pre-commit hooks (pip install pre-commit && pre-commit install).
  • I verified that my changes work as expected (this may include manually testing them in your local environment, or in other available environments). Cross this out if not relevant (for example, if you're making a documentation change).
  • Please squash merge this pull request (uncheck if you'd like us to merge as multiple commits)

I used Claude Code to write this change.

load_alias_map_file() opened the JSON file in text mode with the locale
encoding. On Windows that is a legacy code page such as cp932 or cp1252,
so a UTF-8 alias map with non-ASCII table names either raised
UnicodeDecodeError (not caught, so pgcli failed to start) or was decoded
into mojibake that never matched the table name. A BOM written by some
Windows editors was also rejected as invalid JSON on every platform.

Open the file in binary mode and let json detect the encoding, which
handles UTF-8 with or without a BOM regardless of the locale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@j-bennet

Copy link
Copy Markdown
Contributor

@b84330808 thanks for the PR! Looks good, but I see tests failing. Can you take a look?

@b84330808

Copy link
Copy Markdown
Author

Thanks for taking a look! The only failure is test_format_array_output_expanded, and it fails on main too (main's CI has been red since #1650): cli_helpers 2.15.2 now pads the CJK column name by display width. #1653 updates that test's expected output. The new alias_map tests pass on every Python version.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants