fix: file-type detection crash on non-UTF-8 text starting with "{" or "[" - #4447
Open
2sumtech wants to merge 1 commit into
Open
fix: file-type detection crash on non-UTF-8 text starting with "{" or "["#44472sumtech wants to merge 1 commit into
2sumtech wants to merge 1 commit into
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
No issues found across 4 files
Shadow auto-approve: would auto-approve. Fixes a crash in file-type detection when non-UTF-8 text starts with '{' or '['. Adds tests for the fix. No tradeoffs requiring human sign-off.
Re-trigger cubic
_TextFileDifferentiator._is_json parsed the raw byte stream with
json.load(), which auto-detects only the UTF-8/16/32 family, and caught
only json.JSONDecodeError. A plain-text file in any other character set
(e.g. cp1252/latin-1) whose first non-whitespace character is { or [ --
a log file or INI-style config -- raised UnicodeDecodeError out of
detect_filetype() and partition() instead of classifying as TXT.
Treat an undecodable payload as not-JSON after retrying with the
declared encoding, so an explicitly-declared legacy-encoded JSON file
still classifies as JSON. Regression of the encoding-aware detection
fixed in Unstructured-IO#707 (issue Unstructured-IO#705), reintroduced when detection was rewritten.
2sumtech
force-pushed
the
fix/json-detect-unicode
branch
from
August 24, 2026 21:44
ee29e99 to
e914534
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4446
Summary
detect_filetype()andpartition()raisedUnicodeDecodeErroron any non-UTF-8 plain-text file (cp1252, latin-1, …) whose first non-whitespace character is{or[— log files with[timestamp]lines, INI-style configs, or legacy-encoded JSON on systems where libmagic reports it astext/plain._TextFileDifferentiator._is_jsonparses the raw byte stream withjson.load(), which auto-detects only the UTF-8/16/32 family (the JSON interchange encodings, RFC 8259), and caught onlyjson.JSONDecodeError.Fix
Keep the
json.load()raw-stream parse first (preserving JSON-spec UTF-16/32 auto-detection exactly), and onUnicodeDecodeError/UnicodeErrorretry once decoding with the declaredencoding— so an explicitly-declared legacy-encoded JSON file still classifies as JSON — otherwise classify as not-JSON instead of raising. This is a regression of the encoding-aware detection from #707 (issue #705), reintroduced when detection was rewritten into the_TextFileDifferentiatorarchitecture.Testing
Two new parametrized
_is_jsoncases (cp1252 JSON-lookalike, latin-1 INI) plus a file-path test and a declared-encoding-JSON test — all four fail with the exactUnicodeDecodeErrorwithout the fix and pass with it. Fulltest_filetype.py: 277 passed.ruff check+ruff format --checkclean on touched files. CHANGELOG entry and version bump (0.26.5 — note: open PR #4444 claims 0.26.4; whichever merges second has a trivial adjacent-line CHANGELOG/version conflict, happy to renumber) included per repo convention.Disclosure: prepared with AI assistance (Claude Code); I reviewed the change and take responsibility for it.