Skip to content

fix(index): normalize invalid UTF-8 source bytes - #4

Merged
davemoore- merged 1 commit into
elastic:mainfrom
simianhacker:fix-invalid-utf8-indexing
Sep 23, 2026
Merged

davemoore- merged 1 commit into
elastic:mainfrom
simianhacker:fix-invalid-utf8-indexing

Conversation

@simianhacker

@simianhacker simianhacker commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

What

Normalize undecodable source bytes to Unicode replacement characters before constructing Elasticsearch file and line documents. This preserves valid UTF-8 exactly while preventing lone surrogate code points from causing bulk indexing failures.

The issue was reproduced while indexing apache/grails-core at commit fa1e14756daba70da13bde862d94056489171080. Legacy-encoded bytes in grails-forge/grails-forge-api/src/test/resources/features_es.properties were decoded with surrogateescape, producing lone surrogates such as \udcf3, \udcae, and \udce1. Elasticsearch rejected those line documents with Invalid UTF-8: Illegal surrogate character, leaving the ref incomplete.

How to Verify

  1. Run uv run --locked pytest tests/test_documents.py -v.
  2. Run uv run --locked pytest tests/ -v.
  3. With local Elasticsearch credentials configured, index the exact repository revision that exposed the issue:
    uv run sourcerer index github/apache/grails-core \
      --commit fa1e14756daba70da13bde862d94056489171080 \
      --force \
      --retry-window 0s \
      --url "$ELASTICSEARCH_URL" \
      --api-key "$ELASTICSEARCH_API_KEY"
  4. Confirm indexing completes without Illegal surrogate character bulk errors. The verified local run completed with 14,026 files and 1,324,487 lines.

Breaking Changes

None

- replace undecodable source bytes with deterministic Unicode replacement characters
- keep valid UTF-8 and existing binary detection behavior unchanged
- cover normalized line documents and JSON-safe deterministic actions
@simianhacker
simianhacker marked this pull request as ready for review September 23, 2026 13:52
@davemoore-
davemoore- self-requested a review September 23, 2026 18:10
@davemoore-
davemoore- merged commit 1139fbc into elastic:main Sep 23, 2026
2 checks passed
@simianhacker
simianhacker deleted the fix-invalid-utf8-indexing branch September 23, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants