Skip to content

fix(toolshed): honour the find() -1 sentinel when rewriting generated stub headers - #2549

Open
LeSingh1 wants to merge 1 commit into
NVIDIA:mainfrom
LeSingh1:toolshed-stubgen-newline-sentinel
Open

fix(toolshed): honour the find() -1 sentinel when rewriting generated stub headers#2549
LeSingh1 wants to merge 1 commit into
NVIDIA:mainfrom
LeSingh1:toolshed-stubgen-newline-sentinel

Conversation

@LeSingh1

@LeSingh1 LeSingh1 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Stacking note: the one-line ci-nightly.yml change (adding toolshed/tests to the tooling test job) is byte-identical to the one in #2539 and #2548, so they merge cleanly in any order.

Problem

_normalize_stub_headers rewrites the OS path separator in a generated stub's first-line comment, so committed .pyi files are identical across platforms. It splits the file on the first newline:

newline = data.find(b"\n")
first_line = data[:newline] if newline != -1 else data
if not first_line.startswith(_HEADER_PREFIX) or b"\\" not in first_line:
    continue
stub.write_bytes(first_line.replace(b"\\", b"/") + data[newline:])

The first slice special-cases the -1 that bytes.find returns when there is no newline. The second does not — and data[-1:] is the file's last byte. For a stub that is a single line with no trailing newline, the rewritten header gets a duplicate of its own final character appended:

in : b'# This file was generated by stubgen-pyx from cuda_core\\cuda\\core\\x.pyx'
out: b'# This file was generated by stubgen-pyx from cuda_core/cuda/core/x.pyxx'
                                                                            ^ duplicated

Two things make this worth fixing rather than shrugging at:

  • A .pyi with no trailing newline is a shape this repo tolerates on purpose. .pyi is in the end-of-file-fixer exclude list in .pre-commit-config.yaml, so nothing adds one.
  • This wrapper is the stubgen-pyx-cuda-core pre-commit hook — a fixer. The corrupted byte is written straight back into the working tree, and the next git commit takes it.

The -1 handling one line above shows the sentinel was already known about; it just was not applied to both slices.

Fix

Slice the remainder with the same sentinel check:

rest = data[newline:] if newline != -1 else b""

No behavior change for any stub that has a newline, which is every stub in the tree today.

Tests

toolshed/run_stubgen_pyx.py had no tests. Added toolshed/tests/test_run_stubgen_pyx.py, which calls _normalize_stub_headers(tmp_path) directly (no stubgen-pyx binary, no subprocess, no CUDA):

  • the separator rewrite with a body and a trailing newline, with a body and no trailing newline, header-only with a newline, and header-only with no newline — asserting the result is byte-for-byte the input with only \/ changed;
  • files that must be left byte-identical: an already-POSIX header (with and without a trailing newline), a hand-written stub containing backslashes but no generated-header prefix, and an empty file.

Verification

Executed in full (pure Python, no GPU):

# with the fix
8 passed

# with toolshed/run_stubgen_pyx.py restored from upstream/main
FAILED test_separator_is_rewritten_without_touching_anything_else[header-only-no-newline]
1 failed, 7 passed

ruff check / ruff format --check clean on both changed Python files. Restore done with cp aside + git show upstream/main:<path> >; index verified clean before committing.

`_normalize_stub_headers` rewrites the OS path separator in a generated
stub's first-line comment. It splits the file on the first newline:

    newline = data.find(b"\n")
    first_line = data[:newline] if newline != -1 else data
    ...
    stub.write_bytes(first_line.replace(b"\\", b"/") + data[newline:])

The first slice special-cases the -1 that `bytes.find` returns when there is
no newline; the second does not. `data[-1:]` is the file's last byte, so for
a stub that is a single line with no trailing newline the rewritten header
gets a duplicate of its own final character appended:

    in : b'...stubgen-pyx from cuda_core\\cuda\\core\\x.pyx'
    out: b'...stubgen-pyx from cuda_core/cuda/core/x.pyxx'

A `.pyi` with no trailing newline is a shape this repo tolerates on purpose:
`.pyi` is excluded from the end-of-file-fixer hook in
.pre-commit-config.yaml, so nothing adds one. And because this wrapper runs
as a pre-commit *fixer* (the stubgen-pyx-cuda-core hook), the corrupted byte
is written back into the working tree and committed.

Slice the remainder with the same sentinel check.

Adds tests under toolshed/tests/ covering the rewrite with and without a
trailing newline, header-only stubs, and the cases that must be left
byte-identical (already-POSIX headers, hand-written stubs containing
backslashes, and empty files).
@copy-pr-bot

copy-pr-bot Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant