From ea57fee757b3c20923d445fa4f4669a68298243a Mon Sep 17 00:00:00 2001 From: JeanExtreme002 Date: Sun, 6 Sep 2026 03:07:36 -0300 Subject: [PATCH 1/2] ci: add a macOS cell so Codecov covers the third backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The merged Codecov view and codecov.yml both describe the report as covering "win32 + linux + macos", but macOS left the matrix in 041bfc0 and the macos backend has not been measured since — so the combined number counted that whole backend as uncovered. macOS comes back as its own job rather than a third row in the matrix: the Python-version sweep stays on Linux and Windows exactly as it is today, and this cell exists only to measure the backend. The one deliberate difference from the matrix is `.[dev,speed]`. That is what makes a macOS cell viable rather than a preference. A macOS process exposes ~390 GiB of sparse address space across ~145 regions — two orders of magnitude more than a Linux one — and the scan tests walk all of it. Measured on an M-series Mac with the identical pytest invocation: with NumPy 79s 562 passed 87% total without NumPy 681s 498 passed 85% total An 8.5x gap on 10 cores, and macos-latest gives a job 3. That is what made macOS look like it hung here before. At 79s this lands in the same range as the existing cells (2.8-4.5 min including setup), which is the whole requirement for it. Nothing is lost by letting this cell take the vectorized path: all eight cells of the matrix still run the pure-Python scan path, and NumPy in fact *enables* 64 tests that skip without it. The backend ends up well covered — functions.py 81%, process.py 91%, libsystem.py and types.py 100%. timeout-minutes: 20 caps the job so a pathological scan can never sit at the 6h default, which is how the original hang presented. --- .github/workflows/python-package.yml | 48 ++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 6523d42..6a7660d 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -133,6 +133,54 @@ jobs: token: ${{ secrets.CODECOV_TOKEN }} fail_ci_if_error: false + # macOS gets a single cell instead of a row in the matrix above. Its only + # purpose is to measure the `macos` backend so the merged Codecov report + # covers all three platforms; the Python-version sweep stays on Linux and + # Windows, where it costs a fraction of the wall clock. + # + # Unlike the matrix, this job installs `.[dev,speed]` (NumPy). That is not a + # preference — it is what makes a macOS cell viable at all. A macOS process + # exposes ~390 GiB of sparse address space across ~145 regions, two orders of + # magnitude more than a Linux one, and the scan tests walk all of it. Measured + # on an M-series Mac, this exact command takes ~82s with NumPy and 681s + # without: an 8x gap that a 3-core runner only widens, which is what made + # macOS look like it hung here before (it was removed in 041bfc0). + # The pure-Python scan path loses nothing — all eight cells of the matrix + # above still exercise it. + build-macos: + needs: lint + runs-on: macos-latest + # Hard ceiling so a pathological scan can never sit at the 6h default. + timeout-minutes: 20 + steps: + - uses: actions/checkout@v4 + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + # No Qt system-library step here: the PySide6 macOS wheel bundles its own + # Qt frameworks, unlike the Linux one. + - name: Install dependencies (with NumPy speed extra) + run: | + python -m pip install --upgrade pip + pip install -e ".[dev,speed]" + - name: Test with pytest + env: + QT_QPA_PLATFORM: offscreen + # Same conservative coverage gate as the main matrix (see that step's note). + run: | + pytest tests -v -s -x --cov=PyMemoryEditor --cov-report=term --cov-report=xml --cov-fail-under=60 + - name: Upload coverage to Codecov + # Flag mirrors the matrix format (`runner.os` is "macOS" here) so Codecov + # groups this cell alongside the Linux and Windows uploads. + if: always() + uses: codecov/codecov-action@v5 + with: + files: ./coverage.xml + flags: macOS-py3.12 + token: ${{ secrets.CODECOV_TOKEN }} + fail_ci_if_error: false + # Isolated job for the optional NumPy-accelerated scan fast path (the # `[speed]` extra). The matrix above deliberately runs WITHOUT NumPy so the # pure-Python scan path stays covered; this single Linux job installs From b9dafd251eaa8dc475b5d00ff5f4f01848188e85 Mon Sep 17 00:00:00 2001 From: JeanExtreme002 Date: Sun, 6 Sep 2026 04:11:22 -0300 Subject: [PATCH 2/2] fix(macos): let a sweep skip a range the kernel refuses to read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The new macOS cell failed on its first run: a plain search_by_value(int, ...) over the current process died with MachReadError: mach_vm_read_overwrite failed: (os/kern) protection failure (kr=2) on a 22 MB region that reported VM_PROT_READ and Shared=False, so default_scan_filter had cleared it. Aborting there loses every region that came after it, which is the whole rest of the scan. KERN_PROTECTION_FAILURE stays out of _PAGE_GONE_KRS, where a test deliberately pinned it, because that tuple feeds three callers and one of them is not a sweep. search_values_by_addresses reads addresses the caller named, and process.scanning.iter_values_for_addresses swallows a "transient" error *even when raise_error=True* — so listing kr=2 there would turn a documented, opt-in "tell me this read failed" into the same (address, None) the caller gets for an address in a gap between regions. So the two full-address-space sweeps get their own, wider classifier instead: search_addresses_by_value and search_addresses_by_pattern walk every region the filter accepted and hold no opinion about any single one — a range the kernel declines to hand over is a range with no matches in it. The address-list read keeps the strict rule. What is *not* established is why the runner answers kr=2 at all, and the comment says so rather than inventing a mechanism. It does not reproduce locally: probing all 135 scannable regions of a live process yields only kr=1 and kr=10, and a page deliberately re-protected to PROT_NONE — or never mapped — answers KERN_INVALID_ADDRESS, so the obvious "protection changed under us" story is ruled out, not confirmed. Full suite on macOS: 570 passed, 87% coverage. --- PyMemoryEditor/macos/functions.py | 37 +++++++++++++++++- tests/platforms/test_macos_transient_reads.py | 39 +++++++++++++++++++ 2 files changed, 74 insertions(+), 2 deletions(-) diff --git a/PyMemoryEditor/macos/functions.py b/PyMemoryEditor/macos/functions.py index f2d13a1..1522dd1 100644 --- a/PyMemoryEditor/macos/functions.py +++ b/PyMemoryEditor/macos/functions.py @@ -311,6 +311,31 @@ def get_memory_regions(task: int, pid: int = 0) -> Generator[MemoryRegion, None, KERN_MEMORY_ERROR, ) +# KERN_PROTECTION_FAILURE is tolerated by a full-address-space *sweep* only, +# never by a read of caller-supplied addresses — hence a second tuple rather +# than another entry above. +# +# The sweeps walk every region the filter accepted and have no opinion about +# any single one: a range the kernel declines to hand over is a range with no +# matches in it, and aborting the whole scan over it loses every region that +# came after. `search_values_by_addresses` is the opposite case — the caller +# named those addresses, and its documented `raise_error=True` must still be +# able to report that one of them could not be read. Anything in the tuple +# above is swallowed even when `raise_error` is set (see +# process.scanning.iter_values_for_addresses), so putting kr=2 there would +# silently turn "permission denied" into the same `(address, None)` the caller +# gets for an address in a gap. +# +# Honest limits of what is known here. This was added because a plain +# search_by_value over the current process died with kr=2 on GitHub's +# virtualized macOS runners, on a 22 MB region that reported VM_PROT_READ and +# Shared=False. The mechanism was not established. It does *not* reproduce +# locally: probing all 135 scannable regions of a live process yields only kr=1 +# and kr=10, and a page deliberately re-protected to PROT_NONE (or never +# mapped) answers KERN_INVALID_ADDRESS, not kr=2 — so the usual +# "protection changed under us" story does not explain it. +_SWEEP_SKIPPABLE_KRS = _PAGE_GONE_KRS + (KERN_PROTECTION_FAILURE,) + class MachReadError(OSError): """OSError subclass that carries the underlying kern_return_t.""" @@ -494,6 +519,14 @@ def _is_transient(exc: BaseException) -> bool: return isinstance(exc, MachReadError) and exc.kr in _PAGE_GONE_KRS +def _is_skippable_by_sweep(exc: BaseException) -> bool: + """ + Same as :func:`_is_transient`, plus the codes only a full-address-space + sweep may skip. See :data:`_SWEEP_SKIPPABLE_KRS`. + """ + return isinstance(exc, MachReadError) and exc.kr in _SWEEP_SKIPPABLE_KRS + + def _query_region(task: int, address: int): """Return the region containing `address`, or None when the query fails.""" addr = mach_vm_address_t(address) @@ -683,7 +716,7 @@ def search_addresses_by_value( scan_type, _make_read_chunk(task), progress_information=progress_information, - transient_error_check=_is_transient, + transient_error_check=_is_skippable_by_sweep, ) @@ -1044,7 +1077,7 @@ def search_addresses_by_pattern( length, _make_read_chunk(task), progress_information=progress_information, - transient_error_check=_is_transient, + transient_error_check=_is_skippable_by_sweep, ) diff --git a/tests/platforms/test_macos_transient_reads.py b/tests/platforms/test_macos_transient_reads.py index b109b25..925541e 100644 --- a/tests/platforms/test_macos_transient_reads.py +++ b/tests/platforms/test_macos_transient_reads.py @@ -25,6 +25,7 @@ from PyMemoryEditor.macos.functions import ( # noqa: E402 MachPartialReadError, MachReadError, + _is_skippable_by_sweep, _is_transient, ) from PyMemoryEditor.macos.types import ( # noqa: E402 @@ -59,6 +60,12 @@ def test_a_vanished_page_lets_the_scan_continue(kr): # KERN_MEMORY_FAILURE directly above KERN_MEMORY_ERROR and documents it as # permanent, which is exactly the distinction being drawn here; KERN_FAILURE is # what task_for_pid returns without the debugger entitlement. +# +# KERN_PROTECTION_FAILURE stays here even though a *sweep* skips it (see +# test_a_sweep_may_skip_a_protected_range below): this classifier also drives +# search_values_by_addresses, whose caller named the addresses and whose +# documented raise_error=True must still be able to report one it could not +# read. @pytest.mark.parametrize( "kr", (KERN_FAILURE, KERN_PROTECTION_FAILURE, KERN_MEMORY_FAILURE) ) @@ -79,3 +86,35 @@ def test_a_short_read_lets_the_scan_continue(): def test_a_non_mach_error_is_never_transient(): assert not _is_transient(OSError("some unrelated failure")) assert not _is_transient(ValueError("not an OSError at all")) + + +# A full-address-space sweep is the one caller allowed to skip a range the +# kernel refuses to hand over. It has no opinion about any single region, and +# aborting on one loses every region that came after it — which is how a plain +# search_by_value died on a virtualized macOS CI runner. +@pytest.mark.parametrize( + "kr", + ( + KERN_INVALID_ADDRESS, + KERN_INVALID_ARGUMENT, + KERN_NO_ACCESS, + KERN_MEMORY_ERROR, + KERN_PROTECTION_FAILURE, + ), +) +def test_a_sweep_may_skip_a_protected_range(kr): + assert _is_skippable_by_sweep(MachReadError(kr, "read failed (kr=%d)" % kr)) + + +# The sweep is more tolerant, not unconditionally tolerant: a task-level +# failure still has to stop it. +@pytest.mark.parametrize("kr", (KERN_FAILURE, KERN_MEMORY_FAILURE)) +def test_a_sweep_still_stops_on_a_task_level_failure(kr): + assert not _is_skippable_by_sweep(MachReadError(kr, "read failed (kr=%d)" % kr)) + + +def test_only_the_sweep_skips_a_protected_range(): + """The two classifiers must disagree on exactly one code, and this is it.""" + exc = MachReadError(KERN_PROTECTION_FAILURE, "read failed") + assert _is_skippable_by_sweep(exc) + assert not _is_transient(exc)