Skip to content

Restore the deferred imports to module scope - #557

Open
jaraco wants to merge 3 commits into
mainfrom
feature/eager-imports
Open

jaraco wants to merge 3 commits into
mainfrom
feature/eager-imports

Conversation

@jaraco

@jaraco jaraco commented Oct 31, 2026

Copy link
Copy Markdown
Member

Closes #554.

Puts all six function-local imports back at module scope and keeps them deferred by declaring __lazy_modules__ (PEP 810).

     def _load_json(self, filename):
-        # Deferred for performance (python/importlib_metadata#503)
-        import json
-
         return pass_none(json.loads)(

Shape

__lazy_modules__ is only a module global, so it is inert before 3.15 and needs no guard — the declaration reads the same in both trees, and __name__ keeps the entries for this package's own modules identical too. The one entry that has to differ is zipfile, next to the import line that already differs for the zipp overlay; there is a comment on it so the sync step is visible.

Same behavior before 3.15 needs the backport, and it lives behind compat/py314 in the shape compat already uses, so the shared module carries only from .compat.py314 import lazy_importer and with lazy_importer():. Both come out at sync time like from .compat import py311 already does. The one real sync cost is that the imports inside the block are indented and have to be dedented.

_compat/localize.py gets the same treatment, not for its own sake but because it is imported eagerly — leaving _adapters at module scope there would have undone the deferral in __init__.py.

Two things worth knowing

from . import _adapters doesn't work. Per PEP 810 that form names the package as the module to defer, and the package is mid-import, so it defers nothing; on the backport it recurses until the stack is exhausted. Importing the member instead keys the declaration on _adapters and works on both, at the cost of _adapters.Message becoming Message at two call sites. Both declarations carry a note, since the next person to add an entry will hit this.

inspect can't be observed as deferred on 3.12 and 3.13, where importlib.abc — eagerly imported here — reaches it through importlib.resources._common. The declaration is still right; the module is just loaded for other reasons. test_declared_modules_stay_unloaded imports importlib.abc before taking its snapshot so that it measures the declaration and not the stdlib's own graph.

Cost

Per the project's own pytest-perf exercise:

import time
3.11 +0.6 ms
3.12 +1.6 to +2.6 ms
3.13 +0.9 to +1.7 ms
3.14 +0.8 to +1.1 ms
3.15 +0.06 ms (+1%)

Ranges because repeated runs on one machine disagree by that much — the honest figure is 1 to 2 ms before 3.15, and nothing measurable on 3.15. The entire cost is importing lazy_importer itself (its module body, plus contextlib, threading and importlib.util); the deferrals it buys back are the ones already in place.

lazy-importer becomes a dependency below 3.15. It is pure Python with no dependencies of its own, but it is at 0.1.0, which is worth weighing for a package this far down the stack — say the word and I'll make compat/py314 fall back to eager imports when it is absent, at the price of uniformity.

Verification

tox -e py,py311,py312,py313,py314 — 266/267 passed on each, mypy clean, coverage 99%; tox -e docs and tox -e diffcov clean, diff coverage 100%. Beyond the test above, test_declared_modules_resolve checks that reading a lazy name yields the real module and not one of the backport's proxies, which is what isinstance and type() comparisons would otherwise trip over.

🤖 Generated with Claude Code

jaraco and others added 2 commits October 31, 2026 15:45
Six imports sat inside the functions that used them, each with a comment
explaining why, and none where a reader looks for a module's
dependencies. PEP 810 lets the language express that instead: declared in
__lazy_modules__, they stay deferred while reading as ordinary imports.

Python 3.15 honors the declaration natively. Earlier versions reach the
same behavior through the lazy_importer backport, kept behind
compat/py314 so the shared module carries only one import and one with
statement, both removed when syncing to the stdlib.

Costs about 1-2 ms of import time before 3.15, the price of importing
lazy_importer, and nothing measurable on 3.15.

Closes #554

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Before 3.15 the backport binds a proxy, which passes attribute access
and calls through but answers isinstance() as no match however the
object relates to the class it stands for. localize.message asks exactly
that, and so adapted an already adapted message a second time on every
Python before 3.15.

Route such uses through compat.py314.resolve, the backport's resolve()
before 3.15 and the identity after, so the behavior matches across
versions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jaraco

jaraco commented Nov 7, 2026

Copy link
Copy Markdown
Member Author

Pushed a fix for a behavior difference this change introduced, found while checking it really does behave alike on 3.14 and 3.15.

Before 3.15 the backport binds a proxy. A proxy passes attribute access and calls through to the object it stands for, which covers almost every use of a lazy name — csv.reader, json.loads, zipfile.Path, Message(...). But isinstance inspects the second argument itself, sees the proxy, and answers no match whatever the object is, without raising. localize.message asks exactly that:

if isinstance(input, Message):
    return input
return Message(input)

So an already adapted message was adapted a second time on every Python before 3.15, while 3.15 passed it through:

$ python3.14 isinst.py          # before the fix
passed through unchanged: False
double-wrapped: True
$ python3.15 isinst.py
passed through unchanged: True
double-wrapped: False

The backport provides resolve() for this, so compat/py314 now exports a resolve alongside lazy_importer — the backport's before 3.15, the identity after — and that one site uses it. localize.py is excluded from the stdlib, so this costs nothing at sync time.

The regression test runs as a subprocess, because reading the name anywhere resolves it for good and any other test in the class would do that first. Without the fix it fails on 3.14 and passes on 3.15, which is what made the bug invisible until I went looking for divergence specifically.

One residual difference is worth recording rather than fixing, since it is the backport's design: before 3.15 the bound name is not the module, so importlib_metadata.inspect is inspect and importlib_metadata.Message is _adapters.Message are False there and True on 3.15. Nothing in the package depends on that identity now, and resolve is the remedy for anything that comes to. test_reading_through_a_lazy_name pins what does hold everywhere.

All five envs green, diff coverage 100%.

Only the package's own __init__ has to import the member: naming the
package as the module to defer is what no emulation can honor from
inside the package itself. Elsewhere `from .. import _adapters` works,
and reaching the class through the module resolves the lazy name, so
isinstance sees the class and not the proxy without being told to.

That leaves compat.py314.resolve with no callers, so it goes, and what
it documented moves to lazy_importer, out of the shared module and into
the shim the stdlib never sees.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jaraco

jaraco commented Nov 8, 2026

Copy link
Copy Markdown
Member Author

Simplified: localize.py goes back to from .. import _adapters, and compat/py314's resolve is gone with it.

The constraint I'd generalized turns out to be narrow. from . import <submodule> names the package as the module to defer, and that is only a problem in the package's own __init__, where the proxy stands for the module being executed and resolves itself through the globals entry that is the proxy. From any other module it is fine:

$ python3.14 -c 'import importlib_metadata'     # from . import _adapters in __init__
RecursionError: maximum recursion depth exceeded
$ python3.14 pkgspell.py                        # from .. import _adapters in localize
_adapters deferred: True
passed through unchanged: True

So only __init__.py has to import the member. localize.py importing the module is also better than what I had: reaching the class as _adapters.Message is attribute access on the proxy, which resolves it, so isinstance sees the class without being told to. The double-adaptation bug cannot arise in that spelling, which is why resolve now has no callers and has been removed.

What resolve's docstring carried has moved onto lazy_importer in compat/py314, which is excluded from the stdlib — a better home for emulation lore than the shared module. It records both traps: the proxy's silent isinstance answer, and the recursion on a package self-reference.

I also corrected the note in __init__.py. It claimed from . import _adapters "would defer nothing, the package being mid-import"; that is wrong — 3.15 defers it natively and correctly. The reason is only that no emulation can honor it from inside the package.

Verified: 267/268 passed on 3.11 through 3.15, mypy clean, docs and diffcov clean at 100% diff coverage. The regression test still earns its place — reinstating the member spelling without resolve makes it fail on 3.14.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Restore the deferred imports to module scope with PEP 810 lazy imports

1 participant