Repository navigation
Fix TypeError when a class type hint uses | with another class from the same script (#875) - #902
HoneyBearTech wants to merge 2 commits into
Conversation
…mponents#875) A class defined at module level is stored in the global symbol table as an EvalLocalVar. ast_name only unwrapped that when the global table was the current symbol table, so inside another class's body a name like Foo came back as the wrapper. Most uses still worked through its __getattr__, but operators didn't, so an annotation such as "z: Foo | None = None" raised TypeError. Unwrap a defined EvalLocalVar when a name is found in the global symbol table after the local ones. A class that's still being defined stays wrapped, so a self-reference like "nxt: Node = None" inside Node keeps working as before. Names declared global inside a function are looked up and assigned as before. Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
| with another class from the same script (#875)
There was a problem hiding this comment.
Thanks @HoneyBearTech, this is a nice fix with a clear explanation!
Unwrapping a defined EvalLocalVar when the lookup falls through to the global table is the right change. It also fixes a couple of related cases beyond #875. Inside another class's body, subclassing a module-level class (class C(Foo):) used to raise TypeError: EvalLocalVar.__init__() takes 2 positional arguments but 4 were given, and Foo is type(Foo()) returned False. Both work now.
The global path is still broken (not blocking)
This bug predates the PR: a module-level class used through a global declaration still comes back wrapped. For example:
class Foo:
x = 1
def read():
return Foo
def f():
global Foo
Foo | None # TypeError: unsupported operand type(s) for |: 'EvalLocalVar' and 'NoneType'
Foo is type(Foo()) # False
Foo.x = 3 # silently sets x on the wrapper, so Foo.x is still 1
def rebind():
global Foo
Foo = Other # read(), defined earlier, still returns the old FooDeleting through global has the same problem: after global Foo; del Foo, functions defined earlier still see the old class. If Foo is then defined again, they don't see the new one.
You were right not to just unwrap the global load. That breaks your keep() test, because global Foo; Foo = Foo then replaces the wrapper with the plain class, and functions defined earlier keep the old wrapper. The root cause is that assignment through global replaces the table entry (self.global_sym_table[var_name] = val in recurse_assign). Module-level assignment instead updates an existing EvalLocalVar in place with .set(). I tried two changes together:
- in
recurse_assign, when the global table already holds anEvalLocalVar, call.set(val)on it instead of replacing it - in
ast_name, unwrap the value in theglobal_nameslookup, just as this PR does for the fall-through lookup
With those changes all 202 tests pass, including keep(), and the |, is, attribute-assignment and rebinding cases above all behave like Python. del would need the matching change in the global branch of ast_delete, calling set_undefined() on the wrapper instead of removing the entry.
If you'd like to add that here, with tests, that would be great. Otherwise we're happy to merge this as is and do the global path in a follow-up commit. Just let us know which you prefer.
Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
|
@craigbarratt I'll add that here with tests. Thank you! |
…lobal (custom-components#875) A module-level class is stored in the global symbol table as an EvalLocalVar, and functions defined at module level hold on to that wrapper. Used through a global declaration, the class still came back wrapped, so inside a function with "global Foo", "Foo | None" raised TypeError, "Foo is type(Foo())" was False and "Foo.x = 3" set x on the wrapper. Assigning or deleting through global replaced or removed the table entry, so functions defined earlier kept seeing the old class after "global Foo; Foo = Other" or "global Foo; del Foo". As suggested in review, assignment through global now calls .set() on an existing EvalLocalVar instead of replacing it, del through global calls set_undefined() on it (and raises NameError if it is already undefined, like the module-level branch), and a load through global unwraps it when it is defined. An undefined one is left wrapped, as the lookup that falls through to the global symbol table already does, so a class defined through global can still refer to itself in its annotations. Updating a wrapper in place is only safe if no other binding shares it, and functions defined earlier should also see a class defined again or imported under the same name. So: - Assignment, import and from-import now bind names through one helper, assign_name(). It stores the value held by a defined EvalLocalVar rather than the wrapper, so "from mod import Foo", "from mod import *" and "Foo = globals()['Original']" no longer share another binding's wrapper. Otherwise "global Foo; Foo = Other" or "del Foo" in the importer would change mod's own Foo, for every script importing mod. A plain module-level "Foo = Other" or "del Foo" after "from mod import Foo" already did this before. - Reading an attribute of a module, as in "mod.Foo", now also gives the value held by mod's EvalLocalVar, so "Foo = mod.Foo" and "globals()['Foo'] = mod.Foo" store the class itself, and "mod.Foo | None", "mod.Foo is mod.modread()" and isinstance(obj, mod.Foo) work. For a name mod has deleted it raises AttributeError, as in Python. "mod.Foo = Other" updates mod's EvalLocalVar in place, so mod's own functions see the new value, as in Python. - Like assignment, import and from-import now update an existing EvalLocalVar for the name, so functions defined earlier see the imported value, and a name declared global in a function is bound in the global table instead of locally. - Names bound by import or from-import inside a function are now local to it, as in Python, so a nested function can use them; for example "import mod as m" inside a function used to be invisible to a nested function. In a function with a nested function, reading any local name before it is bound, or after it is deleted, now raises UnboundLocalError instead of NameError, as in Python. Without a nested function, reading an imported name before the import raises UnboundLocalError when a module-level name of that name exists, instead of finding it. - In a class body, assignment, import, del, and class and def statements bind in the class namespace, even inside a function that declares the name global, and reading a name finds the class's own binding before the global one, as in Python. Assignment and import there replace an EvalLocalVar left by a nested class instead of updating it, so a method doesn't pick up the new value as an enclosing variable. A global declaration in the class body itself applies to the class body only, as in Python: the name is bound and read in the global table there, but no longer becomes global in the enclosing function once the class statement completes. In a class body at module level it no longer raises SyntaxError. - "from mod import Foo" of a name mod has deleted raises ImportError with Python's message, and "from mod import *" skips it, since del now leaves an undefined EvalLocalVar in mod. - A class statement always made a new EvalLocalVar, so functions defined earlier never saw a class defined again under the same name, and inside the new class body the name was undefined. It now reuses an existing wrapper and sets the new class once the class statement completes. Until then the name still holds the old class, also if the class body raises, as in Python. A first definition works as before. This changes how imports and aliases behave: "from mod import Foo" and "Foo = mod.Foo" now take a snapshot of mod's Foo, as in Python, and no longer follow later changes to Foo inside mod; "mod.Foo" itself does. Known limits: - getattr(mod, "Foo"), mod.__dict__ and vars(mod) still hand out mod's EvalLocalVar, and so does "mod.Foo" inside a lambda or @pyscript_compile function; after mod deletes Foo, hasattr(mod, "Foo") and dir(mod) still report it - a lambda or @pyscript_compile function gets a module-level class as its EvalLocalVar: reading an attribute works and follows a later rebind, but calling it or "|" raises TypeError, and "is", "==", isinstance() and issubclass() against the class give False - a function defined with @pyscript_compile in a class body that declares its name global is still bound in the class namespace, not the global table - in a function without a nested function, reading an imported name before the import gives the builtin of that name if there is one, and raises NameError if there is neither a module-level name nor a builtin, instead of UnboundLocalError - a dotted import such as "import os.path" inside a function is not local to it, so a nested function can't use it - a method reads a class nested in its class body (class C: class Foo: ...) as if it were an enclosing variable, instead of the module-level name - "global Zed; del Zed" on a name that was never defined doesn't raise NameError - after "global Foo; del Foo", a bare read of Foo through global gives back the undefined wrapper instead of raising NameError. Reading an attribute, calling it or using it in isinstance() still raises NameError, but "|" raises TypeError, and "is", "==", bool(), str() and setting an attribute don't raise at all Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
|
Thank you @craigbarratt for the careful review and the suggested fix. I've added the global path here as a second commit, so the first commit is unchanged. Your three changes are in, but on their own they bring back a cross-module problem.
This does change one thing users might notice: The tests run each case in CPython and in pyscript and compare the results. The commit message lists what still differs from Python. The full suite, ruff, pylint, mypy and codespell pass on 3.14. It's bigger than your sketch, so if you'd rather take only part of it, or merge the first commit on its own and look at this one separately, that's fine too. |
Closes #875
What was broken
A type hint with
|that names a class defined in the same script failed to load:Why
A class defined at module level is stored in the global symbol table as an
EvalLocalVar.ast_nameonly unwrapped it when the global table was the current symbol table, so insideanother class's body
Foocame back as the wrapper. Most uses still worked through the wrapper's__getattr__, but operators such as|didn't.What I changed
When a name is found in the global symbol table after the local tables,
ast_namenow unwraps a definedEvalLocalVar. A class that is still being defined stays wrapped, so aself-reference like
nxt: Node = NoneinsideNodeworks as before. Names declaredglobalinside a function are read and assigned exactly as before.Testing
tests/test_unit_eval.py:Nodeself-reference;globalinside a function still updates functions defined earlier.Second commit: the
globalpath (as suggested in review)A module-level class used through
globalinside a function still came back wrapped, soFoo | Noneraised TypeError,Foo is type(Foo())was False, andFoo.x = 3setxon the wrapper. Rebinding or deleting throughglobalalso wasn't seen by functions defined earlier.delthroughglobalnow update the existingEvalLocalVarin place, and a load throughglobalunwraps it.import,from ... import) and a read ofmod.Foonow store or return the value, not the wrapper, so one script can no longer change another module'sFoothrough a shared wrapper.from mod import FooandFoo = mod.Fooare now snapshots, as in Python. The new tests run each case in CPython and in pyscript and compare the results.Known limits
Listed in the second commit's message. The main ones:
getattr(mod, "Foo"),mod.__dict__andvars(mod)still return the wrapper, and so does a module-level class read inside a lambda or@pyscript_compilefunction.