Repository navigation
Plan: add the global path fix to #902 - #2
honeybeartech-homelab-automation[bot] wants to merge 5 commits into
Conversation
Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
|
@HoneyBearTech The plan for the follow-up to upstream custom-components#902 is ready. Please approve this PR (or comment "approve") to start the work. |
HoneyBearTech
left a comment
There was a problem hiding this comment.
I'd like to add it to custom-components#902 instead
Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
|
@HoneyBearTech Plan updated as you asked: the |
…lobal (custom-components#875) A module-level class is stored in the global symbol table as an EvalLocalVar, and functions defined at module level hold on to that wrapper. Used through a global declaration, the class still came back wrapped, so inside a function with "global Foo", "Foo | None" raised TypeError, "Foo is type(Foo())" was False and "Foo.x = 3" set x on the wrapper. Assigning or deleting through global replaced or removed the table entry, so functions defined earlier kept seeing the old class after "global Foo; Foo = Other" or "global Foo; del Foo". As suggested in review, assignment through global now calls .set() on an existing EvalLocalVar instead of replacing it, del through global calls set_undefined() on it (and raises NameError if it is already undefined, like the module-level branch), and a load through global unwraps it when it is defined. An undefined one is left wrapped, as the lookup that falls through to the global symbol table already does, so a class defined through global can still refer to itself in its annotations. Updating a wrapper in place is only safe if no other binding shares it, and functions defined earlier should also see a class defined again or imported under the same name. So: - Assignment, import and from-import now bind names through one helper, assign_name(). It stores the value held by a defined EvalLocalVar rather than the wrapper, so "from mod import Foo", "from mod import *" and "Foo = globals()['Original']" no longer share another binding's wrapper. Otherwise "global Foo; Foo = Other" or "del Foo" in the importer would change mod's own Foo, for every script importing mod. A plain module-level "Foo = Other" or "del Foo" after "from mod import Foo" already did this before. - Reading an attribute of a module, as in "mod.Foo", now also gives the value held by mod's EvalLocalVar, so "Foo = mod.Foo" and "globals()['Foo'] = mod.Foo" store the class itself, and "mod.Foo | None", "mod.Foo is mod.modread()" and isinstance(obj, mod.Foo) work. For a name mod has deleted it raises AttributeError, as in Python. "mod.Foo = Other" updates mod's EvalLocalVar in place, so mod's own functions see the new value, as in Python. - Like assignment, import and from-import now update an existing EvalLocalVar for the name, so functions defined earlier see the imported value, and a name declared global in a function is bound in the global table instead of locally. - Names bound by import or from-import inside a function are now local to it, as in Python, so a nested function can use them; for example "import mod as m" inside a function used to be invisible to a nested function. In a function with a nested function, reading any local name before it is bound, or after it is deleted, now raises UnboundLocalError instead of NameError, as in Python. Without a nested function, reading an imported name before the import raises UnboundLocalError when a module-level name of that name exists, instead of finding it. - In a class body, assignment, import, del, and class and def statements bind in the class namespace, even inside a function that declares the name global, and reading a name finds the class's own binding before the global one, as in Python. Assignment and import there replace an EvalLocalVar left by a nested class instead of updating it, so a method doesn't pick up the new value as an enclosing variable. - "from mod import Foo" of a name mod has deleted raises ImportError with Python's message, and "from mod import *" skips it, since del now leaves an undefined EvalLocalVar in mod. - A class statement always made a new EvalLocalVar, so functions defined earlier never saw a class defined again under the same name, and inside the new class body the name was undefined. It now reuses an existing wrapper and sets the new class once the class statement completes. Until then the name still holds the old class, also if the class body raises, as in Python. A first definition works as before. This changes how imports and aliases behave: "from mod import Foo" and "Foo = mod.Foo" now take a snapshot of mod's Foo, as in Python, and no longer follow later changes to Foo inside mod; "mod.Foo" itself does. Known limits: - getattr(mod, "Foo"), mod.__dict__ and vars(mod) still hand out mod's EvalLocalVar, and so does "mod.Foo" inside a lambda or @pyscript_compile function - a lambda or @pyscript_compile function gets a module-level class as its EvalLocalVar: reading an attribute works and follows a later rebind, but calling it or "|" raises TypeError, and "is", "==", isinstance() and issubclass() against the class give False - in a function without a nested function, reading an imported name before the import gives the builtin of that name if there is one, and raises NameError if there is neither a module-level name nor a builtin, instead of UnboundLocalError - a dotted import such as "import os.path" inside a function is not local to it, so a nested function can't use it - a method reads a class nested in its class body (class C: class Foo: ...) as if it were an enclosing variable, instead of the module-level name - "global Zed; del Zed" on a name that was never defined doesn't raise NameError - after "global Foo; del Foo", a bare read of Foo through global gives back the undefined wrapper instead of raising NameError. Reading an attribute, calling it or using it in isinstance() still raises NameError, but "|" raises TypeError, and "is", "==", bool(), str() and setting an attribute don't raise at all Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
|
@HoneyBearTech The DoneOne new commit, ready to add on top of custom-components#902's commit. Using a class through To make the maintainer's idea safe, we had to stop scripts from sharing pyscript's internal "box" for a class. Before, Along the way, these changed too, all to match Python:
The commit message lists what still doesn't match Python. The change is much bigger than the maintainer's three-line sketch: about 100 lines of code and 740 lines of tests. The tests run every case in real Python and in pyscript and check that the two agree. Checks
Two fix rounds is the limit, so these two are still open. Rejected findings
My recommendationDon't approve yet. Comment "fix" here and I'll do one short extra round for the two open problems, re-check, and ask again. The alternative is to approve now, and I'll list both problems in the commit message as known limits. I don't recommend that, because one of them breaks code that works today. Given the size, you could also still tell the maintainer to merge custom-components#902 as is, and we'd send this as its own PR. Just comment "separate" if you'd prefer that. What happens when you approve (after the fix round)
To approveApprove this PR (or comment "approve") to finish. Close it to throw the work away. |
|
fix |
…#875) Adds the class-body global and prepared namespace fixes, their tests and the updated job record on top of the previous version. The commit to send upstream is 4a99336. Signed-off-by: Sean Stoube <31805425+HoneyBearTech@users.noreply.github.com>
|
@HoneyBearTech The extra fix round is done. Both open problems are fixed, and the final audit found nothing new. This is ready for your approval. Done
The change is unchanged otherwise. It is one commit for custom-components#902: about 110 lines of code and 825 lines of tests, each case checked against real Python. Checks
Rejected findings
What happens when you approve
To approveApprove this PR (or comment "approve") to finish. Close it to throw the work away. |
|
@HoneyBearTech Done: the new commit is on custom-components#902 (custom-components#902), and GitHub shows it can merge cleanly. Two things for you to paste there; the app can't write on the upstream repo. 1. Reply comment on custom-components#902 (copy everything inside the box): Thanks for the careful review and the suggested fix. I've added the global path here as a second commit, so the first commit is unchanged.
Your three changes are in, but on their own they bring back a cross-module problem. `from mod import Foo` copied mod's `EvalLocalVar` into the importer, so after that, `global Foo; Foo = Other` (or `del Foo`) in the importer updated the shared wrapper and changed mod's own `Foo`, for every script importing mod. A plain module-level `Foo = Other` after `from mod import Foo` already did this on master. So the commit also:
- stores the value, not the wrapper, whenever a name is bound (assignment, `import`, `from ... import`), and when reading `mod.Foo`, so a binding never shares another module's wrapper; `mod.Foo = Other` now updates mod's wrapper in place
- reuses an existing wrapper in a class statement, so after `global Foo; del Foo` a class defined again is seen by functions defined earlier (with your version that raised NameError)
- makes imports inside a function local to it, and keeps class-body names in the class namespace, both as in Python; these were needed so the in-place updates don't reach the wrong scope
This does change one thing users might notice: `from mod import Foo` and `Foo = mod.Foo` are now snapshots, as in Python, and no longer follow later changes inside mod.
The tests run each case in CPython and in pyscript and compare the results. The commit message lists what still differs from Python. The full suite, ruff, pylint, mypy and codespell pass on 3.14.
It's bigger than your sketch, so if you'd rather take only part of it, or merge the first commit on its own and look at this one separately, that's fine too.2. New description for custom-components#902 (on custom-components#902, click ··· → Edit at the top, replace all the text with this; it fills in the empty "Known limits" heading and adds the second commit): Closes #875
**What was broken**
A type hint with `|` that names a class defined in the same script failed to load:
```python
class Foo:
pass
class Bar:
z: Foo | None = None
```
```
TypeError: unsupported operand type(s) for |: 'EvalLocalVar' and 'NoneType'
```
**Why**
A class defined at module level is stored in the global symbol table as an `EvalLocalVar`. `ast_name` only unwrapped it when the global table was the current symbol table, so inside
another class's body `Foo` came back as the wrapper. Most uses still worked through the wrapper's `__getattr__`, but operators such as `|` didn't.
**What I changed**
When a name is found in the global symbol table after the local tables, `ast_name` now unwraps a defined `EvalLocalVar`. A class that is still being defined stays wrapped, so a
self-reference like `nxt: Node = None` inside `Node` works as before. Names declared `global` inside a function are read and assigned exactly as before.
**Testing**
- New cases in `tests/test_unit_eval.py`:
- the annotation from the issue, both in a plain class and in a dataclass;
- the `Node` self-reference;
- a check that rebinding a class through `global` inside a function still updates functions defined earlier.
- The full test suite, pylint, mypy and ruff all pass on Python 3.14.
**Second commit: the `global` path** (as suggested in review)
A module-level class used through `global` inside a function still came back wrapped, so `Foo | None` raised TypeError, `Foo is type(Foo())` was False, and `Foo.x = 3` set `x` on the wrapper. Rebinding or deleting through `global` also wasn't seen by functions defined earlier.
- Assignment and `del` through `global` now update the existing `EvalLocalVar` in place, and a load through `global` unwraps it.
- A binding (assignment, `import`, `from ... import`) and a read of `mod.Foo` now store or return the value, not the wrapper, so one script can no longer change another module's `Foo` through a shared wrapper.
- A class statement reuses an existing wrapper, so a class defined again is seen by functions defined earlier.
- Imports inside a function are local to it, and names in a class body stay in the class, as in Python.
`from mod import Foo` and `Foo = mod.Foo` are now snapshots, as in Python. The new tests run each case in CPython and in pyscript and compare the results.
**Known limits**
Listed in the second commit's message. The main ones: `getattr(mod, "Foo")`, `mod.__dict__` and `vars(mod)` still return the wrapper, and so does a module-level class read inside a lambda or `@pyscript_compile` function.
|
This PR holds the plan for the next step on upstream custom-components/pyscript#902, the custom-components#875 fix. It stays on this fork and is never sent upstream. Full plan:
docs/plans/global-path-875.md.Where things stand. The maintainer approved custom-components#902. They also suggested fixing a related, older bug: using a class through
globalinside a function. They asked whether we want that in custom-components#902 or in a follow-up.What we found. Their suggested fix works for their examples, but it has a catch. If script B does
from mod import Fooand then reassignsFoothroughglobal, it also changesFooinsidemod, for every script that usesmod. We hit and removed this same problem earlier in custom-components#875. Two extra small changes avoid it, and they also fix a related bug that exists today.The plan, in short (updated: you chose to add this to custom-components#902):
eval.py: the maintainer's three, plus our two.|with another class from the same script (#875) custom-components/pyscript#902's branch. That updates upstream Fix TypeError when a class type hint uses|with another class from the same script (#875) custom-components/pyscript#902 directly. The existing commit stays as it is.|with another class from the same script (#875) custom-components/pyscript#902, and an updated Fix TypeError when a class type hint uses|with another class from the same script (#875) custom-components/pyscript#902 description (the current one ends with an empty "Known limits" heading).Because custom-components#902 gets new commits, the maintainer will look at it again before merging.
Contribution rules followed: the project's
docs/contributing.rstandtests/README.md(dev setup, pre-commit hooks, pytest), pre-commit (ruff, codespell, pylint, mypy), and CI on Python 3.14. There is no PR template; we follow the maintainer's habits: small focused changes with tests. Commits are signed off.Status: built and checked, and the extra fix round is done. No open problems. See the latest report comment.
Approve this PR (or comment "approve") to finish. Close it to throw the work away.