Repository navigation
Load the bridge library from outside the VUnit output path - #13
Conversation
b73cfa1 to
33d74db
Compare
The prebuilt Windows DLL was copied to the output path and other libraries were built there, which fails where the output path does not allow executing files. The prebuilt DLL is now loaded from the installed package and built libraries are cached in the cache directory of the user. VUNIT_PYTHON_BRIDGE_LIBRARY_DIR selects another directory on every platform. The library found its configuration next to itself. The configuration stays in the output path, and the simulator hooks of NVC, GHDL and Questa/ModelSim pass its path in VUNIT_PYTHON_BRIDGE_CONFIG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loading it from the package gave GHDL its versioned file name as the VHPIDIRECT library token, which exceeds the 32 characters GHDL accepts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The simulator cannot load the bridge library from a file system mounted noexec (e.g. a tmpfs in a rootless podman container) and GHDL only says "cannot load VHPIDIRECT shared library". Check the mount when preparing the library and point at VUNIT_PYTHON_BRIDGE_LIBRARY_DIR instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A linked simulation found the bridge library only through the LD_LIBRARY_PATH the run environment hook sets, so a ghdl wrapper that resets it, or a re-run by hand, failed to load it. GHDL passes what follows -Wl, to the compiler driver, hence -Wl,-Wl,-rpath. mcode and llvm-jit cannot be helped this way: GHDL limits the library name of a VHPIDIRECT attribute to 32 characters, too short for an absolute path. Document that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ritable A container run with a random UID often has HOME=/ or a read-only home, and the setup then failed with a bare OSError creating the cache directory. Raise a PythonBridgeError naming the directory, whether it is the default or set by VUNIT_PYTHON_BRIDGE_LIBRARY_DIR, and asking for that variable to be set to a writable directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows has no noexec mount, but an execution policy (AppLocker, Windows Defender Application Control, Smart App Control) can block libraries in a directory, and NVC and GHDL then only report that loading failed. Load the VHPIDIRECT library once at setup, with the LoadLibrary search order the simulators use, and fail early: a policy error points to VUNIT_PYTHON_BRIDGE_LIBRARY_DIR, any other error is reported with its reason. The library is unloaded again so the file is not kept open. The FLI variant is not checked, it needs the libraries of vsim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NVC and GHDL report a bridge library they cannot load as "cannot load VHPIDIRECT shared library" and leave out the reason the loader gives. The setup now loads the VHPIDIRECT variant itself on Linux and macOS, the counterpart of the Windows check, and fails with that reason, for example a missing libpython. Taken from VUnit#20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| # Questa finds the library by the absolute path in its FLI attributes | ||
| self.assertIsNone(context.hooks["modelsim"]["run_env"]) | ||
| # Every simulator is told where the configuration of the project is | ||
| for simulator in ("nvc", "ghdl", "modelsim"): |
There was a problem hiding this comment.
Also include riviera pro
There was a problem hiding this comment.
Claude (AI assistant) here, on behalf of @ru551n:
Riviera-PRO/Active-HDL use the VHPI application rather than the bridge library, and the application reads no configuration file, so these hooks don't apply to it. Moving the application out of the output path, which is what including Riviera-PRO amounts to here, goes into #21 with the rest of the Riviera-PRO/Active-HDL support.
|
|
||
| - `import_module_from_file` resolves a relative path against the directory of the testbench file, like `exec_file`, rather than against the working directory of the simulator. It must be called after `test_runner_setup`. Callers that relied on a path relative to the working directory must pass an absolute path instead. See #7. | ||
| - A relative file name given to `exec_file` or `import_module_from_file` before `test_runner_setup` has completed is reported as a failure, instead of being taken from the working directory of the simulator. The new `base_path` returns a base directory set with `vunit_python_bridge.set_relative_file_base()` or `VUNIT_PYTHON_BRIDGE_FILE_BASE`, for file names that work at any time, for example in verification components. | ||
| - The bridge library is no longer built in or copied to the VUnit output path, which may not allow executing files, but to the cache directory of the user. This includes the prebuilt Windows DLLs. `VUNIT_PYTHON_BRIDGE_LIBRARY_DIR` selects another directory on every platform. When the directory cannot be written, the setup fails and asks for the variable to be set. |
There was a problem hiding this comment.
Also mention the XDG variable here.
| if configured: | ||
| return Path(configured).absolute() | ||
| if sys.platform == "win32": | ||
| cache = os.environ.get("LOCALAPPDATA") or str(Path.home() / "AppData" / "Local") |
There was a problem hiding this comment.
I think most users would prefer the user directory, such as a sub directory to it as many other config directories and cache stuff hit. This is too hidden.
There was a problem hiding this comment.
Claude (AI assistant) here, on behalf of @ru551n:
Done in 0043ca0. The default is now ~/.vunit-python-bridge on every platform (%USERPROFILE%\.vunit-python-bridge on Windows). The user guide, the README, the release note and the Windows wheel test are updated. Rechecked with GHDL: the library lands in ~/.vunit-python-bridge, and with HOME=/ the setup fails with Cannot build the Python bridge library in /.vunit-python-bridge (the default): ....
|
|
||
| class TestWindowsLoadCheck(unittest.TestCase): | ||
| """ | ||
| The Windows counterpart of the noexec check: loading the library, with ctypes mocked. |
There was a problem hiding this comment.
Where is the linux counterpart?
There was a problem hiding this comment.
Claude (AI assistant) here, on behalf of @ru551n:
It's in TestPosixBuildAndCache: test_noexec_output_path_raises_before_compiling for the noexec check and test_library_that_does_not_load_raises_with_the_loader_error for the load check. The docstring of TestWindowsLoadCheck now points to them (a527b1c).
| mock.patch("ctypes.windll", mock.Mock(kernel32=kernel32), create=True), | ||
| ): | ||
| # A 64-bit handle, which FreeLibrary must get as a pointer rather than a C int | ||
| cdll_mock.return_value._handle = 0x7FF812340000 |
There was a problem hiding this comment.
Is thus just a random number?
There was a problem hiding this comment.
Claude (AI assistant) here, on behalf of @ru551n:
Not random, but the value didn't show why. The test is about truncation: FreeLibrary must get the handle as a pointer, and a 64-bit handle passed as a C int would be cut off. It's now 2**32 + 0x1000, just above the range of a C int, with a comment saying so (a527b1c).
|
|
||
| def test_every_simulator_is_told_where_the_configuration_is(self): | ||
| hooks = self._hooks(Path("/some/dir/libvunit_python_bridge.so")) | ||
| for simulator in ("nvc", "ghdl", "modelsim"): |
There was a problem hiding this comment.
Claude (AI assistant) here, on behalf of @ru551n:
Riviera-PRO/Active-HDL use the VHPI application rather than the bridge library, and the application reads no configuration file, so these hooks don't apply to it. Moving the application out of the output path, which is what including Riviera-PRO amounts to here, goes into #21 with the rest of the Riviera-PRO/Active-HDL support.
The default was the cache directory of the platform (~/.cache, or XDG_CACHE_HOME, on Linux, ~/Library/Caches on macOS, %LOCALAPPDATA% on Windows), which is hidden from most users. A dot-directory in the home directory, like many other tools use, is easier to find and the same on every platform. VUNIT_PYTHON_BRIDGE_LIBRARY_DIR still selects another directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Point to the Linux and macOS counterparts of the check, and use a handle whose purpose is evident: just above the range of a C int. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 participant
The bridge library ended up in the VUnit output path: the prebuilt Windows DLL was copied to
<output path>/python_bridge/..., and Linux/macOS and Windows gcc builds were compiled there. Where the output path doesn't allow executing files (anoexecmount, AppLocker or similar), the simulator can't load the library.Changes
All bridge libraries go to
~/.vunit-python-bridgein the home directory of the user instead of the output path, on every platform (%USERPROFILE%\.vunit-python-bridgeon Windows). That covers the prebuilt Windows DLLs, copied under the short namevunit_python_bridge.dllas before, and the libraries built for Linux/macOS, Windows gcc and Questa FLI.Builds are keyed by what they were built from, so projects can share the directory.
VUNIT_PYTHON_BRIDGE_LIBRARY_DIRselects another directory on every platform.Configuration. The library used to read
vunit_python_bridge.cfgfrom its own directory, which is why it had to live in the output path. The config is per project, so it stays in<output path>/python_bridge. Newrun_envhooks for NVC, GHDL and Questa/ModelSim pass its path inVUNIT_PYTHON_BRIDGE_CONFIG.config.creads that variable, with_wgetenvon Windows so non-ASCII paths survive, and falls back to the library's directory when it isn't set.Why the DLL is still copied. The first version of this PR loaded the DLL straight from the package. GHDL's VHPIDIRECT library token then became the versioned file name, which is longer than the 32 characters GHDL accepts, and Windows GHDL CI crashed.
Docs and release note. README, the "How it works" section of the user guide, and
release_notes/0.2.0.md.Unwritable library directory. When the library directory cannot be created or written (e.g.
HOME=/with a random UID in a container), the setup fails with aPythonBridgeErrornaming the directory, whether it is the default or set byVUNIT_PYTHON_BRIDGE_LIBRARY_DIR, and asking for that variable to be set to a writable directory. There is no fallback to the output path.Windows load check. Windows has no
noexecmount, but an execution policy (AppLocker, Windows Defender Application Control, Smart App Control) can block libraries in a directory, and NVC and GHDL then only report that loading failed. The setup loads the VHPIDIRECT library once, withwinmode=0so dependencies are searched like the simulators'LoadLibrarydoes, and unloads it again. Errors 1260, 577 and 4551 (blocked by policy, invalid image hash, integrity policy violation) fail with a message pointing toVUNIT_PYTHON_BRIDGE_LIBRARY_DIR. Any other load error fails with its reason. The FLI variant is not checked, since it needs vsim's libraries. Overhead: one load and unload per setup, well under a millisecond (measured 0.16 ms for the Linux equivalent).Linux/macOS load check (taken from Test on RHEL 9 and 10 with every GHDL backend and NVC, and report why the bridge does not load #20). After the noexec check, the setup loads the VHPIDIRECT library with
ctypes.CDLL, the counterpart of the Windows check. Any other load failure, for example a missing libpython, fails with the loader's reason, which NVC and GHDL leave out of "cannot load VHPIDIRECT shared library". The FLI variant is not checked, since it needs vsim's symbols. It does not catch aLD_LIBRARY_PATHthat is dropped between VUnit and GHDL mcode/llvm-jit, since it loads the library by absolute path; that case is documented below.noexec check. On Linux and macOS, a library directory on a file system mounted
noexec(e.g. a tmpfs in a rootless podman container) now fails the setup with aPythonBridgeErrornaming the directory andVUNIT_PYTHON_BRIDGE_LIBRARY_DIR. Without it, GHDL only sayscannot load VHPIDIRECT shared library 'libvunit_python_bridge.so'.rpath for GHDL llvm and gcc. The elaboration hook also passes
-Wl,-Wl,-rpath,<library dir>(GHDL hands what follows-Wl,to the compiler driver). A linked simulation then finds the bridge without theLD_LIBRARY_PATHof the run environment, so aghdlwrapper that drops it, or a re-run by hand, works. It still needsVUNIT_PYTHON_BRIDGE_CONFIG, and fails with a message naming it when that is missing. mcode and llvm-jit can't be helped this way: GHDL stores the VHPIDIRECT library name in a 32-character field, too short for an absolute path. The user guide documents that a wrapper must keepLD_LIBRARY_PATHandVUNIT_PYTHON_BRIDGE_CONFIG.Tests
After rebasing onto
mainwith Report relative file names used before test_runner_setup and add base_path #12, Lint the Python code with ruff and check license headers #22 and Mark the re-exports of __init__.py for ruff #23, and the review (a527b1c):VUNIT_PYTHON_BRIDGE_LIBRARY_DIRset: 109 of 109, andtests/file_base7 of 7 with the base directory set by the run script and 7 of 7 with it set by the environment variable.pytest:
run_envhook for every simulator.The suite sets
VUNIT_PYTHON_BRIDGE_LIBRARY_DIRto temporary directories, so it never writes to the real home directory.CI:
~/.vunit-python-bridgeand nothing was compiled.Locally (Linux):
~/.vunit-python-bridge(rechecked with GHDL after the default changed) and the output path only holds the config and generated VHDL.Linux, rootless podman, GHDL mcode, Python 3.11:
noexectmpfs: the bridge now loads (107 of 108 feature tests; the remaining failure, "Test importing a module with a relative file name", comes from runningrun.pyfrom thetestsdirectory, which that test rejects; it passes from the repository root, as CI runs it).noexectmpfs: the setup fails with the new message.Linux/macOS load check: a new unit test puts a broken library in a fresh library directory and checks the message names it with the loader's reason. The two tests with a stub compiler mock the load. In podman, with Python 3.11 and GHDL 6.0.0 mcode, and with Python 3.12 and NVC 1.22.1: pytest 95 passed and feature tests 108 of 108 on both, and a corrupt cached library fails the setup with
does not load: ... file too short.Windows load check: three unit tests with
ctypesmocked (policy errors, other errors, unload with a 64-bit handle). The 13 Windows CI jobs (prebuilt DLL and gcc build, NVC and GHDL, wheel tests) pass, so the check does not reject a library that loads. The policy-blocked case is not tested on a real machine.Linux, GHDL mcode, unwritable library directory:
HOME=/:Cannot build the Python bridge library in /.vunit-python-bridge (the default): [Errno 13] Permission denied: '/.vunit-python-bridge'. Set VUNIT_PYTHON_BRIDGE_LIBRARY_DIR to a writable directory.(set by VUNIT_PYTHON_BRIDGE_LIBRARY_DIR).Linux, GHDL master llvm (LLVM 19):
run.pywas run from thetestsdirectory).ghdlwrapper droppingLD_LIBRARY_PATH: before,error while loading shared libraries: libvunit_python_bridge.so; after, passes.ghdlwrapper runningenv -i: fails withFailed to open the Python bridge configuration file (VUNIT_PYTHON_BRIDGE_CONFIG is not set).-Wl,), left to CI.Not tested locally: the Windows code in
config.c(left to the Windows CI jobs), and GHDL llvm/gcc (left to CI).Not in this PR
The Riviera-PRO/Active-HDL VHPI application is still built under
<output path>/<simulator>/librariesbyforeign_application.py, so it has the same problem. Moving it to the library directory belongs to the Riviera-PRO/Active-HDL support in #21.🤖 Generated with Claude Code