Add llvm-snippy random test generator flow#2444
Conversation
Use the RV32ZC configuration parameter when constructing the ISA string passed to the cosimulator. This lets DV runs distinguish between Zca, ZcaZcb, ZcaZcmp and ZcaZcbZcmp configurations.
Add an optional RV32ZC Make variable that overrides the compressed extension subset from the selected Ibex configuration. Thread this value through regression metadata into compile and simulation option generation, and teach ibex_config.py to apply validated FIELD=VALUE overrides via --set. This keeps the default IBEX_CONFIG behaviour unchanged while allowing the DV flow to build and run the same configuration with a different RV32ZC subset when needed.
Add a small ELF-to-manifest conversion utility and a UVM test that can load the generated manifest into both the DUT memory model and the cosim memory before releasing fetch. This provides a generic path for running prebuilt ELF images without going through the existing raw binary loader. The test remains compatible with the existing +bin flow, but also accepts +elf_manifest=<path> for externally generated programs.
Add the riscv-dv extension pieces needed to generate a small wrapper program for llvm-snippy runs. This includes a custom assembly program generator, helper instructions for fixing up tp/ra, the user extension registration, and a metadata test entry. This commit only adds the riscv-dv/SV hooks. It does not add the Snippy runtime input files, RTL runner, CMake orchestration, YAML layouts, or coverage flow yet.
Add scripts for running pre-generated llvm-snippy test artifacts through the Ibex RTL simulation flow and for post-processing the resulting logs. The runner writes standard Ibex TestRunResult-compatible artifacts and registers them through snippy_tests.mk so later Ibex Make goals can discover the generated runs. This does not add the CMake orchestration, Snippy input files, YAML layouts, or coverage exclusions yet.
Add the llvm-snippy CMake flow and the Snippy input files used to run Ibex RTL simulations from generated ELF payloads. The flow wires together riscv-dv metadata generation, llvm-snippy invocation, linker/manifest generation, RTL simulation, log analysis, and aggregate CMake/Make targets. The Snippy directory contains the static assembly/linker inputs, YAML test layouts, and coverage exclusion files consumed by the flow.
Document the llvm-snippy flow from the core_ibex user-facing README, including the required tools, Make entry points, test selection knobs, and coverage targets.
AnastasiyaChernikova
force-pushed
the
add_snippy_flow
branch
from
722976b to
d2fc73f
Compare
|
Hi @AnastasiyaChernikova, thanks for putting this PR together! It is really interesting to see an llvm-snippy integration working with Ibex. As I mentioned over on the issue, we can't promise yet that we will be able to merge this as officially integrating a new generator would be a significant ongoing maintenance commitment for us. That being said, we really appreciate the effort, and it's a cool concept. Could you elaborate a bit more on the specific advantages of having Snippy as a second generator? For instance, what are some concrete things or scenarios that Snippy can handle better, or more easily, than riscv-dv? Also, regarding the Zcmp issue (#2445) mentioned in the description: as I noted in the issue thread, those mismatches are very likely just due to the mainline cosim environment not being fully updated for the experimental Zc* extensions yet. |
|
@SamuelRiedel Thank you very much for the comments and for taking a look at the PR. I fully understand the concern about maintenance cost. I am not proposing Snippy as a replacement for riscv-dv. Rather, I see it as an additional CI flow that could complement the existing riscv-dv-based regressions and provide a different instruction stimulus profile. In my opinion, Snippy can bring value to Ibex in several ways.
Snippy is built on a different infrastructure and uses a different approach to program generation compared to riscv-dv. As a result, it can generate tests with a different instruction distribution, different program structure, and different corner cases. For CI, this is useful as an additional independent check: even if Snippy runs in the same Ibex testbench/cosimulation environment, it may activate scenarios that do not appear as frequently in the main riscv-dv flow.
Snippy is based on LLVM target descriptions. If an ISA extension is already supported in LLVM, in many cases it does not need to be implemented again inside the generator. For example, Snippy documentation describes support/configuration for extensions and features such as B/Zb*, C/Zc*, F/D, V, Zicbom/Zicboz, Zicond, scalar crypto extensions Zk*, vector crypto extensions Zvkn*/Zvks*. This can be useful for Ibex if new extensions are added in the future: the generator-side effort is usually lower than manually adding support for a new extension to a separate generator description.
Snippy allows instruction histograms and patterns to be specified explicitly in YAML configurations. This is convenient for targeted tests where we want to focus on a specific instruction subset, ISA extension, or operation class.
One of Snippy’s strengths is the ability to control memory access patterns. It supports address ranges, stride/offset-based accesses, explicit addresses, burst-like memory sequences, and other memory schemes. This makes it possible to create denser and more directed stress on the load/store unit, the data memory path, and related pipeline scenarios.
Snippy allows branches, loops, and call graphs to be configured separately, including loop ratio, nesting depth, branch distance, function calls, and the overall structure of the generated program. This is useful when we do not just want random branch/jump instructions, but want to control the shape of the control-flow graph and the dynamic behavior of the program. In particular, this can be useful for stress testing the BPU and fetch/control-flow logic, which could be valuable for Ibex as well.
Snippy is also convenient for dependency-oriented scenarios. For example, the available register set can be restricted to increase the probability of RAW/dependency cases, or dedicated configurations can be created for hazard-oriented testing. This helps generate denser dependency patterns than a fully general random instruction stream.
Snippy is well suited for tests targeting rare operand values and special situations. For example, configurations can be built for specific operand values, sign-related cases, dense memory accesses, self-modifying code, and other scenarios that are difficult to obtain reliably by simply increasing the length of a random test.
Snippy generates regular executable artifacts that can be run in the existing Ibex DV environment, and the resulting coverage can be merged with coverage from riscv-dv and Snippy runs. The goal of the integration is not to rewrite the testbench or replace the riscv-dv infrastructure. The goal is to add another source of programs that go through the existing execution, cosimulation, and checking mechanisms. For CI, I think the practical option would be to add Snippy as a second or optional job with a limited set of configurations. Such a job could run a compact set of targeted tests: arithmetic/bitmanip, memory patterns, branches/loops/calls, dependency/hazard scenarios, and a few corner-case configurations. The generation time itself should not be a major CI bottleneck. Snippy is implemented in C++, and in our measurements it generated 50,000 instructions in under 2 seconds. Therefore, most of the additional runtime would come from cosimulation, not from test generation. If the maintainers think this direction is useful, I would be happy to help bring this PR to an acceptable state, prepare a compact set of Snippy configurations for CI, help integrate this flow into CI, and support it afterwards. I can also help investigate possible failures, update configurations when the Ibex DV flow changes, and keep the integration aligned with changes in Snippy. There is also a good upstream support story on the Snippy side: the project is actively developed, issues are usually handled quickly, and questions about the generator are answered promptly. Therefore, if generator-side problems appear, there is a realistic path to get help or fixes upstream. |
2 participants
This patch adds an experimental llvm-snippy integration to the Ibex DV flow.
The main motivation is to complement the existing riscv-dv tests with another generator that uses a different approach to program generation. riscv-dv remains the primary and well-integrated random instruction generator for Ibex, but llvm-snippy can be useful as an independent source of programs with a different generation profile, including different ways of controlling memory layout, call graph structure, register initialization, and instruction distribution. As a result, it can produce scenarios that are not necessarily well covered by the existing riscv-dv tests, or that would require additional dedicated riscv-dv configuration.
The practical value of this approach has already been demonstrated: during experiments with Snippy, an issue was found in the implementation of the Zcmp extension, described in issue #2445. This shows that an independent generator can be useful not as a replacement for riscv-dv, but as a complementary source of test programs that increases the chance of finding corner cases in new or complex ISA extensions.
The integration is implemented as a separate Snippy flow and should not change the behavior of regular riscv-dv runs. Snippy scenarios are described by separate YAML files, can be run through the Makefile bridge, and reuse the existing Ibex infrastructure for build, simulation, and coverage. This makes it possible to use Snippy as an additional tool for targeted/random testing without polluting the main riscv-dv flow.
In its current form, this patch adds infrastructure for running Snippy tests in Ibex:
The goal of this patch is not to replace the current regression tests, but to give Ibex DV one more independent and configurable way to generate programs for RTL simulation and coverage-driven exploration.
Coverage results for unprivileged isa only with llvm-snippy tests: