Define how to run tests

[furtib]

Why:
The requirements for running tests are not well defined.
This patch aims to define which targets should be run for testing.
This patch does not define how the user should set up their environment, only how the tests should be run. That is a todo for the next patch.

What:

• Added a script that runs all checks.
• Modified the CONTRIBUTING.md to specify which tests must be run

Addresses:
none?

[nettle]

Good start! Thanks @furtib!

[nettle]

I'm sure, before running anything we must crealy describe how to initialize the environment.
See Development Environment above

[nettle]

Hm... discussable...
Maybe we should setup correct environment in this script?

[Szelethus]

Set up through program arguments/environmental variables? I understand that our clientbase have their own bazel and CodeChecker distributions, so we can't generalize too much here.

This sounds like a reasonable compromise:
USE_BAZEL_VERSION=8.5.0 CODECHECKER_BIN_PATH=/path/to/codechecker/25 run_tests.sh

[nettle]

This is about Development Environment not Execution Environment since we run tests, right?
Here we have to find common ground, common environment that works for all possible contributors, starting from you and me, but extend to anyone. Which already means at least different versions of Ubuntu and RedHat, different version sets of Bazel and Clang etc.
That's exactly why I suggested Mise and/or Micromamba to solve this problem.
Both are still not ideal, but at least something. Maybe some alternatives are possible too.
(BTW, native bazel dependency manager may work, but we did not try)

[furtib]

We could very well set up the python virtual environment for testing in this script, but I would be hesitant to also do that for the llvm toolchain (maybe thats where im wrong?).

My first idea is whether a bazel provided llvm toolchain exist and can we use that?

My experience is that most of these crashes should be due to diagtool not being found by clang. (You can check the codechecker log for the failing analyzed file to confirm this, even though the plist files arent getting created the log still does)

[nettle]

We could very well set up the python virtual environment for testing in this script, but I would be hesitant to also do that for the llvm toolchain (maybe thats where im wrong?).

Right, unfortunately, venv (virtual environment) is not sufficient, that's exactly why I suggested mise and/or micromamba.

My first idea is whether a bazel provided llvm toolchain exist and can we use that?

Yes, it is theoretically possible!
Practically... I dont know, it might be difficult to find llvm package that we could use.

My experience is that most of these crashes should be due to diagtool not being found by clang. (You can check the codechecker log for the failing analyzed file to confirm this, even though the plist files arent getting created the log still does)

There are so many different failures depending on versions and environments, so did not check them.
We need to fix the environment first, then set of tests and the way we run them.
After that I can report and investigate fails.

[furtib]

I have investigated the possibility of using the bazel module: https://github.com/bazel-contrib/toolchains_llvm
And I found it to be surprisingly doable. I don't even think the big download size is a huge blocker, since it won't redownload it even after a bazel clean.
With that said, it introduces the complexity of whether we require all users to use it or not. (How do we test both methods if we do not require it?)

So instead, I tried this different approach, setting up the environment "manually".
I try not to set any dependency that we need to vary, like Bazel; I'm still figuring out how to handle that.
I chose this because it produces an environment that I think is more easily reproducible to end users compared to what Micromamba provides.

[nettle]

This looks to me as introduction of venv + llvm environment :)
It should go under .ci/ as yet another dev env along with mise and micromamba.
And it should be a separate review, not a part of "Define how to run tests".

[furtib]

Fair!
I will strip this back to just the script.

That raises some questions about whether the file test/test.sh is still being used in CI workflows, or if I can remove that entirely.

[nettle]

Fair! I will strip this back to just the script.

Actually I think venv or uv could be useful :)
But again as separate PRs

That raises some questions about whether the file test/test.sh is still being used in CI workflows, or if I can remove that entirely.

To be honest just adding such script does not bring much value without defining the environment I think.
Without defining reliable environment (micromamba, mise, venv, uv or something else)
we cannot say we have answer to the question "how to run tests?"

[nettle]

I have investigated the possibility of using the bazel module: https://github.com/bazel-contrib/toolchains_llvm And I found it to be surprisingly doable. I don't even think the big download size is a huge blocker, since it won't redownload it even after a bazel clean. With that said, it introduces the complexity of whether we require all users to use it or not. (How do we test both methods if we do not require it?)

This is actually interesting!
Can you please do a prototype? (as a separate PR)

So instead, I tried this different approach, setting up the environment "manually". I try not to set any dependency that we need to vary, like Bazel; I'm still figuring out how to handle that. I chose this because it produces an environment that I think is more easily reproducible to end users compared to what Micromamba provides.

Well, setting up the environment "manually" sounds extremely unreliable.
To make sure we have "common grounds" for tests the setup must be automated.
But of course it starts from describing "manual" steps and conditions.