Skip to content

improvement update test levels #853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RolandJentschETAS wants to merge 3 commits into
base: main
Choose a base branch
from
Open

improvement update test levels #853

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6aa7559
improvement update test levels
RolandJentschETAS Oct 26, 2026
63bb863
revert deletion of test methods
RolandJentschETAS Oct 29, 2026
4210c5a
fix review findings
RolandJentschETAS Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 29 additions & 19 deletions process/process_areas/verification/guidance/verification_process_reqs.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Process Requirements
.. gd_req:: Linking Requirements to Tests
:id: gd_req__verification_link_tests
:status: valid
:version: 1
:version: 2
:tags: prio_1_automation, verification
:satisfies: wf__verification_unit_test[version==1],
wf__verification_comp_int_test[version==1],
Expand All @@ -29,26 +29,26 @@ Process Requirements
:complies: std_req__iso26262__support_6432[version==1]


For linking test suites to requirements following metadata shall be used:
For linking test suites to requirements following metadata shall be used. Each field may contain multiple
comma-separated values from the applicable levels below:

* Verifies
* PartiallyVerifies
* FullyVerifies
* Description
* TestType
* Fault Injection (fault-injection)
* Interface Test (interface-test)
* Requirements-based Test (requirements-based)
* Resource Usage Evaluation (resource-usage)
* Fault Injection (fault-injection): unit, component, feature
* Interface Test (interface-test): unit, component, feature
* Requirements-based Test (requirements-based): component, feature, platform

@pahmann pahmann Oct 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember we discussed that a unit test can also cover some part of requirement, but maybe as it already uses linking other requirements it automatically transfers into a component level test.

* Resource Usage Evaluation (resource-usage): feature
* DerivationTechnique
* Analysis of requirements (requirements-analysis)
* Analysis of design (design-analysis)
* Analysis of boundary values (boundary-values)
* Analysis of equivalence classes (equivalence-classes)
* Fuzzy testing (fuzz-testing)
* Error guessing based on knowledge or experience (error-guessing)
* Explorative testing (explorative-testing)

* Analysis of requirements (requirements-analysis): all levels
* Analysis of design (design-analysis): all levels
* Analysis of boundary values (boundary-values): component, feature, platform
* Analysis of equivalence classes (equivalence-classes): all levels
* Fuzzy testing (fuzz-testing): unit, component, feature
* Error guessing based on knowledge or experience (error-guessing): all levels
* Explorative testing (explorative-testing): all levels

More information can be found in the :need:`gd_guidl__verification_guide`, :need:`doc_concept__verification_process`,
and :need:`gd_guidl__verification_specification` .
Expand Down Expand Up @@ -179,7 +179,7 @@ Process Requirements
.. gd_req:: Verification Documentation Checks
:id: gd_req__verification_checks
:status: valid
:version: 1
:version: 2
:tags: prio_1_automation, verification
:satisfies: wf__verification_unit_test[version==1],
wf__verification_comp_int_test[version==1],
Expand All @@ -191,10 +191,20 @@ Process Requirements

- TestType and DerivationTechnique shall be set
- Description shall not be empty
- In a Platform Integration Test Partially/FullyVerifies shall be set to at least one Platform Requirement
- If Partially/FullyVerifies are set in Feature Integration Test these shall link to at least one Feature Requirement or Architectural element
- If Partially/FullyVerifies are set in Component Integration Test these shall link to at least one Component Requirement or Architectural element
- If Partially/FullyVerifies are set in Unit Test these shall link to at least one Component Requirement

The test level is determined by the test's folder path. Verifies links shall use only the allowed types for that level:

- ``platform_integration_tests/`` (Platform): ``stkh_req``
- ``score/tests/`` (Feature): ``feat``, ``feat_req``, ``feat_arc_dyn``, ``feat_arc_sta``, ``mod``, ``mod_view_dyn``, ``mod_view_sta``, ``tool_req``

@pahmann pahmann Oct 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really play it that hard and make it mandatory?
Will it work with the idea that feature integration test can live in the reference integration and in the feature repository (we discussed this for better linking and testing depending on whether they link to architecture or requirements )

- ``score/.*/tests/`` (Component): ``comp``, ``comp_req``, ``comp_arc_dyn``, ``comp_arc_sta``, ``real_arc_int``, ``real_arc_int_op``
- Anything else (Unit): ``comp_req``, ``feat_req``

The minimum link coverage shall also be enforced:

- Platform Integration Tests shall link to at least one stakeholder requirement.
- If Partially/FullyVerifies are set in a Feature Integration Test, these shall use one or more allowed Feature links.
- If Partially/FullyVerifies are set in a Component Integration Test, these shall use one or more allowed Component links.
- If Partially/FullyVerifies are set in a Unit Test, these shall use one or more allowed Unit links.

.. gd_req:: Verification Documentation Checks Extended
:id: gd_req__verification_checks_extended
Expand Down
10 changes: 6 additions & 4 deletions process/process_areas/verification/verification_concept.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,9 @@ The rigor is described in the implementation of :need:`wp__verification_plan`.

Automated test cases (as well as manual, where applicable) shall contain further information about which methods have been applied.
The corresponding guidance is given here: :need:`gd_guidl__verification_guide`.
The identifier of the respective method is to be used as meta data (*TestType* and *DerivationTechnique*).
For Sphinx-Needs test metadata, use only the identifiers and test levels listed in

@AlexanderLanin AlexanderLanin Oct 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

process should not mention specific tools?!

:need:`gd_req__verification_link_tests`. Other verification methods described here remain applicable, but their
identifiers are not supported as *TestType* or *DerivationTechnique* values.

Test Case Development
=====================
Expand Down Expand Up @@ -155,9 +157,9 @@ Following aspect should be considered when developing test cases:
General Traceability Concept
============================

To allow a traceability of code to a written requirement, unit tests are linked to other unit
tests or component tests. This linking is done using metatags. This is also true for component
integration tests linking to the component requirements and architecture.
To allow a traceability of code to a written requirement, unit tests use ``Verifies`` links to component or feature
requirements as allowed by :need:`gd_req__verification_checks`. They may also link to related unit or component tests.
Component integration tests link to the component requirements and architecture as allowed by the same requirement.

Traceability of feature integration tests shall be established through linking those test cases to
feature requirements and architecture as features describe the integrated behavior of all components.
Expand Down
8 changes: 4 additions & 4 deletions process/process_areas/verification/verification_workflows.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ For a detailed explanation of workflows and their role within the process model,
:supported_by: rl__safety_manager[version==1]
:input: wp__sw_implementation[version==1], wp__verification_plan[version==1]
:output: wp__verification_sw_unit_test[version==1]
:contains: gd_req__verification_link_tests[version==1],
:contains: gd_req__verification_link_tests[version==2],
gd_req__verification_link_tests_cpp[version==1],
gd_req__verification_link_tests_python[version==1],
gd_req__verification_link_tests_rust[version==1],
Expand Down Expand Up @@ -71,7 +71,7 @@ For a detailed explanation of workflows and their role within the process model,
wp__requirements_comp_aou[version==1],
wp__verification_plan[version==1]
:output: wp__verification_comp_int_test[version==1]
:contains: gd_req__verification_link_tests[version==1],
:contains: gd_req__verification_link_tests[version==2],
gd_req__verification_link_tests_cpp[version==1],
gd_req__verification_link_tests_python[version==1],
gd_req__verification_link_tests_rust[version==1],
Expand Down Expand Up @@ -103,7 +103,7 @@ For a detailed explanation of workflows and their role within the process model,
wp__requirements_feat_aou[version==1],
wp__verification_plan[version==1]
:output: wp__verification_feat_int_test[version==1]
:contains: gd_req__verification_link_tests[version==1],
:contains: gd_req__verification_link_tests[version==2],
gd_req__verification_link_tests_cpp[version==1],
gd_req__verification_link_tests_python[version==1],
gd_req__verification_link_tests_rust[version==1],
Expand All @@ -128,7 +128,7 @@ For a detailed explanation of workflows and their role within the process model,
:supported_by: rl__safety_manager[version==1]
:input: wp__requirements_stkh[version==1], wp__verification_plan[version==1]
:output: wp__verification_platform_int_test[version==1]
:contains: gd_req__verification_link_tests[version==1],
:contains: gd_req__verification_link_tests[version==2],
gd_req__verification_link_tests_cpp[version==1],
gd_req__verification_link_tests_python[version==1],
gd_req__verification_link_tests_rust[version==1],
Expand Down
Loading