diff --git a/process/process_areas/verification/guidance/verification_methods.rst b/process/process_areas/verification/guidance/verification_methods.rst index 9130e59845..dcc93de3e1 100644 --- a/process/process_areas/verification/guidance/verification_methods.rst +++ b/process/process_areas/verification/guidance/verification_methods.rst @@ -20,6 +20,74 @@ Verification methods and derivation techniques This guide helps to get an understanding what the different methods and derivation techniques mean and how to create test cases using the same. +.. _verification_quick_guide: + +Quick Guide: What to use when +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Each test case needs at least one ``TestType`` and one ``DerivationTechnique`` +(see :need:`gd_req__verification_link_tests`). +Multiple values are always possible for both and should be combined where it makes sense. + +The test types ``requirements-based``, ``interface-test``, ``fault-injection`` and ``resource-usage`` and the +derivation techniques ``requirements-analysis``, ``equivalence-classes``, ``boundary-values`` and ``error-guessing`` +correspond to the methods of ISO 26262-6 (software unit and software integration verification). +``design-analysis``, ``fuzz-testing`` and ``explorative-testing`` are additional S-CORE techniques. + +.. list-table:: Test Type + :header-rows: 1 + :widths: 25 20 55 + + * - TestType + - Applicable levels + - Use when + * - ``requirements-based`` + - unit, component, feature, platform + - A functional, process, tool or other non-functional requirement (not resource-related) is tested. + Units have no requirements of their own, but unit tests can verify component or feature requirements. + * - ``interface-test`` + - unit, component, feature + - An interface or interface requirement is tested. + * - ``resource-usage`` + - feature + - A non-functional requirement on resources (e.g. execution time, memory, CPU load, + communication bandwidth) is tested. + * - ``fault-injection`` + - unit, component, feature + - The behavior of the software in case of faults or failures of external parts influencing it is tested. + +.. list-table:: Derivation Technique + :header-rows: 1 + :widths: 25 20 55 + + * - DerivationTechnique + - Applicable levels + - Use when + * - ``requirements-analysis`` + - all + - Testing requirements (positive and negative cases). + * - ``design-analysis`` + - unit, component, feature + - Testing architectural and design aspects. + * - ``boundary-values`` + - all + - Interfaces, interface requirements or functional requirements contain values. + Can be combined with ``equivalence-classes``. + * - ``equivalence-classes`` + - all + - Interfaces, interface requirements or functional requirements contain values. + Can be combined with ``boundary-values``. + * - ``fuzz-testing`` + - unit, component, feature + - Interfaces, interface requirements or functional requirements contain values + which are tested with random values. + * - ``error-guessing`` + - all + - Known error-prone areas shall be tested, based on knowledge or experience. Fits all test types. + * - ``explorative-testing`` + - feature, platform + - The system and its boundaries are explored without predefined test cases. + Methods ^^^^^^^ @@ -178,10 +246,10 @@ Derivation Techniques * :ref:`Requirements analysis ` * :ref:`Boundary Values ` * :ref:`Equivalence Classes ` - * :ref:`Fuzzy Testing ` + * :ref:`Fuzz Testing ` Note: More than one derivation technique can be used in a single test case. - + .. _ver_req_anal: Requirements analysis @@ -199,7 +267,7 @@ be also considered: i.e. testing what happens when input x = C (if not already s in other requirements). Note that requirement-based derived test case may also include other techniques and derivation methods -mentioned here, such as Boundary Values, Equivalence Classes, Fuzzy Testing, Interface +mentioned here, such as Boundary Values, Equivalence Classes, Fuzz Testing, Interface Testing and Fault Injection. .. _ver_boundary: @@ -380,48 +448,48 @@ as well as during audits and assessments. .. _ver_fuzzy: -Fuzzy Testing -""""""""""""" +Fuzz Testing +"""""""""""" -Fuzzy Testing, often called Fuzz Testing, is a software testing technique that involves providing +Fuzz Testing is a software testing technique that involves providing invalid, unexpected, or random data as inputs to a component or system to detect defects. The goal -of fuzzy testing is to expose unhandled exceptions, memory leaks, assertion failures, and other +of fuzz testing is to expose unhandled exceptions, memory leaks, assertion failures, and other critical issues by generating semi-random inputs and observing the system's behaviour. The following perspectives are included in ``fuzz-testing``: **Detect Security Vulnerabilities** - By feeding malformed or unexpected inputs, fuzzy testing can reveal security vulnerabilities + By feeding malformed or unexpected inputs, fuzz testing can reveal security vulnerabilities like buffer overflows, SQL injections, and other input validation issues. **Uncover Edge Cases** - Fuzzy testing can discover edge cases that might not be covered by traditional testing methods, + Fuzz testing can discover edge cases that might not be covered by traditional testing methods, as it generates inputs that developers might not have anticipated. **Automate Input Generation** - Instead of manually creating test inputs, fuzzy testing automates the process, allowing for a + Instead of manually creating test inputs, fuzz testing automates the process, allowing for a much larger number of test cases to be executed. **Improve Robustness** - By exposing the system to a wide variety of inputs, fuzzy testing helps ensure that the + By exposing the system to a wide variety of inputs, fuzz testing helps ensure that the software can handle unexpected situations gracefully. -How to perform fuzzy testing: +How to perform fuzz testing: **1. Identify Target Functions** Determine which functions or interfaces are suitable for - fuzzy testing, focusing on those that process external inputs. + fuzz testing, focusing on those that process external inputs. **2. Define Input Structure** Understand the basic structure or format of valid inputs to create effective fuzzers that can produce semi-valid inputs. - **3. Generate Fuzzy Inputs** Create inputs that are slightly malformed or unexpected, either + **3. Generate Fuzz Inputs** Create inputs that are slightly malformed or unexpected, either through complete randomization or by mutating valid inputs. - **4. Monitor System behaviour** Execute the system with fuzzy inputs and monitor for crashes, + **4. Monitor System behaviour** Execute the system with fuzz inputs and monitor for crashes, hangs, memory leaks, or other unexpected behaviours. **5. Analyze and Fix Issues** When a problem is detected, analyze the input that caused it diff --git a/process/process_areas/verification/guidance/verification_process_reqs.rst b/process/process_areas/verification/guidance/verification_process_reqs.rst index c9cd7cfb69..b488a9bd54 100644 --- a/process/process_areas/verification/guidance/verification_process_reqs.rst +++ b/process/process_areas/verification/guidance/verification_process_reqs.rst @@ -20,7 +20,7 @@ Process Requirements .. gd_req:: Linking Requirements to Tests :id: gd_req__verification_link_tests :status: valid - :version: 1 + :version: 3 :tags: prio_1_automation, verification :satisfies: wf__verification_unit_test[version==1], wf__verification_comp_int_test[version==1], @@ -29,26 +29,26 @@ Process Requirements :complies: std_req__iso26262__support_6432[version==1] - For linking test suites to requirements following metadata shall be used: + For linking test suites to requirements following metadata shall be used. Each field may contain multiple + comma-separated values from the applicable levels below: * Verifies * PartiallyVerifies * FullyVerifies * Description * TestType - * Fault Injection (fault-injection) - * Interface Test (interface-test) - * Requirements-based Test (requirements-based) - * Resource Usage Evaluation (resource-usage) + * Requirements-based Test (requirements-based): unit, component, feature, platform -> for testing functional, process, tool or other non-functional requirements which are not resource-related (units have no requirements of their own, but unit tests can verify component or feature requirements) + * Interface Test (interface-test): unit, component, feature -> for testing interfaces or interface requirements + * Resource Usage Evaluation (resource-usage): feature -> for testing non-functional requirements on resources (e.g. execution time, memory, CPU load, communication bandwidth) + * Fault Injection (fault-injection): unit, component, feature -> for testing the behavior of the software in case of faults or failures of external parts influencing it * DerivationTechnique - * Analysis of requirements (requirements-analysis) - * Analysis of design (design-analysis) - * Analysis of boundary values (boundary-values) - * Analysis of equivalence classes (equivalence-classes) - * Fuzzy testing (fuzz-testing) - * Error guessing based on knowledge or experience (error-guessing) - * Explorative testing (explorative-testing) - + * Analysis of design (design-analysis): unit, component, feature -> for testing architectural and design aspects + * Analysis of requirements (requirements-analysis): all levels -> for testing requirements (positive and negative cases) + * Analysis of boundary values (boundary-values): all levels -> for interfaces, interface requirements or functional requirements containing values; can be combined with equivalence-classes + * Analysis of equivalence classes (equivalence-classes): all levels -> for interfaces, interface requirements or functional requirements containing values; can be combined with boundary-values + * Fuzz testing (fuzz-testing): unit, component, feature -> for interfaces, interface requirements or functional requirements containing values which are tested with random values + * Error guessing based on knowledge or experience (error-guessing): all levels -> for testing known error-prone areas; applicable for all test types + * Explorative testing (explorative-testing): feature, platform -> for exploring the system and its boundaries without predefined test cases More information can be found in the :need:`gd_guidl__verification_guide`, :need:`doc_concept__verification_process`, and :need:`gd_guidl__verification_specification` . @@ -179,7 +179,7 @@ Process Requirements .. gd_req:: Verification Documentation Checks :id: gd_req__verification_checks :status: valid - :version: 1 + :version: 2 :tags: prio_1_automation, verification :satisfies: wf__verification_unit_test[version==1], wf__verification_comp_int_test[version==1], @@ -191,10 +191,23 @@ Process Requirements - TestType and DerivationTechnique shall be set - Description shall not be empty - - In a Platform Integration Test Partially/FullyVerifies shall be set to at least one Platform Requirement - - If Partially/FullyVerifies are set in Feature Integration Test these shall link to at least one Feature Requirement or Architectural element - - If Partially/FullyVerifies are set in Component Integration Test these shall link to at least one Component Requirement or Architectural element - - If Partially/FullyVerifies are set in Unit Test these shall link to at least one Component Requirement + + The test level is determined by the test's location as configured for the tooling (e.g. per repository and folder), + so that tests of one level can reside in different repositories (e.g. feature repository or reference integration). + Verifies links shall use only the allowed types for that level. The following folder structure is the default: + + - ``platform_integration_tests/`` (Platform): ``stkh_req`` + - ``score/tests/`` in the feature repository (Feature) or ``feature_integration_tests/`` in the reference integration repository : ``feat``, ``feat_req``, ``feat_arc_dyn``, ``feat_arc_sta``, ``mod``, ``mod_view_dyn``, ``mod_view_sta``, ``tool_req`` + - ``score/.*/tests/`` (Component): ``comp``, ``comp_req``, ``comp_arc_dyn``, ``comp_arc_sta``, ``real_arc_int``, ``real_arc_int_op`` + - Anything else (Unit): ``comp_req``, ``feat_req`` (units have no requirements of their own, but unit tests can verify component or feature requirements) + + The minimum link coverage shall also be enforced: + + - Platform Integration Tests shall link to at least one stakeholder requirement. + - If Partially/FullyVerifies are set in a Feature Integration Test, these shall use one or more allowed Feature links. + Requirements and architecture links are equally allowed, none of them is mandatory. + - If Partially/FullyVerifies are set in a Component Integration Test, these shall use one or more allowed Component links. + - If Partially/FullyVerifies are set in a Unit Test, these shall use one or more allowed Unit links. .. gd_req:: Verification Documentation Checks Extended :id: gd_req__verification_checks_extended diff --git a/process/process_areas/verification/guidance/verification_specification.rst b/process/process_areas/verification/guidance/verification_specification.rst index 0a5ad1e956..41d3afbbd7 100644 --- a/process/process_areas/verification/guidance/verification_specification.rst +++ b/process/process_areas/verification/guidance/verification_specification.rst @@ -64,7 +64,6 @@ A test specification contains the following attributes. - requirements-based - interface-test - - coverage (various types apply, shall be tool supported) - These are example values and an incomplete list. A full list of test types is available in :need:`doc_concept__verification_process` at :ref:`verification_concept_types_methods`. diff --git a/process/process_areas/verification/guidance/verification_templates.rst b/process/process_areas/verification/guidance/verification_templates.rst index d9245a195b..ce7811eaf5 100644 --- a/process/process_areas/verification/guidance/verification_templates.rst +++ b/process/process_areas/verification/guidance/verification_templates.rst @@ -32,6 +32,12 @@ Verification Templates The sections below are seen as typical ways when writing tests and their specification. Their usage differs based on the selected testing framework and the implementation language of the component(s). + Allowed values for *TestType* and *DerivationTechnique* and a short guide when to use which are available in the + :ref:`verification_quick_guide`. Multiple values can be given for both. + + The Ids used in *PartiallyVerifies* and *FullyVerifies* are the Ids of component, feature or stakeholder requirements, + depending on the test level (see :need:`gd_req__verification_checks`). + C++ --- diff --git a/process/process_areas/verification/verification_concept.rst b/process/process_areas/verification/verification_concept.rst index 6f05409f67..175e481166 100644 --- a/process/process_areas/verification/verification_concept.rst +++ b/process/process_areas/verification/verification_concept.rst @@ -117,7 +117,7 @@ The derivation of test cases can also be based on certain methods. * Analysis of requirements (requirements-analysis) * Analysis of design (design-analysis) * Error guessing based on knowledge or experience (error-guessing) - * Fuzzy testing (fuzz-testing) + * Fuzz testing (fuzz-testing) * Explorative testing (explorative-testing) Usually the defined methods are not applied on each verification level between unit and platform level. @@ -126,7 +126,9 @@ The rigor is described in the implementation of :need:`wp__verification_plan`. Automated test cases (as well as manual, where applicable) shall contain further information about which methods have been applied. The corresponding guidance is given here: :need:`gd_guidl__verification_guide`. -The identifier of the respective method is to be used as meta data (*TestType* and *DerivationTechnique*). +For test metadata, use only the identifiers and test levels listed in +:need:`gd_req__verification_link_tests`. Other verification methods described here remain applicable, but their +identifiers are not supported as *TestType* or *DerivationTechnique* values. Test Case Development ===================== @@ -155,9 +157,9 @@ Following aspect should be considered when developing test cases: General Traceability Concept ============================ -To allow a traceability of code to a written requirement, unit tests are linked to other unit -tests or component tests. This linking is done using metatags. This is also true for component -integration tests linking to the component requirements and architecture. +To allow a traceability of code to a written requirement, unit tests use ``Verifies`` links to component or feature +requirements as allowed by :need:`gd_req__verification_checks`. They may also link to related unit or component tests. +Component integration tests link to the component requirements and architecture as allowed by the same requirement. Traceability of feature integration tests shall be established through linking those test cases to feature requirements and architecture as features describe the integrated behavior of all components. diff --git a/process/process_areas/verification/verification_workflows.rst b/process/process_areas/verification/verification_workflows.rst index cebef53885..292b662cef 100644 --- a/process/process_areas/verification/verification_workflows.rst +++ b/process/process_areas/verification/verification_workflows.rst @@ -29,7 +29,7 @@ For a detailed explanation of workflows and their role within the process model, :supported_by: rl__safety_manager[version==1] :input: wp__sw_implementation[version==1], wp__verification_plan[version==1] :output: wp__verification_sw_unit_test[version==1] - :contains: gd_req__verification_link_tests[version==1], + :contains: gd_req__verification_link_tests[version==3], gd_req__verification_link_tests_cpp[version==1], gd_req__verification_link_tests_python[version==1], gd_req__verification_link_tests_rust[version==1], @@ -71,7 +71,7 @@ For a detailed explanation of workflows and their role within the process model, wp__requirements_comp_aou[version==1], wp__verification_plan[version==1] :output: wp__verification_comp_int_test[version==1] - :contains: gd_req__verification_link_tests[version==1], + :contains: gd_req__verification_link_tests[version==3], gd_req__verification_link_tests_cpp[version==1], gd_req__verification_link_tests_python[version==1], gd_req__verification_link_tests_rust[version==1], @@ -103,7 +103,7 @@ For a detailed explanation of workflows and their role within the process model, wp__requirements_feat_aou[version==1], wp__verification_plan[version==1] :output: wp__verification_feat_int_test[version==1] - :contains: gd_req__verification_link_tests[version==1], + :contains: gd_req__verification_link_tests[version==3], gd_req__verification_link_tests_cpp[version==1], gd_req__verification_link_tests_python[version==1], gd_req__verification_link_tests_rust[version==1], @@ -128,7 +128,7 @@ For a detailed explanation of workflows and their role within the process model, :supported_by: rl__safety_manager[version==1] :input: wp__requirements_stkh[version==1], wp__verification_plan[version==1] :output: wp__verification_platform_int_test[version==1] - :contains: gd_req__verification_link_tests[version==1], + :contains: gd_req__verification_link_tests[version==3], gd_req__verification_link_tests_cpp[version==1], gd_req__verification_link_tests_python[version==1], gd_req__verification_link_tests_rust[version==1],