Feature request: Allow optional test cases to be documented with a comment instead of being executed

[RikkeVestesen]

Background

In our testing process, some test cases are optional because the functionality is not necessarily supported by all systems.
In ITB, an assessment is normally based on the result of executing a test case session. However, when an optional test case covers functionality that the tested system does not support, there is no reason to execute a test session. As a result, there is currently no suitable way to document why the optional test case was not executed as part of the assessment.

Suggested feature

For optional test cases, we would like the user to have the possibility to provide a comment instead of executing a test case session.
The comment should document why the optional test case is not being executed.

The important requirements are:
• A comment can be provided as an alternative to executing an optional test case.
• When this alternative is used, the comment MUST be mandatory.
• The comment must be saved as part of the assessment/documentation for the test case.

Why is this important?

The purpose is to ensure that there is explicit documentation (needed for our certification processes) whenever an optional test case is deliberately not executed. We do not want optional test cases to simply remain untested without an explanation.

This would allow us to distinguish between:

1. An optional test case that has been successfully executed.
2. An optional test case that was executed but did not pass.
3. An optional test case that has not yet been executed.
4. An optional test case that was deliberately not executed, with a documented explanation.

Let us know, if you have any questions and we are looking forward to you feedback on this.

[costas80]

Hi @RikkeVestesen.

Supporting a special workflow for such cases could be foreseen along the lines of what you suggest. I'm wondering however whether you could not already cover all your needs with the existing capabilities. In brief:

1. Define your optional test cases as normal, non-optional ones.
2. Make a simple adaptation to them (a one-liner addition of a scriptlet call) that will present an interaction asking whether they will execute it or not. If they choose to not execute it then you can show a mandatory comment input (with Similar functionality for request elements as 'depensOn' on the endpoint config #255 available you can have dynamically presented inputs depending on a choice).
3. To highlight these specific test cases use a tag as opposed to the optional status (given Feature request : Filter by tags #250 you can also filter using tags so you can easily pinpoint them).
4. To avoid a party that would anyway execute all optional tests, needing to make the discussed choice every time, you can foresee e.g. a statement-level configuration property. If this is set to e.g. "Yes - execute all" then the scriptlet I mentioned completes without any prompt and just proceeds with the regular test steps. When set to "No - skip" you only see the presented interaction for a comment input. Also, when such tests are executed via REST API you can foresee input(s) that will have the same effect.

The thing is that these tests are optional but in the end they require mandatory action for your conformance process. You could argue that this makes them required, albeit with the possibility to complete with a comment input. You can then keep the comments feature as a supplementary means of providing further information.

With this approach in place:

1. An optional test case that has been successfully executed.

Achieved. Also with the extra config options we discussed you can even skip superfluous interactions.

2. An optional test case that was executed but did not pass.

Achieved. See also previous point.

3. An optional test case that has not yet been executed.

Achieved. These are just the tagged "optional" test cases that have not been executed.

4. An optional test case that was deliberately not executed, with a documented explanation.

Achieved. These are actually executed as usual and set to capture the provided explanation.

From my perspective this seems to cover all points, and is immediately available without delay or effort from our end. It also avoids workarounds implementation-wise on our end whereby these optional tests are not ignored when calculating the conformance status (as they are currently). From a test development perspective (i.e. your perspective) this seems trivial, whereas for your users there remains one consistent approach for everything.

What do you think?