test: Use initial and latest tedge-p11-sever version in PKCS11 feature integration tests

[Bravo555]

Proposed changes

To ensure new versions of tedge remain compatible with older versions of tedge-p11-server, all PKCS11 features will be tested using both initial (i.e. first tedge-p11-server version that supports them) and latest tedge-p11-server versions.

This will supersede the previous approach of separate compatibility suite which only tested a subset of the features from version 1.5.1. With this new approach, all features will be tested for compatibility.

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
• I ran just format as mentioned in CODING_GUIDELINES
• I used just check as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
749	0	3	749	100	2h30m4.169284s

[didier-wenzek]

I guess that p11tool is used instead of tedge cert create-key-hsm because we want to run the tests even with an old version of tedge. Being explicit about that requirement would be better than commenting out the tedge command.

[didier-wenzek]

No 1.6.2 tag has been created and later removed.

Since this commit #2543:

• 1.6.2~300+g5cfe216 means in preparation of 1.6.2, 300 commits after the previous tag i.e. 1.6.1 in that case.

However, thin-edge documentation is not up to date and is still referencing the old scheme https://thin-edge.github.io/thin-edge.io/contribute/package-hosting/#version-syntax

[didier-wenzek]

The proposed test hierarchy is really clear (with a resource file and two test suites per feature - one for the initial reference release and one for the HEAD). I don't know if there is a better way to run with robot framework the same test suite on two different version of the software under test. But the current proposal nicely shares the test source code for the initial and current versions in a resource file per feature.

I will be happy to approve, but I would like first discuss the reference version to be used for the "create hsm keys" feature.

[didier-wenzek]

Even if 1.6.2~300+g5cfe216 is the precise commit which introduced the tedge cert create-key-hsm command, I would use the official release aka 1.7.0 (which retention policy is longer).

Suggested change

	${TEDGE_P11_SERVER_VERSION} 1.6.2~300+g5cfe216
	${TEDGE_P11_SERVER_VERSION} 1.7.0

[didier-wenzek]

Approved