Fix RSA-PSS signing and RSA-OAEP decryption when using OpenSSL 3.x.

[olszomal]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Issue number: Fixes #638

Current Behavior

When using the PKCS#11 provider with OpenSSL 3.x, RSA-PSS signing and RSA-OAEP decryption do not work correctly with PKCS#11-backed keys.

New Behavior

RSA-PSS signing and RSA-OAEP decryption work correctly when using the PKCS#11 provider with OpenSSL 3.x.

Scope of Changes

• Register a custom RSA EVP_PKEY_METHOD wrapper for the PKCS#11 provider on OpenSSL 3.x
• Ensure correct support for RSA-PSS signing and RSA-OAEP decryption
• Preserve PKCS#11 key association via EVP_PKEY ex_data
• Properly initialize and clean up the global method and ex_data index
• Fix X509_check_private_key() usage in tests
• Relax version guards for Ed25519/Ed448 provider tests
• Fix compatibility for OpenSSL < 1.0.2m / LibreSSL

Testing

• Existing tests
• New tests added
• Manual testing

Additional Notes

License Declaration

• I hereby agree to license my contribution under the project's license.