Replace global RSA methods with per-context callbacks

[mtrojnar]

Pull Request Type

• Bug fix
• New feature
• Code style / formatting / renaming
• Refactoring (no functional or API changes)
• Build / CI related changes
• Documentation
• Other (please describe):

Related Issue

Fixes #672

Current Behavior

On OpenSSL 3.x, PKCS11_get_private_key() registers libp11's custom RSA
EVP_PKEY_METHOD process-wide with EVP_PKEY_meth_add0().

That registration changes method selection for unrelated software RSA keys.
Provider-backed keys created before the PKCS#11 key can subsequently fail in
operations such as X509_sign() with:

error:03000093:digital envelope routines:default_check:command not supported

Software RSA keys created afterward can also silently fall back to legacy
implementations instead of remaining backed by the default provider.

The registration also crosses an architecture boundary: the libp11 library
initializes deprecated OpenSSL method integration that belongs to the ENGINE
component.

New Behavior

PKCS11_get_private_key() no longer registers an RSA EVP_PKEY_METHOD
globally. Loading a PKCS#11 key therefore does not alter method selection for
unrelated keys.

A new per-context callback API lets integrations customize each returned key:

PKCS11_CTX_set_pkey_callback(ctx, callback_type, callback, user_data);

The callback receives borrowed PKCS11_KEY and EVP_PKEY objects and may
customize the returned key in place. The callback must not free either
object; an error aborts key retrieval and libp11 releases the EVP_PKEY.

The OpenSSL integrations use this callback independently:

• The ENGINE attaches its PKCS11_pkey_meths dispatch only to keys loaded by
  that ENGINE. This preserves native RSA-PSS and software-PSS-over-raw-RSA
  behavior without changing process-wide method selection.
• The provider creates P11_KEYDATA in its callback and transfers it to STORE
  through a provider-owned EVP_PKEY ex-data slot. Duplication and cleanup
  callbacks retain explicit ownership, and STORE detaches the slot before
  releasing the temporary carrier key.

The provider-owned slot avoids assuming that process-local ex-data indexes
are shared between the external libp11 library and the provider's embedded
libp11 copy. Provider operations continue through provider dispatch and do
not depend on ENGINE support.

Scope of Changes

• Add and export PKCS11_CTX_set_pkey_callback() and the
  PKCS11_PKEY_CALLBACK_GET_PRIVATE_KEY event.
• Preserve callback registration when UTIL_CTX creates or reloads its
  internal libp11 context.
• Remove process-wide RSA EVP_PKEY_METHOD registration and its global state
  from the libp11 library.
• Select ENGINE PKEY methods per loaded key from eng_* code.
• Integrate provider private-key loading through a provider callback and a
  reference-counted ex-data handoff to KEYMGMT.
• Add regression coverage for unrelated provider-backed software RSA keys,
  callback-selected ENGINE dispatch, raw-RSA-only ENGINE signing, and
  native-PSS-only signing through both ENGINE and provider dispatch.

This adds a public libp11 API symbol and callback type. No existing public
symbol or function signature is removed.

Testing

• Existing tests
• New tests added
• Manual testing

Built with OpenSSL 3.6.3 using strict configuration and ran:

./configure --enable-strict
make
make check

Result: 51 total, 46 passed, 5 skipped due to unavailable ML-DSA/ML-KEM
features, 0 failed.

The provider RSA-PSS test restricts SoftHSM2 to CKM_RSA_PKCS_PSS, verifies
that CKM_RSA_X_509 is unavailable, and signs with explicit SHA-256, MGF1,
and digest-length salt parameters. Its success therefore covers the target
PSS-only-device behavior directly.

The ENGINE scenario also sets the MGF1 digest and digest-length salt
explicitly, and runs against both raw-RSA-only and native-PSS-only SoftHSM2
configurations.

A clean strict build with CPPFLAGS=-DOPENSSL_NO_ENGINE also succeeded.
Focused tests in that configuration produced 3 provider passes
(RSA-PSS, RSA-OAEP, and EC signing) and 1 expected ENGINE-dependent skip.

Additional Notes

ENGINE-specific APIs remain confined to eng_* sources. The provider does
not route operations through ENGINE, preserving the provider-only OpenSSL 4
architecture.

The provider callback implements key construction and ownership handoff.
Provider signature and asymmetric-cipher mechanism selection remain in the
existing provider operation implementation; this change does not add a
software-PSS-over-CKM_RSA_X_509 provider fallback.

License Declaration

• I hereby agree to license my contribution under the project's license.