Fix buffer overruns in certificate signing and the EVP_PKEY populate paths - #11061
Merged
Conversation
|
Frauschi
force-pushed
the
fix-cert-evp-buffer-overruns
branch
from
August 5, 2026 14:17
617a569 to
d140776
Compare
wolfSSL-Fenrir-bot
left a comment
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #11061
Scan targets checked: wolfcrypt-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src
Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Findings are non-blocking.
Frauschi
force-pushed
the
fix-cert-evp-buffer-overruns
branch
from
August 5, 2026 15:16
d140776 to
faf8440
Compare
Frauschi
commented
Aug 5, 2026
wolfSSL-Fenrir-bot
left a comment
wolfSSL-Fenrir-bot
left a comment
There was a problem hiding this comment.
Fenrir Automated Review — PR #11061
Scan targets checked: wolfcrypt-bugs, wolfcrypt-rs-bugs, wolfcrypt-src, wolfssl-bugs, wolfssl-src
Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)
This review was generated automatically by Fenrir. Findings are non-blocking.
Contributor
Author
|
Jenkins retest this please |
dgarske
requested changes
Aug 6, 2026
Frauschi
force-pushed
the
fix-cert-evp-buffer-overruns
branch
from
August 6, 2026 06:47
faf8440 to
879c3f9
Compare
SignCert() checked the output buffer with
requestSz + MAX_SEQ_SZ * 2 + sigSz > buffSz
before handing the buffer to AddSignature(). That accounts for the outer
SEQUENCE header but not for the signatureAlgorithm AlgorithmIdentifier
(OID plus optional NULL parameters) or the signatureValue BIT STRING
header that AddSignature() also writes, an under-count of about 13 bytes.
AddSignature() takes no buffer size of its own, so any certificate whose
final encoding lands in that narrow band just under buffSz passed the
check and was written past the end of the buffer.
The same estimate was used in wc_SignCert_cb().
Both call sites now ask AddSignature() for the exact encoding size by
passing a NULL buffer first, then compare that against buffSz. This is
the two-pass idiom already used when signing CRLs in SignCrl(), in
wolfssl_x509_make_der() and in wolfSSL_X509_CRL_sign(). Both the template
and the original ASN.1 encoders support the NULL buffer sizing call.
The comparison is made unsigned, matching the pre-flight in SignCrl().
Casting buffSz to int made a buffer larger than INT_MAX compare negative
and rejected every signature for it.
Both functions also now bound requestSz against buffSz up front.
MakeSignature() and MakeSignatureCb() hash requestSz bytes out of buf
before any size check runs, so a caller passing the two mismatched got an
out of bounds read of up to requestSz - buffSz bytes before the function
returned. Only a negative requestSz was rejected before. Reaching this
needs the application to pass values that disagree, so it is API misuse
rather than attacker controlled input, but the read side now carries the
same guarantee as the write side.
Reachable from the OpenSSL compatibility layer through
wolfSSL_X509_sign() and wolfSSL_X509_REQ_sign(), where the caller
controls the certificate contents that steer the encoded size into the
band.
Adds test_wc_SignCert_buffer_bounds(), which signs into buffers sized
across the band below the exact encoding size and requires BUFFER_E and
an untouched guard region for each, while still accepting the exact size.
test_wc_SignCert_cb() gains the same check for the callback entry point,
using its RSA half where the PKCS#1 v1.5 signature is fixed length, in
both directions so that an over-conservative estimate is caught too.
The bounds test covers ECDSA as well as RSA. IsSigAlgoNoParams() drops
the NULL parameters from the AlgorithmIdentifier, so the width an
estimate under-counts by differs between the two: 24 bytes of wrapper
against the 12 byte estimate for RSA, but only 19 for ECDSA, putting the
capacities that used to be accepted and overrun within 8 bytes of the
exact size.
An ECDSA encoding size cannot be measured once and reused, because the
DER INTEGERs holding r and s change length with the leading zero bytes of
each new signature. The sweep measures a fresh reference size every
iteration and, rather than requiring BUFFER_E for a capacity that the
next signature might genuinely fit, asserts what has to hold either way:
the call returns BUFFER_E or a size within the capacity, and the guard
region past the capacity is untouched. That covers the whole band instead
of trading it away for a margin wide enough to absorb the jitter.
The prerequisites are split into one condition macro per algorithm rather
than one shared list. Gating the whole test on the RSA prerequisites
would have compiled the ECDSA sweep out of a build without RSA, which is
exactly where it is the only coverage that exists.
Both tests set an explicit serial number. wc_InitCert() leaves serialSz at
zero, so wc_MakeCert() generates a random serial, and GenerateInteger()
does not shrink the length after dropping leading zero bytes, which lets
the promoted byte carry the MSB and makes the encoder pad the INTEGER with
an extra 0x00. Measured over 200000 generated bodies, 813 of them, 0.406
percent, came out one byte longer, which would have made the swept
capacities disagree with the reference size for roughly one run in 128.
The RSA branch of wolfSSL_EVP_PKEY_keygen() passed &pkey->pkey.ptr
directly to wolfSSL_i2d_RSAPrivateKey():
pkey->pkey_sz = wolfSSL_i2d_RSAPrivateKey(pkey->rsa,
(unsigned char**)&pkey->pkey.ptr);
Following the i2d convention, wolfSSL_RSA_To_Der_ex() treats a non-NULL
*outBuf as a caller supplied buffer: it encodes into it with no size
check and then advances the pointer past the encoding. When ppkey points
at an EVP_PKEY that already carries a DER encoding, the generated private
key is written into that older, typically smaller allocation and
pkey.ptr is left pointing into the middle of it, which the eventual
XFREE() then trips over. Decoding a 2048-bit public key and calling
keygen on the same EVP_PKEY writes about 1190 bytes into a 294 byte
buffer.
The branch now installs the generated key on the pkey and hands the
encoding to PopulateRSAEvpPkeyDer(), which is what
wolfSSL_EVP_PKEY_set1_RSA() already does and what the sibling EC branch
does through ECC_populate_EVP_PKEY(). That function sizes the encoding
first and allocates its own buffer, so i2d is never shown a populated
pkey, and the three copies of the free, encode and assign sequence become
one. It also allocates against pkey->heap, which is the heap every site
that later releases pkey.ptr passes, while i2d deliberately allocates
with a NULL hint because its result is returned to the user.
The old RSA key is released before the new one is installed, which fixes
the previous unconditional overwrite of pkey->rsa leaking the old object,
and success is no longer reported when the encoding fails.
pkcs8HeaderSz is taken from the newly generated key rather than left as
it was. It describes the DER currently held in pkey.ptr, and
PopulateRSAEvpPkeyDer() adds a PKCS#8 wrapper only when the RSA key
carries a header size. d2i_PrivateKey(), d2i_AutoPrivateKey() and
PEM_read_bio_PrivateKey() set the field to 26 for a wrapped RSA key, and a
pkey obtained that way and then reused for keygen kept the 26 while the
encoding underneath was no longer wrapped. Every export path that trusts
the pair then sliced 26 bytes off the front of the new key:
wolfssl_i_evp_pkey_get_der() behind i2d_PrivateKey(), pkcs8_encode()
behind i2d_PKCS8PrivateKey(), and wolfssl_pkey_encrypt() behind
PEM_write_bio_PrivateKey(), returning a corrupt encoding under a success
return. wolfSSL_EVP_PKEY_set1_RSA() already maintains this field.
Adds test_wolfSSL_EVP_PKEY_keygen_reuse(), which runs keygen on an
EVP_PKEY populated from a public key DER and requires the resulting
encoding to decode back to the generated key, plus a second pass seeded
from a PKCS#8 key since the public key seed leaves pkcs8HeaderSz at zero
and cannot catch the stale header.
The test is gated on OPENSSL_EXTRA rather than OPENSSL_ALL, since nothing
it calls needs the latter, and on !NO_ASN and !NO_PWDBASED because
wolfSSL_i2d_PrivateKey() is compiled only under those. The PKCS#8 pass
additionally needs !NO_CERTS, which is what load_file() is gated on.
The forward declaration is gated on the same condition as the definition
rather than on WOLFSSL_KEY_GEN, which settings.h only happens to derive
WOLFSSL_KEY_TO_DER from today.
The sibling cases in the same switch are left alone deliberately. The DH
case does not free a previous pkey->dh, and the EC case promotes a
borrowed pkey->ecc to owned, both of which are the same ownership class
this change fixes for RSA. They are pre-existing, they need their own
tests, and folding them in here would widen a buffer overrun fix into a
rework of EVP_PKEY_keygen ownership across four algorithms.
PopulateRSAEvpPkeyDer() installs the newly allocated DER buffer on the EVP_PKEY before encoding into it, but only updates pkey_sz on the success path at the end. Every failure return in between left pkey_sz describing the previous encoding while pkey.ptr pointed at a buffer that holds no encoding at all and can be smaller than the old one. Callers such as wolfssl_i_evp_pkey_get_der() copy pkey_sz bytes out of pkey.ptr, so they would read past the new allocation. The reachable paths are wc_RsaKeyToDer() or wc_RsaKeyToPublicDer() failing after their size query succeeded, and, under HAVE_PKCS8, the PKCS#8 buffer allocation or wc_CreatePKCS8Key() failing. Reset pkey_sz when the new buffer is installed so a failure return leaves the pkey describing an empty encoding rather than a stale one. Verified by fault injection, having wc_RsaKeyToDer() fail whenever asked to write: wolfSSL_EVP_PKEY_set1_RSA() on a populated EVP_PKEY left pkey_sz at 1192 before this change and leaves it 0 after. pkcs8HeaderSz is cleared on the same return. It describes an offset into the encoding pkey_sz measures, and wolfssl_i_evp_pkey_get_der() already guards the subtraction of one from the other, but pkcs8_encode() and pkcs8_encrypt() in src/pk.c do not: with pkey_sz reset and a header size of 26 left over from a PKCS#8 wrapped predecessor, they would compute a length of 0 - 26 as a word32.
Frauschi
force-pushed
the
fix-cert-evp-buffer-overruns
branch
3 times, most recently
from
August 6, 2026 10:06
7a218ff to
c9499a5
Compare
dgarske
requested changes
Aug 6, 2026
Under WOLFSSL_NO_REALLOC, PopulateRSAEvpPkeyDer() and
ECC_populate_EVP_PKEY() emulate XREALLOC by allocating a buffer sized for
the NEW encoding and then copying pkey_sz bytes, the size of the OLD one,
into it:
derBuf = (byte*)XMALLOC((size_t)derSz, pkey->heap, DYNAMIC_TYPE_DER);
if (derBuf != NULL) {
XMEMCPY(derBuf, pkey->pkey.ptr, (size_t)pkey->pkey_sz);
Whenever the replacement key encodes shorter than the one already on the
EVP_PKEY the copy runs past the end of the new allocation. Putting a
public key on a pkey holding a 2048-bit private key copies 1192 bytes
into a 294 byte buffer.
The copy serves no purpose: both functions fill the new buffer with a
fresh encoding immediately afterwards. It is removed rather than bounded.
ECC_populate_EVP_PKEY() also gains the pkey_sz reset that
PopulateRSAEvpPkeyDer() already has, so a failure between the allocation
and the encoding cannot leave the size describing a buffer that holds no
encoding.
The outgoing buffer is now wiped with ForceZero() before it is
reallocated or freed. On a private key it holds a complete RSA or ECC DER,
so returning it to the allocator intact leaves the key recoverable from
the free pool through a later heap over-read, a core dump or a swap page.
wolfSSL_RSA_To_Der_ex() establishes the same convention two frames away.
wolfSSL_EVP_PKEY_free() gets the same treatment, since it releases that
buffer on every normal teardown, as does the PKCS#8 branch of
PopulateRSAEvpPkeyDer(), which frees the unwrapped PKCS#1 key on its
success path once the wrapped copy has been built.
In ECC_populate_EVP_PKEY() that covers all three sites which release the
previous encoding, the two private-key branches as well as the public
one. clearEVPPkeyKeys() leaves pkey.ptr in place, so a pkey decoded from
a private key still carries that DER when a public-only key replaces it.
The wipe there happens before the allocation, since XREALLOC consumes the
old pointer, and pkey_sz and pkcs8HeaderSz are dropped with the contents
so a failed allocation cannot leave either describing a buffer that no
longer holds an encoding.
Where the allocation of the new buffer fails, pkcs8HeaderSz is cleared
along with pkey_sz for the reason given in the previous commit.
ECC_populate_EVP_PKEY() clears pkcs8HeaderSz when it installs a public
key. A SubjectPublicKeyInfo has no PKCS#8 wrapper, but neither
wolfSSL_EVP_PKEY_set1_EC_KEY() nor clearEVPPkeyKeys() resets the field, so
putting a public key on a pkey decoded from a PKCS#8 EC key left the
export paths starting that many bytes inside the new encoding and
returning it short under a success return.
The traditional private-key branch needs the same reset. It runs whenever
the incoming EC key carries no header size of its own, a generated key for
instance, and writes a bare SEC1 ECPrivateKey. Seeding an EVP_PKEY from
certs/ecc-keyPkcs8.der and then calling wolfSSL_EVP_PKEY_set1_EC_KEY()
with a generated key made wolfSSL_i2d_PrivateKey() return 92 bytes
beginning in the middle of the private scalar rather than the 121 byte
encoding. Every export path is affected, including the PKCS#8 encryption
in wolfSSL_PEM_write_bio_PKCS8PrivateKey(), which encrypts that same
misaligned slice.
Adds test_wolfSSL_EVP_PKEY_set1_shrinking_der(), which replaces the key
on an EVP_PKEY with a public-only one for both RSA and ECC and requires
the stored encoding to shrink. The smoke-test job
opensslextra-norealloc-asan builds exactly this configuration under
AddressSanitizer, which is where the over-copy is caught.
The test gates each algorithm on its own prerequisites rather than on one
shared list. WOLFSSL_KEY_TO_DER is defined by settings.h only when RSA is
enabled, so requiring it for the whole test compiled the ECC half out of
any build without RSA, and that half is the only coverage the ECC
over-copy has. The ECC half is seeded from a PKCS#8 wrapped key so that
pkcs8HeaderSz starts non-zero, and its size assertion is exact rather than
a comparison against the previous size, so an export starting at a stale
header shows up as a mismatch rather than passing.
test_wolfSSL_EVP_PKEY_set1_EC_KEY_no_pkcs8() covers the private-key case.
It compares the encoding exported after the replacement against the one a
pkey that never held a wrapped key produces from the same EC key, so a
carried over header size shows up as a size and content mismatch.
WOLFSSL_CERT_SIGN_CB is set only by --enable-certsigncb, and that flag appears nowhere under .github/. wc_SignCert_cb() is therefore compiled by no CI job, and neither is test_wc_SignCert_cb(), the only test covering it. The buffer bounds check the preceding commits add to that function, and the test assertions that go with it, would have merged without anything building them. Added as its own entry in the os-check Linux config list, in sorted position. "minutes" is seeded at 8.3 from the sibling --enable-all entries rather than omitted: the omission defaults it to 1.0, which sorts an eight minute build last in a list scheduled longest-first and deals it into whichever shard is already fullest. It also suppresses the stale estimate annotation that would otherwise prompt the refresh. Replace it with the real number from the Minutes column of the first run. Verified by running the entry through the workflow's own driver, .github/scripts/parallel-make-check.py, with the CFLAGS the workflow applies at make time (-pedantic -Wdeclaration-after-statement -Wnull-dereference -Wno-overlength-strings -DTEST_LIBWOLFSSL_SOURCES_INCLUSION_SEQUENCE) and --private-dir=certs. The out-of-tree build produces no compiler warnings and make check reports 17 passed, 6 skipped, 0 failed. Both test_wc_SignCert_buffer_bounds and test_wc_SignCert_cb run rather than skip.
The DH case of wolfSSL_EVP_PKEY_keygen() assigned straight over
pkey->dh:
case WC_EVP_PKEY_DH:
pkey->dh = wolfSSL_DH_new();
A caller supplied EVP_PKEY can already hold a DH object.
wolfSSL_EVP_PKEY_set1_DH() takes a reference and sets ownDh, and
wolfSSL_EVP_PKEY_assign_DH() installs one outright, so keygen on such a
pkey dropped the only pointer the EVP_PKEY had to that object without
releasing its reference, and nothing freed it afterwards.
The case now generates into a temporary and frees the previous key when
the pkey owned it, which is the shape the RSA case in the same switch
uses.
Adds test_wolfSSL_EVP_PKEY_keygen_dh_reuse(), which loads DH parameters,
puts them on an EVP_PKEY with set1_DH so the pkey holds a reference, and
then runs keygen on that same pkey. The leak itself is not asserted by the
test: it needs an allocation tracker, and the smoke-test sanitize-asan job
provides one, since it builds with AddressSanitizer and sets no
ASAN_OPTIONS, so LeakSanitizer runs by default there. What the test does
locally is drive the path and show it stays free of double frees under
AddressSanitizer.
This is hardening rather than a fix for a reachable defect. Both branches below are wrong as written, but no entry point tested reaches them: wolfSSL_d2i_PrivateKey() and wolfSSL_d2i_PUBKEY() both leave pkey->ecc populated, so wolfSSL_EVP_PKEY_get1_EC_KEY() always takes its up_ref path. wolfSSL_EVP_PKEY_get1_EC_KEY() has a branch that builds an EC_KEY when the pkey does not carry one, caches it on the pkey and returns it. It did that without taking a second reference and without setting ownEcc, so the single reference the key was created with was handed to the caller while the pkey kept an unowned pointer to it. A caller releasing what get1 gave it, as the contract requires, would leave pkey->ecc dangling. The pkey now keeps the reference the key was created with and the caller gets one of its own. The same branch freed the key when neither DER load succeeded but left pkey->ecc pointing at it. That pointer is now cleared. wolfSSL_EVP_PKEY_keygen() set ownEcc on the EC path whether or not it had created the key, so a key placed on the pkey by something that did not transfer ownership would gain a second owner. Ownership is now claimed where the key is created. Adds test_wolfSSL_EVP_PKEY_get1_EC_KEY_reuse(), which releases the reference get1 returns and then calls get1 again. It covers the path a decoded pkey actually takes and pins the reference contract; it passes with and without the change above, which the comment on the test says plainly so it is not mistaken for a regression test.
d2i_make_pkey() replaces the key data, size and type of a caller-supplied
WOLFSSL_EVP_PKEY, but left pkcs8HeaderSz, mldsaOID, pkey_curve and
save_type describing the key the object held before.
A stale pkcs8HeaderSz is the damaging one. pkcs8_encode() and
wolfssl_i_evp_pkey_get_der() both encode from pkey.ptr + pkcs8HeaderSz,
so after
d2i_PrivateKey(EVP_PKEY_RSA, NULL, &p, pkcs8RsaDer);
d2i_PrivateKey_EVP(&pkey, &q, traditionalEccDer);
PEM_write_bio_PKCS8PrivateKey() reports success while wrapping the ECC
key with its first 26 bytes cut off, and the resulting PEM cannot be
read back.
Only the d2i_PUBKEY and d2i_PrivateKey_EVP routes are affected;
d2i_PrivateKey and d2i_AutoPrivateKey go through d2i_evp_pkey(), which
allocates a fresh object and recomputes the header size.
The same branch also drops the data and the key object of the previous
key without releasing either. pkey.ptr is overwritten with a fresh
allocation, and wolfSSL_EVP_PKEY_free() only disposes of the object
matching the type currently set, so the object of a key whose type has
since changed is never freed. The sequence above leaks the 1219 byte RSA
encoding together with the WOLFSSL_RSA and its bignums, 13 allocations
in all. The data is released after the new encoding has been copied in,
since the caller may be decoding out of it.
Clear the metadata and dispose of the previous key on the reuse branch,
so a reused object decodes to the same state as a new one, and add a
regression test comparing the PKCS#8 output of a reused key against a
freshly decoded one.
Frauschi
force-pushed
the
fix-cert-evp-buffer-overruns
branch
from
August 6, 2026 20:20
c9499a5 to
63dc0dc
Compare
dgarske
self-requested a review
August 6, 2026 20:22
dgarske
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix buffer overruns in certificate signing and the EVP_PKEY populate paths
Three heap buffer overruns plus a set of related
EVP_PKEYstate defects, all reproducible on master and none introduced by recent work.1.
SignCert()under-counts the signature wrapper. The checkrequestSz + MAX_SEQ_SZ * 2 + sigSz > buffSzcovers the outer SEQUENCE but not thesignatureAlgorithmAlgorithmIdentifier orsignatureValueBIT STRING header thatAddSignature()also writes. The real wrapper is 24 bytes for RSA and 19 for ECDSA against a 12 byte estimate, so certificates landing in that band were accepted and written past the end. Reachable fromwolfSSL_X509_sign()andwolfSSL_X509_REQ_sign(). Reproduced with plain RSA: an RSA-2048 leaf, RSA-2048 CA and 64 SANs writes 4099 bytes into the 4096 byteWC_MAX_X509_GENbuffer. Both call sites now askAddSignature()for the exact size with a NULL buffer first, the idiom already used inSignCrl().2.
EVP_PKEY_keygen()on a populatedEVP_PKEY. The RSA branch passed&pkey->pkey.ptrtowolfSSL_i2d_RSAPrivateKey(), which treats a non-NULL*outBufas a caller supplied buffer and encodes into it with no size check. Keygen on a pkey decoded from a 2048-bit public key writes about 1190 bytes into a 294 byte buffer and leavespkey.ptrinterior. The branch now goes throughPopulateRSAEvpPkeyDer(), aswolfSSL_EVP_PKEY_set1_RSA()and the sibling EC case already do.3.
WOLFSSL_NO_REALLOCpopulate paths.PopulateRSAEvpPkeyDer()andECC_populate_EVP_PKEY()allocated for the new encoding then copiedpkey_szbytes of the old one into it, 1192 into 294 when replacing a private key with a public one. The copy is redundant and is removed. Covered by theopensslextra-norealloc-asanjob.4.
EVP_PKEY_keygen()drops a DH key without releasing it. The DH case assigned straight overpkey->dh.EVP_PKEY_set1_DH()takes a reference andEVP_PKEY_assign_DH()installs one outright, so keygen on such a pkey dropped the EVP_PKEY's only pointer to that object without releasing its reference. It now generates into a temporary and frees the previous key when the pkey owned it, matching the RSA case in the same switch.5.
d2i_make_pkey()keeps the previous key's metadata, data and key object when reusing anEVP_PKEY. The reuse branch replaces the key data, size and type of a caller suppliedWOLFSSL_EVP_PKEYbut leavespkcs8HeaderSz,mldsaOID,pkey_curveandsave_typedescribing the key the object held before. The stalepkcs8HeaderSzis the damaging one:pkcs8_encode()andwolfssl_i_evp_pkey_get_der()both encode frompkey.ptr + pkcs8HeaderSz, so decoding a PKCS#8 RSA key and then reusing that object for a traditional ECC key makesPEM_write_bio_PKCS8PrivateKey()report success while wrapping the ECC key with its first 26 bytes cut off, and the PEM cannot be read back. Only thed2i_PUBKEYandd2i_PrivateKey_EVProutes are affected;d2i_PrivateKeyandd2i_AutoPrivateKeygo throughd2i_evp_pkey(), which allocates a fresh object and recomputes the header size. The same branch also drops the data and the key object of the previous key without releasing either:pkey.ptris overwritten with a fresh allocation, andwolfSSL_EVP_PKEY_free()disposes only of the object matching the type currently set, so the object of a key whose type has since changed is never freed. The sequence above leaks the 1219 byte RSA encoding together with theWOLFSSL_RSAand its bignums, 13 allocations in all. The reuse branch now clears the metadata and disposes of the previous key, so a reused object decodes to the same state as a new one. The data is released after the new encoding has been copied in, since the caller may be decoding out of it, and the call clearing the key object is underOPENSSL_EXTRAbecauseclearEVPPkeyKeys()is defined there whiled2i_make_pkey()also compiles underWOLFSSL_WPAS_SMALL.Also fixed:
pkey_szandpkcs8HeaderSzleft describing a buffer that no longer holds an encoding on the failure returns; a stalepkcs8HeaderSzin the traditional private branch ofECC_populate_EVP_PKEY(), making export paths return a short encoding under a success return; unsigned buffer comparison; arequestSzbound so the signature hash cannot over-read; andForceZero()on the private key DERs before they are reallocated or freed.One commit is hardening rather than a fix.
EVP_PKEY_get1_EC_KEY()has a branch that builds an EC_KEY when the pkey carries none, caches it and returns it without taking a second reference or settingownEcc, and on load failure frees it while leavingpkey->eccpointing at it;EVP_PKEY_keygen()claimedownEccwhether or not it created the key. Both are wrong as written, but no entry point checked reaches them, sinced2i_PrivateKey()andd2i_PUBKEY()both leavepkey->eccpopulated. The commit message says so.Tests
test_wc_SignCert_buffer_bounds,test_wc_SignCert_cb,test_wolfSSL_EVP_PKEY_keygen_reuse,test_wolfSSL_EVP_PKEY_set1_shrinking_der,test_wolfSSL_EVP_PKEY_set1_EC_KEY_no_pkcs8andtest_wolfSSL_d2i_PrivateKey_reuse_resets_state. Each was confirmed to fail against the unpatched library: restoring master'sasn.cfails both sweeps, master'sevp.caborts under ASan insp_to_unsigned_bin_len, dropping thepkcs8HeaderSzreset fails the shrinking-DER assertion, reverting theECC_populate_EVP_PKEY()assignment fails on a 92 byte encoding starting mid scalar, and reverting thed2i_make_pkey()reset fails thepkcs8HeaderSz == 0assertion. The last one compares the PKCS#8 output of a reused key against a freshly decoded one, which are byte identical with the fix in place.The reuse leak is asserted by the sanitizer jobs rather than by the test itself, since it needs an allocation tracker: any job building with AddressSanitizer and no
ASAN_OPTIONSruns LeakSanitizer, and the same test drives the path. It was measured directly by toggling only thed2i_make_pkey()hunk in a static build and running theevp_pkeygroup under a leak checker: 13 leaks and 22240 bytes without it, 0 leaks with it. The 13 allocations match what LeakSanitizer reports, and the direct leak is the size ofcerts/server-keyPkcs8.der.Two further tests do not assert their defect and are labelled as such in the source.
test_wolfSSL_EVP_PKEY_keygen_dh_reusedrives the DH path but cannot assert the leak, which needs an allocation tracker; the smoke-testsanitize-asanjob supplies one, since it builds with AddressSanitizer and sets noASAN_OPTIONS, so LeakSanitizer runs there by default.test_wolfSSL_EVP_PKEY_get1_EC_KEY_reusepins the get1 reference contract on the path a decoded pkey takes, and passes with and without the EC change.Verified with
--enable-all --enable-certsigncb --enable-debugunder ASan (make check17 passed, 6 skipped, 0 failed), an ECC-only--disable-rsabuild, and--enable-opensslextra -DWOLFSSL_NO_REALLOCunder ASan. All build without warnings. The last commit was additionally built under--enable-opensslextra -DWOLFSSL_NO_REALLOC, where its test compiles out, and under--enable-wpas=small, which is the build withoutOPENSSL_EXTRAthat reachesd2i_make_pkey().Two guard notes, since both cost coverage when they were wrong. The bounds test gates each algorithm on its own prerequisites, because
WOLFSSL_KEY_TO_DERandUSE_CERT_BUFFERS_2048imply RSA and a shared guard silently removes the ECDSA and ECC halves from RSA-less builds. The ECDSA sweep assertsBUFFER_Eor a size within the capacity plus an untouched guard region, rather than a fixed return, because the signature length varies per signing; a margin wide enough to absorb that jitter would sit outside the band entirely.CI
WOLFSSL_CERT_SIGN_CBis set only by--enable-certsigncb, which appears nowhere under.github/, sowc_SignCert_cb()and its test are built by no job today.