rsa: hold the key mutex on every shared-key wolfCrypt call

[yosuke-wolfssl]

Problem

wp_Rsa objects are reference-shared between operation contexts via wp_rsa_up_ref(), but only 4 of 13 wolfCrypt call sites took the per-key mutex.

RsaKey carries state, data, dataLen and dataIsAlloc, which every RSA call mutates — this is not gated on async. wc_RsaSSL_Verify() and wc_RsaPSS_Verify_ex() route through RsaPrivateDecryptEx(), which XMALLOCs into key->data and lets wc_RsaCleanup() free it. The locked sign path (RsaPublicEncryptEx()) calls wc_RsaCleanup() as well.

So the existing mutex only serialized sign against sign. A concurrent verify, encrypt or KEM operation on the same key raced that malloc/free pair — use-after-free and double-free, plus spurious BAD_STATE_E verification failures. Locking the public side is what makes the private-side lock mean anything.

Closes f-8702.

Fix (src/wp_rsa_sig.c)

Ten call sites now hold wp_rsa_get_mutex() across their wc_* call:

File	Sites
wp_rsa_sig.c	verify_pkcs1, verify_pss, verify_no_pad, verify_x931, verify_recover
wp_rsa_asym.c	wp_rsaa_encrypt (fail-closed ladder); wp_rsaa_decrypt no longer conditional on WC_RSA_BLINDING
wp_rsa_kem.c	wp_rsasve_generate, wp_rsasve_recover
wp_rsa_kmgmt.c	wp_rsa_validate around wc_CheckRsaKey

Size queries (wc_RsaEncryptSize) and key->n reads stay outside the lock — nothing writes n during an operation, and TSan confirms they are not racy.

Tests (test/test_rsa.c)

test_rsa_concurrent_ops runs four threads over one shared EVP_PKEY. All ten locked sites are exercised under contention, verified by instrumented call counts.

Verification

• ThreadSanitizer, wolfSSL and wolfProvider both instrumented: 34 data races and an abort without the locks, zero with them. A representative pair shows the mechanism exactly — the sign side holding the mutex, the verify side not:

Read  by T1: RsaPrivateDecryptEx <- wc_RsaSSL_Verify <- wp_rsa_verify_pkcs1
Write by T4: RsaPublicEncryptEx  <- wc_RsaSSL_Sign   <- wp_rsa_sign_pkcs1
             (mutexes: write M0)

• RSA suite 31/31 pass under TSan; build clean under the project's -Werror set.
• A plain build passes with or without the locks — the race needs a sanitizer to surface, which the existing TSan CI job provides.

Not in this PR

wp_ecdsa_sig.c has the same asymmetry (sign locks, verify does not); wp_ecx_sig.c and the ML-DSA paths warrant the same audit.

[Copilot]

🟡 Changes recommended

The new concurrency test’s key-size guard should enforce an exact modulus-size match to avoid passing the guard while later failing in RSA_NO_PADDING operations.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens RSA operations against shared-key concurrency races by consistently holding the per-key mutex around wolfCrypt calls that mutate RsaKey state, and adds a multi-threaded regression test to exercise the contended paths under ThreadSanitizer.

Changes:

• Wrap RSA verify paths (PKCS1, PSS, no-pad, X9.31, verify-recover) with wp_rsa_get_mutex() locking.
• Extend the same locking discipline to RSA encrypt/decrypt, RSASVE KEM encapsulate/decapsulate, and wc_CheckRsaKey() validation.
• Add test_rsa_concurrent_ops to drive one shared EVP_PKEY from multiple threads across the affected operations.

File summaries

File	Description
test/unit.h	Registers the new RSA concurrency test prototype.
test/unit.c	Adds the RSA concurrency test to the unit test list.
test/test_rsa.c	Implements a pthread-based concurrent RSA sign/verify/enc/dec/KEM test over a shared key.
src/wp_rsa_sig.c	Adds per-key mutex coverage for RSA verification-related wolfCrypt calls.
src/wp_rsa_asym.c	Ensures encrypt/decrypt operations hold the per-key mutex (including non-blinding builds).
src/wp_rsa_kem.c	Serializes RSASVE KEM wolfCrypt calls with the per-key mutex.
src/wp_rsa_kmgmt.c	Documents expanded mutex intent and locks wc_CheckRsaKey() under the per-key mutex.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #487

Scan targets checked: wolfprovider-bugs, wolfprovider-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.