TLS Extensions: add more extensions and improve current

[SparkiDev]

Description

Added support for record_size_limit in TLS 1.3 and TLS 1.2. Added compress_certificate support for TLS 1.3.
Added signed_certificate_timestamp TLS 1.2 send and TLS 1.2 and 1.3 receive. Accepts server_name in CertificateRequest.
Added API for setting signature algorithms for signature_algorithms_cert.

Tests added and interop performed where possible.

Testing

Regression tested TLS.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-dtls13

• FLASH: .text +256 B (+0.1%, 186,812 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +576 B (+0.1%, 781,484 B / 1,048,576 B, total: 75% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +512 B (+0.2%, 302,084 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +448 B (+0.1%, 332,256 B / 1,048,576 B, total: 32% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +512 B (+0.2%, 242,703 B / 262,144 B, total: 93% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +512 B (+0.2%, 303,044 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +512 B (+0.2%, 242,703 B / 262,144 B, total: 93% used)

linuxkm-standard

• Data: __patchable_function_entries +104 B (+0.2%, 49,872 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7
• linuxkm-pie
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11362

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11362

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

record_size_limit default ignores a CTX-level max_fragment_length · Logic errors

The guard that suppresses the default record_size_limit when the application asked for max_fragment_length only searches ssl->extensions, but wolfSSL_CTX_UseMaxFragment() stores the extension on ctx->extensions. Such a client advertises both; a peer following RFC 8449 Sect. 5 then ignores max_fragment_length, so it is never echoed and ssl->max_fragment never takes effect.

Related known finding #7004 (similar but distinct): Both concern record_size_limit support, but #7004 alleges the extension is entirely absent from registry parsing/emission, whereas this faults default-extension suppression when a CTX-level max_fragment_length is configured. The operations, root causes, and required patches differ.

Fix: Fall back to TLSX_Find(ssl->ctx->extensions, TLSX_MAX_FRAGMENT_LENGTH) as TLSX_MFL_Parse() does at src/tls.c:3387.

[wolfSSL-Fenrir-bot]

Compressed Certificate path skips the client's CertReqCtx release · Resource leaks on error paths

The compressed-certificate branch returns before the epilogue at src/tls13.c:11220 that pops and frees ssl->certReqCtx on a client. In-handshake client auth allocates a zero-length node (src/tls13.c:6388), and UseCompressedCertificate() permits compression for it, so the node stays allocated on the list for the life of the connection.

Fix: Perform the same client-side certReqCtx pop and free before returning from the compressed branch, when the send did not end in WANT_WRITE.

[wolfSSL-Fenrir-bot]

Server sets certCompAdvertised without sending compress_certificate, defeating the unsolicited-CompressedCertificate… · TLS protocol issues

ssl->certCompAdvertised = 1 is set with no isServer check, so a TLS 1.3 server sets it while parsing every ClientHello — before deciding whether to send a CertificateRequest (the only message in which a server advertises compress_certificate). The !ssl->certCompAdvertised guard in SanityCheckTls13MsgReceived therefore never fires server-side, and a client that was never asked for a certificate can drive DoTls13CompressedCertificate into a WOLFSSL_MAX_CERT_COMP_SZ allocation plus wc_DeCompress.

Related known finding #7644 (similar but distinct): Both allow an unadvertised peer-selected certificate-related protocol choice to pass a missing solicitation invariant, but #7644 parses unsolicited certificate_type values on the client while this incorrectly sets server certCompAdvertised and admits an unsolicited CompressedCertificate. The roles, operations, causes, and patches are separate.

Fix: Set certCompAdvertised only when the extension is actually emitted: for the client in the ClientHello, and for the server when the CertificateRequest is written.