dh: reject generation controls wolfSSL cannot apply

[yosuke-wolfssl]

Problem

The DH generation interface advertises and accepts DH_GENERATOR and
DH_PRIV_LEN, stores them in wp_DhGenCtx, and never reads them — the struct
comments said "value ignored" outright. wolfCrypt has no landing spot for
either: wc_DhGenerateParams() takes only a modulus size and derives g
itself, and wc_DhGenerateKeyPair() derives the private length from q (or
from the size of p). A caller asking for a generator or a private-key length
therefore got success and backend-default output. Reachable from
openssl genpkey -pkeyopt dh_paramgen_generator:N / -pkeyopt priv_len:N and
from EVP_PKEY_CTX_set_params(). Closes 12485.

This is an API contract violation, not a crypto weakness: the generated g is a
valid generator of the order-q subgroup, and an ignored priv_len yields a
longer private key, never a shorter one.

Fix (src/wp_dh_kmgmt.c)

Requests that cannot be applied now fail; requests that are already satisfied
still succeed.

Request	Result
priv_len non-zero	PROV_R_NOT_SUPPORTED at set-params
priv_len 0	accepted (means "unset")
generator < 2	PROV_R_INVALID_DATA at set-params
generator 3, 5, … during parameter generation	PROV_R_NOT_SUPPORTED before the prime search
generator 2 during parameter generation	accepted, see limitation below
generator, named group or parameters key	accepted when it equals the group's g, else PROV_R_NOT_SUPPORTED

Both parameters stay in gen_settable_params() so the provider's own error
reaches the caller: delisting would make OpenSSL return -2 before calling us,
with nothing on the error queue, while a direct EVP_PKEY_CTX_set_params()
bypasses the settable list entirely and would be silently ignored again.

Known limitation: generator 2 is tolerated, not applied

apps/dhparam.c sets g = 2 whenever the caller passes none of -2/-3/-5, and
always forwards it, so openssl dhparam 2048 and openssl dhparam -2 2048 are
indistinguishable to a provider. Refusing 2 broke every dhparam generation
run, so it is accepted — but wolfSSL still derives its own generator, and the
parameters come back with a full-size g, not 2. Values that can only have come
from the user (3, 5) are refused rather than silently replaced.

Tests

test_dh_pgen_controls covers both controls over parameter and key generation:
generator 5 refused by paramgen and 2 accepted, generator 1 refused when set,
generator 2 and 5 accepted/refused against both a named group and a parameters
key, and priv_len 256 refused / 0 accepted.

Verification

• Build clean, no new warnings.
• Unit suite 225/225 pass.
• Negative control: each assertion fails against the pre-fix provider.
• CLI: openssl dhparam 2048 and -2 2048 generate parameters; -3 and -5
  fail with wp_dh_gen_parameters:not supported; genpkey -genparam unchanged.

[Copilot]

🟡 Changes recommended

The new DH control test can produce false-negatives by not asserting that setting a valid generator succeeds before expecting paramgen to fail.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR tightens the DH generation API contract by rejecting DH generation controls (DH_GENERATOR, DH_PRIV_LEN) that wolfSSL cannot actually honor, and adds unit coverage to ensure callers see deterministic failures instead of silent backend defaults.

Changes:

• Reject non-zero DH_PRIV_LEN at set-params time with PROV_R_NOT_SUPPORTED.
• Validate and enforce DH_GENERATOR behavior: reject invalid (<2), reject during parameter generation, and only accept during keygen when it matches the group/template generator.
• Add unit tests covering generator/priv_len behavior across paramgen and keygen flows.

File summaries

File	Description
test/unit.h	Registers the new DH control test declaration.
test/unit.c	Adds the new test case to the unit test table.
test/test_dh.c	Adds coverage for DH generator and private-length controls across paramgen/keygen.
src/wp_dh_kmgmt.c	Enforces rejection/acceptance rules for unsupported DH generation controls.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #486

Scan targets checked: wolfprovider-bugs, wolfprovider-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #486

Scan targets checked: wolfprovider-bugs, wolfprovider-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.