Repository navigation
fix(upstream): verify against ssl_trusted_certificate when only tls.verify is set - #14113
Conversation
…erify is set nginx loads proxy_ssl_trusted_certificate into the upstream SSL context only when proxy_ssl_verify is on. With it off (the default), enabling tls.verify per upstream without tls.ca_certs verified the certificate against an empty store, so every upstream failed with "unable to get local issuer certificate". Load apisix.ssl.ssl_trusted_certificate into a trusted store once per worker and apply it when tls.verify is true and tls.ca_certs is unset.
Workers may run as a different user than the master, which reads ssl_trusted_certificate for nginx itself. Loading the store in http_init lets workers inherit it and turns a load failure into a startup error instead of a 503 on the first request.
Assert that the route is created, so the case cannot pass against a route left over from a previous case.
membphis
left a comment
membphis
left a comment
There was a problem hiding this comment.
Blocking P1: applying the default CA store currently replaces the NGINX verification store and drops configured upstream CRL enforcement. Please preserve the existing revocation policy and add a real-gateway regression before merging. Details are attached to apisix/upstream.lua.
| store, err = trusted_store_cache(tls.ca_certs, up_conf.resource_version, | ||
| create_trusted_store, tls.ca_certs) | ||
| elseif tls.verify then | ||
| store = default_trusted_store |
There was a problem hiding this comment.
[P1] Preserve configured CRL checks when applying default CAs
When proxy_ssl_verify on and proxy_ssl_crl are configured, an upstream with tls.verify: true and no tls.ca_certs previously used NGINX's CA+CRL store. This branch now replaces it with default_trusted_store, which is created using only store:load_file(path). The CRL entries and X509_V_FLAG_CRL_CHECK | X509_V_FLAG_CRL_CHECK_ALL flags are lost, so a certificate signed by a trusted CA but already revoked can pass verification.
NGINX loads the CRL and its flags into the original store in ngx_ssl_crl. The runtime applies the replacement using SSL_set1_verify_cert_store, rather than merging it with the existing store.
An isolated in-memory TLS handshake experiment with OpenSSL 3.6.3 confirmed the mechanism for the same revoked certificate:
NGINX-style CA+CRL store: verify_result=23 (certificate revoked)
CA-only replacement store: verify_result=0 (ok)
This was a standalone OpenSSL experiment, not a full gateway reproduction. Please load the default CAs into the existing SSL_CTX store, or otherwise preserve its CRLs and verification flags when applying the fallback. Add a real-gateway regression with proxy_ssl_verify on, proxy_ssl_crl, and upstream tls.verify: true without ca_certs, asserting that a valid certificate returns 200 and a revoked certificate still returns 502.
There was a problem hiding this comment.
Fixed in the latest revision. The Lua default-store replacement has been removed. APISIX now uses runtime 1.4.1, which includes api7/apisix-nginx-module#137: trusted CAs and CRLs are loaded into NGINX's existing upstream SSL context even when static verification is off, preserving the CRL entries and verification flags. This does not enable verification or change the verification depth when static verification is off.
Added t/node/upstream-tls-trusted-ca.t coverage for both HTTPS and gRPCS, including static verification on with a configured CRL and tls.verify: true without tls.ca_certs: a valid certificate returns 200, and a revoked certificate returns 502 with certificate revoked. The regression suite passes in CI. Explicit tls.ca_certs retains its existing behavior.
…verify-default-ca
…copy Replacing the verify store from Lua drops the CRL and CRL check flags that nginx puts on its own store, so with proxy_ssl_verify on and proxy_ssl_crl configured a revoked upstream certificate passed tls.verify. The runtime now loads proxy_ssl_trusted_certificate and the CRL into the upstream SSL context without proxy_ssl_verify on, so the default trusted store is no longer needed. Render grpc_ssl_trusted_certificate next to proxy_ssl_trusted_certificate so grpcs upstreams get the same trusted certificates.
Description
An upstream that sets
tls.verify: truewithouttls.ca_certsfails withupstream SSL certificate verify error: (20:unable to get local issuer certificate)for every upstream certificate, including ones signed by a CA listed inapisix.ssl.ssl_trusted_certificate. Turning onproxy_ssl_verifyin nginx makes the same request succeed.Root cause: nginx loads
proxy_ssl_trusted_certificate(andproxy_ssl_crl) into the upstream SSL context only whenproxy_ssl_verifyis on at configuration time. APISIX rendersproxy_ssl_trusted_certificatebut leavesproxy_ssl_verifyoff, so whentls.verifyenables verification per request, the handshake is verified against an empty trusted store. gRPC has the same problem, and APISIX does not rendergrpc_ssl_trusted_certificateat all.An earlier revision of this PR loaded
ssl_trusted_certificateinto a Lua X509 store and applied it withset_ssl_trusted_store. That replaces nginx's store, so a CRL configured withproxy_ssl_crl(together withproxy_ssl_verify on) stopped being checked and a revoked certificate was accepted. That approach is dropped.The fix now lives in the runtime: api7/apisix-nginx-module#137 loads the trusted certificates and the CRL into the upstream SSL context whenever
*_ssl_trusted_certificateis set. This turns on neither verification nor*_ssl_verify_depth, which still applies only with*_ssl_verify on. With that, this PR:grpc_ssl_trusted_certificatenext toproxy_ssl_trusted_certificate, so grpcs upstreams get the same trusted certificates;APISIX_RUNTIMEto1.4.1(apisix-nginx-module1.20.1, which contains fix(upstream): load trusted certificates and CRL without static verify api7/apisix-nginx-module#137) and updates the pinned checksums of theapisix-runtime-debugpackages inci/linux-install-openresty.sh.tls.ca_certsis not set.tls.ca_certskeeps its current behaviour: it replaces the verify store for that upstream.Upgrade note: before 3.19.0,
tls.verifywas not applied to HTTP upstreams. After upgrading, upstreams that setverify: trueand present a certificate from an untrusted CA are rejected. In addition,grpc_ssl_trusted_certificateis now rendered at server scope fromssl_trusted_certificate. Likeproxy_ssl_trusted_certificate, it overrides the same directive set inhttp_configuration_snippet. A deployment that configured a separate gRPC CA there has to add that CA tossl_trusted_certificate.Tests:
t/node/upstream-tls-trusted-ca.t(new):httpsandgrpcsupstreams, withproxy_ssl_trusted_certificate/grpc_ssl_trusted_certificateandproxy_ssl_crl/grpc_ssl_crlconfigured in nginx:tls.verify: true: trusted certificate → 200, revoked → 502(23:certificate revoked), untrusted CA → 502tls.verifyunset: all three → 200 (unchanged)tls.verify: true: trusted → 200, revoked → 502. This covers the case where the Lua store dropped the CRL.tls.verifyunset: revoked → 502tls.verify: false, static verify on or off: untrusted CA → 200tls.verify: truewith static verify off → 200 (no depth limit is added); static verify on (default depth 1), withtls.verifyeithertrueor unset → 502On runtime 1.4.0, the trusted and revoked cases with
tls.verify: trueand static verify off fail. Both return(20:unable to get local issuer certificate).t/cli/test_upstream_mtls.sh: new case using the real renderednginx.confwith noproxy_ssl_verifysnippet.tls.verify: truereturns 200 whenssl_trusted_certificatecovers the upstream certificate and 502 when it does not. It also checks thatgrpc_ssl_trusted_certificateis rendered.Which issue(s) this PR fixes:
N/A
Checklist