feat(backend): global TLS CA bundle for outbound components [INFP-586]

[fatih-acar]

Summary

Operators running Infrahub behind a private PKI had to rebuild the Docker image to add their root CA: git trusted only the system store, and the git credential helper and S3 storage had no CA setting at all, while every other component carried its own. This PR adds one setting, INFRAHUB_TLS_CA_BUNDLE, that every outbound TLS connection honours, closes the three gaps, and documents how to mount a bundle into the stock image.

Resolves INFP-586 (split from INFP-434).

Key Changes

• Operators set a single INFRAHUB_TLS_CA_BUNDLE path and git, the HTTP client (webhooks, SSO, telemetry, task manager), Neo4j, the broker, the cache, S3, the trace exporter, log forwarding and LDAP all trust it. Precedence: component setting, then the global bundle, then the system store. A component with *_TLS_INSECURE is left alone.
• Git repositories over HTTPS work against a private CA: new INFRAHUB_GIT_TLS_CA_FILE and INFRAHUB_GIT_TLS_INSECURE, written to the global git config at task-worker startup and cleared when unset, so a persisted gitconfig cannot keep a stale value. INFRAHUB_GIT_TLS_INSECURE replaces the custom-image http.sslVerify=false recipe the docs used to teach.
• The git credential helper now honours the HTTP TLS settings when it calls the Infrahub API, so an internal address served with a private CA no longer breaks credential lookups.
• S3 object storage gains INFRAHUB_STORAGE_TLS_CA_FILE (alias AWS_CA_BUNDLE), passed to boto3.
• Both compose env blocks expose the new variables; no image change is needed, a read-only bind mount is enough.

Notable implementation details

• The global bundle is resolved once at config load by a Settings validator that fills every unset component CA field, so adapters keep reading their own section and the resolved values are what shows up when inspecting settings.
• The bundle is path-only and validated as a loadable PEM at startup in every process. A container that reads the setting without the file mounted refuses to start; the guide tells operators to mount into both the server and the task worker.
• The trace exporter inherits the global bundle only when its connection is already encrypted: on gRPC a CA bundle switches a plaintext exporter to TLS.
• The S3 driver now builds the boto3 resource itself, mirroring the library constructor, because that constructor offers no hook for a CA bundle.
• High-risk areas touched: task-worker startup (git config writes), storage driver init. No database, ACP, PTY or updater changes.
• Review follow-ups folded in: the pre-existing safe.directory write appended a duplicate entry on every worker start because a trailing --replace-all is parsed by git as a value pattern; it now uses the helper's replace_all flag. The configuration reference generator dropped optional scalar settings nested in a sub-section (the new INFRAHUB_STORAGE_TLS_CA_FILE was invisible) and showed None as the type of every optional setting; both are fixed and the reference regenerated.

Open question for reviewers: naming overlap with the SDK's INFRAHUB_TLS_CA_FILE

The Python SDK's Config uses the INFRAHUB_ env prefix, so every SDK client built from the environment already reads INFRAHUB_TLS_CA_FILE and INFRAHUB_TLS_INSECURE (documented in the SDK reference). Inside the task-worker container that covers the infrahub CLI, infrahubctl, generators or transforms that instantiate Config() themselves, and the askpass helper. This PR introduces a server-side INFRAHUB_TLS_CA_BUNDLE with a broader meaning (every outbound connection), so two similarly named variables now coexist in the same container:

• INFRAHUB_TLS_CA_FILE (SDK): CA trusted when an SDK client talks to the Infrahub API. Not read by the server.
• INFRAHUB_TLS_CA_BUNDLE (server, this PR): CA trusted by every outbound connection of the server and task worker. Not read by SDK clients built from the environment, so setting only this variable leaves those SDK-based tools on the system trust store.

Options considered:

1. Rename the new setting to INFRAHUB_TLS_CA_FILE so one variable configures both. It is also the more accurate name: the global is path-only, and most components already use *_TLS_CA_FILE for a path while *_TLS_CA_BUNDLE is used where PEM text is accepted too. The cost is a scope expansion for anyone who already sets INFRAHUB_TLS_CA_FILE in the server environment to make the credential and askpass helpers reach a private-CA-served internal API: after the rename it would also apply to git, the database, the broker, the cache and S3. Those are the private-PKI users this feature targets, but it must be called out in the changelog.
2. Keep both names (this PR as opened) and document the difference in the private CA guide. No behavior change for existing SDK users, but the confusion stays and SDK-from-env tools in the container do not pick up the server bundle.
3. Use the ticket's INFRAHUB_CA_BUNDLE: avoids the TLS_ collision but keeps two variables with the same blindness as option 2.

The PR is opened with option 2 to keep the change reviewable; option 1 is the recommended follow-up. Related gap found during this review: git_credential/askpass.py still builds its SDK client from the environment without the HTTP TLS settings the credential helper now applies; it should reuse build_client_config() in the same follow-up.

Documentation Updates

• New guide: deploy-manage/install-configure/production-deployment/private-ca (Compose, Docker and Helm mounts, precedence, verification), registered in the sidebar.
• git-integration/connect-repository: the two custom-image sections now use the settings.
• production-deployment/overview: the TLS block shows the global bundle.
• reference/configuration regenerated; dev/knowledge/backend/tls.md added for contributors.

Test Plan

• New unit tests: config precedence and validation, git global-config writes and unsets, credential helper TLS context, S3 verify wiring.
• End-to-end: a bare repo served over HTTPS with a throwaway private CA is rejected without configuration, clones with git.tls_ca_file, and clones with git.tls_insecure.
• Git TLS error enrichment now also recognises the curl-GnuTLS 8.x wording ("server verification failed: certificate signer not trusted") the shipped image's git emits, so an untrusted certificate yields error-connection with the certificate hint instead of a generic error; all three wordings are covered by the enrichment test.
• Local gate: ruff, ruff format, ty, mypy, generated files, GraphQL/OpenAPI schema, docs validate, markdownlint, Vale, yamllint, compose env validation, biome, betterer all clean. Backend unit suite: 2446 passed; the 4 failures are the Prefect ephemeral-server tests, which pass with a fresh PREFECT_HOME (stale ~/.prefect database on the host). knip and the docs build could not run locally (Node 18 host, repo needs 24); sidebar id and links verified by script.
• Manual: mount a bundle, set INFRAHUB_TLS_CA_BUNDLE, then docker compose exec task-worker git config --file /opt/infrahub/.gitconfig --get http.sslCAInfo prints the path (the worker selects that file through GIT_CONFIG_GLOBAL in its own process, so --global from an exec shell reads a different file) and a repository on an internal Git server reaches Active.

[codspeed-hq]

Merging this PR will not alter performance

✅ 13 untouched benchmarks

---

_{Comparing fac/custom-ca-bundle-406td (312f0f0) with stable (85569da)}

[cubic-dev-ai]

1 issue found across 20 files

Confidence score: 4/5

• In backend/infrahub/storage.py, initialization permits AWS_S3_USE_SSL=False alongside a configured CA bundle, causing the endpoint to use plaintext while silently ignoring the CA setting; reject this incompatible configuration during initialization.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="backend/infrahub/storage.py">

<violation number="1" location="backend/infrahub/storage.py:46">
P2: When `AWS_S3_USE_SSL=False` and a component or global CA bundle is configured, this endpoint is plaintext and the CA setting is silently ignored. Reject this incompatible combination during initialization so operators do not believe S3 certificate verification is active.

(Based on your team's feedback about rejecting TLS settings that are ignored for plaintext endpoints.)</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When AWS_S3_USE_SSL=False and a component or global CA bundle is configured, this endpoint is plaintext and the CA setting is silently ignored. Reject this incompatible combination during initialization so operators do not believe S3 certificate verification is active.

(Based on your team's feedback about rejecting TLS settings that are ignored for plaintext endpoints.)

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At backend/infrahub/storage.py, line 46:

<comment>When `AWS_S3_USE_SSL=False` and a component or global CA bundle is configured, this endpoint is plaintext and the CA setting is silently ignored. Reject this incompatible combination during initialization so operators do not believe S3 certificate verification is active.

(Based on your team's feedback about rejecting TLS settings that are ignored for plaintext endpoints.) </comment>

<file context>
@@ -36,7 +40,20 @@ def __init__(self, **kwargs: Any) -> None:
+        # Mirrors fastapi_storages.S3Storage.__init__, which offers no hook to pass a CA bundle to boto3.
+        if self.AWS_S3_ENDPOINT_URL.startswith("http"):
+            raise ValueError("AWS_S3_ENDPOINT_URL should not contain the protocol")
+        self._http_scheme = "https" if self.AWS_S3_USE_SSL else "http"
+        self._url = f"{self._http_scheme}://{self.AWS_S3_ENDPOINT_URL}"
+        self._s3 = boto3.resource(
</file context>

Suggested change

	self._http_scheme = "https" if self.AWS_S3_USE_SSL else "http"
	if self.AWS_CA_BUNDLE and not self.AWS_S3_USE_SSL:
	raise ValueError("AWS_CA_BUNDLE cannot be combined with AWS_S3_USE_SSL=False")
	self._http_scheme = "https" if self.AWS_S3_USE_SSL else "http"

[fatih-acar]

Valid point — with use_ssl=false a configured CA bundle was passed to boto3 and silently ignored. An explicit INFRAHUB_STORAGE_TLS_CA_FILE / AWS_CA_BUNDLE with use_ssl=false is now rejected at startup by S3StorageSettings, the same rule the trace exporter applies to a plaintext endpoint, and the global INFRAHUB_TLS_CA_BUNDLE no longer fills S3 when the endpoint is plaintext, in c01f64f.

[cubic-dev-ai]

0 issues found across 7 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 5 unresolved issues from previous reviews.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 4 unresolved issues from previous reviews.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 8 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would require human review. This introduces a broad outbound TLS trust policy and git certificate-verification controls across services. Human review is needed for the security scope and unresolved SDK environment-variable naming tradeoff.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would require human review. Introduces a global outbound TLS CA bundle and git TLS settings across components; requires human sign-off on the security scope and the unresolved INFRAHUB_TLS_CA_BUNDLE vs SDK INFRAHUB_TLS_CA_FILE naming overlap.

Re-trigger cubic</sub>