Skip to content

fix: configure HTTP connection pools and surface CDA batch errors - #312

Merged
krowvin merged 10 commits into
mainfrom
fix/endpoint-error-handling
Sep 28, 2026
Merged

krowvin merged 10 commits into
mainfrom
fix/endpoint-error-handling

Conversation

@krowvin

@krowvin krowvin commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Internal HTTP CDA loads use Requests' default pool of 10 because cwms-python previously mounted its configured adapter only on HTTPS. Multi-series operations can exceed that pool, producing repeated connection-discard warnings. This PR applies the configured pool and retry policy to both protocols and surfaces request/batch failures with their original diagnostic details.

  • Mount the adapter for HTTP and HTTPS in the initial session and init_session(). Both use the default pool size of 100 or the caller's pool_connections value.
  • Include HTTP status, method, URL, and CDA response body in errors, including custom user-management errors. Preserve exception causes and route role deletion through the shared API layer. Avoid the misleading "empty query" hint for failed writes returning 404.
  • Raise BatchError (a RuntimeError subclass) for failed concurrent reads/writes, retaining every original exception in failures. Propagate extent lookup failures and invalid JSON; restrict chunk retries to transient failures. Successful writes are not rolled back.
  • Narrow rating-validation catches and document pooling, errors, and debug logging. Request diagnostics omit request bodies and authentication headers.

Batch script evidence

During September 28, 2026 MVP CWMS Batch Events testing of a CLI time-series load into dev, the job's CDA_API_ROOT was confirmed to use an unencrypted internal HTTP URL, structurally:

CDA_API_ROOT=http://internal-cda.example/cwms-data/

The hostname above is a placeholder. Configuring only the HTTPS adapter leaves this HTTP destination on Requests' default pool. The existing Batch run matched 147 time series and logged repeated Connection pool is full ... Connection pool size: 10 warnings during concurrent stores.

The run also reported three failed stores with HTTP 404 database responses and empty error details. These are separate request failures: the pool warning concerns connection reuse and does not itself indicate discarded response data. Preserving the response cannot reconstruct server internals that CDA did not return.

Batch marked the run successful with exit code 0 because cwms-cli caught the aggregate store exception and only printed it. That process-exit behavior is addressed in cwms-cli #279; the library must raise exceptions rather than terminate its caller.

Validation

  • 223 mock/doctest tests passed on Python 3.14; strict mypy passed for 41 source files.
  • Black, isort, and git diff --check passed.
  • Six new cases fail before this update: HTTP default/custom pool sizes and 404 hints for POST/PATCH/DELETE. All pass with the fix.
  • Pool regressions exercise the real urllib3 connection-return path with 30 connections, without network sockets, and check HTTP/HTTPS retry configuration.
  • Existing error regressions cover endpoint failures, partial concurrent reads/writes, retained causes, malformed responses, and chunk retry behavior.
  • Batch evidence above is from the existing deployment, inspected read-only. The patched library has not been deployed or rerun in Batch. No live CDA/database integration tests were run locally.

Related: #277, #287, and #310. Callers now receive errors where incomplete results or fallback text were previously returned; existing RuntimeError handlers still catch batch failures. HTTP now uses the same retry policy already configured for HTTPS.

@krowvin
krowvin requested a review from msweier September 10, 2026 12:35
@sonarqubecloud

sonarqubecloud Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
2 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Comment thread cwms/api.py Dismissed
Comment thread cwms/api.py Dismissed
@krowvin krowvin changed the title fix: surface endpoint errors and bump patch version to 1.0.9 fix: configure HTTP connection pools and surface CDA batch errors Sep 28, 2026
krowvin added a commit to HydrologicEngineeringCenter/cwms-cli that referenced this pull request Sep 28, 2026
A time-series copy can report failed stores and still exit 0, causing
AWS Batch to mark the job successful. Raise a ClickException with the
original error as its cause instead of printing the error and continuing
to the completion message. Existing fatal-service error handling remains
intact.

Observed during September 28, 2026 MVP Batch Events testing: a load into
dev used an unencrypted internal HTTP CDA_API_ROOT, matched 147 series,
and reported three failed stores, but Batch received exit code 0. The
HTTP connection-pool warning is addressed separately in [cwms-python
#312](HydrologicEngineeringCenter/cwms-python#312);
this companion change fixes the CLI process status.

Validation: 38 related loader and CLI error-handling tests passed on
Python 3.14. Regression cases demonstrate that both read and aggregate
write failures previously exited 0 and now exit 1 while retaining the
failure text. Successful, empty-data, and dry-run commands still exit 0.
Black, isort, and git diff --check passed. No patched Batch run or live
CDA writes were performed; the dev container was unavailable locally.
@krowvin
krowvin merged commit dd32eb1 into main Sep 28, 2026
11 of 12 checks passed
@krowvin
krowvin deleted the fix/endpoint-error-handling branch September 28, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants