Improved HTTP body handling

[Thom Chiovoloni (thomcc-work)]

This PR adds body limits to HTTP, as well as control over response retention. It works for all request types (df.http and df.http_multipart, both with and without df.endpoint).

The body limits are enforced through both max_request_bytes and max_response_bytes options (through df.with_http_options). The request check handles encoding, substitutions (secrets, variables), and enpoints. The response check is after decompression. Oversized bodies fail rather than being truncated.

Response handling can be done in one of 4 modes.

• inline: Body is returned and recorded in history, as before.
• metadata: Body is discarded, we return the size, metadata, and body sha256.
• discard: As before, but no hash (maybe not worth having?)
• sink: Stored in caller-provided table, history receives metadata and a row reference.

Headers can independently retain everything, use a 'safe' preset, select specific names, or retain nothing. Non-inline modes also omit body previews from 5xx errors.

The sink writes use the permission of the submitter, as well as RLS, target DB, and search path. The rows contain raw bytes and a fresh UUID per attempt, and commit before the reference is returned.

The defaults are the same as before. Responses are inline, keep all headers and add no caps. Existing SQL signatures are unchanged, and the extension schema is the same too.

Fixes #376.

[Copilot]

Copilot review overview

🟡 Changes recommended

The configured reqwest features do not support the documented and tested gzip decompression behavior.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

• Enable response decompression to enforce limits on decoded bytes · New

What changed in this PR

Extends pg_durable HTTP activities with configurable body limits and retention modes to keep sensitive payloads out of durable history.

Changes:

• Adds request/response limits, header filtering, and metadata/discard response modes.
• Adds SQL table sinks for response bodies.
• Adds extensive tests and documentation.

File	Description
USER_GUIDE.md	Documents body policies and sinks.
tests/​e2e/​sql/​75_http_sink.sql	Tests sink behavior and security.
tests/​e2e/​sql/​74_secret_bindings.sql	Tests limits with secrets.
tests/​e2e/​sql/​69_http_options.sql	Tests response modes and limits.
src/​types.rs	Defines body options and validation.
src/​secrets.rs	Applies limits to configured forms.
src/​orchestrations/​execute_function_graph.rs	Checks expanded request sizes.
src/​endpoints.rs	Secures sink execution context.
src/​dsl.rs	Extends HTTP DSL options.
src/​activities/​http_response.rs	Reads, filters, hashes, and stores responses.
src/​activities/​execute_multipart.rs	Enforces multipart limits and retention.
src/​activities/​execute_http.rs	Enforces ordinary HTTP policies.
docs/​upgrade-testing.md	Documents compatibility implications.
docs/​http-security.md	Documents security considerations.
docs/​api-reference.md	Documents the new API options.
CHANGELOG.md	Records the feature.
Cargo.toml	Adds HTTP and hashing dependencies.
Cargo.lock	Locks new dependencies.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Security-sensitive retention, RLS, cross-database writes, and replay behavior warrant final human validation.

Review effort: Balanced
Findings: 1

Open (1)

• Correct PR description to state non-inline bodies are omitted · New

Resolved since last review (1)

• Enable response decompression to enforce limits on decoded bytes

[Copilot]

Copilot review overview

🔵 Needs a closer look

Security-sensitive HTTP enforcement, replay behavior, and transactional cross-database sinks warrant final human validation despite extensive tests.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Correct PR description to state non-inline bodies are omitted

[Pino de Candia (pinodeca)]

Reviewed with Astra, medium effort and 272k context, 12 experts/AI personas, using ATV starter kit. No defects found.

I also skimmed the implementation and read the documentation from an ergonomics perspective and this looks good to me.