From b5b4c1b72be4290e11c0973306eff17f193a1743 Mon Sep 17 00:00:00 2001 From: Atharva Naik Date: Wed, 7 Oct 2026 02:40:29 +0000 Subject: [PATCH 1/3] docs: document S3/GCS cloud storage tiers and the CAE S3 connector Covers the user-facing configuration surface of iowarp/clio-core#1093 (CLIO-S3 benchmark): the Poco+SigV4 S3 bdev, s3/gcs as CTE storage-tier types, the CAE S3 importer's in-process credential resolution, the cloud environment variables (AWS_*, S3_*, GCS_*), and CLIO_STRICT_RESPONSE_IDENTITY. Adds a dedicated S3 connector page modelled on the Globus one, and fixes the CTE storage-device-type list, which named types that do not exist. Co-Authored-By: Claude Opus 5.5 --- docs/deployment/configuration.md | 184 ++++++++++- docs/deployment/hpc-cluster.md | 12 + docs/deployment/performance.md | 1 + docs/sdk/context-assimilation-engine/omni.md | 7 +- .../context-assimilation-engine/overview.md | 20 ++ docs/sdk/context-assimilation-engine/s3.md | 308 ++++++++++++++++++ .../context-runtime/20.base-modules/2.bdev.md | 12 + docs/sdk/context-transfer-engine/cte.md | 11 +- 8 files changed, 543 insertions(+), 12 deletions(-) create mode 100644 docs/sdk/context-assimilation-engine/s3.md diff --git a/docs/deployment/configuration.md b/docs/deployment/configuration.md index bb8f6848..621b82bf 100644 --- a/docs/deployment/configuration.md +++ b/docs/deployment/configuration.md @@ -264,8 +264,8 @@ Block devices provide the shared memory allocator used by other modules. At leas | Parameter | Required | Description | |-----------|----------|-------------| -| `bdev_type` | Yes | `"ram"` for DRAM-backed, `"file"` for filesystem-backed. | -| `capacity` | Yes | Maximum capacity (e.g., `"512MB"`, `"100GB"`). | +| `bdev_type` | Yes | `"ram"` for DRAM-backed, `"file"` for filesystem-backed, `"s3"` for an Amazon S3 (or S3-compatible) bucket, `"gcs"` for a Google Cloud Storage bucket. `hbm`, `pinned`, and `noop` are also accepted. | +| `capacity` | Yes | Maximum capacity (e.g., `"512MB"`, `"100GB"`). For `s3` / `gcs` this is a logical cap on how many bytes the pool hands out — the bucket has no quota — and `0` means 1 TB. | ```yaml compose: @@ -286,7 +286,62 @@ compose: # capacity: "100GB" ``` -For DRAM devices the `pool_name` uses the `ram::` convention. For file-backed devices the `pool_name` is the filesystem path where data is stored. +For DRAM devices the `pool_name` uses the `ram::` convention. For file-backed devices the `pool_name` is the filesystem path where data is stored. For `s3` / `gcs` devices the `pool_name` names the bucket and a key prefix — see below. + +### Cloud object-store block devices (`s3`, `gcs`) {#cloud-bdevs} + +An `s3` or `gcs` block device stores every block as one object in a bucket, under the key `[/]block_`. It is usually reached through a [CTE storage tier](#storage-tiers-storage) rather than composed directly, but both work. + +```yaml +compose: + # Amazon S3 (or MinIO) bucket + - mod_name: clio_bdev + pool_name: "s3://my-bucket/clio/pool_0" # bucket "my-bucket", prefix "clio/pool_0" + pool_query: local + pool_id: "303.0" + bdev_type: s3 + capacity: "1TB" + + # Google Cloud Storage bucket + # - mod_name: clio_bdev + # pool_name: "gcs://my-bucket/clio/pool_0" + # pool_query: local + # pool_id: "304.0" + # bdev_type: gcs + # capacity: "1TB" +``` + +**`pool_name` format.** `[s3://|gcs://][/]`. The scheme is optional, everything after the first `/` is the key prefix, and trailing slashes are dropped. Use a prefix: CTE storage tiers append `_node` to the path, and only a prefix gives each node its own key space. + +**Behavior.** +- A block that was never written reads back as zeros, and an object shorter than the block is zero-filled. +- Freeing a block deletes its object (best effort). +- `s3` only: each runtime worker keeps one HTTPS keep-alive connection, so steady-state I/O does not reconnect per block. +- Credentials and endpoints come **only** from the runtime daemon's environment — there are no YAML keys for them. See [Cloud object stores](#cloud-object-stores) for every variable. + +**Bucket creation differs between the two.** + +| Type | Bucket missing at pool creation | +|------|-------------------------------| +| `s3` | Fails with `S3 bucket '' does not exist`, unless `S3_ALLOW_BUCKET_CREATE=1` — so a typo cannot silently create a new billed bucket. A bucket owned by another account (HTTP 403) always fails. | +| `gcs` | Created automatically in project `GCS_PROJECT_ID`. An existing bucket (HTTP 409) is accepted. | + +**Build requirements.** Neither type is in a default build. Both are HTTPS clients built on `Poco::Net`, and both are switched off (with a CMake warning) when Poco's Net/NetSSL/Crypto components are not found — the same rule as the [web dashboard](#web-dashboard-viz). The S3 device deliberately does **not** use the AWS SDK, which cannot be loaded into a process that starts the runtime. + +| Type | CMake option | Configure output when enabled | Spack | +|------|-------------|-------------------------------|-------| +| `s3` | `-DCLIO_ENABLE_AMAZON_DRIVE=ON` | `Amazon S3 bdev support: ENABLED (Poco found)` | `spack install iowarp +s3_bdev` | +| `gcs` | `-DCLIO_ENABLE_GOOGLE_CLOUD=ON` | `Google Cloud Storage bdev support: ENABLED (Poco found)` | *(no variant — use CMake)* | + +A runtime built without the option still parses `bdev_type: s3` / `gcs`, but pool creation fails with `CLIO_ENABLE_AMAZON_DRIVE is not defined. Cannot use S3 bdev.` (or the `CLIO_ENABLE_GOOGLE_CLOUD` equivalent). + +:::caution Composing one cloud pool on several nodes +A `clio_bdev` entry is created with the same `pool_name` on every node it reaches, and each node allocates blocks independently. With `pool_query: dynamic` or `broadcast`, every node therefore writes `block_` keys under the **same** prefix and overwrites the others' data. For multi-node deployments, use a CTE storage tier, which gives each node its own `_node`. +::: + +:::note Importing from S3 is a different feature +These devices make a bucket a **storage tier**. Importing existing objects into IOWarp is CAE's job — see the [S3 connector](../sdk/context-assimilation-engine/s3). +::: --- @@ -298,9 +353,9 @@ Array of storage targets. At least one entry is required when CTE is enabled. | Parameter | Required | Description | |-----------|----------|-------------| -| `path` | Yes | `ram::` for DRAM storage, or a filesystem path for disk. Supports `${HOME}` expansion. | -| `bdev_type` | Yes | `file`, `ram`, `hbm`, `pinned`, or `noop`. | -| `capacity_limit` | Yes | Maximum capacity (e.g., `"512MB"`, `"200GB"`). `0` / `"0g"` = 80% of total system DRAM. For file tiers this is a cap, not an upfront allocation — the file grows lazily in 1 GB units. | +| `path` | Yes | `ram::` for DRAM storage, a filesystem path for disk, or `s3:///` / `gcs:///` for a cloud bucket. Supports `${HOME}` expansion. | +| `bdev_type` | Yes | `file`, `ram`, `hbm`, `pinned`, `noop`, `s3`, or `gcs`. | +| `capacity_limit` | Yes | Maximum capacity (e.g., `"512MB"`, `"200GB"`). `0` / `"0g"` = 80% of total system DRAM, for `ram` tiers only; every other type must be greater than 0. For file tiers this is a cap, not an upfront allocation — the file grows lazily in 1 GB units. For `s3` / `gcs` it is a logical cap; the bucket has no quota. | | `score` | No | Placement priority (0.0–1.0). Higher = preferred. `-1.0` (default) = automatic scoring. | | `persistence_level` | No | `"volatile"` (default), `"temporary"`, or `"long_term"`. | | `existing_pool_id` | No | Bind this target to an already-composed bdev pool instead of creating one. Skips `path` / `capacity_limit` validation — routing is purely by pool id. | @@ -327,8 +382,25 @@ storage: capacity_limit: 2TB score: 0.3 persistence_level: long_term + + # Cloud tier — an S3 (or MinIO) bucket; use gcs:// + bdev_type: gcs for GCS + - path: s3://my-bucket/clio-cte + bdev_type: s3 + capacity_limit: 10TB + score: 0.1 + persistence_level: long_term ``` +**Cloud tiers (`s3`, `gcs`).** The path is a URL, not a filesystem path, so CTE skips the directory creation it does for `file` tiers. CTE registers one target per node as `_node`, so the tier above writes node 0's blocks under `s3://my-bucket/clio-cte_node0/block_`. Each tier is a [cloud object-store block device](#cloud-bdevs) — the build flags, bucket-creation rules, and credentials described there all apply. + +:::caution Always give a cloud tier a key prefix +Write `s3://my-bucket/clio-cte`, never `s3://my-bucket`. The `_node` suffix is appended to the whole path: with no prefix it lands on the **bucket name** (`my-bucket_node0`), which names a bucket that does not exist (and is not a valid S3 bucket name). The config parses either way; the bare form only fails later, when the target is created. +::: + +:::caution Credentials must reach the runtime daemon +Cloud tiers are written from runtime worker threads, so `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (or `GCS_ACCESS_TOKEN`) must be in the environment of the process that runs `clio_run start` **on every node**. If they are missing, target creation fails, CTE registers no target on that tier, and later `PutBlob` calls fail from a different process than the one that logged the cause. See [Cloud object stores](#cloud-object-stores). +::: + :::caution `persistence_level` is load-bearing Placement filters (`Context::min_persistence_level_`) and durable replicas (`REPLICA_PERSISTENT`, used by the @@ -800,6 +872,53 @@ compose: poll_period_ms: 5000 ``` +### RAM + Amazon S3 Tier + +A DRAM cache in front of an S3 bucket. Needs a runtime built with `-DCLIO_ENABLE_AMAZON_DRIVE=ON` (see [Cloud object-store block devices](#cloud-bdevs)), and credentials exported where the runtime starts: + +```bash +export AWS_ACCESS_KEY_ID=... +export AWS_SECRET_ACCESS_KEY=... +export AWS_DEFAULT_REGION=us-east-2 # the bucket's real region +# export S3_ENDPOINT=http://127.0.0.1:9000 # MinIO / S3-compatible store +clio_run start +``` + +```yaml +networking: + port: 9413 + +runtime: + num_threads: 8 + +compose: + - mod_name: clio_bdev + pool_name: "ram::chi_default_bdev" + pool_query: local + pool_id: "301.0" + bdev_type: ram + capacity: "2GB" + + - mod_name: clio_cte_core + pool_name: cte_main + pool_query: local + pool_id: "512.0" + storage: + - path: "ram::cte_cache" + bdev_type: ram + capacity_limit: 1GB + score: 1.0 + - path: s3://my-bucket/clio-cte # prefix is mandatory + bdev_type: s3 + capacity_limit: 10TB + score: 0.1 + persistence_level: long_term + dpe: + dpe_type: max_bw +``` + +For Google Cloud Storage, use `path: gcs://my-bucket/clio-cte` with `bdev_type: gcs`, build with `-DCLIO_ENABLE_GOOGLE_CLOUD=ON`, and export `GCS_ACCESS_TOKEN` instead of the AWS variables. + --- ## Docker Deployment @@ -878,9 +997,12 @@ a YAML key, the **environment wins** — it is applied after the config file is parsed, so a deployment can retune without editing a config. :::note -Every variable uses the `CLIO_` prefix (or `CTP_` for transport-primitive -concerns). The old `CHI_` prefix is **no longer recognized** — see -[Deprecation Notes](../deprecation-notes). +IOWarp's own variables use the `CLIO_` prefix (or `CTP_` for +transport-primitive concerns). The old `CHI_` prefix is **no longer +recognized** — see [Deprecation Notes](../deprecation-notes). The exceptions +are `IOWARP_PPN` and the [cloud object-store](#cloud-object-stores) variables, +which keep the standard AWS (`AWS_*`) names, plus `S3_*` / `GCS_*`, so existing +cloud tooling and credentials work unchanged. ::: ### Configuration and startup @@ -913,6 +1035,7 @@ concerns). The old `CHI_` prefix is **no longer recognized** — see | `CLIO_ZMQ_LOCAL_IPC` | *(on for macOS)* | Run the local client↔runtime ROUTER/DEALER over `ipc://` instead of TCP. The cross-node ROUTER is untouched, so multi-node TCP is unaffected. | | `CLIO_LBM_THALLIUM_PROTOCOL` | — | Thallium/Mercury protocol string (e.g. `ofi+verbs`). | | `CLIO_LBM_THALLIUM_RPC_THREADS` | — | Thallium RPC handler threads. | +| `CLIO_STRICT_RESPONSE_IDENTITY` | `0` | Read by **client** processes (TCP / IPC transports), not the daemon. Every response echoes the identity of the task that asked; a reply whose identity does not match the task waiting at that address is a late reply to an earlier task that reused the address. At `0` the mismatch is logged as `RESPONSE IDENTITY MISMATCH` and the reply is still delivered (observe-only). `1` rejects it and keeps waiting for the right reply. Set it once the log confirms mismatches are happening; it is read once per client process. | ### Shared-memory ingest tuning @@ -949,6 +1072,49 @@ concerns). The old `CHI_` prefix is **no longer recognized** — see | `CLIO_BDEV_STATS_DIR` | `~/.clio/bdev_perf` | Directory holding per-bdev `.perf` files (measured latency/bandwidth plus the learned wall-clock model). | | `CLIO_BDEV_PERSIST_STATS` | `1` | `0` disables persistence, so each start begins from a cold model. | +### Cloud object stores (S3 / GCS) {#cloud-object-stores} + +Credentials and endpoints for the [`s3` / `gcs` block devices](#cloud-bdevs) +(and therefore CTE cloud tiers) and for the CAE +[S3](../sdk/context-assimilation-engine/s3) and GCS importers. There are no +YAML keys for any of these. All of them are read **inside the runtime daemon**, +so set them in the environment that runs `clio_run start` on every node, not in +the client application's environment. + +:::caution Launchers do not forward these by default +Tools that start the daemon remotely (pssh, Slurm prologs, Jarvis) pass a +curated environment, so `AWS_*` exported in a job script can reach your +benchmark but not `clio_run`. With the `jarvis_clio_core` runtime package, list +the names in its `forward_env` parameter. Only names are logged, never values. +::: + +**Amazon S3 and S3-compatible stores** + +| Variable | Default | Read by | Description | +|----------|---------|---------|-------------| +| `AWS_ACCESS_KEY_ID` | *(none)* | S3 bdev, CAE S3 | Access key. **Required** by the S3 bdev, which reads credentials only from the environment. CAE falls back to a credentials-file profile when it is unset. | +| `AWS_SECRET_ACCESS_KEY` | *(none)* | S3 bdev, CAE S3 | Secret key. Same rules as `AWS_ACCESS_KEY_ID`. | +| `AWS_SESSION_TOKEN` | *(none)* | S3 bdev, CAE S3 | Session token for temporary (STS) credentials. | +| `AWS_DEFAULT_REGION` | S3 bdev: `us-east-1`; CAE: *(required)* | S3 bdev, CAE S3 | Bucket region. Must be the bucket's **real** region — a mismatch is answered with HTTP 301, which is not followed. The S3 bdev falls back to `us-east-1`; CAE has no fallback and fails instead. | +| `AWS_REGION` | *(none)* | CAE S3 | Region fallback when `AWS_DEFAULT_REGION` is unset. | +| `AWS_PROFILE` | `default` | CAE S3 | Profile read from the credentials file when no key pair is in the environment. The S3 bdev has **no** profile support. | +| `AWS_SHARED_CREDENTIALS_FILE` | `~/.aws/credentials` | CAE S3 | Credentials file to read profiles from. | +| `AWS_CONFIG_FILE` | `~/.aws/config` | CAE S3 | Config file consulted for the profile's `region` as a last resort. | +| `S3_ENDPOINT` | *(unset = real AWS)* | S3 bdev, CAE S3 | S3-compatible endpoint (e.g. `http://127.0.0.1:9000` for MinIO). Setting it switches to path-style addressing. `AWS_ENDPOINT_URL` is **not** read by the runtime. | +| `S3_ALLOW_BUCKET_CREATE` | `0` | S3 bdev | `1` lets the S3 bdev create a missing bucket. Otherwise a missing bucket fails pool creation, so a typo cannot create a new billed bucket. CAE never creates buckets. | + +**Google Cloud Storage** + +| Variable | Default | Read by | Description | +|----------|---------|---------|-------------| +| `GCS_ACCESS_TOKEN` | *(none — required)* | GCS bdev | OAuth2 bearer token, e.g. from `gcloud auth print-access-token`. Read once when the pool is created; `gcloud` access tokens are short-lived (typically one hour), so long runs need a fresh token at each start. | +| `GCS_ENDPOINT` | `https://storage.googleapis.com` | GCS bdev, CAE GCS | GCS-compatible endpoint (e.g. `fake-gcs-server`). For the CAE importer, setting it also switches to unauthenticated requests. | +| `GCS_PROJECT_ID` | `clio-prototype` | GCS bdev | Project a missing bucket is created in. Only used when the bucket does not exist. | + +The CAE GCS importer (`gs://`) authenticates with Google +[Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) +rather than `GCS_ACCESS_TOKEN`. + ### ADIOS2 adapter startup (large-scale MPI) At 512+ nodes the local daemon is busy serving cross-node SWIM probes and can diff --git a/docs/deployment/hpc-cluster.md b/docs/deployment/hpc-cluster.md index b483d2b1..b2ae2a49 100644 --- a/docs/deployment/hpc-cluster.md +++ b/docs/deployment/hpc-cluster.md @@ -79,6 +79,18 @@ export CLIO_IPC_MODE=TCP This variable is read by `CLIO_RUNTIME_INIT()`. If unset, the value of the `default_with_runtime` argument passed to `CLIO_RUNTIME_INIT()` is used instead. +### Cloud Storage Credentials + +If the config has an `s3` or `gcs` storage tier, or CAE imports from `s3://` / `gs://`, the **runtime daemon on every node** needs the cloud credentials in its own environment: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` (plus `S3_ENDPOINT` for MinIO), or `GCS_ACCESS_TOKEN`. Exporting them in the job script is not always enough. Launchers that start the daemons over SSH (pssh, Jarvis) pass a curated environment rather than your shell's, so make sure the variables reach each node's `clio_run start`. With Jarvis, list them in the runtime package's `forward_env`. + +```bash +export AWS_ACCESS_KEY_ID=... # keep keys out of committed scripts +export AWS_SECRET_ACCESS_KEY=... +export AWS_DEFAULT_REGION=us-east-2 # the bucket's real region +``` + +See [Configuration → Cloud object stores](./configuration#cloud-object-stores) for every variable and which component reads it. + --- ## Single-Node Deployment diff --git a/docs/deployment/performance.md b/docs/deployment/performance.md index c160daf3..acad4aa6 100644 --- a/docs/deployment/performance.md +++ b/docs/deployment/performance.md @@ -81,6 +81,7 @@ Scoring is typically set based on performance. - Use RAM tiers (`bdev_type: ram`) for critical high-priority data with `score: 1.0` (most selective, most precious) - Add NVMe/SSD tiers for balanced selectivity with `score: 0.3-0.7` - Use HDDs as overflow tiers with `score: 0.0` (least selective, accepts any data) +- Treat cloud tiers (`bdev_type: s3` / `gcs`) as the coldest overflow tier, with a low score below every local tier. Each block is a separate object and each access is a network round trip, so they trade latency for capacity. Put a RAM or NVMe tier in front of them. See [Cloud object-store block devices](./configuration#cloud-bdevs). - Align `capacity_limit` with available hardware capacity - Multiple tiers with the same score will be used with equal priority diff --git a/docs/sdk/context-assimilation-engine/omni.md b/docs/sdk/context-assimilation-engine/omni.md index 257f844a..999ab67d 100644 --- a/docs/sdk/context-assimilation-engine/omni.md +++ b/docs/sdk/context-assimilation-engine/omni.md @@ -52,15 +52,20 @@ The source URI specifying where to read data from. The URI format determines the - `file::` - Local filesystem - `hdf5://:` - HDF5 file and dataset - `globus:///` - Globus endpoint -- `s3:///` - S3 object storage +- `s3:///` - Amazon S3 or an S3-compatible store (also `s3::/`). Requires `-DCAE_ENABLE_S3=ON`; see the [S3 Connector](s3.md) for credentials. +- `gs:///` - Google Cloud Storage (also `gcs://`). Requires `-DCAE_ENABLE_GCS=ON`. **Examples:** ```yaml src: file::/data/satellite/TERRA_2024.bin src: hdf5::/data/climate.h5:/temperature/surface src: globus://82f1b5c6-6e9b-11e5-ba47-22000b92c6ec/dataset.nc +src: s3://my-bucket/climate/TERRA_2024.bin +src: gs://my-bucket/climate/TERRA_2024.bin ``` +Cloud sources (`s3://`, `gs://`) carry no credentials in the OMNI file. They are resolved inside the runtime daemon from its environment, so export them where `clio_run` starts, not where `clio_cae` runs. + ### `dst` (string, required) The destination URI specifying where to write data. Currently supports the `iowarp` scheme for CTE tag-based storage. diff --git a/docs/sdk/context-assimilation-engine/overview.md b/docs/sdk/context-assimilation-engine/overview.md index 93219df5..77d088c1 100644 --- a/docs/sdk/context-assimilation-engine/overview.md +++ b/docs/sdk/context-assimilation-engine/overview.md @@ -72,6 +72,12 @@ The CAE uses a factory pattern to select the correct assimilator based on the so | `file` | `file::/path/to/file` | `BinaryFileAssimilator` | Always enabled | | `hdf5` | `hdf5::/path/file.h5:/dataset` | `Hdf5FileAssimilator` | `-DWRP_CORE_ENABLE_HDF5=ON` | | `globus` | `globus:///` | `GlobusFileAssimilator` | `-DCAE_ENABLE_GLOBUS=ON` | +| `s3` | `s3:///` | `S3FileAssimilator` | `-DCAE_ENABLE_S3=ON` | +| `gs` / `gcs` | `gs:///` | `GcsFileAssimilator` | `-DCAE_ENABLE_GCS=ON` | + +When a protocol's build flag is off, the factory logs `... support not compiled in. Rebuild with -D=ON` and returns no assimilator. + +The factory is constructed by the CAE runtime with its CTE client and, when S3 support is built, a runtime-owned S3 connection pool that `S3FileAssimilator` borrows keep-alive connections from. The factory also detects Globus web URLs (`https://app.globus.org/...`) and routes them to `GlobusFileAssimilator`. @@ -105,6 +111,10 @@ class BaseAssimilator { **GlobusFileAssimilator** handles Globus transfers. It supports Globus-to-Globus transfers (via the Globus transfer API with submission IDs and polling) and Globus-to-local downloads (via HTTPS). Authentication tokens are passed through `ctx.src_token`. +**S3FileAssimilator** imports objects from Amazon S3 or an S3-compatible store (MinIO, via `S3_ENDPOINT`). It reads in-process: requests are SigV4-signed and streamed over POCO HTTPS, and the AWS SDK is never loaded into the runtime. The body is stored as 1 MB `chunk_` blobs plus a `description` blob, honoring `range_off` / `range_size`. A connection dropped mid-object is resumed with a ranged GET. Credentials and region are resolved inside the runtime from the `AWS_*` environment or a `~/.aws/credentials` profile; see the [S3 Connector](s3.md). + +**GcsFileAssimilator** imports objects from Google Cloud Storage (`gs://` or `gcs://`) using `google-cloud-cpp` and Application Default Credentials. Setting `GCS_ENDPOINT` points it at a GCS-compatible server such as `fake-gcs-server`. + ## AssimilationCtx `AssimilationCtx` is the serializable descriptor for a single data transfer: @@ -124,10 +134,14 @@ struct AssimilationCtx { std::string src_data; // Inline payload for "string::..." sources std::vector include_patterns; // Glob patterns to include std::vector exclude_patterns; // Glob patterns to exclude + std::string s3_region; // S3 region override (empty = resolve from env/config) + std::string s3_profile; // AWS profile NAME (empty = AWS_PROFILE or "default") }; } // namespace clio::cae::core ``` +`s3_region` and `s3_profile` are used only by `S3FileAssimilator`. When set, `s3_region` overrides `AWS_DEFAULT_REGION` / `AWS_REGION`, and `s3_profile` overrides `AWS_PROFILE`. A key pair in the runtime's environment still wins over any profile. They carry names, never secrets: the keys themselves stay in the runtime's environment or credentials file. OMNI files do not set them; they are for programmatic callers that build `AssimilationCtx` directly. + Serialization uses the [cereal](https://uscilab.github.io/cereal/) library with binary archives. The client serializes a `std::vector` into the `ParseOmniTask`, and the runtime deserializes it on the server side. ## Method IDs @@ -326,9 +340,15 @@ class MyAssimilator : public clio::cae::core::BaseAssimilator { |-------------|---------|-------------| | `WRP_CORE_ENABLE_HDF5` | OFF | Enable HDF5 assimilator (requires libhdf5) | | `CAE_ENABLE_GLOBUS` | OFF | Enable Globus assimilator (requires POCO) | +| `CAE_ENABLE_S3` | OFF | Enable S3 assimilator (requires POCO for the in-process read path **and** the AWS SDK for C++ for the `cae_s3_tool` helper). Spack: `+s3_cae`. | +| `CAE_ENABLE_GCS` | OFF | Enable GCS assimilator (requires `google-cloud-cpp` storage). Spack: `+gcs`. | + +The S3 and GCS **storage tiers** (block devices) are separate features with their own flags (`CLIO_ENABLE_AMAZON_DRIVE`, `CLIO_ENABLE_GOOGLE_CLOUD`). See [Configuration → Cloud object-store block devices](../../deployment/configuration#cloud-bdevs). ## Related Documentation - [OMNI File Format](omni.md) - YAML configuration for data transfers +- [S3 Connector](s3.md) - Importing from Amazon S3 and S3-compatible stores +- [Globus Connector](globus.md) - Importing from Globus endpoints - [Module Development Guide](../context-runtime/2.module_dev_guide.md) - Module development - [CTE Documentation](../context-transfer-engine/cte.md) - CTE storage documentation diff --git a/docs/sdk/context-assimilation-engine/s3.md b/docs/sdk/context-assimilation-engine/s3.md new file mode 100644 index 00000000..abf3c272 --- /dev/null +++ b/docs/sdk/context-assimilation-engine/s3.md @@ -0,0 +1,308 @@ +--- +sidebar_position: 4 +--- + +# Amazon S3 Connector + +## Overview + +The S3 connector lets the Context Assimilation Engine (CAE) import objects from [Amazon S3](https://aws.amazon.com/s3/), or any S3-compatible object store such as [MinIO](https://min.io/), into IOWarp. Each object named in an OMNI file is streamed straight from S3 into a CTE tag. + +The connector is implemented by `S3FileAssimilator` and is conditionally compiled with the `CAE_ENABLE_S3` CMake flag. + +Objects are read **in-process** inside the runtime: the assimilator signs each request itself (AWS Signature Version 4) and streams it over HTTPS with [POCO](https://pocoproject.org/). The AWS SDK for C++ is never loaded into the runtime, because its load-time initialization corrupts CLIO Runtime startup. The SDK is used only by the standalone [`cae_s3_tool`](#the-cae_s3_tool-helper) helper. + +:::note Not the S3 storage tier +This page covers **importing** S3 objects through CAE. Using an S3 bucket as a CTE **storage tier** (the `s3` block device) is a separate feature with its own build flag and its own credential rules. See [Configuration → Cloud object-store block devices](../../deployment/configuration#cloud-bdevs). +::: + +## Prerequisites + +### Build Dependencies + +| Dependency | Purpose | +|-----------|---------| +| [POCO](https://pocoproject.org/) (Net, NetSSL, Crypto, Foundation) | In-process HTTPS reads and SigV4 signing inside the runtime | +| [AWS SDK for C++](https://github.com/aws/aws-sdk-cpp) (`s3` component) | Builds the out-of-process `cae_s3_tool` helper only | + +Both are required. If either is missing, CMake warns and S3 import is disabled. + +### Build with S3 Support + +```bash +cmake --preset release -DCAE_ENABLE_S3=ON +cmake --build build +``` + +Verify that the configure output includes: + +``` +found AWS SDK + Poco (CAE S3 support: ENABLED) +``` + +If you instead see `S3 import will be disabled.`, the warning just above it names the missing dependency: the AWS SDK (`aws-sdk-cpp`) or POCO (`libpoco-dev` on Debian/Ubuntu). + +With Spack, enable the `s3_cae` variant (CAE itself is on by default): + +```bash +spack install iowarp +s3_cae +``` + +:::note Variant rename +This variant used to be called `+s3`. It was renamed to `+s3_cae` so it is not confused with `+s3_bdev`, which builds the S3 **storage tier** and has different dependencies. +::: + +### AWS Account + +You need a bucket and an access key whose IAM policy allows `s3:GetObject` on the objects you import. Also grant `s3:ListBucket` on the bucket: without it, S3 reports a missing key as HTTP 403 instead of 404, which hides the real cause. + +## Credentials and Region + +Credentials and region are resolved **inside the runtime process** (the `clio_run` daemon hosting the CAE pool), once per imported object. The environment of the `clio_cae` client that submits the OMNI file is **not** consulted, so export the variables in the shell (or job script) that starts the runtime. + +Only the profile **name** ever travels in a task. Secret keys stay in the runtime's environment or credentials file. + +### Credential resolution + +Highest priority first: + +1. **Environment keys.** `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (plus `AWS_SESSION_TOKEN` for temporary credentials). Both the key ID and the secret must be set for this step to apply. +2. **Credentials file.** Otherwise, read a profile section from the shared credentials file: + - **Profile:** `AWS_PROFILE`, else `default`. + - **File:** `AWS_SHARED_CREDENTIALS_FILE`, else `~/.aws/credentials` (the runtime process's `$HOME`). + - **Keys read:** `aws_access_key_id`, `aws_secret_access_key`, and the optional `aws_session_token`. + +Only static keys are supported. SSO profiles, `credential_process`, EC2/ECS instance roles and `role_arn` assume-role chains are **not** resolved. Export temporary keys into the environment instead. + +### Region resolution + +Highest priority first: + +1. `AWS_DEFAULT_REGION` +2. `AWS_REGION` +3. The `region` key in the config file: `AWS_CONFIG_FILE`, else `~/.aws/config`. Section `[default]` for the default profile, `[profile ]` (or `[]`) for a named one. + +There is **no fallback region**. If none of these is set, the import fails rather than guessing `us-east-1`, because a wrong region makes S3 answer HTTP 301, which the signer does not follow. + +### Example + +```ini +# ~/.aws/credentials (chmod 600) +[clio-bench] +aws_access_key_id = AKIA... +aws_secret_access_key = ... +``` + +```ini +# ~/.aws/config +[profile clio-bench] +region = us-east-2 +``` + +```bash +# In the shell that starts the runtime +export AWS_PROFILE=clio-bench +clio_run start +``` + +## S3-Compatible Endpoints (MinIO) + +Set `S3_ENDPOINT` in the runtime's environment to target an S3-compatible server: + +```bash +export S3_ENDPOINT=http://127.0.0.1:9000 +export AWS_DEFAULT_REGION=us-east-1 # still required; MinIO accepts us-east-1 +export AWS_ACCESS_KEY_ID=minioadmin +export AWS_SECRET_ACCESS_KEY=minioadmin +clio_run start +``` + +| `S3_ENDPOINT` | Addressing | Request URL | +|---------------|-----------|-------------| +| unset | Virtual-hosted (real AWS) | `https://.s3..amazonaws.com/` | +| set | Path-style | `//` | + +A trailing `/` on the endpoint is stripped. + +:::caution `AWS_ENDPOINT_URL` is not read by the runtime +The in-process assimilator reads only `S3_ENDPOINT`. `AWS_ENDPOINT_URL` is honored by `cae_s3_tool` (through the AWS SDK) but **not** by the runtime, so a setup that relies on it will seed objects into MinIO with the helper and then try to read them from real AWS. +::: + +## OMNI File Format for S3 + +### `s3://` URI + +```yaml +name: s3_import + +transfers: + - src: s3://my-bucket/path/to/object.bin + dst: iowarp::my_dataset + format: binary + depends_on: "" + range_off: 0 + range_size: 0 +``` + +| Field | Notes | +|-------|-------| +| `src` | `s3:///`. The `s3::/` form is also accepted. The key must be non-empty: a bare `s3://` is rejected. | +| `dst` | Must be `iowarp::`. No other destination protocol is supported. | +| `format` | `binary`. The object is stored as opaque bytes. | +| `range_off` / `range_size` | `range_size: 0` imports the whole object. A positive `range_size` imports bytes `[range_off, range_off + range_size)` with a ranged GET. A bare `range_off` with `range_size: 0` still means the whole object. | + +Submit it with `clio_cae`, the same as any OMNI file: + +```bash +clio_cae /path/to/s3_import.yaml +``` + +### What lands in CTE + +For each transfer, the assimilator gets or creates the destination tag and writes: + +| Blob | Contents | +|------|----------| +| `description` | `binary, offset=>` | +| `chunk_0`, `chunk_1`, … | The object body, in chunks of up to 1 MB | + +Body chunks are written with up to 32 `PutBlob` operations in flight. If the HTTPS connection drops mid-object, the assimilator resumes with a ranged GET from the last byte received (up to 16 times per object) instead of failing the import. + +Keep-alive connections are pooled per runtime and reused across objects, so a batch of many small objects does not pay a TLS handshake each. The assimilator never creates a bucket. + +## Quick Start + +### 1. Build with S3 Support + +```bash +cmake --preset release -DCAE_ENABLE_S3=ON +cmake --build build && cmake --install build +``` + +### 2. Provide Credentials to the Runtime + +Either export keys: + +```bash +export AWS_ACCESS_KEY_ID=... +export AWS_SECRET_ACCESS_KEY=... +export AWS_DEFAULT_REGION=us-east-2 +``` + +or point at a profile in `~/.aws/credentials`: + +```bash +export AWS_PROFILE=my-profile +``` + +### 3. Start the Runtime with CTE and CAE + +Start `clio_run` **from the same shell**, with a config whose `compose` section includes `clio_cte_core` and `clio_cae_core` (the shipped default does): + +```bash +clio_run start +``` + +### 4. Import + +```bash +clio_cae s3_import.yaml +``` + +## The `cae_s3_tool` Helper + +When `CAE_ENABLE_S3` is on, the build also installs `cae_s3_tool` to `bin/`. It is a small standalone program that links the AWS SDK, so it never runs inside the runtime. It is handy for seeding a bucket, fetching an object to compare against what landed in CTE, or checking credentials outside the runtime. + +```text +cae_s3_tool get [range_off range_size] +cae_s3_tool put +cae_s3_tool del +``` + +- **`put`** also creates the bucket if it is missing (best effort). +- **`del`** treats a missing object as success. +- **Exit status:** `0` on success; non-zero with a message on stderr otherwise. + +The helper resolves its configuration through the **AWS SDK**, so it accepts more than the runtime does: + +| Setting | Read from | +|---------|-----------| +| Credentials | The SDK's default provider chain (environment, profiles, SSO, instance roles, …) | +| Region | `AWS_DEFAULT_REGION`, default `us-east-1` | +| Endpoint | `S3_ENDPOINT` or `AWS_ENDPOINT_URL` (path-style) | +| Anonymous access | `AWS_NO_SIGN_REQUEST=1` forces unsigned requests (public buckets such as the AWS Open Data registry). If the chain finds no credentials, it also falls back to anonymous. | + +:::caution +Because the helper's credential chain is broader, `cae_s3_tool get` can succeed where an import fails. A working helper proves the bucket and key are readable. It does not prove the runtime can resolve the same credentials (see [Credentials and Region](#credentials-and-region)). +::: + +## Error Codes + +The `S3FileAssimilator::Schedule` method returns the following error codes: + +| Code | Description | +|------|-------------| +| 0 | Success | +| -1 | Invalid destination protocol (not `iowarp`) | +| -2 | Destination URI has no tag name | +| -3 | Failed to get or create the destination tag in CTE | +| -4 | Invalid S3 source URI (missing bucket or key) | +| -6 | Credentials or region could not be resolved (the log names the profile and file tried) | +| -7 | Object not found, or the GET request failed | +| -8 | Short or failed read on a body chunk (after resume attempts) | +| -9 | Failed to store the `description` blob | +| -10 | A body `PutBlob` failed | + +If the build lacks S3 support, the factory logs `S3 protocol requested but S3 support not compiled in. Rebuild with -DCAE_ENABLE_S3=ON to enable S3 support.` and no assimilator is created. + +## Troubleshooting + +### `CAE S3 support` is missing from the configure output + +The configure step printed `S3 import will be disabled.` Install whichever dependency the preceding warning names (`aws-sdk-cpp`, or POCO's Net/NetSSL/Crypto components) and reconfigure. + +### `AWS credentials: no profile [] in the credentials file` + +The runtime found no `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` pair, fell back to the credentials file, and that file has no `[]` section. Check: +- `AWS_PROFILE` is spelled correctly in the **runtime's** environment. +- The file exists under the runtime user's `$HOME`, or set `AWS_SHARED_CREDENTIALS_FILE`. +- Credentials files use `[name]`, not `[profile name]`; that form belongs in `~/.aws/config`. + +### `profile [] is missing aws_access_key_id / aws_secret_access_key` + +The section exists, but one of the two keys is absent, or it is indented, which makes the parser treat it as a nested setting. Keys must start at column 0. + +### `AWS region unresolved for profile []` + +Set `AWS_DEFAULT_REGION` (or `AWS_REGION`) in the runtime's environment, or add `region = ...` to the profile in `~/.aws/config`. + +### HTTP 301 + +The region does not match the bucket's region. Find the bucket's real region (the S3 console shows it) and set `AWS_DEFAULT_REGION` to it. + +### HTTP 403 + +The keys are wrong or expired (temporary credentials need `AWS_SESSION_TOKEN`), the IAM policy lacks `s3:GetObject`, or the host clock is more than 15 minutes off, which invalidates SigV4 signatures. A missing key also returns 403 when the policy lacks `s3:ListBucket`. + +### `object not found: s3:///` + +The key does not exist. Keys are case-sensitive and have no leading `/`. With `S3_ENDPOINT` set, confirm you are pointed at the store you seeded. + +### Imports work in your shell but fail under a launcher + +The runtime daemon did not inherit your environment. Launchers that start `clio_run` on other nodes (pssh, Slurm prologs, Jarvis) pass a curated environment, so `AWS_*` variables exported in your shell or job script may never reach it. Export them where the daemon is started. With the `jarvis_clio_core` runtime package, list them in its `forward_env` parameter. + +## Test Files Reference + +| File | Description | +|------|-------------| +| `context-assimilation-engine/test/unit/s3_assim/test_s3_assim.cc` | End-to-end import against an S3-compatible store: seeds an object with `cae_s3_tool`, runs `ParseOmni`, verifies the bytes in CTE. Self-skips unless `S3_ENDPOINT` is set (bucket via `S3_TEST_BUCKET`, default `clio-cae-test`). | +| `context-assimilation-engine/test/unit/s3_assim/s3_assim_omni.yaml` | Example OMNI file for an S3 import | +| `context-assimilation-engine/test/unit/aws_creds/test_aws_creds.cc` | Offline tests for credential and region resolution precedence | + +## Related Documentation + +- [OMNI File Format](omni.md) - Full OMNI YAML specification +- [CAE Overview](overview.md) - CAE architecture and API reference +- [Configuration Reference](../../deployment/configuration#cloud-object-stores) - Every S3/GCS environment variable, and the S3 storage tier +- [CTE Documentation](../context-transfer-engine/cte.md) - CTE storage documentation diff --git a/docs/sdk/context-runtime/20.base-modules/2.bdev.md b/docs/sdk/context-runtime/20.base-modules/2.bdev.md index a75c5c1c..dca611f1 100644 --- a/docs/sdk/context-runtime/20.base-modules/2.bdev.md +++ b/docs/sdk/context-runtime/20.base-modules/2.bdev.md @@ -431,6 +431,10 @@ enum class BdevType : clio::run::u32 { **Backend Characteristics:** - **kFile**: Uses file-based storage with libaio for asynchronous I/O, supports alignment requirements, persistent data - **kRam**: Uses malloc-allocated RAM buffer, synchronous operations, volatile data (lost on restart) +- **kS3**: Stores each block as one object (`[prefix/]block_`) in an Amazon S3 or S3-compatible bucket. A HTTPS client built on `Poco::Net` signs requests with AWS SigV4; the AWS SDK is deliberately not used, because loading it into a process that calls `CLIO_INIT` corrupts runtime startup. Each worker keeps its own keep-alive connection, unwritten blocks read back as zeros, and freeing a block deletes its object (best effort). Persistent. Compiled only with `-DCLIO_ENABLE_AMAZON_DRIVE=ON`. +- **kGcs**: The same object-per-block layout on Google Cloud Storage, authenticated with an OAuth2 bearer token (`GCS_ACCESS_TOKEN`). Persistent. Compiled only with `-DCLIO_ENABLE_GOOGLE_CLOUD=ON`. + +Credentials, endpoints, and bucket-creation policy for `kS3` / `kGcs` come from environment variables only. See [Configuration → Cloud object-store block devices](../../../deployment/configuration#cloud-bdevs). ### Block Structure Represents an allocated block of storage. @@ -547,9 +551,11 @@ struct CreateParams { - **pool_name**: - For kFile: **IS the file path** (can be block device `/dev/nvme0n1` or regular file) - For kRam: Unique identifier for the RAM device + - For kS3 / kGcs: `[s3://|gcs://][/]` — the bucket, plus an optional key prefix after the first `/` (trailing slashes dropped) - **total_size_**: - For kFile: Set to 0 to use full file/device size, or specify limit - For kRam: **Required** - specifies the RAM buffer size to allocate + - For kS3 / kGcs: Logical capacity the allocator hands out (the bucket has no quota); 0 means 1 TB - **io_depth_**: Higher values improve parallelism for kFile but use more memory (typical: 16-128), ignored for kRam - **alignment_**: Must match device requirements for kFile (typically 512 or 4096 bytes), less critical for kRam @@ -861,6 +867,7 @@ void example(clio::run::bdev::Client& bdev_client) { - **cereal**: Serialization library for network communication - **libaio**: Linux asynchronous I/O library for high-performance block operations - **Boost.Fiber** and **Boost.Context**: Coroutine support +- **POCO** (Net, NetSSL, Crypto) — *optional*: HTTPS transport for the `kS3` and `kGcs` backends. When it is not found, `CLIO_ENABLE_AMAZON_DRIVE` and `CLIO_ENABLE_GOOGLE_CLOUD` are switched off with a CMake warning. ## Installation @@ -972,6 +979,11 @@ void example() { - Integration with existing storage infrastructure - Need for data durability across restarts +**Use a Cloud Backend (`BdevType::kS3` / `BdevType::kGcs`) when:** +- Capacity needs exceed local disks, or data must outlive the cluster's storage +- The tier is a cold or overflow tier — every block is a network round trip to the object store +- Nodes share no filesystem but can all reach the same bucket + ### Performance Tuning 1. **Block Size Selection**: Choose appropriate block sizes based on I/O patterns diff --git a/docs/sdk/context-transfer-engine/cte.md b/docs/sdk/context-transfer-engine/cte.md index 13fdfe6e..fcb7cdcd 100644 --- a/docs/sdk/context-transfer-engine/cte.md +++ b/docs/sdk/context-transfer-engine/cte.md @@ -1238,10 +1238,17 @@ void example() { ### Storage Device Types +These are the values the config parser accepts; anything else is rejected at load time. + - `"file"` - File-based block device - `"ram"` - RAM-based block device (for caching) -- `"dev_dax"` - Persistent memory device -- `"posix"` - POSIX file system interface +- `"hbm"` - GPU high-bandwidth (device) memory +- `"pinned"` - Pinned host memory +- `"noop"` - No-op backend for latency testing (no actual I/O) +- `"s3"` - Amazon S3 or S3-compatible bucket; `path` is `s3:///` +- `"gcs"` - Google Cloud Storage bucket; `path` is `gcs:///` + +The cloud types need a runtime built with `CLIO_ENABLE_AMAZON_DRIVE` / `CLIO_ENABLE_GOOGLE_CLOUD`, and credentials in the runtime's environment. See [Configuration → Cloud object-store block devices](../../deployment/configuration#cloud-bdevs). ### Manual Tier Scoring From c470c30e539f674380da4bd9d732b2d8d95b373a Mon Sep 17 00:00:00 2001 From: Atharva Naik Date: Thu, 8 Oct 2026 02:03:45 +0000 Subject: [PATCH 2/3] docs: trim S3/GCS cloud storage docs to the precedented parts - Fold the s3/gcs compose examples into the existing clio_bdev block and drop the separate cloud-bdev section (behavior, bucket creation, build requirements); retarget its links. - Describe hbm/pinned/noop in the bdev_type row. - Drop the cloud-tier paragraph; keep the key-prefix caution. - State why credentials are env-only in two places (CTE storage tiers, CAE S3 page) and remove the other repetitions. - Revert the env-var prefix note; shorten CLIO_STRICT_RESPONSE_IDENTITY; remove the cloud-object-stores preamble. - hpc-cluster: table instead of prose for cloud credentials. - CAE overview/omni: cut factory internals and credential notes. Co-Authored-By: Claude Opus 5.5 --- docs/deployment/configuration.md | 92 ++++--------------- docs/deployment/hpc-cluster.md | 10 +- docs/deployment/performance.md | 1 - docs/sdk/context-assimilation-engine/omni.md | 2 - .../context-assimilation-engine/overview.md | 14 +-- docs/sdk/context-assimilation-engine/s3.md | 4 +- .../context-runtime/20.base-modules/2.bdev.md | 2 +- docs/sdk/context-transfer-engine/cte.md | 2 +- 8 files changed, 35 insertions(+), 92 deletions(-) diff --git a/docs/deployment/configuration.md b/docs/deployment/configuration.md index 621b82bf..493b2e14 100644 --- a/docs/deployment/configuration.md +++ b/docs/deployment/configuration.md @@ -264,8 +264,8 @@ Block devices provide the shared memory allocator used by other modules. At leas | Parameter | Required | Description | |-----------|----------|-------------| -| `bdev_type` | Yes | `"ram"` for DRAM-backed, `"file"` for filesystem-backed, `"s3"` for an Amazon S3 (or S3-compatible) bucket, `"gcs"` for a Google Cloud Storage bucket. `hbm`, `pinned`, and `noop` are also accepted. | -| `capacity` | Yes | Maximum capacity (e.g., `"512MB"`, `"100GB"`). For `s3` / `gcs` this is a logical cap on how many bytes the pool hands out — the bucket has no quota — and `0` means 1 TB. | +| `bdev_type` | Yes | `"ram"` for DRAM-backed, `"file"` for filesystem-backed, `"s3"` for an Amazon S3 (or S3-compatible) bucket, `"gcs"` for a Google Cloud Storage bucket, `"hbm"` for GPU device memory, `"pinned"` for page-locked host memory, `"noop"` for no-I/O latency testing. | +| `capacity` | Yes | Maximum capacity (e.g., `"512MB"`, `"100GB"`). | ```yaml compose: @@ -284,25 +284,16 @@ compose: # pool_id: "302.0" # bdev_type: file # capacity: "100GB" -``` - -For DRAM devices the `pool_name` uses the `ram::` convention. For file-backed devices the `pool_name` is the filesystem path where data is stored. For `s3` / `gcs` devices the `pool_name` names the bucket and a key prefix — see below. - -### Cloud object-store block devices (`s3`, `gcs`) {#cloud-bdevs} -An `s3` or `gcs` block device stores every block as one object in a bucket, under the key `[/]block_`. It is usually reached through a [CTE storage tier](#storage-tiers-storage) rather than composed directly, but both work. - -```yaml -compose: - # Amazon S3 (or MinIO) bucket - - mod_name: clio_bdev - pool_name: "s3://my-bucket/clio/pool_0" # bucket "my-bucket", prefix "clio/pool_0" - pool_query: local - pool_id: "303.0" - bdev_type: s3 - capacity: "1TB" + # Amazon S3 (or MinIO) bucket (optional) + # - mod_name: clio_bdev + # pool_name: "s3://my-bucket/clio/pool_0" # bucket "my-bucket", prefix "clio/pool_0" + # pool_query: local + # pool_id: "303.0" + # bdev_type: s3 + # capacity: "1TB" - # Google Cloud Storage bucket + # Google Cloud Storage bucket (optional) # - mod_name: clio_bdev # pool_name: "gcs://my-bucket/clio/pool_0" # pool_query: local @@ -311,38 +302,12 @@ compose: # capacity: "1TB" ``` -**`pool_name` format.** `[s3://|gcs://][/]`. The scheme is optional, everything after the first `/` is the key prefix, and trailing slashes are dropped. Use a prefix: CTE storage tiers append `_node` to the path, and only a prefix gives each node its own key space. - -**Behavior.** -- A block that was never written reads back as zeros, and an object shorter than the block is zero-filled. -- Freeing a block deletes its object (best effort). -- `s3` only: each runtime worker keeps one HTTPS keep-alive connection, so steady-state I/O does not reconnect per block. -- Credentials and endpoints come **only** from the runtime daemon's environment — there are no YAML keys for them. See [Cloud object stores](#cloud-object-stores) for every variable. - -**Bucket creation differs between the two.** - -| Type | Bucket missing at pool creation | -|------|-------------------------------| -| `s3` | Fails with `S3 bucket '' does not exist`, unless `S3_ALLOW_BUCKET_CREATE=1` — so a typo cannot silently create a new billed bucket. A bucket owned by another account (HTTP 403) always fails. | -| `gcs` | Created automatically in project `GCS_PROJECT_ID`. An existing bucket (HTTP 409) is accepted. | - -**Build requirements.** Neither type is in a default build. Both are HTTPS clients built on `Poco::Net`, and both are switched off (with a CMake warning) when Poco's Net/NetSSL/Crypto components are not found — the same rule as the [web dashboard](#web-dashboard-viz). The S3 device deliberately does **not** use the AWS SDK, which cannot be loaded into a process that starts the runtime. - -| Type | CMake option | Configure output when enabled | Spack | -|------|-------------|-------------------------------|-------| -| `s3` | `-DCLIO_ENABLE_AMAZON_DRIVE=ON` | `Amazon S3 bdev support: ENABLED (Poco found)` | `spack install iowarp +s3_bdev` | -| `gcs` | `-DCLIO_ENABLE_GOOGLE_CLOUD=ON` | `Google Cloud Storage bdev support: ENABLED (Poco found)` | *(no variant — use CMake)* | - -A runtime built without the option still parses `bdev_type: s3` / `gcs`, but pool creation fails with `CLIO_ENABLE_AMAZON_DRIVE is not defined. Cannot use S3 bdev.` (or the `CLIO_ENABLE_GOOGLE_CLOUD` equivalent). +For DRAM devices the `pool_name` uses the `ram::` convention. For file-backed devices the `pool_name` is the filesystem path where data is stored. For `s3` / `gcs` devices the `pool_name` is `[s3://|gcs://][/]`. :::caution Composing one cloud pool on several nodes A `clio_bdev` entry is created with the same `pool_name` on every node it reaches, and each node allocates blocks independently. With `pool_query: dynamic` or `broadcast`, every node therefore writes `block_` keys under the **same** prefix and overwrites the others' data. For multi-node deployments, use a CTE storage tier, which gives each node its own `_node`. ::: -:::note Importing from S3 is a different feature -These devices make a bucket a **storage tier**. Importing existing objects into IOWarp is CAE's job — see the [S3 connector](../sdk/context-assimilation-engine/s3). -::: - --- ## CTE Module Parameters (`clio_cte_core`) @@ -355,7 +320,7 @@ Array of storage targets. At least one entry is required when CTE is enabled. |-----------|----------|-------------| | `path` | Yes | `ram::` for DRAM storage, a filesystem path for disk, or `s3:///` / `gcs:///` for a cloud bucket. Supports `${HOME}` expansion. | | `bdev_type` | Yes | `file`, `ram`, `hbm`, `pinned`, `noop`, `s3`, or `gcs`. | -| `capacity_limit` | Yes | Maximum capacity (e.g., `"512MB"`, `"200GB"`). `0` / `"0g"` = 80% of total system DRAM, for `ram` tiers only; every other type must be greater than 0. For file tiers this is a cap, not an upfront allocation — the file grows lazily in 1 GB units. For `s3` / `gcs` it is a logical cap; the bucket has no quota. | +| `capacity_limit` | Yes | Maximum capacity (e.g., `"512MB"`, `"200GB"`). `0` / `"0g"` = 80% of total system DRAM for `ram` tiers. For `file` tiers this is a cap, not an upfront allocation — the file grows lazily in 1 GB units. For `s3` / `gcs` it is a logical cap; the bucket has no quota. | | `score` | No | Placement priority (0.0–1.0). Higher = preferred. `-1.0` (default) = automatic scoring. | | `persistence_level` | No | `"volatile"` (default), `"temporary"`, or `"long_term"`. | | `existing_pool_id` | No | Bind this target to an already-composed bdev pool instead of creating one. Skips `path` / `capacity_limit` validation — routing is purely by pool id. | @@ -391,14 +356,12 @@ storage: persistence_level: long_term ``` -**Cloud tiers (`s3`, `gcs`).** The path is a URL, not a filesystem path, so CTE skips the directory creation it does for `file` tiers. CTE registers one target per node as `_node`, so the tier above writes node 0's blocks under `s3://my-bucket/clio-cte_node0/block_`. Each tier is a [cloud object-store block device](#cloud-bdevs) — the build flags, bucket-creation rules, and credentials described there all apply. - :::caution Always give a cloud tier a key prefix -Write `s3://my-bucket/clio-cte`, never `s3://my-bucket`. The `_node` suffix is appended to the whole path: with no prefix it lands on the **bucket name** (`my-bucket_node0`), which names a bucket that does not exist (and is not a valid S3 bucket name). The config parses either way; the bare form only fails later, when the target is created. +Write `s3://my-bucket/clio-cte`, never `s3://my-bucket`. CTE appends `_node` to the whole path, one target per node: with no prefix it lands on the **bucket name** (`my-bucket_node0`), which names a bucket that does not exist (and is not a valid S3 bucket name). The config parses either way; the bare form only fails later, when the target is created. ::: :::caution Credentials must reach the runtime daemon -Cloud tiers are written from runtime worker threads, so `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (or `GCS_ACCESS_TOKEN`) must be in the environment of the process that runs `clio_run start` **on every node**. If they are missing, target creation fails, CTE registers no target on that tier, and later `PutBlob` calls fail from a different process than the one that logged the cause. See [Cloud object stores](#cloud-object-stores). +Cloud credentials have no YAML keys, so secret keys never end up in a config file that gets shared or committed. Export them where `clio_run start` runs, on every node. See [Cloud object stores](#cloud-object-stores). ::: :::caution `persistence_level` is load-bearing @@ -874,7 +837,7 @@ compose: ### RAM + Amazon S3 Tier -A DRAM cache in front of an S3 bucket. Needs a runtime built with `-DCLIO_ENABLE_AMAZON_DRIVE=ON` (see [Cloud object-store block devices](#cloud-bdevs)), and credentials exported where the runtime starts: +A DRAM cache in front of an S3 bucket. Needs a runtime built with `-DCLIO_ENABLE_AMAZON_DRIVE=ON`, and credentials exported where the runtime starts: ```bash export AWS_ACCESS_KEY_ID=... @@ -997,12 +960,9 @@ a YAML key, the **environment wins** — it is applied after the config file is parsed, so a deployment can retune without editing a config. :::note -IOWarp's own variables use the `CLIO_` prefix (or `CTP_` for -transport-primitive concerns). The old `CHI_` prefix is **no longer -recognized** — see [Deprecation Notes](../deprecation-notes). The exceptions -are `IOWARP_PPN` and the [cloud object-store](#cloud-object-stores) variables, -which keep the standard AWS (`AWS_*`) names, plus `S3_*` / `GCS_*`, so existing -cloud tooling and credentials work unchanged. +Every variable uses the `CLIO_` prefix (or `CTP_` for transport-primitive +concerns). The old `CHI_` prefix is **no longer recognized** — see +[Deprecation Notes](../deprecation-notes). ::: ### Configuration and startup @@ -1035,7 +995,7 @@ cloud tooling and credentials work unchanged. | `CLIO_ZMQ_LOCAL_IPC` | *(on for macOS)* | Run the local client↔runtime ROUTER/DEALER over `ipc://` instead of TCP. The cross-node ROUTER is untouched, so multi-node TCP is unaffected. | | `CLIO_LBM_THALLIUM_PROTOCOL` | — | Thallium/Mercury protocol string (e.g. `ofi+verbs`). | | `CLIO_LBM_THALLIUM_RPC_THREADS` | — | Thallium RPC handler threads. | -| `CLIO_STRICT_RESPONSE_IDENTITY` | `0` | Read by **client** processes (TCP / IPC transports), not the daemon. Every response echoes the identity of the task that asked; a reply whose identity does not match the task waiting at that address is a late reply to an earlier task that reused the address. At `0` the mismatch is logged as `RESPONSE IDENTITY MISMATCH` and the reply is still delivered (observe-only). `1` rejects it and keeps waiting for the right reply. Set it once the log confirms mismatches are happening; it is read once per client process. | +| `CLIO_STRICT_RESPONSE_IDENTITY` | `0` | Client-side. `0` logs replies meant for an earlier task (`RESPONSE IDENTITY MISMATCH`); `1` drops them and keeps waiting. | ### Shared-memory ingest tuning @@ -1074,20 +1034,6 @@ cloud tooling and credentials work unchanged. ### Cloud object stores (S3 / GCS) {#cloud-object-stores} -Credentials and endpoints for the [`s3` / `gcs` block devices](#cloud-bdevs) -(and therefore CTE cloud tiers) and for the CAE -[S3](../sdk/context-assimilation-engine/s3) and GCS importers. There are no -YAML keys for any of these. All of them are read **inside the runtime daemon**, -so set them in the environment that runs `clio_run start` on every node, not in -the client application's environment. - -:::caution Launchers do not forward these by default -Tools that start the daemon remotely (pssh, Slurm prologs, Jarvis) pass a -curated environment, so `AWS_*` exported in a job script can reach your -benchmark but not `clio_run`. With the `jarvis_clio_core` runtime package, list -the names in its `forward_env` parameter. Only names are logged, never values. -::: - **Amazon S3 and S3-compatible stores** | Variable | Default | Read by | Description | diff --git a/docs/deployment/hpc-cluster.md b/docs/deployment/hpc-cluster.md index b2ae2a49..8a94041e 100644 --- a/docs/deployment/hpc-cluster.md +++ b/docs/deployment/hpc-cluster.md @@ -81,10 +81,16 @@ This variable is read by `CLIO_RUNTIME_INIT()`. If unset, the value of the `defa ### Cloud Storage Credentials -If the config has an `s3` or `gcs` storage tier, or CAE imports from `s3://` / `gs://`, the **runtime daemon on every node** needs the cloud credentials in its own environment: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` (plus `S3_ENDPOINT` for MinIO), or `GCS_ACCESS_TOKEN`. Exporting them in the job script is not always enough. Launchers that start the daemons over SSH (pssh, Jarvis) pass a curated environment rather than your shell's, so make sure the variables reach each node's `clio_run start`. With Jarvis, list them in the runtime package's `forward_env`. +| Variable | Default | Description | +|----------|---------|-------------| +| `AWS_ACCESS_KEY_ID` | *(none)* | S3 access key, for `s3` storage tiers and CAE `s3://` imports. | +| `AWS_SECRET_ACCESS_KEY` | *(none)* | S3 secret key. | +| `AWS_DEFAULT_REGION` | `us-east-1` | The bucket's region. CAE has no default and fails if it is unset. | +| `S3_ENDPOINT` | *(AWS)* | S3-compatible endpoint, e.g. MinIO. | +| `GCS_ACCESS_TOKEN` | *(none)* | OAuth2 bearer token, for `gcs` storage tiers. | ```bash -export AWS_ACCESS_KEY_ID=... # keep keys out of committed scripts +export AWS_ACCESS_KEY_ID=... export AWS_SECRET_ACCESS_KEY=... export AWS_DEFAULT_REGION=us-east-2 # the bucket's real region ``` diff --git a/docs/deployment/performance.md b/docs/deployment/performance.md index acad4aa6..c160daf3 100644 --- a/docs/deployment/performance.md +++ b/docs/deployment/performance.md @@ -81,7 +81,6 @@ Scoring is typically set based on performance. - Use RAM tiers (`bdev_type: ram`) for critical high-priority data with `score: 1.0` (most selective, most precious) - Add NVMe/SSD tiers for balanced selectivity with `score: 0.3-0.7` - Use HDDs as overflow tiers with `score: 0.0` (least selective, accepts any data) -- Treat cloud tiers (`bdev_type: s3` / `gcs`) as the coldest overflow tier, with a low score below every local tier. Each block is a separate object and each access is a network round trip, so they trade latency for capacity. Put a RAM or NVMe tier in front of them. See [Cloud object-store block devices](./configuration#cloud-bdevs). - Align `capacity_limit` with available hardware capacity - Multiple tiers with the same score will be used with equal priority diff --git a/docs/sdk/context-assimilation-engine/omni.md b/docs/sdk/context-assimilation-engine/omni.md index 999ab67d..d292e9d0 100644 --- a/docs/sdk/context-assimilation-engine/omni.md +++ b/docs/sdk/context-assimilation-engine/omni.md @@ -64,8 +64,6 @@ src: s3://my-bucket/climate/TERRA_2024.bin src: gs://my-bucket/climate/TERRA_2024.bin ``` -Cloud sources (`s3://`, `gs://`) carry no credentials in the OMNI file. They are resolved inside the runtime daemon from its environment, so export them where `clio_run` starts, not where `clio_cae` runs. - ### `dst` (string, required) The destination URI specifying where to write data. Currently supports the `iowarp` scheme for CTE tag-based storage. diff --git a/docs/sdk/context-assimilation-engine/overview.md b/docs/sdk/context-assimilation-engine/overview.md index 77d088c1..0edb2792 100644 --- a/docs/sdk/context-assimilation-engine/overview.md +++ b/docs/sdk/context-assimilation-engine/overview.md @@ -6,7 +6,7 @@ sidebar_position: 1 ## Introduction -The Context Assimilation Engine (CAE) is a CLIO Runtime module (`clio_cae::core`) that ingests external data sources into the CLIO Runtime. It reads data from files, HDF5 datasets, or remote Globus endpoints and stores them as blobs in the Context Transfer Engine (CTE). The CAE is registered as a Module container with pool ID `400.0`. +The Context Assimilation Engine (CAE) is a CLIO Runtime module (`clio_cae::core`) that ingests external data sources into the CLIO Runtime. It reads data from files, HDF5 datasets, remote Globus endpoints, or Amazon S3 buckets and stores them as blobs in the Context Transfer Engine (CTE). The CAE is registered as a Module container with pool ID `400.0`. CAE ships a second, optional ChiMod: the **summarizer** (`clio_cae_summarizer`, pool `401.0`). It interposes on the CTE core's task interface and attaches an LLM-generated summary to each matching blob on the way through. See [the interposition chain](../context-transfer-engine/chimod-chain#summarizer-chimod-clio_cae_summarizer) for its behavior and [deployment configuration](../../deployment/configuration) for its keys. @@ -75,10 +75,6 @@ The CAE uses a factory pattern to select the correct assimilator based on the so | `s3` | `s3:///` | `S3FileAssimilator` | `-DCAE_ENABLE_S3=ON` | | `gs` / `gcs` | `gs:///` | `GcsFileAssimilator` | `-DCAE_ENABLE_GCS=ON` | -When a protocol's build flag is off, the factory logs `... support not compiled in. Rebuild with -D=ON` and returns no assimilator. - -The factory is constructed by the CAE runtime with its CTE client and, when S3 support is built, a runtime-owned S3 connection pool that `S3FileAssimilator` borrows keep-alive connections from. - The factory also detects Globus web URLs (`https://app.globus.org/...`) and routes them to `GlobusFileAssimilator`. Protocol extraction supports two URI styles: @@ -140,7 +136,7 @@ struct AssimilationCtx { } // namespace clio::cae::core ``` -`s3_region` and `s3_profile` are used only by `S3FileAssimilator`. When set, `s3_region` overrides `AWS_DEFAULT_REGION` / `AWS_REGION`, and `s3_profile` overrides `AWS_PROFILE`. A key pair in the runtime's environment still wins over any profile. They carry names, never secrets: the keys themselves stay in the runtime's environment or credentials file. OMNI files do not set them; they are for programmatic callers that build `AssimilationCtx` directly. +`s3_region` and `s3_profile` are used only by `S3FileAssimilator`. When set, `s3_region` overrides `AWS_DEFAULT_REGION` / `AWS_REGION`, and `s3_profile` overrides `AWS_PROFILE`. A key pair in the runtime's environment still wins over any profile. OMNI files do not set them; they are for programmatic callers that build `AssimilationCtx` directly. Serialization uses the [cereal](https://uscilab.github.io/cereal/) library with binary archives. The client serializes a `std::vector` into the `ParseOmniTask`, and the runtime deserializes it on the server side. @@ -340,10 +336,8 @@ class MyAssimilator : public clio::cae::core::BaseAssimilator { |-------------|---------|-------------| | `WRP_CORE_ENABLE_HDF5` | OFF | Enable HDF5 assimilator (requires libhdf5) | | `CAE_ENABLE_GLOBUS` | OFF | Enable Globus assimilator (requires POCO) | -| `CAE_ENABLE_S3` | OFF | Enable S3 assimilator (requires POCO for the in-process read path **and** the AWS SDK for C++ for the `cae_s3_tool` helper). Spack: `+s3_cae`. | -| `CAE_ENABLE_GCS` | OFF | Enable GCS assimilator (requires `google-cloud-cpp` storage). Spack: `+gcs`. | - -The S3 and GCS **storage tiers** (block devices) are separate features with their own flags (`CLIO_ENABLE_AMAZON_DRIVE`, `CLIO_ENABLE_GOOGLE_CLOUD`). See [Configuration → Cloud object-store block devices](../../deployment/configuration#cloud-bdevs). +| `CAE_ENABLE_S3` | OFF | Enable S3 assimilator (requires POCO and the AWS SDK). | +| `CAE_ENABLE_GCS` | OFF | Enable GCS assimilator (requires `google-cloud-cpp` storage) | ## Related Documentation diff --git a/docs/sdk/context-assimilation-engine/s3.md b/docs/sdk/context-assimilation-engine/s3.md index abf3c272..eaa7828e 100644 --- a/docs/sdk/context-assimilation-engine/s3.md +++ b/docs/sdk/context-assimilation-engine/s3.md @@ -13,7 +13,7 @@ The connector is implemented by `S3FileAssimilator` and is conditionally compile Objects are read **in-process** inside the runtime: the assimilator signs each request itself (AWS Signature Version 4) and streams it over HTTPS with [POCO](https://pocoproject.org/). The AWS SDK for C++ is never loaded into the runtime, because its load-time initialization corrupts CLIO Runtime startup. The SDK is used only by the standalone [`cae_s3_tool`](#the-cae_s3_tool-helper) helper. :::note Not the S3 storage tier -This page covers **importing** S3 objects through CAE. Using an S3 bucket as a CTE **storage tier** (the `s3` block device) is a separate feature with its own build flag and its own credential rules. See [Configuration → Cloud object-store block devices](../../deployment/configuration#cloud-bdevs). +This page covers **importing** S3 objects through CAE. Using an S3 bucket as a CTE **storage tier** (the `s3` block device) is a separate feature with its own build flag and its own credential rules. See [Configuration → Storage Tiers](../../deployment/configuration#storage-tiers-storage). ::: ## Prerequisites @@ -60,7 +60,7 @@ You need a bucket and an access key whose IAM policy allows `s3:GetObject` on th Credentials and region are resolved **inside the runtime process** (the `clio_run` daemon hosting the CAE pool), once per imported object. The environment of the `clio_cae` client that submits the OMNI file is **not** consulted, so export the variables in the shell (or job script) that starts the runtime. -Only the profile **name** ever travels in a task. Secret keys stay in the runtime's environment or credentials file. +Only the profile **name** ever travels in a task, so secret keys never appear in an OMNI file or task payload. They stay in the runtime's environment or credentials file. ### Credential resolution diff --git a/docs/sdk/context-runtime/20.base-modules/2.bdev.md b/docs/sdk/context-runtime/20.base-modules/2.bdev.md index dca611f1..512df284 100644 --- a/docs/sdk/context-runtime/20.base-modules/2.bdev.md +++ b/docs/sdk/context-runtime/20.base-modules/2.bdev.md @@ -434,7 +434,7 @@ enum class BdevType : clio::run::u32 { - **kS3**: Stores each block as one object (`[prefix/]block_`) in an Amazon S3 or S3-compatible bucket. A HTTPS client built on `Poco::Net` signs requests with AWS SigV4; the AWS SDK is deliberately not used, because loading it into a process that calls `CLIO_INIT` corrupts runtime startup. Each worker keeps its own keep-alive connection, unwritten blocks read back as zeros, and freeing a block deletes its object (best effort). Persistent. Compiled only with `-DCLIO_ENABLE_AMAZON_DRIVE=ON`. - **kGcs**: The same object-per-block layout on Google Cloud Storage, authenticated with an OAuth2 bearer token (`GCS_ACCESS_TOKEN`). Persistent. Compiled only with `-DCLIO_ENABLE_GOOGLE_CLOUD=ON`. -Credentials, endpoints, and bucket-creation policy for `kS3` / `kGcs` come from environment variables only. See [Configuration → Cloud object-store block devices](../../../deployment/configuration#cloud-bdevs). +Credentials, endpoints, and bucket-creation policy for `kS3` / `kGcs` come from environment variables only. See [Configuration → Cloud object stores](../../../deployment/configuration#cloud-object-stores). ### Block Structure Represents an allocated block of storage. diff --git a/docs/sdk/context-transfer-engine/cte.md b/docs/sdk/context-transfer-engine/cte.md index fcb7cdcd..70f5f468 100644 --- a/docs/sdk/context-transfer-engine/cte.md +++ b/docs/sdk/context-transfer-engine/cte.md @@ -1248,7 +1248,7 @@ These are the values the config parser accepts; anything else is rejected at loa - `"s3"` - Amazon S3 or S3-compatible bucket; `path` is `s3:///` - `"gcs"` - Google Cloud Storage bucket; `path` is `gcs:///` -The cloud types need a runtime built with `CLIO_ENABLE_AMAZON_DRIVE` / `CLIO_ENABLE_GOOGLE_CLOUD`, and credentials in the runtime's environment. See [Configuration → Cloud object-store block devices](../../deployment/configuration#cloud-bdevs). +The cloud types need a runtime built with `CLIO_ENABLE_AMAZON_DRIVE` / `CLIO_ENABLE_GOOGLE_CLOUD`, and credentials in the runtime's environment. See [Configuration → Cloud object stores](../../deployment/configuration#cloud-object-stores). ### Manual Tier Scoring From d64c4a006371a551c041e74ff644c5dd963d451d Mon Sep 17 00:00:00 2001 From: Atharva Naik Date: Thu, 8 Oct 2026 02:33:08 +0000 Subject: [PATCH 3/3] docs(hpc-cluster): drop redundant cloud credential export example Co-Authored-By: Claude Opus 5.5 --- docs/deployment/hpc-cluster.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/deployment/hpc-cluster.md b/docs/deployment/hpc-cluster.md index 8a94041e..a0a84fb3 100644 --- a/docs/deployment/hpc-cluster.md +++ b/docs/deployment/hpc-cluster.md @@ -89,12 +89,6 @@ This variable is read by `CLIO_RUNTIME_INIT()`. If unset, the value of the `defa | `S3_ENDPOINT` | *(AWS)* | S3-compatible endpoint, e.g. MinIO. | | `GCS_ACCESS_TOKEN` | *(none)* | OAuth2 bearer token, for `gcs` storage tiers. | -```bash -export AWS_ACCESS_KEY_ID=... -export AWS_SECRET_ACCESS_KEY=... -export AWS_DEFAULT_REGION=us-east-2 # the bucket's real region -``` - See [Configuration → Cloud object stores](./configuration#cloud-object-stores) for every variable and which component reads it. ---