diff --git a/docs/deployment/configuration.md b/docs/deployment/configuration.md index bb8f6848..493b2e14 100644 --- a/docs/deployment/configuration.md +++ b/docs/deployment/configuration.md @@ -264,7 +264,7 @@ Block devices provide the shared memory allocator used by other modules. At leas | Parameter | Required | Description | |-----------|----------|-------------| -| `bdev_type` | Yes | `"ram"` for DRAM-backed, `"file"` for filesystem-backed. | +| `bdev_type` | Yes | `"ram"` for DRAM-backed, `"file"` for filesystem-backed, `"s3"` for an Amazon S3 (or S3-compatible) bucket, `"gcs"` for a Google Cloud Storage bucket, `"hbm"` for GPU device memory, `"pinned"` for page-locked host memory, `"noop"` for no-I/O latency testing. | | `capacity` | Yes | Maximum capacity (e.g., `"512MB"`, `"100GB"`). | ```yaml @@ -284,9 +284,29 @@ compose: # pool_id: "302.0" # bdev_type: file # capacity: "100GB" + + # Amazon S3 (or MinIO) bucket (optional) + # - mod_name: clio_bdev + # pool_name: "s3://my-bucket/clio/pool_0" # bucket "my-bucket", prefix "clio/pool_0" + # pool_query: local + # pool_id: "303.0" + # bdev_type: s3 + # capacity: "1TB" + + # Google Cloud Storage bucket (optional) + # - mod_name: clio_bdev + # pool_name: "gcs://my-bucket/clio/pool_0" + # pool_query: local + # pool_id: "304.0" + # bdev_type: gcs + # capacity: "1TB" ``` -For DRAM devices the `pool_name` uses the `ram::` convention. For file-backed devices the `pool_name` is the filesystem path where data is stored. +For DRAM devices the `pool_name` uses the `ram::` convention. For file-backed devices the `pool_name` is the filesystem path where data is stored. For `s3` / `gcs` devices the `pool_name` is `[s3://|gcs://][/]`. + +:::caution Composing one cloud pool on several nodes +A `clio_bdev` entry is created with the same `pool_name` on every node it reaches, and each node allocates blocks independently. With `pool_query: dynamic` or `broadcast`, every node therefore writes `block_` keys under the **same** prefix and overwrites the others' data. For multi-node deployments, use a CTE storage tier, which gives each node its own `_node`. +::: --- @@ -298,9 +318,9 @@ Array of storage targets. At least one entry is required when CTE is enabled. | Parameter | Required | Description | |-----------|----------|-------------| -| `path` | Yes | `ram::` for DRAM storage, or a filesystem path for disk. Supports `${HOME}` expansion. | -| `bdev_type` | Yes | `file`, `ram`, `hbm`, `pinned`, or `noop`. | -| `capacity_limit` | Yes | Maximum capacity (e.g., `"512MB"`, `"200GB"`). `0` / `"0g"` = 80% of total system DRAM. For file tiers this is a cap, not an upfront allocation — the file grows lazily in 1 GB units. | +| `path` | Yes | `ram::` for DRAM storage, a filesystem path for disk, or `s3:///` / `gcs:///` for a cloud bucket. Supports `${HOME}` expansion. | +| `bdev_type` | Yes | `file`, `ram`, `hbm`, `pinned`, `noop`, `s3`, or `gcs`. | +| `capacity_limit` | Yes | Maximum capacity (e.g., `"512MB"`, `"200GB"`). `0` / `"0g"` = 80% of total system DRAM for `ram` tiers. For `file` tiers this is a cap, not an upfront allocation — the file grows lazily in 1 GB units. For `s3` / `gcs` it is a logical cap; the bucket has no quota. | | `score` | No | Placement priority (0.0–1.0). Higher = preferred. `-1.0` (default) = automatic scoring. | | `persistence_level` | No | `"volatile"` (default), `"temporary"`, or `"long_term"`. | | `existing_pool_id` | No | Bind this target to an already-composed bdev pool instead of creating one. Skips `path` / `capacity_limit` validation — routing is purely by pool id. | @@ -327,8 +347,23 @@ storage: capacity_limit: 2TB score: 0.3 persistence_level: long_term + + # Cloud tier — an S3 (or MinIO) bucket; use gcs:// + bdev_type: gcs for GCS + - path: s3://my-bucket/clio-cte + bdev_type: s3 + capacity_limit: 10TB + score: 0.1 + persistence_level: long_term ``` +:::caution Always give a cloud tier a key prefix +Write `s3://my-bucket/clio-cte`, never `s3://my-bucket`. CTE appends `_node` to the whole path, one target per node: with no prefix it lands on the **bucket name** (`my-bucket_node0`), which names a bucket that does not exist (and is not a valid S3 bucket name). The config parses either way; the bare form only fails later, when the target is created. +::: + +:::caution Credentials must reach the runtime daemon +Cloud credentials have no YAML keys, so secret keys never end up in a config file that gets shared or committed. Export them where `clio_run start` runs, on every node. See [Cloud object stores](#cloud-object-stores). +::: + :::caution `persistence_level` is load-bearing Placement filters (`Context::min_persistence_level_`) and durable replicas (`REPLICA_PERSISTENT`, used by the @@ -800,6 +835,53 @@ compose: poll_period_ms: 5000 ``` +### RAM + Amazon S3 Tier + +A DRAM cache in front of an S3 bucket. Needs a runtime built with `-DCLIO_ENABLE_AMAZON_DRIVE=ON`, and credentials exported where the runtime starts: + +```bash +export AWS_ACCESS_KEY_ID=... +export AWS_SECRET_ACCESS_KEY=... +export AWS_DEFAULT_REGION=us-east-2 # the bucket's real region +# export S3_ENDPOINT=http://127.0.0.1:9000 # MinIO / S3-compatible store +clio_run start +``` + +```yaml +networking: + port: 9413 + +runtime: + num_threads: 8 + +compose: + - mod_name: clio_bdev + pool_name: "ram::chi_default_bdev" + pool_query: local + pool_id: "301.0" + bdev_type: ram + capacity: "2GB" + + - mod_name: clio_cte_core + pool_name: cte_main + pool_query: local + pool_id: "512.0" + storage: + - path: "ram::cte_cache" + bdev_type: ram + capacity_limit: 1GB + score: 1.0 + - path: s3://my-bucket/clio-cte # prefix is mandatory + bdev_type: s3 + capacity_limit: 10TB + score: 0.1 + persistence_level: long_term + dpe: + dpe_type: max_bw +``` + +For Google Cloud Storage, use `path: gcs://my-bucket/clio-cte` with `bdev_type: gcs`, build with `-DCLIO_ENABLE_GOOGLE_CLOUD=ON`, and export `GCS_ACCESS_TOKEN` instead of the AWS variables. + --- ## Docker Deployment @@ -913,6 +995,7 @@ concerns). The old `CHI_` prefix is **no longer recognized** — see | `CLIO_ZMQ_LOCAL_IPC` | *(on for macOS)* | Run the local client↔runtime ROUTER/DEALER over `ipc://` instead of TCP. The cross-node ROUTER is untouched, so multi-node TCP is unaffected. | | `CLIO_LBM_THALLIUM_PROTOCOL` | — | Thallium/Mercury protocol string (e.g. `ofi+verbs`). | | `CLIO_LBM_THALLIUM_RPC_THREADS` | — | Thallium RPC handler threads. | +| `CLIO_STRICT_RESPONSE_IDENTITY` | `0` | Client-side. `0` logs replies meant for an earlier task (`RESPONSE IDENTITY MISMATCH`); `1` drops them and keeps waiting. | ### Shared-memory ingest tuning @@ -949,6 +1032,35 @@ concerns). The old `CHI_` prefix is **no longer recognized** — see | `CLIO_BDEV_STATS_DIR` | `~/.clio/bdev_perf` | Directory holding per-bdev `.perf` files (measured latency/bandwidth plus the learned wall-clock model). | | `CLIO_BDEV_PERSIST_STATS` | `1` | `0` disables persistence, so each start begins from a cold model. | +### Cloud object stores (S3 / GCS) {#cloud-object-stores} + +**Amazon S3 and S3-compatible stores** + +| Variable | Default | Read by | Description | +|----------|---------|---------|-------------| +| `AWS_ACCESS_KEY_ID` | *(none)* | S3 bdev, CAE S3 | Access key. **Required** by the S3 bdev, which reads credentials only from the environment. CAE falls back to a credentials-file profile when it is unset. | +| `AWS_SECRET_ACCESS_KEY` | *(none)* | S3 bdev, CAE S3 | Secret key. Same rules as `AWS_ACCESS_KEY_ID`. | +| `AWS_SESSION_TOKEN` | *(none)* | S3 bdev, CAE S3 | Session token for temporary (STS) credentials. | +| `AWS_DEFAULT_REGION` | S3 bdev: `us-east-1`; CAE: *(required)* | S3 bdev, CAE S3 | Bucket region. Must be the bucket's **real** region — a mismatch is answered with HTTP 301, which is not followed. The S3 bdev falls back to `us-east-1`; CAE has no fallback and fails instead. | +| `AWS_REGION` | *(none)* | CAE S3 | Region fallback when `AWS_DEFAULT_REGION` is unset. | +| `AWS_PROFILE` | `default` | CAE S3 | Profile read from the credentials file when no key pair is in the environment. The S3 bdev has **no** profile support. | +| `AWS_SHARED_CREDENTIALS_FILE` | `~/.aws/credentials` | CAE S3 | Credentials file to read profiles from. | +| `AWS_CONFIG_FILE` | `~/.aws/config` | CAE S3 | Config file consulted for the profile's `region` as a last resort. | +| `S3_ENDPOINT` | *(unset = real AWS)* | S3 bdev, CAE S3 | S3-compatible endpoint (e.g. `http://127.0.0.1:9000` for MinIO). Setting it switches to path-style addressing. `AWS_ENDPOINT_URL` is **not** read by the runtime. | +| `S3_ALLOW_BUCKET_CREATE` | `0` | S3 bdev | `1` lets the S3 bdev create a missing bucket. Otherwise a missing bucket fails pool creation, so a typo cannot create a new billed bucket. CAE never creates buckets. | + +**Google Cloud Storage** + +| Variable | Default | Read by | Description | +|----------|---------|---------|-------------| +| `GCS_ACCESS_TOKEN` | *(none — required)* | GCS bdev | OAuth2 bearer token, e.g. from `gcloud auth print-access-token`. Read once when the pool is created; `gcloud` access tokens are short-lived (typically one hour), so long runs need a fresh token at each start. | +| `GCS_ENDPOINT` | `https://storage.googleapis.com` | GCS bdev, CAE GCS | GCS-compatible endpoint (e.g. `fake-gcs-server`). For the CAE importer, setting it also switches to unauthenticated requests. | +| `GCS_PROJECT_ID` | `clio-prototype` | GCS bdev | Project a missing bucket is created in. Only used when the bucket does not exist. | + +The CAE GCS importer (`gs://`) authenticates with Google +[Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials) +rather than `GCS_ACCESS_TOKEN`. + ### ADIOS2 adapter startup (large-scale MPI) At 512+ nodes the local daemon is busy serving cross-node SWIM probes and can diff --git a/docs/deployment/hpc-cluster.md b/docs/deployment/hpc-cluster.md index b483d2b1..a0a84fb3 100644 --- a/docs/deployment/hpc-cluster.md +++ b/docs/deployment/hpc-cluster.md @@ -79,6 +79,18 @@ export CLIO_IPC_MODE=TCP This variable is read by `CLIO_RUNTIME_INIT()`. If unset, the value of the `default_with_runtime` argument passed to `CLIO_RUNTIME_INIT()` is used instead. +### Cloud Storage Credentials + +| Variable | Default | Description | +|----------|---------|-------------| +| `AWS_ACCESS_KEY_ID` | *(none)* | S3 access key, for `s3` storage tiers and CAE `s3://` imports. | +| `AWS_SECRET_ACCESS_KEY` | *(none)* | S3 secret key. | +| `AWS_DEFAULT_REGION` | `us-east-1` | The bucket's region. CAE has no default and fails if it is unset. | +| `S3_ENDPOINT` | *(AWS)* | S3-compatible endpoint, e.g. MinIO. | +| `GCS_ACCESS_TOKEN` | *(none)* | OAuth2 bearer token, for `gcs` storage tiers. | + +See [Configuration → Cloud object stores](./configuration#cloud-object-stores) for every variable and which component reads it. + --- ## Single-Node Deployment diff --git a/docs/sdk/context-assimilation-engine/omni.md b/docs/sdk/context-assimilation-engine/omni.md index 257f844a..d292e9d0 100644 --- a/docs/sdk/context-assimilation-engine/omni.md +++ b/docs/sdk/context-assimilation-engine/omni.md @@ -52,13 +52,16 @@ The source URI specifying where to read data from. The URI format determines the - `file::` - Local filesystem - `hdf5://:` - HDF5 file and dataset - `globus:///` - Globus endpoint -- `s3:///` - S3 object storage +- `s3:///` - Amazon S3 or an S3-compatible store (also `s3::/`). Requires `-DCAE_ENABLE_S3=ON`; see the [S3 Connector](s3.md) for credentials. +- `gs:///` - Google Cloud Storage (also `gcs://`). Requires `-DCAE_ENABLE_GCS=ON`. **Examples:** ```yaml src: file::/data/satellite/TERRA_2024.bin src: hdf5::/data/climate.h5:/temperature/surface src: globus://82f1b5c6-6e9b-11e5-ba47-22000b92c6ec/dataset.nc +src: s3://my-bucket/climate/TERRA_2024.bin +src: gs://my-bucket/climate/TERRA_2024.bin ``` ### `dst` (string, required) diff --git a/docs/sdk/context-assimilation-engine/overview.md b/docs/sdk/context-assimilation-engine/overview.md index 93219df5..0edb2792 100644 --- a/docs/sdk/context-assimilation-engine/overview.md +++ b/docs/sdk/context-assimilation-engine/overview.md @@ -6,7 +6,7 @@ sidebar_position: 1 ## Introduction -The Context Assimilation Engine (CAE) is a CLIO Runtime module (`clio_cae::core`) that ingests external data sources into the CLIO Runtime. It reads data from files, HDF5 datasets, or remote Globus endpoints and stores them as blobs in the Context Transfer Engine (CTE). The CAE is registered as a Module container with pool ID `400.0`. +The Context Assimilation Engine (CAE) is a CLIO Runtime module (`clio_cae::core`) that ingests external data sources into the CLIO Runtime. It reads data from files, HDF5 datasets, remote Globus endpoints, or Amazon S3 buckets and stores them as blobs in the Context Transfer Engine (CTE). The CAE is registered as a Module container with pool ID `400.0`. CAE ships a second, optional ChiMod: the **summarizer** (`clio_cae_summarizer`, pool `401.0`). It interposes on the CTE core's task interface and attaches an LLM-generated summary to each matching blob on the way through. See [the interposition chain](../context-transfer-engine/chimod-chain#summarizer-chimod-clio_cae_summarizer) for its behavior and [deployment configuration](../../deployment/configuration) for its keys. @@ -72,6 +72,8 @@ The CAE uses a factory pattern to select the correct assimilator based on the so | `file` | `file::/path/to/file` | `BinaryFileAssimilator` | Always enabled | | `hdf5` | `hdf5::/path/file.h5:/dataset` | `Hdf5FileAssimilator` | `-DWRP_CORE_ENABLE_HDF5=ON` | | `globus` | `globus:///` | `GlobusFileAssimilator` | `-DCAE_ENABLE_GLOBUS=ON` | +| `s3` | `s3:///` | `S3FileAssimilator` | `-DCAE_ENABLE_S3=ON` | +| `gs` / `gcs` | `gs:///` | `GcsFileAssimilator` | `-DCAE_ENABLE_GCS=ON` | The factory also detects Globus web URLs (`https://app.globus.org/...`) and routes them to `GlobusFileAssimilator`. @@ -105,6 +107,10 @@ class BaseAssimilator { **GlobusFileAssimilator** handles Globus transfers. It supports Globus-to-Globus transfers (via the Globus transfer API with submission IDs and polling) and Globus-to-local downloads (via HTTPS). Authentication tokens are passed through `ctx.src_token`. +**S3FileAssimilator** imports objects from Amazon S3 or an S3-compatible store (MinIO, via `S3_ENDPOINT`). It reads in-process: requests are SigV4-signed and streamed over POCO HTTPS, and the AWS SDK is never loaded into the runtime. The body is stored as 1 MB `chunk_` blobs plus a `description` blob, honoring `range_off` / `range_size`. A connection dropped mid-object is resumed with a ranged GET. Credentials and region are resolved inside the runtime from the `AWS_*` environment or a `~/.aws/credentials` profile; see the [S3 Connector](s3.md). + +**GcsFileAssimilator** imports objects from Google Cloud Storage (`gs://` or `gcs://`) using `google-cloud-cpp` and Application Default Credentials. Setting `GCS_ENDPOINT` points it at a GCS-compatible server such as `fake-gcs-server`. + ## AssimilationCtx `AssimilationCtx` is the serializable descriptor for a single data transfer: @@ -124,10 +130,14 @@ struct AssimilationCtx { std::string src_data; // Inline payload for "string::..." sources std::vector include_patterns; // Glob patterns to include std::vector exclude_patterns; // Glob patterns to exclude + std::string s3_region; // S3 region override (empty = resolve from env/config) + std::string s3_profile; // AWS profile NAME (empty = AWS_PROFILE or "default") }; } // namespace clio::cae::core ``` +`s3_region` and `s3_profile` are used only by `S3FileAssimilator`. When set, `s3_region` overrides `AWS_DEFAULT_REGION` / `AWS_REGION`, and `s3_profile` overrides `AWS_PROFILE`. A key pair in the runtime's environment still wins over any profile. OMNI files do not set them; they are for programmatic callers that build `AssimilationCtx` directly. + Serialization uses the [cereal](https://uscilab.github.io/cereal/) library with binary archives. The client serializes a `std::vector` into the `ParseOmniTask`, and the runtime deserializes it on the server side. ## Method IDs @@ -326,9 +336,13 @@ class MyAssimilator : public clio::cae::core::BaseAssimilator { |-------------|---------|-------------| | `WRP_CORE_ENABLE_HDF5` | OFF | Enable HDF5 assimilator (requires libhdf5) | | `CAE_ENABLE_GLOBUS` | OFF | Enable Globus assimilator (requires POCO) | +| `CAE_ENABLE_S3` | OFF | Enable S3 assimilator (requires POCO and the AWS SDK). | +| `CAE_ENABLE_GCS` | OFF | Enable GCS assimilator (requires `google-cloud-cpp` storage) | ## Related Documentation - [OMNI File Format](omni.md) - YAML configuration for data transfers +- [S3 Connector](s3.md) - Importing from Amazon S3 and S3-compatible stores +- [Globus Connector](globus.md) - Importing from Globus endpoints - [Module Development Guide](../context-runtime/2.module_dev_guide.md) - Module development - [CTE Documentation](../context-transfer-engine/cte.md) - CTE storage documentation diff --git a/docs/sdk/context-assimilation-engine/s3.md b/docs/sdk/context-assimilation-engine/s3.md new file mode 100644 index 00000000..eaa7828e --- /dev/null +++ b/docs/sdk/context-assimilation-engine/s3.md @@ -0,0 +1,308 @@ +--- +sidebar_position: 4 +--- + +# Amazon S3 Connector + +## Overview + +The S3 connector lets the Context Assimilation Engine (CAE) import objects from [Amazon S3](https://aws.amazon.com/s3/), or any S3-compatible object store such as [MinIO](https://min.io/), into IOWarp. Each object named in an OMNI file is streamed straight from S3 into a CTE tag. + +The connector is implemented by `S3FileAssimilator` and is conditionally compiled with the `CAE_ENABLE_S3` CMake flag. + +Objects are read **in-process** inside the runtime: the assimilator signs each request itself (AWS Signature Version 4) and streams it over HTTPS with [POCO](https://pocoproject.org/). The AWS SDK for C++ is never loaded into the runtime, because its load-time initialization corrupts CLIO Runtime startup. The SDK is used only by the standalone [`cae_s3_tool`](#the-cae_s3_tool-helper) helper. + +:::note Not the S3 storage tier +This page covers **importing** S3 objects through CAE. Using an S3 bucket as a CTE **storage tier** (the `s3` block device) is a separate feature with its own build flag and its own credential rules. See [Configuration → Storage Tiers](../../deployment/configuration#storage-tiers-storage). +::: + +## Prerequisites + +### Build Dependencies + +| Dependency | Purpose | +|-----------|---------| +| [POCO](https://pocoproject.org/) (Net, NetSSL, Crypto, Foundation) | In-process HTTPS reads and SigV4 signing inside the runtime | +| [AWS SDK for C++](https://github.com/aws/aws-sdk-cpp) (`s3` component) | Builds the out-of-process `cae_s3_tool` helper only | + +Both are required. If either is missing, CMake warns and S3 import is disabled. + +### Build with S3 Support + +```bash +cmake --preset release -DCAE_ENABLE_S3=ON +cmake --build build +``` + +Verify that the configure output includes: + +``` +found AWS SDK + Poco (CAE S3 support: ENABLED) +``` + +If you instead see `S3 import will be disabled.`, the warning just above it names the missing dependency: the AWS SDK (`aws-sdk-cpp`) or POCO (`libpoco-dev` on Debian/Ubuntu). + +With Spack, enable the `s3_cae` variant (CAE itself is on by default): + +```bash +spack install iowarp +s3_cae +``` + +:::note Variant rename +This variant used to be called `+s3`. It was renamed to `+s3_cae` so it is not confused with `+s3_bdev`, which builds the S3 **storage tier** and has different dependencies. +::: + +### AWS Account + +You need a bucket and an access key whose IAM policy allows `s3:GetObject` on the objects you import. Also grant `s3:ListBucket` on the bucket: without it, S3 reports a missing key as HTTP 403 instead of 404, which hides the real cause. + +## Credentials and Region + +Credentials and region are resolved **inside the runtime process** (the `clio_run` daemon hosting the CAE pool), once per imported object. The environment of the `clio_cae` client that submits the OMNI file is **not** consulted, so export the variables in the shell (or job script) that starts the runtime. + +Only the profile **name** ever travels in a task, so secret keys never appear in an OMNI file or task payload. They stay in the runtime's environment or credentials file. + +### Credential resolution + +Highest priority first: + +1. **Environment keys.** `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (plus `AWS_SESSION_TOKEN` for temporary credentials). Both the key ID and the secret must be set for this step to apply. +2. **Credentials file.** Otherwise, read a profile section from the shared credentials file: + - **Profile:** `AWS_PROFILE`, else `default`. + - **File:** `AWS_SHARED_CREDENTIALS_FILE`, else `~/.aws/credentials` (the runtime process's `$HOME`). + - **Keys read:** `aws_access_key_id`, `aws_secret_access_key`, and the optional `aws_session_token`. + +Only static keys are supported. SSO profiles, `credential_process`, EC2/ECS instance roles and `role_arn` assume-role chains are **not** resolved. Export temporary keys into the environment instead. + +### Region resolution + +Highest priority first: + +1. `AWS_DEFAULT_REGION` +2. `AWS_REGION` +3. The `region` key in the config file: `AWS_CONFIG_FILE`, else `~/.aws/config`. Section `[default]` for the default profile, `[profile ]` (or `[]`) for a named one. + +There is **no fallback region**. If none of these is set, the import fails rather than guessing `us-east-1`, because a wrong region makes S3 answer HTTP 301, which the signer does not follow. + +### Example + +```ini +# ~/.aws/credentials (chmod 600) +[clio-bench] +aws_access_key_id = AKIA... +aws_secret_access_key = ... +``` + +```ini +# ~/.aws/config +[profile clio-bench] +region = us-east-2 +``` + +```bash +# In the shell that starts the runtime +export AWS_PROFILE=clio-bench +clio_run start +``` + +## S3-Compatible Endpoints (MinIO) + +Set `S3_ENDPOINT` in the runtime's environment to target an S3-compatible server: + +```bash +export S3_ENDPOINT=http://127.0.0.1:9000 +export AWS_DEFAULT_REGION=us-east-1 # still required; MinIO accepts us-east-1 +export AWS_ACCESS_KEY_ID=minioadmin +export AWS_SECRET_ACCESS_KEY=minioadmin +clio_run start +``` + +| `S3_ENDPOINT` | Addressing | Request URL | +|---------------|-----------|-------------| +| unset | Virtual-hosted (real AWS) | `https://.s3..amazonaws.com/` | +| set | Path-style | `//` | + +A trailing `/` on the endpoint is stripped. + +:::caution `AWS_ENDPOINT_URL` is not read by the runtime +The in-process assimilator reads only `S3_ENDPOINT`. `AWS_ENDPOINT_URL` is honored by `cae_s3_tool` (through the AWS SDK) but **not** by the runtime, so a setup that relies on it will seed objects into MinIO with the helper and then try to read them from real AWS. +::: + +## OMNI File Format for S3 + +### `s3://` URI + +```yaml +name: s3_import + +transfers: + - src: s3://my-bucket/path/to/object.bin + dst: iowarp::my_dataset + format: binary + depends_on: "" + range_off: 0 + range_size: 0 +``` + +| Field | Notes | +|-------|-------| +| `src` | `s3:///`. The `s3::/` form is also accepted. The key must be non-empty: a bare `s3://` is rejected. | +| `dst` | Must be `iowarp::`. No other destination protocol is supported. | +| `format` | `binary`. The object is stored as opaque bytes. | +| `range_off` / `range_size` | `range_size: 0` imports the whole object. A positive `range_size` imports bytes `[range_off, range_off + range_size)` with a ranged GET. A bare `range_off` with `range_size: 0` still means the whole object. | + +Submit it with `clio_cae`, the same as any OMNI file: + +```bash +clio_cae /path/to/s3_import.yaml +``` + +### What lands in CTE + +For each transfer, the assimilator gets or creates the destination tag and writes: + +| Blob | Contents | +|------|----------| +| `description` | `binary, offset=>` | +| `chunk_0`, `chunk_1`, … | The object body, in chunks of up to 1 MB | + +Body chunks are written with up to 32 `PutBlob` operations in flight. If the HTTPS connection drops mid-object, the assimilator resumes with a ranged GET from the last byte received (up to 16 times per object) instead of failing the import. + +Keep-alive connections are pooled per runtime and reused across objects, so a batch of many small objects does not pay a TLS handshake each. The assimilator never creates a bucket. + +## Quick Start + +### 1. Build with S3 Support + +```bash +cmake --preset release -DCAE_ENABLE_S3=ON +cmake --build build && cmake --install build +``` + +### 2. Provide Credentials to the Runtime + +Either export keys: + +```bash +export AWS_ACCESS_KEY_ID=... +export AWS_SECRET_ACCESS_KEY=... +export AWS_DEFAULT_REGION=us-east-2 +``` + +or point at a profile in `~/.aws/credentials`: + +```bash +export AWS_PROFILE=my-profile +``` + +### 3. Start the Runtime with CTE and CAE + +Start `clio_run` **from the same shell**, with a config whose `compose` section includes `clio_cte_core` and `clio_cae_core` (the shipped default does): + +```bash +clio_run start +``` + +### 4. Import + +```bash +clio_cae s3_import.yaml +``` + +## The `cae_s3_tool` Helper + +When `CAE_ENABLE_S3` is on, the build also installs `cae_s3_tool` to `bin/`. It is a small standalone program that links the AWS SDK, so it never runs inside the runtime. It is handy for seeding a bucket, fetching an object to compare against what landed in CTE, or checking credentials outside the runtime. + +```text +cae_s3_tool get [range_off range_size] +cae_s3_tool put +cae_s3_tool del +``` + +- **`put`** also creates the bucket if it is missing (best effort). +- **`del`** treats a missing object as success. +- **Exit status:** `0` on success; non-zero with a message on stderr otherwise. + +The helper resolves its configuration through the **AWS SDK**, so it accepts more than the runtime does: + +| Setting | Read from | +|---------|-----------| +| Credentials | The SDK's default provider chain (environment, profiles, SSO, instance roles, …) | +| Region | `AWS_DEFAULT_REGION`, default `us-east-1` | +| Endpoint | `S3_ENDPOINT` or `AWS_ENDPOINT_URL` (path-style) | +| Anonymous access | `AWS_NO_SIGN_REQUEST=1` forces unsigned requests (public buckets such as the AWS Open Data registry). If the chain finds no credentials, it also falls back to anonymous. | + +:::caution +Because the helper's credential chain is broader, `cae_s3_tool get` can succeed where an import fails. A working helper proves the bucket and key are readable. It does not prove the runtime can resolve the same credentials (see [Credentials and Region](#credentials-and-region)). +::: + +## Error Codes + +The `S3FileAssimilator::Schedule` method returns the following error codes: + +| Code | Description | +|------|-------------| +| 0 | Success | +| -1 | Invalid destination protocol (not `iowarp`) | +| -2 | Destination URI has no tag name | +| -3 | Failed to get or create the destination tag in CTE | +| -4 | Invalid S3 source URI (missing bucket or key) | +| -6 | Credentials or region could not be resolved (the log names the profile and file tried) | +| -7 | Object not found, or the GET request failed | +| -8 | Short or failed read on a body chunk (after resume attempts) | +| -9 | Failed to store the `description` blob | +| -10 | A body `PutBlob` failed | + +If the build lacks S3 support, the factory logs `S3 protocol requested but S3 support not compiled in. Rebuild with -DCAE_ENABLE_S3=ON to enable S3 support.` and no assimilator is created. + +## Troubleshooting + +### `CAE S3 support` is missing from the configure output + +The configure step printed `S3 import will be disabled.` Install whichever dependency the preceding warning names (`aws-sdk-cpp`, or POCO's Net/NetSSL/Crypto components) and reconfigure. + +### `AWS credentials: no profile [] in the credentials file` + +The runtime found no `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` pair, fell back to the credentials file, and that file has no `[]` section. Check: +- `AWS_PROFILE` is spelled correctly in the **runtime's** environment. +- The file exists under the runtime user's `$HOME`, or set `AWS_SHARED_CREDENTIALS_FILE`. +- Credentials files use `[name]`, not `[profile name]`; that form belongs in `~/.aws/config`. + +### `profile [] is missing aws_access_key_id / aws_secret_access_key` + +The section exists, but one of the two keys is absent, or it is indented, which makes the parser treat it as a nested setting. Keys must start at column 0. + +### `AWS region unresolved for profile []` + +Set `AWS_DEFAULT_REGION` (or `AWS_REGION`) in the runtime's environment, or add `region = ...` to the profile in `~/.aws/config`. + +### HTTP 301 + +The region does not match the bucket's region. Find the bucket's real region (the S3 console shows it) and set `AWS_DEFAULT_REGION` to it. + +### HTTP 403 + +The keys are wrong or expired (temporary credentials need `AWS_SESSION_TOKEN`), the IAM policy lacks `s3:GetObject`, or the host clock is more than 15 minutes off, which invalidates SigV4 signatures. A missing key also returns 403 when the policy lacks `s3:ListBucket`. + +### `object not found: s3:///` + +The key does not exist. Keys are case-sensitive and have no leading `/`. With `S3_ENDPOINT` set, confirm you are pointed at the store you seeded. + +### Imports work in your shell but fail under a launcher + +The runtime daemon did not inherit your environment. Launchers that start `clio_run` on other nodes (pssh, Slurm prologs, Jarvis) pass a curated environment, so `AWS_*` variables exported in your shell or job script may never reach it. Export them where the daemon is started. With the `jarvis_clio_core` runtime package, list them in its `forward_env` parameter. + +## Test Files Reference + +| File | Description | +|------|-------------| +| `context-assimilation-engine/test/unit/s3_assim/test_s3_assim.cc` | End-to-end import against an S3-compatible store: seeds an object with `cae_s3_tool`, runs `ParseOmni`, verifies the bytes in CTE. Self-skips unless `S3_ENDPOINT` is set (bucket via `S3_TEST_BUCKET`, default `clio-cae-test`). | +| `context-assimilation-engine/test/unit/s3_assim/s3_assim_omni.yaml` | Example OMNI file for an S3 import | +| `context-assimilation-engine/test/unit/aws_creds/test_aws_creds.cc` | Offline tests for credential and region resolution precedence | + +## Related Documentation + +- [OMNI File Format](omni.md) - Full OMNI YAML specification +- [CAE Overview](overview.md) - CAE architecture and API reference +- [Configuration Reference](../../deployment/configuration#cloud-object-stores) - Every S3/GCS environment variable, and the S3 storage tier +- [CTE Documentation](../context-transfer-engine/cte.md) - CTE storage documentation diff --git a/docs/sdk/context-runtime/20.base-modules/2.bdev.md b/docs/sdk/context-runtime/20.base-modules/2.bdev.md index a75c5c1c..512df284 100644 --- a/docs/sdk/context-runtime/20.base-modules/2.bdev.md +++ b/docs/sdk/context-runtime/20.base-modules/2.bdev.md @@ -431,6 +431,10 @@ enum class BdevType : clio::run::u32 { **Backend Characteristics:** - **kFile**: Uses file-based storage with libaio for asynchronous I/O, supports alignment requirements, persistent data - **kRam**: Uses malloc-allocated RAM buffer, synchronous operations, volatile data (lost on restart) +- **kS3**: Stores each block as one object (`[prefix/]block_`) in an Amazon S3 or S3-compatible bucket. A HTTPS client built on `Poco::Net` signs requests with AWS SigV4; the AWS SDK is deliberately not used, because loading it into a process that calls `CLIO_INIT` corrupts runtime startup. Each worker keeps its own keep-alive connection, unwritten blocks read back as zeros, and freeing a block deletes its object (best effort). Persistent. Compiled only with `-DCLIO_ENABLE_AMAZON_DRIVE=ON`. +- **kGcs**: The same object-per-block layout on Google Cloud Storage, authenticated with an OAuth2 bearer token (`GCS_ACCESS_TOKEN`). Persistent. Compiled only with `-DCLIO_ENABLE_GOOGLE_CLOUD=ON`. + +Credentials, endpoints, and bucket-creation policy for `kS3` / `kGcs` come from environment variables only. See [Configuration → Cloud object stores](../../../deployment/configuration#cloud-object-stores). ### Block Structure Represents an allocated block of storage. @@ -547,9 +551,11 @@ struct CreateParams { - **pool_name**: - For kFile: **IS the file path** (can be block device `/dev/nvme0n1` or regular file) - For kRam: Unique identifier for the RAM device + - For kS3 / kGcs: `[s3://|gcs://][/]` — the bucket, plus an optional key prefix after the first `/` (trailing slashes dropped) - **total_size_**: - For kFile: Set to 0 to use full file/device size, or specify limit - For kRam: **Required** - specifies the RAM buffer size to allocate + - For kS3 / kGcs: Logical capacity the allocator hands out (the bucket has no quota); 0 means 1 TB - **io_depth_**: Higher values improve parallelism for kFile but use more memory (typical: 16-128), ignored for kRam - **alignment_**: Must match device requirements for kFile (typically 512 or 4096 bytes), less critical for kRam @@ -861,6 +867,7 @@ void example(clio::run::bdev::Client& bdev_client) { - **cereal**: Serialization library for network communication - **libaio**: Linux asynchronous I/O library for high-performance block operations - **Boost.Fiber** and **Boost.Context**: Coroutine support +- **POCO** (Net, NetSSL, Crypto) — *optional*: HTTPS transport for the `kS3` and `kGcs` backends. When it is not found, `CLIO_ENABLE_AMAZON_DRIVE` and `CLIO_ENABLE_GOOGLE_CLOUD` are switched off with a CMake warning. ## Installation @@ -972,6 +979,11 @@ void example() { - Integration with existing storage infrastructure - Need for data durability across restarts +**Use a Cloud Backend (`BdevType::kS3` / `BdevType::kGcs`) when:** +- Capacity needs exceed local disks, or data must outlive the cluster's storage +- The tier is a cold or overflow tier — every block is a network round trip to the object store +- Nodes share no filesystem but can all reach the same bucket + ### Performance Tuning 1. **Block Size Selection**: Choose appropriate block sizes based on I/O patterns diff --git a/docs/sdk/context-transfer-engine/cte.md b/docs/sdk/context-transfer-engine/cte.md index 13fdfe6e..70f5f468 100644 --- a/docs/sdk/context-transfer-engine/cte.md +++ b/docs/sdk/context-transfer-engine/cte.md @@ -1238,10 +1238,17 @@ void example() { ### Storage Device Types +These are the values the config parser accepts; anything else is rejected at load time. + - `"file"` - File-based block device - `"ram"` - RAM-based block device (for caching) -- `"dev_dax"` - Persistent memory device -- `"posix"` - POSIX file system interface +- `"hbm"` - GPU high-bandwidth (device) memory +- `"pinned"` - Pinned host memory +- `"noop"` - No-op backend for latency testing (no actual I/O) +- `"s3"` - Amazon S3 or S3-compatible bucket; `path` is `s3:///` +- `"gcs"` - Google Cloud Storage bucket; `path` is `gcs:///` + +The cloud types need a runtime built with `CLIO_ENABLE_AMAZON_DRIVE` / `CLIO_ENABLE_GOOGLE_CLOUD`, and credentials in the runtime's environment. See [Configuration → Cloud object stores](../../deployment/configuration#cloud-object-stores). ### Manual Tier Scoring