diff --git a/content/patterns/secure-agent-workspace/_index.adoc b/content/patterns/secure-agent-workspace/_index.adoc new file mode 100644 index 000000000..52bf6bc83 --- /dev/null +++ b/content/patterns/secure-agent-workspace/_index.adoc @@ -0,0 +1,42 @@ +--- +title: Secure Agent Workspace +date: 2026-09-30 +tier: sandbox +summary: This pattern gives each user an isolated AI agent workspace in their own OpenShift Virtualization VM, running NVIDIA OpenShell and OpenClaw agents with OIDC sign-in, governed sandbox policy, and API keys kept in Vault. +rh_products: + - Red Hat OpenShift Container Platform + - Red Hat OpenShift Virtualization + - Red Hat OpenShift GitOps + - Red Hat build of Keycloak +partners: + - NVIDIA +industries: + - General +focus_areas: + - AI + - Security + - Virtualization +aliases: /secure-agent-workspace/ +links: + github: https://github.com/validatedpatterns-sandbox/secure-agent-workspace + install: getting-started + arch: /images/secure-agent-workspace/saw-architecture.png + bugs: https://github.com/validatedpatterns-sandbox/secure-agent-workspace/issues + feedback: https://docs.google.com/forms/d/e/1FAIpQLScI76b6tD1WyPu2-d_9CCVDr3Fu5jYERthqLKJDUGwqBg7Vcg/viewform +--- +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] + +include::modules/secure-agent-workspace-about.adoc[leveloffset=+1] + +include::modules/secure-agent-workspace-architecture.adoc[leveloffset=+1] + +[id="next-steps-secure-agent-workspace"] +== Next steps + +* link:getting-started[Getting started] +* link:cluster-sizing[Cluster sizing] +* link:ideas-for-customization[Ideas for customization] +* link:troubleshooting[Troubleshooting] diff --git a/content/patterns/secure-agent-workspace/cluster-sizing.adoc b/content/patterns/secure-agent-workspace/cluster-sizing.adoc new file mode 100644 index 000000000..ce5b6518c --- /dev/null +++ b/content/patterns/secure-agent-workspace/cluster-sizing.adoc @@ -0,0 +1,68 @@ +--- +title: Cluster sizing +weight: 20 +aliases: /secure-agent-workspace/cluster-sizing/ +--- + +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] +include::modules/secure-agent-workspace/metadata-secure-agent-workspace.adoc[] + +[id="secure-agent-workspace-bare-metal"] +== Hardware virtualization requirements + +Each workspace is a {VirtProductName} virtual machine. {VirtProductName} uses the hardware virtualization of the node (Intel VT-x or AMD-V through KVM), so the worker nodes that run the workspace VMs must be bare metal: + +* *On premises*: {ocp} installed on bare-metal servers, with virtualization enabled in the firmware. +* *Public cloud*: bare-metal instance types, for example `m5.metal`, `m6i.metal`, or `c5n.metal` on {AWS}. Virtualized instance types cannot run the VMs, because nested virtualization is not supported. + +The control plane nodes can be virtual machines or virtualized instances. For the full list of requirements, see link:https://docs.redhat.com/en/documentation/openshift_container_platform/latest/html/virtualization/installing#preparing-cluster-for-virt[Preparing your cluster for OpenShift Virtualization]. + +[id="secure-agent-workspace-sizing"] +== Sizing requirements + +Size the bare-metal workers for the shared services plus one VM per user. Each VM requests its full memory from the node, so the amount of memory on the node limits the number of workspaces the node can run. + +.Resources per component +[cols="2,1,1,1",options="header"] +|=== +| Component | vCPU | Memory | Storage + +| Shared services: Operators, Argo CD, Keycloak and its database, Vault, governance interceptor +| 8 +| 16 GiB +| 50 GiB, including the template VM image + +| Each workspace VM (default, set in the `openshell-saw` chart as `vm.cores`, `vm.memory`, `vm.diskSize`) +| 4 +| 8 GiB +| 40 GiB +|=== + +.Example minimum cluster +[cols="<,^,<",options="header"] +|=== +| Node type | Number of nodes | Example (AWS) + +| Control plane +| 3 +| `m5.xlarge` + +| Bare-metal worker +| 2 +| `m5.metal` (96 vCPU, 384 GiB memory) +|=== + +Two bare-metal workers of this size hold the shared services and dozens of workspaces. With smaller servers, plan for the shared services plus 4 vCPU and 8 GiB of memory per user. Ensure that there is room to move VMs from one node to another in case of node failure. + +[id="secure-agent-workspace-storage"] +== Storage requirements + +The cluster needs a default storage class for the VM disks (`ReadWriteOnce`, 40 GiB per user by default) and for the template image that the disks are cloned from. Storage that supports smart cloning, such as {rh-ocp-data-first} or a CSI driver with volume snapshots, makes new workspaces start faster. + +[id="secure-agent-workspace-model-serving"] +== Model serving + +By default, the agents use NVIDIA Nemotron on build.nvidia.com and need no GPU in the cluster. If you serve models on the cluster instead, for example with vLLM, add the GPU nodes that the model server needs. For more information, see link:ideas-for-customization[Ideas for customization]. diff --git a/content/patterns/secure-agent-workspace/getting-started.adoc b/content/patterns/secure-agent-workspace/getting-started.adoc new file mode 100644 index 000000000..a56876887 --- /dev/null +++ b/content/patterns/secure-agent-workspace/getting-started.adoc @@ -0,0 +1,275 @@ +--- +title: Getting started +weight: 10 +aliases: /secure-agent-workspace/getting-started/ +--- + +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] + +[id="deploying-secure-agent-workspace-pattern"] +== Deploying the Secure Agent Workspace pattern + +.Prerequisites + +* An {ocp} cluster with *bare-metal worker nodes* to provide hardware virtualization to run a separate {VirtProductName} VM for each workspace. {VirtProductName} does not support nested virtualization. +** *For on-premises deployments*, install {ocp} on bare-metal servers with Intel VT-x or AMD-V enabled in the firmware. +** *For public cloud deployments*, select bare-metal instance types for workers, for example, on AWS, select `m5.metal` or `c5n.metal`. +** See link:cluster-sizing[Cluster sizing] for advice on the size and number of nodes required. +** You can create an {ocp} cluster by using the link:https://console.redhat.com/openshift/create[Red{nbsp}Hat Hybrid Cloud Console]. +* A default storage class that can provision `ReadWriteOnce` volumes for the VM disks. +* Cluster administrator access. End users need no {ocp} access: they sign in to the workspaces with Keycloak. +* An API key for each service that the default profile uses: build.nvidia.com (NVIDIA Nemotron) and Brave Search. To use a model server of your own instead, see link:ideas-for-customization[Ideas for customization]. +* The `oc` CLI. For instructions, see link:https://docs.openshift.com/container-platform/latest/cli_reference/openshift_cli/getting-started-cli.html[Getting started with the OpenShift CLI]. +* The Helm binary. For instructions, see link:https://helm.sh/docs/intro/install/[Installing Helm]. +* The `openshell` CLI, version 0.1.x, from the link:https://github.com/NVIDIA/OpenShell/releases[OpenShell releases]. A 0.0.x CLI cannot connect to the 0.1.x gateway that this pattern installs. +* Additional installation tool dependencies. For details, see link:https://validatedpatterns.io/learn/quickstart/[Patterns quick start]. + +[id="preparing-for-deployment-saw"] +== Preparing for deployment +.Procedure + +. Fork the link:https://github.com/validatedpatterns-sandbox/secure-agent-workspace[secure-agent-workspace] repository on GitHub. You must fork the repository to add users and to customize this pattern. + +. Clone the forked copy of this repository. ++ +[source,terminal,subs="+quotes"] +---- +$ git clone git@github.com:____/secure-agent-workspace.git +---- + +. Go to the root directory of your Git repository: ++ +[source,terminal] +---- +$ cd secure-agent-workspace +---- + +. Run the following command to set the upstream repository: ++ +[source,terminal] +---- +$ git remote add -f upstream git@github.com:validatedpatterns-sandbox/secure-agent-workspace.git +---- + +. Generate the SSH key pair that the pattern uses to reach the workspace VMs for troubleshooting: ++ +[source,terminal] +---- +$ make generate-keys +---- ++ +The keys are written to `~/.generated-ssh-keys/`. + +. Save your API keys in files outside the repository, one key per line: ++ +[source,terminal] +---- +$ echo '' > ~/.nvidia-api-key +$ echo '' > ~/.brave-api-key +$ chmod 600 ~/.nvidia-api-key ~/.brave-api-key +---- + +. Make a local copy of the secrets template outside your repository to hold credentials for the pattern. ++ +[WARNING] +==== +Do not add, commit, or push this file to your repository. Doing so might expose personal credentials to GitHub. +==== ++ +Run the following command: ++ +[source,terminal] +---- +$ cp values-secret.yaml.template ~/values-secret-secure-agent-workspace.yaml +---- ++ +The template reads the SSH keys and the two API key files from the paths above. Edit your copy of the secrets file to store your API key files in a different location, or to use a different model provider. + +. Optional: Choose who gets a workspace. Edit `overrides/saw-users.yaml`. Each entry in the `users` list defines one user. The `name` of each user is also used for that user's namespace and for their VM: ++ +[source,yaml] +---- +users: + - name: alice + profiles: + - data-science +---- ++ +The name must also be a user in Keycloak. The pattern's realm includes the test users `alice` and `bob`. Each name must be a lowercase DNS label no more than 19 characters long. + +. Optional: To customize the deployment, create and switch to a new branch by running the following command: ++ +[source,terminal] +---- +$ git checkout -b my-branch +---- ++ +Make your changes, then stage, commit, and push them: ++ +[source,terminal] +---- +$ git add +$ git commit -m "Customize deployment" +$ git push origin my-branch +---- ++ +The branch that you deploy must exist on your fork, because {rh-gitops} syncs from it. + +[id="deploying-cluster-using-patternsh-file-saw"] +== Deploying the pattern by using the pattern.sh file + +To deploy the pattern by using the `pattern.sh` file, complete the following steps: + +. Log in to your cluster. + +.. Obtain an API token by visiting `https://oauth-openshift.apps../oauth/token/request`. + +.. Log in to the cluster by running the following command: ++ +[source,terminal] +---- +$ oc login --token= --server=https://api..:6443 +---- ++ +Or log in by running the following command: ++ +[source,terminal] +---- +$ export KUBECONFIG=~/ +---- + +. Copy the prebuilt template VM image into the cluster's internal registry. This takes about 5 minutes: ++ +[source,terminal] +---- +$ make copy-images +---- + +. Deploy the pattern to your cluster. Run the following command: ++ +[source,terminal] +---- +$ ./pattern.sh make install +---- ++ +To deploy a branch other than the one that is checked out, export `TARGET_BRANCH` before you run the command, and `TARGET_ORIGIN` if the branch is on a remote other than `origin`. For example: ++ +[source,terminal] +---- +$ export TARGET_BRANCH=my-branch +$ export TARGET_ORIGIN=origin +$ ./pattern.sh make install +---- + +.Verification + +. Check the health of the Argo CD applications: ++ +[source,terminal] +---- +$ ./pattern.sh make argo-healthcheck +---- ++ +It might take 20 to 30 minutes for all applications to synchronize, the Operators to install, and the first workspace VM to boot. + +. Verify that the Operators are installed. In the {ocp} web console, go to *Operators \-> Installed Operators* and confirm that the following Operators are present: ++ +* {VirtProductName} +* Red{nbsp}Hat build of Keycloak +* {eso} + +. Verify that each user's VM is running. For the user `alice`: ++ +[source,terminal] +---- +$ oc get vm -n saw-alice +---- ++ +.Example output ++ +[source,terminal] +---- +NAME AGE STATUS READY +alice 12m Running True +---- + +. Check that the installer in the VM finished. Both steps must show `"phase": "Done"`: ++ +[source,terminal] +---- +$ make openshell-saw-status OPENSHELL_SAW_NAME=alice +---- ++ +To follow the installer as it runs, use `make openshell-saw-logs OPENSHELL_SAW_NAME=alice`. + +[id="accessing-a-workspace-saw"] +== Accessing a workspace + +. Register the user's gateway and log in. ++ +Run the following commands, then use the browser window that opens to log in using Keycloak. The test user is `alice` with password `alice`. ++ +[source,terminal] +---- +$ export OPENSHELL_SAW_NAME=alice +$ make openshell-saw-configure-gateway +$ openshell gateway login alice +---- + +. List the sandboxes of the default profile: ++ +[source,terminal] +---- +$ openshell sandbox list +$ openshell sandbox list --workspace cuda-dev +---- ++ +.Example output ++ +[source,terminal] +---- +NAME CREATED PHASE +notebook 2026-09-30 14:40:42 Ready +---- + +. Open the OpenClaw agent in the `notebook` sandbox, in the terminal or in the browser: ++ +[source,terminal] +---- +$ make openclaw-tui SANDBOX_NAME=notebook +$ make openclaw-gui SANDBOX_NAME=notebook GUI_PORT=28789 +---- ++ +The browser UI opens through an SSH tunnel on `http://localhost:28789`. + +. Open the OpenShell web UI, which lists the user's workspaces and sandboxes: ++ +[source,terminal] +---- +$ oc get route alice-webui -n saw-alice -o jsonpath='https://{.spec.host}{"\n"}' +---- + +. Optional: See the sandbox policy in effect. GitHub is not one of the hosts that the default profile allows, so the sandbox's egress proxy blocks the agent's request. In the OpenClaw agent of the `notebook` sandbox, enter the following prompt: ++ +[source,text] +---- +Use your terminal tool to run curl -sS --max-time 10 https://api.github.com/zen. Show the command and its exact output. +---- ++ +The command fails because the connection is denied. To see the denial, view the sandbox logs: ++ +[source,terminal] +---- +$ openshell --gateway alice --workspace default \ + logs notebook --since 5m --source sandbox +---- + +[id="next-steps-getting-started-saw"] +== Next steps + +* link:ideas-for-customization[Ideas for customization] +* link:cluster-sizing[Cluster sizing] +* link:troubleshooting[Troubleshooting] diff --git a/content/patterns/secure-agent-workspace/ideas-for-customization.adoc b/content/patterns/secure-agent-workspace/ideas-for-customization.adoc new file mode 100644 index 000000000..e963f9344 --- /dev/null +++ b/content/patterns/secure-agent-workspace/ideas-for-customization.adoc @@ -0,0 +1,122 @@ +--- +title: Ideas for customization +weight: 30 +aliases: /secure-agent-workspace/ideas-for-customization/ +--- + +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] + +[id="ideas-for-customization"] +== Ideas for customizing the Secure Agent Workspace pattern + +Everything in this pattern is configured in Git: who gets a workspace, what each workspace contains, which providers and hosts agents can use, and which OpenShell release the VMs run. Commit and push a change to the branch that the pattern deploys, and {rh-gitops} applies it. + +[id="adding-users-saw"] +=== Adding and removing users + +Each entry in `overrides/saw-users.yaml` is one workspace: + +[source,yaml] +---- +users: + - name: alice + profiles: + - data-science + - name: bob + profiles: + - data-science + # Optional: + vaultPrefix: secret/data/hub/saw-bob # bob's own API keys in Vault + ownerSubject: # from `openshell whoami` after bob's first sign-in +---- + +* `name` is the user's Keycloak user name, the VM name, and the namespace suffix (`saw-`). It must be a lowercase DNS label of at most 19 characters. +* `profiles` lists the SAW-BOM profiles the workspace gets (default: `data-science`). +* `ownerSubject` makes that Keycloak user an administrator of the workspaces in the VM. +* `vaultPrefix` points the workspace at its own API keys. Load them into {hashicorp-vault-short} under that prefix, as the commented example in `values-secret.yaml.template` shows. By default all workspaces share the keys under `secret/data/hub`. + +The pattern does not create Keycloak users. Create each user in the `openshell` realm of the Keycloak instance in the `saw-keycloak` namespace, and give them the `openshell-user` role. + +Removing an entry deletes that user's Argo CD applications but keeps the VM and the namespace. To delete them as well, first set `pruneOnRemove: true` on the entry and push, then remove the entry and push. + +[id="changing-profiles-saw"] +=== Changing what a workspace contains + +SAW-BOM profiles are directories in `charts/saw-bom/profiles///`, one per OpenShell workspace, with three files: + +`workspace.yaml`:: +The workspace name and description. + +`providers.yaml`:: +The providers: a type from the governance catalog, the Secret and key that hold the API key, and for model providers the model name. + +`sandbox.yaml`:: +The sandboxes: name, type (`openclaw`, `nemoclaw`, or `generic`), container image, and the providers to attach. + +To add a profile, copy `data-science`, change it, and list it under `profiles` for the users that need it. A workspace VM applies profile changes on its next restart: + +[source,terminal] +---- +$ make openshell-saw-restart OPENSHELL_SAW_NAME=alice +---- + +[id="changing-model-provider-saw"] +=== Changing the model provider + +The default profile uses NVIDIA Nemotron on build.nvidia.com. To use another provider, change the `inference` secret in your `values-secret` file (the `provider`, `model`, and `api_key` fields) and a matching provider in the profile. Then load the secrets again: + +[source,terminal] +---- +$ ./pattern.sh make load-secrets +---- + +To use an alternative model server on your cluster, such as vLLM or Ollama, give users the `custom-inference` profile and complete the details of the `inference` example in `values-secret.yaml.template`. This section is commented out by default. You must specify `provider: openai`, and must provide valid values for the `model`, `url`, and `api_key` parameters. The URL must be reachable from the workspace VMs, for example a cluster Service. It cannot be `localhost`. + +[IMPORTANT] +==== +Agents call the model endpoint directly, and the sandbox's egress proxy allows only the hosts that the provider profile names. For a custom endpoint, set the endpoint's `host` and `port` in `charts/governance-policy/profiles/openai.yaml`. Without that change, requests to your server are denied. +==== + +[id="changing-governance-saw"] +=== Changing the governance policy and provider profiles + +The governance interceptor serves two things from the `governance-policy` chart: + +`policy.yaml`:: +The sandbox policy: filesystem paths that are read-only or writable, the user that agents run as, and Landlock settings. It applies to every sandbox. + +`profiles/.yaml`:: +The provider profiles. You must define a provider profile in this directory before you can create a provider of that type. You cannot create providers whose type has no profile here. Each profile lists the credentials, the `endpoints` (hosts, ports, and protocols) that sandboxes can reach with it, and the `binaries` that can connect, for example `node` for OpenClaw or `curl`. + +For example, to let agents use GitHub, keep the `github` profile and add a provider of type `github` to a profile. To allow a new API, add a profile file with its hosts and binaries. + +[NOTE] +==== +A profile that does not list `binaries` blocks all connections. List the real path of each program: the egress proxy resolves symbolic links before it matches a program, and you can use wildcards. For example, in the default image `/usr/bin/node` is a link to `/usr/bin/node-`, so for OpenClaw the profile must list `/usr/bin/node-*`. +==== + +Changes take effect when {rh-gitops} syncs the chart. Profile changes affect running sandboxes without restarting the sandbox. + +[id="upgrading-openshell-saw"] +=== Upgrading OpenShell + +The OpenShell release that the VMs run is pinned by image digest in the InstallerBOM, the `bom` block in `charts/openshell-saw/values.yaml`. To upgrade OpenShell, complete the following steps: + +. Change the component versions and digests in the `bom` block. All components (gateway, CLI, supervisor, and sandbox runtime) must come from the same release. +. Build the governance interceptor from the same OpenShell release. It is set as `source.git.ref` in `image-builder-charts/helm/governance-interceptor-image/values.yaml`, and the chart's image tag must match. +. Push, and restart the VMs. Each VM installs the new release on boot. + +[WARNING] +==== +Updating to a new release series, for example from 0.0.x to 0.1.x, causes the gateway to lose its state, and the data in the `/sandbox` directory of each sandbox is lost. This occurs because the OpenShell installer moves the old gateway database aside and creates fresh workspaces, providers, and sandboxes. After an update to a new release series, users also need to update to a compatible version of the `openshell` CLI. +==== + +[id="optional-apf-saw"] +=== Additional customization options + +* *Custom container images*: sandbox images are set per sandbox in the profile's `sandbox.yaml`. The VM pulls them from their registry. +* *Custom VM size*: `vm.cores`, `vm.memory`, and `vm.diskSize` in `charts/openshell-saw/values.yaml`, or per user under `values` in `overrides/saw-users.yaml`. +* *Signing*: the installer can require signed OpenShell images (`signing.mode: enforce` in `charts/openshell-saw/values.yaml`) after the images you pin are signed. diff --git a/content/patterns/secure-agent-workspace/troubleshooting.adoc b/content/patterns/secure-agent-workspace/troubleshooting.adoc new file mode 100644 index 000000000..2afd8e40f --- /dev/null +++ b/content/patterns/secure-agent-workspace/troubleshooting.adoc @@ -0,0 +1,201 @@ +--- +title: Troubleshooting +weight: 40 +aliases: /secure-agent-workspace/troubleshooting/ +--- + +:toc: +:imagesdir: /images +:_content-type: ASSEMBLY +include::modules/comm-attributes.adoc[] + +[id="troubleshooting-secure-agent-workspace"] +== Troubleshooting the Secure Agent Workspace pattern + +Use this page to diagnose common issues when you deploy or use this pattern. Most commands take the user's name as `OPENSHELL_SAW_NAME`; the examples use `alice`. + +[id="troubleshooting-vm-saw"] +=== The workspace VM does not start + +Symptom:: +`oc get vm -n saw-alice` shows `ErrorUnschedulable` or stays in `Scheduling`, or the {VirtProductName} Operator reports that no node supports virtualization. + +Cause:: +The worker nodes are not bare metal, or hardware virtualization is turned off in the firmware. {VirtProductName} cannot run the VMs on virtualized nodes. + +Resolution:: +Add bare-metal worker nodes (on public clouds, `*.metal` instance types) and check that the nodes are labeled as able to run VMs: ++ +[source,terminal] +---- +$ oc get nodes -l kubevirt.io/schedulable=true +---- ++ +For more information, see link:cluster-sizing[Cluster sizing]. + +[id="troubleshooting-installer-saw"] +=== The installer in the VM fails + +Symptom:: +`make openshell-saw-status OPENSHELL_SAW_NAME=alice` shows the `install` or `apply` step in a phase other than `Done`. The command prints each step with its phase and the first line of the error. + +Cause:: +Common causes are the following: ++ +* *A provider key is missing.* The workspace needs the `inference` and `web-search` Secrets in `saw-alice`. +* *The provider in the Secret does not match the profile.* The installer refuses an `inference` Secret whose `provider` does not match the profile's provider type. +* *An image cannot be pulled.* The VM pulls the OpenShell images from quay.io and the sandbox images from their registries. + +Resolution:: +Read the installer's log to find the cause: ++ +[source,terminal] +---- +$ make openshell-saw-logs OPENSHELL_SAW_NAME=alice +---- ++ +For a shell on the VM, run `make openshell-saw-vm-ssh OPENSHELL_SAW_NAME=alice`. It adds your SSH key to the VM when needed. ++ +* For a missing key, check that the ExternalSecrets synced (`oc get externalsecret -n saw-alice`) and that the keys are in {hashicorp-vault-short}. After you change your `values-secret` file, run `./pattern.sh make load-secrets`. +* For a provider mismatch, set the `provider` of the `inference` secret in your `values-secret` file to the provider type of the profile, and load the secrets again. +* For an image that cannot be pulled, check that the cluster can reach the registry. ++ +After you fix the cause, restart the VM to run the installer again: ++ +[source,terminal] +---- +$ make openshell-saw-restart OPENSHELL_SAW_NAME=alice +---- + +[id="troubleshooting-sandbox-saw"] +=== A sandbox stays in Error + +Symptom:: +`openshell sandbox list` shows a sandbox in `Error`, and its supervisor log reports `Startup configuration did not stabilize after 5 attempts`. + +Cause:: +With OpenShell 0.1.x, a provider profile that carries more than one annotation makes the gateway compute a different provider revision on every request (link:https://github.com/NVIDIA/OpenShell/issues/3929[NVIDIA/OpenShell#3929]). The governance interceptor that this pattern builds keeps only one annotation per profile. An interceptor built without that change, or from another OpenShell release, causes this error. + +Resolution:: +Check that the governance interceptor runs the image defined in the chart, and that the image and gateways are built using the same OpenShell release: ++ +[source,terminal] +---- +$ oc get deploy governance-interceptor -n openshell-agents -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +---- ++ +Then restart the VM. The installer recreates sandboxes that are in `Error`. + +[id="troubleshooting-interceptor-saw"] +=== Sandboxes and providers cannot be created + +Symptom:: +`openshell sandbox create` or `openshell provider create` fails, or the installer logs that the gateway has no profile for a provider type. + +Cause:: +The governance interceptor is unreachable or does not serve that profile. The gateways fail closed. + +Resolution:: +Check the interceptor pod and its log. The log lists the profiles it loaded: ++ +[source,terminal] +---- +$ oc get pods -n openshell-agents -l app.kubernetes.io/name=governance-interceptor +$ oc logs -n openshell-agents -l app.kubernetes.io/name=governance-interceptor --tail=20 +---- ++ +If the pod is in `ImagePullBackOff`, check that the interceptor image exists and that the cluster can pull it without credentials. A provider type that has no profile in `charts/governance-policy/profiles/` cannot be created. + +[id="troubleshooting-egress-saw"] +=== An agent cannot reach a model or tool API + +Symptom:: +The agent reports a network error or `403`, or the sandbox log shows a denied connection. + +Cause:: +Sandboxes can reach only the hosts that their attached providers' profiles list, and only from the programs that the profiles list. + +Resolution:: +Check that the provider is attached to the sandbox (`openshell sandbox provider list `). Check also that its profile lists the host and port and the real path of the program that connects, such as `/usr/bin/node-*` for OpenClaw. For a custom model endpoint, the `openai` profile must name the endpoint's host. See link:ideas-for-customization[Ideas for customization]. + +[id="troubleshooting-cli-version-saw"] +=== The openshell CLI reports a version mismatch + +Symptom:: +The `openshell` CLI cannot connect to the gateway. + +Cause:: +The CLI is from a different OpenShell release series than the gateway. The gateway that this pattern installs is from the 0.1.x series. + +Resolution:: +Check the CLI version with `openshell --version`, and install a 0.1.x CLI from the link:https://github.com/NVIDIA/OpenShell/releases[OpenShell releases]. + +[id="troubleshooting-cli-login-saw"] +=== The openshell CLI reports an expired sign-in + +Symptom:: +The `openshell` CLI reports an expired or inactive token. + +Cause:: +The Keycloak session that the CLI signed in with has ended. + +Resolution:: +Sign in again: ++ +[source,terminal] +---- +$ openshell gateway login alice +---- + +[id="troubleshooting-cli-tls-saw"] +=== The openshell CLI reports a TLS handshake error + +Symptom:: +The `openshell` CLI fails with a TLS handshake error. + +Cause:: +The gateway was registered with an old route. + +Resolution:: +Register the gateway again. The route is `alice-gateway` in `saw-alice`. ++ +[source,terminal] +---- +$ make openshell-saw-configure-gateway OPENSHELL_SAW_NAME=alice +---- + +[id="troubleshooting-webui-cache-saw"] +=== The web UI is blank after an upgrade + +Symptom:: +The OpenShell web UI shows a blank page after the pattern is upgraded. + +Cause:: +The browser is using cached files from the previous build. + +Resolution:: +Reload the page without the cache (for example, Shift+Reload), or clear the site data. + +[id="troubleshooting-webui-version-saw"] +=== The web UI reports that a workspace is not found + +Symptom:: +The OpenShell web UI shows `workspace ' ' not found`. + +Cause:: +The web UI build and the gateway release do not match. The OpenShell web UI and the gateway must use the same API version. + +Resolution:: +Upgrade the gateway, or pin a web UI image that matches it. + +[id="troubleshooting-argo-saw"] +=== Argo CD applications are not healthy + +Symptom:: +`./pattern.sh make argo-healthcheck` reports applications that are not `Synced` and `Healthy`. + +Cause:: +An application failed to sync or one of its resources is not ready. Each user has three applications, `saw--secrets`, `saw--bom`, and `saw-`. The shared ones are `openshift-cnv`, `openshell-keycloak`, `vault`, `openshift-external-secrets`, `governance-policy`, `governance-interceptor`, and `saw-users`. + +Resolution:: +Open the application in the Argo CD console and check the sync status and the resources that are not healthy. For a user's applications, also check the sections above about the VM and its installer. diff --git a/modules/secure-agent-workspace-about.adoc b/modules/secure-agent-workspace-about.adoc new file mode 100644 index 000000000..411c79899 --- /dev/null +++ b/modules/secure-agent-workspace-about.adoc @@ -0,0 +1,66 @@ +:_content-type: CONCEPT +:imagesdir: ../../images +include::comm-attributes.adoc[] + +[id="about-secure-agent-workspace"] += About the Secure Agent Workspace pattern + +The Secure Agent Workspace Validated Pattern creates isolated AI agent workspaces on {rh-ocp} for each user. Each workspace contains a dedicated virtual machine that runs NVIDIA OpenShell and its agent sandboxes. Workspaces are consistent and secure, with single sign-on (SSO), centrally governed sandbox policy, and API keys that never enter the agent environment. + +Use case:: + +* Run AI coding and knowledge agents, which execute code and call tools on a user's behalf, inside a boundary that belongs to one user only. +* Control what agents can reach: which hosts, which programs, and which credentials, from policy kept in Git. +* Onboard and remove users by editing one file in Git, with {rh-gitops} creating or deleting each user's workspace. + +Background:: + +This pattern implements the NVIDIA link:https://docs.nvidia.com/enterprise-reference-architectures/secure-agent-workspace-reference-design/latest/[Secure Agent Workspace reference design] on {rh-ocp} with {VirtProductName}, following its link:https://docs.nvidia.com/enterprise-reference-architectures/secure-agent-workspace-reference-design/latest/openshift-virtualization-reference-implementation.html[OpenShift Virtualization reference implementation]. Agents can run arbitrary code, so container isolation alone is not enough: each user gets a virtual machine, and inside it every agent runs in an OpenShell sandbox whose network access and credentials are enforced by a supervisor, not by the agent. This pattern generalizes one or more successful deployments of this use case. Implementation details might vary depending on your specific environment and requirements. + +[id="about-secure-agent-workspace-solution"] +== About the solution + +The {solution-name-upstream} framework installs the Operators and shared services, then creates one workspace per user listed in `overrides/saw-users.yaml`. Each workspace contains the following: + +* a namespace, `saw-` +* a {VirtProductName} VM named after the user +* routes for the user VM +* inputs to the VM installer + +Each VM installs a pinned OpenShell release on every boot from a versioned bill of materials (the _InstallerBOM_), then applies the user's _SAW-BOM profiles_: the OpenShell workspaces, model and tool providers, and agent sandboxes the user gets. The default `data-science` profile creates two sandboxes running the OpenClaw agent with the NVIDIA Nemotron model on build.nvidia.com, and Brave web search in the `default` workspace. + +Security comes in layers: + +* *VM isolation:* one VM per user, so no agent shares a kernel or process space with another user's agents. +* *Identity:* users sign in to Keycloak with OIDC. The OpenShell gateway in each VM accepts only tokens for its realm and maps Keycloak roles to OpenShell roles. +* *Governance:* every sandbox and provider change goes through a governance interceptor that serves a signed sandbox policy and the approved provider profiles. Policy changes are made in Git. +* *Credentials:* API keys are stored in {hashicorp-vault} and reach the user's namespace through the {eso}. Inside a sandbox, the agent receives only a placeholder. The sandbox's egress proxy substitutes the real key, but only for the hosts and programs that the provider profile allows. + +[id="about-secure-agent-workspace-technology"] +== About the technology + +This solution uses the following technologies: + +https://www.redhat.com/en/technologies/cloud-computing/openshift/try-it[{rh-ocp}]:: +An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. + +https://www.redhat.com/en/technologies/cloud-computing/openshift/virtualization[Red{nbsp}Hat {VirtProductName}]:: +Runs virtual machines alongside containers on {ocp}. This pattern runs one workspace VM per user, cloned from a template image with a known, stable configuration. {VirtProductName} requires bare-metal worker nodes. + +https://www.redhat.com/en/technologies/cloud-computing/openshift/gitops[{rh-gitops}]:: +A declarative application continuous delivery tool for Kubernetes based on the Argo CD project. Application definitions, configurations, and environments are declarative and version controlled in Git. + +https://access.redhat.com/products/red-hat-build-of-keycloak[Red{nbsp}Hat build of Keycloak]:: +Identity and access management based on Keycloak. The pattern creates an `openshell` realm with the roles and OIDC clients that the OpenShell CLI, gateway, and web UI use. + +https://www.vaultproject.io/[{hashicorp-vault}] and the {eso}:: +Store the model and tool API keys and the SSH key, and copy them into each user's namespace. + +https://github.com/NVIDIA/OpenShell[NVIDIA OpenShell]:: +The runtime for agent sandboxes: a gateway that manages workspaces, providers, and sandboxes, and a supervisor in each sandbox that enforces filesystem, process, and network policy. The pattern uses the Red{nbsp}Hat build of OpenShell (`quay.io/opendatahub/odh-openshell-*`). + +https://github.com/openclaw/openclaw[OpenClaw] and https://github.com/NVIDIA/NemoClaw[NVIDIA NemoClaw]:: +The AI agents that run in the sandboxes, with a terminal UI and a web UI. + +https://build.nvidia.com/[NVIDIA models on build.nvidia.com]:: +The default model provider (NVIDIA Nemotron). You can use any OpenAI-compatible endpoint on the cluster instead, such as vLLM or Ollama. diff --git a/modules/secure-agent-workspace-architecture.adoc b/modules/secure-agent-workspace-architecture.adoc new file mode 100644 index 000000000..df8f4585f --- /dev/null +++ b/modules/secure-agent-workspace-architecture.adoc @@ -0,0 +1,118 @@ +:_content-type: CONCEPT +:imagesdir: ../../images +include::comm-attributes.adoc[] + +[id="secure-agent-workspace-architecture"] += Secure Agent Workspace architecture + +The following figure shows the pattern on one cluster, with the workspace of one user, `alice`, in detail. + +.Secure Agent Workspace architecture +image::secure-agent-workspace/saw-architecture.png["An architectural and workflow diagram showing a workspace created for hypothetical user Alice using the Secure Agent Workspace Validated Pattern",link="/images/secure-agent-workspace/saw-architecture.png"] + +The following list describes the numbered flows shown in the _Secure Agent Workspace architecture_ figure: + +. {rh-gitops} syncs the pattern from your fork of the pattern repository. +. {rh-gitops} installs the shared services and, for each user in `overrides/saw-users.yaml`, a namespace, a VM, and the VM's inputs. +. Users sign in to Keycloak with OIDC. The OpenShell gateway in each VM accepts those tokens. +. The `openshell` CLI and the web UI reach the user's own gateway through the routes in the user's namespace. +. The gateway checks every sandbox and provider change with the governance interceptor. +. The {eso} copies the provider API keys from {hashicorp-vault} into the user's namespace. +. Sandbox traffic leaves through the sandbox supervisor's egress proxy. Only the hosts specified as allowed in the provider profiles are reachable, and the egress proxy adds the real API key to those requests. +. Every time the VM boots, it installs the OpenShell release that is pinned in its InstallerBOM. + +[IMPORTANT] +==== +{VirtProductName} runs virtual machines with hardware virtualization, so the worker nodes that run the workspace VMs must be *bare metal*: an on-premise bare-metal cluster, or bare-metal instance types in a public cloud (for example, `*.metal` instances on AWS). Nested virtualization is not supported. For more information, see link:cluster-sizing[Cluster sizing]. +==== + +[id="secure-agent-workspace-shared-services"] +== Shared services + +The pattern installs these once per cluster: + +{VirtProductName} (`openshift-cnv`):: +Runs the workspace VMs. The pattern creates each VM by cloning from a template image with a known, stable configuration. The template image is based on Fedora and includes podman and OpenShell service units. The pattern stores the template image as a Containerized Data Importer (CDI) `DataSource`. + +Red{nbsp}Hat build of Keycloak (`saw-keycloak`):: +Provides the `openshell` realm with the `openshell-user` and `openshell-admin` roles and the OIDC clients for the CLI (`openshell-cli`, device code and PKCE) and the web UI (`openshell-dashboard`). The realm includes test users (`alice`, `bob`, `developer`, `admin`). Replace these test users before you use the pattern in production. + +{hashicorp-vault} and the {eso} (`vault`, `external-secrets`):: +Store the model provider key, the web search key, and the SSH key that the pattern loads from your `values-secret` file. The {eso} copies them into each user's namespace. By default, all users share one set of keys, but you can also configure an individual Vault path for each user as needed. + +Governance interceptor (`openshell-agents`):: +A gRPC service that every OpenShell gateway calls before it creates a sandbox or a provider, or changes a sandbox's configuration. It serves the signed sandbox policy (filesystem, process, and Landlock rules) and the only provider profiles that users can use: `nvidia`, `openai`, `brave`, `github`, `slack`, `gemini`, and `web-search`. The policy and the profiles are files in the `governance-policy` chart, so changing them is a Git change. The interceptor fails closed: if it is unreachable, the gateways refuse those operations. + +[id="secure-agent-workspace-per-user"] +== Per-user workspace + +For each entry in `overrides/saw-users.yaml`, the `saw-users` chart creates a namespace `saw-` and three Argo CD applications: the user's secrets, the user's bill of materials, and the VM. + +Routes:: +`-gateway` exposes the OpenShell gateway (port 17670, TLS passthrough, gRPC) for the `openshell` CLI and the web UI. `-webui` exposes the OpenShell web UI through an OAuth2 proxy that signs in with Keycloak. + +Installer inputs:: +The user's provider Secrets (`inference`, `web-search`), the InstallerBOM, and the user's SAW-BOM profiles. They are attached to the VM as read-only disks, so the installer reads them without any access to the cluster API. + +VirtualMachine:: +A Fedora VM (4 vCPU, 8 GiB memory, and a 40 GiB disk by default). The VM runs the following tasks every time it boots: + +* `saw-install` pulls each OpenShell component (gateway, CLI, supervisor, and sandbox runtime) by digest, checks its version, starts the gateway, and, when the gateway moves to a new OpenShell release series, recreates the gateway state. +* `saw-apply` signs in to the gateway with its own mTLS identity and creates the workspaces, providers, and sandboxes of the user's profiles. It then onboards the OpenClaw agent in each agent sandbox against its model provider. + +Sandboxes:: +Each sandbox is a rootless podman container that the OpenShell gateway creates, with a supervisor that enforces the signed policy. The agent's environment holds only placeholders for API keys. The supervisor's egress proxy only adds the real API key to requests that are from allowed programs (for example `node` or `curl`) and to allowed hosts, which are defined in the sandbox's provider profiles lists. This means that when an agent calls a model or tool API, the supervisor's egress proxy can deny outbound traffic by withholding the real API key. All other outbound traffic is denied. + +[id="secure-agent-workspace-default-profile"] +== Default profile + +The `data-science` SAW-BOM profile, which every user gets by default, creates: + +[cols="1,1,1,2",options="header"] +|=== +| Workspace | Sandbox | Agent | Providers + +| `default` +| `notebook` +| OpenClaw +| `nvidia` (NVIDIA Nemotron on build.nvidia.com), `brave` (web search) + +| `cuda-dev` +| `cuda-sandbox` +| OpenClaw in the NemoClaw sandbox image +| `nvidia` +|=== + +The `custom-inference` profile uses an OpenAI-compatible endpoint of your own, such as vLLM or Ollama on the cluster, instead of build.nvidia.com. For more information, see link:ideas-for-customization[Ideas for customization]. + +[id="secure-agent-workspace-technologies"] +== Implementation technologies + +[cols="1,2",options="header"] +|=== +| Component | Technology + +| VM isolation +| {VirtProductName} (KubeVirt, CDI) + +| GitOps +| {solution-name-upstream} Operator, {rh-gitops} + +| Identity +| Red{nbsp}Hat build of Keycloak (OIDC) + +| Secrets +| {hashicorp-vault}, {eso} + +| Agent runtime +| NVIDIA OpenShell (gateway, supervisor), podman + +| Agents +| OpenClaw, NVIDIA NemoClaw sandbox image + +| Governance +| OpenShell governance interceptor (signed sandbox policy, provider profiles) + +| Models +| NVIDIA Nemotron on build.nvidia.com, or any OpenAI-compatible endpoint +|=== diff --git a/modules/secure-agent-workspace/metadata-secure-agent-workspace.adoc b/modules/secure-agent-workspace/metadata-secure-agent-workspace.adoc new file mode 100644 index 000000000..a9ebf70ef --- /dev/null +++ b/modules/secure-agent-workspace/metadata-secure-agent-workspace.adoc @@ -0,0 +1,17 @@ +// This file defines metadata attributes for the Secure Agent Workspace pattern. +// The pattern repository has no pattern-metadata.yaml yet, so these values are defined manually. +:metadata_version: 1.0 +:name: secure-agent-workspace +:pattern_version: 1.0 +:description: Give each user an isolated AI agent workspace in their own OpenShift Virtualization VM, with NVIDIA OpenShell, OIDC sign-in, and governed sandbox policy. +:display_name: Secure Agent Workspace +:repo_url: https://github.com/validatedpatterns-sandbox/secure-agent-workspace +:docs_repo_url: https://github.com/validatedpatterns/docs +:issues_url: https://github.com/validatedpatterns-sandbox/secure-agent-workspace/issues +:docs_url: https://validatedpatterns.io/patterns/secure-agent-workspace/ +:ci_url: https://validatedpatterns.io/ci/?pattern=secure-agent-workspace +:tier: sandbox +:owners: sauagarwa +:extra_features_hypershift_support: false +:extra_features_spoke_support: false +:external_requirements: Bare-metal worker nodes for OpenShift Virtualization; API keys for build.nvidia.com and Brave Search diff --git a/static/images/secure-agent-workspace/saw-architecture.png b/static/images/secure-agent-workspace/saw-architecture.png new file mode 100644 index 000000000..01e2c3114 Binary files /dev/null and b/static/images/secure-agent-workspace/saw-architecture.png differ diff --git a/static/images/secure-agent-workspace/saw-architecture.svg b/static/images/secure-agent-workspace/saw-architecture.svg new file mode 100644 index 000000000..76f236dc3 --- /dev/null +++ b/static/images/secure-agent-workspace/saw-architecture.svg @@ -0,0 +1,153 @@ + + + + + + + +Git repository +validatedpatterns-sandbox/ +secure-agent-workspace (fork): +values-*.yaml, charts/, +overrides/saw-users.yaml + +Workspace user +openshell CLI (0.1.x) +browser: OpenShell web UI +and OpenClaw UI +signs in with OIDC + +Container registries +quay.io/opendatahub (OpenShell) +quay.io/rh-ai-quickstart +ghcr.io (interceptor) + +Model provider +build.nvidia.com (NVIDIA) +or an OpenAI-compatible +endpoint (vLLM, Ollama) + +Web search API +Brave Search + +Red Hat OpenShift on bare metal +OpenShift Virtualization runs the VMs, so worker nodes must be bare metal + +Validated Patterns Operator + Red Hat OpenShift GitOps (Argo CD) +installs the operators and syncs every application from Git, including one namespace, VM and inputs per user + +saw-keycloak +Red Hat build of Keycloak: +realm openshell, users, +roles, OIDC clients + +vault, external-secrets +HashiCorp Vault and the +External Secrets Operator: +provider API keys, SSH key + +openshell-agents +Governance interceptor: +signed sandbox policy and +provider profiles; golden image + +openshift-cnv +OpenShift Virtualization +(KubeVirt, CDI): VMs, +disks, golden DataSource + + +saw-alice +one namespace per user + +Installer inputs +Secrets from Vault: +inference, web-search +ConfigMaps: BOM, +SAW-BOM profiles + +Routes +alice-gateway :17670 +(TLS passthrough) +alice-webui (edge) + +VM disk (PVC) +cloned from the +golden image + +VirtualMachine alice +Fedora, KubeVirt + +In-guest installer (every boot) +saw-install: OpenShell by digest from the +InstallerBOM +saw-apply: workspaces, providers and +sandboxes from the SAW-BOM profiles + +OpenShell gateway :17670 + web UI +OIDC tokens from Keycloak for users, +mTLS for the installer; podman driver; +every sandbox and provider change +goes through the governance interceptor +Sandboxes: one container each, created by the gateway + +workspace default +notebook +OpenClaw + +Agent (OpenClaw, node) +providers attached: nvidia, brave +API key in its env is a placeholder + +Supervisor + egress proxy +enforces the signed policy; allows only +the provider profiles' hosts and binaries, +and puts the real key in only there + +workspace cuda-dev +cuda-sandbox +NemoClaw image + +Agent (OpenClaw, node) +providers attached: nvidia +API key in its env is a placeholder + +Supervisor + egress proxy +enforces the signed policy; allows only +the provider profiles' hosts and binaries, +and puts the real key in only there + + +1 + + + +2 + + +3 + +OIDC sign-in + + +4 + + + +5 + +policy, profiles (gRPC) + + +6 + +secrets sync + + + + +7 + + +8 + \ No newline at end of file