feat(eval): grouped help and command examples for the eval commands

[jariy17]

I have decided to implement help groups using the Commander helpGroup feature. As shown below, organizing flags into groups makes it easier for users to understand how each flag functions within the command.

agentcore eval batch-evaluation evaluate

Before (refactor):

Usage: agentcore eval batch-evaluation evaluate [options]

evaluate existing sessions service-side (async; returns a job ID)

Options:
  --agent <agent>                            source: harness ID or Runtime ID whose sessions to use
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --online-eval <online-eval>                source: use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  source: raw DataSourceConfig JSON (inline, file://<path>, or -); escape hatch
  --start-time <start-time>                  time filter: window start (ISO-8601, with --end-time)
  --end-time <end-time>                      time filter: window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             filter: specific session IDs (only with --agent)
  --evaluators <evaluators...>               evaluator ID(s) to apply
  --ground-truth <ground-truth>              session ground truth (JSON SessionMetadataShape[]; inline, file://<path>, or -)
  --name <name>                              batch evaluation name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt evaluation data at rest
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

After (this PR):

Usage: agentcore eval batch-evaluation evaluate [options]

evaluate existing sessions service-side (async; returns a job ID)

Configuration:
  --name <name>                              batch evaluation name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt evaluation data at rest

Session source (choose exactly one):
  --agent <agent>                            harness ID or Runtime ID whose sessions to use
  --online-eval <online-eval>                use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  the traces to read (JSON DataSourceConfig); escape hatch

Source filters:
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --start-time <start-time>                  window start (ISO-8601, with --end-time)
  --end-time <end-time>                      window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             specific session IDs (only with --agent)

Evaluation:
  --evaluators <evaluators...>               evaluator ID(s) to apply
  --ground-truth <ground-truth>              expected answers for the sessions (JSON SessionMetadataShape[])

Other options:
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Where sessions and traces are read from, for sources the --agent and
      --online-eval convenience flags cannot express. Only top-level key:
      cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "logGroupNames": ["string", ...],  // [required] groups holding the traces
            "serviceNames": ["string", ...],   // e.g. "my_agent.DEFAULT"
            "filterConfig": {
              "sessionIds": ["string", ...],
              "sessionFilterConfig": {
                "startTime": "timestamp",
                "endTime": "timestamp"
              }
            }
          }
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/support_agent-AbC123XyZ9-DEFAULT"],"serviceNames":["support_agent.DEFAULT"],"filterConfig":{"sessionIds":["session-123"]}}}'

  --ground-truth (JSON: list of objects)
      Expected answers for the sessions being evaluated, so an evaluator can score a
      response against a reference instead of judging it on its own. Each entry names
      one session; omit an entry for a session that has no reference answer.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "sessionId": "string",         // [required] the session the reference applies to
            "testScenarioId": "string",    // groups sessions replaying the same scenario
            "groundTruth": {
              "inline": "string"           // the expected answer
            }
          },
          ...
        ]

      Example:
        --ground-truth '[{"sessionId":"session-123","groundTruth":{"inline":"The order shipped on Tuesday."}}]'

        --ground-truth file://ground-truth.json

agentcore eval batch-evaluation simulate

Before (refactor):

Usage: agentcore eval batch-evaluation simulate [options]

replay a dataset against a Runtime, then batch-evaluate the resulting sessions

Options:
  --runtime-id <runtime-id>                Runtime ID to invoke per scenario
  --qualifier <qualifier>                  Runtime endpoint qualifier (default DEFAULT)
  --payload-template <payload-template>    JSON payload template; {input} is the scenario input, e.g. {"prompt":"{input}"}
  --header <header...>                     an ordered application header (repeatable)
  --bearer-token <bearer-token>            CUSTOM_JWT bearer token (for JWT-auth Runtimes)
  --user-id <user-id>                      Runtime user ID
  --dataset <dataset>                      dataset source: local JSONL path or a dataset ID
  --dataset-version <dataset-version>      dataset version (with a dataset ID)
  --evaluators <evaluators...>             evaluator ID(s) to apply
  --name <name>                            batch evaluation name (unique in the account)
  --description <description>              description for the batch evaluation
  --kms-key-arn <kms-key-arn>              KMS key to encrypt evaluation data at rest
  --ingestion-wait-ms <ingestion-wait-ms>  ms to wait for span ingestion before grading (default 180000; 0 to skip)
  -h, --help                               display help for command

Global Options:
  --region <region>                        AWS region
  --debug                                  debug logging (default: false)
  --json                                   JSON output (default: false)
  --endpoint-url <endpoint-url>            endpoint URL override

After (this PR):

Usage: agentcore eval batch-evaluation simulate [options]

replay a dataset against a Runtime, then batch-evaluate the resulting sessions

Runtime invocation:
  --runtime-id <runtime-id>                Runtime ID to invoke per scenario
  --qualifier <qualifier>                  Runtime endpoint qualifier (default DEFAULT)
  --payload-template <payload-template>    request body per example (JSON object); {input} is replaced with the input
  --header <header...>                     an ordered application header (repeatable)
  --bearer-token <bearer-token>            CUSTOM_JWT bearer token (for JWT-auth Runtimes)
  --user-id <user-id>                      Runtime user ID

Dataset:
  --dataset <dataset>                      dataset source: local JSONL path or a dataset ID
  --dataset-version <dataset-version>      dataset version (with a dataset ID)
  --ingestion-wait-ms <ingestion-wait-ms>  ms to wait for span ingestion before grading (default 180000; 0 to skip)

Configuration:
  --name <name>                            batch evaluation name (unique in the account)
  --description <description>              description for the batch evaluation
  --kms-key-arn <kms-key-arn>              KMS key to encrypt evaluation data at rest

Evaluation:
  --evaluators <evaluators...>             evaluator ID(s) to apply

Other options:
  -h, --help                               display help for command

Global Options:
  --region <region>                        AWS region
  --debug                                  debug logging (default: false)
  --json                                   JSON output (default: false)
  --endpoint-url <endpoint-url>            endpoint URL override

Parameter details:

  --payload-template (JSON object)
      The request body sent to the Runtime for each dataset example. Every occurrence
      of {input} is replaced with that example's input, so the template describes the
      shape your agent expects and {input} marks where the prompt goes.

      Example:
        --payload-template '{"prompt":"{input}"}'

        --payload-template '{"messages":[{"role":"user","content":"{input}"}],"stream":false}'

agentcore eval batch-insights run

Before (refactor):

Usage: agentcore eval batch-insights run [options]

start an asynchronous batch insights run over existing sessions

Options:
  --agent <agent>                            source: harness ID or Runtime ID whose sessions to use
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --online-eval <online-eval>                source: use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  source: raw DataSourceConfig JSON (inline, file://<path>, or -); escape hatch
  --start-time <start-time>                  time filter: window start (ISO-8601, with --end-time)
  --end-time <end-time>                      time filter: window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             filter: specific session IDs (only with --agent)
  --insight <insight...>                     insight ID(s) to run (default: ["Builtin.Insight.FailureAnalysis"])
  --evaluators <evaluators...>               optional evaluator ID(s) to run alongside the insights
  --name <name>                              batch insights name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt insights data at rest
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

After (this PR):

Usage: agentcore eval batch-insights run [options]

start an asynchronous batch insights run over existing sessions

Configuration:
  --name <name>                              batch insights name (must be unique in the account)
  --description <description>                optional description
  --kms-key-arn <kms-key-arn>                KMS key to encrypt insights data at rest

Session source (choose exactly one):
  --agent <agent>                            harness ID or Runtime ID whose sessions to use
  --online-eval <online-eval>                use sessions an online-eval config already sampled
  --data-source-config <data-source-config>  the traces to read (JSON DataSourceConfig); escape hatch

Source filters:
  --endpoint <endpoint>                      Runtime endpoint qualifier (default DEFAULT; only with --agent)
  --start-time <start-time>                  window start (ISO-8601, with --end-time)
  --end-time <end-time>                      window end (ISO-8601, with --start-time)
  --session-ids <session-ids...>             specific session IDs (only with --agent)

Analysis:
  --insight <insight...>                     insight ID(s) to run (default: ["Builtin.Insight.FailureAnalysis"])
  --evaluators <evaluators...>               optional evaluator ID(s) to run alongside the insights

Other options:
  -h, --help                                 display help for command

Global Options:
  --region <region>                          AWS region
  --debug                                    debug logging (default: false)
  --json                                     JSON output (default: false)
  --endpoint-url <endpoint-url>              endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Where sessions and traces are read from, for sources the --agent and
      --online-eval convenience flags cannot express. Only top-level key:
      cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "logGroupNames": ["string", ...],  // [required] groups holding the traces
            "serviceNames": ["string", ...],   // e.g. "my_agent.DEFAULT"
            "filterConfig": {
              "sessionIds": ["string", ...],
              "sessionFilterConfig": {
                "startTime": "timestamp",
                "endTime": "timestamp"
              }
            }
          }
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/support_agent-AbC123XyZ9-DEFAULT"],"serviceNames":["support_agent.DEFAULT"],"filterConfig":{"sessionIds":["session-123"]}}}'

agentcore eval online-eval create

Before (refactor):

Usage: agentcore eval online-eval create [options]

create an online evaluation config

Options:
  --name <name>                                        the name of the online evaluation config
  --agent <agent>                                      harness ID or Runtime ID whose traffic to sample
  --endpoint <endpoint>                                the agent endpoint qualifier to scope monitoring to (default DEFAULT)
  --data-source-config <data-source-config>            the traces to evaluate (JSON DataSourceConfig; inline, file://<path>, or - for stdin), as an alternative to --agent
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440, default 15)
  --filters <filters>                                  trace filters (JSON Filter[]; inline, file://<path>, or - for stdin)
  --role-arn <role-arn>                                IAM role the online evaluation assumes (default auto-provisioned)
  --enable-on-create <enable-on-create>                whether to enable evaluation immediately (default true; pass false to create it paused)
  --description <description>                          a description of the config's monitoring purpose
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

After (this PR):

Usage: agentcore eval online-eval create [options]

create an online evaluation config

Configuration:
  --name <name>                                        the name of the online evaluation config
  --description <description>                          a description of the config's monitoring purpose
  --enable-on-create <enable-on-create>                whether to enable evaluation immediately (default true; pass false to create it paused)

Session source (choose exactly one):
  --agent <agent>                                      harness ID or Runtime ID whose traffic to sample
  --data-source-config <data-source-config>            the traces to sample (JSON DataSourceConfig), as an alternative to --agent
  --endpoint <endpoint>                                the agent endpoint qualifier to scope monitoring to (default DEFAULT)

Evaluation:
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440, default 15)
  --filters <filters>                                  trace filters (JSON Filter[])

Execution:
  --role-arn <role-arn>                                IAM role the online evaluation assumes (default auto-provisioned)

Other options:
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Which traces are sampled, for sources the --agent convenience flag cannot
      express. Only top-level key: cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "serviceNames": ["string", ...],          // [required] e.g. "my_agent.DEFAULT"
            "logGroupNames": ["string", ...],         // exact group names
            "logGroupNamePrefixes": ["string", ...]   // or match by prefix instead
          }                                           // supply one of the two name lists
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore-control/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/my-runtime-DEFAULT"],"serviceNames":["my_agent.DEFAULT"]}}'

  --filters (JSON: list of objects)
      Narrows which sampled traces are evaluated. A trace must match every filter in
      the list. Omit it to evaluate every sampled trace.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "key": "string",       // [required] trace field to filter on
            "operator": "Equals" | "NotEquals" | "Contains" | "NotContains"
                      | "GreaterThan" | "GreaterThanOrEqual"
                      | "LessThan" | "LessThanOrEqual",   // [required]
            "value": {                                    // [required] exactly one key
              "stringValue": "string",
              "doubleValue": number,
              "booleanValue": true | false
            }
          },
          ...
        ]

      Example:
        --filters '[{"key":"attributes.customer_tier","operator":"Equals","value":{"stringValue":"enterprise"}}]'

        --filters file://filters.json

agentcore eval online-eval update

Before (refactor):

Usage: agentcore eval online-eval update [options]

update an online evaluation config

Options:
  --id <id>                                            the ID of the online evaluation config to update
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440)
  --filters <filters>                                  trace filters (JSON Filter[]; inline, file://<path>, or - for stdin)
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply (replaces the existing list)
  --agent <agent>                                      repoint at a different harness ID or Runtime ID
  --endpoint <endpoint>                                re-scope monitoring to a different agent endpoint qualifier
  --clear-endpoint <clear-endpoint>                    reset the endpoint scope to the default qualifier (pass true)
  --data-source-config <data-source-config>            replace the traces to evaluate (JSON DataSourceConfig; inline, file://<path>, or - for stdin)
  --role-arn <role-arn>                                replace the IAM role the online evaluation assumes
  --update-role <update-role>                          whether to re-scope an auto-provisioned execution role when the data source changes (default true)
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

After (this PR):

Usage: agentcore eval online-eval update [options]

update an online evaluation config

Target:
  --id <id>                                            the ID of the online evaluation config to update

Session source:
  --agent <agent>                                      repoint at a different harness ID or Runtime ID
  --data-source-config <data-source-config>            replace the traces to sample (JSON DataSourceConfig)

Source filters:
  --endpoint <endpoint>                                re-scope monitoring to a different agent endpoint qualifier
  --clear-endpoint <clear-endpoint>                    reset the endpoint scope to the default qualifier (pass true)

Evaluation:
  --evaluators <evaluators...>                         the ID(s) of the evaluators to apply (replaces the existing list)
  --sampling-rate <sampling-rate>                      percentage of sessions to sample (0.01-100)
  --session-timeout-minutes <session-timeout-minutes>  minutes of inactivity before a session is considered complete (1-1440)
  --filters <filters>                                  replace the trace filters (JSON Filter[])

Execution:
  --role-arn <role-arn>                                replace the IAM role the online evaluation assumes
  --update-role <update-role>                          whether to re-scope an auto-provisioned execution role when the data source changes (default true)

Other options:
  -h, --help                                           display help for command

Global Options:
  --region <region>                                    AWS region
  --debug                                              debug logging (default: false)
  --json                                               JSON output (default: false)
  --endpoint-url <endpoint-url>                        endpoint URL override

Parameter details:

  --data-source-config (JSON: tagged union object)
      Which traces are sampled, for sources the --agent convenience flag cannot
      express. Only top-level key: cloudWatchLogs.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        {
          "cloudWatchLogs": {
            "serviceNames": ["string", ...],          // [required] e.g. "my_agent.DEFAULT"
            "logGroupNames": ["string", ...],         // exact group names
            "logGroupNamePrefixes": ["string", ...]   // or match by prefix instead
          }                                           // supply one of the two name lists
        }

      API reference:
        https://docs.aws.amazon.com/bedrock-agentcore-control/latest/APIReference/API_DataSourceConfig.html

      Example:
        --data-source-config '{"cloudWatchLogs":{"logGroupNames":["/aws/bedrock-agentcore/runtimes/my-runtime-DEFAULT"],"serviceNames":["my_agent.DEFAULT"]}}'

  --filters (JSON: list of objects)
      Narrows which sampled traces are evaluated. A trace must match every filter in
      the list. Omit it to evaluate every sampled trace.

      Accepts inline JSON, file://<path>, or - to read stdin.

      JSON syntax:
        [
          {
            "key": "string",       // [required] trace field to filter on
            "operator": "Equals" | "NotEquals" | "Contains" | "NotContains"
                      | "GreaterThan" | "GreaterThanOrEqual"
                      | "LessThan" | "LessThanOrEqual",   // [required]
            "value": {                                    // [required] exactly one key
              "stringValue": "string",
              "doubleValue": number,
              "booleanValue": true | false
            }
          },
          ...
        ]

      Example:
        --filters '[{"key":"attributes.customer_tier","operator":"Equals","value":{"stringValue":"enterprise"}}]'

        --filters file://filters.json

Tests

Router (6 new): heading names and ordering, ungrouped flags staying in Options:, -h placement with and without groups, examples rendering exactly once after Parameter details:, backslash continuation for array commands vs single-line for strings, and the no-examples case.

Handlers (14 new, src/handlers/eval/helpGroups.test.ts): asserts what the router's rendering tests cannot — that every flag on these five commands carries a group, that the group comes from the shared vocabulary, that a heading's flags are declared contiguously (a stray one would reorder the headings), and that no example names a flag its command doesn't declare.

Mutation-checked: ungrouping a flag, hand-typing a heading that drifts from the vocabulary, and misspelling a flag inside an example each fail a test.

Verification

• bun test — 3199 pass, 0 fail (225 files)
• bun run typecheck, bun run lint:check, bun run format:check — clean
• No existing test needed changing: the *.screen.test.tsx suites don't assert on these commands' option layout

Follow-ups

#2265 adds --output-config to batch-evaluation, renames --qualifier → --endpoint, and makes --start-time/--end-time require a timezone. #2266 does the online-eval equivalents plus the managed-role widening.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

Reviewed the router mechanism (flags.tsx, handler.tsx, router.tsx, index.tsx), the --evaluator → --evaluators rename across the 8 handlers, and the SDK bump.

• The Flag.group plumbing is minimal and correctly optional; ungrouped flags stay in Commander's default section.
• Handler.examples is optional and read off the authored node in compile(), so middleware wrappers (which only need to forward handle) don't have to carry it. withEffectiveTuiSupport also forwards it via handler.examples?.() so nothing gets dropped through TUI wrapping.
• The -h, --help relocation to Other options: correctly triggers only when ownFlags.some(f => f.group), keeping unchanged behavior for commands that don't opt in.
• New router tests cover: heading ordering, ungrouped flags remaining in Options:, -h placement in both cases, examples rendering exactly once after Parameter details:, and the no-examples case.
• The rename is complete: no leftover "--evaluator" argv, flag("evaluator", …), or flags["evaluator"] reads in src/, and the singular evaluator resource-type enum in project remove is correctly left alone (it's a resource selector, not a flag list). Error messages updated to '--evaluators <evaluators...>'. All corresponding fixture/unit tests updated in the same PR.
• SDK bump to 3.1129.0 on both agentcore clients is scoped and justified for the follow-up PRs.

No changes requested.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.06%. Comparing base (551ab96) to head (c8936e9).

Additional details and impacted files

@@            Coverage Diff             @@
##           refactor    #2262    +/-   ##
==========================================
  Coverage     97.05%   97.06%            
==========================================
  Files           566      568     +2     
  Lines         39228    39343   +115     
==========================================
+ Hits          38073    38188   +115     
  Misses         1155     1155            

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

remove this comment.

[jariy17]

remove code comments everywhere

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)