Define the atomic-action benchmark measurement standard

[Yuan-Xinyi]

Description

Defines how atomic-skill benchmarks measure a skill, as a document only. No
benchmark code changes — the implementation follows in a separate change, so
the standard can be argued about before anything is built on it.

scripts/benchmark/atomic_action/BENCHMARK_STANDARD.md, +158, nothing else.

The problem

Per-skill benchmarks report one boolean per case. That cannot distinguish a
plan that was never produced, a robot that never reached the object, and a
skill that reached the object and missed the goal. They are different problems
with different owners.

What the standard says

Three dimensions. success_rate is specified here. robustness and
adaptability are success_rate recomputed over a perturbation set and a
generalization set, so they inherit every definition. robustness has its axes
listed and its magnitudes unfixed. adaptability now has a target as well: a
frozen, versioned held-out domain set that did not inform case selection,
tolerances or skill parameters, scored per domain and reported as
adaptability_coverage_rate, adaptability_mean, adaptability_worst,
adaptability_variance and adaptability_retention, with eligibility at 100 %
coverage of mandatory cases plus suite-owned thresholds on the worst domain and
on retention. The domains and the two threshold values land with the suite.

success_rate is a per-skill ordered list of stages. Each stage is a scene
measurement taken at the end of the segment where the skill stops acting on
what the stage is about. A stage is reached only when every earlier stage
passed, so a report localizes the failure instead of collapsing it:

stage_success_rate[i] = passed(i) / reached(i)      reached(i) = passed(i-1)
success_rate          = passed(last) / total

Closing stages count. A skill that lifts an object and then drops it did not
pick it up. All 15 built-in skills have stages defined, including the three
that have no benchmark module yet.

Two tolerance sets, split by binding contract — not per skill.
MoveEndEffector and MoveJoints bind only a motion endpoint and are scored
after the terminal settle with the drive converged; the other thirteen also
bind a grasp endpoint and are scored mid-trajectory, with the arm
contact-loaded. That is a physical difference, not a preference:

Set	Skills	Scored at	Position	Rotation
PRIMITIVE_*	motion endpoint only (2)	trajectory end + settle	3 mm	0.3°
TASK_*	motion and grasp endpoints (13)	end of the actuating segment	1 cm	5°

Every value is 1.8x the worst error measured for its set. Each pair describes
one physical error at a characteristic lever arm: 0.57 m (arm reach) and
0.115 m (handle, drawer front, grasped object). Tolerances are absolute, never
scaled by commanded magnitude, so cases stay comparable.

A skill that breaks and re-establishes contact gets a derived budget rather
than its own constant: n * TASK_POSITION_TOLERANCE_M for n contact phases.
HandOver has three, so 3 cm — it measures 11.2 cm and therefore fails, at the
placed stage, with grasped, transferred and handed_over all passing.
That is the decomposition doing its job, not a tolerance to relax.

Trajectory-level criteria are excluded on purpose

Whether a plan is finite, stays inside the joint limits, avoids collisions, or
is tracked faithfully by the drive belongs to
scripts/benchmark/motion_generation/, whose own design document labels that
layer L1 trajectory. Here those properties are diagnostic columns only. A
skill that opened the door is not failed because the drive lagged the plan.

max_tracking_error_rad is still reported per case, so a reader can judge
whether a scene measurement is trustworthy.

Tolerances are calibrated against physics, not against the planner

The shipped MoveJoints and MoveEndEffector criteria compute their error
from traj[:, -1] — the planner's own endpoint, with no physics. They report
0.000000 rad and 0.00002 m, which describes the solver, not the robot.

Replaying the same coverage cases in physics and reading the achieved state:

Skill	Case	From traj[:, -1]	After physical replay + settle
MoveJoints	all three	0.000000 rad	0.002944 rad (0.169°)
MoveEndEffector	near_center	0.000021 m	0.000642 m
	front_left	0.000014 m	0.001268 m
	front_right	0.000026 m	0.001327 m
	far_center	0.000064 m	0.001537 m

Hence 3 mm / 0.3°, not the 0.01 m / 1e-4 rad in the shipped benchmarks.

For TASK_*, the worst measured errors are Slide 0.0056 m and AxisAlign
0.0480 rad (2.75°), against a contact-loaded arm tracking error of 0.0532 rad
(3.05°). A task rotation tolerance below about 3° is unreachable in this
simulation at any replay rate, because the arm itself lags the plan by that
much while loaded.

Vocabulary

The failure reasons are the task-level subset of the taxonomy in
motion_generation/BENCHMARK_DESIGN.md section 7, used verbatim. stage and
peak signed are that document's words too. The two tolerance sets are named
after their constants rather than a new tier vocabulary, and
PHYSICAL_PICK_MIN_LIFT_M is the existing constant, reused.
TASK_POSITION_TOLERANCE_M replaces PHYSICAL_PLACE_XY_TOLERANCE_M (0.10) and
PHYSICAL_MOVE_HELD_OBJECT_XYZ_TOLERANCE_M (0.12).

Type of change

• Documentation

Screenshots

N/A

Checklist

• I have run the black . command to format the code base (no Python changed)
• I have made corresponding changes to the documentation (this is the documentation)
• Public API changes are reflected in the API docs (none)
• I have added tests that prove my feature works (no code; tests land with the implementation)
• Dependencies have been updated, if applicable (none)

Validation

Rebased onto current main, so the earlier revision's incidental diff is gone:
ten of its twelve non-document files were already identical to main and only
appeared because the branch predated #620. Two others would have removed
affordance_sampling support that landed on main after the branch point
(parse_affordance_sampling_arguments, create_affordance_sampling_context,
log_affordance_branch_diagnostics and their CLI). The branch is now two
commits adding a single file.

Calibration runs behind the PRIMITIVE_* numbers:

python -m scripts.benchmark.atomic_action.move_joints_benchmark --profile coverage --device cpu
python -m scripts.benchmark.atomic_action.move_end_effector_benchmark --profile coverage --device cpu

plus a physical-replay pass over the same cases at 16 steps per waypoint with a
terminal settle, which is what produced the right-hand column above.

Open

1. Twist's twisted stage gates on the executed end-effector rotation
   about the knob axis, which measures exactly 0.7854 rad for a 0.7854 rad
   command. The knob joint itself over-rotates to 2.38–3.14 rad for
   0.52–1.57 rad commanded, because the gripper wedges against the 5 cm knob
   and the joint offers no resistance; it is reported as a diagnostic. The
   alternative was to keep gating on the knob joint and let Twist fail on
   this asset.
2. Press's criterion only shows that the button moved. Commanding 3 mm
   still drives it to 5.66 mm of its 6 mm stroke, so the 80 % ratio does not
   discriminate commanded depth on this asset.
3. robustness magnitudes, and adaptability's domains and thresholds, are
   not set. For robustness the axes are listed and how much to perturb is
   not decided. adaptability is defined above, but which
   meshes, assets, layouts, end effectors and arms make up version 1 of the
   held-out set, and what adaptability_worst and adaptability_retention
   must reach, are owned by the suite and land with it.
4. PRIMITIVE_* rotation is calibrated at one terminal pose. All three
   MoveJoints coverage cases end at home, and all three settle to the same
   0.002944 rad — a systematic steady-state drive offset, reproducible but
   single-pose.
5. Single-sample estimates. Everything is --repeat 1.

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 3/5

The PR does not appear safe to merge while the standard can credit skill success despite invalid or unfaithfully executed motion and remains out of sync with the shipped benchmark behavior.

Findings

1.  Invalid Motion Can Pass ▶
2.  Standard Describes Unimplemented Behavior ▶
3.  Focused tests are missing ▶
4.  Missing segments are mis-scored ▶
5.  Timing mode breaks calibration ▶

Fix with agent prompt

### Issue 1
scripts/benchmark/atomic_action/BENCHMARK_STANDARD.md:3-6
The standard says motion validity and faithful tracking are diagnostic only and cannot fail a skill. That conflicts with the referenced motion-generation success ladder, where `task_success` requires both `motion_valid` and `execution_success`, and with the PR description's `max_tracking_error_rad <= 0.10` gate. Under this definition, an unsafe trajectory or motion the robot did not perform can still be credited as a successful skill.

### Issue 2
scripts/benchmark/atomic_action/BENCHMARK_STANDARD.md:58-67
The document presents staged rates, failure-stage metadata, replacement tolerances, 16-step replay, and `max_tracking_error_rad` reporting as current behavior. The shipped benchmarks still produce one success boolean with legacy reasons and tolerances, replay four physics steps per waypoint, and do not compute the promised tracking diagnostic. For example, Place still uses a 10 cm tolerance while this standard specifies 1 cm. As a result, existing reports cannot be reproduced or interpreted according to this standard.

### Issue 3
scripts/benchmark/atomic_action/common.py:1202-1205
The new trajectory validation, replay tracking, and success-ladder behavior has no focused test or in-repository caller. The reported smoke run exercises an unchanged benchmark path, leaving cases such as non-finite trajectories, limit violations, tracking failures, and ladder transitions unverified. This violates the repository directive that new features include focused tests proving their behavior, so the requirement must be satisfied before merging.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 4
scripts/benchmark/atomic_action/common.py:1529-1532
A missing or misspelled `actuation_segment` is silently replaced with the trajectory's final waypoint. For contact plans that release and retract, this scores the target after control has ended-the exact rebound condition this helper is intended to avoid. The benchmark can therefore report a misleading task result instead of identifying invalid segment metadata.

### Issue 5
scripts/benchmark/atomic_action/common.py:1345-1347
The advertised `waypoint_dt` mode replays at planner timing, while the new benchmark standard establishes that this cadence produces tracking errors above one radian and invalidates contact measurements. A caller using these documented parameters will trigger controller failures instead of using the calibrated 16-step replay. The mode should be removed or made consistent with the standard.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Defines a documentation-only measurement standard for atomic-action benchmarks:

• Establishes primitive and task tolerance sets.
• Decomposes skill success into ordered, scene-measured stages.
• Defines robustness and adaptability reporting, coverage, and eligibility.
• Documents replay configuration, diagnostics, and known calibration limitations.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[All benchmark cases] --> B{Earlier stages passed?}
  B -- No --> C[Record failure stage and reason]
  B -- Yes --> D[Measure current stage]
  D -- Fail --> C
  D -- Pass --> E{Final stage?}
  E -- No --> B
  E -- Yes --> F[Count case as skill success]
  F --> G[success_rate]
  G --> H[Nominal cases]
  G --> I[Perturbation set: robustness]
  G --> J[Held-out domains: adaptability]
  J --> K{100% mandatory coverage?}
  K -- No --> L[Report result as ineligible]
  K -- Yes --> M[Apply worst-domain and retention thresholds]

Loading

_{Reviews (6) · Last reviewed commit: "docs(bench): define a testable adaptabil..."}

[greptile-apps]

Focused tests are missing

The new trajectory validation, replay tracking, and success-ladder behavior has no focused test or in-repository caller. The reported smoke run exercises an unchanged benchmark path, leaving cases such as non-finite trajectories, limit violations, tracking failures, and ladder transitions unverified. This violates the repository directive that new features include focused tests proving their behavior, so the requirement must be satisfied before merging.

Context Used: AGENTS.md (source)

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/common.py
Line: 1202-1205

Comment:
**Focused tests are missing**

The new trajectory validation, replay tracking, and success-ladder behavior has no focused test or in-repository caller. The reported smoke run exercises an unchanged benchmark path, leaving cases such as non-finite trajectories, limit violations, tracking failures, and ladder transitions unverified. This violates the repository directive that new features include focused tests proving their behavior, so the requirement must be satisfied before merging.

**Context Used:** AGENTS.md ([source](https://github.com/dexforce/embodichain/blob/main/AGENTS.md))

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Missing segments are mis-scored

A missing or misspelled actuation_segment is silently replaced with the trajectory's final waypoint. For contact plans that release and retract, this scores the target after control has ended—the exact rebound condition this helper is intended to avoid. The benchmark can therefore report a misleading task result instead of identifying invalid segment metadata.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/common.py
Line: 1529-1532

Comment:
**Missing segments are mis-scored**

A missing or misspelled `actuation_segment` is silently replaced with the trajectory's final waypoint. For contact plans that release and retract, this scores the target after control has ended—the exact rebound condition this helper is intended to avoid. The benchmark can therefore report a misleading task result instead of identifying invalid segment metadata.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Timing mode breaks calibration

The advertised waypoint_dt mode replays at planner timing, while the new benchmark standard establishes that this cadence produces tracking errors above one radian and invalidates contact measurements. A caller using these documented parameters will trigger controller failures instead of using the calibrated 16-step replay. The mode should be removed or made consistent with the standard.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/common.py
Line: 1345-1347

Comment:
**Timing mode breaks calibration**

The advertised `waypoint_dt` mode replays at planner timing, while the new benchmark standard establishes that this cadence produces tracking errors above one radian and invalidates contact measurements. A caller using these documented parameters will trigger controller failures instead of using the calibrated 16-step replay. The mode should be removed or made consistent with the standard.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Invalid Motion Can Pass

The standard says motion validity and faithful tracking are diagnostic only and cannot fail a skill. That conflicts with the referenced motion-generation success ladder, where task_success requires both motion_valid and execution_success, and with the PR description's max_tracking_error_rad <= 0.10 gate. Under this definition, an unsafe trajectory or motion the robot did not perform can still be credited as a successful skill.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/BENCHMARK_STANDARD.md
Line: 3-6

Comment:
**Invalid Motion Can Pass**

The standard says motion validity and faithful tracking are diagnostic only and cannot fail a skill. That conflicts with the referenced motion-generation success ladder, where `task_success` requires both `motion_valid` and `execution_success`, and with the PR description's `max_tracking_error_rad <= 0.10` gate. Under this definition, an unsafe trajectory or motion the robot did not perform can still be credited as a successful skill.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Standard Describes Unimplemented Behavior

The document presents staged rates, failure-stage metadata, replacement tolerances, 16-step replay, and max_tracking_error_rad reporting as current behavior. The shipped benchmarks still produce one success boolean with legacy reasons and tolerances, replay four physics steps per waypoint, and do not compute the promised tracking diagnostic. For example, Place still uses a 10 cm tolerance while this standard specifies 1 cm. As a result, existing reports cannot be reproduced or interpreted according to this standard.

Prompt To Fix With AI

This is a comment left during a code review.
Path: scripts/benchmark/atomic_action/BENCHMARK_STANDARD.md
Line: 58-67

Comment:
**Standard Describes Unimplemented Behavior**

The document presents staged rates, failure-stage metadata, replacement tolerances, 16-step replay, and `max_tracking_error_rad` reporting as current behavior. The shipped benchmarks still produce one success boolean with legacy reasons and tolerances, replay four physics steps per waypoint, and do not compute the promised tracking diagnostic. For example, Place still uses a 10 cm tolerance while this standard specifies 1 cm. As a result, existing reports cannot be reproduced or interpreted according to this standard.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[yangchen73]

Looks great.

[yuecideng]

The adaptability section currently lists generalization axes, but it does not yet define a testable generalization target. In particular, it does not freeze a held-out domain set, state what skill semantics must remain invariant, or define coverage and pass thresholds.

A concise addition could be:

• Robustness varies conditions within the nominal domain; adaptability evaluates held-out objects, scenes, or embodiments outside the calibration domain.
• Adaptability measures the same skill contract on a frozen, versioned held-out domain set that was not used to select cases, tolerances, or skill parameters. Every case preserves the semantic goal and satisfies the binding contract.
• For each held-out domain g, recompute section 1 as success_rate[g] and report adaptability_coverage, macro adaptability_mean, adaptability_worst, adaptability_retention = adaptability_worst / nominal_success_rate, and variance across domains.
• A suite passes only when coverage is 100% and both the worst-domain score and retention meet suite-owned, versioned thresholds. Unsupported mandatory cases reduce coverage and cannot be silently excluded.

This keeps the standard compact while turning the current axis list into a comparable and reproducible generalization objective.

[Yuan-Xinyi]

@yuecideng Done in 5d84c56 — section 3 rewritten along your four points.

• The two dimensions are now separated explicitly. Section 2 states that
  every robustness axis stays inside the nominal domain (same assets, same
  scene, same embodiment; only their conditions vary). Section 3 names the
  calibration domain — the assets, cases and runs section 0 used to fix the
  tolerances and section 1 used to fix the stages — and evaluates outside it.
• Frozen, versioned held-out domain set. A domain is one held-out variation
  of the nominal setup (object mesh, articulated asset, scene layout, end
  effector, arm) carrying its own cases. Frozen means three things: the set did
  not inform case selection, tolerances or skill parameters; every case
  preserves the semantic goal and satisfies the binding contract, so section 1
  applies unchanged — same stages, same tolerance set, no per-domain
  recalibration; the set is versioned and the version is reported with the
  numbers, so results are comparable only within one version.
• Per-domain metrics. success_rate[g], adaptability_coverage_rate,
  adaptability_mean (equal weight per domain), adaptability_worst,
  adaptability_variance, adaptability_retention.
• Eligibility. 100 % coverage of mandatory cases, plus suite-owned
  versioned thresholds on the worst domain and on retention. Mean and variance
  are reported but do not decide eligibility, since a mean can hide one domain
  at zero — which is what the worst-domain threshold exists to catch.

Two deviations from your wording, both to avoid a second vocabulary for
something motion_generation/BENCHMARK_DESIGN.md section 5 already names, as
this document borrows its taxonomy verbatim:

1. adaptability_coverage → adaptability_coverage_rate, after the existing
   coverage_rate; "a suite passes" → eligible/eligibility, after "Formal
   track eligibility requires 100% coverage of mandatory cases" and
   eligible=false. Likewise measured rather than "scored", and
   applicable / N/A rather than "out of scope".
2. Unsupported mandatory cases reuse the existing required_capabilities rule
   instead of a new one: an unsupported case is recorded with its reason and
   left out of that domain's denominator, but it lowers
   adaptability_coverage_rate; a case that passed the capability check and
   then failed counts as a failure. The effect you asked for is unchanged —
   dropping a hard domain shows up as missing coverage, not as a higher
   minimum — and an ineligible result is still reported, together with the
   coverage rate that made it ineligible.

One addition on top of your text: when nominal_success_rate is 0, retention
divides by zero, so it is reported as N/A (not zero) and eligibility rests on
adaptability_worst alone. That state already fails section 1, so no
adaptability result can credit it.

Still deliberately unset, and now stated that way in Open item 3: which meshes,
assets, layouts, end effectors and arms make up version 1 of the held-out set,
and what the two thresholds must reach. Those are owned by the suite and land
with it.