You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add the diversity metrics and the start-position bias diagnostic - #93
share of the 5 cm voxels two episodes' grasp points visit that only one of them visits
approach_spread
angle between the directions two lifts of an object come from, in its frame, averaged over the lifted objects
behavior_diversity
RMS distance between two lifts' approach paths, averaged over the lifted objects, with each object's novelty against the seeds
start_spread
RMS distance between two episodes' object start layouts in the horizontal plane
exact_duplicates
share of episodes whose actions repeat an earlier episode's bit for bit
generation_bias
start-position cells whose success rate is significantly below the overall rate
The four spread metrics average over pairs of episodes or lifts, at most 2,000 drawn with a fixed seed, so they apply from two episodes on and do not grow with the dataset. The source column holds the same average over the seeds.
They follow each hand's grasp point, where the objects it carries sit in its frame at lift-off, so fingertip, flange and wrist frames compare alike. Hand and object poses come from obs/datagen_info, in one frame before each step. Generation now records the object poses with --record_metric_inputs; datasets generated without them read these three metrics as not computed.
generation_bias reads the start positions the attempt log already records. With a Bonferroni correction over nine cells, a cell that never succeeds is flagged from 10 attempts at a 50% success rate and from 56 at 10%; its details name the worst tested cell either way.
Episode checks run episode by episode, sharing one listing of each, and an episode that cannot be read fails its own checks. A 1,000-episode report, source included, takes 24 s on one core.
On a 50-demo Franka MimicGen run, two demos leave 80.3% of the space their hands visit unshared and approach a cube 27.9° apart, against 78.5% and 28.9° for two seeds; 1,000 demos read the same. The six-demo GR1 run varies less than its seeds on every metric.
Summary by CodeRabbit
New Features
Added metrics for demonstration diversity, including workspace coverage, approach and behavior variation, starting-position spread, and exact duplicates.
Added a metric that identifies potential generation bias across object starting positions.
Generated episodes now consistently include object poses alongside end-effector and target poses.
Bug Fixes
Improved metric reporting for unreadable episodes so issues can be isolated and reported more clearly.
Documentation
Documented the new diversity metrics and object-pose recording.
The diversity metrics each average, over pairs of episodes or lifts, how
much the two differ, so they do not grow with the number of episodes:
- workspace_coverage: the share of the 5 cm voxels two episodes' grasp
points visit that only one of them visits.
- approach_spread: the angle between the directions two lifts of an
object come from, in the object's frame.
- behavior_diversity: the distance between two lifts' approach paths in
the object's frame, with their novelty against the seed paths.
- start_spread: the distance between two episodes' object layouts.
exact_duplicates counts episodes whose actions repeat an earlier
episode's, and generation_bias counts start-position cells whose success
rate is significantly below the overall rate.
A hand's grasp point is where the objects it carries sit in its frame at
lift-off, so fingertip, flange and wrist frames compare alike. Object
frames read the quaternion order of the file's format version. The
episode checks run episode by episode, sharing one listing of each, which
brings a 1,000-episode report from 29 s to 20 s on one core.
Auto reviews are disabled on base/target branches other than the default branch.
Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.
You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.
Use the checkbox below for a quick retry:
Trigger review
Walkthrough
The change makes object poses part of per-step recorded data, adds episode evidence and report-handling updates, and introduces diversity and generation-bias metrics. Tests and documentation cover the new data handling and metric behavior.
Changes: Recorded Evidence and Dataset Metrics
Layer / File(s)
Summary
Record object poses and attempt positions autodata_interfaces/env/datagen_info_recorder.py, autodata_metrics/records.py, autodata_tests/interfaces/env/test_datagen_info_recorder.py, autodata_tests/metrics/test_records.py, docs/pages/concepts/data_generator.rst, scripts/generate_dataset.py
The pre-step recorder always includes object poses, and its configuration no longer has record_object_poses. Attempt records accept initial object positions. Related tests, documentation, and command help text reflect the recorded data.
Load episodes and evaluate reports autodata_metrics/evidence.py, autodata_metrics/report.py, autodata_tests/metrics/test_report.py
Episode listings can be cached within a context. Episode-open errors become InvalidRecord, and camera-path collection skips episodes whose listing fails. Report checks aggregate failures by episode and use the cached evidence baseline.
The package adds workspace coverage, approach spread, behavior diversity, start spread, and exact-duplicate metrics. Tests and documentation cover their calculations, and metric assertions account for pair-based metrics.
The viability module adds a binomial CDF helper and a generation-bias metric that tests sufficiently sampled start-position cells. A test checks the flagged-cell result.
sequenceDiagram
participant Dataset as Dataset file
participant Evidence as Evidence
participant Checks as run_checks
participant MetricChecks as Metric checks
participant Report as Report
Dataset->>Evidence: Load episode groups
Evidence->>Checks: Provide episodes
Checks->>MetricChecks: Evaluate applicable checks per episode
MetricChecks-->>Checks: Return result or NotApplicable
Checks->>Report: Store failures for applied checks
Loading
Possibly related PRs
isaac-sim/AutoData#84: Adds per-step pose data and initial object positions used by the metrics in this change.
isaac-sim/AutoData#92: Extends the same metrics evidence and report framework, including baseline evaluation.
isaac-sim/AutoData#90: Introduces the metric registry, evidence, and report evaluation extended by this change.
Merge Risk: Low · previous review b52df
The change adds diversity and generation-bias metrics and always records object poses. No merge-blocking defect was found. Optional diagnostics improvements remain for owner awareness.
Pre-merge checks: 3 · 1 · 1
Failed checks: 1 warning · 1 inconclusive
Check name
Status
Explanation
Resolution
Docstring Coverage
Docstring coverage is 65.91% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…
Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check
The description states the stacking dependency, lists each new metric with its meaning, and explains the record-metric-inputs requirement, the Bonferroni correction, the per-episode failure handling, …
Add a '## Summary' heading with a one-line description of 50 characters or fewer, for example 'Add diversity metrics and generation-bias diagnostic'. Add a '## Detailed description' heading and state explicitly the reason for the change (wh…
Passed checks: 3 passed
Check name
Status
Explanation
Title check
The title names the two main additions in the changeset: the diversity metrics in autodata_metrics/diversity.py and the generation_bias start-position diagnostic in autodata_metrics/viability.py. It i…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the two main additions in the changeset: the diversity metrics in autodata_metrics/diversity.py and the generation_bias start-position diagnostic in autodata_metrics/viability.py. It is specific and matches the PR objectives, and it does not describe anything absent from the changes.
Full details: Docstring Coverage
Explanation
Docstring coverage is 65.91% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 44 functions across 16 files. (2 skipped: 2 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
Full details: Description check
Explanation
The description states the stacking dependency, lists each new metric with its meaning, and explains the record-metric-inputs requirement, the Bonferroni correction, the per-episode failure handling, and the reported validation runs. It covers the substance of the template's detailed-description questions (reason, changes, impact). However, the template's Summary section is missing as a labeled heading with a short summary of 50 characters or fewer, and the description does not have the template's section structure. The reason for the change is implied (the metrics are the purpose of the PR) but not stated as an explicit motivation.
Resolution
Add a '## Summary' heading with a one-line description of 50 characters or fewer, for example 'Add diversity metrics and generation-bias diagnostic'. Add a '## Detailed description' heading and state explicitly the reason for the change (why these metrics are needed), what changed, and the impact on users, such as the new requirement to record object poses. Keep the existing content under that heading.
Finishing Touches
Generate docstrings
Commit to this branch
Create a new PR
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
…r objects
- The lift metrics read hand and object poses from obs/datagen_info, which
records both in one frame before each step, and find lift-off from the
world-frame states, which record each step's outcome. Generation records
the object poses with --record_metric_inputs.
- approach_spread and behavior_diversity average over the lifted objects;
the lowest object's average rises with the number of lifts.
- An approach is measured against the object's pose at each step, so a
robot frame that moves does not shift it.
- An episode that cannot be listed fails its checks instead of the report,
and objects without a start or root pose are left out.
- The source dataset is walked once per report.
- A lift goes to the hand, among those at most 15 cm farther from the
object than the nearest one at lift-off, in whose frame the object moves
least. An idle hand far from a slowly lifted object no longer takes it.
- workspace_coverage needs object poses and states, as its grasp points
come from lifts, so a dataset without them is not compared with a source
that has them.
- An episode that cannot be read fails its own checks only, and an episode
group that cannot be opened is refused by name.
- The help of --record_metric_inputs and the docs list the object poses
generation now records.
Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.
The reason will be displayed to describe this comment to others. Learn more.
Additional comments: 2
autodata_metrics/report.py-143-144: 1
143-144: Trivial · Maintainability & Code Quality · Low value
Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.
Keep the traceback when a check raises.
The handler turns the exception into "Type: message" and discards the traceback. The failure appears in the report, so it is not silent. A bug in check code, as opposed to damaged data, then shows only a one-line reason and is hard to debug.
Log the traceback once per failing check and episode. Keep the report reason as it is.
♻️ Proposed change
except Exception as exc: # A broken check fails its episodes visibly instead of sinking the report.
+ logger.warning("Check %s failed on episode %s", name, episode.name, exc_info=True)
reason = f"{type(exc).__name__}: {exc}"
Define logger = logging.getLogger(__name__) at module level.
Based on learnings: in batch processing, log each per-item exception with its full traceback and continue, so one failure does not abort the run.
Sources: Learnings, Linters/SAST tools
autodata_metrics/evidence.py-180-181: 1
180-181: Trivial · Maintainability & Code Quality · Low value
Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.
Log the swallowed exception in camera_paths.
The except Exception: continue block drops the error with no trace. The comment says the episode "fails its own checks". That holds only if an applicable check touches the unreadable episode, and the check might also run in a different way than camera_paths.
Record the error so a broken episode stays diagnosable. Ruff S112 and BLE001 flag this block too.
♻️ Proposed change
- except Exception: # An episode that cannot be read fails its own checks.- continue+ except Exception: # An episode that cannot be read fails its own checks.+ logger.warning("Episode %s cannot be listed for camera paths", episode.name, exc_info=True)+ continue
Define logger = logging.getLogger(__name__) at module level.
Based on learnings: in batch loops, catch and log each exception with a full traceback, continue with the remaining items, and do not swallow failures silently.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Stacked on #92; review it first.
Adds the diversity metrics and one diagnostic:
workspace_coverageapproach_spreadbehavior_diversitystart_spreadexact_duplicatesgeneration_biasobs/datagen_info, in one frame before each step. Generation now records the object poses with--record_metric_inputs; datasets generated without them read these three metrics as not computed.generation_biasreads the start positions the attempt log already records. With a Bonferroni correction over nine cells, a cell that never succeeds is flagged from 10 attempts at a 50% success rate and from 56 at 10%; its details name the worst tested cell either way.On a 50-demo Franka MimicGen run, two demos leave 80.3% of the space their hands visit unshared and approach a cube 27.9° apart, against 78.5% and 28.9° for two seeds; 1,000 demos read the same. The six-demo GR1 run varies less than its seeds on every metric.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation