Skip to content

Add planner-oriented atomic skill benchmark - #520

Draft
yuecideng wants to merge 1 commit into
mainfrom
feat/atomic-skill-benchmark
Draft

Add planner-oriented atomic skill benchmark#520
yuecideng wants to merge 1 commit into
mainfrom
feat/atomic-skill-benchmark

Conversation

@yuecideng

Copy link
Copy Markdown
Contributor

Description

This PR adds a shared planner benchmark framework for physics-backed Atomic Actions and implements the first Franka + PGI and cuRobo vertical slice proposed in #437.

The benchmark now:

  • routes Atomic Actions through an adapter-owned MotionGenerator with MotionPolicy(strategy="motion_gen", planner="curobo");
  • registers planners, robots, scenarios, atomic skills, and object kinds behind shared extension points;
  • freezes robot, object, grasp, difficulty, and independent IK evidence in case manifests;
  • separates planning, motion-validity, execution, and physical task success;
  • reports planning and execution efficiency, controller tracking, path quality, object lift, memory, and a complete leaderboard in exactly three tables;
  • provides a deterministic atomic_franka_pgi_curobo smoke suite covering MoveEndEffector and antipodal-grasp PickUp;
  • preserves the existing free-space benchmark through the same runner.

Current Atomic Task scope is deliberately limited to batch size 1, an empty external cuRobo collision world, and the two initial skills. Additional planners opt in by exposing a MotionGenerator and the atomic_action capability.

Dependencies: cuRobo and CUDA are required to execute the new suite. No dependency manifests are changed.

Refs #437

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (non-breaking change which improves an existing functionality)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (existing functionality will not work without user modification)
  • Documentation update

Screenshots

Not applicable. The feature produces JSON/JSONL artifacts and a Markdown benchmark report.

Validation

  • black --check --diff --color ./ — 671 files unchanged
  • pytest -q tests/benchmark/motion_generation — 30 passed, 1 skipped
  • python -m compileall -q scripts/benchmark/motion_generation tests/benchmark/motion_generation/test_atomic_task_benchmark.py
  • python -m scripts.benchmark.motion_generation.run_benchmark --suite atomic_franka_pgi_curobo --device cuda — MoveEndEffector and PickUp both reached 100% planning, motion, execution, and task success; PickUp lifted the object by approximately 0.1594 m

Checklist

  • I have run the black . command to format the code base.
  • I have made corresponding changes to the documentation
  • I have added tests that prove my feature works
  • Dependencies have been updated, if applicable. No dependency changes are required.

Add shared planner, robot, scenario, skill, and object extension points for physics-backed Atomic Action evaluation. Introduce a Franka + PGI cuRobo smoke suite with MoveEndEffector and PickUp, staged success metrics, execution timing, manifests, reports, documentation, and tests.
@yuecideng yuecideng added enhancement New feature or request motion gen Things related to motion generation for robot atomic action atomic action related functionality robot Module related to robot physics Things related to physics labels Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

atomic action atomic action related functionality enhancement New feature or request motion gen Things related to motion generation for robot physics Things related to physics robot Module related to robot

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant