Manipulation

Motion planning and teleoperation for robotic manipulators. Drake remains the default world backend, RoboPlan is available as an optional planning backend, and manipulation visualization supports Meshcat or Viser.

Quick Start

Recent addition: the A-750 keyboard teleop blueprint is now available via:

dimos run keyboard-teleop-a750

Keyboard Teleop (single command)

Each blueprint launches the full stack — keyboard UI, mock controller, IK solver, and Drake visualization:

dimos run keyboard-teleop-a750    # A-750 6-DOF
dimos run keyboard-teleop-piper   # Piper 6-DOF
dimos run keyboard-teleop-xarm6   # XArm6 6-DOF
dimos run keyboard-teleop-xarm7   # XArm7 7-DOF

Open the Meshcat URL printed in the terminal (default http://localhost:7000) to see the robot.

Keyboard controls:

Key	Action
W/S	+X/-X (forward/back)
A/D	+Y/-Y (left/right)
Q/E	+Z/-Z (up/down)
R/F	+Roll/-Roll
T/G	+Pitch/-Pitch
Y/H	+Yaw/-Yaw
ESC	Quit

Motion Planning (two terminals)

# Terminal 1: Mock coordinator
dimos run coordinator-mock

# Terminal 2: Planner with Drake visualization
dimos run xarm7-planner-coordinator

Pink IK is the default solver. Tune it with nested module config overrides:

dimos run xarm7-planner-coordinator \
  -o manipulationmodule.kinematics.backend=pink \
  -o manipulationmodule.kinematics.max_iterations=100 \
  -o manipulationmodule.kinematics.dt=0.02

For blueprints that instantiate PickAndPlaceModule, use the corresponding module prefix:

dimos run xarm-perception-sim \
  -o pickandplacemodule.kinematics.backend=pink

Then use the IPython client:

python -m dimos.manipulation.planning.examples.manipulation_client

skip

joints()                # Get current joints
plan([0.1] * 7)         # Plan to target
preview()               # Preview in Meshcat
execute()               # Execute via coordinator

Planning backend selection

Manipulation planning separates the world backend from the planner algorithm:

• world_backend selects the robot/world/collision representation.
• planner_name selects the path-planning algorithm.
• kinematics.backend selects the IK backend. The legacy kinematics_name field remains available as a compatibility shim.

Drake remains the default:

dimos run xarm7-planner-coordinator

RoboPlan is available as an optional backend for evaluating a non-Drake world implementation. Select it explicitly with module options:

dimos run xarm7-planner-coordinator \
  -o manipulationmodule.world_backend=roboplan \
  -o manipulationmodule.planner_name=rrt_connect

Valid combinations:

world_backend	planner_name	kinematics.backend	Status
drake	rrt_connect	pink	Default path
drake	rrt_connect	jacobian	Legacy Jacobian IK
drake	rrt_connect	drake_optimization	Drake-only IK
roboplan	rrt_connect	pink or jacobian	Generic RRT over RoboPlan collision checks
roboplan	roboplan	pink or jacobian	RoboPlan-native planner, using the RoboPlan world object

Invalid combinations fail during startup instead of waiting for the first plan request. For example, planner_name=roboplan requires world_backend=roboplan, and kinematics.backend=drake_optimization requires world_backend=drake.

Install the manipulation dependencies:

uv sync --extra manipulation --inexact

The manipulation extra includes RoboPlan via roboplan from PyPI. The --inexact flag preserves other extras already installed in your current environment.

Safety behavior for unsupported RoboPlan features:

• Planning-critical unsupported inputs fail loudly before planning. Examples include unsupported obstacle geometry, unavailable robot loading APIs, or unavailable collision query APIs. RoboPlan worlds generate a minimal SRDF from the DimOS robot config, including configured collision-exclusion pairs.
• Unverified non-critical query methods raise explicit NotImplementedError. In particular, signed minimum-distance semantics are not implemented for RoboPlan until a safe equivalent is verified.
• Embedded Meshcat visualization requires a world implementing VisualizationSpec; use Viser or none with the RoboPlan backend.

Planning Visualization

Manipulation visualization is configured on ManipulationModuleConfig.visualization. It is independent from the global Rerun stream viewer in docs/usage/visualization.md.

Backend choices:

• meshcat: embedded Drake/Meshcat visualizer. The planning world must be created with embedded visualization enabled, so this is selected through the visualization config.
• viser: in-process Viser visualizer. It renders pushed current robot state, target controls, transient preview ghosts, synchronized trajectory previews, and optional panel controls.
• none: no manipulation planning visualization.

CLI example:

uv run dimos run xarm7-planner-coordinator \
  -o manipulationmodule.visualization.backend=viser

Blueprint example:

skip

from dimos.manipulation.manipulation_module import ManipulationModule, ManipulationModuleConfig

manipulation = ManipulationModule.blueprint(
    config=ManipulationModuleConfig(
        robots=[...],
        visualization={
            "backend": "viser",
            "host": "127.0.0.1",
            "port": 8095,
            "open_browser": True,
            "panel_enabled": True,  # default; set False for scene-only Viser
        },
    )
)

Viser support is included in the manipulation extra:

uv sync --extra manipulation --inexact

The Viser panel talks to the concrete ManipulationOperator bound into its VisualizationSession. GUI callbacks enqueue operations through that operator for target evaluation, planning, preview, execution, cancellation, reset, and clear-plan actions. The panel owns only target drafts, selection state, and callback generations; it does not touch WorldSpec, IK, planner objects, ManipulationModule, WorldMonitor, or live Drake contexts directly.

External manipulation visualizers are initialized from a backend-neutral VisualizationSession after the planning world has added its robots. The session contains static PlanningSceneInfo metadata: world robot IDs, RobotModelConfig values, and resolved planning groups. Runtime joint state is then pushed through VisualizationStateFrame updates so renderers do not poll world/module state or own freshness policy. Embedded Meshcat visualization does not need extra setup because it observes the Drake world directly.

Previews use the stored synchronized JointTrajectory from the generated plan. Viser projects the globally named trajectory into robot-local preview ghosts and plays the stored timestamped points directly; optional preview duration only scales the stored delays. Execute freshness is enforced by the manipulation module/operator immediately before dispatch, not by Viser-side telemetry snapshots.

Perception + Agent

# Coordinator + perception + manipulation + LLM agent (single command)
XARM7_IP=<ip> dimos run coordinator-xarm7 xarm-perception-agent

For a simulation walkthrough, see Agentic xArm simulation.

Architecture

KeyboardTeleopModule ──→ ControlCoordinator ──→ ManipulationModule
  (pygame UI)              (100Hz tick loop)      (WorldSpec backend)
       │                        │                       │
  TwistStamped           EEFTwistTask             RRT planner
  spatial EEF twist      (Pinocchio FK/IK)        JacobianIK
                               │                   DrakeWorld
                          JointState ────────────→ (visualization)

• KeyboardTeleopModule — Pygame UI publishing routed spatial EEF twist intent
• ControlCoordinator — 100Hz control loop with mock or real hardware adapters
• ManipulationModule — world backend, optional visualization, RRT motion planning, obstacle management

Internally, planning code depends on WorldSpec for world, collision, and kinematics behavior. Meshcat preview and publishing are exposed separately through VisualizationSpec, so non-visual planning paths do not require a visualization backend.

Blueprints

Blueprint	Description
keyboard-teleop-a750	A750 6-DOF keyboard teleop with Drake viz
keyboard-teleop-piper	Piper 6-DOF keyboard teleop with Drake viz
keyboard-teleop-xarm6	XArm6 6-DOF keyboard teleop with Drake viz
keyboard-teleop-xarm7	XArm7 7-DOF keyboard teleop with Drake viz
xarm6-planner-only	XArm6 standalone planner (no coordinator)
xarm7-planner-coordinator	XArm7 planner with coordinator integration
dual-xarm6-planner	Dual XArm6 planning
xarm-perception	XArm7 + RealSense camera for perception
xarm-perception-agent	XArm7 perception + LLM agent
xarm-perception-sim	XArm7 simulation perception stack
xarm-perception-sim-agent	XArm7 simulation perception stack + LLM agent

Supported Robots

Robot	DOF	Teleop	Planning	Perception
A-750	6	Y	Y	—
Piper	6	Y	Y	—
XArm6	6	Y	Y	—
XArm7	7	Y	Y	Y

Adding a Custom Arm

guide is here

Key Files

File	Description
manipulation_module.py	Main module (RPC interface, state machine)
robot/manipulators/common/blueprints.py	Shared coordinator, planner, and task helpers
robot/manipulators/a750/config.py	A-750 model and hardware config
robot/manipulators/a750/blueprints/teleop.py	A-750 keyboard teleop blueprint
robot/manipulators/piper/blueprints/basic.py	Piper coordinator blueprint
robot/manipulators/piper/blueprints/teleop.py	Piper teleop blueprints
robot/manipulators/xarm/blueprints/basic.py	XArm coordinator and planner blueprints
robot/manipulators/xarm/blueprints/perception.py	XArm perception blueprint
teleop/keyboard/keyboard_teleop_module.py	Keyboard teleop module
planning/world/drake_world.py	Drake physics backend
planning/planners/rrt_planner.py	RRT-Connect motion planner