Pick

Typical Workflow

Preferred (inspect, then execute)

predict_grasps(text_prompt="mug", max_grasps=5)

After calling predict_grasps:

• Check Rerun Detections / Segmentation to confirm the target object label and mask are correct.
• Check world/grasps/* to inspect the refreshed grasp poses.
• Choose the desired candidate id, then execute:

execute_grasp(grasp_id=0, lift_height=0.15)

The execute step performs the actual motion: open → approach → descend → close → lift.

Quick (one call, less controllable)

pick_object(object_label="mug", lift_height=0.15)

Use this only when you do not need to inspect detections and grasp candidates manually.

Failure Policy

• Do not call pick_object or execute_grasp unless the target has been positively detected in the current view, or predict_grasps has returned at least one candidate.
• If perception_detect(...) returns an empty list twice in a row for the same object, stop retrying prompt variations. Report that the target is not visible or not detected reliably in the current frame.
• If predict_grasps(...) returns zero candidates, stop and ask for a better view or a different prompt. Do not keep retrying blind grasp attempts.
• Do not use whole-scene fallback grasps as if they were target-specific grasps.

Grasp Selection Hints

• Default to grasp_id=0 (highest score).
• For tall or elongated objects, prefer candidates where approach_world[2] is strongly negative (more top-down approach).
• Avoid candidates with translation_world[2] near zero (too close to the table surface — likely to hit the table on descent).
• If the first attempt fails, retry with grasp_id=1 or increase approach_steps.

Key Parameters

Backend

Configured by the GRASP_BACKEND environment variable on the grasp node:

• open3d (default) — YOLO bbox + SAM2 segmentation + geometric grasp generation. This is the preferred interactive workflow because it updates both segmentation and grasp poses in Rerun.
• heuristic — geometric fallback with limited scene understanding. Avoid this backend when you need reliable target selection.

Known Limitations

• The heuristic backend always generates a top-down grasp; it will fail for objects that must be grasped from the side.
• Detection/segmentation overlays are only meaningful if the target is actually detected in the current camera view.
• The current demo assumes the robot/controller pairing configured in env_node.py. If you change robot or controller, revisit the action layout assumptions in the grasp executor.