Stretch 4 Grasping Demo
This repository contains a grasping demo for the Stretch 4 mobile manipulator from Hello Robot. Grasping is based on RGB and depth images from Stretch 4's gripper camera.
Perception uses the Molmo 2 Vision-Language Model (VLM) to output pixel coordinates for a target object described with text. The pixel coordinates are then used to prompt the Segment Anything Model 2 (SAM 2) to segment the target object, after which SAM 2 tracks and segments the target object over time.
A finite state machine (FSM) consisting of a sequence of visual servoing behaviors controls the robot's motions. The behaviors use the segmentation mask output by SAM 2, the depth image from the wrist-mounted camera, and estimates of the gripper's fingertip frames of reference to decide how to move the robot.
The FSM behaviors use three control modes provided by flying gripper control:
Mode 1: Gripper Frame Relative Motions
Mode 2: Gripper Frame Projected into the Base Frame Relative Motions
Mode 3: Direct Joint-Space Control of the Joints (Relative and Absolute Motions)
More details about the FSM and tunable parameters for the behaviors can be found in visual_servo_fsm_params.py.
Installation
Other Dependencies
Please clone and install the following dependencies in the virtual environment associated with this package:
stretch4_flying_gripper: https://github.com/hello-robot/stretch4_flying_gripperstretch4_compliant_gripper: https://github.com/hello-robot/stretch4_compliant_gripper
[!NOTE] Architecture Note: Both Molmo 2 and SAM 2.1 are loaded dynamically on the fly using the Hugging Face
transformerslibrary. You do not need to clone their respective repositories or manually download checkpoints.
Installation on Stretch 4
Install the standard package requirements on the robot as follows; no need for a specific torch installation.
1. Create a Virtual Environment
Navigate to the stretch4_grasping_demo directory and create a new Python virtual environment:
2. Install Standard Dependencies
Install the remaining required packages using the provided requirements.txt:
3. Install Local Repositories
Install the related Stretch 4 packages in editable mode:
Installation on Remote Desktop (Ubuntu 24.04 + RTX 5090)
Install both the standard package requirements and the visual servoing perception system packages on your desktop computer. The visual servoing perception system used by grasping_demo.py and recv_and_molmo_sam2_gripper_images.py requires a powerful GPU. These instructions are tailored for an NVIDIA GeForce RTX 5090 running on Ubuntu 24.04.
1. Create a Virtual Environment
Navigate to the stretch4_grasping_demo directory and create a new Python virtual environment:
2. Install PyTorch (CUDA 12.8)
The RTX 5090 (Blackwell architecture) requires at least CUDA 12.8. Ensure your .venv is activated, then run:
(Note: Once a stable release of PyTorch with built-in cu128 or higher support becomes available, you can replace this with the standard stable installation command.)
3. Install Standard Dependencies
Install the remaining required packages using the provided requirements.txt:
4. Install Local Repositories
Install the related Stretch 4 packages in editable mode:
Running the Code
Networking Configuration
This grasping demo relies on a high-bandwidth, low-latency connection between the robot and the remote computer. A dedicated, high-performance WiFi access point is recommended.
After you have ensured that the robot and the remote computer are connected and able to communicate with each other, you can proceed to edit the IP addresses in gripper_networking.py to match the robot's IP address and the remote computer's IP address.
This file should be edited on both the robot and the remote computer. The file is in the src/stretch4_compliant_gripper/ directory of the stretch4_compliant_gripper repository.
You can see the file on GitHub via the following link:
The two variable to change are at the top of the file, as shown in the following excerpt:
Once networking is configured, you can proceed to copy the gripper calibration files from the robot to the remote computer.
Sync the Gripper Calibration Files
On your remote computer, run the following script to copy the gripper calibration files from the robot to the desktop:
This tool automatically connects to the robot using the IP specified in gripper_networking.py, downloads the calibration files in a single scp command (minimizing password prompts), and extracts the correct HELLO_FLEET_ID directly from the downloaded data. It then automatically organizes the models into your desktop's local ~/stretch_user/<robot_id>/calibration_gripper directory, keeping both machines in sync without requiring you to manually specify the robot's ID.
Start Robot-Side Services
On the Stretch robot (in the stretch4_compliant_gripper repository directory), open two separate terminals and run:
Terminal 1 (Receive Commands):
Terminal 2 (Publish Images & States):
Test the Perception System (Desktop)
Prior to making the robot move via closed-loop control, you should first test the perception system. On your remote desktop computer, ensure your .venv is activated, then run the following script with an OBJECT_DESCRIPTION that describes the object you want the robot to grasp. Prior to running the script, the target object should be in view of the robot's gripper camera. This script will prompt Molmo 2, use the pixel coordinates it provides to prompt SAM 2, and then use SAM 2 to track and segment the target object over time. The results will be visualized in a window.
While the script is running, you can move the object around, occlude it, deform it, and otherwise manipulate it to test the robustness of the perception system.
A text description of the target object (OBJECT_DESCRIPTION) should be provided on the command line.
The object description text is used to prompt the Molmo 2 VLM. Examples of object descriptions that have been used successfully follow.
Example Object Descriptions:
"sunscreen"
"cleaning wipe container"
"coffee mug"
"white plastic cup"
"black rubber rocket with red nozzle"
"brown paper cup"
The VLM Prompt
The full Molmo 2 prompt is defined by the get_molmo_pointing_prompt(object_description) function in vlm_utils.py. Advanced users can edit this prompt to better match their application.
Run Visual Servoing (Desktop)
If the previous test of the perception system was successful at tracking and segmenting the target object at a high frame rate with low latency, you can proceed to run the visual servoing finite state machine (FSM) that commands the robot to move and attempts to grasp the object.
First, make sure that the perception system test code is no longer running.
Then, in a terminal on your remote desktop, activate the environment and run the visual servoing FSM. As with the perception test, a text description of the target object (OBJECT_DESCRIPTION) should be provided on the command line.
[WARNING] This visual servoing demo does not avoid obstacles. It is strictly driven by joint states and depth and RGB camera images from the wrist-mounted camera. Make sure that the scene is clear of obstacles before running this script. You should also be prepared to stop the code and run-stop the robot via the button on the robot's head.
Optional Arguments:
--move_to_front: By default, the robot will approach the object directly. Passing this flag enables an exploration phase where the robot moves sideways to find the front face of the object (minimizing the apparent width) before approaching.
Testing Manual Gripper Control (Optional)
To test the gripper control manually, you can plug a gamepad dongle into the desktop and run the send_gripper_commands.py script from the installed model repository:
Last updated
Was this helpful?