# Stretch Docs
*** ## Overview Looking to learn about Stretch 4? You've come to the right place. {% hint style="info" %} ### Documentation for Stretch 3 | 2 | RE1 has moved! Stretch 3 DocsStretch 2 | RE1 Docs {% endhint %} ### Ask an Agent The Stretch Docs Agent (via GitBook) can answer questions about the documentation. ### Agent Endpoint Agents can connect to this endpoint to browse and query Stretch 4 documentation. Use it in any MCP-compatible client. \ `https://docs.hello-robot.com/~gitbook/mcp` {% tabs %} {% tab title="Claude Code CLI" icon="claude" %} Open a terminal and run this command to register your site as a user-scoped MCP server in Claude Code. \ `claude mcp add gitbook-documentation --scope user --transport http https://docs.hello-robot.com/~gitbook/mcp` \ Claude can call into the Hello Robot documentation server directly when answering questions or exploring your docs. {% endtab %} {% tab title="Codex CLI" icon="openai" %} Open a terminal and run this command to add your site as an MCP server in Codex. \ `codex mcp add gitbook-documentation --url https://docs.hello-robot.com/~gitbook/mcp`
Codex can keep your docs available alongside the rest of your local development tools. {% endtab %} {% tab title="Antigravity" icon="vscode" %} Antigravity handles remote MCP hosting using a global or workspace configuration file. 1. Inside **Antigravity**, open the command palette (`Cmd/Ctrl+Shift+P`), type `mcp`, and select **Antigravity: Manage MCP Servers**. 2. Click **View raw config** to open the `mcp_config.json` configuration file. 3. Paste the following configuration snippet inside your `mcpServers` object: ```json { "mcpServers": { "hello-robot-docs": { "serverUrl": "https://docs.hello-robot.com/~gitbook/mcp" } } } ``` > ⚠️ **Important:** Google Antigravity uses `serverUrl` as its configuration key, not `url` or `httpUrl` like other IDEs. If you change this key, the environment will throw an initialization error. 4. Save the file and return to the MCP server manager interface. Click **Refresh** to activate the tools. {% endtab %} {% endtabs %} ### Other Resources The Stretch 4 documentation is spread across numerous Git repositories. This Stretch Docs site provides an organized view of this information. In addition, you can learn more at these resources: Hello RobotStretch User ForumGitHub Repos {% hint style="info" %} **This site is still under development.** If you don't see what you need, email us at . {% endhint %}
*** ## Getting Started
Cover image

:leaf:

Day One GuideNew to Stretch? Start here!/spaces/A7odvp6Q3OcylK96Pt41/pages/r1iGUi4spGcVoyWWDtsq/files/JTYDEiLUcY7RODUspbQt

:head-side-gear:

Hardware GuideLearn about hardware specifications, use, and care./spaces/5az5ocOmcIzrlNNAT5eH/files/QCzjbqk9QZAi6RdI3yZ0

:terminal:

Safety GuideLearn best practices for safe operation/spaces/Fw3I6nfQuCkv1r6I5Ty9/files/poRtTgYXf39I6YUI86To
***
{% columns %} {% column width="50%" %}
{% endcolumn %} {% column width="50%" valign="middle" %} ## Working with Stretch Learn the basics — like keeping the robot charged, logging data, and contrlling the joints. {% endcolumn %} {% endcolumns %}
:robot: Getting Started
/spaces/A7odvp6Q3OcylK96Pt41/pages/r1iGUi4spGcVoyWWDtsqRecommended all new users start here with Stretch - safety info, basics, and gamepad teleoperation
/spaces/A7odvp6Q3OcylK96Pt41/pages/wjJQ1QtzOzqbXue00ramConnect to Stretch and set it up for successful development
/spaces/A7odvp6Q3OcylK96Pt41/pages/Fk71kMGZHgSNsZ2S3I2DExplore Stretch's joints and sensors
/spaces/A7odvp6Q3OcylK96Pt41/pages/Du7j6IwJt2HfhwwRZr0rMap a room and navigate autonomously using Nav2
/spaces/A7odvp6Q3OcylK96Pt41/pages/xy4SIlhuBH8U0KRKMN7xControl Stretch from a phone or PC
/spaces/A7odvp6Q3OcylK96Pt41/pages/xsI64TNWpqrT65iKwqM0Software development basics for Stretch
/spaces/A7odvp6Q3OcylK96Pt41/pages/KzaAzLfRjPWE13LQ2dMqGet some support, or share your work with the community
:gears: General Use
/spaces/A7odvp6Q3OcylK96Pt41/pages/J0NLvY6STlFIOesvIifsUnboxing your new Stretch
/spaces/A7odvp6Q3OcylK96Pt41/pages/y1tPSiwM7A9JRJpOGlerHow to keep your battery charged and healthy
/spaces/A7odvp6Q3OcylK96Pt41/pages/LApYW3VeICngbcdLKQw5How to re-pack your Stretch for return shipping or transporting in a car.
/spaces/A7odvp6Q3OcylK96Pt41/pages/Rsb21zltdXGCVGpA7aQTHow to develop on Stretch remotely
/spaces/A7odvp6Q3OcylK96Pt41/pages/HHaMKgOgLUj2I6U9N5PNConfiguring the Stretch LAN, etc
:camera: Sensor Basics
/spaces/A7odvp6Q3OcylK96Pt41/pages/qx6xw5qVWUMKDCqh19SgHow to access the RGB cameras
/spaces/A7odvp6Q3OcylK96Pt41/pages/GO8gRv4ebEZab8OWxMLaHow to access the dual 3D Lidars
/spaces/A7odvp6Q3OcylK96Pt41/pages/kkqX8KxZ2iM0b7gftFQLHow to access the base laser-line sensor array
:gear: Motion Basics
/spaces/A7odvp6Q3OcylK96Pt41/pages/JRnFNmlNA301A26Fso62How to program simple joint motions
/spaces/A7odvp6Q3OcylK96Pt41/pages/LevKhpFqocY4j2VQFEoPCommon CLI tools for joint motion
/spaces/A7odvp6Q3OcylK96Pt41/pages/ZU17sizHE5uEbIczKtAuWays to adjust the motion characteristics
:gamepad-modern: Teleoperation
/spaces/A7odvp6Q3OcylK96Pt41/pages/826wuxDKQ5m9HMLTZt3JTelop via the mobile phone controller
/spaces/A7odvp6Q3OcylK96Pt41/pages/YHLQYSelRBPe5Pmu0GoqTeleop via the gamepad controller
/spaces/A7odvp6Q3OcylK96Pt41/pages/0ZXgTaokHBYxLX5qzhFRTeleop via the puppet controller
:map: Navigation
/spaces/A7odvp6Q3OcylK96Pt41/pages/1WBZGXnGyedBumTw8Zo5Quick start mapping and navigation
/spaces/A7odvp6Q3OcylK96Pt41/pages/iwslVDpH8IEM7fipZYRZTutorial on working with the Nav2 system
:layer-plus: Common Tasks
/spaces/A7odvp6Q3OcylK96Pt41/pages/rfWX1lFJJGj9RKd7LZroHow to change the end-of-arm tool
/spaces/A7odvp6Q3OcylK96Pt41/pages/r8DR4vz5jm6cg55CLbdkHow to log status data from the server
/spaces/A7odvp6Q3OcylK96Pt41/pages/XIWgBhrbecu71LnWD9J5Understanding the base frames
/spaces/A7odvp6Q3OcylK96Pt41/pages/Zvl5ThEwcpuR5lAYTdNgHow to manage the robot model
/spaces/A7odvp6Q3OcylK96Pt41/pages/Ua7GaMDFoxXrkmT0qAWwWorking with ArUCo IDs
*** ## Demos and Tutorials We are still developing exciting new demos and tutorials for Stretch 4. Check back periodically for new documentation.
Cover image
Data Collection with Puppet TeleopLearn how to use the puppet tool to collect on-robot manipulation data/files/akl2IRfMEskRNrIrOaSW/spaces/A7odvp6Q3OcylK96Pt41/pages/0ZXgTaokHBYxLX5qzhFR
VLM Based GraspingLearn how to use a the Molmo2 Vision Language Model to grasp objects/files/nJG9rvqKeFNIYcpPYanw/spaces/9DmMukkLxY3zG5WEzi3S
Sense and Follow PeopleLearn how to do real-time human pose detection and use that to have Stretch follow people./files/hg7LJdffP9fgKypx9ann/spaces/VDRzQppzRi0RGMCHcT0O
Flying Gripper ControlLearn how to use the Jacobian to control the robot from the tool frame/files/FVSfbBYePrQBXSoZic31/spaces/HfMeDcxEWF9hyYdOhtvN
Hybrid Marker DemoLearn how to integrate the reflective ArUco markers into a real-time controller/files/q0S5NcyuUKe3A8d5Wjxz/spaces/i6YjBUfk6eBB9NOVUrrA
Navigation UniversityLearn the details of working with Nav2 and Stretch 4/files/lIToytEe8Tlpg5lyGkMH/spaces/A7odvp6Q3OcylK96Pt41/pages/iwslVDpH8IEM7fipZYRZ
***
{% columns %} {% column width="50%" %}
{% endcolumn %} {% column width="50%" valign="middle" %} ## Explore the Open Source Software Stack The Stretch 4 software is organized into a multi-layered stack designed to transition seamlessly from low-level hardware control to high-level autonomous behaviors. The [Hello Robot GitHub ](https://github.com/hello-robot)page provide an overview of the key repositories. Each repository is intended to be self-documenting through its internal README.md and other Markdown files. Explore the Code {% endcolumn %} {% endcolumns %} ***
{% columns %} {% column width="50%" %}

Join the community of 100s of developers

Join our the Stretch community to post questions, get help, and share resources with over 100s of like-minded developers. Join the Forum {% endcolumn %} {% column width="50%" valign="middle" %}
{% endcolumn %} {% endcolumns %} # Quick Start Guide
Cover image
Getting Started with Stretch/files/tWQnrZtwDTf4ob0Ownso/spaces/A7odvp6Q3OcylK96Pt41/pages/r1iGUi4spGcVoyWWDtsq
# Setting Up Stretch This documentation is currently in beta. Please proceed carefully, and if you run into issues or need additional information, reach out to Hello Robot directly for support. ### Connecting to Stretch - Tethered In order to work with Stretch's software, or to develop and test your own code, you'll first need to connect to the computer inside. The simplest way to do this is to directly connect a monitor, keyboard, and mouse to the robot using the exposed ports in the robot trunk.
Stretch 4 contains a mini PC (model: Asus NUC 15) that can be accessed using the ports as shown above. Connect a monitor to the HDMI port, and a mouse and keyboard (we like to use a wireless dongle) to any of the USB ports, and make sure the robot is powered on (power button illuminated green). The Ubuntu desktop environment should appear on the connected monitor. Stretch 4 is running the Ubuntu 24.04 operating system. The default user login credentials came in the box with the robot. By default, the robot is not configured to ask for your password on boot, but may ask for it later if the NUC goes to sleep.
{% columns %} {% column %}
robot trunk
{% endcolumn %} {% column %}
{% endcolumn %} {% endcolumns %} #### Setting up Wi-Fi One of the first things you'll probably want to do with Stretch is to connect it to the internet. While there is an Ethernet port in the trunk as well, its much more likely that you'll want to use Wi-Fi. A Wi-Fi connection will also enable you to use untethered connections to the robot in the future (more details below). This is as simple as opening the Wi-Fi menu at the top right, selecting your network from the list, and inputting your network password if necessary. We recommend selecting the option to auto-connect to the network, in order to make working with the robot easier in the future.
### Connecting to Stretch - Untethered Once your robot is on the network, you can access it remotely from another computer. This is the preferred way to work with Stretch for most users. There are multiple options for achieving this, including remote desktop software like RustDesk or TeamViewer, SSH connections, Tailscale, ROS2 networking, PyZMQ, etc. For the purposes of this tutorial, we will focus on Ubuntu Remote Desktop Protocol (RDP) over a local network. #### RDP with Remote Login Ubuntu 24.04 ships with a built-in Remote Login feature that allows you to access your Ubuntu User Account remotely using Remote Desktop Protocol (RDP). Multiple users can even log into the robot at the same time for development, if necessary. The official Ubuntu docs for setting up Remote Login are here: You do not need a HDMI Dummy Dongle to use Remote Login. Note that Remote Login is different from Remote Desktop, which shares the same menu in Ubuntu's Settings. Remote Desktop does require a Dummy Dongle and requires you to remain logged in. #### Robot setup instructions The following instructions will walk you through setting up Remote Login. {% hint style="info" %} Please follow these steps while you are on the hello-robot or default user account - setting up Remote Login on multiple accounts might cause conflicts if the username-password pair is the same. {% endhint %} 1. On your robot, navigate to Settings -> System -> Remote Desktop
2. Select the Remote Login tab, click Unlock and type in your password. Toggle to Enable Remote Login
3. At the bottom of the window, enter a username and strong password combination that will be shared by all users remotely connecting to this robot; it’s a good idea to NOT use the password for your User Account because anyone on the network can connect to the login screen after setting up Remote Login. This password will be shared by all users for remote access to the Login Screen (not a particular user account). 4. Retrieve the ip address of this robot using `hostname -I`. {% hint style="warning" %} It is important to follow your institution or organization's security best practices for setting up remote access and Ubuntu User Account Credentials. {% endhint %} #### Client Setup Instructions On the computer you want to connect to the robot, follow the below instructions based on your operating system: {% tabs %} {% tab title="Linux Client Setup" %} 1. Run `sudo apt install remmina` 2. Open Remmina and click the "+" icon. Enter your robot's IP, username and password you configured in the previous step. Click Save and Connect to test your connection 3. Click the Toggle Dynamic Resolution button on the sidebar to make the RDP window use your monitor’s resolution. 4. Login to your User Account to start using your robot. 5. To configure audio: on the client, right click the Remmina connection -> Edit -> go to the Advanced tab -> Audio output mode. Choose Remote to play audio through the robot’s speakers. Local to play it using the client speakers. Then click Save. {% endtab %} {% tab title="MacOS Client Setup" %} 1. Get the Windows App, offered by Microsoft Corporation, from the [Apple Store](https://apps.apple.com/us/app/windows-app/id1295203466) 2. Create a new Computer connection and change the PC name to the IP Address of the robot. You can add a Friendly Name to help identify the robot. Under Credentials, you can click to “Add Credentials” and save your password so you do not have to enter it every time you connect. > Note: if you encounter a blank screen error on MacOS, export your connection and change the line that says `use redirection server name:i:1`, and re-import your configuration, as suggested in: > {% endtab %} {% tab title="Windows Client Setup" %} 1. Open a Command Prompt window by pressing Win + R, then run `mstsc` 2. In the "Computer" field, type the IP address as retrieved from the robot above 3. Click "Connect". You may receive a certificate warning; click Yes to proceed. 4. When the login screen appears, enter your robot username and password {% endtab %} {% endtabs %} You are now connected remotely to Stretch! This can be a convenient way to use and develop on the robot. To learn more about different ways of connecting to Stretch, see the full guide here: [Connecting to Stretch](/stretch4_docs/working-with-stretch/general_use/connecting-to-stretch). ## Turning off Gamepad Teleoperation Out of the box, Stretch is configured to launch the gamepad teleoperation demo in the background at startup. While this is running, other code cannot use the robot. You will need to free the robot process so that your code can use it. Run the below command in the terminal to kill the server and stop the startup script. ```bash stretch_body_server --kill ``` You can also disable the autostart feature entirely. Search for "**Startup Applications**" from the Apps menu and uncheck the box for `hello_robot_xbox_teleop`.
In the next section, we'll learn about Stretch's joints and sensors and the basic commands for controlling them. # Robot Overview This documentation is currently in beta. Please proceed carefully, and if you run into issues or need additional information, reach out to Hello Robot directly for support. ## System Check Run a full hardware and software check to confirm the system is ready: ```bash stretch_system_check ``` If all checks pass, your robot is ready to go. Use `stretch_about` to print robot identity and configuration, and `stretch_params` to inspect all robot parameters. ## Homing If your robot needs to be homed, the system check will tell you. The homing procedure takes approximately 30 seconds and finds the zero position of all joints. It must be run once every time the robot is powered on: To home the robot: ```bash stretch_robot_home ``` \--# insert video of robot homing The robot will beep when finished. The motors will remember their homing state while the robot remains powered on — using the runstop, backdriving the robot, or restarting the NUC will not require rehoming. Powering the robot down completely will require homing again on the next boot. To stow the robot to its compact travel pose: ```bash stretch_robot_stow ``` \--# insert video of robot stowing ## Runstop The physical runstop button on the side of the robot's head immediately cuts power to all motors. The `stretch_runstop` tool lets you toggle the runstop programmatically from the terminal. \--# we already mentioned this in quick start should we point to it? ## Monitoring To monitor all robot devices, sensors, and activity in real time, including joint states, battery, and subsystem status: ```bash stretch_monitor_devices ``` This tool provides a unified live dashboard of the robot's state. \--# this tool exists in stretch\_production\_tools\_ii, worth it to port to stretch4\_body ## Motors and Joints \--# add videos for each ### Omnibase Stretch 4 has a triangular omnibase, consisting of three holonomic closed-loop stepper motors each driving one wheel. Wheel numbering increases counter-clockwise, with wheel 0 to the left of the forward direction. This is consistent with the ROS frame convention where X+ is forward and Y+ is to the left, forming a right-handed coordinate frame. The omnibase supports full planar motion — forward, sideways, diagonal, and rotation in place, all simultaneously. It also supports guarded contact sensitivity, which can be calibrated and tuned per use case. To jog the base: ```bash stretch_omni_base_jog ``` ### Lift The lift provides vertical translation of the arm, reaching up to 47 inches high and all the way down to the ground. It is driven by a closed-loop stepper motor through a low gear-ratio belt drive, providing smooth and precise motion. ```bash stretch_lift_jog ``` ### Arm The arm comprises 4 telescoping links set on rollers, extending 21.6 inches beyond the base footprint and retracting to stow within it. Its proprietary drivetrain is driven by a stepper motor with closed-loop control and current sensing, enabling contact detection during motion. In combination, the lift, arm, and mobile base provide three orthogonal axes of motion — a Cartesian system for end-effector placement. ```bash stretch_arm_jog ``` ### Contact Sensitivity (Guarded Contact) Stretch's lift, arm, and omnibase joints have a contact detection system called Guarded Contact. This safety feature limits the forces Stretch can apply to a person or its environment, and can be tuned for your application. Guarded Contact uses current sensing to detect when actuator effort exceeds a user-specified threshold during motion. When triggered, the safety controller halts the joint until a new movement command is received. ### Feetech Motors The wrist and gripper joints use Feetech servo motors, controlled over a TTL serial bus. To monitor Feetech motor status: ```bash stretch_feetech_monitor ``` #### Feetech Errors and Reset Feetech motors can enter an error state from over-force or over-temperature events. When in this state, the motor becomes backdrivable, stops responding to commands, and the LED on the motor body blinks red. Powering the robot down completely will clear this error. Note: rebooting only the NUC will NOT clear it, as the Feetech motors remain powered. A faster way to clear the error: ```bash stretch_feetech_reboot ``` This reboots all Feetech motors and resets their error status. You will need to re-run homing after doing this since the wrist\_yaw and gripper joints lose their homed positions. ## Head Cameras Stretch 4 has a Luxonis OAK-FFC-3P camera module mounted in the head with three cameras: | Camera | Type | Resolution | Target FPS | | ------ | --------------------- | ---------- | ---------- | | Left | Fish-eye wide angle | 1920×1200 | 30 | | Right | Fish-eye wide angle | 1920×1200 | 30 | | Center | High-resolution color | 4032×3040 | 10 | The left and right cameras are fish-eye wide-angle cameras providing a broad environmental field of view. The center camera is a high-resolution color camera. All three are angled to maximize combined visual coverage in every frame. To display any combination of camera feeds and open them in `rerun` or `opencv` : ```bash stretch_camera_show --left --rerun stretch_camera_show --center --opencv stretch_camera_show --right --rerun stretch_camera_show --left_right_center --opencv ``` {% hint style="info" %} 1. use either --rerun or --opencv (not both) in conjunction with the choice of camera combination. 2. use the flag -h to see all options {% endhint %} ### Emulated RGBD from Head Cameras The head cameras can be fused with the head Hesai lidars to produce RGBD (color + depth) point clouds. To visualize this: ```bash stretch_rgbd_show ``` ### Camera Focus and Calibration Camera focus is adjusted and intrinsics are calibrated using the factory calibration pipeline. After calibration, cross-calibration between cameras and lidars is also performed during the bringup process to align all sensor modalities spatially. ## Dexterous Wrist Stretch 4 has a three degree-of-freedom wrist with yaw, pitch, and roll actuation — all driven by Feetech actuators. | Axis | Raw Servo Range | Notes | | ----------- | --------------- | -------------------------------------------------------------------------- | | Wrist Yaw | 310° | `range_deg: [-65, 245]` | | Wrist Pitch | 310° | `range_deg: [-65, 245]` — effective range reduced by self-collision limits | | Wrist Roll | 310° | `range_deg: [-65, 245]` | To jog individual wrist joints: ```bash stretch_dex_wrist_jog ``` To home all wrist joints: ```bash stretch_dex_wrist_home ``` Individual axis homing is also available with `stretch_wrist_yaw_home`, `stretch_wrist_pitch_home`, and `stretch_wrist_roll_home`. ## Gripper The compliant gripper is a robust single-degree-of-freedom end-effector. A Feetech actuator drives the center of the spring mechanism, which causes the outer fingers to flex and provide a grasping force. ```bash stretch_gripper_jog stretch_gripper_home ``` ### Gripper Cameras Stretch 4 has two OAK-D-SR (Short Range) cameras mounted at the gripper, providing stereo depth for in-hand manipulation tasks. These are detected as a separate Luxonis device from the head cameras (2-sensor device vs. the 3-sensor head device). ```bash stretch_camera_show --gripper ``` ## Sensors Stretch 4 includes the following sensors: * Hesai JT128 3D LiDAR × 2 (Left and Right, mounted on the head) * Luxonis OAK-FFC-3P Camera Module (Head: 3-camera array) + OAK-D-SR (Gripper: stereo pair) * Speaker (mounted at the bottom of the head) * Pixart J3 Line Sensor Array (base, floor-facing) ### Hesai JT128 3D LiDAR (Head) Stretch 4 has two Hesai JT128 3D LiDAR units mounted on the head — a left lidar and a right lidar — providing full 3D point cloud coverage of the environment. These are used for mapping, navigation, and RGBD fusion with the head cameras. Each lidar communicates over Ethernet. The NUC holds a single network profile with two IP addresses to communicate with both lidars and the Jetson simultaneously:
DeviceIP AddressDescription
Left lidar192.168.1.202Hesai JT128 left
Right lidar192.168.1.201Hesai JT128 right
NUC (Lidar subnet)192.168.1.2Onboard computer — lidar communications
NUC (Jetson subnet)192.168.1.100Onboard computer — Jetson communications
Jetson192.168.1.101AI co-processor
### Speaker and Microphone The robot has a speaker and noise-cancelling microphone mounted at the bottom of the head, allowing Stretch to communicate from across a room. To test audio output: ```bash stretch_audio_test ``` ### Line Sensor Array Stretch 4 includes a floor-facing GreatScott GS2 line sensor array on the base. This array continuously scans the floor in front of the robot and uses an on-robot model to classify the surface as floor or obstacle, enabling low-latency hazard detection independent of the lidar. The line sensor runs in a dedicated background worker process at approximately 30 Hz, and its output is used by the omnibase to automatically limit velocity when an obstacle is detected in the direction of travel. To visualize the line sensor: ```bash stretch_line_sensor_viz_3d ``` ## LED Eyes and Lightbar The robot head has two LED eye displays. These support a set of built-in animations including idle glow, blinking, directional gaze, rainbow spin, alert, and happy states: ```bash stretch_eye_animations ``` The head also has a programmable RGB LED lightbar. The lightbar can be triggered and tested via `stretch_power_periph_jog`. ## NVIDIA Jetson (AI Co-processor) Stretch 4 includes an NVIDIA Jetson Orin module as a dedicated AI co-processor, directly connected to the NUC via Ethernet. ### Software Environment The Jetson runs with: * **OS:** Ubuntu 22.04 + JetPack 6.1 * **Custom Docker container** with: ROS 2 Jazzy, PyTorch, CUDA (cu129), Zenoh, Ultralytics, and OpenCV Bridge for NVIDIA Jetson ### Accessibility The NVIDIA Jetson boots up automatically whenever the robot is powered on. Passwordless SSH is pre-configured, so you can seamlessly access the Jetson from the NUC using the following command: ```bash ssh jetson1@192.168.1.101 ``` {% hint style="warning" %} **Note:** *If the Jetson fails to boot, you can manually power it on from the NUC using `stretch_power_periph_jog -d`. To forcefully power it off from the NUC, use `stretch_power_periph_jog -c`)* {% endhint %} ### Internet Connectivity To enable system updates and software package installations, there is a Wi-Fi dongle attached directly to the Jetson processor. You can connect the Jetson's Wi-Fi dongle to your local internet using the command line interface (CLI). Once logged into the Jetson via SSH, use the `nmcli` network manager to connect to your Wi-Fi network: ```bash sudo nmcli device wifi connect "" password "" ``` Once connected, you can verify internet access by pinging an external server: ```bash ping 8.8.8.8 ``` ### GPU Offloading The Jetson communicates with the NUC via Zenoh for ROS 2 topic bridging. This enables offloading compute-heavy tasks such as YOLO object detection or pose estimation to the Jetson's GPU, while the NUC handles robot control. The Jetson subscribes to image topics published by the NUC and returns inference results. ## Software Architecture Stretch 4 uses a client/server architecture in `stretch4_body`: * **`stretch_body_server`** runs a 100 Hz control loop on the NUC, managing all hardware state * Application code connects via `RobotClient` to issue commands * A C++ transport backend handles non-blocking USB communication to all motor controllers in parallel The 100 Hz control loop follows this sequence every tick: 1. Pull status from all devices 2. Update sentries (safety watchdogs) 3. Ingest commands from the Robot Client 4. Run active controllers and behaviors 5. Compute safe motion limits 6. Push safe commands to motor controllers To launch the body server: ```bash stretch_body_server --launch ``` ## Gamepad Teleoperation Stretch 4 ships with an Xbox controller and supports two control mappings: ### 1. Joint Space (default) Direct joint-level control of the robot. All controls are summarized below:
ControlAction
Base Controls
Left StickTranslate base (XY)
LB / RBRotate base
Hold LB + RB, Right StickAnalog base rotation
RT + Left StickStraight-line base movement
Arm Controls
Right StickWrist Pitch (Y) and Yaw (X)
D-Pad Up / DownLift up / down
D-Pad Left / RightArm retract / extend
A / B ButtonsClose / Open gripper
RT + LB / RBWrist Roll
Modifiers
LTPrecision mode (reduce speed)
### 2. Flying Gripper IK IK-based Cartesian control of the gripper. Point the gripper toward your target with the Right Stick, then move toward it with the Left Stick: | Control | Action | | ------------------ | ------------------------------------- | | Left Stick | Move toward target (translation) | | Right Stick | Point gripper at target (orientation) | | D-Pad Up / Down | Lift up / down | | D-Pad Left / Right | Wrist Roll | | A / B Buttons | Close / Open gripper | | LT | Precision mode (reduce speed) | ### Special Functions
InputAction
Y Button (tap)Cycle control mapping (Joint Space ↔ Flying Gripper IK)
RT + A (tap)Cycle motion speed profile (Slow / Medium / Fast)
RT + B (tap)Cycle contact sensitivity profile
Start Button (tap, unhomed)Home the robot
Start Button (tap, homed)Switch gripper handedness (without motion)
Start Button (hold 3s, homed)Switch gripper handedness (with motion)
RT + Select (tap)Announce current settings (handedness, speed, sensitivity, mapping)
RT + Select (hold 2s)Stow the robot
X Button (hold 0.5s)Execute custom function command
To launch gamepad teleoperation: ```bash stretch_gamepad_teleop ``` ## Developer I/O Stretch 4 contains additional ports connected to the onboard NUC that can be used for accessories: * **Trunk:** 1× USB-A 3.0 ports, 2x USB-A 2.0 ports, 1× Ethernet port, 1× HDMI port. * **Head (top):** 1× USB-A 2.0 port, 1x USC-C Port * **End-of-arm:** 1× USB-A 2.0 port * **Wrist:** Quick-connect mechanism for tool attachment. There are also threaded mounting points on the head to add additional sensors. ## Key Tools Reference
ToolDescription
stretch_body_serverStart / stop / release the 100 Hz robot control server
stretch_robot_homeHome all robot joints
stretch_robot_stowStow the robot to its compact travel pose
stretch_runstopToggle the runstop programmatically
stretch_system_checkFull hardware and software system check
stretch_aboutPrint robot identity and configuration
stretch_paramsPrint all robot parameters
stretch_battery_checkCheck battery state of charge
stretch_statusLive robot status display
stretch_joint_vizJoint state visualization
stretch_collision_vizVisualize self-collision safety margins
stretch_gamepad_teleopLaunch Xbox gamepad teleoperation
stretch_puppet_teleopLaunch puppet (backdriving) teleoperation
stretch_omni_base_jogJog the omnibase
stretch_lift_jogJog the lift
stretch_arm_jogJog the arm
stretch_arm_homeHome the arm
stretch_lift_homeHome the lift
stretch_gripper_jogJog the gripper
stretch_gripper_homeHome the gripper
stretch_dex_wrist_jogJog dex wrist joints
stretch_dex_wrist_homeHome all dex wrist joints
stretch_feetech_rebootReboot Feetech motors and clear errors
stretch_feetech_monitorMonitor Feetech motor status
stretch_camera_showLive camera feed viewer (configurable)
stretch_rgbd_showEmulated RGBD from head cameras + lidar
stretch_line_sensor_viz_3d3D visualization of line sensor data
stretch_power_periph_jogManually control power periph (Jetson, fans, lidar, etc.)
stretch_eoa_powerToggle end-of-arm power
stretch_eye_animationsRun LED eye animations
stretch_audio_testTest speaker output
stretch_pose_playPlay back recorded robot poses
stretch_pose_recordRecord robot poses
stretch_pose_editEdit recorded pose files
# Writing Code for Stretch This tutorial introduces the two primary ways to develop software with Stretch SE4 — Python and ROS 2 — and walks you through writing your first Python programs using the Stretch 4 Body API. By the end, you will be able to command every joint on the robot, read sensor status, control the omnibase holonomically, and use the client/server architecture. *** ### Background Stretch 4 supports two approaches to software development: **Python (Stretch 4 Body)** is the low-level direct interface to the robot hardware. It gives you fine-grained control over every joint and sensor, and is the fastest way to get started. The `stretch4_body` package is pre-installed on every Stretch SE4. **ROS 2 (Robot Operating System 2)** is a robotics middleware framework providing a collection of tools, libraries, and conventions for building robot applications. Stretch SE4 ships with ROS 2 Jazzy and a full driver stack. It is well suited for navigation, SLAM, MoveIt, and multi-node architectures. You can learn more about when to use each approach in the Developing with Stretch guide. This tutorial focuses on the Python API. ROS 2 examples are covered in the Demos section. *** ### Software Architecture All application code on Stretch SE4 uses the `RobotClient` class to communicate with a background `stretch_body_server` process running a 100 Hz control loop on the NUC. The 100 Hz control loop follows this sequence every tick: 1. Pulls status from all hardware devices 2. Updates safety sentries (watchdogs) 3. Ingests commands from the Robot Client 4. Runs active controllers and behaviors 5. Computes safe motion limits 6. Pushes safe commands to motor controllers Your code connects via `RobotClient`, which queues commands that the server loop ingests on the next tick. This means **motion is asynchronous by default** — your code keeps running while the robot moves. Use `push_command()` to flush the command queue and `wait_on_motion_finish()` to block until motion is done. *** ### Prerequisites Before running any code, the robot must be set up and ready. **1. Make sure the body server is running.** The body server is launched automatically at startup. Verify it is active: ```bash stretch_body_server --status ``` If it is not running, start it: ```bash stretch_body_server --daemon ``` **2. Free up robot control from the gamepad demo.** If the Xbox gamepad teleoperation demo is running at startup, your code cannot control the robot. Free the robot: ```bash stretch_body_server --free_up_control ``` **3. Confirm the system is healthy.** ```bash stretch_system_check ``` **4. Home the robot (required once per power cycle).** ```bash stretch_robot_home ``` The robot will beep when homing is complete. Joints will not accept motion commands until they are homed. Once homed, the motors remember their homed state as long as the robot remains powered on. *** ### Your First Program with RobotClient Open a terminal and launch iPython, an interactive Python console where each line runs immediately: ```bash ipython3 ``` Import and start the client: ```python from stretch4_body.robot.robot_client import RobotClient robot = RobotClient() robot.startup() ``` `startup()` connects to the running body server. If successful, you now have full access to all robot subsystems. Stow the robot to its compact travel pose: ```python robot.stow() ``` This is a blocking call — it returns only when the robot is fully stowed. Check if the robot is homed: ```python robot.is_homed() ``` When you are done with a session, always call: ```python robot.stop() ``` This cleanly disconnects from the server. **Skipping `stop()` can leave stale connections** that block other processes from controlling the robot. #### Using RobotClient as a Context Manager For scripts, the cleanest pattern is to use `RobotClient` as a context manager. This ensures `stop()` is always called, even if an exception occurs: ```python from stretch4_body.robot.robot_client import RobotClient with RobotClient() as robot: robot.stow() # ... your code ... # robot.stop() is called automatically here ``` *** ### Moving Individual Joints #### Arm The arm telescopes horizontally, extending up to \~0.52 m (21.6 inches) beyond the base. Position is in meters (0.0 = fully retracted, \~0.52 = fully extended). Move the arm to an absolute position: ```python robot.arm.move_to(0.25) robot.push_command() ``` Move the arm by a relative amount: ```python robot.arm.move_by(0.05) robot.push_command() ``` Move the arm back: ```python robot.arm.move_by(-0.05) robot.push_command() ``` #### Lift The lift translates the arm vertically, from floor level up to \~1.20 m (47 inches). Position is in meters. Move the lift to mid height: ```python robot.lift.move_to(0.6) robot.push_command() ``` Move the lift up a little: ```python robot.lift.move_by(0.1) robot.push_command() ``` #### Commanding Multiple Joints Simultaneously You can queue commands to several joints before pushing. They will execute simultaneously: ```python robot.lift.move_to(0.7) robot.arm.move_to(0.3) robot.push_command() ``` > \[!NOTE] If you queue two motion commands for the **same joint** before pushing, only the last one executes. Earlier commands are overwritten. #### Velocity and Acceleration Limits Every motion command accepts optional velocity and acceleration limits (in m/s and m/s² respectively): ```python robot.arm.move_to(0.2, v_m=0.05, a_m=0.1) robot.push_command() ``` If not specified, the robot uses its default motion profile. *** ### The Omnibase (Holonomic Base) Stretch SE4 has a triangular holonomic omnibase — three holonomic closed-loop stepper motors each driving one omnidirectional wheel. Unlike a differential drive, the omnibase can move **forward, sideways, diagonally, and rotate in place — all simultaneously**. Wheel numbering increases counter-clockwise, with wheel 0 to the left of the forward direction. This is consistent with the ROS convention where X+ is forward and Y+ is left. The base is accessed as `robot.base` or `robot.omnibase`. #### Translate by a Relative Amount Move the base 0.3 m forward (X): ```python robot.base.translate_by(x_m=0.3, y_m=0.0) robot.push_command() ``` Move sideways to the left (positive Y): ```python robot.base.translate_by(x_m=0.0, y_m=0.15) robot.push_command() ``` #### Rotate in Place Rotate counter-clockwise by 90 degrees (π/2 radians): ```python import math robot.base.rotate_by(w_r=math.pi / 2) robot.push_command() ``` #### Set Continuous Velocity Set a continuous velocity (useful for teleoperation or reactive control). Units: m/s for linear, rad/s for rotation: ```python # Drive forward at 0.2 m/s while rotating at 0.1 rad/s robot.base.set_velocity(vx_m=0.2, vy_m=0.0, w_r=0.1) robot.push_command() # Stop robot.base.set_velocity(vx_m=0.0, vy_m=0.0, w_r=0.0) robot.push_command() ``` #### Hard Stop Immediately stop all base motion: ```python robot.base.hard_stop() robot.push_command() ``` *** ### The Dexterous Wrist and Gripper Stretch SE4 has a 3-DOF dexterous wrist (yaw, pitch, roll) driven by Feetech servo motors over a TTL serial bus, plus a compliant spring-mechanism gripper. All wrist and gripper joints are accessed through `robot.end_of_arm`. Positions are in **radians** unless noted. | Joint | Range | | ----------------- | --------------------------------------------- | | `wrist_yaw` | ±170 deg (340 deg total) | | `wrist_pitch` | \~100 deg | | `wrist_roll` | ±170 deg (340 deg total) | | `stretch_gripper` | -100 to +100 (percent, where positive = open) | #### Move Wrist Joints Move wrist yaw to 0.5 rad: ```python robot.end_of_arm.move_to('wrist_yaw', 0.5) robot.push_command() ``` Move wrist pitch: ```python robot.end_of_arm.move_to('wrist_pitch', -0.5) robot.push_command() ``` Move wrist roll by a relative amount: ```python robot.end_of_arm.move_by('wrist_roll', 1.0) robot.push_command() robot.end_of_arm.move_by('wrist_roll', -1.0) robot.push_command() ``` #### Gripper Open the gripper (positive values = open): ```python robot.end_of_arm.move_to('stretch_gripper', 100) # or `parallel_gripper` robot.push_command() ``` Close the gripper (negative values = closed): ```python robot.end_of_arm.move_to('stretch_gripper', -100) robot.push_command() ``` Move to neutral (zero): ```python robot.end_of_arm.move_to('stretch_gripper', 0) robot.push_command() ``` > Note: To use the parallel jaw gripper or a custom tool, replace `stretch_gripper` with `parallel_gripper` or the name of your custom tool. #### Disable/Enable Torque on a Wrist Joint Making a joint backdrivable (torque off): ```python robot.end_of_arm.disable_torque('wrist_yaw') robot.push_command() ``` Re-enabling: ```python robot.end_of_arm.enable_torque('wrist_yaw') robot.push_command() ``` #### Homing the Wrist If a wrist joint loses its homed state (e.g., after a Feetech motor error and reboot), home it: ```python robot.routines.routine_wrist_joint_home('wrist_yaw') ``` Or home the entire end of arm at once: ```python robot.routines.routine_end_of_arm_home() ``` > \[!NOTE] **After a Feetech error:** If a wrist or gripper motor enters an error state (LED blinks red, motor becomes limp), clear it with `stretch_feetech_reboot` from a terminal, then re-home the wrist. *** ### Reading Robot Status #### Print Full Robot Status Print all subsystem statuses in a human-readable format: ```python robot.pretty_print() ``` This outputs a lot of data. To read individual subsystem statuses, access `robot.status` directly: ```python print(robot.status['lift']) print(robot.status['arm']) print(robot.status['omnibase']) print(robot.status['power_periph']) print(robot.status['end_of_arm']) ``` #### Key Status Fields **Lift/Arm:** ```python robot.status['lift']['pos'] # Current position (m) robot.status['lift']['vel'] # Current velocity (m/s) robot.status['arm']['pos'] # Current position (m) robot.status['arm']['motor']['pos_calibrated'] # True if homed ``` **Omnibase:** ```python robot.status['omnibase']['x'] # X position estimate (m) robot.status['omnibase']['y'] # Y position estimate (m) robot.status['omnibase']['theta'] # Heading (rad) robot.status['omnibase']['x_vel'] # X velocity (m/s) robot.status['omnibase']['y_vel'] # Y velocity (m/s) ``` **Power Periph (IMU, battery, runstop):** ```python robot.status['power_periph']['voltage_cpu'] # NUC supply voltage (V) robot.status['power_periph']['current_cpu'] # NUC current (A) robot.status['power_periph']['battery_soc'] # Battery state of charge (%) robot.status['power_periph']['runstop_event'] # True if runstop active robot.status['power_periph']['imu']['ax'] # IMU accelerometer X (m/s²) robot.status['power_periph']['imu']['gravity_tilt'] # Gravity tilt angle (rad) robot.status['power_periph']['over_tilt_type'] # Tilt direction if tilted (e.g. 'Left Tilt') ``` **Wrist Joint (via end\_of\_arm):** ```python robot.status['end_of_arm']['wrist_yaw']['pos'] # Position (rad) robot.status['end_of_arm']['wrist_yaw']['effort'] # Effort (%) robot.status['end_of_arm']['wrist_yaw']['temp'] # Motor temperature robot.status['end_of_arm']['wrist_yaw']['overtemp_error'] # Overtemp flag robot.status['end_of_arm']['wrist_yaw']['pos_calibrated'] # True if homed ``` *** ### Waiting for Motion to Complete Motion commands are asynchronous. Use these methods to synchronize: #### Wait for Specific Subsystems to Finish Moving ```python robot.arm.move_to(0.4) robot.push_command() robot.wait_on_motion_finish(['arm'], timeout=10.0) print('Arm done moving') ``` Wait for multiple joints at once: ```python robot.lift.move_to(0.8) robot.arm.move_to(0.2) robot.push_command() robot.wait_on_motion_finish(['lift', 'arm'], timeout=15.0) ``` #### Wait for All Motion to Complete To wait for the arm, lift, base, and end of arm to all finish moving: ```python robot.arm.move_to(0.3) robot.push_command() robot.wait_command(timeout=10.0) ``` *** ### Guarded Contact Sensitivity Stretch SE4's lift, arm, and omnibase joints have a **Guarded Contact** safety system. This uses current sensing to detect when actuator effort exceeds a configurable threshold. When triggered, the joint halts until a new command is received. This protects people and objects from excessive force. By default, contacts are set to a moderate sensitivity. You can change this per use case: ```python # See available modes print(robot.get_guarded_contact_modes()) # Set a robot-wide mode robot.set_guarded_contact_sensitivity('high_sensitivity_manipulation') # Or set per-joint robot.arm.set_guarded_contact_sensitivity('high_sensitivity_manipulation') robot.base.set_guarded_contact_sensitivity('default') ``` You can also pass per-move contact thresholds directly: ```python # Move arm with high contact sensitivity in the positive direction robot.arm.move_to(0.3, contact_sensitivity_pos=0.9) robot.push_command() ``` *** ### Power Periph: Beeps, Eyes, and Fan The Power Periph board controls the speaker buzzer, LED eye animations, RGB lightbar, and chassis fans. #### Trigger a Beep ```python robot.power_periph.trigger_beep() robot.push_command() ``` #### Control LED Eyes Set animations on the left and right eye rings. Animation indices are defined in the firmware (0 = off, see `stretch_eye_animations` for the full set): ```python # Set left eye animation index 3, right eye animation index 3 robot.power_periph.set_eye_animation(left_idx=3, right_idx=3) robot.push_command() ``` #### Toggle the Runstop Programmatically ```python robot.power_periph.trigger_runstop() robot.push_command() # Later, clear it robot.power_periph.clear_runstop() robot.push_command() ``` #### Control the Fan ```python robot.power_periph.set_fan_on() robot.push_command() robot.power_periph.set_fan_off() robot.push_command() ``` *** ### Writing a Complete Script Here is a complete standalone Python script that demonstrates a sequence of moves on Stretch SE4. Save this as `my_first_stretch_script.py`: ```python #!/usr/bin/env python3 """ Simple Stretch SE4 demo script. Run with: python3 my_first_stretch_script.py Prerequisites: - stretch_body_server must be running - Robot must be homed """ import time import math from stretch4_body.robot.robot_client import RobotClient def main(): with RobotClient() as robot: if not robot.is_homed(): print('Robot is not homed. Homing now...') robot.home() print('Stowing robot...') robot.stow() # --- Lift --- print('Raising lift to 0.6 m...') robot.lift.move_to(0.6) robot.push_command() robot.wait_on_motion_finish(['lift'], timeout=15.0) # --- Arm --- print('Extending arm to 0.3 m...') robot.arm.move_to(0.3) robot.push_command() robot.wait_on_motion_finish(['arm'], timeout=10.0) # --- Simultaneous move --- print('Moving lift and arm together...') robot.lift.move_to(0.8) robot.arm.move_by(0.1) robot.push_command() robot.wait_on_motion_finish(['lift', 'arm'], timeout=15.0) # --- Wrist --- print('Moving wrist yaw...') robot.end_of_arm.move_to('wrist_yaw', 0.5) robot.push_command() print('Opening gripper...') robot.end_of_arm.move_to('stretch_gripper', 100) robot.push_command() time.sleep(1.0) print('Closing gripper...') robot.end_of_arm.move_to('stretch_gripper', -50) robot.push_command() time.sleep(1.0) # --- Base --- print('Translating base forward 0.2 m...') robot.base.translate_by(x_m=0.2, y_m=0.0) robot.push_command() robot.wait_on_motion_finish(['omnibase'], timeout=10.0) print('Rotating base 90 degrees...') robot.base.rotate_by(w_r=math.pi / 2) robot.push_command() robot.wait_on_motion_finish(['omnibase'], timeout=10.0) # --- Beep to signal completion --- robot.power_periph.trigger_beep() robot.push_command() print('Demo complete. Stowing...') robot.stow() if __name__ == '__main__': main() ``` Run it with: ```bash python3 my_first_stretch_script.py ``` *** ### Feetech Motor Errors Wrist and gripper Feetech motors can enter an error state after over-force or over-temperature events. When this happens: * The motor becomes limp and backdrivable * The LED on the motor body blinks red * Motion commands are ignored Clear the error without powering down: ```bash stretch_feetech_reboot ``` After rebooting, re-home the wrist and gripper: ```bash stretch_dex_wrist_home stretch_gripper_home ``` You can also check Feetech motor health at any time: ```bash stretch_feetech_monitor ``` *** ### ROS 2 with Stretch SE4 Stretch SE4 ships with ROS 2 Jazzy. The ROS 2 driver communicates with `stretch_body_server` via the same `RobotClient` interface, making Python and ROS 2 development fully interoperable. Key ROS 2 topics published by the driver include: * `/joint_states` — positions and velocities for all joints * `/wheel_odom` — odometry from the omnibase * `/scan_filtered` — point clouds from the Hesai QT128 lidars * `/color/image_raw` — RGB images from the head cameras Key ROS 2 services include: * `/home` — trigger full robot homing * `/stow` — stow the robot The Jetson add-on (when present) subscribes to camera topics and returns inference results via Zenoh topic bridging. This enables GPU-accelerated perception (e.g., YOLO, pose estimation) without burdening the NUC. For full ROS 2 examples, refer to the ROS 2 with Stretch tutorial track on Stretch Docs. *** ### Learn More * [**Robot Overview**](/stretch-4-quick-start-guide/robot-overview) — complete reference for all hardware, sensors, and CLI tools on Stretch SE4 * [**Stretch Safety Guide**](https://docs.hello-robot.com/stretch-4-safety-guide/) — essential safety information before operating the robot *** ### Next Steps In the next tutorial, **Demo #1 - Mapping & Navigation**, we will look at two ways that Stretch can navigate within a map. # Demo - Mapping and Navigation This documentation is currently in beta. Please proceed carefully, and if you run into issues or need additional information, reach out to Hello Robot directly for support. A fundamental feature of mobile robots like Stretch is the ability to create a map of an environment and then navigate around it autonomously. The following tutorial will explain how to quickly build a 2-dimensional map that the robot can use to navigate around your current environment, using ROS 2 software called Nav2. ## Building a Map Building a map of the environment allows the robot to understand where it is currently located, and where you would like it to travel to. Stretch builds maps using the 3D LiDAR sensors located in the robot's head. These sensors allow Stretch to understand the structure of the space around the robot, including the floor, walls, and obstacles. By driving the robot slowly around the space, Stretch builds its understanding of the space, and will create a 2D map of all of the area that it can traverse freely. {% stepper %} {% step %} ### Home and Stow the Robot To make a map accurately, it is important that Stretch's arm and gripper are tucked out of the way so that they do not block the view of the sensors. To do this, run the following commands on Stretch, making sure the arm and wrist have free space to move. ``` stretch_robot_home stretch_robot_stow ``` {% endstep %} {% step %} ### Launch the ROS 2 Mapping Node Start the offline mapping launch file by running the following command in a terminal: ``` ros2 launch stretch_nav2 offline_mapping.launch.py ``` If you want to display the live feed from the robot cameras while driving, you can also run the following in a separate terminal window: ``` stretch_camera_show --opencv --right ``` {% endstep %} {% step %} ### Begin Teleoperation In a new terminal, start gamepad teleop by running: ``` stretch_gamepad_teleop ``` Just as you did in the [Quick Start Guide](/stretch-4-quick-start-guide), use the gamepad left stick and bumpers to move the robot around the space. You can watch the laser scan in RViz to understand what areas the robot has already scanned. Blank sections should be filled in by driving the robot around to get a better view. Once the map of your space looks fairly complete, you can move to the next step. We recommend starting with just one or two rooms for this initial demonstration. {% endstep %} {% step %} ### Save the Map Open a new terminal (leaving the one with the mapping node running) and use the following command to save the map to your fleet directory, replacing `` with the name you want to use (eg `stretch_demo_map`) ``` ros2 run nav2_map_server map_saver_cli -f ${HELLO_FLEET_PATH}/maps/ ``` {% endstep %} {% step %} ### Close the Mapping Node Go to the terminal window with the mapping launch file (the one from Step 2), and press Ctrl+C to close the stop the mapping script. {% endstep %} {% endstepper %} {% embed url="" %} ## Navigating the Map #### Phase 2: Navigating the Generated Map {% hint style="warning" %} Avoid initiating navigation while the robot is docked in the **Stretch Docking Station**. Always undock the robot completely before sending a navigation goal. {% endhint %} {% hint style="danger" %} This demo does **NOT** utilize the line sensors for small object or cliff detection. Keep the area clear, and **keep the robot away from stairs or ledges at all times** during this demo. {% endhint %} Now that we have saved a map, we can use it to tell the robot to autonomously navigate through your environment. Stretch will still look for new obstacles around its body, so if an object or person moves through the space, it will adjust and adapt to avoid collisions and plan its movements intelligently. {% stepper %} {% step %} ### Launch the ROS 2 Navigation Node Use the following terminal command to start Nav2 and point it to the .YAML file for the map you just created. Don't forget to update the `` parameter to the name you chose above. ``` ros2 launch stretch_nav2 navigation_mppi.launch.py map:=${HELLO_FLEET_PATH}/maps/.yaml ``` {% endstep %} {% step %} ### Set the Initial Robot Position You should now see the map open on Stretch in the RViz2 software. However, the robot doesn't yet know where it is currently positioned on the map. We'll need to tell it approximately where it is located and which direction it is facing. In the RViz2 window showing the map, click on the "2D Pose Estimate" button, then click on the robot's current location in the map, and hold and drag in the direction the robot is currently facing. Release to set the robot's orientation. \#this definitely needs a gif or short video {% endstep %} {% step %} ### Set a Navigation Goal In RViz2, click the "Nav2 Goal" button, then click a position on the map where you want the robot to navigate to. Make sure the position is inside the free area on the map. Stretch should now start driving to the location you selected autonomously! Try sending Stretch to different points in the room, then interact with it by moving objects around or standing in front of the robot to force it to react to these new obstacles and find alternative routes. {% endstep %} {% endstepper %}
Troubleshooting and Debugging 1. **Isolate node errors:** If navigation fails at startup, try adding `use_composition:=false` to the navigation launch command. This starts nodes outside a shared container, which makes errors easier to spot. 2. **Remote operation:** If you use a remote desktop session, you may need to plug in a dummy HDMI adapter so the display server starts correctly. One of these shipped in the accessory box along with your robot 3. **Navigation does not start after setting the pose:** If too much time passes before you set the robot's initial pose, the navigation stack may need to be reset. A common symptom is that the local costmap appears in RViz2, but the global costmap does not. To recover: 1. In the Navigation panel in RViz2, click **Startup**. 2. Click **Reset**. 3. Set the robot's initial pose again. Once the global costmap appears, the robot is ready to accept navigation goals.
#### Learn More To learn more information about ROS 2 Nav2 software, check out the official [Nav2 Getting Started guide](https://docs.nav2.org/getting_started/index.html) to learn the concepts in simulation. For more advanced topics and tuning guides for navigation on Stretch, you can also explore the tutorials in [Navigation University](/stretch4_docs/working-with-stretch/nav_u). In the next section, we'll learn how to launch the Web Teleoperation demo to drive Stretch with a user-friendly interface from a remote PC or mobile phone. # Demo - Web Teleoperation This documentation is currently in beta. Please proceed carefully, and if you run into issues or need additional information, reach out to Hello Robot directly for support. A common use case for Stretch is to operate the robot remotely, either on a local network or across the Internet. Hello Robot's Web Teleop software makes this possible, allowing an operator to control the robot while streaming the view through any of the robot's cameras. In the following demo, you will learn how to launch and use the Web Teleop demo that comes with Stretch. ### Launching the Demo On the robot, web teleop can be launched in two ways. {% stepper %} {% step %} ### Command Line Open a Terminal window and type the following to navigate to the correct folder: ``` colcon_cd stretch4_web_teleop ``` Then to launch the demo: ``` ./launch_interface.sh ``` {% endstep %} {% step %} ### Stretch Tray Click the Hello Robot icon at the top right of the screen, then select "Launch Web Teleop" from the dropdown menu. A Terminal window will open and launch the demo. {% endstep %} {% endstepper %} When the demo launches, the URL for connecting to the robot will be displayed at the bottom of the window (example: ). Record this for use in the next step. ### Connecting over a local network On a mobile phone or PC that is connected to the same Wi-Fi network as Stretch, open a web browser (Firefox recommended for best compatibility) and type in the URL as it appeared in the robot's Terminal window. This will connect you to the robot. {% hint style="info" %} You might receive an 'invalid certificate' error that asks you if you want to proceed with the connection - this is expected, and it is OK to proceed. {% endhint %} After a few seconds, you should see the Web Teleop interface display in your browser:
\#Insert web teleop default image {% hint style="info" %} Add hint about homing the robot if unhomed? {% endhint %} Using this interface, you can directly control all of the robot's joints, switch between camera views, change speed settings, . \#Labeled diagram of interface options \#Insert video screengrab of \~ 1 minute of web teleop ### Advanced capabilities - autonomous navigation {% hint style="info" icon="book-open" %} Documentation Coming Soon {% endhint %} ### Advanced capabilities - click-to-pregrasp {% hint style="info" icon="book-open" %} Documentation Coming Soon {% endhint %} ### Connecting remotely over the Internet It is possible to use Web Teleop entirely remotely over an internet connection, using a service like ngrok for secure tunneling. Instructions and tips can be found in the [stretch4\_web\_teleop](https://docs.hello-robot.com/stretch-4-web-teleop/) repository. # Additional Resources This documentation is currently in beta. Please proceed carefully, and if you run into issues or need additional information, reach out to Hello Robot directly for support. While we hope the documentation found here gets you successfully started developing with your robot, there are a number of other resources worth noting that provide additional information and ways to interact with Stretch and the larger Hello Robot community. ### :speech\_balloon: Forum Stretch has a diverse and vibrant user community, and one of our goals at Hello Robot is to help our customers connect, collaborate, and share with one another. To this end, we have created a [public Hello Robot Forum](https://forum.hello-robot.com/) where you can ask questions, post about your work, request support, or search an archive of resolved issues. Our engineers regularly read and respond to threads here. ### :computer: GitHub Nearly all the code we write for Stretch is open-source and freely available on our [Hello Robot GitHub](https://github.com/hello-robot). Feel free to browse through the repos or dig deeper into the code. For information on how to contribute to Stretch software, see the \[Contribution Guide]. #### Simulating Stretch Stretch 4 is simulated in Google Deepmind's [MuJoCo](https://mujoco.org/) using `stretch4_mujoco`: You can also control Stretch 4's stretch4\_mujoco in ROS2 using the `stretch_simulation` ROS2 package: #### Additional Demos There are many demos that are published to GitHub that are not shipped on the robot. Typically, you would clone these demos to your robot's `~/repos` directory and follow the instructions in the README to set them up. | [`stretch4_kinematics`](https://github.com/hello-robot/stretch4_kinematics) | Kinematics and task-space control library for Stretch 4 | Python | | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------ | | [`stretch4_grasping_demo`](https://github.com/hello-robot/stretch4_grasping_demo) | AI-powered demo utilizing VLMs and object tracking for visual servoing. | Python | | [`stretch4_hybrid_marker_demo`](https://github.com/hello-robot/stretch4_hybrid_marker_demo) | LiDAR-reflective material + visible light ArUco marker demo. | Python | | [`stretch4_rgbd`](https://github.com/hello-robot/stretch4_rgbd) | Methods to enhance and process RGB-D imagery from onboard cameras. | Python | | [`stretch4_human_perception`](https://github.com/hello-robot/stretch4_human_perception) | Specialized tools to enable the robot to perceive and track humans. | Python | | [`stretch4_compliant_gripper`](https://github.com/hello-robot/stretch4_compliant_gripper) | Modeling and control code for the standard Stretch 4 compliant gripper. | Python | ### :people\_holding\_hands: Community Updates We love to help spread the work about the great work our community is constantly publishing! Every month, we publish a \[Stretch Community Update] with some highlights of the recent research done by Stretch customers on our website, mailing list and social media. When you have work that is ready to make public, \[let us know] and we'll make sure it's seen by our community! ### :wrench: Get Some Support If you've run into a hardware problem, need urgent assistance, or just can't find the answer to your questions and want to chat directly with a Hello Robot engineer, please feel free to contact us directly at support \[at] hello-robot.com. We love talking to Stretch users, and your feedback helps us improve the product for all our users.
# Stretch 4 Safety Guide This documentation is currently in beta. Please proceed carefully, and if you run into issues or need additional information, reach out to Hello Robot directly for support. {% hint style="warning" %} Warning: There are many [safety features](#safety-features) that are built into Stretch. At the time of writing, two important features are still not implemented to run on most demo's by default: 1. Cliff detection: Stretch will not stop if driven off a threshold or a flight of stairs. Please do not navigate or teleoperate the robot near stairs or sharp drops. 2. Ramps: Stretch will classify a ramp as a tilt event, and the wheels will go into freewheel. This means that if the robot is at the top of the ramp, it will roll down freely and quickly. It will not be able to climb ramps. {% endhint %} \ Stretch is a potentially dangerous machine with safety hazards. If improperly used it can cause injury or death. Stretch is intended for use by researchers to conduct research in controlled indoor environments. This product is not intended for other uses and lacks the required certifications for other uses, such as use in the home by consumers. ### Safety Hazards We have designed Stretch to be safer than previous commercially-available human-scale mobile manipulators, so that researchers can explore the future of mobile manipulation. For example, we have made it smaller and lighter weight with backdrivable torque-sensing joints that can stop when they detect contact. Nonetheless, Stretch is a research robot that can be dangerous. It is essential that researchers use Stretch carefully to avoid damage, injury, or death. Here, we list a number of safety hazards that researchers must consider prior to use and during use of Stretch. #### **Stretch Can Put People And Animals At Risk** As described in more detail later, Stretch can put people and animals at risk. People and animals near the robot must be closely supervised at all times. At all times, an experienced researcher must carefully monitor the robot and be prepared to stop it. Any people near the robot must be made aware that the robot could be dangerous. Prior to any use of the robot near people or animals, researchers must carefully assess and minimize risks. #### **Stretch Can Topple Onto A Person** The robot may drive off stairs, push or pull itself over with its telescoping arm, fall over while attempting to traverse a threshold, or encounter obstacles that cause it to fall on or otherwise collide with people, causing injury. Operate the robot only on flat surfaces away from stairs or other obstacles that may cause it to topple, and do not allow the robot to push or pull itself over. #### **Stretch Should Not Be Lifted By A Single Person** Stretch weighs 85 lbs (\~60lbs with the battery removed), so two or more people should lift and carry the robot. A single person can move the robot around by enabling the runstop button and rolling it on flat ground, or using a hand truck or dolly. At least two people should lift and carry the robot when needed. The battery should always be removed from the robot when lifting the robot. #### **Stretch Can Cause Lacerations** The robot may have sharp edges that can cause lacerations or punctures to skin or the eyes. Operate the robot away from eyes and other sensitive body parts.
#### **Stretch Can Trap, Crush, Or Pinch Body Parts** The robot has moving joints that can trap, crush or pinch hands, fingers, or other body parts. The robot could also injure a person or animal by driving over a body part. Keep body parts away from trap, crush, and pinch points during robot motion, including underneath the wheels. #### **Stretch Can Entrap Loose Clothing Or Hair** The robot's shoulder and arm have rollers that can pull in and entrap loose clothing or hair. The robot wheels are composed of rotating parts that could also trap loose hair. Keep loose clothing and long hair away from the robot's shoulder, telescoping arm, and wheels when in motion. #### **Stretch Has Flammable Components** The robot has PLA and polyurethane covers that are flammable and must be kept away from potential ignition sources, such as open flames and hot surfaces. Keep the robot away from potential ignition sources and always have a working fire extinguisher nearby. #### **Stretch Is An Electrical Device** Stretch has batteries, electronics, wires, and other electrical components throughout its body. It also provides uncovered connectors that provide power. While the robot has fuses to reduce electrical risks, users must be careful. Keep the robot dry and away from liquids, avoid electrical shocks, ensure power cables and wires are in good condition, be careful with the robot’s connectors, and generally exercise caution while working with this electrical device. #### **Stretch Can Perform Dangerous Activities** Stretch is a versatile robot capable of performing many actions, including actions that would be dangerous to people. For example, if a dangerous object is held by or affixed to the robot, such as a knife, a heavy object, or breakable glass, the robot can become very dangerous. Likewise, the robot is capable of physically altering the environment in ways that would be dangerous, such as turning a knob that releases gas from a gas stove. Users must be cautious while using the robot to ensure it interacts safely with people and the surrounding environment. #### **Stretch Is An Open Platform That Can Be Made More Dangerous** Stretch is an open platform with user-modifiable and user-extensible hardware and software. User changes to the hardware or software can entail serious risks. For example, when shipped, the robot has conservative settings that restrict its speed and the forces it applies to reduce the risks associated with the robot. By modifying the robot, users could enable the robot to move at unsafe speeds and apply unsafe forces. As another example, improper electrical connections could result in a fire. Researchers who choose to modify or extend the robot’s hardware or software do so at their own risk, and should be careful to understand the implications of their modifications or extensions. Changes to the robot could result in dangerous situations that cause injury or death. ### Additional Risks The most important aspects of safety with Stretch are to use good judgment and common sense. Additional important considerations follow: * If the robot appears to be damaged, stop the robot immediately. * Always be ready to stop the robot. * Do not operate the robot unless an experienced user is present and attentive. * Be aware that the robot can move in unexpected ways. * Do not put fingers or other objects into the channel that runs along the length of the mast. A belt moves within this channel. * Keep an eye on cords, rugs, and any other floor hazards as the robot drives. * Keep the robot at least 3 meters from ledges, curbs, stairs, and any other toppling hazard. * Do not operate the robot outdoors. * Do not attempt to ride the robot. * Do not have the robot hold sharp objects. * Do not attempt to service the robot without supervision by Hello Robot. ### Other Problems Will Likely Occur “Anticipate potential problems and hazards. Always imagine what might happen if the robot malfunctions or behaves in a way different from the desired action. Be vigilant.” - [PR2 User Manual](https://www.clearpathrobotics.com/assets/downloads/pr2/pr2_manual_r321.pdf) by Willow Garage from October 5, 2012 Stretch is a complex device that includes many mechanical, electrical, and computational systems that have been designed to work together. Be prepared for something to go wrong. For example, a motor control board might fail, software might not operate as anticipated, an unexpected process might still be running on the robot, or the batteries for the Xbox-style controller or the robot itself might run out. ### Safety Features {% hint style="warning" %} Warning: At the time of writing, two important features are still not implemented to run on most demo's by default: 1. Cliff detection: Stretch will not stop if driven off a threshold or a flight of stairs. Please do not navigate or teleoperate the robot near stairs or sharp drops. 2. Ramps: Stretch will classify a ramp as a tilt event, and the wheels will go into freewheel. This means that if the robot is at the top of the ramp, it will roll down freely and quickly. It will not be able to climb ramps. {% endhint %} We have considered safety from the outset in the design of Stretch. * Runstop: The illuminated runstop button on Stretch’s head can be used to pause operation of the robot joints when it is in motion. * Lightweight design: The overall mass of Stretch is less than 100lbs, and the majority of the mass is in the base. While this reduces the risk of crushing, crushing injury can still occur and should be carefully monitored. * Gravity friendly: The arrangement of Stretch’s design means that it doesn’t have to counteract gravity on a larger lever arm. As a result, the motors and gearboxes are much lower torque and lower weight than a traditional robot, allowing us to avoid the often dangerous strong joints. * Low gear ratio: The primary joints of Stretch have low gear-ratios (approx 5:1), allowing for backdriving of joints when powered off. A low gear-ratio also reduces the effective inertia of each joint, limiting the impacted force during undesired contacts with people and the environment. * Contact Sensitivity: The primary joints of Stretch have contact sensitivity. We measure motor currents to estimate contact forces. Because Stretch is a low gear-ratio robot, current sensing provides a fairly sensitive measure of contact forces. * Firmware limits: Motor torques are limited at the lowest level of the firmware to configured bounds. * Velocity limits: Fast motions of the base are restricted when the lift is up high. This limits the likelihood of toppling. * Tilt detection: The robot can detect when its body is tilted beyond a safe threshold. The robot can be configured to trigger a runstop event during an over-tilt event. #### **Runstop** The runstop allows the user to pause the motion of the primary actuators by tapping the illuminated button on the head. An experienced operator should always keep the runstop within reach, allowing them to stop the motion of the robot if it is deemed unsafe. NOTE: The runstop is not equivalent to an Emergency Stop found on industrial equipment and no safety guarantees are made by its function. When the runstop is enabled, these actuators are in a ‘Safety Mode’ that inhibits the motion controller at the firmware level. Disabling the runstop allows normal operation to resume. The runstop logic is:
| Action | Runstop State | Button Illumination | | -------------------------------- | --------------- | ------------------- | | Robot startup | Motion enabled | Solid | | Tap runstop button | Motion disabled | Flashing at 1Hz | | Hold down runstop button for >2s | Motion enabled | Solid | ### **Safety Hazard Details** #### Sharp Edges Stretch is a piece of laboratory equipment. As such, its structure has moderately sharp edges and corners that can be unsafe. These edges can get snagged during motion, or they may cause lacerations when sufficient force is applied to a person. Care should be taken when grasping or otherwise making contact with Stretch that a sharp corner or edge is not contacted. #### **Toppling** Stretch is a relatively lightweight robot. In some kinematic configurations a high center of gravity can make it prone to toppling. Toppling can occur when: * The mobile base is moving at moderate or fast speed and hits a bump, threshold, or other change in floor property. * The lift is raised up high and pushes on the environment with sufficient force. * The robot drives over a drop-off such as a stair or curb. #### **Pinch Points** Pinch points around the robot's shoulder can cause discomfort and care should be taken when handling these joints as they move. The shoulder, which travels up and down on the lift, has a series of rollers that ride along the mast. While the shoulder shells can prevent large objects from getting pinched by the rollers, small and thin objects can be pulled into and crushed. Extra care should be taken with long hair, clothing, and small fingers around the shoulder rollers. #### **Crush Points** The lift degree of freedom is the strongest joint on the robot and as such can apply potentially unsafe forces to a person. The lift, while in motion, may trap or crush objects between the ‘shoulder’ and another surface. As such, best practices for lift safety should always be used when using the lift degree of freedom. The lift has a max theoretical strength of over 200N of linear force. In practice, this force is limited by the lift’s Guarded Move function, which places the lift in Safety Mode when the actuator forces exceed a threshold. \ \ \
# Stretch 4 Hardware Guide This manual provides the engineering data and user guidance for working with the Hello Robot Stretch 4 hardware. ## Disclaimer {% hint style="warning" %} The Hello Robot Stretch is intended for use in the research of mobile manipulation applications by users experienced in the use and programming of research robots. This product is not intended for general use in the home by consumers, and lacks the required certifications for such use. Please see the section on Regulatory Compliance for further details. {% endhint %} ## Functional Specification ![](/files/uzOOTL74ZW2kA7Fr5Mla)
Full system datasheet [available here](https://drive.google.com/file/d/1lyNPdvGHvn5w1UVGlV-HEfL53Mco89LW/view?usp=sharing). ## Body Plan ![](/files/AsLW3VksmjkcTyU5TPvP) ## Hardware Architecture ![](/files/G3i9ZfXt60XglC0B3wOa) ## Robot Subsystems ### Base The base is a 3-omniwheel holonomic drive. It includes 6 Laser Line Sensors to allow detection of stairs, thresholds, etc. Charging region and I/O ports are located in the "trunk" on one side of the base. ![](/files/aYJ3JSgD7VjDoTRi8WjU) | | | | ---------------------------------- | ---------------------------------------------------------------------------------------------- | | Item | Notes | | Holonomic Drive | 3X 8" diameter Omniwheels. | | Line Sensor Array | 6X Pixart Laser Line Sensors for cliff and hazard monitoring. | | Removable Shell for Battery Access | Please refer to the Battery Swap section. | | Docking Region | Spring-loaded contacts mating with the Docking Station for wireless charging. | | Charge Port |

Barrel Jack Connector.
!! Only use the Official Hello Robot 36V Charger!!

| | ON/SLEEP/OFF |

ON/OFF Button.
ON : Green solid LED.
SLEEP : Red blinking LED.
OFF : No LED.

| | ETH/USB/HDMI Ports |

ETH : 1X RJ45 Port.
USB : 2X USB-A 2.0 & 1X USB-A 3.0 Ports.
HDMI : 4K HDMI Port.

| ![](/files/RMcFjBKxrRS9TeUNc37w) ### Battery Swap Stretch 4 features a removable battery that can be accessed by removing the top shell of the base. ![](/files/TbOkLI4ViIhpIUJWCWtm) | | | | | ------ | ------------------------------------------------------ | ------------------------------------------------------------------------------- | | Step 1 | Turn off robot. | Hold the power button for \~6 seconds until the LED turns off. | | Step 2 | Remove base top shell. | Shell has magnets that keep it in place. No tool required. | | Step 3 | Unplug the cable from the battery. | Press the latch and pull the cable. | | Step 4 | Unlatch the battery from the top plate. | Flip both draw latches to release the battery from the strike plates. | | Step 5 | Remove / Swap battery. | Be mindful when manipulating the battery: **it weighs 30lbs**!! | | Step 6 | Secure the battery on the top plate with both latches. | Make sure the battery is securely held in place before moving to the next step. | | Step 7 | Plug the cable to the battery. | Make sure the cable is latched to the battery before closing the robot. | | Step 8 | Install top shell and turn the robot on. | That's it :) | {% hint style="danger" %} DO NOT charge the battery with any other charger that is not an official Hello Robot charger. {% endhint %} {% hint style="warning" %} Replacing Stretch 4's battery is NOT a "hot swap". The robot will cut power during the swap. {% endhint %} Battery removal tutorial: {% embed url="" %} Battery installation tutorial: {% embed url="" %} {% hint style="warning" %} **When lifting the robot (with or without the shipping box), please remove the battery.** The robot weighs \~60lbs without the battery, making it more comfortable to transport. {% endhint %} ### Head The head includes a high definition sensor suite with 2 Hemispherical LiDARs and 3 RGB Cameras. It also has a runstop, battery level indicators and a developer interface to allow for additional user hardware. The eye rings can be used for animations. ![](/files/3rCrnxFVSSIsgn7VZd0A) | | | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Item | Notes | | Hemispherical LiDAR (2X) | Dual Hemispherical 3D LiDAR (360° x 189°) provide a depth map of the world around the robot at all times with over 2,000,000 measurements per second. | | Fisheye Camera (2X) |

Dual Global Shutter RGB Cameras (2.3MP) provide wide FOV; calibrated to LiDARs provide RGBD data all around the body.
Eye LEDs for animation.

| | Teleoperation Camera | High-Res RGB Camera (12MP) for remote teleoperation. | | Runstop | The runstop allows the user to pause the motion of the four primary DOF (base, lift, and arm) by tapping the illuminated button on the head. When the runstop is enabled, these DOF are in a ‘Safety Mode’ that inhibits the motion controller at the firmware level. Disabling the runstop allows normal operation to resume. | | State of Charge LEDs | 8X Discrete LEDs that indicate the battery State of Charge. | | Speaker | Noise reduction microphone and speaker. | | USB Hub | 1X USB-A 3.2 & 1X USB-C 3.2 Ports. | #### Mounting Points The top of the head includes 3x M4 threaded and 4x M5 threaded mounting points as shown below: ![](/files/iCKR8LaXkKSTVSiABlry) ### Lift The lift degree of freedom provides vertical translation of the arm. It is driven by a closed loop stepper motor, providing smooth and precise motion through a low gear-ratio belt drive. The brake release button allows the motion of the arm when the robot is off. ![](/files/59qQpNhPA56qLpk9o96U) ### Telescopic Arm ![](/files/hfhCGybP4yTuVVe1F5Tn) The arm comprises 5 telescoping aluminum links set on rollers. Its proprietary drive train is driven by a stepper motor with closed loop control and current sensing, allowing contact sensitivity during motion. The retracted arm and wrist combined are designed to fit within the footprint of the base. The arm is designed to have: * Reach: 0.55m * Max payload (Retracted) : 4kg * Max payload (Extended) : 2.5kg
### Dexterous Wrist The Stretch 4 dexterous wrist is 3 DOF (Yaw, Pitch, Roll) fully internally cabled system with an OAK-D SR camera at the roll joint for precise grasping and servoing. The quick connect interface at the roll joint gives freedom to the user to use different end effector tools. A USB-A 2.0 Port for accessory mounting can be found at the top of the yaw joint shell. ![](/files/Ds4cvUss9gqzr8iX9em3) Each joint is powered by a FeeTech SM80BL. The range of motion of each joint 310°. {% embed url="" %} #### Quick Connect Interface Pressing the pink button next to the OAK-D SR Camera allows the tool to be removed. If you would like to use your own end-effector tool with the quick connect interface, please contact for CAD and hardware support. {% embed url="" %} ### Stretch Compliant Gripper The Stretch Compliant Gripper utilizes a FeeTech SM80BL servo to drive the spring grasper mechanism. It also includes Aruco tags to enable precise visual servoing, as well as mounting points to enable ‘Puppet’ based on-device data collection.
## Robot Accessories ### Parallel Jaw Gripper The Parallel Jaw Gripper utilizes a FeeTech SM80BL servo to drive the linkage mechanism. The fingers can be changed to a design of your choice. It comes standard with the ‘Aloha’ style fingers that have shown success in Physical AI for precise, dexterous manipulation. It also includes Aruco tags to enable precise visual servoing, as well as mounting points to enable ‘Puppet’ based on-device data collection.
### Puppet Teleop Pistol Accessory Stretch 4 comes with an accessory that attaches to the bottom of the Stretch Compliant Gripper or the Parallel Jaw Gripper. It can be used for data collection and remote control of a second Stretch 4 robot.
### NVIDIA Jetson Orin NX The NVIDIA Jetson Orin NX comes installed within the Stretch base and is ready for development. It includes 128GB NVME and 16GB memory. It has its own WiFi as well as Ethernet networking to the main compute.
### Docking Station The docking station sits on a flat floor against a wall, allowing Stretch to autonomously dock itself and begin charging. It includes 3X reflective Aruco tags, enabling Stretch to detect the station in low-light or dark environment using its LiDARs.
## Robot Care ### Transporting the Robot Stretch was designed to be transported in the back of a car, up a staircase, or around a building. For short trips, the robot can be simply rolled around by grabbing its mast. {% hint style="warning" %} **For safety, please use two people to lift the robot.** {% endhint %} {% hint style="warning" %} **When lifting the robot (with or without the shipping box), please remove the battery.** The robot weighs \~60lbs without the battery, making it more comfortable to transport. {% endhint %} {% hint style="warning" %} In a car or in any other means of transportation, it is heavily recommended to transport Stretch 4 in its shipping box to not damage the robot or alter the calibration of the system. {% endhint %} {% hint style="info" %} It may be picked up by its mast and carried up stairs as well. You can use the eye hook located on the top plate to attach a handle. You can also use the handle located on the bottom of the robot for a better grasp. {% endhint %} ### Belt Tension A neoprene timing belt drives the arm up and down the lift. It may loosen over long periods of time if it experiences sustained loading. In this case, slack will become visually apparent in the belt as the lift moves. The belt is very straightforward to re-tension. Please contact for tensioning instructions. ### Keeping the Robot Clean The robot surfaces can be wiped down with an alcohol wipe or a moist rag from time to time in order to remove any debris or oils that accumulate on the shells or mast. The drive wheels can accumulate dust over time and begin to lose traction. They should be periodically wiped down as well. If the robot cameras or LiDARs requires cleaning, use appropriate lens cleaning fluid and a microfiber cloth. Fan filters at the bottom of the robot can accumulate dust over time and cause overheating in the base. Removing the dust with a soft brush will help keep thermals inside the base normal. ### Keeping the Robot Calibrated The robot comes pre-calibrated with a robot-specific URDF. This calibration allows the LiDARs and cameras to accurately estimate its environment and own body. The robot may become slightly uncalibrated over time for a variety of reasons, including normal wear and tear and loosening of robot joints, or accidental collisions or falls leading to high loads on the robot joints. The calibration accuracy can be checked using the provided ROS tools. If necessary, the user can recalibrate the robot. See the [Stretch URDF Calibration Guide](https://github.com/hello-robot/stretch_ros2/tree/humble/stretch_calibration#overview) for more information. ### System Check It is useful to periodically run stretch\_system\_check.py. This will check that the robot's hardware devices are present and within normal operating conditions. ## Regulatory Compliance Stretch is not certified for use as a consumer device in the U.S., EU, or elsewhere. Unless stated otherwise, Stretch is not subjected to compliance testing nor certified to meet any requirements, such as requirements for EMI, EMC, or ESD. Per [FCC 47 CFR, Part 15, Subpart B, section 15.103(c)](https://www.law.cornell.edu/cfr/text/47/15.103), we claim Stretch as an exempted device, since it is a digital device used exclusively as industrial, commercial, or medical test equipment, where test equipment is equipment intended primarily for purposes of performing scientific investigations. [OET BULLETIN NO. 62](https://transition.fcc.gov/bureaus/oet/info/documents/bulletins/oet62/oet62rev.pdf), titled "UNDERSTANDING THE FCC REGULATIONS FOR COMPUTERS AND OTHER DIGITAL DEVICES" from December 1993 provides further clarification of the Section 15.103(c) exemption: “*Test equipment* includes devices used for maintenance, research, evaluation, simulation and other analytical or scientific applications in areas such as industrial plants, public utilities, hospitals, universities, laboratories, automotive service centers and electronic repair shops.” *** All materials are Copyright 2026 by Hello Robot Inc. Hello Robot and Stretch are registered trademarks. # README ### Overview This repository holds ROS 2 Jazzy packages for the Stretch 4 mobile manipulator from Hello Robot Inc. On a fresh robot, these packages are built and available through the `~/ament_ws` workspace. Stretch 4 ROS packages contain breaking changes from Stretch 3 and earlier hardware versions. ROS packages for earlier versions can be found in [stretch\_ros2](https://github.com/hello-robot/stretch_ros2). ### Packages | Resource | Description | | --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | [stretch\_core](/stretch4-ros2-repo/stretch_core) | ROS 2 drivers for Stretch 4 | | [stretch\_description](/stretch4-ros2-repo/stretch_description) | Visualize Stretch 4's URDF | | [stretch\_nav2](/stretch4-ros2-repo/stretch_nav2) | Navigation stack built on Nav2 | | [stretch\_simulation](/stretch4-ros2-repo/stretch_simulation) | Simulation of Stretch 4, built on [Stretch4 Mujoco](https://github.com/hello-robot/stretch4_mujoco) | | [stretch\_tag\_perception](/stretch4-ros2-repo/stretch_python_bridge) | Detect aruco tags with Stretch 4 | | [stretch\_python\_bridge](/stretch4-ros2-repo/stretch_python_bridge) | A high-level Python API that abstracts away rclpy | | [hello\_helpers](/stretch4-ros2-repo/hello_helpers) | Miscellaneous helper code used across the stretch\_ros2 repository | ### Licenses For license details for this repository, see the LICENSE files found in the directories. A summary of the licenses follows: | Directory | License | | ------------------------ | ------------------------------------------------------------------------------------- | | stretch\_core | [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0) | | stretch\_description | [BSD 3-Clause Clear License](https://choosealicense.com/licenses/bsd-3-clause-clear/) | | stretch\_nav2 | [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0) | | stretch\_simulation | [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0) | | stretch\_tag\_perception | [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0) | | stretch\_python\_bridge | [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0) | | hello\_helpers | [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0) | ### Common ROS2 commands Below is an unordered list of common or useful ROS2 commands: * Start Stretch Driver: `ros2 launch stretch_core stretch_driver.launch.py` * Teleop twist keyboard: `ros2 run teleop_twist_keyboard teleop_twist_keyboard` * Start Lidars: `ros2 launch stretch_core dual_hesai.launch.py` * Start Cameras: `ros2 launch stretch_core luxonis.launch.py use_center:=true use_left:=true use_right:=true` for the head cameras and `ros2 launch stretch_core gripper_camera.launch.py` for the gripper camera. * Navigation (See ): * Start Mapping: `ros2 launch stretch_nav2 offline_mapping.launch.py` and `stretch_gamepad_teleop` * Save a map: `ros2 run nav2_map_server map_saver_cli -f ${HELLO_FLEET_PATH}/maps/` * Start nav2: `ros2 launch stretch_nav2 navigation_mppi.launch.py map:=${HELLO_FLEET_PATH}/maps/.yaml` * Web teleop (See ): `~/ament_ws/src/stretch4_web_teleop/launch_interface.sh` and navigate to `https://localhost/operator` * Record rosbags/mcap files: `ros2 bag record -a` . Note: `-a` records all the topics, if lidar and cameras are streaming, a few minutes of recording are hundreds of gigabytes large. * Play rosbags/mcap files: `ros2 bag play ./path/to/recording` * View topics: `ros2 topic list` and `ros2 topic echo /joint_states` * View kinematic links: `ros2 run tf2_tools view_frames` * View transform between two links: `ros2 run tf2_ros tf2_echo base_link head_link` ### Troubleshooting #### Zenoh middleware not working Zenoh should start as a linux user service on Stretch 4. Run `systemctl --user status zenoh.service` to see if it has failed to start. Run `systemctl --user restart zenoh.service` to restart it. If you would like to manually start it, use `ros2 run rmw_zenoh_cpp rmw_zenohd`. #### rviz2 is not launching with a Launch file There is a known conflict between opencv and rviz2 in ROS2 Jazzy that is documented and a solution is proposed here: #### Topics are not appearing There are a number of reasons for topics to not appear. 1. Check that you are running the ros2 node and the client on the same computer. If you are trying to connect from a different computer, you may need to add the following environment variables to your client session: ``` export RMW_IMPLEMENTATION=rmw_zenoh_cpp # Replace ROBOT_IP with your Stretch's IP address ROBOT_IP=192.168.X.XXX export ZENOH_CONFIG_OVERRIDE='mode="client";connect/endpoints=["tcp/$ROBOT_IP:7447"]' ``` 2. Stretch 4 is configured with `export RMW_IMPLEMENTATION=rmw_zenoh_cpp` in the .bashrc file to tell ROS2 to use the Zenoh middleware. Make sure your terminal session with this env variable. 3. Use `ros2 topic list` to list topics. Also, `ros2 topic echo /joint_states` to listen to a topic 4. Check that the TF tree has all the links: `ros2 run tf2_tools view_frames`
# hello\_helpers ### Overview *hello\_helpers* mostly consists of the hello\_helpers Python module. This module provides various Python files used across stretch\_ros2 that have not attained sufficient status to stand on their own. ### Capabilities *fit\_plane.py* : Fits planes to 3D data. *hello\_misc.py* : Various functions, including a helpful Python object with which to create ROS nodes. *hello\_ros\_viz.py* : Various helper functions for vizualizations using RViz. ### Typical Usage ```python import hello_helpers.fit_plane as fp ``` ```python import hello_helpers.hello_misc as hm ``` ```python import hello_helpers.hello_ros_viz as hr ``` ## API ### Classes #### [HelloNode](https://github.com/hello-robot/stretch4_ros2/blob/jazzy/hello_helpers/src/hello_helpers/hello_misc.py) This class is a convenience class for creating a ROS 2 node for Stretch. The most common way to use this class is to extend it. In your extending class, the main function would call `HelloNode`'s main function. This would look like: ```python import hello_helpers.hello_misc as hm class MyNode(hm.HelloNode): def __init__(self): hm.HelloNode.__init__(self) def main(self): hm.HelloNode.main(self, 'my_node', 'my_node', wait_for_first_pointcloud=False) # my_node's main logic goes here node = MyNode() node.main() ``` There is also a one-liner class method for instantiating a `HelloNode` for easy prototyping. One example where this is handy in sending pose commands from iPython: ```python # roslaunch the stretch launch file beforehand import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') temp.move_to_pose({'lift_joint': 0.4}) ``` **Attributes** **`dryrun`** This attribute allows you to control whether the robot actually moves when calling `move_to_pose()`, `home_the_robot()`, `stow_the_robot()`, or other motion methods in this class. When `dryrun` is set to True, these motion methods return immediately. This attribute is helpful when you want to run just the perception/planning part of your node without actually moving the robot. For example, you could replace the following verbose snippet: ```python # launch the stretch driver launch file beforehand import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') actually_move = False [...] if actually_move: temp.move_to_pose({'translate_mobile_base': 1.0}) ``` to be more consise: ```python # launch the stretch driver launch file beforehand import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') [...] temp.dryrun = True temp.move_to_pose({'translate_mobile_base': 1.0}) ``` **Methods** **`move_to_pose(pose, blocking=False, custom_contact_thresholds=False, duration=2.0)`** This method takes in a dictionary that describes a desired pose for the robot and communicates with [stretch\_driver](/stretch4-ros2-repo/stretch_core#stretchdrivernodesstretchdriver) to execute it. The basic format of this dictionary is string/number key/value pairs, where the keys are joint names and the values are desired position goals. For example, `{'lift_joint': 0.5}` would put the lift at 0.5m in its joint range. A full list of command-able joints is published to the `/stretch/joint_states` topic. Used within a node extending `HelloNode`, calling this method would look like: ```python self.move_to_pose({'lift_joint': 0.5}) ``` Internally, this dictionary is converted into a [JointTrajectory](https://docs.ros2.org/latest/api/trajectory_msgs/msg/JointTrajectory.html) message that is sent to a [FollowJointTrajectory action](http://docs.ros.org/en/noetic/api/control_msgs/html/action/FollowJointTrajectory.html) server in stretch\_driver. This method waits by default for the server to report that the goal has completed executing. However, you can return before the goal has completed by setting the `blocking` argument to False. This can be useful for preempting goals. When the robot is in `position` mode, if you set `custom_contact_thresholds` to True, this method expects a different format dictionary: string/tuple key/value pairs, where the keys are still joint names, but the values are `(position_goal, effort_threshold)`. The addition of a effort threshold enables you to detect when a joint has made contact with something in the environment, which is useful for manipulation or safe movements. For example, `{'arm_joint': (0.5, 20)}` commands the telescoping arm fully out (the arm is nearly fully extended at 0.5 meters) but with a low enough effort threshold (20% of the arm motor's max effort) that the motor will stop when the end of arm has made contact with something. Again, in a node, this would look like: ```python self.move_to_pose({'arm_joint': (0.5, 40)}, custom_contact_thresholds=True) ``` When the robot is in `trajectory` mode, if you set argument `duration` as `ts`, this method will ensure that the target joint positions are achieved over `ts` seconds. For example, the below would put the lift at 0.5m from its current position in `5.0` seconds: ```python self.move_to_pose({'lift_joint': 0.5}, duration=5.0) ``` **`home_the_robot()`** This is a convenience method to interact with the driver's [`/home_the_robot` service](/stretch4-ros2-repo/stretch_core#home_the_robot-std_srvstrigger). **`stow_the_robot()`** This is a convenience method to interact with the driver's [`/stow_the_robot` service](/stretch4-ros2-repo/stretch_core#stow_the_robot-std_srvstrigger). **`stop_the_robot()`** This is a convenience method to interact with the driver's [`/stop_the_robot` service](/stretch4-ros2-repo/stretch_core#stop_the_robot-std_srvstrigger). **`get_tf(from_frame, to_frame)`** Use this method to get the transform ([geometry\_msgs/TransformStamped](https://docs.ros2.org/latest/api/geometry_msgs/msg/TransformStamped.html)) between two frames. This method is blocking. For example, this method can do forward kinematics from the base\_link to the link between the gripper fingers, grasp\_center\_link, using: ```python # launch the stretch driver launch file beforehand import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') t = temp.get_tf('base_link', 'grasp_center_link') print(t.transform.translation) ``` **`get_robot_floor_pose_xya(floor_frame='odom')`** Returns the current estimated x, y position and angle of the robot on the floor. This is typically called with respect to the odom frame or the map frame. x and y are in meters and the angle is in radians. ```python # launch the stretch driver launch file beforehand import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') t = temp.get_robot_floor_pose_xya(floor_frame='odom') print(t) ``` **`main(node_name, node_topic_namespace, wait_for_first_pointcloud=True)`** When extending the `HelloNode` class, call this method at the very beginning of your `main()` method. This method handles setting up a few ROS components, including registering the node with the ROS server, creating a TF listener, creating a [FollowJointTrajectory](http://docs.ros.org/en/noetic/api/control_msgs/html/action/FollowJointTrajectory.html) client for the [`move_to_pose()`](#movetoposepose-returnbeforedonefalse-customcontactthresholdsfalse-customfullgoalfalse) method, subscribing to depth camera point cloud topic, and connecting to the quick-stop service. Since it takes up to 30 seconds for the head camera to start streaming data, the `wait_for_first_pointcloud` argument will get the node to wait until it has seen camera data, which is helpful if your node is processing camera data. **`quick_create(name, wait_for_first_pointcloud=False)`** A class level method for quick testing. This allows you to avoid having to extend `HelloNode` to use it. ```python # launch the stretch driver launch file beforehand import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') temp.move_to_pose({'lift_joint': 0.4}) ``` **Subscribed Topics** **/camera/depth/color/points (**[**sensor\_msgs/PointCloud2**](https://docs.ros2.org/latest/api/sensor_msgs/msg/PointCloud2.html)**)** Provides a point cloud as currently seen by the Realsense depth camera in Stretch's head. Accessible from the `self.point_cloud` attribute. ```python # launch the stretch driver launch file beforehand import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp', wait_for_first_pointcloud=True) print(temp.point_cloud) ``` **/stretch/joint\_states (**[**sensor\_msgs/JointState**](https://docs.ros2.org/latest/api/sensor_msgs/msg/JointState.html)**)** Provides the current state of robot joints that includes joint names, positions, velocities, efforts. Accessible from the `self.joint_state` attribute. ```python print(temp.joint_state) ``` **/mode (**[**std\_msgs/String**](https://docs.ros2.org/latest/api/std_msgs/msg/String.html)**)** Provides the mode the stretch driver is currently in. Possible values include `position`, `trajectory`, `navigation`, `homing`, `stowing`. ```python print(temp.mode) ``` **/tool (**[**std\_msgs/String**](https://docs.ros2.org/latest/api/std_msgs/msg/String.html)**)** Provides the end of arm tool attached to the robot. ```python print(temp.tool) ``` **Subscribed Services** **/stop\_the\_robot (**[**std\_srvs/Trigger**](https://docs.ros2.org/latest/api/std_srvs/srv/Trigger.html)**)** Provides a service to quickly stop any motion currently executing on the robot. ```python # launch the stretch driver launch file beforehand from std_srvs.srv import TriggerRequest import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') temp.stop_the_robot_service(TriggerRequest()) ``` **/stow\_the\_robot (**[**std\_srvs/Trigger**](https://docs.ros2.org/latest/api/std_srvs/srv/Trigger.html)**)** Provides a service to stow the robot arm. ```python # launch the stretch driver launch file beforehand from std_srvs.srv import TriggerRequest import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') temp.stow_the_robot_service(TriggerRequest()) ``` **/home\_the\_robot (**[**std\_srvs/Trigger**](https://docs.ros2.org/latest/api/std_srvs/srv/Trigger.html)**)** Provides a service to home the robot joints. ```python # launch the stretch driver launch file beforehand from std_srvs.srv import TriggerRequest import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') temp.home_the_robot_service(TriggerRequest()) ``` **/switch\_to\_trajectory\_mode (**[**std\_srvs/Trigger**](https://docs.ros2.org/latest/api/std_srvs/srv/Trigger.html)**)** Provides a service to quickly stop any motion currently executing on the robot. ```python # launch the stretch driver launch file beforehand from std_srvs.srv import TriggerRequest import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') temp.switch_to_trajectory_mode_service(TriggerRequest()) ``` **/switch\_to\_position\_mode (**[**std\_srvs/Trigger**](https://docs.ros2.org/latest/api/std_srvs/srv/Trigger.html)**)** Provides a service to quickly stop any motion currently executing on the robot. ```python # launch the stretch driver launch file beforehand from std_srvs.srv import TriggerRequest import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') temp.switch_to_position_mode_service(TriggerRequest()) ``` **/switch\_to\_navigation\_mode (**[**std\_srvs/Trigger**](https://docs.ros2.org/latest/api/std_srvs/srv/Trigger.html)**)** Provides a service to quickly stop any motion currently executing on the robot. ```python # launch the stretch driver launch file beforehand from std_srvs.srv import TriggerRequest import hello_helpers.hello_misc as hm temp = hm.HelloNode.quick_create('temp') temp.switch_to_navigation_mode_service(TriggerRequest()) ``` ### License For license information, please see the LICENSE files. # LICENSE The following license applies to the entire contents of this directory (the "Contents"), which contains software for use with the Stretch mobile manipulators, which are robots produced and sold by Hello Robot Inc. Copyright 2020-2026 Hello Robot Inc. The Contents are licensed under the Apache License, Version 2.0 (the "License"). You may not use the Contents except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, the Contents are distributed under the License are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. For further information about the Contents including inquiries about dual licensing, please contact Hello Robot Inc. # Overview *stretch\_core* provides the drivers and navigation-facing sensor processing for the Stretch mobile manipulator. ## Dual Hesai point cloud merger The `dual_lidar_pointcloud_merger` node synchronizes the left and right Hesai `PointCloud2` streams, transforms both clouds into a target frame, preserves the original per-point fields such as `ring` and `timestamp`, and publishes one merged cloud. It is intended for consumers that need a merged point cloud rather than a projected navigation scan. The node caches the static lidar transforms after they become available, supports an optional pre-transform voxel downsample with `merger_voxel_leaf_size`, and can blank points inside a base-centered cylinder with `cylinder_filter_radius`. Example: ```bash ros2 run stretch_core dual_lidar_pointcloud_merger --ros-args \ -p left_topic:=/lidar_points_left \ -p right_topic:=/lidar_points_right \ -p output_topic:=/lidar_points \ -p target_frame:=base_link ``` ## Dual Hesai LaserScan filtering The `dual_lidar_laserscan` node (`pointcloud_to_laserscan`) fuses the left and right Hesai point clouds, runs a configurable filter pipeline, and publishes `/scan_filtered` (`LaserScan`) in `base_footprint`. The scan is only published while synchronized pairs arrive from **both** lidars: if either lidar stops (or their stamps drift more than 0.2 s apart), `/scan_filtered` goes silent and the node reports the stale input on `/diagnostics`. * `pub_pointcloud` - optional debug output: publish a filtered merged xyz cloud on `pointcloud_topic` (default `/lidar_pointcloud`, off by default) ### Filter presets Processing is handled by `DualLidarPipeline`. The easiest way to choose behavior is with `filter_type`: | `filter_type` | What it enables | Typical use | | ------------- | --------------------------------------------------------------- | --------------------------------------------------- | | `region` | Region crop + robot self-filter | Mapping and general filtered scans | | `sor` | Region crop + robot self-filter + near-robot SOR | Navigation | | `sor_ransac` | Region crop + robot self-filter + near-robot SOR + floor RANSAC | Navigation when floor returns need explicit removal | | `self` | Robot self-filter only | Debugging robot/self-hit removal | | `none` | No filtering before projection | Baseline/debug comparison | | `custom` | Uses the individual `enable_*` booleans | Experiments and tuning | ### Available filtering techniques Each technique is optional except the final LaserScan projection: | Technique | What it does | User-facing knobs | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | Region crop | Removes points outside the configured height/range limits. This is the simple floor/ceiling/far-range crop. | `z_min`, `z_max`, `range_max` | | Robot self-filter | Removes returns from the robot itself: base cylinder, arm capsule, and required URDF-derived arm/wrist/gripper/tool boxes. | `base_radius`, `arm_filter_radius`, `self_filter_*_buffer`, `self_filter_box_*` | | Self-filter spatial gate | Cheap pre-check before robot geometry. Points outside this XY/Z gate skip the expensive robot shape checks. It does not remove points by itself. | `self_filter_spatial_gate_enabled`, `self_filter_gate_radius_m`, `self_filter_gate_z_min_m`, `self_filter_gate_z_max_m` | | Internal voxel downsample | Near-robot voxel grid reduction used before expensive work. It is not a user-selectable `filter_type`; it runs automatically when robot self-filter or SOR is enabled. | `dist_rob`, `leaf_size` | | SOR | PCL StatisticalOutlierRemoval for flying/noisy near-robot points. In the `sor` preset, voxel downsample runs first, then SOR runs inside `dist_rob`; far points pass through. | `dist_rob`, `sor_mean_k`, `sor_stddev` | | Floor RANSAC | Fits the floor plane with RANSAC from the configured floor-detection Z band, then removes points close to that plane. It is only floor removal, not general obstacle detection. | `floor_detect_z_min`, `floor_detect_z_max`, `plane_fitting_threshold`, `angle` | | Speckle filter | Post-projection cleanup on LaserScan bins. It removes weak isolated bins that do not have angular support from neighboring bins. | `speckle_filter_enabled`, `speckle_min_points`, `speckle_neighbor_window`, `speckle_min_neighbors`, `speckle_range_tolerance` | The runtime order is: ``` transform -> region crop -> internal voxel -> robot self-filter -> SOR -> floor RANSAC -> LaserScan projection -> speckle filter ``` Only enabled stages run. Projection always runs because it produces `/scan_filtered`. Use `region` for mapping, `sor` for navigation, and `self` with `pub_pointcloud:=true` to visualize robot/self-hit removal without the region crop. ### Launch ```bash # Mapping-style filtering: region + self-filter ros2 launch stretch_core dual_hesai.launch.py filter_type:=region tool_preset:=auto # Navigation-style filtering: region + self-filter + near-robot SOR ros2 launch stretch_core dual_hesai.launch.py filter_type:=sor tool_preset:=auto # With RViz ros2 launch stretch_core dual_hesai.launch.py filter_type:=sor tool_preset:=auto use_rviz:=true ``` `tool_preset` selects URDF-derived self-filter geometry for the mounted tool. Use `auto` to read Stretch robot params when available, or pass `sg4`, `pg4`, `tablet`, or `nil` explicitly. Nav2 navigation launch (`stretch_nav2`) typically includes this stack with `filter_type:=sor` and starts `robot_footprint_publisher` for a dynamic costmap footprint. ### Robot self-filter and footprint geometry Self-filter geometry is generated from URDF collision geometry at launch time. The generated boxes cover the arm/lift details, wrist chain, gripper camera, and the selected SG4/PG4/tablet tool geometry. Empty frames such as grasp frames and ArUco marker frames are not used as collision boxes. The base cylinder and arm capsule are always active when the self-filter stage runs. The URDF-derived boxes are required; launch through `dual_hesai.launch.py` or `robot_footprint.launch.py` so `self_filter_config.py` can generate the temporary `self_filter_box_*` parameter YAML. The dynamic footprint publisher receives the same generated geometry as the lidar self-filter. By default, footprint buffers match the self-filter buffers so Nav2 plans conservatively around the same arm/tool volume that lidar filtering treats as robot geometry. ### Debug self-filter geometry Run the lidar filter and enable marker publication on the node: ```bash ros2 launch stretch_core dual_hesai.launch.py filter_type:=self tool_preset:=auto pub_pointcloud:=true use_rviz:=true ros2 param set /pointcloud_to_laserscan pub_self_filter_markers true ros2 param set /pointcloud_to_laserscan publish_raw_urdf_self_filter_markers true ros2 param set /pointcloud_to_laserscan publish_buffered_self_filter_markers true ``` RViz markers are published on `/self_filter_markers`. Raw URDF boxes show the collision geometry from the URDF. Buffered boxes show the effective lidar self-filter geometry. ### Configuration | File | Purpose | | ------------------------------- | ------------------------------------------------------------------------------------ | | `config/dual_lidar_filter.yaml` | `filter_type`, region limits, SOR, speckle, floor RANSAC | | `config/robot_self_filter.yaml` | Shared self-filter policy: base cylinder, arm capsule, spatial gate, marker controls | | launch-generated temp YAML | Required URDF-derived `self_filter_box_*` geometry for the selected `tool_preset` | | `config/robot_footprint.yaml` | Dynamic footprint publisher topics, base polygon, and joint update thresholds | Tuning notes for filter order, gate radius, URDF box buffers, and markers: see [config/README.md](/stretch4-ros2-repo/stretch_core/config). ### Head lidar PTP check Read-only verification of JT128 return mode (Last + Strongest), point-cloud filter (Strong), PTP lock offset (350 µs), locked PTP status (single read), and jitter p95 ≤ 350 µs over 30 s (direct PTC TCP, no ROS topics): ```bash ros2 run stretch_core stretch_lidar_check ``` Options: `--left`, `--right`, `--duration 30`, `--json`, `--verbose`. ## API For comprehensive API documentation, please refer to [Coming soon](#TODO). ## Testing Colcon is used to run the system/perf tests in the */test* folder. The command to run the entire suite of tests is: ```bash $ cd ~/ament_ws $ colcon test --packages-select stretch_core ``` You can run individual tests using the following command: ```bash $ colcon test --packages-select stretch_core --pytest-args -k test_trajectory_server -s --event-handlers console_direct+ Test suites: - test_trajectory_server - test_pub_topics - test_sub_topics - test_services - test_parameters ``` ## Head lidar PTC check `stretch_lidar_check` verifies JT128 return mode (Last + Strongest), point-cloud filter (Strong), PTP lock offset (350 µs), locked PTP status, and jitter p95 over PTC TCP (port 9347): ## License Please see the [LICENSE](/stretch4-ros2-repo/stretch_core/license) file. # LICENSE The following license applies to the entire contents of this directory (the "Contents"), which contains software for use with the Stretch mobile manipulators, which are robots produced and sold by Hello Robot Inc. Copyright 2020-2026 Hello Robot Inc. The Contents are licensed under the Apache License, Version 2.0 (the "License"). You may not use the Contents except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, the Contents are distributed under the License are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. For further information about the Contents including inquiries about dual licensing, please contact Hello Robot Inc. # stretch\_core filter configuration Shared YAML under this directory configures the dual-lidar pipeline, robot self-filter policy, and Nav2 footprint publisher. Arm, wrist, gripper-camera, and tool boxes are generated from URDF collision geometry at launch time by `launch/self_filter_config.py`. ## Filter presets and techniques The node always publishes `LaserScan` on `output_topic` (default `/scan_filtered`). Set `pub_pointcloud: true` to also publish a debug xyz `PointCloud2` on `pointcloud_topic`. `filter_type` selects a preset: | `filter_type` | Enabled techniques | | ------------- | ------------------------------------------------------------------------------------------------------------------- | | `region` | Region crop, robot self-filter | | `sor` | Region crop, robot self-filter, SOR | | `sor_ransac` | Region crop, robot self-filter, SOR, floor RANSAC | | `self` | Robot self-filter only | | `none` | No filters before LaserScan projection | | `custom` | Controlled by `enable_self_robot_filter`, `enable_region_filter`, `enable_sor_filter`, `enable_floor_ransac_filter` | Available techniques: | Technique | What it does | Main parameters | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | Region crop | Removes points outside the configured height/range limits. | `z_min`, `z_max`, `range_max` | | Robot self-filter | Removes robot returns using the base cylinder, arm capsule, and required URDF-derived arm/wrist/gripper/tool boxes. | `base_radius`, `arm_filter_radius`, `self_filter_*_buffer`, `self_filter_box_*` | | Self-filter spatial gate | Cheap XY/Z pre-check before exact robot geometry checks. It reduces computation; it is not a standalone removal filter. | `self_filter_gate_radius_m`, `self_filter_gate_z_min_m`, `self_filter_gate_z_max_m` | | Internal voxel downsample | Near-robot voxel grid reduction before self-filter or SOR. This is an implementation detail, not a `filter_type`. | `dist_rob`, `leaf_size` | | SOR | PCL StatisticalOutlierRemoval for noisy/flying points near the robot. In the `sor` preset, voxel downsample runs first, then SOR runs inside `dist_rob`; far points pass through. | `dist_rob`, `sor_mean_k`, `sor_stddev` | | Floor RANSAC | Fits the floor plane from the floor-detection Z band and removes points close to that plane. It is only floor removal, not general obstacle detection. | `floor_detect_z_min`, `floor_detect_z_max`, `plane_fitting_threshold`, `angle` | | Speckle filter | Post-projection cleanup that removes weak isolated LaserScan bins. | `speckle_*` | The implementation order in `dual_lidar_pipeline.cpp` is: ``` transform -> region crop -> internal voxel -> robot self-filter -> SOR -> floor RANSAC -> LaserScan projection -> speckle filter ``` Only enabled stages run. Projection always runs because it produces `/scan_filtered`. The internal voxel step runs automatically when robot self-filter or SOR is enabled because those are the expensive near-field stages. ## Spatial gate vs other radii | Parameter | Default | Shape | Purpose | | --------------------------- | ------- | -------------- | ---------------------------------------- | | `self_filter_gate_radius_m` | 1.5 m | Cylinder in XY | Near-field ROI for robot geometry checks | | `base_radius` | 0.25 m | Cylinder in XY | Base robot volume removed as self-hit | | `dist_rob` | 2.5 m | Square in XY | SOR denoise ROI when `filter_type:=sor` | Tune `self_filter_gate_radius_m` to at least max arm+tool XY reach, usually about 1.0-1.2 m. Increase it if extended-arm returns leak into `/scan_filtered`. Gate parameters in `robot_self_filter.yaml`: * `self_filter_spatial_gate_enabled` - master switch * `self_filter_gate_radius_m` - XY circle radius * `self_filter_gate_z_min_m` / `self_filter_gate_z_max_m` - optional height band inside the cylinder ## URDF box generation `self_filter_config.py` generates temporary `self_filter_box_*` parameters from URDF collision geometry. This replaces the old manual `self_filter_.yaml` files and the old `wrist_chain_*` parameter names. `tool_preset:=auto` tries Stretch robot params first, then fleet/user YAML, and falls back to `sg4` if the mounted tool cannot be detected. You can also pass `sg4`, `pg4`, `tablet`, or `nil` explicitly. Generated boxes include physical robot/tool collision links only: | Group | Typical links | | ---------------- | --------------------------------------------------------------------- | | `arm` | `arm_l0_link` through `arm_l4_link`, `lift_link` | | `wrist` | `wrist_link`, `wrist_yaw_link`, `wrist_pitch_link`, `wrist_roll_link` | | `gripper_camera` | `gripper_camera_link` | | `tool` | selected SG4, PG4, or tablet tool collision links | Empty/reference frames such as grasp frames, ArUco marker frames, and attachment-site frames are not used as collision boxes. The C++ self-filter requires generated URDF boxes. If a node is started without the generated `self_filter_box_*` parameters, configuration will fail instead of silently running with incomplete robot geometry. ## URDF box buffers Runtime tuning is by group: | Parameter | Applies to | | -------------------------------- | ---------------------------------- | | `self_filter_arm_buffer` | `arm` URDF boxes | | `self_filter_wrist_buffer` | `wrist` URDF boxes | | `self_filter_gripper_cam_buffer` | `gripper_camera` URDF boxes | | `self_filter_tool_buffer` | selected SG4/PG4/tablet tool boxes | Example: ```bash ros2 param set /pointcloud_to_laserscan self_filter_wrist_buffer 0.06 ros2 param set /pointcloud_to_laserscan self_filter_tool_buffer 0.06 ``` `self_filter_box_buffers` remains available as a full per-box override. Leave it empty for group tuning. By default, the Nav2 footprint uses the same effective buffer as the self-filter so planning remains conservative around the arm and tool. `self_filter_box_footprint_buffers` is available as an expert override when the footprint must intentionally differ from lidar self-filter geometry. ## RViz self-filter markers Markers are published by `pointcloud_to_laserscan` on `/self_filter_markers`. Start the lidar filter, then enable the marker parameters on that node: ```bash ros2 launch stretch_core dual_hesai.launch.py filter_type:=self tool_preset:=auto pub_pointcloud:=true use_rviz:=true ros2 param set /pointcloud_to_laserscan pub_self_filter_markers true ros2 param set /pointcloud_to_laserscan publish_raw_urdf_self_filter_markers true ros2 param set /pointcloud_to_laserscan publish_buffered_self_filter_markers true ``` | Namespace | Meaning | | ----------------------------------------------- | ------------------------------------------------------------------------------------ | | `self_filter/gate` | Spatial gate volume, which bounds expensive robot geometry checks | | `self_filter/gate_ring` | Ground circle at gate radius for top-down RViz inspection | | `self_filter/base` | Base cylinder from `base_radius` | | `self_filter/arm` | Arm capsule from `arm_l0_link`/`lift_link` to `wrist_link` | | `self_filter/urdf_raw//` | Raw URDF collision bounding box, when `publish_raw_urdf_self_filter_markers` is true | | `self_filter/urdf_buffered//` | Buffered filtering box used for lidar self-hit/artifact removal | Useful live commands: ```bash ros2 param list /pointcloud_to_laserscan ros2 param set /pointcloud_to_laserscan self_filter_gate_radius_m 1.7 ros2 param set /pointcloud_to_laserscan self_filter_wrist_buffer 0.05 ros2 param set /pointcloud_to_laserscan self_filter_tool_buffer 0.05 ``` # stretch\_deep\_perception # LICENSE The following license applies to the entire contents of this directory (the "Contents"), which contains software for use with the Stretch mobile manipulators, which are robots produced and sold by Hello Robot Inc. Copyright 2020-2026 Hello Robot Inc. The Contents are licensed under the Apache License, Version 2.0 (the "License"). You may not use the Contents except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, the Contents are distributed under the License are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. For further information about the Contents including inquiries about dual licensing, please contact Hello Robot Inc. # stretch\_description ### Overview The purpose of *stretch\_description* is simply to provide a launch file to visualize the URDF in Rviz. None of the files (`.xacro`, `.urdf`, `.stl`, etc.) live in this repo anymore; they have been moved to [stretch4\_urdf](https://github.com/hello-robot/stretch4_urdf). You can launch the visualization using: ```bash ros2 launch stretch_description display.launch.py ``` By default, the visualization uses your robot's URDF. To visualize another end-effector, batch, or variant of robot, you can run: ```bash ros2 launch stretch_description display.launch.py urdf_file:= ``` ### Details `display.launch.py` loads the URDF from [*stretch4\_urdf*](https://github.com/hello-robot/stretch4_urdf) using: ```python from stretch4_urdf import get_robot_params, get_urdf model_name, batch_name, tool_name = get_robot_params() robot_description_content = get_urdf(model_name=model_name, batch_name=batch_name, tool_name=tool_name) robot_state_publisher = Node(package='robot_state_publisher', executable='robot_state_publisher', parameters=[{'robot_description': robot_description_content}]) ``` `model` and `tool` is defined for your robot in Stretch's parameters: ``` import stretch4_body.robot r = stretch4_body.robot.Robot() model = r.params['model_name'] tool = r.params['tool'] ``` To access other URDFs, XACROs, or individual meshes from *stretch4\_urdf*, look through [Coming Soon](#TODO) for details on these assets are organized. ### License and Patents Patents are pending that cover aspects of the Stretch mobile manipulator. For license information, please see the LICENSE files. # LICENSE The following license applies to the entire contents of this directory (the "Contents"), which contains software and data for use with the Stretch mobile manipulators, which are robots produced and sold by Hello Robot Inc. *** The Clear BSD License Copyright (c) 2021-2026 Hello Robot Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted (subject to the limitations in the disclaimer below) provided that the following conditions are met: ``` * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. ``` NO EXPRESS OR IMPLIED LICENSES TO ANY PARTY'S PATENT RIGHTS ARE GRANTED BY THIS LICENSE. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # stretch\_nav2 ### Overview The *stretch\_nav2* package provides the standard ROS 2 navigation stack (Nav2) with its launch files. This package utilizes slam\_toolbox and Nav2 to drive Stretch around a mapped space. Running this code will require the robot to be untethered. We recommend stowing the arm while running navigation on the robot. ### Quickstart The first step is to map the space that the robot will navigate in. The `offline_mapping.launch.py` will enable you to do this. First, run: ```bash ros2 launch stretch_nav2 offline_mapping.launch.py ``` Rviz will show the robot and the map that is being constructed. With the terminal open, use the joystick (see instructions below for using a keyboard) to teleoperate the robot around. Avoid sharp or fast turns and revisit previously visited spots to form loop closures. In Rviz, once you see a map that has reconstructed the space well enough, open a new terminal and run the following commands to save the map to the `stretch_user/` directory. ```bash mkdir ${HELLO_FLEET_PATH}/maps ros2 run nav2_map_server map_saver_cli -f ${HELLO_FLEET_PATH}/maps/ ``` **NOTE**: The `` does not include an extension. The map\_saver node will save two files as `.pgm` and `.yaml`. **Tip**: For a quick sanity check, you can inspect the saved map using a pre-installed tool called Eye of Gnome (eog) by running the following command: ```bash eog ${HELLO_FLEET_PATH}/maps/.pgm ``` Next, with `.yaml`, we can navigate the robot around the mapped space. Run: ```bash ros2 launch stretch_nav2 navigation_mppi.launch.py map:=${HELLO_FLEET_PATH}/maps/.yaml ``` A new RViz window should pop up with a `Startup` button in a menu at the bottom left of the window. Press the `Startup` button to kick-start all navigation related lifecycle nodes. Rviz will show the robot in the previously mapped space, however, it's likely that the robot's location on the map does not match the robot's location in the real space. To correct this, from the top bar of Rviz, use `2D Pose Estimate` to lay an arrow down roughly where the robot is located in the real space. This gives an initial estimate of the robot's location to AMCL, the localization package. AMCL will better localize the robot once we pass the robot a `2D Nav Goal`. In the top bar of Rviz, use `2D Nav Goal` to lay down an arrow where you'd like the robot to navigate. In the terminal, you'll see Nav2 go through the planning phases and then navigate the robot to the goal. If planning fails, the robot will begin a recovery behavior - spinning around 180 degrees in place or backing up. **Tip**: If navigation fails or the robot becomes unresponsive to subsequent goals through RViz, you can still teleoperate the robot using the Xbox controller. *** *** ### Launch structure + Nav2 params overlay #### Which launch files to use Top-level launch files live directly under `launch/` — these are the ones you run: | Launch file | Purpose | | ----------------------------------------- | ------------------------------------------------------ | | `offline_mapping.launch.py` | Build a map with SLAM | | `navigation_mppi.launch.py` | Navigate on a saved map (main entry point) | | `navigation_mppi_nav2_filters.launch.py` | Navigate with the keepout and/or speed costmap filters | | `navigation_mppi_binary_filter.launch.py` | Navigate with binary-filter adaptive params | | `binary_filter_launch.py` | Start the binary costmap filter servers | | `global_plan_demo.launch.py` | Standalone global planner demo | Supporting launch files that are included by the above live under `launch/include/` (`nav_core`, `bringup`, `navigation_launch`, `nav2_filters`, `slam_toolbox`). You normally do not run these directly. #### Launch ordering The usual entry point is `navigation_mppi.launch.py`, which includes launch files in this order: `navigation_mppi.launch.py` → `include/nav_core.launch.py` → `include/bringup_launch.py` → `include/navigation_launch.py` What each file contains: * **`navigation_mppi.launch.py`**: top-level “run navigation on the robot” launcher. Starts the Stretch driver, starts the dual-lidar filter that publishes `/scan_filtered`, then launches `include/nav_core.launch.py` with the merged Nav2 params. * **`include/nav_core.launch.py`**: Stretch wrapper around bringup. Validates the map file, includes `include/bringup_launch.py`, and optionally launches RViz. * **`include/bringup_launch.py`**: Nav2 bringup orchestrator. Loads/rewrites the `params_file`, then includes localization/SLAM (from `nav2_bringup`) and navigation (from `stretch_nav2`). * **`include/navigation_launch.py`**: Nav2 navigation servers (controller/planner/BT/etc). Runs either as composed components in a container or as separate ROS nodes depending on `use_composition`. #### Nav2 parameter overlay order (`MultiYaml`) `navigation_mppi.launch.py` passes a `params_file` built with `MultiYaml([...])`. YAML files are merged **in order**; later files override earlier ones at nested keys. Overlay order: * `config/original_nav2_params.yaml`: upstream Nav2 baseline * `config/nav2_params_core.yaml`: Stretch-specific changes (e.g., `/scan_filtered`, omni AMCL, costmap scan topics) * `config/nav2_params_mppi.yaml`: MPPI controller selection + core controller/costmap changes * `config/mppi_params.yaml`: MPPI tuning parameters #### Debugging Nav2 components: `use_composition:=False` By default, Nav2 may run as composable components inside a single container node, which makes per-component logs harder to follow. For debugging, set: `use_composition:=False` This runs each Nav2 component as its own ROS node so its logs are visible directly. Note: the `use_composition` launch argument is declared in `include/bringup_launch.py` / `include/navigation_launch.py` and must be passed through from the top-level launch file to take effect. ### Navigation Launch Options: Different environments often require different navigation strategies. There’s no single setup that works best everywhere. Below are options you can try to adapt navigation performance to your environment. **NOTE:** use can use the argument map:=/\.yaml with all the navigation launch commands #### 1) Handling Noisy Laser Scans If your LaserScan data contains a lot of noise, use the denoise layer: ```bash ros2 launch stretch_nav2 navigation_mppi.launch.py \ params_file:=/home/hello-robot/ament_ws/src/stretch4_ros2/stretch_nav2/config/nav2_params_mppi_denoise.yaml ``` #### 2) Preventing Unnecessary Replanning In environments with many possible paths (e.g., highly connected spaces), Nav2 may constantly switch paths. You can avoid this by using a behavior tree (set via the `default_nav_to_pose_bt_xml` parameter) that only replans when the current path becomes invalid: ```bash ros2 launch stretch_nav2 navigation_mppi.launch.py params_file:=/home/hello-robot/ament_ws/src/stretch4_ros2/stretch_nav2/config/nav2_params_mppi_bt.yaml ``` #### 3) Forcing the Robot to Always Face Forward You can set the motion\_model parameter of the MPPI to diffdrive. ⚠️ Not recommended unless you specifically want the robot to always face forward. This disables omni-motion, which is usually helpful for obstacle avoidance (the robot can slide sideways without rotating making response faster). #### 4) Using a Binary Filter for Adaptive Navigation The **binary filter** detects when the robot **enters or exits specific areas** of the map.\ It publishes to the `/binary_state` topic, which outputs `true` or `false` only when the robot transitions into or out of a marked region (not continuously). In our setup, the binary filter was used to **dynamically adjust navigation parameters** to improve doorway navigation: * **Entering a doorway (narrow, cluttered area):** speed is reduced and costmap inflation is lowered. * **Exiting the doorway:** normal speed and inflation are restored. By lowering inflation in narrow areas, the robot could maneuver without being overly conservative. To maintain safety, speed was also reduced in these areas. In open spaces, higher inflation and normal speed were used to prevent collisions while enabling faster movement. This approach balances safety and efficiency by adjusting both inflation and speed according to the environment. **Launching the Binary Filter** 1. Start the filter node: ```bash ros2 launch stretch_nav2 binary_filter_launch.py ``` 2. Launch navigation with the filter configuration: ```bash ros2 launch stretch_nav2 navigation_mppi_binary_filter.launch.py ``` Alternative for running the navigation\_mppi\_binary\_filter: ```bash ros2 launch stretch_nav2 navigation_mppi.launch.py \ params_file:=/home/hello-robot/ament_ws/src/stretch4_ros2/stretch_nav2/config/nav2_params_mppi_binary_filter.yaml ros2 run stretch_nav2 binary_filter_switch.py ``` ⚠️ Remember: **binary\_filter\_launch.py** must always be started first. You can verify that the filter is active by subscribing to the map topic defined under mask\_topic in binary\_filter\_param.yaml. Ensure the QoS settings match the publisher. **How the Filter Works** The binary\_filter\_switch.py node listens to /binary\_state and updates navigation parameters whenever the robot enters or leaves a marked area. You can modify this node (located in the stretch\_nav2 subfolder) to adjust any parameter or trigger additional actions, such as disabling cameras in certain zones. **How Set Up Your Own Binary Filter** 1. Annotate your map image with the areas where the filter should activate. 2. Make sure your global or local costmap includes the filter under the filters: parameter. #### 5) Keepout and Speed Filters (`nav2_filters`) Two Nav2 costmap filters, driven by mask images you paint over your map: * **Keepout filter** — marks regions the planner must not route through. Applied to both the global and local costmaps. * **Speed filter** — marks regions with a speed limit. Applied to the global costmap only. Both live in one launch file and are selected with launch arguments, so you can run either one or both together. **Launching** Both filters (the default): ```bash ros2 launch stretch_nav2 navigation_mppi_nav2_filters.launch.py \ map:=/path/to/map.yaml \ keepout_mask:=/path/to/keepout_mask.yaml \ speed_mask:=/path/to/speed_mask.yaml ``` Keepout only — no `speed_mask` needed: ```bash ros2 launch stretch_nav2 navigation_mppi_nav2_filters.launch.py \ map:=/path/to/map.yaml \ keepout_mask:=/path/to/keepout_mask.yaml \ enable_speed:=false ``` Speed only — no `keepout_mask` needed: ```bash ros2 launch stretch_nav2 navigation_mppi_nav2_filters.launch.py \ map:=/path/to/map.yaml \ speed_mask:=/path/to/speed_mask.yaml \ enable_keepout:=false ``` Setting both `enable_keepout:=false` and `enable_speed:=false` starts no filter servers and skips the overlay entirely, leaving you with plain `navigation_mppi` behaviour. **Arguments** | Argument | Default | Purpose | | ----------------- | ------------ | ----------------------------------------------------------------------------- | | `map` | *(required)* | Occupancy map yaml | | `enable_keepout` | `true` | Start the keepout mask/info servers and enable the plugin | | `enable_speed` | `true` | Start the speed mask/info servers and enable the plugin | | `keepout_mask` | `''` | Keepout mask yaml. **Required when `enable_keepout` is true** | | `speed_mask` | `''` | Speed mask yaml. **Required when `enable_speed` is true** | | `tool_preset` | `auto` | Mounted tool for the lidar self-filter: `auto`, `sg4`, `pg4`, `tablet`, `nil` | | `use_rviz` | `true` | Start RViz | | `use_composition` | `True` | Run Nav2 composed. Set `False` to debug individual nodes | Enabling a filter without giving it a mask fails fast at launch: ``` RuntimeError: enable_keepout=true but keepout_mask is empty ``` Resources for \[Keepout and Speed Filters] ### ### ArUco Tag-Based Localization and Calibration The `stretch_nav2` package supports seeding the robot's initial pose using a pre-calibrated ArUco tag on the map. This is useful for instantly localizing the robot without manually estimating its pose in RViz using the `2D Pose Estimate` tool. The services for calibrating the tag location (`/calibrate_tag_pose` with `std_srv Trigger`) and seeding localization when the robot can see the tag (`/seed_localization` with `std_srv Trigger`) are provided by the `aruco_tag_localization.py` node. The workflow consists of two phases: 1. **Calibration**: Measure and save the static transform between the `map` and the ArUco tag (default ID: `999`, 150mm, 6x6x1000 dictionary). 2. **Localization Seeding**: Whenever the robot is unlocalized, look at the tag and trigger initial pose estimation. #### 1. Calibration Setup & Launch To run the calibration process, place the robot in a well-localized state on your map (using AMCL or RViz) facing your statically mounted localization ArUco tag. **Step 1: Launch the Tag Calibration Stack** Run the bringup launch file to start the driver, cameras, tag perception, Nav2, and the tag localization node: ```bash ros2 launch stretch_nav2 tag_calibration_bringup.launch.py map:=${HELLO_FLEET_PATH}/maps/.yaml ``` **Step 2: Launch the Interactive Calibration GUI** In a separate terminal, start the interactive OpenCV calibration interface: ```bash ros2 run stretch_nav2 calibrate_tag_cli.py ``` * Interactive Window Controls: * Adjust the robot's head or position until the target ArUco tag is highlighted with a green bounding box in the GUI. * Press c or SPACE in the GUI window, or press ENTER in your terminal to trigger the calibration. * Press ESC in the GUI or Ctrl+C in the terminal to exit. Once triggered, the terminal will print a structured comparison table of the calibrated pose and automatically save it to `~/stretch_user/maps/tag_localization/_tag_pose.yaml`. #### 2. Seeding Robot Localization Once your tag is calibrated and saved, you can use it to instantly localize the robot. Power on the robot and start navigation or tag-calibration stacks: ```bash ros2 launch stretch_nav2 tag_calibration_bringup.launch.py map:=${HELLO_FLEET_PATH}/maps/.yaml ``` Position the robot so that its camera can see the calibrated tag. Call the seed localization service to calculate and publish the robot's initial pose to Nav2 (/initialpose): ```bash ros2 service call /seed_localization std_srvs/srv/Trigger ``` If the robot can see the calibration tag, AMCL will automatically ingest this initial pose estimate and align the robot's position on the map. *** ### Things to Note and Design Decisions * **Filtering Methods:** We use two filters for point cloud processing: 1. **voxel\_sor filter** – Applies a voxel grid followed by a StatisticalOutlierRemoval within a configurable radius (e.g., \~2 m works well) to clean the point cloud before converting it into a laser scan. For navigation, `voxel_sor` improves localization by reducing outliers, but it performs poorly for mapping because the map builder expects a consistent, full environment. 2. **region\_filter** – Removes points based on position (e.g., base, ground, ceiling). Mapping launch files use `region_filter`, while navigation uses `voxel_sor` (selected via the `filter_type` parameter). The reason for applying `voxel_sor` only within a certain distance is that farther points have a different distribution than closer ones so they tend to be more distant, and noise from closer points affects motion more significantly so one parameter doesn't fit close and far away points so we focus only on the closer points. Dual-lidar filter nodes live in `stretch_core` (`region_dual_lidar_laserscan`, `voxel_dual_lidar_laserscan`, `voxel_dual_lidar_laserscan_RANSAC`), launched via `stretch_core/launch/dual_hesai.launch.py`. * **LaserScan Topic:** The filtered laser scan is published to `/scan_filtered` with **Best Effort QoS** (not reliable). * **Debugging Tips:** If a topic appears inactive in RViz2: 1. Check that `ros2 topic echo` shows messages. 2. Verify that RViz is subscribed to the correct topic. 3. Run `ros2 topic info /topic_name -v` to inspect publishers, subscribers, and QoS. Ensure subscriber and publisher QoS match. *** ### License For license information, please see the LICENSE files. # LICENSE The following license applies to the entire contents of this directory (the "Contents"), which contains software for use with the Stretch mobile manipulators, which are robots produced and sold by Hello Robot Inc. Copyright 2020-2026 Hello Robot Inc. The Contents are licensed under the Apache License, Version 2.0 (the "License"). You may not use the Contents except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, the Contents are distributed under the License are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. For further information about the Contents including inquiries about dual licensing, please contact Hello Robot Inc. # Stretch Python Bridge The `stretch_python_bridge` package provides Python generators that allow simple, synchronous access to asynchronous ROS 2 topics like images, pointclouds, and TF transforms. This is particularly useful for quickly writing scripts or for integration into standard python loop structures without the need for explicitly setting up ROS2 subscriptions and callbacks for topics. ## Prerequisites For the streams to work properly, ensure the primary robot nodes and Zenoh router are running in the background: ```bash ros2 run rmw_zenoh_cpp rmw_zenohd ros2 launch stretch_core stretch_driver.launch.py ros2 launch stretch_core dual_hesai.launch.py ros2 launch stretch_core luxonis.launch.py ros2 launch stretch_core gripper_camera.launch.py ``` ## Features * **Generators for ROS 2 Topics**: Access the latest message from a topic directly without defining subscriptions or explicit callbacks. * **Image Bridge**: Convert ROS 2 `sensor_msgs/Image` topis to NumPy arrays seamlessly. Yields a numpy array. * **PointCloud2 Bridge**: Convert ROS 2 `sensor_msgs/PointCloud2` streams to structured NumPy arrays, making it easy to access fields like x, y, z, and intensity. Yields a numpy array. * **Transforms Bridge**: Query TF2 transforms asynchronously and yield the 4x4 homogenous transformation matrix from a base frame to a target frame. Yields a 4x4 tracking numpy ndarray. * **CameraInfo Bridge**: Convert `sensor_msgs/CameraInfo` streams to fetch identical timestamped intrinsic parameter arrays cleanly. * **IMU Bridge**: Convert `sensor_msgs/Imu` streams to fetch identical timestamped IMU data arrays cleanly. ## Installation This package is a standard `ament_python` package. Compile the ROS workspace with `colcon build`: ```bash cd ~/ament_ws colcon build --packages-select stretch_python_bridge source install/setup.bash ``` ## Typical usage ## Prepared Methods for Known Topics To make scripts exceptionally concise, all primary robot cameras and lidars have prepared wrapper methods inside the module export that stream the payloads locally. They natively support `(timeout: float | None = 10.0, blocking: bool = True)`. * `stream_camera_center()` → `/cameras_head/center/image_raw` * `stream_camera_center_rotated()` → `/cameras_head/center/rotated_image` * `stream_camera_center_info()` → `/cameras_head/center/camera_info` * `stream_camera_left()` → `/cameras_head/left/image_raw` * `stream_camera_left_rotated()` → `/cameras_head/left/rotated_image` * `stream_camera_left_info()` → `/cameras_head/left/camera_info` * `stream_camera_right()` → `/cameras_head/right/image_raw` * `stream_camera_right_rotated()` → `/cameras_head/right/rotated_image` * `stream_camera_right_info()` → `/cameras_head/right/camera_info` * `stream_lidar_points_left()` → `/lidar_points_left` * `stream_lidar_points_right()` → `/lidar_points_right` * `stream_gripper_imu()` → `/cameras_gripper/imu/data` * `stream_gripper_right_info()` → `/cameras_gripper/right/camera_info` * `stream_gripper_right()` → `/cameras_gripper/right/image_raw` * `stream_gripper_stereo_info()` → `/cameras_gripper/stereo/camera_info` * `stream_gripper_stereo()` → `/cameras_gripper/stereo/image_raw` * `stream_gripper_stereo_points()` → `/cameras_gripper/stereo_left_rgbd/points` ``` import cv2 from stretch_python_bridge import * for rgb_frame in stream_camera_right_rotated(): cv2.namedWindow("RGB Stream", cv2.WINDOW_NORMAL) if rgb_frame is not None: print(f"Got a frame with timestamp: {rgb_frame.timestamp}") cv2.imshow("RGB Stream", rgb_frame.image) cv2.waitKey(1) ``` ``` import rerun as rr import numpy as np from stretch_python_bridge import * rr.init("pointcloud_stream", spawn=True) for pc_frame in stream_lidar_points_left(): if pc_frame is not None: rr.log("pointcloud", rr.Points3D(pc_frame.points)) ``` ### Examples: * [Lidar Pointcloud with intensity coloring](https://github.com/hello-robot/stretch4_ros2/blob/jazzy/stretch_python_bridge/examples/lidar_pointcloud.py) * [Gripper Colored Pointcloud](https://github.com/hello-robot/stretch4_ros2/blob/jazzy/stretch_python_bridge/examples/gripper_pointcloud.py) * [Right camera image with cv2](https://github.com/hello-robot/stretch4_ros2/blob/jazzy/stretch_python_bridge/examples/right_camera_image.py) ### 1. Typical usage using Stream Manager If your script requires monitoring multiple ROS topics simultaneously (e.g., streaming images from 3 cameras and a TF listener at the same time), using the individual generator methods will spawn heavy background threads for each topic. Instead, use `StreamManager` to monitor an unlimited number of topics using only a single internal execution thread. ```python from stretch_python_bridge import StreamManager from stretch_python_bridge import * def main(): # 1. Initialize the manager manager = StreamManager() # 2. Pass the manager to the stream generators to use the manager. center_stream = stream_camera_center(stream_manager=manager) lidar_stream = stream_lidar_points_left(stream_manager=manager) # You can also add as many subscriptions as you want explicitly: manager.add_tf_stream(target_frame="grasp_center_link", base_frame="base_link") manager.add_image_topic("/cameras_head/left/image_raw") print("Waiting for any data to arrive...") try: # 3. Use the stream as an iterable generator, this will still use the manager under-the-hood for frame in center_stream: if frame is not None: print(f"Got camera image at {frame.timestamp}") break # 4. you could also use `next()` on the generator camera_frame = next(center_stream) # 5. Get a specific topic from the manager, with options for blocking and timeout: camera_frame = manager.get(center_stream, block=True, timeout=5.0) lidar_frame = manager.get(lidar_stream, block=True, timeout=5.0) # 6. Stream from the unified generator # If block is True, this will wait until AT LEAST ONE new frame arrives across the registered streams. for frames_dict in manager.stream(block=True): # frames_dict contains the absolute *latest* cached data for ALL topics # If a topic hasn't received any data yet, its value will be None # Access the topic using .get() left_camera_frame = frames_dict.get("/cameras_head/left/image_raw") if left_camera_frame is not None: print(f"Got left camera image at {left_camera_frame.timestamp}") # Use the returned generators to cleanly retrieve the frames camera_frame = next(center_gen) if camera_frame is not None: print(f"Got camera image at {camera_frame.timestamp}") # Access the TF transform tf_data = frames_dict.get("grasp_center_link") if tf_data is not None: x = tf_data.transform_4x4[0, 3] print(f"Got Grasp Center X: {x:.3f}") except KeyboardInterrupt: pass finally: # Important: shut down the internal threads cleanly manager.close() if __name__ == '__main__': main() ``` ## For custom topics, you can use your own stream generators ### Blocking vs Non-blocking Generators The bridging functions are split into two variants for each data type: * **Blocking Generators** (e.g. `image_stream_blocking`, `tf_stream_blocking`): Guarantee a non-`None` return. Calling `next()` or using them in a `for` loop will block indefinitely until a *new* frame or message arrives. * **Non-blocking Generators** (e.g. `image_stream`): Can be configured to be completely non-blocking, or blocking with a specific timeout window. They yield `Frame | None`. ### 1. Image Stream Example Use the `image_stream_blocking` to effortlessly fetch frames from an active image topic. ```python import cv2 from stretch_python_bridge import image_stream_blocking def main(): rgb_topic = "/cameras_head/center/image_raw" rgb_gen = image_stream_blocking(rgb_topic) print(f"Subscribed to {rgb_topic}. Press CTRL+C to stop.") try: # Continuously fetch the newest image frame for rgb_frame in rgb_gen: cv2.imshow("RGB Stream", rgb_frame.image) key = cv2.waitKey(1) if key == 27: break except KeyboardInterrupt: pass finally: cv2.destroyAllWindows() if __name__ == '__main__': main() ``` ### 2. Pointcloud Stream Example Use the `pointcloud_stream_blocking` generator to get the most recent pointcloud as a structured Numpy array. Given the structured output, you can query specific fields using keys. ```python from stretch_python_bridge import pointcloud_stream_blocking def main(): pc_topic = '/lidar_points_left' pc_gen = pointcloud_stream_blocking(pc_topic) print(f"Waiting for pointclouds on {pc_topic}...") # Grab just the very first available pointcloud frame try: first_cloud = next(pc_gen) print("\nSuccessfully received 1 point cloud!") print(f"Type: {type(first_cloud.points)}") print(f"Shape: {first_cloud.points.shape}") # Access structured fields x_vals = first_cloud.points['x'] y_vals = first_cloud.points['y'] z_vals = first_cloud.points['z'] intensity_vals = first_cloud.points['intensity'] print(f"First point: x={x_vals[0]:.3f}, y={y_vals[0]:.3f}, z={z_vals[0]:.3f}, intensity={intensity_vals[0]:.3f}") except StopIteration: print("Generator stopped unexpectedly.") if __name__ == '__main__': main() ``` ### 3. Transforms Stream Example Use the `tf_stream` to track the geometric relationship and 6DoF pose of arbitrary frames. The matrix yielded is a 4x4 affine homogenous transformation containing rotation and translation data. ```python import numpy as np from stretch_python_bridge import tf_stream def main(): target_frame = "grasp_center_link" base_frame = "base_link" # Creates a generator that yields 4x4 matrices 5 times a second pose_gen = tf_stream(target_frame, base_frame=base_frame) print(f"Tracking {target_frame} with respect to {base_frame}") try: for pose_frame in pose_gen: if pose_frame is None: print("Transform lookup failed this tick...") continue # Extract out the translation components x = pose_frame.transform_4x4[0, 3] y = pose_frame.transform_4x4[1, 3] z = pose_frame.transform_4x4[2, 3] print(f"[{target_frame} pose in {base_frame}]: X={x:.3f}, Y={y:.3f}, Z={z:.3f}") except KeyboardInterrupt: pass if __name__ == '__main__': main() ``` ## Frame Dataclasses All generator wrappers from this bridge yield native Python dataclasses, providing clean and strongly-typed interfaces to underlying ROS 2 messages. Here is the structure and available bindings for each underlying datum: ### `ImageFrame` Returned by `image_stream`, `stream_camera_center`, etc. * `image: np.ndarray`: A standard NumPy array representing the image data. Its dimensions and encoding match the original topic config. * `timestamp: float`: The absolute floating-point timestamp acquired from the ROS 2 message header (`sec + nanosec * 1e-9`). ### `PointCloudFrame` Returned by `pointcloud_stream`, `stream_lidar_points_left`, etc. * `points: np.ndarray`: A structured NumPy array output from `ros2_numpy`. Common accessible channels include `.points['x']`, `.points['y']`, `.points['z']`, and `.points['intensity']`. * `timestamp: float`: The absolute floating-point timestamp acquired from the ROS 2 message header. ### `TransformsFrame` Returned by `tf_stream`. * `transform_4x4: np.ndarray`: A 4x4 homogenous affine transformation matrix containing the resolved translation and rotation from the queried base frame to target frame. * `timestamp: float`: The absolute floating-point timestamp acquired from the ROS 2 TF buffer. ### `ImuFrame` Returned by `imu_stream`, `stream_gripper_imu`. * `orientation: np.ndarray`: 1D array natively ordered as `[x, y, z, w]`. * `angular_velocity: np.ndarray`: 1D array ordered as `[x, y, z]`. * `linear_acceleration: np.ndarray`: 1D array ordered as `[x, y, z]`. * `timestamp: float`: The absolute floating-point timestamp acquired from the ROS 2 message header. ### `CameraInfoFrame` Returned by `camera_info_stream`, `stream_camera_center_info`, etc. * `camera_matrix: np.ndarray`: A 3x3 NumPy array of the intrinsic `K` parameters. * `distortion_coefficients: np.ndarray`: A 1D NumPy array covering the `D` vector. * `distortion_model: str`: The specific distortion model (e.g. `plumb_bob`) utilized by this camera logic. * `timestamp: float`: The absolute floating-point timestamp acquired from the ROS 2 message header. # LICENSE The following license applies to the entire contents of this directory (the "Contents"), which contains software for use with the Stretch mobile manipulators, which are robots produced and sold by Hello Robot Inc. Copyright 2020-2026 Hello Robot Inc. The Contents are licensed under the Apache License, Version 2.0 (the "License"). You may not use the Contents except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, the Contents are distributed under the License are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. For further information about the Contents including inquiries about dual licensing, please contact Hello Robot Inc. # Stretch Simulation in ROS2 Use this package to use ROS2 with Stretch in Mujoco. ## System Requirements Stretch Simulation supports the following environments: 1. Native Ubuntu 24.04 workstation with an Nvidia GPU 2. Docker environment with GPU support Minimum: 16GB RAM. Recommended: 32GB RAM. > Note: This package is not supported on Metal (MacOS) at this time due to the lack of GPU acceleration and OpenGL 1.5+ support in Docker, and slow performance in UTM with a virtual machine. ## Launching The main launch file is `stretch_mujoco_driver.launch.py` which you can invoke by running: ```shell ros2 launch stretch_simulation stretch_mujoco_driver.launch.py ``` This launch file has many arguments that you can pass via the command-line. ### Driver Options * `mode` (default=`position`, options: `position`, `navigation`, `trajectory`, `gamepad`) - command mode for the robot * `use_cameras` (default=false) - Enabling the cameras uses a bit of CPU ### Scene Options * By default, the Stretch spawns into a simple three-walled scene with objects on a table. * By setting `use_robocasa` to True, you can spawn into a number of different [RoboCasa](https://robocasa.ai/) environments. * You can also use your own custom scene by providing the absolute path to its XML to the `scene_xml` path. The Stretch 4 robot will be included at the origin if you have `` in the scene xml. ### Visualization Options * `use_mujoco_viewer` (default=true) * `use_rviz` (default=true) > Note: If using zenoh middleware, you should run `ros2 run rmw_zenoh_cpp rmw_zenohd` in its own terminal. ## Nav2 First go through the [Getting Started](#getting-started) guide to set up your environment. ### Mapping This section is similar to , but uses the Stretch simulation environment. To map the simulated environment, run the following: ```shell # Terminal 1: Zenoh Router ros2 run rmw_zenoh_cpp rmw_zenohd # Terminal 2: Slam Toolbox ros2 launch stretch_nav2 online_async_launch.py use_sim_time:=true # Optional: Navigation bringup: ros2 launch stretch_nav2 navigation_mppi.launch.py use_slam:=true use_sim_time:=true use_rviz:=true teleop_type:=none # Terminal 3: Stretch Mujoco Driver export MUJOCO_GL=egl # On Ubuntu, tell Mujoco to use the GPU ros2 launch stretch_simulation stretch_mujoco_driver.launch.py use_mujoco_viewer:=true mode:=navigation use_robocasa:=false # Terminal 4: Keyboard Teleop ros2 service call /switch_to_navigation_mode std_srvs/srv/Trigger ros2 run teleop_twist_keyboard teleop_twist_keyboard --ros-args --remap cmd_vel:=/stretch/cmd_vel ``` To save your map, run: ```sh mkdir ${HELLO_FLEET_PATH}/maps ros2 run nav2_map_server map_saver_cli -f ${HELLO_FLEET_PATH}/maps/ ``` ### Navigation To run navigation on a previously generated map, run: ```sh # Terminal 1: Stretch Mujoco Driver ros2 launch stretch_simulation stretch_mujoco_driver.launch.py use_mujoco_viewer:=true use_rviz:=false mode:=navigation # Terminal 2: Navigation ros2 service call /switch_to_navigation_mode std_srvs/srv/Trigger ros2 launch stretch_nav2 navigation_mppi.launch.py map:=${HELLO_FLEET_PATH}/maps/.yaml use_sim_time:=true use_rviz:=true teleop_type:=none ``` You may want to dynamically reduce the cost\_map inflation radius for most Robocasa environments: ```shell ros2 param get /global_costmap/global_costmap inflation_layer.inflation_radius ros2 param get /local_costmap/local_costmap inflation_layer.inflation_radius ros2 param set /global_costmap/global_costmap inflation_layer.inflation_radius 0.20 ros2 param set /local_costmap/local_costmap inflation_layer.inflation_radius 0.20 ``` #### Pre-mapped scene There are [maps](https://github.com/hello-robot/stretch4_ros2/blob/jazzy/stretch_simulation/maps/README.md) included in this package that you can use with navigation out of the box. Launch the pre-mapped environment using the following commands: ```shell # Terminal 1 ros2 launch stretch_simulation stretch_mujoco_driver.launch.py use_mujoco_viewer:=true mode:=navigation robocasa_layout:='G-shaped' robocasa_style:=Modern_1 # Terminal 2 ros2 launch stretch_nav2 navigation_mppi.launch.py map:=~/ament_ws/src/stretch4_ros2/stretch_simulation/maps/gshaped_modern1_robocasa.yaml use_sim_time:=true use_rviz:=true teleop_type:=none # Terminal 3 ros2 service call /stow_the_robot std_srvs/srv/Trigger ros2 param set /global_costmap/global_costmap inflation_layer.inflation_radius 0.20 ros2 param set /local_costmap/local_costmap inflation_layer.inflation_radius 0.20 ``` ## Web Teleop You can use Stretch Web Teleop with the Stretch Simulation environment! Before you start, install the dependencies for Stretch Web Teleop by following these [instructions](#setting-up-stretch-web-teleop). Use the following commands to start Stretch Mujoco with Web Teleop: ```shell parallel_terminal="gnome-terminal --tab -- /bin/bash -c " # or "xterm -e" # Terminal 1 $parallel_terminal "MUJOCO_GL=egl ros2 launch stretch_simulation stretch_mujoco_driver.launch.py use_mujoco_viewer:=false mode:=position robocasa_layout:='G-shaped' robocasa_style:=Modern_1 use_rviz:=false use_cameras:=true map:=~/ament_ws/src/stretch4_ros2/stretch_simulation/maps/gshaped_modern1_robocasa.yaml" & # Terminal 2 $parallel_terminal "ros2 launch stretch_simulation stretch_simulation_web_interface.launch.py" & # Terminal 3 $parallel_terminal "cd ~/ament_ws/src/stretch4_web_teleop; npm run localstorage" & # Terminal 4 $parallel_terminal "cd ~/ament_ws/src/stretch4_web_teleop; sudo node ./server.js" & # Terminal 4 $parallel_terminal "cd ~/ament_ws/src/stretch4_web_teleop; node start_robot_browser.js" & ``` ## Cameras and PointClouds Please use the `use_cameras:=true` argument to enable cameras and pointclouds. e.g. `ros2 launch stretch_simulation stretch_mujoco_driver.launch.py use_mujoco_viewer:=true mode:=navigation use_cameras:=true` There are five camera topics being published: * RGB and Depth for the D405 camera in the gripper. * RGB and Depth or the D435i camera in the head. * RGB for the wide-lens camera in the head. The RGB and Depth frames are used to create two PointCloud2 topics as well. ## Stretch Drivers Simulation Drivers interface with the simulator to read and write data. Simulation Drivers mimic the StretchDriver in `stretch_core`, which talks to the real robot. You could display all the launch options available to the Stretch Mujoco Driver using: `ros2 launch stretch_simulation stretch_mujoco_driver.launch.py --show-args`: ``` 'broadcast_odom_tf': Whether to broadcast the odom TF. Valid choices are: ['True', 'False'] (default: 'True') 'fail_out_of_range_goal': Whether the motion action servers fail on out-of-range commands. Valid choices are: ['True', 'False'] (default: 'False') 'mode': The mode in which the ROS driver commands the robot. Valid choices are: ['position', 'navigation', 'trajectory', 'gamepad'] (default: 'position') 'use_rviz': One of: ['true', 'false'] (default: 'true') 'use_mujoco_viewer': One of: ['true', 'false'] (default: 'true') 'use_cameras': One of: ['true', 'false'] (default: 'false') 'use_robocasa': One of: ['true', 'false'] (default: 'true') 'robocasa_task': no description given (default: 'PnPCounterToCab') 'robocasa_layout': One of: ['Random', 'One wall', 'One wall w/ island', 'L-shaped', 'L-shaped w/ island', 'Galley', 'U-shaped', 'U-shaped w/ island', 'G-shaped', 'G-shaped (large)', 'Wraparound'] (default: 'Random') 'robocasa_style': One of: ['Random', 'Industrial', 'Scandanavian', 'Coastal', 'Modern_1', 'Modern_2', 'Traditional_1', 'Traditional_2', 'Farmhouse', 'Rustic', 'Mediterranean', 'Transitional_1', 'Transitional_2'] (default: 'Random') ``` You can also set the node's argument `arguments=["--ros-args", "--log-level", "debug"]` in the launch file to display Sim-to-Real time and other useful debug information. ## Getting Started You should go through all the sections in Getting Started to run this package correctly. > Note: If you are running on a Stretch robot, you may not need to run Stretch Simulation, unless you are trying to test out the simulation environment. Estimated install time: `~1-2hrs`. ### Docker Install (Recommended) If you are on Linux or Windows, you can use the [Docker setup](/stretch4-ros2-repo/stretch_simulation/readme_docker) to get started using Docker with hardware acceleration. This is not supported on MacOS due to the lack of OpenGL 1.5+ support in Docker. ### Native Install If you would like to install Stretch Simulation directly on your host machine, please follow the [README\_SETUP](/stretch4-ros2-repo/stretch_simulation/readme_setup) instructions. # LICENSE The following license applies to the entire contents of this directory (the "Contents"), which contains software for use with the Stretch mobile manipulators, which are robots produced and sold by Hello Robot Inc. Copyright 2020-2026 Hello Robot Inc. The Contents are licensed under the Apache License, Version 2.0 (the "License"). You may not use the Contents except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, the Contents are distributed under the License are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. For further information about the Contents including inquiries about dual licensing, please contact Hello Robot Inc. # Docker Setup for Stretch Simulation with Nvidia GPU Support This guide provides instructions for setting up and running the Stretch Simulation environment in Docker with Nvidia GPU acceleration. ## Prerequisites * Ubuntu host system (22.04 or newer recommended) * Nvidia GPU with CUDA >=12.6 drivers installed * Docker * Minimum 16GB RAM (32GB recommended) ## Quick Setup The following commands will install the Nvidia Container Toolkit, test GPU access, build the Docker image, and run the container: 1. Follow [Install Docker](#1-install-docker) and [Install Nvidia Container Toolkit](#2-install-nvidia-container-toolkit) to set up your system. 2. Run the following commands: **Note:** The build process may take 1-2 hours depending on your internet connection and system performance. ```bash # 1. Install Nvidia Container Toolkit make install-nvidia-toolkit # 2. Test GPU access make test-gpu # 3. Build the Docker image (takes 1-2 hours) make build # 4. Run the container make run ``` After you have the container running, you can follow the [main README](/stretch4-ros2-repo/stretch_simulation) for commands to run the simulation. If you would like to manually go through the setup steps, please follow the instructions starting at [Build the Docker Image](#3-build-the-docker-image). ## 1. Install Docker If you don't have Docker installed, run: ```bash # Update package index sudo apt-get update # Install dependencies sudo apt-get install -y \ ca-certificates \ curl \ gnupg \ lsb-release # Add Docker's official GPG key sudo mkdir -p /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg # Set up the repository echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null # Install Docker Engine sudo apt-get update sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin # Add your user to the docker group (optional, to run docker without sudo) sudo usermod -aG docker $USER newgrp docker ``` ## 2. Install Nvidia Container Toolkit The Nvidia Container Toolkit allows Docker containers to access your GPU for hardware acceleration. ```bash # Configure the production repository distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list # Update package list sudo apt-get update # Install the Nvidia Container Toolkit sudo apt-get install -y nvidia-container-toolkit # Configure Docker to use Nvidia runtime sudo nvidia-ctk runtime configure --runtime=docker # Restart Docker daemon sudo systemctl restart docker ``` ### Verify Nvidia Container Toolkit Installation Test that GPU access works in Docker: ```bash docker run --rm --gpus all nvidia/cuda:12.6.1-base-ubuntu24.04 nvidia-smi ``` You should see your GPU information displayed. ## 3. Build the Docker Image Navigate to the stretch\_simulation directory and build the image: ```bash cd /path/to/stretch4_ros2/stretch_simulation docker build -t stretch-simulation:latest . ``` **Note:** The build process may take 1-2 hours depending on your internet connection and system performance. **Technical Note:** The Dockerfile sets the `DOCKER_BUILD=1` environment variable when running the workspace setup script. This enables Docker mode which: * Bypasses sudo checks (running as root is expected in Docker) * Skips interactive prompts for automated builds * Uses the local script instead of downloading from the internet The same `stretch_create_ament_workspace.sh` script works in both Docker and manual modes based on the `DOCKER_BUILD` environment variable. ## 4. Run the Docker Container ### Basic Usage Run the container with GPU support and X11 forwarding for GUI applications: ```bash # Allow X11 connections from Docker xhost +local:docker # Run the container docker run -it --rm \ --gpus all \ --env="DISPLAY=$DISPLAY" \ --env="QT_X11_NO_MITSHM=1" \ --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \ --volume="$HOME/.Xauthority:/root/.Xauthority:rw" \ --network=host \ --privileged \ stretch-simulation:latest ``` ### Run with Persistent Storage To persist maps and other data: ```bash docker run -it --rm \ --gpus all \ --env="DISPLAY=$DISPLAY" \ --env="QT_X11_NO_MITSHM=1" \ --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \ --volume="$HOME/.Xauthority:/root/.Xauthority:rw" \ --volume="$HOME/stretch_data:/root/stretch_user:rw" \ --network=host \ --privileged \ stretch-simulation:latest ``` ### Run Specific Launch Commands Launch the Mujoco driver directly: ```bash docker run -it --rm \ --gpus all \ --env="DISPLAY=$DISPLAY" \ --env="QT_X11_NO_MITSHM=1" \ --env="MUJOCO_GL=egl" \ --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \ --volume="$HOME/.Xauthority:/root/.Xauthority:rw" \ --network=host \ --privileged \ stretch-simulation:latest \ ros2 launch stretch_simulation stretch_mujoco_driver.launch.py mode:=navigation use_mujoco_viewer:=true ``` ## 5. Using Docker Compose (Optional) Create a `docker-compose.yml` file for easier management: ```yaml version: '3.8' services: stretch-simulation: image: stretch-simulation:latest container_name: stretch-sim privileged: true network_mode: host environment: - DISPLAY=${DISPLAY} - QT_X11_NO_MITSHM=1 - MUJOCO_GL=egl volumes: - /tmp/.X11-unix:/tmp/.X11-unix:rw - ${HOME}/.Xauthority:/root/.Xauthority:rw - ${HOME}/stretch_data:/root/stretch_user:rw deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] stdin_open: true tty: true ``` Run with: ```bash xhost +local:docker docker compose up ``` ## 6. Common Usage Examples ### Navigation with Pre-mapped Scene ```bash # Terminal 1: Launch Mujoco driver ros2 launch stretch_simulation stretch_mujoco_driver.launch.py \ use_mujoco_viewer:=true \ mode:=navigation \ robocasa_layout:='G-shaped' \ robocasa_style:=Modern_1 # Terminal 2: Launch navigation ros2 launch stretch_nav2 navigation_mppi.launch.py \ map:=/root/ament_ws/src/stretch4_ros2/stretch_simulation/maps/gshaped_modern1_robocasa.yaml \ use_sim_time:=true \ use_rviz:=true \ teleop_type:=none # Terminal 3: Configure and stow ros2 service call /stow_the_robot std_srvs/srv/Trigger ros2 param set /global_costmap/global_costmap inflation_layer.inflation_radius 0.20 ros2 param set /local_costmap/local_costmap inflation_layer.inflation_radius 0.20 ``` ### Enable Cameras and PointClouds ```bash ros2 launch stretch_simulation stretch_mujoco_driver.launch.py \ use_mujoco_viewer:=true \ mode:=navigation \ use_cameras:=true ``` ## 7. Troubleshooting ### GPU Not Detected If the container can't access the GPU: ```bash # Check Nvidia driver installation nvidia-smi # Verify Docker can see the GPU docker run --rm --gpus all nvidia/cuda:12.6.1-base-ubuntu24.04 nvidia-smi # Check Nvidia Container Toolkit configuration sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker ``` ### X11 Display Issues If GUI applications don't appear: ```bash # Allow X11 connections xhost +local:docker # Check DISPLAY variable echo $DISPLAY # Try running with --net=host docker run -it --rm --gpus all --net=host -e DISPLAY=$DISPLAY ... ``` ### OpenGL/EGL Issues If you encounter OpenGL errors: ```bash # Make sure MUJOCO_GL is set to egl docker run -it --rm --gpus all -e MUJOCO_GL=egl ... ``` ## 8. Performance Tips * Use `--shm-size=2g` or higher if you encounter shared memory issues * For better performance, use `--ipc=host` * Consider using `--cpuset-cpus` to dedicate specific CPU cores * Monitor GPU usage with `nvidia-smi` while running simulations ## 9. Cleaning Up Remove containers and images: ```bash # Remove stopped containers docker container prune # Remove unused images docker image prune # Remove the stretch-simulation image docker rmi stretch-simulation:latest ``` ## Additional Resources * [Stretch ROS2 Documentation](https://docs.hello-robot.com/0.3/ros2/) * [Nvidia Container Toolkit Documentation](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) * [Docker Documentation](https://docs.docker.com/) # Setting up Stretch Simulation You should go through all the sections in this setup guide to run this package correctly. > NOTE: If you are running on a Stretch robot, you can skip to [Setting up Mujoco](#setting-up-mujoco-15-minutes) > NOTE: If you are on Linux or Windows, you can use the [Docker setup](/stretch4-ros2-repo/stretch_simulation/readme_docker) to get started using Docker with hardware acceleration. Estimated install time: `~1-2hrs`. ## Prerequisites * Ubuntu 24.04 host system * Nvidia GPU with CUDA >=12.6 drivers installed * Minimum 16GB RAM (32GB recommended) ## Install ROS2 Jazzy (10 minutes) > NOTE: Please do not run this step if you are running on a Stretch robot. The commands below are taken from this guide: ```shell sudo apt install software-properties-common sudo add-apt-repository universe sudo apt update && sudo apt install curl -y sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null sudo apt update sudo apt install ros-jazzy-desktop ros-jazzy-rmw-zenoh-cpp ros-dev-tools python3-pip source /opt/ros/jazzy/setup.bash ``` ## Setting up `ament_ws` (1 hour) > NOTE: Please do not run this step if you are running on a Stretch robot. If you are not running this package on a robot NUC (which is *not* [recommended](#system-requirements)), you will need to set up a ROS2 environment similar to the environment that ships with Stretch. Please run these commands to install the environment. This will delete the existing `~/ament_ws` directory, so please proceed with caution. First you should install `NodeJS>=21.x` and `npm` if you don't already have them: ```shell curl -sL https://deb.nodesource.com/setup_22.x | sudo -E bash - sudo apt install -y nodejs ``` ```sh curl -sL https://raw.githubusercontent.com/hello-robot/stretch4_ros2/refs/heads/jazzy/stretch_simulation/stretch_ros2_jazzy.repos > /tmp/stretch_ros2_jazzy.repos curl -sL https://raw.githubusercontent.com/hello-robot/stretch4_ros2/refs/heads/jazzy/stretch_simulation/stretch_create_ament_workspace.sh > /tmp/stretch_create_ament_workspace.sh bash /tmp/stretch_create_ament_workspace.sh # Optional: add source install/setup.bash to .bashrc: echo 'source ~/ament_ws/install/setup.bash' >> ~/.bashrc ``` > Note: If you run into a colcon build error: "fatal error: numpy/ndarrayobject.h: No such file or directory", run `sudo ln -s ~/.local/lib/python3.10/site-packages/numpy/core/ /usr/include/numpy` to resolve it. A successful `ament_ws` setup will look like this: ``` $ ls ~/ament_ws/src airy_lidar_filter_cpp depthai-core depthai-ros ros2_numpy stretch4_ros2 stretch4_web_teleop ``` ## Setting up URDF (15 minutes) Run the commands below or follow the instruction in the [`stretch_description #updating-the-urdf`](/stretch4-ros2-repo/stretch_description#updating-the-urdf) README file to set up the URDF meshes. ```shell source ~/ament_ws/install/setup.bash git clone https://github.com/hello-robot/stretch4_urdf.git --depth 1 /tmp/stretch4_urdf cd /tmp/stretch4_urdf pip install -e . ``` A successful URDF update will look like this: ``` $ ls ~/ament_ws/src/stretch4_ros2/stretch_description/urdf/ d405 stretch_aruco.xacro stretch_description_SE3_eoa_wrist_dw3_tool_nil.xacro stretch_head_nav_cam.xacro stretch_tool_sg3.xacro d435i stretch_base_imu.xacro stretch_description_SE3_eoa_wrist_dw3_tool_sg3.xacro stretch_laser_range_finder.xacro stretch_tool_tablet_12in.xacro export_urdf_license_template.md stretch_d405_sg3.xacro stretch_description_SE3_eoa_wrist_dw3_tool_tablet_12in.xacro stretch_main.xacro stretch_uncalibrated.urdf export_urdf.sh stretch_d435i.xacro stretch_description.xacro stretch_respeaker.xacro stretch_wrist_dw3.xacro ``` ## Setting up Mujoco (15 minutes) This ROS 2 package includes nodes and launch files that use the [`stretch4_mujoco`](https://github.com/hello-robot/stretch4_mujoco) repo to interface with Mujoco. Run the following, after having done the previous ament\_ws setup steps, to start interacting with Stretch in Mujoco using ROS 2: ```shell pip3 install --upgrade pip #This is important after a fresh install of Ubuntu, for edittable installation of dependencies source ~/ament_ws/install/setup.bash # Set-up script to install mujoco and dependencies: # Note: if you get an externally managed environment error, first run:export PIP_BREAK_SYSTEM_PACKAGES=1 bash ~/ament_ws/src/stretch4_ros2/stretch_simulation/stretch_mujoco_driver/setup.bash pip install pyquaternion PyOpenGL==3.1.4 # Fixes AttributeError: module 'OpenGL.EGL' has no attribute 'EGLDeviceEXT' cd ~/ament_ws source ./install/setup.bash colcon build ros2 launch stretch_simulation stretch_mujoco_driver.launch.py mode:=navigation ``` ## Setting up Stretch Web Teleop Make sure you've already completed everything under [Setting up `ament_ws`](#setting-up-ament_ws) above. Run the following commands to get IK for the gripper working: ```shell cd stretch_description/urdf cp ./stretch_uncalibrated.urdf stretch.urdf sudo apt install rpl ./export_urdf.sh # It's okay if it fails on calibrated params mkdir -p $HELLO_FLEET_PATH/$HELLO_FLEET_ID/exported_urdf cp -r ./exported_urdf/* $HELLO_FLEET_PATH/$HELLO_FLEET_ID/exported_urdf ``` # Introduction The `stretch4_body` repository contains the core Python software stack that allows developers to interact with the hardware of Stretch 4 robots. The repository for Stretch 3 and below can be found in the [stretch\_body](https://github.com/hello-robot/stretch_body) repo. This repo provides a robust, soft real-time capable framework for managing low-level motor communication, subsystem coordination, autonomous behaviors, and a high-level API for user applications. This repository is intended to be imported by other code that needs access to these features. This package can be installed by: ``` python3 -m pip install -U hello-robot-stretch4-body ``` ## Architecture Block Diagram At its heart, the architecture is built around a Client-Server model. A dedicated `RobotServer` runs as a background daemon managing the physical hardware at 100Hz, executing safety monitoring, self-collision detection, and hardware command multiplexing. Developers build their applications using the `RobotClient`, which asynchronously communicates with the server over ZeroMQ. This decouples user scripts from strict hardware timing constraints and allows for safe, concurrent control of the robot. ```mermaid graph TD ClientCode["User Application RobotClient"] Server["Robot Server 100Hz Loop"] Subsystems["Hardware Subsystems"] Arm Lift Omnibase PowerPeriph EndOfArm Behaviors["Behaviors"] Sentries["Sentries Safety Monitors"] SafeMotions["Safe Motions Collision Avoidance"] Routines["Routines Autonomous Actions"] Workers["Background Workers"] LineSensorLoop["Line Sensor Loop"] CollisionLoop["Self Collision Loop"] EOALoop["End Of Arm Loop"] ClientCode -->|ZeroMQ Commands and Status| Server Server --> Behaviors Behaviors --> Sentries Behaviors --> SafeMotions Behaviors --> Routines Server --> Subsystems Subsystems --> Arm Subsystems --> Lift Subsystems --> Omnibase Subsystems --> PowerPeriph Subsystems --> EndOfArm Server --> Workers Workers --> LineSensorLoop Workers --> CollisionLoop Workers --> EOALoop ``` ## Technical Primers For an in-depth understanding of how specific parts of the system are designed, refer to the following technical primers: | Primer | Description | | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- | | [Core Architecture](/stretch4-body-repo/core-framework/primer_core) | Maps out the foundational classes, IPC communication, and file organization of the core library. | | [Robot Parameters](/stretch4-body-repo/core-framework/primer_robot_params) | Explains the multi-layered parameter system (default vs user) and dynamic runtime generation. | | [Robot Client API](/stretch4-body-repo/core-framework/primer_robot_client) | A guide to using the RobotClient API for reading status and commanding motion asynchronously. | | [Hardware Subsystems](/stretch4-body-repo/subsystems-and-hardware/primer_subsystems) | Overview of the primary hardware abstractions (Arm, Lift, Base) and how they are instantiated. | | [End-Of-Arm EOA](/stretch4-body-repo/subsystems-and-hardware/primer_end_of_arm) | Details the dynamically instantiated, multi-process architecture for interchangeable tool attachments. | | [Line Sensors](/stretch4-body-repo/subsystems-and-hardware/primer_line_sensor) | Details the operation and background processing for the downward-facing Pixart line sensors. | | [Server Behaviors](/stretch4-body-repo/behaviors-and-safety/primer_behaviors) | Explains the plugin architecture for Sentries, Safe Motions, and Routines within the 100Hz server loop. | | [Self-Collision](/stretch4-body-repo/behaviors-and-safety/primer_self_collision) | Details the MuJoCo-based collision checking system, its background loop, and configuration parameters. | | [Gamepad Teleop](/stretch4-body-repo/teleoperation/primer_gamepad_teleop) | Explains how different control schemes can be mapped onto a standard gamepad controller + how to extend it. | | [Cameras](/stretch4-body-repo/subsystems-and-hardware/primer_cameras) | A guide to the cameras on Stretch 4's head and wrist, with an overview of the CLIs and API. | ## Installation 1. `pip3 install -e .` 2. `stretch_body_server --launch` *Note: The C++ shared libraries for `transport` and `SCSerial` will compile automatically via Meson during the `pip install`.* If you want to install the object detection dependencies: ```bash pip3 install -e .[object_detection] ``` ### Troubleshooting Editable Installs If you make a C++ syntax error or typo in the source files and attempt to run a command while in editable mode (e.g., launching `stretch_body_server`), you may encounter an obscure Python exception instead of the actual C++ compiler error message: ``` subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1. ``` Because `meson-python` editable builds run quietly in the background on import, it drops the standard output of the C++ compiler natively, hiding your C++ syntax error. To see the actual compiler output and locate the line where C++ failed, prepend your command with the verbose flag: ```bash MESONPY_EDITABLE_VERBOSE=1 stretch_body_server --launch ``` ## Custom User End-of-Arm Tools Stretch 4 supports dynamic user-defined custom end-of-arm tools. Users can define, process, register, and switch to their own tools without modifying the core software stack. ### 1. Directory Structure Custom tools should be placed in your fleet's `user_tools` directory: * If environment variables `HELLO_FLEET_PATH` and `HELLO_FLEET_ID` are set: `//user_tools/` * Otherwise (fallback): `~/stretch_user/user_tools/` Create a subdirectory named after your tool (e.g., `user_eoa_mytool`): ```yaml > user_eoa_mytool > meshes my_tool_mesh.stl # Visual/Collision mesh files user_eoa_mytool.urdf # Tool URDF file describing joints & links user_eoa_mytool.py # Optional custom Python driver class ``` If your tool has custom driver code, the main Python file must match the tool directory name (e.g., `user_eoa_mytool.py`) or be named `tool.py`, and contain a class matching the tool name in PascalCase (e.g., `class UserEoaMytool`). ### 2. Mesh Preprocessing and Registration Once your files are in place, process the tool using the automatic registration utility. This script simplifies visual meshes, generates collision meshes, and appends the default baseline configuration (including serial devices, joint exclusion, and collision management) to `stretch_user_params.yaml`: ```bash stretch_configure_tool --add_user_tool ``` The tool will prompt you to select your custom tool subdirectory, process its URDF/meshes, and generate the parameters. ### 3. Switching to Your Tool To switch your robot to use the custom tool: ```bash stretch_configure_tool --quick --tool user_eoa_mytool ``` This updates `stretch_user_params.yaml` to make `user_eoa_mytool` the active tool. The `RobotClient`, `stretch_status`, and `stretch_system_check` utilities will automatically recognize, load, and poll your custom tool. # Core Architecture The `stretch4_body/core` directory contains the foundational base classes, utilities, and communication protocols that power the `stretch4_body` software stack. These classes are typically **not used directly by end-user developers**. Instead, they provide the internal scaffolding that the higher-level robot subsystems (like `OmniBase`, `Arm`, `Lift`, etc.) build upon to interact with the hardware and network. ## Organizational Structure The contents of the `core` directory can be grouped into six main functional areas: 1. Device and Hardware Abstractions 2. Teleoperation and Gamepad Control 3. Client/Server and IPC (Inter-Process Communication) 4. Diagnostics, Visualization, and Tracing 5. General Utilities and Configuration 6. Subdirectories (`factory`, `feetech`, `transport`) *** ### 1. Device and Hardware Abstractions These files define the base Python representations of the physical robot hardware. * **`device.py`**: Defines the foundational `Device` base class inherited by all hardware subsystems. It manages basic lifecycle execution (startup, stop), parses configuration dictionaries, and handles logging initialization. * **`prismatic_joint.py`**: An abstraction layer for linear joints, inherited specifically by the `Arm` and `Lift` subsystems. It handles the mathematical conversions from motor rotation to linear translation, manages soft-limits, and coordinates homing procedures. * **`stepper.py`**: The API interface for the custom Hello Robot stepper motor controllers (used by the mobile base wheels, arm, and lift). It manages the serial command protocols for position/velocity trajectory control, guarded motion (collision detection), and motor telemetry reporting. * **`worker_loop.py`**: A utility wrapper for instantiating high-frequency, non-blocking background multiprocessing workers (used extensively to isolate I/O tasks in the `end_of_arm` and `line_sensor` subsystems). ### 2. Teleoperation and Gamepad Control This module handles interpreting user input from USB/Bluetooth gamepads (like Xbox controllers) into smooth robotic motion. * **`gamepad_teleop.py` & `gamepad_controller.py`**: The primary drivers and classes for reading asynchronous events from the controller and smoothing them into velocity or position commands for the robot's joints. * **`gamepad_control_mappings.py` & `gamepad_enums.py` & `gamepad_joints.py`**: Define the specific button mappings, state machines, and joint-specific kinematics required to drive the robot intuitively. ### 3. Client/Server and IPC These files manage the multi-process architecture of Stretch Body, allowing multiple user scripts to communicate with the robot hardware simultaneously safely. * **`client_server.py`**: Contains the `StretchBodyServer` and `StretchBodyClient` classes. The server uses ZeroMQ (ZMQ) to publish high-frequency robot status, subscribe to incoming commands, and multiplex access by managing client priorities and leases (preventing command collisions). * **`subsystem_client.py`**: Provides the base ZMQ client used by isolated worker processes to push commands and pull status data to/from the main server loop. ### 4. Diagnostics, Visualization, and Tracing Tools for ensuring the robot operates safely and providing developers with visual debugging aids. * **`robot_monitor.py`**: A background thread that continually checks for hardware faults—such as low battery state-of-charge, runstop events, or over-tilting—and can autonomously trigger safety behaviors. * **`robot_trace.py` & `scope.py`**: Utilities for capturing, logging, and visualizing high-frequency telemetry. `scope.py` uses Matplotlib to generate oscilloscope-like plots for tuning motor control loops. * **`rerun_plot.py` & `rerun_dynamic_plotter.py`**: Integrations with the Rerun SDK for advanced, real-time 3D visual telemetry and state debugging. * **`mujoco_urdf.py`**: Uses the MuJoCo physics engine to load the robot's URDF, facilitating dynamic self-collision detection and collision visualization. ### 5. General Utilities and Configuration * **`robot_params.py`**: The central configuration manager. It loads, merges, and resolves the robot's layered YAML configuration files (`stretch_factory_params`, `stretch_configuration_params`, and `stretch_user_yaml`), providing the unified dictionary used by every subsystem. * **`hello_utils.py`**: A collection of miscellaneous, widely-used helpers for thread management, POSIX file locking, timestamping, and parsing configuration paths. *** ### 6. Subdirectories The `core` folder also contains three deeply integrated subdirectories handling low-level communication and manufacturing tooling: * **`factory/`**: A suite of low-level utility classes and scripts strictly intended for manufacturing, hardware testing, and firmware flashing. Developers should rarely need to interact with these unless debugging deep hardware issues. * **`feetech/`**: Communication wrappers and SDK implementations for interfacing with the Feetech smart serial servos. These are used exclusively to actuate the `end_of_arm` tools (like the dexterous wrist and gripper). * **`transport/`**: Contains the critical Python and C++ extensions that manage the raw, high-speed serial packet communication to the custom motor controllers and microcontrollers distributed across the robot. # Robot Client API The `stretch4_body/robot/robot_client.py` file defines the `RobotClient`, which serves as the **user-facing Python API** for interacting with the Stretch 4 robot. Whether you are writing a simple script to move an arm or developing a complex autonomous behavior, `RobotClient` is the entry point. The `RobotClient`, in your python script, talks to the `RobotServer` (started with the `stretch_body_server --daemon` command in a terminal on the robot) using [ZMQ](https://zeromq.org/). This is a **many-to-one relationship** with leases and priorities; you can have multiple RobotClient instances running at the same time, but the first one to connect to the server will hold the lease until it stops sending commands. > Note: `stretch_gamepad_teleop` also uses `RobotClient` and has a higher priority than the default priorty setting for client instances. Commands from gamepad teleop will override commands from any other script. ## 1. Typical Usage of the `RobotClient` The typical usage involves instantiating the `RobotClient`, calling `startup()` to establish connections to the robot server, executing your behavior, and finally calling `stop()` to cleanly close the connections: ```python from stretch4_body.robot.robot_client import RobotClient robot = RobotClient() if not robot.startup(): raise Exception("Could not start the robot client.") if not robot.is_homed(): if input('Home Stretch? Note: Joints will move! [y/n]') == 'y': robot.home() # Warning! This will move the robot joints! print("Stretch 4 is ready!") robot.stop() ``` Alternatively, you can use it as a context manager to handle the cleanup automatically: ```python import time from stretch4_body.robot.robot_client import RobotClient # Using a context manager ensures stop() is called automatically with RobotClient() as robot: # Check if the robot needs to be homed if not robot.is_homed(): if input('Home Stretch? Note: Joints will move! [y/n]') == 'y': robot.home() # Warning! This will move the robot joints! print("Stretch 4 is ready!") ``` ## 2. Accessing Subsystems The `RobotClient` aggregates all of the robot's hardware components into individual **subsystem clients**. The primary subsystems relate directly to the `RobotClient` as attributes. For example, `robot.arm`, `robot.lift`, `robot.omnibase`, and `robot.end_of_arm`. ### Example: Commanding Subsystems ```python with RobotClient() as robot: # Command the arm to extend to 0.3 meters robot.arm.move_to(0.3) # Command the lift to move up by 0.1 meters robot.lift.move_by(0.1) ``` > Note: If you command a joint to move, but the joint is not homed, **it will not move.** The subsystem command will return `False` and print a warning that the joint is not homed. To home the robot, run `robot.home()` in your code, or use the cli command `stretch_robot_home` in a terminal. ## 3. The User Control Loop Design Pattern Stretch is designed around Client and Server control loops. The Client relies on two primary functions: `pull_status()` and `push_command()`to communicate with the Server. 1. **`pull_status()`**: Fetches the latest sensor data, joint positions, and state from the robot server and updates the `robot.status` dictionary. 2. **`push_command()`**: Takes all commands queued up in the various subsystems and flushes them to the robot hardware simultaneously. ### Deep Copy Note that the dictionary returned by `pull_status()` is **deep-copied**. A reference to a key in an older `pull_status()` will retain the value of the old dictionary and will not be updated by a subsequent `pull_status()`. Therefore, it it important to pull\_status() before acting on the status dictionary: ```python with RobotClient() as robot: while True: # 1. Always pull_status inside the loop before accessing it robot.pull_status() # 2. Access the status dictionary print(f"{robot.arm.status['pos']=}") time.sleep(1/15) # 15Hz loop ``` ### Rate Expectations A typical user control loop runs between **10 Hz and 50 Hz**. Running faster than the Server's control loop (100 Hz) is unnecessary. ### Example: Control Loop ```python with RobotClient() as robot: rate_hz = 20.0 dt = 1.0 / rate_hz while True: # 1. Update the status dictionary with the latest hardware state robot.pull_status() # 2. Read the current position of the lift current_lift_pos = robot.status['lift']['pos'] # 3. Calculate a new position based on some logic (e.g., following a target) target_lift_pos = current_lift_pos + 0.01 # 4. Queue the command (does not move the robot yet) robot.lift.move_to(target_lift_pos) # 5. Flush all queued commands to the hardware simultaneously robot.push_command() # 6. Sleep to maintain the loop rate time.sleep(dt) ``` ## 4. The Status Dictionary Structure When you call `robot.pull_status()`, the `RobotClient` populates a master dictionary accessible via `robot.status`. This dictionary acts as a snapshot of the robot's state at that exact moment. The `robot.status` dictionary is organized by subsystem. Each subsystem provides its own set of keys reflecting its physical state. ### Common Subsystem Status Keys For prismatic or revolute joints like the `arm` or `lift`, the status dictionary generally includes: * **`pos`**: The current position (meters or radians). * **`vel`**: The current velocity (m/s or rad/s). * **`effort`**: The measured effort/torque. For the `omnibase`, you will typically find odometry information such as `x`, `y`, and `theta`. For the `power_periph` (power system and IMU), you'll find system health data like `voltage` and `current`. ### Example: Working with Status Dictionaries ```python with RobotClient() as robot: # Always pull the latest status before reading! robot.pull_status() # --- Reading Arm Status --- arm_pos = robot.status['arm']['pos'] arm_effort = robot.status['arm']['effort'] print(f"Arm Position: {arm_pos:.3f} m, Effort: {arm_effort:.2f}") # --- Reading End-Of-Arm (EOA) Status --- # The EOA structure depends on the tool configured (e.g., a gripper) if 'stretch_gripper' in robot.status['end_of_arm']: gripper_pos = robot.status['end_of_arm']['stretch_gripper']['pos'] print(f"Gripper Position: {gripper_pos:.2f} rad") # --- Reading Power Status --- battery_v = robot.status['power_periph']['voltage'] print(f"Battery Voltage: {battery_v:.2f} V") ``` ### Direct Subsystem Status Access You can also access a subsystem's status directly via the subsystem object, which points to the exact same dictionary: ```python # These two lines return the identical value pos_1 = robot.status['lift']['pos'] pos_2 = robot.lift.status['pos'] ``` ## 5. Blocking vs. Non-Blocking Calls A crucial concept in Stretch's API is understanding when a command blocks the execution of your Python script and when it does not. ### Non-Blocking Calls (Asynchronous) By default, subsystem commands like `move_to()`, `move_by()`, and `set_velocity()` simply queue the intent. When you call `robot.push_command()`, the command is sent to the server, and your script continues executing immediately. **The robot will move in the background.** This is essential for control loops (like the one shown above) where you need to continuously read sensors while the robot is moving. ### Blocking Calls (Synchronous) Sometimes you want your script to wait until a motion is physically finished before executing the next line of code. You can achieve this by explicitly waiting for the motion to finish. High-level routines, like `robot.home()` or `robot.stow()`, are typically blocking by default. ### Example: Waiting for Motion ```python with RobotClient() as robot: # Command the arm to move (Non-blocking) robot.arm.move_to(0.5) robot.push_command() # Wait until the arm has finished its trajectory (Blocking) robot.wait_on_motion_finish(['arm']) print("Arm has reached its destination. Moving the lift.") # Now command the lift robot.lift.move_to(0.8) robot.push_command() robot.wait_on_motion_finish(['lift']) ``` ## 6. Remote Connection Clients The `RobotClient` sits on top of a network communication layer (ZMQ), and doesn't have to be running on the robot's physical computer. You can run your Python scripts from your laptop to control the robot remotely over WiFi or a tunnel. To do this, simply instantiate the `RobotClient` with the robot's IP address: ```python # Connect to a Stretch robot over the local network REMOTE_IP = "192.168.1.105" with RobotClient(ip_address=REMOTE_IP) as robot: robot.pull_status() print("Successfully connected to the remote robot!") print(f"Current battery voltage: {robot.status['power_periph']['voltage']}") ``` ### Multiple User Access If multiple Ubuntu User Accounts are logged in to the robot at the same time (e.g. using [RDP or remote access](https://docs.hello-robot.com/stretch4_docs/working-with-stretch/general_use/connecting-to-stretch?q=user#untethered_setup)), there might only be **one instance** of the Server (`stretch_body_server --daemon`) running at a time. By default, inter-user access to the server is disabled to avoid versioning conflicts. However, if you would still like to connect to another user's Server, you can pass the `allow_different_user_connection=True` parameter during startup: ```python robot = RobotClient() robot.startup(allow_different_user_connection=True) robot.pull_status() ``` Note that if you try to connect to another user's server without the `allow_different_user_connection` parameter, you will get this error: ``` StretchBodyClient: A server is already running, but it was started by a different user (Username). StretchBodyClient: You can run `stretch_body_server --kill` to forcefully end the other user's session. ``` ## 7. Architecture Visualization The following diagram illustrates the hierarchy and data flow from your user application down to the physical hardware. ```mermaid graph TD USER_CODE["User Application Code
(Your Script)"]:::user RC["RobotClient
(Aggregator)"]:::client subgraph Subsystems ARM["ArmClient"]:::sub LIFT["LiftClient"]:::sub BASE["OmniBaseClient"]:::sub EOA["EndOfArmClient"]:::sub ROUTINES["RoutinesClient"]:::sub end COMM_LAYER["Communication Layer
(SubsystemClient / RPC)"]:::comm ROBOT_SERVER["Robot Server
(Runs on Robot Hardware)"]:::server %% API Interactions USER_CODE -->|"Calls pull_status() / push_command()"| RC USER_CODE -->|"Calls move_to(), etc."| ARM USER_CODE -->|Calls routines| ROUTINES %% Aggregation RC --> ARM RC --> LIFT RC --> BASE RC --> EOA RC --> ROUTINES %% Communication ARM --> COMM_LAYER LIFT --> COMM_LAYER BASE --> COMM_LAYER EOA --> COMM_LAYER ROUTINES --> COMM_LAYER RC --> COMM_LAYER %% Network Boundary COMM_LAYER <-->|ZeroMQ / TCP / IPC| ROBOT_SERVER classDef user fill:#e88d3e,stroke:#333,stroke-width:2px,color:#fff; classDef client fill:#57a661,stroke:#333,stroke-width:2px,color:#fff; classDef sub fill:#4f81c7,stroke:#333,stroke-width:2px,color:#fff; classDef comm fill:#a859b3,stroke:#333,stroke-width:2px,color:#fff; classDef server fill:#d9534f,stroke:#333,stroke-width:2px,color:#fff; ``` ## RobotClient Best Practices When writing code to control the Stretch robot via `RobotClient`: 1. **Always Pull Before Reading:** The status dictionaries are not magically updated in the background. You MUST call `robot.pull_status()` at the start of your loop before reading `robot.status` or `robot.subsystem.status`. 2. **Commands are Queued:** Calling `robot.arm.move_to()` simply queues the command locally. It does nothing until you call `robot.push_command()`. 3. **Execution is Asynchronous:** `robot.push_command()` returns immediately. If you need to wait for a motion to finish before executing the next step (e.g. a simple sequence script), you must use `robot.wait_on_motion_finish(['subsystem_name'])`. 4. **End of Arm (EOA) Dynamism:** Be aware that `robot.end_of_arm` and `robot.status['end_of_arm']` are dynamic based on the tool attached. Do not hardcode a specific gripper key without checking if it exists (e.g. check for `'stretch_gripper'`). 5. **Always Cleanup:** Use `with RobotClient() as robot:` or explicitly call `robot.stop()` to ensure the connection to the server is terminated cleanly. ## Troubleshooting ### robot.startup() is returning False If you are not able to call RobotClient's `startup()` method or are getting a message, such as the snippet below, it means that the client is unable to connect to the Robot Server. ``` =============================================== StretchBodyClient: Not able to connect to Stretch Body Server. Check that server is running StretchBodyClient: Try running the server with stretch_body_server --launch =============================================== ``` You can resolve this by running `stretch_body_server --print` in a terminal to see the server logs and check for any errors. If you do not see any errors, you can run `stretch_body_server --restart` to restart it and tail the logs. Check the logs for any startup errors and resolve them. #### Common Causes of Server Start Failure **Detached or wrong gripper or end-of-arm tool** A common cause of the server failing to start is a detached gripper or end-of-arm tool. You can run `stretch_configure_tool` or (`stretch_configure_tool -d` if the server is offline) to select the right end-of-arm tool or no tool. **E-fuse reset** In very rare cases, your robot's joints may have triggered an e-fuse, which is like a circuit breaker to protect your robot's electrical components. To reset an e-fuse, turn off the robot and unplug the battery for 10 seconds, then plug it in again. You could also run `REx_actuator_control --all` to power cycle all the robot's joints while the robot is turned on. You will need to call `stretch_robot_home` to recalibrate the joints after doing this. ### move\_by or move\_to is not working If your joint's move\_by or move\_to is not responding, check that: 1. You have called `robot.push_command()` to tell the robot serverto execute queue'd commands. 2. Your joint is homed. You can home by running `stretch_robot_home` from a terminal. Warning: the robot's joints will move when you run this command! # Robot Parameters This document provides a comprehensive overview of the parameter system in the Stretch robot codebase, including its organizational structure, runtime plug-in mechanisms, and the tools available for managing parameters. It is designed to be easily accessible to both users and AI agents. ## 1. High-Level Organizational Structure The parameter system in Stretch uses a multi-layered dictionary approach, prioritized from base defaults to user-specific overrides. Parameters are resolved dynamically at runtime by `stretch4_body.core.robot_params.RobotParams`. The parameter dictionaries are loaded and overwritten in the following order (ascending priority): 1. **`robot_params_.py` (Python):** Model-specific nominal parameters (e.g., `robot_params_SE4.py`). Defines the baseline configuration for a specific robot model. 2. **`stretch_configuration_params.yaml` (YAML):** Robot-specific data (e.g., serial numbers, hardware offsets, and factory calibration data). Typically updated by factory or calibration tools. 3. **`stretch_user_params.yaml` (YAML):** User-specific overrides (e.g., custom velocity limits, contact thresholds, controller tunings). **This is the highest priority.** > \[!WARNING] **Common Trap:** A frequent issue is that a robot may have `user_params` overriding the factory params (perhaps set by another user previously), generating undesired or unexpected behavior. Always check `stretch_user_params.yaml` for active user overrides if the robot behaves unexpectedly. > \[!NOTE] **YAML vs. Python:** Python files serve as the rigid, heavily-structured "default" settings shipped by the manufacturer. YAML files act as local configuration files used to store specific calibrations (`configuration_params.yaml`) or explicit user preferences (`user_params.yaml`) without modifying the tracked source code. ### Plug-in System and Runtime Generation Stretch utilizes a highly modular plug-in architecture to dynamically generate configurations at runtime based on the physical hardware attached to the robot. * **Plug-in Managers (`SentryManager`, `SafeMotionManager`, `RoutineManager`):** The system uses various managers to dynamically load plug-ins. * `SentryManager` handles safety sentries or monitors (e.g., `self_collision_loop`) that protect the robot or monitor state. * `SafeMotionManager` manages plug-ins that restrict motions of the motors to help avoid hazards (e.g., limiting velocity/acceleration, triggering safe stops). * `RoutineManager` manages plug-ins that execute complex routines or behaviors. These plug-ins are defined in the parameters under `['controllers']`. At runtime, the respective manager reads the `py_module_name` and `py_class_name` for each enabled plug-in and instantiates them dynamically. * **End-Of-Arm (EOA) Tool Generation:** The End-Of-Arm tool configuration is entirely generated at runtime. When `robot.tool` is specified in a YAML file (e.g., `'eoa_wrist_dw4_tool_sg4'`), the `RobotParams` class looks up the corresponding tool template in `nominal_params`. It expands the parameter dictionary by pulling in the relevant joint properties (like `SE4_wrist_yaw_DW4` or `SE4_stretch_gripper_DW4`) into the active `devices` list. ## 2. Managing Parameters with CLI Tools The codebase provides two primary Python scripts to inspect and modify parameters: ### `stretch_params.py` This tool recursively traverses the resolved parameter tree and prints it to the console, importantly noting the **Origin** of each parameter (e.g., whether it came from nominal python defaults, or was overridden by `stretch_user_params.yaml`). **Using with `grep`:** Since the parameter list is extensive, `stretch_params.py` is best used alongside `grep` to quickly check the active value of a specific parameter and ensure user overrides are being respected. ```bash # Check the active value and origin of the arm's velocity stretch_params.py | grep arm | grep vel_m ``` ### `stretch_change_param.py` An interactive command-line utility used to safely modify the robot parameters. It dynamically explores the `RobotParams` tree. When a user modifies a value, the script automatically writes the override to `stretch_user_params.yaml` without destroying existing overrides in the file. ```bash # Launch the interactive parameter modification tool stretch_change_param.py ``` ## 3. Detailed Parameter Sections in `robot_params_SE4.py` The `robot_params_SE4.py` file contains the baseline configuration for the Stretch SE4 model. It is organized into several distinct logical blocks: ### 3.1. EOA Joint Templates Variables like `SE4_wrist_yaw_DW4` and `SE4_stretch_gripper_DW4` define the physical attributes of individual dynamixel/feetech servos. * **Details:** Contains `eeprom_cfg` (limits, PIDs, protections) and `motion` profiles (`default`, `fast`, `max`, `slow`). * **Relation to Structure:** These serve as building blocks. They are not directly loaded into the root parameter dictionary until an EOA tool explicitly requests them. ### 3.2. EndOfArm Defn (Tool Configurations) Dictionaries like `SE4_eoa_wrist_dw4_tool_sg4` group multiple EOA joints together into a cohesive tool. * **Details:** Defines the `wrist`, the `tool`, `stow` positions, collision management offsets, and points to the `py_class_name` and `py_module_name` for the software driver. It also enumerates the required joint templates under the `devices` key. * **Relation to Structure:** This represents the plug-in schema. When a tool is selected, `RobotParams` iterates through the `devices` key and merges the associated EOA Joint Templates (from 3.1) into the active parameter tree. ### 3.3. `nominal_params` (Root Dictionary) This is the master dictionary that gets injected into `RobotParams`. It contains base configurations for major subsystems and managers: * **Hardware Subsystems:** Configurations for hardware like `omnibase` (kinematics, wheel diameter), `arm` (gearing, homing thresholds), and generic motor templates (e.g., `hello-motor-omni-2` or stepper motors). * **Loops and Rates:** Settings for control loops like `line_sensor_loop` and `end_of_arm_loop` (e.g., `loop_rate_Hz`). * **Plug-in Configurations:** Defines the parameter profiles for the various plug-ins managed by `SentryManager`, `SafeMotionManager`, and `RoutineManager` (e.g., settings for `routine_docking` or `sentry_self_collision`). * **Relation to Structure:** It acts as the baseline priority hierarchy. It holds the `supported_eoa` lists and templates that the runtime generation system uses to build the final dictionary. ## 4. Organizational Structure Visualization The following diagram illustrates how parameters flow into the resolved parameter tree and how plug-ins are generated. ```mermaid graph TD %% Parameter Sources RP_PY["robot_params_SE4.py
(Nominal Defaults)"]:::python CONF_YAML["stretch_configuration_params.yaml
(Factory Calibrations)"]:::yaml USER_YAML["stretch_user_params.yaml
(User Overrides)"]:::yaml %% Core System R_PARAMS{"RobotParams
(Runtime Dictionary)"}:::core %% Injection RP_PY -->|1. Base| R_PARAMS CONF_YAML -->|2. Overrides| R_PARAMS USER_YAML -->|3. Highest Priority Overrides| R_PARAMS %% Dynamic Generation TOOL_YAML["robot.tool specified
(e.g., 'eoa_wrist_dw4_tool_sg4')"] -.->|Triggers| EOA_GEN subgraph "Dynamic Runtime Generation" EOA_GEN["EOA Tool Generation"]:::gen PLUG_GEN["Plug-in Generation
(Sentry, SafeMotion, Routine)"]:::gen end R_PARAMS --> EOA_GEN R_PARAMS --> PLUG_GEN %% Class/System usage EOA_GEN -->|Injects Tool Devices| R_PARAMS PLUG_GEN --> |Loads py_module_name| MANAGERS["Plug-in Managers"] classDef python fill:#4f81c7,stroke:#333,stroke-width:2px,color:#fff; classDef yaml fill:#e88d3e,stroke:#333,stroke-width:2px,color:#fff; classDef core fill:#57a661,stroke:#333,stroke-width:2px,color:#fff; classDef gen fill:#a859b3,stroke:#333,stroke-width:2px,color:#fff; ``` ## Summary for AI Agents When interacting with the Stretch parameter system: 1. **Never edit `robot_params_SE4.py` to change a local behavior.** Always instruct the user to use `stretch_change_param.py` or modify `stretch_user_params.yaml`. 2. **Dynamic Resolution:** Be aware that the dictionary structure changes based on the configured tool. If a joint isn't physically attached (defined by `robot.tool`), its parameters will not exist in the resolved tree. 3. **Debugging:** Use `stretch_params.py` to trace where a parameter is coming from if you suspect a YAML file isn't applying correctly. # Server Behaviors The `stretch4_body/behavior` directory contains the logic for advanced autonomous functionality and safety monitoring within the `RobotServer`. It is structured into three primary types of behaviors: **Sentries**, **Routines**, and **Safe Motions**. These behaviors operate within a dynamic plug-in architecture, allowing new safety checks or autonomous sequences to be added without modifying the core server loop. *** ## 1. High-Level Roles ### Sentries **Role:** Continuous background monitoring and limit enforcement. Sentries constantly watch the robot's state (telemetry, odometry, current draw, CPU temperature) and enforce dynamic limits on the robot's capabilities. For example, the `sentry_limit_vel_on_pose` reduces the maximum allowed base velocity if the arm is extended far out, preventing tipping. The `sentry_self_collision` monitors for self-collision and updates safety limits. Sentries generally do not create motion commands; they enforce the bounds within which motion commands must operate. ### Safe Motions **Role:** Final-check command modification and hazard avoidance. Safe motions act as a final layer of defense. They intercept the pending commands immediately before they are sent to the hardware. If a command would cause a hazard (e.g., `safe_motion_overtilt_avoid`), the safe motion plug-in can actively overwrite the setpoint (e.g., zero out the velocity) or trigger a system-wide safe stop. ### Routines **Role:** Predefined, autonomous macro sequences. Routines handle complex, multi-step actions like homing (`routine_homing`), stowing the arm (`routine_stow`), or docking to the charger (`routine_blind_dock`). When a routine is active, it takes over control of the robot, rejecting motion commands from external clients until the routine finishes or is explicitly canceled. *** ## 2. Plug-in Architecture and Management The `RobotServer` does not hardcode which behaviors run. Instead, it relies on three manager classes (`SentryManager`, `SafeMotionManager`, and `RoutineManager`). During startup, each manager reads the `controllers` list from the active parameter configuration (`robot_params`). It then uses `importlib` to dynamically import the corresponding `py_module_name` and instantiate the `py_class_name`. ```yaml # Example from YAML configuration sentry_cpu_temp: py_module_name: stretch4_body.behavior.sentries.sentry_cpu_temp py_class_name: SentryCPUTemp enabled: 1 ``` If a behavior is marked with `enabled: 1`, it is instantiated and passed a reference to the `Robot` instance. This allows developers to easily create and inject custom sentries or routines simply by updating the YAML configuration. *** ## 3. Flow of Control within the Server Loop Understanding the differences between these three behaviors requires looking at exactly *when* they execute inside the `RobotServer`'s 100Hz control loop: 1. **Pull Status:** The server asynchronously pulls the latest state from the hardware. 2. **Step Sentries (`sentry_manager.step()`):** Sentries run immediately after new state data arrives. They analyze the state and update internal limits (like `max_vel` or `max_accel`). They *do not* see incoming commands. 3. **Ingest Client Commands:** The server receives new commands from the ZMQ network (e.g., user scripts, ROS). 4. **Step Routines (`routine_manager.step()`):** If a Routine is active, it overrides step 3. The routine takes over generating the motion commands for this cycle and external client commands are rejected. 5. **Step Safe Motions (`safe_motion_manager.step()`):** Right before the commands are sent to the motors, Safe Motions analyze the pending trajectory. If the command violates a safety condition, the Safe Motion will rewrite the command (e.g., override the target velocity to 0) or trigger a safe stop. 6. **Push Command:** The server pushes the finalized commands to the hardware via serial/ZMQ. 7. **Publish Status:** The server broadcasts the updated state and results to listening clients. ### Summary of Differences * **Sentries** run *early* in the loop. They analyze state and set bounds. * **Routines** run in the *middle* of the loop. They *generate* commands and block the client. * **Safe Motions** run at the *end* of the loop. They analyze and *overwrite/veto* commands just before execution. # Self-Collision The self-collision system is a critical safety behavior implemented as a Sentry (`SentrySelfCollision`). It continuously monitors the robot's joint states to prevent the arm, lift, and end-of-arm tools from colliding with the robot's own body (e.g., the mast, base, or head). ## 1. High-Level Architecture The self-collision system is organized into several distinct layers: * **MuJoCo Physics Engine:** At the core, the system uses the MuJoCo physics engine to perform fast forward kinematics and collision checking. It dynamically loads the robot's exact URDF and collision meshes using the `stretch4_urdf` package, matching the specific robot model, batch, and attached end-of-arm tool. * **Multiprocessing (`SelfCollisionLoop`):** Collision checking is computationally intensive. To prevent it from blocking the high-frequency (100Hz) main control loop, the `SelfCollisionLoop` spawns a dedicated background worker process. The main server loop sends the current joint configuration into an asynchronous command queue, and the background worker continuously processes these states, pushing the resulting collision status back into a status queue. * **Motion Clipping Response:** When a collision is imminent, MuJoCo calculates the "collision direction"—the gradient indicating which way the joint must move to escape the collision. The `SentrySelfCollision` uses this to set `pos` or `neg` collision flags for the affected joints. These flags are fed into each subsystem's `step_collision_avoidance` method (e.g., in the `Arm` or `Lift` classes), which actively zeroes out any commanded velocity that would push the joint further into the collision, stopping the joint while allowing the user to back away safely. ## 2. Configuration Parameters The system is highly configurable via the `self_collision_mujoco` dictionary in the robot's parameter files (e.g., `stretch4_body/robot/robot_params_SE4.py`). These parameters dictate what is checked and how sensitive the system is: * **`k_brake_distance`:** The system doesn't just check the robot's exact current position; it checks a "virtual" position slightly ahead of the robot based on its current velocity and braking capabilities. The `k_brake_distance` parameter (e.g., `{'lift': 1.1, 'arm': 1.1}`) acts as a multiplier. A value of 1.1 means the system pads the joint's position by 110% of its required braking distance, ensuring the robot stops *before* making contact. * **`ignore_links`:** A list of URDF link names (e.g., `wheel_0_link`, `camera_center_link`) that will be completely ignored by the collision engine. Their collision properties are disabled internally to save computation time and ignore benign hardware. * **`exclusions`:** A list of link pairs (e.g., `["head_link", "lift_link"]`). These specific pairs are allowed to intersect or touch without triggering a collision event. This is necessary because many adjacent links in a URDF naturally overlap at their joints during normal operation. ## 3. Flow of Control The following diagram illustrates how data flows through the system, from the main control loop down to the physics engine, and back up to stop the joint motion. ```mermaid graph TD ServerControlLoop --> SentryManager SentryManager --> SentrySelfCollision SentrySelfCollision --> SendJointConfigurationQueue SendJointConfigurationQueue --> SelfCollisionLoopWorker SelfCollisionLoopWorker --> MujocoPhysicsEngine MujocoPhysicsEngine --> ComputeCollisionsAsync ComputeCollisionsAsync --> ReturnCollisionStatus ReturnCollisionStatus --> SelfCollisionLoopWorker SelfCollisionLoopWorker --> ReadLatestStatusQueue ReadLatestStatusQueue --> SentrySelfCollision SentrySelfCollision --> IdentifyResolveDirection IdentifyResolveDirection --> SetCollisionStopFlags SetCollisionStopFlags --> Subsystems Subsystems --> ApplyBrakeAndStopMotion ``` # Hardware Subsystems This document provides an overview of the core hardware subsystems that make up the Stretch robot. It details how these subsystems are instantiated, managed, and configured, along with key parameters, status fields, and API calls for each. ## Subsystem Instantiation and Management Subsystems are managed collectively by the `RobotCore` base class (and its descendants, like `Robot` or the `RobotServer`). During startup, `RobotCore` reads the `subsystems` list from the active parameter configuration (`robot_params`). It then instantiates the corresponding Python objects and stores them in its `self.subsystems` dictionary. * `RobotCore` handles the base instantiation of `arm`, `lift`, `omnibase`, and `power_periph`. * The `end_of_arm` and `line_sensor` subsystems are handled specifically by `Robot` or the server, as they rely on additional configuration (such as `robot.tool` to determine the active class) or separate multiprocess workers. During the control loop, `RobotCore` aggregates data by calling `pull_status` and `push_command` on every active subsystem in the list. ### Disabling a Subsystem If a hardware subsystem is physically removed or malfunctioning, you can prevent the software from attempting to instantiate and communicate with it by overriding the `subsystems` list in your `stretch_user_yaml`. For example, to run the robot without the `end_of_arm` subsystem, you would add the following to your user YAML: ```yaml robot: subsystems: - arm - lift - omnibase - power_periph ``` *(By omitting `end_of_arm`, the software will bypass its initialization entirely.)* > **Note on Cameras:** The `cameras` directory is included within the `subsystems` folder structure, but cameras are not actively managed by the `RobotServer` or `RobotCore` control loop. The directory exists primarily to manage camera configuration parameters (like framerates, exposure limits, and intrinsic calibrations) within the unified parameter system. *** ## Subsystems Overview ### 1. Arm (`stretch4_body.subsystem.arm.Arm`) Controls the horizontal prismatic joint (in/out extension) of the robot. It translates motor rotation into linear motion using a chain and sprocket mechanism. ### 2. Lift (`stretch4_body.subsystem.lift.Lift`) Controls the vertical prismatic joint (up/down) of the robot, driven by a belt. It supports payload-based gravity feedforward compensation to safely lift and hold various end-of-arm tools. ### 3. OmniBase (`stretch4_body.subsystem.omnibase.OmniBase`) Controls the differential/omnidirectional base using three stepper motor wheels. It provides real-time odometry and handles blended translational and rotational commands, as well as guarded collision detection. ### 4. PowerPeriph (`stretch4_body.subsystem.power_periph.PowerPeriphBase`) Serves as the central hub for the robot's power state, IMU data, battery monitoring, and peripheral GPIO (LEDs, buzzers, fans, and runstop management). ### 5. EndOfArm (`stretch4_body.subsystem.end_of_arm.EndOfArm`) A modular tool system dynamically instantiated based on the active `robot.tool` configuration (e.g., dexterous wrists, parallel grippers). It typically runs its hardware communication inside a separate high-frequency background worker process to decouple from the main server loop. ### 6. LineSensor (`stretch4_body.subsystem.line_sensor.LineSensorLoop`) Interfaces with the downward-facing Pixart hardware sensors used for line following or drop-off detection. Similar to the EndOfArm, it runs in a dedicated background worker process for high-speed, non-blocking serial acquisition. *** ## Class Architecture The following diagram illustrates how `RobotCore` manages the subsystems, and how specific hardware subsystems inherit from shared base classes (like `PrismaticJoint`). ```mermaid classDiagram class RobotCore { +dict subsystems } class Robot class Arm class Lift class OmniBase class PowerPeriph class EndOfArm class LineSensor class PrismaticJoint RobotCore <|-- Robot RobotCore *-- Arm RobotCore *-- Lift RobotCore *-- OmniBase RobotCore *-- PowerPeriph RobotCore *-- EndOfArm RobotCore *-- LineSensor PrismaticJoint <|-- Arm PrismaticJoint <|-- Lift ``` # End-Of-Arm (EOA) The `end_of_arm` subsystem is responsible for controlling the various tools and joints attached to the distal end of the robot's arm, such as the wrist (yaw, pitch, roll) and the gripper. It is designed to be highly modular, supporting a wide range of custom and factory tool attachments. ## Modular Architecture and Tool Determination The modularity of the end-of-arm system is heavily reliant on the `RobotParams` system. At runtime, the `EndOfArmLoop` reads the user's YAML configuration to determine exactly which tool is installed on the robot. It does this by looking up the `eoa_name` property and then instantiating the class dynamically via Python's `importlib`: ```python rp = RobotParams._robot_params eoa_name = RobotParams.eoa_name module_name = rp[eoa_name]['py_module_name'] class_name = rp[eoa_name]['py_class_name'] eoa = getattr(importlib.import_module(module_name), class_name)() ``` This architecture allows users to define custom end-of-arm tools by simply creating a Python class that inherits from `EndOfArm` and specifying its module/class name in their `stretch_user_params.yaml`. Note that changing the active tool is done by setting ``` robot: tool: eoa_name ``` in the user yaml. ### Actuation Component: Feetech Servos All actuation at the end-of-arm is driven by **Feetech** smart serial servos. The base `EndOfArm` class inherits from `FeetechSMChain`, meaning that the end-of-arm is treated as a daisy-chained serial bus of Feetech motors. Each individual joint (like `wrist_pitch`, `wrist_roll`, or `stretch_gripper`) corresponds to a `FeetechSMHello` instance that sits on this chain. ## Class Hierarchy The following diagram illustrates the class hierarchy and ownership from the low-level Feetech chain up to the client application: ```mermaid classDiagram FeetechSMChain <|-- EndOfArm EndOfArm <|-- EOA_Wrist_DW4_Tool_NIL EndOfArm <|-- EOA_Wrist_DW4_Tool_SG4 EndOfArm <|-- EOA_Wrist_DW4_Tool_PG4 EndOfArmLoop *-- EndOfArm : Instantiated in Worker Process RobotServer *-- EndOfArmLoop : Manages Lifecycle RobotClient ..> RobotServer : Communicates over ZMQ ``` ## Multiprocessing and Control Rate Communicating with multiple serial servos on a shared bus can cause I/O latency. To maintain a solid **50Hz** control and status update rate, the `end_of_arm` subsystem relies on Python multiprocessing. The `EndOfArmLoop` spins up a separate background `Process` that runs the `end_of_arm_loop_worker`. Inside this worker, a high-frequency loop constantly calls `eoa.pull_status()` to read from the servos over the serial port. Because this happens in an isolated process, any serial port blocking or I/O delays do not interrupt the main `RobotServer` application loop. The server manages this by providing two `CircularMultiprocessingQueue` objects: * `q_status`: The worker drops fresh state dictionaries here; the main process reads them. * `q_cmd`: The main process drops RPC commands (like `move_to`) here; the worker executes them against the `EndOfArm` instance. ## Control Flow State Diagram The flow of commands and data moves bidirectionally through the multiprocess queues and ZMQ transport. ```mermaid flowchart TD A[Feetech Hardware Servos] <-->|UART and Serial| B[FeetechSMHello] B <--> C[EndOfArm Instance] subgraph Background Worker Process C -->|status polling| D[end_of_arm_loop_worker] D -->|method execution| C end D -->|status dict| E[q_status CircularMultiprocessingQueue] I[q_cmd CircularMultiprocessingQueue] -->|command tuple| D subgraph Main Process E -->|pull_status| F[EndOfArmLoop] F -->|push_command| I F -->|Aggregated Status| G[RobotServer] G -->|RPC and Commands| F end G -->|ZMQ Pub and Sub| H[RobotClient] H -->|ZMQ Req and Rep| G ``` ## Available Tools There are several command-line tools available for interacting with the `end_of_arm` subsystem, testing joints, and calibrating the system, including: * `stretch_dex_wrist_home.py`: Homes the full 3-DOF dexterous wrist. * `stretch_dex_wrist_jog.py`: Interactively jog the yaw, pitch, and roll axes of the wrist. * `stretch_gripper_home.py`: Homes the end-of-arm gripper. * `stretch_gripper_jog.py`: Interactively open and close the gripper. * `stretch_wrist_yaw_home.py`, `stretch_wrist_pitch_home.py`, `stretch_wrist_roll_home.py`: Individually home specific wrist joints. * `REx_feetech_backlash_measure.py`: Factory tool designed to measure mechanical backlash in the Feetech servos. * `REx_feetech_id_change.py`: Factory tool to change the ID of a Feetech servo. * `REx_feetech_id_scan.py`: Factory tool to scan the serial bus for active Feetech IDs. * `REx_feetech_jog.py`: Factory tool to interactively jog individual Feetech servos. * `REx_feetech_reboot.py`: Factory tool to soft-reboot a Feetech servo. * `REx_feetech_set_baud.py`: Factory tool to change the baud rate of a Feetech servo. # Cameras ## Preview from the Cameras from the command line You can use the following tool to display the camera feeds in opencv, rerun and store them locally anywhere from the command line: [stretch\_camera\_show](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/tools/stretch_camera_show.py). Run `stretch_camera_show --help` for all the available options. Examples: ``` stretch_camera_show # Display the left, right and center head cameras in rerun (Default) stretch_camera_show --left # Display the left head camera only in rerun stretch_camera_show --left --opencv # Display the left head camera only in opencv stretch_camera_show --left --no-rotate # Display the left head camera without rotation in rerun stretch_camera_show --gripper # Display the gripper camera feeds and the point cloud from the stereo depth in rerun stretch_camera_show --left --recording_directory ./recordings # Store the left head camera to disk as a video (and display in rerun) stretch_camera_show --left --recording_directory ./recordings --record_format .png # Store one .png per frame instead stretch_camera_show --recording_directory ./recordings --recording_chunk_seconds 60 # Split the videos into one-minute files ``` Recording turns the display off, because drawing the imagery costs frames: rerun cannot keep up with the head cameras and backpressures the pipeline that feeds it, which drops the synced left/right pair from 30 fps to around 11 a few minutes into a recording, and has been seen to stall capture altogether. Pass `--rerun` or `--opencv` alongside `--recording_directory` to display it anyway and accept the dropped frames. Recordings are written to `RECORDING_DIRECTORY///`. `--record_format` takes `.mp4` (the default), which writes video per camera at the camera's configured frame rate, or `.png` and `.jpg`, which write one file per frame, named after the frame's timestamp. ### Video recordings Video is encoded as HEVC (H.265), on the GPU when the machine has one that can encode it and on the CPU otherwise. This needs `ffmpeg` on the `PATH`; without it recording falls back to OpenCV's mpeg4 encoder, which produces files roughly ten times larger. A video recording is split into chunks of `--recording_chunk_seconds` (300 by default), named `video_0000.mp4`, `video_0001.mp4` and so on. Chunking matters because an mp4 that never gets closed has no index and will not play back at all, so without it an interrupted recording is lost entirely rather than just its last chunk. Pass `--recording_chunk_seconds 0` to write a single `video.mp4` instead. Chunks rotate on elapsed real time, not on the length of the recorded video. The two differ whenever a camera delivers below its configured frame rate: the video's own timeline then advances slower than the clock, so a chunk measured in recorded seconds would take proportionally longer to fill. The gap is small when nothing is competing for the frames, but it is wide enough to matter with a display attached, where a "300 second" chunk took over half an hour to close. Note that this also means the video plays back faster than real time when frames were dropped, because it is written at the camera's configured rate. Long recordings are practical at these settings. Measured on the 12MP center camera, HEVC writes about 10 Mbps where the mpeg4 encoder wrote 112 Mbps, so recording the left, right and center cameras together costs roughly 4.5 GB an hour instead of 63 GB. A static scene compresses much further - the three head cameras together measured 4.1 Mbps, about 1.8 GB an hour - so budget by how much the scene moves. With the display off, the head cameras hold their configured rates for the whole recording: the left and right cameras measured 30.0, 29.9 and 29.9 fps over successive chunks, and the center camera 96% of its configured 10 fps. To play the chunks of a recording as one video: ```bash printf "file '%s'\n" video_*.mp4 > chunks.txt ffmpeg -f concat -safe 0 -i chunks.txt -c copy whole_recording.mp4 ``` ## Using the Cameras with Python ### Stream-API The camera subsystem exposes Python [generators](https://book.pythontips.com/en/latest/generators.html) for streaming frames from the cameras. These generators yield [ImageFrame](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/models/image_frame.py) or [SyncedImageFrame](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/models/image_frame.py) objects. [ImageFrame](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/models/image_frame.py) is a container for a single camera's image and metadata. Access the image using the `image` attribute, timestamp with `timestamp`, and [AI model results](#custom-ai-models-eg-rtmo-pose-estimation) with `ai_model_results`. [SyncedImageFrame](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/models/image_frame.py) is a container for multiple cameras' images and metadata. You can access the individual camera frames using the `left`, `right`, and `center` attributes of the `SyncedImageFrame` object, which are `ImageFrame` objects. Using one camera yields an ImageFrame. ```python from stretch4_body.subsystem.cameras import * for image_frame in stream_left_camera(): if image_frame is None: print("No frame returned") continue print(f"Got image: {image_frame.image.shape=}, {image_frame.timestamp=}") ``` Using multiple cameras from the head cameras simultaneously yields a `SyncedImageFrame`. ```python from stretch4_body.subsystem.cameras import * for image_frame in stream_left_right_camera(): if image_frame is None: print("No frame returned") continue print(f"Got left image: {image_frame.left.image.shape=}, {image_frame.left.timestamp=}") print(f"Got right image: {image_frame.right.image.shape=}, {image_frame.right.timestamp=}") ``` Stream from the gripper RGBD camera. This yields a SyncedImageFrame and populates the pointcloud field. ```python from stretch4_body.subsystem.cameras import * for image_frame in stream_gripper_camera(): if image_frame is None: print("No frame returned") continue print(f"Got left image: {image_frame.left.image.shape=}, {image_frame.left.timestamp=}") if image_frame.pointcloud is not None: print(f"Got pointcloud image: {image_frame.pointcloud.shape=}") ``` ### Compressed streams Every stream function has a `*_compressed()` twin (`stream_left_camera_compressed()`, `stream_left_right_camera_compressed()`, `stream_gripper_camera_compressed()`, and so on). They open the same cameras, but the camera MJPEG-encodes frames on-chip, so a head frame costs a couple hundred kilobytes instead of \~7 MB. Use them whenever frames have to leave the process; the head cameras are 1920x1200 and the center camera is 12MP, so raw pixels saturate a transport long before the sensors do. The default `is_run_pipeline=True` decodes frames for you, so `image` is ordinary BGR and the compressed variants are a drop-in replacement: ```python from stretch4_body.subsystem.cameras import * for image_frame in stream_left_right_camera_compressed(): print(f"Got left image: {image_frame.left.image.shape=}") ``` Pass `is_run_pipeline=False` to keep the JPEG bitstream instead. `ImageFrame.is_compressed()` then returns True, `image` is a 1-D buffer, and `uncompress()` decodes it when you actually need pixels. This is what the ROS 2 camera node does: it forwards the bitstream to the compressed topics without ever decoding it. Camera configuration (resolution, frame rate, JPEG quality, rotation, exposure) lives in `CAMERA_CONFIGS` in [rgb\_camera\_config.py](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/models/rgb_camera_config.py) and is read through `RGBCameras..config`. A compressed variant reuses its base camera's entry with compression forced on, and `RGBCameras..base` gets you back to the camera it derives from. ## Preview RGBD Cameras from the command line Run `stretch_rgbd_show --help` for all the available options. Examples: ``` stretch_rgbd_show --left_right # Display the left, right head cameras only stretch_rgbd_show --left_right_center # Display the left, right and center head cameras stretch_rgbd_show --left # Display the left head camera only stretch_rgbd_show --right # Display the right head camera only stretch_rgbd_show --center # Display the center head camera only ``` If you have ROS2 launch files running for cameras or lidars, you can use the `--use_ros_for_cameras` or `--use_ros_for_lidar` flags to use the ROS2 data instead of direct python-api access to those sensors. ``` stretch_rgbd_show --left --use_ros_for_cameras --use_ros_for_lidar ``` ## Using Emulated RGBD with Python You can capture RGBD data from the head cameras using the following scripts: ```python from stretch4_body.subsystem.cameras import * for frame in stream_left_rgbd(): if frame is None: print("No frame returned") continue print(f"""Got a point cloud using the left camera and both lidar. Number of points: {frame.pointcloud.shape[0]} Depth size: {frame.depth_image.shape}""") for frame in stream_left_right_center_rgbd(): if frame is None: print("No frame returned") continue print(f"""Got a point cloud using the left, right and center cameras and both lidar. Left: Number of points: {frame.left.pointcloud.shape[0]} Depth size: {frame.left.depth_image.shape} Right: Number of points: {frame.right.pointcloud.shape[0]} Depth size: {frame.right.depth_image.shape} Center: Number of points: {frame.center.pointcloud.shape[0] if frame.center is not None else 'N/A'} Depth size: {frame.center.depth_image.shape if frame.center is not None else 'N/A'}""") ``` ## Custom AI Models (e.g. RTMO Pose Estimation) You can pass custom AI models to the camera pipeline by wrapping them in an `AIModelWrapper` instance. The pipeline will automatically route imagery to your model. The results are packed in the `image_frame.ai_model_results` field. Here is an example wrapping an RTMO model: First install ``` git clone https://github.com/hello-robot/stretch4_human_pose_estimation.git pip install ./stretch4_human_pose_estimation human_pose_estimation_install_dependencies human_pose_estimation_setup_models human_pose_estimation_setup_models --size m ``` Then run this example: ```python import cv2 import numpy as np from stretch4_body.subsystem.cameras import stream_left_camera from stretch4_body.subsystem.cameras.detectors.detector_ai_models import AIModelWrapper from stretch4_human_pose_estimation.rtmo import RTMOPipeline class RTMOWrapper(AIModelWrapper): def __init__(self): self.model = self.init_model() def name(self) -> str: return "RTMO Pose" def init_model(self): return RTMOPipeline(size="m", device="AUTO") def run_model(self, img: np.ndarray, conf_threshold: float = 0.5): return self.model.predict(img, conf_threshold=conf_threshold) def visualize_results(self, img: np.ndarray, result_from_run_model) -> np.ndarray: return self.model.visualize(img, result_from_run_model, kpt_thr=0.3, style="cvpr") @staticmethod def get_joint(person_res: dict, joint: BodyJoint) -> tuple[float, float]: """Returns the (x, y) coordinates of the specified joint for a single person.""" kpts = person_res.get("keypoints", []) if joint < len(kpts): return float(kpts[joint][0]), float(kpts[joint][1]) return 0.0, 0.0 @staticmethod def get_joint_score(person_res: dict, joint: BodyJoint) -> float: """Returns the confidence score of the specified joint for a single person.""" kpts = person_res.get("keypoints", []) if joint < len(kpts): return float(kpts[joint][2]) return 0.0 # Instantiate the custom model wrapper rtmo = RTMOWrapper() # Start the left camera stream passing the AI model to the pipeline for image_frame in stream_left_camera(ai_models_to_use=[rtmo]): if image_frame is None: continue results = image_frame.ai_model_results[0] annotated_image = rtmo.visualize_results(image_frame.image.copy(), results) cv2.namedWindow(rtmo.name(), cv2.WINDOW_NORMAL) cv2.imshow(rtmo.name(), annotated_image) if cv2.waitKey(1) == ord('q'): break ``` Here is another example wrapping a YOLOX object detection model (from rtmlib) configured to identify general COCO objects like cups, tables, desks, and chairs. First install rtmlib: ``` pip install rtmlib pip install openvino # To utilize the NPU on Stretch's NUC pip install onnxruntime-openvino ``` The model weights will be automatically downloaded by rtmlib on the first run. ```python import cv2 import numpy as np from stretch4_body.subsystem.cameras import * from stretch4_body.subsystem.cameras.detectors.detector_ai_models import AIModelWrapper # RTMLIB imports from rtmlib import YOLOX from rtmlib.tools.base import RTMLIB_SETTINGS # Configure rtmlib to utilize the Intel GPU or NPU via OpenVINO Execution Provider RTMLIB_SETTINGS['onnxruntime']['npu'] = ('OpenVINOExecutionProvider', {'device_type': 'NPU'}) RTMLIB_SETTINGS['onnxruntime']['gpu'] = ('OpenVINOExecutionProvider', {'device_type': 'GPU'}) class YOLOXWrapper(AIModelWrapper): def __init__(self): self.coco_classes = [ "person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", "microwave", "oven", "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush", "monitor", "can", "bottle", "tennis ball", "ball", "tape", "mug", "remote control", "desk" ] self.model = self.init_model() def name(self) -> str: return "YOLOX Object Detection" def init_model(self): device = 'gpu' # npu, cpu, gpu backend = 'onnxruntime' # We use onnxruntime to route to the OpenVINO NPU Provider # Provide a URL to a COCO yolox model. rtmlib will cache it automatically. url = 'https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_tiny.onnx' return YOLOX(url, mode='multiclass',model_input_size=(416, 416), backend=backend, device=device) def run_model(self, img: np.ndarray): return self.model(img) def visualize_results(self, img: np.ndarray, result_from_run_model) -> np.ndarray: bboxes, cls_inds = result_from_run_model h, w = img.shape[:2] img_area = h * w for bbox, cls_id in zip(bboxes, cls_inds): x1, y1, x2, y2 = map(int, bbox[:4]) area = (x2 - x1) * (y2 - y1) # Color based on relative size (Green for small, Red for large) ratio = min(1.0, max(0.0, np.sqrt(area / img_area))) hue = int((1.0 - ratio) * 120) hsv = np.uint8([[[hue, 255, 255]]]) color = tuple(map(int, cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)[0][0])) cv2.rectangle(img, (x1, y1), (x2, y2), color, 3) label = self.coco_classes[int(cls_id)] if int(cls_id) < len(self.coco_classes) else str(cls_id) label = f"{label.capitalize()}" # Render a solid background for the text font_scale = 0.8 thickness = 2 (t_w, t_h), baseline = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, font_scale, thickness) y_label_bg = max(y1, t_h + 10) cv2.rectangle(img, (x1, y_label_bg - t_h - 10), (x1 + t_w + 10, y_label_bg), color, -1) # Draw white text over the solid background (anti-aliased) cv2.putText(img, label, (x1 + 5, y_label_bg - 5), cv2.FONT_HERSHEY_SIMPLEX, font_scale, (255, 255, 255), thickness, cv2.LINE_AA) return img # Instantiate the custom model wrapper yolox_model = YOLOXWrapper() # Start the left camera stream passing the AI model to the pipeline # for image_frame in stream_center_camera(ai_models_to_use=[yolox_model]): # if image_frame is None: # continue # results = image_frame.ai_model_results[0] # annotated_image = yolox_model.visualize_results(image_frame.image.copy(), results) # cv2.namedWindow(yolox_model.name(), cv2.WINDOW_NORMAL) # cv2.imshow(yolox_model.name(), annotated_image) # if cv2.waitKey(1) == ord('q'): # break for synced_frame in stream_gripper_camera(ai_models_to_use=[yolox_model]): if synced_frame is None: continue image_frame =synced_frame.left results = image_frame.ai_model_results[0] annotated_image = yolox_model.visualize_results(image_frame.image.copy(), results) cv2.namedWindow(yolox_model.name(), cv2.WINDOW_NORMAL) cv2.imshow(yolox_model.name(), annotated_image) if cv2.waitKey(1) == ord('q'): break ``` ## Calibrating your cameras Cameras are calibrated in the factory and the calibration files are stored in the robot's home directory, under `$HELLO_FLEET_PATH/$HELLO_FLEET_ID/calibration_cameras/`. If you need to re-calibrate your cameras, you can use the following tools. First focus your camera lens using [REx\_camera\_focus](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/tools/factory/REx_camera_focus.py). Then calibrate the camera intrinsics and extrinsics using [REx\_camera\_calibrate](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/tools/factory/REx_camera_calibrate.py). The calibration process does the following: 1. Calibrates the camera intrinsics using [calibrate\_camera\_intrinsics](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/calibrate_intrinsics.py). This saves the calibration yaml file that contains `head_center`, `head_left`, and `head_right` keys with the K and D matrices, along with other information, at `$HELLO_FLEET_PATH/$HELLO_FLEET_ID/calibration_cameras/calibration_rgb_head_camera.yaml` and a few other yaml files for ROS2 to work correctly. 2. Verifies the camera intrinsics using [camera\_intrinsics\_validate\_l2\_distance](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/camera_intrinsics_validate_l2_distance.py). This uses pre-tape-measured values to verify the camera intrinsics are correct. The values are expected to vary a little across robots, but it's a good "sanity check". 3. Calibrates the camera-camera extrinsics using [calibrate\_extrinsics\_cameras](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/calibrate_extrinsics_cameras.py). This saves the calibration transforms as a yaml file containing `left_to_center` and `right_to_center` keys at `$HELLO_FLEET_PATH/$HELLO_FLEET_ID/calibration_cameras/camera_extrinsics.yaml` and the urdf calibration file at `$HELLO_FLEET_PATH/$HELLO_FLEET_ID/calibration_values.yaml`. 4. Calibrates the camera-lidar extrinsics using [calibrate\_extrinsics\_lidars](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/calibrate_extrinsics_lidars.py). This appends the camera-lidar extrinsics `transform_right_lidar_to_head_center` key to the `$HELLO_FLEET_PATH/$HELLO_FLEET_ID/calibration_cameras/camera_extrinsics.yaml` file and the urdf calibration file at `$HELLO_FLEET_PATH/$HELLO_FLEET_ID/calibration_values.yaml`. These values are used to estimate the distance to ArUco markers of known size using [detector\_aruco.py](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/detectors/detector_aruco.py) that uses `cv2.solve_pnp` or `cv2.fisheye.solve_pnp` depending on the lens type. These values are also used by [emulated\_rgbd.py](https://github.com/hello-robot/stretch4_body/blob/main/stretch4_body/subsystem/cameras/emulated_rgbd.py) to create an colored point clouds and depth images using the left and right lidars, and each head camera using `cv2.projectPoints` or `cv2.fisheye.projectPoints` depending on the lens type. This also requires the lidar extrinsics to be calibrated using . ### Calibration Controls & Offline Replay During execution of the calibration tools (`REx_camera_calibrate` for camera-lidar or camera intrinsics), progression and commands can be input using either the gamepad controller or terminal keyboard inputs: * **Gamepad Control**: Tap `X` to capture a frame / unpause automatic movement, and hold `X` (for 3-4 seconds) to save the calibration to disk. * **Keyboard Control**: Type `x` + Enter to capture/unpause, type `s` + Enter to save/exit, and type `q` + Enter to quit without saving. This allows running calibration without a gamepad connected. For camera-lidar calibration, you can also replay a previous session offline: ```bash REx_camera_calibrate --extrinsics_lidar --replay_last ``` This offline replay automatically bypasses the gamepad pauses and runs through all poses automatically without requiring user input. # Line Sensors Six PixArt J3 sensors ring the base, each pointing down and forward, each reporting 320 range bins at \~30 Hz. Together they see the floor immediately around the robot: obstacles, and cliffs. The firmware streams newline-delimited JSON over a USB CDC port (`/dev/hello-pixart-j3`). A background process decodes it; the robot server publishes it; clients consume it over ZMQ. ## The status codes come first A bin does not always carry a distance. The chip has two status codes: | raw | metres | meaning | | -------- | ------ | ----------------------------------------------------------------------------------------------------------- | | `0xFF80` | 5.11 | **No detection at all.** No reflection came back. Ambiguous: a dark floor, a lighting condition, or a void. | | `0xFE80` | 5.09 | **Detected, but beyond the range limit.** Something returned from further than the sensor can measure. | Because the sensors point *down*, 5.09 is the stronger cliff evidence: the beam travelled past where the floor should have been and found something far away. These are classified **once**, at decode, in `line_sensor/protocol.py`, before any conversion or correction: * `ranges[bin]` is set to `NaN` wherever the bin is not a distance; * `codes[bin]` carries the identity (`CODE_VALID`, `CODE_BEYOND_LIMIT`, `CODE_NO_RETURN`, `CODE_OTHER_INVALID`). ## Status schema ### `health` — subsystem-wide | field | meaning | | --------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | `streaming` | decoding is running. **False means no cliff detection.** | | `port_open` | the serial port is open. False during a dropout. | | `disabled_sensors` | names switched off at runtime, by choice | | `sensors_dead` | names not reporting, by fault. Empty during normal operation. | | `rate_hz` | whole-frame rate, \~30 Hz | | `reader_restarts` | serial recoveries so far. A climbing number means a flaky cable. | | `decode_errors`, `frame_advance_err`, `frame_not_full_err`, `not_six_sensors_err` | counters; all should stay flat | `disabled` and `dead` are deliberately different: one is a choice, the other is a fault. A disabled sensor never appears in `sensors_dead`. ### `sensor_0` … `sensor_5` — per sensor | field | meaning | | ------------------------------------- | -------------------------------------------------------------- | | `ranges` | float64\[320], metres, `NaN` at every non-measurement bin | | `codes` | uint8\[320], `protocol.CODE_*` — the authority on 5.09 vs 5.11 | | `n_no_return`, `n_beyond_limit` | counts of each code this report | | `missed_frames` | consecutive frames not reported; 0 while healthy | | `enabled` | False if switched off at runtime | | `frame_id`, `ts_last_read`, `rate_hz` | | ### `calibration` — served by the body The tare is loaded, fingerprint-checked and published **by the robot**, so no consumer opens a YAML file or repeats the validation | field | meaning | | ---------- | --------------------------------------------------- | | `loaded` | names with an accepted tare | | `rejected` | `{name: why}` for every refusal | | `id` | changes when the calibration changes; cache on this | | `sensor_N` | packed tare | ## Parameters Under `line_sensor_loop` in the robot params: * `loop_rate_Hz` — 250. The reader polls far faster than the sensors report. * `sensor_names` — `sensor_0` … `sensor_5`, clockwise from robot forward. * `bus_sensor_map` — **`[[1, 0], [3, 2], [5, 4]]`**. Maps `distances` in the JSON to a logical sensor index. Change it if the cables are plugged in differently. * `flip_range_ordering` — whether to reverse each 320-bin array. * `line_sensor_geometry` — FOV, mounting angles, emitter height and pitch diameter, `pixart_report_num`. `bus_sensor_map` and `flip_range_ordering` **raise if missing** rather than defaulting. A wrong bin order mirrors every reading left-to-right and nothing downstream can detect it; a wrong bus map rotates the whole ring. Six identical 60-degree wedges tile the circle, so *any* mapping error produces a complete, plausible-looking result. Only a physical observation can catch it — occlude one wedge and confirm the expected sensor index responds. ## Two lossy hops, and what that forces ```mermaid flowchart TD A[PixArt hardware] -->|SPI / I2C| B[hello_pixart_j3 firmware in stretch_firmware_ii] B -->|newline-delimited JSON over USB CDC| C[PixartJ3Reader, background process at 250 Hz] C -->|complete status| D[q_status: CircularMultiprocessingQueue depth 3] D -->|pull_status| E[LineSensorLoop in the robot server] E -->|whole robot status at 100 Hz| F[ZMQ PUB, CONFLATE=1] F --> G[RobotClient / LineSensorLoopClient] ``` Both transports **drop**: * `q_status.put()` discards the *oldest* message when full. The reader puts at 250 Hz; the control loop drains at 48–100 Hz; the queue is 3 deep. * The ZMQ status socket sets `CONFLATE=1` on both ends, so a subscriber only ever holds the newest message. That single fact drives two rules: 1. **Every status message carries all six sensors.** a dropped delta loses that sensor's frame permanently, and the symptom — one sensor frozen while five stream — is indistinguishable from a hardware fault. 2. **The calibration rides every message.** Publishing it once would be conflated away for any client that connected a moment later, leaving it silently uncalibrated. `q_cmd` is the exception: it is 100 deep, because dropping the oldest *command* means a `set_streaming(False)` quietly does not happen. ## Calibration Flat-floor tare: park on a clean, light, flat floor with nothing within \~0.5 m, record, and subtract the difference between what each bin measures and the ideal floor depth. ``` REx_line_sensor_calibrate --all REx_line_sensor_calibrate -s sensor_1 --print-per-bin REx_line_sensor_calibrate --all --dry-run REx_line_sensor_calibrate --recompute # no robot needed ``` What it will and will not do: * Status-code samples never enter the arithmetic. A bin that mostly returns 5.09/5.11 is rejected, not averaged. * A run is **refused** if too many bins fail — a partial tare is worse than none, because nothing downstream can tell a bad correction from a good one. A dark or glossy floor produces exactly this. * Each raw session is one `session.npz`, so a tare can be recomputed later without the robot. * A stored tare records a **configuration fingerprint** (bus map, flip, report count, geometry, code mapping). On mismatch it is *refused*, never downgraded to a warning and never quietly replaced with an older file. ## Runtime control ```python from stretch4_body.robot.robot_client import RobotClient r = RobotClient(); r.startup() ls = r.line_sensor_loop ls.set_streaming(False) # pause all six ls.set_sensor_enabled('sensor_3', False) # or just one r.push_command() # nothing happens without this ls.is_streaming(); ls.disabled_sensors(); ls.dead_sensors() ``` Neither setting persists. A restart always comes up streaming with all six enabled. While paused the port is still read and the bytes discarded, so the kernel buffer cannot fill and hand back a corrupt part-report on resume. A disabled sensor is skipped *before* `json.loads`, which is where the reader's time actually goes, so it costs less rather than merely publishing less. ## Recovery A serial error closes the port, marks every enabled sensor dead, and retries the open on a 0.5 → 5 s backoff, incrementing `reader_restarts` on success. Verified on hardware: deauthorizing the USB node drops the subsystem and it returns by itself within \~3 s. Closing the port matters. It is opened `exclusive=True`, so a descriptor left open blocks every later reopen with "device busy" — which is how a single transient fault used to kill the subsystem until the next reboot, silently, while the process stayed alive. ## Tools * `REx_line_sensor_calibrate` — flat-floor tare. * `stretch_line_sensor_ranges` — live per-bin plot of what each bin reports. Status codes get their own coloured rows so 5.09 and 5.11 are distinguishable at a glance. `--calib` overlays the tare the body serves. * `stretch_line_sensor_viz_3d` — interactive 3D view of the projected points, with clustering and cost-map overlays. Both viewers run on the robot and need a display. ## Consumer checklist * Read `codes`, never a float comparison, to ask what a bin was. * Check `health['streaming']` and `health['disabled_sensors']` before trusting a clear floor. A disabled sensor reports nothing, which is not the same as reporting that nothing is there. * Check `sensors_dead`. * Use `is_sensor_updated(name)` to skip repeats: status publishes at 100 Hz while sensors report at \~30 Hz, so about 70% of polls carry no new frame. * Apply the tare with `LineSensorLoopClient.apply_tare()` rather than subtracting offsets yourself — it leaves status-code bins untouched. # Gamepad Teleop ## Gamepad Teleop for Stretch Stretch 4 ships with a gamepad controller for teleoperation. You can start the gamepad teleop script by running `stretch_gamepad_teleop` in the command line, or from the Stretch Tray icon in the system tray. > Note: `stretch_gamepad_teleop` uses a RobotClient. Make sure the Stretch Body Server is running using `stretch_body_server --status`. ### Homing from Gamepad Teleop When you boot Stretch 4, the robot needs to be homed (auto-calibrated). You may notice that the controller vibrates and does not allow you to move the robot. If this happens, make sure the robot's surroundings are clear, and press the `Start` button on the controller to Home (calibrate) the joints. ### Running in the background You can keep the `stretch_gamepad_teleop` application script running in the background while you run other RobotClient applications. The `stretch_gamepad_teleop` script will timeout and stop sending commands to the server after a few milliseconds of not receiving inputs. > Note that `stretch_gamepad_teleop` has a higher priority on the RobotClient command queue, so commands from it will override other scripts by default. ### Controller Mappings By default, `stretch_gamepad_teleop` has two control mapping modes that you can switch between using `Y`. The first is Joint Control, which allows you to control each joint individually. The second is Flying Gripper Control, which allows you to use the `Right Stick` to move the wrist, and move towards where the gripper is pointing using the `Left Stick`. This mode is useful for manipulation and grasping objects. {% tabs %} {% tab title="Joint Control Mode" %}
{% endtab %} {% tab title="Flying Gripper Mode" %}
{% endtab %} {% endtabs %} ### Changing Speed and Strength Tap or hold the `Right Trigger + A` to change or cycle the speed of Stretch 4. By default it is Medium speed. This cycles between Low, Medium and High speed presets. > Be careful when navigating with Medium or High speed. It is a good idea to keep the arm retracted or Stow the robot (Hold `Select`) while the robot is moving in the environment. Tap or hold the `Right Trigger + B` to change or cycle the strength of Stretch 4. By default it is Medium strength. This cycles between Low, Medium and High strength presets. Use Low strength when manipulating the robot near people, and High strength when trying to manipulate objects that require more force, like opening a door. > Strength affects the contact sensitivity thresholds. Low strength will stop joint motion when encountering a small motor effort. ### Special Functions * **Homing**: If the robot is not homed, press the `Start` button to home the robot. * **Gripper Handedness**: If the robot is homed, hold the `Start` button for 3 seconds to switch gripper handedness (Left/Right). * **Stowing**: Hold the `Select` button for 3 seconds to stow the robot. ## Customization You can customize the behavior by: * Modifying `gamepad_control_mappings.py` to add or edit mappings. * Setting `params['function_cmd']` in user parameters to execute a shell command via the `X` button. # Puppet Teleop The `stretch_puppet_teleop.py` tool allows a user to control a remote Stretch robot (the "Puppet") by physically moving the joints of a local Stretch robot (the "Controller"). ## 1. Overview and Architecture When the script is launched, it connects to the local Controller robot and places its joints (lift, arm, wrist, base) into a backdrivable "freewheel" or "safety" mode. This allows a human operator to physically push and pull the robot's links. The script continuously reads the joint positions of the Controller at a high frequency (80 Hz) and streams them as motion commands to the remote Puppet robot over the network. Additionally, the Controller tool supports a custom "Pistol Grip" hardware attachment (`/dev/hello-gripper-pistol`) with a slider potentiometer. This slider can be used to control the Puppet's gripper aperture, independent of the Controller's physical gripper state. ### Architecture Block Diagram ```mermaid graph TD User([Human Operator]) -->|Physically moves links| Controller[Local Controller Robot] User -->|Slides potentiometer| Pistol[Gripper Pistol Slider] subgraph Controller Side Controller Pistol Script[stretch_puppet_teleop.py] end Pistol -->|Serial USB| Script Controller -->|Local ZMQ / Status Pull| Script subgraph Network ZMQ[TCP / ZMQ Connection] end Script -->|Motion Commands| ZMQ subgraph Puppet Side PuppetServer[Stretch Body Server] PuppetRobot[Remote Puppet Robot] end ZMQ --> PuppetServer PuppetServer -->|Actuates motors| PuppetRobot PuppetRobot -->|Interacts with| Environment([Physical Environment]) ``` ## 2. Controller Robot Setup * 2 Stretch * 1 controller Stretch with pistol gripper, and gripper mapped to /dev/hello-gripper-pistol * 2nd Stretch with SG4 or PG4 tool Add the following to `/etc/udev/rules.d/95-hello-arduino.rules` `KERNEL=="ttyACM*", ATTRS{idVendor}=="239a", ATTRS{idProduct}=="8101",MODE:="0666", SYMLINK+="hello-gripper-pistol", ENV{ID_MM_DEVICE_IGNORE}="1"` Then run `sudo udevadm control --reload` and confirm that /dev/hello-gripper-pistol is there ## 2. Networking Setup To successfully teleoperate the Puppet robot, both robots must be connected to the same network (e.g., the same Wi-Fi router or a direct Ethernet connection), and you must know the IP address of the Puppet robot. **Step 1: Find the Puppet's IP Address** On the remote Puppet robot, open a terminal and run: ```bash hostname -I ``` Alternatively, you can use `ip a` or `ifconfig`. Note the IPv4 address (e.g., `192.168.1.15`). **Step 2: Start the Robot Servers** Both robots require the Stretch Body Server to be running. * **On the Puppet robot:** Start the server in a terminal: ```bash stretch_body_server ``` * **On the Controller robot:** Start the server in a terminal: ```bash stretch_body_server ``` **Step 3: Run the Teleop Script** On the Controller robot, run the teleop script and provide the Puppet's IP address (see Usage Examples below). ## 3. Command Line Arguments The `stretch_puppet_teleop.py` script provides several arguments to customize the teleoperation experience: | Argument | Description | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--puppet_ip` | IP address of the remote Puppet robot running Stretch Body Server (e.g., `192.168.1.10`). Required unless `--no_puppet` is set. | | `--joints` | List of joints to mimic. Default is: `omnibase lift arm wrist gripper`. Example: `--joints lift arm` | | `--no_puppet` | Runs the script in a local-only testing mode without connecting to a puppet robot. Useful for verifying the controller's backdrivability and joint sensors. | | `--no_pistol` | Runs the script without attempting to connect to the pistol grip slider hardware. | | `--pg4` | Indicates that the Puppet robot is equipped with a Parallel Gripper 4. Automatically maps standard gripper slider inputs to PG4 translation commands. | | `--pg4c` | Indicates that the Controller robot is equipped with a Parallel Gripper 4. | | `--print_only` | Connects to both robots and prints their joint positions to the terminal, but does *not* command any motion on the Puppet. Good for safe network testing. | > \[!IMPORTANT] The wrist\_pitch, wrist\_yaw, and wrist\_roll joints on the Controller robot must have `enable_torque_after_runstop: 0` configured in their `stretch_user_params.yaml`. This ensures the wrist remains backdrivable and doesn't snap to a position after a runstop event. ## 4. Usage Examples Here are common ways to launch the teleoperation script. For these examples, we assume the Puppet robot's IP address is `192.168.1.15`. **Standard Teleoperation** Connects to the puppet and starts mimicking all default joints, assuming the pistol grip slider is attached. ```bash stretch_puppet_teleop.py --puppet_ip 192.168.1.15 ``` **Teleoperation Without Pistol Grip** If you are using a standard Controller robot without the custom pistol grip hardware installed. ```bash stretch_puppet_teleop.py --puppet_ip 192.168.1.15 --no_pistol ``` **Teleoperation with a Parallel Gripper (PG4)** If the remote Puppet robot has the Parallel Gripper 4 attached instead of the standard Stretch Gripper. ```bash stretch_puppet_teleop.py --puppet_ip 192.168.1.15 --pg4 ``` **Partial Teleoperation (Arm and Lift Only)** Only mirror the lift and arm extensions. The base, wrist, and gripper will remain stationary. ```bash stretch_puppet_teleop.py --puppet_ip 192.168.1.15 --joints lift arm --no_pistol ``` **Safe Dry-Run** Connects to the Puppet over the network and displays the real-time joint positions of both robots in a terminal table, without actually moving the Puppet. ```bash stretch_puppet_teleop.py --puppet_ip 192.168.1.15 --print_only ``` **Controller Hardware Test** Places the local Controller robot into backdrivable mode and prints joint values to the terminal. No network connection or Puppet robot is required. ```bash stretch_puppet_teleop.py --no_puppet ``` **Base Rotation Only** Filters out translation commands from the Controller's base. Pushing the Controller base will only generate pure rotation commands on the Puppet. ```bash stretch_puppet_teleop.py --puppet_ip 192.168.1.15 --base_rotate_only ``` # README ## Overview This interface enables a user to remotely teleoperate a Stretch robot through a web browser. This website can be set up to teleoperate the robot remotely from anywhere in the world with an internet connection, or simply eyes-off teleop from the next room on a local network. The codebase is built on ROS2, WebRTC, Nav2, and TypeScript. ## Setup & Installation The interface is compatible with the Stretch 4. It currently only supports Ubuntu 24.04 and ROS2 Humble. Upgrade your operating system if necessary ([instructions](https://docs.hello-robot.com/0.3/installation/robot_install/)) and create/update the Stretch ROS2 Humble workspace ([instructions](https://docs.hello-robot.com/0.3/installation/ros_workspace/)). This will install all package dependencies and install the web teleop interface. ## Launching the Interface First, navigate to the folder containing the codebase using: ``` colcon_cd stretch4_web_teleop ``` Next, launch the interface: ``` ./launch_interface.sh ``` In the terminal, you will see output similar to: ``` Visit the URL(s) below to see the web interface: https://localhost/operator https://192.168.1.14/operator ``` Look for a URL like `https:///operator`. Visit this URL in a web browser on your personal laptop or desktop to see the web interface. Ensure your personal computer is connected to the same network as Stretch. You might see a warning that says "Your connection is not private". If you do, click `Advanced` and `Proceed`. Once you're done with the interface, close the browser and run: ``` ./stop_interface.sh ``` **Note:** Only one browser can be connected to the interface at a time. ## Using the Interface Remotely **WARNING: This is prototype code and there are security issues. Deploy this code at your own risk.** We recommend setting up the interface for remote use using [ngrok](https://ngrok.com/docs/what-is-ngrok/). First, create an account with `ngrok` and follow the Linux installation instructions in the `Setup & Installation` tab in your ngrok account dashboard. Navigate to the `Domains` tab and click `Create Domain`. ngrok will automatically generate a domain name for your free account. You will see a domain similar to `deciding-hornet-purely.ngrok-free.app`. Follow the interface launch instructions and then start the ngrok tunnel by running the following command (replace `` with your account's domain and `user:password` with a secure username and password): ``` ngrok http --basic-auth="user:password" --domain= 443 ``` In your browser, open `https:///operator` to see the interface. You will then be prompted to enter the appropriate username and password. Note, anyone in the world with internet access can open this link. ### Storing Ngrok Tunnel Configuration To store this configuration, open the ngrok config file: ``` ngrok config edit ``` Add the following configuration to the file. Make sure to update ``, ``, and `admin:password` with the appropriate values. ``` authtoken: version: 2 tunnels: stretch-web-teleop: proto: http domain: addr: 443 basic_auth: - "admin:password" host_header: rewrite inspect: true ``` Now run `ngrok start stretch-web-teleop` to start the tunnel and navigate to `https:///operator`. You will then be prompted to enter the appropriate username and password. ## Developer Docs The following primers provide a high-level introduction to the codebase for developers. | Primer | Description | | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Software Architecture](/stretch-4-web-teleop/src/primer_software_architecture) | Overview of the system architecture — how the robot browser, operator browser, ROS2, and WebRTC fit together. Includes a render logic flow diagram showing how `MobileOperator` is structured into scenes, function providers, and the global footer. | | [Operator Page](/stretch-4-web-teleop/src/primer_operator) | Deep dive into the operator page codebase — directory layout, the shared layer (`commands`, `RemoteRobot`, `util`, `webrtcconnections`), the entry point (`index.tsx`), function providers, component folders, storage handler, and a step-by-step guide for adding new features. | | [Robot Page](/stretch-4-web-teleop/src/primer_robot) | Deep dive into the robot page codebase — the `Robot` class, ROS2 subscriptions/services/actions, robot modes, joint state processing, video streams, and the full bidirectional data flow between ROS2 and the operator browser. | ## Contributing * This repository uses pre-commit hooks to enforce consistent formatting and style. * Install pre-commit: `python3 -m pip install pre-commit` * Install the hooks locally: `cd` to the top-level of this repository and run `pre-commit install`. * Moving forward, pre-commit hooks will run before you create any commit. ## Troubleshooting ### Collecting logs First, ensure that your robot has the latest version of Web Teleop by [updating your ROS workspace](https://docs.hello-robot.com/0.3/installation/ros_workspace/). Then, launch the program normally, and if you see "FAILURE. COULD NOT LAUNCH WEB TELEOP.", then locate the zipped-up logs file and send them to Hello Robot Support (). To locate the logs, open a file explorer, go into "Home", go into "stretch\_user", go into "log", go into "web\_teleop", locate the folder with the latest timestamp, and send "stretch4\_web\_teleop\_logs.zip" to the support team. ## Licenses The following license applies to the contents of this directory written by Vinitha Ranganeni, Noah Ponto, authors associated with the University of Washington, and authors associated with Hello Robot Inc. (the "Contents"). This software is intended for use with Stretch ® mobile manipulators produced and sold by Hello Robot ®. Copyright 2023 Vinitha Ranganeni, Noah Ponto, the University of Washington, and Hello Robot Inc. The Contents are licensed under the Apache License, Version 2.0 (the "License"). You may not use the Contents except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, the Contents are distributed under the License are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. \============================================================ Some of the contents of this directory derive from the following repositories: Text from relevant license files found in these repositories. # LICENSE From The following license applies to the contents of this directory created by Hello Robot Inc. (the "Contents"), but does not cover materials from other sources. This software is intended for use with the Stretch RE1 mobile manipulator, which is a robot produced and sold by Hello Robot Inc. Copyright 2020 Hello Robot Inc. The Contents are licensed under the Apache License, Version 2.0 (the "License"). You may not use the Contents except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, the Contents are distributed under the License are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. For further information about the Contents including inquiries about dual licensing, please contact Hello Robot Inc. # WEBRTC\_PROJECT\_LICENSE From The following license covers the original code from which some of the web interface code was derived (e.g., operator\_acquire\_av.js, robot\_acquire\_av.js). The original code was released in the following repository, which contains WebRTC example code. \====================================== Copyright (c) 2014, The WebRTC project authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Neither the name of Google nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # src # Primer: Operator Page The **operator page** is the browser-based UI through which a user teleoperates the Stretch 4 robot. It connects to the robot browser over WebRTC, receives sensor/camera data, and sends movement commands back. > For a worked example of adding a brand-new feature end-to-end (the "Home the Robot" button), see [`documentation/development.md`](https://github.com/hello-robot/stretch4_web_teleop/blob/main/documentation/development.md). *** ## Directory Layout ``` src/pages/operator/ ├── html/ # Entry HTML page ├── css/ # Page and component stylesheets ├── icons/ # SVG icon assets └── tsx/ # All TypeScript/React source ├── index.tsx # Module entry point — bootstraps WebRTC and renders root ├── MobileOperator.tsx # Mobile operator root component ├── function_providers/ # Logic layer connecting UI to the robot ├── layout_components/ # Composite, stateful UI components ├── basic_components/ # Small, reusable UI primitives ├── static_components/ # Mostly-fixed UI panels (header, sidebar, etc.) ├── storage_handler/ # Layout persistence (local or Firebase) ├── utils/ # Shared types, enums, and small utilities └── default_layouts/ # JSON default layout configs ``` *** ## Shared Layer (`src/shared/`) Before describing the operator page itself, it helps to understand the shared layer it builds on. These four files define the common language between the two browser peers. ### `commands.tsx` This file defines the common language between the operator and robot browsers. It defines every **command** the interface can transmit — the full vocabulary of robot actions. For example, you can ask Nav2 to navigate the robot using the MoveBaseCommand, or you can ask the driver for the battery's current voltage using GetBatteryVoltageCommand. Every action taken in the web interface is converted into a command and transmitted through WebRTC's data channel to the robot browser, where it is interpreted and further transmitted to Stretch's ROS2 packages. Each command is a TypeScript interface with a `type` discriminant: ```ts export interface HomeTheRobotCommand { type: "homeTheRobot"; } ``` All commands are collected into the `cmd` union type, which is what WebRTC's data channel actually carries: ```ts export type cmd = DriveCommand | IncrementalMove | HomeTheRobotCommand | ...; ``` **When adding a new capability**, define its command here first. ### `remoterobot.tsx` This file wraps the logic for receiving and transmitting commands, sensor streams, etc. over the WebRTC channels in a nice API for the operator browser. You can think of `RemoteRobot` as the bridge between the operator browser and the robot. The interface creates a single instance of `RemoteRobot` and uses it for the duration of the session that it is connected to the robot. On the other end, the robot browser is listening for commands from / transmitting data to `RemoteRobot`. Each method packages a command and sends it through the WebRTC data channel: ```ts homeTheRobot() { let cmd: HomeTheRobotCommand = { type: "homeTheRobot" }; this.robotChannel(cmd); } ``` **When adding a new capability**, add a method to `RemoteRobot` that dispatches the new command. ### `util.tsx` This file contains shared custom message types, variables, and enums For example, this file defines the WebRTC message types: ```js export type WebRTCMessage = ... | cmd; ``` `cmd` is imported from `commands.tsx`. This means all commands defined in `commands.tsx` can be transmitted over WebRTC. You do not need to edit this file for the homing functionality, you just need to define your command as described [in this section](#commandstsx). You generally do not need to edit this file unless adding a new WebRTC message type. ### `webrtcconnections.tsx` *Note: You should never touch this file!* This file contains the code used for establishing a peer connection between the operator and robot browser and setting up data channels such that they can communicate. *** ## Entry Point: `tsx/index.tsx` This file is the module entry. It: 1. **Creates** all `FunctionProvider` singletons (see below). 2. **Establishes** the WebRTC connection to the robot browser. 3. **Handles two WebRTC callbacks**: * `handleRemoteTrackAdded()` — subscribes to video/audio media channels so they feed the camera views. * `handleWebRTCMessage()` — a switch statement that routes incoming robot data to the right function provider or state variable: ```ts case "isRunStopped": remoteRobot.sensors.setRunStopState(message.enabled); case "moveBaseState": underMapFunctionProvider.setMoveBaseState(message.message); case "occupancyGrid": // chunked and assembled into occupancyGrid module-level var ``` 4. **Calls** `initializeOperator()` once the data channel is open, which: * Creates the `RemoteRobot` instance and sets up its sensor callbacks. * Creates the `StorageHandler` (loads the saved layout). * Instantiates providers that require storage (e.g. `MovementRecorderFunctionProvider`, `UnderMapFunctionProvider`, etc.). * Renders `MobileOperator` (Note, we currently only have the mobile version of the operator page. On desktop, the mobile operator page is scaled up to fit the screen). Key module-level exports that components import directly (no prop-drilling): | Export | Type | Description | | ---------------------------------- | ---------------------------------- | --------------------------- | | `buttonFunctionProvider` | `ButtonFunctionProvider` | Main movement logic | | `runStopFunctionProvider` | `RunStopFunctionProvider` | Run-stop state | | `batteryVoltageFunctionProvider` | `BatteryVoltageFunctionProvider` | Battery display | | `movementRecorderFunctionProvider` | `MovementRecorderFunctionProvider` | Playback/recording | | `underMapFunctionProvider` | `UnderMapFunctionProvider` | Autonomous navigation | | `homeTheRobotFunctionProvider` | `HomeTheRobotFunctionProvider` | Homing sequence | | `cameraSwitcherFunctionProvider` | `CameraSwitcherFunctionProvider` | Camera perspective | | `occupancyGrid` | `ROSOccupancyGrid \| undefined` | Current map data | | `stretchTool` | `StretchTool` | Currently mounted tool | | `storageHandler` | `StorageHandler` | Persisted layout/recordings | *** ## Root Components ### `MobileOperator.tsx` The active root component rendered on the user's device. Renders two the **Pilot Mode** scene by default (live camera + robot controls) and the **Auto Nav** scene can but accessed from the scene menu (map-based autonomous navigation). *** ## Function Providers (`tsx/function_providers/`) The **function provider** pattern is the core architectural idea: React components are kept logic-free. Each component is given a function provider object at render time, and calls methods on it in response to user interactions. Components define an enum naming the actions they need, and the provider returns an anonymous function for each: ```ts // In the component: export enum HomeTheRobotFunction { Home } // In the provider: public provideFunctions(fn: HomeTheRobotFunction) { switch (fn) { case HomeTheRobotFunction.Home: return () => { FunctionProvider.remoteRobot?.homeTheRobot(); }; } } // In the component's render: const Home = homeTheRobotFunctionProvider.provideFunctions(HomeTheRobotFunction.Home); ``` ### `FunctionProvider` (base class) All providers extend this. It holds: * `static remoteRobot` — the single shared `RemoteRobot` instance (set via `addRemoteRobot()`) * `static velocityScale`, `actionMode`, `pilotControlsCurrent` — global control settings shared across all providers * `setBaseVelocity()` / `incrementalJointMovement()` / `continuousJointMovement()` — shared motion helpers that manage the velocity heartbeat interval (sent every 25 ms) * `stopCurrentAction()` — cancels any active velocity send loop or timeout ### `ButtonFunctionProvider` The heaviest provider. Maps every `ButtonPadButton` enum value to robot actions. Handles: * Different **action modes**: `StepActions` (tap once, move discrete amount), `PressRelease` (hold for continuous), `ClickClick` (tap to start, tap to stop) * **Joint state callbacks** from the robot → updates which buttons are at their limits/in collision → updates `ButtonStateMap` for the UI * Drive, arm, lift, wrist, gripper, head, and camera-perspective commands ### `MovementRecorderFunctionProvider` Manages recording and playing back sequences of joint poses. Interfaces with `StorageHandler` to persist recordings. Streams playback state back via operator callback. ### `UnderMapFunctionProvider` Handles autonomous move-base navigation. Sends navigation goals via `RemoteRobot`, receives action state feedback, and exposes an operator callback for alert rendering. ### `HomeTheRobotFunctionProvider` Tracks homing and robot mode state. Exposes `updateModeState()` and `updateIsHomedState()` for the WebRTC message handler to call. ### `RunStopFunctionProvider` Tracks the software run-stop state. Exposes `updateRunStopState()`. ### `BatteryVoltageFunctionProvider` Tracks battery voltage. Exposes `updateVoltage()`. ### `MapFunctionProvider` Handles click-on-map → navigation goal conversion. ### `CameraSwitcherFunctionProvider` Manages which camera perspective is active. *** ## Component Folders There are three component folders, each with a different intended scope: * **`layout_components/`** — customizable components that form the interactive body of the interface (camera views, button pads, map, movement recorder). In the desktop `Operator`, these can be added, removed, or rearranged by toggling "Customize". * **`static_components/`** — fixed components found in the header, footer, or sidebar (speed controls, action mode selector, run-stop, occupancy grid canvas, etc.). They cannot be moved by the user. * **`basic_components/`** — generic, robot-agnostic UI primitives (dropdowns, modals, carousels, tab groups) used to build the above two. ### Notable Layout Components | Component | Purpose | | ------------------ | ------------------------------------------------------------------------------------ | | `PilotMode` | Top-level pilot scene: camera view + controls tab group + movement recorder + footer | | `SimpleCameraView` | Renders a single camera video stream | | `GripperCamPIP` | Gripper camera picture-in-picture overlay | | `ButtonPad` | Grid of robot control buttons; receives functions from `ButtonFunctionProvider` | | `MovementRecorder` | Feature for recording, naming, editing, and replaying pose sequences | | `AutoNav` | Map view + click-to-navigate goal setting | | `FooterPilotMode` | Bottom toolbar: camera switcher, action speed, action mode, scene navigation | | `FooterAutoNav` | Bottom toolbar in auto-nav scene: goal input, move-base controls | | `FooterGlobal` | Persistent bottom bar switching between pilot/auto-nav scenes | | `HomeTheRobot` | Prominent banner shown when robot is un-homed, with a "Home" button | ### Notable Static Components | Component | Purpose | | --------------------- | -------------------------------------------------------------------------------------- | | `ActionMode` | Dropdown: Step Actions / Press-Release / Click-Click | | `ActionSpeed` | Speed selector (Slowest → Fastest) | | `PilotControlsToggle` | Switches between button pad layouts (Omni Drive, Arm, etc.) | | `RunStop` | Software run-stop indicator/button | | `OccupancyGrid` | Canvas-based 2D map renderer (CreateJS); robot marker, goal marker, saved pose markers | | `CameraSwitcher` | Buttons to switch camera perspective | ### Notable Basic Components | Component | Purpose | | ------------------- | ----------------------------------------------- | | `SceneCarousel` | Horizontally scrollable scene switcher carousel | | `PlaybackStatusbar` | Progress bar for movement playback | | `Flex` | Lightweight flexbox layout wrapper | *** ## Storage Handler (`tsx/storage_handler/`) Persists the operator's layout and recordings across sessions. Two backends: * **`LocalStorageHandler`** — uses browser `localStorage`; default for on-robot use. * **`FirebaseStorageHandler`** — syncs to Firestore; used when `process.env.storage === "firebase"`. Both implement the common `StorageHandler` interface. *** ## Utils (`tsx/utils/`) | File | Contents | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `component_definitions.tsx` | Enums and types for the layout system: `ComponentType`, `CameraViewId`, `ActionModeType`, `LayoutDefinition`, `ComponentDefinition`, etc. | | `genUUID.tsx` | Thin wrapper around the uuid library | | `hex-to-rgb-array.tsx` | `#rrggbb` → `[r, g, b]` color conversion | | `svg.tsx` | Inline SVG strings for all button pad icons (arm directions, gripper, head, etc.) | *** ## Full Data Flow ``` WebRTC media channels │ ▼ handleRemoteTrackAdded() ← feeds camera