> For the complete documentation index, see [llms.txt](https://docs.hello-robot.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.hello-robot.com/stretch4_mujoco/docs/architecture.md).
# Using the Mujoco Simulator with Stretch
When using Mujoco to simulate Stretch, you can command [joints](https://github.com/hello-robot/stretch4_mujoco/blob/main/stretch4_mujoco/enums/actuators.py) and access joint poses and [camera](https://github.com/hello-robot/stretch4_mujoco/blob/main/stretch4_mujoco/enums/stretch_cameras.py) data.
## Getting Started
1. Read the [README](/stretch4_mujoco/readme.md) to install the required dependencies.
2. Check out the controller examples, such as:
* [keyboard\_teleop.py](https://github.com/hello-robot/stretch4_mujoco/blob/main/examples/keyboard_teleop.py)
* [gamepad\_teleopy.py](https://github.com/hello-robot/stretch4_mujoco/blob/main/examples/gamepad_teleop.py)
3. Check out the headless examples, such as:
* [draw\_circles.py](https://github.com/hello-robot/stretch4_mujoco/blob/main/examples/draw_circles.py)
* [camera\_feeds.py](https://github.com/hello-robot/stretch4_mujoco/blob/main/examples/camera_feeds.py)
4. Check out the sensor example: [laser\_scan.py](https://github.com/hello-robot/stretch4_mujoco/blob/main/examples/laser_scan.py)
### Terminology
The following words apply to this document only, to make it easier to read:
* [Mujoco](https://mujoco.readthedocs.io/en/stable/overview.html): An open-source physics engine.
* [Mujoco Viewer](https://mujoco.readthedocs.io/en/stable/programming/samples.html#sasimulate): An interactive Mujoco GUI that ships with Mujoco. This is spawned when you don't use `headless` mode.
* [Headless Mode](https://mujoco.readthedocs.io/en/stable/APIreference/APIfunctions.html#main-simulation): Using the Mujoco simulation without the Mujoco Viewer. This calls `mj_step` directly to step the simulation. For the purposes of Stretch Mujoco Simulations, this is a performant mode to run simulations in.
* [Stretch Mujoco Simulator](https://github.com/hello-robot/stretch4_mujoco/blob/main/stretch4_mujoco/stretch4_mujoco_simulator.py): A scaffolding that enables you to send commands and receive sensor data to and from Stretch in a Mujoco environment.
## Control Flow
All simulations using [Stretch Mujoco Simulator](https://github.com/hello-robot/stretch4_mujoco/blob/main/stretch4_mujoco/stretch4_mujoco_simulator.py) should have an entry point that looks similar to this:
```python
if __name__ == "__main__":
# You can use all the camera's, but it takes longer to render, and may affect the overall simulation FPS.
# cameras_to_use = StretchCameras.all_stretch4()
cameras_to_use = [StretchCameras.cam_gripper_rgb]
sim = StretchMujocoSimulator(cameras_to_use=cameras_to_use)
sim.start(headless=True)
```
> You can use the `start()` method to specify headless or UI view modes. The Passive Viewer is the default and recommended non-headless mode.
> If you are using the simulation for machine-learning applications, it is recommended to use the headless mode for better performance.
> If you are displaying camera data or doing heavy computations on your Control Loop, it is recommended to move your control commands to a thread, and display the camera data on the main thread. See [Displaying camera data using OpenCV](#displaying-camera-data-using-opencv) below and the [camera\_feeds.py](https://github.com/hello-robot/stretch4_mujoco/blob/main/examples/camera_feeds.py) example for more information.
### Commanding Stretch
When the simulator is running, you can use your script to send commands to Stretch, or read data from the simulation.
Use the following command to move the lift to `0.5m`: `sim.move_to(Actuators.lift, 0.5)`
### Reading data from Stretch
There are two methods for pulling data from the simulation: `sim.pull_status()` and `sim.pull_camera_data()`.
#### Stretch Status
Use `sim.pull_status()` to fetch the joint states of the robot.
This method returns a `StatusStretchJoints` [dataclass](https://github.com/hello-robot/stretch4_mujoco/blob/main/stretch4_mujoco/datamodels/status_stretch_joints.py) with the names of all the joints populated.
The statuses of all the joints are fetched at the same time.
#### Stretch Sensors
Use `sim.pull_sensor_data()` to fetch data from sensors on Stretch.
This methods returns a `StatusStretchSensors` [dataclass](https://github.com/hello-robot/stretch4_mujoco/blob/main/stretch4_mujoco/datamodels/status_stretch_sensors.py)
The statuses of all the sensors are fetched at the same time.
All the sensors defined in [`stretch.xml`](https://github.com/hello-robot/stretch4_mujoco/blob/main/stretch4_mujoco/models/scene.xml) are fetched:
```
```
> Note: The Lidar sensor (implemented as ``) is compute intensive. Comment it out in the XML, if you are not using it.
#### Stretch Cameras
Use `sim.pull_camera_data()` to fetch the pixel values from Stretch's cameras.
This method returns a `StatusStretchCameras` [dataclass](https://github.com/hello-robot/stretch4_mujoco/blob/main/stretch4_mujoco/datamodels/status_stretch_camera.py) with the names of all the cameras populated.
The pixel values of all the renderings of the cameras are fetched at the same time.
Note: this operation is computationally heavy. The more cameras that are requested, the slower the simulation may run.
**Displaying camera data using OpenCV**
The [camera\_feeds.py](https://github.com/hello-robot/stretch4_mujoco/blob/main/examples/camera_feeds.py) example shows a sample to display camera data using `cv2.imshow()`.
```python
camera_data = sim.pull_camera_data()
for camera in cameras_to_use:
cv2.imshow(camera.name, camera_data.get_camera_data(camera))
cv2.waitKey(1)
```
> Important Note: you should call `cv2.imshow()` on the MAIN THREAD to avoid getting graphics library (GL) related errors from your OS.
### Misc Stretch Mujoco Simulator API calls
#### World Coordinate Frame Arrows
You can use `sim.add_world_frame((0.1, 0.0, 0.0))` to add arrows dynamically to the Mujoco viewer:
This also supports rotating the frame:
```
sim.add_world_frame((0.1,0,0), (0,0,0))
sim.add_world_frame((0.2,0,0), (1.57,0,0)) # (x, y, z), (r, p, y)
```
## More to know
### Behind the scenes
When you call `sim.start()`, the following process diagram explains how Mujoco is launched and your Control Loop are managed.
> tl;dr The Mujoco simulator is started on a spawned process, and data is communicated between your main process and the Mujoco process using a [Multiprocesing Manager](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Manager).
### Mujoco rendering locks
Mujoco's [passive](https://mujoco.readthedocs.io/en/stable/python.html#passive-viewer) and headless modes need some access governance over the `mjdata` and `mjmodel` objects to behave correctly.
When using either of these modes, we are responsible for calling `mj_step` to step the simulation. This also means we're responsible for managing when Mujoco should render scenes. Calling `mj_step` is not a thread-safe operation, and if it's done while a render is rendering, bad things happen. So we use some mutexes to ensure these steps are in sync.
> Note: ignoring the use of mutexes to lock `mjdata` and `mjmodel` before rendering could cause Mujoco to crash, or instability errors such as "WARNING: Inertia matrix is too close to singular at DOF 11. Check model. WARNING: Nan, Inf or huge value in QPOS at DOF 0. The simulation is unstable."
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.hello-robot.com/stretch4_mujoco/docs/architecture.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.