Experience remote assistant in the simulator with simulation_remote_assistant

The simulation_remote_assistant sample application is the ROS package that uses an AMR as a remote assistant within a virtual office environment. Users can interact with the robot by inputting natural language commands, such as “Go to the office to check the person.” The robot can autonomously navigate to the specified location and perform object detection tasks as instructed.

NoteFor more information, see simulation_remote_assistant on GitHub.

Figure: Sample effects of simulation_remote_assistant

Pipeline flow for Simulation Remote Assistant

Figure: Pipeline flow for simulation_remote_assistant

ROS nodes used in the simulation_remote_assistant pipeline

ROS node	Description
`qrb_ros_simulation`	`qrb_ros_simulation` is a ROS package that sets up the Qualcomm robotic simulation environment. For details, see QRB ROS Simulation.
`nn_inference_node`	Loads a trained AI model, receives preprocessed images, performs inference, and publishes results. For details, see NN Inference Node.
`yolo_preprocess_node`	Executes preprocess and postprocess for Yolo model. For details, see QRB ROS Tensor Process.
`yolo_detection_postprocess_node`
`yolo_detection_overlay_node`
`cartographer_node`	Processes sensor data to perform SLAM.
`cartographer_occupancy_grid_node`	Publishes a ROS occupancy grid map.
`nav2_bringup`	The Navigation2 stack for robot navigation.
`build_map_node`	Builds the map of office world.
`nav_preparation_node`	Relocalization and launches the navigation node.
`task_manager_node`	Parses the input commands and executes the action.

ROS topics used in the Simulation Remote Assistant pipeline

ROS topic	Type	Published by	Description
`/scan`	`sensor_msgs::msg::LaserScan`	`qrb_ros_simulation`	2D lidar scan data for SLAM and mapping.
`/map`	`nav_msgs::msg::OccupancyGrid`	`cartographer_occupancy_grid_node`	Occupancy grid map for navigation.
`/tf`	`tf2_msgs::msg::TFMessage`	`qrb_ros_simulation` / `cartographer_node`	Frame transforms between coordinate systems.
`/odom`	`nav_msgs::msg::Odometry`	`qrb_ros_simulation`	Odometry data for pose estimation.
`/cmd_vel`	`geometry_msgs::msg::Twist`	`qrb_ros_simulation`	Velocity commands for robot motion.
`/goal_pose`	`geometry_msgs::msg::PoseStamped`	`task_manager_node`	Target goal pose for navigation.
`/camera/color/image_raw`	`sensor_msgs::msg::Image`	`qrb_ros_simulation`	RGB image from the simulation camera.
`/qrb_inference_input_tensor`	`qrb_ros_tensor_list_msgs::msg::TensorList`	`yolo_preprocess_node`	Preprocessed input tensor for inference.
`/yolo_detect_tensor_output`	`qrb_ros_tensor_list_msgs::msg::TensorList`	`nn_inference_node`	Inference output tensor for detection.
`/yolo_detect_result`	`vision_msgs::msg::Detection2DArray`	`yolo_detection_postprocess_node`	The detected objects as 2D bounding-box results.
`/yolo_detect_overlay`	`sensor_msgs::msg::Image`	`yolo_detection_overlay_node`	The detection overlay image with bounding boxes.

Prerequisites

You have completed the following settings in Set up the environment for running sample applications (Skip the settings of Set up host docker).
- Set up the device
- Set up the host computer
Follow the steps in Generate YOLOv8 model using QAIHUB to generate the model files and download the coco-labels YAML file. Then, push them to the /opt/model/ directory on the device.
mkdir /opt/model/ mv coco8.yaml yolov8_det_qcs9075.bin yolov8_det_qcs8275.bin /opt/model/

Set up the simulation environment

On your host computer, complete the following to build the Qualcomm robotic simulation environment according to the README of QRB ROS Simulation. Ensure that the device and the host are on the same local network and can communicate with each other through ROS communication.
1. Set up development environment on Ubuntu24.04 host or Docker.
2. Build from source.

On the host, run the QRB Robot Base AMR in the simulated office world.

# Set the ROS_DOMAIN_ID
export ROS_DOMAIN_ID=78
ros2 launch qrb_ros_sim_gazebo gazebo_robot_base_mini.launch.py \
world_model:=office \
initial_x:=1.0 \
initial_y:=6.0 \
enable_depth_camera:=false

Run out-of-the-box `simulation_remote_assistant`

On the development kit, run the following commands:

# setup runtime environment
(ssh) mount -o remount rw /usr
(ssh) source /opt/ros/jazzy/setup.bash
(ssh) export ROS_DOMAIN_ID=78

# In /opt/ros/jazzy/share/nav2_bringup/params/nav2_params.yaml,
# change nav2_mppi_controller::MPPIController to nav2_graceful_controller::GracefulController.

# launch the map_nav_setup.launch.py for auto mapping
(ssh) ros2 launch simulation_remote_assistant map_nav_setup.launch.py

# launch the yolo_detectcion.launch.py
# qcs9075 RB8 bin htp
(ssh) ros2 launch simulation_remote_assistant yolo_detectcion.launch.py model:=/opt/model/yolov8_det_qcs9075.bin backend_option:=libQnnHtp.so label_file:=/opt/model/coco8.yaml
# qcs8275 RB4 bin htp
(ssh) ros2 launch simulation_remote_assistant yolo_detectcion.launch.py model:=/opt/model/yolov8_det_qcs8275.bin backend_option:=libQnnHtp.so label_file:=/opt/model/coco8.yaml


# launch the task_manager node
(ssh) ros2 run simulation_remote_assistant task_manager_node

# input the fix commands in the task_manager_node terminal like
(ssh) Go to office to check person

Build and run `simulation_remote_assistant`

In a terminal of the host computer, run the following commands:

Note

<qir_version>: Version of the QIR SDK, see QIR SDK 2.0 Release Notes.
<qir_decompressed_path>: The path where the QIR SDK is decompressed.

Build the sample application project.

# set up the QIR SDK environment
cd <decompressed_workspace>/target/qcs6490-rb3gen2-vision-kit/qirpsdk_artifacts/qcs6490-rb3gen2-vision-kit
tar -zxf qirp-sdk_<qir_version>.tar.gz
cd <qir_decompressed_path>/qirp-sdk
source setup.sh
# build sample
cd <qir_decompressed_path>/qirp-sdk/qirp-samples/robotics/simulation_remote_assistant
> colcon build
> ```

Package and push the sample application to the device.

# package and push build result of sample
cd <qir_decompressed_path>/qirp-sdk/qirp-amples/robotics/simulation_remote_assistant/install/simulation_remote_assistant
tar -czf simulation_remote_assistant.tar.gz lib share
scp simulation_remote_assistant.tar.gz root@[ip-addr]:/opt/

On the development kit, run the following commands:
1. Install the sample application.
  # install sample package (ssh) tar --no-overwrite-dir --no-same-owner -zxf /opt/simulation_remote_assistant.tar.gz -C /usr/ros/jazzy/
2. Run the sample application with the steps in Run out-of-the-box simulation_remote_assistant.

​Pipeline flow for Simulation Remote Assistant

​ROS nodes used in the simulation_remote_assistant pipeline

​ROS topics used in the Simulation Remote Assistant pipeline

​Prerequisites

​Set up the simulation environment

​Run out-of-the-box simulation_remote_assistant

​Build and run simulation_remote_assistant

Pipeline flow for Simulation Remote Assistant

ROS nodes used in the simulation_remote_assistant pipeline

ROS topics used in the Simulation Remote Assistant pipeline

Prerequisites

Set up the simulation environment

Run out-of-the-box `simulation_remote_assistant`

Build and run `simulation_remote_assistant`