Experience remote assistant in the simulator with simulation_remote_assistant

Users can interact with the robot by inputting natural language commands, such as “Go to the office to check the person.” The robot then autonomously goes to the specified location and perform object detection tasks as instructed. The following figure shows the autonomous mobile robot (AMR) running the remote assistant sample:

Simulated office in Gazebo where the robot navigates to a desk area while a terminal logs the navigation goal and the detection task. — Remote assistant running result in a simulated office.

Pipeline flow

The following figure shows the pipeline of simulation_remote_assistant:

Pipeline in three stages: build map, navigation preparation, and task execution with the YOLO object detection nodes. — simulation-remote-assistant pipeline.

ROS nodes used in the `simulation_remote_assistant` pipeline

The following table describes the ROS nodes used in the simulation_remote_assistant pipeline.

Node name	Description
`qrb_ros_simulation`	Sets up the Qualcomm® Robotics simulation environment. See `qrb_ros_simulation`.
`nn_inference_node`	Loads a trained AI model, receives preprocessed images, performs inference, and publishes results. See nn_inference_node on GitHub.
`yolo_preprocess_node`	Runs preprocessing and postprocessing for the Yolo model. See qrb_ros_tensor_process on GitHub.
`yolo_detection_postprocess_node`	Runs preprocessing and postprocessing for the Yolo model. See qrb_ros_tensor_process on GitHub.
`yolo_detection_overlay_node`	Runs preprocessing and postprocessing for the Yolo model. See qrb_ros_tensor_process on GitHub.
`cartographer_node`	Processes sensor data to perform simultaneous localization and mapping (SLAM).
`cartographer_occupancy_grid_node`	Publishes a ROS occupancy grid map.
`nav2_bringup`	The Navigation2 stack for robot navigation.
`build_map_node`	Builds the map of office world.
`nav_preparation_node`	Relocalization and launches the navigation node.
`task_manager_node`	Parses the input commands and runs the action.

ROS topics used in the `simulation_remote_assistant` pipeline

The following table describes the ROS topics used in the simulation_remote_assistant pipeline.

ROS topic	Type	Description
`/scan`	`<sensor_msgs::msg::LaserScan>`	2D lidar data for SLAM and mapping.
`/map`	`<nav_msgs::msg::OccupancyGrid>`	Occupancy grid map for navigation.
`/tf`	`<tf2_msgs::msg::TFMessage>`	Transforms between coordinate frames.
`/odom`	`<nav_msgs::msg::Odometry>`	Odometry data for pose estimation.
`/cmd_vel`	`<geometry_msgs::msg::Twist>`	Velocity commands for robot movement.
`/goal_pose`	`<geometry_msgs::msg::PoseStamped>`	Target goal pose for navigation.
`/camera/color/image_raw`	`<sensor_msgs::msg::Image>`	RGB image from simulation camera.
`/qrb_inference_input_tensor`	`<qrb_ros_tensor_list_msgs::msg::TensorList>`	`yolo_preprocess_node` preprocessed tensor.
`/yolo_detect_result`	`<vision_msgs::msg::Detection2DArray>`	`nn_inference_node` publishes the detected result.
`/yolo_detect_tensor_output`	`<qrb_ros_tensor_list_msgs::msg::TensorList>`	`yolo_detection_postprocess_node` postprocessed tensor.
`/yolo_detect_overlay`	`<sensor_msgs::msg::Image>`	The detected result with the bounding box.

Prerequisites

You have set up the device, installed ROS2 Jazzy and the Qualcomm Intelligent Robotics (QIR) SDK on the device according to Install the QIR SDK.

Run out-of-the-box `simulation_remote_assistant`

Set up the simulation sample environment on host computer

a. Prepare the simulation environment according to the qrb_ros_simulation README.b. Set up the simulation office world.

Set up environment

source /opt/ros/jazzy/setup.bash
export ROS_DOMAIN_ID=78       

Launch the simulation office world

ros2 launch qrb_ros_sim_gazebo gazebo_robot_base_mini.launch.py \
   world_model:=office \
   initial_x:=1.0 \
   initial_y:=6.0 \
   enable_depth_camera:=false

Install the simulation-remote-assistant packages

sudo apt install ros-jazzy-simulation-remote-assistant

Download the yolo object detection model

a. Build and download the yolo model according to qrb_ros_tensor_process README.b. Run the following commands:

Prepare the model and move to the default model path

sudo chmod -R 777 /opt/
mkdir /opt/model/
mv coco8.yaml yolov8_det_qcs9075.bin yolov8_det_qcs8275.bin /opt/model/

Run the remote assistant sample on the device

a. Set up the environment and run the map_nav_setup.launch.py script for auto mapping.

Set up the environment

source /opt/ros/jazzy/setup.bash
export ROS_DOMAIN_ID=78

Launch the map_nav_setup.launch.py scripts

ros2 launch simulation_remote_assistant map_nav_setup.launch.py

b. Launch the yolo object detection script.

ros2 launch simulation_remote_assistant yolo_detectcion.launch.py model:=/opt/model/yolov8_det_qcs8275.bin backend_option:=libQnnHtp.so label_file:=/opt/model/coco8.yaml    

ros2 launch simulation_remote_assistant yolo_detectcion.launch.py model:=/opt/model/yolov8_det_qcs9075.bin backend_option:=libQnnHtp.so label_file:=/opt/model/coco8.yaml       

c. Run the task manager to parse the location and object.

ros2 run simulation_remote_assistant task_manager_node

d. Input the task commands in terminal.

go to office to check person

Build from the source of `simulation_remote_assistant`

Build simulation_remote_assistant from source when you need to change the sample code. The device steps install the dependencies, clone the source repository, and build the package.

Device steps

Install the dependencies

sudo apt install ros-dev-tools

Download source code from qrb-ros-sample repository

mkdir -p ~/qrb_ros_sample_ws/src && cd ~/qrb_ros_sample_ws/src
git clone -b jazzy-rel https://github.com/qualcomm-qrb-ros/qrb_ros_samples.git

Build sample from source code

cd ~/qrb_ros_sample_ws/src/qrb_ros_samples/robotics/simulation_remote_assistant
rosdep install -i --from-path ./ --rosdistro jazzy -y
colcon build
source install/setup.bash

Run and test the sample

Run and test according to steps 2-4 of Run out-of-the-box simulation_remote_assistant.

​Pipeline flow

​ROS nodes used in the simulation_remote_assistant pipeline

​ROS topics used in the simulation_remote_assistant pipeline

​Prerequisites

​Run out-of-the-box simulation_remote_assistant

​Build from the source of simulation_remote_assistant

​Device steps