Implement model inference with qrb_ros_nn_inference - Qualcomm Dragonwing Documentation

The qrb_ros_nn_inference package provides AI-based perception for robotics applications. It provides the following:

Model inference API which supports three model formats: .tflite, .so, .bin
Model inference acceleration based on Qualcomm platforms

Architecture of `qrb_ros_nn_inference`

The following figure shows the architecture of qrb_ros_nn_interface.

Layered stack: application, qrb_ros_nn_inference node, QrbInferenceManager, QNN and TFLite SDKs, running on CPU, GPU, and Hexagon Tensor Processor. — Architecture of qrb_ros_nn_inference.

qrb_ros_nn_inference is a ROS2 package based on qrb_inference_manager, which is a C++ library encapsulating the APIs of Qualcomm AI Engine Direct and Qualcomm Neural Network (QNN) Delegate for TensorFlow Lite. qrb_ros_nn_inference receives data from a specific topic, then without any processing, directly uses the received data for model inference. Then, the results of the model inference are sent out directly through another specific topic.

ROS node parameters

The following table describes the ROS node parameters for qrb_ros_nn_inference.

Parameter	Type	Default value	Description
`backend_option`	`String`	`""`	For the hardware acceleration options for model inference and valid values, see qrb_inference_manager documentation.
`model_path`	`String`	`""`	Path of model file.

ROS topics

The following table describes the ROS topics used by qrb_ros_nn_inference.

Topic name	Message type	Description
`qrb_inference_input_tensor`	`TensorList`	Subscribed topic
`qrb_inference_output_tensor`	`TensorList`	Published topic

Prerequisites

You have set up the device, installed ROS2 Jazzy and Qualcomm Intelligent Robotics (QIR) SDK on the device according to Install the QIR SDK.

Run out-of-the-box `qrb_ros_nn_inference`

Install the qrb_ros_transport packages

Install the packages

sudo apt install ros-jazzy-qrb-ros-nn-inference

Prepare the preprocess node and postprocess node for model inference

Download the test nodes

# qrb_ros_nn_inference/test includes the pre-process node and post-process node
mkdir -p ~/ros-ws/src && cd ~/ros-ws/src && \
git clone https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference && \

Test qrb_ros_nn_inference with the YOLOv8 detection model

a. Download the yolov8.tflite model by following QC AI hub Getting Started.b. Download the test image for object detection.

wget -P ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_pre_process/image/ \
https://ultralytics.com/images/bus.jpg && \
python3 ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_post_process/scripts/yolov8_input_pre_process.py

c. Point out the raw image path and model path in ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_post_process/launch/nn_node_test.launch.py.

nn_node_test.launch.py

pre_process_node = ComposableNode(
   package = "qrb_ros_pre_process",
   plugin = "qrb_ros::pre_process::QrbRosPreProcessNode",
   name = "pre_process_node",
   parameters=[
     {
       "image_path": os.environ['HOME'] + "/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_pre_process/image/bus.raw"
     }
   ]
)
nn_inference_node = ComposableNode(
   package = "qrb_ros_nn_inference",
   plugin = "qrb_ros::nn_inference::QrbRosInferenceNode",
   name = "nn_inference_node",
   parameters=[
     {
       "backend_option": "",
       "model_path": "/path/to/model"
     }
   ]
)

d. Build the preprocess node and postprocess node.

Build and run the inference

source /opt/ros/jazzy/setup.bash && \
  cd ~/ros-ws && \
  rm ./src/qrb_ros_nn_inference/test/qrb_ros_post_process/COLCON_IGNORE && \
  rm ./src/qrb_ros_nn_inference/test/qrb_ros_pre_process/COLCON_IGNORE && \
colcon build --packages-select qrb_ros_pre_process qrb_ros_post_process
execute the inference
  cd ~/ros-ws && \
source install/local_setup.bash && \
ros2 launch qrb_ros_post_process nn_node_test.launch.py
visualize the detection result
python3 ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_post_process/scripts/qrb_ros_yolo_detection_visualizer.py \
  --original_image ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_pre_process/image/bus.jpg

The output result image is in ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_post_process/inference_result.

Build and run `qrb_ros_nn_inference`

Install dependencies

Install the dependencies

sudo apt install -y software-properties-common colcon
sudo apt install -y libtensorflow-lite-c-qcom1 libtensorflow-lite-qcom-dev libqnn-dev libqnn1

Clone the repository to the device

Download the source code

mkdir -p ~/qrb_ros_ws/src
cd ~/qrb_ros_ws/src
git clone https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference
git clone https://github.com/qualcomm-qrb-ros/qrb_ros_interfaces

Build and run qrb_ros_nn_inference

Build and launch

cd ~/qrb_ros_ws
colcon build --packages-up-to qrb_ros_nn_inference
source install/setup.bash
ros2 launch qrb_ros_post_process nn_node_test.launch.py

​Architecture of qrb_ros_nn_inference

​ROS node parameters

​ROS topics

​Prerequisites

​Run out-of-the-box qrb_ros_nn_inference

​Build and run qrb_ros_nn_inference