Classify images with sample_resnet101 - Qualcomm Dragonwing Documentation

The sample_resnet101 sample application classifies images using a ResNet-101 model with ROS 2 nodes. It uses Qualcomm® AI Engine Direct SDK (QNN) for model inference. resnet101 is a machine learning model that can classify images from the ImageNet dataset. It also serves as the basis for building more advanced models tailored to specific use cases.

NoteFor more information, see sample_resnet101 on GitHub.

Figure: Sample effects of sample_resnet101

Pipeline flow for `sample_resnet101`

Figure: Pipeline flow for sample_resnet101

ROS nodes used in `sample_resnet101`

ROS node	Description
qrb_ros_camera	Qualcomm ROS 2 package that captures images with parameters and publishes them to ROS topics. Runs as node `camera_node`.
`qrb_ros_resnet101`	Preprocess node (`resnet101_node`) from the `sample_resnet101` package. Subscribes to the image topic, converts and resizes it to 224x224x3, and publishes a tensor to `/qrb_inference_input_tensor`.
qrb_ros_nn_inference	Loads the `ResNet101_w8a8.bin` model, receives the preprocessed tensor, runs inference on the QNN HTP backend, and publishes the output tensor. Runs as node `nn_inference_node`.
`qrb_ros_resnet101_posprocess`	Postprocess node (`resnet101_postprocess_node`) from the `sample_resnet101` package. Maps the output tensor to an ImageNet label and publishes the classification result.
image_publisher	ROS 2 Jazzy package that publishes the image ROS topic with a local path. Used instead of `qrb_ros_camera` when the sample is launched with `launch_with_image_publisher.py`.

ROS topics used in `sample_resnet101`

ROS topic	Type	Published by
`/cam0_stream1`	`sensor_msgs.msg.Image`	`camera_node`
`/image_raw`	`sensor_msgs.msg.Image`	`image_publisher`
`/qrb_inference_input_tensor`	`qrb_ros_tensor_list_msgs.msg.TensorList`	`resnet101_node`
`/qrb_inference_output_tensor`	`qrb_ros_tensor_list_msgs.msg.TensorList`	`nn_inference_node`
`/resnet101_output`	`std_msgs.msg.String`	`resnet101_postprocess_node`

NoteThe preprocess node subscribes to image_raw. With launch_with_qrb_ros_camera.py this subscription is remapped to /cam0_stream1; with launch_with_image_publisher.py it keeps the default /image_raw.

Prerequisites

You have completed the settings in Set up the environment for running sample applications.

Run out-of-the-box `sample_resnet101`

Try me

Run the sample on the development kit

SSH Session

# Set up the runtime environment
source /usr/share/qirp-setup.sh -m
# Set the ROS_DOMAIN_ID
export ROS_DOMAIN_ID=123
ros2 launch sample_resnet101 launch_with_qrb_ros_camera.py

Check the classification result on the host computer

SSH Session

export ROS_DOMAIN_ID=123
ros2 topic echo /resnet101_output

Build and run `sample_resnet101`

The following steps build the sample_resnet101 package using Dragonwing IQ-9075 Evaluation Kit as an example.

Try me

Build and package on the host computer

Build the sample application project.

cd <decompressed_workspace>/images/iq-9075-evk/qirpsdk_artifacts/iq-9075-evk/
tar -zxvf qirp-sdk_<qirp_version>.tar.gz
cd qirp-sdk
source setup.sh
# build sample
cd <qirp_decompressed_path>/qirp-samples/ai_vision/sample_resnet101
colcon build

Package and push the sample application to the device.

# package and push build result of sample
cd <qirp_decompressed_path>/qirp-samples/ai_vision/sample_resnet101/install/sample_resnet101
tar -czvf sample_resnet101.tar.gz lib share
scp sample_resnet101.tar.gz root@[ip-addr]:/opt/

Install and run on the development kit

Install the sample application.

SSH Session

# Install sample package
tar --no-overwrite-dir --no-same-owner -zxf /opt/sample_resnet101.tar.gz -C /usr/ros/jazzy/

Run the sample application with the steps in Run out-of-the-box sample_resnet101.

​Pipeline flow for sample_resnet101

​ROS nodes used in sample_resnet101

​ROS topics used in sample_resnet101

​Prerequisites

​Run out-of-the-box sample_resnet101

​Build and run sample_resnet101

Pipeline flow for `sample_resnet101`

ROS nodes used in `sample_resnet101`

ROS topics used in `sample_resnet101`

Prerequisites

Run out-of-the-box `sample_resnet101`

Build and run `sample_resnet101`