Classify images with sample_resnet101 - Qualcomm Dragonwing Documentation

The following figure shows an example image classification result.

Live camera view of sunglasses with a pink label and a terminal window echoing the classification result on the /resnet101_output topic. — Image classification result.

ResNet101 is a machine learning model that can classify images from the ImageNet data set. It can also be used as a base network in building more complex models for specific use cases.

Image classification pipeline flow

The following figure shows the pipeline flow for the image classification sample application.

Image classification pipeline running from camera capture and image publishing through preprocessing, inference, and postprocessing to labeled output. — Image classification pipeline.

ROS nodes used in the image classification pipeline

The following table lists the ROS nodes used in the image classification pipeline.

Node name	Description
qrb ros camera	Qualcomm ROS 2 package that captures images with parameters and publishes them to ROS topics.
Image publisher	Publishes image data to a ROS topic—can be camera frames, local files, or processed outputs.
Image classification preprocess	Subscribes to image data, reshapes and resizes it, and republishes it to a downstream topic.
qrb ros nn interface	Loads a trained AI model, receives preprocessed images, performs inference, and publishes results.
Image classification postprocess	Transforms raw inference outputs into human-readable results by mapping predicted indices to corresponding labels.

ROS topics used in the image classification pipeline

The following table lists the ROS topics used in the image classification pipeline.

ROS topic	Type	Description
`/image_raw`	`<sensor_msgs.msg.Image>`	Publishes image information.
`/qrb_inference_input_tensor`	`<qrb_ros_tensor_list_msgs.msg.TensorList>`	Preprocesses messages.
`/qrb_inference_output_tensor`	`<qrb_ros_tensor_list_msgs.msg.TensorList>`	This is the nn interface result with model.
`/resnet101_results`	`<sensor_msgs.msg.String>`	This is the model output label.

Prerequisites

You have set up the device, installed ROS2 Jazzy and Qualcomm Intelligent Robotics (QIR) SDK on the device according to Install the QIR SDK.

Run out-of-the-box image classification

Install the image classification packages

Install the packages

sudo apt install ros-jazzy-sample-resnet101

Set up the sample environment on the device

Launch the sample application

source /opt/ros/jazzy/setup.bash
ros2 launch sample_resnet101 launch_with_image_publisher.py
or # You can also replace this with a custom image file
ros2 launch sample_resnet101 launch_with_image_publisher.py image_path:=<your image path>
or # You can launch with qrb ros camera
ros2 launch sample_resnet101 launch_with_qrb_ros_camera.py

The launch_with_image_publisher.py launch script uses the default parameters:

DeclareLaunchArgument(
'image_path',
default_value=os.path.join(package_path, 'glasses.jpg'),
description='Path to the image file'
)
# Node for image_publisher
image_publisher_node = Node(
package='image_publisher',
executable='image_publisher_node',
namespace=namespace,
name='image_publisher_node',
output='screen',
parameters=[
{'filename': image_path},
{'rate': 10.0},  # Set the publishing rate to 10 Hz
]
)

a. It sends the local glasses.jpg file, and outputs the image at 10 Hz.b. Then, you can check ROS topics with the name/resnet101_output in another shell terminal.

ros2 topic echo /resnet101_output

Output

data: 'sunglass

Build from the source of image classification

Build the image classification sample from source when you need to change the sample code. The device steps install the dependencies, clone the source repository, and build the package.

Device steps

Install the dependencies

sudo apt install ros-jazzy-rclpy \
ros-jazzy-sensor-msgs \
ros-jazzy-std-msgs \
ros-jazzy-cv-bridge \
ros-jazzy-ament-index-python \
ros-jazzy-qrb-ros-tensor-list-msgs \
python3-opencv \
python3-numpy \
ros-jazzy-image-publisher \
ros-jazzy-qrb-ros-nn-inference \
ros-jazzy-qrb-ros-camera \
ros-jazzy-image-publisher

Download the source code from the qrb_ros_samples repository

mkdir -p ~/qrb_ros_ws/src && cd ~/qrb_ros_ws/src
git clone https://github.com/qualcomm-qrb-ros/qrb_ros_samples.git

Build the object_detection sample application from the source code

Build the sample application

cd ~/qrb_ros_ws/src/qrb_ros_samples/ai_vision/sample_resnet101
colcon build
source install/setup.bash

Run and test the sample application

Run and test according to step 2 of Run out-of-the-box image classification.

​Image classification pipeline flow

​ROS nodes used in the image classification pipeline

​ROS topics used in the image classification pipeline

​Prerequisites

​Run out-of-the-box image classification

​Build from the source of image classification

​Device steps