Segment objects with sample_object_segmentation - Qualcomm Dragonwing Documentation

The sample_object_segmentation sample application detects and segments objects in images using ROS 2 nodes. It uses Qualcomm® AI Engine Direct SDK (QNN) for model inference. The sample application uses the Ultralytics YOLOv8 model to predict bounding boxes, segmentation masks, and classes of objects in an image.

NoteFor more information, see sample_object_segmentation on GitHub.

Figure: Sample effects of sample_object_segmentation

Pipeline flow for `sample_object_segmentation`

Figure: Pipeline flow for sample_object_segmentation

ROS nodes used in `sample_object_segmentation`

ROS node	Description
qrb_ros_camera	Qualcomm ROS 2 package that captures images with parameters and publishes them to ROS topics.
yolo_preprocess	Subscribes to image data, reshapes and resizes it, and republishes it to a downstream topic.
qrb_ros_nn_inference	Loads a trained AI model, receives preprocessed images, performs inference, and publishes results.
yolo_postprocess	Matches inference output with YOLO label files.
yolo_overlay	Subscribes to YOLO postprocess and image data, and shows the object segmentation results as a ROS topic.

ROS topics used in `sample_object_segmentation`

ROS Topic	Type	Published By
`/cam0_stream1`	`sensor_msgs.msg.Image`	`camera_node`
`/qrb_inference_input_tensor`	`qrb_ros_tensor_list_msgs.msg.TensorList`	`yolo_preprocess_node`
`/resized_image`	`sensor_msgs.msg.Image`	`yolo_preprocess_node`
`/yolo_segment_tensor_output`	`qrb_ros_tensor_list_msgs.msg.TensorList`	`nn_inference_node`
`/yolo_segment_result`	`qrb_ros_vision_msgs.msg.Detection2DWithMaskArray`	`yolo_segment_postprocess_node`
`/yolo_segment_overlay`	`sensor_msgs.msg.Image`	`yolo_segment_overlay_node`

Noteyolo_segment_overlay_node synchronizes two inputs, /yolo_segment_result and /resized_image, using an exact-time policy. Both must be publishing for /yolo_segment_overlay to be produced.

Prerequisites

You have completed the settings in Set up the environment for running sample applications.
You have generated the YOLOv8 model using QAI-Hub on the host computer with the following steps.

Generate YOLOv8 model

Set up QAI-Hub on the host computer.

Install QAI-Hub.

## It is recommended to use python venv to avoid impacting the host environment.
python3 -m venv venv_qaihub
source venv_qaihub/bin/activate
## Install QAI-Hub related python packages.
pip3 install qai-hub
## Configure the QAI-Hub token, replacing xxx with your own token obtained from the QAI-Hub page.
qai-hub configure --api_token xxx
pip3 install "qai-hub-models[yolov8_seg]"

Export the model on the host computer.

# For IQ-9075, after executing the following commands, the model file and coco8.yaml are generated in the host computer.
python -m qai_hub_models.models.yolov8_seg.export --target-runtime "tflite" --chipset "qualcomm-qcs9075-proxy" --skip-profiling --skip-inferencing

SSH Session

# Push model files to the device's default path according to the preceding export log.
# On the device, create the path if it doesn't exist.
mkdir /opt/model

# On the host computer, scp all model files to the device.
scp <Host_Export_model_Path>/yolov8_det_float/yolov8_seg.tflite root@<device_ip>:/opt/model
scp <Host_Export_model_Path>/coco8.yaml root@<device_ip>:/opt/

For more information, see qrb_ros_tensor_process on GitHub.

Run out-of-the-box `sample_object_segmentation`

Try me

Run the sample on the development kit

SSH Session

# Set up the runtime environment
source /usr/share/qirp-setup.sh -m
# Set the ROS_DOMAIN_ID
export ROS_DOMAIN_ID=123
# Launch the sample object segmentation
ros2 launch sample_object_segmentation launch_with_qrb_ros_camera.py model:=/opt/model/yolov8_seg.tflite

Check the results on the host computer

In a terminal of the host computer, check the ROS topics with the name /yolo_segment_overlay in rqt.

Build and run `sample_object_segmentation`

The following steps build the sample_object_segmentation package using Dragonwing IQ-9075 Evaluation Kit as an example.

Try me

Build and package on the host computer

Build the sample application project.

cd <decompressed_workspace>/images/iq-9075-evk/qirpsdk_artifacts/iq-9075-evk/
tar -zxvf qirp-sdk_<qirp_version>.tar.gz
cd qirp-sdk
source setup.sh
# build sample
cd <qirp_decompressed_path>/qirp-samples/ai_vision/sample_object_segmentation
colcon build

Package and push the sample application to the device.

# package and push build result of sample
cd <qirp_decompressed_path>/qirp-samples/ai_vision/sample_object_segmentation/install/sample_object_segmentation
tar -czvf sample_object_segmentation.tar.gz lib share
scp sample_object_segmentation.tar.gz root@[ip-addr]:/opt/

Install and run on the development kit

Install the sample application.

SSH Session

# Install sample package
tar --no-overwrite-dir --no-same-owner -zxf /opt/sample_object_segmentation.tar.gz -C /usr/ros/jazzy/

Run the sample application with the steps in Run out-of-the-box sample_object_segmentation.

​Pipeline flow for sample_object_segmentation

​ROS nodes used in sample_object_segmentation

​ROS topics used in sample_object_segmentation

​Prerequisites

​Generate YOLOv8 model

​Run out-of-the-box sample_object_segmentation

​Build and run sample_object_segmentation

Pipeline flow for `sample_object_segmentation`

ROS nodes used in `sample_object_segmentation`

ROS topics used in `sample_object_segmentation`

Prerequisites

Generate YOLOv8 model

Run out-of-the-box `sample_object_segmentation`

Build and run `sample_object_segmentation`