Segment objects with sample_object_segmentation - Qualcomm Dragonwing Documentation

Ultralytics YOLOv8 is a machine learning model that predicts bounding boxes, segmentation masks and classes of objects in an image.

`sample_object_segmentation` pipeline flow

The following figure shows the pipeline flow for sample_object_segmentation.

Object segmentation pipeline from camera capture through YOLO preprocessing, inference, postprocessing, and overlay to the segmentation result. — sample_object_segmentation pipeline.

ROS nodes used in the `sample_object_segmentation` pipeline

The following table lists the ROS nodes used in the sample_object_segmentation pipeline.

Node name	Description
qrb ros camera	Qualcomm ROS 2 package that captures images with parameters and publishes them to ROS topics.
yolo preprocess	Subscribes to image data, reshapes/resizes it, and republishes it to a downstream topic.
qrb ros nn interface	Loads a trained AI model, receives preprocessed images, performs inference, and publishes results.
yolo postprocess	Matches inference output with yolo label files.
yolo overlay	Subscribes to yolo postprocess and image data, shows the object detection results with a ROS topic.

ROS topics used in the `sample_object_segmentation`

The following table lists the ROS topics used in the sample_object_segmentation pipeline.

ROS topic	Type	Published by
`/camera/color/image_raw`	`< sensor_msgs.msg.Image>`	`orbbec_camera`
`/qrb_inference_input_tensor`	`< qrb_ros_tensor_list_msgs.msg.TensorList>`	`yolo_preprocess_node`
`/yolo_segment_result`	`<vision_msgs.msg.Detection2DArray>`	`nn_inference_node`
`/yolo_segment_tensor_output`	`<qrb_ros_tensor_list_msgs.msg.TensorList>`	`yolo_segment_postprocess_node`
`/yolo_segment_overlay`	`<sensor_msgs.msg.Image>`	`yolo_segment_overlay_node`

Prerequisites

You have set up the device, installed ROS2 Jazzy and Qualcomm Intelligent Robotics (QIR) SDK on the device according to Install the QIR SDK.
On the host computer, you have downloaded and built the yolo model according to steps in the README of qrb_ros_tensor_process.
When downloading the yolo model, use target-runtime and device as follows:

Generate YOLOv8 model

Set up the QAI-Hub on the host computer.

a. Sign in to QAI-Hub and get your API token.b. Install QAI-Hub on the host computer.

 ## It is recommended to use python venv to avoid impacting the host environment.
 python3 -m venv venv_qaihub
 source venv_qaihub/bin/activate

 ## Install QAI-Hub related python packages.
 pip3 install qai-hub

 ## Configure QAI-Hub token, replacing xxx with your own token got from QAI-Hub page.
 qai-hub configure --api_token xxx

 pip3 install "qai-hub-models[yolov8-seg]"

Export the model on the host computer.

After executing the following commands, the model file and coco8.yaml are generated in the host computer.

python -m qai_hub_models.models.yolov8_seg.export --target-runtime "tflite" --chipset "qualcomm-qcs9075" --skip-profiling --skip-inferencing

# Push model files to the device's default path according to the preceding export log.

# On the device, create the path if it doesn't exist.
mkdir /opt/model

# On the host computer, scp all model files to the device.
scp <Host_Export_model_Path>/yolov8_seg.tflite ubuntu@<device_ip>:/opt/model

sudo find / -name "coco8.yaml" -print -quit
scp <label-file-path>/coco8.yaml ubuntu@<device_ip>:/opt/

python -m qai_hub_models.models.yolov8_seg.export --target-runtime "tflite" --chipset "qualcomm-qcs8275" --skip-profiling --skip-inferencing

# Push model files to the device's default path according to the preceding export log.

# On the device, create the path if it doesn't exist.
mkdir /opt/model

# On the host computer, scp all model files to the device.
scp <Host_Export_model_Path>/yolov8_seg.tflite ubuntu@<device_ip>:/opt/model

sudo find / -name "coco8.yaml" -print -quit
scp <label-file-path>/coco8.yaml ubuntu@<device_ip>:/opt/

Run out-of-the-box `sample_object_segmentation`

On the device, run the sample application:

Run the sample application

source /opt/ros/jazzy/setup.bash
ros2 launch sample_object_segmentation launch_with_qrb_ros_camera.py model:=/opt/model/yolov8_seg.tflite

Now, you can check ROS topics with the name /yolo_segment_overlay in rviz2.

Build from the source of `sample_object_segmentation`

Ensure that you run the Prerequisites first before performing the following steps.

Download the source code from the qrb_ros_samples repository

Download the source code

mkdir -p ~/qrb_ros_ws/src && cd ~/qrb_ros_ws/src
git clone https://github.com/qualcomm-qrb-ros/qrb_ros_samples.git

Build the sample_sample_segmentation sample application from the source code

Build the sample application

cd ~/qrb_ros_ws/src/qrb_ros_samples/ai_vision/sample_object_segmentation
colcon build
source install/setup.bash

Run and test the sample application

Run and test according to steps 2–3 of Run out-of-the-box sample_object_segmentation.

​sample_object_segmentation pipeline flow

​ROS nodes used in the sample_object_segmentation pipeline

​ROS topics used in the sample_object_segmentation

​Prerequisites

​Generate YOLOv8 model

​Run out-of-the-box sample_object_segmentation

​Build from the source of sample_object_segmentation

`sample_object_segmentation` pipeline flow

ROS nodes used in the `sample_object_segmentation` pipeline

ROS topics used in the `sample_object_segmentation`

Prerequisites

Generate YOLOv8 model

Run out-of-the-box `sample_object_segmentation`

Build from the source of `sample_object_segmentation`