Detect hands with sample_hand_detection - Qualcomm Dragonwing Documentation

The model is sourced from the MediaPipe Hand Landmark Detector, which is a machine learning pipeline that predicts bounding boxes and pose skeletons of hands in an image. This sample application allows you to input an image named input_image.jpg, and then it publishes the result as ROS topic handlandmark_result. It uses Qualcomm AI Engine Direct SDK (QNN) for model inference.

For model information, see MediaPipe-Hand-Detection.For more information, see the sample_hand_detection GitHub repository.

The following figure shows an example hand detection result.

Hand detection result showing an open palm with a skeletal overlay and bounding boxes drawn around the hand and the palm.

`sample_hand_detection` pipeline flow

The following figure shows the pipeline flow for sample_hand_detection.

Hand detection pipeline from the image publisher through the hand detector node to the published hand landmark result topic. — sample_hand_detection pipeline.

ROS nodes used in the `sample_hand_detection` pipeline

The following table lists the ROS nodes used in the sample_hand_detection pipeline.

ROS node	Description
`qrb_ros_hand_detector`	The `qrb_ros_hand_detector` node is a python-based ROS Jazzy package that achieves image classification. It uses the QNN SDK for model inference. This ROS node subscribes to the image topic and publishes the classification result topic after the pre-post process.
`image_publisher`	`image_publisher` is a ROS Jazzy package. It publishes the image ROS topic with a local path. For more information, see image_publisher.

ROS topics used in the `sample_hand_detection` pipeline

The following table lists the ROS topics used in the sample_hand_detection pipeline.

ROS topic	Type	Published by
`/handlandmark_result`	`sensor_msgs.msg.Image`	`qrb_ros_hand_detector`
`/image_raw`	`sensor_msgs.msg.Image`	`image_publisher`

Prerequisites

You have set up the device, installed ROS2 Jazzy and Qualcomm Intelligent Robotics (QIR) SDK on the device according to Install the QIR SDK.

Run out-of-the-box `sample_hand_detection`

Create necessary directories for running the sample application

Create the output directories

sudo mkdir -p /opt/ros/jazzy/share/sample_hand_detection/output_MediaPipeHandDetector/
sudo chmod 777 /opt/ros/jazzy/share/sample_hand_detection/output_MediaPipeHandDetector/
sudo mkdir -p /opt/ros/jazzy/share/sample_hand_detection/output_MediaPipeHandLandmarkDetector
sudo chmod 777 /opt/ros/jazzy/share/sample_hand_detection/output_MediaPipeHandLandmarkDetector
sudo chmod 777 -R /opt/ros/jazzy/share/sample_hand_detection/

On the development kit, run the following commands

Launch the sample application

export ROS_DOMAIN_ID=123
source /opt/ros/jazzy/setup.bash
ros2 launch sample_hand_detection launch_with_image_publisher.py

On the host, run the following commands

a. Start rqt.

# YOUR_HOST_IP is the IP address of the Host where you want to view the sample output.
export DISPLAY=YOUR_HOST_IP:0
export ROS_DOMAIN_ID=123
rqt

b. Select the following buttons in sequence:

Plugins --> Visualization --> Image View

c. Select handlandmark_result to see the results.

​sample_hand_detection pipeline flow

​ROS nodes used in the sample_hand_detection pipeline

​ROS topics used in the sample_hand_detection pipeline

​Prerequisites

​Run out-of-the-box sample_hand_detection

`sample_hand_detection` pipeline flow

ROS nodes used in the `sample_hand_detection` pipeline

ROS topics used in the `sample_hand_detection` pipeline

Prerequisites

Run out-of-the-box `sample_hand_detection`