> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.
# 使用 qrb_ros_nn_inference 实现模型推理
> qrb_ros_nn_inference 是一个用于执行神经网络模型的 ROS2 包，为机器人应用提供基于 AI 的感知能力。
`qrb_ros_nn_inference` 包为机器人应用提供基于 AI 的感知能力。它提供以下功能：
* 支持 `.tflite`、`.so`、`.bin` 三种模型格式的模型推理 API
* 基于 Qualcomm 平台的模型推理加速
## `qrb_ros_nn_inference` 的架构
下图展示了 `qrb_ros_nn_interface` 的架构。
`qrb_ros_nn_inference` 是基于 [qrb\_inference\_manager](https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference/blob/main/qrb_inference_manager/README.md) 的 ROS2 包，后者是一个封装了 [Qualcomm AI Engine Direct](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-50/overview.html) 和 [Qualcomm Neural Network (QNN) Delegate for TensorFlow Lite](https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/overview.html) API 的 C++ 库。
`qrb_ros_nn_inference` 从特定话题接收数据，然后不做任何处理直接使用接收到的数据进行模型推理。随后，模型推理的结果会通过另一个特定话题直接发送出去。
## ROS 节点参数
下表描述了 `qrb_ros_nn_inference` 的 ROS 节点参数。
| 参数 | 类型 | 默认值 | 说明 |
| ---------------- | -------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `backend_option` | `String` | `""` | 关于模型推理的硬件加速选项和有效值，请参阅 [qrb\_inference\_manager documentation](https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference/blob/main/qrb_inference_manager/Documentation.md)。 |
| `model_path` | `String` | `""` | 模型文件路径。 |
## ROS 话题
下表描述了 `qrb_ros_nn_inference` 使用的 ROS 话题。
| 话题名 | 消息类型 | 说明 |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ----- |
| `qrb_inference_input_tensor` | [`TensorList`](https://github.com/qualcomm-qrb-ros/qrb_ros_interfaces/blob/main/qrb_ros_tensor_list_msgs/msg/TensorList.msg) | 订阅的话题 |
| `qrb_inference_output_tensor` | [`TensorList`](https://github.com/qualcomm-qrb-ros/qrb_ros_interfaces/blob/main/qrb_ros_tensor_list_msgs/msg/TensorList.msg) | 发布的话题 |
## 前提条件
你已根据[安装 QIR SDK](./install-the-qir-sdk)完成设备设置，并在设备上安装了 ROS2 Jazzy 和 Qualcomm Intelligent Robotics (QIR) SDK。
## 开箱即用运行 `qrb_ros_nn_inference`
```bash Install the packages theme={null}
sudo apt install ros-jazzy-qrb-ros-nn-inference
```
```bash Download the test nodes theme={null}
# qrb_ros_nn_inference/test includes the pre-process node and post-process node
mkdir -p ~/ros-ws/src && cd ~/ros-ws/src && \
git clone https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference && \
```
a. 按照 [QC AI hub Getting Started](https://app.aihub.qualcomm.com/docs/hub/getting_started.html) 下载 `yolov8.tflite` 模型。
b. 下载用于目标检测的测试图像。
```bash theme={null}
wget -P ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_pre_process/image/ \
https://ultralytics.com/images/bus.jpg && \
python3 ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_post_process/scripts/yolov8_input_pre_process.py
```
c. 在 `~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_post_process/launch/nn_node_test.launch.py` 中指出原始图像路径和模型路径。
```python title="nn_node_test.launch.py" theme={null}
pre_process_node = ComposableNode(
package = "qrb_ros_pre_process",
plugin = "qrb_ros::pre_process::QrbRosPreProcessNode",
name = "pre_process_node",
parameters=[
{
"image_path": os.environ['HOME'] + "/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_pre_process/image/bus.raw"
}
]
)
nn_inference_node = ComposableNode(
package = "qrb_ros_nn_inference",
plugin = "qrb_ros::nn_inference::QrbRosInferenceNode",
name = "nn_inference_node",
parameters=[
{
"backend_option": "",
"model_path": "/path/to/model"
}
]
)
```
d. 构建预处理节点和后处理节点。
```bash Build and run the inference theme={null}
source /opt/ros/jazzy/setup.bash && \
cd ~/ros-ws && \
rm ./src/qrb_ros_nn_inference/test/qrb_ros_post_process/COLCON_IGNORE && \
rm ./src/qrb_ros_nn_inference/test/qrb_ros_pre_process/COLCON_IGNORE && \
colcon build --packages-select qrb_ros_pre_process qrb_ros_post_process
execute the inference
cd ~/ros-ws && \
source install/local_setup.bash && \
ros2 launch qrb_ros_post_process nn_node_test.launch.py
visualize the detection result
python3 ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_post_process/scripts/qrb_ros_yolo_detection_visualizer.py \
--original_image ~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_pre_process/image/bus.jpg
```
输出结果图像位于 `~/ros-ws/src/qrb_ros_nn_inference/test/qrb_ros_post_process/inference_result` 中。
## 构建并运行 `qrb_ros_nn_inference`
```bash Install the dependencies theme={null}
sudo apt install -y software-properties-common colcon
sudo apt install -y libtensorflow-lite-c-qcom1 libtensorflow-lite-qcom-dev libqnn-dev libqnn1
```
```bash Download the source code theme={null}
mkdir -p ~/qrb_ros_ws/src
cd ~/qrb_ros_ws/src
git clone https://github.com/qualcomm-qrb-ros/qrb_ros_nn_inference
git clone https://github.com/qualcomm-qrb-ros/qrb_ros_interfaces
```
```bash Build and launch theme={null}
cd ~/qrb_ros_ws
colcon build --packages-up-to qrb_ros_nn_inference
source install/setup.bash
ros2 launch qrb_ros_post_process nn_node_test.launch.py
```