> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Qualcomm AI Hub
> 使用 LiteRT 或 ONNX Runtime 将 Qualcomm AI Hub 中的预训练 AI 模型部署到 Dragonwing 开发板的 NPU 上。
Qualcomm [AI Hub](https://aihub.qualcomm.com) 包含大量经过优化、可在 Dragonwing 硬件上运行的预训练 AI 模型。
## 端到端示例
以下是实现了 AI Hub 模型、可直接在 Dragonwing 开发板 NPU 上运行的示例应用程序（Python）列表：
* [Lightweight-Face-Detection](https://github.com/qualcomm/ai-hub-models/tree/main/qai_hub_models/models/face_det_lite)
要运行其他模型，请继续阅读！
## 查找支持的模型
AI Hub 中的模型按支持的 Qualcomm 芯片组分类。要查看可在您的开发套件上运行的模型：
进入[模型列表](https://aihub.qualcomm.com/iot/models)。
在 'Chipset' 下选择：
* RB3 Gen 2 Vision Kit：'Qualcomm QCS6490 (Proxy)'
* RUBIK Pi 3：'Qualcomm QCS6490 (Proxy)'
* IQ-9075 EVK：'Qualcomm QCS9075 (Proxy)'
在 'Model precision' 下选择：'Quantized'。您 Dragonwing 开发板上的 NPU 仅运行量化模型。
## 将模型部署到 NPU（Python）
作为示例，我们来部署 [Lightweight-Face-Detection](https://aihub.qualcomm.com/iot/models/face_det_lite) 模型。
### 运行示例仓库
所有 AI Hub 模型都附带一个示例仓库。这是一个很好的起点，因为它准确展示了如何*运行*该模型。它展示了神经网络的输入应该是什么样子，以及如何解释输出（在这里，是将输出张量映射为边界框）。示例仓库*尚未*在 NPU 或 GPU 上运行——它们在没有加速的情况下运行。在将这个模型迁移到 NPU 之前，先看看我们的输入/输出应该是什么样子。
在 [Lightweight-Face-Detection](https://aihub.qualcomm.com/iot/models/face_det_lite) 的 AI Hub 页面上，点击 "Model repository"。这会跳转到一个 [README](https://github.com/quic/ai-hub-models/blob/main/qai_hub_models/models/face_det_lite/README.md) 文件，其中包含运行示例仓库的说明。
要部署此模型，请打开开发板上的终端，或与开发板建立 ssh 会话：
创建一个新的 `venv` 并安装一些基础软件包：
```shell theme={null}
mkdir -p ~/aihub-demo
cd ~/aihub-demo
python3 -m venv .venv
source .venv/bin/activate
pip3 install numpy setuptools Cython shapely
```
将一张包含人脸的图像（640x480 分辨率，JPG 格式）下载到您的开发板上：
```shell theme={null}
wget https://cdn.edgeimpulse.com/qc-ai-docs/example-images/three-people-640-480.jpg
```
**输入分辨率：** AI Hub 模型要求输入尺寸正确。您可以在 "Technical Details > Input resolution" 下找到所需分辨率（格式为 *HEIGHT x WIDTH*（此处 480x640 => 宽x高为 640x480））；或检查 TFLite 或 ONNX 文件中输入张量的尺寸。
按照 Facial Landmark Detection 模型 'Example & Usage' 下的说明操作：
```shell theme={null}
# Install the example (add --no-build-isolation)
pip3 install --no-build-isolation "qai-hub-models[face-det-lite]"
# Run the example
# Use --help to see all options
python3 -m qai_hub_models.models.face_det_lite.demo --quantize w8a8 --image ./three-people-640-480.jpg --output-dir out/
```
您可以在 `out/FaceDetLitebNet_output.png` 中找到输出图像。
如果您是通过 ssh 连接的，可以通过以下方式将输出图像复制回您的主机：
```shell theme={null}
# Find IP via: ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'
# Then: (replace 192.168.1.148 by the IP address of your development kit)
scp ubuntu@192.168.1.148:~/aihub-demo/out/FaceDetLitebNet_output.png ~/Downloads/FaceDetLitebNet_output.png
```
我们已经有了一个可用的模型。作为参考，在 IQ9-EVK 上运行此模型每次推理耗时 106.86ms。
### 将模型移植到 NPU
现在我们有了一个可用的参考模型，接下来让它在 NPU 上运行。您需要实现三个部分：
1. **预处理**数据 — 将图像转换为可传递给神经网络的特征。
2. **运行推理** — 将模型导出为 ONNX 或 TFLite，并通过 [LiteRT](/zh/ai-workflows/lite-rt) 或 [ONNX Runtime](/zh/ai-workflows/onnxruntime) 运行模型。
3. **后处理**输出 — 将神经网络的输出转换为人脸边界框。
正如您在 [LiteRT](/zh/ai-workflows/lite-rt) 和 [ONNX Runtime](/zh/ai-workflows/onnxruntime) 页面中所看到的，模型本身很简单。然而，预处理和后处理代码可能就不那么简单了……
#### 预处理输入
对于图像模型，大多数 AI Hub 模型接受形状为 `(HEIGHT, WIDTH, CHANNELS)`（LiteRT）或 `(CHANNELS, HEIGHT, WIDTH)`（ONNX）、缩放到 0..1 范围的矩阵。如果只有 1 个通道，请先将图像转换为灰度图。如果您的模型是量化的（很可能是），您还需要读取 zero\_point 和 scale，并相应地缩放像素（在 LiteRT 中这很容易，因为其中包含量化参数，但 ONNX 没有这些参数）。通常，对于量化模型，您最终会得到线性缩放到 0..255（uint8）或 -128..127（int8）的数据——所以这相对容易。下面的示例代码中有一个用 Python 演示所有这些操作的函数（`def load_image_litert`）。
*但是……* 这并不是绝对的；这正是 AI Hub 示例代码的用武之地。每个 AI Hub 示例都包含用于缩放输入的确切代码。在我们当前的示例——Lightweight-Face-Detection 中，输入的形状为 `(480, 640, 1)`。然而，如果您查看[预处理代码](https://github.com/quic/ai-hub-models/blob/8cdeb11df6cc835b9b0b0cf9b602c7aa83ebfaf8/qai_hub_models/models/face_det_lite/app.py#L70)，数据并没有被转换为灰度图，而是只取了 RGB 图像的蓝色通道：
```python theme={null}
img_array = img_array.astype("float32") / 255.0
img_array = img_array[np.newaxis, ...]
img_tensor = torch.Tensor(img_array)
img_tensor = img_tensor[:, :, :, -1] # HERE WE TAKE BLUE CHANNEL, NOT CONVERT TO GRAYSCALE
```
这类细节很容易出错。因此，如果您发现自己的实现与 AI Hub 示例的结果不一致：请阅读代码。对于非图像输入（例如音频）更是如此。请使用演示代码来理解模型实际期望的输入。
#### 后处理输出
后处理也是如此。例如，没有标准的方法将神经网络的输出映射为边界框（在本例中用于检测人脸）。对于 Lightweight-Face-Detection，您可以在这里找到代码：[face\_det\_lite/app.py#L77](https://github.com/quic/ai-hub-models/blob/8cdeb11df6cc835b9b0b0cf9b602c7aa83ebfaf8/qai_hub_models/models/face_det_lite/app.py#L77)。
如果您的目标是 Python，通常最简单的方法是将后处理代码复制到您的应用程序中；因为 AI Hub 有许多您可能不想要的依赖项。此外，后处理代码基于 PyTorch 张量运算，而您的推理在 LiteRT 或 ONNX Runtime 下运行；因此，您需要做一些小的修改。我们将在下面的端到端示例中展示这一点。
### 端到端示例（Python）
说明部分已经结束，让我们来看一些代码。
在开发板上打开终端：
```shell theme={null}
# Create a new fresh directory
mkdir -p ~/aihub-npu
cd ~/aihub-npu
# Create a new venv
python3 -m venv .venv
source .venv/bin/activate
# Install the LiteRT runtime (to run models) and Pillow (to parse images)
pip3 install ai-edge-litert==1.3.0 Pillow
# Download an example image
wget https://cdn.edgeimpulse.com/qc-ai-docs/example-images/three-people-640-480.jpg
```
NPU 仅支持 uint8/int8 量化模型。幸运的是，AI Hub 已经包含预量化和优化过的模型。您可以：
* 下载本教程使用的模型（已镜像到 CDN）：
```shell theme={null}
wget https://cdn.edgeimpulse.com/qc-ai-docs/models/face_det_lite-lightweight-face-detection-w8a8.tflite
```
* 或者，对于其他任何模型，从 AI Hub 下载模型并推送到您的开发板：
1. 前往 [Lightweight-Face-Detection](https://aihub.qualcomm.com/iot/models/face_det_lite)。
2. 点击 "Download model"。
3. 运行时选择 "TFLite"，精度选择 "w8a8"。
如果您的模型仅提供 ONNX 格式，请参阅[使用 ONNX Runtime 运行模型](/zh/ai-workflows/onnxruntime)中的说明。本教程中的原则同样适用。
4. 下载模型。
5. 如果您不是直接在 Dragonwing 开发板上下载模型，请通过 ssh 推送模型：
```shell theme={null}
# Find your board's IP
ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'
# Push the .tflite file (replace IP)
scp face_det_lite-lightweight-face-detection-w8a8.tflite ubuntu@192.168.1.253:~/face_det_lite-lightweight-face-detection-w8a8.tflite
```
创建一个新文件 `face_detection.py`。该文件包含模型调用，以及来自 AI Hub 示例的预处理和后处理代码（见内联注释）。
```python theme={null}
import numpy as np
from ai_edge_litert.interpreter import Interpreter, load_delegate
from PIL import Image, ImageDraw
import os, time, sys
def curr_ms():
return round(time.time() * 1000)
# Paths
IMAGE_IN = 'three-people-640-480.jpg'
IMAGE_OUT = 'three-people-640-480-overlay.jpg'
MODEL_PATH = 'face_det_lite-lightweight-face-detection-w8a8.tflite'
# If we pass in --use-npu we offload to NPU
use_npu = True if len(sys.argv) >= 2 and sys.argv[1] == '--use-npu' else False
experimental_delegates = []
if use_npu:
experimental_delegates = [load_delegate("libQnnTFLiteDelegate.so", options={"backend_type":"htp"})]
# Load TFLite model and allocate tensors
interpreter = Interpreter(
model_path=MODEL_PATH,
experimental_delegates=experimental_delegates
)
interpreter.allocate_tensors()
# Get input and output tensor details
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# === BEGIN PREPROCESSING ===
# Load an image (using Pillow) and make it in the right format that the interpreter expects (e.g. quantize)
# All AI Hub image models use 0..1 inputs to start.
def load_image_litert(interpreter, path, single_channel_behavior: str = 'grayscale'):
d = interpreter.get_input_details()[0]
shape = [int(x) for x in d["shape"]] # e.g. [1, H, W, C] or [1, C, H, W]
dtype = d["dtype"]
scale, zp = d.get("quantization", (0.0, 0))
if len(shape) != 4 or shape[0] != 1:
raise ValueError(f"Unexpected input shape: {shape}")
# Detect layout
if shape[1] in (1, 3): # [1, C, H, W]
layout, C, H, W = "NCHW", shape[1], shape[2], shape[3]
elif shape[3] in (1, 3): # [1, H, W, C]
layout, C, H, W = "NHWC", shape[3], shape[1], shape[2]
else:
raise ValueError(f"Cannot infer layout from shape {shape}")
# Load & resize
img = Image.open(path).convert("RGB").resize((W, H), Image.BILINEAR)
arr = np.array(img)
if C == 1:
if single_channel_behavior == 'grayscale':
gray = np.asarray(Image.fromarray(arr).convert('L'))
elif single_channel_behavior in ('red', 'green', 'blue'):
ch_idx = {'red': 0, 'green': 1, 'blue': 2}[single_channel_behavior]
gray = arr[:, :, ch_idx]
else:
raise ValueError(f"Invalid single_channel_behavior: {single_channel_behavior}")
arr = gray[..., np.newaxis]
# HWC -> correct layout
if layout == "NCHW":
arr = np.transpose(arr, (2, 0, 1)) # (C,H,W)
# Scale 0..1 (all AI Hub image models use this)
arr = (arr / 255.0).astype(np.float32)
# Quantize if needed
if scale and float(scale) != 0.0:
q = np.rint(arr / float(scale) + int(zp))
if dtype == np.uint8:
arr = np.clip(q, 0, 255).astype(np.uint8)
else:
arr = np.clip(q, -128, 127).astype(np.int8)
return np.expand_dims(arr, 0) # add batch
# This model looks like grayscale, but AI Hub inference actually takes the BLUE channel
# see https://github.com/quic/ai-hub-models/blob/8cdeb11df6cc835b9b0b0cf9b602c7aa83ebfaf8/qai_hub_models/models/face_det_lite/app.py#L70
input_data = load_image_litert(interpreter, IMAGE_IN, single_channel_behavior='blue')
# === END PREPROCESSING (input_data contains right data) ===
# Set tensor and run inference
interpreter.set_tensor(input_details[0]['index'], input_data)
# Run once to warmup
interpreter.invoke()
# Then run 10x
start = curr_ms()
for i in range(0, 10):
interpreter.invoke()
end = curr_ms()
# === BEGIN POSTPROCESSING ===
# Grab 3 output tensors and dequantize
q_output_0 = interpreter.get_tensor(output_details[0]['index'])
scale_0, zero_point_0 = output_details[0]['quantization']
hm = ((q_output_0.astype(np.float32) - zero_point_0) * scale_0)[0]
q_output_1 = interpreter.get_tensor(output_details[1]['index'])
scale_1, zero_point_1 = output_details[1]['quantization']
box = ((q_output_1.astype(np.float32) - zero_point_1) * scale_1)[0]
q_output_2 = interpreter.get_tensor(output_details[2]['index'])
scale_2, zero_point_2 = output_details[2]['quantization']
landmark = ((q_output_2.astype(np.float32) - zero_point_2) * scale_2)[0]
# Taken from https://github.com/quic/ai-hub-models/blob/8cdeb11df6cc835b9b0b0cf9b602c7aa83ebfaf8/qai_hub_models/utils/bounding_box_processing.py#L369
def get_iou(boxA: np.ndarray, boxB: np.ndarray) -> float:
xA = max(boxA[0], boxB[0])
yA = max(boxA[1], boxB[1])
xB = min(boxA[2], boxB[2])
yB = min(boxA[3], boxB[3])
inter_area = max(0, xB - xA + 1) * max(0, yB - yA + 1)
boxA_area = (boxA[2] - boxA[0] + 1) * (boxA[3] - boxA[1] + 1)
boxB_area = (boxB[2] - boxB[0] + 1) * (boxB[3] - boxB[1] + 1)
return inter_area / float(boxA_area + boxB_area - inter_area)
# Taken from https://github.com/quic/ai-hub-models/blob/8cdeb11df6cc835b9b0b0cf9b602c7aa83ebfaf8/qai_hub_models/models/face_det_lite/utils.py
class BBox:
def __init__(self, label, xyrb, score=0, landmark=None, rotate=False):
self.label = label
self.score = score
self.landmark = landmark
self.x, self.y, self.r, self.b = xyrb
self.rotate = rotate
minx = min(self.x, self.r)
maxx = max(self.x, self.r)
miny = min(self.y, self.b)
maxy = max(self.y, self.b)
self.x, self.y, self.r, self.b = minx, miny, maxx, maxy
@property
def width(self): return self.r - self.x + 1
@property
def height(self): return self.b - self.y + 1
@property
def box(self): return [self.x, self.y, self.r, self.b]
@box.setter
def box(self, newvalue): self.x, self.y, self.r, self.b = newvalue
@property
def haslandmark(self): return self.landmark is not None
@property
def xywh(self): return [self.x, self.y, self.width, self.height]
def nms(objs, iou=0.5):
if objs is None or len(objs) <= 1:
return objs
objs = sorted(objs, key=lambda obj: obj.score, reverse=True)
keep = []
flags = [0] * len(objs)
for index, obj in enumerate(objs):
if flags[index] != 0:
continue
keep.append(obj)
for j in range(index + 1, len(objs)):
if flags[j] == 0 and get_iou(np.array(obj.box), np.array(objs[j].box)) > iou:
flags[j] = 1
return keep
def detect(hm, box, landmark, threshold=0.2, nms_iou=0.2, stride=8):
def _sigmoid(x):
out = np.empty_like(x, dtype=np.float32)
np.negative(x, out=out)
np.exp(out, out=out)
out += 1.0
np.divide(1.0, out, out=out)
return out
def _maxpool3x3_same(x_hw):
H, W = x_hw.shape
pad = 1
xpad = np.pad(x_hw, ((pad, pad), (pad, pad)), mode='constant', constant_values=-np.inf)
s0, s1 = xpad.strides
shape = (H, W, 3, 3)
strides = (s0, s1, s0, s1)
windows = np.lib.stride_tricks.as_strided(xpad, shape=shape, strides=strides, writeable=False)
return windows.max(axis=(2, 3))
def _topk_desc(values_flat, k):
if k <= 0:
return np.array([], dtype=values_flat.dtype), np.array([], dtype=np.int64)
k = min(k, values_flat.size)
idx_part = np.argpartition(-values_flat, k - 1)[:k]
order = np.argsort(-values_flat[idx_part])
idx_sorted = idx_part[order]
return values_flat[idx_sorted], idx_sorted
hm = _sigmoid(hm.astype(np.float32, copy=False))
hm_hw = hm[..., 0]
hm_pool = _maxpool3x3_same(hm_hw)
keep = (hm_hw >= hm_pool)
candidate_scores = np.where(keep, hm_hw, 0.0).ravel()
num_candidates = int(keep.sum())
k = min(num_candidates, 2000)
scores_k, flat_idx_k = _topk_desc(candidate_scores, k)
H, W = hm_hw.shape
ys = (flat_idx_k // W).astype(np.int32)
xs = (flat_idx_k % W).astype(np.int32)
objs = []
for cx, cy, score in zip(xs, ys, scores_k):
if score < threshold:
break
x, y, r, b = box[cy, cx].astype(np.float32, copy=False)
cxcycxcy = np.array([cx, cy, cx, cy], dtype=np.float32)
xyrb = (cxcycxcy + np.array([-x, -y, r, b], dtype=np.float32)) * float(stride)
xyrb = xyrb.astype(np.int32, copy=False).tolist()
x5y5 = landmark[cy, cx].astype(np.float32, copy=False)
x5y5 = x5y5 + np.array([cx]*5 + [cy]*5, dtype=np.float32)
x5y5 *= float(stride)
box_landmark = list(zip(x5y5[:5].tolist(), x5y5[5:].tolist()))
objs.append(BBox("0", xyrb=xyrb, score=float(score), landmark=box_landmark))
if nms_iou != -1:
return nms(objs, iou=nms_iou)
return objs
dets = detect(hm, box, landmark, threshold=0.55, nms_iou=-1, stride=8)
res = []
for n in range(0, len(dets)):
xmin, ymin, w, h = dets[n].xywh
score = dets[n].score
L, R, T, B = int(xmin), int(xmin + w), int(ymin), int(ymin + h)
W, H = int(w), int(h)
if L < 0: L = 0
if T < 0: T = 0
if R >= 640: R = 640 - 1
if B >= 480: B = 480 - 1
b_Left = L - int(W * 0.05)
b_Top = T - int(H * 0.05)
b_Width = int(W * 1.1)
b_Height = int(H * 1.1)
if b_Left >= 0 and b_Top >= 0 and b_Width - 1 + b_Left < 640 and b_Height - 1 + b_Top < 480:
L, T, W, H = b_Left, b_Top, b_Width, b_Height
R, B = W - 1 + L, H - 1 + T
print(f'Found face: x={L}, y={T}, w={W}, h={H}, score={score}')
res.append([L, T, W, H, score])
# === END POSTPROCESSING ===
# Create output image with bounding boxes
input_reshaped = input_data.reshape(input_data.shape[1:])
if input_reshaped.shape[2] == 1:
input_reshaped = np.squeeze(input_reshaped, axis=-1)
img_out = Image.fromarray(input_reshaped).convert("RGB")
draw = ImageDraw.Draw(img_out)
for bb in res:
L, T, W, H, score = bb
draw.rectangle([L, T, L + W, T + H], outline="#00FF00", width=3)
img_out.save(IMAGE_OUT)
print('')
print(f'Inference took (on average): {(end - start) / 10}ms. per image')
```
```shell theme={null}
python3 face_detection.py
# INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
# Found face: x=120, y=186, w=62, h=79, score=0.8306506276130676
# Found face: x=311, y=125, w=66, h=81, score=0.8148472309112549
# Found face: x=424, y=173, w=64, h=86, score=0.8093323111534119
#
# Inference took (on average): 29.8ms. per image
```
这已经将每次推理的时间从 106.86ms 降至 29.8ms。
```shell theme={null}
python3 face_detection.py --use-npu
# Found face: x=312, y=125, w=64, h=81, score=0.8202381134033203
# Found face: x=120, y=186, w=62, h=78, score=0.8202381134033203
# Found face: x=421, y=173, w=67, h=86, score=0.8093323111534119
#
# Inference took (on average): 1.5ms. per image
```
通过量化此模型并将其移植到 NPU，我们将模型速度提升了 71 倍。您也不仅限于使用 Python — [LiteRT](/zh/ai-workflows/lite-rt) 页面还提供了 C++ 示例。