> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# 运行 ONNX 模型

> 使用 ONNX Runtime 与 AI Engine Direct 在 Dragonwing 设备的 NPU 上运行 ONNX 模型。

ONNX(Open Neural Network Exchange)是一种用于导出模型的标准格式 —— 这些模型通常是
在 PyTorch 等框架中创建的 —— 使其可以在任何地方运行。在 Dragonwing 设备上,您可以
借助 ONNX Runtime 与 AI Engine Direct,直接在 NPU 上执行 ONNX 模型以获得最佳性能。

## 准备您的 onnx 文件

NPU 仅支持具有固定输入形状的 uint8/int8 量化模型。如果您的模型未量化,或者具有动态
输入形状,模型将自动回退到 CPU。以下是一些准备模型的建议。

<Tip>PyTorch 文档中提供了将 PyTorch 模型导出为 ONNX 的[完整教程](https://docs.pytorch.org/tutorials/beginner/onnx/export_simple_model_to_onnx_tutorial.html)。</Tip>

### 动态形状

如果您的模型具有动态形状,则需要先将其转换为固定形状。您可以通过 [Netron](https://netron.app)
查看网络的形状。

例如,下面这个模型具有动态形状:

<Frame caption="一个具有动态形状的 ONNX 模型。此处输入张量名为 `pixel_values`。">
  <img src="https://mintlify.s3.us-west-1.amazonaws.com/qualcomm-prod/zh/AI-Developer-Workflow-Ubuntu/_images/onnxruntime1.png" />
</Frame>

您可以通过 `onnxruntime.tools.make_dynamic_shape_fixed` 设置固定形状:

```
python3 -m onnxruntime.tools.make_dynamic_shape_fixed \
    model_without_shapes.onnx \
    model_with_shapes.onnx \
    --input_name pixel_values \
    --input_shape 1,3,224,224
```

处理后,您的模型将具有固定形状,可以在 NPU 上运行。

<Frame caption="一个具有固定形状的 ONNX 模型">
  <img src="https://mintlify.s3.us-west-1.amazonaws.com/qualcomm-prod/zh/AI-Developer-Workflow-Ubuntu/_images/onnxruntime2.png" />
</Frame>

### 量化模型

NPU 仅支持 uint8/int8 量化模型。不受支持的模型或层将自动回退到 CPU。有关量化模型的
指南,请参见 [ONNX Runtime 文档:Quantize ONNX Models](https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html)。

<Tip>**不想自己量化?** 您可以从 [Qualcomm AI Hub](https://aihub.qualcomm.com) 下载各种预量化模型,或者使用 [Edge Impulse](/zh/AI-Developer-Workflow-Ubuntu/topic/edge-impulse) 对新模型或已有模型进行量化。</Tip>

## 在 NPU 上运行模型(Python)

要将模型卸载到 NPU,只需加载 `QNNExecutionProvider`,并在创建 `InferenceSession` 时
传入即可。例如:

```
import onnxruntime as ort

providers = (("QNNExecutionProvider", {
    "backend_type": "htp",
    "profiling_level": "detailed",
}))

so = ort.SessionOptions()

sess = ort.InferenceSession(MODEL_PATH, sess_options=so, providers=providers)
actual_providers = sess.get_providers()
print(f"Using providers: {actual_providers}")   # will show QNNExecutionProvider,CPUExecutionProvider if QNN can be loaded
```

注意:请确保使用带有 AI Engine Direct 绑定的 onnxruntime wheel,请参见本页顶部。

## 示例:SqueezeNet-1.1(Python)

在开发板上打开终端,或通过 SSH 连接到开发板,然后执行:

<Steps>
  <Step title="创建 venv 并安装 onnxruntime 与 Pillow">
    ```bash theme={null}
    mkdir -p ~/onnxruntime-demo/
    cd ~/onnxruntime-demo/

    python3.12 -m venv .venv
    source .venv/bin/activate

    # Other dependencies
    pip3 install Pillow
    pip3 install onnxruntime-qnn==2.5.0
    ```
  </Step>

  <Step title="保存端到端的 SqueezeNet-1.1 示例脚本">
    以下是一个运行 [SqueezeNet-1.1](https://aihub.qualcomm.com/models/squeezenet1_1) 的端到端示例。请将此文件保存为 `inference_onnx.py`:

    ```python theme={null}
    import os, sys, time, urllib.request, numpy as np, onnxruntime as ort
    from PIL import Image

    use_npu = True if len(sys.argv) >= 2 and sys.argv[1] == '--use-npu' else False

    def download_file_if_not_exists(path, url):
        if not os.path.exists(path):
            os.makedirs(os.path.dirname(path), exist_ok=True)
            print(f"Downloading {path} from {url}...")
            urllib.request.urlretrieve(url, path)
        return path

    # Path to your model/label/test image (will be download automatically)
    MODEL_PATH = download_file_if_not_exists('models/squeezenet-1.1/model.onnx', 'https://cdn.edgeimpulse.com/qc-ai-docs/models/SqueezeNet-1.1_w8a8.onnx')
    MODEL_DATA_PATH = download_file_if_not_exists('models/squeezenet-1.1/model.data', 'https://cdn.edgeimpulse.com/qc-ai-docs/models/SqueezeNet-1.1_w8a8.data')
    LABELS_PATH = download_file_if_not_exists('models/squeezenet-1.1_labels.txt', 'https://cdn.edgeimpulse.com/qc-ai-docs/models/SqueezeNet-1.1_labels.txt')
    IMAGE_PATH = download_file_if_not_exists('images/boa-constrictor.jpg', 'https://cdn.edgeimpulse.com/qc-ai-docs/examples/boa-constrictor.jpg')

    # Parse labels
    with open(LABELS_PATH, 'r') as f:
        labels = [line for line in f.read().splitlines() if line.strip()]

    # Use HTP backend of libQnnTFLiteDelegate.so (NPU) when --use-npu is passed in (otherwise CPU)
    providers = []
    if use_npu:
        providers.append(("QNNExecutionProvider", {
            "backend_type": "htp",
        }))
    else:
        providers.append("CPUExecutionProvider")

    so = ort.SessionOptions()

    sess = ort.InferenceSession(MODEL_PATH, sess_options=so, providers=providers)
    actual_providers = sess.get_providers()
    print(f"Using providers: {actual_providers}") # Show which providers are actually loaded

    inputs  = sess.get_inputs()
    outputs = sess.get_outputs()

    # !! Quantization parameters (cannot read these params from the onnx model I believe) - update these if you have another model
    scale = 1.0 / 255.0
    zero_point = 0
    dtype = np.uint8

    # Load, preprocess and quantize image
    def load_image_for_onnx(path, H, W):
        # Load image
        img = Image.open(path).convert("RGB").resize((W, H))
        img_np = np.array(img, dtype=np.float32)
        # !! Normalize... this model is 0..1 scaled (no further normalization); but that depends on your model !!
        img_np = img_np / 255
        # HWC -> CHW
        img_np = np.transpose(img_np, (2, 0, 1))
        # Add batch dim
        img_np = np.expand_dims(img_np, 0)

        # Quantize input if needed
        if dtype == np.float32:
            return img_np
        elif dtype == np.uint8:
            # q = round(x/scale + zp)
            q = np.round(img_np / scale + zero_point)
            return np.clip(q, 0, 255).astype(np.uint8)
        elif dtype == np.int8:
            # Commonly zero_point ≈ 0 (symmetric), but use provided zp anyway
            q = np.round(img_np / scale + zero_point)
            return np.clip(q, -128, 127).astype(np.int8)
        else:
            raise Exception('Unexpected dtype: ' + str(dtype))

    # input data scaled 0..1
    input_data = load_image_for_onnx(path=IMAGE_PATH, H=224, W=224)

    # Warmup once
    _ = sess.run(None, { sess.get_inputs()[0].name: input_data })

    # Run 10x so we can calculate avg. runtime per inference
    start = time.perf_counter()
    for i in range(10):
        out = sess.run(None, { sess.get_inputs()[0].name: input_data })
    end = time.perf_counter()

    # Image classification models in AI Hub miss a Softmax() layer at the end of the model, so add it manually
    def softmax(x, axis=-1):
        # subtract max for numerical stability
        x_max = np.max(x, axis=axis, keepdims=True)
        e_x = np.exp(x - x_max)
        return e_x / np.sum(e_x, axis=axis, keepdims=True)

    scores = softmax(np.squeeze(out[0], axis=0))

    # Take top 5
    top_k_idx = scores.argsort()[-5:][::-1]

    print("\nTop-5 predictions:")
    for i in top_k_idx:
        label = labels[i] if i < len(labels) else f"Class {i}"
        print(f"{label}: score={scores[i]}")

    print("")
    print(f'Inference took (on average): {((end - start) * 1000) / 10:.4g}ms. per image')
    ```

    <Warning>该脚本中硬编码了量化参数。如果您替换了模型,可能需要相应修改这些参数。</Warning>
  </Step>

  <Step title="在 CPU 上运行模型">
    ```
    python3 inference_onnx.py

    # Top-5 predictions:
    # common iguana: score=0.3682704567909241
    # night snake: score=0.1186317503452301
    # water snake: score=0.1186317503452301
    # boa constrictor: score=0.0813227966427803
    # bullfrog: score=0.0813227966427803
    #
    # Inference took (on average): 6.50 ms per image
    ```
  </Step>

  <Step title="在 NPU 上运行模型">
    ```
    python3 inference_onnx.py --use-npu

    # Top-5 predictions:
    # common iguana: score=0.30427297949790955
    # water snake: score=0.11838366836309433
    # night snake: score=0.11838366836309433
    # boa constrictor: score=0.11838366836309433
    # rock python: score=0.08115273714065552
    #
    # Inference took (on average): 1.60 ms per image
    ```
  </Step>
</Steps>

可以看到,该模型在 NPU 上运行速度显著更快 —— 但模型输出略有变化。

## 提示与技巧

### 禁用 CPU 回退

为了便于调试,您可能希望禁用 CPU 回退:

```python theme={null}
so = ort.SessionOptions()
so.add_session_config_entry("session.disable_cpu_ep_fallback", "1")
```

### 构建新版本的 onnxruntime 包

请参见 [edgeimpulse/onnxruntime-qnn-linux-aarch64](https://github.com/edgeimpulse/onnxruntime-qnn-linux-aarch64)。