> ## Documentation Index > Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt > Use this file to discover all available pages before exploring further. # 运行 ONNX 模型 > 在 Dragonwing 设备上使用 ONNX Runtime 与 AI Engine Direct 在 NPU 上运行 ONNX 模型。 ONNX(开放神经网络交换,Open Neural Network Exchange)是一种用于导出模型的标准格式 — 模型通常由 PyTorch 等框架创建 — 以便在任何地方运行。在 Dragonwing 设备上,您可以将 ONNX Runtime 与 AI Engine Direct 配合使用,直接在 NPU 上执行 ONNX 模型,以获得最佳性能。 ## 带 AI Engine Direct 的 onnxruntime wheel `onnxruntime` 目前没有发布带 AI Engine Direct 绑定的 aarch64 Linux 预构建 wheel — 因此您无法从 PyPI 安装 onnxruntime。不过,您可以在这里下载预构建的 wheel: * [onnxruntime\_qnn-1.23.0-cp312-cp312-linux\_aarch64.whl](https://cdn.edgeimpulse.com/qc-ai-docs/wheels/onnxruntime_qnn-1.23.0-cp312-cp312-linux_aarch64.whl) (通过 `pip3 install onnxruntime_qnn-*-linux_aarch64.whl` 安装) 要为其他 onnxruntime 或 Python 版本构建 wheel,请参见 [edgeimpulse/onnxruntime-qnn-linux-aarch64](https://github.com/edgeimpulse/onnxruntime-qnn-linux-aarch64)。 ## 准备您的 onnx 文件 NPU 仅支持具有固定输入形状的量化 uint8/int8 模型。如果您的模型未经量化,或输入形状是动态的,模型将自动被卸载到 CPU。以下是一些准备模型的技巧。 [PyTorch 文档中](https://docs.pytorch.org/tutorials/beginner/onnx/export_simple_model_to_onnx_tutorial.html)提供了将 PyTorch 模型导出为 ONNX 的完整教程。 ### 动态形状 如果您的模型具有动态形状,则需要先将其转换为固定形状。您可以通过 [Netron](https://netron.app) 查看网络的形状。 例如,这个模型具有动态形状: 您可以通过 `onnxruntime.tools.make_dynamic_shape_fixed` 设置固定形状: ``` python3 -m onnxruntime.tools.make_dynamic_shape_fixed \ model_without_shapes.onnx \ model_with_shapes.onnx \ --input_name pixel_values \ --input_shape 1,3,224,224 ``` 之后,您的模型将具有固定形状,并可以在 NPU 上运行。 ### 量化模型 NPU 仅支持 uint8/int8 量化模型。不受支持的模型或不受支持的层会自动回退到 CPU 上运行。有关模型量化的指南,请参见 [ONNX Runtime 文档:量化 ONNX 模型](https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html)。 **不想自己进行量化?** 您可以从 [Qualcomm AI Hub](https://aihub.qualcomm.com) 下载一系列预量化模型,或使用 [Edge Impulse](/zh/ai-workflows/edge-impulse) 对新模型或现有模型进行量化。 ## 在 NPU 上运行模型(Python) 要将模型卸载到 NPU,只需加载 `QNNExecutionProvider`,并在创建 `InferenceSession` 时传入。例如: ``` import onnxruntime as ort providers = (("QNNExecutionProvider", { "backend_type": "htp", "profiling_level": "detailed", })) so = ort.SessionOptions() sess = ort.InferenceSession(MODEL_PATH, sess_options=so, providers=providers) actual_providers = sess.get_providers() print(f"Using providers: {actual_providers}") # will show QNNExecutionProvider,CPUExecutionProvider if QNN can be loaded ``` 注意:请确保使用带 AI Engine Direct 绑定的 onnxruntime wheel,参见本页顶部。 ## 示例:SqueezeNet-1.1(Python) 在开发板上打开终端,或通过 SSH 会话连接到开发板,然后: 1. 创建一个新的 venv,并安装 onnxruntime 和 Pillow: ```bash theme={null} mkdir -p ~/onnxruntime-demo/ cd ~/onnxruntime-demo/ python3.12 -m venv .venv source .venv/bin/activate # onnxruntime with AI Engine Direct bindings (only works on Python3.12) wget https://cdn.edgeimpulse.com/qc-ai-docs/wheels/onnxruntime_qnn-1.23.0-cp312-cp312-linux_aarch64.whl pip3 install onnxruntime_qnn-1.23.0-cp312-cp312-linux_aarch64.whl rm onnxruntime*.whl # Other dependencies pip3 install Pillow ``` 2. 以下是运行 [SqueezeNet-1.1](https://aihub.qualcomm.com/models/squeezenet1_1) 的端到端示例。将此文件保存为 `inference_onnx.py`: ```python theme={null} import os, sys, time, urllib.request, numpy as np, onnxruntime as ort from PIL import Image use_npu = True if len(sys.argv) >= 2 and sys.argv[1] == '--use-npu' else False def download_file_if_not_exists(path, url): if not os.path.exists(path): os.makedirs(os.path.dirname(path), exist_ok=True) print(f"Downloading {path} from {url}...") urllib.request.urlretrieve(url, path) return path # Path to your model/label/test image (will be download automatically) MODEL_PATH = download_file_if_not_exists('models/squeezenet-1.1/model.onnx', 'https://cdn.edgeimpulse.com/qc-ai-docs/models/SqueezeNet-1.1_w8a8.onnx') MODEL_DATA_PATH = download_file_if_not_exists('models/squeezenet-1.1/model.data', 'https://cdn.edgeimpulse.com/qc-ai-docs/models/SqueezeNet-1.1_w8a8.data') LABELS_PATH = download_file_if_not_exists('models/squeezenet-1.1_labels.txt', 'https://cdn.edgeimpulse.com/qc-ai-docs/models/SqueezeNet-1.1_labels.txt') IMAGE_PATH = download_file_if_not_exists('images/boa-constrictor.jpg', 'https://cdn.edgeimpulse.com/qc-ai-docs/examples/boa-constrictor.jpg') # Parse labels with open(LABELS_PATH, 'r') as f: labels = [line for line in f.read().splitlines() if line.strip()] # Use HTP backend of libQnnTFLiteDelegate.so (NPU) when --use-npu is passed in (otherwise CPU) providers = [] if use_npu: providers.append(("QNNExecutionProvider", { "backend_type": "htp", })) else: providers.append("CPUExecutionProvider") so = ort.SessionOptions() sess = ort.InferenceSession(MODEL_PATH, sess_options=so, providers=providers) actual_providers = sess.get_providers() print(f"Using providers: {actual_providers}") # Show which providers are actually loaded inputs = sess.get_inputs() outputs = sess.get_outputs() # !! Quantization parameters (cannot read these params from the onnx model I believe) - update these if you have another model scale = 1.0 / 255.0 zero_point = 0 dtype = np.uint8 # Load, preprocess and quantize image def load_image_for_onnx(path, H, W): # Load image img = Image.open(path).convert("RGB").resize((W, H)) img_np = np.array(img, dtype=np.float32) # !! Normalize... this model is 0..1 scaled (no further normalization); but that depends on your model !! img_np = img_np / 255 # HWC -> CHW img_np = np.transpose(img_np, (2, 0, 1)) # Add batch dim img_np = np.expand_dims(img_np, 0) # Quantize input if needed if dtype == np.float32: return img_np elif dtype == np.uint8: # q = round(x/scale + zp) q = np.round(img_np / scale + zero_point) return np.clip(q, 0, 255).astype(np.uint8) elif dtype == np.int8: # Commonly zero_point ≈ 0 (symmetric), but use provided zp anyway q = np.round(img_np / scale + zero_point) return np.clip(q, -128, 127).astype(np.int8) else: raise Exception('Unexpected dtype: ' + str(dtype)) # input data scaled 0..1 input_data = load_image_for_onnx(path=IMAGE_PATH, H=224, W=224) # Warmup once _ = sess.run(None, { sess.get_inputs()[0].name: input_data }) # Run 10x so we can calculate avg. runtime per inference start = time.perf_counter() for i in range(10): out = sess.run(None, { sess.get_inputs()[0].name: input_data }) end = time.perf_counter() # Image classification models in AI Hub miss a Softmax() layer at the end of the model, so add it manually def softmax(x, axis=-1): # subtract max for numerical stability x_max = np.max(x, axis=axis, keepdims=True) e_x = np.exp(x - x_max) return e_x / np.sum(e_x, axis=axis, keepdims=True) scores = softmax(np.squeeze(out[0], axis=0)) # Take top 5 top_k_idx = scores.argsort()[-5:][::-1] print("\nTop-5 predictions:") for i in top_k_idx: label = labels[i] if i < len(labels) else f"Class {i}" print(f"{label}: score={scores[i]}") print("") print(f'Inference took (on average): {((end - start) * 1000) / 10:.4g}ms. per image') ``` 此脚本使用了硬编码的量化参数。如果您更换模型,可能需要修改这些参数。 3. 在 CPU 上运行模型: ``` python3 inference_onnx.py # Top-5 predictions: # common iguana: score=0.3682704567909241 # night snake: score=0.1186317503452301 # water snake: score=0.1186317503452301 # boa constrictor: score=0.0813227966427803 # bullfrog: score=0.0813227966427803 # # Inference took (on average): 6.50 ms per image ``` 4. 在 NPU 上运行模型: ``` python3 inference_onnx.py --use-npu # Top-5 predictions: # common iguana: score=0.30427297949790955 # water snake: score=0.11838366836309433 # night snake: score=0.11838366836309433 # boa constrictor: score=0.11838366836309433 # rock python: score=0.08115273714065552 # # Inference took (on average): 1.60 ms per image ``` 如您所见,该模型在 NPU 上运行速度显著更快 — 但模型输出会有轻微变化。 ## 技巧与窍门 ### 禁用 CPU 回退 在调试时,您可能希望通过以下方式禁用回退到 CPU: ```python theme={null} so = ort.SessionOptions() so.add_session_config_entry("session.disable_cpu_ep_fallback", "1") ``` ### 构建新版本的 onnxruntime 软件包 参见 [edgeimpulse/onnxruntime-qnn-linux-aarch64](https://github.com/edgeimpulse/onnxruntime-qnn-linux-aarch64)。