Live YOLO depth on the NPU with AI Hub and native QNN - Qualcomm Dragonwing Documentation

Qualcomm Linux · AI Hub · QNN · NPU

A complete prototype path for live monocular depth on Dragonwing IQ-8275: Ultralytics YOLO depth → AI Hub W8A16 quantization → QNN DLC → generated QNN context binary → persistent native C++ QNN runner → Python/OpenCV live camera UI.

Heath Blandford·Jul 29, 2026·← All posts

This tutorial shows how to run a live USB-camera monocular depth demo on the Hexagon NPU of a Dragonwing IQ-8275 EVK using a model compiled by Qualcomm AI Hub and executed through a native C++ QNN app. The final demo is intentionally split into two parts:

Part	Language	Responsibility
Native QNN runner	C++	Load the QNN context once, execute `QnnGraph_execute()` repeatedly on the NPU
Live application	Python/OpenCV	Capture camera frames, letterbox/normalize input, send frames to the runner, colorize depth, display RGB/depth/overlay

That split keeps the NPU path realistic and persistent while still making the live demo easy to modify.

Target used for this prototype: Dragonwing IQ-8275 EVK running Ubuntu 24.04 on aarch64, QCS8275/QCS8300-class platform, Hexagon V75, USB camera, and an attached display. The same pattern applies to other Dragonwing targets, but the AI Hub target, QAIRT version, and generated context binary must match your hardware/runtime.

What you will build

By the end, the live path looks like this:

USB camera
  ↓
Python live app
  ↓
320×320 NHWC float32 RGB input.raw
  ↓
persistent native C++ QNN server
  ↓
AI Hub quantized + compiled QNN context
  ↓
Hexagon NPU / HTP backend
  ↓
1×1×320×320 float32 depth output.raw
  ↓
Python unletterbox + colorize + overlay
  ↓
live RGB | DEPTH | OVERLAY window

In the prototype run, steady-state native QNN inference was about 16 ms/inference, or roughly 62 FPS for the model execution itself. The live display FPS is lower because it also includes camera capture, Python preprocessing, file I/O between Python and the native process, colorization, and OpenCV display.

Reference benchmark from the prototype

All measurements below used imgsz=320.

Path	Approx. latency	Approx. FPS	Notes
PyTorch CPU	629 ms	1.6 FPS	Baseline
NCNN CPU	52 ms	19 FPS	Fixed-shape NCNN export
Local QNN / ONNX Runtime	29 ms	35 FPS	Persistent ORT QNN session
AI Hub QNN native C++ app	16 ms	62 FPS	Persistent QNN context and graph

Benchmark numbers depend on the board image, QAIRT version, model version, camera resolution, display resolution, thermal state, and power mode. Treat these as a reference, not a product specification.

Prerequisites

Hardware

Dragonwing IQ-8275 EVK with HTP/NPU support. If your board is not set up yet, follow Set up the IQ-8275 EVK on Ubuntu.
USB camera
Display attached to the board for the live OpenCV window
Network connectivity for package/model downloads and AI Hub job submission

Software on the board

Install the runtime, development headers, and Python packages:

sudo apt update
sudo apt install -y python3.12-venv build-essential cmake pkg-config \
  qairt-headers libqnn-dev

Confirm the HTP backend is available:

qnn-platform-validator \
  --backend dsp \
  --coreVersion \
  --libVersion \
  --testBackend \
  --targetPath /tmp/qnnval

A healthy setup reports a Hexagon architecture and that the backend DSP test passed, for example:

Core Version of the backend DSP: Hexagon Architecture V75
Unit Test on the backend DSP: Passed.
QNN is supported for backend DSP on the device.

Set up the Python environment:

mkdir -p ~/yolo-depth-run
cd ~/yolo-depth-run

python3.12 -m venv ~/yolo-depth-venv
source ~/yolo-depth-venv/bin/activate

pip install --upgrade pip
pip install ultralytics opencv-python-headless onnx onnxslim onnxruntime-qnn qai-hub numpy

Keep your AI Hub token private. If a token is ever pasted into a chat, issue tracker, terminal recording, or shared log, revoke or rotate it from your AI Hub account.

Configure AI Hub once:

source ~/yolo-depth-venv/bin/activate
qai-hub configure --api_token "$QAI_HUB_API_TOKEN"

Step 1: Export YOLO depth to ONNX

Download/load the Ultralytics YOLO depth model and export a fixed 320×320 ONNX model.

cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate

python - <<'PY'
from ultralytics import YOLO
model = YOLO("yolo26n-depth.pt")
model.export(format="onnx", imgsz=320, device="cpu", simplify=True)
PY

The expected output is:

~/yolo-depth-run/yolo26n-depth.onnx

Clean duplicate ONNX metadata if necessary

Some exporters can place a graph output in both graph.output and graph.value_info. AI Hub can reject that with a duplicate-name error. This small sanitizer removes any duplicated value_info entries.

cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate

python - <<'PY'
import onnx
src = "yolo26n-depth.onnx"
dst = "yolo26n-depth-aihub-clean.onnx"
model = onnx.load(src)
outputs = {o.name for o in model.graph.output}
kept = [v for v in model.graph.value_info if v.name not in outputs]
removed = [v.name for v in model.graph.value_info if v.name in outputs]
del model.graph.value_info[:]
model.graph.value_info.extend(kept)
onnx.checker.check_model(model)
onnx.save(model, dst)
print("removed duplicate value_info:", removed)
print("wrote", dst)
PY

Step 2: Capture calibration images

The model input is NHWC float32 RGB, normalized to [0, 1], with Ultralytics-style square letterboxing to 320×320. Create make_aihub_calib.py:

import argparse, time
import cv2
import numpy as np


def preprocess(frame, size=320):
    h, w = frame.shape[:2]
    scale = min(size / h, size / w)
    nh, nw = int(round(h * scale)), int(round(w * scale))
    resized = cv2.resize(frame, (nw, nh), interpolation=cv2.INTER_LINEAR)
    canvas = np.full((size, size, 3), 114, dtype=np.uint8)
    top = (size - nh) // 2
    left = (size - nw) // 2
    canvas[top:top+nh, left:left+nw] = resized
    rgb = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB)
    return (rgb.astype(np.float32) / 255.0)[None, ...]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--camera", default="0")
    ap.add_argument("--frames", type=int, default=64)
    ap.add_argument("--size", type=int, default=320)
    ap.add_argument("--skip", type=int, default=3)
    ap.add_argument("--out", default="aihub_calib_images.npz")
    args = ap.parse_args()

    cam = int(args.camera) if args.camera.isdigit() else args.camera
    cap = cv2.VideoCapture(cam, cv2.CAP_V4L2)
    if not cap.isOpened():
        raise SystemExit(f"Could not open camera {cam!r}")

    cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
    cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
    cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)

    arrays = []
    i = 0
    while len(arrays) < args.frames:
        ok, frame = cap.read()
        if ok and i % args.skip == 0:
            arrays.append(preprocess(frame, args.size))
            print(f"{len(arrays)}/{args.frames}")
        i += 1
        time.sleep(0.03)
    cap.release()

    arr = np.concatenate(arrays, axis=0).astype(np.float32)
    np.savez_compressed(args.out, images=arr)
    print("saved", args.out, arr.shape, arr.dtype, float(arr.min()), float(arr.max()))


if __name__ == "__main__":
    main()

Capture calibration data:

cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate
python make_aihub_calib.py --camera 0 --frames 64 --out aihub_calib_images.npz

For better visual quality, use more frames from representative scenes:

python make_aihub_calib.py --camera 0 --frames 300 --out aihub_calib_images_300.npz

Step 3: Quantize with AI Hub

This example uses W8A16 quantization: 8-bit weights and 16-bit activations. Create submit_aihub_quant.py:

from pathlib import Path
import numpy as np
import qai_hub as hub

client = hub.Client()
cal = np.load("aihub_calib_images.npz")["images"].astype("float32")

# Calibration entries are keyed by ONNX input name.
# This ONNX input is NHWC: images [1,320,320,3].
calibration_data = {"images": [cal[i:i+1] for i in range(cal.shape[0])]}
print("calibration samples", len(calibration_data["images"]), calibration_data["images"][0].shape)

job = client.submit_quantize_job(
    model="yolo26n-depth-aihub-clean.onnx",
    calibration_data=calibration_data,
    weights_dtype=hub.QuantizeDtype.INT8,
    activations_dtype=hub.QuantizeDtype.INT16,
    name="yolo-depth-w8a16-qcs8275",
)
print("job_id", job.job_id)
print("job_url", getattr(job, "url", None) or getattr(job, "_url", None))
job.wait()
print("status", job.get_status())

result = job.get_target_model()
out_dir = Path("aihub_quantized")
out_dir.mkdir(exist_ok=True)
if hasattr(result, "download"):
    print("downloaded", result.download(str(out_dir / "yolo-depth-w8a16.onnx")))
elif hasattr(result, "download_model"):
    print("downloaded", result.download_model(str(out_dir / "yolo-depth-w8a16.onnx")))
else:
    print("target model", result)

Run it:

cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate
python submit_aihub_quant.py

Quantized ONNX graphs often contain integer tensors internally. In this prototype, the public output remained FLOAT [1,1,320,320]. The internal tensor before dequantization was uint16, followed by DequantizeLinear to the public float output. Check your own graph before assuming output type or layout.

Step 4: Compile the quantized model to QNN DLC

In AI Hub, compile the quantized model for a target that matches your board’s SoC/NPU generation and select the Qualcomm AI Runtime / QNN DLC target runtime. Download the compiled DLC to the board:

~/yolo-depth-run/aihub_compiled/yolo26n-depth-aihub-qcs8275-qnn-dlc.dlc

Sanity-check it with qnn-net-run before writing a native app:

cd ~/yolo-depth-run
mkdir -p aihub_dlc_test/input aihub_dlc_test/out

# input/images.raw must be one preprocessed [1,320,320,3] float32 tensor.
# Use your own preprocessing script or the live app's letterbox function.

echo "images:=aihub_dlc_test/input/images.raw" > aihub_dlc_test/input_list.txt

qnn-net-run \
  --backend /usr/lib/libQnnHtp.so \
  --dlc_path aihub_compiled/yolo26n-depth-aihub-qcs8275-qnn-dlc.dlc \
  --input_list aihub_dlc_test/input_list.txt \
  --output_dir aihub_dlc_test/out \
  --log_level warn

If this fails, fix the model/runtime issue before building the native app.

Step 5: Generate a QNN context binary

The downloaded DLC may contain topology, parameters, and weights, not a prebuilt HTP context cache. A native app can compose that, but the standard fast path is to generate a QNN context binary once and load it directly.

cd ~/yolo-depth-run
mkdir -p qnn_context

qnn-context-binary-generator \
  --backend /usr/lib/libQnnHtp.so \
  --dlc_path aihub_compiled/yolo26n-depth-aihub-qcs8275-qnn-dlc.dlc \
  --binary_file yolo26n-depth-aihub-qcs8275-context.bin \
  --output_dir qnn_context \
  --log_level warn 2>&1 | tee qnn_context/generate_context.log

The generated file may have a doubled suffix, depending on the tool version:

~/yolo-depth-run/qnn_context/yolo26n-depth-aihub-qcs8275-context.bin.bin

Treat a QNN context binary as target-specific. It is tied to the hardware target and QAIRT/QNN version. Regenerate it when you change the board image, QAIRT version, target device, or model.

Step 6: Build a persistent native QNN runner

The native runner does three things:

Dynamically loads QNN providers from /usr/lib.
Loads the generated context binary with QnnContext_createFromBinary().
Reuses the graph and tensors for repeated QnnGraph_execute() calls.

It also has a simple line-based server mode so Python can stream frames without reloading the model:

READY <input_bytes> <output_bytes>
RUN <input.raw> <output.raw>
OK <inference_ms> <output_bytes>
QUIT

Create a folder:

cd ~/yolo-depth-run
mkdir -p qnn_app

Create qnn_app/Makefile:

CXX ?= g++
CXXFLAGS ?= -std=c++17 -O3 -Wall -Wextra -I/usr/include -I/usr/include/QNN
LDFLAGS ?= -ldl

all: qnn_dlc_runner

qnn_dlc_runner: qnn_dlc_runner.cpp
	$(CXX) $(CXXFLAGS) $< -o $@ $(LDFLAGS)

clean:
	rm -f qnn_dlc_runner

Copy the complete qnn_dlc_runner.cpp source from the companion files page. These are the key implementation requirements:

Include QNN headers from /usr/include/QNN.
Load libQnnHtp.so with dlopen().
Load QnnInterface_getProviders and choose a provider that exposes QNN_API_VERSION_MAJOR.
Call the QNN backend/device/context APIs in the same sequence used by QNN sample apps.
Load the generated context binary, not the original DLC, using QnnContext_createFromBinary().
Use the graph/tensor metadata from the context or the known model contract:
- graph: graph_ymndtmzg
- input: images, shape [1,320,320,3], float32, 1,228,800 bytes
- output: output_0, shape [1,1,320,320], float32, 409,600 bytes
In server mode, copy each new input buffer into the registered input tensor, call QnnGraph_execute(), and write the output tensor to disk.

Build:

cd ~/yolo-depth-run/qnn_app
make -j$(nproc)

Run a one-shot benchmark:

cd ~/yolo-depth-run/qnn_app

./qnn_dlc_runner \
  --dlc ../qnn_context/yolo26n-depth-aihub-qcs8275-context.bin.bin \
  --input ../aihub_dlc_test/input/images.raw \
  --output qnn_app_output.raw \
  --warmup 20 \
  --loops 500

Expected output resembles:

graph: graph_ymndtmzg inputs=1 outputs=1
input: images bytes=1228800 dtype=0x232
output: output_0 bytes=409600 dtype=0x232
loops=500 warmup=20
avg_ms=16.1 p50_ms=16.1 p90_ms=16.6 p99_ms=17.4 fps=62.0
wrote=qnn_app_output.raw bytes=409600

Compare against qnn-net-run output to verify correctness:

cd ~/yolo-depth-run/qnn_app
python3 - <<'PY'
import numpy as np
app = np.fromfile("qnn_app_output.raw", dtype=np.float32)
ref = np.fromfile("../aihub_dlc_test/out/Result_0/output_0.raw", dtype=np.float32)
print("app", app.shape, app.min(), app.mean(), app.max())
print("ref", ref.shape, ref.min(), ref.mean(), ref.max())
print("max_abs_diff", np.max(np.abs(app-ref)))
print("mean_abs_diff", np.mean(np.abs(app-ref)))
PY

A correct native runner should match qnn-net-run exactly or within normal floating-point tolerance. In the prototype, max_abs_diff was 0.0.

Step 7: Add the live Python camera app

The live Python process owns the camera and display. The native C++ process owns the QNN context and graph. The Python app should:

Start qnn_dlc_runner --server.
Wait for READY.
Open the USB camera with OpenCV.
For each frame:
- letterbox to 320×320, BGR → RGB, float32 [0,1], NHWC batch
- write input.raw
- send RUN input.raw output.raw to the native process
- read output.raw as float32 [1,1,320,320]
- crop away the letterbox padding and resize depth back to camera resolution
- colorize depth and show RGB | DEPTH | OVERLAY

Copy the complete live Python app from the companion files page. A minimal loop looks like this:

proc.stdin.write(f"RUN {in_raw} {out_raw}\n")
proc.stdin.flush()
line = proc.stdout.readline().strip()
if not line.startswith("OK "):
    raise RuntimeError(line)
qnn_ms = float(line.split()[1])

depth_320 = np.fromfile(out_raw, dtype=np.float32).reshape(1, 1, 320, 320)[0, 0]
depth = unletterbox_depth(depth_320, meta)
color = colorize_depth(depth, cmap="spectral", mode="disparity")
overlay = cv2.addWeighted(frame, 0.45, color, 0.55, 0)
cv2.imshow("AI Hub QNN Native Live Depth", np.hstack([frame, color, overlay]))

Run the live app from the board’s desktop session:

cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate
python live_aihub_qnn_native.py --camera 0 --display-width 1280 --font-scale 1.2

If camera index 0 is not correct:

ls /dev/video*
python live_aihub_qnn_native.py --camera /dev/video0

Suggested controls:

Key	Action
`q` or `Esc`	Quit
`s`	Save RGB image, colored depth PNG, overlay PNG, and depth `.npy`

Output contract and dtype checks

For this prototype:

Tensor	Shape	Runtime dtype	Notes
`images`	`[1,320,320,3]`	float32	NHWC RGB normalized to `[0,1]`
`output_0`	`[1,1,320,320]`	float32	Depth/disparity-like output, no output splitting needed

Even though W8A16 quantization uses integer tensors internally, the compiled QNN runtime output was float32. Check this for every model. A common pattern is:

output0_q -> DequantizeLinear -> output0

where output0_q may be 16-bit but the public model/runtime output is float32.

Troubleshooting

`qnn-platform-validator` fails

The NPU backend is not ready. Check that the correct board image, firmware, FastRPC devices, and QNN packages are installed before debugging the model.

`qnn-context-binary-generator` succeeds but the app cannot load the context

Regenerate the context on the same board/runtime that will run the app. Context binaries are not portable across arbitrary QAIRT versions and targets.

The native app exits before printing `READY`

In server mode, make sure the initial --input path exists and has the correct byte size. The prototype runner validates the input path during startup, even though later frames are supplied through RUN commands. Create a dummy input before launching the server:

np.zeros((1, 320, 320, 3), dtype=np.float32).tofile(in_raw)

Camera opens but display fails

Run from a terminal attached to the board’s graphical session, not a headless SSH shell. If using SSH for logs, keep the OpenCV display on the board’s monitor.

Depth colors look unstable

Use more representative calibration frames and recapture in the actual lighting/camera setup. For live demos, 100–300 frames is a better starting point than 32 frames.

Why not run `qnn-net-run` per frame?

qnn-net-run is excellent for validation, but it is a command-line test tool. If you spawn it per frame, most of your time goes into process startup, context setup, and teardown. For a live app, keep the QNN context alive:

bad live path:
  Python frame → spawn qnn-net-run → load model → run once → exit

good live path:
  start native runner once → load context once → Python sends RUN commands per frame

That is the main reason the persistent native app measured around 16 ms while the qnn-net-run path measured around 48 ms amortized in the prototype.

Summary

This demo runs a Qualcomm AI Hub W8A16-quantized YOLO depth model on the Dragonwing IQ-8275 Hexagon NPU through a persistent native C++ QNN app, while Python handles camera capture, preprocessing, visualization, and the live UI. This gives you Python’s flexibility for the application layer with the performance characteristics of native QNN model execution.

​What you will build

​Reference benchmark from the prototype

​Prerequisites

​Hardware

​Software on the board

​Step 1: Export YOLO depth to ONNX

​Clean duplicate ONNX metadata if necessary

​Step 2: Capture calibration images

​Step 3: Quantize with AI Hub

​Step 4: Compile the quantized model to QNN DLC

​Step 5: Generate a QNN context binary

​Step 6: Build a persistent native QNN runner

​Step 7: Add the live Python camera app

​Output contract and dtype checks

​Troubleshooting

​qnn-platform-validator fails

​qnn-context-binary-generator succeeds but the app cannot load the context

​The native app exits before printing READY

​Camera opens but display fails

​Depth colors look unstable

​Why not run qnn-net-run per frame?

​Summary

What you will build

Reference benchmark from the prototype

Prerequisites

Hardware

Software on the board

Step 1: Export YOLO depth to ONNX

Clean duplicate ONNX metadata if necessary

Step 2: Capture calibration images

Step 3: Quantize with AI Hub

Step 4: Compile the quantized model to QNN DLC

Step 5: Generate a QNN context binary

Step 6: Build a persistent native QNN runner

Step 7: Add the live Python camera app

Output contract and dtype checks

Troubleshooting

`qnn-platform-validator` fails

`qnn-context-binary-generator` succeeds but the app cannot load the context

The native app exits before printing `READY`

Camera opens but display fails

Depth colors look unstable

Why not run `qnn-net-run` per frame?

Summary