> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Live YOLO depth on the NPU with AI Hub and native QNN
> Quantize YOLO depth with Qualcomm AI Hub, compile it to a QNN DLC/context binary, run it from a persistent native C++ QNN app, and use Python/OpenCV for a live USB-camera depth display on Dragonwing IQ-8275.
Qualcomm Linux · AI Hub · QNN · NPU
A complete prototype path for live monocular depth on Dragonwing IQ-8275: Ultralytics YOLO depth → AI Hub W8A16 quantization → QNN DLC → generated QNN context binary → persistent native C++ QNN runner → Python/OpenCV live camera UI.
This tutorial shows how to run a **live USB-camera monocular depth demo** on the **Hexagon NPU** of a Dragonwing **IQ-8275 EVK** using a model compiled by **Qualcomm AI Hub** and executed through a **native C++ QNN app**.
The final demo is intentionally split into two parts:
| Part | Language | Responsibility |
| ----------------- | ------------: | ---------------------------------------------------------------------------------------------------------------------- |
| Native QNN runner | C++ | Load the QNN context once, execute `QnnGraph_execute()` repeatedly on the NPU |
| Live application | Python/OpenCV | Capture camera frames, letterbox/normalize input, send frames to the runner, colorize depth, display RGB/depth/overlay |
That split keeps the NPU path realistic and persistent while still making the live demo easy to modify.
**Target used for this prototype:** Dragonwing IQ-8275 EVK running Ubuntu 24.04 on `aarch64`, QCS8275/QCS8300-class platform, Hexagon V75, USB camera, and an attached display. The same pattern applies to other Dragonwing targets, but the AI Hub target, QAIRT version, and generated context binary must match your hardware/runtime.
## What you will build
By the end, the live path looks like this:
```text theme={null}
USB camera
↓
Python live app
↓
320×320 NHWC float32 RGB input.raw
↓
persistent native C++ QNN server
↓
AI Hub quantized + compiled QNN context
↓
Hexagon NPU / HTP backend
↓
1×1×320×320 float32 depth output.raw
↓
Python unletterbox + colorize + overlay
↓
live RGB | DEPTH | OVERLAY window
```
In the prototype run, steady-state native QNN inference was about **16 ms/inference**, or roughly **62 FPS** for the model execution itself. The live display FPS is lower because it also includes camera capture, Python preprocessing, file I/O between Python and the native process, colorization, and OpenCV display.
## Reference benchmark from the prototype
All measurements below used `imgsz=320`.
| Path | Approx. latency | Approx. FPS | Notes |
| ------------------------- | --------------: | ----------: | -------------------------------- |
| PyTorch CPU | 629 ms | 1.6 FPS | Baseline |
| NCNN CPU | 52 ms | 19 FPS | Fixed-shape NCNN export |
| Local QNN / ONNX Runtime | 29 ms | 35 FPS | Persistent ORT QNN session |
| AI Hub QNN native C++ app | 16 ms | 62 FPS | Persistent QNN context and graph |
Benchmark numbers depend on the board image, QAIRT version, model version, camera resolution, display resolution, thermal state, and power mode. Treat these as a reference, not a product specification.
## Prerequisites
### Hardware
* Dragonwing IQ-8275 EVK with HTP/NPU support. If your board is not set up yet, follow [Set up the IQ-8275 EVK on Ubuntu](/Ubuntu/devices/iq8275-evk/set-up-the-device).
* USB camera
* Display attached to the board for the live OpenCV window
* Network connectivity for package/model downloads and AI Hub job submission
### Software on the board
Install the runtime, development headers, and Python packages:
```bash theme={null}
sudo apt update
sudo apt install -y python3.12-venv build-essential cmake pkg-config \
qairt-headers libqnn-dev
```
Confirm the HTP backend is available:
```bash theme={null}
qnn-platform-validator \
--backend dsp \
--coreVersion \
--libVersion \
--testBackend \
--targetPath /tmp/qnnval
```
A healthy setup reports a Hexagon architecture and that the backend DSP test passed, for example:
```text theme={null}
Core Version of the backend DSP: Hexagon Architecture V75
Unit Test on the backend DSP: Passed.
QNN is supported for backend DSP on the device.
```
Set up the Python environment:
```bash theme={null}
mkdir -p ~/yolo-depth-run
cd ~/yolo-depth-run
python3.12 -m venv ~/yolo-depth-venv
source ~/yolo-depth-venv/bin/activate
pip install --upgrade pip
pip install ultralytics opencv-python-headless onnx onnxslim onnxruntime-qnn qai-hub numpy
```
Keep your AI Hub token private. If a token is ever pasted into a chat, issue tracker, terminal recording, or shared log, revoke or rotate it from your AI Hub account.
Configure AI Hub once:
```bash theme={null}
source ~/yolo-depth-venv/bin/activate
qai-hub configure --api_token "$QAI_HUB_API_TOKEN"
```
## Step 1: Export YOLO depth to ONNX
Download/load the Ultralytics YOLO depth model and export a fixed `320×320` ONNX model.
```bash theme={null}
cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate
python - <<'PY'
from ultralytics import YOLO
model = YOLO("yolo26n-depth.pt")
model.export(format="onnx", imgsz=320, device="cpu", simplify=True)
PY
```
The expected output is:
```text theme={null}
~/yolo-depth-run/yolo26n-depth.onnx
```
### Clean duplicate ONNX metadata if necessary
Some exporters can place a graph output in both `graph.output` and `graph.value_info`. AI Hub can reject that with a duplicate-name error. This small sanitizer removes any duplicated `value_info` entries.
```bash theme={null}
cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate
python - <<'PY'
import onnx
src = "yolo26n-depth.onnx"
dst = "yolo26n-depth-aihub-clean.onnx"
model = onnx.load(src)
outputs = {o.name for o in model.graph.output}
kept = [v for v in model.graph.value_info if v.name not in outputs]
removed = [v.name for v in model.graph.value_info if v.name in outputs]
del model.graph.value_info[:]
model.graph.value_info.extend(kept)
onnx.checker.check_model(model)
onnx.save(model, dst)
print("removed duplicate value_info:", removed)
print("wrote", dst)
PY
```
## Step 2: Capture calibration images
The model input is NHWC float32 RGB, normalized to `[0, 1]`, with Ultralytics-style square letterboxing to `320×320`.
Create `make_aihub_calib.py`:
```python theme={null}
import argparse, time
import cv2
import numpy as np
def preprocess(frame, size=320):
h, w = frame.shape[:2]
scale = min(size / h, size / w)
nh, nw = int(round(h * scale)), int(round(w * scale))
resized = cv2.resize(frame, (nw, nh), interpolation=cv2.INTER_LINEAR)
canvas = np.full((size, size, 3), 114, dtype=np.uint8)
top = (size - nh) // 2
left = (size - nw) // 2
canvas[top:top+nh, left:left+nw] = resized
rgb = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB)
return (rgb.astype(np.float32) / 255.0)[None, ...]
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--camera", default="0")
ap.add_argument("--frames", type=int, default=64)
ap.add_argument("--size", type=int, default=320)
ap.add_argument("--skip", type=int, default=3)
ap.add_argument("--out", default="aihub_calib_images.npz")
args = ap.parse_args()
cam = int(args.camera) if args.camera.isdigit() else args.camera
cap = cv2.VideoCapture(cam, cv2.CAP_V4L2)
if not cap.isOpened():
raise SystemExit(f"Could not open camera {cam!r}")
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
arrays = []
i = 0
while len(arrays) < args.frames:
ok, frame = cap.read()
if ok and i % args.skip == 0:
arrays.append(preprocess(frame, args.size))
print(f"{len(arrays)}/{args.frames}")
i += 1
time.sleep(0.03)
cap.release()
arr = np.concatenate(arrays, axis=0).astype(np.float32)
np.savez_compressed(args.out, images=arr)
print("saved", args.out, arr.shape, arr.dtype, float(arr.min()), float(arr.max()))
if __name__ == "__main__":
main()
```
Capture calibration data:
```bash theme={null}
cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate
python make_aihub_calib.py --camera 0 --frames 64 --out aihub_calib_images.npz
```
For better visual quality, use more frames from representative scenes:
```bash theme={null}
python make_aihub_calib.py --camera 0 --frames 300 --out aihub_calib_images_300.npz
```
## Step 3: Quantize with AI Hub
This example uses W8A16 quantization: 8-bit weights and 16-bit activations. Create `submit_aihub_quant.py`:
```python theme={null}
from pathlib import Path
import numpy as np
import qai_hub as hub
client = hub.Client()
cal = np.load("aihub_calib_images.npz")["images"].astype("float32")
# Calibration entries are keyed by ONNX input name.
# This ONNX input is NHWC: images [1,320,320,3].
calibration_data = {"images": [cal[i:i+1] for i in range(cal.shape[0])]}
print("calibration samples", len(calibration_data["images"]), calibration_data["images"][0].shape)
job = client.submit_quantize_job(
model="yolo26n-depth-aihub-clean.onnx",
calibration_data=calibration_data,
weights_dtype=hub.QuantizeDtype.INT8,
activations_dtype=hub.QuantizeDtype.INT16,
name="yolo-depth-w8a16-qcs8275",
)
print("job_id", job.job_id)
print("job_url", getattr(job, "url", None) or getattr(job, "_url", None))
job.wait()
print("status", job.get_status())
result = job.get_target_model()
out_dir = Path("aihub_quantized")
out_dir.mkdir(exist_ok=True)
if hasattr(result, "download"):
print("downloaded", result.download(str(out_dir / "yolo-depth-w8a16.onnx")))
elif hasattr(result, "download_model"):
print("downloaded", result.download_model(str(out_dir / "yolo-depth-w8a16.onnx")))
else:
print("target model", result)
```
Run it:
```bash theme={null}
cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate
python submit_aihub_quant.py
```
Quantized ONNX graphs often contain integer tensors internally. In this prototype, the public output remained `FLOAT [1,1,320,320]`. The internal tensor before dequantization was `uint16`, followed by `DequantizeLinear` to the public float output. Check your own graph before assuming output type or layout.
## Step 4: Compile the quantized model to QNN DLC
In AI Hub, compile the quantized model for a target that matches your board's SoC/NPU generation and select the **Qualcomm AI Runtime / QNN DLC** target runtime.
Download the compiled DLC to the board:
```text theme={null}
~/yolo-depth-run/aihub_compiled/yolo26n-depth-aihub-qcs8275-qnn-dlc.dlc
```
Sanity-check it with `qnn-net-run` before writing a native app:
```bash theme={null}
cd ~/yolo-depth-run
mkdir -p aihub_dlc_test/input aihub_dlc_test/out
# input/images.raw must be one preprocessed [1,320,320,3] float32 tensor.
# Use your own preprocessing script or the live app's letterbox function.
echo "images:=aihub_dlc_test/input/images.raw" > aihub_dlc_test/input_list.txt
qnn-net-run \
--backend /usr/lib/libQnnHtp.so \
--dlc_path aihub_compiled/yolo26n-depth-aihub-qcs8275-qnn-dlc.dlc \
--input_list aihub_dlc_test/input_list.txt \
--output_dir aihub_dlc_test/out \
--log_level warn
```
If this fails, fix the model/runtime issue before building the native app.
## Step 5: Generate a QNN context binary
The downloaded DLC may contain topology, parameters, and weights, not a prebuilt HTP context cache. A native app can compose that, but the standard fast path is to generate a QNN context binary once and load it directly.
```bash theme={null}
cd ~/yolo-depth-run
mkdir -p qnn_context
qnn-context-binary-generator \
--backend /usr/lib/libQnnHtp.so \
--dlc_path aihub_compiled/yolo26n-depth-aihub-qcs8275-qnn-dlc.dlc \
--binary_file yolo26n-depth-aihub-qcs8275-context.bin \
--output_dir qnn_context \
--log_level warn 2>&1 | tee qnn_context/generate_context.log
```
The generated file may have a doubled suffix, depending on the tool version:
```text theme={null}
~/yolo-depth-run/qnn_context/yolo26n-depth-aihub-qcs8275-context.bin.bin
```
Treat a QNN context binary as target-specific. It is tied to the hardware target and QAIRT/QNN version. Regenerate it when you change the board image, QAIRT version, target device, or model.
## Step 6: Build a persistent native QNN runner
The native runner does three things:
1. Dynamically loads QNN providers from `/usr/lib`.
2. Loads the generated context binary with `QnnContext_createFromBinary()`.
3. Reuses the graph and tensors for repeated `QnnGraph_execute()` calls.
It also has a simple line-based server mode so Python can stream frames without reloading the model:
```text theme={null}
READY
RUN
OK
QUIT
```
Create a folder:
```bash theme={null}
cd ~/yolo-depth-run
mkdir -p qnn_app
```
Create `qnn_app/Makefile`:
```makefile theme={null}
CXX ?= g++
CXXFLAGS ?= -std=c++17 -O3 -Wall -Wextra -I/usr/include -I/usr/include/QNN
LDFLAGS ?= -ldl
all: qnn_dlc_runner
qnn_dlc_runner: qnn_dlc_runner.cpp
$(CXX) $(CXXFLAGS) $< -o $@ $(LDFLAGS)
clean:
rm -f qnn_dlc_runner
```
Copy the complete `qnn_dlc_runner.cpp` source from the [companion files page](/tutorials/aihub-qnn-native-yolo-depth-files). These are the key implementation requirements:
* Include QNN headers from `/usr/include/QNN`.
* Load `libQnnHtp.so` with `dlopen()`.
* Load `QnnInterface_getProviders` and choose a provider that exposes `QNN_API_VERSION_MAJOR`.
* Call the QNN backend/device/context APIs in the same sequence used by QNN sample apps.
* Load the generated context binary, not the original DLC, using `QnnContext_createFromBinary()`.
* Use the graph/tensor metadata from the context or the known model contract:
* graph: `graph_ymndtmzg`
* input: `images`, shape `[1,320,320,3]`, float32, `1,228,800` bytes
* output: `output_0`, shape `[1,1,320,320]`, float32, `409,600` bytes
* In server mode, copy each new input buffer into the registered input tensor, call `QnnGraph_execute()`, and write the output tensor to disk.
Build:
```bash theme={null}
cd ~/yolo-depth-run/qnn_app
make -j$(nproc)
```
Run a one-shot benchmark:
```bash theme={null}
cd ~/yolo-depth-run/qnn_app
./qnn_dlc_runner \
--dlc ../qnn_context/yolo26n-depth-aihub-qcs8275-context.bin.bin \
--input ../aihub_dlc_test/input/images.raw \
--output qnn_app_output.raw \
--warmup 20 \
--loops 500
```
Expected output resembles:
```text theme={null}
graph: graph_ymndtmzg inputs=1 outputs=1
input: images bytes=1228800 dtype=0x232
output: output_0 bytes=409600 dtype=0x232
loops=500 warmup=20
avg_ms=16.1 p50_ms=16.1 p90_ms=16.6 p99_ms=17.4 fps=62.0
wrote=qnn_app_output.raw bytes=409600
```
Compare against `qnn-net-run` output to verify correctness:
```bash theme={null}
cd ~/yolo-depth-run/qnn_app
python3 - <<'PY'
import numpy as np
app = np.fromfile("qnn_app_output.raw", dtype=np.float32)
ref = np.fromfile("../aihub_dlc_test/out/Result_0/output_0.raw", dtype=np.float32)
print("app", app.shape, app.min(), app.mean(), app.max())
print("ref", ref.shape, ref.min(), ref.mean(), ref.max())
print("max_abs_diff", np.max(np.abs(app-ref)))
print("mean_abs_diff", np.mean(np.abs(app-ref)))
PY
```
A correct native runner should match `qnn-net-run` exactly or within normal floating-point tolerance. In the prototype, `max_abs_diff` was `0.0`.
## Step 7: Add the live Python camera app
The live Python process owns the camera and display. The native C++ process owns the QNN context and graph.
The Python app should:
1. Start `qnn_dlc_runner --server`.
2. Wait for `READY`.
3. Open the USB camera with OpenCV.
4. For each frame:
* letterbox to `320×320`, BGR → RGB, float32 `[0,1]`, NHWC batch
* write `input.raw`
* send `RUN input.raw output.raw` to the native process
* read `output.raw` as float32 `[1,1,320,320]`
* crop away the letterbox padding and resize depth back to camera resolution
* colorize depth and show `RGB | DEPTH | OVERLAY`
Copy the complete live Python app from the [companion files page](/tutorials/aihub-qnn-native-yolo-depth-files). A minimal loop looks like this:
```python theme={null}
proc.stdin.write(f"RUN {in_raw} {out_raw}\n")
proc.stdin.flush()
line = proc.stdout.readline().strip()
if not line.startswith("OK "):
raise RuntimeError(line)
qnn_ms = float(line.split()[1])
depth_320 = np.fromfile(out_raw, dtype=np.float32).reshape(1, 1, 320, 320)[0, 0]
depth = unletterbox_depth(depth_320, meta)
color = colorize_depth(depth, cmap="spectral", mode="disparity")
overlay = cv2.addWeighted(frame, 0.45, color, 0.55, 0)
cv2.imshow("AI Hub QNN Native Live Depth", np.hstack([frame, color, overlay]))
```
Run the live app from the board's desktop session:
```bash theme={null}
cd ~/yolo-depth-run
source ~/yolo-depth-venv/bin/activate
python live_aihub_qnn_native.py --camera 0 --display-width 1280 --font-scale 1.2
```
If camera index `0` is not correct:
```bash theme={null}
ls /dev/video*
python live_aihub_qnn_native.py --camera /dev/video0
```
Suggested controls:
| Key | Action |
| ------------ | ---------------------------------------------------------------- |
| `q` or `Esc` | Quit |
| `s` | Save RGB image, colored depth PNG, overlay PNG, and depth `.npy` |
## Output contract and dtype checks
For this prototype:
| Tensor | Shape | Runtime dtype | Notes |
| ---------- | --------------- | ------------- | ------------------------------------------------------- |
| `images` | `[1,320,320,3]` | float32 | NHWC RGB normalized to `[0,1]` |
| `output_0` | `[1,1,320,320]` | float32 | Depth/disparity-like output, no output splitting needed |
Even though W8A16 quantization uses integer tensors internally, the compiled QNN runtime output was float32. Check this for every model. A common pattern is:
```text theme={null}
output0_q -> DequantizeLinear -> output0
```
where `output0_q` may be 16-bit but the public model/runtime output is float32.
## Troubleshooting
### `qnn-platform-validator` fails
The NPU backend is not ready. Check that the correct board image, firmware, FastRPC devices, and QNN packages are installed before debugging the model.
### `qnn-context-binary-generator` succeeds but the app cannot load the context
Regenerate the context on the same board/runtime that will run the app. Context binaries are not portable across arbitrary QAIRT versions and targets.
### The native app exits before printing `READY`
In server mode, make sure the initial `--input` path exists and has the correct byte size. The prototype runner validates the input path during startup, even though later frames are supplied through `RUN` commands.
Create a dummy input before launching the server:
```python theme={null}
np.zeros((1, 320, 320, 3), dtype=np.float32).tofile(in_raw)
```
### Camera opens but display fails
Run from a terminal attached to the board's graphical session, not a headless SSH shell. If using SSH for logs, keep the OpenCV display on the board's monitor.
### Depth colors look unstable
Use more representative calibration frames and recapture in the actual lighting/camera setup. For live demos, 100–300 frames is a better starting point than 32 frames.
## Why not run `qnn-net-run` per frame?
`qnn-net-run` is excellent for validation, but it is a command-line test tool. If you spawn it per frame, most of your time goes into process startup, context setup, and teardown.
For a live app, keep the QNN context alive:
```text theme={null}
bad live path:
Python frame → spawn qnn-net-run → load model → run once → exit
good live path:
start native runner once → load context once → Python sends RUN commands per frame
```
That is the main reason the persistent native app measured around **16 ms** while the `qnn-net-run` path measured around **48 ms** amortized in the prototype.
## Summary
This demo runs a Qualcomm AI Hub W8A16-quantized YOLO depth model on the Dragonwing IQ-8275 Hexagon NPU through a persistent native C++ QNN app, while Python handles camera capture, preprocessing, visualization, and the live UI. This gives you Python's flexibility for the application layer with the performance characteristics of native QNN model execution.