> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Porting a custom YOLO detector from Jetson to Dragonwing (Part 3 of 7)
> Port a custom YOLO detector from a TensorRT engine to a Qualcomm QAIRT/QNN context binary with validation at every stage.
Assume you have a Jetson app with a custom detector and a TensorRT `.engine`. This is the migration case most Jetson teams actually care about:
```text theme={null}
I trained a custom model.
It runs on Jetson.
I have a TensorRT engine.
How do I run it on Dragonwing?
```
The short answer:
> Best practice: recover the source model and rebuild the deployment artifact for Qualcomm instead of treating the TensorRT engine as portable.
For this post, assume the running case study: a custom YOLO face detector used inside a Jetson smart-camera app. On Jetson, it may run through PyTorch, Ultralytics, TensorRT, or DeepStream. On Dragonwing, we will rebuild the deployment path through ONNX, QAIRT/QNN, quantization, a context binary, and app-side postprocessing.
Before you start:
```text theme={null}
[ ] source checkpoint or clean ONNX export
[ ] validation images/video and expected product metric
[ ] representative calibration images
[ ] preprocessing and postprocessing code
[ ] labels/class map
[ ] target board, SoC, OS/BSP, and QAIRT/QNN SDK version
[ ] latency/FPS/power target
```
***
## The migration path
A practical custom model migration is a set of gates, not one magic conversion command:
```text theme={null}
0. Inventory the Jetson app and artifacts
1. Recover the source model
2. Check AI Hub for a compatible model
3. Export static ONNX and convert to a Qualcomm artifact
4. Handle unsupported operators
5. Quantize with representative data
6. Build target-specific QNN context binary
7. Deploy model and runtime bundle
8. Run inference on device
9. Validate against ONNX FP32 baseline
10. Move postprocessing into the app
11. Benchmark after functional validation
```
That may look longer than the Jetson path, and that difference is part of the product story. The fastest Jetson path often reaches inference with fewer visible hardware-specific stages. Qualcomm’s custom-model path exposes more of the work: conversion, quantization, context generation, packaging, and backend validation.
Custom operators and specialized HTP work are not routine prerequisites for every model. They are escalation paths:
```text theme={null}
AI Hub / validated runtime artifact -> skip most conversion work
Supported ONNX graph -> use the standard pipeline
Unsupported operator/pattern -> rewrite first; escalate to custom op/UDO or expert help
```
Each gate removes one failure class, but it also adds onboarding time. That trade-off is the reason to check AI Hub first and to prove a known-good model before porting the production detector.
***
## Step 0: inventory what you have
Start by separating portable source artifacts from Jetson-specific deployment artifacts.
Useful artifacts:
```text theme={null}
best.pt / checkpoint.pth
model.onnx
TensorFlow SavedModel
TFLite model
preprocessing code
postprocessing code
labels / class map
validation dataset
calibration images
accuracy SLA
latency / FPS / power target
```
Jetson-specific artifacts to replace during migration:
```text theme={null}
TensorRT .engine / .plan
TensorRT INT8 calibration cache
CUDA preprocessing kernels
DeepStream config as-is
JetPack-pinned Docker image
```
If all you have is this:
```text theme={null}
model.engine
```
then the best next step is to recover the training checkpoint or a neutral export such as ONNX.
That is the most important migration gate:
```text theme={null}
Have source model? continue
Only have TensorRT engine? recover source first
```
***
## Step 1: recover and export the source model
For a YOLO detector, start from the training checkpoint:
```text theme={null}
best.pt
```
Export static ONNX with NMS disabled:
```bash theme={null}
yolo export \
model=best.pt \
format=onnx \
imgsz=640 \
opset=17 \
simplify=True \
dynamic=False \
nms=False
```
For HTP/NPU deployment, static shapes are the simplest starting point. For a standard YOLO image model, that usually means:
```text theme={null}
input: [1, 3, 640, 640]
```
Keep NMS out of the exported graph at first. Postprocessing is easier to control in the application, especially when you need to compare CPU, GPU, and HTP outputs.
After export, validate ONNX before touching Qualcomm tooling:
```bash theme={null}
python3 -c "import onnx; onnx.checker.check_model('best.onnx')"
```
Then create a golden output with ONNX Runtime FP32. Save this output. You will compare every later stage against it.
If ONNX export or validation fails, fix that before touching QNN:
| Symptom | First place to look |
| --------------------------------- | ---------------------------------------------------------- |
| Export fails | unsupported framework op, training wrapper, opset mismatch |
| Shape errors | dynamic batch/image size, missing static input shape |
| ONNX Runtime differs from PyTorch | preprocessing, layout, simplification, exported NMS |
| Converter rejects the graph later | unsupported op, dynamic shape, postprocessing inside graph |
Shortest rule: make PyTorch and ONNX Runtime match first. QNN cannot fix a bad ONNX export.
***
## Step 2: check AI Hub first
Before building a custom conversion pipeline, check whether your model architecture already exists in AI Hub.
For YOLO-style models:
```bash theme={null}
python3 -m venv ~/qaihub-venv
source ~/qaihub-venv/bin/activate
pip install qai-hub-models
qai-hub-models perf yolov8-det \
--device "Dragonwing IQ-9075 EVK"
qai-hub-models perf yolov8-det \
--device "Dragonwing IQ-9075 EVK" \
--runtime qnn_dlc
```
The current Dragonwing docs point to the AI Hub IoT model catalog as the up-to-date source for model availability. It includes validated coverage across LLMs, VLMs, detection, segmentation, classification, embeddings, audio, depth, restoration, and robotics.
If your exact model is not available, a compatible architecture may still be useful:
```text theme={null}
Use AI Hub model for first app integration
Then swap in custom model after the pipeline works
```
That avoids debugging conversion and application integration at the same time.
***
## Step 3: convert ONNX to a Qualcomm artifact
Qualcomm QAIRT/QNN conversion is not the same shape as `trtexec`.
Jetson often feels like one build step:
```bash theme={null}
trtexec --onnx=model.onnx --saveEngine=model.engine --fp16
```
Qualcomm separates the work into one of two lanes:
```text theme={null}
Local QNN SDK lane:
ONNX -> QNN .cpp/.bin -> model .so -> context binary
AI Hub / DLC-style lane:
AI Hub artifact or DLC -> QNN/QAIRT runtime path -> context binary where supported
```
Public Dragonwing docs commonly show the local QNN path as `qnn-onnx-converter` followed by `qnn-model-lib-generator`. AI Hub and some SDK flows may hand you a DLC instead. Use the flow that matches your SDK release, and verify exact flags with `--help`.
Local QNN model-library flow:
```bash theme={null}
"$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-onnx-converter" \
--input_network best.optimized.onnx \
--output_path out/best.cpp \
--input_dim "images" 1,3,640,640
"$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-model-lib-generator" \
-c out/best.cpp \
-b out/best.bin \
-o out/libs \
-t x86_64-linux-clang
```
Now validate the converted FP32 artifact on the host CPU. This catches conversion problems before quantization adds noise.
```bash theme={null}
"$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-net-run" \
--model out/libs/x86_64-linux-clang/libbest.so \
--backend "$QAIRT_SDK_ROOT/lib/x86_64-linux-clang/libQnnCpu.so" \
--input_list calibration_input_list.txt \
--output_dir out/cpu_validation/output/
```
Suggested gate:
```text theme={null}
FP32 QNN artifact vs ONNX Runtime FP32:
cosine >= 0.999
SQNR >= 30 dB
```
If this fails, we recommend fixing export, graph shape, preprocessing, or unsupported operator issues before quantizing.
***
## Step 4: handle unsupported operators
This is the point where the Qualcomm onboarding gap can become visible. Do not hide it behind a generic “debug the converter” instruction. Record the operator, graph location, affected backend, and fallback cost; then use this order:
If the converter rejects an operator, use this order:
```text theme={null}
1. Graph rewrite
2. QNN Custom Op Package / UDO
3. CPU fallback
```
Graph rewrite is usually the best first attempt when an unsupported op has a supported equivalent. Custom ops are more work but keep the model on the intended backend. CPU fallback can be acceptable for low-frequency or non-critical pieces, but document the latency penalty.
Our best practice is to classify the failure before choosing a fix.
Useful failure buckets:
| Failure | Bucket | Route |
| ------------------------------- | ----------------------------- | ------------------------------- |
| Converter rejects op | `C1_UNSUPPORTED_OP_TYPE` | Graph rewrite or custom op |
| HTP-incompatible pattern | `C2_HTP_INCOMPATIBLE_PATTERN` | Repair graph pattern |
| Accuracy below threshold | `E3_ACCURACY_REGRESSION` | Quantization retry ladder |
| CPU fallback layers | `P1_CPU_FALLBACK_DETECTED` | Graph optimization |
| Context build fails from memory | `G2_VTCM_EXCEEDED` | Reduce VTCM or change precision |
| SDK/SoC mismatch | `H3_ABI_MISMATCH` | Rebuild for target SoC/SDK |
| Unknown | `Z1_UNCLASSIFIED` | Preserve evidence and escalate |
For `Z1_UNCLASSIFIED`, the safest path is to save logs, model, inputs, outputs, and tool versions before escalating. This preserves evidence for the real fix.
***
## Step 5: quantize with real calibration data
If you want efficient HTP/NPU execution, quantization is central.
A TensorRT calibration cache is not useful here. It is TensorRT-specific. Recalibrate for Qualcomm.
For a YOLO face detector, use real images from the target environment:
```text theme={null}
target camera frames
normal lighting
low light
backlight
near faces
far faces
partial occlusion
empty scenes
crowded scenes
motion blur if expected
```
A small first calibration set might be 64-200 images. More is not automatically better if the images miss real operating conditions. Representative beats random.
In the public QNN converter flow, static quantization is usually part of conversion by passing `--input_list calibration_input_list.txt` to `qnn-onnx-converter`, then compiling the generated graph with `qnn-model-lib-generator`:
```bash theme={null}
"$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-onnx-converter" \
--input_network best.optimized.onnx \
--output_path out/best_a8w8.cpp \
--input_list calibration_input_list.txt \
--input_dim "images" 1,3,640,640
"$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-model-lib-generator" \
-c out/best_a8w8.cpp \
-b out/best_a8w8.bin \
-o out/libs \
-t x86_64-linux-clang
```
If AI Hub or your QAIRT SDK gives you a quantized DLC instead, keep the same validation gates and feed that DLC into the context-binary step with `libQnnModelDlc.so`.
Important gotcha from the current docs:
> `a8w16` is not a valid HTP mode. Use `a8w8`, `a16w8`, `a16w16`, or `fp16`.
If accuracy drops, use a retry ladder instead of random flag changes:
| Order | Strategy | Example flag / action |
| ----: | ---------------------------------- | ----------------------------------------------------------------------------- |
| 1 | Asymmetric activations | `--act_quantizer_schema asymmetric` |
| 2 | Per-channel + enhanced calibration | `--use_per_channel_quantization --act_quantizer_calibration enhanced` |
| 3 | Mixed precision | Mark sensitive layers |
| 4 | Precision upgrade | Try `a16w8`, `a16w16`, or `fp16` after review |
| 5 | Outlier clipping | `--act_quantizer_calibration percentile --percentile_calibration_value 99.99` |
| 6 | QAT | Escalate to AIMET QAT |
Use the first strategy that passes your accuracy SLA.
***
## Step 6: build the context binary
A QNN context binary is the target deployment artifact for this path.
Resolve the target SoC first:
```text theme={null}
IQ-9075 / QCS9075 -> dsp_arch=v73
IQ-8275 / QCS8275 -> dsp_arch=v75
```
Set `dsp_arch`, VTCM, and other backend options from the SDK examples for your exact model and target; do not copy tuning values across SoCs without validating them. In published commands, include the tested `context_config.json` or point readers to the exact SDK sample config used for that board.
Build the context binary:
```bash theme={null}
"$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-context-binary-generator" \
--model out/libs/x86_64-linux-clang/libbest_a8w8.so \
--backend "$QAIRT_SDK_ROOT/lib/x86_64-linux-clang/libQnnHtp.so" \
--output_dir out/context_binary/ \
--binary_file best_ctx \
--config_file context_config.json
```
If your input is an AI Hub or DLC artifact, use the SDK's DLC model loader path, commonly `--model libQnnModelDlc.so --dlc_path out/best_a8w8.dlc`, with the same backend and config.
Expected output:
```text theme={null}
best_ctx.bin
```
This artifact is target-sensitive. Rebuild when you change SDK, BSP, SoC, precision, or model.
If context generation fails, check the small things first:
| Symptom | First place to look |
| ------------------------------ | ------------------------------------------------------------------- |
| Backend load fails | wrong host/target library or missing `ADSP_LIBRARY_PATH` equivalent |
| HTP compile fails | wrong `dsp_arch`, stale backend config, unsupported graph pattern |
| Memory/VTCM error | reduce model size, precision, batch, or VTCM setting |
| Runs on CPU but not HTP | unsupported operator path or quantization/backend mismatch |
| Worked on old image, fails now | SDK/BSP/runtime drift; rebuild the artifact |
***
## Step 7: deploy the runtime bundle
A deployment is not just the model file. It also needs compatible runtime libraries and config.
At minimum, expect a bundle like this:
```text theme={null}
best_ctx.bin
bin/
qnn-net-run or app binary
lib/ # ARM64 host-side libraries, found through LD_LIBRARY_PATH
libQnnHtp.so
libQnnHtpV73Stub.so # IQ-9/QCS9075 example
libQnnHtpNetRunExtensions.so
libQnnSystem.so
dsp/ # DSP/skel libraries, found through ADSP_LIBRARY_PATH
libQnnHtpV73Skel.so
configs/
backend_ext.json
htp_settings.json
input_list.txt
```
Verify you are deploying ARM64 libraries to the device:
```bash theme={null}
file lib/aarch64-oe-linux-gcc11.2/libQnnHtpNetRunExtensions.so
```
You want:
```text theme={null}
ELF 64-bit LSB shared object, ARM aarch64
```
Copy the bundle:
```bash theme={null}
scp -r deployment_package/ ubuntu@DEVICE_IP:/data/qairt_runtime/
```
Use `scp` for Linux targets. Reserve `adb push` for Android targets.
***
## Step 8: run inference on device
On the device:
```bash theme={null}
cd /data/qairt_runtime
export LD_LIBRARY_PATH=/data/qairt_runtime/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
export ADSP_LIBRARY_PATH="/data/qairt_runtime/dsp;/usr/lib/rfsa/adsp;/dsp"
./bin/qnn-net-run \
--retrieve_context best_ctx.bin \
--backend libQnnHtp.so \
--input_list input_list.txt \
--output_dir inference_results/ \
--config_file configs/backend_ext.json
```
Keep `backend_ext.json` minimal:
```json theme={null}
{
"backend_extensions": {
"shared_library_path": "libQnnHtpNetRunExtensions.so",
"config_file_path": "configs/htp_settings.json"
}
}
```
Keep the top-level backend-extension file aligned with the schema shown for your installed QAIRT release. Put graph and device tuning in the referenced HTP configuration file rather than inventing sibling keys beside `backend_extensions`. Start from the matching SDK example and add settings incrementally.
***
## Step 9: validate against the right baseline
Always compare against the ONNX Runtime FP32 golden baseline, not TensorRT FP16.
Why? TensorRT output already includes NVIDIA-specific graph transformations and precision behavior. The neutral reference is the source model export.
Suggested validation gates:
| Stage | Gate |
| --------------------------------- | ---------------------------------------------- |
| ONNX simplify parity | cosine >= 0.9999 |
| FP32 Qualcomm artifact validation | cosine >= 0.999, SQNR >= 30 dB |
| Quantized/context output | cosine >= 0.99, SQNR >= 20 dB, plus app metric |
| YOLO app metric | mAP/recall/precision against validation set |
A tiny comparison helper:
```python theme={null}
import numpy as np
ref = np.load("ort_float_output.npy").flatten()
test = np.fromfile("device_output.raw", dtype=np.float32).flatten()
cosine = np.dot(ref, test) / (np.linalg.norm(ref) * np.linalg.norm(test))
sqnr = 10 * np.log10(
np.sum(ref ** 2) / (np.sum((ref - test) ** 2) + 1e-10)
)
print(f"Cosine: {cosine:.4f}")
print(f"SQNR: {sqnr:.1f} dB")
```
For object detection, tensor similarity is not enough. You also need task-level metrics:
```text theme={null}
face recall
false positives per frame
mAP if labeled validation data exists
post-NMS box agreement
latency / FPS / power
```
***
## Step 10: move YOLO postprocessing into the app
Jetson Python examples often hide YOLO decode and NMS behind Ultralytics. Once you export and deploy, you may receive raw tensors.
Make postprocessing explicit:
```text theme={null}
raw output decode
confidence thresholding
class filtering
NMS
box scaling back to original frame
```
For a face-only model, keep it simple. There is one class. A generic COCO postprocessor is only worth adding when you actually need it.
Also lock preprocessing:
```text theme={null}
letterbox vs resize
RGB vs BGR
0..1 vs mean/std normalization
NCHW vs NHWC
uint8 vs float32 input
```
Preprocessing mismatches can look like quantization problems. Validate preprocessing numerically before blaming the accelerator.
***
## Step 11: benchmark only after functional validation
First prove HTP works:
```bash theme={null}
qnn-platform-validator --backend dsp --testBackend
```
Then run with profiling:
```bash theme={null}
export ADSP_LIBRARY_PATH="/data/qairt_runtime/dsp;/usr/lib/rfsa/adsp;/dsp"
./bin/qnn-net-run \
--retrieve_context best_ctx.bin \
--backend libQnnHtp.so \
--perf_profile burst \
--profiling_level detailed \
--input_list input_list.txt \
--output_dir outputs/
```
Parse on the host:
```bash theme={null}
scp ubuntu@DEVICE_IP:/data/qairt_runtime/outputs/qnn-profiling-data.log ./
$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-profile-viewer \
--input_log qnn-profiling-data.log \
--reader $QAIRT_SDK_ROOT/lib/x86_64-linux-clang/libQnnHtpProfilingReader.so
```
Look for accelerator execution time, CPU fallback layers, and bottleneck ops. If model latency is good but app FPS is bad, the bottleneck is outside inference.
| Symptom | Likely bottleneck |
| -------------------------------- | --------------------------------------------------------------- |
| QNN profile is fast, app is slow | camera, preprocessing, postprocessing, display, or encode |
| CPU load is high | copies, fallback ops, Python glue, or NMS/postprocessing |
| HTP time is low but FPS is low | pipeline starvation or queueing |
| Latency spikes | queues, synchronization, thermal throttling, or memory pressure |
Capture memory delta, not just raw peak:
```bash theme={null}
ssh ubuntu@DEVICE_IP "cat /proc/meminfo | grep MemAvailable" > mem_baseline.txt
```
Then measure during inference and report the delta from idle.
***
## Benchmark plan for the YOLO migration
Use the same input clip, preprocessing contract, postprocessing thresholds, and accuracy set on both devices. Capture this matrix before making platform claims:
| Platform | Artifact | Runtime | Precision | Measure |
| ------------------------ | -------------- | ---------------- | ------------------------- | ----------------------------- |
| Jetson Orin Nano/Orin NX | `best.engine` | TensorRT | FP16/INT8 | latency, FPS, power, accuracy |
| Dragonwing target | `best_ctx.bin` | QNN/HTP | a8w8 | latency, FPS, power, accuracy |
| Dragonwing target | FP32 path | CPU/GPU fallback | FP32/FP16 where supported | correctness/fallback behavior |
If you cite external benchmark numbers, label them as source-document references and keep them separate from your own device results. The benchmark method matters more than a borrowed headline number.
***
## Common pitfalls
| Pitfall | Fix |
| ------------------------------------- | ------------------------------------------------------- |
| Only have `.engine`, no source model | Recover `.pt`, `.onnx`, or training checkpoint first |
| Using dynamic ONNX shapes | Export static shapes for HTP path |
| Comparing against TensorRT output | Compare against ONNX Runtime FP32 baseline |
| Reusing TRT calibration cache | Recalibrate with Qualcomm tooling |
| Synthetic calibration tensors | Use real target-domain images |
| Runtime library architecture mismatch | `file` the library; target builds should be ARM aarch64 |
| Treating context binary as portable | Rebuild for SoC/SDK/BSP/model changes |
| Skipping HTP validator | Run `qnn-platform-validator` before benchmarking |
| Hidden YOLO postprocessing | Make decode/NMS explicit in app code |
| Unknown error bucket | Preserve logs and escalate |
***
## What this migration really changes
The application goal stays the same:
```text theme={null}
camera frame -> face boxes -> overlay/dashboard
```
The deployment machinery changes:
```text theme={null}
Jetson:
best.pt -> best.engine -> TensorRT/CUDA
Dragonwing:
best.pt -> ONNX -> QNN model library or DLC-style artifact -> context binary -> QNN/HTP
```
The biggest mindset shift is that model deployment becomes a staged validation pipeline. Each stage has an artifact and a gate. That is not ceremony. It is how you avoid debugging quantization, operator support, preprocessing, runtime libraries, and app code all at once.
In the next post, we will zoom in on the most underestimated stage: quantization. That is where accuracy regressions usually appear, where calibration data quality matters, and where the difference between “runs” and “ships” becomes obvious.