Quantization is the migration step people underestimate (Part 4 of 7)

← Previous: Part 3 Next: Part 5 →

The fastest way to make a model run slowly on an edge AI accelerator is to treat quantization as an export checkbox. On Jetson, many teams reach for TensorRT FP16 first and add INT8 later when they need more throughput. On Qualcomm Dragonwing, the HTP/NPU path is much more often an integer path from the beginning. That changes the migration plan. The happy path still starts with the same model you already trained:

best.pt
  -> static ONNX
  -> FP32 validation
  -> QAIRT/QNN conversion
  -> quantization with representative data
  -> context binary for the target SoC
  -> validation against the FP32 baseline

The risky shortcut is this:

TensorRT INT8 calibration cache
  -> copy to Qualcomm

That does not carry over. A TensorRT calibration cache is a TensorRT artifact. The safer path is to re-quantize from the source model or FP32 intermediate, using real calibration samples from your product domain. For the running YOLO smart-camera case study, this means using real frames from the target camera path, not a generic image folder. Before you start:

[ ] FP32 source model or validated ONNX
[ ] representative calibration set
[ ] separate validation set with product metric
[ ] target SoC/HTP architecture and SDK version
[ ] acceptable accuracy loss and latency/FPS target

Why this matters more than the export

Exporting PyTorch to ONNX gets you most of the way to a portable graph. Quantization decides whether that graph is accurate and fast on the target accelerator. A migration can fail in three different ways that all look like “the model is bad”:

The model was exported incorrectly.
The model was quantized with unrepresentative data.
The runtime artifact was built for the wrong target/runtime combination.

That is why our best practice is to validate at every boundary:

Boundary	What you compare
PyTorch -> ONNX	PyTorch output vs ONNX Runtime output
ONNX -> FP32 QAIRT/QNN artifact	ONNX Runtime output vs converted artifact output
FP32 -> quantized artifact	ONNX Runtime output vs quantized output
Artifact -> application	Source app metric vs device app metric

For a classifier, that might be top-1/top-5 accuracy. For YOLO, it might be mAP, recall at a fixed confidence threshold, and a few hand-inspected edge cases. For speech, it might be WER. For embeddings, it might be cosine similarity and retrieval quality. The exact metric depends on the product. The important part is picking it before tuning. Use the failing boundary to avoid blaming the wrong stage:

Failing comparison	Likely problem
PyTorch vs ONNX	export, layout, preprocessing, or unsupported graph rewrite
ONNX vs FP32 QNN	converter behavior, operator support, tensor names, or layout
FP32 QNN vs quantized QNN	calibration data, quantization settings, or precision choice
Tensors match but app differs	camera format, color conversion, postprocessing, thresholds, or timestamps

PTQ first, QAT when needed

There are two practical quantization paths:

Path	Meaning	When to use
PTQ	Post-training quantization	First attempt for most models
QAT	Quantization-aware training	When PTQ misses the accuracy target

PTQ is the simplest good path: no retraining loop, quick iteration, and often enough for vision models with clean operators and representative calibration data. QAT is the heavier path. AIMET can help train the model with quantization effects in the loop, but that adds training infrastructure, model-owner time, and a new validation cycle. We recommend saving QAT for cases where PTQ misses a real product gate. A good decision rule:

PTQ passes accuracy and latency gates -> ship candidate
PTQ misses accuracy but FP32 artifact is correct -> try quantization fixes
Quantization fixes fail -> use AIMET QAT
FP32 artifact is wrong -> go back to export/conversion, not quantization

Start with real calibration data

Calibration data is not a formality. It teaches the quantizer the activation ranges the model will see on device. Use samples that match production:

real cameras
real lighting
real compression
real crops
real resolutions
real class imbalance
real empty scenes
real hard negatives

For a camera detector, a better calibration set is usually 100-500 boring product frames than 5,000 generic internet images. Include dark frames, motion blur, glare, crowded scenes, background-only scenes, and the edge cases that normally trigger false positives. For LLM/VLM/audio models, calibration is more runtime- and model-specific, but the same principle applies: prompts, context lengths, images, or audio clips should resemble the product workload.

Pick a precision lane

The common Qualcomm migration precisions are:

Precision	Use case
`a8w8`	Default first try for many vision models on HTP
`a16w8`	Useful when activation quantization hurts accuracy
`a16w16`	Larger/slower, but can preserve more accuracy
`fp16`	Useful for GPU paths or fallback validation where supported
`w4a16`	Common for GenAI/LLM NPU-optimized artifacts

For a standard object detector, our best practice is to start with a8w8, validate, and only move up in precision if the product metric requires it. That keeps the search small:

a8w8
  -> a16w8 if activations are sensitive
  -> a16w16 / fp16 if accuracy matters more than size or throughput
  -> QAT if post-training options miss the gate

A practical QAIRT quantization flow

Assume you already exported and validated a static ONNX model. This is a compact recap of the artifact flow so the quantization decisions have context. Public Dragonwing docs commonly show a local QNN model-library lane with qnn-onnx-converter and qnn-model-lib-generator; DLC-style QAIRT/AI Hub flows use DLC artifacts. Verify tool names against your installed SDK. Local QNN model-library flow:

$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-onnx-converter \
  --input_network model.optimized.onnx \
  --output_path out/model.cpp \
  --input_dim "images" 1,3,640,640

$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-model-lib-generator \
  -c out/model.cpp \
  -b out/model.bin \
  -o out/libs \
  -t x86_64-linux-clang

Create a calibration input list with raw inputs that match the model input tensor:

calibration/frame_0001.raw
calibration/frame_0002.raw
calibration/frame_0003.raw
...

Then quantize by passing the calibration list during conversion:

$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-onnx-converter \
  --input_network model.optimized.onnx \
  --output_path out/model_a8w8.cpp \
  --input_list calibration_input_list.txt \
  --input_dim "images" 1,3,640,640

$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-model-lib-generator \
  -c out/model_a8w8.cpp \
  -b out/model_a8w8.bin \
  -o out/libs \
  -t x86_64-linux-clang

If AI Hub or your QAIRT SDK gives you a DLC instead, use the equivalent DLC quantization path from that SDK release and record the tool version in the artifact manifest. Build the target context binary only after quantization passes your functional checks:

$QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-context-binary-generator \
  --model out/libs/x86_64-linux-clang/libmodel_a8w8.so \
  --backend $QAIRT_SDK_ROOT/lib/x86_64-linux-clang/libQnnHtp.so \
  --output_dir out/context_binary \
  --binary_file model_ctx \
  --config_file context_config.json

For a DLC artifact, use the SDK’s DLC model loader path, commonly --model libQnnModelDlc.so --dlc_path out/model_a8w8.dlc. For IQ-9/QCS9075, plan around HTP v73. For IQ-8275/QCS8275, plan around HTP v75. Set that architecture through the tested SDK sample config for your board, not by guessing. Context binaries are target-sensitive, so rebuild when the target SoC, SDK, BSP, or runtime package changes.

The retry ladder

When quantization hurts accuracy, avoid random flag hunting. Use a small ladder and take the first option that passes the metric.

Step	Try	Why
1	More representative calibration data	Most common fix, least code
2	Asymmetric activation quantization	Helps when activation ranges are not centered
3	Per-channel weight quantization	Helps convolution-heavy models
4	Enhanced or percentile calibration	Helps outliers that stretch ranges
5	Mixed precision for sensitive layers	Keeps fragile layers wider
6	`a16w8` or `a16w16`	Trades size/perf for accuracy
7	AIMET QAT	Heavier, but often the right answer for stubborn models

AIMET becomes attractive when the model owner can retrain or fine-tune and the PTQ loss is real, not a preprocessing bug.

Validate the application, not just tensors

Tensor metrics are useful, but product metrics win. For a detector, compare:

mAP / recall / precision
false positives per hour
misses on critical classes
box jitter frame to frame
NMS behavior
latency and FPS at camera resolution
power under sustained stream

For an LLM, compare:

TTFT
prefill tokens/sec
decode tokens/sec
context length
memory delta
answer quality on product prompts
power under long sessions

For audio, compare:

WER or character error rate
streaming chunk latency
first-token / first-audio latency
memory growth over long runs

A quantized tensor can look close while the application still fails because preprocessing, output decoding, thresholds, or temporal logic changed. For the running YOLO case study, keep the quantization report small and concrete:

Candidate	Calibration set	Precision	Tensor gate	Product gate	Decision
FP32 baseline	validation set	FP32	reference	reference mAP/recall	baseline
PTQ attempt 1	target camera frames v1	a8w8	cosine/SQNR vs FP32	mAP/recall delta	keep or retry
PTQ retry	target camera frames v2 or wider precision	a16w8/a16w16 if needed	cosine/SQNR vs FP32	mAP/recall delta	ship candidate or escalate to QAT

The key field is the final decision: which artifact passed, which metric failed, and which retry step changed the result.

The smallest useful acceptance gate

For a first migration, keep the gate simple:

[ ] ONNX Runtime matches PyTorch or training framework baseline
[ ] FP32 converted artifact matches ONNX Runtime closely
[ ] Quantized artifact passes tensor similarity thresholds
[ ] Product metric is inside agreed tolerance
[ ] HTP runtime validation passes
[ ] Latency/FPS/power measured after correctness passes
[ ] Calibration set archived with the model build
[ ] SDK, BSP, SoC, and precision recorded in the artifact manifest

Suggested tensor gates:

Stage	Gate
ONNX simplification	cosine >= 0.9999
FP32 converted artifact	cosine >= 0.999, SQNR >= 30 dB
Quantized artifact	cosine >= 0.99, SQNR >= 20 dB, plus product metric

These are starting points, not universal truth. A detector, segmenter, embedding model, or speech model can need different tolerances. The best gate is the smallest one that predicts field behavior.

Takeaway

The trained model is usually portable. The deployment artifact is not. Quantization is where the portable model becomes a Qualcomm-ready model. Start with PTQ, use real calibration data, compare against the FP32 ONNX baseline, and keep the retry ladder short. If the simple path passes, take it. If it does not, AIMET QAT is the next serious tool, not a bag of random export flags.

​Why this matters more than the export

​PTQ first, QAT when needed

​Start with real calibration data

​Pick a precision lane

​A practical QAIRT quantization flow

​The retry ladder

​Validate the application, not just tensors

​The smallest useful acceptance gate

​Takeaway