> ## Documentation Index > Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt > Use this file to discover all available pages before exploring further. # Quantization is the migration step people underestimate (Part 4 of 7) > Use representative data, PTQ, and a measured retry ladder to turn an FP32 model into an accurate, efficient HTP/NPU artifact.
← Previous: Part 3 Next: Part 5 →
The fastest way to make a model run slowly on an edge AI accelerator is to treat quantization as an export checkbox. On Jetson, many teams reach for TensorRT FP16 first and add INT8 later when they need more throughput. On Qualcomm Dragonwing, the HTP/NPU path is much more often an integer path from the beginning. That changes the migration plan. The happy path still starts with the same model you already trained: ```text theme={null} best.pt -> static ONNX -> FP32 validation -> QAIRT/QNN conversion -> quantization with representative data -> context binary for the target SoC -> validation against the FP32 baseline ``` The risky shortcut is this: ```text theme={null} TensorRT INT8 calibration cache -> copy to Qualcomm ``` That does not carry over. A TensorRT calibration cache is a TensorRT artifact. The safer path is to re-quantize from the source model or FP32 intermediate, using real calibration samples from your product domain. For the running YOLO smart-camera case study, this means using real frames from the target camera path, not a generic image folder. Before you start: ```text theme={null} [ ] FP32 source model or validated ONNX [ ] representative calibration set [ ] separate validation set with product metric [ ] target SoC/HTP architecture and SDK version [ ] acceptable accuracy loss and latency/FPS target ``` *** ## Why this matters more than the export Exporting PyTorch to ONNX gets you most of the way to a portable graph. Quantization decides whether that graph is accurate and fast on the target accelerator. A migration can fail in three different ways that all look like “the model is bad”: ```text theme={null} The model was exported incorrectly. The model was quantized with unrepresentative data. The runtime artifact was built for the wrong target/runtime combination. ``` That is why our best practice is to validate at every boundary: | Boundary | What you compare | | ------------------------------- | ------------------------------------------------ | | PyTorch -> ONNX | PyTorch output vs ONNX Runtime output | | ONNX -> FP32 QAIRT/QNN artifact | ONNX Runtime output vs converted artifact output | | FP32 -> quantized artifact | ONNX Runtime output vs quantized output | | Artifact -> application | Source app metric vs device app metric | For a classifier, that might be top-1/top-5 accuracy. For YOLO, it might be mAP, recall at a fixed confidence threshold, and a few hand-inspected edge cases. For speech, it might be WER. For embeddings, it might be cosine similarity and retrieval quality. The exact metric depends on the product. The important part is picking it before tuning. Use the failing boundary to avoid blaming the wrong stage: | Failing comparison | Likely problem | | ----------------------------- | -------------------------------------------------------------------------- | | PyTorch vs ONNX | export, layout, preprocessing, or unsupported graph rewrite | | ONNX vs FP32 QNN | converter behavior, operator support, tensor names, or layout | | FP32 QNN vs quantized QNN | calibration data, quantization settings, or precision choice | | Tensors match but app differs | camera format, color conversion, postprocessing, thresholds, or timestamps | *** ## PTQ first, QAT when needed There are two practical quantization paths: | Path | Meaning | When to use | | ---- | --------------------------- | ----------------------------------- | | PTQ | Post-training quantization | First attempt for most models | | QAT | Quantization-aware training | When PTQ misses the accuracy target | PTQ is the simplest good path: no retraining loop, quick iteration, and often enough for vision models with clean operators and representative calibration data. QAT is the heavier path. AIMET can help train the model with quantization effects in the loop, but that adds training infrastructure, model-owner time, and a new validation cycle. We recommend saving QAT for cases where PTQ misses a real product gate. A good decision rule: ```text theme={null} PTQ passes accuracy and latency gates -> ship candidate PTQ misses accuracy but FP32 artifact is correct -> try quantization fixes Quantization fixes fail -> use AIMET QAT FP32 artifact is wrong -> go back to export/conversion, not quantization ``` *** ## Start with real calibration data Calibration data is not a formality. It teaches the quantizer the activation ranges the model will see on device. Use samples that match production: ```text theme={null} real cameras real lighting real compression real crops real resolutions real class imbalance real empty scenes real hard negatives ``` For a camera detector, a better calibration set is usually 100-500 boring product frames than 5,000 generic internet images. Include dark frames, motion blur, glare, crowded scenes, background-only scenes, and the edge cases that normally trigger false positives. For LLM/VLM/audio models, calibration is more runtime- and model-specific, but the same principle applies: prompts, context lengths, images, or audio clips should resemble the product workload. *** ## Pick a precision lane The common Qualcomm migration precisions are: | Precision | Use case | | --------- | ----------------------------------------------------------- | | `a8w8` | Default first try for many vision models on HTP | | `a16w8` | Useful when activation quantization hurts accuracy | | `a16w16` | Larger/slower, but can preserve more accuracy | | `fp16` | Useful for GPU paths or fallback validation where supported | | `w4a16` | Common for GenAI/LLM NPU-optimized artifacts | For a standard object detector, our best practice is to start with `a8w8`, validate, and only move up in precision if the product metric requires it. That keeps the search small: ```text theme={null} a8w8 -> a16w8 if activations are sensitive -> a16w16 / fp16 if accuracy matters more than size or throughput -> QAT if post-training options miss the gate ``` *** ## A practical QAIRT quantization flow Assume you already exported and validated a static ONNX model. This is a compact recap of the artifact flow so the quantization decisions have context. Public Dragonwing docs commonly show a local QNN model-library lane with `qnn-onnx-converter` and `qnn-model-lib-generator`; DLC-style QAIRT/AI Hub flows use DLC artifacts. Verify tool names against your installed SDK. Local QNN model-library flow: ```bash theme={null} $QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-onnx-converter \ --input_network model.optimized.onnx \ --output_path out/model.cpp \ --input_dim "images" 1,3,640,640 $QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-model-lib-generator \ -c out/model.cpp \ -b out/model.bin \ -o out/libs \ -t x86_64-linux-clang ``` Create a calibration input list with raw inputs that match the model input tensor: ```text theme={null} calibration/frame_0001.raw calibration/frame_0002.raw calibration/frame_0003.raw ... ``` Then quantize by passing the calibration list during conversion: ```bash theme={null} $QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-onnx-converter \ --input_network model.optimized.onnx \ --output_path out/model_a8w8.cpp \ --input_list calibration_input_list.txt \ --input_dim "images" 1,3,640,640 $QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-model-lib-generator \ -c out/model_a8w8.cpp \ -b out/model_a8w8.bin \ -o out/libs \ -t x86_64-linux-clang ``` If AI Hub or your QAIRT SDK gives you a DLC instead, use the equivalent DLC quantization path from that SDK release and record the tool version in the artifact manifest. Build the target context binary only after quantization passes your functional checks: ```bash theme={null} $QAIRT_SDK_ROOT/bin/x86_64-linux-clang/qnn-context-binary-generator \ --model out/libs/x86_64-linux-clang/libmodel_a8w8.so \ --backend $QAIRT_SDK_ROOT/lib/x86_64-linux-clang/libQnnHtp.so \ --output_dir out/context_binary \ --binary_file model_ctx \ --config_file context_config.json ``` For a DLC artifact, use the SDK's DLC model loader path, commonly `--model libQnnModelDlc.so --dlc_path out/model_a8w8.dlc`. For IQ-9/QCS9075, plan around HTP v73. For IQ-8275/QCS8275, plan around HTP v75. Set that architecture through the tested SDK sample config for your board, not by guessing. Context binaries are target-sensitive, so rebuild when the target SoC, SDK, BSP, or runtime package changes. *** ## The retry ladder When quantization hurts accuracy, avoid random flag hunting. Use a small ladder and take the first option that passes the metric. | Step | Try | Why | | ---- | ------------------------------------ | ------------------------------------------------------- | | 1 | More representative calibration data | Most common fix, least code | | 2 | Asymmetric activation quantization | Helps when activation ranges are not centered | | 3 | Per-channel weight quantization | Helps convolution-heavy models | | 4 | Enhanced or percentile calibration | Helps outliers that stretch ranges | | 5 | Mixed precision for sensitive layers | Keeps fragile layers wider | | 6 | `a16w8` or `a16w16` | Trades size/perf for accuracy | | 7 | AIMET QAT | Heavier, but often the right answer for stubborn models | AIMET becomes attractive when the model owner can retrain or fine-tune and the PTQ loss is real, not a preprocessing bug. *** ## Validate the application, not just tensors Tensor metrics are useful, but product metrics win. For a detector, compare: ```text theme={null} mAP / recall / precision false positives per hour misses on critical classes box jitter frame to frame NMS behavior latency and FPS at camera resolution power under sustained stream ``` For an LLM, compare: ```text theme={null} TTFT prefill tokens/sec decode tokens/sec context length memory delta answer quality on product prompts power under long sessions ``` For audio, compare: ```text theme={null} WER or character error rate streaming chunk latency first-token / first-audio latency memory growth over long runs ``` A quantized tensor can look close while the application still fails because preprocessing, output decoding, thresholds, or temporal logic changed. For the running YOLO case study, keep the quantization report small and concrete: | Candidate | Calibration set | Precision | Tensor gate | Product gate | Decision | | ------------- | ------------------------------------------ | ---------------------- | ------------------- | -------------------- | --------------------------------- | | FP32 baseline | validation set | FP32 | reference | reference mAP/recall | baseline | | PTQ attempt 1 | target camera frames v1 | a8w8 | cosine/SQNR vs FP32 | mAP/recall delta | keep or retry | | PTQ retry | target camera frames v2 or wider precision | a16w8/a16w16 if needed | cosine/SQNR vs FP32 | mAP/recall delta | ship candidate or escalate to QAT | The key field is the final decision: which artifact passed, which metric failed, and which retry step changed the result. *** ## The smallest useful acceptance gate For a first migration, keep the gate simple: ```text theme={null} [ ] ONNX Runtime matches PyTorch or training framework baseline [ ] FP32 converted artifact matches ONNX Runtime closely [ ] Quantized artifact passes tensor similarity thresholds [ ] Product metric is inside agreed tolerance [ ] HTP runtime validation passes [ ] Latency/FPS/power measured after correctness passes [ ] Calibration set archived with the model build [ ] SDK, BSP, SoC, and precision recorded in the artifact manifest ``` Suggested tensor gates: | Stage | Gate | | ----------------------- | -------------------------------------------------- | | ONNX simplification | cosine >= 0.9999 | | FP32 converted artifact | cosine >= 0.999, SQNR >= 30 dB | | Quantized artifact | cosine >= 0.99, SQNR >= 20 dB, plus product metric | These are starting points, not universal truth. A detector, segmenter, embedding model, or speech model can need different tolerances. The best gate is the smallest one that predicts field behavior. *** ## Takeaway The trained model is usually portable. The deployment artifact is not. Quantization is where the portable model becomes a Qualcomm-ready model. Start with PTQ, use real calibration data, compare against the FP32 ONNX baseline, and keep the retry ladder short. If the simple path passes, take it. If it does not, AIMET QAT is the next serious tool, not a bag of random export flags.