> ## Documentation Index > Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt > Use this file to discover all available pages before exploring further. # Porting the full multimedia application, not just the model (Part 6 of 7) > Move the complete Jetson camera application—not just its model—to a Qualcomm Dragonwing multimedia and GStreamer pipeline.
← Previous: Part 5 Next: Part 7 →
Assume your model artifact now runs on Qualcomm Dragonwing. That is necessary, but it is rarely the whole product. Most Jetson edge AI products are applications wrapped around a model: ```text theme={null} camera capture preprocessing inference postprocessing tracking overlay display encode streaming control messages health monitoring ``` On Jetson, that application might be DeepStream, CUDA kernels, TensorRT, OpenCV, and a Docker image pinned to JetPack. On Qualcomm, the equivalent product path is usually built around Qualcomm Intelligent Multimedia SDK, GStreamer, QNN/QAIRT, LiteRT/QNN delegate, camera ISP, Adreno GPU, video encode/decode, and HTP/NPU. The key migration idea: > Port the pipeline, not just the neural net. In the running case study, this is where the YOLO context binary stops being a standalone model artifact and becomes part of the live camera, metadata, overlay, encode, and health-monitoring application. Before you start: ```text theme={null} [ ] model artifact runs on the target runtime [ ] sample video for file-input validation [ ] target camera/sensor path identified on the BSP [ ] GStreamer/IM SDK plugins installed and inspected [ ] expected camera format, resolution, FPS, and timestamps known [ ] end-to-end latency/FPS/power gates defined ``` *** ## Why a model-only port is not enough A model can pass every tensor-level validation gate and still fail inside the product. Common causes: ```text theme={null} camera format changed resize behavior changed RGB/BGR swap NCHW/NHWC mismatch normalization changed letterbox padding changed NMS moved or changed tracking IDs behave differently overlay draws stale metadata encoder adds too much latency GPU is busy with display work power or thermals drift after 30 minutes ``` That is why our best practice is to freeze the application contract before optimizing individual pieces. For a vision application, write down: ```text theme={null} input source: camera / file / RTSP input resolution and frame rate pixel format preprocessing math model input tensor shape and layout model output tensor names and shapes postprocessing steps tracking behavior overlay fields output stream/display requirements latency/FPS/power targets ``` Then port one stage at a time. *** ## The DeepStream-to-IM SDK map A practical mapping looks like this: | Jetson / DeepStream | Qualcomm / IM SDK direction | | ----------------------- | -------------------------------------------------- | | `nvv4l2camerasrc` | `qticamsrc` | | `nvinfer` | `qtimlqnn` or `qtimltflite` with QNN delegate | | `nvtracker` | `qtiobjtracker` | | `nvosd` | `qtivoverlay` | | `nveglglessink` | `waylandsink` | | CUDA resize/normalize | `qtimlvconverter` or CPU OpenCV | | DeepStream metadata | IM SDK ML metadata / `qtimetamux` | | NVENC encode path | V4L2 hardware encode path | | DeepStream config files | IM SDK/GStreamer pipeline plus model JSON/settings | This is not a one-to-one replacement for every property. It is a starting map so the team can identify which code disappears, which config moves, and which behavior needs a new validation check. Current IM SDK examples use `qticamsrc`; older examples may show `qtiqmmfsrc`. Check `gst-inspect-1.0` on your image before copying a pipeline. *** ## Start with a file, then move to a camera The cheapest path is to remove live-camera variability at first. Use the same input video on Jetson and Qualcomm: ```text theme={null} sample.mp4 -> decode -> preprocess -> inference -> postprocess -> overlay or metadata dump ``` Once the file path matches, move to the real camera: ```text theme={null} camera -> same preprocess -> same inference -> same postprocess -> same overlay / stream ``` This makes failures cheaper to classify. If the file path matches and the camera path does not, the bug is likely capture format, timestamping, frame rate, exposure, buffer memory, or color conversion. If both fail, look earlier at model export, preprocessing, or postprocessing. Before blaming the model, check the camera path: ```bash theme={null} gst-inspect-1.0 | grep -E 'qti|v4l2|wayland' ``` ```text theme={null} [ ] sensor is supported by the target BSP/image [ ] ISP/camera service is configured for the module [ ] requested format/resolution/FPS actually comes from the camera [ ] timestamps survive capture, inference, overlay, and encode [ ] hardware/DMABUF path is used where the SDK supports it [ ] one camera is stable before adding multi-camera sync ``` *** ## A minimal Qualcomm camera AI pipeline A simplified IM SDK-style pipeline for a live camera detector looks like this. It follows the current IM SDK object-detection examples, but exact properties are model- and release-specific; verify them with `gst-inspect-1.0 qticamsrc qtiqmmfsrc qtimlqnn qtimlpostprocess` on your image. On the IQ-9075 lab image used for this series, `qtiqmmfsrc` is present and `qticamsrc` is not, so treat the source element as image-dependent: ```bash theme={null} gst-launch-1.0 -e --gst-debug=2 \ qticamsrc ! \ video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! \ queue ! tee name=t \ t. ! queue ! qtimetamux name=obj_mux ! qtivoverlay ! waylandsink fullscreen=true sync=false \ t. ! queue ! qtimlvconverter ! queue ! \ qtimlqnn model=$HOME/models/best_ctx.bin backend=/usr/lib/libQnnHtp.so tensors="" ! queue ! \ qtimlpostprocess module=yolov8 labels=$HOME/labels/yolov8.json settings="{\"confidence\": 51.0}" ! \ text/x-raw ! queue ! obj_mux. ``` For LiteRT models, the inference stage can use the QNN delegate path instead: ```bash theme={null} qtimltflite \ model=$HOME/models/model_w8a8.tflite \ delegate=external \ external-delegate-path=libQnnTFLiteDelegate.so \ external-delegate-options="QNNExternalDelegate,backend_type=htp;" ``` The exact pipeline depends on your model, sensor, display, and output requirement. The pattern is the point: ```text theme={null} source -> preprocess -> inference -> postprocess -> metadata -> overlay/stream ``` *** ## Keep preprocessing boring Most migration bugs hide in preprocessing. For every model, record these fields in a model-side config file: ```json theme={null} { "input_width": 640, "input_height": 640, "input_layout": "NCHW", "input_color": "RGB", "scale": 0.00392156862745098, "mean": [0.0, 0.0, 0.0], "std": [1.0, 1.0, 1.0], "letterbox": true, "pad_value": 114, "nms_in_graph": false } ``` Then make Jetson and Qualcomm read the same contract. If the old app has CUDA preprocessing, isolate its behavior before replacing it: ```text theme={null} save preprocessed tensor from Jetson save preprocessed tensor from Qualcomm compare shape, dtype, min/max, mean/std, and a few pixels ``` A one-line color swap can cost more accuracy than the runtime port. *** ## Treat postprocessing as application code For YOLO-style models, keeping NMS outside the graph is often the simplest migration path. Postprocessing usually includes: ```text theme={null} output tensor decode confidence threshold class filtering box conversion NMS box scaling back to source image metadata formatting ``` Keep this code explicit and shared when possible. If Jetson has Python postprocessing and Qualcomm has C++/GStreamer postprocessing, build a small golden-output test from saved tensors. The smallest useful test: ```text theme={null} same raw output tensor same labels same thresholds same NMS settings same expected boxes ``` That catches layout and threshold drift before you blame the accelerator. *** ## Metadata replaces glue code DeepStream applications often depend on metadata attached to buffers. IM SDK has the same broad idea: inference produces tensors, postprocessing converts tensors into metadata or masks, and downstream elements use that metadata for tracking, overlay, streaming, or another inference stage. A common pattern: ```text theme={null} video branch -> display/encode path inference branch -> qtimlvconverter -> qtimlqnn or qtimltflite -> qtimlpostprocess -> metadata metadata + video -> qtimetamux -> qtiobjtracker -> qtivoverlay -> display or stream ``` This keeps video buffers and inference metadata aligned without turning the app into a pile of custom glue. *** ## Multi-stream is where Qualcomm can shine Jetson apps often rely on one GPU for inference, CUDA preprocessing, display, and encode. Qualcomm Dragonwing devices have a more heterogeneous pipeline: camera ISP, HTP/NPU, Adreno GPU, video encode/decode, CPU, and DSP resources can each carry different pieces. That does not make performance automatic. It gives you more placement choices. For multi-stream products, measure: ```text theme={null} per-stream FPS end-to-end latency frame drops queue depth encoder latency HTP utilization GPU/display contention CPU load memory bandwidth pressure power and thermal behavior ``` The best pipeline is often the one that avoids unnecessary copies, not the one with the fastest single model invocation. Use symptoms to find the bottleneck: | Symptom | Likely bottleneck | | ------------------------------------- | ------------------------------------------------------- | | Model profile is fast, app FPS is low | camera, queues, postprocessing, display, or encode | | HTP utilization is low | pipeline is starving inference | | CPU load is high | copies, color conversion, fallback ops, or app-side NMS | | GPU is high | display, overlay, or resize contention | | FPS drops over time | thermal throttling, power mode, or memory growth | | Multi-camera drifts | timestamp/sync policy, not the detector | *** ## Deployment bundle A model-only bundle is too small for a real multimedia app. Bundle the contract around the model: ```text theme={null} models/ detector.bin or detector.tflite classifier.bin or classifier.tflite labels/ detector_labels.json classifier_labels.json config/ preprocess.json postprocess.json tracker.json pipeline.env app/ start.sh healthcheck.sh manifest.json ``` The manifest should record: ```text theme={null} model version source checkpoint or ONNX hash AI Hub job or build ID QAIRT/QNN SDK version BSP/OS image version target SoC precision expected input/output tensor names known validation dataset ``` That turns “it worked on my EVK” into something another engineer can reproduce. *** ## Validation checklist Before calling the app migrated, make these checks boring: ```text theme={null} [ ] File-input pipeline runs with known test video [ ] Camera-input pipeline runs at target resolution/FPS [ ] Preprocessed tensors match the Jetson/reference behavior [ ] Model outputs match the FP32 ONNX baseline within tolerance [ ] Postprocessing produces expected boxes/classes/masks [ ] Tracking IDs remain stable enough for the product [ ] Overlay matches source coordinates [ ] Encode/stream path meets latency target [ ] End-to-end FPS measured under sustained load [ ] Power and thermals measured for a real run length [ ] App starts, stops, and recovers cleanly ``` The first demo can be a one-line `gst-launch-1.0` command. The production version should still keep the same pipeline shape, just with better configuration, logging, service management, and update behavior. *** ## Takeaway A successful Jetson-to-Qualcomm migration is not “TensorRT model becomes QNN model.” It is: ```text theme={null} DeepStream/CUDA product pipeline -> IM SDK/GStreamer/QNN product pipeline ``` Keep the model contract explicit, validate preprocessing and postprocessing separately, start with file input, then move to the camera. The less custom glue you carry over, the easier the product is to debug when it runs all day.