> ## Documentation Index > Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt > Use this file to discover all available pages before exploring further. # Choosing Qualcomm runtimes for LLM, VLM, audio, and VLA workloads (Part 5 of 7) > Choose Qualcomm runtime paths for LLM, VLM, audio, and VLA workloads, and measure the metrics that matter for each product.
← Previous: Part 4 Next: Part 6 →
Classical vision migration is mostly about tensors, preprocessing, quantization, and camera pipelines. GenAI migration adds another layer: ```text theme={null} model weights runtime format prompt template tokenizer context length prefill speed decode speed memory footprint serving API multimodal inputs ``` That is why our best practice for LLM, VLM, audio, and VLA workloads is to begin with a supported reference path before moving your own app. For most teams, that means **GenieX first**. GenieX gives you a Qualcomm-facing path for local LLM/VLM inference with a CLI, Python SDK, Docker on Linux ARM64, and an OpenAI-compatible local server. It also exposes two useful runtime lanes: | Runtime lane | Model format | Best for | | -------------------- | ----------------------------------- | ------------------------------------------------------- | | GenieX + `llama_cpp` | GGUF | Broad Hugging Face model coverage and quick experiments | | GenieX + `qairt` | Qualcomm AI Hub precompiled bundles | NPU-focused path for supported models | Raw `llama.cpp` is still useful when you need low-level debugging or a direct GGUF experiment. For a migration story, GenieX is the cleaner first recommendation. This post is a decision guide, not a full recipe for every modality. In the running case study, it covers the optional local LLM/VLM sidecar that may sit beside the migrated camera pipeline. Before you start: ```text theme={null} [ ] target board and available memory known [ ] runtime lane chosen or narrowed: GenieX qairt, GenieX llama_cpp, raw llama.cpp, or app runtime [ ] model format available: AI Hub bundle, GGUF, ONNX, or framework checkpoint [ ] tokenizer, prompt template, and context length recorded [ ] workload-specific metrics defined: TTFT, decode tok/s, WER, first-audio latency, or action latency ``` *** ## Start with the runtime, not the model name On Jetson, a GenAI prototype might be built around vLLM, TensorRT-LLM, llama.cpp, Python Transformers, or a custom service. The first Qualcomm decision is not “which exact framework replaces this?” It is: ```text theme={null} Do I need broad GGUF coverage, or a precompiled Qualcomm NPU artifact? ``` That gives you a small decision tree: ```text theme={null} Need fastest Day 0 demo or broad model choice? -> GenieX + llama_cpp runtime Need supported NPU-optimized artifact from AI Hub? -> GenieX + qairt runtime Need app compatibility with existing local LLM service? -> GenieX local server / OpenAI-compatible API Need custom model architecture not covered yet? -> Treat it as a model-porting project, not a command-line swap ``` This is the same theme as the vision posts: use the ready artifact when one exists, and only build the conversion path when the model requires it. For the running smart-camera case study, the smallest useful move is: keep the existing sidecar API, prove a GenieX local server on Dragonwing, then swap only the service URL or backend. Skip audio and VLA unless they are real product requirements. *** ## LLMs: measure prefill and decode separately A single “tokens per second” number hides too much. For user-facing LLM workloads, track at least: | Metric | What it tells you | | ------------------ | ---------------------------------------------- | | TTFT | How long the user waits before the first token | | Prefill tokens/sec | How fast the prompt/context is processed | | Decode tokens/sec | How fast new tokens are generated | | Context length | Whether the real prompt fits | | Memory delta | Whether the app survives long sessions | | Sustained power | Whether the product can run continuously | A model can have good decode speed and still feel slow if TTFT is high. A model can run a short demo and still fail when the production context length doubles. Use GenieX benchmarking or runtime logs to capture the same measurements across models and OS images. `geniex-bench` is a standalone binary, separate from the `geniex` CLI. Install it from the GenieX benchmarking tutorial, set `BENCH` to the extracted binary, then run examples like these: ```bash theme={null} curl -fsSL \ https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-geniex/geniex-bench-linux-arm64.tar.gz \ | tar xz BENCH_DIR=$(ls -d geniex-bench-linux-arm64-*) export LD_LIBRARY_PATH="$BENCH_DIR/lib:$BENCH_DIR/lib/llama_cpp${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" export GENIEX_PLUGIN_PATH="$BENCH_DIR/lib" BENCH="$BENCH_DIR/bin/geniex-bench" "$BENCH" \ --plugin llama_cpp \ --device npu \ -m unsloth/Qwen3.5-0.8B-GGUF:Q4_0 \ -p 512 \ -n 128 "$BENCH" \ --plugin qairt \ --device npu \ -m ai-hub-models/Qwen3-4B \ -p 512 \ -n 128 ``` For GGUF/`llama_cpp`, context length can be adjusted at runtime within model and memory limits. For `qairt` bundles, context length is fixed by the compiled bundle; use a longer-context bundle if the product prompt requires it. For a simple first run, using a current GenieX docs example: ```bash theme={null} geniex infer ai-hub-models/Qwen3-4B ``` For broad GGUF experiments, using a current GenieX docs example: ```bash theme={null} geniex infer unsloth/Qwen3.5-0.8B-GGUF:Q4_0 ``` For GGUF models, `Q4_0` is the usual Day 0 choice for Hexagon NPU support. *** ## Pick models that match the board Model support changes quickly, so use AI Hub and GenieX docs as the source of truth. The current validated model families include examples like: | Family | Examples | Notes | | ------------ | -------------------------------------------- | ----------------------------------------------------- | | Small LLM | Qwen3 0.6B/1.7B, Llama 3.2 1B/3B | Good first local assistant tests | | Larger LLM | Qwen3 4B/8B, Llama 3.1 8B, Gemma, Phi-4 Mini | Watch memory, context, and TTFT | | VLM | Qwen2.5-VL, Qwen3-VL, SmolVLM, Florence-2 | Validate image path and projector/runtime support | | Audio | Whisper Tiny/Base/Small, MeloTTS, PiperTTS | Measure streaming latency, not just final output | | Robotics/VLA | ACT, Pi0.5 | Validate full closed-loop latency and safety behavior | A smaller model that runs reliably on-device is often more useful than a larger model that only works in a lab prompt. *** ## VLMs: split the text path from the vision path Vision-language models are easy to over-simplify. They are not just LLMs with an image parameter. A VLM has at least two performance paths: ```text theme={null} image encode / projector path text prefill path text decode path ``` Those paths can have different memory footprints and runtime behavior. Validate them separately: ```text theme={null} [ ] text-only prompt works [ ] single image prompt works [ ] expected image resolution works [ ] repeated image prompts keep memory stable [ ] TTFT is acceptable with real image payloads [ ] answer quality is acceptable on product images ``` For a Jetson migration, the main application question is usually whether your app already talks to an OpenAI-compatible endpoint. If it does, a GenieX local server can keep the app-side change small. Verify the exact AI Hub model ID for your release: ```bash theme={null} geniex pull ai-hub-models/Qwen3-4B-Instruct-2507 geniex serve ``` Then test locally on the device against: ```text theme={null} http://127.0.0.1:18181/v1/chat/completions ``` For another machine on the network, either start the server with the documented bind/host option for your GenieX release or use an SSH tunnel. That keeps the migration boundary clean: model serving changes first, application logic later. *** ## Audio: treat streaming as the product path Audio demos often pass as batch jobs and fail as products. For ASR models such as Whisper, measure: ```text theme={null} clip length chunk size first partial result latency final result latency WER CPU load around audio capture memory growth over repeated clips ``` For TTS models such as MeloTTS or PiperTTS, measure: ```text theme={null} first audio latency real-time factor audio underruns voice quality CPU load around playback ``` If the Jetson version uses a Python service, keep that boundary at first. Swap the inference backend, preserve the request/response contract, and only then optimize the capture/playback path. The smallest useful audio migration plan is: ```text theme={null} known-good model on Qualcomm -> same test clips as Jetson -> same scoring script -> same service API -> then optimize streaming ``` Minimum deliverable before calling audio migrated: a repeatable script that runs the same clips on Jetson and Dragonwing, prints WER or real-time factor, and records first-result latency. *** ## VLA and robotics: latency is a safety input Vision-language-action models and robotics policies have a different success condition. A nice answer is not enough. The action loop has to be stable. For ACT, Pi0.5, or similar workloads, measure the full loop: ```text theme={null} camera frame arrival preprocessing model inference action decode control output robot actuation next frame feedback ``` Useful gates: | Gate | Why | | ------------------------- | ---------------------------------------- | | End-to-end action latency | Determines control stability | | Jitter | Spikes can be worse than average latency | | Memory growth | Long rollouts expose leaks | | Thermal behavior | Robots run continuously | | Safe fallback | Needed when inference misses a deadline | | Deterministic logging | Needed to debug a bad rollout | The model runtime is only one piece. The product is the closed loop. Minimum deliverable before calling VLA migrated: a logged replay or closed-loop run that records input timestamps, inference timestamps, action timestamps, missed deadlines, and the safe fallback behavior. *** ## Migration patterns from Jetson | Jetson pattern | Qualcomm starting point | | -------------------------------- | ---------------------------------------------------------- | | Local `llama.cpp` server | GenieX local server or GenieX `llama_cpp` runtime | | TensorRT-LLM / vLLM prototype | GenieX `qairt` if the model is supported | | Python Transformers script | GenieX Python SDK or CLI wrapper | | Custom REST service | Preserve REST contract, change runtime behind it | | DeepStream + LLM sidecar | Keep the sidecar boundary, move vision pipeline separately | | ROS node calling a model service | Keep ROS message contracts, move inference service first | The practical migration is not a rewrite. It is preserving the service boundary that already works. *** ## Benchmark fairly Before comparing Jetson and Qualcomm numbers, record the setup: ```text theme={null} model name and exact revision runtime precision context length batch/concurrency prompt length output token count image/audio resolution OS image SDK/runtime version power mode / thermal state ``` Then report the metrics that match the workload: | Workload | Primary metrics | | -------- | ------------------------------------------------------------- | | LLM | TTFT, prefill tok/s, decode tok/s, memory, power | | VLM | image TTFT, decode tok/s, image size, memory, power | | ASR | WER, first partial latency, final latency, real-time factor | | TTS | first audio latency, real-time factor, underruns | | VLA | end-to-end action latency, jitter, sustained thermal behavior | A Qualcomm result that uses less power at acceptable latency can be the better product result even if a Jetson GPU wins a narrow synthetic benchmark. *** ## Takeaway For GenAI and robotics workloads, the model file is only part of the migration. Runtime choice, serving API, tokenizer behavior, context length, memory, and sustained power all matter. Start with GenieX. Use the `llama_cpp` lane for broad GGUF coverage and fast experiments. Use the `qairt` lane when AI Hub provides a supported NPU-optimized bundle. Keep your existing service boundary when possible, measure TTFT and decode separately, and validate the full product loop before calling the migration done.