Skip to main content
Robotics
Rami Mouro·Jul 29, 2026·← All posts
Build up a vision-language-action model on a Qualcomm Dragonwing IQ-9075 EVK, one step at a time, until it generates robot motion on the board’s Hexagon NPUs. Everything here is self-contained — every command and every line of code you need is on this page, and each step ends in output you can check against. We use Pi0.5, a 3-billion-parameter robot foundation model: give it camera images and a plain-English instruction, and it outputs joint motion. The first NPU inference takes about ten minutes to reach. The last third of the page is the interesting part — why the four pieces of this model end up distributed across the hardware the way they do.
Every number on this page was measured on a real IQ-9075 EVK running Ubuntu 24.04 Server. Figures taken from Qualcomm AI Hub’s own published profiling are labelled as such and never blended with ours.

​
What you will end up with

​
Before you start

Every command below runs on the board, over SSH or a serial console. No Qualcomm AI Hub account is needed. No cloud compile job. No browser. Set a working directory that the rest of the page refers to:

​
1. Install ROS 2 Jazzy and the Qualcomm AI runtime

Start with ROS 2. These are the commands from the Software Setup page:
http://, not https://, is deliberate. packages.ros.org is a CNAME to ftp.osuosl.org, whose TLS certificate covers only *.osuosl.org. On networks where no CDN papers over that mismatch, https:// fails certificate verification. Package integrity comes from the GPG signature, which is what apt actually verifies — the official Qualcomm setup page uses http:// for the same reason.
Now the Qualcomm AI Runtime (QAIRT), which contains the QNN libraries that talk to the Hexagon NPU:
ppa:ubuntu-qcom-iot/qcom-ppa is already present on the stock IQ-9075 image, so you do not need to add it. Adding it a second time produces a fatal E: Conflicting values set for option Trusted. If you hit that, list /etc/apt/sources.list.d/ and delete the duplicate.
Confirm the NPU is reachable and you have the right Hexagon architecture. The IQ-9075 is v73:
Expected output:
Note that there are two CDSP device nodes. Hold that thought — it becomes the whole story later.
Inference works as a normal user; you do not need root. If you get permission errors on /dev/fastrpc-cdsp, add yourself to the fastrpc group with sudo usermod -aG fastrpc $(id -un) and log out and back in.

​
2. Get Pi0.5 without an AI Hub account

A per-user cloud compile job in AI Hub is what you need to bring your own model to the NPU. For published models it is not required: Qualcomm precompiles QNN context binaries for a wide range of its chips and serves them through AI Hub Models, and anyone can download them without an account. qualcomm-qcs9075 — the IQ-9075 EVK — is Pi0.5’s default_device, so getting the model is a plain HTTPS download:
Expected output — four .bin files and a metadata file, 2.9 GB in total:
Pi0.5 is not one model — it is four. That shapes everything that follows: Quantization is mixed: w4a16 backbone, w8a16 vision encoder and action expert.
The bundle records SDK build=v2.45.0.260326154327. It runs unmodified on QAIRT 2.46.0 from apt, so you do not need to match SDK versions exactly.

​
3. Run your first piece of the model on the NPU

Before writing any code, prove the hardware works using qnn-net-run, which ships with qairt-tools. The vision encoder takes one RGB image, 224×224, channels-first, float32, normalized to [-1, 1]. Make one:
Expected output ends with:
And you have a real tensor:
Expected output: 2097152 bytes — exactly 1 × 256 × 2048 float32 values.

​
4. Find out what you actually have to wire up

You now have four models and no idea how they connect. Do not guess, and do not trust metadata.json’s key order — ask the binaries themselves. qnn-context-binary-utility dumps a context binary’s declared inputs and outputs, in the order the compiled graph expects them:
Then print the graph order:
Expected output (abridged — action_expert has 41 inputs):
Read action_expert’s input order again. It is l0, l1, l10, l11 … l17, l2, l3 … l9 — string-sorted, not numeric. The backbone that produces those caches emits them as l0, l1, l2 … l17. Wire them up in producer order and you silently scramble 12 of 18 layers. Nothing errors. The model runs. The actions are garbage. This is why you generate the ordering from the binaries rather than typing it out.

​
How the pipeline fits together

One action chunk is 15 NPU invocations: three vision passes (one per camera slot), one token embedding, one backbone prefill, then ten denoising steps through the action expert. Constants worth knowing: Two renames to watch, on top of the ordering trap: token_emb emits prefix_emb but backbone calls it hidden_state, and prefix_att_2d becomes prefix_att_2d_masks. suffix_sin/suffix_cos arrive at the action expert as rope_emb_sin/rope_emb_cos.

​
Where the robot’s state goes

There is no state input tensor anywhere in those four graphs. Pi0.5 discretizes proprioception into 256 buckets and splices it into the language prompt. This is the format, reproduced from the upstream openpi implementation:
So the prompt — and therefore the token ids — change on every control step. You cannot precompute a lookup table of tokenized instructions and use it in a closed loop; the tokenizer has to run inline. The tokenizer is the standard PaliGemma SentencePiece model. google/paligemma-3b-pt-224 on Hugging Face is gated (HTTP 401 without an accepted licence), but the identical file is served unauthenticated from Google’s big_vision bucket — which is where openpi fetches it too:
Expected output: 8986bb4f423f07f8c7f70d0dbe3526fb2316056c17bae71b1ea975e77a168fc6. Generate the token ids for a task, with a state vector folded in:
Expected output:
If you feed these to qnn-net-run, you must pass --use_native_input_files. By default it parses every input file as float32 and casts to the graph’s dtype. For int32 lang_tokens that turns real token ids into zeros — i.e. all padding — and the model then silently conditions on nothing. The symptom is subtle: output that differs only in the rows corresponding to real language tokens, whose values equal the padding embedding.

​
5. Run a model from your own code

qnn-net-run is a test harness; it reads files and writes files. To chain four models you need them in one process. The QRB ROS package that does this is qrb_inference_manager, and its entire API is three calls. Install it, plus the ROS node that wraps it:
The apt version cannot run this model. ros-jazzy-qrb-inference-manager in apt is 1.1.1; upstream is 2.2.0. Version 1.1.1 rejects int32 tensor inputs, which is exactly what token_emb takes for lang_tokens. We install it here for the headers and to see the API, then build from source in step 8 — which you need anyway.
Write this file as $WORK/minimal_npu.cpp. It is the complete program:
Build and run it — one g++ line, no CMake:
Expected output:
Three API calls to put a 3-billion-parameter model’s vision encoder on an NPU. There is no session setup, no delegate registration, no graph builder, and no device management to write.
QNN_HTP_BURST=1 locks the HTP to its TURBO performance level. Without it the NPU runs at a default DCVS setting and every stage is measurably slower. It is read at context creation, so it must be in the environment before the process starts.

​
6. Chain the four models

This is the part you write yourself, and it is mostly bookkeeping. The shape of it:
Two implementation notes that save real time:
  • Everything except x_t and time_step is constant across the ten denoise steps. Pack the ~34 MB of KV cache into the expert’s input buffer once, then rewrite only 6404 bytes per step.
  • The expert returns x_{t+dt} directly. It applies the Euler update internally, so there is no host-side x += dt * v.
Build the flat input buffer for each stage by concatenating tensors in the graph order you dumped in step 4 — that is the contract inference_execute() enforces, and it will tell you if the total size is wrong:

​
7. Hit the wall: one NPU cannot hold Pi0.5

With all four contexts constructed in one process, you get this:
The four binaries hold 2875 MiB of weights. This is DSP address space, not host RAM — it fails with 34 GB of system memory free, as root, and each binary loads perfectly well on its own. Measured on device 0: It is not a clean volume limit either: repeated map/unmap fragments the address space, so identical totals succeed or fail depending on what the process did earlier.
A context whose weights fail to map still reports success. You get Initialize Qnn graph from binary file successfully in the log, and the handle it hands back produces a garbage-length output vector — which surfaces much later as a memory error in your own code. Defend against it: compare the number of output tensors you got against the number the graph declared in step 4, and treat any mismatch as a hard error.
The obvious workaround is to create and destroy contexts around each use. It works, and it is slow — weight loading dominates: Two thirds of your time is spent not computing.

​
8. Unlock the second NPU

Remember the two device nodes from step 1. The question is whether QNN will let you target them. Ask it directly — write $WORK/devices.cpp:
Expected output:
Two addressable HTP devices, each with its own mapping budget. Split the four contexts across both and all of them stay resident — the paging problem disappears entirely. There is one obstacle: qrb_inference_manager calls deviceCreate(nullptr, nullptr, …), which always lands on device 0. Selecting a device means handing it a single-entry QnnDevice_PlatformInfo_t. Build the library from source and add it:
Building from source is required regardless — it is what gets you int32 support (token_emb), multi-input tensors, and HTP burst mode, none of which are in the apt 1.1.1 release. In qrb_ros_nn_inference/qrb_inference_manager/src/qnn_inference/qnn_inference.cpp, find create_device() and replace the deviceCreate call with this:
Add a const uint32_t device_id_ = 0; member and a device_id constructor parameter on both QnnInference and QrbInferenceManager, then build:
OutputTensor is an ABI break between apt 1.1.1 and upstream. Upstream added four DMA-buf fields, changing sizeof(OutputTensor). Anything compiled against the apt headers but linked against the rebuilt library reads std::vector<OutputTensor> at the wrong stride and crashes in std::bad_array_new_length. Your include path for the overlay must come before /opt/ros/jazzy/include, and dependent code needs a clean rebuild rather than a relink.
Now construct each of the four managers with an explicit device id and keep all four resident.

​
Why we split things the way we did

With both NPUs available there are several ways to distribute four models. The obvious one is to balance by size — put roughly 1.4 GB on each device. That is not the best answer.

Placement of Pi0.5's four components across the IQ-9075's two Hexagon NPUs, drawn to measured latency. One NPU pays 2258 ms of weight paging (3277 ms total). Two NPUs keep everything resident (1323 ms). Re-placing the expensive stages onto the faster NPU — same bytes per device — reaches 1111 ms.

The animation makes one thing visible that a static diagram cannot: the four stages are sequentially dependent. vision_encoder feeds token_emb, which feeds backbone, which feeds action_expert. Nothing runs concurrently. Spreading the model across two NPUs does not parallelise anything — it just means every component can stay mapped, so the chain stops paying to swap weights in and out. That is where the first and largest win comes from. Start from what the components cost, measured per action chunk: The two NPUs are not equally fast. Every component runs 25–30% slower on device 1 than on device 0. We measured this repeatedly and consistently; we do not have a confirmed cause, and we are not going to speculate about one. That asymmetry decides the placement. backbone and action_expert together are ~888 ms of the ~1030 ms of compute — so they belong on the fast device, and the two cheap components go on the slow one. Balance by cost, not by size: Note that the first and last rows put the same number of bytes on each device. The 212 ms between them comes purely from which device does the expensive work. The end state, measured over 12 iterations after 3 warmup: Context paging is now exactly zero, and the whole pipeline is 4× faster than the single-NPU path it started from — from 4421 ms down to 1111 ms.
Latency on this board is history-dependent, so benchmark a fresh one. Re-running this same benchmark on an idle board with 25 hours of uptime and several thousand context map/unmap cycles behind it gave 1152 ms, not 1111 — every stage slower, and token_emb (the largest weight map at 1006 MiB) slower by 29%. That is the opposite of what removing contention should do. The likely mechanism is the address-space fragmentation described above, showing up as degraded performance rather than as a failed mapping. We have not confirmed it — the experiment is a reboot and an immediate re-run — so treat any single figure here as the low end of a 1111–1152 ms range and state your board’s uptime alongside your own numbers.

​
How that compares to AI Hub’s published figures

AI Hub profiles each component in isolation. We run the whole chain with host-side marshalling between stages, so ours are necessarily higher. This is a like-for-unlike comparison, shown to locate the overhead rather than to claim a win: The ~223 ms gap is dominated by moving tensors between stages on the CPU. backbone alone emits 36 KV tensors totalling ~34 MB that have to be read out and repacked into the expert’s input buffer. That is the obvious next target, and qrb_ros_transport DMA-buf fd passing is the mechanism — qrb_inference_manager 2.x already exposes an inference_execute_dmabuf() entry point for it.

​
9. Prove it is numerically correct

Fast and wrong is easy to build here — a scrambled KV cache produces confident, plausible, useless actions. So check against the reference implementation you already used in step 3. Dump each stage’s exact flat input buffer and its outputs from your chained pipeline, split the input back into per-tensor files using the graph order from step 4, replay them through qnn-net-run, and diff:
Then compare ref/Result_0/<name>.raw against your pipeline’s output for each tensor. Ours match bitwise — all 45 output tensors across all four components, and they still match with components pinned to different NPUs:
Four independent signals say the NPU ran and not the CPU: AI Hub reports 100% NPU layer placement for all four components (3835/3835, 2473/2473, 1120/1120, 34/34); the loaded backend is libQnnHtp.so; the outputs are bitwise identical to qnn-net-run on the HTP backend; and context creation drops to zero once everything is resident.

​
Wrapping it in a ROS 2 node

Once the chain works, the ROS layer is unremarkable, which is the point: Two decisions worth copying. Inference takes ~1.1 s, far too long for an executor callback, so run it on a dedicated worker thread. And drop frames that arrive mid-inference rather than queueing them — a VLA acting on stale observations is worse than one acting at a lower rate. Publish the per-stage latency breakdown alongside every chunk. It costs nothing and it means no performance claim you make later is detached from a live measurement.

​
Is the output any good?

Latency and bitwise fidelity prove the pipeline is correct. They say nothing about whether the actions are useful. To check that without a robot, replay a real episode from the LIBERO dataset — the data Pi0.5 was calibrated on — feeding the NPU exactly the observations a human demonstrator saw, and compare the predicted chunk against what they actually did. Both state and action use mean/std normalization, and the statistics ship with the dataset in meta/stats.json. Scoring 15 replanning steps over a 20-step horizon: MAE / action std is the scale-free number: 0.145 means the error is about 15% of the natural spread of actions in this dataset. Predicting the dataset mean scores 1.0 by construction, so the model is genuinely tracking the demonstrator. The gripper — effectively binary at ±1, and the one dimension where being wrong is unambiguous — matches to 0.022.
Publish state before images, or the prompt is silently stale. The node fires as soon as every camera has a fresh frame; robot state is not part of that trigger, and with no state the tokenizer builds a structurally different prompt than the one Pi0.5 was calibrated on. Because DDS guarantees ordering per topic but not across topics, publishing images first lets inference run on frame k while the prompt still encodes state k−1 — with no error and no log line. Publishing state first, then the task, then the images moved this table from 0.149 to 0.145 overall and the gripper from 0.036 to 0.022.
This is open-loop teacher-forced comparison on one episode, not a task success rate. The policy never sees the consequences of its own actions. That is what the next section fixes.

​
Can it actually finish the task?

The section above has a hole in it: the human’s actions decided what happened next, so the policy was never held responsible for its own. A subtly wrong policy and a good one score about the same, because neither ever compounds its mistakes. Closing the loop removes the human. The simulator renders what the arm can see, the policy decides, the simulator executes that decision, and the task’s own goal predicate says whether it worked. And it turns out this runs entirely on the board — which is not what we expected, having just established that Gazebo cannot render here. The difference is one word in a requirement: Mesa’s software rasterizer advertises a 4.5 compatibility profile. Ogre2 demands core and dies; MuJoCo’s classic renderer is happy. So no laptop, no cross-host bridge, no DDS across machines:

​
Rendering is the expensive half, so only render when you replan

A two-camera 256×256 observation costs 340 ms on that software rasterizer. A physics step costs 31 ms. The NPU produces fifty actions in 1126 ms. Render every sim step and the numbers inseparably invert — the simulator costs 3.2× the 3B VLA: So render on replan. That is not a shortcut — it is precisely what a 50-step action chunk buys you, since the policy only needs an observation when it plans. It turns a 220-step episode from 84.8 s into 15.3 s, and it moves the bottleneck back onto the NPU where you would expect it.

NPU chunk production against simulator step consumption, drawn to measured cost. One chunk buys 50 actions for 1126 ms; rendering every sim step makes the simulator 3.2× the policy, while rendering only at replan boundaries collapses it to 18.7 s and puts the NPU back in charge. The replan horizon is the dial: H=1 costs 426 s per episode, H=50 costs 18 s.

The dial has a cost at both ends, and only one end is measured. A long horizon is cheap but acts on staler observations, because the arm executes further into a plan made from an older image. We have not measured where that starts to hurt accuracy, so --replan-horizon 10 is a working choice rather than a tuned one.

​
Four conventions that silently destroy the result

None of these throw. Get one wrong and the loop still runs, still renders, and still reports a rate — of zero. Each was settled by comparing against the recorded dataset rather than by reading documentation:
The axis-angle branch is a trap, and the obvious fix is the wrong one. LIBERO’s home pose points the gripper straight down, which puts the rotation angle at almost exactly π — the discontinuity of the axis-angle representation. Pinning the branch by the sign of the quaternion’s scalar part is the natural move, and it fails: a rollout flips from +3.14 to −3.14 partway through, a jump of 2π, while describing the same physical orientation. Since state is spliced into the language prompt as discretized bins, that reads to the policy as the wrist having spun a full turn between two control steps. The recorded dataset never wraps, so the correct rule is continuity against the previous state, not a fixed sign test.

​
The result

All ten LIBERO-10 tasks — the long-horizon suite — ten initial states each: Those ten initial states are two independent bands of five — LIBERO’s indices 0–4 and 20–24 — run as separate sweeps on purpose. A single band cannot tell you whether the states you happened to pick were easy ones, and this is a suite where difficulty varies a lot: The intervals overlap comfortably, so the bands are consistent and pooling them is legitimate. Half the ten tasks were solved 10/10. The one genuinely hard task is “put the yellow and white mug in the microwave and close it” at 4/10 — the only task that scored low in both bands. “Put both moka pots on the stove” went 2/5 then 5/5, which is a useful reminder of how little a five-episode sample resolves.

One closed-loop episode: 'pick up the book and place it in the back compartment of the caddy', solved in 252 sim steps and 26 chunks. The strip under the video is the 50-action chunk — shaded cells are the ten that will be executed before the next replan, the amber cell is the action being applied right now, and the outlined remainder is discarded. Per-stage NPU latency and the loaded backend are burned into every frame.

The HUD is doing something specific: backend libQnnHtp.so sits on every frame, so proof that the NPU ran travels with the footage instead of being asserted next to it. The action strip is there because it makes the discard visible — forty of every fifty predicted actions are thrown away at the next replan, which looks wasteful until you notice it is what lets the loop run at all. Recording that video is not free: extra frames cost 340 ms a pair, so the harness prints a warning that a run with --save-frames is not a timing measurement. The video is composed by a separate offline pass over saved frames, so the compositing never lands inside a measured episode. Two details matter more than the headline: Every one of the fourteen failures ran to exactly 520 steps. None diverged, thrashed, or produced nonsense — they ran out of step budget mid-task. The failure mode is “too slow for the cap”, not “wrong”, which makes the cap part of the result rather than an incidental setting. A null policy scores 0 out of 24 on the same harness. Zeros and uniform-random actions both fail every task, and the goal predicate is never already satisfied at reset. Without that check, a high success rate could just as easily have been measuring a broken predicate — and it would have looked identical from the outside.
This is a reduced protocol. LIBERO’s full evaluation is 4 suites × 10 tasks × 50 initial states. This is one suite and ten initial states per task, so 86% carries a roughly ±7-point interval and is not a benchmark figure. Two agreeing bands raise confidence that the sampled states are not pathological, but twenty percent of the available states is still a sample. We also draw no comparison against published pi0-class results — that would need their replan horizon and step cap to match ours, which we have not verified.
Every episode is one JSON record carrying the suite, task, initial-state index, horizon, step cap, library versions, per-chunk backend and latency distribution, so the number is auditable rather than asserted.

​
Honest limitations

  • The task success rate is on a reduced protocol. A full LIBERO evaluation is 4 suites × 10 tasks × 50 episodes, which is not affordable on one board. Any rate here states its episode count and confidence interval and is never presented as a suite-level benchmark figure.
  • Gazebo will not render on this board, but MuJoCo will. gz-harmonic installs fine on arm64, but Ogre2 requires desktop OpenGL 3.3 core and the Adreno driver here exposes only OpenGL ES; Ogre v1 aborts and forcing Mesa llvmpipe segfaults inside Ogre2RenderEngine::LoadImpl. LIBERO’s MuJoCo renderer is satisfied by the OpenGL 4.5 compatibility profile that same llvmpipe advertises, which is why the closed-loop demo runs entirely on the board with no laptop involved. Rendering is still software: there is no hardware path, because /dev/dri/renderD128 is the display controller rather than the GPU, so Mesa’s freedreno cannot bind it, and zink is refused by both Vulkan ICDs.
  • Action semantics are embodiment-specific. The published export targets LIBERO’s 7-DoF Franka Panda. Its output is not valid joint commands for a different arm.
  • Normalization statistics are external. Raw joint values will produce meaningless prompts; you need the statistics the policy was trained with.
  • No power or thermal measurements, and no sustained-load soak test. All figures come from short runs on a thermally unstressed board.
  • Never measured with a live camera. Inputs were synthetic or replayed from disk.
  • The NPUs are exclusive. Two processes each mapping the full bundle contend for the same CDSP budget and both slow down. Measure with nothing else on the NPU.
  • The ~1600 MiB per-device ceiling we work to is empirical and conservative, not a documented limit.

​
What this says about the platform

100 TOPS makes a 3B vision-language-action model plausible. What the exercise showed is that it is also practical — at 4.3–4.5× real time — but only if you treat the two NPUs as a resource to schedule rather than as one accelerator. The gap between the naive path and the informed one is 4×, and the naive path also silently swallows a fatal mapping error. The remaining headroom is on the CPU, not the NPU: 223 ms per chunk of host-side tensor marshalling that zero-copy DMA-buf should largely remove.