LLMs/VLMs using Llama.cpp - Qualcomm Dragonwing Documentation

You can run a wide range of Large Language Models (LLMs) and Vision Language Models (VLMs) on your Dragonwing development boards using llama.cpp. llama.cpp supports three backends on Dragonwing devices: CPU, GPU (via OpenCL), and NPU (via the Hexagon HTP backend). You can run a subset of models on the NPU via GENIE as well.

Choose a backend

CPU: no special drivers or extra build flags needed. Useful as a baseline to compare against GPU or NPU performance.
GPU (OpenCL): build llama.cpp with the OpenCL backend to offload layers to the Adreno GPU. The same build also runs on CPU by skipping GPU offload.
NPU (Hexagon HTP): build llama.cpp with the Snapdragon toolchain container to offload layers to the Hexagon HTP device (HTP0) for the best performance on supported models.

NPU (Hexagon HTP)
GPU (OpenCL)
CPU

Overview

Build llama.cpp for Dragonwing devices with the Hexagon HTP backend, then run GGUF large language models on HTP0.

These instructions focus on the Snapdragon and Dragonwing llama.cpp build that exposes the Hexagon HTP device as HTP0. This is different from the OpenCL GPU workflow in the GPU tab.

Prerequisites

Before you begin, make sure you have:

Completed the first time setup for your Dragonwing device:
- Dragonwing IQ8 setup
- Dragonwing IQ9 setup
Access to the device by SSH, or by a directly connected display, keyboard, and mouse.
- The setup guides linked above include instructions for networking, serial console access, display setup, and SSH access.
Installed the required Dragonwing software packages on the device:
- Dragonwing IQ8
- Dragonwing IQ9
Installed Docker on the build host.
Enough free space for the build output and models. Plan for several GB per model.

The required software package setup installs the QNN runtime and tools, including libqnn-dev and qnn-tools, that llama.cpp needs for accelerated inference.

Prepare the build host

On your build host, clone llama.cpp or update an existing checkout.

mkdir -p ~/src
cd ~/src

# Clone if this is your first build
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

# For repeat builds, update to the latest upstream changes
git fetch origin
git checkout master
git pull --ff-only

If you need a reproducible build, record the commit you built:

git rev-parse HEAD

Build llama.cpp with the Snapdragon toolchain container

The easiest way to build llama.cpp for Dragonwing is to use the Snapdragon ARM64 Linux toolchain container. The container includes the ARM64 cross compiler, CMake, OpenCL SDK, and Hexagon SDK pieces needed by the Snapdragon preset. The Docker command below explicitly requests the linux/amd64 image.From the root of your llama.cpp checkout, start the container:

docker run -it \
  -u $(id -u):$(id -g) \
  --volume $(pwd):/workspace \
  --platform linux/amd64 \
  ghcr.io/snapdragon-toolchain/arm64-linux:v0.1

Inside the container, configure and build llama.cpp:

cd /workspace
cp docs/backend/snapdragon/CMakeUserPresets.json .

cmake --preset arm64-linux-snapdragon-release -B build-snapdragon
cmake --build build-snapdragon -j $(nproc)

Create an installable package:

cmake --install build-snapdragon --prefix pkg-snapdragon
zip -r pkg-snapdragon.zip pkg-snapdragon

Exit the container when the package is complete:

exit

The package archive is now available on the host at:

~/src/llama.cpp/pkg-snapdragon.zip

Rebuild when upstream llama.cpp changes

llama.cpp changes frequently. To rebuild with the latest upstream code, repeat this update and build flow from your host checkout:

cd ~/src/llama.cpp
git fetch origin
git checkout master
git pull --ff-only

docker run -it \
  -u $(id -u):$(id -g) \
  --volume $(pwd):/workspace \
  --platform linux/amd64 \
  ghcr.io/snapdragon-toolchain/arm64-linux:v0.1

Then, inside the container:

cd /workspace
cp docs/backend/snapdragon/CMakeUserPresets.json .

rm -rf build-snapdragon pkg-snapdragon pkg-snapdragon.zip
cmake --preset arm64-linux-snapdragon-release -B build-snapdragon
cmake --build build-snapdragon -j $(nproc)
cmake --install build-snapdragon --prefix pkg-snapdragon
zip -r pkg-snapdragon.zip pkg-snapdragon

If you are testing a branch, tag, or local llama.cpp changes, check out that source before running the Docker build command.

Copy the package to the Dragonwing device

Replace ubuntu@DEVICE_IP with your SSH user and target IP address.

scp ~/src/llama.cpp/pkg-snapdragon.zip ubuntu@DEVICE_IP:/home/ubuntu/

ssh ubuntu@DEVICE_IP

Unpack the package:

cd ~
unzip pkg-snapdragon.zip
cd pkg-snapdragon

Set the runtime library paths for the current shell:

export LD_LIBRARY_PATH=$PWD/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
export ADSP_LIBRARY_PATH=$PWD/lib${ADSP_LIBRARY_PATH:+:$ADSP_LIBRARY_PATH}

Verify that the package runs:

./bin/llama-cli --version

List the available llama.cpp devices:

./bin/llama-cli --list-devices

Expected output includes:

Available devices:
  HTP0: Hexagon

Make this the default llama.cpp install on the device

The packaged binaries need LD_LIBRARY_PATH and ADSP_LIBRARY_PATH so they can find the packaged llama.cpp and Hexagon backend libraries. The safest way to make this the default install is to move the package into /opt and create wrapper commands in /usr/local/bin.Run the following on the Dragonwing device:

cd ~
sudo rm -rf /opt/llama.cpp-snapdragon
sudo mkdir -p /opt
sudo cp -a ~/pkg-snapdragon /opt/llama.cpp-snapdragon
sudo chown -R root:root /opt/llama.cpp-snapdragon

Create a shared environment file:

sudo tee /etc/llama.cpp-snapdragon.env >/dev/null <<'EOF'
LLAMA_CPP_SNAPDRAGON_HOME=/opt/llama.cpp-snapdragon
LD_LIBRARY_PATH=/opt/llama.cpp-snapdragon/lib
ADSP_LIBRARY_PATH=/opt/llama.cpp-snapdragon/lib
EOF

Create wrapper commands for llama-cli and llama-server:

sudo tee /usr/local/bin/llama-cli >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
source /etc/llama.cpp-snapdragon.env
exec "$LLAMA_CPP_SNAPDRAGON_HOME/bin/llama-cli" "$@"
EOF

sudo tee /usr/local/bin/llama-server >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
source /etc/llama.cpp-snapdragon.env
exec "$LLAMA_CPP_SNAPDRAGON_HOME/bin/llama-server" "$@"
EOF

sudo chmod +x /usr/local/bin/llama-cli /usr/local/bin/llama-server
hash -r

Confirm that the default commands now resolve to the wrappers:

which llama-cli
which llama-server
llama-cli --version
llama-cli --list-devices

Expected paths:

/usr/local/bin/llama-cli
/usr/local/bin/llama-server

/usr/local/bin usually appears before /usr/bin in PATH, so these wrappers become the default commands without replacing system packages.

Download a model

llama.cpp uses models in GGUF format. A small instruct model is a good first test.Create a model directory on the Dragonwing device:

mkdir -p ~/models
cd ~/models

Download a Llama 3.2 3B instruct GGUF model:

wget https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_0.gguf

Model support and performance vary by architecture, quantization, context length, and llama.cpp commit. If you find a model or quantization that runs particularly well on Dragonwing devices, share it with the community.

Run your first prompt on HTP0

Run llama-cli and offload layers to the Hexagon HTP device:

llama-cli \
  -m ~/models/Llama-3.2-3B-Instruct-Q4_0.gguf \
  --device HTP0 \
  -ngl 99 \
  -p "What is the most popular cookie in the world?"

Useful options:

--device HTP0 selects the Hexagon HTP backend.
-ngl 99 asks llama.cpp to offload model layers to the selected device.
-m points to your GGUF model file.
-p passes a prompt for single prompt testing.

Start llama-server

llama-server exposes a local web UI and an OpenAI compatible API.Start a server on the Dragonwing device:

llama-server \
  -m ~/models/Llama-3.2-3B-Instruct-Q4_0.gguf \
  --device HTP0 \
  -ngl 99 \
  --host 0.0.0.0 \
  --port 8080

Find the device IP address:

hostname -I

From another machine on the same network, open:

http://DEVICE_IP:8080

You can also test the API with curl:

curl http://DEVICE_IP:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain Dragonwing in one sentence."}
    ],
    "temperature": 0.7,
    "max_tokens": 128
  }'

Update the default install after a rebuild

After you rebuild and copy a new pkg-snapdragon.zip to the device, update /opt/llama.cpp-snapdragon:

cd ~
rm -rf pkg-snapdragon
unzip pkg-snapdragon.zip

sudo rm -rf /opt/llama.cpp-snapdragon
sudo cp -a ~/pkg-snapdragon /opt/llama.cpp-snapdragon
sudo chown -R root:root /opt/llama.cpp-snapdragon

llama-cli --version
llama-cli --list-devices

The wrapper commands in /usr/local/bin do not need to be recreated unless you change the install path.

Troubleshooting

error while loading shared librariesIf you run binaries directly from the package directory, set the library paths first:

cd ~/pkg-snapdragon
export LD_LIBRARY_PATH=$PWD/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
export ADSP_LIBRARY_PATH=$PWD/lib${ADSP_LIBRARY_PATH:+:$ADSP_LIBRARY_PATH}
./bin/llama-cli --version

If you are using the default install wrappers, confirm the wrapper is being used:

which llama-cli

It should print /usr/local/bin/llama-cli.HTP0 does not appearConfirm that the required Dragonwing software packages are installed, especially libqnn-dev and qnn-tools. Then check devices again:

llama-cli --list-devices

Also confirm the package contains Hexagon backend libraries:

ls /opt/llama.cpp-snapdragon/lib/libggml-hexagon.so*
ls /opt/llama.cpp-snapdragon/lib/libggml-htp-*.so

The command uses the wrong llama.cpp binaryCheck command resolution:

type -a llama-cli
type -a llama-server

If another path appears before /usr/local/bin, update your PATH or call /usr/local/bin/llama-cli explicitly.

Building llama.cpp

You’ll need to build some dependencies for llama.cpp. Open the terminal on your development board, or an ssh session to your development board, and run:

Install build dependencies:

sudo apt update
sudo apt install -y cmake ninja-build curl libcurl4-openssl-dev build-essential

Install the OpenCL headers and ICD loader library:

mkdir -p ~/dev/llm

# Symlink the OpenCL shared library
sudo rm -f /usr/lib/libOpenCL.so
sudo ln -s /lib/aarch64-linux-gnu/libOpenCL.so.1.0.0 /usr/lib/libOpenCL.so

# OpenCL headers
cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-Headers
cd OpenCL-Headers
git checkout 5d52989617e7ca7b8bb83d7306525dc9f58cdd46
mkdir -p build && cd build
cmake .. -G Ninja \
    -DBUILD_TESTING=OFF \
    -DOPENCL_HEADERS_BUILD_TESTING=OFF \
    -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF \
    -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

# ICD Loader
cd ~/dev/llm
git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader
cd OpenCL-ICD-Loader
git checkout 02134b05bdff750217bf0c4c11a9b13b63957b04
mkdir -p build && cd build
cmake .. -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \
    -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
cmake --build . --target install

# Symlink OpenCL headers
sudo rm -rf /usr/include/CL
sudo ln -s ~/dev/llm/opencl/include/CL/ /usr/include/CL

Build llama.cpp with the OpenCL backend:

cd ~/dev/llm

# Clone repository
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

# We've tested this commit explicitly, you can try master if you want bleeding edge
git checkout f6da8cb86a28f0319b40d9d2a957a26a7d875f8c
git rev-parse HEAD
# Expected: f6da8cb86a28f0319b40d9d2a957a26a7d875f8c

# Build
mkdir -p build
cd build
cmake .. -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=OFF \
    -DGGML_OPENCL=ON
ninja -j`nproc`

Add the llama.cpp paths to your PATH:

cd ~/dev/llm/llama.cpp/build/bin

echo "" >> ~/.bash_profile
echo "# Begin llama.cpp" >> ~/.bash_profile
echo "export PATH=\$PATH:$PWD" >> ~/.bash_profile
echo "# End llama.cpp" >> ~/.bash_profile
echo "" >> ~/.bash_profile

# To use the llama.cpp files in your current session
source ~/.bash_profile

You now have llama.cpp:

llama-cli --version
# ggml_opencl: selected platform: 'QUALCOMM Snapdragon(TM)'
# ggml_opencl: device: 'QUALCOMM Adreno(TM) 663 (OpenCL 3.0 Adreno(TM) 663)'
# ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: 0808.0.7 Compiler E031.49.02.00
# ggml_opencl: vector subgroup broadcast support: true

Downloading and quantizing a model

To run GPU-accelerated models you’ll want pure 4-bit quantized (Q4_0) models in GGUF format (the llama.cpp format, conversion guide). You can either find pre-quantized models, or quantize a model yourself using llama-quantize. For example, for Qwen2-1.5B-Instruct:

# Download fp16 model (https://huggingface.co/Qwen/Qwen2-1.5B-Instruct-GGUF)
wget https://huggingface.co/Qwen/Qwen2-1.5B-Instruct-GGUF/resolve/main/qwen2-1_5b-instruct-fp16.gguf

# Quantize (pure Q4_0)
llama-quantize --pure qwen2-1_5b-instruct-fp16.gguf qwen2-1_5b-instruct-q4_0-pure.gguf Q4_0

Running your first LLM using llama-cli

You’re now ready to run the LLM via llama-cli. It’ll automatically offload layers to the GPU:

llama-cli -m ./qwen2-1_5b-instruct-q4_0-pure.gguf -no-cnv --no-warmup -b 128 -c 2048 -s 11 -n 128 -p "Knock knock, " -fa off

# ... You'll see:
# load_tensors: offloaded 29/29 layers to GPU
# ...
# Knock knock, 11:59 pm ... rest of the story

🚀 You now have an LLM running on the GPU of your device!

Serving LLMs using llama-server

Next, you can use llama-server to start a web server with a chat interface, and an OpenAI compatible chat completions API.

First, find the IP address of your development board:

ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'

# ... Example:
# 192.168.1.253

Start the server via:

llama-server -m ./qwen2-1_5b-instruct-q4_0-pure.gguf --no-warmup -b 128 -c 2048 -s 11 -n 128 --host 0.0.0.0 --port 9876

On your computer, open a web browser and navigate to http://192.168.1.253:9876 (replace the IP address with the one you found in 1.):
Serving LLMs using llama-server

You can also programmatically access this server using the OpenAI Chat Completions API. E.g. from Python:

Create a new venv and install requests:

python3 -m venv .venv-chat
source .venv-chat/bin/activate
pip3 install requests

Create a new file chat.py:

import requests

# if running from your own computer, replace localhost with the IP address of your development board
url = "http://localhost:9876/v1/chat/completions"

payload = {
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain Qualcomm in one sentence."}
    ],
    "temperature": 0.7,
    "max_tokens": 200
}

response = requests.post(url, headers={ "Content-Type": "application/json" }, json=payload)
print(response.json())

Run chat.py:

python3 chat.py

# ...
# {'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': 'Qualcomm is a leading global technology company that designs, develops, licenses, and markets semiconductor-based products and mobile platform technologies to major telecommunications and consumer electronics manufacturers worldwide.'}}], 'created': 1757073340, 'model': 'gpt-3.5-turbo', 'system_fingerprint': 'b6362-f6da8cb8', 'object': 'chat.completion', 'usage': {'completion_tokens': 34, 'prompt_tokens': 26, 'total_tokens': 60}, 'id': 'chatcmpl-3O7l005WG1DzN191FTNomJNweHMoH8Is', 'timings': {'prompt_n': 12, 'prompt_ms': 303.581, 'prompt_per_token_ms': 25.298416666666668, 'prompt_per_second': 39.52816546490064, 'predicted_n': 34, 'predicted_ms': 4052.23, 'predicted_per_token_ms': 119.18323529411765, 'predicted_per_second': 8.390441806116632}}

You can also use multi-modal LLMs. For example SmolVLM-500M-Instruct-GGUF. Download both the Q4_0 quantized weights (or quantize them yourself), and download the CLIP encoder mmproj-*.gguf file. For example:

# Download weights
wget https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF/resolve/main/SmolVLM-500M-Instruct-f16.gguf
wget https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF/resolve/main/mmproj-SmolVLM-500M-Instruct-f16.gguf

# Quantize model (mmproj- models are not quantizable via llama-quantize, see below)
llama-quantize --pure SmolVLM-500M-Instruct-f16.gguf SmolVLM-500M-Instruct-q4_0-pure.gguf Q4_0

# Serve the model
llama-server -m ./SmolVLM-500M-Instruct-q4_0-pure.gguf --mmproj ./mmproj-SmolVLM-500M-Instruct-f16.gguf --no-warmup -b 128 -c 2048 -s 11 -n 128 --host 0.0.0.0 --port 9876

Serving multi-modal LLMs using llama-server

CLIP model is still fp16: The mmproj model is still fp16; and thus processing images will be slow. There is code to quantize the CLIP encoder in older versions of llama.cpp, that you can explore.

Running on CPU

You don’t need a separate build to try CPU inference. The same OpenCL-enabled build from the GPU tab runs on CPU too. Add -ngl 0 to any llama-* command to skip offloading layers to the GPU, which is a quick way to compare CPU and GPU performance on the same model and build.For example, the Qwen2-1.5B-Instruct Q4_0 model (see the GPU tab for how to download and quantize it):GPU:

llama-cli -m ./qwen2-1_5b-instruct-q4_0-pure.gguf -no-cnv --no-warmup -b 128 -c 2048 -s 11 -n 128 -p "Knock knock, " -fa off

# llama_perf_sampler_print:    sampling time =      26.33 ms /   133 runs   (    0.20 ms per token,  5050.70 tokens per second)
# llama_perf_context_print:        load time =    3535.69 ms
# llama_perf_context_print: prompt eval time =     192.38 ms /     5 tokens (   38.48 ms per token,    25.99 tokens per second)
# llama_perf_context_print:        eval time =    5679.81 ms /   127 runs   (   44.72 ms per token,    22.36 tokens per second)
# llama_perf_context_print:       total time =    9276.10 ms /   132 tokens
# llama_perf_context_print:    graphs reused =        122

CPU:

llama-cli -m ./qwen2-1_5b-instruct-q4_0-pure.gguf -no-cnv --no-warmup -b 128 -ngl 0 -c 2048 -s 11 -n 128 -p "Knock knock, " -fa off

# llama_perf_sampler_print:    sampling time =      15.44 ms /   133 runs   (    0.12 ms per token,  8615.66 tokens per second)
# llama_perf_context_print:        load time =    1061.95 ms
# llama_perf_context_print: prompt eval time =      51.75 ms /     5 tokens (   10.35 ms per token,    96.62 tokens per second)
# llama_perf_context_print:        eval time =    2789.13 ms /   127 runs   (   21.96 ms per token,    45.53 tokens per second)
# llama_perf_context_print:       total time =    3885.55 ms /   132 tokens
# llama_perf_context_print:    graphs reused =        122

Here the CPU evaluates tokens about twice as fast as the GPU for this small model. Smaller models can be memory-bandwidth bound enough that CPU keeps up with or beats the GPU, so it’s worth comparing both for your specific model and quantization.llama-server also honors -ngl 0, so you can serve on CPU using the same command shown in the GPU tab’s serving section.

​Choose a backend

​Overview

​Prerequisites

​Prepare the build host

​Build llama.cpp with the Snapdragon toolchain container

​Rebuild when upstream llama.cpp changes

​Copy the package to the Dragonwing device

​Make this the default llama.cpp install on the device

​Download a model

​Run your first prompt on HTP0

​Start llama-server

​Update the default install after a rebuild

​Troubleshooting

​Building llama.cpp

​Downloading and quantizing a model

​Running your first LLM using llama-cli

​Serving LLMs using llama-server

​Serving multi-modal LLMs

​Running on CPU

Choose a backend

Overview

Prerequisites

Prepare the build host

Build llama.cpp with the Snapdragon toolchain container

Rebuild when upstream llama.cpp changes

Copy the package to the Dragonwing device

Make this the default llama.cpp install on the device

Download a model

Run your first prompt on HTP0

Start llama-server

Update the default install after a rebuild

Troubleshooting

Building llama.cpp

Downloading and quantizing a model

Running your first LLM using llama-cli

Serving LLMs using llama-server

Serving multi-modal LLMs

Running on CPU