> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# LLMs/VLMs using GenieX

> Install the GenieX CLI on a Qualcomm Dragonwing IQ9 device, pull the Gemma 4 E4B bundle, and run text, image, and audio inference on the Hexagon NPU.

GenieX is the Qualcomm AI Hub command line and serving layer for generative models. On
a Dragonwing ARM64 Linux host it runs models natively against the device NPU through
the Qualcomm driver libraries.

This page takes you from a freshly imaged IQ9 board to Gemma 4 E4B answering prompts on
the NPU. Expect around 15 minutes of setup, most of it spent downloading the \~5 GB model
bundle.

You will:

1. Install the system and Qualcomm driver packages, then the `geniex` CLI.
2. Pull the Gemma 4 E4B bundle from AI Hub.
3. Run text, image, and audio inference, and optionally serve an OpenAI-compatible API.

<Warning>
  Every step on this page runs **directly on the IQ9 device**, not on a development host.
</Warning>

Open a session on the board:

```bash theme={null}
ssh ubuntu@<device-ip>
```

## Prerequisites

| Requirement      | Value                                                                      |
| ---------------- | -------------------------------------------------------------------------- |
| Architecture     | Linux ARM64 (`aarch64`)                                                    |
| Operating system | Ubuntu image for Dragonwing IQ-9075 EVK, or any device with a Qualcomm BSP |
| Chipset          | Dragonwing IoT chipset — IQ-9075 / QCS9075 in this blog                    |
| Free storage     | ≥ 12 GB                                                                    |
| RAM              | ≥ 12 GB                                                                    |
| Network          | Outbound HTTPS to `qaihub-public-assets.s3.us-west-2.amazonaws.com`        |

Confirm the architecture and available space before continuing:

```bash theme={null}
uname -m      # expected: aarch64
df -h /       # confirm at least 12 GB available
```

<Note>
  If the board is not yet brought up, complete device setup first — see the
  [IQ-9075 EVK setup guide](https://dragonwingdocs.qualcomm.com/Ubuntu/devices/iq9075-evk/set-up-the-device).
  This is only applicable to IQ9 and please refer [GenieX](https://aihub.qualcomm.com/geniex) page for other platforms and select the supported model.
</Note>

## Install GenieX

<Steps>
  <Step title="Install standard APT packages">
    ```bash theme={null}
    sudo apt update
    sudo apt install -y libatomic1 libglib2.0-0 ocl-icd-libopencl1
    ```

    <Note>
      `ocl-icd-libopencl1` is replaced by `qcom-adreno1` in the next step; installing it
      first keeps the dependency resolver happy on minimal images.
    </Note>
  </Step>

  <Step title="Install the Qualcomm driver packages">
    `geniex` reaches the NPU through the Qualcomm proprietary driver libraries,
    delivered from the `ubuntu-qcom-iot` PPA.

    ```bash theme={null}
    sudo apt-get install -y qcom-adreno1 qcom-fastrpc1 libqnn1
    ```

    | Library                                                                      | Provided by     |
    | ---------------------------------------------------------------------------- | --------------- |
    | `libOpenCL_adreno.so.1`, `libCB.so.1`, `libadreno_utils.so.1`, `libgsl.so.1` | `qcom-adreno1`  |
    | `libcdsprpc.so.1.0.0`                                                        | `qcom-fastrpc1` |
    | QNN backend libraries (`libQnnHtp.so`, `libQnnSystem.so`)                    | `libqnn1`       |

    <Note>
      The PPA (`ppa:ubuntu-qcom-iot/qcom-ppa`) is pre-configured on the IQ9075 EVK Ubuntu
      image. On a custom image, add it with
      `sudo add-apt-repository -y ppa:ubuntu-qcom-iot/qcom-ppa` followed by `sudo apt update`.
    </Note>

    These three packages are the most common cause of installation failure. If a later
    step reports a missing `.so`, return here.
  </Step>

  <Step title="Run the installer">
    If `HOME` is unset — common in minimal containers or `sudo -s` sessions — export it
    first:

    ```bash theme={null}
    export HOME=/home/ubuntu
    ```

    ```bash theme={null}
    curl -fsSL https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-geniex/install.sh | sh
    ```

    The script downloads the latest stable release, verifies its SHA256 checksum, and
    installs without `sudo`.
  </Step>

  <Step title="Add GenieX to your PATH">
    The installer prints the exact export line if the launcher directory is not already
    on your `PATH`. For example:

    ```bash theme={null}
    echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc
    source ~/.bashrc
    ```
  </Step>

  <Step title="Optional: install SoX for microphone input">
    Needed only for the `/mic` command in an interactive session. Loading audio from
    disk works without it.

    ```bash theme={null}
    sudo apt install -y sox
    ```
  </Step>
</Steps>

### Verify the installation

```bash theme={null}
geniex --help
```

If `geniex` is not found, open a new shell or apply the `PATH` line the installer
printed. If the command fails on a missing shared library such as `libCB.so.1` or
`libcdsprpc.so.1.0.0`, the Qualcomm driver packages are not installed — see step 2.

Add `--log` to any command to raise the log level. The flag is equivalent to the
`GENIEX_LOG` environment variable and takes precedence over it.

| Level   | Emits                            |
| ------- | -------------------------------- |
| `none`  | nothing (default)                |
| `error` | errors only                      |
| `warn`  | warnings and errors              |
| `info`  | informational messages and above |
| `debug` | debug messages and above         |
| `trace` | everything                       |

## Download the model

```bash theme={null}
geniex pull google/gemma-4-E4B-it-qat-q4_0-gguf
```

This transfers roughly 5 GB, so allow several minutes on a typical link. If the transfer
is interrupted, re-run the same command — it resumes rather than starting over.

The general syntax is:

```bash theme={null}
geniex pull <model-name>[:<precision>]
```

| Flag           | Purpose                                                                |
| -------------- | ---------------------------------------------------------------------- |
| `--model-hub`  | Model source: `aihub`, `hf`, or `localfs`. Auto-detected when omitted. |
| `--local-path` | Register a bundle already present on disk.                             |

For GGUF models the CLI prompts for a precision when more than one is published. Choose
**`Q4_0`** for IQ9 — it is the quantization-aware-trained build and gives the best
accuracy per byte on the NPU. Pin it inline to skip the prompt in scripts:

```bash theme={null}
geniex pull google/gemma-4-E4B-it-qat-q4_0-gguf:Q4_0
```

Confirm the result:

```bash theme={null}
geniex list
```

<Note>
  `pull` copies files into the GenieX cache. After a successful `--local-path` pull you can
  delete the source directory rather than keeping two copies of a \~5 GB model.
</Note>

### Bundle contents

GenieX manages the GGUF bundle for you. It contains the quantized `*.gguf` weights, an
`mmproj-*.gguf` multimodal projector for image and audio input, tokenizer metadata
embedded in the GGUF container, and a manifest recording precision, runtime, and compute
defaults.

A **Genie/QAIRT** bundle — the `w4a16` build, or output from the Jupyter path — is
instead explicit, and must contain:

| File                          | Purpose                                                                |
| ----------------------------- | ---------------------------------------------------------------------- |
| `genie_config.json`           | Backend selection, model and tokenizer paths, context and token limits |
| `htp_backend_ext_config.json` | HTP/NPU backend settings (SoC ID, DSP architecture, performance mode)  |
| `tokenizer.json`              | Tokenizer vocabulary and merge rules                                   |
| `*.bin`                       | Prompt-processor and token-generator context binaries                  |

<Note>
  Sample Genie configurations are published in the
  [AI Hub Apps repository](https://github.com/qualcomm/ai-hub-apps/tree/main/tutorials/llm_on_genie/configs/genie);
  for field definitions see the
  [Genie dialog JSON reference](https://docs.qualcomm.com/doc/80-63442-10/topic/json.html#genie-dialog-json-config-string).
</Note>

## Run inference

Start an interactive chat session:

```bash theme={null}
geniex infer google/gemma-4-E4B-it-qat-q4_0-gguf
```

The model loads onto the NPU and you get a prompt. Type a message and press Enter.

Pass a single prompt and exit instead — useful for scripts and smoke tests:

```bash theme={null}
geniex infer google/gemma-4-E4B-it-qat-q4_0-gguf \
  -p "Summarize what a Hexagon NPU does in three sentences."
```

### Common flags

| Flag                              | Effect                                                                                                       |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `-p "<prompt>"`                   | Run one prompt and exit instead of opening a session.                                                        |
| `--think`                         | Show intermediate reasoning before the answer.                                                               |
| `--think=false`                   | Answer directly. Recommended in production.                                                                  |
| `--compute npu`                   | Run on the Hexagon NPU. This is the default.                                                                 |
| `--compute cpu` / `--compute gpu` | Run on CPU or GPU. Valid for GGUF builds only; useful for A/B comparison or isolating an NPU driver problem. |
| `--log <level>`                   | Raise the log level for diagnostics.                                                                         |

### Multimodal prompts

The `q4_0` bundle ships an audio-capable projector, so a single prompt can carry both an
image and an audio clip. Download two sample files into your home directory:

```bash theme={null}
cd ~
curl -L -o jfk.wav https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav
curl -L -o landmark.jpg "https://images.pexels.com/photos/402028/pexels-photo-402028.jpeg?w=1024"
```

Reference them by absolute path in the prompt:

```bash theme={null}
geniex infer google/gemma-4-E4B-it-qat-q4_0-gguf \
  -p "Describe the image and transcribe the audio. Image: $HOME/landmark.jpg Audio: $HOME/jfk.wav"
```

Expected output:

```
**Image Description:**
This is a scenic, panoramic photograph that features a traditional Japanese temple ...

**Audio Transcription:**
And so my fellow Americans, ask not what your country can do for you, ask what you
can do for your country.
```

<Note>
  Always use **absolute** paths for image and audio inputs. Relative paths resolve against
  the process working directory and are a frequent source of "file not found" errors.
</Note>

In an interactive session, `/mic` records a clip instead of loading one from disk;
`Ctrl-C` stops recording and transcribes it. This requires SoX on your `PATH`.

<Warning>
  QAIRT models report `audio: false`. Passing one an audio file fails with
  `GenieXError(-201201): Multimodal generation failed`. Use the GGUF build for audio.
</Warning>

### Serve an OpenAI-compatible endpoint

For application integration, run the local server instead of the interactive CLI:

```bash theme={null}
geniex serve
```

The server prints the address and port it is listening on at startup. Substitute those
values below:

```bash theme={null}
curl http://<device-ip>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E4B-it-qat-q4_0-gguf",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of Italy?"}
    ],
    "stream": false
  }'
```

Any OpenAI-compatible client or framework works, including LangChain and Open WebUI. Set
the per-request `reasoning_format` field to move a thinking model's chain-of-thought out
of `message.content` into `message.reasoning_content`, which keeps the rendered answer
clean while preserving the reasoning for logs.

<Note>
  Test with `curl` from the device itself first. If the endpoint works locally but not
  remotely, the server is bound to loopback or a firewall is blocking the port.
</Note>

## Performance and best practices

On IQ9 the reference figures are a 4096-token context, \~660 tokens/s prefill, and
\~17.9 tokens/s decode. Because decode is roughly 37× slower than prefill, output length
dominates response time far more than prompt length.

* **Reuse the loaded model.** Model load is the largest fixed cost — use `geniex serve`
  or a long-lived interactive session rather than invoking `geniex infer` per request.
* **Cap output length.** Ask for "three bullet points" rather than "explain in detail"
  when latency matters.
* **Keep prompts short.** Prefill is fast, but prompt tokens still consume context that
  generation needs.
* **Disable thinking mode** (`--think=false`) unless you consume the reasoning trace; it
  can multiply the number of generated tokens.
* **Right-size the context.** KV-cache memory scales with context length, so do not
  configure a large window for a task that never needs it.
* **Watch thermals.** Sustained generation raises SoC temperature and triggers
  throttling; measure steady-state throughput, not just the first request.
* **Keep storage headroom.** Provision roughly double the \~5 GB bundle size to allow for
  upgrades.

## Verify the result

| # | Check                  | Command                                                                | Expected                                            |
| - | ---------------------- | ---------------------------------------------------------------------- | --------------------------------------------------- |
| 1 | Correct architecture   | `uname -m`                                                             | `aarch64`                                           |
| 2 | Drivers installed      | `apt list --installed qcom-adreno1 qcom-fastrpc1 libqnn1`              | All three present                                   |
| 3 | CLI works              | `geniex --help`                                                        | Usage text, no library errors                       |
| 4 | Model in cache         | `geniex list`                                                          | Gemma 4 E4B listed                                  |
| 5 | Inference returns text | `geniex infer google/gemma-4-E4B-it-qat-q4_0-gguf -p "Reply with OK."` | A short coherent reply within seconds of model load |
| 6 | NPU backend selected   | `geniex --log debug infer google/gemma-4-E4B-it-qat-q4_0-gguf -p "Hi"` | NPU backend in the log, not a CPU fallback          |

If any check fails, see Troubleshooting.

## Next step

The published bundle is now running on the NPU. If it meets your accuracy, context
length, and licensing requirements, you are done — move on to integrating it with your
application.

If you need a custom quantization recipe, a longer context, or your own checkpoint,
continue to the Jupyter notebook path. Otherwise go to Troubleshooting and next steps.