> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# 使用 Llama.cpp 运行 LLM/VLM

您可以借助 [llama.cpp](https://github.com/ggml-org/llama.cpp) 在 Dragonwing 开发板上
运行各种大语言模型 (LLM) 和视觉语言模型 (VLM)。在 llama.cpp 下运行的模型跑在 *GPU*
上,而非 *NPU* 上。您可以通过 [GENIE](https://docs.qualcomm.com/doc/80-63442-10/topic/introduction.html)
在 NPU 上运行部分模型。

## 构建 llama.cpp

您需要为 llama.cpp 构建若干依赖。在开发板上打开终端,或通过 ssh 连接到开发板,然后
运行:

<Steps>
  <Step title="安装构建依赖">
    ```
    sudo apt update
    sudo apt install -y cmake ninja-build curl libcurl4-openssl-dev build-essential
    ```
  </Step>

  <Step title="安装 OpenCL 头文件和 ICD 加载库">
    ```shell theme={null}
    mkdir -p ~/dev/llm

    # Symlink the OpenCL shared library
    sudo rm -f /usr/lib/libOpenCL.so
    sudo ln -s /lib/aarch64-linux-gnu/libOpenCL.so.1.0.0 /usr/lib/libOpenCL.so

    # OpenCL headers
    cd ~/dev/llm
    git clone https://github.com/KhronosGroup/OpenCL-Headers
    cd OpenCL-Headers
    git checkout 5d52989617e7ca7b8bb83d7306525dc9f58cdd46
    mkdir -p build && cd build
    cmake .. -G Ninja \
        -DBUILD_TESTING=OFF \
        -DOPENCL_HEADERS_BUILD_TESTING=OFF \
        -DOPENCL_HEADERS_BUILD_CXX_TESTS=OFF \
        -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
    cmake --build . --target install

    # ICD Loader
    cd ~/dev/llm
    git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader
    cd OpenCL-ICD-Loader
    git checkout 02134b05bdff750217bf0c4c11a9b13b63957b04
    mkdir -p build && cd build
    cmake .. -G Ninja \
        -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_PREFIX_PATH="$HOME/dev/llm/opencl" \
        -DCMAKE_INSTALL_PREFIX="$HOME/dev/llm/opencl"
    cmake --build . --target install

    # Symlink OpenCL headers
    sudo rm -rf /usr/include/CL
    sudo ln -s ~/dev/llm/opencl/include/CL/ /usr/include/CL
    ```
  </Step>

  <Step title="使用 OpenCL 后端构建 llama.cpp">
    ```
    cd ~/dev/llm

    # Clone repository
    git clone https://github.com/ggml-org/llama.cpp
    cd llama.cpp

    # We've tested this commit explicitly, you can try master if you want bleeding edge
    git checkout f6da8cb86a28f0319b40d9d2a957a26a7d875f8c
    git rev-parse HEAD
    # Expected: f6da8cb86a28f0319b40d9d2a957a26a7d875f8c

    # Build
    mkdir -p build
    cd build
    cmake .. -G Ninja \
        -DCMAKE_BUILD_TYPE=Release \
        -DBUILD_SHARED_LIBS=OFF \
        -DGGML_OPENCL=ON
    ninja -j`nproc`
    ```
  </Step>

  <Step title="将 llama.cpp 路径添加到 PATH">
    ```
    cd ~/dev/llm/llama.cpp/build/bin

    echo "" >> ~/.bash_profile
    echo "# Begin llama.cpp" >> ~/.bash_profile
    echo "export PATH=\$PATH:$PWD" >> ~/.bash_profile
    echo "# End llama.cpp" >> ~/.bash_profile
    echo "" >> ~/.bash_profile

    # To use the llama.cpp files in your current session
    source ~/.bash_profile
    ```
  </Step>

  <Step title="确认 llama.cpp 已安装">
    ```
    llama-cli --version
    # ggml_opencl: selected platform: 'QUALCOMM Snapdragon(TM)'
    # ggml_opencl: device: 'QUALCOMM Adreno(TM) 663 (OpenCL 3.0 Adreno(TM) 663)'
    # ggml_opencl: OpenCL driver: OpenCL 3.0 QUALCOMM build: 0808.0.7 Compiler E031.49.02.00
    # ggml_opencl: vector subgroup broadcast support: true
    ```
  </Step>
</Steps>

### 下载并量化模型

要运行 GPU 加速的模型,您需要 GGUF 格式的纯 4 位量化(`Q4_0`)模型(llama.cpp 的格式,
参见[转换指南](https://github.com/ggml-org/llama.cpp/discussions/2948))。您可以查找
已量化好的模型,也可以使用 `llama-quantize` 自行量化。例如,针对 Qwen2-1.5B-Instruct:

```
# Download fp16 model (https://huggingface.co/Qwen/Qwen2-1.5B-Instruct-GGUF)
wget https://huggingface.co/Qwen/Qwen2-1.5B-Instruct-GGUF/resolve/main/qwen2-1_5b-instruct-fp16.gguf

# Quantize (pure Q4_0)
llama-quantize --pure qwen2-1_5b-instruct-fp16.gguf qwen2-1_5b-instruct-q4_0-pure.gguf Q4_0
```

### 使用 llama-cli 运行您的第一个 LLM

您现在可以通过 `llama-cli` 运行 LLM。它会自动将部分层卸载到 GPU:

```
llama-cli -m ./qwen2-1_5b-instruct-q4_0-pure.gguf -no-cnv --no-warmup -b 128 -c 2048 -s 11 -n 128 -p "Knock knock, " -fa off

# ... You'll see:
# load_tensors: offloaded 29/29 layers to GPU
# ...
# Knock knock, 11:59 pm ... rest of the story
```

🚀 您的设备 GPU 上现在已经运行起了一个 LLM!

### 使用 llama-server 提供 LLM 服务

接下来,您可以使用 `llama-server` 启动一个带聊天界面和兼容 OpenAI 的 chat completions
API 的 Web 服务器。

<Steps>
  <Step title="查找开发板的 IP 地址">
    ```
    ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'

    # ... Example:
    # 192.168.1.253
    ```
  </Step>

  <Step title="启动服务器">
    ```
    llama-server -m ./qwen2-1_5b-instruct-q4_0-pure.gguf --no-warmup -b 128 -c 2048 -s 11 -n 128 --host 0.0.0.0 --port 9876
    ```
  </Step>

  <Step title="在浏览器中打开聊天界面">
    在您的电脑上打开 Web 浏览器并访问 `http://192.168.1.253:9876`(请将 IP 地址替换为步骤 1 中找到的地址):

    <Frame caption="使用 llama-server 提供 LLM 服务">
      <img src="https://mintlify.s3.us-west-1.amazonaws.com/qualcomm-prod/zh/AI-Developer-Workflow-Ubuntu/_images/llamacpp1.png" />
    </Frame>
  </Step>

  <Step title="通过 OpenAI Chat Completions API 以编程方式访问服务器">
    以 Python 为例:

    <Steps>
      <Step title="创建新的 venv 并安装 `requests`">
        ```
        python3 -m venv .venv-chat
        source .venv-chat/bin/activate
        pip3 install requests
        ```
      </Step>

      <Step title="创建新文件 `chat.py`">
        ```
        import requests

        # if running from your own computer, replace localhost with the IP address of your development board
        url = "http://localhost:9876/v1/chat/completions"

        payload = {
            "messages": [
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": "Explain Qualcomm in one sentence."}
            ],
            "temperature": 0.7,
            "max_tokens": 200
        }

        response = requests.post(url, headers={ "Content-Type": "application/json" }, json=payload)
        print(response.json())
        ```
      </Step>

      <Step title="运行 `chat.py`">
        ```
        python3 chat.py

        # ...
        # {'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'role': 'assistant', 'content': 'Qualcomm is a leading global technology company that designs, develops, licenses, and markets semiconductor-based products and mobile platform technologies to major telecommunications and consumer electronics manufacturers worldwide.'}}], 'created': 1757073340, 'model': 'gpt-3.5-turbo', 'system_fingerprint': 'b6362-f6da8cb8', 'object': 'chat.completion', 'usage': {'completion_tokens': 34, 'prompt_tokens': 26, 'total_tokens': 60}, 'id': 'chatcmpl-3O7l005WG1DzN191FTNomJNweHMoH8Is', 'timings': {'prompt_n': 12, 'prompt_ms': 303.581, 'prompt_per_token_ms': 25.298416666666668, 'prompt_per_second': 39.52816546490064, 'predicted_n': 34, 'predicted_ms': 4052.23, 'predicted_per_token_ms': 119.18323529411765, 'predicted_per_second': 8.390441806116632}}
        ```
      </Step>
    </Steps>
  </Step>
</Steps>

### 提供多模态 LLM 服务

您也可以使用多模态 LLM,例如
[SmolVLM-500M-Instruct-GGUF](https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF)。
下载 Q4\_0 量化权重(或自行量化),并下载 CLIP 编码器 `mmproj-*.gguf` 文件。例如:

```
# Download weights
wget https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF/resolve/main/SmolVLM-500M-Instruct-f16.gguf
wget https://huggingface.co/ggml-org/SmolVLM-500M-Instruct-GGUF/resolve/main/mmproj-SmolVLM-500M-Instruct-f16.gguf

# Quantize model (mmproj- models are not quantizable via llama-quantize, see below)
llama-quantize --pure SmolVLM-500M-Instruct-f16.gguf SmolVLM-500M-Instruct-q4_0-pure.gguf Q4_0

# Serve the model
llama-server -m ./SmolVLM-500M-Instruct-q4_0-pure.gguf --mmproj ./mmproj-SmolVLM-500M-Instruct-f16.gguf --no-warmup -b 128 -c 2048 -s 11 -n 128 --host 0.0.0.0 --port 9876
```

<Frame caption="使用 llama-server 提供多模态 LLM 服务">
  <img src="https://mintlify.s3.us-west-1.amazonaws.com/qualcomm-prod/zh/AI-Developer-Workflow-Ubuntu/_images/llamacpp2.png" />
</Frame>

**CLIP 模型仍为 fp16:** `mmproj` 模型仍然是 fp16;因此图像处理速度较慢。在
[较旧版本的 llama.cpp](https://github.com/ggml-org/llama.cpp/pull/11644) 中有量化
CLIP 编码器的代码,您可以参考。

## 提示与技巧

### 对比 CPU 性能

在 `llama-*` 命令中加上 `-ngl 0` 可跳过将层卸载到 GPU。模型将在 CPU 上运行,您可以将
性能与 GPU 对比。

以 Qwen2-1.5B-Instruct Q4\_0 为例:

**GPU:**

```
llama-cli -m ./qwen2-1_5b-instruct-q4_0-pure.gguf -no-cnv --no-warmup -b 128 -c 2048 -s 11 -n 128 -p "Knock knock, " -fa off

# llama_perf_sampler_print:    sampling time =      26.33 ms /   133 runs   (    0.20 ms per token,  5050.70 tokens per second)
# llama_perf_context_print:        load time =    3535.69 ms
# llama_perf_context_print: prompt eval time =     192.38 ms /     5 tokens (   38.48 ms per token,    25.99 tokens per second)
# llama_perf_context_print:        eval time =    5679.81 ms /   127 runs   (   44.72 ms per token,    22.36 tokens per second)
# llama_perf_context_print:       total time =    9276.10 ms /   132 tokens
# llama_perf_context_print:    graphs reused =        122
```

**CPU:**

```
llama-cli -m ./qwen2-1_5b-instruct-q4_0-pure.gguf -no-cnv --no-warmup -b 128 -ngl 99 -c 2048 -s 11 -n 128 -p "Knock knock, " -fa off -ngl 0

# llama_perf_sampler_print:    sampling time =      15.44 ms /   133 runs   (    0.12 ms per token,  8615.66 tokens per second)
# llama_perf_context_print:        load time =    1061.95 ms
# llama_perf_context_print: prompt eval time =      51.75 ms /     5 tokens (   10.35 ms per token,    96.62 tokens per second)
# llama_perf_context_print:        eval time =    2789.13 ms /   127 runs   (   21.96 ms per token,    45.53 tokens per second)
# llama_perf_context_print:       total time =    3885.55 ms /   132 tokens
# llama_perf_context_print:    graphs reused =        122
```

在这里,CPU 对 token 的评估速度大约是 GPU 的两倍。