> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Whisper

> 在 Dragonwing 设备上运行 OpenAI 的 Whisper 语音识别模型,可选择使用 NPU(VoiceAI ASR)或 CPU/GPU(whisper.cpp)。

Whisper 是 OpenAI 推出的通用自动语音识别 (ASR) 模型。您可以将其用于音频转写、翻译和
语种识别。您可以借助 Qualcomm 的
[VoiceAI ASR](https://softwarecenter.qualcomm.com/catalog/item/VoiceAI_ASR_Community)
在 Dragonwing 开发板的 NPU 上运行 Whisper,也可以借助
[whisper.cpp](https://github.com/ggml-org/whisper.cpp) 在 CPU 上运行 Whisper。

<Note>
  通过 AIHUB 对 NPU 上 Whisper 模型的支持正在推进中,不久将更新。
</Note>

## 使用 VoiceAI ASR 在 NPU 上运行 Whisper

### 1. 安装 SDK

<Steps>
  <Step title="准备基础环境">
    在开发板上打开终端,并为本示例准备基础依赖:

    ```shell theme={null}
    sudo apt install -y cmake pulseaudio-utils
    ```
  </Step>

  <Step title="安装 VoiceAI ASR - Community Edition">
    安装 [VoiceAI ASR - Community Edition](https://softwarecenter.qualcomm.com/catalog/item/VoiceAI_ASR_Community):

    ```shell theme={null}
    cd ~/

    wget https://softwarecenter.qualcomm.com/api/download/software/sdks/VoiceAI_ASR_Community/All/2.3.0.0/VoiceAI_ASR_Community_v2.3.0.0.zip
    unzip VoiceAI_ASR_Community_v2.3.0.0.zip -d voiceai_asr
    rm VoiceAI_ASR_Community_v2.3.0.0.zip

    cd voiceai_asr/2.3.0.0/

    # Put the path to VoiceAI ASR in your bash_profile (so it's available under VOICEAI_ROOT)
    echo "" >> ~/.bash_profile
    echo "# Begin VoiceAI ASR" >> ~/.bash_profile
    echo "export VOICEAI_ROOT=$PWD" >> ~/.bash_profile
    echo "# End VoiceAI ASR" >> ~/.bash_profile
    echo "" >> ~/.bash_profile

    # Re-load the environment variables
    source ~/.bash_profile

    # Symlink Whisper libraries
    cd $VOICEAI_ROOT/whisper_sdk/libs/npu/rpc_libraries/linux/whisper_all_quantized/
    sudo ln -s $PWD/*.so /usr/lib/
    ```
  </Step>
</Steps>

### 2. 从 AI Hub 下载模型

安装好 SDK 后,您可以从 [AI Hub](https://aihub.qualcomm.com/models?searchTerm=whisper)
下载已编译的 Whisper 模型。下载模型时,请选择以下设备:

| 开发板                  | AI Hub 设备                |
| -------------------- | ------------------------ |
| RB3 Gen 2 Vision Kit | Qualcomm QCS6490 (Proxy) |
| RUBIK Pi 3           | Qualcomm QCS6490 (Proxy) |
| IQ-9075 EVK          | Qualcomm QCS9075 (Proxy) |

下载后,将 encoder 模型重命名为 `encoder_model_htp.bin`,将 decoder 模型重命名为
`decoder_model_htp.bin`。

要直接在开发板上下载
[Whisper-Small-Quantized](https://aihub.qualcomm.com/models/whisper_small_quantized?searchTerm=whisper\&chipsets=qualcomm-qcs6490-proxy)
模型:

<Tabs>
  <Tab title="RB3 Gen 2 Vision Kit / Rubik Pi 3">
    ```shell theme={null}
    mkdir -p ~/whisper_models/ai_hub_small_quantized/
    cd ~/whisper_models/ai_hub_small_quantized/

    # Models from https://aihub.qualcomm.com/models/whisper_small_quantized for QCS6490 (Proxy) target
    wget -O encoder_model_htp.bin https://huggingface.co/qualcomm/Whisper-Small-Quantized/resolve/0e21411/precompiled/qualcomm-qcs6490-proxy/Whisper-Small-Quantized_WhisperSmallEncoderQuantizable_w8a16.bin
    wget -O decoder_model_htp.bin https://huggingface.co/qualcomm/Whisper-Small-Quantized/resolve/0e21411/precompiled/qualcomm-qcs6490-proxy/Whisper-Small-Quantized_WhisperSmallDecoderQuantizable_w8a16.bin

    # Vocab file is not in AI Hub yet, grab from our CDN
    wget -O vocab.bin https://cdn.edgeimpulse.com/qc-ai-docs/models/whisper/vocab.bin
    ```
  </Tab>

  <Tab title="IQ-9075 EVK">
    ```shell theme={null}
    mkdir -p ~/whisper_models/ai_hub_small_quantized/
    cd ~/whisper_models/ai_hub_small_quantized/

    # Models from https://aihub.qualcomm.com/models/whisper_small_quantized for QCS9075 (Proxy) target
    wget -O encoder_model_htp.bin https://huggingface.co/qualcomm/Whisper-Small-Quantized/resolve/0e21411/precompiled/qualcomm-qcs9075-proxy/Whisper-Small-Quantized_WhisperSmallEncoderQuantizable_w8a16.bin
    wget -O decoder_model_htp.bin https://huggingface.co/qualcomm/Whisper-Small-Quantized/resolve/0e21411/precompiled/qualcomm-qcs9075-proxy/Whisper-Small-Quantized_WhisperSmallDecoderQuantizable_w8a16.bin

    # Vocab file is not in AI Hub yet, grab from our CDN
    wget -O vocab.bin https://cdn.edgeimpulse.com/qc-ai-docs/models/whisper/vocab.bin
    ```
  </Tab>
</Tabs>

### 3. 编译并运行示例

<Steps>
  <Step title="构建 npu_rpc_linux_sample/voice-ai-ref 示例">
    ```shell theme={null}
    cd $VOICEAI_ROOT/whisper_sdk/sampleapp/npu_rpc_linux_sample/voice-ai-ref

    # Overwrite the LogUtil.h function to log to stdout
    wget -O include/LogUtil.h https://cdn.edgeimpulse.com/qc-ai-docs/code/voiceai_asr/2.3.0.0/whisper_sdk/sampleapp/npu_rpc_linux_sample/voice-ai-ref/include/LogUtil.h
    # Overwrite the main.cpp example to add microphone selection
    wget -O src/main.cpp https://cdn.edgeimpulse.com/qc-ai-docs/code/voiceai_asr/2.3.0.0/whisper_sdk/sampleapp/npu_rpc_linux_sample/voice-ai-ref/src/main.cpp

    # Symlink Whisper libraries for build
    mkdir -p libs/arm64-v8a/
    cd libs/arm64-v8a/
    ln -s $VOICEAI_ROOT/whisper_sdk/libs/npu/rpc_libraries/linux/whisper_all_quantized/*.so .
    cd ../../

    mkdir -p build
    cd build
    cmake ..
    make -j`nproc`
    ```
  </Step>

  <Step title="转写 .WAV 文件">
    您现在可以转写 .WAV 文件:

    ```shell theme={null}
    cd $VOICEAI_ROOT/whisper_sdk/sampleapp/npu_rpc_linux_sample/voice-ai-ref/build

    # Download sample file
    wget -O jfk.wav https://raw.githubusercontent.com/ggml-org/whisper.cpp/refs/heads/master/samples/jfk.wav

    # Transcribe:
    ./voice-ai-ref -f jfk.wav -l en -t transcribe -m ~/whisper_models/ai_hub_small_quantized/

    # ... Expected result:
    # VoiceAIRef final result =  And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country. [language: English]

    # Press Control-C to exit the running application.
    ```
  </Step>

  <Step title="或者进行实时转写">
    <Steps>
      <Step title="将麦克风连接到开发板">
        将麦克风连接到开发板。
      </Step>

      <Step title="查找您的麦克风名称">
        ```shell theme={null}
        pactl list short sources
        # 49	alsa_output.platform-sound.stereo-fallback.monitor	PipeWire	s24-32le 2ch 48000Hz	SUSPENDED
        # 76	alsa_input.usb-046d_C922_Pro_Stream_Webcam_C72F6EDF-02.analog-stereo	PipeWire	s16le 2ch 32000Hz	SUSPENDED

        # To use the USB webcam, use "alsa_input.usb-046d_C922_Pro_Stream_Webcam_C72F6EDF-02.analog-stereo" as the name
        ```
      </Step>

      <Step title="运行实时转写">
        ```shell theme={null}
        ./voice-ai-ref -r -l en -t transcribe -m ~/whisper_models/ai_hub_small_quantized/ -d "alsa_input.usb-046d_C922_Pro_Stream_Webcam_C72F6EDF-02.analog-stereo"

        # VoiceAIRef final result =  Hi, this is to see if I can do live transcription. [language: English]
        ```
      </Step>
    </Steps>
  </Step>
</Steps>

<Warning>在 VAD 判定当前无语音后,实时转写会立即报错退出,希望在未来更新中得到修复。</Warning>

🚀 您现在已经可以在开发板上完全离线地进行音频转写了!VoiceAI ASR 没有面向更高级
语言(如 Python)的绑定,因此如果您希望在应用中使用 Whisper,最简单的方法就是启动
`voice-ai-ref` 二进制文件,并从其 `stdout` 中读取数据。

## 使用 whisper.cpp 在 CPU 上运行 Whisper

或者,您也可以借助 whisper.cpp(或其他流行的 Whisper 库)在 CPU 上运行 Whisper(性能
稍差)。

下面是 [whisper.cpp](https://github.com/ggml-org/whisper.cpp) 的使用说明。在开发板上
打开终端,或通过 ssh 连接到开发板,然后运行:

<Steps>
  <Step title="安装构建依赖">
    ```shell theme={null}
    sudo apt update
    sudo apt install -y libsdl2-dev libsdl2-2.0-0 libasound2-dev
    ```
  </Step>

  <Step title="构建 whisper.cpp">
    ```shell theme={null}
    mkdir -p ~/dev/llm/
    cd ~/dev/llm/

    git clone https://github.com/ggml-org/whisper.cpp.git
    cd whisper.cpp
    git checkout v1.7.6

    # Build (CPU)
    cmake -B build-cpu -DWHISPER_SDL2=ON
    cmake --build build-cpu -j`nproc` --config Release
    ```
  </Step>

  <Step title="将 whisper.cpp 路径添加到 PATH">
    ```shell theme={null}
    cd ~/dev/llm/whisper.cpp/build-cpu/bin

    echo "" >> ~/.bash_profile
    echo "# Begin whisper.cpp" >> ~/.bash_profile
    echo "export PATH=\$PATH:$PWD" >> ~/.bash_profile
    echo "# End whisper.cpp" >> ~/.bash_profile
    echo "" >> ~/.bash_profile

    # To use the whisper.cpp files in your current session
    source ~/.bash_profile
    ```
  </Step>

  <Step title="使用 whisper.cpp 转写一段音频">
    ```shell theme={null}
    # Download model
    cd ~/dev/llm/whisper.cpp
    sh ./models/download-ggml-model.sh tiny.en-q5_1

    # Transcribe text
    whisper-cli -m models/ggml-tiny.en-q5_1.bin -f samples/jfk.wav

    # [00:00:00.000 --> 00:00:10.480]
    # and so my fellow Americans ask not what your country can do for you ask what you can do for your country
    ```
  </Step>

  <Step title="实时转写音频">
    <Steps>
      <Step title="将麦克风连接到开发板">
        将麦克风连接到开发板。
      </Step>

      <Step title="查找您的麦克风 ID">
        ```shell theme={null}
        SDL_AUDIODRIVER=alsa whisper-stream -m models/ggml-tiny.en-q5_1.bin
        # init: found 2 capture devices:
        # init:    - Capture device #0: 'qcm6490-rb3-vision-snd-card, '
        # init:    - Capture device #1: 'Yeti Stereo Microphone, USB Audio'

        # If you want "Yeti Stereo Microphone, USB Audio" then the ID is 1
        ```
      </Step>

      <Step title="开始实时转写">
        ```shell theme={null}
        SDL_AUDIODRIVER=alsa whisper-stream -m models/ggml-tiny.en-q5_1.bin -c 1

        # main: processing 48000 samples (step = 3.0 sec / len = 10.0 sec / keep = 0.2 sec), 4 threads, lang = en, task = transcribe, timestamps = 0 ...
        # main: n_new_line = 2, no_context = 1
        #
        # [Start speaking]
        # This is a test to see if you can transcribe text live on your Qualcomm device
        ```
      </Step>
    </Steps>
  </Step>
</Steps>

### 通过 OpenCL 在 GPU 上运行

您也可以构建在 GPU 上运行的二进制文件:

<Steps>
  <Step title="安装 OpenCL 头文件和 ICD 加载库">
    首先按 [llama.cpp](/zh/AI-Developer-Workflow-Ubuntu/topic/llama-cpp) 中"安装 OpenCL 头文件和 ICD 加载库"的步骤操作。
  </Step>

  <Step title="构建带 OpenCL 的二进制文件">
    ```shell theme={null}
    cd ~/dev/llm/whisper.cpp

    cmake -B build-gpu -DGGML_OPENCL=ON  -DWHISPER_SDL2=ON
    cmake --build build-gpu -j`nproc` --config Release

    # Find the binary in:
    #     build-gpu/bin/whisper-cli
    ```
  </Step>
</Steps>