> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Whisper

> Run OpenAI's Whisper speech recognition model on Dragonwing devices using the NPU (VoiceAI ASR) or CPU/GPU (whisper.cpp).

Whisper is OpenAI's general-purpose automatic speech recognition (ASR) model. You can use it for audio transcription, translation, and language identification. You can run Whisper on the NPU of your Dragonwing development board using Qualcomm's [VoiceAI ASR](https://softwarecenter.qualcomm.com/catalog/item/VoiceAI_ASR_Community), or on the CPU using [whisper.cpp](https://github.com/ggml-org/whisper.cpp).

<Note>
  Whisper model support on NPU is in progress via AIHUB and will be updated soon.
</Note>

## Running Whisper on the NPU with VoiceAI ASR

### 1. Installing SDKs

<Steps>
  <Step title="Set up the base requirements">
    Open a terminal on your development board, and set up the base requirements for this example:

    ```shell theme={null}
    sudo apt install -y cmake pulseaudio-utils
    ```
  </Step>

  <Step title="Install VoiceAI ASR - Community Edition">
    Install [VoiceAI ASR - Community Edition](https://softwarecenter.qualcomm.com/catalog/item/VoiceAI_ASR_Community):

    ```shell theme={null}
    cd ~/

    wget https://softwarecenter.qualcomm.com/api/download/software/sdks/VoiceAI_ASR_Community/All/2.3.0.0/VoiceAI_ASR_Community_v2.3.0.0.zip
    unzip VoiceAI_ASR_Community_v2.3.0.0.zip -d voiceai_asr
    rm VoiceAI_ASR_Community_v2.3.0.0.zip

    cd voiceai_asr/2.3.0.0/

    # Put the path to VoiceAI ASR in your bash_profile (so it's available under VOICEAI_ROOT)
    echo "" >> ~/.bash_profile
    echo "# Begin VoiceAI ASR" >> ~/.bash_profile
    echo "export VOICEAI_ROOT=$PWD" >> ~/.bash_profile
    echo "# End VoiceAI ASR" >> ~/.bash_profile
    echo "" >> ~/.bash_profile

    # Re-load the environment variables
    source ~/.bash_profile

    # Symlink Whisper libraries
    cd $VOICEAI_ROOT/whisper_sdk/libs/npu/rpc_libraries/linux/whisper_all_quantized/
    sudo ln -s $PWD/*.so /usr/lib/
    ```
  </Step>
</Steps>

### 2. Download models from AI Hub

With the SDKs installed, you can download precompiled Whisper models from [AI Hub](https://aihub.qualcomm.com/models?searchTerm=whisper). When downloading a model, select the following device:

| Development board    | AI Hub device            |
| -------------------- | ------------------------ |
| RB3 Gen 2 Vision Kit | Qualcomm QCS6490 (Proxy) |
| RUBIK Pi 3           | Qualcomm QCS6490 (Proxy) |
| IQ-9075 EVK          | Qualcomm QCS9075 (Proxy) |

After downloading, rename the encoder model to `encoder_model_htp.bin` and the decoder model to `decoder_model_htp.bin`.

To download the [Whisper-Small-Quantized](https://aihub.qualcomm.com/models/whisper_small_quantized?searchTerm=whisper\&chipsets=qualcomm-qcs6490-proxy) model directly on your development board:

<Tabs>
  <Tab title="RB3 Gen 2 Vision Kit / Rubik Pi 3">
    ```shell theme={null}
    mkdir -p ~/whisper_models/ai_hub_small_quantized/
    cd ~/whisper_models/ai_hub_small_quantized/

    # Models from https://aihub.qualcomm.com/models/whisper_small_quantized for QCS6490 (Proxy) target
    wget -O encoder_model_htp.bin https://huggingface.co/qualcomm/Whisper-Small-Quantized/resolve/0e21411/precompiled/qualcomm-qcs6490-proxy/Whisper-Small-Quantized_WhisperSmallEncoderQuantizable_w8a16.bin
    wget -O decoder_model_htp.bin https://huggingface.co/qualcomm/Whisper-Small-Quantized/resolve/0e21411/precompiled/qualcomm-qcs6490-proxy/Whisper-Small-Quantized_WhisperSmallDecoderQuantizable_w8a16.bin

    # Vocab file is not in AI Hub yet, grab from our CDN
    wget -O vocab.bin https://cdn.edgeimpulse.com/qc-ai-docs/models/whisper/vocab.bin
    ```
  </Tab>

  <Tab title="IQ-9075 EVK">
    ```shell theme={null}
    mkdir -p ~/whisper_models/ai_hub_small_quantized/
    cd ~/whisper_models/ai_hub_small_quantized/

    # Models from https://aihub.qualcomm.com/models/whisper_small_quantized for QCS9075 (Proxy) target
    wget -O encoder_model_htp.bin https://huggingface.co/qualcomm/Whisper-Small-Quantized/resolve/0e21411/precompiled/qualcomm-qcs9075-proxy/Whisper-Small-Quantized_WhisperSmallEncoderQuantizable_w8a16.bin
    wget -O decoder_model_htp.bin https://huggingface.co/qualcomm/Whisper-Small-Quantized/resolve/0e21411/precompiled/qualcomm-qcs9075-proxy/Whisper-Small-Quantized_WhisperSmallDecoderQuantizable_w8a16.bin

    # Vocab file is not in AI Hub yet, grab from our CDN
    wget -O vocab.bin https://cdn.edgeimpulse.com/qc-ai-docs/models/whisper/vocab.bin
    ```
  </Tab>
</Tabs>

### 3. Compiling and running examples

<Steps>
  <Step title="Build the npu_rpc_linux_sample/voice-ai-ref example">
    ```shell theme={null}
    cd $VOICEAI_ROOT/whisper_sdk/sampleapp/npu_rpc_linux_sample/voice-ai-ref

    # Overwrite the LogUtil.h function to log to stdout
    wget -O include/LogUtil.h https://cdn.edgeimpulse.com/qc-ai-docs/code/voiceai_asr/2.3.0.0/whisper_sdk/sampleapp/npu_rpc_linux_sample/voice-ai-ref/include/LogUtil.h
    # Overwrite the main.cpp example to add microphone selection
    wget -O src/main.cpp https://cdn.edgeimpulse.com/qc-ai-docs/code/voiceai_asr/2.3.0.0/whisper_sdk/sampleapp/npu_rpc_linux_sample/voice-ai-ref/src/main.cpp

    # Symlink Whisper libraries for build
    mkdir -p libs/arm64-v8a/
    cd libs/arm64-v8a/
    ln -s $VOICEAI_ROOT/whisper_sdk/libs/npu/rpc_libraries/linux/whisper_all_quantized/*.so .
    cd ../../

    mkdir -p build
    cd build
    cmake ..
    make -j`nproc`
    ```
  </Step>

  <Step title="Transcribe .WAV files">
    You can now transcribe .WAV files:

    ```shell theme={null}
    cd $VOICEAI_ROOT/whisper_sdk/sampleapp/npu_rpc_linux_sample/voice-ai-ref/build

    # Download sample file
    wget -O jfk.wav https://raw.githubusercontent.com/ggml-org/whisper.cpp/refs/heads/master/samples/jfk.wav

    # Transcribe:
    ./voice-ai-ref -f jfk.wav -l en -t transcribe -m ~/whisper_models/ai_hub_small_quantized/

    # ... Expected result:
    # VoiceAIRef final result =  And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country. [language: English]

    # Press Control-C to exit the running application.
    ```
  </Step>

  <Step title="Or even do live transcription">
    <Steps>
      <Step title="Connect a microphone to your development board">
        Connect a microphone to your development board.
      </Step>

      <Step title="Find the name of your microphone">
        ```shell theme={null}
        pactl list short sources
        # 49	alsa_output.platform-sound.stereo-fallback.monitor	PipeWire	s24-32le 2ch 48000Hz	SUSPENDED
        # 76	alsa_input.usb-046d_C922_Pro_Stream_Webcam_C72F6EDF-02.analog-stereo	PipeWire	s16le 2ch 32000Hz	SUSPENDED

        # To use the USB webcam, use "alsa_input.usb-046d_C922_Pro_Stream_Webcam_C72F6EDF-02.analog-stereo" as the name
        ```
      </Step>

      <Step title="Run live transcription">
        ```shell theme={null}
        ./voice-ai-ref -r -l en -t transcribe -m ~/whisper_models/ai_hub_small_quantized/ -d "alsa_input.usb-046d_C922_Pro_Stream_Webcam_C72F6EDF-02.analog-stereo"

        # VoiceAIRef final result =  Hi, this is to see if I can do live transcription. [language: English]
        ```
      </Step>
    </Steps>
  </Step>
</Steps>

<Warning>Live transcription errors out immediately after the VAD determines that there is no speech, hopefully this will be fixed in a future update.</Warning>

🚀 You now have fully offline transcription of audio on your development board! VoiceAI ASR does not have bindings to higher level languages (like Python), so if you want to use Whisper in your application it's easiest to just spawn the `voice-ai-ref` binary, and read data from `stdout`.

## Running Whisper on the CPU with whisper.cpp

Alternatively you can run Whisper on the CPU (with less performance) using whisper.cpp (or any of the other popular Whisper libraries).

Here's instructions for [whisper.cpp](https://github.com/ggml-org/whisper.cpp). Open the terminal on your development board, or an ssh session to your development board, and run:

<Steps>
  <Step title="Install build dependencies">
    ```shell theme={null}
    sudo apt update
    sudo apt install -y libsdl2-dev libsdl2-2.0-0 libasound2-dev
    ```
  </Step>

  <Step title="Build whisper.cpp">
    ```shell theme={null}
    mkdir -p ~/dev/llm/
    cd ~/dev/llm/

    git clone https://github.com/ggml-org/whisper.cpp.git
    cd whisper.cpp
    git checkout v1.7.6

    # Build (CPU)
    cmake -B build-cpu -DWHISPER_SDL2=ON
    cmake --build build-cpu -j`nproc` --config Release
    ```
  </Step>

  <Step title="Add the whisper.cpp paths to your PATH">
    ```shell theme={null}
    cd ~/dev/llm/whisper.cpp/build-cpu/bin

    echo "" >> ~/.bash_profile
    echo "# Begin whisper.cpp" >> ~/.bash_profile
    echo "export PATH=\$PATH:$PWD" >> ~/.bash_profile
    echo "# End whisper.cpp" >> ~/.bash_profile
    echo "" >> ~/.bash_profile

    # To use the whisper.cpp files in your current session
    source ~/.bash_profile
    ```
  </Step>

  <Step title="Transcribe some audio using whisper.cpp">
    ```shell theme={null}
    # Download model
    cd ~/dev/llm/whisper.cpp
    sh ./models/download-ggml-model.sh tiny.en-q5_1

    # Transcribe text
    whisper-cli -m models/ggml-tiny.en-q5_1.bin -f samples/jfk.wav

    # [00:00:00.000 --> 00:00:10.480]
    # and so my fellow Americans ask not what your country can do for you ask what you can do for your country
    ```
  </Step>

  <Step title="Live transcribe audio">
    <Steps>
      <Step title="Connect a microphone to your development board">
        Connect a microphone to your development board.
      </Step>

      <Step title="Find your microphone ID">
        ```shell theme={null}
        SDL_AUDIODRIVER=alsa whisper-stream -m models/ggml-tiny.en-q5_1.bin
        # init: found 2 capture devices:
        # init:    - Capture device #0: 'qcm6490-rb3-vision-snd-card, '
        # init:    - Capture device #1: 'Yeti Stereo Microphone, USB Audio'

        # If you want "Yeti Stereo Microphone, USB Audio" then the ID is 1
        ```
      </Step>

      <Step title="Start live transcribing">
        ```shell theme={null}
        SDL_AUDIODRIVER=alsa whisper-stream -m models/ggml-tiny.en-q5_1.bin -c 1

        # main: processing 48000 samples (step = 3.0 sec / len = 10.0 sec / keep = 0.2 sec), 4 threads, lang = en, task = transcribe, timestamps = 0 ...
        # main: n_new_line = 2, no_context = 1
        #
        # [Start speaking]
        # This is a test to see if you can transcribe text live on your Qualcomm device
        ```
      </Step>
    </Steps>
  </Step>
</Steps>

### Running on the GPU with OpenCL

You can also build binaries that run on the GPU:

<Steps>
  <Step title="Install the OpenCL headers and ICD loader library">
    First follow the steps in [llama.cpp](../topic/llama-cpp) under "Install the OpenCL headers and ICD loader library".
  </Step>

  <Step title="Build a binary with OpenCL">
    ```shell theme={null}
    cd ~/dev/llm/whisper.cpp

    cmake -B build-gpu -DGGML_OPENCL=ON  -DWHISPER_SDL2=ON
    cmake --build build-gpu -j`nproc` --config Release

    # Find the binary in:
    #     build-gpu/bin/whisper-cli
    ```
  </Step>
</Steps>