> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Prepare a GenAI model using AI Hub

> Use Qualcomm AI Hub to prepare and deploy large language models (LLMs) on Qualcomm Dragonwing IoT platforms using the Genie framework.

Qualcomm AI Hub provides a streamlined workflow to prepare and deploy
large language models (LLMs) on Qualcomm Dragonwing™ products using
Qualcomm GenerativeAI Inference Extensions (Genie).

This approach enables efficient on-device execution of generative AI
models by leveraging the neural processing unit (NPU) and optimized binaries.

The following image shows the high-level GenAI model workflow from preparation to
execution.

<img src="https://mintcdn.com/qualcomm-prod/Xo6U_rj4CrsoQjNT/AI-Developer-Workflow-Ubuntu/_images/genai-prepare-ai-hub.png?fit=max&auto=format&n=Xo6U_rj4CrsoQjNT&q=85&s=85c1548722c304e594709001bc5f179f" alt="High-level GenAI model workflow from preparation to execution using AI Hub" width="2929" height="411" data-path="AI-Developer-Workflow-Ubuntu/_images/genai-prepare-ai-hub.png" />

The following are the steps to create LLM model binaries using AI Hub:

* [Prerequisites](https://github.com/qualcomm/ai-hub-apps/tree/main/tutorials/llm_on_genie#requirements)

* [Detailed instructions](https://github.com/qualcomm/ai-hub-apps/tree/main/tutorials/llm_on_genie#step-2-prepare-a-genie-bundle-on-the-host-machine)

The following is an overview of LLM on-device deployment:

<Steps>
  <Step title="Prepare the model">
    <Steps>
      <Step title="Start with the desired LLM">
        Start with the desired LLM (for example, Llama 3.x series) from Hugging Face or another source.
      </Step>

      <Step title="Export the model with the qai_hub_models Python package">
        Use the `qai_hub_models` Python package to export the model.

        This process:

        * Downloads the model weights.<br />
        * Uploads them to AI Hub for compilation.<br />
        * Generates QNN binaries split into multiple parts for NPU execution.<br />
        * Creates a deployable folder (`genie_bundle`) with all required assets (context binaries, configs, tokenizer).

        <Note>
          AI Hub compiles models into optimized binaries for [Qualcomm AI Runtime (QAIRT) SDK](https://docs.qualcomm.com/doc/80-63442-10/):

          1. AI Hub supports quantization (typically 4-bit internally, though weights may be stored as 8-bit for compatibility).
          2. Export scripts handle splitting large models into prompt processors and token generator components.
        </Note>
      </Step>
    </Steps>
  </Step>

  <Step title="Deploy the model">
    The following are high-level steps of the deployment process. For detailed instructions
    and commands, see [Run LLMs with Genie](https://github.com/qualcomm/ai-hub-apps/tree/main/tutorials/llm_on_genie#step-3-run-the-llm-on-device).

    <Steps>
      <Step title="Install the Qualcomm AI Runtime (QAIRT) SDK on the target device">
        Install the Qualcomm AI Runtime (QAIRT) SDK on the target device (Android, Windows, Linux).
      </Step>

      <Step title="Copy the compiled binaries and configuration files to the device">
        Copy the compiled binaries and configuration files to the device.
      </Step>

      <Step title="Use Genie CLI tools or the Genie dialog API for inference">
        Use [Qualcomm GenerativeAI Inference Extensions (Genie) CLI tools](https://docs.qualcomm.com/doc/80-63442-10/topic/tools_tools.html)
        (for example, `genie-t2t-run`) or [Genie dialog API](https://docs.qualcomm.com/doc/80-63442-10/topic/api-rst_file_include_Genie_GenieDialog_h.html#file-include-Genie-GenieDialog.h) for inference.
      </Step>

      <Step title="Ensure the target device meets the requirements">
        The steps in this section are validated for QCS9100, which uses Hexagon architecture V73.

        * Hexagon architecture: v73 or newer
        * Required RAM:
          * 16 GB for 7B models
          * \~12 GB for 3B models
      </Step>
    </Steps>
  </Step>

  <Step title="Run the model on-device using Genie APIs">
    Run the model on-device using Genie APIs integrated with [Qualcomm AI Engine Direct](https://docs.qualcomm.com/doc/80-63442-10/topic/index_QNN.html).

    * Genie manages multiple binaries and execution orders for optimal NPU utilization.
  </Step>
</Steps>

### Important notes

* AI Hub advantages

  * automatically handles model compilation, quantization, and splitting.
  * Provides pre-optimized models and bring your own model (BYOM) support.

* Genie

  * Simplifies inference by abstracting complex execution steps.
  * Offers APIs for text-to-text and dialogue-based interactions.

* Customization

  * Export flow defaults to 4-bit quantization for runtime efficiency.
  * No direct option to store weights as 4-bit; they remain 8-bit but load as 4-bit during execution.