> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Profile your AI model using QAIRT SDK

> Profile AI model performance on Qualcomm Dragonwing IoT platforms using QNN tools including qnn-net-run, qnn-throughput-net-run, and qnn-profile-viewer.

Profile your AI model using QAIRT SDK to measure the total runtime of model execution on a specified backend and optimize performance. Profiling provides detailed insights into  latency and hardware utilization during execution.

Enable additional profiling options to view execution times at different levels, such as per operation or per layer. Use this profiling information to identify bottlenecks and inefficiencies in graph execution, so you can optimize QNN runtime and reduce model latency.

## Prerequisites

* Set up the QAIRT SDK on your host computer.

  For detailed installation and configuration instructions, see [Set up Qualcomm AI Runtime SDK](../topic/qairt-setup).

* Select the model for profiling.

  You can either convert and quantize a custom model using QAIRT tools or generate a quantized
  model through AI Hub. For detailed guidance on compiling and optimizing models, see
  [Compile and optimize an AI model](../map/compile-and-optimize-model).

  The following instructions use the Inception V3 model from AI Hub.

* Enable Wi-Fi and SSH on the device.

  The device requires an internet connection to download the artifacts needed to run sample applications. If SSH and Wi-Fi are already configured, skip this step.

  Follow [Setup an SSH connection](https://dragonwingdocs.qualcomm.com/Technologies/Ethernet/get-started-with-ethernet#set-up-an-ssh-connection) to enable Wi-Fi and SSH on the device.

* Ensure that you have installed the following QNN tools on the target device as part of the build.

  * qnn-net-run
  * qnn-throughput-net-run
  * qnn-context-binary-generator
  * qnn-profile-viewer

## Profiling levels on HTP

The following table provides the profiling levels, their description, and configuration:

| Profiling levels | Description                                                                                                                                                                      | Configuration                                                                                                              |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Basic            | • Total model execution time in microseconds<br />• Use for latency measurements                                                                                                 | `--profiling_level=basic` with `qnn-net-run`                                                                               |
| Detailed         | Provides basic information along with the per-operation execution time in cycles.                                                                                                | `--profiling_level=detailed` with `qnn-net-run`                                                                            |
| Lint             | • Provides per-op cycle count on main thread and background execution information.<br />• Enables chrometrace for deeper analysis.                                               | `--profiling_level=backend` with `qnn-net-run` and `--profiling_level=linting` inside `backend_extension_config.json` file |
| Opttrace         | • Provides extremely detailed operation-level HTP execution status.<br />• Provides HVX/HMX utilization and VTCM usage.<br />• Use for in-depth performance bottleneck analysis. | `--profiling_level=detailed` and `--profiling_options=optrace` with `qnn-net-run`                                          |

The following figure shows the primary execution profiling events for HTP and how these events are measured during inference:

<img src="https://mintcdn.com/qualcomm-prod/9xxk39ukdMdHcCv-/AI-Developer-Workflow-Ubuntu/_images/htp-basic-profiling-events.png?fit=max&auto=format&n=9xxk39ukdMdHcCv-&q=85&s=36f0c3ee55a443eff2ec3e00756105f1" alt="HTP basic profiling events diagram" width="964" height="758" data-path="AI-Developer-Workflow-Ubuntu/_images/htp-basic-profiling-events.png" />

**Figure: HTP basic profiling events**

## Perform Lint profile with `qnn-net-run`

Lint profiling provides detailed per-operation cycle counts on the main thread along with background execution information. The following steps perform lint profiling on the Inception-v3 AI Hub model. Follow these steps and replace the model with your custom model.

<Steps>
  <Step title="SSH into your target device">
    ```shell theme={null}
    ssh ubuntu@<IP ADDRESS OF THE TARGET DEVICE>
    ```

    When prompted, enter the password.
  </Step>

  <Step title="Download the quantized (w8a8) inception_v3 dlc model from AI Hub on the target device for profiling">
    ```shell theme={null}
    curl -L https://huggingface.co/qualcomm/Inception-v3/resolve/v0.42.0/Inception-v3_w8a8.dlc -o /etc/models/inception_v3_quantized.dlc 
    ```
  </Step>

  <Step title="Profile the model using generated input files">
    For demonstration purposes, you can profile the model using generated input files.

    Generate these input files using the following Python script tailored for the `inception_v3_quantized.dlc` model.

    <Steps>
      <Step title="Save the following script as generate_random_input.py in the /etc/models directory">
        ```python theme={null}
        import os
        import numpy as np

        input_path_list =[]
        BASE_PATH = "/tmp/RandomInputsForInceptionV3Profiling/"

        if not os.path.exists(BASE_PATH):
           os.mkdir(BASE_PATH)

        # generate 10 random inputs and save as raw
        NUM_IMAGES = 10

        #binary files
        for img in range(NUM_IMAGES):
           filename = "input_{}.raw".format(img)
           randomTensor = np.random.random((1, 224, 224, 3)).astype(np.float32)
           filename = os.path.join(BASE_PATH, filename)
           randomTensor.tofile(filename)
           input_path_list.append(filename)

        #for saving as input_list text
              with open("input_list_profiling.txt", "w") as f:
                 for path in input_path_list:
                    f.write(path)
                    f.write("\n")
        ```

        This script generates 10 sample input files saved in the `/tmp/RandomInputsForInceptionV3Profiling/`
        directory and an `input_list_profiling.txt` file that contains the path to each sample generated.
      </Step>

      <Step title="Run the script on the target device">
        ```shell theme={null}
        python3 /etc/models/generate_random_input.py
        ```
      </Step>
    </Steps>
  </Step>

  <Step title="Create the backend_extension_config_file.json and htp_config.json files">
    Create the `backend_extension_config_file.json` and `htp_config.json` files in the `/etc/models` directory
    of the target device to profile the model using the HTP runtime.

    * `backend_extension_config_file.json`

      ```json theme={null}
      {
         "backend_extensions": {
            "shared_library_path" : "libQnnHtpNetRunExtensions.so",
            "config_file_path" : "./htp_config.json"
         }
      }
      ```

    * `htp_config.json`

      ```json theme={null}
      {
      "graphs": [
            {
                  "vtcm_mb": 2,
                  "fp16_relaxed_precision": 0,
                  "graph_names": [
                     "graph_name_1"
                  ],
                  "O": 3.0
            }
         ],
         "devices": [
            {
                  "dsp_arch": "v68",
                  "profiling_level": "linting",
                  "cores": [
                     {
                        "perf_profile": "burst"
                     }
                  ]
            }
         ]
      }
      ```

          <Note>
            * Use `"dsp_arch": "v68"` for Qualcomm Dragonwing™ RB3 Gen 2
            * Use `"dsp_arch": "v75"` for Dragonwing IQ-8275
            * Use `"dsp_arch": "v73"` for Dragonwing IQ-9075
          </Note>
  </Step>

  <Step title="Go to the /etc/models directory and run the qnn-net-run command on the target device">
    ```shell theme={null}
    qnn-net-run --model libQnnModelDlc.so \
                --backend libQnnHtp.so \
                --input_list input_list_profiling.txt \
                --config_file backend_extension_config_file.json \
                --output_dir output_htp \
                --profiling_level backend \
                --dlc_path /etc/models/inception_v3_quantized.dlc
    ```
  </Step>

  <Step title="Enable lint profiling by specifying --profiling_level=backend">
    This step ensures that the profiling level defined in the backend-specific configuration file is applied.

    The `execution_metadata.yaml` and `qnn-profiling-data_0.log` files should be created in the `/etc/models/output_htp` directory.

    To view logs from the qnn-profiling-data\_0.log file, use qnn-profile-viewer.

    <img src="https://mintcdn.com/qualcomm-prod/9xxk39ukdMdHcCv-/AI-Developer-Workflow-Ubuntu/_images/lint-profling.png?fit=max&auto=format&n=9xxk39ukdMdHcCv-&q=85&s=400a50c09280c9064de51bbac8c695f8" alt="Lint profiling output files in the output directory" width="1430" height="733" data-path="AI-Developer-Workflow-Ubuntu/_images/lint-profling.png" />
  </Step>
</Steps>

## View lint profiling logs using qnn-profile-viewer

View the profile outputs generated at the backend profiling level by using the qnn-profile-viewer tool with the following plugins:

<Tabs>
  <Tab title="libQnnHtpProfilingReader.so">
    To retrieve linting information from an inference, run qnn-profile-viewer with the `libQnnHtpProfilingReader.so` plugin. This plugin provides raw output of every single run.

    ```shell theme={null}
    qnn-profile-viewer --reader libQnnHtpProfilingReader.so --input_log /etc/models/output_htp/qnn-profiling-data_0.log --output /etc/models/output_htp/profile_htp.csv
    ```

    The following is the sample output:

    <img src="https://mintcdn.com/qualcomm-prod/9xxk39ukdMdHcCv-/AI-Developer-Workflow-Ubuntu/_images/lint-profling-output.png?fit=max&auto=format&n=9xxk39ukdMdHcCv-&q=85&s=29c7364cb7d380f627e33fa29100ae37" width="623" height="323" data-path="AI-Developer-Workflow-Ubuntu/_images/lint-profling-output.png" />

    **Figure: Sample output of Lint profiling with libQnnHtpProfilingReader.so**

    In the linting profiling report, each operation has:

    * **Cycle count**: the time spent executing on the main thread.
    * **Wait entry**: the cycles spent waiting before execution starts.
    * **Overlap**: the cycles spent on at least one background operation while the main thread executes the current operation.
    * **Overlap (wait)**: the cycles spent on at least one background operation during the main thread’s wait period.

    <Note>
      Every operation on the main thread has a wait period before its executed, which only begins once the previous operation has ended. This delay may be caused by scheduling issues or by waiting for background activities like HVX or DMA to finish.
    </Note>
  </Tab>

  <Tab title="libQnnChrometraceProfilingReader.so">
    The `libQnnChrometraceProfilingReader.so` plugin provides average output of all the runs.
    Additionally, if an output file is specified with the `--output` option, generates a file
    containing the profiling data in chrometrace format.

    ```
    qnn-profile-viewer --reader libQnnChrometraceProfilingReader.so --input_log /etc/models/output_htp/qnn-profiling-data_0.log --output /etc/models/output_htp/chromeTrace.json
    ```

    The following figures shows the sample output:

    <img src="https://mintcdn.com/qualcomm-prod/9xxk39ukdMdHcCv-/AI-Developer-Workflow-Ubuntu/_images/lint-libQnnChrometraceProfilingReader.png?fit=max&auto=format&n=9xxk39ukdMdHcCv-&q=85&s=6ff0b06ed1e786f4633dcaaf3ff657c0" width="1428" height="581" data-path="AI-Developer-Workflow-Ubuntu/_images/lint-libQnnChrometraceProfilingReader.png" />

    **Figure: Sample output of Lint profiling with libQnnChrometraceProfilingReader.so**

    <Note>
      To view the chrome trace, use Google Chrome.
    </Note>
  </Tab>
</Tabs>

## Perform advanced profiling with QNN HTP Optrace

Use QNN optrace profiling to understand detailed internal operations of QNN HTP hardware blocks. This capability helps you:

* Identify problematic operations that may not be parallelized well.
* See how operations are scheduled throughout execution.
* Observe the interaction between various operators.
* Evaluate how efficiently HVX parallelism works for each operation.

To understand more about QNN HTP optrace profiling, see [QNN HTP Optrace Profiling](https://docs.qualcomm.com/nav/home/htp_backend.html?product=1601111740009302#qnn-htp-optrace-profiling).

## Perform profiling with `qnn-throughput-net-run`

Use qnn-throughput-net-run for multi-threaded execution across one or more QNN backends. This profiling supports multi-threaded execution and lets you run models repeatedly for a specified duration or a set number of iterations. Use this profiling for scenarios where you need concurrent or repeated execution of multiple models for performance benchmarking.

<Steps>
  <Step title="SSH into your target device">
    ```shell theme={null}
    ssh ubuntu@<IP ADDRESS OF THE TARGET DEVICE>
    ```

    When prompted, enter the password.
  </Step>

  <Step title="On the target device, create a working directory">
    ```shell theme={null}
    mkdir -p /etc/models
    ```
  </Step>

  <Step title="On the target device, download the quantized (w8a8) inception_v3 dlc model from AI Hub">
    ```shell theme={null}
    curl -L https://huggingface.co/qualcomm/Inception-v3/resolve/v0.42.0/Inception-v3_w8a8.dlc -o /etc/models/inception_v3_quantized.dlc
    ```
  </Step>

  <Step title="On the target device, create the backend_extension_config.json and htp_config.json files">
    Create the files in the `/etc/models` directory.

    These files are required to generate the context binary in the next step.

    * **backend\_extension\_config.json**

      ```json theme={null}
      {
         "backend_extensions": {
            "shared_library_path": "libQnnHtpNetRunExtensions.so",
            "config_file_path": "./htp_config.json"
         }
      }
      ```

    * **htp\_config.json**

      ```json theme={null}
      {
      "graphs": [
            {
                  "vtcm_mb": 2,
                  "fp16_relaxed_precision": 0,
                  "graph_names": [
                     "graph_name_2"
                  ],
                  "O": 3.0
            }
         ],
         "devices": [
            {
                  "dsp_arch": "v68",
                  "profiling_level": "linting",
                  "cores": [
                     {
                        "perf_profile": "burst"
                     }
                  ]
            }
         ]
      }
      ```

    <Note>
      - Use `"dsp_arch": "v68"` for Qualcomm Dragonwing™ RB3 Gen 2.
      - Use `"dsp_arch": "v75"` for Dragonwing IQ-8275.
      - Use `"dsp_arch": "v73"` for Dragonwing IQ-9075.
    </Note>
  </Step>

  <Step title="On the target device, generate the context binary (.bin file) using the qnn-context-binary-generator tool">
    ```shell theme={null}
    qnn-context-binary-generator --log_level=info --backend libQnnHtp.so --model libQnnModelDlc.so --config_file /etc/models/backend_extension_config.json --output_dir context_bin_dir --dlc_path /etc/models/inception_v3_quantized.dlc --binary_file inception_v3
    ```

    The `qnn-throughput-net-run` command will ingest the generated context binary.
  </Step>

  <Step title="To profile the model using qnn-throughput-net-run, create the qtnr_config.json and htp_backend.json files">
    Create these files in the `/etc/models/` directory on the target device.

    * `htp_backend.json`:

      ```json theme={null}
      {
         "devices": [
            {
            "dsp_arch": "v68",
            "device_id" : 0
            }
         ]
      }
      ```
    * `qtnr_config.json`:

      ```json theme={null}
      {
      "backends": [
         {
         "backendName": "htp_backend",
         "backendPath": "libQnnHtp.so",
         "profilingLevel": "BASIC",
         "backendExtensions": "libQnnHtpNetRunExtensions.so",
         "perfProfile": "burst"
         }
      ],
      "models": [
         {
         "modelName": "inception_v3",
         "modelPath": "/etc/models/context_bin_dir/inception_v3.bin",
         "loadFromCachedBinary": true,
         "outputPath": "output_original"
         }
      ],
      "contexts": [
         {
         "contextName": "htp_context_1",
         "priority": "HIGH"
         }
      ],
      "testCase": {
         "iteration": 1,
         "logLevel": "info",
         "threads": [
            {
               "threadName": "htp_thread_1",
               "backend": "htp_backend",
               "context": "htp_context_1",
               "model": "inception_v3",
               "interval": 0,
               "loopUnit": "second",
               "loop": 10,
               "backendConfig": "htp_backend.json"
            }
         ]
         }
      }
      ```

    <Note>
      - Use  `"dsp_arch"`: `"v68"` for Qualcomm Dragonwing™ RB3 Gen 2
      - Use `"dsp_arch"`: `"v75"` for Dragonwing IQ-8275
      - Use `"dsp_arch"`: `"v73"` for Dragonwing IQ-9075
    </Note>
  </Step>

  <Step title="To perform profiling, on the target device, run the following commands">
    ```shell theme={null}
    cd /etc/models
    ```

    ```shell theme={null}
    qnn-throughput-net-run --config /etc/models/qtnr_config.json --output /etc/models/output_qtnr.json
    ```

    The profiling information is generated in the `/etc/models` directory.

    <img src="https://mintcdn.com/qualcomm-prod/PpKFrrqBJ7iBLtKw/AI-Developer-Workflow-Ubuntu/_images/qnn-throughput-net-run-profiling-output.png?fit=max&auto=format&n=PpKFrrqBJ7iBLtKw&q=85&s=f0c76ce6ed66472b449f6e0c7b25b702" alt="Sample output of qnn-throughput-net-run profiling" width="624" height="96" data-path="AI-Developer-Workflow-Ubuntu/_images/qnn-throughput-net-run-profiling-output.png" />

    **Figure: Sample output of qnn-throughput-net-run profiling**
  </Step>
</Steps>