> ## Documentation Index
> Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshoot kernel issues

To troubleshoot kernel issues, use the following methods:

## **Boot failure due to incorrect or no DTB picked**

When troubleshooting boot failure due to DTB load issues, verify authentication and DTB availability, and follow the steps to fix the problem.

The following scenarios lead to boot failure:

* **Authentication failure**
  The following example shows the logs for failure due to incorrect authentication:
  ```text theme={null}
  Platform Subtype : 0
  DtPlatformLoadDtb qcs6490-rb3gen2.dtb is loaded
  Platform Subtype : -2090817768
  DtPlatformLoadSign qcs6490-rb3gen2.sgn is loaded
  failed to authenticate image !
  ```
* **Failure due to unavailability of DTB**
  The following example shows the logs for failure due to the missing DTB:
  ```text theme={null}
  DtPlatformLoadDtb qcs6490-rb3gen2.dtb is loading failed with Status = E
  DtPlatformDxeEntryPoint: no DTB blob could be loaded, defaulting to ACPI (Status == Not Found)
  ```
  To fix the unavailability of DTB issues, do the following:
  * Verify if the corresponding DTB file is a part of a packaged `dtb.bin` file.
  * Mount the `dtb.bin` file locally on the host development machine to verify the following:
    ```text theme={null}
    mount -t vfat dtb.bin /mnt/
    fdtdump /mnt/combined-dtb.bin | grep -i model
    ```
* **To verify loaded DTB from the kernel log**
  Use the following command to verify DTB loaded during boot from the kernel log:
  ```text theme={null}
  dmesg | grep -i model
  ```

## **Serial console not working**

To troubleshoot serial console log failure, enable specific drivers in the kernel configuration and add relevant parameters to the kernel boot arguments.

* Enable the following driver in the kernel configuration file:
  * `CONFIG_SERIAL_QCOM_GENI=y`
  * `CONFIG_SERIAL_QCOM_GENI_CONSOLE=y`
* Add the following parameter to the kernel boot arguments:
  ```text theme={null}
  console=ttyMSM0,115200n8
  ```
* To get early boot messages from the kernel, add the following parameter to the kernel boot arguments:
  ```text theme={null}
  earlycon
  ```

## **Debug remote processor failure**

To troubleshoot remoteproc failure, capture kernel logs, use matching firmware signature files, and verify firmware locations in the device.

* Ensure that all the matching subsystem images are flashed without any errors.
* Verify if there are any errors in the kernel log.
  The following example shows a sample kernel log:
  ```text theme={null}
  0x000000000A27652C |   5198.790423:   qcom_q6v5_pas 3000000.remoteproc: fatal error received: err_inject_crash.c:413:Crash injected via Diag
  0x000000000A276689 |   5198.801061:   remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error
  0x000000000A2767A1 |   5198.809602:   remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc
  0x000000000A27688E |   5198.816837:   remoteproc remoteproc2: recovering 3000000.remoteproc
  0x000000000A276971 |   5198.823784:   qcom_q6v5_pas 8a00000.remoteproc: subsystem event rejected
  ```
* Disable the following subsystem restart feature to ensure that the remoteproc crash signature is visible in the kernel log:
  ```text theme={null}
  echo disabled > /sys/kernel/debug/remoteproc/remoteprocN/recovery
  ```
* Confirm if all the necessary firmware files are present in the `/lib/firmware/qcom/<SoC>` in rootfs file system.

## **Debug configurations or symbols not reflecting in images**

To debug kernel configuration change issues where configurations or symbols are not reflecting in the images, follow these steps:

1. Add the debug configuration driver to the `kernel/configs/debug.config` file.
2. Export the `DEBUG_BUILD=1` macro before running the `bitbake` command.

## **System memory is too low**

To debug the cases where a system runs out of free memory, see [Debugging linux kernel](https://dragonwingdocs.qualcomm.com/System/Debug-Subsystem/Debugging%20linux%20kernel).

## **Identify DTB during boot**

When the device boots, the boot loader verifies the following IDs and loads the corresponding DTB file:

```text theme={null}
qcom,msm-id = <x z>;
qcom,board-id = <y y'>;
```

* x = ID for SoC
* z = ID for SoC revision (reserved field)
* y = ID for CDP, MTP (hardware variants), and platform
* y’ = ID for subtype (assumed 0 if absent)

## **Debug RT kernel**

The following methods are used to troubleshooting issues in the RT kernel:

* How to verify that the underlying kernel is real time?
  After the boot is complete, run the following commands to verify the kernel type:
  ```text theme={null}
  uname -r
  6.6-rt15
  uname -v
  SMP PREEMPT_RT
  ```
* How to make drivers RT compliant?
  Ensure that synchronization primitives do not break RT assumptions.
  The following are examples of the scenarios that can break RT assumptions and cause unexpected system behavior:
  ```text theme={null}
  /* Acquiring a preemptible lock in non preemptible context */
  preempt_disable( )
  ……
  spin_lock( )
  /* Acquiring a preemptible lock in non preemptible context /
  raw_spin_lock( )
  ……
  spin_lock( )
  /* Acquiring a non preemptible lock in preemptible context /
  local_lock_irq( )
     …..
  raw_spin_lock( )
  ```
* Debug RT kernel configurations
  The following are the debug configurations for the RT kernel:
  * `CONFIG_DEBUG_ATOMIC_SLEEP` - Verify for sleep inside an atomic section.
  * `CONFIG_PROVE_RAW_LOCK_NESTING` - Allows the `raw_spinlock` vs. spinlock nesting. Ensures that the lock nesting rules for `PREEMPT_RT` kernels are not violated.
    While debugging, you can switch from an RT kernel to a non-RT kernel. To switch, make changes to the `meta-qcom/ci/linux-qcom-rt-6.18.yml` file to change virtual/kernel settings in `linux-qcom-rt` and compile again.
    To make the changes, run the following commands:
    ```text theme={null}
    - PREFERRED_PROVIDER_virtual/kernel = "linux-qcom-rt"
    + PREFERRED_PROVIDER_virtual/kernel = "linux-qcom"
    ```

For more information about locking primitives with the RT kernel, see [Lock types and their rules—The Linux Kernel documentation](https://www.kernel.org/doc/html/latest/locking/locktypes.html).

## **Troubleshooting virtio issues**

Use the following methods to troubleshoot virtio issues:

* Ensure that the kernel is compiled with the required configurations.
* Verify the QEMU command-line options or libvirt XML configuration.
* Verify the system logs for errors.
* Enable traces in the QEMU command-line for virtio using the available trace back ends. For more information about KVM traces, see [KVM traces](https://docs.qualcomm.com/doc/80-80021-3/topic/virtualization.html?product=895724676033554725\&facet=Kernel\&version=2.0-rc2#kvm-traces).