Skip to main content
A subsystem processes independently within its own execution environment on the Qualcomm® SoC. This section describes the tools, sample logs, and troubleshooting methods for impacted subsystems. Understanding the discrepancies in subsystems and methods for examining errors is useful for diagnosing and troubleshooting such anomalies. The following table lists the important subsystems on the SoC. Table: Subsystems on SoC
SubsystemDescription
Application processor subsystem (APSS)This primary subsystem executes the Qualcomm® Linux® kernel as the high-level OS (HLOS).
Application digital signal processor (aDSP)This subsystem, also known as the low-power audio subsystem (LPASS), handles digital signal processing tasks, such as audio encoding/decoding, and voice recognition.
Compute digital signal processor (cDSP)This subsystem performs compute-intensive tasks, such as neural network-related calculations.
Qualcomm® Trusted Execution Environment (TEE)/TrustZone (TZ)This subsystem performs secure operations and leverages the Arm® TrustZone® architecture. For more information, see Qualcomm Linux Security Guide.
Wireless local area network processor subsystem (WPSS)This subsystem connects to a Wi-Fi network, transmits and receives data packets, handles security protocols, and ensures stable network performance. For more information, see Qualcomm Linux Wi-Fi Guide.
Always-on Processor (AOP)This subsystem regulates the power for the device.
The term non-HLOS refers to software for all subsystems other than the application processor.To understand the boot flow of the subsystems, see Qualcomm Linux Boot Guide.
See Hardware SoCs that are supported on Qualcomm® Linux®.

​
Debug workflow

Identify the subsystem where the issue first occurs, as the Qualcomm SoC has many subsystems. This helps you to debug the correct subsystem. The application processor, as the primary subsystem, detects when other subsystems crash. For example, if the aDSP processor subsystem crashes, the kernel log captures the subsystem restart (SSR) crash error log. Therefore, to identify the subsystem that needs debugging, first verify the kernel debug messages. The following figure shows the workflow to identify errors and the components you need to debug.
Workflow for diagnosing kernel panics, software bugs, and subsystem crash issues.
When the kernel dmesg logs show no panic signatures, verify the Qualcomm TEE diagnostic (diag) logs for errors such as nonsecure watchdog bite and network-on-chip (NoC) errors. The following sections describe example kernel messages that indicate kernel panic, bugs, and subsystem crash issues.

​
Identify kernel panic and bugs

You can determine from the kernel dmesg logs whether the reset was a kernel panic or a bug. The following example logs indicate a kernel panic or a bug:
  • Panic (Pattern: General error) 18.800936: <6> Kernel panic - not syncing: Fatal exception 18.800938: <6> SMP: stopping secondary CPUs 18.800947: <6> CPU0: stopping Kernel panic - not syncing: Apps watchdog Bark received!
  • Bug (Pattern 1: Forced crash caused by memory corruption) 12.899532: <6> BUG kmalloc-128 (Not tainted): Redzone overwritten 12.905418: <6> -----------------------------------------------------------------------------
  • Bug (Pattern 2: Forced crash from driver) 320.510769: <6> ------------[ cut here ]------------ 320.510781: <6> kernel BUG at /local/mnt/workspace/lnxbuild/project/trees_in_use/free_tree_platform_manifest_refs_tags/drivers/platform/msm/ipa/ipa_v3/ipa_qmi_service.c:955!
  • Bug (Pattern 3: Forced crash due to lockup issue) [ 180.993861] BUG: spinlock lockup suspected on CPU#0, swapper/0/0 [ 180.993883] lock: stop_lock+0x0/0x18, .magic: dead4ead, .owner: swapper/6/0, .owner_cpu: 6 [ 181.015629] Causing a watchdog bite!
Most bugs typically result in a kernel panic. For more information about debugging kernel issues, see Qualcomm Linux Kernel Guide.

​
Identify subsystem crash issues

You can verify the kernel dmesg logs to determine whether the restart was due to a subsystem crash. For example, the following log indicates that the aDSP subsystem crashed, according to the sc7280.dtsi (node: remoteproc_adsp: remoteproc@3700000): 0x000000000A27652C | 5198.790423: qcom_q6v5_pas 3700000.remoteproc: fatal error received: err_inject_crash.c:413:Crash injected via Diag 0x000000000A276689 | 5198.801061: remoteproc remoteproc2: crash detected in 3000000.remoteproc: type fatal error 0x000000000A2767A1 | 5198.809602: remoteproc remoteproc2: handling crash #1 in 3000000.remoteproc 0x000000000A27688E | 5198.816837: remoteproc remoteproc2: recovering 3000000.remoteproc 0x000000000A276971 | 5198.823784: qcom_q6v5_pas 8a00000.remoteproc: subsystem event rejected

​
Identify system issues from Qualcomm TEE logs

This feature is available to licensed developers with authorized access. For sample logs that indicate errors in the Qualcomm TEE, see Qualcomm Linux Debug Guide - Addendum. If the Qualcomm TEE diag logs show no error, the issue may be due to a secure watchdog bite. For more information about watchdog issues, see Hardware reset. For more information about the debugging issues in Qualcomm TEE, see Qualcomm Linux Security Guide.

​
Debug methods

The following are the two approaches for debugging:
  • On-target debugging On-target debugging resolves software issues. This approach allows access to most information directly from the live device. Connect the Linux host computer to the live device using SSH. To set up SSH, see Sign in using SSH. For more information about various on-target debug features, see On-target debug features.
  • Off-target debuggingi This approach uses logs instead of an actual device, making it an efficient method of debugging. You can debug using memory dumps or logging tools and various types of logs for offline debugging. Although the RAM dump captures most of the memory regions, it includes only limited hardware register information, so debugging a hardware-related issue can be a challenge. For more information about various off-target debug features, see Off-target debug features.

​
On-target debug features

You can use the following debug features on the device at runtime:

​
Perf utility

The Perf utility in Linux facilitates performance analysis and profiling. The Linux kernel includes Perf utility in the tools/perf directory. You can use the Perf utility to debug various aspects of system behavior, including CPU performance counters, tracepoints, kprobes, and uprobes (for dynamic tracing). The following table lists the key features of the Perf utility. Table: Key features of Perf utility
FeatureDescription
CPU performance countersThese CPU hardware registers track events, such as executed instructions, cache misses, and branch mispredictions. These events form the basis for profiling applications and identifying performance bottlenecks.
TracepointsYou can place these tracepoints at logical locations in the code, such as system calls, network events, and file system operations. These tracepoints provide information, such as timestamps and stack traces, with minimal overhead.
Dynamic tracingThis feature dynamically creates tracepoints using the kprobes and uprobes frameworks, allowing tracing in both kernel space and user space.
For more information about the Perf utility, see the following resources:

​
Subsystem restart

Subsystems can restart independently when they crash without requiring a device restart called subsystem restart (SSR). The SSR feature uses the remoteproc framework available in the Linux kernel. To enable a full memory dump for debugging purposes, disable the SSR feature by default.
It’s recommended to use the SSR feature on commercial devices and not during the development phase.
For more information about the SSR feature, see Configure the remoteprocessor subsystems. Enable the SSR feature to generate the SSR dump on a subsystem restart. Consider using the SSR dump for debugging, as its size is smaller than the full memory dump. However, you need the entire RAM dump to debug some subsystem crashes. For more information about how to enable and capture subsystem dumps, see Capture the subsystem dumps.

​
Force subsystem reset

Use this feature to debug use cases that require restarting a subsystem.
This feature is available to licensed developers with authorized access.
To force a subsystem reset, you can run the diag command using the QXDM Professional™ tool. For more information about QXDM Professional commands and their usage, see Qualcomm Linux Debug Guide - Addendum.

​
Off-target debug features

You can use RAM dump or QXDM Professional for offline debugging.

​
RAM dump

The RAM dump is a snapshot of the entire memory at the time of failure. You can analyze the RAM dump using various tools, including Qualcomm® Crash Analysis Portal (QCAP), RAMParser, Lauterbach TRACE32 simulator, and crash utility. For more information about how to collect and parse RAM dumps, see Collect and parse RAM dump.

​
QXDM Professional

Use QXDM Professional to debug various subsystems. This tool is available to licensed developers with authorized access. For more information about QXDM Professional commands and their usage, see Qualcomm Linux Debug Guide - Addendum.

​
Common system issues

You can categorize system issues into one of the following types:

​
Miscellaneous issues

You may run into the following miscellaneous issues:
  • Device-freeze issues due to software bugs
  • Linux application-related issues
  • Random resets due to hardware issues related to the PCB

​
Next steps