> ## Documentation Index > Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt > Use this file to discover all available pages before exploring further. # Configure the kernel for debugging The following table lists the common kernel configuration options that are useful for debugging. **Table: Common kernel debug configuration options** | **Kernel debug configuration option** | **Description** | | :-----------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | CONFIG\_DEBUG\_LIST | Turns on checks for performing standard linked list manipulations with the `list.h` header file. If the pointers don't match, the system prints a warning, followed by a `BUG_ON` crash. | | CONFIG\_PAGE\_POISONING | Fills the pages with the poison pattern `(PAGE_POISON 0xaa)`, after calling `free_pages()`. | | CONFIG\_DEBUG\_PAGEALLOC | Verifies the patterns before calling `alloc_pages()`. | | CONFIG\_DEBUG\_USER | Prints a message when the system kills a user-space process due to a segmentation fault (`segfault`) or an invalid instruction, such as `user_debug=31` in the `arch/arm/Kconfig.debug` file. Add the Kernel boot parameter to the `BoardConfig.mk` file. | | CONFIG\_DEBUG\_SPINLOCK | Identifies missing spinlock initialization and common spinlock errors, such as: | > * Waiting for more than one second on a spinlock > * Freeing an already freed lock > * Reinitializing a lock that was already used | CONFIG\_DEBUG\_MUTEXES | Detects Mutex semantic violations. | | :------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DEBUG\_LOCK\_ALLOC | Detects wrong freeing of live locks. | | CONFIG\_SLUB CONFIG\_SLUB\_DEBUG | Performs extra checks to detect the corruption of internal kernel memory allocation structures by adding poison for `use-after-free (0x6b)` and `buffer-overflow-padding (0xbb)`. | | Kernel debug configuration options for extra debugging that can be verbose and can make the system slow. | | | CONFIG\_DEBUG\_ATOMIC\_SLEEP | Causes routines that may sleep to become noisy when they're called within the atomic sections. | | DEBUG\_SPINLOCK\_SLEEP | Causes routines that may sleep to become noisy when they're called with a spinlock held. | | CONFIG\_DEBUG\_VM, CONFIG\_DEBUG\_HIGHMEM | Provides an extra debugging support for virtual memory management corruption issues. | | CONFIG\_DEBUG\_OBJECTS | Tracks the lifetime of various objects and validates the operations on those objects. | The following are the example logs indicating common memory issues: ## **List corruption issues** To identify the following crash signatures that indicate list corruption issues, enable the `CONFIG_DEBUG_LIST` option. **Sample crash signature 1** ```bash theme={null} \<4\> WARNING: at kernel/lib/list\_debug.c:60 \_\_list\_del\_entry\+0xa0/0xd0() \<6\> list\_del corruption. prev-\>next should be c6fc374c, but was c18d804c \<6\> Modules linked in: adsprpc \<6\> \[\\] (unwind\_backtrace\+0x0/0x138) from \[\\] (warn\_slowpath\_common\+0x4c/0x64) \<6\> \[\\] (warn\_slowpath\_common\+0x4c/0x64) from \[\\] (warn\_slowpath\_fmt\+0x30/0x40) \<6\> \[\\] (warn\_slowpath\_fmt\+0x30/0x40) from \[\\] (\_\_list\_del\_entry\+0xa0/0xd0) \<6\> \[\\] (\_\_list\_del\_entry\+0xa0/0xd0) from \[\\] (account\_entity\_dequeue\+0x84/0x94) \<6\> \[\\] (account\_entity\_dequeue\+0x84/0x94) from \[\\] (dequeue\_task\_fair\+0x64/0x190) ``` **Sample crash signature 2** ```bash theme={null} \<4\>WARNING: at kernel/lib/list\_debug.c:52 \_\_list\_del\_entry\+0x8c/0xac() \<4\> list\_del corruption, e240c500-\>prev is LIST\_POISON2 (00200200) \<4\> Modules linked in: wlan(PO) cfg80211 adsp\_loader \<4\> \[\\] (unwind\_backtrace\+0x0/0x11c) from \[\\] (warn\_slowpath\_common\+0x4c/0x64) \<4\> \[\\] (warn\_slowpath\_common\+0x4c/0x64) from \[\\] (warn\_slowpath\_fmt\+0x2c/0x3c) \<4\> \[\\] (warn\_slowpath\_fmt\+0x2c/0x3c) from \[\\] (\_\_list\_del\_entry\+0x8c/0xac) \<4\> \[\\] (\_\_list\_del\_entry\+0x8c/0xac) from \[\\] (list\_del\+0xc/0x24) \<4\> \[\\] (list\_del\+0xc/0x24) from \[\\] (binder\_thread\_read\+0x488/0xb70) \<4\> \[\\] (binder\_thread\_read\+0x488/0xb70) from \[\\] (binder\_ioctl\+0x220/0x5b8) ``` **Sample crash signature 3** ```bash theme={null} \<4\> WARNING: at kernel/lib/list\_debug.c:47 \_\_list\_del\_entry\+0x90/0xb0() \<4\> list\_del corruption, d6ed3720-\>next is LIST\_POISON1 (00100100) \<4\> Modules linked in: \<4\> \[\] (unwind\_backtrace\+0x0/0x12c) from \[\] (warn\_slowpath\_common\+0x4c/0x64) \<4\> \[\] (warn\_slowpath\_common\+0x4c/0x64) from \[\] (warn\_slowpath\_fmt\+0x2c/0x3c) \<4\> \[\] (warn\_slowpath\_fmt\+0x2c/0x3c) from \[\] (\_\_list\_del\_entry\+0x90/0xb0) \<4\> \[\] (\_\_list\_del\_entry\+0x90/0xb0) from \[\] (list\_del\+0xc/0x28) \<4\> \[\] (list\_del\+0xc/0x28) from \[\] (bam\_mux\_write\_done\+0x34/0x11c) ``` ## **Identify spinlock corruption issues** To identify crash signatures that indicate spinlock corruption issues, enable the `CONFIG_DEBUG_SPINLOCK` and `CONFIG_DEBUG_MUTEXES` kernel configuration options. **Sample crash signature 1** ```bash theme={null} \<0\>BUG: spinlock lockup on CPU#1, ndroid.launcher/1071 \<0\> lock: 0xd5e8f480, .magic: dead4ead, .owner: \/-1, .owner\_cpu: -1 \<4\> \[\\] (spin\_dump\+0x74/0x84) from \[\\] (do\_raw\_spin\_lock\+0x144/0x188) \<4\> \[\\] (do\_raw\_spin\_lock\+0x144/0x188) from \[\\] (kgsl\_mmu\_pt\_get\_flags\+0x18/0x44) \<4\> \[\\] (kgsl\_mmu\_pt\_get\_flags\+0x18/0x44) from \[\\] (adreno\_ringbuffer\_submitcmd\+0x168/0x228) \<4\> \[\\] (adreno\_ringbuffer\_submitcmd\+0x168/0x228) from \[\\] (sendcmd\+0x3c/0x254) ``` **Sample crash signature 2** ```bash theme={null} \<0\> BUG: spinlock lockup on CPU#2, kworker/2:0H/2910 \<0\> lock: kpss\_clock\_reg\_lock\+0x0/0x10, .magic: dead4ead, .owner: kworker/3:0H/2904, .owner\_cpu: 3 \<6\> kworker/2:0H (2910): undefined instruction: pc=c0963098 kernel BUG at kernel/lib/spinlock\_debug.c:95! \<4\> \[\\] (spin\_dump\+0x7c/0x94) from \[\\] (do\_raw\_spin\_lock\+0xcc/0x164) \<4\> \[\\] (do\_raw\_spin\_lock\+0xcc/0x164) from \[\\] (\_raw\_spin\_lock\_irqsave\+0x20/0x28) \<4\> \[\\] (\_raw\_spin\_lock\_irqsave\+0x20/0x28) from \[\\] (\_\_kpss\_mux\_set\_sel\+0x14/0x80) \<4\> \[\\] (\_\_kpss\_mux\_set\_sel\+0x14/0x80) from \[\\] (kpss\_mux\_set\_sel\+0x18/0x20) ``` ## **Identify slub poisoning issues** To identify crash signatures that indicate slub poisoning issues, enable the `CONFIG_SLUB` and `CONFIG_SLUB_DEBUG` kernel configuration options. ```bash theme={null} <3\>\[ 3438.930472\] ===================================================================== <3\>\[ 3438.937628\] BUG kmalloc-64 (Tainted: G W O): Poison overwritten <3\>\[ 3438.944223\] --------------------------------------------------------------------- <3\>\[ 3438.944228\] <3\>\[ 3438.953861\] INFO: 0xce308408-0xce30840b. First byte 0x0 instead of 0x6b <3\>\[ 3438.960470\] INFO: Allocated in kgsl\_ioctl\_drawctxt\_create\+0x2c/0x2ac age=4426 cpu=0 pid=21702 <3\>\[ 3438.968970\] \_\_slab\_alloc.isra.37.constprop.43\+0x4d4/0x534 <3\>\[ 3438.974435\] kmem\_cache\_alloc\_trace\+0x240/0x258 <3\>\[ 3438.978947\] kgsl\_ioctl\_drawctxt\_create\+0x2c/0x2ac <3\>\[ 3438.983721\] kgsl\_ioctl\+0xfc/0x324 <3\>\[ 3438.987108\] do\_vfs\_ioctl\+0x80/0x54c <3\>\[ 3438.990669\] sys\_ioctl\+0x38/0x5c <3\>\[ 3438.993880\] ret\_fast\_syscall\+0x0/0x30 <3\>\[ 3438.997615\] INFO: Freed in kgsl\_release\+0xb8/0xc0 age=10 cpu=3 pid=21842 <3\>\[ 3439.004298\] \_\_slab\_free\+0x30/0x308 <3\>\[ 3439.007770\] kgsl\_release\+0xb8/0xc0 <3\>\[ 3439.011242\] fput\+0xcc/0x23c <3\>\[ 3439.014106\] filp\_close\+0x68/0x80 <3\>\[ 3439.017407\] put\_files\_struct\+0xd8/0x110 <3\>\[ 3439.021312\] do\_exit\+0x164/0x860 \<3\>\[ 3439.024524\] do\_group\_exit\+0x3c/0xb0 \<3\>\[ 3439.028084\] get\_signal\_to\_deliver\+0x2c4/0x59c \<3\>\[ 3439.032510\] do\_signal\+0x90/0x480 <3\>\[ 3439.035809\] do\_notify\_resume\+0x50/0x5c <3\>\[ 3439.039629\] work\_pending\+0x24/0x28 <3\>\[ 3439.043101\] INFO: Slab 0xc0fbcd20 objects=16 used=16 fp=0x (null) flags=0x0080 <3\>\[ 3439.050401\] INFO: Object 0xce308400 @offset=1024 fp=0xce308900 <3\>\[ 3439.050405\] <3\>\[ 3439.057694\] Bytes b4 ce3083f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ <3\>\[ 3439.066457\] Object ce308400: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b kkkkkkkk....kkkk <3\>\[ 3439.075051\] Object ce308410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk <3\>\[ 3439.083639\] Object ce308420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk <3\>\[ 3439.092234\] Object ce308430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk. <3\>\[ 3439.100828\] Redzone ce308440: bb bb bb bb .... <3\>\[ 3439.108463\] Padding ce3084e8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ <3\>\[ 3439.117145\] Padding ce3084f8: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ <4\>\[ 3439.125144\] \[\\] (unwind\_backtrace\+0x0/0x138) from \[\\] (check\_bytes\_and\_report\+0xc0/0xe4) <4\>\[ 3439.134771\] \[\\] (check\_bytes\_and\_report\+0xc0/0xe4) from \[\\] (check\_object\+0x1b8/0x214) <4\>\[ 3439.144237\] \[\\] (check\_object\+0x1b8/0x214) from \[\\] (alloc\_debug\_processing\+0x7c/0x150) <4\>\[ 3439.153782\] \[\\] (alloc\_debug\_processing\+0x7c/0x150) from \[\\] (\_\_slab\_alloc.isra.37.constprop.43\+0x4d4/0x534) <4\>\[ 3439.165153\] \[\\] (\_\_slab\_alloc.isra.37.constprop.43\+0x4d4/0x534) from \[\\] (kmem\_cache\_alloc\_trace\+0x240/0x258) <4\>\[ 3439.176611\] \[\\] (kmem\_cache\_alloc\_trace\+0x240/0x258) from \[\\] (\_\_get\_vm\_area\_node.isra.26\+0x84/0x174) <4\>\[ 3439.187376\] \[\\] (\_\_get\_vm\_area\_node.isra.26\+0x84/0x174) from \[\\] (get\_vm\_area\_caller\+0x44/0x4c) <4\>\[ 3439.197615\] \[\\] (get\_vm\_area\_caller\+0x44/0x4c) from \[\\] (vmap\+0x50/0x90) <4\>\[ 3439.205866\] \[\\] (vmap\+0x50/0x90) from \[\\] (\_kgsl\_sharedmem\_page\_alloc\+0x238/0x3d4) <4\>\[ 3439.214979\] \[\\] (\_kgsl\_sharedmem\_page\_alloc\+0x238/0x3d4) from \[\\] (\_gpumem\_alloc\+0xb0/0xfc) <4\>\[ 3439.224874\] \[\\] (\_gpumem\_alloc\+0xb0/0xfc) from \[\\] (kgsl\_ioctl\_gpumem\_alloc\_id\+0x40/0x1a8) <4\>\[ 3439.234684\] \[\\] (kgsl\_ioctl\_gpumem\_alloc\_id\+0x40/0x1a8) from \[\\] (kgsl\_ioctl\+0xfc/0x324) <4\>\[ 3439.244319\] \[\\] (kgsl\_ioctl\+0xfc/0x324) from \[\\] (do\_vfs\_ioctl\+0x80/0x54c) <4\>\[ 3439.252738\] \[\\] (do\_vfs\_ioctl\+0x80/0x54c) from \[\\] (sys\_ioctl\+0x38/0x5c) <4\>\[ 3439.260986\] \[\\] (sys\_ioctl\+0x38/0x5c) from \[\\] (ret\_fast\_syscall\+0x0/0x30) <3\>\[ 3439.269400\] FIX kmalloc-64: Restoring 0xce308408-0xce30840b=0x6b ``` ## **Identify page poisoning issues** To identify a crash signature that indicates page poisoning issues, enable the `CONFIG_DEBUG_PAGEALLOC` and `CONFIG_PAGE_POISONING` kernel configuration options. ```bash theme={null} \<1\> Unable to handle kernel paging request at virtual address aaaaaaae \<1\> pgd = e98b4000 \<1\> \[aaaaaaae\] [\*](https://docs.qualcomm.com/doc/80-80021-12/topic/configure_the_kernel_for_debugging.html?product=895724676033554725&facet=Debug%20Subsystem&version=2.0-rc2#id2)pgd=00000000 \<0\> Internal error: Oops: 5 \[#1\] PREEMPT SMP ARM \<4\> Modules linked in: adsp\_loader exfat\_fs(P) exfat\_core(P) \<4\> CPU: 1 Tainted: P W (3.4.0-628250-eng #1) \<4\> PC is at pid\_nr\_ns\+0xc/0x3c \<4\> LR is at do\_task\_stat\+0x248/0x83c \<4\> pc : \[\\] lr : \[\\] psr: a0000093 ```