Configure the kernel for debugging - Qualcomm Dragonwing Documentation

The following table lists the common kernel configuration options that are useful for debugging. Table: Common kernel debug configuration options

Kernel debug configuration option	Description
CONFIG_DEBUG_LIST	Turns on checks for performing standard linked list manipulations with the `list.h` header file. If the pointers don’t match, the system prints a warning, followed by a `BUG_ON` crash.
CONFIG_PAGE_POISONING	Fills the pages with the poison pattern `(PAGE_POISON 0xaa)`, after calling `free_pages()`.
CONFIG_DEBUG_PAGEALLOC	Verifies the patterns before calling `alloc_pages()`.
CONFIG_DEBUG_USER	Prints a message when the system kills a user-space process due to a segmentation fault (`segfault`) or an invalid instruction, such as `user_debug=31` in the `arch/arm/Kconfig.debug` file. Add the Kernel boot parameter to the `BoardConfig.mk` file.
CONFIG_DEBUG_SPINLOCK	Identifies missing spinlock initialization and common spinlock errors, such as:

Waiting for more than one second on a spinlock

Freeing an already freed lock

Reinitializing a lock that was already used

CONFIG_DEBUG_MUTEXES	Detects Mutex semantic violations.
DEBUG_LOCK_ALLOC	Detects wrong freeing of live locks.
CONFIG_SLUB CONFIG_SLUB_DEBUG	Performs extra checks to detect the corruption of internal kernel memory allocation structures by adding poison for `use-after-free (0x6b)` and `buffer-overflow-padding (0xbb)`.
Kernel debug configuration options for extra debugging that can be verbose and can make the system slow.
CONFIG_DEBUG_ATOMIC_SLEEP	Causes routines that may sleep to become noisy when they’re called within the atomic sections.
DEBUG_SPINLOCK_SLEEP	Causes routines that may sleep to become noisy when they’re called with a spinlock held.
CONFIG_DEBUG_VM, CONFIG_DEBUG_HIGHMEM	Provides an extra debugging support for virtual memory management corruption issues.
CONFIG_DEBUG_OBJECTS	Tracks the lifetime of various objects and validates the operations on those objects.

The following are the example logs indicating common memory issues:

List corruption issues

To identify the following crash signatures that indicate list corruption issues, enable the CONFIG_DEBUG_LIST option. Sample crash signature 1

 \<4\> WARNING: at kernel/lib/list\_debug.c:60 \_\_list\_del\_entry\+0xa0/0xd0()

 \<6\> list\_del corruption. prev-\>next should be c6fc374c, but was c18d804c

 \<6\> Modules linked in: adsprpc

 \<6\> \[\<c010cc94\>\] (unwind\_backtrace\+0x0/0x138) from \[\<c018ad54\>\] (warn\_slowpath\_common\+0x4c/0x64)

 \<6\> \[\<c018ad54\>\] (warn\_slowpath\_common\+0x4c/0x64) from \[\<c018ae00\>\] (warn\_slowpath\_fmt\+0x30/0x40)

 \<6\> \[\<c018ae00\>\] (warn\_slowpath\_fmt\+0x30/0x40) from \[\<c03b3128\>\] (\_\_list\_del\_entry\+0xa0/0xd0)

 \<6\> \[\<c03b3128\>\] (\_\_list\_del\_entry\+0xa0/0xd0) from \[\<c01bc968\>\] (account\_entity\_dequeue\+0x84/0x94)

 \<6\> \[\<c01bc968\>\] (account\_entity\_dequeue\+0x84/0x94) from \[\<c01bdaa4\>\] (dequeue\_task\_fair\+0x64/0x190)

Sample crash signature 2

 \<4\>WARNING: at kernel/lib/list\_debug.c:52 \_\_list\_del\_entry\+0x8c/0xac()

 \<4\> list\_del corruption, e240c500-\>prev is LIST\_POISON2 (00200200)

 \<4\> Modules linked in: wlan(PO) cfg80211 adsp\_loader

 \<4\> \[\<c001498c\>\] (unwind\_backtrace\+0x0/0x11c)

 from \[\<c0068c90\>\] (warn\_slowpath\_common\+0x4c/0x64)

 \<4\> \[\<c0068c90\>\] (warn\_slowpath\_common\+0x4c/0x64)

 from \[\<c0068d28\>\] (warn\_slowpath\_fmt\+0x2c/0x3c)

 \<4\> \[\<c0068d28\>\] (warn\_slowpath\_fmt\+0x2c/0x3c)

 from \[\<c023ef94\>\] (\_\_list\_del\_entry\+0x8c/0xac)

 \<4\> \[\<c023ef94\>\] (\_\_list\_del\_entry\+0x8c/0xac)

 from \[\<c023efc0\>\] (list\_del\+0xc/0x24)

 \<4\> \[\<c023efc0\>\] (list\_del\+0xc/0x24)

 from \[\<c043c78c\>\] (binder\_thread\_read\+0x488/0xb70)

 \<4\> \[\<c043c78c\>\] (binder\_thread\_read\+0x488/0xb70)

 from \[\<c043d094\>\] (binder\_ioctl\+0x220/0x5b8)

Sample crash signature 3

 \<4\> WARNING: at kernel/lib/list\_debug.c:47 \_\_list\_del\_entry\+0x90/0xb0()

 \<4\> list\_del corruption, d6ed3720-\>next is LIST\_POISON1 (00100100)

 \<4\> Modules linked in:

 \<4\> \[\] (unwind\_backtrace\+0x0/0x12c) from \[\] (warn\_slowpath\_common\+0x4c/0x64)

 \<4\> \[\] (warn\_slowpath\_common\+0x4c/0x64) from \[\] (warn\_slowpath\_fmt\+0x2c/0x3c)

 \<4\> \[\] (warn\_slowpath\_fmt\+0x2c/0x3c) from \[\] (\_\_list\_del\_entry\+0x90/0xb0)

 \<4\> \[\] (\_\_list\_del\_entry\+0x90/0xb0) from \[\] (list\_del\+0xc/0x28)

 \<4\> \[\] (list\_del\+0xc/0x28) from \[\] (bam\_mux\_write\_done\+0x34/0x11c)

Identify spinlock corruption issues

To identify crash signatures that indicate spinlock corruption issues, enable the CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_MUTEXES kernel configuration options. Sample crash signature 1

 \<0\>BUG: spinlock lockup on CPU#1, ndroid.launcher/1071

 \<0\> lock: 0xd5e8f480, .magic: dead4ead, .owner: \<none\>/-1, .owner\_cpu: -1

 \<4\> \[\<c0734d84\>\] (spin\_dump\+0x74/0x84) from \[\<c028f8fc\>\]

 (do\_raw\_spin\_lock\+0x144/0x188)

 \<4\> \[\<c028f8fc\>\] (do\_raw\_spin\_lock\+0x144/0x188) from \[\<c0332594\>\] (kgsl\_mmu\_pt\_get\_flags\+0x18/0x44)

 \<4\> \[\<c0332594\>\] (kgsl\_mmu\_pt\_get\_flags\+0x18/0x44) from \[\<c033c6c8\>\] (adreno\_ringbuffer\_submitcmd\+0x168/0x228)

 \<4\> \[\<c033c6c8\>\] (adreno\_ringbuffer\_submitcmd\+0x168/0x228) from \[\<c033d9a4\>\] (sendcmd\+0x3c/0x254)

Sample crash signature 2

 \<0\> BUG: spinlock lockup on CPU#2, kworker/2:0H/2910

 \<0\> lock: kpss\_clock\_reg\_lock\+0x0/0x10, .magic: dead4ead, .owner:

 kworker/3:0H/2904, .owner\_cpu: 3

 \<6\> kworker/2:0H (2910): undefined instruction: pc=c0963098

 kernel BUG at kernel/lib/spinlock\_debug.c:95!

 \<4\> \[\<c0963098\>\] (spin\_dump\+0x7c/0x94) from \[\<c038e030\>\]

 (do\_raw\_spin\_lock\+0xcc/0x164)

 \<4\> \[\<c038e030\>\] (do\_raw\_spin\_lock\+0xcc/0x164) from \[\<c0972228\>\] (\_raw\_spin\_lock\_irqsave\+0x20/0x28)

 \<4\> \[\<c0972228\>\] (\_raw\_spin\_lock\_irqsave\+0x20/0x28) from \[\<c011d4ac\>\] (\_\_kpss\_mux\_set\_sel\+0x14/0x80)

 \<4\> \[\<c011d4ac\>\] (\_\_kpss\_mux\_set\_sel\+0x14/0x80) from \[\<c011d54c\>\] (kpss\_mux\_set\_sel\+0x18/0x20)

Identify slub poisoning issues

To identify crash signatures that indicate slub poisoning issues, enable the CONFIG_SLUB and CONFIG_SLUB_DEBUG kernel configuration options.

<3\>\[ 3438.930472\] =====================================================================

<3\>\[ 3438.937628\] BUG kmalloc-64 (Tainted: G W O): Poison overwritten

<3\>\[ 3438.944223\] ---------------------------------------------------------------------

<3\>\[ 3438.944228\]

<3\>\[ 3438.953861\] INFO: 0xce308408-0xce30840b. First byte 0x0 instead of 0x6b

<3\>\[ 3438.960470\] INFO: Allocated in kgsl\_ioctl\_drawctxt\_create\+0x2c/0x2ac age=4426 cpu=0 pid=21702

<3\>\[ 3438.968970\] \_\_slab\_alloc.isra.37.constprop.43\+0x4d4/0x534

<3\>\[ 3438.974435\] kmem\_cache\_alloc\_trace\+0x240/0x258

<3\>\[ 3438.978947\] kgsl\_ioctl\_drawctxt\_create\+0x2c/0x2ac

<3\>\[ 3438.983721\] kgsl\_ioctl\+0xfc/0x324

<3\>\[ 3438.987108\] do\_vfs\_ioctl\+0x80/0x54c

<3\>\[ 3438.990669\] sys\_ioctl\+0x38/0x5c

<3\>\[ 3438.993880\] ret\_fast\_syscall\+0x0/0x30

<3\>\[ 3438.997615\] INFO: Freed in kgsl\_release\+0xb8/0xc0 age=10 cpu=3 pid=21842

<3\>\[ 3439.004298\] \_\_slab\_free\+0x30/0x308

<3\>\[ 3439.007770\] kgsl\_release\+0xb8/0xc0

<3\>\[ 3439.011242\] fput\+0xcc/0x23c

<3\>\[ 3439.014106\] filp\_close\+0x68/0x80

<3\>\[ 3439.017407\] put\_files\_struct\+0xd8/0x110

<3\>\[ 3439.021312\] do\_exit\+0x164/0x860

\<3\>\[ 3439.024524\] do\_group\_exit\+0x3c/0xb0

\<3\>\[ 3439.028084\] get\_signal\_to\_deliver\+0x2c4/0x59c

\<3\>\[ 3439.032510\] do\_signal\+0x90/0x480

<3\>\[ 3439.035809\] do\_notify\_resume\+0x50/0x5c

<3\>\[ 3439.039629\] work\_pending\+0x24/0x28

<3\>\[ 3439.043101\] INFO: Slab 0xc0fbcd20 objects=16 used=16 fp=0x (null) flags=0x0080

<3\>\[ 3439.050401\] INFO: Object 0xce308400 @offset=1024 fp=0xce308900

<3\>\[ 3439.050405\]

<3\>\[ 3439.057694\] Bytes b4 ce3083f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ

<3\>\[ 3439.066457\] Object ce308400: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b 6b kkkkkkkk....kkkk

<3\>\[ 3439.075051\] Object ce308410: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk

<3\>\[ 3439.083639\] Object ce308420: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk

<3\>\[ 3439.092234\] Object ce308430: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.

<3\>\[ 3439.100828\] Redzone ce308440: bb bb bb bb ....

<3\>\[ 3439.108463\] Padding ce3084e8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ

<3\>\[ 3439.117145\] Padding ce3084f8: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ

<4\>\[ 3439.125144\] \[\<c0014f78\>\] (unwind\_backtrace\+0x0/0x138) from \[\<c0150250\>\] (check\_bytes\_and\_report\+0xc0/0xe4)

<4\>\[ 3439.134771\] \[\<c0150250\>\] (check\_bytes\_and\_report\+0xc0/0xe4) from \[\<c015042c\>\] (check\_object\+0x1b8/0x214)

<4\>\[ 3439.144237\] \[\<c015042c\>\] (check\_object\+0x1b8/0x214) from \[\<c071b0bc\>\] (alloc\_debug\_processing\+0x7c/0x150)

<4\>\[ 3439.153782\] \[\<c071b0bc\>\] (alloc\_debug\_processing\+0x7c/0x150) from \[\<c071bcc0\>\] (\_\_slab\_alloc.isra.37.constprop.43\+0x4d4/0x534)

<4\>\[ 3439.165153\] \[\<c071bcc0\>\] (\_\_slab\_alloc.isra.37.constprop.43\+0x4d4/0x534) from \[\<c015221c\>\] (kmem\_cache\_alloc\_trace\+0x240/0x258)

<4\>\[ 3439.176611\] \[\<c015221c\>\] (kmem\_cache\_alloc\_trace\+0x240/0x258) from \[\<c014668c\>\] (\_\_get\_vm\_area\_node.isra.26\+0x84/0x174)

<4\>\[ 3439.187376\] \[\<c014668c\>\] (\_\_get\_vm\_area\_node.isra.26\+0x84/0x174) from \[\<c0147038\>\] (get\_vm\_area\_caller\+0x44/0x4c)

<4\>\[ 3439.197615\] \[\<c0147038\>\] (get\_vm\_area\_caller\+0x44/0x4c) from \[\<c014739c\>\] (vmap\+0x50/0x90)

<4\>\[ 3439.205866\] \[\<c014739c\>\] (vmap\+0x50/0x90) from \[\<c032a694\>\] (\_kgsl\_sharedmem\_page\_alloc\+0x238/0x3d4)

<4\>\[ 3439.214979\] \[\<c032a694\>\] (\_kgsl\_sharedmem\_page\_alloc\+0x238/0x3d4) from \[\<c0323e44\>\] (\_gpumem\_alloc\+0xb0/0xfc)

<4\>\[ 3439.224874\] \[\<c0323e44\>\] (\_gpumem\_alloc\+0xb0/0xfc) from \[\<c0323ed0\>\] (kgsl\_ioctl\_gpumem\_alloc\_id\+0x40/0x1a8)

<4\>\[ 3439.234684\] \[\<c0323ed0\>\] (kgsl\_ioctl\_gpumem\_alloc\_id\+0x40/0x1a8) from \[\<c0321874\>\] (kgsl\_ioctl\+0xfc/0x324)

<4\>\[ 3439.244319\] \[\<c0321874\>\] (kgsl\_ioctl\+0xfc/0x324) from \[\<c0167fe8\>\] (do\_vfs\_ioctl\+0x80/0x54c)

<4\>\[ 3439.252738\] \[\<c0167fe8\>\] (do\_vfs\_ioctl\+0x80/0x54c) from \[\<c01684ec\>\] (sys\_ioctl\+0x38/0x5c)

<4\>\[ 3439.260986\] \[\<c01684ec\>\] (sys\_ioctl\+0x38/0x5c) from \[\<c000eb00\>\] (ret\_fast\_syscall\+0x0/0x30)

<3\>\[ 3439.269400\] FIX kmalloc-64: Restoring 0xce308408-0xce30840b=0x6b

Identify page poisoning issues

To identify a crash signature that indicates page poisoning issues, enable the CONFIG_DEBUG_PAGEALLOC and CONFIG_PAGE_POISONING kernel configuration options.

\<1\> Unable to handle kernel paging request at virtual address aaaaaaae

\<1\> pgd = e98b4000

\<1\> \[aaaaaaae\] [\*](https://docs.qualcomm.com/doc/80-80021-12/topic/configure_the_kernel_for_debugging.html?product=895724676033554725&facet=Debug%20Subsystem&version=2.0-rc2#id2)pgd=00000000

\<0\> Internal error: Oops: 5 \[#1\] PREEMPT SMP ARM

\<4\> Modules linked in: adsp\_loader exfat\_fs(P) exfat\_core(P)

\<4\> CPU: 1 Tainted: P W (3.4.0-628250-eng #1)

\<4\> PC is at pid\_nr\_ns\+0xc/0x3c

\<4\> LR is at do\_task\_stat\+0x248/0x83c

\<4\> pc : \[\<c00abda8\>\] lr : \[\<c018c15c\>\] psr: a0000093

​List corruption issues

​Identify spinlock corruption issues

​Identify slub poisoning issues

​Identify page poisoning issues

List corruption issues

Identify spinlock corruption issues

Identify slub poisoning issues

Identify page poisoning issues