Assess memory usage

Linux uses a virtual memory system. Therefore, the addresses that the user program accesses don’t correspond to the physical addresses that the hardware uses directly. Virtual memory introduces a layer of indirection, allowing programs to assign extra memory beyond the physically available memory. Memory management implementation covers the following areas:

Management of physical pages in the memory
Buddy system to assign memory in large chunks
Slab, slub, and slob allocators to assign smaller chunks of memory
vmalloc mechanism to assign noncontiguous blocks of memory
Address space of the processes

/proc file system

The /proc file system provides the following files:

/proc/meminfo

This file provides information about distribution and usage of memory.

To view the contents of this file, run the following command:

cat /proc/meminfo

Sample output:

MemTotal:        3813532 kB
MemFree:          624836 kB
MemAvailable:    2098008 kB
Buffers:           40416 kB
Cached:          1484320 kB
SwapCached:            0 kB
Active:          1334816 kB
.
.
.
.
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     2431048 kB
Committed_AS:   99995284 kB
VmallocTotal:  258867136 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:         163840 kB
CmaFree:            1368 kB

For more information about the parameters available in this file, see /proc filesystem.

/proc/vmstat

This file shows detailed virtual memory statistics from the kernel. Most of the statistics are available only if you enable the CONFIG_VM_EVENT_COUNTERS option in the init/Kconfig file.

To view the contents of this file, run the following command:

cat /proc/vmstat

Sample output:

nr_free_pages 156290
nr_alloc_batch 132
nr_inactive_anon 108
nr_active_anon 165006
nr_inactive_file 212275
nr_active_file 168709
nr_unevictable 64
nr_mlock 64
nr_anon_pages 164982
nr_mapped 90366
nr_file_pages 381184:
:
unevictable_pgs_mlocked 0
unevictable_pgs_munlocked 0
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 0

For more information about the parameters available in this file, see Linux manual page.

/proc/iomem

This file shows the memory map of the system for its various device drivers.

To view the contents of this file, run the following command:

cat /proc/iomem

Sample output:

007781b8-007791b7 : vmpm
010aa000-010abfff : tsens_physical
010ac000-010ac003 : pshold-base
010ad000-010aefff : tsens_physical
01680000-0168ffff : /soc/arm,smmu-anoc1@1680000
016c0000-016fffff : /soc/arm,smmu-anoc2@16c0000
01d0101c-01d0101f : sp2soc_irq_status
01d01024-01d01027 : sp2soc_irq_clr
01d01028-01d0102b : sp2soc_irq_mask
.
.
.
0caa0000-0caa3fff : jpeg_hw
0caa4000-0caa47ff : fd_core
0caa5000-0caa53ff : fd_misc
0cd00000-0cd3ffff : /soc/arm,smmu-mmss@cd00000
17817000-17817fff : msm-watchdog
17900000-1790dfff : msm-gladiator-erp
80000000-857fffff : System RAM
80080000-817fffff : Kernel code
82330000-82945fff : Kernel data
88f00000-8aafffff : System RAM
95300000-17e3bffff : System RAM

/proc/vmallocinfo

This file shows detailed information about virtual address allocation through vmalloc or ioremap.

To view the contents of this file, run the following command:

cat /proc/vmallocinfo

Sample output:

0xbf000000-0xbf002000     8192            module_alloc_update_bounds+0xc/0x5c     pages=1 vmalloc
0xbf004000-0xbf008000     16384          module_alloc_update_bounds+0xc/0x5c     pages=3 vmalloc
0xee800000-0xef800000    16777216   iotable_init+0x0/0xb0   phys=36800000   ioremap
0xf0000000-0xf0002000     8192            of_iomap+0x30/0x38   ioremap
0xf0002000-0xf0004000     8192            of_iomap+0x30/0x38   ioremap
0xf0004000-0xf000c000     32768          gen_pool_add_virt+0x48/0xb8   pages= 7    vmalloc
0xf000c000-0xf000e000     8192            msm_pm_setup_saved_state+0xcc/0x1bc   ioremap
………………………
0xf0174000-0xf0176000    8192            msm_cpu_status_probe+0xd8/0x20c   ioremap
0xf0f24000-0xf0f28000      16384         _kgsl_sharedmem_page_alloc+0xa0/0x41c   pages=3   vmalloc
0xf0f39000-0xf0f3e000      20480         _kgsl_sharedmem_page_alloc+0xa0/0x41c   pages=4   vmalloc
0xf0f61000-0xf0f66000      20480         _kgsl_sharedmem_page_alloc+0xa0/0x41c   pages=4   vmalloc
……….
0xfa400000-0xfa600000    2097152     iotable_init+0x0/0xb0 phys=fa00000    ioremap
0xfa71e000-0xfa71f000     4096            iotable_init+0x0/0xb0 phys=f991e000     ioremap
0xfefd8000-0xff000000     163840        pcpu_get_vm_areas+0x0/0x56c  vmalloc

memblock interface

The memblock interface on the debugfs file system provides details about the available and reserved memory regions in the system. This interface provides the following files:

/sys/kernel/debug/memblock/memory

This file provides information about all the available memory regions (HLOS and non-HLOS) visible to the Linux kernel. To determine the overall memory accessible to the kernel, calculate the difference between the start and end addresses of each of the region and sum these values to get the total occupied RAM. The remaining memory, which is the difference between the RAM size of the device and the occupied RAM, is the non-HLOS memory or the memory occupied by other subsystems.

To view the contents of this file, run the following command:

cat /sys/kernel/debug/memblock/memory

Sample output:

0x0000000080000000..0x00000000857fffff

0x0000000088f00000..0x000000008aafffff

0x0000000095300000..0x000000017e3bffff

/sys/kernel/debug/memblock/reserved

This file provides information about all the reserved memory regions in the system.

To view the contents of this file, run the following command:

cat /sys/kernel/debug/memblock/reserved

Sample output:

0x0000000080080000..0x0000000082944fff
0x0000000083200000..0x0000000083259bb4
0x0000000083400000..0x00000000839506c9
0x00000000f5800000..0x00000000ffbfffff
0x00000000ffff7000..0x00000000ffffefff
0x00000000ffffff40..0x00000000ffffff77
0x00000000ffffff80..0x00000000ffffffb7
0x00000000ffffffc0..0x00000000fffffff7
0x0000000179258000..0x000000017d9fffff
0x000000017da17000..0x000000017da1ffff
0x000000017da20e00..0x000000017da26fff
0x000000017da27300..0x000000017da2735f
0x000000017da27380..0x000000017da273df
0x000000017da27400..0x000000017da2755f
0x000000017da27580..0x000000017da27587
0x000000017da275c0..0x000000017da275c7
0x000000017da29600..0x000000017da29924
0x000000017da29940..0x000000017da29c64
0x000000017da29c80..0x000000017da29fa4
0x000000017da29fac..0x000000017da2a3f8
0x000000017da2a3fc..0x000000017da2a42e
0x000000017da2a430..0x000000017da2a45e
0x000000017da2a460..0x000000017e3bffff

Debug memory leak issues

To debug kernel memory leak issues, enable the following configuration options:

CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE=4000
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y

By default, a kernel thread scans the memory every 10 minutes and prints the number of new unreferenced objects found. For example,

unreferenced object 0xec26f000 (size 4096):

comm "Binder_2", pid 4592, jiffies 8848 (age 336.710s)

hex dump (first 32 bytes):

ec 4d f8 c0 02 00 00 00 00 00 00 00 00 00 00 00 .M..............

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

backtrace:

[<c0126f70>] kmem_cache_alloc_trace+0x17c/0x238

[<c03059c4>] ddl_client_transact+0xd0/0x158

[<c0314a3c>] ddl_open+0x4c/0x194

[<c0302288>] vcd_init_client_context+0x14/0x9c

[<c02ffd10>] vcd_open_in_ready+0x3c/0x94

[<c02fd31c>] vcd_open+0x214/0x274

[<c031b244>] vid_dec_open_client+0x1d0/0x288

[<c031b3ac>] vid_dec_open+0x30/0x7c

[<c012ff70>] chrdev_open+0x10c/0x134

[<c012aa7c>] __dentry_open.isra.12+0x190/0x29c

[<c0138a78>] do_last.isra.29+0x690/0x6c0

[<c0138c70>] path_openat+0xb8/0x35c

[<c0138ff4>] do_filp_open+0x2c/0x78

[<c012b7a0>] do_sys_open+0xd8/0x170

[<c000df20>] ret_fast_syscall+0x0/0x30

[<ffffffff>] 0xffffffff

Disable the KMEMLEAK option at boot time by passing KMEMLEAK=off on the kernel command line. For more information about the kmemleak.txt file, see kernel documentation. The following extra kernel configuration options are available to track the allocator of each page of memory:

CONFIG_PAGE_OWNER
CONFIG_PAGE_OWNER_ENABLE_DEFAULT
CONFIG_PAGE_EXTENSION

Enabling these options can help identify several allocations, which may indicate a memory leak issue.

Identify memory corruption issues

Enable the following kernel configuration options to identify memory corruption issues:

CONFIG_PAGE_POISONING
CONFIG_SLUB_DEBUG_ON
CONFIG_DEBUG_LIST
CONFIG_SLUB_DEBUG

Sample log:

BUG <slab cache affected>: <What went wrong>
-----------------------------------------------
INFO: <corruption start>-<corruption_end> <more info>
INFO: Slab <address> <slab information>
INFO: Object <address> <object information>
INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by cpu> pid=<pid of the process>
INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu> pid=<pid of the process>

For more information about slub debugging, see the Documentation/vm/slub.txt file available in the kernel documentation.

Out of memory

When the system fails to assign a page, the kernel logs display a message such as the following:

<4>[12146.861355] Thread-430: page allocation failure: order:0, mode:0x10d2

<CALL STACK>

<4>[12146.951687] Mem-info:

<4>[12146.953909] Normal per-cpu:

<4>[12146.956686] CPU 0: hi: 186, btch: 31 usd: 61

<4>[12146.961489] CPU 1: hi: 186, btch: 31 usd: 0

<4>[12146.966235] HighMem per-cpu:

<4>[12146.969122] CPU 0: hi: 186, btch: 31 usd: 54

<4>[12146.973877] CPU 1: hi: 186, btch: 31 usd: 0

…….

<4>[12147.010770] Normal free:53192kB min:3508kB low:4384kB high:5260kB ……..

<4>[12147.050805] lowmem_reserve[]: 0 9022 9022

<4>[12147.054610] HighMem free:153360kB min:512kB low:1824kB high:3140kB ………

<4>[12147.095453] lowmem_reserve[]: 0 0 0

<4>[12147.098617] Normal: 118*4kB 232*8kB 161*16kB 110*32kB 34*64kB 9*128kB 10*256kB 6*512kB 7*1024kB 6*2048kB 4*4096kB = 53224kB

<4>[12147.115455] HighMem: 2774*4kB 11769*8kB 3005*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153360kB

………………

These messages indicate that the system couldn’t assign the requested page. The top line of the log message provides several details about the out-of-memory issue. For example, in the following log:

<4>[1214.855361] Thread-4: page allocation failure: order:2, mode:0x10d2

order: Indicates the size of the page. In this example, 2 ² x PAGE_SIZE (4 K) = 16 K
- Linux uses a buddy allocator that allocates pages in powers of 2.
  - The maximum size of the buddy allocator is the order of 10 = 2 ¹⁰ x 4 kB = 4 MB.
  - For allocation > 4 MB, use an alternate allocation method such as the contiguous memory allocator (CMA).
- For failure of higher-order allocations, examine whether the memory can be virtually contiguous, instead of being physically contiguous.
mode: Indicates the type of page requested
- mode provides information about get free pages (GFP) flags. In this example, mode is 0x10d2.
- mode is the result of OR operation on all the GFP flags in the allocation.

Pages available in the system A page allocation failure message prints details about the size of pages that were available in the system.

Normal: 118*4kB 232*8kB 161*16kB 110*32kB 34*64kB 9*128kB 10*256kB 6*512kB 7*1024kB 6*2048kB 4*4096kB = 53224kB
HighMem: 2774*4kB 11769*8kB 3005*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153360kB

IOMMU page fault

Input-output memory management unit (IOMMU), also known as the system memory management unit (SMMU), performs memory management functions on behalf of subsystems that don’t have their own MMU. The IOMMU hardware block allows physically noncontiguous pages to support virtually contiguous memory. The memory translation logic in the IOMMU is the same as the logic in the CPU MMU. The IOMMU page fault is the most common IOMMU issue. The fault occurs when the requested page is mapped in the page table but isn’t found in the memory. The fault handler receives the context bank instance of the IOMMU and dumps the registers for this context. The following log indicates an IOMMU page fault.

[ 47.228992] msm_iommu_v1: Unexpected IOMMU page fault!

[ 47.233115] msm_iommu_v1: name = mdp_iommu

[ 47.237238] msm_iommu_v1: context = mdp_0 (0)

[ 47.241507] msm_iommu_v1: Interesting registers:

[ 47.246149] msm_iommu_v1: FAR = 0000000000000000

[ 47.250970] msm_iommu_v1: PAR = 0000000000000000

[ 47.255834] msm_iommu_v1: FSR = 00000002 [TF ]

[ 47.260540] msm_iommu_v1: FSYNR0 = 000005a1 FSYNR1 = 00030005

[ 47.266528] msm_iommu_v1: TTBR0 = 0000000071a28000

[ 47.271370] msm_iommu_v1: TTBR1 = 0000000000000000

[ 47.276248] msm_iommu_v1: SCTLR = 00001043 ACTLR = 70000000

[ 47.282221] msm_iommu_v1: CBAR = 00000000 CBFRSYNRA = 00000000

[ 47.288521] msm_iommu_v1: PRRR = ff0a81a8 NMRR = 40e040e0

[ 47.294461] msm_iommu_v1: NOTE: Value actually unknown for CBAR

[ 47.300394] msm_iommu_v1: NOTE: Value actually unknown for CBFRSYNRA

[ 47.306717] msm_iommu_v1: Page table in DDR shows PA = 0

The following table describes the fields captured in the log message. Table: Information in IOMMU page fault log

Item	Description
`name`	Name of the hardware block that caused the fault.
`FAR`	Fault address register (FAR) indicates the address at which the fault occurred.
`FSR`	Fault status register (FSR) indicates the following:

Translation fault (TF)

Access permission fault (APF)

Stalled status (SS)

The FSR is one of the most important registers in IOMMU debugging. This register has read/write-clear access. The read operation on this register reads the value in the register while the write operation clears the bits corresponding to 1s in the written data and leaves the bits corresponding to 0s unchanged. This process prevents inadvertent clearing of new faults when writing the register to clear an old fault. Some useful bits in this register are: Table: Bits in fault status register

Bit	Description
`[Bit 1]: TF`	Translation fault (invalid page table entry)
`[Bit 2]: AFF`	Access fault
`[Bit 3]: APF`	Permission fault (write to read only region)
`[Bit 4]: TLBMF`	TLB miss fault
`[Bit 5]: HTWDEEF`	Hardware table walk decode error external fault
`[Bit 6]: HTWSEEF`	Hardware table walk subordinate error external fault
`[Bit 7]: MHF`	Many hits in TLB
`[Bit 16]: SL`	Second-level fault (fault occurred in second level of page table)
`[Bit 30]: SS`	Stalled status
`[Bit 31]: MULTI`	Multiple faults

The TF, APF, and SL flags indicate normal operation, whereas the TLBMF, HTWDEEF, HTWSEEF, and MHF flags indicate that there is an issue.

IOMMU page table

The IOMMU page table dump provides a faulting address from the FAR and the register dump in the kernel log. This faulting address represents the virtual address, and you can acquire the corresponding physical address from the page table. From the page table dump, you can identify whether the requested address is mapped or not mapped. Each IOMMU domain has a page table, and the dump includes page tables for each of the domains. Currently, there are six domains. The following is the sample dump of the Domain: 2 page table.

Domain: 2 [L2 cache redirect for page tables is OFF]

0x00000000--0x0001ffff [0x00020000] [UNMAPPED]

0x00020000--0x01807fff [0x017e8000] A:0x82a8e000--0x84275fff [0x017e8000] [R/W][4K]

0x01808000--0x01939fff [0x00132000] A:0xf0c24000--0xf0d55fff [0x00132000] [R/W][4K]

0x0193a000--0x0199ffff [0x00066000] A:0xf13fa000--0xf145ffff [0x00066000] [R/W][4K]

0x019a0000--0x01e85fff [0x004e6000] A:0xf966e000--0xf9b53fff [0x004e6000] [R/W][4K]

0x01e86000--0x01ffffff [0x0017a000] [UNMAPPED]

0x02000000--0x02feffff [0x00ff0000] A:0xf5a22000--0xf6a11fff [0x00ff0000] [R/W][4K]

In this example, the first column represents the virtual address, the second column represents the number of bytes in the corresponding region of contiguous physical addresses, and the third column represents the physical addresses. The permissions are also mentioned for each of these regions.

Memory map

For more information about the memory map, see the latest Release Notes.

Stack corruption

Wrong coding logic accesses memory locations in the stack, leading to changes in values at those memory locations, causing stack corruption. Stack corruption can occur in the following ways:

Due to bad code logic, the program consumes all the stack memory and writes memory beyond the stack boundaries, resulting in a stack overflow.
Accessing an array that’s out of bounds.
An undefined or freed pointer that points at a stack address.
Corrupted return address of a caller function.

To identify the stack corruption issues, enable the following kernel configuration options:

CONFIG_STACKPROTECTOR
CONFIG_STACKPROTECTOR_STRONG

The Kernel address sanitizer (KASAN) utility also helps in identifying some stack corruption issues.

Introduction

Architecture

Key Documents

Technologies

System

Release Notes

/proc file system

/proc/meminfo

/proc/vmstat

/proc/iomem

/proc/vmallocinfo

memblock interface

/sys/kernel/debug/memblock/memory

/sys/kernel/debug/memblock/reserved

Debug memory leak issues

Identify memory corruption issues

Out of memory

IOMMU page fault

IOMMU page table

Memory map

Stack corruption

​/proc file system

​/proc/meminfo

​/proc/vmstat

​/proc/iomem

​/proc/vmallocinfo

​memblock interface

​/sys/kernel/debug/memblock/memory

​/sys/kernel/debug/memblock/reserved

​Debug memory leak issues

​Identify memory corruption issues

​Out of memory

​IOMMU page fault

​IOMMU page table

​Memory map

​Stack corruption

/proc file system

/proc/meminfo

/proc/vmstat

/proc/iomem

/proc/vmallocinfo

memblock interface

/sys/kernel/debug/memblock/memory

/sys/kernel/debug/memblock/reserved

Debug memory leak issues

Identify memory corruption issues

Out of memory

IOMMU page fault

IOMMU page table

Memory map

Stack corruption