Kernel probes and tracing

The Linux kernel provides several dynamic instrumentation mechanisms that let you insert probes, collect traces, and measure timing without patching or recompiling the kernel. This page covers kprobes for code-path breakpoints, ftrace for function-level profiling, and MMIO traces for debugging driver register access failures.

Kprobes

Kprobes lets you break into any kernel instruction address and run a custom handler when execution reaches that point. The technique is non-disruptive — the running kernel is not stopped and other CPUs continue executing.

How kprobes works

You register a probe on a symbol or address.
The kernel replaces the target instruction with a trap.
When the trap fires, the kprobe handler runs in the same context as the interrupted code.
The kernel restores the original instruction and continues execution.

Kprobes are particularly useful for:

Tracing scheduler events (schedule(), try_to_wake_up())
Counting how often a slow code path is taken
Capturing call arguments without adding printk and rebuilding

Kconfig

CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y   # enables kprobe trace events via tracefs

Verify on target:

zcat /proc/config.gz | grep CONFIG_KPROBE

Kprobe trace events via tracefs

The easiest way to use kprobes is through the tracefs kprobe_events interface:

# Register a kprobe on qcom_geni_serial_probe
echo "p:qcom_probe qcom_geni_serial_probe" > /sys/kernel/tracing/kprobe_events

# Enable the event
echo 1 > /sys/kernel/tracing/events/kprobes/qcom_probe/enable

# Start tracing
echo 1 > /sys/kernel/tracing/tracing_on

# ... trigger the probe by loading the driver or booting ...

# Read results
cat /sys/kernel/tracing/trace

# Clean up
echo 0 > /sys/kernel/tracing/tracing_on
echo "-:qcom_probe" >> /sys/kernel/tracing/kprobe_events

For full kprobes documentation, see Kernel Probes (kprobes).

Ftrace

Ftrace is the kernel’s built-in function tracer. It can record every function call in the kernel, trace specific subsystems, or profile latency-sensitive code paths. Results are read from the tracefs interface (/sys/kernel/tracing/).

Function tracer

The function tracer records the name and CPU of every kernel function called:

echo function > /sys/kernel/tracing/current_tracer
echo 1 > /sys/kernel/tracing/tracing_on
sleep 1
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace | head -30

Function graph tracer

The function_graph tracer records entry and exit of each function, including execution time. Useful for identifying slow code paths:

echo function_graph > /sys/kernel/tracing/current_tracer
echo 1 > /sys/kernel/tracing/tracing_on
# ... trigger the code path ...
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace | grep -E "qcom|remoteproc" | head -20

Sample output:

 2)   0.432 us    |          qcom_geni_serial_isr();
 2)   1.201 us    |        handle_irq_event_percpu();
 2)   3.567 us    |      handle_fasteoi_irq();

Trace boot initcalls

Add to the kernel command line to trace all initcalls at boot:

initcall_debug

Parse results with:

dmesg | grep "initcall.*after" \
      | sed 's/.*initcall \(.*\) returned.* after \([0-9]*\) usecs/\2 \1/' \
      | sort -rn | head -20

For full ftrace documentation, see Function Tracer.

MMIO trace events

Memory-mapped I/O (MMIO) traces record every register read and write performed by the kernel, using __raw_{read,write}{b,l,w,q} accessors. They are essential for diagnosing the following crash categories on Qualcomm^® SoCs: Table: MMIO crash scenarios

Scenario	Explanation	Typical symptom
Unclocked access	Register access attempted before the clock to that block is enabled	Instant reboot / SError
Protected register space	Register region requires EL3 privilege; access from EL1/EL2 is rejected by TrustZone	Synchronous exception / NoC error
xPU violation	Memory Protection Unit blocks the access from this bus master	Interconnect hang or synchronous abort

Enable MMIO traces

Kconfig:

CONFIG_TRACE_MMIO_ACCESS=y

At runtime, enable rwmmio trace events via tracefs:

# List available rwmmio events
cat /sys/kernel/tracing/available_events | grep rwmmio

# Enable all rwmmio events
cat /sys/kernel/tracing/available_events | grep rwmmio >> /sys/kernel/tracing/set_event

# Enable tracing
echo 1 > /sys/kernel/tracing/tracing_on

# ... reproduce the crash or trigger the driver ...

# Read the trace
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace

Sample output:

rwmmio_read:  gic_peek_irq+0xd0/0xd8       readl  addr=0xffff800010040104
rwmmio_write: gic_poke_irq+0xe4/0xf0       writel addr=0xffff800010040184
rwmmio_read:  gic_do_wait_for_rwp+0x54/0x90 readl  addr=0xffff800010040000
rwmmio_write: gic_set_affinity+0x1bc/0x1e8  writeq addr=0xffff800010046130

Each line identifies the calling function (with offset), the access type (readl/writel/writeq), and the physical address. Cross-reference the address against the SoC Technical Reference Manual to identify the register block involved in a crash.

Dump MMIO trace on console at crash

To capture MMIO traces even when the system crashes before you can read tracefs, enable console dumping of the ftrace buffer:

echo 1 > /proc/sys/kernel/ftrace_dump_on_oops

The last trace entries are then included in the kernel panic output on the serial console.

Introduction

Architecture

Key Documents

Technologies

System

Release Notes

Kprobes

How kprobes works

Kconfig

Kprobe trace events via tracefs

Ftrace

Function tracer

Function graph tracer

Trace boot initcalls

MMIO trace events

Enable MMIO traces

Dump MMIO trace on console at crash

​Kprobes

​How kprobes works

​Kconfig

​Kprobe trace events via tracefs

​Ftrace

​Function tracer

​Function graph tracer

​Trace boot initcalls

​MMIO trace events

​Enable MMIO traces

​Dump MMIO trace on console at crash

Kprobes

How kprobes works

Kconfig

Kprobe trace events via tracefs

Ftrace

Function tracer

Function graph tracer

Trace boot initcalls

MMIO trace events

Enable MMIO traces

Dump MMIO trace on console at crash