Skip to main content
The Linux kernel provides several dynamic instrumentation mechanisms that let you insert probes, collect traces, and measure timing without patching or recompiling the kernel. This page covers kprobes for code-path breakpoints, ftrace for function-level profiling, and MMIO traces for debugging driver register access failures.

​
Kprobes

Kprobes lets you break into any kernel instruction address and run a custom handler when execution reaches that point. The technique is non-disruptive — the running kernel is not stopped and other CPUs continue executing.

​
How kprobes works

  1. You register a probe on a symbol or address.
  2. The kernel replaces the target instruction with a trap.
  3. When the trap fires, the kprobe handler runs in the same context as the interrupted code.
  4. The kernel restores the original instruction and continues execution.
Kprobes are particularly useful for:
  • Tracing scheduler events (schedule(), try_to_wake_up())
  • Counting how often a slow code path is taken
  • Capturing call arguments without adding printk and rebuilding

​
Kconfig

CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y   # enables kprobe trace events via tracefs
Verify on target: 
zcat /proc/config.gz | grep CONFIG_KPROBE

​
Kprobe trace events via tracefs

The easiest way to use kprobes is through the tracefs kprobe_events interface: 
# Register a kprobe on qcom_geni_serial_probe
echo "p:qcom_probe qcom_geni_serial_probe" > /sys/kernel/tracing/kprobe_events

# Enable the event
echo 1 > /sys/kernel/tracing/events/kprobes/qcom_probe/enable

# Start tracing
echo 1 > /sys/kernel/tracing/tracing_on

# ... trigger the probe by loading the driver or booting ...

# Read results
cat /sys/kernel/tracing/trace

# Clean up
echo 0 > /sys/kernel/tracing/tracing_on
echo "-:qcom_probe" >> /sys/kernel/tracing/kprobe_events
For full kprobes documentation, see Kernel Probes (kprobes).

​
Ftrace

Ftrace is the kernel’s built-in function tracer. It can record every function call in the kernel, trace specific subsystems, or profile latency-sensitive code paths. Results are read from the tracefs interface (/sys/kernel/tracing/).

​
Function tracer

The function tracer records the name and CPU of every kernel function called: 
echo function > /sys/kernel/tracing/current_tracer
echo 1 > /sys/kernel/tracing/tracing_on
sleep 1
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace | head -30

​
Function graph tracer

The function_graph tracer records entry and exit of each function, including execution time. Useful for identifying slow code paths: 
echo function_graph > /sys/kernel/tracing/current_tracer
echo 1 > /sys/kernel/tracing/tracing_on
# ... trigger the code path ...
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace | grep -E "qcom|remoteproc" | head -20
Sample output: 
 2)   0.432 us    |          qcom_geni_serial_isr();
 2)   1.201 us    |        handle_irq_event_percpu();
 2)   3.567 us    |      handle_fasteoi_irq();

​
Trace boot initcalls

Add to the kernel command line to trace all initcalls at boot: 
initcall_debug
Parse results with: 
dmesg | grep "initcall.*after" \
      | sed 's/.*initcall \(.*\) returned.* after \([0-9]*\) usecs/\2 \1/' \
      | sort -rn | head -20
For full ftrace documentation, see Function Tracer.

​
MMIO trace events

Memory-mapped I/O (MMIO) traces record every register read and write performed by the kernel, using __raw_{read,write}{b,l,w,q} accessors. They are essential for diagnosing the following crash categories on Qualcomm® SoCs: Table: MMIO crash scenarios
ScenarioExplanationTypical symptom
Unclocked accessRegister access attempted before the clock to that block is enabledInstant reboot / SError
Protected register spaceRegister region requires EL3 privilege; access from EL1/EL2 is rejected by TrustZoneSynchronous exception / NoC error
xPU violationMemory Protection Unit blocks the access from this bus masterInterconnect hang or synchronous abort

​
Enable MMIO traces

Kconfig: 
CONFIG_TRACE_MMIO_ACCESS=y
At runtime, enable rwmmio trace events via tracefs: 
# List available rwmmio events
cat /sys/kernel/tracing/available_events | grep rwmmio

# Enable all rwmmio events
cat /sys/kernel/tracing/available_events | grep rwmmio >> /sys/kernel/tracing/set_event

# Enable tracing
echo 1 > /sys/kernel/tracing/tracing_on

# ... reproduce the crash or trigger the driver ...

# Read the trace
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace
Sample output: 
rwmmio_read:  gic_peek_irq+0xd0/0xd8       readl  addr=0xffff800010040104
rwmmio_write: gic_poke_irq+0xe4/0xf0       writel addr=0xffff800010040184
rwmmio_read:  gic_do_wait_for_rwp+0x54/0x90 readl  addr=0xffff800010040000
rwmmio_write: gic_set_affinity+0x1bc/0x1e8  writeq addr=0xffff800010046130
Each line identifies the calling function (with offset), the access type (readl/writel/writeq), and the physical address. Cross-reference the address against the SoC Technical Reference Manual to identify the register block involved in a crash.

​
Dump MMIO trace on console at crash

To capture MMIO traces even when the system crashes before you can read tracefs, enable console dumping of the ftrace buffer: 
echo 1 > /proc/sys/kernel/ftrace_dump_on_oops
The last trace entries are then included in the kernel panic output on the serial console.