> ## Documentation Index > Fetch the complete documentation index at: https://dragonwingdocs.qualcomm.com/llms.txt > Use this file to discover all available pages before exploring further. # Kernel probes and tracing The Linux kernel provides several dynamic instrumentation mechanisms that let you insert probes, collect traces, and measure timing without patching or recompiling the kernel. This page covers kprobes for code-path breakpoints, ftrace for function-level profiling, and MMIO traces for debugging driver register access failures. ## Kprobes Kprobes lets you break into any kernel instruction address and run a custom handler when execution reaches that point. The technique is non-disruptive — the running kernel is not stopped and other CPUs continue executing. ### How kprobes works 1. You register a probe on a symbol or address. 2. The kernel replaces the target instruction with a trap. 3. When the trap fires, the kprobe handler runs in the same context as the interrupted code. 4. The kernel restores the original instruction and continues execution. Kprobes are particularly useful for: * Tracing scheduler events (`schedule()`, `try_to_wake_up()`) * Counting how often a slow code path is taken * Capturing call arguments without adding `printk` and rebuilding ### Kconfig ```text theme={null} CONFIG_KPROBES=y CONFIG_KPROBE_EVENTS=y # enables kprobe trace events via tracefs ``` Verify on target: ```bash theme={null} zcat /proc/config.gz | grep CONFIG_KPROBE ``` ### Kprobe trace events via tracefs The easiest way to use kprobes is through the tracefs `kprobe_events` interface: ```bash theme={null} # Register a kprobe on qcom_geni_serial_probe echo "p:qcom_probe qcom_geni_serial_probe" > /sys/kernel/tracing/kprobe_events # Enable the event echo 1 > /sys/kernel/tracing/events/kprobes/qcom_probe/enable # Start tracing echo 1 > /sys/kernel/tracing/tracing_on # ... trigger the probe by loading the driver or booting ... # Read results cat /sys/kernel/tracing/trace # Clean up echo 0 > /sys/kernel/tracing/tracing_on echo "-:qcom_probe" >> /sys/kernel/tracing/kprobe_events ``` For full kprobes documentation, see [Kernel Probes (kprobes)](https://docs.kernel.org/trace/kprobes.html). ## Ftrace Ftrace is the kernel's built-in function tracer. It can record every function call in the kernel, trace specific subsystems, or profile latency-sensitive code paths. Results are read from the tracefs interface (`/sys/kernel/tracing/`). ### Function tracer The `function` tracer records the name and CPU of every kernel function called: ```bash theme={null} echo function > /sys/kernel/tracing/current_tracer echo 1 > /sys/kernel/tracing/tracing_on sleep 1 echo 0 > /sys/kernel/tracing/tracing_on cat /sys/kernel/tracing/trace | head -30 ``` ### Function graph tracer The `function_graph` tracer records entry and exit of each function, including execution time. Useful for identifying slow code paths: ```bash theme={null} echo function_graph > /sys/kernel/tracing/current_tracer echo 1 > /sys/kernel/tracing/tracing_on # ... trigger the code path ... echo 0 > /sys/kernel/tracing/tracing_on cat /sys/kernel/tracing/trace | grep -E "qcom|remoteproc" | head -20 ``` Sample output: ```text theme={null} 2) 0.432 us | qcom_geni_serial_isr(); 2) 1.201 us | handle_irq_event_percpu(); 2) 3.567 us | handle_fasteoi_irq(); ``` ### Trace boot initcalls Add to the kernel command line to trace all initcalls at boot: ```text theme={null} initcall_debug ``` Parse results with: ```bash theme={null} dmesg | grep "initcall.*after" \ | sed 's/.*initcall \(.*\) returned.* after \([0-9]*\) usecs/\2 \1/' \ | sort -rn | head -20 ``` For full ftrace documentation, see [Function Tracer](https://docs.kernel.org/trace/ftrace.html). ## MMIO trace events Memory-mapped I/O (MMIO) traces record every register read and write performed by the kernel, using `__raw_{read,write}{b,l,w,q}` accessors. They are essential for diagnosing the following crash categories on Qualcomm® SoCs: **Table: MMIO crash scenarios** | **Scenario** | **Explanation** | **Typical symptom** | | :----------------------- | :----------------------------------------------------------------------------------- | :------------------------------------- | | Unclocked access | Register access attempted before the clock to that block is enabled | Instant reboot / SError | | Protected register space | Register region requires EL3 privilege; access from EL1/EL2 is rejected by TrustZone | Synchronous exception / NoC error | | xPU violation | Memory Protection Unit blocks the access from this bus master | Interconnect hang or synchronous abort | ### Enable MMIO traces Kconfig: ```text theme={null} CONFIG_TRACE_MMIO_ACCESS=y ``` At runtime, enable `rwmmio` trace events via tracefs: ```bash theme={null} # List available rwmmio events cat /sys/kernel/tracing/available_events | grep rwmmio # Enable all rwmmio events cat /sys/kernel/tracing/available_events | grep rwmmio >> /sys/kernel/tracing/set_event # Enable tracing echo 1 > /sys/kernel/tracing/tracing_on # ... reproduce the crash or trigger the driver ... # Read the trace echo 0 > /sys/kernel/tracing/tracing_on cat /sys/kernel/tracing/trace ``` Sample output: ```text theme={null} rwmmio_read: gic_peek_irq+0xd0/0xd8 readl addr=0xffff800010040104 rwmmio_write: gic_poke_irq+0xe4/0xf0 writel addr=0xffff800010040184 rwmmio_read: gic_do_wait_for_rwp+0x54/0x90 readl addr=0xffff800010040000 rwmmio_write: gic_set_affinity+0x1bc/0x1e8 writeq addr=0xffff800010046130 ``` Each line identifies the calling function (with offset), the access type (`readl`/`writel`/`writeq`), and the physical address. Cross-reference the address against the SoC Technical Reference Manual to identify the register block involved in a crash. ### Dump MMIO trace on console at crash To capture MMIO traces even when the system crashes before you can read tracefs, enable console dumping of the ftrace buffer: ```bash theme={null} echo 1 > /proc/sys/kernel/ftrace_dump_on_oops ``` The last trace entries are then included in the kernel panic output on the serial console.