PCIe - Qualcomm Dragonwing Documentation

PCIe uses a bidirectional connection to send and receive information at the same time, as shown in the following figure.

Figure: PCIe device connection link

The path between the devices is called a Link. It is made up of one or more transmit and receive pairs. One pair of the Link is called a Lane. The PCIe device connection in Qualcomm Linux devices supports 16 lanes. The number of lanes or the Link width is x16. The following table lists the types of PCIe connections for devices.

Table : PCIe connections

PCIe type	Description
Root complex (RC)	Connects the CPU to the PCIe topology
Switch	Connects more than 2 ports and acts as a packet router
Bridge	Connects different buses: for example, PCIe to PCIe, or PCIe to peripheral component interconnect (PCI)
Endpoint (EP)	Resides at the bottom of the PCIe topology tree structure and has only an upstream port
Legacy endpoint	Uses older PCI bus operations to support backward compatibility

PCIe host mode enumeration feature

When a system first powers up, the configuration software running on the system host processor is aware of the existence of only Bus 0 (if PCIe is supported). The software is not aware of the bus topology or any device connected to the bus. The enumeration process discovers the various buses, devices, and functions present in the system. When enumeration is complete, each bus in the system is numbered as follows:

The primary bus number indicates the bus that directly connects to the primary interface of the bridge (towards the root complex).
The secondary bus number indicates the bus that directly connects to the secondary interface of the bridge (away from the root complex).
The subordinate bus number indicates the highest numbered bus that exists on the downstream side of the bridge.

The BDF number uniquely identifies each device. The ID-based routing method of the transaction layer packet (TLP) uses this number. The PCIe host mode enumeration process involves the following:

Link training
Scanning for devices on the bus
Registration

For more information on the PCIe device initialization, enumeration process, see https://www.kernel.org/doc/html/latest/PCI/index.html.

PCIe layered architecture

The following figure shows the PCIe software architecture.

Figure : PCIe software architecture

The following figure shows the layered architecture model of PCIe.

Figure : PCIe layered architecture

The transmission units exchanged are as follows.

Ordered set between the physical layer entities.
Data link layer packet (DLLP) between data link layer entities.
Transaction layer packet (TLP) between transaction layer entities.

The following table lists the three layers, with the respective functions, in the PCIe architecture.

Table : Layers in PCIe architecture

Layer	Features
Physical layer	Logical sub-block: Link training, initialization, and maintenance.
Physical layer	Physical sub-block: 8b/10b encoding and decoding, and parallel-to-serial and serial-to-parallel conversion.
Data link layer	Assembly and disassembly of the DLLP packet.
	Generation and validation of the link layer CRC (LCRC).
	Acknowledgment and no acknowledgment protocol (replay of TLPs in error).
Transaction layer	Assembly and disassembly of the TLP packet.
	Generation and validation of end-to-end CRC (ECRC).
	Flow control receives entity advertises for the available receive buffer size information using DLLPs.
	Quality of service (QoS): traffic class (TC) to virtual channel (VC) mapping.
	Transaction ordering: implements the transaction ordering rule within a VC.

PCI defines a dedicated block of configuration address space for each function as shown in the following figure. The software determines the presence of a function, configures it, and checks and controls its status.

Figure : PCIe configuration address space

PCIe software driver configuration

The PCIe controller driver initializes the PCIe resources and performs link training. After successful training, the controller driver calls the PCIe framework for link enumeration, such as endpoint discovery, identifying the client driver, and probing those drivers. For more information about the PCIe framework and client driver PCIe registrations, see https://www.kernel.org/doc/html/latest/PCI/index.html.

Link training

Link training comprises the following operations:

The PCIe driver pcie-qcom.c file at https://github.com/torvalds/linux/blob/master/drivers/pci/controller/dwc/pcie-qcom.c obtains the required resources such as regulators, clocks, from the device tree.
The PCIe driver calls Synopsys DesignWare® Core host driver pcie-designware-host.c file at https://github.com/torvalds/linux/blob/master/drivers/pci/controller/dwc/pcie-designware-host.c to initialize the root complex.
The Synopsys DesignWare Core driver performs all necessary initializations.
The Synopsys DesignWare Core driver calls a function pointer to perform host initialization.
The Qualcomm PCIe driver performs PHY power-on, enables all regulators, clocks.
The Synopsys DesignWare Core driver starts the link training by calling the function pointer to start the link.

Hardware initialization

The driver initializes and configures the PCIe hardware block and performs link training. The initialization occurs after the platform _probe() driver function is called.

static int qcom_pcie_probe(struct platform_device *pdev)
{

        // get PCIe resources

        ret = dw_pcie_host_init(pp);  // call DWC framework to host intialisation like MSI, MSIx and controller init 
        if (ret) {
                dev_err(dev, "cannot initialize host\n");
                goto err_phy_exit;
        }

};

static const struct of_device_id qcom_pcie_match[] = {
        { .compatible = "qcom,pcie-sc7280", .data = &cfg_1_9_0 },
        {}

};

static struct platform_driver qcom_pcie_driver = {
        .probe = qcom_pcie_probe,
        .driver = {
                .name = "qcom-pcie",
                .suppress_bind_attrs = true,
                .of_match_table = qcom_pcie_match,
        },
};

When the Synopsys DesignWare Core driver is initialized, it also initializes MSI, MSIx, and controllers. The driver calls a Qualcomm function pointer to start the link training.

static int qcom_pcie_start_link(struct dw_pcie *pci)
{
        struct qcom_pcie *pcie = to_qcom_pcie(pci);

        /* Enable Link Training state machine */
        if (pcie->cfg->ops->ltssm_enable)
                pcie->cfg->ops->ltssm_enable(pcie);

        return 0;
}

static int qcom_pcie_link_up(struct dw_pcie *pci)
{
        u16 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
        u16 val = readw(pci->dbi_base + offset + PCI_EXP_LNKSTA);

        return !!(val & PCI_EXP_LNKSTA_DLLLA); // checks the link is up or not
}
static const struct dw_pcie_ops dw_pcie_ops = {
        .link_up = qcom_pcie_link_up,
        .start_link = qcom_pcie_start_link,
};

The Synopsys DesignWare Core driver waits for the link to be active to enumerate the PCI framework. The Qualcomm PCIe driver enables only the link training.

Note: The PCIE_0 root complex instance is enabled by default for the WLAN EP connection.

Enable PCIe endpoint (EP) mode for Dragonwing IQ-9075 (Lemans)

Each PCIe controller instance on Dragonwing IQ-9075 (Lemans) can operate in either root complex (RC) or endpoint (EP) mode, but not both at the same time. The device tree defines a separate node for each mode at the same base address, and only one of the two may be enabled per PCIe instance.

Note: Monaco also supports PCIe EP mode, but that support isn’t in this kernel tree yet and is planned for a future release.

Table : Lemans PCIe EP nodes

EP node	RC node (mutually exclusive)	Compatible	num-lanes	linux,pci-domain
pcie0_ep	pcie0	qcom,sa8775p-pcie-ep	2	0
pcie1_ep	pcie1	qcom,sa8775p-pcie-ep	4	1

Both pcie0_ep and pcie1_ep are defined in arch/arm64/boot/dts/qcom/lemans.dtsi with status = "disabled" by default, since the corresponding RC nodes (pcie0, pcie1) are enabled instead. The Qualcomm PCIe EP controller driver that binds to these nodes is drivers/pci/controller/dwc/pcie-qcom-ep.c (driver name qcom-pcie-ep).

PCIe EP mode kernel configuration

CONFIG_PCI_ENDPOINT and CONFIG_PCI_ENDPOINT_CONFIGFS are already enabled in the Lemans defconfig. CONFIG_PCIE_QCOM_EP, which builds the Qualcomm EP controller driver, is not enabled by default and must be turned on to bring up EP mode. CONFIG_PCI_EPF_MHI and its dependency CONFIG_MHI_BUS_EP also aren’t enabled by default and must be turned on to use the MHI endpoint function. To enable these configs, apply the following patch to the arch/arm64/configs/defconfig file.

QCLINUX: arm64: defconfig: qcom:
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 7293680..730122c 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -242,6 +242,7 @@
 CONFIG_PCIE_QCOM=y
+CONFIG_PCIE_QCOM_EP=y
 CONFIG_PCI_ENDPOINT=y
 CONFIG_PCI_ENDPOINT_CONFIGFS=y
 CONFIG_PCI_EPF_TEST=m
+CONFIG_PCI_EPF_MHI=m
+CONFIG_MHI_BUS_EP=m

Note: CONFIG_PCIE_QCOM_EP depends on CONFIG_PCI_ENDPOINT and selects CONFIG_PCIE_DW_EP and CONFIG_PCIE_QCOM_COMMON, defined in drivers/pci/controller/dwc/Kconfig. CONFIG_PCI_EPF_MHI, defined in drivers/pci/endpoint/functions/Kconfig, depends on CONFIG_MHI_BUS_EP, the MHI endpoint bus implementation in drivers/bus/mhi/ep/Kconfig.

Enable the EP node in the device tree

Both pcie1 (RC) and pcie1_ep (EP) are defined with labels in lemans.dtsi, and pcie1 defaults to status = "disabled" there. However, lemans-evk.dts re-enables pcie1 in RC mode for the onboard M.2 E-key connector (&pcie1 { ... status = "okay"; };). Since a PCIe instance can only be RC or EP at a given time, switching PCIE_1 to EP mode on the Lemans EVK means disabling that RC override and enabling pcie1_ep instead. Apply the following changes to arch/arm64/boot/dts/qcom/lemans-evk.dts:

diff --git a/arch/arm64/boot/dts/qcom/lemans-evk.dts b/arch/arm64/boot/dts/qcom/lemans-evk.dts
--- a/arch/arm64/boot/dts/qcom/lemans-evk.dts
+++ b/arch/arm64/boot/dts/qcom/lemans-evk.dts
@@ -856,13 +856,14 @@
 };
 
 &pcie1 {
-	perst-gpios = <&tlmm 4 GPIO_ACTIVE_LOW>;
-	wake-gpios = <&tlmm 5 GPIO_ACTIVE_HIGH>;
+	status = "disabled";
+};
 
-	pinctrl-0 = <&pcie1_default_state>;
+&pcie1_ep {
+	reset-gpios = <&tlmm 4 GPIO_ACTIVE_LOW>;
+	wake-gpios = <&tlmm 5 GPIO_ACTIVE_HIGH>;
+	pinctrl-0 = <&pcie1_default_state>;
 	pinctrl-names = "default";
 
 	status = "okay";
 };

pcie1_default_state (defined in the &tlmm node further down in lemans-evk.dts) sets the pin function, drive strength, and bias for GPIO 4 (PERST#) and GPIO 5 (WAKE#), but doesn’t fix their direction, so the EP node can reuse the same pinctrl group and physical GPIOs as RC mode. However, the signal direction on each GPIO reverses between modes:

In RC mode, pcie-qcom.c requests the PERST# GPIO as GPIOD_OUT_HIGH — the root complex drives PERST# to reset the downstream device. RC mode doesn’t drive a WAKE# GPIO.
In EP mode, pcie-qcom-ep.c requests reset-gpios as GPIOD_IN — the endpoint reads PERST# as an input driven by the connected root complex — and requests wake-gpios as GPIOD_OUT_LOW — the endpoint drives WAKE# as an output to the host.

The GPIO direction is set by the driver at probe time based on which mode’s binding it matches, not by the pinctrl group, so no additional device tree change is needed for the direction switch.

Note: The upstream qcom,pcie-ep.yaml binding requires reset-gpios, the GPIO wired to the PERST# input driven by the connected root complex. The pcie1_ep node in lemans.dtsi doesn’t define this property, since it’s board-specific.

The same procedure applies to PCIE_0: disable the pcie0 RC override and enable pcie0_ep instead, reusing the pcie0_default_state pinctrl group and its PERST#/WAKE# GPIOs.

Note: Disabling pcie0 also disables the M.2 E-key connector’s PCIe link (pcieport0), since that connector is wired to PCIE_0 in RC mode. Only switch PCIE_0 to EP mode if you don’t need the M.2 connector for a PCIe card.

phys, iommus, interconnects, power-domains, clocks, and resets are already populated for both EP nodes and don’t need to be modified for a standard bring-up.

PCIe EP driver initialization

qcom_pcie_ep_probe() in pcie-qcom-ep.c acquires the parf, dbi, mmio, and dma register regions, all five instance clocks, the core reset, the reset (PERST#) and optional wake (WAKE#) GPIOs, and the optional pciephy PHY, then calls the common DesignWare dw_pcie_ep_init() to initialize the endpoint core. Lemans matches the qcom,sa8775p-pcie-ep compatible string, which selects the cfg_1_34_0 configuration (HDMA support enabled, NO_SNOOP override, and the MHI RAM parity check disabled). A global IRQ thread (qcom_pcie_ep_global_irq_thread()) handles link-up and link-down events, bus master enable (BME), and PM turn-off/D-state notifications raised by the connected root complex.

Bind the MHI endpoint function using configfs

The pci-epf-mhi.c driver registers one device ID per SoC in pci_epf_mhi_ids[], each with its own header (vendor/device ID) and MHI channel configuration. Lemans matches the pci_epf_mhi_sa8775p entry, which sets device ID 0x0116, uses BAR_0, 32 MSI vectors, an MRU of 0x8000, and enables DMA-assisted transfers (MHI_EPF_USE_DMA). This entry also selects the standard mhi_v1_channels table (LOOPBACK, SAHARA, DIAG, SSR, QDSS, EFS, MBIM, QMI, IP-CTRL-1, IPCR, DUN, IP_SW0). The header and channel configuration are fixed by this entry, so no vendorid/deviceid configfs writes are needed. After the kernel boots with pcie1_ep enabled, use the PCI endpoint configfs interface to bind the MHI endpoint function to the qcom-pcie-ep controller. The function directory name must match the pci_epf_mhi_sa8775p entry so the correct device ID and channel table are probed.

mount -t configfs none /sys/kernel/config
mkdir -p /sys/kernel/config/pci_ep/functions/pci_epf_mhi_sa8775p/func1

ln -s /sys/kernel/config/pci_ep/functions/pci_epf_mhi_sa8775p/func1 \
      /sys/kernel/config/pci_ep/controllers/1c10000.pcie-ep/

echo 1 > /sys/kernel/config/pci_ep/controllers/1c10000.pcie-ep/start

Writing 1 to start asserts the endpoint’s readiness and lets the link train with the connected root complex. pci_epf_mhi_bind() looks up the mmio register region and doorbell interrupt from the parent qcom-pcie-ep platform device, and pci_epf_mhi_link_up() registers the MHI endpoint controller once the link comes up.

Note: Load mhi_bus_ep before pci_epf_mhi if they’re built as separate modules, since CONFIG_PCI_EPF_MHI depends on CONFIG_MHI_BUS_EP.

Verify MHI EP mode enumeration

On the host connected to the Lemans EP port, confirm the device enumerates as a PCI endpoint with the MHI device ID.

lspci -d 17cb:0116

Output:

0001:01:00.0 Unassigned class [ff00]: Qualcomm Device 0116

On the target, confirm the MHI endpoint channels are registered.

cat /sys/bus/mhi_ep/devices/*/channels

For more information about the configfs-based endpoint bring-up flow, see https://www.kernel.org/doc/html/latest/PCI/endpoint/pci-endpoint-cfs.html.

Enable QPS615 PCIe switch

This section describes how to enable a QPS615 PCIe switch in the Qualcomm Linux hardware SoCs. The QPS615 switch endpoint is supported on the PCIe1 instance. The following figure shows the QPS615 endpoint and connections.

Figure : QPS615 PCIe switch connection diagram

The Qualcomm PCIe driver documentation can be accessed at the following locations:

The following configurations are enabled by default to support the QPS615 switch. Disable the QPS615 switch default support, by reverting the code changes, to use it for a different PCIe endpoint. To enable PCIe-related configs, apply the following patch to the /arch/arm64/configs/qcom_addons.config file.

QCLINUX: arm64: defconfig: qcom:
diff --git a/arch/arm64/configs/qcom_defconfig b/arch/arm64/configs/qcom_defconfig
index 7293680..730122c 100644
--- a/arch/arm64/configs/qcom_defconfig
+++ b/arch/arm64/configs/qcom_defconfig
@@ -173,6 +173,7 @@
 CONFIG_HOTPLUG_PCI=y
 CONFIG_HOTPLUG_PCI_ACPI=y
 CONFIG_PCI_HOST_GENERIC=y
+CONFIG_PCIE_DW_PLAT_HOST=y
 CONFIG_PCIE_QCOM=y
 CONFIG_PCI_ENDPOINT=y
 CONFIG_PCI_ENDPOINT_CONFIGFS=y
@@ -793,3 +794,5 @@
 CONFIG_CORESIGHT_TPDM=m
 CONFIG_CORESIGHT_DUMMY=m
 CONFIG_MEMTEST=y
+CONFIG_PCIEASPM=y
+CONFIG_PCIEASPM_POWER_SUPERSAVE=y

Message signaled interrupt (MSI)

The current MSI mapping doesn’t have all the vectors. The Qualcomm Linux hardware SoCs support eight vectors. Each vector in turn supports 32 MSIs. Therefore, the total MSIs supported are 256. For information about adding all the MSI groups supported for this PCIe instance, see https://lore.kernel.org/linux-arm-msm/f1168212-bc6e-4570-869c-2870d6f248ad@linaro.org/T/.

Sample PCIe kernel driver log

The following is a sample PCIe kernel driver log from the QPS615 device enumeration.

Ethernet interfaces supported through QPS615 PCIe switch

The QPS615 PCIe switch enables Ethernet connectivity for the device. When the device loads the QPS615 driver and establishes the PCIe link, it automatically activates the supported Ethernet interfaces, by default, during device startup. To customize the default configuration or enable extra MAC/PHY components beyond Qualcomm’s hardware setup, see the Bring up Ethernet section in the Qualcomm Linux Ethernet guide.

Table : Supported Ethernet interfaces

Interface type	Speed	Connector type	Description
QEP PHY (SGMII)	2.5 GbE	IX/RJ45 connector (QEP8121)	Enabled by default.
AQR PHY (USXGMII)	10 GbE	IX/RJ45 connector (AQR113C)	The 10 Gigabit Ethernet AQR PHY is an optional component in the RB3 Gen 2 Development Kit, and the Evaluation Kits (EVK) for Dragonwing IQ-8275 and Dragonwing IQ-9075. It’s validated on AQR113C.

Bring up alternate hardware components

You can attach MAC/PHY components other than the hardware configuration provided by Qualcomm and bring them up. To replace QPS615 with other PCIe based MAC/PHY, see Enable QPS615 PCIe switch.

Note: You must obtain the MAC/PHY driver and firmware from the respective vendor. Qualcomm isn’t responsible for these configuration changes.

Enable USB interface through PCIe switch

This section provides instructions on how to activate a USB interface through a PCIe switch in the Qualcomm Linux hardware SoCs. The PCIE1 instance is connected to the endpoint of the QPS615 switch, and the downstream port of the QPS615 is connected to the PCIe to USB endpoint. For the PCIe to USB endpoint connections using the QPS615, see the mainboard and interposer block diagram at https://docs.qualcomm.com/bundle/publicresource/topics/80-80021-251/rb3_hardware_overview.html.

Note: Dragonwing IQ-9075 PCIe software doesn’t support USB.

Power to usb hub is enabled through pwrctrl driver, Please refer below upstream links for these details. https://elixir.bootlin.com/linux/v7.1/source/arch/arm64/boot/dts/qcom/qcs6490-rb3gen2.dts#L896 https://elixir.bootlin.com/linux/v7.1/source/Documentation/devicetree/bindings/usb/renesas,upd720201-pci.yaml https://elixir.bootlin.com/linux/v7.1/source/drivers/pci/pwrctrl/generic.c

### **Avoid early handoff in PCIe to USB controller**

Ensure that `usb_early-handoff` is skipped for the PCIe to USB controller since the firmware isn't loaded. This precaution is necessary to prevent any adverse impact on the bootup duration.

```text
diff --git a/drivers/usb/host/pci-quirks.c b/drivers/usb/host/pci-quirks.c
index 2665832..c3d488f 100644
--- a/drivers/usb/host/pci-quirks.c
+++ b/drivers/usb/host/pci-quirks.c
@@ -1260,6 +1260,11 @@
 			return;
 	}
 
+	/* Skip handoff for Renesas PCI USB controller on QCOM SOC */
+	if ((pdev->vendor == PCI_VENDOR_ID_RENESAS) &&
+		(pcie_find_root_port(pdev)->vendor == PCI_VENDOR_ID_QCOM))
+		return;
+
 	if (pdev->class != PCI_CLASS_SERIAL_USB_UHCI &&
 			pdev->class != PCI_CLASS_SERIAL_USB_OHCI &&
 			pdev->class != PCI_CLASS_SERIAL_USB_EHCI &&

Download PCIe to USB controller firmware

To download the firmware from https://www.renesas.com/us/en/products/interface/usb-switches-hubs/upd720201-usb-30-host-controller#design_development, register and log in to https://www.renesas.com/. Rename the downloaded firmware file to renesas_usb_fw.mem.

Note: To prevent command failures, update the software as described in the Set up the device section before updating the Renesas firmware.

Option 1 (Ubuntu): Push the firmware files using SCP or similar tools. For example,

scp renesas_usb_fw.mem root@10.92.175.138:/lib/firmware

Option 2 (Yocto):

Connect device to Host PC via USB cable for adb.

Push firmware to device.

mount -o remount,rw /usr
adb push renesas_usb_fw.mem /lib/firmware

To activate the firmware do either of the following options.
- Option A: Reboot target for USB type A ports.
- Option B: Manually bind the Renesas xHCI driver.
  echo "0001:04:00.0" > /sys/bus/pci/drivers/xhci-pci-renesas/bind
Verify firmware enumeration.
dmesg

PCIe kernel driver logs for PCIe to USB device enumeration reference

You can run the following commands to view the device information:

To display device information in USB, run the following command.

lsusb

The following message is displayed.

Bus 002 Device 003: ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
Bus 002 Device 002: ID 05e3:0625 Genesys Logic, Inc. USB3.2 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

To display device information in PCIe, run the following command.

lspci

The following message is displayed.

00:00.0 PCI bridge: Qualcomm Device 010b
01:00.0 PCI bridge: Toshiba Corporation Device 0623
02:01.0 PCI bridge: Toshiba Corporation Device 0623
02:02.0 PCI bridge: Toshiba Corporation Device 0623
02:03.0 PCI bridge: Toshiba Corporation Device 0623
04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
05:00.0 Ethernet controller: Toshiba Corporation Device 0220
05:00.1 Ethernet controller: Toshiba Corporation Device 0220

Connect QPS615 switches in cascade

Connect the QPS615 switches in cascade to enable additional Ethernet, PCIe, and USB ports.

Note: This feature is supported only in QCS6490.

The following image shows the two QPS615 switches connected in cascade to PCIE1 and another QPS615 switch connected to PCIE2.

QPS615 cascade connection — Figure : QPS615 switches in cascade connection

To initiate the link training and enumeration for all endpoints, do the following:

To reset the QPS615 switch, toggle the RESX GPIOs for both QPS615 #1 and QPS615 #2.
To control the endpoint reset, trigger PERST. Both the switches share the PERST.

The devices can be attached directly to the QPS615 switch. The PCIe RC0 is attached to the WLAN endpoint. The following differences are due to the PCIe node hierarchy.

Switch-attached devices:
- The QPS615 PCIe tree node hierarchy is statically fixed.
- All nodes for switching USP and DSP ports are created during PCIe initialization.
- One of the switch DSP ports represents WLAN.
- If you disable the WLAN node, it disables the WLAN device, but the PCIe downstream port remains enabled and returns a default maximum link width.
Directly-attached WLAN devices:
- Disables only a single node.
- Returns the Invalid argument, when WLAN is disabled.

Enable NVMe through PCIe interface

This section describes how to enable NVMe using PCIe for storage expansion. To verify if NVMe is connected over a PCIe interface, do the following:

To display PCIe device information, run the following command.

lspci

Output:

00:00.0 PCI bridge: Qualcomm Device 0115
01:00.0 Network controller: Qualcomm QCNFA765 Wireless Network Adapter (rev 01)
00:00.0 PCI bridge: Qualcomm Device 0115
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO

Locate the PCIe logs.

dmesg | grep pcie

Output:

[    0.000000] Kernel command line:  root=/dev/disk/by-partlabel/system rw rootwait console=ttyMSM0,115200n8 
pcie_pme=nomsi net.ifnames=0 pci=noaer kpti=off kasan=off kasan.stacktrace=off swiotlb=128 earlycon reboot=panic_warm page_owner=on qcom_scm.download_mode=1 slub_debug=FZP,
zs_handle,zspage;FZPU mitigations=auto kernel.sched_pelt_multiplier=4 rcupdate.rcu_expedited=1 rcu_nocbs=0-7 no-steal-acc vfio_iommu_type1.allow_unsafe_interrupts=1 fw_devlink.strict=1
	[    3.842071] qcom-pcie 1c00000.pci: supply vdda not found, using dummy regulator
	[    3.845339] qcom-pcie 1c10000.pci: supply vdda not found, using dummy regulator
	[    3.845428] qcom-pcie 1c10000.pci: supply vddpe-3v3 not found, using dummy regulator
	[    3.845508] qcom-pcie 1c10000.pci: host bridge /pci@1c10000 ranges:
	[    3.845519] qcom-pcie 1c10000.pci:       IO 0x0060200000..0x00602fffff -> 0x0000000000
	[    3.845526] qcom-pcie 1c10000.pci:      MEM 0x0060300000..0x007fffffff -> 0x0060300000
	[    3.848154] qcom-pcie 1c00000.pci: supply vddpe-3v3 not found, using dummy regulator
	[    3.854556] qcom-pcie 1c00000.pci: host bridge /pci@1c00000 ranges:
	[    3.860574] qcom-pcie 1c00000.pci:       IO 0x0040200000..0x00402fffff -> 0x0000000000
	[    3.867362] qcom-pcie 1c00000.pci:      MEM 0x0040300000..0x005fffffff -> 0x0040300000
	[    3.963275] qcom-pcie 1c10000.pci: iATU: unroll T, 32 ob, 8 ib, align 4K, limit 1024G
	[    3.990385] qcom-pcie 1c00000.pci: iATU: unroll T, 32 ob, 8 ib, align 4K, limit 1024G
	[    4.072091] qcom-pcie 1c10000.pci: PCIe Gen.4 x2 link up
	[    4.080241] qcom-pcie 1c10000.pci: PCI host bridge to bus 0001:00
	[    4.291824] pcieport 0001:00:00.0: Adding to iommu group 9
	[    4.551677] pcieport 0001:00:00.0: PME: Signaling with IRQ 256
	[    4.572484] WARNING: CPU: 3 PID: 89 at drivers/pci/controller/dwc/pcie-qcom.c:1430 qcom_pcie_icc_update+0xf8/0x144
	[    4.595291] pc : qcom_pcie_icc_update+0xf8/0x144
	[    4.595293] lr : qcom_pcie_icc_update+0x9c/0x144
	[    4.640742]  qcom_pcie_icc_update+0xf8/0x144
	[    4.640745]  qcom_pcie_probe+0x234/0x30c
	[    4.988519] qcom-pcie 1c00000.pci: Phy link never came up
	[    5.302179] qcom-pcie 1c00000.pci: PCI host bridge to bus 0000:00
	[    5.372008] pcieport 0000:00:00.0: Adding to iommu group 10
	[    5.378905] pcieport 0000:00:00.0: PME: Signaling with IRQ 257
	[   11.564590] pcieport 0000:00:00.0: BAR 14: assigned [mem 0x40400000-0x405fffff]

To locate the NVMe directories, run the following command.

ls -lh /dev/nvme*

Output:

crw-------. 1 root root 508,  0 Jan  1 00:00 /dev/nvme0
	brw-rw----. 1 root disk 259, 30 Jan  1 00:00 /dev/nvme0n1
	

Update the iommu-map property for SMMUv2 targets

Targets that use an SMMUv2 IOMMU for PCIe (for example, Dragonwing IQ-9075 (Lemans), Monaco, and Kodiak) translate each PCIe Requester ID (RID) to a Stream ID (SID) through the iommu-map property in the PCIe host bridge node. Getting this property wrong is a common source of no iommu-map translation for id failures when adding devices, switches, or endpoints downstream of a PCIe root complex.

How iommu-map works

Each iommu-map entry has the form:

iommu-map = <rid_base  &smmu_phandle  sid_base  count>;

The kernel (drivers/of/base.c: of_map_id()) applies the following rule.

if RID is in [rid_base, rid_base + count):
    SID = RID - rid_base + sid_base

Multiple entries are checked in order, and the first match wins. With count = 1, the mapping simplifies to RID == rid_base -> SID = sid_base. Using count = 1 per entry (sparse point mapping) pins exactly one RID to exactly one SID, regardless of how large the RID value is, and avoids exhausting the SMMU’s implemented SID range.

Note: A naive single-range entry, such as iommu-map = <0x0 &pcie_smmu 0x0000 0x500>;, maps count consecutive RIDs to count consecutive SIDs, so the SID offset grows proportionally with the RID value. This fails once RIDs from a downstream switch or additional endpoint exceed the SMMU’s available SID range.

PCIe RID and SID basics

A PCIe RID is a 16-bit value that encodes the bus, device, and function.

RID = (Bus << 8) | (Device << 3) | Function

Table : Common RID values

Bus	Dev	Fn	RID (hex)	Description
0	0	0	0x0000	Root complex, always present
1	0	0	0x0100	First device downstream of the RC (endpoint or switch upstream)
2	1	0	0x0208	Switch downstream port, device 1
2	16	1	0x0281	lspci address 02:10.1

lemans.dtsi reference example

Lemans defines two PCIe controllers, each mapped to its own SID subrange within the shared pcie_smmu (qcom,sa8775p-smmu-500, an SMMUv2-family IOMMU). pcie0 uses SID base 0x0000 and pcie1 uses SID base 0x0080, so their SID spaces don’t overlap.

&pcie0 {
	/* RC on bus 0, WiFi endpoint on bus 1 */
	iommu-map = <0x0   &pcie_smmu 0x0000 0x0 0x1>,
		    <0x100 &pcie_smmu 0x0001 0x0 0x1>;
};

&pcie1 {
	/* RC on bus 0, endpoint on bus 1 */
	iommu-map = <0x0   &pcie_smmu 0x0080 0x0 0x1>,
		    <0x100 &pcie_smmu 0x0081 0x0 0x1>;
};

RID 0x0000 (bus 0, dev 0, fn 0) maps to SID 0x0000 on pcie0 and SID 0x0080 on pcie1. RID 0x0100 (bus 1, dev 0, fn 0) maps to SID 0x0001 on pcie0 and SID 0x0081 on pcie1. Monaco follows the same pattern with its own pcie_smmu instance, and Kodiak follows the same pattern using apps_smmu (qcom,sc7280-smmu-500) instead of a dedicated pcie_smmu.

Note: lemans-ride-common.dtsi places a WiFi chip on pcieport0 (bus 1, dev 0, fn 0 — RID 0x0100) and inherits the iommu-map from lemans.dtsi without an override, since the base file already covers that RID.

Extending iommu-map for a switch topology

If a PCIe switch is fitted downstream of a controller (for example, adding an NVMe SSD on bus 4 through two switch ports), extend the entry list with one line per RID, keeping count = 1 for each:

&pcie0 {
	iommu-map = <0x0000 &pcie_smmu 0x0000 0x0 0x1>,   /* RC              */
		    <0x0100 &pcie_smmu 0x0001 0x0 0x1>,   /* switch upstream */
		    <0x0200 &pcie_smmu 0x0002 0x0 0x1>,   /* switch port A   */
		    <0x0300 &pcie_smmu 0x0003 0x0 0x1>,   /* switch port B   */
		    <0x0400 &pcie_smmu 0x0004 0x0 0x1>;   /* NVMe SSD        */
};

This uses 5 SIDs total (0x0000 through 0x0004), all within pcie0’s subrange, instead of a single range entry that would need count = 0x500 and could exceed the SMMU’s implemented SID range.

Note: SID values assigned to each RID don’t need to be sequential or in RID order. They only need to be unique within the controller’s iommu-map and within the SMMU’s implemented SID range.

Diagnose a missing iommu-map entry

When a device’s RID has no matching iommu-map entry, the kernel logs an error similar to the following.

OF: /pci@1c10000: no iommu-map translation for id 0x281 on (null)
vfio-pci: probe of 0001:02:10.1 failed with error -22

Read the RID from the log. In this example, the RID is 0x0281.
Decode the RID into bus, device, and function.
Bus = (RID >> 8) & 0xFF = 0x02 = 2 Device = (RID >> 3) & 0x1F = 0x10 = 16 Fn = RID & 0x07 = 0x01 = 1
Cross-check against the lspci address in the same log line (0001:02:10.1 → bus 2, device 0x10, function 1). The fields match, confirming the RID decodes correctly.
Choose a free SID for this device. Look at the existing iommu-map entries and pick a SID that isn’t already used by another entry in this controller’s map and is within the SMMU’s implemented SID range.

Add the entry with count = 1.

iommu-map = <0x0000 &apps_smmu 0x1400 0x1>,   /* RC              */
	    <0x0100 &apps_smmu 0x1401 0x1>,   /* bus 1 dev0 fn0  */
	    <0x0200 &apps_smmu 0x1402 0x1>,   /* bus 2 dev0 fn0  */
	    <0x0280 &apps_smmu 0x1403 0x1>,   /* bus 2 dev16 fn0 */
	    <0x0281 &apps_smmu 0x1404 0x1>;   /* bus 2 dev16 fn1 <- added */

If fn0 and fn1 are consecutive and neither has a SID yet, a single count = 2 entry can cover both instead: <0x0280 &apps_smmu 0x1403 0x2> maps RID 0x0280 to SID 0x1403 and RID 0x0281 to SID 0x1404.

Rebuild the DTB, flash, and reboot, then confirm the fix.
dmesg | grep "0x281"
The error line must no longer appear.

Note: lspci shows a device regardless of whether an iommu-map entry exists, since lspci reads PCI configuration space directly and isn’t affected by IOMMU mapping. Use dmesg, not lspci, to confirm a missing entry is fixed.

For more information on the kernel’s RID-to-SID translation, see drivers/of/base.c: of_map_id().

PCIe client driver sample

The client driver defines the device-id table and pci_driver structures, and registers with the PCIe framework. The following are a few PCIe client driver samples for reference.

Sample data structure to hold client-specific private data.

struct sample_driver_data {
    int driver_data;
};

Sample driver: You can provide data according to your driver-specific data structure.
struct sample_driver_data info = {};

Sample device ID table with the driver-specific data. The client driver registers with the 0x306 device ID.

static const struct pci_device_id sample_pci_id_table[] = {
    { PCI_DEVICE_SUB(PCI_VENDOR_ID_QCOM, 0x0306, PCI_VENDOR_ID_QCOM, 0x010c),
        .driver_data = (kernel_ulong_t) &info },
    {  } 
};
MODULE_DEVICE_TABLE(pci, sample_pci_id_table);

Note: MODULE_DEVICE_TABLE(pci, sample_pci_id_table); is mandatory.

Sample pci_driver data structure with client driver name, pci-id table, and callbacks. The pointer to this structure is passed while registering with the PCI frame work.

static struct pci_driver sample_pci_driver = {
      .name           = "Sample-PCI-Client-driver",        /* Give the name that suites your driver description */
      .id_table       = sample_pci_id_table,               /* pci core driver will compare from ids's supplied in this table with the enumerated endpoints */
      .probe          = sample_pci_probe,                  /* probe function is invoked by the PCI f/w when id matches with the endpoint id's */
      .remove         = sample_pci_remove,                 /* Remove function is invoked by PCI f/w when senses the attached endpoint/function is detached or down */
      .driver.pm      = &sample_pci_pm_ops                 /* pm ops that will be called from the pci core driver*/
};

To register with PCI firmware, call pci_register_driver(&sample_pci_driver) from module_init().
module_pci_driver(sample_pci_driver);

PCIe bringup

For information about PCIe bringup, see PCIe-related configurations and QPS615 switch support.

PCIe power optimization

PCIe defines two types of power management methods.

Power management software that determines the power management capability of each device and manages each device individually
System that doesn’t require software intervention such as active state power management (ASPM)

During a time period when no packet is transmitted through the link, a device places the link into a power-saving state.

PCIe L0 link states

PCIe power management defines the following L0 link states:

L0: active state where all PCIe transactions and other operations are enabled
L0s: ASPM state with low-resume latency (energy saving standby state)

PCIe device states

PCIe power management defines the following device states:

D0 (mandatory): The device is in full ON state, where there are two substates
- D0_{uninitialized}: The function is present in the D0_{uninitialized} state after the device comes out of reset, waiting to be enumerated and configured.
- D0_active
  - The function is present in the D0_active state following the completion of the enumeration and configuration process.
  - The function enters the D0_active state when the system software enables one or more (in any combination) function parameters, such as memory space enable, I/O space enable, or BME bits.
D1 (optional): light-sleep state
- The function can’t initiate a TLP except for the PME message
- The function can’t act as the target of transactions other than for configuration transactions.
- The function issues a software command to enter the D1 state by programming the PM control and status register.
D2 (optional): deep-sleep state
- The function can’t initiate a TLP except for the PME message
- The function can’t act as the target of transactions other than configuration transactions.
- The function issues a software command to enter the D2 state by programming the PM control and status register.
D3 (mandatory): device is the lowest power state, where the function must support both the D3 states
D3_hot
- The function can’t initiate a TLP except for the PME message.
- The function can’t act as the target of transactions other than configuration transactions.
- The function issues a software command to enter the D3_hot state by programming the power state field.
D3_cold: device enters the D3_cold state and power is removed; when power is restored, the device enters the D0_{uninitialized} state.

PCIe verification

For information about PCIe verification, see PCIe-related configurations and QPS615 switch support.

Debug PCIe issues

The lspci and setpci commands are native to Linux distributions. These commands have various levels of output. These commands also provide a useful point-in-time look at the capabilities and status of the different components trained on the PCI bus. Most of these capabilities are reflections of the configuration space registers required by the PCIe base specification. For more details, see https://pcisig.com/specifications. To view the usage instructions, run the following command.

lspci --help

The following features are useful in troubleshooting PCIe issues.

Display device information

lspci

The following message is displayed.

00:00.0 PCI bridge: Qualcomm Device 010b
01:00.0 PCI bridge: Toshiba Corporation Device 0623
02:01.0 PCI bridge: Toshiba Corporation Device 0623
02:02.0 PCI bridge: Toshiba Corporation Device 0623
02:03.0 PCI bridge: Toshiba Corporation Device 0623
04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
05:00.0 Ethernet controller: Toshiba Corporation Device 0220
05:00.1 Ethernet controller: Toshiba Corporation Device 0220

Display PCIe device and vendor IDs in the device control register.

lspci -nvmm

The following message is displayed.

Slot:   0001:00:00.0
Class:  0604
Vendor: 17cb
Device: 010b
DTNode: /sys/firmware/devicetree/base/soc@0/pci@1c08000/pcie@0
IOMMUGroup:     33
 
Slot:   0001:01:00.0
Class:  0604
Vendor: 1179
Device: 0623
DTNode: /sys/firmware/devicetree/base/soc@0/pci@1c08000/pcie@0/pcie1_bus1_dev0_fn0
IOMMUGroup:     33
 
Slot:   0001:02:01.0
Class:  0604
Vendor: 1179
Device: 0623
DTNode: /sys/firmware/devicetree/base/soc@0/pci@1c08000/pcie@0/pcie1_bus1_dev0_fn0/pcie1_bus2_dev1_fn0
IOMMUGroup:     33
 
Slot:   0001:02:02.0
Class:  0604
Vendor: 1179
Device: 0623
DTNode: /sys/firmware/devicetree/base/soc@0/pci@1c08000/pcie@0/pcie1_bus1_dev0_fn0/pcie1_bus2_dev2_fn0
IOMMUGroup:     33
 
Slot:   0001:02:03.0
Class:  0604
Vendor: 1179
Device: 0623
DTNode: /sys/firmware/devicetree/base/soc@0/pci@1c08000/pcie@0/pcie1_bus1_dev0_fn0/pcie1_bus2_dev3_fn0
IOMMUGroup:     33
 
Slot:   0001:04:00.0
Class:  0c03
Vendor: 1912
Device: 0014
Rev:    03
ProgIf: 30
DTNode: /sys/firmware/devicetree/base/soc@0/pci@1c08000/pcie@0/pcie1_bus1_dev0_fn0/pcie1_bus2_dev2_fn0/pcie1_bus4_dev0_fn0
IOMMUGroup:     33
 
Slot:   0001:05:00.0
Class:  0200
Vendor: 1179
Device: 0220
SVendor:        1179
SDevice:        0001
DTNode: /sys/firmware/devicetree/base/soc@0/pci@1c08000/pcie@0/pcie1_bus1_dev0_fn0/pcie1_bus2_dev3_fn0/qps615_eth0,qps615_eth0@pcie1_rp
IOMMUGroup:     33
 
Slot:   0001:05:00.1
Class:  0200
Vendor: 1179
Device: 0220
SVendor:        1179
SDevice:        0001
DTNode: /sys/firmware/devicetree/base/soc@0/pci@1c08000/pcie@0/pcie1_bus1_dev0_fn0/pcie1_bus2_dev3_fn0/qps615_eth1,qps615_eth1@pcie1_rp
IOMMUGroup:     33

For more info on PCIe debugging, see https://www.kernel.org/doc/html/v4.17/driver-api/pci.html.

PCIe examples

For information about the upstream device tree reference, see the following files.

For information about device-tree node for the Qualcomm Linux hardware SoCs, see the following DTSI files.

QCS6490 and QCS5430: https://elixir.bootlin.com/linux/v7.0-rc7/source/arch/arm64/boot/dts/qcom/kodiak.dtsi
Dragonwing IQ-9075: https://elixir.bootlin.com/linux/v7.0-rc7/source/arch/arm64/boot/dts/qcom/lemans.dtsi
Dragonwing IQ-615: https://git.kernel.org/pub/scm/linux/kernel/git/qcom/linux.git/tree/arch/arm64/boot/dts/qcom/qcs615.dtsi?h=arm64-for-6.16

Client and PCI driver operation flow example

The following figure shows the sequence that the PCIe client driver follows to configure the PCIe driver for a client.

Client and PCI driver high-level call flow example

The following figure shows the high-level call flow and call details between the PCIe client driver and PCIe driver.

​PCIe host mode enumeration feature

​PCIe layered architecture

​PCIe software driver configuration

​Link training

​Hardware initialization

​Enable PCIe endpoint (EP) mode for Dragonwing IQ-9075 (Lemans)

​PCIe EP mode kernel configuration

​Enable the EP node in the device tree

​PCIe EP driver initialization

​Bind the MHI endpoint function using configfs

​Verify MHI EP mode enumeration

​Enable QPS615 PCIe switch

​PCIe-related configurations

​Message signaled interrupt (MSI)

​Sample PCIe kernel driver log

​Ethernet interfaces supported through QPS615 PCIe switch

​Bring up alternate hardware components

​Enable USB interface through PCIe switch

​Download PCIe to USB controller firmware

​PCIe kernel driver logs for PCIe to USB device enumeration reference

​Connect QPS615 switches in cascade

​Enable NVMe through PCIe interface

​Update the iommu-map property for SMMUv2 targets

​How iommu-map works

​PCIe RID and SID basics

​lemans.dtsi reference example

​Extending iommu-map for a switch topology

​Diagnose a missing iommu-map entry

​PCIe client driver sample

​PCIe bringup

​PCIe power optimization

​PCIe L0 link states

​PCIe device states

​PCIe verification

​Debug PCIe issues

​PCIe examples

​Client and PCI driver operation flow example

​Client and PCI driver high-level call flow example

PCIe host mode enumeration feature

PCIe layered architecture

PCIe software driver configuration

Link training

Hardware initialization

Enable PCIe endpoint (EP) mode for Dragonwing IQ-9075 (Lemans)

PCIe EP mode kernel configuration

Enable the EP node in the device tree

PCIe EP driver initialization

Bind the MHI endpoint function using configfs

Verify MHI EP mode enumeration

Enable QPS615 PCIe switch

PCIe-related configurations

Message signaled interrupt (MSI)

Sample PCIe kernel driver log

Ethernet interfaces supported through QPS615 PCIe switch

Bring up alternate hardware components

Enable USB interface through PCIe switch

Download PCIe to USB controller firmware

PCIe kernel driver logs for PCIe to USB device enumeration reference

Connect QPS615 switches in cascade

Enable NVMe through PCIe interface

Update the iommu-map property for SMMUv2 targets

How iommu-map works

PCIe RID and SID basics

lemans.dtsi reference example

Extending iommu-map for a switch topology

Diagnose a missing iommu-map entry

PCIe client driver sample

PCIe bringup

PCIe power optimization

PCIe L0 link states

PCIe device states

PCIe verification

Debug PCIe issues

PCIe examples

Client and PCI driver operation flow example

Client and PCI driver high-level call flow example