Figure: PCIe device connection link
The path between the devices is called a Link. It is made up of one or more transmit and receive pairs. One pair of the Link is called a Lane. The PCIe device connection in Qualcomm Linux devices supports 16 lanes. The number of lanes or the Link width is x16. The following table lists the types of PCIe connections for devices.Table : PCIe connections
| PCIe type | Description |
|---|---|
| Root complex (RC) | Connects the CPU to the PCIe topology |
| Switch | Connects more than 2 ports and acts as a packet router |
| Bridge | Connects different buses: for example, PCIe to PCIe, or PCIe to peripheral component interconnect (PCI) |
| Endpoint (EP) | Resides at the bottom of the PCIe topology tree structure and has only an upstream port |
| Legacy endpoint | Uses older PCI bus operations to support backward compatibility |
PCIe host mode enumeration feature
When a system first powers up, the configuration software running on the system host processor is aware of the existence of only Bus 0 (if PCIe is supported). The software is not aware of the bus topology or any device connected to the bus. The enumeration process discovers the various buses, devices, and functions present in the system. When enumeration is complete, each bus in the system is numbered as follows:- The primary bus number indicates the bus that directly connects to the primary interface of the bridge (towards the root complex).
- The secondary bus number indicates the bus that directly connects to the secondary interface of the bridge (away from the root complex).
- The subordinate bus number indicates the highest numbered bus that exists on the downstream side of the bridge.
- Link training
- Scanning for devices on the bus
- Registration
PCIe layered architecture
The following figure shows the PCIe software architecture.
Figure : PCIe software architecture
The following figure shows the layered architecture model of PCIe.Figure : PCIe layered architecture
The transmission units exchanged are as follows.- Ordered set between the physical layer entities.
- Data link layer packet (DLLP) between data link layer entities.
- Transaction layer packet (TLP) between transaction layer entities.
Table : Layers in PCIe architecture
| Layer | Features |
|---|---|
| Physical layer | Logical sub-block: Link training, initialization, and maintenance. |
| Physical sub-block: 8b/10b encoding and decoding, and parallel-to-serial and serial-to-parallel conversion. | |
| Data link layer | Assembly and disassembly of the DLLP packet. |
| Generation and validation of the link layer CRC (LCRC). | |
| Acknowledgment and no acknowledgment protocol (replay of TLPs in error). | |
| Transaction layer | Assembly and disassembly of the TLP packet. |
| Generation and validation of end-to-end CRC (ECRC). | |
| Flow control receives entity advertises for the available receive buffer size information using DLLPs. | |
| Quality of service (QoS): traffic class (TC) to virtual channel (VC) mapping. | |
| Transaction ordering: implements the transaction ordering rule within a VC. |
Figure : PCIe configuration address space
PCIe software driver configuration
The PCIe controller driver initializes the PCIe resources and performs link training. After successful training, the controller driver calls the PCIe framework for link enumeration, such as endpoint discovery, identifying the client driver, and probing those drivers. For more information about the PCIe framework and client driver PCIe registrations, see https://www.kernel.org/doc/html/latest/PCI/index.html.Link training
Link training comprises the following operations:- The PCIe driver
pcie-qcom.cfile at https://github.com/torvalds/linux/blob/master/drivers/pci/controller/dwc/pcie-qcom.c obtains the required resources such as regulators, clocks, from the device tree. - The PCIe driver calls Synopsys DesignWare® Core host driver pcie-designware-host.c file at https://github.com/torvalds/linux/blob/master/drivers/pci/controller/dwc/pcie-designware-host.c to initialize the root complex.
- The Synopsys DesignWare Core driver performs all necessary initializations.
- The Synopsys DesignWare Core driver calls a function pointer to perform host initialization.
- The Qualcomm PCIe driver performs PHY power-on, enables all regulators, clocks.
- The Synopsys DesignWare Core driver starts the link training by calling the function pointer to start the link.
Hardware initialization
The driver initializes and configures the PCIe hardware block and performs link training. The initialization occurs after theplatform _probe() driver function is called.
Note: The PCIE_0 root complex instance is enabled by default for the WLAN EP connection.
Enable QPS615 PCIe switch
This section describes how to enable a QPS615 PCIe switch in the Qualcomm Linux hardware SoCs. The QPS615 switch endpoint is supported on thePCIe1 instance. The following figure shows the QPS615 endpoint and connections.
Figure : QPS615 PCIe switch connection diagram
The Qualcomm PCIe driver documentation can be accessed at the following locations:- https://github.com/torvalds/linux/blob/master/Documentation/devicetree/bindings/pci/qcom%2Cpcie-sc8280xp.yaml
- https://elixir.bootlin.com/linux/v6.6.48/source/Documentation/devicetree/bindings/pci/qcom,pcie.yaml
- https://github.com/torvalds/linux/blob/master/Documentation/devicetree/bindings/phy/qcom%2Csc8280xp-qmp-pcie-phy.yaml
PCIe-related configurations
The following configurations are enabled by default to support the QPS615 switch. Disable the QPS615 switch default support, by reverting the code changes, to use it for a different PCIe endpoint. To enable PCIe-relatedconfigs, apply the following patch to the /arch/arm64/configs/qcom_addons.config file.
Message signaled interrupt (MSI)
The current MSI mapping doesn’t have all the vectors. The Qualcomm Linux hardware SoCs support eight vectors. Each vector in turn supports 32 MSIs. Therefore, the total MSIs supported are 256. For information about adding all the MSI groups supported for this PCIe instance, see https://lore.kernel.org/linux-arm-msm/f1168212-bc6e-4570-869c-2870d6f248ad@linaro.org/T/.Sample PCIe kernel driver log
The following is a sample PCIe kernel driver log from the QPS615 device enumeration.Ethernet interfaces supported through QPS615 PCIe switch
The QPS615 PCIe switch enables Ethernet connectivity for the device. When the device loads the QPS615 driver and establishes the PCIe link, it automatically activates the supported Ethernet interfaces, by default, during device startup. To customize the default configuration or enable extra MAC/PHY components beyond Qualcomm’s hardware setup, see the Bring up Ethernet section in the Qualcomm Linux Ethernet guide.Table : Supported Ethernet interfaces
| Interface type | Speed | Connector type | Description |
|---|---|---|---|
| QEP PHY (SGMII) | 2.5 GbE | IX/RJ45 connector (QEP8121) |
|
| AQR PHY (USXGMII) | 10 GbE | IX/RJ45 connector (AQR113C) |
|
Bring up alternate hardware components
You can attach MAC/PHY components other than the hardware configuration provided by Qualcomm and bring them up. To replace QPS615 with other PCIe based MAC/PHY, see Enable QPS615 PCIe switch.Note: You must obtain the MAC/PHY driver and firmware from the respective vendor. Qualcomm isn’t responsible for these configuration changes.
Enable USB interface through PCIe switch
This section provides instructions on how to activate a USB interface through a PCIe switch in the Qualcomm Linux hardware SoCs. The PCIE1 instance is connected to the endpoint of the QPS615 switch, and the downstream port of the QPS615 is connected to the PCIe to USB endpoint. For the PCIe to USB endpoint connections using the QPS615, see the mainboard and interposer block diagram at https://docs.qualcomm.com/bundle/publicresource/topics/80-80021-251/rb3_hardware_overview.html.Note: Dragonwing IQ-9075 PCIe software doesn’t support USB.
Download PCIe to USB controller firmware
To download the firmware from https://www.renesas.com/us/en/products/interface/usb-switches-hubs/upd720201-usb-30-host-controller#design_development, register and log in to https://www.renesas.com/. Rename the downloaded firmware file to renesas_usb_fw.mem.Note: To prevent command failures, update the software as described in the Set up the device section before updating the Renesas firmware.
- Connect device to Host PC via USB cable for adb.
- Push firmware to device.
- To activate the firmware do either of the following options.
- Option A: Reboot target for USB type A ports.
- Option B: Manually bind the Renesas xHCI driver.
- Verify firmware enumeration.
PCIe kernel driver logs for PCIe to USB device enumeration reference
You can run the following commands to view the device information:- To display device information in USB, run the following command.
The following message is displayed.
- To display device information in PCIe, run the following command.
The following message is displayed.
Connect QPS615 switches in cascade
Connect the QPS615 switches in cascade to enable additional Ethernet, PCIe, and USB ports.Note: This feature is supported only in QCS6490.
- To reset the QPS615 switch, toggle the RESX GPIOs for both QPS615 #1 and QPS615 #2.
- To control the endpoint reset, trigger PERST. Both the switches share the PERST.
- Switch-attached devices:
- The QPS615 PCIe tree node hierarchy is statically fixed.
- All nodes for switching USP and DSP ports are created during PCIe initialization.
- One of the switch DSP ports represents WLAN.
- If you disable the WLAN node, it disables the WLAN device, but the PCIe downstream port remains enabled and returns a default maximum link width.
- Directly-attached WLAN devices:
- Disables only a single node.
- Returns the Invalid argument, when WLAN is disabled.
Enable NVMe through PCIe interface
This section describes how to enable NVMe using PCIe for storage expansion. To verify if NVMe is connected over a PCIe interface, do the following:- To display PCIe device information, run the following command.
Output:
- Locate the PCIe logs.
Output:
- To locate the NVMe directories, run the following command.
Output:
PCIe client driver sample
The client driver defines thedevice-id table and pci_driver structures, and registers with the PCIe framework. The following are a few PCIe client driver samples for reference.
- Sample data structure to hold client-specific private data.
- Sample driver: You can provide data according to your driver-specific data structure.
- Sample device ID table with the driver-specific data. The client driver registers with the
0x306device ID.
Note:
MODULE_DEVICE_TABLE(pci, sample_pci_id_table); is mandatory.- Sample
pci_driverdata structure with client driver name,pci-idtable, and callbacks. The pointer to this structure is passed while registering with the PCI frame work. - To register with PCI firmware, call
pci_register_driver(&sample_pci_driver)frommodule_init().
PCIe bringup
For information about PCIe bringup, see PCIe-related configurations and QPS615 switch support.PCIe power optimization
PCIe defines two types of power management methods.- Power management software that determines the power management capability of each device and manages each device individually
- System that doesn’t require software intervention such as active state power management (ASPM)
PCIe L0 link states
PCIe power management defines the following L0 link states:- L0: active state where all PCIe transactions and other operations are enabled
- L0s: ASPM state with low-resume latency (energy saving standby state)
PCIe device states
PCIe power management defines the following device states:- D0 (mandatory): The device is in full ON state, where there are two substates
- D0uninitialized: The function is present in the D0uninitialized state after the device comes out of reset, waiting to be enumerated and configured.
- D0active
- The function is present in the D0active state following the completion of the enumeration and configuration process.
- The function enters the D0active state when the system software enables one or more (in any combination) function parameters, such as memory space enable, I/O space enable, or BME bits.
- D1 (optional): light-sleep state
- The function can’t initiate a TLP except for the PME message
- The function can’t act as the target of transactions other than for configuration transactions.
- The function issues a software command to enter the D1 state by programming the PM control and status register.
- D2 (optional): deep-sleep state
- The function can’t initiate a TLP except for the PME message
- The function can’t act as the target of transactions other than configuration transactions.
- The function issues a software command to enter the D2 state by programming the PM control and status register.
- D3 (mandatory): device is the lowest power state, where the function must support both the D3 states
- D3hot
- The function can’t initiate a TLP except for the PME message.
- The function can’t act as the target of transactions other than configuration transactions.
- The function issues a software command to enter the D3hot state by programming the power state field.
- D3cold: device enters the D3cold state and power is removed; when power is restored, the device enters the D0uninitialized state.
PCIe verification
For information about PCIe verification, see PCIe-related configurations and QPS615 switch support.Debug PCIe issues
Thelspci and setpci commands are native to Linux distributions. These commands have various levels of output. These commands also provide a useful point-in-time look at the capabilities and status of the different components trained on the PCI bus. Most of these capabilities are reflections of the configuration space registers required by the PCIe base specification. For more details, see https://pcisig.com/specifications. To view the usage instructions, run the following command.
- Display device information
The following message is displayed.
- Display PCIe device and vendor IDs in the device control register.
The following message is displayed.
PCIe examples
For information about the upstream device tree reference, see the following files.- QCS6490 and QCS5430: https://elixir.bootlin.com/linux/v7.0-rc7/source/arch/arm64/boot/dts/qcom/kodiak.dtsi
- Dragonwing IQ-9075: https://elixir.bootlin.com/linux/v7.0-rc7/source/arch/arm64/boot/dts/qcom/lemans.dtsi
- Dragonwing IQ-615: https://git.kernel.org/pub/scm/linux/kernel/git/qcom/linux.git/tree/arch/arm64/boot/dts/qcom/qcs615.dtsi?h=arm64-for-6.16
- https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-qcom.c?h=v6.8-rc6#n1634
- QCS6490 and QCS5430: https://elixir.bootlin.com/linux/v7.0-rc7/source/arch/arm64/boot/dts/qcom/kodiak.dtsi
- Dragonwing IQ-9075: https://elixir.bootlin.com/linux/v7.0-rc7/source/arch/arm64/boot/dts/qcom/lemans.dtsi
- Dragonwing IQ-615: https://git.kernel.org/pub/scm/linux/kernel/git/qcom/linux.git/tree/arch/arm64/boot/dts/qcom/qcs615.dtsi?h=arm64-for-6.16
Client and PCI driver operation flow example
The following figure shows the sequence that the PCIe client driver follows to configure the PCIe driver for a client.
Client and PCI driver high-level call flow example
The following figure shows the high-level call flow and call details between the PCIe client driver and PCIe driver.

