Skip to contentSkip to conversation

Add AMD Versal CPM/QDMA PCIe support

PATCH Applied Pranav Sanwal 2 participants 7 messages Aug 24 – Sep 7

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover Add AMD Versal CPM/QDMA PCIe support

This series adds PCIe root port support for the AMD Versal CPM4, CPM5 and QDMA PL (soft IP) host bridges, and enables it along with NVMe storage on the xilinx_versal_virt platform.

Patch 1 adds a generic mmap config-space write helper for host bridges that only decode 32-bit accesses.
Patches 2 and 3 add the QDMA PL and CPM4/CPM5 host bridge drivers, respectively.
Patch 4 maps the CPM5 ECAM windows as device memory so config space accesses are not cached or reordered.
Patch 5 wires everything up in the versal_virt defconfig and adds NVMe as a distro boot target.

Pranav Sanwal (5):
pci: Add pci_generic_mmap_write_config32() helper
pci: Add support for AMD QDMA PL PCIe root port
pci: Add support for CPM4 and CPM5 PCIe controllers
arm64: versal: Add MMU mappings for CPM5 ECAM config space
arm64: versal: Enable PCIe/NVMe support and add NVMe boot target

 MAINTAINERS                          |  10 +
 arch/arm/mach-versal/cpu.c           |  20 ++
 configs/xilinx_versal_virt_defconfig |   6 +
 drivers/pci/Kconfig                  |  19 ++
 drivers/pci/Makefile                 |   2 +
 drivers/pci/pci-uclass.c             |  42 ++++
 drivers/pci/pcie_amd_cpm.c           | 363 +++++++++++++++++++++++++++
 drivers/pci/pcie_amd_qdma_pl.c       | 205 +++++++++++++++
 include/configs/xilinx_versal.h      |   7 +
 include/pci.h                        |  26 ++
 10 files changed, 700 insertions(+)
 create mode 100644 drivers/pci/pcie_amd_cpm.c
 create mode 100644 drivers/pci/pcie_amd_qdma_pl.c
Signature · 1 line
2.43.7
#1 lore raw

PATCH 1/5 pci: Add pci_generic_mmap_write_config32() helper

Some memory-mapped PCI host bridges only decode 32-bit accesses to their configuration space and either drop or corrupt neighbouring bytes on a narrower access. Because such hardware only ever reliably commits a full 32-bit transaction, the only safe way to perform a narrower write is to synthesize it in software: read the containing word, patch in the requested bits, and write the whole word back.

Add pci_generic_mmap_write_config32(), a counterpart to pci_generic_mmap_write_config(), that does exactly this: it transparently promotes 8-bit and 16-bit writes to a 32-bit read-modify-write of the word containing the requested offset, leaving 32-bit writes untouched. This mirrors Linux's pci_generic_config_write32() (drivers/pci/access.c), used for the same class of host bridge.

---
 drivers/pci/pci-uclass.c | 42 ++++++++++++++++++++++++++++++++++++++++
 include/pci.h            | 26 +++++++++++++++++++++++++
 2 files changed, 68 insertions(+)
drivers/pci/pci-uclass.c +42 -0
index 4bdd1f7477f..f4ce630118f 100644
@@ -618,6 +618,48 @@ int pci_generic_mmap_write_config(
 	}
 }
 
+int pci_generic_mmap_write_config32(
+	const struct udevice *bus,
+	int (*addr_f)(const struct udevice *bus, pci_dev_t bdf, uint offset,
+		      void **addrp),
+	pci_dev_t bdf,
+	uint offset,
+	ulong value,
+	enum pci_size_t size)
+{
+	u32 shift, mask, tmp;
+	void *address;
+
+	if (addr_f(bus, bdf, ALIGN_DOWN(offset, 4), &address) < 0) {
+		debug("%s: failed to get config address for offset 0x%x\n",
+		      __func__, offset);
+		return 0;
+	}
+
+	switch (size) {
+	case PCI_SIZE_32:
+		writel(value, address);
+		return 0;
+	case PCI_SIZE_8:
+	case PCI_SIZE_16:
+		/*
+		 * Some host bridges only decode 32-bit accesses to their
+		 * config space and silently corrupt neighbouring bytes on a
+		 * narrower write. Widen the access to a 32-bit
+		 * read-modify-write of the word containing the requested
+		 * offset.
+		 */
+		shift = (offset % 4) * BITS_PER_BYTE;
+		mask = pci_get_ff(size) << shift;
+		tmp = readl(address) & ~mask;
+		tmp |= (value << shift) & mask;
+		writel(tmp, address);
+		return 0;
+	default:
+		return -EINVAL;
+	}
+}
+
 int pci_generic_mmap_read_config(
 	const struct udevice *bus,
 	int (*addr_f)(const struct udevice *bus, pci_dev_t bdf, uint offset,
include/pci.h +26 -0
index 4b0facd6dcf..108d97a5d54 100644
@@ -1151,6 +1151,32 @@ int pci_generic_mmap_write_config(
 	ulong value,
 	enum pci_size_t size);
 
+/**
+ * pci_generic_mmap_write_config32() - Generic helper for writing to
+ * memory-mapped PCI configuration space that only supports 32-bit accesses.
+ * @bus: Pointer to the PCI bus
+ * @addr_f: Callback for calculating the config space address
+ * @bdf: Identifies the PCI device to access
+ * @offset: The offset into the device's configuration space
+ * @value: The value to write
+ * @size: Indicates the size of access to perform
+ *
+ * Like pci_generic_mmap_write_config(), but for host bridges whose
+ * configuration space does not decode 8-bit or 16-bit accesses correctly.
+ * Accesses narrower than 32 bits are promoted to a 32-bit
+ * read-modify-write of the word containing @offset.
+ *
+ * Return: 0 on success, else -EINVAL
+ */
+int pci_generic_mmap_write_config32(
+	const struct udevice *bus,
+	int (*addr_f)(const struct udevice *bus, pci_dev_t bdf, uint offset,
+		      void **addrp),
+	pci_dev_t bdf,
+	uint offset,
+	ulong value,
+	enum pci_size_t size);
+
 /**
  * pci_generic_mmap_read_config() - Generic helper for reading from
  * memory-mapped PCI configuration space.
Signature · 1 line
2.43.7
#2 lore raw

PATCH 2/5 pci: Add support for AMD QDMA PL PCIe root port

In reply to Pranav Sanwal

Add PCIe host bridge driver for AMD QDMA PL (Programmable Logic) soft IP PCIe5 controller. This driver enables PCIe root port functionality for QDMA IP instantiated in the PL fabric.

---
 MAINTAINERS                    |   5 +
 drivers/pci/Kconfig            |  10 ++
 drivers/pci/Makefile           |   1 +
 drivers/pci/pcie_amd_qdma_pl.c | 205 +++++++++++++++++++++++++++++++++
 4 files changed, 221 insertions(+)
 create mode 100644 drivers/pci/pcie_amd_qdma_pl.c
MAINTAINERS +5 -0
index eb48eea55c5..4f184a39a9d 100644
@@ -71,6 +71,11 @@ F:	include/alist.h
 F:	lib/alist.c
 F:	test/lib/alist.c
 
+AMD QDMA PL PCIE DRIVER
+M:	Pranav Sanwal <[email protected]>
+S:	Maintained
+F:	drivers/pci/pcie_amd_qdma_pl.c
+
 AMD VERSAL2 PCIE DRIVER
 M:	Pranav Sanwal <[email protected]>
 S:	Maintained
drivers/pci/Kconfig +10 -0
index 2b2ccdf4745..da2bebafe9a 100644
@@ -444,6 +444,16 @@ config PCIE_XILINX_NWL
 	  Say 'Y' here if you want support for Xilinx / AMD NWL PCIe
 	  controller as Root Port.
 
+config PCIE_AMD_QDMA_PL
+	bool "AMD QDMA PL PCIe host bridge support"
+	depends on ARCH_VERSAL
+	select SYS_PCI_64BIT
+	help
+	  Say 'Y' here if you want support for the Xilinx/AMD QDMA
+	  PL PCIe host bridge. The controller is a Soft IP instantiated
+	  in the PL fabric which can act as a PCIe Root Port. It provides
+	  ECAM-based configuration space access for device enumeration.
+
 config PCIE_PLDA_COMMON
 	bool
 
drivers/pci/Makefile +1 -0
index e6d71fd172b..a6891154e9e 100644
@@ -54,6 +54,7 @@ obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o
 obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o
 obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o
 obj-$(CONFIG_PCIE_XILINX_NWL) += pcie-xilinx-nwl.o
+obj-$(CONFIG_PCIE_AMD_QDMA_PL) += pcie_amd_qdma_pl.o
 obj-$(CONFIG_PCIE_PLDA_COMMON) += pcie_plda_common.o
 obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.o
 obj-$(CONFIG_PCIE_DW_AMD) += pcie_dw_amd.o
drivers/pci/pcie_amd_qdma_pl.c new file +205 -0

205 lines not on this page · raw message

Signature · 1 line
2.43.7
#3 lore raw

PATCH 3/5 pci: Add support for CPM4 and CPM5 PCIe controllers

In reply to Pranav Sanwal

Add a host controller driver for the AMD Versal CPM4 and CPM5 PCIe bridges, selected via the DT compatible string and enabled with CONFIG_PCIE_AMD_CPM. The driver maps the bridge/ECAM regions, services ECAM-based config space accesses, masks off all local interrupts, and brings up the root port bridge. CPM5 variants also support an optional PERST#-driven IP reset sequence ahead of link bring-up.

---
 MAINTAINERS                |   5 +
 drivers/pci/Kconfig        |   9 +
 drivers/pci/Makefile       |   1 +
 drivers/pci/pcie_amd_cpm.c | 363 +++++++++++++++++++++++++++++++++++++
 4 files changed, 378 insertions(+)
 create mode 100644 drivers/pci/pcie_amd_cpm.c
MAINTAINERS +5 -0
index 4f184a39a9d..0cc8670cfe4 100644
@@ -71,6 +71,11 @@ F:	include/alist.h
 F:	lib/alist.c
 F:	test/lib/alist.c
 
+AMD CPM PCIE DRIVER
+M:	Pranav Sanwal <[email protected]>
+S:	Maintained
+F:	drivers/pci/pcie_amd_cpm.c
+
 AMD QDMA PL PCIE DRIVER
 M:	Pranav Sanwal <[email protected]>
 S:	Maintained
drivers/pci/Kconfig +9 -0
index da2bebafe9a..64bab2b9dd6 100644
@@ -444,6 +444,15 @@ config PCIE_XILINX_NWL
 	  Say 'Y' here if you want support for Xilinx / AMD NWL PCIe
 	  controller as Root Port.
 
+config PCIE_AMD_CPM
+	bool "AMD Versal CPM PCIe host bridge support"
+	depends on ARCH_VERSAL
+	depends on DM_GPIO
+	select SYS_PCI_64BIT
+	help
+	  Say 'Y' here if you want support for the AMD Versal
+	  CPM PCIe host bridge.
+
 config PCIE_AMD_QDMA_PL
 	bool "AMD QDMA PL PCIe host bridge support"
 	depends on ARCH_VERSAL
drivers/pci/Makefile +1 -0
index a6891154e9e..3873f357a45 100644
@@ -54,6 +54,7 @@ obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o
 obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o
 obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o
 obj-$(CONFIG_PCIE_XILINX_NWL) += pcie-xilinx-nwl.o
+obj-$(CONFIG_PCIE_AMD_CPM) += pcie_amd_cpm.o
 obj-$(CONFIG_PCIE_AMD_QDMA_PL) += pcie_amd_qdma_pl.o
 obj-$(CONFIG_PCIE_PLDA_COMMON) += pcie_plda_common.o
 obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.o
drivers/pci/pcie_amd_cpm.c new file +363 -0

363 lines not on this page · raw message

Signature · 1 line
2.43.7
#4 lore raw

PATCH 4/5 arm64: versal: Add MMU mappings for CPM5 ECAM config space

In reply to Pranav Sanwal

CPM5 Host0 and Host1 each expose PCIe config space (ECAM) inside their 4 GiB NoC address slot, at 0x600000000 and 0x700000000 respectively. Per the DT "cfg" reg and the PCIe ECAM spec (256 bus numbers x 1 MiB per bus), each window is 256 MiB. Config space accesses must not be cached, reordered, or speculatively issued, so map each ECAM window as Device memory (MT_DEVICE_NGNRNE).

Rather than mapping 0x600000000-0xe00000000 only when CONFIG_PCIE_AMD_CPM is enabled, add it as a baseline Normal/cacheable entry unconditionally: this NoC/PL address space is shared by more than CPM (PL fabric AXI space, other NoC-attached peripherals, etc.), so it needs coverage regardless of which PCIe controllers are built in. The two CPM5 ECAM sub-ranges are then overlaid as Device memory on top of that baseline only when PCIE_AMD_CPM is enabled.

Update VERSAL_MEM_MAP_USED to match: 3 entries when PCIE_AMD_CPM is disabled, 5 when it's enabled (+2 ECAM overlays).

---
 arch/arm/mach-versal/cpu.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
arch/arm/mach-versal/cpu.c +20 -0
index 7521d45bc1a..5dbd0fcc438 100644
@@ -20,7 +20,11 @@
 
 DECLARE_GLOBAL_DATA_PTR;
 
+#if CONFIG_IS_ENABLED(PCIE_AMD_CPM)
+#define VERSAL_MEM_MAP_USED	5
+#else
 #define VERSAL_MEM_MAP_USED	3
+#endif
 
 #define DRAM_BANKS CONFIG_NR_DRAM_BANKS
 
@@ -55,6 +59,22 @@ static struct mm_region versal_mem_map[VERSAL_MEM_MAP_MAX] = {
 		.attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) |
 			 PTE_BLOCK_NON_SHARE |
 			 PTE_BLOCK_PXN | PTE_BLOCK_UXN
+#if CONFIG_IS_ENABLED(PCIE_AMD_CPM)
+	}, {
+		.virt = 0x600000000UL,
+		.phys = 0x600000000UL,
+		.size = 0x010000000UL,
+		.attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) |
+			 PTE_BLOCK_NON_SHARE |
+			 PTE_BLOCK_PXN | PTE_BLOCK_UXN
+	}, {
+		.virt = 0x700000000UL,
+		.phys = 0x700000000UL,
+		.size = 0x010000000UL,
+		.attrs = PTE_BLOCK_MEMTYPE(MT_DEVICE_NGNRNE) |
+			 PTE_BLOCK_NON_SHARE |
+			 PTE_BLOCK_PXN | PTE_BLOCK_UXN
+#endif
 	}
 };
 
Signature · 1 line
2.43.7
#5 lore raw

PATCH 5/5 arm64: versal: Enable PCIe/NVMe support and add NVMe boot target

In reply to Pranav Sanwal

Enable the AMD Versal CPM/QDMA PCIe host controllers and NVMe storage in the virtual platform defconfig.

Add BOOT_TARGET_DEVICES_NVME to xilinx_versal.h so NVMe is included in the distro boot target list when CONFIG_NVME is enabled.

---
 configs/xilinx_versal_virt_defconfig | 6 ++++++
 include/configs/xilinx_versal.h      | 7 +++++++
 2 files changed, 13 insertions(+)
configs/xilinx_versal_virt_defconfig +6 -0
index a94468d9b9b..647e0455f85 100644
@@ -16,6 +16,7 @@ CONFIG_SYS_LOAD_ADDR=0x8000000
 CONFIG_ENV_OFFSET_REDUND=0x7F00000
 CONFIG_CMD_FRU=y
 CONFIG_DEFINE_TCM_OCM_MMAP=y
+CONFIG_PCI=y
 CONFIG_SYS_MEMTEST_START=0x00000000
 CONFIG_SYS_MEMTEST_END=0x00001000
 CONFIG_REMAKE_ELF=y
@@ -47,9 +48,11 @@ CONFIG_CMD_CLK=y
 CONFIG_CMD_DFU=y
 CONFIG_CMD_GPIO=y
 CONFIG_CMD_I2C=y
+CONFIG_CMD_LSBLK=y
 CONFIG_CMD_MMC=y
 CONFIG_MMC_SPEED_MODE_SET=y
 CONFIG_CMD_MTD=y
+CONFIG_CMD_PCI=y
 CONFIG_CMD_SF_TEST=y
 CONFIG_CMD_USB=y
 CONFIG_CMD_NFS=y
@@ -135,6 +138,9 @@ CONFIG_PHY_NATSEMI=y
 CONFIG_PHY_REALTEK=y
 CONFIG_PHY_TI_DP83867=y
 CONFIG_PHY_FIXED=y
+CONFIG_NVME_PCI=y
+CONFIG_PCIE_AMD_CPM=y
+CONFIG_PCIE_AMD_QDMA_PL=y
 CONFIG_POWER_DOMAIN=y
 CONFIG_ZYNQMP_POWER_DOMAIN=y
 CONFIG_RESET_ZYNQMP=y
include/configs/xilinx_versal.h +7 -0
index 64f123424cf..cea3340cf2e 100644
@@ -118,10 +118,17 @@
 #define BOOTENV_DEV_NAME_USB_THOR(devtypeu, devtypel, instance) \
 	""
 
+#if defined(CONFIG_NVME)
+# define BOOT_TARGET_DEVICES_NVME(func)	func(NVME, nvme, 0)
+#else
+# define BOOT_TARGET_DEVICES_NVME(func)
+#endif
+
 #define BOOT_TARGET_DEVICES(func) \
 	BOOT_TARGET_DEVICES_JTAG(func) \
 	BOOT_TARGET_DEVICES_MMC(func) \
 	BOOT_TARGET_DEVICES_XSPI(func) \
+	BOOT_TARGET_DEVICES_NVME(func) \
 	BOOT_TARGET_DEVICES_USB_DFU(func) \
 	BOOT_TARGET_DEVICES_USB_THOR(func) \
 	BOOT_TARGET_DEVICES_USB(func) \
Signature · 1 line
2.43.7
maintainer Applied
#6 lore raw
In reply to Pranav Sanwal
Quoted 31 lines · Pranav Sanwal

This series adds PCIe root port support for the AMD Versal CPM4, CPM5 and QDMA PL (soft IP) host bridges, and enables it along with NVMe storage on the xilinx_versal_virt platform.

Patch 1 adds a generic mmap config-space write helper for host bridges that only decode 32-bit accesses.
Patches 2 and 3 add the QDMA PL and CPM4/CPM5 host bridge drivers, respectively.
Patch 4 maps the CPM5 ECAM windows as device memory so config space accesses are not cached or reordered.
Patch 5 wires everything up in the versal_virt defconfig and adds NVMe as a distro boot target.

Pranav Sanwal (5):
pci: Add pci_generic_mmap_write_config32() helper
pci: Add support for AMD QDMA PL PCIe root port
pci: Add support for CPM4 and CPM5 PCIe controllers
arm64: versal: Add MMU mappings for CPM5 ECAM config space
arm64: versal: Enable PCIe/NVMe support and add NVMe boot target

  MAINTAINERS                          |  10 +
  arch/arm/mach-versal/cpu.c           |  20 ++
  configs/xilinx_versal_virt_defconfig |   6 +
  drivers/pci/Kconfig                  |  19 ++
  drivers/pci/Makefile                 |   2 +
  drivers/pci/pci-uclass.c             |  42 ++++
  drivers/pci/pcie_amd_cpm.c           | 363 +++++++++++++++++++++++++++
  drivers/pci/pcie_amd_qdma_pl.c       | 205 +++++++++++++++
  include/configs/xilinx_versal.h      |   7 +
  include/pci.h                        |  26 ++
  10 files changed, 700 insertions(+)
Quoted 3 lines

create mode 100644 drivers/pci/pcie_amd_cpm.c create mode 100644 drivers/pci/pcie_amd_qdma_pl.c

Applied.
M