Skip to contentSkip to conversation

Add StarFive JH7110 PCIe drvier support

PATCH Superseded Minda Chen 3 participants 10 messages Apr 11, 2023 – Apr 12, 2023

Subsystems (2)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
author
#8 lore raw

PATCH v4 2/3 configs: starfive-jh7110: Add support for PCIe host driver

In reply to Minda Chen

From: Mason Huo <[email protected]>

also add the nvme driver and rtl8169 support.

---
 configs/starfive_visionfive2_12a_defconfig | 10 ++++++++++
 configs/starfive_visionfive2_13b_defconfig | 10 ++++++++++
 2 files changed, 20 insertions(+)
configs/starfive_visionfive2_12a_defconfig +10 -0
index e0f98292ff..bd5b25f9d4 100644
@@ -16,6 +16,7 @@ CONFIG_SPL=y
 CONFIG_SPL_SPI_FLASH_SUPPORT=y
 CONFIG_SPL_SPI=y
 CONFIG_SYS_LOAD_ADDR=0x82000000
+CONFIG_SYS_PCI_64BIT=y
 CONFIG_TARGET_STARFIVE_VISIONFIVE2=y
 CONFIG_SPL_OPENSBI_LOAD_ADDR=0x40000000
 CONFIG_ARCH_RV64I=y
@@ -49,9 +50,12 @@ CONFIG_SPL_SPI_LOAD=y
 CONFIG_SYS_CBSIZE=256
 CONFIG_SYS_PBSIZE=276
 CONFIG_SYS_BOOTM_LEN=0x4000000
+CONFIG_CMD_PCI=y
 CONFIG_CMD_MEMINFO=y
 CONFIG_CMD_TFTPPUT=y
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
+CONFIG_REGMAP=y
+CONFIG_SYSCON=y
 CONFIG_SPL_CLK_COMPOSITE_CCF=y
 CONFIG_CLK_COMPOSITE_CCF=y
 CONFIG_SPL_CLK_JH7110=y
@@ -65,6 +69,12 @@ CONFIG_SPI_FLASH_EON=y
 CONFIG_SPI_FLASH_GIGADEVICE=y
 CONFIG_SPI_FLASH_ISSI=y
 CONFIG_SPI_FLASH_MACRONIX=y
+CONFIG_RTL8169=y
+CONFIG_NVME_PCI=y
+CONFIG_PCI=y
+CONFIG_DM_PCI_COMPAT=y
+CONFIG_PCI_REGION_MULTI_ENTRY=y
+CONFIG_PCIE_STARFIVE_JH7110=y
 CONFIG_PINCTRL=y
 CONFIG_PINCONF=y
 CONFIG_SPL_PINCTRL=y
configs/starfive_visionfive2_13b_defconfig +10 -0
index 550d0ff3ab..7247992796 100644
@@ -16,6 +16,7 @@ CONFIG_SPL=y
 CONFIG_SPL_SPI_FLASH_SUPPORT=y
 CONFIG_SPL_SPI=y
 CONFIG_SYS_LOAD_ADDR=0x82000000
+CONFIG_SYS_PCI_64BIT=y
 CONFIG_TARGET_STARFIVE_VISIONFIVE2=y
 CONFIG_SPL_OPENSBI_LOAD_ADDR=0x40000000
 CONFIG_ARCH_RV64I=y
@@ -50,8 +51,11 @@ CONFIG_SYS_CBSIZE=256
 CONFIG_SYS_PBSIZE=276
 CONFIG_SYS_BOOTM_LEN=0x4000000
 CONFIG_CMD_MEMINFO=y
+CONFIG_CMD_PCI=y
 CONFIG_CMD_TFTPPUT=y
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
+CONFIG_REGMAP=y
+CONFIG_SYSCON=y
 CONFIG_SPL_CLK_COMPOSITE_CCF=y
 CONFIG_CLK_COMPOSITE_CCF=y
 CONFIG_SPL_CLK_JH7110=y
@@ -65,6 +69,12 @@ CONFIG_SPI_FLASH_EON=y
 CONFIG_SPI_FLASH_GIGADEVICE=y
 CONFIG_SPI_FLASH_ISSI=y
 CONFIG_SPI_FLASH_MACRONIX=y
+CONFIG_RTL8169=y
+CONFIG_NVME_PCI=y
+CONFIG_PCI=y
+CONFIG_DM_PCI_COMPAT=y
+CONFIG_PCI_REGION_MULTI_ENTRY=y
+CONFIG_PCIE_STARFIVE_JH7110=y
 CONFIG_PINCTRL=y
 CONFIG_PINCONF=y
 CONFIG_SPL_PINCTRL=y
Signature · 1 line
2.17.1

Minda Chen

author
#0 lore raw

v4 cover Add StarFive JH7110 PCIe drvier support

This patchset needs to apply after patchset in [1]. These PCIe series patches are based on the JH7110 RISC-V SoC and VisionFive V2 board.

[1] https://patchwork.ozlabs.org/project/uboot/cover/[email protected]

The PCIe driver depends on gpio, pinctrl, clk and reset driver to do init. The PCIe dts configuation includes all these setting.

The PCIe drivers codes has been tested on the VisionFive V2 boards. The test devices includes M.2 NVMe SSD and Realtek 8169 Ethernet adapter.

previous patch version

v1: https://patchwork.ozlabs.org/project/uboot/cover/[email protected]/
v2: https://patchwork.ozlabs.org/project/uboot/cover/[email protected]/
v3: https://patchwork.ozlabs.org/project/uboot/cover/[email protected]/

changes
v4
patch 1
1. Remove the IDS_REVISION_ID macros.
2. Replace sec_busno to first_busno in starfive_pcie
3. Remove starfive_pcie_off_conf function.
4. Replace "imply" to "depends on" in PCIe Kconfig. 5 .Check sec_busno in starfive_pcie_addr_valid.

v3
patch 1
1. remove the read vendor ID delay
2. remove starfive_pcie_hide_rc_bar function. do not hide host
bridge BAR write.
3. Using PCIE_ECAM_OFFSET and PCI_CLASS_BRIDGE_PCI_NORMAL macros.
4. Add comments for bus and address limitation reason in function starfive_pcie_addr_valid
5. Change the multiple line comments in Line 373
6. Using gpio_request_by_name to get PCIe reset gpio,and using dm_gpio_set_value set GPIO value.
patch 2
1. support PCIeboth 12a and 13b vf2 board.
patch 3
1. reset dts change to reset-gpio.

v2
1. remove clock commit. The pcie clocks change has been includeded in [1].
2. Using GENMASK marco1 in patch1.
3. remove the syscon dts node in patch3. The syscon dts dts node has been included in [1].

---
Mason Huo (3):
starfive: pci: Add StarFive JH7110 pcie driver
configs: starfive-jh7110: Add support for PCIe host driver
riscv: dts: starfive: Enable PCIe host controller

 .../dts/jh7110-starfive-visionfive-2.dtsi     |  11 +
 arch/riscv/dts/jh7110.dtsi                    |  74 +++
 configs/starfive_visionfive2_12a_defconfig    |  10 +
 configs/starfive_visionfive2_13b_defconfig    |  10 +
 drivers/pci/Kconfig                           |   9 +
 drivers/pci/Makefile                          |   1 +
 drivers/pci/pcie_starfive_jh7110.c            | 465 ++++++++++++++++++
 7 files changed, 580 insertions(+)
 create mode 100644 drivers/pci/pcie_starfive_jh7110.c

base-commit: 41a88ad529b3943b1e465846eb24fe2c29203e35
prerequisite-patch-id: a84452ba131408ff842b65ae19a3a05f64b3ff60
prerequisite-patch-id: 4abb46d56dbaf36017b2866688f31a73c2cacd0d
prerequisite-patch-id: ef5e9d61f7392c7d2e5321aa8a10996ca8eae5fc
prerequisite-patch-id: 26dfdb401d680ecd9449acf09131cf4790e987b7
prerequisite-patch-id: a0cfa5d8cdb49bbe9d4739afa90991f882950881
prerequisite-patch-id: 61ceecb41cfd9029bfb267f7b7c5330a2d7e5edc
prerequisite-patch-id: d834ece14ffb525b8c3e661e78736692f33fca9b
prerequisite-patch-id: 88c9f9637335fdc4107d42db67b9110c9b73ead3
prerequisite-patch-id: ebc26ed3f97f7babc0fc805152acb578f5eff922
prerequisite-patch-id: e85a9ca22facac4d5b5562a2708b7332478d6db4
prerequisite-patch-id: ff5bd799e4cb905fef6cf1cd0ce757980e1d34b7
prerequisite-patch-id: a341d4e8a8e25cb711309123939235fcb3c10c1a
prerequisite-patch-id: 749e4eafb431857ee2bec4ac8e78a682f6a57588
prerequisite-patch-id: d4a3666ce2e7135402537cea60e9e6f0435230bf
prerequisite-patch-id: fa8ad1bfce7d6ab74ad63ff8d31dff3c2ccb5e43
prerequisite-patch-id: cc352634b8dfaf2d4243819362326a579e010c94
prerequisite-patch-id: 8e0fa3d987020ca06e22401bc4c69f834c38c3dc

Signature · 1 line
2.17.1
author
#9 lore raw

PATCH v4 3/3 riscv: dts: starfive: Enable PCIe host controller

From: Mason Huo <[email protected]>

Enable and add pinctrl configuration for PCIe host controller. Also add JH7110 stg syscon configuration.

---
 .../dts/jh7110-starfive-visionfive-2.dtsi     | 11 +++
 arch/riscv/dts/jh7110.dtsi                    | 74 +++++++++++++++++++
 2 files changed, 85 insertions(+)
arch/riscv/dts/jh7110-starfive-visionfive-2.dtsi +11 -0
index c6b6dfa940..12245576ac 100644
@@ -7,6 +7,7 @@
 
 #include "jh7110.dtsi"
 #include <dt-bindings/pinctrl/pinctrl-starfive-jh7110.h>
+#include <dt-bindings/gpio/gpio.h>
 / {
 	aliases {
 		serial0 = &uart0;
@@ -300,6 +301,16 @@
 	};
 };
 
+&pcie0 {
+	reset-gpios = <&sysgpio 26 GPIO_ACTIVE_LOW>;
+	status = "disabled";
+};
+
+&pcie1 {
+	reset-gpios = <&sysgpio 28 GPIO_ACTIVE_LOW>;
+	status = "okay";
+};
+
 &syscrg {
 	assigned-clocks = <&syscrg JH7110_SYSCLK_CPU_ROOT>,
 			  <&syscrg JH7110_SYSCLK_BUS_ROOT>,
arch/riscv/dts/jh7110.dtsi +74 -0

79 lines not on this page · raw message

Signature · 1 line
2.17.1
author
#1 lore raw

PATCH v4 1/3 starfive: pci: Add StarFive JH7110 pcie driver

In reply to Minda Chen

From: Mason Huo <[email protected]>

Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.

Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.

---
 drivers/pci/Kconfig                |   9 +
 drivers/pci/Makefile               |   1 +
 drivers/pci/pcie_starfive_jh7110.c | 465 +++++++++++++++++++++++++++++
 3 files changed, 475 insertions(+)
 create mode 100644 drivers/pci/pcie_starfive_jh7110.c
drivers/pci/Kconfig +9 -0
index ef328d2652..f37b6baa25 100644
@@ -374,4 +374,13 @@ config PCIE_UNIPHIER
 	  Say Y here if you want to enable PCIe controller support on
 	  UniPhier SoCs.
 
+config PCIE_STARFIVE_JH7110
+	bool "Enable Starfive JH7110 PCIe driver"
+	imply STARFIVE_JH7110
+	imply CLK_JH7110
+	imply RESET_JH7110
+	help
+	  Say Y here if you want to enable PCIe controller support on
+	  StarFive JH7110 SoC.
+
 endif
drivers/pci/Makefile +1 -0
index 49506e7ba5..bbe3323bb5 100644
@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o
 obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o
 obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o
 obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o
+obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.o
drivers/pci/pcie_starfive_jh7110.c new file +465 -0

465 lines not on this page · raw message

Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH v4 1/3] starfive: pci: Add StarFive JH7110 pcie driver

Quoted 496 lines · Minda Chen

From: Mason Huo <[email protected]>

Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.

Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.

---
 drivers/pci/Kconfig                |   9 +
 drivers/pci/Makefile               |   1 +
 drivers/pci/pcie_starfive_jh7110.c | 465 +++++++++++++++++++++++++++++
 3 files changed, 475 insertions(+)
 create mode 100644 drivers/pci/pcie_starfive_jh7110.c
drivers/pci/Kconfig +9 -0
index ef328d2652..f37b6baa25 100644
@@ -374,4 +374,13 @@ config PCIE_UNIPHIER
          Say Y here if you want to enable PCIe controller support on
          UniPhier SoCs.
 
+config PCIE_STARFIVE_JH7110
+       bool "Enable Starfive JH7110 PCIe driver"
+       imply STARFIVE_JH7110
+       imply CLK_JH7110
+       imply RESET_JH7110
+       help
+         Say Y here if you want to enable PCIe controller support on
+         StarFive JH7110 SoC.
+
 endif
drivers/pci/Makefile +1 -0
index 49506e7ba5..bbe3323bb5 100644
@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o
 obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o
 obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o
 obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o
+obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.o
drivers/pci/pcie_starfive_jh7110.c new file +444 -0
new file mode 100644
index 0000000000..130181013e
@@ -0,0 +1,465 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * StarFive PLDA PCIe host controller driver
+ *
+ * Copyright (c) 2023 Starfive, Inc.
+ * Author: Mason Huo <[email protected]>
+ *
+ */
+
+#include <common.h>
+#include <clk.h>
+#include <dm.h>
+#include <generic-phy.h>
+#include <pci.h>
+#include <pci_ids.h>
+#include <power-domain.h>
+#include <regmap.h>
+#include <reset.h>
+#include <syscon.h>
+#include <asm/global_data.h>
+#include <asm/io.h>
+#include <asm-generic/gpio.h>
+#include <dm/device_compat.h>
+#include <dm/pinctrl.h>
+#include <linux/delay.h>
+#include <linux/iopoll.h>
+#include <power/regulator.h>
+
+DECLARE_GLOBAL_DATA_PTR;
+
+#define GEN_SETTINGS                   0x80
+#define PCIE_PCI_IDS                   0x9C
+#define PCIE_WINROM                    0xFC
+#define PMSG_SUPPORT_RX                        0x3F0
+#define PCI_MISC                       0xB4
+
+#define PLDA_EP_ENABLE                 0
+#define PLDA_RP_ENABLE                 1
+
+#define IDS_CLASS_CODE_SHIFT           8
+
+#define PREF_MEM_WIN_64_SUPPORT                BIT(3)
+#define PMSG_LTR_SUPPORT               BIT(2)
+#define PLDA_FUNCTION_DIS              BIT(15)
+#define PLDA_FUNC_NUM                  4
+#define PLDA_PHY_FUNC_SHIFT            9
+
+#define XR3PCI_ATR_AXI4_SLV0           0x800
+#define XR3PCI_ATR_SRC_ADDR_LOW                0x0
+#define XR3PCI_ATR_SRC_ADDR_HIGH       0x4
+#define XR3PCI_ATR_TRSL_ADDR_LOW       0x8
+#define XR3PCI_ATR_TRSL_ADDR_HIGH      0xc
+#define XR3PCI_ATR_TRSL_PARAM          0x10
+#define XR3PCI_ATR_TABLE_OFFSET                0x20
+#define XR3PCI_ATR_MAX_TABLE_NUM       8
+
+#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT  1
+#define XR3PCI_ATR_SRC_ADDR_MASK       GENMASK(31, 12)
+#define XR3PCI_ATR_TRSL_ADDR_MASK      GENMASK(31, 12)
+#define XR3_PCI_ECAM_SIZE              28
+#define XR3PCI_ATR_TRSL_DIR            BIT(22)
+/* IDs used in the XR3PCI_ATR_TRSL_PARAM */
+#define XR3PCI_ATR_TRSLID_PCIE_MEMORY  0x0
+#define XR3PCI_ATR_TRSLID_PCIE_CONFIG  0x1
+
+/* system control */
+#define STG_SYSCON_K_RP_NEP_MASK               BIT(8)
+#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK       GENMASK(22, 8)
+#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT      8
+#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK       GENMASK(14, 0)
+#define STG_SYSCON_CLKREQ_MASK                 BIT(22)
+#define STG_SYSCON_CKREF_SRC_SHIFT             18
+#define STG_SYSCON_CKREF_SRC_MASK              GENMASK(19, 18)
+
+struct starfive_pcie {
+       struct udevice *dev;
+
+       void __iomem *reg_base;
+       void __iomem *cfg_base;
+
+       struct regmap *regmap;
+       u32 stg_arfun;
+       u32 stg_awfun;
+       u32 stg_rp_nep;
+
+       struct clk_bulk clks;
+       struct reset_ctl_bulk   rsts;
+       struct gpio_desc        reset_gpio;
+
+       int atr_table_num;
+       int sec_busno;
+};
+
+static bool starfive_pcie_addr_valid(pci_dev_t bdf, struct starfive_pcie *priv)
+{
+       /*
+        * Single device limitation.
+        * For JH7110 SoC limitation, one bus can only connnect one device.
+        * And PCIe controller contain HW issue that secondary bus of
+        * host bridge emumerate duplicate devices.
+        * Only can access device 0 in secondary bus.
+        */
+       if (PCI_BUS(bdf) == priv->sec_busno && PCI_DEV(bdf) > 0)
+               return false;
+
+       return true;
+}
+
+static int starfive_pcie_conf_address(const struct udevice *udev, pci_dev_t bdf,
+                                     uint offset, void **paddr)
+{
+       struct starfive_pcie *priv = dev_get_priv(udev);
+       int where = PCIE_ECAM_OFFSET(PCI_BUS(bdf), PCI_DEV(bdf),
+                                    PCI_FUNC(bdf), offset);
+
+       if (!starfive_pcie_addr_valid(bdf, priv))
+               return -ENODEV;
+
+       *paddr = (void *)(priv->cfg_base + where);
+       return 0;
+}
+
+static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf,
+                                    uint offset, ulong *valuep,
+                                    enum pci_size_t size)
+{
+       return pci_generic_mmap_read_config(udev, starfive_pcie_conf_address,
+                                           bdf, offset, valuep, size);
+}
+
+int starfive_pcie_config_write(struct udevice *udev, pci_dev_t bdf,
+                              uint offset, ulong value,
+                              enum pci_size_t size)
+{
+       struct starfive_pcie *priv = dev_get_priv(udev);
+       int ret;
+
+       ret = pci_generic_mmap_write_config(udev, starfive_pcie_conf_address,
+                                           bdf, offset, value, size);
+
+       if (!ret && offset == PCI_SECONDARY_BUS) {
+               priv->sec_busno = value & 0xff;
+               debug("Secondary bus number was changed to %d\n",
+                     priv->sec_busno);
+       }
+       return ret;
+}
+
+static int starfive_pcie_set_atr_entry(struct starfive_pcie *priv, phys_addr_t src_addr,
+                                      phys_addr_t trsl_addr, size_t window_size,
+                                      int trsl_param)
+{
+       void __iomem *base =
+               priv->reg_base + XR3PCI_ATR_AXI4_SLV0;
+
+       /* Support AXI4 Slave 0 Address Translation Tables 0-7. */
+       if (priv->atr_table_num >= XR3PCI_ATR_MAX_TABLE_NUM) {
+               dev_err(priv->dev, "ATR table number %d exceeds max num\n",
+                       priv->atr_table_num);
+               return -EINVAL;
+       }
+       base +=  XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num;
+       priv->atr_table_num++;
+
+       /*
+        * X3PCI_ATR_SRC_ADDR_LOW:
+        *   - bit 0: enable entry,
+        *   - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1)
+        *   - bits 7-11: reserved
+        *   - bits 12-31: start of source address
+        */
+       writel((lower_32_bits(src_addr) & XR3PCI_ATR_SRC_ADDR_MASK) |
+                       (fls(window_size) - 1) << XR3PCI_ATR_SRC_WIN_SIZE_SHIFT | 1,
+                       base + XR3PCI_ATR_SRC_ADDR_LOW);
+       writel(upper_32_bits(src_addr), base + XR3PCI_ATR_SRC_ADDR_HIGH);
+       writel((lower_32_bits(trsl_addr) & XR3PCI_ATR_TRSL_ADDR_MASK),
+              base + XR3PCI_ATR_TRSL_ADDR_LOW);
+       writel(upper_32_bits(trsl_addr), base + XR3PCI_ATR_TRSL_ADDR_HIGH);
+       writel(trsl_param, base + XR3PCI_ATR_TRSL_PARAM);
+
+       dev_dbg(priv->dev, "ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n",
+               src_addr, (trsl_param & XR3PCI_ATR_TRSL_DIR) ? "<-" : "->",
+               trsl_addr, (u64)window_size, trsl_param);
+       return 0;
+}
+
+static int starfive_pcie_atr_init(struct starfive_pcie *priv)
+{
+       struct udevice *ctlr = pci_get_controller(priv->dev);
+       struct pci_controller *hose = dev_get_uclass_priv(ctlr);
+       int i, ret;
+
+       /*
+        * As the two host bridges in JH7110 soc have the same default
+        * address translation table, this cause the second root port can't
+        * access it's host bridge config space correctly.
+        * To workaround, config the ATR of host bridge config space by SW.
+        */
+
+       ret = starfive_pcie_set_atr_entry(priv,
+                                         (phys_addr_t)priv->cfg_base,
+                                         0,
+                                         1 << XR3_PCI_ECAM_SIZE,
+                                         XR3PCI_ATR_TRSLID_PCIE_CONFIG);
+       if (ret)
+               return ret;
+
+       for (i = 0; i < hose->region_count; i++) {
+               if (hose->regions[i].flags == PCI_REGION_SYS_MEMORY)
+                       continue;
+
+               /* Only support identity mappings. */
+               if (hose->regions[i].bus_start !=
+                   hose->regions[i].phys_start)
+                       return -EINVAL;
+
+               ret = starfive_pcie_set_atr_entry(priv,
+                                                 hose->regions[i].phys_start,
+                                                 hose->regions[i].bus_start,
+                                                 hose->regions[i].size,
+                                                 XR3PCI_ATR_TRSLID_PCIE_MEMORY);
+               if (ret)
+                       return ret;
+       }
+
+       return 0;
+}
+
+static int starfive_pcie_get_syscon(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       struct udevice *syscon;
+       struct ofnode_phandle_args syscfg_phandle;
+       u32 cells[4];
+       int ret;
+
+       /* get corresponding syscon phandle */
+       ret = dev_read_phandle_with_args(dev, "starfive,stg-syscon", NULL, 0, 0,
+                                        &syscfg_phandle);
+
+       if (ret < 0) {
+               dev_err(dev, "Can't get syscfg phandle: %d\n", ret);
+               return ret;
+       }
+
+       ret = uclass_get_device_by_ofnode(UCLASS_SYSCON, syscfg_phandle.node,
+                                         &syscon);
+       if (ret) {
+               dev_err(dev, "Unable to find syscon device (%d)\n", ret);
+               return ret;
+       }
+
+       priv->regmap = syscon_get_regmap(syscon);
+       if (!priv->regmap) {
+               dev_err(dev, "Unable to find regmap\n");
+               return -ENODEV;
+       }
+
+       /* get syscon register offset */
+       ret = dev_read_u32_array(dev, "starfive,stg-syscon",
+                                cells, ARRAY_SIZE(cells));
+       if (ret) {
+               dev_err(dev, "Get syscon register err %d\n", ret);
+               return -EINVAL;
+       }
+
+       dev_dbg(dev, "Get syscon values: %x, %x, %x\n",
+               cells[1], cells[2], cells[3]);
+       priv->stg_arfun = cells[1];
+       priv->stg_awfun = cells[2];
+       priv->stg_rp_nep = cells[3];
+
+       return 0;
+}
+
+static int starfive_pcie_parse_dt(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       int ret;
+
+       priv->reg_base = (void *)dev_read_addr_name(dev, "reg");
+       if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE) {
+               dev_err(dev, "Missing required reg address range\n");
+               return -EINVAL;
+       }
+
+       priv->cfg_base = (void *)dev_read_addr_name(dev, "config");
+       if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE) {
+               dev_err(dev, "Missing required config address range");
+               return -EINVAL;
+       }
+
+       ret = starfive_pcie_get_syscon(dev);
+       if (ret) {
+               dev_err(dev, "Can't get syscon: %d\n", ret);
+               return ret;
+       }
+
+       ret = reset_get_bulk(dev, &priv->rsts);
+       if (ret) {
+               dev_err(dev, "Can't get reset: %d\n", ret);
+               return ret;
+       }
+
+       ret = clk_get_bulk(dev, &priv->clks);
+       if (ret) {
+               dev_err(dev, "Can't get clock: %d\n", ret);
+               return ret;
+       }
+
+       ret = gpio_request_by_name(dev, "reset-gpios", 0, &priv->reset_gpio,
+                                  GPIOD_IS_OUT);
+       if (ret) {
+               dev_err(dev, "Can't get reset-gpio: %d\n", ret);
+               return ret;
+       }
+
+       if (!dm_gpio_is_valid(&priv->reset_gpio)) {
+               dev_err(dev, "reset-gpio is not valid\n");
+               return -EINVAL;
+       }
+       return 0;
+}
+
+static int starfive_pcie_init_port(struct udevice *dev)
+{
+       int ret, i;
+       unsigned int value;
+       struct starfive_pcie *priv = dev_get_priv(dev);
+
+       ret = clk_enable_bulk(&priv->clks);
+       if (ret) {
+               dev_err(dev, "Failed to enable clks (ret=%d)\n", ret);
+               return ret;
+       }
+
+       ret = reset_deassert_bulk(&priv->rsts);
+       if (ret) {
+               dev_err(dev, "Failed to deassert resets (ret=%d)\n", ret);
+               goto err_deassert_clk;
+       }
+
+       dm_gpio_set_value(&priv->reset_gpio, 1);
+       /* Disable physical functions except #0 */
+       for (i = 1; i < PLDA_FUNC_NUM; i++) {
+               regmap_update_bits(priv->regmap,
+                                  priv->stg_arfun,
+                                  STG_SYSCON_AXI4_SLVL_ARFUNC_MASK,
+                                  (i << PLDA_PHY_FUNC_SHIFT) <<
+                                  STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT);
+               regmap_update_bits(priv->regmap,
+                                  priv->stg_awfun,
+                                  STG_SYSCON_AXI4_SLVL_AWFUNC_MASK,
+                                  i << PLDA_PHY_FUNC_SHIFT);
+
+               value = readl(priv->reg_base + PCI_MISC);
+               value |= PLDA_FUNCTION_DIS;
+               writel(value, priv->reg_base + PCI_MISC);
+       }
+
+       /* Disable physical functions */
+       regmap_update_bits(priv->regmap,
+                          priv->stg_arfun,
+                          STG_SYSCON_AXI4_SLVL_ARFUNC_MASK,
+                          0);
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_AXI4_SLVL_AWFUNC_MASK,
+                          0);
+
+       /* Enable root port */
+       value = readl(priv->reg_base + GEN_SETTINGS);
+       value |= PLDA_RP_ENABLE;
+       writel(value, priv->reg_base + GEN_SETTINGS);
+
+       /* PCIe PCI Standard Configuration Identification Settings. */
+       value = readl(priv->reg_base + PCIE_PCI_IDS);
+       value &= 0xff;
+       value |= (PCI_CLASS_BRIDGE_PCI_NORMAL << IDS_CLASS_CODE_SHIFT);
+       writel(value, priv->reg_base + PCIE_PCI_IDS);
+
+       /*
+        * The LTR message forwarding of PCIe Message Reception was set by core
+        * as default, but the forward id & addr are also need to be reset.
+        * If we do not disable LTR message forwarding here, or set a legal
+        * forwarding address, the kernel will get stuck after this driver probe.
+        * To workaround, disable the LTR message forwarding support on
+        * PCIe Message Reception.
+        */
+       value = readl(priv->reg_base + PMSG_SUPPORT_RX);
+       value &= ~PMSG_LTR_SUPPORT;
+       writel(value, priv->reg_base + PMSG_SUPPORT_RX);
+
+       /* Prefetchable memory window 64-bit addressing support */
+       value = readl(priv->reg_base + PCIE_WINROM);
+       value |= PREF_MEM_WIN_64_SUPPORT;
+       writel(value, priv->reg_base + PCIE_WINROM);
+
+       starfive_pcie_atr_init(priv);
+
+       dm_gpio_set_value(&priv->reset_gpio, 0);
+       /* Ensure that PERST in default at least 300 ms */
+       mdelay(300);
+
+       return 0;
+
+err_deassert_clk:
+       clk_disable_bulk(&priv->clks);
+       return ret;
+}
+
+static int starfive_pcie_probe(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       int ret;
+
+       priv->atr_table_num = 0;
+       priv->dev = dev;
+
+       ret = starfive_pcie_parse_dt(dev);
+       if (ret)
+               return ret;
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_rp_nep,
+                          STG_SYSCON_K_RP_NEP_MASK,
+                          STG_SYSCON_K_RP_NEP_MASK);
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_CKREF_SRC_MASK,
+                          2 << STG_SYSCON_CKREF_SRC_SHIFT);
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_CLKREQ_MASK,
+                          STG_SYSCON_CLKREQ_MASK);
+
+       ret = starfive_pcie_init_port(dev);
+       if (ret)
+               return ret;
+
+       dev_err(dev, "Starfive PCIe bus probed.\n");
+

drivers/pci/pcie_starfive_jh7110.c line 454

Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10 -0
+       return 0;
+}
+
+static const struct dm_pci_ops starfive_pcie_ops = {
+       .read_config    = starfive_pcie_config_read,
+       .write_config   = starfive_pcie_config_write,
+};
+
+static const struct udevice_id starfive_pcie_ids[] = {
+       { .compatible = "starfive,jh7110-pcie" },

My previous comments were not addressed.

Is this PCIe controller a StarFive IP? Or some 3rd party licensed IP? Is this compatible string approved by the DT community?

drivers/pci/pcie_starfive_jh7110.c line 465

Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11 -0
+       { }
+};
+
+U_BOOT_DRIVER(starfive_pcie_drv) = {
+       .name                   = "starfive_7110_pcie",
+       .id                     = UCLASS_PCI,
+       .of_match               = starfive_pcie_ids,
+       .ops                    = &starfive_pcie_ops,
+       .probe                  = starfive_pcie_probe,
+       .priv_auto      = sizeof(struct starfive_pcie),
+};

Regards,
Bin

author
#3 lore raw

Re: [PATCH v4 1/3] starfive: pci: Add StarFive JH7110 pcie driver

Quoted 501 lines · Bin Meng
Quoted 500 lines · Minda Chen

From: Mason Huo <[email protected]>

Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.

Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.

---
 drivers/pci/Kconfig                |   9 +
 drivers/pci/Makefile               |   1 +
 drivers/pci/pcie_starfive_jh7110.c | 465 +++++++++++++++++++++++++++++
 3 files changed, 475 insertions(+)
 create mode 100644 drivers/pci/pcie_starfive_jh7110.c
drivers/pci/Kconfig +9 -0
index ef328d2652..f37b6baa25 100644
@@ -374,4 +374,13 @@ config PCIE_UNIPHIER
          Say Y here if you want to enable PCIe controller support on
          UniPhier SoCs.
 
+config PCIE_STARFIVE_JH7110
+       bool "Enable Starfive JH7110 PCIe driver"
+       imply STARFIVE_JH7110
+       imply CLK_JH7110
+       imply RESET_JH7110
+       help
+         Say Y here if you want to enable PCIe controller support on
+         StarFive JH7110 SoC.
+
 endif
drivers/pci/Makefile +1 -0
index 49506e7ba5..bbe3323bb5 100644
@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o
 obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o
 obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o
 obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o
+obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.o
drivers/pci/pcie_starfive_jh7110.c new file +448 -0
new file mode 100644
index 0000000000..130181013e
@@ -0,0 +1,465 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * StarFive PLDA PCIe host controller driver
+ *
+ * Copyright (c) 2023 Starfive, Inc.
+ * Author: Mason Huo <[email protected]>
+ *
+ */
+
+#include <common.h>
+#include <clk.h>
+#include <dm.h>
+#include <generic-phy.h>
+#include <pci.h>
+#include <pci_ids.h>
+#include <power-domain.h>
+#include <regmap.h>
+#include <reset.h>
+#include <syscon.h>
+#include <asm/global_data.h>
+#include <asm/io.h>
+#include <asm-generic/gpio.h>
+#include <dm/device_compat.h>
+#include <dm/pinctrl.h>
+#include <linux/delay.h>
+#include <linux/iopoll.h>
+#include <power/regulator.h>
+
+DECLARE_GLOBAL_DATA_PTR;
+
+#define GEN_SETTINGS                   0x80
+#define PCIE_PCI_IDS                   0x9C
+#define PCIE_WINROM                    0xFC
+#define PMSG_SUPPORT_RX                        0x3F0
+#define PCI_MISC                       0xB4
+
+#define PLDA_EP_ENABLE                 0
+#define PLDA_RP_ENABLE                 1
+
+#define IDS_CLASS_CODE_SHIFT           8
+
+#define PREF_MEM_WIN_64_SUPPORT                BIT(3)
+#define PMSG_LTR_SUPPORT               BIT(2)
+#define PLDA_FUNCTION_DIS              BIT(15)
+#define PLDA_FUNC_NUM                  4
+#define PLDA_PHY_FUNC_SHIFT            9
+
+#define XR3PCI_ATR_AXI4_SLV0           0x800
+#define XR3PCI_ATR_SRC_ADDR_LOW                0x0
+#define XR3PCI_ATR_SRC_ADDR_HIGH       0x4
+#define XR3PCI_ATR_TRSL_ADDR_LOW       0x8
+#define XR3PCI_ATR_TRSL_ADDR_HIGH      0xc
+#define XR3PCI_ATR_TRSL_PARAM          0x10
+#define XR3PCI_ATR_TABLE_OFFSET                0x20
+#define XR3PCI_ATR_MAX_TABLE_NUM       8
+
+#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT  1
+#define XR3PCI_ATR_SRC_ADDR_MASK       GENMASK(31, 12)
+#define XR3PCI_ATR_TRSL_ADDR_MASK      GENMASK(31, 12)
+#define XR3_PCI_ECAM_SIZE              28
+#define XR3PCI_ATR_TRSL_DIR            BIT(22)
+/* IDs used in the XR3PCI_ATR_TRSL_PARAM */
+#define XR3PCI_ATR_TRSLID_PCIE_MEMORY  0x0
+#define XR3PCI_ATR_TRSLID_PCIE_CONFIG  0x1
+
+/* system control */
+#define STG_SYSCON_K_RP_NEP_MASK               BIT(8)
+#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK       GENMASK(22, 8)
+#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT      8
+#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK       GENMASK(14, 0)
+#define STG_SYSCON_CLKREQ_MASK                 BIT(22)
+#define STG_SYSCON_CKREF_SRC_SHIFT             18
+#define STG_SYSCON_CKREF_SRC_MASK              GENMASK(19, 18)
+
+struct starfive_pcie {
+       struct udevice *dev;
+
+       void __iomem *reg_base;
+       void __iomem *cfg_base;
+
+       struct regmap *regmap;
+       u32 stg_arfun;
+       u32 stg_awfun;
+       u32 stg_rp_nep;
+
+       struct clk_bulk clks;
+       struct reset_ctl_bulk   rsts;
+       struct gpio_desc        reset_gpio;
+
+       int atr_table_num;
+       int sec_busno;
+};
+
+static bool starfive_pcie_addr_valid(pci_dev_t bdf, struct starfive_pcie *priv)
+{
+       /*
+        * Single device limitation.
+        * For JH7110 SoC limitation, one bus can only connnect one device.
+        * And PCIe controller contain HW issue that secondary bus of
+        * host bridge emumerate duplicate devices.
+        * Only can access device 0 in secondary bus.
+        */
+       if (PCI_BUS(bdf) == priv->sec_busno && PCI_DEV(bdf) > 0)
+               return false;
+
+       return true;
+}
+
+static int starfive_pcie_conf_address(const struct udevice *udev, pci_dev_t bdf,
+                                     uint offset, void **paddr)
+{
+       struct starfive_pcie *priv = dev_get_priv(udev);
+       int where = PCIE_ECAM_OFFSET(PCI_BUS(bdf), PCI_DEV(bdf),
+                                    PCI_FUNC(bdf), offset);
+
+       if (!starfive_pcie_addr_valid(bdf, priv))
+               return -ENODEV;
+
+       *paddr = (void *)(priv->cfg_base + where);
+       return 0;
+}
+
+static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf,
+                                    uint offset, ulong *valuep,
+                                    enum pci_size_t size)
+{
+       return pci_generic_mmap_read_config(udev, starfive_pcie_conf_address,
+                                           bdf, offset, valuep, size);
+}
+
+int starfive_pcie_config_write(struct udevice *udev, pci_dev_t bdf,
+                              uint offset, ulong value,
+                              enum pci_size_t size)
+{
+       struct starfive_pcie *priv = dev_get_priv(udev);
+       int ret;
+
+       ret = pci_generic_mmap_write_config(udev, starfive_pcie_conf_address,
+                                           bdf, offset, value, size);
+
+       if (!ret && offset == PCI_SECONDARY_BUS) {
+               priv->sec_busno = value & 0xff;
+               debug("Secondary bus number was changed to %d\n",
+                     priv->sec_busno);
+       }
+       return ret;
+}
+
+static int starfive_pcie_set_atr_entry(struct starfive_pcie *priv, phys_addr_t src_addr,
+                                      phys_addr_t trsl_addr, size_t window_size,
+                                      int trsl_param)
+{
+       void __iomem *base =
+               priv->reg_base + XR3PCI_ATR_AXI4_SLV0;
+
+       /* Support AXI4 Slave 0 Address Translation Tables 0-7. */
+       if (priv->atr_table_num >= XR3PCI_ATR_MAX_TABLE_NUM) {
+               dev_err(priv->dev, "ATR table number %d exceeds max num\n",
+                       priv->atr_table_num);
+               return -EINVAL;
+       }
+       base +=  XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num;
+       priv->atr_table_num++;
+
+       /*
+        * X3PCI_ATR_SRC_ADDR_LOW:
+        *   - bit 0: enable entry,
+        *   - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1)
+        *   - bits 7-11: reserved
+        *   - bits 12-31: start of source address
+        */
+       writel((lower_32_bits(src_addr) & XR3PCI_ATR_SRC_ADDR_MASK) |
+                       (fls(window_size) - 1) << XR3PCI_ATR_SRC_WIN_SIZE_SHIFT | 1,
+                       base + XR3PCI_ATR_SRC_ADDR_LOW);
+       writel(upper_32_bits(src_addr), base + XR3PCI_ATR_SRC_ADDR_HIGH);
+       writel((lower_32_bits(trsl_addr) & XR3PCI_ATR_TRSL_ADDR_MASK),
+              base + XR3PCI_ATR_TRSL_ADDR_LOW);
+       writel(upper_32_bits(trsl_addr), base + XR3PCI_ATR_TRSL_ADDR_HIGH);
+       writel(trsl_param, base + XR3PCI_ATR_TRSL_PARAM);
+
+       dev_dbg(priv->dev, "ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n",
+               src_addr, (trsl_param & XR3PCI_ATR_TRSL_DIR) ? "<-" : "->",
+               trsl_addr, (u64)window_size, trsl_param);
+       return 0;
+}
+
+static int starfive_pcie_atr_init(struct starfive_pcie *priv)
+{
+       struct udevice *ctlr = pci_get_controller(priv->dev);
+       struct pci_controller *hose = dev_get_uclass_priv(ctlr);
+       int i, ret;
+
+       /*
+        * As the two host bridges in JH7110 soc have the same default
+        * address translation table, this cause the second root port can't
+        * access it's host bridge config space correctly.
+        * To workaround, config the ATR of host bridge config space by SW.
+        */
+
+       ret = starfive_pcie_set_atr_entry(priv,
+                                         (phys_addr_t)priv->cfg_base,
+                                         0,
+                                         1 << XR3_PCI_ECAM_SIZE,
+                                         XR3PCI_ATR_TRSLID_PCIE_CONFIG);
+       if (ret)
+               return ret;
+
+       for (i = 0; i < hose->region_count; i++) {
+               if (hose->regions[i].flags == PCI_REGION_SYS_MEMORY)
+                       continue;
+
+               /* Only support identity mappings. */
+               if (hose->regions[i].bus_start !=
+                   hose->regions[i].phys_start)
+                       return -EINVAL;
+
+               ret = starfive_pcie_set_atr_entry(priv,
+                                                 hose->regions[i].phys_start,
+                                                 hose->regions[i].bus_start,
+                                                 hose->regions[i].size,
+                                                 XR3PCI_ATR_TRSLID_PCIE_MEMORY);
+               if (ret)
+                       return ret;
+       }
+
+       return 0;
+}
+
+static int starfive_pcie_get_syscon(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       struct udevice *syscon;
+       struct ofnode_phandle_args syscfg_phandle;
+       u32 cells[4];
+       int ret;
+
+       /* get corresponding syscon phandle */
+       ret = dev_read_phandle_with_args(dev, "starfive,stg-syscon", NULL, 0, 0,
+                                        &syscfg_phandle);
+
+       if (ret < 0) {
+               dev_err(dev, "Can't get syscfg phandle: %d\n", ret);
+               return ret;
+       }
+
+       ret = uclass_get_device_by_ofnode(UCLASS_SYSCON, syscfg_phandle.node,
+                                         &syscon);
+       if (ret) {
+               dev_err(dev, "Unable to find syscon device (%d)\n", ret);
+               return ret;
+       }
+
+       priv->regmap = syscon_get_regmap(syscon);
+       if (!priv->regmap) {
+               dev_err(dev, "Unable to find regmap\n");
+               return -ENODEV;
+       }
+
+       /* get syscon register offset */
+       ret = dev_read_u32_array(dev, "starfive,stg-syscon",
+                                cells, ARRAY_SIZE(cells));
+       if (ret) {
+               dev_err(dev, "Get syscon register err %d\n", ret);
+               return -EINVAL;
+       }
+
+       dev_dbg(dev, "Get syscon values: %x, %x, %x\n",
+               cells[1], cells[2], cells[3]);
+       priv->stg_arfun = cells[1];
+       priv->stg_awfun = cells[2];
+       priv->stg_rp_nep = cells[3];
+
+       return 0;
+}
+
+static int starfive_pcie_parse_dt(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       int ret;
+
+       priv->reg_base = (void *)dev_read_addr_name(dev, "reg");
+       if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE) {
+               dev_err(dev, "Missing required reg address range\n");
+               return -EINVAL;
+       }
+
+       priv->cfg_base = (void *)dev_read_addr_name(dev, "config");
+       if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE) {
+               dev_err(dev, "Missing required config address range");
+               return -EINVAL;
+       }
+
+       ret = starfive_pcie_get_syscon(dev);
+       if (ret) {
+               dev_err(dev, "Can't get syscon: %d\n", ret);
+               return ret;
+       }
+
+       ret = reset_get_bulk(dev, &priv->rsts);
+       if (ret) {
+               dev_err(dev, "Can't get reset: %d\n", ret);
+               return ret;
+       }
+
+       ret = clk_get_bulk(dev, &priv->clks);
+       if (ret) {
+               dev_err(dev, "Can't get clock: %d\n", ret);
+               return ret;
+       }
+
+       ret = gpio_request_by_name(dev, "reset-gpios", 0, &priv->reset_gpio,
+                                  GPIOD_IS_OUT);
+       if (ret) {
+               dev_err(dev, "Can't get reset-gpio: %d\n", ret);
+               return ret;
+       }
+
+       if (!dm_gpio_is_valid(&priv->reset_gpio)) {
+               dev_err(dev, "reset-gpio is not valid\n");
+               return -EINVAL;
+       }
+       return 0;
+}
+
+static int starfive_pcie_init_port(struct udevice *dev)
+{
+       int ret, i;
+       unsigned int value;
+       struct starfive_pcie *priv = dev_get_priv(dev);
+
+       ret = clk_enable_bulk(&priv->clks);
+       if (ret) {
+               dev_err(dev, "Failed to enable clks (ret=%d)\n", ret);
+               return ret;
+       }
+
+       ret = reset_deassert_bulk(&priv->rsts);
+       if (ret) {
+               dev_err(dev, "Failed to deassert resets (ret=%d)\n", ret);
+               goto err_deassert_clk;
+       }
+
+       dm_gpio_set_value(&priv->reset_gpio, 1);
+       /* Disable physical functions except #0 */
+       for (i = 1; i < PLDA_FUNC_NUM; i++) {
+               regmap_update_bits(priv->regmap,
+                                  priv->stg_arfun,
+                                  STG_SYSCON_AXI4_SLVL_ARFUNC_MASK,
+                                  (i << PLDA_PHY_FUNC_SHIFT) <<
+                                  STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT);
+               regmap_update_bits(priv->regmap,
+                                  priv->stg_awfun,
+                                  STG_SYSCON_AXI4_SLVL_AWFUNC_MASK,
+                                  i << PLDA_PHY_FUNC_SHIFT);
+
+               value = readl(priv->reg_base + PCI_MISC);
+               value |= PLDA_FUNCTION_DIS;
+               writel(value, priv->reg_base + PCI_MISC);
+       }
+
+       /* Disable physical functions */
+       regmap_update_bits(priv->regmap,
+                          priv->stg_arfun,
+                          STG_SYSCON_AXI4_SLVL_ARFUNC_MASK,
+                          0);
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_AXI4_SLVL_AWFUNC_MASK,
+                          0);
+
+       /* Enable root port */
+       value = readl(priv->reg_base + GEN_SETTINGS);
+       value |= PLDA_RP_ENABLE;
+       writel(value, priv->reg_base + GEN_SETTINGS);
+
+       /* PCIe PCI Standard Configuration Identification Settings. */
+       value = readl(priv->reg_base + PCIE_PCI_IDS);
+       value &= 0xff;
+       value |= (PCI_CLASS_BRIDGE_PCI_NORMAL << IDS_CLASS_CODE_SHIFT);
+       writel(value, priv->reg_base + PCIE_PCI_IDS);
+
+       /*
+        * The LTR message forwarding of PCIe Message Reception was set by core
+        * as default, but the forward id & addr are also need to be reset.
+        * If we do not disable LTR message forwarding here, or set a legal
+        * forwarding address, the kernel will get stuck after this driver probe.
+        * To workaround, disable the LTR message forwarding support on
+        * PCIe Message Reception.
+        */
+       value = readl(priv->reg_base + PMSG_SUPPORT_RX);
+       value &= ~PMSG_LTR_SUPPORT;
+       writel(value, priv->reg_base + PMSG_SUPPORT_RX);
+
+       /* Prefetchable memory window 64-bit addressing support */
+       value = readl(priv->reg_base + PCIE_WINROM);
+       value |= PREF_MEM_WIN_64_SUPPORT;
+       writel(value, priv->reg_base + PCIE_WINROM);
+
+       starfive_pcie_atr_init(priv);
+
+       dm_gpio_set_value(&priv->reset_gpio, 0);
+       /* Ensure that PERST in default at least 300 ms */
+       mdelay(300);
+
+       return 0;
+
+err_deassert_clk:
+       clk_disable_bulk(&priv->clks);
+       return ret;
+}
+
+static int starfive_pcie_probe(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       int ret;
+
+       priv->atr_table_num = 0;
+       priv->dev = dev;
+
+       ret = starfive_pcie_parse_dt(dev);
+       if (ret)
+               return ret;
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_rp_nep,
+                          STG_SYSCON_K_RP_NEP_MASK,
+                          STG_SYSCON_K_RP_NEP_MASK);
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_CKREF_SRC_MASK,
+                          2 << STG_SYSCON_CKREF_SRC_SHIFT);
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_CLKREQ_MASK,
+                          STG_SYSCON_CLKREQ_MASK);
+
+       ret = starfive_pcie_init_port(dev);
+       if (ret)
+               return ret;
+
+       dev_err(dev, "Starfive PCIe bus probed.\n");
+
+       return 0;
+}
+
+static const struct dm_pci_ops starfive_pcie_ops = {
Quoted 9 lines

drivers/pci/pcie_starfive_jh7110.c line 454

Quoted 6 lines
drivers/pci/pcie_starfive_jh7110.c new file +6 -0
+       .read_config    = starfive_pcie_config_read,
+       .write_config   = starfive_pcie_config_write,
+};
+
+static const struct udevice_id starfive_pcie_ids[] = {
+       { .compatible = "starfive,jh7110-pcie" },

My previous comments were not addressed.

I am sorry. I overlooked your previous comments.

Quoted 3 lines

Is this PCIe controller a StarFive IP? Or some 3rd party licensed IP? Is this compatible string approved by the DT community? The PCIe controller is 3rd party (PLDA) licensed IP.

Both kernel and u-boot can not find PLDA IP compatible string.
So this compatible string is not approved.
I can split PCIe controller ops code to new pcie-plda-common.c like designeware codes.

Quoted 15 lines

drivers/pci/pcie_starfive_jh7110.c line 465

Quoted 12 lines
drivers/pci/pcie_starfive_jh7110.c new file +11 -0
+       { }
+};
+
+U_BOOT_DRIVER(starfive_pcie_drv) = {
+       .name                   = "starfive_7110_pcie",
+       .id                     = UCLASS_PCI,
+       .of_match               = starfive_pcie_ids,
+       .ops                    = &starfive_pcie_ops,
+       .probe                  = starfive_pcie_probe,
+       .priv_auto      = sizeof(struct starfive_pcie),
+};

Regards,
Bin

#4 lore raw

Re: [PATCH v4 1/3] starfive: pci: Add StarFive JH7110 pcie driver

Quoted 514 lines · Minda Chen
Quoted 501 lines · Bin Meng
Quoted 500 lines · Minda Chen

From: Mason Huo <[email protected]>

Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.

Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.

---
 drivers/pci/Kconfig                |   9 +
 drivers/pci/Makefile               |   1 +
 drivers/pci/pcie_starfive_jh7110.c | 465 +++++++++++++++++++++++++++++
 3 files changed, 475 insertions(+)
 create mode 100644 drivers/pci/pcie_starfive_jh7110.c
drivers/pci/Kconfig +9 -0
index ef328d2652..f37b6baa25 100644
@@ -374,4 +374,13 @@ config PCIE_UNIPHIER
          Say Y here if you want to enable PCIe controller support on
          UniPhier SoCs.
 
+config PCIE_STARFIVE_JH7110
+       bool "Enable Starfive JH7110 PCIe driver"
+       imply STARFIVE_JH7110
+       imply CLK_JH7110
+       imply RESET_JH7110
+       help
+         Say Y here if you want to enable PCIe controller support on
+         StarFive JH7110 SoC.
+
 endif
drivers/pci/Makefile +1 -0
index 49506e7ba5..bbe3323bb5 100644
@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o
 obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o
 obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o
 obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o
+obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.o
drivers/pci/pcie_starfive_jh7110.c new file +448 -0
new file mode 100644
index 0000000000..130181013e
@@ -0,0 +1,465 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * StarFive PLDA PCIe host controller driver
+ *
+ * Copyright (c) 2023 Starfive, Inc.
+ * Author: Mason Huo <[email protected]>
+ *
+ */
+
+#include <common.h>
+#include <clk.h>
+#include <dm.h>
+#include <generic-phy.h>
+#include <pci.h>
+#include <pci_ids.h>
+#include <power-domain.h>
+#include <regmap.h>
+#include <reset.h>
+#include <syscon.h>
+#include <asm/global_data.h>
+#include <asm/io.h>
+#include <asm-generic/gpio.h>
+#include <dm/device_compat.h>
+#include <dm/pinctrl.h>
+#include <linux/delay.h>
+#include <linux/iopoll.h>
+#include <power/regulator.h>
+
+DECLARE_GLOBAL_DATA_PTR;
+
+#define GEN_SETTINGS                   0x80
+#define PCIE_PCI_IDS                   0x9C
+#define PCIE_WINROM                    0xFC
+#define PMSG_SUPPORT_RX                        0x3F0
+#define PCI_MISC                       0xB4
+
+#define PLDA_EP_ENABLE                 0
+#define PLDA_RP_ENABLE                 1
+
+#define IDS_CLASS_CODE_SHIFT           8
+
+#define PREF_MEM_WIN_64_SUPPORT                BIT(3)
+#define PMSG_LTR_SUPPORT               BIT(2)
+#define PLDA_FUNCTION_DIS              BIT(15)
+#define PLDA_FUNC_NUM                  4
+#define PLDA_PHY_FUNC_SHIFT            9
+
+#define XR3PCI_ATR_AXI4_SLV0           0x800
+#define XR3PCI_ATR_SRC_ADDR_LOW                0x0
+#define XR3PCI_ATR_SRC_ADDR_HIGH       0x4
+#define XR3PCI_ATR_TRSL_ADDR_LOW       0x8
+#define XR3PCI_ATR_TRSL_ADDR_HIGH      0xc
+#define XR3PCI_ATR_TRSL_PARAM          0x10
+#define XR3PCI_ATR_TABLE_OFFSET                0x20
+#define XR3PCI_ATR_MAX_TABLE_NUM       8
+
+#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT  1
+#define XR3PCI_ATR_SRC_ADDR_MASK       GENMASK(31, 12)
+#define XR3PCI_ATR_TRSL_ADDR_MASK      GENMASK(31, 12)
+#define XR3_PCI_ECAM_SIZE              28
+#define XR3PCI_ATR_TRSL_DIR            BIT(22)
+/* IDs used in the XR3PCI_ATR_TRSL_PARAM */
+#define XR3PCI_ATR_TRSLID_PCIE_MEMORY  0x0
+#define XR3PCI_ATR_TRSLID_PCIE_CONFIG  0x1
+
+/* system control */
+#define STG_SYSCON_K_RP_NEP_MASK               BIT(8)
+#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK       GENMASK(22, 8)
+#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT      8
+#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK       GENMASK(14, 0)
+#define STG_SYSCON_CLKREQ_MASK                 BIT(22)
+#define STG_SYSCON_CKREF_SRC_SHIFT             18
+#define STG_SYSCON_CKREF_SRC_MASK              GENMASK(19, 18)
+
+struct starfive_pcie {
+       struct udevice *dev;
+
+       void __iomem *reg_base;
+       void __iomem *cfg_base;
+
+       struct regmap *regmap;
+       u32 stg_arfun;
+       u32 stg_awfun;
+       u32 stg_rp_nep;
+
+       struct clk_bulk clks;
+       struct reset_ctl_bulk   rsts;
+       struct gpio_desc        reset_gpio;
+
+       int atr_table_num;
+       int sec_busno;
+};
+
+static bool starfive_pcie_addr_valid(pci_dev_t bdf, struct starfive_pcie *priv)
+{
+       /*
+        * Single device limitation.
+        * For JH7110 SoC limitation, one bus can only connnect one device.
+        * And PCIe controller contain HW issue that secondary bus of
+        * host bridge emumerate duplicate devices.
+        * Only can access device 0 in secondary bus.
+        */
+       if (PCI_BUS(bdf) == priv->sec_busno && PCI_DEV(bdf) > 0)
+               return false;
+
+       return true;
+}
+
+static int starfive_pcie_conf_address(const struct udevice *udev, pci_dev_t bdf,
+                                     uint offset, void **paddr)
+{
+       struct starfive_pcie *priv = dev_get_priv(udev);
+       int where = PCIE_ECAM_OFFSET(PCI_BUS(bdf), PCI_DEV(bdf),
+                                    PCI_FUNC(bdf), offset);
+
+       if (!starfive_pcie_addr_valid(bdf, priv))
+               return -ENODEV;
+
+       *paddr = (void *)(priv->cfg_base + where);
+       return 0;
+}
+
+static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf,
+                                    uint offset, ulong *valuep,
+                                    enum pci_size_t size)
+{
+       return pci_generic_mmap_read_config(udev, starfive_pcie_conf_address,
+                                           bdf, offset, valuep, size);
+}
+
+int starfive_pcie_config_write(struct udevice *udev, pci_dev_t bdf,
+                              uint offset, ulong value,
+                              enum pci_size_t size)
+{
+       struct starfive_pcie *priv = dev_get_priv(udev);
+       int ret;
+
+       ret = pci_generic_mmap_write_config(udev, starfive_pcie_conf_address,
+                                           bdf, offset, value, size);
+
+       if (!ret && offset == PCI_SECONDARY_BUS) {
+               priv->sec_busno = value & 0xff;
+               debug("Secondary bus number was changed to %d\n",
+                     priv->sec_busno);
+       }
+       return ret;
+}
+
+static int starfive_pcie_set_atr_entry(struct starfive_pcie *priv, phys_addr_t src_addr,
+                                      phys_addr_t trsl_addr, size_t window_size,
+                                      int trsl_param)
+{
+       void __iomem *base =
+               priv->reg_base + XR3PCI_ATR_AXI4_SLV0;
+
+       /* Support AXI4 Slave 0 Address Translation Tables 0-7. */
+       if (priv->atr_table_num >= XR3PCI_ATR_MAX_TABLE_NUM) {
+               dev_err(priv->dev, "ATR table number %d exceeds max num\n",
+                       priv->atr_table_num);
+               return -EINVAL;
+       }
+       base +=  XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num;
+       priv->atr_table_num++;
+
+       /*
+        * X3PCI_ATR_SRC_ADDR_LOW:
+        *   - bit 0: enable entry,
+        *   - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1)
+        *   - bits 7-11: reserved
+        *   - bits 12-31: start of source address
+        */
+       writel((lower_32_bits(src_addr) & XR3PCI_ATR_SRC_ADDR_MASK) |
+                       (fls(window_size) - 1) << XR3PCI_ATR_SRC_WIN_SIZE_SHIFT | 1,
+                       base + XR3PCI_ATR_SRC_ADDR_LOW);
+       writel(upper_32_bits(src_addr), base + XR3PCI_ATR_SRC_ADDR_HIGH);
+       writel((lower_32_bits(trsl_addr) & XR3PCI_ATR_TRSL_ADDR_MASK),
+              base + XR3PCI_ATR_TRSL_ADDR_LOW);
+       writel(upper_32_bits(trsl_addr), base + XR3PCI_ATR_TRSL_ADDR_HIGH);
+       writel(trsl_param, base + XR3PCI_ATR_TRSL_PARAM);
+
+       dev_dbg(priv->dev, "ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n",
+               src_addr, (trsl_param & XR3PCI_ATR_TRSL_DIR) ? "<-" : "->",
+               trsl_addr, (u64)window_size, trsl_param);
+       return 0;
+}
+
+static int starfive_pcie_atr_init(struct starfive_pcie *priv)
+{
+       struct udevice *ctlr = pci_get_controller(priv->dev);
+       struct pci_controller *hose = dev_get_uclass_priv(ctlr);
+       int i, ret;
+
+       /*
+        * As the two host bridges in JH7110 soc have the same default
+        * address translation table, this cause the second root port can't
+        * access it's host bridge config space correctly.
+        * To workaround, config the ATR of host bridge config space by SW.
+        */
+
+       ret = starfive_pcie_set_atr_entry(priv,
+                                         (phys_addr_t)priv->cfg_base,
+                                         0,
+                                         1 << XR3_PCI_ECAM_SIZE,
+                                         XR3PCI_ATR_TRSLID_PCIE_CONFIG);
+       if (ret)
+               return ret;
+
+       for (i = 0; i < hose->region_count; i++) {
+               if (hose->regions[i].flags == PCI_REGION_SYS_MEMORY)
+                       continue;
+
+               /* Only support identity mappings. */
+               if (hose->regions[i].bus_start !=
+                   hose->regions[i].phys_start)
+                       return -EINVAL;
+
+               ret = starfive_pcie_set_atr_entry(priv,
+                                                 hose->regions[i].phys_start,
+                                                 hose->regions[i].bus_start,
+                                                 hose->regions[i].size,
+                                                 XR3PCI_ATR_TRSLID_PCIE_MEMORY);
+               if (ret)
+                       return ret;
+       }
+
+       return 0;
+}
+
+static int starfive_pcie_get_syscon(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       struct udevice *syscon;
+       struct ofnode_phandle_args syscfg_phandle;
+       u32 cells[4];
+       int ret;
+
+       /* get corresponding syscon phandle */
+       ret = dev_read_phandle_with_args(dev, "starfive,stg-syscon", NULL, 0, 0,
+                                        &syscfg_phandle);
+
+       if (ret < 0) {
+               dev_err(dev, "Can't get syscfg phandle: %d\n", ret);
+               return ret;
+       }
+
+       ret = uclass_get_device_by_ofnode(UCLASS_SYSCON, syscfg_phandle.node,
+                                         &syscon);
+       if (ret) {
+               dev_err(dev, "Unable to find syscon device (%d)\n", ret);
+               return ret;
+       }
+
+       priv->regmap = syscon_get_regmap(syscon);
+       if (!priv->regmap) {
+               dev_err(dev, "Unable to find regmap\n");
+               return -ENODEV;
+       }
+
+       /* get syscon register offset */
+       ret = dev_read_u32_array(dev, "starfive,stg-syscon",
+                                cells, ARRAY_SIZE(cells));
+       if (ret) {
+               dev_err(dev, "Get syscon register err %d\n", ret);
+               return -EINVAL;
+       }
+
+       dev_dbg(dev, "Get syscon values: %x, %x, %x\n",
+               cells[1], cells[2], cells[3]);
+       priv->stg_arfun = cells[1];
+       priv->stg_awfun = cells[2];
+       priv->stg_rp_nep = cells[3];
+
+       return 0;
+}
+
+static int starfive_pcie_parse_dt(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       int ret;
+
+       priv->reg_base = (void *)dev_read_addr_name(dev, "reg");
+       if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE) {
+               dev_err(dev, "Missing required reg address range\n");
+               return -EINVAL;
+       }
+
+       priv->cfg_base = (void *)dev_read_addr_name(dev, "config");
+       if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE) {
+               dev_err(dev, "Missing required config address range");
+               return -EINVAL;
+       }
+
+       ret = starfive_pcie_get_syscon(dev);
+       if (ret) {
+               dev_err(dev, "Can't get syscon: %d\n", ret);
+               return ret;
+       }
+
+       ret = reset_get_bulk(dev, &priv->rsts);
+       if (ret) {
+               dev_err(dev, "Can't get reset: %d\n", ret);
+               return ret;
+       }
+
+       ret = clk_get_bulk(dev, &priv->clks);
+       if (ret) {
+               dev_err(dev, "Can't get clock: %d\n", ret);
+               return ret;
+       }
+
+       ret = gpio_request_by_name(dev, "reset-gpios", 0, &priv->reset_gpio,
+                                  GPIOD_IS_OUT);
+       if (ret) {
+               dev_err(dev, "Can't get reset-gpio: %d\n", ret);
+               return ret;
+       }
+
+       if (!dm_gpio_is_valid(&priv->reset_gpio)) {
+               dev_err(dev, "reset-gpio is not valid\n");
+               return -EINVAL;
+       }
+       return 0;
+}
+
+static int starfive_pcie_init_port(struct udevice *dev)
+{
+       int ret, i;
+       unsigned int value;
+       struct starfive_pcie *priv = dev_get_priv(dev);
+
+       ret = clk_enable_bulk(&priv->clks);
+       if (ret) {
+               dev_err(dev, "Failed to enable clks (ret=%d)\n", ret);
+               return ret;
+       }
+
+       ret = reset_deassert_bulk(&priv->rsts);
+       if (ret) {
+               dev_err(dev, "Failed to deassert resets (ret=%d)\n", ret);
+               goto err_deassert_clk;
+       }
+
+       dm_gpio_set_value(&priv->reset_gpio, 1);
+       /* Disable physical functions except #0 */
+       for (i = 1; i < PLDA_FUNC_NUM; i++) {
+               regmap_update_bits(priv->regmap,
+                                  priv->stg_arfun,
+                                  STG_SYSCON_AXI4_SLVL_ARFUNC_MASK,
+                                  (i << PLDA_PHY_FUNC_SHIFT) <<
+                                  STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT);
+               regmap_update_bits(priv->regmap,
+                                  priv->stg_awfun,
+                                  STG_SYSCON_AXI4_SLVL_AWFUNC_MASK,
+                                  i << PLDA_PHY_FUNC_SHIFT);
+
+               value = readl(priv->reg_base + PCI_MISC);
+               value |= PLDA_FUNCTION_DIS;
+               writel(value, priv->reg_base + PCI_MISC);
+       }
+
+       /* Disable physical functions */
+       regmap_update_bits(priv->regmap,
+                          priv->stg_arfun,
+                          STG_SYSCON_AXI4_SLVL_ARFUNC_MASK,
+                          0);
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_AXI4_SLVL_AWFUNC_MASK,
+                          0);
+
+       /* Enable root port */
+       value = readl(priv->reg_base + GEN_SETTINGS);
+       value |= PLDA_RP_ENABLE;
+       writel(value, priv->reg_base + GEN_SETTINGS);
+
+       /* PCIe PCI Standard Configuration Identification Settings. */
+       value = readl(priv->reg_base + PCIE_PCI_IDS);
+       value &= 0xff;
+       value |= (PCI_CLASS_BRIDGE_PCI_NORMAL << IDS_CLASS_CODE_SHIFT);
+       writel(value, priv->reg_base + PCIE_PCI_IDS);
+
+       /*
+        * The LTR message forwarding of PCIe Message Reception was set by core
+        * as default, but the forward id & addr are also need to be reset.
+        * If we do not disable LTR message forwarding here, or set a legal
+        * forwarding address, the kernel will get stuck after this driver probe.
+        * To workaround, disable the LTR message forwarding support on
+        * PCIe Message Reception.
+        */
+       value = readl(priv->reg_base + PMSG_SUPPORT_RX);
+       value &= ~PMSG_LTR_SUPPORT;
+       writel(value, priv->reg_base + PMSG_SUPPORT_RX);
+
+       /* Prefetchable memory window 64-bit addressing support */
+       value = readl(priv->reg_base + PCIE_WINROM);
+       value |= PREF_MEM_WIN_64_SUPPORT;
+       writel(value, priv->reg_base + PCIE_WINROM);
+
+       starfive_pcie_atr_init(priv);
+
+       dm_gpio_set_value(&priv->reset_gpio, 0);
+       /* Ensure that PERST in default at least 300 ms */
+       mdelay(300);
+
+       return 0;
+
+err_deassert_clk:
+       clk_disable_bulk(&priv->clks);
+       return ret;
+}
+
+static int starfive_pcie_probe(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       int ret;
+
+       priv->atr_table_num = 0;
+       priv->dev = dev;
+
+       ret = starfive_pcie_parse_dt(dev);
+       if (ret)
+               return ret;
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_rp_nep,
+                          STG_SYSCON_K_RP_NEP_MASK,
+                          STG_SYSCON_K_RP_NEP_MASK);
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_CKREF_SRC_MASK,
+                          2 << STG_SYSCON_CKREF_SRC_SHIFT);
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_CLKREQ_MASK,
+                          STG_SYSCON_CLKREQ_MASK);
+
+       ret = starfive_pcie_init_port(dev);
+       if (ret)
+               return ret;
+
+       dev_err(dev, "Starfive PCIe bus probed.\n");
+
+       return 0;
+}
+
+static const struct dm_pci_ops starfive_pcie_ops = {
Quoted 9 lines

drivers/pci/pcie_starfive_jh7110.c line 454

Quoted 6 lines
drivers/pci/pcie_starfive_jh7110.c new file +6 -0
+       .read_config    = starfive_pcie_config_read,
+       .write_config   = starfive_pcie_config_write,
+};
+
+static const struct udevice_id starfive_pcie_ids[] = {
+       { .compatible = "starfive,jh7110-pcie" },

My previous comments were not addressed.

Quoted 7 lines

I am sorry. I overlooked your previous comments.

Quoted 3 lines

Is this PCIe controller a StarFive IP? Or some 3rd party licensed IP? Is this compatible string approved by the DT community? The PCIe controller is 3rd party (PLDA) licensed IP.

Both kernel and u-boot can not find PLDA IP compatible string.
So this compatible string is not approved.
I can split PCIe controller ops code to new pcie-plda-common.c like designeware codes.

Yes, please do. This will allow future SoCs that use the same PLDA PCIe IP to reuse most of the codes.

In the meantime, I suggest you send patch to the Linux kernel for adding the compatible string of the PLDA PCIe controller.

Is the datasheet public avaiable?

Regards,
Bin

#6 lore raw

Re: [PATCH v4 1/3] starfive: pci: Add StarFive JH7110 pcie driver

In reply to Minda Chen

Hello!

Quoted 5 lines · On Tuesday 11 April 2023 09:02:07 Minda Chen wrote:
drivers/pci/pcie_starfive_jh7110.c new file +5 -0
+int starfive_pcie_config_write(struct udevice *udev, pci_dev_t bdf,
+			       uint offset, ulong value,
+			       enum pci_size_t size)
+{
+	struct starfive_pcie *priv = dev_get_priv(udev);

drivers/pci/pcie_starfive_jh7110.c line 145

Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10 -0
+	int ret;
+
+	ret = pci_generic_mmap_write_config(udev, starfive_pcie_conf_address,
+					    bdf, offset, value, size);
+
+	if (!ret && offset == PCI_SECONDARY_BUS) {
+		priv->sec_busno = value & 0xff;
+		debug("Secondary bus number was changed to %d\n",
+		      priv->sec_busno);
+	}

This block of code contains two issues:

1) If secondary bus is changed by the 16-bit or 32-bit write operation then this condition does not catch it.

2) priv->sec_busno is used just for checking if driver is going to access device on secondary bus of the Root Port. But this code updates priv->sec_busno also for write to _any_ device on any bus, not just when updating Root Port device. So it breaks support for non-trivial PCIe hierarchy which contains e.g. PCIe switch (e.g. when changing configuration of the virtual PCI-to-PCI bridge device of PCIe switch, which is behind the secondary bus of the Root Port).

So you need something like this:

    if (!ret &&
        PCI_BUS(bdf) == dev_seq(udev) && PCI_DEV(bdf) == 0 && PCI_FUNC(bdf) == 0 &&
        (offset == PCI_SECONDARY_BUS || (offset == PCI_PRIMARY_BUS && size != PCI_SIZE_8)) {
        priv->sec_busno = ((offset == PCI_PRIMARY_BUS) ? (value >> 8) : value) & 0xff;
        debug("Secondary bus number was changed to %d\n", pcie->sec_busno);
    }

You have to update priv->sec_busno only when write request is for the Root Port. And you need to catch also 16-bit or 32-bit write operation to the PCI_PRIMARY_BUS register. It is because PCI_SECONDARY_BUS reg is (PCI_PRIMARY_BUS+2) and (PCI_SECONDARY_BUS & ~3) == PCI_PRIMARY_BUS

Quoted 2 lines
(quoted diff) +2 -0
+	return ret;
+}
author
#5 lore raw

Re: [PATCH v4 1/3] starfive: pci: Add StarFive JH7110 pcie driver

In reply to Bin Meng
Quoted 523 lines · Bin Meng
Quoted 514 lines · Minda Chen
Quoted 501 lines · Bin Meng
Quoted 500 lines · Minda Chen

From: Mason Huo <[email protected]>

Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.

Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.

---
 drivers/pci/Kconfig                |   9 +
 drivers/pci/Makefile               |   1 +
 drivers/pci/pcie_starfive_jh7110.c | 465 +++++++++++++++++++++++++++++
 3 files changed, 475 insertions(+)
 create mode 100644 drivers/pci/pcie_starfive_jh7110.c
drivers/pci/Kconfig +9 -0
index ef328d2652..f37b6baa25 100644
@@ -374,4 +374,13 @@ config PCIE_UNIPHIER
          Say Y here if you want to enable PCIe controller support on
          UniPhier SoCs.
 
+config PCIE_STARFIVE_JH7110
+       bool "Enable Starfive JH7110 PCIe driver"
+       imply STARFIVE_JH7110
+       imply CLK_JH7110
+       imply RESET_JH7110
+       help
+         Say Y here if you want to enable PCIe controller support on
+         StarFive JH7110 SoC.
+
 endif
drivers/pci/Makefile +1 -0
index 49506e7ba5..bbe3323bb5 100644
@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o
 obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o
 obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o
 obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o
+obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.o
drivers/pci/pcie_starfive_jh7110.c new file +448 -0
new file mode 100644
index 0000000000..130181013e
@@ -0,0 +1,465 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * StarFive PLDA PCIe host controller driver
+ *
+ * Copyright (c) 2023 Starfive, Inc.
+ * Author: Mason Huo <[email protected]>
+ *
+ */
+
+#include <common.h>
+#include <clk.h>
+#include <dm.h>
+#include <generic-phy.h>
+#include <pci.h>
+#include <pci_ids.h>
+#include <power-domain.h>
+#include <regmap.h>
+#include <reset.h>
+#include <syscon.h>
+#include <asm/global_data.h>
+#include <asm/io.h>
+#include <asm-generic/gpio.h>
+#include <dm/device_compat.h>
+#include <dm/pinctrl.h>
+#include <linux/delay.h>
+#include <linux/iopoll.h>
+#include <power/regulator.h>
+
+DECLARE_GLOBAL_DATA_PTR;
+
+#define GEN_SETTINGS                   0x80
+#define PCIE_PCI_IDS                   0x9C
+#define PCIE_WINROM                    0xFC
+#define PMSG_SUPPORT_RX                        0x3F0
+#define PCI_MISC                       0xB4
+
+#define PLDA_EP_ENABLE                 0
+#define PLDA_RP_ENABLE                 1
+
+#define IDS_CLASS_CODE_SHIFT           8
+
+#define PREF_MEM_WIN_64_SUPPORT                BIT(3)
+#define PMSG_LTR_SUPPORT               BIT(2)
+#define PLDA_FUNCTION_DIS              BIT(15)
+#define PLDA_FUNC_NUM                  4
+#define PLDA_PHY_FUNC_SHIFT            9
+
+#define XR3PCI_ATR_AXI4_SLV0           0x800
+#define XR3PCI_ATR_SRC_ADDR_LOW                0x0
+#define XR3PCI_ATR_SRC_ADDR_HIGH       0x4
+#define XR3PCI_ATR_TRSL_ADDR_LOW       0x8
+#define XR3PCI_ATR_TRSL_ADDR_HIGH      0xc
+#define XR3PCI_ATR_TRSL_PARAM          0x10
+#define XR3PCI_ATR_TABLE_OFFSET                0x20
+#define XR3PCI_ATR_MAX_TABLE_NUM       8
+
+#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT  1
+#define XR3PCI_ATR_SRC_ADDR_MASK       GENMASK(31, 12)
+#define XR3PCI_ATR_TRSL_ADDR_MASK      GENMASK(31, 12)
+#define XR3_PCI_ECAM_SIZE              28
+#define XR3PCI_ATR_TRSL_DIR            BIT(22)
+/* IDs used in the XR3PCI_ATR_TRSL_PARAM */
+#define XR3PCI_ATR_TRSLID_PCIE_MEMORY  0x0
+#define XR3PCI_ATR_TRSLID_PCIE_CONFIG  0x1
+
+/* system control */
+#define STG_SYSCON_K_RP_NEP_MASK               BIT(8)
+#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK       GENMASK(22, 8)
+#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT      8
+#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK       GENMASK(14, 0)
+#define STG_SYSCON_CLKREQ_MASK                 BIT(22)
+#define STG_SYSCON_CKREF_SRC_SHIFT             18
+#define STG_SYSCON_CKREF_SRC_MASK              GENMASK(19, 18)
+
+struct starfive_pcie {
+       struct udevice *dev;
+
+       void __iomem *reg_base;
+       void __iomem *cfg_base;
+
+       struct regmap *regmap;
+       u32 stg_arfun;
+       u32 stg_awfun;
+       u32 stg_rp_nep;
+
+       struct clk_bulk clks;
+       struct reset_ctl_bulk   rsts;
+       struct gpio_desc        reset_gpio;
+
+       int atr_table_num;
+       int sec_busno;
+};
+
+static bool starfive_pcie_addr_valid(pci_dev_t bdf, struct starfive_pcie *priv)
+{
+       /*
+        * Single device limitation.
+        * For JH7110 SoC limitation, one bus can only connnect one device.
+        * And PCIe controller contain HW issue that secondary bus of
+        * host bridge emumerate duplicate devices.
+        * Only can access device 0 in secondary bus.
+        */
+       if (PCI_BUS(bdf) == priv->sec_busno && PCI_DEV(bdf) > 0)
+               return false;
+
+       return true;
+}
+
+static int starfive_pcie_conf_address(const struct udevice *udev, pci_dev_t bdf,
+                                     uint offset, void **paddr)
+{
+       struct starfive_pcie *priv = dev_get_priv(udev);
+       int where = PCIE_ECAM_OFFSET(PCI_BUS(bdf), PCI_DEV(bdf),
+                                    PCI_FUNC(bdf), offset);
+
+       if (!starfive_pcie_addr_valid(bdf, priv))
+               return -ENODEV;
+
+       *paddr = (void *)(priv->cfg_base + where);
+       return 0;
+}
+
+static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf,
+                                    uint offset, ulong *valuep,
+                                    enum pci_size_t size)
+{
+       return pci_generic_mmap_read_config(udev, starfive_pcie_conf_address,
+                                           bdf, offset, valuep, size);
+}
+
+int starfive_pcie_config_write(struct udevice *udev, pci_dev_t bdf,
+                              uint offset, ulong value,
+                              enum pci_size_t size)
+{
+       struct starfive_pcie *priv = dev_get_priv(udev);
+       int ret;
+
+       ret = pci_generic_mmap_write_config(udev, starfive_pcie_conf_address,
+                                           bdf, offset, value, size);
+
+       if (!ret && offset == PCI_SECONDARY_BUS) {
+               priv->sec_busno = value & 0xff;
+               debug("Secondary bus number was changed to %d\n",
+                     priv->sec_busno);
+       }
+       return ret;
+}
+
+static int starfive_pcie_set_atr_entry(struct starfive_pcie *priv, phys_addr_t src_addr,
+                                      phys_addr_t trsl_addr, size_t window_size,
+                                      int trsl_param)
+{
+       void __iomem *base =
+               priv->reg_base + XR3PCI_ATR_AXI4_SLV0;
+
+       /* Support AXI4 Slave 0 Address Translation Tables 0-7. */
+       if (priv->atr_table_num >= XR3PCI_ATR_MAX_TABLE_NUM) {
+               dev_err(priv->dev, "ATR table number %d exceeds max num\n",
+                       priv->atr_table_num);
+               return -EINVAL;
+       }
+       base +=  XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num;
+       priv->atr_table_num++;
+
+       /*
+        * X3PCI_ATR_SRC_ADDR_LOW:
+        *   - bit 0: enable entry,
+        *   - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1)
+        *   - bits 7-11: reserved
+        *   - bits 12-31: start of source address
+        */
+       writel((lower_32_bits(src_addr) & XR3PCI_ATR_SRC_ADDR_MASK) |
+                       (fls(window_size) - 1) << XR3PCI_ATR_SRC_WIN_SIZE_SHIFT | 1,
+                       base + XR3PCI_ATR_SRC_ADDR_LOW);
+       writel(upper_32_bits(src_addr), base + XR3PCI_ATR_SRC_ADDR_HIGH);
+       writel((lower_32_bits(trsl_addr) & XR3PCI_ATR_TRSL_ADDR_MASK),
+              base + XR3PCI_ATR_TRSL_ADDR_LOW);
+       writel(upper_32_bits(trsl_addr), base + XR3PCI_ATR_TRSL_ADDR_HIGH);
+       writel(trsl_param, base + XR3PCI_ATR_TRSL_PARAM);
+
+       dev_dbg(priv->dev, "ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n",
+               src_addr, (trsl_param & XR3PCI_ATR_TRSL_DIR) ? "<-" : "->",
+               trsl_addr, (u64)window_size, trsl_param);
+       return 0;
+}
+
+static int starfive_pcie_atr_init(struct starfive_pcie *priv)
+{
+       struct udevice *ctlr = pci_get_controller(priv->dev);
+       struct pci_controller *hose = dev_get_uclass_priv(ctlr);
+       int i, ret;
+
+       /*
+        * As the two host bridges in JH7110 soc have the same default
+        * address translation table, this cause the second root port can't
+        * access it's host bridge config space correctly.
+        * To workaround, config the ATR of host bridge config space by SW.
+        */
+
+       ret = starfive_pcie_set_atr_entry(priv,
+                                         (phys_addr_t)priv->cfg_base,
+                                         0,
+                                         1 << XR3_PCI_ECAM_SIZE,
+                                         XR3PCI_ATR_TRSLID_PCIE_CONFIG);
+       if (ret)
+               return ret;
+
+       for (i = 0; i < hose->region_count; i++) {
+               if (hose->regions[i].flags == PCI_REGION_SYS_MEMORY)
+                       continue;
+
+               /* Only support identity mappings. */
+               if (hose->regions[i].bus_start !=
+                   hose->regions[i].phys_start)
+                       return -EINVAL;
+
+               ret = starfive_pcie_set_atr_entry(priv,
+                                                 hose->regions[i].phys_start,
+                                                 hose->regions[i].bus_start,
+                                                 hose->regions[i].size,
+                                                 XR3PCI_ATR_TRSLID_PCIE_MEMORY);
+               if (ret)
+                       return ret;
+       }
+
+       return 0;
+}
+
+static int starfive_pcie_get_syscon(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       struct udevice *syscon;
+       struct ofnode_phandle_args syscfg_phandle;
+       u32 cells[4];
+       int ret;
+
+       /* get corresponding syscon phandle */
+       ret = dev_read_phandle_with_args(dev, "starfive,stg-syscon", NULL, 0, 0,
+                                        &syscfg_phandle);
+
+       if (ret < 0) {
+               dev_err(dev, "Can't get syscfg phandle: %d\n", ret);
+               return ret;
+       }
+
+       ret = uclass_get_device_by_ofnode(UCLASS_SYSCON, syscfg_phandle.node,
+                                         &syscon);
+       if (ret) {
+               dev_err(dev, "Unable to find syscon device (%d)\n", ret);
+               return ret;
+       }
+
+       priv->regmap = syscon_get_regmap(syscon);
+       if (!priv->regmap) {
+               dev_err(dev, "Unable to find regmap\n");
+               return -ENODEV;
+       }
+
+       /* get syscon register offset */
+       ret = dev_read_u32_array(dev, "starfive,stg-syscon",
+                                cells, ARRAY_SIZE(cells));
+       if (ret) {
+               dev_err(dev, "Get syscon register err %d\n", ret);
+               return -EINVAL;
+       }
+
+       dev_dbg(dev, "Get syscon values: %x, %x, %x\n",
+               cells[1], cells[2], cells[3]);
+       priv->stg_arfun = cells[1];
+       priv->stg_awfun = cells[2];
+       priv->stg_rp_nep = cells[3];
+
+       return 0;
+}
+
+static int starfive_pcie_parse_dt(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       int ret;
+
+       priv->reg_base = (void *)dev_read_addr_name(dev, "reg");
+       if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE) {
+               dev_err(dev, "Missing required reg address range\n");
+               return -EINVAL;
+       }
+
+       priv->cfg_base = (void *)dev_read_addr_name(dev, "config");
+       if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE) {
+               dev_err(dev, "Missing required config address range");
+               return -EINVAL;
+       }
+
+       ret = starfive_pcie_get_syscon(dev);
+       if (ret) {
+               dev_err(dev, "Can't get syscon: %d\n", ret);
+               return ret;
+       }
+
+       ret = reset_get_bulk(dev, &priv->rsts);
+       if (ret) {
+               dev_err(dev, "Can't get reset: %d\n", ret);
+               return ret;
+       }
+
+       ret = clk_get_bulk(dev, &priv->clks);
+       if (ret) {
+               dev_err(dev, "Can't get clock: %d\n", ret);
+               return ret;
+       }
+
+       ret = gpio_request_by_name(dev, "reset-gpios", 0, &priv->reset_gpio,
+                                  GPIOD_IS_OUT);
+       if (ret) {
+               dev_err(dev, "Can't get reset-gpio: %d\n", ret);
+               return ret;
+       }
+
+       if (!dm_gpio_is_valid(&priv->reset_gpio)) {
+               dev_err(dev, "reset-gpio is not valid\n");
+               return -EINVAL;
+       }
+       return 0;
+}
+
+static int starfive_pcie_init_port(struct udevice *dev)
+{
+       int ret, i;
+       unsigned int value;
+       struct starfive_pcie *priv = dev_get_priv(dev);
+
+       ret = clk_enable_bulk(&priv->clks);
+       if (ret) {
+               dev_err(dev, "Failed to enable clks (ret=%d)\n", ret);
+               return ret;
+       }
+
+       ret = reset_deassert_bulk(&priv->rsts);
+       if (ret) {
+               dev_err(dev, "Failed to deassert resets (ret=%d)\n", ret);
+               goto err_deassert_clk;
+       }
+
+       dm_gpio_set_value(&priv->reset_gpio, 1);
+       /* Disable physical functions except #0 */
+       for (i = 1; i < PLDA_FUNC_NUM; i++) {
+               regmap_update_bits(priv->regmap,
+                                  priv->stg_arfun,
+                                  STG_SYSCON_AXI4_SLVL_ARFUNC_MASK,
+                                  (i << PLDA_PHY_FUNC_SHIFT) <<
+                                  STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT);
+               regmap_update_bits(priv->regmap,
+                                  priv->stg_awfun,
+                                  STG_SYSCON_AXI4_SLVL_AWFUNC_MASK,
+                                  i << PLDA_PHY_FUNC_SHIFT);
+
+               value = readl(priv->reg_base + PCI_MISC);
+               value |= PLDA_FUNCTION_DIS;
+               writel(value, priv->reg_base + PCI_MISC);
+       }
+
+       /* Disable physical functions */
+       regmap_update_bits(priv->regmap,
+                          priv->stg_arfun,
+                          STG_SYSCON_AXI4_SLVL_ARFUNC_MASK,
+                          0);
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_AXI4_SLVL_AWFUNC_MASK,
+                          0);
+
+       /* Enable root port */
+       value = readl(priv->reg_base + GEN_SETTINGS);
+       value |= PLDA_RP_ENABLE;
+       writel(value, priv->reg_base + GEN_SETTINGS);
+
+       /* PCIe PCI Standard Configuration Identification Settings. */
+       value = readl(priv->reg_base + PCIE_PCI_IDS);
+       value &= 0xff;
+       value |= (PCI_CLASS_BRIDGE_PCI_NORMAL << IDS_CLASS_CODE_SHIFT);
+       writel(value, priv->reg_base + PCIE_PCI_IDS);
+
+       /*
+        * The LTR message forwarding of PCIe Message Reception was set by core
+        * as default, but the forward id & addr are also need to be reset.
+        * If we do not disable LTR message forwarding here, or set a legal
+        * forwarding address, the kernel will get stuck after this driver probe.
+        * To workaround, disable the LTR message forwarding support on
+        * PCIe Message Reception.
+        */
+       value = readl(priv->reg_base + PMSG_SUPPORT_RX);
+       value &= ~PMSG_LTR_SUPPORT;
+       writel(value, priv->reg_base + PMSG_SUPPORT_RX);
+
+       /* Prefetchable memory window 64-bit addressing support */
+       value = readl(priv->reg_base + PCIE_WINROM);
+       value |= PREF_MEM_WIN_64_SUPPORT;
+       writel(value, priv->reg_base + PCIE_WINROM);
+
+       starfive_pcie_atr_init(priv);
+
+       dm_gpio_set_value(&priv->reset_gpio, 0);
+       /* Ensure that PERST in default at least 300 ms */
+       mdelay(300);
+
+       return 0;
+
+err_deassert_clk:
+       clk_disable_bulk(&priv->clks);
+       return ret;
+}
+
+static int starfive_pcie_probe(struct udevice *dev)
+{
+       struct starfive_pcie *priv = dev_get_priv(dev);
+       int ret;
+
+       priv->atr_table_num = 0;
+       priv->dev = dev;
+
+       ret = starfive_pcie_parse_dt(dev);
+       if (ret)
+               return ret;
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_rp_nep,
+                          STG_SYSCON_K_RP_NEP_MASK,
+                          STG_SYSCON_K_RP_NEP_MASK);
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_CKREF_SRC_MASK,
+                          2 << STG_SYSCON_CKREF_SRC_SHIFT);
+
+       regmap_update_bits(priv->regmap,
+                          priv->stg_awfun,
+                          STG_SYSCON_CLKREQ_MASK,
+                          STG_SYSCON_CLKREQ_MASK);
+
+       ret = starfive_pcie_init_port(dev);
+       if (ret)
+               return ret;
+
+       dev_err(dev, "Starfive PCIe bus probed.\n");
+
+       return 0;
+}
+
+static const struct dm_pci_ops starfive_pcie_ops = {
Quoted 9 lines

drivers/pci/pcie_starfive_jh7110.c line 454

Quoted 6 lines
drivers/pci/pcie_starfive_jh7110.c new file +6 -0
+       .read_config    = starfive_pcie_config_read,
+       .write_config   = starfive_pcie_config_write,
+};
+
+static const struct udevice_id starfive_pcie_ids[] = {
+       { .compatible = "starfive,jh7110-pcie" },

My previous comments were not addressed.

Quoted 7 lines

I am sorry. I overlooked your previous comments.

Quoted 3 lines

Is this PCIe controller a StarFive IP? Or some 3rd party licensed IP? Is this compatible string approved by the DT community? The PCIe controller is 3rd party (PLDA) licensed IP.

Both kernel and u-boot can not find PLDA IP compatible string.
So this compatible string is not approved.
I can split PCIe controller ops code to new pcie-plda-common.c like designeware codes.

Quoted 6 lines

Yes, please do. This will allow future SoCs that use the same PLDA PCIe IP to reuse most of the codes.

In the meantime, I suggest you send patch to the Linux kernel for adding the compatible string of the PLDA PCIe controller.

I am preparing JH7110 PCIe Linux kernel driver upstream. At the same time, both PLDA PCIe controller compatible string and "starfive,jh7110-pcie" will be added to Linux dt-binding documents.

Quoted 2 lines

Is the datasheet public avaiable?

I am sorry. the datasheet is not public available.

Quoted 2 lines

Regards,
Bin

author
#7 lore raw

Re: [PATCH v4 1/3] starfive: pci: Add StarFive JH7110 pcie driver

In reply to Pali Rohár
Quoted 41 lines · Pali Rohár

Hello!

Quoted 5 lines · On Tuesday 11 April 2023 09:02:07 Minda Chen wrote:
drivers/pci/pcie_starfive_jh7110.c new file +5 -0
+int starfive_pcie_config_write(struct udevice *udev, pci_dev_t bdf,
+			       uint offset, ulong value,
+			       enum pci_size_t size)
+{
+	struct starfive_pcie *priv = dev_get_priv(udev);

drivers/pci/pcie_starfive_jh7110.c line 145

Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10 -0
+	int ret;
+
+	ret = pci_generic_mmap_write_config(udev, starfive_pcie_conf_address,
+					    bdf, offset, value, size);
+
+	if (!ret && offset == PCI_SECONDARY_BUS) {
+		priv->sec_busno = value & 0xff;
+		debug("Secondary bus number was changed to %d\n",
+		      priv->sec_busno);
+	}

This block of code contains two issues:

1) If secondary bus is changed by the 16-bit or 32-bit write operation then this condition does not catch it.

2) priv->sec_busno is used just for checking if driver is going to access device on secondary bus of the Root Port. But this code updates priv->sec_busno also for write to _any_ device on any bus, not just when updating Root Port device. So it breaks support for non-trivial PCIe hierarchy which contains e.g. PCIe switch (e.g. when changing configuration of the virtual PCI-to-PCI bridge device of PCIe switch, which is behind the secondary bus of the Root Port).

So you need something like this:

    if (!ret &&
        PCI_BUS(bdf) == dev_seq(udev) && PCI_DEV(bdf) == 0 && PCI_FUNC(bdf) == 0 &&
        (offset == PCI_SECONDARY_BUS || (offset == PCI_PRIMARY_BUS && size != PCI_SIZE_8)) {
        priv->sec_busno = ((offset == PCI_PRIMARY_BUS) ? (value >> 8) : value) & 0xff;
        debug("Secondary bus number was changed to %d\n", pcie->sec_busno);
    }
Quoted 5 lines

You have to update priv->sec_busno only when write request is for the Root Port. And you need to catch also 16-bit or 32-bit write operation to the PCI_PRIMARY_BUS register. It is because PCI_SECONDARY_BUS reg is (PCI_PRIMARY_BUS+2) and (PCI_SECONDARY_BUS & ~3) == PCI_PRIMARY_BUS

I will change like this. Thank you very much.

Quoted 2 lines
Quoted 2 lines
(quoted diff) +2 -0
+	return ret;
+}