Add StarFive JH7110 PCIe drvier support
v2 cover Add StarFive JH7110 PCIe drvier support
This patchset needs to apply after patchset in [1]. These PCIe series patches are based on the JH7110 RISC-V SoC and VisionFive V2 board.
[1] https://patchwork.ozlabs.org/project/uboot/cover/[email protected]/
The PCIe driver depends on gpio, pinctrl, clk and reset driver to do init. The PCIe dts configuation includes all these setting.
The PCIe drivers codes has been tested on the VisionFive V2 boards. The test devices includes M.2 NVMe SSD and Realtek 8169 Ethernet adapter.
previous patch version
v1: https://patchwork.ozlabs.org/project/uboot/cover/[email protected]/
changesv1->v2
1. remove clock commit. The pcie clocks change has been includeded in [1].
2. Using GENMASK marco1 in patch1.
3. remove the syscon dts node in patch3. The syscon dts dts node has been included in [1].
---
The JH7110 minimal system patchset upstream is in progress. PCIe driver patches is based on this patchset.
The JH7110 minimal system patchset details are listed in [1].
For more JH7110 and visionFive v2 information and support, you can visit RVspace wiki[2].
Mason Huo (3):
starfive: pci: Add StarFive JH7110 pcie driver
configs: starfive-jh7110: Add support for PCIe host driver
riscv: dts: starfive: Enable PCIe host controller
.../dts/jh7110-starfive-visionfive-2.dtsi | 99 ++++ arch/riscv/dts/jh7110.dtsi | 75 +++ configs/starfive_visionfive2_defconfig | 9 + drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 ++++++++++++++++++ 6 files changed, 673 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.c
base-commit: b0eda49bc9b00503366f2ec431be0178caf9e9b5
prerequisite-patch-id: a84452ba131408ff842b65ae19a3a05f64b3ff60
prerequisite-patch-id: bb320a2102f46d2b2d08b4467cbcdb92050767b8
prerequisite-patch-id: ef5e9d61f7392c7d2e5321aa8a10996ca8eae5fc
prerequisite-patch-id: 26dfdb401d680ecd9449acf09131cf4790e987b7
prerequisite-patch-id: a0cfa5d8cdb49bbe9d4739afa90991f882950881
prerequisite-patch-id: a0509f01172e24b3d163f4701496b0349435bdc0
prerequisite-patch-id: d834ece14ffb525b8c3e661e78736692f33fca9b
prerequisite-patch-id: aa95c1c047d0b7dae469586873a42932d613411d
prerequisite-patch-id: ebc26ed3f97f7babc0fc805152acb578f5eff922
prerequisite-patch-id: 81a83d6fc1e71ce190f9151577deb0c5cd382a20
prerequisite-patch-id: ff5bd799e4cb905fef6cf1cd0ce757980e1d34b7
prerequisite-patch-id: a341d4e8a8e25cb711309123939235fcb3c10c1a
prerequisite-patch-id: 749e4eafb431857ee2bec4ac8e78a682f6a57588
prerequisite-patch-id: 04cd7779c7464c7be683cd2718baee8c6554791d
prerequisite-patch-id: 25cb7c602018ec89abce43d8835926ac3356f5db
prerequisite-patch-id: cf1af96990b01e1025ec50d0b49c79f92f64f352
prerequisite-patch-id: cc5878da5f03e7e1d2c15e88537fd2a93bf47e53
Signature · 1 line
2.17.1
PATCH v2 1/3 starfive: pci: Add StarFive JH7110 pcie driver
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.
--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.c
drivers/pci/Kconfig +11 -0
@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endif
drivers/pci/Makefile +1 -0
@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.o
drivers/pci/pcie_starfive_jh7110.c new file +478 -0
478 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
Hello! See few comments below.
Quoted 84 lines · On Wednesday 08 March 2023 13:48:31 Minda Chen wrote:
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +31
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C
drivers/pci/pcie_starfive_jh7110.c line 41
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0 +#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8
Please do not duplicate standard PCI macros and constants. In U-Boot they are already available in include/pci_ids.h header file.
Quoted 17 lines
drivers/pci/pcie_starfive_jh7110.c new file +17
-0+#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12)
drivers/pci/pcie_starfive_jh7110.c line 69
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12
Please do not implement duplicate PCIe ECAM code. U-Boot and also Linux kernel already provides PCIE_ECAM_OFFSET() macro.
drivers/pci/pcie_starfive_jh7110.c line 71
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19
Unused.
Quoted 24 lines
drivers/pci/pcie_starfive_jh7110.c new file +24
-0+ +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; +
drivers/pci/pcie_starfive_jh7110.c line 105
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ; +if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ;
If fisrt_busno is the bus of the root port then first_busno+1 does not have to bus of the PCIe device on the other end of the link from the root port. See for example pci_mvebu.c how this logic is handled. You probably want to also limit access to non-zero function of root port unless you have multifunction root-port device.
Quoted 8 lines
drivers/pci/pcie_starfive_jh7110.c new file +8
-0+ +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf);
drivers/pci/pcie_starfive_jh7110.c line 123
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ + if ((PCI_BUS(bdf) == 0) &&
This looks like an incorrect check for the bus of he root port. See for example pci_mvebu.c where logic should be correct.
I guess you wanted to use: PCI_BUS(bdf)-dev_seq(udev)
Quoted 4 lines
drivers/pci/pcie_starfive_jh7110.c new file +4
-0+ (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1)) +return true; + +return false;
drivers/pci/pcie_starfive_jh7110.c line 137
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev); +int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ;
Is this check needed? Does it mean that ECAM returns bogus information for disconnected devices, that it needs to be manually filtered? (Which is against PCIe spec).
drivers/pci/pcie_starfive_jh7110.c line 149
Quoted 12 lines
drivers/pci/pcie_starfive_jh7110.c new file +12
-0+ + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read (const struct udevice *udev,pci_dev_t bdf, +uint offset,ulong *valuep, +enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 );
This looks strange. There is some bug / errata that reading vendor id from ECAM does not work correctly?
drivers/pci/pcie_starfive_jh7110.c line 160
Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-0+ +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ;
Why you are ignoring writes to some offset, but not ignoring reads? And ignoring results in no-error (return value 0)?
Quoted 321 lines
drivers/pci/pcie_starfive_jh7110.c new file +318
-0+ +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); + +/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core +* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); + +return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" }, + { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +};Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
Quoted 92 lines · Pali Rohár
Hello! See few comments below.
Quoted 89 lines · On Wednesday 08 March 2023 13:48:31 Minda Chen wrote:
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +36
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C +#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0
Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 41
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8Please do not duplicate standard PCI macros and constants. In U-Boot they are already available in
include/pci_ids.hheader file.
ok
Quoted 31 lines
Quoted 17 lines
drivers/pci/pcie_starfive_jh7110.c new file +17
-0+#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12)drivers/pci/pcie_starfive_jh7110.c line 69
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12Please do not implement duplicate PCIe ECAM code. U-Boot and also Linux kernel already provides
PCIE_ECAM_OFFSET()macro.
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c line 71
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19Unused.
ok
Quoted 32 lines
Quoted 32 lines
drivers/pci/pcie_starfive_jh7110.c new file +32
-0+ +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; + +int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ;
Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 104
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ;If
fisrt_busnois the bus of the root port then first_busno+1 does not have to bus of the PCIe device on the other end of the link from the root port. See for examplepci_mvebu.chow this logic is handled. You probably want to also limit access to non-zero function of root port unless you have multifunction root-port device.
root port limitation can follow the pci_mvebu.c
Quoted 22 lines
Quoted 8 lines
drivers/pci/pcie_starfive_jh7110.c new file +8
-0+ +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf);drivers/pci/pcie_starfive_jh7110.c line 123
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ + if ((PCI_BUS(bdf) == 0) &&This looks like an incorrect check for the bus of he root port. See for example
pci_mvebu.cwhere logic should be correct.
Quoted 2 lines
I guess you wanted to use: PCI_BUS(bdf)-dev_seq(udev)
[1] This is a work around method. Our PCIe support 64bit. But the config
space is 16KB. If no limitation here, uboot will set high 32 bit base address to 0xfffffff after init. But this will make kernel pcie failed. This value will be not cleared.
While kernel read the base address first. kernel pcie can not work.
And our IC we can not reset the PCIe to clear this value.
I dont know how to fix it expcept this method.
Quoted 10 lines
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+ (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1)) +return true; + +return false; +} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev);
Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 137
Quoted 4 lines
drivers/pci/pcie_starfive_jh7110.c new file +4
-0+int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ;Is this check needed? Does it mean that ECAM returns bogus information for disconnected devices, that it needs to be manually filtered? (Which is against PCIe spec).
I found nvme device can not work if the non-zero function of root port without limitation. But I will try to fix it. Maybe I can remove this.
Quoted 7 lines
Quoted 7 lines
drivers/pci/pcie_starfive_jh7110.c new file +7
-0+ + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf, + uint offset, ulong *valuep,
Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 149
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+ enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 );This looks strange. There is some bug / errata that reading vendor id from ECAM does not work correctly?
I will remove this. It is not a errata of reading the vendor ID.
Quoted 336 lines
drivers/pci/pcie_starfive_jh7110.c line 160
Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-0+ +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ;Why you are ignoring writes to some offset, but not ignoring reads? And ignoring results in no-error (return value 0)?
The same as [1]Quoted 321 lines
drivers/pci/pcie_starfive_jh7110.c new file +318
-0+ +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); + +/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core +* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); + +return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" }, + { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +};Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
Quoted 181 lines · On Friday 10 March 2023 18:36:44 Minda Chen wrote:
Quoted 92 lines · Pali Rohár
Hello! See few comments below.
Quoted 89 lines · On Wednesday 08 March 2023 13:48:31 Minda Chen wrote:
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +36
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C +#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 41
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8Please do not duplicate standard PCI macros and constants. In U-Boot they are already available in
include/pci_ids.hheader file.ok
Quoted 31 lines
Quoted 17 lines
drivers/pci/pcie_starfive_jh7110.c new file +17
-0+#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12)drivers/pci/pcie_starfive_jh7110.c line 69
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12Please do not implement duplicate PCIe ECAM code. U-Boot and also Linux kernel already provides
PCIE_ECAM_OFFSET()macro.Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c line 71
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19Unused.
ok
Quoted 32 lines
Quoted 32 lines
drivers/pci/pcie_starfive_jh7110.c new file +32
-0+ +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; + +int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ;Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 104
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ;If
fisrt_busnois the bus of the root port then first_busno+1 does not have to bus of the PCIe device on the other end of the link from the root port. See for examplepci_mvebu.chow this logic is handled. You probably want to also limit access to non-zero function of root port unless you have multifunction root-port device.
Quoted 1 line
root port limitation can follow the
pci_mvebu.c
It can follow, but question is why it should follow pci_mvebu.c. Marvell PCIe controller does not use ECAM, instead it uses Intel PCI conf1 mechanism and moreover it contains HW bugs which are workarounded in that filter code.
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
drivers/pci/pcie_starfive_jh7110.c new file +8
-0+ +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf);drivers/pci/pcie_starfive_jh7110.c line 123
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ + if ((PCI_BUS(bdf) == 0) &&This looks like an incorrect check for the bus of he root port. See for example
pci_mvebu.cwhere logic should be correct.
Quoted 3 lines
Quoted 2 lines
I guess you wanted to use: PCI_BUS(bdf)-dev_seq(udev)
[1] This is a work around method. Our PCIe support 64bit. But the configspace is 16KB. If no limitation here, uboot will set high 32 bit base address to
0xfffffffafter init.
I just do not understand. BAR configures PCIe MEM space mapping and it is fully independent of the PCIe config space. I do not see reason for limitation.
Quoted 4 lines
But this will make kernel pcie failed. This value will be not cleared. While kernel read the base address first. kernel pcie can not work. And our IC we can not reset the PCIe to clear this value.
I dont know how to fix it expcept this method.
Well, this sounds like a bug in kernel, which should be fixed or at least reported.
And if in the u-boot is workaround for particular kernel bug then it should be properly documented with information in which kernel version was bug fixed. Otherwise u-boot code stay in non-maintenanceble state.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+ (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1)) +return true; + +return false; +} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev);Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 137
Quoted 4 lines
drivers/pci/pcie_starfive_jh7110.c new file +4
-0+int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ;Is this check needed? Does it mean that ECAM returns bogus information for disconnected devices, that it needs to be manually filtered? (Which is against PCIe spec).
Quoted 2 lines
I found nvme device can not work if the non-zero function of root port without limitation. But I will try to fix it. Maybe I can remove this.
So you have multifunction root port device (if you are talking about non-zero function)? And what does it mean that NVMe device cannot work? Can you connect your device to second function of root port if it works?
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/pci/pcie_starfive_jh7110.c new file +7
-0+ + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf, + uint offset, ulong *valuep,Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 149
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+ enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 );This looks strange. There is some bug / errata that reading vendor id from ECAM does not work correctly?
Quoted 16 lines
I will remove this. It is not a errata of reading the vendor ID.
Quoted 15 lines
drivers/pci/pcie_starfive_jh7110.c line 160
Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-0+ +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ;Why you are ignoring writes to some offset, but not ignoring reads? And ignoring results in no-error (return value 0)?
The same as [1]
But this does not explain why is result ignored. I'm just lost what is aim or expected result here.
Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
drivers/pci/pcie_starfive_jh7110.c new file +318
-0+ +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); + +/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core +* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); + +return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" }, + { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +};Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
Quoted 182 lines · Pali Rohár
Quoted 181 lines · On Friday 10 March 2023 18:36:44 Minda Chen wrote:
Quoted 92 lines · Pali Rohár
Hello! See few comments below.
Quoted 89 lines · On Wednesday 08 March 2023 13:48:31 Minda Chen wrote:
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +36
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C +#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 41
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8Please do not duplicate standard PCI macros and constants. In U-Boot they are already available in
include/pci_ids.hheader file.ok
Quoted 31 lines
Quoted 17 lines
drivers/pci/pcie_starfive_jh7110.c new file +17
-0+#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12)drivers/pci/pcie_starfive_jh7110.c line 69
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12Please do not implement duplicate PCIe ECAM code. U-Boot and also Linux kernel already provides
PCIE_ECAM_OFFSET()macro.Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c line 71
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19Unused.
ok
Quoted 41 lines
Quoted 24 lines
drivers/pci/pcie_starfive_jh7110.c new file +24
-0+ +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; +drivers/pci/pcie_starfive_jh7110.c line 105
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ; +if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ;If
fisrt_busnois the bus of the root port then first_busno+1 does not have to bus of the PCIe device on the other end of the link from the root port. See for examplepci_mvebu.chow this logic is handled. You probably want to also limit access to non-zero function of root port unless you have multifunction root-port device.
Quoted 7 lines
Quoted 1 line
root port limitation can follow the
pci_mvebu.cIt can follow, but question is why it should follow
pci_mvebu.c. Marvell PCIe controller does not use ECAM, instead it uses Intel PCI conf1 mechanism and moreover it contains HW bugs which are workarounded in that filter code.
starfive_pcie_addr_valid limit the access the non-zero device.
The limitaion reason is list below
For PCIe bus of JH7110 SoC limitation, The PCIe host bridge of just support one device.
(The simple topic graph)
================ JH7110 SoC ==================
[Processor]
|
(Root Complex)
|
------------------------------
[Host Bridge Domain 0000]
|
(Bus 0)
|
[Bus 0 Device 0]
|
================ JH7110 SoC ==================
|
(BUS 1)
|
[BUS 1 Device 0]To avoid access non-zero device config space, add this limitation.
And I can see designware PCIe code (pcie_dw_common.c) have the same code logic(pcie_dw_addr_valid).
Quoted 35 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
drivers/pci/pcie_starfive_jh7110.c new file +8
-0+ +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf);drivers/pci/pcie_starfive_jh7110.c line 123
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ + if ((PCI_BUS(bdf) == 0) &&This looks like an incorrect check for the bus of he root port. See for example
pci_mvebu.cwhere logic should be correct.Quoted 3 lines
Quoted 2 lines
I guess you wanted to use: PCI_BUS(bdf)-dev_seq(udev)
[1] This is a work around method. Our PCIe support 64bit. But the configspace is 16KB. If no limitation here, uboot will set high 32 bit base address to
0xfffffffafter init.I just do not understand. BAR configures PCIe MEM space mapping and it is fully independent of the PCIe config space. I do not see reason for limitation.
Quoted 4 lines
But this will make kernel pcie failed. This value will be not cleared. While kernel read the base address first. kernel pcie can not work. And our IC we can not reset the PCIe to clear this value.
I dont know how to fix it expcept this method.
Quoted 6 lines
Well, this sounds like a bug in kernel, which should be fixed or at least reported.
Quoted 1 line
And if in the u-boot is workaround for particular kernel bug then it
should be properly documented with information in which kernel version was bug fixed. Otherwise u-boot code stay in non-maintenanceble state.
I am not sure whether it is a kernel issue. For this are workaround codes,
I will remove this function (starfive_pcie_hide_rc_bar) in next version. Remove this codes uboot PCIe still can work.
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 4 lines
drivers/pci/pcie_starfive_jh7110.c new file +4
-0+ (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1)) +return true; + +return false;drivers/pci/pcie_starfive_jh7110.c line 137
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev); +int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ;Is this check needed? Does it mean that ECAM returns bogus information for disconnected devices, that it needs to be manually filtered? (Which is against PCIe spec).
Quoted 2 lines
Quoted 2 lines
I found nvme device can not work if the non-zero function of root port without limitation. But I will try to fix it. Maybe I can remove this.
Please ignore this reply. I have replied wrong message. I am sorry about this It is not 'non-zero function'. I should said 'non-zero device'. The check reason see starfive_pcie_addr_valid reply.
Quoted 4 lines
So you have multifunction root port device (if you are talking about non-zero function)? And what does it mean that NVMe device cannot work? Can you connect your device to second function of root port if it works?
Please ignore previous reply. And the test NVMe devices do not support multi-function.
Quoted 33 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/pci/pcie_starfive_jh7110.c new file +7
-0+ + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf, + uint offset, ulong *valuep,Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 149
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+ enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 );This looks strange. There is some bug / errata that reading vendor id from ECAM does not work correctly?
Quoted 16 lines
I will remove this. It is not a errata of reading the vendor ID.
Quoted 15 lines
drivers/pci/pcie_starfive_jh7110.c line 160
Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-0+ +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ;Why you are ignoring writes to some offset, but not ignoring reads? And ignoring results in no-error (return value 0)?
The same as [1]
Quoted 3 lines
But this does not explain why is result ignored. I'm just lost what is aim or expected result here.
Since it is a workaround codes for booting kernel, (Writing root port base0 and base 1 register cause kernel PCIe driver can not work, So just limtation writing base0 and bas1). I will remove starfive_pcie_hide_rc_bar as previous said.
Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
drivers/pci/pcie_starfive_jh7110.c new file +318
-0+ +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); + +/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core +* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); + +return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" }, + { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +};Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
Quoted 214 lines · On Thursday 23 March 2023 18:51:38 Minda Chen wrote:
Quoted 182 lines · Pali Rohár
Quoted 181 lines · On Friday 10 March 2023 18:36:44 Minda Chen wrote:
Quoted 92 lines · Pali Rohár
Hello! See few comments below.
Quoted 89 lines · On Wednesday 08 March 2023 13:48:31 Minda Chen wrote:
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +36
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C +#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 41
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8Please do not duplicate standard PCI macros and constants. In U-Boot they are already available in
include/pci_ids.hheader file.ok
Quoted 31 lines
Quoted 17 lines
drivers/pci/pcie_starfive_jh7110.c new file +17
-0+#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12)drivers/pci/pcie_starfive_jh7110.c line 69
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12Please do not implement duplicate PCIe ECAM code. U-Boot and also Linux kernel already provides
PCIE_ECAM_OFFSET()macro.Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c line 71
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19Unused.
ok
Quoted 41 lines
Quoted 24 lines
drivers/pci/pcie_starfive_jh7110.c new file +24
-0+ +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; +drivers/pci/pcie_starfive_jh7110.c line 105
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ; +if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ;If
fisrt_busnois the bus of the root port then first_busno+1 does not have to bus of the PCIe device on the other end of the link from the root port. See for examplepci_mvebu.chow this logic is handled. You probably want to also limit access to non-zero function of root port unless you have multifunction root-port device.Quoted 7 lines
Quoted 1 line
root port limitation can follow the
pci_mvebu.cIt can follow, but question is why it should follow
pci_mvebu.c. Marvell PCIe controller does not use ECAM, instead it uses Intel PCI conf1 mechanism and moreover it contains HW bugs which are workarounded in that filter code.
starfive_pcie_addr_validlimit the access the non-zero device.The limitaion reason is list below
For PCIe bus of JH7110 SoC limitation, The PCIe host bridge of just support one device.(The simple topic graph)
================ JH7110 SoC ==================[Processor] | (Root Complex) | ------------------------------ [Host Bridge Domain 0000] | (Bus 0) | [Bus 0 Device 0] | ================ JH7110 SoC ================== | (BUS 1) | [BUS 1 Device 0]
Quoted 1 line
To avoid access non-zero device config space, add this limitation.
Ok. The hierarchy is quite standard. But question is why you need to do explicit filtering in the driver? ECAM should correctly work also when you ask non-existent device (e.g. by returning all-ones).
Is the driver working fine also without filtering?
Quoted 2 lines
And I can see designware PCIe code (
pcie_dw_common.c) have the same codelogic(pcie_dw_addr_valid).
In mvebu is explicit filtering because of HW bug which cause that HW ignores device number. And so device 0 is aliased to device 2, 3, 4... So to prevent kernel to see tons of non-existent devices, mvebu driver has to filter access.
But normally any special logic should not be needed unless there is a HW bug. HW just normally returns to kernel that there is no other device.
mvebu driver is not something useful for taking as a template driver for building a new driver because it contains tons of code for workarounding real HW issues without which other kernel (and userspace) PCIe parts do not work.
Quoted 35 lines
Quoted 35 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
drivers/pci/pcie_starfive_jh7110.c new file +8
-0+ +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf);drivers/pci/pcie_starfive_jh7110.c line 123
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ + if ((PCI_BUS(bdf) == 0) &&This looks like an incorrect check for the bus of he root port. See for example
pci_mvebu.cwhere logic should be correct.Quoted 3 lines
Quoted 2 lines
I guess you wanted to use: PCI_BUS(bdf)-dev_seq(udev)
[1] This is a work around method. Our PCIe support 64bit. But the configspace is 16KB. If no limitation here, uboot will set high 32 bit base address to
0xfffffffafter init.I just do not understand. BAR configures PCIe MEM space mapping and it is fully independent of the PCIe config space. I do not see reason for limitation.
Quoted 4 lines
But this will make kernel pcie failed. This value will be not cleared. While kernel read the base address first. kernel pcie can not work. And our IC we can not reset the PCIe to clear this value.
I dont know how to fix it expcept this method.
Quoted 8 lines
Quoted 6 lines
Well, this sounds like a bug in kernel, which should be fixed or at least reported.
Quoted 1 line
And if in the u-boot is workaround for particular kernel bug then it
should be properly documented with information in which kernel version was bug fixed. Otherwise u-boot code stay in non-maintenanceble state.
I am not sure whether it is a kernel issue. For this are workaround codes,
I will remove this function (starfive_pcie_hide_rc_bar) in next version. Remove this codes uboot PCIe still can work.
If everything is working fine then I do not see reason for having it there.
Quoted 390 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 4 lines
drivers/pci/pcie_starfive_jh7110.c new file +4
-0+ (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1)) +return true; + +return false;drivers/pci/pcie_starfive_jh7110.c line 137
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev); +int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ;Is this check needed? Does it mean that ECAM returns bogus information for disconnected devices, that it needs to be manually filtered? (Which is against PCIe spec).
Quoted 2 lines
Quoted 2 lines
I found nvme device can not work if the non-zero function of root port without limitation. But I will try to fix it. Maybe I can remove this.
Please ignore this reply. I have replied wrong message. I am sorry about this It is not 'non-zero function'. I should said 'non-zero device'. The check reason see
starfive_pcie_addr_validreply.Quoted 4 lines
So you have multifunction root port device (if you are talking about non-zero function)? And what does it mean that NVMe device cannot work? Can you connect your device to second function of root port if it works?
Please ignore previous reply. And the test NVMe devices do not support multi-function.
Quoted 33 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/pci/pcie_starfive_jh7110.c new file +7
-0+ + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf, + uint offset, ulong *valuep,Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 149
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+ enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 );This looks strange. There is some bug / errata that reading vendor id from ECAM does not work correctly?
Quoted 16 lines
I will remove this. It is not a errata of reading the vendor ID.
Quoted 15 lines
drivers/pci/pcie_starfive_jh7110.c line 160
Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-0+ +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ;Why you are ignoring writes to some offset, but not ignoring reads? And ignoring results in no-error (return value 0)?
The same as [1]Quoted 3 lines
But this does not explain why is result ignored. I'm just lost what is aim or expected result here.
Since it is a workaround codes for booting kernel, (Writing root port base0 and base 1 register cause kernel PCIe driver can not work, So just limtation writing base0 and bas1). I will remove
starfive_pcie_hide_rc_baras previous said.Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
drivers/pci/pcie_starfive_jh7110.c new file +318
-0+ +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); + +/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core +* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); + +return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" }, + { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +};Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
Quoted 231 lines · Pali Rohár
Quoted 214 lines · On Thursday 23 March 2023 18:51:38 Minda Chen wrote:
Quoted 182 lines · Pali Rohár
Quoted 181 lines · On Friday 10 March 2023 18:36:44 Minda Chen wrote:
Quoted 92 lines · Pali Rohár
Hello! See few comments below.
Quoted 89 lines · On Wednesday 08 March 2023 13:48:31 Minda Chen wrote:
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +36
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C +#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 41
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8Please do not duplicate standard PCI macros and constants. In U-Boot they are already available in
include/pci_ids.hheader file.ok
Quoted 31 lines
Quoted 17 lines
drivers/pci/pcie_starfive_jh7110.c new file +17
-0+#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12)drivers/pci/pcie_starfive_jh7110.c line 69
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12Please do not implement duplicate PCIe ECAM code. U-Boot and also Linux kernel already provides
PCIE_ECAM_OFFSET()macro.Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c line 71
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19Unused.
ok
Quoted 41 lines
Quoted 24 lines
drivers/pci/pcie_starfive_jh7110.c new file +24
-0+ +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; +drivers/pci/pcie_starfive_jh7110.c line 105
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ; +if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ;If
fisrt_busnois the bus of the root port then first_busno+1 does not have to bus of the PCIe device on the other end of the link from the root port. See for examplepci_mvebu.chow this logic is handled. You probably want to also limit access to non-zero function of root port unless you have multifunction root-port device.Quoted 7 lines
Quoted 1 line
root port limitation can follow the
pci_mvebu.cIt can follow, but question is why it should follow
pci_mvebu.c. Marvell PCIe controller does not use ECAM, instead it uses Intel PCI conf1 mechanism and moreover it contains HW bugs which are workarounded in that filter code.
starfive_pcie_addr_validlimit the access the non-zero device.The limitaion reason is list below
For PCIe bus of JH7110 SoC limitation, The PCIe host bridge of just support one device.(The simple topic graph)
================ JH7110 SoC ==================[Processor] | (Root Complex) | ------------------------------ [Host Bridge Domain 0000] | (Bus 0) | [Bus 0 Device 0] | ================ JH7110 SoC ================== | (BUS 1) | [BUS 1 Device 0]Quoted 1 line
To avoid access non-zero device config space, add this limitation.
Ok. The hierarchy is quite standard. But question is why you need to do explicit filtering in the driver? ECAM should correctly work also when you ask non-existent device (e.g. by returning all-ones).
Is the driver working fine also without filtering?
Quoted 2 lines
And I can see designware PCIe code (
pcie_dw_common.c) have the same codelogic(pcie_dw_addr_valid).In mvebu is explicit filtering because of HW bug which cause that HW ignores device number. And so device 0 is aliased to device 2, 3, 4... So to prevent kernel to see tons of non-existent devices, mvebu driver has to filter access.
Quoted 8 lines
But normally any special logic should not be needed unless there is a HW bug. HW just normally returns to kernel that there is no other device.
mvebu driver is not something useful for taking as a template driver for building a new driver because it contains tons of code for workarounding real HW issues without which other kernel (and userspace) PCIe parts do not work.
Uboot PCIe can not work well. If without filtering, uboot get duplicate devices. device 0 is aliased to device 2, 3, 4...31
Quoted 437 lines
Quoted 35 lines
Quoted 35 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
drivers/pci/pcie_starfive_jh7110.c new file +8
-0+ +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf);drivers/pci/pcie_starfive_jh7110.c line 123
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ + if ((PCI_BUS(bdf) == 0) &&This looks like an incorrect check for the bus of he root port. See for example
pci_mvebu.cwhere logic should be correct.Quoted 3 lines
Quoted 2 lines
I guess you wanted to use: PCI_BUS(bdf)-dev_seq(udev)
[1] This is a work around method. Our PCIe support 64bit. But the configspace is 16KB. If no limitation here, uboot will set high 32 bit base address to
0xfffffffafter init.I just do not understand. BAR configures PCIe MEM space mapping and it is fully independent of the PCIe config space. I do not see reason for limitation.
Quoted 4 lines
But this will make kernel pcie failed. This value will be not cleared. While kernel read the base address first. kernel pcie can not work. And our IC we can not reset the PCIe to clear this value.
I dont know how to fix it expcept this method.Quoted 8 lines
Quoted 6 lines
Well, this sounds like a bug in kernel, which should be fixed or at least reported.
Quoted 1 line
And if in the u-boot is workaround for particular kernel bug then it
should be properly documented with information in which kernel version was bug fixed. Otherwise u-boot code stay in non-maintenanceble state.
I am not sure whether it is a kernel issue. For this are workaround codes,
I will remove this function (starfive_pcie_hide_rc_bar) in next version. Remove this codes uboot PCIe still can work.If everything is working fine then I do not see reason for having it there.
Quoted 390 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 4 lines
drivers/pci/pcie_starfive_jh7110.c new file +4
-0+ (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1)) +return true; + +return false;drivers/pci/pcie_starfive_jh7110.c line 137
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev); +int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ;Is this check needed? Does it mean that ECAM returns bogus information for disconnected devices, that it needs to be manually filtered? (Which is against PCIe spec).
Quoted 2 lines
Quoted 2 lines
I found nvme device can not work if the non-zero function of root port without limitation. But I will try to fix it. Maybe I can remove this.
Please ignore this reply. I have replied wrong message. I am sorry about this It is not 'non-zero function'. I should said 'non-zero device'. The check reason see
starfive_pcie_addr_validreply.Quoted 4 lines
So you have multifunction root port device (if you are talking about non-zero function)? And what does it mean that NVMe device cannot work? Can you connect your device to second function of root port if it works?
Please ignore previous reply. And the test NVMe devices do not support multi-function.
Quoted 33 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/pci/pcie_starfive_jh7110.c new file +7
-0+ + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf, + uint offset, ulong *valuep,Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 149
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+ enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 );This looks strange. There is some bug / errata that reading vendor id from ECAM does not work correctly?
Quoted 16 lines
I will remove this. It is not a errata of reading the vendor ID.
Quoted 15 lines
drivers/pci/pcie_starfive_jh7110.c line 160
Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-0+ +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ;Why you are ignoring writes to some offset, but not ignoring reads? And ignoring results in no-error (return value 0)?
The same as [1]Quoted 3 lines
But this does not explain why is result ignored. I'm just lost what is aim or expected result here.
Since it is a workaround codes for booting kernel, (Writing root port base0 and base 1 register cause kernel PCIe driver can not work, So just limtation writing base0 and bas1). I will remove
starfive_pcie_hide_rc_baras previous said.Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
drivers/pci/pcie_starfive_jh7110.c new file +318
-0+ +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); + +/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core +* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); + +return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" }, + { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +};Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
Quoted 235 lines · On Friday 24 March 2023 18:57:33 Minda Chen wrote:
Quoted 234 lines · Pali Rohár
Quoted 214 lines · On Thursday 23 March 2023 18:51:38 Minda Chen wrote:
Quoted 182 lines · Pali Rohár
Quoted 181 lines · On Friday 10 March 2023 18:36:44 Minda Chen wrote:
Quoted 92 lines · Pali Rohár
Hello! See few comments below.
Quoted 89 lines · On Wednesday 08 March 2023 13:48:31 Minda Chen wrote:
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +36
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C +#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 41
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8Please do not duplicate standard PCI macros and constants. In U-Boot they are already available in
include/pci_ids.hheader file.ok
Quoted 31 lines
Quoted 17 lines
drivers/pci/pcie_starfive_jh7110.c new file +17
-0+#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12)drivers/pci/pcie_starfive_jh7110.c line 68
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12Please do not implement duplicate PCIe ECAM code. U-Boot and also Linux kernel already provides
PCIE_ECAM_OFFSET()macro.Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c line 70
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19Unused.
ok
Quoted 41 lines
Quoted 24 lines
drivers/pci/pcie_starfive_jh7110.c new file +24
-0+ +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; +drivers/pci/pcie_starfive_jh7110.c line 104
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ; +if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ;If
fisrt_busnois the bus of the root port then first_busno+1 does not have to bus of the PCIe device on the other end of the link from the root port. See for examplepci_mvebu.chow this logic is handled. You probably want to also limit access to non-zero function of root port unless you have multifunction root-port device.Quoted 7 lines
Quoted 1 line
root port limitation can follow the
pci_mvebu.cIt can follow, but question is why it should follow
pci_mvebu.c. Marvell PCIe controller does not use ECAM, instead it uses Intel PCI conf1 mechanism and moreover it contains HW bugs which are workarounded in that filter code.
starfive_pcie_addr_validlimit the access the non-zero device.The limitaion reason is list below
For PCIe bus of JH7110 SoC limitation, The PCIe host bridge of just support one device.(The simple topic graph)
================ JH7110 SoC ==================[Processor] | (Root Complex) | ------------------------------ [Host Bridge Domain 0000] | (Bus 0) | [Bus 0 Device 0] | ================ JH7110 SoC ================== | (BUS 1) | [BUS 1 Device 0]Quoted 1 line
To avoid access non-zero device config space, add this limitation.
Ok. The hierarchy is quite standard. But question is why you need to do explicit filtering in the driver? ECAM should correctly work also when you ask non-existent device (e.g. by returning all-ones).
Is the driver working fine also without filtering?
Quoted 2 lines
And I can see designware PCIe code (
pcie_dw_common.c) have the same codelogic(pcie_dw_addr_valid).In mvebu is explicit filtering because of HW bug which cause that HW ignores device number. And so device 0 is aliased to device 2, 3, 4... So to prevent kernel to see tons of non-existent devices, mvebu driver has to filter access.
But normally any special logic should not be needed unless there is a HW bug. HW just normally returns to kernel that there is no other device.
Quoted 7 lines
Quoted 5 lines
mvebu driver is not something useful for taking as a template driver for building a new driver because it contains tons of code for workarounding real HW issues without which other kernel (and userspace) PCIe parts do not work.
Uboot PCIe can not work well. If without filtering, uboot get duplicate devices. device 0 is aliased to device 2, 3, 4...31
Ok, so you are seeing same issue as on mvebu HW which I described above, right? And which bus is affected by this issue? Root (zero) bus? Or the bus behind it (1)?
Quoted 437 lines
Quoted 437 lines
Quoted 35 lines
Quoted 35 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
drivers/pci/pcie_starfive_jh7110.c new file +8
-0+ +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf);drivers/pci/pcie_starfive_jh7110.c line 122
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ + if ((PCI_BUS(bdf) == 0) &&This looks like an incorrect check for the bus of he root port. See for example
pci_mvebu.cwhere logic should be correct.Quoted 3 lines
Quoted 2 lines
I guess you wanted to use: PCI_BUS(bdf)-dev_seq(udev)
[1] This is a work around method. Our PCIe support 64bit. But the configspace is 16KB. If no limitation here, uboot will set high 32 bit base address to
0xfffffffafter init.I just do not understand. BAR configures PCIe MEM space mapping and it is fully independent of the PCIe config space. I do not see reason for limitation.
Quoted 4 lines
But this will make kernel pcie failed. This value will be not cleared. While kernel read the base address first. kernel pcie can not work. And our IC we can not reset the PCIe to clear this value.
I dont know how to fix it expcept this method.Quoted 8 lines
Quoted 6 lines
Well, this sounds like a bug in kernel, which should be fixed or at least reported.
Quoted 1 line
And if in the u-boot is workaround for particular kernel bug then it
should be properly documented with information in which kernel version was bug fixed. Otherwise u-boot code stay in non-maintenanceble state.
I am not sure whether it is a kernel issue. For this are workaround codes,
I will remove this function (starfive_pcie_hide_rc_bar) in next version. Remove this codes uboot PCIe still can work.If everything is working fine then I do not see reason for having it there.
Quoted 390 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 4 lines
drivers/pci/pcie_starfive_jh7110.c new file +4
-0+ (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1)) +return true; + +return false;drivers/pci/pcie_starfive_jh7110.c line 136
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev); +int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ;Is this check needed? Does it mean that ECAM returns bogus information for disconnected devices, that it needs to be manually filtered? (Which is against PCIe spec).
Quoted 2 lines
Quoted 2 lines
I found nvme device can not work if the non-zero function of root port without limitation. But I will try to fix it. Maybe I can remove this.
Please ignore this reply. I have replied wrong message. I am sorry about this It is not 'non-zero function'. I should said 'non-zero device'. The check reason see
starfive_pcie_addr_validreply.Quoted 4 lines
So you have multifunction root port device (if you are talking about non-zero function)? And what does it mean that NVMe device cannot work? Can you connect your device to second function of root port if it works?
Please ignore previous reply. And the test NVMe devices do not support multi-function.
Quoted 33 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/pci/pcie_starfive_jh7110.c new file +7
-0+ + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf, + uint offset, ulong *valuep,Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 148
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+ enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 );This looks strange. There is some bug / errata that reading vendor id from ECAM does not work correctly?
Quoted 16 lines
I will remove this. It is not a errata of reading the vendor ID.
Quoted 15 lines
drivers/pci/pcie_starfive_jh7110.c line 159
Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-0+ +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ;Why you are ignoring writes to some offset, but not ignoring reads? And ignoring results in no-error (return value 0)?
The same as [1]Quoted 3 lines
But this does not explain why is result ignored. I'm just lost what is aim or expected result here.
Since it is a workaround codes for booting kernel, (Writing root port base0 and base 1 register cause kernel PCIe driver can not work, So just limtation writing base0 and bas1). I will remove
starfive_pcie_hide_rc_baras previous said.Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
drivers/pci/pcie_starfive_jh7110.c new file +318
-0+ +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); + +/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core +* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); + +return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" }, + { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +};Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
Quoted 241 lines · Pali Rohár
Quoted 240 lines · On Friday 24 March 2023 18:57:33 Minda Chen wrote:
Quoted 239 lines · Pali Rohár
Quoted 214 lines · On Thursday 23 March 2023 18:51:38 Minda Chen wrote:
Quoted 182 lines · Pali Rohár
Quoted 181 lines · On Friday 10 March 2023 18:36:44 Minda Chen wrote:
Quoted 92 lines · Pali Rohár
Hello! See few comments below.
Quoted 89 lines · On Wednesday 08 March 2023 13:48:31 Minda Chen wrote:
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +36
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C +#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 41
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8Please do not duplicate standard PCI macros and constants. In U-Boot they are already available in
include/pci_ids.hheader file.ok
Quoted 31 lines
Quoted 17 lines
drivers/pci/pcie_starfive_jh7110.c new file +17
-0+#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12)drivers/pci/pcie_starfive_jh7110.c line 68
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12Please do not implement duplicate PCIe ECAM code. U-Boot and also Linux kernel already provides
PCIE_ECAM_OFFSET()macro.Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c line 70
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19Unused.
ok
Quoted 41 lines
Quoted 24 lines
drivers/pci/pcie_starfive_jh7110.c new file +24
-0+ +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; +drivers/pci/pcie_starfive_jh7110.c line 104
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ; +if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ;If
fisrt_busnois the bus of the root port then first_busno+1 does not have to bus of the PCIe device on the other end of the link from the root port. See for examplepci_mvebu.chow this logic is handled. You probably want to also limit access to non-zero function of root port unless you have multifunction root-port device.Quoted 7 lines
Quoted 1 line
root port limitation can follow the
pci_mvebu.cIt can follow, but question is why it should follow
pci_mvebu.c. Marvell PCIe controller does not use ECAM, instead it uses Intel PCI conf1 mechanism and moreover it contains HW bugs which are workarounded in that filter code.
starfive_pcie_addr_validlimit the access the non-zero device.The limitaion reason is list below
For PCIe bus of JH7110 SoC limitation, The PCIe host bridge of just support one device.(The simple topic graph)
================ JH7110 SoC ==================[Processor] | (Root Complex) | ------------------------------ [Host Bridge Domain 0000] | (Bus 0) | [Bus 0 Device 0] | ================ JH7110 SoC ================== | (BUS 1) | [BUS 1 Device 0]Quoted 1 line
To avoid access non-zero device config space, add this limitation.
Ok. The hierarchy is quite standard. But question is why you need to do explicit filtering in the driver? ECAM should correctly work also when you ask non-existent device (e.g. by returning all-ones).
Is the driver working fine also without filtering?
Quoted 2 lines
And I can see designware PCIe code (
pcie_dw_common.c) have the same codelogic(pcie_dw_addr_valid).In mvebu is explicit filtering because of HW bug which cause that HW ignores device number. And so device 0 is aliased to device 2, 3, 4... So to prevent kernel to see tons of non-existent devices, mvebu driver has to filter access.
But normally any special logic should not be needed unless there is a HW bug. HW just normally returns to kernel that there is no other device.
mvebu driver is not something useful for taking as a template driver for building a new driver because it contains tons of code for workarounding real HW issues without which other kernel (and userspace) PCIe parts do not work.
Quoted 7 lines
Quoted 2 lines
Uboot PCIe can not work well. If without filtering, uboot get duplicate devices. device 0 is aliased to device 2, 3, 4...31
Ok, so you are seeing same issue as on mvebu HW which I described above, right? And which bus is affected by this issue? Root (zero) bus? Or the bus behind it (1)?
bus 1.
Quoted 437 lines
Quoted 437 lines
Quoted 437 lines
Quoted 35 lines
Quoted 35 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
drivers/pci/pcie_starfive_jh7110.c new file +8
-0+ +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf);drivers/pci/pcie_starfive_jh7110.c line 122
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ + if ((PCI_BUS(bdf) == 0) &&This looks like an incorrect check for the bus of he root port. See for example
pci_mvebu.cwhere logic should be correct.Quoted 3 lines
Quoted 2 lines
I guess you wanted to use: PCI_BUS(bdf)-dev_seq(udev)
[1] This is a work around method. Our PCIe support 64bit. But the configspace is 16KB. If no limitation here, uboot will set high 32 bit base address to
0xfffffffafter init.I just do not understand. BAR configures PCIe MEM space mapping and it is fully independent of the PCIe config space. I do not see reason for limitation.
Quoted 4 lines
But this will make kernel pcie failed. This value will be not cleared. While kernel read the base address first. kernel pcie can not work. And our IC we can not reset the PCIe to clear this value.
I dont know how to fix it expcept this method.Quoted 8 lines
Quoted 6 lines
Well, this sounds like a bug in kernel, which should be fixed or at least reported.
Quoted 1 line
And if in the u-boot is workaround for particular kernel bug then it
should be properly documented with information in which kernel version was bug fixed. Otherwise u-boot code stay in non-maintenanceble state.
I am not sure whether it is a kernel issue. For this are workaround codes,
I will remove this function (starfive_pcie_hide_rc_bar) in next version. Remove this codes uboot PCIe still can work.If everything is working fine then I do not see reason for having it there.
Quoted 390 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 4 lines
drivers/pci/pcie_starfive_jh7110.c new file +4
-0+ (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1)) +return true; + +return false;drivers/pci/pcie_starfive_jh7110.c line 136
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev); +int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ;Is this check needed? Does it mean that ECAM returns bogus information for disconnected devices, that it needs to be manually filtered? (Which is against PCIe spec).
Quoted 2 lines
Quoted 2 lines
I found nvme device can not work if the non-zero function of root port without limitation. But I will try to fix it. Maybe I can remove this.
Please ignore this reply. I have replied wrong message. I am sorry about this It is not 'non-zero function'. I should said 'non-zero device'. The check reason see
starfive_pcie_addr_validreply.Quoted 4 lines
So you have multifunction root port device (if you are talking about non-zero function)? And what does it mean that NVMe device cannot work? Can you connect your device to second function of root port if it works?
Please ignore previous reply. And the test NVMe devices do not support multi-function.
Quoted 33 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/pci/pcie_starfive_jh7110.c new file +7
-0+ + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf, + uint offset, ulong *valuep,Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 148
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+ enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 );This looks strange. There is some bug / errata that reading vendor id from ECAM does not work correctly?
Quoted 16 lines
I will remove this. It is not a errata of reading the vendor ID.
Quoted 15 lines
drivers/pci/pcie_starfive_jh7110.c line 159
Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-0+ +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ;Why you are ignoring writes to some offset, but not ignoring reads? And ignoring results in no-error (return value 0)?
The same as [1]Quoted 3 lines
But this does not explain why is result ignored. I'm just lost what is aim or expected result here.
Since it is a workaround codes for booting kernel, (Writing root port base0 and base 1 register cause kernel PCIe driver can not work, So just limtation writing base0 and bas1). I will remove
starfive_pcie_hide_rc_baras previous said.Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
drivers/pci/pcie_starfive_jh7110.c new file +318
-0+ +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); + +/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core +* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); + +return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" }, + { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +};Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
Quoted 244 lines · On Monday 27 March 2023 09:03:46 Minda Chen wrote:
Quoted 241 lines · Pali Rohár
Quoted 240 lines · On Friday 24 March 2023 18:57:33 Minda Chen wrote:
Quoted 239 lines · Pali Rohár
Quoted 214 lines · On Thursday 23 March 2023 18:51:38 Minda Chen wrote:
Quoted 182 lines · Pali Rohár
Quoted 181 lines · On Friday 10 March 2023 18:36:44 Minda Chen wrote:
Quoted 92 lines · Pali Rohár
Hello! See few comments below.
Quoted 89 lines · On Wednesday 08 March 2023 13:48:31 Minda Chen wrote:
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +36
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C +#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 41
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8Please do not duplicate standard PCI macros and constants. In U-Boot they are already available in
include/pci_ids.hheader file.ok
Quoted 31 lines
Quoted 17 lines
drivers/pci/pcie_starfive_jh7110.c new file +17
-0+#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12)drivers/pci/pcie_starfive_jh7110.c line 68
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12Please do not implement duplicate PCIe ECAM code. U-Boot and also Linux kernel already provides
PCIE_ECAM_OFFSET()macro.Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c line 70
Quoted 2 lines
drivers/pci/pcie_starfive_jh7110.c new file +2
-0+/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19Unused.
ok
Quoted 41 lines
Quoted 24 lines
drivers/pci/pcie_starfive_jh7110.c new file +24
-0+ +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; +drivers/pci/pcie_starfive_jh7110.c line 104
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ; +if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ;If
fisrt_busnois the bus of the root port then first_busno+1 does not have to bus of the PCIe device on the other end of the link from the root port. See for examplepci_mvebu.chow this logic is handled. You probably want to also limit access to non-zero function of root port unless you have multifunction root-port device.Quoted 7 lines
Quoted 1 line
root port limitation can follow the
pci_mvebu.cIt can follow, but question is why it should follow
pci_mvebu.c. Marvell PCIe controller does not use ECAM, instead it uses Intel PCI conf1 mechanism and moreover it contains HW bugs which are workarounded in that filter code.
starfive_pcie_addr_validlimit the access the non-zero device.The limitaion reason is list below
For PCIe bus of JH7110 SoC limitation, The PCIe host bridge of just support one device.(The simple topic graph)
================ JH7110 SoC ==================[Processor] | (Root Complex) | ------------------------------ [Host Bridge Domain 0000] | (Bus 0) | [Bus 0 Device 0] | ================ JH7110 SoC ================== | (BUS 1) | [BUS 1 Device 0]Quoted 1 line
To avoid access non-zero device config space, add this limitation.
Ok. The hierarchy is quite standard. But question is why you need to do explicit filtering in the driver? ECAM should correctly work also when you ask non-existent device (e.g. by returning all-ones).
Is the driver working fine also without filtering?
Quoted 2 lines
And I can see designware PCIe code (
pcie_dw_common.c) have the same codelogic(pcie_dw_addr_valid).In mvebu is explicit filtering because of HW bug which cause that HW ignores device number. And so device 0 is aliased to device 2, 3, 4... So to prevent kernel to see tons of non-existent devices, mvebu driver has to filter access.
But normally any special logic should not be needed unless there is a HW bug. HW just normally returns to kernel that there is no other device.
mvebu driver is not something useful for taking as a template driver for building a new driver because it contains tons of code for workarounding real HW issues without which other kernel (and userspace) PCIe parts do not work.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Uboot PCIe can not work well. If without filtering, uboot get duplicate devices. device 0 is aliased to device 2, 3, 4...31
Ok, so you are seeing same issue as on mvebu HW which I described above, right? And which bus is affected by this issue? Root (zero) bus? Or the bus behind it (1)?
bus 1.
Ok, in this case filter out bus behind the root bus and describe this HW bug in code comment, so it would be clear why it is needed.
And anyway, could you report this ECAM bug to HW engineers, so they can fix it in new version of PCIe controller? Because this is violation of ECAM as described in the PCIe base standard.
Quoted 437 lines
Quoted 437 lines
Quoted 437 lines
Quoted 437 lines
Quoted 35 lines
Quoted 35 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
drivers/pci/pcie_starfive_jh7110.c new file +8
-0+ +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf);drivers/pci/pcie_starfive_jh7110.c line 122
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ + if ((PCI_BUS(bdf) == 0) &&This looks like an incorrect check for the bus of he root port. See for example
pci_mvebu.cwhere logic should be correct.Quoted 3 lines
Quoted 2 lines
I guess you wanted to use: PCI_BUS(bdf)-dev_seq(udev)
[1] This is a work around method. Our PCIe support 64bit. But the configspace is 16KB. If no limitation here, uboot will set high 32 bit base address to
0xfffffffafter init.I just do not understand. BAR configures PCIe MEM space mapping and it is fully independent of the PCIe config space. I do not see reason for limitation.
Quoted 4 lines
But this will make kernel pcie failed. This value will be not cleared. While kernel read the base address first. kernel pcie can not work. And our IC we can not reset the PCIe to clear this value.
I dont know how to fix it expcept this method.Quoted 8 lines
Quoted 6 lines
Well, this sounds like a bug in kernel, which should be fixed or at least reported.
Quoted 1 line
And if in the u-boot is workaround for particular kernel bug then it
should be properly documented with information in which kernel version was bug fixed. Otherwise u-boot code stay in non-maintenanceble state.
I am not sure whether it is a kernel issue. For this are workaround codes,
I will remove this function (starfive_pcie_hide_rc_bar) in next version. Remove this codes uboot PCIe still can work.If everything is working fine then I do not see reason for having it there.
Quoted 390 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 4 lines
drivers/pci/pcie_starfive_jh7110.c new file +4
-0+ (offset == PCI_BASE_ADDRESS_0 || offset == PCI_BASE_ADDRESS_1)) +return true; + +return false;drivers/pci/pcie_starfive_jh7110.c line 136
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev); +int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ;Is this check needed? Does it mean that ECAM returns bogus information for disconnected devices, that it needs to be manually filtered? (Which is against PCIe spec).
Quoted 2 lines
Quoted 2 lines
I found nvme device can not work if the non-zero function of root port without limitation. But I will try to fix it. Maybe I can remove this.
Please ignore this reply. I have replied wrong message. I am sorry about this It is not 'non-zero function'. I should said 'non-zero device'. The check reason see
starfive_pcie_addr_validreply.Quoted 4 lines
So you have multifunction root port device (if you are talking about non-zero function)? And what does it mean that NVMe device cannot work? Can you connect your device to second function of root port if it works?
Please ignore previous reply. And the test NVMe devices do not support multi-function.
Quoted 33 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
drivers/pci/pcie_starfive_jh7110.c new file +7
-0+ + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read(const struct udevice *udev, pci_dev_t bdf, + uint offset, ulong *valuep,Quoted 9 lines
drivers/pci/pcie_starfive_jh7110.c line 148
Quoted 5 lines
drivers/pci/pcie_starfive_jh7110.c new file +5
-0+ enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 );This looks strange. There is some bug / errata that reading vendor id from ECAM does not work correctly?
Quoted 16 lines
I will remove this. It is not a errata of reading the vendor ID.
Quoted 15 lines
drivers/pci/pcie_starfive_jh7110.c line 159
Quoted 11 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-0+ +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ;Why you are ignoring writes to some offset, but not ignoring reads? And ignoring results in no-error (return value 0)?
The same as [1]Quoted 3 lines
But this does not explain why is result ignored. I'm just lost what is aim or expected result here.
Since it is a workaround codes for booting kernel, (Writing root port base0 and base 1 register cause kernel PCIe driver can not work, So just limtation writing base0 and bas1). I will remove
starfive_pcie_hide_rc_baras previous said.Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
Quoted 321 lines
drivers/pci/pcie_starfive_jh7110.c new file +318
-0+ +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); + +/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core +* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); + +return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" }, + { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +};Signature · 1 line
2.17.1
Re: [PATCH v2 1/3] starfive: pci: Add StarFive JH7110 pcie driver
In reply to Minda ChenQuoted 433 lines · Minda Chen
From: Mason Huo <[email protected]>
Add pcie driver for StarFive JH7110, the driver depends on starfive gpio, pinctrl, clk and reset driver to do init.
Several devices are tested:
a) M.2 NVMe SSD
b) Realtek 8169 Ethernet adapter.Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- drivers/pci/Kconfig | 11 + drivers/pci/Makefile | 1 + drivers/pci/pcie_starfive_jh7110.c | 478 +++++++++++++++++++++++++++++ 3 files changed, 490 insertions(+) create mode 100644 drivers/pci/pcie_starfive_jh7110.cdrivers/pci/Kconfig +11
-0@@ -374,4 +374,15 @@ config PCIE_UNIPHIER Say Y here if you want to enable PCIe controller support on UniPhier SoCs. +config PCIE_STARFIVE_JH7110 + bool "Enable Starfive JH7110 PCIe driver" + depends on STARFIVE_JH7110 + depends on PINCTRL_STARFIVE_JH7110 + depends on CLK_JH7110 + depends on RESET_JH7110 + default y + help + Say Y here if you want to enable PCIe controller support on + StarFive JH7110 SoC. + endifdrivers/pci/Makefile +1
-0@@ -49,3 +49,4 @@ obj-$(CONFIG_PCI_OCTEONTX) += pci_octeontx.o obj-$(CONFIG_PCIE_OCTEON) += pcie_octeon.o obj-$(CONFIG_PCIE_DW_SIFIVE) += pcie_dw_sifive.o obj-$(CONFIG_PCIE_UNIPHIER) += pcie_uniphier.o +obj-$(CONFIG_PCIE_STARFIVE_JH7110) += pcie_starfive_jh7110.odrivers/pci/pcie_starfive_jh7110.c new file +379
-0@@ -0,0 +1,478 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* StarFive PLDA PCIe host controller driver +* +* Copyright (c) 2023 Starfive, Inc. +* Author: Mason Huo <[email protected]> +* +*/ + +#include <common.h> +#include <clk.h> +#include <dm.h> +#include <generic-phy.h> +#include <pci.h> +#include <power-domain.h> +#include <regmap.h> +#include <reset.h> +#include <syscon.h> +#include <asm/global_data.h> +#include <asm/io.h> +#include <asm-generic/gpio.h> +#include <dm/device_compat.h> +#include <dm/pinctrl.h> +#include <linux/delay.h> +#include <linux/iopoll.h> +#include <power/regulator.h> + +DECLARE_GLOBAL_DATA_PTR ; + +#define GEN_SETTINGS 0x80 +#define PCIE_PCI_IDS 0x9C +#define PCIE_WINROM 0xFC +#define PMSG_SUPPORT_RX 0x3F0 +#define PCI_MISC 0xB4 + +#define PLDA_EP_ENABLE 0 +#define PLDA_RP_ENABLE 1 + +#define IDS_REVISION_ID 0x02 +#define IDS_PCI_TO_PCI_BRIDGE 0x060400 +#define IDS_CLASS_CODE_SHIFT 8 + +#define PREF_MEM_WIN_64_SUPPORT BIT(3) +#define PMSG_LTR_SUPPORT BIT(2) +#define PLDA_FUNCTION_DIS BIT(15) +#define PLDA_FUNC_NUM 4 +#define PLDA_PHY_FUNC_SHIFT 9 + +#define XR3PCI_ATR_AXI4_SLV0 0x800 +#define XR3PCI_ATR_SRC_ADDR_LOW 0x0 +#define XR3PCI_ATR_SRC_ADDR_HIGH 0x4 +#define XR3PCI_ATR_TRSL_ADDR_LOW 0x8 +#define XR3PCI_ATR_TRSL_ADDR_HIGH 0xc +#define XR3PCI_ATR_TRSL_PARAM 0x10 +#define XR3PCI_ATR_TABLE_OFFSET 0x20 +#define XR3PCI_ATR_MAX_TABLE_NUM 8 + +#define XR3PCI_ATR_SRC_WIN_SIZE_SHIFT 1 +#define XR3PCI_ATR_SRC_ADDR_MASK GENMASK(31, 12) +#define XR3PCI_ATR_TRSL_ADDR_MASK GENMASK(31, 12) +#define XR3_PCI_ECAM_SIZE 28 +#define XR3PCI_ATR_TRSL_DIR BIT(22) +/* IDs used in the XR3PCI_ATR_TRSL_PARAM */ +#define XR3PCI_ATR_TRSLID_PCIE_MEMORY 0x0 +#define XR3PCI_ATR_TRSLID_PCIE_CONFIG 0x1 + +#define ECAM_BUS_SHIFT 20 +#define ECAM_DEV_SHIFT 15 +#define ECAM_FUNC_SHIFT 12 +/* Secondary bus number offset in config space */ +#define PCI_SECONDARY_BUS 0x19 + +/* system control */ +#define STG_SYSCON_K_RP_NEP_MASK BIT(8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_MASK GENMASK(22, 8) +#define STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT 8 +#define STG_SYSCON_AXI4_SLVL_AWFUNC_MASK GENMASK(14, 0) +#define STG_SYSCON_CLKREQ_MASK BIT(22) +#define STG_SYSCON_CKREF_SRC_SHIFT 18 +#define STG_SYSCON_CKREF_SRC_MASK GENMASK(19, 18) + +struct starfive_pcie { +struct udevice *dev; + +void __iomem *reg_base; +void __iomem *cfg_base; + +struct regmap *regmap; +u32 stg_arfun; +u32 stg_awfun; +u32 stg_rp_nep; + +struct clk_bulk clks; +struct reset_ctl_bulk rsts; + +int atr_table_num; +int first_busno; +}; + +static int starfive_pcie_addr_valid (pci_dev_t bdf,int first_busno) +{ +if ((PCI_BUS (bdf) == first_busno) && (PCI_DEV (bdf) >0 )) +return 0 ; +if ((PCI_BUS (bdf) == first_busno +1 ) && (PCI_DEV (bdf) >0 )) +return 0 ; + +return 1 ; +} + +static int starfive_pcie_off_conf (pci_dev_t bdf,uint offset) +{ +unsigned int bus =PCI_BUS (bdf); +unsigned int dev =PCI_DEV (bdf); +unsigned int func =PCI_FUNC (bdf); + +return (bus <<ECAM_BUS_SHIFT ) | (dev <<ECAM_DEV_SHIFT ) | + (func <<ECAM_FUNC_SHIFT ) | offset; +} + +static bool starfive_pcie_hide_rc_bar (pci_dev_t bdf,int offset) +{ +/* Only root port 1 is in use */ +if ((PCI_BUS (bdf) ==0 ) && + (offset ==PCI_BASE_ADDRESS_0 || offset ==PCI_BASE_ADDRESS_1 )) +return true; + +return false; +} + +static int starfive_pcie_conf_address (const struct udevice *udev,pci_dev_t bdf, +uint offset,void **paddr) +{ +struct starfive_pcie *priv =dev_get_priv (udev); +int where =starfive_pcie_off_conf (bdf, offset); + +if (!starfive_pcie_addr_valid (bdf, priv->first_busno)) +return -EINVAL ; + + *paddr = (void *)(priv->cfg_base + where); +return 0 ; +} + +static int starfive_pcie_config_read (const struct udevice *udev,pci_dev_t bdf, +uint offset,ulong *valuep, +enum pci_size_t size) +{ +/* Make sure the LAST TLP is finished, before reading vendor ID. */ +if (offset ==PCI_VENDOR_ID ) +mdelay (20 ); + +return pci_generic_mmap_read_config (udev, starfive_pcie_conf_address, + bdf, offset, valuep, size); +} + +int starfive_pcie_config_write (struct udevice *udev,pci_dev_t bdf, +uint offset,ulong value, +enum pci_size_t size) +{ +if (starfive_pcie_hide_rc_bar (bdf, offset)) +return 0 ; + +return pci_generic_mmap_write_config (udev, starfive_pcie_conf_address, + bdf, offset, value, size); +} + +static int starfive_pcie_set_atr_entry (struct starfive_pcie *priv,phys_addr_t src_addr, +phys_addr_t trsl_addr,size_t window_size, +int trsl_param) +{ +void __iomem *base = + priv->reg_base +XR3PCI_ATR_AXI4_SLV0 ; + +/* Support AXI4 Slave 0 Address Translation Tables 0-7. */ +if (priv->atr_table_num >=XR3PCI_ATR_MAX_TABLE_NUM ) { +dev_err (priv->dev,"ATR table number %d exceeds max num\n" , + priv->atr_table_num); +return -EINVAL ; + } + base +=XR3PCI_ATR_TABLE_OFFSET * priv->atr_table_num; + priv->atr_table_num++; + +/* X3PCI_ATR_SRC_ADDR_LOW: +* - bit 0: enable entry, +* - bits 1-6: ATR window size: total size in bytes: 2^(ATR_WSIZE + 1) +* - bits 7-11: reserved +* - bits 12-31: start of source address +*/ +writel ((lower_32_bits (src_addr) &XR3PCI_ATR_SRC_ADDR_MASK ) | + (fls (window_size) -1 ) <<XR3PCI_ATR_SRC_WIN_SIZE_SHIFT |1 , + base +XR3PCI_ATR_SRC_ADDR_LOW ); +writel (upper_32_bits (src_addr), base +XR3PCI_ATR_SRC_ADDR_HIGH ); +writel ((lower_32_bits (trsl_addr) &XR3PCI_ATR_TRSL_ADDR_MASK ), + base +XR3PCI_ATR_TRSL_ADDR_LOW ); +writel (upper_32_bits (trsl_addr), base +XR3PCI_ATR_TRSL_ADDR_HIGH ); +writel (trsl_param, base +XR3PCI_ATR_TRSL_PARAM ); + +dev_dbg (priv->dev,"ATR entry: 0x%010llx %s 0x%010llx [0x%010llx] (param: 0x%06x)\n" , + src_addr, (trsl_param &XR3PCI_ATR_TRSL_DIR ) ?"<-" :"->" , + trsl_addr, (u64 )window_size, trsl_param); +return 0 ; +} + +static int starfive_pcie_atr_init (struct starfive_pcie *priv) +{ +struct udevice *ctlr =pci_get_controller (priv->dev); +struct pci_controller *hose =dev_get_uclass_priv (ctlr); +int i, ret; + +/* As the two host bridges in JH7110 soc have the same default +* address translation table, this cause the second root port can't +* access it's host bridge config space correctly. +* To workaround, config the ATR of host bridge config space by SW. +*/ + + ret =starfive_pcie_set_atr_entry (priv, + (phys_addr_t )priv->cfg_base, +0 , +1 <<XR3_PCI_ECAM_SIZE , +XR3PCI_ATR_TRSLID_PCIE_CONFIG ); +if (ret) +return ret; + +for (i =0 ; i < hose->region_count; i++) { +if (hose->regions[i].flags ==PCI_REGION_SYS_MEMORY ) +continue ; + +/* Only support identity mappings. */ +if (hose->regions[i].bus_start != + hose->regions[i].phys_start) +return -EINVAL ; + + ret =starfive_pcie_set_atr_entry (priv, + hose->regions[i].phys_start, + hose->regions[i].bus_start, + hose->regions[i].size, +XR3PCI_ATR_TRSLID_PCIE_MEMORY ); +if (ret) +return ret; + } + +return 0 ; +} + +static int starfive_pcie_get_syscon (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +struct udevice *syscon; +struct ofnode_phandle_args syscfg_phandle; +u32 cells[4 ]; +int ret; + +/* get corresponding syscon phandle */ + ret =dev_read_phandle_with_args (dev,"starfive,stg-syscon" ,NULL ,0 ,0 , + &syscfg_phandle); + +if (ret <0 ) { +dev_err (dev,"Can't get syscfg phandle: %d\n" , ret); +return ret; + } + + ret =uclass_get_device_by_ofnode (UCLASS_SYSCON , syscfg_phandle.node, + &syscon); +if (ret) { +dev_err (dev,"Unable to find syscon device (%d)\n" , ret); +return ret; + } + + priv->regmap =syscon_get_regmap (syscon); +if (!priv->regmap) { +dev_err (dev,"Unable to find regmap\n" ); +return -ENODEV ; + } + +/* get syscon register offset */ + ret =dev_read_u32_array (dev,"starfive,stg-syscon" , + cells,ARRAY_SIZE (cells)); +if (ret) { +dev_err (dev,"Get syscon register err %d\n" , ret); +return -EINVAL ; + } + +dev_dbg (dev,"Get syscon values: %x, %x, %x\n" , + cells[1 ], cells[2 ], cells[3 ]); + priv->stg_arfun = cells[1 ]; + priv->stg_awfun = cells[2 ]; + priv->stg_rp_nep = cells[3 ]; + +return 0 ; +} + +static int starfive_pcie_parse_dt (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->reg_base = (void *)dev_read_addr_name (dev,"reg" ); +if (priv->reg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required reg address range\n" ); +return -EINVAL ; + } + + priv->cfg_base = (void *)dev_read_addr_name (dev,"config" ); +if (priv->cfg_base == (void __iomem *)FDT_ADDR_T_NONE ) { +dev_err (dev,"Missing required config address range" ); +return -EINVAL ; + } + + ret =starfive_pcie_get_syscon (dev); +if (ret) { +dev_err (dev,"Can't get syscon: %d\n" , ret); +return ret; + } + + ret =reset_get_bulk (dev, &priv->rsts); +if (ret) { +dev_err (dev,"Can't get reset: %d\n" , ret); +return ret; + } + + ret =clk_get_bulk (dev, &priv->clks); +if (ret) { +dev_err (dev,"Can't get clock: %d\n" , ret); +return ret; + } + +return 0 ; +} + +static int starfive_pcie_init_port (struct udevice *dev) +{ +int ret, i; +unsigned int value; +struct starfive_pcie *priv =dev_get_priv (dev); + + ret =clk_enable_bulk (&priv->clks); +if (ret) { +dev_err (dev,"Failed to enable clks (ret=%d)\n" , ret); +return ret; + } + + ret =reset_deassert_bulk (&priv->rsts); +if (ret) { +dev_err (dev,"Failed to deassert resets (ret=%d)\n" , ret); + goto err_deassert_clk; + } + + ret =pinctrl_select_state (dev,"perst-active" ); +if (ret) { +dev_err (dev,"Set perst-active pinctrl failed: %d\n" , ret); + goto err_deassert_reset; + } + +/* Disable physical functions except #0 */ +for (i =1 ; i <PLDA_FUNC_NUM ; i++) { +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , + (i <<PLDA_PHY_FUNC_SHIFT ) << +STG_SYSCON_AXI4_SLVL_ARFUNC_SHIFT ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , + i <<PLDA_PHY_FUNC_SHIFT ); + + value =readl (priv->reg_base +PCI_MISC ); + value |=PLDA_FUNCTION_DIS ; +writel (value, priv->reg_base +PCI_MISC ); + } + +/* Disable physical functions */ +regmap_update_bits (priv->regmap, + priv->stg_arfun, +STG_SYSCON_AXI4_SLVL_ARFUNC_MASK , +0 ); +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_AXI4_SLVL_AWFUNC_MASK , +0 ); +
drivers/pci/pcie_starfive_jh7110.c line 389
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+/* Enable root port */ + value =readl (priv->reg_base +GEN_SETTINGS ); + value |=PLDA_RP_ENABLE ; +writel (value, priv->reg_base +GEN_SETTINGS ); + +/* PCIe PCI Standard Configuration Identification Settings. */ + value = (IDS_PCI_TO_PCI_BRIDGE <<IDS_CLASS_CODE_SHIFT ) |IDS_REVISION_ID ; +writel (value, priv->reg_base +PCIE_PCI_IDS ); + +/* The LTR message forwarding of PCIe Message Reception was set by core
Wrong multi-line comment format.
Quoted 68 lines
drivers/pci/pcie_starfive_jh7110.c new file +68
-0+* as default, but the forward id & addr are also need to be reset. +* If we do not disable LTR message forwarding here, or set a legal +* forwarding address, the kernel will get stuck after this driver probe. +* To workaround, disable the LTR message forwarding support on +* PCIe Message Reception. +*/ + value =readl (priv->reg_base +PMSG_SUPPORT_RX ); + value &= ~PMSG_LTR_SUPPORT ; +writel (value, priv->reg_base +PMSG_SUPPORT_RX ); + +/* Prefetchable memory window 64-bit addressing support */ + value =readl (priv->reg_base +PCIE_WINROM ); + value |=PREF_MEM_WIN_64_SUPPORT ; +writel (value, priv->reg_base +PCIE_WINROM ); + +starfive_pcie_atr_init (priv); + +/* Ensure that PERST has been asserted for at least 300 ms */ +mdelay (300 ); + ret =pinctrl_select_state (dev,"perst-default" ); +if (ret) { +dev_err (dev,"Set perst-default pinctrl failed: %d\n" , ret); +return ret; + } + +return 0 ; + +err_deassert_clk: +clk_disable_bulk (&priv->clks); +err_deassert_reset: +reset_assert_bulk (&priv->rsts); +return ret; +} + +static int starfive_pcie_probe (struct udevice *dev) +{ +struct starfive_pcie *priv =dev_get_priv (dev); +int ret; + + priv->atr_table_num =0 ; + priv->dev = dev; + priv->first_busno =dev_seq (dev); + + ret =starfive_pcie_parse_dt (dev); +if (ret) +return ret; + +regmap_update_bits (priv->regmap, + priv->stg_rp_nep, +STG_SYSCON_K_RP_NEP_MASK , +STG_SYSCON_K_RP_NEP_MASK ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CKREF_SRC_MASK , +2 <<STG_SYSCON_CKREF_SRC_SHIFT ); + +regmap_update_bits (priv->regmap, + priv->stg_awfun, +STG_SYSCON_CLKREQ_MASK , +STG_SYSCON_CLKREQ_MASK ); + + ret =starfive_pcie_init_port (dev); +if (ret) +return ret; + +dev_err (dev,"Starfive PCIe bus probed.\n" ); +
drivers/pci/pcie_starfive_jh7110.c line 467
Quoted 10 lines
drivers/pci/pcie_starfive_jh7110.c new file +10
-0+return 0 ; +} + +static const struct dm_pci_ops starfive_pcie_ops = { + .read_config = starfive_pcie_config_read, + .write_config = starfive_pcie_config_write, +}; + +static const struct udevice_id starfive_pcie_ids[] = { + { .compatible ="starfive,jh7110-pcie" },
Is this PCIe controller a StarFive IP? Or some 3rd party licensed IP? Is this compatible string approved by the DT community?
drivers/pci/pcie_starfive_jh7110.c
Quoted 12 lines
drivers/pci/pcie_starfive_jh7110.c new file +11
-1+ { } +}; + +U_BOOT_DRIVER (starfive_pcie_drv) = { + .name ="starfive_7110_pcie" , + .id =UCLASS_PCI , + .of_match = starfive_pcie_ids, + .ops = &starfive_pcie_ops, + .probe = starfive_pcie_probe, + .priv_auto =sizeof (struct starfive_pcie ), +}; --
Regards,
Bin
PATCH v2 2/3 configs: starfive-jh7110: Add support for PCIe host driver
In reply to Minda ChenFrom: Mason Huo <[email protected]>
also add the nvme driver and rtl8169 support.
--- configs/starfive_visionfive2_defconfig | 9 +++++++++ 1 file changed, 9 insertions(+)
configs/starfive_visionfive2_defconfig +9 -0
@@ -16,6 +16,7 @@ CONFIG_SPL=y CONFIG_SPL_SPI_FLASH_SUPPORT=y CONFIG_SPL_SPI=y CONFIG_SYS_LOAD_ADDR=0x82000000 +CONFIG_SYS_PCI_64BIT=y CONFIG_TARGET_STARFIVE_VISIONFIVE2=y CONFIG_SPL_OPENSBI_LOAD_ADDR=0x40000000 CONFIG_ARCH_RV64I=y
@@ -50,8 +51,11 @@ CONFIG_SYS_CBSIZE=256 CONFIG_SYS_PBSIZE=276 CONFIG_SYS_BOOTM_LEN=0x4000000 CONFIG_CMD_MEMINFO=y +CONFIG_CMD_PCI=y CONFIG_CMD_TFTPPUT=y CONFIG_SYS_RELOC_GD_ENV_ADDR=y +CONFIG_REGMAP=y +CONFIG_SYSCON=y CONFIG_SPL_CLK_COMPOSITE_CCF=y CONFIG_CLK_COMPOSITE_CCF=y CONFIG_SPL_CLK_JH7110=y
@@ -65,6 +69,11 @@ CONFIG_SPI_FLASH_EON=y CONFIG_SPI_FLASH_GIGADEVICE=y CONFIG_SPI_FLASH_ISSI=y CONFIG_SPI_FLASH_MACRONIX=y +CONFIG_RTL8169=y +CONFIG_NVME_PCI=y +CONFIG_PCI=y +CONFIG_DM_PCI_COMPAT=y +CONFIG_PCI_REGION_MULTI_ENTRY=y CONFIG_PINCTRL=y CONFIG_PINCONF=y CONFIG_SPL_PINCTRL=y
Signature · 1 line
2.17.1
PATCH v2 3/3 riscv: dts: starfive: Enable PCIe host controller
In reply to Minda ChenFrom: Mason Huo <[email protected]>
Enable and add pinctrl configuration for PCIe host controller. Also add JH7110 stg syscon configuration.
--- .../dts/jh7110-starfive-visionfive-2.dtsi | 99 +++++++++++++++++++ arch/riscv/dts/jh7110.dtsi | 75 ++++++++++++++ 2 files changed, 174 insertions(+)
arch/riscv/dts/jh7110-starfive-visionfive-2.dtsi +99 -0
@@ -178,6 +178,87 @@ slew-rate = <0>; }; }; + + pcie0_perst_default: pcie0_perst_default { + perst-pins { + pinmux = <GPIOMUX(26, GPOUT_HIGH, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-disable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + + pcie0_perst_active: pcie0_perst_active { + perst-pins { + pinmux = <GPIOMUX(26, GPOUT_LOW, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-disable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + + pcie0_wake_default: pcie0_wake_default { + wake-pins { + pinmux = <GPIOMUX(32, GPOUT_HIGH, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-enable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + + pcie0_clkreq_default: pcie0_clkreq_default { + clkreq-pins { + pinmux = <GPIOMUX(27, GPOUT_HIGH, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-enable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + + pcie1_perst_default: pcie1_perst_default { + perst-pins { + pinmux = <GPIOMUX(28, GPOUT_HIGH, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-disable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + + pcie1_perst_active: pcie1_perst_active { + perst-pins { + pinmux = <GPIOMUX(28, GPOUT_LOW, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-disable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + + pcie1_wake_default: pcie1_wake_default { + wake-pins { + pinmux = <GPIOMUX(21, GPOUT_HIGH, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-enable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + + pcie1_clkreq_default: pcie1_clkreq_default { + clkreq-pins { + pinmux = <GPIOMUX(29, GPOUT_HIGH, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-enable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + }; &mmc0 {
@@ -234,6 +315,24 @@ }; }; +&pcie0 { + pinctrl-names = "perst-default", "perst-active", "wake-default", "clkreq-default"; + pinctrl-0 = <&pcie0_perst_default>; + pinctrl-1 = <&pcie0_perst_active>; + pinctrl-2 = <&pcie0_wake_default>; + pinctrl-3 = <&pcie0_clkreq_default>; + status = "disabled"; +}; + +&pcie1 { + pinctrl-names = "perst-default", "perst-active", "wake-default", "clkreq-default"; + pinctrl-0 = <&pcie1_perst_default>; + pinctrl-1 = <&pcie1_perst_active>; + pinctrl-2 = <&pcie1_wake_default>; + pinctrl-3 = <&pcie1_clkreq_default>; + status = "okay"; +}; + &syscrg { assigned-clocks = <&syscrg JH7110_SYSCLK_CPU_ROOT>, <&syscrg JH7110_SYSCLK_BUS_ROOT>,
arch/riscv/dts/jh7110.dtsi +75 -0
80 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v2 3/3] riscv: dts: starfive: Enable PCIe host controller
Quoted 40 lines · On Wednesday 08 March 2023 13:48:33 Minda Chen wrote:
From: Mason Huo <[email protected]>
Enable and add pinctrl configuration for PCIe host controller. Also add JH7110 stg syscon configuration.
Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- .../dts/jh7110-starfive-visionfive-2.dtsi | 99 +++++++++++++++++++ arch/riscv/dts/jh7110.dtsi | 75 ++++++++++++++ 2 files changed, 174 insertions(+)arch/riscv/dts/jh7110-starfive-visionfive-2.dtsi +20
-0@@ -178,6 +178,87 @@ slew-rate = <0>; }; }; + + pcie0_perst_default: pcie0_perst_default { + perst-pins { + pinmux = <GPIOMUX(26, GPOUT_HIGH, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-disable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + + pcie0_perst_active: pcie0_perst_active { + perst-pins { + pinmux = <GPIOMUX(26, GPOUT_LOW, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-disable; + input-schmitt-disable; + slew-rate = <0>; + }; + };
...
arch/riscv/dts/jh7110-starfive-visionfive-2.dtsi line 325
Quoted 8 lines
+&pcie0 {
arch/riscv/dts/jh7110-starfive-visionfive-2.dtsi +7
-0+ pinctrl-names = "perst-default", "perst-active", "wake-default", "clkreq-default"; + pinctrl-0 = <&pcie0_perst_default>; + pinctrl-1 = <&pcie0_perst_active>; + pinctrl-2 = <&pcie0_wake_default>; + pinctrl-3 = <&pcie0_clkreq_default>; + status = "disabled"; +};
This is not correct declaration of PERST# signal. You should define in &pcie0 node "reset-gpios" property with plain gpio definition with active state.
"reset-gpios" is standard PCIe property for controlling PERST# signal over GPIO. Look for example into drivers/pci/pci-aardvark.c and arch/arm/dts/armada-3720-espressobin.dtsi files how it is used.
Re: [PATCH v2 3/3] riscv: dts: starfive: Enable PCIe host controller
Quoted 55 lines · Pali Rohár
Quoted 40 lines · On Wednesday 08 March 2023 13:48:33 Minda Chen wrote:
From: Mason Huo <[email protected]>
Enable and add pinctrl configuration for PCIe host controller. Also add JH7110 stg syscon configuration.
Signed-off-by Mason Huo <[email protected]>Signed-off-by Minda Chen <[email protected]>--- .../dts/jh7110-starfive-visionfive-2.dtsi | 99 +++++++++++++++++++ arch/riscv/dts/jh7110.dtsi | 75 ++++++++++++++ 2 files changed, 174 insertions(+)arch/riscv/dts/jh7110-starfive-visionfive-2.dtsi +20
-0@@ -178,6 +178,87 @@ slew-rate = <0>; }; }; + + pcie0_perst_default: pcie0_perst_default { + perst-pins { + pinmux = <GPIOMUX(26, GPOUT_HIGH, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-disable; + input-schmitt-disable; + slew-rate = <0>; + }; + }; + + pcie0_perst_active: pcie0_perst_active { + perst-pins { + pinmux = <GPIOMUX(26, GPOUT_LOW, GPOEN_ENABLE, GPI_NONE)>; + drive-strength = <2>; + input-disable; + input-schmitt-disable; + slew-rate = <0>; + }; + };...
arch/riscv/dts/jh7110-starfive-visionfive-2.dtsi line 325
Quoted 8 lines
+&pcie0 {
arch/riscv/dts/jh7110-starfive-visionfive-2.dtsi +7
-0+ pinctrl-names = "perst-default", "perst-active", "wake-default", "clkreq-default"; + pinctrl-0 = <&pcie0_perst_default>; + pinctrl-1 = <&pcie0_perst_active>; + pinctrl-2 = <&pcie0_wake_default>; + pinctrl-3 = <&pcie0_clkreq_default>; + status = "disabled"; +};This is not correct declaration of PERST# signal. You should define in &pcie0 node "reset-gpios" property with plain gpio definition with active state.
Quoted 3 lines
"reset-gpios" is standard PCIe property for controlling PERST# signal over GPIO. Look for example into
drivers/pci/pci-aardvark.candarch/arm/dts/armada-3720-espressobin.dtsifiles how it is used.
OK, I will change this, thanks