Skip to contentSkip to conversation

PCI: brcmstb: enable PCIe for STB chips

PATCH Superseded Jim Quinlan 3 participants 24 messages Jun 16, 2020 – Jun 18, 2020

Subsystems (30)

v5 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v5 cover PCI: brcmstb: enable PCIe for STB chips

Patchset Summary:
Enhance a PCIe host controller driver. Because of its unusual design we are foced to change dev->dma_pfn_offset into a more general role allowing multiple offsets.

v5:
Commit "device core: Introduce multiple dma pfn offsets" -- in of/address.c: "map_size = 0" => "*map_size = 0" -- use kcalloc instead of kzalloc (AndyS)
-- use PHYS_ADDR_MAX instead of "~(phys_addr_t)0" Commit "PCI: brcmstb: Set internal memory viewport sizes" -- now gives error on missing dma-ranges property. Commit "dt-bindings: PCI: Add bindings for more Brcmstb chips" -- removed "Allof:" from brcm,scb-sizes definition (RobH) All Commits:
-- indentation style, use max chars 100 (AndyS)
-- rebased to torvalds master

v4:
Commit "device core: Introduce multiple dma pfn offsets" -- of_dma_get_range() does not take a dev param but instead
takes two "out" params: map and map_size. We do this so that the code that parses dma-ranges is separate from
the code that modifies 'dev'. (Nicolas)
-- the separate case of having a single pfn offset has
been removed and is now processed by going through the map array. (Nicolas)
-- move attach_uniform_dma_pfn_offset() from of/address.c to
dma/mapping.c so that it does not depend on CONFIG_OF. (Nicolas)
-- devm_kcalloc => devm_kzalloc (DanC)
-- add/fix assignment to dev->dma_pfn_offset_map for func
attach_uniform_dma_pfn_offset() (DanC, Nicolas)
-- s/struct dma_pfn_offset_region/struct bus_dma_region/ (Nicolas) -- s/attach_uniform_dma_pfn_offset/dma_attach_uniform_pfn_offset/ -- s/attach_dma_pfn_offset_map/dma_attach_pfn_offset_map/ -- More use of PFN_{PHYS,DOWN,UP}. (AndyS)
Commit "of: Include a dev param in of_dma_get_range()" -- this commit was sqaushed with "device core: Introduce ..."

v3:
Commit "device core: Introduce multiple dma pfn offsets" Commit "arm: dma-mapping: Invoke dma offset func if needed" -- The above two commits have been squashed. More importantly,
the code has been modified so that the functionality for multiple pfn offsets subsumes the use of dev->dma_pfn_offset. In fact, dma_pfn_offset is removed and supplanted by dma_pfn_offset_map, which is a pointer to an array. The more common case of a uniform offset is now handled as a map with a single entry, while cases requiring multiple pfn offsets use a map with multiple entries. Code paths that used to do this:

dev->dma_pfn_offset = mydrivers_pfn_offset;

have been changed to do this:

attach_uniform_dma_pfn_offset(dev, pfn_offset);

Commit "dt-bindings: PCI: Add bindings for more Brcmstb chips" -- Add if/then clause for required props: resets, reset-names (RobH) -- Change compatible list from const to enum (RobH)
-- Change list of u32-tuples to u64 (RobH)

Commit "of: Include a dev param in of_dma_get_range()"
-- modify of/unittests.c to add NULL param in of_dma_get_range() call.

Commit "device core: Add ability to handle multiple dma offsets" -- align comment in device.h (AndyS).
-- s/cpu_beg/cpu_start/ and s/dma_beg/dma_start/ in struct
dma_pfn_offset_region (AndyS).

v2:
Commit: "device core: Add ability to handle multiple dma offsets"
o Added helper func attach_dma_pfn_offset_map() in address.c (Chistoph)
o Helpers funcs added to __phys_to_dma() & __dma_to_phys() (Christoph)
o Added warning when multiple offsets are needed and !DMA_PFN_OFFSET_MAP
o dev->dma_pfn_map => dev->dma_pfn_offset_map
o s/frm/from/ for dma_pfn_offset_frm_{phys,dma}_addr() (Christoph)
o In device.h: s/const void */const struct dma_pfn_offset_region */
o removed 'unlikely' from unlikely(dev->dma_pfn_offset_map) since guarded by CONFIG_DMA_PFN_OFFSET_MAP (Christoph)
o Since dev->dma_pfn_offset is copied in usb/core/{usb,message}.c, now

    dev->dma_pfn_offset_map is copied as well.

o Merged two of the DMA commits into one (Christoph).

Commit "arm: dma-mapping: Invoke dma offset func if needed":
o Use helper functions instead of #if CONFIG_DMA_PFN_OFFSET

Other commits' changes:
o Removed need for carrying of_id var in priv (Nicolas)
o Commit message rewordings (Bjorn)
o Commit log messages filled to 75 chars (Bjorn)
o devm_reset_control_get_shared())
=> devm_reset_control_get_optional_shared (Philipp)
o Add call to reset_control_assert() in PCIe remove routines (Philipp)

v1:
This patchset expands the usefulness of the Broadcom Settop Box PCIe controller by building upon the PCIe driver used currently by the Raspbery Pi. Other forms of this patchset were submitted by me years ago and not accepted; the major sticking point was the code required for the DMA remapping needed for the PCIe driver to work [1].

There have been many changes to the DMA and OF subsystems since that time, making a cleaner and less intrusive patchset possible. This patchset implements a generalization of "dev->dma_pfn_offset", except that instead of a single scalar offset it provides for multiple offsets via a function which depends upon the "dma-ranges" property of the PCIe host controller. This is required for proper functionality of the BrcmSTB PCIe controller and possibly some other devices.

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Jim Quinlan (12):
PCI: brcmstb: PCIE_BRCMSTB depends on ARCH_BRCMSTB
ata: ahci_brcm: Fix use of BCM7216 reset controller
dt-bindings: PCI: Add bindings for more Brcmstb chips
PCI: brcmstb: Add bcm7278 register info
PCI: brcmstb: Add suspend and resume pm_ops
PCI: brcmstb: Add bcm7278 PERST support
PCI: brcmstb: Add control of rescal reset
device core: Introduce multiple dma pfn offsets
PCI: brcmstb: Set internal memory viewport sizes
PCI: brcmstb: Accommodate MSI for older chips
PCI: brcmstb: Set bus max burst size by chip type
PCI: brcmstb: Add bcm7211, bcm7216, bcm7445, bcm7278 to match list

 .../bindings/pci/brcm,stb-pcie.yaml           |  56 ++-
 arch/arm/include/asm/dma-mapping.h            |   9 +-
 arch/arm/mach-keystone/keystone.c             |   8 +-
 arch/sh/drivers/pci/pcie-sh7786.c             |   3 +-
 arch/sh/kernel/dma-coherent.c                 |  14 +-
 arch/x86/pci/sta2x11-fixup.c                  |   7 +-
 drivers/acpi/arm64/iort.c                     |   4 +-
 drivers/ata/ahci_brcm.c                       |  11 +-
 drivers/gpu/drm/sun4i/sun4i_backend.c         |   5 +-
 drivers/iommu/io-pgtable-arm.c                |   2 +-
 .../platform/sunxi/sun4i-csi/sun4i_csi.c      |   5 +-
 .../platform/sunxi/sun6i-csi/sun6i_csi.c      |   4 +-
 drivers/of/address.c                          |  71 ++-
 drivers/of/device.c                           |  19 +-
 drivers/of/of_private.h                       |  11 +-
 drivers/of/unittest.c                         |   8 +-
 drivers/pci/controller/Kconfig                |   3 +-
 drivers/pci/controller/pcie-brcmstb.c         | 403 +++++++++++++++---
 drivers/remoteproc/remoteproc_core.c          |   2 +-
 .../staging/media/sunxi/cedrus/cedrus_hw.c    |   7 +-
 drivers/usb/core/message.c                    |   4 +-
 drivers/usb/core/usb.c                        |   2 +-
 include/linux/device.h                        |   4 +-
 include/linux/dma-direct.h                    |  14 +-
 include/linux/dma-mapping.h                   |  38 ++
 kernel/dma/coherent.c                         |  11 +-
 kernel/dma/mapping.c                          |  39 ++
 27 files changed, 632 insertions(+), 132 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH v5 1/12 PCI: brcmstb: PCIE_BRCMSTB depends on ARCH_BRCMSTB

From: Jim Quinlan <[email protected]>

Have PCIE_BRCMSTB depend on ARCH_BRCMSTB. Also set the default value to ARCH_BRCMSTB.

---
 drivers/pci/controller/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/pci/controller/Kconfig +2 -1
index adddf21fa381..c165328d5fbb 100644
@@ -262,9 +262,10 @@ config VMD
 
 config PCIE_BRCMSTB
 	tristate "Broadcom Brcmstb PCIe host controller"
-	depends on ARCH_BCM2835 || COMPILE_TEST
+	depends on ARCH_BRCMSTB || ARCH_BCM2835 || COMPILE_TEST
 	depends on OF
 	depends on PCI_MSI_IRQ_DOMAIN
+	default ARCH_BRCMSTB
 	help
 	  Say Y here to enable PCIe host controller support for
 	  Broadcom STB based SoCs, like the Raspberry Pi 4.
Signature · 1 line
2.17.1
#2 lore raw

PATCH v5 2/12 ata: ahci_brcm: Fix use of BCM7216 reset controller

In reply to Jim Quinlan

From: Jim Quinlan <[email protected]>

A reset controller "rescal" is shared between the AHCI driver and the PCIe driver for the BrcmSTB 7216 chip. The code is modified to allow this sharing and to deassert() properly.

Fixes 272ecd60a636 ("ata: ahci_brcm: BCM7216 reset is self de-asserting")
Fixes c345ec6a50e9 ("ata: ahci_brcm: Support BCM7216 reset controller

name")

---
 drivers/ata/ahci_brcm.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)
drivers/ata/ahci_brcm.c +3 -8
index 6853dbb4131d..d6115bc04b09 100644
@@ -428,7 +428,6 @@ static int brcm_ahci_probe(struct platform_device *pdev)
 {
 	const struct of_device_id *of_id;
 	struct device *dev = &pdev->dev;
-	const char *reset_name = NULL;
 	struct brcm_ahci_priv *priv;
 	struct ahci_host_priv *hpriv;
 	struct resource *res;
@@ -452,11 +451,10 @@ static int brcm_ahci_probe(struct platform_device *pdev)
 
 	/* Reset is optional depending on platform and named differently */
 	if (priv->version == BRCM_SATA_BCM7216)
-		reset_name = "rescal";
+		priv->rcdev = devm_reset_control_get_optional_shared(&pdev->dev, "rescal");
 	else
-		reset_name = "ahci";
+		priv->rcdev = devm_reset_control_get_optional(&pdev->dev, "ahci");
 
-	priv->rcdev = devm_reset_control_get_optional(&pdev->dev, reset_name);
 	if (IS_ERR(priv->rcdev))
 		return PTR_ERR(priv->rcdev);
 
@@ -479,10 +477,7 @@ static int brcm_ahci_probe(struct platform_device *pdev)
 		break;
 	}
 
-	if (priv->version == BRCM_SATA_BCM7216)
-		ret = reset_control_reset(priv->rcdev);
-	else
-		ret = reset_control_deassert(priv->rcdev);
+	ret = reset_control_deassert(priv->rcdev);
 	if (ret)
 		return ret;
 
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH v5 02/12] ata: ahci_brcm: Fix use of BCM7216 reset controller

Quoted 5 lines · Jim Quinlan

From: Jim Quinlan <[email protected]>

A reset controller "rescal" is shared between the AHCI driver and the PCIe driver for the BrcmSTB 7216 chip. The code is modified to allow this sharing and to deassert() properly.

Use imperative mood, e.g.,

A reset controller "rescal" is shared between the AHCI driver and the PCIe driver for the BrcmSTB 7216 chip. Use devm_reset_control_get_optional_shared control() to handle this sharing.

Quoted 5 lines
Fixes 272ecd60a636 ("ata: ahci_brcm: BCM7216 reset is self de-asserting")
Fixes c345ec6a50e9 ("ata: ahci_brcm: Support BCM7216 reset controller

name")

Don't wrap "Fixes:" lines so it's easier to search for them.

#4 lore raw

PATCH v5 3/12 dt-bindings: PCI: Add bindings for more Brcmstb chips

In reply to Jim Quinlan

From: Jim Quinlan <[email protected]>

- Add compatible strings for three more Broadcom STB chips: 7278, 7216, 7211 (STB version of RPi4).
- add new property 'brcm,scb-sizes'
- add new property 'resets'
- add new property 'reset-names' for 7216 only
- allow 'ranges' and 'dma-ranges' to have more than one item and update the example to show this.

---
 .../bindings/pci/brcm,stb-pcie.yaml           | 56 ++++++++++++++++---
 1 file changed, 49 insertions(+), 7 deletions(-)
Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml +49 -7
index 8680a0f86c5a..807694b4f41f 100644
@@ -9,12 +9,15 @@ title: Brcmstb PCIe Host Controller Device Tree Bindings
 maintainers:
   - Nicolas Saenz Julienne <[email protected]>
 
-allOf:
-  - $ref: /schemas/pci/pci-bus.yaml#
-
 properties:
   compatible:
-    const: brcm,bcm2711-pcie # The Raspberry Pi 4
+    items:
+      - enum:
+          - brcm,bcm2711-pcie # The Raspberry Pi 4
+          - brcm,bcm7211-pcie # Broadcom STB version of RPi4
+          - brcm,bcm7278-pcie # Broadcom 7278 Arm
+          - brcm,bcm7216-pcie # Broadcom 7216 Arm
+          - brcm,bcm7445-pcie # Broadcom 7445 Arm
 
   reg:
     maxItems: 1
@@ -34,10 +37,12 @@ properties:
       - const: msi
 
   ranges:
-    maxItems: 1
+    minItems: 1
+    maxItems: 4
 
   dma-ranges:
-    maxItems: 1
+    minItems: 1
+    maxItems: 6
 
   clocks:
     maxItems: 1
@@ -58,8 +63,31 @@ properties:
 
   aspm-no-l0s: true
 
+  resets:
+    description: for "brcm,bcm7216-pcie", must be a valid reset
+      phandle pointing to the RESCAL reset controller provider node.
+    $ref: "/schemas/types.yaml#/definitions/phandle"
+
+  reset-names:
+    items:
+      - const: rescal
+
+  brcm,scb-sizes:
+    description: u64 giving the 64bit PCIe memory
+      viewport size of a memory controller.  There may be up to
+      three controllers, and each size must be a power of two
+      with a size greater or equal to the amount of memory the
+      controller supports.  Note that each memory controller
+      may have two component regions -- base and extended -- so
+      this information cannot be deduced from the dma-ranges.
+    $ref: /schemas/types.yaml#/definitions/uint64-array
+    items:
+      minItems: 1
+      maxItems: 3
+
 required:
   - reg
+  - ranges
   - dma-ranges
   - "#interrupt-cells"
   - interrupts
@@ -68,6 +96,18 @@ required:
   - interrupt-map
   - msi-controller
 
+allOf:
+  - $ref: /schemas/pci/pci-bus.yaml#
+  - if:
+      properties:
+        compatible:
+          contains:
+            const: brcm,bcm7216-pcie
+    then:
+      required:
+        - resets
+        - reset-names
+
 unevaluatedProperties: false
 
 examples:
@@ -93,7 +133,9 @@ examples:
                     msi-parent = <&pcie0>;
                     msi-controller;
                     ranges = <0x02000000 0x0 0xf8000000 0x6 0x00000000 0x0 0x04000000>;
-                    dma-ranges = <0x02000000 0x0 0x00000000 0x0 0x00000000 0x0 0x80000000>;
+                    dma-ranges = <0x42000000 0x1 0x00000000 0x0 0x40000000 0x0 0x80000000>,
+                                 <0x42000000 0x1 0x80000000 0x3 0x00000000 0x0 0x80000000>;
                     brcm,enable-ssc;
+                    brcm,scb-sizes =  <0x0000000080000000 0x0000000080000000>;
             };
     };
Signature · 7 lines
2.17.1


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#5 lore raw

Re: [PATCH v5 03/12] dt-bindings: PCI: Add bindings for more Brcmstb chips

Quoted 9 lines · Jim Quinlan

From: Jim Quinlan <[email protected]>

- Add compatible strings for three more Broadcom STB chips: 7278, 7216, 7211 (STB version of RPi4).
- add new property 'brcm,scb-sizes'
- add new property 'resets'
- add new property 'reset-names' for 7216 only
- allow 'ranges' and 'dma-ranges' to have more than one item and update the example to show this.

Capitalize and add periods to the list items consistently.

#6 lore raw

PATCH v5 4/12 PCI: brcmstb: Add bcm7278 register info

In reply to Jim Quinlan

From: Jim Quinlan <[email protected]>

Add in compatibility strings and code for three Broadcom STB chips. Some of the register locations, shifts, and masks are different for certain chips, requiring the use of different constants based on of_id.

We would like to add the following at this time to the match list but we need to wait until the end of this patchset so that everything works.

    { .compatible = "brcm,bcm7211-pcie", .data = &generic_cfg },
    { .compatible = "brcm,bcm7278-pcie", .data = &bcm7278_cfg },
    { .compatible = "brcm,bcm7216-pcie", .data = &bcm7278_cfg },
    { .compatible = "brcm,bcm7445-pcie", .data = &generic_cfg },
---
 drivers/pci/controller/pcie-brcmstb.c | 105 +++++++++++++++++++++++---
 1 file changed, 93 insertions(+), 12 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +93 -12
index 7730ea845ff2..b7a222fde3c4 100644
@@ -122,9 +122,8 @@
 #define  PCIE_EXT_SLOT_SHIFT				15
 #define  PCIE_EXT_FUNC_SHIFT				12
 
-#define PCIE_RGR1_SW_INIT_1				0x9210
 #define  PCIE_RGR1_SW_INIT_1_PERST_MASK			0x1
-#define  PCIE_RGR1_SW_INIT_1_INIT_MASK			0x2
+#define  PCIE_RGR1_SW_INIT_1_PERST_SHIFT		0x0
 
 /* PCIe parameters */
 #define BRCM_NUM_PCIE_OUT_WINS		0x4
@@ -154,6 +153,73 @@
 #define SSC_STATUS_SSC_MASK		0x400
 #define SSC_STATUS_PLL_LOCK_MASK	0x800
 
+#define IDX_ADDR(pcie)			(pcie->reg_offsets[EXT_CFG_INDEX])
+#define DATA_ADDR(pcie)			(pcie->reg_offsets[EXT_CFG_DATA])
+#define PCIE_RGR1_SW_INIT_1(pcie)	(pcie->reg_offsets[RGR1_SW_INIT_1])
+
+enum {
+	RGR1_SW_INIT_1,
+	EXT_CFG_INDEX,
+	EXT_CFG_DATA,
+};
+
+enum {
+	RGR1_SW_INIT_1_INIT_MASK,
+	RGR1_SW_INIT_1_INIT_SHIFT,
+};
+
+enum pcie_type {
+	GENERIC,
+	BCM7278,
+	BCM2711,
+};
+
+struct pcie_cfg_data {
+	const int *reg_field_info;
+	const int *offsets;
+	const enum pcie_type type;
+};
+
+static const int pcie_reg_field_info[] = {
+	[RGR1_SW_INIT_1_INIT_MASK] = 0x2,
+	[RGR1_SW_INIT_1_INIT_SHIFT] = 0x1,
+};
+
+static const int pcie_reg_field_info_bcm7278[] = {
+	[RGR1_SW_INIT_1_INIT_MASK] = 0x1,
+	[RGR1_SW_INIT_1_INIT_SHIFT] = 0x0,
+};
+
+static const int pcie_offsets[] = {
+	[RGR1_SW_INIT_1] = 0x9210,
+	[EXT_CFG_INDEX]  = 0x9000,
+	[EXT_CFG_DATA]   = 0x9004,
+};
+
+static const struct pcie_cfg_data generic_cfg = {
+	.reg_field_info	= pcie_reg_field_info,
+	.offsets	= pcie_offsets,
+	.type		= GENERIC,
+};
+
+static const int pcie_offset_bcm7278[] = {
+	[RGR1_SW_INIT_1] = 0xc010,
+	[EXT_CFG_INDEX] = 0x9000,
+	[EXT_CFG_DATA] = 0x9004,
+};
+
+static const struct pcie_cfg_data bcm7278_cfg = {
+	.reg_field_info = pcie_reg_field_info_bcm7278,
+	.offsets	= pcie_offset_bcm7278,
+	.type		= BCM7278,
+};
+
+static const struct pcie_cfg_data bcm2711_cfg = {
+	.reg_field_info	= pcie_reg_field_info,
+	.offsets	= pcie_offsets,
+	.type		= BCM2711,
+};
+
 struct brcm_msi {
 	struct device		*dev;
 	void __iomem		*base;
@@ -178,6 +244,9 @@ struct brcm_pcie {
 	int			gen;
 	u64			msi_target_addr;
 	struct brcm_msi		*msi;
+	const int		*reg_offsets;
+	const int		*reg_field_info;
+	enum pcie_type		type;
 };
 
 /*
@@ -604,20 +673,21 @@ static struct pci_ops brcm_pcie_ops = {
 
 static inline void brcm_pcie_bridge_sw_init_set(struct brcm_pcie *pcie, u32 val)
 {
-	u32 tmp;
+	u32 tmp, mask =  pcie->reg_field_info[RGR1_SW_INIT_1_INIT_MASK];
+	u32 shift = pcie->reg_field_info[RGR1_SW_INIT_1_INIT_SHIFT];
 
-	tmp = readl(pcie->base + PCIE_RGR1_SW_INIT_1);
-	u32p_replace_bits(&tmp, val, PCIE_RGR1_SW_INIT_1_INIT_MASK);
-	writel(tmp, pcie->base + PCIE_RGR1_SW_INIT_1);
+	tmp = readl(pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
+	tmp = (tmp & ~mask) | ((val << shift) & mask);
+	writel(tmp, pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
 }
 
 static inline void brcm_pcie_perst_set(struct brcm_pcie *pcie, u32 val)
 {
 	u32 tmp;
 
-	tmp = readl(pcie->base + PCIE_RGR1_SW_INIT_1);
+	tmp = readl(pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
 	u32p_replace_bits(&tmp, val, PCIE_RGR1_SW_INIT_1_PERST_MASK);
-	writel(tmp, pcie->base + PCIE_RGR1_SW_INIT_1);
+	writel(tmp, pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
 }
 
 static inline int brcm_pcie_get_rc_bar2_size_and_offset(struct brcm_pcie *pcie,
@@ -927,11 +997,17 @@ static int brcm_pcie_remove(struct platform_device *pdev)
 	return 0;
 }
 
+static const struct of_device_id brcm_pcie_match[] = {
+	{ .compatible = "brcm,bcm2711-pcie", .data = &bcm2711_cfg },
+	{},
+};
+
 static int brcm_pcie_probe(struct platform_device *pdev)
 {
 	struct device_node *np = pdev->dev.of_node, *msi_np;
 	struct pci_host_bridge *bridge;
 	struct device_node *fw_np;
+	const struct pcie_cfg_data *data;
 	struct brcm_pcie *pcie;
 	struct pci_bus *child;
 	struct resource *res;
@@ -955,9 +1031,18 @@ static int brcm_pcie_probe(struct platform_device *pdev)
 	if (!bridge)
 		return -ENOMEM;
 
+	data = of_device_get_match_data(&pdev->dev);
+	if (!data) {
+		pr_err("failed to look up compatible string\n");
+		return -EINVAL;
+	}
+
 	pcie = pci_host_bridge_priv(bridge);
 	pcie->dev = &pdev->dev;
 	pcie->np = np;
+	pcie->reg_offsets = data->offsets;
+	pcie->reg_field_info = data->reg_field_info;
+	pcie->type = data->type;
 
 	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
 	pcie->base = devm_ioremap_resource(&pdev->dev, res);
@@ -1023,10 +1108,6 @@ static int brcm_pcie_probe(struct platform_device *pdev)
 	return ret;
 }
 
-static const struct of_device_id brcm_pcie_match[] = {
-	{ .compatible = "brcm,bcm2711-pcie" },
-	{},
-};
 MODULE_DEVICE_TABLE(of, brcm_pcie_match);
 
 static struct platform_driver brcm_pcie_driver = {
Signature · 1 line
2.17.1
#7 lore raw

PATCH v5 5/12 PCI: brcmstb: Add suspend and resume pm_ops

In reply to Jim Quinlan

From: Jim Quinlan <[email protected]>

Broadcom Set-top (BrcmSTB) boards typically support S2, S3, and S5 suspend and resume. Now the PCIe driver may do so as well.

---
 drivers/pci/controller/pcie-brcmstb.c | 47 +++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
drivers/pci/controller/pcie-brcmstb.c +47 -0
index b7a222fde3c4..7c148eb65170 100644
@@ -979,6 +979,47 @@ static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
 	brcm_pcie_bridge_sw_init_set(pcie, 1);
 }
 
+static int brcm_pcie_suspend(struct device *dev)
+{
+	struct brcm_pcie *pcie = dev_get_drvdata(dev);
+
+	brcm_pcie_turn_off(pcie);
+	clk_disable_unprepare(pcie->clk);
+
+	return 0;
+}
+
+static int brcm_pcie_resume(struct device *dev)
+{
+	struct brcm_pcie *pcie = dev_get_drvdata(dev);
+	void __iomem *base;
+	u32 tmp;
+	int ret;
+
+	base = pcie->base;
+	clk_prepare_enable(pcie->clk);
+
+	/* Take bridge out of reset so we can access the SERDES reg */
+	brcm_pcie_bridge_sw_init_set(pcie, 0);
+
+	/* SERDES_IDDQ = 0 */
+	tmp = readl(base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
+	u32p_replace_bits(&tmp, 0, PCIE_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK);
+	writel(tmp, base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
+
+	/* wait for serdes to be stable */
+	udelay(100);
+
+	ret = brcm_pcie_setup(pcie);
+	if (ret)
+		return ret;
+
+	if (pcie->msi)
+		brcm_msi_set_regs(pcie->msi);
+
+	return 0;
+}
+
 static void __brcm_pcie_remove(struct brcm_pcie *pcie)
 {
 	brcm_msi_remove(pcie);
@@ -1110,12 +1151,18 @@ static int brcm_pcie_probe(struct platform_device *pdev)
 
 MODULE_DEVICE_TABLE(of, brcm_pcie_match);
 
+static const struct dev_pm_ops brcm_pcie_pm_ops = {
+	.suspend_noirq = brcm_pcie_suspend,
+	.resume_noirq = brcm_pcie_resume,
+};
+
 static struct platform_driver brcm_pcie_driver = {
 	.probe = brcm_pcie_probe,
 	.remove = brcm_pcie_remove,
 	.driver = {
 		.name = "brcm-pcie",
 		.of_match_table = brcm_pcie_match,
+		.pm = &brcm_pcie_pm_ops,
 	},
 };
 module_platform_driver(brcm_pcie_driver);
Signature · 1 line
2.17.1
#8 lore raw

PATCH v5 6/12 PCI: brcmstb: Add bcm7278 PERST support

In reply to Jim Quinlan

From: Jim Quinlan <[email protected]>

The PERST bit was moved to a different register in 7278-type STB chips. In addition, the polarity of the bit was also changed; for other chips writing a 1 specified assert; for 7278-type chips, writing a 0 specifies assert.

Signal-wise, PERST is an asserted-low signal.

---
 drivers/pci/controller/pcie-brcmstb.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +11 -3
index 7c148eb65170..d0e256d8578a 100644
@@ -83,6 +83,7 @@
 
 #define PCIE_MISC_PCIE_CTRL				0x4064
 #define  PCIE_MISC_PCIE_CTRL_PCIE_L23_REQUEST_MASK	0x1
+#define PCIE_MISC_PCIE_CTRL_PCIE_PERSTB_MASK		0x4
 
 #define PCIE_MISC_PCIE_STATUS				0x4068
 #define  PCIE_MISC_PCIE_STATUS_PCIE_PORT_MASK		0x80
@@ -685,9 +686,16 @@ static inline void brcm_pcie_perst_set(struct brcm_pcie *pcie, u32 val)
 {
 	u32 tmp;
 
-	tmp = readl(pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
-	u32p_replace_bits(&tmp, val, PCIE_RGR1_SW_INIT_1_PERST_MASK);
-	writel(tmp, pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
+	if (pcie->type == BCM7278) {
+		/* Perst bit has moved and assert value is 0 */
+		tmp = readl(pcie->base + PCIE_MISC_PCIE_CTRL);
+		u32p_replace_bits(&tmp, !val, PCIE_MISC_PCIE_CTRL_PCIE_PERSTB_MASK);
+		writel(tmp, pcie->base +  PCIE_MISC_PCIE_CTRL);
+	} else {
+		tmp = readl(pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
+		u32p_replace_bits(&tmp, val, PCIE_RGR1_SW_INIT_1_PERST_MASK);
+		writel(tmp, pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
+	}
 }
 
 static inline int brcm_pcie_get_rc_bar2_size_and_offset(struct brcm_pcie *pcie,
Signature · 1 line
2.17.1
#9 lore raw

Re: [PATCH v5 06/12] PCI: brcmstb: Add bcm7278 PERST support

Quoted 7 lines · Jim Quinlan

From: Jim Quinlan <[email protected]>

The PERST bit was moved to a different register in 7278-type STB chips. In addition, the polarity of the bit was also changed; for other chips writing a 1 specified assert; for 7278-type chips, writing a 0 specifies assert.

Signal-wise, PERST is an asserted-low signal.

s/PERST/PERST#/ to match usage of the signal name in spec.

The PERST bit above is the name of a register bit, so use whatever matches the STB spec.

Quoted 19 lines
---
 drivers/pci/controller/pcie-brcmstb.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +1 -0
index 7c148eb65170..d0e256d8578a 100644
@@ -83,6 +83,7 @@
 
 #define PCIE_MISC_PCIE_CTRL				0x4064
 #define  PCIE_MISC_PCIE_CTRL_PCIE_L23_REQUEST_MASK	0x1
+#define PCIE_MISC_PCIE_CTRL_PCIE_PERSTB_MASK		0x4
 
 #define PCIE_MISC_PCIE_STATUS				0x4068
 #define  PCIE_MISC_PCIE_STATUS_PCIE_PORT_MASK		0x80

drivers/pci/controller/pcie-brcmstb.c static inline void brcm_pcie_perst_set(struct brcm_pcie *pcie, u32 val) · line 690

Quoted 8 lines
drivers/pci/controller/pcie-brcmstb.c +2 -3
@@ -685,9 +686,16 @@ static inline void brcm_pcie_perst_set(struct brcm_pcie *pcie, u32 val)
 {
 	u32 tmp;
 
-	tmp = readl(pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
-	u32p_replace_bits(&tmp, val, PCIE_RGR1_SW_INIT_1_PERST_MASK);
-	writel(tmp, pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
+	if (pcie->type == BCM7278) {
+		/* Perst bit has moved and assert value is 0 */

s/Perst/PERST/ or PERST# so it doesn't look like an English word and to match the STB spec.

Quoted 14 lines
drivers/pci/controller/pcie-brcmstb.c +8 -0
+		tmp = readl(pcie->base + PCIE_MISC_PCIE_CTRL);
+		u32p_replace_bits(&tmp, !val, PCIE_MISC_PCIE_CTRL_PCIE_PERSTB_MASK);
+		writel(tmp, pcie->base +  PCIE_MISC_PCIE_CTRL);
+	} else {
+		tmp = readl(pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
+		u32p_replace_bits(&tmp, val, PCIE_RGR1_SW_INIT_1_PERST_MASK);
+		writel(tmp, pcie->base + PCIE_RGR1_SW_INIT_1(pcie));
+	}
 }
 
 static inline int brcm_pcie_get_rc_bar2_size_and_offset(struct brcm_pcie *pcie,
Signature · 1 line
2.17.1
#10 lore raw

PATCH v5 7/12 PCI: brcmstb: Add control of rescal reset

In reply to Jim Quinlan

From: Jim Quinlan <[email protected]>

Some STB chips have a special purpose reset controller named RESCAL (reset calibration). The PCIe HW can now control RESCAL to start and stop its operation.

---
 drivers/pci/controller/pcie-brcmstb.c | 81 ++++++++++++++++++++++++++-
 1 file changed, 80 insertions(+), 1 deletion(-)
drivers/pci/controller/pcie-brcmstb.c +80 -1
index d0e256d8578a..9189406fd35c 100644
@@ -23,6 +23,7 @@
 #include <linux/of_platform.h>
 #include <linux/pci.h>
 #include <linux/printk.h>
+#include <linux/reset.h>
 #include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/string.h>
@@ -158,6 +159,16 @@
 #define DATA_ADDR(pcie)			(pcie->reg_offsets[EXT_CFG_DATA])
 #define PCIE_RGR1_SW_INIT_1(pcie)	(pcie->reg_offsets[RGR1_SW_INIT_1])
 
+/* Rescal registers */
+#define PCIE_DVT_PMU_PCIE_PHY_CTRL				0xc700
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS			0x3
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_MASK		0x4
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_SHIFT	0x2
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_MASK		0x2
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_SHIFT		0x1
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_MASK		0x1
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_SHIFT		0x0
+
 enum {
 	RGR1_SW_INIT_1,
 	EXT_CFG_INDEX,
@@ -248,6 +259,7 @@ struct brcm_pcie {
 	const int		*reg_offsets;
 	const int		*reg_field_info;
 	enum pcie_type		type;
+	struct reset_control	*rescal;
 };
 
 /*
@@ -963,6 +975,47 @@ static void brcm_pcie_enter_l23(struct brcm_pcie *pcie)
 		dev_err(pcie->dev, "failed to enter low-power link state\n");
 }
 
+static int brcm_phy_cntl(struct brcm_pcie *pcie, const int start)
+{
+	static const u32 shifts[PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS] = {
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_SHIFT,
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_SHIFT,
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_SHIFT,};
+	static const u32 masks[PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS] = {
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_MASK,
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_MASK,
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_MASK,};
+	const int beg = start ? 0 : PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS - 1;
+	const int end = start ? PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS : -1;
+	u32 tmp, combined_mask = 0;
+	u32 val = !!start;
+	void __iomem *base = pcie->base;
+	int i;
+
+	for (i = beg; i != end; start ? i++ : i--) {
+		tmp = readl(base + PCIE_DVT_PMU_PCIE_PHY_CTRL);
+		tmp = (tmp & ~masks[i]) | ((val << shifts[i]) & masks[i]);
+		writel(tmp, base + PCIE_DVT_PMU_PCIE_PHY_CTRL);
+		usleep_range(50, 200);
+		combined_mask |= masks[i];
+	}
+
+	tmp = readl(base + PCIE_DVT_PMU_PCIE_PHY_CTRL);
+	val = start ? combined_mask : 0;
+
+	return (tmp & combined_mask) == val ? 0 : -EIO;
+}
+
+static inline int brcm_phy_start(struct brcm_pcie *pcie)
+{
+	return pcie->rescal ? brcm_phy_cntl(pcie, 1) : 0;
+}
+
+static inline int brcm_phy_stop(struct brcm_pcie *pcie)
+{
+	return pcie->rescal ? brcm_phy_cntl(pcie, 0) : 0;
+}
+
 static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
 {
 	void __iomem *base = pcie->base;
@@ -990,11 +1043,15 @@ static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
 static int brcm_pcie_suspend(struct device *dev)
 {
 	struct brcm_pcie *pcie = dev_get_drvdata(dev);
+	int ret;
 
 	brcm_pcie_turn_off(pcie);
+	ret = brcm_phy_stop(pcie);
+	if (ret)
+		dev_err(pcie->dev, "failed to stop phy\n");
 	clk_disable_unprepare(pcie->clk);
 
-	return 0;
+	return ret;
 }
 
 static int brcm_pcie_resume(struct device *dev)
@@ -1007,6 +1064,12 @@ static int brcm_pcie_resume(struct device *dev)
 	base = pcie->base;
 	clk_prepare_enable(pcie->clk);
 
+	ret = brcm_phy_start(pcie);
+	if (ret) {
+		dev_err(pcie->dev, "failed to start phy\n");
+		return ret;
+	}
+
 	/* Take bridge out of reset so we can access the SERDES reg */
 	brcm_pcie_bridge_sw_init_set(pcie, 0);
 
@@ -1032,6 +1095,9 @@ static void __brcm_pcie_remove(struct brcm_pcie *pcie)
 {
 	brcm_msi_remove(pcie);
 	brcm_pcie_turn_off(pcie);
+	if (brcm_phy_stop(pcie))
+		dev_err(pcie->dev, "failed to stop phy\n");
+	reset_control_assert(pcie->rescal);
 	clk_disable_unprepare(pcie->clk);
 }
 
@@ -1117,6 +1183,19 @@ static int brcm_pcie_probe(struct platform_device *pdev)
 		dev_err(&pdev->dev, "could not enable clock\n");
 		return ret;
 	}
+	pcie->rescal = devm_reset_control_get_optional_shared(&pdev->dev, "rescal");
+	if (IS_ERR(pcie->rescal))
+		return PTR_ERR(pcie->rescal);
+
+	ret = reset_control_deassert(pcie->rescal);
+	if (ret)
+		dev_err(&pdev->dev, "failed to deassert 'rescal'\n");
+
+	ret = brcm_phy_start(pcie);
+	if (ret) {
+		dev_err(pcie->dev, "failed to start phy\n");
+		return ret;
+	}
 
 	ret = brcm_pcie_setup(pcie);
 	if (ret)
Signature · 1 line
2.17.1
#11 lore raw

Re: [PATCH v5 07/12] PCI: brcmstb: Add control of rescal reset

Quoted 5 lines · Jim Quinlan

From: Jim Quinlan <[email protected]>

Some STB chips have a special purpose reset controller named RESCAL (reset calibration). The PCIe HW can now control RESCAL to start and stop its operation.

The HW *can* now control RESCAL, but what does this patch do?

I guess maybe this patch uses RESCAL to turn on the PHY in probe and resume and turn it off in suspend and remove?

Quoted 154 lines
---
 drivers/pci/controller/pcie-brcmstb.c | 81 ++++++++++++++++++++++++++-
 1 file changed, 80 insertions(+), 1 deletion(-)
drivers/pci/controller/pcie-brcmstb.c +80 -1
index d0e256d8578a..9189406fd35c 100644
@@ -23,6 +23,7 @@
 #include <linux/of_platform.h>
 #include <linux/pci.h>
 #include <linux/printk.h>
+#include <linux/reset.h>
 #include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/string.h>
@@ -158,6 +159,16 @@
 #define DATA_ADDR(pcie)			(pcie->reg_offsets[EXT_CFG_DATA])
 #define PCIE_RGR1_SW_INIT_1(pcie)	(pcie->reg_offsets[RGR1_SW_INIT_1])
 
+/* Rescal registers */
+#define PCIE_DVT_PMU_PCIE_PHY_CTRL				0xc700
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS			0x3
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_MASK		0x4
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_SHIFT	0x2
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_MASK		0x2
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_SHIFT		0x1
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_MASK		0x1
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_SHIFT		0x0
+
 enum {
 	RGR1_SW_INIT_1,
 	EXT_CFG_INDEX,
@@ -248,6 +259,7 @@ struct brcm_pcie {
 	const int		*reg_offsets;
 	const int		*reg_field_info;
 	enum pcie_type		type;
+	struct reset_control	*rescal;
 };
 
 /*
@@ -963,6 +975,47 @@ static void brcm_pcie_enter_l23(struct brcm_pcie *pcie)
 		dev_err(pcie->dev, "failed to enter low-power link state\n");
 }
 
+static int brcm_phy_cntl(struct brcm_pcie *pcie, const int start)
+{
+	static const u32 shifts[PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS] = {
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_SHIFT,
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_SHIFT,
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_SHIFT,};
+	static const u32 masks[PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS] = {
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_MASK,
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_MASK,
+		PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_MASK,};
+	const int beg = start ? 0 : PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS - 1;
+	const int end = start ? PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS : -1;
+	u32 tmp, combined_mask = 0;
+	u32 val = !!start;
+	void __iomem *base = pcie->base;
+	int i;
+
+	for (i = beg; i != end; start ? i++ : i--) {
+		tmp = readl(base + PCIE_DVT_PMU_PCIE_PHY_CTRL);
+		tmp = (tmp & ~masks[i]) | ((val << shifts[i]) & masks[i]);
+		writel(tmp, base + PCIE_DVT_PMU_PCIE_PHY_CTRL);
+		usleep_range(50, 200);
+		combined_mask |= masks[i];
+	}
+
+	tmp = readl(base + PCIE_DVT_PMU_PCIE_PHY_CTRL);
+	val = start ? combined_mask : 0;
+
+	return (tmp & combined_mask) == val ? 0 : -EIO;
+}
+
+static inline int brcm_phy_start(struct brcm_pcie *pcie)
+{
+	return pcie->rescal ? brcm_phy_cntl(pcie, 1) : 0;
+}
+
+static inline int brcm_phy_stop(struct brcm_pcie *pcie)
+{
+	return pcie->rescal ? brcm_phy_cntl(pcie, 0) : 0;
+}
+
 static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
 {
 	void __iomem *base = pcie->base;
@@ -990,11 +1043,15 @@ static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
 static int brcm_pcie_suspend(struct device *dev)
 {
 	struct brcm_pcie *pcie = dev_get_drvdata(dev);
+	int ret;
 
 	brcm_pcie_turn_off(pcie);
+	ret = brcm_phy_stop(pcie);
+	if (ret)
+		dev_err(pcie->dev, "failed to stop phy\n");
 	clk_disable_unprepare(pcie->clk);
 
-	return 0;
+	return ret;
 }
 
 static int brcm_pcie_resume(struct device *dev)
@@ -1007,6 +1064,12 @@ static int brcm_pcie_resume(struct device *dev)
 	base = pcie->base;
 	clk_prepare_enable(pcie->clk);
 
+	ret = brcm_phy_start(pcie);
+	if (ret) {
+		dev_err(pcie->dev, "failed to start phy\n");
+		return ret;
+	}
+
 	/* Take bridge out of reset so we can access the SERDES reg */
 	brcm_pcie_bridge_sw_init_set(pcie, 0);
 
@@ -1032,6 +1095,9 @@ static void __brcm_pcie_remove(struct brcm_pcie *pcie)
 {
 	brcm_msi_remove(pcie);
 	brcm_pcie_turn_off(pcie);
+	if (brcm_phy_stop(pcie))
+		dev_err(pcie->dev, "failed to stop phy\n");
+	reset_control_assert(pcie->rescal);
 	clk_disable_unprepare(pcie->clk);
 }
 
@@ -1117,6 +1183,19 @@ static int brcm_pcie_probe(struct platform_device *pdev)
 		dev_err(&pdev->dev, "could not enable clock\n");
 		return ret;
 	}
+	pcie->rescal = devm_reset_control_get_optional_shared(&pdev->dev, "rescal");
+	if (IS_ERR(pcie->rescal))
+		return PTR_ERR(pcie->rescal);
+
+	ret = reset_control_deassert(pcie->rescal);
+	if (ret)
+		dev_err(&pdev->dev, "failed to deassert 'rescal'\n");
+
+	ret = brcm_phy_start(pcie);
+	if (ret) {
+		dev_err(pcie->dev, "failed to start phy\n");
+		return ret;
+	}
 
 	ret = brcm_pcie_setup(pcie);
 	if (ret)
Signature · 1 line
2.17.1
#12 lore raw

Re: [PATCH v5 07/12] PCI: brcmstb: Add control of rescal reset

Quoted 4 lines · Bjorn Helgaas
Quoted 2 lines · Jim Quinlan

From: Jim Quinlan <[email protected]>

Quoted 8 lines
Quoted 3 lines

Some STB chips have a special purpose reset controller named RESCAL (reset calibration). The PCIe HW can now control RESCAL to start and stop its operation.

The HW *can* now control RESCAL, but what does this patch do?

I guess maybe this patch uses RESCAL to turn on the PHY in probe and resume and turn it off in suspend and remove?

Yes, I will redo this text with a better description.

Thanks,
Jim

PS Will attend to your other responses as well in the V6.

Quoted 150 lines
Quoted 150 lines
---
 drivers/pci/controller/pcie-brcmstb.c | 81 ++++++++++++++++++++++++++-
 1 file changed, 80 insertions(+), 1 deletion(-)
drivers/pci/controller/pcie-brcmstb.c +80 -1
index d0e256d8578a..9189406fd35c 100644
@@ -23,6 +23,7 @@
 #include <linux/of_platform.h>
 #include <linux/pci.h>
 #include <linux/printk.h>
+#include <linux/reset.h>
 #include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/string.h>
@@ -158,6 +159,16 @@
 #define DATA_ADDR(pcie)                      (pcie->reg_offsets[EXT_CFG_DATA])
 #define PCIE_RGR1_SW_INIT_1(pcie)    (pcie->reg_offsets[RGR1_SW_INIT_1])
 
+/* Rescal registers */
+#define PCIE_DVT_PMU_PCIE_PHY_CTRL                           0xc700
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS                       0x3
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_MASK              0x4
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_SHIFT     0x2
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_MASK          0x2
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_SHIFT         0x1
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_MASK          0x1
+#define  PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_SHIFT         0x0
+
 enum {
      RGR1_SW_INIT_1,
      EXT_CFG_INDEX,
@@ -248,6 +259,7 @@ struct brcm_pcie {
      const int               *reg_offsets;
      const int               *reg_field_info;
      enum pcie_type          type;
+     struct reset_control    *rescal;
 };
 
 /*
@@ -963,6 +975,47 @@ static void brcm_pcie_enter_l23(struct brcm_pcie *pcie)
              dev_err(pcie->dev, "failed to enter low-power link state\n");
 }
 
+static int brcm_phy_cntl(struct brcm_pcie *pcie, const int start)
+{
+     static const u32 shifts[PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS] = {
+             PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_SHIFT,
+             PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_SHIFT,
+             PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_SHIFT,};
+     static const u32 masks[PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS] = {
+             PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_PWRDN_MASK,
+             PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_RESET_MASK,
+             PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_DIG_RESET_MASK,};
+     const int beg = start ? 0 : PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS - 1;
+     const int end = start ? PCIE_DVT_PMU_PCIE_PHY_CTRL_DAST_NFLDS : -1;
+     u32 tmp, combined_mask = 0;
+     u32 val = !!start;
+     void __iomem *base = pcie->base;
+     int i;
+
+     for (i = beg; i != end; start ? i++ : i--) {
+             tmp = readl(base + PCIE_DVT_PMU_PCIE_PHY_CTRL);
+             tmp = (tmp & ~masks[i]) | ((val << shifts[i]) & masks[i]);
+             writel(tmp, base + PCIE_DVT_PMU_PCIE_PHY_CTRL);
+             usleep_range(50, 200);
+             combined_mask |= masks[i];
+     }
+
+     tmp = readl(base + PCIE_DVT_PMU_PCIE_PHY_CTRL);
+     val = start ? combined_mask : 0;
+
+     return (tmp & combined_mask) == val ? 0 : -EIO;
+}
+
+static inline int brcm_phy_start(struct brcm_pcie *pcie)
+{
+     return pcie->rescal ? brcm_phy_cntl(pcie, 1) : 0;
+}
+
+static inline int brcm_phy_stop(struct brcm_pcie *pcie)
+{
+     return pcie->rescal ? brcm_phy_cntl(pcie, 0) : 0;
+}
+
 static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
 {
      void __iomem *base = pcie->base;
@@ -990,11 +1043,15 @@ static void brcm_pcie_turn_off(struct brcm_pcie *pcie)
 static int brcm_pcie_suspend(struct device *dev)
 {
      struct brcm_pcie *pcie = dev_get_drvdata(dev);
+     int ret;
 
      brcm_pcie_turn_off(pcie);
+     ret = brcm_phy_stop(pcie);
+     if (ret)
+             dev_err(pcie->dev, "failed to stop phy\n");
      clk_disable_unprepare(pcie->clk);
 
-     return 0;
+     return ret;
 }
 
 static int brcm_pcie_resume(struct device *dev)
@@ -1007,6 +1064,12 @@ static int brcm_pcie_resume(struct device *dev)
      base = pcie->base;
      clk_prepare_enable(pcie->clk);
 
+     ret = brcm_phy_start(pcie);
+     if (ret) {
+             dev_err(pcie->dev, "failed to start phy\n");
+             return ret;
+     }
+
      /* Take bridge out of reset so we can access the SERDES reg */
      brcm_pcie_bridge_sw_init_set(pcie, 0);
 
@@ -1032,6 +1095,9 @@ static void __brcm_pcie_remove(struct brcm_pcie *pcie)
 {
      brcm_msi_remove(pcie);
      brcm_pcie_turn_off(pcie);
+     if (brcm_phy_stop(pcie))
+             dev_err(pcie->dev, "failed to stop phy\n");
+     reset_control_assert(pcie->rescal);
      clk_disable_unprepare(pcie->clk);
 }
 
@@ -1117,6 +1183,19 @@ static int brcm_pcie_probe(struct platform_device *pdev)
              dev_err(&pdev->dev, "could not enable clock\n");
              return ret;
      }
+     pcie->rescal = devm_reset_control_get_optional_shared(&pdev->dev, "rescal");
+     if (IS_ERR(pcie->rescal))
+             return PTR_ERR(pcie->rescal);
+
+     ret = reset_control_deassert(pcie->rescal);
+     if (ret)
+             dev_err(&pdev->dev, "failed to deassert 'rescal'\n");
+
+     ret = brcm_phy_start(pcie);
+     if (ret) {
+             dev_err(pcie->dev, "failed to start phy\n");
+             return ret;
+     }
 
Quoted 5 lines
Quoted 5 lines
drivers/pci/controller/pcie-brcmstb.c +0 -1
      ret = brcm_pcie_setup(pcie);
      if (ret)
--

2.17.1

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#13 lore raw

PATCH v5 8/12 device core: Introduce multiple dma pfn offsets

In reply to Jim Quinlan

The new field in struct device 'dma_pfn_offset_map' is used to facilitate the use of single or multiple pfn offsets between cpu addrs and dma addrs. It subsumes the role of dev->dma_pfn_offset -- a uniform offset.

The function of_dma_get_range() has been modified to take two additional
arguments: the "map", which is an array that holds the information regarding the pfn offset regions, and map_size, which is the size in bytes of the map array.

of_dma_configure() is the typical manner to set pfn offsets but there are a number of ad hoc assignments to dev->dma_pfn_offset in the kernel driver code. These cases now invoke the function dma_attach_uniform_pfn_offset(dev, pfn_offset).

---
 arch/arm/include/asm/dma-mapping.h            |  9 +--
 arch/arm/mach-keystone/keystone.c             |  8 ++-
 arch/sh/drivers/pci/pcie-sh7786.c             |  3 +-
 arch/sh/kernel/dma-coherent.c                 | 14 ++--
 arch/x86/pci/sta2x11-fixup.c                  |  7 +-
 drivers/acpi/arm64/iort.c                     |  4 +-
 drivers/gpu/drm/sun4i/sun4i_backend.c         |  5 +-
 drivers/iommu/io-pgtable-arm.c                |  2 +-
 .../platform/sunxi/sun4i-csi/sun4i_csi.c      |  5 +-
 .../platform/sunxi/sun6i-csi/sun6i_csi.c      |  4 +-
 drivers/of/address.c                          | 71 ++++++++++++++++---
 drivers/of/device.c                           | 19 +++--
 drivers/of/of_private.h                       | 11 +--
 drivers/of/unittest.c                         |  8 ++-
 drivers/remoteproc/remoteproc_core.c          |  2 +-
 .../staging/media/sunxi/cedrus/cedrus_hw.c    |  7 +-
 drivers/usb/core/message.c                    |  4 +-
 drivers/usb/core/usb.c                        |  2 +-
 include/linux/device.h                        |  4 +-
 include/linux/dma-direct.h                    | 14 +++-
 include/linux/dma-mapping.h                   | 38 ++++++++++
 kernel/dma/coherent.c                         | 11 +--
 kernel/dma/mapping.c                          | 39 ++++++++++
 23 files changed, 231 insertions(+), 60 deletions(-)
arch/arm/include/asm/dma-mapping.h +5 -4
index bdd80ddbca34..f1e72f99468b 100644
@@ -35,8 +35,9 @@ static inline const struct dma_map_ops *get_arch_dma_ops(struct bus_type *bus)
 #ifndef __arch_pfn_to_dma
 static inline dma_addr_t pfn_to_dma(struct device *dev, unsigned long pfn)
 {
-	if (dev)
-		pfn -= dev->dma_pfn_offset;
+	if (dev && dev->dma_pfn_offset_map)
+		pfn -= dma_pfn_offset_from_phys_addr(dev, PFN_PHYS(pfn));
+
 	return (dma_addr_t)__pfn_to_bus(pfn);
 }
 
@@ -44,8 +45,8 @@ static inline unsigned long dma_to_pfn(struct device *dev, dma_addr_t addr)
 {
 	unsigned long pfn = __bus_to_pfn(addr);
 
-	if (dev)
-		pfn += dev->dma_pfn_offset;
+	if (dev && dev->dma_pfn_offset_map)
+		pfn += dma_pfn_offset_from_dma_addr(dev, addr);
 
 	return pfn;
 }
arch/arm/mach-keystone/keystone.c +5 -3
index 638808c4e122..5890ec90599e 100644
@@ -8,6 +8,7 @@
  */
 #include <linux/io.h>
 #include <linux/of.h>
+#include <linux/dma-mapping.h>
 #include <linux/init.h>
 #include <linux/of_platform.h>
 #include <linux/of_address.h>
@@ -38,9 +39,10 @@ static int keystone_platform_notifier(struct notifier_block *nb,
 		return NOTIFY_BAD;
 
 	if (!dev->of_node) {
-		dev->dma_pfn_offset = keystone_dma_pfn_offset;
-		dev_err(dev, "set dma_pfn_offset%08lx\n",
-			dev->dma_pfn_offset);
+		int ret = dma_attach_uniform_pfn_offset(dev, keystone_dma_pfn_offset);
+
+		dev_err(dev, "set dma_pfn_offset%08lx%s\n", dev->dma_pfn_offset,
+			ret ? " failed" : "");
 	}
 	return NOTIFY_OK;
 }
arch/sh/drivers/pci/pcie-sh7786.c +2 -1
index e0b568aaa701..3e63c6b6e070 100644
@@ -12,6 +12,7 @@
 #include <linux/io.h>
 #include <linux/async.h>
 #include <linux/delay.h>
+#include <linux/dma-mapping.h>
 #include <linux/slab.h>
 #include <linux/clk.h>
 #include <linux/sh_clk.h>
@@ -487,7 +488,7 @@ int pcibios_map_platform_irq(const struct pci_dev *pdev, u8 slot, u8 pin)
 
 void pcibios_bus_add_device(struct pci_dev *pdev)
 {
-	pdev->dev.dma_pfn_offset = dma_pfn_offset;
+	dma_attach_uniform_pfn_offset(&pdev->dev, dma_pfn_offset);
 }
 
 static int __init sh7786_pcie_core_init(void)
arch/sh/kernel/dma-coherent.c +8 -6
index d4811691b93c..f4a092e74910 100644
@@ -14,6 +14,7 @@ void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle,
 {
 	void *ret, *ret_nocache;
 	int order = get_order(size);
+	phys_addr_t phys;
 
 	gfp |= __GFP_ZERO;
 
@@ -34,11 +35,12 @@ void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle,
 		return NULL;
 	}
 
-	split_page(pfn_to_page(virt_to_phys(ret) >> PAGE_SHIFT), order);
+	phys = virt_to_phys(ret);
+	split_page(pfn_to_page(PFN_DOWN(phys)), order);
 
-	*dma_handle = virt_to_phys(ret);
-	if (!WARN_ON(!dev))
-		*dma_handle -= PFN_PHYS(dev->dma_pfn_offset);
+	*dma_handle = (dma_addr_t)phys;
+	if (!WARN_ON(!dev) && dev->dma_pfn_offset_map)
+		*dma_handle -= PFN_PHYS(dma_pfn_offset_from_phys_addr(dev, phys));
 
 	return ret_nocache;
 }
@@ -50,8 +52,8 @@ void arch_dma_free(struct device *dev, size_t size, void *vaddr,
 	unsigned long pfn = (dma_handle >> PAGE_SHIFT);
 	int k;
 
-	if (!WARN_ON(!dev))
-		pfn += dev->dma_pfn_offset;
+	if (!WARN_ON(!dev) && dev->dma_pfn_offset_map)
+		pfn += dma_pfn_offset_from_dma_addr(dev, dma_handle);
 
 	for (k = 0; k < (1 << order); k++)
 		__free_pages(pfn_to_page(pfn + k), 0);
arch/x86/pci/sta2x11-fixup.c +5 -2
index c313d784efab..7815af63f870 100644
@@ -12,6 +12,7 @@
 #include <linux/export.h>
 #include <linux/list.h>
 #include <linux/dma-direct.h>
+#include <linux/dma-mapping.h>
 #include <asm/iommu.h>
 
 #define STA2X11_SWIOTLB_SIZE (4*1024*1024)
@@ -133,7 +134,7 @@ static void sta2x11_map_ep(struct pci_dev *pdev)
 	struct sta2x11_instance *instance = sta2x11_pdev_to_instance(pdev);
 	struct device *dev = &pdev->dev;
 	u32 amba_base, max_amba_addr;
-	int i;
+	int i, ret;
 
 	if (!instance)
 		return;
@@ -141,7 +142,9 @@ static void sta2x11_map_ep(struct pci_dev *pdev)
 	pci_read_config_dword(pdev, AHB_BASE(0), &amba_base);
 	max_amba_addr = amba_base + STA2X11_AMBA_SIZE - 1;
 
-	dev->dma_pfn_offset = PFN_DOWN(-amba_base);
+	ret = dma_attach_uniform_pfn_offset(dev, PFN_DOWN(-amba_base));
+	if (ret)
+		dev_err(dev, "sta2x11: could not set PFN offset\n");
 
 	dev->bus_dma_limit = max_amba_addr;
 	pci_set_consistent_dma_mask(pdev, max_amba_addr);
drivers/acpi/arm64/iort.c +2 -2
index 28a6b387e80e..38d1cc63034a 100644
@@ -1142,8 +1142,8 @@ void iort_dma_setup(struct device *dev, u64 *dma_addr, u64 *dma_size)
 	*dma_addr = dmaaddr;
 	*dma_size = size;
 
-	dev->dma_pfn_offset = PFN_DOWN(offset);
-	dev_dbg(dev, "dma_pfn_offset(%#08llx)\n", offset);
+	ret = dma_attach_uniform_pfn_offset(dev, PFN_DOWN(offset));
+	dev_dbg(dev, "dma_pfn_offset(%#08llx)%s\n", offset, ret ? " failed!" : "");
 }
 
 static void __init acpi_iort_register_irq(int hwirq, const char *name,
drivers/gpu/drm/sun4i/sun4i_backend.c +4 -1
index 072ea113e6be..772e6cc74be8 100644
@@ -11,6 +11,7 @@
 #include <linux/module.h>
 #include <linux/of_device.h>
 #include <linux/of_graph.h>
+#include <linux/dma-mapping.h>
 #include <linux/platform_device.h>
 #include <linux/reset.h>
 
@@ -812,7 +813,9 @@ static int sun4i_backend_bind(struct device *dev, struct device *master,
 		 * on our device since the RAM mapping is at 0 for the DMA bus,
 		 * unlike the CPU.
 		 */
-		drm->dev->dma_pfn_offset = PHYS_PFN_OFFSET;
+		ret = dma_attach_uniform_pfn_offset(dev, PHYS_PFN_OFFSET);
+		if (ret)
+			return ret;
 	}
 
 	backend->engine.node = dev->of_node;
drivers/iommu/io-pgtable-arm.c +1 -1
index 04fbd4bf0ff9..e9cc1c2d47cd 100644
@@ -754,7 +754,7 @@ arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg)
 	if (cfg->oas > ARM_LPAE_MAX_ADDR_BITS)
 		return NULL;
 
-	if (!selftest_running && cfg->iommu_dev->dma_pfn_offset) {
+	if (!selftest_running && cfg->iommu_dev->dma_pfn_offset_map) {
 		dev_err(cfg->iommu_dev, "Cannot accommodate DMA offset for IOMMU page tables\n");
 		return NULL;
 	}
drivers/media/platform/sunxi/sun4i-csi/sun4i_csi.c +4 -1
index eff34ded6305..d0d337ab9f1b 100644
@@ -7,6 +7,7 @@
  */
 
 #include <linux/clk.h>
+#include <linux/dma-mapping.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/mutex.h>
@@ -183,7 +184,9 @@ static int sun4i_csi_probe(struct platform_device *pdev)
 			return ret;
 	} else {
 #ifdef PHYS_PFN_OFFSET
-		csi->dev->dma_pfn_offset = PHYS_PFN_OFFSET;
+		ret = dma_attach_uniform_pfn_offset(dev, PHYS_PFN_OFFSET);
+		if (ret)
+			return ret;
 #endif
 	}
 
drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c +3 -1
index 055eb0b8e396..e91cc511f625 100644
@@ -898,7 +898,9 @@ static int sun6i_csi_probe(struct platform_device *pdev)
 
 	sdev->dev = &pdev->dev;
 	/* The DMA bus has the memory mapped at 0 */
-	sdev->dev->dma_pfn_offset = PHYS_OFFSET >> PAGE_SHIFT;
+	ret = dma_attach_uniform_pfn_offset(sdev->dev, PFN_DOWN(PHYS_OFFSET));
+	if (ret)
+		return ret;
 
 	ret = sun6i_csi_resource_request(sdev, pdev);
 	if (ret)
drivers/of/address.c +60 -11

116 lines not on this page · raw message

drivers/of/device.c +12 -7
index 27203bfd0b22..1501d92589b8 100644
@@ -91,11 +91,12 @@ int of_dma_configure(struct device *dev, struct device_node *np, bool force_dma)
 	u64 dma_addr, paddr, size = 0;
 	int ret;
 	bool coherent;
-	unsigned long offset;
 	const struct iommu_ops *iommu;
 	u64 mask, end;
+	size_t map_size;
+	struct bus_dma_region *map;
 
-	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
+	ret = of_dma_get_range(np, &dma_addr, &paddr, &size, &map, &map_size);
 	if (ret < 0) {
 		/*
 		 * For legacy reasons, we have to assume some devices need
@@ -105,9 +106,16 @@ int of_dma_configure(struct device *dev, struct device_node *np, bool force_dma)
 		if (!force_dma)
 			return ret == -ENODEV ? 0 : ret;
 
-		dma_addr = offset = 0;
+		dma_addr = 0;
 	} else {
-		offset = PFN_DOWN(paddr - dma_addr);
+		if (map) {
+			/* We want the offset map to be device-managed */
+			dev->dma_pfn_offset_map = devm_kzalloc(dev, map_size, GFP_KERNEL);
+			if (!dev->dma_pfn_offset_map)
+				return -ENOMEM;
+			memcpy(dev->dma_pfn_offset_map, map, map_size);
+			kfree(map);
+		}
 
 		/*
 		 * Add a work around to treat the size as mask + 1 in case
@@ -123,7 +131,6 @@ int of_dma_configure(struct device *dev, struct device_node *np, bool force_dma)
 			dev_err(dev, "Adjusted size 0x%llx invalid\n", size);
 			return -EINVAL;
 		}
-		dev_dbg(dev, "dma_pfn_offset(%#08lx)\n", offset);
 	}
 
 	/*
@@ -142,8 +149,6 @@ int of_dma_configure(struct device *dev, struct device_node *np, bool force_dma)
 	else if (!size)
 		size = 1ULL << 32;
 
-	dev->dma_pfn_offset = offset;
-
 	/*
 	 * Limit coherent and dma mask based on size and default mask
 	 * set by the driver.
drivers/of/of_private.h +7 -4
index edc682249c00..63bb7b0e94aa 100644
@@ -157,12 +157,15 @@ extern void __of_sysfs_remove_bin_file(struct device_node *np,
 extern int of_bus_n_addr_cells(struct device_node *np);
 extern int of_bus_n_size_cells(struct device_node *np);
 
+struct bus_dma_region;
 #ifdef CONFIG_OF_ADDRESS
-extern int of_dma_get_range(struct device_node *np, u64 *dma_addr,
-			    u64 *paddr, u64 *size);
+extern int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr,
+			    u64 *size, struct bus_dma_region **map,
+			    size_t *map_size);
 #else
-static inline int of_dma_get_range(struct device_node *np, u64 *dma_addr,
-				   u64 *paddr, u64 *size)
+static inline int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr,
+				   u64 *size, struct bus_dma_region **map,
+				   size_t *map_size)
 {
 	return -ENODEV;
 }
drivers/of/unittest.c +6 -2
index 398de04fd19c..33fa3947951f 100644
@@ -873,6 +873,8 @@ static void __init of_unittest_dma_ranges_one(const char *path,
 {
 	struct device_node *np;
 	u64 dma_addr, paddr, size;
+	size_t map_size;
+	struct bus_dma_region *map = NULL;
 	int rc;
 
 	np = of_find_node_by_path(path);
@@ -881,7 +883,7 @@ static void __init of_unittest_dma_ranges_one(const char *path,
 		return;
 	}
 
-	rc = of_dma_get_range(np, &dma_addr, &paddr, &size);
+	rc = of_dma_get_range(np, &dma_addr, &paddr, &size, &map, &map_size);
 
 	unittest(!rc, "of_dma_get_range failed on node %pOF rc=%i\n", np, rc);
 	if (!rc) {
@@ -891,6 +893,7 @@ static void __init of_unittest_dma_ranges_one(const char *path,
 			 "of_dma_get_range wrong phys addr (%llx) on node %pOF", paddr, np);
 		unittest(dma_addr == expect_dma_addr,
 			 "of_dma_get_range wrong DMA addr (%llx) on node %pOF", dma_addr, np);
+		kfree(map);
 	}
 	of_node_put(np);
 }
@@ -901,8 +904,9 @@ static void __init of_unittest_parse_dma_ranges(void)
 		0x0, 0x20000000, 0x40000000);
 	of_unittest_dma_ranges_one("/testcase-data/address-tests/bus@80000000/device@1000",
 		0x10000000, 0x20000000, 0x40000000);
+	/* This one has two ranges in the dma-range property */
 	of_unittest_dma_ranges_one("/testcase-data/address-tests/pci@90000000",
-		0x80000000, 0x20000000, 0x10000000);
+		0x80000000, 0x20000000, 0x50000000);
 }
 
 static void __init of_unittest_pci_dma_ranges(void)
drivers/remoteproc/remoteproc_core.c +1 -1
index 9f04c30c4aaf..ec543b5fa50d 100644
@@ -519,7 +519,7 @@ static int rproc_handle_vdev(struct rproc *rproc, struct fw_rsc_vdev *rsc,
 	/* Initialise vdev subdevice */
 	snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
 	rvdev->dev.parent = &rproc->dev;
-	rvdev->dev.dma_pfn_offset = rproc->dev.parent->dma_pfn_offset;
+	rvdev->dev.dma_pfn_offset_map = rproc->dev.parent->dma_pfn_offset_map;
 	rvdev->dev.release = rproc_rvdev_release;
 	dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev->dev.parent), name);
 	dev_set_drvdata(&rvdev->dev, rvdev);
drivers/staging/media/sunxi/cedrus/cedrus_hw.c +5 -2
index 1744e6fcc999..d821dcb2b632 100644
@@ -230,8 +230,11 @@ int cedrus_hw_probe(struct cedrus_dev *dev)
 	 */
 
 #ifdef PHYS_PFN_OFFSET
-	if (!(variant->quirks & CEDRUS_QUIRK_NO_DMA_OFFSET))
-		dev->dev->dma_pfn_offset = PHYS_PFN_OFFSET;
+	if (!(variant->quirks & CEDRUS_QUIRK_NO_DMA_OFFSET)) {
+		ret = dma_attach_uniform_pfn_offset(dev->dev, PHYS_PFN_OFFSET);
+		if (ret)
+			return ret;
+	}
 #endif
 
 	ret = of_reserved_mem_device_init(dev->dev);
drivers/usb/core/message.c +2 -2
index 6197938dcc2d..071856000428 100644
@@ -1956,10 +1956,10 @@ int usb_set_configuration(struct usb_device *dev, int configuration)
 		intf->dev.groups = usb_interface_groups;
 		/*
 		 * Please refer to usb_alloc_dev() to see why we set
-		 * dma_mask and dma_pfn_offset.
+		 * dma_mask and dma_pfn_offset_map.
 		 */
 		intf->dev.dma_mask = dev->dev.dma_mask;
-		intf->dev.dma_pfn_offset = dev->dev.dma_pfn_offset;
+		intf->dev.dma_pfn_offset_map = dev->dev.dma_pfn_offset_map;
 		INIT_WORK(&intf->reset_ws, __usb_queue_reset_device);
 		intf->minor = -1;
 		device_initialize(&intf->dev);
drivers/usb/core/usb.c +1 -1
index f16c26dc079d..3fbc0c06ce9c 100644
@@ -611,7 +611,7 @@ struct usb_device *usb_alloc_dev(struct usb_device *parent,
 	 * mask for the entire HCD, so don't do that.
 	 */
 	dev->dev.dma_mask = bus->sysdev->dma_mask;
-	dev->dev.dma_pfn_offset = bus->sysdev->dma_pfn_offset;
+	dev->dev.dma_pfn_offset_map = bus->sysdev->dma_pfn_offset_map;
 	set_dev_node(&dev->dev, dev_to_node(bus->sysdev));
 	dev->state = USB_STATE_ATTACHED;
 	dev->lpm_disable_count = 1;
include/linux/device.h +2 -2
index 15460a5ac024..a17da8e271a2 100644
@@ -492,7 +492,7 @@ struct dev_links_info {
  * 		such descriptors.
  * @bus_dma_limit: Limit of an upstream bridge or bus which imposes a smaller
  *		DMA limit than the device itself supports.
- * @dma_pfn_offset: offset of DMA memory range relatively of RAM
+ * @dma_pfn_offset_map: offset map for DMA memory range relatively of RAM
  * @dma_parms:	A low level driver may set these to teach IOMMU code about
  * 		segment limitations.
  * @dma_pools:	Dma pools (if dma'ble device).
@@ -577,7 +577,7 @@ struct device {
 					     64 bit addresses for consistent
 					     allocations such descriptors. */
 	u64		bus_dma_limit;	/* upstream dma constraint */
-	unsigned long	dma_pfn_offset;
+	struct bus_dma_region *dma_pfn_offset_map;
 
 	struct device_dma_parameters *dma_parms;
 
include/linux/dma-direct.h +12 -2
index 136f984df0d9..d52fc351e073 100644
@@ -15,14 +15,24 @@ static inline dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t paddr)
 {
 	dma_addr_t dev_addr = (dma_addr_t)paddr;
 
-	return dev_addr - ((dma_addr_t)dev->dma_pfn_offset << PAGE_SHIFT);
+	if (dev->dma_pfn_offset_map) {
+		unsigned long dma_pfn_offset = dma_pfn_offset_from_phys_addr(dev, paddr);
+
+		dev_addr -= PFN_PHYS((dma_addr_t)dma_pfn_offset);
+	}
+	return dev_addr;
 }
 
 static inline phys_addr_t __dma_to_phys(struct device *dev, dma_addr_t dev_addr)
 {
 	phys_addr_t paddr = (phys_addr_t)dev_addr;
 
-	return paddr + ((phys_addr_t)dev->dma_pfn_offset << PAGE_SHIFT);
+	if (dev->dma_pfn_offset_map) {
+		unsigned long dma_pfn_offset = dma_pfn_offset_from_dma_addr(dev, dev_addr);
+
+		paddr += PFN_PHYS((phys_addr_t)dma_pfn_offset);
+	}
+	return paddr;
 }
 #endif /* !CONFIG_ARCH_HAS_PHYS_TO_DMA */
 
include/linux/dma-mapping.h +38 -0
index 78f677cf45ab..81fab7a37c31 100644
@@ -256,6 +256,39 @@ static inline void dma_direct_sync_sg_for_cpu(struct device *dev,
 size_t dma_direct_max_mapping_size(struct device *dev);
 
 #ifdef CONFIG_HAS_DMA
+struct bus_dma_region {
+	phys_addr_t	cpu_start;
+	phys_addr_t	cpu_end;
+	dma_addr_t	dma_start;
+	dma_addr_t	dma_end;
+	unsigned long	pfn_offset;
+};
+
+int dma_attach_uniform_pfn_offset(struct device *dev,
+				  unsigned long dma_pfn_offset);
+
+static inline unsigned long dma_pfn_offset_from_dma_addr(struct device *dev,
+							 dma_addr_t dma_addr)
+{
+	const struct bus_dma_region *m = dev->dma_pfn_offset_map;
+
+	for (; m->cpu_end; m++)
+		if (dma_addr >= m->dma_start && dma_addr <= m->dma_end)
+			return m->pfn_offset;
+	return 0;
+}
+
+static inline unsigned long dma_pfn_offset_from_phys_addr(struct device *dev,
+							  phys_addr_t paddr)
+{
+	const struct bus_dma_region *m = dev->dma_pfn_offset_map;
+
+	for (; m->cpu_end; m++)
+		if (paddr >= m->cpu_start && paddr <= m->cpu_end)
+			return m->pfn_offset;
+	return 0;
+}
+
 #include <asm/dma-mapping.h>
 
 static inline const struct dma_map_ops *get_dma_ops(struct device *dev)
@@ -463,6 +496,11 @@ u64 dma_get_required_mask(struct device *dev);
 size_t dma_max_mapping_size(struct device *dev);
 unsigned long dma_get_merge_boundary(struct device *dev);
 #else /* CONFIG_HAS_DMA */
+static inline int dma_attach_uniform_pfn_offset(struct device *dev,
+						unsigned long dma_pfn_offset)
+{
+	return -EIO;
+}
 static inline dma_addr_t dma_map_page_attrs(struct device *dev,
 		struct page *page, size_t offset, size_t size,
 		enum dma_data_direction dir, unsigned long attrs)
kernel/dma/coherent.c +7 -4
index 2a0c4985f38e..a5403bf996c1 100644
@@ -31,10 +31,13 @@ static inline struct dma_coherent_mem *dev_get_coherent_memory(struct device *de
 static inline dma_addr_t dma_get_device_base(struct device *dev,
 					     struct dma_coherent_mem * mem)
 {
-	if (mem->use_dev_dma_pfn_offset)
-		return (mem->pfn_base - dev->dma_pfn_offset) << PAGE_SHIFT;
-	else
-		return mem->device_base;
+	if (mem->use_dev_dma_pfn_offset && dev->dma_pfn_offset_map) {
+		unsigned long dma_pfn_offset
+			= dma_pfn_offset_from_phys_addr(dev, PFN_PHYS(mem->pfn_base));
+
+		return (dma_addr_t)PFN_PHYS(mem->pfn_base - dma_pfn_offset);
+	}
+	return mem->device_base;
 }
 
 static int dma_init_coherent_memory(phys_addr_t phys_addr,
kernel/dma/mapping.c +39 -0
index 98e3d873792e..20149a29d136 100644
@@ -11,6 +11,7 @@
 #include <linux/dma-noncoherent.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/limits.h>
 #include <linux/of_device.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
@@ -407,3 +408,41 @@ unsigned long dma_get_merge_boundary(struct device *dev)
 	return ops->get_merge_boundary(dev);
 }
 EXPORT_SYMBOL_GPL(dma_get_merge_boundary);
+
+/**
+ * dma_attach_uniform_pfn_offset - Assign scalar offset for all addresses.
+ * @dev:	device pointer; needed to "own" the alloced memory.
+ * @pfn_offset:	offset to apply when converting from phys addr
+ *		to dma addr and vice versa.
+ *
+ * This is for the simple case of a uniform offset which requires
+ * no bounds calculations for its use.  It is the equivalent the role
+ * of the former dev->dma_pfn_offset; ie it is just blindly added
+ * or subtracted in all cases.
+ *
+ * It returns -ENOMEM if out of memory, -ENODEV if dev == NULL, otherwise 0.
+ */
+int dma_attach_uniform_pfn_offset(struct device *dev, unsigned long pfn_offset)
+{
+	struct bus_dma_region *map;
+
+	if (!dev)
+		return -ENODEV;
+
+	if (!pfn_offset)
+		return 0;
+
+	map = devm_kcalloc(dev, 2, sizeof(*map), GFP_KERNEL);
+	if (!map)
+		return -ENOMEM;
+
+	map->pfn_offset = pfn_offset;
+	map->cpu_start = 0;
+	map->cpu_end = PHYS_ADDR_MAX;
+	map->dma_start = 0;
+	map->dma_end = ~(dma_addr_t)0;
+	dev->dma_pfn_offset_map = map;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dma_attach_uniform_pfn_offset);
Signature · 6 lines
2.17.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
reviewer
#14 lore raw

Re: [PATCH v5 08/12] device core: Introduce multiple dma pfn offsets

Hi Jim,

Thanks for taking this on!

Quoted 3 lines · Jim Quinlan

The new field in struct device 'dma_pfn_offset_map' is used to facilitate the use of single or multiple pfn offsets between cpu addrs and dma addrs. It subsumes the role of dev->dma_pfn_offset -- a uniform offset.

This isn't just about offsets - it should (eventually) subsume bus_dma_limit as well, so I'd be inclined to call it something like "dma_ranges"/"dma_range_map"/"dma_regions"/etc.

Quoted 9 lines

The function of_dma_get_range() has been modified to take two additional
arguments: the "map", which is an array that holds the information regarding the pfn offset regions, and map_size, which is the size in bytes of the map array.

of_dma_configure() is the typical manner to set pfn offsets but there are a number of ad hoc assignments to dev->dma_pfn_offset in the kernel driver code. These cases now invoke the function dma_attach_uniform_pfn_offset(dev, pfn_offset).

I'm also not convinced that sticking to the PFN paradigm is necessarily the right way to go - when there's only a single nicely-aligned offset to consider then an unsigned long that's immune to PAE/LPAE/etc. disruption is indeed the cheapest and easiest option from core code's PoV. However it already means that all the users have to do some degree of conversion back and forth between PFNs and usable addresses; once the core code itself also has to start bouncing back and forth between addresses and PFNs internally then we end up effectively just doing work to cancel out other work, and the whole lot would end up simpler and more efficient if the API worked purely in terms of addresses.

[...]

Quoted 15 lines
drivers/of/address.c +7 -0
index 8eea3f6e29a4..767fa3b492c8 100644
@@ -918,12 +918,48 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  }
  EXPORT_SYMBOL(of_io_request_and_map);
 
+static int dma_attach_pfn_offset_map(struct device_node *node, int num_ranges,
+				     struct bus_dma_region **map, size_t *map_size)
+{
+	struct of_range_parser parser;
+	struct of_range range;
+	struct bus_dma_region *r;
+

drivers/of/address.c void __iomem *of_io_request_and_map(struct device_node *np, int index, · line 937

Quoted 10 lines
drivers/of/address.c +10 -0
+	*map_size = (num_ranges + 1) * sizeof(**map);
+	r = kcalloc(num_ranges + 1, sizeof(**map), GFP_KERNEL);
+	if (!r)
+		return -ENOMEM;
+	*map = r;
+
+	of_dma_range_parser_init(&parser, node);
+	/*
+	 * Record all info for DMA ranges array.  We could
+	 * just use the of_range struct, but if we did that it

Not making the entire DMA API depend on OF is a far better justification for having its own dedicated structure.

drivers/of/address.c void __iomem *of_io_request_and_map(struct device_node *np, int index, · line 940

Quoted 3 lines
drivers/of/address.c +3 -0
+	 * would require more calculations for phys_to_dma and
+	 * dma_to_phys conversions.
+	 */

However that part is pretty much nonsense. Consider your "efficient" operation for looking up and consuming a DMA offset:

	API				caller

1. load cpu_start
2. compare addr >= cpu_start
3. load cpu_end
4. compare addr <= cpu_end
5. load pfn_offset
6. shift pfn_offset << PAGE_SHIFT
7. add "offset" + addr
8. [use the result]

versus the "more calculations" approach (once the PFN cruft is peeled away):

	API				caller

1. load cpu_addr
2. compare addr >= cpu_addr
3. subtract addr - cpu_addr
4. load size
5. compare "addr_offset" < size
6. load dma_start
7. add dma_start + "addr_offset"
8. [use the result]

Oh look, it's the exact same number of memory accesses and ALU operations, but with a smaller code footprint (assuming, reasonably, more than one caller) and less storage overhead ;)

Basically, having this degree of redundancy is somewhere between silly and actively harmful (what if pfn_offset gets out of sync with cpu_start/dma_start? What if cpu_end/dma_end don't represent equivalent lengths?)

Quoted 27 lines
drivers/of/address.c +16 -0
+	for_each_of_range(&parser, &range) {
+		r->cpu_start = range.cpu_addr;
+		r->cpu_end = r->cpu_start + range.size - 1;
+		r->dma_start = range.bus_addr;
+		r->dma_end = r->dma_start + range.size - 1;
+		r->pfn_offset = PFN_DOWN(range.cpu_addr) - PFN_DOWN(range.bus_addr);
+		r++;
+	}
+	return 0;
+}
+
  /**
   * of_dma_get_range - Get DMA range info
   * @np:		device node to get DMA range info
   * @dma_addr:	pointer to store initial DMA address of DMA range
   * @paddr:	pointer to store initial CPU address of DMA range
   * @size:	pointer to store size of DMA range
+ * @map:	pointer to a pointer of an array of structs.  This is updated
+ *		to point to NULL (no offsets needed) or kmalloc'd array of
+ *		structs.  In the latter case, it is the caller's obligation to
+ *		kfree the array in the case it is no longer in use.
+ * @map_size:	updated to be the size in bytes of memory allocated to the map
   *
   * Look in bottom up direction for the first "dma-ranges" property
   * and parse it.
@@ -932,10 +968,11 @@ EXPORT_SYMBOL(of_io_request_and_map);
   *	CPU addr (phys_addr_t)	: pna cells

drivers/of/address.c EXPORT_SYMBOL(of_io_request_and_map); · line 975

Quoted 10 lines
drivers/of/address.c +4 -3
   *	size			: nsize cells
   *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * It returns -ENODEV if "dma-ranges" property was not found for this
+ * device in the DT.
   */
-int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
+int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr,
+		     u64 *size, struct bus_dma_region **map, size_t *map_size)

dma_addr, paddr and size don't really make much sense any more when we could be handling multiple ranges - it would seem logical to simplify the interface entirely:

struct bus_dma_region *of_dma_get_range(struct device_node *np);

which returns either a valid map, NULL if dma-ranges is absent, or an ERR_PTR.

Quoted 15 lines
drivers/of/address.c +4 -1
  {
  	struct device_node *node = of_node_get(np);
  	const __be32 *ranges = NULL;
@@ -944,7 +981,10 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
  	bool found_dma_ranges = false;
  	struct of_range_parser parser;
  	struct of_range range;
-	u64 dma_start = U64_MAX, dma_end = 0, dma_offset = 0;
+	phys_addr_t cpu_start = ~(phys_addr_t)0;
+	u64 dma_start = U64_MAX, dma_end = 0;
+	bool offset_map_needed = false;
+	int num_ranges = 0;
 
  	while (node) {
  		ranges = of_get_property(node, "dma-ranges", &len);

[...]

Quoted 14 lines
include/linux/device.h +1 -1
index 15460a5ac024..a17da8e271a2 100644
@@ -492,7 +492,7 @@ struct dev_links_info {
   * 		such descriptors.
   * @bus_dma_limit: Limit of an upstream bridge or bus which imposes a smaller
   *		DMA limit than the device itself supports.
- * @dma_pfn_offset: offset of DMA memory range relatively of RAM
+ * @dma_pfn_offset_map: offset map for DMA memory range relatively of RAM
   * @dma_parms:	A low level driver may set these to teach IOMMU code about
   * 		segment limitations.
   * @dma_pools:	Dma pools (if dma'ble device).

include/linux/device.h struct device { · line 580

Quoted 5 lines
include/linux/device.h +1 -1
@@ -577,7 +577,7 @@ struct device {
  					     64 bit addresses for consistent
  					     allocations such descriptors. */
  	u64		bus_dma_limit;	/* upstream dma constraint */
-	unsigned long	dma_pfn_offset;
+	struct bus_dma_region *dma_pfn_offset_map;

I think it's a very good idea for this to be const.

Quoted 2 lines
include/linux/device.h +0 -0
  	struct device_dma_parameters *dma_parms;
 

[...]

Quoted 15 lines
kernel/dma/mapping.c +1 -0
index 98e3d873792e..20149a29d136 100644
@@ -11,6 +11,7 @@
  #include <linux/dma-noncoherent.h>
  #include <linux/export.h>
  #include <linux/gfp.h>
+#include <linux/limits.h>
  #include <linux/of_device.h>
  #include <linux/slab.h>
  #include <linux/vmalloc.h>
@@ -407,3 +408,41 @@ unsigned long dma_get_merge_boundary(struct device *dev)
  	return ops->get_merge_boundary(dev);
  }

kernel/dma/mapping.c unsigned long dma_get_merge_boundary(struct device *dev) · line 419

Quoted 10 lines
kernel/dma/mapping.c +9 -0
  EXPORT_SYMBOL_GPL(dma_get_merge_boundary);
+
+/**
+ * dma_attach_uniform_pfn_offset - Assign scalar offset for all addresses.
+ * @dev:	device pointer; needed to "own" the alloced memory.
+ * @pfn_offset:	offset to apply when converting from phys addr
+ *		to dma addr and vice versa.
+ *
+ * This is for the simple case of a uniform offset which requires
+ * no bounds calculations for its use.

That's crap - just because we've been (mostly) getting away without them doesn't mean bounds checks aren't relevant to a uniform offset. Take a look at this saga for an example of phys_to_dma() on something *below* the offset being a very nasty problem indeed:

https://lore.kernel.org/linux-arm-kernel/[email protected]/

Quoted 7 lines
kernel/dma/mapping.c +6 -0
 It is the equivalent the role
+ * of the former dev->dma_pfn_offset; ie it is just blindly added
+ * or subtracted in all cases.
+ *
+ * It returns -ENOMEM if out of memory, -ENODEV if dev == NULL, otherwise 0.
+ */
+int dma_attach_uniform_pfn_offset(struct device *dev, unsigned long pfn_offset)

kernel/dma/mapping.c unsigned long dma_get_merge_boundary(struct device *dev) · line 435

Quoted 10 lines
kernel/dma/mapping.c +10 -0
+{
+	struct bus_dma_region *map;
+
+	if (!dev)
+		return -ENODEV;
+
+	if (!pfn_offset)
+		return 0;
+
+	map = devm_kcalloc(dev, 2, sizeof(*map), GFP_KERNEL);

Nope, if you want to call this from bus notifiers then it can't use devres - see the list_empty(&dev->devres_head) check in really_probe(). However, I think those platform-level cases are probably better off statically allocating a single map for all devices to share (I had an idea about doing similar in the DT code, but that could be a future optimisation); this function is really only needed for nasty driver-level hacks like the sunxi ones.

And for those driver-level hacks, I think it would also be worth being a bit cleverer here, i.e. not blindly replacing any existing ranges (but returning success if the desired offset is already in place).

kernel/dma/mapping.c unsigned long dma_get_merge_boundary(struct device *dev) · line 443

Quoted 8 lines
kernel/dma/mapping.c +8 -0
+	if (!map)
+		return -ENOMEM;
+
+	map->pfn_offset = pfn_offset;
+	map->cpu_start = 0;
+	map->cpu_end = PHYS_ADDR_MAX;
+	map->dma_start = 0;
+	map->dma_end = ~(dma_addr_t)0;

As above, allowing these to be out-of-sync with the fundamental constraints of dma_pfn_offset seems really bad - there is no valid DMA address corresponding to cpu_start and no representable physical address corresponding to dma_end, and no good can come of that. But of course getting rid of the redundancy entirely is even better.

Robin.

kernel/dma/mapping.c unsigned long dma_get_merge_boundary(struct device *dev) · line 448

Quoted 6 lines
kernel/dma/mapping.c +5 -0
+	dev->dma_pfn_offset_map = map;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(dma_attach_uniform_pfn_offset);

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#15 lore raw

Re: [PATCH v5 08/12] device core: Introduce multiple dma pfn offsets

Quoted 4 lines · Robin Murphy

Hi Jim,

Thanks for taking this on!

Hi Robin,

Quoted 9 lines · Jim Quinlan
Quoted 3 lines

The new field in struct device 'dma_pfn_offset_map' is used to facilitate the use of single or multiple pfn offsets between cpu addrs and dma addrs. It subsumes the role of dev->dma_pfn_offset -- a uniform offset.

This isn't just about offsets - it should (eventually) subsume bus_dma_limit as well, so I'd be inclined to call it something like "dma_ranges"/"dma_range_map"/"dma_regions"/etc.

I will change my wording here.

Quoted 11 lines
Quoted 10 lines

The function of_dma_get_range() has been modified to take two additional
arguments: the "map", which is an array that holds the information regarding the pfn offset regions, and map_size, which is the size in bytes of the map array.

of_dma_configure() is the typical manner to set pfn offsets but there are a number of ad hoc assignments to dev->dma_pfn_offset in the kernel driver code. These cases now invoke the function dma_attach_uniform_pfn_offset(dev, pfn_offset).

Quoted 10 lines

I'm also not convinced that sticking to the PFN paradigm is necessarily the right way to go - when there's only a single nicely-aligned offset to consider then an unsigned long that's immune to PAE/LPAE/etc. disruption is indeed the cheapest and easiest option from core code's PoV. However it already means that all the users have to do some degree of conversion back and forth between PFNs and usable addresses; once the core code itself also has to start bouncing back and forth between addresses and PFNs internally then we end up effectively just doing work to cancel out other work, and the whole lot would end up simpler and more efficient if the API worked purely in terms of addresses.

If you want me to change the paradigm to an address offset, I will. If so please specify its type: dma_addr_t, phys_addr_t, or u64?

Quoted 60 lines

[...]

Quoted 15 lines
drivers/of/address.c +7 -0
index 8eea3f6e29a4..767fa3b492c8 100644
@@ -918,12 +918,48 @@ void __iomem *of_io_request_and_map(struct device_node *np, int index,
  }
  EXPORT_SYMBOL(of_io_request_and_map);
 
+static int dma_attach_pfn_offset_map(struct device_node *node, int num_ranges,
+                                  struct bus_dma_region **map, size_t *map_size)
+{
+     struct of_range_parser parser;
+     struct of_range range;
+     struct bus_dma_region *r;
+

drivers/of/address.c void __iomem *of_io_request_and_map(struct device_node *np, int index, · line 937

Quoted 10 lines
drivers/of/address.c +10 -0
+     *map_size = (num_ranges + 1) * sizeof(**map);
+     r = kcalloc(num_ranges + 1, sizeof(**map), GFP_KERNEL);
+     if (!r)
+             return -ENOMEM;
+     *map = r;
+
+     of_dma_range_parser_init(&parser, node);
+     /*
+      * Record all info for DMA ranges array.  We could
+      * just use the of_range struct, but if we did that it

Not making the entire DMA API depend on OF is a far better justification for having its own dedicated structure.

drivers/of/address.c void __iomem *of_io_request_and_map(struct device_node *np, int index, · line 940

Quoted 3 lines
drivers/of/address.c +3 -0
+      * would require more calculations for phys_to_dma and
+      * dma_to_phys conversions.
+      */

However that part is pretty much nonsense. Consider your "efficient" operation for looking up and consuming a DMA offset:

API caller
1. load cpu_start
2. compare addr >= cpu_start
3. load cpu_end
4. compare addr <= cpu_end
5. load pfn_offset

6.                              shift pfn_offset << PAGE_SHIFT
7.                              add "offset" + addr
8.                              [use the result]

versus the "more calculations" approach (once the PFN cruft is peeled away):

API caller
1. load cpu_addr
2. compare addr >= cpu_addr
3. subtract addr - cpu_addr
4. load size
5. compare "addr_offset" < size
6. load dma_start
7. add dma_start + "addr_offset"
8. [use the result]

Quoted 3 lines

Oh look, it's the exact same number of memory accesses and ALU operations, but with a smaller code footprint (assuming, reasonably, more than one caller) and less storage overhead ;)

I feel you have conflated two independent issues to get your comparison. Issue one is having two extra fields -- cpu_end, dma_end -- and issue two is using address verses PFN offsets. If you look at both code sequences when using an address offset verses PFN offset, the of_range way requires an extra computation. Do you agree?

Quoted 4 lines

Basically, having this degree of redundancy is somewhere between silly and actively harmful (what if pfn_offset gets out of sync with cpu_start/dma_start?

How can this happen if the range map is set once and passed as a const structure?

Quoted 1 line

What if cpu_end/dma_end don't represent equivalent lengths?)

They are set once using the same +size operation so they must be equal. They are not changed after they are set.

Nonetheless, I really don't care which way we do it. If you want me to have an of_range-like structure I am fine with that.

Quoted 43 lines
Quoted 28 lines
drivers/of/address.c +16 -0
+     for_each_of_range(&parser, &range) {
+             r->cpu_start = range.cpu_addr;
+             r->cpu_end = r->cpu_start + range.size - 1;
+             r->dma_start = range.bus_addr;
+             r->dma_end = r->dma_start + range.size - 1;
+             r->pfn_offset = PFN_DOWN(range.cpu_addr) - PFN_DOWN(range.bus_addr);
+             r++;
+     }
+     return 0;
+}
+
  /**
   * of_dma_get_range - Get DMA range info
   * @np:             device node to get DMA range info
   * @dma_addr:       pointer to store initial DMA address of DMA range
   * @paddr:  pointer to store initial CPU address of DMA range
   * @size:   pointer to store size of DMA range
+ * @map:     pointer to a pointer of an array of structs.  This is updated
+ *           to point to NULL (no offsets needed) or kmalloc'd array of
+ *           structs.  In the latter case, it is the caller's obligation to
+ *           kfree the array in the case it is no longer in use.
+ * @map_size:        updated to be the size in bytes of memory allocated to the map
   *
   * Look in bottom up direction for the first "dma-ranges" property
   * and parse it.
@@ -932,10 +968,11 @@ EXPORT_SYMBOL(of_io_request_and_map);
   *  CPU addr (phys_addr_t)  : pna cells

drivers/of/address.c EXPORT_SYMBOL(of_io_request_and_map); · line 975

Quoted 10 lines
drivers/of/address.c +4 -3
   *  size                    : nsize cells
   *
- * It returns -ENODEV if "dma-ranges" property was not found
- * for this device in DT.
+ * It returns -ENODEV if "dma-ranges" property was not found for this
+ * device in the DT.
   */
-int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *size)
+int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr,
+                  u64 *size, struct bus_dma_region **map, size_t *map_size)

dma_addr, paddr and size don't really make much sense any more when we could be handling multiple ranges - it would seem logical to simplify the interface entirely:

Quoted 4 lines

struct bus_dma_region *of_dma_get_range(struct device_node *np);

which returns either a valid map, NULL if dma-ranges is absent, or an ERR_PTR.

Fair enough, I'll change this. As you may have guessed I typically lean towards just changing the minimal code I need to change and preserving the rest.

Quoted 30 lines
Quoted 16 lines
drivers/of/address.c +4 -1
  {
      struct device_node *node = of_node_get(np);
      const __be32 *ranges = NULL;
@@ -944,7 +981,10 @@ int of_dma_get_range(struct device_node *np, u64 *dma_addr, u64 *paddr, u64 *siz
      bool found_dma_ranges = false;
      struct of_range_parser parser;
      struct of_range range;
-     u64 dma_start = U64_MAX, dma_end = 0, dma_offset = 0;
+     phys_addr_t cpu_start = ~(phys_addr_t)0;
+     u64 dma_start = U64_MAX, dma_end = 0;
+     bool offset_map_needed = false;
+     int num_ranges = 0;
 
      while (node) {
              ranges = of_get_property(node, "dma-ranges", &len);

[...]

Quoted 13 lines
include/linux/device.h +1 -1
index 15460a5ac024..a17da8e271a2 100644
@@ -492,7 +492,7 @@ struct dev_links_info {
   *          such descriptors.
   * @bus_dma_limit: Limit of an upstream bridge or bus which imposes a smaller
   *          DMA limit than the device itself supports.
- * @dma_pfn_offset: offset of DMA memory range relatively of RAM
+ * @dma_pfn_offset_map: offset map for DMA memory range relatively of RAM
   * @dma_parms:      A low level driver may set these to teach IOMMU code about
   *          segment limitations.
   * @dma_pools:      Dma pools (if dma'ble device).
Quoted 8 lines

include/linux/device.h struct device { · line 580

Quoted 6 lines
include/linux/device.h +1 -1
@@ -577,7 +577,7 @@ struct device {
                                           64 bit addresses for consistent
                                           allocations such descriptors. */
      u64             bus_dma_limit;  /* upstream dma constraint */
-     unsigned long   dma_pfn_offset;
+     struct bus_dma_region *dma_pfn_offset_map;

I think it's a very good idea for this to be const.

Agree.

Quoted 35 lines
Quoted 3 lines
include/linux/device.h +0 -0
      struct device_dma_parameters *dma_parms;
 

[...]

Quoted 15 lines
kernel/dma/mapping.c +1 -0
index 98e3d873792e..20149a29d136 100644
@@ -11,6 +11,7 @@
  #include <linux/dma-noncoherent.h>
  #include <linux/export.h>
  #include <linux/gfp.h>
+#include <linux/limits.h>
  #include <linux/of_device.h>
  #include <linux/slab.h>
  #include <linux/vmalloc.h>
@@ -407,3 +408,41 @@ unsigned long dma_get_merge_boundary(struct device *dev)
      return ops->get_merge_boundary(dev);
  }

kernel/dma/mapping.c unsigned long dma_get_merge_boundary(struct device *dev) · line 419

Quoted 10 lines
kernel/dma/mapping.c +9 -0
  EXPORT_SYMBOL_GPL(dma_get_merge_boundary);
+
+/**
+ * dma_attach_uniform_pfn_offset - Assign scalar offset for all addresses.
+ * @dev:     device pointer; needed to "own" the alloced memory.
+ * @pfn_offset:      offset to apply when converting from phys addr
+ *           to dma addr and vice versa.
+ *
+ * This is for the simple case of a uniform offset which requires
+ * no bounds calculations for its use.

That's crap - just because we've been (mostly) getting away without them doesn't mean bounds checks aren't relevant to a uniform offset. Take a look at this saga for an example of phys_to_dma() on something *below* the offset being a very nasty problem indeed:

Quoted 1 line

https://lore.kernel.org/linux-arm-kernel/[email protected]/

Fair enough, I would love to have bounds for this and enforce them. The problem is that all of the drivers that are setting the offset do not indicate what the bounds actually are. How do I get around this (other than removing my comment)?

Quoted 19 lines
Quoted 8 lines
kernel/dma/mapping.c +6 -0
 It is the equivalent the role
+ * of the former dev->dma_pfn_offset; ie it is just blindly added
+ * or subtracted in all cases.
+ *
+ * It returns -ENOMEM if out of memory, -ENODEV if dev == NULL, otherwise 0.
+ */
+int dma_attach_uniform_pfn_offset(struct device *dev, unsigned long pfn_offset)

kernel/dma/mapping.c unsigned long dma_get_merge_boundary(struct device *dev) · line 435

Quoted 10 lines
kernel/dma/mapping.c +10 -0
+{
+     struct bus_dma_region *map;
+
+     if (!dev)
+             return -ENODEV;
+
+     if (!pfn_offset)
+             return 0;
+
+     map = devm_kcalloc(dev, 2, sizeof(*map), GFP_KERNEL);
Quoted 7 lines

Nope, if you want to call this from bus notifiers then it can't use devres - see the list_empty(&dev->devres_head) check in really_probe(). However, I think those platform-level cases are probably better off statically allocating a single map for all devices to share (I had an idea about doing similar in the DT code, but that could be a future optimisation); this function is really only needed for nasty driver-level hacks like the sunxi ones.

Yes, but there are a decent number of these drivers. To be clear, you have reservations but no change in code is required?

Quoted 4 lines

And for those driver-level hacks, I think it would also be worth being a bit cleverer here, i.e. not blindly replacing any existing ranges (but returning success if the desired offset is already in place).

Got it.

Quoted 10 lines

kernel/dma/mapping.c unsigned long dma_get_merge_boundary(struct device *dev) · line 443

Quoted 9 lines
kernel/dma/mapping.c +8 -0
+     if (!map)
+             return -ENOMEM;
+
+     map->pfn_offset = pfn_offset;
+     map->cpu_start = 0;
+     map->cpu_end = PHYS_ADDR_MAX;
+     map->dma_start = 0;
+     map->dma_end = ~(dma_addr_t)0;
Quoted 5 lines

As above, allowing these to be out-of-sync with the fundamental constraints of dma_pfn_offset seems really bad - there is no valid DMA address corresponding to cpu_start and no representable physical address corresponding to dma_end, and no good can come of that. But of course getting rid of the redundancy entirely is even better.

Understood.

Thanks much for your time and review,
Jim Quinlan

Quoted 9 lines

Robin.

kernel/dma/mapping.c unsigned long dma_get_merge_boundary(struct device *dev) · line 448

Quoted 6 lines
kernel/dma/mapping.c +5 -0
+     dev->dma_pfn_offset_map = map;
+
+     return 0;
+}
+EXPORT_SYMBOL_GPL(dma_attach_uniform_pfn_offset);

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#16 lore raw

PATCH v5 9/12 PCI: brcmstb: Set internal memory viewport sizes

In reply to Jim Quinlan

BrcmSTB PCIe controllers are intimately connected to the memory controller(s) on the SOC. There is a "viewport" for each memory controller that allows inbound accesses to CPU memory. Each viewport's size must be set to a power of two, and that size must be equal to or larger than the amount of memory each controller supports.

---
 drivers/pci/controller/pcie-brcmstb.c | 68 ++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 18 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +50 -18
index 9189406fd35c..39f77709c6a2 100644
@@ -57,6 +57,8 @@
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_MASK	0x300000
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_128		0x0
 #define  PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK		0xf8000000
+#define  PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK		0x07c00000
+#define  PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK		0x0000001f
 
 #define PCIE_MISC_CPU_2_PCIE_MEM_WIN0_LO		0x400c
 #define PCIE_MEM_WIN0_LO(win)	\
@@ -154,6 +156,7 @@
 #define SSC_STATUS_OFFSET		0x1
 #define SSC_STATUS_SSC_MASK		0x400
 #define SSC_STATUS_PLL_LOCK_MASK	0x800
+#define PCIE_BRCM_MAX_MEMC		3
 
 #define IDX_ADDR(pcie)			(pcie->reg_offsets[EXT_CFG_INDEX])
 #define DATA_ADDR(pcie)			(pcie->reg_offsets[EXT_CFG_DATA])
@@ -260,6 +263,8 @@ struct brcm_pcie {
 	const int		*reg_field_info;
 	enum pcie_type		type;
 	struct reset_control	*rescal;
+	int			num_memc;
+	u64			memc_size[PCIE_BRCM_MAX_MEMC];
 };
 
 /*
@@ -715,22 +720,44 @@ static inline int brcm_pcie_get_rc_bar2_size_and_offset(struct brcm_pcie *pcie,
 							u64 *rc_bar2_offset)
 {
 	struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
-	struct device *dev = pcie->dev;
 	struct resource_entry *entry;
+	struct device *dev = pcie->dev;
+	u64 lowest_pcie_addr = ~(u64)0;
+	int ret, i = 0;
+	u64 size = 0;
 
-	entry = resource_list_first_type(&bridge->dma_ranges, IORESOURCE_MEM);
-	if (!entry)
-		return -ENODEV;
+	resource_list_for_each_entry(entry, &bridge->dma_ranges) {
+		u64 pcie_beg = entry->res->start - entry->offset;
 
+		size += entry->res->end - entry->res->start + 1;
+		if (pcie_beg < lowest_pcie_addr)
+			lowest_pcie_addr = pcie_beg;
+	}
 
-	/*
-	 * The controller expects the inbound window offset to be calculated as
-	 * the difference between PCIe's address space and CPU's. The offset
-	 * provided by the firmware is calculated the opposite way, so we
-	 * negate it.
-	 */
-	*rc_bar2_offset = -entry->offset;
-	*rc_bar2_size = 1ULL << fls64(entry->res->end - entry->res->start);
+	if (lowest_pcie_addr == ~(u64)0) {
+		dev_err(dev, "DT node has no dma-ranges\n");
+		return -EINVAL;
+	}
+
+	ret = of_property_read_variable_u64_array(pcie->np, "brcm,scb-sizes", pcie->memc_size, 1,
+						  PCIE_BRCM_MAX_MEMC);
+
+	if (ret <= 0) {
+		/* Make an educated guess */
+		pcie->num_memc = 1;
+		pcie->memc_size[0] = 1 << fls64(size - 1);
+	} else {
+		pcie->num_memc = ret;
+	}
+
+	/* Each memc is viewed through a "port" that is a power of 2 */
+	for (i = 0, size = 0; i < pcie->num_memc; i++)
+		size += pcie->memc_size[i];
+
+	/* System memory starts at this address in PCIe-space */
+	*rc_bar2_offset = lowest_pcie_addr;
+	/* The sum of all memc views must also be a power of 2 */
+	*rc_bar2_size = 1ULL << fls64(size - 1);
 
 	/*
 	 * We validate the inbound memory view even though we should trust
@@ -782,12 +809,11 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	void __iomem *base = pcie->base;
 	struct device *dev = pcie->dev;
 	struct resource_entry *entry;
-	unsigned int scb_size_val;
 	bool ssc_good = false;
 	struct resource *res;
 	int num_out_wins = 0;
 	u16 nlw, cls, lnksta;
-	int i, ret;
+	int i, ret, memc;
 	u32 tmp, aspm_support;
 
 	/* Reset the bridge */
@@ -824,11 +850,17 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	writel(upper_32_bits(rc_bar2_offset),
 	       base + PCIE_MISC_RC_BAR2_CONFIG_HI);
 
-	scb_size_val = rc_bar2_size ?
-		       ilog2(rc_bar2_size) - 15 : 0xf; /* 0xf is 1GB */
 	tmp = readl(base + PCIE_MISC_MISC_CTRL);
-	u32p_replace_bits(&tmp, scb_size_val,
-			  PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+	for (memc = 0; memc < pcie->num_memc; memc++) {
+		u32 scb_size_val = ilog2(pcie->memc_size[memc]) - 15;
+
+		if (memc == 0)
+			u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+		else if (memc == 1)
+			u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK);
+		else if (memc == 2)
+			u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK);
+	}
 	writel(tmp, base + PCIE_MISC_MISC_CTRL);
 
 	/*
Signature · 1 line
2.17.1
#17 lore raw

Re: [PATCH v5 09/12] PCI: brcmstb: Set internal memory viewport sizes

Quoted 5 lines · Jim Quinlan

BrcmSTB PCIe controllers are intimately connected to the memory controller(s) on the SOC. There is a "viewport" for each memory controller that allows inbound accesses to CPU memory. Each viewport's size must be set to a power of two, and that size must be equal to or larger than the amount of memory each controller supports.

This describes some requirements, but doesn't actually say what this patch *does*.

I *think* it reads the viewport sizes from the "brcm,scb-sizes" DT property instead of computing something from "dma-ranges". Looks like it also adds support for SCB1 and SCB2.

Those seem interesting, but don't really come through in the subject or even the commit log.

If I understand correctly, this is all for DMA ("inbound accesses to CPU memory"). I think it would be worth mentioning "DMA", since that's the common term for this.

Quoted 132 lines
---
 drivers/pci/controller/pcie-brcmstb.c | 68 ++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 18 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +50 -18
index 9189406fd35c..39f77709c6a2 100644
@@ -57,6 +57,8 @@
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_MASK	0x300000
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_128		0x0
 #define  PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK		0xf8000000
+#define  PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK		0x07c00000
+#define  PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK		0x0000001f
 
 #define PCIE_MISC_CPU_2_PCIE_MEM_WIN0_LO		0x400c
 #define PCIE_MEM_WIN0_LO(win)	\
@@ -154,6 +156,7 @@
 #define SSC_STATUS_OFFSET		0x1
 #define SSC_STATUS_SSC_MASK		0x400
 #define SSC_STATUS_PLL_LOCK_MASK	0x800
+#define PCIE_BRCM_MAX_MEMC		3
 
 #define IDX_ADDR(pcie)			(pcie->reg_offsets[EXT_CFG_INDEX])
 #define DATA_ADDR(pcie)			(pcie->reg_offsets[EXT_CFG_DATA])
@@ -260,6 +263,8 @@ struct brcm_pcie {
 	const int		*reg_field_info;
 	enum pcie_type		type;
 	struct reset_control	*rescal;
+	int			num_memc;
+	u64			memc_size[PCIE_BRCM_MAX_MEMC];
 };
 
 /*
@@ -715,22 +720,44 @@ static inline int brcm_pcie_get_rc_bar2_size_and_offset(struct brcm_pcie *pcie,
 							u64 *rc_bar2_offset)
 {
 	struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
-	struct device *dev = pcie->dev;
 	struct resource_entry *entry;
+	struct device *dev = pcie->dev;
+	u64 lowest_pcie_addr = ~(u64)0;
+	int ret, i = 0;
+	u64 size = 0;
 
-	entry = resource_list_first_type(&bridge->dma_ranges, IORESOURCE_MEM);
-	if (!entry)
-		return -ENODEV;
+	resource_list_for_each_entry(entry, &bridge->dma_ranges) {
+		u64 pcie_beg = entry->res->start - entry->offset;
 
+		size += entry->res->end - entry->res->start + 1;
+		if (pcie_beg < lowest_pcie_addr)
+			lowest_pcie_addr = pcie_beg;
+	}
 
-	/*
-	 * The controller expects the inbound window offset to be calculated as
-	 * the difference between PCIe's address space and CPU's. The offset
-	 * provided by the firmware is calculated the opposite way, so we
-	 * negate it.
-	 */
-	*rc_bar2_offset = -entry->offset;
-	*rc_bar2_size = 1ULL << fls64(entry->res->end - entry->res->start);
+	if (lowest_pcie_addr == ~(u64)0) {
+		dev_err(dev, "DT node has no dma-ranges\n");
+		return -EINVAL;
+	}
+
+	ret = of_property_read_variable_u64_array(pcie->np, "brcm,scb-sizes", pcie->memc_size, 1,
+						  PCIE_BRCM_MAX_MEMC);
+
+	if (ret <= 0) {
+		/* Make an educated guess */
+		pcie->num_memc = 1;
+		pcie->memc_size[0] = 1 << fls64(size - 1);
+	} else {
+		pcie->num_memc = ret;
+	}
+
+	/* Each memc is viewed through a "port" that is a power of 2 */
+	for (i = 0, size = 0; i < pcie->num_memc; i++)
+		size += pcie->memc_size[i];
+
+	/* System memory starts at this address in PCIe-space */
+	*rc_bar2_offset = lowest_pcie_addr;
+	/* The sum of all memc views must also be a power of 2 */
+	*rc_bar2_size = 1ULL << fls64(size - 1);
 
 	/*
 	 * We validate the inbound memory view even though we should trust
@@ -782,12 +809,11 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	void __iomem *base = pcie->base;
 	struct device *dev = pcie->dev;
 	struct resource_entry *entry;
-	unsigned int scb_size_val;
 	bool ssc_good = false;
 	struct resource *res;
 	int num_out_wins = 0;
 	u16 nlw, cls, lnksta;
-	int i, ret;
+	int i, ret, memc;
 	u32 tmp, aspm_support;
 
 	/* Reset the bridge */
@@ -824,11 +850,17 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	writel(upper_32_bits(rc_bar2_offset),
 	       base + PCIE_MISC_RC_BAR2_CONFIG_HI);
 
-	scb_size_val = rc_bar2_size ?
-		       ilog2(rc_bar2_size) - 15 : 0xf; /* 0xf is 1GB */
 	tmp = readl(base + PCIE_MISC_MISC_CTRL);
-	u32p_replace_bits(&tmp, scb_size_val,
-			  PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+	for (memc = 0; memc < pcie->num_memc; memc++) {
+		u32 scb_size_val = ilog2(pcie->memc_size[memc]) - 15;
+
+		if (memc == 0)
+			u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+		else if (memc == 1)
+			u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK);
+		else if (memc == 2)
+			u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK);
+	}
 	writel(tmp, base + PCIE_MISC_MISC_CTRL);
 
 	/*
Signature · 1 line
2.17.1
#18 lore raw

Re: [PATCH v5 09/12] PCI: brcmstb: Set internal memory viewport sizes

Hello Bjorn,

Quoted 15 lines · Bjorn Helgaas
Quoted 5 lines · Jim Quinlan

BrcmSTB PCIe controllers are intimately connected to the memory controller(s) on the SOC. There is a "viewport" for each memory controller that allows inbound accesses to CPU memory. Each viewport's size must be set to a power of two, and that size must be equal to or larger than the amount of memory each controller supports.

This describes some requirements, but doesn't actually say what this patch *does*.

I *think* it reads the viewport sizes from the "brcm,scb-sizes" DT property instead of computing something from "dma-ranges". Looks like it also adds support for SCB1 and SCB2.

Quoted 6 lines

Those seem interesting, but don't really come through in the subject or even the commit log.

If I understand correctly, this is all for DMA ("inbound accesses to CPU memory"). I think it would be worth mentioning "DMA", since that's the common term for this.

I have changed the commit message to the text below. Please let me know if it requires more work
Thanks, Jim

PCI: brcmstb: Set internal memory DMA viewport sizes

BrcmSTB PCIe controllers are intimately connected to the memory controller(s) on the SOC. There is a "viewport" for each memory controller that allows inbound DMA acceses to CPU memory. Each viewport's size must be set to a power of two, and that size must be equal to or larger than the amount of memory each controller supports. Unfortunately the viewport sizes cannot be ascertained from the "dma-ranges" property so they have their own property, "brcm,scb-sizes".

There may be one to three memory controllers; they are indicated by the term SCBi. Each controller has a base region and an optional extension region. In physical memory, the base and extension regions are not adjacent, but in PCIe-space they are. Further, the 1-3 viewports are also adjacent in PCIe-space.

The SCB settings work in conjunction with the "dma-ranges' offsets to enable non-identity mappings between system memory and PCIe space.

Quoted 134 lines
Quoted 134 lines
---
 drivers/pci/controller/pcie-brcmstb.c | 68 ++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 18 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +50 -19
index 9189406fd35c..39f77709c6a2 100644
@@ -57,6 +57,8 @@
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_MASK     0x300000
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_128              0x0
 #define  PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK          0xf8000000
+#define  PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK          0x07c00000
+#define  PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK          0x0000001f
 
 #define PCIE_MISC_CPU_2_PCIE_MEM_WIN0_LO             0x400c
 #define PCIE_MEM_WIN0_LO(win)        \
@@ -154,6 +156,7 @@
 #define SSC_STATUS_OFFSET            0x1
 #define SSC_STATUS_SSC_MASK          0x400
 #define SSC_STATUS_PLL_LOCK_MASK     0x800
+#define PCIE_BRCM_MAX_MEMC           3
 
 #define IDX_ADDR(pcie)                       (pcie->reg_offsets[EXT_CFG_INDEX])
 #define DATA_ADDR(pcie)                      (pcie->reg_offsets[EXT_CFG_DATA])
@@ -260,6 +263,8 @@ struct brcm_pcie {
      const int               *reg_field_info;
      enum pcie_type          type;
      struct reset_control    *rescal;
+     int                     num_memc;
+     u64                     memc_size[PCIE_BRCM_MAX_MEMC];
 };
 
 /*
@@ -715,22 +720,44 @@ static inline int brcm_pcie_get_rc_bar2_size_and_offset(struct brcm_pcie *pcie,
                                                      u64 *rc_bar2_offset)
 {
      struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
-     struct device *dev = pcie->dev;
      struct resource_entry *entry;
+     struct device *dev = pcie->dev;
+     u64 lowest_pcie_addr = ~(u64)0;
+     int ret, i = 0;
+     u64 size = 0;
 
-     entry = resource_list_first_type(&bridge->dma_ranges, IORESOURCE_MEM);
-     if (!entry)
-             return -ENODEV;
+     resource_list_for_each_entry(entry, &bridge->dma_ranges) {
+             u64 pcie_beg = entry->res->start - entry->offset;
 
+             size += entry->res->end - entry->res->start + 1;
+             if (pcie_beg < lowest_pcie_addr)
+                     lowest_pcie_addr = pcie_beg;
+     }
 
-     /*
-      * The controller expects the inbound window offset to be calculated as
-      * the difference between PCIe's address space and CPU's. The offset
-      * provided by the firmware is calculated the opposite way, so we
-      * negate it.
-      */
-     *rc_bar2_offset = -entry->offset;
-     *rc_bar2_size = 1ULL << fls64(entry->res->end - entry->res->start);
+     if (lowest_pcie_addr == ~(u64)0) {
+             dev_err(dev, "DT node has no dma-ranges\n");
+             return -EINVAL;
+     }
+
+     ret = of_property_read_variable_u64_array(pcie->np, "brcm,scb-sizes", pcie->memc_size, 1,
+                                               PCIE_BRCM_MAX_MEMC);
+
+     if (ret <= 0) {
+             /* Make an educated guess */
+             pcie->num_memc = 1;
+             pcie->memc_size[0] = 1 << fls64(size - 1);
+     } else {
+             pcie->num_memc = ret;
+     }
+
+     /* Each memc is viewed through a "port" that is a power of 2 */
+     for (i = 0, size = 0; i < pcie->num_memc; i++)
+             size += pcie->memc_size[i];
+
+     /* System memory starts at this address in PCIe-space */
+     *rc_bar2_offset = lowest_pcie_addr;
+     /* The sum of all memc views must also be a power of 2 */
+     *rc_bar2_size = 1ULL << fls64(size - 1);
 
      /*
       * We validate the inbound memory view even though we should trust
@@ -782,12 +809,11 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
      void __iomem *base = pcie->base;
      struct device *dev = pcie->dev;
      struct resource_entry *entry;
-     unsigned int scb_size_val;
      bool ssc_good = false;
      struct resource *res;
      int num_out_wins = 0;
      u16 nlw, cls, lnksta;
-     int i, ret;
+     int i, ret, memc;
      u32 tmp, aspm_support;
 
      /* Reset the bridge */
@@ -824,11 +850,17 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
      writel(upper_32_bits(rc_bar2_offset),
             base + PCIE_MISC_RC_BAR2_CONFIG_HI);
 
-     scb_size_val = rc_bar2_size ?
-                    ilog2(rc_bar2_size) - 15 : 0xf; /* 0xf is 1GB */
      tmp = readl(base + PCIE_MISC_MISC_CTRL);
-     u32p_replace_bits(&tmp, scb_size_val,
-                       PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+     for (memc = 0; memc < pcie->num_memc; memc++) {
+             u32 scb_size_val = ilog2(pcie->memc_size[memc]) - 15;
+
+             if (memc == 0)
+                     u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+             else if (memc == 1)
+                     u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK);
+             else if (memc == 2)
+                     u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK);
+     }
      writel(tmp, base + PCIE_MISC_MISC_CTRL);
 
      /*
--

2.17.1

#19 lore raw

Re: [PATCH v5 09/12] PCI: brcmstb: Set internal memory viewport sizes

Quoted 26 lines · Jim Quinlan

Hello Bjorn,

Quoted 15 lines · Bjorn Helgaas
Quoted 5 lines · Jim Quinlan

BrcmSTB PCIe controllers are intimately connected to the memory controller(s) on the SOC. There is a "viewport" for each memory controller that allows inbound accesses to CPU memory. Each viewport's size must be set to a power of two, and that size must be equal to or larger than the amount of memory each controller supports.

This describes some requirements, but doesn't actually say what this patch *does*.

I *think* it reads the viewport sizes from the "brcm,scb-sizes" DT property instead of computing something from "dma-ranges". Looks like it also adds support for SCB1 and SCB2.

Quoted 6 lines

Those seem interesting, but don't really come through in the subject or even the commit log.

If I understand correctly, this is all for DMA ("inbound accesses to CPU memory"). I think it would be worth mentioning "DMA", since that's the common term for this.

Quoted 5 lines

I have changed the commit message to the text below. Please let me know if it requires more work
Thanks, Jim

PCI: brcmstb: Set internal memory DMA viewport sizes

Did you not set the viewport sizes before?

Quoted 7 lines

BrcmSTB PCIe controllers are intimately connected to the memory controller(s) on the SOC. There is a "viewport" for each memory controller that allows inbound DMA acceses to CPU memory. Each viewport's size must be set to a power of two, and that size must be equal to or larger than the amount of memory each controller supports. Unfortunately the viewport sizes cannot be ascertained from the "dma-ranges" property so they have their own property, "brcm,scb-sizes".

s/inbound DMA acceses to CPU memory/DMA/

"Accesses" is redundant since the "A" in "DMA" stands for "access". I'm not sure "inbound" adds anything and might confuse since DMA may be either a read or write of CPU memory.

I assume *all* drivers need to know the address and size of regions in "dma-ranges". Is there something special about this device that means it needs something different?

I guess it's the base/extension split? That couldn't be described as two separate DMA ranges?

Could/should the new property have a name somehow related to "dma-ranges"?

Should "dma-ranges" be documented in Documentation/devicetree/bindings/pci/pci.txt instead of the individual device bindings?

Quoted 8 lines

There may be one to three memory controllers; they are indicated by the term SCBi. Each controller has a base region and an optional extension region. In physical memory, the base and extension regions are not adjacent, but in PCIe-space they are. Further, the 1-3 viewports are also adjacent in PCIe-space.

The SCB settings work in conjunction with the "dma-ranges' offsets to enable non-identity mappings between system memory and PCIe space.

s/ranges'/ranges"/ (mismatched quotes)

This describes the hardware, but still doesn't actually say what this patch *does*.

If I'm a user, why do I want this patch? Does it fix something that didn't work before? Does it increase the amount of DMA-able memory?

What does this mean in terms of backwards compatibility with old DTs? Does this work with old DTs that don't have "brcm,scb-sizes"? Maybe this is all related to specific devices that weren't supported before, so there *are* no old DTs for them? I can't tell from the binding update or the patch that this is related to specific devices.

Quoted 132 lines
Quoted 132 lines
Quoted 132 lines
---
 drivers/pci/controller/pcie-brcmstb.c | 68 ++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 18 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +50 -19
index 9189406fd35c..39f77709c6a2 100644
@@ -57,6 +57,8 @@
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_MASK     0x300000
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_128              0x0
 #define  PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK          0xf8000000
+#define  PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK          0x07c00000
+#define  PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK          0x0000001f
 
 #define PCIE_MISC_CPU_2_PCIE_MEM_WIN0_LO             0x400c
 #define PCIE_MEM_WIN0_LO(win)        \
@@ -154,6 +156,7 @@
 #define SSC_STATUS_OFFSET            0x1
 #define SSC_STATUS_SSC_MASK          0x400
 #define SSC_STATUS_PLL_LOCK_MASK     0x800
+#define PCIE_BRCM_MAX_MEMC           3
 
 #define IDX_ADDR(pcie)                       (pcie->reg_offsets[EXT_CFG_INDEX])
 #define DATA_ADDR(pcie)                      (pcie->reg_offsets[EXT_CFG_DATA])
@@ -260,6 +263,8 @@ struct brcm_pcie {
      const int               *reg_field_info;
      enum pcie_type          type;
      struct reset_control    *rescal;
+     int                     num_memc;
+     u64                     memc_size[PCIE_BRCM_MAX_MEMC];
 };
 
 /*
@@ -715,22 +720,44 @@ static inline int brcm_pcie_get_rc_bar2_size_and_offset(struct brcm_pcie *pcie,
                                                      u64 *rc_bar2_offset)
 {
      struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
-     struct device *dev = pcie->dev;
      struct resource_entry *entry;
+     struct device *dev = pcie->dev;
+     u64 lowest_pcie_addr = ~(u64)0;
+     int ret, i = 0;
+     u64 size = 0;
 
-     entry = resource_list_first_type(&bridge->dma_ranges, IORESOURCE_MEM);
-     if (!entry)
-             return -ENODEV;
+     resource_list_for_each_entry(entry, &bridge->dma_ranges) {
+             u64 pcie_beg = entry->res->start - entry->offset;
 
+             size += entry->res->end - entry->res->start + 1;
+             if (pcie_beg < lowest_pcie_addr)
+                     lowest_pcie_addr = pcie_beg;
+     }
 
-     /*
-      * The controller expects the inbound window offset to be calculated as
-      * the difference between PCIe's address space and CPU's. The offset
-      * provided by the firmware is calculated the opposite way, so we
-      * negate it.
-      */
-     *rc_bar2_offset = -entry->offset;
-     *rc_bar2_size = 1ULL << fls64(entry->res->end - entry->res->start);
+     if (lowest_pcie_addr == ~(u64)0) {
+             dev_err(dev, "DT node has no dma-ranges\n");
+             return -EINVAL;
+     }
+
+     ret = of_property_read_variable_u64_array(pcie->np, "brcm,scb-sizes", pcie->memc_size, 1,
+                                               PCIE_BRCM_MAX_MEMC);
+
+     if (ret <= 0) {
+             /* Make an educated guess */
+             pcie->num_memc = 1;
+             pcie->memc_size[0] = 1 << fls64(size - 1);
+     } else {
+             pcie->num_memc = ret;
+     }
+
+     /* Each memc is viewed through a "port" that is a power of 2 */
+     for (i = 0, size = 0; i < pcie->num_memc; i++)
+             size += pcie->memc_size[i];
+
+     /* System memory starts at this address in PCIe-space */
+     *rc_bar2_offset = lowest_pcie_addr;
+     /* The sum of all memc views must also be a power of 2 */
+     *rc_bar2_size = 1ULL << fls64(size - 1);
 
      /*
       * We validate the inbound memory view even though we should trust
@@ -782,12 +809,11 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
      void __iomem *base = pcie->base;
      struct device *dev = pcie->dev;
      struct resource_entry *entry;
-     unsigned int scb_size_val;
      bool ssc_good = false;
      struct resource *res;
      int num_out_wins = 0;
      u16 nlw, cls, lnksta;
-     int i, ret;
+     int i, ret, memc;
      u32 tmp, aspm_support;
 
      /* Reset the bridge */
@@ -824,11 +850,17 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
      writel(upper_32_bits(rc_bar2_offset),
             base + PCIE_MISC_RC_BAR2_CONFIG_HI);
 
-     scb_size_val = rc_bar2_size ?
-                    ilog2(rc_bar2_size) - 15 : 0xf; /* 0xf is 1GB */
      tmp = readl(base + PCIE_MISC_MISC_CTRL);
-     u32p_replace_bits(&tmp, scb_size_val,
-                       PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+     for (memc = 0; memc < pcie->num_memc; memc++) {
+             u32 scb_size_val = ilog2(pcie->memc_size[memc]) - 15;
+
+             if (memc == 0)
+                     u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+             else if (memc == 1)
+                     u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK);
+             else if (memc == 2)
+                     u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK);
+     }
      writel(tmp, base + PCIE_MISC_MISC_CTRL);
 
      /*
--

2.17.1

#20 lore raw

Re: [PATCH v5 09/12] PCI: brcmstb: Set internal memory viewport sizes

Hello Bjorn,

Quoted 28 lines · Bjorn Helgaas
Quoted 26 lines · Jim Quinlan

Hello Bjorn,

Quoted 21 lines · Bjorn Helgaas
Quoted 5 lines · Jim Quinlan

BrcmSTB PCIe controllers are intimately connected to the memory controller(s) on the SOC. There is a "viewport" for each memory controller that allows inbound accesses to CPU memory. Each viewport's size must be set to a power of two, and that size must be equal to or larger than the amount of memory each controller supports.

This describes some requirements, but doesn't actually say what this patch *does*.

I *think* it reads the viewport sizes from the "brcm,scb-sizes" DT property instead of computing something from "dma-ranges". Looks like it also adds support for SCB1 and SCB2.

Those seem interesting, but don't really come through in the subject or even the commit log.

If I understand correctly, this is all for DMA ("inbound accesses to CPU memory"). I think it would be worth mentioning "DMA", since that's the common term for this.

Quoted 7 lines
Quoted 5 lines

I have changed the commit message to the text below. Please let me know if it requires more work
Thanks, Jim

PCI: brcmstb: Set internal memory DMA viewport sizes

Did you not set the viewport sizes before?

Only for SCB0, and that was set from the size of the first and only dma-range region. This was enough for the Raspberry Pi to work but it cannot handle BrcmSTB SOCs requiring more than one dma-range region.

Quoted 11 lines
Quoted 8 lines

BrcmSTB PCIe controllers are intimately connected to the memory controller(s) on the SOC. There is a "viewport" for each memory controller that allows inbound DMA acceses to CPU memory. Each viewport's size must be set to a power of two, and that size must be equal to or larger than the amount of memory each controller supports. Unfortunately the viewport sizes cannot be ascertained from the "dma-ranges" property so they have their own property, "brcm,scb-sizes".

s/inbound DMA acceses to CPU memory/DMA/

Quoted 7 lines

"Accesses" is redundant since the "A" in "DMA" stands for "access". I'm not sure "inbound" adds anything and might confuse since DMA may be either a read or write of CPU memory.

I assume *all* drivers need to know the address and size of regions in "dma-ranges". Is there something special about this device that means it needs something different?

All previous Linux devices required at most one dma-range (to be precise, there could be multiple dma-ranges but they had to have the same offset). This device may have up to six dma-ranges, most of them having unique offsets. This is explained in a reference given in my cover letter under v1:
https://lore.kernel.org/linux-arm-kernel/[email protected]/

Quoted 3 lines

I guess it's the base/extension split? That couldn't be described as two separate DMA ranges?

Using just dma-ranges one cannot tell the difference between

dma-range[0] -- from memc0 base, 2GB
dma-range[1] -- from memc0 extension, 1GB
/* Action: SCB0 is set to 4GB */

and

dma-range[0] -- from memc0 base 2GB dma-range[1] -- from memc1 base 1GB
/* Action: SCB0 is set to 2GB, SCB1 is set to 1GB */

Quoted 3 lines

Could/should the new property have a name somehow related to "dma-ranges"?

Even though they are related, I can't think of a different name which would be helpful. The property name "brcm,scb-sizes" describes exactly what it is. The mapping that is necessary for BrcmSTB PCIe drivers is best described by a picture and I don't think a better name is going to help anyone figure out its necessity. I will greatly enhance my commit message for this patch to describe the details you have asked about.

Quoted 4 lines

Should "dma-ranges" be documented in Documentation/devicetree/bindings/pci/pci.txt instead of the individual device bindings?

That file references
http://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf, which defines "ranges". So the "dma-ranges" should be described in the PDF file but is not.

"dma-ranges" is certainly defined and enforced in the PCI YAML
description (aren't we moving towards YAML anyway?).

Quoted 10 lines
Quoted 9 lines

There may be one to three memory controllers; they are indicated by the term SCBi. Each controller has a base region and an optional extension region. In physical memory, the base and extension regions are not adjacent, but in PCIe-space they are. Further, the 1-3 viewports are also adjacent in PCIe-space.

The SCB settings work in conjunction with the "dma-ranges' offsets to enable non-identity mappings between system memory and PCIe space.

Quoted 7 lines

s/ranges'/ranges"/ (mismatched quotes)

This describes the hardware, but still doesn't actually say what this patch *does*.

If I'm a user, why do I want this patch? Does it fix something that didn't work before? Does it increase the amount of DMA-able memory?

BrcmSTB SOCs, with the exception of the simple memory configuration of the Raspberry Pi, will not work without this patch. The RPI sets the SCB0 size which is enough for it to function, but this patch is needed for other Broadcom STB SOCs.

Quoted 2 lines

What does this mean in terms of backwards compatibility with old DTs?

Should be backwards compatible -- there is only the RPI to worry about and I hope at some point Nicolas can send a "Tested-by" to confirm.

Quoted 1 line

Does this work with old DTs that don't have "brcm,scb-sizes"?

It should work for RPI.

Quoted 3 lines

Maybe
this is all related to specific devices that weren't supported before, so there *are* no old DTs for them?

There is only the RPI DT and they do not use "brcm,scb-sizes".. I've been trying to upstream this driver for ~3 years and it always got NACKed because of this DMA mapping issue. Now that the OF system parses the dma-ranges all the way up from an EP -- with possibly no DT node -- up to the PCIe host controller, the solution in this patchset might have a chance.

Quoted 2 lines

I can't tell from the binding
update or the patch that this is related to specific devices.

Fair enough, I will describe this in the next rev.

Thanks,
Jim Quinlan
Broadcom STB

Quoted 133 lines
Quoted 133 lines
Quoted 133 lines
Quoted 133 lines
---
 drivers/pci/controller/pcie-brcmstb.c | 68 ++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 18 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +50 -19
index 9189406fd35c..39f77709c6a2 100644
@@ -57,6 +57,8 @@
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_MASK     0x300000
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_128              0x0
 #define  PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK          0xf8000000
+#define  PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK          0x07c00000
+#define  PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK          0x0000001f
 
 #define PCIE_MISC_CPU_2_PCIE_MEM_WIN0_LO             0x400c
 #define PCIE_MEM_WIN0_LO(win)        \
@@ -154,6 +156,7 @@
 #define SSC_STATUS_OFFSET            0x1
 #define SSC_STATUS_SSC_MASK          0x400
 #define SSC_STATUS_PLL_LOCK_MASK     0x800
+#define PCIE_BRCM_MAX_MEMC           3
 
 #define IDX_ADDR(pcie)                       (pcie->reg_offsets[EXT_CFG_INDEX])
 #define DATA_ADDR(pcie)                      (pcie->reg_offsets[EXT_CFG_DATA])
@@ -260,6 +263,8 @@ struct brcm_pcie {
      const int               *reg_field_info;
      enum pcie_type          type;
      struct reset_control    *rescal;
+     int                     num_memc;
+     u64                     memc_size[PCIE_BRCM_MAX_MEMC];
 };
 
 /*
@@ -715,22 +720,44 @@ static inline int brcm_pcie_get_rc_bar2_size_and_offset(struct brcm_pcie *pcie,
                                                      u64 *rc_bar2_offset)
 {
      struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
-     struct device *dev = pcie->dev;
      struct resource_entry *entry;
+     struct device *dev = pcie->dev;
+     u64 lowest_pcie_addr = ~(u64)0;
+     int ret, i = 0;
+     u64 size = 0;
 
-     entry = resource_list_first_type(&bridge->dma_ranges, IORESOURCE_MEM);
-     if (!entry)
-             return -ENODEV;
+     resource_list_for_each_entry(entry, &bridge->dma_ranges) {
+             u64 pcie_beg = entry->res->start - entry->offset;
 
+             size += entry->res->end - entry->res->start + 1;
+             if (pcie_beg < lowest_pcie_addr)
+                     lowest_pcie_addr = pcie_beg;
+     }
 
-     /*
-      * The controller expects the inbound window offset to be calculated as
-      * the difference between PCIe's address space and CPU's. The offset
-      * provided by the firmware is calculated the opposite way, so we
-      * negate it.
-      */
-     *rc_bar2_offset = -entry->offset;
-     *rc_bar2_size = 1ULL << fls64(entry->res->end - entry->res->start);
+     if (lowest_pcie_addr == ~(u64)0) {
+             dev_err(dev, "DT node has no dma-ranges\n");
+             return -EINVAL;
+     }
+
+     ret = of_property_read_variable_u64_array(pcie->np, "brcm,scb-sizes", pcie->memc_size, 1,
+                                               PCIE_BRCM_MAX_MEMC);
+
+     if (ret <= 0) {
+             /* Make an educated guess */
+             pcie->num_memc = 1;
+             pcie->memc_size[0] = 1 << fls64(size - 1);
+     } else {
+             pcie->num_memc = ret;
+     }
+
+     /* Each memc is viewed through a "port" that is a power of 2 */
+     for (i = 0, size = 0; i < pcie->num_memc; i++)
+             size += pcie->memc_size[i];
+
+     /* System memory starts at this address in PCIe-space */
+     *rc_bar2_offset = lowest_pcie_addr;
+     /* The sum of all memc views must also be a power of 2 */
+     *rc_bar2_size = 1ULL << fls64(size - 1);
 
      /*
       * We validate the inbound memory view even though we should trust
@@ -782,12 +809,11 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
      void __iomem *base = pcie->base;
      struct device *dev = pcie->dev;
      struct resource_entry *entry;
-     unsigned int scb_size_val;
      bool ssc_good = false;
      struct resource *res;
      int num_out_wins = 0;
      u16 nlw, cls, lnksta;
-     int i, ret;
+     int i, ret, memc;
      u32 tmp, aspm_support;
 
      /* Reset the bridge */
@@ -824,11 +850,17 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
      writel(upper_32_bits(rc_bar2_offset),
             base + PCIE_MISC_RC_BAR2_CONFIG_HI);
 
-     scb_size_val = rc_bar2_size ?
-                    ilog2(rc_bar2_size) - 15 : 0xf; /* 0xf is 1GB */
      tmp = readl(base + PCIE_MISC_MISC_CTRL);
-     u32p_replace_bits(&tmp, scb_size_val,
-                       PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+     for (memc = 0; memc < pcie->num_memc; memc++) {
+             u32 scb_size_val = ilog2(pcie->memc_size[memc]) - 15;
+
+             if (memc == 0)
+                     u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK);
+             else if (memc == 1)
+                     u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK);
+             else if (memc == 2)
+                     u32p_replace_bits(&tmp, scb_size_val, PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK);
+     }
      writel(tmp, base + PCIE_MISC_MISC_CTRL);
 
      /*
--

2.17.1

#21 lore raw

PATCH v5 10/12 PCI: brcmstb: Accommodate MSI for older chips

In reply to Jim Quinlan

From: Jim Quinlan <[email protected]>

Older BrcmSTB chips do not have a separate register for MSI interrupts; the MSIs are in a register that also contains unrelated interrupts. In addition, the interrupts lie in bits [31..24] for these legacy chips. This commit provides common code for both legacy and non-legacy MSI interrupt registers.

---
 drivers/pci/controller/pcie-brcmstb.c | 71 +++++++++++++++++++--------
 1 file changed, 50 insertions(+), 21 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +50 -21
index 39f77709c6a2..29fad717f619 100644
@@ -82,7 +82,8 @@
 #define PCIE_MISC_MSI_BAR_CONFIG_HI			0x4048
 
 #define PCIE_MISC_MSI_DATA_CONFIG			0x404c
-#define  PCIE_MISC_MSI_DATA_CONFIG_VAL			0xffe06540
+#define  PCIE_MISC_MSI_DATA_CONFIG_VAL_32		0xffe06540
+#define  PCIE_MISC_MSI_DATA_CONFIG_VAL_8		0xfff86540
 
 #define PCIE_MISC_PCIE_CTRL				0x4064
 #define  PCIE_MISC_PCIE_CTRL_PCIE_L23_REQUEST_MASK	0x1
@@ -94,6 +95,9 @@
 #define  PCIE_MISC_PCIE_STATUS_PCIE_PHYLINKUP_MASK	0x10
 #define  PCIE_MISC_PCIE_STATUS_PCIE_LINK_IN_L23_MASK	0x40
 
+#define PCIE_MISC_REVISION				0x406c
+#define  BRCM_PCIE_HW_REV_33				0x0303
+
 #define PCIE_MISC_CPU_2_PCIE_MEM_WIN0_BASE_LIMIT		0x4070
 #define  PCIE_MISC_CPU_2_PCIE_MEM_WIN0_BASE_LIMIT_LIMIT_MASK	0xfff00000
 #define  PCIE_MISC_CPU_2_PCIE_MEM_WIN0_BASE_LIMIT_BASE_MASK	0xfff0
@@ -114,10 +118,14 @@
 #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK	0x2
 #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK		0x08000000
 
-#define PCIE_MSI_INTR2_STATUS				0x4500
-#define PCIE_MSI_INTR2_CLR				0x4508
-#define PCIE_MSI_INTR2_MASK_SET				0x4510
-#define PCIE_MSI_INTR2_MASK_CLR				0x4514
+
+#define PCIE_INTR2_CPU_BASE		0x4300
+#define PCIE_MSI_INTR2_BASE		0x4500
+/* Offsets from PCIE_INTR2_CPU_BASE and PCIE_MSI_INTR2_BASE */
+#define  MSI_INT_STATUS			0x0
+#define  MSI_INT_CLR			0x8
+#define  MSI_INT_MASK_SET		0x10
+#define  MSI_INT_MASK_CLR		0x14
 
 #define PCIE_EXT_CFG_DATA				0x8000
 
@@ -132,6 +140,8 @@
 /* PCIe parameters */
 #define BRCM_NUM_PCIE_OUT_WINS		0x4
 #define BRCM_INT_PCI_MSI_NR		32
+#define BRCM_INT_PCI_MSI_LEGACY_NR	8
+#define BRCM_INT_PCI_MSI_SHIFT		0
 
 /* MSI target adresses */
 #define BRCM_MSI_TARGET_ADDR_LT_4GB	0x0fffffffcULL
@@ -246,6 +256,12 @@ struct brcm_msi {
 	int			irq;
 	/* used indicates which MSI interrupts have been alloc'd */
 	unsigned long		used;
+	bool			legacy;
+	/* Some chips have MSIs in bits [31..24] of a shared register. */
+	int			legacy_shift;
+	int			nr; /* No. of MSI available, depends on chip */
+	/* This is the base pointer for interrupt status/set/clr regs */
+	void __iomem		*intr_base;
 };
 
 /* Internal PCIe Host Controller Information.*/
@@ -265,6 +281,7 @@ struct brcm_pcie {
 	struct reset_control	*rescal;
 	int			num_memc;
 	u64			memc_size[PCIE_BRCM_MAX_MEMC];
+	u32			hw_rev;
 };
 
 /*
@@ -455,8 +472,10 @@ static void brcm_pcie_msi_isr(struct irq_desc *desc)
 	msi = irq_desc_get_handler_data(desc);
 	dev = msi->dev;
 
-	status = readl(msi->base + PCIE_MSI_INTR2_STATUS);
-	for_each_set_bit(bit, &status, BRCM_INT_PCI_MSI_NR) {
+	status = readl(msi->intr_base + MSI_INT_STATUS);
+	status >>= msi->legacy_shift;
+
+	for_each_set_bit(bit, &status, msi->nr) {
 		virq = irq_find_mapping(msi->inner_domain, bit);
 		if (virq)
 			generic_handle_irq(virq);
@@ -473,7 +492,7 @@ static void brcm_msi_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 
 	msg->address_lo = lower_32_bits(msi->target_addr);
 	msg->address_hi = upper_32_bits(msi->target_addr);
-	msg->data = (0xffff & PCIE_MISC_MSI_DATA_CONFIG_VAL) | data->hwirq;
+	msg->data = (0xffff & PCIE_MISC_MSI_DATA_CONFIG_VAL_32) | data->hwirq;
 }
 
 static int brcm_msi_set_affinity(struct irq_data *irq_data,
@@ -485,8 +504,9 @@ static int brcm_msi_set_affinity(struct irq_data *irq_data,
 static void brcm_msi_ack_irq(struct irq_data *data)
 {
 	struct brcm_msi *msi = irq_data_get_irq_chip_data(data);
+	const int shift_amt = data->hwirq + msi->legacy_shift;
 
-	writel(1 << data->hwirq, msi->base + PCIE_MSI_INTR2_CLR);
+	writel(1 << shift_amt, msi->intr_base + MSI_INT_CLR);
 }
 
 
@@ -502,7 +522,7 @@ static int brcm_msi_alloc(struct brcm_msi *msi)
 	int hwirq;
 
 	mutex_lock(&msi->lock);
-	hwirq = bitmap_find_free_region(&msi->used, BRCM_INT_PCI_MSI_NR, 0);
+	hwirq = bitmap_find_free_region(&msi->used, msi->nr, 0);
 	mutex_unlock(&msi->lock);
 
 	return hwirq;
@@ -551,8 +571,7 @@ static int brcm_allocate_domains(struct brcm_msi *msi)
 	struct fwnode_handle *fwnode = of_node_to_fwnode(msi->np);
 	struct device *dev = msi->dev;
 
-	msi->inner_domain = irq_domain_add_linear(NULL, BRCM_INT_PCI_MSI_NR,
-						  &msi_domain_ops, msi);
+	msi->inner_domain = irq_domain_add_linear(NULL, msi->nr, &msi_domain_ops, msi);
 	if (!msi->inner_domain) {
 		dev_err(dev, "failed to create IRQ domain\n");
 		return -ENOMEM;
@@ -589,7 +608,10 @@ static void brcm_msi_remove(struct brcm_pcie *pcie)
 
 static void brcm_msi_set_regs(struct brcm_msi *msi)
 {
-	writel(0xffffffff, msi->base + PCIE_MSI_INTR2_MASK_CLR);
+	u32 val = __GENMASK(31, msi->legacy_shift);
+
+	writel(val, msi->intr_base + MSI_INT_MASK_CLR);
+	writel(val, msi->intr_base + MSI_INT_CLR);
 
 	/*
 	 * The 0 bit of PCIE_MISC_MSI_BAR_CONFIG_LO is repurposed to MSI
@@ -600,8 +622,8 @@ static void brcm_msi_set_regs(struct brcm_msi *msi)
 	writel(upper_32_bits(msi->target_addr),
 	       msi->base + PCIE_MISC_MSI_BAR_CONFIG_HI);
 
-	writel(PCIE_MISC_MSI_DATA_CONFIG_VAL,
-	       msi->base + PCIE_MISC_MSI_DATA_CONFIG);
+	val = msi->legacy ? PCIE_MISC_MSI_DATA_CONFIG_VAL_8 : PCIE_MISC_MSI_DATA_CONFIG_VAL_32;
+	writel(val, msi->base + PCIE_MISC_MSI_DATA_CONFIG);
 }
 
 static int brcm_pcie_enable_msi(struct brcm_pcie *pcie)
@@ -626,6 +648,17 @@ static int brcm_pcie_enable_msi(struct brcm_pcie *pcie)
 	msi->np = pcie->np;
 	msi->target_addr = pcie->msi_target_addr;
 	msi->irq = irq;
+	msi->legacy = pcie->hw_rev < BRCM_PCIE_HW_REV_33;
+
+	if (msi->legacy) {
+		msi->intr_base = msi->base + PCIE_INTR2_CPU_BASE;
+		msi->nr = BRCM_INT_PCI_MSI_LEGACY_NR;
+		msi->legacy_shift = 24;
+	} else {
+		msi->intr_base = msi->base + PCIE_MSI_INTR2_BASE;
+		msi->nr = BRCM_INT_PCI_MSI_NR;
+		msi->legacy_shift = 0;
+	}
 
 	ret = brcm_allocate_domains(msi);
 	if (ret)
@@ -885,12 +918,6 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	tmp &= ~PCIE_MISC_RC_BAR3_CONFIG_LO_SIZE_MASK;
 	writel(tmp, base + PCIE_MISC_RC_BAR3_CONFIG_LO);
 
-	/* Mask all interrupts since we are not handling any yet */
-	writel(0xffffffff, pcie->base + PCIE_MSI_INTR2_MASK_SET);
-
-	/* clear any interrupts we find on boot */
-	writel(0xffffffff, pcie->base + PCIE_MSI_INTR2_CLR);
-
 	if (pcie->gen)
 		brcm_pcie_set_gen(pcie, pcie->gen);
 
@@ -1233,6 +1260,8 @@ static int brcm_pcie_probe(struct platform_device *pdev)
 	if (ret)
 		goto fail;
 
+	pcie->hw_rev = readl(pcie->base + PCIE_MISC_REVISION);
+
 	msi_np = of_parse_phandle(pcie->np, "msi-parent", 0);
 	if (pci_msi_enabled() && msi_np == pcie->np) {
 		ret = brcm_pcie_enable_msi(pcie);
Signature · 1 line
2.17.1
#22 lore raw

PATCH v5 11/12 PCI: brcmstb: Set bus max burst size by chip type

In reply to Jim Quinlan

From: Jim Quinlan <[email protected]>

The proper value of the parameter SCB_MAX_BURST_SIZE varies per chip. The 2711 family requires 128B whereas other devices can employ 512. The assignment is complicated by the fact that the values for this two-bit field have different meanings;

Value Type_Generic Type_7278

     00       Reserved         128B
     01           128B         256B
     10           256B         512B

11 512B Reserved

---
 drivers/pci/controller/pcie-brcmstb.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
drivers/pci/controller/pcie-brcmstb.c +15 -4
index 29fad717f619..02b769534e53 100644
@@ -55,7 +55,7 @@
 #define  PCIE_MISC_MISC_CTRL_SCB_ACCESS_EN_MASK		0x1000
 #define  PCIE_MISC_MISC_CTRL_CFG_READ_UR_MODE_MASK	0x2000
 #define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_MASK	0x300000
-#define  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_128		0x0
+
 #define  PCIE_MISC_MISC_CTRL_SCB0_SIZE_MASK		0xf8000000
 #define  PCIE_MISC_MISC_CTRL_SCB1_SIZE_MASK		0x07c00000
 #define  PCIE_MISC_MISC_CTRL_SCB2_SIZE_MASK		0x0000001f
@@ -847,7 +847,7 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	int num_out_wins = 0;
 	u16 nlw, cls, lnksta;
 	int i, ret, memc;
-	u32 tmp, aspm_support;
+	u32 tmp, burst, aspm_support;
 
 	/* Reset the bridge */
 	brcm_pcie_bridge_sw_init_set(pcie, 1);
@@ -864,11 +864,22 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	/* Wait for SerDes to be stable */
 	usleep_range(100, 200);
 
+	/*
+	 * SCB_MAX_BURST_SIZE is a two bit field.  For GENERIC chips it
+	 * is encoded as 0=128, 1=256, 2=512, 3=Rsvd, for BCM7278 it
+	 * is encoded as 0=Rsvd, 1=128, 2=256, 3=512.
+	 */
+	if (pcie->type == BCM2711)
+		burst = 0x0; /* 128B */
+	else if (pcie->type == BCM7278)
+		burst = 0x3; /* 512 bytes */
+	else
+		burst = 0x2; /* 512 bytes */
+
 	/* Set SCB_MAX_BURST_SIZE, CFG_READ_UR_MODE, SCB_ACCESS_EN */
 	u32p_replace_bits(&tmp, 1, PCIE_MISC_MISC_CTRL_SCB_ACCESS_EN_MASK);
 	u32p_replace_bits(&tmp, 1, PCIE_MISC_MISC_CTRL_CFG_READ_UR_MODE_MASK);
-	u32p_replace_bits(&tmp, PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_128,
-			  PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_MASK);
+	u32p_replace_bits(&tmp, burst, PCIE_MISC_MISC_CTRL_MAX_BURST_SIZE_MASK);
 	writel(tmp, base + PCIE_MISC_MISC_CTRL);
 
 	ret = brcm_pcie_get_rc_bar2_size_and_offset(pcie, &rc_bar2_size,
Signature · 1 line
2.17.1
#23 lore raw

PATCH v5 12/12 PCI: brcmstb: Add bcm7211, bcm7216, bcm7445, bcm7278 to match list

In reply to Jim Quinlan

Now that the support is in place with previous commits, we add several chips that use the BrcmSTB driver.

---
 drivers/pci/controller/pcie-brcmstb.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/pci/controller/pcie-brcmstb.c +4 -0
index 02b769534e53..1437ad1a26c8 100644
@@ -1184,6 +1184,10 @@ static int brcm_pcie_remove(struct platform_device *pdev)
 
 static const struct of_device_id brcm_pcie_match[] = {
 	{ .compatible = "brcm,bcm2711-pcie", .data = &bcm2711_cfg },
+	{ .compatible = "brcm,bcm7211-pcie", .data = &generic_cfg },
+	{ .compatible = "brcm,bcm7278-pcie", .data = &bcm7278_cfg },
+	{ .compatible = "brcm,bcm7216-pcie", .data = &bcm7278_cfg },
+	{ .compatible = "brcm,bcm7445-pcie", .data = &generic_cfg },
 	{},
 };
 
Signature · 1 line
2.17.1