Skip to contentSkip to conversation

PCI: Add support for Cavium ThunderX RC and on-SoC devices.

PATCH In review David Daney 6 participants 18 messages Sep 17, 2015 – Sep 23, 2015

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover PCI: Add support for Cavium ThunderX RC and on-SoC devices.

From: David Daney <[email protected]>

The Cavium ThunderX arm64 based SoC needs a little bit of special handling for both its PCIe Root Complexes as well as on-SoC devices (which all appear as PCIe devices).

1/3 - Small change to allow SRIOV BARs to be given fixed addresses in
the header fixup.

2/3 - Add quirks to support fixed BAR addresses for all on-SoC devices,
including SRIOV BARs in the NIC.

3/3 - Add config spaces accessors to pci-host-generic driver for ThunderX RC.

This patch set depends on:

https://lkml.org/lkml/2015/9/17/799

David Daney (3):
PCI: Allow quirks to override SRIOV BARs.
PCI: Add quirks for devices found on Cavium ThunderX SoCs.
PCI: generic: Add support for Cavium ThunderX PCIe root complexes.

 .../devicetree/bindings/pci/host-generic-pci.txt   |  8 +-
 drivers/pci/host/Kconfig                           |  6 ++
 drivers/pci/host/Makefile                          |  1 +
 drivers/pci/host/pci-host-generic.c                | 29 +++++++
 drivers/pci/host/quirks-thunder.c                  | 95 ++++++++++++++++++++++
 drivers/pci/iov.c                                  |  9 +-
 6 files changed, 143 insertions(+), 5 deletions(-)
 create mode 100644 drivers/pci/host/quirks-thunder.c
Signature · 1 line
1.9.1
#1 lore raw

PATCH 1/3 PCI: Allow quirks to override SRIOV BARs.

From: David Daney <[email protected]>

Devices with fixed BARs can install BAR resources with IORESOURCE_PCI_FIXED from the header fixup. Allow this to work with the SRIOV BARs as well by testing if the BAR resource has already been set before attempting to read it from the config space.

---
 drivers/pci/iov.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/pci/iov.c +7 -2
index ee0ebff..f8a6e1e 100644
@@ -436,8 +436,13 @@ found:
 	nres = 0;
 	for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
 		res = &dev->resource[i + PCI_IOV_RESOURCES];
-		bar64 = __pci_read_base(dev, pci_bar_unknown, res,
-					pos + PCI_SRIOV_BAR + i * 4);
+		if (res->flags)
+			/* Already populated by quirks, just set bar64. */
+			bar64 = (res->flags & IORESOURCE_MEM_64) ? 1 : 0;
+		else
+			bar64 = __pci_read_base(dev, pci_bar_unknown, res,
+						pos + PCI_SRIOV_BAR + i * 4);
+
 		if (!res->flags)
 			continue;
 		if (resource_size(res) & (PAGE_SIZE - 1)) {
Signature · 1 line
1.9.1
#2 lore raw

PATCH 2/3 PCI: Add quirks for devices found on Cavium ThunderX SoCs.

In reply to David Daney

From: David Daney <[email protected]>

The on-chip devices all have fixed bars. So, fix them up.

---
 drivers/pci/host/Kconfig          |  6 +++
 drivers/pci/host/Makefile         |  1 +
 drivers/pci/host/quirks-thunder.c | 95 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 102 insertions(+)
 create mode 100644 drivers/pci/host/quirks-thunder.c
drivers/pci/host/Kconfig +6 -0
index d5e58ba..20d700d 100644
@@ -145,4 +145,10 @@ config PCIE_IPROC_BCMA
 	  Say Y here if you want to use the Broadcom iProc PCIe controller
 	  through the BCMA bus interface
 
+config PCI_QUIRKS_THUNDER
+	bool "PCI quirks for Cavium ThunderX SoCs"
+	depends on ARM64
+	help
+	  Say Y here to support ThunderX SoCs
+
 endmenu
drivers/pci/host/Makefile +1 -0
index 140d66f..3ce7456 100644
@@ -17,3 +17,4 @@ obj-$(CONFIG_PCI_VERSATILE) += pci-versatile.o
 obj-$(CONFIG_PCIE_IPROC) += pcie-iproc.o
 obj-$(CONFIG_PCIE_IPROC_PLATFORM) += pcie-iproc-platform.o
 obj-$(CONFIG_PCIE_IPROC_BCMA) += pcie-iproc-bcma.o
+obj-$(CONFIG_PCI_QUIRKS_THUNDER) += quirks-thunder.o
drivers/pci/host/quirks-thunder.c new file +95 -0

95 lines not on this page · raw message

Signature · 1 line
1.9.1
#3 lore raw

Re: [PATCH 2/3] PCI: Add quirks for devices found on Cavium ThunderX SoCs.

Quoted 6 lines · On Thursday 17 September 2015 15:41:33 David Daney wrote:

From: David Daney <[email protected]>

The on-chip devices all have fixed bars. So, fix them up.

You should be able to just mark the BARs as fixed in DT and not need this hack. If that doesn't work with current kernels, it would be better to fix the kernel to respect the flags.

Arnd

#4 lore raw

Re: [PATCH 2/3] PCI: Add quirks for devices found on Cavium ThunderX SoCs.

Quoted 9 lines · Arnd Bergmann
Quoted 6 lines · On Thursday 17 September 2015 15:41:33 David Daney wrote:

From: David Daney <[email protected]>

The on-chip devices all have fixed bars. So, fix them up.

You should be able to just mark the BARs as fixed in DT

In the case of ACPI, there is no DT. So we would need a different solution for ACPI. What would you recommend for ACPI?

Also, can you point me to the OF device tree specification where it tells how to specify PCI BAR addresses, I would especially be interested in knowing how to specify fixed SRIOV BAR addresses in the device tree.

Thanks,
David Daney

Quoted 2 lines

and not need
this hack.

Yes, it is a bit of a hack. That is why I put it in its own file, and only try to hack up PCI devices that exactly match the vendor and device ids that need fixing.

IMHO, putting infrastructure into drivers/pci/probe.c, et al. to handle this would be much more intrusive.

For the record: The PCI Enhanced Allocation (EA) capability (approved by PCI SIG on 23 October 2014) is the proper way to handle this going forward. However, this is not yet implemented in the SoCs that this patch addresses. Our plan is to implement the EA capability in the core PCI code, so that we do not need to keep adding devices to this fixup code.

David Daney

#5 lore raw

Re: [PATCH 2/3] PCI: Add quirks for devices found on Cavium ThunderX SoCs.

Quoted 6 lines · On Friday 18 September 2015 10:00:32 David Daney wrote:
Quoted 5 lines · Arnd Bergmann
Quoted 4 lines · On Thursday 17 September 2015 15:41:33 David Daney wrote:

From: David Daney <[email protected]>

The on-chip devices all have fixed bars. So, fix them up.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

You should be able to just mark the BARs as fixed in DT

In the case of ACPI, there is no DT. So we would need a different solution for ACPI. What would you recommend for ACPI?

I would expect that this does not matter at all on ACPI, because the devices that need it are not hot-plugged, and all boot-time devices are probed by the firmware: the ACPI PCI implementation does not reassign any BARs, except for the hotplug case.

Quoted 3 lines

Also, can you point me to the OF device tree specification where it tells how to specify PCI BAR addresses, I would especially be interested in knowing how to specify fixed SRIOV BAR addresses in the device tree.

This is the 'n' bit mentioned sections 2.1.3 and 2.2.1.1 of the PCI binding. When it is set, the OS is not supposed to try to reassign the BAR even on machines that otherwise do a complete rescan.

The PCI binding traditionally requires you to list all PCI devices in DT, Linux as an extension (for the flattened DT format) allows leaving out the devices, but in this case you probably need to list every device that has a fixed BAR.

Quoted 6 lines

Yes, it is a bit of a hack. That is why I put it in its own file, and only try to hack up PCI devices that exactly match the vendor and device ids that need fixing.

IMHO, putting infrastructure into drivers/pci/probe.c, et al. to handle this would be much more intrusive.

My guess is that it's already there, but even if it's not, this is a generic well-defined case that has a standardized binding, and we should implement that.

Quoted 5 lines

For the record: The PCI Enhanced Allocation (EA) capability (approved by PCI SIG on 23 October 2014) is the proper way to handle this going forward. However, this is not yet implemented in the SoCs that this patch addresses. Our plan is to implement the EA capability in the core PCI code, so that we do not need to keep adding devices to this fixup code.

Good, but still this should only be required for the embedded case where you don't have a firmware to probe the bus.

Arnd

#6 lore raw

Re: [PATCH 2/3] PCI: Add quirks for devices found on Cavium ThunderX SoCs.

Quoted 11 lines · Arnd Bergmann
Quoted 10 lines · On Friday 18 September 2015 10:00:32 David Daney wrote:
Quoted 9 lines · Arnd Bergmann
Quoted 6 lines · On Thursday 17 September 2015 15:41:33 David Daney wrote:

From: David Daney <[email protected]>

The on-chip devices all have fixed bars. So, fix them up.

You should be able to just mark the BARs as fixed in DT

I think we can switch to PCI_PROBE_ONLY, and have all non-fixed BAR devices configured by firmware. This may significantly simplify any quirks required in the kernel.

Quoted 45 lines
Quoted 3 lines

In the case of ACPI, there is no DT. So we would need a different solution for ACPI. What would you recommend for ACPI?

I would expect that this does not matter at all on ACPI, because the devices that need it are not hot-plugged, and all boot-time devices are probed by the firmware: the ACPI PCI implementation does not reassign any BARs, except for the hotplug case.

Quoted 3 lines

Also, can you point me to the OF device tree specification where it tells how to specify PCI BAR addresses, I would especially be interested in knowing how to specify fixed SRIOV BAR addresses in the device tree.

This is the 'n' bit mentioned sections 2.1.3 and 2.2.1.1 of the PCI binding. When it is set, the OS is not supposed to try to reassign the BAR even on machines that otherwise do a complete rescan.

The PCI binding traditionally requires you to list all PCI devices in DT, Linux as an extension (for the flattened DT format) allows leaving out the devices, but in this case you probably need to list every device that has a fixed BAR.

Quoted 6 lines

Yes, it is a bit of a hack. That is why I put it in its own file, and only try to hack up PCI devices that exactly match the vendor and device ids that need fixing.

IMHO, putting infrastructure into drivers/pci/probe.c, et al. to handle this would be much more intrusive.

My guess is that it's already there, but even if it's not, this is a generic well-defined case that has a standardized binding, and we should implement that.

Quoted 5 lines

For the record: The PCI Enhanced Allocation (EA) capability (approved by PCI SIG on 23 October 2014) is the proper way to handle this going forward. However, this is not yet implemented in the SoCs that this patch addresses. Our plan is to implement the EA capability in the core PCI code, so that we do not need to keep adding devices to this fixup code.

Good, but still this should only be required for the embedded case where you don't have a firmware to probe the bus.

Arnd

maintainer
#7 lore raw

Re: [PATCH 2/3] PCI: Add quirks for devices found on Cavium ThunderX SoCs.

Hi David,

Quoted 8 lines · David Daney
Quoted 7 lines · Arnd Bergmann
Quoted 6 lines · On Friday 18 September 2015 10:00:32 David Daney wrote:
Quoted 5 lines · Arnd Bergmann
Quoted 4 lines · On Thursday 17 September 2015 15:41:33 David Daney wrote:

From: David Daney <[email protected]>

The on-chip devices all have fixed bars. So, fix them up.

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

You should be able to just mark the BARs as fixed in DT

I think we can switch to PCI_PROBE_ONLY, and have all non-fixed BAR devices configured by firmware. This may significantly simplify any quirks required in the kernel.

I don't like PCI_PROBE_ONLY, and I'd like to avoid it when we can.

Your original patch description said the on-chip devices have "fixed BARs." In what sense are they "fixed"? I assume they are writable enough so we can learn their sizes? If we can't learn their sizes, we have bigger problems because we can't tell what space is used.

Are there other parts of the system, e.g., run-time firmware, that depend on the devices not being moved?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

For the record: The PCI Enhanced Allocation (EA) capability (approved by PCI SIG on 23 October 2014) is the proper way to handle this going forward. However, this is not yet implemented in the SoCs that this patch addresses. Our plan is to implement the EA capability in the core PCI code, so that we do not need to keep adding devices to this fixup code.

Sean Stalley has posted some patches to add EA support to Linux, but I haven't merged them yet. If we had that, another option would be to hook into your config accessors and fabricate an EA capability.

Bjorn

#8 lore raw

Re: [PATCH 2/3] PCI: Add quirks for devices found on Cavium ThunderX SoCs.

Quoted 14 lines · Bjorn Helgaas

Hi David,

Quoted 11 lines · David Daney
Quoted 10 lines · Arnd Bergmann
Quoted 9 lines · On Friday 18 September 2015 10:00:32 David Daney wrote:
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · On Thursday 17 September 2015 15:41:33 David Daney wrote:

From: David Daney <[email protected]>

The on-chip devices all have fixed bars. So, fix them up.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

You should be able to just mark the BARs as fixed in DT

I think we can switch to PCI_PROBE_ONLY, and have all non-fixed BAR devices configured by firmware. This may significantly simplify any quirks required in the kernel.

I don't like PCI_PROBE_ONLY, and I'd like to avoid it when we can.

I don't like it either, but if it were the only way the PCI maintainers would allow us to support the hardware, I would rewrite the firmware to make it possible. However, as you say below ...

Quoted 4 lines

Your original patch description said the on-chip devices have "fixed BARs." In what sense are they "fixed"? I assume they are writable enough so we can learn their sizes?

Yes. The BAR registers are writable, but ignored. So, the size is correctly probed. The BAR registers are initialized by hardware/firmware to the proper value.

Quoted 5 lines

If we can't learn their sizes, we
have bigger problems because we can't tell what space is used.

Are there other parts of the system, e.g., run-time firmware, that depend on the devices not being moved?

The devices cannot move, the address decoding is not programmable. The BAR registers are provided as an aid in integrating the devices with OS PCI infrastructure. Although, one might argue that they don't do a very good job of adhering to the PCI specifications...

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

For the record: The PCI Enhanced Allocation (EA) capability (approved by PCI SIG on 23 October 2014) is the proper way to handle this going forward. However, this is not yet implemented in the SoCs that this patch addresses. Our plan is to implement the EA capability in the core PCI code, so that we do not need to keep adding devices to this fixup code.

Sean Stalley has posted some patches to add EA support to Linux, but I haven't merged them yet. If we had that, another option would be to hook into your config accessors and fabricate an EA capability.

To me, this is the most interesting part of your message. If you really would accept a config read accessor that presented a synthetic EA capability, that would be ideal. We know exactly which roots contain the fixed devices, so it would be a trivial exercise to provide a custom config accessor.

I am going to work on this in hope of eventual acceptance of this strategy.

Thanks,
David Daney

#9 lore raw

Re: [PATCH 2/3] PCI: Add quirks for devices found on Cavium ThunderX SoCs.

In reply to Arnd Bergmann
Quoted 12 lines · Arnd Bergmann
Quoted 11 lines · On Friday 18 September 2015 10:00:32 David Daney wrote:
Quoted 9 lines · Arnd Bergmann
Quoted 6 lines · On Thursday 17 September 2015 15:41:33 David Daney wrote:

From: David Daney <[email protected]>

The on-chip devices all have fixed bars. So, fix them up.

You should be able to just mark the BARs as fixed in DT

Quoted 7 lines
Quoted 2 lines

In the case of ACPI, there is no DT. So we would need a different solution for ACPI. What would you recommend for ACPI?

I would expect that this does not matter at all on ACPI, because the devices that need it are not hot-plugged, and all boot-time devices are probed by the firmware: the ACPI PCI implementation does not reassign any BARs, except for the hotplug case.

What do you mean by "the ACPI PCI implementation does not reassign any BARs" ? Do you mean on x86 ? The resource assignment is part of the resource survey on x86, where all resources that can be claimed are claimed, but still, some of them may be still reassigned IIUC.

On arm64 we do not carry out any resource survey at present, but if we do (and we should), it will have to work for both DT and ACPI systems.

Quoted 8 lines
Quoted 3 lines

Also, can you point me to the OF device tree specification where it tells how to specify PCI BAR addresses, I would especially be interested in knowing how to specify fixed SRIOV BAR addresses in the device tree.

This is the 'n' bit mentioned sections 2.1.3 and 2.2.1.1 of the PCI binding. When it is set, the OS is not supposed to try to reassign the BAR even on machines that otherwise do a complete rescan.

I do not see any code in the kernel taking care of that bit and if I am not mistaken the code that allows creating pci devices exists in PowerPC arch code (arch/powerpc/kernel/pci_of_scan.c) and should be moved out of there for the other arches to use it.

Or maybe we can use DT just to fix-up the resource flags (ie every PCI device will have its own of_node set if it is present in the DT, we can use it to fixup its resources and related flags, pcibios_add_device() ?).

Quoted 9 lines

The PCI binding traditionally requires you to list all PCI devices in DT, Linux as an extension (for the flattened DT format) allows leaving out the devices, but in this case you probably need to list every device that has a fixed BAR.

Quoted 4 lines

Yes, it is a bit of a hack. That is why I put it in its own file, and only try to hack up PCI devices that exactly match the vendor and device ids that need fixing.

Quoted 6 lines
Quoted 2 lines

IMHO, putting infrastructure into drivers/pci/probe.c, et al. to handle this would be much more intrusive.

My guess is that it's already there, but even if it's not, this is a generic well-defined case that has a standardized binding, and we should implement that.

See above.

Thanks,
Lorenzo

Quoted 15 lines
Quoted 5 lines

Forthe record: The PCI Enhanced Allocation (EA) capability (approved by PCI SIG on 23 October 2014) is the proper way to handle this going forward. However, this is not yet implemented in the SoCs that this patch addresses. Our plan is to implement the EA capability in the core PCI code, so that we do not need to keep adding devices to this fixup code.

Good, but still this should only be required for the embedded case where you don't have a firmware to probe the bus.

Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw

Re: [PATCH 2/3] PCI: Add quirks for devices found on Cavium ThunderX SoCs.

Quoted 21 lines · On Tuesday 22 September 2015 16:39:31 Lorenzo Pieralisi wrote:
Quoted 12 lines · Arnd Bergmann
Quoted 11 lines · On Friday 18 September 2015 10:00:32 David Daney wrote:
Quoted 9 lines · Arnd Bergmann
Quoted 6 lines · On Thursday 17 September 2015 15:41:33 David Daney wrote:

From: David Daney <[email protected]>

The on-chip devices all have fixed bars. So, fix them up.

You should be able to just mark the BARs as fixed in DT

Quoted 7 lines
Quoted 2 lines

In the case of ACPI, there is no DT. So we would need a different solution for ACPI. What would you recommend for ACPI?

I would expect that this does not matter at all on ACPI, because the devices that need it are not hot-plugged, and all boot-time devices are probed by the firmware: the ACPI PCI implementation does not reassign any BARs, except for the hotplug case.

Quoted 4 lines

What do you mean by "the ACPI PCI implementation does not reassign any BARs" ? Do you mean on x86 ? The resource assignment is part of the resource survey on x86, where all resources that can be claimed are claimed, but still, some of them may be still reassigned IIUC.

The ACPI PCI implementation should be completely architecture independent and do the same thing everywhere. The code is spread over drivers/acpi/pci*, with small parts currently residing in arch/{x86,ia64} that need to be moved to drivers/acpi/

Quoted 3 lines

On arm64 we do not carry out any resource survey at present, but if we do (and we should), it will have to work for both DT and ACPI systems.

This is not a difference between DT and ACPI, but a difference between embedded and server. Server platforms that have a proper firmware to scan the PCI bus should not set PCI_REASSIGN_ALL_RSRC and PCI_REASSIGN_ALL_BUS, while embedded systems that don't set up the PCI bus before boot have to set those.

On ACPI, reassigning the resources would actually be dangerous because the firmware may rely on devices to be at a certain location (both bus number and mmio address), or may play tricks here to hide stuff from the OS.

Quoted 14 lines
Quoted 8 lines
Quoted 3 lines

Also, can you point me to the OF device tree specification where it tells how to specify PCI BAR addresses, I would especially be interested in knowing how to specify fixed SRIOV BAR addresses in the device tree.

This is the 'n' bit mentioned sections 2.1.3 and 2.2.1.1 of the PCI binding. When it is set, the OS is not supposed to try to reassign the BAR even on machines that otherwise do a complete rescan.

I do not see any code in the kernel taking care of that bit and if I am not mistaken the code that allows creating pci devices exists in PowerPC arch code (arch/powerpc/kernel/pci_of_scan.c) and should be moved out of there for the other arches to use it.

Quoted 4 lines

Or maybe we can use DT just to fix-up the resource flags (ie every PCI device will have its own of_node set if it is present in the DT, we can use it to fixup its resources and related flags, pcibios_add_device() ?).

I haven't looked at the code in detail now, but I'm pretty sure that a device node that refers to a PCI device is connected to the dev->of_node pointer already on all architectures.

It's quite possible that we never implemented the fixed BAR logic though, but it can be done based on those pointers.

Arnd

#11 lore raw

PATCH 3/3 PCI: generic: Add support for Cavium ThunderX PCIe root complexes.

In reply to David Daney

From: David Daney <[email protected]>

The config space for external PCIe root complexes on some Cavium ThunderX SoCs is very similar to CAM and ECAM, but differs in the shift values that have to be applied to the bus and devfn numbers to compose that address window offset. These root complexes also have the interesting property that there is no root bridge, so the standard manner of limiting scanning to only the first device doesn't work. We can use the standard pci-host-generic driver if we make a minor addition to handle these differences, so we...

Add a mapping function for ThunderX PCIe root complexes with a bus shift of 24 and devfn shift of 16. Ignore accesses for devices other than the first device on the primary bus.

Document the whole thing in devicetree/bindings/pci/host-generic-pci.txt

---
 .../devicetree/bindings/pci/host-generic-pci.txt   |  8 +++---
 drivers/pci/host/pci-host-generic.c                | 29 ++++++++++++++++++++++
 2 files changed, 34 insertions(+), 3 deletions(-)
Documentation/devicetree/bindings/pci/host-generic-pci.txt +5 -3
index 105a968..a5aed0f 100644
@@ -14,9 +14,11 @@ tree bindings communicated in pci.txt:
 
 Properties of the host controller node:
 
-- compatible     : Must be "pci-host-cam-generic" or "pci-host-ecam-generic"
-                   depending on the layout of configuration space (CAM vs
-                   ECAM respectively).
+- compatible     : One of the following with bus:devfn:reg mapped to the
+                   PCI config space address window in the bit positions shown:
+                   "pci-host-cam-generic" -- 'CAM'  bits 16:8:0
+                   "pci-host-ecam-generic" -- 'ECAM'  bits 20:12:0
+                   "cavium,pci-host-thunder-pem" --  bits 24:16:0
 
 - device_type    : Must be "pci".
 
drivers/pci/host/pci-host-generic.c +29 -0
index e364232..e1d8d5b 100644
@@ -91,6 +91,32 @@ static struct gen_pci_cfg_bus_ops gen_pci_cfg_ecam_bus_ops = {
 	}
 };
 
+static void __iomem *gen_pci_map_cfg_bus_thunder_pem(struct pci_bus *bus,
+						     unsigned int devfn,
+						     int where)
+{
+	struct gen_pci *pci = bus->sysdata;
+	resource_size_t idx = bus->number - pci->cfg.bus_range->start;
+
+	/*
+	 * Thunder PEM is a PCIe RC, but without a root bridge.  On
+	 * the primary bus, ignore accesses for devices other than
+	 * the first device.
+	 */
+	if (idx == 0 && (devfn & ~7u))
+		return NULL;
+	return pci->cfg.win[idx] + ((devfn << 16) | where);
+}
+
+static struct gen_pci_cfg_bus_ops gen_pci_cfg_thunder_pem_bus_ops = {
+	.bus_shift	= 24,
+	.ops		= {
+		.map_bus	= gen_pci_map_cfg_bus_thunder_pem,
+		.read		= pci_generic_config_read,
+		.write		= pci_generic_config_write,
+	}
+};
+
 static const struct of_device_id gen_pci_of_match[] = {
 	{ .compatible = "pci-host-cam-generic",
 	  .data = &gen_pci_cfg_cam_bus_ops },
@@ -98,6 +124,9 @@ static const struct of_device_id gen_pci_of_match[] = {
 	{ .compatible = "pci-host-ecam-generic",
 	  .data = &gen_pci_cfg_ecam_bus_ops },
 
+	{ .compatible = "cavium,pci-host-thunder-pem",
+	  .data = &gen_pci_cfg_thunder_pem_bus_ops },
+
 	{ },
 };
 MODULE_DEVICE_TABLE(of, gen_pci_of_match);
Signature · 1 line
1.9.1
#12 lore raw

Re: [PATCH 3/3] PCI: generic: Add support for Cavium ThunderX PCIe root complexes.

Quoted 29 lines · David Daney

From: David Daney <[email protected]>

The config space for external PCIe root complexes on some Cavium ThunderX SoCs is very similar to CAM and ECAM, but differs in the shift values that have to be applied to the bus and devfn numbers to compose that address window offset. These root complexes also have the interesting property that there is no root bridge, so the standard manner of limiting scanning to only the first device doesn't work. We can use the standard pci-host-generic driver if we make a minor addition to handle these differences, so we...

Add a mapping function for ThunderX PCIe root complexes with a bus shift of 24 and devfn shift of 16. Ignore accesses for devices other than the first device on the primary bus.

Document the whole thing in devicetree/bindings/pci/host-generic-pci.txt

---
 .../devicetree/bindings/pci/host-generic-pci.txt   |  8 +++---
 drivers/pci/host/pci-host-generic.c                | 29 ++++++++++++++++++++++
 2 files changed, 34 insertions(+), 3 deletions(-)
Documentation/devicetree/bindings/pci/host-generic-pci.txt +0 -0
index 105a968..a5aed0f 100644
@@ -14,9 +14,11 @@ tree bindings communicated in pci.txt:
 

Documentation/devicetree/bindings/pci/host-generic-pci.txt tree bindings communicated in pci.txt: · line 21

Quoted 10 lines
Documentation/devicetree/bindings/pci/host-generic-pci.txt +5 -3
 Properties of the host controller node:
 
-- compatible     : Must be "pci-host-cam-generic" or "pci-host-ecam-generic"
-                   depending on the layout of configuration space (CAM vs
-                   ECAM respectively).
+- compatible     : One of the following with bus:devfn:reg mapped to the
+                   PCI config space address window in the bit positions shown:
+                   "pci-host-cam-generic" -- 'CAM'  bits 16:8:0
+                   "pci-host-ecam-generic" -- 'ECAM'  bits 20:12:0
+                   "cavium,pci-host-thunder-pem" --  bits 24:16:0

To me that's ECAM left shifted by 4. Is not it just a matter of defining config space base address (ie reg property) differently ?

Lorenzo

Quoted 58 lines
Documentation/devicetree/bindings/pci/host-generic-pci.txt +0 -0
 - device_type    : Must be "pci".
 
drivers/pci/host/pci-host-generic.c +29 -0
index e364232..e1d8d5b 100644
@@ -91,6 +91,32 @@ static struct gen_pci_cfg_bus_ops gen_pci_cfg_ecam_bus_ops = {
 	}
 };
 
+static void __iomem *gen_pci_map_cfg_bus_thunder_pem(struct pci_bus *bus,
+						     unsigned int devfn,
+						     int where)
+{
+	struct gen_pci *pci = bus->sysdata;
+	resource_size_t idx = bus->number - pci->cfg.bus_range->start;
+
+	/*
+	 * Thunder PEM is a PCIe RC, but without a root bridge.  On
+	 * the primary bus, ignore accesses for devices other than
+	 * the first device.
+	 */
+	if (idx == 0 && (devfn & ~7u))
+		return NULL;
+	return pci->cfg.win[idx] + ((devfn << 16) | where);
+}
+
+static struct gen_pci_cfg_bus_ops gen_pci_cfg_thunder_pem_bus_ops = {
+	.bus_shift	= 24,
+	.ops		= {
+		.map_bus	= gen_pci_map_cfg_bus_thunder_pem,
+		.read		= pci_generic_config_read,
+		.write		= pci_generic_config_write,
+	}
+};
+
 static const struct of_device_id gen_pci_of_match[] = {
 	{ .compatible = "pci-host-cam-generic",
 	  .data = &gen_pci_cfg_cam_bus_ops },
@@ -98,6 +124,9 @@ static const struct of_device_id gen_pci_of_match[] = {
 	{ .compatible = "pci-host-ecam-generic",
 	  .data = &gen_pci_cfg_ecam_bus_ops },
 
+	{ .compatible = "cavium,pci-host-thunder-pem",
+	  .data = &gen_pci_cfg_thunder_pem_bus_ops },
+
 	{ },
 };
 MODULE_DEVICE_TABLE(of, gen_pci_of_match);
Signature · 6 lines
1.9.1

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#13 lore raw

Re: [PATCH 3/3] PCI: generic: Add support for Cavium ThunderX PCIe root complexes.

Quoted 32 lines · Lorenzo Pieralisi
Quoted 31 lines · David Daney

From: David Daney <[email protected]>

The config space for external PCIe root complexes on some Cavium ThunderX SoCs is very similar to CAM and ECAM, but differs in the shift values that have to be applied to the bus and devfn numbers to compose that address window offset. These root complexes also have the interesting property that there is no root bridge, so the standard manner of limiting scanning to only the first device doesn't work. We can use the standard pci-host-generic driver if we make a minor addition to handle these differences, so we...

Add a mapping function for ThunderX PCIe root complexes with a bus shift of 24 and devfn shift of 16. Ignore accesses for devices other than the first device on the primary bus.

Document the whole thing in devicetree/bindings/pci/host-generic-pci.txt

---
  .../devicetree/bindings/pci/host-generic-pci.txt   |  8 +++---
  drivers/pci/host/pci-host-generic.c                | 29 ++++++++++++++++++++++
  2 files changed, 34 insertions(+), 3 deletions(-)
Documentation/devicetree/bindings/pci/host-generic-pci.txt +0 -0
index 105a968..a5aed0f 100644
@@ -14,9 +14,11 @@ tree bindings communicated in pci.txt:
 
  Properties of the host controller node:
 
Quoted 11 lines

Documentation/devicetree/bindings/pci/host-generic-pci.txt tree bindings communicated in pci.txt: · line 21

Quoted 8 lines
Documentation/devicetree/bindings/pci/host-generic-pci.txt +5 -3
-- compatible     : Must be "pci-host-cam-generic" or "pci-host-ecam-generic"
-                   depending on the layout of configuration space (CAM vs
-                   ECAM respectively).
+- compatible     : One of the following with bus:devfn:reg mapped to the
+                   PCI config space address window in the bit positions shown:
+                   "pci-host-cam-generic" -- 'CAM'  bits 16:8:0
+                   "pci-host-ecam-generic" -- 'ECAM'  bits 20:12:0
+                   "cavium,pci-host-thunder-pem" --  bits 24:16:0

To me that's ECAM left shifted by 4. Is not it just a matter of defining config space base address (ie reg property) differently ?

No.

That's like saying ECAM is just CAM shifted by 4, can't we just specify the "reg" differently.

The "reg" property describes the base of the config space access window.
CAM/ECAM/cavium,pci-host-thunder-pem describe how to generate offsets
from that base, The offset calculation requires information that is not present in the "reg" property, it is enumerated in the various "compatible" values listed above.

David Daney

Quoted 61 lines

Lorenzo

Quoted 58 lines
Documentation/devicetree/bindings/pci/host-generic-pci.txt +0 -0
  - device_type    : Must be "pci".
 
drivers/pci/host/pci-host-generic.c +29 -1
index e364232..e1d8d5b 100644
@@ -91,6 +91,32 @@ static struct gen_pci_cfg_bus_ops gen_pci_cfg_ecam_bus_ops = {
  	}
  };
 
+static void __iomem *gen_pci_map_cfg_bus_thunder_pem(struct pci_bus *bus,
+						     unsigned int devfn,
+						     int where)
+{
+	struct gen_pci *pci = bus->sysdata;
+	resource_size_t idx = bus->number - pci->cfg.bus_range->start;
+
+	/*
+	 * Thunder PEM is a PCIe RC, but without a root bridge.  On
+	 * the primary bus, ignore accesses for devices other than
+	 * the first device.
+	 */
+	if (idx == 0 && (devfn & ~7u))
+		return NULL;
+	return pci->cfg.win[idx] + ((devfn << 16) | where);
+}
+
+static struct gen_pci_cfg_bus_ops gen_pci_cfg_thunder_pem_bus_ops = {
+	.bus_shift	= 24,
+	.ops		= {
+		.map_bus	= gen_pci_map_cfg_bus_thunder_pem,
+		.read		= pci_generic_config_read,
+		.write		= pci_generic_config_write,
+	}
+};
+
  static const struct of_device_id gen_pci_of_match[] = {
  	{ .compatible = "pci-host-cam-generic",
  	  .data = &gen_pci_cfg_cam_bus_ops },
@@ -98,6 +124,9 @@ static const struct of_device_id gen_pci_of_match[] = {
  	{ .compatible = "pci-host-ecam-generic",
  	  .data = &gen_pci_cfg_ecam_bus_ops },
 
+	{ .compatible = "cavium,pci-host-thunder-pem",
+	  .data = &gen_pci_cfg_thunder_pem_bus_ops },
+
  	{ },
  };
  MODULE_DEVICE_TABLE(of, gen_pci_of_match);
--

1.9.1

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw

Re: [PATCH 3/3] PCI: generic: Add support for Cavium ThunderX PCIe root complexes.

Quoted 2 lines · David Daney
Quoted 1 line · Lorenzo Pieralisi

On Thu, Sep 17, 2015 at 11:41:34PM +0100, David Daney wrote:

[...]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

Properties of the host controller node:

-- compatible : Must be "pci-host-cam-generic" or "pci-host-ecam-generic"

Documentation/devicetree/bindings/pci/host-generic-pci.txt +5 -2
-                   depending on the layout of configuration space (CAM vs
-                   ECAM respectively).
+- compatible     : One of the following with bus:devfn:reg mapped to the
+                   PCI config space address window in the bit positions shown:
+                   "pci-host-cam-generic" -- 'CAM'  bits 16:8:0
+                   "pci-host-ecam-generic" -- 'ECAM'  bits 20:12:0
+                   "cavium,pci-host-thunder-pem" --  bits 24:16:0
Quoted 7 lines
Quoted 2 lines

To me that's ECAM left shifted by 4. Is not it just a matter of defining config space base address (ie reg property) differently ?

No.

That's like saying ECAM is just CAM shifted by 4, can't we just specify the "reg" differently.

Ok, sorry I was not clear, apologies. I meant to ask if it is something that can be configured in your platform and apparently it is not, I worded it wrongly.

Quoted 5 lines

The "reg" property describes the base of the config space access window.
CAM/ECAM/cavium,pci-host-thunder-pem describe how to generate offsets
from that base, The offset calculation requires information that is not present in the "reg" property, it is enumerated in the various "compatible" values listed above.

Agreed, this also forces you to map in more virtual address space for a given bus, right ? I am also a bit worried about how we can make this work with ACPI, where MCFG accessors take the bus and devfn shifts for granted (ie ECAM specs), so I asked.

Thanks,
Lorenzo

Quoted 65 lines

David Daney

Quoted 61 lines

Lorenzo

Quoted 58 lines

- device_type : Must be "pci".

drivers/pci/host/pci-host-generic.c +29 -1
index e364232..e1d8d5b 100644
@@ -91,6 +91,32 @@ static struct gen_pci_cfg_bus_ops gen_pci_cfg_ecam_bus_ops = {
  	}
  };
 
+static void __iomem *gen_pci_map_cfg_bus_thunder_pem(struct pci_bus *bus,
+						     unsigned int devfn,
+						     int where)
+{
+	struct gen_pci *pci = bus->sysdata;
+	resource_size_t idx = bus->number - pci->cfg.bus_range->start;
+
+	/*
+	 * Thunder PEM is a PCIe RC, but without a root bridge.  On
+	 * the primary bus, ignore accesses for devices other than
+	 * the first device.
+	 */
+	if (idx == 0 && (devfn & ~7u))
+		return NULL;
+	return pci->cfg.win[idx] + ((devfn << 16) | where);
+}
+
+static struct gen_pci_cfg_bus_ops gen_pci_cfg_thunder_pem_bus_ops = {
+	.bus_shift	= 24,
+	.ops		= {
+		.map_bus	= gen_pci_map_cfg_bus_thunder_pem,
+		.read		= pci_generic_config_read,
+		.write		= pci_generic_config_write,
+	}
+};
+
  static const struct of_device_id gen_pci_of_match[] = {
  	{ .compatible = "pci-host-cam-generic",
  	  .data = &gen_pci_cfg_cam_bus_ops },
@@ -98,6 +124,9 @@ static const struct of_device_id gen_pci_of_match[] = {
  	{ .compatible = "pci-host-ecam-generic",
  	  .data = &gen_pci_cfg_ecam_bus_ops },
 
+	{ .compatible = "cavium,pci-host-thunder-pem",
+	  .data = &gen_pci_cfg_thunder_pem_bus_ops },
+
  	{ },
  };
  MODULE_DEVICE_TABLE(of, gen_pci_of_match);
--

1.9.1

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#15 lore raw

Re: [PATCH 3/3] PCI: generic: Add support for Cavium ThunderX PCIe root complexes.

Quoted 20 lines · Lorenzo Pieralisi
Quoted 2 lines · David Daney
Quoted 1 line · Lorenzo Pieralisi

On Thu, Sep 17, 2015 at 11:41:34PM +0100, David Daney wrote:

[...]

Quoted 14 lines
Quoted 13 lines

Documentation/devicetree/bindings/pci/host-generic-pci.txt tree bindings communicated in pci.txt: · line 21

Quoted 10 lines

Properties of the host controller node:

-- compatible : Must be "pci-host-cam-generic" or "pci-host-ecam-generic"

Documentation/devicetree/bindings/pci/host-generic-pci.txt +5 -2
-                   depending on the layout of configuration space (CAM vs
-                   ECAM respectively).
+- compatible     : One of the following with bus:devfn:reg mapped to the
+                   PCI config space address window in the bit positions shown:
+                   "pci-host-cam-generic" -- 'CAM'  bits 16:8:0
+                   "pci-host-ecam-generic" -- 'ECAM'  bits 20:12:0
+                   "cavium,pci-host-thunder-pem" --  bits 24:16:0

To me that's ECAM left shifted by 4. Is not it just a matter of defining config space base address (ie reg property) differently ?

Quoted 7 lines
Quoted 4 lines

No.

That's like saying ECAM is just CAM shifted by 4, can't we just specify the "reg" differently.

Ok, sorry I was not clear, apologies. I meant to ask if it is something that can be configured in your platform and apparently it is not,

No, the hardware cannot be adjusted.

Quoted 1 line

I worded it wrongly.

OK, I guess I also misunderstood what you were suggesting.

Quoted 9 lines
Quoted 6 lines

The "reg" property describes the base of the config space access window.
CAM/ECAM/cavium,pci-host-thunder-pem describe how to generate offsets
from that base, The offset calculation requires information that is not present in the "reg" property, it is enumerated in the various "compatible" values listed above.

Agreed, this also forces you to map in more virtual address space for a given bus, right ?

Yes. It takes a full 32-bits of address space to have access to all PCI buses on the root complex.

Quoted 3 lines

I am also a bit worried about how we can make this
work with ACPI, where MCFG accessors take the bus and devfn shifts for granted (ie ECAM specs), so I asked.

If ACPI is not flexible enough to describe actual hardware, it will either have to be enhanced, or not used.

Quoted 69 lines

Thanks,
Lorenzo

Quoted 65 lines

David Daney

Quoted 61 lines

Lorenzo

Quoted 58 lines

- device_type : Must be "pci".

drivers/pci/host/pci-host-generic.c +29 -1
index e364232..e1d8d5b 100644
@@ -91,6 +91,32 @@ static struct gen_pci_cfg_bus_ops gen_pci_cfg_ecam_bus_ops = {
   	}
   };
 
+static void __iomem *gen_pci_map_cfg_bus_thunder_pem(struct pci_bus *bus,
+						     unsigned int devfn,
+						     int where)
+{
+	struct gen_pci *pci = bus->sysdata;
+	resource_size_t idx = bus->number - pci->cfg.bus_range->start;
+
+	/*
+	 * Thunder PEM is a PCIe RC, but without a root bridge.  On
+	 * the primary bus, ignore accesses for devices other than
+	 * the first device.
+	 */
+	if (idx == 0 && (devfn & ~7u))
+		return NULL;
+	return pci->cfg.win[idx] + ((devfn << 16) | where);
+}
+
+static struct gen_pci_cfg_bus_ops gen_pci_cfg_thunder_pem_bus_ops = {
+	.bus_shift	= 24,
+	.ops		= {
+		.map_bus	= gen_pci_map_cfg_bus_thunder_pem,
+		.read		= pci_generic_config_read,
+		.write		= pci_generic_config_write,
+	}
+};
+
   static const struct of_device_id gen_pci_of_match[] = {
   	{ .compatible = "pci-host-cam-generic",
   	  .data = &gen_pci_cfg_cam_bus_ops },
@@ -98,6 +124,9 @@ static const struct of_device_id gen_pci_of_match[] = {
   	{ .compatible = "pci-host-ecam-generic",
   	  .data = &gen_pci_cfg_ecam_bus_ops },
 
+	{ .compatible = "cavium,pci-host-thunder-pem",
+	  .data = &gen_pci_cfg_thunder_pem_bus_ops },
+
   	{ },
   };
   MODULE_DEVICE_TABLE(of, gen_pci_of_match);
--

1.9.1

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Acked-by on v1 3/3
#16 lore raw

Re: [PATCH 3/3] PCI: generic: Add support for Cavium ThunderX PCIe root complexes.

In reply to David Daney
Quoted 15 lines · David Daney

From: David Daney <[email protected]>

The config space for external PCIe root complexes on some Cavium ThunderX SoCs is very similar to CAM and ECAM, but differs in the shift values that have to be applied to the bus and devfn numbers to compose that address window offset. These root complexes also have the interesting property that there is no root bridge, so the standard manner of limiting scanning to only the first device doesn't work. We can use the standard pci-host-generic driver if we make a minor addition to handle these differences, so we...

Add a mapping function for ThunderX PCIe root complexes with a bus shift of 24 and devfn shift of 16. Ignore accesses for devices other than the first device on the primary bus.

Quoted 7 lines

Document the whole thing in devicetree/bindings/pci/host-generic-pci.txt

---
 .../devicetree/bindings/pci/host-generic-pci.txt   |  8 +++---
 drivers/pci/host/pci-host-generic.c                | 29 ++++++++++++++++++++++
 2 files changed, 34 insertions(+), 3 deletions(-)

Thanks, this looks better now:

Will

#17 lore raw

Re: [PATCH 3/3] PCI: generic: Add support for Cavium ThunderX PCIe root complexes.

Quoted 24 lines · Will Deacon
Quoted 15 lines · David Daney

From: David Daney <[email protected]>

The config space for external PCIe root complexes on some Cavium ThunderX SoCs is very similar to CAM and ECAM, but differs in the shift values that have to be applied to the bus and devfn numbers to compose that address window offset. These root complexes also have the interesting property that there is no root bridge, so the standard manner of limiting scanning to only the first device doesn't work. We can use the standard pci-host-generic driver if we make a minor addition to handle these differences, so we...

Add a mapping function for ThunderX PCIe root complexes with a bus shift of 24 and devfn shift of 16. Ignore accesses for devices other than the first device on the primary bus.

Quoted 7 lines

Document the whole thing in devicetree/bindings/pci/host-generic-pci.txt

---
  .../devicetree/bindings/pci/host-generic-pci.txt   |  8 +++---
  drivers/pci/host/pci-host-generic.c                | 29 ++++++++++++++++++++++
  2 files changed, 34 insertions(+), 3 deletions(-)
Quoted 4 lines

Thanks, this looks better now:

Thanks Will.

Because patches 1/3 and 2/3 will be reworked, I will re-send this as a stand-alone patch.

David Daney

Quoted 2 lines

Will