Skip to contentSkip to conversation

PCI: Configure PCIe MPS settings

PATCH New Bjorn Helgaas 4 participants 11 messages Feb 8, 2017 – Feb 24, 2017

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover PCI: Configure PCIe MPS settings

[Some of you will get this twice because Gmail and I aren't getting along today; sorry]

The PCI core doesn't configure the PCIe MPS settings by itself. Each host bridge driver has to call pcie_bus_configure_settings() to make this happen.

Jon fixed this already for pcie-iproc.c. I propose these similar patches for other drivers.

HV guys, I included you because create_root_hv_pci_bus() is one place that calls pci_scan_child_bus() but does not call pcie_bus_configure_settings(). I know you probably don't strictly *need* to configure MPS settings in a paravirtual front-end, but the PCI core does other device configuration in this path:

pci_scan_child_bus
pci_scan_slot
pci_scan_single_device
pci_device_add
pci_configure_device
pci_init_capabilities

and I would like to eventually migrate the MPS configuration into that same path. Since we do this other configuration for HV devices already, I think pcie_bus_configure_settings() should also work (even if it ends up not doing anything to real devices).

It would make that eventual migration easier if all the pci_scan_child_bus() callers had the same pattern of calling pcie_bus_configure_settings().

So would it make sense to make a similar patch for HV? It looks easy to add it to create_root_hv_pci_bus(), but I don't know exactly what to do about pci_devices_present_work().

---

Bjorn Helgaas (3):
PCI: xilinx: Configure PCIe MPS settings
PCI: versatile: Configure PCIe MPS settings
PCI: xgene: Configure PCIe MPS settings

 drivers/pci/host/pci-versatile.c |    4 +++-
 drivers/pci/host/pci-xgene.c     |    4 +++-
 drivers/pci/host/pcie-xilinx.c   |    4 +++-
 3 files changed, 9 insertions(+), 3 deletions(-)
#1 lore raw

PATCH 1/3 PCI: xilinx: Configure PCIe MPS settings

From: Bjorn Helgaas <[email protected]>

Make sure PCIe MPS settings are valid when we enumerate a new hierarchy.

---
 drivers/pci/host/pcie-xilinx.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/host/pcie-xilinx.c +3 -1
index c8616fadccf1..7f030f5d750b 100644
@@ -632,7 +632,7 @@ static int xilinx_pcie_probe(struct platform_device *pdev)
 {
 	struct device *dev = &pdev->dev;
 	struct xilinx_pcie_port *port;
-	struct pci_bus *bus;
+	struct pci_bus *bus, *child;
 	int err;
 	resource_size_t iobase = 0;
 	LIST_HEAD(res);
@@ -686,6 +686,8 @@ static int xilinx_pcie_probe(struct platform_device *pdev)
 #ifndef CONFIG_MICROBLAZE
 	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
 #endif
+	list_for_each_entry(child, &bus->children, node)
+		pcie_bus_configure_settings(child);
 	pci_bus_add_devices(bus);
 	return 0;
 
#2 lore raw

PATCH 1/3 PCI: xilinx: Configure PCIe MPS settings

drivers/pci/host/pcie-xilinx.c static int xilinx_pcie_probe(struct platform_device *pdev) · line 690

Quoted 2 lines · Bjorn Helgaas
drivers/pci/host/pcie-xilinx.c +2 -0
+	list_for_each_entry(child, &bus->children, node)
+		pcie_bus_configure_settings(child);

This loop is duplicated in just about every driver, so it it might be a good idea to provide a littler helper for it.

#3 lore raw

PATCH 1/3 PCI: xilinx: Configure PCIe MPS settings

Quoted 6 lines · Christoph Hellwig

drivers/pci/host/pcie-xilinx.c static int xilinx_pcie_probe(struct platform_device *pdev) · line 690

Quoted 2 lines · Bjorn Helgaas
drivers/pci/host/pcie-xilinx.c +2 -0
+	list_for_each_entry(child, &bus->children, node)
+		pcie_bus_configure_settings(child);

This loop is duplicated in just about every driver, so it it might be a good idea to provide a littler helper for it.

Yeah, you're right. One reason I didn't is because I would like to remove these if we can move the MPS setup into the pci_device_add() path.

We ought to be able to do at least the minimal "configure this new device to match the existing hierarchy; if that's impossible, disable the device" sort of thing there.

We probably would still want some sort of bus- or root port- or host bridge-level configuration that can take a broader view, and that might be done where we currently have this loop. But I don't know exactly what that should look like.

Bjorn

#4 lore raw

PATCH 2/3 PCI: versatile: Configure PCIe MPS settings

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Make sure PCIe MPS settings are valid when we enumerate a new hierarchy.

---
 drivers/pci/host/pci-versatile.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/host/pci-versatile.c +3 -1
index b7dc07002f13..5ebee7d37ff5 100644
@@ -124,7 +124,7 @@ static int versatile_pci_probe(struct platform_device *pdev)
 	int ret, i, myslot = -1;
 	u32 val;
 	void __iomem *local_pci_cfg_base;
-	struct pci_bus *bus;
+	struct pci_bus *bus, *child;
 	LIST_HEAD(pci_res);
 
 	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
@@ -204,6 +204,8 @@ static int versatile_pci_probe(struct platform_device *pdev)
 
 	pci_fixup_irqs(pci_common_swizzle, of_irq_parse_and_map_pci);
 	pci_assign_unassigned_bus_resources(bus);
+	list_for_each_entry(child, &bus->children, node)
+		pcie_bus_configure_settings(child);
 	pci_bus_add_devices(bus);
 
 	return 0;
#5 lore raw

PATCH 3/3 PCI: xgene: Configure PCIe MPS settings

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Make sure PCIe MPS settings are valid when we enumerate a new hierarchy.

---
 drivers/pci/host/pci-xgene.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/pci/host/pci-xgene.c +3 -1
index 7c3b54b9eb17..8091a8778ae7 100644
@@ -638,7 +638,7 @@ static int xgene_pcie_probe_bridge(struct platform_device *pdev)
 	struct device_node *dn = dev->of_node;
 	struct xgene_pcie_port *port;
 	resource_size_t iobase = 0;
-	struct pci_bus *bus;
+	struct pci_bus *bus, *child;
 	int ret;
 	LIST_HEAD(res);
 
@@ -681,6 +681,8 @@ static int xgene_pcie_probe_bridge(struct platform_device *pdev)
 
 	pci_scan_child_bus(bus);
 	pci_assign_unassigned_bus_resources(bus);
+	list_for_each_entry(child, &bus->children, node)
+		pcie_bus_configure_settings(child);
 	pci_bus_add_devices(bus);
 	return 0;
 
#6 lore raw

cover PCI: Configure PCIe MPS settings

In reply to Bjorn Helgaas
Quoted 42 lines

-----Original Message-----
From: Bjorn Helgaas [mailto:helgaas at kernel.org]
Sent: Wednesday, February 8, 2017 2:49 PM
To: linux-pci at vger.kernel.org
Cc: Rob Herring <[email protected]>; Haiyang Zhang <[email protected]>; Michal Simek <[email protected]>; Jon Mason <[email protected]>; Ray Jui <[email protected]>; S?ren Brinkmann <[email protected]>;
devel at linuxdriverproject.org; KY Srinivasan <[email protected]>; linux- arm-kernel at lists.infradead.org; Tanmay Inamdar <[email protected]>
Subject: [PATCH v1 0/3] PCI: Configure PCIe MPS settings

[Some of you will get this twice because Gmail and I aren't getting along today; sorry]

The PCI core doesn't configure the PCIe MPS settings by itself. Each host bridge driver has to call pcie_bus_configure_settings() to make this happen.

Jon fixed this already for pcie-iproc.c. I propose these similar patches for other drivers.

HV guys, I included you because create_root_hv_pci_bus() is one place that calls pci_scan_child_bus() but does not call pcie_bus_configure_settings(). I know you probably don't strictly *need* to configure MPS settings in a paravirtual front-end, but the PCI core does other device configuration in this path:

pci_scan_child_bus
pci_scan_slot
pci_scan_single_device
pci_device_add
pci_configure_device
pci_init_capabilities

and I would like to eventually migrate the MPS configuration into that same path. Since we do this other configuration for HV devices already, I think pcie_bus_configure_settings() should also work (even if it ends up not doing anything to real devices).

Quoted 7 lines

It would make that eventual migration easier if all the pci_scan_child_bus() callers had the same pattern of calling pcie_bus_configure_settings().

So would it make sense to make a similar patch for HV? It looks easy to add it to create_root_hv_pci_bus(), but I don't know exactly what to do about pci_devices_present_work().

Bjorn,

We will take a look.

K. Y

Quoted 13 lines

---

Bjorn Helgaas (3):
PCI: xilinx: Configure PCIe MPS settings
PCI: versatile: Configure PCIe MPS settings
PCI: xgene: Configure PCIe MPS settings

 drivers/pci/host/pci-versatile.c |    4 +++-
 drivers/pci/host/pci-xgene.c     |    4 +++-
 drivers/pci/host/pcie-xilinx.c   |    4 +++-
 3 files changed, 9 insertions(+), 3 deletions(-)
author Applied
#7 lore raw

cover PCI: Configure PCIe MPS settings

In reply to Bjorn Helgaas
Quoted 44 lines · Bjorn Helgaas

[Some of you will get this twice because Gmail and I aren't getting along today; sorry]

The PCI core doesn't configure the PCIe MPS settings by itself. Each host bridge driver has to call pcie_bus_configure_settings() to make this happen.

Jon fixed this already for pcie-iproc.c. I propose these similar patches for other drivers.

HV guys, I included you because create_root_hv_pci_bus() is one place that calls pci_scan_child_bus() but does not call pcie_bus_configure_settings(). I know you probably don't strictly *need* to configure MPS settings in a paravirtual front-end, but the PCI core does other device configuration in this path:

pci_scan_child_bus
pci_scan_slot
pci_scan_single_device
pci_device_add
pci_configure_device
pci_init_capabilities

and I would like to eventually migrate the MPS configuration into that same path. Since we do this other configuration for HV devices already, I think pcie_bus_configure_settings() should also work (even if it ends up not doing anything to real devices).

It would make that eventual migration easier if all the pci_scan_child_bus() callers had the same pattern of calling pcie_bus_configure_settings().

So would it make sense to make a similar patch for HV? It looks easy to add it to create_root_hv_pci_bus(), but I don't know exactly what to do about pci_devices_present_work().

---

Bjorn Helgaas (3):
PCI: xilinx: Configure PCIe MPS settings
PCI: versatile: Configure PCIe MPS settings
PCI: xgene: Configure PCIe MPS settings

Quoted 4 lines
 drivers/pci/host/pci-versatile.c |    4 +++-
 drivers/pci/host/pci-xgene.c     |    4 +++-
 drivers/pci/host/pcie-xilinx.c   |    4 +++-
 3 files changed, 9 insertions(+), 3 deletions(-)

I applied these to pci/host-xilinx, pci/host-versatile, and pci/host-xgene, respectively, for v4.11.

#8 lore raw

cover PCI: Configure PCIe MPS settings

Quoted 36 lines

From: devel [mailto:driverdev-devel.. at ...] On Behalf Of Bjorn Helgaas
Sent: Saturday, February 11, 2017 06:00
...

Quoted 32 lines · Bjorn Helgaas

The PCI core doesn't configure the PCIe MPS settings by itself. Each host bridge driver has to call pcie_bus_configure_settings() to make this happen.

Jon fixed this already for pcie-iproc.c. I propose these similar patches for other drivers.

HV guys, I included you because create_root_hv_pci_bus() is one place that calls pci_scan_child_bus() but does not call pcie_bus_configure_settings(). I know you probably don't strictly *need* to configure MPS settings in a paravirtual front-end, but the PCI core does other device configuration in this path:

pci_scan_child_bus
pci_scan_slot
pci_scan_single_device
pci_device_add
pci_configure_device
pci_init_capabilities

and I would like to eventually migrate the MPS configuration into that same path. Since we do this other configuration for HV devices already, I think pcie_bus_configure_settings() should also work (even if it ends up not doing anything to real devices).

It would make that eventual migration easier if all the pci_scan_child_bus() callers had the same pattern of calling pcie_bus_configure_settings().

So would it make sense to make a similar patch for HV? It looks easy to add it to create_root_hv_pci_bus(), but I don't know exactly what to do about pci_devices_present_work().

(I'm sorry for the really late response. I neglected the thread somehow...)

Hi Bjorn,
IMO we don't need to make any change to pci-hyperv.c: when a PCIe device is passed through to a guest on Hyper-V, in the guest the device only directly appears on a new root bus, and the new root bus doesn't have any child bus, and the new root bus itself is actually only a logical concept instead of a concrete device (i.e. it doesn't have its own BDF at all).

So, in the path pci_scan_child_bus -> ... -> pci_device_add -> pci_configure_device -> pci_configure_mps, we're sure pci_upstream_bridge() is NULL, so pci_configure_mps does nothing.

And since the root bus has no child bus, we don't need to call pcie_bus_configure_settings(child).

Then you may ask how the MPS is set.
I suppose the para-virtual PCI back-end driver on the host should take care of this properly.

FWIW, after passing through a Mellanox VF to Linux guest, in the guest I get this for the device by lspci:
DevCap: MaxPayload 512 bytes
DevCtl: MaxPayload 128 bytes, MaxReadReq 128 bytes

Thanks
-- Dexuan

#9 lore raw

cover PCI: Configure PCIe MPS settings

Quoted 33 lines · Dexuan Cui
Quoted 33 lines

From: devel [mailto:driverdev-devel.. at ...] On Behalf Of Bjorn Helgaas
Sent: Saturday, February 11, 2017 06:00
...

Quoted 29 lines · Bjorn Helgaas

The PCI core doesn't configure the PCIe MPS settings by itself. Each host bridge driver has to call pcie_bus_configure_settings() to make this happen.

Jon fixed this already for pcie-iproc.c. I propose these similar patches for other drivers.

HV guys, I included you because create_root_hv_pci_bus() is one place that calls pci_scan_child_bus() but does not call pcie_bus_configure_settings(). I know you probably don't strictly *need* to configure MPS settings in a paravirtual front-end, but the PCI core does other device configuration in this path:

pci_scan_child_bus
pci_scan_slot
pci_scan_single_device
pci_device_add
pci_configure_device
pci_init_capabilities

and I would like to eventually migrate the MPS configuration into that same path. Since we do this other configuration for HV devices already, I think pcie_bus_configure_settings() should also work (even if it ends up not doing anything to real devices).

It would make that eventual migration easier if all the pci_scan_child_bus() callers had the same pattern of calling pcie_bus_configure_settings().

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

So would it make sense to make a similar patch for HV? It looks easy to add it to create_root_hv_pci_bus(), but I don't know exactly what to do about pci_devices_present_work().

(I'm sorry for the really late response. I neglected the thread somehow...)

No problem, I'm kind of bad about that, too.

Quoted 15 lines

Hi Bjorn,
IMO we don't need to make any change to pci-hyperv.c: when a PCIe device is passed through to a guest on Hyper-V, in the guest the device only directly appears on a new root bus, and the new root bus doesn't have any child bus, and the new root bus itself is actually only a logical concept instead of a concrete device (i.e. it doesn't have its own BDF at all).

So, in the path pci_scan_child_bus -> ... -> pci_device_add -> pci_configure_device -> pci_configure_mps, we're sure pci_upstream_bridge() is NULL, so pci_configure_mps does nothing.

And since the root bus has no child bus, we don't need to call pcie_bus_configure_settings(child).

Quoted 8 lines

Then you may ask how the MPS is set.
I suppose the para-virtual PCI back-end driver on the host should take care of this properly.

FWIW, after passing through a Mellanox VF to Linux guest, in the guest I get this for the device by lspci:
DevCap: MaxPayload 512 bytes
DevCtl: MaxPayload 128 bytes, MaxReadReq 128 bytes

My question wasn't so much whether pci-hyperv.c *needs* to set MPS; I don't expect a guest to need to or even be able to configure things like that.

My question is whether it would do any harm if we did eventually do MPS configuration in the generic pci_scan_child_bus() path. If/when we do that, the Hyper-V guest will attempt to do MPS configuration. I think this would be a no-op. And I think it would also be a no-op if we made pci-hyperv.c call pcie_bus_configure_settings() today.

I'm really just looking for confirmation of that. If it is true that these are no-ops, I would consider adding a pci-hyperv.c call to pcie_bus_configure_settings() simply to make it conform to the pattern that "everything that calls pci_scan_child_bus() should also call pcie_bus_configure_settings()".

If every caller of pci_scan_child_bus() also calls pcie_bus_configure_settings(), it will make it easier in the future to fold pcie_bus_configure_settings() into pci_scan_child_bus().

Bjorn

#10 lore raw

cover PCI: Configure PCIe MPS settings

Quoted 5 lines

From: Bjorn Helgaas
...
My question wasn't so much whether pci-hyperv.c *needs* to set MPS; I don't expect a guest to need to or even be able to configure things like that.

I agree.

Quoted 4 lines

My question is whether it would do any harm if we did eventually do MPS configuration in the generic pci_scan_child_bus() path. If/when we do that, the Hyper-V guest will attempt to do MPS configuration. I think this would be a no-op.

Yes. It's a no-op, as I explained in my previous mail.

Quoted 2 lines

And I think it would also be a no-op if
we made pci-hyperv.c call pcie_bus_configure_settings() today.

Yes, I can confirm it's a no-op too.

Quoted 5 lines

I'm really just looking for confirmation of that. If it is true that these are no-ops, I would consider adding a pci-hyperv.c call to pcie_bus_configure_settings() simply to make it conform to the pattern that "everything that calls pci_scan_child_bus() should also call pcie_bus_configure_settings()".

Please help to add that. Thanks!

Quoted 5 lines

If every caller of pci_scan_child_bus() also calls pcie_bus_configure_settings(), it will make it easier in the future to fold pcie_bus_configure_settings() into pci_scan_child_bus().

Bjorn

Thanks! I see.

Thanks,
-- Dexuan