VMD: Configuring bus settings

PATCH New Jon Derrick 4 participants 6 messages Dec 13, 2018 – Jan 30, 2019

Subsystems (3)

PCI PCI native host bridge and endpoint drivers PCI driver for Intel volume management device (vmd)

lore

Share & tools

lore

Jon Derrick

author

2018-12-13 16:32 UTC #0 lore raw

PATCH 1/1 VMD: Configuring bus settings

Hi Bjorn, Lorenzo, Keith,

This patch supercedes the previous MPS on rescan patch:
https://patchwork.kernel.org/patch/10671355/

The previous patch shouldn't have negatively effected already-added devices, but it became clear that some devices required special MPS handling provided by quirks, and thus wasn't safe to perform generically.

This patch instead open-codes pci_rescan_bus() in VMD and performs bus settings prior to driver attach

Jon Derrick (1):
PCI/VMD: Configure MPS settings before adding devices

 drivers/pci/controller/vmd.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

Signature · 1 line

1.8.3.1

Jon Derrick

author

2018-12-13 16:32 UTC #1 lore raw

PATCH 1/1 PCI/VMD: Configure MPS settings before adding devices

In order to provide the most performance and/or compatible settings, ensure VMD root buses observe the pcie bus tuning settings by configuring those settings prior to adding the devices to the pcie tree.

This patch open-codes pci_rescan_bus() and configures the buses prior to adding devices and attaching drivers.

Signed-off-by Jon Derrick <[email protected]>

---
 drivers/pci/controller/vmd.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

drivers/pci/controller/vmd.c +14 -1

index e50b0b5..818dda8 100644
@@ -585,6 +585,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000, busn_start = 0;
+	struct pci_bus *child;
 
 	/*
 	 * Shadow registers may exist in certain VMD device ids which allow

@@ -712,7 +713,19 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	vmd_setup_dma_ops(vmd);
 	dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-	pci_rescan_bus(vmd->bus);
+
+	pci_scan_child_bus(vmd->bus);
+	pci_assign_unassigned_bus_resources(vmd->bus);
+
+	/*
+	 * VMD root buses are virtual and don't return true on pci_is_pcie()
+	 * and will fail pcie_bus_configure_settings() early. It can instead be
+	 * run on each of the real root ports.
+	 */
+	list_for_each_entry(child, &vmd->bus->children, node)
+		pcie_bus_configure_settings(child);
+
+	pci_bus_add_devices(vmd->bus);
 
 	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
 			       "domain"), "Can't create symlink to domain\n");

Signature · 1 line

1.8.3.1

Keith Busch

Reviewed-by on v1 1/1

2018-12-14 14:46 UTC #2 lore raw

Re: [PATCH] PCI/VMD: Configure MPS settings before adding devices

Quoted 8 lines · Jon Derrick

In order to provide the most performance and/or compatible settings, ensure VMD root buses observe the pcie bus tuning settings by configuring those settings prior to adding the devices to the pcie tree.
This patch open-codes pci_rescan_bus() and configures the buses prior to adding devices and attaching drivers.
Signed-off-by Jon Derrick <[email protected]>

Makes sense.

Reviewed-by Keith Busch <[email protected]>

Quoted 40 lines

---
 drivers/pci/controller/vmd.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

drivers/pci/controller/vmd.c +14 -1

index e50b0b5..818dda8 100644
@@ -585,6 +585,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000, busn_start = 0;
+	struct pci_bus *child;
 
 	/*
 	 * Shadow registers may exist in certain VMD device ids which allow

@@ -712,7 +713,19 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	vmd_setup_dma_ops(vmd);
 	dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-	pci_rescan_bus(vmd->bus);
+
+	pci_scan_child_bus(vmd->bus);
+	pci_assign_unassigned_bus_resources(vmd->bus);
+
+	/*
+	 * VMD root buses are virtual and don't return true on pci_is_pcie()
+	 * and will fail pcie_bus_configure_settings() early. It can instead be
+	 * run on each of the real root ports.
+	 */
+	list_for_each_entry(child, &vmd->bus->children, node)
+		pcie_bus_configure_settings(child);
+
+	pci_bus_add_devices(vmd->bus);
 
 	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
 			       "domain"), "Can't create symlink to domain\n");

Signature · 1 line

1.8.3.1

Derrick, Jonathan

author

2019-01-14 17:56 UTC #3 lore raw

Re: [PATCH] PCI/VMD: Configure MPS settings before adding devices

Ping?

Quoted 65 lines · Keith Busch

Quoted 11 lines · Jon Derrick
In order to provide the most performance and/or compatible settings,
ensure VMD root buses observe the pcie bus tuning settings by configuring those settings prior to adding the devices to the pcie tree.
This patch open-codes pci_rescan_bus() and configures the buses prior to
adding devices and attaching drivers.
Signed-off-by Jon Derrick <[email protected]>
Makes sense.
Reviewed-by Keith Busch <[email protected]>
Quoted 48 lines
---
 drivers/pci/controller/vmd.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
drivers/pci/controller/vmd.c +1 -0
index e50b0b5..818dda8 100644
@@ -585,6 +585,7 @@ static int vmd_enable_domain(struct vmd_dev
 *vmd, unsigned long features)
 	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000, busn_start = 0;
+	struct pci_bus *child;
 
 	/*
* Shadow registers may exist in certain VMD device ids
which allow
drivers/pci/controller/vmd.c +7 -1
@@ -712,7 +713,19 @@ static int vmd_enable_domain(struct vmd_dev
 *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	vmd_setup_dma_ops(vmd);
 	dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-	pci_rescan_bus(vmd->bus);
+
+	pci_scan_child_bus(vmd->bus);
+	pci_assign_unassigned_bus_resources(vmd->bus);
+
+	/*
+	 * VMD root buses are virtual and don't return true on
 pci_is_pcie()
+	 * and will fail pcie_bus_configure_settings() early. It
can instead be
drivers/pci/controller/vmd.c +6 -0
+	 * run on each of the real root ports.
+	 */
+	list_for_each_entry(child, &vmd->bus->children, node)
+		pcie_bus_configure_settings(child);
+
+	pci_bus_add_devices(vmd->bus);
 
 	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus-
Quoted 1 line
dev.kobj,
"domain"), "Can't create symlink to
domain\n");
Signature · 1 line
1.8.3.1

Bjorn Helgaas

2019-01-29 23:14 UTC #4 lore raw

Re: [PATCH] PCI/VMD: Configure MPS settings before adding devices

In reply to Jon Derrick

Quoted 8 lines · Jon Derrick

In order to provide the most performance and/or compatible settings, ensure VMD root buses observe the pcie bus tuning settings by configuring those settings prior to adding the devices to the pcie tree.
This patch open-codes pci_rescan_bus() and configures the buses prior to adding devices and attaching drivers.
Signed-off-by Jon Derrick <[email protected]>

Sorry, Lorenzo takes care of drivers/pci/controller/*, but I had mistakenly delegated this to myself, so it probably wasn't on his radar.

My dream is that someday pcie_bus_configure_settings() will somehow be done as part of enumeration and host bridge drivers won't need to care about it, but that's obviously fanciful, so this looks fine to me.

Quoted 40 lines

---
 drivers/pci/controller/vmd.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

drivers/pci/controller/vmd.c +14 -1

index e50b0b5..818dda8 100644
@@ -585,6 +585,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000, busn_start = 0;
+	struct pci_bus *child;
 
 	/*
 	 * Shadow registers may exist in certain VMD device ids which allow

@@ -712,7 +713,19 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	vmd_setup_dma_ops(vmd);
 	dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-	pci_rescan_bus(vmd->bus);
+
+	pci_scan_child_bus(vmd->bus);
+	pci_assign_unassigned_bus_resources(vmd->bus);
+
+	/*
+	 * VMD root buses are virtual and don't return true on pci_is_pcie()
+	 * and will fail pcie_bus_configure_settings() early. It can instead be
+	 * run on each of the real root ports.
+	 */
+	list_for_each_entry(child, &vmd->bus->children, node)
+		pcie_bus_configure_settings(child);
+
+	pci_bus_add_devices(vmd->bus);
 
 	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
 			       "domain"), "Can't create symlink to domain\n");

Signature · 1 line

1.8.3.1

Lorenzo Pieralisi

Applied

2019-01-30 11:15 UTC #5 lore raw

Re: [PATCH] PCI/VMD: Configure MPS settings before adding devices

In reply to Jon Derrick

Quoted 11 lines · Jon Derrick

In order to provide the most performance and/or compatible settings, ensure VMD root buses observe the pcie bus tuning settings by configuring those settings prior to adding the devices to the pcie tree.
This patch open-codes pci_rescan_bus() and configures the buses prior to adding devices and attaching drivers.
Signed-off-by Jon Derrick <[email protected]>
---
 drivers/pci/controller/vmd.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

Applied to pci/vmd for v5.1, thanks and apologies for the delay.

Lorenzo

Quoted 36 lines

drivers/pci/controller/vmd.c +14 -1

index e50b0b5..818dda8 100644
@@ -585,6 +585,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	LIST_HEAD(resources);
 	resource_size_t offset[2] = {0};
 	resource_size_t membar2_offset = 0x2000, busn_start = 0;
+	struct pci_bus *child;
 
 	/*
 	 * Shadow registers may exist in certain VMD device ids which allow

@@ -712,7 +713,19 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 	vmd_attach_resources(vmd);
 	vmd_setup_dma_ops(vmd);
 	dev_set_msi_domain(&vmd->bus->dev, vmd->irq_domain);
-	pci_rescan_bus(vmd->bus);
+
+	pci_scan_child_bus(vmd->bus);
+	pci_assign_unassigned_bus_resources(vmd->bus);
+
+	/*
+	 * VMD root buses are virtual and don't return true on pci_is_pcie()
+	 * and will fail pcie_bus_configure_settings() early. It can instead be
+	 * run on each of the real root ports.
+	 */
+	list_for_each_entry(child, &vmd->bus->children, node)
+		pcie_bus_configure_settings(child);
+
+	pci_bus_add_devices(vmd->bus);
 
 	WARN(sysfs_create_link(&vmd->dev->dev.kobj, &vmd->bus->dev.kobj,
 			       "domain"), "Can't create symlink to domain\n");

Signature · 1 line

1.8.3.1