Skip to contentSkip to conversation

PCI: vmd: VMD to control Hotplug on its rootports

PATCH In review Nirmal Patel 3 participants 5 messages Jul 5, 2023 – Jul 24, 2023

Subsystems (3)
lore
Share & tools
lore

Nirmal Patel

author maintainer
#0 lore raw

PATCH 1/1 PCI: vmd: VMD to control Hotplug on its rootports

The hotplug functionality is broken in various combinations of guest OSes i.e. RHEL, SlES and hypervisors i.e. KVM and ESXI.

VMD enabled on Intel ADL cpus caused interrupt storm for smasung drives due to AER being enabled on VMD controlled root ports. The patch 04b12ef163d10e348db664900ae7f611b83c7a0e
("PCI: vmd: Honor APCI _OSC on PCIe features.") was added to the VMD driver to correct the issue based on the following assumption:
“Since VMD is an aperture to regular PCIe root ports, honor ACPI _OSC to disable PCIe features accordingly to resolve the issue.”

VMD as a PCIe device is an end point(type 0), not a PCIe aperture (pcie bridge). In fact VMD is a type 0 raid controller(class code). When BIOS boots, all root ports under VMD is inaccessible by BIOS, and as such, they maintain their power on default states. The VMD UEFI DXE driver loads and configure all devices under VMD. This is how AER, power management and hotplug gets enabled in UEFI, since the BIOS pci driver cannot access the root ports.

The patch worked around the interrupt storm by assigning the native ACPI states to the root ports under VMD. It assigns AER, hotplug, PME, etc. These have been restored back to the power on default state in guest OS, which says the root port hot plug enable is default OFF. At most, the work around should have only assigned AER state. An additional patch should be added to exclude hot plug from the original patch.
This will cause hot plug to start working again in the guest, as the settings implemented by the UEFI VMD DXE driver will remain in effect in Linux.

---
 drivers/pci/controller/vmd.c | 2 --
 1 file changed, 2 deletions(-)
drivers/pci/controller/vmd.c +0 -2
index 769eedeb8802..52c2461b4761 100644
@@ -701,8 +701,6 @@ static int vmd_alloc_irqs(struct vmd_dev *vmd)
 static void vmd_copy_host_bridge_flags(struct pci_host_bridge *root_bridge,
 				       struct pci_host_bridge *vmd_bridge)
 {
-	vmd_bridge->native_pcie_hotplug = root_bridge->native_pcie_hotplug;
-	vmd_bridge->native_shpc_hotplug = root_bridge->native_shpc_hotplug;
 	vmd_bridge->native_aer = root_bridge->native_aer;
 	vmd_bridge->native_pme = root_bridge->native_pme;
 	vmd_bridge->native_ltr = root_bridge->native_ltr;
Signature · 1 line
2.31.1
#1 lore raw

[+cc Jonathan, Lorenzo, Krzysztof, Rob (from MAINTAINERS)]

Can you make the subject line say what the patch does? Repeating "VMD" is probably unnecessary.

Quoted 2 lines · Nirmal Patel

The hotplug functionality is broken in various combinations of guest OSes i.e. RHEL, SlES and hypervisors i.e. KVM and ESXI.

"SLES"

Quoted 2 lines

VMD enabled on Intel ADL cpus caused interrupt storm for smasung drives due to AER being enabled on VMD controlled root ports.

"Samsung"

Enabling AER should not cause an interrupt storm. There must be something else going on in addition. Are you saying the Samsung drives have some AER-related defect, like generating Correctable Errors when they shouldn't? It doesn't sound like "Intel ADL" would be relevant here.

It's not clear to me if this is directly related to *hotplug* or if it's an AER issue that may happen because of hotplug and possible for other reasons.

Quoted 1 line

The patch 04b12ef163d10e348db664900ae7f611b83c7a0e

12 char SHA1 is sufficient.

Quoted 6 lines

("PCI: vmd: Honor APCI _OSC on PCIe features.") was added to the VMD driver to correct the issue based on the following assumption:
“Since VMD is an aperture to regular PCIe root ports, honor ACPI _OSC to disable PCIe features accordingly to resolve the issue.”

Quoted 7 lines

VMD as a PCIe device is an end point(type 0), not a PCIe aperture (pcie bridge). In fact VMD is a type 0 raid controller(class code). When BIOS boots, all root ports under VMD is inaccessible by BIOS, and as such, they maintain their power on default states. The VMD UEFI DXE driver loads and configure all devices under VMD. This is how AER, power management and hotplug gets enabled in UEFI, since the BIOS pci driver cannot access the root ports.

s/pcie/PCIe/
s/pci/PCI/
s/raid/RAID/

Quoted 1 line

The patch worked around the interrupt storm by assigning the native

I assume "the patch" means 04b12ef163d1. Sometimes people write "the patch does X" in the commit log for the current patch, so it's nice to be specific to avoid confusion.

Quoted 6 lines

ACPI states to the root ports under VMD. It assigns AER, hotplug, PME, etc. These have been restored back to the power on default state in guest OS, which says the root port hot plug enable is default OFF. At most, the work around should have only assigned AER state. An additional patch should be added to exclude hot plug from the original patch.

Add blank line between paragraphs.

Quoted 3 lines

This will cause hot plug to start working again in the guest, as the settings implemented by the UEFI VMD DXE driver will remain in effect in Linux.

This is a lot of description that doesn't seem to say what the actual underlying issue is. It's basically "if we treat hotplug differently from other negotiated features, things work better." And it seems like it depends on what the UEFI driver did, and we should try to avoid a dependency there.

If the issue is too many Correctable Errors being reported via AER, we have that problem regardless of VMD, and we should handle it by rate-limiting and/or suppressing them completely for particularly offensive devices. We have some issues like this that are still outstanding:

[1] https://lore.kernel.org/linux-pci/DM6PR04MB6473197DBD89FF4643CC391F8BC19@DM6PR04MB6473.namprd04.prod.outlook.com/
[2] https://lore.kernel.org/linux-pci/[email protected]/

Quoted 21 lines
---
 drivers/pci/controller/vmd.c | 2 --
 1 file changed, 2 deletions(-)
drivers/pci/controller/vmd.c +0 -2
index 769eedeb8802..52c2461b4761 100644
@@ -701,8 +701,6 @@ static int vmd_alloc_irqs(struct vmd_dev *vmd)
 static void vmd_copy_host_bridge_flags(struct pci_host_bridge *root_bridge,
 				       struct pci_host_bridge *vmd_bridge)
 {
-	vmd_bridge->native_pcie_hotplug = root_bridge->native_pcie_hotplug;
-	vmd_bridge->native_shpc_hotplug = root_bridge->native_shpc_hotplug;
 	vmd_bridge->native_aer = root_bridge->native_aer;
 	vmd_bridge->native_pme = root_bridge->native_pme;
 	vmd_bridge->native_ltr = root_bridge->native_ltr;
Signature · 1 line
2.31.1
#2 lore raw
Quoted 54 lines · Bjorn Helgaas

[+cc Jonathan, Lorenzo, Krzysztof, Rob (from MAINTAINERS)]

Can you make the subject line say what the patch does? Repeating "VMD" is probably unnecessary.

Quoted 2 lines · Nirmal Patel

The hotplug functionality is broken in various combinations of guest OSes i.e. RHEL, SlES and hypervisors i.e. KVM and ESXI.

"SLES"

Quoted 2 lines

VMD enabled on Intel ADL cpus caused interrupt storm for smasung drives due to AER being enabled on VMD controlled root ports.

"Samsung"

Enabling AER should not cause an interrupt storm. There must be something else going on in addition. Are you saying the Samsung drives have some AER-related defect, like generating Correctable Errors when they shouldn't? It doesn't sound like "Intel ADL" would be relevant here.

It's not clear to me if this is directly related to *hotplug* or if it's an AER issue that may happen because of hotplug and possible for other reasons.

Quoted 1 line

The patch 04b12ef163d10e348db664900ae7f611b83c7a0e

12 char SHA1 is sufficient.

Quoted 6 lines

("PCI: vmd: Honor APCI _OSC on PCIe features.") was added to the VMD driver to correct the issue based on the following assumption:
“Since VMD is an aperture to regular PCIe root ports, honor ACPI _OSC to disable PCIe features accordingly to resolve the issue.”

Quoted 7 lines

VMD as a PCIe device is an end point(type 0), not a PCIe aperture (pcie bridge). In fact VMD is a type 0 raid controller(class code). When BIOS boots, all root ports under VMD is inaccessible by BIOS, and as such, they maintain their power on default states. The VMD UEFI DXE driver loads and configure all devices under VMD. This is how AER, power management and hotplug gets enabled in UEFI, since the BIOS pci driver cannot access the root ports.

s/pcie/PCIe/
s/pci/PCI/
s/raid/RAID/

Quoted 1 line

The patch worked around the interrupt storm by assigning the native

I assume "the patch" means 04b12ef163d1. Sometimes people write "the patch does X" in the commit log for the current patch, so it's nice to be specific to avoid confusion.

Quoted 12 lines
Quoted 6 lines

ACPI states to the root ports under VMD. It assigns AER, hotplug, PME, etc. These have been restored back to the power on default state in guest OS, which says the root port hot plug enable is default OFF. At most, the work around should have only assigned AER state. An additional patch should be added to exclude hot plug from the original patch.

Add blank line between paragraphs.

Quoted 3 lines

This will cause hot plug to start working again in the guest, as the settings implemented by the UEFI VMD DXE driver will remain in effect in Linux.

So hotplug will work in the guest per UEFI VMD DXE driver, but AER will be determined by the host native_aer state?

Quoted 7 lines

This is a lot of description that doesn't seem to say what the actual underlying issue is. It's basically "if we treat hotplug differently from other negotiated features, things work better." And it seems like it depends on what the UEFI driver did, and we should try to avoid a dependency there.

Quoted 5 lines

If the issue is too many Correctable Errors being reported via AER, we have that problem regardless of VMD, and we should handle it by rate-limiting and/or suppressing them completely for particularly offensive devices. We have some issues like this that are still outstanding:

I'm curious if Non-VMD root ports on the system see the same CE storm. Either way, rate-limiting per device seems like a good idea.

Quoted 25 lines

[1] https://lore.kernel.org/linux-pci/DM6PR04MB6473197DBD89FF4643CC391F8BC19@DM6PR04MB6473.namprd04.prod.outlook.com/
[2] https://lore.kernel.org/linux-pci/[email protected]/

Quoted 21 lines
---
  drivers/pci/controller/vmd.c | 2 --
  1 file changed, 2 deletions(-)
drivers/pci/controller/vmd.c +0 -2
index 769eedeb8802..52c2461b4761 100644
@@ -701,8 +701,6 @@ static int vmd_alloc_irqs(struct vmd_dev *vmd)
  static void vmd_copy_host_bridge_flags(struct pci_host_bridge *root_bridge,
  				       struct pci_host_bridge *vmd_bridge)
  {
-	vmd_bridge->native_pcie_hotplug = root_bridge->native_pcie_hotplug;
-	vmd_bridge->native_shpc_hotplug = root_bridge->native_shpc_hotplug;
  	vmd_bridge->native_aer = root_bridge->native_aer;
  	vmd_bridge->native_pme = root_bridge->native_pme;
  	vmd_bridge->native_ltr = root_bridge->native_ltr;
Signature · 1 line
2.31.1
author maintainer
#3 lore raw
Quoted 64 lines · Jonathan Derrick
Quoted 61 lines · Bjorn Helgaas

[+cc Jonathan, Lorenzo, Krzysztof, Rob (from MAINTAINERS)]

Can you make the subject line say what the patch does?  Repeating "VMD" is probably unnecessary.

Quoted 2 lines · Nirmal Patel

The hotplug functionality is broken in various combinations of guest OSes i.e. RHEL, SlES and hypervisors i.e. KVM and ESXI.

"SLES"

Quoted 2 lines

VMD enabled on Intel ADL cpus caused interrupt storm for smasung drives due to AER being enabled on VMD controlled root ports.

"Samsung"

Enabling AER should not cause an interrupt storm.  There must be something else going on in addition.  Are you saying the Samsung drives have some AER-related defect, like generating Correctable Errors when they shouldn't?  It doesn't sound like "Intel ADL" would be relevant here.

It's not clear to me if this is directly related to *hotplug* or if it's an AER issue that may happen because of hotplug and possible for other reasons.

Quoted 1 line

The patch 04b12ef163d10e348db664900ae7f611b83c7a0e

12 char SHA1 is sufficient.

Quoted 6 lines

("PCI: vmd: Honor APCI _OSC on PCIe features.") was added to the VMD driver to correct the issue based on the following assumption:
    “Since VMD is an aperture to regular PCIe root ports, honor ACPI _OSC to disable PCIe features accordingly to resolve the issue.”

Quoted 7 lines

VMD as a PCIe device is an end point(type 0), not a PCIe aperture (pcie bridge). In fact VMD is a type 0 raid controller(class code). When BIOS boots, all root ports under VMD is inaccessible by BIOS, and as such, they maintain their power on default states. The VMD UEFI DXE driver loads and configure all devices under VMD. This is how AER, power management and hotplug gets enabled in UEFI, since the BIOS pci driver cannot access the root ports.

s/pcie/PCIe/
s/pci/PCI/
s/raid/RAID/

Quoted 1 line

The patch worked around the interrupt storm by assigning the native

I assume "the patch" means 04b12ef163d1.  Sometimes people write "the patch does X" in the commit log for the current patch, so it's nice to be specific to avoid confusion.

Quoted 6 lines

ACPI states to the  root ports under VMD. It assigns AER, hotplug, PME, etc. These have been restored back to the power on default state in guest OS, which says the root port hot plug enable is default OFF. At most, the work around should have only assigned AER state. An additional patch should be added to exclude hot plug from the original patch.

Quoted 6 lines
Quoted 5 lines

Add blank line between paragraphs.

Quoted 3 lines

This will cause hot plug to start working again in the guest, as the settings implemented by the UEFI VMD DXE driver will remain in effect in Linux.

So hotplug will work in the guest per UEFI VMD DXE driver, but AER will be determined by the host native_aer state?

Yes. Guest doesn't have UEFI driver to configure rootports as in case of Host.

Quoted 8 lines
Quoted 8 lines

This is a lot of description that doesn't seem to say what the actual underlying issue is.  It's basically "if we treat hotplug differently from other negotiated features, things work better."  And it seems like it depends on what the UEFI driver did, and we should try to avoid a dependency there.

Quoted 7 lines
Quoted 5 lines

If the issue is too many Correctable Errors being reported via AER, we have that problem regardless of VMD, and we should handle it by rate-limiting and/or suppressing them completely for particularly offensive devices.  We have some issues like this that are still outstanding:

I'm curious if Non-VMD root ports on the system see the same CE storm. Either way, rate-limiting per device seems like a good idea.

AER is not the issue here and we can leave them as it is. I added unnecessary description which caused more confusion. My apologies.

Quoted 26 lines
Quoted 26 lines

[1] https://lore.kernel.org/linux-pci/DM6PR04MB6473197DBD89FF4643CC391F8BC19@DM6PR04MB6473.namprd04.prod.outlook.com/
[2] https://lore.kernel.org/linux-pci/[email protected]/

Quoted 21 lines

---
  drivers/pci/controller/vmd.c | 2 --

  1 file changed, 2 deletions(-)
drivers/pci/controller/vmd.c +0 -2
index 769eedeb8802..52c2461b4761 100644
@@ -701,8 +701,6 @@ static int vmd_alloc_irqs(struct vmd_dev *vmd)
   static void vmd_copy_host_bridge_flags(struct pci_host_bridge *root_bridge,
                          struct pci_host_bridge *vmd_bridge)
   {
-    vmd_bridge->native_pcie_hotplug = root_bridge->native_pcie_hotplug;
-    vmd_bridge->native_shpc_hotplug = root_bridge->native_shpc_hotplug;
       vmd_bridge->native_aer = root_bridge->native_aer;
       vmd_bridge->native_pme = root_bridge->native_pme;
       vmd_bridge->native_ltr = root_bridge->native_ltr;

--
2.31.1

author maintainer
#4 lore raw
In reply to Bjorn Helgaas
Quoted 9 lines · Bjorn Helgaas

[+cc Jonathan, Lorenzo, Krzysztof, Rob (from MAINTAINERS)]

Can you make the subject line say what the patch does? Repeating "VMD" is probably unnecessary.

Quoted 2 lines · Nirmal Patel

The hotplug functionality is broken in various combinations of guest OSes i.e. RHEL, SlES and hypervisors i.e. KVM and ESXI.

"SLES"

Will fix it.

Quoted 4 lines
Quoted 3 lines

VMD enabled on Intel ADL cpus caused interrupt storm for smasung drives due to AER being enabled on VMD controlled root ports.

"Samsung"

Will fix it.

Quoted 7 lines

Enabling AER should not cause an interrupt storm. There must be something else going on in addition. Are you saying the Samsung drives have some AER-related defect, like generating Correctable Errors when they shouldn't? It doesn't sound like "Intel ADL" would be relevant here.

Quoted 6 lines

It's not clear to me if this is directly related to *hotplug* or if it's an AER issue that may happen because of hotplug and possible for other reasons.

Quoted 1 line

The patch 04b12ef163d10e348db664900ae7f611b83c7a0e

12 char SHA1 is sufficient.

Will fix it.

Quoted 14 lines
Quoted 7 lines

("PCI: vmd: Honor APCI _OSC on PCIe features.") was added to the VMD driver to correct the issue based on the following assumption:
“Since VMD is an aperture to regular PCIe root ports, honor ACPI _OSC to disable PCIe features accordingly to resolve the issue.”

Quoted 7 lines

VMD as a PCIe device is an end point(type 0), not a PCIe aperture (pcie bridge). In fact VMD is a type 0 raid controller(class code). When BIOS boots, all root ports under VMD is inaccessible by BIOS, and as such, they maintain their power on default states. The VMD UEFI DXE driver loads and configure all devices under VMD. This is how AER, power management and hotplug gets enabled in UEFI, since the BIOS pci driver cannot access the root ports.

Quoted 3 lines

s/pcie/PCIe/
s/pci/PCI/
s/raid/RAID/

Will fix it.

Quoted 6 lines
Quoted 2 lines

The patch worked around the interrupt storm by assigning the native

I assume "the patch" means 04b12ef163d1. Sometimes people write "the patch does X" in the commit log for the current patch, so it's nice to be specific to avoid confusion.

Quoted 7 lines
Quoted 6 lines

ACPI states to the root ports under VMD. It assigns AER, hotplug, PME, etc. These have been restored back to the power on default state in guest OS, which says the root port hot plug enable is default OFF. At most, the work around should have only assigned AER state. An additional patch should be added to exclude hot plug from the original patch.

Add blank line between paragraphs.

Will fix it.

Quoted 10 lines
Quoted 4 lines

This will cause hot plug to start working again in the guest, as the settings implemented by the UEFI VMD DXE driver will remain in effect in Linux.

This is a lot of description that doesn't seem to say what the actual underlying issue is. It's basically "if we treat hotplug differently from other negotiated features, things work better." And it seems like it depends on what the UEFI driver did, and we should try to avoid a dependency there.

Quoted 8 lines

If the issue is too many Correctable Errors being reported via AER, we have that problem regardless of VMD, and we should handle it by rate-limiting and/or suppressing them completely for particularly offensive devices. We have some issues like this that are still outstanding:

[1] https://lore.kernel.org/linux-pci/DM6PR04MB6473197DBD89FF4643CC391F8BC19@DM6PR04MB6473.namprd04.prod.outlook.com/
[2] https://lore.kernel.org/linux-pci/[email protected]/

Sorry for the long description. I think your understanding of treating hotplug differently from other features is correct. The hotplug issue is seen only in guest because guest BIOS doesn't know about such settings nor have an implementation. Ideally, when hotplug is enabled in Host BIOS, VMD root ports in guest should obtain same values with the help of direct assign feature. The patch (04b12ef163d1) overwrites these values and assigns default values(i.e 0h for hotplug) resulting in hotplug disabled.

Regarding AER, I think we should leave other settings as it is. During the initialization, AER is enabled on VMD root-port without knowing if the platform firmware has a handler. Thanks to the patch(04b12ef163d1), this mistake was corrected.

Quoted 21 lines
Quoted 21 lines
---
 drivers/pci/controller/vmd.c | 2 --
 1 file changed, 2 deletions(-)
drivers/pci/controller/vmd.c +0 -2
index 769eedeb8802..52c2461b4761 100644
@@ -701,8 +701,6 @@ static int vmd_alloc_irqs(struct vmd_dev *vmd)
 static void vmd_copy_host_bridge_flags(struct pci_host_bridge *root_bridge,
 				       struct pci_host_bridge *vmd_bridge)
 {
-	vmd_bridge->native_pcie_hotplug = root_bridge->native_pcie_hotplug;
-	vmd_bridge->native_shpc_hotplug = root_bridge->native_shpc_hotplug;
 	vmd_bridge->native_aer = root_bridge->native_aer;
 	vmd_bridge->native_pme = root_bridge->native_pme;
 	vmd_bridge->native_ltr = root_bridge->native_ltr;
Signature · 1 line
2.31.1