PCI: vmd: Enable Hotplug based on BIOS setting on VMD rootports
Subsystems (3)
Nirmal Patel
maintainerPATCH v2 1/1 PCI: vmd: Enable Hotplug based on BIOS setting on VMD rootports
Currently VMD copies root bridge setting to enable Hotplug on its rootports. This mechanism works fine for Host OS and no issue has been observed. However in case of VM, all the HyperVisors don't pass the Hotplug setting to the guest BIOS which results in assigning default values and disabling Hotplug capability in the guest which have been observed by many OEMs.
VMD Hotplug can be enabled or disabled based on the VMD rootports' Hotplug configuration in BIOS. is_hotplug_bridge is set on each VMD rootport based on Hotplug capable bit in SltCap in probe.c. Check is_hotplug_bridge and enable or disable native_pcie_hotplug based on that value.
This patch will make sure that Hotplug is enabled properly in Host as well as in VM while honoring _OSC settings as well as VMD hotplug setting.
---v1->v2: Updating commit message.
--- drivers/pci/controller/vmd.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/pci/controller/vmd.c +10 -1
@@ -720,6 +720,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)resource_size_t membar2_offset =0x2000 ;struct pci_bus *child;struct pci_dev *dev; +struct pci_host_bridge *vmd_bridge;int ret;/*
@@ -886,8 +887,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)* and will fail pcie_bus_configure_settings() early. It can instead be * run on each of the real root ports. */ -list_for_each_entry (child, &vmd->bus->children, node) + vmd_bridge =to_pci_host_bridge (vmd->bus->bridge); +list_for_each_entry (child, &vmd->bus->children, node) {pcie_bus_configure_settings (child); +/* +* When Hotplug is enabled on vmd root-port, enable it on vmd +* bridge. +*/ +if (child->self->is_hotplug_bridge) + vmd_bridge->native_pcie_hotplug =1 ; + }pci_bus_add_devices (vmd->bus);
Signature · 1 line
2.31.1
Quoted 6 lines · Nirmal Patel
Currently VMD copies root bridge setting to enable Hotplug on its rootports. This mechanism works fine for Host OS and no issue has been observed. However in case of VM, all the HyperVisors don't pass the Hotplug setting to the guest BIOS which results in assigning default values and disabling Hotplug capability in the guest which have been observed by many OEMs.
Can we make this a little more specific? I'm lacking a lot of virtualization background here, so I'm going to ask a bunch of stupid questions here:
"VMD copies root bridge setting" refers to _OSC and the copying done in vmd_copy_host_bridge_flags()? Obviously this must be in the Host OS.
"This mechanism works fine for Host OS and no issue has been observed." I guess this means that if the platform grants hotplug control to the Host OS via _OSC, pciehp in the Host OS works fine to manage hotplug on Root Ports below the VMD?
If the platform does *not* grant hotplug control to the Host OS, I guess it means the Host OS pciehp can *not* manage hotplug below VMD? Is that also what you expect?
"However in case of VM, all the HyperVisors don't pass the Hotplug setting to the guest BIOS" What is the mechanism by which they would pass the hotplug setting? I guess the guest probably doesn't see the VMD endpoint itself, right? The guest sees either virtualized or passed-through VMD Root Ports?
I assume the guest OS sees a constructed ACPI system (different from the ACPI environment the host sees), so it sees a PNP0A03 host bridge with _OSC? And presumably VMD Root Ports below that host bridge?
So are you saying the _OSC seen by the guest doesn't grant hotplug control to the guest OS? And it doesn't do that because the guest BIOS hasn't implemented _OSC? Or maybe the guest BIOS relies on the Slot Capabilities registers of VMD Root Ports to decide whether _OSC should grant hotplug control? And those Slot Capabilities don't advertise hotplug support?
"... which results in assigning default values and disabling Hotplug capability in the guest." What default values are these? Is this talking about the default host_bridge->native_pcie_hotplug value set in acpi_pci_root_create(), where the OS assumes hotplug is owned by the platform unless _OSC grants control to the OS?
Quoted 9 lines
VMD Hotplug can be enabled or disabled based on the VMD rootports' Hotplug configuration in BIOS.
is_hotplug_bridgeis set on each VMD rootport based on Hotplug capable bit in SltCap inprobe.c. Checkis_hotplug_bridgeand enable or disablenative_pcie_hotplugbased on that value.This patch will make sure that Hotplug is enabled properly in Host as well as in VM while honoring
_OSCsettings as well as VMD hotplug setting.
"Hotplug is enabled properly in Host as well as in VM" sounds like we're talking about both the host OS and the guest OS.
In the host OS, this overrides whatever was negotiated via _OSC, I guess on the principle that _OSC doesn't apply because the host BIOS doesn't know about the Root Ports below the VMD. (I'm not sure it's 100% resolved that _OSC doesn't apply to them, so we should mention that assumption here.)
In the guest OS, this still overrides whatever was negotiated via _OSC, but presumably the guest BIOS *does* know about the Root Ports, so I don't think the same principle about overriding _OSC applies there.
I think it's still a problem that we handle host_bridge->native_pcie_hotplug differently than all the other features negotiated via _OSC. We should at least explain this somehow.
I suspect part of the reason for doing it differently is to avoid the interrupt storm that we avoid via 04b12ef163d1 ("PCI: vmd: Honor ACPI _OSC on PCIe features"). If so, I think 04b12ef163d1 should be mentioned here because it's an important piece of the picture.
Quoted 23 lines
Signed-off-by Nirmal Patel <[email protected]>---
v1->v2: Updating commit message.--- drivers/pci/controller/vmd.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)drivers/pci/controller/vmd.c +1
-0@@ -720,6 +720,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)resource_size_t membar2_offset =0x2000 ;struct pci_bus *child;struct pci_dev *dev; +struct pci_host_bridge *vmd_bridge;int ret;/* @@ -886,8 +887,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)* and will fail pcie_bus_configure_settings() early. It can instead be * run on each of the real root ports. */
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 898
Quoted 10 lines
drivers/pci/controller/vmd.c +8
-1-list_for_each_entry (child, &vmd->bus->children, node) + vmd_bridge =to_pci_host_bridge (vmd->bus->bridge); +list_for_each_entry (child, &vmd->bus->children, node) {pcie_bus_configure_settings (child); +/* +* When Hotplug is enabled on vmd root-port, enable it on vmd +* bridge. +*/ +if (child->self->is_hotplug_bridge) + vmd_bridge->native_pcie_hotplug =1 ;
"is_hotplug_bridge" basically means PCI_EXP_SLTCAP_HPC is set, which means "the hardware of this slot is hot-plug *capable*."
Whether hotplug is *enabled* is a separate policy decision about whether the OS is allowed to operate the hotplug functionality, so I think saying "When Hotplug is enabled" in the comment is a little bit misleading.
Bjorn
Quoted 7 lines
drivers/pci/controller/vmd.c +1
-0+ }pci_bus_add_devices (vmd->bus);Signature · 1 line
2.31.1
Nirmal Patel
maintainerQuoted 8 lines · Bjorn Helgaas
Quoted 6 lines · Nirmal Patel
Currently VMD copies root bridge setting to enable Hotplug on its rootports. This mechanism works fine for Host OS and no issue has been observed. However in case of VM, all the HyperVisors don't pass the Hotplug setting to the guest BIOS which results in assigning default values and disabling Hotplug capability in the guest which have been observed by many OEMs.
Quoted 7 lines
Can we make this a little more specific? I'm lacking a lot of virtualization background here, so I'm going to ask a bunch of stupid questions here:
"VMD copies root bridge setting" refers to
_OSCand the copying done invmd_copy_host_bridge_flags()? Obviously this must be in the Host OS.
Yes. we are talking about vmd_copy_host_bridge_flags. It gets called in both Host and Guest. But it assigns different values. In Host, it assigns correct values, while in guest it assigns power-on default value 0 which in simple term disable everything including hotplug.
Quoted 5 lines
"This mechanism works fine for Host OS and no issue has been observed." I guess this means that if the platform grants hotplug control to the Host OS via
_OSC, pciehp in the Host OS works fine to manage hotplug on Root Ports below the VMD?
Yes. When platform hotplug setting is enabled, then _OSC assigns correct value to vmd root ports via vmd_copy_host_bridge_flags.
Quoted 10 lines
If the platform does *not* grant hotplug control to the Host OS, I guess it means the Host OS pciehp can *not* manage hotplug below VMD? Is that also what you expect?
"However in case of VM, all the HyperVisors don't pass the Hotplug setting to the guest BIOS" What is the mechanism by which they would pass the hotplug setting? I guess the guest probably doesn't see the VMD endpoint itself, right? The guest sees either virtualized or passed-through VMD Root Ports?
In guest, vmd is passthrough including pci topology behind vmd. The way we verify Hotplug capability is to read Slot Capabilities registers' hotplug enabled bit set on vmd root ports in Guest.
The values copied in vmd_copy_host_bridge_flags are different in Host than in Guest. I do not know what component is responsible for this in every HyperVisor. But I tested this in ESXi, HyperV, KVM and they all had the same behavior where the _OSC flags are set to power-on default values of 0 by vmd_copy_host_bridge_flags in Guest OS which is disabling hotplug.
Quoted 11 lines
I assume the guest OS sees a constructed ACPI system (different from the ACPI environment the host sees), so it sees a PNP0A03 host bridge with
_OSC? And presumably VMD Root Ports below that host bridge?So are you saying the
_OSCseen by the guest doesn't grant hotplug control to the guest OS? And it doesn't do that because the guest BIOS hasn't implemented_OSC? Or maybe the guest BIOS relies on the Slot Capabilities registers of VMD Root Ports to decide whether_OSCshould grant hotplug control? And those Slot Capabilities don't advertise hotplug support?
Currently Hotplug is controlled by _OSC in both host and Guest. In case of guest, it seems guest BIOS hasn't implemented _OSC since all the flags are set to 0 which is not the case in host.
VMD is passthrough in Guest so the slot capabilities registers are still same as what Host driver would see. That is how we can still control hotplug on vmd on both Guest and Host.
Quoted 3 lines
"... which results in assigning default values and disabling Hotplug capability in the guest." What default values are these?
0. Everything is disabled in guest. So basically _OSC is writing wrong values in guest.
Quoted 4 lines
Is this
talking about the defaulthost_bridge->native_pcie_hotplugvalue set inacpi_pci_root_create(), where the OS assumes hotplug is owned by the platform unless_OSCgrants control to the OS?
vmd driver calls pci_create_root_bus which eventually ends up calling pci_init_host_bridge where native_pcie_hotplug is set to 1. Now vmd_copy_host_bridge_flags overwrites the native_pcie_hotplug to _OSC setting of 0 in guest.
Quoted 7 lines
Quoted 7 lines
VMD Hotplug can be enabled or disabled based on the VMD rootports' Hotplug configuration in BIOS.
is_hotplug_bridgeis set on each VMD rootport based on Hotplug capable bit in SltCap inprobe.c. Checkis_hotplug_bridgeand enable or disablenative_pcie_hotplugbased on that value.
Quoted 7 lines
Quoted 4 lines
This patch will make sure that Hotplug is enabled properly in Host as well as in VM while honoring
_OSCsettings as well as VMD hotplug
setting."Hotplug is enabled properly in Host as well as in VM" sounds like we're talking about both the host OS and the guest OS.
Yes. VM means guest.
Quoted 6 lines
In the host OS, this overrides whatever was negotiated via
_OSC, I guess on the principle that_OSCdoesn't apply because the host BIOS doesn't know about the Root Ports below the VMD. (I'm not sure it's 100% resolved that_OSCdoesn't apply to them, so we should mention that assumption here.)
_OSC still controls every flag including Hotplug. I have observed that slot capabilities register has hotplug enabled only when platform has enabled the hotplug. So technically not overriding it in the host. It overrides in guest because _OSC is passing wrong/different information than the _OSC information in Host.
Also like I mentioned, slot capabilities registers are not changed in Guest because vmd is passthrough.
Quoted 10 lines
In the guest OS, this still overrides whatever was negotiated via
_OSC, but presumably the guest BIOS *does* know about the Root Ports, so I don't think the same principle about overriding_OSCapplies there.I think it's still a problem that we handle
host_bridge->native_pcie_hotplugdifferently than all the other features negotiated via_OSC. We should at least explain this somehow.
Right now this is the only way to know in Guest OS if platform has enabled Hotplug or not. We have many customers complaining about Hotplug being broken in Guest. So just trying to honor _OSC but also fixing its limitation in Guest.
Quoted 5 lines
I suspect part of the reason for doing it differently is to avoid the interrupt storm that we avoid via
04b12ef163d1("PCI: vmd: Honor ACPI_OSCon PCIe features"). If so, I think04b12ef163d1should be mentioned here because it's an important piece of the picture.
I can add a note about this patch as well. Let me know if you want me to add something specific.
Thank you for taking the time. This is a very critical issue for us and we have many customers complaining about it, I would appreciate to get it resolved as soon as possible.
Thanks
nirmal
Quoted 43 lines
Quoted 35 lines
Signed-off-by Nirmal Patel <[email protected]>---
v1->v2: Updating commit message.--- drivers/pci/controller/vmd.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)drivers/pci/controller/vmd.c +1
-0@@ -720,6 +720,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) resource_size_t membar2_offset = 0x2000;struct pci_bus *child;struct pci_dev *dev; +struct pci_host_bridge *vmd_bridge;int ret;drivers/pci/controller/vmd.c +0
-0/* @@ -886,8 +887,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) * and will fail pcie_bus_configure_settings() early. It caninstead be
drivers/pci/controller/vmd.c +4
-1* run on each of the real root ports. */ -list_for_each_entry (child, &vmd->bus->children, node) + vmd_bridge =to_pci_host_bridge (vmd->bus->bridge); +list_for_each_entry (child, &vmd->bus->children, node) {pcie_bus_configure_settings (child); +/* +* When Hotplug is enabled on vmd root-port, enable it on vmd
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 898
Quoted 4 lines
drivers/pci/controller/vmd.c +4
-0+* bridge. +*/ +if (child->self->is_hotplug_bridge) + vmd_bridge->native_pcie_hotplug =1 ;"
is_hotplug_bridge" basically meansPCI_EXP_SLTCAP_HPCis set, which means "the hardware of this slot is hot-plug *capable*."
Quoted 4 lines
Whether hotplug is *enabled* is a separate policy decision about whether the OS is allowed to operate the hotplug functionality, so I think saying "When Hotplug is enabled" in the comment is a little bit misleading.
I will rewrite the comment.
Quoted 10 lines
Bjorn
Quoted 7 lines
drivers/pci/controller/vmd.c +1
-0+ }pci_bus_add_devices (vmd->bus);Signature · 2 lines
-- 2.31.1
Quoted 13 lines · Nirmal Patel
Quoted 12 lines · Bjorn Helgaas
Quoted 6 lines · Nirmal Patel
Currently VMD copies root bridge setting to enable Hotplug on its rootports. This mechanism works fine for Host OS and no issue has been observed. However in case of VM, all the HyperVisors don't pass the Hotplug setting to the guest BIOS which results in assigning default values and disabling Hotplug capability in the guest which have been observed by many OEMs.
Can we make this a little more specific? I'm lacking a lot of virtualization background here, so I'm going to ask a bunch of stupid questions here:
Quoted 8 lines
Quoted 3 lines
"VMD copies root bridge setting" refers to
_OSCand the copying done invmd_copy_host_bridge_flags()? Obviously this must be in the Host OS.Yes. we are talking about
vmd_copy_host_bridge_flags. It gets called in both Host and Guest. But it assigns different values. In Host, it assigns correct values, while in guest it assigns power-on default value 0 which in simple term disable everything including hotplug.
If vmd_copy_host_bridge_flags() is called both in the Host and the Guest, I guess that means both the VMD endpoint and the VMD Root Ports below it are passed through to the Guest? I expected only the VMD Root Ports would be passed through, but maybe that's my misunderstanding.
The VMD endpoint is under host bridge A, and the VMD creates a new PCI domain under a new host bridge B. vmd_copy_host_bridge_flags() copies all the feature ownership flags from A to B.
On ACPI systems (like both Host and Guest) all these flags are initially cleared for host bridge A in acpi_pci_root_create() because these features are owned by the platform until _OSC grants ownership to the OS. They are initially *set* for host bridge B in pci_init_host_bridge() because it's created by the vmd driver instead of being enumerated via ACPI.
If the Host _OSC grants ownership to the OS, A's native_pcie_hotplug will be set, and vmd_copy_host_bridge_flags() leaves it set for B as well. If the Host _OSC does *not* grant hotplug ownership to the OS, native_pcie_hotplug will be cleared for both A and B.
Apparently the Guest doesn't supply _OSC (seems like a spec violation; could tell from the dmesg), so A's native_pcie_hotplug remains cleared, which means vmd_copy_host_bridge_flags() will also clear it for B.
[The lack of a Guest BIOS _OSC would also mean that if you passed through a normal non-VMD PCIe Switch Downstream Port to the Guest, the Guest OS would never be able to manage hotplug below that Port. Maybe nobody passes through Switch Ports.]
Quoted 18 lines
Quoted 4 lines
"This mechanism works fine for Host OS and no issue has been observed." I guess this means that if the platform grants hotplug control to the Host OS via
_OSC, pciehp in the Host OS works fine to manage hotplug on Root Ports below the VMD?Yes. When platform hotplug setting is enabled, then
_OSCassigns correct value to vmd root ports viavmd_copy_host_bridge_flags.Quoted 9 lines
If the platform does *not* grant hotplug control to the Host OS, I guess it means the Host OS pciehp can *not* manage hotplug below VMD? Is that also what you expect?
"However in case of VM, all the HyperVisors don't pass the Hotplug setting to the guest BIOS" What is the mechanism by which they would pass the hotplug setting? I guess the guest probably doesn't see the VMD endpoint itself, right? The guest sees either virtualized or passed-through VMD Root Ports?
Quoted 3 lines
In guest, vmd is passthrough including pci topology behind vmd. The way we verify Hotplug capability is to read Slot Capabilities registers' hotplug enabled bit set on vmd root ports in Guest.
I guess maybe this refers to set_pcie_hotplug_bridge(), which checks PCI_EXP_SLTCAP_HPC? If it's not set, pciehp won't bind to the device. This behavior is the same in Host and Guest.
Quoted 18 lines
The values copied in
vmd_copy_host_bridge_flagsare different in Host than in Guest. I do not know what component is responsible for this in every HyperVisor. But I tested this in ESXi, HyperV, KVM and they all had the same behavior where the_OSCflags are set to power-on default values of 0 byvmd_copy_host_bridge_flagsin Guest OS which is disabling hotplug.Quoted 10 lines
I assume the guest OS sees a constructed ACPI system (different from the ACPI environment the host sees), so it sees a PNP0A03 host bridge with
_OSC? And presumably VMD Root Ports below that host bridge?So are you saying the
_OSCseen by the guest doesn't grant hotplug control to the guest OS? And it doesn't do that because the guest BIOS hasn't implemented_OSC? Or maybe the guest BIOS relies on the Slot Capabilities registers of VMD Root Ports to decide whether_OSCshould grant hotplug control? And those Slot Capabilities don't advertise hotplug support?
Quoted 3 lines
Currently Hotplug is controlled by
_OSCin both host and Guest. In case of guest, it seems guest BIOS hasn't implemented_OSCsince all the flags are set to 0 which is not the case in host.
I think you want pciehp to work on the VMD Root Ports in the Guest, no matter what, on the assumption that any _OSC that applies to host bridge A does not apply to the host bridge created by the vmd driver.
If so, we should just revert 04b12ef163d1 ("PCI: vmd: Honor ACPI _OSC on PCIe features"). Since host bridge B was not enumerated via ACPI, the OS owns all those features under B by default, so reverting 04b12ef163d1 would leave that state.
Obviously we'd have to first figure out another solution for the AER message flood that 04b12ef163d1 resolved.
Quoted 137 lines
VMD is passthrough in Guest so the slot capabilities registers are still same as what Host driver would see. That is how we can still control hotplug on vmd on both Guest and Host.
Quoted 3 lines
"... which results in assigning default values and disabling Hotplug capability in the guest." What default values are these?
0. Everything is disabled in guest. So basically
_OSCis writing wrong values in guest.Quoted 3 lines
Is this talking about the default
host_bridge->native_pcie_hotplugvalue set inacpi_pci_root_create(), where the OS assumes hotplug is owned by the platform unless_OSCgrants control to the OS?vmd driver calls
pci_create_root_buswhich eventually ends up callingpci_init_host_bridgewherenative_pcie_hotplugis set to 1. Nowvmd_copy_host_bridge_flagsoverwrites thenative_pcie_hotplugto_OSCsetting of 0 in guest.Quoted 6 lines
Quoted 6 lines
VMD Hotplug can be enabled or disabled based on the VMD rootports' Hotplug configuration in BIOS.
is_hotplug_bridgeis set on each VMD rootport based on Hotplug capable bit in SltCap inprobe.c. Checkis_hotplug_bridgeand enable or disablenative_pcie_hotplugbased on that value.Quoted 6 lines
Quoted 3 lines
This patch will make sure that Hotplug is enabled properly in Host as well as in VM while honoring
_OSCsettings as well as VMD hotplug setting."Hotplug is enabled properly in Host as well as in VM" sounds like we're talking about both the host OS and the guest OS.
Yes. VM means guest.
Quoted 6 lines
In the host OS, this overrides whatever was negotiated via
_OSC, I guess on the principle that_OSCdoesn't apply because the host BIOS doesn't know about the Root Ports below the VMD. (I'm not sure it's 100% resolved that_OSCdoesn't apply to them, so we should mention that assumption here.)
_OSCstill controls every flag including Hotplug. I have observed that slot capabilities register has hotplug enabled only when platform has enabled the hotplug. So technically not overriding it in the host.It overrides in guest because
_OSCis passing wrong/different information than the_OSCinformation in Host. Also like I mentioned, slot capabilities registers are not changed in Guest because vmd is passthrough.Quoted 10 lines
In the guest OS, this still overrides whatever was negotiated via
_OSC, but presumably the guest BIOS *does* know about the Root Ports, so I don't think the same principle about overriding_OSCapplies there.I think it's still a problem that we handle
host_bridge->native_pcie_hotplugdifferently than all the other features negotiated via_OSC. We should at least explain this somehow.Right now this is the only way to know in Guest OS if platform has enabled Hotplug or not. We have many customers complaining about Hotplug being broken in Guest. So just trying to honor
_OSCbut also fixing its limitation in Guest.Quoted 6 lines
I suspect part of the reason for doing it differently is to avoid the interrupt storm that we avoid via
04b12ef163d1("PCI: vmd:
Honor ACPI_OSCon PCIe features"). If so, I think04b12ef163d1should be mentioned here because it's an important piece of the picture.I can add a note about this patch as well. Let me know if you want me to add something specific.
Thank you for taking the time. This is a very critical issue for us and we have many customers complaining about it, I would appreciate to get it resolved as soon as possible.
Quoted 42 lines
Quoted 34 lines
Signed-off-by Nirmal Patel <[email protected]>---
v1->v2: Updating commit message.--- drivers/pci/controller/vmd.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)drivers/pci/controller/vmd.c +1
-0@@ -720,6 +720,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) resource_size_t membar2_offset = 0x2000;struct pci_bus *child;struct pci_dev *dev; +struct pci_host_bridge *vmd_bridge;int ret;drivers/pci/controller/vmd.c +0
-0/* @@ -886,8 +887,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) * and will fail pcie_bus_configure_settings() early. It caninstead be
drivers/pci/controller/vmd.c +4
-1* run on each of the real root ports. */ -list_for_each_entry (child, &vmd->bus->children, node) + vmd_bridge =to_pci_host_bridge (vmd->bus->bridge); +list_for_each_entry (child, &vmd->bus->children, node) {pcie_bus_configure_settings (child); +/* +* When Hotplug is enabled on vmd root-port, enable it on vmd
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 898
Quoted 4 lines
drivers/pci/controller/vmd.c +4
-0+* bridge. +*/ +if (child->self->is_hotplug_bridge) + vmd_bridge->native_pcie_hotplug =1 ;"
is_hotplug_bridge" basically meansPCI_EXP_SLTCAP_HPCis set, which means "the hardware of this slot is hot-plug *capable*."Quoted 4 lines
Whether hotplug is *enabled* is a separate policy decision about whether the OS is allowed to operate the hotplug functionality, so I think saying "When Hotplug is enabled" in the comment is a little bit misleading.
I will rewrite the comment.
Quoted 10 lines
Bjorn
Quoted 7 lines
drivers/pci/controller/vmd.c +1
-0+ }pci_bus_add_devices (vmd->bus);Signature · 2 lines
-- 2.31.1
Nirmal Patel
maintainerQuoted 141 lines · Bjorn Helgaas
Quoted 23 lines · Nirmal Patel
Quoted 16 lines · Bjorn Helgaas
Quoted 9 lines · Nirmal Patel
Currently VMD copies root bridge setting to enable Hotplug on its
rootports. This mechanism works fine for Host OS and no issue has
been observed. However in case of VM, all the HyperVisors don't pass the Hotplug setting to the guest BIOS which results in assigning default values and disabling Hotplug capability in the
guest which have been observed by many OEMs.Can we make this a little more specific? I'm lacking a lot of virtualization background here, so I'm going to ask a bunch of stupid
questions here:Quoted 5 lines
"VMD copies root bridge setting" refers to
_OSCand the copying done
invmd_copy_host_bridge_flags()? Obviously this must be in the Host
OS.Quoted 5 lines
Yes. we are talking about
vmd_copy_host_bridge_flags. It gets called in
both Host and Guest. But it assigns different values. In Host, it assigns correct values, while in guest it assigns power-on default value 0 which in simple term disable everything including hotplug.If
vmd_copy_host_bridge_flags()is called both in the Host and the Guest, I guess that means both the VMD endpoint and the VMD Root Ports
below it are passed through to the Guest? I expected only the VMD Root Ports would be passed through, but maybe that's my misunderstanding.The VMD endpoint is under host bridge A, and the VMD creates a new PCI
domain under a new host bridge B.vmd_copy_host_bridge_flags()copies
all the feature ownership flags from A to B.On ACPI systems (like both Host and Guest) all these flags are initially cleared for host bridge A in
acpi_pci_root_create()because these features are owned by the platform until_OSCgrants ownership to the OS. They are initially *set* for host bridge B inpci_init_host_bridge()because it's created by the vmd driver instead of being enumerated via ACPI.If the Host
_OSCgrants ownership to the OS, A'snative_pcie_hotplugwill be set, andvmd_copy_host_bridge_flags()leaves it set for B as well. If the Host_OSCdoes *not* grant hotplug ownership to the OS,native_pcie_hotplugwill be cleared for both A and B.Apparently the Guest doesn't supply
_OSC(seems like a spec violation;
could tell from the dmesg), so A'snative_pcie_hotplugremains cleared, which meansvmd_copy_host_bridge_flags()will also clear it for B.[The lack of a Guest BIOS
_OSCwould also mean that if you passed through a normal non-VMD PCIe Switch Downstream Port to the Guest, the
Guest OS would never be able to manage hotplug below that Port. Maybe
nobody passes through Switch Ports.]Quoted 25 lines
Quoted 6 lines
"This mechanism works fine for Host OS and no issue has been observed." I guess this means that if the platform grants hotplug
control to the Host OS via_OSC, pciehp in the Host OS works fine to
manage hotplug on Root Ports below the VMD?Yes. When platform hotplug setting is enabled, then
_OSCassigns correct value to vmd root ports viavmd_copy_host_bridge_flags.Quoted 6 lines
If the platform does *not* grant hotplug control to the Host OS, I
guess it means the Host OS pciehp can *not* manage hotplug below VMD?
Is that also what you expect?Quoted 8 lines
"However in case of VM, all the HyperVisors don't pass the Hotplug
setting to the guest BIOS" What is the mechanism by which they would
pass the hotplug setting? I guess the guest probably doesn't see the
VMD endpoint itself, right? The guest sees either virtualized or passed-through VMD Root Ports?Quoted 5 lines
In guest, vmd is passthrough including pci topology behind vmd. The way
we verify Hotplug capability is to read Slot Capabilities registers'
hotplug enabled bit set on vmd root ports in Guest.I guess maybe this refers to
set_pcie_hotplug_bridge(), which checksPCI_EXP_SLTCAP_HPC? If it's not set, pciehp won't bind to the device.
This behavior is the same in Host and Guest.Quoted 28 lines
The values copied in
vmd_copy_host_bridge_flagsare different in Host
than in Guest. I do not know what component is responsible for this in
every HyperVisor. But I tested this in ESXi, HyperV, KVM and they all
had the same behavior where the_OSCflags are set to power-on default
values of 0 byvmd_copy_host_bridge_flagsin Guest OS which is disabling hotplug.Quoted 6 lines
I assume the guest OS sees a constructed ACPI system (different from
the ACPI environment the host sees), so it sees a PNP0A03 host bridge
with_OSC? And presumably VMD Root Ports below that host bridge?Quoted 10 lines
So are you saying the
_OSCseen by the guest doesn't grant hotplug
control to the guest OS? And it doesn't do that because the guest
BIOS hasn't implemented_OSC? Or maybe the guest BIOS relies on the
Slot Capabilities registers of VMD Root Ports to decide whether_OSC
should grant hotplug control? And those Slot Capabilities don't advertise hotplug support?Quoted 4 lines
Currently Hotplug is controlled by
_OSCin both host and Guest. In case
of guest, it seems guest BIOS hasn't implemented_OSCsince all the flags are set to 0 which is not the case in host.I think you want pciehp to work on the VMD Root Ports in the Guest, no matter what, on the assumption that any
_OSCthat applies to host bridge A does not apply to the host bridge created by the vmd driver.
Quoted 7 lines
If so, we should just revert
04b12ef163d1("PCI: vmd: Honor ACPI_OSCon PCIe features"). Since host bridge B was not enumerated via ACPI, the OS owns all those features under B by default, so reverting04b12ef163d1would leave that state.Obviously we'd have to first figure out another solution for the AER message flood that
04b12ef163d1resolved.
Hi Bjorn,
If we revert 04b12ef163d1, then VMD driver will still enable AER blindly which is a bug. So we need to find a way to make VMD driver aware of AER platform setting and use that information to enable or disable AER for its child devices.
There is a setting in BIOS that allows us to enable OS native AER support on the platform. This setting is located in EDK Menu -> Platform configuration -> system event log -> IIO error enabling -> OS native AER support.
This setting is assigned to VMD bridge by vmd_copy_host_bridge_flags in patch 04b12ef163d1. Without the patch 04b12ef163d1, VMD driver will enable AER even if platform has disabled OS native AER support as mentioned earlier. This will result in an AER flood mentioned in 04b12e f163d1 since there is no AER handlers.
I believe the rate limit will not alone fix the issue rather will act as a work around. If VMD is aware of OS native AER support setting, then we will not see AER flooding issue.
Do you have any suggestion on how to make VMD driver aware of AER setting and keep it in sync with platform setting.
Thanks
nirmal
Quoted 147 lines
Quoted 147 lines
VMD is passthrough in Guest so the slot capabilities registers are still same as what Host driver would see. That is how we can still control hotplug on vmd on both Guest and Host.
Quoted 3 lines
"... which results in assigning default values and disabling Hotplug
capability in the guest." What default values are these?0. Everything is disabled in guest. So basically
_OSCis writing wrong
values in guest.Quoted 4 lines
Is this talking about the default host_bridge-
Quoted 1 line
native_pcie_hotplugvalue set in
acpi_pci_root_create(), where the OS assumes hotplug is owned by the platform unless_OSCgrants control to the OS?vmd driver calls
pci_create_root_buswhich eventually ends up callingpci_init_host_bridgewherenative_pcie_hotplugis set to 1. Nowvmd_copy_host_bridge_flagsoverwrites thenative_pcie_hotplugto_OSCsetting of 0 in guest.Quoted 6 lines
Quoted 6 lines
VMD Hotplug can be enabled or disabled based on the VMD rootports' Hotplug configuration in BIOS.
is_hotplug_bridgeis set on each VMD rootport based on Hotplug capable bit in SltCap inprobe.c. Checkis_hotplug_bridgeand enable or disablenative_pcie_hotplugbased on that value.Quoted 7 lines
Quoted 3 lines
This patch will make sure that Hotplug is enabled properly in Host as well as in VM while honoring
_OSCsettings as well as VMD hotplug setting."Hotplug is enabled properly in Host as well as in VM" sounds like
we're talking about both the host OS and the guest OS.Yes. VM means guest.
Quoted 9 lines
In the host OS, this overrides whatever was negotiated via
_OSC, I
guess on the principle that_OSCdoesn't apply because the host BIOS
doesn't know about the Root Ports below the VMD. (I'm not sure it's
100% resolved that_OSCdoesn't apply to them, so we should mention
that assumption here.)
_OSCstill controls every flag including Hotplug. I have observed that slot capabilities register has hotplug enabled only when platform has enabled the hotplug. So technically not overriding it in the host.It overrides in guest because
_OSCis passing wrong/different information than the_OSCinformation in Host. Also like I mentioned, slot capabilities registers are not changed in Guest because vmd is passthrough.Quoted 9 lines
In the guest OS, this still overrides whatever was negotiated via
_OSC, but presumably the guest BIOS *does* know about the Root Ports, so I don't think the same principle about overriding_OSCapplies there.I think it's still a problem that we handle
host_bridge->native_pcie_hotplugdifferently than all the other features negotiated via_OSC. We should at least explain this somehow.Right now this is the only way to know in Guest OS if platform has enabled Hotplug or not. We have many customers complaining about Hotplug being broken in Guest. So just trying to honor
_OSCbut also
fixing its limitation in Guest.Quoted 5 lines
I suspect part of the reason for doing it differently is to avoid the interrupt storm that we avoid via
04b12ef163d1("PCI: vmd:
Honor ACPI_OSCon PCIe features"). If so, I think04b12ef163d1should be mentioned here because it's an important piece of the picture.I can add a note about this patch as well. Let me know if you want me to add something specific.
Thank you for taking the time. This is a very critical issue for us and we have many customers complaining about it, I would appreciate to get it resolved as soon as possible.
Quoted 46 lines
Quoted 37 lines
Signed-off-by Nirmal Patel <[email protected]>---
v1->v2: Updating commit message.--- drivers/pci/controller/vmd.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)drivers/pci/controller/vmd.c +1
-0@@ -720,6 +720,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) resource_size_t membar2_offset = 0x2000;struct pci_bus *child;struct pci_dev *dev; +struct pci_host_bridge *vmd_bridge;int ret;drivers/pci/controller/vmd.c +0
-0/* @@ -886,8 +887,16 @@ static int vmd_enable_domain(struct
vmd_dev
*vmd, unsigned long features)
* and will failpcie_bus_configure_settings()early.
It can
instead bedrivers/pci/controller/vmd.c +4
-1* run on each of the real root ports. */ -list_for_each_entry (child, &vmd->bus->children, node) + vmd_bridge =to_pci_host_bridge (vmd->bus->bridge); +list_for_each_entry (child, &vmd->bus->children, node) {pcie_bus_configure_settings (child); +/* +* When Hotplug is enabled on vmd root-port, enable it
on vmddrivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 898
Quoted 4 lines
drivers/pci/controller/vmd.c +4
-0+* bridge. +*/ +if (child->self->is_hotplug_bridge) + vmd_bridge->native_pcie_hotplug =1 ;"
is_hotplug_bridge" basically meansPCI_EXP_SLTCAP_HPCis set, which
means "the hardware of this slot is hot-plug *capable*."Quoted 6 lines
Whether hotplug is *enabled* is a separate policy decision about whether the OS is allowed to operate the hotplug functionality, so I
think saying "When Hotplug is enabled" in the comment is a little bit
misleading.I will rewrite the comment.
Quoted 9 lines
Bjorn
Quoted 7 lines
drivers/pci/controller/vmd.c +1
-0+ }pci_bus_add_devices (vmd->bus);Signature · 2 lines
-- 2.31.1
Quoted 31 lines · Nirmal Patel
Quoted 12 lines · Bjorn Helgaas
On Tue, Dec 05, 2023 at 03:20:27PM -0700, Nirmal Patel wrote:
...Quoted 3 lines
Currently Hotplug is controlled by
_OSCin both host and Guest. In case of guest, it seems guest BIOS hasn't implemented_OSCsince all the flags are set to 0 which is not the case in host.I think you want pciehp to work on the VMD Root Ports in the Guest, no matter what, on the assumption that any
_OSCthat applies to host bridge A does not apply to the host bridge created by the vmd driver.Quoted 7 lines
If so, we should just revert
04b12ef163d1("PCI: vmd: Honor ACPI_OSCon PCIe features"). Since host bridge B was not enumerated via ACPI, the OS owns all those features under B by default, so reverting04b12ef163d1would leave that state.Obviously we'd have to first figure out another solution for the AER message flood that
04b12ef163d1resolved.If we revert
04b12ef163d1, then VMD driver will still enable AER blindly which is a bug. So we need to find a way to make VMD driver aware of AER platform setting and use that information to enable or disable AER for its child devices.There is a setting in BIOS that allows us to enable OS native AER support on the platform. This setting is located in EDK Menu -> Platform configuration -> system event log -> IIO error enabling -> OS native AER support.
Quoted 8 lines
This setting is assigned to VMD bridge by
vmd_copy_host_bridge_flagsin patch04b12ef163d1. Without the patch04b12ef163d1, VMD driver will enable AER even if platform has disabled OS native AER support as mentioned earlier. This will result in an AER flood mentioned in 04b12e f163d1 since there is no AER handlers.I believe the rate limit will not alone fix the issue rather will act as a work around.
I agree with this part. A rate limit would not solve the problem of an interrupt storm consuming the CPU. That could be addressed by switching to polling when the rate is high or possibly other ways.
Quoted 5 lines
If VMD is aware of OS native AER support setting, then we will not see AER flooding issue.
Do you have any suggestion on how to make VMD driver aware of AER setting and keep it in sync with platform setting.
Well, this is the whole problem. IIUC, you're saying we should use _OSC to negotiate for AER control below the VMD bridge, but ignore _OSC for hotplug control.
I don't see how to read the _OSC spec and decide which things apply below a VMD bridge and which things don't.
But I think a proposal that "_OSC doesn't apply to any devices below a VMD bridge," that *is* a pretty generic principle. If we think that's the right way to use _OSC, shouldn't the vmd driver be able to configure AER for devices below the VMD bridge in any way it wants?
I'm not sure what "_OSC doesn't apply below a VMD" would mean for things like firmware-first error logging below the VMD. Maybe firmware-first doesn't make sense below VMD anyway because firmware and the OS have different ideas about the Segment for those devices?
Bjorn
Nirmal Patel
maintainerQuoted 49 lines · Bjorn Helgaas
Quoted 38 lines · Nirmal Patel
Quoted 12 lines · Bjorn Helgaas
On Tue, Dec 05, 2023 at 03:20:27PM -0700, Nirmal Patel wrote:
...Quoted 3 lines
Currently Hotplug is controlled by
_OSCin both host and Guest. In case of guest, it seems guest BIOS hasn't implemented_OSCsince all the flags are set to 0 which is not the case in host.I think you want pciehp to work on the VMD Root Ports in the Guest, no matter what, on the assumption that any
_OSCthat applies to host bridge A does not apply to the host bridge created
by the vmd driver.Quoted 7 lines
If so, we should just revert
04b12ef163d1("PCI: vmd: Honor ACPI_OSCon PCIe features"). Since host bridge B was not enumerated via ACPI, the OS owns all those features under B by default, so reverting04b12ef163d1would leave that state.Obviously we'd have to first figure out another solution for the AER message flood that
04b12ef163d1resolved.If we revert
04b12ef163d1, then VMD driver will still enable AER blindly which is a bug. So we need to find a way to make VMD driver aware of AER platform setting and use that information to enable or disable AER for its child devices.There is a setting in BIOS that allows us to enable OS native AER support on the platform. This setting is located in EDK Menu -> Platform configuration -> system event log -> IIO error enabling -> OS native AER support.
This setting is assigned to VMD bridge by
vmd_copy_host_bridge_flags
in patch04b12ef163d1. Without the patch04b12ef163d1, VMD driver will enable AER even if platform has disabled OS native AER support as mentioned earlier. This will result in an AER flood mentioned in 04b12e f163d1 since there is no AER handlers.Quoted 2 lines
I believe the rate limit will not alone fix the issue rather will act as a work around.
I agree with this part. A rate limit would not solve the problem of an interrupt storm consuming the CPU. That could be addressed by switching to polling when the rate is high or possibly other ways.
Quoted 3 lines
If VMD is aware of OS native AER support setting, then we will not see AER flooding issue.
Quoted 6 lines
Quoted 2 lines
Do you have any suggestion on how to make VMD driver aware of AER setting and keep it in sync with platform setting.
Well, this is the whole problem. IIUC, you're saying we should use
_OSCto negotiate for AER control below the VMD bridge, but ignore_OSCfor hotplug control.
Because VMD has it's own hotplug BIOS setting which allows vmd to enable or disable hotplug on it's domain. However we dont have VMD specific AER, DPC, LTR settings. Is it okay if we include an additional check for hotplug? i.e. Hotplug capable bit in SltCap register which reflects VMD BIOS hotplug setting.
Another way is to overwrite _OSC setting for hotplug only in Guest OS. If we make VMD driver aware of Host or Guest environment, only in case of Guest we overwrite _OSC hotplug using SltCap register and we don't revert the 04b12ef163d1.
Quoted 17 lines
I don't see how to read the
_OSCspec and decide which things apply below a VMD bridge and which things don't.But I think a proposal that "
_OSCdoesn't apply to any devices below a
VMD bridge," that *is* a pretty generic principle. If we think that's
the right way to use_OSC, shouldn't the vmd driver be able to configure AER for devices below the VMD bridge in any way it wants?I'm not sure what "
_OSCdoesn't apply below a VMD" would mean for things like firmware-first error logging below the VMD. Maybe firmware-first doesn't make sense below VMD anyway because firmware and the OS have different ideas about the Segment for those devices?Bjorn
Quoted 33 lines · Nirmal Patel
Quoted 32 lines · Bjorn Helgaas
Quoted 31 lines · Nirmal Patel
Quoted 12 lines · Bjorn Helgaas
On Tue, Dec 05, 2023 at 03:20:27PM -0700, Nirmal Patel wrote:
...Quoted 3 lines
Currently Hotplug is controlled by
_OSCin both host and Guest. In case of guest, it seems guest BIOS hasn't implemented_OSCsince all the flags are set to 0 which is not the case in host.I think you want pciehp to work on the VMD Root Ports in the Guest, no matter what, on the assumption that any
_OSCthat applies to host bridge A does not apply to the host bridge created
by the vmd driver.Quoted 7 lines
If so, we should just revert
04b12ef163d1("PCI: vmd: Honor ACPI_OSCon PCIe features"). Since host bridge B was not enumerated via ACPI, the OS owns all those features under B by default, so reverting04b12ef163d1would leave that state.Obviously we'd have to first figure out another solution for the AER message flood that
04b12ef163d1resolved.If we revert
04b12ef163d1, then VMD driver will still enable AER blindly which is a bug. So we need to find a way to make VMD driver aware of AER platform setting and use that information to enable or disable AER for its child devices.There is a setting in BIOS that allows us to enable OS native AER support on the platform. This setting is located in EDK Menu -> Platform configuration -> system event log -> IIO error enabling -> OS native AER support.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
This setting is assigned to VMD bridge by
vmd_copy_host_bridge_flagsin patch04b12ef163d1. Without the patch04b12ef163d1, VMD driver will enable AER even if platform has disabled OS native AER support as mentioned earlier. This will result in an AER flood mentioned in04b12ef163d1since there is no AER handlers.
I missed this before. What does "there are no AER handlers" mean? Do you mean there are no *firmware* AER handlers?
The dmesg log at https://bugzilla.kernel.org/attachment.cgi?id=299571
(from https://bugzilla.kernel.org/show_bug.cgi?id=215027, the bug fixed by 04b12ef163d1), shows this:
acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] acpi PNP0A08:00: _OSC: platform does not support [AER]
acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME PCIeCapability LTR]
so the firmware did not grant AER control to the OS (I think "platform does not support" is a confusing description).
Prior to 04b12ef163d1, Linux applied _OSC above the VMD bridge but not below it, so Linux had native control below VMD and it did handle AER interrupts from the 10000:e0:06.0 Root Port below VMD:
vmd 0000:00:0e.0: PCI host bridge to bus 10000:e0 pcieport 10000:e0:06.0: AER: Corrected error received: 10000:e1:00.0
After 04b12ef163d1, Linux applied _OSC below the VMD bridge as well, so it did not enable or handle any AER interrupts. I suspect the platform didn't handle AER interrupts below VMD either, so those errors were probably just ignored.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
If VMD is aware of OS native AER support setting, then we will not see AER flooding issue.
Do you have any suggestion on how to make VMD driver aware of AER setting and keep it in sync with platform setting.
Quoted 7 lines
Quoted 3 lines
Well, this is the whole problem. IIUC, you're saying we should use
_OSCto negotiate for AER control below the VMD bridge, but ignore_OSCfor hotplug control.Because VMD has its own hotplug BIOS setting which allows vmd to enable or disable hotplug on its domain. However we don't have VMD specific AER, DPC, LTR settings.
I don't quite follow. The OS knows nothing about whether BIOS settings exist, so they can't be used to determine where _OSC applies.
Quoted 3 lines
Is it okay if we include an additional check for hotplug? i.e. Hotplug capable bit in SltCap register which reflects VMD BIOS hotplug setting.
I don't know what check you have in mind, but the OS can definitely use SltCap to decide what features to enable.
You suggest above that you want vmd to be aware of AER ownership per _OSC, but it sounds more like you just want AER disabled below VMD regardless. Or do you have platforms that can actually handle AER interrupts from Root Ports below VMD?
Quoted 4 lines
Another way is to overwrite
_OSCsetting for hotplug only in Guest OS. If we make VMD driver aware of Host or Guest environment, only in case of Guest we overwrite_OSChotplug using SltCap register and we don't revert the04b12ef163d1.
Making vmd aware of being in host or guest sounds kind of ugly to me.
Bjorn
Nirmal Patel
maintainerQuoted 53 lines · Bjorn Helgaas
Quoted 45 lines · Nirmal Patel
Quoted 44 lines · Bjorn Helgaas
Quoted 43 lines · Nirmal Patel
Quoted 23 lines · Bjorn Helgaas
On Tue, Dec 05, 2023 at 03:20:27PM -0700, Nirmal Patel wrote:
...Quoted 6 lines
Currently Hotplug is controlled by
_OSCin both host and Guest.
In case of guest, it seems guest BIOS hasn't implemented_OSC
since all the flags are set to 0 which is not the case in host.I think you want pciehp to work on the VMD Root Ports in the Guest, no matter what, on the assumption that any
_OSCthat applies to host bridge A does not apply to the host bridge created
by the vmd driver.If so, we should just revert
04b12ef163d1("PCI: vmd: Honor ACPI_OSCon PCIe features"). Since host bridge B was not enumerated
via ACPI, the OS owns all those features under B by default, so
reverting04b12ef163d1would leave that state.Quoted 3 lines
Obviously we'd have to first figure out another solution for the
AER message flood that04b12ef163d1resolved.If we revert
04b12ef163d1, then VMD driver will still enable AER
blindly which is a bug. So we need to find a way to make VMD driver
aware of AER platform setting and use that information to enable or
disable AER for its child devices.There is a setting in BIOS that allows us to enable OS native AER
support on the platform. This setting is located in EDK Menu -> Platform configuration -> system event log -> IIO error enabling ->
OS native AER support.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
This setting is assigned to VMD bridge by
vmd_copy_host_bridge_flagsin patch04b12ef163d1. Without the patch04b12ef163d1, VMD driver will enable AER even if platform has disabled OS native AER support as mentioned earlier. This will result in an AER flood mentioned in04b12ef163d1since there is no AER handlers.
Quoted 3 lines
I missed this before. What does "there are no AER handlers" mean? Do
you mean there are no *firmware* AER handlers?
Sorry for confusing wordings. Your understanding is correct. The patch 04b12ef163d1 is used to disable AER on vmd and avoid the AER flooding by applying _OSC settings since vmd driver doesn't give a choice to toggle AER, DPC, LTR, etc.
Quoted 51 lines
The dmesg log at https://bugzilla.kernel.org/attachment.cgi?id=299571
(from https://bugzilla.kernel.org/show_bug.cgi?id=215027, the bug fixed by04b12ef163d1), shows this:acpi PNP0A08:00:
_OSC: OS supports [ExtendedConfig ASPM ClockPM
Segments MSI HPX-Type3]
acpi PNP0A08:00:_OSC: platform does not support [AER] acpi PNP0A08:00:_OSC: OS now controls [PCIeHotplug SHPCHotplug PME
PCIeCapability LTR]so the firmware did not grant AER control to the OS (I think "platform
does not support" is a confusing description).Prior to
04b12ef163d1, Linux applied_OSCabove the VMD bridge but not
below it, so Linux had native control below VMD and it did handle AER interrupts from the 10000:e0:06.0 Root Port below VMD:vmd 0000:00:0e.0: PCI host bridge to bus 10000:e0 pcieport 10000:e0:06.0: AER: Corrected error received:
10000:e1:00.0After
04b12ef163d1, Linux applied_OSCbelow the VMD bridge as well, so it did not enable or handle any AER interrupts. I suspect the platform didn't handle AER interrupts below VMD either, so those errors were probably just ignored.Quoted 14 lines
Quoted 8 lines
Quoted 7 lines
If VMD is aware of OS native AER support setting, then we will not
see AER flooding issue.Do you have any suggestion on how to make VMD driver aware of AER
setting and keep it in sync with platform setting.Quoted 5 lines
Well, this is the whole problem. IIUC, you're saying we should use
_OSCto negotiate for AER control below the VMD bridge, but ignore_OSCfor hotplug control.Quoted 3 lines
Because VMD has its own hotplug BIOS setting which allows vmd to enable or disable hotplug on its domain. However we don't have VMD specific AER, DPC, LTR settings.
I don't quite follow. The OS knows nothing about whether BIOS settings exist, so they can't be used to determine where
_OSCapplies.
Quoted 6 lines
Quoted 3 lines
Is it okay if we include an additional check for hotplug? i.e. Hotplug capable bit in SltCap register which reflects VMD BIOS hotplug setting.
I don't know what check you have in mind, but the OS can definitely use SltCap to decide what features to enable.
Yes I agree. That is what I am also suggesting in this patch.
Quoted 5 lines
You suggest above that you want vmd to be aware of AER ownership per
_OSC, but it sounds more like you just want AER disabled below VMD regardless. Or do you have platforms that can actually handle AER interrupts from Root Ports below VMD?
No I dont want AER disabled below VMD all the time. I am suggesting vmd should be in sync with all the _OSC flags since vmd doesn't give a choice to toggle. However, the issue arises in guest where _OSC setting for hotplug is always 0 eventhough hotplug is 1 in host and hotplug enable bit in SltCap register is 1 in both host and guest. So we can use that to enable hotplug in guest. Like you suggested in your above comment.
Quoted 8 lines
Quoted 6 lines
Another way is to overwrite
_OSCsetting for hotplug only in Guest OS. If we make VMD driver aware of Host or Guest environment, only in case of Guest we overwrite_OSChotplug using SltCap register and
we don't revert the04b12ef163d1.Making vmd aware of being in host or guest sounds kind of ugly to me.
I agree.
Quoted 2 lines
Bjorn
Nirmal Patel
maintainerQuoted 153 lines · Nirmal Patel
Quoted 59 lines · Bjorn Helgaas
Quoted 50 lines · Nirmal Patel
Quoted 49 lines · Bjorn Helgaas
Quoted 48 lines · Nirmal Patel
Quoted 26 lines · Bjorn Helgaas
On Tue, Dec 05, 2023 at 03:20:27PM -0700, Nirmal Patel wrote:
...Quoted 6 lines
Currently Hotplug is controlled by
_OSCin both host and Guest.
In case of guest, it seems guest BIOS hasn't implemented_OSC
since all the flags are set to 0 which is not the case in host.I think you want pciehp to work on the VMD Root Ports in the
Guest, no matter what, on the assumption that any_OSCthat applies to host bridge A does not apply to the host bridge created
by the vmd driver.If so, we should just revert
04b12ef163d1("PCI: vmd: Honor ACPI_OSCon PCIe features"). Since host bridge B was not enumerated
via ACPI, the OS owns all those features under B by default,
so
reverting04b12ef163d1would leave that state.Quoted 4 lines
Obviously we'd have to first figure out another solution for
the
AER message flood that04b12ef163d1resolved.If we revert
04b12ef163d1, then VMD driver will still enable AER
blindly which is a bug. So we need to find a way to make VMD driver
aware of AER platform setting and use that information to enable or
disable AER for its child devices.There is a setting in BIOS that allows us to enable OS native AER
support on the platform. This setting is located in EDK Menu ->
Platform configuration -> system event log -> IIO error enabling ->
OS native AER support.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
This setting is assigned to VMD bridge by
vmd_copy_host_bridge_flagsin patch04b12ef163d1. Without the patch04b12ef163d1, VMD driver will enable AER even if platform
has disabled OS native AER support as mentioned earlier. This will result in an AER flood mentioned in04b12ef163d1since there is no AER handlers.Quoted 3 lines
I missed this before. What does "there are no AER handlers" mean? Do
you mean there are no *firmware* AER handlers?Sorry for confusing wordings. Your understanding is correct. The patch
04b12ef163d1is used to disable AER on vmd and avoid the AER flooding by applying_OSCsettings since vmd driver doesn't give a choice to toggle AER, DPC, LTR, etc.Quoted 56 lines
The dmesg log at
https://bugzilla.kernel.org/attachment.cgi?id=299571
(from https://bugzilla.kernel.org/show_bug.cgi?id=215027, the bug fixed by04b12ef163d1), shows this:acpi PNP0A08:00:
_OSC: OS supports [ExtendedConfig ASPM ClockPM
Segments MSI HPX-Type3]
acpi PNP0A08:00:_OSC: platform does not support [AER] acpi PNP0A08:00:_OSC: OS now controls [PCIeHotplug SHPCHotplug
PME
PCIeCapability LTR]so the firmware did not grant AER control to the OS (I think "platform
does not support" is a confusing description).Prior to
04b12ef163d1, Linux applied_OSCabove the VMD bridge but not
below it, so Linux had native control below VMD and it did handle AER
interrupts from the 10000:e0:06.0 Root Port below VMD:vmd 0000:00:0e.0: PCI host bridge to bus 10000:e0 pcieport 10000:e0:06.0: AER: Corrected error received:
10000:e1:00.0After
04b12ef163d1, Linux applied_OSCbelow the VMD bridge as well,
so it did not enable or handle any AER interrupts. I suspect the platform didn't handle AER interrupts below VMD either, so those errors were probably just ignored.Quoted 15 lines
Quoted 9 lines
Quoted 8 lines
If VMD is aware of OS native AER support setting, then we will
not
see AER flooding issue.Do you have any suggestion on how to make VMD driver aware of AER
setting and keep it in sync with platform setting.Quoted 5 lines
Well, this is the whole problem. IIUC, you're saying we should use
_OSCto negotiate for AER control below the VMD bridge, but ignore_OSCfor hotplug control.Quoted 4 lines
Because VMD has its own hotplug BIOS setting which allows vmd to enable or disable hotplug on its domain. However we don't have VMD
specific AER, DPC, LTR settings.I don't quite follow. The OS knows nothing about whether BIOS settings exist, so they can't be used to determine where
_OSCapplies.Quoted 6 lines
Quoted 3 lines
Is it okay if we include an additional check for hotplug? i.e. Hotplug capable bit in SltCap register which reflects VMD BIOS hotplug setting.
I don't know what check you have in mind, but the OS can definitely use SltCap to decide what features to enable.
Yes I agree. That is what I am also suggesting in this patch.
Quoted 5 lines
You suggest above that you want vmd to be aware of AER ownership per
_OSC, but it sounds more like you just want AER disabled below VMD regardless. Or do you have platforms that can actually handle AER interrupts from Root Ports below VMD?No I dont want AER disabled below VMD all the time. I am suggesting vmd
should be in sync with all the_OSCflags since vmd doesn't give a choice to toggle. However, the issue arises in guest where_OSCsetting
for hotplug is always 0 eventhough hotplug is 1 in host and hotplug enable bit in SltCap register is 1 in both host and guest. So we can use that to enable hotplug in guest. Like you suggested in your above comment.Quoted 8 lines
Quoted 7 lines
Another way is to overwrite
_OSCsetting for hotplug only in Guest
OS. If we make VMD driver aware of Host or Guest environment, only
in case of Guest we overwrite_OSChotplug using SltCap register and
we don't revert the04b12ef163d1.
Quoted 4 lines
Quoted 2 lines
Making vmd aware of being in host or guest sounds kind of ugly to me.
I agree.
Quoted 1 line
Bjorn
Hi Bjorn,
Happy new year. sending a gentle reminder to help conclude the discussion.
Thanks
nirmal
[+cc Rafael, Kai-Heng]
Quoted 95 lines · Nirmal Patel
Quoted 42 lines · Bjorn Helgaas
Quoted 34 lines · Nirmal Patel
Quoted 33 lines · Bjorn Helgaas
Quoted 32 lines · Nirmal Patel
Quoted 12 lines · Bjorn Helgaas
On Tue, Dec 05, 2023 at 03:20:27PM -0700, Nirmal Patel wrote:
...Quoted 4 lines
Currently Hotplug is controlled by
_OSCin both host and Guest. In case of guest, it seems guest BIOS hasn't implemented_OSCsince all the flags are set to 0 which is not the case in host.I think you want pciehp to work on the VMD Root Ports in the Guest, no matter what, on the assumption that any
_OSCthat applies to host bridge A does not apply to the host bridge created by the vmd driver.Quoted 8 lines
If so, we should just revert
04b12ef163d1("PCI: vmd:
Honor ACPI_OSCon PCIe features"). Since host bridge B was not enumerated via ACPI, the OS owns all those features under B by default, so reverting04b12ef163d1would leave that state.Obviously we'd have to first figure out another solution for the AER message flood that
04b12ef163d1resolved.If we revert
04b12ef163d1, then VMD driver will still enable AER blindly which is a bug. So we need to find a way to make VMD driver aware of AER platform setting and use that information to enable or disable AER for its child devices.There is a setting in BIOS that allows us to enable OS native AER support on the platform. This setting is located in EDK Menu -> Platform configuration -> system event log -> IIO error enabling -> OS native AER support.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
This setting is assigned to VMD bridge by
vmd_copy_host_bridge_flagsin patch04b12ef163d1. Without the patch04b12ef163d1, VMD driver will enable AER even if platform has disabled OS native AER support as mentioned earlier. This will result in an AER flood mentioned in04b12ef163d1since there is no AER handlers.Quoted 2 lines
I missed this before. What does "there are no AER handlers" mean? Do you mean there are no *firmware* AER handlers?
Sorry for confusing wordings. Your understanding is correct. The patch
04b12ef163d1is used to disable AER on vmd and avoid the AER flooding by applying_OSCsettings since vmd driver doesn't give a choice to toggle AER, DPC, LTR, etc.Quoted 45 lines
The dmesg log at https://bugzilla.kernel.org/attachment.cgi?id=299571
(from https://bugzilla.kernel.org/show_bug.cgi?id=215027, the bug fixed by04b12ef163d1), shows this:acpi PNP0A08:00:
_OSC: OS supports [ExtendedConfig ASPM ClockPM
Segments MSI HPX-Type3]
acpi PNP0A08:00:_OSC: platform does not support [AER] acpi PNP0A08:00:_OSC: OS now controls [PCIeHotplug SHPCHotplug PME
PCIeCapability LTR]so the firmware did not grant AER control to the OS (I think "platform does not support" is a confusing description).
Prior to
04b12ef163d1, Linux applied_OSCabove the VMD bridge but not below it, so Linux had native control below VMD and it did handle AER interrupts from the 10000:e0:06.0 Root Port below VMD:vmd 0000:00:0e.0: PCI host bridge to bus 10000:e0 pcieport 10000:e0:06.0: AER: Corrected error received:
10000:e1:00.0After
04b12ef163d1, Linux applied_OSCbelow the VMD bridge as well, so it did not enable or handle any AER interrupts. I suspect the platform didn't handle AER interrupts below VMD either, so those errors were probably just ignored.Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
If VMD is aware of OS native AER support setting, then we will not see AER flooding issue.
Do you have any suggestion on how to make VMD driver aware of AER setting and keep it in sync with platform setting.
Quoted 7 lines
Quoted 3 lines
Well, this is the whole problem. IIUC, you're saying we should use
_OSCto negotiate for AER control below the VMD bridge, but ignore_OSCfor hotplug control.Because VMD has its own hotplug BIOS setting which allows vmd to enable or disable hotplug on its domain. However we don't have VMD specific AER, DPC, LTR settings.
I don't quite follow. The OS knows nothing about whether BIOS settings exist, so they can't be used to determine where
_OSCapplies.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Is it okay if we include an additional check for hotplug? i.e. Hotplug capable bit in SltCap register which reflects VMD BIOS hotplug setting.
I don't know what check you have in mind, but the OS can definitely use SltCap to decide what features to enable.
Yes I agree. That is what I am also suggesting in this patch.
I should have said "the OS can use SltCap to *help* decide what features to enable." For ACPI host bridges, _OSC determines whether hotplug should be operated by the platform or the OS.
I think we're converging on the idea that since VMD is effectively *not* an ACPI host bridge and doesn't have its own _OSC, the _OSC that applies to the VMD endpoint does *not* apply to the hierarchy below the VMD. In that case, the default is that the OS owns all the features (hotplug, AER, etc) below the VMD.
Quoted 8 lines
Quoted 4 lines
You suggest above that you want vmd to be aware of AER ownership per
_OSC, but it sounds more like you just want AER disabled below VMD regardless. Or do you have platforms that can actually handle AER interrupts from Root Ports below VMD?No I dont want AER disabled below VMD all the time. I am suggesting vmd should be in sync with all the
_OSCflags since vmd doesn't give a choice to toggle.
This is what I don't understand. If we decide that _OSC doesn't apply below VMD because the VMD host bridge isn't described in ACPI, the idea of being in sync with _OSC doesn't make sense.
Quoted 4 lines
However, the issue arises in guest where
_OSCsetting for hotplug is always 0 even though hotplug is 1 in host and hotplug enable bit in SltCap register is 1 in both host and guest. So we can use that to enable hotplug in guest. Like you suggested in your above comment.
All this got paged out over the holidays, so I need to refresh my understanding here. Maybe it will help if we can make the situation more concrete. I'm basing this on the logs at
https://bugzilla.kernel.org/show_bug.cgi?id=215027. I assume the 10000:e0:06.0 Root Port and the 10000:e1:00.0 NVMe device are both passed through to the guest. I'm sure I got lots wrong here, so please correct me :)
Host OS sees:
PNP0A08 host bridge to 0000 [bus 00-ff]
_OSC applies to domain 0000
OS owns [PCIeHotplug SHPCHotplug PME PCIeCapability LTR] in domain 0000
vmd 0000:00:0e.0: PCI host bridge to domain 10000 [bus e0-ff]
no _OSC applies in domain 10000;
host OS owns all PCIe features in domain 10000
pci 10000:e0:06.0: [8086:464d] # VMD root port
pci 10000:e0:06.0: PCI bridge to [bus e1]
pci 10000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable
pci 10000:e1:00.0: [144d:a80a] # nvmeGuest OS sees:
PNP0A03 host bridge to 0000 [bus 00-ff]
_OSC applies to domain 0000
platform owns [PCIeHotplug ...] # _OSC doesn't grant to OS
pci 0000:e0:06.0: [8086:464d] # VMD root port
pci 0000:e0:06.0: PCI bridge to [bus e1]
pci 0000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable
pci 0000:e1:00.0: [144d:a80a] # nvmeSo the guest OS sees that the VMD Root Port supports hotplug, but it can't use it because it was not granted ownership of the feature?
Bjorn
Nirmal Patel
maintainerQuoted 124 lines · Bjorn Helgaas
[+cc Rafael, Kai-Heng]
Quoted 108 lines · Nirmal Patel
Quoted 48 lines · Bjorn Helgaas
Quoted 40 lines · Nirmal Patel
Quoted 39 lines · Bjorn Helgaas
Quoted 38 lines · Nirmal Patel
Quoted 15 lines · Bjorn Helgaas
On Tue, Dec 05, 2023 at 03:20:27PM -0700, Nirmal Patel wrote:
...Quoted 5 lines
Currently Hotplug is controlled by
_OSCin both host and
Guest. In case of guest, it seems guest BIOS hasn't implemented_OSCsince all the flags are set to 0 which is not the case in host.I think you want pciehp to work on the VMD Root Ports in the Guest, no matter what, on the assumption that any
_OSC
that applies to host bridge A does not apply to the host bridge created by the vmd driver.Quoted 8 lines
If so, we should just revert
04b12ef163d1("PCI: vmd:
Honor ACPI_OSCon PCIe features"). Since host bridge B was not enumerated via ACPI, the OS owns all those features under B by default, so reverting04b12ef163d1would leave that state.Obviously we'd have to first figure out another solution for the AER message flood that
04b12ef163d1resolved.If we revert
04b12ef163d1, then VMD driver will still enable
AER blindly which is a bug. So we need to find a way to make
VMD driver aware of AER platform setting and use that information to enable or disable AER for its child devices.There is a setting in BIOS that allows us to enable OS native AER support on the platform. This setting is located in EDK Menu -> Platform configuration -> system event log ->
IIO error enabling -> OS native AER support.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
This setting is assigned to VMD bridge by
vmd_copy_host_bridge_flagsin patch04b12ef163d1. Without the patch04b12ef163d1, VMD driver will enable AER even if platform has disabled OS native AER support as mentioned earlier. This will result in an AER flood mentioned in04b12ef163d1since there is no AER handlers.Quoted 3 lines
I missed this before. What does "there are no AER handlers" mean?
Do you mean there are no *firmware* AER handlers?Sorry for confusing wordings. Your understanding is correct. The patch
04b12ef163d1is used to disable AER on vmd and avoid the AER flooding
by applying_OSCsettings since vmd driver doesn't give a choice to toggle AER, DPC, LTR, etc.Quoted 49 lines
The dmesg log at
https://bugzilla.kernel.org/attachment.cgi?id=299571
(from https://bugzilla.kernel.org/show_bug.cgi?id=215027, the bug fixed by04b12ef163d1), shows this:acpi PNP0A08:00:
_OSC: OS supports [ExtendedConfig ASPM ClockPM
Segments MSI HPX-Type3]
acpi PNP0A08:00:_OSC: platform does not support [AER] acpi PNP0A08:00:_OSC: OS now controls [PCIeHotplug SHPCHotplug
PME
PCIeCapability LTR]so the firmware did not grant AER control to the OS (I think "platform does not support" is a confusing description).
Prior to
04b12ef163d1, Linux applied_OSCabove the VMD bridge but
not below it, so Linux had native control below VMD and it did handle AER interrupts from the 10000:e0:06.0 Root Port below VMD:vmd 0000:00:0e.0: PCI host bridge to bus 10000:e0 pcieport 10000:e0:06.0: AER: Corrected error received:
10000:e1:00.0After
04b12ef163d1, Linux applied_OSCbelow the VMD bridge as well,
so it did not enable or handle any AER interrupts. I suspect the platform didn't handle AER interrupts below VMD either, so those errors were probably just ignored.Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
If VMD is aware of OS native AER support setting, then we will not see AER flooding issue.
Do you have any suggestion on how to make VMD driver aware of AER setting and keep it in sync with platform setting.
Quoted 8 lines
Quoted 3 lines
Well, this is the whole problem. IIUC, you're saying we should use
_OSCto negotiate for AER control below the VMD bridge, but ignore_OSCfor hotplug control.Because VMD has its own hotplug BIOS setting which allows vmd to
enable or disable hotplug on its domain. However we don't have VMD specific AER, DPC, LTR settings.I don't quite follow. The OS knows nothing about whether BIOS settings exist, so they can't be used to determine where
_OSCapplies.Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Is it okay if we include an additional check for hotplug? i.e. Hotplug capable bit in SltCap register which reflects VMD BIOS hotplug setting.
I don't know what check you have in mind, but the OS can definitely use SltCap to decide what features to enable.
Yes I agree. That is what I am also suggesting in this patch.
I should have said "the OS can use SltCap to *help* decide what features to enable." For ACPI host bridges,
_OSCdetermines whether hotplug should be operated by the platform or the OS.
Quoted 6 lines
I think we're converging on the idea that since VMD is effectively *not* an ACPI host bridge and doesn't have its own
_OSC, the_OSCthat
applies to the VMD endpoint does *not* apply to the hierarchy below the VMD. In that case, the default is that the OS owns all the features (hotplug, AER, etc) below the VMD.
Well there will be few problems if VMD owns/assigns all the flags by itself. We discussed all of the potential problems but due to the holidays I think I should summarize them again. #1 : Currently there is no way to pass the information about AER, DPC,
etc to VMD driver from BIOS or from boot parameter. For example, If VMD
blindly enables AER and platform has AER disabled, then we will see
AERs from devices under VMD but user have no way to toggle it since we
rejected idea of adding boot parameter for AER, DPC under VMD. I
believe you also didn't like the idea of sysfs knob suggested by Kai-
Heng.
#2 : Even if we use VMD hardware register to store AER, DPC and make UEFI VMD driver to write information about Hotplug, DPC, AER, we still dont have a way to change the setting if user wants to alter them. Also the issue will still persist in client platforms since we don't own their UEFI VMD driver. It will be a huge effort.
#3 : At this moment, user can enable/disable only Hotplug in VMD BIOS settings (meaning no AER, DPC, LTR, etc)and VMD driver can read it from SltCap register. This means BIOS needs to add other settings and VMD needs to figure out to read them which at this moment VMD can't do it.
#4 : consistency with Host OS and Guest OS.
I believe the current propesed patch is the best option which requires minimal changes without breaking other platform features and unblock our customers. This issues has been a blocker for us.
For your concerns regarding how VMD can own all the _OSC features, i am open to other ideas and will discuss with the architect if they have any suggestions.
Quoted 53 lines
Quoted 9 lines
Quoted 8 lines
You suggest above that you want vmd to be aware of AER ownership per
_OSC, but it sounds more like you just want AER disabled below VMD
regardless. Or do you have platforms that can actually handle AER
interrupts from Root Ports below VMD?Quoted 4 lines
No I dont want AER disabled below VMD all the time. I am suggesting vmd should be in sync with all the
_OSCflags since vmd doesn't give
a choice to toggle.This is what I don't understand. If we decide that
_OSCdoesn't apply
below VMD because the VMD host bridge isn't described in ACPI, the idea of being in sync with_OSCdoesn't make sense.Quoted 5 lines
However, the issue arises in guest where
_OSCsetting for hotplug is
always 0 even though hotplug is 1 in host and hotplug enable bit in SltCap register is 1 in both host and guest. So we can use that to enable hotplug in guest. Like you suggested in your above comment.All this got paged out over the holidays, so I need to refresh my understanding here. Maybe it will help if we can make the situation more concrete. I'm basing this on the logs at
https://bugzilla.kernel.org/show_bug.cgi?id=215027. I assume the 10000:e0:06.0 Root Port and the 10000:e1:00.0 NVMe device are both passed through to the guest. I'm sure I got lots wrong here, so please correct me :)Host OS sees:
PNP0A08 host bridge to 0000 [bus 00-ff]
_OSCapplies to domain 0000
OS owns [PCIeHotplug SHPCHotplug PME PCIeCapability LTR] in
domain 0000vmd 0000:00:0e.0: PCI host bridge to domain 10000 [bus e0-ff] no _OSC applies in domain 10000; host OS owns all PCIe features in domain 10000 pci 10000:e0:06.0: [8086:464d] # VMD root port pci 10000:e0:06.0: PCI bridge to [bus e1] pci 10000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable pci 10000:e1:00.0: [144d:a80a] # nvmeGuest OS sees:
PNP0A03 host bridge to 0000 [bus 00-ff]
_OSCapplies to domain 0000
platform owns [PCIeHotplug ...] #_OSCdoesn't grant
to OS
Quoted 7 lines
pci 0000:e0:06.0: [8086:464d] # VMD root port pci 0000:e0:06.0: PCI bridge to [bus e1] pci 0000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable pci 0000:e1:00.0: [144d:a80a] # nvmeSo the guest OS sees that the VMD Root Port supports hotplug, but it can't use it because it was not granted ownership of the feature?
You are correct about _OSC not granting access in Guest. This is what I see on my setup.
Host OS: ACPI: PCI Root Bridge [PC11] (domain 0000 [bus e2-fa]) acpi PNP0A08:0b: _OSC: OS supports [ExtendedConfig ASPM ClockPM
Segments MSI EDR HPX-Type3]
acpi PNP0A08:0b: _OSC: platform does not support [SHPCHotplug
AER DPC]
acpi PNP0A08:0b: _OSC: OS now controls [PCIeHotplug PME
PCIeCapability LTR]
PCI host bridge to bus 0000:e2 vmd 0000:e2:00.5: PCI host bridge to bus 10007:00 vmd 0000:e2:00.5: Bound to PCI domain 10007 Guest OS: ACPI: PCI Root Bridge [PC0G] (domain 0000 [bus 03]) acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM
Segments MSI EDR HPX-Type3]
acpi PNP0A08:01: _OSC: platform does not support [PCIeHotplug
SHPCHotplug PME AER LTR DPC]
acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability]
vmd 0000:03:00.0: Bound to PCI domain 10000
SltCap: PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Quoted 2 lines
Bjorn
Quoted 35 lines · Nirmal Patel
On Fri, 2024-01-12 at 16:55 -0600, Bjorn Helgaas wrote:
...Quoted 32 lines
Maybe it will help if we can make the
situation more concrete. I'm basing this on the logs at
https://bugzilla.kernel.org/show_bug.cgi?id=215027. I assume the 10000:e0:06.0 Root Port and the 10000:e1:00.0 NVMe device are both passed through to the guest. I'm sure I got lots wrong here, so please correct me :)Host OS sees:
PNP0A08 host bridge to 0000 [bus 00-ff]
_OSCapplies to domain 0000
OS owns [PCIeHotplug SHPCHotplug PME PCIeCapability LTR] in
domain 0000vmd 0000:00:0e.0: PCI host bridge to domain 10000 [bus e0-ff] no _OSC applies in domain 10000; host OS owns all PCIe features in domain 10000 pci 10000:e0:06.0: [8086:464d] # VMD root port pci 10000:e0:06.0: PCI bridge to [bus e1] pci 10000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable pci 10000:e1:00.0: [144d:a80a] # nvmeGuest OS sees:
PNP0A03 host bridge to 0000 [bus 00-ff]
_OSCapplies to domain 0000
platform owns [PCIeHotplug ...] #_OSCdoesn't grant
to OSpci 0000:e0:06.0: [8086:464d] # VMD root port pci 0000:e0:06.0: PCI bridge to [bus e1] pci 0000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable pci 0000:e1:00.0: [144d:a80a] # nvme
Quoted 5 lines
Quoted 3 lines
So the guest OS sees that the VMD Root Port supports hotplug, but it can't use it because it was not granted ownership of the feature?
You are correct about
_OSCnot granting access in Guest.
I was assuming the VMD Endpoint itself was not visible in the guest and the VMD Root Ports appeared in domain 0000 described by the guest PNP0A03. The PNP0A03 _OSC would then apply to the VMD Root Ports. But my assumption appears to be wrong:
Quoted 15 lines
This is what I see on my setup.
Host OS:
ACPI: PCI Root Bridge [PC11] (domain 0000 [bus e2-fa])
acpi PNP0A08:0b:_OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] acpi PNP0A08:0b:_OSC: platform does not support [SHPCHotplug AER DPC]
acpi PNP0A08:0b:_OSC: OS now controls [PCIeHotplug PME PCIeCapability LTR] PCI host bridge to bus 0000:e2vmd 0000:e2:00.5: PCI host bridge to bus 10007:00 vmd 0000:e2:00.5: Bound to PCI domain 10007
Guest OS:
Quoted 8 lines
ACPI: PCI Root Bridge [PC0G] (domain 0000 [bus 03])
acpi PNP0A08:01:_OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] acpi PNP0A08:01:_OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR DPC] acpi PNP0A08:01:_OSC: OS now controls [PCIeCapability]vmd 0000:03:00.0: Bound to PCI domain 10000
SltCap: PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Your example above suggests that the guest has a PNP0A08 device for domain 0000, with an _OSC, the guest sees the VMD Endpoint at 0000:03:00.0, and the VMD Root Ports and devices below them are in domain 10000. Right?
If we decide the _OSC that covers the VMD Endpoint does *not* apply to the domain below the VMD bridge, the guest has no _OSC for domain 10000, so the guest OS should default to owning all the PCIe features in that domain.
Bjorn
Quoted 66 lines · Bjorn Helgaas
Quoted 34 lines · Nirmal Patel
On Fri, 2024-01-12 at 16:55 -0600, Bjorn Helgaas wrote:
...Quoted 32 lines
Maybe it will help if we can make the
situation more concrete. I'm basing this on the logs at
https://bugzilla.kernel.org/show_bug.cgi?id=215027. I assume the 10000:e0:06.0 Root Port and the 10000:e1:00.0 NVMe device are both passed through to the guest. I'm sure I got lots wrong here, so please correct me :)Host OS sees:
PNP0A08 host bridge to 0000 [bus 00-ff]
_OSCapplies to domain 0000
OS owns [PCIeHotplug SHPCHotplug PME PCIeCapability LTR] in
domain 0000vmd 0000:00:0e.0: PCI host bridge to domain 10000 [bus e0-ff] no _OSC applies in domain 10000; host OS owns all PCIe features in domain 10000 pci 10000:e0:06.0: [8086:464d] # VMD root port pci 10000:e0:06.0: PCI bridge to [bus e1] pci 10000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable pci 10000:e1:00.0: [144d:a80a] # nvmeGuest OS sees:
PNP0A03 host bridge to 0000 [bus 00-ff]
_OSCapplies to domain 0000
platform owns [PCIeHotplug ...] #_OSCdoesn't grant
to OSpci 0000:e0:06.0: [8086:464d] # VMD root port pci 0000:e0:06.0: PCI bridge to [bus e1] pci 0000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable pci 0000:e1:00.0: [144d:a80a] # nvmeQuoted 5 lines
Quoted 3 lines
So the guest OS sees that the VMD Root Port supports hotplug, but it can't use it because it was not granted ownership of the feature?
You are correct about
_OSCnot granting access in Guest.I was assuming the VMD Endpoint itself was not visible in the guest and the VMD Root Ports appeared in domain 0000 described by the guest PNP0A03. The PNP0A03
_OSCwould then apply to the VMD Root Ports. But my assumption appears to be wrong:Quoted 20 lines
This is what I see on my setup.
Host OS:
ACPI: PCI Root Bridge [PC11] (domain 0000 [bus e2-fa])
acpi PNP0A08:0b:_OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] acpi PNP0A08:0b:_OSC: platform does not support [SHPCHotplug AER DPC]
acpi PNP0A08:0b:_OSC: OS now controls [PCIeHotplug PME PCIeCapability LTR] PCI host bridge to bus 0000:e2vmd 0000:e2:00.5: PCI host bridge to bus 10007:00 vmd 0000:e2:00.5: Bound to PCI domain 10007
Guest OS:
ACPI: PCI Root Bridge [PC0G] (domain 0000 [bus 03])
acpi PNP0A08:01:_OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3] acpi PNP0A08:01:_OSC: platform does not support [PCIeHotplug SHPCHotplug PME AER LTR DPC] acpi PNP0A08:01:_OSC: OS now controls [PCIeCapability]
Quoted 8 lines
Quoted 3 lines
vmd 0000:03:00.0: Bound to PCI domain 10000
SltCap: PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Your example above suggests that the guest has a PNP0A08 device for domain 0000, with an
_OSC, the guest sees the VMD Endpoint at 0000:03:00.0, and the VMD Root Ports and devices below them are in domain 10000. Right?
Is my assumption here correct? Do the VMD Endpoint, the VMD Root Ports, and the devices below those Root Ports all visible to the guest?
If so, it sounds like the VMD Endpoint is visible to both the host and the guest? I assume you only want it claimed by the vmd driver in *one* of them? If it's visible in both, how do you prevent one from claiming it?
Quoted 4 lines
If we decide the
_OSCthat covers the VMD Endpoint does *not* apply to the domain below the VMD bridge, the guest has no_OSCfor domain 10000, so the guest OS should default to owning all the PCIe features in that domain.
I assume the guest sees the VMD Endpoint in domain 0000, right?
Does the guest have an _OSC for domain 0000? If so, that would clearly apply to the VMD Endpoint.
I think the guest sees the VMD Root Ports in domain 10000, along with the devices below them, right?
I'm pretty sure the guest has no _OSC for domain 10000, i.e., for the the VMD Root Ports and below, right?
Bjorn
Nirmal Patel
maintainerQuoted 282 lines · Nirmal Patel
Quoted 145 lines · Bjorn Helgaas
[+cc Rafael, Kai-Heng]
Quoted 132 lines · Nirmal Patel
Quoted 57 lines · Bjorn Helgaas
Quoted 48 lines · Nirmal Patel
Quoted 47 lines · Bjorn Helgaas
Quoted 45 lines · Nirmal Patel
Quoted 25 lines · Bjorn Helgaas
On Tue, Dec 05, 2023 at 03:20:27PM -0700, Nirmal Patel wrote:
...Quoted 6 lines
Currently Hotplug is controlled by
_OSCin both host and
Guest. In case of guest, it seems guest BIOS hasn't implemented_OSCsince all the flags are set to 0 which
is not the case in host.I think you want pciehp to work on the VMD Root Ports in
the Guest, no matter what, on the assumption that any_OSC
that applies to host bridge A does not apply to the host
bridge created by the vmd driver.If so, we should just revert
04b12ef163d1("PCI: vmd:
Honor ACPI_OSCon PCIe features"). Since host bridge B
was not enumerated via ACPI, the OS owns all those features under B by default, so reverting04b12ef163d1would leave that state.Quoted 3 lines
Obviously we'd have to first figure out another solution
for the AER message flood that04b12ef163d1resolved.If we revert
04b12ef163d1, then VMD driver will still enable
AER blindly which is a bug. So we need to find a way to make
VMD driver aware of AER platform setting and use that information to enable or disable AER for its child devices.There is a setting in BIOS that allows us to enable OS native AER support on the platform. This setting is located
in EDK Menu -> Platform configuration -> system event log ->
IIO error enabling -> OS native AER support.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
This setting is assigned to VMD bridge by
vmd_copy_host_bridge_flagsin patch04b12ef163d1. Without the patch04b12ef163d1, VMD driver will enable AER even if
platform has disabled OS native AER support as mentioned earlier. This will result in an AER flood mentioned in04b12ef163d1since there is no AER handlers.Quoted 3 lines
I missed this before. What does "there are no AER handlers" mean?
Do you mean there are no *firmware* AER handlers?Sorry for confusing wordings. Your understanding is correct. The patch
04b12ef163d1is used to disable AER on vmd and avoid the AER flooding
by applying_OSCsettings since vmd driver doesn't give a choice to
toggle AER, DPC, LTR, etc.Quoted 63 lines
The dmesg log at
https://bugzilla.kernel.org/attachment.cgi?id=299571
(from https://bugzilla.kernel.org/show_bug.cgi?id=215027, the bug
fixed by04b12ef163d1), shows this:acpi PNP0A08:00:
_OSC: OS supports [ExtendedConfig ASPM
ClockPM
Segments MSI HPX-Type3]
acpi PNP0A08:00:_OSC: platform does not support [AER] acpi PNP0A08:00:_OSC: OS now controls [PCIeHotplug
SHPCHotplug
PME
PCIeCapability LTR]so the firmware did not grant AER control to the OS (I think "platform does not support" is a confusing description).
Prior to
04b12ef163d1, Linux applied_OSCabove the VMD bridge but
not below it, so Linux had native control below VMD and it did handle AER interrupts from the 10000:e0:06.0 Root Port below VMD:vmd 0000:00:0e.0: PCI host bridge to bus 10000:e0 pcieport 10000:e0:06.0: AER: Corrected error received:
10000:e1:00.0After
04b12ef163d1, Linux applied_OSCbelow the VMD bridge as well,
so it did not enable or handle any AER interrupts. I suspect the
platform didn't handle AER interrupts below VMD either, so those
errors were probably just ignored.Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
If VMD is aware of OS native AER support setting, then we will not see AER flooding issue.
Do you have any suggestion on how to make VMD driver aware
of AER setting and keep it in sync with platform setting.Well, this is the whole problem. IIUC, you're saying we should use
_OSCto negotiate for AER control below the VMD bridge, but ignore_OSCfor hotplug control.Quoted 5 lines
Because VMD has its own hotplug BIOS setting which allows vmd to
enable or disable hotplug on its domain. However we don't have
VMD specific AER, DPC, LTR settings.I don't quite follow. The OS knows nothing about whether BIOS settings exist, so they can't be used to determine where
_OSCapplies.Quoted 5 lines
Is it okay if we include an additional check for hotplug? i.e.
Hotplug capable bit in SltCap register which reflects VMD BIOS
hotplug setting.Quoted 4 lines
Quoted 2 lines
I don't know what check you have in mind, but the OS can definitely use SltCap to decide what features to enable.
Yes I agree. That is what I am also suggesting in this patch.
I should have said "the OS can use SltCap to *help* decide what features to enable." For ACPI host bridges,
_OSCdetermines whether
hotplug should be operated by the platform or the OS.Quoted 6 lines
I think we're converging on the idea that since VMD is effectively *not* an ACPI host bridge and doesn't have its own
_OSC, the_OSCthat
applies to the VMD endpoint does *not* apply to the hierarchy below the VMD. In that case, the default is that the OS owns all the features (hotplug, AER, etc) below the VMD.Well there will be few problems if VMD owns/assigns all the flags by itself. We discussed all of the potential problems but due to the holidays I think I should summarize them again.
#1 : Currently there is no way to pass the information about AER, DPC,
etc to VMD driver from BIOS or from boot parameter. For example, If VMD
blindly enables AER and platform has AER disabled, then we will see AERs from devices under VMD but user have no way to toggle it since we
rejected idea of adding boot parameter for AER, DPC under VMD. I believe you also didn't like the idea of sysfs knob suggested by Kai- Heng.#2 : Even if we use VMD hardware register to store AER, DPC and make UEFI VMD driver to write information about Hotplug, DPC, AER, we still
dont have a way to change the setting if user wants to alter them. Also
the issue will still persist in client platforms since we don't own their UEFI VMD driver. It will be a huge effort.#3 : At this moment, user can enable/disable only Hotplug in VMD BIOS settings (meaning no AER, DPC, LTR, etc)and VMD driver can read it from
SltCap register. This means BIOS needs to add other settings and VMD needs to figure out to read them which at this moment VMD can't do it.#4 : consistency with Host OS and Guest OS.
I believe the current propesed patch is the best option which requires
minimal changes without breaking other platform features and unblock our customers. This issues has been a blocker for us.For your concerns regarding how VMD can own all the
_OSCfeatures, i am
open to other ideas and will discuss with the architect if they have any suggestions.Quoted 58 lines
Quoted 9 lines
Quoted 8 lines
You suggest above that you want vmd to be aware of AER ownership
per_OSC, but it sounds more like you just want AER disabled below VMD
regardless. Or do you have platforms that can actually handle AER
interrupts from Root Ports below VMD?Quoted 5 lines
No I dont want AER disabled below VMD all the time. I am suggesting
vmd should be in sync with all the_OSCflags since vmd doesn't give
a choice to toggle.This is what I don't understand. If we decide that
_OSCdoesn't apply
below VMD because the VMD host bridge isn't described in ACPI, the idea of being in sync with_OSCdoesn't make sense.Quoted 8 lines
However, the issue arises in guest where
_OSCsetting for hotplug is
always 0 even though hotplug is 1 in host and hotplug enable bit in
SltCap register is 1 in both host and guest. So we can use that to
enable hotplug in guest. Like you suggested in your above comment.All this got paged out over the holidays, so I need to refresh my understanding here. Maybe it will help if we can make the situation
more concrete. I'm basing this on the logs at
https://bugzilla.kernel.org/show_bug.cgi?id=215027. I assume the 10000:e0:06.0 Root Port and the 10000:e1:00.0 NVMe device are both passed through to the guest. I'm sure I got lots wrong here, so please correct me :)Host OS sees:
PNP0A08 host bridge to 0000 [bus 00-ff]
_OSCapplies to domain 0000
OS owns [PCIeHotplug SHPCHotplug PME PCIeCapability LTR] in
domain 0000vmd 0000:00:0e.0: PCI host bridge to domain 10000 [bus e0-ff] no _OSC applies in domain 10000; host OS owns all PCIe features in domain 10000 pci 10000:e0:06.0: [8086:464d] # VMD root port pci 10000:e0:06.0: PCI bridge to [bus e1] pci 10000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable pci 10000:e1:00.0: [144d:a80a] # nvmeGuest OS sees:
PNP0A03 host bridge to 0000 [bus 00-ff]
_OSCapplies to domain 0000
platform owns [PCIeHotplug ...] #_OSCdoesn't grant
to OSQuoted 8 lines
pci 0000:e0:06.0: [8086:464d] # VMD root port pci 0000:e0:06.0: PCI bridge to [bus e1] pci 0000:e0:06.0: SltCap: HotPlug+ # Hotplug Capable pci 0000:e1:00.0: [144d:a80a] # nvmeSo the guest OS sees that the VMD Root Port supports hotplug, but it
can't use it because it was not granted ownership of the feature?You are correct about
_OSCnot granting access in Guest. This is what I
see on my setup.Host OS: ACPI: PCI Root Bridge [PC11] (domain 0000 [bus e2-fa]) acpi PNP0A08:0b: _OSC: OS supports [ExtendedConfig ASPM ClockPMSegments MSI EDR HPX-Type3]
acpi PNP0A08:0b:_OSC: platform does not support [SHPCHotplug
AER DPC]
acpi PNP0A08:0b:_OSC: OS now controls [PCIeHotplug PME
PCIeCapability LTR]PCI host bridge to bus 0000:e2 vmd 0000:e2:00.5: PCI host bridge to bus 10007:00 vmd 0000:e2:00.5: Bound to PCI domain 10007 Guest OS: ACPI: PCI Root Bridge [PC0G] (domain 0000 [bus 03]) acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM
Quoted 12 lines
Segments MSI EDR HPX-Type3]
acpi PNP0A08:01:_OSC: platform does not support [PCIeHotplug
SHPCHotplug PME AER LTR DPC]
acpi PNP0A08:01:_OSC: OS now controls [PCIeCapability]vmd 0000:03:00.0: Bound to PCI domain 10000
SltCap: PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Quoted 1 line
Bjorn
Hi Bjorn, Lorenzo,
Gentle ping.
Thanks.
Quoted 1 line · Nirmal Patel
Gentle ping.
Thanks for the ping, I was waiting for you, so we were deadlocked ;)
Nirmal Patel
maintainerQuoted 4 lines · Bjorn Helgaas
Quoted 1 line · Nirmal Patel
Gentle ping.
Thanks for the ping, I was waiting for you, so we were deadlocked ;)
Hi Bjorn,
Sorry I missed that.
Did you have a chance to look at my response on January 16th to your questions? I tried to summarize all of the potential problems and issues with different fixes. Please let me know if it is easier if I resend the explanation. Thanks.
-nirmal
Quoted 5 lines · Nirmal Patel
...
Did you have a chance to look at my response on January 16th to your questions? I tried to summarize all of the potential problems and issues with different fixes. Please let me know if it is easier if I resend the explanation. Thanks.
I did see your Jan 16 response, thanks.
I had more questions after reading it, but they're more about understanding the topology seen by the host and the guest:
Jan 16: https://lore.kernel.org/r/20240117004933.GA108810@bhelgaas
Feb 1: https://lore.kernel.org/r/20240201211620.GA650432@bhelgaas
As I mentioned in my second Feb 1 response
(https://lore.kernel.org/r/20240201222245.GA650725@bhelgaas), the usual plan envisioned by the PCI Firmware spec is that an OS can use a PCIe feature if the platform has granted the OS ownership via _OSC and a device advertises the feature via a Capability in config space.
My main concern with the v2 patch
(https://lore.kernel.org/r/[email protected])
is that it overrides _OSC for native_pcie_hotplug, but not for any of the other features (AER, PME, LTR, DPC, etc.)
I think it's hard to read the specs and conclude that PCIe hotplug is a special case, and I think we're likely to have similar issues with other features in the future.
But if you think this is the best solution, I'm OK with merging it.
Bjorn
Nirmal Patel
maintainerQuoted 31 lines · Bjorn Helgaas
Quoted 7 lines · Nirmal Patel
...
Did you have a chance to look at my response on January 16th to your
questions? I tried to summarize all of the potential problems and issues with different fixes. Please let me know if it is easier if I
resend the explanation. Thanks.I did see your Jan 16 response, thanks.
I had more questions after reading it, but they're more about understanding the topology seen by the host and the guest:
Jan 16: https://lore.kernel.org/r/20240117004933.GA108810@bhelgaas
Feb 1: https://lore.kernel.org/r/20240201211620.GA650432@bhelgaasAs I mentioned in my second Feb 1 response
(https://lore.kernel.org/r/20240201222245.GA650725@bhelgaas), the usual plan envisioned by the PCI Firmware spec is that an OS can use a
PCIe feature if the platform has granted the OS ownership via_OSCand
a device advertises the feature via a Capability in config space.My main concern with the v2 patch
(
https://lore.kernel.org/r/[email protected]
)
is that it overrides_OSCfornative_pcie_hotplug, but not for any of the other features (AER, PME, LTR, DPC, etc.)
Quoted 5 lines
I think it's hard to read the specs and conclude that PCIe hotplug is a special case, and I think we're likely to have similar issues with other features in the future.
But if you think this is the best solution, I'm OK with merging it.
I sincerely apologize for late responses. I just found out that my evolution mail client is automatically sending linux-pci emails to junk since January 2024.
At the moment Hotplug is an exception for us, but I do share your concern about other flags. We have done lot of testing and so far patch v2 is the best solution we have.
Thanks
nirmal
Quoted 2 lines
Bjorn
Nirmal Patel
maintainerQuoted 37 lines · Nirmal Patel
Quoted 36 lines · Bjorn Helgaas
Quoted 8 lines · Nirmal Patel
...
Did you have a chance to look at my response on January 16th to your
questions? I tried to summarize all of the potential problems and issues with different fixes. Please let me know if it is easier if
I
resend the explanation. Thanks.I did see your Jan 16 response, thanks.
I had more questions after reading it, but they're more about understanding the topology seen by the host and the guest:
Jan 16:
https://lore.kernel.org/r/20240117004933.GA108810@bhelgaas
Feb 1:
https://lore.kernel.org/r/20240201211620.GA650432@bhelgaasAs I mentioned in my second Feb 1 response
(https://lore.kernel.org/r/20240201222245.GA650725@bhelgaas), the usual plan envisioned by the PCI Firmware spec is that an OS can use
a
PCIe feature if the platform has granted the OS ownership via_OSCand
a device advertises the feature via a Capability in config space.My main concern with the v2 patch
(
https://lore.kernel.org/r/[email protected]
)
is that it overrides_OSCfornative_pcie_hotplug, but not for any of
the other features (AER, PME, LTR, DPC, etc.)
Quoted 7 lines
Quoted 7 lines
I think it's hard to read the specs and conclude that PCIe hotplug is
a special case, and I think we're likely to have similar issues with
other features in the future.But if you think this is the best solution, I'm OK with merging it.
Hi Bjorn,
We tried some other ways to pass the information about all of the flags but I couldn't retrieve it in guest OS and VMD hardware doesn't have empty registers to write this information. So as of now we have this solution which only overwrites Hotplug flag. If you can accept it that would be great.
Thanks
nirmal.
Quoted 13 lines
I sincerely apologize for late responses. I just found out that my evolution mail client is automatically sending linux-pci emails to junk
since January 2024.At the moment Hotplug is an exception for us, but I do share your concern about other flags. We have done lot of testing and so far patch
v2 is the best solution we have.Thanks
nirmalQuoted 1 line
Bjorn
Hi Nirmal,
Quoted 45 lines · Nirmal Patel
Quoted 43 lines · Nirmal Patel
Quoted 42 lines · Bjorn Helgaas
Quoted 8 lines · Nirmal Patel
...
Did you have a chance to look at my response on January 16th to your
questions? I tried to summarize all of the potential problems and issues with different fixes. Please let me know if it is easier if
I
resend the explanation. Thanks.I did see your Jan 16 response, thanks.
I had more questions after reading it, but they're more about understanding the topology seen by the host and the guest:
Jan 16:
https://lore.kernel.org/r/20240117004933.GA108810@bhelgaas
Feb 1:
https://lore.kernel.org/r/20240201211620.GA650432@bhelgaasAs I mentioned in my second Feb 1 response
(https://lore.kernel.org/r/20240201222245.GA650725@bhelgaas), the usual plan envisioned by the PCI Firmware spec is that an OS can use
a
PCIe feature if the platform has granted the OS ownership via_OSCand
a device advertises the feature via a Capability in config space.My main concern with the v2 patch
(
https://lore.kernel.org/r/[email protected]
)
is that it overrides_OSCfornative_pcie_hotplug, but not for any of
the other features (AER, PME, LTR, DPC, etc.)I think it's hard to read the specs and conclude that PCIe hotplug is
a special case, and I think we're likely to have similar issues with
other features in the future.
Quoted 8 lines
Quoted 1 line
Quoted 1 line
But if you think this is the best solution, I'm OK with merging it.
Hi Bjorn,
We tried some other ways to pass the information about all of the flags but I couldn't retrieve it in guest OS and VMD hardware doesn't have empty registers to write this information. So as of now we have this solution which only overwrites Hotplug flag. If you can accept it that would be great.
My original commit "PCI: vmd: Honor ACPI _OSC on PCIe features" was a logically conclusion based on VMD ports are just apertures. Apparently there are more nuances than that, and people outside of Intel can't possibly know. And yes VMD creates lots of headaches during hardware enablement.
So is it possible to document the right way to use _OSC, as Bjorn suggested?
Kai-Heng
Quoted 44 lines
Quoted 5 lines
In the host OS, this overrides whatever was negotiated via
_OSC, I guess on the principle that_OSCdoesn't apply because the host BIOS doesn't know about the Root Ports below the VMD. (I'm not sure it's 100% resolved that_OSCdoesn't apply to them, so we should mention that assumption here.)
_OSCstill controls every flag including Hotplug. I have observed that slot capabilities register has hotplug enabled only when platform has enabled the hotplug. So technically not overriding it in the host. It overrides in guest because_OSCis passing wrong/different information than the_OSCinformation in Host.
Also like I mentioned, slot capabilities registers are not changed in Guest because vmd is passthrough.Quoted 10 lines
In the guest OS, this still overrides whatever was negotiated via
_OSC, but presumably the guest BIOS *does* know about the Root Ports, so I don't think the same principle about overriding_OSCapplies there.I think it's still a problem that we handle
host_bridge->native_pcie_hotplugdifferently than all the other features negotiated via_OSC. We should at least explain this somehow.Right now this is the only way to know in Guest OS if platform has enabled Hotplug or not. We have many customers complaining about Hotplug being broken in Guest. So just trying to honor
_OSCbut also fixing its limitation in Guest.Thanks
nirmal.Quoted 13 lines
I sincerely apologize for late responses. I just found out that my evolution mail client is automatically sending linux-pci emails to junk
since January 2024.At the moment Hotplug is an exception for us, but I do share your concern about other flags. We have done lot of testing and so far patch
v2 is the best solution we have.Thanks
nirmalQuoted 1 line
Bjorn
Nirmal Patel
maintainerQuoted 66 lines · On Thu, 7 Mar 2024 14:44:23 +0800 Kai-Heng Feng <[email protected]>…
Hi Nirmal,
Quoted 48 lines · Nirmal Patel
Quoted 38 lines · Nirmal Patel
Quoted 37 lines · Bjorn Helgaas
Quoted 8 lines · Nirmal Patel
...
Did you have a chance to look at my response on January 16th to your
questions? I tried to summarize all of the potential problems and issues with different fixes. Please let me know if it is easier if
I
resend the explanation. Thanks.I did see your Jan 16 response, thanks.
I had more questions after reading it, but they're more about understanding the topology seen by the host and the guest:
Jan 16:
https://lore.kernel.org/r/20240117004933.GA108810@bhelgaas
Feb 1:
https://lore.kernel.org/r/20240201211620.GA650432@bhelgaasAs I mentioned in my second Feb 1 response
(https://lore.kernel.org/r/20240201222245.GA650725@bhelgaas),
the usual plan envisioned by the PCI Firmware spec is that an OS can use
a
PCIe feature if the platform has granted the OS ownership via_OSCand
a device advertises the feature via a Capability in config space.My main concern with the v2 patch
(
https://lore.kernel.org/r/[email protected]
)
is that it overrides_OSCfornative_pcie_hotplug, but not for any of
the other features (AER, PME, LTR, DPC, etc.)Quoted 8 lines
Quoted 8 lines
I think it's hard to read the specs and conclude that PCIe hotplug is
a special case, and I think we're likely to have similar issues with
other features in the future.But if you think this is the best solution, I'm OK with merging it.
Quoted 7 lines
Hi Bjorn,
We tried some other ways to pass the information about all of the flags but I couldn't retrieve it in guest OS and VMD hardware doesn't have empty registers to write this information. So as of now we have this solution which only overwrites Hotplug flag. If you can accept it that would be great.
My original commit "PCI: vmd: Honor ACPI
_OSCon PCIe features" was a logically conclusion based on VMD ports are just apertures. Apparently there are more nuances than that, and people outside of Intel can't possibly know. And yes VMD creates lots of headaches during hardware enablement.
Quoted 4 lines
So is it possible to document the right way to use
_OSC, as Bjorn suggested?Kai-Heng
Well we are stuck here with two issues which are kind of chicken and egg problem.
1) VMD respects _OSC; which works excellent in Host but _OSC gives wrong information in Guest OS which ends up disabling Hotplug, AER, DPC, etc. We can live with AER, DPC disabled in VM but not the Hotplug.
2) VMD takes ownership of all the flags. For this to work VMD needs to know these settings from BIOS. VMD hardware doesn't have empty register where VMD UEFI driver can write this information. So honoring user settings for AER, Hotplug, etc from BIOS is not possible.
Where do you want me to add documentation? Will more information in the comment section of the Sltcap code change help?
Architecture wise VMD must own all of these flags but we have a hardware limitation to successfully pass the necessary information to the Host OS and the Guest OS.
Thanks
nirmal
Quoted 45 lines
Quoted 45 lines
Quoted 6 lines
In the host OS, this overrides whatever was negotiated via
_OSC, I guess on the principle that_OSCdoesn't apply because the host BIOS doesn't know about the Root Ports below the VMD. (I'm not sure it's 100% resolved that_OSCdoesn't apply to them, so we should mention that assumption here.)
_OSCstill controls every flag including Hotplug. I have observed that slot capabilities register has hotplug enabled only when platform has enabled the hotplug. So technically not overriding it in the host. It overrides in guest because_OSCis passing wrong/different information than the_OSCinformation in Host. Also like I mentioned, slot capabilities registers are not changed in Guest because vmd is passthrough.Quoted 10 lines
In the guest OS, this still overrides whatever was negotiated via
_OSC, but presumably the guest BIOS *does* know about the Root Ports, so I don't think the same principle about overriding_OSCapplies there.I think it's still a problem that we handle
host_bridge->native_pcie_hotplugdifferently than all the other features negotiated via_OSC. We should at least explain this somehow.Right now this is the only way to know in Guest OS if platform has enabled Hotplug or not. We have many customers complaining about Hotplug being broken in Guest. So just trying to honor
_OSCbut also fixing its limitation in Guest.Thanks
nirmal.Quoted 13 lines
I sincerely apologize for late responses. I just found out that my evolution mail client is automatically sending linux-pci emails to junk
since January 2024.At the moment Hotplug is an exception for us, but I do share your concern about other flags. We have done lot of testing and so far patch
v2 is the best solution we have.Thanks
nirmalQuoted 1 line
Bjorn
Quoted 85 lines · Nirmal Patel
Quoted 66 lines · On Thu, 7 Mar 2024 14:44:23 +0800 Kai-Heng Feng <[email protected]>…
Hi Nirmal,
Quoted 48 lines · Nirmal Patel
Quoted 38 lines · Nirmal Patel
Quoted 37 lines · Bjorn Helgaas
Quoted 8 lines · Nirmal Patel
...
Did you have a chance to look at my response on January 16th to your
questions? I tried to summarize all of the potential problems and issues with different fixes. Please let me know if it is easier if
I
resend the explanation. Thanks.I did see your Jan 16 response, thanks.
I had more questions after reading it, but they're more about understanding the topology seen by the host and the guest:
Jan 16:
https://lore.kernel.org/r/20240117004933.GA108810@bhelgaas
Feb 1:
https://lore.kernel.org/r/20240201211620.GA650432@bhelgaasAs I mentioned in my second Feb 1 response
(https://lore.kernel.org/r/20240201222245.GA650725@bhelgaas),
the usual plan envisioned by the PCI Firmware spec is that an OS can use
a
PCIe feature if the platform has granted the OS ownership via_OSCand
a device advertises the feature via a Capability in config space.My main concern with the v2 patch
(
https://lore.kernel.org/r/[email protected]
)
is that it overrides_OSCfornative_pcie_hotplug, but not for any of
the other features (AER, PME, LTR, DPC, etc.)Quoted 8 lines
Quoted 8 lines
I think it's hard to read the specs and conclude that PCIe hotplug is
a special case, and I think we're likely to have similar issues with
other features in the future.But if you think this is the best solution, I'm OK with merging it.
Quoted 7 lines
Hi Bjorn,
We tried some other ways to pass the information about all of the flags but I couldn't retrieve it in guest OS and VMD hardware doesn't have empty registers to write this information. So as of now we have this solution which only overwrites Hotplug flag. If you can accept it that would be great.
My original commit "PCI: vmd: Honor ACPI
_OSCon PCIe features" was a logically conclusion based on VMD ports are just apertures. Apparently there are more nuances than that, and people outside of Intel can't possibly know. And yes VMD creates lots of headaches during hardware enablement.Quoted 4 lines
So is it possible to document the right way to use
_OSC, as Bjorn suggested?Kai-Heng
Well we are stuck here with two issues which are kind of chicken and egg problem.
1) VMD respects_OSC; which works excellent in Host but_OSCgives wrong information in Guest OS which ends up disabling Hotplug, AER, DPC, etc. We can live with AER, DPC disabled in VM but not the Hotplug.2) VMD takes ownership of all the flags. For this to work VMD needs to know these settings from BIOS. VMD hardware doesn't have empty register where VMD UEFI driver can write this information. So honoring user settings for AER, Hotplug, etc from BIOS is not possible.
Quoted 6 lines
Where do you want me to add documentation? Will more information in the comment section of the Sltcap code change help?
Architecture wise VMD must own all of these flags but we have a hardware limitation to successfully pass the necessary information to the Host OS and the Guest OS.
If there's an official document on intel.com, it can make many things clearer and easier.
States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.
Kai-Heng
Quoted 49 lines
Thanks
nirmalQuoted 45 lines
Quoted 45 lines
Quoted 6 lines
In the host OS, this overrides whatever was negotiated via
_OSC, I guess on the principle that_OSCdoesn't apply because the host BIOS doesn't know about the Root Ports below the VMD. (I'm not sure it's 100% resolved that_OSCdoesn't apply to them, so we should mention that assumption here.)
_OSCstill controls every flag including Hotplug. I have observed that slot capabilities register has hotplug enabled only when platform has enabled the hotplug. So technically not overriding it in the host. It overrides in guest because_OSCis passing wrong/different information than the_OSCinformation in Host. Also like I mentioned, slot capabilities registers are not changed in Guest because vmd is passthrough.Quoted 10 lines
In the guest OS, this still overrides whatever was negotiated via
_OSC, but presumably the guest BIOS *does* know about the Root Ports, so I don't think the same principle about overriding_OSCapplies there.I think it's still a problem that we handle
host_bridge->native_pcie_hotplugdifferently than all the other features negotiated via_OSC. We should at least explain this somehow.Right now this is the only way to know in Guest OS if platform has enabled Hotplug or not. We have many customers complaining about Hotplug being broken in Guest. So just trying to honor
_OSCbut also fixing its limitation in Guest.Thanks
nirmal.Quoted 13 lines
I sincerely apologize for late responses. I just found out that my evolution mail client is automatically sending linux-pci emails to junk
since January 2024.At the moment Hotplug is an exception for us, but I do share your concern about other flags. We have done lot of testing and so far patch
v2 is the best solution we have.Thanks
nirmalQuoted 1 line
Bjorn
Nirmal Patel
maintainerQuoted 98 lines · On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]>…
Quoted 89 lines · Nirmal Patel
Quoted 68 lines · On Thu, 7 Mar 2024 14:44:23 +0800 Kai-Heng Feng <[email protected]>…
Hi Nirmal,
Quoted 49 lines · Nirmal Patel
Quoted 39 lines · Nirmal Patel
Quoted 38 lines · Bjorn Helgaas
Quoted 8 lines · Nirmal Patel
...
Did you have a chance to look at my response on January 16th to your
questions? I tried to summarize all of the potential problems and issues with different fixes. Please let me know if it is easier if
I
resend the explanation. Thanks.I did see your Jan 16 response, thanks.
I had more questions after reading it, but they're more about understanding the topology seen by the host and the
guest: Jan 16:
https://lore.kernel.org/r/20240117004933.GA108810@bhelgaas
Feb 1:
https://lore.kernel.org/r/20240201211620.GA650432@bhelgaasAs I mentioned in my second Feb 1 response
(https://lore.kernel.org/r/20240201222245.GA650725@bhelgaas),
the usual plan envisioned by the PCI Firmware spec is that an OS can use
a
PCIe feature if the platform has granted the OS ownership via_OSCand
a device advertises the feature via a Capability in config space.My main concern with the v2 patch
(
https://lore.kernel.org/r/[email protected]
)
is that it overrides_OSCfornative_pcie_hotplug, but not for any of
the other features (AER, PME, LTR, DPC, etc.)Quoted 8 lines
Quoted 8 lines
I think it's hard to read the specs and conclude that PCIe hotplug is
a special case, and I think we're likely to have similar issues with
other features in the future.But if you think this is the best solution, I'm OK with merging it.
Quoted 8 lines
Hi Bjorn,
We tried some other ways to pass the information about all of the flags but I couldn't retrieve it in guest OS and VMD hardware doesn't have empty registers to write this information. So as of now we have this solution which only overwrites Hotplug flag. If you can accept it that would be great.
My original commit "PCI: vmd: Honor ACPI
_OSCon PCIe features" was a logically conclusion based on VMD ports are just apertures. Apparently there are more nuances than that, and people outside of Intel can't possibly know. And yes VMD creates lots of headaches during hardware enablement.Quoted 4 lines
So is it possible to document the right way to use
_OSC, as Bjorn suggested?Kai-Heng
Well we are stuck here with two issues which are kind of chicken and egg problem.
1) VMD respects_OSC; which works excellent in Host but_OSCgives wrong information in Guest OS which ends up disabling Hotplug, AER, DPC, etc. We can live with AER, DPC disabled in VM but not the Hotplug.2) VMD takes ownership of all the flags. For this to work VMD needs to know these settings from BIOS. VMD hardware doesn't have empty register where VMD UEFI driver can write this information. So honoring user settings for AER, Hotplug, etc from BIOS is not possible.
Quoted 6 lines
Where do you want me to add documentation? Will more information in the comment section of the Sltcap code change help?
Architecture wise VMD must own all of these flags but we have a hardware limitation to successfully pass the necessary information to the Host OS and the Guest OS.
Quoted 6 lines
If there's an official document on intel.com, it can make many things clearer and easier.
States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.Kai-Heng
Hi Kai-Heng/ Bjorn,
Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to Documentation/PCI/controller/vmd.rst. Will that be okay?
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7:
"This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing _OSC settings from BIOS to VMD OS driver?
These 8 bits are more than enough for UEFI VMD driver to store all _OSC flags and VMD OS driver can read it during OS boot up. This will solve all of our issues.
Thanks
nirmal
Quoted 50 lines
Quoted 50 lines
Thanks
nirmalQuoted 45 lines
Quoted 45 lines
Quoted 6 lines
In the host OS, this overrides whatever was negotiated via
_OSC, I guess on the principle that_OSCdoesn't apply because the host BIOS doesn't know about the Root Ports below the VMD. (I'm not sure it's 100% resolved that_OSCdoesn't apply to them, so we should mention that assumption here.)
_OSCstill controls every flag including Hotplug. I have observed that slot capabilities register has hotplug enabled only when platform has enabled the hotplug. So technically not overriding it in the host. It overrides in guest because_OSCis passing wrong/different information than the_OSCinformation in Host. Also like I mentioned, slot capabilities registers are not changed in Guest because vmd is passthrough.Quoted 10 lines
In the guest OS, this still overrides whatever was negotiated via
_OSC, but presumably the guest BIOS *does* know about the Root Ports, so I don't think the same principle about overriding_OSCapplies there.I think it's still a problem that we handle
host_bridge->native_pcie_hotplugdifferently than all the other features negotiated via_OSC. We should at least explain this somehow.Right now this is the only way to know in Guest OS if platform has enabled Hotplug or not. We have many customers complaining about Hotplug being broken in Guest. So just trying to honor
_OSCbut also fixing its limitation in Guest.Thanks
nirmal.Quoted 13 lines
I sincerely apologize for late responses. I just found out that my evolution mail client is automatically sending linux-pci emails to junk
since January 2024.At the moment Hotplug is an exception for us, but I do share your concern about other flags. We have done lot of testing and so far patch
v2 is the best solution we have.Thanks
nirmalQuoted 1 line
Bjorn
Quoted 4 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote:
...
Quoted 8 lines
Quoted 4 lines
If there's an official document on intel.com, it can make many things clearer and easier.
States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?
I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 9 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7:
"This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing_OSCsettings from BIOS to VMD OS driver?
Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.
Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where PCI_CACHE_LINE_SIZE is still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.
Bjorn
Quoted 30 lines · Bjorn Helgaas
Quoted 4 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote:
...Quoted 8 lines
Quoted 4 lines
If there's an official document on intel.com, it can make many things clearer and easier.
States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 9 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7:
"This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing_OSCsettings from BIOS to VMD OS driver?Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.
Quoted 6 lines
Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where
PCI_CACHE_LINE_SIZEis still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.
I think the suggestion here is to use the VMD devices Cache Line Size register, not the other PCI devices. In that case we don't have to worry about conventional PCI devices because we aren't touching them.
Paul
Quoted 37 lines · Paul M Stillwell Jr
Quoted 30 lines · Bjorn Helgaas
Quoted 4 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote:
...Quoted 8 lines
Quoted 4 lines
If there's an official document on intel.com, it can make many things clearer and easier.
States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 9 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7:
"This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing_OSCsettings from BIOS to VMD OS driver?Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.Quoted 5 lines
Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where
PCI_CACHE_LINE_SIZEis still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.
Quoted 3 lines
I think the suggestion here is to use the VMD devices Cache Line Size register, not the other PCI devices. In that case we don't have to worry about conventional PCI devices because we aren't touching them.
Yes, but there is generic code that writes PCI_CACHE_LINE_SIZE for every device in some cases. If we wrote the VMD PCI_CACHE_LINE_SIZE, it would obliterate whatever you want to pass there.
Bjorn
Quoted 38 lines · Bjorn Helgaas
Quoted 37 lines · Paul M Stillwell Jr
Quoted 35 lines · Bjorn Helgaas
Quoted 4 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote:
...Quoted 8 lines
Quoted 4 lines
If there's an official document on intel.com, it can make many things clearer and easier.
States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 9 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7:
"This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing_OSCsettings from BIOS to VMD OS driver?Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where
PCI_CACHE_LINE_SIZEis still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.
Quoted 8 lines
Quoted 3 lines
I think the suggestion here is to use the VMD devices Cache Line Size register, not the other PCI devices. In that case we don't have to worry about conventional PCI devices because we aren't touching them.
Yes, but there is generic code that writes
PCI_CACHE_LINE_SIZEfor every device in some cases. If we wrote the VMDPCI_CACHE_LINE_SIZE, it would obliterate whatever you want to pass there.
Ah, got it. Thanks for clarifying.
Paul
Nirmal Patel
maintainerQuoted 44 lines · On Fri, 22 Mar 2024 18:36:37 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 38 lines · Paul M Stillwell Jr
Quoted 31 lines · Bjorn Helgaas
Quoted 10 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote:
...Quoted 5 lines
If there's an official document on intel.com, it can make many things clearer and easier.
States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.Quoted 3 lines
Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 9 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7: "This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing
_OSCsettings from BIOS to VMD OS driver?Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.Quoted 5 lines
Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where
PCI_CACHE_LINE_SIZEis still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.Quoted 4 lines
I think the suggestion here is to use the VMD devices Cache Line Size register, not the other PCI devices. In that case we don't have to worry about conventional PCI devices because we aren't touching them.
Quoted 5 lines
Yes, but there is generic code that writes
PCI_CACHE_LINE_SIZEfor every device in some cases. If we wrote the VMDPCI_CACHE_LINE_SIZE, it would obliterate whatever you want to pass there.Bjorn
Our initial testing showed no change in value by overwrite from pci. The registers didn't change in Host as well as in Guest OS when some data was written from BIOS. I will perform more testing and also make sure to write every register just in case.
Thanks for the response.
-nirmal
Quoted 52 lines · Nirmal Patel
Quoted 48 lines · On Fri, 22 Mar 2024 18:36:37 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 38 lines · Paul M Stillwell Jr
Quoted 31 lines · Bjorn Helgaas
Quoted 10 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote:
...Quoted 5 lines
If there's an official document on intel.com, it can make many things clearer and easier.
States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.Quoted 3 lines
Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 9 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7: "This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing
_OSCsettings from BIOS to VMD OS driver?Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.Quoted 5 lines
Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where
PCI_CACHE_LINE_SIZEis still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.Quoted 4 lines
I think the suggestion here is to use the VMD devices Cache Line Size register, not the other PCI devices. In that case we don't have to worry about conventional PCI devices because we aren't touching them.
Yes, but there is generic code that writes
PCI_CACHE_LINE_SIZEfor every device in some cases. If we wrote the VMDPCI_CACHE_LINE_SIZE, it would obliterate whatever you want to pass there.
Quoted 5 lines
Quoted 1 line
Bjorn
Our initial testing showed no change in value by overwrite from pci. The registers didn't change in Host as well as in Guest OS when some data was written from BIOS. I will perform more testing and also make sure to write every register just in case.
If the VMD hardware is designed in this way and there's an official document states that "VMD ports should follow _OSC expect for hotplugging" then IMO there's no need to find alternative.
Kai-Heng
Quoted 4 lines
Thanks for the response.
-nirmal
Quoted 57 lines · Kai-Heng Feng
Quoted 56 lines · Nirmal Patel
Quoted 52 lines · On Fri, 22 Mar 2024 18:36:37 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 40 lines · Paul M Stillwell Jr
Quoted 32 lines · Bjorn Helgaas
Quoted 8 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote: ...
Quoted 4 lines
If there's an official document on intel.com, it can make many things clearer and easier. States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.
Quoted 4 lines
Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 10 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7: "This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing
_OSCsettings from BIOS to VMD OS driver?Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.Quoted 6 lines
Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where
PCI_CACHE_LINE_SIZEis still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.Quoted 4 lines
I think the suggestion here is to use the VMD devices Cache Line Size register, not the other PCI devices. In that case we don't have to worry about conventional PCI devices because we aren't touching them.
Yes, but there is generic code that writes
PCI_CACHE_LINE_SIZEfor every device in some cases. If we wrote the VMDPCI_CACHE_LINE_SIZE, it would obliterate whatever you want to pass there.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Bjorn
Our initial testing showed no change in value by overwrite from pci. The registers didn't change in Host as well as in Guest OS when some data was written from BIOS. I will perform more testing and also make sure to write every register just in case.
If the VMD hardware is designed in this way and there's an official document states that "VMD ports should follow
_OSCexpect for hotplugging" then IMO there's no need to find alternative.
There isn't any official documentation for this, it's just the way VMD is designed :). VMD assumes that everything behind it is hotpluggable so the bits don't *really* matter. In other words, VMD supports hotplug and you can't turn that off so hotplug is always on regardless of what the bits say.
I believe prior to 04b12ef163d1 ("PCI: vmd: Honor ACPI _OSC on PCIe
features") VMD ignored the flags (which was ok, but led to the issue
that the patch corrected). I think that patch is fine except for the hotplug bits because the hypervisor BIOS isn't filling them in correctly (if I understand the problem correctly). As I mentioned earlier the VMD design is such that VMD is going to handle all the hotplug events so the bits in the root bridge for hotplug are irrelevant WRT VMD.
Paul
Quoted 6 lines
Kai-Heng
Quoted 4 lines
Thanks for the response.
-nirmal
Quoted 73 lines · Paul M Stillwell Jr
Quoted 57 lines · Kai-Heng Feng
Quoted 56 lines · Nirmal Patel
Quoted 52 lines · On Fri, 22 Mar 2024 18:36:37 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 40 lines · Paul M Stillwell Jr
Quoted 32 lines · Bjorn Helgaas
Quoted 8 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote: ...
Quoted 4 lines
If there's an official document on intel.com, it can make many things clearer and easier. States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.
Quoted 4 lines
Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 10 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7: "This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing
_OSCsettings from BIOS to VMD OS driver?Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.Quoted 6 lines
Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where
PCI_CACHE_LINE_SIZEis still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.Quoted 4 lines
I think the suggestion here is to use the VMD devices Cache Line Size register, not the other PCI devices. In that case we don't have to worry about conventional PCI devices because we aren't touching them.
Yes, but there is generic code that writes
PCI_CACHE_LINE_SIZEfor every device in some cases. If we wrote the VMDPCI_CACHE_LINE_SIZE, it would obliterate whatever you want to pass there.Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Bjorn
Our initial testing showed no change in value by overwrite from pci. The registers didn't change in Host as well as in Guest OS when some data was written from BIOS. I will perform more testing and also make sure to write every register just in case.
If the VMD hardware is designed in this way and there's an official document states that "VMD ports should follow
_OSCexpect for hotplugging" then IMO there's no need to find alternative.There isn't any official documentation for this, it's just the way VMD is designed :). VMD assumes that everything behind it is hotpluggable so the bits don't *really* matter. In other words, VMD supports hotplug and you can't turn that off so hotplug is always on regardless of what the bits say.
Quoted 4 lines
I believe prior to
04b12ef163d1("PCI: vmd: Honor ACPI_OSCon PCIe
features") VMD ignored the flags (which was ok, but led to the issue
that the patch corrected). I think that patch is fine except for the hotplug bits because the hypervisor BIOS isn't filling them in correctly (if I understand the problem correctly). As I mentioned earlier the VMD design is such that VMD is going to handle all the hotplug events so the bits in the root bridge for hotplug are irrelevant WRT VMD.
I should have been clearer in one aspect of this response: the hypervisor BIOS sets all the _OSC bits to zero, not just the hotplug bits. It's just that the hotplug bits cause us issues when VMD is being used in a VM because it disables hotplug and we need it to be enabled because our customers expect to always be able to hotplug devices owned by VMD.
Quoted 10 lines
Paul
Quoted 6 lines
Kai-Heng
Quoted 4 lines
Thanks for the response.
-nirmal
Quoted 68 lines · Paul M Stillwell Jr
Quoted 58 lines · Kai-Heng Feng
Quoted 56 lines · Nirmal Patel
Quoted 52 lines · On Fri, 22 Mar 2024 18:36:37 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 40 lines · Paul M Stillwell Jr
Quoted 32 lines · Bjorn Helgaas
Quoted 8 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote: ...
Quoted 4 lines
If there's an official document on intel.com, it can make many things clearer and easier. States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.
Quoted 4 lines
Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 10 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7: "This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing
_OSCsettings from BIOS to VMD OS driver?Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.Quoted 6 lines
Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where
PCI_CACHE_LINE_SIZEis still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.Quoted 4 lines
I think the suggestion here is to use the VMD devices Cache Line Size register, not the other PCI devices. In that case we don't have to worry about conventional PCI devices because we aren't touching them.
Yes, but there is generic code that writes
PCI_CACHE_LINE_SIZEfor every device in some cases. If we wrote the VMDPCI_CACHE_LINE_SIZE, it would obliterate whatever you want to pass there.Quoted 8 lines
Quoted 4 lines
Our initial testing showed no change in value by overwrite from pci. The registers didn't change in Host as well as in Guest OS when some data was written from BIOS. I will perform more testing and also make sure to write every register just in case.
If the VMD hardware is designed in this way and there's an official document states that "VMD ports should follow
_OSCexpect for hotplugging" then IMO there's no need to find alternative.
Quoted 5 lines
There isn't any official documentation for this, it's just the way VMD is designed :). VMD assumes that everything behind it is hotpluggable so the bits don't *really* matter. In other words, VMD supports hotplug and you can't turn that off so hotplug is always on regardless of what the bits say.
This whole discussion is about who *owns* the feature, not whether the hardware supports the feature.
The general rule is "if _OSC covers the feature, the platform owns the feature and the OS shouldn't touch it unless the OS requests and is granted control of it."
VMD wants special treatment, and we'd like a simple description of what that treatment is. Right now it sounds like you want the OS to own *hotplug* below VMD regardless of what _OSC says.
But I still have no idea about other features like AER, etc., so we're kind of in no man's land about how to handle them.
Bjorn
Quoted 69 lines · Bjorn Helgaas
Quoted 68 lines · Paul M Stillwell Jr
Quoted 66 lines · Kai-Heng Feng
Quoted 56 lines · Nirmal Patel
Quoted 47 lines · On Fri, 22 Mar 2024 18:36:37 -0500 Bjorn Helgaas <[email protected]> wrote:
Quoted 40 lines · Paul M Stillwell Jr
Quoted 32 lines · Bjorn Helgaas
Quoted 8 lines · Nirmal Patel
On Fri, 15 Mar 2024 09:29:32 +0800 Kai-Heng Feng <[email protected]> wrote: ...
Quoted 4 lines
If there's an official document on intel.com, it can make many things clearer and easier. States what VMD does and what VMD expect OS to do can be really helpful. Basically put what you wrote in an official document.
Quoted 4 lines
Thanks for the suggestion. I can certainly find official VMD architecture document and add that required information to
Documentation/PCI/controller/vmd.rst. Will that be okay?I'd definitely be interested in whatever you can add to illuminate these issues.
Quoted 10 lines
I also need your some help/suggestion on following alternate solution. We have been looking at VMD HW registers to find some empty registers. Cache Line Size register offset OCh is not being used by VMD. This is the explanation in PCI spec 5.0 section 7.5.1.1.7: "This read-write register is implemented for legacy compatibility purposes but has no effect on any PCI Express device behavior." Can these registers be used for passing
_OSCsettings from BIOS to VMD OS driver?Quoted 3 lines
These 8 bits are more than enough for UEFI VMD driver to store all
_OSCflags and VMD OS driver can read it during OS boot up. This will solve all of our issues.Quoted 6 lines
Interesting idea. I think you'd have to do some work to separate out the conventional PCI devices, where
PCI_CACHE_LINE_SIZEis still relevant, to make sure nothing breaks. But I think we overwrite it in some cases even for PCIe devices where it's pointless, and it would be nice to clean that up.Quoted 4 lines
I think the suggestion here is to use the VMD devices Cache Line Size register, not the other PCI devices. In that case we don't have to worry about conventional PCI devices because we aren't touching them.
Quoted 5 lines
Yes, but there is generic code that writes
PCI_CACHE_LINE_SIZEfor every device in some cases. If we wrote the VMDPCI_CACHE_LINE_SIZE, it would obliterate whatever you want to pass there.Quoted 4 lines
Our initial testing showed no change in value by overwrite from pci. The registers didn't change in Host as well as in Guest OS when some data was written from BIOS. I will perform more testing and also make sure to write every register just in case.
If the VMD hardware is designed in this way and there's an official document states that "VMD ports should follow
_OSCexpect for hotplugging" then IMO there's no need to find alternative.
Quoted 9 lines
Quoted 5 lines
There isn't any official documentation for this, it's just the way VMD is designed :). VMD assumes that everything behind it is hotpluggable so the bits don't *really* matter. In other words, VMD supports hotplug and you can't turn that off so hotplug is always on regardless of what the bits say.
This whole discussion is about who *owns* the feature, not whether the hardware supports the feature.
I'm not disagreeing about who owns the feature :) I'm trying to find a solution when the data that the driver gets is wrong.
Quoted 4 lines
The general rule is "if
_OSCcovers the feature, the platform owns the feature and the OS shouldn't touch it unless the OS requests and is granted control of it."
The code is fine in a bare metal system, but completely broken in a hypervisor because all the _OSC bits are set to 0. So I am trying to find a solution where the hotplug bits can be set correctly in this situation.
Quoted 4 lines
VMD wants special treatment, and we'd like a simple description of what that treatment is. Right now it sounds like you want the OS to own *hotplug* below VMD regardless of what
_OSCsays.
What I want is for hotplug to be enabled in a hypervisor :) The slot capability register has the correct bits set, but it doesn't make sense (to me) to use a register in a root port to set the bridge settings. That's why this patch just removed the hotplug bits from the code. Everything is set up correctly (as far as I can tell) in both bare metal and hypervisors if we omit those 2 lines.
Does anyone have a better suggestion on how to handle the case where the _OSC bits are incorrect?
Paul
Quoted 4 lines
But I still have no idea about other features like AER, etc., so we're kind of in no man's land about how to handle them.
Bjorn
Quoted 35 lines · Nirmal Patel
On Fri, 2024-01-12 at 16:55 -0600, Bjorn Helgaas wrote:
...Quoted 5 lines
I think we're converging on the idea that since VMD is effectively *not* an ACPI host bridge and doesn't have its own
_OSC, the_OSCthat applies to the VMD endpoint does *not* apply to the hierarchy below the VMD. In that case, the default is that the OS owns all the features (hotplug, AER, etc) below the VMD.Well there will be few problems if VMD owns/assigns all the flags by itself. We discussed all of the potential problems but due to the holidays I think I should summarize them again.
#1 : Currently there is no way to pass the information about AER, DPC, etc to VMD driver from BIOS or from boot parameter. For example, if VMD blindly enables AER and platform has AER disabled, then we will see AERs from devices under VMD but user have no way to toggle it since we rejected idea of adding boot parameter for AER, DPC under VMD. I believe you also didn't like the idea of sysfs knob suggested by Kai-Heng.
#2 : Even if we use VMD hardware register to store AER, DPC and make UEFI VMD driver to write information about Hotplug, DPC, AER, we still dont have a way to change the setting if user wants to alter them. Also the issue will still persist in client platforms since we don't own their UEFI VMD driver. It will be a huge effort.
#3 : At this moment, user can enable/disable only Hotplug in VMD BIOS settings (meaning no AER, DPC, LTR, etc)and VMD driver can read it from SltCap register. This means BIOS needs to add other settings and VMD needs to figure out to read them which at this moment VMD can't do it.
#4 : consistency with Host OS and Guest OS.
Quoted 7 lines
I believe the current proposed patch is the best option which requires minimal changes without breaking other platform features and unblock our customers. This issues has been a blocker for us.
For your concerns regarding how VMD can own all the
_OSCfeatures, i am open to other ideas and will discuss with the architect if they have any suggestions.
As I understand it, the basic model of PCIe features is:
- If platform doesn't have a way to negotiate ownership of PCIe features, the OS owns them by default, e.g., on non-ACPI systems.
- If platform does have a way to negotiate, e.g., ACPI _OSC, the platform owns the features until it grants ownership to the OS.
- If the OS has ownership (either by default or granted by platform), it can use the feature if the hardware device advertises support.
I think this model applies to all PCIe features, including hotplug, AER, DPC, etc., and the OS uses _OSC and the Capabilities in device config space to determine whether to use the features.
_OSC is the obvious way for a platform to use BIOS settings to influence what the OS does. I think the problem with VMD is that you have a guest OS running on a platform (the guest VM) where you want a host BIOS setting to control things in that guest platform, but there's currently no way to influence the _OSC seen by the guest.
I think we need to:
- Clarify whether _OSC only applies in the domain of the host bridge where it appears, i.e., an _OSC in a host bridge to domain 0000 obviously applies to a VMD Endpoint in domain 0000; does it also apply to devices in the domain 10000 *below* the VMD Endpoint?
- Describe what the VMD bridge does with events below it. E.g., normally a Root Port that receives an error Message generates an interrupt depending on its Interrupt Disable and Error Reporting Enable bits. What happens when it's a VMD Root Port? Does it forward an error Message up via the VMD Endpoint? Generate an interrupt via the VMD Endpoint? If so, which interrupt?
The question of where _OSC applies feels like an architectural thing.
The question of how AER, DPC, hotplug, etc. events are forwarded across the VMD Root Port/Endpoint might be, too, or maybe that's all driver-specific, I dunno. Either way, I think it's worth documenting somehow.
Bjorn
Quoted 3 lines · Nirmal Patel
...
In guest, vmd is passthrough including pci topology behind vmd. ...
IIUC this says the VMD bridge itself is passed through to the guest, as well as the VMD Root Ports and devices below them.
Why is the VMD bridge itself passed through to the guest? I assume the host OS vmd driver enables the VMD bridge functionality, i.e., in vmd_enable_domain()?
I gleaned this from the dmesg logs at
https://bugzilla.kernel.org/show_bug.cgi?id=215027. I assume these are from the host:
pci 0000:00:0e.0: [8086:467f] type 00 # VMD Endpoint
vmd 0000:00:0e.0: PCI host bridge to bus 10000:e0 pci_bus 10000:e0: root bus resource [bus e0-ff]
pci 10000:e0:06.0: [8086:464d] type 01 # VMD Root Port
pci 10000:e0:06.0: PCI bridge to [bus e1]
pci 10000:e1:00.0: [144d:a80a] type 00 # NVMe
nvme 10000:e1:00.0: PCI INT A: not connected
What does it look like in the guest? Does the guest see all three of these devices (VMD Endpoint, VMD Root Port, NVMe)?
If the guest sees the VMD Endpoint, what does it do with it? Does the host vmd driver coordinate with the guest vmd driver?
Bjorn
Lorenzo Pieralisi
maintainerOn Tue, Dec 05, 2023 at 03:20:27PM -0700, Nirmal Patel wrote:
[...]
Quoted 18 lines
Quoted 5 lines
In the host OS, this overrides whatever was negotiated via
_OSC, I guess on the principle that_OSCdoesn't apply because the host BIOS doesn't know about the Root Ports below the VMD. (I'm not sure it's 100% resolved that_OSCdoesn't apply to them, so we should mention that assumption here.)
_OSCstill controls every flag including Hotplug. I have observed that slot capabilities register has hotplug enabled only when platform has enabled the hotplug. So technically not overriding it in the host. It overrides in guest because_OSCis passing wrong/different information than the_OSCinformation in Host.
Also like I mentioned, slot capabilities registers are not changed in Guest because vmd is passthrough.Quoted 6 lines
In the guest OS, this still overrides whatever was negotiated via
_OSC, but presumably the guest BIOS *does* know about the Root Ports, so I don't think the same principle about overriding_OSCapplies there.
Quoted 8 lines
Quoted 4 lines
I think it's still a problem that we handle
host_bridge->native_pcie_hotplugdifferently than all the other features negotiated via_OSC. We should at least explain this somehow.Right now this is the only way to know in Guest OS if platform has enabled Hotplug or not. We have many customers complaining about Hotplug being broken in Guest. So just trying to honor
_OSCbut also fixing its limitation in Guest.
The question is: should _OSC settings apply to the VMD hierarchy ?
Yes or no (not maybe) ?
If yes, the guest firmware is buggy (and I appreciate it is hard to fix). If no this patch should also change vmd_copy_host_bridge_flags() and remove the native_pcie_hotplug initialization from the root bridge.
As a matter of fact this patch overrides the _OSC settings in host and guest and it is not acceptable because it is not based on any documented policy (if there is a policy please describe it).
Quoted 6 lines
Quoted 4 lines
I suspect part of the reason for doing it differently is to avoid the interrupt storm that we avoid via
04b12ef163d1("PCI: vmd: Honor ACPI_OSCon PCIe features"). If so, I think04b12ef163d1should be mentioned here because it's an important piece of the picture.I can add a note about this patch as well. Let me know if you want me to add something specific.
At the very least you need to explain the problem you are solving in the commit log - sentences like:
"This patch will make sure that Hotplug is enabled properly in Host as well as in VM while honoring _OSC settings as well as VMD hotplug setting."
aren't useful to someone who will look into this in the future.
If _OSC does not grant the host kernel HP handling and *any* (that's another choice that should be explained) slot
capability reports that the VMD root port is HP capable we do override _OSC, we are not honouring it, AFAICS.
Then we can say that in your user experience the stars always align and what I mention above is not a problem because it can't happen but it is hard to grok by just reading the code and more importantly, it is not based on any standard documentation.
Quoted 3 lines
Thank you for taking the time. This is a very critical issue for us and we have many customers complaining about it, I would appreciate to get it resolved as soon as possible.
Sure but we need to solve it in a way that does not break tomorrow otherwise we will keep applying plasters on top of plasters.
Ignoring _OSC hotplug settings for VMD bridges in *both* host and guests may be an acceptable - though a tad controversial - policy (if based on software/hardware guidelines).
A mixture thereof in my opinion is not acceptable.
Thanks,
Lorenzo