PCI: vmd: Fix boot failure when trying to clean up domain before enumeration
Subsystems (3)
PATCH 1/1 PCI: vmd: Fix boot failure when trying to clean up domain before enumeration
From: Adrian Huang <[email protected]>
Commit 6aab5622296b ("PCI: vmd: Clean up domain before enumeration") clears PCI configuration space of VMD root ports. However, the host OS cannot boot successfully with the following error message:
vmd 0000:64:05.5: PCI host bridge to bus 10000:00
...
vmd 0000:64:05.5: Bound to PCI domain 10000
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: Invalidation Time-out Error (ITE) cleared
The root cause is that memset_io() clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This might enable prefetchable memory if the device disables prefetchable memory originally. Here is an example (before memset_io()):
PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
00 00 00 00 40 00 00 00 00 00 00 00 00 01 02 00
So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:
The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.
When memset_io() clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled. This behavior causes that the content of PCI configuration space of VMD root ports is 0xff after invoking memset_io() in vmd_domain_reset():
10000:00:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev ff) (prog-if ff)
!!! Unknown header type 7f
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
...
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
10000:00:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port B (rev ff) (prog-if ff)
!!! Unknown header type 7f
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
...
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
To fix the issue, prefetchable limit upper 32 bits register needs to be cleared firstly. This also adheres to the implementation of pci_setup_bridge_mmio_pref(). Please see the function for detail.
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=216644
--- drivers/pci/controller/vmd.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/pci/controller/vmd.c +3 -0
@@ -526,6 +526,9 @@ static void vmd_domain_reset(struct vmd_dev *vmd) PCI_CLASS_BRIDGE_PCI))continue ; +/* Clear the upper 32 bits of PREF limit. */ +memset_io (base +PCI_PREF_LIMIT_UPPER32 ,0 ,4 ); +memset_io (base +PCI_IO_BASE ,0 ,PCI_ROM_ADDRESS1 -PCI_IO_BASE ); }
Signature · 1 line
2.31.1
Quoted 78 lines · Adrian Huang
From: Adrian Huang <[email protected]>
Commit
6aab5622296b("PCI: vmd: Clean up domain before enumeration") clears PCI configuration space of VMD root ports. However, the host OS cannot boot successfully with the following error message:vmd 0000:64:05.5: PCI host bridge to bus 10000:00
...
vmd 0000:64:05.5: Bound to PCI domain 10000
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 =0x0, qw1 =0x0
DMAR: QI PRIOR: UNKNOWN qw0 =0x0, qw1 =0x0
DMAR: Invalidation Time-out Error (ITE) clearedThe root cause is that
memset_io()clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This might enable prefetchable memory if the device disables prefetchable memory originally. Here is an example (beforememset_io()):PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
00 00 00 00 40 00 00 00 00 00 00 00 00 01 02 00So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:
The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.
When
memset_io()clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled. This behavior causes that the content of PCI configuration space of VMD root ports is0xffafter invokingmemset_io()invmd_domain_reset():10000:00:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev ff) (prog-if ff)
!!! Unknown header type 7f
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
...
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff10000:00:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port B (rev ff) (prog-if ff)
!!! Unknown header type 7f
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
...
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ffTo fix the issue, prefetchable limit upper 32 bits register needs to be cleared firstly. This also adheres to the implementation of
pci_setup_bridge_mmio_pref(). Please see the function for detail.Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=216644
Fixes 6aab5622296b ("PCI: vmd: Clean up domain before enumeration")Cc Nirmal Patel <[email protected]>Signed-off-by Adrian Huang <[email protected]>--- drivers/pci/controller/vmd.c | 3 +++ 1 file changed, 3 insertions(+)drivers/pci/controller/vmd.c +3
-0@@ -526,6 +526,9 @@ static void vmd_domain_reset(struct vmd_dev *vmd) PCI_CLASS_BRIDGE_PCI))continue ; +/* Clear the upper 32 bits of PREF limit. */ +memset_io (base +PCI_PREF_LIMIT_UPPER32 ,0 ,4 ); +memset_io (base +PCI_IO_BASE ,0 ,PCI_ROM_ADDRESS1 -PCI_IO_BASE ); }
Can we make the subject line any more specific about what this patch does? Apparently this is really about avoiding accidental enablement of the window because the base & limit can't be updated atomically?
Quoted 31 lines · Adrian Huang
From: Adrian Huang <[email protected]>
Commit
6aab5622296b("PCI: vmd: Clean up domain before enumeration") clears PCI configuration space of VMD root ports. However, the host OS cannot boot successfully with the following error message:vmd 0000:64:05.5: PCI host bridge to bus 10000:00
...
vmd 0000:64:05.5: Bound to PCI domain 10000
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 =0x0, qw1 =0x0
DMAR: QI PRIOR: UNKNOWN qw0 =0x0, qw1 =0x0
DMAR: Invalidation Time-out Error (ITE) clearedThe root cause is that
memset_io()clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This might enable prefetchable memory if the device disables prefetchable memory originally. Here is an example (beforememset_io()):PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
00 00 00 00 40 00 00 00 00 00 00 00 00 01 02 00So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:
Quoted 6 lines
The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.
When
memset_io()clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled.
#define PCI_IO_BASE 0x1c #define PCI_PREF_LIMIT_UPPER32 0x2c #define PCI_ROM_ADDRESS1 0x38
memset_io(base + PCI_IO_BASE, 0, PCI_ROM_ADDRESS1 - PCI_IO_BASE);
The memset does clear PCI_PREF_LIMIT_UPPER32 already, but I think you're saying that PCI_PREF_MEMORY_BASE, PCI_PREF_MEMORY_LIMIT, and PCI_PREF_BASE_UPPER32 are cleared first, so there is a time when the prefetchable base is zero and the limit is non-zero, so the window is enabled.
I would expect that to be a transient thing that you wouldn't be likely to trip over, but you seem to see it consistently.
Quoted 2 lines
This behavior causes that the content of PCI configuration space of VMD root ports is
0xffafter invokingmemset_io()invmd_domain_reset():
Well, it doesn't actually change the content of config space, does it? I assume these config accesses get routed the wrong place because the window is enabled, and some PCI error like Unsupported Request is getting turned into ~0?
I don't understand why this seems to be a stable state, though.
Seems sort of unexpected to me that config space would be in a *prefetchable* window, but I guess that must be a VMD thing.
Quoted 37 lines
10000:00:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev ff) (prog-if ff)
!!! Unknown header type 7f
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
...
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff10000:00:01.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port B (rev ff) (prog-if ff)
!!! Unknown header type 7f
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
...
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ffTo fix the issue, prefetchable limit upper 32 bits register needs to be cleared firstly. This also adheres to the implementation of
pci_setup_bridge_mmio_pref(). Please see the function for detail.Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=216644
Fixes 6aab5622296b ("PCI: vmd: Clean up domain before enumeration")Cc Nirmal Patel <[email protected]>Signed-off-by Adrian Huang <[email protected]>--- drivers/pci/controller/vmd.c | 3 +++ 1 file changed, 3 insertions(+)drivers/pci/controller/vmd.c +3
-0@@ -526,6 +526,9 @@ static void vmd_domain_reset(struct vmd_dev *vmd) PCI_CLASS_BRIDGE_PCI))continue ; +/* Clear the upper 32 bits of PREF limit. */ +memset_io (base +PCI_PREF_LIMIT_UPPER32 ,0 ,4 ); +memset_io (base +PCI_IO_BASE ,0 ,PCI_ROM_ADDRESS1 -PCI_IO_BASE );
Quoted 4 lines · Bjorn Helgaas
Can we make the subject line any more specific about what this patch does? Apparently this is really about avoiding accidental enablement of the window because the base & limit can't be updated atomically?
Sure, I will rephrase the subject.
Quoted 11 lines
#define PCI_IO_BASE 0x1c #define PCI_PREF_LIMIT_UPPER32 0x2c #define PCI_ROM_ADDRESS1 0x38memset_io(base + PCI_IO_BASE, 0, PCI_ROM_ADDRESS1 - PCI_IO_BASE);The memset does clear
PCI_PREF_LIMIT_UPPER32already, but I think you're saying thatPCI_PREF_MEMORY_BASE,PCI_PREF_MEMORY_LIMIT, andPCI_PREF_BASE_UPPER32are cleared first, so there is a time when the prefetchable base is zero and the limit is non-zero, so the window is enabled.
Yes, your understanding is correct.
Quoted 10 lines
I would expect that to be a transient thing that you wouldn't be likely to trip over, but you seem to see it consistently.
Quoted 2 lines
This behavior causes that the content of PCI configuration space of VMD root ports is
0xffafter invokingmemset_io()invmd_domain_reset():Well, it doesn't actually change the content of config space, does it? I assume these config accesses get routed the wrong place because the window is enabled, and some PCI error like Unsupported Request is getting turned into ~0?
Yes, I think so.
I'll rephrase the commit message accordingly.
-- Adrian