PCI: vmd: Disable bridge window for domain reset
Subsystems (3)
Nirmal Patel
maintainerPATCH v3 1/1 PCI: vmd: Disable bridge window for domain reset
During domain reset process vmd_domain_reset() clears PCI configuration space of VMD root ports. But certain platform has observed following errors and failed to boot.
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: Invalidation Time-out Error (ITE) cleared
The root cause is that memset_io() clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This seems to be enabling prefetchable memory if the device disabled prefetchable memory originally.
Here is an example (before memset_io()):
PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
...
So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. When memset_io() clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled.
This is believed to be the reason for the failure and in addition the sequence of operation in vmd_domain_reset() is not following the PCIe specs.
Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:
The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridg
Disable the bridge window by executing a sequence of operations borrowed from pci_disable_bridge_window(), that comply with the PCI specifications.
---v3->v3: Add more information to commit description. v1->v2: Follow same chain of operation as pci_disable_bridge_window
and update commit log.
--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +10 -2
@@ -526,8 +526,16 @@ static void vmd_domain_reset(struct vmd_dev *vmd) PCI_CLASS_BRIDGE_PCI))continue ; -memset_io (base +PCI_IO_BASE ,0 , -PCI_ROM_ADDRESS1 -PCI_IO_BASE ); +writel (0 , base +PCI_IO_BASE ); +/* MMIO Base/Limit */ +writel (0x0000fff0 , base +PCI_MEMORY_BASE ); + +/* Prefetchable MMIO Base/Limit */ +writel (0 , base +PCI_PREF_LIMIT_UPPER32 ); +writel (0x0000fff0 , base +PCI_PREF_MEMORY_BASE ); +writel (0xffffffff , base +PCI_PREF_BASE_UPPER32 ); +writel (0 , base +PCI_IO_BASE_UPPER16 ); +writeb (0 , base +PCI_CAPABILITY_LIST ); } } }
Signature · 1 line
2.31.1
Quoted 11 lines · Nirmal Patel
During domain reset process
vmd_domain_reset()clears PCI configuration space of VMD root ports. But certain platform has observed following errors and failed to boot.
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 =0x0, qw1 =0x0
DMAR: QI PRIOR: UNKNOWN qw0 =0x0, qw1 =0x0
DMAR: Invalidation Time-out Error (ITE) cleared
Quoted 5 lines
The root cause is that
memset_io()clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This seems to be enabling prefetchable memory if the device disabled prefetchable memory originally.
Here is an example (beforememset_io()):
Paragraph separation or wrapping error.
Quoted 15 lines
PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
...So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. When
memset_io()clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled.This is believed to be the reason for the failure and in addition the sequence of operation in
vmd_domain_reset()is not following the PCIe specs.
Quoted 5 lines
Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:
The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridg
s/bridg/bridge/
The [mem 0x0-0x575000fffff] state is transient, right? After the memset_io() completes, the window is [mem 0x0-0x000fffff], which still not what you want, since it's enabled, while you want to *disable* the window.
I don't know what the connection between this and the DMAR invalidation queue errors is. Maybe those can happen with either the transient [mem 0x0-0x575000fffff] state or the [mem 0x0-0x000fffff] end state?
IIUC there are two problems with the memset_io():
1) The memset_io() writes 0 to all the base and limit registers. For address decoding purposes, the low bits of the base are implicitly clear while the low bits of the limit are implicitly set, so setting the base to zero always makes the windows *enabled*, e.g., [io 0x0000-0x0fff].
2) The I/O and prefetchable base/limit can't be configured with a single config write, so we have to write them in a specific order to avoid transient enabled windows that could cause conflicts.
Quoted 18 lines
Disable the bridge window by executing a sequence of operations borrowed from
pci_disable_bridge_window(), that comply with the PCI specifications.Signed-off-by Nirmal Patel <[email protected]>---
v3->v3: Add more information to commit description.v1->v2: Follow same chain of operation aspci_disable_bridge_window
and update commit log.--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)drivers/pci/controller/vmd.c +0
-0
drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 529
Quoted 6 lines
drivers/pci/controller/vmd.c +1
-2@@ -526,8 +526,16 @@ static void vmd_domain_reset(struct vmd_dev *vmd) PCI_CLASS_BRIDGE_PCI))continue ; -memset_io (base +PCI_IO_BASE ,0 , -PCI_ROM_ADDRESS1 -PCI_IO_BASE ); +writel (0 , base +PCI_IO_BASE );
This is a 32-bit write, so it writes zero to PCI_IO_BASE, PCI_IO_LIMIT, and PCI_SEC_STATUS. Writing zero to PCI_SEC_STATUS is probably harmless since everything there is RO or RW1C, but is unnecessary and seems a little sloppy.
Writing zero to PCI_IO_BASE and PCI_IO_LIMIT enables it as [io 0x0000-0x0fff]. I think the code in pci_setup_bridge_io() is more like what you want.
drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 537
Quoted 8 lines
drivers/pci/controller/vmd.c +8
-0+/* MMIO Base/Limit */ +writel (0x0000fff0 , base +PCI_MEMORY_BASE ); + +/* Prefetchable MMIO Base/Limit */ +writel (0 , base +PCI_PREF_LIMIT_UPPER32 ); +writel (0x0000fff0 , base +PCI_PREF_MEMORY_BASE ); +writel (0xffffffff , base +PCI_PREF_BASE_UPPER32 ); +writel (0 , base +PCI_IO_BASE_UPPER16 );
kkkk
Quoted 7 lines
drivers/pci/controller/vmd.c +1
-0+writeb (0 , base +PCI_CAPABILITY_LIST ); } } }Signature · 1 line
2.31.1
Patel, Nirmal
maintainerQuoted 12 lines · Bjorn Helgaas
Quoted 11 lines · Nirmal Patel
During domain reset process
vmd_domain_reset()clears PCI configuration space of VMD root ports. But certain platform has observed following errors and failed to boot.
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 =0x0, qw1 =0x0
DMAR: QI PRIOR: UNKNOWN qw0 =0x0, qw1 =0x0
DMAR: Invalidation Time-out Error (ITE) cleared
Quoted 6 lines
Quoted 5 lines
The root cause is that
memset_io()clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This seems to be enabling prefetchable memory if the device disabled prefetchable memory originally.
Here is an example (beforememset_io()):Paragraph separation or wrapping error.
I will fix it.
Quoted 41 lines
Quoted 16 lines
PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
...So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. When
memset_io()clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled.This is believed to be the reason for the failure and in addition the sequence of operation in
vmd_domain_reset()is not following the PCIe specs.Quoted 5 lines
Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:
The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridg
s/bridg/bridge/
The [mem 0x0-0x575000fffff] state is transient, right? After the
memset_io()completes, the window is [mem 0x0-0x000fffff], which still not what you want, since it's enabled, while you want to *disable* the window.I don't know what the connection between this and the DMAR invalidation queue errors is. Maybe those can happen with either the transient [mem 0x0-0x575000fffff] state or the [mem 0x0-0x000fffff] end state?
IIUC there are two problems with the
memset_io():1) The
memset_io()writes 0 to all the base and limit registers. For address decoding purposes, the low bits of the base are implicitly clear while the low bits of the limit are implicitly set, so setting the base to zero always makes the windows *enabled*, e.g., [io 0x0000-0x0fff].
Quoted 3 lines
2) The I/O and prefetchable base/limit can't be configured with a single config write, so we have to write them in a specific order to avoid transient enabled windows that could cause conflicts.
Yes. Your explanation is very accurate.
Quoted 30 lines
Quoted 19 lines
Disable the bridge window by executing a sequence of operations borrowed from
pci_disable_bridge_window(), that comply with the PCI specifications.Signed-off-by Nirmal Patel <[email protected]>---
v3->v3: Add more information to commit description.v1->v2: Follow same chain of operation aspci_disable_bridge_window
and update commit log.--- drivers/pci/controller/vmd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)drivers/pci/controller/vmd.c +0
-0drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 529
Quoted 6 lines
drivers/pci/controller/vmd.c +1
-2@@ -526,8 +526,16 @@ static void vmd_domain_reset(struct vmd_dev *vmd) PCI_CLASS_BRIDGE_PCI))continue ; -memset_io (base +PCI_IO_BASE ,0 , -PCI_ROM_ADDRESS1 -PCI_IO_BASE ); +writel (0 , base +PCI_IO_BASE );This is a 32-bit write, so it writes zero to
PCI_IO_BASE,PCI_IO_LIMIT, andPCI_SEC_STATUS. Writing zero toPCI_SEC_STATUSis probably harmless since everything there is RO or RW1C, but is unnecessary and seems a little sloppy.
Quoted 3 lines
Writing zero to
PCI_IO_BASEandPCI_IO_LIMITenables it as [io 0x0000-0x0fff]. I think the code inpci_setup_bridge_io()is more like what you want.
I will make changes according to pci_setup_bridge_io() ASAP.
Thanks.
Quoted 17 lines
drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 537
Quoted 9 lines
drivers/pci/controller/vmd.c +8
-0+/* MMIO Base/Limit */ +writel (0x0000fff0 , base +PCI_MEMORY_BASE ); + +/* Prefetchable MMIO Base/Limit */ +writel (0 , base +PCI_PREF_LIMIT_UPPER32 ); +writel (0x0000fff0 , base +PCI_PREF_MEMORY_BASE ); +writel (0xffffffff , base +PCI_PREF_BASE_UPPER32 ); +writel (0 , base +PCI_IO_BASE_UPPER16 );kkkk
Quoted 7 lines
drivers/pci/controller/vmd.c +1
-0+writeb (0 , base +PCI_CAPABILITY_LIST ); } } }Signature · 1 line
2.31.1