Skip to contentSkip to conversation

PCI: vmd: Disable bridge window for domain reset

PATCH Superseded Nirmal Patel 2 participants 5 messages Aug 9, 2023 – Aug 10, 2023

Subsystems (3)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Nirmal Patel

author maintainer
#0 lore raw

PATCH v4 1/1 PCI: vmd: Disable bridge window for domain reset

During domain reset process vmd_domain_reset() clears PCI configuration space of VMD root ports. But certain platform has observed following errors and failed to boot.
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: Invalidation Time-out Error (ITE) cleared

The root cause is that memset_io() clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This seems to be enabling prefetchable memory if the device disabled prefetchable memory originally.

Here is an example (before memset_io()):

PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
...

So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. When memset_io() clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled and incorrect.

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

This is believed to be the reason for the failure and in addition the sequence of operation in vmd_domain_reset() is not following the PCIe specs.

Disable the bridge window by executing a sequence of operations borrowed from pci_disable_bridge_window() and pci_setup_bridge_io(), that comply with the PCI specifications.

---
v3->v4: Following same operation as pci_setup_bridge_io. v2->v3: Add more information to commit description. v1->v2: Follow same chain of operation as pci_disable_bridge_window
and update commit log.

---
 drivers/pci/controller/vmd.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +15 -2
index 769eedeb8802..ae5b4c1704e4 100644
@@ -526,8 +526,21 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				/* Temporarily disable the I/O range before updating PCI_IO_BASE */
+				writel(0x0000ffff, base + PCI_IO_BASE_UPPER16);
+				/* Update lower 16 bits of I/O base/limit */
+				writew(0x00f0, base + PCI_IO_BASE);
+				/* Update upper 16 bits of I/O base/limit */
+				writel(0, base + PCI_IO_BASE_UPPER16);
+
+				/* MMIO Base/Limit */
+				writel(0x0000fff0, base + PCI_MEMORY_BASE);
+
+				/* Prefetchable MMIO Base/Limit */
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+				writel(0x0000fff0, base + PCI_PREF_MEMORY_BASE);
+				writel(0xffffffff, base + PCI_PREF_BASE_UPPER32);
+				writeb(0, base + PCI_CAPABILITY_LIST);
 			}
 		}
 	}
Signature · 1 line
2.31.1
#1 lore raw
Quoted 24 lines · Nirmal Patel

During domain reset process vmd_domain_reset() clears PCI configuration space of VMD root ports. But certain platform has observed following errors and failed to boot.
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: Invalidation Time-out Error (ITE) cleared

The root cause is that memset_io() clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This seems to be enabling prefetchable memory if the device disabled prefetchable memory originally.

Here is an example (before memset_io()):

PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
...

Quoted 4 lines

So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. When memset_io() clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled and incorrect.

It's not clear to me how this window config causes the VT-d errors. But empirically it seems to be related, and maybe that's enough.

Quoted 39 lines

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

This is believed to be the reason for the failure and in addition the sequence of operation in vmd_domain_reset() is not following the PCIe specs.

Disable the bridge window by executing a sequence of operations borrowed from pci_disable_bridge_window() and pci_setup_bridge_io(), that comply with the PCI specifications.

---
v3->v4: Following same operation as pci_setup_bridge_io. v2->v3: Add more information to commit description. v1->v2: Follow same chain of operation as pci_disable_bridge_window
and update commit log.

---
 drivers/pci/controller/vmd.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +5 -2
index 769eedeb8802..ae5b4c1704e4 100644
@@ -526,8 +526,21 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				/* Temporarily disable the I/O range before updating PCI_IO_BASE */
+				writel(0x0000ffff, base + PCI_IO_BASE_UPPER16);
+				/* Update lower 16 bits of I/O base/limit */
+				writew(0x00f0, base + PCI_IO_BASE);
+				/* Update upper 16 bits of I/O base/limit */

drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 543

Quoted 10 lines
drivers/pci/controller/vmd.c +10 -0
+				writel(0, base + PCI_IO_BASE_UPPER16);
+
+				/* MMIO Base/Limit */
+				writel(0x0000fff0, base + PCI_MEMORY_BASE);
+
+				/* Prefetchable MMIO Base/Limit */
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+				writel(0x0000fff0, base + PCI_PREF_MEMORY_BASE);
+				writel(0xffffffff, base + PCI_PREF_BASE_UPPER32);
+				writeb(0, base + PCI_CAPABILITY_LIST);

What's the purpose of this PCI_CAPABILITY_LIST write? I guess you don't want to find PM, MSI, MSI-X, PCIe, etc. capabilities?

It's been there since the v1 patch, but the commit log only mentions disabling bridge windows.

Bjorn

author maintainer
#2 lore raw
Quoted 84 lines · Bjorn Helgaas
Quoted 24 lines · Nirmal Patel

During domain reset process vmd_domain_reset() clears PCI configuration space of VMD root ports. But certain platform has observed following errors and failed to boot.
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: Invalidation Time-out Error (ITE) cleared

The root cause is that memset_io() clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This seems to be enabling prefetchable memory if the device disabled prefetchable memory originally.

Here is an example (before memset_io()):

PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
...

Quoted 4 lines

So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. When memset_io() clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled and incorrect.

It's not clear to me how this window config causes the VT-d errors. But empirically it seems to be related, and maybe that's enough.

Quoted 39 lines

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

This is believed to be the reason for the failure and in addition the sequence of operation in vmd_domain_reset() is not following the PCIe specs.

Disable the bridge window by executing a sequence of operations borrowed from pci_disable_bridge_window() and pci_setup_bridge_io(), that comply with the PCI specifications.

---
v3->v4: Following same operation as pci_setup_bridge_io. v2->v3: Add more information to commit description. v1->v2: Follow same chain of operation as pci_disable_bridge_window
and update commit log.

---
 drivers/pci/controller/vmd.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +5 -2
index 769eedeb8802..ae5b4c1704e4 100644
@@ -526,8 +526,21 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				/* Temporarily disable the I/O range before updating PCI_IO_BASE */
+				writel(0x0000ffff, base + PCI_IO_BASE_UPPER16);
+				/* Update lower 16 bits of I/O base/limit */
+				writew(0x00f0, base + PCI_IO_BASE);
+				/* Update upper 16 bits of I/O base/limit */

drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 543

Quoted 10 lines
drivers/pci/controller/vmd.c +10 -0
+				writel(0, base + PCI_IO_BASE_UPPER16);
+
+				/* MMIO Base/Limit */
+				writel(0x0000fff0, base + PCI_MEMORY_BASE);
+
+				/* Prefetchable MMIO Base/Limit */
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+				writel(0x0000fff0, base + PCI_PREF_MEMORY_BASE);
+				writel(0xffffffff, base + PCI_PREF_BASE_UPPER32);
+				writeb(0, base + PCI_CAPABILITY_LIST);

What's the purpose of this PCI_CAPABILITY_LIST write? I guess you don't want to find PM, MSI, MSI-X, PCIe, etc. capabilities?

Quoted 4 lines

It's been there since the v1 patch, but the commit log only mentions disabling bridge windows.

Bjorn

I added it since it was part of original memset_io range. However from your previous comment, I checked the lspci output for PCI_CAPABILITY_LIST with and without the change and it doesn't seem to make any difference.

#3 lore raw
Quoted 85 lines · Nirmal
Quoted 84 lines · Bjorn Helgaas
Quoted 24 lines · Nirmal Patel

During domain reset process vmd_domain_reset() clears PCI configuration space of VMD root ports. But certain platform has observed following errors and failed to boot.
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: Invalidation Time-out Error (ITE) cleared

The root cause is that memset_io() clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This seems to be enabling prefetchable memory if the device disabled prefetchable memory originally.

Here is an example (before memset_io()):

PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
...

Quoted 4 lines

So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. When memset_io() clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled and incorrect.

It's not clear to me how this window config causes the VT-d errors. But empirically it seems to be related, and maybe that's enough.

Quoted 39 lines

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

This is believed to be the reason for the failure and in addition the sequence of operation in vmd_domain_reset() is not following the PCIe specs.

Disable the bridge window by executing a sequence of operations borrowed from pci_disable_bridge_window() and pci_setup_bridge_io(), that comply with the PCI specifications.

---
v3->v4: Following same operation as pci_setup_bridge_io. v2->v3: Add more information to commit description. v1->v2: Follow same chain of operation as pci_disable_bridge_window
and update commit log.

---
 drivers/pci/controller/vmd.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +5 -2
index 769eedeb8802..ae5b4c1704e4 100644
@@ -526,8 +526,21 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				/* Temporarily disable the I/O range before updating PCI_IO_BASE */
+				writel(0x0000ffff, base + PCI_IO_BASE_UPPER16);
+				/* Update lower 16 bits of I/O base/limit */
+				writew(0x00f0, base + PCI_IO_BASE);
+				/* Update upper 16 bits of I/O base/limit */

drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 543

Quoted 10 lines
drivers/pci/controller/vmd.c +10 -0
+				writel(0, base + PCI_IO_BASE_UPPER16);
+
+				/* MMIO Base/Limit */
+				writel(0x0000fff0, base + PCI_MEMORY_BASE);
+
+				/* Prefetchable MMIO Base/Limit */
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+				writel(0x0000fff0, base + PCI_PREF_MEMORY_BASE);
+				writel(0xffffffff, base + PCI_PREF_BASE_UPPER32);
+				writeb(0, base + PCI_CAPABILITY_LIST);

What's the purpose of this PCI_CAPABILITY_LIST write? I guess you don't want to find PM, MSI, MSI-X, PCIe, etc. capabilities?

Quoted 7 lines
Quoted 2 lines

It's been there since the v1 patch, but the commit log only mentions disabling bridge windows.

I added it since it was part of original memset_io range. However from your previous comment, I checked the lspci output for PCI_CAPABILITY_LIST with and without the change and it doesn't seem to make any difference.

Ah, I see. My guess is that was a mistake in the original memset_io() because I don't see a reason to clear PCI_CAPABILITY_LIST.

PCI_CAPABILITY_LIST is HwInit, so should be read-only in terms of config accesses, and if lspci sees the same capability list before and after writing a zero to it, it sounds like it *is* read-only.

Bjorn

author maintainer
#4 lore raw
Quoted 92 lines · Bjorn Helgaas
Quoted 85 lines · Nirmal
Quoted 84 lines · Bjorn Helgaas
Quoted 24 lines · Nirmal Patel

During domain reset process vmd_domain_reset() clears PCI configuration space of VMD root ports. But certain platform has observed following errors and failed to boot.
...
DMAR: VT-d detected Invalidation Queue Error: Reason f
DMAR: VT-d detected Invalidation Time-out Error: SID ffff
DMAR: VT-d detected Invalidation Completion Error: SID ffff
DMAR: QI HEAD: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: QI PRIOR: UNKNOWN qw0 = 0x0, qw1 = 0x0
DMAR: Invalidation Time-out Error (ITE) cleared

The root cause is that memset_io() clears prefetchable memory base/limit registers and prefetchable base/limit 32 bits registers sequentially. This seems to be enabling prefetchable memory if the device disabled prefetchable memory originally.

Here is an example (before memset_io()):

PCI configuration space for 10000:00:00.0:
86 80 30 20 06 00 10 00 04 00 04 06 00 00 01 00
00 00 00 00 00 00 00 00 00 01 01 00 00 00 00 20
00 00 00 00 01 00 01 00 ff ff ff ff 75 05 00 00
...

Quoted 4 lines

So, prefetchable memory is ffffffff00000000-575000fffff, which is disabled. When memset_io() clears prefetchable base 32 bits register, the prefetchable memory becomes 0000000000000000-575000fffff, which is enabled and incorrect.

It's not clear to me how this window config causes the VT-d errors. But empirically it seems to be related, and maybe that's enough.

Quoted 39 lines

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

This is believed to be the reason for the failure and in addition the sequence of operation in vmd_domain_reset() is not following the PCIe specs.

Disable the bridge window by executing a sequence of operations borrowed from pci_disable_bridge_window() and pci_setup_bridge_io(), that comply with the PCI specifications.

---
v3->v4: Following same operation as pci_setup_bridge_io. v2->v3: Add more information to commit description. v1->v2: Follow same chain of operation as pci_disable_bridge_window
and update commit log.

---
 drivers/pci/controller/vmd.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +5 -2
index 769eedeb8802..ae5b4c1704e4 100644
@@ -526,8 +526,21 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				/* Temporarily disable the I/O range before updating PCI_IO_BASE */
+				writel(0x0000ffff, base + PCI_IO_BASE_UPPER16);
+				/* Update lower 16 bits of I/O base/limit */
+				writew(0x00f0, base + PCI_IO_BASE);
+				/* Update upper 16 bits of I/O base/limit */

drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 543

Quoted 10 lines
drivers/pci/controller/vmd.c +10 -0
+				writel(0, base + PCI_IO_BASE_UPPER16);
+
+				/* MMIO Base/Limit */
+				writel(0x0000fff0, base + PCI_MEMORY_BASE);
+
+				/* Prefetchable MMIO Base/Limit */
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+				writel(0x0000fff0, base + PCI_PREF_MEMORY_BASE);
+				writel(0xffffffff, base + PCI_PREF_BASE_UPPER32);
+				writeb(0, base + PCI_CAPABILITY_LIST);

What's the purpose of this PCI_CAPABILITY_LIST write? I guess you don't want to find PM, MSI, MSI-X, PCIe, etc. capabilities?

Quoted 6 lines
Quoted 2 lines

It's been there since the v1 patch, but the commit log only mentions disabling bridge windows.

I added it since it was part of original memset_io range. However from your previous comment, I checked the lspci output for PCI_CAPABILITY_LIST with and without the change and it doesn't seem to make any difference.

Quoted 8 lines

Ah, I see. My guess is that was a mistake in the original memset_io() because I don't see a reason to clear PCI_CAPABILITY_LIST.

PCI_CAPABILITY_LIST is HwInit, so should be read-only in terms of config accesses, and if lspci sees the same capability list before and after writing a zero to it, it sounds like it *is* read-only.

Bjorn

Yes, I will remove it. Thanks.