Skip to contentSkip to conversation

PCI: vmd: Fix domain reset operation

PATCH In review Nirmal Patel 3 participants 7 messages May 30, 2023 – Jul 5, 2023

Subsystems (3)
lore
Share & tools
lore

Nirmal Patel

author maintainer
#0 lore raw

PATCH 1/1 PCI: vmd: Fix domain reset operation

During domain reset process we are accidentally enabling the prefetchable memory by writing 0x0 to Prefetchable Memory Base and Prefetchable Memory Limit registers. As a result certain platforms failed to boot up.

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

When clearing Prefetchable Memory Base, Prefetchable Memory Limit and Prefetchable Base Upper 32 bits, the prefetchable memory range becomes 0x0-0x575000fffff. As a result the prefetchable memory is enabled accidentally.

Implementing correct operation by writing a value to Prefetchable Base Memory larger than the value of Prefetchable Memory Limit.

---
 drivers/pci/controller/vmd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +12 -2
index 769eedeb8802..f3eb740e3028 100644
@@ -526,8 +526,18 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				writel(0, base + PCI_IO_BASE);
+				writew(0xFFF0, base + PCI_MEMORY_BASE);
+				writew(0, base + PCI_MEMORY_LIMIT);
+
+				writew(0xFFF1, base + PCI_PREF_MEMORY_BASE);
+				writew(0, base + PCI_PREF_MEMORY_LIMIT);
+
+				writel(0xFFFFFFFF, base + PCI_PREF_BASE_UPPER32);
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+
+				writel(0, base + PCI_IO_BASE_UPPER16);
+				writeb(0, base + PCI_CAPABILITY_LIST);
 			}
 		}
 	}
Signature · 1 line
2.27.0
author maintainer
#1 lore raw
Quoted 39 lines · Nirmal Patel

During domain reset process we are accidentally enabling the prefetchable memory by writing 0x0 to Prefetchable Memory Base and Prefetchable Memory Limit registers. As a result certain platforms failed to boot up.

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

When clearing Prefetchable Memory Base, Prefetchable Memory Limit and Prefetchable Base Upper 32 bits, the prefetchable memory range becomes 0x0-0x575000fffff. As a result the prefetchable memory is enabled accidentally.

Implementing correct operation by writing a value to Prefetchable Base Memory larger than the value of Prefetchable Memory Limit.

---
 drivers/pci/controller/vmd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +5 -2
index 769eedeb8802..f3eb740e3028 100644
@@ -526,8 +526,18 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				writel(0, base + PCI_IO_BASE);
+				writew(0xFFF0, base + PCI_MEMORY_BASE);
+				writew(0, base + PCI_MEMORY_LIMIT);
+
+				writew(0xFFF1, base + PCI_PREF_MEMORY_BASE);

drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 543

Quoted 10 lines
drivers/pci/controller/vmd.c +7 -0
+				writew(0, base + PCI_PREF_MEMORY_LIMIT);
+
+				writel(0xFFFFFFFF, base + PCI_PREF_BASE_UPPER32);
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+
+				writel(0, base + PCI_IO_BASE_UPPER16);
+				writeb(0, base + PCI_CAPABILITY_LIST);
 			}
 		}
 	}

Gentle reminder!
Thanks.

#2 lore raw
Quoted 9 lines · Nirmal
Quoted 7 lines · Nirmal Patel

During domain reset process we are accidentally enabling the prefetchable memory by writing 0x0 to Prefetchable Memory Base and Prefetchable Memory Limit registers. As a result certain platforms failed to boot up.

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

Quoted 7 lines
Quoted 7 lines

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

When clearing Prefetchable Memory Base, Prefetchable Memory Limit and Prefetchable Base Upper 32 bits, the prefetchable memory range becomes 0x0-0x575000fffff. As a result the

I don't get why the top 32 bits aren't cleared. The patch is fine to me.

Lorenzo

Quoted 39 lines
Quoted 25 lines

prefetchable memory is enabled accidentally.

Implementing correct operation by writing a value to Prefetchable Base Memory larger than the value of Prefetchable Memory Limit.

---
 drivers/pci/controller/vmd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +5 -2
index 769eedeb8802..f3eb740e3028 100644
@@ -526,8 +526,18 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				writel(0, base + PCI_IO_BASE);
+				writew(0xFFF0, base + PCI_MEMORY_BASE);
+				writew(0, base + PCI_MEMORY_LIMIT);
+
+				writew(0xFFF1, base + PCI_PREF_MEMORY_BASE);

drivers/pci/controller/vmd.c static void vmd_domain_reset(struct vmd_dev *vmd) · line 543

Quoted 10 lines
drivers/pci/controller/vmd.c +7 -0
+				writew(0, base + PCI_PREF_MEMORY_LIMIT);
+
+				writel(0xFFFFFFFF, base + PCI_PREF_BASE_UPPER32);
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+
+				writel(0, base + PCI_IO_BASE_UPPER16);
+				writeb(0, base + PCI_CAPABILITY_LIST);
 			}
 		}
 	}

Gentle reminder!
Thanks.

#3 lore raw
In reply to Nirmal Patel
Quoted 11 lines · Nirmal Patel

During domain reset process we are accidentally enabling the prefetchable memory by writing 0x0 to Prefetchable Memory Base and Prefetchable Memory Limit registers. As a result certain platforms failed to boot up.

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

Quoted 7 lines

When clearing Prefetchable Memory Base, Prefetchable Memory Limit and Prefetchable Base Upper 32 bits, the prefetchable memory range becomes 0x0-0x575000fffff. As a result the prefetchable memory is enabled accidentally.

Implementing correct operation by writing a value to Prefetchable Base Memory larger than the value of Prefetchable Memory Limit.

Can you make the subject line more descriptive than "Fix X"?

There are a zillion ways X might be broken, so it's useful to have a hint about which one it is.

Actually, vmd_domain_reset() is sort of weirdly named, because I don't see any *reset* there. It looks like it's actually disabling all bridge windows (not just the prefetchable one, as the log suggests).

The fact that this makes a difference suggests that the bridges had PCI_COMMAND_MEMORY set, which sounds like something you may not have wanted.

Quoted 33 lines
---
 drivers/pci/controller/vmd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +12 -2
index 769eedeb8802..f3eb740e3028 100644
@@ -526,8 +526,18 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				writel(0, base + PCI_IO_BASE);
+				writew(0xFFF0, base + PCI_MEMORY_BASE);
+				writew(0, base + PCI_MEMORY_LIMIT);
+
+				writew(0xFFF1, base + PCI_PREF_MEMORY_BASE);
+				writew(0, base + PCI_PREF_MEMORY_LIMIT);
+
+				writel(0xFFFFFFFF, base + PCI_PREF_BASE_UPPER32);
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+
+				writel(0, base + PCI_IO_BASE_UPPER16);
+				writeb(0, base + PCI_CAPABILITY_LIST);
 			}
 		}
 	}
Signature · 1 line
2.27.0
#4 lore raw
In reply to Nirmal Patel
Quoted 11 lines · Nirmal Patel

During domain reset process we are accidentally enabling the prefetchable memory by writing 0x0 to Prefetchable Memory Base and Prefetchable Memory Limit registers. As a result certain platforms failed to boot up.

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

Quoted 7 lines

When clearing Prefetchable Memory Base, Prefetchable Memory Limit and Prefetchable Base Upper 32 bits, the prefetchable memory range becomes 0x0-0x575000fffff. As a result the prefetchable memory is enabled accidentally.

Implementing correct operation by writing a value to Prefetchable Base Memory larger than the value of Prefetchable Memory Limit.

Oh, I forgot: better to use imperative mood here instead of present participle ("-ing" form), e.g., "Write ... so that ..."

This should probably use the same form as pci_disable_bridge_window(), since I think it's doing the same thing.

Quoted 33 lines
---
 drivers/pci/controller/vmd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +12 -2
index 769eedeb8802..f3eb740e3028 100644
@@ -526,8 +526,18 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				writel(0, base + PCI_IO_BASE);
+				writew(0xFFF0, base + PCI_MEMORY_BASE);
+				writew(0, base + PCI_MEMORY_LIMIT);
+
+				writew(0xFFF1, base + PCI_PREF_MEMORY_BASE);
+				writew(0, base + PCI_PREF_MEMORY_LIMIT);
+
+				writel(0xFFFFFFFF, base + PCI_PREF_BASE_UPPER32);
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+
+				writel(0, base + PCI_IO_BASE_UPPER16);
+				writeb(0, base + PCI_CAPABILITY_LIST);
 			}
 		}
 	}
Signature · 1 line
2.27.0
author maintainer
#5 lore raw
Quoted 17 lines · Bjorn Helgaas
Quoted 16 lines · Nirmal Patel

During domain reset process we are accidentally enabling the prefetchable memory by writing 0x0 to Prefetchable Memory Base and Prefetchable Memory Limit registers. As a result certain platforms failed to boot up.

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

When clearing Prefetchable Memory Base, Prefetchable Memory Limit and Prefetchable Base Upper 32 bits, the prefetchable memory range becomes 0x0-0x575000fffff. As a result the prefetchable memory is enabled accidentally.

Quoted 7 lines
Quoted 2 lines

Implementing correct operation by writing a value to Prefetchable Base Memory larger than the value of Prefetchable Memory Limit.

Oh, I forgot: better to use imperative mood here instead of present participle ("-ing" form), e.g., "Write ... so that ..."

This should probably use the same form as pci_disable_bridge_window(), since I think it's doing the same thing.

I will make the adjustment.

Quoted 34 lines
Quoted 34 lines
---
 drivers/pci/controller/vmd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +12 -2
index 769eedeb8802..f3eb740e3028 100644
@@ -526,8 +526,18 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				writel(0, base + PCI_IO_BASE);
+				writew(0xFFF0, base + PCI_MEMORY_BASE);
+				writew(0, base + PCI_MEMORY_LIMIT);
+
+				writew(0xFFF1, base + PCI_PREF_MEMORY_BASE);
+				writew(0, base + PCI_PREF_MEMORY_LIMIT);
+
+				writel(0xFFFFFFFF, base + PCI_PREF_BASE_UPPER32);
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+
+				writel(0, base + PCI_IO_BASE_UPPER16);
+				writeb(0, base + PCI_CAPABILITY_LIST);
 			}
 		}
 	}
Signature · 1 line
2.27.0
author maintainer
#6 lore raw
In reply to Bjorn Helgaas
Quoted 17 lines · Bjorn Helgaas
Quoted 16 lines · Nirmal Patel

During domain reset process we are accidentally enabling the prefetchable memory by writing 0x0 to Prefetchable Memory Base and Prefetchable Memory Limit registers. As a result certain platforms failed to boot up.

Here is the quote from section 7.5.1.3.9 of PCI Express Base 6.0 spec:

The Prefetchable Memory Limit register must be programmed to a smaller value than the Prefetchable Memory Base register if there is no prefetchable memory on the secondary side of the bridge.

When clearing Prefetchable Memory Base, Prefetchable Memory Limit and Prefetchable Base Upper 32 bits, the prefetchable memory range becomes 0x0-0x575000fffff. As a result the prefetchable memory is enabled accidentally.

Quoted 7 lines
Quoted 2 lines

Implementing correct operation by writing a value to Prefetchable Base Memory larger than the value of Prefetchable Memory Limit.

Oh, I forgot: better to use imperative mood here instead of present participle ("-ing" form), e.g., "Write ... so that ..."

This should probably use the same form as pci_disable_bridge_window(), since I think it's doing the same thing.

will do that. Thanks.

Quoted 34 lines
Quoted 34 lines
---
 drivers/pci/controller/vmd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/pci/controller/vmd.c +12 -2
index 769eedeb8802..f3eb740e3028 100644
@@ -526,8 +526,18 @@ static void vmd_domain_reset(struct vmd_dev *vmd)
 				     PCI_CLASS_BRIDGE_PCI))
 					continue;
 
-				memset_io(base + PCI_IO_BASE, 0,
-					  PCI_ROM_ADDRESS1 - PCI_IO_BASE);
+				writel(0, base + PCI_IO_BASE);
+				writew(0xFFF0, base + PCI_MEMORY_BASE);
+				writew(0, base + PCI_MEMORY_LIMIT);
+
+				writew(0xFFF1, base + PCI_PREF_MEMORY_BASE);
+				writew(0, base + PCI_PREF_MEMORY_LIMIT);
+
+				writel(0xFFFFFFFF, base + PCI_PREF_BASE_UPPER32);
+				writel(0, base + PCI_PREF_LIMIT_UPPER32);
+
+				writel(0, base + PCI_IO_BASE_UPPER16);
+				writeb(0, base + PCI_CAPABILITY_LIST);
 			}
 		}
 	}
Signature · 1 line
2.27.0