Skip to contentSkip to conversation

PCI: Fix extended config space regression

PATCH Applied Bjorn Helgaas 9 participants 25 messages Jan 10, 2023 – Dec 6, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover PCI: Fix extended config space regression

From: Bjorn Helgaas <[email protected]>

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.

This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.

This happened because mmconfig-shared.c checks to see that ECAM space is reserved in E820 or ACPI motherboard resources. If it's not, it assumes ECAM doesn't work. 07eab0901ede removed some E820 entries, so it looked like ECAM was no longer reserved, so we stopped using it.

The machines where this broke don't reserve the ECAM in ACPI PNP0C02 devices (which seems to be what the PCI Firmware spec requires), but they do mention it in the EFI memory map as EfiMemoryMappedIO.

Bjorn Helgaas (2):
x86/pci: Simplify is_mmconf_reserved() messages x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

 arch/x86/pci/mmconfig-shared.c | 44 +++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 6 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/2 x86/pci: Simplify is_mmconf_reserved() messages

From: Bjorn Helgaas <[email protected]>

is_mmconf_reserved() takes a "with_e820" parameter that only determines the message logged if it finds the MMCONFIG region is reserved. Pass the message directly, which will simplify a future patch that adds a new way of looking for that reservation. No functional change intended.

---
 arch/x86/pci/mmconfig-shared.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
arch/x86/pci/mmconfig-shared.c +7 -6
index 758cbfe55daa..cd16bef5f2d9 100644
@@ -446,13 +446,12 @@ typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type);
 
 static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,
 				     struct pci_mmcfg_region *cfg,
-				     struct device *dev, int with_e820)
+				     struct device *dev, char *method)
 {
 	u64 addr = cfg->res.start;
 	u64 size = resource_size(&cfg->res);
 	u64 old_size = size;
 	int num_buses;
-	char *method = with_e820 ? "E820" : "ACPI motherboard resources";
 
 	while (!is_reserved(addr, addr + size, E820_TYPE_RESERVED)) {
 		size >>= 1;
@@ -464,10 +463,10 @@ static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,
 		return false;
 
 	if (dev)
-		dev_info(dev, "MMCONFIG at %pR reserved in %s\n",
+		dev_info(dev, "MMCONFIG at %pR reserved as %s\n",
 			 &cfg->res, method);
 	else
-		pr_info(PREFIX "MMCONFIG at %pR reserved in %s\n",
+		pr_info(PREFIX "MMCONFIG at %pR reserved as %s\n",
 		       &cfg->res, method);
 
 	if (old_size != size) {
@@ -500,7 +499,8 @@ static bool __ref
 pci_mmcfg_check_reserved(struct device *dev, struct pci_mmcfg_region *cfg, int early)
 {
 	if (!early && !acpi_disabled) {
-		if (is_mmconf_reserved(is_acpi_reserved, cfg, dev, 0))
+		if (is_mmconf_reserved(is_acpi_reserved, cfg, dev,
+				       "ACPI motherboard resource"))
 			return true;
 
 		if (dev)
@@ -527,7 +527,8 @@ pci_mmcfg_check_reserved(struct device *dev, struct pci_mmcfg_region *cfg, int e
 	/* Don't try to do this check unless configuration
 	   type 1 is available. how about type 2 ?*/
 	if (raw_pci_ops)
-		return is_mmconf_reserved(e820__mapped_all, cfg, dev, 1);
+		return is_mmconf_reserved(e820__mapped_all, cfg, dev,
+					  "E820 entry");
 
 	return false;
 }
Signature · 1 line
2.25.1
Reviewed-by on v1 1/2
#2 lore raw

RE: [PATCH 1/2] x86/pci: Simplify is_mmconf_reserved() messages

Quoted 17 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

is_mmconf_reserved() takes a "with_e820" parameter that only determines the message logged if it finds the MMCONFIG region is reserved. Pass the message directly, which will simplify a future patch that adds a new way of looking for that reservation. No functional change intended.

---
 arch/x86/pci/mmconfig-shared.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
arch/x86/pci/mmconfig-shared.c +0 -0
index 758cbfe55daa..cd16bef5f2d9 100644

arch/x86/pci/mmconfig-shared.c typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type); · line 449

Quoted 5 lines
arch/x86/pci/mmconfig-shared.c +1 -1
@@ -446,13 +446,12 @@ typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type);
 
 static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,
 				     struct pci_mmcfg_region *cfg,
-				     struct device *dev, int with_e820)
+				     struct device *dev, char *method)

@method could be 'const char *', but either way:

#3 lore raw

Re: [PATCH 1/2] x86/pci: Simplify is_mmconf_reserved() messages

Quoted 19 lines · Dan Williams
Quoted 18 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

is_mmconf_reserved() takes a "with_e820" parameter that only determines the message logged if it finds the MMCONFIG region is reserved. Pass the message directly, which will simplify a future patch that adds a new way of looking for that reservation. No functional change intended.

---
 arch/x86/pci/mmconfig-shared.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
arch/x86/pci/mmconfig-shared.c +0 -0
index 758cbfe55daa..cd16bef5f2d9 100644
@@ -446,13 +446,12 @@ typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type);
 
Quoted 6 lines

arch/x86/pci/mmconfig-shared.c typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type); · line 449

Quoted 4 lines
arch/x86/pci/mmconfig-shared.c +1 -1
 static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,
 				     struct pci_mmcfg_region *cfg,
-				     struct device *dev, int with_e820)
+				     struct device *dev, char *method)

@method could be 'const char *', but either way:

Of course, thanks! Folded in.

#4 lore raw

PATCH 2/2 x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.

Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.

Reported by Kan Liang, Tony Luck, and Giovanni Cabiddu.

Fixes 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")
---
 arch/x86/pci/mmconfig-shared.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/pci/mmconfig-shared.c +31 -0
index cd16bef5f2d9..da4b6e8e9df0 100644
@@ -12,6 +12,7 @@
  */
 
 #include <linux/acpi.h>
+#include <linux/efi.h>
 #include <linux/pci.h>
 #include <linux/init.h>
 #include <linux/bitmap.h>
@@ -442,6 +443,32 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)
 	return mcfg_res.flags;
 }
 
+static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)
+{
+#ifdef CONFIG_EFI
+	efi_memory_desc_t *md;
+	u64 size, mmio_start, mmio_end;
+
+	for_each_efi_memory_desc(md) {
+		if (md->type == EFI_MEMORY_MAPPED_IO) {
+			size = md->num_pages << EFI_PAGE_SHIFT;
+			mmio_start = md->phys_addr;
+			mmio_end = mmio_start + size;
+
+			/*
+			 * N.B. Caller supplies (start, start + size),
+			 * so to match, mmio_end is the first address
+			 * *past* the EFI_MEMORY_MAPPED_IO area.
+			 */
+			if (mmio_start <= start && end <= mmio_end)
+				return true;
+		}
+	}
+#endif
+
+	return false;
+}
+
 typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type);
 
 static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,
@@ -513,6 +540,10 @@ pci_mmcfg_check_reserved(struct device *dev, struct pci_mmcfg_region *cfg, int e
 			       "MMCONFIG at %pR not reserved in "
 			       "ACPI motherboard resources\n",
 			       &cfg->res);
+
+		if (is_mmconf_reserved(is_efi_mmio, cfg, dev,
+				       "EfiMemoryMappedIO"))
+			return true;
 	}
 
 	/*
Signature · 1 line
2.25.1
Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

Quoted 19 lines · CET Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.

Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.

Quoted 8 lines

Reported by Kan Liang, Tony Luck, and Giovanni Cabiddu.

Fixes 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")
Quoted 61 lines
---
 arch/x86/pci/mmconfig-shared.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/pci/mmconfig-shared.c +31 -0
index cd16bef5f2d9..da4b6e8e9df0 100644
@@ -12,6 +12,7 @@
  */
 
 #include <linux/acpi.h>
+#include <linux/efi.h>
 #include <linux/pci.h>
 #include <linux/init.h>
 #include <linux/bitmap.h>
@@ -442,6 +443,32 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)
 	return mcfg_res.flags;
 }
 
+static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)
+{
+#ifdef CONFIG_EFI
+	efi_memory_desc_t *md;
+	u64 size, mmio_start, mmio_end;
+
+	for_each_efi_memory_desc(md) {
+		if (md->type == EFI_MEMORY_MAPPED_IO) {
+			size = md->num_pages << EFI_PAGE_SHIFT;
+			mmio_start = md->phys_addr;
+			mmio_end = mmio_start + size;
+
+			/*
+			 * N.B. Caller supplies (start, start + size),
+			 * so to match, mmio_end is the first address
+			 * *past* the EFI_MEMORY_MAPPED_IO area.
+			 */
+			if (mmio_start <= start && end <= mmio_end)
+				return true;
+		}
+	}
+#endif
+
+	return false;
+}
+
 typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type);
 
 static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,
@@ -513,6 +540,10 @@ pci_mmcfg_check_reserved(struct device *dev, struct pci_mmcfg_region *cfg, int e
 			       "MMCONFIG at %pR not reserved in "
 			       "ACPI motherboard resources\n",
 			       &cfg->res);
+
+		if (is_mmconf_reserved(is_efi_mmio, cfg, dev,
+				       "EfiMemoryMappedIO"))
+			return true;
 	}
 
 	/*
Reviewed-by on v1 2/2
#6 lore raw

RE: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

In reply to Bjorn Helgaas
Quoted 59 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.

Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.

Reported by Kan Liang, Tony Luck, and Giovanni Cabiddu.

Fixes 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")
---
 arch/x86/pci/mmconfig-shared.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/pci/mmconfig-shared.c +13 -0
index cd16bef5f2d9..da4b6e8e9df0 100644
@@ -12,6 +12,7 @@
  */
 
 #include <linux/acpi.h>
+#include <linux/efi.h>
 #include <linux/pci.h>
 #include <linux/init.h>
 #include <linux/bitmap.h>
@@ -442,6 +443,32 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)
 	return mcfg_res.flags;
 }
 
+static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)
+{
+#ifdef CONFIG_EFI
+	efi_memory_desc_t *md;
+	u64 size, mmio_start, mmio_end;
+
+	for_each_efi_memory_desc(md) {
+		if (md->type == EFI_MEMORY_MAPPED_IO) {
+			size = md->num_pages << EFI_PAGE_SHIFT;
+			mmio_start = md->phys_addr;
+			mmio_end = mmio_start + size;
+

arch/x86/pci/mmconfig-shared.c static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used) · line 467

Quoted 10 lines
arch/x86/pci/mmconfig-shared.c +10 -0
+			/*
+			 * N.B. Caller supplies (start, start + size),
+			 * so to match, mmio_end is the first address
+			 * *past* the EFI_MEMORY_MAPPED_IO area.
+			 */
+			if (mmio_start <= start && end <= mmio_end)
+				return true;
+		}
+	}
+#endif

Perhaps the following trick (compile tested), but either way:

arch/x86/pci/mmconfig-shared.c +0 -2
index da4b6e8e9df0..ae95d1b073c6 100644
@@ -445,7 +445,6 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)
 
 static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)
 {
-#ifdef CONFIG_EFI
 	efi_memory_desc_t *md;
 	u64 size, mmio_start, mmio_end;
 
@@ -464,7 +463,6 @@ static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)
 				return true;
 		}
 	}
-#endif
 
 	return false;
 }
include/linux/efi.h +4 -0
index 4b27519143f5..3ab0c255b791 100644
@@ -790,8 +790,12 @@ extern int efi_memattr_apply_permissions(struct mm_struct *mm,
  *
  * Once the loop finishes @md must not be accessed.
  */
+#ifdef CONFIG_EFI
 #define for_each_efi_memory_desc(md) \
 	for_each_efi_memory_desc_in_map(&efi.memmap, md)
+#else
+#define for_each_efi_memory_desc(md) for (; 0;)
+#endif
 
 /*
  * Format an EFI memory descriptor's type and attributes to a user-provided
#7 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

Quoted 73 lines · Dan Williams
Quoted 59 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.

Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.

Reported by Kan Liang, Tony Luck, and Giovanni Cabiddu.

Fixes 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")
---
 arch/x86/pci/mmconfig-shared.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/pci/mmconfig-shared.c +13 -0
index cd16bef5f2d9..da4b6e8e9df0 100644
@@ -12,6 +12,7 @@
  */
 
 #include <linux/acpi.h>
+#include <linux/efi.h>
 #include <linux/pci.h>
 #include <linux/init.h>
 #include <linux/bitmap.h>
@@ -442,6 +443,32 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)
 	return mcfg_res.flags;
 }
 
+static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)
+{
+#ifdef CONFIG_EFI
+	efi_memory_desc_t *md;
+	u64 size, mmio_start, mmio_end;
+
+	for_each_efi_memory_desc(md) {
+		if (md->type == EFI_MEMORY_MAPPED_IO) {
+			size = md->num_pages << EFI_PAGE_SHIFT;
+			mmio_start = md->phys_addr;
+			mmio_end = mmio_start + size;
+

arch/x86/pci/mmconfig-shared.c static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used) · line 467

Quoted 10 lines
arch/x86/pci/mmconfig-shared.c +10 -0
+			/*
+			 * N.B. Caller supplies (start, start + size),
+			 * so to match, mmio_end is the first address
+			 * *past* the EFI_MEMORY_MAPPED_IO area.
+			 */
+			if (mmio_start <= start && end <= mmio_end)
+				return true;
+		}
+	}
+#endif

Perhaps the following trick (compile tested), but either way:

Quoted 1 line

That's a great trick, and I wish I'd thought of it. I have some follow-on patches I'm considering for v6.3, so in the interest of streamlining the path of this one to v6.2-rc4, I think I'll wait on this until v6.3.

Quoted 37 lines
arch/x86/pci/mmconfig-shared.c +0 -2
index da4b6e8e9df0..ae95d1b073c6 100644
@@ -445,7 +445,6 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)
 
 static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)
 {
-#ifdef CONFIG_EFI
 	efi_memory_desc_t *md;
 	u64 size, mmio_start, mmio_end;
 
@@ -464,7 +463,6 @@ static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)
 				return true;
 		}
 	}
-#endif
 
 	return false;
 }
include/linux/efi.h +4 -0
index 4b27519143f5..3ab0c255b791 100644
@@ -790,8 +790,12 @@ extern int efi_memattr_apply_permissions(struct mm_struct *mm,
  *
  * Once the loop finishes @md must not be accessed.
  */
+#ifdef CONFIG_EFI
 #define for_each_efi_memory_desc(md) \
 	for_each_efi_memory_desc_in_map(&efi.memmap, md)
+#else
+#define for_each_efi_memory_desc(md) for (; 0;)
+#endif
 
 /*
  * Format an EFI memory descriptor's type and attributes to a user-provided
#8 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

In reply to Bjorn Helgaas

Hello,

Quoted 8 lines · Bjorn Helgaas

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.

Quoted 8 lines

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.

Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.

I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.

efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[...]
[mem 0x7f800000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
[...]
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0: pci_request_selected_regions failed 0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16

After disabling the code causing this (using always-false condition:

		if (size >= 256*1024 && 0) {

) in the chunk:

https://lore.kernel.org/lkml/[email protected]/

the BAR starts at 0x90000000 (not 0x80000000):

efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry

and everything seems to work again.

I've got full system bootup logs from the upstream and worked around, but I'm not sure if this is OK to attach them (the CC list is long).

Also, this is my test machine so I can run some experiments.

best regards,

Signature · 1 line
Tomasz Pala <[email protected]>
#9 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

Quoted 14 lines · Tomasz Pala

I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.

efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[...]
[mem 0x7f800000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
[...]
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0: pci_request_selected_regions failed 0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16

FWIW, as I got no response - there were other people facing the issue as well:

https://forum.proxmox.com/threads/proxmox-8-kernel-6-2-16-4-pve-ixgbe-driver-fails-to-load-due-to-pci-device-probing-failure.131203/

Apparently this might be some hardware quirk, therefore I'm not sure if the internal EfiMemoryMappedIO reservation logic should be reviewed, or some quirk handling to be added, or maybe some CONFIG_option introduced.

Anyone please?

Signature · 1 line
Tomasz Pala <[email protected]>
#10 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

In reply to Tomasz Pala
Quoted 14 lines · Tomasz Pala
Quoted 13 lines · Bjorn Helgaas

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.

Quoted 5 lines
Quoted 3 lines

Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.

I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.

Thanks very much for the report, and sorry for the inconvenience and my delay in looking at it.

Quoted 9 lines

efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[mem 0x7f800000-0xfed1bfff] available for PCI devices
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0: pci_request_selected_regions failed 0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16

Something is wrong with our allocation scheme. Both the MMCONFIG region and the ixgbe BAR 0 are at 0x80000000, which obviously cannot work. Maybe the full dmesg log will have a clue about why we didn't move ixgbe out of the way.

Quoted 15 lines

After disabling the code causing this (using always-false condition:

		if (size >= 256*1024 && 0) {

) in the chunk:

https://lore.kernel.org/lkml/[email protected]/

the BAR starts at 0x90000000 (not 0x80000000):

efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry

Quoted 5 lines

and everything seems to work again.

I've got full system bootup logs from the upstream and worked around, but I'm not sure if this is OK to attach them (the CC list is long).

Would you mind opening a new report at https://bugzilla.kernel.org,
attaching those logs, and responding here with the URL?

I looked at the proxmox thread you mentioned, but sometimes people strip out parts of the log they think are irrelevant, and in this case, the stripped-out parts *are* relevant.

Bjorn

#11 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

Quoted 54 lines · Bjorn Helgaas
Quoted 14 lines · Tomasz Pala
Quoted 13 lines · Bjorn Helgaas

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.

Quoted 5 lines
Quoted 3 lines

Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.

I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.

Thanks very much for the report, and sorry for the inconvenience and my delay in looking at it.

Quoted 9 lines

efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[mem 0x7f800000-0xfed1bfff] available for PCI devices
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0: pci_request_selected_regions failed 0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16

Something is wrong with our allocation scheme. Both the MMCONFIG region and the ixgbe BAR 0 are at 0x80000000, which obviously cannot work. Maybe the full dmesg log will have a clue about why we didn't move ixgbe out of the way.

Quoted 15 lines

After disabling the code causing this (using always-false condition:

		if (size >= 256*1024 && 0) {

) in the chunk:

https://lore.kernel.org/lkml/[email protected]/

the BAR starts at 0x90000000 (not 0x80000000):

efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry

Quoted 8 lines
Quoted 5 lines

and everything seems to work again.

I've got full system bootup logs from the upstream and worked around, but I'm not sure if this is OK to attach them (the CC list is long).

Would you mind opening a new report at https://bugzilla.kernel.org,
attaching those logs, and responding here with the URL?

Thanks for the report and the logs, which are attached at
https://bugzilla.kernel.org/show_bug.cgi?id=218050

I think the problem is that the MMCONFIG region is at
[mem 0x80000000-0x8fffffff], and that is *also* included in one of the host bridge windows reported via _CRS:

PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) pci_bus 0000:00: root bus resource [mem 0x80000000-0xfbffffff window]

I'll try to figure out how to deal with that. In the meantime, would you mind attaching the contents of /proc/iomem to the bugzilla? I think you have to cat it as root to get the actual values included.

Bjorn

#12 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

[+cc Sebastian]

Quoted 76 lines · Bjorn Helgaas
Quoted 59 lines · Bjorn Helgaas
Quoted 18 lines · Tomasz Pala
Quoted 17 lines · Bjorn Helgaas

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.

Quoted 6 lines
Quoted 3 lines

Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.

I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.

Thanks very much for the report, and sorry for the inconvenience and my delay in looking at it.

Quoted 9 lines

efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[mem 0x7f800000-0xfed1bfff] available for PCI devices
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0: pci_request_selected_regions failed 0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16

Something is wrong with our allocation scheme. Both the MMCONFIG region and the ixgbe BAR 0 are at 0x80000000, which obviously cannot work. Maybe the full dmesg log will have a clue about why we didn't move ixgbe out of the way.

Quoted 15 lines

After disabling the code causing this (using always-false condition:

		if (size >= 256*1024 && 0) {

) in the chunk:

https://lore.kernel.org/lkml/[email protected]/

the BAR starts at 0x90000000 (not 0x80000000):

efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry

Quoted 8 lines
Quoted 5 lines

and everything seems to work again.

I've got full system bootup logs from the upstream and worked around, but I'm not sure if this is OK to attach them (the CC list is long).

Would you mind opening a new report at https://bugzilla.kernel.org,
attaching those logs, and responding here with the URL?

Thanks for the report and the logs, which are attached at
https://bugzilla.kernel.org/show_bug.cgi?id=218050

I think the problem is that the MMCONFIG region is at
[mem 0x80000000-0x8fffffff], and that is *also* included in one of the host bridge windows reported via _CRS:

Quoted 6 lines

PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) pci_bus 0000:00: root bus resource [mem 0x80000000-0xfbffffff window]

I'll try to figure out how to deal with that. In the meantime, would you mind attaching the contents of /proc/iomem to the bugzilla? I think you have to cat it as root to get the actual values included.

I attached a debug patch to both bugzilla entries. If you could attach the "acpidump" output and (if practical) boot a kernel with the debug patch and attach the dmesg logs, that would be great.

Bjorn

#13 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

Quoted 9 lines · Bjorn Helgaas
Quoted 9 lines

https://bugzilla.kernel.org/show_bug.cgi?id=218050

I think the problem is that the MMCONFIG region is at
[mem 0x80000000-0x8fffffff], and that is *also* included in one of the host bridge windows reported via _CRS:

PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) pci_bus 0000:00: root bus resource [mem 0x80000000-0xfbffffff window]

Quoted 6 lines
Quoted 2 lines

I'll try to figure out how to deal with that. In the meantime, would you mind attaching the contents of /proc/iomem to the bugzilla? I

I attached a debug patch to both bugzilla entries. If you could attach the "acpidump" output and (if practical) boot a kernel with the debug patch and attach the dmesg logs, that would be great.

I've posted the files. There are signs of buggy BIOS, but I don't expect any firmware update to be released for this hw anymore.

DMI: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.4 11/20/2019

.text .data .bss are not marked as E820_TYPE_RAM!
tboot: non-0 tboot_addr but it is not of type E820_TYPE_RESERVED

DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000df243000-0x00000000df251fff], contact BIOS vendor for fixes
DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000df243000-0x00000000df251fff]

BTW is there a reason for this logging discrepancy?

efi: Remove mem173: MMIO range=[0xe0000000-0xefffffff] (256MB) from e820 map
efi: Not removing mem71: MMIO range=[0xe0000000-0xefffffff] (262144KB) from e820 map

efi: Not removing mem74: MMIO range=[0xff000000-0xffffffff] (16384KB) from e820 map
efi: Remove mem176: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map

This is arch/x86/platform/efi/efi.c:
static void __init efi_remove_e820_mmio(void)

Remove mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluMB) ... size >> 20 Not removing mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluKB) ... size >> 10

Signature · 1 line
Tomasz Pala <[email protected]>
#14 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

Quoted 14 lines · Tomasz Pala
Quoted 12 lines · Bjorn Helgaas
Quoted 11 lines

https://bugzilla.kernel.org/show_bug.cgi?id=218050

I think the problem is that the MMCONFIG region is at
[mem 0x80000000-0x8fffffff], and that is *also* included in one of the host bridge windows reported via _CRS:

PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) pci_bus 0000:00: root bus resource [mem 0x80000000-0xfbffffff window]

I'll try to figure out how to deal with that. In the meantime, would you mind attaching the contents of /proc/iomem to the bugzilla? I

Quoted 6 lines
Quoted 3 lines

I attached a debug patch to both bugzilla entries. If you could attach the "acpidump" output and (if practical) boot a kernel with the debug patch and attach the dmesg logs, that would be great.

I've posted the files. There are signs of buggy BIOS, but I don't expect any firmware update to be released for this hw anymore.

Thank you! A BIOS update is almost never the answer because even if an update exists, we have to assume that most users in the field will never install the update.

I want to look at the BIOS info in case we can learn about something *Linux* is doing wrong. This most likely works fine with Windows, so I assume Linux is doing something wrong or at least differently than Windows.

Quoted 3 lines

DMI: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.4 11/20/2019

.text .data .bss are not marked as E820_TYPE_RAM!

Added by 4eea6aa581ab ("x86, mm: if kernel .text .data .bss are not marked as E820_RAM, complain and fix"). No idea. A shame we didn't include the .text/.data values in the message.

Quoted 1 line

tboot: non-0 tboot_addr but it is not of type E820_TYPE_RESERVED

Added by 316253406959 ("x86, intel_txt: Intel TXT boot support"). No idea about this either.

Quoted 2 lines

DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000df243000-0x00000000df251fff], contact BIOS vendor for fixes
DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000df243000-0x00000000df251fff]

Both related to arch_rmrr_sanity_check(), added by f036c7fa0ab6 ("iommu/vt-d: Check VT-d RMRR region in BIOS is reported as reserved") and f5a68bb0752e ("iommu/vt-d: Mark firmware tainted if RMRR fails sanity check").

No idea about this one either. The VT-d spec (r1.3, sec 8.4) says "BIOS must report the RMRR reported memory addresses as reserved in the system memory map returned through methods such as INT15, EFI GetMemoryMap etc."

arch_rmrr_sanity_check() only looks at your e820 map, which only has this:

BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
BIOS-e820: [mem 0x0000000000100000-0x00000000d1f36fff] usable

I think Linux basically converts the info from EFI GetMemoryMap to an e820 format; I think booting with "efi=debug" would show more details of this.

Anyway, this is all a tangent.

Quoted 5 lines

BTW is there a reason for this logging discrepancy?

efi: Remove mem173: MMIO range=[0xe0000000-0xefffffff] (256MB) from e820 map
efi: Not removing mem71: MMIO range=[0xe0000000-0xefffffff] (262144KB) from e820 map

Quoted 8 lines

efi: Not removing mem74: MMIO range=[0xff000000-0xffffffff] (16384KB) from e820 map
efi: Remove mem176: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map

This is arch/x86/platform/efi/efi.c:
static void __init efi_remove_e820_mmio(void)

Remove mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluMB) ... size >> 20 Not removing mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluKB) ... size >> 10

You mean the MB vs KB difference? That's my fault. I guess I used KB for the "Not removing" message because those are smaller (< 256KB) so the size in MB wouldn't be useful there. We could use KB for both, but I guess I used MB for the "Remove" case because it's a little easier to read and I expected "Not removing" to be a relatively unusual case.

Bjorn

#15 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

Quoted 3 lines · Bjorn Helgaas

Thank you! A BIOS update is almost never the answer because even if an update exists, we have to assume that most users in the field will never install the update.

Not to mention enabling 64-bit BARs, which is even more cumbersome ixgbe-specific magic that requires entirely dedicated tools...

Quoted 1 line
Quoted 1 line

.text .data .bss are not marked as E820_TYPE_RAM!

and

Quoted 2 lines
Quoted 2 lines

DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000df243000-0x00000000df251fff], contact BIOS vendor for fixes
DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000df243000-0x00000000df251fff]

[...]

Quoted 3 lines

I think Linux basically converts the info from EFI GetMemoryMap to an e820 format; I think booting with "efi=debug" would show more details of this.

The dmesg I've attached today is with efi=debug, but the weird thing is
- both of the above warnings manifested themself only once, with the first (verbose debugging: "MCFG debug") patch applied... Anyway.

The "memremap attempted on mixed range 0x0000000000000000 size: 0x8000

WARNING: CPU: 0 PID: 1 at kernel/iomem.c:78 memremap+0x154/0x170" also

seems to be triggered by "efi=debug", so my guess is that it's unrelated.

Signature · 1 line
Tomasz Pala <[email protected]>
#16 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

Quoted 13 lines · Tomasz Pala
Quoted 3 lines · Bjorn Helgaas

Thank you! A BIOS update is almost never the answer because even if an update exists, we have to assume that most users in the field will never install the update.

Not to mention enabling 64-bit BARs, which is even more cumbersome ixgbe-specific magic that requires entirely dedicated tools...

Quoted 1 line
Quoted 1 line

.text .data .bss are not marked as E820_TYPE_RAM!

and

Quoted 2 lines
Quoted 2 lines

DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000df243000-0x00000000df251fff], contact BIOS vendor for fixes
DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000df243000-0x00000000df251fff]

Quoted 8 lines

[...]

Quoted 3 lines

I think Linux basically converts the info from EFI GetMemoryMap to an e820 format; I think booting with "efi=debug" would show more details of this.

The dmesg I've attached today is with efi=debug, but the weird thing is
- both of the above warnings manifested themself only once, with the first (verbose debugging: "MCFG debug") patch applied... Anyway.

OK. I don't know what (if anything) to do about the above.

Quoted 3 lines

The "memremap attempted on mixed range 0x0000000000000000 size: 0x8000

WARNING: CPU: 0 PID: 1 at kernel/iomem.c:78 memremap+0x154/0x170" also

seems to be triggered by "efi=debug", so my guess is that it's unrelated.

Yes, I think so. This is from efi_debugfs_init(), which we only run when "efi=debug", and I think it comes from memremapping this area:

efi: mem00: [Boot Code | | | | | | | | | | |WB|WT|WC|UC] range=[0x0000000000000000-0x0000000000007fff] (0MB)

Bjorn

#17 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

In reply to Tomasz Pala
Quoted 43 lines · Tomasz Pala
Quoted 8 lines · Bjorn Helgaas

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.

Quoted 8 lines

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.

Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.

I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.

efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[...]
[mem 0x7f800000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
[...]
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0: pci_request_selected_regions failed 0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16

After disabling the code causing this (using always-false condition:

		if (size >= 256*1024 && 0) {

) in the chunk:

https://lore.kernel.org/lkml/[email protected]/

the BAR starts at 0x90000000 (not 0x80000000):

Quoted 8 lines

efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry

and everything seems to work again.

Adding to regression tracking:

#regzbot ^introduced: 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=218107

#18 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

Quoted 6 lines · Bjorn Helgaas

...

Adding to regression tracking:

#regzbot ^introduced: 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=218107

#regzbot title: PCI BAR vs MCFG/ECAM resource conflict
#regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=218050
#regzbot link: https://forum.proxmox.com/threads/proxmox-8-kernel-6-2-16-4-pve-ixgbe-driver-fails-to-load-due-to-pci-device-probing-failure.131203/

#19 lore raw

Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

In reply to Tomasz Pala

[TLDR: This mail in primarily relevant for Linux kernel regression tracking. See link in footer if these mails annoy you.]

Quoted 9 lines · Tomasz Pala

Hello,

Quoted 3 lines · Bjorn Helgaas

Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.

I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.

#regzbot fix: x86/pci: Reserve ECAM if BIOS didn't include it in PNP0C02 _CRS
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Tested-by on the v1 series
#20 lore raw
In reply to Bjorn Helgaas
Quoted 13 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.

This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.

This happened because mmconfig-shared.c checks to see that ECAM space is reserved in E820 or ACPI motherboard resources. If it's not, it assumes ECAM doesn't work. 07eab0901ede removed some E820 entries, so it looked like ECAM was no longer reserved, so we stopped using it.

Quoted 8 lines

The machines where this broke don't reserve the ECAM in ACPI PNP0C02 devices (which seems to be what the PCI Firmware spec requires), but they do mention it in the EFI memory map as EfiMemoryMappedIO.

Bjorn Helgaas (2):
x86/pci: Simplify is_mmconf_reserved() messages x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

This patch series fixes the issue on my machine.

Thanks,
Kan

Quoted 3 lines
 arch/x86/pci/mmconfig-shared.c | 44 +++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 6 deletions(-)
Tested-by on the v1 series
#22 lore raw
In reply to Bjorn Helgaas
Quoted 7 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.

This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.

For QAT:

Signature · 1 line
Giovanni
author Applied
#23 lore raw
In reply to Bjorn Helgaas
Quoted 17 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.

This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.

This happened because mmconfig-shared.c checks to see that ECAM space is reserved in E820 or ACPI motherboard resources. If it's not, it assumes ECAM doesn't work. 07eab0901ede removed some E820 entries, so it looked like ECAM was no longer reserved, so we stopped using it.

The machines where this broke don't reserve the ECAM in ACPI PNP0C02 devices (which seems to be what the PCI Firmware spec requires), but they do mention it in the EFI memory map as EfiMemoryMappedIO.

Quoted 6 lines

Bjorn Helgaas (2):
x86/pci: Simplify is_mmconf_reserved() messages x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

 arch/x86/pci/mmconfig-shared.c | 44 +++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 6 deletions(-)

Applied to for-linus for v6.2. Sorry for the breakage and thank you very much for the debugging and testing help!

Tested-by on the v1 series
#24 lore raw
In reply to Bjorn Helgaas

With this updated patches, perf uncore works fine on SPR(tested on both DNP and MCC).

-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Wednesday, 11 January, 2023 02:03
To: [email protected]
Cc: Williams, Dan J <[email protected]>; Kan Liang <[email protected]>; Luck, Tony <[email protected]>; Box, David E <[email protected]>; Sun, Yunying <[email protected]>; Jiang, Dave <[email protected]>; Mika Westerberg <[email protected]>; Cabiddu, Giovanni <[email protected]>; Herbert Xu <[email protected]>; Hans de Goede <[email protected]>; Florent DELAHAYE <[email protected]>; Konrad J Hambrick <[email protected]>; Matt Hansen <[email protected]>; Nicholas Johnson <[email protected]>; Benoit Grégoire <[email protected]>; Werner Sembach <[email protected]>; [email protected]; [email protected]; Bjorn Helgaas <[email protected]>
Subject: [PATCH 0/2] PCI: Fix extended config space regression

From: Bjorn Helgaas <[email protected]>

07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.

This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.

This happened because mmconfig-shared.c checks to see that ECAM space is reserved in E820 or ACPI motherboard resources. If it's not, it assumes ECAM doesn't work. 07eab0901ede removed some E820 entries, so it looked like ECAM was no longer reserved, so we stopped using it.

The machines where this broke don't reserve the ECAM in ACPI PNP0C02 devices (which seems to be what the PCI Firmware spec requires), but they do mention it in the EFI memory map as EfiMemoryMappedIO.

Bjorn Helgaas (2):
x86/pci: Simplify is_mmconf_reserved() messages x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space

 arch/x86/pci/mmconfig-shared.c | 44 +++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 6 deletions(-)

--
2.25.1