PCI: Fix extended config space regression
cover PCI: Fix extended config space regression
From: Bjorn Helgaas <[email protected]>
07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.
This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.
This happened because mmconfig-shared.c checks to see that ECAM space is reserved in E820 or ACPI motherboard resources. If it's not, it assumes ECAM doesn't work. 07eab0901ede removed some E820 entries, so it looked like ECAM was no longer reserved, so we stopped using it.
The machines where this broke don't reserve the ECAM in ACPI PNP0C02 devices (which seems to be what the PCI Firmware spec requires), but they do mention it in the EFI memory map as EfiMemoryMappedIO.
Bjorn Helgaas (2):
x86/pci: Simplify is_mmconf_reserved() messages x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
arch/x86/pci/mmconfig-shared.c | 44 +++++++++++++++++++++++++++++----- 1 file changed, 38 insertions(+), 6 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/2 x86/pci: Simplify is_mmconf_reserved() messages
From: Bjorn Helgaas <[email protected]>
is_mmconf_reserved() takes a "with_e820" parameter that only determines the message logged if it finds the MMCONFIG region is reserved. Pass the message directly, which will simplify a future patch that adds a new way of looking for that reservation. No functional change intended.
--- arch/x86/pci/mmconfig-shared.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
arch/x86/pci/mmconfig-shared.c +7 -6
@@ -446,13 +446,12 @@ typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type);static bool __ref is_mmconf_reserved (check_reserved_t is_reserved,struct pci_mmcfg_region *cfg, -struct device *dev,int with_e820) +struct device *dev,char *method) {u64 addr = cfg->res.start;u64 size =resource_size (&cfg->res);u64 old_size = size;int num_buses; -char *method = with_e820 ?"E820" :"ACPI motherboard resources" ;while (!is_reserved (addr, addr + size,E820_TYPE_RESERVED )) { size >>=1 ;
@@ -464,10 +463,10 @@ static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,return false;if (dev) -dev_info (dev,"MMCONFIG at %pR reserved in %s\n" , +dev_info (dev,"MMCONFIG at %pR reserved as %s\n" , &cfg->res, method);else -pr_info (PREFIX "MMCONFIG at %pR reserved in %s\n" , +pr_info (PREFIX "MMCONFIG at %pR reserved as %s\n" , &cfg->res, method);if (old_size != size) {
@@ -500,7 +499,8 @@ static bool __ref pci_mmcfg_check_reserved(struct device *dev, struct pci_mmcfg_region *cfg, int early) { if (!early && !acpi_disabled) { - if (is_mmconf_reserved(is_acpi_reserved, cfg, dev, 0)) + if (is_mmconf_reserved(is_acpi_reserved, cfg, dev, + "ACPI motherboard resource")) return true; if (dev)
@@ -527,7 +527,8 @@ pci_mmcfg_check_reserved(struct device *dev, struct pci_mmcfg_region *cfg, int e/* Don't try to do this check unless configuration type 1 is available. how about type 2 ?*/ if (raw_pci_ops) -return is_mmconf_reserved (e820__mapped_all, cfg, dev,1 ); +return is_mmconf_reserved (e820__mapped_all, cfg, dev, +"E820 entry" );return false; }
Signature · 1 line
2.25.1
RE: [PATCH 1/2] x86/pci: Simplify is_mmconf_reserved() messages
Quoted 17 lines · Bjorn Helgaas
From: Bjorn Helgaas <[email protected]>
is_mmconf_reserved()takes a "with_e820" parameter that only determines the message logged if it finds the MMCONFIG region is reserved. Pass the message directly, which will simplify a future patch that adds a new way of looking for that reservation. No functional change intended.Signed-off-by Bjorn Helgaas <[email protected]>--- arch/x86/pci/mmconfig-shared.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)arch/x86/pci/mmconfig-shared.c +0
-0
arch/x86/pci/mmconfig-shared.c typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type); · line 449
Quoted 5 lines
arch/x86/pci/mmconfig-shared.c +1
-1@@ -446,13 +446,12 @@ typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type);static bool __ref is_mmconf_reserved (check_reserved_t is_reserved,struct pci_mmcfg_region *cfg, -struct device *dev,int with_e820) +struct device *dev,char *method)
@method could be 'const char *', but either way:
Re: [PATCH 1/2] x86/pci: Simplify is_mmconf_reserved() messages
Quoted 19 lines · Dan Williams
Quoted 18 lines · Bjorn Helgaas
From: Bjorn Helgaas <[email protected]>
is_mmconf_reserved()takes a "with_e820" parameter that only determines the message logged if it finds the MMCONFIG region is reserved. Pass the message directly, which will simplify a future patch that adds a new way of looking for that reservation. No functional change intended.Signed-off-by Bjorn Helgaas <[email protected]>--- arch/x86/pci/mmconfig-shared.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)arch/x86/pci/mmconfig-shared.c +0
-0@@ -446,13 +446,12 @@ typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type);
Quoted 6 lines
arch/x86/pci/mmconfig-shared.c typedef bool (*check_reserved_t)(u64 start, u64 end, enum e820_type type); · line 449
Quoted 4 lines
arch/x86/pci/mmconfig-shared.c +1
-1static bool __ref is_mmconf_reserved (check_reserved_t is_reserved,struct pci_mmcfg_region *cfg, -struct device *dev,int with_e820) +struct device *dev,char *method)@method could be 'const char *', but either way:
Of course, thanks! Folded in.
PATCH 2/2 x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
In reply to Bjorn HelgaasFrom: Bjorn Helgaas <[email protected]>
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02 _CRS method (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.
Some firmware doesn't report ECAM space via PNP0C02 _CRS methods, but does mention it as an EfiMemoryMappedIO region via EFI GetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.
07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.
Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02 _CRS.
Reported by Kan Liang, Tony Luck, and Giovanni Cabiddu.
--- arch/x86/pci/mmconfig-shared.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
arch/x86/pci/mmconfig-shared.c +31 -0
@@ -12,6 +12,7 @@*/ #include <linux/acpi.h> +#include <linux/efi.h> #include <linux/pci.h> #include <linux/init.h> #include <linux/bitmap.h>
@@ -442,6 +443,32 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)return mcfg_res.flags; } +static bool is_efi_mmio (u64 start,u64 end,enum e820_type not_used) +{ +#ifdef CONFIG_EFI +efi_memory_desc_t *md; +u64 size, mmio_start, mmio_end; + +for_each_efi_memory_desc (md) { +if (md->type ==EFI_MEMORY_MAPPED_IO ) { + size = md->num_pages <<EFI_PAGE_SHIFT ; + mmio_start = md->phys_addr; + mmio_end = mmio_start + size; + +/* +* N.B. Caller supplies (start, start + size), +* so to match, mmio_end is the first address +* *past* the EFI_MEMORY_MAPPED_IO area. +*/ +if (mmio_start <= start && end <= mmio_end) +return true; + } + } +#endif + +return false; +} +typedef bool (*check_reserved_t )(u64 start,u64 end,enum e820_type type); static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,
@@ -513,6 +540,10 @@ pci_mmcfg_check_reserved(struct device *dev, struct pci_mmcfg_region *cfg, int e"MMCONFIG at %pR not reserved in " "ACPI motherboard resources\n" , &cfg->res); + +if (is_mmconf_reserved (is_efi_mmio, cfg, dev, +"EfiMemoryMappedIO" )) +return true; }/*
Signature · 1 line
2.25.1
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Quoted 19 lines · CET Bjorn Helgaas
From: Bjorn Helgaas <[email protected]>
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02
_CRSmethod (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.Some firmware doesn't report ECAM space via PNP0C02
_CRSmethods, but does mention it as an EfiMemoryMappedIO region via EFIGetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
_CRS.
Quoted 8 lines
Reported by Kan Liang, Tony Luck, and Giovanni Cabiddu.
Fixes 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")Reported-by Kan Liang <[email protected]>Reported-by Tony Luck <[email protected]>Reported-by Giovanni Cabiddu <[email protected]>Signed-off-by Bjorn Helgaas <[email protected]>
Quoted 61 lines
--- arch/x86/pci/mmconfig-shared.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)arch/x86/pci/mmconfig-shared.c +31
-0@@ -12,6 +12,7 @@*/ #include <linux/acpi.h> +#include <linux/efi.h> #include <linux/pci.h> #include <linux/init.h> #include <linux/bitmap.h> @@ -442,6 +443,32 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)return mcfg_res.flags; } +static bool is_efi_mmio (u64 start,u64 end,enum e820_type not_used) +{ +#ifdef CONFIG_EFI +efi_memory_desc_t *md; +u64 size, mmio_start, mmio_end; + +for_each_efi_memory_desc (md) { +if (md->type ==EFI_MEMORY_MAPPED_IO ) { + size = md->num_pages <<EFI_PAGE_SHIFT ; + mmio_start = md->phys_addr; + mmio_end = mmio_start + size; + +/* +* N.B. Caller supplies (start, start + size), +* so to match, mmio_end is the first address +* *past* the EFI_MEMORY_MAPPED_IO area. +*/ +if (mmio_start <= start && end <= mmio_end) +return true; + } + } +#endif + +return false; +} +typedef bool (*check_reserved_t )(u64 start,u64 end,enum e820_type type); static bool __ref is_mmconf_reserved(check_reserved_t is_reserved,@@ -513,6 +540,10 @@ pci_mmcfg_check_reserved(struct device *dev, struct pci_mmcfg_region *cfg, int e"MMCONFIG at %pR not reserved in " "ACPI motherboard resources\n" , &cfg->res); + +if (is_mmconf_reserved (is_efi_mmio, cfg, dev, +"EfiMemoryMappedIO" )) +return true; }/*
RE: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
In reply to Bjorn HelgaasQuoted 59 lines · Bjorn Helgaas
From: Bjorn Helgaas <[email protected]>
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02
_CRSmethod (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.Some firmware doesn't report ECAM space via PNP0C02
_CRSmethods, but does mention it as an EfiMemoryMappedIO region via EFIGetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
_CRS.Reported by Kan Liang, Tony Luck, and Giovanni Cabiddu.
Fixes 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")Reported-by Kan Liang <[email protected]>Reported-by Tony Luck <[email protected]>Reported-by Giovanni Cabiddu <[email protected]>Signed-off-by Bjorn Helgaas <[email protected]>--- arch/x86/pci/mmconfig-shared.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)arch/x86/pci/mmconfig-shared.c +13
-0@@ -12,6 +12,7 @@*/ #include <linux/acpi.h> +#include <linux/efi.h> #include <linux/pci.h> #include <linux/init.h> #include <linux/bitmap.h> @@ -442,6 +443,32 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)return mcfg_res.flags; } +static bool is_efi_mmio (u64 start,u64 end,enum e820_type not_used) +{ +#ifdef CONFIG_EFI +efi_memory_desc_t *md; +u64 size, mmio_start, mmio_end; + +for_each_efi_memory_desc (md) { +if (md->type ==EFI_MEMORY_MAPPED_IO ) { + size = md->num_pages <<EFI_PAGE_SHIFT ; + mmio_start = md->phys_addr; + mmio_end = mmio_start + size; +
arch/x86/pci/mmconfig-shared.c static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used) · line 467
Quoted 10 lines
arch/x86/pci/mmconfig-shared.c +10
-0+/* +* N.B. Caller supplies (start, start + size), +* so to match, mmio_end is the first address +* *past* the EFI_MEMORY_MAPPED_IO area. +*/ +if (mmio_start <= start && end <= mmio_end) + return true; + } + } +#endif
Perhaps the following trick (compile tested), but either way:
arch/x86/pci/mmconfig-shared.c +0 -2
@@ -445,7 +445,6 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)static bool is_efi_mmio (u64 start,u64 end,enum e820_type not_used) { -#ifdef CONFIG_EFI efi_memory_desc_t *md;u64 size, mmio_start, mmio_end;
@@ -464,7 +463,6 @@ static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)return true; } } -#endif return false; }
include/linux/efi.h +4 -0
@@ -790,8 +790,12 @@ extern int efi_memattr_apply_permissions(struct mm_struct *mm,* * Once the loop finishes @md must not be accessed. */ +#ifdef CONFIG_EFI #define for_each_efi_memory_desc (md) \ for_each_efi_memory_desc_in_map(&efi.memmap, md) +#else +#define for_each_efi_memory_desc (md) for (; 0;) +#endif /* * Format an EFI memory descriptor's type and attributes to a user-provided
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Quoted 73 lines · Dan Williams
Quoted 59 lines · Bjorn Helgaas
From: Bjorn Helgaas <[email protected]>
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02
_CRSmethod (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.Some firmware doesn't report ECAM space via PNP0C02
_CRSmethods, but does mention it as an EfiMemoryMappedIO region via EFIGetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
_CRS.Reported by Kan Liang, Tony Luck, and Giovanni Cabiddu.
Fixes 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map")Reported-by Kan Liang <[email protected]>Reported-by Tony Luck <[email protected]>Reported-by Giovanni Cabiddu <[email protected]>Signed-off-by Bjorn Helgaas <[email protected]>--- arch/x86/pci/mmconfig-shared.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)arch/x86/pci/mmconfig-shared.c +13
-0@@ -12,6 +12,7 @@*/ #include <linux/acpi.h> +#include <linux/efi.h> #include <linux/pci.h> #include <linux/init.h> #include <linux/bitmap.h> @@ -442,6 +443,32 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)return mcfg_res.flags; } +static bool is_efi_mmio (u64 start,u64 end,enum e820_type not_used) +{ +#ifdef CONFIG_EFI +efi_memory_desc_t *md; +u64 size, mmio_start, mmio_end; + +for_each_efi_memory_desc (md) { +if (md->type ==EFI_MEMORY_MAPPED_IO ) { + size = md->num_pages <<EFI_PAGE_SHIFT ; + mmio_start = md->phys_addr; + mmio_end = mmio_start + size; +arch/x86/pci/mmconfig-shared.c static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used) · line 467
Quoted 10 lines
arch/x86/pci/mmconfig-shared.c +10
-0+/* +* N.B. Caller supplies (start, start + size), +* so to match, mmio_end is the first address +* *past* the EFI_MEMORY_MAPPED_IO area. +*/ +if (mmio_start <= start && end <= mmio_end) + return true; + } + } +#endif Perhaps the following trick (compile tested), but either way:
Quoted 1 line
Reviewed-by Dan Williams <[email protected]>
That's a great trick, and I wish I'd thought of it. I have some follow-on patches I'm considering for v6.3, so in the interest of streamlining the path of this one to v6.2-rc4, I think I'll wait on this until v6.3.
Quoted 37 lines
arch/x86/pci/mmconfig-shared.c +0
-2@@ -445,7 +445,6 @@ static bool is_acpi_reserved(u64 start, u64 end, enum e820_type not_used)static bool is_efi_mmio (u64 start,u64 end,enum e820_type not_used) { -#ifdef CONFIG_EFI efi_memory_desc_t *md;u64 size, mmio_start, mmio_end;@@ -464,7 +463,6 @@ static bool is_efi_mmio(u64 start, u64 end, enum e820_type not_used)return true; } } -#endif return false; }include/linux/efi.h +4
-0@@ -790,8 +790,12 @@ extern int efi_memattr_apply_permissions(struct mm_struct *mm,* * Once the loop finishes @md must not be accessed. */ +#ifdef CONFIG_EFI #define for_each_efi_memory_desc (md) \ for_each_efi_memory_desc_in_map(&efi.memmap, md) +#else +#define for_each_efi_memory_desc (md) for (; 0;) +#endif /* * Format an EFI memory descriptor's type and attributes to a user-provided
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
In reply to Bjorn HelgaasHello,
Quoted 8 lines · Bjorn Helgaas
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02
_CRSmethod (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.Some firmware doesn't report ECAM space via PNP0C02
_CRSmethods, but does mention it as an EfiMemoryMappedIO region via EFIGetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.
Quoted 8 lines
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
_CRS.
I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.
efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[...]
[mem 0x7f800000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
[...]
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0: pci_request_selected_regions failed 0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16
After disabling the code causing this (using always-false condition:
if (size >= 256*1024 && 0) {) in the chunk:
https://lore.kernel.org/lkml/[email protected]/
the BAR starts at 0x90000000 (not 0x80000000):
efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry
and everything seems to work again.
I've got full system bootup logs from the upstream and worked around, but I'm not sure if this is OK to attach them (the CC list is long).
Also, this is my test machine so I can run some experiments.
best regards,
Signature · 1 line
Tomasz Pala <[email protected]>
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Quoted 14 lines · Tomasz Pala
I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.
efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[...]
[mem 0x7f800000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
[...]
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0:pci_request_selected_regionsfailed0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16
FWIW, as I got no response - there were other people facing the issue as well:
Apparently this might be some hardware quirk, therefore I'm not sure if the internal EfiMemoryMappedIO reservation logic should be reviewed, or some quirk handling to be added, or maybe some CONFIG_option introduced.
Anyone please?
Signature · 1 line
Tomasz Pala <[email protected]>
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
In reply to Tomasz PalaQuoted 14 lines · Tomasz Pala
Quoted 13 lines · Bjorn Helgaas
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02
_CRSmethod (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.Some firmware doesn't report ECAM space via PNP0C02
_CRSmethods, but does mention it as an EfiMemoryMappedIO region via EFIGetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.
Quoted 5 lines
Quoted 3 lines
Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
_CRS.I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.
Thanks very much for the report, and sorry for the inconvenience and my delay in looking at it.
Quoted 9 lines
efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[mem 0x7f800000-0xfed1bfff] available for PCI devices
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0:pci_request_selected_regionsfailed0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16
Something is wrong with our allocation scheme. Both the MMCONFIG region and the ixgbe BAR 0 are at 0x80000000, which obviously cannot work. Maybe the full dmesg log will have a clue about why we didn't move ixgbe out of the way.
Quoted 15 lines
After disabling the code causing this (using always-false condition:
if (size >= 256*1024 && 0) {) in the chunk:
https://lore.kernel.org/lkml/[email protected]/
the BAR starts at
0x90000000(not0x80000000):efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry
Quoted 5 lines
and everything seems to work again.
I've got full system bootup logs from the upstream and worked around, but I'm not sure if this is OK to attach them (the CC list is long).
Would you mind opening a new report at https://bugzilla.kernel.org,
attaching those logs, and responding here with the URL?
I looked at the proxmox thread you mentioned, but sometimes people strip out parts of the log they think are irrelevant, and in this case, the stripped-out parts *are* relevant.
Bjorn
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Quoted 54 lines · Bjorn Helgaas
Quoted 14 lines · Tomasz Pala
Quoted 13 lines · Bjorn Helgaas
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02
_CRSmethod (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.Some firmware doesn't report ECAM space via PNP0C02
_CRSmethods, but does mention it as an EfiMemoryMappedIO region via EFIGetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.Quoted 5 lines
Quoted 3 lines
Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
_CRS.I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.
Thanks very much for the report, and sorry for the inconvenience and my delay in looking at it.
Quoted 9 lines
efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[mem 0x7f800000-0xfed1bfff] available for PCI devices
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0:pci_request_selected_regionsfailed0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16Something is wrong with our allocation scheme. Both the MMCONFIG region and the ixgbe BAR 0 are at
0x80000000, which obviously cannot work. Maybe the full dmesg log will have a clue about why we didn't move ixgbe out of the way.Quoted 15 lines
After disabling the code causing this (using always-false condition:
if (size >= 256*1024 && 0) {) in the chunk:
https://lore.kernel.org/lkml/[email protected]/
the BAR starts at
0x90000000(not0x80000000):efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entry
Quoted 8 lines
Quoted 5 lines
and everything seems to work again.
I've got full system bootup logs from the upstream and worked around, but I'm not sure if this is OK to attach them (the CC list is long).
Would you mind opening a new report at https://bugzilla.kernel.org,
attaching those logs, and responding here with the URL?
Thanks for the report and the logs, which are attached at
https://bugzilla.kernel.org/show_bug.cgi?id=218050
I think the problem is that the MMCONFIG region is at
[mem 0x80000000-0x8fffffff], and that is *also* included in one of the host bridge windows reported via _CRS:
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) pci_bus 0000:00: root bus resource [mem 0x80000000-0xfbffffff window]
I'll try to figure out how to deal with that. In the meantime, would you mind attaching the contents of /proc/iomem to the bugzilla? I think you have to cat it as root to get the actual values included.
Bjorn
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
[+cc Sebastian]
Quoted 76 lines · Bjorn Helgaas
Quoted 59 lines · Bjorn Helgaas
Quoted 18 lines · Tomasz Pala
Quoted 17 lines · Bjorn Helgaas
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02
_CRSmethod (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.Some firmware doesn't report ECAM space via PNP0C02
_CRSmethods, but does mention it as an EfiMemoryMappedIO region via EFIGetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.Quoted 6 lines
Quoted 3 lines
Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
_CRS.I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.
Thanks very much for the report, and sorry for the inconvenience and my delay in looking at it.
Quoted 9 lines
efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[mem 0x7f800000-0xfed1bfff] available for PCI devices
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0:pci_request_selected_regionsfailed0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16Something is wrong with our allocation scheme. Both the MMCONFIG region and the ixgbe BAR 0 are at
0x80000000, which obviously cannot work. Maybe the full dmesg log will have a clue about why we didn't move ixgbe out of the way.Quoted 15 lines
After disabling the code causing this (using always-false condition:
if (size >= 256*1024 && 0) {) in the chunk:
https://lore.kernel.org/lkml/[email protected]/
the BAR starts at
0x90000000(not0x80000000):efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entryQuoted 8 lines
Quoted 5 lines
and everything seems to work again.
I've got full system bootup logs from the upstream and worked around, but I'm not sure if this is OK to attach them (the CC list is long).
Would you mind opening a new report at https://bugzilla.kernel.org,
attaching those logs, and responding here with the URL?Thanks for the report and the logs, which are attached at
https://bugzilla.kernel.org/show_bug.cgi?id=218050I think the problem is that the MMCONFIG region is at
[mem 0x80000000-0x8fffffff], and that is *also* included in one of the host bridge windows reported via_CRS:
Quoted 6 lines
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base
0x80000000)pci_bus0000:00: root bus resource [mem 0x80000000-0xfbffffff window]I'll try to figure out how to deal with that. In the meantime, would you mind attaching the contents of
/proc/iomemto the bugzilla? I think you have to cat it as root to get the actual values included.
I attached a debug patch to both bugzilla entries. If you could attach the "acpidump" output and (if practical) boot a kernel with the debug patch and attach the dmesg logs, that would be great.
Bjorn
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Quoted 9 lines · Bjorn Helgaas
Quoted 9 lines
https://bugzilla.kernel.org/show_bug.cgi?id=218050
I think the problem is that the MMCONFIG region is at
[mem 0x80000000-0x8fffffff], and that is *also* included in one of the host bridge windows reported via_CRS:PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base
0x80000000)pci_bus0000:00: root bus resource [mem 0x80000000-0xfbffffff window]
Quoted 6 lines
Quoted 2 lines
I'll try to figure out how to deal with that. In the meantime, would you mind attaching the contents of
/proc/iomemto the bugzilla? II attached a debug patch to both bugzilla entries. If you could attach the "acpidump" output and (if practical) boot a kernel with the debug patch and attach the dmesg logs, that would be great.
I've posted the files. There are signs of buggy BIOS, but I don't expect any firmware update to be released for this hw anymore.
DMI: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.4 11/20/2019
.text .data .bss are not marked as E820_TYPE_RAM!
tboot: non-0 tboot_addr but it is not of type E820_TYPE_RESERVED
DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000df243000-0x00000000df251fff], contact BIOS vendor for fixes
DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000df243000-0x00000000df251fff]
BTW is there a reason for this logging discrepancy?
efi: Remove mem173: MMIO range=[0xe0000000-0xefffffff] (256MB) from e820 map
efi: Not removing mem71: MMIO range=[0xe0000000-0xefffffff] (262144KB) from e820 map
efi: Not removing mem74: MMIO range=[0xff000000-0xffffffff] (16384KB) from e820 map
efi: Remove mem176: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map
This is arch/x86/platform/efi/efi.c:
static void __init efi_remove_e820_mmio(void)
Remove mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluMB) ... size >> 20 Not removing mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluKB) ... size >> 10
Signature · 1 line
Tomasz Pala <[email protected]>
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Quoted 14 lines · Tomasz Pala
Quoted 12 lines · Bjorn Helgaas
Quoted 11 lines
https://bugzilla.kernel.org/show_bug.cgi?id=218050
I think the problem is that the MMCONFIG region is at
[mem 0x80000000-0x8fffffff], and that is *also* included in one of the host bridge windows reported via_CRS:PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base
0x80000000)pci_bus0000:00: root bus resource [mem 0x80000000-0xfbffffff window]I'll try to figure out how to deal with that. In the meantime, would you mind attaching the contents of
/proc/iomemto the bugzilla? I
Quoted 6 lines
Quoted 3 lines
I attached a debug patch to both bugzilla entries. If you could attach the "acpidump" output and (if practical) boot a kernel with the debug patch and attach the dmesg logs, that would be great.
I've posted the files. There are signs of buggy BIOS, but I don't expect any firmware update to be released for this hw anymore.
Thank you! A BIOS update is almost never the answer because even if an update exists, we have to assume that most users in the field will never install the update.
I want to look at the BIOS info in case we can learn about something *Linux* is doing wrong. This most likely works fine with Windows, so I assume Linux is doing something wrong or at least differently than Windows.
Quoted 3 lines
DMI: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.4 11/20/2019
.text .data .bss are not marked as
E820_TYPE_RAM!
Added by 4eea6aa581ab ("x86, mm: if kernel .text .data .bss are not marked as E820_RAM, complain and fix"). No idea. A shame we didn't include the .text/.data values in the message.
Quoted 1 line
tboot: non-0
tboot_addrbut it is not of typeE820_TYPE_RESERVED
Added by 316253406959 ("x86, intel_txt: Intel TXT boot support"). No idea about this either.
Quoted 2 lines
DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000df243000-0x00000000df251fff], contact BIOS vendor for fixes
DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000df243000-0x00000000df251fff]
Both related to arch_rmrr_sanity_check(), added by f036c7fa0ab6 ("iommu/vt-d: Check VT-d RMRR region in BIOS is reported as reserved") and f5a68bb0752e ("iommu/vt-d: Mark firmware tainted if RMRR fails sanity check").
No idea about this one either. The VT-d spec (r1.3, sec 8.4) says "BIOS must report the RMRR reported memory addresses as reserved in the system memory map returned through methods such as INT15, EFI GetMemoryMap etc."
arch_rmrr_sanity_check() only looks at your e820 map, which only has this:
BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
BIOS-e820: [mem 0x0000000000100000-0x00000000d1f36fff] usable
I think Linux basically converts the info from EFI GetMemoryMap to an e820 format; I think booting with "efi=debug" would show more details of this.
Anyway, this is all a tangent.
Quoted 5 lines
BTW is there a reason for this logging discrepancy?
efi: Remove mem173: MMIO range=[0xe0000000-0xefffffff] (256MB) from e820 map
efi: Not removing mem71: MMIO range=[0xe0000000-0xefffffff] (262144KB) from e820 map
Quoted 8 lines
efi: Not removing mem74: MMIO range=[0xff000000-0xffffffff] (16384KB) from e820 map
efi: Remove mem176: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 mapThis is
arch/x86/platform/efi/efi.c:
static void__initefi_remove_e820_mmio(void)Remove mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluMB) ... size >> 20 Not removing mem%02u: MMIO range=[0x%08llx-0x%08llx] (%lluKB) ... size >> 10
You mean the MB vs KB difference? That's my fault. I guess I used KB for the "Not removing" message because those are smaller (< 256KB) so the size in MB wouldn't be useful there. We could use KB for both, but I guess I used MB for the "Remove" case because it's a little easier to read and I expected "Not removing" to be a relatively unusual case.
Bjorn
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Quoted 3 lines · Bjorn Helgaas
Thank you! A BIOS update is almost never the answer because even if an update exists, we have to assume that most users in the field will never install the update.
Not to mention enabling 64-bit BARs, which is even more cumbersome ixgbe-specific magic that requires entirely dedicated tools...
Quoted 1 line
Quoted 1 line
.text .data .bss are not marked as
E820_TYPE_RAM!
and
Quoted 2 lines
Quoted 2 lines
DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000df243000-0x00000000df251fff], contact BIOS vendor for fixes
DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000df243000-0x00000000df251fff]
[...]
Quoted 3 lines
I think Linux basically converts the info from EFI GetMemoryMap to an e820 format; I think booting with "efi=debug" would show more details of this.
The dmesg I've attached today is with efi=debug, but the weird thing is
- both of the above warnings manifested themself only once, with the first (verbose debugging: "MCFG debug") patch applied... Anyway.
The "memremap attempted on mixed range 0x0000000000000000 size: 0x8000
WARNING: CPU: 0 PID: 1 at kernel/iomem.c:78 memremap+0x154/0x170" also
seems to be triggered by "efi=debug", so my guess is that it's unrelated.
Signature · 1 line
Tomasz Pala <[email protected]>
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Quoted 13 lines · Tomasz Pala
Quoted 3 lines · Bjorn Helgaas
Thank you! A BIOS update is almost never the answer because even if an update exists, we have to assume that most users in the field will never install the update.
Not to mention enabling 64-bit BARs, which is even more cumbersome ixgbe-specific magic that requires entirely dedicated tools...
Quoted 1 line
Quoted 1 line
.text .data .bss are not marked as
E820_TYPE_RAM!and
Quoted 2 lines
Quoted 2 lines
DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000df243000-0x00000000df251fff], contact BIOS vendor for fixes
DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000df243000-0x00000000df251fff]
Quoted 8 lines
[...]
Quoted 3 lines
I think Linux basically converts the info from EFI GetMemoryMap to an e820 format; I think booting with "efi=debug" would show more details of this.
The dmesg I've attached today is with efi=debug, but the weird thing is
- both of the above warnings manifested themself only once, with the first (verbose debugging: "MCFG debug") patch applied... Anyway.
OK. I don't know what (if anything) to do about the above.
Quoted 3 lines
The "memremap attempted on mixed range
0x0000000000000000size:0x8000WARNING: CPU: 0 PID: 1 at kernel/iomem.c:78 memremap+0x154/0x170" alsoseems to be triggered by "efi=debug", so my guess is that it's unrelated.
Yes, I think so. This is from efi_debugfs_init(), which we only run when "efi=debug", and I think it comes from memremapping this area:
efi: mem00: [Boot Code | | | | | | | | | | |WB|WT|WC|UC] range=[0x0000000000000000-0x0000000000007fff] (0MB)
Bjorn
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
In reply to Tomasz PalaQuoted 43 lines · Tomasz Pala
Quoted 8 lines · Bjorn Helgaas
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02
_CRSmethod (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.Some firmware doesn't report ECAM space via PNP0C02
_CRSmethods, but does mention it as an EfiMemoryMappedIO region via EFIGetMemoryMap(), which is normally converted to an E820 entry by a bootloader or EFI stub.Quoted 8 lines
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map"), removes E820 entries that correspond to EfiMemoryMappedIO regions because some other firmware uses EfiMemoryMappedIO for PCI host bridge windows, and the E820 entries prevent Linux from allocating BAR space for hot-added devices.Allow use of ECAM for extended config space when the region is covered by an EfiMemoryMappedIO region, even if it's not included in E820 or PNP0C02
_CRS.I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.
efi: Remove mem63: MMIO range=[0x80000000-0x8fffffff] (256MB) from e820 map
[...]
[mem 0x7f800000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base0x80000000) [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as EfiMemoryMappedIO
[...]
ixgbe 0000:02:00.0: enabling device (0140 -> 0142)
ixgbe 0000:02:00.0: BAR 0: can't reserve [mem 0x80000000-0x8007ffff 64bit]
ixgbe 0000:02:00.0:pci_request_selected_regionsfailed0xfffffff0
ixgbe: probe of 0000:02:00.0 failed with error -16After disabling the code causing this (using always-false condition:
if (size >= 256*1024 && 0) {) in the chunk:
https://lore.kernel.org/lkml/[email protected]/
the BAR starts at
0x90000000(not0x80000000):
Quoted 8 lines
efi: Not removing mem63: MMIO range=[0x80000000-0x8fffffff] (262144KB) from e820 map [...]
[mem 0x90000000-0xfed1bfff] available for PCI devices
[...]
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base0x80000000)
PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved as E820 entryand everything seems to work again.
Adding to regression tracking:
#regzbot ^introduced: 07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=218107
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
Quoted 6 lines · Bjorn Helgaas
...
Adding to regression tracking:
#regzbot ^introduced:
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map") #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=218107
#regzbot title: PCI BAR vs MCFG/ECAM resource conflict
#regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=218050
#regzbot link: https://forum.proxmox.com/threads/proxmox-8-kernel-6-2-16-4-pve-ixgbe-driver-fails-to-load-due-to-pci-device-probing-failure.131203/
Re: [PATCH 2/2] x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
In reply to Tomasz Pala[TLDR: This mail in primarily relevant for Linux kernel regression tracking. See link in footer if these mails annoy you.]
Quoted 9 lines · Tomasz Pala
Hello,
Quoted 3 lines · Bjorn Helgaas
Normally we reject ECAM space unless it is reported as reserved in the E820 table or via a PNP0C02
_CRSmethod (PCI Firmware, r3.3, sec 4.1.2). This means PCI extended config space (offsets 0x100-0xfff) may not be accessible.I'm still having a problem initializing ixgbe NICs with pristine 6.5.7 kernel.
#regzbot fix: x86/pci: Reserve ECAM if BIOS didn't include it in PNP0C02 _CRS
#regzbot ignore-activity
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.
Quoted 13 lines · Bjorn Helgaas
From: Bjorn Helgaas <[email protected]>
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.
This happened because
mmconfig-shared.cchecks to see that ECAM space is reserved in E820 or ACPI motherboard resources. If it's not, it assumes ECAM doesn't work.07eab0901ederemoved some E820 entries, so it looked like ECAM was no longer reserved, so we stopped using it.
Quoted 8 lines
The machines where this broke don't reserve the ECAM in ACPI PNP0C02 devices (which seems to be what the PCI Firmware spec requires), but they do mention it in the EFI memory map as EfiMemoryMappedIO.
Bjorn Helgaas (2):
x86/pci: Simplifyis_mmconf_reserved()messages x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
This patch series fixes the issue on my machine.
Thanks,
Kan
Quoted 3 lines
arch/x86/pci/mmconfig-shared.c | 44 +++++++++++++++++++++++++++++----- 1 file changed, 38 insertions(+), 6 deletions(-)
Quoted 2 lines
This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.
For the EDAC on Broadwell case I reported:
Quoted 7 lines · Bjorn Helgaas
From: Bjorn Helgaas <[email protected]>
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.
For QAT:
Signature · 1 line
Giovanni
Bjorn Helgaas
AppliedQuoted 17 lines · Bjorn Helgaas
From: Bjorn Helgaas <[email protected]>
07eab0901ede("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.
This happened because
mmconfig-shared.cchecks to see that ECAM space is reserved in E820 or ACPI motherboard resources. If it's not, it assumes ECAM doesn't work.07eab0901ederemoved some E820 entries, so it looked like ECAM was no longer reserved, so we stopped using it.The machines where this broke don't reserve the ECAM in ACPI PNP0C02 devices (which seems to be what the PCI Firmware spec requires), but they do mention it in the EFI memory map as EfiMemoryMappedIO.
Quoted 6 lines
Bjorn Helgaas (2):
x86/pci: Simplifyis_mmconf_reserved()messages x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM spacearch/x86/pci/mmconfig-shared.c | 44 +++++++++++++++++++++++++++++----- 1 file changed, 38 insertions(+), 6 deletions(-)
Applied to for-linus for v6.2. Sorry for the breakage and thank you very much for the debugging and testing help!
With this updated patches, perf uncore works fine on SPR(tested on both DNP and MCC).
-----Original Message-----
From: Bjorn Helgaas <[email protected]>
Sent: Wednesday, 11 January, 2023 02:03
To: [email protected]
Cc: Williams, Dan J <[email protected]>; Kan Liang <[email protected]>; Luck, Tony <[email protected]>; Box, David E <[email protected]>; Sun, Yunying <[email protected]>; Jiang, Dave <[email protected]>; Mika Westerberg <[email protected]>; Cabiddu, Giovanni <[email protected]>; Herbert Xu <[email protected]>; Hans de Goede <[email protected]>; Florent DELAHAYE <[email protected]>; Konrad J Hambrick <[email protected]>; Matt Hansen <[email protected]>; Nicholas Johnson <[email protected]>; Benoit Grégoire <[email protected]>; Werner Sembach <[email protected]>; [email protected]; [email protected]; Bjorn Helgaas <[email protected]>
Subject: [PATCH 0/2] PCI: Fix extended config space regression
From: Bjorn Helgaas <[email protected]>
07eab0901ede ("efi/x86: Remove EfiMemoryMappedIO from E820 map") appeared in v6.2-rc1 and broke extended config space on several machines.
This broke drivers that use things in extended config space, e.g., perf, VSEC telemetry, EDAC, QAT, etc.
This happened because mmconfig-shared.c checks to see that ECAM space is reserved in E820 or ACPI motherboard resources. If it's not, it assumes ECAM doesn't work. 07eab0901ede removed some E820 entries, so it looked like ECAM was no longer reserved, so we stopped using it.
The machines where this broke don't reserve the ECAM in ACPI PNP0C02 devices (which seems to be what the PCI Firmware spec requires), but they do mention it in the EFI memory map as EfiMemoryMappedIO.
Bjorn Helgaas (2):
x86/pci: Simplify is_mmconf_reserved() messages x86/pci: Treat EfiMemoryMappedIO as reservation of ECAM space
arch/x86/pci/mmconfig-shared.c | 44 +++++++++++++++++++++++++++++----- 1 file changed, 38 insertions(+), 6 deletions(-)
--
2.25.1