Skip to contentSkip to conversation

x86/PCI: Log E820 clipping

PATCH In review Bjorn Helgaas 3 participants 12 messages Apr 14, 2022 – May 5, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover x86/PCI: Log E820 clipping

From: Bjorn Helgaas <[email protected]>

This is still work-in-progress on the issue of PNP0A03 _CRS methods that are buggy or not interpreted correctly by Linux.

The previous try at:
https://lore.kernel.org/r/[email protected]
caused regressions on some Chromebooks:
https://lore.kernel.org/r/[email protected]

This v2 drops the commit that caused the Chromebook regression, so it also doesn't fix the issue we were *trying* to fix on Lenovo Yoga and Clevo Barebones.

The point of this v2 update is to split the logging patch into (1) a pure logging addition and (2) the change to only clip PCI windows, which was previously hidden inside the logging patch and not well documented.

Bjorn Helgaas (3):
x86/PCI: Eliminate remove_e820_regions() common subexpressions
x86: Log resource clipping for E820 regions
x86/PCI: Clip only host bridge windows for E820 regions

 arch/x86/include/asm/e820/api.h |  5 +++++
 arch/x86/kernel/resource.c      | 23 ++++++++++++++++-------
 arch/x86/pci/acpi.c             |  5 +++++
 3 files changed, 26 insertions(+), 7 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH v2 1/3 x86/PCI: Eliminate remove_e820_regions() common subexpressions

From: Bjorn Helgaas <[email protected]>

Add local variables to reduce repetition later. No functional change intended.

---
 arch/x86/kernel/resource.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86/kernel/resource.c +4 -2
index 9b9fb7882c20..8ffe68437744 100644
@@ -27,12 +27,14 @@ static void remove_e820_regions(struct resource *avail)
 {
 	int i;
 	struct e820_entry *entry;
+	u64 e820_start, e820_end;
 
 	for (i = 0; i < e820_table->nr_entries; i++) {
 		entry = &e820_table->entries[i];
+		e820_start = entry->addr;
+		e820_end = entry->addr + entry->size - 1;
 
-		resource_clip(avail, entry->addr,
-			      entry->addr + entry->size - 1);
+		resource_clip(avail, e820_start, e820_end);
 	}
 }
 
Signature · 1 line
2.25.1
#2 lore raw

PATCH v2 2/3 x86: Log resource clipping for E820 regions

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

When remove_e820_regions() clips a resource because an E820 region overlaps it, log a note in dmesg to add in debugging.

---
 arch/x86/kernel/resource.c | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/kernel/resource.c +7 -0
index 8ffe68437744..30d524adb012 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <linux/ioport.h>
+#include <linux/printk.h>
 #include <asm/e820/api.h>
 
 static void resource_clip(struct resource *res, resource_size_t start,
@@ -28,6 +29,7 @@ static void remove_e820_regions(struct resource *avail)
 	int i;
 	struct e820_entry *entry;
 	u64 e820_start, e820_end;
+	struct resource orig = *avail;
 
 	for (i = 0; i < e820_table->nr_entries; i++) {
 		entry = &e820_table->entries[i];
@@ -35,6 +37,11 @@ static void remove_e820_regions(struct resource *avail)
 		e820_end = entry->addr + entry->size - 1;
 
 		resource_clip(avail, e820_start, e820_end);
+		if (orig.start != avail->start || orig.end != avail->end) {
+			pr_info("clipped %pR to %pR for e820 entry [mem %#010Lx-%#010Lx]\n",
+				 &orig, avail, e820_start, e820_end);
+			orig = *avail;
+		}
 	}
 }
 
Signature · 1 line
2.25.1
maintainer Acked-by on v2 2/3
#3 lore raw

Re: [PATCH v2 2/3] x86: Log resource clipping for E820 regions

Quoted 7 lines · Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

When remove_e820_regions() clips a resource because an E820 region overlaps it, log a note in dmesg to add in debugging.

Quoted 38 lines
---
 arch/x86/kernel/resource.c | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/kernel/resource.c +7 -1
index 8ffe68437744..30d524adb012 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <linux/ioport.h>
+#include <linux/printk.h>
 #include <asm/e820/api.h>
 
 static void resource_clip(struct resource *res, resource_size_t start,
@@ -28,6 +29,7 @@ static void remove_e820_regions(struct resource *avail)
        int i;
        struct e820_entry *entry;
        u64 e820_start, e820_end;
+       struct resource orig = *avail;
 
        for (i = 0; i < e820_table->nr_entries; i++) {
                entry = &e820_table->entries[i];
@@ -35,6 +37,11 @@ static void remove_e820_regions(struct resource *avail)
                e820_end = entry->addr + entry->size - 1;
 
                resource_clip(avail, e820_start, e820_end);
+               if (orig.start != avail->start || orig.end != avail->end) {
+                       pr_info("clipped %pR to %pR for e820 entry [mem %#010Lx-%#010Lx]\n",
+                                &orig, avail, e820_start, e820_end);
+                       orig = *avail;
+               }
        }
 }
 
--

2.25.1

#4 lore raw

PATCH v2 3/3 x86/PCI: Clip only host bridge windows for E820 regions

In reply to Bjorn Helgaas

From: Bjorn Helgaas <[email protected]>

ACPI firmware advertises PCI host bridge resources via PNP0A03 _CRS methods. Some BIOSes include non-window address space in _CRS, and if we allocate that non-window space for PCI devices, they don't work.

4dc2287c1805 ("x86: avoid E820 regions when allocating address space") works around this issue by clipping out any regions mentioned in the E820 table in the allocate_resource() path, but the implementation has a couple issues:

- The clipping is done for *all* allocations, not just those for PCI address space, and

- The clipping is done at each allocation instead of being done once when setting up the host bridge windows.

Rework the implementation so we only clip PCI host bridge windows, and we do it once when setting them up.

Example output changes:

    BIOS-e820: [mem 0x00000000b0000000-0x00000000c00fffff] reserved

+ acpi PNP0A08:00: clipped [mem 0xc0000000-0xfebfffff window] to [mem 0xc0100000-0xfebfffff window] for e820 entry [mem 0xb0000000-0xc00fffff]
- pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
+ pci_bus 0000:00: root bus resource [mem 0xc0100000-0xfebfffff window]

---
 arch/x86/include/asm/e820/api.h |  5 +++++
 arch/x86/kernel/resource.c      | 14 +++++++-------
 arch/x86/pci/acpi.c             |  5 +++++
 3 files changed, 17 insertions(+), 7 deletions(-)
arch/x86/include/asm/e820/api.h +5 -0
index e8f58ddd06d9..5a39ed59b6db 100644
@@ -4,6 +4,9 @@
 
 #include <asm/e820/types.h>
 
+struct device;
+struct resource;
+
 extern struct e820_table *e820_table;
 extern struct e820_table *e820_table_kexec;
 extern struct e820_table *e820_table_firmware;
@@ -43,6 +46,8 @@ extern void e820__register_nosave_regions(unsigned long limit_pfn);
 
 extern int  e820__get_entry_type(u64 start, u64 end);
 
+extern void remove_e820_regions(struct device *dev, struct resource *avail);
+
 /*
  * Returns true iff the specified range [start,end) is completely contained inside
  * the ISA region.
arch/x86/kernel/resource.c +7 -7
index 30d524adb012..db2b350a37b7 100644
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
+#include <linux/dev_printk.h>
 #include <linux/ioport.h>
-#include <linux/printk.h>
 #include <asm/e820/api.h>
 
 static void resource_clip(struct resource *res, resource_size_t start,
@@ -24,13 +24,16 @@ static void resource_clip(struct resource *res, resource_size_t start,
 		res->start = end + 1;
 }
 
-static void remove_e820_regions(struct resource *avail)
+void remove_e820_regions(struct device *dev, struct resource *avail)
 {
 	int i;
 	struct e820_entry *entry;
 	u64 e820_start, e820_end;
 	struct resource orig = *avail;
 
+	if (!(avail->flags & IORESOURCE_MEM))
+		return;
+
 	for (i = 0; i < e820_table->nr_entries; i++) {
 		entry = &e820_table->entries[i];
 		e820_start = entry->addr;
@@ -38,7 +41,7 @@ static void remove_e820_regions(struct resource *avail)
 
 		resource_clip(avail, e820_start, e820_end);
 		if (orig.start != avail->start || orig.end != avail->end) {
-			pr_info("clipped %pR to %pR for e820 entry [mem %#010Lx-%#010Lx]\n",
+			dev_info(dev, "clipped %pR to %pR for e820 entry [mem %#010Lx-%#010Lx]\n",
 				 &orig, avail, e820_start, e820_end);
 			orig = *avail;
 		}
@@ -52,9 +55,6 @@ void arch_remove_reservations(struct resource *avail)
 	 * the low 1MB unconditionally, as this area is needed for some ISA
 	 * cards requiring a memory range, e.g. the i82365 PCMCIA controller.
 	 */
-	if (avail->flags & IORESOURCE_MEM) {
+	if (avail->flags & IORESOURCE_MEM)
 		resource_clip(avail, BIOS_ROM_BASE, BIOS_ROM_END);
-
-		remove_e820_regions(avail);
-	}
 }
arch/x86/pci/acpi.c +5 -0
index 052f1d78a562..562c81a51ea0 100644
@@ -8,6 +8,7 @@
 #include <linux/pci-acpi.h>
 #include <asm/numa.h>
 #include <asm/pci_x86.h>
+#include <asm/e820/api.h>
 
 struct pci_root_info {
 	struct acpi_pci_root_info common;
@@ -299,6 +300,10 @@ static int pci_acpi_root_prepare_resources(struct acpi_pci_root_info *ci)
 	int status;
 
 	status = acpi_pci_probe_root_resources(ci);
+
+	resource_list_for_each_entry(entry, &ci->resources)
+		remove_e820_regions(&device->dev, entry->res);
+
 	if (pci_use_crs) {
 		resource_list_for_each_entry_safe(entry, tmp, &ci->resources)
 			if (resource_is_pcicfg_ioport(entry->res))
Signature · 1 line
2.25.1
Reviewed-by on the v2 series
#5 lore raw
In reply to Bjorn Helgaas

Hi All,

Quoted 12 lines · Bjorn Helgaas

This is still work-in-progress on the issue of PNP0A03 _CRS methods that are buggy or not interpreted correctly by Linux.

The previous try at:
https://lore.kernel.org/r/[email protected]
caused regressions on some Chromebooks:
https://lore.kernel.org/r/[email protected]

This v2 drops the commit that caused the Chromebook regression, so it also doesn't fix the issue we were *trying* to fix on Lenovo Yoga and Clevo Barebones.

Quoted 8 lines

The point of this v2 update is to split the logging patch into (1) a pure logging addition and (2) the change to only clip PCI windows, which was previously hidden inside the logging patch and not well documented.

Bjorn Helgaas (3):
x86/PCI: Eliminate remove_e820_regions() common subexpressions
x86: Log resource clipping for E820 regions
x86/PCI: Clip only host bridge windows for E820 regions

Thanks, the entire series looks good to me:

For the series.

So what is the plan to actually fix the issue seen on some Lenovo models
and Clevo Barebones ? As I mentioned previously I think that since all
our efforts have failed so far that we should maybe reconsider just using DMI quirks to ignore the E820 reservation windows for host bridges on affected models ?

That or implement 3 of:

Quoted 11 lines

So I think the progression is:

1) Remove anything mentioned in E820 from _CRS (4dc2287c1805 [7]). This worked around some issues on Dell systems.

2) Remove things mentioned in E820 unless they cover the entire

    window (5949965ec934 [8])

3) Coalesce adjacent _CRS windows, *then* remove things mentioned in E820 unless they cover the entire (coalesced) window (current proposal)

Regards,

Hans

#6 lore raw
Quoted 17 lines · Hans de Goede
Quoted 16 lines · Bjorn Helgaas

This is still work-in-progress on the issue of PNP0A03 _CRS methods that are buggy or not interpreted correctly by Linux.

The previous try at:
https://lore.kernel.org/r/[email protected]
caused regressions on some Chromebooks:
https://lore.kernel.org/r/[email protected]

This v2 drops the commit that caused the Chromebook regression, so it also doesn't fix the issue we were *trying* to fix on Lenovo Yoga and Clevo Barebones.

The point of this v2 update is to split the logging patch into (1) a pure logging addition and (2) the change to only clip PCI windows, which was previously hidden inside the logging patch and not well documented.

Quoted 8 lines
Quoted 4 lines

Bjorn Helgaas (3):
x86/PCI: Eliminate remove_e820_regions() common subexpressions
x86: Log resource clipping for E820 regions
x86/PCI: Clip only host bridge windows for E820 regions

Thanks, the entire series looks good to me:

Thank you!

Quoted 5 lines

So what is the plan to actually fix the issue seen on some Lenovo models
and Clevo Barebones ? As I mentioned previously I think that since all
our efforts have failed so far that we should maybe reconsider just using DMI quirks to ignore the E820 reservation windows for host bridges on affected models ?

I have been resisting DMI quirks but I'm afraid there's no other way. I think the web we've gotten into, where vendors have used E820 to interact with _CRS in incompatible and undocumented ways, is not sustainable.

I'm not aware of any spec that says the OS should use E820 to clip things out of _CRS, so I think the long term plan should be to decouple them by default.

Straw man:

- Disable E820 clipping by default.

- Add a quirk to enable E820 clipping for machines older than X, e.g., 2023, to avoid breaking machines that currently work.

- Add quirks to disable E820 clipping for individual machines like the Lenovo and Clevos that predate X, but E820 clipping breaks them.

- Add quirks to enable E820 clipping for individual machines like the Chromebooks (and probably machines we don't know about yet) that have devices that consume part of _CRS but are not enumerable.

- Communicate this to OEMs to try to prevent future machines that need quirks.

Bjorn

#7 lore raw

Hi,

Quoted 29 lines · Bjorn Helgaas
Quoted 17 lines · Hans de Goede
Quoted 16 lines · Bjorn Helgaas

This is still work-in-progress on the issue of PNP0A03 _CRS methods that are buggy or not interpreted correctly by Linux.

The previous try at:
https://lore.kernel.org/r/[email protected]
caused regressions on some Chromebooks:
https://lore.kernel.org/r/[email protected]

This v2 drops the commit that caused the Chromebook regression, so it also doesn't fix the issue we were *trying* to fix on Lenovo Yoga and Clevo Barebones.

The point of this v2 update is to split the logging patch into (1) a pure logging addition and (2) the change to only clip PCI windows, which was previously hidden inside the logging patch and not well documented.

Quoted 8 lines
Quoted 4 lines

Bjorn Helgaas (3):
x86/PCI: Eliminate remove_e820_regions() common subexpressions
x86: Log resource clipping for E820 regions
x86/PCI: Clip only host bridge windows for E820 regions

Thanks, the entire series looks good to me:

Thank you!

Quoted 7 lines
Quoted 5 lines

So what is the plan to actually fix the issue seen on some Lenovo models
and Clevo Barebones ? As I mentioned previously I think that since all
our efforts have failed so far that we should maybe reconsider just using DMI quirks to ignore the E820 reservation windows for host bridges on affected models ?

I have been resisting DMI quirks but I'm afraid there's no other way.

Well there is the first match adjacent windows returned by _CRS and only then do the "covers whole region" exception check. I still think that would work at least for the chromebook regression...

So do you want me to give that a try; or shall I write a patch using DMI quirks. And if we go the DMI quirks, what about matching cmdline arguments? If we add matching cmdline arguments, which seems to be the sensible thing to do then to allow users to test if they need the quirk, then we basically end up with my first attempt at fixing this from 6 months ago:

https://lore.kernel.org/linux-pci/[email protected]/

Quoted 7 lines

I think the web we've gotten into, where vendors have used E820 to interact with _CRS in incompatible and undocumented ways, is not sustainable.

I'm not aware of any spec that says the OS should use E820 to clip things out of _CRS, so I think the long term plan should be to decouple them by default.

Right and AFAICT the reason Windows is getting away with this is the same as with the original Dell _CRS has overlap with physical RAM issue (1), Linux assigns address to unassigneds BAR-s starting with the lowest available address in the bridge window, where as Windows assigns addresses from the highest available address in the window. So the real fix here might very well be to rework the BAR assignment code to switch to fill the window from the top rather then from the bottom. AFAICT all issues where excluding _E820 reservations have helped are with _E820 - bridge window overlaps at the bottom of the window.

IOW these are really all bugs in the _CRS method for the bridge, which Windows does not hit because it never actually uses the lowest address(es) of the _CRS returned window.

Regards,

Hans

1) At least I read in either a bugzilla, or email thread about this that Windows allocating bridge window space from the top was assumed to be why Windows was not impacted.

Quoted 21 lines

Straw man:

- Disable E820 clipping by default.

- Add a quirk to enable E820 clipping for machines older than X, e.g., 2023, to avoid breaking machines that currently work.

- Add quirks to disable E820 clipping for individual machines like the Lenovo and Clevos that predate X, but E820 clipping breaks them.

- Add quirks to enable E820 clipping for individual machines like the Chromebooks (and probably machines we don't know about yet) that have devices that consume part of _CRS but are not enumerable.

- Communicate this to OEMs to try to prevent future machines that need quirks.

Bjorn

#8 lore raw
Quoted 38 lines · Hans de Goede

Hi,

Quoted 35 lines · Bjorn Helgaas
Quoted 17 lines · Hans de Goede
Quoted 16 lines · Bjorn Helgaas

This is still work-in-progress on the issue of PNP0A03 _CRS methods that are buggy or not interpreted correctly by Linux.

The previous try at:
https://lore.kernel.org/r/[email protected]
caused regressions on some Chromebooks:
https://lore.kernel.org/r/[email protected]

This v2 drops the commit that caused the Chromebook regression, so it also doesn't fix the issue we were *trying* to fix on Lenovo Yoga and Clevo Barebones.

The point of this v2 update is to split the logging patch into (1) a pure logging addition and (2) the change to only clip PCI windows, which was previously hidden inside the logging patch and not well documented.

Quoted 8 lines
Quoted 4 lines

Bjorn Helgaas (3):
x86/PCI: Eliminate remove_e820_regions() common subexpressions
x86: Log resource clipping for E820 regions
x86/PCI: Clip only host bridge windows for E820 regions

Thanks, the entire series looks good to me:

Thank you!

Quoted 5 lines

So what is the plan to actually fix the issue seen on some Lenovo models
and Clevo Barebones ? As I mentioned previously I think that since all
our efforts have failed so far that we should maybe reconsider just using DMI quirks to ignore the E820 reservation windows for host bridges on affected models ?

Quoted 5 lines
Quoted 1 line

I have been resisting DMI quirks but I'm afraid there's no other way.

Well there is the first match adjacent windows returned by _CRS and only then do the "covers whole region" exception check. I still think that would work at least for the chromebook regression...

Without a crystal clear strategy, I think we're going to be tweaking the algorithm forever as the _CRS/E820 mix changes. That's why I think that in the long term, a "use _CRS only, with quirks for exceptions" strategy will be simplest.

Quoted 8 lines

So do you want me to give that a try; or shall I write a patch using DMI quirks. And if we go the DMI quirks, what about matching cmdline arguments? If we add matching cmdline arguments, which seems to be the sensible thing to do then to allow users to test if they need the quirk, then we basically end up with my first attempt at fixing this from 6 months ago:

https://lore.kernel.org/linux-pci/[email protected]/

So I think we should go ahead with DMI quirks instead of trying to make the algorithm smarter, and yes, I think we will need commandline arguments, probably one to force E820 clipping for future machines, and one to disable it for old machines.

Quoted 8 lines
Quoted 7 lines

I think the web we've gotten into, where vendors have used E820 to interact with _CRS in incompatible and undocumented ways, is not sustainable.

I'm not aware of any spec that says the OS should use E820 to clip things out of _CRS, so I think the long term plan should be to decouple them by default.

Quoted 6 lines

Right and AFAICT the reason Windows is getting away with this is the same as with the original Dell _CRS has overlap with physical RAM issue (1), Linux assigns address to unassigneds BAR-s starting with the lowest available address in the bridge window, where as Windows assigns addresses from the highest available address in the window.

Right, I agree. I'm guessing Chromebooks don't get tested with Windows at all, so we don't even have that level of testing to help.

Quoted 9 lines

So the real fix here might very well be
to rework the BAR assignment code to switch to fill the window from the top rather then from the bottom. AFAICT all issues where excluding _E820 reservations have helped are with _E820 - bridge window overlaps at the bottom of the window.

IOW these are really all bugs in the _CRS method for the bridge, which Windows does not hit because it never actually uses the lowest address(es) of the _CRS returned window.

Yes. We actually did try this
(https://git.kernel.org/linus/1af3c2e45e7a), but unfortunately we had to revert it. Even more unfortunately, the revert
(https://git.kernel.org/linus/5e52f1c5e85f) doesn't have any details about what went wrong.

Quoted 26 lines

1) At least I read in either a bugzilla, or email thread about this that Windows allocating bridge window space from the top was assumed to be why Windows was not impacted.

Quoted 21 lines

Straw man:

- Disable E820 clipping by default.

- Add a quirk to enable E820 clipping for machines older than X, e.g., 2023, to avoid breaking machines that currently work.

- Add quirks to disable E820 clipping for individual machines like the Lenovo and Clevos that predate X, but E820 clipping breaks them.

- Add quirks to enable E820 clipping for individual machines like the Chromebooks (and probably machines we don't know about yet) that have devices that consume part of _CRS but are not enumerable.

- Communicate this to OEMs to try to prevent future machines that need quirks.

Bjorn

#9 lore raw

Hi,

Sorry for the late reply.

Quoted 41 lines · Bjorn Helgaas
Quoted 40 lines · Hans de Goede

Hi,

Quoted 36 lines · Bjorn Helgaas
Quoted 17 lines · Hans de Goede
Quoted 16 lines · Bjorn Helgaas

This is still work-in-progress on the issue of PNP0A03 _CRS methods that are buggy or not interpreted correctly by Linux.

The previous try at:
https://lore.kernel.org/r/[email protected]
caused regressions on some Chromebooks:
https://lore.kernel.org/r/[email protected]

This v2 drops the commit that caused the Chromebook regression, so it also doesn't fix the issue we were *trying* to fix on Lenovo Yoga and Clevo Barebones.

The point of this v2 update is to split the logging patch into (1) a pure logging addition and (2) the change to only clip PCI windows, which was previously hidden inside the logging patch and not well documented.

Quoted 8 lines
Quoted 4 lines

Bjorn Helgaas (3):
x86/PCI: Eliminate remove_e820_regions() common subexpressions
x86: Log resource clipping for E820 regions
x86/PCI: Clip only host bridge windows for E820 regions

Thanks, the entire series looks good to me:

Thank you!

Quoted 5 lines

So what is the plan to actually fix the issue seen on some Lenovo models
and Clevo Barebones ? As I mentioned previously I think that since all
our efforts have failed so far that we should maybe reconsider just using DMI quirks to ignore the E820 reservation windows for host bridges on affected models ?

I have been resisting DMI quirks but I'm afraid there's no other way.

Quoted 8 lines
Quoted 3 lines

Well there is the first match adjacent windows returned by _CRS and only then do the "covers whole region" exception check. I still think that would work at least for the chromebook regression...

Without a crystal clear strategy, I think we're going to be tweaking the algorithm forever as the _CRS/E820 mix changes. That's why I think that in the long term, a "use _CRS only, with quirks for exceptions" strategy will be simplest.

Looking at the amount of exception we already now about I'm not sure if that will work well.

Quoted 8 lines
Quoted 8 lines

So do you want me to give that a try; or shall I write a patch using DMI quirks. And if we go the DMI quirks, what about matching cmdline arguments? If we add matching cmdline arguments, which seems to be the sensible thing to do then to allow users to test if they need the quirk, then we basically end up with my first attempt at fixing this from 6 months ago:

Quoted 6 lines
Quoted 1 line

https://lore.kernel.org/linux-pci/[email protected]/

So I think we should go ahead with DMI quirks instead of trying to make the algorithm smarter, and yes, I think we will need commandline arguments, probably one to force E820 clipping for future machines, and one to disable it for old machines.

So what you are suggesting is to go back to a bios-date based approach (to determine old vs new machines) combined with DMI quirks to force E820 clipping on new machines which turn out to need it despite them being new ?

Quoted 29 lines
Quoted 9 lines
Quoted 8 lines

I think the web we've gotten into, where vendors have used E820 to interact with _CRS in incompatible and undocumented ways, is not sustainable.

I'm not aware of any spec that says the OS should use E820 to clip things out of _CRS, so I think the long term plan should be to decouple them by default.

Quoted 6 lines

Right and AFAICT the reason Windows is getting away with this is the same as with the original Dell _CRS has overlap with physical RAM issue (1), Linux assigns address to unassigneds BAR-s starting with the lowest available address in the bridge window, where as Windows assigns addresses from the highest available address in the window.

Right, I agree. I'm guessing Chromebooks don't get tested with Windows at all, so we don't even have that level of testing to help.

Quoted 9 lines

So the real fix here might very well be
to rework the BAR assignment code to switch to fill the window from the top rather then from the bottom. AFAICT all issues where excluding _E820 reservations have helped are with _E820 - bridge window overlaps at the bottom of the window.

IOW these are really all bugs in the _CRS method for the bridge, which Windows does not hit because it never actually uses the lowest address(es) of the _CRS returned window.

Quoted 5 lines

Yes. We actually did try this
(https://git.kernel.org/linus/1af3c2e45e7a), but unfortunately we had to revert it. Even more unfortunately, the revert
(https://git.kernel.org/linus/5e52f1c5e85f) doesn't have any details about what went wrong.

When I first started working on this I did read the entire old email thread and IIRC this approach was reverted because the e820 based approach was deemed to be a cleaner fix. Also the single resource_alloc_from_bottom flag influenced all types of resource allocations, not just PCI host bridge window allocations.

Note that the current kernel no longer has the resource_alloc_from_bottom flag. Still I think it might be worthwhile to give switching to top-down allocating for host bridge window allocs a try. Maybe we can make the desired allocation strategy a flag in the resource ?

I have the feeling that if we switch to top-down allocating that we can then switch to just using _CRS and that everything will then just work, because we then match what Windows is doing...

Regards,

Hans

Quoted 28 lines
Quoted 27 lines

1) At least I read in either a bugzilla, or email thread about this that Windows allocating bridge window space from the top was assumed to be why Windows was not impacted.

Quoted 21 lines

Straw man:

- Disable E820 clipping by default.

- Add a quirk to enable E820 clipping for machines older than X, e.g., 2023, to avoid breaking machines that currently work.

- Add quirks to disable E820 clipping for individual machines like the Lenovo and Clevos that predate X, but E820 clipping breaks them.

- Add quirks to enable E820 clipping for individual machines like the Chromebooks (and probably machines we don't know about yet) that have devices that consume part of _CRS but are not enumerable.

- Communicate this to OEMs to try to prevent future machines that need quirks.

Bjorn

#10 lore raw
Quoted 19 lines · Hans de Goede
Quoted 18 lines · Bjorn Helgaas
Quoted 16 lines · Hans de Goede
Quoted 10 lines · Bjorn Helgaas
Quoted 5 lines · Hans de Goede

So what is the plan to actually fix the issue seen on some
Lenovo models and Clevo Barebones ? As I mentioned previously
I think that since all our efforts have failed so far that we should maybe reconsider just using DMI quirks to ignore the E820 reservation windows for host bridges on affected models ?

I have been resisting DMI quirks but I'm afraid there's no other way.

Well there is the first match adjacent windows returned by _CRS and only then do the "covers whole region" exception check. I still think that would work at least for the chromebook regression...

Quoted 7 lines
Quoted 4 lines

Without a crystal clear strategy, I think we're going to be tweaking the algorithm forever as the _CRS/E820 mix changes. That's why I think that in the long term, a "use _CRS only, with quirks for exceptions" strategy will be simplest.

Looking at the amount of exception we already now about I'm not sure if that will work well.

It's possible that many quirks will be required. But I think in the long run the value of the simplest, most obvious strategy is huge. It's laid out in the spec already and it's the clearest way to agreement between firmware and OS. When we trip over something, it's very easy to determine whether _CRS is wrong or Linux is using it wrong. If we have to bring in question of looking at E820 entries, possibly merging them, using them or not based on overlaps ... that's a much more difficult conversation without a clear resolution.

Quoted 9 lines
Quoted 4 lines

So I think we should go ahead with DMI quirks instead of trying to make the algorithm smarter, and yes, I think we will need commandline arguments, probably one to force E820 clipping for future machines, and one to disable it for old machines.

So what you are suggesting is to go back to a bios-date based approach (to determine old vs new machines) combined with DMI quirks to force E820 clipping on new machines which turn out to need it despite them being new ?

Yes. It's ugly but I think the 10-year outlook is better.

Quoted 3 lines

I have the feeling that if we switch to top-down allocating that we can then switch to just using _CRS and that everything will then just work, because we then match what Windows is doing...

Yes, it might. But I'm not 100% comfortable because it basically sweeps _CRS bugs under the rug, and we may trip over them as we do more hotplug and (eventually) resource rebalancing. I think we need to work toward getting _CRS more reliable.

Bjorn

#11 lore raw

Hi,

Quoted 42 lines · Bjorn Helgaas
Quoted 19 lines · Hans de Goede
Quoted 18 lines · Bjorn Helgaas
Quoted 16 lines · Hans de Goede
Quoted 10 lines · Bjorn Helgaas
Quoted 5 lines · Hans de Goede

So what is the plan to actually fix the issue seen on some
Lenovo models and Clevo Barebones ? As I mentioned previously
I think that since all our efforts have failed so far that we should maybe reconsider just using DMI quirks to ignore the E820 reservation windows for host bridges on affected models ?

I have been resisting DMI quirks but I'm afraid there's no other way.

Well there is the first match adjacent windows returned by _CRS and only then do the "covers whole region" exception check. I still think that would work at least for the chromebook regression...

Quoted 7 lines
Quoted 4 lines

Without a crystal clear strategy, I think we're going to be tweaking the algorithm forever as the _CRS/E820 mix changes. That's why I think that in the long term, a "use _CRS only, with quirks for exceptions" strategy will be simplest.

Looking at the amount of exception we already now about I'm not sure if that will work well.

It's possible that many quirks will be required. But I think in the long run the value of the simplest, most obvious strategy is huge. It's laid out in the spec already and it's the clearest way to agreement between firmware and OS. When we trip over something, it's very easy to determine whether _CRS is wrong or Linux is using it wrong. If we have to bring in question of looking at E820 entries, possibly merging them, using them or not based on overlaps ... that's a much more difficult conversation without a clear resolution.

Quoted 5 lines
Quoted 4 lines

So I think we should go ahead with DMI quirks instead of trying to make the algorithm smarter, and yes, I think we will need commandline arguments, probably one to force E820 clipping for future machines, and one to disable it for old machines.

Quoted 6 lines
Quoted 4 lines

So what you are suggesting is to go back to a bios-date based approach (to determine old vs new machines) combined with DMI quirks to force E820 clipping on new machines which turn out to need it despite them being new ?

Yes. It's ugly but I think the 10-year outlook is better.

Ok, I've brushed off one of my earlier patches doing this and added DMI quirks for the "Lenovo X1 Carbon 2nd gen" suspend issue + the Asus C523NA / Google Coral Chromebook not booting issues which we already know will get triggered by this based on earlier testing.

I'll send this out right after this email.

Note this new patch is based on top of your:

https://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git/log/?h=pci/resource

Quoted 8 lines
Quoted 3 lines

I have the feeling that if we switch to top-down allocating that we can then switch to just using _CRS and that everything will then just work, because we then match what Windows is doing...

Yes, it might. But I'm not 100% comfortable because it basically sweeps _CRS bugs under the rug, and we may trip over them as we do more hotplug and (eventually) resource rebalancing. I think we need to work toward getting _CRS more reliable.

Ok.

Regards,

Hans