Skip to contentSkip to conversation

x86-64: improve e820_search_gap()

PATCH In review Jan Beulich 3 participants 12 messages May 6, 2009 – May 10, 2009 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86-64: improve e820_search_gap()

Impact: bug fix

Blindly putting the gap close after max_pfn is in conflict with that same memory range potentially being used by hotplugged memory.

Also, make the function static to ensure there are no other users that could depend on the previous behavior regarding the way start_addr gets specified.

---
 arch/x86/include/asm/e820.h |    2 --
 arch/x86/kernel/e820.c      |   11 +++++++++--
 2 files changed, 9 insertions(+), 4 deletions(-)
arch/x86/include/asm/e820.h +0 -2
@@ -79,8 +79,6 @@ extern u64 e820_remove_range(u64 start, 
 			     int checktype);
 extern void update_e820(void);
 extern void e820_setup_gap(void);
-extern int e820_search_gap(unsigned long *gapstart, unsigned long *gapsize,
-			unsigned long start_addr, unsigned long long end_addr);
 struct setup_data;
 extern void parse_e820_ext(struct setup_data *data, unsigned long pa_data);
 
arch/x86/kernel/e820.c +9 -2
@@ -574,13 +574,19 @@ static void __init update_e820_saved(voi
 /*
  * Search for a gap in the e820 memory space from start_addr to end_addr.
  */
-__init int e820_search_gap(unsigned long *gapstart, unsigned long *gapsize,
+static int __init
+e820_search_gap(unsigned long *gapstart, unsigned long *gapsize,
 		unsigned long start_addr, unsigned long long end_addr)
 {
 	unsigned long long last;
 	int i = e820.nr_map;
 	int found = 0;
 
+#ifdef CONFIG_X86_64
+	if (start_addr >= MAX_GAP_END)
+		last = end_addr ?: (1UL << boot_cpu_data.x86_phys_bits);
+	else
+#endif
 	last = (end_addr && end_addr < MAX_GAP_END) ? end_addr : MAX_GAP_END;
 
 	while (--i >= 0) {
@@ -626,11 +632,12 @@ __init void e820_setup_gap(void)
 
 #ifdef CONFIG_X86_64
 	if (!found) {
-		gapstart = (max_pfn << PAGE_SHIFT) + 1024*1024;
 		printk(KERN_ERR "PCI: Warning: Cannot find a gap in the 32bit "
 		       "address range\n"
 		       KERN_ERR "PCI: Unassigned devices with 32bit resource "
 		       "registers may break!\n");
+		found = e820_search_gap(&gapstart, &gapsize, MAX_GAP_END, 0);
+		WARN_ON(!found);
 	}
 #endif
 
#1 lore raw
Quoted 11 lines · Jan Beulich

Impact: bug fix

Blindly putting the gap close after max_pfn is in conflict with that same memory range potentially being used by hotplugged memory.

Also, make the function static to ensure there are no other users that could depend on the previous behavior regarding the way start_addr gets specified.

So blindly locate it somewhere else? How is that inherently better? Wouldn't a machine with hotplug memory (which doesn't bother advertising that fact so we can reserve the address space) be just as likely to use a sparse memory space, since one can hardly expect the hardware to pack the space (packing in hardware is why PCs generally have a mostly-contiguous RAM space) when the memory is hotplugged?

I think I'm missing something fundamental...

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#2 lore raw
Quoted 14 lines
Quoted 1 line
Quoted 1 line

"H. Peter Anvin" <[email protected]> 08.05.09 06:59 >>>

Quoted 11 lines · Jan Beulich

Impact: bug fix

Blindly putting the gap close after max_pfn is in conflict with that same memory range potentially being used by hotplugged memory.

Also, make the function static to ensure there are no other users that could depend on the previous behavior regarding the way start_addr gets specified.

Quoted 6 lines

So blindly locate it somewhere else? How is that inherently better? Wouldn't a machine with hotplug memory (which doesn't bother advertising that fact so we can reserve the address space) be just as likely to use a sparse memory space, since one can hardly expect the hardware to pack the space (packing in hardware is why PCs generally have a mostly-contiguous RAM space) when the memory is hotplugged?

Why blindly? Aren't hotpluggable memory ranges supposed to be reserved in the E820 map?

Jan

#3 lore raw
Quoted 11 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line

So blindly locate it somewhere else? How is that inherently better? Wouldn't a machine with hotplug memory (which doesn't bother advertising that fact so we can reserve the address space) be just as likely to use a sparse memory space, since one can hardly expect the hardware to pack the space (packing in hardware is why PCs generally have a mostly-contiguous RAM space) when the memory is hotplugged?

Why blindly? Aren't hotpluggable memory ranges supposed to be reserved in the E820 map?

That's was my reaction too... so I guess I'm misreading the patch. Could you perhaps add a bit more to the description, specifically of what policy your patch implements.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#4 lore raw
In reply to Jan Beulich
Quoted 3 lines · Jan Beulich

Why blindly? Aren't hotpluggable memory ranges supposed to be reserved in the E820 map?

They are supposed to be reserved in SRAT, but not in e820.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#5 lore raw
Quoted 5 lines · Andi Kleen
Quoted 2 lines · Jan Beulich

Why blindly? Aren't hotpluggable memory ranges supposed to be reserved in the E820 map?

They are supposed to be reserved in SRAT, but not in e820.

Ah, okay. Perhaps we should fold this information into our internal "e820" map? If so I guess the question is how soon we can do that.

-hpa

#6 lore raw
Quoted 9 lines · H. Peter Anvin
Quoted 5 lines · Andi Kleen
Quoted 2 lines · Jan Beulich

Why blindly? Aren't hotpluggable memory ranges supposed to be reserved in the E820 map?

They are supposed to be reserved in SRAT, but not in e820.

Ah, okay. Perhaps we should fold this information into our internal "e820" map? If so I guess the question is how soon we can do that.

You could do that or just query SRAT too, but ...

The problem is really that there are still systems which have hidden holes which are not reserved anywhere. When I last hacked on the gap algorithm it triggered subtle bugs. So this will never be fully reliable.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#7 lore raw
Quoted 5 lines · Andi Kleen
Quoted 4 lines · H. Peter Anvin
Quoted 3 lines · Andi Kleen
Quoted 2 lines · Jan Beulich

Why blindly? Aren't hotpluggable memory ranges supposed to be reserved in the E820 map?

Quoted 9 lines
Quoted 2 lines
Quoted 1 line

They are supposed to be reserved in SRAT, but not in e820.

Ah, okay. Perhaps we should fold this information into our internal "e820" map? If so I guess the question is how soon we can do that.

You could do that or just query SRAT too, but ...

The problem is really that there are still systems which have hidden holes which are not reserved anywhere. When I last hacked on the gap algorithm it triggered subtle bugs. So this will never be fully reliable.

That's without a question.

However, as far as querying SRAT, I don't like the idea of spreading the knowledge of the system memory map out between a bunch of different places, each of which have a little piece of the puzzle. It puts a huge onus on the user to know what mechanisms are actually available, and really makes a shitty interface.

-hpa

#8 lore raw
Quoted 5 lines

However, as far as querying SRAT, I don't like the idea of spreading the knowledge of the system memory map out between a bunch of different places, each of which have a little piece of the puzzle. It puts a huge onus on the user to know what mechanisms are actually available, and really makes a shitty interface.

AFAIK another popular OS always combines mappings from all sources (e820, SRAT, PCI, PNP, ACPI etc.) in the query before allocating anything. Something like that might be a reasonable long term direction for Linux too, but it's probably also a can of worms to handle the conflicts between the various sources (e.g. e820 reserves a lot of things in other sources too). It would be a rather large change. Maybe that would handle the systems I thought of above.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#9 lore raw
Quoted 6 lines · Andi Kleen
Quoted 5 lines

However, as far as querying SRAT, I don't like the idea of spreading the knowledge of the system memory map out between a bunch of different places, each of which have a little piece of the puzzle. It puts a huge onus on the user to know what mechanisms are actually available, and really makes a shitty interface.

Quoted 7 lines

AFAIK another popular OS always combines mappings from all sources (e820, SRAT, PCI, PNP, ACPI etc.) in the query before allocating anything. Something like that might be a reasonable long term direction for Linux too, but it's probably also a can of worms to handle the conflicts between the various sources (e.g. e820 reserves a lot of things in other sources too). It would be a rather large change. Maybe that would handle the systems I thought of above.

You *always* have a conflict resolution policy... whether or not it is explicit or accidental, and whether or not it is the result of merging the data or just accessing multiple data sources is another matter. It might be hard to replicate an accidental policy in an explicit way, or the accidental policy really might make no sense, which may mean behavior changes.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#10 lore raw
Quoted 15 lines · H. Peter Anvin
Quoted 6 lines · Andi Kleen
Quoted 5 lines

However, as far as querying SRAT, I don't like the idea of spreading the knowledge of the system memory map out between a bunch of different places, each of which have a little piece of the puzzle. It puts a huge onus on the user to know what mechanisms are actually available, and really makes a shitty interface.

Quoted 7 lines

AFAIK another popular OS always combines mappings from all sources (e820, SRAT, PCI, PNP, ACPI etc.) in the query before allocating anything. Something like that might be a reasonable long term direction for Linux too, but it's probably also a can of worms to handle the conflicts between the various sources (e.g. e820 reserves a lot of things in other sources too). It would be a rather large change. Maybe that would handle the systems I thought of above.

Quoted 1 line

You *always* have a conflict resolution policy... whether or not it is

Well right now we ignore ACPI/PNP data, sometimes ignore PCI data and only look in e820 and sometimes only use PCI/e820 and sometimes only use SRAT (or at least it was like this at some point)

BTW SRAT hot range can be also wrong, some BIOS always had a full 512GB range even though they don't support hotplug.
However I haven't seen a system which supports hotplug (not that there are very many of those) where the entry was not there, so at least that one should be safe.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#11 lore raw
Quoted 5 lines · Andi Kleen

Well right now we ignore ACPI/PNP data, sometimes ignore PCI data and only look in e820 and sometimes only use PCI/e820 and sometimes only use SRAT (or at least it was like this at some point)

That *is* a conflict resolution policy. Quite probably an accidental one.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.