fix for hot-add enabled SRAT/BIOS and numa KVA areas

RFC In review keith 3 participants 10 messages Nov 17, 2004 – Nov 18, 2004 Memory management

Share & tools

keith

author

2004-11-17 02:37 UTC #0 lore raw

RFC 1/1 fix for hot-add enabled SRAT/BIOS and numa KVA areas

My numa hardware (IBM x445 summit based) supports hot-add memory.
When this feature is enabled in the bios and booted as with CONFIG_NUMA I get a memory range exposed by the SRAT/ACPI parsing. This range express the amount of memory the "could" be added to the system.

This chunk extends from the end of physical memory to the end of the
i386 address space. If the following my physical memory is 0x2C0000.

(From the boot messages)
Memory range 0x80000 to 0xC0000 (type 0x0) in proximity domain 0x01 enabled Memory range 0x100000 to 0x2C0000 (type 0x0) in proximity domain 0x01 enabled Memory range 0x2C0000 to 0x1000000 (type 0x0) in proximity domain 0x01 enabled and removable

These memory ranges I believe to be valid according to what I know
about the SRAT and the ACPI 2.0c specs. (I am not an ACPI expert please correct me if I am wrong!)

The numa KVA code used the node_start and node_end values (obtained
from the above memory ranges) to make it's lowmem reservations. The problem is that the lowmem area reserved is quite large. It reserves the entire a lmem_map large enough for 0x1000000 address space. I don't feel this is a great use of lowmem on my system :)

Thankfully as we know the e820 shows what memory is really in the
system. My simple fix it to find the max_pfn from the e820 earlier and set the numa KVA areas accordingly.

Don't trust the SRAT for this info only the e820.

Thanks,
Keith Mannthey

arch/i386/mm/discontig.c +10 -1

diff -urN linux-2.6.9/arch/i386/mm/discontig.c linux-2.6.9-fix2/arch/i386/mm/discontig.c
@@ -199,6 +199,15 @@
 	unsigned long size, reserve_pages = 0;
 
 	for (nid = 1; nid < numnodes; nid++) {
+		/*
+		* The acpi/srat node info can show hot-add memroy zones
+		* where memory could be added but not currently present.
+		*/
+		if (node_start_pfn[nid] > max_pfn)
+			continue;
+		if (node_end_pfn[nid] > max_pfn)
+			node_end_pfn[nid] = max_pfn;
+
 		/* calculate the size of the mem_map needed in bytes */
 		size = (node_end_pfn[nid] - node_start_pfn[nid] + 1) 
 			* sizeof(struct page) + sizeof(pg_data_t);

@@ -261,12 +270,12 @@
 		printk("\n");
 	}
 
+	find_max_pfn();
 	reserve_pages = calculate_numa_remap_pages();
 
 	/* partially used pages are not usable - thus round upwards */
 	system_start_pfn = min_low_pfn = PFN_UP(init_pg_tables_end);
 
-	find_max_pfn();
 	system_max_low_pfn = max_low_pfn = find_max_low_pfn() - reserve_pages;
 	printk("reserve_pages = %ld find_max_low_pfn() ~ %ld\n",
 			reserve_pages, max_low_pfn + reserve_pages);

Dave Hansen

2004-11-17 17:12 UTC #1 lore raw

Quoted 5 lines · keith

The numa KVA code used the node_start and node_end values (obtained
from the above memory ranges) to make it's lowmem reservations. The problem is that the lowmem area reserved is quite large. It reserves the entire a lmem_map large enough for 0x1000000 address space. I don't feel this is a great use of lowmem on my system :)

It does seem silly to waste all of that lowmem for memory that *might* be there, but what do you plan to do for contiguous address space (for mem_map) once the memory addition occurs? We've always talked about having to preallocate mem_map space on 32-bit platforms and by your patch it appears that this isn't what you want to do.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

keith

author

2004-11-18 02:08 UTC #2 lore raw

Quoted 7 lines · Dave Hansen

Quoted 5 lines · keith
The numa KVA code used the node_start and node_end values (obtained
from the above memory ranges) to make it's lowmem reservations. The problem is that the lowmem area reserved is quite large. It reserves the entire a lmem_map large enough for 0x1000000 address space. I don't feel this is a great use of lowmem on my system :)

Quoted 7 lines

It does seem silly to waste all of that lowmem for memory that *might* be there, but what do you plan to do for contiguous address space (for mem_map) once the memory addition occurs? We've always talked about having to preallocate mem_map space on 32-bit platforms and by your patch it appears that this isn't what you want to do.
-- Dave

I am not anticipating to support hot-add without config_nonlinear or
something similar which should provide more flexibility in allocation of smaller section mem_maps. This is only a issue when booted as a discontig system. We don't even consult the SRAT when we boot flat (contiguous address space) so it is a non-issue.

Wasting 500k of lowmem for memory that "might" be there is no good. I
don't think having to preallocate the mem_map for a hot-add is really that good. What if the system never adds memory? What if it only adds 8gig not 49g? The system is crippled because it reserves the lmem_map it "might" do a hot add with?

I forgot the mention that without this patch my system does not boot
with the hot-add support enabled in the bios.

Thanks,
Keith

Dave Hansen

2004-11-18 02:24 UTC #3 lore raw

Quoted 5 lines · keith

I am not anticipating to support hot-add without config_nonlinear or
something similar which should provide more flexibility in allocation of smaller section mem_maps. This is only a issue when booted as a discontig system. We don't even consult the SRAT when we boot flat (contiguous address space) so it is a non-issue.

Once a system has been running for any length of time, finding any multi-order pages gets somewhat hard. For a 16M section, you're still talking about ~128k of mem_map, which is still an order 5 allocation. Nick's kswapd higher-order patches should help with this, though.

Quoted 5 lines

Wasting 500k of lowmem for memory that "might" be there is no good. I
don't think having to preallocate the mem_map for a hot-add is really that good. What if the system never adds memory? What if it only adds 8gig not 49g? The system is crippled because it reserves the lmem_map it "might" do a hot add with?

I have the feeling we'll eventually need a boot-time option for this reservation. Your patch, of course will work for now. Do you want me to pick it up in my tree?

Quoted 2 lines

I forgot the mention that without this patch my system does not boot
with the hot-add support enabled in the bios.

Why not? I'm just curious what caused the actual failure.

-- Dave

keith

author

2004-11-18 19:18 UTC #4 lore raw

Quoted 11 lines · Dave Hansen

Quoted 5 lines · keith
I am not anticipating to support hot-add without config_nonlinear or
something similar which should provide more flexibility in allocation of smaller section mem_maps. This is only a issue when booted as a discontig system. We don't even consult the SRAT when we boot flat (contiguous address space) so it is a non-issue.
Once a system has been running for any length of time, finding any multi-order pages gets somewhat hard. For a 16M section, you're still talking about ~128k of mem_map, which is still an order 5 allocation. Nick's kswapd higher-order patches should help with this, though.

I think it is safe to deal with this issue later. I think i386 hot add is going to be confined to a very small pool of boxes.

Quoted 9 lines

Quoted 5 lines
Wasting 500k of lowmem for memory that "might" be there is no good. I
don't think having to preallocate the mem_map for a hot-add is really that good. What if the system never adds memory? What if it only adds 8gig not 49g? The system is crippled because it reserves the lmem_map it "might" do a hot add with?
I have the feeling we'll eventually need a boot-time option for this reservation. Your patch, of course will work for now. Do you want me to pick it up in my tree?

I agree a boot time option to reserve a chunk of virtual address space may be the way to go for i386. I would hate to reserve too much space that ends up just being wasted. I think this can be save off for future work. I would like to get it into mm, I'll ping Andrew.

Quoted 7 lines

Quoted 3 lines
I forgot the mention that without this patch my system does not boot
with the hot-add support enabled in the bios.
Why not? I'm just curious what caused the actual failure.
-- Dave

(snip)
Summit chipset: Starting Cyclone Counter.
Detected 1997.351 MHz processor.
Using cyclone for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) Initializing HighMem for node 0
Initializing HighMem for node 1
Bad page state at free_hot_cold_page (in process 'swapper', page d8e0d000)
flags:0xfffff7ff mapping:00000000 mapcount:0 count:266321921

Backtrace:
 [<c0140e09>] bad_page+0x81/0xb4
 [<c01415fd>] free_hot_cold_page+0x7e/0x100
 [<c0492849>] one_highpage_init+0xda/0x172
 [<c0493842>] set_highmem_pages_init+0xb0/0xf5
 [<c03691eb>] _etext+0x0/0x10d5
 [<c0492dcb>] mem_init+0x14f/0x2ba
 [<c0495c78>] alloc_large_system_hash+0x128/0x173
 [<c04827c7>] start_kernel+0x118/0x221
 [<c048231a>] unknown_bootoption+0x0/0x145

Trying to fix it up, but a reboot is needed
Bad page state at free_hot_cold_page (in process 'swapper', page d8e0d020)
flags:0xfffff7ff mapping:00000000 mapcount:0 count:-523233279

Backtrace:
 [<c0140e09>] bad_page+0x81/0xb4
 [<c01415fd>] free_hot_cold_page+0x7e/0x100
 [<c0492849>] one_highpage_init+0xda/0x172
 [<c0493842>] set_highmem_pages_init+0xb0/0xf5
 [<c03691eb>] _etext+0x0/0x10d5
 [<c0492dcb>] mem_init+0x14f/0x2ba
 [<c0495c78>] alloc_large_system_hash+0x128/0x173
 [<c04827c7>] start_kernel+0x118/0x221
 [<c048231a>] unknown_bootoption+0x0/0x145

Trying to fix it up, but a reboot is needed
Bad page state at free_hot_cold_page (in process 'swapper', page d8e0d040)
flags:0xfffff7ff mapping:00000000 mapcount:0 count:-523233279

Backtrace:
 [<c0140e09>] bad_page+0x81/0xb4
 [<c01415fd>] free_hot_cold_page+0x7e/0x100
 [<c0492849>] one_highpage_init+0xda/0x172

(and on and on and on)

Thanks,
Keith Mannthey

Yasunori Goto

2004-11-17 22:34 UTC #5 lore raw

In reply to keith

Hello, Keith-san.

Quoted 11 lines

This chunk extends from the end of physical memory to the end of the
i386 address space. If the following my physical memory is 0x2C0000.
(From the boot messages)
Memory range 0x80000 to 0xC0000 (type 0x0) in proximity domain 0x01 enabled Memory range 0x100000 to 0x2C0000 (type 0x0) in proximity domain 0x01 enabled Memory range 0x2C0000 to 0x1000000 (type 0x0) in proximity domain 0x01 enabled and removable
These memory ranges I believe to be valid according to what I know
about the SRAT and the ACPI 2.0c specs. (I am not an ACPI expert please correct me if I am wrong!)

I also think this is valid. Probably the firmware of x445 thought, if enabled bit of SRAT is off, any other information of its area will not be trusted.
So, there is no way to distinguish the machine can't attach more memory from it can do it (just there is no memory AT BOOT TIME.)
So, third area in this boot message just indicates "possibility" of hotadd memory.
But e820 probably indicates just memory areas which
are already connected on the board, right?

(IIRC, there is no mention about if enable bit of SRAT is off
in SRAT spec.)

BTW, I have a question.
- Can x445 be attached memory without removing the node? In my concern machine, there is no physical space to hot add or exchange memory without physical removing the node. But, this SRAT table indicate that all of proximity is 0x01....
Or is it just logical attachment?

Thanks.

Signature · 8 lines

Yasunori Goto <ygoto at us.fujitsu.com>


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dave Hansen

2004-11-17 22:42 UTC #6 lore raw

Quoted 2 lines · Yasunori Goto

But e820 probably indicates just memory areas which are already connected on the board, right?

It's more than that. It indicates which were connected the first time that the machine was powered on. If you suspend or hibernate the system for some reason, it has to always present the e820 as it initially appeared.

Quoted 7 lines

BTW, I have a question.
- Can x445 be attached memory without removing the node? In my concern machine, there is no physical space to hot add or exchange memory without physical removing the node. But, this SRAT table indicate that all of proximity is 0x01....
Or is it just logical attachment?

You can't remove nodes, just DIMMs. The x440 hotplug is more like the SMP case that I've always been concerned with.

-- Dave

Yasunori Goto

2004-11-17 23:21 UTC #7 lore raw

Quoted 2 lines

You can't remove nodes, just DIMMs. The x440 hotplug is more like the SMP case that I've always been concerned with.

OK. I understood.
Thanks.

Signature · 8 lines

Yasunori Goto <ygoto at us.fujitsu.com>


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

keith

author

2004-11-18 02:18 UTC #8 lore raw

In reply to Dave Hansen

Quoted 8 lines · Dave Hansen

Quoted 2 lines · Yasunori Goto
But e820 probably indicates just memory areas which are already connected on the board, right?
It's more than that. It indicates which were connected the first time that the machine was powered on. If you suspend or hibernate the system for some reason, it has to always present the e820 as it initially appeared.

You use the acpi events to handle the addition of memory. The e820 is what you use to boot with.

Quoted 11 lines

Quoted 7 lines
BTW, I have a question.
- Can x445 be attached memory without removing the node? In my concern machine, there is no physical space to hot add or exchange memory without physical removing the node. But, this SRAT table indicate that all of proximity is 0x01....
Or is it just logical attachment?
You can't remove nodes, just DIMMs. The x440 hotplug is more like the SMP case that I've always been concerned with.
-- Dave

My hardware only supports addition of memory not removal.

Thanks,
Keith Mannthey

keith

author

2004-11-18 02:16 UTC #9 lore raw

In reply to Yasunori Goto

Quoted 14 lines · Yasunori Goto

Hello, Keith-san.
Quoted 11 lines
This chunk extends from the end of physical memory to the end of the
i386 address space. If the following my physical memory is 0x2C0000.
(From the boot messages)
Memory range 0x80000 to 0xC0000 (type 0x0) in proximity domain 0x01 enabled Memory range 0x100000 to 0x2C0000 (type 0x0) in proximity domain 0x01 enabled Memory range 0x2C0000 to 0x1000000 (type 0x0) in proximity domain 0x01 enabled and removable
These memory ranges I believe to be valid according to what I know
about the SRAT and the ACPI 2.0c specs. (I am not an ACPI expert please correct me if I am wrong!)

Quoted 9 lines

I also think this is valid. Probably the firmware of x445 thought, if enabled bit of SRAT is off, any other information of its area will not be trusted.
So, there is no way to distinguish the machine can't attach more memory from it can do it (just there is no memory AT BOOT TIME.)
So, third area in this boot message just indicates "possibility" of hotadd memory.
But e820 probably indicates just memory areas which
are already connected on the board, right?

This is what I believe. The e820 is the only true source of how much memory is present in the system. The SRAT just shows what the memory layout is. The SRAT is correct there is the possibility of memory being in this zone.

BTW: I can't toggle the SRAT enable bit in my bios. I get it all the time.

Quoted 10 lines

(IIRC, there is no mention about if enable bit of SRAT is off
in SRAT spec.)
BTW, I have a question.
- Can x445 be attached memory without removing the node? In my concern machine, there is no physical space to hot add or exchange memory without physical removing the node. But, this SRAT table indicate that all of proximity is 0x01....
Or is it just logical attachment?

My memory add works as follows. I can physically add dimms in one of my nodes. I do not have to remove any node(cec) from my system. Just open up the case in insert dmmms into a empty memory bank.

Thanks,
Keith Mannthey