Skip to contentSkip to conversation

x86: early boot crash: "alloc_low_page: ran out of memory" (bisected)

PATCH In review Dave Hansen 4 participants 11 messages Aug 9, 2013 – Aug 20, 2013

Subsystems (3)
Stable branch x86 x86 MM
lore
Share & tools
lore

Dave Hansen

author
#0 lore raw

I'm getting a 100% reproducible panic early in boot:

Quoted 1 line
[    0.000000] Kernel panic - not syncing: alloc_low_page: ran out of memory

I'm not sure why I didn't run in to this until now. I think there are a couple of config options that need to get set just right to trigger it, but CONFIG_DEBUG_PAGEALLOC seems to be the main one. Full config is here:

	http://sr71.net/~dave/intel/foo/config-bigbox-crash-20130809.txt

I bisected it back to this commit (which I seem to remember causing some other probems):

Quoted 5 lines

commit 8170e6bed465b4b0c7687f93e9948aca4358a33b

Author: H. Peter Anvin <[email protected]>
Date:   Thu Jan 24 12:19:52 2013 -0800

x86, 64bit: Use a #PF handler to materialize early mappings on demand

I need somewhere between 500G and 600G of memory to trigger it, but it can be triggered using qemu with much less _actual_ RAM than that. From looking at the dmesg diffs, I suspect that the delta in memory use between using 1G and 4k ptes for the identity mapping (DEBUG_PAGEALLOC forces 4k pages) is the proximate trigger.

I also suspect that alloc_low_pages() is buggy in the way it manipulates min/max_pfn_mapped. I'm quite baffled how 'max_pfn_mapped' is supposed to get set up correctly. Current code says:

max_pfn_mapped = 0; /* will get exact value next */

but I certainly don't see it getting set later on in that function, or _ever_ as adding some printk()'s shows:

Quoted 8 lines
+[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
+[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 0 max_pfn_mapped: 0
+[    0.000000] BRK [0x02086000, 0x02086fff] PGTABLE
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 0 max_pfn_mapped: 0
+[    0.000000] BRK [0x02087000, 0x02087fff] PGTABLE
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 0 max_pfn_mapped: 0
+[    0.000000] BRK [0x02088000, 0x02088fff] PGTABLE
Quoted 8 lines
+[    0.000000] init_memory_mapping: [mem 0xf07fe00000-0xf07fffffff]
+[    0.000000]  [mem 0xf07fe00000-0xf07fffffff] page 4k
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 252182528 max_pfn_mapped: 0
+[    0.000000] BRK [0x02089000, 0x02089fff] PGTABLE
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 252182528 max_pfn_mapped: 0
+[    0.000000] BRK [0x0208a000, 0x0208afff] PGTABLE
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 252182528 max_pfn_mapped: 0
+[    0.000000] Kernel panic - not syncing: alloc_low_page: ran out of memory

I'll take a closer look at it next week, but figured I'd report it first.

Full dmesg:

Quoted 66 lines

early console in setup code

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.8.0-rc5-00059-g8170e6b ([email protected]) (gcc version 4.6.3 20120306 (Red Hat 4.6.3-2) (GCC) ) #29 SMP Fri Aug 9 15:56:12 PDT 2013
[    0.000000] Command line: root=/dev/sda1 console=ttyS0,115200 earlyprintk=ttyS0,115200 debug
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f3ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009f400-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000dfffbfff] usable
[    0.000000] BIOS-e820: [mem 0x00000000dfffc000-0x00000000dfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000929bffffff] usable
[    0.000000] bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] DMI: Bochs Bochs, BIOS Bochs 01/01/2007
[    0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0x929c000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 00E0000000 mask FFE0000000 uncachable
[    0.000000]   1 disabled
[    0.000000]   2 disabled
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] PAT not supported by CPU.
[    0.000000] e820: last_pfn = 0xdfffc max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000fdb00-0x000fdb0f] mapped at [ffff8800000fdb00]
[    0.000000] initial memory mapped: [mem 0x00000000-0xffffffffffffffff]
[    0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x0205a000, 0x0205afff] PGTABLE
[    0.000000] BRK [0x0205b000, 0x0205bfff] PGTABLE
[    0.000000] BRK [0x0205c000, 0x0205cfff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x929be00000-0x929bffffff]
[    0.000000]  [mem 0x929be00000-0x929bffffff] page 4k
[    0.000000] BRK [0x0205d000, 0x0205dfff] PGTABLE
[    0.000000] BRK [0x0205e000, 0x0205efff] PGTABLE
[    0.000000] Kernel panic - not syncing: alloc_low_page: ran out of memory
[    0.000000] Pid: 0, comm: swapper Not tainted 3.8.0-rc5-00059-g8170e6b #29
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff81639b47>] panic+0xbb/0x1cb
[    0.000000]  [<ffffffff816257aa>] alloc_low_pages+0x15a/0x160
[    0.000000]  [<ffffffff81634d46>] phys_pmd_init+0x1f1/0x290
[    0.000000]  [<ffffffff81634fb7>] phys_pud_init+0x1d2/0x24f
[    0.000000]  [<ffffffff81635132>] kernel_physical_mapping_init+0xfe/0x16e
[    0.000000]  [<ffffffff81625993>] init_memory_mapping+0x1e3/0x350
[    0.000000]  [<ffffffff81cf5c5d>] init_range_memory_mapping+0xc2/0x10b
[    0.000000]  [<ffffffff81cf5dd9>] init_mem_mapping+0x133/0x1c8
[    0.000000]  [<ffffffff81ce77ad>] setup_arch+0x6ef/0xbe4
[    0.000000]  [<ffffffff81639ca4>] ? printk+0x4d/0x4f
[    0.000000]  [<ffffffff81ce3b4d>] start_kernel+0xce/0x3b3
[    0.000000]  [<ffffffff81ce3592>] x86_64_start_reservations+0x91/0x95
[    0.000000]  [<ffffffff81ce3681>] x86_64_start_kernel+0xeb/0xf2
#1 lore raw
Quoted 51 lines · Dave Hansen

I'm getting a 100% reproducible panic early in boot:

Quoted 1 line
[    0.000000] Kernel panic - not syncing: alloc_low_page: ran out of memory

I'm not sure why I didn't run in to this until now. I think there are a couple of config options that need to get set just right to trigger it, but CONFIG_DEBUG_PAGEALLOC seems to be the main one. Full config is here:

        http://sr71.net/~dave/intel/foo/config-bigbox-crash-20130809.txt

I bisected it back to this commit (which I seem to remember causing some other probems):

Quoted 5 lines

commit 8170e6bed465b4b0c7687f93e9948aca4358a33b

Author: H. Peter Anvin <[email protected]>
Date:   Thu Jan 24 12:19:52 2013 -0800

x86, 64bit: Use a #PF handler to materialize early mappings on demand

I need somewhere between 500G and 600G of memory to trigger it, but it can be triggered using qemu with much less _actual_ RAM than that. From looking at the dmesg diffs, I suspect that the delta in memory use between using 1G and 4k ptes for the identity mapping (DEBUG_PAGEALLOC forces 4k pages) is the proximate trigger.

I also suspect that alloc_low_pages() is buggy in the way it manipulates min/max_pfn_mapped. I'm quite baffled how 'max_pfn_mapped' is supposed to get set up correctly. Current code says:

max_pfn_mapped = 0; /* will get exact value next */

but I certainly don't see it getting set later on in that function, or _ever_ as adding some printk()'s shows:

Quoted 8 lines
+[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
+[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 0 max_pfn_mapped: 0
+[    0.000000] BRK [0x02086000, 0x02086fff] PGTABLE
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 0 max_pfn_mapped: 0
+[    0.000000] BRK [0x02087000, 0x02087fff] PGTABLE
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 0 max_pfn_mapped: 0
+[    0.000000] BRK [0x02088000, 0x02088fff] PGTABLE
Quoted 8 lines
+[    0.000000] init_memory_mapping: [mem 0xf07fe00000-0xf07fffffff]
+[    0.000000]  [mem 0xf07fe00000-0xf07fffffff] page 4k
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 252182528 max_pfn_mapped: 0
+[    0.000000] BRK [0x02089000, 0x02089fff] PGTABLE
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 252182528 max_pfn_mapped: 0
+[    0.000000] BRK [0x0208a000, 0x0208afff] PGTABLE
+[    0.000000] alloc_low_pages(1) min_pfn_mapped: 252182528 max_pfn_mapped: 0
+[    0.000000] Kernel panic - not syncing: alloc_low_page: ran out of memory
Quoted 8 lines

I'll take a closer look at it next week, but figured I'd report it first.

Full dmesg:

Quoted 4 lines

early console in setup code

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.8.0-rc5-00059-g8170e6b

so how about v3.10?

We should have some fixes in 3.10 already.

Thanks

Yinghai

#2 lore raw
Quoted 2 lines · Yinghai Lu
Quoted 1 line · Dave Hansen

I'm getting a 100% reproducible panic early in boot:

...

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

early console in setup code

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.8.0-rc5-00059-g8170e6b

so how about v3.10?

We should have some fixes in 3.10 already.

I was hitting it on Linus's current tree today (3.11-rcwhatever). I pasted the panic() from your patch's commit specifically, but the same behavior is happening on current kernels, and it looked consistent as I bisected between the 3.11-rc's and your commit.

#3 lore raw
Quoted 9 lines · Dave Hansen
Quoted 2 lines · Yinghai Lu
Quoted 1 line · Dave Hansen

I'm getting a 100% reproducible panic early in boot:

...

Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

early console in setup code

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.8.0-rc5-00059-g8170e6b
Quoted 8 lines
Quoted 3 lines

so how about v3.10?

We should have some fixes in 3.10 already.

I was hitting it on Linus's current tree today (3.11-rcwhatever). I pasted the panic() from your patch's commit specifically, but the same behavior is happening on current kernels, and it looked consistent as I bisected between the 3.11-rc's and your commit.

Can you post 3.11-rc with "debug ignore_loglevel memblock=debug" ?

Thanks

Yinghai

#4 lore raw
Quoted 12 lines · Yinghai Lu
Quoted 11 lines · Dave Hansen
Quoted 2 lines · Yinghai Lu
Quoted 1 line · Dave Hansen

I'm getting a 100% reproducible panic early in boot:

...

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

early console in setup code

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.8.0-rc5-00059-g8170e6b

so how about v3.10?

Quoted 9 lines
Quoted 6 lines
Quoted 1 line

We should have some fixes in 3.10 already.

I was hitting it on Linus's current tree today (3.11-rcwhatever). I pasted the panic() from your patch's commit specifically, but the same behavior is happening on current kernels, and it looked consistent as I bisected between the 3.11-rc's and your commit.

Can you post 3.11-rc with "debug ignore_loglevel memblock=debug" ?

Can you try attached patch ?

Thanks

Yinghai

arch/x86/mm/init.c +2 -2
index 2ec29ac..f9eec80 100644
@@ -78,8 +78,8 @@ __ref void *alloc_low_pages(unsigned int num)
 	return __va(pfn << PAGE_SHIFT);
 }
 
-/* need 4 4k for initial PMD_SIZE, 4k for 0-ISA_END_ADDRESS */
-#define INIT_PGT_BUF_SIZE	(5 * PAGE_SIZE)
+/* need 4 4k for initial PMD_SIZE,  3 4k for 0-ISA_END_ADDRESS */
+#define INIT_PGT_BUF_SIZE	(7 * PAGE_SIZE)
 RESERVE_BRK(early_pgt_alloc, INIT_PGT_BUF_SIZE);
 void  __init early_alloc_pgt_buf(void)
 {
#5 lore raw

arch/x86/mm/init.c __ref void *alloc_low_pages(unsigned int num) · line 84

Quoted 6 lines · Yinghai Lu
arch/x86/mm/init.c +2 -2
-/* need 4 4k for initial PMD_SIZE, 4k for 0-ISA_END_ADDRESS */
-#define INIT_PGT_BUF_SIZE	(5 * PAGE_SIZE)
+/* need 4 4k for initial PMD_SIZE,  3 4k for 0-ISA_END_ADDRESS */
+#define INIT_PGT_BUF_SIZE	(7 * PAGE_SIZE)
 RESERVE_BRK(early_pgt_alloc, INIT_PGT_BUF_SIZE);
 void  __init early_alloc_pgt_buf(void)

That patch allows me to boot again. I've also attached a boot with the debug options that you asked for.

I'm really curious to see the full changelog for why this patch helps any why it's only triggered for large memory sizes. :)

Quoted 86 lines
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.11.0-rc4-00153-g14e9419-dirty ([email protected]) (gcc version 4.6.3 20120306 (Red Hat 4.6.3-2) (GCC) ) #33 SMP Mon Aug 12 09:22:46 PDT 2013
[    0.000000] Command line: root=/dev/sda1 console=ttyS0,115200 earlyprintk=ttyS0,115200 debug debug ignore_loglevel memblock=debug
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009f3ff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009f400-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000dfffbfff] usable
[    0.000000] BIOS-e820: [mem 0x00000000dfffc000-0x00000000dfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000e80effffff] usable
[    0.000000] bootconsole [earlyser0] enabled
[    0.000000] debug: ignoring loglevel setting.
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] DMI: Bochs Bochs, BIOS Bochs 01/01/2007
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] No AGP bridge found
[    0.000000] e820: last_pfn = 0xe80f000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 00E0000000 mask FFE0000000 uncachable
[    0.000000]   1 disabled
[    0.000000]   2 disabled
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] PAT not supported by CPU.
[    0.000000] e820: last_pfn = 0xdfffc max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000fdb00-0x000fdb0f] mapped at [ffff8800000fdb00]
[    0.000000] memblock_reserve: [0x000000000fdb00-0x000000000fdb10] smp_scan_config+0x101/0x135
[    0.000000] memblock_reserve: [0x000000000fdb10-0x000000000fdbf0] smp_scan_config+0x11a/0x135
[    0.000000] memblock_reserve: [0x00000002085000-0x0000000208b000] setup_arch+0x5ed/0xc79
[    0.000000] MEMBLOCK configuration:
[    0.000000]  memory size = 0xe7eef9a400 reserved size = 0x10ec000
[    0.000000]  memory.cnt  = 0x3
[    0.000000]  memory[0x0]	[0x00000000001000-0x0000000009efff], 0x9e000 bytes
[    0.000000]  memory[0x1]	[0x00000000100000-0x000000dfffbfff], 0xdfefc000 bytes
[    0.000000]  memory[0x2]	[0x00000100000000-0x0000e80effffff], 0xe70f000000 bytes
[    0.000000]  reserved.cnt  = 0x2
[    0.000000]  reserved[0x0]	[0x0000000009f000-0x000000000fffff], 0x61000 bytes
[    0.000000]  reserved[0x1]	[0x00000001000000-0x0000000208afff], 0x108b000 bytes
[    0.000000] memblock_reserve: [0x00000000099000-0x0000000009f000] reserve_real_mode+0x61/0x87
[    0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576
[    0.000000] memblock_reserve: [0x00000000000000-0x00000000010000] setup_arch+0x6cf/0xc79
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x02086000, 0x02086fff] PGTABLE
[    0.000000] BRK [0x02087000, 0x02087fff] PGTABLE
[    0.000000] BRK [0x02088000, 0x02088fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0xe80ee00000-0xe80effffff]
[    0.000000]  [mem 0xe80ee00000-0xe80effffff] page 4k
[    0.000000] BRK [0x02089000, 0x02089fff] PGTABLE
[    0.000000] BRK [0x0208a000, 0x0208afff] PGTABLE
[    0.000000] Kernel panic - not syncing: alloc_low_page: ran out of memory
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 3.11.0-rc4-00153-g14e9419-dirty #33
[    0.000000] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[    0.000000]  0000000000000001 ffffffff81c01ad8 ffffffff8165b550 000000000000019b
[    0.000000]  ffffffff819ee0b8 ffffffff81c01b58 ffffffff81658600 0000000000000000
[    0.000000]  0000000000000008 ffffffff81c01b68 ffffffff81c01b08 0000000000000006
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff8165b550>] dump_stack+0x55/0x76
[    0.000000]  [<ffffffff81658600>] panic+0xbb/0x1cb
[    0.000000]  [<ffffffff8164ee0a>] alloc_low_pages+0x15a/0x160
[    0.000000]  [<ffffffff816533bb>] phys_pmd_init+0x1e2/0x28d
[    0.000000]  [<ffffffff81658c46>] ? printk+0x4d/0x4f
[    0.000000]  [<ffffffff81653638>] phys_pud_init+0x1d2/0x25c
[    0.000000]  [<ffffffff81653fc7>] kernel_physical_mapping_init+0x14c/0x1ea
[    0.000000]  [<ffffffff8164eff3>] init_memory_mapping+0x1e3/0x350
[    0.000000]  [<ffffffff81d09d89>] init_range_memory_mapping+0xc2/0x10b
[    0.000000]  [<ffffffff81d09f05>] init_mem_mapping+0x133/0x1e2
[    0.000000]  [<ffffffff81cfaf11>] setup_arch+0x6d4/0xc79
[    0.000000]  [<ffffffff81658c46>] ? printk+0x4d/0x4f
[    0.000000]  [<ffffffff81cf4b59>] start_kernel+0xc9/0x3f3
[    0.000000]  [<ffffffff81cf45a6>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81cf4694>] x86_64_start_kernel+0xec/0xf3
#6 lore raw

PATCH 1/1 x86: Fix booting with DEBUG_PAGE_ALLOC with more than 512G RAM

Dave reported that system have early crash if DEBUG_PAGEALLOC is selected, when system have between 500G and 600G.

Quoted 10 lines
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x02086000, 0x02086fff] PGTABLE
[    0.000000] BRK [0x02087000, 0x02087fff] PGTABLE
[    0.000000] BRK [0x02088000, 0x02088fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0xe80ee00000-0xe80effffff]
[    0.000000]  [mem 0xe80ee00000-0xe80effffff] page 4k
[    0.000000] BRK [0x02089000, 0x02089fff] PGTABLE
[    0.000000] BRK [0x0208a000, 0x0208afff] PGTABLE
[    0.000000] Kernel panic - not syncing: alloc_low_page: ran out of memory

It turns out that we missed increasing needed pages in BRK to mapping initial 2M and [0,1M) when we switch to use #PF handler set mem mapping.

Quoted 5 lines

commit 8170e6bed465b4b0c7687f93e9948aca4358a33b

Author: H. Peter Anvin <[email protected]>
Date:   Thu Jan 24 12:19:52 2013 -0800

x86, 64bit: Use a #PF handler to materialize early mappings on demand

Before that, we have maping from [0,512M) in head_64.S, and we can spare two pages [0-1M). After that change, we can not reuse pages anymore.

When we have more than 512M ram, we need extra page for pgd page with [512G, 1024g).

Increase pages in BRK for page table to solve the booting problem.

Reported-by Dave Hansen <[email protected]>
Bisected-by Dave Hansen <[email protected]>
Tested-by Dave Hansen <[email protected]>
Signed-off-by Yinghai Lu <[email protected]>
Cc <[email protected]> 3.9+
---
 arch/x86/mm/init.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/mm/init.c +2 -2
Index: linux-2.6/arch/x86/mm/init.c
===================================================================
@@ -78,8 +78,8 @@ __ref void *alloc_low_pages(unsigned int
 	return __va(pfn << PAGE_SHIFT);
 }
 
-/* need 4 4k for initial PMD_SIZE, 4k for 0-ISA_END_ADDRESS */
-#define INIT_PGT_BUF_SIZE	(5 * PAGE_SIZE)
+/* need 3 4k for initial PMD_SIZE,  3 4k for 0-ISA_END_ADDRESS */
+#define INIT_PGT_BUF_SIZE	(6 * PAGE_SIZE)
 RESERVE_BRK(early_pgt_alloc, INIT_PGT_BUF_SIZE);
 void  __init early_alloc_pgt_buf(void)
 {
author
#7 lore raw

Re: [PATCH] x86: Fix booting with DEBUG_PAGE_ALLOC with more than 512G RAM

Quoted 4 lines · Yinghai Lu
Quoted 4 lines
Quoted 4 lines

commit 8170e6bed465b4b0c7687f93e9948aca4358a33b

Author: H. Peter Anvin <[email protected]>
Date:   Thu Jan 24 12:19:52 2013 -0800
Quoted 8 lines
Quoted 1 line
Quoted 1 line

x86, 64bit: Use a #PF handler to materialize early mappings on demand

Before that, we have maping from [0,512M) in head_64.S, and we can spare two pages [0-1M). After that change, we can not reuse pages anymore.

When we have more than 512M ram, we need extra page for pgd page with [512G, 1024g).

Increase pages in BRK for page table to solve the booting problem.

So how much does this get us up to? 1TB? That's actually _fairly_ small today. I've got a fairly old machine with that much in it, and it's only half full of DIMMs.

It's also a bit worrying that this is completely disconnected from the other code in the kernel that is concerned with the amount of total address space in the system: MAX_PHYSADDR_BITS.

#8 lore raw

Re: [PATCH] x86: Fix booting with DEBUG_PAGE_ALLOC with more than 512G RAM

Quoted 14 lines · Dave Hansen
Quoted 4 lines · Yinghai Lu
Quoted 4 lines
Quoted 4 lines

commit 8170e6bed465b4b0c7687f93e9948aca4358a33b

Author: H. Peter Anvin <[email protected]>
Date:   Thu Jan 24 12:19:52 2013 -0800
Quoted 8 lines
Quoted 1 line
Quoted 1 line

x86, 64bit: Use a #PF handler to materialize early mappings on demand

Before that, we have maping from [0,512M) in head_64.S, and we can spare two pages [0-1M). After that change, we can not reuse pages anymore.

When we have more than 512M ram, we need extra page for pgd page with [512G, 1024g).

Increase pages in BRK for page table to solve the booting problem.

Quoted 7 lines

So how much does this get us up to? 1TB? That's actually _fairly_ small today. I've got a fairly old machine with that much in it, and it's only half full of DIMMs.

It's also a bit worrying that this is completely disconnected from the other code in the kernel that is concerned with the amount of total address space in the system: MAX_PHYSADDR_BITS.

3 pages for [0,1M)
3 pages for initial 2M. ( it is 2M alignment).
are enough.

one page for PGD page (cover 512g), one page for PUD page (cover 1G) and one page for PMD page (cover 2M).

After initial 2M is mapped, we will use that mapped 2M for other memory range page table buffer.

Thanks

Yinghai

merge bot
#9 lore raw

[tip:x86/urgent] x86/mm: Fix boot crash with DEBUG_PAGE_ALLOC= y and more than 512G RAM

In reply to Yinghai Lu
x86/mm: Fix boot crash with DEBUG_PAGE_ALLOC=y and more than 512G RAM
Commit-ID:  527bf129f9a780e11b251cf2467dc30118a57d16
Gitweb:     http://git.kernel.org/tip/527bf129f9a780e11b251cf2467dc30118a57d16
Author:     Yinghai Lu <[email protected]>
AuthorDate: Mon, 12 Aug 2013 16:43:24 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 20 Aug 2013 10:06:50 +0200

x86/mm: Fix boot crash with DEBUG_PAGE_ALLOC=y and more than 512G RAM

Dave Hansen reported that systems between 500G and 600G RAM crash early if DEBUG_PAGEALLOC is selected.

Quoted 10 lines
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x02086000, 0x02086fff] PGTABLE
[    0.000000] BRK [0x02087000, 0x02087fff] PGTABLE
[    0.000000] BRK [0x02088000, 0x02088fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0xe80ee00000-0xe80effffff]
[    0.000000]  [mem 0xe80ee00000-0xe80effffff] page 4k
[    0.000000] BRK [0x02089000, 0x02089fff] PGTABLE
[    0.000000] BRK [0x0208a000, 0x0208afff] PGTABLE
[    0.000000] Kernel panic - not syncing: alloc_low_page: ran out of memory

It turns out that we missed increasing needed pages in BRK to mapping initial 2M and [0,1M) when we switched to use the #PF handler to set memory mappings:

Quoted 5 lines

commit 8170e6bed465b4b0c7687f93e9948aca4358a33b

Author: H. Peter Anvin <[email protected]>
Date:   Thu Jan 24 12:19:52 2013 -0800

x86, 64bit: Use a #PF handler to materialize early mappings on demand

Before that, we had the maping from [0,512M) in head_64.S, and we can spare two pages [0-1M). After that change, we can not reuse pages anymore.

When we have more than 512M ram, we need an extra page for pgd page with [512G, 1024g).

Increase pages in BRK for page table to solve the boot crash.

Reported-by Dave Hansen <[email protected]>
Bisected-by Dave Hansen <[email protected]>
Tested-by Dave Hansen <[email protected]>
Signed-off-by Yinghai Lu <[email protected]>
Cc <[email protected]> # v3.9 and later
Link http://lkml.kernel.org/r/[email protected] open here
Signed-off-by Ingo Molnar <[email protected]>
---
 arch/x86/mm/init.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/mm/init.c +2 -2
index 2ec29ac..04664cd 100644
@@ -78,8 +78,8 @@ __ref void *alloc_low_pages(unsigned int num)
 	return __va(pfn << PAGE_SHIFT);
 }
 
-/* need 4 4k for initial PMD_SIZE, 4k for 0-ISA_END_ADDRESS */
-#define INIT_PGT_BUF_SIZE	(5 * PAGE_SIZE)
+/* need 3 4k for initial PMD_SIZE,  3 4k for 0-ISA_END_ADDRESS */
+#define INIT_PGT_BUF_SIZE	(6 * PAGE_SIZE)
 RESERVE_BRK(early_pgt_alloc, INIT_PGT_BUF_SIZE);
 void  __init early_alloc_pgt_buf(void)
 {
#10 lore raw
In reply to Dave Hansen
Quoted 11 lines · Dave Hansen

arch/x86/mm/init.c __ref void *alloc_low_pages(unsigned int num) · line 84

Quoted 6 lines · Yinghai Lu
arch/x86/mm/init.c +2 -2
-/* need 4 4k for initial PMD_SIZE, 4k for 0-ISA_END_ADDRESS */
-#define INIT_PGT_BUF_SIZE    (5 * PAGE_SIZE)
+/* need 4 4k for initial PMD_SIZE,  3 4k for 0-ISA_END_ADDRESS */
+#define INIT_PGT_BUF_SIZE    (7 * PAGE_SIZE)
 RESERVE_BRK(early_pgt_alloc, INIT_PGT_BUF_SIZE);
 void  __init early_alloc_pgt_buf(void)

That patch allows me to boot again. I've also attached a boot with the debug options that you asked for.

Quoted 2 lines

I'm really curious to see the full changelog for why this patch helps any why it's only triggered for large memory sizes. :)

Thanks, please check changelog at

https://patchwork.kernel.org/patch/2843321/

Yinghai