kvm_intel: Could not allocate 42 bytes percpu data

DISCUSSION Chegu Vinod 4 participants 8 messages Jun 24, 2013 – Jul 3, 2013 KVM

Share & tools

Chegu Vinod

author

2013-06-24 19:01 UTC #0 lore raw

Hello,

Lots (~700+) of the following messages are showing up in the dmesg of a 3.10-rc1 based kernel (Host OS is running on a large socket count box with HT-on).

[   82.270682] PERCPU: allocation failed, size=42 align=16, alloc from reserved chunk failed
[   82.272633] kvm_intel: Could not allocate 42 bytes percpu data

... also call traces like the following...

[  101.852136]  ffffc901ad5aa090 ffff88084675dd08 ffffffff81633743 ffff88084675ddc8
[  101.860889]  ffffffff81145053 ffffffff81f3fa78 ffff88084809dd40 ffff8907d1cfd2e8
[  101.869466]  ffff8907d1cfd280 ffff88087fffdb08 ffff88084675c010 ffff88084675dfd8
[  101.878190] Call Trace:
[  101.880953]  [<ffffffff81633743>] dump_stack+0x19/0x1e
[  101.886679]  [<ffffffff81145053>] pcpu_alloc+0x9a3/0xa40
[  101.892754]  [<ffffffff81145103>] __alloc_reserved_percpu+0x13/0x20
[  101.899733]  [<ffffffff810b2d7f>] load_module+0x35f/0x1a70
[  101.905835]  [<ffffffff8163ad6e>] ? do_page_fault+0xe/0x10
[  101.911953]  [<ffffffff810b467b>] SyS_init_module+0xfb/0x140
[  101.918287]  [<ffffffff8163f542>] system_call_fastpath+0x16/0x1b
[  101.924981] kvm_intel: Could not allocate 42 bytes percpu data

Wondering if anyone else has seen this with the recent [3.10] based kernels esp. on larger boxes?

There was a similar issue that was reported earlier (where modules were being loaded per cpu without checking if an instance was already loaded/being-loaded). That issue seems to have been addressed in the recent past (e.g. https://lkml.org/lkml/2013/1/24/659 along with a couple of follow on cleanups) Is the above yet another variant of the original issue or perhaps some race condition that got exposed when there are lot more threads ?

Vinod

Prarit Bhargava

2013-06-24 22:52 UTC #1 lore raw

Quoted 9 lines · Chegu Vinod

Hello,
Lots (~700+) of the following messages are showing up in the dmesg of a 3.10-rc1 based kernel (Host OS is running on a large socket count box with HT-on).
[   82.270682] PERCPU: allocation failed, size=42 align=16, alloc from reserved
chunk failed
[   82.272633] kvm_intel: Could not allocate 42 bytes percpu data

On 3.10? Geez. I thought we had fixed this. I'll grab a big machine and see if I can debug.

Rusty -- any ideas off the top of your head?'

Quoted 20 lines

... also call traces like the following...

[  101.852136]  ffffc901ad5aa090 ffff88084675dd08 ffffffff81633743 ffff88084675ddc8
[  101.860889]  ffffffff81145053 ffffffff81f3fa78 ffff88084809dd40 ffff8907d1cfd2e8
[  101.869466]  ffff8907d1cfd280 ffff88087fffdb08 ffff88084675c010 ffff88084675dfd8
[  101.878190] Call Trace:
[  101.880953]  [<ffffffff81633743>] dump_stack+0x19/0x1e
[  101.886679]  [<ffffffff81145053>] pcpu_alloc+0x9a3/0xa40
[  101.892754]  [<ffffffff81145103>] __alloc_reserved_percpu+0x13/0x20
[  101.899733]  [<ffffffff810b2d7f>] load_module+0x35f/0x1a70
[  101.905835]  [<ffffffff8163ad6e>] ? do_page_fault+0xe/0x10
[  101.911953]  [<ffffffff810b467b>] SyS_init_module+0xfb/0x140
[  101.918287]  [<ffffffff8163f542>] system_call_fastpath+0x16/0x1b
[  101.924981] kvm_intel: Could not allocate 42 bytes percpu data

Wondering if anyone else has seen this with the recent [3.10] based kernels esp. on larger boxes?

Quoted 6 lines

There was a similar issue that was reported earlier (where modules were being loaded per cpu without checking if an instance was already loaded/being-loaded). That issue seems to have been addressed in the recent past (e.g.
https://lkml.org/lkml/2013/1/24/659 along with a couple of follow on cleanups) Is the above yet another variant of the original issue or perhaps some race condition that got exposed when there are lot more threads ?

Hmm ... not sure but yeah, that's the likely culprit.

Marcelo Tosatti

2013-06-27 00:53 UTC #2 lore raw

Quoted 9 lines · Prarit Bhargava

Quoted 6 lines · Chegu Vinod
Hello,
Lots (~700+) of the following messages are showing up in the dmesg of a 3.10-rc1 based kernel (Host OS is running on a large socket count box with HT-on).

Quoted 8 lines

Quoted 3 lines
[   82.270682] PERCPU: allocation failed, size=42 align=16, alloc from reserved
chunk failed
[   82.272633] kvm_intel: Could not allocate 42 bytes percpu data
On 3.10? Geez. I thought we had fixed this. I'll grab a big machine and see if I can debug.
Rusty -- any ideas off the top of your head?'

As far as my limited understanding goes, the reserved space setup by arch code for percpu allocations, is limited and subject to exhaustion.

It would be best if the allocator could handle the allocation, but otherwise, switching vmx.c to dynamic allocations for the percpu regions is an option (see 013f6a5d3dd9e4).

It should be similar to convert these two larger data structures:

static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu); static DEFINE_PER_CPU(struct desc_ptr, host_gdt);

Quoted 30 lines

Quoted 20 lines
... also call traces like the following...
[  101.852136]  ffffc901ad5aa090 ffff88084675dd08 ffffffff81633743 ffff88084675ddc8
[  101.860889]  ffffffff81145053 ffffffff81f3fa78 ffff88084809dd40 ffff8907d1cfd2e8
[  101.869466]  ffff8907d1cfd280 ffff88087fffdb08 ffff88084675c010 ffff88084675dfd8
[  101.878190] Call Trace:
[  101.880953]  [<ffffffff81633743>] dump_stack+0x19/0x1e
[  101.886679]  [<ffffffff81145053>] pcpu_alloc+0x9a3/0xa40
[  101.892754]  [<ffffffff81145103>] __alloc_reserved_percpu+0x13/0x20
[  101.899733]  [<ffffffff810b2d7f>] load_module+0x35f/0x1a70
[  101.905835]  [<ffffffff8163ad6e>] ? do_page_fault+0xe/0x10
[  101.911953]  [<ffffffff810b467b>] SyS_init_module+0xfb/0x140
[  101.918287]  [<ffffffff8163f542>] system_call_fastpath+0x16/0x1b
[  101.924981] kvm_intel: Could not allocate 42 bytes percpu data
Wondering if anyone else has seen this with the recent [3.10] based kernels esp. on larger boxes?
Quoted 6 lines
There was a similar issue that was reported earlier (where modules were being loaded per cpu without checking if an instance was already loaded/being-loaded). That issue seems to have been addressed in the recent past (e.g.
https://lkml.org/lkml/2013/1/24/659 along with a couple of follow on cleanups) Is the above yet another variant of the original issue or perhaps some race condition that got exposed when there are lot more threads ?
Hmm ... not sure but yeah, that's the likely culprit.
P.

Rusty Russell

Tested-by Noone it seems.

2013-07-01 08:53 UTC #3 lore raw

In reply to Chegu Vinod

Quoted 9 lines · Chegu Vinod

Hello,
Lots (~700+) of the following messages are showing up in the dmesg of a 3.10-rc1 based kernel (Host OS is running on a large socket count box with HT-on).
[   82.270682] PERCPU: allocation failed, size=42 align=16, alloc from 
reserved chunk failed
[   82.272633] kvm_intel: Could not allocate 42 bytes percpu data

Woah, weird....

Oh. Shit. Um, this is embarrassing.

Thanks,
Rusty.
===
module: do percpu allocation after uniqueness check. No, really!

v3.8-rc1-5-g1fb9341 was supposed to stop parallel kvm loads exhausting percpu memory on large machines:

Now we have a new state MODULE_STATE_UNFORMED, we can insert the module into the list (and thus guarantee its uniqueness) before we allocate the per-cpu region.

In my defence, it didn't actually say the patch did this. Just that we "can".

This patch actually *does* it.

Signed-off-by Rusty Russell <[email protected]>

Tested-by Noone it seems.

kernel/module.c +18 ~~-16~~

index cab4bce..fa53db8 100644
@@ -2927,7 +2927,6 @@ static struct module *layout_and_allocate(struct load_info *info, int flags)
 {
 	/* Module within temporary copy. */
 	struct module *mod;
-	Elf_Shdr *pcpusec;
 	int err;
 
 	mod = setup_load_info(info, flags);

@@ -2942,17 +2941,10 @@ static struct module *layout_and_allocate(struct load_info *info, int flags)
 	err = module_frob_arch_sections(info->hdr, info->sechdrs,
 					info->secstrings, mod);
 	if (err < 0)
-		goto out;
+		return ERR_PTR(err);
 
-	pcpusec = &info->sechdrs[info->index.pcpu];
-	if (pcpusec->sh_size) {
-		/* We have a special allocation for this section. */
-		err = percpu_modalloc(mod,
-				      pcpusec->sh_size, pcpusec->sh_addralign);
-		if (err)
-			goto out;
-		pcpusec->sh_flags &= ~(unsigned long)SHF_ALLOC;
-	}
+	/* We will do a special allocation for per-cpu sections later. */
+	info->sechdrs[info->index.pcpu].sh_flags &= ~(unsigned long)SHF_ALLOC;
 
 	/* Determine total sizes, and put offsets in sh_entsize.  For now
 	   this is done generically; there doesn't appear to be any

@@ -2963,17 +2955,22 @@ static struct module *layout_and_allocate(struct load_info *info, int flags)
 	/* Allocate and move to the final place */
 	err = move_module(mod, info);
 	if (err)
-		goto free_percpu;
+		return ERR_PTR(err);
 
 	/* Module has been copied to its final place now: return it. */
 	mod = (void *)info->sechdrs[info->index.mod].sh_addr;
 	kmemleak_load_module(mod, info);
 	return mod;
+}
 
-free_percpu:
-	percpu_modfree(mod);
-out:
-	return ERR_PTR(err);
+static int alloc_module_percpu(struct module *mod, struct load_info *info)
+{
+	Elf_Shdr *pcpusec = &info->sechdrs[info->index.pcpu];
+	if (!pcpusec->sh_size)
+		return 0;
+
+	/* We have a special allocation for this section. */
+	return percpu_modalloc(mod, pcpusec->sh_size, pcpusec->sh_addralign);
 }
 
 /* mod is no longer valid after this! */

@@ -3237,6 +3234,11 @@ static int load_module(struct load_info *info, const char __user *uargs,
 	}
 #endif
 
+	/* To avoid stressing percpu allocator, do this once we're unique. */
+	err = alloc_module_percpu(mod, info);
+	if (err)
+		goto unlink_mod;
+
 	/* Now module is in final location, initialize linked lists, etc. */
 	err = module_unload_init(mod);
 	if (err)

Chegu Vinod

author

2013-07-02 01:13 UTC #4 lore raw

Quoted 10 lines · Rusty Russell

Quoted 9 lines · Chegu Vinod
Hello,
Lots (~700+) of the following messages are showing up in the dmesg of a 3.10-rc1 based kernel (Host OS is running on a large socket count box with HT-on).
[   82.270682] PERCPU: allocation failed, size=42 align=16, alloc from
reserved chunk failed
[   82.272633] kvm_intel: Could not allocate 42 bytes percpu data

Quoted 6 lines

Woah, weird....
Oh. Shit. Um, this is embarrassing.
Thanks,
Rusty.

Thanks for your response!

Quoted 10 lines

===
module: do percpu allocation after uniqueness check. No, really!
v3.8-rc1-5-g1fb9341 was supposed to stop parallel kvm loads exhausting percpu memory on large machines:
Now we have a new state MODULE_STATE_UNFORMED, we can insert the module into the list (and thus guarantee its uniqueness) before we allocate the per-cpu region.

Quoted 7 lines

In my defence, it didn't actually say the patch did this. Just that we "can".
This patch actually *does* it.
Signed-off-by Rusty Russell <[email protected]>
Tested-by Noone it seems.

Your following "updated" fix seems to be working fine on the larger socket count machine with HT-on.

Thx
Vinod

Quoted 76 lines

kernel/module.c +18 ~~-16~~

index cab4bce..fa53db8 100644
@@ -2927,7 +2927,6 @@ static struct module *layout_and_allocate(struct load_info *info, int flags)
  {
  	/* Module within temporary copy. */
  	struct module *mod;
-	Elf_Shdr *pcpusec;
  	int err;
 
  	mod = setup_load_info(info, flags);

@@ -2942,17 +2941,10 @@ static struct module *layout_and_allocate(struct load_info *info, int flags)
  	err = module_frob_arch_sections(info->hdr, info->sechdrs,
  					info->secstrings, mod);
  	if (err < 0)
-		goto out;
+		return ERR_PTR(err);
 
-	pcpusec = &info->sechdrs[info->index.pcpu];
-	if (pcpusec->sh_size) {
-		/* We have a special allocation for this section. */
-		err = percpu_modalloc(mod,
-				      pcpusec->sh_size, pcpusec->sh_addralign);
-		if (err)
-			goto out;
-		pcpusec->sh_flags &= ~(unsigned long)SHF_ALLOC;
-	}
+	/* We will do a special allocation for per-cpu sections later. */
+	info->sechdrs[info->index.pcpu].sh_flags &= ~(unsigned long)SHF_ALLOC;
 
  	/* Determine total sizes, and put offsets in sh_entsize.  For now
  	   this is done generically; there doesn't appear to be any

@@ -2963,17 +2955,22 @@ static struct module *layout_and_allocate(struct load_info *info, int flags)
  	/* Allocate and move to the final place */
  	err = move_module(mod, info);
  	if (err)
-		goto free_percpu;
+		return ERR_PTR(err);
 
  	/* Module has been copied to its final place now: return it. */
  	mod = (void *)info->sechdrs[info->index.mod].sh_addr;
  	kmemleak_load_module(mod, info);
  	return mod;
+}
 
-free_percpu:
-	percpu_modfree(mod);
-out:
-	return ERR_PTR(err);
+static int alloc_module_percpu(struct module *mod, struct load_info *info)
+{
+	Elf_Shdr *pcpusec = &info->sechdrs[info->index.pcpu];
+	if (!pcpusec->sh_size)
+		return 0;
+
+	/* We have a special allocation for this section. */
+	return percpu_modalloc(mod, pcpusec->sh_size, pcpusec->sh_addralign);
  }
 
  /* mod is no longer valid after this! */

@@ -3237,6 +3234,11 @@ static int load_module(struct load_info *info, const char __user *uargs,
  	}
  #endif
 
+	/* To avoid stressing percpu allocator, do this once we're unique. */
+	err = alloc_module_percpu(mod, info);
+	if (err)
+		goto unlink_mod;
+
  	/* Now module is in final location, initialize linked lists, etc. */
  	err = module_unload_init(mod);
  	if (err)

Rusty Russell

2013-07-02 05:49 UTC #5 lore raw

Quoted 34 lines · Chegu Vinod

Quoted 10 lines · Rusty Russell
Quoted 9 lines · Chegu Vinod
Hello,
Lots (~700+) of the following messages are showing up in the dmesg of a 3.10-rc1 based kernel (Host OS is running on a large socket count box with HT-on).
[   82.270682] PERCPU: allocation failed, size=42 align=16, alloc from
reserved chunk failed
[   82.272633] kvm_intel: Could not allocate 42 bytes percpu data
Quoted 6 lines
Woah, weird....
Oh. Shit. Um, this is embarrassing.
Thanks,
Rusty.
Thanks for your response!
Quoted 13 lines
===
module: do percpu allocation after uniqueness check. No, really!
v3.8-rc1-5-g1fb9341 was supposed to stop parallel kvm loads exhausting percpu memory on large machines:
Now we have a new state MODULE_STATE_UNFORMED, we can insert the module into the list (and thus guarantee its uniqueness) before we allocate the per-cpu region.
In my defence, it didn't actually say the patch did this. Just that we "can".

Quoted 7 lines

Quoted 4 lines
This patch actually *does* it.
Signed-off-by Rusty Russell <[email protected]>
Tested-by Noone it seems.
Your following "updated" fix seems to be working fine on the larger socket count machine with HT-on.

OK, did you definitely revert every other workaround?

If so, please give me a Tested-by: line...

Thanks,
Rusty.

Chegu Vinod

author Tested-by Jim Hull

2013-07-02 16:34 UTC #6 lore raw

Quoted 34 lines · Rusty Russell

Quoted 33 lines · Chegu Vinod
Quoted 10 lines · Rusty Russell
Quoted 9 lines · Chegu Vinod
Hello,
Lots (~700+) of the following messages are showing up in the dmesg of a 3.10-rc1 based kernel (Host OS is running on a large socket count box with HT-on).
[   82.270682] PERCPU: allocation failed, size=42 align=16, alloc from
reserved chunk failed
[   82.272633] kvm_intel: Could not allocate 42 bytes percpu data
Quoted 6 lines
Woah, weird....
Oh. Shit. Um, this is embarrassing.
Thanks,
Rusty.
Thanks for your response!
Quoted 13 lines
===
module: do percpu allocation after uniqueness check. No, really!
v3.8-rc1-5-g1fb9341 was supposed to stop parallel kvm loads exhausting percpu memory on large machines:
Now we have a new state MODULE_STATE_UNFORMED, we can insert the module into the list (and thus guarantee its uniqueness) before we allocate the per-cpu region.
In my defence, it didn't actually say the patch did this. Just that we "can".

Quoted 7 lines

Quoted 6 lines
Quoted 4 lines
This patch actually *does* it.
Signed-off-by Rusty Russell <[email protected]>
Tested-by Noone it seems.
Your following "updated" fix seems to be working fine on the larger socket count machine with HT-on.
OK, did you definitely revert every other workaround?

Yes no other workarounds were there when your change was tested.

Quoted 2 lines

If so, please give me a Tested-by: line...

FYI.... The actual verification of your change was done by my esteemed colleague :Jim Hull (cc'd) who had access to this larger socket count box.

Tested-by Jim Hull <[email protected]>

Thanks
Vinod

Quoted 5 lines

Thanks,
Rusty.
.

Rusty Russell

2013-07-03 00:37 UTC #7 lore raw

Quoted 26 lines · Chegu Vinod

Quoted 14 lines · Rusty Russell
Quoted 13 lines · Chegu Vinod
Quoted 12 lines · Rusty Russell
module: do percpu allocation after uniqueness check. No, really!
v3.8-rc1-5-g1fb9341 was supposed to stop parallel kvm loads exhausting percpu memory on large machines:
Now we have a new state MODULE_STATE_UNFORMED, we can insert the module into the list (and thus guarantee its uniqueness) before we allocate the per-cpu region.
In my defence, it didn't actually say the patch did this. Just that we "can".
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
This patch actually *does* it.
Signed-off-by Rusty Russell <[email protected]>
Tested-by Noone it seems.
Your following "updated" fix seems to be working fine on the larger socket count machine with HT-on.
OK, did you definitely revert every other workaround?
Yes no other workarounds were there when your change was tested.
Quoted 1 line

Quoted 8 lines

Quoted 1 line
If so, please give me a Tested-by: line...
FYI.... The actual verification of your change was done by my esteemed colleague :Jim Hull (cc'd) who had access to this larger socket count box.
Tested-by Jim Hull <[email protected]>

Thanks, I've put this in my -next tree, and CC'd stable.

Cheers,
Rusty.