Skip to contentSkip to conversation

sparc64 pcpu failures...

DISCUSSION David Miller 2 participants 4 messages Sep 18, 2009

lore
Share & tools
lore

David Miller

author Acked-by
#0 lore raw

Tejun, I just started seeing the following on sparc64:

[   56.422005] WARNING: at mm/vmalloc.c:1991 pcpu_get_vm_areas+0x1b4/0x5fc()
[   56.430881] Modules linked in: ipv6(+) snd_ali5451(+) snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device snd soundcore snd_page_alloc ac97_bus tg3 libphy
[   56.450490] Call Trace:
[   56.460021]  [00000000004d769c] pcpu_get_vm_areas+0x1b4/0x5fc
[   56.469645]  [00000000004ea5d8] pcpu_alloc+0x204/0x878
[   56.479212]  [00000000006d6a10] snmp_mib_init+0x3c/0x68
[   56.488813]  [0000000010184324] ipv6_add_dev+0x174/0x390 [ipv6]
[   56.498383]  [00000000101c44e0] addrconf_init+0x4c/0x16c [ipv6]
[   56.507913]  [00000000101c42fc] inet6_init+0x1cc/0x34c [ipv6]
[   56.517450]  [0000000000426acc] do_one_initcall+0x44/0x154
[   56.526975]  [0000000000490f7c] SyS_init_module+0xbc/0x1f4
[   56.536466]  [0000000000406214] linux_sparc_syscall32+0x34/0x40
[   56.545943] ---[ end trace 557118e4f3ad7a51 ]---
[   56.609093] NET: Unregistered protocol family 10
[   56.872235] NET: Registered protocol family 10

Might this be a result of:

commit bcb2107fdbecef3de55d597d23453747af81ba88

Author: Tejun Heo <[email protected]>
Date:   Fri Aug 14 15:00:53 2009 +0900

sparc64: use embedding percpu first chunk allocator

sparc64 currently allocates a large page for each cpu and partially remap them into vmalloc area much like what lpage first chunk allocator did. As a 4M page is used for each cpu, this results in very large unit size and also adds TLB pressure due to the double mapping of pages in the first chunk.

This patch converts sparc64 to use the embedding percpu first chunk allocator which now knows how to handle NUMA configurations. This simplifies the code a lot, doesn't incur any extra TLB pressure and results in better utilization of address space.

Do you think?

#1 lore raw

From: David Miller <[email protected]>
Date: Thu, 17 Sep 2009 22:18:07 -0700 (PDT)

Quoted 4 lines

Tejun, I just started seeing the following on sparc64:

[   56.422005] WARNING: at mm/vmalloc.c:1991 pcpu_get_vm_areas+0x1b4/0x5fc()

...

Quoted 19 lines

Might this be a result of:

commit bcb2107fdbecef3de55d597d23453747af81ba88

Author: Tejun Heo <[email protected]>
Date:   Fri Aug 14 15:00:53 2009 +0900

sparc64: use embedding percpu first chunk allocator

sparc64 currently allocates a large page for each cpu and partially remap them into vmalloc area much like what lpage first chunk allocator did. As a 4M page is used for each cpu, this results in very large unit size and also adds TLB pressure due to the double mapping of pages in the first chunk.

This patch converts sparc64 to use the embedding percpu first chunk allocator which now knows how to handle NUMA configurations. This simplifies the code a lot, doesn't incur any extra TLB pressure and results in better utilization of address space.

Quoted 4 lines

Do you think?

I've verified that reverting this makes the problem go away.

#2 lore raw

Hello, David.

Quoted 26 lines · David Miller

From: David Miller <[email protected]>
Date: Thu, 17 Sep 2009 22:18:07 -0700 (PDT)

Quoted 3 lines

Tejun, I just started seeing the following on sparc64:

[   56.422005] WARNING: at mm/vmalloc.c:1991 pcpu_get_vm_areas+0x1b4/0x5fc()

...

Quoted 19 lines

Might this be a result of:

commit bcb2107fdbecef3de55d597d23453747af81ba88

Author: Tejun Heo <[email protected]>
Date:   Fri Aug 14 15:00:53 2009 +0900

sparc64: use embedding percpu first chunk allocator

sparc64 currently allocates a large page for each cpu and partially remap them into vmalloc area much like what lpage first chunk allocator did. As a 4M page is used for each cpu, this results in very large unit size and also adds TLB pressure due to the double mapping of pages in the first chunk.

This patch converts sparc64 to use the embedding percpu first chunk allocator which now knows how to handle NUMA configurations. This simplifies the code a lot, doesn't incur any extra TLB pressure and results in better utilization of address space.

Quoted 6 lines
Quoted 4 lines

Do you think?

I've verified that reverting this makes the problem go away.

Yes, this would be the result of the new congruent sparse allocator. Heh... I didn't really expect that WARN_ON() to actually trigger. What it means is that the farthest percpu units were too far to fit into vmalloc area so percpu chunks couldn't be allocated in the vmalloc area. Checking... vmalloc area on sparc64 is only 4G so yeap that is entirely possible.

Hmmm... the congruent sparse allocator assumes vmalloc area to be larger with some margin than the farthest physical memory nodes. On x86, powerpc and ia64, this holds with sufficient margin. Would it be possible to modify sparc64 memory layout so that the assumption can be satisfied? If that's not possible, going back to lpage is an option but in many ways congruent sparse allocator is better.

Thanks.

Signature · 1 line
tejun
#3 lore raw

From: Tejun Heo <[email protected]>
Date: Fri, 18 Sep 2009 17:38:58 +0900

Quoted 7 lines

Yes, this would be the result of the new congruent sparse allocator. Heh... I didn't really expect that WARN_ON() to actually trigger. What it means is that the farthest percpu units were too far to fit into vmalloc area so percpu chunks couldn't be allocated in the vmalloc area. Checking... vmalloc area on sparc64 is only 4G so yeap that is entirely possible.

Quoted 6 lines

Hmmm... the congruent sparse allocator assumes vmalloc area to be larger with some margin than the farthest physical memory nodes. On x86, powerpc and ia64, this holds with sufficient margin. Would it be possible to modify sparc64 memory layout so that the assumption can be satisfied? If that's not possible, going back to lpage is an option but in many ways congruent sparse allocator is better.

I'll think about this, thanks.