mm: percpu: Add PCPU_FC_FIXED to pcpu_fc for setting fixed pcpu_atom_size.

From: ShuoX Liu <[email protected]>

We are enabling Android on Medfield. On i386, if the board has more physical memory, it means the vmalloc space is small. If the vmalloc space is big, it means physical memory is small. Dynamic percpu allocation is based on VM space. On i386, by default, the chunk size is 4MB. As vmalloc space is <= 128M, percpu allocation often fails.

If using PERCPU_FC_PAGE, system can't go to deep sleep states.

arch/x86/kernel/setup_percpu.c +0 -1

index 71f4727..824bc41 100644
@@ -185,9 +185,13 @@ void __init setup_per_cpu_areas(void)
 #endif
 	rc = -EINVAL;
 	if (pcpu_chosen_fc != PCPU_FC_PAGE) {
-		const size_t atom_size = cpu_has_pse ? PMD_SIZE : PAGE_SIZE;

Hello, ShouX.

Quoted 7 lines · ShuoX Liu

From: ShuoX Liu <[email protected]>

We are enabling Android on Medfield. On i386, if the board has more physical memory, it means the vmalloc space is small. If the vmalloc space is big, it means physical memory is small. Dynamic percpu allocation is based on VM space. On i386, by default, the chunk size is 4MB. As vmalloc space is <= 128M, percpu allocation often fails.

Can you please provide more details - which kernel/user split was used, how much memory was there and so on? Also, can you please attach boot log and the output of "cat /proc/vmallocinfo"?

Quoted 2 lines

If using PERCPU_FC_PAGE, system can't go to deep sleep states.

Why?

Quoted 18 lines

arch/x86/kernel/setup_percpu.c +5 -1

index 71f4727..824bc41 100644
@@ -185,9 +185,13 @@ void __init setup_per_cpu_areas(void)
 #endif
 	rc = -EINVAL;
 	if (pcpu_chosen_fc != PCPU_FC_PAGE) {
-		const size_t atom_size = cpu_has_pse ? PMD_SIZE : PAGE_SIZE;
+		size_t atom_size;
 		const size_t dyn_size = PERCPU_MODULE_RESERVE +
 			PERCPU_DYNAMIC_RESERVE - PERCPU_FIRST_CHUNK_RESERVE;
+		if (pcpu_chosen_fc == PCPU_FC_FIXED && pcpu_atom_size)
+			atom_size = pcpu_atom_size;
+		else
+			atom_size = cpu_has_pse ? PMD_SIZE : PAGE_SIZE;
 

arch/x86/kernel/setup_percpu.c void __init setup_per_cpu_areas(void) · line 197

Quoted 2 lines

arch/x86/kernel/setup_percpu.c +0 -0

 		rc = pcpu_embed_first_chunk(PERCPU_FIRST_CHUNK_RESERVE,
 					    dyn_size, atom_size,

Umm... this is way too hacky. atom_size can't be an arbitrary value

and the param is meaningful only to x86 yet defined globally.

Also,
while atom_size has effect on vmalloc area usage, way more important factor is distance between units.

What we probably need to do is
tighten the rejection criteria of pcpu_embed_first_chunk()

and fix
whatever problem FC_PAGE is causing.

[    0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 85
[    0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:bec00000)
[    0.000000] setup_percpu: NR_CPUS:2 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 12 pages/cpu @f6400000 s25280 r0 d23872 u2097152
[    0.000000] pcpu-alloc: s25280 r0 d23872 u2097152 alloc=1*4194304
[    0.000000] pcpu-alloc: [0] 0 1

PERCPU: allocation failed, size=252 align=4, failed to allocate new chunk

vmallocinfo is attached. From the vmallocinfo, we could find the VM space is fragmented. We would write another patch to clean it up.

Quoted 3 lines

Quoted 1 line

If using PERCPU_FC_PAGE, system can't go to deep sleep states.

Why?

Medfield has 2 cpu threads. Only when all the 2 threads enter deep C states, for example, C6, the core would enter C6. If booting kernel with percpu_alloc=page, cpu core often aborts the C6 entering. We don't know why. C6 is aborted under many conditions. One is when there is pending interrupt. I suspect with page size alloc, it might trigger more cache miss. Just before calls mwait to enter C6, we record some statistics data and that might trigger the cache miss to abort the C6. It's just a _GUESS_.

We tried atom_size with 32k, 128k, 256k. There is no power regression.

We can't fix FC_PAGE power regression. If we do so, we need contact many hardware architects. Current kernel supports FC_PAGE and PMD_SIZE, why not to allow admin to choose other values?

Hello,

Quoted 7 lines · Yanmin Zhang

[    0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 85
[    0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:bec00000)
[    0.000000] setup_percpu: NR_CPUS:2 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 12 pages/cpu @f6400000 s25280 r0 d23872 u2097152
[    0.000000] pcpu-alloc: s25280 r0 d23872 u2097152 alloc=1*4194304
[    0.000000] pcpu-alloc: [0] 0 1

Heh, I was getting confused, forget the distance thing, so it's single group w/ 4MiB allocation size.

Quoted 1 line

PERCPU: allocation failed, size=252 align=4, failed to allocate new chunk

Which later fails percpu allocation due to vmalloc space exhaustion. How long does that take to happen?

Quoted 3 lines

vmallocinfo is attached. From the vmallocinfo, we could find the VM space is fragmented. We would write another patch to clean it up.

Whee... ah well, 128M isn't that big after all.

Quoted 13 lines

Quoted 4 lines

Quoted 2 lines

If using PERCPU_FC_PAGE, system can't go to deep sleep states.

Why?

Medfield has 2 cpu threads. Only when all the 2 threads enter deep C states, for example, C6, the core would enter C6. If booting kernel with percpu_alloc=page, cpu core often aborts the C6 entering. We don't know why. C6 is aborted under many conditions. One is when there is pending interrupt. I suspect with page size alloc, it might trigger more cache miss. Just before calls mwait to enter C6, we record some statistics data and that might trigger the cache miss to abort the C6. It's just a _GUESS_.

Quoted 1 line

We tried atom_size with 32k, 128k, 256k. There is no power regression.

So, the difference between EMBED and PAGE is how the first chunk which contains all the static percpu variables and some dynamic area for optimization is allocated. For EMBED, it's just kmallocd which means that it piggy backs on the default kernel linear mapping thus avoiding adding any extra TLB pressure. For PAGE, all those percpu areas end up getting re-mapped in vmalloc area using 4k pages, so if TLB pressure can affect entering C6, that could be it.

Quoted 4 lines

We can't fix FC_PAGE power regression. If we do so, we need contact many hardware architects. Current kernel supports FC_PAGE and PMD_SIZE, why not to allow admin to choose other values?

If this is something which is met in the field commonly, we need to fix the default behavior rather than introducing some arcane boot param.

IIRC, the reasons PMD_SIZE is used for atom_size are so that
percpu areas are aligned to PSE mapping, maybe later we can make use of PSE mapping in vmalloc area too, and it didn't seem to hurt anything.

If the large unit size is becoming a problem on i386, we can just use PAGE_SIZE as atom_size. Can you please verify that atom_size of 4k w/ EMBED also resolves the power issue?

Quoted 11 lines · Tejun Heo

Hello,

Quoted 7 lines · Yanmin Zhang

[    0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 85
[    0.000000] Allocating PCI resources starting at 40000000 (gap: 40000000:bec00000)
[    0.000000] setup_percpu: NR_CPUS:2 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 12 pages/cpu @f6400000 s25280 r0 d23872 u2097152
[    0.000000] pcpu-alloc: s25280 r0 d23872 u2097152 alloc=1*4194304
[    0.000000] pcpu-alloc: [0] 0 1

Quoted 7 lines

Heh, I was getting confused, forget the distance thing, so it's single group w/ 4MiB allocation size.

Quoted 1 line

PERCPU: allocation failed, size=252 align=4, failed to allocate new chunk

Which later fails percpu allocation due to vmalloc space exhaustion. How long does that take to happen?

It depends. Sometimes it fails in 400 seconds after booting. We run MTBF and other stress testing. Sometimes even with other non-stress testing, pecpu allocation fails. Most drivers or upper layers expect the percpu allocation should succeed. If not, although mostly there is no OOPS in kernel, upper applications wouldn't work.

Quoted 5 lines

Quoted 3 lines

vmallocinfo is attached. From the vmallocinfo, we could find the VM space is fragmented. We would write another patch to clean it up.

Whee... ah well, 128M isn't that big after all.

Indeed, so we need tune the memory utilization carefully on i386. We did work out other patches at other places/drivers to fix other OOM issues.

Quoted 15 lines

Quoted 13 lines

Quoted 4 lines

Quoted 2 lines

If using PERCPU_FC_PAGE, system can't go to deep sleep states.

Why?

Medfield has 2 cpu threads. Only when all the 2 threads enter deep C states, for example, C6, the core would enter C6. If booting kernel with percpu_alloc=page, cpu core often aborts the C6 entering. We don't know why. C6 is aborted under many conditions. One is when there is pending interrupt. I suspect with page size alloc, it might trigger more cache miss. Just before calls mwait to enter C6, we record some statistics data and that might trigger the cache miss to abort the C6. It's just a _GUESS_.

Quoted 1 line

We tried atom_size with 32k, 128k, 256k. There is no power regression.

Quoted 7 lines

So, the difference between EMBED and PAGE is how the first chunk which contains all the static percpu variables and some dynamic area for optimization is allocated. For EMBED, it's just kmallocd which means that it piggy backs on the default kernel linear mapping thus avoiding adding any extra TLB pressure. For PAGE, all those percpu areas end up getting re-mapped in vmalloc area using 4k pages, so if TLB pressure can affect entering C6, that could be it.

Thanks for the explanation.

Quoted 8 lines

Quoted 4 lines

We can't fix FC_PAGE power regression. If we do so, we need contact many hardware architects. Current kernel supports FC_PAGE and PMD_SIZE, why not to allow admin to choose other values?

If this is something which is met in the field commonly, we need to fix the default behavior rather than introducing some arcane boot param.

We just add a new value input method instead of introducing new parameter.

Quoted 4 lines

IIRC, the reasons PMD_SIZE is used for atom_size are so that
percpu areas are aligned to PSE mapping, maybe later we can make use of PSE mapping in vmalloc area too, and it didn't seem to hurt anything.

Well, vmalloc area might use different prot to map physical pages. So sharing one PMD huge page by many vmalloc areas might be not good.

Quoted 4 lines

If the large unit size is becoming a problem on i386, we can just use PAGE_SIZE as atom_size. Can you please verify that atom_size of 4k w/ EMBED also resolves the power issue?

We are enable Android ICS which is based on kernel 3.0.8. It seems there is no much change between 3.0.8 and the latest kernel.
With 3.0.8, although we could set percpualloc=embed, atom_size would becomes PMD_SIZE.
With our patch, we could do the experiment as we could configure percpu_alloc=4K easily. We would let you know the testing result of atom_size=4K && first_chunk_embedded.

arch/x86/kernel/setup_percpu.c void __init setup_per_cpu_areas(void) · line 203

Quoted 13 lines · Tejun Heo

arch/x86/kernel/setup_percpu.c +12 -0

+		/*
+		 * If PSE is available, use PMD_SIZE for atom_size so that
+		 * embedded percpu areas are aligned to PMD.  This, in the
+		 * future, can also allow using PMD mappings in vmalloc
+		 * area.  Use PAGE_SIZE on 32bit as vmalloc space is highly
+		 * contended and large vmalloc area alloc can easily fail.
+		 */
+		atom_size = PAGE_SIZE;
+#ifdef CONFIG_X86_64
+		if (cpu_has_pse)
+			atom_size = PMD_SIZE;
+#endif

Just make it PAGE_SIZE on i386 and PMD_SIZE on x86-64; keep the behavior consistent. !cpu_has_pse on x86-64 is an abnormal situation (I think it applies to Xen, pretty much.)

Other than that,

Acked-by H. Peter Anvin <[email protected]>

This patch makes x86_32 always use PAGE_SIZE as atom_size for embed first chunk allocator.

Quoted 2 lines · Tejun Heo

This patch makes x86_32 always use PAGE_SIZE as atom_size for embed first chunk allocator.

Would it not be easier to use the first #ifdef in the function to set pcpu_chosen_fc = PCPU_FC_PAGE?

You could just skip the whole PCPU_FC_PAGE block in the CONFIG_X86_32 case.

With the embed percpu first chunk allocator, x86 uses either PAGE_SIZE or PMD_SIZE for atom_size. PMD_SIZE is used when CPU supports PSE so that percpu areas are aligned to PMD mappings and possibly allow using PMD mappings in vmalloc areas in the future. Using larger atom_size doesn't waste actual memory; however, it does require larger vmalloc space allocation later on for !first chunks.

With reasonably sized vmalloc area, PMD_SIZE shouldn't be a problem but x86_32 at this point is anything but reasonable in terms of address space and using larger atom_size reportedly leads to frequent percpu allocation failures on certain setups.

As there is no reason to not use PMD_SIZE on x86_64 as vmalloc space is aplenty and most x86_64 configurations support PSE, fix the issue by always using PMD_SIZE on x86_64 and PAGE_SIZE on x86_32.

v2: drop cpu_has_pse test and make x86_64 always use PMD_SIZE and
x86_32 PAGE_SIZE as suggested by hpa.

Signed-off-by Tejun Heo <[email protected]>

Reported-by Yanmin Zhang <[email protected]>

Reported-by ShuoX Liu <[email protected]>

Acked-by H. Peter Anvin <[email protected]>

LKML-Reference: <[email protected]>
Cc: [email protected]

Quoted 19 lines · Tejun Heo

With the embed percpu first chunk allocator, x86 uses either PAGE_SIZE or PMD_SIZE for atom_size. PMD_SIZE is used when CPU supports PSE so that percpu areas are aligned to PMD mappings and possibly allow using PMD mappings in vmalloc areas in the future. Using larger atom_size doesn't waste actual memory; however, it does require larger vmalloc space allocation later on for !first chunks.

With reasonably sized vmalloc area, PMD_SIZE shouldn't be a problem but x86_32 at this point is anything but reasonable in terms of address space and using larger atom_size reportedly leads to frequent percpu allocation failures on certain setups.

As there is no reason to not use PMD_SIZE on x86_64 as vmalloc space is aplenty and most x86_64 configurations support PSE, fix the issue by always using PMD_SIZE on x86_64 and PAGE_SIZE on x86_32.

v2: drop cpu_has_pse test and make x86_64 always use PMD_SIZE and
x86_32 PAGE_SIZE as suggested by hpa.

Quoted 6 lines

Signed-off-by Tejun Heo <[email protected]>

Reported-by Yanmin Zhang <[email protected]>

Reported-by ShuoX Liu <[email protected]>

Acked-by H. Peter Anvin <[email protected]>

LKML-Reference: <[email protected]>
Cc: [email protected]

You're carrying this, or should I?

With the embed percpu first chunk allocator, x86 uses either PAGE_SIZE or PMD_SIZE for atom_size. PMD_SIZE is used when CPU supports PSE so that percpu areas are aligned to PMD mappings and possibly allow using PMD mappings in vmalloc areas in the future. Using larger atom_size doesn't waste actual memory; however, it does require larger vmalloc space allocation later on for !first chunks.

With reasonably sized vmalloc area, PMD_SIZE shouldn't be a problem but x86_32 at this point is anything but reasonable in terms of address space and using later atom_size reportedly leads to frequent percpu allocation failures on certain setups.

This patch makes x86_32 always use PAGE_SIZE as atom_size for embed first chunk allocator.

Signed-off-by Tejun Heo <[email protected]>

Reported-by Yanmin Zhang <[email protected]>

Reported-by ShuoX Liu <[email protected]>

LKML-Reference: <[email protected]>
Cc: [email protected]
---
Can you please verify this resolves the issue you guys are seeing? Once verified, I'll push it through percpu/for-3.5-fixes.

Thanks.

 arch/x86/kernel/setup_percpu.c |   14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

arch/x86/kernel/setup_percpu.c +13 -1

index 71f4727..ed96573 100644
@@ -185,10 +185,22 @@ void __init setup_per_cpu_areas(void)
 #endif
 	rc = -EINVAL;
 	if (pcpu_chosen_fc != PCPU_FC_PAGE) {
-		const size_t atom_size = cpu_has_pse ? PMD_SIZE : PAGE_SIZE;
 		const size_t dyn_size = PERCPU_MODULE_RESERVE +
 			PERCPU_DYNAMIC_RESERVE - PERCPU_FIRST_CHUNK_RESERVE;
+		size_t atom_size;
 
+		/*
+		 * If PSE is available, use PMD_SIZE for atom_size so that
+		 * embedded percpu areas are aligned to PMD.  This, in the
+		 * future, can also allow using PMD mappings in vmalloc
+		 * area.  Use PAGE_SIZE on 32bit as vmalloc space is highly
+		 * contended and large vmalloc area alloc can easily fail.
+		 */
+		atom_size = PAGE_SIZE;
+#ifdef CONFIG_X86_64
+		if (cpu_has_pse)
+			atom_size = PMD_SIZE;
+#endif
 		rc = pcpu_embed_first_chunk(PERCPU_FIRST_CHUNK_RESERVE,
 					    dyn_size, atom_size,
 					    pcpu_cpu_distance,

Quoted 7 lines

Quoted 3 lines

We can't fix FC_PAGE power regression. If we do so, we need contact many hardware architects. Current kernel supports FC_PAGE and PMD_SIZE, why not to allow admin to choose other values?

If this is something which is met in the field commonly, we need to fix the default behavior rather than introducing some arcane boot param.

We just add a new value input method instead of introducing new parameter.

Quoted 4 lines

IIRC, the reasons PMD_SIZE is used for atom_size are so that
percpu areas are aligned to PSE mapping, maybe later we can make use of PSE mapping in vmalloc area too, and it didn't seem to hurt anything.

Well, vmalloc area might use different prot to map physical pages. So sharing one PMD huge page by many vmalloc areas might be not good.