Skip to contentSkip to conversation

32/64-bit NUMA consolidation behavior regresion

DISCUSSION Dave Hansen 2 participants 3 messages Nov 29, 2012 – Nov 30, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi Tejun,

I was bisecting a boot problem on a 32-bit NUMA kernel and it bisected down to commit 8db78cc4. It turns out that, with this patch, pcpu_need_numa() changed its return value on my system from 1 to 0. What that basically meant was that we stopped using the remapped lowmem areas for percpu data.

My system is just qemu booted with:

-smp 8 -m 8192 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7

Watch the "PERCPU:" line early in boot, and you can see the "Embedded" come and go with or without your patch:

[    0.000000] PERCPU: Embedded 11 pages/cpu @f3000000 s30592 r0 d14464
vs
[    0.000000] PERCPU: 11 4K pages/cpu @f83fe000 s30592 r0 d14464

I believe this has to do with the hunks in your patch that do:

-#ifdef CONFIG_X86_64

        init_cpu_to_node();

-#endif
...
-#ifdef CONFIG_X86_32
-DEFINE_EARLY_PER_CPU(int, x86_cpu_to_node_map, 0); -#else
DEFINE_EARLY_PER_CPU(int, x86_cpu_to_node_map, NUMA_NO_NODE);
-#endif
EXPORT_EARLY_PER_CPU_SYMBOL(x86_cpu_to_node_map);

I don't have a fix handy because I'm working on the original problem, but I just happened to run across this during a bisect.

#1 lore raw

Hello, Dave.

Quoted 10 lines · Dave Hansen

My system is just qemu booted with:

-smp 8 -m 8192 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7

Watch the "PERCPU:" line early in boot, and you can see the "Embedded" come and go with or without your patch:

[    0.000000] PERCPU: Embedded 11 pages/cpu @f3000000 s30592 r0 d14464
vs
[    0.000000] PERCPU: 11 4K pages/cpu @f83fe000 s30592 r0 d14464

...

Quoted 2 lines

I don't have a fix handy because I'm working on the original problem, but I just happened to run across this during a bisect.

Just tested 3.7-rc7 w/ qemu and it works as expected here.

Can you please boot with the following debug patch and report the boot message before and after?

Thanks.

arch/x86/kernel/setup_percpu.c +5 -1
index 5cdff03..1133dc8 100644
@@ -71,9 +71,13 @@ static bool __init pcpu_need_numa(void)
 	for_each_possible_cpu(cpu) {
 		int node = early_cpu_to_node(cpu);
 
+		printk("XXX pcpu_need_numa: cpu%d@%d online=%d ND=%p\n",
+		       cpu, node, node_online(node), NODE_DATA(node));
 		if (node_online(node) && NODE_DATA(node) &&
-		    last && last != NODE_DATA(node))
+		    last && last != NODE_DATA(node)) {
+			printk("XXX need numa\n");
 			return true;
+		}
 
 		last = NODE_DATA(node);
 	}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
Quoted 11 lines · Tejun Heo
Quoted 10 lines · Dave Hansen

My system is just qemu booted with:

-smp 8 -m 8192 -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7

Watch the "PERCPU:" line early in boot, and you can see the "Embedded" come and go with or without your patch:

[    0.000000] PERCPU: Embedded 11 pages/cpu @f3000000 s30592 r0 d14464
vs
[    0.000000] PERCPU: 11 4K pages/cpu @f83fe000 s30592 r0 d14464
Quoted 8 lines

...

Quoted 2 lines

I don't have a fix handy because I'm working on the original problem, but I just happened to run across this during a bisect.

Just tested 3.7-rc7 w/ qemu and it works as expected here.

Can you please boot with the following debug patch and report the boot message before and after?

Hi Tejun,

I just tested with 3.7-rc7 and I'm seeing the expected behavior now. Looks like it got fixed along the way somewhere. I was bisecting way back in the 2.6.3x's. Sorry of the noise.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>