Skip to contentSkip to conversation

SMP busted on non-cpu-hotplug systems

DISCUSSION David S. Miller 3 participants 8 messages Mar 25, 2006

lore
Share & tools
lore
#0 lore raw

I just noticed this on sparc64, as I lost 31 cpus on my Niagara box due to it :)

boot_cpu_init() sets the boot processor ID in cpu_present_map.

But fixup_cpu_present_map() will only populate the cpu_present_map if it is empty, which it won't be because of what boot_cpu_init() just did.

#1 lore raw
Quoted 10 lines · David S. Miller

I just noticed this on sparc64, as I lost 31 cpus on my Niagara box due to it :)

boot_cpu_init() sets the boot processor ID in cpu_present_map.

But fixup_cpu_present_map() will only populate the cpu_present_map if it is empty, which it won't be because of what boot_cpu_init() just did.

oops. I guess most architectures set cpu_present_map while bringing up the APs.

I think it'd be cleanest to require that the arch do that - fixup_cpu_present_map() looks like a bit of a hack.

I guess if we want to perpetuate fixup_cpu_present_map() then we should
teach it to ignore the boot cpu. (cpus_weight(&cpu_present_map) == 1)
would do that.

#2 lore raw

From: Andrew Morton <[email protected]>
Date: Sat, 25 Mar 2006 03:47:44 -0800

Quoted 2 lines

I think it'd be cleanest to require that the arch do that - fixup_cpu_present_map() looks like a bit of a hack.

Indeed it does. I'm planning on doing someting like this for sparc64:

arch/sparc64/kernel/smp.c +4 -1
index 1b6e2ad..7dc28a4 100644
@@ -1298,6 +1298,7 @@ void __init smp_prepare_cpus(unsigned in
 		while (!cpu_find_by_instance(instance, NULL, &mid)) {
 			if (mid != boot_cpu_id) {
 				cpu_clear(mid, phys_cpu_present_map);
+				cpu_clear(mid, cpu_present_map);
 				if (num_possible_cpus() <= max_cpus)
 					break;
 			}
@@ -1332,8 +1333,10 @@ void __init smp_setup_cpu_possible_map(v
 
 	instance = 0;
 	while (!cpu_find_by_instance(instance, NULL, &mid)) {
-		if (mid < NR_CPUS)
+		if (mid < NR_CPUS) {
 			cpu_set(mid, phys_cpu_present_map);
+			cpu_set(mid, cpu_present_map);
+		}
 		instance++;
 	}
 }
#3 lore raw
Quoted 9 lines · David S. Miller

From: Andrew Morton <[email protected]>
Date: Sat, 25 Mar 2006 03:47:44 -0800

Quoted 2 lines

I think it'd be cleanest to require that the arch do that - fixup_cpu_present_map() looks like a bit of a hack.

Indeed it does. I'm planning on doing someting like this for sparc64:

Fair enough. fixup_cpu_present_map() is an elaborate no-op now. I'll nuke it and will send a heads-up to the arch maintainers.

#4 lore raw
In reply to Andrew Morton
Quoted 15 lines · Andrew Morton
Quoted 10 lines · David S. Miller

I just noticed this on sparc64, as I lost 31 cpus on my Niagara box due to it :)

boot_cpu_init() sets the boot processor ID in cpu_present_map.

But fixup_cpu_present_map() will only populate the cpu_present_map if it is empty, which it won't be because of what boot_cpu_init() just did.

oops. I guess most architectures set cpu_present_map while bringing up the APs.

Quoted 6 lines

I think it'd be cleanest to require that the arch do that - fixup_cpu_present_map() looks like a bit of a hack.

I guess if we want to perpetuate fixup_cpu_present_map() then we should
teach it to ignore the boot cpu. (cpus_weight(&cpu_present_map) == 1)
would do that.

At setup_arch() time, we initialise cpu_possible_map to contain the CPUs the system might have.

We then call smp_prepare_boot_cpu() which marks the boot cpu in both cpu_present_map and cpu_online_map.

Eventually, we call smp_prepare_cpus(), where an architecture may populate cpu_present_map to indicate which cpus are actually present, and following this we call fixup_cpu_present_map().

With your proposed change, if a SMP system with has 4 possible CPUs was passed maxcpus=1, cpu_possible_map may well have 4 CPUs, and cpu_present_map will only contain the one. However, due to the fixup_cpu_present_map(), it will say "oh only one CPU, we need to populate the others" and so you'd actually try to boot all 4.

So no, this doesn't work. Isn't it about time the pre-CPU hotplug SMP stuff was updated, rather than trying to messily support two different SMP initialisation methodologies in generic code with band aid plasters all over?

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core
#5 lore raw

From: Russell King <[email protected]>
Date: Sat, 25 Mar 2006 12:05:46 +0000

Quoted 4 lines

So no, this doesn't work. Isn't it about time the pre-CPU hotplug SMP stuff was updated, rather than trying to messily support two different SMP initialisation methodologies in generic code with band aid plasters all over?

Agreed.

#6 lore raw
In reply to Russell King
Quoted 20 lines · Russell King
Quoted 18 lines · Andrew Morton
Quoted 10 lines · David S. Miller

I just noticed this on sparc64, as I lost 31 cpus on my Niagara box due to it :)

boot_cpu_init() sets the boot processor ID in cpu_present_map.

But fixup_cpu_present_map() will only populate the cpu_present_map if it is empty, which it won't be because of what boot_cpu_init() just did.

oops. I guess most architectures set cpu_present_map while bringing up the APs.

I think it'd be cleanest to require that the arch do that - fixup_cpu_present_map() looks like a bit of a hack.

Quoted 6 lines
Quoted 3 lines

I guess if we want to perpetuate fixup_cpu_present_map() then we should
teach it to ignore the boot cpu. (cpus_weight(&cpu_present_map) == 1)
would do that.

At setup_arch() time, we initialise cpu_possible_map to contain the CPUs the system might have.

OK.

Quoted 2 lines

We then call smp_prepare_boot_cpu() which marks the boot cpu in both cpu_present_map and cpu_online_map.

OK.

Quoted 3 lines

Eventually, we call smp_prepare_cpus(), where an architecture may populate cpu_present_map to indicate which cpus are actually present, and following this we call fixup_cpu_present_map().

OK.

Quoted 1 line

With your proposed change,

Which proposed change? I proposed two.

Quoted 5 lines

if a SMP system with has 4 possible CPUs
was passed maxcpus=1, cpu_possible_map may well have 4 CPUs, and cpu_present_map will only contain the one. However, due to the fixup_cpu_present_map(), it will say "oh only one CPU, we need to populate the others" and so you'd actually try to boot all 4.

The change we appear to be going with is to remove fixup_cpu_present_map() which appears to address this.

Quoted 1 line

So no, this doesn't work.

What doesn't work?

Quoted 4 lines

Isn't it about time the pre-CPU hotplug SMP
stuff was updated, rather than trying to messily support two different SMP initialisation methodologies in generic code with band aid plasters all over?

What two methodologies? arch-doing-it and fixup_cpu_present_map() doing it?

#7 lore raw
Quoted 4 lines · Andrew Morton
Quoted 1 line · Russell King

With your proposed change,

Which proposed change? I proposed two.

The latter change.

Quoted 6 lines
Quoted 5 lines

if a SMP system with has 4 possible CPUs
was passed maxcpus=1, cpu_possible_map may well have 4 CPUs, and cpu_present_map will only contain the one. However, due to the fixup_cpu_present_map(), it will say "oh only one CPU, we need to populate the others" and so you'd actually try to boot all 4.

Quoted 6 lines

The change we appear to be going with is to remove fixup_cpu_present_map() which appears to address this.

Quoted 1 line

So no, this doesn't work.

What doesn't work?

The situation I described and you've quoted in the bulk of the above quote.

Quoted 6 lines
Quoted 4 lines

Isn't it about time the pre-CPU hotplug SMP
stuff was updated, rather than trying to messily support two different SMP initialisation methodologies in generic code with band aid plasters all over?

What two methodologies? arch-doing-it and fixup_cpu_present_map() doing it?

What I'm referring to is the pre-CPU hotplug SMP initialisation methodology and the post-CPU hotplug SMP initialisation methodology, which I think is covered by "two different SMP initialisation methodologies".

The two methodologies had entirely different ways of bringing up the non-boot CPUs to the extent of using the cpu_*_map variables in different ways. However, now that I come to look again at x86, the situation does appear to have improved somewhat over the last year or so since I last looked (which was when I sorted out the ARM SMP support.)

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core