2.6.2-rc2-mm2 CPU Hotplug: cpu_active_map
PATCH 1/4 2.6.2-rc2-mm2 CPU Hotplug: cpu_active_map
Hi Andrew et al.
Forward ported to -mm tree, so there might still be bugs.
Wanted to get these aired more widely. Especially get Nick exposed to the sched parts.
Name: Create cpu_active_map
Author: Rusty Russell
Status: Booted on 2.6.2-rc2-bk2
D: When CPUs are going down, there is a time when cpu_online(cpu) is
D: false, but they are still scheduling and responding to interrupts
D: (we are migrating things off the CPU, shutting down per-cpu
D: threads, etc). It turns out that RCU cares about these CPUs, so
D: the decision was made to expose this mask (previously internal to x86,
D: and only used for IPIs).
D:
D: The semantics of this mask are as follows: D: 1) For platforms without hot unplug of CPUs: cpu_active_map == D: cpu_online_map. D: 2) For the others, they are equal except for a CPU which is going D: down: cpu_online_map gets cleared by __cpu_disable(), cpu_ipi_map D: gets cleared in __cpu_die() once the CPU is no longer responding D: to interrupts.
include/linux/cpumask.h +7 -0
@@ -43,6 +43,13 @@ typedef unsigned long cpumask_t;extern cpumask_t cpu_online_map;extern cpumask_t cpu_possible_map; +#ifdef CONFIG_HOTPLUG_CPU +/* Online, or on its way down but still receiving interrupts. */ +extern cpumask_t cpu_active_map; +#else +#define cpu_active_map cpu_online_map +#endif +#define num_online_cpus () cpus_weight(cpu_online_map)#define cpu_online (cpu) cpu_isset(cpu, cpu_online_map)#define cpu_possible (cpu) cpu_isset(cpu, cpu_possible_map)
kernel/rcupdate.c +1 -1
@@ -110,7 +110,7 @@ static void rcu_start_batch(long newbatc !cpus_empty(rcu_ctrlblk.rcu_cpu_mask)) {return ; } - rcu_ctrlblk.rcu_cpu_mask = cpu_online_map; + rcu_ctrlblk.rcu_cpu_mask = cpu_active_map; }/*
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Quoted 6 lines · Rusty Russell
D: When CPUs are going down, there is a time when
cpu_online(cpu)is
D: false, but they are still scheduling and responding to interrupts
D: (we are migrating things off the CPU, shutting down per-cpu
D: threads, etc). It turns out that RCU cares about these CPUs, so
D: the decision was made to expose this mask (previously internal to x86,
D: and only used for IPIs).
these kinds of problems could be avoided by making the CPU-off as much of an atomic operation as possible. The less atomic it is, the more kernel code is exposed to the transitional state - and since this is a rare situation it will always have quality problems. Is there any killer argument that makes it impossible to down a CPU atomically? Kernel threads can get their callbacks on other CPUs just fine. Other tasks should not care. If the migrate-off operation is done 100% atomically then zero knowledge is needed by unrelated scheduler code about the act of disabling a CPU.
Ingo
In message <[email protected]> you write:
Quoted 10 lines · Rusty Russell
Quoted 7 lines
D: When CPUs are going down, there is a time when
cpu_online(cpu)is
D: false, but they are still scheduling and responding to interrupts
D: (we are migrating things off the CPU, shutting down per-cpu
D: threads, etc). It turns out that RCU cares about these CPUs, so
D: the decision was made to expose this mask (previously internal to x86,
D: and only used for IPIs).
Quoted 6 lines
these kinds of problems could be avoided by making the CPU-off as much of an atomic operation as possible. The less atomic it is, the more kernel code is exposed to the transitional state - and since this is a rare situation it will always have quality problems. Is there any killer argument that makes it impossible to down a CPU atomically? Kernel threads can get their callbacks on other CPUs just fine.
Well, we could grab control of the machine, unconditionally move all the threads and kill the cpu.
We'd need a lot of locks, but we could just use a bogolock (a-la module.c's schedule a thread per cpu and tell them all to disable interrupts) which has the same effect of grabbing every spinlock.
It means that kernel threads must not use smp_processor_id(), because it might change under them (get_cpu() would protect from this). You'd still need explicit shutdown code.
Quoted 3 lines
Other tasks should not care. If the migrate-off operation is done 100% atomically then zero knowledge is needed by unrelated scheduler code about the act of disabling a CPU.
No, migration code still needs to check the cpu is still up after you've slept.
I'll sleep on it and see if it still sounds like a good idea in the morning 8)
Rusty.
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
Quoted 7 lines · Rusty Russell
D: The semantics of this mask are as follows: D: 1) For platforms without hot unplug of CPUs: cpu_active_map == D: cpu_online_map. D: 2) For the others, they are equal except for a CPU which is going D: down: cpu_online_map gets cleared by __cpu_disable(), cpu_ipi_map D: gets cleared in __cpu_die() once the CPU is no longer responding D: to interrupts.
Small typo, but I believe you meant to type 'cpu_active_map' in part 2, not 'cpu_ipi_map'.
Cheers!
-Matt