Hotplug CPU Infrastructure
Subsystems (13)
PATCH 1/1 Hotplug CPU Infrastructure
A fairly big patch from the -mm tree (with a slightly different scheduler part), but 99% is under CONFIG_HOTPLUG_CPU or similar noops.
Sorry for the delay, I tested this with and without CONFIG_HOTPLUG_CPU to make sure I didn't break your tree again.
Name: Hotplug CPUs for Linus with Atomic Takedown of CPUs
Status: Tested on 2.6.4-rc2-bk5
Depends: Hotcpu/bogolock.patch.gz
Depends: Hotcpu/drain_local_pages.patch.gz
Depends: Hotcpu/more-minor-cleanups.patch.gz
Depends: Hotcpu/minor-slab-cleanups.patch.gz
Depends: Hotcpu/bogolock-do_stop-race.patch.gz
Depends: Hotcpu/hotcpu-radix-tree.patch.gz
We take CPUs down atomically: freeze the machine, and then call __cpu_disable(), which should arrange that this CPU no longer receives interrupts (by interrupt controller reprogramming etc). The machine is released, the dead cpu returns to the idle thread, and it can be given the coup de grace from another cpu in __cpu_die().
Any code which refers to num_online_cpus(), cpu_online(), or uses cpu_online_map in any way should do so with preempt disabled, so that CPUs don't go down.
Kernel threads may be rudely torn off CPUs they were bound to: they should not rely on smp_processor_id() without checking this hasn't happened (eg. disable preempt, check cpu still online, do work).
Thanks to Ingo Molnar for the atomicity idea, and everyone else on the hotplug mailing list for testing, bugfixing and brainsweat.
Changes which are not obvious noops without CONFIG_HOTPLUG_CPU:
- cpu.c: minor bugfix: grab the cpucontrol sem around notifier registration.
- sched.c: hold irqs off across move_task_away call in migration thread, so CPUs don't go offline. So move_task_away doesn't need local_irq_save.
- sched.c: migration thread can't assume we're still on CPU: it might have gone straight back down.
- sched.c: make sure migration thread has reached high prio before returning from CPU_ONLINE, otherwise stop_machine can't use it.
- sched.c: no reason to do sched notifier last, remove priority.
- softirq.c: don't assume cpu hasn't gone straight back down. Hold preempt disabled over do_softirq so cpu doesn't go down, and check beforehand so we don't do_softirq() on wrong cpu.
- softirq.c: remove double assignment
- workqueue.c: remove BUG_ON() in case cpu goes back down, and hence unused cpu var.
- workqueue.c: Add name and list to workqueue structure.
Changes in detail:
- drivers/base/cpu.c: Add an "online" attribute.
- drivers/scsi/scsi.c: drain the scsi_done_q into current CPU on CPU_DEAD
- fs/buffer.c: release buffer heads from bh_lrus on CPU_DEAD
- linux/cpu.h: Add cpu_down(). Add hotcpu_notifier convenience macro. Add cpu_is_offline() for when you know a cpu was (once) online: this is a constant if !CONFIG_HOTPLUG_CPU.
- linux/interrupt.h: Add tasklet_kill_immediate for RCU tasklet takedown on dead CPU.
- linux/mmzone.h: keep pointer to kswapd, in case we need to move it.
- linux/notifier.h: remove CPU_OFFLINE: we just get CPU_DEAD now.
- linux/sched.h: Add migrate_all_tasks() for cpu-down. Add migrate_to_cpu for kernel threads to escape.
- kernel/cpu.c: Take cpucontrol lock around notifiers: notifiers have no locking themselves. Implement cpu_down() and code to call sbin_hotplug.
- kernel/rcupdate.c: Comment on why online map use is OK, and move rcu batch when cpu dies.
- kernel/sched.c:
Take hotplug lock in sys_sched_setaffinity.
Return cpus_allowed masked by cpu_possible_map, not cpu_online_map in sys_sched_getaffinity, otherwise tasks can't know about whether they can run on currently-offline cpus.
Migration thread can be taken off CPU when CPU goes down: remove two uses of smp_processor_id() to make that safe.
Implement migrate_all_tasks() to push tasks off the dying cpu.
Add callbacks to stop migration thread.
- kernel/softirq.c:
Handle case where ksoftirqd is kicked off dead CPU: disable preempt to prevent it, then check if cpu already offline. Simpler than recoding do_softirq() for this wierd case.
Implement tasklet_kill_immediate for RCU.
Code to shut down ksoftirqd, and take over dead cpu's tasklets.
Remove double initialization in CPU_UP_PREPARE.
- kernel/timer.c:
Code to migrate timers off dead CPU.
- kernel/workqueue.c:
Keep all workqueue is list, with their name: when cpus come up we need to create new threads for each one.
Lock cpu hotplug in flush_workqueue for simplicity.
create_workqueue_thread doesn't need name arg, now in struct.
Lock cpu hotplug when creating and destroying workqueues.
cleanup_workqueue_thread needs to block irqs: can now be called on live workqueues.
Implement callbacks to add and delete threads, and take over work.
- mm/page_alloc.c:
Drain local pages on CPU which is dead.
- mm/slab.c:
Free cache blocks and correct stats one CPU is dead.
- mm/swap.c:
Fix up committed stats and drain lru cache when cpu dead.
- mm/vmscan.c:
Keep kswapd on cpus within node unless all CPUs go offline, and restore when they come back.
Record kswapd tasks for each pgdat upon creation.
- net/core/dev.c:
Drain skb queues to this cpu when another cpu is dead.
- net/core/flow.c:
Call __flow_cache_shrink() to shrink cache to zero when CPU dead.
drivers/base/cpu.c +43 -0
@@ -7,6 +7,7 @@#include <linux/init.h> #include <linux/cpu.h> #include <linux/topology.h> +#include <linux/device.h> struct sysdev_class cpu_sysdev_class = {
@@ -14,6 +15,46 @@ struct sysdev_class cpu_sysdev_class = { };EXPORT_SYMBOL (cpu_sysdev_class); +#ifdef CONFIG_HOTPLUG_CPU +static ssize_t show_online (struct sys_device *dev,char *buf) +{ + struct cpu *cpu = container_of(dev, struct cpu, sysdev); + +return sprintf (buf,"%u\n" , !!cpu_online (cpu->sysdev.id)); +} + +static ssize_t store_online (struct sys_device *dev,const char *buf, +size_t count) +{ + struct cpu *cpu = container_of(dev, struct cpu, sysdev); +ssize_t ret; + +switch (buf[0 ]) { +case '0' : + ret =cpu_down (cpu->sysdev.id); +break ; +case '1' : + ret =cpu_up (cpu->sysdev.id); +break ; +default : + ret = -EINVAL ; + } + +if (ret >=0 ) + ret = count; +return ret; +} +static SYSDEV_ATTR(online, 0600, show_online, store_online); + +static void __init register_cpu_control (struct cpu *cpu) +{ +sysdev_create_file (&cpu->sysdev, &attr_online); +} +#else /* ... !CONFIG_HOTPLUG_CPU */ +static inline void register_cpu_control (struct cpu *cpu) +{ +} +#endif /* CONFIG_HOTPLUG_CPU */ /* * register_cpu - Setup a driverfs device for a CPU.
@@ -34,6 +75,8 @@ int __init register_cpu(struct cpu *cpu, error =sysfs_create_link (&root->sysdev.kobj, &cpu->sysdev.kobj,kobject_name (&cpu->sysdev.kobj)); +if (!error) +register_cpu_control (cpu);return error; }
drivers/scsi/scsi.c +36 -0
@@ -53,6 +53,8 @@#include <linux/spinlock.h> #include <linux/kmod.h> #include <linux/interrupt.h> +#include <linux/notifier.h> +#include <linux/cpu.h> #include <scsi/scsi_host.h> #include "scsi.h"
@@ -1130,6 +1132,38 @@ int scsi_device_cancel(struct scsi_devicreturn 0 ; } +#ifdef CONFIG_HOTPLUG_CPU +static int scsi_cpu_notify (struct notifier_block *self, +unsigned long action,void *hcpu) +{ +int cpu = (unsigned long )hcpu; + +switch (action) { +case CPU_DEAD : +/* Drain scsi_done_q. */ +local_irq_disable (); +list_splice_init (&per_cpu (scsi_done_q, cpu), + &__get_cpu_var (scsi_done_q)); +raise_softirq_irqoff (SCSI_SOFTIRQ ); +local_irq_enable (); +break ; +default : +break ; + } +return NOTIFY_OK ; +} + +static struct notifier_block __devinitdata scsi_cpu_nb = { + .notifier_call = scsi_cpu_notify, +}; + +#define register_scsi_cpu () register_cpu_notifier(&scsi_cpu_nb) +#define unregister_scsi_cpu () unregister_cpu_notifier(&scsi_cpu_nb) +#else +#define register_scsi_cpu () +#define unregister_scsi_cpu () +#endif /* CONFIG_HOTPLUG_CPU */ +MODULE_DESCRIPTION ("SCSI core" );MODULE_LICENSE ("GPL" );
@@ -1164,6 +1198,7 @@ static int __init init_scsi(void)devfs_mk_dir ("scsi" );open_softirq (SCSI_SOFTIRQ , scsi_softirq,NULL ); +register_scsi_cpu ();printk (KERN_NOTICE "SCSI subsystem initialized\n" );return 0 ;
@@ -1191,6 +1226,7 @@ static void __exit exit_scsi(void)devfs_remove ("scsi" );scsi_exit_procfs ();scsi_exit_queue (); +unregister_scsi_cpu (); }subsys_initcall (init_scsi);
fs/buffer.c +21 -0
@@ -3024,6 +3024,26 @@ init_buffer_head(void *data, kmem_cache_ } } +#ifdef CONFIG_HOTPLUG_CPU +static void buffer_exit_cpu (int cpu) +{ +int i; +struct bh_lru *b = &per_cpu (bh_lrus, cpu); + +for (i =0 ; i <BH_LRU_SIZE ; i++) { +brelse (b->bhs[i]); + b->bhs[i] =NULL ; + } +} + +static int buffer_cpu_notify (struct notifier_block *self, +unsigned long action,void *hcpu) +{ +if (action ==CPU_DEAD ) +buffer_exit_cpu ((unsigned long )hcpu); +return NOTIFY_OK ; +} +#endif /* CONFIG_HOTPLUG_CPU */ void __init buffer_init (void ) {
@@ -3041,6 +3061,7 @@ void __init buffer_init(void)*/ nrpages = (nr_free_buffer_pages () *10 ) /100 ; max_buffer_heads = nrpages * (PAGE_SIZE /sizeof (struct buffer_head )); +hotcpu_notifier (buffer_cpu_notify,0 ); }EXPORT_SYMBOL (__bforget);
include/linux/cpu.h +1 -0
@@ -63,6 +63,7 @@ extern struct semaphore cpucontrol; static struct notifier_block fn##_nb = { fn, pri }; \ register_cpu_notifier(&fn##_nb); \ } +int cpu_down (unsigned int cpu);#define cpu_is_offline (cpu) unlikely(!cpu_online(cpu))#else #define lock_cpu_hotplug () do { } while (0)
include/linux/interrupt.h +1 -0
@@ -211,6 +211,7 @@ static inline void tasklet_hi_enable(str }extern void tasklet_kill (struct tasklet_struct *t); +extern void tasklet_kill_immediate (struct tasklet_struct *t,unsigned int cpu);extern void tasklet_init (struct tasklet_struct *t,void (*func)(unsigned long ),unsigned long data);
include/linux/mmzone.h +1 -0
@@ -213,6 +213,7 @@ typedef struct pglist_data {int node_id;struct pglist_data *pgdat_next;wait_queue_head_t kswapd_wait; +struct task_struct *kswapd; } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)
include/linux/notifier.h +0 -1
@@ -63,7 +63,6 @@ extern int notifier_call_chain(struct no#define CPU_ONLINE 0x0002/* CPU (unsigned)v is up */ #define CPU_UP_PREPARE 0x0003/* CPU (unsigned)v coming up */ #define CPU_UP_CANCELED 0x0004/* CPU (unsigned)v NOT coming up */ -#define CPU_OFFLINE 0x0005/* CPU (unsigned)v offline (still scheduling) */ #define CPU_DEAD 0x0006/* CPU (unsigned)v dead */ #endif /* __KERNEL__ */
include/linux/sched.h +4 -0
@@ -547,6 +547,10 @@ extern void node_nr_running_init(void);#define node_nr_running_init () {}#endif +/* Move tasks off this (offline) CPU onto another. */ +extern void migrate_all_tasks (void ); +/* Try to move me here, if possible. */ +void migrate_to_cpu (int dest_cpu);extern void set_user_nice (task_t *p,long nice);extern int task_prio (task_t *p);extern int task_nice (task_t *p);
kernel/cpu.c +140 -6
180 lines not on this page · raw message
kernel/rcupdate.c +60 -1
79 lines not on this page · raw message
kernel/sched.c +83 -13
187 lines not on this page · raw message
kernel/softirq.c +81 -5
122 lines not on this page · raw message
kernel/timer.c +72 -1
85 lines not on this page · raw message
kernel/workqueue.c +106 -9
206 lines not on this page · raw message
lib/radix-tree.c +25 -0
@@ -24,6 +24,8 @@#include <linux/radix-tree.h> #include <linux/percpu.h> #include <linux/slab.h> +#include <linux/notifier.h> +#include <linux/cpu.h> #include <linux/gfp.h> #include <linux/string.h>
@@ -420,6 +422,28 @@ static __init void radix_tree_init_maxin height_to_maxindex[i] =__maxindex (i); } +#ifdef CONFIG_HOTPLUG_CPU +static int radix_tree_callback (struct notifier_block *nfb, +unsigned long action, +void *hcpu) +{ +int cpu = (long )hcpu; +struct radix_tree_preload *rtp; + +/* Free per-cpu pool of perloaded nodes */ +if (action ==CPU_DEAD ) { + rtp = &per_cpu (radix_tree_preloads, cpu); +while (rtp->nr) { +kmem_cache_free (radix_tree_node_cachep, + rtp->nodes[rtp->nr-1 ]); + rtp->nodes[rtp->nr-1 ] =NULL ; + rtp->nr--; + } + } +return NOTIFY_OK ; +} +#endif /* CONFIG_HOTPLUG_CPU */ +void __init radix_tree_init (void ) { radix_tree_node_cachep = kmem_cache_create("radix_tree_node",
@@ -428,4 +452,5 @@ void __init radix_tree_init(void)if (!radix_tree_node_cachep)panic ("Failed to create radix_tree_node cache\n" );radix_tree_init_maxindex (); +hotcpu_notifier (radix_tree_callback,0 ); }
mm/page_alloc.c +20 -0
@@ -1550,9 +1550,29 @@ struct seq_operations vmstat_op = {#endif /* CONFIG_PROC_FS */ +#ifdef CONFIG_HOTPLUG_CPU +static int page_alloc_cpu_notify (struct notifier_block *self, +unsigned long action,void *hcpu) +{ +int cpu = (unsigned long )hcpu; +long *count; + +if (action ==CPU_DEAD ) { +/* Drain local pagecache count. */ + count = &per_cpu (nr_pagecache_local, cpu); +atomic_add (*count, &nr_pagecache); + *count =0 ; +local_irq_disable (); +__drain_pages (cpu); +local_irq_enable (); + } +return NOTIFY_OK ; +} +#endif /* CONFIG_HOTPLUG_CPU */ void __init page_alloc_init (void ) { +hotcpu_notifier (page_alloc_cpu_notify,0 ); }/*
mm/slab.c +26 -6
@@ -589,12 +589,19 @@ static void start_cpu_timer(int cpu) } } -/* -* Note: if someone calls kmem_cache_alloc() on the new -* cpu before the cpuup callback had a chance to allocate -* the head arrays, it will oops. -* Is CPU_ONLINE early enough? -*/ +#ifdef CONFIG_HOTPLUG_CPU +static void stop_cpu_timer (int cpu) +{ +struct timer_list *rt = &per_cpu (reap_timers, cpu); + +if (rt->function) { +del_timer_sync (rt); +WARN_ON (timer_pending (rt)); + rt->function =NULL ; + } +} +#endif +static int __devinit cpuup_callback (struct notifier_block *nfb,unsigned long action,void *hcpu)
@@ -630,18 +637,28 @@ static int __devinit cpuup_callback(strucase CPU_ONLINE :start_cpu_timer (cpu);break ; +#ifdef CONFIG_HOTPLUG_CPU +case CPU_DEAD : +stop_cpu_timer (cpu); +/* fall thru */ case CPU_UP_CANCELED :down (&cache_chain_sem);list_for_each_entry (cachep, &cache_chain, next) {struct array_cache *nc; +spin_lock_irq (&cachep->spinlock); +/* cpu is dead; no one can alloc from it. */ nc = cachep->array[cpu]; cachep->array[cpu] =NULL ; + cachep->free_limit -= cachep->batchcount; +free_block (cachep,ac_entry (nc), nc->avail); +spin_unlock_irq (&cachep->spinlock);kfree (nc); }up (&cache_chain_sem);break ; +#endif }return NOTIFY_OK ; bad:
@@ -1486,6 +1503,9 @@ int kmem_cache_destroy (kmem_cache_t * creturn 1 ; } +/* no cpu_online check required here since we clear the percpu +* array on cpu offline and set this to NULL. +*/ for (i =0 ; i <NR_CPUS ; i++)kfree (cachep->array[i]);
mm/swap.c +35 -1
@@ -27,6 +27,9 @@#include <linux/module.h> #include <linux/percpu_counter.h> #include <linux/percpu.h> +#include <linux/cpu.h> +#include <linux/notifier.h> +#include <linux/init.h> /* How many pages do we try to swap or page in/out together? */ int page_cluster;
@@ -381,7 +384,37 @@ void vm_acct_memory(long pages)preempt_enable (); }EXPORT_SYMBOL (vm_acct_memory); -#endif + +#ifdef CONFIG_HOTPLUG_CPU +static void lru_drain_cache (unsigned int cpu) +{ +struct pagevec *pvec = &per_cpu (lru_add_pvecs, cpu); + +/* CPU is dead, so no locking needed. */ +if (pagevec_count (pvec)) +__pagevec_lru_add (pvec); + pvec = &per_cpu (lru_add_active_pvecs, cpu); +if (pagevec_count (pvec)) +__pagevec_lru_add_active (pvec); +} + +/* Drop the CPU's cached committed space back into the central pool. */ +static int cpu_swap_callback (struct notifier_block *nfb, +unsigned long action, +void *hcpu) +{ +long *committed; + + committed = &per_cpu (committed_space, (long )hcpu); +if (action ==CPU_DEAD ) { +atomic_add (*committed, &vm_committed_space); + *committed =0 ; +lru_drain_cache ((long )hcpu); + } +return NOTIFY_OK ; +} +#endif /* CONFIG_HOTPLUG_CPU */ +#endif /* CONFIG_SMP */ #ifdef CONFIG_SMP void percpu_counter_mod (struct percpu_counter *fbc,long amount)
@@ -420,4 +453,5 @@ void __init swap_setup(void)* Right now other parts of the system means that we * _really_ don't want to cluster much more */ +hotcpu_notifier (cpu_swap_callback,0 ); }
mm/vmscan.c +29 -1
@@ -30,6 +30,8 @@#include <linux/backing-dev.h> #include <linux/rmap-locking.h> #include <linux/topology.h> +#include <linux/cpu.h> +#include <linux/notifier.h> #include <asm/pgalloc.h> #include <asm/tlbflush.h>
@@ -1085,13 +1087,39 @@ int shrink_all_memory(int nr_pages) }#endif +#ifdef CONFIG_HOTPLUG_CPU +/* It's optimal to keep kswapds on the same CPUs as their memory, but +not required for correctness. So if the last cpu in a node goes +away, we get changed to run anywhere: as the first one comes back, +restore their cpu bindings. */ +static int __devinit cpu_callback (struct notifier_block *nfb, +unsigned long action, +void *hcpu) +{ +pg_data_t *pgdat; +cpumask_t mask; + +if (action ==CPU_ONLINE ) { +for_each_pgdat (pgdat) { + mask =node_to_cpumask (pgdat->node_id); +if (any_online_cpu (mask) !=NR_CPUS ) +/* One of our CPUs online: restore mask */ +set_cpus_allowed (pgdat->kswapd, mask); + } + } +return NOTIFY_OK ; +} +#endif /* CONFIG_HOTPLUG_CPU */ +static int __init kswapd_init (void ) {pg_data_t *pgdat;swap_setup ();for_each_pgdat (pgdat) -kernel_thread (kswapd, pgdat,CLONE_KERNEL ); + pgdat->kswapd + =find_task_by_pid (kernel_thread (kswapd, pgdat,CLONE_KERNEL )); total_memory =nr_free_pagecache_pages (); +hotcpu_notifier (cpu_callback,0 );return 0 ; }
net/core/dev.c +48 -0
66 lines not on this page · raw message
net/core/flow.c +12 -0
@@ -326,6 +326,17 @@ static void __devinit flow_cache_cpu_pretasklet_init (tasklet, flow_cache_flush_tasklet,0 ); } +#ifdef CONFIG_HOTPLUG_CPU +static int flow_cache_cpu (struct notifier_block *nfb, +unsigned long action, +void *hcpu) +{ +if (action ==CPU_DEAD ) +__flow_cache_shrink ((unsigned long )hcpu,0 ); +return NOTIFY_OK ; +} +#endif /* CONFIG_HOTPLUG_CPU */ +static int __init flow_cache_init (void ) {int i;
@@ -350,6 +361,7 @@ static int __init flow_cache_init(void)for_each_cpu (i)flow_cache_cpu_prepare (i); +hotcpu_notifier (flow_cache_cpu,0 );return 0 ; }
Signature · 1 line
Anyone who quotes me in their signature is an idiot -- Rusty Russell