rcu_barrier VS cpu_hotplug: make rcu_barrier can be called on CPU_DEAD
PATCH 1/1 rcu_barrier VS cpu_hotplug: make rcu_barrier can be called on CPU_DEAD
cpu hotplug notifier_call for CPU_DEAD are very complex, some notifier_call may call rcu_barrier(), if this notifier_call is called earlier than rcu_cpu_notify(), it is buggy.
_cpu_down() -->
some_cpu_hotplug_notify() --> case CPU_DEAD: rcu_barrier() rcu_cpu_notify() --> rcu_offline_cpu()
When rcu_barrier() is called, rcu_barrier() does not know how to finish his work: rcu callbacks in dead cpu have not been migrated to online cpu, so rcu_barrier() cannot wait for these rcu callbacks, but rcu_barrier() should wait for all queued rcu callbacks.
This patch ensure that rcu_cpu_notify() is called earlier than any other notifier_call.
---
include/linux/notifier.h +6 -0
@@ -238,6 +238,12 @@ static inline int notifier_to_errno(int ret)#define CPU_DYING_FROZEN (CPU_DYING | CPU_TASKS_FROZEN)#define CPU_STARTING_FROZEN (CPU_STARTING | CPU_TASKS_FROZEN) +/* Register at highest priority so that task migration (migrate_all_tasks) +* and rcu callbacks migration happens before everything else. +*/ +#define CPUHOTPLUG_TASK_MIGRATE_PRIORITY 10 +#define CPUHOTPLUG_RCU_MIGRATE_PRIORITY 11 +/* Hibernation and suspend events */ #define PM_HIBERNATION_PREPARE 0x0001/* Going to hibernate */ #define PM_POST_HIBERNATION 0x0002/* Hibernation finished */
kernel/rcuclassic.c +1 -0
@@ -764,6 +764,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,static struct notifier_block __cpuinitdata rcu_nb = { .notifier_call = rcu_cpu_notify, + .priority =CPUHOTPLUG_RCU_MIGRATE_PRIORITY , };/*
kernel/rcupreempt.c +1 -0
@@ -1407,6 +1407,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,static struct notifier_block __cpuinitdata rcu_nb = { .notifier_call = rcu_cpu_notify, + .priority =CPUHOTPLUG_RCU_MIGRATE_PRIORITY , };void __init __rcu_init (void )
kernel/rcutree.c +1 -0
@@ -1503,6 +1503,7 @@ do { \static struct notifier_block __cpuinitdata rcu_nb = { .notifier_call = rcu_cpu_notify, + .priority =CPUHOTPLUG_RCU_MIGRATE_PRIORITY , };void __init __rcu_init (void )
kernel/sched.c +1 -4
@@ -6740,12 +6740,9 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)return NOTIFY_OK ; } -/* Register at highest priority so that task migration (migrate_all_tasks) -* happens before everything else. -*/ static struct notifier_block __cpuinitdata migration_notifier = { .notifier_call = migration_call, - .priority =10 + .priority =CPUHOTPLUG_TASK_MIGRATE_PRIORITY , };static int __init migration_init (void )
Quoted 11 lines · Lai Jiangshan
cpu hotplug
notifier_callforCPU_DEADare very complex, somenotifier_callmay callrcu_barrier(), if thisnotifier_callis called earlier thanrcu_cpu_notify(), it is buggy.
_cpu_down()-->some_cpu_hotplug_notify() --> case CPU_DEAD: rcu_barrier() rcu_cpu_notify() --> rcu_offline_cpu()
Quoted 4 lines
When
rcu_barrier()is called,rcu_barrier()does not know how to finish his work: rcu callbacks in dead cpu have not been migrated to online cpu, sorcu_barrier()cannot wait for these rcu callbacks, butrcu_barrier()should wait for all queued rcu callbacks.
Hmmm...
What should happen in this case is that rcu_offline_cpu() moves the rcu_barrier() callback to a surviving CPU, and then the rcu_barrier() should complete just fine at the end of a subsequent grace period. In fact, this is why it is so important to keep RCU callbacks ordered through CPU-hotplug operations.
Or do you have a test that demonstrates a failure in this case? If so, please share!
Thanx, Paul
Quoted 82 lines
This patch ensure that
rcu_cpu_notify()is called earlier than any othernotifier_call.Signed-off-by Lai Jiangshan <[email protected]>---
include/linux/notifier.h +6
-0@@ -238,6 +238,12 @@ static inline int notifier_to_errno(int ret)#define CPU_DYING_FROZEN (CPU_DYING | CPU_TASKS_FROZEN)#define CPU_STARTING_FROZEN (CPU_STARTING | CPU_TASKS_FROZEN) +/* Register at highest priority so that task migration (migrate_all_tasks) +* and rcu callbacks migration happens before everything else. +*/ +#define CPUHOTPLUG_TASK_MIGRATE_PRIORITY 10 +#define CPUHOTPLUG_RCU_MIGRATE_PRIORITY 11 +/* Hibernation and suspend events */ #define PM_HIBERNATION_PREPARE 0x0001/* Going to hibernate */ #define PM_POST_HIBERNATION 0x0002/* Hibernation finished */ kernel/rcuclassic.c +1
-0@@ -764,6 +764,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,static struct notifier_block __cpuinitdata rcu_nb = { .notifier_call = rcu_cpu_notify, + .priority =CPUHOTPLUG_RCU_MIGRATE_PRIORITY , };/* kernel/rcupreempt.c +1
-0@@ -1407,6 +1407,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,static struct notifier_block __cpuinitdata rcu_nb = { .notifier_call = rcu_cpu_notify, + .priority =CPUHOTPLUG_RCU_MIGRATE_PRIORITY , };void __init __rcu_init (void )kernel/rcutree.c +1
-0@@ -1503,6 +1503,7 @@ do { \static struct notifier_block __cpuinitdata rcu_nb = { .notifier_call = rcu_cpu_notify, + .priority =CPUHOTPLUG_RCU_MIGRATE_PRIORITY , };void __init __rcu_init (void )kernel/sched.c +1
-5@@ -6740,12 +6740,9 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)return NOTIFY_OK ; } -/* Register at highest priority so that task migration (migrate_all_tasks) -* happens before everything else. -*/ static struct notifier_block __cpuinitdata migration_notifier = { .notifier_call = migration_call, - .priority =10 + .priority =CPUHOTPLUG_TASK_MIGRATE_PRIORITY , };static int __init migration_init (void ) --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 16 lines · Paul E. McKenney
Quoted 10 lines · Lai Jiangshan
cpu hotplug
notifier_callforCPU_DEADare very complex, somenotifier_callmay callrcu_barrier(), if thisnotifier_callis called earlier thanrcu_cpu_notify(), it is buggy.
_cpu_down()-->some_cpu_hotplug_notify() --> case CPU_DEAD: rcu_barrier() rcu_cpu_notify() --> rcu_offline_cpu()Quoted 4 lines
When
rcu_barrier()is called,rcu_barrier()does not know how to finish his work: rcu callbacks in dead cpu have not been migrated to online cpu, sorcu_barrier()cannot wait for these rcu callbacks, butrcu_barrier()should wait for all queued rcu callbacks.
Quoted 4 lines
Hmmm...
What should happen in this case is that
rcu_offline_cpu()moves thercu_barrier()callback to a surviving CPU, and then thercu_barrier()
It's not about the rcu_barrier() callback, It's about rcu callbacks in dead cpu. rcu_barrier() should also wait for these rcu callbacks to complete. But these rcu callbacks are not migrated to online cpu, rcu_barrier() cannot wait for them.
Thanks, Lai.
Quoted 7 lines
should complete just fine at the end of a subsequent grace period. In fact, this is why it is so important to keep RCU callbacks ordered through CPU-hotplug operations.
Or do you have a test that demonstrates a failure in this case? If so, please share!