Skip to contentSkip to conversation

rcu_barrier VS cpu_hotplug: make rcu_barrier can be called on CPU_DEAD

PATCH In review Lai Jiangshan 2 participants 3 messages Mar 7, 2009 – Mar 8, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rcu_barrier VS cpu_hotplug: make rcu_barrier can be called on CPU_DEAD

cpu hotplug notifier_call for CPU_DEAD are very complex, some notifier_call may call rcu_barrier(), if this notifier_call is called earlier than rcu_cpu_notify(), it is buggy.

_cpu_down() -->

	some_cpu_hotplug_notify() -->
		case CPU_DEAD: rcu_barrier()
	rcu_cpu_notify() -->
		rcu_offline_cpu()

When rcu_barrier() is called, rcu_barrier() does not know how to finish his work: rcu callbacks in dead cpu have not been migrated to online cpu, so rcu_barrier() cannot wait for these rcu callbacks, but rcu_barrier() should wait for all queued rcu callbacks.

This patch ensure that rcu_cpu_notify() is called earlier than any other notifier_call.

---

include/linux/notifier.h +6 -0
index b86fa2f..f7ae5a3 100644
@@ -238,6 +238,12 @@ static inline int notifier_to_errno(int ret)
 #define CPU_DYING_FROZEN	(CPU_DYING | CPU_TASKS_FROZEN)
 #define CPU_STARTING_FROZEN	(CPU_STARTING | CPU_TASKS_FROZEN)
 
+/* Register at highest priority so that task migration (migrate_all_tasks)
+ * and rcu callbacks migration happens before everything else.
+ */
+#define CPUHOTPLUG_TASK_MIGRATE_PRIORITY	10
+#define CPUHOTPLUG_RCU_MIGRATE_PRIORITY		11
+
 /* Hibernation and suspend events */
 #define PM_HIBERNATION_PREPARE	0x0001 /* Going to hibernate */
 #define PM_POST_HIBERNATION	0x0002 /* Hibernation finished */
kernel/rcuclassic.c +1 -0
index 654c640..040b894 100644
@@ -764,6 +764,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
 
 static struct notifier_block __cpuinitdata rcu_nb = {
 	.notifier_call	= rcu_cpu_notify,
+	.priority	= CPUHOTPLUG_RCU_MIGRATE_PRIORITY,
 };
 
 /*
kernel/rcupreempt.c +1 -0
index 5d59e85..8ffcbf0 100644
@@ -1407,6 +1407,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
 
 static struct notifier_block __cpuinitdata rcu_nb = {
 	.notifier_call = rcu_cpu_notify,
+	.priority = CPUHOTPLUG_RCU_MIGRATE_PRIORITY,
 };
 
 void __init __rcu_init(void)
kernel/rcutree.c +1 -0
index 97ce315..fcef06c 100644
@@ -1503,6 +1503,7 @@ do { \
 
 static struct notifier_block __cpuinitdata rcu_nb = {
 	.notifier_call	= rcu_cpu_notify,
+	.priority	= CPUHOTPLUG_RCU_MIGRATE_PRIORITY,
 };
 
 void __init __rcu_init(void)
kernel/sched.c +1 -4
index 8e2558c..56f225e 100644
@@ -6740,12 +6740,9 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)
 	return NOTIFY_OK;
 }
 
-/* Register at highest priority so that task migration (migrate_all_tasks)
- * happens before everything else.
- */
 static struct notifier_block __cpuinitdata migration_notifier = {
 	.notifier_call = migration_call,
-	.priority = 10
+	.priority = CPUHOTPLUG_TASK_MIGRATE_PRIORITY,
 };
 
 static int __init migration_init(void)
#1 lore raw
Quoted 11 lines · Lai Jiangshan

cpu hotplug notifier_call for CPU_DEAD are very complex, some notifier_call may call rcu_barrier(), if this notifier_call is called earlier than rcu_cpu_notify(), it is buggy.

_cpu_down() -->

	some_cpu_hotplug_notify() -->
		case CPU_DEAD: rcu_barrier()
	rcu_cpu_notify() -->
		rcu_offline_cpu()
Quoted 4 lines

When rcu_barrier() is called, rcu_barrier() does not know how to finish his work: rcu callbacks in dead cpu have not been migrated to online cpu, so rcu_barrier() cannot wait for these rcu callbacks, but rcu_barrier() should wait for all queued rcu callbacks.

Hmmm...

What should happen in this case is that rcu_offline_cpu() moves the rcu_barrier() callback to a surviving CPU, and then the rcu_barrier() should complete just fine at the end of a subsequent grace period. In fact, this is why it is so important to keep RCU callbacks ordered through CPU-hotplug operations.

Or do you have a test that demonstrates a failure in this case? If so, please share!

Thanx, Paul

Quoted 82 lines

This patch ensure that rcu_cpu_notify() is called earlier than any other notifier_call.

---

include/linux/notifier.h +6 -0
index b86fa2f..f7ae5a3 100644
@@ -238,6 +238,12 @@ static inline int notifier_to_errno(int ret)
 #define CPU_DYING_FROZEN	(CPU_DYING | CPU_TASKS_FROZEN)
 #define CPU_STARTING_FROZEN	(CPU_STARTING | CPU_TASKS_FROZEN)
 
+/* Register at highest priority so that task migration (migrate_all_tasks)
+ * and rcu callbacks migration happens before everything else.
+ */
+#define CPUHOTPLUG_TASK_MIGRATE_PRIORITY	10
+#define CPUHOTPLUG_RCU_MIGRATE_PRIORITY		11
+
 /* Hibernation and suspend events */
 #define PM_HIBERNATION_PREPARE	0x0001 /* Going to hibernate */
 #define PM_POST_HIBERNATION	0x0002 /* Hibernation finished */
kernel/rcuclassic.c +1 -0
index 654c640..040b894 100644
@@ -764,6 +764,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
 
 static struct notifier_block __cpuinitdata rcu_nb = {
 	.notifier_call	= rcu_cpu_notify,
+	.priority	= CPUHOTPLUG_RCU_MIGRATE_PRIORITY,
 };
 
 /*
kernel/rcupreempt.c +1 -0
index 5d59e85..8ffcbf0 100644
@@ -1407,6 +1407,7 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
 
 static struct notifier_block __cpuinitdata rcu_nb = {
 	.notifier_call = rcu_cpu_notify,
+	.priority = CPUHOTPLUG_RCU_MIGRATE_PRIORITY,
 };
 
 void __init __rcu_init(void)
kernel/rcutree.c +1 -0
index 97ce315..fcef06c 100644
@@ -1503,6 +1503,7 @@ do { \
 
 static struct notifier_block __cpuinitdata rcu_nb = {
 	.notifier_call	= rcu_cpu_notify,
+	.priority	= CPUHOTPLUG_RCU_MIGRATE_PRIORITY,
 };
 
 void __init __rcu_init(void)
kernel/sched.c +1 -5
index 8e2558c..56f225e 100644
@@ -6740,12 +6740,9 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)
 	return NOTIFY_OK;
 }
 
-/* Register at highest priority so that task migration (migrate_all_tasks)
- * happens before everything else.
- */
 static struct notifier_block __cpuinitdata migration_notifier = {
 	.notifier_call = migration_call,
-	.priority = 10
+	.priority = CPUHOTPLUG_TASK_MIGRATE_PRIORITY,
 };
 
 static int __init migration_init(void)
 
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
Quoted 16 lines · Paul E. McKenney
Quoted 10 lines · Lai Jiangshan

cpu hotplug notifier_call for CPU_DEAD are very complex, some notifier_call may call rcu_barrier(), if this notifier_call is called earlier than rcu_cpu_notify(), it is buggy.

_cpu_down() -->

	some_cpu_hotplug_notify() -->
		case CPU_DEAD: rcu_barrier()
	rcu_cpu_notify() -->
		rcu_offline_cpu()
Quoted 4 lines

When rcu_barrier() is called, rcu_barrier() does not know how to finish his work: rcu callbacks in dead cpu have not been migrated to online cpu, so rcu_barrier() cannot wait for these rcu callbacks, but rcu_barrier() should wait for all queued rcu callbacks.

Quoted 4 lines

Hmmm...

What should happen in this case is that rcu_offline_cpu() moves the rcu_barrier() callback to a surviving CPU, and then the rcu_barrier()

It's not about the rcu_barrier() callback, It's about rcu callbacks in dead cpu. rcu_barrier() should also wait for these rcu callbacks to complete. But these rcu callbacks are not migrated to online cpu, rcu_barrier() cannot wait for them.

Thanks, Lai.

Quoted 7 lines

should complete just fine at the end of a subsequent grace period. In fact, this is why it is so important to keep RCU callbacks ordered through CPU-hotplug operations.

Or do you have a test that demonstrates a failure in this case? If so, please share!