Skip to contentSkip to conversation

generic-ipi: Allow cpus not yet online to call smp_call_function with irqs disabled

tip:x86/pat Nick Piggin 2 participants 4 messages Aug 24, 2009 – Aug 25, 2009 1 message not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 9 lines · Suresh B
Commit-ID:  269c861baa2fe7c114c3bc7831292758d29eb336
Gitweb:     http://git.kernel.org/tip/269c861baa2fe7c114c3bc7831292758d29eb336
Author:     Suresh Siddha <[email protected]>
AuthorDate: Wed, 19 Aug 2009 18:05:35 -0700
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Fri, 21 Aug 2009 16:25:43 -0700

generic-ipi: Allow cpus not yet online to call smp_call_function with irqs disabled

Quoted 8 lines

Because of deadlock possiblities smp_call_function() is not allowed to be called with interrupts disabled. Add an exception for the cpu not yet online, as no one else can send smp call function interrupt to this cpu that is not yet online and as such deadlock condition is not possible.

I don't know if we should allow the use of smp_call_function here -- only call_function_single. CPU hotplug code is required to set up some call_function data and if the cpu is offline then it might not be set up correctly.

Also, I would say that we should just restrict this to wait==1 case because in that case the stack can trivially be used for data. In the wait==0 case, it is more complex. In the current implementation it should be OK (it uses per-cpu data), but we've used kmalloc there in the past, which probably wouldn't work either.

Quoted 83 lines
---
 kernel/smp.c |   40 ++++++++++++++++++++++++++++++++++------
 1 files changed, 34 insertions(+), 6 deletions(-)
kernel/smp.c +34 -6
index ad63d85..2accdf6 100644
@@ -177,6 +177,11 @@ void generic_smp_call_function_interrupt(void)
 	int cpu = get_cpu();
 
 	/*
+	 * Shouldn't receive this interrupt on a cpu that is not yet online.
+	 */
+	WARN_ON_ONCE(!cpu_online(cpu));
+
+	/*
 	 * Ensure entry is visible on call_function_queue after we have
 	 * entered the IPI. See comment in smp_call_function_many.
 	 * If we don't have this, then we may miss an entry on the list
@@ -230,6 +235,11 @@ void generic_smp_call_function_single_interrupt(void)
 	unsigned int data_flags;
 	LIST_HEAD(list);
 
+	/*
+	 * Shouldn't receive this interrupt on a cpu that is not yet online.
+	 */
+	WARN_ON_ONCE(!cpu_online(smp_processor_id()));
+
 	spin_lock(&q->lock);
 	list_replace_init(&q->list, &list);
 	spin_unlock(&q->lock);
@@ -285,8 +295,14 @@ int smp_call_function_single(int cpu, void (*func) (void *info), void *info,
 	 */
 	this_cpu = get_cpu();
 
-	/* Can deadlock when called with interrupts disabled */
-	WARN_ON_ONCE(irqs_disabled() && !oops_in_progress);
+	/*
+	 * Can deadlock when called with interrupts disabled.
+	 * We allow cpu's that are not yet online though, as no one else can
+	 * send smp call function interrupt to this cpu and as such deadlocks
+	 * can't happen.
+	 */
+	WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled()
+		     && !oops_in_progress);
 
 	if (cpu == this_cpu) {
 		local_irq_save(flags);
@@ -329,8 +345,14 @@ void __smp_call_function_single(int cpu, struct call_single_data *data,
 {
 	csd_lock(data);
 
-	/* Can deadlock when called with interrupts disabled */
-	WARN_ON_ONCE(wait && irqs_disabled() && !oops_in_progress);
+	/*
+	 * Can deadlock when called with interrupts disabled.
+	 * We allow cpu's that are not yet online though, as no one else can
+	 * send smp call function interrupt to this cpu and as such deadlocks
+	 * can't happen.
+	 */
+	WARN_ON_ONCE(cpu_online(smp_processor_id()) && wait && irqs_disabled()
+		     && !oops_in_progress);
 
 	generic_exec_single(cpu, data, wait);
 }
@@ -365,8 +387,14 @@ void smp_call_function_many(const struct cpumask *mask,
 	unsigned long flags;
 	int cpu, next_cpu, this_cpu = smp_processor_id();
 
-	/* Can deadlock when called with interrupts disabled */
-	WARN_ON_ONCE(irqs_disabled() && !oops_in_progress);
+	/*
+	 * Can deadlock when called with interrupts disabled.
+	 * We allow cpu's that are not yet online though, as no one else can
+	 * send smp call function interrupt to this cpu and as such deadlocks
+	 * can't happen.
+	 */
+	WARN_ON_ONCE(cpu_online(this_cpu) && irqs_disabled()
+		     && !oops_in_progress);
 
 	/* So, what's a CPU they want? Ignoring this one. */
 	cpu = cpumask_first_and(mask, cpu_online_mask);
#2 lore raw
Quoted 15 lines · Nick Piggin
Quoted 14 lines · Suresh B
Commit-ID:  269c861baa2fe7c114c3bc7831292758d29eb336
Gitweb:     http://git.kernel.org/tip/269c861baa2fe7c114c3bc7831292758d29eb336
Author:     Suresh Siddha <[email protected]>
AuthorDate: Wed, 19 Aug 2009 18:05:35 -0700
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Fri, 21 Aug 2009 16:25:43 -0700

generic-ipi: Allow cpus not yet online to call smp_call_function with irqs disabled

Because of deadlock possiblities smp_call_function() is not allowed to be called with interrupts disabled. Add an exception for the cpu not yet online, as no one else can send smp call function interrupt to this cpu that is not yet online and as such deadlock condition is not possible.

Quoted 8 lines
Quoted 3 lines

I don't know if we should allow the use of smp_call_function here -- only call_function_single. CPU hotplug code is required to set up some call_function data and if the cpu is offline then it might not be set up correctly.

We are doing the required allocations in CPU_UP_PREPARE. So we should be okay for any smp_call_function usage.

Quoted 5 lines

Also, I would say that we should just restrict this to wait==1 case because in that case the stack can trivially be used for data. In the wait==0 case, it is more complex. In the current implementation it should be OK (it uses per-cpu data), but we've used kmalloc there in the past, which probably wouldn't work either.

In future if we add any kmalloc, we already have checks in kmalloc() that can be easily caught. I would like to make this change as generic as possible.

thanks,
suresh

#3 lore raw
Quoted 20 lines · Suresh B
Quoted 19 lines · Nick Piggin
Quoted 17 lines · Suresh B
Commit-ID:  269c861baa2fe7c114c3bc7831292758d29eb336
Gitweb:     http://git.kernel.org/tip/269c861baa2fe7c114c3bc7831292758d29eb336
Author:     Suresh Siddha <[email protected]>
AuthorDate: Wed, 19 Aug 2009 18:05:35 -0700
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Fri, 21 Aug 2009 16:25:43 -0700

generic-ipi: Allow cpus not yet online to call smp_call_function with irqs disabled

Because of deadlock possiblities smp_call_function() is not allowed to be called with interrupts disabled. Add an exception for the cpu not yet online, as no one else can send smp call function interrupt to this cpu that is not yet online and as such deadlock condition is not possible.

Quoted 7 lines
Quoted 4 lines

I don't know if we should allow the use of smp_call_function here -- only call_function_single. CPU hotplug code is required to set up some call_function data and if the cpu is offline then it might not be set up correctly.

We are doing the required allocations in CPU_UP_PREPARE. So we should be okay for any smp_call_function usage.

OK

Quoted 9 lines
Quoted 5 lines

Also, I would say that we should just restrict this to wait==1 case because in that case the stack can trivially be used for data. In the wait==0 case, it is more complex. In the current implementation it should be OK (it uses per-cpu data), but we've used kmalloc there in the past, which probably wouldn't work either.

In future if we add any kmalloc, we already have checks in kmalloc() that can be easily caught. I would like to make this change as generic as possible.

Why? You think there will be much demand for it?

#4 lore raw
Quoted 8 lines · Nick Piggin
Quoted 7 lines · Suresh B
Quoted 5 lines · Nick Piggin

Also, I would say that we should just restrict this to wait==1 case because in that case the stack can trivially be used for data. In the wait==0 case, it is more complex. In the current implementation it should be OK (it uses per-cpu data), but we've used kmalloc there in the past, which probably wouldn't work either.

Quoted 5 lines
Quoted 3 lines

In future if we add any kmalloc, we already have checks in kmalloc() that can be easily caught. I would like to make this change as generic as possible.

Why? You think there will be much demand for it?

In the current mtrr case, we currently use wait==0

Also unless there are issues, I would like to keep it open. So that it will encourage more usages and expose any other bugs that we perhaps overlooked.

With all the recent generic-ipi changes, even for online cpu's, we should be able to allow smp_call_function() with interrupts disabled for wait==0 case, right?

thanks,
suresh