Skip to contentSkip to conversation

generic-ipi: fix deadlock in __smp_call_function_single

PATCH In review Heiko Carstens 7 participants 18 messages Sep 9, 2010 – Sep 26, 2010

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic-ipi: fix deadlock in __smp_call_function_single

From: Heiko Carstens <[email protected]>

Just got my 6 way machine to a state where cpu 0 is in an endless loop within __smp_call_function_single.
All other cpus are idle.

The call trace on cpu 0 looks like this:

__smp_call_function_single
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
clock_comparator_work
do_extint
ext_int_handler
----> timer irq
cpu_idle

__smp_call_function_single got called from nohz_balancer_kick (inlined) with the remote cpu being 1, wait being 0 and the per cpu variable remote_sched_softirq_cb (call_single_data) of the current cpu (0).

Then it loops forever when it tries to grab the lock of the call_single_data, since it is already locked and enqueued on cpu 0.

My theory how this could have happened: for some reason the scheduler decided to call __smp_call_function_single on it's own cpu, and sends an IPI to itself. The interrupt stays pending since IRQs are disabled. If then the hypervisor schedules the cpu away it might happen that upon rescheduling both the IPI and the timer IRQ are pending.
If then interrupts are enabled again it depends which one gets scheduled first.
If the timer interrupt gets delivered first we end up with the local deadlock as seen in the calltrace above.

Let's make __smp_call_function_single check if the target cpu is the current cpu and execute the function immediately just like smp_call_function_single does. That should prevent at least the scenario described here.

It might also be that the scheduler is not supposed to call __smp_call_function_single with the remote cpu being the current cpu, but that is a different issue.

---
 kernel/smp.c |   14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
kernel/smp.c +12 -2
index 75c970c..f1427d8 100644
@@ -376,8 +376,10 @@ EXPORT_SYMBOL_GPL(smp_call_function_any);
 void __smp_call_function_single(int cpu, struct call_single_data *data,
 				int wait)
 {
-	csd_lock(data);
+	unsigned int this_cpu;
+	unsigned long flags;
 
+	this_cpu = get_cpu();
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -387,7 +389,15 @@ void __smp_call_function_single(int cpu, struct call_single_data *data,
 	WARN_ON_ONCE(cpu_online(smp_processor_id()) && wait && irqs_disabled()
 		     && !oops_in_progress);
 
-	generic_exec_single(cpu, data, wait);
+	if (cpu == this_cpu) {
+		local_irq_save(flags);
+		data->func(data->info);
+		local_irq_restore(flags);
+	} else {
+		csd_lock(data);
+		generic_exec_single(cpu, data, wait);
+	}
+	put_cpu();
 }
 
 /**
Acked-by on v1 1/1
#1 lore raw
Quoted 42 lines · Heiko Carstens

From: Heiko Carstens <[email protected]>

Just got my 6 way machine to a state where cpu 0 is in an endless loop within __smp_call_function_single.
All other cpus are idle.

The call trace on cpu 0 looks like this:

__smp_call_function_single
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
clock_comparator_work
do_extint
ext_int_handler
----> timer irq
cpu_idle

__smp_call_function_single got called from nohz_balancer_kick (inlined) with the remote cpu being 1, wait being 0 and the per cpu variable remote_sched_softirq_cb (call_single_data) of the current cpu (0).

Then it loops forever when it tries to grab the lock of the call_single_data, since it is already locked and enqueued on cpu 0.

My theory how this could have happened: for some reason the scheduler decided to call __smp_call_function_single on it's own cpu, and sends an IPI to itself. The interrupt stays pending since IRQs are disabled. If then the hypervisor schedules the cpu away it might happen that upon rescheduling both the IPI and the timer IRQ are pending.
If then interrupts are enabled again it depends which one gets scheduled first.
If the timer interrupt gets delivered first we end up with the local deadlock as seen in the calltrace above.

Let's make __smp_call_function_single check if the target cpu is the current cpu and execute the function immediately just like smp_call_function_single does. That should prevent at least the scenario described here.

Quoted 5 lines

It might also be that the scheduler is not supposed to call __smp_call_function_single with the remote cpu being the current cpu, but that is a different issue.

Right, so it looks like all other users of __smp_call_function_single() do indeed ensure not to call it on self, but your patch does make sense.

Quoted 37 lines
---
 kernel/smp.c |   14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
kernel/smp.c +12 -2
index 75c970c..f1427d8 100644
@@ -376,8 +376,10 @@ EXPORT_SYMBOL_GPL(smp_call_function_any);
 void __smp_call_function_single(int cpu, struct call_single_data *data,
 				int wait)
 {
-	csd_lock(data);
+	unsigned int this_cpu;
+	unsigned long flags;
 
+	this_cpu = get_cpu();
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -387,7 +389,15 @@ void __smp_call_function_single(int cpu, struct call_single_data *data,
 	WARN_ON_ONCE(cpu_online(smp_processor_id()) && wait && irqs_disabled()
 		     && !oops_in_progress);
 
-	generic_exec_single(cpu, data, wait);
+	if (cpu == this_cpu) {
+		local_irq_save(flags);
+		data->func(data->info);
+		local_irq_restore(flags);
+	} else {
+		csd_lock(data);
+		generic_exec_single(cpu, data, wait);
+	}
+	put_cpu();
 }
 
 /**
Acked-by
#2 lore raw
Quoted 43 lines · Peter Zijlstra
Quoted 42 lines · Heiko Carstens

From: Heiko Carstens <[email protected]>

Just got my 6 way machine to a state where cpu 0 is in an endless loop within __smp_call_function_single.
All other cpus are idle.

The call trace on cpu 0 looks like this:

__smp_call_function_single
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
clock_comparator_work
do_extint
ext_int_handler
----> timer irq
cpu_idle

__smp_call_function_single got called from nohz_balancer_kick (inlined) with the remote cpu being 1, wait being 0 and the per cpu variable remote_sched_softirq_cb (call_single_data) of the current cpu (0).

Then it loops forever when it tries to grab the lock of the call_single_data, since it is already locked and enqueued on cpu 0.

My theory how this could have happened: for some reason the scheduler decided to call __smp_call_function_single on it's own cpu, and sends an IPI to itself. The interrupt stays pending since IRQs are disabled. If then the hypervisor schedules the cpu away it might happen that upon rescheduling both the IPI and the timer IRQ are pending.
If then interrupts are enabled again it depends which one gets scheduled first.
If the timer interrupt gets delivered first we end up with the local deadlock as seen in the calltrace above.

Let's make __smp_call_function_single check if the target cpu is the current cpu and execute the function immediately just like smp_call_function_single does. That should prevent at least the scenario described here.

Quoted 8 lines
Quoted 5 lines

It might also be that the scheduler is not supposed to call __smp_call_function_single with the remote cpu being the current cpu, but that is a different issue.

Right, so it looks like all other users of __smp_call_function_single() do indeed ensure not to call it on self, but your patch does make sense.

I guess it depends on whether how bullet proof you want that (core) API to be. We've traditionally had this kind of support in similar functions so the caller doesn't have to check, so I guess the patch is fine with me too.

For extra credit, the function documentation should be modified as well:

* __smp_call_function_single(): Run a function on another CPU

Signature · 1 line
Jens Axboe
author Acked-by Peter Zijlstra Acked-by Jens Axboe
#3 lore raw
Quoted 4 lines · Jens Axboe
Quoted 2 lines · Peter Zijlstra

Right, so it looks like all other users of __smp_call_function_single() do indeed ensure not to call it on self, but your patch does make sense.

Quoted 8 lines

I guess it depends on whether how bullet proof you want that (core) API to be. We've traditionally had this kind of support in similar functions so the caller doesn't have to check, so I guess the patch is fine with me too.

For extra credit, the function documentation should be modified as well:

* __smp_call_function_single(): Run a function on another CPU

Upated patch below. I also added documentation for the wait flag.

Subject: [PATCH] generic-ipi: fix deadlock in __smp_call_function_single

From: Heiko Carstens <[email protected]>

Just got my 6 way machine to a state where cpu 0 is in an endless loop within __smp_call_function_single.
All other cpus are idle.

The call trace on cpu 0 looks like this:

__smp_call_function_single
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
clock_comparator_work
do_extint
ext_int_handler
----> timer irq
cpu_idle

__smp_call_function_single got called from nohz_balancer_kick (inlined) with the remote cpu being 1, wait being 0 and the per cpu variable remote_sched_softirq_cb (call_single_data) of the current cpu (0).

Then it loops forever when it tries to grab the lock of the call_single_data, since it is already locked and enqueued on cpu 0.

My theory how this could have happened: for some reason the scheduler decided to call __smp_call_function_single on it's own cpu, and sends an IPI to itself. The interrupt stays pending since IRQs are disabled. If then the hypervisor schedules the cpu away it might happen that upon rescheduling both the IPI and the timer IRQ are pending.
If then interrupts are enabled again it depends which one gets scheduled first.
If the timer interrupt gets delivered first we end up with the local deadlock as seen in the calltrace above.

Let's make __smp_call_function_single check if the target cpu is the current cpu and execute the function immediately just like smp_call_function_single does. That should prevent at least the scenario described here.

It might also be that the scheduler is not supposed to call __smp_call_function_single with the remote cpu being the current cpu, but that is a different issue.

---
 kernel/smp.c |   17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
kernel/smp.c +14 -3
index 75c970c..ed6aacf 100644
@@ -365,9 +365,10 @@ call:
 EXPORT_SYMBOL_GPL(smp_call_function_any);
 
 /**
- * __smp_call_function_single(): Run a function on another CPU
+ * __smp_call_function_single(): Run a function on a specific CPU
  * @cpu: The CPU to run on.
  * @data: Pre-allocated and setup data structure
+ * @wait: If true, wait until function has completed on specified CPU.
  *
  * Like smp_call_function_single(), but allow caller to pass in a
  * pre-allocated data structure. Useful for embedding @data inside
@@ -376,8 +377,10 @@ EXPORT_SYMBOL_GPL(smp_call_function_any);
 void __smp_call_function_single(int cpu, struct call_single_data *data,
 				int wait)
 {
-	csd_lock(data);
+	unsigned int this_cpu;
+	unsigned long flags;
 
+	this_cpu = get_cpu();
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -387,7 +390,15 @@ void __smp_call_function_single(int cpu, struct call_single_data *data,
 	WARN_ON_ONCE(cpu_online(smp_processor_id()) && wait && irqs_disabled()
 		     && !oops_in_progress);
 
-	generic_exec_single(cpu, data, wait);
+	if (cpu == this_cpu) {
+		local_irq_save(flags);
+		data->func(data->info);
+		local_irq_restore(flags);
+	} else {
+		csd_lock(data);
+		generic_exec_single(cpu, data, wait);
+	}
+	put_cpu();
 }
 
 /**
#4 lore raw

[tip:core/urgent] generic-ipi: Fix deadlock in __smp_call_function_single

Commit-ID:  27c379f7f89a4d558c685b5d89b5ba2fe79ae701
Gitweb:     http://git.kernel.org/tip/27c379f7f89a4d558c685b5d89b5ba2fe79ae701
Author:     Heiko Carstens <[email protected]>
AuthorDate: Fri, 10 Sep 2010 13:47:29 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 10 Sep 2010 16:48:40 +0200

generic-ipi: Fix deadlock in __smp_call_function_single

Just got my 6 way machine to a state where cpu 0 is in an endless loop within __smp_call_function_single. All other cpus are idle.

The call trace on cpu 0 looks like this:

__smp_call_function_single
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
clock_comparator_work
do_extint
ext_int_handler
----> timer irq
cpu_idle

__smp_call_function_single() got called from nohz_balancer_kick() (inlined) with the remote cpu being 1, wait being 0 and the per cpu variable remote_sched_softirq_cb (call_single_data) of the current cpu (0).

Then it loops forever when it tries to grab the lock of the call_single_data, since it is already locked and enqueued on cpu 0.

My theory how this could have happened: for some reason the scheduler decided to call __smp_call_function_single() on it's own cpu, and sends an IPI to itself. The interrupt stays pending since IRQs are disabled. If then the hypervisor schedules the cpu away it might happen that upon rescheduling both the IPI and the timer IRQ are pending. If then interrupts are enabled again it depends which one gets scheduled first.
If the timer interrupt gets delivered first we end up with the local deadlock as seen in the calltrace above.

Let's make __smp_call_function_single() check if the target cpu is the current cpu and execute the function immediately just like smp_call_function_single does. That should prevent at least the scenario described here.

It might also be that the scheduler is not supposed to call __smp_call_function_single with the remote cpu being the current cpu, but that is a different issue.

LKML-Reference: <[email protected]>

---
 kernel/smp.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
kernel/smp.c +14 -3
index 75c970c..ed6aacf 100644
@@ -365,9 +365,10 @@ call:
 EXPORT_SYMBOL_GPL(smp_call_function_any);
 
 /**
- * __smp_call_function_single(): Run a function on another CPU
+ * __smp_call_function_single(): Run a function on a specific CPU
  * @cpu: The CPU to run on.
  * @data: Pre-allocated and setup data structure
+ * @wait: If true, wait until function has completed on specified CPU.
  *
  * Like smp_call_function_single(), but allow caller to pass in a
  * pre-allocated data structure. Useful for embedding @data inside
@@ -376,8 +377,10 @@ EXPORT_SYMBOL_GPL(smp_call_function_any);
 void __smp_call_function_single(int cpu, struct call_single_data *data,
 				int wait)
 {
-	csd_lock(data);
+	unsigned int this_cpu;
+	unsigned long flags;
 
+	this_cpu = get_cpu();
 	/*
 	 * Can deadlock when called with interrupts disabled.
 	 * We allow cpu's that are not yet online though, as no one else can
@@ -387,7 +390,15 @@ void __smp_call_function_single(int cpu, struct call_single_data *data,
 	WARN_ON_ONCE(cpu_online(smp_processor_id()) && wait && irqs_disabled()
 		     && !oops_in_progress);
 
-	generic_exec_single(cpu, data, wait);
+	if (cpu == this_cpu) {
+		local_irq_save(flags);
+		data->func(data->info);
+		local_irq_restore(flags);
+	} else {
+		csd_lock(data);
+		generic_exec_single(cpu, data, wait);
+	}
+	put_cpu();
 }
 
 /**
#5 lore raw
In reply to Peter Zijlstra
Quoted 43 lines · On Fri, 10 Sep 2010 13:06:57 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 42 lines · Heiko Carstens

From: Heiko Carstens <[email protected]>

Just got my 6 way machine to a state where cpu 0 is in an endless loop within __smp_call_function_single.
All other cpus are idle.

The call trace on cpu 0 looks like this:

__smp_call_function_single
scheduler_tick
update_process_times
tick_sched_timer
__run_hrtimer
hrtimer_interrupt
clock_comparator_work
do_extint
ext_int_handler
----> timer irq
cpu_idle

__smp_call_function_single got called from nohz_balancer_kick (inlined) with the remote cpu being 1, wait being 0 and the per cpu variable remote_sched_softirq_cb (call_single_data) of the current cpu (0).

Then it loops forever when it tries to grab the lock of the call_single_data, since it is already locked and enqueued on cpu 0.

My theory how this could have happened: for some reason the scheduler decided to call __smp_call_function_single on it's own cpu, and sends an IPI to itself. The interrupt stays pending since IRQs are disabled. If then the hypervisor schedules the cpu away it might happen that upon rescheduling both the IPI and the timer IRQ are pending.
If then interrupts are enabled again it depends which one gets scheduled first.
If the timer interrupt gets delivered first we end up with the local deadlock as seen in the calltrace above.

Let's make __smp_call_function_single check if the target cpu is the current cpu and execute the function immediately just like smp_call_function_single does. That should prevent at least the scenario described here.

Quoted 8 lines
Quoted 5 lines

It might also be that the scheduler is not supposed to call __smp_call_function_single with the remote cpu being the current cpu, but that is a different issue.

Right, so it looks like all other users of __smp_call_function_single() do indeed ensure not to call it on self

Yes, it's a cross-CPU call only. If the scheduler called it for the current CPU then that's a scheduler bug.

Where is this scheduler bug? Did it occur because someone didn't understand __smp_call_function_single()? Or did it occur because the scheduler code is doing something which its implementors did not expect or intend?

Quoted 1 line

but your patch does make sense.

Maybe. Or maybe it papers over a scheduler bug by gratuitously adding additional code which no present callsites actually need.

The patch didn't update the __smp_call_function_single() kerneldoc. Compare it with smp_call_function_single() and note the subtle difference between "a specific CPU" and the now incorrect "on another CPU".

#6 lore raw
Quoted 4 lines · Andrew Morton

Where is this scheduler bug? Did it occur because someone didn't understand __smp_call_function_single()? Or did it occur because the scheduler code is doing something which its implementors did not expect or intend?

It comes from 83cd4fe2 (sched: Change nohz idle load balancing logic to push model), where nohz_balance_kick() simply needs to kick the designated driver into action.

I take it Venki assumed __smp_call_function_single() works like smp_call_function_single() where you can use it for the local cpu as well.

I guess we could do something like the below as well, which would be slightly faster since we don't actually need to call raise_softirq() since we already set it for self a bit earlier in order to have it do the regular load-balance actions.

---
 kernel/sched_fair.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/sched_fair.c +6 -4
index 9b5b4f8..c8ca1cb 100644
@@ -3404,11 +3404,13 @@ static void nohz_balancer_kick(int cpu)
 	}
 
 	if (!cpu_rq(ilb_cpu)->nohz_balance_kick) {
-		struct call_single_data *cp;
-
 		cpu_rq(ilb_cpu)->nohz_balance_kick = 1;
-		cp = &per_cpu(remote_sched_softirq_cb, cpu);
-		__smp_call_function_single(ilb_cpu, cp, 0);
+
+		if (ilb_cpu != cpu) {
+			struct call_single_data *cp;
+			cp = &per_cpu(remote_sched_softirq_cb, cpu);
+			__smp_call_function_single(ilb_cpu, cp, 0);
+		}
 	}
 	return;
 }
#7 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Andrew Morton

Where is this scheduler bug?  Did it occur because someone didn't understand __smp_call_function_single()?  Or did it occur because the scheduler code is doing something which its implementors did not expect or intend?

Quoted 7 lines

It comes from 83cd4fe2 (sched: Change nohz idle load balancing logic to push model), where nohz_balance_kick() simply needs to kick the designated driver into action.

I take it Venki assumed __smp_call_function_single() works like smp_call_function_single() where you can use it for the local cpu as well.

Yes. This was an oversight while moving from using send_remote_softirq to using __smp_call_function_single.
Also, as we don't have rq lock around this point, it seems possible that the CPU that was busy and wants to kick idle load balance on remote CPU, could have become idle and nominated itself as idle load balancer.

Below patch looks good to me.

I guess, we also need a WARN_ON_ONCE for (cpu == smp_processor_id()) in __smp_call_function_single(), as the eventual result of this bug that Heiko saw was a deadlock

Thanks,
Venki

Quoted 35 lines

I guess we could do something like the below as well, which would be slightly faster since we don't actually need to call raise_softirq() since we already set it for self a bit earlier in order to have it do the regular load-balance actions.

---
 kernel/sched_fair.c |   10 ++++++----

 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/sched_fair.c +6 -4
index 9b5b4f8..c8ca1cb 100644
@@ -3404,11 +3404,13 @@ static void nohz_balancer_kick(int cpu)
        }
 
        if (!cpu_rq(ilb_cpu)->nohz_balance_kick) {
-               struct call_single_data *cp;
-
                cpu_rq(ilb_cpu)->nohz_balance_kick = 1;
-               cp = &per_cpu(remote_sched_softirq_cb, cpu);
-               __smp_call_function_single(ilb_cpu, cp, 0);
+
+               if (ilb_cpu != cpu) {
+                       struct call_single_data *cp;
+                       cp = &per_cpu(remote_sched_softirq_cb, cpu);
+                       __smp_call_function_single(ilb_cpu, cp, 0);
+               }
        }
        return;
  }
#8 lore raw
Quoted 23 lines · Venkatesh Pallipadi
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Andrew Morton

Where is this scheduler bug?  Did it occur because someone didn't understand __smp_call_function_single()?  Or did it occur because the scheduler code is doing something which its implementors did not expect or intend?

Quoted 7 lines

It comes from 83cd4fe2 (sched: Change nohz idle load balancing logic to push model), where nohz_balance_kick() simply needs to kick the designated driver into action.

I take it Venki assumed __smp_call_function_single() works like smp_call_function_single() where you can use it for the local cpu as well.

Yes. This was an oversight while moving from using send_remote_softirq to using __smp_call_function_single.
Also, as we don't have rq lock around this point, it seems possible that the CPU that was busy and wants to kick idle load balance on remote CPU, could have become idle and nominated itself as idle load balancer.

Quoted 7 lines

Below patch looks good to me.

I guess, we also need a WARN_ON_ONCE for (cpu == smp_processor_id()) in __smp_call_function_single(), as the eventual result of this bug that Heiko saw was a deadlock

Either that or my generic IPI patch should be applied. At least to me it was rather surprising to see that smp_call_function_single() and __smp_call_function_single() behave differently when the 'remote' cpu is the current cpu.

#9 lore raw
In reply to Venkatesh Pallipadi
Quoted 4 lines · Venkatesh Pallipadi

Also, as we don't have rq lock around this point, it seems possible that the CPU that was busy and wants to kick idle load balance on remote CPU, could have become idle and nominated itself as idle load balancer.

A busy cpu (currently running something -- one task on the rq atleast) can't become idle in the middle of trigger_load_balance().

What might be happening is similar what you said but the opposite of it.

cpu-x is idle which is also ilb_cpu
got a scheduler tick during idle
and the nohz_kick_needed() in trigger_load_balance() checks for rq_x->nr_running which might not be zero (because of someone waking a task on this rq etc) and this leads to the situation of the cpu-x sending a kick to itself.

Perhaps the more appropriate patch would be(?):

---

kernel/sched_fair.c +1 -1
index 134f7ed..5b5aa97 100644
@@ -3632,7 +3632,7 @@ static inline int nohz_kick_needed(struct rq *rq, int cpu)
 	if (time_before(now, nohz.next_balance))
 		return 0;
 
-	if (!rq->nr_running)
+	if (rq->idle_at_tick)
 		return 0;
 
 	first_pick_cpu = atomic_read(&nohz.first_pick_cpu);
#10 lore raw
Quoted 9 lines · Suresh Siddha
Quoted 4 lines · Venkatesh Pallipadi

Also, as we don't have rq lock around this point, it seems possible that the CPU that was busy and wants to kick idle load balance on remote CPU, could have become idle and nominated itself as idle load balancer.

A busy cpu (currently running something -- one task on the rq atleast) can't become idle in the middle of trigger_load_balance().

Quoted 8 lines

What might be happening is similar what you said but the opposite of it.

cpu-x is idle which is also ilb_cpu
got a scheduler tick during idle
and the nohz_kick_needed() in trigger_load_balance() checks for rq_x->nr_running which might not be zero (because of someone waking a task on this rq etc) and this leads to the situation of the cpu-x sending a kick to itself.

So what patches are we going to merge?

I share Heiko's opinion on that its somewhat surprising to have __smp_call_function_single() differ in this detail from smp_call_function_single() and think that merging his patch would be good in that respect. But Andrew seemed to have reservations.

We can also merge either my or Suresh's patch (which I think makes sense, but is kinda subtle) to avoid the needless self kick.

Hmm?

#11 lore raw
Quoted 21 lines · Peter Zijlstra
Quoted 9 lines · Suresh Siddha
Quoted 4 lines · Venkatesh Pallipadi

Also, as we don't have rq lock around this point, it seems possible that the CPU that was busy and wants to kick idle load balance on remote CPU, could have become idle and nominated itself as idle load balancer.

A busy cpu (currently running something -- one task on the rq atleast) can't become idle in the middle of trigger_load_balance().

Quoted 8 lines

What might be happening is similar what you said but the opposite of it.

cpu-x is idle which is also ilb_cpu
got a scheduler tick during idle
and the nohz_kick_needed() in trigger_load_balance() checks for rq_x->nr_running which might not be zero (because of someone waking a task on this rq etc) and this leads to the situation of the cpu-x sending a kick to itself.

So what patches are we going to merge?

Quoted 7 lines

I share Heiko's opinion on that its somewhat surprising to have __smp_call_function_single() differ in this detail from smp_call_function_single() and think that merging his patch would be good in that respect. But Andrew seemed to have reservations.

We can also merge either my or Suresh's patch (which I think makes sense, but is kinda subtle) to avoid the needless self kick.

I would prefer to see your's or Suresh's scheduler patch to be merged to fix the bug.
My patch could be merged for 2.6.37 or be dropped in favour of a WARN_ON in __smp_call_function_single() if remote cpu == current cpu. However I think it would be better if smp_call_function_single() and __smp_call_function_single() wouldn't differ here.

#12 lore raw
Quoted 10 lines · Heiko Carstens
Quoted 10 lines

So what patches are we going to merge?

I share Heiko's opinion on that its somewhat surprising to have __smp_call_function_single() differ in this detail from smp_call_function_single() and think that merging his patch would be good in that respect. But Andrew seemed to have reservations.

We can also merge either my or Suresh's patch (which I think makes sense, but is kinda subtle) to avoid the needless self kick.

Peter, Can you please merge my patch instead of yours, as mine is more appropriate here. And also I would like Heiko's patch also to be merged as that brings smp_call_function_single() and __smp_call_function_single() to similar behavior.

thanks,
suresh

Quoted 7 lines

I would prefer to see your's or Suresh's scheduler patch to be merged to fix the bug.
My patch could be merged for 2.6.37 or be dropped in favour of a WARN_ON in __smp_call_function_single() if remote cpu == current cpu. However I think it would be better if smp_call_function_single() and __smp_call_function_single() wouldn't differ here.

#13 lore raw
Quoted 9 lines · Suresh Siddha
Quoted 8 lines · Heiko Carstens
Quoted 8 lines

So what patches are we going to merge?

I share Heiko's opinion on that its somewhat surprising to have __smp_call_function_single() differ in this detail from smp_call_function_single() and think that merging his patch would be good in that respect. But Andrew seemed to have reservations.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

We can also merge either my or Suresh's patch (which I think makes sense, but is kinda subtle) to avoid the needless self kick.

Peter, Can you please merge my patch instead of yours, as mine is more appropriate here. And also I would like Heiko's patch also to be merged as that brings smp_call_function_single() and __smp_call_function_single() to similar behavior.

Sure can do. Thanks!

#14 lore raw

[tip:sched/urgent] sched: Fix nohz balance kick

In reply to Suresh Siddha
Commit-ID:  f6c3f1686e7ec1dd8725a9a3dcb857dfd0c7a5bf
Gitweb:     http://git.kernel.org/tip/f6c3f1686e7ec1dd8725a9a3dcb857dfd0c7a5bf
Author:     Suresh Siddha <[email protected]>
AuthorDate: Mon, 13 Sep 2010 11:02:21 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 21 Sep 2010 13:50:50 +0200

sched: Fix nohz balance kick

There's a situation where the nohz balancer will try to wake itself:

cpu-x is idle which is also ilb_cpu
got a scheduler tick during idle
and the nohz_kick_needed() in trigger_load_balance() checks for rq_x->nr_running which might not be zero (because of someone waking a task on this rq etc) and this leads to the situation of the cpu-x sending a kick to itself.

And this can cause a lockup.

Avoid this by not marking ourself eligible for kicking.

LKML-Reference: <[email protected]>

---
 kernel/sched_fair.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched_fair.c +1 -1
index a171138..db3f674 100644
@@ -3630,7 +3630,7 @@ static inline int nohz_kick_needed(struct rq *rq, int cpu)
 	if (time_before(now, nohz.next_balance))
 		return 0;
 
-	if (!rq->nr_running)
+	if (rq->idle_at_tick)
 		return 0;
 
 	first_pick_cpu = atomic_read(&nohz.first_pick_cpu);
#15 lore raw
In reply to Andrew Morton
Quoted 2 lines · Andrew Morton

Maybe. Or maybe it papers over a scheduler bug by gratuitously adding additional code which no present callsites actually need.

Hm, indeed.

We now have the scheduler bug fixed upstream. Do we really need this patch?

Quoted 4 lines

The patch didn't update the __smp_call_function_single() kerneldoc. Compare it with smp_call_function_single() and note the subtle difference between "a specific CPU" and the now incorrect "on another CPU".

In any case this feedback didnt get addressed AFAICS.

Ingo

#16 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 2 lines · Andrew Morton

Maybe. Or maybe it papers over a scheduler bug by gratuitously adding additional code which no present callsites actually need.

Hm, indeed.

We now have the scheduler bug fixed upstream. Do we really need this patch?

General consensus was that it is good if smp_call_function_single() and __smp_call_function_single() would behave the same if remote cpu == current cpu.

If you're not applying this patch then at least at a WARN_ON() which triggers when remote cpu == current cpu. I don't want to debug something like this again.

Quoted 6 lines
Quoted 4 lines

The patch didn't update the __smp_call_function_single() kerneldoc. Compare it with smp_call_function_single() and note the subtle difference between "a specific CPU" and the now incorrect "on another CPU".

In any case this feedback didnt get addressed AFAICS.

It did get addressed in an updated patch which is the one which you
applied: http://lkml.org/lkml/2010/9/10/245

#17 lore raw
Quoted 12 lines · Heiko Carstens
Quoted 10 lines · Ingo Molnar
Quoted 2 lines · Andrew Morton

Maybe. Or maybe it papers over a scheduler bug by gratuitously adding additional code which no present callsites actually need.

Hm, indeed.

We now have the scheduler bug fixed upstream. Do we really need this patch?

Quoted 7 lines

General consensus was that it is good if smp_call_function_single() and __smp_call_function_single() would behave the same if remote cpu == current cpu.

If you're not applying this patch then at least at a WARN_ON() which triggers when remote cpu == current cpu. I don't want to debug something like this again.

Would be nice to hear from Andrew whether he's still opposed to this patch. I've got the patch queued up, but dont want to send it to Linus against Andrew's objections.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

The patch didn't update the __smp_call_function_single() kerneldoc. Compare it with smp_call_function_single() and note the subtle difference between "a specific CPU" and the now incorrect "on another CPU".

In any case this feedback didnt get addressed AFAICS.

It did get addressed in an updated patch which is the one which you
applied: http://lkml.org/lkml/2010/9/10/245

Indeed :-)

Thanks,

Ingo