hrtimer: remove deadlock due to waiting on IPI in softirq context
RFC PATCH 1/1 hrtimer: remove deadlock due to waiting on IPI in softirq context
There appears to be a deadlock in the hrtimer code. Specifically, clock_was_set() calls an IPI with wait=1, from softirq context.
Waiting for IPIs to complete in irq context can lead to a deadlock, because the current code (that was interrupted) might be holding some kind of lock, that another CPU is waiting for with spin_lock_irq or similar.
In other words, the current CPU may need to release a resource, before the IPI can be handled by one of the destination CPUs.
To my untrained eye, it does not look like this patch introduces a new bug to the timer code, but that is hard to ascertain with the timer code. so I am posting this as an RFC for the timer gods to hurt their brains on :)
This bug was introduced by 54cdfdb4 in early 2007 (the original hrtimer code patch).
--- kernel/hrtimer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/hrtimer.c +1 -1
@@ -771,7 +771,7 @@ void clock_was_set(void) {#ifdef CONFIG_HIGH_RES_TIMERS /* Retrigger the CPU local events everywhere */ -on_each_cpu (retrigger_next_event,NULL ,1 ); +on_each_cpu (retrigger_next_event,NULL ,0 );#endif timerfd_clock_was_set (); }
Quoted 2 lines · Rik van Riel
There appears to be a deadlock in the hrtimer code. Specifically,
clock_was_set()calls an IPI with wait=1, from softirq context.
This should not be called from softirq context.
Quoted 8 lines
Waiting for IPIs to complete in irq context can lead to a deadlock, because the current code (that was interrupted) might be holding some kind of lock, that another CPU is waiting for with
spin_lock_irqor similar.In other words, the current CPU may need to release a resource, before the IPI can be handled by one of the destination CPUs.
Quoted 7 lines
To my untrained eye, it does not look like this patch introduces a new bug to the timer code, but that is hard to ascertain with the timer code. so I am posting this as an RFC for the timer gods to hurt their brains on :)
This bug was introduced by
54cdfdb4in early 2007 (the original hrtimer code patch).
Right and we had some issues with that until we moved the calls to clock_was_set() out of lock held regions.
The only call which happens from interrupt context is in update_wall_time(). And that one definitely holds no locks which are relevant.
On which kernel are you observing the issue?
Can you provide the debug info which made you look into this?
Thanks,
tglx
Quoted 26 lines
Not-yet-signed-off-by Rik van Riel <[email protected]>Reported-by Mateusz Guzik <[email protected]>Cc Benjamin Herrenschmidt <[email protected]>Cc Ingo Molnar <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Prarit Bhargava <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc Clark Williams <[email protected]>--- kernel/hrtimer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/hrtimer.c +1
-1@@ -771,7 +771,7 @@ void clock_was_set(void) {#ifdef CONFIG_HIGH_RES_TIMERS /* Retrigger the CPU local events everywhere */ -on_each_cpu (retrigger_next_event,NULL ,1 ); +on_each_cpu (retrigger_next_event,NULL ,0 );#endif timerfd_clock_was_set (); }
Quoted 19 lines · Thomas Gleixner
Quoted 2 lines · Rik van Riel
There appears to be a deadlock in the hrtimer code. Specifically,
clock_was_set()calls an IPI with wait=1, from softirq context.This should not be called from softirq context.
Quoted 13 lines
Waiting for IPIs to complete in irq context can lead to a deadlock, because the current code (that was interrupted) might be holding some kind of lock, that another CPU is waiting for with
spin_lock_irqor similar.In other words, the current CPU may need to release a resource, before the IPI can be handled by one of the destination CPUs.
To my untrained eye, it does not look like this patch introduces a new bug to the timer code, but that is hard to ascertain with the timer code. so I am posting this as an RFC for the timer gods to hurt their brains on :)
Quoted 5 lines
Quoted 2 lines
This bug was introduced by
54cdfdb4in early 2007 (the original hrtimer code patch).Right and we had some issues with that until we moved the calls to
clock_was_set()out of lock held regions.
Ahh indeed, the bug got fixed already :)
Quoted 5 lines
The only call which happens from interrupt context is in
update_wall_time(). And that one definitely holds no locks which are relevant.On which kernel are you observing the issue?
This was RHEL6, and I saw that the immediate function was still the same upstream.
I forgot to check that clock_was_set() is now called in a different way. My bad.