"perf top" results in "NOHZ: local_softirq_pending 100"
While playing around with perf I realized that "perf top" immediatly results in a "NOHZ: local_softirq_pending 100" message to the console.
0x100 means that a HRTIMER_SOFTIRQ is pending when the cpu tries to disable the tick.
In perf_event.c we have a call to __hrtimer_start_range_ns() in perf_swevent_start_hrtimer() where its wakeup parameter is zero. __hrtimer_start_range_ns() in turn will call hrtimer_enqueue_reprogram() which will call __raise_softirq_irqoff(HRTIMER_SOFTIRQ) (since wakeup is zero).
That means that just the HRTIMER_SOFTIRQ bit gets set in the softirq pending field, but wakeup_softirqd() doesn't get called.
As far as I could see this function gets called from process context with a spinlock held and hence we don't have any guarantee that this pending softirq get executed before the idle task gets scheduled and tries to disable the tick.
The easiest fix would be to set wakeup to one (see patch below), but I guess there is a reason why its zero. Anybody?
--- kernel/perf_event.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/perf_event.c +1 -1
@@ -4942,7 +4942,7 @@ static void perf_swevent_start_hrtimer(struct perf_event *event) }__hrtimer_start_range_ns (&hwc->hrtimer,ns_to_ktime (period),0 , -HRTIMER_MODE_REL_PINNED ,0 ); +HRTIMER_MODE_REL_PINNED ,1 ); } }
Quoted 30 lines · Heiko Carstens
While playing around with perf I realized that "perf top" immediatly results in a "NOHZ:
local_softirq_pending100" message to the console.
0x100means that aHRTIMER_SOFTIRQis pending when the cpu tries to disable the tick.
Inperf_event.cwe have a call to__hrtimer_start_range_ns()inperf_swevent_start_hrtimer()where its wakeup parameter is zero.__hrtimer_start_range_ns()in turn will callhrtimer_enqueue_reprogram()which will call__raise_softirq_irqoff(HRTIMER_SOFTIRQ)(since wakeup is zero).
That means that just theHRTIMER_SOFTIRQbit gets set in the softirq pending field, butwakeup_softirqd()doesn't get called.As far as I could see this function gets called from process context with a spinlock held and hence we don't have any guarantee that this pending softirq get executed before the idle task gets scheduled and tries to disable the tick.
The easiest fix would be to set wakeup to one (see patch below), but I guess there is a reason why its zero. Anybody?
--- kernel/perf_event.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/perf_event.c +0
-0
kernel/perf_event.c static void perf_swevent_start_hrtimer(struct perf_event *event) · line 4945
Quoted 5 lines
kernel/perf_event.c +1
-1@@ -4942,7 +4942,7 @@ static void perf_swevent_start_hrtimer(struct perf_event *event) }__hrtimer_start_range_ns (&hwc->hrtimer,ns_to_ktime (period),0 , -HRTIMER_MODE_REL_PINNED ,0 ); +HRTIMER_MODE_REL_PINNED ,1 );
Ah. With this patch "perf record" might deadlock.
That's the reason why wakeup is zero. Tough luck.
Quoted 7 lines · Heiko Carstens
Quoted 7 lines
As far as I could see this function gets called from process context with a spinlock held and hence we don't have any guarantee that this pending softirq get executed before the idle task gets scheduled and tries to disable the tick.
The easiest fix would be to set wakeup to one (see patch below), but I guess there is a reason why its zero. Anybody?
We can start that hrtimer from within the scheduler function while holding the rq->lock, doing a wakeup from there is not sane.
The best solution would be to fix the hrtimer_start*() interface, something Thomas and I have wanted to do for ages but because we've procrastinated is now a much larger job than it was :/
The whole HRTIMER_SOFTIRQ thing should die.. but for that to happen its only use-case today must first go.
The problem is trying to start a timer with already elapsed time. Preferably hrtimer_start*() would simply return -ETIME and let the caller sort it, sadly the current behaviour is to 'fix' it for the caller by enqueueing the timer onto the softirq list and raising the softirq.
I guess we could make hrtimer_start*(.wakeup=false) return the -ENOTIME thing and audit those few use-cases.
Quoted 19 lines · Peter Zijlstra
Quoted 7 lines · Heiko Carstens
Quoted 7 lines
As far as I could see this function gets called from process context with a spinlock held and hence we don't have any guarantee that this pending softirq get executed before the idle task gets scheduled and tries to disable the tick.
The easiest fix would be to set wakeup to one (see patch below), but I guess there is a reason why its zero. Anybody?
We can start that hrtimer from within the scheduler function while holding the
rq->lock, doing a wakeup from there is not sane.The best solution would be to fix the hrtimer_start*() interface, something Thomas and I have wanted to do for ages but because we've procrastinated is now a much larger job than it was :/
The whole
HRTIMER_SOFTIRQthing should die.. but for that to happen its only use-case today must first go.
Quoted 8 lines
The problem is trying to start a timer with already elapsed time. Preferably hrtimer_start*() would simply return
-ETIMEand let the caller sort it, sadly the current behaviour is to 'fix' it for the caller by enqueueing the timer onto the softirq list and raising the softirq.I guess we could make hrtimer_start*(.wakeup=false) return the
-ENOTIMEthing and audit those few use-cases.
That would be sensible anyway.