tick: make sleep length calculation more accurate
PATCH 1/1 tick: make sleep length calculation more accurate
The sleep_length is computed in the tick_nohz_stop_sched_tick function but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter [ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...
arch_cpu_idle
menu_select [ uses here 'sleep_length' ]
...
Between the computation of the sleep length and its usage, some interrupts can occur, making the sleep length shorter than actually it is.
This patch fixes that by moving the sleep_length computation in the tick_nohz_get_sleep_length function and store the next_event for the device instead of the sleep_length.
--- include/linux/tick.h | 2 +- kernel/time/tick-sched.c | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/tick.h +1 -1
@@ -67,7 +67,7 @@ struct tick_sched {ktime_t idle_exittime;ktime_t idle_sleeptime;ktime_t iowait_sleeptime; -ktime_t sleep_length; +ktime_t next_event;unsigned long last_jiffies;unsigned long next_jiffies;ktime_t idle_expires;
kernel/time/tick-sched.c +3 -2
@@ -673,7 +673,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, out: ts->next_jiffies = next_jiffies; ts->last_jiffies = last_jiffies; - ts->sleep_length =ktime_sub (dev->next_event, now); + ts->next_event = dev->next_event;return ret; }
@@ -837,8 +837,9 @@ void tick_nohz_irq_exit(void)ktime_t tick_nohz_get_sleep_length (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +ktime_t now =ktime_get (); -return ts->sleep_length; +return ktime_sub (ts->next_event, now); }static void tick_nohz_restart (struct tick_sched *ts,ktime_t now)
Signature · 1 line
1.7.9.5
Quoted 31 lines · Daniel Lezcano
The
sleep_lengthis computed in thetick_nohz_stop_sched_tickfunction but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter[ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...
arch_cpu_idle
menu_select[ uses here 'sleep_length' ]
...Between the computation of the sleep length and its usage, some interrupts can occur, making the sleep length shorter than actually it is.
This patch fixes that by moving the
sleep_lengthcomputation in thetick_nohz_get_sleep_lengthfunction and store thenext_eventfor the device instead of thesleep_length.Signed-off-by Daniel Lezcano <[email protected]>--- include/linux/tick.h | 2 +- kernel/time/tick-sched.c | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-)include/linux/tick.h +0
-0
include/linux/tick.h struct tick_sched { · line 73
Quoted 8 lines
include/linux/tick.h +1
-1@@ -67,7 +67,7 @@ struct tick_sched {ktime_t idle_exittime;ktime_t idle_sleeptime;ktime_t iowait_sleeptime; -ktime_t sleep_length; +ktime_t next_event;unsigned long last_jiffies;unsigned long next_jiffies;ktime_t idle_expires;
Documentation update?
Quoted 14 lines
kernel/time/tick-sched.c +1
-1@@ -673,7 +673,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, out: ts->next_jiffies = next_jiffies; ts->last_jiffies = last_jiffies; - ts->sleep_length =ktime_sub (dev->next_event, now); + ts->next_event = dev->next_event;return ret; }
kernel/time/tick-sched.c void tick_nohz_irq_exit(void) · line 845
Quoted 10 lines
kernel/time/tick-sched.c +2
-1@@ -837,8 +837,9 @@ void tick_nohz_irq_exit(void)ktime_t tick_nohz_get_sleep_length (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +ktime_t now =ktime_get (); -return ts->sleep_length; +return ktime_sub (ts->next_event, now); }static void tick_nohz_restart (struct tick_sched *ts,ktime_t now)
What happens if the idling CPU's next_event is updated via that interrupt? Say if the interrupt handler schedules a timer to fire before the next timer on the CPU? It looks like we won't notice that.
Perhaps it's better to do this instead?
ktime_t tick_nohz_get_sleep_length(void)
{
struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);
+ ktime_t now = ktime_get();
+ struct clock_event_device *dev = __get_cpu_var(tick_cpu_device).evtdev;
- return ts->sleep_length;
+ return ktime_sub(dev->next_event, now);
}
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Quoted 66 lines · Stephen Boyd
Quoted 31 lines · Daniel Lezcano
The
sleep_lengthis computed in thetick_nohz_stop_sched_tickfunction but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter[ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...
arch_cpu_idle
menu_select[ uses here 'sleep_length' ]
...Between the computation of the sleep length and its usage, some interrupts can occur, making the sleep length shorter than actually it is.
This patch fixes that by moving the
sleep_lengthcomputation in thetick_nohz_get_sleep_lengthfunction and store thenext_eventfor the device instead of thesleep_length.Signed-off-by Daniel Lezcano <[email protected]>--- include/linux/tick.h | 2 +- kernel/time/tick-sched.c | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-)include/linux/tick.h +0
-0include/linux/tick.h struct tick_sched { · line 73
Quoted 8 lines
include/linux/tick.h +1
-1@@ -67,7 +67,7 @@ struct tick_sched {ktime_t idle_exittime;ktime_t idle_sleeptime;ktime_t iowait_sleeptime; -ktime_t sleep_length; +ktime_t next_event;unsigned long last_jiffies;unsigned long next_jiffies;ktime_t idle_expires;Documentation update?
Quoted 23 lines
kernel/time/tick-sched.c +3
-2@@ -673,7 +673,7 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, out: ts->next_jiffies = next_jiffies; ts->last_jiffies = last_jiffies; - ts->sleep_length =ktime_sub (dev->next_event, now); + ts->next_event = dev->next_event;return ret; }@@ -837,8 +837,9 @@ void tick_nohz_irq_exit(void)ktime_t tick_nohz_get_sleep_length (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +ktime_t now =ktime_get (); -return ts->sleep_length; +return ktime_sub (ts->next_event, now); }
Quoted 5 lines
kernel/time/tick-sched.c void tick_nohz_irq_exit(void) · line 845
Quoted 1 line
kernel/time/tick-sched.c +0
-0static void tick_nohz_restart (struct tick_sched *ts,ktime_t now)What happens if the idling CPU's
next_eventis updated via that interrupt? Say if the interrupt handler schedules a timer to fire before the next timer on the CPU? It looks like we won't notice that.
Yes, or after.
It sounds like this issue also occurs with the current code, no ?
kernel/time/tick-sched.c void tick_nohz_irq_exit(void) · line 843
Quoted 11 lines
Perhaps it's better to do this instead?
ktime_t tick_nohz_get_sleep_length(void) { struct tick_sched *ts = &__get_cpu_var(tick_cpu_sched);kernel/time/tick-sched.c +3
-1+ktime_t now =ktime_get (); +struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev; -return ts->sleep_length; +return ktime_sub (dev->next_event, now); }
Yes, I agree.
Thanks for the review.
-- Daniel
Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook | <http://twitter.com/#!/linaroorg> Twitter | <http://www.linaro.org/linaro-blog/> Blog