tick: Make sleep length calculation more accurate
PATCH v2 1/1 tick: Make sleep length calculation more accurate
The sleep_length is computed in the tick_nohz_stop_sched_tick function but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter [ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...
arch_cpu_idle
menu_select [ uses here 'sleep_length' ]
...
Between the computation of the sleep length and its usage, some interrupts may occur, making the sleep length shorter than actually it is because of the interrupt processing, or different if the timer itself expired.
This patch fixes that by moving the sleep_length computation in the tick_nohz_get_sleep_length function and using the tick device's next_event.
As the sleep_length field is no longer needed, it is removed from the tick_sched structure.
--- include/linux/tick.h | 2 -- kernel/time/tick-sched.c | 5 +++-- 2 files changed, 3 insertions(+), 4 deletions(-)
include/linux/tick.h +0 -2
@@ -48,7 +48,6 @@ enum tick_nohz_mode {* @idle_exittime: Time when the idle state was left * @idle_sleeptime: Sum of the time slept in idle with sched tick stopped * @iowait_sleeptime: Sum of the time slept in idle with sched tick stopped, with IO outstanding -* @sleep_length: Duration of the current idle sleep * @do_timer_lst: CPU was the last one doing do_timer before going idle */ struct tick_sched {
@@ -67,7 +66,6 @@ struct tick_sched {ktime_t idle_exittime;ktime_t idle_sleeptime;ktime_t iowait_sleeptime; -ktime_t sleep_length;unsigned long last_jiffies;unsigned long next_jiffies;ktime_t idle_expires;
kernel/time/tick-sched.c +3 -2
@@ -673,7 +673,6 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, out: ts->next_jiffies = next_jiffies; ts->last_jiffies = last_jiffies; - ts->sleep_length =ktime_sub (dev->next_event, now);return ret; }
@@ -837,8 +836,10 @@ void tick_nohz_irq_exit(void)ktime_t tick_nohz_get_sleep_length (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev; +ktime_t now =ktime_get (); -return ts->sleep_length; +return ktime_sub (dev->next_event, now); }static void tick_nohz_restart (struct tick_sched *ts,ktime_t now)
Signature · 1 line
1.7.9.5
Quoted 17 lines · Daniel Lezcano
The
sleep_lengthis computed in thetick_nohz_stop_sched_tickfunction but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter[ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...
arch_cpu_idle
menu_select[ uses here 'sleep_length' ]
...Between the computation of the sleep length and its usage, some interrupts may occur, making the sleep length shorter than actually it is because of the interrupt processing, or different if the timer itself expired.
Quoted 8 lines
This patch fixes that by moving the
sleep_lengthcomputation in thetick_nohz_get_sleep_lengthfunction and using the tick device'snext_event.As the
sleep_lengthfield is no longer needed, it is removed from thetick_schedstructure.Signed-off-by Daniel Lezcano <[email protected]>Signed-off-by Stephen Boyd <[email protected]>
Stephen,
I added your signed-off as the change in tick_nohz_get_sleep_length is the one you proposed previously.
-- Daniel
Quoted 50 lines
--- include/linux/tick.h | 2 -- kernel/time/tick-sched.c | 5 +++-- 2 files changed, 3 insertions(+), 4 deletions(-)include/linux/tick.h +0
-2@@ -48,7 +48,6 @@ enum tick_nohz_mode {* @idle_exittime: Time when the idle state was left * @idle_sleeptime: Sum of the time slept in idle with sched tick stopped * @iowait_sleeptime: Sum of the time slept in idle with sched tick stopped, with IO outstanding -* @sleep_length: Duration of the current idle sleep * @do_timer_lst: CPU was the last one doing do_timer before going idle */ struct tick_sched {@@ -67,7 +66,6 @@ struct tick_sched {ktime_t idle_exittime;ktime_t idle_sleeptime;ktime_t iowait_sleeptime; -ktime_t sleep_length;unsigned long last_jiffies;unsigned long next_jiffies;ktime_t idle_expires;kernel/time/tick-sched.c +3
-2@@ -673,7 +673,6 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, out: ts->next_jiffies = next_jiffies; ts->last_jiffies = last_jiffies; - ts->sleep_length =ktime_sub (dev->next_event, now);return ret; }@@ -837,8 +836,10 @@ void tick_nohz_irq_exit(void)ktime_t tick_nohz_get_sleep_length (void ) {struct tick_sched *ts = &__get_cpu_var (tick_cpu_sched); +struct clock_event_device *dev =__get_cpu_var (tick_cpu_device).evtdev; +ktime_t now =ktime_get (); -return ts->sleep_length; +return ktime_sub (dev->next_event, now); }static void tick_nohz_restart (struct tick_sched *ts,ktime_t now)
Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook | <http://twitter.com/#!/linaroorg> Twitter | <http://www.linaro.org/linaro-blog/> Blog
2013/10/2 Daniel Lezcano <[email protected]>:
Quoted 9 lines
The
sleep_lengthis computed in thetick_nohz_stop_sched_tickfunction but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter[ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...
Quoted 7 lines
arch_cpu_idle
menu_select[ uses here 'sleep_length' ]
...Between the computation of the sleep length and its usage, some interrupts may occur, making the sleep length shorter than actually it is because of the interrupt processing
So, do you mean that the ts->sleep_length would return a value that is too long given that the CPU already spent some time to service the irqs since we computed the sleep length in tick_nohz_idle_enter()?
But then tick_nohz_irq_exit() should take care of that as it calls again tick_nohz_stop_sched_tick().
So I'm a bit confused.
Quoted 1 line
or different if the timer itself expired.
Same here, if the timer expired, it triggers an interrupt which can do two things:
1) reprogram a new timer and this recompute sleep_length
2) set_need_resched() and then exit the idle loop, so arch_cpu_idle() won't even be called. Or the timer interrupts hlt, but then menu_select() was called before.
So I probably missed something here.
Thanks.
Quoted 31 lines · Frederic Weisbecker
2013/10/2 Daniel Lezcano <[email protected]>:
Quoted 9 lines
The
sleep_lengthis computed in thetick_nohz_stop_sched_tickfunction but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter[ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...Quoted 7 lines
arch_cpu_idle
menu_select[ uses here 'sleep_length' ]
...Between the computation of the sleep length and its usage, some interrupts may occur, making the sleep length shorter than actually it is because of the interrupt processing
So, do you mean that the
ts->sleep_lengthwould return a value that is too long given that the CPU already spent some time to service the irqs since we computed the sleep length intick_nohz_idle_enter()?But then
tick_nohz_irq_exit()should take care of that as it calls againtick_nohz_stop_sched_tick().
So I'm a bit confused.Quoted 1 line
or different if the timer itself expired.
Same here, if the timer expired, it triggers an interrupt which can do two things:
Quoted 6 lines
1) reprogram a new timer and this recompute
sleep_length
2)set_need_resched()and then exit the idle loop, soarch_cpu_idle()won't even be called. Or the timer interrupts hlt, but thenmenu_select()was called before.So I probably missed something here.
No you did not :)
Indeed... At the first glance, this issue sounded so obvious I suspected there must be a trick somewhere but I did not think to look at the irq_exit, the code is very complex. Thanks for clarifying this.
For my personal information, is there any particular reason to set an intermediate 'sleep_length' in tick_nohz_stop_sched_tick instead of doing what does this patch ?
Thanks
-- Daniel
Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook | <http://twitter.com/#!/linaroorg> Twitter | <http://www.linaro.org/linaro-blog/> Blog
Quoted 41 lines · Daniel Lezcano
Quoted 31 lines · Frederic Weisbecker
2013/10/2 Daniel Lezcano <[email protected]>:
Quoted 9 lines
The
sleep_lengthis computed in thetick_nohz_stop_sched_tickfunction but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter[ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...Quoted 7 lines
arch_cpu_idle
menu_select[ uses here 'sleep_length' ]
...Between the computation of the sleep length and its usage, some interrupts may occur, making the sleep length shorter than actually it is because of the interrupt processing
So, do you mean that the
ts->sleep_lengthwould return a value that is too long given that the CPU already spent some time to service the irqs since we computed the sleep length intick_nohz_idle_enter()?But then
tick_nohz_irq_exit()should take care of that as it calls againtick_nohz_stop_sched_tick().
So I'm a bit confused.Quoted 1 line
or different if the timer itself expired.
Same here, if the timer expired, it triggers an interrupt which can do two things:
Quoted 6 lines
1) reprogram a new timer and this recompute
sleep_length
2)set_need_resched()and then exit the idle loop, soarch_cpu_idle()won't even be called. Or the timer interrupts hlt, but thenmenu_select()was called before.So I probably missed something here.
No you did not :)
Quoted 8 lines
Indeed... At the first glance, this issue sounded so obvious I suspected there must be a trick somewhere but I did not think to look at the
irq_exit, the code is very complex. Thanks for clarifying this.For my personal information, is there any particular reason to set an intermediate '
sleep_length' intick_nohz_stop_sched_tickinstead of doing what does this patch ?
May be we could do it that way yeah. Is menu_select() called only there? I don't know how much difference that would make.
Quoted 11 lines
Thanks
-- DanielSignature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook | <http://twitter.com/#!/linaroorg> Twitter | <http://www.linaro.org/linaro-blog/> Blog
Quoted 47 lines · Frederic Weisbecker
Quoted 46 lines · Daniel Lezcano
Quoted 31 lines · Frederic Weisbecker
2013/10/2 Daniel Lezcano <[email protected]>:
Quoted 9 lines
The
sleep_lengthis computed in thetick_nohz_stop_sched_tickfunction but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter[ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...Quoted 7 lines
arch_cpu_idle
menu_select[ uses here 'sleep_length' ]
...Between the computation of the sleep length and its usage, some interrupts may occur, making the sleep length shorter than actually it is because of the interrupt processing
So, do you mean that the
ts->sleep_lengthwould return a value that is too long given that the CPU already spent some time to service the irqs since we computed the sleep length intick_nohz_idle_enter()?But then
tick_nohz_irq_exit()should take care of that as it calls againtick_nohz_stop_sched_tick().
So I'm a bit confused.Quoted 1 line
or different if the timer itself expired.
Same here, if the timer expired, it triggers an interrupt which can do two things:
Quoted 6 lines
1) reprogram a new timer and this recompute
sleep_length
2)set_need_resched()and then exit the idle loop, soarch_cpu_idle()won't even be called. Or the timer interrupts hlt, but thenmenu_select()was called before.So I probably missed something here.
No you did not :)
Indeed... At the first glance, this issue sounded so obvious I suspected there must be a trick somewhere but I did not think to look at the
irq_exit, the code is very complex. Thanks for clarifying this.
Quoted 6 lines
Quoted 3 lines
For my personal information, is there any particular reason to set an intermediate '
sleep_length' intick_nohz_stop_sched_tickinstead of doing what does this patch ?May be we could do it that way yeah. Is
menu_select()called only there? I don't know how much difference that would make.
Yes, it is called just one time in all the code. The benefit would be just to cleanup a field in the struct tick_sched.
Signature · 5 lines
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook | <http://twitter.com/#!/linaroorg> Twitter | <http://www.linaro.org/linaro-blog/> Blog
Quoted 52 lines · Daniel Lezcano
Quoted 51 lines · Frederic Weisbecker
Quoted 49 lines · Daniel Lezcano
Quoted 31 lines · Frederic Weisbecker
2013/10/2 Daniel Lezcano <[email protected]>:
Quoted 9 lines
The
sleep_lengthis computed in thetick_nohz_stop_sched_tickfunction but it is used later in the code with in between the local irq enabled.
cpu_idle_loop
tick_nohz_idle_enter[ exits with local irq enabled ]
__tick_nohz_idle_enter
tick_nohz_stop_sched_tick
...Quoted 7 lines
arch_cpu_idle
menu_select[ uses here 'sleep_length' ]
...Between the computation of the sleep length and its usage, some interrupts may occur, making the sleep length shorter than actually it is because of the interrupt processing
So, do you mean that the
ts->sleep_lengthwould return a value that is too long given that the CPU already spent some time to service the irqs since we computed the sleep length intick_nohz_idle_enter()?But then
tick_nohz_irq_exit()should take care of that as it calls againtick_nohz_stop_sched_tick().
So I'm a bit confused.Quoted 1 line
or different if the timer itself expired.
Same here, if the timer expired, it triggers an interrupt which can do two things:
Quoted 6 lines
1) reprogram a new timer and this recompute
sleep_length
2)set_need_resched()and then exit the idle loop, soarch_cpu_idle()won't even be called. Or the timer interrupts hlt, but thenmenu_select()was called before.So I probably missed something here.
No you did not :)
Indeed... At the first glance, this issue sounded so obvious I suspected there must be a trick somewhere but I did not think to look at the
irq_exit, the code is very complex. Thanks for clarifying this.For my personal information, is there any particular reason to set an intermediate '
sleep_length' intick_nohz_stop_sched_tickinstead of doing what does this patch ?
Quoted 5 lines
Quoted 2 lines
May be we could do it that way yeah. Is
menu_select()called only there? I don't know how much difference that would make.Yes, it is called just one time in all the code. The benefit would be just to cleanup a field in the struct
tick_sched.
Yeah, why not.
Thanks.
Quoted 9 lines
Signature · 6 lines
-- <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook | <http://twitter.com/#!/linaroorg> Twitter | <http://www.linaro.org/linaro-blog/> Blog