timer: Fix missing dynticks kick v2
cover timer: Fix missing dynticks kick v2
Hi,
It's a 2nd set that fixes some missing dyntick kicks in the timer's code. This new version also handles missing kicks in the hrtimers subsystem.
The patches are also available at:
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
timers/missing-kick-v2
Thanks,
Frederic
---
Viresh Kumar (5):
timer: Store cpu-number in 'struct tvec_base'
timer: Kick dynticks targets on mod_timer*() calls
hrtimer: Store cpu-number in 'struct hrtimer_cpu_base'
hrtimer: Kick lowres dynticks targets on timer enqueue
hrtimer: Remove hrtimer_enqueue_reprogram()
include/linux/hrtimer.h | 2 ++ kernel/hrtimer.c | 49 +++++++++++++++++++++++++++---------------------- kernel/timer.c | 34 ++++++++++++++++++---------------- 3 files changed, 47 insertions(+), 38 deletions(-)
PATCH 1/5 timer: Store cpu-number in 'struct tvec_base'
From: Viresh Kumar <[email protected]>
Timers are serviced by the tick. But when a timer is enqueued on a dynticks target, we need to kick it in order to make it reconsider the next tick to schedule to correctly handle the timer's expiring time.
Now while this kick is correctly performed for add_timer_on(), the mod_timer*() family has been a bit neglected.
To prepare for fixing this, we need internal_add_timer() to be able to resolve the CPU target associated to a timer's object 'base' so that the kick can be centralized there.
This can't be passed as an argument as not all the callers know the CPU number of a timer's base. So lets store it in the struct tvec_base to resolve the CPU without much overhead. It is set once for good at every CPU's first boot.
--- kernel/timer.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/timer.c +2 -0
@@ -82,6 +82,7 @@ struct tvec_base {unsigned long next_timer;unsigned long active_timers;unsigned long all_timers; +int cpu;struct tvec_root tv1;struct tvec tv2;struct tvec tv3;
@@ -1568,6 +1569,7 @@ static int init_timers_cpu(int cpu) }spin_lock_init (&base->lock); tvec_base_done[cpu] =1 ; + base->cpu = cpu; }else { base =per_cpu (tvec_bases, cpu); }
Signature · 1 line
1.8.3.1
tip-bot for Viresh Kumar
merge bot[tip:timers/core] timer: Store cpu-number in struct tvec_base
timer: Store cpu-number in struct tvec_base
Commit-ID: d6f93829811a3e74f58e3c3823d507411eed651a Gitweb: http://git.kernel.org/tip/d6f93829811a3e74f58e3c3823d507411eed651a Author: Viresh Kumar <[email protected]> AuthorDate: Sun, 22 Jun 2014 01:29:13 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Mon, 23 Jun 2014 11:23:46 +0200
timer: Store cpu-number in struct tvec_base
Timers are serviced by the tick. But when a timer is enqueued on a dynticks target, we need to kick it in order to make it reconsider the next tick to schedule to correctly handle the timer's expiring time.
Now while this kick is correctly performed for add_timer_on(), the mod_timer*() family has been a bit neglected.
To prepare for fixing this, we need internal_add_timer() to be able to resolve the CPU target associated to a timer's object 'base' so that the kick can be centralized there.
This can't be passed as an argument as not all the callers know the CPU number of a timer's base. So lets store it in the struct tvec_base to resolve the CPU without much overhead. It is set once for good at every CPU's first boot.
--- kernel/time/timer.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/time/timer.c +2 -0
@@ -82,6 +82,7 @@ struct tvec_base {unsigned long next_timer;unsigned long active_timers;unsigned long all_timers; +int cpu;struct tvec_root tv1;struct tvec tv2;struct tvec tv3;
@@ -1568,6 +1569,7 @@ static int init_timers_cpu(int cpu) }spin_lock_init (&base->lock); tvec_base_done[cpu] =1 ; + base->cpu = cpu; }else { base =per_cpu (tvec_bases, cpu); }
PATCH 2/5 timer: Kick dynticks targets on mod_timer*() calls
In reply to Frederic WeisbeckerFrom: Viresh Kumar <[email protected]>
When a timer is enqueued or modified on a dynticks target, that CPU must re-evaluate the next tick to service that timer.
The tick re-evaluation is performed by an IPI kick on the target. Now while we correctly call wake_up_nohz_cpu() from add_timer_on(), the mod_timer*() API family doesn't support so well dynticks targets.
The reason for this is likely that __mod_timer() isn't supposed to select an idle target for a timer, unless that target is the current CPU, in which case a dynticks idle kick isn't actually needed.
But there is a small race window lurking behind that assumption: the elected target has all the time to turn dynticks idle between the call to get_nohz_timer_target() and the locking of its base. Hence a risk that we enqueue a timer on a dynticks idle destination without kicking it. As a result, the timer might be serviced too late in the future.
Also a target elected by __mod_timer() can be in full dynticks mode and thus require to be kicked as well. And unlike idle dynticks, this concern both local and remote targets.
To fix this whole issue, lets centralize the dynticks kick to internal_add_timer() so that it is well handled for all sort of timer enqueue. Even timer migration is concerned so that a full dynticks target is correctly kicked as needed when timers are migrating to it.
--- kernel/timer.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
kernel/timer.c +16 -16
@@ -410,6 +410,22 @@ static void internal_add_timer(struct tvec_base *base, struct timer_list *timer) base->next_timer = timer->expires; } base->all_timers++; + +/* +* Check whether the other CPU is in dynticks mode and needs +* to be triggered to reevaluate the timer wheel. +* We are protected against the other CPU fiddling +* with the timer by holding the timer base lock. This also +* makes sure that a CPU on the way to stop its tick can not +* evaluate the timer wheel. +* +* Spare the IPI for deferrable timers on idle targets though. +* The next busy ticks will take care of it. Except full dynticks +* require special care against races with idle_cpu(), lets deal +* with that later. +*/ +if (!tbase_get_deferrable (base) ||tick_nohz_full_cpu (base->cpu)) +wake_up_nohz_cpu (base->cpu); }#ifdef CONFIG_TIMER_STATS
@@ -949,22 +965,6 @@ void add_timer_on(struct timer_list *timer, int cpu)timer_set_base (timer, base);debug_activate (timer, timer->expires);internal_add_timer (base, timer); -/* -* Check whether the other CPU is in dynticks mode and needs -* to be triggered to reevaluate the timer wheel. -* We are protected against the other CPU fiddling -* with the timer by holding the timer base lock. This also -* makes sure that a CPU on the way to stop its tick can not -* evaluate the timer wheel. -* -* Spare the IPI for deferrable timers on idle targets though. -* The next busy ticks will take care of it. Except full dynticks -* require special care against races with idle_cpu(), lets deal -* with that later. -*/ -if (!tbase_get_deferrable (timer->base) ||tick_nohz_full_cpu (cpu)) -wake_up_nohz_cpu (cpu); -spin_unlock_irqrestore (&base->lock, flags); }EXPORT_SYMBOL_GPL (add_timer_on);
Signature · 1 line
1.8.3.1
tip-bot for Viresh Kumar
merge bot[tip:timers/core] timer: Kick dynticks targets on mod_timer*() calls
timer: Kick dynticks targets on mod_timer*() calls
Commit-ID: 9f6d9baaa8ca94b48aea495261cadaf2967c7784 Gitweb: http://git.kernel.org/tip/9f6d9baaa8ca94b48aea495261cadaf2967c7784 Author: Viresh Kumar <[email protected]> AuthorDate: Sun, 22 Jun 2014 01:29:14 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Mon, 23 Jun 2014 11:23:47 +0200
timer: Kick dynticks targets on mod_timer*() calls
When a timer is enqueued or modified on a dynticks target, that CPU must re-evaluate the next tick to service that timer.
The tick re-evaluation is performed by an IPI kick on the target. Now while we correctly call wake_up_nohz_cpu() from add_timer_on(), the mod_timer*() API family doesn't support so well dynticks targets.
The reason for this is likely that __mod_timer() isn't supposed to select an idle target for a timer, unless that target is the current CPU, in which case a dynticks idle kick isn't actually needed.
But there is a small race window lurking behind that assumption: the elected target has all the time to turn dynticks idle between the call to get_nohz_timer_target() and the locking of its base. Hence a risk that we enqueue a timer on a dynticks idle destination without kicking it. As a result, the timer might be serviced too late in the future.
Also a target elected by __mod_timer() can be in full dynticks mode and thus require to be kicked as well. And unlike idle dynticks, this concern both local and remote targets.
To fix this whole issue, lets centralize the dynticks kick to internal_add_timer() so that it is well handled for all sort of timer enqueue. Even timer migration is concerned so that a full dynticks target is correctly kicked as needed when timers are migrating to it.
--- kernel/time/timer.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
kernel/time/timer.c +16 -16
@@ -410,6 +410,22 @@ static void internal_add_timer(struct tvec_base *base, struct timer_list *timer) base->next_timer = timer->expires; } base->all_timers++; + +/* +* Check whether the other CPU is in dynticks mode and needs +* to be triggered to reevaluate the timer wheel. +* We are protected against the other CPU fiddling +* with the timer by holding the timer base lock. This also +* makes sure that a CPU on the way to stop its tick can not +* evaluate the timer wheel. +* +* Spare the IPI for deferrable timers on idle targets though. +* The next busy ticks will take care of it. Except full dynticks +* require special care against races with idle_cpu(), lets deal +* with that later. +*/ +if (!tbase_get_deferrable (base) ||tick_nohz_full_cpu (base->cpu)) +wake_up_nohz_cpu (base->cpu); }#ifdef CONFIG_TIMER_STATS
@@ -949,22 +965,6 @@ void add_timer_on(struct timer_list *timer, int cpu)timer_set_base (timer, base);debug_activate (timer, timer->expires);internal_add_timer (base, timer); -/* -* Check whether the other CPU is in dynticks mode and needs -* to be triggered to reevaluate the timer wheel. -* We are protected against the other CPU fiddling -* with the timer by holding the timer base lock. This also -* makes sure that a CPU on the way to stop its tick can not -* evaluate the timer wheel. -* -* Spare the IPI for deferrable timers on idle targets though. -* The next busy ticks will take care of it. Except full dynticks -* require special care against races with idle_cpu(), lets deal -* with that later. -*/ -if (!tbase_get_deferrable (timer->base) ||tick_nohz_full_cpu (cpu)) -wake_up_nohz_cpu (cpu); -spin_unlock_irqrestore (&base->lock, flags); }EXPORT_SYMBOL_GPL (add_timer_on);
PATCH 3/5 hrtimer: Store cpu-number in 'struct hrtimer_cpu_base'
In reply to Frederic WeisbeckerFrom: Viresh Kumar <[email protected]>
In lowres mode, hrtimers are serviced by the tick instead of a clock event. Now it works well as long as the tick stays periodic but we must also make sure that the hrtimers are serviced in dynticks mode.
Part of that job consist in kicking a dynticks hrtimer target in order to make it reconsider the next tick to schedule to correctly handle the hrtimer's expiring time. And that part isn't handled by the hrtimers subsystem.
To prepare for fixing this, we need __hrtimer_start_range_ns() to be able to resolve the CPU target associated to a hrtimer's object 'cpu_base' so that the kick can be centralized there.
So lets store it in the 'struct hrtimer_cpu_base' to resolve the CPU without overhead. It is set once at CPU's online notification.
--- include/linux/hrtimer.h | 2 ++ kernel/hrtimer.c | 1 + 2 files changed, 3 insertions(+)
include/linux/hrtimer.h +2 -0
@@ -165,6 +165,7 @@ enum hrtimer_base_type {* struct hrtimer_cpu_base - the per cpu clock bases * @lock: lock protecting the base and associated clock bases * and timers +* @cpu: cpu number * @active_bases: Bitfield to mark bases with active timers * @clock_was_set: Indicates that clock was set from irq context. * @expires_next: absolute time of the next event which was scheduled
@@ -179,6 +180,7 @@ enum hrtimer_base_type {*/ struct hrtimer_cpu_base {raw_spinlock_t lock; +unsigned int cpu;unsigned int active_bases;unsigned int clock_was_set;#ifdef CONFIG_HIGH_RES_TIMERS
kernel/hrtimer.c +1 -0
@@ -1680,6 +1680,7 @@ static void init_hrtimers_cpu(int cpu)timerqueue_init_head (&cpu_base->clock_base[i].active); } + cpu_base->cpu = cpu;hrtimer_init_hres (cpu_base); }
Signature · 1 line
1.8.3.1
tip-bot for Viresh Kumar
merge bot[tip:timers/core] hrtimer: Store cpu-number in struct hrtimer_cpu_base
hrtimer: Store cpu-number in struct hrtimer_cpu_base
Commit-ID: cddd02489f52ccf635ed65931214729a23b93cd6 Gitweb: http://git.kernel.org/tip/cddd02489f52ccf635ed65931214729a23b93cd6 Author: Viresh Kumar <[email protected]> AuthorDate: Sun, 22 Jun 2014 01:29:15 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Mon, 23 Jun 2014 11:23:47 +0200
hrtimer: Store cpu-number in struct hrtimer_cpu_base
In lowres mode, hrtimers are serviced by the tick instead of a clock event. Now it works well as long as the tick stays periodic but we must also make sure that the hrtimers are serviced in dynticks mode.
Part of that job consist in kicking a dynticks hrtimer target in order to make it reconsider the next tick to schedule to correctly handle the hrtimer's expiring time. And that part isn't handled by the hrtimers subsystem.
To prepare for fixing this, we need __hrtimer_start_range_ns() to be able to resolve the CPU target associated to a hrtimer's object 'cpu_base' so that the kick can be centralized there.
So lets store it in the 'struct hrtimer_cpu_base' to resolve the CPU without overhead. It is set once at CPU's online notification.
--- include/linux/hrtimer.h | 2 ++ kernel/time/hrtimer.c | 1 + 2 files changed, 3 insertions(+)
include/linux/hrtimer.h +2 -0
@@ -165,6 +165,7 @@ enum hrtimer_base_type {* struct hrtimer_cpu_base - the per cpu clock bases * @lock: lock protecting the base and associated clock bases * and timers +* @cpu: cpu number * @active_bases: Bitfield to mark bases with active timers * @clock_was_set: Indicates that clock was set from irq context. * @expires_next: absolute time of the next event which was scheduled
@@ -179,6 +180,7 @@ enum hrtimer_base_type {*/ struct hrtimer_cpu_base {raw_spinlock_t lock; +unsigned int cpu;unsigned int active_bases;unsigned int clock_was_set;#ifdef CONFIG_HIGH_RES_TIMERS
kernel/time/hrtimer.c +1 -0
@@ -1680,6 +1680,7 @@ static void init_hrtimers_cpu(int cpu)timerqueue_init_head (&cpu_base->clock_base[i].active); } + cpu_base->cpu = cpu;hrtimer_init_hres (cpu_base); }
PATCH 4/5 hrtimer: Kick lowres dynticks targets on timer enqueue
In reply to Frederic WeisbeckerFrom: Viresh Kumar <[email protected]>
In lowres mode, hrtimers are serviced by the tick instead of a clock event. It works well as long as the tick stays periodic but we must also make sure that the hrtimers are serviced in dynticks mode targets, pretty much like timer list timers do.
Note that all dynticks modes are concerned: get_nohz_timer_target() tries not to return remote idle CPUs but there is nothing to prevent the elected target from entering dynticks idle mode until we lock its base. It's also prefectly legal to enqueue hrtimers on full dynticks CPU.
So there are two requirements to correctly handle dynticks:
1) On target's tick stop time, we must not delay the next tick further the next hrtimer.
2) On hrtimer queue time. If the tick of the target is stopped, we must wake up that CPU such that it sees the new hrtimer and recalculate the next tick accordingly.
The point 1 is well handled currently through get_nohz_timer_interrupt() and cmp_next_hrtimer_event().
But the point 2 isn't handled at all.
Fixing this is easy though as we have the necessary API ready for that. All we need is to call wake_up_nohz_cpu() on a target when a newly enqueued hrtimer requires tick rescheduling, like timer list timer do.
--- kernel/hrtimer.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
kernel/hrtimer.c +19 -8
@@ -1013,14 +1013,25 @@ int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim, leftmost =enqueue_hrtimer (timer, new_base); -/* -* Only allow reprogramming if the new base is on this CPU. -* (it might still be on another CPU if the timer was pending) -* -* XXX send_remote_softirq() ? -*/ -if (leftmost && new_base->cpu_base == &__get_cpu_var (hrtimer_bases) - &&hrtimer_enqueue_reprogram (timer, new_base)) { +if (!leftmost) { +unlock_hrtimer_base (timer, &flags); +return ret; + } + +if (!hrtimer_is_hres_active (timer)) { +/* +* Kick to reschedule the next tick to handle the new timer +* on dynticks target. +*/ +wake_up_nohz_cpu (base->cpu_base->cpu); + }else if (new_base->cpu_base == &__get_cpu_var (hrtimer_bases) && +hrtimer_enqueue_reprogram (timer, new_base)) { +/* +* Only allow reprogramming if the new base is on this CPU. +* (it might still be on another CPU if the timer was pending) +* +* XXX send_remote_softirq() ? +*/ if (wakeup) {/* * We need to drop cpu_base->lock to avoid a
Signature · 1 line
1.8.3.1
Re: [PATCH 4/5] hrtimer: Kick lowres dynticks targets on timer enqueue
Quoted 53 lines · Frederic Weisbecker
From: Viresh Kumar <[email protected]>
In lowres mode, hrtimers are serviced by the tick instead of a clock event. It works well as long as the tick stays periodic but we must also make sure that the hrtimers are serviced in dynticks mode targets, pretty much like timer list timers do.
Note that all dynticks modes are concerned:
get_nohz_timer_target()tries not to return remote idle CPUs but there is nothing to prevent the elected target from entering dynticks idle mode until we lock its base. It's also prefectly legal to enqueue hrtimers on full dynticks CPU.So there are two requirements to correctly handle dynticks:
1) On target's tick stop time, we must not delay the next tick further the next hrtimer.
2) On hrtimer queue time. If the tick of the target is stopped, we must wake up that CPU such that it sees the new hrtimer and recalculate the next tick accordingly.
The point 1 is well handled currently through
get_nohz_timer_interrupt()andcmp_next_hrtimer_event().But the point 2 isn't handled at all.
Fixing this is easy though as we have the necessary API ready for that. All we need is to call
wake_up_nohz_cpu()on a target when a newly enqueued hrtimer requires tick rescheduling, like timer list timer do.Signed-off-by Viresh Kumar <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/hrtimer.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)kernel/hrtimer.c +1
-8@@ -1013,14 +1013,25 @@ int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim, leftmost =enqueue_hrtimer (timer, new_base); -/* -* Only allow reprogramming if the new base is on this CPU. -* (it might still be on another CPU if the timer was pending) -* -* XXX send_remote_softirq() ? -*/ -if (leftmost && new_base->cpu_base == &__get_cpu_var (hrtimer_bases) - &&hrtimer_enqueue_reprogram (timer, new_base)) { +if (!leftmost) {
kernel/hrtimer.c int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim, · line 1026
Quoted 10 lines
kernel/hrtimer.c +10
-0+unlock_hrtimer_base (timer, &flags); +return ret; + } + +if (!hrtimer_is_hres_active (timer)) { +/* +* Kick to reschedule the next tick to handle the new timer +* on dynticks target. +*/ +wake_up_nohz_cpu (base->cpu_base->cpu);
The timer is queued on new_base not on base.
Thanks,
tglx
Re: [PATCH 4/5] hrtimer: Kick lowres dynticks targets on timer enqueue
Quoted 5 lines · Thomas Gleixner
Quoted 1 line · Frederic Weisbecker
+
wake_up_nohz_cpu(base->cpu_base->cpu);The timer is queued on
new_basenot on base.
Oops, thanks for spotting this bug. Will be fixed in next version.
PATCH v2 4/5 hrtimer: Kick lowres dynticks targets on timer enqueue
In reply to Frederic WeisbeckerIn lowres mode, hrtimers are serviced by the tick instead of a clock event. It works well as long as the tick stays periodic but we must also make sure that the hrtimers are serviced in dynticks mode targets, pretty much like timer list timers do.
Note that all dynticks modes are concerned: get_nohz_timer_target() tries not to return remote idle CPUs but there is nothing to prevent the elected target from entering dynticks idle mode until we lock its base. It's also prefectly legal to enqueue hrtimers on full dynticks CPU.
So there are two requirements to correctly handle dynticks:
1) On target's tick stop time, we must not delay the next tick further the next hrtimer.
2) On hrtimer queue time. If the tick of the target is stopped, we must wake up that CPU such that it sees the new hrtimer and recalculate the next tick accordingly.
The point 1 is well handled currently through get_nohz_timer_interrupt() and cmp_next_hrtimer_event().
But the point 2 isn't handled at all.
Fixing this is easy though as we have the necessary API ready for that. All we need is to call wake_up_nohz_cpu() on a target when a newly enqueued hrtimer requires tick rescheduling, like timer list timer do.
---
Hi Thomas,
Frederic is on vacations and so sending it directly.
V1->V2: s/base/new_base as pointed out by Thomas.
Pushed here as well:git://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git timers/missing-kick-v2
kernel/hrtimer.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
kernel/hrtimer.c +19 -8
@@ -1013,14 +1013,25 @@ int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim, leftmost =enqueue_hrtimer (timer, new_base); -/* -* Only allow reprogramming if the new base is on this CPU. -* (it might still be on another CPU if the timer was pending) -* -* XXX send_remote_softirq() ? -*/ -if (leftmost && new_base->cpu_base == &__get_cpu_var (hrtimer_bases) - &&hrtimer_enqueue_reprogram (timer, new_base)) { +if (!leftmost) { +unlock_hrtimer_base (timer, &flags); +return ret; + } + +if (!hrtimer_is_hres_active (timer)) { +/* +* Kick to reschedule the next tick to handle the new timer +* on dynticks target. +*/ +wake_up_nohz_cpu (new_base->cpu_base->cpu); + }else if (new_base->cpu_base == &__get_cpu_var (hrtimer_bases) && +hrtimer_enqueue_reprogram (timer, new_base)) { +/* +* Only allow reprogramming if the new base is on this CPU. +* (it might still be on another CPU if the timer was pending) +* +* XXX send_remote_softirq() ? +*/ if (wakeup) {/* * We need to drop cpu_base->lock to avoid a
Signature · 1 line
2.0.0.rc2
tip-bot for Viresh Kumar
merge bot[tip:timers/core] hrtimer: Kick lowres dynticks targets on timer enqueue
hrtimer: Kick lowres dynticks targets on timer enqueue
Commit-ID: 49a2a07514a3a2ea4a02482fa60575e106d960f9 Gitweb: http://git.kernel.org/tip/49a2a07514a3a2ea4a02482fa60575e106d960f9 Author: Viresh Kumar <[email protected]> AuthorDate: Mon, 23 Jun 2014 13:39:37 +0530 Committer: Thomas Gleixner <[email protected]> CommitDate: Mon, 23 Jun 2014 11:23:47 +0200
hrtimer: Kick lowres dynticks targets on timer enqueue
In lowres mode, hrtimers are serviced by the tick instead of a clock event. It works well as long as the tick stays periodic but we must also make sure that the hrtimers are serviced in dynticks mode targets, pretty much like timer list timers do.
Note that all dynticks modes are concerned: get_nohz_timer_target() tries not to return remote idle CPUs but there is nothing to prevent the elected target from entering dynticks idle mode until we lock its base. It's also prefectly legal to enqueue hrtimers on full dynticks CPU.
So there are two requirements to correctly handle dynticks:
1) On target's tick stop time, we must not delay the next tick further the next hrtimer.
2) On hrtimer queue time. If the tick of the target is stopped, we must wake up that CPU such that it sees the new hrtimer and recalculate the next tick accordingly.
The point 1 is well handled currently through get_nohz_timer_interrupt() and cmp_next_hrtimer_event().
But the point 2 isn't handled at all.
Fixing this is easy though as we have the necessary API ready for that. All we need is to call wake_up_nohz_cpu() on a target when a newly enqueued hrtimer requires tick rescheduling, like timer list timer do.
--- kernel/time/hrtimer.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
kernel/time/hrtimer.c +19 -8
@@ -1013,14 +1013,25 @@ int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim, leftmost =enqueue_hrtimer (timer, new_base); -/* -* Only allow reprogramming if the new base is on this CPU. -* (it might still be on another CPU if the timer was pending) -* -* XXX send_remote_softirq() ? -*/ -if (leftmost && new_base->cpu_base == &__get_cpu_var (hrtimer_bases) - &&hrtimer_enqueue_reprogram (timer, new_base)) { +if (!leftmost) { +unlock_hrtimer_base (timer, &flags); +return ret; + } + +if (!hrtimer_is_hres_active (timer)) { +/* +* Kick to reschedule the next tick to handle the new timer +* on dynticks target. +*/ +wake_up_nohz_cpu (new_base->cpu_base->cpu); + }else if (new_base->cpu_base == &__get_cpu_var (hrtimer_bases) && +hrtimer_enqueue_reprogram (timer, new_base)) { +/* +* Only allow reprogramming if the new base is on this CPU. +* (it might still be on another CPU if the timer was pending) +* +* XXX send_remote_softirq() ? +*/ if (wakeup) {/* * We need to drop cpu_base->lock to avoid a
PATCH 5/5 hrtimer: Remove hrtimer_enqueue_reprogram()
In reply to Frederic WeisbeckerFrom: Viresh Kumar <[email protected]>
We call hrtimer_enqueue_reprogram() only when we are in high resolution mode now so we don't need to check that again in hrtimer_enqueue_reprogram().
Once the check is removed, hrtimer_enqueue_reprogram() turns to be an useless wrapper over hrtimer_reprogram() and can be dropped.
--- kernel/hrtimer.c | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-)
kernel/hrtimer.c +8 -15
@@ -602,6 +602,11 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)* timers, we have to check, whether it expires earlier than the timer for * which the clock event device was armed. * +* Note, that in case the state has HRTIMER_STATE_CALLBACK set, no reprogramming +* and no expiry check happens. The timer gets enqueued into the rbtree. The +* reprogramming and expiry check is done in the hrtimer_interrupt or in the +* softirq. +* * Called with interrupts disabled and base->cpu_base.lock held */ static int hrtimer_reprogram(struct hrtimer *timer,
@@ -662,18 +667,6 @@ static inline void hrtimer_init_hres(struct hrtimer_cpu_base *base) base->hres_active =0 ; } -/* -* When High resolution timers are active, try to reprogram. Note, that in case -* the state has HRTIMER_STATE_CALLBACK set, no reprogramming and no expiry -* check happens. The timer gets enqueued into the rbtree. The reprogramming -* and expiry check is done in the hrtimer_interrupt or in the softirq. -*/ -static inline int hrtimer_enqueue_reprogram (struct hrtimer *timer, -struct hrtimer_clock_base *base) -{ -return base->cpu_base->hres_active &&hrtimer_reprogram (timer, base); -} -static inline ktime_t hrtimer_update_base (struct hrtimer_cpu_base *base) {ktime_t *offs_real = &base->clock_base[HRTIMER_BASE_REALTIME ].offset;
@@ -755,8 +748,8 @@ static inline int hrtimer_is_hres_enabled(void) { return 0; }static inline int hrtimer_switch_to_hres (void ) {return 0 ; }static inline void hrtimer_force_reprogram (struct hrtimer_cpu_base *base,int skip_equal) { } -static inline int hrtimer_enqueue_reprogram (struct hrtimer *timer, -struct hrtimer_clock_base *base) +static inline int hrtimer_reprogram (struct hrtimer *timer, +struct hrtimer_clock_base *base) {return 0 ; }
@@ -1025,7 +1018,7 @@ int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,*/ wake_up_nohz_cpu (base->cpu_base->cpu); }else if (new_base->cpu_base == &__get_cpu_var (hrtimer_bases) && -hrtimer_enqueue_reprogram (timer, new_base)) { +hrtimer_reprogram (timer, new_base)) {/* * Only allow reprogramming if the new base is on this CPU. * (it might still be on another CPU if the timer was pending)
Signature · 1 line
1.8.3.1
tip-bot for Viresh Kumar
merge bot[tip:timers/core] hrtimer: Remove hrtimer_enqueue_reprogram()
hrtimer: Remove hrtimer_enqueue_reprogram()
Commit-ID: 9e1e01dd79ac4cf936623399abe57dfba4528ae6 Gitweb: http://git.kernel.org/tip/9e1e01dd79ac4cf936623399abe57dfba4528ae6 Author: Viresh Kumar <[email protected]> AuthorDate: Sun, 22 Jun 2014 01:29:17 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Mon, 23 Jun 2014 11:23:47 +0200
hrtimer: Remove hrtimer_enqueue_reprogram()
We call hrtimer_enqueue_reprogram() only when we are in high resolution mode now so we don't need to check that again in hrtimer_enqueue_reprogram().
Once the check is removed, hrtimer_enqueue_reprogram() turns to be an useless wrapper over hrtimer_reprogram() and can be dropped.
--- kernel/time/hrtimer.c | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-)
kernel/time/hrtimer.c +8 -15
@@ -602,6 +602,11 @@ hrtimer_force_reprogram(struct hrtimer_cpu_base *cpu_base, int skip_equal)* timers, we have to check, whether it expires earlier than the timer for * which the clock event device was armed. * +* Note, that in case the state has HRTIMER_STATE_CALLBACK set, no reprogramming +* and no expiry check happens. The timer gets enqueued into the rbtree. The +* reprogramming and expiry check is done in the hrtimer_interrupt or in the +* softirq. +* * Called with interrupts disabled and base->cpu_base.lock held */ static int hrtimer_reprogram(struct hrtimer *timer,
@@ -662,18 +667,6 @@ static inline void hrtimer_init_hres(struct hrtimer_cpu_base *base) base->hres_active =0 ; } -/* -* When High resolution timers are active, try to reprogram. Note, that in case -* the state has HRTIMER_STATE_CALLBACK set, no reprogramming and no expiry -* check happens. The timer gets enqueued into the rbtree. The reprogramming -* and expiry check is done in the hrtimer_interrupt or in the softirq. -*/ -static inline int hrtimer_enqueue_reprogram (struct hrtimer *timer, -struct hrtimer_clock_base *base) -{ -return base->cpu_base->hres_active &&hrtimer_reprogram (timer, base); -} -static inline ktime_t hrtimer_update_base (struct hrtimer_cpu_base *base) {ktime_t *offs_real = &base->clock_base[HRTIMER_BASE_REALTIME ].offset;
@@ -755,8 +748,8 @@ static inline int hrtimer_is_hres_enabled(void) { return 0; }static inline int hrtimer_switch_to_hres (void ) {return 0 ; }static inline void hrtimer_force_reprogram (struct hrtimer_cpu_base *base,int skip_equal) { } -static inline int hrtimer_enqueue_reprogram (struct hrtimer *timer, -struct hrtimer_clock_base *base) +static inline int hrtimer_reprogram (struct hrtimer *timer, +struct hrtimer_clock_base *base) {return 0 ; }
@@ -1025,7 +1018,7 @@ int __hrtimer_start_range_ns(struct hrtimer *timer, ktime_t tim,*/ wake_up_nohz_cpu (new_base->cpu_base->cpu); }else if (new_base->cpu_base == &__get_cpu_var (hrtimer_bases) && -hrtimer_enqueue_reprogram (timer, new_base)) { +hrtimer_reprogram (timer, new_base)) {/* * Only allow reprogramming if the new base is on this CPU. * (it might still be on another CPU if the timer was pending)