timers/nohz fixes
Subsystems (2)
Frederic Weisbecker
maintainercover timers/nohz fixes
A bunch of fixes that don't need to be on the urgent queue since they are not regression fixes. But they are still fixes...
Frederic Weisbecker (2):
timers/nohz: Last resort update jiffies on nohz_full IRQ entry sched/cputime: Fix getrusage(RUSAGE_THREAD) with nohz_full
include/linux/sched/cputime.h | 5 +++-- kernel/sched/cputime.c | 12 +++++++++--- kernel/softirq.c | 3 ++- kernel/time/tick-sched.c | 7 +++++++ 4 files changed, 21 insertions(+), 6 deletions(-)
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH RESEND 1/2 timers/nohz: Last resort update jiffies on nohz_full IRQ entry
When at least one CPU runs in nohz_full mode, a dedicated timekeeper CPU is guaranteed to stay online and to never stop its tick.
Meanwhile on some rare case, the dedicated timekeeper may be running with interrupts disabled for a while, such as in stop_machine.
If jiffies stop being updated, a nohz_full CPU may end up endlessly programming the next tick in the past, taking the last jiffies update monotonic timestamp as a stale base, resulting in an tick storm.
Here is a scenario where it matters:
0) CPU 0 is the timekeeper and CPU 1 a nohz_full CPU.
1) A stop machine callback is queued to execute somewhere.
2) CPU 0 reaches MULTI_STOP_DISABLE_IRQ while CPU 1 is still in MULTI_STOP_PREPARE. Hence CPU 0 can't do its timekeeping duty. CPU 1 can still take IRQs.
3) CPU 1 receives an IRQ which queues a timer callback one jiffy forward.
4) On IRQ exit, CPU 1 schedules the tick one jiffy forward, taking last_jiffies_update as a base. But last_jiffies_update hasn't been updated for 2 jiffies since the timekeeper has interrupts disabled.
5) clockevents_program_event(), which relies on ktime_get(), observes that the expiration is in the past and therefore programs the min delta event on the clock.
6) The tick fires immediately, goto 3)
7) Tick storm, the nohz_full CPU is drown and takes ages to reach MULTI_STOP_DISABLE_IRQ, which is the only way out of this situation.
Solve this with unconditionally updating jiffies if the value is stale on nohz_full IRQ entry. IRQs and other disturbances are expected to be rare enough on nohz_full for the unconditional call to ktime_get() to actually matter.
--- kernel/softirq.c | 3 ++- kernel/time/tick-sched.c | 7 +++++++ 2 files changed, 9 insertions(+), 1 deletion(-)
kernel/softirq.c +2 -1
@@ -595,7 +595,8 @@ void irq_enter_rcu(void) {__irq_enter_raw (); -if (is_idle_task (current) && (irq_count () ==HARDIRQ_OFFSET )) +if (tick_nohz_full_cpu (smp_processor_id ()) || + (is_idle_task (current) && (irq_count () ==HARDIRQ_OFFSET )))tick_irq_enter ();account_hardirq_enter (current);
kernel/time/tick-sched.c +7 -0
@@ -1375,6 +1375,13 @@ static inline void tick_nohz_irq_enter(void) now =ktime_get ();if (ts->idle_active)tick_nohz_stop_idle (ts, now); +/* +* If all CPUs are idle. We may need to update a stale jiffies value. +* Note nohz_full is a special case: a timekeeper is guaranteed to stay +* alive but it might be busy looping with interrupts disabled in some +* rare case (typically stop machine). So we must make sure we have a +* last resort. +*/ if (ts->tick_stopped)tick_nohz_update_jiffies (now); }
Signature · 1 line
2.25.1
[tip: timers/urgent] timers/nohz: Last resort update jiffies on nohz_full IRQ entry
The following commit has been merged into the timers/urgent branch of tip:
The following commit has been merged into the timers/urgent branch of tip:
Commit-ID: 53e87e3cdc155f20c3417b689df8d2ac88d79576 Gitweb: https://git.kernel.org/tip/53e87e3cdc155f20c3417b689df8d2ac88d79576 Author: Frederic Weisbecker <[email protected]> AuthorDate: Tue, 26 Oct 2021 16:10:54 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Thu, 02 Dec 2021 15:07:22 +01:00
timers/nohz: Last resort update jiffies on nohz_full IRQ entry
When at least one CPU runs in nohz_full mode, a dedicated timekeeper CPU is guaranteed to stay online and to never stop its tick.
Meanwhile on some rare case, the dedicated timekeeper may be running with interrupts disabled for a while, such as in stop_machine.
If jiffies stop being updated, a nohz_full CPU may end up endlessly programming the next tick in the past, taking the last jiffies update monotonic timestamp as a stale base, resulting in an tick storm.
Here is a scenario where it matters:
0) CPU 0 is the timekeeper and CPU 1 a nohz_full CPU.
1) A stop machine callback is queued to execute somewhere.
2) CPU 0 reaches MULTI_STOP_DISABLE_IRQ while CPU 1 is still in MULTI_STOP_PREPARE. Hence CPU 0 can't do its timekeeping duty. CPU 1 can still take IRQs.
3) CPU 1 receives an IRQ which queues a timer callback one jiffy forward.
4) On IRQ exit, CPU 1 schedules the tick one jiffy forward, taking last_jiffies_update as a base. But last_jiffies_update hasn't been updated for 2 jiffies since the timekeeper has interrupts disabled.
5) clockevents_program_event(), which relies on ktime_get(), observes that the expiration is in the past and therefore programs the min delta event on the clock.
6) The tick fires immediately, goto 3)
7) Tick storm, the nohz_full CPU is drown and takes ages to reach MULTI_STOP_DISABLE_IRQ, which is the only way out of this situation.
Solve this with unconditionally updating jiffies if the value is stale on nohz_full IRQ entry. IRQs and other disturbances are expected to be rare enough on nohz_full for the unconditional call to ktime_get() to actually matter.
--- kernel/softirq.c | 3 ++- kernel/time/tick-sched.c | 7 +++++++ 2 files changed, 9 insertions(+), 1 deletion(-)
kernel/softirq.c +2 -1
@@ -595,7 +595,8 @@ void irq_enter_rcu(void) {__irq_enter_raw (); -if (is_idle_task (current) && (irq_count () ==HARDIRQ_OFFSET )) +if (tick_nohz_full_cpu (smp_processor_id ()) || + (is_idle_task (current) && (irq_count () ==HARDIRQ_OFFSET )))tick_irq_enter ();account_hardirq_enter (current);
kernel/time/tick-sched.c +7 -0
@@ -1375,6 +1375,13 @@ static inline void tick_nohz_irq_enter(void) now =ktime_get ();if (ts->idle_active)tick_nohz_stop_idle (ts, now); +/* +* If all CPUs are idle. We may need to update a stale jiffies value. +* Note nohz_full is a special case: a timekeeper is guaranteed to stay +* alive but it might be busy looping with interrupts disabled in some +* rare case (typically stop machine). So we must make sure we have a +* last resort. +*/ if (ts->tick_stopped)tick_nohz_update_jiffies (now); }
Frederic Weisbecker
maintainerPATCH 2/2 sched/cputime: Fix getrusage(RUSAGE_THREAD) with nohz_full
In reply to Frederic Weisbeckergetrusage(RUSAGE_THREAD) with nohz_full may return shorter utime/stime than the actual time.
task_cputime_adjusted() snapshots utime and stime and then adjust their sum to match the scheduler maintained cputime.sum_exec_runtime. Unfortunately in nohz_full, sum_exec_runtime is only updated once per second in the worst case, causing a discrepancy against utime and stime that can be updated anytime by the reader using vtime.
To fix this situation, perform an update of cputime.sum_exec_runtime when the cputime snapshot reports the task as actually running while the tick is disabled. The related overhead is then contained within the relevant situations.
--- include/linux/sched/cputime.h | 5 +++-- kernel/sched/cputime.c | 12 +++++++++--- 2 files changed, 12 insertions(+), 5 deletions(-)
include/linux/sched/cputime.h +3 -2
@@ -18,15 +18,16 @@#endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN -extern void task_cputime (struct task_struct *t, +extern bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime);extern u64 task_gtime (struct task_struct *t);#else -static inline void task_cputime (struct task_struct *t, +static inline bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime) { *utime = t->utime; *stime = t->stime; +return false; }static inline u64 task_gtime (struct task_struct *t)
kernel/sched/cputime.c +9 -3
@@ -615,7 +615,8 @@ void task_cputime_adjusted(struct task_struct *p, u64 *ut, u64 *st) .sum_exec_runtime = p->se.sum_exec_runtime, }; -task_cputime (p, &cputime.utime, &cputime.stime); +if (task_cputime (p, &cputime.utime, &cputime.stime)) + cputime.sum_exec_runtime =task_sched_runtime (p);cputime_adjust (&cputime, &p->prev_cputime, ut, st); }EXPORT_SYMBOL_GPL (task_cputime_adjusted);
@@ -828,19 +829,21 @@ u64 task_gtime(struct task_struct *t)* add up the pending nohz execution time since the last * cputime snapshot. */ -void task_cputime (struct task_struct *t,u64 *utime,u64 *stime) +bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime) {struct vtime *vtime = &t->vtime;unsigned int seq;u64 delta; +int ret;if (!vtime_accounting_enabled ()) { *utime = t->utime; *stime = t->stime; -return ; +return false; } do { + ret = false; seq = read_seqcount_begin(&vtime->seqcount); *utime = t->utime;
@@ -850,6 +853,7 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime)if (vtime->state <VTIME_SYS )continue ; + ret = true; delta =vtime_delta (vtime);/*
@@ -861,6 +865,8 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime)else *utime += vtime->utime + delta; }while (read_seqcount_retry (&vtime->seqcount, seq)); + +return ret; }static int vtime_state_fetch (struct vtime *vtime,int cpu)
Signature · 1 line
2.25.1
Re: [PATCH 2/2] sched/cputime: Fix getrusage(RUSAGE_THREAD) with nohz_full
Quoted 14 lines · Frederic Weisbecker
getrusage(RUSAGE_THREAD)withnohz_fullmay return shorter utime/stime than the actual time.
task_cputime_adjusted()snapshots utime and stime and then adjust their sum to match the scheduler maintained cputime.sum_exec_runtime. Unfortunately innohz_full,sum_exec_runtimeis only updated once per second in the worst case, causing a discrepancy against utime and stime that can be updated anytime by the reader using vtime.To fix this situation, perform an update of cputime.sum_exec_runtime when the cputime snapshot reports the task as actually running while the tick is disabled. The related overhead is then contained within the relevant situations.
Quoted 5 lines
Reported-by Hasegawa Hitomi <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Mel Gorman <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by Hasegawa Hitomi <[email protected]>
Thank you for this patch. getrusage(RUSAGE_THREAD) with nohz_full works well! Please feel free to add:
Thanks,
Masa
Quoted 86 lines
--- include/linux/sched/cputime.h | 5 +++-- kernel/sched/cputime.c | 12 +++++++++--- 2 files changed, 12 insertions(+), 5 deletions(-)include/linux/sched/cputime.h +3
-2@@ -18,15 +18,16 @@#endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN -extern void task_cputime (struct task_struct *t, +extern bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime);extern u64 task_gtime (struct task_struct *t);#else -static inline void task_cputime (struct task_struct *t, +static inline bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime) { *utime = t->utime; *stime = t->stime; +return false; }static inline u64 task_gtime (struct task_struct *t)kernel/sched/cputime.c +9
-3@@ -615,7 +615,8 @@ void task_cputime_adjusted(struct task_struct *p, u64 *ut, u64 *st) .sum_exec_runtime = p->se.sum_exec_runtime, }; -task_cputime (p, &cputime.utime, &cputime.stime); +if (task_cputime (p, &cputime.utime, &cputime.stime)) + cputime.sum_exec_runtime =task_sched_runtime (p);cputime_adjust (&cputime, &p->prev_cputime, ut, st); }EXPORT_SYMBOL_GPL (task_cputime_adjusted);@@ -828,19 +829,21 @@ u64 task_gtime(struct task_struct *t)* add up the pending nohz execution time since the last * cputime snapshot. */ -void task_cputime (struct task_struct *t,u64 *utime,u64 *stime) +bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime) {struct vtime *vtime = &t->vtime;unsigned int seq;u64 delta; +int ret;if (!vtime_accounting_enabled ()) { *utime = t->utime; *stime = t->stime; -return ; +return false; } do { + ret = false; seq = read_seqcount_begin(&vtime->seqcount); *utime = t->utime;@@ -850,6 +853,7 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime)if (vtime->state <VTIME_SYS )continue ; + ret = true; delta =vtime_delta (vtime);/* @@ -861,6 +865,8 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime)else *utime += vtime->utime + delta; }while (read_seqcount_retry (&vtime->seqcount, seq)); + +return ret; }static int vtime_state_fetch (struct vtime *vtime,int cpu)Signature · 1 line
2.25.1
Re: [PATCH 2/2] sched/cputime: Fix getrusage(RUSAGE_THREAD) with nohz_full
In reply to Frederic WeisbeckerHi,
Quoted 105 lines · On Tue, Oct 26, 2021 at 04:10:55PM +0200 Frederic Weisbecker wrote:
getrusage(RUSAGE_THREAD)withnohz_fullmay return shorter utime/stime than the actual time.
task_cputime_adjusted()snapshots utime and stime and then adjust their sum to match the scheduler maintained cputime.sum_exec_runtime. Unfortunately innohz_full,sum_exec_runtimeis only updated once per second in the worst case, causing a discrepancy against utime and stime that can be updated anytime by the reader using vtime.To fix this situation, perform an update of cputime.sum_exec_runtime when the cputime snapshot reports the task as actually running while the tick is disabled. The related overhead is then contained within the relevant situations.
Reported-by Hasegawa Hitomi <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Mel Gorman <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by Hasegawa Hitomi <[email protected]>--- include/linux/sched/cputime.h | 5 +++-- kernel/sched/cputime.c | 12 +++++++++--- 2 files changed, 12 insertions(+), 5 deletions(-)include/linux/sched/cputime.h +3
-2@@ -18,15 +18,16 @@#endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN -extern void task_cputime (struct task_struct *t, +extern bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime);extern u64 task_gtime (struct task_struct *t);#else -static inline void task_cputime (struct task_struct *t, +static inline bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime) { *utime = t->utime; *stime = t->stime; +return false; }static inline u64 task_gtime (struct task_struct *t)kernel/sched/cputime.c +9
-3@@ -615,7 +615,8 @@ void task_cputime_adjusted(struct task_struct *p, u64 *ut, u64 *st) .sum_exec_runtime = p->se.sum_exec_runtime, }; -task_cputime (p, &cputime.utime, &cputime.stime); +if (task_cputime (p, &cputime.utime, &cputime.stime)) + cputime.sum_exec_runtime =task_sched_runtime (p);cputime_adjust (&cputime, &p->prev_cputime, ut, st); }EXPORT_SYMBOL_GPL (task_cputime_adjusted);@@ -828,19 +829,21 @@ u64 task_gtime(struct task_struct *t)* add up the pending nohz execution time since the last * cputime snapshot. */ -void task_cputime (struct task_struct *t,u64 *utime,u64 *stime) +bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime) {struct vtime *vtime = &t->vtime;unsigned int seq;u64 delta; +int ret;if (!vtime_accounting_enabled ()) { *utime = t->utime; *stime = t->stime; -return ; +return false; } do { + ret = false; seq = read_seqcount_begin(&vtime->seqcount); *utime = t->utime;@@ -850,6 +853,7 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime)if (vtime->state <VTIME_SYS )continue ; + ret = true; delta =vtime_delta (vtime);/* @@ -861,6 +865,8 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime)else *utime += vtime->utime + delta; }while (read_seqcount_retry (&vtime->seqcount, seq)); + +return ret; }static int vtime_state_fetch (struct vtime *vtime,int cpu)Signature · 1 line
2.25.1
Could someone please pick this (or, rather, these) up?
Thanks!
Phil
[tip: sched/urgent] sched/cputime: Fix getrusage(RUSAGE_THREAD) with nohz_full
In reply to Frederic WeisbeckerThe following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: e7f2be115f0746b969c0df14c0d182f65f005ca5 Gitweb: https://git.kernel.org/tip/e7f2be115f0746b969c0df14c0d182f65f005ca5 Author: Frederic Weisbecker <[email protected]> AuthorDate: Tue, 26 Oct 2021 16:10:55 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Thu, 02 Dec 2021 15:08:22 +01:00
sched/cputime: Fix getrusage(RUSAGE_THREAD) with nohz_full
getrusage(RUSAGE_THREAD) with nohz_full may return shorter utime/stime than the actual time.
task_cputime_adjusted() snapshots utime and stime and then adjust their sum to match the scheduler maintained cputime.sum_exec_runtime. Unfortunately in nohz_full, sum_exec_runtime is only updated once per second in the worst case, causing a discrepancy against utime and stime that can be updated anytime by the reader using vtime.
To fix this situation, perform an update of cputime.sum_exec_runtime when the cputime snapshot reports the task as actually running while the tick is disabled. The related overhead is then contained within the relevant situations.
--- include/linux/sched/cputime.h | 5 +++-- kernel/sched/cputime.c | 12 +++++++++--- 2 files changed, 12 insertions(+), 5 deletions(-)
include/linux/sched/cputime.h +3 -2
@@ -18,15 +18,16 @@#endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN -extern void task_cputime (struct task_struct *t, +extern bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime);extern u64 task_gtime (struct task_struct *t);#else -static inline void task_cputime (struct task_struct *t, +static inline bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime) { *utime = t->utime; *stime = t->stime; +return false; }static inline u64 task_gtime (struct task_struct *t)
kernel/sched/cputime.c +9 -3
@@ -615,7 +615,8 @@ void task_cputime_adjusted(struct task_struct *p, u64 *ut, u64 *st) .sum_exec_runtime = p->se.sum_exec_runtime, }; -task_cputime (p, &cputime.utime, &cputime.stime); +if (task_cputime (p, &cputime.utime, &cputime.stime)) + cputime.sum_exec_runtime =task_sched_runtime (p);cputime_adjust (&cputime, &p->prev_cputime, ut, st); }EXPORT_SYMBOL_GPL (task_cputime_adjusted);
@@ -828,19 +829,21 @@ u64 task_gtime(struct task_struct *t)* add up the pending nohz execution time since the last * cputime snapshot. */ -void task_cputime (struct task_struct *t,u64 *utime,u64 *stime) +bool task_cputime (struct task_struct *t,u64 *utime,u64 *stime) {struct vtime *vtime = &t->vtime;unsigned int seq;u64 delta; +int ret;if (!vtime_accounting_enabled ()) { *utime = t->utime; *stime = t->stime; -return ; +return false; } do { + ret = false; seq = read_seqcount_begin(&vtime->seqcount); *utime = t->utime;
@@ -850,6 +853,7 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime)if (vtime->state <VTIME_SYS )continue ; + ret = true; delta =vtime_delta (vtime);/*
@@ -861,6 +865,8 @@ void task_cputime(struct task_struct *t, u64 *utime, u64 *stime)else *utime += vtime->utime + delta; }while (read_seqcount_retry (&vtime->seqcount, seq)); + +return ret; }static int vtime_state_fetch (struct vtime *vtime,int cpu)