context_tracking: remove local_irq_save from __acct_update_integrals
PATCH 1/1 context_tracking: remove local_irq_save from __acct_update_integrals
The function __acct_update_integrals() is called both from irq context and task context. This creates a race where irq context can advance tsk->acct_timexpd to a value larger than time, leading to a negative value, which causes a divide error. See commit 6d5b5acca9e5 ("Fix fixpoint divide exception in acct_update_integrals")
In 2012, __acct_update_integrals() was changed to get utime and stime as function parameters. This re-introduced the bug, because an irq can hit in-between the call to task_cputime() and where irqs actually get disabled.
However, this race condition was originally reproduced on Hercules, and I have not seen any reports of it re-occurring since it was re-introduced 3 years ago.
On the other hand, the irq disabling and re-enabling, which no longer even protects us against the race today, show up prominently in the perf profile of a program that makes a very large number of system calls in a short period of time, when nohz_full= (and context tracking) is enabled.
This patch replaces the (now ineffective) irq blocking with a cheaper way to test for the race condition, and speeds up my microbenchmark with 10 million iterations:
run time system time
vanilla 5.49s 2.08s
patch 5.21s 1.92s
The above shows a reduction in system time of about 7%. The standard deviation is mostly in the third digit after the decimal point.
--- kernel/tsacct.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
kernel/tsacct.c +11 -5
@@ -126,23 +126,29 @@ static void __acct_update_integrals(struct task_struct *tsk,if (likely (tsk->mm)) {cputime_t time, dtime;struct timeval value; -unsigned long flags;u64 delta; -local_irq_save (flags); time = stime + utime; dtime = time - tsk->acct_timexpd; +/* +* This code is called both from irq context and from +* task context. There is a race where irq context advances +* tsk->acct_timexpd to a value larger than time, creating +* a negative value. In that case, the irq has already +* updated the statistics. +*/ +if (unlikely ((signed long )dtime <=0 )) +return ; +jiffies_to_timeval (cputime_to_jiffies (dtime), &value); delta = value.tv_sec; delta = delta *USEC_PER_SEC + value.tv_usec;if (delta ==0 ) - goto out; +return ; tsk->acct_timexpd = time; tsk->acct_rss_mem1 += delta *get_mm_rss (tsk->mm); tsk->acct_vm_mem1 += delta * tsk->mm->total_vm; - out: -local_irq_restore (flags); } }
Quoted 13 lines · Rik van Riel
kernel/tsacct.c +0
-2@@ -126,23 +126,29 @@ static void __acct_update_integrals(struct task_struct *tsk,if (likely (tsk->mm)) {cputime_t time, dtime;struct timeval value; -unsigned long flags;u64 delta; -local_irq_save (flags); time = stime + utime;
kernel/tsacct.c static void __acct_update_integrals(struct task_struct *tsk, · line 141
Quoted 10 lines
kernel/tsacct.c +9
-0dtime = time - tsk->acct_timexpd; +/* +* This code is called both from irq context and from +* task context. There is a race where irq context advances +* tsk->acct_timexpd to a value larger than time, creating +* a negative value. In that case, the irq has already +* updated the statistics. +*/ +if (unlikely ((signed long )dtime <=0 )) +return ;
FWIW, I think you either need a barrier() before the if-statement or use READ_ONCE() when reading tsk->acct_timexpd above.
Otherwise the compiler could (in theory at least) generate code which would translate to
if (unlikely(time <= tsk->acct_timexpd))
in order to achieve the same result, no?
Besides that cputime_t might be 64 bit in size, therefore you don't have much of a guarentee that reading tsk->acct_timexpd happens atomically on 32 bit architectures, so you _may_ end up with garbage, no?
Quoted 28 lines · Heiko Carstens
Quoted 13 lines · Rik van Riel
kernel/tsacct.c +0
-2@@ -126,23 +126,29 @@ static void __acct_update_integrals(struct task_struct *tsk,if (likely (tsk->mm)) {cputime_t time, dtime;struct timeval value; -unsigned long flags;u64 delta; -local_irq_save (flags); time = stime + utime;kernel/tsacct.c static void __acct_update_integrals(struct task_struct *tsk, · line 141
Quoted 10 lines
kernel/tsacct.c +9
-0dtime = time - tsk->acct_timexpd; +/* +* This code is called both from irq context and from +* task context. There is a race where irq context advances +* tsk->acct_timexpd to a value larger than time, creating +* a negative value. In that case, the irq has already +* updated the statistics. +*/ +if (unlikely ((signed long )dtime <=0 )) +return ;FWIW, I think you either need a
barrier()before the if-statement or useREAD_ONCE()when readingtsk->acct_timexpdabove.
Quoted 8 lines
Otherwise the compiler could (in theory at least) generate code which would translate to
if (unlikely(time <= tsk->acct_timexpd))in order to achieve the same result, no?
Besides that
cputime_tmight be 64 bit in size, therefore you don't have much of a guarentee that readingtsk->acct_timexpdhappens atomically on 32 bit architectures, so you _may_ end up with garbage, no?
You are right on both counts. Thank you for pointing out what should have been obvious...
Let me post a new patch :)
Signature · 1 line
All rights reversed.