timer: Reduce unnecessary sighand lock contention
Subsystems (2)
PATCH 3/3 timer: Reduce unnecessary sighand lock contention
And some more comments on the series...
Review struct task_cputime_atomic { · line 632
Quoted 6 lines
(quoted diff) +1
-0@@ -626,6 +628,7 @@ struct task_cputime_atomic { struct thread_group_cputimer { struct task_cputime_atomic cputime_atomic; int running; + int checking_timer; };
Why not make both running and checking_timer actual booleans, so the pair is only 16 bits rather than 64?
I suppose since sum_exec_runtime in struct task_cputime_atomic is 64 bits, alignment means there's no actual improvement. It's just my personal preference (Linus disagrees with me!) for the bool type for documentation.
(Compile-tested patch appended.)
But when it comes to really understanding this code, I'm struggling. It seems that this changes the return value of of fastpath_timer_check. I'm tryng to wrap my head around exactly why that is safe.
Any time I see things like READ_ONCE and WRITE_ONCE, I wonder if there need to be memory barriers as well. (Because x86 has strong default memory ordering, testing there doesn't prove the code right on other platforms.)
For the writes, the surrounding spinlock does the job.
But on the read side (fastpath_timer_check), I'm not sure what the rules should be.
Or is it basically okay if this is massively racey, since process-wide CPU timers are inherently sloppy. A race will just cause an expiration check to be missed, but it will be retried soon anyway.
Other half-baked thoughts that went through my head:
If the problem is that you have contention for read access to sig->cputimer.cputime, can that be changed to a seqlock?
Another possibility is to use a trylock before testing checking_timer:
if (sig->cputimer.running) {
struct task_cputime group_sample;- raw_spin_lock(&sig->cputimer.lock);
+ if (!raw_spin_trylock(&sig->cputimer.lock)) {
+ if (READ_ONCE(sig->cputimer.checking_timer))
+ return false; /* Lock holder's doing it for us */
+ raw_spin_lock(&sig->cputimer.lock);
+ }
group_sample = sig->cputimer.cputime; raw_spin_unlock(&sig->cputimer.lock); if (task_cputime_expired(&group_sample, &sig->cputime_expires)) return true; } return false; }
----8<---- This is the patch mentioned above ----8<----
Quoted 1 line
From
0058bf5ed6a9e483ae33746685332e3ef9562ed5Mon Sep 17 00:00:00 2001
From: George Spelvin <[email protected]>
Date: Wed, 26 Aug 2015 19:15:54 +0000
Subject: [PATCH] timer: Use bool more in kernel/time/posix-cpu-timers.c
This is mostly function return values, for documentation.
One structure field is changed (from int), but alignment padding precludes any actual space saving.
include/linux/sched.h +2 -2
@@ -585,7 +585,7 @@ struct task_cputime {/** * struct thread_group_cputimer - thread group interval timer counts * @cputime: thread group interval timers. -* @running: non-zero when there are timers running and +* @running: %true when there are timers running and * @cputime receives updates. * @lock: lock for fields in this struct. *
@@ -594,7 +594,7 @@ struct task_cputime {*/ struct thread_group_cputimer {struct task_cputime cputime; -int running; +bool running;raw_spinlock_t lock; };
kernel/time/posix-cpu-timers.c +21 -22
105 lines not on this page · raw message
Quoted 12 lines · George Spelvin
And some more comments on the series...
Review struct task_cputime_atomic { · line 632
Quoted 6 lines
(quoted diff) +1
-0@@ -626,6 +628,7 @@ struct task_cputime_atomic { struct thread_group_cputimer { struct task_cputime_atomic cputime_atomic; int running; + int checking_timer; };Why not make both running and
checking_timeractual booleans, so the pair is only 16 bits rather than 64?
Quoted 5 lines
I suppose since
sum_exec_runtimein structtask_cputime_atomicis 64 bits, alignment means there's no actual improvement. It's just my personal preference (Linus disagrees with me!) for the bool type for documentation.(Compile-tested patch appended.)
I can include your patch in the series and then use boolean for the new checking_timer field. However, it looks like this applies on an old kernel. For example, the spin_lock field has already been removed from the structure.
Quoted 9 lines
But when it comes to really understanding this code, I'm struggling. It seems that this changes the return value of of
fastpath_timer_check. I'm tryng to wrap my head around exactly why that is safe.Any time I see things like
READ_ONCEandWRITE_ONCE, I wonder if there need to be memory barriers as well. (Because x86 has strong default memory ordering, testing there doesn't prove the code right on other platforms.)
Quoted 8 lines
For the writes, the surrounding spinlock does the job.
But on the read side (
fastpath_timer_check), I'm not sure what the rules should be.Or is it basically okay if this is massively racey, since process-wide CPU timers are inherently sloppy. A race will just cause an expiration check to be missed, but it will be retried soon anyway.
Yes, the worst case scenario is that we wait until the next thread to come along and handle the next expired timer. However, this "delay" already occurs now (for example: a timer can expire right after a thread exits check_process_timers()).
Quoted 11 lines
Other half-baked thoughts that went through my head:
If the problem is that you have contention for read access to
sig->cputimer.cputime, can that be changed to a seqlock?Another possibility is to use a trylock before testing
checking_timer:if (sig->cputimer.running) { struct task_cputime group_sample;
Review
Quoted 6 lines
(quoted diff) +5
-1- raw_spin_lock(&sig->cputimer.lock); + if (!raw_spin_trylock(&sig->cputimer.lock)) { + if (READ_ONCE(sig->cputimer.checking_timer)) + return false; /* Lock holder's doing it for us */ + raw_spin_lock(&sig->cputimer.lock); + }
The spinlock call has already been removed from a previous patch. The issue now is with contention with the sighand lock.
Thanks,
Jason
Quoted 4 lines
I can include your patch in the series and then use boolean for the new
checking_timerfield. However, it looks like this applies on an old kernel. For example, thespin_lockfield has already been removed from the structure.
Apologies; that was 4.1.6. A 4.2-rc8 patch is appended (it's a pretty trivial merge once you look at it).
Quoted 2 lines
The spinlock call has already been removed from a previous patch. The issue now is with contention with the sighand lock.
I'll look some more and try to wrap my head around it.
Quoted 8 lines
Quoted 3 lines
Or is it basically okay if this is massively racey, since process-wide CPU timers are inherently sloppy. A race will just cause an expiration check to be missed, but it will be retried soon anyway.
Yes, the worst case scenario is that we wait until the next thread to come along and handle the next expired timer. However, this "delay" already occurs now (for example: a timer can expire right after a thread exits
check_process_timers()).
Ad is this polled, or is there some non-polled system that will trigger another call to check_process_timers().
E.g. suppose a process fails to notice that it blew past a CPU time timeout before blocking. Does anything guarantee that it will get the timeout signal in finite real time?
Quoted 1 line
From
95349f9b16c30aea518ce79d72e8e0f0c5d12069Mon Sep 17 00:00:00 2001
From: George Spelvin <[email protected]>
Date: Wed, 26 Aug 2015 19:15:54 +0000
Subject: [PATCH] timer: Use bool more in kernel/time/posix-cpu-timers.c
This is mostly function return values, for documentation.
One structure field is changed (from int), but alignment padding precludes any actual space saving.
include/linux/sched.h +2 -2
@@ -607,7 +607,7 @@ struct task_cputime_atomic {/** * struct thread_group_cputimer - thread group interval timer counts * @cputime_atomic: atomic thread group interval timers. -* @running: non-zero when there are timers running and +* @running: %true when there are timers running and * @cputime receives updates. * * This structure contains the version of task_cputime, above, that is
@@ -615,7 +615,7 @@ struct task_cputime_atomic {*/ struct thread_group_cputimer {struct task_cputime_atomic cputime_atomic; -int running; +bool running; };#include <linux/rwsem.h>
kernel/time/posix-cpu-timers.c +21 -22
105 lines not on this page · raw message
Quoted 7 lines · George Spelvin
Quoted 4 lines
I can include your patch in the series and then use boolean for the new
checking_timerfield. However, it looks like this applies on an old kernel. For example, thespin_lockfield has already been removed from the structure.Apologies; that was 4.1.6. A 4.2-rc8 patch is appended (it's a pretty trivial merge once you look at it).
Frederic suggested that we just use a single "status" variable and access the bits for the running and checking field. I am leaning towards that method, so I might not include the rest of the boolean changes in this patchset.
Quoted 14 lines
Quoted 2 lines
The spinlock call has already been removed from a previous patch. The issue now is with contention with the sighand lock.
I'll look some more and try to wrap my head around it.
Quoted 8 lines
Quoted 3 lines
Or is it basically okay if this is massively racey, since process-wide CPU timers are inherently sloppy. A race will just cause an expiration check to be missed, but it will be retried soon anyway.
Yes, the worst case scenario is that we wait until the next thread to come along and handle the next expired timer. However, this "delay" already occurs now (for example: a timer can expire right after a thread exits
check_process_timers()).
Quoted 6 lines
Ad is this polled, or is there some non-polled system that will trigger another call to
check_process_timers().E.g. suppose a process fails to notice that it blew past a CPU time timeout before blocking. Does anything guarantee that it will get the timeout signal in finite real time?
Yep, the check_process_timers will get called again during the next scheduler interrupt (approximately after 1 jiffy) and send the signal if it finds that the timer expired then.
Quoted 4 lines · Jason Low
Frederic suggested that we just use a single "status" variable and access the bits for the running and checking field. I am leaning towards that method, so I might not include the rest of the boolean changes in this patchset.
Don't worry, I'm not offended. I just started editing and figured I might as well share it.
Whichever solution is easier. My only complaint about bitmapped variables is that so many are "unsigned long" because the Linux atomic access primitives, originally designed for file system bitmaps, use that type.
But using that for a non array wastes 4 bytes on 64-bit platforms that can't be used if the code is to work on 32-bit ones.
Quoted 7 lines
Quoted 3 lines
E.g. suppose a process fails to notice that it blew past a CPU time timeout before blocking. Does anything guarantee that it will get the timeout signal in finite real time?
Yep, the
check_process_timerswill get called again during the next scheduler interrupt (approximately after 1 jiffy) and send the signal if it finds that the timer expired then.
Will it? I thought it got called on the running process only. Which is not the (blocked by assumption) process of interest.
I don't suspect that this would be a problem in practice, as CPU-time timers are used on activities which use a *lot* of it. But it seemed like a flaw worth either acknowledging or disproving.
Quoted 24 lines · George Spelvin
Quoted 4 lines · Jason Low
Frederic suggested that we just use a single "status" variable and access the bits for the running and checking field. I am leaning towards that method, so I might not include the rest of the boolean changes in this patchset.
Don't worry, I'm not offended. I just started editing and figured I might as well share it.
Whichever solution is easier. My only complaint about bitmapped variables is that so many are "unsigned long" because the Linux atomic access primitives, originally designed for file system bitmaps, use that type.
But using that for a non array wastes 4 bytes on 64-bit platforms that can't be used if the code is to work on 32-bit ones.
Quoted 7 lines
Quoted 3 lines
E.g. suppose a process fails to notice that it blew past a CPU time timeout before blocking. Does anything guarantee that it will get the timeout signal in finite real time?
Yep, the
check_process_timerswill get called again during the next scheduler interrupt (approximately after 1 jiffy) and send the signal if it finds that the timer expired then.
Quoted 6 lines
Will it? I thought it got called on the running process only. Which is not the (blocked by assumption) process of interest.
I don't suspect that this would be a problem in practice, as CPU-time timers are used on activities which use a *lot* of it. But it seemed like a flaw worth either acknowledging or disproving.
You're right, this is only called on running processes. If the process is blocked, sending the signal can get delayed. However, in practice, this is not really a problem as you mentioned. Also, this "issue" is also not really avoidable even without this patch. For instance, a timer may expire and the process can get block before the next scheduler interrupt.
Thanks,
Jason