Skip to contentSkip to conversation

timer: Reduce unnecessary sighand lock contention

PATCH In review George Spelvin 2 participants 6 messages Aug 26, 2015 – Aug 28, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 3/3 timer: Reduce unnecessary sighand lock contention

And some more comments on the series...

Review struct task_cputime_atomic { · line 632

Quoted 6 lines
(quoted diff) +1 -0
@@ -626,6 +628,7 @@ struct task_cputime_atomic {
 struct thread_group_cputimer {
 	struct task_cputime_atomic cputime_atomic;
 	int running;
+	int checking_timer;
 };

Why not make both running and checking_timer actual booleans, so the pair is only 16 bits rather than 64?

I suppose since sum_exec_runtime in struct task_cputime_atomic is 64 bits, alignment means there's no actual improvement. It's just my personal preference (Linus disagrees with me!) for the bool type for documentation.

(Compile-tested patch appended.)

But when it comes to really understanding this code, I'm struggling. It seems that this changes the return value of of fastpath_timer_check. I'm tryng to wrap my head around exactly why that is safe.

Any time I see things like READ_ONCE and WRITE_ONCE, I wonder if there need to be memory barriers as well. (Because x86 has strong default memory ordering, testing there doesn't prove the code right on other platforms.)

For the writes, the surrounding spinlock does the job.

But on the read side (fastpath_timer_check), I'm not sure what the rules should be.

Or is it basically okay if this is massively racey, since process-wide CPU timers are inherently sloppy. A race will just cause an expiration check to be missed, but it will be retried soon anyway.

Other half-baked thoughts that went through my head:

If the problem is that you have contention for read access to sig->cputimer.cputime, can that be changed to a seqlock?

Another possibility is to use a trylock before testing checking_timer:

	if (sig->cputimer.running) {
		struct task_cputime group_sample;

- raw_spin_lock(&sig->cputimer.lock);
+ if (!raw_spin_trylock(&sig->cputimer.lock)) {
+ if (READ_ONCE(sig->cputimer.checking_timer))
+ return false; /* Lock holder's doing it for us */
+ raw_spin_lock(&sig->cputimer.lock);
+ }

		group_sample = sig->cputimer.cputime;
		raw_spin_unlock(&sig->cputimer.lock);

		if (task_cputime_expired(&group_sample, &sig->cputime_expires))
			return true;
	}
	return false;
}

----8<---- This is the patch mentioned above ----8<----

Quoted 1 line

From 0058bf5ed6a9e483ae33746685332e3ef9562ed5 Mon Sep 17 00:00:00 2001

From: George Spelvin <[email protected]>
Date: Wed, 26 Aug 2015 19:15:54 +0000
Subject: [PATCH] timer: Use bool more in kernel/time/posix-cpu-timers.c

This is mostly function return values, for documentation.

One structure field is changed (from int), but alignment padding precludes any actual space saving.

include/linux/sched.h +2 -2
index 26a2e612..fac3a7e2 100644
@@ -585,7 +585,7 @@ struct task_cputime {
 /**
  * struct thread_group_cputimer - thread group interval timer counts
  * @cputime:		thread group interval timers.
- * @running:		non-zero when there are timers running and
+ * @running:		%true when there are timers running and
  * 			@cputime receives updates.
  * @lock:		lock for fields in this struct.
  *
@@ -594,7 +594,7 @@ struct task_cputime {
  */
 struct thread_group_cputimer {
 	struct task_cputime cputime;
-	int running;
+	bool running;
 	raw_spinlock_t lock;
 };
 
kernel/time/posix-cpu-timers.c +21 -22

105 lines not on this page · raw message

#1 lore raw
Quoted 12 lines · George Spelvin

And some more comments on the series...

Review struct task_cputime_atomic { · line 632

Quoted 6 lines
(quoted diff) +1 -0
@@ -626,6 +628,7 @@ struct task_cputime_atomic {
 struct thread_group_cputimer {
 	struct task_cputime_atomic cputime_atomic;
 	int running;
+	int checking_timer;
 };

Why not make both running and checking_timer actual booleans, so the pair is only 16 bits rather than 64?

Quoted 5 lines

I suppose since sum_exec_runtime in struct task_cputime_atomic is 64 bits, alignment means there's no actual improvement. It's just my personal preference (Linus disagrees with me!) for the bool type for documentation.

(Compile-tested patch appended.)

I can include your patch in the series and then use boolean for the new checking_timer field. However, it looks like this applies on an old kernel. For example, the spin_lock field has already been removed from the structure.

Quoted 9 lines

But when it comes to really understanding this code, I'm struggling. It seems that this changes the return value of of fastpath_timer_check. I'm tryng to wrap my head around exactly why that is safe.

Any time I see things like READ_ONCE and WRITE_ONCE, I wonder if there need to be memory barriers as well. (Because x86 has strong default memory ordering, testing there doesn't prove the code right on other platforms.)

Quoted 8 lines

For the writes, the surrounding spinlock does the job.

But on the read side (fastpath_timer_check), I'm not sure what the rules should be.

Or is it basically okay if this is massively racey, since process-wide CPU timers are inherently sloppy. A race will just cause an expiration check to be missed, but it will be retried soon anyway.

Yes, the worst case scenario is that we wait until the next thread to come along and handle the next expired timer. However, this "delay" already occurs now (for example: a timer can expire right after a thread exits check_process_timers()).

Quoted 11 lines

Other half-baked thoughts that went through my head:

If the problem is that you have contention for read access to sig->cputimer.cputime, can that be changed to a seqlock?

Another possibility is to use a trylock before testing checking_timer:

	if (sig->cputimer.running) {
		struct task_cputime group_sample;

Review

Quoted 6 lines
(quoted diff) +5 -1
-		raw_spin_lock(&sig->cputimer.lock);
+		if (!raw_spin_trylock(&sig->cputimer.lock)) {
+			if (READ_ONCE(sig->cputimer.checking_timer))
+				return false;	/* Lock holder's doing it for us */
+			raw_spin_lock(&sig->cputimer.lock);
+		}

The spinlock call has already been removed from a previous patch. The issue now is with contention with the sighand lock.

Thanks,
Jason

#2 lore raw
Quoted 4 lines

I can include your patch in the series and then use boolean for the new checking_timer field. However, it looks like this applies on an old kernel. For example, the spin_lock field has already been removed from the structure.

Apologies; that was 4.1.6. A 4.2-rc8 patch is appended (it's a pretty trivial merge once you look at it).

Quoted 2 lines

The spinlock call has already been removed from a previous patch. The issue now is with contention with the sighand lock.

I'll look some more and try to wrap my head around it.

Quoted 8 lines
Quoted 3 lines

Or is it basically okay if this is massively racey, since process-wide CPU timers are inherently sloppy. A race will just cause an expiration check to be missed, but it will be retried soon anyway.

Yes, the worst case scenario is that we wait until the next thread to come along and handle the next expired timer. However, this "delay" already occurs now (for example: a timer can expire right after a thread exits check_process_timers()).

Ad is this polled, or is there some non-polled system that will trigger another call to check_process_timers().

E.g. suppose a process fails to notice that it blew past a CPU time timeout before blocking. Does anything guarantee that it will get the timeout signal in finite real time?

Quoted 1 line

From 95349f9b16c30aea518ce79d72e8e0f0c5d12069 Mon Sep 17 00:00:00 2001

From: George Spelvin <[email protected]>
Date: Wed, 26 Aug 2015 19:15:54 +0000
Subject: [PATCH] timer: Use bool more in kernel/time/posix-cpu-timers.c

This is mostly function return values, for documentation.

One structure field is changed (from int), but alignment padding precludes any actual space saving.

include/linux/sched.h +2 -2
index 04b5ada4..0cd80f76 100644
@@ -607,7 +607,7 @@ struct task_cputime_atomic {
 /**
  * struct thread_group_cputimer - thread group interval timer counts
  * @cputime_atomic:	atomic thread group interval timers.
- * @running:		non-zero when there are timers running and
+ * @running:		%true when there are timers running and
  * 			@cputime receives updates.
  *
  * This structure contains the version of task_cputime, above, that is
@@ -615,7 +615,7 @@ struct task_cputime_atomic {
  */
 struct thread_group_cputimer {
 	struct task_cputime_atomic cputime_atomic;
-	int running;
+	bool running;
 };
 
 #include <linux/rwsem.h>
kernel/time/posix-cpu-timers.c +21 -22

105 lines not on this page · raw message

#3 lore raw
Quoted 7 lines · George Spelvin
Quoted 4 lines

I can include your patch in the series and then use boolean for the new checking_timer field. However, it looks like this applies on an old kernel. For example, the spin_lock field has already been removed from the structure.

Apologies; that was 4.1.6. A 4.2-rc8 patch is appended (it's a pretty trivial merge once you look at it).

Frederic suggested that we just use a single "status" variable and access the bits for the running and checking field. I am leaning towards that method, so I might not include the rest of the boolean changes in this patchset.

Quoted 14 lines
Quoted 2 lines

The spinlock call has already been removed from a previous patch. The issue now is with contention with the sighand lock.

I'll look some more and try to wrap my head around it.

Quoted 8 lines
Quoted 3 lines

Or is it basically okay if this is massively racey, since process-wide CPU timers are inherently sloppy. A race will just cause an expiration check to be missed, but it will be retried soon anyway.

Yes, the worst case scenario is that we wait until the next thread to come along and handle the next expired timer. However, this "delay" already occurs now (for example: a timer can expire right after a thread exits check_process_timers()).

Quoted 6 lines

Ad is this polled, or is there some non-polled system that will trigger another call to check_process_timers().

E.g. suppose a process fails to notice that it blew past a CPU time timeout before blocking. Does anything guarantee that it will get the timeout signal in finite real time?

Yep, the check_process_timers will get called again during the next scheduler interrupt (approximately after 1 jiffy) and send the signal if it finds that the timer expired then.

#4 lore raw
Quoted 4 lines · Jason Low

Frederic suggested that we just use a single "status" variable and access the bits for the running and checking field. I am leaning towards that method, so I might not include the rest of the boolean changes in this patchset.

Don't worry, I'm not offended. I just started editing and figured I might as well share it.

Whichever solution is easier. My only complaint about bitmapped variables is that so many are "unsigned long" because the Linux atomic access primitives, originally designed for file system bitmaps, use that type.

But using that for a non array wastes 4 bytes on 64-bit platforms that can't be used if the code is to work on 32-bit ones.

Quoted 7 lines
Quoted 3 lines

E.g. suppose a process fails to notice that it blew past a CPU time timeout before blocking. Does anything guarantee that it will get the timeout signal in finite real time?

Yep, the check_process_timers will get called again during the next scheduler interrupt (approximately after 1 jiffy) and send the signal if it finds that the timer expired then.

Will it? I thought it got called on the running process only. Which is not the (blocked by assumption) process of interest.

I don't suspect that this would be a problem in practice, as CPU-time timers are used on activities which use a *lot* of it. But it seemed like a flaw worth either acknowledging or disproving.

#5 lore raw
Quoted 24 lines · George Spelvin
Quoted 4 lines · Jason Low

Frederic suggested that we just use a single "status" variable and access the bits for the running and checking field. I am leaning towards that method, so I might not include the rest of the boolean changes in this patchset.

Don't worry, I'm not offended. I just started editing and figured I might as well share it.

Whichever solution is easier. My only complaint about bitmapped variables is that so many are "unsigned long" because the Linux atomic access primitives, originally designed for file system bitmaps, use that type.

But using that for a non array wastes 4 bytes on 64-bit platforms that can't be used if the code is to work on 32-bit ones.

Quoted 7 lines
Quoted 3 lines

E.g. suppose a process fails to notice that it blew past a CPU time timeout before blocking. Does anything guarantee that it will get the timeout signal in finite real time?

Yep, the check_process_timers will get called again during the next scheduler interrupt (approximately after 1 jiffy) and send the signal if it finds that the timer expired then.

Quoted 6 lines

Will it? I thought it got called on the running process only. Which is not the (blocked by assumption) process of interest.

I don't suspect that this would be a problem in practice, as CPU-time timers are used on activities which use a *lot* of it. But it seemed like a flaw worth either acknowledging or disproving.

You're right, this is only called on running processes. If the process is blocked, sending the signal can get delayed. However, in practice, this is not really a problem as you mentioned. Also, this "issue" is also not really avoidable even without this patch. For instance, a timer may expire and the process can get block before the next scheduler interrupt.

Thanks,
Jason