Skip to contentSkip to conversation

task_work: only grab task signal lock when needed

PATCH In review Jens Axboe 3 participants 7 messages Aug 11, 2020 – Aug 13, 2020

lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

PATCH 1/1 task_work: only grab task signal lock when needed

If JOBCTL_TASK_WORK is already set on the targeted task, then we need not go through {lock,unlock}_task_sighand() to set it again and queue a signal wakeup. This is safe as we're checking it _after adding the new task_work with cmpxchg().

---

Tested this with an intensive task_work based io_uring workload, and the benefits are quite large.

kernel/task_work.c +2 -1
index 5c0848ca1287..cbf8cab6e864 100644
@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)
 		set_notify_resume(task);
 		break;
 	case TWA_SIGNAL:
-		if (lock_task_sighand(task, &flags)) {
+		if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&
+		    lock_task_sighand(task, &flags)) {
 			task->jobctl |= JOBCTL_TASK_WORK;
 			signal_wake_up(task, 0);
 			unlock_task_sighand(task, &flags);
Signature · 1 line
Jens Axboe
#1 lore raw

kernel/task_work.c task_work_add(struct task_struct *task, struct callback_head *work, int notify) · line 46

Quoted 10 lines · Jens Axboe
kernel/task_work.c +2 -1
@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)
 		set_notify_resume(task);
 		break;
 	case TWA_SIGNAL:
-		if (lock_task_sighand(task, &flags)) {
+		if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&
+		    lock_task_sighand(task, &flags)) {

Aaaaah, sorry Jens, now I think this is racy. So I am glad I didn't add this optimization into the initial version ;)

It is possible that JOBCTL_TASK_WORK is set but ->task_works == NULL. Say, task_work_add(TWA_SIGNAL) + task_work_cancel(), or the target task can call task_work_run() before it enters get_signal().

And in this case another task_work_add(tsk, TWA_SIGNAL) can actually race with get_signal() which does

	current->jobctl &= ~JOBCTL_TASK_WORK;
	if (unlikely(current->task_works)) {
		spin_unlock_irq(&sighand->siglock);
		task_work_run();

nothing guarantees that get_signal() sees ->task_works != NULL. Probably this is what Jann meant.

We can probably add a barrier into get_signal() but I didn't sleep today, I'll try to think tomorrow.

Oleg.

author
#2 lore raw
Quoted 27 lines · Oleg Nesterov

kernel/task_work.c task_work_add(struct task_struct *task, struct callback_head *work, int notify) · line 46

Quoted 10 lines · Jens Axboe
kernel/task_work.c +2 -1
@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)
 		set_notify_resume(task);
 		break;
 	case TWA_SIGNAL:
-		if (lock_task_sighand(task, &flags)) {
+		if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&
+		    lock_task_sighand(task, &flags)) {

Aaaaah, sorry Jens, now I think this is racy. So I am glad I didn't add this optimization into the initial version ;)

It is possible that JOBCTL_TASK_WORK is set but ->task_works == NULL. Say, task_work_add(TWA_SIGNAL) + task_work_cancel(), or the target task can call task_work_run() before it enters get_signal().

And in this case another task_work_add(tsk, TWA_SIGNAL) can actually race with get_signal() which does

	current->jobctl &= ~JOBCTL_TASK_WORK;
	if (unlikely(current->task_works)) {
		spin_unlock_irq(&sighand->siglock);
		task_work_run();
Quoted 5 lines

nothing guarantees that get_signal() sees ->task_works != NULL. Probably this is what Jann meant.

We can probably add a barrier into get_signal() but I didn't sleep today, I'll try to think tomorrow.

Appreciate you looking into this! Would be pretty critical for me to get this working.

Signature · 1 line
Jens Axboe
#3 lore raw
In reply to Oleg Nesterov
Quoted 28 lines · Oleg Nesterov

kernel/task_work.c task_work_add(struct task_struct *task, struct callback_head *work, int notify) · line 46

Quoted 10 lines · Jens Axboe
kernel/task_work.c +2 -1
@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)
 		set_notify_resume(task);
 		break;
 	case TWA_SIGNAL:
-		if (lock_task_sighand(task, &flags)) {
+		if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&
+		    lock_task_sighand(task, &flags)) {

Aaaaah, sorry Jens, now I think this is racy. So I am glad I didn't add this optimization into the initial version ;)

It is possible that JOBCTL_TASK_WORK is set but ->task_works == NULL. Say, task_work_add(TWA_SIGNAL) + task_work_cancel(), or the target task can call task_work_run() before it enters get_signal().

And in this case another task_work_add(tsk, TWA_SIGNAL) can actually race with get_signal() which does

	current->jobctl &= ~JOBCTL_TASK_WORK;
	if (unlikely(current->task_works)) {
		spin_unlock_irq(&sighand->siglock);
		task_work_run();
Quoted 5 lines

nothing guarantees that get_signal() sees ->task_works != NULL. Probably this is what Jann meant.

We can probably add a barrier into get_signal() but I didn't sleep today, I'll try to think tomorrow.

I see nothing better than the additional change below. Peter, do you see another solution?

This needs a comment to explain that this mb() pairs with another barrier provided by cmpxchg() in task_work_add(). It ensures that either get_signal() sees the new work added by task_work_add(), or task_work_add() sees the result of "&= ~JOBCTL_TASK_WORK".

Oleg.

kernel/signal.c +1 -1
@@ -2541,7 +2541,7 @@ bool get_signal(struct ksignal *ksig)
 
 relock:
 	spin_lock_irq(&sighand->siglock);
-	current->jobctl &= ~JOBCTL_TASK_WORK;
+	smp_store_mb(current->jobctl, current->jobctl & ~JOBCTL_TASK_WORK);
 	if (unlikely(current->task_works)) {
 		spin_unlock_irq(&sighand->siglock);
 		task_work_run();
#4 lore raw
Quoted 2 lines · Oleg Nesterov

I see nothing better than the additional change below. Peter, do you see another solution?

Nope -- although I don't claim to understand the signal code much.

Quoted 11 lines

This needs a comment to explain that this mb() pairs with another barrier provided by cmpxchg() in task_work_add(). It ensures that either get_signal() sees the new work added by task_work_add(), or task_work_add() sees the result of "&= ~JOBCTL_TASK_WORK".

Oleg.

kernel/signal.c bool get_signal(struct ksignal *ksig) · line 2547

Quoted 8 lines
kernel/signal.c +1 -1
@@ -2541,7 +2541,7 @@ bool get_signal(struct ksignal *ksig)
 
 relock:
 	spin_lock_irq(&sighand->siglock);
-	current->jobctl &= ~JOBCTL_TASK_WORK;
+	smp_store_mb(current->jobctl, current->jobctl & ~JOBCTL_TASK_WORK);
 	if (unlikely(current->task_works)) {
 		spin_unlock_irq(&sighand->siglock);
 		task_work_run();

I agree this should work; smp_store_mb() isn't my favourite primitive, but yes, this seems as good a use of it as there is so why not.

author
#5 lore raw
In reply to Oleg Nesterov
Quoted 49 lines · Oleg Nesterov
Quoted 28 lines · Oleg Nesterov

kernel/task_work.c task_work_add(struct task_struct *task, struct callback_head *work, int notify) · line 46

Quoted 10 lines · Jens Axboe
kernel/task_work.c +2 -1
@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)
 		set_notify_resume(task);
 		break;
 	case TWA_SIGNAL:
-		if (lock_task_sighand(task, &flags)) {
+		if (!(READ_ONCE(task->jobctl) & JOBCTL_TASK_WORK) &&
+		    lock_task_sighand(task, &flags)) {

Aaaaah, sorry Jens, now I think this is racy. So I am glad I didn't add this optimization into the initial version ;)

It is possible that JOBCTL_TASK_WORK is set but ->task_works == NULL. Say, task_work_add(TWA_SIGNAL) + task_work_cancel(), or the target task can call task_work_run() before it enters get_signal().

And in this case another task_work_add(tsk, TWA_SIGNAL) can actually race with get_signal() which does

	current->jobctl &= ~JOBCTL_TASK_WORK;
	if (unlikely(current->task_works)) {
		spin_unlock_irq(&sighand->siglock);
		task_work_run();
Quoted 5 lines

nothing guarantees that get_signal() sees ->task_works != NULL. Probably this is what Jann meant.

We can probably add a barrier into get_signal() but I didn't sleep today, I'll try to think tomorrow.

I see nothing better than the additional change below. Peter, do you see another solution?

This needs a comment to explain that this mb() pairs with another barrier provided by cmpxchg() in task_work_add(). It ensures that either get_signal() sees the new work added by task_work_add(), or task_work_add() sees the result of "&= ~JOBCTL_TASK_WORK".

Oleg.

kernel/signal.c bool get_signal(struct ksignal *ksig) · line 2547

Quoted 8 lines
kernel/signal.c +1 -1
@@ -2541,7 +2541,7 @@ bool get_signal(struct ksignal *ksig)
 
 relock:
 	spin_lock_irq(&sighand->siglock);
-	current->jobctl &= ~JOBCTL_TASK_WORK;
+	smp_store_mb(current->jobctl, current->jobctl & ~JOBCTL_TASK_WORK);
 	if (unlikely(current->task_works)) {
 		spin_unlock_irq(&sighand->siglock);
 		task_work_run();

I think this should work when paired with the READ_ONCE() on the task_work_add() side. I haven't managed to reproduce badness with the existing one that doesn't have the smp_store_mb() here, so can't verify much beyond that...

Are you going to send this out as a complete patch?

Signature · 1 line
Jens Axboe
#6 lore raw
Quoted 7 lines · Jens Axboe
Quoted 5 lines · Oleg Nesterov
kernel/signal.c +0 -0
@@ -2541,7 +2541,7 @@ bool get_signal(struct ksignal *ksig)
 
Quoted 11 lines

kernel/signal.c bool get_signal(struct ksignal *ksig) · line 2547

Quoted 8 lines
kernel/signal.c +1 -1
 relock:
 	spin_lock_irq(&sighand->siglock);
-	current->jobctl &= ~JOBCTL_TASK_WORK;
+	smp_store_mb(current->jobctl, current->jobctl & ~JOBCTL_TASK_WORK);
 	if (unlikely(current->task_works)) {
 		spin_unlock_irq(&sighand->siglock);
 		task_work_run();

I think this should work when paired with the READ_ONCE() on the task_work_add() side.

It pairs with mb (implied by cmpxchg) before READ_ONCE. So we roughly have

	task_work_add:				get_signal:

	STORE(task->task_works, new_work);	STORE(task->jobctl);
	mb();					mb();
	LOAD(task->jobctl);			LOAD(task->task_works);

and we can rely on STORE-MB-LOAD.

Quoted 3 lines

I haven't managed to reproduce badness with the
existing one that doesn't have the smp_store_mb() here, so can't verify much beyond that...

Yes, the race is very unlikely. And the problem is minor, the target task can miss the new work added by TWA_SIGNAL and return from get_signal() without TIF_SIGPENDING.

Quoted 1 line

Are you going to send this out as a complete patch?

Jens, could you please send the patch? I am on vacation and travelling. Feel free to add my ACK.

Oleg.