task_work: only grab task signal lock when needed
PATCH 1/1 task_work: only grab task signal lock when needed
If JOBCTL_TASK_WORK is already set on the targeted task, then we need not go through {lock,unlock}_task_sighand() to set it again and queue a signal wakeup. This is safe as we're checking it _after adding the new task_work with cmpxchg().
---
Tested this with an intensive task_work based io_uring workload, and the benefits are quite large.
kernel/task_work.c +2 -1
@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)set_notify_resume (task);break ;case TWA_SIGNAL : -if (lock_task_sighand (task, &flags)) { +if (!(READ_ONCE (task->jobctl) &JOBCTL_TASK_WORK ) && +lock_task_sighand (task, &flags)) { task->jobctl |=JOBCTL_TASK_WORK ;signal_wake_up (task,0 );unlock_task_sighand (task, &flags);
Signature · 1 line
Jens Axboe
kernel/task_work.c task_work_add(struct task_struct *task, struct callback_head *work, int notify) · line 46
Quoted 10 lines · Jens Axboe
kernel/task_work.c +2
-1@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)set_notify_resume (task);break ;case TWA_SIGNAL : -if (lock_task_sighand (task, &flags)) { +if (!(READ_ONCE (task->jobctl) &JOBCTL_TASK_WORK ) && +lock_task_sighand (task, &flags)) {
Aaaaah, sorry Jens, now I think this is racy. So I am glad I didn't add this optimization into the initial version ;)
It is possible that JOBCTL_TASK_WORK is set but ->task_works == NULL. Say, task_work_add(TWA_SIGNAL) + task_work_cancel(), or the target task can call task_work_run() before it enters get_signal().
And in this case another task_work_add(tsk, TWA_SIGNAL) can actually race with get_signal() which does
current->jobctl &= ~JOBCTL_TASK_WORK;
if (unlikely(current->task_works)) {
spin_unlock_irq(&sighand->siglock);
task_work_run();nothing guarantees that get_signal() sees ->task_works != NULL. Probably this is what Jann meant.
We can probably add a barrier into get_signal() but I didn't sleep today, I'll try to think tomorrow.
Oleg.
Quoted 27 lines · Oleg Nesterov
kernel/task_work.c task_work_add(struct task_struct *task, struct callback_head *work, int notify) · line 46
Quoted 10 lines · Jens Axboe
kernel/task_work.c +2
-1@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)set_notify_resume (task);break ;case TWA_SIGNAL : -if (lock_task_sighand (task, &flags)) { +if (!(READ_ONCE (task->jobctl) &JOBCTL_TASK_WORK ) && +lock_task_sighand (task, &flags)) {Aaaaah, sorry Jens, now I think this is racy. So I am glad I didn't add this optimization into the initial version ;)
It is possible that
JOBCTL_TASK_WORKis set but ->task_works == NULL. Say,task_work_add(TWA_SIGNAL)+task_work_cancel(), or the target task can calltask_work_run()before it entersget_signal().And in this case another
task_work_add(tsk, TWA_SIGNAL)can actually race withget_signal()which doescurrent->jobctl &= ~JOBCTL_TASK_WORK; if (unlikely(current->task_works)) { spin_unlock_irq(&sighand->siglock); task_work_run();
Quoted 5 lines
nothing guarantees that
get_signal()sees ->task_works != NULL. Probably this is what Jann meant.We can probably add a barrier into
get_signal()but I didn't sleep today, I'll try to think tomorrow.
Appreciate you looking into this! Would be pretty critical for me to get this working.
Signature · 1 line
Jens Axboe
Quoted 28 lines · Oleg Nesterov
kernel/task_work.c task_work_add(struct task_struct *task, struct callback_head *work, int notify) · line 46
Quoted 10 lines · Jens Axboe
kernel/task_work.c +2
-1@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)set_notify_resume (task);break ;case TWA_SIGNAL : -if (lock_task_sighand (task, &flags)) { +if (!(READ_ONCE (task->jobctl) &JOBCTL_TASK_WORK ) && +lock_task_sighand (task, &flags)) {Aaaaah, sorry Jens, now I think this is racy. So I am glad I didn't add this optimization into the initial version ;)
It is possible that
JOBCTL_TASK_WORKis set but ->task_works == NULL. Say,task_work_add(TWA_SIGNAL)+task_work_cancel(), or the target task can calltask_work_run()before it entersget_signal().And in this case another
task_work_add(tsk, TWA_SIGNAL)can actually race withget_signal()which doescurrent->jobctl &= ~JOBCTL_TASK_WORK; if (unlikely(current->task_works)) { spin_unlock_irq(&sighand->siglock); task_work_run();
Quoted 5 lines
nothing guarantees that
get_signal()sees ->task_works != NULL. Probably this is what Jann meant.We can probably add a barrier into
get_signal()but I didn't sleep today, I'll try to think tomorrow.
I see nothing better than the additional change below. Peter, do you see another solution?
This needs a comment to explain that this mb() pairs with another barrier provided by cmpxchg() in task_work_add(). It ensures that either get_signal() sees the new work added by task_work_add(), or task_work_add() sees the result of "&= ~JOBCTL_TASK_WORK".
Oleg.
kernel/signal.c +1 -1
@@ -2541,7 +2541,7 @@ bool get_signal(struct ksignal *ksig) relock:spin_lock_irq (&sighand->siglock); - current->jobctl &= ~JOBCTL_TASK_WORK ; +smp_store_mb (current->jobctl, current->jobctl & ~JOBCTL_TASK_WORK );if (unlikely (current->task_works)) {spin_unlock_irq (&sighand->siglock);task_work_run ();
Quoted 2 lines · Oleg Nesterov
I see nothing better than the additional change below. Peter, do you see another solution?
Nope -- although I don't claim to understand the signal code much.
Quoted 11 lines
This needs a comment to explain that this
mb()pairs with another barrier provided bycmpxchg()intask_work_add(). It ensures that eitherget_signal()sees the new work added bytask_work_add(), ortask_work_add()sees the result of "&= ~JOBCTL_TASK_WORK".Oleg.
kernel/signal.c bool get_signal(struct ksignal *ksig) · line 2547
Quoted 8 lines
kernel/signal.c +1
-1@@ -2541,7 +2541,7 @@ bool get_signal(struct ksignal *ksig) relock:spin_lock_irq (&sighand->siglock); - current->jobctl &= ~JOBCTL_TASK_WORK ; +smp_store_mb (current->jobctl, current->jobctl & ~JOBCTL_TASK_WORK );if (unlikely (current->task_works)) {spin_unlock_irq (&sighand->siglock);task_work_run ();
I agree this should work; smp_store_mb() isn't my favourite primitive, but yes, this seems as good a use of it as there is so why not.
Quoted 49 lines · Oleg Nesterov
Quoted 28 lines · Oleg Nesterov
kernel/task_work.c task_work_add(struct task_struct *task, struct callback_head *work, int notify) · line 46
Quoted 10 lines · Jens Axboe
kernel/task_work.c +2
-1@@ -42,7 +42,8 @@ task_work_add(struct task_struct *task, struct callback_head *work, int notify)set_notify_resume (task);break ;case TWA_SIGNAL : -if (lock_task_sighand (task, &flags)) { +if (!(READ_ONCE (task->jobctl) &JOBCTL_TASK_WORK ) && +lock_task_sighand (task, &flags)) {Aaaaah, sorry Jens, now I think this is racy. So I am glad I didn't add this optimization into the initial version ;)
It is possible that
JOBCTL_TASK_WORKis set but ->task_works == NULL. Say,task_work_add(TWA_SIGNAL)+task_work_cancel(), or the target task can calltask_work_run()before it entersget_signal().And in this case another
task_work_add(tsk, TWA_SIGNAL)can actually race withget_signal()which doescurrent->jobctl &= ~JOBCTL_TASK_WORK; if (unlikely(current->task_works)) { spin_unlock_irq(&sighand->siglock); task_work_run();Quoted 5 lines
nothing guarantees that
get_signal()sees ->task_works != NULL. Probably this is what Jann meant.We can probably add a barrier into
get_signal()but I didn't sleep today, I'll try to think tomorrow.I see nothing better than the additional change below. Peter, do you see another solution?
This needs a comment to explain that this
mb()pairs with another barrier provided bycmpxchg()intask_work_add(). It ensures that eitherget_signal()sees the new work added bytask_work_add(), ortask_work_add()sees the result of "&= ~JOBCTL_TASK_WORK".Oleg.
kernel/signal.c bool get_signal(struct ksignal *ksig) · line 2547
Quoted 8 lines
kernel/signal.c +1
-1@@ -2541,7 +2541,7 @@ bool get_signal(struct ksignal *ksig) relock:spin_lock_irq (&sighand->siglock); - current->jobctl &= ~JOBCTL_TASK_WORK ; +smp_store_mb (current->jobctl, current->jobctl & ~JOBCTL_TASK_WORK );if (unlikely (current->task_works)) {spin_unlock_irq (&sighand->siglock);task_work_run ();
I think this should work when paired with the READ_ONCE() on the task_work_add() side. I haven't managed to reproduce badness with the existing one that doesn't have the smp_store_mb() here, so can't verify much beyond that...
Are you going to send this out as a complete patch?
Signature · 1 line
Jens Axboe
Quoted 7 lines · Jens Axboe
Quoted 5 lines · Oleg Nesterov
kernel/signal.c +0
-0@@ -2541,7 +2541,7 @@ bool get_signal(struct ksignal *ksig)
Quoted 11 lines
kernel/signal.c bool get_signal(struct ksignal *ksig) · line 2547
Quoted 8 lines
kernel/signal.c +1
-1relock:spin_lock_irq (&sighand->siglock); - current->jobctl &= ~JOBCTL_TASK_WORK ; +smp_store_mb (current->jobctl, current->jobctl & ~JOBCTL_TASK_WORK );if (unlikely (current->task_works)) {spin_unlock_irq (&sighand->siglock);task_work_run ();I think this should work when paired with the
READ_ONCE()on thetask_work_add()side.
It pairs with mb (implied by cmpxchg) before READ_ONCE. So we roughly have
task_work_add: get_signal: STORE(task->task_works, new_work); STORE(task->jobctl); mb(); mb(); LOAD(task->jobctl); LOAD(task->task_works);
and we can rely on STORE-MB-LOAD.
Quoted 3 lines
I haven't managed to reproduce badness with the
existing one that doesn't have thesmp_store_mb()here, so can't verify much beyond that...
Yes, the race is very unlikely. And the problem is minor, the target task can miss the new work added by TWA_SIGNAL and return from get_signal() without TIF_SIGPENDING.
Quoted 1 line
Are you going to send this out as a complete patch?
Jens, could you please send the patch? I am on vacation and travelling. Feel free to add my ACK.
Oleg.