kernel: make TIF_NOTIFY_SIGNAL and core dumps co-exist
PATCH 1/1 kernel: make TIF_NOTIFY_SIGNAL and core dumps co-exist
task_work being added with notify == TWA_SIGNAL will utilize TIF_NOTIFY_SIGNAL for signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.
Have task_work_add() with notify == TWA_SIGNAL check if a task has been signaled for a core dump, and refuse to add the work if that is the case. When a core dump is invoked, explicitly check for TIF_NOTIFY_SIGNAL and run any pending task_work if that is set. This is similar to how an exiting task will not get new task_work added, and we return the same error for the core dump case. As we return success or failure from task_work_add(), the caller has to be prepared to handle this case already.
Currently this manifests itself in that io_uring tasks that end up using task_work will experience truncated core dumps.
---
fs/coredump.c +8 -0
@@ -602,6 +602,14 @@ void do_coredump(const kernel_siginfo_t *siginfo) .mm_flags = mm->flags, }; +/* +* task_work_add() will refuse to add work after PF_SIGNALED has +* been set, ensure that we flush any pending TIF_NOTIFY_SIGNAL work +* if any was queued before that. +*/ +if (test_thread_flag (TIF_NOTIFY_SIGNAL )) +tracehook_notify_signal (); +audit_core_dumps (siginfo->si_signo); binfmt = mm->binfmt;
kernel/task_work.c +6 -0
@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work, head =READ_ONCE (task->task_works);if (unlikely (head == &work_exited))return -ESRCH ; +/* +* TIF_NOTIFY_SIGNAL notifications will interfere with +* a core dump in progress, reject them. +*/ +if (notify ==TWA_SIGNAL && (task->flags &PF_SIGNALED )) +return -ESRCH ; work->next = head; }while (cmpxchg (&task->task_works, head, work) != head);
Signature · 1 line
Jens Axboe
Quoted 5 lines · Jens Axboe
task_workbeing added with notify ==TWA_SIGNALwill utilizeTIF_NOTIFY_SIGNALfor signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.
This patch seems (a) buggy and (b) hacky.
kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49
Quoted 12 lines
kernel/task_work.c +6
-0@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work, head =READ_ONCE (task->task_works);if (unlikely (head == &work_exited))return -ESRCH ; +/* +* TIF_NOTIFY_SIGNAL notifications will interfere with +* a core dump in progress, reject them. +*/ +if (notify ==TWA_SIGNAL && (task->flags &PF_SIGNALED )) +return -ESRCH ;
This basically seems to check task->flags with no serialization.
I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.
Also, the main user that actually triggered the problem already has
if (unlikely(tsk->flags & PF_EXITING))
goto fail;just above the call to task_work_add(), so this all seems very hacky indeed.
Of course, I don't see what the serialization for _that_ one is either.
Pls explain. You can't just randomly add tests for random flags that get modified by other random code.
Linus
Quoted 33 lines · Linus Torvalds
Quoted 5 lines · Jens Axboe
task_workbeing added with notify ==TWA_SIGNALwill utilizeTIF_NOTIFY_SIGNALfor signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.This patch seems (a) buggy and (b) hacky.
kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49
Quoted 12 lines
kernel/task_work.c +6
-0@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work, head =READ_ONCE (task->task_works);if (unlikely (head == &work_exited))return -ESRCH ; +/* +* TIF_NOTIFY_SIGNAL notifications will interfere with +* a core dump in progress, reject them. +*/ +if (notify ==TWA_SIGNAL && (task->flags &PF_SIGNALED )) +return -ESRCH ;This basically seems to check
task->flagswith no serialization.I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.
Also, the main user that actually triggered the problem already has
if (unlikely(tsk->flags & PF_EXITING)) goto fail;
Quoted 6 lines
just above the call to
task_work_add(), so this all seems very hacky indeed.Of course, I don't see what the serialization for _that_ one is either.
Pls explain. You can't just randomly add tests for random flags that get modified by other random code.
You're absolutely right. On the io_uring side, in the current tree, there's only one check where current != task being checked - and that's in the poll rewait arming. That one should likely just go away. It may be fine as it is, as it just pertains to ring exit cancelations. We want to ensure that we don't rearm poll requests if the process is canceling and going away. I'll take a closer look at that one.
For this particular patch, I agree it's racy. I'll see if I can come up with something better...
Signature · 1 line
Jens Axboe
Quoted 12 lines · Jens Axboe
You're absolutely right. On the
io_uringside, in the current tree, there's only one check where current != task being checked - and that's
in the poll rewait arming. That one should likely just go away. It may
be fine as it is, as it just pertains to ring exit cancelations. We want
to ensure that we don't rearm poll requests if the process is canceling
and going away. I'll take a closer look at that one.
Quoted 4 lines
For this particular patch, I agree it's racy. I'll see if I can come up
with something better...
I have finally found the patch that you wanted me to test. I'm going to do it ASAP despite still having issue.
I do have a different approach to solve the same core dump issue.
Feel free to consider it if this can avoid the race condition described here.
Quoted 44 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe
task_workbeing added with notify ==TWA_SIGNALwill utilizeTIF_NOTIFY_SIGNALfor signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.This patch seems (a) buggy and (b) hacky.
kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49
Quoted 12 lines
kernel/task_work.c +6
-0@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work, head =READ_ONCE (task->task_works);if (unlikely (head == &work_exited))return -ESRCH ; +/* +* TIF_NOTIFY_SIGNAL notifications will interfere with +* a core dump in progress, reject them. +*/ +if (notify ==TWA_SIGNAL && (task->flags &PF_SIGNALED )) +return -ESRCH ;This basically seems to check
task->flagswith no serialization.I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.
Also, the main user that actually triggered the problem already has
if (unlikely(tsk->flags & PF_EXITING)) goto fail;Quoted 6 lines
just above the call to
task_work_add(), so this all seems very hacky indeed.Of course, I don't see what the serialization for _that_ one is either.
Pls explain. You can't just randomly add tests for random flags that get modified by other random code.
You're absolutely right. On the
io_uringside, in the current tree, there's only one check where current != task being checked - and that's in the poll rewait arming. That one should likely just go away. It may be fine as it is, as it just pertains to ring exit cancelations. We want to ensure that we don't rearm poll requests if the process is canceling and going away. I'll take a closer look at that one.
Quoted 3 lines
For this particular patch, I agree it's racy. I'll see if I can come up with something better...
Continuing this thread from August 2021:
I previously tested a version of Jens' patch backported to 5.10 and it fixed my problem. Now I am trying to upgrade kernels, and 5.17 still has the same problem - coredumps from an io_uring program to a pipe are truncated. Jens' patch applied to 5.17 again fixes the problem. Has there been any progress with fixing the problem upstream?
Reference:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
Tony Battersby
Cybernetics
Quoted 57 lines · Tony Battersby
Quoted 44 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe
task_workbeing added with notify ==TWA_SIGNALwill utilizeTIF_NOTIFY_SIGNALfor signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.This patch seems (a) buggy and (b) hacky.
kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49
Quoted 12 lines
kernel/task_work.c +6
-0@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work, head =READ_ONCE (task->task_works);if (unlikely (head == &work_exited))return -ESRCH ; +/* +* TIF_NOTIFY_SIGNAL notifications will interfere with +* a core dump in progress, reject them. +*/ +if (notify ==TWA_SIGNAL && (task->flags &PF_SIGNALED )) +return -ESRCH ;This basically seems to check
task->flagswith no serialization.I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.
Also, the main user that actually triggered the problem already has
if (unlikely(tsk->flags & PF_EXITING)) goto fail;Quoted 6 lines
just above the call to
task_work_add(), so this all seems very hacky indeed.Of course, I don't see what the serialization for _that_ one is either.
Pls explain. You can't just randomly add tests for random flags that get modified by other random code.
You're absolutely right. On the
io_uringside, in the current tree, there's only one check where current != task being checked - and that's in the poll rewait arming. That one should likely just go away. It may be fine as it is, as it just pertains to ring exit cancelations. We want to ensure that we don't rearm poll requests if the process is canceling and going away. I'll take a closer look at that one.Quoted 3 lines
For this particular patch, I agree it's racy. I'll see if I can come up with something better...
Continuing this thread from August 2021:
I previously tested a version of Jens' patch backported to 5.10 and it fixed my problem. Now I am trying to upgrade kernels, and 5.17 still has the same problem - coredumps from an
io_uringprogram to a pipe are truncated. Jens' patch applied to 5.17 again fixes the problem. Has there been any progress with fixing the problem upstream?
Quoted 4 lines
Reference:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
I am still slowly working on this. (I was unfortunately preempted by some painful to track down and fix regressions elsewhere).
When I was doubly checking to be certain I understood the problem the case you describe is one of the easy cases that needs to be handled.
There is at least one more difficult interaction that is not solved by squelching task_work_add after PF_SIGNALED is set, and I am not 100% convinced that it is even correct to squelch task_work_add at that point in the code.
The progress I have made to date that I am sending to Linus for v5.18 is the removal of tracehook.h which makes the code much more understandable.
I think I have a general solution that I am planning to post after v5.18-rc1 that I have not tested yet on the cases that I know about, but I expect it will work.
So I think that puts a good general fix 2-3 weeks out.
This is quite possibly a case where perfection is getting in the way of the good, but I honestly can't judge anything except a fix that cleans up everything and is complete. There are too many weird and subtle interactions that I don't understand.
So I am going to continue concentrating on a good general solution so that the code is readable and makes sense.
Eric
Quoted 88 lines · Eric W. Biederman
Quoted 56 lines · Tony Battersby
Quoted 44 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe
task_workbeing added with notify ==TWA_SIGNALwill utilizeTIF_NOTIFY_SIGNALfor signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.This patch seems (a) buggy and (b) hacky.
kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49
Quoted 12 lines
kernel/task_work.c +6
-0@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work, head =READ_ONCE (task->task_works);if (unlikely (head == &work_exited))return -ESRCH ; +/* +* TIF_NOTIFY_SIGNAL notifications will interfere with +* a core dump in progress, reject them. +*/ +if (notify ==TWA_SIGNAL && (task->flags &PF_SIGNALED )) +return -ESRCH ;This basically seems to check
task->flagswith no serialization.I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.
Also, the main user that actually triggered the problem already has
if (unlikely(tsk->flags & PF_EXITING)) goto fail;Quoted 6 lines
just above the call to
task_work_add(), so this all seems very hacky indeed.Of course, I don't see what the serialization for _that_ one is either.
Pls explain. You can't just randomly add tests for random flags that get modified by other random code.
You're absolutely right. On the
io_uringside, in the current tree, there's only one check where current != task being checked - and that's in the poll rewait arming. That one should likely just go away. It may be fine as it is, as it just pertains to ring exit cancelations. We want to ensure that we don't rearm poll requests if the process is canceling and going away. I'll take a closer look at that one.Quoted 3 lines
For this particular patch, I agree it's racy. I'll see if I can come up with something better...
Continuing this thread from August 2021:
I previously tested a version of Jens' patch backported to 5.10 and it fixed my problem. Now I am trying to upgrade kernels, and 5.17 still has the same problem - coredumps from an
io_uringprogram to a pipe are truncated. Jens' patch applied to 5.17 again fixes the problem. Has there been any progress with fixing the problem upstream?Quoted 4 lines
Reference:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/I am still slowly working on this. (I was unfortunately preempted by some painful to track down and fix regressions elsewhere).
When I was doubly checking to be certain I understood the problem the case you describe is one of the easy cases that needs to be handled.
There is at least one more difficult interaction that is not solved by squelching
task_work_addafterPF_SIGNALEDis set, and I am not 100% convinced that it is even correct to squelchtask_work_addat that point in the code.The progress I have made to date that I am sending to Linus for v5.18 is the removal of
tracehook.hwhich makes the code much more understandable.I think I have a general solution that I am planning to post after v5.18-rc1 that I have not tested yet on the cases that I know about, but I expect it will work.
So I think that puts a good general fix 2-3 weeks out.
This is quite possibly a case where perfection is getting in the way of the good, but I honestly can't judge anything except a fix that cleans up everything and is complete. There are too many weird and subtle interactions that I don't understand.
Quoted 5 lines
So I am going to continue concentrating on a good general solution so that the code is readable and makes sense.
Eric
Again continuing this thread from March 2022:
This problem has never been fixed; it is still present in 6.16-rc3. To
recap: core dumps from a program using io_uring get truncated, especially if the core is piped to a program. I have a forward-port of Jen's patch that fixes it, but apparently that patch did not pass muster with Linus back in 2021. Any chance of getting this fixed properly upstream?
Reference:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/
Tony Battersby
Cybernetics
Quoted 65 lines · Jens Axboe
task_workbeing added with notify ==TWA_SIGNALwill utilizeTIF_NOTIFY_SIGNALfor signaling the targeted task that work is available.
If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.Have
task_work_add()with notify ==TWA_SIGNALcheck if a task has been
signaled for a core dump, and refuse to add the work if that is the case.
When a core dump is invoked, explicitly check forTIF_NOTIFY_SIGNALand
run any pendingtask_workif that is set. This is similar to how an exiting task will not get newtask_workadded, and we return the same error for the core dump case. As we return success or failure fromtask_work_add(), the caller has to be prepared to handle this case already.Currently this manifests itself in that
io_uringtasks that end up usingtask_workwill experience truncated core dumps.Reported-by Tony Battersby <[email protected]>Reported-by Olivier Langlois <[email protected]>Cc Eric W. Biederman <[email protected]>Cc Oleg Nesterov <[email protected]>Cc Linus Torvalds <[email protected]>Cc [email protected] # 5.10+Signed-off-by Jens Axboe <[email protected]>---
fs/coredump.c +0
-0@@ -602,6 +602,14 @@ void do_coredump(const kernel_siginfo_t *siginfo) .mm_flags = mm->flags, };
+ /*
+ *task_work_add()will refuse to add work afterPF_SIGNALEDhas
+ * been set, ensure that we flush any pendingTIF_NOTIFY_SIGNALwork
+ * if any was queued before that.
+ */
+ if (test_thread_flag(TIF_NOTIFY_SIGNAL))
+ tracehook_notify_signal();
+
audit_core_dumps(siginfo->si_signo);
binfmt =mm->binfmt;kernel/task_work.c +2
-0@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, structcallback_head *work, head = READ_ONCE(task->task_works); if (unlikely(head == &work_exited)) return -ESRCH; + /* +* TIF_NOTIFY_SIGNAL notifications will interfere
Quoted 9 lines
with
+ * a core dump in progress, reject them. + */
+ if (notify ==TWA_SIGNAL&& (task->flags&PF_SIGNALED))
+ return-ESRCH;
work->next = head;
} while (cmpxchg(&task->task_works, head, work)!= head);
tested successfully on 5.12.19