Skip to contentSkip to conversation

kernel: make TIF_NOTIFY_SIGNAL and core dumps co-exist

PATCH In review Jens Axboe 5 participants 8 messages Aug 18, 2021 – Jun 27, 2025 VFS

lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

PATCH 1/1 kernel: make TIF_NOTIFY_SIGNAL and core dumps co-exist

task_work being added with notify == TWA_SIGNAL will utilize TIF_NOTIFY_SIGNAL for signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.

Have task_work_add() with notify == TWA_SIGNAL check if a task has been signaled for a core dump, and refuse to add the work if that is the case. When a core dump is invoked, explicitly check for TIF_NOTIFY_SIGNAL and run any pending task_work if that is set. This is similar to how an exiting task will not get new task_work added, and we return the same error for the core dump case. As we return success or failure from task_work_add(), the caller has to be prepared to handle this case already.

Currently this manifests itself in that io_uring tasks that end up using task_work will experience truncated core dumps.

---

fs/coredump.c +8 -0
index 07afb5ddb1c4..ca7c1ee44ada 100644
@@ -602,6 +602,14 @@ void do_coredump(const kernel_siginfo_t *siginfo)
 		.mm_flags = mm->flags,
 	};
 
+	/*
+	 * task_work_add() will refuse to add work after PF_SIGNALED has
+	 * been set, ensure that we flush any pending TIF_NOTIFY_SIGNAL work
+	 * if any was queued before that.
+	 */
+	if (test_thread_flag(TIF_NOTIFY_SIGNAL))
+		tracehook_notify_signal();
+
 	audit_core_dumps(siginfo->si_signo);
 
 	binfmt = mm->binfmt;
kernel/task_work.c +6 -0
index 1698fbe6f0e1..1ab28904adc4 100644
@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work,
 		head = READ_ONCE(task->task_works);
 		if (unlikely(head == &work_exited))
 			return -ESRCH;
+		/*
+		 * TIF_NOTIFY_SIGNAL notifications will interfere with
+		 * a core dump in progress, reject them.
+		 */
+		if (notify == TWA_SIGNAL && (task->flags & PF_SIGNALED))
+			return -ESRCH;
 		work->next = head;
 	} while (cmpxchg(&task->task_works, head, work) != head);
 
Signature · 1 line
Jens Axboe
#1 lore raw
Quoted 5 lines · Jens Axboe

task_work being added with notify == TWA_SIGNAL will utilize TIF_NOTIFY_SIGNAL for signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.

This patch seems (a) buggy and (b) hacky.

kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49

Quoted 12 lines
kernel/task_work.c +6 -0
@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work,
                head = READ_ONCE(task->task_works);
                if (unlikely(head == &work_exited))
                        return -ESRCH;
+               /*
+                * TIF_NOTIFY_SIGNAL notifications will interfere with
+                * a core dump in progress, reject them.
+                */
+               if (notify == TWA_SIGNAL && (task->flags & PF_SIGNALED))
+                       return -ESRCH;

This basically seems to check task->flags with no serialization.

I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.

Also, the main user that actually triggered the problem already has

        if (unlikely(tsk->flags & PF_EXITING))
                goto fail;

just above the call to task_work_add(), so this all seems very hacky indeed.

Of course, I don't see what the serialization for _that_ one is either.

Pls explain. You can't just randomly add tests for random flags that get modified by other random code.

Linus

author
#2 lore raw
Quoted 33 lines · Linus Torvalds
Quoted 5 lines · Jens Axboe

task_work being added with notify == TWA_SIGNAL will utilize TIF_NOTIFY_SIGNAL for signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.

This patch seems (a) buggy and (b) hacky.

kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49

Quoted 12 lines
kernel/task_work.c +6 -0
@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work,
                head = READ_ONCE(task->task_works);
                if (unlikely(head == &work_exited))
                        return -ESRCH;
+               /*
+                * TIF_NOTIFY_SIGNAL notifications will interfere with
+                * a core dump in progress, reject them.
+                */
+               if (notify == TWA_SIGNAL && (task->flags & PF_SIGNALED))
+                       return -ESRCH;

This basically seems to check task->flags with no serialization.

I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.

Also, the main user that actually triggered the problem already has

        if (unlikely(tsk->flags & PF_EXITING))
                goto fail;
Quoted 6 lines

just above the call to task_work_add(), so this all seems very hacky indeed.

Of course, I don't see what the serialization for _that_ one is either.

Pls explain. You can't just randomly add tests for random flags that get modified by other random code.

You're absolutely right. On the io_uring side, in the current tree, there's only one check where current != task being checked - and that's in the poll rewait arming. That one should likely just go away. It may be fine as it is, as it just pertains to ring exit cancelations. We want to ensure that we don't rearm poll requests if the process is canceling and going away. I'll take a closer look at that one.

For this particular patch, I agree it's racy. I'll see if I can come up with something better...

Signature · 1 line
Jens Axboe
#3 lore raw
Quoted 12 lines · Jens Axboe

You're absolutely right. On the io_uring side, in the current tree, there's only one check where current != task being checked - and that's
in the poll rewait arming. That one should likely just go away. It may
be fine as it is, as it just pertains to ring exit cancelations. We want
to ensure that we don't rearm poll requests if the process is canceling
and going away. I'll take a closer look at that one.

Quoted 4 lines

For this particular patch, I agree it's racy. I'll see if I can come up
with something better...

I have finally found the patch that you wanted me to test. I'm going to do it ASAP despite still having issue.

I do have a different approach to solve the same core dump issue.

Feel free to consider it if this can avoid the race condition described here.

#4 lore raw
In reply to Jens Axboe
Quoted 44 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe

task_work being added with notify == TWA_SIGNAL will utilize TIF_NOTIFY_SIGNAL for signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.

This patch seems (a) buggy and (b) hacky.

kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49

Quoted 12 lines
kernel/task_work.c +6 -0
@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work,
                head = READ_ONCE(task->task_works);
                if (unlikely(head == &work_exited))
                        return -ESRCH;
+               /*
+                * TIF_NOTIFY_SIGNAL notifications will interfere with
+                * a core dump in progress, reject them.
+                */
+               if (notify == TWA_SIGNAL && (task->flags & PF_SIGNALED))
+                       return -ESRCH;

This basically seems to check task->flags with no serialization.

I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.

Also, the main user that actually triggered the problem already has

        if (unlikely(tsk->flags & PF_EXITING))
                goto fail;
Quoted 6 lines

just above the call to task_work_add(), so this all seems very hacky indeed.

Of course, I don't see what the serialization for _that_ one is either.

Pls explain. You can't just randomly add tests for random flags that get modified by other random code.

You're absolutely right. On the io_uring side, in the current tree, there's only one check where current != task being checked - and that's in the poll rewait arming. That one should likely just go away. It may be fine as it is, as it just pertains to ring exit cancelations. We want to ensure that we don't rearm poll requests if the process is canceling and going away. I'll take a closer look at that one.

Quoted 3 lines

For this particular patch, I agree it's racy. I'll see if I can come up with something better...

Continuing this thread from August 2021:

I previously tested a version of Jens' patch backported to 5.10 and it fixed my problem.  Now I am trying to upgrade kernels, and 5.17 still has the same problem - coredumps from an io_uring program to a pipe are truncated.  Jens' patch applied to 5.17 again fixes the problem.  Has there been any progress with fixing the problem upstream?

Reference:

https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/

Tony Battersby
Cybernetics

#5 lore raw
Quoted 57 lines · Tony Battersby
Quoted 44 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe

task_work being added with notify == TWA_SIGNAL will utilize TIF_NOTIFY_SIGNAL for signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.

This patch seems (a) buggy and (b) hacky.

kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49

Quoted 12 lines
kernel/task_work.c +6 -0
@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work,
                head = READ_ONCE(task->task_works);
                if (unlikely(head == &work_exited))
                        return -ESRCH;
+               /*
+                * TIF_NOTIFY_SIGNAL notifications will interfere with
+                * a core dump in progress, reject them.
+                */
+               if (notify == TWA_SIGNAL && (task->flags & PF_SIGNALED))
+                       return -ESRCH;

This basically seems to check task->flags with no serialization.

I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.

Also, the main user that actually triggered the problem already has

        if (unlikely(tsk->flags & PF_EXITING))
                goto fail;
Quoted 6 lines

just above the call to task_work_add(), so this all seems very hacky indeed.

Of course, I don't see what the serialization for _that_ one is either.

Pls explain. You can't just randomly add tests for random flags that get modified by other random code.

You're absolutely right. On the io_uring side, in the current tree, there's only one check where current != task being checked - and that's in the poll rewait arming. That one should likely just go away. It may be fine as it is, as it just pertains to ring exit cancelations. We want to ensure that we don't rearm poll requests if the process is canceling and going away. I'll take a closer look at that one.

Quoted 3 lines

For this particular patch, I agree it's racy. I'll see if I can come up with something better...

Continuing this thread from August 2021:

I previously tested a version of Jens' patch backported to 5.10 and it fixed my problem.  Now I am trying to upgrade kernels, and 5.17 still has the same problem - coredumps from an io_uring program to a pipe are truncated.  Jens' patch applied to 5.17 again fixes the problem.  Has there been any progress with fixing the problem upstream?

Quoted 4 lines

Reference:

https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/

I am still slowly working on this. (I was unfortunately preempted by some painful to track down and fix regressions elsewhere).

When I was doubly checking to be certain I understood the problem the case you describe is one of the easy cases that needs to be handled.

There is at least one more difficult interaction that is not solved by squelching task_work_add after PF_SIGNALED is set, and I am not 100% convinced that it is even correct to squelch task_work_add at that point in the code.

The progress I have made to date that I am sending to Linus for v5.18 is the removal of tracehook.h which makes the code much more understandable.

I think I have a general solution that I am planning to post after v5.18-rc1 that I have not tested yet on the cases that I know about, but I expect it will work.

So I think that puts a good general fix 2-3 weeks out.

This is quite possibly a case where perfection is getting in the way of the good, but I honestly can't judge anything except a fix that cleans up everything and is complete. There are too many weird and subtle interactions that I don't understand.

So I am going to continue concentrating on a good general solution so that the code is readable and makes sense.

Eric

#6 lore raw
Quoted 88 lines · Eric W. Biederman
Quoted 56 lines · Tony Battersby
Quoted 44 lines · Jens Axboe
Quoted 30 lines · Linus Torvalds
Quoted 4 lines · Jens Axboe

task_work being added with notify == TWA_SIGNAL will utilize TIF_NOTIFY_SIGNAL for signaling the targeted task that work is available. If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.

This patch seems (a) buggy and (b) hacky.

kernel/task_work.c int task_work_add(struct task_struct *task, struct callback_head *work, · line 49

Quoted 12 lines
kernel/task_work.c +6 -0
@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct callback_head *work,
                head = READ_ONCE(task->task_works);
                if (unlikely(head == &work_exited))
                        return -ESRCH;
+               /*
+                * TIF_NOTIFY_SIGNAL notifications will interfere with
+                * a core dump in progress, reject them.
+                */
+               if (notify == TWA_SIGNAL && (task->flags & PF_SIGNALED))
+                       return -ESRCH;

This basically seems to check task->flags with no serialization.

I'm sure it works 99.9% of the time in practice, since you'd be really unlucky to hit any races, but I really don't see what the serialization logic is.

Also, the main user that actually triggered the problem already has

        if (unlikely(tsk->flags & PF_EXITING))
                goto fail;
Quoted 6 lines

just above the call to task_work_add(), so this all seems very hacky indeed.

Of course, I don't see what the serialization for _that_ one is either.

Pls explain. You can't just randomly add tests for random flags that get modified by other random code.

You're absolutely right. On the io_uring side, in the current tree, there's only one check where current != task being checked - and that's in the poll rewait arming. That one should likely just go away. It may be fine as it is, as it just pertains to ring exit cancelations. We want to ensure that we don't rearm poll requests if the process is canceling and going away. I'll take a closer look at that one.

Quoted 3 lines

For this particular patch, I agree it's racy. I'll see if I can come up with something better...

Continuing this thread from August 2021:

I previously tested a version of Jens' patch backported to 5.10 and it fixed my problem.  Now I am trying to upgrade kernels, and 5.17 still has the same problem - coredumps from an io_uring program to a pipe are truncated.  Jens' patch applied to 5.17 again fixes the problem.  Has there been any progress with fixing the problem upstream?

Quoted 4 lines

Reference:

https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/

I am still slowly working on this. (I was unfortunately preempted by some painful to track down and fix regressions elsewhere).

When I was doubly checking to be certain I understood the problem the case you describe is one of the easy cases that needs to be handled.

There is at least one more difficult interaction that is not solved by squelching task_work_add after PF_SIGNALED is set, and I am not 100% convinced that it is even correct to squelch task_work_add at that point in the code.

The progress I have made to date that I am sending to Linus for v5.18 is the removal of tracehook.h which makes the code much more understandable.

I think I have a general solution that I am planning to post after v5.18-rc1 that I have not tested yet on the cases that I know about, but I expect it will work.

So I think that puts a good general fix 2-3 weeks out.

This is quite possibly a case where perfection is getting in the way of the good, but I honestly can't judge anything except a fix that cleans up everything and is complete. There are too many weird and subtle interactions that I don't understand.

Quoted 5 lines

So I am going to continue concentrating on a good general solution so that the code is readable and makes sense.

Eric

Again continuing this thread from March 2022:

This problem has never been fixed; it is still present in 6.16-rc3.  To
recap: core dumps from a program using io_uring get truncated, especially if the core is piped to a program.  I have a forward-port of Jen's patch that fixes it, but apparently that patch did not pass muster with Linus back in 2021.  Any chance of getting this fixed properly upstream?

Reference:

https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/all/[email protected]/

Tony Battersby
Cybernetics

Tested-by on v1 1/1
#7 lore raw
In reply to Jens Axboe
Quoted 65 lines · Jens Axboe

task_work being added with notify == TWA_SIGNAL will utilize TIF_NOTIFY_SIGNAL for signaling the targeted task that work is available.
If this happens while a task is going through a core dump, it'll potentially disturb and truncate the dump as a signal interruption.

Have task_work_add() with notify == TWA_SIGNAL check if a task has been
signaled for a core dump, and refuse to add the work if that is the case.
When a core dump is invoked, explicitly check for TIF_NOTIFY_SIGNAL and
run any pending task_work if that is set. This is similar to how an exiting task will not get new task_work added, and we return the same error for the core dump case. As we return success or failure from task_work_add(), the caller has to be prepared to handle this case already.

Currently this manifests itself in that io_uring tasks that end up using
task_work will experience truncated core dumps.

---

fs/coredump.c +0 -0
index 07afb5ddb1c4..ca7c1ee44ada 100644
@@ -602,6 +602,14 @@ void do_coredump(const kernel_siginfo_t
 *siginfo)
                 .mm_flags = mm->flags,
         };


+       /*
+        * task_work_add() will refuse to add work after PF_SIGNALED has
+        * been set, ensure that we flush any pending TIF_NOTIFY_SIGNAL work
+        * if any was queued before that.
+        */
+       if (test_thread_flag(TIF_NOTIFY_SIGNAL))
+               tracehook_notify_signal();
+
        audit_core_dumps(siginfo->si_signo);

        binfmt = mm->binfmt;

kernel/task_work.c +2 -0
index 1698fbe6f0e1..1ab28904adc4 100644
@@ -41,6 +41,12 @@ int task_work_add(struct task_struct *task, struct
 callback_head *work,
                 head = READ_ONCE(task->task_works);
                 if (unlikely(head == &work_exited))
                         return -ESRCH;
+               /*
+                * TIF_NOTIFY_SIGNAL notifications will interfere
Quoted 9 lines

with
+                * a core dump in progress, reject them. +                */
+               if (notify == TWA_SIGNAL && (task->flags & PF_SIGNALED))
+                       return -ESRCH;
                work->next = head;
        } while (cmpxchg(&task->task_works, head, work) != head);

tested successfully on 5.12.19