exec: don't force_sigsegv processes with a pending fatal signal
Subsystems (2)
PATCH RESEND 1/1 exec: don't force_sigsegv processes with a pending fatal signal
We were seeing unexplained segfaults in coreutils processes and other basic utilities that we tracked down to binfmt_elf failing to load segments for ld.so. Digging further, the actual problem seems to occur when a process gets sigkilled while it is still being loaded by the kernel. In our case when _do_page_fault goes for a retry it will return early as it first checks for fatal_signal_pending(), so load_elf_interp also returns with error and as a result search_binary_handler will force_sigsegv() which is pretty confusing as nothing actually failed here.
Reference: https://lkml.org/lkml/2013/2/14/5
--- fs/exec.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/exec.c +2 -1
@@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)if (retval <0 && !bprm->mm) {/* we got to flush_old_exec() and failed after it */ read_unlock (&binfmt_lock); -force_sigsegv (SIGSEGV , current); +if (!fatal_signal_pending (current)) +force_sigsegv (SIGSEGV , current);return retval; }if (retval != -ENOEXEC || !bprm->file) {
Signature · 1 line
2.18.0
Hi Ivan,
2018-07-31 1:56 GMT+01:00 Ivan Delalande <[email protected]>:
Quoted 10 lines
We were seeing unexplained segfaults in coreutils processes and other basic utilities that we tracked down to
binfmt_elffailing to load segments for ld.so. Digging further, the actual problem seems to occur when a process gets sigkilled while it is still being loaded by the kernel. In our case when_do_page_faultgoes for a retry it will return early as it first checks forfatal_signal_pending(), soload_elf_interpalso returns with error and as a resultsearch_binary_handlerwillforce_sigsegv()which is pretty confusing as nothing actually failed here.
Quoted 3 lines
Fixes 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")Reference: https://lkml.org/lkml/2013/2/14/5
Signed-off-by Ivan Delalande <[email protected]>
+Cc: Oleg Nesterov <[email protected]>
+Cc: Andy Lutomirski <[email protected]>
Quoted 9 lines
--- fs/exec.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)fs/exec.c +0
-0
fs/exec.c int search_binary_handler(struct linux_binprm *bprm) · line 1660
Quoted 6 lines
fs/exec.c +2
-1@@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)if (retval <0 && !bprm->mm) {/* we got to flush_old_exec() and failed after it */ read_unlock (&binfmt_lock); -force_sigsegv (SIGSEGV , current); +if (!fatal_signal_pending (current)) +force_sigsegv (SIGSEGV , current);
I would suggest to add something like:
: if (print_fatal_signals)
: pr_info("load_binary() failed: %d\n", retval);
It was interesting to catch that it actually segfaults during loading, probably will save someone a couple of minutes too ;-)
fs/exec.c int search_binary_handler(struct linux_binprm *bprm) · line 1663
Quoted 3 lines
fs/exec.c +0
-0return retval; }if (retval != -ENOEXEC || !bprm->file) {
Thanks,
Dmitry
2018-08-02 20:53 GMT+01:00 Dmitry Safonov <[email protected]>:
Quoted 17 lines
Hi Ivan,
2018-07-31 1:56 GMT+01:00 Ivan Delalande <[email protected]>:
Quoted 13 lines
We were seeing unexplained segfaults in coreutils processes and other basic utilities that we tracked down to
binfmt_elffailing to load segments for ld.so. Digging further, the actual problem seems to occur when a process gets sigkilled while it is still being loaded by the kernel. In our case when_do_page_faultgoes for a retry it will return early as it first checks forfatal_signal_pending(), soload_elf_interpalso returns with error and as a resultsearch_binary_handlerwillforce_sigsegv()which is pretty confusing as nothing actually failed here.Fixes 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")Reference: https://lkml.org/lkml/2013/2/14/5
Signed-off-by Ivan Delalande <[email protected]>
Review
Quoted 2 lines
(quoted diff) +2
-0+Cc: Oleg Nesterov <[email protected]> +Cc: Andy Lutomirski <[email protected]>
Also worth to add to commit message an example of previously user-visible message in dmesg:
[ 1545.889604] potentially unexpected fatal signal 11. [ 1545.889614] CPU: 2 PID: 7462 Comm: grep Tainted: P O 3.18.28 #1 [ 1545.889617] Hardware name: Celestica D4040/D4040, BIOS 5.6.5 08/18/2016 [ 1545.889621] task: ffff880011282280 ti: ffff880100938000 task.ti: ffff880100938000 [ 1545.889624] RIP: 0023:[<00000000f760eb70>] [<00000000f760eb70>] 0xf760eb70 [ 1545.889641] RSP: 002b:00000000fffa3454 EFLAGS: 00000296 [ 1545.889644] RAX: fffffffffffffff2 RBX: 00000000f7a5c3e8 RCX: 00000000f7a5c718 [ 1545.889647] RDX: 00000000f7a5b230 RSI: 00000000f7a5c718 RDI: 00000000f757c000 [ 1545.889650] RBP: 00000000f7a5c3e8 R08: 0000000000000000 R09: 0000000000000000 [ 1545.889653] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 1545.889656] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1545.889659] FS: 0000000000000000(0000) GS:ffff88017fb00000(0000) knlGS:0000000000000000 [ 1545.889662] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b [ 1545.889665] CR2: 00000000f77d7838 CR3: 000000010bb90000 CR4: 00000000001007e0
(which now will be suppressed if there was a fatal signal)
Quoted 21 lines
Quoted 10 lines
--- fs/exec.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)fs/exec.c +0
-0fs/exec.c int search_binary_handler(struct linux_binprm *bprm) · line 1660
Quoted 6 lines
fs/exec.c +2
-1@@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)if (retval <0 && !bprm->mm) {/* we got to flush_old_exec() and failed after it */ read_unlock (&binfmt_lock); -force_sigsegv (SIGSEGV , current); +if (!fatal_signal_pending (current)) +force_sigsegv (SIGSEGV , current);I would suggest to add something like:
: if (print_fatal_signals)
:pr_info("load_binary() failed: %d\n", retval);
Quoted 2 lines
It was interesting to catch that it actually segfaults during loading, probably will save someone a couple of minutes too ;-)
Not sure if it's easy to trigger, but it might require a ratelimit too..
Quoted 4 lines
Quoted 4 lines
fs/exec.c +0
-0return retval; }if (retval != -ENOEXEC || !bprm->file) {
Signature · 1 line
Dmitry
Quoted 18 lines · Dmitry Safonov
Hi Ivan,
2018-07-31 1:56 GMT+01:00 Ivan Delalande <[email protected]>:
Quoted 13 lines
We were seeing unexplained segfaults in coreutils processes and other basic utilities that we tracked down to
binfmt_elffailing to load segments for ld.so. Digging further, the actual problem seems to occur when a process gets sigkilled while it is still being loaded by the kernel. In our case when_do_page_faultgoes for a retry it will return early as it first checks forfatal_signal_pending(), soload_elf_interpalso returns with error and as a resultsearch_binary_handlerwillforce_sigsegv()which is pretty confusing as nothing actually failed here.Fixes 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")Reference: https://lkml.org/lkml/2013/2/14/5
Signed-off-by Ivan Delalande <[email protected]>
Review
Quoted 2 lines
(quoted diff) +2
-0+Cc: Oleg Nesterov <[email protected]> +Cc: Andy Lutomirski <[email protected]>
Thanks...
and sorry, I fail to understand the problem and what/how this patch tries to fix.
Hmm. After I read the next email from Dmitry it seems to me that the whole purpose of this patch is to avoid print_fatal_signal()? If yes, the changelog should clearly explain this.
Quoted 9 lines
fs/exec.c int search_binary_handler(struct linux_binprm *bprm) · line 1660
Quoted 9 lines
fs/exec.c +2
-1@@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)if (retval <0 && !bprm->mm) {/* we got to flush_old_exec() and failed after it */ read_unlock (&binfmt_lock); -force_sigsegv (SIGSEGV , current); +if (!fatal_signal_pending (current)) +force_sigsegv (SIGSEGV , current);
I won't argue, but may be force_sigsegv() should check fatal_signal_pending() itself. setup_rt_frame() can too fail if fatal_signal_pending() by the same reason.
Oleg.
Hi,
Quoted 21 lines · Oleg Nesterov
Quoted 15 lines · Dmitry Safonov
2018-07-31 1:56 GMT+01:00 Ivan Delalande <[email protected]>:
Quoted 13 lines
We were seeing unexplained segfaults in coreutils processes and other basic utilities that we tracked down to
binfmt_elffailing to load segments for ld.so. Digging further, the actual problem seems to occur when a process gets sigkilled while it is still being loaded by the kernel. In our case when_do_page_faultgoes for a retry it will return early as it first checks forfatal_signal_pending(), soload_elf_interpalso returns with error and as a resultsearch_binary_handlerwillforce_sigsegv()which is pretty confusing as nothing actually failed here.Fixes 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")Reference: https://lkml.org/lkml/2013/2/14/5
Signed-off-by Ivan Delalande <[email protected]>Review
Quoted 2 lines
(quoted diff) +2
-0+Cc: Oleg Nesterov <[email protected]> +Cc: Andy Lutomirski <[email protected]>Thanks...
Quoted 5 lines
and sorry, I fail to understand the problem and what/how this patch tries to fix.
Hmm. After I read the next email from Dmitry it seems to me that the whole purpose of this patch is to avoid
print_fatal_signal()? If yes, the changelog should clearly explain this.
Sorry about that, yes this is purely to avoid printing the segfault messages for these processes when they were in fact killed. I'll definitely send a v2 to clarify that, and probably add the helpful message Dimitry suggested as well.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
fs/exec.c +0
-0@@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)if (retval <0 && !bprm->mm) {/* we got to flush_old_exec() and failed after it */
Quoted 8 lines
Quoted 4 lines
fs/exec.c int search_binary_handler(struct linux_binprm *bprm) · line 1660
Quoted 4 lines
fs/exec.c +2
-1read_unlock (&binfmt_lock); -force_sigsegv (SIGSEGV , current); +if (!fatal_signal_pending (current)) +force_sigsegv (SIGSEGV , current);I won't argue, but may be
force_sigsegv()should checkfatal_signal_pending()itself.setup_rt_frame()can too fail iffatal_signal_pending()by the same reason.
I'm not sure, I think it would feel out of place in force_sigsegv() as other callers might not expect this check in different contexts. I could add a similar call to fatal_signal_pending() in signal_setup_done() though, if you think we can hit the same problem from setup_rt_frame().
Thanks,
Signature · 2 lines
Ivan Delalande Arista Networks