exec: Fix a deadlock in ptrace
Subsystems (17)
PATCH 1/1 exec: Fix a deadlock in ptrace
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in vm_access.
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the PTRACE_EVENT_EXIT is handled and the WIFEXITED event is received:
strace D 0 30614 30584 0x00000000
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
expect D 0 31933 30876 0x80004003
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
The proposed solution is to have a second mutex that is used in mm_access, so it is allowed to continue while the dying threads are not yet terminated.
I also took the opportunity to improve the documentation of prepare_creds, which is obviously out of sync.
--- Documentation/security/credentials.rst | 18 ++++++++++-------- fs/exec.c | 9 +++++++++ include/linux/binfmts.h | 6 +++++- include/linux/sched/signal.h | 1 + init/init_task.c | 1 + kernel/cred.c | 2 +- kernel/fork.c | 5 +++-- mm/process_vm_access.c | 2 +- 8 files changed, 31 insertions(+), 13 deletions(-)
Documentation/security/credentials.rst +10 -8
@@ -437,9 +437,13 @@ new set of credentials by calling:: struct cred *prepare_creds(void); -this locks current->cred_replace_mutex and then allocates and constructs a -duplicate of the current process's credentials, returning with the mutex still -held if successful. It returns NULL if not successful (out of memory). +this allocates and constructs a duplicate of the current process's credentials. +It returns NULL if not successful (out of memory). + +If called from __do_execve_file, the mutex current->signal->cred_guard_mutex +is acquired before this function gets called, and the mutex +current->signal->cred_change_mutex is acquired later, while the credentials +and the process mmap are actually changed. The mutex prevents ``ptrace()`` from altering the ptrace state of a process while security checks on credentials construction and changing is taking place
@@ -466,9 +470,8 @@ by calling:: This will alter various aspects of the credentials and the process, giving the LSM a chance to do likewise, then it will use ``rcu_assign_pointer()`` to -actually commit the new credentials to ``current->cred``, it will release -``current->cred_replace_mutex`` to allow ``ptrace()`` to take place, and it -will notify the scheduler and others of the changes. +actually commit the new credentials to ``current->cred``, and it will notify +the scheduler and others of the changes. This function is guaranteed to return 0, so that it can be tail-called at the end of such functions as ``sys_setresuid()``.
@@ -486,8 +489,7 @@ invoked:: void abort_creds(struct cred *new); -This releases the lock on ``current->cred_replace_mutex`` that -``prepare_creds()`` got and then releases the new credentials. +This releases the new credentials. A typical credentials alteration function would look something like this::
fs/exec.c +9 -0
@@ -1266,6 +1266,12 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; + retval =mutex_lock_killable (¤t->signal->cred_change_mutex); +if (retval) + goto out; + + bprm->called_flush_old_exec =1 ; +/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update
@@ -1420,6 +1426,8 @@ static void free_bprm(struct linux_binprm *bprm) {free_arg_pages (bprm);if (bprm->cred) { +if (bprm->called_flush_old_exec) +mutex_unlock (¤t->signal->cred_change_mutex);mutex_unlock (¤t->signal->cred_guard_mutex);abort_creds (bprm->cred); }
@@ -1469,6 +1477,7 @@ void install_exec_creds(struct linux_binprm *bprm)* credentials; any time after this it may be unlocked. */ security_bprm_committed_creds (bprm); +mutex_unlock (¤t->signal->cred_change_mutex);mutex_unlock (¤t->signal->cred_guard_mutex); }EXPORT_SYMBOL (install_exec_creds);
include/linux/binfmts.h +5 -1
@@ -44,7 +44,11 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1 , +/* +* Set by flush_old_exec, when the cred_change_mutex is taken. +*/ + called_flush_old_exec:1;#ifdef __alpha__ unsigned int taso:1;#endif
include/linux/sched/signal.h +1 -0
@@ -225,6 +225,7 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations * (notably. ptrace) */ +struct mutex cred_change_mutex;/* guard against credentials change */ } __randomize_layout;/*
init/init_task.c +1 -0
@@ -26,6 +26,7 @@ .multiprocess = HLIST_HEAD_INIT, .rlim = INIT_RLIMITS, .cred_guard_mutex = __MUTEX_INITIALIZER(init_signals.cred_guard_mutex), + .cred_change_mutex =__MUTEX_INITIALIZER (init_signals.cred_change_mutex), #ifdef CONFIG_POSIX_TIMERS .posix_timers =LIST_HEAD_INIT (init_signals.posix_timers), .cputimer = {
kernel/cred.c +1 -1
@@ -676,7 +676,7 @@ void __init cred_init(void)* * Returns the new credentials or NULL if out of memory. * -* Does not take, and does not return holding current->cred_replace_mutex. +* Does not take, and does not return holding ->cred_guard_mutex. */ struct cred *prepare_kernel_cred (struct task_struct *daemon) {
kernel/fork.c +3 -2
@@ -1224,7 +1224,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)struct mm_struct *mm;int err; - err =mutex_lock_killable (&task->signal->cred_guard_mutex); + err =mutex_lock_killable (&task->signal->cred_change_mutex);if (err)return ERR_PTR (err);
@@ -1234,7 +1234,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)mmput (mm); mm =ERR_PTR (-EACCES ); } -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->cred_change_mutex);return mm; }
@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); +mutex_init (&sig->cred_change_mutex);return 0 ; }
mm/process_vm_access.c +1 -1
@@ -204,7 +204,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter,if (!mm ||IS_ERR (mm)) { rc =IS_ERR (mm) ?PTR_ERR (mm) : -ESRCH ;/* -* Explicitly map EACCES to EPERM as EPERM is a more a +* Explicitly map EACCES to EPERM as EPERM is a more * appropriate error code for process_vw_readv/writev */ if (rc == -EACCES )
Signature · 1 line
1.9.1
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 8 lines
The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.I also took the opportunity to improve the documentation of
prepare_creds, which is obviously out of sync.Signed-off-by Bernd Edlinger <[email protected]>
I can't comment on the validity of the patch, but I also found and reported this issue in 2016[1] and the discussion quickly veered into the problem being more complicated (and uglier) than it seems at first glance.
You should probably also Cc stable, given this has been a long-standing issue and your patch doesn't look (too) invasive.
[1]: https://lore.kernel.org/lkml/[email protected]/
Signature · 4 lines
Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH <https://www.cyphar.com/>
Quoted 55 lines · Aleksa Sarai
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9Quoted 8 lines
The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.I also took the opportunity to improve the documentation of
prepare_creds, which is obviously out of sync.Signed-off-by Bernd Edlinger <[email protected]>I can't comment on the validity of the patch, but I also found and reported this issue in 2016[1] and the discussion quickly veered into the problem being more complicated (and uglier) than it seems at first glance.
Quoted 4 lines
You should probably also Cc stable, given this has been a long-standing issue and your patch doesn't look (too) invasive.
Yeah, I remember you mentioning this a while back.
Bernd, we really want a reproducer for this sent alongside with this patch added to:tools/testing/selftests/ptrace/
Having a test for this bug irrespective of whether or not we go with this as fix seems really worth it.
Oleg seems to have suggested that a potential alternative fix is to wait in de_thread() until all other threads in the thread-group have passed exit_notiy(). Right now we only kill them but don't wait. Currently de_thread() only waits for the thread-group leader to pass exit_notify() whenever a non-thread-group leader thread execs (because the exec'ing thread becomes the new thread-group leader with the same pid as the former thread-group leader).
Christian
Quoted 61 lines · Christian Brauner
Quoted 55 lines · Aleksa Sarai
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9Quoted 8 lines
The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.I also took the opportunity to improve the documentation of
prepare_creds, which is obviously out of sync.Signed-off-by Bernd Edlinger <[email protected]>I can't comment on the validity of the patch, but I also found and reported this issue in 2016[1] and the discussion quickly veered into the problem being more complicated (and uglier) than it seems at first glance.
Quoted 4 lines
You should probably also Cc stable, given this has been a long-standing issue and your patch doesn't look (too) invasive.
Quoted 8 lines
Yeah, I remember you mentioning this a while back.
Bernd, we really want a reproducer for this sent alongside with this patch added to:
tools/testing/selftests/ptrace/
Having a test for this bug irrespective of whether or not we go with this as fix seems really worth it.
I ran into this issue, because I wanted to fix an issue in the gcc testsuite, namely why it forgets to remove some temp files,
so I did the following:
strace -ftt -o trace.txt make check-gcc-c -k -j4
I reproduced with v4.20 and v5.5 kernel, and I don't know why but it is not happening on all systems I tested, maybe it is something that the expect program does, because, always when I try to reproduce this, the deadlock was always in "expect".
I use expect version 5.45 on the computer where the above test freezes after a couple of minutes.
I think the issue with strace is that it is using vm_access to get the parameters of a syscall that is going on in one thread, and that races with another thread that calls execve, and blocks the cred_guard_mutex.
While Olg's test case here, will certainly not be fixed:
https://lore.kernel.org/lkml/[email protected]/
he mentions the access to "anything else which needs ->cred_guard_mutex, say open(/proc/$pid/mem)", I don't know for sure how that can be done, but if that is possible, it would probably work as a test case.
What do you think?
Bernd.
Quoted 10 lines
Oleg seems to have suggested that a potential alternative fix is to wait in
de_thread()until all other threads in the thread-group have passedexit_notiy(). Right now we only kill them but don't wait. Currentlyde_thread()only waits for the thread-group leader to passexit_notify()whenever a non-thread-group leader thread execs (because the exec'ing thread becomes the new thread-group leader with the same pid as the former thread-group leader).Christian
Quoted 90 lines · Bernd Edlinger
Quoted 61 lines · Christian Brauner
Quoted 55 lines · Aleksa Sarai
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9Quoted 8 lines
The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.I also took the opportunity to improve the documentation of
prepare_creds, which is obviously out of sync.Signed-off-by Bernd Edlinger <[email protected]>I can't comment on the validity of the patch, but I also found and reported this issue in 2016[1] and the discussion quickly veered into the problem being more complicated (and uglier) than it seems at first glance.
Quoted 4 lines
You should probably also Cc stable, given this has been a long-standing issue and your patch doesn't look (too) invasive.
Quoted 8 lines
Yeah, I remember you mentioning this a while back.
Bernd, we really want a reproducer for this sent alongside with this patch added to:
tools/testing/selftests/ptrace/
Having a test for this bug irrespective of whether or not we go with this as fix seems really worth it.I ran into this issue, because I wanted to fix an issue in the gcc testsuite, namely why it forgets to remove some temp files,
so I did the following:strace -ftt -o trace.txt make check-gcc-c -k -j4
I reproduced with v4.20 and v5.5 kernel, and I don't know why but it is not happening on all systems I tested, maybe it is something that the expect program does, because, always when I try to reproduce this, the deadlock was always in "expect".
I use expect version 5.45 on the computer where the above test freezes after a couple of minutes.
I think the issue with strace is that it is using
vm_accessto get the parameters of a syscall that is going on in one thread, and that races with another thread that calls execve, and blocks thecred_guard_mutex.While Olg's test case here, will certainly not be fixed:
Quoted 7 lines
https://lore.kernel.org/lkml/[email protected]/
he mentions the access to "anything else which needs ->cred_guard_mutex, say open(/proc/$pid/mem)", I don't know for sure how that can be done, but if that is possible, it would probably work as a test case.
What do you think?
Yeah, anything that calls ptrace_may_access() is fine and open(/proc/$pid/mem) will work so long as $pid is not in the same thread-group as the caller. A polished version of the reproducer you linked in would probably be good.
Hi Aleksa,
Quoted 50 lines · Aleksa Sarai
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9Quoted 8 lines
The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.I also took the opportunity to improve the documentation of
prepare_creds, which is obviously out of sync.Signed-off-by Bernd Edlinger <[email protected]>
Quoted 8 lines
I can't comment on the validity of the patch, but I also found and reported this issue in 2016[1] and the discussion quickly veered into the problem being more complicated (and uglier) than it seems at first glance.
You should probably also Cc stable, given this has been a long-standing issue and your patch doesn't look (too) invasive.
I am fully aware that this patch won't fix the case then PTRACE_ACCESS is racing with de_thread. But I don't see a problem with allowing vm access based on the current credentials as they are still the same until de_thread is done with it's job. And in a practical way this fixes 99% of the real problem here, as it only happens since strace is currently tracing something and needs access to the parameters in the tracee's vm space.
Of course you could fork the strace process to do any PTRACE_ACCESS when necessary, and, well, maybe that would fix the remaining problem here...
However before I considered changing the kernel for this I tried to fix this within strace. First I tried to wait in the signal handler. See attached strace-patch-1.diff, but that did not work, BUT I think it is possible that your patch you proposed previously would actually make it work.
I tried then another approach, using a worker thread to wait for the childs, but it did only work when I remove PTRACE_O_TRACEEXIT from the ptrace options, because the ptrace(PTRACE_SYSCALL, pid, 0L, 0L) does not work in the worker thread, rv = -1, errno = 3 there, and unfortunately the main thread is blocked and unable to do the ptrace call, that makes the thread continue.
So I consider that second patch really ugly, and wouldn't propose something like that seriously.
@@ -69,7 +71,7 @@
cflag_t cflag = CFLAG_NONE;
unsigned int followfork;
unsigned int ptrace_setoptions = PTRACE_O_TRACESYSGOOD | PTRACE_O_TRACEEXEC
- | PTRACE_O_TRACEEXIT;
+ ;//| PTRACE_O_TRACEEXIT;
unsigned int xflag;
bool debug_flag;
bool Tflag;
so it only works because of this line, without that it is not able to make the thread continue after the PTRACE_EVENT_EXIT.
Thanks
Bernd.
Quoted 2 lines
delay.h +1 -0
@@ -14,5 +14,6 @@void delay_timer_expired (void );void arm_delay_timer (const struct tcb *);void delay_tcb (struct tcb *,uint16_t delay_idx,bool isenter); +int my_waitpid (int ,int *,int );#endif /* !STRACE_DELAY_H */
filter_seccomp.c +3 -2
@@ -19,6 +19,7 @@#include "number_set.h" #include "syscall.h" #include "scno.h" +#include "delay.h" bool seccomp_filtering;bool seccomp_before_sysentry;
@@ -136,7 +137,7 @@int status;for (;;) { -long rc =waitpid (pid, &status,0 ); +long rc =my_waitpid (pid, &status,0 );if (rc <0 && errno ==EINTR )continue ;if (rc == pid)
@@ -272,7 +273,7 @@if (pid) {kill (pid,SIGKILL );for (;;) { -long rc =waitpid (pid,NULL ,0 ); +long rc =my_waitpid (pid,NULL ,0 );if (rc <0 && errno ==EINTR )continue ;break ;
Makefile.am +1 -1
@@ -45,7 +45,7 @@ strace_CPPFLAGS = $(AM_CPPFLAGS) strace_CFLAGS = $(AM_CFLAGS) strace_LDFLAGS = -strace_LDADD = libstrace.a $(clock_LIBS) $(timer_LIBS) +strace_LDADD = libstrace.a -lpthread $(clock_LIBS) $(timer_LIBS) noinst_LIBRARIES = libstrace.a libstrace_a_CPPFLAGS = $(strace_CPPFLAGS)
Makefile.in +1 -1
@@ -1631,7 +1631,7 @@ $(am__append_11) $(CODE_COVERAGE_CPPFLAGS) strace_CFLAGS = $(AM_CFLAGS) $(am__append_4) $(CODE_COVERAGE_CFLAGS) strace_LDFLAGS = $(am__append_5) $(am__append_9) $(am__append_12) -strace_LDADD = libstrace.a $(clock_LIBS) $(timer_LIBS) $(am__append_6) \ +strace_LDADD = libstrace.a -lpthread $(clock_LIBS) $(timer_LIBS) $(am__append_6) \ $(am__append_10) $(am__append_13) $(CODE_COVERAGE_LIBS) \ $(am__append_14) $(am__append_18) noinst_LIBRARIES = libstrace.a $(am__append_15) $(am__append_19)
ptrace_syscall_info.c +3 -2
@@ -12,6 +12,7 @@#include "ptrace.h" #include "ptrace_syscall_info.h" #include "scno.h" +#include "delay.h" #include <signal.h> #include <sys/wait.h>
@@ -118,7 +119,7 @@ };const size_t size =sizeof (info);int status; -long rc =waitpid (pid, &status,0 ); +long rc =my_waitpid (pid, &status,0 );if (rc != pid) {/* cannot happen */ kill_tracee (pid);
@@ -247,7 +248,7 @@ done:if (pid) {kill_tracee (pid); -waitpid (pid,NULL ,0 ); +my_waitpid (pid,NULL ,0 ); ptrace_stop =-1U ; }
strace.c +103 -10
168 lines not on this page · raw message
delay.h +1 -0
@@ -14,5 +14,6 @@void delay_timer_expired (void );void arm_delay_timer (const struct tcb *);void delay_tcb (struct tcb *,uint16_t delay_idx,bool isenter); +int my_waitpid (int ,int *,int );#endif /* !STRACE_DELAY_H */
filter_seccomp.c +3 -2
@@ -19,6 +19,7 @@#include "number_set.h" #include "syscall.h" #include "scno.h" +#include "delay.h" bool seccomp_filtering;bool seccomp_before_sysentry;
@@ -136,7 +137,7 @@int status;for (;;) { -long rc =waitpid (pid, &status,0 ); +long rc =my_waitpid (pid, &status,0 );if (rc <0 && errno ==EINTR )continue ;if (rc == pid)
@@ -272,7 +273,7 @@if (pid) {kill (pid,SIGKILL );for (;;) { -long rc =waitpid (pid,NULL ,0 ); +long rc =my_waitpid (pid,NULL ,0 );if (rc <0 && errno ==EINTR )continue ;break ;
Makefile.am +1 -1
@@ -45,7 +45,7 @@ strace_CPPFLAGS = $(AM_CPPFLAGS) strace_CFLAGS = $(AM_CFLAGS) strace_LDFLAGS = -strace_LDADD = libstrace.a $(clock_LIBS) $(timer_LIBS) +strace_LDADD = libstrace.a -lpthread $(clock_LIBS) $(timer_LIBS) noinst_LIBRARIES = libstrace.a libstrace_a_CPPFLAGS = $(strace_CPPFLAGS)
Makefile.in +1 -1
@@ -1631,7 +1631,7 @@ $(am__append_11) $(CODE_COVERAGE_CPPFLAGS) strace_CFLAGS = $(AM_CFLAGS) $(am__append_4) $(CODE_COVERAGE_CFLAGS) strace_LDFLAGS = $(am__append_5) $(am__append_9) $(am__append_12) -strace_LDADD = libstrace.a $(clock_LIBS) $(timer_LIBS) $(am__append_6) \ +strace_LDADD = libstrace.a -lpthread $(clock_LIBS) $(timer_LIBS) $(am__append_6) \ $(am__append_10) $(am__append_13) $(CODE_COVERAGE_LIBS) \ $(am__append_14) $(am__append_18) noinst_LIBRARIES = libstrace.a $(am__append_15) $(am__append_19)
strace.c +129 -9
201 lines not on this page · raw message
Quoted 3 lines · Bernd Edlinger
The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.
Just for context: When I proposed something similar back in 2016,
https://lore.kernel.org/linux-fsdevel/[email protected]/
was the resulting discussion thread. At least back then, I looked through the various existing users of cred_guard_mutex, and the only places that couldn't be converted to the new second mutex were PTRACE_ATTACH and SECCOMP_FILTER_FLAG_TSYNC.
The ideal solution would IMO be something like this: Decide what the new task's credentials should be *before* reaching de_thread(), install them into a second cred* on the task (together with the new dumpability), drop the cred_guard_mutex, and let ptrace_may_access() check against both. After that, some further restructuring might even allow the cred_guard_mutex to not be held across all of the VFS operations that happen early on in execve, which may block indefinitely. But that would be pretty complicated, so I think your proposed solution makes sense for now, given that nobody has managed to implement anything better in the last few years.
Quoted 14 lines · Jann Horn
Quoted 3 lines · Bernd Edlinger
The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.Just for context: When I proposed something similar back in 2016,
https://lore.kernel.org/linux-fsdevel/[email protected]/
was the resulting discussion thread. At least back then, I looked through the various existing users ofcred_guard_mutex, and the only places that couldn't be converted to the new second mutex werePTRACE_ATTACHandSECCOMP_FILTER_FLAG_TSYNC.
Quoted 5 lines
The ideal solution would IMO be something like this: Decide what the new task's credentials should be *before* reaching
de_thread(), install them into a second cred* on the task (together with the new dumpability), drop thecred_guard_mutex, and letptrace_may_access()check against both. After that, some further restructuring might even
Hm, so essentially a private ptrace_access_cred member in task_struct? That would presumably also involve altering various LSM hooks to look at ptrace_access_cred.
(Minor side-note, de_thread() takes a struct task_struct argument but
only ever is passed current.)
Quoted 5 lines
allow the
cred_guard_mutexto not be held across all of the VFS operations that happen early on in execve, which may block indefinitely. But that would be pretty complicated, so I think your proposed solution makes sense for now, given that nobody has managed to implement anything better in the last few years.
Reading through the old threads and how often this issue came up, I tend to agree.
Quoted 28 lines · Christian Brauner
Quoted 14 lines · Jann Horn
Quoted 3 lines · Bernd Edlinger
The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.Just for context: When I proposed something similar back in 2016,
https://lore.kernel.org/linux-fsdevel/[email protected]/
was the resulting discussion thread. At least back then, I looked through the various existing users ofcred_guard_mutex, and the only places that couldn't be converted to the new second mutex werePTRACE_ATTACHandSECCOMP_FILTER_FLAG_TSYNC.Quoted 5 lines
The ideal solution would IMO be something like this: Decide what the new task's credentials should be *before* reaching
de_thread(), install them into a second cred* on the task (together with the new dumpability), drop thecred_guard_mutex, and letptrace_may_access()check against both. After that, some further restructuring might evenHm, so essentially a private
ptrace_access_credmember intask_struct? That would presumably also involve altering various LSM hooks to look atptrace_access_cred.(Minor side-note,
de_thread()takes a structtask_structargument but
only ever is passed current.)
Quoted 9 lines
Quoted 5 lines
allow the
cred_guard_mutexto not be held across all of the VFS operations that happen early on in execve, which may block indefinitely. But that would be pretty complicated, so I think your proposed solution makes sense for now, given that nobody has managed to implement anything better in the last few years.Reading through the old threads and how often this issue came up, I tend to agree.
Okay, fine.
I managed to change Oleg's test case, into one that shows what exactly is changed with this patch:
$ cat t.c #include <stdio.h> #include <fcntl.h> #include <unistd.h> #include <pthread.h> #include <sys/signal.h> #include <sys/ptrace.h>
void *thread(void *arg)
{
ptrace(PTRACE_TRACEME, 0,0,0);
return NULL;
}int main(void)
{
int f, pid = fork();
char mm[64];
if (!pid) {
pthread_t pt;
pthread_create(&pt, NULL, thread, NULL);
pthread_join(pt, NULL);
execlp("echo", "echo", "passed", NULL);
}
sleep(1);
sprintf(mm, "/proc/%d/mem", pid);
printf("open(%s)\n", mm);
f = open(mm, O_RDONLY);
printf("f = %d\n", f);
// this is not fixed! ptrace(PTRACE_ATTACH, pid, 0,0);
kill(pid, SIGCONT);
if (f >= 0)
close(f);
return 0;
}
$ gcc -pthread -Wall t.c
$ ./a.out
open(/proc/2802/mem)
f = 3
$ passedpreviously this did block, how can I make a test case for this? I am not so experienced in this matter.
Thanks
Bernd.
Quoted 15 lines · Christian Brauner
Quoted 14 lines · Jann Horn
Quoted 3 lines · Bernd Edlinger
The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.Just for context: When I proposed something similar back in 2016,
https://lore.kernel.org/linux-fsdevel/[email protected]/
was the resulting discussion thread. At least back then, I looked through the various existing users ofcred_guard_mutex, and the only places that couldn't be converted to the new second mutex werePTRACE_ATTACHandSECCOMP_FILTER_FLAG_TSYNC.
Quoted 7 lines
Quoted 5 lines
The ideal solution would IMO be something like this: Decide what the new task's credentials should be *before* reaching
de_thread(), install them into a second cred* on the task (together with the new dumpability), drop thecred_guard_mutex, and letptrace_may_access()check against both. After that, some further restructuring might evenHm, so essentially a private
ptrace_access_credmember intask_struct?
And a second dumpability field, because that changes together with the creds during execve. (Btw, currently the dumpability is in the mm_struct, but that's kinda wrong. The mm_struct is removed from a task on exit while access checks can still be performed against it, and currently ptrace_may_access() just lets the access go through in that case, which weakens the protection offered by PR_SET_DUMPABLE when used for security purposes. I think it ought to be moved over into the task_struct.)
Quoted 2 lines
That would presumably also involve altering various LSM hooks to look at
ptrace_access_cred.
When I tried to implement this in the past, I changed the LSM hook to take the target task's cred* as an argument, and then called the LSM hook twice from ptrace_may_access(). IIRC having the target task's creds as an argument works for almost all the LSMs, with the exception of Yama, which doesn't really care about the target task's creds, so you have to pass in both the task_struct* and the cred*.
PATCH v2 1/1 exec: Fix a deadlock in ptrace
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in vm_access.
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the PTRACE_EVENT_EXIT is handled and the WIFEXITED event is received:
strace D 0 30614 30584 0x00000000
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
expect D 0 31933 30876 0x80004003
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
The proposed solution is to have a second mutex that is used in mm_access, so it is allowed to continue while the dying threads are not yet terminated.
I also took the opportunity to improve the documentation of prepare_creds, which is obviously out of sync.
--- Documentation/security/credentials.rst | 18 ++++++------ fs/exec.c | 9 ++++++ include/linux/binfmts.h | 6 +++- include/linux/sched/signal.h | 1 + init/init_task.c | 1 + kernel/cred.c | 2 +- kernel/fork.c | 5 ++-- mm/process_vm_access.c | 2 +- tools/testing/selftests/ptrace/Makefile | 4 +-- tools/testing/selftests/ptrace/vmaccess.c | 46 +++++++++++++++++++++++++++++++ 10 files changed, 79 insertions(+), 15 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.c
v2: adds a test case which passes when this patch is applied.
Documentation/security/credentials.rst +10 -8
@@ -437,9 +437,13 @@ new set of credentials by calling:: struct cred *prepare_creds(void); -this locks current->cred_replace_mutex and then allocates and constructs a -duplicate of the current process's credentials, returning with the mutex still -held if successful. It returns NULL if not successful (out of memory). +this allocates and constructs a duplicate of the current process's credentials. +It returns NULL if not successful (out of memory). + +If called from __do_execve_file, the mutex current->signal->cred_guard_mutex +is acquired before this function gets called, and the mutex +current->signal->cred_change_mutex is acquired later, while the credentials +and the process mmap are actually changed. The mutex prevents ``ptrace()`` from altering the ptrace state of a process while security checks on credentials construction and changing is taking place
@@ -466,9 +470,8 @@ by calling:: This will alter various aspects of the credentials and the process, giving the LSM a chance to do likewise, then it will use ``rcu_assign_pointer()`` to -actually commit the new credentials to ``current->cred``, it will release -``current->cred_replace_mutex`` to allow ``ptrace()`` to take place, and it -will notify the scheduler and others of the changes. +actually commit the new credentials to ``current->cred``, and it will notify +the scheduler and others of the changes. This function is guaranteed to return 0, so that it can be tail-called at the end of such functions as ``sys_setresuid()``.
@@ -486,8 +489,7 @@ invoked:: void abort_creds(struct cred *new); -This releases the lock on ``current->cred_replace_mutex`` that -``prepare_creds()`` got and then releases the new credentials. +This releases the new credentials. A typical credentials alteration function would look something like this::
fs/exec.c +9 -0
@@ -1266,6 +1266,12 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; + retval =mutex_lock_killable (¤t->signal->cred_change_mutex); +if (retval) + goto out; + + bprm->called_flush_old_exec =1 ; +/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update
@@ -1420,6 +1426,8 @@ static void free_bprm(struct linux_binprm *bprm) {free_arg_pages (bprm);if (bprm->cred) { +if (bprm->called_flush_old_exec) +mutex_unlock (¤t->signal->cred_change_mutex);mutex_unlock (¤t->signal->cred_guard_mutex);abort_creds (bprm->cred); }
@@ -1469,6 +1477,7 @@ void install_exec_creds(struct linux_binprm *bprm)* credentials; any time after this it may be unlocked. */ security_bprm_committed_creds (bprm); +mutex_unlock (¤t->signal->cred_change_mutex);mutex_unlock (¤t->signal->cred_guard_mutex); }EXPORT_SYMBOL (install_exec_creds);
include/linux/binfmts.h +5 -1
@@ -44,7 +44,11 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1 , +/* +* Set by flush_old_exec, when the cred_change_mutex is taken. +*/ + called_flush_old_exec:1;#ifdef __alpha__ unsigned int taso:1;#endif
include/linux/sched/signal.h +1 -0
@@ -225,6 +225,7 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations * (notably. ptrace) */ +struct mutex cred_change_mutex;/* guard against credentials change */ } __randomize_layout;/*
init/init_task.c +1 -0
@@ -26,6 +26,7 @@ .multiprocess = HLIST_HEAD_INIT, .rlim = INIT_RLIMITS, .cred_guard_mutex = __MUTEX_INITIALIZER(init_signals.cred_guard_mutex), + .cred_change_mutex =__MUTEX_INITIALIZER (init_signals.cred_change_mutex), #ifdef CONFIG_POSIX_TIMERS .posix_timers =LIST_HEAD_INIT (init_signals.posix_timers), .cputimer = {
kernel/cred.c +1 -1
@@ -676,7 +676,7 @@ void __init cred_init(void)* * Returns the new credentials or NULL if out of memory. * -* Does not take, and does not return holding current->cred_replace_mutex. +* Does not take, and does not return holding ->cred_guard_mutex. */ struct cred *prepare_kernel_cred (struct task_struct *daemon) {
kernel/fork.c +3 -2
@@ -1224,7 +1224,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)struct mm_struct *mm;int err; - err =mutex_lock_killable (&task->signal->cred_guard_mutex); + err =mutex_lock_killable (&task->signal->cred_change_mutex);if (err)return ERR_PTR (err);
@@ -1234,7 +1234,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)mmput (mm); mm =ERR_PTR (-EACCES ); } -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->cred_change_mutex);return mm; }
@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); +mutex_init (&sig->cred_change_mutex);return 0 ; }
mm/process_vm_access.c +1 -1
@@ -204,7 +204,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter,if (!mm ||IS_ERR (mm)) { rc =IS_ERR (mm) ?PTR_ERR (mm) : -ESRCH ;/* -* Explicitly map EACCES to EPERM as EPERM is a more a +* Explicitly map EACCES to EPERM as EPERM is a more * appropriate error code for process_vw_readv/writev */ if (rc == -EACCES )
tools/testing/selftests/ptrace/Makefile +2 -2
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -iquote../../../../include/uapi -Wall +CFLAGS += -std=c99 -pthread -iquote../../../../include/uapi -Wall -TEST_GEN_PROGS := get_syscall_info peeksiginfo +TEST_GEN_PROGS := get_syscall_info peeksiginfo vmaccess include ../lib.mk
tools/testing/selftests/ptrace/vmaccess.c new file +46 -0
46 lines not on this page · raw message
Signature · 1 line
1.9.1
Quoted 8 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.
Quoted 4 lines
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:
I think your patch works, but I don't think to solve your case another mutex is necessary. Possibly it is justified, but I hesitate to introduce yet another concept in the code.
Having read elsewhere in the thread that this does not solve the problem Oleg has mentioned I am really hesitant to add more complexity to the situation.
For your case there is a straight forward and local workaround.
When the current task is ptracing the target task don't bother with cred_gaurd_mutex and ptrace_may_access in access_mm as those tests have already passed. Instead just confirm the ptrace status. AKA the permission check in ptraces_access_vm.
I think something like this is all we need.
kernel/fork.c +10 -0
@@ -1224,6 +1224,16 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)struct mm_struct *mm;int err; +if (task->ptrace && (current == task->parent)) { + mm =get_task_mm (task); +if ((get_dumpable (mm) !=SUID_DUMP_USER ) && + !ptracer_capable (task, mm->user_ns)) { +mmput (mm); + mm =ERR_PTR (-EACCESS ); + } +return mm; + } + err =mutex_lock_killable (&task->signal->cred_guard_mutex);if (err)return ERR_PTR (err);
Does this solve your test case?
The patch above is short the approriate locking for the ptrace attached check. (tasklist_lock I think). But is enough to illustrate the idea, and it is probably a check we want in any event so that if the tracer starts dropping privileges process_vm_readv and process_vm_writev will still be usable by the tracer.
Eric
Quoted 291 lines
strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9The proposed solution is to have a second mutex that is used in
mm_access, so it is allowed to continue while the dying threads are not yet terminated.I also took the opportunity to improve the documentation of
prepare_creds, which is obviously out of sync.Signed-off-by Bernd Edlinger <[email protected]>--- Documentation/security/credentials.rst | 18 ++++++------ fs/exec.c | 9 ++++++ include/linux/binfmts.h | 6 +++- include/linux/sched/signal.h | 1 + init/init_task.c | 1 + kernel/cred.c | 2 +- kernel/fork.c | 5 ++-- mm/process_vm_access.c | 2 +- tools/testing/selftests/ptrace/Makefile | 4 +-- tools/testing/selftests/ptrace/vmaccess.c | 46 +++++++++++++++++++++++++++++++ 10 files changed, 79 insertions(+), 15 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.cv2: adds a test case which passes when this patch is applied.
Documentation/security/credentials.rst +10
-8@@ -437,9 +437,13 @@ new set of credentials by calling:: struct cred *prepare_creds(void); -this locks current->cred_replace_mutex and then allocates and constructs a -duplicate of the current process's credentials, returning with the mutex still -held if successful. It returns NULL if not successful (out of memory). +this allocates and constructs a duplicate of the current process's credentials. +It returns NULL if not successful (out of memory). + +If called from __do_execve_file, the mutex current->signal->cred_guard_mutex +is acquired before this function gets called, and the mutex +current->signal->cred_change_mutex is acquired later, while the credentials +and the process mmap are actually changed. The mutex prevents ``ptrace()`` from altering the ptrace state of a process while security checks on credentials construction and changing is taking place@@ -466,9 +470,8 @@ by calling:: This will alter various aspects of the credentials and the process, giving the LSM a chance to do likewise, then it will use ``rcu_assign_pointer()`` to -actually commit the new credentials to ``current->cred``, it will release -``current->cred_replace_mutex`` to allow ``ptrace()`` to take place, and it -will notify the scheduler and others of the changes. +actually commit the new credentials to ``current->cred``, and it will notify +the scheduler and others of the changes. This function is guaranteed to return 0, so that it can be tail-called at the end of such functions as ``sys_setresuid()``.@@ -486,8 +489,7 @@ invoked:: void abort_creds(struct cred *new); -This releases the lock on ``current->cred_replace_mutex`` that -``prepare_creds()`` got and then releases the new credentials. +This releases the new credentials. A typical credentials alteration function would look something like this::fs/exec.c +9
-0@@ -1266,6 +1266,12 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; + retval =mutex_lock_killable (¤t->signal->cred_change_mutex); +if (retval) + goto out; + + bprm->called_flush_old_exec =1 ; +/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update @@ -1420,6 +1426,8 @@ static void free_bprm(struct linux_binprm *bprm) {free_arg_pages (bprm);if (bprm->cred) { +if (bprm->called_flush_old_exec) +mutex_unlock (¤t->signal->cred_change_mutex);mutex_unlock (¤t->signal->cred_guard_mutex);abort_creds (bprm->cred); }@@ -1469,6 +1477,7 @@ void install_exec_creds(struct linux_binprm *bprm)* credentials; any time after this it may be unlocked. */ security_bprm_committed_creds (bprm); +mutex_unlock (¤t->signal->cred_change_mutex);mutex_unlock (¤t->signal->cred_guard_mutex); }EXPORT_SYMBOL (install_exec_creds);include/linux/binfmts.h +5
-1@@ -44,7 +44,11 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1 , +/* +* Set by flush_old_exec, when the cred_change_mutex is taken. +*/ + called_flush_old_exec:1;#ifdef __alpha__ unsigned int taso:1;#endif include/linux/sched/signal.h +1
-0@@ -225,6 +225,7 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations * (notably. ptrace) */ +struct mutex cred_change_mutex;/* guard against credentials change */ } __randomize_layout;/* init/init_task.c +1
-0@@ -26,6 +26,7 @@ .multiprocess = HLIST_HEAD_INIT, .rlim = INIT_RLIMITS, .cred_guard_mutex = __MUTEX_INITIALIZER(init_signals.cred_guard_mutex), + .cred_change_mutex =__MUTEX_INITIALIZER (init_signals.cred_change_mutex), #ifdef CONFIG_POSIX_TIMERS .posix_timers =LIST_HEAD_INIT (init_signals.posix_timers), .cputimer = {kernel/cred.c +1
-1@@ -676,7 +676,7 @@ void __init cred_init(void)* * Returns the new credentials or NULL if out of memory. * -* Does not take, and does not return holding current->cred_replace_mutex. +* Does not take, and does not return holding ->cred_guard_mutex. */ struct cred *prepare_kernel_cred (struct task_struct *daemon) {kernel/fork.c +3
-2@@ -1224,7 +1224,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)struct mm_struct *mm;int err; - err =mutex_lock_killable (&task->signal->cred_guard_mutex); + err =mutex_lock_killable (&task->signal->cred_change_mutex);if (err)return ERR_PTR (err);@@ -1234,7 +1234,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)mmput (mm); mm =ERR_PTR (-EACCES ); } -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->cred_change_mutex);return mm; }@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); +mutex_init (&sig->cred_change_mutex);return 0 ; }mm/process_vm_access.c +1
-1@@ -204,7 +204,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter,if (!mm ||IS_ERR (mm)) { rc =IS_ERR (mm) ?PTR_ERR (mm) : -ESRCH ;/* -* Explicitly map EACCES to EPERM as EPERM is a more a +* Explicitly map EACCES to EPERM as EPERM is a more * appropriate error code for process_vw_readv/writev */ if (rc == -EACCES )tools/testing/selftests/ptrace/Makefile +2
-2@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -iquote../../../../include/uapi -Wall +CFLAGS += -std=c99 -pthread -iquote../../../../include/uapi -Wall -TEST_GEN_PROGS := get_syscall_info peeksiginfo +TEST_GEN_PROGS := get_syscall_info peeksiginfo vmaccess include ../lib.mktools/testing/selftests/ptrace/vmaccess.c new file +46
-0@@ -0,0 +1,46 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (c) 2020 Bernd Edlinger <[email protected]> +* All rights reserved. +* +* Check whether /proc/$pid/mem can be accessed without causing deadlocks +* when de_thread is blocked with ->cred_guard_mutex held. +*/ + +#include "../kselftest_harness.h" +#include <stdio.h> +#include <fcntl.h> +#include <pthread.h> +#include <signal.h> +#include <unistd.h> +#include <sys/ptrace.h> + +static void *thread (void *arg) +{ +ptrace (PTRACE_TRACEME ,0 ,0 ,0 ); +return NULL ; +} + +TEST (vmaccess) +{ +int f, pid =fork (); +char mm[64 ]; + +if (!pid) { +pthread_t pt; +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); +sprintf (mm,"/proc/%d/mem" , pid); + f =open (mm,O_RDONLY ); + ASSERT_LE(0, f) +close (f); +/* this is not fixed! ptrace(PTRACE_ATTACH, pid, 0,0); */ + f =kill (pid,SIGCONT ); +ASSERT_EQ (0 , f); +} + +TEST_HARNESS_MAIN
Quoted 55 lines · Eric W. Biederman
Quoted 8 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.Quoted 4 lines
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:I think your patch works, but I don't think to solve your case another mutex is necessary. Possibly it is justified, but I hesitate to introduce yet another concept in the code.
Having read elsewhere in the thread that this does not solve the problem Oleg has mentioned I am really hesitant to add more complexity to the situation.
For your case there is a straight forward and local workaround.
When the current task is ptracing the target task don't bother with
cred_gaurd_mutexandptrace_may_accessinaccess_mmas those tests have already passed. Instead just confirm the ptrace status. AKA the permission check inptraces_access_vm.I think something like this is all we need.
kernel/fork.c +10
-0@@ -1224,6 +1224,16 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)struct mm_struct *mm;int err; +if (task->ptrace && (current == task->parent)) { + mm =get_task_mm (task); +if ((get_dumpable (mm) !=SUID_DUMP_USER ) && + !ptracer_capable (task, mm->user_ns)) { +mmput (mm); + mm =ERR_PTR (-EACCESS ); + } +return mm; + } + err =mutex_lock_killable (&task->signal->cred_guard_mutex);if (err)return ERR_PTR (err);
Quoted 2 lines
Does this solve your test case?
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable. That is obviously because it bypasses the cred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.
However that smells like a denial of service, that this simple test case which can be executed by guest, creates a /proc/$pid/mem that freezes any process, even root, when it looks at it.
I mean: "ln -s README /proc/$pid/mem" would be a nice bomb.
Bernd.
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.
Yes. Your the test case in your patch a variant of the original problem.
I have been staring at this trying to understand the fundamentals of the original deeper problem.
The current scope of cred_guard_mutex in exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.
If that case did not exist we could reduce the scope of the cred_guard_mutex in exec to where your patch puts the cred_change_mutex.
I am starting to think reworking how we deal with ptrace and exec is the way to solve this problem.
Eric
Quoted 26 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
Quoted 7 lines
I have been staring at this trying to understand the fundamentals of the original deeper problem.
The current scope of
cred_guard_mutexin exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.
It has a second use, that it prevents two threads entering execve, which would probably result in disaster.
Quoted 8 lines
If that case did not exist we could reduce the scope of the
cred_guard_mutexin exec to where your patch puts thecred_change_mutex.I am starting to think reworking how we deal with ptrace and exec is the way to solve this problem.
Eric
Quoted 30 lines · Bernd Edlinger
Quoted 29 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
I have been staring at this trying to understand the fundamentals of the original deeper problem.
Quoted 7 lines
Quoted 4 lines
The current scope of
cred_guard_mutexin exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.It has a second use, that it prevents two threads entering execve, which would probably result in disaster.
Exec can fail with an error code up until de_thread. de_thread causes exec to fail with the error code -EAGAIN for the second thread to get into de_thread.
So no. The cred_guard_mutex is not needed for that case at all.
Quoted 5 lines
Quoted 5 lines
If that case did not exist we could reduce the scope of the
cred_guard_mutexin exec to where your patch puts thecred_change_mutex.I am starting to think reworking how we deal with ptrace and exec is the way to solve this problem.
I am 99% convinced that the fix is to move cred_guard_mutex down.
Then right after we take cred_guard_mutex do:
if (ptraced) {
use_original_creds();
}And call it a day.
The details suck but I am 99% certain that would solve everyones problems, and not be too bad to audit either.
Eric
Quoted 47 lines · Eric W. Biederman
Quoted 30 lines · Bernd Edlinger
Quoted 29 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
I have been staring at this trying to understand the fundamentals of the original deeper problem.
Quoted 7 lines
Quoted 4 lines
The current scope of
cred_guard_mutexin exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.It has a second use, that it prevents two threads entering execve, which would probably result in disaster.
Exec can fail with an error code up until
de_thread.de_threadcauses exec to fail with the error code-EAGAINfor the second thread to get intode_thread.So no. The
cred_guard_mutexis not needed for that case at all.
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
If that case did not exist we could reduce the scope of the
cred_guard_mutexin exec to where your patch puts thecred_change_mutex.I am starting to think reworking how we deal with ptrace and exec is the way to solve this problem.
I am 99% convinced that the fix is to move
cred_guard_mutexdown.
"move cred_guard_mutex down" as in "take it once we've already set up the new process, past the point of no return"?
Quoted 9 lines
Then right after we take
cred_guard_mutexdo:if (ptraced) { use_original_creds(); }And call it a day.
The details suck but I am 99% certain that would solve everyones problems, and not be too bad to audit either.
Ah, hmm, that sounds like it'll work fine at least when no LSMs are involved.
SELinux normally doesn't do the execution-degrading thing, it just blocks the execution completely - see their selinux_bprm_set_creds() hook. So I think they'd still need to set some state on the task that says "we're currently in the middle of an execution where the target task will run in context X", and then check against that in the ptrace_may_access hook. Or I suppose they could just kill the task near the end of execve, although that'd be kinda ugly.
Quoted 72 lines · Jann Horn
Quoted 47 lines · Eric W. Biederman
Quoted 30 lines · Bernd Edlinger
Quoted 29 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
I have been staring at this trying to understand the fundamentals of the original deeper problem.
Quoted 7 lines
Quoted 4 lines
The current scope of
cred_guard_mutexin exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.It has a second use, that it prevents two threads entering execve, which would probably result in disaster.
Exec can fail with an error code up until
de_thread.de_threadcauses exec to fail with the error code-EAGAINfor the second thread to get intode_thread.So no. The
cred_guard_mutexis not needed for that case at all.Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
If that case did not exist we could reduce the scope of the
cred_guard_mutexin exec to where your patch puts thecred_change_mutex.I am starting to think reworking how we deal with ptrace and exec is the way to solve this problem.
I am 99% convinced that the fix is to move
cred_guard_mutexdown."move
cred_guard_mutexdown" as in "take it once we've already set up the new process, past the point of no return"?Quoted 9 lines
Then right after we take
cred_guard_mutexdo:if (ptraced) { use_original_creds(); }And call it a day.
The details suck but I am 99% certain that would solve everyones problems, and not be too bad to audit either.
Ah, hmm, that sounds like it'll work fine at least when no LSMs are involved.
Quoted 8 lines
SELinux normally doesn't do the execution-degrading thing, it just blocks the execution completely - see their
selinux_bprm_set_creds()hook. So I think they'd still need to set some state on the task that says "we're currently in the middle of an execution where the target task will run in context X", and then check against that in theptrace_may_accesshook. Or I suppose they could just kill the task near the end of execve, although that'd be kinda ugly.
We have current->in_execve for that, right?
I think when the cred_guard_mutex is taken only in the critical section, then PTRACE_ATTACH could take the guard_mutex, and look at current->in_execve, and just return -EAGAIN in that case, right, everybody happy :)
Bernd.
Quoted 82 lines · Bernd Edlinger
Quoted 72 lines · Jann Horn
Quoted 47 lines · Eric W. Biederman
Quoted 30 lines · Bernd Edlinger
Quoted 29 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
I have been staring at this trying to understand the fundamentals of the original deeper problem.
Quoted 7 lines
Quoted 4 lines
The current scope of
cred_guard_mutexin exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.It has a second use, that it prevents two threads entering execve, which would probably result in disaster.
Exec can fail with an error code up until
de_thread.de_threadcauses exec to fail with the error code-EAGAINfor the second thread to get intode_thread.So no. The
cred_guard_mutexis not needed for that case at all.Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
If that case did not exist we could reduce the scope of the
cred_guard_mutexin exec to where your patch puts thecred_change_mutex.I am starting to think reworking how we deal with ptrace and exec is the way to solve this problem.
I am 99% convinced that the fix is to move
cred_guard_mutexdown."move
cred_guard_mutexdown" as in "take it once we've already set up the new process, past the point of no return"?Quoted 9 lines
Then right after we take
cred_guard_mutexdo:if (ptraced) { use_original_creds(); }And call it a day.
The details suck but I am 99% certain that would solve everyones problems, and not be too bad to audit either.
Ah, hmm, that sounds like it'll work fine at least when no LSMs are involved.
Quoted 8 lines
SELinux normally doesn't do the execution-degrading thing, it just blocks the execution completely - see their
selinux_bprm_set_creds()hook. So I think they'd still need to set some state on the task that says "we're currently in the middle of an execution where the target task will run in context X", and then check against that in theptrace_may_accesshook. Or I suppose they could just kill the task near the end of execve, although that'd be kinda ugly.
Quoted 4 lines
We have
current->in_execvefor that, right?
I think when thecred_guard_mutexis taken only in the critical section, thenPTRACE_ATTACHcould take theguard_mutex, and look atcurrent->in_execve, and just return-EAGAINin that case, right, everybody happy :)
It's probably going to mean that things like strace will just randomly fail to attach to processes if they happen to be in the middle of execve... but I guess that works?
<[email protected]>,"[email protected]" <[email protected]>,"[email protected]" <[email protected]>,"[email protected]" <[email protected]>,"[email protected]" <[email protected]>,linux-security-module <[email protected]>
From: Christian Brauner <[email protected]>
Message-ID: <[email protected]>
Quoted 103 lines · Jann Horn
Quoted 2 lines · Bernd Edlinger
Quoted 1 line · Jann Horn
On Mon, Mar 2, 2020 at 5:19 PM Eric W. Biederman
Quoted 10 lines · <[email protected]> wrote:
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines · Bernd Edlinger
Quoted 6 lines · Eric W. Biederman
Quoted 4 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not
freeze,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 2 lines
That is
obviously because it bypasses thecred_guard_mutex. But allother
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
process that access this file still freeze, and cannot be interrupted except with kill -9.
However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
that freezes any process, even root, when it looks at it. I mean: "ln -s README
/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
I have been staring at this trying to understand the
fundamentals of the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
original deeper problem.
The current scope of
cred_guard_mutexin exec is because beingptraced
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
causes suid exec to act differently. So we need to know early
if we are
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
ptraced.
It has a second use, that it prevents two threads entering
execve,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
which would probably result in disaster.
Exec can fail with an error code up until
de_thread.de_threadcauses
Quoted 1 line
Quoted 1 line
Quoted 1 line
exec to fail with the error code
-EAGAINfor the second thread toget
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
into
de_thread.So no. The
cred_guard_mutexis not needed for that case at all.Quoted 2 lines
Quoted 2 lines
If that case did not exist we could reduce the scope of the
cred_guard_mutexin exec to where your patch puts the
cred_change_mutex.Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I am starting to think reworking how we deal with ptrace and
exec is the
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
way to solve this problem.
I am 99% convinced that the fix is to move
cred_guard_mutexdown."move
cred_guard_mutexdown" as in "take it once we've already setup
Quoted 7 lines
Quoted 7 lines
the new process, past the point of no return"?
Quoted 5 lines
Then right after we take
cred_guard_mutexdo:if (ptraced) { use_original_creds(); }Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
And call it a day.
The details suck but I am 99% certain that would solve everyones problems, and not be too bad to audit either.
Ah, hmm, that sounds like it'll work fine at least when no LSMs are
involved.
Quoted 3 lines
Quoted 3 lines
SELinux normally doesn't do the execution-degrading thing, it just blocks the execution completely - see their
selinux_bprm_set_creds()Quoted 1 line
Quoted 1 line
hook. So I think they'd still need to set some state on the task
that
Quoted 1 line
Quoted 1 line
says "we're currently in the middle of an execution where the
target
Quoted 7 lines
Quoted 4 lines
task will run in context X", and then check against that in the
ptrace_may_accesshook. Or I suppose they could just kill the task near the end of execve, although that'd be kinda ugly.We have
current->in_execvefor that, right?
I think when thecred_guard_mutexis taken only in the critical
Quoted 8 lines
section,
Quoted 1 line
then
PTRACE_ATTACHcould take theguard_mutex, and look at
current->in_execve,Quoted 1 line
and just return
-EAGAINin that case, right, everybody happy :)It's probably going to mean that things like strace will just randomly fail to attach to processes if they happen to be in the middle of execve... but I guess that works?
That sounds like an acceptable outcome. We can at least risk it and if we regress revert or come up with the more complex solution suggested in another mail here?
Quoted 6 lines · On Mon, Mar 2, 2020 at 6:43 PM <[email protected]> wrote:
Quoted 5 lines · Jann Horn
Quoted 2 lines · Bernd Edlinger
Quoted 1 line · Jann Horn
On Mon, Mar 2, 2020 at 5:19 PM Eric W. Biederman
<[email protected]> wrote:
[...]
Quoted 38 lines
Quoted 38 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
I am 99% convinced that the fix is to move
cred_guard_mutexdown."move
cred_guard_mutexdown" as in "take it once we've already setup
Quoted 7 lines
Quoted 7 lines
the new process, past the point of no return"?
Quoted 5 lines
Then right after we take
cred_guard_mutexdo:if (ptraced) { use_original_creds(); }Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
And call it a day.
The details suck but I am 99% certain that would solve everyones problems, and not be too bad to audit either.
Ah, hmm, that sounds like it'll work fine at least when no LSMs are
involved.
Quoted 3 lines
Quoted 3 lines
SELinux normally doesn't do the execution-degrading thing, it just blocks the execution completely - see their
selinux_bprm_set_creds()Quoted 1 line
Quoted 1 line
hook. So I think they'd still need to set some state on the task
that
Quoted 1 line
Quoted 1 line
says "we're currently in the middle of an execution where the
target
Quoted 7 lines
Quoted 4 lines
task will run in context X", and then check against that in the
ptrace_may_accesshook. Or I suppose they could just kill the task near the end of execve, although that'd be kinda ugly.We have
current->in_execvefor that, right?
I think when thecred_guard_mutexis taken only in the criticalsection,
Quoted 1 line
then
PTRACE_ATTACHcould take theguard_mutex, and look at
current->in_execve,Quoted 1 line
and just return
-EAGAINin that case, right, everybody happy :)
Quoted 8 lines
Quoted 3 lines
It's probably going to mean that things like strace will just randomly fail to attach to processes if they happen to be in the middle of execve... but I guess that works?
That sounds like an acceptable outcome. We can at least risk it and if we regress revert or come up with the more complex solution suggested in another mail here?
Yeah, sounds reasonable, I guess.
PATCH v3 1/1 exec: Fix a deadlock in ptrace
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in vm_access.
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the PTRACE_EVENT_EXIT is handled and the WIFEXITED event is received:
strace D 0 30614 30584 0x00000000
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
expect D 0 31933 30876 0x80004003
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
The proposed solution is to take the cred_guard_mutex only in a critical section at the beginning, and at the end of the execve function, and let PTRACE_ATTACH fail with EAGAIN while execve is not complete, but other functions like vm_access are allowed to complete normally.
I also took the opportunity to improve the documentation of prepare_creds, which is obviously out of sync.
--- Documentation/security/credentials.rst | 19 +++++---- fs/exec.c | 28 ++++++++++++-- include/linux/binfmts.h | 6 ++- include/linux/sched/signal.h | 1 + init/init_task.c | 1 + kernel/cred.c | 2 +- kernel/fork.c | 1 + kernel/ptrace.c | 4 ++ mm/process_vm_access.c | 2 +- tools/testing/selftests/ptrace/Makefile | 4 +- tools/testing/selftests/ptrace/vmaccess.c | 64 +++++++++++++++++++++++++++++++ 11 files changed, 115 insertions(+), 17 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.c
v2: adds a test case which passes when this patch is applied.
v3: fixes the issue without introducing a new mutex.
Documentation/security/credentials.rst +11 -8
@@ -437,9 +437,14 @@ new set of credentials by calling:: struct cred *prepare_creds(void); -this locks current->cred_replace_mutex and then allocates and constructs a -duplicate of the current process's credentials, returning with the mutex still -held if successful. It returns NULL if not successful (out of memory). +this allocates and constructs a duplicate of the current process's credentials. +It returns NULL if not successful (out of memory). + +If called from __do_execve_file, the mutex current->signal->cred_guard_mutex +is acquired before this function gets called, and released after setting +current->signal->cred_locked_for_ptrace. The same mutex is acquired later, +while the credentials and the process mmap are actually changed, and +current->signal->cred_locked_for_ptrace is reset again. The mutex prevents ``ptrace()`` from altering the ptrace state of a process while security checks on credentials construction and changing is taking place
@@ -466,9 +471,8 @@ by calling:: This will alter various aspects of the credentials and the process, giving the LSM a chance to do likewise, then it will use ``rcu_assign_pointer()`` to -actually commit the new credentials to ``current->cred``, it will release -``current->cred_replace_mutex`` to allow ``ptrace()`` to take place, and it -will notify the scheduler and others of the changes. +actually commit the new credentials to ``current->cred``, and it will notify +the scheduler and others of the changes. This function is guaranteed to return 0, so that it can be tail-called at the end of such functions as ``sys_setresuid()``.
@@ -486,8 +490,7 @@ invoked:: void abort_creds(struct cred *new); -This releases the lock on ``current->cred_replace_mutex`` that -``prepare_creds()`` got and then releases the new credentials. +This releases the new credentials. A typical credentials alteration function would look something like this::
fs/exec.c +24 -4
@@ -1266,6 +1266,12 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; + retval =mutex_lock_killable (¤t->signal->cred_guard_mutex); +if (retval) + goto out; + + bprm->called_flush_old_exec =1 ; +/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update
@@ -1398,28 +1404,41 @@ void finalize_exec(struct linux_binprm *bprm)EXPORT_SYMBOL (finalize_exec);/* -* Prepare credentials and lock ->cred_guard_mutex. +* Prepare credentials and set ->cred_locked_for_ptrace. * install_exec_creds() commits the new creds and drops the lock. * Or, if exec fails before, free_bprm() should release ->cred and * and unlock. */ static int prepare_bprm_creds (struct linux_binprm *bprm) { +int ret; +if (mutex_lock_interruptible (¤t->signal->cred_guard_mutex))return -ERESTARTNOINTR ; + ret = -EAGAIN ; +if (unlikely (current->signal->cred_locked_for_ptrace)) + goto out; + + ret = -ENOMEM ; bprm->cred =prepare_exec_creds (); -if (likely (bprm->cred)) -return 0 ; +if (likely (bprm->cred)) { + current->signal->cred_locked_for_ptrace = true; + ret =0 ; + } +out:mutex_unlock (¤t->signal->cred_guard_mutex); -return -ENOMEM ; +return ret; }static void free_bprm (struct linux_binprm *bprm) {free_arg_pages (bprm);if (bprm->cred) { +if (!bprm->called_flush_old_exec) +mutex_lock (¤t->signal->cred_guard_mutex); + current->signal->cred_locked_for_ptrace = false;mutex_unlock (¤t->signal->cred_guard_mutex);abort_creds (bprm->cred); }
@@ -1469,6 +1488,7 @@ void install_exec_creds(struct linux_binprm *bprm)* credentials; any time after this it may be unlocked. */ security_bprm_committed_creds (bprm); + current->signal->cred_locked_for_ptrace = false;mutex_unlock (¤t->signal->cred_guard_mutex); }EXPORT_SYMBOL (install_exec_creds);
include/linux/binfmts.h +5 -1
@@ -44,7 +44,11 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1 , +/* +* Set by flush_old_exec, when the cred_change_mutex is taken. +*/ + called_flush_old_exec:1;#ifdef __alpha__ unsigned int taso:1;#endif
include/linux/sched/signal.h +1 -0
@@ -225,6 +225,7 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations * (notably. ptrace) */ +bool cred_locked_for_ptrace;/* set while in execve */ } __randomize_layout;/*
init/init_task.c +1 -0
@@ -26,6 +26,7 @@ .multiprocess = HLIST_HEAD_INIT, .rlim = INIT_RLIMITS, .cred_guard_mutex = __MUTEX_INITIALIZER(init_signals.cred_guard_mutex), + .cred_locked_for_ptrace = false, #ifdef CONFIG_POSIX_TIMERS .posix_timers =LIST_HEAD_INIT (init_signals.posix_timers), .cputimer = {
kernel/cred.c +1 -1
@@ -676,7 +676,7 @@ void __init cred_init(void)* * Returns the new credentials or NULL if out of memory. * -* Does not take, and does not return holding current->cred_replace_mutex. +* Does not take, and does not return holding ->cred_guard_mutex. */ struct cred *prepare_kernel_cred (struct task_struct *daemon) {
kernel/fork.c +1 -0
@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); + sig->cred_locked_for_ptrace = false;return 0 ; }
kernel/ptrace.c +4 -0
@@ -395,6 +395,10 @@ static int ptrace_attach(struct task_struct *task, long request,if (mutex_lock_interruptible (&task->signal->cred_guard_mutex)) goto out; + retval = -EAGAIN ; +if (task->signal->cred_locked_for_ptrace) + goto unlock_creds; +task_lock (task); retval =__ptrace_may_access (task,PTRACE_MODE_ATTACH_REALCREDS );task_unlock (task);
mm/process_vm_access.c +1 -1
@@ -204,7 +204,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter,if (!mm ||IS_ERR (mm)) { rc =IS_ERR (mm) ?PTR_ERR (mm) : -ESRCH ;/* -* Explicitly map EACCES to EPERM as EPERM is a more a +* Explicitly map EACCES to EPERM as EPERM is a more * appropriate error code for process_vw_readv/writev */ if (rc == -EACCES )
tools/testing/selftests/ptrace/Makefile +2 -2
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -iquote../../../../include/uapi -Wall +CFLAGS += -std=c99 -pthread -iquote../../../../include/uapi -Wall -TEST_GEN_PROGS := get_syscall_info peeksiginfo +TEST_GEN_PROGS := get_syscall_info peeksiginfo vmaccess include ../lib.mk
tools/testing/selftests/ptrace/vmaccess.c new file +64 -0
64 lines not on this page · raw message
Signature · 1 line
1.9.1
include/linux/binfmts.h struct linux_binprm { · line 49
Quoted 10 lines · Bernd Edlinger
include/linux/binfmts.h +3
-1@@ -44,7 +44,11 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1, +/* +* Set by flush_old_exec, when the cred_change_mutex is taken.
Oops, missed to update this comment, should be "when the cred_guard_mutex is taken".
I'll send a new patch later.
Bernd.
Quoted 5 lines
include/linux/binfmts.h +2
-0+*/ + called_flush_old_exec:1 ;#ifdef __alpha__ unsigned int taso:1;#endif
Quoted 72 lines · Jann Horn
Quoted 47 lines · Eric W. Biederman
Quoted 30 lines · Bernd Edlinger
Quoted 29 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
I have been staring at this trying to understand the fundamentals of the original deeper problem.
Quoted 7 lines
Quoted 4 lines
The current scope of
cred_guard_mutexin exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.It has a second use, that it prevents two threads entering execve, which would probably result in disaster.
Exec can fail with an error code up until
de_thread.de_threadcauses exec to fail with the error code-EAGAINfor the second thread to get intode_thread.So no. The
cred_guard_mutexis not needed for that case at all.Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
If that case did not exist we could reduce the scope of the
cred_guard_mutexin exec to where your patch puts thecred_change_mutex.I am starting to think reworking how we deal with ptrace and exec is the way to solve this problem.
I am 99% convinced that the fix is to move
cred_guard_mutexdown."move
cred_guard_mutexdown" as in "take it once we've already set up the new process, past the point of no return"?Quoted 9 lines
Then right after we take
cred_guard_mutexdo:if (ptraced) { use_original_creds(); }And call it a day.
The details suck but I am 99% certain that would solve everyones problems, and not be too bad to audit either.
Ah, hmm, that sounds like it'll work fine at least when no LSMs are involved.
Quoted 7 lines
SELinux normally doesn't do the execution-degrading thing, it just blocks the execution completely - see their
selinux_bprm_set_creds()hook. So I think they'd still need to set some state on the task that says "we're currently in the middle of an execution where the target task will run in context X", and then check against that in theptrace_may_accesshook. Or I suppose they could just kill the task near the end of execve, although that'd be kinda ugly.
So I looked and it the selinux thing is complicated. It is not as simple as selinux does not do the execution degrading thing.
In the normal case selinux retains the same sid for no_new_privs
and for no_suid. I don't know when selinux policy says to change sids
on exec.
My sense right now is that selinux is completely inconsistent in this area and it might not be a bit deal at all to write a patch to make selinux match the rest of exec and unconditionally do the execution degrading.
Certainly attempting it and having the conversation is probably the best way to illuminate this dark corner.
Eric
Quoted 37 lines · Eric W. Biederman
Quoted 35 lines · Bernd Edlinger
Quoted 33 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
I have been staring at this trying to understand the fundamentals of the original deeper problem.
The current scope of
cred_guard_mutexin exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.
Quoted 9 lines
Quoted 2 lines
It has a second use, that it prevents two threads entering execve, which would probably result in disaster.
Exec can fail with an error code up until
de_thread.de_threadcauses exec to fail with the error code-EAGAINfor the second thread to get intode_thread.So no. The
cred_guard_mutexis not needed for that case at all.
Okay, but that will reset current->in_execve, right?
Quoted 21 lines
Quoted 5 lines
Quoted 5 lines
If that case did not exist we could reduce the scope of the
cred_guard_mutexin exec to where your patch puts thecred_change_mutex.I am starting to think reworking how we deal with ptrace and exec is the way to solve this problem.
I am 99% convinced that the fix is to move
cred_guard_mutexdown.Then right after we take
cred_guard_mutexdo:if (ptraced) { use_original_creds(); }And call it a day.
The details suck but I am 99% certain that would solve everyones problems, and not be too bad to audit either.
Eric
Quoted 41 lines · Bernd Edlinger
Quoted 40 lines · Eric W. Biederman
Quoted 37 lines · Bernd Edlinger
Quoted 26 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
Quoted 7 lines
I have been staring at this trying to understand the fundamentals of the original deeper problem.
The current scope of
cred_guard_mutexin exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.It has a second use, that it prevents two threads entering execve, which would probably result in disaster.
Quoted 8 lines
Quoted 6 lines
Exec can fail with an error code up until
de_thread.de_threadcauses exec to fail with the error code-EAGAINfor the second thread to get intode_thread.So no. The
cred_guard_mutexis not needed for that case at all.Okay, but that will reset
current->in_execve, right?
Absolutely.
The error handling kicks in and exec_binprm fails with a negative return code. Then __do_excve_file cleans up and clears current->in_execve.
Eric
Quoted 50 lines · Eric W. Biederman
Quoted 48 lines · Bernd Edlinger
Quoted 46 lines · Eric W. Biederman
Quoted 30 lines · Bernd Edlinger
Quoted 29 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
I tried this with s/EACCESS/EACCES/.
The test case in this patch is not fixed, but strace does not freeze, at least with my setup where it did freeze repeatable.
Thanks, That is what I was aiming at.
So we have one method we can pursue to fix this in practice.
Quoted 9 lines
That is
obviously because it bypasses thecred_guard_mutex. But all other process that access this file still freeze, and cannot be interrupted except with kill -9.However that smells like a denial of service, that this simple test case which can be executed by guest, creates a
/proc/$pid/memthat freezes any process, even root, when it looks at it.
I mean: "ln -s README/proc/$pid/mem" would be a nice bomb.Yes. Your the test case in your patch a variant of the original problem.
I have been staring at this trying to understand the fundamentals of the original deeper problem.
Quoted 7 lines
Quoted 4 lines
The current scope of
cred_guard_mutexin exec is because being ptraced causes suid exec to act differently. So we need to know early if we are ptraced.It has a second use, that it prevents two threads entering execve, which would probably result in disaster.
Exec can fail with an error code up until
de_thread.de_threadcauses exec to fail with the error code-EAGAINfor the second thread to get intode_thread.So no. The
cred_guard_mutexis not needed for that case at all.
Quoted 8 lines
Quoted 1 line
Okay, but that will reset
current->in_execve, right?Absolutely.
The error handling kicks in and
exec_binprmfails with a negative return code. Then__do_excve_filecleans up and clearscurrent->in_execve.
Yes of course. I was under the wrong impression that that value is a kind of global, but it is a thread local.
So I think I need a new boolean see v3 of the patch, and soon v4 (with just one comment fixed).
I'm currently executing the strace v5.5 testsuite, and every test is passed so far. I'll also look at gdb testsuite, before I send the next version.
Thanks
Bernd.
PATCH v4 1/1 exec: Fix a deadlock in ptrace
In reply to Eric W. BiedermanThis fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in vm_access.
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the PTRACE_EVENT_EXIT is handled and the WIFEXITED event is received:
strace D 0 30614 30584 0x00000000
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
expect D 0 31933 30876 0x80004003
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
The proposed solution is to take the cred_guard_mutex only in a critical section at the beginning, and at the end of the execve function, and let PTRACE_ATTACH fail with EAGAIN while execve is not complete, but other functions like vm_access are allowed to complete normally.
I also took the opportunity to improve the documentation of prepare_creds, which is obviously out of sync.
--- Documentation/security/credentials.rst | 19 +++++---- fs/exec.c | 28 +++++++++++-- include/linux/binfmts.h | 6 ++- include/linux/sched/signal.h | 1 + init/init_task.c | 1 + kernel/cred.c | 2 +- kernel/fork.c | 1 + kernel/ptrace.c | 4 ++ mm/process_vm_access.c | 2 +- tools/testing/selftests/ptrace/Makefile | 4 +- tools/testing/selftests/ptrace/vmaccess.c | 66 +++++++++++++++++++++++++++++++ 11 files changed, 117 insertions(+), 17 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.c
v2: adds a test case which passes when this patch is applied.
v3: fixes the issue without introducing a new mutex.
v4: fixes one comment and a formatting issue found by checkpatch.pl in the test case.
Documentation/security/credentials.rst +11 -8
@@ -437,9 +437,14 @@ new set of credentials by calling:: struct cred *prepare_creds(void); -this locks current->cred_replace_mutex and then allocates and constructs a -duplicate of the current process's credentials, returning with the mutex still -held if successful. It returns NULL if not successful (out of memory). +this allocates and constructs a duplicate of the current process's credentials. +It returns NULL if not successful (out of memory). + +If called from __do_execve_file, the mutex current->signal->cred_guard_mutex +is acquired before this function gets called, and released after setting +current->signal->cred_locked_for_ptrace. The same mutex is acquired later, +while the credentials and the process mmap are actually changed, and +current->signal->cred_locked_for_ptrace is reset again. The mutex prevents ``ptrace()`` from altering the ptrace state of a process while security checks on credentials construction and changing is taking place
@@ -466,9 +471,8 @@ by calling:: This will alter various aspects of the credentials and the process, giving the LSM a chance to do likewise, then it will use ``rcu_assign_pointer()`` to -actually commit the new credentials to ``current->cred``, it will release -``current->cred_replace_mutex`` to allow ``ptrace()`` to take place, and it -will notify the scheduler and others of the changes. +actually commit the new credentials to ``current->cred``, and it will notify +the scheduler and others of the changes. This function is guaranteed to return 0, so that it can be tail-called at the end of such functions as ``sys_setresuid()``.
@@ -486,8 +490,7 @@ invoked:: void abort_creds(struct cred *new); -This releases the lock on ``current->cred_replace_mutex`` that -``prepare_creds()`` got and then releases the new credentials. +This releases the new credentials. A typical credentials alteration function would look something like this::
fs/exec.c +24 -4
@@ -1266,6 +1266,12 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; + retval =mutex_lock_killable (¤t->signal->cred_guard_mutex); +if (retval) + goto out; + + bprm->called_flush_old_exec =1 ; +/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update
@@ -1398,28 +1404,41 @@ void finalize_exec(struct linux_binprm *bprm)EXPORT_SYMBOL (finalize_exec);/* -* Prepare credentials and lock ->cred_guard_mutex. +* Prepare credentials and set ->cred_locked_for_ptrace. * install_exec_creds() commits the new creds and drops the lock. * Or, if exec fails before, free_bprm() should release ->cred and * and unlock. */ static int prepare_bprm_creds (struct linux_binprm *bprm) { +int ret; +if (mutex_lock_interruptible (¤t->signal->cred_guard_mutex))return -ERESTARTNOINTR ; + ret = -EAGAIN ; +if (unlikely (current->signal->cred_locked_for_ptrace)) + goto out; + + ret = -ENOMEM ; bprm->cred =prepare_exec_creds (); -if (likely (bprm->cred)) -return 0 ; +if (likely (bprm->cred)) { + current->signal->cred_locked_for_ptrace = true; + ret =0 ; + } +out:mutex_unlock (¤t->signal->cred_guard_mutex); -return -ENOMEM ; +return ret; }static void free_bprm (struct linux_binprm *bprm) {free_arg_pages (bprm);if (bprm->cred) { +if (!bprm->called_flush_old_exec) +mutex_lock (¤t->signal->cred_guard_mutex); + current->signal->cred_locked_for_ptrace = false;mutex_unlock (¤t->signal->cred_guard_mutex);abort_creds (bprm->cred); }
@@ -1469,6 +1488,7 @@ void install_exec_creds(struct linux_binprm *bprm)* credentials; any time after this it may be unlocked. */ security_bprm_committed_creds (bprm); + current->signal->cred_locked_for_ptrace = false;mutex_unlock (¤t->signal->cred_guard_mutex); }EXPORT_SYMBOL (install_exec_creds);
include/linux/binfmts.h +5 -1
@@ -44,7 +44,11 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1 , +/* +* Set by flush_old_exec, when the cred_guard_mutex is taken. +*/ + called_flush_old_exec:1;#ifdef __alpha__ unsigned int taso:1;#endif
include/linux/sched/signal.h +1 -0
@@ -225,6 +225,7 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations * (notably. ptrace) */ +bool cred_locked_for_ptrace;/* set while in execve */ } __randomize_layout;/*
init/init_task.c +1 -0
@@ -26,6 +26,7 @@ .multiprocess = HLIST_HEAD_INIT, .rlim = INIT_RLIMITS, .cred_guard_mutex = __MUTEX_INITIALIZER(init_signals.cred_guard_mutex), + .cred_locked_for_ptrace = false, #ifdef CONFIG_POSIX_TIMERS .posix_timers =LIST_HEAD_INIT (init_signals.posix_timers), .cputimer = {
kernel/cred.c +1 -1
@@ -676,7 +676,7 @@ void __init cred_init(void)* * Returns the new credentials or NULL if out of memory. * -* Does not take, and does not return holding current->cred_replace_mutex. +* Does not take, and does not return holding ->cred_guard_mutex. */ struct cred *prepare_kernel_cred (struct task_struct *daemon) {
kernel/fork.c +1 -0
@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); + sig->cred_locked_for_ptrace = false;return 0 ; }
kernel/ptrace.c +4 -0
@@ -395,6 +395,10 @@ static int ptrace_attach(struct task_struct *task, long request,if (mutex_lock_interruptible (&task->signal->cred_guard_mutex)) goto out; + retval = -EAGAIN ; +if (task->signal->cred_locked_for_ptrace) + goto unlock_creds; +task_lock (task); retval =__ptrace_may_access (task,PTRACE_MODE_ATTACH_REALCREDS );task_unlock (task);
mm/process_vm_access.c +1 -1
@@ -204,7 +204,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter,if (!mm ||IS_ERR (mm)) { rc =IS_ERR (mm) ?PTR_ERR (mm) : -ESRCH ;/* -* Explicitly map EACCES to EPERM as EPERM is a more a +* Explicitly map EACCES to EPERM as EPERM is a more * appropriate error code for process_vw_readv/writev */ if (rc == -EACCES )
tools/testing/selftests/ptrace/Makefile +2 -2
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -iquote../../../../include/uapi -Wall +CFLAGS += -std=c99 -pthread -iquote../../../../include/uapi -Wall -TEST_GEN_PROGS := get_syscall_info peeksiginfo +TEST_GEN_PROGS := get_syscall_info peeksiginfo vmaccess include ../lib.mk
tools/testing/selftests/ptrace/vmaccess.c new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
1.9.1
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 5 lines
The proposed solution is to take the
cred_guard_mutexonly in a critical section at the beginning, and at the end of the execve function, and letPTRACE_ATTACHfail with EAGAIN while execve is not complete, but other functions likevm_accessare allowed to complete normally.
Sorry to be bummer, but I don't think this will work. A few more things during the exec process depend on cred_guard_mutex being held.
If I'm reading this patch correctly, this changes the lifetime of the cred_guard_mutex lock to be:
- during prepare_bprm_creds()
- from flush_old_exec() through install_exec_creds()
Before, cred_guard_mutex was held from prepare_bprm_creds() through install_exec_creds().
That means, for example, that check_unsafe_exec()'s documented invariant is violated:
/*
* determine how safe it is to execute the proposed program
* - the caller must hold ->cred_guard_mutex to protect against
* PTRACE_ATTACH or seccomp thread-sync
*/
static void check_unsafe_exec(struct linux_binprm *bprm) ...
which is looking at no_new_privs as well as other details, and making decisions about the bprm state from the current state.
I think it also means that the potentially multiple invocations of bprm_fill_uid() (via prepare_binprm() via binfmt_script.c and binfmt_misc.c) would be changing bprm->cred details (uid, gid) without a lock (another place where current's no_new_privs is evaluated).
Related, it also means that cred_guard_mutex is unheld for every invocation of search_binary_handler() (which can loop via the previously mentioned binfmt_script.c and binfmt_misc.c), if any of them have hidden dependencies on cred_guard_mutex. (Thought I only see bprm_fill_uid() currently.)
For seccomp, the expectations about existing thread states risks races too. There are two locks held for TSYNC:
- current->sighand->siglock is held to keep new threads from appearing/disappearing, which would destroy filter refcounting and lead to memory corruption.
- cred_guard_mutex is held to keep no_new_privs in sync with filters to avoid no_new_privs and filter confusion during exec, which could lead to exploitable setuid conditions (see below).
Just racing a malicious thread during TSYNC is not a very strong example (a malicious thread could do lots of fun things to "current" before it ever got near calling TSYNC), but I think there is the risk of mismatched/confused states that we don't want to allow. One is a particularly bad state that could lead to privilege escalations (in the form of the old "sendmail doesn't check setuid" flaw; if a setuid process has a filter attached that silently fails a priv-dropping setuid call and continues execution with elevated privs, it can be tricked into doing bad things on behalf of the unprivileged parent, which was the primary goal of the original use of cred_guard_mutex with TSYNC[1]):
thread A clones thread B
thread B starts setuid exec
thread A sets no_new_privs
thread A calls seccomp with TSYNC
thread A in seccomp_sync_threads() sets seccomp filter on self and thread B thread B passes check_unsafe_exec() with no_new_privs unset thread B reaches bprm_fill_uid() with no_new_privs unset and gains privs thread A still in seccomp_sync_threads() sets no_new_privs on thread B thread B finishes exec, now running with elevated privs, a filter chosen
by thread A, _and_ nnp set (which doesn't matter)
With the original locking, thread B will fail check_unsafe_exec() because filter and nnp state are changed together, with "atomicity" protected by the cred_guard_mutex.
And this is just the bad state I _can_ see. I'm worried there are more...
All this said, I do see a small similarity here to the work I did to stabilize stack rlimits (there was an ongoing problem with making multiple decisions for the bprm based on current's state -- but current's state was mutable during exec). For this, I saved rlim_stack to bprm and ignored current's copy until exec ended and then stored bprm's copy into current. If the only problem anyone can see here is the handling of no_new_privs, we might be able to solve that similarly, at least disentangling tsync/nnp from cred_guard_mutex.
-Kees
[1] https://lore.kernel.org/lkml/[email protected]/
Signature · 1 line
Kees Cook
Quoted 57 lines · Kees Cook
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9Quoted 5 lines
The proposed solution is to take the
cred_guard_mutexonly in a critical section at the beginning, and at the end of the execve function, and letPTRACE_ATTACHfail with EAGAIN while execve is not complete, but other functions likevm_accessare allowed to complete normally.Sorry to be bummer, but I don't think this will work. A few more things during the exec process depend on
cred_guard_mutexbeing held.If I'm reading this patch correctly, this changes the lifetime of the
cred_guard_mutexlock to be:
- duringprepare_bprm_creds()
- fromflush_old_exec()throughinstall_exec_creds()
Before,cred_guard_mutexwas held fromprepare_bprm_creds()throughinstall_exec_creds().
Quoted 7 lines
That means, for example, that check_unsafe_exec()'s documented invariant is violated:
/*
* determine how safe it is to execute the proposed program
* - the caller must hold ->cred_guard_mutex to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/
Oh, right, I haven't understood that hint...
Quoted 8 lines
static void
check_unsafe_exec(struct linux_binprm *bprm)...
which is looking atno_new_privsas well as other details, and making decisions about the bprm state from the current state.I think it also means that the potentially multiple invocations of
bprm_fill_uid()(viaprepare_binprm()viabinfmt_script.candbinfmt_misc.c) would be changingbprm->creddetails (uid, gid) without a lock (another place where current'sno_new_privsis evaluated).
So no_new_privs can change from 0->1, but should not when execve is running.
As long as the calling thread is in execve it won't do this, and the only other place, where it may set for other threads is in seccomp_sync_threads, but that can easily be avoided see below.
Quoted 7 lines
Related, it also means that
cred_guard_mutexis unheld for every invocation ofsearch_binary_handler()(which can loop via the previously mentionedbinfmt_script.candbinfmt_misc.c), if any of them have hidden dependencies oncred_guard_mutex. (Thought I only seebprm_fill_uid()currently.)
Quoted 5 lines
For seccomp, the expectations about existing thread states risks races too. There are two locks held for TSYNC:
-current->sighand->siglockis held to keep new threads from appearing/disappearing, which would destroy filter refcounting and lead to memory corruption.
I don't understand what you mean here.
How can this lead to memory corruption?
Quoted 26 lines
-
cred_guard_mutexis held to keepno_new_privsin sync with filters to avoidno_new_privsand filter confusion during exec, which could lead to exploitable setuid conditions (see below).Just racing a malicious thread during TSYNC is not a very strong example (a malicious thread could do lots of fun things to "current" before it ever got near calling TSYNC), but I think there is the risk of mismatched/confused states that we don't want to allow. One is a particularly bad state that could lead to privilege escalations (in the form of the old "sendmail doesn't check setuid" flaw; if a setuid process has a filter attached that silently fails a priv-dropping setuid call and continues execution with elevated privs, it can be tricked into doing bad things on behalf of the unprivileged parent, which was the primary goal of the original use of
cred_guard_mutexwith TSYNC[1]):thread A clones thread B
thread B starts setuid exec
thread A setsno_new_privs
thread A calls seccomp with TSYNC
thread A inseccomp_sync_threads()sets seccomp filter on self and thread B thread B passescheck_unsafe_exec()withno_new_privsunset thread B reachesbprm_fill_uid()withno_new_privsunset and gains privs thread A still inseccomp_sync_threads()setsno_new_privson thread B thread B finishes exec, now running with elevated privs, a filter chosen
by thread A, _and_ nnp set (which doesn't matter)
Quoted 4 lines
With the original locking, thread B will fail
check_unsafe_exec()because filter and nnp state are changed together, with "atomicity" protected by thecred_guard_mutex.
Ah, good point, thanks!
This can be fixed by checking current->signal->cred_locked_for_ptrace while the cred_guard_mutex is locked, like this for instance:
kernel/seccomp.c +3 -0
@@ -342,6 +342,9 @@ static inline pid_t seccomp_can_sync_threads(void)BUG_ON (!mutex_is_locked (¤t->signal->cred_guard_mutex));assert_spin_locked (¤t->sighand->siglock); +if (current->signal->cred_locked_for_ptrace) +return -EAGAIN ; +/* Validate all threads being eligible for synchronization. */ caller = current;for_each_thread (caller, thread) {
Quoted 11 lines
And this is just the bad state I _can_ see. I'm worried there are more...
All this said, I do see a small similarity here to the work I did to stabilize stack rlimits (there was an ongoing problem with making multiple decisions for the bprm based on current's state -- but current's state was mutable during exec). For this, I saved
rlim_stackto bprm and ignored current's copy until exec ended and then stored bprm's copy into current. If the only problem anyone can see here is the handling ofno_new_privs, we might be able to solve that similarly, at least disentangling tsync/nnp fromcred_guard_mutex.
I still think that is solvable with using cred_locked_for_ptrace and simply make the tsync fail if it would otherwise be blocked.
Thanks
Bernd.
Quoted 10 lines · Bernd Edlinger
Quoted 9 lines · Kees Cook
Quoted 1 line · Bernd Edlinger
[...]
If I'm reading this patch correctly, this changes the lifetime of the
cred_guard_mutexlock to be:
- duringprepare_bprm_creds()
- fromflush_old_exec()throughinstall_exec_creds()
Before,cred_guard_mutexwas held fromprepare_bprm_creds()throughinstall_exec_creds().
BTW, I think the effect of this change (i.e. my paragraph above) should be distinctly called out in the commit log if this solution moves forward.
Quoted 9 lines
Quoted 7 lines
That means, for example, that check_unsafe_exec()'s documented invariant is violated:
/*
* determine how safe it is to execute the proposed program
* - the caller must hold ->cred_guard_mutex to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/Oh, right, I haven't understood that hint...
I know no_new_privs is checked there, but I haven't studied the PTRACE_ATTACH part of that comment. If that is handled with the new check, this comment should be updated.
Quoted 11 lines
Quoted 4 lines
I think it also means that the potentially multiple invocations of
bprm_fill_uid()(viaprepare_binprm()viabinfmt_script.candbinfmt_misc.c) would be changingbprm->creddetails (uid, gid) without a lock (another place where current'sno_new_privsis evaluated).So
no_new_privscan change from 0->1, but should not when execve is running.As long as the calling thread is in execve it won't do this, and the only other place, where it may set for other threads is in
seccomp_sync_threads, but that can easily be avoided see below.
Yeah, everything was fine until I had to go complicate things with TSYNC. ;) The real goal is making sure an exec cannot gain privs while later gaining a seccomp filter from an unpriv process. The no_new_privs flag was used to control this, but it required that the filter not get applied during exec.
Quoted 6 lines
Quoted 6 lines
Related, it also means that
cred_guard_mutexis unheld for every invocation ofsearch_binary_handler()(which can loop via the previously mentionedbinfmt_script.candbinfmt_misc.c), if any of them have hidden dependencies oncred_guard_mutex. (Thought I only seebprm_fill_uid()currently.)
Quoted 8 lines
Quoted 5 lines
For seccomp, the expectations about existing thread states risks races too. There are two locks held for TSYNC:
-current->sighand->siglockis held to keep new threads from appearing/disappearing, which would destroy filter refcounting and lead to memory corruption.I don't understand what you mean here.
How can this lead to memory corruption?
Mainly this is a matter of how seccomp manages its filter hierarchy (since the filters are shared through process ancestry), so if a thread appears in the middle of TSYNC it may be racing another TSYNC and break ancestry, leading to bad reference counting on process death, etc. (Though, yes, with refcount_t now, things should never corrupt, just waste memory.)
Quoted 41 lines
Quoted 26 lines
-
cred_guard_mutexis held to keepno_new_privsin sync with filters to avoidno_new_privsand filter confusion during exec, which could lead to exploitable setuid conditions (see below).Just racing a malicious thread during TSYNC is not a very strong example (a malicious thread could do lots of fun things to "current" before it ever got near calling TSYNC), but I think there is the risk of mismatched/confused states that we don't want to allow. One is a particularly bad state that could lead to privilege escalations (in the form of the old "sendmail doesn't check setuid" flaw; if a setuid process has a filter attached that silently fails a priv-dropping setuid call and continues execution with elevated privs, it can be tricked into doing bad things on behalf of the unprivileged parent, which was the primary goal of the original use of
cred_guard_mutexwith TSYNC[1]):thread A clones thread B
thread B starts setuid exec
thread A setsno_new_privs
thread A calls seccomp with TSYNC
thread A inseccomp_sync_threads()sets seccomp filter on self and thread B thread B passescheck_unsafe_exec()withno_new_privsunset thread B reachesbprm_fill_uid()withno_new_privsunset and gains privs thread A still inseccomp_sync_threads()setsno_new_privson thread B thread B finishes exec, now running with elevated privs, a filter chosen
by thread A, _and_ nnp set (which doesn't matter)Quoted 4 lines
With the original locking, thread B will fail
check_unsafe_exec()because filter and nnp state are changed together, with "atomicity" protected by thecred_guard_mutex.Ah, good point, thanks!
This can be fixed by checking
current->signal->cred_locked_for_ptracewhile thecred_guard_mutexis locked, like this for instance:kernel/seccomp.c +0
-0
kernel/seccomp.c static inline pid_t seccomp_can_sync_threads(void) · line 347
Quoted 6 lines
kernel/seccomp.c +3
-0@@ -342,6 +342,9 @@ static inline pid_t seccomp_can_sync_threads(void)BUG_ON (!mutex_is_locked (¤t->signal->cred_guard_mutex));assert_spin_locked (¤t->sighand->siglock); +if (current->signal->cred_locked_for_ptrace) +return -EAGAIN ; +
Hmm. I guess something like that could work. TSYNC expects to be able to report _which_ thread wrecked the call, though... I wonder if in_execve could be used to figure out the offending thread. Hm, nope, that would be outside of lock too (and all users are "current" right now, so the lock wasn't needed before).
Quoted 17 lines
kernel/seccomp.c +0
-0/* Validate all threads being eligible for synchronization. */ caller = current;for_each_thread (caller, thread) {Quoted 4 lines
And this is just the bad state I _can_ see. I'm worried there are more...
Quoted 9 lines
All this said, I do see a small similarity here to the work I did to stabilize stack rlimits (there was an ongoing problem with making multiple decisions for the bprm based on current's state -- but current's state was mutable during exec). For this, I saved
rlim_stackto bprm and ignored current's copy until exec ended and then stored bprm's copy into current. If the only problem anyone can see here is the handling ofno_new_privs, we might be able to solve that similarly, at least disentangling tsync/nnp fromcred_guard_mutex.
Quoted 2 lines
I still think that is solvable with using
cred_locked_for_ptraceand simply make the tsync fail if it would otherwise be blocked.
I wonder if we can find a better name than "cred_locked_for_ptrace"? Maybe "cred_unfinished" or "cred_locked_in_exec" or something?
And the comment on bool cred_locked_for_ptrace should mention that access is only allowed under cred_guard_mutex lock.
Quoted 1 line
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +1
-0+ sig->cred_locked_for_ptrace = false;
This is redundant to the zalloc -- I think you can drop it (unless someone wants to keep it for clarify?)
Also, I think cred_locked_for_ptrace needs checking deeper, in __ptrace_may_access(), not in ptrace_attach(), since LOTS of things make calls to ptrace_may_access() holding cred_guard_mutex, expecting that to be sufficient to see a stable version of the thread...
(I remain very nervous about weakening cred_guard_mutex without addressing the many many users...)
Signature · 1 line
Kees Cook
Quoted 12 lines · Kees Cook
Quoted 10 lines · Bernd Edlinger
Quoted 9 lines · Kees Cook
Quoted 1 line · Bernd Edlinger
[...]
If I'm reading this patch correctly, this changes the lifetime of the
cred_guard_mutexlock to be:
- duringprepare_bprm_creds()
- fromflush_old_exec()throughinstall_exec_creds()
Before,cred_guard_mutexwas held fromprepare_bprm_creds()throughinstall_exec_creds().
Quoted 4 lines
BTW, I think the effect of this change (i.e. my paragraph above) should be distinctly called out in the commit log if this solution moves forward.
Okay, will do.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
That means, for example, that check_unsafe_exec()'s documented invariant is violated:
/*
* determine how safe it is to execute the proposed program
* - the caller must hold ->cred_guard_mutex to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/
Quoted 6 lines
Quoted 1 line
Oh, right, I haven't understood that hint...
I know
no_new_privsis checked there, but I haven't studied thePTRACE_ATTACHpart of that comment. If that is handled with the new check, this comment should be updated.
Okay, I change that comment to:
/*
* determine how safe it is to execute the proposed program
* - the caller must have set ->cred_locked_in_execve to protect against
* PTRACE_ATTACH or seccomp thread-sync
*/
Quoted 33 lines
Quoted 11 lines
Quoted 4 lines
I think it also means that the potentially multiple invocations of
bprm_fill_uid()(viaprepare_binprm()viabinfmt_script.candbinfmt_misc.c) would be changingbprm->creddetails (uid, gid) without a lock (another place where current'sno_new_privsis evaluated).So
no_new_privscan change from 0->1, but should not when execve is running.As long as the calling thread is in execve it won't do this, and the only other place, where it may set for other threads is in
seccomp_sync_threads, but that can easily be avoided see below.Yeah, everything was fine until I had to go complicate things with TSYNC. ;) The real goal is making sure an exec cannot gain privs while later gaining a seccomp filter from an unpriv process. The
no_new_privsflag was used to control this, but it required that the filter not get applied during exec.Quoted 6 lines
Quoted 6 lines
Related, it also means that
cred_guard_mutexis unheld for every invocation ofsearch_binary_handler()(which can loop via the previously mentionedbinfmt_script.candbinfmt_misc.c), if any of them have hidden dependencies oncred_guard_mutex. (Thought I only seebprm_fill_uid()currently.)Quoted 8 lines
Quoted 5 lines
For seccomp, the expectations about existing thread states risks races too. There are two locks held for TSYNC:
-current->sighand->siglockis held to keep new threads from appearing/disappearing, which would destroy filter refcounting and lead to memory corruption.I don't understand what you mean here.
How can this lead to memory corruption?
Quoted 7 lines
Mainly this is a matter of how seccomp manages its filter hierarchy (since the filters are shared through process ancestry), so if a thread appears in the middle of TSYNC it may be racing another TSYNC and break ancestry, leading to bad reference counting on process death, etc. (Though, yes, with
refcount_tnow, things should never corrupt, just waste memory.)
I assume for now, that the current->sighand->siglock held while iterating all threads is sufficient here.
Quoted 44 lines
Quoted 44 lines
Quoted 26 lines
-
cred_guard_mutexis held to keepno_new_privsin sync with filters to avoidno_new_privsand filter confusion during exec, which could lead to exploitable setuid conditions (see below).Just racing a malicious thread during TSYNC is not a very strong example (a malicious thread could do lots of fun things to "current" before it ever got near calling TSYNC), but I think there is the risk of mismatched/confused states that we don't want to allow. One is a particularly bad state that could lead to privilege escalations (in the form of the old "sendmail doesn't check setuid" flaw; if a setuid process has a filter attached that silently fails a priv-dropping setuid call and continues execution with elevated privs, it can be tricked into doing bad things on behalf of the unprivileged parent, which was the primary goal of the original use of
cred_guard_mutexwith TSYNC[1]):thread A clones thread B
thread B starts setuid exec
thread A setsno_new_privs
thread A calls seccomp with TSYNC
thread A inseccomp_sync_threads()sets seccomp filter on self and thread B thread B passescheck_unsafe_exec()withno_new_privsunset thread B reachesbprm_fill_uid()withno_new_privsunset and gains privs thread A still inseccomp_sync_threads()setsno_new_privson thread B thread B finishes exec, now running with elevated privs, a filter chosen
by thread A, _and_ nnp set (which doesn't matter)Quoted 4 lines
With the original locking, thread B will fail
check_unsafe_exec()because filter and nnp state are changed together, with "atomicity" protected by thecred_guard_mutex.Ah, good point, thanks!
This can be fixed by checking
current->signal->cred_locked_for_ptracewhile thecred_guard_mutexis locked, like this for instance:kernel/seccomp.c +0
-0@@ -342,6 +342,9 @@ static inline pid_t seccomp_can_sync_threads(void)BUG_ON (!mutex_is_locked (¤t->signal->cred_guard_mutex));assert_spin_locked (¤t->sighand->siglock);
Quoted 10 lines
kernel/seccomp.c static inline pid_t seccomp_can_sync_threads(void) · line 347
Quoted 3 lines
kernel/seccomp.c +3
-0+if (current->signal->cred_locked_for_ptrace) +return -EAGAIN ; +Hmm. I guess something like that could work. TSYNC expects to be able to report _which_ thread wrecked the call, though... I wonder if
in_execvecould be used to figure out the offending thread. Hm, nope, that would be outside of lock too (and all users are "current" right now, so the lock wasn't needed before).
I could move that in_execve = 1 to prepare_bprm_creds, if it really matters, but the caller will die quickly and cannot do anything with that information when another thread executes execve, right?
Quoted 17 lines
Quoted 17 lines
kernel/seccomp.c +0
-0/* Validate all threads being eligible for synchronization. */ caller = current;for_each_thread (caller, thread) {Quoted 13 lines
And this is just the bad state I _can_ see. I'm worried there are more...
All this said, I do see a small similarity here to the work I did to stabilize stack rlimits (there was an ongoing problem with making multiple decisions for the bprm based on current's state -- but current's state was mutable during exec). For this, I saved
rlim_stackto bprm and ignored current's copy until exec ended and then stored bprm's copy into current. If the only problem anyone can see here is the handling ofno_new_privs, we might be able to solve that similarly, at least disentangling tsync/nnp fromcred_guard_mutex.
Quoted 6 lines
Quoted 2 lines
I still think that is solvable with using
cred_locked_for_ptraceand simply make the tsync fail if it would otherwise be blocked.I wonder if we can find a better name than "
cred_locked_for_ptrace"? Maybe "cred_unfinished" or "cred_locked_in_exec" or something?
Yeah, I'd go with "cred_locked_in_execve".
Quoted 3 lines
And the comment on bool
cred_locked_for_ptraceshould mention that access is only allowed undercred_guard_mutexlock.
okay.
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +1
-0+ sig->cred_locked_for_ptrace = false;This is redundant to the zalloc -- I think you can drop it (unless someone wants to keep it for clarify?)
I'll remove that here and in init/init_task.c
Quoted 5 lines
Also, I think
cred_locked_for_ptraceneeds checking deeper, in__ptrace_may_access(), not inptrace_attach(), since LOTS of things make calls toptrace_may_access()holdingcred_guard_mutex, expecting that to be sufficient to see a stable version of the thread...
No, these need to be addressed individually, but most users just want to know if the current credentials are sufficient at this moment, but will not change the credentials, as ptrace and TSYNC do.
BTW: Not all users have cred_guard_mutex, see mm/migrate.c, mm/mempolicy.c, kernel/futex.c, fs/proc/namespaces.c etc. So adding an access to cred_locked_for_execve in ptrace_may_access is probably not an option.
However, one nice added value by this change is this:
void *thread(void *arg)
{
ptrace(PTRACE_TRACEME, 0,0,0);
return NULL;
}int main(void)
{
int pid = fork();
if (!pid) {
pthread_t pt;
pthread_create(&pt, NULL, thread, NULL);
pthread_join(pt, NULL);
execlp("echo", "echo", "passed", NULL);
}
sleep(1000);
ptrace(PTRACE_ATTACH, pid, 0,0);
kill(pid, SIGCONT);
return 0;
}cat /proc/3812/stack
[<0>] flush_old_exec+0xbf/0x760 [<0>] load_elf_binary+0x35a/0x16c0 [<0>] search_binary_handler+0x97/0x1d0 [<0>] __do_execve_file.isra.40+0x624/0x920 [<0>] __x64_sys_execve+0x49/0x60 [<0>] do_syscall_64+0x64/0x220 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 3 lines
(I remain very nervous about weakening
cred_guard_mutexwithout addressing the many many users...)
They need to be looked at closely, that's pretty clear. Most fall in the class, that just the current credentials need to stay stable for a certain time.
Bernd.
Quoted 236 lines · Bernd Edlinger
Quoted 12 lines · Kees Cook
Quoted 10 lines · Bernd Edlinger
Quoted 9 lines · Kees Cook
Quoted 1 line · Bernd Edlinger
[...]
If I'm reading this patch correctly, this changes the lifetime of the
cred_guard_mutexlock to be:
- duringprepare_bprm_creds()
- fromflush_old_exec()throughinstall_exec_creds()
Before,cred_guard_mutexwas held fromprepare_bprm_creds()throughinstall_exec_creds().Quoted 4 lines
BTW, I think the effect of this change (i.e. my paragraph above) should be distinctly called out in the commit log if this solution moves forward.
Okay, will do.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
That means, for example, that check_unsafe_exec()'s documented invariant is violated:
/*
* determine how safe it is to execute the proposed program
* - the caller must hold ->cred_guard_mutex to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/Quoted 6 lines
Quoted 1 line
Oh, right, I haven't understood that hint...
I know
no_new_privsis checked there, but I haven't studied thePTRACE_ATTACHpart of that comment. If that is handled with the new check, this comment should be updated.Okay, I change that comment to:
/*
* determine how safe it is to execute the proposed program
* - the caller must have set ->cred_locked_in_execve to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/Quoted 33 lines
Quoted 11 lines
Quoted 4 lines
I think it also means that the potentially multiple invocations of
bprm_fill_uid()(viaprepare_binprm()viabinfmt_script.candbinfmt_misc.c) would be changingbprm->creddetails (uid, gid) without a lock (another place where current'sno_new_privsis evaluated).So
no_new_privscan change from 0->1, but should not when execve is running.As long as the calling thread is in execve it won't do this, and the only other place, where it may set for other threads is in
seccomp_sync_threads, but that can easily be avoided see below.Yeah, everything was fine until I had to go complicate things with TSYNC. ;) The real goal is making sure an exec cannot gain privs while later gaining a seccomp filter from an unpriv process. The
no_new_privsflag was used to control this, but it required that the filter not get applied during exec.Quoted 6 lines
Quoted 6 lines
Related, it also means that
cred_guard_mutexis unheld for every invocation ofsearch_binary_handler()(which can loop via the previously mentionedbinfmt_script.candbinfmt_misc.c), if any of them have hidden dependencies oncred_guard_mutex. (Thought I only seebprm_fill_uid()currently.)Quoted 8 lines
Quoted 5 lines
For seccomp, the expectations about existing thread states risks races too. There are two locks held for TSYNC:
-current->sighand->siglockis held to keep new threads from appearing/disappearing, which would destroy filter refcounting and lead to memory corruption.I don't understand what you mean here.
How can this lead to memory corruption?Quoted 7 lines
Mainly this is a matter of how seccomp manages its filter hierarchy (since the filters are shared through process ancestry), so if a thread appears in the middle of TSYNC it may be racing another TSYNC and break ancestry, leading to bad reference counting on process death, etc. (Though, yes, with
refcount_tnow, things should never corrupt, just waste memory.)I assume for now, that the
current->sighand->siglockheld while iterating all threads is sufficient here.Quoted 44 lines
Quoted 44 lines
Quoted 26 lines
-
cred_guard_mutexis held to keepno_new_privsin sync with filters to avoidno_new_privsand filter confusion during exec, which could lead to exploitable setuid conditions (see below).Just racing a malicious thread during TSYNC is not a very strong example (a malicious thread could do lots of fun things to "current" before it ever got near calling TSYNC), but I think there is the risk of mismatched/confused states that we don't want to allow. One is a particularly bad state that could lead to privilege escalations (in the form of the old "sendmail doesn't check setuid" flaw; if a setuid process has a filter attached that silently fails a priv-dropping setuid call and continues execution with elevated privs, it can be tricked into doing bad things on behalf of the unprivileged parent, which was the primary goal of the original use of
cred_guard_mutexwith TSYNC[1]):thread A clones thread B
thread B starts setuid exec
thread A setsno_new_privs
thread A calls seccomp with TSYNC
thread A inseccomp_sync_threads()sets seccomp filter on self and thread B thread B passescheck_unsafe_exec()withno_new_privsunset thread B reachesbprm_fill_uid()withno_new_privsunset and gains privs thread A still inseccomp_sync_threads()setsno_new_privson thread B thread B finishes exec, now running with elevated privs, a filter chosen
by thread A, _and_ nnp set (which doesn't matter)Quoted 4 lines
With the original locking, thread B will fail
check_unsafe_exec()because filter and nnp state are changed together, with "atomicity" protected by thecred_guard_mutex.Ah, good point, thanks!
This can be fixed by checking
current->signal->cred_locked_for_ptracewhile thecred_guard_mutexis locked, like this for instance:kernel/seccomp.c +0
-0@@ -342,6 +342,9 @@ static inline pid_t seccomp_can_sync_threads(void)BUG_ON (!mutex_is_locked (¤t->signal->cred_guard_mutex));assert_spin_locked (¤t->sighand->siglock);Quoted 10 lines
kernel/seccomp.c static inline pid_t seccomp_can_sync_threads(void) · line 347
Quoted 3 lines
kernel/seccomp.c +3
-0+if (current->signal->cred_locked_for_ptrace) +return -EAGAIN ; +Hmm. I guess something like that could work. TSYNC expects to be able to report _which_ thread wrecked the call, though... I wonder if
in_execvecould be used to figure out the offending thread. Hm, nope, that would be outside of lock too (and all users are "current" right now, so the lock wasn't needed before).I could move that
in_execve= 1 toprepare_bprm_creds, if it really matters, but the caller will die quickly and cannot do anything with that information when another thread executes execve, right?Quoted 17 lines
Quoted 17 lines
kernel/seccomp.c +0
-0/* Validate all threads being eligible for synchronization. */ caller = current;for_each_thread (caller, thread) {Quoted 13 lines
And this is just the bad state I _can_ see. I'm worried there are more...
All this said, I do see a small similarity here to the work I did to stabilize stack rlimits (there was an ongoing problem with making multiple decisions for the bprm based on current's state -- but current's state was mutable during exec). For this, I saved
rlim_stackto bprm and ignored current's copy until exec ended and then stored bprm's copy into current. If the only problem anyone can see here is the handling ofno_new_privs, we might be able to solve that similarly, at least disentangling tsync/nnp fromcred_guard_mutex.Quoted 6 lines
Quoted 2 lines
I still think that is solvable with using
cred_locked_for_ptraceand simply make the tsync fail if it would otherwise be blocked.I wonder if we can find a better name than "
cred_locked_for_ptrace"? Maybe "cred_unfinished" or "cred_locked_in_exec" or something?Yeah, I'd go with "
cred_locked_in_execve".Quoted 3 lines
And the comment on bool
cred_locked_for_ptraceshould mention that access is only allowed undercred_guard_mutexlock.okay.
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +1
-0+ sig->cred_locked_for_ptrace = false;This is redundant to the zalloc -- I think you can drop it (unless someone wants to keep it for clarify?)
I'll remove that here and in
init/init_task.cQuoted 5 lines
Also, I think
cred_locked_for_ptraceneeds checking deeper, in__ptrace_may_access(), not inptrace_attach(), since LOTS of things make calls toptrace_may_access()holdingcred_guard_mutex, expecting that to be sufficient to see a stable version of the thread...No, these need to be addressed individually, but most users just want to know if the current credentials are sufficient at this moment, but will not change the credentials, as ptrace and TSYNC do.
BTW: Not all users have
cred_guard_mutex, seemm/migrate.c,mm/mempolicy.c,kernel/futex.c,fs/proc/namespaces.cetc. So adding an access tocred_locked_for_execveinptrace_may_accessis probably not an option.However, one nice added value by this change is this:
void *thread(void *arg)
{ ptrace(PTRACE_TRACEME, 0,0,0); return NULL; }int main(void)
{ int pid = fork(); if (!pid) { pthread_t pt; pthread_create(&pt, NULL, thread, NULL); pthread_join(pt, NULL); execlp("echo", "echo", "passed", NULL); } sleep(1000); ptrace(PTRACE_ATTACH, pid, 0,0); kill(pid, SIGCONT); return 0; }cat
/proc/3812/stack[<0>] flush_old_exec+0xbf/0x760 [<0>] load_elf_binary+0x35a/0x16c0 [<0>] search_binary_handler+0x97/0x1d0 [<0>] __do_execve_file.isra.40+0x624/0x920 [<0>] __x64_sys_execve+0x49/0x60 [<0>] do_syscall_64+0x64/0x220 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 7 lines
Quoted 3 lines
(I remain very nervous about weakening
cred_guard_mutexwithout addressing the many many users...)They need to be looked at closely, that's pretty clear. Most fall in the class, that just the current credentials need to stay stable for a certain time.
I remain rather set on wanting some very basic tests with this change. Imho, looking through tools/testing/selftests again we don't have nearly enough for these codepaths; not to say none. Basically, if someone wants to make a change affecting the current problem we should really have at least a single simple test/reproducer that can be run without digging through lore. And hopefully over time we'll have more tests.
Christian
Quoted 245 lines · Christian Brauner
Quoted 236 lines · Bernd Edlinger
Quoted 12 lines · Kees Cook
Quoted 10 lines · Bernd Edlinger
Quoted 9 lines · Kees Cook
Quoted 1 line · Bernd Edlinger
[...]
If I'm reading this patch correctly, this changes the lifetime of the
cred_guard_mutexlock to be:
- duringprepare_bprm_creds()
- fromflush_old_exec()throughinstall_exec_creds()
Before,cred_guard_mutexwas held fromprepare_bprm_creds()throughinstall_exec_creds().Quoted 4 lines
BTW, I think the effect of this change (i.e. my paragraph above) should be distinctly called out in the commit log if this solution moves forward.
Okay, will do.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
That means, for example, that check_unsafe_exec()'s documented invariant is violated:
/*
* determine how safe it is to execute the proposed program
* - the caller must hold ->cred_guard_mutex to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/Quoted 6 lines
Quoted 1 line
Oh, right, I haven't understood that hint...
I know
no_new_privsis checked there, but I haven't studied thePTRACE_ATTACHpart of that comment. If that is handled with the new check, this comment should be updated.Okay, I change that comment to:
/*
* determine how safe it is to execute the proposed program
* - the caller must have set ->cred_locked_in_execve to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/Quoted 33 lines
Quoted 11 lines
Quoted 4 lines
I think it also means that the potentially multiple invocations of
bprm_fill_uid()(viaprepare_binprm()viabinfmt_script.candbinfmt_misc.c) would be changingbprm->creddetails (uid, gid) without a lock (another place where current'sno_new_privsis evaluated).So
no_new_privscan change from 0->1, but should not when execve is running.As long as the calling thread is in execve it won't do this, and the only other place, where it may set for other threads is in
seccomp_sync_threads, but that can easily be avoided see below.Yeah, everything was fine until I had to go complicate things with TSYNC. ;) The real goal is making sure an exec cannot gain privs while later gaining a seccomp filter from an unpriv process. The
no_new_privsflag was used to control this, but it required that the filter not get applied during exec.Quoted 6 lines
Quoted 6 lines
Related, it also means that
cred_guard_mutexis unheld for every invocation ofsearch_binary_handler()(which can loop via the previously mentionedbinfmt_script.candbinfmt_misc.c), if any of them have hidden dependencies oncred_guard_mutex. (Thought I only seebprm_fill_uid()currently.)Quoted 8 lines
Quoted 5 lines
For seccomp, the expectations about existing thread states risks races too. There are two locks held for TSYNC:
-current->sighand->siglockis held to keep new threads from appearing/disappearing, which would destroy filter refcounting and lead to memory corruption.I don't understand what you mean here.
How can this lead to memory corruption?Quoted 7 lines
Mainly this is a matter of how seccomp manages its filter hierarchy (since the filters are shared through process ancestry), so if a thread appears in the middle of TSYNC it may be racing another TSYNC and break ancestry, leading to bad reference counting on process death, etc. (Though, yes, with
refcount_tnow, things should never corrupt, just waste memory.)I assume for now, that the
current->sighand->siglockheld while iterating all threads is sufficient here.Quoted 44 lines
Quoted 44 lines
Quoted 26 lines
-
cred_guard_mutexis held to keepno_new_privsin sync with filters to avoidno_new_privsand filter confusion during exec, which could lead to exploitable setuid conditions (see below).Just racing a malicious thread during TSYNC is not a very strong example (a malicious thread could do lots of fun things to "current" before it ever got near calling TSYNC), but I think there is the risk of mismatched/confused states that we don't want to allow. One is a particularly bad state that could lead to privilege escalations (in the form of the old "sendmail doesn't check setuid" flaw; if a setuid process has a filter attached that silently fails a priv-dropping setuid call and continues execution with elevated privs, it can be tricked into doing bad things on behalf of the unprivileged parent, which was the primary goal of the original use of
cred_guard_mutexwith TSYNC[1]):thread A clones thread B
thread B starts setuid exec
thread A setsno_new_privs
thread A calls seccomp with TSYNC
thread A inseccomp_sync_threads()sets seccomp filter on self and thread B thread B passescheck_unsafe_exec()withno_new_privsunset thread B reachesbprm_fill_uid()withno_new_privsunset and gains privs thread A still inseccomp_sync_threads()setsno_new_privson thread B thread B finishes exec, now running with elevated privs, a filter chosen
by thread A, _and_ nnp set (which doesn't matter)Quoted 4 lines
With the original locking, thread B will fail
check_unsafe_exec()because filter and nnp state are changed together, with "atomicity" protected by thecred_guard_mutex.Ah, good point, thanks!
This can be fixed by checking
current->signal->cred_locked_for_ptracewhile thecred_guard_mutexis locked, like this for instance:kernel/seccomp.c +0
-0@@ -342,6 +342,9 @@ static inline pid_t seccomp_can_sync_threads(void)BUG_ON (!mutex_is_locked (¤t->signal->cred_guard_mutex));assert_spin_locked (¤t->sighand->siglock);Quoted 10 lines
kernel/seccomp.c static inline pid_t seccomp_can_sync_threads(void) · line 347
Quoted 3 lines
kernel/seccomp.c +3
-0+if (current->signal->cred_locked_for_ptrace) +return -EAGAIN ; +Hmm. I guess something like that could work. TSYNC expects to be able to report _which_ thread wrecked the call, though... I wonder if
in_execvecould be used to figure out the offending thread. Hm, nope, that would be outside of lock too (and all users are "current" right now, so the lock wasn't needed before).I could move that
in_execve= 1 toprepare_bprm_creds, if it really matters, but the caller will die quickly and cannot do anything with that information when another thread executes execve, right?Quoted 17 lines
Quoted 17 lines
kernel/seccomp.c +0
-0/* Validate all threads being eligible for synchronization. */ caller = current;for_each_thread (caller, thread) {Quoted 13 lines
And this is just the bad state I _can_ see. I'm worried there are more...
All this said, I do see a small similarity here to the work I did to stabilize stack rlimits (there was an ongoing problem with making multiple decisions for the bprm based on current's state -- but current's state was mutable during exec). For this, I saved
rlim_stackto bprm and ignored current's copy until exec ended and then stored bprm's copy into current. If the only problem anyone can see here is the handling ofno_new_privs, we might be able to solve that similarly, at least disentangling tsync/nnp fromcred_guard_mutex.Quoted 6 lines
Quoted 2 lines
I still think that is solvable with using
cred_locked_for_ptraceand simply make the tsync fail if it would otherwise be blocked.I wonder if we can find a better name than "
cred_locked_for_ptrace"? Maybe "cred_unfinished" or "cred_locked_in_exec" or something?Yeah, I'd go with "
cred_locked_in_execve".Quoted 3 lines
And the comment on bool
cred_locked_for_ptraceshould mention that access is only allowed undercred_guard_mutexlock.okay.
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +1
-0+ sig->cred_locked_for_ptrace = false;This is redundant to the zalloc -- I think you can drop it (unless someone wants to keep it for clarify?)
I'll remove that here and in
init/init_task.cQuoted 5 lines
Also, I think
cred_locked_for_ptraceneeds checking deeper, in__ptrace_may_access(), not inptrace_attach(), since LOTS of things make calls toptrace_may_access()holdingcred_guard_mutex, expecting that to be sufficient to see a stable version of the thread...No, these need to be addressed individually, but most users just want to know if the current credentials are sufficient at this moment, but will not change the credentials, as ptrace and TSYNC do.
BTW: Not all users have
cred_guard_mutex, seemm/migrate.c,mm/mempolicy.c,kernel/futex.c,fs/proc/namespaces.cetc. So adding an access tocred_locked_for_execveinptrace_may_accessis probably not an option.However, one nice added value by this change is this:
void *thread(void *arg)
{ ptrace(PTRACE_TRACEME, 0,0,0); return NULL; }int main(void)
{ int pid = fork(); if (!pid) { pthread_t pt; pthread_create(&pt, NULL, thread, NULL); pthread_join(pt, NULL); execlp("echo", "echo", "passed", NULL); } sleep(1000); ptrace(PTRACE_ATTACH, pid, 0,0); kill(pid, SIGCONT); return 0; }cat
/proc/3812/stack[<0>] flush_old_exec+0xbf/0x760 [<0>] load_elf_binary+0x35a/0x16c0 [<0>] search_binary_handler+0x97/0x1d0 [<0>] __do_execve_file.isra.40+0x624/0x920 [<0>] __x64_sys_execve+0x49/0x60 [<0>] do_syscall_64+0x64/0x220 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9Quoted 7 lines
Quoted 3 lines
(I remain very nervous about weakening
cred_guard_mutexwithout addressing the many many users...)They need to be looked at closely, that's pretty clear. Most fall in the class, that just the current credentials need to stay stable for a certain time.
Quoted 6 lines
I remain rather set on wanting some very basic tests with this change. Imho, looking through
tools/testing/selftestsagain we don't have nearly enough for these codepaths; not to say none. Basically, if someone wants to make a change affecting the current problem we should really have at least a single simple test/reproducer that can be run without digging through lore. And hopefully over time we'll have more tests.
Which you added in v4. Which is great! (I should've mentioned this in my first mail.)
Christian
Quoted 192 lines · Bernd Edlinger
Quoted 12 lines · Kees Cook
Quoted 10 lines · Bernd Edlinger
Quoted 9 lines · Kees Cook
Quoted 1 line · Bernd Edlinger
[...]
If I'm reading this patch correctly, this changes the lifetime of the
cred_guard_mutexlock to be:
- duringprepare_bprm_creds()
- fromflush_old_exec()throughinstall_exec_creds()
Before,cred_guard_mutexwas held fromprepare_bprm_creds()throughinstall_exec_creds().Quoted 4 lines
BTW, I think the effect of this change (i.e. my paragraph above) should be distinctly called out in the commit log if this solution moves forward.
Okay, will do.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
That means, for example, that check_unsafe_exec()'s documented invariant is violated:
/*
* determine how safe it is to execute the proposed program
* - the caller must hold ->cred_guard_mutex to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/Quoted 6 lines
Quoted 1 line
Oh, right, I haven't understood that hint...
I know
no_new_privsis checked there, but I haven't studied thePTRACE_ATTACHpart of that comment. If that is handled with the new check, this comment should be updated.Okay, I change that comment to:
/*
* determine how safe it is to execute the proposed program
* - the caller must have set ->cred_locked_in_execve to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/Quoted 33 lines
Quoted 11 lines
Quoted 4 lines
I think it also means that the potentially multiple invocations of
bprm_fill_uid()(viaprepare_binprm()viabinfmt_script.candbinfmt_misc.c) would be changingbprm->creddetails (uid, gid) without a lock (another place where current'sno_new_privsis evaluated).So
no_new_privscan change from 0->1, but should not when execve is running.As long as the calling thread is in execve it won't do this, and the only other place, where it may set for other threads is in
seccomp_sync_threads, but that can easily be avoided see below.Yeah, everything was fine until I had to go complicate things with TSYNC. ;) The real goal is making sure an exec cannot gain privs while later gaining a seccomp filter from an unpriv process. The
no_new_privsflag was used to control this, but it required that the filter not get applied during exec.Quoted 6 lines
Quoted 6 lines
Related, it also means that
cred_guard_mutexis unheld for every invocation ofsearch_binary_handler()(which can loop via the previously mentionedbinfmt_script.candbinfmt_misc.c), if any of them have hidden dependencies oncred_guard_mutex. (Thought I only seebprm_fill_uid()currently.)Quoted 8 lines
Quoted 5 lines
For seccomp, the expectations about existing thread states risks races too. There are two locks held for TSYNC:
-current->sighand->siglockis held to keep new threads from appearing/disappearing, which would destroy filter refcounting and lead to memory corruption.I don't understand what you mean here.
How can this lead to memory corruption?Quoted 7 lines
Mainly this is a matter of how seccomp manages its filter hierarchy (since the filters are shared through process ancestry), so if a thread appears in the middle of TSYNC it may be racing another TSYNC and break ancestry, leading to bad reference counting on process death, etc. (Though, yes, with
refcount_tnow, things should never corrupt, just waste memory.)I assume for now, that the
current->sighand->siglockheld while iterating all threads is sufficient here.Quoted 44 lines
Quoted 44 lines
Quoted 26 lines
-
cred_guard_mutexis held to keepno_new_privsin sync with filters to avoidno_new_privsand filter confusion during exec, which could lead to exploitable setuid conditions (see below).Just racing a malicious thread during TSYNC is not a very strong example (a malicious thread could do lots of fun things to "current" before it ever got near calling TSYNC), but I think there is the risk of mismatched/confused states that we don't want to allow. One is a particularly bad state that could lead to privilege escalations (in the form of the old "sendmail doesn't check setuid" flaw; if a setuid process has a filter attached that silently fails a priv-dropping setuid call and continues execution with elevated privs, it can be tricked into doing bad things on behalf of the unprivileged parent, which was the primary goal of the original use of
cred_guard_mutexwith TSYNC[1]):thread A clones thread B
thread B starts setuid exec
thread A setsno_new_privs
thread A calls seccomp with TSYNC
thread A inseccomp_sync_threads()sets seccomp filter on self and thread B thread B passescheck_unsafe_exec()withno_new_privsunset thread B reachesbprm_fill_uid()withno_new_privsunset and gains privs thread A still inseccomp_sync_threads()setsno_new_privson thread B thread B finishes exec, now running with elevated privs, a filter chosen
by thread A, _and_ nnp set (which doesn't matter)Quoted 4 lines
With the original locking, thread B will fail
check_unsafe_exec()because filter and nnp state are changed together, with "atomicity" protected by thecred_guard_mutex.Ah, good point, thanks!
This can be fixed by checking
current->signal->cred_locked_for_ptracewhile thecred_guard_mutexis locked, like this for instance:kernel/seccomp.c +0
-0@@ -342,6 +342,9 @@ static inline pid_t seccomp_can_sync_threads(void)BUG_ON (!mutex_is_locked (¤t->signal->cred_guard_mutex));assert_spin_locked (¤t->sighand->siglock);Quoted 10 lines
kernel/seccomp.c static inline pid_t seccomp_can_sync_threads(void) · line 347
Quoted 3 lines
kernel/seccomp.c +3
-0+if (current->signal->cred_locked_for_ptrace) +return -EAGAIN ; +Hmm. I guess something like that could work. TSYNC expects to be able to report _which_ thread wrecked the call, though... I wonder if
in_execvecould be used to figure out the offending thread. Hm, nope, that would be outside of lock too (and all users are "current" right now, so the lock wasn't needed before).I could move that
in_execve= 1 toprepare_bprm_creds, if it really matters, but the caller will die quickly and cannot do anything with that information when another thread executes execve, right?Quoted 17 lines
Quoted 17 lines
kernel/seccomp.c +0
-0/* Validate all threads being eligible for synchronization. */ caller = current;for_each_thread (caller, thread) {Quoted 13 lines
And this is just the bad state I _can_ see. I'm worried there are more...
All this said, I do see a small similarity here to the work I did to stabilize stack rlimits (there was an ongoing problem with making multiple decisions for the bprm based on current's state -- but current's state was mutable during exec). For this, I saved
rlim_stackto bprm and ignored current's copy until exec ended and then stored bprm's copy into current. If the only problem anyone can see here is the handling ofno_new_privs, we might be able to solve that similarly, at least disentangling tsync/nnp fromcred_guard_mutex.Quoted 6 lines
Quoted 2 lines
I still think that is solvable with using
cred_locked_for_ptraceand simply make the tsync fail if it would otherwise be blocked.I wonder if we can find a better name than "
cred_locked_for_ptrace"? Maybe "cred_unfinished" or "cred_locked_in_exec" or something?Yeah, I'd go with "
cred_locked_in_execve".Quoted 3 lines
And the comment on bool
cred_locked_for_ptraceshould mention that access is only allowed undercred_guard_mutexlock.okay.
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +1
-0+ sig->cred_locked_for_ptrace = false;This is redundant to the zalloc -- I think you can drop it (unless someone wants to keep it for clarify?)
I'll remove that here and in
init/init_task.cQuoted 5 lines
Also, I think
cred_locked_for_ptraceneeds checking deeper, in__ptrace_may_access(), not inptrace_attach(), since LOTS of things make calls toptrace_may_access()holdingcred_guard_mutex, expecting that to be sufficient to see a stable version of the thread...
Quoted 8 lines
No, these need to be addressed individually, but most users just want to know if the current credentials are sufficient at this moment, but will not change the credentials, as ptrace and TSYNC do.
BTW: Not all users have
cred_guard_mutex, seemm/migrate.c,mm/mempolicy.c,kernel/futex.c,fs/proc/namespaces.cetc. So adding an access tocred_locked_for_execveinptrace_may_accessis probably not an option.
That could be solved by e.g. adding ptrace_may_access_{no}exec() taking cred_guard_mutex.
Quoted 73 lines · Kees Cook
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9Quoted 5 lines
The proposed solution is to take the
cred_guard_mutexonly in a critical section at the beginning, and at the end of the execve function, and letPTRACE_ATTACHfail with EAGAIN while execve is not complete, but other functions likevm_accessare allowed to complete normally.Sorry to be bummer, but I don't think this will work. A few more things during the exec process depend on
cred_guard_mutexbeing held.If I'm reading this patch correctly, this changes the lifetime of the
cred_guard_mutexlock to be:
- duringprepare_bprm_creds()
- fromflush_old_exec()throughinstall_exec_creds()
Before,cred_guard_mutexwas held fromprepare_bprm_creds()throughinstall_exec_creds().That means, for example, that check_unsafe_exec()'s documented invariant is violated:
/*
* determine how safe it is to execute the proposed program
* - the caller must hold ->cred_guard_mutex to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/
static voidcheck_unsafe_exec(struct linux_binprm *bprm)...
which is looking atno_new_privsas well as other details, and making decisions about the bprm state from the current state.I think it also means that the potentially multiple invocations of
bprm_fill_uid()(viaprepare_binprm()viabinfmt_script.candbinfmt_misc.c) would be changingbprm->creddetails (uid, gid) without a lock (another place where current'sno_new_privsis evaluated).
Quoted 5 lines
Related, it also means that
cred_guard_mutexis unheld for every invocation ofsearch_binary_handler()(which can loop via the previously mentionedbinfmt_script.candbinfmt_misc.c), if any of them have hidden dependencies oncred_guard_mutex. (Thought I only seebprm_fill_uid()currently.)
So one issue I see with having to reacquire the cred_guard_mutex might be that this would allow tasks holding the cred_guard_mutex to block a killed exec'ing task from exiting, right?
Quoted 80 lines · Christian Brauner
Quoted 73 lines · Kees Cook
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9Quoted 5 lines
The proposed solution is to take the
cred_guard_mutexonly in a critical section at the beginning, and at the end of the execve function, and letPTRACE_ATTACHfail with EAGAIN while execve is not complete, but other functions likevm_accessare allowed to complete normally.Sorry to be bummer, but I don't think this will work. A few more things during the exec process depend on
cred_guard_mutexbeing held.If I'm reading this patch correctly, this changes the lifetime of the
cred_guard_mutexlock to be:
- duringprepare_bprm_creds()
- fromflush_old_exec()throughinstall_exec_creds()
Before,cred_guard_mutexwas held fromprepare_bprm_creds()throughinstall_exec_creds().That means, for example, that check_unsafe_exec()'s documented invariant is violated:
/*
* determine how safe it is to execute the proposed program
* - the caller must hold ->cred_guard_mutex to protect against
*PTRACE_ATTACHor seccomp thread-sync
*/
static voidcheck_unsafe_exec(struct linux_binprm *bprm)...
which is looking atno_new_privsas well as other details, and making decisions about the bprm state from the current state.I think it also means that the potentially multiple invocations of
bprm_fill_uid()(viaprepare_binprm()viabinfmt_script.candbinfmt_misc.c) would be changingbprm->creddetails (uid, gid) without a lock (another place where current'sno_new_privsis evaluated).Quoted 5 lines
Related, it also means that
cred_guard_mutexis unheld for every invocation ofsearch_binary_handler()(which can loop via the previously mentionedbinfmt_script.candbinfmt_misc.c), if any of them have hidden dependencies oncred_guard_mutex. (Thought I only seebprm_fill_uid()currently.)
Quoted 4 lines
So one issue I see with having to reacquire the
cred_guard_mutexmight be that this would allow tasks holding thecred_guard_mutexto block a killed exec'ing task from exiting, right?
Yes maybe, but I think it will not be worse than it is now. Since the second time the mutex is acquired it is done with mutex_lock_killable, so at least kill -9 should get it terminated.
Bernd.
Quoted 15 lines · Bernd Edlinger
Quoted 4 lines · Christian Brauner
So one issue I see with having to reacquire the
cred_guard_mutexmight be that this would allow tasks holding thecred_guard_mutexto block a killed exec'ing task from exiting, right?Yes maybe, but I think it will not be worse than it is now. Since the second time the mutex is acquired it is done with
mutex_lock_killable, so at least kill -9 should get it terminated.static void
free_bprm(struct linux_binprm *bprm)
{
fs/exec.c void finalize_exec(struct linux_binprm *bprm) · line 1442
Quoted 6 lines
fs/exec.c +3
-0free_arg_pages (bprm);if (bprm->cred) { +if (!bprm->called_flush_old_exec) +mutex_lock (¤t->signal->cred_guard_mutex); + current->signal->cred_locked_for_ptrace = false;mutex_unlock (¤t->signal->cred_guard_mutex);
Hmm, cough...
actually when the mutex_lock_killable fails, due to kill -9, in flush_old_exec free_bprm locks the same mutex, this time unkillable, but I should better do mutex_lock_killable here, and if that fails, I can leave cred_locked_for_ptrace, it shouldn't matter, since this is a fatal signal anyway, right?
Bernd.
Quoted 14 lines · Bernd Edlinger
Quoted 10 lines · Bernd Edlinger
Quoted 4 lines · Christian Brauner
So one issue I see with having to reacquire the
cred_guard_mutexmight be that this would allow tasks holding thecred_guard_mutexto block a killed exec'ing task from exiting, right?Yes maybe, but I think it will not be worse than it is now. Since the second time the mutex is acquired it is done with
mutex_lock_killable, so at least kill -9 should get it terminated.
Quoted 15 lines
fs/exec.c void finalize_exec(struct linux_binprm *bprm) · line 1442
Quoted 8 lines
static void
free_bprm(struct linux_binprm *bprm)
{fs/exec.c +3
-0free_arg_pages (bprm);if (bprm->cred) { +if (!bprm->called_flush_old_exec) +mutex_lock (¤t->signal->cred_guard_mutex); + current->signal->cred_locked_for_ptrace = false;mutex_unlock (¤t->signal->cred_guard_mutex);Hmm, cough...
actually when themutex_lock_killablefails, due to kill -9, inflush_old_execfree_bprmlocks the same mutex, this time unkillable, but I should better domutex_lock_killablehere, and if that fails, I can leavecred_locked_for_ptrace, it shouldn't matter, since this is a fatal signal anyway, right?
I think so, yes.
PATCH v5 1/1 exec: Fix a deadlock in ptrace
In reply to Christian BraunerThis fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in vm_access.
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the PTRACE_EVENT_EXIT is handled and the WIFEXITED event is received:
strace D 0 30614 30584 0x00000000
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
expect D 0 31933 30876 0x80004003
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
The proposed solution is to take the cred_guard_mutex only in a critical section at the beginning, and at the end of the execve function, and let PTRACE_ATTACH fail with EAGAIN while execve is not complete, but other functions like vm_access are allowed to complete normally.
This changes the lifetime of the cred_guard_mutex lock to be:
- during prepare_bprm_creds()
- from flush_old_exec() through install_exec_creds()
Before, cred_guard_mutex was held from prepare_bprm_creds() through install_exec_creds().
I also took the opportunity to improve the documentation of prepare_creds, which is obviously out of sync.
--- Documentation/security/credentials.rst | 19 +++++---- fs/exec.c | 41 ++++++++++++++++--- include/linux/binfmts.h | 6 ++- include/linux/sched/signal.h | 2 + kernel/cred.c | 2 +- kernel/ptrace.c | 4 ++ kernel/seccomp.c | 3 ++ mm/process_vm_access.c | 2 +- tools/testing/selftests/ptrace/Makefile | 4 +- tools/testing/selftests/ptrace/vmaccess.c | 66 +++++++++++++++++++++++++++++++ 10 files changed, 130 insertions(+), 19 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.c
v2: adds a test case which passes when this patch is applied.
v3: fixes the issue without introducing a new mutex.
v4: fixes one comment and a formatting issue found by checkpatch.pl in the test case.
v5: addresses review comments.
Documentation/security/credentials.rst +11 -8
@@ -437,9 +437,14 @@ new set of credentials by calling:: struct cred *prepare_creds(void); -this locks current->cred_replace_mutex and then allocates and constructs a -duplicate of the current process's credentials, returning with the mutex still -held if successful. It returns NULL if not successful (out of memory). +this allocates and constructs a duplicate of the current process's credentials. +It returns NULL if not successful (out of memory). + +If called from __do_execve_file, the mutex current->signal->cred_guard_mutex +is acquired before this function gets called, and released after setting +current->signal->cred_locked_in_execve. The same mutex is acquired later, +while the credentials and the process mmap are actually changed, and +current->signal->cred_locked_in_execve is reset again. The mutex prevents ``ptrace()`` from altering the ptrace state of a process while security checks on credentials construction and changing is taking place
@@ -466,9 +471,8 @@ by calling:: This will alter various aspects of the credentials and the process, giving the LSM a chance to do likewise, then it will use ``rcu_assign_pointer()`` to -actually commit the new credentials to ``current->cred``, it will release -``current->cred_replace_mutex`` to allow ``ptrace()`` to take place, and it -will notify the scheduler and others of the changes. +actually commit the new credentials to ``current->cred``, and it will notify +the scheduler and others of the changes. This function is guaranteed to return 0, so that it can be tail-called at the end of such functions as ``sys_setresuid()``.
@@ -486,8 +490,7 @@ invoked:: void abort_creds(struct cred *new); -This releases the lock on ``current->cred_replace_mutex`` that -``prepare_creds()`` got and then releases the new credentials. +This releases the new credentials. A typical credentials alteration function would look something like this::
fs/exec.c +35 -6
83 lines not on this page · raw message
include/linux/binfmts.h +5 -1
@@ -44,7 +44,11 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1 , +/* +* Set by flush_old_exec, when the cred_guard_mutex is taken. +*/ + called_flush_old_exec:1;#ifdef __alpha__ unsigned int taso:1;#endif
include/linux/sched/signal.h +2 -0
@@ -225,6 +225,8 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations * (notably. ptrace) */ +bool cred_locked_in_execve;/* set while in execve, only valid when +* cred_guard_mutex is held */ } __randomize_layout;/*
kernel/cred.c +1 -1
@@ -676,7 +676,7 @@ void __init cred_init(void)* * Returns the new credentials or NULL if out of memory. * -* Does not take, and does not return holding current->cred_replace_mutex. +* Does not take, and does not return holding ->cred_guard_mutex. */ struct cred *prepare_kernel_cred (struct task_struct *daemon) {
kernel/ptrace.c +4 -0
@@ -395,6 +395,10 @@ static int ptrace_attach(struct task_struct *task, long request,if (mutex_lock_interruptible (&task->signal->cred_guard_mutex)) goto out; + retval = -EAGAIN ; +if (task->signal->cred_locked_in_execve) + goto unlock_creds; +task_lock (task); retval =__ptrace_may_access (task,PTRACE_MODE_ATTACH_REALCREDS );task_unlock (task);
kernel/seccomp.c +3 -0
@@ -342,6 +342,9 @@ static inline pid_t seccomp_can_sync_threads(void)BUG_ON (!mutex_is_locked (¤t->signal->cred_guard_mutex));assert_spin_locked (¤t->sighand->siglock); +if (current->signal->cred_locked_in_execve) +return -EAGAIN ; +/* Validate all threads being eligible for synchronization. */ caller = current;for_each_thread (caller, thread) {
mm/process_vm_access.c +1 -1
@@ -204,7 +204,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter,if (!mm ||IS_ERR (mm)) { rc =IS_ERR (mm) ?PTR_ERR (mm) : -ESRCH ;/* -* Explicitly map EACCES to EPERM as EPERM is a more a +* Explicitly map EACCES to EPERM as EPERM is a more * appropriate error code for process_vw_readv/writev */ if (rc == -EACCES )
tools/testing/selftests/ptrace/Makefile +2 -2
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -iquote../../../../include/uapi -Wall +CFLAGS += -std=c99 -pthread -iquote../../../../include/uapi -Wall -TEST_GEN_PROGS := get_syscall_info peeksiginfo +TEST_GEN_PROGS := get_syscall_info peeksiginfo vmaccess include ../lib.mk
tools/testing/selftests/ptrace/vmaccess.c new file +66 -0
66 lines not on this page · raw message
Signature · 1 line
1.9.1
Quoted 8 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.
Quoted 4 lines
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:
A couple of things.
Why do we think it is safe to change the behavior exposed to userspace? Not the deadlock but all of the times the current code would not deadlock?
Especially given that this is a small window it might be hard for people to track down and report so we need a strong argument that this won't break existing userspace before we just change things.
Usually surveying all of the users of a system call that we can find and checking to see if they might be affected by the change in behavior is difficult enough that we usually opt for not being lazy and preserving the behavior.
This patch is up to two changes in behavior now, that could potentially affect a whole array of programs. Adding linux-api so that this change in behavior can be documented if/when this change goes through.
If you can split the documentation and test fixes out into separate patches that would help reviewing this code, or please make it explicit that the your are changing documentation about behavior that is changing with this patch.
Eric
Quoted 55 lines
tools/testing/selftests/ptrace/vmaccess.c new file +49
-0@@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (c) 2020 Bernd Edlinger <[email protected]> +* All rights reserved. +* +* Check whether /proc/$pid/mem can be accessed without causing deadlocks +* when de_thread is blocked with ->cred_guard_mutex held. +*/ + +#include "../kselftest_harness.h" +#include <stdio.h> +#include <fcntl.h> +#include <pthread.h> +#include <signal.h> +#include <unistd.h> +#include <sys/ptrace.h> + +static void *thread (void *arg) +{ +ptrace (PTRACE_TRACEME ,0 ,0L ,0L ); +return NULL ; +} + +TEST (vmaccess) +{ +int f, pid =fork (); +char mm[64 ]; + +if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); +sprintf (mm,"/proc/%d/mem" , pid); + f =open (mm,O_RDONLY ); +ASSERT_LE (0 , f); +close (f); + f =kill (pid,SIGCONT ); +ASSERT_EQ (0 , f); +} + +TEST (attach) +{ +int f, pid =fork (); +
tools/testing/selftests/ptrace/vmaccess.c line 59
Quoted 10 lines
tools/testing/selftests/ptrace/vmaccess.c new file +10
-0+if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); + f =ptrace (PTRACE_ATTACH , pid,0L ,0L );
To be meaningful this code needs to learn to loop when ptrace returns -EAGAIN.
Because that is pretty much what any self respecting user space process will do.
At which point I am not certain we can say that the behavior has sufficiently improved not to be a deadlock.
tools/testing/selftests/ptrace/vmaccess.c line 66
Quoted 7 lines
tools/testing/selftests/ptrace/vmaccess.c new file +7
-0+ASSERT_EQ (EAGAIN , errno); +ASSERT_EQ (f,-1 ); + f =kill (pid,SIGCONT ); +ASSERT_EQ (0 , f); +} + +TEST_HARNESS_MAIN
Eric
Quoted 17 lines · Eric W. Biederman
Quoted 8 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.Quoted 4 lines
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:A couple of things.
Quoted 8 lines
Why do we think it is safe to change the behavior exposed to userspace? Not the deadlock but all of the times the current code would not deadlock?
Especially given that this is a small window it might be hard for people to track down and report so we need a strong argument that this won't break existing userspace before we just change things.
Hmm, I tend to agree.
Quoted 9 lines
Usually surveying all of the users of a system call that we can find and checking to see if they might be affected by the change in behavior is difficult enough that we usually opt for not being lazy and preserving the behavior.
This patch is up to two changes in behavior now, that could potentially affect a whole array of programs. Adding linux-api so that this change in behavior can be documented if/when this change goes through.
One is PTRACE_ACCESS possibly returning EAGAIN, yes.
We could try to restrict that behavior change to when any thread is ptraced when execve starts, can't be too complicated.
But the other is only SYS_seccomp returning EAGAIN, when a different thread of the current process is calling execve at the same time.
I would consider it completely impossible to have any user-visual effect, since de_thread is just terminating all threads, including the thread where the -EAGAIN was returned, so we will never know what happened.
Quoted 5 lines
If you can split the documentation and test fixes out into separate patches that would help reviewing this code, or please make it explicit that the your are changing documentation about behavior that is changing with this patch.
I am not sure if I have touched the right user documentation.
I only saw a document referring to a non-existent "current->cred_replace_mutex" I haven't digged the git history, but that must be pre-historic IMHO. It appears to me that is some developer documentation, but it's nevertheless worth to keep up to date when the code changes.
So where would I add the possibility for PTRACE_ATTACH to return -EAGAIN ?
Bernd.
Quoted 71 lines
Eric
Quoted 55 lines
tools/testing/selftests/ptrace/vmaccess.c new file +49
-0@@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (c) 2020 Bernd Edlinger <[email protected]> +* All rights reserved. +* +* Check whether /proc/$pid/mem can be accessed without causing deadlocks +* when de_thread is blocked with ->cred_guard_mutex held. +*/ + +#include "../kselftest_harness.h" +#include <stdio.h> +#include <fcntl.h> +#include <pthread.h> +#include <signal.h> +#include <unistd.h> +#include <sys/ptrace.h> + +static void *thread (void *arg) +{ +ptrace (PTRACE_TRACEME ,0 ,0L ,0L ); +return NULL ; +} + +TEST (vmaccess) +{ +int f, pid =fork (); +char mm[64 ]; + +if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); +sprintf (mm,"/proc/%d/mem" , pid); + f =open (mm,O_RDONLY ); +ASSERT_LE (0 , f); +close (f); + f =kill (pid,SIGCONT ); +ASSERT_EQ (0 , f); +} + +TEST (attach) +{ +int f, pid =fork (); +tools/testing/selftests/ptrace/vmaccess.c line 59
Quoted 10 lines
tools/testing/selftests/ptrace/vmaccess.c new file +10
-0+if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); + f =ptrace (PTRACE_ATTACH , pid,0L ,0L );To be meaningful this code needs to learn to loop when ptrace returns
-EAGAIN.
Quoted 6 lines
Because that is pretty much what any self respecting user space process will do.
At which point I am not certain we can say that the behavior has sufficiently improved not to be a deadlock.
In this special dead-duck test it won't work, but it would still be lots more transparent what is going on, since previously you had two zombie process, and no way to even output debug messages, which also all self respecting user space processes should do.
So yes, I can at least give a good example and re-try it several times together with wait4 which a tracer is expected to do.
Bernd.
Quoted 10 lines
tools/testing/selftests/ptrace/vmaccess.c line 66
Quoted 7 lines
tools/testing/selftests/ptrace/vmaccess.c new file +7
-0+ASSERT_EQ (EAGAIN , errno); +ASSERT_EQ (f,-1 ); + f =kill (pid,SIGCONT ); +ASSERT_EQ (0 , f); +} + +TEST_HARNESS_MAIN Eric
Quoted 45 lines · Bernd Edlinger
Quoted 17 lines · Eric W. Biederman
Quoted 8 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.Quoted 4 lines
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:A couple of things.
Quoted 8 lines
Why do we think it is safe to change the behavior exposed to userspace? Not the deadlock but all of the times the current code would not deadlock?
Especially given that this is a small window it might be hard for people to track down and report so we need a strong argument that this won't break existing userspace before we just change things.
Hmm, I tend to agree.
Quoted 9 lines
Usually surveying all of the users of a system call that we can find and checking to see if they might be affected by the change in behavior is difficult enough that we usually opt for not being lazy and preserving the behavior.
This patch is up to two changes in behavior now, that could potentially affect a whole array of programs. Adding linux-api so that this change in behavior can be documented if/when this change goes through.
One is
PTRACE_ACCESSpossibly returning EAGAIN, yes.We could try to restrict that behavior change to when any thread is ptraced when execve starts, can't be too complicated.
Quoted 6 lines
But the other is only
SYS_seccompreturning EAGAIN, when a different thread of the current process is calling execve at the same time.I would consider it completely impossible to have any user-visual effect, since
de_threadis just terminating all threads, including the thread where the-EAGAINwas returned, so we will never know what happened.
I think if we risk a user-space facing change we should try the simple thing first before making the fix more convoluted? But it's a tough call...
Quoted 8 lines
Quoted 7 lines
If you can split the documentation and test fixes out into separate patches that would help reviewing this code, or please make it explicit that the your are changing documentation about behavior that is changing with this patch.
Quoted 8 lines
I am not sure if I have touched the right user documentation.
I only saw a document referring to a non-existent "
current->cred_replace_mutex" I haven't digged the git history, but that must be pre-historic IMHO. It appears to me that is some developer documentation, but it's nevertheless worth to keep up to date when the code changes.So where would I add the possibility for
PTRACE_ATTACHto return-EAGAIN?
Since that would be a potentially user-visible change it would make the most sense to add it to man ptrace(2) if/when we land this change.
For developers, placing a comment in kernel/ptrace.c:ptrace_attach() would make the most sense? We already have something about exec protection in there.
Christian
Quoted 49 lines · Christian Brauner
Quoted 48 lines · Bernd Edlinger
Quoted 17 lines · Eric W. Biederman
Quoted 8 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.Quoted 4 lines
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:A couple of things.
Quoted 8 lines
Why do we think it is safe to change the behavior exposed to userspace? Not the deadlock but all of the times the current code would not deadlock?
Especially given that this is a small window it might be hard for people to track down and report so we need a strong argument that this won't break existing userspace before we just change things.
Hmm, I tend to agree.
Quoted 9 lines
Usually surveying all of the users of a system call that we can find and checking to see if they might be affected by the change in behavior is difficult enough that we usually opt for not being lazy and preserving the behavior.
This patch is up to two changes in behavior now, that could potentially affect a whole array of programs. Adding linux-api so that this change in behavior can be documented if/when this change goes through.
One is
PTRACE_ACCESSpossibly returning EAGAIN, yes.We could try to restrict that behavior change to when any thread is ptraced when execve starts, can't be too complicated.
But the other is only
SYS_seccompreturning EAGAIN, when a different thread of the current process is calling execve at the same time.
Quoted 7 lines
Quoted 3 lines
I would consider it completely impossible to have any user-visual effect, since
de_threadis just terminating all threads, including the thread where the-EAGAINwas returned, so we will never know what happened.I think if we risk a user-space facing change we should try the simple thing first before making the fix more convoluted? But it's a tough call...
Actually, to get a _rough_ estimate of the possible impact I would recommend you run the criu test suite (and possible the strace test-suite) on a kernel with and without your fix. That's what I tend to do when I touch code I fear will have impact on APIs that very deeply touch core kernel. Criu's test-suite makes heavy use of ptrace and usually runs into a bunch of interesting (exec) races too, and does have tests for handling zombies processes etc. pp.
Should be relatively simple: create a vm and then criu build-dependencies, git clone criu; cd criu; make; cd test; ./zdtm.py run -a --keep-going If your system doesn't support Selinux properly, you need to disable it when running the tests and you also need to make sure that you're using python3 or change the shebang in zdtm.py to python3.
Just a recommendation.
Christian
Quoted 78 lines · Bernd Edlinger
Quoted 70 lines · Eric W. Biederman
Quoted 55 lines · Bernd Edlinger
tools/testing/selftests/ptrace/vmaccess.c new file +49
-0@@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (c) 2020 Bernd Edlinger <[email protected]> +* All rights reserved. +* +* Check whether /proc/$pid/mem can be accessed without causing deadlocks +* when de_thread is blocked with ->cred_guard_mutex held. +*/ + +#include "../kselftest_harness.h" +#include <stdio.h> +#include <fcntl.h> +#include <pthread.h> +#include <signal.h> +#include <unistd.h> +#include <sys/ptrace.h> + +static void *thread (void *arg) +{ +ptrace (PTRACE_TRACEME ,0 ,0L ,0L ); +return NULL ; +} + +TEST (vmaccess) +{ +int f, pid =fork (); +char mm[64 ]; + +if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); +sprintf (mm,"/proc/%d/mem" , pid); + f =open (mm,O_RDONLY ); +ASSERT_LE (0 , f); +close (f); + f =kill (pid,SIGCONT ); +ASSERT_EQ (0 , f); +} + +TEST (attach) +{ +int f, pid =fork (); +tools/testing/selftests/ptrace/vmaccess.c line 59
Quoted 10 lines
tools/testing/selftests/ptrace/vmaccess.c new file +10
-0+if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); + f =ptrace (PTRACE_ATTACH , pid,0L ,0L );To be meaningful this code needs to learn to loop when ptrace returns
-EAGAIN.Quoted 6 lines
Because that is pretty much what any self respecting user space process will do.
At which point I am not certain we can say that the behavior has sufficiently improved not to be a deadlock.
Quoted 5 lines
In this special dead-duck test it won't work, but it would still be lots more transparent what is going on, since previously you had two zombie process, and no way to even output debug messages, which also all self respecting user space processes should do.
Agreed it is more transparent. So if you are going to deadlock it is better.
My previous proposal (which I admit is more work to implement) would actually allow succeeding in this case and so it would not be subject to a dead lock (even via -EGAIN) at this point.
Quoted 2 lines
So yes, I can at least give a good example and re-try it several times together with wait4 which a tracer is expected to do.
Thank you,
Eric
Quoted 93 lines · Eric W. Biederman
Quoted 78 lines · Bernd Edlinger
Quoted 70 lines · Eric W. Biederman
Quoted 55 lines · Bernd Edlinger
tools/testing/selftests/ptrace/vmaccess.c new file +49
-0@@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (c) 2020 Bernd Edlinger <[email protected]> +* All rights reserved. +* +* Check whether /proc/$pid/mem can be accessed without causing deadlocks +* when de_thread is blocked with ->cred_guard_mutex held. +*/ + +#include "../kselftest_harness.h" +#include <stdio.h> +#include <fcntl.h> +#include <pthread.h> +#include <signal.h> +#include <unistd.h> +#include <sys/ptrace.h> + +static void *thread (void *arg) +{ +ptrace (PTRACE_TRACEME ,0 ,0L ,0L ); +return NULL ; +} + +TEST (vmaccess) +{ +int f, pid =fork (); +char mm[64 ]; + +if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); +sprintf (mm,"/proc/%d/mem" , pid); + f =open (mm,O_RDONLY ); +ASSERT_LE (0 , f); +close (f); + f =kill (pid,SIGCONT ); +ASSERT_EQ (0 , f); +} + +TEST (attach) +{ +int f, pid =fork (); +tools/testing/selftests/ptrace/vmaccess.c line 59
Quoted 10 lines
tools/testing/selftests/ptrace/vmaccess.c new file +10
-0+if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); + f =ptrace (PTRACE_ATTACH , pid,0L ,0L );To be meaningful this code needs to learn to loop when ptrace returns
-EAGAIN.Quoted 6 lines
Because that is pretty much what any self respecting user space process will do.
At which point I am not certain we can say that the behavior has sufficiently improved not to be a deadlock.
Quoted 5 lines
In this special dead-duck test it won't work, but it would still be lots more transparent what is going on, since previously you had two zombie process, and no way to even output debug messages, which also all self respecting user space processes should do.
Agreed it is more transparent. So if you are going to deadlock it is better.
My previous proposal (which I admit is more work to implement) would actually allow succeeding in this case and so it would not be subject to a dead lock (even via
-EGAIN) at this point.
Quoted 7 lines
Quoted 2 lines
So yes, I can at least give a good example and re-try it several times together with wait4 which a tracer is expected to do.
Thank you,
Eric
Okay, I think it can be done with minimal API changes, but it needs two mutexes, one that guards the execve, and one that guards only the credentials.
If no traced sibling thread exists, the mutexes are used this way:lock(exec_guard_mutex)cred_locked_in_execve = true;de_thread()lock(cred_guard_mutex)unlock(cred_guard_mutex)cred_locked_in_execve = false;unlock(exec_guard_mutex)
so effectively no API change at all.
If a traced sibling thread exists, the mutexes are used differently:lock(exec_guard_mutex)cred_locked_in_execve = true;unlock(exec_guard_mutex)de_thread()lock(cred_guard_mutex)unlock(cred_guard_mutex)lock(exec_guard_mutex)cred_locked_in_execve = false;unlock(exec_guard_mutex)
Only the case changes that would deadlock anyway.
Bernd.
Quoted 128 lines · Bernd Edlinger
Quoted 93 lines · Eric W. Biederman
Quoted 78 lines · Bernd Edlinger
Quoted 70 lines · Eric W. Biederman
Quoted 55 lines · Bernd Edlinger
tools/testing/selftests/ptrace/vmaccess.c new file +49
-0@@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (c) 2020 Bernd Edlinger <[email protected]> +* All rights reserved. +* +* Check whether /proc/$pid/mem can be accessed without causing deadlocks +* when de_thread is blocked with ->cred_guard_mutex held. +*/ + +#include "../kselftest_harness.h" +#include <stdio.h> +#include <fcntl.h> +#include <pthread.h> +#include <signal.h> +#include <unistd.h> +#include <sys/ptrace.h> + +static void *thread (void *arg) +{ +ptrace (PTRACE_TRACEME ,0 ,0L ,0L ); +return NULL ; +} + +TEST (vmaccess) +{ +int f, pid =fork (); +char mm[64 ]; + +if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); +sprintf (mm,"/proc/%d/mem" , pid); + f =open (mm,O_RDONLY ); +ASSERT_LE (0 , f); +close (f); + f =kill (pid,SIGCONT ); +ASSERT_EQ (0 , f); +} + +TEST (attach) +{ +int f, pid =fork (); +tools/testing/selftests/ptrace/vmaccess.c line 59
Quoted 10 lines
tools/testing/selftests/ptrace/vmaccess.c new file +10
-0+if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); + f =ptrace (PTRACE_ATTACH , pid,0L ,0L );To be meaningful this code needs to learn to loop when ptrace returns
-EAGAIN.Quoted 6 lines
Because that is pretty much what any self respecting user space process will do.
At which point I am not certain we can say that the behavior has sufficiently improved not to be a deadlock.
Quoted 5 lines
In this special dead-duck test it won't work, but it would still be lots more transparent what is going on, since previously you had two zombie process, and no way to even output debug messages, which also all self respecting user space processes should do.
Agreed it is more transparent. So if you are going to deadlock it is better.
My previous proposal (which I admit is more work to implement) would actually allow succeeding in this case and so it would not be subject to a dead lock (even via
-EGAIN) at this point.Quoted 7 lines
Quoted 2 lines
So yes, I can at least give a good example and re-try it several times together with wait4 which a tracer is expected to do.
Thank you,
Eric
Okay, I think it can be done with minimal API changes, but it needs two mutexes, one that guards the execve, and one that guards only the credentials.
If no traced sibling thread exists, the mutexes are used this way:
lock(exec_guard_mutex)cred_locked_in_execve= true;de_thread()lock(cred_guard_mutex)unlock(cred_guard_mutex)cred_locked_in_execve= false;unlock(exec_guard_mutex)so effectively no API change at all.
If a traced sibling thread exists, the mutexes are used differently:
lock(exec_guard_mutex)cred_locked_in_execve= true;unlock(exec_guard_mutex)de_thread()lock(cred_guard_mutex)unlock(cred_guard_mutex)lock(exec_guard_mutex)cred_locked_in_execve= false;unlock(exec_guard_mutex)
Quoted 1 line
Only the case changes that would deadlock anyway.
Let me propose a slight alternative that I think sets us up for long term success.
Leave cred_guard_mutex as is, but declare it undesirable. The cred_guard_mutex as designed really is something we should get rid of. As it it can sleep over several different userspace accesses. The copying of the exec arguments is technically as prone to deadlock as the ptrace case.
Add a new mutex with a better name perhaps "exec_change_mutex" that is used to guard the changes that exec makes to a process.
Then we gradually shift all the cred_guard_mutex users over to the new mutex. AKA one patch per user of cred_guard_mutex. At each patch that shifts things over we will have the opportunity to review the code to see that there no funny dependencies that were missed.
I will sign up for working on the no_new_privs and ptrace_attach cases as I think I can make those happen. Especially no_new_privs.
Getting the easier cases will resolve your issues and put things on a better footing.
Eric
Quoted 150 lines · Eric W. Biederman
Quoted 128 lines · Bernd Edlinger
Quoted 93 lines · Eric W. Biederman
Quoted 78 lines · Bernd Edlinger
Quoted 70 lines · Eric W. Biederman
Quoted 55 lines · Bernd Edlinger
tools/testing/selftests/ptrace/vmaccess.c new file +49
-0@@ -0,0 +1,66 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (c) 2020 Bernd Edlinger <[email protected]> +* All rights reserved. +* +* Check whether /proc/$pid/mem can be accessed without causing deadlocks +* when de_thread is blocked with ->cred_guard_mutex held. +*/ + +#include "../kselftest_harness.h" +#include <stdio.h> +#include <fcntl.h> +#include <pthread.h> +#include <signal.h> +#include <unistd.h> +#include <sys/ptrace.h> + +static void *thread (void *arg) +{ +ptrace (PTRACE_TRACEME ,0 ,0L ,0L ); +return NULL ; +} + +TEST (vmaccess) +{ +int f, pid =fork (); +char mm[64 ]; + +if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); +sprintf (mm,"/proc/%d/mem" , pid); + f =open (mm,O_RDONLY ); +ASSERT_LE (0 , f); +close (f); + f =kill (pid,SIGCONT ); +ASSERT_EQ (0 , f); +} + +TEST (attach) +{ +int f, pid =fork (); +tools/testing/selftests/ptrace/vmaccess.c line 59
Quoted 10 lines
tools/testing/selftests/ptrace/vmaccess.c new file +10
-0+if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); + f =ptrace (PTRACE_ATTACH , pid,0L ,0L );To be meaningful this code needs to learn to loop when ptrace returns
-EAGAIN.Quoted 6 lines
Because that is pretty much what any self respecting user space process will do.
At which point I am not certain we can say that the behavior has sufficiently improved not to be a deadlock.
Quoted 5 lines
In this special dead-duck test it won't work, but it would still be lots more transparent what is going on, since previously you had two zombie process, and no way to even output debug messages, which also all self respecting user space processes should do.
Agreed it is more transparent. So if you are going to deadlock it is better.
My previous proposal (which I admit is more work to implement) would actually allow succeeding in this case and so it would not be subject to a dead lock (even via
-EGAIN) at this point.Quoted 7 lines
Quoted 2 lines
So yes, I can at least give a good example and re-try it several times together with wait4 which a tracer is expected to do.
Thank you,
Eric
Okay, I think it can be done with minimal API changes, but it needs two mutexes, one that guards the execve, and one that guards only the credentials.
If no traced sibling thread exists, the mutexes are used this way:
lock(exec_guard_mutex)cred_locked_in_execve= true;de_thread()lock(cred_guard_mutex)unlock(cred_guard_mutex)cred_locked_in_execve= false;unlock(exec_guard_mutex)so effectively no API change at all.
If a traced sibling thread exists, the mutexes are used differently:
lock(exec_guard_mutex)cred_locked_in_execve= true;unlock(exec_guard_mutex)de_thread()lock(cred_guard_mutex)unlock(cred_guard_mutex)lock(exec_guard_mutex)cred_locked_in_execve= false;unlock(exec_guard_mutex)Quoted 1 line
Only the case changes that would deadlock anyway.
Let me propose a slight alternative that I think sets us up for long term success.
Leave
cred_guard_mutexas is, but declare it undesirable. Thecred_guard_mutexas designed really is something we should get rid of. As it it can sleep over several different userspace accesses. The copying of the exec arguments is technically as prone to deadlock as the ptrace case.Add a new mutex with a better name perhaps "
exec_change_mutex" that is used to guard the changes that exec makes to a process.Then we gradually shift all the
cred_guard_mutexusers over to the new mutex. AKA one patch per user ofcred_guard_mutex. At each patch that shifts things over we will have the opportunity to review the code to see that there no funny dependencies that were missed.
Quoted 8 lines
I will sign up for working on the
no_new_privsandptrace_attachcases as I think I can make those happen. Especiallyno_new_privs.Getting the easier cases will resolve your issues and put things on a better footing.
Eric
Okay, however I think we will need two mutexes in the long term.
So currently I have reduced the cred_guard_mutex to protect just the loading of the executable code in the process vm, since that is what works for vm_access, (one of the test cases). And another mutex that protects the whole execve function, that is need for ptrace, (and seccomp).
But I have only a test case for ptrace.
If I understand that right, I should not recycle cred_guard_mutex but leave it as is, and create two additional mutexes which will take over step by step.
Sounds reasonable, indeed.
I will send an update (v6) what I have right now,
but just for information, so you can see how my minimal API-Change approach works.
Bernd.
PATCH v6 1/1 exec: Fix a deadlock in ptrace
In reply to Eric W. BiedermanThis fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in vm_access.
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the PTRACE_EVENT_EXIT is handled and the WIFEXITED event is received:
strace D 0 30614 30584 0x00000000
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
expect D 0 31933 30876 0x80004003
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
The proposed solution is to detect if a sibling thread exists that is traced and in this case to make PTRACE_ACCESS fail with -EAGAIN instead of dead-lock.
But other functions like vm_access are allowed to complete normally.
This changes the lifetime of the cred_guard_mutex lock to be from flush_old_exec() through install_exec_creds(). Before, cred_guard_mutex was held from prepare_bprm_creds() through install_exec_creds().
Additionally a new mutex exec_guard_mutex is introduced that is used for PTRACE_ACCESS and SECCOMP_FILTER_FLAG_TSYNC.
--- Documentation/security/credentials.rst | 29 ++++++++--- fs/exec.c | 58 ++++++++++++++++++--- include/linux/binfmts.h | 15 +++++- include/linux/sched/signal.h | 10 ++-- init/init_task.c | 1 + kernel/cred.c | 4 +- kernel/fork.c | 1 + kernel/ptrace.c | 20 ++++++-- kernel/seccomp.c | 15 +++--- mm/process_vm_access.c | 2 +- tools/testing/selftests/ptrace/Makefile | 4 +- tools/testing/selftests/ptrace/vmaccess.c | 85 +++++++++++++++++++++++++++++++ 12 files changed, 210 insertions(+), 34 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.c
v2: adds a test case which passes when this patch is applied.
v3: fixes the issue without introducing a new mutex.
v4: fixes one comment and a formatting issue found by checkpatch.pl in the test case.
v5: addresses review comments.
v6: minimal API changes, using a second mutex, improved test case.
Documentation/security/credentials.rst +21 -8
@@ -437,15 +437,30 @@ new set of credentials by calling:: struct cred *prepare_creds(void); -this locks current->cred_replace_mutex and then allocates and constructs a -duplicate of the current process's credentials, returning with the mutex still -held if successful. It returns NULL if not successful (out of memory). +this allocates and constructs a duplicate of the current process's credentials. +It returns NULL if not successful (out of memory). + +If called from __do_execve_file, the mutex current->signal->exec_guard_mutex +is acquired before this function gets called, and usually released after +the new process mmap and credentials are installed. However if one of the +sibling threads are being traced when the execve is invoked, there is no +guarantee how long it takes to terminate all sibling threads, and therefore +the variable current->signal->cred_locked_in_execve is set, and the +exec_guard_mutex is released immediately. Functions that may have effect +on the credentials of a different thread need to lock the exec_guard_mutex +and additionally check the cred_locked_in_execve status, and fail with +-EAGAIN if that variable is set. The mutex prevents ``ptrace()`` from altering the ptrace state of a process while security checks on credentials construction and changing is taking place as the ptrace state may alter the outcome, particularly in the case of ``execve()``. +The mutex current->signal->cred_guard_mutex is acquired when only a single thread +is remaining, and the credentials and the process mmap are actually changed. +Functions that only need to access to a consistent state of the credentials +and the process mmap do only need to aquire this mutex. + The new credentials set should be altered appropriately, and any security checks and hooks done. Both the current and the proposed sets of credentials are available for this purpose as current_cred() will return the current set
@@ -466,9 +481,8 @@ by calling:: This will alter various aspects of the credentials and the process, giving the LSM a chance to do likewise, then it will use ``rcu_assign_pointer()`` to -actually commit the new credentials to ``current->cred``, it will release -``current->cred_replace_mutex`` to allow ``ptrace()`` to take place, and it -will notify the scheduler and others of the changes. +actually commit the new credentials to ``current->cred``, and it will notify +the scheduler and others of the changes. This function is guaranteed to return 0, so that it can be tail-called at the end of such functions as ``sys_setresuid()``.
@@ -486,8 +500,7 @@ invoked:: void abort_creds(struct cred *new); -This releases the lock on ``current->cred_replace_mutex`` that -``prepare_creds()`` got and then releases the new credentials. +This releases the new credentials. A typical credentials alteration function would look something like this::
fs/exec.c +50 -8
103 lines not on this page · raw message
include/linux/binfmts.h +14 -1
@@ -44,7 +44,20 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1, + /* + * Set by prepare_bprm_creds, if a sibling thread is being + * traced and the exec_guard_mutex is therefore not taken. + */ + detected_unsafe_exec:1, + /* + * Set when the cred_guard_mutex is taken. + */ + holding_cred_guard_mutex:1, + /* + * Set when the exec_guard_mutex is taken. + */ + holding_exec_guard_mutex:1;#ifdef __alpha__ unsigned int taso:1;#endif
include/linux/sched/signal.h +7 -3
@@ -222,9 +222,13 @@ struct signal_struct {struct mm_struct *oom_mm;/* recorded mm when the thread group got * killed by the oom killer */ -struct mutex cred_guard_mutex;/* guard against foreign influences on -* credential calculations -* (notably. ptrace) */ +struct mutex cred_guard_mutex;/* guard against changing credentials */ +struct mutex exec_guard_mutex;/* guard against foreign influences on +* execve (notably. ptrace) +*/ +bool cred_locked_in_execve;/* set while in execve, only valid when +* exec_guard_mutex is held +*/ } __randomize_layout;/*
init/init_task.c +1 -0
@@ -26,6 +26,7 @@ .multiprocess = HLIST_HEAD_INIT, .rlim = INIT_RLIMITS, .cred_guard_mutex = __MUTEX_INITIALIZER(init_signals.cred_guard_mutex), + .exec_guard_mutex =__MUTEX_INITIALIZER (init_signals.exec_guard_mutex), #ifdef CONFIG_POSIX_TIMERS .posix_timers =LIST_HEAD_INIT (init_signals.posix_timers), .cputimer = {
kernel/cred.c +2 -2
@@ -295,7 +295,7 @@ struct cred *prepare_creds(void)/* * Prepare credentials for current to perform an execve() -* - The caller must hold ->cred_guard_mutex +* - The caller must hold ->exec_guard_mutex */ struct cred *prepare_exec_creds (void ) {
@@ -676,7 +676,7 @@ void __init cred_init(void)* * Returns the new credentials or NULL if out of memory. * -* Does not take, and does not return holding current->cred_replace_mutex. +* Does not take, and does not return holding ->cred_guard_mutex. */ struct cred *prepare_kernel_cred (struct task_struct *daemon) {
kernel/fork.c +1 -0
@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); +mutex_init (&sig->exec_guard_mutex);return 0 ; }
kernel/ptrace.c +17 -3
@@ -392,9 +392,13 @@ static int ptrace_attach(struct task_struct *task, long request,* under ptrace. */ retval = -ERESTARTNOINTR ; -if (mutex_lock_interruptible (&task->signal->cred_guard_mutex)) +if (mutex_lock_interruptible (&task->signal->exec_guard_mutex)) goto out; + retval = -EAGAIN ; +if (task->signal->cred_locked_in_execve) + goto unlock_creds; +task_lock (task); retval =__ptrace_may_access (task,PTRACE_MODE_ATTACH_REALCREDS );task_unlock (task);
@@ -447,7 +451,7 @@ static int ptrace_attach(struct task_struct *task, long request, unlock_tasklist:write_unlock_irq (&tasklist_lock); unlock_creds: -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_guard_mutex); out:if (!retval) {/*
@@ -472,10 +476,18 @@ static int ptrace_attach(struct task_struct *task, long request,*/ static int ptrace_traceme (void ) { -int ret = -EPERM ; +int ret; + +if (mutex_lock_interruptible (¤t->signal->exec_guard_mutex)) +return -ERESTARTNOINTR ; + + ret = -EAGAIN ; +if (current->signal->cred_locked_in_execve) + goto unlock_creds;write_lock_irq (&tasklist_lock);/* Are we already being traced? */ + ret = -EPERM ;if (!current->ptrace) { ret =security_ptrace_traceme (current->parent);/*
@@ -490,6 +502,8 @@ static int ptrace_traceme(void) }write_unlock_irq (&tasklist_lock); +unlock_creds: +mutex_unlock (¤t->signal->exec_guard_mutex);return ret; }
kernel/seccomp.c +9 -6
@@ -329,7 +329,7 @@ static int is_ancestor(struct seccomp_filter *parent,/** * seccomp_can_sync_threads: checks if all threads can be synchronized * -* Expects sighand and cred_guard_mutex locks to be held. +* Expects sighand and exec_guard_mutex locks to be held. * * Returns 0 on success, -ve on error, or the pid of a thread which was * either not in the correct seccomp mode or did not have an ancestral
@@ -339,9 +339,12 @@ static inline pid_t seccomp_can_sync_threads(void) {struct task_struct *thread, *caller; -BUG_ON (!mutex_is_locked (¤t->signal->cred_guard_mutex)); +BUG_ON (!mutex_is_locked (¤t->signal->exec_guard_mutex));assert_spin_locked (¤t->sighand->siglock); +if (current->signal->cred_locked_in_execve) +return -EAGAIN ; +/* Validate all threads being eligible for synchronization. */ caller = current;for_each_thread (caller, thread) {
@@ -371,7 +374,7 @@ static inline pid_t seccomp_can_sync_threads(void)/** * seccomp_sync_threads: sets all threads to use current's filter * -* Expects sighand and cred_guard_mutex locks to be held, and for +* Expects sighand and exec_guard_mutex locks to be held, and for * seccomp_can_sync_threads() to have returned success already * without dropping the locks. *
@@ -380,7 +383,7 @@ static inline void seccomp_sync_threads(unsigned long flags) {struct task_struct *thread, *caller; -BUG_ON (!mutex_is_locked (¤t->signal->cred_guard_mutex)); +BUG_ON (!mutex_is_locked (¤t->signal->exec_guard_mutex));assert_spin_locked (¤t->sighand->siglock);/* Synchronize all threads. */
@@ -1319,7 +1322,7 @@ static long seccomp_set_mode_filter(unsigned int flags,* while another thread is in the middle of calling exec. */ if (flags &SECCOMP_FILTER_FLAG_TSYNC && -mutex_lock_killable (¤t->signal->cred_guard_mutex)) +mutex_lock_killable (¤t->signal->exec_guard_mutex)) goto out_put_fd;spin_lock_irq (¤t->sighand->siglock);
@@ -1337,7 +1340,7 @@ static long seccomp_set_mode_filter(unsigned int flags, out:spin_unlock_irq (¤t->sighand->siglock);if (flags &SECCOMP_FILTER_FLAG_TSYNC ) -mutex_unlock (¤t->signal->cred_guard_mutex); +mutex_unlock (¤t->signal->exec_guard_mutex); out_put_fd:if (flags &SECCOMP_FILTER_FLAG_NEW_LISTENER ) {if (ret) {
mm/process_vm_access.c +1 -1
@@ -204,7 +204,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter,if (!mm ||IS_ERR (mm)) { rc =IS_ERR (mm) ?PTR_ERR (mm) : -ESRCH ;/* -* Explicitly map EACCES to EPERM as EPERM is a more a +* Explicitly map EACCES to EPERM as EPERM is a more * appropriate error code for process_vw_readv/writev */ if (rc == -EACCES )
tools/testing/selftests/ptrace/Makefile +2 -2
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -iquote../../../../include/uapi -Wall +CFLAGS += -std=c99 -pthread -iquote../../../../include/uapi -Wall -TEST_GEN_PROGS := get_syscall_info peeksiginfo +TEST_GEN_PROGS := get_syscall_info peeksiginfo vmaccess include ../lib.mk
tools/testing/selftests/ptrace/vmaccess.c new file +85 -0
85 lines not on this page · raw message
Signature · 1 line
1.9.1
Quoted 54 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9The proposed solution is to detect if a sibling thread exists that is traced and in this case to make
PTRACE_ACCESSfail with-EAGAINinstead of dead-lock.
But other functions likevm_accessare allowed to complete normally.This changes the lifetime of the
cred_guard_mutexlock to be fromflush_old_exec()throughinstall_exec_creds(). Before,cred_guard_mutexwas held fromprepare_bprm_creds()throughinstall_exec_creds().Additionally a new mutex
exec_guard_mutexis introduced that is used forPTRACE_ACCESSandSECCOMP_FILTER_FLAG_TSYNC.Signed-off-by Bernd Edlinger <[email protected]>
Quoted 16 lines
--- Documentation/security/credentials.rst | 29 ++++++++--- fs/exec.c | 58 ++++++++++++++++++--- include/linux/binfmts.h | 15 +++++- include/linux/sched/signal.h | 10 ++-- init/init_task.c | 1 + kernel/cred.c | 4 +- kernel/fork.c | 1 + kernel/ptrace.c | 20 ++++++-- kernel/seccomp.c | 15 +++--- mm/process_vm_access.c | 2 +- tools/testing/selftests/ptrace/Makefile | 4 +- tools/testing/selftests/ptrace/vmaccess.c | 85 +++++++++++++++++++++++++++++++ 12 files changed, 210 insertions(+), 34 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.c
Okay, I think there is consensus about the next steps to be as follows:
- post the Documentation/security/credentials.rst changes as an independent patch.
- post a infrastructure patch which only introduces two new mutexes, one exec_guard_mutex, and one the "cred_change_mutex" (I am unhappy with that name, because credentials can change without the cred_guard_mutex, this appears more to guarantee that the credentials of the process and the process memory map are consistent, so I think I need to think of a better name first...) This keeps cred_guard_mutex as is, just deprecates it, and adds a note that it will go away.
- post one patch that fixes the mm_access code path
- post one patch that fixes the PTRACE_ATTACH code path
- post one patch that introduces the new test cases
Thanks
Bernd.
cover Infrastructure to allow fixing exec deadlocks
Bernd, everyone
This is how I think the infrastructure change should look that makes way for fixing this issue.
- Correct the point of no return.
- Add a new mutex to replace cred_guard_mutex
Then I think it is just going through the existing users of cred_guard_mutex and fixing them to use the new one.
There really aren't that many users of cred_guard_mutex so we should be able to get through the easy ones fairly quickly. And anything that isn't easy we can wait until we have a good fix.
The users of cred_guard_mutex that I saw were:
fs/proc/base.c:
proc_pid_attr_write
do_io_accounting
proc_pid_stack
proc_pid_syscall
proc_pid_personality
perf_event_open
mm_access
kcmp
pidfd_fget
seccomp_set_mode_filterBernd does this make sense to you?
I think we can fix the seccomp/no_new_privs issue with some careful refactoring. We can probably do the same for ptrace but that appears to need a little lsm bug fixing.
My goal here is to allow us to fix the uncontroversial easy bits. While still allowing the difficult tricky bits to be fixed.
Eric W. Biederman (2):
exec: Properly mark the point of no return
exec: Add a exec_update_mutex to replace cred_guard_mutex
fs/exec.c | 11 ++++++++--- include/linux/binfmts.h | 7 ++++++- include/linux/sched/signal.h | 9 ++++++++- kernel/fork.c | 1 + 4 files changed, 23 insertions(+), 5 deletions(-)
Eric
PATCH 1/2 exec: Properly mark the point of no return
Add a flag binfmt->unrecoverable to mark when execution has gotten to the point where it is impossible to return to userspace with the calling process unchanged.
While techinically this state starts as soon as de_thread starts killing threads, the only return path at that point is if there is a fatal signal pending. I have choosen instead to set unrecoverable when the killing stops, and there are possibilities of failures other than fatal signals. In particular it is possible for the allocation of a new sighand structure to fail.
Setting unrecoverable at this point has the benefit that other actions can be taken after the other threads are all dead, and the unrecoverable flag can double as a flag that those actions have been taken.
--- fs/exec.c | 7 ++++--- include/linux/binfmts.h | 7 ++++++- 2 files changed, 10 insertions(+), 4 deletions(-)
fs/exec.c +4 -3
@@ -1061,7 +1061,7 @@ static int exec_mmap(struct mm_struct *mm)* disturbing other processes. (Other processes might share the signal * table via the CLONE_SIGHAND option to clone().) */ -static int de_thread (struct task_struct *tsk) +static int de_thread (struct linux_binprm *bprm,struct task_struct *tsk) {struct signal_struct *sig = tsk->signal;struct sighand_struct *oldsighand = tsk->sighand;
@@ -1182,6 +1182,7 @@ static int de_thread(struct task_struct *tsk)release_task (leader); } + bprm->unrecoverable = true; sig->group_exit_task =NULL ; sig->notify_count =0 ;
@@ -1266,7 +1267,7 @@ int flush_old_exec(struct linux_binprm * bprm)* Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (bprm, current);if (retval) goto out;
@@ -1664,7 +1665,7 @@ int search_binary_handler(struct linux_binprm *bprm)read_lock (&binfmt_lock);put_binfmt (fmt); -if (retval <0 && !bprm->mm) { +if (retval <0 && bprm->unrecoverable) {/* we got to flush_old_exec() and failed after it */ read_unlock (&binfmt_lock);force_sigsegv (SIGSEGV );
include/linux/binfmts.h +6 -1
@@ -44,7 +44,12 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1 , +/* +* Set when changes have been made that prevent returning +* to userspace. +*/ + unrecoverable:1;#ifdef __alpha__ unsigned int taso:1;#endif
Signature · 1 line
2.25.0
Re: [PATCH 1/2] exec: Properly mark the point of no return
Quoted 45 lines · Eric W. Biederman
Add a flag
binfmt->unrecoverableto mark when execution has gotten to the point where it is impossible to return to userspace with the calling process unchanged.While techinically this state starts as soon as
de_threadstarts killing threads, the only return path at that point is if there is a fatal signal pending. I have choosen instead to set unrecoverable when the killing stops, and there are possibilities of failures other than fatal signals. In particular it is possible for the allocation of a new sighand structure to fail.Setting unrecoverable at this point has the benefit that other actions can be taken after the other threads are all dead, and the unrecoverable flag can double as a flag that those actions have been taken.
Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 7 ++++--- include/linux/binfmts.h | 7 ++++++- 2 files changed, 10 insertions(+), 4 deletions(-)fs/exec.c +2
-1@@ -1061,7 +1061,7 @@ static int exec_mmap(struct mm_struct *mm)* disturbing other processes. (Other processes might share the signal * table via the CLONE_SIGHAND option to clone().) */ -static int de_thread (struct task_struct *tsk) +static int de_thread (struct linux_binprm *bprm,struct task_struct *tsk) {struct signal_struct *sig = tsk->signal;struct sighand_struct *oldsighand = tsk->sighand;@@ -1182,6 +1182,7 @@ static int de_thread(struct task_struct *tsk)release_task (leader); } + bprm->unrecoverable = true; sig->group_exit_task =NULL ; sig->notify_count =0 ;
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1270
Quoted 5 lines
fs/exec.c +1
-1@@ -1266,7 +1267,7 @@ int flush_old_exec(struct linux_binprm * bprm)* Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (bprm, current);
can we get rid of passing current as parameter here?
Thanks
Bernd.
Quoted 31 lines
fs/exec.c +1
-1if (retval) goto out;@@ -1664,7 +1665,7 @@ int search_binary_handler(struct linux_binprm *bprm)read_lock (&binfmt_lock);put_binfmt (fmt); -if (retval <0 && !bprm->mm) { +if (retval <0 && bprm->unrecoverable) {/* we got to flush_old_exec() and failed after it */ read_unlock (&binfmt_lock);force_sigsegv (SIGSEGV );include/linux/binfmts.h +6
-1@@ -44,7 +44,12 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1 , +/* +* Set when changes have been made that prevent returning +* to userspace. +*/ + unrecoverable:1;#ifdef __alpha__ unsigned int taso:1;#endif
Re: [PATCH 1/2] exec: Properly mark the point of no return
Quoted 9 lines · Bernd Edlinger
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1270
Quoted 6 lines · Eric W. Biederman
fs/exec.c +1
-1@@ -1266,7 +1267,7 @@ int flush_old_exec(struct linux_binprm * bprm)* Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (bprm, current);can we get rid of passing current as parameter here?
With a separate patch. It makes the patch less clear if I make that change in this one.
Eric
Re: [PATCH 1/2] exec: Properly mark the point of no return
In reply to Eric W. BiedermanQuoted 37 lines · Eric W. Biederman
Add a flag
binfmt->unrecoverableto mark when execution has gotten to the point where it is impossible to return to userspace with the calling process unchanged.While techinically this state starts as soon as
de_threadstarts killing threads, the only return path at that point is if there is a fatal signal pending. I have choosen instead to set unrecoverable when the killing stops, and there are possibilities of failures other than fatal signals. In particular it is possible for the allocation of a new sighand structure to fail.Setting unrecoverable at this point has the benefit that other actions can be taken after the other threads are all dead, and the unrecoverable flag can double as a flag that those actions have been taken.
Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 7 ++++--- include/linux/binfmts.h | 7 ++++++- 2 files changed, 10 insertions(+), 4 deletions(-)fs/exec.c +1
-1@@ -1061,7 +1061,7 @@ static int exec_mmap(struct mm_struct *mm)* disturbing other processes. (Other processes might share the signal * table via the CLONE_SIGHAND option to clone().) */ -static int de_thread (struct task_struct *tsk) +static int de_thread (struct linux_binprm *bprm,struct task_struct *tsk) {struct signal_struct *sig = tsk->signal;struct sighand_struct *oldsighand = tsk->sighand;
fs/exec.c static int de_thread(struct task_struct *tsk) · line 1188
Quoted 7 lines
fs/exec.c +1
-0@@ -1182,6 +1182,7 @@ static int de_thread(struct task_struct *tsk)release_task (leader); } + bprm->unrecoverable = true; sig->group_exit_task =NULL ; sig->notify_count =0 ;
ah, sorry,
if (thread_group_empty(tsk))
goto no_thread_group;will skip this:
sig->group_exit_task = NULL;
sig->notify_count = 0;no_thread_group:
/* we have changed execution domain */
tsk->exit_signal = SIGCHLD;so I think the bprm->unrecoverable = true; should be here?
Bernd.
Quoted 37 lines
fs/exec.c +2
-2@@ -1266,7 +1267,7 @@ int flush_old_exec(struct linux_binprm * bprm)* Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (bprm, current);if (retval) goto out;@@ -1664,7 +1665,7 @@ int search_binary_handler(struct linux_binprm *bprm)read_lock (&binfmt_lock);put_binfmt (fmt); -if (retval <0 && !bprm->mm) { +if (retval <0 && bprm->unrecoverable) {/* we got to flush_old_exec() and failed after it */ read_unlock (&binfmt_lock);force_sigsegv (SIGSEGV );include/linux/binfmts.h +6
-1@@ -44,7 +44,12 @@ struct linux_binprm {* exec has happened. Used to sanitize execution environment * and to set AT_SECURE auxv for glibc. */ - secureexec:1 ; + secureexec:1 , +/* +* Set when changes have been made that prevent returning +* to userspace. +*/ + unrecoverable:1;#ifdef __alpha__ unsigned int taso:1;#endif
Re: [PATCH 1/2] exec: Properly mark the point of no return
Quoted 51 lines · Bernd Edlinger
Quoted 37 lines · Eric W. Biederman
Add a flag
binfmt->unrecoverableto mark when execution has gotten to the point where it is impossible to return to userspace with the calling process unchanged.While techinically this state starts as soon as
de_threadstarts killing threads, the only return path at that point is if there is a fatal signal pending. I have choosen instead to set unrecoverable when the killing stops, and there are possibilities of failures other than fatal signals. In particular it is possible for the allocation of a new sighand structure to fail.Setting unrecoverable at this point has the benefit that other actions can be taken after the other threads are all dead, and the unrecoverable flag can double as a flag that those actions have been taken.
Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 7 ++++--- include/linux/binfmts.h | 7 ++++++- 2 files changed, 10 insertions(+), 4 deletions(-)fs/exec.c +1
-1@@ -1061,7 +1061,7 @@ static int exec_mmap(struct mm_struct *mm)* disturbing other processes. (Other processes might share the signal * table via the CLONE_SIGHAND option to clone().) */ -static int de_thread (struct task_struct *tsk) +static int de_thread (struct linux_binprm *bprm,struct task_struct *tsk) {struct signal_struct *sig = tsk->signal;struct sighand_struct *oldsighand = tsk->sighand;fs/exec.c static int de_thread(struct task_struct *tsk) · line 1188
Quoted 7 lines
fs/exec.c +1
-0@@ -1182,6 +1182,7 @@ static int de_thread(struct task_struct *tsk)release_task (leader); } + bprm->unrecoverable = true; sig->group_exit_task =NULL ; sig->notify_count =0 ;ah, sorry,
if (thread_group_empty(tsk)) goto no_thread_group;will skip this:
Quoted 8 lines
sig->group_exit_task = NULL; sig->notify_count = 0;
no_thread_group:/* we have changed execution domain */ tsk->exit_signal = SIGCHLD;so I think the
bprm->unrecoverable= true; should be here?
Absolutely. Thank you very much.
This is why I try and keep things to one clear simple thing per patch so silly thinkos like that can be caught.
Eric
Re: [PATCH 1/2] exec: Properly mark the point of no return
Quoted 9 lines · Eric W. Biederman
Quoted 7 lines · Bernd Edlinger
Quoted 6 lines · Eric W. Biederman
Add a flag
binfmt->unrecoverableto mark when execution has gotten to the point where it is impossible to return to userspace with the calling process unchanged.While techinically this state starts as soon as
de_threadstarts
typo: s/techinically/technically/
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
killing threads, the only return path at that point is if there is a fatal signal pending. I have choosen instead to set unrecoverable
I'm not good at english, is this chosen ?
Bernd.
Re: [PATCH 1/2] exec: Properly mark the point of no return
Quoted 9 lines · Bernd Edlinger
Quoted 8 lines · Eric W. Biederman
Quoted 6 lines · Bernd Edlinger
Quoted 5 lines · Eric W. Biederman
Add a flag
binfmt->unrecoverableto mark when execution has gotten to the point where it is impossible to return to userspace with the calling process unchanged.
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
While techinically this state starts as soon as
de_threadstartstypo: s/techinically/technically/
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
killing threads, the only return path at that point is if there is a fatal signal pending. I have choosen instead to set unrecoverable
I'm not good at english, is this chosen ?
Yes. Defintley worth fixing.
Eric
PATCH 2/2 exec: Add a exec_update_mutex to replace cred_guard_mutex
In reply to Eric W. BiedermanThe cred_guard_mutex is problematic. The cred_guard_mutex is held over the userspace accesses as the arguments from userspace are read. The cred_guard_mutex is held of PTRACE_EVENT_EXIT as the the other threads are killed. The cred_guard_mutex is held over "put_user(0, tsk->clear_child_tid)" in exit_mm().
Any of those can result in deadlock, as the cred_guard_mutex is held over a possible indefinite userspace waits for userspace.
Add exec_update_mutex that is only held over exec updating process with the new contents of exec, so that code that needs not to be confused by exec changing the mm and the cred in ways that can not happen during ordinary execution of a process can take.
The plan is to switch the users of cred_guard_mutex to exed_udpate_mutex one by one. This lets us move forward while still being careful and not introducing any regressions.
Ref: 45c1a159b85b ("Add PTRACE_O_TRACEVFORKDONE and PTRACE_O_TRACEEXIT facilities.")
Ref: 456f17cd1a28 ("[PATCH] user-vm-unlock-2.5.31-A2")
--- fs/exec.c | 4 ++++ include/linux/sched/signal.h | 9 ++++++++- kernel/fork.c | 1 + 3 files changed, 13 insertions(+), 1 deletion(-)
fs/exec.c +4 -0
@@ -1182,6 +1182,7 @@ static int de_thread(struct linux_binprm *bprm, struct task_struct *tsk)release_task (leader); } +mutex_lock (¤t->signal->exec_update_mutex); bprm->unrecoverable = true; sig->group_exit_task =NULL ; sig->notify_count =0 ;
@@ -1425,6 +1426,8 @@ static void free_bprm(struct linux_binprm *bprm) {free_arg_pages (bprm);if (bprm->cred) { +if (bprm->unrecoverable) +mutex_unlock (¤t->signal->exec_update_mutex);mutex_unlock (¤t->signal->cred_guard_mutex);abort_creds (bprm->cred); }
@@ -1474,6 +1477,7 @@ void install_exec_creds(struct linux_binprm *bprm)* credentials; any time after this it may be unlocked. */ security_bprm_committed_creds (bprm); +mutex_unlock (¤t->signal->exec_update_mutex);mutex_unlock (¤t->signal->cred_guard_mutex); }EXPORT_SYMBOL (install_exec_creds);
include/linux/sched/signal.h +8 -1
@@ -224,7 +224,14 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations -* (notably. ptrace) */ +* (notably. ptrace) +* Deprecated do not use in new code. +* Use exec_update_mutex instead. +*/ +struct mutex exec_update_mutex;/* Held while task_struct is being +* updated during exec, and may have +* inconsistent permissions. +*/ } __randomize_layout;/*
kernel/fork.c +1 -0
@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); +mutex_init (&sig->exec_update_mutex);return 0 ; }
Signature · 1 line
2.25.0
Re: [PATCH 2/2] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 88 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other threads are killed. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Any of those can result in deadlock, as the
cred_guard_mutexis held over a possible indefinite userspace waits for userspace.Add
exec_update_mutexthat is only held over exec updating process with the new contents of exec, so that code that needs not to be confused by exec changing the mm and the cred in ways that can not happen during ordinary execution of a process can take.The plan is to switch the users of
cred_guard_mutextoexed_udpate_mutexone by one. This lets us move forward while still being careful and not introducing any regressions.Ref:
45c1a159b85b("AddPTRACE_O_TRACEVFORKDONEandPTRACE_O_TRACEEXITfacilities.")
Ref:456f17cd1a28("[PATCH] user-vm-unlock-2.5.31-A2")Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 4 ++++ include/linux/sched/signal.h | 9 ++++++++- kernel/fork.c | 1 + 3 files changed, 13 insertions(+), 1 deletion(-)fs/exec.c +4
-0@@ -1182,6 +1182,7 @@ static int de_thread(struct linux_binprm *bprm, struct task_struct *tsk)release_task (leader); } +mutex_lock (¤t->signal->exec_update_mutex); bprm->unrecoverable = true; sig->group_exit_task =NULL ; sig->notify_count =0 ;@@ -1425,6 +1426,8 @@ static void free_bprm(struct linux_binprm *bprm) {free_arg_pages (bprm);if (bprm->cred) { +if (bprm->unrecoverable) +mutex_unlock (¤t->signal->exec_update_mutex);mutex_unlock (¤t->signal->cred_guard_mutex);abort_creds (bprm->cred); }@@ -1474,6 +1477,7 @@ void install_exec_creds(struct linux_binprm *bprm)* credentials; any time after this it may be unlocked. */ security_bprm_committed_creds (bprm); +mutex_unlock (¤t->signal->exec_update_mutex);mutex_unlock (¤t->signal->cred_guard_mutex); }EXPORT_SYMBOL (install_exec_creds);include/linux/sched/signal.h +8
-1@@ -224,7 +224,14 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations -* (notably. ptrace) */ +* (notably. ptrace) +* Deprecated do not use in new code. +* Use exec_update_mutex instead. +*/ +struct mutex exec_update_mutex;/* Held while task_struct is being +* updated during exec, and may have +* inconsistent permissions. +*/ } __randomize_layout;/* kernel/fork.c +0
-0
kernel/fork.c static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) · line 1600
Quoted 7 lines
kernel/fork.c +1
-0@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); +mutex_init (&sig->exec_update_mutex);return 0 ; }
Don't you need to add something like this to init/init_task.c ?
.exec_update_mutex = __MUTEX_INITIALIZER(init_signals.exec_update_mutex),
Bernd.
Re: [PATCH 2/2] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 93 lines · Bernd Edlinger
Quoted 92 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other threads are killed. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Any of those can result in deadlock, as the
cred_guard_mutexis held over a possible indefinite userspace waits for userspace.Add
exec_update_mutexthat is only held over exec updating process with the new contents of exec, so that code that needs not to be confused by exec changing the mm and the cred in ways that can not happen during ordinary execution of a process can take.The plan is to switch the users of
cred_guard_mutextoexed_udpate_mutexone by one. This lets us move forward while still being careful and not introducing any regressions.Ref:
45c1a159b85b("AddPTRACE_O_TRACEVFORKDONEandPTRACE_O_TRACEEXITfacilities.")
Ref:456f17cd1a28("[PATCH] user-vm-unlock-2.5.31-A2")Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 4 ++++ include/linux/sched/signal.h | 9 ++++++++- kernel/fork.c | 1 + 3 files changed, 13 insertions(+), 1 deletion(-)fs/exec.c +4
-0@@ -1182,6 +1182,7 @@ static int de_thread(struct linux_binprm *bprm, struct task_struct *tsk)release_task (leader); } +mutex_lock (¤t->signal->exec_update_mutex); bprm->unrecoverable = true; sig->group_exit_task =NULL ; sig->notify_count =0 ;@@ -1425,6 +1426,8 @@ static void free_bprm(struct linux_binprm *bprm) {free_arg_pages (bprm);if (bprm->cred) { +if (bprm->unrecoverable) +mutex_unlock (¤t->signal->exec_update_mutex);mutex_unlock (¤t->signal->cred_guard_mutex);abort_creds (bprm->cred); }@@ -1474,6 +1477,7 @@ void install_exec_creds(struct linux_binprm *bprm)* credentials; any time after this it may be unlocked. */ security_bprm_committed_creds (bprm); +mutex_unlock (¤t->signal->exec_update_mutex);mutex_unlock (¤t->signal->cred_guard_mutex); }EXPORT_SYMBOL (install_exec_creds);include/linux/sched/signal.h +8
-1@@ -224,7 +224,14 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations -* (notably. ptrace) */ +* (notably. ptrace) +* Deprecated do not use in new code. +* Use exec_update_mutex instead. +*/ +struct mutex exec_update_mutex;/* Held while task_struct is being +* updated during exec, and may have +* inconsistent permissions. +*/ } __randomize_layout;/* kernel/fork.c +1
-0@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); +mutex_init (&sig->exec_update_mutex);
Quoted 6 lines
kernel/fork.c static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) · line 1600
Quoted 3 lines
kernel/fork.c +0
-0return 0 ; }Don't you need to add something like this to
init/init_task.c?.exec_update_mutex =
__MUTEX_INITIALIZER(init_signals.exec_update_mutex),
Yes. I overlooked that. Thank you.
Eric
Re: [PATCH 2/2] exec: Add a exec_update_mutex to replace cred_guard_mutex
Am 06.03.20 um 06:17 schrieb Eric W. Biederman:
Quoted 44 lines
Quoted 42 lines · Bernd Edlinger
Quoted 41 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other threads are killed. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Any of those can result in deadlock, as the
cred_guard_mutexis held over a possible indefinite userspace waits for userspace.Add
exec_update_mutexthat is only held over exec updating process with the new contents of exec, so that code that needs not to be confused by exec changing the mm and the cred in ways that can not happen during ordinary execution of a process can take.The plan is to switch the users of
cred_guard_mutextoexed_udpate_mutexone by one. This lets us move forward while still being careful and not introducing any regressions.Ref:
45c1a159b85b("AddPTRACE_O_TRACEVFORKDONEandPTRACE_O_TRACEEXITfacilities.")
Ref:456f17cd1a28("[PATCH] user-vm-unlock-2.5.31-A2")Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 4 ++++ include/linux/sched/signal.h | 9 ++++++++- kernel/fork.c | 1 + 3 files changed, 13 insertions(+), 1 deletion(-)fs/exec.c +0
-0@@ -1182,6 +1182,7 @@ static int de_thread(struct linux_binprm *bprm, struct task_struct *tsk)release_task (leader); }
Quoted 1 line
Quoted 1 line
fs/exec.c static int de_thread(struct linux_binprm *bprm, struct task_struct *tsk) · line 1185
Quoted 1 line
fs/exec.c +1
-0+mutex_lock (¤t->signal->exec_update_mutex);
And by the way, could you make this mutex_lock_killable?
Bernd.
Re: [PATCH 2/2] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 45 lines · Bernd Edlinger
Am 06.03.20 um 06:17 schrieb Eric W. Biederman:
Quoted 44 lines
Quoted 42 lines · Bernd Edlinger
Quoted 41 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other threads are killed. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Any of those can result in deadlock, as the
cred_guard_mutexis held over a possible indefinite userspace waits for userspace.Add
exec_update_mutexthat is only held over exec updating process with the new contents of exec, so that code that needs not to be confused by exec changing the mm and the cred in ways that can not happen during ordinary execution of a process can take.The plan is to switch the users of
cred_guard_mutextoexed_udpate_mutexone by one. This lets us move forward while still being careful and not introducing any regressions.Ref:
45c1a159b85b("AddPTRACE_O_TRACEVFORKDONEandPTRACE_O_TRACEEXITfacilities.")
Ref:456f17cd1a28("[PATCH] user-vm-unlock-2.5.31-A2")Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 4 ++++ include/linux/sched/signal.h | 9 ++++++++- kernel/fork.c | 1 + 3 files changed, 13 insertions(+), 1 deletion(-)fs/exec.c +0
-0@@ -1182,6 +1182,7 @@ static int de_thread(struct linux_binprm *bprm, struct task_struct *tsk)release_task (leader); }
Quoted 3 lines
Quoted 1 line
Quoted 1 line
fs/exec.c static int de_thread(struct linux_binprm *bprm, struct task_struct *tsk) · line 1185
Quoted 1 line
fs/exec.c +1
-0+mutex_lock (¤t->signal->exec_update_mutex);And by the way, could you make this
mutex_lock_killable?
For some reason when I first read this suggestion I thought making this mutex_lock_killable would cause me to rework the logic of when I set unrecoverable and when I unlock the mutex. I blame a tired brain. If a process has received a fatal signal none of that matters.
So yes I will do that just to make things robust in case we miss something that would still make it possible to deadlock in with the new mutex.
I am a little worried that the new mutex might still cover a little too much. But past a certain point I we are not being able to make this code perfect in the first change. The best we can do is to be careful and avoid regressions. Whatever slips through we can fix when we spot the problem.
Eric
Re: [PATCH 2/2] exec: Add a exec_update_mutex to replace cred_guard_mutex
In reply to Eric W. BiedermanQuoted 10 lines · Eric W. Biederman
Quoted 8 lines · Bernd Edlinger
Quoted 7 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other threads are killed. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().
I am all for this patch, and the direction it is heading, Eric.
I just wanted to add a note that I think it is
possible that exec_mm_release can also invoke put_user(0, tsk->clear_child_tid), under the new exec_update_mutex, since vm_access increments the mm->mm_users, under the cred_update_mutex, but releases the mutex, and the caller can hold the reference for a while and then exec_mmap is not releasing the last reference.
Bernd.
Re: [PATCH 2/2] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 12 lines · Bernd Edlinger
Quoted 10 lines · Eric W. Biederman
Quoted 8 lines · Bernd Edlinger
Quoted 7 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other threads are killed. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().
Quoted 8 lines
I am all for this patch, and the direction it is heading, Eric.
I just wanted to add a note that I think it is
possible thatexec_mm_releasecan also invokeput_user(0, tsk->clear_child_tid), under the newexec_update_mutex, sincevm_accessincrements themm->mm_users, under thecred_update_mutex, but releases the mutex, and the caller can hold the reference for a while and thenexec_mmapis not releasing the last reference.
Good catch. I really appreciate your close look at the details.
I am wondering if process_vm_readv and process_vm_writev could be safely changed to use mmgrab and mmdrop, instead of mmget and mmput.
That would resolve the potential issue you have pointed out. I just haven't figured out if it is safe. The mm code has been seriously refactored since I knew how it all worked.
Eric
Re: [PATCH 2/2] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 23 lines · [email protected] (Eric W. Biederman) writes:
Quoted 12 lines · Bernd Edlinger
Quoted 10 lines · Eric W. Biederman
Quoted 8 lines · Bernd Edlinger
Quoted 7 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other threads are killed. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Quoted 8 lines
I am all for this patch, and the direction it is heading, Eric.
I just wanted to add a note that I think it is
possible thatexec_mm_releasecan also invokeput_user(0, tsk->clear_child_tid), under the newexec_update_mutex, sincevm_accessincrements themm->mm_users, under thecred_update_mutex, but releases the mutex, and the caller can hold the reference for a while and thenexec_mmapis not releasing the last reference.
Quoted 8 lines
Good catch. I really appreciate your close look at the details.
I am wondering if
process_vm_readvandprocess_vm_writevcould be safely changed to use mmgrab and mmdrop, instead of mmget and mmput.That would resolve the potential issue you have pointed out. I just haven't figured out if it is safe. The mm code has been seriously refactored since I knew how it all worked.
Nope, mmget can not be replaced by mmgrab.
It might be possible to do something creative like store a cred in place of the userns on the mm and use that for mm_access permission checks. Still we are talking a pretty narrow window, and a case that no one has figured out how to trigger yet. So I will leave that corner case as something for future improvements.
Eric
Re: [PATCH 2/2] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 34 lines · [email protected] (Eric W. Biederman) writes:
Quoted 23 lines · [email protected] (Eric W. Biederman) writes:
Quoted 12 lines · Bernd Edlinger
Quoted 10 lines · Eric W. Biederman
Quoted 8 lines · Bernd Edlinger
Quoted 7 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other threads are killed. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Quoted 8 lines
I am all for this patch, and the direction it is heading, Eric.
I just wanted to add a note that I think it is
possible thatexec_mm_releasecan also invokeput_user(0, tsk->clear_child_tid), under the newexec_update_mutex, sincevm_accessincrements themm->mm_users, under thecred_update_mutex, but releases the mutex, and the caller can hold the reference for a while and thenexec_mmapis not releasing the last reference.Quoted 8 lines
Good catch. I really appreciate your close look at the details.
I am wondering if
process_vm_readvandprocess_vm_writevcould be safely changed to use mmgrab and mmdrop, instead of mmget and mmput.That would resolve the potential issue you have pointed out. I just haven't figured out if it is safe. The mm code has been seriously refactored since I knew how it all worked.
Quoted 7 lines
Nope, mmget can not be replaced by mmgrab.
It might be possible to do something creative like store a cred in place of the userns on the mm and use that for
mm_accesspermission checks. Still we are talking a pretty narrow window, and a case that no one has figured out how to trigger yet. So I will leave that corner case as something for future improvements.
My brain is restless and keep looking at it.
The worst case is processes created with CLONE_VM|CLONE_CHILD_CLEARTID but not CLONE_THREAD. For those that put_user will occur ever time in exec_mmap.
The only solution that I can see is to move taking the new mutex after exec_mm_release. Which may be feasible given how close exec_mmap follows de_thread.
I am going to sleep on that and perhaps I will be able to see how to move taking the mutex lower.
It would be very nice not to have a known issue going into this set of changes.
Eric
PATCH 1/1 exec: make de_thread alloc new signal struct earlier
It was pointed out that de_thread may return -ENOMEM when it already terminated threads, and returning an error from execve, except when a fatal signal is being delivered is not an option any more.
Allocate the memory for the signal table earlier, and make sure that -ENOMEM is returned before the unrecoverable actions are started.
---
Eric, what do you think, might this be helpful to move the "point of no return" lower, and simplify your patch?
fs/exec.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)
fs/exec.c +23 -8
@@ -1057,16 +1057,26 @@ static int exec_mmap(struct mm_struct *mm)* disturbing other processes. (Other processes might share the signal * table via the CLONE_SIGHAND option to clone().) */ -static int de_thread (struct task_struct *tsk) +static int de_thread (void ) { +struct task_struct *tsk = current;struct signal_struct *sig = tsk->signal;struct sighand_struct *oldsighand = tsk->sighand;spinlock_t *lock = &oldsighand->siglock; +struct sighand_struct *newsighand =NULL ;if (thread_group_empty (tsk)) goto no_thread_group;/* +* This is the last time for an out of memory error. +* After this point only fatal signals are are okay. +*/ + newsighand =kmem_cache_alloc (sighand_cachep,GFP_KERNEL ); +if (!newsighand) +return -ENOMEM ; + +/* * Kill all other threads in the thread group. */ spin_lock_irq (lock);
@@ -1076,7 +1086,7 @@ static int de_thread(struct task_struct *tsk)* return so that the signal is processed. */ spin_unlock_irq (lock); -return -EAGAIN ; + goto err_free; } sig->group_exit_task = tsk;
@@ -1191,14 +1201,16 @@ static int de_thread(struct task_struct *tsk)#endif if (refcount_read (&oldsighand->count) !=1 ) { -struct sighand_struct *newsighand;/* * This ->sighand is shared with the CLONE_SIGHAND * but not CLONE_THREAD task, switch to the new one. */ - newsighand =kmem_cache_alloc (sighand_cachep,GFP_KERNEL ); -if (!newsighand) -return -ENOMEM ; +if (!newsighand) { + newsighand =kmem_cache_alloc (sighand_cachep, +GFP_KERNEL ); +if (!newsighand) +return -ENOMEM ; + }refcount_set (&newsighand->count,1 ); memcpy(newsighand->action, oldsighand->action,
@@ -1211,7 +1223,8 @@ static int de_thread(struct task_struct *tsk)write_unlock_irq (&tasklist_lock);__cleanup_sighand (oldsighand); - } + }else if (newsighand) +kmem_cache_free (sighand_cachep, newsighand);BUG_ON (!thread_group_leader (tsk));return 0 ;
@@ -1222,6 +1235,8 @@ static int de_thread(struct task_struct *tsk) sig->group_exit_task =NULL ; sig->notify_count =0 ;read_unlock (&tasklist_lock); +err_free: +kmem_cache_free (sighand_cachep, newsighand);return -EAGAIN ; }
@@ -1262,7 +1277,7 @@ int flush_old_exec(struct linux_binprm * bprm)* Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread ();if (retval) goto out;
Signature · 1 line
1.9.1
Re: [PATCH] exec: make de_thread alloc new signal struct earlier
Quoted 9 lines · Bernd Edlinger
It was pointed out that
de_threadmay return-ENOMEMwhen it already terminated threads, and returning an error from execve, except when a fatal signal is being delivered is not an option any more.Allocate the memory for the signal table earlier, and make sure that
-ENOMEMis returned before the unrecoverable actions are started.
Quoted 5 lines
Signed-off-by Bernd Edlinger <[email protected]>---
Eric, what do you think, might this be helpful to move the "point of no return" lower, and simplify your patch?
Good thinking but no. In this case it is possible to move the entire allocation lower. As well as the posix timer cleanup. That code is actually much clearer outside of de_thread. I will post a patch in that direction in a moment.
It is something of a bad idea to move the new sighand allocation sooner because in practice it does not happen. It only exists to support the CLONE_VM | CLONE_SIGHAND without CLONE_SIGNAL case which is not used by the modern posix thread libraries.
There are just enough old executables floating out there that I don't think we can remove the CLONE_SIGHAND case in general but I keep dreaming about it. We get a lot of complexity in the code to support something that no one really does anymore.
Eric
Quoted 92 lines
fs/exec.c | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-)fs/exec.c +23
-8@@ -1057,16 +1057,26 @@ static int exec_mmap(struct mm_struct *mm)* disturbing other processes. (Other processes might share the signal * table via the CLONE_SIGHAND option to clone().) */ -static int de_thread (struct task_struct *tsk) +static int de_thread (void ) { +struct task_struct *tsk = current;struct signal_struct *sig = tsk->signal;struct sighand_struct *oldsighand = tsk->sighand;spinlock_t *lock = &oldsighand->siglock; +struct sighand_struct *newsighand =NULL ;if (thread_group_empty (tsk)) goto no_thread_group;/* +* This is the last time for an out of memory error. +* After this point only fatal signals are are okay. +*/ + newsighand =kmem_cache_alloc (sighand_cachep,GFP_KERNEL ); +if (!newsighand) +return -ENOMEM ; + +/* * Kill all other threads in the thread group. */ spin_lock_irq (lock);@@ -1076,7 +1086,7 @@ static int de_thread(struct task_struct *tsk)* return so that the signal is processed. */ spin_unlock_irq (lock); -return -EAGAIN ; + goto err_free; } sig->group_exit_task = tsk;@@ -1191,14 +1201,16 @@ static int de_thread(struct task_struct *tsk)#endif if (refcount_read (&oldsighand->count) !=1 ) { -struct sighand_struct *newsighand;/* * This ->sighand is shared with the CLONE_SIGHAND * but not CLONE_THREAD task, switch to the new one. */ - newsighand =kmem_cache_alloc (sighand_cachep,GFP_KERNEL ); -if (!newsighand) -return -ENOMEM ; +if (!newsighand) { + newsighand =kmem_cache_alloc (sighand_cachep, +GFP_KERNEL ); +if (!newsighand) +return -ENOMEM ; + }refcount_set (&newsighand->count,1 ); memcpy(newsighand->action, oldsighand->action,@@ -1211,7 +1223,8 @@ static int de_thread(struct task_struct *tsk)write_unlock_irq (&tasklist_lock);__cleanup_sighand (oldsighand); - } + }else if (newsighand) +kmem_cache_free (sighand_cachep, newsighand);BUG_ON (!thread_group_leader (tsk));return 0 ;@@ -1222,6 +1235,8 @@ static int de_thread(struct task_struct *tsk) sig->group_exit_task =NULL ; sig->notify_count =0 ;read_unlock (&tasklist_lock); +err_free: +kmem_cache_free (sighand_cachep, newsighand);return -EAGAIN ; }@@ -1262,7 +1277,7 @@ int flush_old_exec(struct linux_binprm * bprm)* Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread ();if (retval) goto out;
Re: [PATCH 0/2] Infrastructure to allow fixing exec deadlocks
In reply to Eric W. BiedermanQuoted 30 lines · Eric W. Biederman
Bernd, everyone
This is how I think the infrastructure change should look that makes way for fixing this issue.
- Correct the point of no return.
- Add a new mutex to replacecred_guard_mutexThen I think it is just going through the existing users of
cred_guard_mutexand fixing them to use the new one.There really aren't that many users of
cred_guard_mutexso we should be able to get through the easy ones fairly quickly. And anything that isn't easy we can wait until we have a good fix.The users of
cred_guard_mutexthat I saw were:fs/proc/base.c: proc_pid_attr_write do_io_accounting proc_pid_stack proc_pid_syscall proc_pid_personality perf_event_open mm_access kcmp pidfd_fget seccomp_set_mode_filter
Quoted 6 lines
Bernd does this make sense to you?
I think we can fix the seccomp/no_new_privs issue with some careful refactoring. We can probably do the same for ptrace but that appears to need a little lsm bug fixing.
Yes, for most functions the proposed "exec_update_mutex" is fine, but we will need a longer-time block for ptrace_attach, seccomp_set_mode_filter and proc_pid_attr_write need to be blocked for the whole exec duration so they need a second "mutex", with deadlock-detection as in my previous patch, if I see that right.
Unfortunately only one of the two test cases can be fixed without the second mutex, of course the mm_access is what cause the practical problem.
Currently for the unlimited user space delay, I have only the case of a ptraced sibling thread on my radar, de_thread waits for the parent to call wait in this case, that can literally take forever. But I know that also PTRACE_CONT may be needed after a PTRACE_EVENT_EXIT.
Can you explain what else in the user space can go wrong to make an unlimited delay in the execve?
Bernd.
Re: [PATCH 0/2] Infrastructure to allow fixing exec deadlocks
Quoted 38 lines · Bernd Edlinger
Quoted 30 lines · Eric W. Biederman
Bernd, everyone
This is how I think the infrastructure change should look that makes way for fixing this issue.
- Correct the point of no return.
- Add a new mutex to replacecred_guard_mutexThen I think it is just going through the existing users of
cred_guard_mutexand fixing them to use the new one.There really aren't that many users of
cred_guard_mutexso we should be able to get through the easy ones fairly quickly. And anything that isn't easy we can wait until we have a good fix.The users of
cred_guard_mutexthat I saw were:fs/proc/base.c: proc_pid_attr_write do_io_accounting proc_pid_stack proc_pid_syscall proc_pid_personality perf_event_open mm_access kcmp pidfd_fget seccomp_set_mode_filterQuoted 6 lines
Bernd does this make sense to you?
I think we can fix the seccomp/no_new_privs issue with some careful refactoring. We can probably do the same for ptrace but that appears to need a little lsm bug fixing.
Quoted 5 lines
Yes, for most functions the proposed "
exec_update_mutex" is fine, but we will need a longer-time block forptrace_attach,seccomp_set_mode_filterandproc_pid_attr_writeneed to be blocked for the whole exec duration so they need a second "mutex", with deadlock-detection as in my previous patch, if I see that right.
So far I am leaving "cred_guard_mutex" as that second "mutex". My sense is that when all we have left are the hard cases we can take those cases out in detail, examine them and see what really can be done.
Quoted 2 lines
Unfortunately only one of the two test cases can be fixed without the second mutex, of course the
mm_accessis what cause the practical problem.
Fixing the practical problems are foremost on my agenda. That and clearing away enough of the noise that we can really focus on the hard problems when we begin to address them.
That way I am hoping we can really solve some of these issues and make them go away.
Quoted 7 lines
Currently for the unlimited user space delay, I have only the case of a ptraced sibling thread on my radar,
de_threadwaits for the parent to call wait in this case, that can literally take forever. But I know that alsoPTRACE_CONTmay be needed after aPTRACE_EVENT_EXIT.Can you explain what else in the user space can go wrong to make an unlimited delay in the execve?
Triggering a page fault. Depending on the backing store or possibly with the use of userfaultfd that page fault can be delayed indefinitely and pretty much be as bad as the ptrace case.
Eric
cover Infrastructure to allow fixing exec deadlocks
In reply to Eric W. BiedermanBernd, everyone
This is how I think the infrastructure change should look that makes way for fixing this issue.
- Cleanup and reorder the code so code that can potentially wait indefinitely for userspace comes at the beginning for flush_old_exec.
- Add a new mutex and take it after we have passed any potential indefinite waits for userspace.
Then I think it is just going through the existing users of cred_guard_mutex and fixing them to use the new one.
There really aren't that many users of cred_guard_mutex so we should be able to get through the easy ones fairly quickly. And anything that isn't easy we can wait until we have a good fix.
The users of cred_guard_mutex that I saw were:
fs/proc/base.c:
proc_pid_attr_write
do_io_accounting
proc_pid_stack
proc_pid_syscall
proc_pid_personality
perf_event_open
mm_access
kcmp
pidfd_fget
seccomp_set_mode_filterBernd I think I have addressed the issues you pointed out in v1. Please let me know if you see anything else.
Eric W. Biederman (5):
exec: Only compute current once in flush_old_exec
exec: Factor unshare_sighand out of de_thread and call it separately
exec: Move cleanup of posix timers on exec out of de_thread
exec: Move exec_mmap right after de_thread in flush_old_exec
exec: Add a exec_update_mutex to replace cred_guard_mutex
fs/exec.c | 65 ++++++++++++++++++++++++++++++-------------- include/linux/sched/signal.h | 9 +++++- init/init_task.c | 1 + kernel/fork.c | 1 + 4 files changed, 54 insertions(+), 22 deletions(-)
PATCH v2 1/5 exec: Only compute current once in flush_old_exec
Make it clear that current only needs to be computed once in flush_old_exec. This may have some efficiency improvements and it makes the code easier to change.
--- fs/exec.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
fs/exec.c +5 -4
@@ -1260,13 +1260,14 @@ void __set_task_comm(struct task_struct *tsk, const char *buf, bool exec)*/ int flush_old_exec (struct linux_binprm * bprm) { +struct task_struct *me = current;int retval;/* * Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (me);if (retval) goto out;
@@ -1294,10 +1295,10 @@ int flush_old_exec(struct linux_binprm * bprm) bprm->mm =NULL ;set_fs (USER_DS ); - current->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD | + me->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD |PF_NOFREEZE |PF_NO_SETAFFINITY );flush_thread (); - current->personality &= ~bprm->per_clear; + me->personality &= ~bprm->per_clear;/* * We have to apply CLOEXEC before we change whether the process is
@@ -1305,7 +1306,7 @@ int flush_old_exec(struct linux_binprm * bprm)* trying to access the should-be-closed file descriptors of a process * undergoing exec(2). */ -do_close_on_exec (current->files); +do_close_on_exec (me->files);return 0 ; out:
Signature · 1 line
2.25.0
Re: [PATCH v2 1/5] exec: Only compute current once in flush_old_exec
Quoted 31 lines · Eric W. Biederman
Make it clear that current only needs to be computed once in
flush_old_exec. This may have some efficiency improvements and it makes the code easier to change.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)fs/exec.c +2
-1@@ -1260,13 +1260,14 @@ void __set_task_comm(struct task_struct *tsk, const char *buf, bool exec)*/ int flush_old_exec (struct linux_binprm * bprm) { +struct task_struct *me = current;int retval;/* * Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (me);if (retval) goto out;
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1299
Quoted 6 lines
fs/exec.c +1
-1@@ -1294,10 +1295,10 @@ int flush_old_exec(struct linux_binprm * bprm) bprm->mm =NULL ;set_fs (USER_DS ); - current->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD | + me->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD |PF_NOFREEZE |PF_NO_SETAFFINITY );
I wonder if this line should be aligned with the previous?
Bernd.
Re: [PATCH v2 1/5] exec: Only compute current once in flush_old_exec
Quoted 34 lines · Bernd Edlinger
Quoted 33 lines · Eric W. Biederman
Make it clear that current only needs to be computed once in
flush_old_exec. This may have some efficiency improvements and it makes the code easier to change.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)fs/exec.c +2
-1@@ -1260,13 +1260,14 @@ void __set_task_comm(struct task_struct *tsk, const char *buf, bool exec)*/ int flush_old_exec (struct linux_binprm * bprm) { +struct task_struct *me = current;int retval;/* * Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (me);if (retval) goto out;@@ -1294,10 +1295,10 @@ int flush_old_exec(struct linux_binprm * bprm) bprm->mm =NULL ;
Quoted 6 lines
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1299
Quoted 4 lines
fs/exec.c +1
-1set_fs (USER_DS ); - current->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD | + me->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD |PF_NOFREEZE |PF_NO_SETAFFINITY );I wonder if this line should be aligned with the previous?
In this case I don't think so. The style used for second line is indent with tabs as much as possible to the right. I haven't changed that.
Further mixing a change in indentation style with just a variable rename will make the patch confusing to read because two things have to be verified at the same time.
So while I see why you ask I think this bit needs to stay as is.
Eric
Re: [PATCH v2 1/5] exec: Only compute current once in flush_old_exec
Quoted 43 lines · Eric W. Biederman
Quoted 34 lines · Bernd Edlinger
Quoted 33 lines · Eric W. Biederman
Make it clear that current only needs to be computed once in
flush_old_exec. This may have some efficiency improvements and it makes the code easier to change.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)fs/exec.c +2
-1@@ -1260,13 +1260,14 @@ void __set_task_comm(struct task_struct *tsk, const char *buf, bool exec)*/ int flush_old_exec (struct linux_binprm * bprm) { +struct task_struct *me = current;int retval;/* * Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (me);if (retval) goto out;@@ -1294,10 +1295,10 @@ int flush_old_exec(struct linux_binprm * bprm) bprm->mm =NULL ;Quoted 6 lines
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1299
Quoted 4 lines
fs/exec.c +1
-1set_fs (USER_DS ); - current->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD | + me->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD |PF_NOFREEZE |PF_NO_SETAFFINITY );I wonder if this line should be aligned with the previous?
Quoted 9 lines
In this case I don't think so. The style used for second line is indent with tabs as much as possible to the right. I haven't changed that.
Further mixing a change in indentation style with just a variable rename will make the patch confusing to read because two things have to be verified at the same time.
So while I see why you ask I think this bit needs to stay as is.
Ah, okay, I see.
Thanks for explaining this rule, I was not aware of it, but I am still new here :)
Thanks
Bernd.
Bernd Edlinger
Reviewed-byRe: [PATCH v2 1/5] exec: Only compute current once in flush_old_exec
Quoted 51 lines · Bernd Edlinger
Quoted 50 lines · Eric W. Biederman
Quoted 34 lines · Bernd Edlinger
Quoted 33 lines · Eric W. Biederman
Make it clear that current only needs to be computed once in
flush_old_exec. This may have some efficiency improvements and it makes the code easier to change.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)fs/exec.c +2
-1@@ -1260,13 +1260,14 @@ void __set_task_comm(struct task_struct *tsk, const char *buf, bool exec)*/ int flush_old_exec (struct linux_binprm * bprm) { +struct task_struct *me = current;int retval;/* * Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (me);if (retval) goto out;@@ -1294,10 +1295,10 @@ int flush_old_exec(struct linux_binprm * bprm) bprm->mm =NULL ;Quoted 6 lines
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1299
Quoted 4 lines
fs/exec.c +1
-1set_fs (USER_DS ); - current->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD | + me->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD |PF_NOFREEZE |PF_NO_SETAFFINITY );I wonder if this line should be aligned with the previous?
In this case I don't think so. The style used for second line is indent with tabs as much as possible to the right. I haven't changed that.
Further mixing a change in indentation style with just a variable rename will make the patch confusing to read because two things have to be verified at the same time.
Quoted 7 lines
Quoted 2 lines
So while I see why you ask I think this bit needs to stay as is.
Ah, okay, I see.
Thanks for explaining this rule, I was not aware of it, but I am still new here :)
Bernd.
Re: [PATCH v2 1/5] exec: Only compute current once in flush_old_exec
In reply to Eric W. BiedermanQuoted 6 lines · Eric W. Biederman
Make it clear that current only needs to be computed once in
flush_old_exec. This may have some efficiency improvements and it makes the code easier to change.Signed-off-by "Eric W. Biederman" <[email protected]>
modulo my suggestion of adding more comments (it could even be kerndoc!) that explicitly states that "me" should always be "current", yup, looks good:
-Kees
Quoted 49 lines
--- fs/exec.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)fs/exec.c +5
-4@@ -1260,13 +1260,14 @@ void __set_task_comm(struct task_struct *tsk, const char *buf, bool exec)*/ int flush_old_exec (struct linux_binprm * bprm) { +struct task_struct *me = current;int retval;/* * Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread (me);if (retval) goto out;@@ -1294,10 +1295,10 @@ int flush_old_exec(struct linux_binprm * bprm) bprm->mm =NULL ;set_fs (USER_DS ); - current->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD | + me->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD |PF_NOFREEZE |PF_NO_SETAFFINITY );flush_thread (); - current->personality &= ~bprm->per_clear; + me->personality &= ~bprm->per_clear;/* * We have to apply CLOEXEC before we change whether the process is @@ -1305,7 +1306,7 @@ int flush_old_exec(struct linux_binprm * bprm)* trying to access the should-be-closed file descriptors of a process * undergoing exec(2). */ -do_close_on_exec (current->files); +do_close_on_exec (me->files);return 0 ; out:Signature · 1 line
2.25.0
Signature · 1 line
Kees Cook
Re: [PATCH v2 1/5] exec: Only compute current once in flush_old_exec
In reply to Eric W. BiedermanQuoted 6 lines · Eric W. Biederman
Make it clear that current only needs to be computed once in
flush_old_exec. This may have some efficiency improvements and it makes the code easier to change.Signed-off-by "Eric W. Biederman" <[email protected]>
PATCH v2 2/5 exec: Factor unshare_sighand out of de_thread and call it separately
In reply to Eric W. BiedermanThis makes the code clearer and makes it easier to implement a mutex that is not taken over any locations that may block indefinitely waiting for userspace.
--- fs/exec.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-)
fs/exec.c +26 -13
@@ -1194,6 +1194,23 @@ static int de_thread(struct task_struct *tsk)flush_itimer_signals ();#endif +BUG_ON (!thread_group_leader (tsk)); +return 0 ; + +killed: +/* protects against exit_notify() and __exit_signal() */ +read_lock (&tasklist_lock); + sig->group_exit_task =NULL ; + sig->notify_count =0 ; +read_unlock (&tasklist_lock); +return -EAGAIN ; +} + + +static int unshare_sighand (struct task_struct *me) +{ +struct sighand_struct *oldsighand = me->sighand; +if (refcount_read (&oldsighand->count) !=1 ) {struct sighand_struct *newsighand;/*
@@ -1210,23 +1227,13 @@ static int de_thread(struct task_struct *tsk)write_lock_irq (&tasklist_lock);spin_lock (&oldsighand->siglock); -rcu_assign_pointer (tsk->sighand, newsighand); +rcu_assign_pointer (me->sighand, newsighand);spin_unlock (&oldsighand->siglock);write_unlock_irq (&tasklist_lock);__cleanup_sighand (oldsighand); } - -BUG_ON (!thread_group_leader (tsk));return 0 ; - -killed: -/* protects against exit_notify() and __exit_signal() */ -read_lock (&tasklist_lock); - sig->group_exit_task =NULL ; - sig->notify_count =0 ; -read_unlock (&tasklist_lock); -return -EAGAIN ; }char *__get_task_comm (char *buf,size_t buf_size,struct task_struct *tsk)
@@ -1264,13 +1271,19 @@ int flush_old_exec(struct linux_binprm * bprm)int retval;/* -* Make sure we have a private signal table and that -* we are unassociated from the previous thread group. +* Make this the only thread in the thread group. */ retval =de_thread (me);if (retval) goto out; +/* +* Make the signal table private. +*/ + retval =unshare_sighand (me); +if (retval) + goto out; +/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update
Signature · 1 line
2.25.0
Re: [PATCH v2 2/5] exec: Factor unshare_sighand out of de_thread and call it separately
Quoted 6 lines · Eric W. Biederman
This makes the code clearer and makes it easier to implement a mutex that is not taken over any locations that may block indefinitely waiting for userspace.
Signed-off-by "Eric W. Biederman" <[email protected]>
Bernd.
Quoted 80 lines
--- fs/exec.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-)fs/exec.c +26
-13@@ -1194,6 +1194,23 @@ static int de_thread(struct task_struct *tsk)flush_itimer_signals ();#endif +BUG_ON (!thread_group_leader (tsk)); +return 0 ; + +killed: +/* protects against exit_notify() and __exit_signal() */ +read_lock (&tasklist_lock); + sig->group_exit_task =NULL ; + sig->notify_count =0 ; +read_unlock (&tasklist_lock); +return -EAGAIN ; +} + + +static int unshare_sighand (struct task_struct *me) +{ +struct sighand_struct *oldsighand = me->sighand; +if (refcount_read (&oldsighand->count) !=1 ) {struct sighand_struct *newsighand;/* @@ -1210,23 +1227,13 @@ static int de_thread(struct task_struct *tsk)write_lock_irq (&tasklist_lock);spin_lock (&oldsighand->siglock); -rcu_assign_pointer (tsk->sighand, newsighand); +rcu_assign_pointer (me->sighand, newsighand);spin_unlock (&oldsighand->siglock);write_unlock_irq (&tasklist_lock);__cleanup_sighand (oldsighand); } - -BUG_ON (!thread_group_leader (tsk));return 0 ; - -killed: -/* protects against exit_notify() and __exit_signal() */ -read_lock (&tasklist_lock); - sig->group_exit_task =NULL ; - sig->notify_count =0 ; -read_unlock (&tasklist_lock); -return -EAGAIN ; }char *__get_task_comm (char *buf,size_t buf_size,struct task_struct *tsk)@@ -1264,13 +1271,19 @@ int flush_old_exec(struct linux_binprm * bprm)int retval;/* -* Make sure we have a private signal table and that -* we are unassociated from the previous thread group. +* Make this the only thread in the thread group. */ retval =de_thread (me);if (retval) goto out; +/* +* Make the signal table private. +*/ + retval =unshare_sighand (me); +if (retval) + goto out; +/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update
Re: [PATCH v2 2/5] exec: Factor unshare_sighand out of de_thread and call it separately
In reply to Eric W. BiedermanQuoted 15 lines · Eric W. Biederman
This makes the code clearer and makes it easier to implement a mutex that is not taken over any locations that may block indefinitely waiting for userspace.
Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-)fs/exec.c +0
-0
fs/exec.c static int de_thread(struct task_struct *tsk) · line 1195
Quoted 2 lines
fs/exec.c +0
-0@@ -1194,6 +1194,23 @@ static int de_thread(struct task_struct *tsk)flush_itimer_signals ();#endif
Semi-related (existing behavior): in de_thread(), what keeps the thread group from changing? i.e.:
if (thread_group_empty(tsk))
goto no_thread_group;
/*
* Kill all other threads in the thread group.
*/
spin_lock_irq(lock);
... kill other threads under lock ...Why is the thread_group_emtpy() test not under lock?
fs/exec.c static int de_thread(struct task_struct *tsk) · line 1201
Quoted 6 lines
fs/exec.c +5
-0+BUG_ON (!thread_group_leader (tsk)); +return 0 ; + +killed: +/* protects against exit_notify() and __exit_signal() */
I wonder if include/linux/sched/task.h's definition of tasklist_lock should explicitly gain note about group_exit_task and notify_count, or, alternatively, signal.h's section on these fields should gain a comment? tasklist_lock is unmentioned in signal.h... :(
Quoted 64 lines
fs/exec.c +21
-13+read_lock (&tasklist_lock); + sig->group_exit_task =NULL ; + sig->notify_count =0 ; +read_unlock (&tasklist_lock); +return -EAGAIN ; +} + + +static int unshare_sighand (struct task_struct *me) +{ +struct sighand_struct *oldsighand = me->sighand; +if (refcount_read (&oldsighand->count) !=1 ) {struct sighand_struct *newsighand;/* @@ -1210,23 +1227,13 @@ static int de_thread(struct task_struct *tsk)write_lock_irq (&tasklist_lock);spin_lock (&oldsighand->siglock); -rcu_assign_pointer (tsk->sighand, newsighand); +rcu_assign_pointer (me->sighand, newsighand);spin_unlock (&oldsighand->siglock);write_unlock_irq (&tasklist_lock);__cleanup_sighand (oldsighand); } - -BUG_ON (!thread_group_leader (tsk));return 0 ; - -killed: -/* protects against exit_notify() and __exit_signal() */ -read_lock (&tasklist_lock); - sig->group_exit_task =NULL ; - sig->notify_count =0 ; -read_unlock (&tasklist_lock); -return -EAGAIN ; }char *__get_task_comm (char *buf,size_t buf_size,struct task_struct *tsk)@@ -1264,13 +1271,19 @@ int flush_old_exec(struct linux_binprm * bprm)int retval;/* -* Make sure we have a private signal table and that -* we are unassociated from the previous thread group. +* Make this the only thread in the thread group. */ retval =de_thread (me);if (retval) goto out; +/* +* Make the signal table private. +*/ + retval =unshare_sighand (me); +if (retval) + goto out; +/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update Signature · 1 line
2.25.0
Otherwise, yes, sensible separation.
Signature · 1 line
Kees Cook
Re: [PATCH v2 2/5] exec: Factor unshare_sighand out of de_thread and call it separately
Quoted 31 lines · Kees Cook
Quoted 15 lines · Eric W. Biederman
This makes the code clearer and makes it easier to implement a mutex that is not taken over any locations that may block indefinitely waiting for userspace.
Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-)fs/exec.c +0
-0fs/exec.c static int de_thread(struct task_struct *tsk) · line 1195
Quoted 2 lines
fs/exec.c +0
-0@@ -1194,6 +1194,23 @@ static int de_thread(struct task_struct *tsk)flush_itimer_signals ();#endif Semi-related (existing behavior): in
de_thread(), what keeps the thread group from changing? i.e.:if (thread_group_empty(tsk)) goto no_thread_group; /* * Kill all other threads in the thread group. */ spin_lock_irq(lock); ... kill other threads under lock ...
Quoted 2 lines
Why is the
thread_group_emtpy()test not under lock?
A new thread cannot created when only one thread is executing, right?
Quoted 81 lines
fs/exec.c static int de_thread(struct task_struct *tsk) · line 1201
Quoted 6 lines
fs/exec.c +5
-0+BUG_ON (!thread_group_leader (tsk)); +return 0 ; + +killed: +/* protects against exit_notify() and __exit_signal() */ I wonder if include/linux/sched/task.h's definition of
tasklist_lockshould explicitly gain note aboutgroup_exit_taskandnotify_count, or, alternatively, signal.h's section on these fields should gain a comment?tasklist_lockis unmentioned insignal.h... :(Quoted 64 lines
fs/exec.c +21
-13+read_lock (&tasklist_lock); + sig->group_exit_task =NULL ; + sig->notify_count =0 ; +read_unlock (&tasklist_lock); +return -EAGAIN ; +} + + +static int unshare_sighand (struct task_struct *me) +{ +struct sighand_struct *oldsighand = me->sighand; +if (refcount_read (&oldsighand->count) !=1 ) {struct sighand_struct *newsighand;/* @@ -1210,23 +1227,13 @@ static int de_thread(struct task_struct *tsk)write_lock_irq (&tasklist_lock);spin_lock (&oldsighand->siglock); -rcu_assign_pointer (tsk->sighand, newsighand); +rcu_assign_pointer (me->sighand, newsighand);spin_unlock (&oldsighand->siglock);write_unlock_irq (&tasklist_lock);__cleanup_sighand (oldsighand); } - -BUG_ON (!thread_group_leader (tsk));return 0 ; - -killed: -/* protects against exit_notify() and __exit_signal() */ -read_lock (&tasklist_lock); - sig->group_exit_task =NULL ; - sig->notify_count =0 ; -read_unlock (&tasklist_lock); -return -EAGAIN ; }char *__get_task_comm (char *buf,size_t buf_size,struct task_struct *tsk)@@ -1264,13 +1271,19 @@ int flush_old_exec(struct linux_binprm * bprm)int retval;/* -* Make sure we have a private signal table and that -* we are unassociated from the previous thread group. +* Make this the only thread in the thread group. */ retval =de_thread (me);if (retval) goto out; +/* +* Make the signal table private. +*/ + retval =unshare_sighand (me); +if (retval) + goto out; +/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update Signature · 1 line
2.25.0Otherwise, yes, sensible separation.
Reviewed-by Kees Cook <[email protected]>
Re: [PATCH v2 2/5] exec: Factor unshare_sighand out of de_thread and call it separately
Quoted 32 lines · Bernd Edlinger
Quoted 31 lines · Kees Cook
Quoted 15 lines · Eric W. Biederman
This makes the code clearer and makes it easier to implement a mutex that is not taken over any locations that may block indefinitely waiting for userspace.
Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-)fs/exec.c +0
-0fs/exec.c static int de_thread(struct task_struct *tsk) · line 1195
Quoted 2 lines
fs/exec.c +0
-0@@ -1194,6 +1194,23 @@ static int de_thread(struct task_struct *tsk)flush_itimer_signals ();#endif Semi-related (existing behavior): in
de_thread(), what keeps the thread group from changing? i.e.:if (thread_group_empty(tsk)) goto no_thread_group; /* * Kill all other threads in the thread group. */ spin_lock_irq(lock); ... kill other threads under lock ...
Quoted 5 lines
Quoted 2 lines
Why is the
thread_group_emtpy()test not under lock?A new thread cannot created when only one thread is executing, right?
*face palm* Yes, of course. :) I'm thinking too hard.
Signature · 1 line
Kees Cook
Re: [PATCH v2 2/5] exec: Factor unshare_sighand out of de_thread and call it separately
In reply to Eric W. BiedermanQuoted 39 lines · Eric W. Biederman
This makes the code clearer and makes it easier to implement a mutex that is not taken over any locations that may block indefinitely waiting for userspace.
Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-)fs/exec.c +17
-0@@ -1194,6 +1194,23 @@ static int de_thread(struct task_struct *tsk)flush_itimer_signals ();#endif +BUG_ON (!thread_group_leader (tsk)); +return 0 ; + +killed: +/* protects against exit_notify() and __exit_signal() */ +read_lock (&tasklist_lock); + sig->group_exit_task =NULL ; + sig->notify_count =0 ; +read_unlock (&tasklist_lock); +return -EAGAIN ; +} + + +static int unshare_sighand (struct task_struct *me) +{ +struct sighand_struct *oldsighand = me->sighand; +if (refcount_read (&oldsighand->count) !=1 ) {struct sighand_struct *newsighand;/*
fs/exec.c static int de_thread(struct task_struct *tsk) · line 1235
Quoted 10 lines
fs/exec.c +1
-1@@ -1210,23 +1227,13 @@ static int de_thread(struct task_struct *tsk)write_lock_irq (&tasklist_lock);spin_lock (&oldsighand->siglock); -rcu_assign_pointer (tsk->sighand, newsighand); +rcu_assign_pointer (me->sighand, newsighand);spin_unlock (&oldsighand->siglock);write_unlock_irq (&tasklist_lock);__cleanup_sighand (oldsighand); }
This is fine for now but we share an aweful lot of code with copy_sighand(). We should earmark this to look into consolidating the core operations into a common helper called from both copy_sighand() and unshare_sighand() maybe even dumbing it down to one helper. But not needed for now.
Otherwise:
PATCH v2 3/5 exec: Move cleanup of posix timers on exec out of de_thread
In reply to Eric W. BiedermanThese functions have very little to do with de_thread move them out of de_thread an into flush_old_exec proper so it can be more clearly seen what flush_old_exec is doing.
--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
fs/exec.c +5 -5
@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)/* we have changed execution domain */ tsk->exit_signal =SIGCHLD ; -#ifdef CONFIG_POSIX_TIMERS -exit_itimers (sig); -flush_itimer_signals (); -#endif -BUG_ON (!thread_group_leader (tsk));return 0 ;
@@ -1277,6 +1272,11 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; +#ifdef CONFIG_POSIX_TIMERS +exit_itimers (me->signal); +flush_itimer_signals (); +#endif +/* * Make the signal table private. */
Signature · 1 line
2.25.0
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
Quoted 6 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>
Bernd.
Quoted 33 lines
--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)fs/exec.c +5
-5@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)/* we have changed execution domain */ tsk->exit_signal =SIGCHLD ; -#ifdef CONFIG_POSIX_TIMERS -exit_itimers (sig); -flush_itimer_signals (); -#endif -BUG_ON (!thread_group_leader (tsk));return 0 ;@@ -1277,6 +1272,11 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; +#ifdef CONFIG_POSIX_TIMERS +exit_itimers (me->signal); +flush_itimer_signals (); +#endif +/* * Make the signal table private. */
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
In reply to Eric W. Biedermanfs/exec.c static int de_thread(struct task_struct *tsk)
Quoted 15 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)fs/exec.c +0
-0@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)
While you're cleaning up de_thread() wouldn't it be good to also take the opportunity and remove the task argument from de_thread(). It's only ever used with current. Could be done in one of your patches or as a separate patch.
fs/exec.c +3 -2
@@ -1061,8 +1061,9 @@ static int exec_mmap(struct mm_struct *mm)* disturbing other processes. (Other processes might share the signal * table via the CLONE_SIGHAND option to clone().) */ -static int de_thread (struct task_struct *tsk) +static int de_thread (void ) { +struct task_struct *tsk = current;struct signal_struct *sig = tsk->signal;struct sighand_struct *oldsighand = tsk->sighand;spinlock_t *lock = &oldsighand->siglock;
@@ -1266,7 +1267,7 @@ int flush_old_exec(struct linux_binprm * bprm)* Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread ();if (retval) goto out;
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
Quoted 11 lines · Christian Brauner
Quoted 10 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
Quoted 10 lines
fs/exec.c static int de_thread(struct task_struct *tsk)
Quoted 5 lines
fs/exec.c +0
-0@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)While you're cleaning up
de_thread()wouldn't it be good to also take the opportunity and remove the task argument fromde_thread(). It's only ever used with current. Could be done in one of your patches or as a separate patch.
How does that affect the code generation?
My sense is that computing current once in flush_old_exec is much better than computing it in each function flush_old_exec calls. I remember that computing current used to be not expensive but noticable.
For clarity I can see renaming tsk to me. So that it is clear we are talking about the current process, and not some arbitrary process.
And for clarity my goal here is not to clean up de_thread. Though I don't mind that result. My goal is to get the extra work out of de_thread so we can do process tear down cleanups that are safe according to the ordinary process rules, before taking a mutex that protects exec mucking with all of the state in exec.
Eric
Quoted 23 lines
fs/exec.c +3
-2@@ -1061,8 +1061,9 @@ static int exec_mmap(struct mm_struct *mm)* disturbing other processes. (Other processes might share the signal * table via the CLONE_SIGHAND option to clone().) */ -static int de_thread (struct task_struct *tsk) +static int de_thread (void ) { +struct task_struct *tsk = current;struct signal_struct *sig = tsk->signal;struct sighand_struct *oldsighand = tsk->sighand;spinlock_t *lock = &oldsighand->siglock;@@ -1266,7 +1267,7 @@ int flush_old_exec(struct linux_binprm * bprm)* Make sure we have a private signal table and that * we are unassociated from the previous thread group. */ - retval =de_thread (current); + retval =de_thread ();if (retval) goto out;
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
Quoted 24 lines · Eric W. Biederman
Quoted 11 lines · Christian Brauner
Quoted 10 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)Quoted 10 lines
fs/exec.c static int de_thread(struct task_struct *tsk)
Quoted 5 lines
fs/exec.c +0
-0@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)While you're cleaning up
de_thread()wouldn't it be good to also take the opportunity and remove the task argument fromde_thread(). It's only ever used with current. Could be done in one of your patches or as a separate patch.
Quoted 1 line
How does that affect the code generation?
The same way renaming "tsk" to "me" does.
Quoted 8 lines
My sense is that computing current once in
flush_old_execis much better than computing it in each functionflush_old_execcalls. I remember that computing current used to be not expensive but noticable.For clarity I can see renaming tsk to me. So that it is clear we are talking about the current process, and not some arbitrary process.
For clarity since de_thread() uses "tsk" giving the impression that any task can be dethreaded while it's only ever used with current. It's just a suggestion since you're doing the rename tsk->me anyway it would fit with the series. You do whatever you want though.
(I just remember that the same request was made once to changes I did:
Don't pass current as arg when it's the only task passed.)
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
Quoted 38 lines · Christian Brauner
Quoted 24 lines · Eric W. Biederman
Quoted 11 lines · Christian Brauner
Quoted 10 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)Quoted 10 lines
fs/exec.c static int de_thread(struct task_struct *tsk)
Quoted 5 lines
fs/exec.c +0
-0@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)While you're cleaning up
de_thread()wouldn't it be good to also take the opportunity and remove the task argument fromde_thread(). It's only ever used with current. Could be done in one of your patches or as a separate patch.Quoted 1 line
How does that affect the code generation?
The same way renaming "tsk" to "me" does.
Quoted 8 lines
My sense is that computing current once in
flush_old_execis much better than computing it in each functionflush_old_execcalls. I remember that computing current used to be not expensive but noticable.For clarity I can see renaming tsk to me. So that it is clear we are talking about the current process, and not some arbitrary process.
Quoted 6 lines
For clarity since
de_thread()uses "tsk" giving the impression that any task can be dethreaded while it's only ever used with current. It's just a suggestion since you're doing the renametsk->meanyway it would fit with the series. You do whatever you want though.
(I just remember that the same request was made once to changes I did:
Don't pass current as arg when it's the only task passed.)
That's fair.
And I completely agree that we should at least rename tsk to me. Just for clarity.
My apologies if I am a little short. My little son has been an extra handful lately.
Eric
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
Quoted 47 lines · Eric W. Biederman
Quoted 38 lines · Christian Brauner
Quoted 24 lines · Eric W. Biederman
Quoted 11 lines · Christian Brauner
Quoted 10 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)Quoted 10 lines
fs/exec.c static int de_thread(struct task_struct *tsk)
Quoted 5 lines
fs/exec.c +0
-0@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)While you're cleaning up
de_thread()wouldn't it be good to also take the opportunity and remove the task argument fromde_thread(). It's only ever used with current. Could be done in one of your patches or as a separate patch.Quoted 1 line
How does that affect the code generation?
The same way renaming "tsk" to "me" does.
Quoted 8 lines
My sense is that computing current once in
flush_old_execis much better than computing it in each functionflush_old_execcalls. I remember that computing current used to be not expensive but noticable.For clarity I can see renaming tsk to me. So that it is clear we are talking about the current process, and not some arbitrary process.
Quoted 6 lines
For clarity since
de_thread()uses "tsk" giving the impression that any task can be dethreaded while it's only ever used with current. It's just a suggestion since you're doing the renametsk->meanyway it would fit with the series. You do whatever you want though.
(I just remember that the same request was made once to changes I did:
Don't pass current as arg when it's the only task passed.)
Quoted 7 lines
That's fair.
And I completely agree that we should at least rename tsk to me. Just for clarity.
My apologies if I am a little short. My little son has been an extra handful lately.
No worries, stress is a thing most of us know too well.
Christian
PATCH 1/1 pidfd: Stop taking cred_guard_mutex
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the cred_guard_mutex as it is useless.
Furthermore he cred_guard_mutex is a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.
--- kernel/pid.c | 6 ------ 1 file changed, 6 deletions(-)
Christian if you don't have any objections I will take this one through my tree.
I tried to figure out why this code path takes the cred_guard_mutex and the archive on lore.kernel.org was not helpful in finding that part of the conversation.
kernel/pid.c +0 -6
@@ -577,17 +577,11 @@ static struct file *__pidfd_fget(struct task_struct *task, int fd)struct file *file;int ret; - ret =mutex_lock_killable (&task->signal->cred_guard_mutex); -if (ret) -return ERR_PTR (ret); -if (ptrace_may_access (task,PTRACE_MODE_ATTACH_REALCREDS )) file =fget_task (task, fd);else file =ERR_PTR (-EPERM ); -mutex_unlock (&task->signal->cred_guard_mutex); -return file ?:ERR_PTR (-EBADF ); }
Signature · 1 line
2.20.1
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
Quoted 15 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.Cc Sargun Dhillon <[email protected]>Cc Christian Brauner <[email protected]>Cc Arnd Bergmann <[email protected]>Fixes 8649c322f75c ("pid: Implement pidfd_getfd syscall")Signed-off-by "Eric W. Biederman" <[email protected]>
Quoted 6 lines
--- kernel/pid.c | 6 ------ 1 file changed, 6 deletions(-)Christian if you don't have any objections I will take this one through my tree.
Sure.
Quoted 4 lines
I tried to figure out why this code path takes the
cred_guard_mutexand the archive on lore.kernel.org was not helpful in finding that part of the conversation.
Let me think a little harder and hopefully get back to you with a sensible explanation.
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
In reply to Eric W. BiedermanQuoted 8 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.
Please don't. Just use the new exec_update_mutex like everywhere else.
Quoted 9 lines
Cc Sargun Dhillon <[email protected]>Cc Christian Brauner <[email protected]>Cc Arnd Bergmann <[email protected]>Fixes 8649c322f75c ("pid: Implement pidfd_getfd syscall")Signed-off-by "Eric W. Biederman" <[email protected]>--- kernel/pid.c | 6 ------ 1 file changed, 6 deletions(-)
Quoted 6 lines
Christian if you don't have any objections I will take this one through my tree.
I tried to figure out why this code path takes the
cred_guard_mutexand the archive on lore.kernel.org was not helpful in finding that part of the conversation.
That was my suggestion.
Quoted 11 lines
kernel/pid.c +0
-3@@ -577,17 +577,11 @@ static struct file *__pidfd_fget(struct task_struct *task, int fd)struct file *file;int ret; - ret =mutex_lock_killable (&task->signal->cred_guard_mutex); -if (ret) -return ERR_PTR (ret);
kernel/pid.c static struct file *__pidfd_fget(struct task_struct *task, int fd) · line 586
Quoted 10 lines
kernel/pid.c +0
-3-if (ptrace_may_access (task,PTRACE_MODE_ATTACH_REALCREDS )) file =fget_task (task, fd);else file =ERR_PTR (-EPERM ); -mutex_unlock (&task->signal->cred_guard_mutex); -return file ?:ERR_PTR (-EBADF ); }
If you make this change, then if this races with execution of a setuid program that afterwards e.g. opens a unix domain socket, an attacker will be able to steal that socket and inject messages into communication with things like DBus. procfs currently has the same race, and that still needs to be fixed, but at least procfs doesn't let you open things like sockets because they don't have a working ->open handler, and it enforces the normal permission check for opening files.
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
Quoted 47 lines · Jann Horn
Quoted 8 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.Please don't. Just use the new
exec_update_mutexlike everywhere else.Quoted 9 lines
Cc Sargun Dhillon <[email protected]>Cc Christian Brauner <[email protected]>Cc Arnd Bergmann <[email protected]>Fixes 8649c322f75c ("pid: Implement pidfd_getfd syscall")Signed-off-by "Eric W. Biederman" <[email protected]>--- kernel/pid.c | 6 ------ 1 file changed, 6 deletions(-)Quoted 6 lines
Christian if you don't have any objections I will take this one through my tree.
I tried to figure out why this code path takes the
cred_guard_mutexand the archive on lore.kernel.org was not helpful in finding that part of the conversation.That was my suggestion.
Quoted 17 lines
kernel/pid.c +0
-4@@ -577,17 +577,11 @@ static struct file *__pidfd_fget(struct task_struct *task, int fd)struct file *file;int ret; - ret =mutex_lock_killable (&task->signal->cred_guard_mutex); -if (ret) -return ERR_PTR (ret); -if (ptrace_may_access (task,PTRACE_MODE_ATTACH_REALCREDS )) file =fget_task (task, fd);else file =ERR_PTR (-EPERM );
Quoted 13 lines
kernel/pid.c static struct file *__pidfd_fget(struct task_struct *task, int fd) · line 586
Quoted 4 lines
kernel/pid.c +0
-2-mutex_unlock (&task->signal->cred_guard_mutex); -return file ?:ERR_PTR (-EBADF ); }If you make this change, then if this races with execution of a setuid program that afterwards e.g. opens a unix domain socket, an attacker will be able to steal that socket and inject messages into communication with things like DBus. procfs currently has the same race, and that still needs to be fixed, but at least procfs doesn't let you open things like sockets because they don't have a working ->open handler, and it enforces the normal permission check for opening files.
It isn't only exec that can change credentials. Do we need a lock for changing credentials?
Wouldn't it be sufficient to simply test ptrace_may_access after we get a copy of the file?
If we need a lock around credential change let's design and build that. Having a mismatch between what a lock is designed to do, and what people use it for can only result in other bugs as people get confused.
Eric
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
Quoted 62 lines · Eric W. Biederman
Quoted 47 lines · Jann Horn
Quoted 8 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.Please don't. Just use the new
exec_update_mutexlike everywhere else.Quoted 9 lines
Cc Sargun Dhillon <[email protected]>Cc Christian Brauner <[email protected]>Cc Arnd Bergmann <[email protected]>Fixes 8649c322f75c ("pid: Implement pidfd_getfd syscall")Signed-off-by "Eric W. Biederman" <[email protected]>--- kernel/pid.c | 6 ------ 1 file changed, 6 deletions(-)Quoted 6 lines
Christian if you don't have any objections I will take this one through my tree.
I tried to figure out why this code path takes the
cred_guard_mutexand the archive on lore.kernel.org was not helpful in finding that part of the conversation.That was my suggestion.
Quoted 17 lines
kernel/pid.c +0
-4@@ -577,17 +577,11 @@ static struct file *__pidfd_fget(struct task_struct *task, int fd)struct file *file;int ret; - ret =mutex_lock_killable (&task->signal->cred_guard_mutex); -if (ret) -return ERR_PTR (ret); -if (ptrace_may_access (task,PTRACE_MODE_ATTACH_REALCREDS )) file =fget_task (task, fd);else file =ERR_PTR (-EPERM );Quoted 13 lines
kernel/pid.c static struct file *__pidfd_fget(struct task_struct *task, int fd) · line 586
Quoted 4 lines
kernel/pid.c +0
-2-mutex_unlock (&task->signal->cred_guard_mutex); -return file ?:ERR_PTR (-EBADF ); }If you make this change, then if this races with execution of a setuid program that afterwards e.g. opens a unix domain socket, an attacker will be able to steal that socket and inject messages into communication with things like DBus. procfs currently has the same race, and that still needs to be fixed, but at least procfs doesn't let you open things like sockets because they don't have a working ->open handler, and it enforces the normal permission check for opening files.
Quoted 2 lines
It isn't only exec that can change credentials. Do we need a lock for changing credentials?
Hmm, I guess so? Normally, a task that's changing credentials becomes nondumpable at the same time (and there are explicit memory barriers in commit_creds() and __ptrace_may_access() to enforce the ordering for this); so you normally don't see tasks becoming ptrace-accessible via anything other than execve(). But I guess if someone opens a root-only file, closes it, drops privileges, and then explicitly does prctl(PR_SET_DUMPABLE, 1), we should probably protect that, too.
Quoted 2 lines
Wouldn't it be sufficient to simply test
ptrace_may_accessafter we get a copy of the file?
There are also setuid helpers that can, after having done privileged stuff, drop privileges and call execve(); after that, ptrace_may_access() succeeds again. In particular, polkit has a helper that does this.
Quoted 3 lines
If we need a lock around credential change let's design and build that. Having a mismatch between what a lock is designed to do, and what people use it for can only result in other bugs as people get confused.
Hmm... what benefits do we get from making it a separate lock? I guess it would allow us to make it a per-task lock instead of a signal_struct-wide one? That might be helpful...
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
Quoted 11 lines · Jann Horn
Quoted 10 lines · Eric W. Biederman
Quoted 9 lines · Jann Horn
Quoted 8 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.
[...]
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
If you make this change, then if this races with execution of a setuid program that afterwards e.g. opens a unix domain socket, an attacker will be able to steal that socket and inject messages into communication with things like DBus. procfs currently has the same race, and that still needs to be fixed, but at least procfs doesn't let you open things like sockets because they don't have a working ->open handler, and it enforces the normal permission check for opening files.
It isn't only exec that can change credentials. Do we need a lock for changing credentials?
[...]
Quoted 7 lines
Quoted 3 lines
If we need a lock around credential change let's design and build that. Having a mismatch between what a lock is designed to do, and what people use it for can only result in other bugs as people get confused.
Hmm... what benefits do we get from making it a separate lock? I guess it would allow us to make it a per-task lock instead of a signal_struct-wide one? That might be helpful...
But actually, isn't the core purpose of the cred_guard_mutex to guard against concurrent credential changes anyway? That's what almost everyone uses it for, and it's in the name...
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
Quoted 29 lines · Jann Horn
Quoted 11 lines · Jann Horn
Quoted 10 lines · Eric W. Biederman
Quoted 9 lines · Jann Horn
Quoted 8 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.[...]
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
If you make this change, then if this races with execution of a setuid program that afterwards e.g. opens a unix domain socket, an attacker will be able to steal that socket and inject messages into communication with things like DBus. procfs currently has the same race, and that still needs to be fixed, but at least procfs doesn't let you open things like sockets because they don't have a working ->open handler, and it enforces the normal permission check for opening files.
It isn't only exec that can change credentials. Do we need a lock for changing credentials?
[...]
Quoted 4 lines
Quoted 3 lines
If we need a lock around credential change let's design and build that. Having a mismatch between what a lock is designed to do, and what people use it for can only result in other bugs as people get confused.
Quoted 8 lines
Quoted 3 lines
Hmm... what benefits do we get from making it a separate lock? I guess it would allow us to make it a per-task lock instead of a signal_struct-wide one? That might be helpful...
But actually, isn't the core purpose of the
cred_guard_mutexto guard against concurrent credential changes anyway? That's what almost everyone uses it for, and it's in the name...
The main reason d'etre of exec_update_mutex is to get a consitent view of task->mm and task credentials.
The reason why you want the cred_guard_mutex, is that some action is changing the resulting credentials that the execve is about to install, and that is the data flow in the opposite direction.
Bernd.
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
Quoted 39 lines · Bernd Edlinger
Quoted 29 lines · Jann Horn
Quoted 11 lines · Jann Horn
Quoted 10 lines · Eric W. Biederman
Quoted 9 lines · Jann Horn
Quoted 8 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.[...]
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
If you make this change, then if this races with execution of a setuid program that afterwards e.g. opens a unix domain socket, an attacker will be able to steal that socket and inject messages into communication with things like DBus. procfs currently has the same race, and that still needs to be fixed, but at least procfs doesn't let you open things like sockets because they don't have a working ->open handler, and it enforces the normal permission check for opening files.
It isn't only exec that can change credentials. Do we need a lock for changing credentials?
[...]
Quoted 4 lines
Quoted 3 lines
If we need a lock around credential change let's design and build that. Having a mismatch between what a lock is designed to do, and what people use it for can only result in other bugs as people get confused.
Quoted 8 lines
Quoted 3 lines
Hmm... what benefits do we get from making it a separate lock? I guess it would allow us to make it a per-task lock instead of a signal_struct-wide one? That might be helpful...
But actually, isn't the core purpose of the
cred_guard_mutexto guard against concurrent credential changes anyway? That's what almost everyone uses it for, and it's in the name...
Quoted 6 lines
The main reason d'etre of
exec_update_mutexis to get a consitent view oftask->mmand task credentials.Quoted 1 line
The reason why you want the
cred_guard_mutex, is that some actionis changing the resulting credentials that the execve is about to install, and that is the data flow in the opposite direction.
So in other words, you need the exec_update_mutex when you access another thread's credentials and possibly the mmap at the same time.
You need the cred_guard_mutex when you *change* the credentials of another thread. (Where you cannot be sure that the other thread just started to execve something)
You need no mutex at all when you are just accessing or even changing the credentials of the current thread. (If another thread is doing execve, your task will be killed, and wether or not the credentials were changed does not matter any more)
Quoted 3 lines
Bernd.
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
Quoted 42 lines · Bernd Edlinger
Quoted 41 lines · Bernd Edlinger
Quoted 29 lines · Jann Horn
Quoted 11 lines · Jann Horn
Quoted 10 lines · Eric W. Biederman
Quoted 9 lines · Jann Horn
Quoted 8 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.[...]
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
If you make this change, then if this races with execution of a setuid program that afterwards e.g. opens a unix domain socket, an attacker will be able to steal that socket and inject messages into communication with things like DBus. procfs currently has the same race, and that still needs to be fixed, but at least procfs doesn't let you open things like sockets because they don't have a working ->open handler, and it enforces the normal permission check for opening files.
It isn't only exec that can change credentials. Do we need a lock for changing credentials?
[...]
Quoted 4 lines
Quoted 3 lines
If we need a lock around credential change let's design and build that. Having a mismatch between what a lock is designed to do, and what people use it for can only result in other bugs as people get confused.
Quoted 8 lines
Quoted 3 lines
Hmm... what benefits do we get from making it a separate lock? I guess it would allow us to make it a per-task lock instead of a signal_struct-wide one? That might be helpful...
But actually, isn't the core purpose of the
cred_guard_mutexto guard against concurrent credential changes anyway? That's what almost everyone uses it for, and it's in the name...The main reason d'etre of
exec_update_mutexis to get a consitent view oftask->mmand task credentials.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
The reason why you want the
cred_guard_mutex, is that some actionis changing the resulting credentials that the execve is about to install, and that is the data flow in the opposite direction.
So in other words, you need the
exec_update_mutexwhen you access another thread's credentials and possibly the mmap at the same time.
Or the file descriptor table, or register state, ...
Quoted 4 lines
You need no mutex at all when you are just accessing or even changing the credentials of the current thread. (If another thread is doing execve, your task will be killed, and wether or not the credentials were changed does not matter any more)
Only if the only access checks you care about are those related to mm access.
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
In reply to Jann HornQuoted 29 lines · Jann Horn
Quoted 11 lines · Jann Horn
Quoted 10 lines · Eric W. Biederman
Quoted 9 lines · Jann Horn
Quoted 8 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.[...]
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
If you make this change, then if this races with execution of a setuid program that afterwards e.g. opens a unix domain socket, an attacker will be able to steal that socket and inject messages into communication with things like DBus. procfs currently has the same race, and that still needs to be fixed, but at least procfs doesn't let you open things like sockets because they don't have a working ->open handler, and it enforces the normal permission check for opening files.
It isn't only exec that can change credentials. Do we need a lock for changing credentials?
[...]
Quoted 4 lines
Quoted 3 lines
If we need a lock around credential change let's design and build that. Having a mismatch between what a lock is designed to do, and what people use it for can only result in other bugs as people get confused.
Quoted 7 lines
Quoted 3 lines
Hmm... what benefits do we get from making it a separate lock? I guess it would allow us to make it a per-task lock instead of a signal_struct-wide one? That might be helpful...
But actually, isn't the core purpose of the
cred_guard_mutexto guard against concurrent credential changes anyway? That's what almost everyone uses it for, and it's in the name...
Having been through all of the users nope.
Maybe someone tried to repurpose for that. I haven't traced through when it went the it was renamed from cred_exec_mutex to cred_guard_mutex.
The original purpose was to make make exec and ptrace deadlock. But it was seen as being there to allow safely calculating the new credentials before the point of now return. Because if a process is ptraced or not affects the new credential calculations. Unfortunately offering that guarantee fundamentally leads to deadlock.
So ptrace_attach and seccomp use the cred_guard_mutex to guarantee a deadlock.
The common use is to take cred_guard_mutex to guard the window when credentials and process details are out of sync in exec. But there is at least do_io_accounting that seems to have the same justification for holding __pidfd_fget.
With effort I suspect we can replace exec_change_mutex with task_lock. When we are guaranteed to be single threaded placing exec_change_mutex in signal_struct doesn't really help us (except maybe in some races?).
The deep problem is no one really understands cred_guard_mutex so it is a mess. Code with poorly defined semantics is always wrong somewhere for someone. Which is part of why I am attacking this and having the conversations to make certain I understand what is going on.
I see your point about commit_creds making a process undumpable. So in practice it really is only exec that changes creds in a way that ptrace_may_access will allow the process to be inspected.
So I guess for now the practical non-regressing course is to change everything to my exec_change_mutex, removing the deadlock. Then we figure out how to cleanly deal with the races inspecting a process with changing credentials has.
Eric
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
Quoted 59 lines · Eric W. Biederman
Quoted 29 lines · Jann Horn
Quoted 11 lines · Jann Horn
Quoted 10 lines · Eric W. Biederman
Quoted 9 lines · Jann Horn
Quoted 8 lines · Eric W. Biederman
During exec some file descriptors are closed and the files struct is unshared. But all of that can happen at other times and it has the same protections during exec as at ordinary times. So stop taking the
cred_guard_mutexas it is useless.Furthermore he
cred_guard_mutexis a bad idea because it is deadlock prone, as it is held in serveral while waiting possibly indefinitely for userspace to do something.[...]
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
If you make this change, then if this races with execution of a setuid program that afterwards e.g. opens a unix domain socket, an attacker will be able to steal that socket and inject messages into communication with things like DBus. procfs currently has the same race, and that still needs to be fixed, but at least procfs doesn't let you open things like sockets because they don't have a working ->open handler, and it enforces the normal permission check for opening files.
It isn't only exec that can change credentials. Do we need a lock for changing credentials?
[...]
Quoted 4 lines
Quoted 3 lines
If we need a lock around credential change let's design and build that. Having a mismatch between what a lock is designed to do, and what people use it for can only result in other bugs as people get confused.
Quoted 7 lines
Quoted 3 lines
Hmm... what benefits do we get from making it a separate lock? I guess it would allow us to make it a per-task lock instead of a signal_struct-wide one? That might be helpful...
But actually, isn't the core purpose of the
cred_guard_mutexto guard against concurrent credential changes anyway? That's what almost everyone uses it for, and it's in the name...Having been through all of the users nope.
Maybe someone tried to repurpose for that. I haven't traced through when it went the it was renamed from
cred_exec_mutextocred_guard_mutex.The original purpose was to make make exec and ptrace deadlock. But it was seen as being there to allow safely calculating the new credentials before the point of now return. Because if a process is ptraced or not affects the new credential calculations. Unfortunately offering that guarantee fundamentally leads to deadlock.
So
ptrace_attachand seccomp use thecred_guard_mutexto guarantee a deadlock.The common use is to take
cred_guard_mutexto guard the window when credentials and process details are out of sync in exec. But there is at leastdo_io_accountingthat seems to have the same justification for holding__pidfd_fget.
Quoted 6 lines
With effort I suspect we can replace
exec_change_mutexwithtask_lock. When we are guaranteed to be single threaded placingexec_change_mutexinsignal_structdoesn't really help us (except maybe in some races?).The deep problem is no one really understands
cred_guard_mutexso it is a mess. Code with poorly defined semantics is always wrong somewhere
This is a good point. When discussing patches sensitive to credential changes cred_guard_mutex was always introduced as having the purpose to guard against concurrent credential changes. And I'm pretty sure that that's how most people have been using it for quite a long time. I mean, it's at least the case for seccomp and proc and probably quite a few more. So the problem seems to me that it has clear _intended_ semantics that runs into issues in all sorts of cases. So if cred_guard_mutex is not that then we seem to need to provide something that serves it's intended purpose.
Re: [PATCH] pidfd: Stop taking cred_guard_mutex
In reply to Eric W. BiedermanQuoted 2 lines · Eric W. Biederman
So
ptrace_attachand seccomp use thecred_guard_mutexto guarantee a deadlock.
Well, that's the result, but seccomp uses it because it wants to be certain that credentials and no_new_privs are changed together "atomically".
Signature · 1 line
Kees Cook
PATCH 1/1 pidfd: Use new infrastructure to fix deadlocks in execve
This changes __pidfd_fget to use the new exec_update_mutex instead of cred_guard_mutex.
This should be safe, as the credentials do not change before exec_update_mutex is locked. Therefore whatever file access is possible with holding the cred_guard_mutex here is also possbile with the exec_update_mutex.
--- kernel/pid.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
This replaces Eric's "[PATCH] pidfd: Stop taking cred_guard_mutex"
kernel/pid.c +2 -2
@@ -584,7 +584,7 @@ static struct file *__pidfd_fget(struct task_struct *task, int fd)struct file *file;int ret; - ret =mutex_lock_killable (&task->signal->cred_guard_mutex); + ret =mutex_lock_killable (&task->signal->exec_update_mutex);if (ret)return ERR_PTR (ret);
@@ -593,7 +593,7 @@ static struct file *__pidfd_fget(struct task_struct *task, int fd)else file =ERR_PTR (-EPERM ); -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return file ?:ERR_PTR (-EBADF ); }
Signature · 1 line
1.9.1
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
In reply to Eric W. BiedermanQuoted 2 lines · Eric W. Biederman
And I completely agree that we should at least rename tsk to me. Just for clarity.
I think it wouldn't hurt to add comments to spell it out explicitly in each of the tsk->me functions, something like:
/*
* The "me" task_struct argument here must only ever refer to "current",
* but it gets passed in to avoid re-calculating "current" in each helper. */
I've found that the exec code in its entirety would be better off with more comments. :) Usually that's the bulk of what I find myself adding when I make changes in this area. ;)
-Kees
Signature · 1 line
Kees Cook
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
In reply to Eric W. BiedermanQuoted 27 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)fs/exec.c +0
-5@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)/* we have changed execution domain */ tsk->exit_signal =SIGCHLD ; -#ifdef CONFIG_POSIX_TIMERS -exit_itimers (sig); -flush_itimer_signals (); -#endif -BUG_ON (!thread_group_leader (tsk));return 0 ;
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1279
Quoted 8 lines
fs/exec.c +5
-0@@ -1277,6 +1272,11 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; +#ifdef CONFIG_POSIX_TIMERS +exit_itimers (me->signal); +flush_itimer_signals (); +#endif +
I twitch at seeing #ifdefs in .c instead of hidden in the .h declarations of these two functions, but as this is a copy/paste, I'll live. ;)
-Kees
Quoted 6 lines
fs/exec.c +0
-0/* * Make the signal table private. */ Signature · 1 line
2.25.0
Signature · 1 line
Kees Cook
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
In reply to Eric W. BiedermanQuoted 26 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)fs/exec.c +0
-5@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)/* we have changed execution domain */ tsk->exit_signal =SIGCHLD ; -#ifdef CONFIG_POSIX_TIMERS -exit_itimers (sig); -flush_itimer_signals (); -#endif -BUG_ON (!thread_group_leader (tsk));return 0 ;
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1278
Quoted 7 lines
fs/exec.c +4
-0@@ -1277,6 +1272,11 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; +#ifdef CONFIG_POSIX_TIMERS +exit_itimers (me->signal); +flush_itimer_signals (); +#endif
nit: exit_itimers() has a comment referring to de_thread, that should probably be updated
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
Quoted 30 lines · Jann Horn
Quoted 29 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)fs/exec.c +0
-5@@ -1189,11 +1189,6 @@ static int de_thread(struct task_struct *tsk)/* we have changed execution domain */ tsk->exit_signal =SIGCHLD ; -#ifdef CONFIG_POSIX_TIMERS -exit_itimers (sig); -flush_itimer_signals (); -#endif -BUG_ON (!thread_group_leader (tsk));return 0 ;@@ -1277,6 +1272,11 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out;
Quoted 7 lines
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1278
Quoted 4 lines
fs/exec.c +4
-0+#ifdef CONFIG_POSIX_TIMERS +exit_itimers (me->signal); +flush_itimer_signals (); +#endif nit:
exit_itimers()has a comment referring tode_thread, that should probably be updated
Good point.
Eric
Re: [PATCH v2 3/5] exec: Move cleanup of posix timers on exec out of de_thread
In reply to Eric W. BiedermanQuoted 6 lines · Eric W. Biederman
These functions have very little to do with
de_threadmove them out ofde_threadan intoflush_old_execproper so it can be more clearly seen whatflush_old_execis doing.Signed-off-by "Eric W. Biederman" <[email protected]>
PATCH v2 4/5 exec: Move exec_mmap right after de_thread in flush_old_exec
In reply to Eric W. BiedermanI have read through the code in exec_mmap and I do not see anything that depends on sighand or the sighand lock, or on signals in anyway so this should be safe.
This rearrangement of code has two siginficant benefits. It makes the determination of passing the point of no return by testing bprm->mm accurate. All failures prior to that point in flush_old_exec are either truly recoverable or they are fatal.
Futher this consolidates all of the possible indefinite waits for userspace together at the top of flush_old_exec. The possible wait for a ptracer on PTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved in clear_child_tid, and the possible wait for a page fault in exit_robust_list.
This consolidation allows the creation of a mutex to replace cred_guard_mutex that is not held of possible indefinite userspace waits. Which will allow removing deadlock scenarios from the kernel.
--- fs/exec.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
fs/exec.c +12 -12
@@ -1272,18 +1272,6 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; -#ifdef CONFIG_POSIX_TIMERS -exit_itimers (me->signal); -flush_itimer_signals (); -#endif - -/* -* Make the signal table private. -*/ - retval =unshare_sighand (me); -if (retval) - goto out; -/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update
@@ -1307,6 +1295,18 @@ int flush_old_exec(struct linux_binprm * bprm)*/ bprm->mm =NULL ; +#ifdef CONFIG_POSIX_TIMERS +exit_itimers (me->signal); +flush_itimer_signals (); +#endif + +/* +* Make the signal table private. +*/ + retval =unshare_sighand (me); +if (retval) + goto out; +set_fs (USER_DS ); me->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD |PF_NOFREEZE |PF_NO_SETAFFINITY );
Signature · 1 line
2.25.0
Re: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
Quoted 6 lines · Eric W. Biederman
I have read through the code in
exec_mmapand I do not see anything that depends on sighand or the sighand lock, or on signals in anyway so this should be safe.This rearrangement of code has two siginficant benefits. It makes
^ typo: significant
Quoted 6 lines
the determination of passing the point of no return by testing
bprm->mmaccurate. All failures prior to that point inflush_old_execare either truly recoverable or they are fatal.Futher this consolidates all of the possible indefinite waits for ^ typo: Further
Quoted 7 lines
userspace together at the top of
flush_old_exec. The possible wait for a ptracer onPTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved inclear_child_tid, and the possible wait for a page fault inexit_robust_list.This consolidation allows the creation of a mutex to replace
cred_guard_mutexthat is not held of possible indefinite userspace
can you also reword this "held of" thing here as well?
Thanks
Bernd.
Eric W. Biederman
Reviewed-by Bernd EdlingerRe: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
Quoted 6 lines · Bernd Edlinger
Quoted 3 lines · Eric W. Biederman
This consolidation allows the creation of a mutex to replace
cred_guard_mutexthat is not held of possible indefinite userspacecan you also reword this "held of" thing here as well?
Done:
exec: Move exec_mmap right after de_thread in flush_old_exec
I have read through the code in exec_mmap and I do not see anything that depends on sighand or the sighand lock, or on signals in anyway so this should be safe.
This rearrangement of code has two siginficant benefits. It makes the determination of passing the point of no return by testing bprm->mm accurate. All failures prior to that point in flush_old_exec are either truly recoverable or they are fatal.
Futher this consolidates all of the possible indefinite waits for userspace together at the top of flush_old_exec. The possible wait for a ptracer on PTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved in clear_child_tid, and the possible wait for a page fault in exit_robust_list.
This consolidation allows the creation of a mutex to replace cred_guard_mutex that is not held over possible indefinite userspace waits. Which will allow removing deadlock scenarios from the kernel.
Eric
Re: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
Quoted 11 lines · Eric W. Biederman
Quoted 6 lines · Bernd Edlinger
Quoted 3 lines · Eric W. Biederman
This consolidation allows the creation of a mutex to replace
cred_guard_mutexthat is not held of possible indefinite userspacecan you also reword this "held of" thing here as well?
Done:
Quoted 7 lines
exec: Move
exec_mmapright afterde_threadinflush_old_execI have read through the code in
exec_mmapand I do not see anything that depends on sighand or the sighand lock, or on signals in anyway so this should be safe.This rearrangement of code has two siginficant benefits. It makes
watch out: sig_i_nificant
Quoted 5 lines
the determination of passing the point of no return by testing
bprm->mmaccurate. All failures prior to that point inflush_old_execare either truly recoverable or they are fatal.Futher this consolidates all of the possible indefinite waits for
Add some r to "Futher", please?
Quoted 14 lines
userspace together at the top of
flush_old_exec. The possible wait for a ptracer onPTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved inclear_child_tid, and the possible wait for a page fault inexit_robust_list.This consolidation allows the creation of a mutex to replace
cred_guard_mutexthat is not held over possible indefinite userspace waits. Which will allow removing deadlock scenarios from the kernel.Reviewed-by Bernd Edlinger <[email protected]>Signed-off-by "Eric W. Biederman" <[email protected]>Eric
Eric W. Biederman
Reviewed-by Bernd EdlingerRe: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
Ok. I think this has it sorted:
exec: Move exec_mmap right after de_thread in flush_old_exec
I have read through the code in exec_mmap and I do not see anything that depends on sighand or the sighand lock, or on signals in anyway so this should be safe.
This rearrangement of code has two significant benefits. It makes the determination of passing the point of no return by testing bprm->mm accurate. All failures prior to that point in flush_old_exec are either truly recoverable or they are fatal.
Further this consolidates all of the possible indefinite waits for userspace together at the top of flush_old_exec. The possible wait for a ptracer on PTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved in clear_child_tid, and the possible wait for a page fault in exit_robust_list.
This consolidation allows the creation of a mutex to replace cred_guard_mutex that is not held over possible indefinite userspace waits. Which will allow removing deadlock scenarios from the kernel.
I don't think I usually have this many typos. Sigh.
Eric
Re: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
Quoted 24 lines · Eric W. Biederman
Ok. I think this has it sorted:
exec: Move
exec_mmapright afterde_threadinflush_old_execI have read through the code in
exec_mmapand I do not see anything that depends on sighand or the sighand lock, or on signals in anyway so this should be safe.This rearrangement of code has two significant benefits. It makes the determination of passing the point of no return by testing
bprm->mmaccurate. All failures prior to that point inflush_old_execare either truly recoverable or they are fatal.Further this consolidates all of the possible indefinite waits for userspace together at the top of
flush_old_exec. The possible wait for a ptracer onPTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved inclear_child_tid, and the possible wait for a page fault inexit_robust_list.This consolidation allows the creation of a mutex to replace
cred_guard_mutexthat is not held over possible indefinite userspace waits. Which will allow removing deadlock scenarios from the kernel.
Quoted 6 lines
Reviewed-by Bernd Edlinger <[email protected]>Signed-off-by "Eric W. Biederman" <[email protected]>I don't think I usually have this many typos. Sigh.
OK.
never mind,
Bernd.
Re: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
Quoted 29 lines · Bernd Edlinger
Quoted 28 lines · Eric W. Biederman
Ok. I think this has it sorted:
exec: Move
exec_mmapright afterde_threadinflush_old_execI have read through the code in
exec_mmapand I do not see anything that depends on sighand or the sighand lock, or on signals in anyway so this should be safe.This rearrangement of code has two significant benefits. It makes the determination of passing the point of no return by testing
bprm->mmaccurate. All failures prior to that point inflush_old_execare either truly recoverable or they are fatal.Further this consolidates all of the possible indefinite waits for userspace together at the top of
flush_old_exec. The possible wait for a ptracer onPTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved inclear_child_tid, and the possible wait for a page fault inexit_robust_list.This consolidation allows the creation of a mutex to replace
cred_guard_mutexthat is not held over possible indefinite userspace waits. Which will allow removing deadlock scenarios from the kernel.Reviewed-by Bernd Edlinger <[email protected]>Signed-off-by "Eric W. Biederman" <[email protected]>
Quoted 6 lines
Quoted 2 lines
I don't think I usually have this many typos. Sigh.
OK.
never mind,
No no. I really appreciate all of the scrutiny. Frequently the issues that will produce typos or poor patch descriptions are also the issues that will produce sloppy patches as well. I was just frustrated with myself.
Eric
Re: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
In reply to Eric W. BiedermanQuoted 9 lines · Eric W. Biederman
I have read through the code in
exec_mmapand I do not see anything that depends on sighand or the sighand lock, or on signals in anyway so this should be safe.This rearrangement of code has two siginficant benefits. It makes the determination of passing the point of no return by testing
bprm->mmaccurate. All failures prior to that point inflush_old_execare either truly recoverable or they are fatal.
Agreed. Though I see a use of "current", which maybe you want to parameterize to a "me" argument in acct_arg_size(). (Though looking at the callers, perhaps there is no benefit?)
Quoted 21 lines
Futher this consolidates all of the possible indefinite waits for userspace together at the top of
flush_old_exec. The possible wait for a ptracer onPTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved inclear_child_tid, and the possible wait for a page fault inexit_robust_list.This consolidation allows the creation of a mutex to replace
cred_guard_mutexthat is not held of possible indefinite userspace waits. Which will allow removing deadlock scenarios from the kernel.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)fs/exec.c +0
-0
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1278
Quoted 7 lines
fs/exec.c +0
-4@@ -1272,18 +1272,6 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; -#ifdef CONFIG_POSIX_TIMERS -exit_itimers (me->signal); -flush_itimer_signals (); -#endif
I think this comment:
/*
* This is called by do_exit or de_thread, only when there are no more
* references to the shared signal_struct.
*/
void exit_itimers(struct signal_struct *sig)
Refers to there being other threads, yes? Not that the signal table is private yet?
Quoted 12 lines
fs/exec.c +0
-8- -/* -* Make the signal table private. -*/ - retval =unshare_sighand (me); -if (retval) - goto out; -/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1301
Quoted 7 lines
fs/exec.c +4
-0@@ -1307,6 +1295,18 @@ int flush_old_exec(struct linux_binprm * bprm)*/ bprm->mm =NULL ; +#ifdef CONFIG_POSIX_TIMERS +exit_itimers (me->signal); +flush_itimer_signals (); +#endif
I've mostly convinced myself that there are no "side-effects" from having these timers expire as the mm is going away. I think some kind of comment of that intent should be explicitly stated here above the timer work.
Beyond that:
-Kees
Quoted 14 lines
fs/exec.c +8
-0+ +/* +* Make the signal table private. +*/ + retval =unshare_sighand (me); +if (retval) + goto out; +set_fs (USER_DS ); me->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD |PF_NOFREEZE |PF_NO_SETAFFINITY );Signature · 1 line
2.25.0
Signature · 1 line
Kees Cook
Re: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
Quoted 6 lines · Kees Cook
Quoted 5 lines · Eric W. Biederman
I have read through the code in
exec_mmapand I do not see anything that depends on sighand or the sighand lock, or on signals in anyway so this should be safe.
Quoted 8 lines
Quoted 4 lines
This rearrangement of code has two siginficant benefits. It makes the determination of passing the point of no return by testing
bprm->mmaccurate. All failures prior to that point inflush_old_execare either truly recoverable or they are fatal.Agreed. Though I see a use of "current", which maybe you want to parameterize to a "me" argument in
acct_arg_size(). (Though looking at the callers, perhaps there is no benefit?)
My testing suggests there is a small benefit on x86.
The code is just "#define current get_current()" and get_current() revoles into a read of "%gs:current_task".
But looking at the code I find gcc can sometimes when the reads are close in the source code can optimize the read away. But gcc does not manage to optimize the extra read of "%gs:current_task" away.
So I think things are much much better than they used to be, code generation wise. But it still helps to cache current in a local variable.
Quoted 30 lines
Quoted 20 lines
Futher this consolidates all of the possible indefinite waits for userspace together at the top of
flush_old_exec. The possible wait for a ptracer onPTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved inclear_child_tid, and the possible wait for a page fault inexit_robust_list.This consolidation allows the creation of a mutex to replace
cred_guard_mutexthat is not held of possible indefinite userspace waits. Which will allow removing deadlock scenarios from the kernel.Signed-off-by "Eric W. Biederman" <[email protected]>--- fs/exec.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)fs/exec.c +0
-0fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1278
Quoted 7 lines
fs/exec.c +0
-4@@ -1272,18 +1272,6 @@ int flush_old_exec(struct linux_binprm * bprm)if (retval) goto out; -#ifdef CONFIG_POSIX_TIMERS -exit_itimers (me->signal); -flush_itimer_signals (); -#endif I think this comment:
Quoted 8 lines
/*
* This is called bydo_exitorde_thread, only when there are no more
* references to the sharedsignal_struct.
*/
voidexit_itimers(struct signal_struct *sig)Refers to there being other threads, yes? Not that the signal table is private yet?
The signal table is in sighand_struct.
So yes that refers to the other threads being gone.
Quoted 15 lines
Quoted 15 lines
fs/exec.c +0
-8- -/* -* Make the signal table private. -*/ - retval =unshare_sighand (me); -if (retval) - goto out; -/* * Must be called _before_ exec_mmap() as bprm->mm is * not visibile until then. This also enables the update @@ -1307,6 +1295,18 @@ int flush_old_exec(struct linux_binprm * bprm)*/ bprm->mm =NULL ;
Quoted 8 lines
fs/exec.c int flush_old_exec(struct linux_binprm * bprm) · line 1301
Quoted 4 lines
fs/exec.c +4
-0+#ifdef CONFIG_POSIX_TIMERS +exit_itimers (me->signal); +flush_itimer_signals (); +#endif I've mostly convinced myself that there are no "side-effects" from having these timers expire as the mm is going away. I think some kind of comment of that intent should be explicitly stated here above the timer work.
The timers can at most generate signals. And we are not handling signals in the middle of exec.
So the only possible interaction would be to set a timeout and then try exec, and have the timer kill the caller.
Maybe we get a killable signal from a scenario like that and maybe this changes the time before the timer expires into the dangerous zone. But that is all I can think of.
We have to return to the edge of userspace before any signals are delivered.
Quoted 6 lines
Beyond that:
Reviewed-by Kees Cook <[email protected]>-Kees
Quoted 14 lines
Quoted 14 lines
fs/exec.c +8
-0+ +/* +* Make the signal table private. +*/ + retval =unshare_sighand (me); +if (retval) + goto out; +set_fs (USER_DS ); me->flags &= ~(PF_RANDOMIZE |PF_FORKNOEXEC |PF_KTHREAD |PF_NOFREEZE |PF_NO_SETAFFINITY );Signature · 1 line
2.25.0
Eric
Re: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
In reply to Eric W. BiedermanQuoted 5 lines · Eric W. Biederman
Futher this consolidates all of the possible indefinite waits for userspace together at the top of
flush_old_exec. The possible wait for a ptracer onPTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved inclear_child_tid, and the possible wait for a page fault inexit_robust_list.
I forgot to mention, just as a point of clarity, there are lots of other page faults possible, but they're _before_ flush_old_exec() (i.e. all the copy_strings() calls). Is it worth clarifying this to "before or at the top of flush_old_exec()" or do you mean something else? (And as always: perhaps expand flush_old_exec()'s comment to describe the newly intended state.)
Signature · 1 line
Kees Cook
Re: [PATCH v2 4/5] exec: Move exec_mmap right after de_thread in flush_old_exec
Quoted 7 lines · Kees Cook
Quoted 5 lines · Eric W. Biederman
Futher this consolidates all of the possible indefinite waits for userspace together at the top of
flush_old_exec. The possible wait for a ptracer onPTRACE_EVENT_EXIT, the possible wait for a page fault to be resolved inclear_child_tid, and the possible wait for a page fault inexit_robust_list.
Quoted 6 lines
I forgot to mention, just as a point of clarity, there are lots of other page faults possible, but they're _before_
flush_old_exec()(i.e. all thecopy_strings()calls). Is it worth clarifying this to "before or at the top offlush_old_exec()" or do you mean something else? (And as always: perhaps expand flush_old_exec()'s comment to describe the newly intended state.)
Yes. Before or at the start of flush_old_exec where the mutex is taken. That is the point. I will see if I can come up with and appropriate comment.
Eric
PATCH v2 5/5 exec: Add a exec_update_mutex to replace cred_guard_mutex
In reply to Eric W. BiedermanThe cred_guard_mutex is problematic. The cred_guard_mutex is held over the userspace accesses as the arguments from userspace are read. The cred_guard_mutex is held of PTRACE_EVENT_EXIT as the the other threads are killed. The cred_guard_mutex is held over "put_user(0, tsk->clear_child_tid)" in exit_mm().
Any of those can result in deadlock, as the cred_guard_mutex is held over a possible indefinite userspace waits for userspace.
Add exec_update_mutex that is only held over exec updating process with the new contents of exec, so that code that needs not to be confused by exec changing the mm and the cred in ways that can not happen during ordinary execution of a process.
The plan is to switch the users of cred_guard_mutex to exec_udpate_mutex one by one. This lets us move forward while still being careful and not introducing any regressions.
Ref: 45c1a159b85b ("Add PTRACE_O_TRACEVFORKDONE and PTRACE_O_TRACEEXIT facilities.")
Ref: 456f17cd1a28 ("[PATCH] user-vm-unlock-2.5.31-A2")
--- fs/exec.c | 9 +++++++++ include/linux/sched/signal.h | 9 ++++++++- init/init_task.c | 1 + kernel/fork.c | 1 + 4 files changed, 19 insertions(+), 1 deletion(-)
fs/exec.c +9 -0
@@ -1014,6 +1014,7 @@ static int exec_mmap(struct mm_struct *mm) {struct task_struct *tsk;struct mm_struct *old_mm, *active_mm; +int ret;/* Notify parent that we're no longer interested in the old VM */ tsk = current;
@@ -1034,6 +1035,11 @@ static int exec_mmap(struct mm_struct *mm)return -EINTR ; } } + + ret =mutex_lock_killable (&tsk->signal->exec_update_mutex); +if (ret) +return ret; +task_lock (tsk); active_mm = tsk->active_mm;membarrier_exec_mmap (mm);
@@ -1438,6 +1444,8 @@ static void free_bprm(struct linux_binprm *bprm) {free_arg_pages (bprm);if (bprm->cred) { +if (!bprm->mm) +mutex_unlock (¤t->signal->exec_update_mutex);mutex_unlock (¤t->signal->cred_guard_mutex);abort_creds (bprm->cred); }
@@ -1487,6 +1495,7 @@ void install_exec_creds(struct linux_binprm *bprm)* credentials; any time after this it may be unlocked. */ security_bprm_committed_creds (bprm); +mutex_unlock (¤t->signal->exec_update_mutex);mutex_unlock (¤t->signal->cred_guard_mutex); }EXPORT_SYMBOL (install_exec_creds);
include/linux/sched/signal.h +8 -1
@@ -224,7 +224,14 @@ struct signal_struct {struct mutex cred_guard_mutex;/* guard against foreign influences on * credential calculations -* (notably. ptrace) */ +* (notably. ptrace) +* Deprecated do not use in new code. +* Use exec_update_mutex instead. +*/ +struct mutex exec_update_mutex;/* Held while task_struct is being +* updated during exec, and may have +* inconsistent permissions. +*/ } __randomize_layout;/*
init/init_task.c +1 -0
@@ -26,6 +26,7 @@ static struct signal_struct init_signals = { .multiprocess = HLIST_HEAD_INIT, .rlim = INIT_RLIMITS, .cred_guard_mutex = __MUTEX_INITIALIZER(init_signals.cred_guard_mutex), + .exec_update_mutex =__MUTEX_INITIALIZER (init_signals.exec_update_mutex), #ifdef CONFIG_POSIX_TIMERS .posix_timers =LIST_HEAD_INIT (init_signals.posix_timers), .cputimer = {
kernel/fork.c +1 -0
@@ -1594,6 +1594,7 @@ static int copy_signal(unsigned long clone_flags, struct task_struct *tsk) sig->oom_score_adj_min = current->signal->oom_score_adj_min;mutex_init (&sig->cred_guard_mutex); +mutex_init (&sig->exec_update_mutex);return 0 ; }
Signature · 1 line
2.25.0
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 4 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other
... is held while waiting for the trace parent to handle PTRACE_EVENT_EXIT or something?
I wonder if we also should mention that
it is held while waiting for the trace parent to
receive the exit code with "wait"?
Quoted 7 lines
threads are killed. The
cred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Any of those can result in deadlock, as the
cred_guard_mutexis held over a possible indefinite userspace waits for userspace.Add
exec_update_mutexthat is only held over exec updating process
Add ?
Quoted 6 lines
with the new contents of exec, so that code that needs not to be confused by exec changing the mm and the cred in ways that can not happen during ordinary execution of a process.
The plan is to switch the users of
cred_guard_mutextoexec_udpate_mutexone by one. This lets us move forward while still
s/udpate/update/
Bernd.
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 5 lines · Bernd Edlinger
Quoted 4 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other
^ over
Quoted 3 lines
... is held while waiting for the trace parent to handle
PTRACE_EVENT_EXITor something?
Yes. Let me see if I can phrase that better.
Quoted 3 lines
I wonder if we also should mention that
it is held while waiting for the trace parent to
receive the exit code with "wait"?
I don't think we have to spell out the details of how it all works, unless that makes things clearer. Kernel developers can be expected to figure out how the kernel works. The critical thing is that it is an indefinite wait for userspace to take action.
But I will look.
Quoted 9 lines
Quoted 7 lines
threads are killed. The
cred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Any of those can result in deadlock, as the
cred_guard_mutexis held over a possible indefinite userspace waits for userspace.Add
exec_update_mutexthat is only held over exec updating processAdd ?
Yes. That is what the change does: add exec_update_mutex.
Quoted 8 lines
Quoted 6 lines
with the new contents of exec, so that code that needs not to be confused by exec changing the mm and the cred in ways that can not happen during ordinary execution of a process.
The plan is to switch the users of
cred_guard_mutextoexec_udpate_mutexone by one. This lets us move forward while stills/udpate/update/
Yes. Very much so.
Eric
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 28 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
Quoted 4 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other^ over
Quoted 3 lines
... is held while waiting for the trace parent to handle
PTRACE_EVENT_EXITor something?Yes. Let me see if I can phrase that better.
Quoted 3 lines
I wonder if we also should mention that
it is held while waiting for the trace parent to
receive the exit code with "wait"?I don't think we have to spell out the details of how it all works, unless that makes things clearer. Kernel developers can be expected to figure out how the kernel works. The critical thing is that it is an indefinite wait for userspace to take action.
But I will look.
Quoted 3 lines
Quoted 3 lines
threads are killed. The
cred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Any of those can result in deadlock, as the
cred_guard_mutexis held over a possible indefinite userspace waits for userspace.Add
exec_update_mutexthat is only held over exec updating processAdd ?
Yes. That is what the change does: add
exec_update_mutex.
I just kind of missed the "subject" in this sentence, like "This patch adds an exec_update_mutex that is ..." but english is a foreign language for me, so may be okay as is.
Bernd.
Quoted 13 lines
Quoted 8 lines
Quoted 6 lines
with the new contents of exec, so that code that needs not to be confused by exec changing the mm and the cred in ways that can not happen during ordinary execution of a process.
The plan is to switch the users of
cred_guard_mutextoexec_udpate_mutexone by one. This lets us move forward while stills/udpate/update/
Yes. Very much so.
Eric
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 34 lines · Bernd Edlinger
Quoted 33 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
Quoted 4 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other^ over
Quoted 3 lines
... is held while waiting for the trace parent to handle
PTRACE_EVENT_EXITor something?Yes. Let me see if I can phrase that better.
Quoted 3 lines
I wonder if we also should mention that
it is held while waiting for the trace parent to
receive the exit code with "wait"?I don't think we have to spell out the details of how it all works, unless that makes things clearer. Kernel developers can be expected to figure out how the kernel works. The critical thing is that it is an indefinite wait for userspace to take action.
But I will look.
Quoted 8 lines
Quoted 7 lines
threads are killed. The
cred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Any of those can result in deadlock, as the
cred_guard_mutexis held over a possible indefinite userspace waits for userspace.Add
exec_update_mutexthat is only held over exec updating process
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Add ?
Yes. That is what the change does: add
exec_update_mutex.I just kind of missed the "subject" in this sentence, like "This patch adds an
exec_update_mutexthat is ..." but english is a foreign language for me, so may be okay as is.
English has a lot of options. I think this is a stylistic difference.
Instead of being an observer and describing what the change does:
"This patch adds exec_update_mutex ..."
I was being there in the moment and saying/commading what is happening:
"Add exec_update_mutex ..."
Using the more immdediate form ends up with more concise and clearer sentences.
Every one of my writing teachers in school emphasized that point and I see the who it works when I write things. But writing is hard and I still tend toward long rambling sentences with many qualifiers that confuse and detract from the point rather than make it clear what is happening.
Eric
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 53 lines · [email protected] (Eric W. Biederman) writes:
Quoted 34 lines · Bernd Edlinger
Quoted 33 lines · Eric W. Biederman
Quoted 5 lines · Bernd Edlinger
Quoted 4 lines · Eric W. Biederman
The
cred_guard_mutexis problematic. Thecred_guard_mutexis held over the userspace accesses as the arguments from userspace are read. Thecred_guard_mutexis held ofPTRACE_EVENT_EXITas the the other^ over
Quoted 3 lines
... is held while waiting for the trace parent to handle
PTRACE_EVENT_EXITor something?Yes. Let me see if I can phrase that better.
Quoted 3 lines
I wonder if we also should mention that
it is held while waiting for the trace parent to
receive the exit code with "wait"?I don't think we have to spell out the details of how it all works, unless that makes things clearer. Kernel developers can be expected to figure out how the kernel works. The critical thing is that it is an indefinite wait for userspace to take action.
But I will look.
Quoted 8 lines
Quoted 7 lines
threads are killed. The
cred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm().Any of those can result in deadlock, as the
cred_guard_mutexis held over a possible indefinite userspace waits for userspace.Add
exec_update_mutexthat is only held over exec updating processQuoted 8 lines
Quoted 4 lines
Quoted 1 line
Add ?
Yes. That is what the change does: add
exec_update_mutex.I just kind of missed the "subject" in this sentence, like "This patch adds an
exec_update_mutexthat is ..." but english is a foreign language for me, so may be okay as is.English has a lot of options. I think this is a stylistic difference.
Instead of being an observer and describing what the change does:
"This patch addsexec_update_mutex..."I was being there in the moment and saying/commading what is happening:
"Addexec_update_mutex..."
Quoted 8 lines
Using the more immdediate form ends up with more concise and clearer sentences.
Every one of my writing teachers in school emphasized that point and I see the who it works when I write things. But writing is hard and I still tend toward long rambling sentences with many qualifiers that confuse and detract from the point rather than make it clear what is happening.
And reading through it all now I can see your confusion. That description of my changes was not well done. Reworking it now.
Eric
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
My rewritten change description reads as follows:
exec: Add a exec_update_mutex to replace cred_guard_mutex
The cred_guard_mutex is problematic as it is held over possibly indefinite waits for userspace. The possilbe indefinite waits for userspace that I have identified are: The cred_guard_mutex is held in PTRACE_EVENT_EXIT waiting for the tracer. The cred_guard_mutex is held over "put_user(0, tsk->clear_child_tid)" in exit_mm(). The cred_guard_mutex is held over "get_user(futex_offset, ...") in exit_robust_list. The cred_guard_mutex held over copy_strings.
The functions get_user and put_user can trigger a page fault which can potentially wait indefinitely in the case of userfaultfd or if userspace implements part of the page fault path.
In any of those cases the userspace process that the kernel is waiting for might userspace might make a different system call that winds up taking the cred_guard_mutex and result in deadlock.
Holding a mutex over any of those possibly indefinite waits for userspace does not appear necessary. Add exec_update_mutex that will just cover updating the process during exec where the permissions and the objects pointed to by the task struct may be out of sync.
The plan is to switch the users of cred_guard_mutex to exec_udpate_mutex one by one. This lets us move forward while still being careful and not introducing any regressions.
Ref: 45c1a159b85b ("Add PTRACE_O_TRACEVFORKDONE and PTRACE_O_TRACEEXIT facilities.")
Ref: 456f17cd1a28 ("[PATCH] user-vm-unlock-2.5.31-A2")
Does that sound better?
Eric
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 4 lines · Eric W. Biederman
My rewritten change description reads as follows:
exec: Add a
exec_update_mutexto replacecred_guard_mutex
is this "an" exec_update_mutex?
Quoted 9 lines
The
cred_guard_mutexis problematic as it is held over possibly indefinite waits for userspace. The possilbe indefinite waits for userspace that I have identified are: Thecred_guard_mutexis held inPTRACE_EVENT_EXITwaiting for the tracer. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm(). Thecred_guard_mutexis held over "get_user(futex_offset, ...")inexit_robust_list. Thecred_guard_mutexheld overcopy_strings.
Quoted 6 lines
The functions
get_userandput_usercan trigger a page fault which can potentially wait indefinitely in the case of userfaultfd or if userspace implements part of the page fault path.In any of those cases the userspace process that the kernel is waiting for might userspace might make a different system call that winds up
^-------------^
^- remove this
Quoted 9 lines
taking the
cred_guard_mutexand result in deadlock.Holding a mutex over any of those possibly indefinite waits for userspace does not appear necessary. Add
exec_update_mutexthat will just cover updating the process during exec where the permissions and the objects pointed to by the task struct may be out of sync.The plan is to switch the users of
cred_guard_mutextoexec_udpate_mutexone by one. This lets us move forward while still
^-- typo: update
Quoted 11 lines
being careful and not introducing any regressions.
Link: https://lore.kernel.org/lkml/[email protected]/
Link: https://lore.kernel.org/lkml/AM6PR03MB5170B06F3A2B75EFB98D071AE4E60@AM6PR03MB5170.eurprd03.prod.outlook.com/
Link: https://lore.kernel.org/linux-fsdevel/[email protected]/
Link: https://lore.kernel.org/lkml/[email protected]/
Link: https://lore.kernel.org/lkml/[email protected]/
Ref: 45c1a159b85b ("Add PTRACE_O_TRACEVFORKDONE and PTRACE_O_TRACEEXIT facilities.")
Ref: 456f17cd1a28 ("[PATCH] user-vm-unlock-2.5.31-A2")
Signed-off-by: "Eric W. Biederman" <[email protected]>Quoted 2 lines
Does that sound better?
almost done.
Quoted 2 lines
Eric
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 7 lines · Bernd Edlinger
Quoted 4 lines · Eric W. Biederman
Does that sound better?
almost done.
I think this text is finally clean.
exec: Add exec_update_mutex to replace cred_guard_mutex
The cred_guard_mutex is problematic as it is held over possibly indefinite waits for userspace. The possilbe indefinite waits for userspace that I have identified are: The cred_guard_mutex is held in PTRACE_EVENT_EXIT waiting for the tracer. The cred_guard_mutex is held over "put_user(0, tsk->clear_child_tid)" in exit_mm(). The cred_guard_mutex is held over "get_user(futex_offset, ...") in exit_robust_list. The cred_guard_mutex held over copy_strings.
The functions get_user and put_user can trigger a page fault which can potentially wait indefinitely in the case of userfaultfd or if userspace implements part of the page fault path.
In any of those cases the userspace process that the kernel is waiting for might make a different system call that winds up taking the cred_guard_mutex and result in deadlock.
Holding a mutex over any of those possibly indefinite waits for userspace does not appear necessary. Add exec_update_mutex that will just cover updating the process during exec where the permissions and the objects pointed to by the task struct may be out of sync.
The plan is to switch the users of cred_guard_mutex to exec_update_mutex one by one. This lets us move forward while still being careful and not introducing any regressions.
Ref: 45c1a159b85b ("Add PTRACE_O_TRACEVFORKDONE and PTRACE_O_TRACEEXIT facilities.")
Ref: 456f17cd1a28 ("[PATCH] user-vm-unlock-2.5.31-A2")
Bernd do you want to give me your Reviewed-by for this part of the series?
After that do you think you can write the obvious patch for mm_access?
I will apply these changes to my tree and push them into linux-next.
Eric
Bernd Edlinger
Reviewed-byRe: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 39 lines · Eric W. Biederman
Quoted 7 lines · Bernd Edlinger
Quoted 4 lines · Eric W. Biederman
Does that sound better?
almost done.
I think this text is finally clean.
exec: Add
exec_update_mutexto replacecred_guard_mutexThe
cred_guard_mutexis problematic as it is held over possibly indefinite waits for userspace. The possilbe indefinite waits for userspace that I have identified are: Thecred_guard_mutexis held inPTRACE_EVENT_EXITwaiting for the tracer. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm(). Thecred_guard_mutexis held over "get_user(futex_offset, ...")inexit_robust_list. Thecred_guard_mutexheld overcopy_strings.The functions
get_userandput_usercan trigger a page fault which can potentially wait indefinitely in the case of userfaultfd or if userspace implements part of the page fault path.In any of those cases the userspace process that the kernel is waiting for might make a different system call that winds up taking the
cred_guard_mutexand result in deadlock.Holding a mutex over any of those possibly indefinite waits for userspace does not appear necessary. Add
exec_update_mutexthat will just cover updating the process during exec where the permissions and the objects pointed to by the task struct may be out of sync.The plan is to switch the users of
cred_guard_mutextoexec_update_mutexone by one. This lets us move forward while still being careful and not introducing any regressions.
Quoted 7 lines
Ref:
45c1a159b85b("AddPTRACE_O_TRACEVFORKDONEandPTRACE_O_TRACEEXITfacilities.")
Ref:456f17cd1a28("[PATCH] user-vm-unlock-2.5.31-A2")
I checked the urls they all work.
Just one last question, are these git references?
I can't find them in my linux git tree (cloned from linus' git)?
Sorry for being pedantically.
Quoted 6 lines
Signed-off-by "Eric W. Biederman" <[email protected]>Bernd do you want to give me your Reviewed-by for this part of the series?
Sure also the other parts of course.
Quoted 2 lines
After that do you think you can write the obvious patch for
mm_access?
Yes, I can do that.
I also have some typos in comments, will make them extra patches as well.
I wonder if the test case is okay to include the ptrace_attach altough that is not yet passing?
Thanks
Bernd.
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 48 lines · Bernd Edlinger
Quoted 39 lines · Eric W. Biederman
Quoted 7 lines · Bernd Edlinger
Quoted 4 lines · Eric W. Biederman
Does that sound better?
almost done.
I think this text is finally clean.
exec: Add
exec_update_mutexto replacecred_guard_mutexThe
cred_guard_mutexis problematic as it is held over possibly indefinite waits for userspace. The possilbe indefinite waits for userspace that I have identified are: Thecred_guard_mutexis held inPTRACE_EVENT_EXITwaiting for the tracer. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm(). Thecred_guard_mutexis held over "get_user(futex_offset, ...")inexit_robust_list. Thecred_guard_mutexheld overcopy_strings.The functions
get_userandput_usercan trigger a page fault which can potentially wait indefinitely in the case of userfaultfd or if userspace implements part of the page fault path.In any of those cases the userspace process that the kernel is waiting for might make a different system call that winds up taking the
cred_guard_mutexand result in deadlock.Holding a mutex over any of those possibly indefinite waits for userspace does not appear necessary. Add
exec_update_mutexthat will just cover updating the process during exec where the permissions and the objects pointed to by the task struct may be out of sync.The plan is to switch the users of
cred_guard_mutextoexec_update_mutexone by one. This lets us move forward while still being careful and not introducing any regressions.Quoted 7 lines
Ref:
45c1a159b85b("AddPTRACE_O_TRACEVFORKDONEandPTRACE_O_TRACEEXITfacilities.")
Ref:456f17cd1a28("[PATCH] user-vm-unlock-2.5.31-A2")
Quoted 5 lines
I checked the urls they all work.
Just one last question, are these git references?
I can't find them in my linux git tree (cloned from linus' git)?Sorry for being pedantically.
You have to track down tglx's historicaly git tree from when everything was in bitkeeper.
But yes they are git references and yes they work. Just that part of the history is not in linux.git.
Quoted 11 lines
Quoted 6 lines
Signed-off-by "Eric W. Biederman" <[email protected]>Bernd do you want to give me your Reviewed-by for this part of the series?
Sure also the other parts of course.
Reviewed-by Bernd Edlinger <[email protected]>
Quoted 8 lines
Quoted 2 lines
After that do you think you can write the obvious patch for
mm_access?Yes, I can do that.
I also have some typos in comments, will make them extra patches as well.I wonder if the test case is okay to include the
ptrace_attachaltough that is not yet passing?
It is an existing kernel but that it doesn't pass.
My sense is that if you include it as a separate patch if it is a problem for someone we can identify it easily via bisect and we do whatever is appropriate.
Eric
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
In reply to Bernd EdlingerQuoted 11 lines · Bernd Edlinger
Quoted 9 lines · Eric W. Biederman
Link: https://lore.kernel.org/lkml/[email protected]/ Link: https://lore.kernel.org/lkml/AM6PR03MB5170B06F3A2B75EFB98D071AE4E60@AM6PR03MB5170.eurprd03.prod.outlook.com/ Link: https://lore.kernel.org/linux-fsdevel/[email protected]/ Link: https://lore.kernel.org/lkml/[email protected]/ Link: https://lore.kernel.org/lkml/[email protected]/ Ref: 45c1a159b85b ("Add PTRACE_O_TRACEVFORKDONE and PTRACE_O_TRACEEXIT facilities.") Ref: 456f17cd1a28 ("[PATCH] user-vm-unlock-2.5.31-A2")
Quoted 3 lines
I checked the urls they all work.
Just one last question, are these git references?
I can't find them in my linux git tree (cloned from linus' git)?
I will add this tag to help people figure out what is going on.
History Tree: https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
Eric
cover Use new infrastructure to fix deadlocks in execve
This is a follow up on Eric's patch series to fix the deadlocks observed with ptracing when execve in multi-threaded applications.
This fixes the simple and most important case where the cred_guard_mutex causes strace to deadlock.
This also adds a test case (which is only partially fixed so far, the rest of the fixes will follow soon).
Two trivial comment fixes are also included.
Bernd Edlinger (4):
exec: Fix a deadlock in ptrace
selftests/ptrace: add test cases for dead-locks
mm: docs: Fix a comment in process_vm_rw_core
kernel: doc: remove outdated comment in prepare_kernel_cred
kernel/cred.c | 2 - kernel/fork.c | 4 +- mm/process_vm_access.c | 2 +- tools/testing/selftests/ptrace/Makefile | 4 +- tools/testing/selftests/ptrace/vmaccess.c | 86 +++++++++++++++++++++++++++++++ 5 files changed, 91 insertions(+), 7 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.c
Signature · 1 line
1.9.1
Eric W. Biederman
AppliedRe: [PATCH 0/4] Use new infrastructure to fix deadlocks in execve
Quoted 19 lines · Bernd Edlinger
This is a follow up on Eric's patch series to fix the deadlocks observed with ptracing when execve in multi-threaded applications.
This fixes the simple and most important case where the
cred_guard_mutexcauses strace to deadlock.This also adds a test case (which is only partially fixed so far, the rest of the fixes will follow soon).
Two trivial comment fixes are also included.
Bernd Edlinger (4):
exec: Fix a deadlock in ptrace
selftests/ptrace: add test cases for dead-locks
mm: docs: Fix a comment inprocess_vm_rw_core
kernel: doc: remove outdated comment inprepare_kernel_cred
Quoted 7 lines
kernel/cred.c | 2 - kernel/fork.c | 4 +- mm/process_vm_access.c | 2 +- tools/testing/selftests/ptrace/Makefile | 4 +- tools/testing/selftests/ptrace/vmaccess.c | 86 +++++++++++++++++++++++++++++++ 5 files changed, 91 insertions(+), 7 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.c
Applied.
Thank you,
Eric
cover Use new infrastructure in more simple cases
This continues the execve anti-deadlock patch and addresses all of the (mostly) simple cases, there the new exec_update_mutex can be used instead of the cred_guard_mutex.
Note: each of these patches is independent of each other, so in case one of them turns out to be controversial, that does not affect the others.
Bernd Edlinger (4):
kernel/kcmp.c: Use new infrastructure to fix deadlocks in execve
proc: Use new infrastructure to fix deadlocks in execve
proc: io_accounting: Use new infrastructure to fix deadlocks in execve
perf: Use new infrastructure to fix deadlocks in execve
fs/proc/base.c | 10 +++++----- kernel/events/core.c | 12 ++++++------ kernel/kcmp.c | 8 ++++---- 3 files changed, 15 insertions(+), 15 deletions(-)
Signature · 1 line
1.9.1
PATCH 1/4 kernel/kcmp.c: Use new infrastructure to fix deadlocks in execve
In reply to Eric W. BiedermanThis changes kcmp_epoll_target to use the new exec_update_mutex instead of cred_guard_mutex.
This should be safe, as the credentials are only used for reading, and furthermore ->mm and ->sighand are updated on execve, but only under the new exec_update_mutex.
--- kernel/kcmp.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/kcmp.c +4 -4
@@ -173,8 +173,8 @@ static int kcmp_epoll_target(struct task_struct *task1,/* * One should have enough rights to inspect task details. */ - ret =kcmp_lock (&task1->signal->cred_guard_mutex, - &task2->signal->cred_guard_mutex); + ret =kcmp_lock (&task1->signal->exec_update_mutex, + &task2->signal->exec_update_mutex);if (ret) goto err; if (!ptrace_may_access(task1, PTRACE_MODE_READ_REALCREDS) ||
@@ -229,8 +229,8 @@ static int kcmp_epoll_target(struct task_struct *task1, } err_unlock: -kcmp_unlock (&task1->signal->cred_guard_mutex, - &task2->signal->cred_guard_mutex); +kcmp_unlock (&task1->signal->exec_update_mutex, + &task2->signal->exec_update_mutex); err:put_task_struct (task1);put_task_struct (task2);
Signature · 1 line
1.9.1
Re: [PATCH 1/4] kernel/kcmp.c: Use new infrastructure to fix deadlocks in execve
Quoted 7 lines · Bernd Edlinger
This changes
kcmp_epoll_targetto use the newexec_update_mutexinstead ofcred_guard_mutex.This should be safe, as the credentials are only used for reading, and furthermore ->mm and ->sighand are updated on execve, but only under the new
exec_update_mutex.
Can you add a comment that the exec_update_mutex is not needed for KCMP_FILE? As both sets of credentials during exec are valid for accessing the files so exec_update_mutex does not matter.
I don't think exec_update_mutex is needed for KCMP_SYSVSEM or KCMP_EPOLL_TFD either. As I don't think exec changes either one of those.
Eric
Quoted 31 lines
Signed-off-by Bernd Edlinger <[email protected]>--- kernel/kcmp.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)kernel/kcmp.c +4
-4@@ -173,8 +173,8 @@ static int kcmp_epoll_target(struct task_struct *task1,/* * One should have enough rights to inspect task details. */ - ret =kcmp_lock (&task1->signal->cred_guard_mutex, - &task2->signal->cred_guard_mutex); + ret =kcmp_lock (&task1->signal->exec_update_mutex, + &task2->signal->exec_update_mutex);if (ret) goto err; if (!ptrace_may_access(task1, PTRACE_MODE_READ_REALCREDS) ||@@ -229,8 +229,8 @@ static int kcmp_epoll_target(struct task_struct *task1, } err_unlock: -kcmp_unlock (&task1->signal->cred_guard_mutex, - &task2->signal->cred_guard_mutex); +kcmp_unlock (&task1->signal->exec_update_mutex, + &task2->signal->exec_update_mutex); err:put_task_struct (task1);put_task_struct (task2);
Re: [PATCH 1/4] kernel/kcmp.c: Use new infrastructure to fix deadlocks in execve
Quoted 5 lines · Eric W. Biederman
Quoted 3 lines · Bernd Edlinger
This changes
kcmp_epoll_targetto use the newexec_update_mutexinstead ofcred_guard_mutex.
Quoted 9 lines
Quoted 4 lines
This should be safe, as the credentials are only used for reading, and furthermore ->mm and ->sighand are updated on execve, but only under the new
exec_update_mutex.Can you add a comment that the
exec_update_mutexis not needed forKCMP_FILE? As both sets of credentials during exec are valid for accessing the files soexec_update_mutexdoes not matter.
some files are closed by do_close_on_exec,
so in theory this allows you to examine files that were open in the old user but closed for the new user with either credential.
It is not a race condition, but it may be a security concern.
Quoted 4 lines
I don't think
exec_update_mutexis needed forKCMP_SYSVSEMorKCMP_EPOLL_TFDeither. As I don't think exec changes either one of those.
KCMP_EPOLL_TFD is also accessing file pointers,
that is possible.
It might be that KCMP_SYSVSEM is a missed optimization, but I may have overlooked something.
I'd rather err on the safe side.
Quoted 34 lines
Eric
Quoted 31 lines
Signed-off-by Bernd Edlinger <[email protected]>--- kernel/kcmp.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)kernel/kcmp.c +4
-4@@ -173,8 +173,8 @@ static int kcmp_epoll_target(struct task_struct *task1,/* * One should have enough rights to inspect task details. */ - ret =kcmp_lock (&task1->signal->cred_guard_mutex, - &task2->signal->cred_guard_mutex); + ret =kcmp_lock (&task1->signal->exec_update_mutex, + &task2->signal->exec_update_mutex);if (ret) goto err; if (!ptrace_may_access(task1, PTRACE_MODE_READ_REALCREDS) ||@@ -229,8 +229,8 @@ static int kcmp_epoll_target(struct task_struct *task1, } err_unlock: -kcmp_unlock (&task1->signal->cred_guard_mutex, - &task2->signal->cred_guard_mutex); +kcmp_unlock (&task1->signal->exec_update_mutex, + &task2->signal->exec_update_mutex); err:put_task_struct (task1);put_task_struct (task2);
PATCH 2/4 proc: Use new infrastructure to fix deadlocks in execve
In reply to Eric W. BiedermanThis changes lock_trace to use the new exec_update_mutex instead of cred_guard_mutex.
This fixes possible deadlocks when the trace is accessing /proc/$pid/stack for instance.
This should be safe, as the credentials are only used for reading, and task->mm is updated on execve under the new exec_update_mutex.
--- fs/proc/base.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/proc/base.c +3 -3
@@ -403,11 +403,11 @@ static int proc_pid_wchan(struct seq_file *m, struct pid_namespace *ns,static int lock_trace (struct task_struct *task) { -int err =mutex_lock_killable (&task->signal->cred_guard_mutex); +int err =mutex_lock_killable (&task->signal->exec_update_mutex);if (err)return err;if (!ptrace_may_access (task,PTRACE_MODE_ATTACH_FSCREDS )) { -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return -EPERM ; }return 0 ;
@@ -415,7 +415,7 @@ static int lock_trace(struct task_struct *task)static void unlock_trace (struct task_struct *task) { -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex); }#ifdef CONFIG_STACKTRACE
Signature · 1 line
1.9.1
Re: [PATCH 2/4] proc: Use new infrastructure to fix deadlocks in execve
Quoted 10 lines · Bernd Edlinger
This changes
lock_traceto use the newexec_update_mutexinstead ofcred_guard_mutex.This fixes possible deadlocks when the trace is accessing
/proc/$pid/stackfor instance.This should be safe, as the credentials are only used for reading, and
task->mmis updated on execve under the newexec_update_mutex.Signed-off-by Bernd Edlinger <[email protected]>
-Kees
Quoted 33 lines
--- fs/proc/base.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)fs/proc/base.c +3
-3@@ -403,11 +403,11 @@ static int proc_pid_wchan(struct seq_file *m, struct pid_namespace *ns,static int lock_trace (struct task_struct *task) { -int err =mutex_lock_killable (&task->signal->cred_guard_mutex); +int err =mutex_lock_killable (&task->signal->exec_update_mutex);if (err)return err;if (!ptrace_may_access (task,PTRACE_MODE_ATTACH_FSCREDS )) { -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return -EPERM ; }return 0 ;@@ -415,7 +415,7 @@ static int lock_trace(struct task_struct *task)static void unlock_trace (struct task_struct *task) { -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex); }#ifdef CONFIG_STACKTRACE Signature · 1 line
1.9.1
Signature · 1 line
Kees Cook
Re: [PATCH 2/4] proc: Use new infrastructure to fix deadlocks in execve
In reply to Bernd EdlingerQuoted 10 lines · Bernd Edlinger
This changes
lock_traceto use the newexec_update_mutexinstead ofcred_guard_mutex.This fixes possible deadlocks when the trace is accessing
/proc/$pid/stackfor instance.This should be safe, as the credentials are only used for reading, and
task->mmis updated on execve under the newexec_update_mutex.Signed-off-by Bernd Edlinger <[email protected]>
I have the same question here as in 3/4. I should probably rescind my Reviewed-by until I'm convinced about the security-safety of this -- why is this not a race against cred changes?
-Kees
Quoted 33 lines
--- fs/proc/base.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)fs/proc/base.c +3
-3@@ -403,11 +403,11 @@ static int proc_pid_wchan(struct seq_file *m, struct pid_namespace *ns,static int lock_trace (struct task_struct *task) { -int err =mutex_lock_killable (&task->signal->cred_guard_mutex); +int err =mutex_lock_killable (&task->signal->exec_update_mutex);if (err)return err;if (!ptrace_may_access (task,PTRACE_MODE_ATTACH_FSCREDS )) { -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return -EPERM ; }return 0 ;@@ -415,7 +415,7 @@ static int lock_trace(struct task_struct *task)static void unlock_trace (struct task_struct *task) { -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex); }#ifdef CONFIG_STACKTRACE Signature · 1 line
1.9.1
Signature · 1 line
Kees Cook
Re: [PATCH 2/4] proc: Use new infrastructure to fix deadlocks in execve
Quoted 7 lines · Kees Cook
Quoted 6 lines · Bernd Edlinger
This changes
lock_traceto use the newexec_update_mutexinstead ofcred_guard_mutex.This fixes possible deadlocks when the trace is accessing
/proc/$pid/stackfor instance.
Quoted 9 lines
Quoted 4 lines
This should be safe, as the credentials are only used for reading, and
task->mmis updated on execve under the newexec_update_mutex.Signed-off-by Bernd Edlinger <[email protected]>I have the same question here as in 3/4. I should probably rescind my Reviewed-by until I'm convinced about the security-safety of this -- why is this not a race against cred changes?
The credentials of a thread that is currently executing execve is already set in the bprm structure, however the credential in the task structure is not yet changed, as well as the process memory map keeps stable until the exec_update_mutex is acquired.
What is done with this functions is access the call stack of the process before the new executable is actually started.
There would immediately be a severe security problem if we did not use any mutex as the check would be then with the old credential, but the stack trace would potentially reveal secret function calls that are done by a setuid program when it starts up.
Bernd.
Quoted 36 lines
-Kees
Quoted 33 lines
--- fs/proc/base.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)fs/proc/base.c +3
-3@@ -403,11 +403,11 @@ static int proc_pid_wchan(struct seq_file *m, struct pid_namespace *ns,static int lock_trace (struct task_struct *task) { -int err =mutex_lock_killable (&task->signal->cred_guard_mutex); +int err =mutex_lock_killable (&task->signal->exec_update_mutex);if (err)return err;if (!ptrace_may_access (task,PTRACE_MODE_ATTACH_FSCREDS )) { -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return -EPERM ; }return 0 ;@@ -415,7 +415,7 @@ static int lock_trace(struct task_struct *task)static void unlock_trace (struct task_struct *task) { -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex); }#ifdef CONFIG_STACKTRACE Signature · 1 line
1.9.1
PATCH 3/4 proc: io_accounting: Use new infrastructure to fix deadlocks in execve
In reply to Eric W. BiedermanThis changes do_io_accounting to use the new exec_update_mutex instead of cred_guard_mutex.
This fixes possible deadlocks when the trace is accessing /proc/$pid/io for instance.
This should be safe, as the credentials are only used for reading.
--- fs/proc/base.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/proc/base.c +2 -2
@@ -2770,7 +2770,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int whunsigned long flags;int result; - result =mutex_lock_killable (&task->signal->cred_guard_mutex); + result =mutex_lock_killable (&task->signal->exec_update_mutex);if (result)return result;
@@ -2806,7 +2806,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int wh result =0 ; out_unlock: -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return result; }
Signature · 1 line
1.9.1
Re: [PATCH 3/4] proc: io_accounting: Use new infrastructure to fix deadlocks in execve
Quoted 7 lines · Bernd Edlinger
This changes
do_io_accountingto use the newexec_update_mutexinstead ofcred_guard_mutex.This fixes possible deadlocks when the trace is accessing
/proc/$pid/iofor instance.This should be safe, as the credentials are only used for reading.
This is an improvement.
We probably want to do this just as an incremental step in making things better but perhaps I am blind but I am not finding the reason for guarding this with the cred_guard_mutex to be at all persuasive.
I think moving the ptrace_may_access check down to after the unlock_task_sighand would be just as effective at addressing the concerns raised in the original commit. I think the task_lock provides all of the barrier we need to make it safe to move the ptrace_may_access checks safe.
The reason I say this is I don't see exec changing ->ioac. Just performing some I/O which would update the io accounting statistics.
Can anyone see if I am wrong?
Eric
commit 293eb1e7772b25a93647c798c7b89bf26c2da2e0
Author: Vasiliy Kulikov <[email protected]> Date: Tue Jul 26 16:08:38 2011 -0700
proc: fix a race in do_io_accounting()
If an inode's mode permits opening /proc/PID/io and the resulting file descriptor is kept across execve() of a setuid or similar binary, the ptrace_may_access() check tries to prevent using this fd against the task with escalated privileges.
Unfortunately, there is a race in the check against execve(). If execve() is processed after the ptrace check, but before the actual io information gathering, io statistics will be gathered from the privileged process. At least in theory this might lead to gathering sensible information (like ssh/ftp password length) that wouldn't be available otherwise.
Holding task->signal->cred_guard_mutex while gathering the io information should protect against the race.
The order of locking is similar to the one inside of ptrace_attach():
first goes cred_guard_mutex, then lock_task_sighand().
Quoted 26 lines
Signed-off-by Bernd Edlinger <[email protected]>--- fs/proc/base.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/proc/base.c +2
-2@@ -2770,7 +2770,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int whunsigned long flags;int result; - result =mutex_lock_killable (&task->signal->cred_guard_mutex); + result =mutex_lock_killable (&task->signal->exec_update_mutex);if (result)return result;@@ -2806,7 +2806,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int wh result =0 ; out_unlock: -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return result; }
Re: [PATCH 3/4] proc: io_accounting: Use new infrastructure to fix deadlocks in execve
Quoted 16 lines · Eric W. Biederman
Quoted 7 lines · Bernd Edlinger
This changes
do_io_accountingto use the newexec_update_mutexinstead ofcred_guard_mutex.This fixes possible deadlocks when the trace is accessing
/proc/$pid/iofor instance.This should be safe, as the credentials are only used for reading.
This is an improvement.
We probably want to do this just as an incremental step in making things better but perhaps I am blind but I am not finding the reason for guarding this with the
cred_guard_mutexto be at all persuasive.
Quoted 9 lines
I think moving the
ptrace_may_accesscheck down to after theunlock_task_sighandwould be just as effective at addressing the concerns raised in the original commit. I think thetask_lockprovides all of the barrier we need to make it safe to move theptrace_may_accesschecks safe.The reason I say this is I don't see exec changing ->ioac. Just performing some I/O which would update the io accounting statistics.
Maybe the suid executable is starting up and doing io or not, and what the program does immediately at startup is a secret, that we want to keep secret but evil eve want to find out. eve is using /proc/alice/io to do that.
It is a bit constructed, but seems like a security concern. when we keep the exec_update_mutex while collecting the data, we cannot see any io of the new process when the new credentials don't allow that.
Bernd.
Quoted 63 lines
Can anyone see if I am wrong?
Eric
commit
293eb1e7772b25a93647c798c7b89bf26c2da2e0Author: Vasiliy Kulikov <[email protected]> Date: Tue Jul 26 16:08:38 2011 -0700proc: fix a race in
do_io_accounting()If an inode's mode permits opening
/proc/PID/ioand the resulting file descriptor is kept acrossexecve()of a setuid or similar binary, theptrace_may_access()check tries to prevent using this fd against the task with escalated privileges.Unfortunately, there is a race in the check against
execve(). Ifexecve()is processed after the ptrace check, but before the actual io information gathering, io statistics will be gathered from the privileged process. At least in theory this might lead to gathering sensible information (like ssh/ftp password length) that wouldn't be available otherwise.Holding
task->signal->cred_guard_mutexwhile gathering the io information should protect against the race.The order of locking is similar to the one inside of
ptrace_attach():
first goescred_guard_mutex, thenlock_task_sighand().Signed-off-by Vasiliy Kulikov <[email protected]>Cc Al Viro <[email protected]>Cc <[email protected]>Signed-off-by Andrew Morton <[email protected]>Signed-off-by Linus Torvalds <[email protected]>Quoted 26 lines
Signed-off-by Bernd Edlinger <[email protected]>--- fs/proc/base.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/proc/base.c +2
-2@@ -2770,7 +2770,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int whunsigned long flags;int result; - result =mutex_lock_killable (&task->signal->cred_guard_mutex); + result =mutex_lock_killable (&task->signal->exec_update_mutex);if (result)return result;@@ -2806,7 +2806,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int wh result =0 ; out_unlock: -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return result; }
Re: [PATCH 3/4] proc: io_accounting: Use new infrastructure to fix deadlocks in execve
Quoted 32 lines · Bernd Edlinger
Quoted 16 lines · Eric W. Biederman
Quoted 7 lines · Bernd Edlinger
This changes
do_io_accountingto use the newexec_update_mutexinstead ofcred_guard_mutex.This fixes possible deadlocks when the trace is accessing
/proc/$pid/iofor instance.This should be safe, as the credentials are only used for reading.
This is an improvement.
We probably want to do this just as an incremental step in making things better but perhaps I am blind but I am not finding the reason for guarding this with the
cred_guard_mutexto be at all persuasive.Quoted 9 lines
I think moving the
ptrace_may_accesscheck down to after theunlock_task_sighandwould be just as effective at addressing the concerns raised in the original commit. I think thetask_lockprovides all of the barrier we need to make it safe to move theptrace_may_accesschecks safe.The reason I say this is I don't see exec changing ->ioac. Just performing some I/O which would update the io accounting statistics.
Maybe the suid executable is starting up and doing io or not, and what the program does immediately at startup is a secret, that we want to keep secret but evil eve want to find out. eve is using
/proc/alice/ioto do that.
Quoted 4 lines
It is a bit constructed, but seems like a security concern. when we keep the
exec_update_mutexwhile collecting the data, we cannot see any io of the new process when the new credentials don't allow that.
Jann Horn has convinced me we should just convert these to the exec_change_mutex today. Because while not 100% correct in theory, the only really interesting case is exec. So the code does something interesting and worth while, and mostly correct. The last thing I want to do is to cause an unnecessary regression.
Eric
Re: [PATCH 3/4] proc: io_accounting: Use new infrastructure to fix deadlocks in execve
In reply to Bernd EdlingerQuoted 7 lines · Bernd Edlinger
This changes
do_io_accountingto use the newexec_update_mutexinstead ofcred_guard_mutex.This fixes possible deadlocks when the trace is accessing
/proc/$pid/iofor instance.This should be safe, as the credentials are only used for reading.
I'd like to see the rationale described better here for why it should be safe. I'm still not seeing why this is safe here, as we might check ptrace_may_access() with one cred and then iterate io accounting with a different credential...
What am I missing?
-Kees
Quoted 30 lines
Signed-off-by Bernd Edlinger <[email protected]>--- fs/proc/base.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/proc/base.c +2
-2@@ -2770,7 +2770,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int whunsigned long flags;int result; - result =mutex_lock_killable (&task->signal->cred_guard_mutex); + result =mutex_lock_killable (&task->signal->exec_update_mutex);if (result)return result;@@ -2806,7 +2806,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int wh result =0 ; out_unlock: -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return result; }Signature · 1 line
1.9.1
Signature · 1 line
Kees Cook
Re: [PATCH 3/4] proc: io_accounting: Use new infrastructure to fix deadlocks in execve
Quoted 7 lines · Kees Cook
Quoted 6 lines · Bernd Edlinger
This changes
do_io_accountingto use the newexec_update_mutexinstead ofcred_guard_mutex.This fixes possible deadlocks when the trace is accessing
/proc/$pid/iofor instance.
Quoted 9 lines
Quoted 1 line
This should be safe, as the credentials are only used for reading.
I'd like to see the rationale described better here for why it should be safe. I'm still not seeing why this is safe here, as we might check
ptrace_may_access()with one cred and then iterate io accounting with a different credential...What am I missing?
The same here, even if execve is already started, the credentials are not actually changed until the execve acquired the exec_update_mutex.
The data flow is from the task->cred => do_io_accounting, if the data flow would be from do_io_accounting => task's no new privs you would see an entirely different patch.
I am open for suggestions how to improve the description, or even add a comment from time to time :)
Thanks
Bernd.
Quoted 33 lines
-Kees
Quoted 30 lines
Signed-off-by Bernd Edlinger <[email protected]>--- fs/proc/base.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/proc/base.c +2
-2@@ -2770,7 +2770,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int whunsigned long flags;int result; - result =mutex_lock_killable (&task->signal->cred_guard_mutex); + result =mutex_lock_killable (&task->signal->exec_update_mutex);if (result)return result;@@ -2806,7 +2806,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int wh result =0 ; out_unlock: -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return result; }Signature · 1 line
1.9.1
Re: [PATCH 3/4] proc: io_accounting: Use new infrastructure to fix deadlocks in execve
In reply to Kees CookQuoted 7 lines · Kees Cook
Quoted 6 lines · Bernd Edlinger
This changes
do_io_accountingto use the newexec_update_mutexinstead ofcred_guard_mutex.This fixes possible deadlocks when the trace is accessing
/proc/$pid/iofor instance.
Quoted 8 lines
Quoted 1 line
This should be safe, as the credentials are only used for reading.
I'd like to see the rationale described better here for why it should be safe. I'm still not seeing why this is safe here, as we might check
ptrace_may_access()with one cred and then iterate io accounting with a different credential...What am I missing?
The rational for non-regression is that exec_update_mutex covers all of the same tsk->cred changes as cred_guard_mutex. Therefore we are not any worse off, and we avoid the deadlock.
As for safety. Jann's argument that the only interesting credential change is in exec applies. All other credential changes that have any effect on permission checks make the new cred non-dumpable (excepions apply see the code).
So I think this is a non-regressing change. A safe change.
I don't think either version of this code is fully correct.
Eric
Quoted 29 lines
Quoted 29 lines
Signed-off-by Bernd Edlinger <[email protected]>--- fs/proc/base.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/proc/base.c +2
-2@@ -2770,7 +2770,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int whunsigned long flags;int result; - result =mutex_lock_killable (&task->signal->cred_guard_mutex); + result =mutex_lock_killable (&task->signal->exec_update_mutex);if (result)return result;@@ -2806,7 +2806,7 @@ static int do_io_accounting(struct task_struct *task, struct seq_file *m, int wh result =0 ; out_unlock: -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return result; }Signature · 1 line
1.9.1
PATCH 4/4 perf: Use new infrastructure to fix deadlocks in execve
In reply to Eric W. BiedermanThis changes perf_event_set_clock to use the new exec_update_mutex instead of cred_guard_mutex.
This should be safe, as the credentials are only used for reading.
--- kernel/events/core.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/events/core.c +6 -6
@@ -1248,7 +1248,7 @@ static void put_ctx(struct perf_event_context *ctx)* function. * * Lock order: -* cred_guard_mutex +* exec_update_mutex * task_struct::perf_event_mutex * perf_event_context::mutex * perf_event::child_mutex;
@@ -11254,14 +11254,14 @@ static int perf_event_set_clock(struct perf_event *event, clockid_t clk_id) }if (task) { - err =mutex_lock_interruptible (&task->signal->cred_guard_mutex); + err =mutex_lock_interruptible (&task->signal->exec_update_mutex);if (err) goto err_task;/* * Reuse ptrace permission checks for now. * -* We must hold cred_guard_mutex across this and any potential +* We must hold exec_update_mutex across this and any potential * perf_install_in_context() call for this new event to * serialize against exec() altering our credentials (and the * perf_event_exit_task() that could imply).
@@ -11550,7 +11550,7 @@ static int perf_event_set_clock(struct perf_event *event, clockid_t clk_id)mutex_unlock (&ctx->mutex);if (task) { -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);put_task_struct (task); }
@@ -11586,7 +11586,7 @@ static int perf_event_set_clock(struct perf_event *event, clockid_t clk_id)free_event (event); err_cred:if (task) -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex); err_task:if (task)put_task_struct (task);
@@ -11891,7 +11891,7 @@ static void perf_event_exit_task_context(struct task_struct *child, int ctxn)/* * When a child task exits, feed back event values to parent events. * -* Can be called with cred_guard_mutex held when called from +* Can be called with exec_update_mutex held when called from * install_exec_creds(). */ void perf_event_exit_task (struct task_struct *child)
Signature · 1 line
1.9.1
PATCH 1/4 exec: Fix a deadlock in ptrace
In reply to Eric W. BiedermanThis fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in vm_access.
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the PTRACE_EVENT_EXIT is handled and the WIFEXITED event is received:
strace D 0 30614 30584 0x00000000
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
expect D 0 31933 30876 0x80004003
Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
This changes mm_access to use the new exec_update_mutex instead of cred_guard_mutex.
This patch is based on the following patch by Eric W. Biederman:
"[PATCH 0/5] Infrastructure to allow fixing exec deadlocks"
--- kernel/fork.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/fork.c +2 -2
@@ -1224,7 +1224,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)struct mm_struct *mm;int err; - err =mutex_lock_killable (&task->signal->cred_guard_mutex); + err =mutex_lock_killable (&task->signal->exec_update_mutex);if (err)return ERR_PTR (err);
@@ -1234,7 +1234,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)mmput (mm); mm =ERR_PTR (-EACCES ); } -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return mm; }
Signature · 1 line
1.9.1
Quoted 8 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.
Quoted 4 lines
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:
Overall this looks good. Mind if I change the subject to:
"exec: Fix a deadlock in strace" ?
Eric
Quoted 62 lines
strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9This changes
mm_accessto use the newexec_update_mutexinstead ofcred_guard_mutex.This patch is based on the following patch by Eric W. Biederman:
"[PATCH 0/5] Infrastructure to allow fixing exec deadlocks"Signed-off-by Bernd Edlinger <[email protected]>--- kernel/fork.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/fork.c +2
-2@@ -1224,7 +1224,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)struct mm_struct *mm;int err; - err =mutex_lock_killable (&task->signal->cred_guard_mutex); + err =mutex_lock_killable (&task->signal->exec_update_mutex);if (err)return ERR_PTR (err);@@ -1234,7 +1234,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)mmput (mm); mm =ERR_PTR (-EACCES ); } -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return mm; }
Quoted 10 lines · Eric W. Biederman
Quoted 8 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.
Quoted 8 lines
Quoted 4 lines
The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:Overall this looks good. Mind if I change the subject to:
"exec: Fix a deadlock in strace" ?
Sure, go ahead.
Thanks
Bernd.
Quoted 65 lines
Eric
Quoted 62 lines
strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9This changes
mm_accessto use the newexec_update_mutexinstead ofcred_guard_mutex.This patch is based on the following patch by Eric W. Biederman:
"[PATCH 0/5] Infrastructure to allow fixing exec deadlocks"Signed-off-by Bernd Edlinger <[email protected]>--- kernel/fork.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/fork.c +2
-2@@ -1224,7 +1224,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)struct mm_struct *mm;int err; - err =mutex_lock_killable (&task->signal->cred_guard_mutex); + err =mutex_lock_killable (&task->signal->exec_update_mutex);if (err)return ERR_PTR (err);@@ -1234,7 +1234,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)mmput (mm); mm =ERR_PTR (-EACCES ); } -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return mm; }
Quoted 40 lines · Bernd Edlinger
This fixes a deadlock in the tracer when tracing a multi-threaded application that calls execve while more than one thread are running.
I observed that when running strace on the gcc test suite, it always blocks after a while, when expect calls execve, because other threads have to be terminated. They send ptrace events, but the strace is no longer able to respond, since it is blocked in
vm_access.The deadlock is always happening when strace needs to access the tracees process mmap, while another thread in the tracee starts to execve a child process, but that cannot continue until the
PTRACE_EVENT_EXITis handled and the WIFEXITED event is received:strace D 0 30614 30584
0x00000000Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 schedule_preempt_disabled+0x15/0x20 __mutex_lock.isra.13+0x1ec/0x520 __mutex_lock_killable_slowpath+0x13/0x20 mutex_lock_killable+0x28/0x30 mm_access+0x27/0xa0 process_vm_rw_core.isra.3+0xff/0x550 process_vm_rw+0xdd/0xf0 __x64_sys_process_vm_readv+0x31/0x40 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9expect D 0 31933 30876
0x80004003Call Trace: __schedule+0x3ce/0x6e0 schedule+0x5c/0xd0 flush_old_exec+0xc4/0x770 load_elf_binary+0x35a/0x16c0 search_binary_handler+0x97/0x1d0 __do_execve_file.isra.40+0x5d4/0x8a0 __x64_sys_execve+0x49/0x60 do_syscall_64+0x64/0x220 entry_SYSCALL_64_after_hwframe+0x44/0xa9
Quoted 8 lines
This changes
mm_accessto use the newexec_update_mutexinstead ofcred_guard_mutex.This patch is based on the following patch by Eric W. Biederman:
"[PATCH 0/5] Infrastructure to allow fixing exec deadlocks"Signed-off-by Bernd Edlinger <[email protected]>
Cool, yes, on top of the new infrastructure this looks correct to me -- the new mutex wraps mm changes and mm_access() is looking at *drum roll* the mm! :)
-Kees
Quoted 28 lines
--- kernel/fork.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/fork.c +2
-2@@ -1224,7 +1224,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)struct mm_struct *mm;int err; - err =mutex_lock_killable (&task->signal->cred_guard_mutex); + err =mutex_lock_killable (&task->signal->exec_update_mutex);if (err)return ERR_PTR (err);@@ -1234,7 +1234,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)mmput (mm); mm =ERR_PTR (-EACCES ); } -mutex_unlock (&task->signal->cred_guard_mutex); +mutex_unlock (&task->signal->exec_update_mutex);return mm; }Signature · 1 line
1.9.1
Signature · 1 line
Kees Cook
PATCH 2/4 selftests/ptrace: add test cases for dead-locks
In reply to Eric W. BiedermanThis adds test cases for ptrace deadlocks.
Additionally fixes a compile problem in get_syscall_info.c, observed with gcc-4.8.4:
get_syscall_info.c: In function 'get_syscall_info':
get_syscall_info.c:93:3: error: 'for' loop initial declarations are only
allowed in C99 mode
for (unsigned int i = 0; i < ARRAY_SIZE(args); ++i) {
^
get_syscall_info.c:93:3: note: use option -std=c99 or -std=gnu99 to compile
your code
--- tools/testing/selftests/ptrace/Makefile | 4 +- tools/testing/selftests/ptrace/vmaccess.c | 86 +++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.c
tools/testing/selftests/ptrace/Makefile +2 -2
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -iquote../../../../include/uapi -Wall +CFLAGS += -std=c99 -pthread -iquote../../../../include/uapi -Wall -TEST_GEN_PROGS := get_syscall_info peeksiginfo +TEST_GEN_PROGS := get_syscall_info peeksiginfo vmaccess include ../lib.mk
tools/testing/selftests/ptrace/vmaccess.c new file +86 -0
86 lines not on this page · raw message
Signature · 1 line
1.9.1
Re: [PATCH 2/4] selftests/ptrace: add test cases for dead-locks
Quoted 5 lines · Bernd Edlinger
This adds test cases for ptrace deadlocks.
Additionally fixes a compile problem in
get_syscall_info.c, observed with gcc-4.8.4:
Quoted 7 lines
get_syscall_info.c: In function 'get_syscall_info':
get_syscall_info.c:93:3: error: 'for' loop initial declarations are only
allowed in C99 modefor (unsigned int i = 0; i < ARRAY_SIZE(args); ++i) { ^get_syscall_info.c:93:3: note: use option -std=c99 or -std=gnu99 to compile
your code
*discomfort noises* (see below)
Quoted 13 lines
Signed-off-by Bernd Edlinger <[email protected]>--- tools/testing/selftests/ptrace/Makefile | 4 +- tools/testing/selftests/ptrace/vmaccess.c | 86 +++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/ptrace/vmaccess.ctools/testing/selftests/ptrace/Makefile +0
-0
tools/testing/selftests/ptrace/Makefile line 2
Quoted 3 lines
tools/testing/selftests/ptrace/Makefile +1
-1@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -iquote../../../../include/uapi -Wall +CFLAGS += -std=c99 -pthread -iquote../../../../include/uapi -Wall
This isn't the common solution in the kernel (the variable declaration would just be lifted out of the loop), but as it's selftest code, which does lots of special things ... I *guess* this is okay.
tools/testing/selftests/ptrace/Makefile line 4
Quoted 3 lines
tools/testing/selftests/ptrace/Makefile +1
-1-TEST_GEN_PROGS := get_syscall_info peeksiginfo +TEST_GEN_PROGS := get_syscall_info peeksiginfo vmaccess
I love having this deadlock test added to the selftests.
I think I need to make an improvement to the test harness, though, as the failure mode right now just blows up after the 30 second timeout and leaves this deadlocked:
$ ./vmaccess [==========] Running 2 tests from 1 test cases. [ RUN ] global.vmaccess Alarm clock $ ps PID TTY TIME CMD 2605 pts/0 00:00:00 bash 23360 pts/0 00:00:00 vmaccess 23361 pts/0 00:00:00 vmaccess 23363 pts/0 00:00:00 ps
But that's mostly unrelated to this code.
-Kees
Quoted 96 lines
tools/testing/selftests/ptrace/Makefile +0
-0include ../lib.mktools/testing/selftests/ptrace/vmaccess.c new file +86
-0@@ -0,0 +1,86 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* +* Copyright (c) 2020 Bernd Edlinger <[email protected]> +* All rights reserved. +* +* Check whether /proc/$pid/mem can be accessed without causing deadlocks +* when de_thread is blocked with ->cred_guard_mutex held. +*/ + +#include "../kselftest_harness.h" +#include <stdio.h> +#include <fcntl.h> +#include <pthread.h> +#include <signal.h> +#include <unistd.h> +#include <sys/ptrace.h> + +static void *thread (void *arg) +{ +ptrace (PTRACE_TRACEME ,0 ,0L ,0L ); +return NULL ; +} + +TEST (vmaccess) +{ +int f, pid =fork (); +char mm[64 ]; + +if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("true" ,"true" ,NULL ); + } + +sleep (1 ); +sprintf (mm,"/proc/%d/mem" , pid); + f =open (mm,O_RDONLY ); +ASSERT_GE (f,0 ); +close (f); + f =kill (pid,SIGCONT ); +ASSERT_EQ (f,0 ); +} + +TEST (attach) +{ +int s, k, pid =fork (); + +if (!pid) { +pthread_t pt; + +pthread_create (&pt,NULL , thread,NULL ); +pthread_join (pt,NULL ); +execlp ("sleep" ,"sleep" ,"2" ,NULL ); + } + +sleep (1 ); + k =ptrace (PTRACE_ATTACH , pid,0L ,0L ); +ASSERT_EQ (errno,EAGAIN ); +ASSERT_EQ (k,-1 ); + k =waitpid (-1 , &s,WNOHANG ); +ASSERT_NE (k,-1 ); +ASSERT_NE (k,0 ); +ASSERT_NE (k, pid); +ASSERT_EQ (WIFEXITED (s),1 ); +ASSERT_EQ (WEXITSTATUS (s),0 ); +sleep (1 ); + k =ptrace (PTRACE_ATTACH , pid,0L ,0L ); +ASSERT_EQ (k,0 ); + k =waitpid (-1 , &s,0 ); +ASSERT_EQ (k, pid); +ASSERT_EQ (WIFSTOPPED (s),1 ); +ASSERT_EQ (WSTOPSIG (s),SIGSTOP ); + k =ptrace (PTRACE_DETACH , pid,0L ,0L ); +ASSERT_EQ (k,0 ); + k =waitpid (-1 , &s,0 ); +ASSERT_EQ (k, pid); +ASSERT_EQ (WIFEXITED (s),1 ); +ASSERT_EQ (WEXITSTATUS (s),0 ); + k =waitpid (-1 ,NULL ,0 ); +ASSERT_EQ (k,-1 ); +ASSERT_EQ (errno,ECHILD ); +} + +TEST_HARNESS_MAIN Signature · 1 line
1.9.1
Signature · 1 line
Kees Cook
Re: [PATCH 2/4] selftests/ptrace: add test cases for dead-locks
In reply to Bernd EdlingerQuoted 12 lines · Bernd Edlinger
This adds test cases for ptrace deadlocks.
Additionally fixes a compile problem in
get_syscall_info.c, observed with gcc-4.8.4:
get_syscall_info.c: In function 'get_syscall_info':
get_syscall_info.c:93:3: error: 'for' loop initial declarations are only
allowed in C99 modefor (unsigned int i = 0; i < ARRAY_SIZE(args); ++i) { ^get_syscall_info.c:93:3: note: use option -std=c99 or -std=gnu99 to compile
your code
[...]
tools/testing/selftests/ptrace/Makefile line 2
Quoted 4 lines
tools/testing/selftests/ptrace/Makefile +1
-1@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS += -iquote../../../../include/uapi -Wall +CFLAGS += -std=c99 -pthread -iquote../../../../include/uapi -Wall
Wouldn't it be better to choose -std=gnu99 over -std=c99?
Signature · 1 line
ldv
PATCH 3/4 mm: docs: Fix a comment in process_vm_rw_core
In reply to Eric W. BiedermanThis removes a duplicate "a" in the comment in process_vm_rw_core.
--- mm/process_vm_access.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/process_vm_access.c +1 -1
@@ -204,7 +204,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter,if (!mm ||IS_ERR (mm)) { rc =IS_ERR (mm) ?PTR_ERR (mm) : -ESRCH ;/* -* Explicitly map EACCES to EPERM as EPERM is a more a +* Explicitly map EACCES to EPERM as EPERM is a more * appropriate error code for process_vw_readv/writev */ if (rc == -EACCES )
Signature · 1 line
1.9.1
Re: [PATCH 3/4] mm: docs: Fix a comment in process_vm_rw_core
Quoted 3 lines · Bernd Edlinger
This removes a duplicate "a" in the comment in
process_vm_rw_core.Signed-off-by Bernd Edlinger <[email protected]>
-Kees
Quoted 19 lines
--- mm/process_vm_access.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/process_vm_access.c +1
-1@@ -204,7 +204,7 @@ static ssize_t process_vm_rw_core(pid_t pid, struct iov_iter *iter,if (!mm ||IS_ERR (mm)) { rc =IS_ERR (mm) ?PTR_ERR (mm) : -ESRCH ;/* -* Explicitly map EACCES to EPERM as EPERM is a more a +* Explicitly map EACCES to EPERM as EPERM is a more * appropriate error code for process_vw_readv/writev */ if (rc == -EACCES )Signature · 1 line
1.9.1
Signature · 1 line
Kees Cook
PATCH 4/4 kernel: doc: remove outdated comment cred.c
In reply to Eric W. BiedermanThis removes an outdated comment in prepare_kernel_cred.
There is no "cred_replace_mutex" any more, so the comment must go away.
--- kernel/cred.c | 2 -- 1 file changed, 2 deletions(-)
kernel/cred.c +0 -2
@@ -675,8 +675,6 @@ void __init cred_init(void)* The caller may change these controls afterwards if desired. * * Returns the new credentials or NULL if out of memory. -* -* Does not take, and does not return holding current->cred_replace_mutex. */ struct cred *prepare_kernel_cred (struct task_struct *daemon) {
Signature · 1 line
1.9.1
Re: [PATCH 4/4] kernel: doc: remove outdated comment cred.c
Quoted 6 lines · Bernd Edlinger
This removes an outdated comment in
prepare_kernel_cred.There is no "
cred_replace_mutex" any more, so the comment must go away.Signed-off-by Bernd Edlinger <[email protected]>
-Kees
Quoted 19 lines
--- kernel/cred.c | 2 -- 1 file changed, 2 deletions(-)kernel/cred.c +0
-2@@ -675,8 +675,6 @@ void __init cred_init(void)* The caller may change these controls afterwards if desired. * * Returns the new credentials or NULL if out of memory. -* -* Does not take, and does not return holding current->cred_replace_mutex. */ struct cred *prepare_kernel_cred (struct task_struct *daemon) {Signature · 1 line
1.9.1
Signature · 1 line
Kees Cook
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
In reply to Eric W. BiedermanQuoted 8 lines · Eric W. Biederman
Quoted 6 lines · Bernd Edlinger
Quoted 4 lines · Eric W. Biederman
Does that sound better?
Quoted 8 lines
Quoted 1 line
almost done.
I think this text is finally clean.
exec: Add
exec_update_mutexto replacecred_guard_mutexThe
cred_guard_mutexis problematic as it is held over possibly indefinite waits for userspace. The possilbe indefinite waits for
-------------------------------------------^^^^^^^^ possible?
Signature · 1 line
ldv
Re: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
Quoted 11 lines · Dmitry V. Levin
Quoted 10 lines · Eric W. Biederman
Quoted 7 lines · Bernd Edlinger
Quoted 4 lines · Eric W. Biederman
Does that sound better?
almost done.
Quoted 8 lines
Quoted 6 lines
I think this text is finally clean.
exec: Add
exec_update_mutexto replacecred_guard_mutexThe
cred_guard_mutexis problematic as it is held over possibly indefinite waits for userspace. The possilbe indefinite waits for-------------------------------------------^^^^^^^^ possible?
Yes. Thank you. Fixed.
Eric
Kees Cook
Reviewed-byRe: [PATCH v2 5/5] exec: Add a exec_update_mutex to replace cred_guard_mutex
In reply to Eric W. BiedermanQuoted 9 lines · Eric W. Biederman
exec: Add
exec_update_mutexto replacecred_guard_mutexThe
cred_guard_mutexis problematic as it is held over possibly indefinite waits for userspace. The possilbe indefinite waits for userspace that I have identified are: Thecred_guard_mutexis held inPTRACE_EVENT_EXITwaiting for the tracer. Thecred_guard_mutexis held over "put_user(0, tsk->clear_child_tid)" inexit_mm(). Thecred_guard_mutexis held over "get_user(futex_offset, ...")inexit_robust_list. Thecred_guard_mutexheld overcopy_strings.
I suspect you're not trying to make a comprehensive list here, but do you want to mention seccomp too (since it's yet another weird case).
Quoted 5 lines
[...]
Holding a mutex over any of those possibly indefinite waits for userspace does not appear necessary. Addexec_update_mutexthat will just cover updating the process during exec where the permissions and the objects pointed to by the task struct may be out of sync.
Should the specific resources be pointed out here? creds, mm, ... ?
But otherwise, yup, looks sane:
Signature · 1 line
Kees Cook