linux-user/syscall.c: Prevent acquiring clone_lock while fork()
PATCH 1/1 linux-user/syscall.c: Prevent acquiring clone_lock while fork()
By the spec, fork() copies only the thread which executes it. So it may happen, what while one thread is doing a fork, another thread is holding clone_lock mutex
(e.g. doing a fork() or exit()).
So the child process is born with the mutex being held, and there are nobody to release it.
As the thread executing do_syscall() is not considered running, start_exclusive() does not protect us from the case.
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3226
--- linux-user/main.c | 2 ++ linux-user/syscall.c | 14 ++++++++++++++ linux-user/user-internals.h | 2 ++ 3 files changed, 18 insertions(+)
linux-user/main.c +2 -0
@@ -145,6 +145,7 @@ unsigned long guest_stack_size = TARGET_DEFAULT_STACK_SIZE;void fork_start (void ) {start_exclusive (); +clone_fork_start ();mmap_fork_start ();cpu_list_lock ();qemu_plugin_user_prefork_lock ();
@@ -174,6 +175,7 @@ void fork_end(pid_t pid)cpu_list_unlock (); }gdbserver_fork_end (thread_cpu, pid); +clone_fork_end (child);/* * qemu_init_cpu_list() reinitialized the child exclusive state, but we * also need to keep current_cpu consistent, so call end_exclusive() for
linux-user/syscall.c +14 -0
@@ -6856,6 +6856,20 @@ static void *clone_func(void *arg)return NULL ; } +void clone_fork_start (void ) +{ +pthread_mutex_lock (&clone_lock); +} + +void clone_fork_end (bool child) +{ +if (child) { +pthread_mutex_init (&clone_lock,NULL ); + }else { +pthread_mutex_unlock (&clone_lock); + } +} +/* do_fork() Must return host values and target errnos (unlike most do_*() functions). */ static int do_fork(CPUArchState *env, unsigned int flags, abi_ulong newsp,
linux-user/user-internals.h +2 -0
@@ -69,6 +69,8 @@ abi_long get_errno(abi_long ret);const char *target_strerror (int err);int get_osversion (void );void init_qemu_uname_release (void ); +void clone_fork_start (void ); +void clone_fork_end (bool child);void fork_start (void );void fork_end (pid_t pid);
Signature · 1 line
2.52.0
Aleksandr Sergeev <[email protected]> writes:
(Adding Paolo/Richard to Cc for exclusive discussion)
Quoted 9 lines
By the spec,
fork()copies only the thread which executes it. So it may happen, what while one thread is doing a fork, another thread is holdingclone_lockmutex
(e.g. doing afork()orexit()).
So the child process is born with the mutex being held, and there are nobody to release it.As the thread executing
do_syscall()is not considered running,start_exclusive()does not protect us from the case.
I see the logic of what you say as:
/* Write pending_cpus before reading other_cpu->running. */
smp_mb();
running_cpus = 0;
CPU_FOREACH(other_cpu) {
if (qatomic_read(&other_cpu->running)) {
other_cpu->has_waiter = true;
running_cpus++;
qemu_cpu_kick(other_cpu);
}
}but...
Quoted 41 lines
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3226
Signed-off-by Aleksandr Sergeev <[email protected]>--- linux-user/main.c | 2 ++ linux-user/syscall.c | 14 ++++++++++++++ linux-user/user-internals.h | 2 ++ 3 files changed, 18 insertions(+)linux-user/main.c +2
-0@@ -145,6 +145,7 @@ unsigned long guest_stack_size = TARGET_DEFAULT_STACK_SIZE;void fork_start (void ) {start_exclusive (); +clone_fork_start ();mmap_fork_start ();cpu_list_lock ();qemu_plugin_user_prefork_lock ();@@ -174,6 +175,7 @@ void fork_end(pid_t pid)cpu_list_unlock (); }gdbserver_fork_end (thread_cpu, pid); +clone_fork_end (child);/* * qemu_init_cpu_list() reinitialized the child exclusive state, but we * also need to keep current_cpu consistent, so call end_exclusive() for linux-user/syscall.c +4
-0@@ -6856,6 +6856,20 @@ static void *clone_func(void *arg)return NULL ; } +void clone_fork_start (void ) +{ +pthread_mutex_lock (&clone_lock); +}
linux-user/syscall.c static void *clone_func(void *arg) · line 6872
Quoted 10 lines
linux-user/syscall.c +10
-0+ +void clone_fork_end (bool child) +{ +if (child) { +pthread_mutex_init (&clone_lock,NULL ); + }else { +pthread_mutex_unlock (&clone_lock); + } +} +
This looks like it should just be open-coded in the:
if (flags & CLONE_VM) {
}leg of do_fork()?
Or maybe I'm missing the subtly of the race condition?
Quoted 15 lines
linux-user/syscall.c +0
-0/* do_fork() Must return host values and target errnos (unlike most do_*() functions). */ static int do_fork(CPUArchState *env, unsigned int flags, abi_ulong newsp,linux-user/user-internals.h +2
-0@@ -69,6 +69,8 @@ abi_long get_errno(abi_long ret);const char *target_strerror (int err);int get_osversion (void );void init_qemu_uname_release (void ); +void clone_fork_start (void ); +void clone_fork_end (bool child);void fork_start (void );void fork_end (pid_t pid);
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
Quoted 33 lines · Alex Bennée
Aleksandr Sergeev <[email protected]> writes:
(Adding Paolo/Richard to Cc for exclusive discussion)
Quoted 15 lines
By the spec,
fork()copies only the thread which executes it. So it may happen, what while one thread is doing a fork, another thread is holdingclone_lockmutex
(e.g. doing afork()orexit()).
So the child process is born with the mutex being held, and there are nobody to release it.As the thread executing
do_syscall()is not considered running,start_exclusive()does not protect us from the case.linux-user/syscall.c +4
-0+void clone_fork_start (void ) +{ +pthread_mutex_lock (&clone_lock); +}linux-user/syscall.c static void *clone_func(void *arg) · line 6872
Quoted 10 lines
linux-user/syscall.c +10
-0+ +void clone_fork_end (bool child) +{ +if (child) { +pthread_mutex_init (&clone_lock,NULL ); + }else { +pthread_mutex_unlock (&clone_lock); + } +} +This looks like it should just be open-coded in the:
Quoted 7 lines
if (flags & CLONE_VM) { }leg of
do_fork()?Or maybe I'm missing the subtly of the race condition?
It's a general thing with fork() -- we need to make sure we hold all the mutexes in QEMU before forking, so that when we do the fork we know the state of the data structures they are protecting. Then after the fork the parent unlocks the mutexes again and the child can re-init the mutex.
We don't want to open-code this because there are quite a lot of things to handle and do_fork() isalready pretty long and complicated. Instead we have the fork_start() and fork_end() functions which sequence all the things that need to be handled across the fork, including the mutex lock/unlock stuff.
I think this patch is doing it the right way, by adding this mutex to the set of things to be handled across fork. Compare mmap_fork_start() and mmap_fork_end() which do this for the mmap_mutex.
-- PMM
Quoted 10 lines · Aleksandr Sergeev
By the spec,
fork()copies only the thread which executes it. So it may happen, what while one thread is doing a fork, another thread is holdingclone_lockmutex
(e.g. doing afork()orexit()).
So the child process is born with the mutex being held, and there are nobody to release it.As the thread executing
do_syscall()is not considered running,start_exclusive()does not protect us from the case.
Quoted 7 lines
Resolves:https://gitlab.com/qemu-project/qemu/-/issues/3226
Signed-off-by Aleksandr Sergeev<[email protected]>--- linux-user/main.c | 2 ++ linux-user/syscall.c | 14 ++++++++++++++ linux-user/user-internals.h | 2 ++ 3 files changed, 18 insertions(+)
r~
Richard Henderson
AppliedQuoted 19 lines · Richard Henderson
Quoted 10 lines · Aleksandr Sergeev
By the spec,
fork()copies only the thread which executes it. So it may happen, what while one thread is doing a fork, another thread is holdingclone_lockmutex
(e.g. doing afork()orexit()).
So the child process is born with the mutex being held, and there are nobody to release it.As the thread executing
do_syscall()is not considered running,start_exclusive()does not protect us from the case.Quoted 7 lines
Resolves:https://gitlab.com/qemu-project/qemu/-/issues/3226
Signed-off-by Aleksandr Sergeev<[email protected]>---
linux-user/main.c | 2 ++
linux-user/syscall.c | 14 ++++++++++++++
linux-user/user-internals.h| 2 ++3 files changed, 18 insertions(+)
Quoted 1 line
Reviewed-by Richard Henderson <[email protected]>
Queued, thanks.
r~
Quoted 7 lines · Aleksandr Sergeev
By the spec,
fork()copies only the thread which executes it. So it may happen, what while one thread is doing a fork, another thread is holdingclone_lockmutex
(e.g. doing afork()orexit()).
So the child process is born with the mutex being held, and there are nobody to release it.
Quoted 5 lines
As the thread executing
do_syscall()is not considered running,start_exclusive()does not protect us from the case.Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3226
Signed-off-by Aleksandr Sergeev <[email protected]>
This feels like qemu-stable material. I'm picking this change for current stable releases. Please let me know if I shouldn't.
Thanks,
/mjt