is_current_single_threaded: don't use ->mmap_sem
PATCH 2/2 is_current_single_threaded: don't use ->mmap_sem
is_current_single_threaded() can safely miss a freshly forked CLONE_VM task, but in this case it must not miss its parent. That is why we take mm->mmap_sem for writing to make sure a thread/task with the same ->mm can't pass exit_mm() and disappear.
However we can avoid ->mmap_sem and rely on rcu/barriers:
- if we do not see the exiting parent on thread/process list we see the result of list_del_rcu(), in this case we must also see the result of list_add_rcu() which does wmb().
- if we do see the parent but its ->mm == NULL, we need rmb() to make sure we can't miss the child.
---
lib/is_single_threaded.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
lib/is_single_threaded.c renamed from lib/is_single_threaded.c~ISS_2_RCU +6 -4
@@ -22,8 +22,6 @@ bool is_current_single_threaded(void)struct task_struct *p, *t;bool ret; -might_sleep (); -if (atomic_read (&task->signal->live) !=1 )return false;
@@ -31,7 +29,6 @@ bool is_current_single_threaded(void)return true; ret = false; -down_write (&mm->mmap_sem);rcu_read_lock ();for_each_process (p) {if (unlikely (p->flags &PF_KTHREAD ))
@@ -45,12 +42,17 @@ bool is_current_single_threaded(void) goto found;if (likely (t->mm))break ; +/* +* t->mm == NULL. Make sure next_thread/next_task +* will see other CLONE_VM tasks which might be +* forked before exiting. +*/ +smp_rmb (); }while_each_thread (p, t); } ret = true; found:rcu_read_unlock (); -up_write (&mm->mmap_sem);return ret; }
Quoted 6 lines · Oleg Nesterov
is_current_single_threaded()can safely miss a freshly forkedCLONE_VMtask, but in this case it must not miss its parent. That is why we takemm->mmap_semfor writing to make sure a thread/task with the same ->mm can't passexit_mm()and disappear.
Quoted 8 lines
However we can avoid ->mmap_sem and rely on rcu/barriers:
- if we do not see the exiting parent on thread/process list we see the result of
list_del_rcu(), in this case we must also see the result oflist_add_rcu()which doeswmb().- if we do see the parent but its ->mm == NULL, we need
rmb()to make sure we can't miss the child.
To simplify the review, below is the code after the patch.
Oleg.
/*
* Returns true if the task does not share ->mm with another thread/process. */
bool is_current_single_threaded(void)
{
struct task_struct *task = current;
struct mm_struct *mm = task->mm;
struct task_struct *p, *t;
bool ret;
if (atomic_read(&task->signal->live) != 1)
return false;
if (atomic_read(&mm->mm_users) == 1)
return true;
ret = false;
rcu_read_lock();
for_each_process(p) {
if (unlikely(p->flags & PF_KTHREAD))
continue;
if (unlikely(p == task->group_leader))
continue;
t = p;
do {
if (unlikely(t->mm == mm))
goto found;
if (likely(t->mm))
break;
/*
* t->mm == NULL. Make sure next_thread/next_task
* will see other CLONE_VM tasks which might be
* forked before exiting.
*/
smp_rmb();
} while_each_thread(p, t);
}
ret = true;
found:
rcu_read_unlock();
return ret;
}