kernel: release ptraced tasks before zap_pid_ns_processes
PATCH v2 1/1 kernel: release ptraced tasks before zap_pid_ns_processes
Currently, exit_ptrace() adds all ptraced tasks in a dead list, than zap_pid_ns_processes() waits all tasks in a current pidns, and only then tasks from the dead list are released.
zap_pid_ns_processes() can stuck on waiting tasks from the dead list. In this case, we will have one unkillable process with one or more dead children.
Thanks to Oleg for the advice to release tasks in find_child_reaper().
---
v2: Oleg showed that ptraced tasks can be released in find_child_reaper(). This allows to avoid additional write_lock/unlock(tasklist), and another list_for_each_entry_safe(dead) loop is called only if it is actually needed.
kernel/exit.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
kernel/exit.c +10 -2
@@ -558,12 +558,14 @@ static struct task_struct *find_alive_thread(struct task_struct *p)return NULL ; } -static struct task_struct *find_child_reaper (struct task_struct *father) +static struct task_struct *find_child_reaper (struct task_struct *father, +struct list_head *dead) __releases(&tasklist_lock) __acquires(&tasklist_lock) {struct pid_namespace *pid_ns =task_active_pid_ns (father);struct task_struct *reaper = pid_ns->child_reaper; +struct task_struct *p, *n;if (likely (reaper != father))return reaper;
@@ -579,6 +581,12 @@ static struct task_struct *find_child_reaper(struct task_struct *father)panic ("Attempted to kill init! exitcode=0x%08x\n" , father->signal->group_exit_code ?: father->exit_code); } + +list_for_each_entry_safe (p, n, dead, ptrace_entry) { +list_del_init (&p->ptrace_entry); +release_task (p); + } +zap_pid_ns_processes (pid_ns);write_lock_irq (&tasklist_lock);
@@ -668,7 +676,7 @@ static void forget_original_parent(struct task_struct *father,exit_ptrace (father, dead);/* Can drop and reacquire tasklist_lock */ - reaper =find_child_reaper (father); + reaper =find_child_reaper (father, dead);if (list_empty (&father->children))return ;
Signature · 1 line
2.17.2
Quoted 8 lines · On Thu, 10 Jan 2019 09:52:00 -0800 Andrei Vagin <[email protected]> wrote:
Currently,
exit_ptrace()adds all ptraced tasks in a dead list, thanzap_pid_ns_processes()waits all tasks in a current pidns, and only then tasks from the dead list are released.
zap_pid_ns_processes()can stuck on waiting tasks from the dead list. In this case, we will have one unkillable process with one or more dead children.
Quoted 8 lines
Thanks to Oleg for the advice to release tasks in
find_child_reaper().Fixes 7c8bd2322c7f ("exit: ptrace: shift "reap dead" code from exit_ptrace() to forget_original_parent()")Cc "Eric W. Biederman" <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Oleg Nesterov <[email protected]>Signed-off-by Andrei Vagin <[email protected]>
Does this warrant a -stable backport? 7c8bd2322c7f was 4+ years ago...
Quoted 9 lines · Andrew Morton
Quoted 7 lines
Fixes 7c8bd2322c7f ("exit: ptrace: shift "reap dead" code from exit_ptrace() to forget_original_parent()")Cc "Eric W. Biederman" <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Oleg Nesterov <[email protected]>Signed-off-by Andrei Vagin <[email protected]>Does this warrant a -stable backport?
7c8bd2322c7fwas 4+ years ago...
Agreed, the problem is trivially reproducible.
Oleg.
Quoted 47 lines · Andrei Vagin
Currently,
exit_ptrace()adds all ptraced tasks in a dead list, thanzap_pid_ns_processes()waits all tasks in a current pidns, and only then tasks from the dead list are released.
zap_pid_ns_processes()can stuck on waiting tasks from the dead list. In this case, we will have one unkillable process with one or more dead children.Thanks to Oleg for the advice to release tasks in
find_child_reaper().Fixes 7c8bd2322c7f ("exit: ptrace: shift "reap dead" code from exit_ptrace() to forget_original_parent()")Cc "Eric W. Biederman" <[email protected]>Cc Andrew Morton <[email protected]>Signed-off-by Oleg Nesterov <[email protected]>Signed-off-by Andrei Vagin <[email protected]>---
v2: Oleg showed that ptraced tasks can be released in
find_child_reaper(). This allows to avoid additional write_lock/unlock(tasklist), and anotherlist_for_each_entry_safe(dead)loop is called only if it is actually needed.kernel/exit.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)kernel/exit.c +3
-1@@ -558,12 +558,14 @@ static struct task_struct *find_alive_thread(struct task_struct *p)return NULL ; } -static struct task_struct *find_child_reaper (struct task_struct *father) +static struct task_struct *find_child_reaper (struct task_struct *father, +struct list_head *dead) __releases(&tasklist_lock) __acquires(&tasklist_lock) {struct pid_namespace *pid_ns =task_active_pid_ns (father);struct task_struct *reaper = pid_ns->child_reaper; +struct task_struct *p, *n;if (likely (reaper != father))return reaper;
kernel/exit.c static struct task_struct *find_child_reaper(struct task_struct *father) · line 589
Quoted 9 lines
kernel/exit.c +6
-0@@ -579,6 +581,12 @@ static struct task_struct *find_child_reaper(struct task_struct *father)panic ("Attempted to kill init! exitcode=0x%08x\n" , father->signal->group_exit_code ?: father->exit_code); } + +list_for_each_entry_safe (p, n, dead, ptrace_entry) { +list_del_init (&p->ptrace_entry); +release_task (p); + } +
Hi,
from our (SUSE) QA we received a report that this patch causes a performance decline in libmicro pthread_* benchmark as reported in:
https://bugzilla.suse.com/show_bug.cgi?id=1126762
I tried myself from the repo:
https://github.com/redhat-performance/libMicro
I ranpthread_create -B 8 -C 200 -S
and with the patch applied:
# STATISTICS usecs/call (raw) usecs/call (outliers removed)
# mean 23.38611 17.29311
Without:
# mean 41.36539 39.21347
The values vary, but they are around 23 and 42, respectively.
The benchmark seems to create 8 (-B above) pthreads, does lock/unlock in them and then the threads exit. The benchmark reaps the threads via pthread_join. This all happens 200 times (-C above).
Any idea how to restore the performance close to the previous state?
kernel/exit.c static void forget_original_parent(struct task_struct *father, · line 681
Quoted 11 lines
kernel/exit.c +1
-1zap_pid_ns_processes (pid_ns);write_lock_irq (&tasklist_lock);@@ -668,7 +676,7 @@ static void forget_original_parent(struct task_struct *father,exit_ptrace (father, dead);/* Can drop and reacquire tasklist_lock */ - reaper =find_child_reaper (father); + reaper =find_child_reaper (father, dead);if (list_empty (&father->children))return ;
thanks,
Signature · 2 lines
js suse labs
Quoted 33 lines · Jiri Slaby
Quoted 19 lines · Andrei Vagin
kernel/exit.c +3
-1@@ -558,12 +558,14 @@ static struct task_struct *find_alive_thread(struct task_struct *p)return NULL ; } -static struct task_struct *find_child_reaper (struct task_struct *father) +static struct task_struct *find_child_reaper (struct task_struct *father, +struct list_head *dead) __releases(&tasklist_lock) __acquires(&tasklist_lock) {struct pid_namespace *pid_ns =task_active_pid_ns (father);struct task_struct *reaper = pid_ns->child_reaper; +struct task_struct *p, *n;if (likely (reaper != father))return reaper;kernel/exit.c static struct task_struct *find_child_reaper(struct task_struct *father) · line 589
Quoted 9 lines
kernel/exit.c +6
-0@@ -579,6 +581,12 @@ static struct task_struct *find_child_reaper(struct task_struct *father)panic ("Attempted to kill init! exitcode=0x%08x\n" , father->signal->group_exit_code ?: father->exit_code); } + +list_for_each_entry_safe (p, n, dead, ptrace_entry) { +list_del_init (&p->ptrace_entry); +release_task (p); + } +Hi,
Quoted 3 lines
from our (SUSE) QA we received a report that this patch causes a performance decline in libmicro pthread_* benchmark as reported in:
https://bugzilla.suse.com/show_bug.cgi?id=1126762
Access Denied
Quoted 6 lines
I tried myself from the repo:
https://github.com/redhat-performance/libMicroI ran
pthread_create-B 8 -C 200 -S
Quoted 6 lines
and with the patch applied:
# STATISTICS usecs/call (raw) usecs/call (outliers removed)
# mean 23.38611 17.29311Without:
# mean 41.36539 39.21347
can't reproduce, I see the same numbers with or without this patch. However, I did "./bin/pthread_create -B 8 -C 200 -S" under KVM.
Quoted 3 lines
The benchmark seems to create 8 (-B above) pthreads, does lock/unlock in them and then the threads exit. The benchmark reaps the threads via
pthread_join. This all happens 200 times (-C above).
Given that this test-case doesn't use CLONE_PID, I fail to understand how this patch can make any noticeable difference performance wise...
with this patch forget_original_parent() just passes the additional argument to find_child_reaper(), nothing else.
The extra list_for_each_entry_safe/release_task loop can't happen, and even if it could it shouldn't cause any performance regression too.
Quoted 1 line
Any idea how to restore the performance close to the previous state?
maybe you can try perf to find out where does this difference come from?
Oleg.
Quoted 16 lines · Oleg Nesterov
Quoted 3 lines
from our (SUSE) QA we received a report that this patch causes a performance decline in libmicro pthread_* benchmark as reported in:
https://bugzilla.suse.com/show_bug.cgi?id=1126762Access Denied
Quoted 10 lines
I tried myself from the repo:
https://github.com/redhat-performance/libMicroI ran
pthread_create-B 8 -C 200 -Sand with the patch applied:
# STATISTICS usecs/call (raw) usecs/call (outliers removed)
# mean 23.38611 17.29311
Quoted 5 lines
Quoted 2 lines
Without:
# mean 41.36539 39.21347can't reproduce, I see the same numbers with or without this patch. However, I did "./bin/pthread_create -B 8 -C 200 -S" under KVM.
Correct. I did the tests also under KVM. The above difference happens when you compare (by stupid mistake) a LOCKDEP and non-LOCKDEP kernel -- sorry. So the proper results are comparably the same -- the difference is on the noise level. So it must be their setup broken too.
Thanks for the input anyway.
Signature · 2 lines
js suse labs