prctl: add PR_{SET,GET}_CHILD_SUBREAPER to allow simple process supervision

Resending this, it got lost last year's September.

We still need it to properly implement init-like service managers.

From: Lennart Poettering <[email protected]>
Subject: prctl: add PR_{SET,GET}_CHILD_SUBREAPER to allow simple process supervision

Users of this will be the systemd per-user instance, which provides init-like functionality for the user's login session and D-Bus, which activates bus services on-demand. Both need init-like capabilities to be able to properly keep track of the services they start.

include/linux/sched.h +1 -0

@@ -552,6 +552,18 @@ struct signal_struct {
 	int			group_stop_count;
 	unsigned int		flags; /* see SIGNAL_* flags below */
 
+	/*

Quoted 18 lines · Kay Sievers

Resending this, it got lost last year's September.

We still need it to properly implement init-like service managers.

From: Lennart Poettering <[email protected]>
Subject: prctl: add PR_{SET,GET}_CHILD_SUBREAPER to allow simple process supervision

Users of this will be the systemd per-user instance, which provides init-like functionality for the user's login session and D-Bus, which activates bus services on-demand. Both need init-like capabilities to be able to properly keep track of the services they start.

include/linux/sched.h +0 -0

@@ -552,6 +552,18 @@ struct signal_struct {
 	int			group_stop_count;
 	unsigned int		flags; /* see SIGNAL_* flags below */
 

include/linux/sched.h struct signal_struct { · line 565

Quoted 11 lines

include/linux/sched.h +11 -0

+	/*
+	 * PR_SET_CHILD_SUBREAPER marks a process, like a service
+	 * manager, to re-parent orphan (double-forking) child processes
+	 * to this process instead of 'init'. The service manager is
+	 * able to receive SIGCHLD signals and is able to investigate
+	 * the process until it calls wait(). All children of this
+	 * process will inherit a flag if they should look for a
+	 * child_subreaper process at exit.
+	 */
+	unsigned int		is_child_subreaper:1;
+	unsigned int		has_child_subreaper:1;

Is there someplace we can stick these two fields where they won't expand the signal_struct? Can we stick them in signal_struct->flags instead?

include/linux/sched.h struct signal_struct { · line 569

Quoted 4 lines

include/linux/sched.h +4 -0

+		/* find the first ancestor marked as child_subreaper */
+		for (reaper = father->real_parent;
+		     reaper != &init_task;
+		     reaper = reaper->real_parent) {

I admit being insufficiently caffienated - does this DTRT in a PID namespace? That &init_task looks fishy to me...

Quoted 7 lines

Review

Quoted 4 lines

(quoted diff) +4 -0

+		/* find the first ancestor marked as child_subreaper */
+		for (reaper = father->real_parent;
+		     reaper != &init_task;
+		     reaper = reaper->real_parent) {

I admit being insufficiently caffienated - does this DTRT in a PID namespace? That &init_task looks fishy to me...

Probably this needs a comment. Initially I was confused too.

Note that the code below checks same_thread_group(reaper, pid_ns->child_reaper), this is what we need to DTRT in a PID namespace. However we still need the check above, see http://marc.info/?l=linux-kernel&m=131385460420380

Quoted 5 lines · On Mon, 9 Jan 2012 16:07:01 +0100 Oleg Nesterov <[email protected]> wrote:

Quoted 5 lines

Quoted 4 lines

+         /* find the first ancestor marked as child_subreaper */
+         for (reaper = father->real_parent;
+              reaper != &init_task;
+              reaper = reaper->real_parent) {

Quoted 8 lines

Quoted 2 lines

I admit being insufficiently caffienated - does this DTRT in a PID namespace? That &init_task looks fishy to me...

Probably this needs a comment. Initially I was confused too.

Note that the code below checks same_thread_group(reaper, pid_ns->child_reaper), this is what we need to DTRT in a PID namespace. However we still need the check above, see http://marc.info/?l=linux-kernel&m=131385460420380

In light of Kay's haughty silence, I did this:

kernel/exit.c renamed from kernel/exit.c~prctl-add-pr_setget_child_subreaper-to-allow-simple-process-supervision-fix-fix +0 -1

@@ -724,7 +724,13 @@ static struct task_struct *find_new_reap
        } else if (father->signal->has_child_subreaper) {
                struct task_struct *reaper;
 
-               /* find the first ancestor marked as child_subreaper */

Resending this, it got lost last year's September.

We still need it to properly implement init-like service managers.

Andrew, care to pick this up again? The issues raised the last year are all expected to be fixed.

Thanks,
Kay

From: Lennart Poettering <[email protected]>
Subject: prctl: add PR_{SET,GET}_CHILD_SUBREAPER to allow simple process supervision

Userspace service managers/supervisors need to track their started services. Many services daemonize by double-forking and get implicitly re-parented to PID 1. The service manager will no longer be able to receive the SIGCHLD signals for them, and is no longer in charge of reaping the children with wait(). All information about the children is lost at the moment PID 1 cleans up the re-parented processes.

With this prctl, a service manager process can mark itself as a sort of 'sub-init', able to stay as the parent for all orphaned processes created by the started services. All SIGCHLD signals will be delivered to the service manager.

Receiving SIGCHLD and doing wait() is in cases of a service-manager much preferred over any possible asynchronous notification about specific PIDs, because the service manager has full access to the child process data in /proc and the PID can not be re-used until the wait(), the service-manager itself is in charge of, has happened.

As a side effect, the relevant parent PID information does not get lost by a double-fork, which results in a more elaborate process tree and 'ps' output:

before:

 # ps afx
 253 ?        Ss     0:00 /bin/dbus-daemon --system --nofork
 294 ?        Sl     0:00 /usr/libexec/polkit-1/polkitd
 328 ?        S      0:00 /usr/sbin/modem-manager
 608 ?        Sl     0:00 /usr/libexec/colord
 658 ?        Sl     0:00 /usr/libexec/upowerd
 819 ?        Sl     0:00 /usr/libexec/imsettings-daemon
 916 ?        Sl     0:00 /usr/libexec/udisks-daemon
 917 ?        S      0:00  \_ udisks-daemon: not polling any devices

after:

 # ps afx
 294 ?        Ss     0:00 /bin/dbus-daemon --system --nofork
 426 ?        Sl     0:00  \_ /usr/libexec/polkit-1/polkitd
 449 ?        S      0:00  \_ /usr/sbin/modem-manager
 635 ?        Sl     0:00  \_ /usr/libexec/colord
 705 ?        Sl     0:00  \_ /usr/libexec/upowerd
 959 ?        Sl     0:00  \_ /usr/libexec/udisks-daemon
 960 ?        S      0:00  |   \_ udisks-daemon: not polling any devices
 977 ?        Sl     0:00  \_ /usr/libexec/packagekitd

This prctl is orthogonal to PID namespaces. PID namespaces are isolated from each other, while a service management process usually requires the services to live in the same namespace, to be able to talk to each other.

Users of this will be the systemd per-user instance, which provides init-like functionality for the user's login session and D-Bus, which activates bus services on-demand. Both need init-like capabilities to be able to properly keep track of the services they start.

Many thanks to Oleg for several rounds of review and insights.

Reviewed-by Oleg Nesterov <[email protected]>

Signed-off-by Lennart Poettering <[email protected]>

Signed-off-by Kay Sievers <[email protected]>

---

 include/linux/prctl.h |    3 +++ include/linux/sched.h |   12 ++++++++++++ kernel/exit.c         |   28 +++++++++++++++++++++++----- kernel/fork.c         |    3 +++
 kernel/sys.c          |    8 ++++++++

 5 files changed, 49 insertions(+), 5 deletions(-)

include/linux/prctl.h +3 -0

@@ -102,4 +102,7 @@
 
  #define PR_MCE_KILL_GET 34
 
+#define PR_SET_CHILD_SUBREAPER 35
+#define PR_GET_CHILD_SUBREAPER 36
+
  #endif /* _LINUX_PRCTL_H */

include/linux/sched.h +0 -0

@@ -552,6 +552,18 @@ struct signal_struct {
        int                     group_stop_count;

       unsigned int            flags; /* see SIGNAL_* flags below */

+       /*
+        * PR_SET_CHILD_SUBREAPER marks a process, like a service
+        * manager, to re-parent orphan (double-forking) child processes
+        * to this process instead of 'init'. The service manager is
+        * able to receive SIGCHLD signals and is able to investigate
+        * the process until it calls wait(). All children of this
+        * process will inherit a flag if they should look for a
+        * child_subreaper process at exit.
+        */
+       unsigned int            is_child_subreaper:1;
+       unsigned int            has_child_subreaper:1;
+
       /* POSIX.1b Interval Timers */
       struct list_head posix_timers;

kernel/exit.c +23 -5

@@ -687,11 +687,12 @@ static void exit_mm(struct task_struct *
  }
 
  /*
- * When we die, we re-parent all our children.
- * Try to give them to another thread in our thread
- * group, and if no such member exists, give it to
- * the child reaper process (ie "init") in our pid
- * space.
+ * When we die, we re-parent all our children, and try to:
+ * 1. give them to another thread in our thread group, if such a
+ *    member exists
+ * 2. give it to the first anchestor process which prctl'd itself
+ *    as a child_subreaper for its children (like a service manager)
+ * 3. give it to the init process (PID 1) in our pid namespace
  */
  static struct task_struct *find_new_reaper(struct task_struct *father)
        __releases(&tasklist_lock)

@@ -722,6 +723,23 @@ static struct task_struct *find_new_reap
                 * forget_original_parent() must move them somewhere.
                 */
                pid_ns->child_reaper = init_pid_ns.child_reaper;
+       } else if (father->signal->has_child_subreaper) {
+               struct task_struct *reaper;
+
+               /* find the first ancestor marked as child_subreaper */
+               for (reaper = father->real_parent;
+                    reaper != &init_task;
+                    reaper = reaper->real_parent) {
+                       if (same_thread_group(reaper, pid_ns->child_reaper))
+                               break;
+                       if (!reaper->signal->is_child_subreaper)
+                               continue;
+                       thread = reaper;
+                       do {
+                               if (!(thread->flags & PF_EXITING))
+                                       return reaper;
+                       } while_each_thread(reaper, thread);
+               }
        }
 
        return pid_ns->child_reaper;

kernel/fork.c +3 -0

@@ -979,6 +979,9 @@ static int copy_signal(unsigned long clo
        sig->oom_score_adj = current->signal->oom_score_adj;
        sig->oom_score_adj_min = current->signal->oom_score_adj_min;
 
+       sig->has_child_subreaper = current->signal->has_child_subreaper ||
+                                  current->signal->is_child_subreaper;
+
        mutex_init(&sig->cred_guard_mutex);
 
        return 0;

kernel/sys.c +0 -0

@@ -1841,6 +1841,14 @@ SYSCALL_DEFINE5(prctl, int, option, unsi

                       else
                               error = PR_MCE_KILL_DEFAULT; break;
+               case PR_SET_CHILD_SUBREAPER:
+                       me->signal->is_child_subreaper = !!arg2;
+                       error = 0;
+                       break;
+               case PR_GET_CHILD_SUBREAPER:
+                       error = put_user(me->signal->is_child_subreaper,
+                                        (int __user *) arg2);
+                       break;
               default:
                       error = -EINVAL;
                       break;

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Lennart,

Quoted 7 lines · Kay Sievers

Resending this, it got lost last year's September.

We still need it to properly implement init-like service managers.

Andrew, care to pick this up again? The issues raised the last year are all expected to be fixed.

Quoted 6 lines

Thanks,
Kay

From: Lennart Poettering <[email protected]>
Subject: prctl: add PR_{SET,GET}_CHILD_SUBREAPER to allow simple process supervision

You have often enthusiastically pointed out to me pieces that are missing from the man pages. When will you be sending me a man-pages patch to document these new prctl() options that you've added?

Thanks,

Michael

Quoted 175 lines

Userspace service managers/supervisors need to track their started services. Many services daemonize by double-forking and get implicitly re-parented to PID 1. The service manager will no longer be able to receive the SIGCHLD signals for them, and is no longer in charge of reaping the children with wait(). All information about the children is lost at the moment PID 1 cleans up the re-parented processes.

With this prctl, a service manager process can mark itself as a sort of 'sub-init', able to stay as the parent for all orphaned processes created by the started services. All SIGCHLD signals will be delivered to the service manager.

Receiving SIGCHLD and doing wait() is in cases of a service-manager much preferred over any possible asynchronous notification about specific PIDs, because the service manager has full access to the child process data in /proc and the PID can not be re-used until the wait(), the service-manager itself is in charge of, has happened.

As a side effect, the relevant parent PID information does not get lost by a double-fork, which results in a more elaborate process tree and 'ps' output:

before:

 # ps afx
 253 ?        Ss     0:00 /bin/dbus-daemon --system --nofork
 294 ?        Sl     0:00 /usr/libexec/polkit-1/polkitd
 328 ?        S      0:00 /usr/sbin/modem-manager
 608 ?        Sl     0:00 /usr/libexec/colord
 658 ?        Sl     0:00 /usr/libexec/upowerd
 819 ?        Sl     0:00 /usr/libexec/imsettings-daemon
 916 ?        Sl     0:00 /usr/libexec/udisks-daemon
 917 ?        S      0:00  \_ udisks-daemon: not polling any devices

after:

 # ps afx
 294 ?        Ss     0:00 /bin/dbus-daemon --system --nofork
 426 ?        Sl     0:00  \_ /usr/libexec/polkit-1/polkitd
 449 ?        S      0:00  \_ /usr/sbin/modem-manager
 635 ?        Sl     0:00  \_ /usr/libexec/colord
 705 ?        Sl     0:00  \_ /usr/libexec/upowerd
 959 ?        Sl     0:00  \_ /usr/libexec/udisks-daemon
 960 ?        S      0:00  |   \_ udisks-daemon: not polling any devices
 977 ?        Sl     0:00  \_ /usr/libexec/packagekitd

This prctl is orthogonal to PID namespaces. PID namespaces are isolated from each other, while a service management process usually requires the services to live in the same namespace, to be able to talk to each other.

Users of this will be the systemd per-user instance, which provides init-like functionality for the user's login session and D-Bus, which activates bus services on-demand. Both need init-like capabilities to be able to properly keep track of the services they start.

Many thanks to Oleg for several rounds of review and insights.

Reviewed-by Oleg Nesterov <[email protected]>

Signed-off-by Lennart Poettering <[email protected]>

Signed-off-by Kay Sievers <[email protected]>

---

 include/linux/prctl.h |    3 +++
 include/linux/sched.h |   12 ++++++++++++
 kernel/exit.c         |   28 +++++++++++++++++++++++-----
 kernel/fork.c         |    3 +++
 kernel/sys.c          |    8 ++++++++
 5 files changed, 49 insertions(+), 5 deletions(-)

include/linux/prctl.h +3 -0

@@ -102,4 +102,7 @@
 
 #define PR_MCE_KILL_GET 34
 
+#define PR_SET_CHILD_SUBREAPER 35
+#define PR_GET_CHILD_SUBREAPER 36
+
 #endif /* _LINUX_PRCTL_H */

include/linux/sched.h +12 -0

@@ -552,6 +552,18 @@ struct signal_struct {
       int                     group_stop_count;
       unsigned int            flags; /* see SIGNAL_* flags below */
 
+       /*
+        * PR_SET_CHILD_SUBREAPER marks a process, like a service
+        * manager, to re-parent orphan (double-forking) child processes
+        * to this process instead of 'init'. The service manager is
+        * able to receive SIGCHLD signals and is able to investigate
+        * the process until it calls wait(). All children of this
+        * process will inherit a flag if they should look for a
+        * child_subreaper process at exit.
+        */
+       unsigned int            is_child_subreaper:1;
+       unsigned int            has_child_subreaper:1;
+
       /* POSIX.1b Interval Timers */
       struct list_head posix_timers;
 

kernel/exit.c +23 -5

@@ -687,11 +687,12 @@ static void exit_mm(struct task_struct *
 }
 
 /*
- * When we die, we re-parent all our children.
- * Try to give them to another thread in our thread
- * group, and if no such member exists, give it to
- * the child reaper process (ie "init") in our pid
- * space.
+ * When we die, we re-parent all our children, and try to:
+ * 1. give them to another thread in our thread group, if such a
+ *    member exists
+ * 2. give it to the first anchestor process which prctl'd itself
+ *    as a child_subreaper for its children (like a service manager)
+ * 3. give it to the init process (PID 1) in our pid namespace
 */
 static struct task_struct *find_new_reaper(struct task_struct *father)
       __releases(&tasklist_lock)

@@ -722,6 +723,23 @@ static struct task_struct *find_new_reap
                * forget_original_parent() must move them somewhere.
                */
               pid_ns->child_reaper = init_pid_ns.child_reaper;
+       } else if (father->signal->has_child_subreaper) {
+               struct task_struct *reaper;
+
+               /* find the first ancestor marked as child_subreaper */
+               for (reaper = father->real_parent;
+                    reaper != &init_task;
+                    reaper = reaper->real_parent) {
+                       if (same_thread_group(reaper, pid_ns->child_reaper))
+                               break;
+                       if (!reaper->signal->is_child_subreaper)
+                               continue;
+                       thread = reaper;
+                       do {
+                               if (!(thread->flags & PF_EXITING))
+                                       return reaper;
+                       } while_each_thread(reaper, thread);
+               }
       }
 
       return pid_ns->child_reaper;

kernel/fork.c +3 -0

@@ -979,6 +979,9 @@ static int copy_signal(unsigned long clo
       sig->oom_score_adj = current->signal->oom_score_adj;
       sig->oom_score_adj_min = current->signal->oom_score_adj_min;
 
+       sig->has_child_subreaper = current->signal->has_child_subreaper ||
+                                  current->signal->is_child_subreaper;
+
       mutex_init(&sig->cred_guard_mutex);
 
       return 0;

kernel/sys.c +8 -1

@@ -1841,6 +1841,14 @@ SYSCALL_DEFINE5(prctl, int, option, unsi
                       else
                               error = PR_MCE_KILL_DEFAULT;
                       break;
+               case PR_SET_CHILD_SUBREAPER:
+                       me->signal->is_child_subreaper = !!arg2;
+                       error = 0;
+                       break;
+               case PR_GET_CHILD_SUBREAPER:
+                       error = put_user(me->signal->is_child_subreaper,
+                                        (int __user *) arg2);
+                       break;
               default:
                       error = -EINVAL;
                       break;
 
 
--

Quoted 4 lines

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
Michael Kerrisk Linux man-pages maintainer;
http://www.kernel.org/doc/man-pages/
Author of "The Linux Programming Interface", http://blog.man7.org/