Skip to contentSkip to conversation

make kernel threads invisible to /sbin/init

PATCH In review Oleg Nesterov 6 participants 26 messages Apr 10, 2007 – Apr 11, 2007 Scheduler

lore
Share & tools
lore
#0 lore raw

PATCH 2/3 make kernel threads invisible to /sbin/init

1. rename reparent_to_init() to reparent_kthread() and export it

2. use init_pid_ns.child_reaper instead of child_reaper(current)

3. set ->exit_signal = -1, so init can't see us and we don't use it to reap the task.

4. add reparent_kthread() to kthread() and stopmachine()

See also

	http://marc.info/?t=117580282200003&r=1
	http://marc.info/?t=95299284800003&r=1
 include/linux/sched.h |    1 +
 kernel/exit.c         |   16 ++++++++--------
 kernel/kthread.c      |    1 +
 kernel/stop_machine.c |    1 +
 4 files changed, 11 insertions(+), 8 deletions(-)
include/linux/sched.h renamed from include/linux/sched.h~2_DETACH +1 -0
@@ -1401,6 +1401,7 @@ extern void exit_itimers(struct signal_s
 
 extern NORET_TYPE void do_group_exit(int);
 
+extern void reparent_kthread(void);
 extern void daemonize(const char *, ...);
 extern int allow_signal(int);
 extern int disallow_signal(int);
kernel/exit.c renamed from kernel/exit.c~2_DETACH +8 -8
@@ -255,7 +255,7 @@ static int has_stopped_jobs(struct pid *
 }
 
 /**
- * reparent_to_init - Reparent the calling kernel thread to the init task of the pid space that the thread belongs to.
+ * reparent_kthread - Reparent the calling kernel thread to the init task of the pid space that the thread belongs to.
  *
  * If a kernel thread is launched as a result of a system call, or if
  * it ever exits, it should generally reparent itself to init so that
@@ -264,20 +264,20 @@ static int has_stopped_jobs(struct pid *
  * The various task state such as scheduling policy and priority may have
  * been inherited from a user process, so we reset them to sane values here.
  *
- * NOTE that reparent_to_init() gives the caller full capabilities.
+ * NOTE that reparent_kthread() gives the caller full capabilities.
  */
-static void reparent_to_init(void)
+void reparent_kthread(void)
 {
 	write_lock_irq(&tasklist_lock);
 
 	ptrace_unlink(current);
 	remove_parent(current);
-	current->parent = child_reaper(current);
-	current->real_parent = child_reaper(current);
+	current->parent = init_pid_ns.child_reaper;
+	current->real_parent = current->parent;
 	add_parent(current);
 
-	/* Set the exit signal to SIGCHLD so we signal init on exit */
-	current->exit_signal = SIGCHLD;
+	/* make the task auto-reap */
+	current->exit_signal = -1;
 
 	security_task_reparent_to_init(current);
 	write_unlock_irq(&tasklist_lock);
@@ -391,7 +391,7 @@ void daemonize(const char *name, ...)
 	current->files = init_task.files;
 	atomic_inc(&current->files->count);
 
-	reparent_to_init();
+	reparent_kthread();
 
 	if (!has_rt_policy(current) && (task_nice(current) < 0))
 		set_user_nice(current, 0);
kernel/kthread.c renamed from kernel/kthread.c~2_DETACH +1 -0
@@ -82,6 +82,7 @@ static int kthread(void *_create)
 	sigset_t blocked;
 	int ret = -EINTR;
 
+	reparent_kthread();
 	kthread_exit_files();
 
 	/* Copy data: it's on keventd's stack */
kernel/stop_machine.c renamed from kernel/stop_machine.c~2_DETACH +1 -0
@@ -33,6 +33,7 @@ static int stopmachine(void *cpu)
 	int irqs_disabled = 0;
 	int prepared = 0;
 
+	reparent_kthread();
 	set_cpus_allowed(current, cpumask_of_cpu((int)(long)cpu));
 
 	/* Ack: we are alive */
#1 lore raw
Quoted 3 lines · Oleg Nesterov

1. rename reparent_to_init() to reparent_kthread() and export it

2. use init_pid_ns.child_reaper instead of child_reaper(current)

Each of these patches looks good to me, but this part in particular is a must-have bugfix.

Just started some tests, if any failures come back I'll report them tonight.

thanks Oleg,
-serge

Quoted 95 lines

3. set ->exit_signal = -1, so init can't see us and we don't use it to reap the task.

4. add reparent_kthread() to kthread() and stopmachine()

See also

	http://marc.info/?t=117580282200003&r=1
	http://marc.info/?t=95299284800003&r=1
 include/linux/sched.h |    1 +
 kernel/exit.c         |   16 ++++++++--------
 kernel/kthread.c      |    1 +
 kernel/stop_machine.c |    1 +
 4 files changed, 11 insertions(+), 8 deletions(-)
include/linux/sched.h renamed from include/linux/sched.h~2_DETACH +1 -0
@@ -1401,6 +1401,7 @@ extern void exit_itimers(struct signal_s
 
 extern NORET_TYPE void do_group_exit(int);
 
+extern void reparent_kthread(void);
 extern void daemonize(const char *, ...);
 extern int allow_signal(int);
 extern int disallow_signal(int);
kernel/exit.c renamed from kernel/exit.c~2_DETACH +8 -8
@@ -255,7 +255,7 @@ static int has_stopped_jobs(struct pid *
 }
 
 /**
- * reparent_to_init - Reparent the calling kernel thread to the init task of the pid space that the thread belongs to.
+ * reparent_kthread - Reparent the calling kernel thread to the init task of the pid space that the thread belongs to.
  *
  * If a kernel thread is launched as a result of a system call, or if
  * it ever exits, it should generally reparent itself to init so that
@@ -264,20 +264,20 @@ static int has_stopped_jobs(struct pid *
  * The various task state such as scheduling policy and priority may have
  * been inherited from a user process, so we reset them to sane values here.
  *
- * NOTE that reparent_to_init() gives the caller full capabilities.
+ * NOTE that reparent_kthread() gives the caller full capabilities.
  */
-static void reparent_to_init(void)
+void reparent_kthread(void)
 {
 	write_lock_irq(&tasklist_lock);
 
 	ptrace_unlink(current);
 	remove_parent(current);
-	current->parent = child_reaper(current);
-	current->real_parent = child_reaper(current);
+	current->parent = init_pid_ns.child_reaper;
+	current->real_parent = current->parent;
 	add_parent(current);
 
-	/* Set the exit signal to SIGCHLD so we signal init on exit */
-	current->exit_signal = SIGCHLD;
+	/* make the task auto-reap */
+	current->exit_signal = -1;
 
 	security_task_reparent_to_init(current);
 	write_unlock_irq(&tasklist_lock);
@@ -391,7 +391,7 @@ void daemonize(const char *name, ...)
 	current->files = init_task.files;
 	atomic_inc(&current->files->count);
 
-	reparent_to_init();
+	reparent_kthread();
 
 	if (!has_rt_policy(current) && (task_nice(current) < 0))
 		set_user_nice(current, 0);
kernel/kthread.c renamed from kernel/kthread.c~2_DETACH +1 -0
@@ -82,6 +82,7 @@ static int kthread(void *_create)
 	sigset_t blocked;
 	int ret = -EINTR;
 
+	reparent_kthread();
 	kthread_exit_files();
 
 	/* Copy data: it's on keventd's stack */
kernel/stop_machine.c renamed from kernel/stop_machine.c~2_DETACH +1 -0
@@ -33,6 +33,7 @@ static int stopmachine(void *cpu)
 	int irqs_disabled = 0;
 	int prepared = 0;
 
+	reparent_kthread();
 	set_cpus_allowed(current, cpumask_of_cpu((int)(long)cpu));
 
 	/* Ack: we are alive */
#2 lore raw
Quoted 7 lines · Serge E. Hallyn
Quoted 3 lines · Oleg Nesterov

1. rename reparent_to_init() to reparent_kthread() and export it

2. use init_pid_ns.child_reaper instead of child_reaper(current)

Each of these patches looks good to me, but this part in particular is a must-have bugfix.

Removing daemonize is a must have bug fix. This falls short of that so it is a good fix, but it doesn't solve the core problem that kernel daemons are being assigned pids inside of child pid namespaces.

It doesn't solve the problem that some kernel daemons are using signals to communicate with user space.

It doesn't solve the problem that we have to do a lot of massaging and maintenance to get kernel threads from grabbing references to namespaces and other kernel pieces they should not be grabbing.

Eric

#3 lore raw
In reply to Oleg Nesterov
Quoted 9 lines · Oleg Nesterov

1. rename reparent_to_init() to reparent_kthread() and export it

2. use init_pid_ns.child_reaper instead of child_reaper(current)

3. set ->exit_signal = -1, so init can't see us and we don't use it to reap the task.

4. add reparent_kthread() to kthread() and stopmachine()

If the goal is to hide from /sbin/init. We don't need to touch kernel/kthread.c or
kernel/stop_machine.c

Their parents are already kernel threads.

Yes. There is a startup issue for kernel/kthread.c I just about have a patch I can stand that addresses that issue.

For the kernel thread they all inherit signals with SIGCHLD set to SIG_IGN, so there is child auto reaping in that form. Adding the ->exit_signal = -1 would be a bonus but is not required.

I am a little concerned with removing the knowledge of who our real parent is as that may compromise debugging. The reason why kernel threads are visible in the first place.

I do support reparenting kernel threads that call daemonize, and the modifications to reparent_kthread here look reasonable.

Eric

#4 lore raw
Quoted 17 lines · Eric W. Biederman
Quoted 9 lines · Oleg Nesterov

1. rename reparent_to_init() to reparent_kthread() and export it

2. use init_pid_ns.child_reaper instead of child_reaper(current)

3. set ->exit_signal = -1, so init can't see us and we don't use it to reap the task.

4. add reparent_kthread() to kthread() and stopmachine()

If the goal is to hide from /sbin/init. We don't need to touch kernel/kthread.c or
kernel/stop_machine.c

Quoted 5 lines

Their parents are already kernel threads.

For the kernel thread they all inherit signals with SIGCHLD set to SIG_IGN, so there is child auto reaping in that form. Adding the ->exit_signal = -1 would be a bonus but is not required.

Unless a kernel thread does kernel_thread() (not kthread_create) and exits. In that case the child will be re-parented to init which doesn't ignore SIGCHLD.

Quoted 6 lines · Robin Holt

wait_task_zombie() is taking many seconds to get through the list. For the case of a modprobe, stop_machine creates one thread per cpu (remember big number). All are parented to init and their exit will cause wait_task_zombie to scan multiple times most of the way through this very long list looking for threads which need to be reaped.

initially, "stopmachine" threads were not parented to init.

However, I agree, your patch is better, and solves most problems in more simple way. Including the above problem, I believe. "stopmachine" likely does exit_notify() and notices SIG_IGN (inherited from kthreadd_setup()) before "do_stop" does forget_original_parent().

Oleg.

#5 lore raw

PATCH 1/1 kthread: Don't depend on work queues

In reply to Oleg Nesterov

Currently there is a circular reference between work queue initialization
and kthread initialization. This prevents the kernel thread
infrastructure from initializing until after work queues have been initialized.

For kernel threads we want something that is as close as possible to the init_task and is not contaminated by user processes. The later we start our kthreadd that forks the rest of the kernel threads the harder this is to do and the more of a mess we have to clean up because the defaults have changed on us.

So this patch modifies the kthread support to not use work queues but to instead use a simple list of structures, and to have kthreadd start from init_task immediately after our kernel thread that execs /sbin/init.

By being a true child of init_task we only have to change those process settings that we want to have different from init_task, such as our process name, blocking all signals and setting SIGCHLD to SIG_IGN so that all of our children are reaped automatically.

By being a tre child of init_task we also naturally get our ppid set to 0 and do not wind up as a child of PID == 1. Ensuring that kernel threads will not slow down the functioning of the wait family of functions.

---
 include/linux/kthread.h |    2 +
 init/main.c             |    2 +
 kernel/kthread.c        |  121 +++++++++++++++++++++++++----------------------
 3 files changed, 69 insertions(+), 56 deletions(-)
include/linux/kthread.h +2 -0
index 1c65e7a..113bd7c 100644
@@ -30,4 +30,6 @@ void kthread_bind(struct task_struct *k, unsigned int cpu);
 int kthread_stop(struct task_struct *k);
 int kthread_should_stop(void);
 
+int kthreadd(void *unused);
+
 #endif /* _LINUX_KTHREAD_H */
init/main.c +2 -0
index a92989e..12a8aba 100644
@@ -54,6 +54,7 @@
 #include <linux/lockdep.h>
 #include <linux/pid_namespace.h>
 #include <linux/device.h>
+#include <linux/kthread.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
@@ -437,6 +438,7 @@ static void noinline rest_init(void)
 {
 	kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
 	numa_default_policy();
+	kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES);
 	unlock_kernel();
 
 	/*
kernel/kthread.c +65 -56

185 lines not on this page · raw message

Signature · 1 line
1.5.0.g53756
#6 lore raw

Re: [PATCH] kthread: Don't depend on work queues

Quoted 15 lines · On Tue, 10 Apr 2007 23:44:36 -0600 [email protected] (Eric W. Biederman)…

Currently there is a circular reference between work queue initialization
and kthread initialization. This prevents the kernel thread
infrastructure from initializing until after work queues have been initialized.

For kernel threads we want something that is as close as possible to the init_task and is not contaminated by user processes. The later we start our kthreadd that forks the rest of the kernel threads the harder this is to do and the more of a mess we have to clean up because the defaults have changed on us.

So this patch modifies the kthread support to not use work queues but to instead use a simple list of structures, and to have kthreadd start from init_task immediately after our kernel thread that execs /sbin/init.

Quoted 8 lines

By being a true child of init_task we only have to change those process settings that we want to have different from init_task, such as our process name, blocking all signals and setting SIGCHLD to SIG_IGN so that all of our children are reaped automatically.

By being a tre child of init_task we also naturally get our ppid set to 0 and do not wind up as a child of PID == 1. Ensuring that kernel threads will not slow down the functioning of the wait family of functions.

argh. Your description freely confuddles the terms "kernel thread" and "kthread". Can we not do that? Henceforth the term "kernel thread" refers to something which was started with kernel_thread() and "kthread" refers to something which was created by kthread_create(), OK?

Your patch gets midly tangled up with Oleg's recent

reduce-reparent_to_init.patch
make-kernel-threads-invisible-to-sbin-init.patch
reparent-kernel-threads-to-swapper.patch

but they seemed fairly unpopular anyway so I'll drop 'em.

Your wait_event() will contribute to load average, I expect. We get mail. I converted it to wait_event_interruptible().

I guess using PF_NOFREEZE rather than try_to_freeze() is OK, but one wonders what thinking led to that?

Often when we have a singleton thread like this it is neater to use wake_up_process() directly on it, rather than creating a rather pointless waitqueue_head for it. I started looking into that but it would have taken more than 30 seconds.

#7 lore raw

Re: [PATCH] kthread: Don't depend on work queues

Quoted 37 lines · Andrew Morton
Quoted 15 lines · On Tue, 10 Apr 2007 23:44:36 -0600 [email protected] (Eric W. Biederman)…

Currently there is a circular reference between work queue initialization
and kthread initialization. This prevents the kernel thread
infrastructure from initializing until after work queues have been initialized.

For kernel threads we want something that is as close as possible to the init_task and is not contaminated by user processes. The later we start our kthreadd that forks the rest of the kernel threads the harder this is to do and the more of a mess we have to clean up because the defaults have changed on us.

So this patch modifies the kthread support to not use work queues but to instead use a simple list of structures, and to have kthreadd start from init_task immediately after our kernel thread that execs /sbin/init.

Quoted 8 lines

By being a true child of init_task we only have to change those process settings that we want to have different from init_task, such as our process name, blocking all signals and setting SIGCHLD to SIG_IGN so that all of our children are reaped automatically.

By being a tre child of init_task we also naturally get our ppid set to 0 and do not wind up as a child of PID == 1. Ensuring that kernel threads will not slow down the functioning of the wait family of functions.

argh. Your description freely confuddles the terms "kernel thread" and "kthread". Can we not do that? Henceforth the term "kernel thread" refers to something which was started with kernel_thread() and "kthread" refers to something which was created by kthread_create(), OK?

Your patch gets midly tangled up with Oleg's recent

reduce-reparent_to_init.patch
make-kernel-threads-invisible-to-sbin-init.patch
reparent-kernel-threads-to-swapper.patch

Quoted 7 lines

but they seemed fairly unpopular anyway so I'll drop 'em.

Your wait_event() will contribute to load average, I expect. We get mail. I converted it to wait_event_interruptible().

I guess using PF_NOFREEZE rather than try_to_freeze() is OK, but one wonders what thinking led to that?

It should be calling try_to_freeze() somewhere anyway. We may need to freeze all tasks in some cases.

Greetings,
Rafael

#8 lore raw

Re: [PATCH] kthread: Don't depend on work queues

Quoted 3 lines · Rafael J. Wysocki

It should be calling try_to_freeze() somewhere anyway. We may need to freeze all tasks in some cases.

How about

for (;;) {
	try_to_freeze();

?

This change allows us to make all the worker threads freezeable by default.

Quoted 1 line

From cpu-hotplug perspective, helper_wq was the only singlethreaded

non-freezeable workqueue.

Quoted 4 lines

Greetings,
Rafael
-

Thanks and Regards
gautham.

Signature · 5 lines
Gautham R Shenoy
Linux Technology Center
IBM India.
"Freedom comes with a price tag of responsibility, which is still a bargain,
because Freedom is priceless!"
#9 lore raw

Re: [PATCH] kthread: Don't depend on work queues

Quoted 11 lines · Gautham R Shenoy
Quoted 3 lines · Rafael J. Wysocki

It should be calling try_to_freeze() somewhere anyway. We may need to freeze all tasks in some cases.

How about

for (;;) {
	try_to_freeze();

?

Why?

Quoted 3 lines

This change allows us to make all the worker threads freezeable by default.

Quoted 1 line

From cpu-hotplug perspective, helper_wq was the only singlethreaded

non-freezeable workqueue.

I think Eric's patch is what you need. We should _not_ freeze kthreadd(), we need kthread_create() after freezing. Now it doesn't depend on workqueues, we can freeze them all, single-thread or not.

I like this patch.

Oleg.

#10 lore raw

Re: [PATCH] kthread: Don't depend on work queues

Quoted 7 lines · Oleg Nesterov
Quoted 6 lines · Gautham R Shenoy
Quoted 3 lines · Rafael J. Wysocki

It should be calling try_to_freeze() somewhere anyway. We may need to freeze all tasks in some cases.

Quoted 7 lines
Quoted 5 lines

How about

for (;;) {
	try_to_freeze();

?

Why?

If some event (defintely NOT cpu hotplug) needs this thread frozen.

Quoted 9 lines
Quoted 4 lines

This change allows us to make all the worker threads freezeable by default.

Quoted 1 line

From cpu-hotplug perspective, helper_wq was the only singlethreaded

non-freezeable workqueue.

I think Eric's patch is what you need. We should _not_ freeze kthreadd(), we need kthread_create() after freezing. Now it doesn't depend on workqueues, we can freeze them all, single-thread or not.

These were my exact thoughts.

Quoted 1 line

I like this patch.

Same here.

Quoted 3 lines

Oleg.

Thanks and Regards
gautham.

Signature · 5 lines
Gautham R Shenoy
Linux Technology Center
IBM India.
"Freedom comes with a price tag of responsibility, which is still a bargain,
because Freedom is priceless!"
#11 lore raw

Re: [PATCH] kthread: Don't depend on work queues

Quoted 12 lines · Gautham R Shenoy
Quoted 11 lines · Oleg Nesterov
Quoted 10 lines · Gautham R Shenoy
Quoted 3 lines · Rafael J. Wysocki

It should be calling try_to_freeze() somewhere anyway. We may need to freeze all tasks in some cases.

How about

for (;;) {
	try_to_freeze();
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

?

Why?

If some event (defintely NOT cpu hotplug) needs this thread frozen.

I didn't mean the CPU hotplug too, BTW, but I'm considering freezing all tasks at the late stage of suspend (ie. after freezing devices).

Greetings,
Rafael

#12 lore raw

Re: [PATCH] kthread: Don't depend on work queues

In reply to Gautham R Shenoy
Quoted 21 lines · Gautham R Shenoy
Quoted 7 lines · Oleg Nesterov
Quoted 6 lines · Gautham R Shenoy
Quoted 3 lines · Rafael J. Wysocki

It should be calling try_to_freeze() somewhere anyway. We may need to freeze all tasks in some cases.

Quoted 7 lines
Quoted 5 lines

How about

for (;;) {
	try_to_freeze();

?

Why?

If some event (defintely NOT cpu hotplug) needs this thread frozen.

Quoted 3 lines
Quoted 3 lines

This change allows us to make all the worker threads freezeable by default.

Quoted 1 line

From cpu-hotplug perspective, helper_wq was the only singlethreaded

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

non-freezeable workqueue.

I think Eric's patch is what you need. We should _not_ freeze kthreadd(), we need kthread_create() after freezing. Now it doesn't depend on workqueues, we can freeze them all, single-thread or not.

These were my exact thoughts.

Sorry, I misunderstood your message.

Yes, we can freeze it with FE_HOTPLUG_CPU. In that case wait_event() should also check !freezing(), and try_to_freeze() should be called after case wait_event().

On the other hand, if "kthreadd" does not sleep on kthread_create_work, we have another unfrozen process waiting for kthread_create_info.done. So, is there any practical reason why kthreadd() should explicitely go to refrigerator?

Oleg.

#13 lore raw

Re: [PATCH] kthread: Don't depend on work queues

Quoted 33 lines · Oleg Nesterov
Quoted 21 lines · Gautham R Shenoy
Quoted 7 lines · Oleg Nesterov
Quoted 6 lines · Gautham R Shenoy
Quoted 3 lines · Rafael J. Wysocki

It should be calling try_to_freeze() somewhere anyway. We may need to freeze all tasks in some cases.

Quoted 7 lines
Quoted 5 lines

How about

for (;;) {
	try_to_freeze();

?

Why?

If some event (defintely NOT cpu hotplug) needs this thread frozen.

Quoted 3 lines
Quoted 3 lines

This change allows us to make all the worker threads freezeable by default.

Quoted 1 line

From cpu-hotplug perspective, helper_wq was the only singlethreaded

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

non-freezeable workqueue.

I think Eric's patch is what you need. We should _not_ freeze kthreadd(), we need kthread_create() after freezing. Now it doesn't depend on workqueues, we can freeze them all, single-thread or not.

These were my exact thoughts.

Sorry, I misunderstood your message.

Quoted 8 lines

Yes, we can freeze it with FE_HOTPLUG_CPU. In that case wait_event() should also check !freezing(), and try_to_freeze() should be called after case wait_event().

On the other hand, if "kthreadd" does not sleep on kthread_create_work, we have another unfrozen process waiting for kthread_create_info.done. So, is there any practical reason why kthreadd() should explicitely go to refrigerator?

Good question. Right now, there probably is not any.

Greetings,
Rafael

#14 lore raw

Re: [PATCH] kthread: Don't depend on work queues

In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

argh. Your description freely confuddles the terms "kernel thread" and "kthread". Can we not do that? Henceforth the term "kernel thread" refers to something which was started with kernel_thread() and "kthread" refers to something which was created by kthread_create(), OK?

Yes. Will fix.

Quoted 7 lines

Your patch gets midly tangled up with Oleg's recent

reduce-reparent_to_init.patch
make-kernel-threads-invisible-to-sbin-init.patch
reparent-kernel-threads-to-swapper.patch

but they seemed fairly unpopular anyway so I'll drop 'em.

Ok. I kind of liked the first one but that is a minor cleanup.

Quoted 2 lines

Your wait_event() will contribute to load average, I expect. We get mail. I converted it to wait_event_interruptible().

Ok. That is more polite.

Quoted 2 lines

I guess using PF_NOFREEZE rather than try_to_freeze() is OK, but one wonders what thinking led to that?

That is what we are currently doing for the work queues, and I was lazy. For people who care they can fix it.

Quoted 4 lines

Often when we have a singleton thread like this it is neater to use wake_up_process() directly on it, rather than creating a rather pointless waitqueue_head for it. I started looking into that but it would have taken more than 30 seconds.

Sure I took a look and it isn't too hard. Updated patch in a minute...

I have left the locking the way it is despite the reasonable chance that Oleg points I can only acquire the lock when deleting the list entry. I'm to lazy to think through the SMP races to make certain that is safe.

Eric

#15 lore raw

PATCH 1/1 kthread: Don't depend on work queues (take 2)

In reply to Andrew Morton

Currently there is a circular reference between work queue initialization
and kthread initialization. This prevents the kthread infrastructure from
initializing until after work queues have been initialized.

We want the properties of tasks created with kthread_create to be as close as possible to the init_task and to not be contaminated by user processes. The later we start our kthreadd that creates these tasks the harder it is to avoid contamination from user processes and the more of a mess we have to clean up because the defaults have changed on us.

So this patch modifies the kthread support to not use work queues but to instead use a simple list of structures, and to have kthreadd start from init_task immediately after our kernel thread that execs /sbin/init.

By being a true child of init_task we only have to change those process settings that we want to have different from init_task, such as our process name, the cpus that are allowed, blocking all signals and setting SIGCHLD to SIG_IGN so that all of our children are reaped automatically.

By being a true child of init_task we also naturally get our ppid set to 0 and do not wind up as a child of PID == 1. Ensuring that tasks generated by kthread_create will not slow down the functioning of the wait family of functions.

---
 include/linux/kthread.h |    3 +
 init/main.c             |    5 ++
 kernel/kthread.c        |  124 ++++++++++++++++++++++++++---------------------
 3 files changed, 76 insertions(+), 56 deletions(-)
include/linux/kthread.h +3 -0
index 1c65e7a..00dd957 100644
@@ -30,4 +30,7 @@ void kthread_bind(struct task_struct *k, unsigned int cpu);
 int kthread_stop(struct task_struct *k);
 int kthread_should_stop(void);
 
+int kthreadd(void *unused);
+extern struct task_struct *kthreadd_task;
+
 #endif /* _LINUX_KTHREAD_H */
init/main.c +5 -0
index a92989e..d05430e 100644
@@ -54,6 +54,7 @@
 #include <linux/lockdep.h>
 #include <linux/pid_namespace.h>
 #include <linux/device.h>
+#include <linux/kthread.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
@@ -435,8 +436,12 @@ static void __init setup_command_line(char *command_line)
 static void noinline rest_init(void)
 	__releases(kernel_lock)
 {
+	int pid;
 	kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
 	numa_default_policy();
+
+	pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES);
+	kthreadd_task = find_task_by_pid(pid);
 	unlock_kernel();
 
 	/*
kernel/kthread.c +68 -56

188 lines not on this page · raw message

Signature · 1 line
1.5.0.g53756
#16 lore raw

PATCH 1/1 Change reparent_to_init to reparent_to_kthreadd

When a kernel thread calls daemonize, instead of reparenting the thread to init reparent the thread to kthreadd next to the threads created by kthread_create.

This is really just a stop gap until daemonize goes away, but it does ensure no kernel threads are under init and they are all in one place that is easy to find.

---
 kernel/exit.c |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)
kernel/exit.c +8 -8
index b55ed4c..1a7c878 100644
@@ -42,6 +42,7 @@
 #include <linux/audit.h> /* for audit_free() */
 #include <linux/resource.h>
 #include <linux/blkdev.h>
+#include <linux/kthread.h>
 
 #include <asm/uaccess.h>
 #include <asm/unistd.h>
@@ -255,26 +256,25 @@ static int has_stopped_jobs(struct pid *pgrp)
 }
 
 /**
- * reparent_to_init - Reparent the calling kernel thread to the init task of the pid space that the thread belongs to.
+ * reparent_to_kthreadd - Reparent the calling kernel thread to kthreadd
  *
  * If a kernel thread is launched as a result of a system call, or if
- * it ever exits, it should generally reparent itself to init so that
- * it is correctly cleaned up on exit.
+ * it ever exits, it should generally reparent itself to kthreadd so it
+ * isn't in the way of other processes and is correctly cleaned up on exit.
  *
  * The various task state such as scheduling policy and priority may have
  * been inherited from a user process, so we reset them to sane values here.
  *
- * NOTE that reparent_to_init() gives the caller full capabilities.
+ * NOTE that reparent_to_kthreadd() gives the caller full capabilities.
  */
-static void reparent_to_init(void)
+static void reparent_to_kthreadd(void)
 {
 	write_lock_irq(&tasklist_lock);
 
 	ptrace_unlink(current);
 	/* Reparent to init */
 	remove_parent(current);
-	current->parent = child_reaper(current);
-	current->real_parent = child_reaper(current);
+	current->real_parent = current->parent = kthreadd_task;
 	add_parent(current);
 
 	/* Set the exit signal to SIGCHLD so we signal init on exit */
@@ -401,7 +401,7 @@ void daemonize(const char *name, ...)
 	current->files = init_task.files;
 	atomic_inc(&current->files->count);
 
-	reparent_to_init();
+	reparent_to_kthreadd();
 }
 
 EXPORT_SYMBOL(daemonize);
Signature · 1 line
1.5.0.g53756
#17 lore raw

Re: [PATCH] kthread: Don't depend on work queues (take 2)

In reply to Eric W. Biederman

kernel/kthread.c int kthread_stop(struct task_struct *k) · line 221

Quoted 1 line · On Wed, 11 Apr 2007 12:27:59 -0600 [email protected] (Eric W. Biederman)…
kernel/kthread.c +1 -0
+	strcpy(tsk->comm, "kthreadd");

We have this dopey set_task_comm() thing which is there to avoid races when userspace is looking at this task's name via /proc.

It obviously doesn't matter in this case, but I suppose one should set a good example. I'll make that change.

#18 lore raw

Re: [PATCH] kthread: Don't depend on work queues (take 2)

In reply to Eric W. Biederman

init/main.c static void __init setup_command_line(char *command_line) · line 445

Quoted 12 lines · Eric W. Biederman
init/main.c +4 -0
@@ -435,8 +436,12 @@ static void __init setup_command_line(char *command_line)
 static void noinline rest_init(void)
 	__releases(kernel_lock)
 {
+	int pid;
 	kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
 	numa_default_policy();
+
+	pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES);
+	kthreadd_task = find_task_by_pid(pid);
 	unlock_kernel();

Just curious. What if kernel/kthread.c declares

	static struct task_struct *kthreadd_task = &init_task;

an then kthreadd_setup() does kthreadd_task = current. I assume it is always safe to try_to_wake_up(idle_thread), because it always TASK_RUNNING. This way we don't need to export kthreadd_task.

kernel/kthread.c struct task_struct *kthread_create(int (*threadfn)(void *data), · line 139

Quoted 4 lines
kernel/kthread.c +4 -0
+	spin_lock(&kthread_create_lock);
+	list_add_tail(&create.list, &kthread_create_list);
+	wake_up_process(kthreadd_task);
+	spin_unlock(&kthread_create_lock);

Very minor nit, but we don't need to do wake_up under spin_unlock().

Oleg.

#19 lore raw

Re: [PATCH] kthread: Don't depend on work queues (take 2)

Quoted 16 lines · Oleg Nesterov

init/main.c static void __init setup_command_line(char *command_line) · line 445

Quoted 12 lines · Eric W. Biederman
init/main.c +4 -0
@@ -435,8 +436,12 @@ static void __init setup_command_line(char *command_line)
 static void noinline rest_init(void)
 	__releases(kernel_lock)
 {
+	int pid;
 	kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
 	numa_default_policy();
+
+	pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES);
+	kthreadd_task = find_task_by_pid(pid);
 	unlock_kernel();

Just curious. What if kernel/kthread.c declares

Quoted 5 lines
	static struct task_struct *kthreadd_task = &init_task;

an then kthreadd_setup() does kthreadd_task = current. I assume it is always safe to try_to_wake_up(idle_thread), because it always TASK_RUNNING. This way we don't need to export kthreadd_task.

I did it this way largely so I could use the export in reparent_to_XXX in daemonize. This way I don't have races in finding kthreadd. Plus I didn't think of the trick of using the idle_thread...

Quoted 6 lines

kernel/kthread.c struct task_struct *kthread_create(int (*threadfn)(void *data), · line 139

Quoted 4 lines
kernel/kthread.c +4 -0
+	spin_lock(&kthread_create_lock);
+	list_add_tail(&create.list, &kthread_create_list);
+	wake_up_process(kthreadd_task);
+	spin_unlock(&kthread_create_lock);

Very minor nit, but we don't need to do wake_up under spin_unlock().

I guess that is true. However it doesn't hurt either. I guess I was keeping the form that I used with wait queues where it may have mattered. Either that or I just copied a bad example.

Eric

#20 lore raw

Re: [PATCH] kthread: Don't depend on work queues (take 2)

Quoted 21 lines · Eric W. Biederman
Quoted 18 lines · Oleg Nesterov

init/main.c static void __init setup_command_line(char *command_line) · line 445

Quoted 12 lines · Eric W. Biederman
init/main.c +4 -0
@@ -435,8 +436,12 @@ static void __init setup_command_line(char *command_line)
 static void noinline rest_init(void)
 	__releases(kernel_lock)
 {
+	int pid;
 	kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
 	numa_default_policy();
+
+	pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES);
+	kthreadd_task = find_task_by_pid(pid);
 	unlock_kernel();

Just curious. What if kernel/kthread.c declares

	static struct task_struct *kthreadd_task = &init_task;
Quoted 7 lines
Quoted 3 lines

an then kthreadd_setup() does kthreadd_task = current. I assume it is always safe to try_to_wake_up(idle_thread), because it always TASK_RUNNING. This way we don't need to export kthreadd_task.

I did it this way largely so I could use the export in reparent_to_XXX in daemonize. This way I don't have races in finding kthreadd. Plus I didn't think of the trick of using the idle_thread...

Ah yes, we still need to export kthreadd_task... I just worried about subtle dependency this patch adds... This "kthreadd_task = find_task_by_pid" assumes that init/main.c:init() takes lock_kernel() before the first kthread_create().

Oleg.

#21 lore raw

Re: [PATCH] kthread: Don't depend on work queues (take 2)

Quoted 3 lines · Oleg Nesterov

Ah yes, we still need to export kthreadd_task... I just worried about subtle dependency this patch adds... This "kthreadd_task = find_task_by_pid" assumes that init/main.c:init() takes lock_kernel() before the first kthread_create().

There is a small element of that. Mostly it assumes we are running in the idle thread until we call schedule. Anyway the assumption and the code is out there in the open in the main code path so it is easy to find, and the code works today.

At this point I am queuing up patches that kill daemonize. I will be happy to revisit this part once daemonize is gone and the only interesting file is kthread.c

Eric

#22 lore raw

Re: [PATCH] kthread: Don't depend on work queues

In reply to Eric W. Biederman
Quoted 4 lines · Eric W. Biederman
kernel/kthread.c +3 -0
+int kthreadd(void *unused)
+{
+	/* Setup a clean context for our children to inherit. */

kernel/kthread.c int kthread_stop(struct task_struct *k) · line 249

Quoted 10 lines
kernel/kthread.c +10 -0
+	kthreadd_setup();
+
+	current->flags |= PF_NOFREEZE;
+
+	for (;;) {
+		wait_event(kthread_create_work,
+			   !list_empty(&kthread_create_list));
+
+		spin_lock(&kthread_create_lock);
+		while (!list_empty(&kthread_create_list)) {

Do we need to check the condition under lock? We can miss an event, but then it will be noticed by wait_event() above.

kernel/kthread.c int kthread_stop(struct task_struct *k) · line 262

Quoted 13 lines
kernel/kthread.c +13 -0
+			struct kthread_create_info *create;
+
+			create = list_entry(kthread_create_list.next,
+					    struct kthread_create_info, list);
+			list_del_init(&create->list);
+			spin_unlock(&kthread_create_lock);
+
+			create_kthread(create);
+
+			spin_lock(&kthread_create_lock);
+		}
+		spin_unlock(&kthread_create_lock);
+	}

IOW,

	for (;;) {
		wait_event(kthread_create_work,
			   !list_empty(&kthread_create_list));

		while (!list_empty(&kthread_create_list)) {
			struct kthread_create_info *create;

			spin_lock(&kthread_create_lock);
			create = list_entry(kthread_create_list.next,
					    struct kthread_create_info, list);
			list_del_init(&create->list);
			spin_unlock(&kthread_create_lock);

			create_kthread(create);
		}
	}

Oleg.

#23 lore raw

Re: [PATCH] kthread: Don't depend on work queues

Quoted 18 lines · Oleg Nesterov
Quoted 4 lines · Eric W. Biederman
kernel/kthread.c +3 -0
+int kthreadd(void *unused)
+{
+	/* Setup a clean context for our children to inherit. */

kernel/kthread.c int kthread_stop(struct task_struct *k) · line 249

Quoted 10 lines
kernel/kthread.c +10 -0
+	kthreadd_setup();
+
+	current->flags |= PF_NOFREEZE;
+
+	for (;;) {
+		wait_event(kthread_create_work,
+			   !list_empty(&kthread_create_list));
+
+		spin_lock(&kthread_create_lock);
+		while (!list_empty(&kthread_create_list)) {

Do we need to check the condition under lock? We can miss an event, but then it will be noticed by wait_event() above.

We need to be certain there is something on the list before we remove it. Otherwise we will start dereferencing bad pointers.

Quoted 18 lines

IOW,

	for (;;) {
		wait_event(kthread_create_work,
			   !list_empty(&kthread_create_list));

		while (!list_empty(&kthread_create_list)) {
			struct kthread_create_info *create;

			spin_lock(&kthread_create_lock);
			create = list_entry(kthread_create_list.next,
					    struct kthread_create_info, list);
			list_del_init(&create->list);
			spin_unlock(&kthread_create_lock);

			create_kthread(create);
		}
	}

I guess since we are the only process to ever remove things from the list that would be safe.

Eric

#24 lore raw

Re: [PATCH] kthread: Don't depend on work queues

In reply to Eric W. Biederman
Quoted 13 lines · Eric W. Biederman

static int kthread(void *_create)
{

kernel/kthread.c +1 -4
 	struct kthread_create_info *create = _create;
 	int (*threadfn)(void *data);
 	void *data;
-	sigset_t blocked;
 	int ret = -EINTR;
 
-	kthread_exit_files();
-
-	/* Copy data: it's on keventd's stack */
+	/* Copy data: it's on kthread's stack */

kernel/kthread.c int kthread_should_stop(void) · line 97

Quoted 10 lines
kernel/kthread.c +0 -7
 	threadfn = create->threadfn;
 	data = create->data;
 
-	/* Block and flush all signals (in case we're not from keventd). */
-	sigfillset(&blocked);
-	sigprocmask(SIG_BLOCK, &blocked, NULL);
-	flush_signals(current);
-
-	/* By default we can run anywhere, unlike keventd. */
-	set_cpus_allowed(current, CPU_MASK_ALL);

The above is OK, but I believe you should add set_cpus_allowed() to kthreadd_setup(). Note that kthreadd() is forked after init_idle().

Oleg.

#25 lore raw

Re: [PATCH] kthread: Don't depend on work queues

Quoted 17 lines · Oleg Nesterov
Quoted 16 lines · Eric W. Biederman

static int kthread(void *_create)
{

kernel/kthread.c +1 -4
 	struct kthread_create_info *create = _create;
 	int (*threadfn)(void *data);
 	void *data;
-	sigset_t blocked;
 	int ret = -EINTR;
 
-	kthread_exit_files();
-
-	/* Copy data: it's on keventd's stack */
+	/* Copy data: it's on kthread's stack */
 	threadfn = create->threadfn;
 	data = create->data;
Quoted 10 lines

kernel/kthread.c int kthread_should_stop(void) · line 97

Quoted 7 lines
kernel/kthread.c +0 -7
-	/* Block and flush all signals (in case we're not from keventd). */
-	sigfillset(&blocked);
-	sigprocmask(SIG_BLOCK, &blocked, NULL);
-	flush_signals(current);
-
-	/* By default we can run anywhere, unlike keventd. */
-	set_cpus_allowed(current, CPU_MASK_ALL);

The above is OK, but I believe you should add set_cpus_allowed() to kthreadd_setup(). Note that kthreadd() is forked after init_idle().

init_idle()? Yep in sched_init. I wondered where that happened for the initial idle thread. Ugh. That is the first thing that we fix after we fork init as well. So yes it would be a good idea to fix that...

I was expecting init_idle() to happen after we forked kthreadd. I almost wonder if I should move init_idle....

Anyway that is one bug caught.

Eric