init's children list is long and slows reaping children.

For testing, Jack Steiner create the following patch. All it does is moves tasks which are transitioning to the zombie state from where they are in the children list to the head of the list. In this way, they will be the first found and reaping does speed up. We will still do a full scan of the list once the rearranged tasks are all removed. This does not seem to be a significant problem.

Quoted 6 lines · Robin Holt

For testing, Jack Steiner create the following patch. All it does is moves tasks which are transitioning to the zombie state from where they are in the children list to the head of the list. In this way, they will be the first found and reaping does speed up. We will still do a full scan of the list once the rearranged tasks are all removed. This does not seem to be a significant problem.

I'd almost prefer to just put the zombie children on a separate list. I wonder how painful that would be..

That would still make it expensive for people who use WUNTRACED to get stopped children (since they'd have to look at all lists), but maybe that's not a big deal.

Another thing we could do is to just make sure that kernel threads simply don't end up as children of init. That whole thing is silly, they're really not children of the user-space init anyway. Comments?

Linus

Quoted 9 lines · Linus Torvalds

Quoted 6 lines · Robin Holt

For testing, Jack Steiner create the following patch. All it does is moves tasks which are transitioning to the zombie state from where they are in the children list to the head of the list. In this way, they will be the first found and reaping does speed up. We will still do a full scan of the list once the rearranged tasks are all removed. This does not seem to be a significant problem.

Quoted 3 lines

I'd almost prefer to just put the zombie children on a separate list. I wonder how painful that would be..

That would still make it expensive for people who use WUNTRACED to get stopped children (since they'd have to look at all lists), but maybe that's not a big deal.

Shouldn't be any worse than it already is.

Quoted 2 lines

Another thing we could do is to just make sure that kernel threads simply don't end up as children of init. That whole thing is silly, they're really not children of the user-space init anyway. Comments?

Quoted 1 line

Linus

Does anyone remember why we started doing this in the first place? I'm sure there are some tools that expect a process tree, rather than a forest, and making it a forest could make them unhappy.

The support angel on my shoulder says we should just put all the kernel threads under a kthread subtree to shorten init's child list and minimize impact. The hacker devil on my other shoulder says that with usermode helpers, containers, etc. it's about time we treat it as a tree, and any tools that have a problem with that need to be fixed.

-- Chris

Quoted 3 lines · Linus Torvalds

Another thing we could do is to just make sure that kernel threads simply don't end up as children of init. That whole thing is silly, they're really not children of the user-space init anyway. Comments?

Does anyone remember why we started doing this in the first place? I'm sure there are some tools that expect a process tree, rather than a forest, and making it a forest could make them unhappy.

The support angel on my shoulder says we should just put all the kernel threads under a kthread subtree to shorten init's child list and minimize impact.

Quoted 9 lines · Chris Snook

Quoted 3 lines · Linus Torvalds

Another thing we could do is to just make sure that kernel threads simply don't end up as children of init. That whole thing is silly, they're really not children of the user-space init anyway. Comments?

Does anyone remember why we started doing this in the first place? I'm sure there are some tools that expect a process tree, rather than a forest, and making it a forest could make them unhappy.

I'm not sure anybody would really be unhappy with pptr pointing to some magic and special task that has pid 0 (which makes it clear to everybody that the parent is something special), and that has SIGCHLD set to SIG_IGN (which should make the exit case not even go through the zombie phase).

I can't even imagine *how* you'd make a tool unhappy with that, since even tools like "ps" (and even more "pstree" won't read all the process states atomically, so they invariably will see parent pointers that don't even exist any more, because by the time they get to the parent, it has exited already.

Quoted 3 lines

The support angel on my shoulder says we should just put all the kernel threads under a kthread subtree to shorten init's child list and minimize impact.

A number are already there, of course, since they use the kthread infrastructure to get there.

So I think we have some options once we get the kernel threads out of the way. Getting the kernel threads out of the way would seem to be the first priority.

Quoted 3 lines

So I think we have some options once we get the kernel threads out of the way. Getting the kernel threads out of the way would seem to be the first priority.

I think both avenues would probably be the right way to proceeed. Getting kthreads to not be parented by init would be an opportunity for optimization.

I think organizing the zombie tasks to be easily reaped also has merit. Rapidly forking/exiting processes like udevd during the boot of a different machine were also shown to benefit significantly from this patch. That machine had 512 cpus and 4608 disk devices, we dropped the device discovery under udevd by 30%. This, honestly, surprised us. It makes some sense now that I think about it. This would be a case where improving the zombie handling would be beneficial to more than just kthreads.

How hard is tasklist_lock hit on these systems?

How hard is the pid hash hit on these systems?

The big benefit of the zombie list over your proposed list reordering is that waitpid can return immediately when we don't have zombies to wait for, but we have lots of children. So it looks like a universal benefit and about as good as it is possible to make zombie handling of waitpid.

Thinking about it I do agree with Linus that two lists sounds like the right solution because it ensures we always have O(1) time when waiting for a zombie.

I'd like to place the list head for the zombie
list in the signal_struct and not in the task_struct so our performance continues to be O(1) when we have a threaded process.

The big benefit of the zombie list over your proposed list reordering is that waitpid can return immediately when we don't have zombies to wait for, but we have lots of children.

Quoted 3 lines

Thinking about it I do agree with Linus that two lists sounds like the right solution because it ensures we always have O(1) time when waiting for a zombie.

Well. I bet this will be painful, and will uglify the code even more.

do_wait() has to iterate over 2 lists, __ptrace_unlink() should check ->exit_state, etc...

no. Two _completely separate_ lists.

i.e. a to-be-reaped task will still be on the main list _too_. The main list is for all the PID semantics rules. The reap-list is just for wait4() processing. The two would be completely separate.

Quoted 5 lines

no. Two _completely separate_ lists.

i.e. a to-be-reaped task will still be on the main list _too_. The main list is for all the PID semantics rules. The reap-list is just for wait4() processing. The two would be completely separate.

I guess this means we add another list head to struct task_struct. Not that big a deal, but it does make me a little sad to think about task_struct getting even bigger....

no. Two _completely separate_ lists.

i.e. a to-be-reaped task will still be on the main list _too_. The main list is for all the PID semantics rules. The reap-list is just for wait4() processing. The two would be completely separate.

Quoted 5 lines · Ingo Molnar

no. Two _completely separate_ lists.

i.e. a to-be-reaped task will still be on the main list _too_. The main list is for all the PID semantics rules. The reap-list is just for wait4() processing. The two would be completely separate.

And what pray tell except for heuristics is the list of children used for?

I could find a use in the scheduler (oldest_child and younger/older_sibling).

I could find a use in mm/oom_kill.

I could find a use in irixsig where it roles it's own version of wait4.

Perhaps I was blind but that was about it.

I didn't see the child list implementing any semantics we really care about to user space.

Quoted 6 lines

Quoted 3 lines

i.e. a to-be-reaped task will still be on the main list _too_. The main list is for all the PID semantics rules. The reap-list is just for wait4() processing. The two would be completely separate.

And what pray tell except for heuristics is the list of children used for?

yeah - by all means get rid of it, but first separate the data structures along uses. Then all the 'why should we iterate two lists in sequence' questions vanish.

Quoted 5 lines · Ingo Molnar

no. Two _completely separate_ lists.

i.e. a to-be-reaped task will still be on the main list _too_. The main list is for all the PID semantics rules. The reap-list is just for wait4() processing. The two would be completely separate.

And what pray tell except for heuristics is the list of children used for?

Quoted 8 lines · Eric W. Biederman

Quoted 5 lines · Ingo Molnar

no. Two _completely separate_ lists.

i.e. a to-be-reaped task will still be on the main list _too_. The main list is for all the PID semantics rules. The reap-list is just for wait4() processing. The two would be completely separate.

Quoted 2 lines

And what pray tell except for heuristics is the list of children used for?

on a second thought: the p->children list is needed for the whole child/parent task tree, which is needed for sys_getppid(). The question is, does anything require us to reparent to within the same thread group?

No! That is why I suggest (a long ago, in fact) to move ->children into ->signal_struct. When sub-thread forks, we set ->parent = group_leader. We don't need forget_original_parent() until the last thead exists. This also simplify do_wait().

However, this breaks the current ->pdeath_signal behaviour. In fact (and Eric thinks the same) this _fixes_ this behaviour, but may break things.

So this patch fixes the pdeath_signal behaviour only sending a signal when the results of getppid would change.

I rely on thread-to-thread pdeath notification for my emulator. I also need notification on loss of a thread-group or mm, so I do my best to emulate those based on thread-to-thread notification.

...

The way I use it pdeath_signal:

The parent is one of many tasks within a task group. The child is not within that thread group, but shares VM. (the child is single threaded, existing only to get the signal) For each task in the thread group, there is a child task waiting for parent death notification. (there are two other tasks sharing VM but not in the task group: one to do async memory searching, and one to communicate with
a user interface for debugging -- like an interactive valgrind)

Quoted 3 lines · Eric W. Biederman

So this patch fixes the pdeath_signal behaviour only sending a signal when the results of getppid would change.

Don't get me wrong, I personally like this patch very much. However,

Does a parent death signal make most sense between separately written programs?

Does a parent death signal make most sense between processes that are part of a larger program.

Quoted 1 line

Does a parent death signal make most sense between separately written programs?

I don't think it does. It has always seemed an utterly cockamamy feature to me, and I've never understood what actually motivated it.

Quoted 2 lines

Does a parent death signal make most sense between processes that are part of a larger program.

That is the only way I can really see it being used. The only actual example of use I know is what Albert Cahalan reported. To my mind, the only semantics that matter for pdeath_signal are what previous uses expected in the past and still need for compatibility. If we started with a fresh rationale from the ground up on what the feature is good for, I am rather skeptical it would pass muster to be added today.

Quoted 8 lines · Eric W. Biederman

Quoted 5 lines · Ingo Molnar

no. Two _completely separate_ lists.

i.e. a to-be-reaped task will still be on the main list _too_. The main list is for all the PID semantics rules. The reap-list is just for wait4() processing. The two would be completely separate.

Quoted 2 lines

And what pray tell except for heuristics is the list of children used for?

on a second thought: the p->children list is needed for the whole child/parent task tree, which is needed for sys_getppid().

The question
is, does anything require us to reparent to within the same thread group?

Quoted 2 lines

on a second thought: the p->children list is needed for the whole child/parent task tree, which is needed for sys_getppid().

Yes, something Oleg said made me realize that.

As long as the reparent isn't to complex it isn't required that we have exactly one list .

Quoted 2 lines

The question is, does anything require us to reparent to within the same thread group?

I think my head is finally on straight about this question.

Currently there is the silly linux specific parent death signal (pdeath_signal). Oleg's memory was a better than mine on this score.

However there is no indication that the parent death signal being sent when a thread leader dies is actually correct, or even interesting. It probably should only be sent when getppid changes.

So with pdeath_signal fixed that is nothing that requires us to reparent within the same thread group.

I'm trying to remember what the story is now. There is a nasty race somewhere with reparenting, a threaded parent setting SIGCHLD to SIGIGN, and non-default signals that results in an zombie that no one can wait for and reap. It requires being reparented twice to trigger.

Anyway it is a real mess and if we can remove the stupid multi-headed child lists things would become much simpler and the problem could not occur.

Plus the code would become much simpler...

utrace appears to have removed the ptrace_children list and the special cases that entailed.

Quoted 32 lines · Eric W. Biederman

Quoted 2 lines

on a second thought: the p->children list is needed for the whole child/parent task tree, which is needed for sys_getppid().

Yes, something Oleg said made me realize that.

As long as the reparent isn't to complex it isn't required that we have exactly one list .

Quoted 2 lines

The question is, does anything require us to reparent to within the same thread group?

I think my head is finally on straight about this question.

Currently there is the silly linux specific parent death signal (pdeath_signal). Oleg's memory was a better than mine on this score.

However there is no indication that the parent death signal being sent when a thread leader dies is actually correct, or even interesting. It probably should only be sent when getppid changes.

So with pdeath_signal fixed that is nothing that requires us to reparent within the same thread group.

I'm trying to remember what the story is now. There is a nasty race somewhere with reparenting, a threaded parent setting SIGCHLD to SIGIGN, and non-default signals that results in an zombie that no one can wait for and reap. It requires being reparented twice to trigger.

Anyway it is a real mess and if we can remove the stupid multi-headed child lists things would become much simpler and the problem could not occur.

Quoted 4 lines

Plus the code would become much simpler...

utrace appears to have removed the ptrace_children list and the special cases that entailed.

so ... is anyone pursuing this? This would allow us to make sys_wait4() faster and more scalable: no tasklist_lock bouncing for example.

Quoted 3 lines

so ... is anyone pursuing this? This would allow us to make sys_wait4() faster and more scalable: no tasklist_lock bouncing for example.

which part?

Quoted 4 lines · Eric W. Biederman

Quoted 3 lines

so ... is anyone pursuing this? This would allow us to make sys_wait4() faster and more scalable: no tasklist_lock bouncing for example.

Quoted 1 line

which part?

all of it :) Everything you mentioned makes sense quite a bit. The thread signal handling of do_wait was added in a pretty arbitrary fashion so i doubt there are strong requirements in that area. Apps might have grown to get used to it meanwhile though, so we've got to do it carefully.

here's a (tested) patch i did that should simplify changes done to p->children and p->sibling handling.

Should we copy Andrew or is someone else going to collect up all of these patches?

Quoted 1 line · On Wed, 11 Apr 2007 01:00:17 -0600 [email protected] (Eric W. Biederman)…

Should we copy Andrew or is someone else going to collect up all of these

patches?

Andrew is cowering in terror, because utrace goes tromping through this code.

It would be rather good to get utrace moving along a bit more quickly. There still seem to be rather a lot of issues.

I'm trying to remember what the story is now. There is a nasty race somewhere with reparenting, a threaded parent setting SIGCHLD to SIGIGN, and non-default signals that results in an zombie that no one can wait for and reap. It requires being reparented twice to trigger.

Quoted 4 lines · Eric W. Biederman

I'm trying to remember what the story is now. There is a nasty race somewhere with reparenting, a threaded parent setting SIGCHLD to SIGIGN, and non-default signals that results in an zombie that no one can wait for and reap. It requires being reparented twice to trigger.

reparent_thread:

	...

        /* If we'd notified the old parent about this child's death,
         * also notify the new parent.
         */
        if (!traced && p->exit_state == EXIT_ZOMBIE &&
            p->exit_signal != -1 && thread_group_empty(p))
                do_notify_parent(p, p->exit_signal);

We notified /sbin/init. If it ignores SIGCHLD, we should release the task. We don't do this.

The best fix I believe is to cleanup the forget_original_parent/reparent_thread interaction and factor out this "exit_state == EXIT_ZOMBIE && exit_signal == -1" checks.

As long as the original parent is preserved for getppid(). There are programs out there which communicate between the parent and child with signals, and if the original parent dies, it undesirable to have the child getppid() and start sending signals to a program not expecting them. Invites undefined behavior.

Quoted 4 lines · Bill Davidsen

As long as the original parent is preserved for getppid(). There are programs out there which communicate between the parent and child with signals, and if the original parent dies, it undesirable to have the child getppid() and start sending signals to a program not expecting them. Invites undefined behavior.

Then the programs are broken. getppid is defined to change if the process is reparented to init.

Quoted 18 lines · Oleg Nesterov

Quoted 4 lines · Eric W. Biederman

I'm trying to remember what the story is now. There is a nasty race somewhere with reparenting, a threaded parent setting SIGCHLD to SIGIGN, and non-default signals that results in an zombie that no one can wait for and reap. It requires being reparented twice to trigger.

reparent_thread:

	...

       /* If we'd notified the old parent about this child's death,
        * also notify the new parent.
        */
       if (!traced && p->exit_state == EXIT_ZOMBIE &&
           p->exit_signal != -1 && thread_group_empty(p))
               do_notify_parent(p, p->exit_signal);

Quoted 9 lines

We notified /sbin/init. If it ignores SIGCHLD, we should release the task. We don't do this.

The best fix I believe is to cleanup the forget_original_parent/reparent_thread
interaction and factor out this "exit_state == EXIT_ZOMBIE && exit_signal == -1"
checks.

As long as the original parent is preserved for getppid(). There are programs out there which communicate between the parent and child with signals, and if the original parent dies, it undesirable to have the child getppid() and start sending signals to a program not expecting them. Invites undefined behavior.

Quoted 4 lines · Eric W. Biederman

Thinking about it I do agree with Linus that two lists sounds like the right solution because it ensures we always have O(1) time when waiting for a zombie.

Well. I bet this will be painful, and will uglify the code even more.

do_wait() has to iterate over 2 lists, __ptrace_unlink() should check ->exit_state, etc...

Quoted 3 lines

I'd like to place the list head for the zombie
list in the signal_struct and not in the task_struct so our performance continues to be O(1) when we have a threaded process.

Sure. It would be nice to move ->children into signal_struct at first. Except this change breaks (in fact fixes) ->pdeath_signal behaviour.

Quoted 3 lines

The big benefit of the zombie list over your proposed list reordering is that waitpid can return immediately when we don't have zombies to wait for, but we have lots of children.

TASK_TRACED/TASK_STOPPED ?

Sure. It would be nice to move ->children into signal_struct at first. Except this change breaks (in fact fixes) ->pdeath_signal behaviour.

Quoted 2 lines · Oleg Nesterov

Sure. It would be nice to move ->children into signal_struct at first. Except this change breaks (in fact fixes) ->pdeath_signal behaviour.

Ohhh, the "signal" struct! Funny name for something that nowadays has probably no more than a 5% affinity with signal-related tasks :/

Quoted 2 lines · Davide Libenzi

On Fri, 6 Apr 2007, Oleg Nesterov wrote:

Quoted 5 lines

Quoted 2 lines

Sure. It would be nice to move ->children into signal_struct at first. Except this change breaks (in fact fixes) ->pdeath_signal behaviour.

Ohhh, the "signal" struct! Funny name for something that nowadays has probably no more than a 5% affinity with signal-related tasks :/

Hmm. I wonder if we should just rename it the struct thread_group, or struct task_group. Those seem slightly more accurate names.

Quoted 6 lines

Quoted 3 lines

Ohhh, the "signal" struct! Funny name for something that nowadays has probably no more than a 5% affinity with signal-related tasks :/

Hmm. I wonder if we should just rename it the struct thread_group, or struct task_group. Those seem slightly more accurate names.

Almost *anything* is better than "signal_struct" ;) A task_group could be fine, so something on the line of task_shared_ctx.

Quoted 4 lines · Davide Libenzi

Quoted 4 lines

Quoted 3 lines

Ohhh, the "signal" struct! Funny name for something that nowadays has probably no more than a 5% affinity with signal-related tasks :/

Quoted 5 lines

Quoted 2 lines

Hmm. I wonder if we should just rename it the struct thread_group, or struct task_group. Those seem slightly more accurate names.

Almost *anything* is better than "signal_struct" ;) A task_group could be fine, so something on the line of task_shared_ctx.

or lets just face it and name it what it is: process_struct ;-)

Quoted 2 lines

or lets just face it and name it what it is: process_struct ;-)

That'd be fine too! Wonder if Linus would swallow a rename patch like that...

Quoted 5 lines · Davide Libenzi

Quoted 2 lines

or lets just face it and name it what it is: process_struct ;-)

That'd be fine too! Wonder if Linus would swallow a rename patch like that...

I don't really see the point. It's not even *true*. A "process" includes more than the shared signal-handling - it would include files and fs etc too.

So it's actually *more* correct to call it the shared signal state than it would be to call it "process" state.

Quoted 7 lines · Linus Torvalds

I don't really see the point. It's not even *true*. A "process" includes more than the shared signal-handling - it would include files and fs etc too.

So it's actually *more* correct to call it the shared signal state than it would be to call it "process" state.

But "signal" has *nothing* to do with what the structure store nowadays, really. It's a pool of "things" that are not Linux task specific.

Quoted 11 lines · Davide Libenzi

Quoted 7 lines · Linus Torvalds

I don't really see the point. It's not even *true*. A "process" includes more than the shared signal-handling - it would include files and fs etc too.

So it's actually *more* correct to call it the shared signal state than it would be to call it "process" state.

But "signal" has *nothing* to do with what the structure store nowadays, really. It's a pool of "things" that are not Linux task specific.

You're ignoring reality. It has more to do with signals than with processes. Look at *all* the fields in the top half of the structure, up to (and including) the "tty" field. They're *all* about signal semantics in one form or another (whether it's directly about shared signal behaviour, or indirectly about *sources* of signals like process control or timers).

And renaming it really has no upsides, even *if* you had a point, which you don't.

Quoted 5 lines · Davide Libenzi

Quoted 2 lines

or lets just face it and name it what it is: process_struct ;-)

That'd be fine too! Wonder if Linus would swallow a rename patch like that...

I don't really see the point. It's not even *true*. A "process" includes more than the shared signal-handling - it would include files and fs etc too.

So it's actually *more* correct to call it the shared signal state than it would be to call it "process" state.

Quoted 2 lines · Linus Torvalds

Quoted 1 line · Davide Libenzi

Quoted 1 line

or lets just face it and name it what it is: process_struct ;-)

Quoted 4 lines

Quoted 1 line

That'd be fine too! Wonder if Linus would swallow a rename patch like that...

I don't really see the point. It's not even *true*. A "process" includes more than the shared signal-handling - it would include files and fs etc too.

So it's actually *more* correct to call it the shared signal state than it would be to call it "process" state.

we could call it "structure for everything that we know to be ugly about POSIX process semantics" ;-) The rest, like files and fs we've abstracted out already.

Ingo

Quoted 3 lines · Ingo Molnar

Quoted 2 lines · Linus Torvalds

Quoted 1 line · Davide Libenzi

Quoted 1 line

or lets just face it and name it what it is: process_struct ;-)

Quoted 5 lines

Quoted 4 lines

Quoted 1 line

That'd be fine too! Wonder if Linus would swallow a rename patch like that...

I don't really see the point. It's not even *true*. A "process" includes more than the shared signal-handling - it would include files and fs etc too.

So it's actually *more* correct to call it the shared signal state than it would be to call it "process" state.

we could call it "structure for everything that we know to be ugly about POSIX process semantics" ;-) The rest, like files and fs we've abstracted out already.

So are you voting for ugly_struct? ;-)

I do think this is still waiting for a more descriptive name, like proc_misc_struct or some such. Kernel code should be treated as literature, intended to be both read and readable.

Maybe "struct posix_process" is more descriptive? "struct process_posix"? "Ugly POSIX process semantics data" seems simple enough to stick in a struct name. "struct uglyposix_process"?

Quoted 4 lines · Kyle Moffett

Maybe "struct posix_process" is more descriptive? "struct process_posix"? "Ugly POSIX process semantics data" seems simple enough to stick in a struct name. "struct uglyposix_process"?

Guys, you didn't read my message.

It's *not* about "process" stuff. Anything that tries to call it a "process" is *by*definition* worse than what it is now. Processes have all the things that we've cleanly separated out for filesystem, VM, SysV semaphore state, namespaces etc.

The "struct signal_struct" is the random *leftovers* from all the other stuff. It's *not* about "processes". Never has been, and never will be.

Quoted 4 lines · Linus Torvalds

Quoted 3 lines · Kyle Moffett

Maybe "struct posix_process" is more descriptive? "struct process_posix"? "Ugly POSIX process semantics data" seems simple enough to stick in a struct name. "struct uglyposix_process"?

Quoted 5 lines

Guys, you didn't read my message.

It's *not* about "process" stuff. Anything that tries to call it a "process" is *by*definition* worse than what it is now. Processes have all the things that we've cleanly separated out for filesystem, VM, SysV semaphore state, namespaces etc.

The "struct signal_struct" is the random *leftovers* from all the other stuff. It's *not* about "processes". Never has been, and never will be.

I proposed "struct task_shared_ctx" but you ducked :)

Quoted 15 lines · Davide Libenzi

Quoted 13 lines · Linus Torvalds

Quoted 5 lines · Kyle Moffett

Maybe "struct posix_process" is more descriptive? "struct process_posix"?
"Ugly POSIX process semantics data" seems simple enough to stick in a struct
name. "struct uglyposix_process"?

Guys, you didn't read my message.

It's *not* about "process" stuff. Anything that tries to call it a "process" is *by*definition* worse than what it is now. Processes have all the things that we've cleanly separated out for filesystem, VM, SysV semaphore state, namespaces etc.

Quoted 5 lines

Quoted 2 lines

The "struct signal_struct" is the random *leftovers* from all the other stuff. It's *not* about "processes". Never has been, and never will be.

I proposed "struct task_shared_ctx" but you ducked :)

Descriptive, correct, I like it!

Maybe "struct posix_process" is more descriptive? "struct process_posix"? "Ugly POSIX process semantics data" seems simple enough to stick in a struct name. "struct uglyposix_process"?

Quoted 3 lines · Linus Torvalds

I'm not sure anybody would really be unhappy with pptr pointing to some magic and special task that has pid 0 (which makes it clear to everybody that the parent is something special), and that has SIGCHLD set to SIG_IGN (which should make the exit case not even go through the zombie phase).

I can't even imagine *how* you'd make a tool unhappy with that, since even tools like "ps" (and even more "pstree" won't read all the process states atomically, so they invariably will see parent pointers that don't even exist any more, because by the time they get to the parent, it has exited already.

Right. pid == 1 being missing might cause some confusing having but having ppid == 0 should be fine. Heck pid == 1 already has ppid == 0, so it is a value user space has had to deal with for a while.

In addition there was a period in 2.6 where most kernel threads and init had a pgid == 0 and a session == 0, and nothing seemed to complain.

We should probably make all of the kernel threads children of init_task. The initial idle thread on the first cpu that is the
parent of pid == 1. That will give the ppid == 0 naturally because
the idle thread has pid == 0.

Quoted 3 lines

The support angel on my shoulder says we should just put all the kernel threads under a kthread subtree to shorten init's child list and minimize impact.

A number are already there, of course, since they use the kthread infrastructure to get there.

As all kernel thread (1) should be converted to kthread anyway for proper containers support and general "let's get rid of a crappy API' cleanups I think that's enough. It would be nice to have SGI helping to convert more drivers over to the proper API as conversions have stalled a little bit.

(1) There's a few very core kernel threads that need to stick to the low-level API, but they're too few to make any differences.

We have been testing a new larger configuration and we are seeing a very large scan time of init's tsk->children list. In the cases we are seeing, there are numerous kernel processes created for each cpu (ie: events/0 ... events/<big number>, xfslogd/0 ... xfslogd/<big number>). These are all on the list ahead of the processes we are currently trying to reap.

I would rather change the implementation under the hood to start per-CPU threads on demand, similar to a thread-pool implementation.

Boxes with $BigNum CPUs probably won't ever use half of those threads.

Quoted 4 lines · Jeff Garzik

I would rather change the implementation under the hood to start per-CPU threads on demand, similar to a thread-pool implementation.

Boxes with $BigNum CPUs probably won't ever use half of those threads.

The counter-argument is that boxes with $BigNum CPU's really don't hurt from it either, and having per-process data structures is often simpler and more efficient than trying to have some thread pool.

IOW, once we get the processes off the global list, there just isn't any downside from them. Sure, they use some memory, but people who buy 1024-cpu machines won't care about a few kB per CPU..

So the *only* downside is literally the process list, and one suggested patch already just removes kernel threads entirely from the parenthood lists.

The other potential downside could be "ps is slow", but on the other hand, having the things stick around and have things like CPU-time accumulate is probably worth it - if there are some issues, they'd show up properly accounted for in a way that process pools would have a hard time doing.

So I really don't think this is worth changing things over, apart from literally removing them from process lists, which I think everybody agrees we should just do - it just never even came up before!

Quoted 10 lines · Linus Torvalds

Quoted 4 lines · Jeff Garzik

I would rather change the implementation under the hood to start per-CPU threads on demand, similar to a thread-pool implementation.

Boxes with $BigNum CPUs probably won't ever use half of those threads.

The counter-argument is that boxes with $BigNum CPU's really don't hurt from it either, and having per-process data structures is often simpler and more efficient than trying to have some thread pool.

Two points here:

* A lot of the users in the current kernel tree don't rely on the per-CPU qualities. They just need multiple threads running.

* Even with per-CPU data structures and code, you don't necessarily have to keep a thread alive and running for each CPU. Reap the ones that haven't been used in $TimeFrame, and add thread creation to the slow path that already exists in the bowels of schedule_work().

Or if some kernel hacker is really motivated, all workqueue users in the kernel would benefit from a "thread audit", looking at working conditions to decide if the new kthread APIs are more appropriate.

Quoted 5 lines

IOW, once we get the processes off the global list, there just isn't any downside from them. Sure, they use some memory, but people who buy 1024-cpu machines won't care about a few kB per CPU..

So the *only* downside is literally the process list, and one suggested patch already just removes kernel threads entirely from the parenthood lists.

The other potential downside could be "ps is slow", but on the other hand, having the things stick around and have things like CPU-time accumulate is probably worth it - if there are some issues, they'd show up properly accounted for in a way that process pools would have a hard time doing.

Regardless of how things are shuffled about internally, there will always be annoying overhead /somewhere/ when you have a metric ton of kernel threads. I think that people should also be working on ways to make the kernel threads a bit more manageable for the average human.

Quoted 1 line

So I really don't think this is worth changing things over, apart from literally removing them from process lists, which I think everybody agrees we should just do - it just never even came up before!

I think there is a human downside. For an admin you have to wade through a ton of processes on your machine, if you are attempting to evaluate the overall state of the machine. Just google around for all the admins complaining about the explosion of kernel threads on production machines :)

Quoted 5 lines · Robin Holt

We have been testing a new larger configuration and we are seeing a very large scan time of init's tsk->children list. In the cases we are seeing, there are numerous kernel processes created for each cpu (ie: events/0 ... events/<big number>, xfslogd/0 ... xfslogd/<big number>). These are all on the list ahead of the processes we are currently trying to reap.

What about attacking the explosion of kernel threads?

As CPU counts increase, the number of per-CPU kernel threads gets really ridiculous.

I would rather change the implementation under the hood to start per-CPU threads on demand, similar to a thread-pool implementation.

Boxes with $BigNum CPUs probably won't ever use half of those threads.

I suspect there are quite a few kernel threads which don't really need to be threads at all: the code would quite happily work if it was changed to use keventd, via schedule_work() and friends. But kernel threads are somewhat easier to code for.

I also suspect that there are a number of workqueue threads which could/should have used create_singlethread_workqueue(). Often this is because the developer just didn't think to do it.

otoh, a lot of these inefficeincies are probably down in scruffy drivers rather than in core or top-level code.

<I also wonder where all these parented-by-init, presumably-not-using-kthread kernel threads are coming from>

From another piece of this thread.

Quoted 1 line

Robin how many kernel thread per cpu are you seeing?

10.

FYI, pid 1539 is kthread.

a01:~ # ps -ef | egrep "\[.*\/255\]"

root       512     1  0 Apr08 ?        00:00:00 [migration/255]
root       513     1  0 Apr08 ?        00:00:00 [ksoftirqd/255]
root      1281     1  0 Apr08 ?        00:00:02 [events/255]
root      2435  1539  0 Apr08 ?        00:00:00 [kblockd/255]
root      3159  1539  0 Apr08 ?        00:00:00 [aio/255]
root      4007  1539  0 Apr08 ?        00:00:00 [cqueue/255]
root      8653  1539  0 Apr08 ?        00:00:00 [ata/255]
root     17438  1539  0 Apr08 ?        00:00:00 [xfslogd/255]
root     17950  1539  0 Apr08 ?        00:00:00 [xfsdatad/255]
root     18426  1539  0 Apr08 ?        00:00:00 [rpciod/255]

Quoted 9 lines · Andrew Morton

I suspect there are quite a few kernel threads which don't really need to be threads at all: the code would quite happily work if it was changed to use keventd, via schedule_work() and friends. But kernel threads are somewhat easier to code for.

I also suspect that there are a number of workqueue threads which could/should have used create_singlethread_workqueue(). Often this is because the developer just didn't think to do it.

Quoted 8 lines

otoh, a lot of these inefficeincies are probably down in scruffy drivers rather than in core or top-level code.

<I also wonder where all these parented-by-init, presumably-not-using-kthread kernel threads are coming from>

From another piece of this thread.

Quoted 7 lines

Quoted 1 line

Robin how many kernel thread per cpu are you seeing?

10.

FYI, pid 1539 is kthread.

a01:~ # ps -ef | egrep "\[.*\/255\]"

Quoted 10 lines

root       512     1  0 Apr08 ?        00:00:00 [migration/255]
root       513     1  0 Apr08 ?        00:00:00 [ksoftirqd/255]
root      1281     1  0 Apr08 ?        00:00:02 [events/255]
root      2435  1539  0 Apr08 ?        00:00:00 [kblockd/255]
root      3159  1539  0 Apr08 ?        00:00:00 [aio/255]
root      4007  1539  0 Apr08 ?        00:00:00 [cqueue/255]
root      8653  1539  0 Apr08 ?        00:00:00 [ata/255]
root     17438  1539  0 Apr08 ?        00:00:00 [xfslogd/255]
root     17950  1539  0 Apr08 ?        00:00:00 [xfsdatad/255]
root     18426  1539  0 Apr08 ?        00:00:00 [rpciod/255]

So it looks like there were about 1500 kernel threads that started up before kthread started.

So the kernel threads appear to have init as their parent is because they started before kthread for the most part.

At 10 kernel threads per cpu there may be a little bloat but it isn't out of control. It is mostly that we are observing the kernel as NR_CPUS approaches infinity. 4096 isn't infinity yet but it's easily a 1000 fold bigger then most people are used to :)

At 10 kernel threads per cpu there may be a little bloat but it isn't out of control. It is mostly that we are observing the kernel as NR_CPUS approaches infinity. 4096 isn't infinity yet but it's easily a 1000 fold bigger then most people are used to :)

Quoted 25 lines · Andrew Morton

I suspect there are quite a few kernel threads which don't really need to be threads at all: the code would quite happily work if it was changed to use keventd, via schedule_work() and friends. But kernel threads are somewhat easier to code for.

I also suspect that there are a number of workqueue threads which could/should have used create_singlethread_workqueue(). Often this is because the developer just didn't think to do it.

otoh, a lot of these inefficeincies are probably down in scruffy drivers rather than in core or top-level code.

<I also wonder where all these parented-by-init, presumably-not-using-kthread kernel threads are coming from>

From another piece of this thread.

Quoted 1 line

Robin how many kernel thread per cpu are you seeing?

10.

FYI, pid 1539 is kthread.

a01:~ # ps -ef | egrep "\[.*\/255\]"

Quoted 10 lines

root       512     1  0 Apr08 ?        00:00:00 [migration/255]
root       513     1  0 Apr08 ?        00:00:00 [ksoftirqd/255]
root      1281     1  0 Apr08 ?        00:00:02 [events/255]
root      2435  1539  0 Apr08 ?        00:00:00 [kblockd/255]
root      3159  1539  0 Apr08 ?        00:00:00 [aio/255]
root      4007  1539  0 Apr08 ?        00:00:00 [cqueue/255]
root      8653  1539  0 Apr08 ?        00:00:00 [ata/255]
root     17438  1539  0 Apr08 ?        00:00:00 [xfslogd/255]
root     17950  1539  0 Apr08 ?        00:00:00 [xfsdatad/255]
root     18426  1539  0 Apr08 ?        00:00:00 [rpciod/255]

So it looks like there were about 1500 kernel threads that started up before kthread started.

So the kernel threads appear to have init as their parent is because they started before kthread for the most part.

At 10 kernel threads per cpu there may be a little bloat but it isn't out of control. It is mostly that we are observing the kernel as NR_CPUS approaches infinity. 4096 isn't infinity yet but it's easily a 1000 fold bigger then most people are used to :)

Eric

I suspect there are quite a few kernel threads which don't really need to be threads at all: the code would quite happily work if it was changed to use keventd, via schedule_work() and friends. But kernel threads are somewhat easier to code for.

otoh, a lot of these inefficeincies are probably down in scruffy drivers rather than in core or top-level code.

Quoted 4 lines · Andrew Morton

I suspect there are quite a few kernel threads which don't really need to be threads at all: the code would quite happily work if it was changed to use keventd, via schedule_work() and friends. But kernel threads are somewhat easier to code for.

Perhaps too easy. We have a bunch of kthreads sitting around that afaict, are there 'just in case', not because they're actually in use.

   10 ?        S<     0:00 [khelper]

Why doesn't this get spawned when it needs to?

  164 ?        S<     0:00 [cqueue/0]
  165 ?        S<     0:00 [cqueue/1]

I'm not even sure wth these are.

...

Quoted 2 lines

otoh, a lot of these inefficeincies are probably down in scruffy drivers rather than in core or top-level code.

You say scruffy, but most of the proliferation of kthreads comes from code written in the last few years. Compare the explosion of kthreads we see coming from 2.4 to 2.6. It's disturbing, and I don't see it slowing down at all.

On the 2-way box I grabbed the above ps output from, I end up with 69 kthreads. It doesn't surprise me at all that bigger iron is starting to see issues.

Quoted 2 lines

   10 ?        S<     0:00 [khelper]

That one's needed to parent the call_usermodehelper() apps. I don't think it does anything else. We used to use keventd for this but that had some problem whcih I forget.

Quoted 4 lines · Andrew Morton

Quoted 1 line

   10 ?        S<     0:00 [khelper]

That one's needed to parent the call_usermodehelper() apps. I don't think it does anything else. We used to use keventd for this but that had some problem whcih I forget.

I think it was one of a long series of deadlocks.

Using a "keventd" for many different things sounds clever and nice, but then sucks horribly when one event triggers another event, and they depend on each other. Solution: use independent threads for the events.

That's why I feel thread creation -- cheap under Linux -- is quite appropriate for many of these situations.

On Mon, 9 Apr 2007 21:59:12 -0400 Dave Jones <[email protected]> wrote:

Quoted 4 lines

  164 ?        S<     0:00 [cqueue/0]
  165 ?        S<     0:00 [cqueue/1]

I'm not even sure wth these are.

Me either.

Quoted 1 line · Andrew Morton

On Mon, 9 Apr 2007 21:59:12 -0400 Dave Jones <[email protected]> wrote:

[possible topic for KS2007]

Quoted 3 lines

Quoted 3 lines

  164 ?        S<     0:00 [cqueue/0]
  165 ?        S<     0:00 [cqueue/1]

Quoted 3 lines

Quoted 1 line

I'm not even sure wth these are.

Me either.

drivers/connector/connector.c:
455 dev->cbdev = cn_queue_alloc_dev("cqueue", dev->nls);

: root       299  0.0  0.0      0     0 ?        S    18:51   0:00 [scsi_eh_0]
: root       300  0.0  0.0      0     0 ?        S    18:51   0:00 [scsi_eh_1]
: root       305  0.0  0.0      0     0 ?        S    18:51   0:00 [scsi_eh_2]
: root       306  0.0  0.0      0     0 ?        S    18:51   0:00 [scsi_eh_3]

This machine has one CPU, one sata disk and one DVD drive. The above is hard to explain.

I don't think it's completely silly to object to all this. Sure, a kernel thread is worth 4k in the best case, but I bet they have associated unused resources and as we've seen, they can cause overhead.

One thread per port, not per device.

 796 ?        S      0:00  \_ [scsi_eh_0]
 797 ?        S      0:00  \_ [scsi_eh_1]
 798 ?        S      0:00  \_ [scsi_eh_2]
 819 ?        S      0:00  \_ [scsi_eh_3]
 820 ?        S      0:00  \_ [scsi_eh_4]
 824 ?        S      0:00  \_ [scsi_eh_5]
 825 ?        S      0:14  \_ [scsi_eh_6]

bardioc ~ # lsscsi -d

[0:0:0:0]    disk    ATA      ST3160827AS      3.42  /dev/sda[8:0]
[1:0:0:0]    disk    ATA      ST3160827AS      3.42  /dev/sdb[8:16]
[5:0:0:0]    disk    ATA      IBM-DHEA-36480   HE8O  /dev/sdc[8:32]
[5:0:1:0]    disk    ATA      Maxtor 6L160P0   BAH4  /dev/sdd[8:48]
[6:0:0:0]    cd/dvd  HL-DT-ST DVDRAM GSA-4081B A100  /dev/sr0[11:0]
bardioc ~ # lsscsi -H
[0]    sata_promise
[1]    sata_promise
[2]    sata_promise
[3]    sata_via
[4]    sata_via
[5]    pata_via
[6]    pata_via

The bad is, that there is always a thread, even if the hardware is not even hotplug capable.
Don't know if the thread is even needed for hotplug...

: root 3 0.0 0.0 0 0 ? S 18:51 0:00 [watchdog/0]

That's the softlockup detector. Confusingly named to look like a, err, watchdog. Could probably use keventd.

: root 5 0.0 0.0 0 0 ? S 18:51 0:00 [khelper]

That's there to parent the kthread_create()d threads. Could presumably use khelper.

: root 152 0.0 0.0 0 0 ? S 18:51 0:00 [ata/0]

Does it need to be per-cpu?

: root 153 0.0 0.0 0 0 ? S 18:51 0:00 [ata_aux]

That's a single-threaded workqueue handler. Perhaps could use keventd.

: root       299  0.0  0.0      0     0 ?        S    18:51   0:00 [scsi_eh_0]
: root       300  0.0  0.0      0     0 ?        S    18:51   0:00 [scsi_eh_1]
: root       305  0.0  0.0      0     0 ?        S    18:51   0:00 [scsi_eh_2]
: root       306  0.0  0.0      0     0 ?        S    18:51   0:00 [scsi_eh_3]

This machine has one CPU, one sata disk and one DVD drive. The above is hard to explain.

: root 319 0.0 0.0 0 0 ? S 18:51 0:00 [pccardd]

hm.

: root 331 0.0 0.0 0 0 ? S 18:51 0:00 [kpsmoused]

hm.

: root 337 0.0 0.0 0 0 ? S 18:51 0:00 [kedac]

hm. I didn't know that the Vaio had EDAC.

: root 1173 0.0 0.0 0 0 ? S 18:51 0:00 [khpsbpkt]

I can't even pronounce that.

: root 1354 0.0 0.0 0 0 ? S 18:51 0:00 [knodemgrd_0]

OK, I do have 1394 hardware, but it hasn't been used.

: root 1636 0.0 0.0 0 0 ? S 18:52 0:00 [kondemand/0]

I blame davej.

Quoted 11 lines

Quoted 2 lines

otoh, a lot of these inefficeincies are probably down in scruffy drivers rather than in core or top-level code.

You say scruffy, but most of the proliferation of kthreads comes from code written in the last few years. Compare the explosion of kthreads we see coming from 2.4 to 2.6. It's disturbing, and I don't see it slowing down at all.

On the 2-way box I grabbed the above ps output from, I end up with 69 kthreads. It doesn't surprise me at all that bigger iron is starting to see issues.

Sure.

I don't think it's completely silly to object to all this. Sure, a kernel thread is worth 4k in the best case, but I bet they have associated unused resources and as we've seen, they can cause overhead.

My main
worry with keventd is that we might get stuck behind an unrelated process for an undefined length of time.

Quoted 1 line · On Tue, 10 Apr 2007 03:05:56 -0400 Jeff Garzik <[email protected]> wrote:

My main worry with keventd is that we might get stuck behind an unrelated process for an undefined length of time.

I don't think it has ever been demonstrated that keventd latency is excessive, or a problem. I guess we could instrument it and fix stuff easily enough.

The main problem with keventd has been flush_scheduled_work() deadlocks: the

The thing to concentrate on here is the per-cpu threads, which is where the proliferation appears to be coming from.

Quoted 9 lines · Andrew Morton

Quoted 3 lines · On Tue, 10 Apr 2007 03:05:56 -0400 Jeff Garzik <[email protected]> wrote:

My main
worry with keventd is that we might get stuck behind an unrelated process for an undefined length of time.

I don't think it has ever been demonstrated that keventd latency is excessive, or a problem. I guess we could instrument it and fix stuff easily enough.

It's simple math, combined with user expectations.

On a 1-CPU or 2-CPU box, if you have three or more tasks, all of which are doing hardware reset tasks that could take 30-60 seconds (realistic for libata, SCSI and network drivers, at least), then OBVIOUSLY you have other tasks blocked for that length of time.

Well that obviously would be a dumb way to use keventd. One would need to do schedule_work(), kick off the reset then do schedule_delayed_work() to wait (or poll) for its termination.

Thus, rather than forcing authors to make their code more complex, we should find another solution.

Quoted 5 lines · Andrew Morton

: root 3 0.0 0.0 0 0 ? S 18:51 0:00
[watchdog/0]

That's the softlockup detector. Confusingly named to look like a, err, watchdog. Could probably use keventd.

I would think this would run into the keventd "problem", where $N processes can lock out another?

IMO a lot of these could potentially be simply started as brand new threads, when an exception arises.

Quoted 9 lines

: root 5 0.0 0.0 0 0 ? S 18:51 0:00
[khelper]

That's there to parent the kthread_create()d threads. Could presumably use khelper.

: root 152 0.0 0.0 0 0 ? S 18:51 0:00 [ata/0]

Does it need to be per-cpu?

No, it does not.

It is used for PIO data transfer, so it merely has to respond quickly, which rules out keventd. You also don't want PIO data xfer for port A blocked, sitting around waiting for PIO data xfer to complete on port C.

So, we merely need fast-reacting threads that keventd will not block. We do not need per-CPU threads.

Again, I think a model where threads are created on demand, and reaped after inactivity, would fit here. As I feel it would fit with many other non-libata drivers.

Quoted 4 lines

: root 153 0.0 0.0 0 0 ? S 18:51 0:00
[ata_aux]

That's a single-threaded workqueue handler. Perhaps could use keventd.

That is used by libata exception handler, for hotpug and such. My main worry with keventd is that we might get stuck behind an unrelated process for an undefined length of time.

IMO the best model would be to create ata_aux thread on demand, and kill it if it hasn't been used recently.

Quoted 11 lines

: root 299 0.0 0.0 0 0 ? S 18:51 0:00
[scsi_eh_0]
: root 300 0.0 0.0 0 0 ? S 18:51 0:00
[scsi_eh_1]
: root 305 0.0 0.0 0 0 ? S 18:51 0:00
[scsi_eh_2]
: root 306 0.0 0.0 0 0 ? S 18:51 0:00
[scsi_eh_3]

This machine has one CPU, one sata disk and one DVD drive. The above is hard to explain.

Nod. I've never thought we needed this many threads. At least it doesn't scale out of control for $BigNum-CPU boxen.

As the name implies, this is SCSI exception handling thread. Although some synchronization is required, this could probably work with an on-demand thread creation model too.

Quoted 9 lines

: root 319 0.0 0.0 0 0 ? S 18:51 0:00
[pccardd]

hm.

: root 331 0.0 0.0 0 0 ? S 18:51 0:00
[kpsmoused]

hm.

This kernel thread is used as a "bottom half" handler for the PS2 mouse interrupt. This one is a bit more justifiable.

Quoted 34 lines

: root 337 0.0 0.0 0 0 ? S 18:51 0:00 [kedac]

hm. I didn't know that the Vaio had EDAC.

: root 1173 0.0 0.0 0 0 ? S 18:51 0:00
[khpsbpkt]

I can't even pronounce that.

: root 1354 0.0 0.0 0 0 ? S 18:51 0:00
[knodemgrd_0]

OK, I do have 1394 hardware, but it hasn't been used.

: root 1636 0.0 0.0 0 0 ? S 18:52 0:00
[kondemand/0]

I blame davej.

Quoted 14 lines

Quoted 1 line

otoh, a lot of these inefficeincies are probably down in scruffy

drivers

Quoted 1 line

rather than in core or top-level code.

You say scruffy, but most of the proliferation of kthreads comes from code written in the last few years. Compare the explosion of kthreads
we see coming from 2.4 to 2.6. It's disturbing, and I don't see it slowing down at all.

On the 2-way box I grabbed the above ps output from, I end up with 69 kthreads.
It doesn't surprise me at all that bigger iron is starting to see issues.

Quoted 5 lines

Sure.

I don't think it's completely silly to object to all this. Sure, a kernel thread is worth 4k in the best case, but I bet they have associated unused resources and as we've seen, they can cause overhead.

Agreed.

: root 319 0.0 0.0 0 0 ? S 18:51 0:00 [pccardd]

hm.

Could probably be eliminated if we had some mechanism to spawn a helper thread to do some task as required which didn't block other helper threads until it completes.

One per PC card socket to avoid the sysfs locking crappyness that would otherwise deadlock, and to convert from the old unreadable state machine implementation to a much more readable linearly coded implementation.

Could probably be eliminated if we had some mechanism to spawn a helper thread to do some task as required which didn't block other helper threads until it completes.

Quoted 3 lines · Russell King

One per PC card socket to avoid the sysfs locking crappyness that would otherwise deadlock, and to convert from the old unreadable state machine implementation to a much more readable linearly coded implementation.

Could probably be eliminated if we had some mechanism to spawn a helper thread to do some task as required which didn't block other helper threads until it completes.

looks like the perfect usecase for threadlets. (threadlets only use up a separate context if necessary and can be coded in the familiar sequential/linear model)