How to handle the rules engine for cgroups

Hi,

While development is going on for cgroup and various controllers, we also need a facility so that an admin/user can specify the group creation and also specify the rules based on which tasks should be placed in respective groups. Group creation part will be handled by libcg which is already under development. We still need to tackle the issue of how to specify the rules and how these rules are enforced (rules engine).

I have gathered few views, with regards to how rule engine can possibly be implemented, I am listing these down.

Proposal 1
==========
Let user space daemon hanle all that. Daemon will open a netlink socket and receive the notifications for various kernel events. Daemon will also parse appropriate admin specified rules config file and place the processes in right cgroup based on rules as and when events happen.

I have written a prototype user space program which does that. Program can be found here. Currently it is in very crude shape.

http://people.redhat.com/vgoyal/misc/rules-engine-daemon/user-id-based-namespaces.patch

Various people have raised two main issues with this approach.

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

- How to handle delays in rule exectuion?
- For example, if an "exec" happens and by the time process is moved to right group, it might have forked off few more processes or might have done quite some amount of memory allocation which will be charged to the wring group. Or, newly exec process might get killed in existing cgroup because of lack of memory (despite the fact that destination cgroup has sufficient memory).

Proposal 2
==========
Implement one or more kernel modules which will implement the rule engine. User space program can parse the config files and pass it to module. Kernel will be patched only on select points to look for the rules (as provided by modules). Very minimal code running inside the kernel if there are no rules loaded.

Concerns:
- Rules can become complex and we don't want to handle that complexity in kernel.

Pros:
- Reliable and precise movement of tasks in right cgroup based on rules.

Proposal 3
==========
How about if additional parameters can be passed to system calls and one can pass destination cgroup as additional parameter. Probably something like sys_indirect proposal. Maybe glibc can act as a wrapper to pass additional parameter so that applications don't need any modifications.

Concerns:
========
- Looks like sys_indirect interface for passing extra flags was rejected.
- Requires extra work in glibc which can also involve parsing of rule files. :-(

Proposal 4
==========
Some vauge thoughts are there regarding how about kind of freezing the process or thread upon fork, exec and unfreeze it once the thread has been placed in right cgroup.

Concerns:
========
- Requires reliable netlink protocol otherwise there is a possibility that a task never gets unfrozen.
- On what basis does one freeze a thread. There might not be any rules to process for that thread we will unnecessarily delay it.

Please provide your inputs regarding what's the best way to handle the rules engine.

To me, letting the rules live in separate module/modules seems to be a reasonable way to move forward which will provide reliable and timely execution of rules and by making it modular, we can remove most of the complexity from core kernel code.

Hi,

While development is going on for cgroup and various controllers, we also need a facility so that an admin/user can specify the group creation and also specify the rules based on which tasks should be placed in respective groups. Group creation part will be handled by libcg which is already under development. We still need to tackle the issue of how to specify the rules and how these rules are enforced (rules engine).

I have gathered few views, with regards to how rule engine can possibly be implemented, I am listing these down.

Proposal 1
==========
Let user space daemon hanle all that. Daemon will open a netlink socket and receive the notifications for various kernel events. Daemon will also parse appropriate admin specified rules config file and place the processes in right cgroup based on rules as and when events happen.

I have written a prototype user space program which does that. Program can be found here. Currently it is in very crude shape.

http://people.redhat.com/vgoyal/misc/rules-engine-daemon/user-id-based-namespaces.patch

Various people have raised two main issues with this approach.

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

- How to handle delays in rule exectuion?
- For example, if an "exec" happens and by the time process is moved to right group, it might have forked off few more processes or might have done quite some amount of memory allocation which will be charged to the wring group. Or, newly exec process might get killed in existing cgroup because of lack of memory (despite the fact that destination cgroup has sufficient memory).

Quoted 30 lines · On Tue, 1 Jul 2008 15:11:26 -0400 Vivek Goyal <[email protected]> wrote:

Hi,

While development is going on for cgroup and various controllers, we also need a facility so that an admin/user can specify the group creation and also specify the rules based on which tasks should be placed in respective groups. Group creation part will be handled by libcg which is already under development. We still need to tackle the issue of how to specify the rules and how these rules are enforced (rules engine).

I have gathered few views, with regards to how rule engine can possibly be implemented, I am listing these down.

Proposal 1
==========
Let user space daemon hanle all that. Daemon will open a netlink socket and receive the notifications for various kernel events. Daemon will also parse appropriate admin specified rules config file and place the processes in right cgroup based on rules as and when events happen.

I have written a prototype user space program which does that. Program can be found here. Currently it is in very crude shape.

http://people.redhat.com/vgoyal/misc/rules-engine-daemon/user-id-based-namespaces.patch

Various people have raised two main issues with this approach.

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

Quoted 8 lines

- How to handle delays in rule exectuion?
- For example, if an "exec" happens and by the time process is moved to right group, it might have forked off few more processes or might have done quite some amount of memory allocation which will be charged to the wring group. Or, newly exec process might get killed in existing cgroup because of lack of memory (despite the fact that destination cgroup has sufficient memory).

Hmm, can't we rework the process event connector to use some reliable fast interface besides netlink ? (I mean an interface like eventpoll.) (Or enhance netlink ? ;)

Because "a child inherits parent's" rule is very strong, I think the amount of events we have to check is much less than we get report. Can't we add some filter/assumption here ?

BTW, the placement of proc_exec_connector() is not too late ? It seems memory for creating exec-image is charged to original group...

Quoted 10 lines · KAMEZAWA Hiroyuki

Quoted 8 lines · On Tue, 1 Jul 2008 15:11:26 -0400 Vivek Goyal <[email protected]> wrote:

- How to handle delays in rule exectuion?
- For example, if an "exec" happens and by the time process is moved to right group, it might have forked off few more processes or might have done quite some amount of memory allocation which will be charged to the wring group. Or, newly exec process might get killed in existing cgroup because of lack of memory (despite the fact that destination cgroup has sufficient memory).

Quoted 3 lines

Hmm, can't we rework the process event connector to use some reliable fast interface besides netlink ? (I mean an interface like eventpoll.) (Or enhance netlink ? ;)

I see following text in netlink man page.

"However, reliable transmissions from kernel to user are impossible in
any case. The kernel can’t send a netlink message if the socket buffer is full: the message will be dropped and the kernel and the userspace process will no longer have the same view of kernel state. It is up to the application to detect when this happens (via the ENOBUFS error returned by recvmsg(2)) and resynchronize."

So at the end of the day, it looks like unreliability comes from the fact that we can not allocate memory currently so we will discard the packet.

Are there alternatives as compared to dropping packets?

- Let sender cache the packet and retry later. So maybe netlink layer can return error if packet can not be queued and connector can cache the event and try sending it later. (Hopefully later memory situation became better because of OOM or some process exited or something else...).

This looks like a band-aid to handle the temporary congestion kind of problems. Will not be able to help if consumer is inherently slow and event generation is faster.

This probably can be one possible enhancement to connector, but at the end of the day, any kind of user space daemon will have to accept the fact that packets can be dropped, leading to lost events. Detect that situation (using ENOBUFS) and then let admin know about it (logging). I am not sure what admin is supposed to do after that.

I am CCing Thomas Graf. He might have a better idea of netlink limitations and is there a way to overcome these.

Quoted 5 lines

Because "a child inherits parent's" rule is very strong, I think the amount of events we have to check is much less than we get report. Can't we add some filter/assumption here ?

I am not sure if proc connector currently allows filtering of various events like fork, exec, exit etc. In a quick look it looks like it does not. But probably that can be worked out. Even then, it will just help reduce the number of messages queued for user space on that socket but will not take away the fact that messages can be dropped under memory pressure.

Quoted 3 lines

BTW, the placement of proc_exec_connector() is not too late ? It seems memory for creating exec-image is charged to original group...

As of today it should happen because newly execed process will run into same cgroup as parent. But that's what probably we need to avoid.

For example, if an admin has created three cgroups "database", "browser" "others" and a user launches "firefox" from shell (assuming shell is running originally in "others" cgroup), then any memory allocation for firefox should come from "browser" cgroup and not from "others".

I am assuming that this will be a requirement for enterprise class systems. Would be good to know the experiences of people who are already doing some kind of work load management.

Thanks
Vivek

Quoted 4 lines

Because "a child inherits parent's" rule is very strong, I think the amount of events we have to check is much less than we get report. Can't we add some filter/assumption here ?

I am not sure if proc connector currently allows filtering of various events like fork, exec, exit etc. In a quick look it looks like it does not. But probably that can be worked out. Even then, it will just help reduce the number of messages queued for user space on that socket but will not take away the fact that messages can be dropped under memory pressure.

Quoted 41 lines · KAMEZAWA Hiroyuki

Quoted 30 lines · On Tue, 1 Jul 2008 15:11:26 -0400 Vivek Goyal <[email protected]> wrote:

Hi,

While development is going on for cgroup and various controllers, we also need a facility so that an admin/user can specify the group creation and also specify the rules based on which tasks should be placed in respective groups. Group creation part will be handled by libcg which is already under development. We still need to tackle the issue of how to specify the rules and how these rules are enforced (rules engine).

I have gathered few views, with regards to how rule engine can possibly be implemented, I am listing these down.

Proposal 1
==========
Let user space daemon hanle all that. Daemon will open a netlink socket and receive the notifications for various kernel events. Daemon will also parse appropriate admin specified rules config file and place the processes in right cgroup based on rules as and when events happen.

I have written a prototype user space program which does that. Program can be found here. Currently it is in very crude shape.

http://people.redhat.com/vgoyal/misc/rules-engine-daemon/user-id-based-namespaces.patch

Various people have raised two main issues with this approach.

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

Quoted 8 lines

- How to handle delays in rule exectuion?
- For example, if an "exec" happens and by the time process is moved to right group, it might have forked off few more processes or might have done quite some amount of memory allocation which will be charged to the wring group. Or, newly exec process might get killed in existing cgroup because of lack of memory (despite the fact that destination cgroup has sufficient memory).

Quoted 3 lines

Hmm, can't we rework the process event connector to use some reliable fast interface besides netlink ? (I mean an interface like eventpoll.) (Or enhance netlink ? ;)

I see following text in netlink man page.

"However, reliable transmissions from kernel to user are impossible in
any case. The kernel can’t send a netlink message if the socket buffer is full: the message will be dropped and the kernel and the userspace process will no longer have the same view of kernel state. It is up to the application to detect when this happens (via the ENOBUFS error returned by recvmsg(2)) and resynchronize."

So at the end of the day, it looks like unreliability comes from the fact that we can not allocate memory currently so we will discard the packet.

Are there alternatives as compared to dropping packets?

- Let sender cache the packet and retry later. So maybe netlink layer can return error if packet can not be queued and connector can cache the event and try sending it later. (Hopefully later memory situation became better because of OOM or some process exited or something else...).

This looks like a band-aid to handle the temporary congestion kind of problems. Will not be able to help if consumer is inherently slow and event generation is faster.

This probably can be one possible enhancement to connector, but at the end of the day, any kind of user space daemon will have to accept the fact that packets can be dropped, leading to lost events. Detect that situation (using ENOBUFS) and then let admin know about it (logging). I am not sure what admin is supposed to do after that.

I am CCing Thomas Graf. He might have a better idea of netlink limitations and is there a way to overcome these.

Quoted 4 lines

Because "a child inherits parent's" rule is very strong, I think the amount of events we have to check is much less than we get report. Can't we add some filter/assumption here ?

I am not sure if proc connector currently allows filtering of various events like fork, exec, exit etc. In a quick look it looks like it does not. But probably that can be worked out. Even then, it will just help reduce the number of messages queued for user space on that socket but will not take away the fact that messages can be dropped under memory pressure.

Quoted 3 lines

BTW, the placement of proc_exec_connector() is not too late ? It seems memory for creating exec-image is charged to original group...

As of today it should happen because newly execed process will run into same cgroup as parent. But that's what probably we need to avoid. For example, if an admin has created three cgroups "database", "browser" "others" and a user launches "firefox" from shell (assuming shell is running originally in "others" cgroup), then any memory allocation for firefox should come from "browser" cgroup and not from "others".

I am assuming that this will be a requirement for enterprise class systems. Would be good to know the experiences of people who are already doing some kind of work load management.

Quoted 76 lines · Vivek Goyal

Quoted 41 lines · KAMEZAWA Hiroyuki

Quoted 30 lines · On Tue, 1 Jul 2008 15:11:26 -0400 Vivek Goyal <[email protected]> wrote:

Hi,

While development is going on for cgroup and various controllers, we also need a facility so that an admin/user can specify the group creation and also specify the rules based on which tasks should be placed in respective groups. Group creation part will be handled by libcg which is already under development. We still need to tackle the issue of how to specify the rules and how these rules are enforced (rules engine).

I have gathered few views, with regards to how rule engine can possibly be implemented, I am listing these down.

Proposal 1
==========
Let user space daemon hanle all that. Daemon will open a netlink socket and receive the notifications for various kernel events. Daemon will also parse appropriate admin specified rules config file and place the processes in right cgroup based on rules as and when events happen.

I have written a prototype user space program which does that. Program can be found here. Currently it is in very crude shape.

http://people.redhat.com/vgoyal/misc/rules-engine-daemon/user-id-based-namespaces.patch

Various people have raised two main issues with this approach.

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

Quoted 8 lines

- How to handle delays in rule exectuion?
- For example, if an "exec" happens and by the time process is moved to right group, it might have forked off few more processes or might have done quite some amount of memory allocation which will be charged to the wring group. Or, newly exec process might get killed in existing cgroup because of lack of memory (despite the fact that destination cgroup has sufficient memory).

Quoted 3 lines

Hmm, can't we rework the process event connector to use some reliable fast interface besides netlink ? (I mean an interface like eventpoll.) (Or enhance netlink ? ;)

I see following text in netlink man page.

"However, reliable transmissions from kernel to user are impossible in
any case. The kernel can’t send a netlink message if the socket buffer is full: the message will be dropped and the kernel and the userspace process will no longer have the same view of kernel state. It is up to the application to detect when this happens (via the ENOBUFS error returned by recvmsg(2)) and resynchronize."

So at the end of the day, it looks like unreliability comes from the fact that we can not allocate memory currently so we will discard the packet.

Are there alternatives as compared to dropping packets?

- Let sender cache the packet and retry later. So maybe netlink layer can return error if packet can not be queued and connector can cache the event and try sending it later. (Hopefully later memory situation became better because of OOM or some process exited or something else...).

This looks like a band-aid to handle the temporary congestion kind of problems. Will not be able to help if consumer is inherently slow and event generation is faster.

This probably can be one possible enhancement to connector, but at the end of the day, any kind of user space daemon will have to accept the fact that packets can be dropped, leading to lost events. Detect that situation (using ENOBUFS) and then let admin know about it (logging). I am not sure what admin is supposed to do after that.

Quoted 3 lines

I am CCing Thomas Graf. He might have a better idea of netlink limitations and is there a way to overcome these.

One thing we did with the delay accounting framework was to add the ability for clients to listen on a per-cpu basis, that helped us scale well (user space buffers per-client in turn per-cpu)

Quoted 23 lines

Quoted 4 lines

Because "a child inherits parent's" rule is very strong, I think the amount of events we have to check is much less than we get report. Can't we add some filter/assumption here ?

I am not sure if proc connector currently allows filtering of various events like fork, exec, exit etc. In a quick look it looks like it does not. But probably that can be worked out. Even then, it will just help reduce the number of messages queued for user space on that socket but will not take away the fact that messages can be dropped under memory pressure.

Quoted 3 lines

BTW, the placement of proc_exec_connector() is not too late ? It seems memory for creating exec-image is charged to original group...

As of today it should happen because newly execed process will run into same cgroup as parent. But that's what probably we need to avoid. For example, if an admin has created three cgroups "database", "browser" "others" and a user launches "firefox" from shell (assuming shell is running originally in "others" cgroup), then any memory allocation for firefox should come from "browser" cgroup and not from "others".

Quoted 3 lines

I am assuming that this will be a requirement for enterprise class systems. Would be good to know the experiences of people who are already doing some kind of work load management.

CKRM had a kernel module for rule based classification - called rule based classification engine (rbce). We should consider a simple cgroups client that can share a database from user space and use the fork callback for classification.

As of today it should happen because newly execed process will run into same cgroup as parent. But that's what probably we need to avoid. For example, if an admin has created three cgroups "database", "browser" "others" and a user launches "firefox" from shell (assuming shell is running originally in "others" cgroup), then any memory allocation for firefox should come from "browser" cgroup and not from "others".

I am assuming that this will be a requirement for enterprise class systems. Would be good to know the experiences of people who are already doing some kind of work load management.

Quoted 7 lines · Vivek Goyal

As of today it should happen because newly execed process will run into same cgroup as parent. But that's what probably we need to avoid. For example, if an admin has created three cgroups "database", "browser" "others" and a user launches "firefox" from shell (assuming shell is running originally in "others" cgroup), then any memory allocation for firefox should come from "browser" cgroup and not from "others".

I think that I'm a little skeptical that anyone would ever want to do that.

Wouldn't it be a simpler mechanism for the admin to simply have wrappers around the "firefox" and "oracle" binaries that move the process into the "browser" or "database" cgroup before running the real binaries?

Quoted 4 lines

I am assuming that this will be a requirement for enterprise class systems. Would be good to know the experiences of people who are already doing some kind of work load management.

I can help there. :-) At Google we have two approaches:

- grid jobs, which are moved into the appropriate cgroup (actually, currently cpuset) by the grid daemon when it starts the job

- ssh logins, which are moved into the appropriate cpuset by a forced-command script specified in the sshd config.

I don't see the rule-based approach being all that useful for our needs.

It's all very well coming up with theoretical cases that a fancy new mechanism solves. But it carries more weight if someone can stand up and say "Yes, I want to use this on my real cluster of machines". (Or even "Yes, if this is implemented I *will* use it on my desktop" would be a start)

So it boils down to.

1) Can we bear the delay in task classification (Especially, exec). If yes, then all the classification job can take place in userspace.

2) If no,
a) Then either we need to implement rule based engine to let kernel do classfication.

b) or we need to do various things in user space as you suggested.
- Pur wrapper around applications.
- Job launcher (ex. Grid daemon) is modified to determine the right cgroup and place application there before actually launching the job.

Well, that would mean first wrappers need to be created around all the applications which needs to be controlled. Then wrapper needs to synchronize with the classification daemon

Quoted 2 lines

- grid jobs, which are moved into the appropriate cgroup (actually, currently cpuset) by the grid daemon when it starts the job

So grid daemon probably first forks off, determines the right cpuset move the job there and then do exec?

I don't see the rule-based approach being all that useful for our needs.

Quoted 1 line · On Thu, 10 Jul 2008 02:23:52 -0700 "Paul Menage" <[email protected]> wrote:

I don't see the rule-based approach being all that useful for our needs.

Agreed, there really is no need for a rule-based approach in kernel space.

There are basically three different cases:

1) daemons get started up in their own process groups, this can be handled by the initscripts

2) user sessions (ssh, etc) start in their own process groups, this can be handled by PAM

3) users fork processes that should go into special process groups - this could be handled by having a small ruleset in userspace handle things, right before calling exec(),

it can even be hidden from the application by hooking into the exec() call

Quoted 3 lines

it can even be hidden from the application by hooking into the exec() call

This means hooking into libc. So libc will parse rules file, determine the right cgroup, place application there and then call exec?

Once the information is available, how is it used? We'd have to pass additional information to the exec syscalls. And it has to happen so that if the exec call fails the original process is not affected (i.e., premature changing isn't an option). The method also must be thread-safe in a limited way: executing failing exec syscalls in multiple threads mustn't disturb the process.

There is one set of problems which I don't care about but others likely
will: what happens if some program uses the syscalls directly? And what happens with old libcs and old statically linked programs? It's exactly the kind of problem why I tell people to never linked statically but some people don't listen.

One easy way is to have a "migrate on exec" option added to the process group code.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 2 lines · Rik van Riel

One easy way is to have a "migrate on exec" option added to the process group code.

That's going to be ugly because the exec functions are signal-safe. I.e., they can happen at any time. This would mean that one always has to set the migration policy before every exec call and that there must be a way to retrieve the currently selected policy so that it can potentially be restored. This policy must be a thread property, not a process property.

Before exec, we should be able to determine the
migration policy related to process/thread (either by reading file or something else etc). Set the policy through cgroup file system. If exec fails for some reason, we just need to go back to cgroup file system to undo the effect of setting migration policy previously set for that thread.

Quoted 12 lines · Rik van Riel

Quoted 1 line · On Thu, 10 Jul 2008 02:23:52 -0700 "Paul Menage" <[email protected]> wrote:

I don't see the rule-based approach being all that useful for our needs.

Agreed, there really is no need for a rule-based approach in kernel space.

There are basically three different cases:

1) daemons get started up in their own process groups, this can be handled by the initscripts

Quoted 6 lines

2) user sessions (ssh, etc) start in their own process groups, this can be handled by PAM

3) users fork processes that should go into special process groups - this could be handled by having a small ruleset in userspace handle things, right before calling exec(),

That means application launcher (ex, shell) is aware of the right cgroup targeted application should go in and then move forked pid to right cgroup and call exec? Or you had something else in mind?

Quoted 3 lines

it can even be hidden from the application by hooking into the exec() call

This means hooking into libc. So libc will parse rules file, determine the right cgroup, place application there and then call exec?

Quoted 24 lines · On Thu, 10 Jul 2008 11:40:35 -0400 Vivek Goyal <[email protected]> wrote:

Quoted 12 lines · Rik van Riel

Quoted 1 line · On Thu, 10 Jul 2008 02:23:52 -0700 "Paul Menage" <[email protected]> wrote:

I don't see the rule-based approach being all that useful for our needs.

Agreed, there really is no need for a rule-based approach in kernel space.

There are basically three different cases:

1) daemons get started up in their own process groups, this can be handled by the initscripts

Quoted 6 lines

2) user sessions (ssh, etc) start in their own process groups, this can be handled by PAM

3) users fork processes that should go into special process groups - this could be handled by having a small ruleset in userspace handle things, right before calling exec(),

That means application launcher (ex, shell) is aware of the right cgroup targeted application should go in and then move forked pid to right cgroup and call exec? Or you had something else in mind?

Quoted 7 lines

Quoted 3 lines

it can even be hidden from the application by hooking into the exec() call

This means hooking into libc. So libc will parse rules file, determine the right cgroup, place application there and then call exec?

Hmm, as I wrote, the rule that the child inherits its own parent't is very strong rule. (Most of case can be handle by this.) So, what I think of is

1. support a new command (in libcg.)
- /bin/change_group_exec ..... read to /etc/cgroup/config and move cgroup and call exec.
2. and libc function
- if necessary.

1. is enough because admin/user can write a wrapper script for their applications if "child inherits parent's" isn't suitable.

no ?

If admin has decided to group applications and has written the rules for it then applications should not know anything about grouping. So I think application writing an script for being placed into the right group should be out of question. Now how does an admin write a wrapper around existing application without breaking anything else.

Quoted 5 lines · Vivek Goyal

If admin has decided to group applications and has written the rules for it then applications should not know anything about grouping. So I think application writing an script for being placed into the right group should be out of question. Now how does an admin write a wrapper around existing application without breaking anything else.

In the Solaris world, processes are placed into cgroups (projects) by one of two mechanisms:

1) inheritance, with everything I create in my existing project. To get this started, there is a mechanism under login/getty/whatever
reading the /etc/projects file and, for example, tossing user davecb into a "user.davecb" project.

2) explicit placement with newtask, which starts a program or moves a process into a project/cgroup

I have a "bg" project which I use for limiting resource consumption of background jobs, and a background command which either starts or moves jobs, thusly:

case "$1" in
[0-9]*) # It's a pid
newtask -p bg -c $1

        ;;
 *) # It's a command-line
        newtask -p bg "$@" &
        ;;

Quoted 7 lines · David Collier-Brown

Quoted 5 lines · Vivek Goyal

If admin has decided to group applications and has written the rules for it then applications should not know anything about grouping. So I think application writing an script for being placed into the right group should be out of question. Now how does an admin write a wrapper around existing application without breaking anything else.

Quoted 8 lines

In the Solaris world, processes are placed into cgroups (projects) by one of two mechanisms:

1) inheritance, with everything I create in my existing project. To get this started, there is a mechanism under login/getty/whatever
reading the /etc/projects file and, for example, tossing user davecb into a "user.davecb" project.

Placing the login sessions in right cgroup based on uid/gid rules is probably easy as check needs to be placed only on system entry upon login (Pam plugin should do). And after that any job started by the user will automatically start in the same cgroup.

Quoted 3 lines

2) explicit placement with newtask, which starts a program or moves a process into a project/cgroup

explicit placement of task based on application type will be tricky.

Quoted 7 lines

I have a "bg" project which I use for limiting resource consumption of background jobs, and a background command which either starts or moves jobs, thusly:

case "$1" in
[0-9]*) # It's a pid
newtask -p bg -c $1

Ok, this is moving of tasks from one cgroup to other based on pid. This is really easy to do through cgroup file system. Just a matter of writing to task file.

Quoted 4 lines

        ;;
 *) # It's a command-line
        newtask -p bg "$@" &
        ;;

So here a user explicitly invokes the wrapper passing it the targeted cgroup and the application to be launched in that cgroup. This should work if there is a facility if user has created its own cgroups (lets say under user controlled cgroup dir in the hierarchy) and user explicitly wants to control the resources of applications under its dir. For example,

/mnt/cgroup

		|	|
		gid1	gid2
		|  |	|  |
	      uid1 uid2	uid3 uid4
	     |  |
	 proj1  proj2

Here probably admin can write the rules for how users are allocated the resources and give ability to users to create subdirs under their cgroups where users can create more cgroups and can do their own resource management based on application tasks and place applications in the right cgroup by writing wrappers as mentioned by you "newtask".

But here there is no discrimination of application type by admin. Admin controls resource divisions only based on uid/gid. And users can manage applications within their user groups. In fact I am having hard time thinking in what kind of scenarios, there is a need for an admin to control resource based on application type? Do we really need setups like, on a system databases should get network bandwidth of 30%. If yes, then it becomes tricky where admin need to write a wrapper to place the task in right cgroup without application/user knowing it.

Quoted 65 lines · Vivek Goyal

Quoted 7 lines · David Collier-Brown

Quoted 5 lines · Vivek Goyal

If admin has decided to group applications and has written the rules for it then applications should not know anything about grouping. So I think application writing an script for being placed into the right group should be out of question. Now how does an admin write a wrapper around existing application without breaking anything else.

Quoted 8 lines

In the Solaris world, processes are placed into cgroups (projects) by one of two mechanisms:

1) inheritance, with everything I create in my existing project. To get this started, there is a mechanism under login/getty/whatever
reading the /etc/projects file and, for example, tossing user davecb into a "user.davecb" project.

Placing the login sessions in right cgroup based on uid/gid rules is probably easy as check needs to be placed only on system entry upon login (Pam plugin should do). And after that any job started by the user will automatically start in the same cgroup.

Quoted 3 lines

2) explicit placement with newtask, which starts a program or moves a process into a project/cgroup

explicit placement of task based on application type will be tricky.

Quoted 7 lines

I have a "bg" project which I use for limiting resource consumption of background jobs, and a background command which either starts or moves jobs, thusly:

case "$1" in
[0-9]*) # It's a pid
newtask -p bg -c $1

Ok, this is moving of tasks from one cgroup to other based on pid. This is really easy to do through cgroup file system. Just a matter of writing to task file.

Quoted 4 lines

        ;;
 *) # It's a command-line
        newtask -p bg "$@" &
        ;;

So here a user explicitly invokes the wrapper passing it the targeted cgroup and the application to be launched in that cgroup. This should work if there is a facility if user has created its own cgroups (lets say under user controlled cgroup dir in the hierarchy) and user explicitly wants to control the resources of applications under its dir. For example,

/mnt/cgroup

		|	|
		gid1	gid2
		|  |	|  |
	      uid1 uid2	uid3 uid4
	     |  |
	 proj1  proj2

Here probably admin can write the rules for how users are allocated the resources and give ability to users to create subdirs under their cgroups where users can create more cgroups and can do their own resource management based on application tasks and place applications in the right cgroup by writing wrappers as mentioned by you "newtask".

Quoted 8 lines

But here there is no discrimination of application type by admin. Admin controls resource divisions only based on uid/gid. And users can manage applications within their user groups. In fact I am having hard time thinking in what kind of scenarios, there is a need for an admin to control resource based on application type? Do we really need setups like, on a system databases should get network bandwidth of 30%. If yes, then it becomes tricky where admin need to write a wrapper to place the task in right cgroup without application/user knowing it.

I think a wrapper (move to right group and calls exec) will run by user, not by admin. In explicit placement, user knows what a type of application he/she launch.

/mnt/cgroup

		|	|
		gid1	gid2
		|  |	|  |
	      uid1 uid2	uid3 uid4
	     |  |
	 proj1  proj2

[uid1/gid1]% newtask.sh proj1app
... proj1app run under /mnt/cgroup/gid1/uid1

[uid1/gid1]% newtask.sh --type proj1type proj1app ... proj1app run under /mnt/cgroup/gid1/uid1/proj1

In this case, admin sets up limitation of proj1type.

Also I guess proj1type has ownership (ex: proj1type allows gid1). Isn't this enough?

Quoted 88 lines · Vivek Goyal

Quoted 78 lines · Kazunaga Ikeno

Quoted 65 lines · Vivek Goyal

Quoted 7 lines · David Collier-Brown

Quoted 5 lines · Vivek Goyal

If admin has decided to group applications and has written the rules for it then applications should not know anything about grouping. So I think application writing an script for being placed into the right group should be out of question. Now how does an admin write a wrapper around existing application without breaking anything else.

Quoted 8 lines

In the Solaris world, processes are placed into cgroups (projects) by one of two mechanisms:

1) inheritance, with everything I create in my existing project. To get this started, there is a mechanism under login/getty/whatever
reading the /etc/projects file and, for example, tossing user davecb into a "user.davecb" project.

Placing the login sessions in right cgroup based on uid/gid rules is probably easy as check needs to be placed only on system entry upon login (Pam plugin should do). And after that any job started by the user will automatically start in the same cgroup.

Quoted 3 lines

2) explicit placement with newtask, which starts a program or moves a process into a project/cgroup

explicit placement of task based on application type will be tricky.

Quoted 7 lines

I have a "bg" project which I use for limiting resource consumption of background jobs, and a background command which either starts or moves jobs, thusly:

case "$1" in
[0-9]*) # It's a pid
newtask -p bg -c $1

Ok, this is moving of tasks from one cgroup to other based on pid. This is really easy to do through cgroup file system. Just a matter of writing to task file.

Quoted 4 lines

        ;;
 *) # It's a command-line
        newtask -p bg "$@" &
        ;;

So here a user explicitly invokes the wrapper passing it the targeted cgroup and the application to be launched in that cgroup. This should work if there is a facility if user has created its own cgroups (lets say under user controlled cgroup dir in the hierarchy) and user explicitly wants to control the resources of applications under its dir. For example,

/mnt/cgroup

		|	|
		gid1	gid2
		|  |	|  |
	      uid1 uid2	uid3 uid4
	     |  |
	 proj1  proj2

Here probably admin can write the rules for how users are allocated the resources and give ability to users to create subdirs under their cgroups where users can create more cgroups and can do their own resource management based on application tasks and place applications in the right cgroup by writing wrappers as mentioned by you "newtask".

Quoted 8 lines

But here there is no discrimination of application type by admin. Admin controls resource divisions only based on uid/gid. And users can manage applications within their user groups. In fact I am having hard time thinking in what kind of scenarios, there is a need for an admin to control resource based on application type? Do we really need setups like, on a system databases should get network bandwidth of 30%. If yes, then it becomes tricky where admin need to write a wrapper to place the task in right cgroup without application/user knowing it.

I think a wrapper (move to right group and calls exec) will run by user, not by admin. In explicit placement, user knows what a type of application he/she launch.

Quoted 8 lines

/mnt/cgroup

		|	|
		gid1	gid2
		|  |	|  |
	      uid1 uid2	uid3 uid4
	     |  |
	 proj1  proj2

Quoted 2 lines

This is the easy to handle situation and I am hoping it will work in many of the cases.

This solution seems ok but this looks only one part of the storey. Here the top level hierarchy is again user based(gid/uid). What if admin wants to manage the system resources per application basis? Say a big server in a university is being shared by everyone in the university for only 3 application
1. http server
2. browsing
3. computing
In case the admin wants the system to be always available for computing, how should he utilize cgroups for managing the server resources among these applications ?
Isn't such scenarios on the priority now?

Quoted 17 lines

Currently I am writting a patch for libcg which allows querying the destination cgroup based on uid/gid and libcg will also migrate the application there. I am also writing a pam plugin which will move all the login sessions to respective cgroup (as mentioned by rule file). Will also modify "init" so that all the system services to into cgroup belonging to root.

Once user is logged in and running into his resource group, he can manage further subgroups at his own based on his application requirements (as you mentioned proj1 and proj2 here).

Quoted 3 lines

[uid1/gid1]% newtask.sh proj1app
... proj1app run under /mnt/cgroup/gid1/uid1

Quoted 3 lines

Yes, so if a user does not specifically launch an application targetted for a particular cgroup, then it will run into default group for that user (as specified by rule file). In this case under /mnt/cgroup/gid1/uid1.

So in this user based approach if admin wants to run 4 major applications each one requiring say 15% cpu he needs to create 4 different gids? Creation of a user account just for running an application does not look very flexible to me.

Quoted 15 lines

Quoted 12 lines

Quoted 11 lines

I think a wrapper (move to right group and calls exec) will run by user, not by admin. In explicit placement, user knows what a type of application he/she launch.

/mnt/cgroup

		|	|
		gid1	gid2
		|  |	|  |
	      uid1 uid2	uid3 uid4
	     |  |
	 proj1  proj2

Quoted 2 lines

This is the easy to handle situation and I am hoping it will work in many of the cases.

Quoted 12 lines

This solution seems ok but this looks only one part of the storey. Here the top level hierarchy is again user based(gid/uid). What if admin wants to manage the system resources per application basis? Say a big server in a university is being shared by everyone in the university for only 3 application
1. http server
2. browsing
3. computing
In case the admin wants the system to be always available for computing, how should he utilize cgroups for managing the server resources among these applications ?
Isn't such scenarios on the priority now?

We only have FS permissions to play around with. Therefore any hierarchy we come up with will be uid/gid based. Such scenarios will be handled by the administrator by ensuring the correct permissions are set for the cgroup.

Quoted 17 lines

Quoted 17 lines

Currently I am writting a patch for libcg which allows querying the destination cgroup based on uid/gid and libcg will also migrate the application there. I am also writing a pam plugin which will move all the login sessions to respective cgroup (as mentioned by rule file). Will also modify "init" so that all the system services to into cgroup belonging to root.

Once user is logged in and running into his resource group, he can manage further subgroups at his own based on his application requirements (as you mentioned proj1 and proj2 here).

Quoted 3 lines

[uid1/gid1]% newtask.sh proj1app
... proj1app run under /mnt/cgroup/gid1/uid1

Quoted 8 lines

Quoted 3 lines

Yes, so if a user does not specifically launch an application targetted for a particular cgroup, then it will run into default group for that user (as specified by rule file). In this case under /mnt/cgroup/gid1/uid1.

So in this user based approach if admin wants to run 4 major applications each one requiring say 15% cpu he needs to create 4 different gids? Creation of a user account just for running an application does not look very flexible to me.

A lot of daemons run as specific users. Also its not a good idea to run daemons/servers as root users. They should run as users who have limited privileges. With such a model in place, Vivek's comments make sense and might be the right way to go ahead.

The problem of placing tasks in respective cgroups seems to be correctly addressed by userspace lib wrappers or classifier daemons [1].

However, this is an attempt to implement an in-kernel classifier.

[ I wrote this patch for a "special purpose" environment, where a lot of short-lived processes belonging to different users are spawned by different daemons, so the main goal here would be to remove the dealy needed by userspace classification and place the tasks in the right cgroup at the time they're created. This is just an ugly hack for now and it works only for uid-based rules, gid-based rules could be implemented in a similar way. ]

Quoted 5 lines · Andrea Righi

The problem of placing tasks in respective cgroups seems to be correctly addressed by userspace lib wrappers or classifier daemons [1].

However, this is an attempt to implement an in-kernel classifier.

Quoted 8 lines

[ I wrote this patch for a "special purpose" environment, where a lot of short-lived processes belonging to different users are spawned by different daemons, so the main goal here would be to remove the dealy needed by userspace classification and place the tasks in the right cgroup at the time they're created. This is just an ugly hack for now and it works only for uid-based rules, gid-based rules could be implemented in a similar way. ]

Hi Andrea,

Recently I introduced the infrastructure in libcgroup to handle the task placement issue based on uid and gid rules. This is what I did.

- Introduced two new APIs in libcgroup to place the task in right cgroup.
- cgroup_change_cgroup_uid_gid
Pleces the task in destination cgroup based on uid/gid rules specified in /etc/cgrules.conf
- cgroup_change_cgroup_path
Puts the task into the cgroup specified by caller

- Provided two command line tools (cgexec and cgclassify) to perform various process placement related tasks.
- cgexec
A tool to launch a task in user specfied cgroup
- cgclassify
A tool to re-classify already running tasks.

- Wrote a pam plugin so that tasks are placed in right user groups upon login or reception of other services which take pam's help.

- Currently work is in progress for a user space daemon which will automatically place the tasks based on notifications.

For your environment, where delay is unbearable, I think you can modify the daemon to use libcgroup to place the forked task in right cgroup before actually executing it. Once the task has been placed in right cgroup, exec() will be called.

We have been doing all the user space development on following mailing list.

https://lists.sourceforge.net/lists/listinfo/libcg-devel

Latest patches which got merged in libcgroup, are here.

http://sourceforge.net/mailarchive/forum.php?thread_name=20080813171720.108005557%40redhat.com&forum_name=libcg-devel

It is accompanied with a decent README file for design details and for how to use it.

I think modifying the daemon to make use of libcgroup is the right way to handle this issue than duplicating the infrastructure in user space as well as kernel space.

[ I wrote this patch for a "special purpose" environment, where a lot of short-lived processes belonging to different users are spawned by different daemons,

Quoted 4 lines · Andrea Righi

[ I wrote this patch for a "special purpose" environment, where a lot of short-lived processes belonging to different users are spawned by different daemons,

What kinds of daemons are these? Is it not possible to add some libcgroup calls to these daemons?

I'm reluctant to add features like this to the kernel side of cgroups due to their "magical" nature - any task that does a setuid() now risks being swept off into a different cgroup.

Having the cgroup attachment done explicitly e.g. by a PAM library at login time is much less likely to cause unexpected behaviour.

Maybe if we had a way to control which tasks the magical setuid switching occurs for, it might be more acceptable. (Perhaps base it on the cgroup of the task that's doing the setuid as well?

Same thing will happen if we implement the daemon in user space. A task who does seteuid(), can be swept away to a different cgroup based on rules specified in /etc/cgrules.conf.

What do you mean by risk? This is the policy set up by system admin and behaviour would seem consistent as per the policy. If an admin decides that tasks of user "apache" should run into /container/cpu/apache cgroup and if a "root" tasks does seteuid(apache), then it manes sense to move task to /container/cpu/apache.

Exactly what kind of scenario do you have in mind when you want the policy to be enforced selectively based on task (tid)?

Quoted 4 lines · Vivek Goyal

Same thing will happen if we implement the daemon in user space. A task who does seteuid(), can be swept away to a different cgroup based on rules specified in /etc/cgrules.conf.

Yes, I'm not so keen on a daemon magically pulling things into a cgroup based on uid either, for the same reasons.

But a user-space based solution can be much more flexible (e.g. easier to configure it to only move tasks from certain source cgroups).

Quoted 6 lines

What do you mean by risk? This is the policy set up by system admin and behaviour would seem consistent as per the policy. If an admin decides that tasks of user "apache" should run into /container/cpu/apache cgroup and if a "root" tasks does seteuid(apache), then it manes sense to move task to /container/cpu/apache.

The kind of unexpected behaviour I was imagining was when some other daemon (e.g. ftpd?) unexpectedly does a setuid to one of the magically-controlled users, and results in that daemon being pulled into the specified cgroup. For something like cpu maybe that's mostly benign (but what moves it back into its original group after it switches back to root?)

but for other subsystems it could be more
painful (memory, device access, etc).

Quoted 1 line

Quoted 2 lines

Exactly what kind of scenario do you have in mind when you want the policy to be enforced selectively based on task (tid)?

I was thinking of something like possibly a per-cgroup file (that also affected child cgroups) rather than a global file. That would also automatically handle multiple hierarchies.

Quoted 15 lines · Paul Menage

Quoted 3 lines · Vivek Goyal

Same thing will happen if we implement the daemon in user space. A task who does seteuid(), can be swept away to a different cgroup based on rules specified in /etc/cgrules.conf.

Yes, I'm not so keen on a daemon magically pulling things into a cgroup based on uid either, for the same reasons.

But a user-space based solution can be much more flexible (e.g. easier to configure it to only move tasks from certain source cgroups).

Quoted 5 lines

What do you mean by risk? This is the policy set up by system admin and behaviour would seem consistent as per the policy. If an admin decides that tasks of user "apache" should run into /container/cpu/apache cgroup and if a "root" tasks does seteuid(apache), then it manes sense to move task to /container/cpu/apache.

Quoted 6 lines

The kind of unexpected behaviour I was imagining was when some other daemon (e.g. ftpd?) unexpectedly does a setuid to one of the magically-controlled users, and results in that daemon being pulled into the specified cgroup. For something like cpu maybe that's mostly benign (but what moves it back into its original group after it switches back to root?)

Once ftpd does seteuid() or setreuid() again to switch effective user to "root", it will be moved back to original group (root's group).

So basic question is if a program changes its effective user id temporarily to user B than all the resource consumption should take place from the resources of user B or should continue to take place from original cgroup.

I would think that we should move the task temporarily to B's cgroup and bring back again upon identity change.

At the same time I can also understand that this behavior can probably be considered over-intrusive and some people might want to avoid that.

Two things come to my mind.

- Users who find it too intrusive, can just shut down the rules based daemon.

- Or, we can implement selective movement of tasks by daemon as suggested by you. This will make system more complex but provides more flexibility in the sense users can keep daemon running at the same time control movement of certain tasks.

Quoted 11 lines

but for other subsystems it could be more
painful (memory, device access, etc).

Quoted 2 lines

Exactly what kind of scenario do you have in mind when you want the policy to be enforced selectively based on task (tid)?

I was thinking of something like possibly a per-cgroup file (that also affected child cgroups) rather than a global file. That would also automatically handle multiple hierarchies.

So there can be two kind of controls.

- Create a per cgroup file say "group_pinned", where if 1 is written to "group_pinned" that means daemon will not move tasks from this cgroup upon effective uid/gid changes.

- Provide more fine grained control where task movement is not controlled per cgroup, rather per thread id. In that case every cgroup will contain another file "tasks_pinned" which will contain all the tids which cannot be moved from this cgroup by daemon. By default this file will be empty and all the tids are movable.

I think initially we can keep things simple and implement "group_pinned" which provides coarse control on the whole group and pins all the tasks in that cgroup.

Applications that really care about moving should use cgroup_attach_task* and move back otherwise with cgrules parsing turned off.

I see control as a two level hierarchy, automatic and controlled, how do we make sure that they don't conflict is something I have not thought about yet.

Hmm... I wonder if we are providing too many knobs. Can't we do something simpler?

Quoted 5 lines · Balbir Singh

Applications that really care about moving should use cgroup_attach_task* and move back otherwise with cgrules parsing turned off.

I see control as a two level hierarchy, automatic and controlled, how do we make sure that they don't conflict is something I have not thought about yet.

[...]

Quoted 1 line

Hmm... I wonder if we are providing too many knobs. Can't we do something simpler?

Solaris doesn't try to change cgroup ("project") on a setuid call, assuming the program is in the proper cgroup initially. For most cases this is trivially true under the very simple default rules, and for the rest one can create a rule or a startup script that sets it with newtask".

The Sun default is

	$ cat /etc/project
	system:0::::
	user.root:1::::
	noproject:2::::
	default:3::::
	group.staff:10::::

Which means that root users are distinguished from users in the staff group, and they are distinguished from daemons and everyone else.

Personally, I add

	user.davecb:101::davecb::
	bg:100:Background jobs:davecb::

which puts me in a separate cgroup, and provides another one for me to put background tasks into. The latter allows me to keep them from reducing the interactive performance of my laptop.

Who executes default rules? IOW, how do you make sure tasks of user.davecb end up in project 101 only and not outside?

So by default all the tasks of user.davecb will run into project 101 until user davecb decides to launch some background jobs in project 100 using newtask?

Quoted 38 lines · Vivek Goyal

Quoted 15 lines · Paul Menage

Quoted 3 lines · Vivek Goyal

Same thing will happen if we implement the daemon in user space. A task who does seteuid(), can be swept away to a different cgroup based on rules specified in /etc/cgrules.conf.

Yes, I'm not so keen on a daemon magically pulling things into a cgroup based on uid either, for the same reasons.

But a user-space based solution can be much more flexible (e.g. easier to configure it to only move tasks from certain source cgroups).

Quoted 5 lines

What do you mean by risk? This is the policy set up by system admin and behaviour would seem consistent as per the policy. If an admin decides that tasks of user "apache" should run into /container/cpu/apache cgroup and if a "root" tasks does seteuid(apache), then it manes sense to move task to /container/cpu/apache.

Quoted 6 lines

The kind of unexpected behaviour I was imagining was when some other daemon (e.g. ftpd?) unexpectedly does a setuid to one of the magically-controlled users, and results in that daemon being pulled into the specified cgroup. For something like cpu maybe that's mostly benign (but what moves it back into its original group after it switches back to root?)

Once ftpd does seteuid() or setreuid() again to switch effective user to "root", it will be moved back to original group (root's group).

So basic question is if a program changes its effective user id temporarily to user B than all the resource consumption should take place from the resources of user B or should continue to take place from original cgroup.

I would think that we should move the task temporarily to B's cgroup and bring back again upon identity change.

At the same time I can also understand that this behavior can probably be considered over-intrusive and some people might want to avoid that.

Two things come to my mind.

Quoted 3 lines

- Users who find it too intrusive, can just shut down the rules based daemon.

Yes, I would say administrators should do that. Classification via setuid(), does make a lot of sense, but at the same time it might be too aggressive if an application frequently uses setuid()

Quoted 19 lines

Quoted 6 lines

Quoted 2 lines

Exactly what kind of scenario do you have in mind when you want the policy to be enforced selectively based on task (tid)?

I was thinking of something like possibly a per-cgroup file (that also affected child cgroups) rather than a global file. That would also automatically handle multiple hierarchies.

So there can be two kind of controls.

- Create a per cgroup file say "group_pinned", where if 1 is written to "group_pinned" that means daemon will not move tasks from this cgroup upon effective uid/gid changes.

- Provide more fine grained control where task movement is not controlled per cgroup, rather per thread id. In that case every cgroup will contain another file "tasks_pinned" which will contain all the tids which cannot be moved from this cgroup by daemon. By default this file will be empty and all the tids are movable.

Quoted 4 lines

I think initially we can keep things simple and implement "group_pinned" which provides coarse control on the whole group and pins all the tasks in that cgroup.

Hmm... I wonder if we are providing too many knobs. Can't we do something simpler?

So there can be two kind of controls.

- Create a per cgroup file say "group_pinned", where if 1 is written to "group_pinned" that means daemon will not move tasks from this cgroup upon effective uid/gid changes.

Quoted 4 lines · Andrea Righi

[ I wrote this patch for a "special purpose" environment, where a lot of short-lived processes belonging to different users are spawned by different daemons,

What kinds of daemons are these? Is it not possible to add some libcgroup calls to these daemons?

I'm reluctant to add features like this to the kernel side of cgroups

due to their "magical" nature - any task that does a setuid() now risks being swept off into a different cgroup.

Having the cgroup attachment done explicitly e.g. by a PAM library at login time is much less likely to cause unexpected behaviour.

Maybe if we had a way to control which tasks the magical setuid switching occurs for, it might be more acceptable. (Perhaps base it on the cgroup of the task that's doing the setuid as well?

Other thoughts:

- what about other uids (euid, fsuid)?

- what about multiple hierarchies?

- if the attach fails, userspace gets no notification.

Paul

unfortunately I don't have too much details for now, so I was just looking for the most generic solution. The PAM lib approach seems reasonable for each daemon that represents an entry point to the system,

Quoted 3 lines

due to their "magical" nature - any task that does a setuid() now risks being swept off into a different cgroup.

If the admin configures so, moving tasks that do setuid() in different cgroups should be an expected behaviour, isn't it?

Quoted 8 lines

Quoted 5 lines · On Thu, 10 Jul 2008 11:40:35 -0400 Vivek Goyal <[email protected]> wrote:

Quoted 4 lines · Rik van Riel

Quoted 1 line · On Thu, 10 Jul 2008 02:23:52 -0700 "Paul Menage" <[email protected]> wrote:

I don't see the rule-based approach being all that useful for our nee

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Agreed, there really is no need for a rule-based approach in kernel spa

Quoted 37 lines

Quoted 17 lines

Quoted 6 lines

There are basically three different cases:

1) daemons get started up in their own process groups, this can be handled by the initscripts

Quoted 6 lines

2) user sessions (ssh, etc) start in their own process groups, this can be handled by PAM

3) users fork processes that should go into special process groups - this could be handled by having a small ruleset in userspace handle things, right before calling exec(),

That means application launcher (ex, shell) is aware of the right cgroup targeted application should go in and then move forked pid to right cgroup and call exec? Or you had something else in mind?

Quoted 7 lines

Quoted 3 lines

it can even be hidden from the application by hooking into the exec() call

This means hooking into libc. So libc will parse rules file, determine the right cgroup, place application there and then call exec?

Hmm, as I wrote, the rule that the child inherits its own parent't is very strong rule. (Most of case can be handle by this.) So, what I think of is

1. support a new command (in libcg.)
- /bin/change_group_exec ..... read to /etc/cgroup/config and move cgroup and call exec.
2. and libc function
- if necessary.

1. is enough because admin/user can write a wrapper script for their applications if "child inherits parent's" isn't suitable.

Quoted 2 lines

no ?

If admin has decided to group applications and has written the rules for it then applications should not know anything about grouping. So I think application writing an script for being placed into the right group should be out of question. Now how does an admin write a wrapper around existing application without breaking anything else.

One thing could be creating soft links where admin created alias points to wrapper and wrapper inturn invokes the executable. But this will not solve the problem if some user decides to invoke the application executable directly and not use admin created alias.

Did you have something else in mind when it came to creating wrappers around applications?

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

Hi Vivek,

Quoted 4 lines · Vivek Goyal

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

One way that you could avoid the unreliability would be to not use netlink, but instead use cgroups itself.

What we're looking for is a way to easily distinguish between processes that are in the right cgroups, and processes that might be in the wrong cgroups. Additionally, we want the children of such processes to inherit the same status until we've dealt with them, and not be able to change their status themselves.

That sounds a bit like a cgroup. How about the following?

- create a cgroup subsystem called "setuid".

- have a uid_changed() hook called by sys_setuid() and friends; this hook would simply attach current to the root cgroup in the "setuid" hierarchy if it wasn't already in that cgroup (which can be determined with a couple of dereferences from current and no locking, so not slowing down the normal case).

- userspace uses this by:

mount the setuid hierarchy, e.g. at /mnt/setuid create a child cgroup /mnt/setuid/processed
while true:
wait for /mnt/setuid/tasks to be non-empty
read a pid from /mnt/setuid/tasks
move that pid to the appropriate cgroups in memory/cpu/etc
hierarchies if necessary
move that pid to /mnt/setuid/processed/tasks

i.e. any pid in the root cgroup of the setuid hierarchy is one that needs attention and may need to be moved to different cgroups

A couple of enhancements to make this more usable might include:

- adding an API (via a new syscall or an eventfd?) to wait for a cgroup to be non-empty, to avoid having to poll /mnt/setuid/tasks more than necessary

- allow the user to designate certain processes and their children as uninteresting so that their setuid calls don't trigger them being moved back to the root (perhaps indicated via membership of an "ignored" cgroup in the setuid hierarchy?)

This should be more reliable than netlink since it doesn't involve userspace having to keep up with a stream of events - we're not queuing up events, we're just shifting process group memberships.

Similar approaches could be used for a "setgid" hierarchy and an "execve" hierarchy.

This looks interesting. So above method should solve atleast the reliability issue of event transport to user space. Got few thougts.

- Hopefully number of hiearchies will not explode as we will be mounting one hierarchies per event type (uid change, gid change, exec, maybe fork etc.).

Quoted 7 lines · Vivek Goyal

This looks interesting. So above method should solve atleast the reliability issue of event transport to user space. Got few thougts.

- Hopefully number of hiearchies will not explode as we will be mounting one hierarchies per event type (uid change, gid change, exec, maybe fork etc.).

In what circumstances would you want to reclassify processes to a different cgroup on a fork?

Quoted 9 lines · Paul Menage

Quoted 7 lines · Vivek Goyal

This looks interesting. So above method should solve atleast the reliability issue of event transport to user space. Got few thougts.

- Hopefully number of hiearchies will not explode as we will be mounting one hierarchies per event type (uid change, gid change, exec, maybe fork etc.).

Quoted 2 lines

In what circumstances would you want to reclassify processes to a different cgroup on a fork?

I don't know. Balbir had mentioned in one of the mails in this thread regarding getting notification on fork.

Hi Vivek,

Quoted 4 lines · Vivek Goyal

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

One way that you could avoid the unreliability would be to not use netlink, but instead use cgroups itself.

What we're looking for is a way to easily distinguish between processes that are in the right cgroups, and processes that might be in the wrong cgroups. Additionally, we want the children of such processes to inherit the same status until we've dealt with them, and not be able to change their status themselves.

That sounds a bit like a cgroup. How about the following?

- create a cgroup subsystem called "setuid".

- have a uid_changed() hook called by sys_setuid() and friends; this hook would simply attach current to the root cgroup in the "setuid" hierarchy if it wasn't already in that cgroup (which can be determined with a couple of dereferences from current and no locking, so not slowing down the normal case).

- userspace uses this by:

mount the setuid hierarchy, e.g. at /mnt/setuid create a child cgroup /mnt/setuid/processed
while true:
wait for /mnt/setuid/tasks to be non-empty
read a pid from /mnt/setuid/tasks
move that pid to the appropriate cgroups in memory/cpu/etc
hierarchies if necessary
move that pid to /mnt/setuid/processed/tasks

i.e. any pid in the root cgroup of the setuid hierarchy is one that needs attention and may need to be moved to different cgroups

A couple of enhancements to make this more usable might include:

- adding an API (via a new syscall or an eventfd?) to wait for a cgroup to be non-empty, to avoid having to poll /mnt/setuid/tasks more than necessary

- allow the user to designate certain processes and their children as uninteresting so that their setuid calls don't trigger them being moved back to the root (perhaps indicated via membership of an "ignored" cgroup in the setuid hierarchy?)

This should be more reliable than netlink since it doesn't involve userspace having to keep up with a stream of events - we're not queuing up events, we're just shifting process group memberships.

Similar approaches could be used for a "setgid" hierarchy and an "execve" hierarchy.

We also need to do something to track all the forked childs after the setuid, setgid or exec till original parent event got classified and children need to meet the same treatment.

Quoted 4 lines · Vivek Goyal

We also need to do something to track all the forked childs after the setuid, setgid or exec till original parent event got classified and children need to meet the same treatment.

You'd get that automatically, since children of the task moved to the root cgroup (indicating "needs attention") would also end up in that cgroup since cgroup are inherited across fork.

I am sorry, I seem to missing something, but who moves the forked children (which got forked during the time between the parent getting classified into the right group and the fork itself) into the correct group?

Quoted 5 lines · Dhaval Giani

I am sorry, I seem to missing something, but who moves the forked children (which got forked during the time between the parent getting classified into the right group and the fork itself) into the correct group?

The classifier daemon would have to do that - my point was that it would be very clear exactly which processes needed this attention, since they'd end up in the root cgroup too.

Hi Vivek,

Quoted 4 lines · Vivek Goyal

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

One way that you could avoid the unreliability would be to not use netlink, but instead use cgroups itself.

What we're looking for is a way to easily distinguish between processes that are in the right cgroups, and processes that might be in the wrong cgroups. Additionally, we want the children of such processes to inherit the same status until we've dealt with them, and not be able to change their status themselves.

That sounds a bit like a cgroup. How about the following?

- create a cgroup subsystem called "setuid".

- have a uid_changed() hook called by sys_setuid() and friends; this hook would simply attach current to the root cgroup in the "setuid" hierarchy if it wasn't already in that cgroup (which can be determined with a couple of dereferences from current and no locking, so not slowing down the normal case).

- userspace uses this by:

mount the setuid hierarchy, e.g. at /mnt/setuid create a child cgroup /mnt/setuid/processed
while true:
wait for /mnt/setuid/tasks to be non-empty
read a pid from /mnt/setuid/tasks
move that pid to the appropriate cgroups in memory/cpu/etc
hierarchies if necessary
move that pid to /mnt/setuid/processed/tasks

i.e. any pid in the root cgroup of the setuid hierarchy is one that needs attention and may need to be moved to different cgroups

Hi,

While development is going on for cgroup and various controllers, we also need a facility so that an admin/user can specify the group creation and also specify the rules based on which tasks should be placed in respective groups. Group creation part will be handled by libcg which is already under development. We still need to tackle the issue of how to specify the rules and how these rules are enforced (rules engine).

I have gathered few views, with regards to how rule engine can possibly be implemented, I am listing these down.

Proposal 1
==========
Let user space daemon hanle all that. Daemon will open a netlink socket and receive the notifications for various kernel events. Daemon will also parse appropriate admin specified rules config file and place the processes in right cgroup based on rules as and when events happen.

I have written a prototype user space program which does that. Program can be found here. Currently it is in very crude shape.

http://people.redhat.com/vgoyal/misc/rules-engine-daemon/user-id-based-namespaces.patch

Various people have raised two main issues with this approach.

- netlink is not a reliable protocol.
- Messages can be dropped and one can loose message. That means a newly forked process might never go into right group as meant.

- How to handle delays in rule exectuion?
- For example, if an "exec" happens and by the time process is moved to right group, it might have forked off few more processes or might have done quite some amount of memory allocation which will be charged to the wring group. Or, newly exec process might get killed in existing cgroup because of lack of memory (despite the fact that destination cgroup has sufficient memory).

Proposal 2
==========
Implement one or more kernel modules which will implement the rule engine. User space program can parse the config files and pass it to module. Kernel will be patched only on select points to look for the rules (as provided by modules). Very minimal code running inside the kernel if there are no rules loaded.

Concerns:
- Rules can become complex and we don't want to handle that complexity in kernel.

Pros:
- Reliable and precise movement of tasks in right cgroup based on rules.

Proposal 3
==========
How about if additional parameters can be passed to system calls and one can pass destination cgroup as additional parameter. Probably something like sys_indirect proposal. Maybe glibc can act as a wrapper to pass additional parameter so that applications don't need any modifications.

Concerns:
========
- Looks like sys_indirect interface for passing extra flags was rejected.
- Requires extra work in glibc which can also involve parsing of rule files. :-(

Proposal 4
==========
Some vauge thoughts are there regarding how about kind of freezing the process or thread upon fork, exec and unfreeze it once the thread has been placed in right cgroup.

Concerns:
========
- Requires reliable netlink protocol otherwise there is a possibility that a task never gets unfrozen.
- On what basis does one freeze a thread. There might not be any rules to process for that thread we will unnecessarily delay it.

Please provide your inputs regarding what's the best way to handle the rules engine.

To me, letting the rules live in separate module/modules seems to be a reasonable way to move forward which will provide reliable and timely execution of rules and by making it modular, we can remove most of the complexity from core kernel code.

Thanks
Vivek
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

For example, create a new file under memory cgroup
==
/opt/memory_cgroup/group_A/notify_at_memory_reach_limit
==
And a user watches the file by inotify.
The kernel modify modified-time of notify_at_memory_reach_limit file and call fs/notify_user.c::notify_change() against this inode. He can catchthe event by inotify.
(I think he can also catch removal of this file, etc...)

Quoted 7 lines · KAMEZAWA Hiroyuki

For example, create a new file under memory cgroup
==
/opt/memory_cgroup/group_A/notify_at_memory_reach_limit
==
And a user watches the file by inotify.
The kernel modify modified-time of notify_at_memory_reach_limit file and ca

Quoted 4 lines

fs/notify_user.c::notify_change() against this inode. He can catchthe event by inotify.
(I think he can also catch removal of this file, etc...)

We've been doing something like this to handle OOMs in userspace, with pretty good success. The approach that we used so far was a custom control file tied to a wait queue, that gets woken when a cgroup triggers OOM, but it's a bit hacky. I've been considering some kind of more generic approach that could be reused by different subsystems for other notifications, maybe using eventfd or maybe netlink.

inotify would be an option too, but that seems like it might be forcing ourselves into filesystem semantics too much.

Paul

Quoted 9 lines · On Tue, 1 Jul 2008 15:11:26 -0400 Vivek Goyal <[email protected]> wrote:

Hi,

While development is going on for cgroup and various controllers, we also need a facility so that an admin/user can specify the group creation and also specify the rules based on which tasks should be placed in respective groups. Group creation part will be handled by libcg which is already under development. We still need to tackle the issue of how to specify the rules and how these rules are enforced (rules engine).

A different topic.

Recently I'm interested in "How to write userland daemon program to control group subsystem." To implement that effectively, we need some notifier between user <-> kernel.

Can we use "inotify" to catch changes in cgroup (by daemon program) ?

For example, create a new file under memory cgroup
==
/opt/memory_cgroup/group_A/notify_at_memory_reach_limit
==
And a user watches the file by inotify.
The kernel modify modified-time of notify_at_memory_reach_limit file and call fs/notify_user.c::notify_change() against this inode. He can catchthe event by inotify.

(I think he can also catch removal of this file, etc...)

Is there some difficulty or problem ? (I'm sorry if we can do this now.)

Quoted 21 lines · KAMEZAWA Hiroyuki

Quoted 9 lines · On Tue, 1 Jul 2008 15:11:26 -0400 Vivek Goyal <[email protected]> wrote:

Hi,

While development is going on for cgroup and various controllers, we also need a facility so that an admin/user can specify the group creation and also specify the rules based on which tasks should be placed in respective groups. Group creation part will be handled by libcg which is already under development. We still need to tackle the issue of how to specify the rules and how these rules are enforced (rules engine).

A different topic.

Recently I'm interested in "How to write userland daemon program to control group subsystem." To implement that effectively, we need some notifier between user <-> kernel.

Can we use "inotify" to catch changes in cgroup (by daemon program) ?

Quoted 8 lines

For example, create a new file under memory cgroup
==
/opt/memory_cgroup/group_A/notify_at_memory_reach_limit
==
And a user watches the file by inotify.
The kernel modify modified-time of notify_at_memory_reach_limit file and call fs/notify_user.c::notify_change() against this inode. He can catchthe event by inotify.

Won't the time latency be an issue (time between exceeding the limit and the user space being notified?).

Since the notification does not use user memory at
the moment (it will not stress the limits futher :)), provided the notification handler is not running under the group that has exceeded its limit. Do we expect the user space application to ACK that it's seen the notification? We could use a netlink channel as well (in the case that we need two way communication).

Quoted 19 lines · KAMEZAWA Hiroyuki

Quoted 9 lines · On Tue, 1 Jul 2008 15:11:26 -0400 Vivek Goyal <[email protected]> wrote:

Hi,

While development is going on for cgroup and various controllers, we also need a facility so that an admin/user can specify the group creation and also specify the rules based on which tasks should be placed in respective groups. Group creation part will be handled by libcg which is already under development. We still need to tackle the issue of how to specify the rules and how these rules are enforced (rules engine).

A different topic.

Recently I'm interested in "How to write userland daemon program to control group subsystem." To implement that effectively, we need some notifier between user <-> kernel.

Quoted 8 lines

Can we use "inotify" to catch changes in cgroup (by daemon program) ?

For example, create a new file under memory cgroup
==
/opt/memory_cgroup/group_A/notify_at_memory_reach_limit
==
And a user watches the file by inotify.
The kernel modify modified-time of notify_at_memory_reach_limit file and ca

Quoted 2 lines

fs/notify_user.c::notify_change() against this inode. He can catchthe event by inotify.

Won't the time latency be an issue (time between exceeding the limit and the user space being notified?). Since the notification does not use user memory

the moment (it will not stress the limits futher :)), provided the notificati

handler is not running under the group that has exceeded its limit. Do we exp

the user space application to ACK that it's seen the notification? We could u

a netlink channel as well (in the case that we need two way communication).

I would prefer to notify on memory.failcnt, if we do use this interface.