cgroup TODOs

1. cpu and cpuacct

They cover the same resources and the scheduler cgroup code ends up having to traverse two separate cgroup trees to update the stats. With nested cgroups, the overhead isn't insignificant and it generally is silly.

While the use cases for having cpuacct on a separate and likely more granular hierarchy, are somewhat valid, the consensus seems that it's just not worth the trouble and cpuacct should be removed in the long term and we shouldn't allow overlapping controllers for the same resource, especially accounting ones.

Solution:

* Whine if cpuacct is not co-mounted with cpu.

* Make sure cpu has all the stats of cpuacct. If cpu and cpuacct are comounted, don't really mount cpuacct but tell cpu that the user requested it. cpu is updated to create aliases for cpuacct.* files in such cases. This involves special casing cpuacct in cgroup core but I much prefer one-off exception case to adding a generic mechanism for this.

* After a while, we can just remove cpuacct completely.

* Later on, phase out the aliases too.

Who:

Me, working on it.

2. memcg's __DEPRECATED_clear_css_refs

This is a remnant of another weird design decision of requiring synchronous draining of refcnts on cgroup removal and allowing subsystems to veto cgroup removal - what's the userspace supposed to do afterwards? Note that this also hinders co-mounting different controllers.

The behavior could be useful for development and debugging but it unnecessarily interlocks userland visible behavior with in-kernel implementation details. To me, it seems outright wrong (either implement proper severing semantics in the controller or do full refcnting) and disallows, for example, lazy drain of caching refs. Also, it complicates the removal path with try / commit / revert logic which has never been fully correct since the beginning.

Currently, the only left user is memcg.

Solution:

* Update memcg->pre_destroy() such that it never fails.

* Drop __DEPRECATED_clear_css_refs and all related logic. Convert pre_destroy() to return void.

Who:

KAMEZAWA, Michal, PLEASE. I will make __DEPRECATED_clear_css_refs trigger WARN sooner or later. Let's please get this settled.

3. cgroup_mutex usage outside cgroup core

This is another thing which is simply broken. Given the way cgroup is structured and used, nesting cgroup_mutex inside any other commonly used lock simply doesn't work - it's held while invoking controller callbacks which then interact and synchronize with various core subsystems.

There are currently three external cgroup_mutex users - cpuset, memcontrol and cgroup_freezer.

Solution:

Well, we should just stop doing it - use a separate nested lock (which seems possible for cgroup_freezer) or track and mange task in/egress some other way.

Who:

I'll do the cgroup_freezer. I'm hoping PeterZ or someone who's familiar with the code base takes care of cpuset. Michal, can you please take care of memcg?

4. Make disabled controllers cheaper

Mostly through the use of static_keys, I suppose. Making this easier AFAICS depends on resolving #2. The lock dependency loop from #2 makes using static_keys from cgroup callbacks extremely nasty.

Solution:

Fix #2 and support common pattern from cgroup core.

Who:

Dunno. Let's see.

5. I CAN HAZ HIERARCHIES?

The cpu ones handle nesting correctly - parent's accounting includes children's, parent's configuration affects children's unless explicitly overridden, and children's limits nest inside parent's.

memcg asked itself the existential question of to be hierarchical or not and then got confused and decided to become both.

When faced with the same question, blkio and cgroup_freezer just gave up and decided to allow nesting and then ignore it - brilliant.

And there are others which kinda sorta try to handle hierarchy but only goes way-half.

This one is screwed up embarrassingly badly. We failed to establish one of the most basic semantics and can't even define what a cgroup hierarchy is - it depends on each controller and they're mostly wacky!

Fortunately, I don't think it will be prohibitively difficult to dig ourselves out of this hole.

Solution:

* cpu ones seem fine.

* For broken controllers, cgroup core will be generating warning messages if the user tries to nest cgroups so that the user at least can know that the behavior may change underneath them later on. For more details,

http://thread.gmane.org/gmane.linux.kernel/1356264/focus=3902

* memcg can be fully hierarchical but we need to phase out the flat hierarchy support. Unfortunately, this involves flipping the behavior for the existing users. Upstream will try to nudge users with warning messages. Most burden would be on the distros and at least SUSE seems to be on board with it. Needs coordination with other distros.

* blkio is the most problematic. It has two sub-controllers - cfq and blk-throttle. Both are utterly broken in terms of hierarchy support and the former is known to have pretty hairy code base. I don't see any other way than just biting the bullet and fixing it.

* cgroup_freezer and others shouldn't be too difficult to fix.

Who:

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

6. Multiple hierarchies

Apart from the apparent wheeeeeeeeness of it (I think I talked about that enough the last time[1]), there's a basic problem when more than one controllers interact - it's impossible to define a resource group when more than two controllers are involved because the intersection of different controllers is only defined in terms of tasks.

IOW, if an entity X is of interest to two controllers, there's no way to map X to the cgroups of the two controllers. X may belong to A and B when viewed by one task but A' and B when viewed by another. This already is a head scratcher in writeback where blkcg and memcg have to interact.

While I am pushing for unified hierarchy, I think it's necessary to have different levels of granularities depending on controllers given that nesting involves significant overhead and noticeable controller-dependent behavior changes.

Solution:

I think a unified hierarchy with the ability to ignore subtrees depending on controllers should work. For example, let's assume the following hierarchy.

          R
	/   \
       A     B
      / \
     AA AB

All controllers are co-mounted. There is per-cgroup knob which controls which controllers nest beyond it. If blkio doesn't want to distinguish AA and AB, the user can specify that blkio doesn't nest beyond A and blkio would see the tree as,

          R
	/   \
       A     B

While other controllers keep seeing the original tree. The exact form of interface, I don't know yet. It could be a single file which the user echoes [-]controller name into it or per-controller boolean file.

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

I *think* this can be achieved by changing where css_set is bound. Currently, a css_set is (conceptually) owned by a task. After the change, a cgroup in the unified hierarchy has its own css_set which tasks point to and can also be used to tag resources as necessary. This way, it should be achieveable without introducing a lot of new code or affecting individual controllers too much.

The headache will be the transition period where we'll probably have to support both modes of operation. Oh well....

Who:

Li, Glauber and me, I guess?

7. Misc issues

* Sort & unique when listing tasks. Even the documentation says it doesn't happen but we have a good hunk of code doing it in cgroup.c. I'm gonna rip it out at some point. Again, if you don't like it, scream.

Quoted 50 lines

2. memcg's __DEPRECATED_clear_css_refs

This is a remnant of another weird design decision of requiring synchronous draining of refcnts on cgroup removal and allowing subsystems to veto cgroup removal - what's the userspace supposed to do afterwards? Note that this also hinders co-mounting different controllers.

The behavior could be useful for development and debugging but it unnecessarily interlocks userland visible behavior with in-kernel implementation details. To me, it seems outright wrong (either implement proper severing semantics in the controller or do full refcnting) and disallows, for example, lazy drain of caching refs. Also, it complicates the removal path with try / commit / revert logic which has never been fully correct since the beginning.

Currently, the only left user is memcg.

Solution:

* Update memcg->pre_destroy() such that it never fails.

* Drop __DEPRECATED_clear_css_refs and all related logic. Convert pre_destroy() to return void.

Who:

KAMEZAWA, Michal, PLEASE. I will make __DEPRECATED_clear_css_refs trigger WARN sooner or later. Let's please get this settled.

3. cgroup_mutex usage outside cgroup core

This is another thing which is simply broken. Given the way cgroup is structured and used, nesting cgroup_mutex inside any other commonly used lock simply doesn't work - it's held while invoking controller callbacks which then interact and synchronize with various core subsystems.

There are currently three external cgroup_mutex users - cpuset, memcontrol and cgroup_freezer.

Solution:

Well, we should just stop doing it - use a separate nested lock (which seems possible for cgroup_freezer) or track and mange task in/egress some other way.

Who:

Quoted 4 lines

I'll do the cgroup_freezer. I'm hoping PeterZ or someone who's familiar with the code base takes care of cpuset. Michal, can you please take care of memcg?

I think this is a pressing problem, yes, but not the only problem with cgroup lock. Even if we restrict its usage to cgroup core, we still can call cgroup functions, which will lock. And then we gain nothing.

And the problem is that people need to lock. cgroup_lock is needed because the data you are accessing is protected by it. The way I see it, it is incredible how we were able to revive the BKL in the form of cgroup_lock after we finally manage to successfully get rid of it!

We should just start to do a more fine grained locking of data, instead of "stop the world, cgroup just started!". If we do that, the problem you are trying to address here will even cease to exist.

Quoted 4 lines

I think this is a pressing problem, yes, but not the only problem with cgroup lock. Even if we restrict its usage to cgroup core, we still can call cgroup functions, which will lock. And then we gain nothing.

Agreed. The biggest issue in cpuset is if hotplug makes a cpuset's cpulist empty the tasks in it will be moved to an ancestor cgroup, which requires holding cgroup lock. We have to either change cpuset's behavior or eliminate the global lock.

Agreed. The biggest issue in cpuset is if hotplug makes a cpuset's cpulist empty the tasks in it will be moved to an ancestor cgroup, which requires holding cgroup lock. We have to either change cpuset's behavior or eliminate the global lock.

Hello,

Quoted 4 lines · Li Zefan

Agreed. The biggest issue in cpuset is if hotplug makes a cpuset's cpulist empty the tasks in it will be moved to an ancestor cgroup, which requires holding cgroup lock. We have to either change cpuset's behavior or eliminate the global lock.

Does that have to happen synchronously? Can't we have a cgroup operation which asynchronously pushes all tasks in a cgroup to its parent from a work item?

Its hotplug, all hotplug stuff is synchronous, the last thing hotplug needs is the added complexity of async callbacks. Also pushing stuff out into worklets just to work around locking issues is vile.

<handwave as I never can remember all the cgroup stuff/>

Can't we play games by pinning both cgroups with a reference and playing games with threadgroup_change / task_lock for the individual tasks being moved about?

Can we please keep some key userspace guys CCd?

Quoted 1 line

1. cpu and cpuacct

Quoted 1 line

Me, working on it.

I can work on it as well if you want. I dealt with it many times in the past, and tried some different approaches, so I am familiar. But if you're already doing it, be my guest...

Quoted 3 lines

I'll do the cgroup_freezer. I'm hoping PeterZ or someone who's familiar with the code base takes care of cpuset. Michal, can you please take care of memcg?

I think this is a pressing problem, yes, but not the only problem with cgroup lock. Even if we restrict its usage to cgroup core, we still can call cgroup functions, which will lock. And then we gain nothing.

And the problem is that people need to lock. cgroup_lock is needed because the data you are accessing is protected by it. The way I see it, it is incredible how we were able to revive the BKL in the form of cgroup_lock after we finally manage to successfully get rid of it!

We should just start to do a more fine grained locking of data, instead of "stop the world, cgroup just started!". If we do that, the problem you are trying to address here will even cease to exist.

Quoted 4 lines

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

I am happy to help where manpower is needed, but I must node I am a bit ignorant of block in general.

Quoted 1 line

6. Multiple hierarchies

Do you realize this is the exact same thing I proposed in our last round, and you keep screaming saying you wanted something else, right?

The only difference is that the discussion at the time started by a forced-comount patch, but that is not the core of the question. For that you are proposing to make sense, the controllers need to be comounted, and at some point we'll have to enforce it. Be it now or in the future. But what to do when they are in fact comounted, I see no difference from what you are saying, and what I said.

Quoted 6 lines

7. Misc issues

* Sort & unique when listing tasks. Even the documentation says it doesn't happen but we have a good hunk of code doing it in cgroup.c. I'm gonna rip it out at some point. Again, if you don't like it, scream.

In all honesty, I never noticed that. ugh

Hello, Glauber.

Quoted 1 line · Glauber Costa

Can we please keep some key userspace guys CCd?

Yeap, thanks for adding the ccs.

Quoted 1 line

Quoted 1 line

1. cpu and cpuacct

...

Quoted 4 lines

Quoted 1 line

Me, working on it.

I can work on it as well if you want. I dealt with it many times in the past, and tried some different approaches, so I am familiar. But if you're already doing it, be my guest...

I'm trying something minimal which can serve as basis for the actual work. I think I figured it out mostly and will probably post it later today. Will squeak if I get stuck.

Quoted 4 lines

Quoted 3 lines

I'll do the cgroup_freezer. I'm hoping PeterZ or someone who's familiar with the code base takes care of cpuset. Michal, can you please take care of memcg?

Quoted 3 lines

I think this is a pressing problem, yes, but not the only problem with cgroup lock. Even if we restrict its usage to cgroup core, we still can call cgroup functions, which will lock. And then we gain nothing.

Can you be a bit more specific?

Quoted 4 lines

And the problem is that people need to lock. cgroup_lock is needed because the data you are accessing is protected by it. The way I see it, it is incredible how we were able to revive the BKL in the form of cgroup_lock after we finally manage to successfully get rid of it!

I wouldn't go as far as comparing it to BKL.

Quoted 9 lines

Do you realize this is the exact same thing I proposed in our last round, and you keep screaming saying you wanted something else, right?

The only difference is that the discussion at the time started by a forced-comount patch, but that is not the core of the question. For that you are proposing to make sense, the controllers need to be comounted, and at some point we'll have to enforce it. Be it now or in the future. But what to do when they are in fact comounted, I see no difference from what you are saying, and what I said.

Maybe I misunderstood you or from still talking about forced co-mounts more likely you're still misunderstanding. From what you told PeterZ, it seemed like you were thinking that this somehow will get rid of differing hierarchies depending on specific controllers and thus will help, for example, the optimization issues between cpu and cpuacct. Going back to the above example,

Unified tree Controller Y's view
controller X's view

R R
/ \ / \
A B A B
/ \
AA AB

If a task assigned to or resourced tagged with AA, for controller X it'll map to AA and for controller Y to A, so we would still need css_set, which actually becomes the primary resource tag and may point to different subsystem states depending on the specific controller.

If that is the direction we're headed, forcing co-mounts at this point doesn't make any sense. We'll make things which are possible today impossible for quite a while and then restore part of it, which is a terrible transition plan. What we need to do is nudging the current users away from practices which hinder implementation of the final form and then transition to it gradually.

If you still don't understand, I don't know what more I can do to help.

Quoted 1 line

Can you be a bit more specific?

What I mean is that if some operation needs to operate locked, they will have to lock. Whether or not the locking is called from cgroup core or not. If the lock is not available outside, people will end up calling a core function that locks.

Quoted 6 lines

Quoted 4 lines

And the problem is that people need to lock. cgroup_lock is needed because the data you are accessing is protected by it. The way I see it, it is incredible how we were able to revive the BKL in the form of cgroup_lock after we finally manage to successfully get rid of it!

I wouldn't go as far as comparing it to BKL.

Of course not, since it is not system-wide. But I think the comparison still holds in spirit...

you seem to hear "comount", and think of unified vision, and that is the reason for this discussion to still be going on. Mounting is all about the root. And if you comount, hierarchies have the same root.

In your example, the different controllers are comounted. They have not the same view, but the possible views are restricted to be a subset of the underlying tree - because they are mounted in the same place, forced or not.

In a situation like this, it makes all the sense in the world to use the css_id as a primary identifier, because it will be guaranteed to be the

same. What makes the tree overly flexible, is that you can have multiple roots, starting in multiple places, with arbitrary topologies downwards.

Hello, Glauber.

Quoted 6 lines · Glauber Costa

Quoted 1 line

Can you be a bit more specific?

What I mean is that if some operation needs to operate locked, they will have to lock. Whether or not the locking is called from cgroup core or not. If the lock is not available outside, people will end up calling a core function that locks.

I was asking whether you have certain specific operations on mind.

Quoted 9 lines

Quoted 6 lines

Quoted 4 lines

And the problem is that people need to lock. cgroup_lock is needed because the data you are accessing is protected by it. The way I see it, it is incredible how we were able to revive the BKL in the form of cgroup_lock after we finally manage to successfully get rid of it!

I wouldn't go as far as comparing it to BKL.

Of course not, since it is not system-wide. But I think the comparison still holds in spirit...

Subsystem-wide locks covering non-hot paths aren't evil things. We have a lot of them and they work fine. BKL was a completely different beast initially with implicit locking on kernel entry and unlocking on sleeping and then got morphed into some chimera inbetween afterwards.

Simple locking is a good thing. If finer-grained locking is necessary, we sure do that but please stop throwing over-generalized half-arguments at it. It doesn't help anything.

Quoted 8 lines

you seem to hear "comount", and think of unified vision, and that is the reason for this discussion to still be going on. Mounting is all about the root. And if you comount, hierarchies have the same root.

In your example, the different controllers are comounted. They have not the same view, but the possible views are restricted to be a subset of the underlying tree - because they are mounted in the same place, forced or not.

Heh, I can't really tell whether you understand it or not. Here and in the previous thread too. You seem to understand that there are different views upto this point.

Quoted 2 lines

In a situation like this, it makes all the sense in the world to use the css_id as a primary identifier, because it will be guaranteed to be the

And then you say something like this (or that this would remove walking different hierarchies in the previous thread - yes, to a certain point but not completely). css_id is a per-css attribute. How can that be the "primariy" identifier when there can be multiple views? For each userland-visible cgroup, there must be a css_set which points to the css's belonging to it, which may not be at the same level - multiple nodes in the userland visible tree may point to the same css.

If you mean that css_id would be the primary identifier for that specific controller's css, why even say that? That's true now and won't ever change.

Quoted 2 lines

same. What makes the tree overly flexible, is that you can have multiple roots, starting in multiple places, with arbitrary topologies downwards.

And now you seem to be on the same page again. But then again, you're asserting that incorporating forced co-mounts *now* is a gradual step towards the goal, which is utterly bonkers. I don't know. I just can't understand what you're thinking at all.

Thanks.

Hello, Glauber.

Quoted 1 line · Glauber Costa

Can we please keep some key userspace guys CCd?

Yeap, thanks for adding the ccs.

Quoted 1 line

Quoted 1 line

1. cpu and cpuacct

...

Quoted 4 lines

Quoted 1 line

Me, working on it.

I can work on it as well if you want. I dealt with it many times in the past, and tried some different approaches, so I am familiar. But if you're already doing it, be my guest...

I'm trying something minimal which can serve as basis for the actual work. I think I figured it out mostly and will probably post it later today. Will squeak if I get stuck.

Quoted 4 lines

Quoted 3 lines

I'll do the cgroup_freezer. I'm hoping PeterZ or someone who's familiar with the code base takes care of cpuset. Michal, can you please take care of memcg?

Quoted 3 lines

I think this is a pressing problem, yes, but not the only problem with cgroup lock. Even if we restrict its usage to cgroup core, we still can call cgroup functions, which will lock. And then we gain nothing.

Can you be a bit more specific?

Quoted 4 lines

And the problem is that people need to lock. cgroup_lock is needed because the data you are accessing is protected by it. The way I see it, it is incredible how we were able to revive the BKL in the form of cgroup_lock after we finally manage to successfully get rid of it!

I wouldn't go as far as comparing it to BKL.

Quoted 9 lines

Do you realize this is the exact same thing I proposed in our last round, and you keep screaming saying you wanted something else, right?

The only difference is that the discussion at the time started by a forced-comount patch, but that is not the core of the question. For that you are proposing to make sense, the controllers need to be comounted, and at some point we'll have to enforce it. Be it now or in the future. But what to do when they are in fact comounted, I see no difference from what you are saying, and what I said.

Maybe I misunderstood you or from still talking about forced co-mounts more likely you're still misunderstanding. From what you told PeterZ, it seemed like you were thinking that this somehow will get rid of differing hierarchies depending on specific controllers and thus will help, for example, the optimization issues between cpu and cpuacct. Going back to the above example,

Unified tree Controller Y's view
controller X's view

R R
/ \ / \
A B A B
/ \
AA AB

If a task assigned to or resourced tagged with AA, for controller X it'll map to AA and for controller Y to A, so we would still need css_set, which actually becomes the primary resource tag and may point to different subsystem states depending on the specific controller.

If that is the direction we're headed, forcing co-mounts at this point doesn't make any sense. We'll make things which are possible today impossible for quite a while and then restore part of it, which is a terrible transition plan. What we need to do is nudging the current users away from practices which hinder implementation of the final form and then transition to it gradually.

If you still don't understand, I don't know what more I can do to help.

7. Misc issues

The cpu ones handle nesting correctly - parent's accounting includes children's, parent's configuration affects children's unless explicitly overridden, and children's limits nest inside parent's.

2. memcg's __DEPRECATED_clear_css_refs

This is a remnant of another weird design decision of requiring synchronous draining of refcnts on cgroup removal and allowing subsystems to veto cgroup removal - what's the userspace supposed to do afterwards? Note that this also hinders co-mounting different controllers.

The behavior could be useful for development and debugging but it unnecessarily interlocks userland visible behavior with in-kernel implementation details. To me, it seems outright wrong (either implement proper severing semantics in the controller or do full refcnting) and disallows, for example, lazy drain of caching refs. Also, it complicates the removal path with try / commit / revert logic which has never been fully correct since the beginning.

Currently, the only left user is memcg.

Solution:

* Update memcg->pre_destroy() such that it never fails.

* Drop __DEPRECATED_clear_css_refs and all related logic. Convert pre_destroy() to return void.

Who:

KAMEZAWA, Michal, PLEASE. I will make __DEPRECATED_clear_css_refs trigger WARN sooner or later. Let's please get this settled.

* memcg can be fully hierarchical but we need to phase out the flat hierarchy support. Unfortunately, this involves flipping the behavior for the existing users. Upstream will try to nudge users with warning messages. Most burden would be on the distros and at least SUSE seems to be on board with it. Needs coordination with other distros.

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

* Sort & unique when listing tasks. Even the documentation says it doesn't happen but we have a good hunk of code doing it in cgroup.c. I'm gonna rip it out at some point. Again, if you don't like it, scream.

6. Multiple hierarchies

Apart from the apparent wheeeeeeeeness of it (I think I talked about that enough the last time[1]), there's a basic problem when more than one controllers interact - it's impossible to define a resource group when more than two controllers are involved because the intersection of different controllers is only defined in terms of tasks.

IOW, if an entity X is of interest to two controllers, there's no way to map X to the cgroups of the two controllers. X may belong to A and B when viewed by one task but A' and B when viewed by another. This already is a head scratcher in writeback where blkcg and memcg have to interact.

While I am pushing for unified hierarchy, I think it's necessary to have different levels of granularities depending on controllers given that nesting involves significant overhead and noticeable controller-dependent behavior changes.

...

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

7. Misc issues

* Sort & unique when listing tasks. Even the documentation says it doesn't happen but we have a good hunk of code doing it in cgroup.c. I'm gonna rip it out at some point. Again, if you don't like it, scream.

* At the PLC, pjt told me that assinging threads of a cgroup to different cgroups is useful for some use cases but if we're to have a unified hierarchy, I don't think we can continue to do that. Paul, can you please elaborate the use case?

* Vivek brought up the issue of distributing resources to tasks and groups in the same cgroup. I don't know. Need to think more about it.

The cpu ones handle nesting correctly - parent's accounting includes children's, parent's configuration affects children's unless explicitly overridden, and children's limits nest inside parent's.

Quoted 3 lines · Tejun Heo

The cpu ones handle nesting correctly - parent's accounting includes children's, parent's configuration affects children's unless explicitly overridden, and children's limits nest inside parent's.

The implementation has some issues with fixed point math limitations on deep hierarchies/large cpu count, but yes.

Doing soft-float/bignum just isn't going to be popular I guess ;-)

People also don't seem to understand that each extra cgroup carries a cost and that nested cgroups are more expensive still, even if the intermediate levels are mostly empty (libvirt is a good example of how not to do things).

Anyway, I guess what I'm saying is that we need to work on the awareness of cost associated with all this cgroup nonsense, people seem to think its all good and free -- or not think at all, which, while depressing, seem the more likely option.

Quoted 15 lines · Peter Zijlstra

Quoted 3 lines · Tejun Heo

The cpu ones handle nesting correctly - parent's accounting includes children's, parent's configuration affects children's unless explicitly overridden, and children's limits nest inside parent's.

The implementation has some issues with fixed point math limitations on deep hierarchies/large cpu count, but yes.

Doing soft-float/bignum just isn't going to be popular I guess ;-)

People also don't seem to understand that each extra cgroup carries a cost and that nested cgroups are more expensive still, even if the intermediate levels are mostly empty (libvirt is a good example of how not to do things).

Quoted 4 lines

Anyway, I guess what I'm saying is that we need to work on the awareness of cost associated with all this cgroup nonsense, people seem to think its all good and free -- or not think at all, which, while depressing, seem the more likely option.

In defense of what libvirt is doing, I'll point out that the kernel docs on cgroups make little to no mention of these performance / cost implications, and the examples of usage given arguably encourage use of deep hierarchies.

Given what we've now learnt about the kernel's lack of scalability wrt cgroup hierarchies, we'll be changing the way libvirt deals with cgroups, to flatten it out to only use 1 level by default. If the kernel docs had clearly expressed the limitations & made better recommendations on app usage we would never have picked the approach we originally chose.

Regards,
Daniel

Quoted 3 lines · Tejun Heo

The cpu ones handle nesting correctly - parent's accounting includes children's, parent's configuration affects children's unless explicitly overridden, and children's limits nest inside parent's.

The implementation has some issues with fixed point math limitations on deep hierarchies/large cpu count, but yes.

Doing soft-float/bignum just isn't going to be popular I guess ;-)

People also don't seem to understand that each extra cgroup carries a cost and that nested cgroups are more expensive still, even if the intermediate levels are mostly empty (libvirt is a good example of how not to do things).

Anyway, I guess what I'm saying is that we need to work on the awareness of cost associated with all this cgroup nonsense, people seem to think its all good and free -- or not think at all, which, while depressing, seem the more likely option.

Quoted 3 lines · Tejun Heo

The cpu ones handle nesting correctly - parent's accounting includes children's, parent's configuration affects children's unless explicitly overridden, and children's limits nest inside parent's.

The implementation has some issues with fixed point math limitations on deep hierarchies/large cpu count, but yes.

Doing soft-float/bignum just isn't going to be popular I guess ;-)

People also don't seem to understand that each extra cgroup carries a cost and that nested cgroups are more expensive still, even if the intermediate levels are mostly empty (libvirt is a good example of how not to do things).

Anyway, I guess what I'm saying is that we need to work on the awareness of cost associated with all this cgroup nonsense, people seem to think its all good and free -- or not think at all, which, while depressing, seem the more likely option.

7. Misc issues

Quoted 2 lines · Tejun Heo

7. Misc issues

* Extract synchronize_rcu() from user interface? Exporting grace
periods to userspace isn't wonderful for dynamic launchers.

5. I CAN HAZ HIERARCHIES?

The cpu ones handle nesting correctly - parent's accounting includes children's, parent's configuration affects children's unless explicitly overridden, and children's limits nest inside parent's.

memcg asked itself the existential question of to be hierarchical or not and then got confused and decided to become both.

When faced with the same question, blkio and cgroup_freezer just gave up and decided to allow nesting and then ignore it - brilliant.

And there are others which kinda sorta try to handle hierarchy but only goes way-half.

This one is screwed up embarrassingly badly. We failed to establish one of the most basic semantics and can't even define what a cgroup hierarchy is - it depends on each controller and they're mostly wacky!

Fortunately, I don't think it will be prohibitively difficult to dig ourselves out of this hole.

Solution:

* cpu ones seem fine.

* For broken controllers, cgroup core will be generating warning messages if the user tries to nest cgroups so that the user at least can know that the behavior may change underneath them later on. For more details,

http://thread.gmane.org/gmane.linux.kernel/1356264/focus=3902

* memcg can be fully hierarchical but we need to phase out the flat hierarchy support. Unfortunately, this involves flipping the behavior for the existing users. Upstream will try to nudge users with warning messages. Most burden would be on the distros and at least SUSE seems to be on board with it. Needs coordination with other distros.

* blkio is the most problematic. It has two sub-controllers - cfq and blk-throttle. Both are utterly broken in terms of hierarchy support and the former is known to have pretty hairy code base. I don't see any other way than just biting the bullet and fixing it.

* cgroup_freezer and others shouldn't be too difficult to fix.

Who:

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

6. Multiple hierarchies

Apart from the apparent wheeeeeeeeness of it (I think I talked about that enough the last time[1]), there's a basic problem when more than one controllers interact - it's impossible to define a resource group when more than two controllers are involved because the intersection of different controllers is only defined in terms of tasks.

IOW, if an entity X is of interest to two controllers, there's no way to map X to the cgroups of the two controllers. X may belong to A and B when viewed by one task but A' and B when viewed by another. This already is a head scratcher in writeback where blkcg and memcg have to interact.

While I am pushing for unified hierarchy, I think it's necessary to have different levels of granularities depending on controllers given that nesting involves significant overhead and noticeable controller-dependent behavior changes.

Solution:

I think a unified hierarchy with the ability to ignore subtrees depending on controllers should work. For example, let's assume the following hierarchy.

          R
	/   \
       A     B
      / \
     AA AB

All controllers are co-mounted. There is per-cgroup knob which controls which controllers nest beyond it. If blkio doesn't want to distinguish AA and AB, the user can specify that blkio doesn't nest beyond A and blkio would see the tree as,

          R
	/   \
       A     B

While other controllers keep seeing the original tree. The exact form of interface, I don't know yet. It could be a single file which the user echoes [-]controller name into it or per-controller boolean file.

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

I *think* this can be achieved by changing where css_set is bound. Currently, a css_set is (conceptually) owned by a task. After the change, a cgroup in the unified hierarchy has its own css_set which tasks point to and can also be used to tag resources as necessary. This way, it should be achieveable without introducing a lot of new code or affecting individual controllers too much.

The headache will be the transition period where we'll probably have to support both modes of operation. Oh well....

Who:

Li, Glauber and me, I guess?

libvirtd will always create its own cgroups starting at the location where libvirtd itself has been placed. This is to co-operate with systemd / initscripts which may place each system service in a dedicated group

Quoted 41 lines

6. Multiple hierarchies

Apart from the apparent wheeeeeeeeness of it (I think I talked about that enough the last time[1]), there's a basic problem when more than one controllers interact - it's impossible to define a resource group when more than two controllers are involved because the intersection of different controllers is only defined in terms of tasks.

IOW, if an entity X is of interest to two controllers, there's no way to map X to the cgroups of the two controllers. X may belong to A and B when viewed by one task but A' and B when viewed by another. This already is a head scratcher in writeback where blkcg and memcg have to interact.

While I am pushing for unified hierarchy, I think it's necessary to have different levels of granularities depending on controllers given that nesting involves significant overhead and noticeable controller-dependent behavior changes.

Solution:

I think a unified hierarchy with the ability to ignore subtrees depending on controllers should work. For example, let's assume the following hierarchy.

          R
	/   \
       A     B
      / \
     AA AB

All controllers are co-mounted. There is per-cgroup knob which controls which controllers nest beyond it. If blkio doesn't want to distinguish AA and AB, the user can specify that blkio doesn't nest beyond A and blkio would see the tree as,

          R
	/   \
       A     B

Quoted 7 lines

While other controllers keep seeing the original tree. The exact form of interface, I don't know yet. It could be a single file which the user echoes [-]controller name into it or per-controller boolean file.

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

I am little concerned about above and wondering how systemd and libvirt will interact and behave out of the box.

Currently systemd does not create its own hierarchy under blkio and libvirt does. So putting all together means there is no way to avoid the overhead of systemd created hierarchy.

\
|

+- system
     |
     +- libvirtd.service
              |
              +- virt-machine1
              +- virt-machine2

So there is now way to avoid the overhead of two levels of hierarchy created by systemd. I really wish that systemd gets rid of "system" cgroup and puts services directly in top level group. Creating deeper hieararchices is expensive.

I just want to mention it clearly that with above model, it will not be possible for libvirt to avoid hierarchy levels created by systemd. So solution would be to keep depth of hierarchy as low as possible and to keep controller overhead as low as possible.

Now I know that with blkio idling kills performance. So one solution could be that on anything fast, don't use CFQ. Use deadline and then group idling overhead goes away and tools like systemd and libvirt don't have to worry about keeping track of disks and what scheduler is running. They don't want to do it and expect kernel to get it right.

Quoted 16 lines · Vivek Goyal

I am little concerned about above and wondering how systemd and libvirt will interact and behave out of the box.

Currently systemd does not create its own hierarchy under blkio and libvirt does. So putting all together means there is no way to avoid the overhead of systemd created hierarchy.

\
|

+- system
     |
     +- libvirtd.service
              |
              +- virt-machine1
              +- virt-machine2

Quoted 4 lines

So there is now way to avoid the overhead of two levels of hierarchy created by systemd. I really wish that systemd gets rid of "system" cgroup and puts services directly in top level group. Creating deeper hieararchices is expensive.

* blkio is the most problematic. It has two sub-controllers - cfq and blk-throttle. Both are utterly broken in terms of hierarchy support and the former is known to have pretty hairy code base. I don't see any other way than just biting the bullet and fixing it.

* cgroup_freezer and others shouldn't be too difficult to fix.

Who:

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

* Vivek brought up the issue of distributing resources to tasks and groups in the same cgroup. I don't know. Need to think more about it.

So while % model is more intutive to users, it is hard to implement.

So
an easier way is to stick to the model of relative weights/share and let user specify relative importance of a virtual machine and actual quota or % will vary dynamically depending on other tasks/components in the system.

Thoughts?

Quoted 1 line · Vivek Goyal

So while % model is more intutive to users, it is hard to implement.

I don't agree with that. The fixed quota thing is counter-intuitive and hard to use. It begets you questions like: why, if everything is idle except my task, am I not getting the full throughput.

It also makes adding entities harder because you're constrained to 100%. This means you have to start each new cgroup with 0% because any !0 value will eventually get you over 100%, it also means you have to do some form of admission control to make sure you never get over that 100%.

Starting with 0% is not convenient for people.. they think this is the wrong default, even though as argued above, it is the only possible value.

Quoted 7 lines

So
an easier way is to stick to the model of relative weights/share and let user specify relative importance of a virtual machine and actual quota or % will vary dynamically depending on other tasks/components in the system.

Quoted 1 line

Thoughts?

cpu does the relative weight, so 'users' will have to deal with it anyway regardless of blk, its effectively free of learning curve for all subsequent controllers.

We don't have to start with 0%. We can keep a pool with dynamic % and launch all the virtual machines from that single pool. So nobody starts with 0%. If we require certain % for a machine, only then we look at peers and see if we have bandwidth free and create cgroup and move virtual machine there, otherwise we deny resources.

So I think it is doable just that it is painful and tricky and I think lot of it will be in user space.

Quoted 8 lines

Quoted 7 lines

So
an easier way is to stick to the model of relative weights/share and let user specify relative importance of a virtual machine and actual quota or % will vary dynamically depending on other tasks/components in the system.

Thoughts?

Quoted 3 lines

cpu does the relative weight, so 'users' will have to deal with it anyway regardless of blk, its effectively free of learning curve for all subsequent controllers.

I am inclined to keep it simple in kernel and just follow cpu model of relative weights and treating tasks and gropu at same level in the hierarchy. It makes behavior consistent across the controllers and I think it might just work for majority of cases.

Quoted 8 lines

Quoted 3 lines

cpu does the relative weight, so 'users' will have to deal with it anyway regardless of blk, its effectively free of learning curve for all subsequent controllers.

I am inclined to keep it simple in kernel and just follow cpu model of relative weights and treating tasks and gropu at same level in the hierarchy. It makes behavior consistent across the controllers and I think it might just work for majority of cases.

I think we need to stick to one model for all controllers; otherwise, it gets confusing and unified hierarchy can't work. That said, I'm not too happy about how cpu is handling it now.

* As I wrote before, the configuration esacpes cgroup proper and the mapping from per-task value to group weight is essentially arbitrary and may not exist depending on the resource type.

* The proportion of each group fluctuates as tasks fork and exit in the parent group, which is confusing.

* cpu deals with tasks but blkcg deals with iocontexts and memcg, which currently doesn't implement proportional control, deals with address spaces (processes). The proportions wouldn't even fluctuate the same way across different controllers.

So, I really don't think the current model used by cpu is a good one and we rather should treat the tasks as a group competing with the rest of child groups. Whether we can change that at this point, I don't know. Peter, what do you think?

I think we need to stick to one model for all controllers; otherwise, it gets confusing and unified hierarchy can't work. That said, I'm not too happy about how cpu is handling it now.

* As I wrote before, the configuration esacpes cgroup proper and the mapping from per-task value to group weight is essentially arbitrary and may not exist depending on the resource type.

* The proportion of each group fluctuates as tasks fork and exit in the parent group, which is confusing.

* cpu deals with tasks but blkcg deals with iocontexts and memcg, which currently doesn't implement proportional control, deals with address spaces (processes). The proportions wouldn't even fluctuate the same way across different controllers.

So, I really don't think the current model used by cpu is a good one and we rather should treat the tasks as a group competing with the rest of child groups. Whether we can change that at this point, I don't know. Peter, what do you think?

Those who really need to implement % model, they will have to do heavy lifting in user space. I am skeptical that will take off but kernel does not prohibit from somebody creating a group, moving all tasks there and making sure tasks and groups are not at same level hence % becomes more predictable. Just that, that's not the default from kernel.

On Thu, Sep 13, 2012 at 01:58:27PM -0700, Tejun Heo wrote:

[..]

Quoted 4 lines

* blkio is the most problematic. It has two sub-controllers - cfq and blk-throttle. Both are utterly broken in terms of hierarchy support and the former is known to have pretty hairy code base. I don't see any other way than just biting the bullet and fixing it.

I am still little concerned about changing the blkio behavior unexpectedly. Can we have some kind of mount time flag which retains the old flat behavior and we warn user that this mode is deprecated and will soon be removed. Move over to hierarchical mode. Then after few release we can drop the flag and cleanup any extra code which supports flat mode in CFQ. This will atleast make transition smooth.

Quoted 4 lines

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

I will try to spend some time on this. Doing changes in blk-throttle should be relatively easy. Painful part if CFQ. It does so much that it is not clear whether a particular change will bite us badly or not. So doing changes becomes hard. There are heuristics, preemptions, queue selection logic, service tree and bringing it all together for full hierarchy becomes interesting.

I think first thing which needs to be done is merge group scheduling and cfqq scheduling. Because of flat hierarchy currently we use two scheduling algorithm. Old logic for queue selection and new logic for group scheduling. If we treat task and group at same level then we have to merge two and come up with single logic.

[..]

Quoted 3 lines

* Vivek brought up the issue of distributing resources to tasks and groups in the same cgroup. I don't know. Need to think more about it.

This one will require some thought. I have heard arguments for both the models. Treating tasks and groups at same level seem to have one disadvantange and that is that people can't think of system resources in terms of %. People often say, give 20% of disk resources to a particular cgroup. But it is not possible as there are all kernel threads running in root cgroup and tasks come and go and that means % share of a group is variable and not fixed.

To make it fixed, we will need to make sure that number of entities fighting for resources are not variable. That means only group fight for resources at a level and tasks with-in groups.

Now the question is should kernel enforce it or should it be left to user space. I think doing it in user space is also messy as different agents control different part of hiearchy. For example, if somebody says that give a particular virtual machine a x% of system resource, libvirt has no way to do that. At max it can ensure x% of parent group but above that hierarchy is controlled by systemd and libvirtd has no control over that.

Only possible way to do this will seem to be that systemd creates libvirt group at top level with a minimum fixed % of quota and then libvirt can figure out % share of each virtual machine. But it is hard to do.

So while % model is more intutive to users, it is hard to implement. So an easier way is to stick to the model of relative weights/share and let user specify relative importance of a virtual machine and actual quota or % will vary dynamically depending on other tasks/components in the system.

Quoted 6 lines

I am still little concerned about changing the blkio behavior unexpectedly. Can we have some kind of mount time flag which retains the old flat behavior and we warn user that this mode is deprecated and will soon be removed. Move over to hierarchical mode. Then after few release we can drop the flag and cleanup any extra code which supports flat mode in CFQ. This will atleast make transition smooth.

I don't know. That essentially is what we're doing with memcg now and it doesn't seem any less messy. Given the already scary complexity, do we really want to support both flat and hierarchy models at the same time?

Quoted 13 lines

Quoted 5 lines

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

I will try to spend some time on this. Doing changes in blk-throttle should be relatively easy. Painful part if CFQ. It does so much that it is not clear whether a particular change will bite us badly or not. So doing changes becomes hard. There are heuristics, preemptions, queue selection logic, service tree and bringing it all together for full hierarchy becomes interesting.

Quoted 5 lines

I think first thing which needs to be done is merge group scheduling and cfqq scheduling. Because of flat hierarchy currently we use two scheduling algorithm. Old logic for queue selection and new logic for group scheduling. If we treat task and group at same level then we have to merge two and come up with single logic.

I think this depends on how we decide to handle tasks vs. groups, right?

Quoted 6 lines

[..]

Quoted 3 lines

* Vivek brought up the issue of distributing resources to tasks and groups in the same cgroup. I don't know. Need to think more about it.

Quoted 7 lines

This one will require some thought. I have heard arguments for both the models. Treating tasks and groups at same level seem to have one disadvantange and that is that people can't think of system resources in terms of %. People often say, give 20% of disk resources to a particular cgroup. But it is not possible as there are all kernel threads running in root cgroup and tasks come and go and that means % share of a group is variable and not fixed.

Another problem is that configuration isn't contained in cgroup proper. We need a way to assign weights to individual tasks which can be somehow directly compared against group weights. cpu cooks priority for this and blkcg may be able to cook ioprio but it's nasty and unobvious. Also, let's say we grow network bandwidth controller for whatever reason. What value are we gonna use?

Quoted 17 lines

To make it fixed, we will need to make sure that number of entities fighting for resources are not variable. That means only group fight for resources at a level and tasks with-in groups.

Now the question is should kernel enforce it or should it be left to user space. I think doing it in user space is also messy as different agents control different part of hiearchy. For example, if somebody says that give a particular virtual machine a x% of system resource, libvirt has no way to do that. At max it can ensure x% of parent group but above that hierarchy is controlled by systemd and libvirtd has no control over that.

Only possible way to do this will seem to be that systemd creates libvirt group at top level with a minimum fixed % of quota and then libvirt can figure out % share of each virtual machine. But it is hard to do.

Quoted 5 lines

So while % model is more intutive to users, it is hard to implement. So an easier way is to stick to the model of relative weights/share and let user specify relative importance of a virtual machine and actual quota or % will vary dynamically depending on other tasks/components in the system.

Why is it hard to implement? You just need to treat tasks in the current node as another group competing with other cgroups on equal terms. If anything, isn't that simpler than treating scheduling "entities"?

As a developer, I will be happy to support only one model and keep code simple. I am only concerned that for blkcg we have still not charted out a clear migration path. The warning message your patch is giving out will work only if we decide to not treat task and groups at same level.

Quoted 6 lines

Another problem is that configuration isn't contained in cgroup proper. We need a way to assign weights to individual tasks which can be somehow directly compared against group weights. cpu cooks priority for this and blkcg may be able to cook ioprio but it's nasty and unobvious. Also, let's say we grow network bandwidth controller for whatever reason. What value are we gonna use?

So if somebody cares about settting SO_PRIORITY for traffic originating from a tasks, move it into a cgroup. Otherwise they all get default priority.

So to me, leaving this decision to userspace based on their requirement makes sense.

Yes, creating a hidden group for tasks in current group should not be hard from implementation point of view. But again, I am concerned about configuration of hidden group and I also don't like the idea of taking flexibility away from user to treat tasks and group at same level.

2. memcg's __DEPRECATED_clear_css_refs

This is a remnant of another weird design decision of requiring synchronous draining of refcnts on cgroup removal and allowing subsystems to veto cgroup removal - what's the userspace supposed to do afterwards? Note that this also hinders co-mounting different controllers.

The behavior could be useful for development and debugging but it unnecessarily interlocks userland visible behavior with in-kernel implementation details. To me, it seems outright wrong (either implement proper severing semantics in the controller or do full refcnting) and disallows, for example, lazy drain of caching refs. Also, it complicates the removal path with try / commit / revert logic which has never been fully correct since the beginning.

Currently, the only left user is memcg.

Solution:

* Update memcg->pre_destroy() such that it never fails.

* Drop __DEPRECATED_clear_css_refs and all related logic. Convert pre_destroy() to return void.

Who:

KAMEZAWA, Michal, PLEASE. I will make __DEPRECATED_clear_css_refs trigger WARN sooner or later. Let's please get this settled.

* memcg can be fully hierarchical but we need to phase out the flat hierarchy support. Unfortunately, this involves flipping the behavior for the existing users. Upstream will try to nudge users with warning messages. Most burden would be on the distros and at least SUSE seems to be on board with it. Needs coordination with other distros.

I am currently planning to add a warning to most of the currenly maintained distributions to have as big coverage as possible. No default switch for obvious reasons but hopefuly we will get some feedback at least.

Quoted 10 lines · Michal Hocko

mm/memcontrol.c +2 -0

index f99f599..b61c34b 100644
@@ -3106,6 +3106,11 @@ mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 	} else {
 		parent = mem_cgroup_from_cont(cont->parent);
 		mem->use_hierarchy = parent->use_hierarchy;
+		WARN_ONCE(!mem->use_hierarchy && parent != root_mem_cgroup,
+				"Creating hierarchies with use_hierarchy==0 "

mm/memcontrol.c mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont) · line 3113

Quoted 3 lines

mm/memcontrol.c +3 -0

+				"(flat hierarchy) is considered deprecated. "
+				"If you believe that your setup is correct, "
+				"we kindly ask you to contact [email protected] and let us know");

When I deprecated /proc/pid/oom_adj (now removed), we did a WARN_ONCE() and others complained that this unnecessarily alters userspace scripts that a serious issue has occurred and Linus agreed that we shouldn't do deprecation in this way. The alternative is to use printk_once() instead.

Yes printk_once is an alternative but I really wanted to have this as much visible as possible. People tend to react to stack traceces more and this one will trigger only if somebody is either doing something wrong or the configuration is the one we are looking for.

Comparing to oom_adj, that one was used much more often so the WARN_ONCE was too verbose especially when you usually had to wait for an userspace update which is not the case here.

Quoted 5 lines · Michal Hocko

Yes printk_once is an alternative but I really wanted to have this as much visible as possible. People tend to react to stack traceces more and this one will trigger only if somebody is either doing something wrong or the configuration is the one we are looking for.

That's the complete opposite of what Linus has said he wants, he said very specifically that he doesn't want WARN_ONCE() or WARN_ON_ONCE() for deprecation of tunables. If you want to have this merged, then please get him to ack it.

Quoted 3 lines

Comparing to oom_adj, that one was used much more often so the WARN_ONCE was too verbose especially when you usually had to wait for an userspace update which is not the case here.

How is WARN_ONCE() too verbose for oom_adj? It's printed once!

And how can you claim that userspace doesn't need to change if it's creating a hierarchy while use_hierarchy == 0?

Quoted 5 lines · Michal Hocko

Yes printk_once is an alternative but I really wanted to have this as much visible as possible. People tend to react to stack traceces more and this one will trigger only if somebody is either doing something wrong or the configuration is the one we are looking for.

That's the complete opposite of what Linus has said he wants, he said very specifically that he doesn't want WARN_ONCE() or WARN_ON_ONCE() for deprecation of tunables. If you want to have this merged, then please get him to ack it.

Quoted 3 lines

Comparing to oom_adj, that one was used much more often so the WARN_ONCE was too verbose especially when you usually had to wait for an userspace update which is not the case here.

How is WARN_ONCE() too verbose for oom_adj? It's printed once!

And how can you claim that userspace doesn't need to change if it's creating a hierarchy while use_hierarchy == 0?

Quoted 10 lines · Michal Hocko

mm/memcontrol.c +2 -0

index f99f599..b61c34b 100644
@@ -3106,6 +3106,11 @@ mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont)
 	} else {
 		parent = mem_cgroup_from_cont(cont->parent);
 		mem->use_hierarchy = parent->use_hierarchy;
+		WARN_ONCE(!mem->use_hierarchy && parent != root_mem_cgroup,
+				"Creating hierarchies with use_hierarchy==0 "

mm/memcontrol.c mem_cgroup_create(struct cgroup_subsys *ss, struct cgroup *cont) · line 3113

Quoted 3 lines

mm/memcontrol.c +3 -0

+				"(flat hierarchy) is considered deprecated. "
+				"If you believe that your setup is correct, "
+				"we kindly ask you to contact [email protected] and let us know");

When I deprecated /proc/pid/oom_adj (now removed), we did a WARN_ONCE() and others complained that this unnecessarily alters userspace scripts that a serious issue has occurred and Linus agreed that we shouldn't do deprecation in this way. The alternative is to use printk_once() instead.

I am currently planning to add a warning to most of the currenly maintained distributions to have as big coverage as possible. No default switch for obvious reasons but hopefuly we will get some feedback at least.

6. Multiple hierarchies

Apart from the apparent wheeeeeeeeness of it (I think I talked about that enough the last time[1]), there's a basic problem when more than one controllers interact - it's impossible to define a resource group when more than two controllers are involved because the intersection of different controllers is only defined in terms of tasks.

IOW, if an entity X is of interest to two controllers, there's no way to map X to the cgroups of the two controllers. X may belong to A and B when viewed by one task but A' and B when viewed by another. This already is a head scratcher in writeback where blkcg and memcg have to interact.

While I am pushing for unified hierarchy, I think it's necessary to have different levels of granularities depending on controllers given that nesting involves significant overhead and noticeable controller-dependent behavior changes.

Solution:

I think a unified hierarchy with the ability to ignore subtrees depending on controllers should work. For example, let's assume the following hierarchy.

          R
	/   \
       A     B
      / \
     AA AB

All controllers are co-mounted. There is per-cgroup knob which controls which controllers nest beyond it. If blkio doesn't want to distinguish AA and AB, the user can specify that blkio doesn't nest beyond A and blkio would see the tree as,

          R
	/   \
       A     B

While other controllers keep seeing the original tree. The exact form of interface, I don't know yet. It could be a single file which the user echoes [-]controller name into it or per-controller boolean file.

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

I am curious that why are you planning to provide capability of controller specific view of hierarchy. To me it sounds pretty close to having separate hierarchies per controller. Just that it is a little more restricted configuration.

IOW, who is is the user of this functionality and who is asking for it. Can we go all out where all controllers have only one hierarchy view.

In addition, for some resources, granularity beyond certain point simply doesn't work. Per-service granularity might make sense for cpu but applying it by default would be silly for blkio.

Hmm.., In that case how libvirt will make use of blkio in the proposed scheme. We can't disable blkio nesting at "system" level. So We will have to disable it at each service level except "libvirtd" so that libvirt can use blkio for its virtual machines.

That means blkio will see each service in a cgroup of its own and if that does not make sense by default, its a problem. In the existing

scheme, atleast every service does not show up in its cgroup from blkio point of view. Everthig is in root and libvirt can create its own cgroups, keeping number of cgroups small.

I think there currently is too much (broken) flexibility and intent to remove it. That doesn't mean that removeing all flexibility is the right direction. It inherently is a balancing act and I think the proposed solution is a reasonable tradeoff. There's important difference between causing full overhead by default for all users and requiring some overhead when the use case at hand calls for the functionality.

That said, if someone can think of a better solution, I'm all ears. One thing that *has* to be maintained is that it should be able to tag a resource in such way that its associated controllers are identifiable regardless of which task is looking at it.

Hello, again.

Quoted 4 lines · Tejun Heo

That said, if someone can think of a better solution, I'm all ears. One thing that *has* to be maintained is that it should be able to tag a resource in such way that its associated controllers are identifiable regardless of which task is looking at it.

So, I thought about it more. How about we do "consider / ignore this node" instead of "(don't) nest beyond this level". For example, let's assume a tree like the following.

R
/ | \
A B C
/ \
AA AB

If we want to differentiate between AA and AB, we'll have to consider the whole tree with the previous sheme - A needs to nest, so R needs to nest and we end up with the whole tree. Instead, if we have honor / ignore this node. We can set the honor bit on A, AA and AB and see the tree as

        R
     /
    A

/ \
AA AB

We still see the intermediate A node but can ignore the other branches. Implementation and concept-wise, it's fairly simple too. For any given node and controller, you travel upwards until you meet a node which has the controller enabled and that's the cgroup the controller considers.

Thanks.

That is exactly what I proposed in our previous discussions around memcg, with files like "available_controllers" , "current_controllers". Name chosen to match what other subsystems already do.

if memcg is not in "available_controllers" for a node, it cannot be seen by anyone bellow that level.

Hello, again.

Quoted 4 lines · Tejun Heo

That said, if someone can think of a better solution, I'm all ears. One thing that *has* to be maintained is that it should be able to tag a resource in such way that its associated controllers are identifiable regardless of which task is looking at it.

So, I thought about it more. How about we do "consider / ignore this node" instead of "(don't) nest beyond this level". For example, let's assume a tree like the following.

R
/ | \
A B C
/ \
AA AB

If we want to differentiate between AA and AB, we'll have to consider the whole tree with the previous sheme - A needs to nest, so R needs to nest and we end up with the whole tree. Instead, if we have honor / ignore this node. We can set the honor bit on A, AA and AB and see the tree as

        R
     /
    A

/ \
AA AB

We still see the intermediate A node but can ignore the other branches. Implementation and concept-wise, it's fairly simple too. For any given node and controller, you travel upwards until you meet a node which has the controller enabled and that's the cgroup the controller considers.

That said, if someone can think of a better solution, I'm all ears. One thing that *has* to be maintained is that it should be able to tag a resource in such way that its associated controllers are identifiable regardless of which task is looking at it.

Hmm.., In that case how libvirt will make use of blkio in the proposed scheme. We can't disable blkio nesting at "system" level. So We will have to disable it at each service level except "libvirtd" so that libvirt can use blkio for its virtual machines.

That means blkio will see each service in a cgroup of its own and if that does not make sense by default, its a problem. In the existing

scheme, atleast every service does not show up in its cgroup from blkio point of view. Everthig is in root and libvirt can create its own cgroups, keeping number of cgroups small.

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

if Serge is not planning to do it already, I can take a look in device_cgroup.

Tejun,

Quoted 4 lines · Tejun Heo

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

if Serge is not planning to do it already, I can take a look in device_cgroup.

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

Quoted 8 lines · Aristeu Rozanski

Tejun,

Quoted 4 lines · Tejun Heo

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

if Serge is not planning to do it already, I can take a look in device_cgroup.

That's fine with me, thanks.

Quoted 3 lines

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

Before going down this road, I'd like to discuss this with at least you, me, and Eric Biederman (cc:d) as to how it relates to a device namespace.

Quoted 12 lines · Serge E. Hallyn

Quoted 8 lines · Aristeu Rozanski

Tejun,

Quoted 4 lines · Tejun Heo

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

if Serge is not planning to do it already, I can take a look in device_cgroup.

That's fine with me, thanks.

Quoted 7 lines

Quoted 3 lines

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

Before going down this road, I'd like to discuss this with at least you, me, and Eric Biederman (cc:d) as to how it relates to a device namespace.

The problem with devices.

- An unrestricted mknod gives you access to effectively any device in the system.

- During process migration if the device number changes using stat to file descriptors can fail on the same file descriptor.

- Devices coming from prexisting filesystems that we mount as unprivileged users are as dangerous as mknod but show that the problem is not limited to mknod.

- udev thinks mknod is a system call we can remove from the kernel.

---

The use cases seem comparitively simple to enumerate.

- Giving unfiltered access to a device to someone not root.

- Virtual devices that everyone uses and have no real privilege
requirements: /dev/null /dev/tty /dev/zero etc.

- Dynamically created devices /dev/loopN /dev/tun /dev/macvtapN, nbd, iscsi, /dev/ptsN, etc

---

There are a couple of solution to these problems.

- The classic solution of creating a /dev for a container before starting it.

- The devpts filesystem. This works well for unprivileged access to ptys. Except for the /dev/ptmx sillines I very like how things are handled today with devpts.

- Device control groups. I am not quite certain what to make of them. The only case I see where they are better than a prebuilt static dev is if there is a hotppluged device that I want to push into my container.

I think the only problem with device control groups and hierarchies is that removing a device from a whitelist does not recurse down the hierarchy.

Can a process inside of a device control group create a child group that has access to a subset of it's devices? The actually checks don't need to be hierarchical but the presence of device nodes should be.

---

I see a couple of holes in the device control picture.

- How do we handle hotplug events?

I think we can do this by relaying events trough userspace, upating the device control groups etc.

- Unprivileged processess interacting with all of this. (possibly with privilege in their user namespace)
What I don't know how to do is how to create a couple of different subhierarchies each for different child processes.

- Dynamically created devices.

My gut feel is that we should replicate the success of devpts and give each type of dynamically created device it's own filesystem and mount point under /dev, and just bend the handful of userspace users into that model.

- Sysfs

My gut says for the container use case we should aim to simply not have dynamically created devices in sysfs and then we can simply not care.

- Migration

Either we need block device numbers that can migrate with us, (possibly a subset of the entire range ala devpts) or we need to send hotplug events to userspace right after a migration so userspace processes that care can invalidate their caches of stat data.

---

With the code in my userns development tree I can create a user namespace, create a new mount namespace, and then if I have access to any block devices mount filesystems, all without needing to have any special privileges. What I haven't figured out is what it would take to get the the device control group into the middle that.

It feels like it should be possible to get the checks straight and use the device control group hooks to control which devices are usable in a user namespace. Unfortunately when I try and work it out the independence of the user namespace and the device control group seem to make that impossible.

Shrug there is most definitely something missing from our model on how to handle devices well. I am hoping we can sprinkling some devpts derived pixie dust at the problem migrate userspace to some new interfaces and have life be good.

Eric

Quoted 28 lines · Eric W. Biederman

Quoted 12 lines · Serge E. Hallyn

Quoted 8 lines · Aristeu Rozanski

Tejun,

Quoted 4 lines · Tejun Heo

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

if Serge is not planning to do it already, I can take a look in device_cgroup.

That's fine with me, thanks.

Quoted 7 lines

Quoted 3 lines

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

Before going down this road, I'd like to discuss this with at least you, me, and Eric Biederman (cc:d) as to how it relates to a device namespace.

The problem with devices.

- An unrestricted mknod gives you access to effectively any device in the system.

Quoted 8 lines

- During process migration if the device number changes using stat to file descriptors can fail on the same file descriptor.

- Devices coming from prexisting filesystems that we mount as unprivileged users are as dangerous as mknod but show that the problem is not limited to mknod.

- udev thinks mknod is a system call we can remove from the kernel.

Also,

- udevadm trigger --action=add

causes all the devices known on the host to be re-sent to everyone (all namespaces). Which floods everyone and causes the host to reset some devices.

Quoted 9 lines

---

The use cases seem comparitively simple to enumerate.

- Giving unfiltered access to a device to someone not root.

- Virtual devices that everyone uses and have no real privilege
requirements: /dev/null /dev/tty /dev/zero etc.

Quoted 2 lines

- Dynamically created devices /dev/loopN /dev/tun /dev/macvtapN, nbd, iscsi, /dev/ptsN, etc

and

- per-namespace uevent filtering.

Quoted 11 lines

---

There are a couple of solution to these problems.

- The classic solution of creating a /dev for a container before starting it.

- The devpts filesystem. This works well for unprivileged access to ptys. Except for the /dev/ptmx sillines I very like how things are handled today with devpts.

Quoted 8 lines

- Device control groups. I am not quite certain what to make of them. The only case I see where they are better than a prebuilt static dev is if there is a hotppluged device that I want to push into my container.

I think the only problem with device control groups and hierarchies is that removing a device from a whitelist does not recurse down the hierarchy.

That's going to be fixed soon thanks to Aristeu :)

Quoted 4 lines

Can a process inside of a device control group create a child group that has access to a subset of it's devices? The actually checks don't need to be hierarchical but the presence of device nodes should be.

If I understand your question right, yes.

Quoted 14 lines

---

I see a couple of holes in the device control picture.

- How do we handle hotplug events?

I think we can do this by relaying events trough userspace, upating the device control groups etc.

- Unprivileged processess interacting with all of this. (possibly with privilege in their user namespace)
What I don't know how to do is how to create a couple of different subhierarchies each for different child processes.

Quoted 6 lines

- Dynamically created devices.

My gut feel is that we should replicate the success of devpts and give each type of dynamically created device it's own filesystem and mount point under /dev, and just bend the handful of userspace users into that model.

Phew. Maybe. Had not considered that. But seems daunting.

Quoted 5 lines

- Sysfs

My gut says for the container use case we should aim to simply not have dynamically created devices in sysfs and then we can simply not care.

Quoted 7 lines

- Migration

Either we need block device numbers that can migrate with us, (possibly a subset of the entire range ala devpts) or we need to send hotplug events to userspace right after a migration so userspace processes that care can invalidate their caches of stat data.

Quoted 8 lines

---

With the code in my userns development tree I can create a user namespace, create a new mount namespace, and then if I have access to any block devices mount filesystems, all without needing to have any special privileges. What I haven't figured out is what it would take to get the the device control group into the middle that.

I'm really not sure that's a question we want to ask. The device control group, like the ns cgroup, was meant as a temporary workaround to not having user and device namespaces.

If we can come up with a device cgroup model that works to fill all the requirements we would have for a devices ns, then great. But I don't want us to be constrained by that.

Quoted 12 lines

It feels like it should be possible to get the checks straight and use the device control group hooks to control which devices are usable in a user namespace. Unfortunately when I try and work it out the independence of the user namespace and the device control group seem to make that impossible.

Shrug there is most definitely something missing from our model on how to handle devices well. I am hoping we can sprinkling some devpts derived pixie dust at the problem migrate userspace to some new interfaces and have life be good.

Quoted 1 line

Eric

Me too!

I'm torn between suggesting that we have a session at UDS to discuss this, and not wanting to so that we can focus on the remaining questions with the user namespace.

Thinking about this a bit more I think we have been asking the wrong question.

I think the correct question should be: How do we safely allow for unprivileged creation of device nodes and devices?

One piece of the puzzle is that we should be able to allow unprivileged device node creation and access for any device on any filesystem for which it unprivileged access is safe.

Something like the current device control group hooks but with the whitelist implemented like:

static bool unpriv_mknod_ok(struct device *dev)

{
	char *tmp, *name;
	umode_t mode = 0;

	name = device_get_devnode(dev, &mode, &tmp);
	if (!name)
        	return false;
	kfree(tmp);
        return mode == 0666;
}

Are there current use cases where people actually want arbitrary access to hardware devices? I really want to say no and get udev and sysfs out of the picture as much as possible.

Quoted 40 lines · Serge E. Hallyn

Quoted 28 lines · Eric W. Biederman

Quoted 12 lines · Serge E. Hallyn

Quoted 8 lines · Aristeu Rozanski

Tejun,

Quoted 4 lines · Tejun Heo

memcg can be handled by memcg people and I can handle cgroup_freezer and others with help from the authors. The problematic one is blkio. If anyone is interested in working on blkio, please be my guest. Vivek? Glauber?

if Serge is not planning to do it already, I can take a look in device_cgroup.

That's fine with me, thanks.

Quoted 7 lines

Quoted 3 lines

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

Before going down this road, I'd like to discuss this with at least you, me, and Eric Biederman (cc:d) as to how it relates to a device namespace.

The problem with devices.

- An unrestricted mknod gives you access to effectively any device in the system.

Quoted 8 lines

- During process migration if the device number changes using stat to file descriptors can fail on the same file descriptor.

- Devices coming from prexisting filesystems that we mount as unprivileged users are as dangerous as mknod but show that the problem is not limited to mknod.

- udev thinks mknod is a system call we can remove from the kernel.

Also,

Quoted 5 lines

- udevadm trigger --action=add

causes all the devices known on the host to be re-sent to everyone (all namespaces). Which floods everyone and causes the host to reset some devices.

I think this is all userspace activity,

and should be largely
fixed by not begin root in a container.

Quoted 10 lines

Quoted 9 lines

The use cases seem comparitively simple to enumerate.

- Giving unfiltered access to a device to someone not root.

- Virtual devices that everyone uses and have no real privilege
requirements: /dev/null /dev/tty /dev/zero etc.

- Dynamically created devices /dev/loopN /dev/tun /dev/macvtapN, nbd, iscsi, /dev/ptsN, etc

Quoted 3 lines

and

- per-namespace uevent filtering.

One possible solution there is to just send the kernel uevents (except for the network ones) into the initial network namespace.

Quoted 25 lines

Quoted 9 lines

There are a couple of solution to these problems.

- The classic solution of creating a /dev for a container before starting it.

- The devpts filesystem. This works well for unprivileged access to ptys. Except for the /dev/ptmx sillines I very like how things are handled today with devpts.

Quoted 8 lines

- Device control groups. I am not quite certain what to make of them. The only case I see where they are better than a prebuilt static dev is if there is a hotppluged device that I want to push into my container.

I think the only problem with device control groups and hierarchies is that removing a device from a whitelist does not recurse down the hierarchy.

That's going to be fixed soon thanks to Aristeu :)

Quoted 4 lines

Can a process inside of a device control group create a child group that has access to a subset of it's devices? The actually checks don't need to be hierarchical but the presence of device nodes should be.

Quoted 1 line

If I understand your question right, yes.

I should also have asked can we do this without any capabilities and without our uid being 0?

Quoted 19 lines

Quoted 18 lines

I see a couple of holes in the device control picture.

- How do we handle hotplug events?

I think we can do this by relaying events trough userspace, upating the device control groups etc.

- Unprivileged processess interacting with all of this. (possibly with privilege in their user namespace)
What I don't know how to do is how to create a couple of different subhierarchies each for different child processes.

- Dynamically created devices.

My gut feel is that we should replicate the success of devpts and give each type of dynamically created device it's own filesystem and mount point under /dev, and just bend the handful of userspace users into that model.

Quoted 1 line

Phew. Maybe. Had not considered that. But seems daunting.

I think the list of device types that we care about here is pretty small. Please correct me if I am wrong.

loop nbd iscsi macvtap

And if we want it to be safe to use these devices in a user namespace without global root privileges we need to go through the code anyway.

So I think it is the gradual safe and sane approach assume we don't run into something like the devpts /dev/ptmx silliness that stalled devpts.

Quoted 5 lines

Quoted 5 lines

- Sysfs

My gut says for the container use case we should aim to simply not have dynamically created devices in sysfs and then we can simply not care.

I guess what I keep thinking for sysfs is that it should be for real hardware backed devices. If we can get away with that like we do with ptys it just makes everyone's life simpler.

Primarily sysfs and uevents are for allowing the system to take automatic action when a new device is created. Do we have an actual need for hotplug support in containers?

One piece of the puzzle is that we should be able to allow unprivileged device node creation and access for any device on any filesystem for which it unprivileged access is safe.

Quoted 3 lines

One piece of the puzzle is that we should be able to allow unprivileged device node creation and access for any device on any filesystem for which it unprivileged access is safe.

Which devices are "safe" is policy for all interesting and useful cases, as are file permissions, security tags, chroot considerations and the like.

It's a complete non starter.

Quoted 9 lines · Alan Cox

Quoted 3 lines

One piece of the puzzle is that we should be able to allow unprivileged device node creation and access for any device on any filesystem for which it unprivileged access is safe.

Which devices are "safe" is policy for all interesting and useful cases, as are file permissions, security tags, chroot considerations and the like.

It's a complete non starter.

Quoted 10 lines · ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected] (Eric W. Biederman) writes:

Quoted 8 lines · Alan Cox

Quoted 3 lines

One piece of the puzzle is that we should be able to allow unprivileged device node creation and access for any device on any filesystem for which it unprivileged access is safe.

Which devices are "safe" is policy for all interesting and useful cases, as are file permissions, security tags, chroot considerations and the like.

Quoted 1 line

Quoted 1 line

It's a complete non starter.

Come to think of it mknod is completely unnecessary.

Without mknod. Without being able to mount filesystems containing device nodes.

The mount namespace is sufficient to prevent all of the cases that the device control group prevents (open and mknod on device nodes).

So I honestly think the device control group is superflous, and it is probably wise to deprecate it and move to a model where it does not exist.

Eric

Quoted 14 lines · Eric W. Biederman

Quoted 11 lines · ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected] (Eric W. Biederman) writes:

Quoted 9 lines · Alan Cox

Quoted 3 lines

One piece of the puzzle is that we should be able to allow unprivileged device node creation and access for any device on any filesystem for which it unprivileged access is safe.

Which devices are "safe" is policy for all interesting and useful cases, as are file permissions, security tags, chroot considerations and the like.

It's a complete non starter.

Quoted 4 lines

Come to think of it mknod is completely unnecessary.

Without mknod. Without being able to mount filesystems containing device nodes.

Hm? That sounds like it will really upset init/udev/upgrades in the container.

Are you saying all filesystems containing device nodes will need to be mounted in advance by the process setting up the container?

Quoted 8 lines

The mount namespace is sufficient to prevent all of the cases that the device control group prevents (open and mknod on device nodes).

So I honestly think the device control group is superflous, and it is probably wise to deprecate it and move to a model where it does not exist.

Quoted 2 lines

Eric

That's what I said a few emails ago :) The device cgroup was meant as a short-term workaround for lack of user (and device) namespaces.

At least with a recent modern distro I can't imagine this to be an issue. I expect we could have a kernel build option that removed the mknod system call and a modern distro wouldn't notice.

For migration with direct access to real hardware devices we must treat it as hardware hotunplug. There is nothing else we can do.

Quoted 3 lines

At least with a recent modern distro I can't imagine this to be an issue. I expect we could have a kernel build option that removed the mknod system call and a modern distro wouldn't notice.

A few things beyond named pipes will break. PCMCIA I believe still depends on ugly mknod hackery of its own. You also need it for some classes of non detectable device.

Basically though you could.

Quoted 2 lines

For migration with direct access to real hardware devices we must treat it as hardware hotunplug. There is nothing else we can do.

That is demonstrably false for a shared bus or a network linked device. Consider a firewire camera wired to two systems at once. Consider SAN storage.

Quoted 15 lines · Serge Hallyn

Quoted 14 lines · Eric W. Biederman

Quoted 11 lines · ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected] (Eric W. Biederman) writes:

Quoted 9 lines · Alan Cox

Quoted 3 lines

One piece of the puzzle is that we should be able to allow unprivileged device node creation and access for any device on any filesystem for which it unprivileged access is safe.

Which devices are "safe" is policy for all interesting and useful cases, as are file permissions, security tags, chroot considerations and the like.

It's a complete non starter.

Quoted 7 lines

Quoted 4 lines

Come to think of it mknod is completely unnecessary.

Without mknod. Without being able to mount filesystems containing device nodes.

Hm? That sounds like it will really upset init/udev/upgrades in the container.

udev does not create device nodes. For an older udev the worst I can see it doing is having mknod failing with EEXIST because the device node already exists.

We should be able to make it look to init like a ramdisk mounted the filesystems.

Why should upgrades care? Package installation shouldn't be calling mknod.

At least with a recent modern distro I can't imagine this to be an issue. I expect we could have a kernel build option that removed the mknod system call and a modern distro wouldn't notice.

Quoted 2 lines

Are you saying all filesystems containing device nodes will need to be mounted in advance by the process setting up the container?

As a general rule.

I think in practice there is wiggle room for special cases like mounting a fresh devpts. devpts at least in always create a new instance on mount mode seems safe, as it can not give you access to any existing devices.

You can also do a lot of what would normally be done with mknod with bind mounts to the original devices location.

Quoted 8 lines

Quoted 8 lines

The mount namespace is sufficient to prevent all of the cases that the device control group prevents (open and mknod on device nodes).

So I honestly think the device control group is superflous, and it is probably wise to deprecate it and move to a model where it does not exist.

Quoted 5 lines

Quoted 2 lines

Eric

That's what I said a few emails ago :) The device cgroup was meant as a short-term workaround for lack of user (and device) namespaces.

I am saying something stronger. The device cgroup doesn't seem to have a practical function now.

That for the general case we don't need any
kernel support. That all of this should be a matter of some user space glue code, and just the tiniest bit of sorting out how hotplug events are sent.

The only thing I can think we would need a device namespace for is for migration.

For migration with direct access to real hardware devices we must treat it as hardware hotunplug. There is nothing else we can do.

If there is any other case where we need to preserve device numbers etc we have the example of devpts.

So at this point I really don't think we need a device namespace or a device control group. (Just emulate devtmpfs, sysfs and uevents).

Eric

Quoted 5 lines

Quoted 2 lines

That's what I said a few emails ago :) The device cgroup was meant as a short-term workaround for lack of user (and device) namespaces.

I am saying something stronger. The device cgroup doesn't seem to have a practical function now.

"Now" is wrong. The user namespace is not complete and not yet usable for a full system container. We still need the device control group.

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

Quoted 3 lines · Aristeu Rozanski

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

That sounds like a really bad idea to me. We've spent ages training users that the actual sd<x> name of their device doesn't matter and they should use UUIDs or WWNs instead ... why should they now care inside containers?

Quoted 3 lines · Aristeu Rozanski

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

That sounds like a really bad idea to me. We've spent ages training users that the actual sd<x> name of their device doesn't matter and they should use UUIDs or WWNs instead ... why should they now care inside containers?

Quoted 3 lines · Aristeu Rozanski

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

That sounds like a really bad idea to me. We've spent ages training users that the actual sd<x> name of their device doesn't matter and they should use UUIDs or WWNs instead ... why should they now care inside containers?

also, heard about the desire of having a device namespace instead with support for translation ("sda" -> "sdf"). If anyone see immediate use for this please let me know.

* Sort & unique when listing tasks. Even the documentation says it doesn't happen but we have a good hunk of code doing it in cgroup.c. I'm gonna rip it out at some point. Again, if you don't like it, scream.

Quoted 5 lines

* Sort & unique when listing tasks. Even the documentation says it doesn't happen but we have a good hunk of code doing it in cgroup.c. I'm gonna rip it out at some point. Again, if you don't like it, scream.

I think some userspace tools do assume the uniq bit. So if we can preserve that, great!

6. Multiple hierarchies

Apart from the apparent wheeeeeeeeness of it (I think I talked about that enough the last time[1]), there's a basic problem when more than one controllers interact - it's impossible to define a resource group when more than two controllers are involved because the intersection of different controllers is only defined in terms of tasks.

IOW, if an entity X is of interest to two controllers, there's no way to map X to the cgroups of the two controllers. X may belong to A and B when viewed by one task but A' and B when viewed by another. This already is a head scratcher in writeback where blkcg and memcg have to interact.

While I am pushing for unified hierarchy, I think it's necessary to have different levels of granularities depending on controllers given that nesting involves significant overhead and noticeable controller-dependent behavior changes.

...

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

[grr. why does gmane scramble addresses?]

Quoted 2 lines

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

OK, I'll bite.

I have a server that has a whole bunch of cores. A small fraction of those cores are general purpose and run whatever they like. The rest are tightly controlled.

For simplicity, we have two cpusets that we use. The root allows all cpus. The other one only allows the general purpose cpus. We shove everything into the general-purpose-only cpuset, and then we move special stuff back to root. (We also shove some kernel threads into a non-root cpuset using the 'cset' tool.)

Enter systemd, which wants a hierarchy corresponding to services. If we were to use it, we might end up violating its hierarchy.

Alternatively, if we started using memcg, then we might have some tasks to have more restrictive memory usage but less restrictive cpu usage.

As long as we can still pull this off, I'm happy.

Hello,

Quoted 1 line · Andy Lutomirski

[grr. why does gmane scramble addresses?]

You can append /raw to the message url and see the raw mssage.

http://article.gmane.org/gmane.linux.kernel.containers/23802/raw

Quoted 10 lines

Quoted 3 lines

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

OK, I'll bite.

I have a server that has a whole bunch of cores. A small fraction of those cores are general purpose and run whatever they like. The rest are tightly controlled.

Quoted 5 lines

For simplicity, we have two cpusets that we use. The root allows all cpus. The other one only allows the general purpose cpus. We shove everything into the general-purpose-only cpuset, and then we move special stuff back to root. (We also shove some kernel threads into a non-root cpuset using the 'cset' tool.)

Using root for special stuff probably isn't a good idea and moving bound kthreads into !root cgroups is already disallowed.

Quoted 7 lines

Enter systemd, which wants a hierarchy corresponding to services. If we were to use it, we might end up violating its hierarchy.

Alternatively, if we started using memcg, then we might have some tasks to have more restrictive memory usage but less restrictive cpu usage.

Quoted 1 line

As long as we can still pull this off, I'm happy.

IIUC, you basically want just two groups w/ cpuset and use it for loose cpu ioslation for high priority jobs. Structure-wise, I don't think it's gonna be a problem although using root for special stuff would need to change.

Hello,

Quoted 1 line · Andy Lutomirski

[grr. why does gmane scramble addresses?]

You can append /raw to the message url and see the raw mssage.

http://article.gmane.org/gmane.linux.kernel.containers/23802/raw

Quoted 10 lines

Quoted 3 lines

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

OK, I'll bite.

I have a server that has a whole bunch of cores. A small fraction of those cores are general purpose and run whatever they like. The rest are tightly controlled.

Quoted 5 lines

For simplicity, we have two cpusets that we use. The root allows all cpus. The other one only allows the general purpose cpus. We shove everything into the general-purpose-only cpuset, and then we move special stuff back to root. (We also shove some kernel threads into a non-root cpuset using the 'cset' tool.)

Using root for special stuff probably isn't a good idea and moving bound kthreads into !root cgroups is already disallowed.

Quoted 7 lines

Enter systemd, which wants a hierarchy corresponding to services. If we were to use it, we might end up violating its hierarchy.

Alternatively, if we started using memcg, then we might have some tasks to have more restrictive memory usage but less restrictive cpu usage.

Quoted 1 line

As long as we can still pull this off, I'm happy.

IIUC, you basically want just two groups w/ cpuset and use it for loose cpu ioslation for high priority jobs. Structure-wise, I don't think it's gonna be a problem although using root for special stuff would need to change.

[grr. why does gmane scramble addresses?]

Quoted 2 lines

I think this level of flexibility should be enough for most use cases. If someone disagrees, please voice your objections now.

OK, I'll bite.

I have a server that has a whole bunch of cores. A small fraction of those cores are general purpose and run whatever they like. The rest are tightly controlled.

For simplicity, we have two cpusets that we use. The root allows all cpus. The other one only allows the general purpose cpus. We shove everything into the general-purpose-only cpuset, and then we move special stuff back to root. (We also shove some kernel threads into a non-root cpuset using the 'cset' tool.)

Enter systemd, which wants a hierarchy corresponding to services. If we were to use it, we might end up violating its hierarchy.

Alternatively, if we started using memcg, then we might have some tasks to have more restrictive memory usage but less restrictive cpu usage.

As long as we can still pull this off, I'm happy.

7. Misc issues

* Sort & unique when listing tasks. Even the documentation says it doesn't happen but we have a good hunk of code doing it in cgroup.c. I'm gonna rip it out at some point. Again, if you don't like it, scream.

* At the PLC, pjt told me that assinging threads of a cgroup to different cgroups is useful for some use cases but if we're to have a unified hierarchy, I don't think we can continue to do that. Paul, can you please elaborate the use case?

* Vivek brought up the issue of distributing resources to tasks and groups in the same cgroup. I don't know. Need to think more about it.