Skip to contentSkip to conversation

Have sane default values for cpusets

PATCH/RFC Dhaval Giani 4 participants 16 messages May 12, 2010 – May 13, 2010

lore
Share & tools
lore
#0 lore raw

Hi folks,

This is a patch (against a somewhat older kernel) which proposes to set a default value for a cpuset cgroup that is created. At this point in time, this is just half done since I would prefer some comments, and see if it is acceptable, and how.

First the description of the patch.

This patch basically sets up default values for the a cpuset that is created. By default right now, cpuset.cpus_allowed and cpuset.mems_allowed is empty. This does not allow a task to be attached to the cpuset. This patch sets the default value of the cpus_allowed and mems_allowed as the same as that of the parent.

TODO:
1. Set the value depending on the exclusive flags set in other cpusets.

This does not break ABI since applications which were explicitly setting up the cpusets will still be setting them up anyway. And if someone was checking if a cpuset was setup or not by checking the state of cpuset.cpus_allowed, then it was broken and should be fixed.

Now the motivation.

Looking from an application programmer's point of view, when using cgroups, he does not want to care about unrelated subsystem and would only manipulate the subsystem which he is concerned with. But this is a decision that is not just limited to the application programmer. It is a decision that is very strongly dependent on the underlying system as well. Cgroups allows multiple subsystems to be mounted together, which then implies they have a common hierarchy.

Now to take an example, consider a system where cpu and memory are mounted together, since the user wants to have the same hierarchy for both cpu and memory. Since the application cares only about memory, it manipulates all those values. But since they are mounted together, every time it creates a cgroup for a task, that task will also be moved to the corresponding cpu cgroup. The solution to this is (and the one we recommend is) to mount all cgroups separately, but this is not always going to happen, because it is quite painful to do this. If you use libcgroup, you need to add additional parameters to your configuration file. If you mount it manually, you have to specify multiple mount commands.

Anyway, coming back to the original issue. Consider that the usecase that the user has is a valid use case, and just mix in cpuset into this case. Now, if the application creates a cgroup, for memory, but not knowing that the user has mounted cpusets together, it is unable to attach a task to its newly created cgroup because cpusets is not setup. Now the programmer is forced to know about cpusets as well.

In order to handle this situation, libcgroup has an API which takes the parameters from the parent cgroup. But that is also broken. Consider this same example. If there is a cgroup, that has its cpu.rt_runtime_us parameter setup in the another child, then the create from parent API will fail since we tried to assign too much rt bandwidth to that cgroup. So you can neither create a cgroup nor can you assign parameters from its parents.

Now rt-cgroups handles this situation quite well. Since real-time is obviously a special case, the default is to have no rt bandwidth for that cgroup. Where cpusets goes wrong is to have a *no* default values. So the question now is, do we expect to have this non uniform policy in implementing subsystems, or do we enforce a policy to have sane defaults for subsystems if they prevent attaching "regular" tasks by default.

Solving it in userspace is just adding another layer, and asking either libcgroup to have a lot of code for just one subsystem, or expecting the programmer to know about every subsystem, just in order to handle every corner case.

Comments?

Thanks!
Dhaval

---
 kernel/cpuset.c |   13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/cpuset.c +13 -0
Index: linux-2.6/kernel/cpuset.c
===================================================================
@@ -1824,6 +1824,17 @@ static void cpuset_post_clone(struct cgr
 }
 
 /*
+ * Inherit the parent's cpus/mems values. Do not inhert the
+ * exclusivity flag
+ *
+ */
+static void cpuset_inherit_parent_values(struct cpuset *child)
+{
+	cpumask_copy(child->cpus_allowed, child->parent->cpus_allowed);
+	child->mems_allowed = child->parent->mems_allowed;
+}
+
+/*
  *	cpuset_create - create a cpuset
  *	ss:	cpuset cgroup subsystem
  *	cont:	control group that the new cpuset will be part of
@@ -1860,6 +1871,8 @@ static struct cgroup_subsys_state *cpuse
 	cs->relax_domain_level = -1;
 
 	cs->parent = parent;
+	cpuset_inherit_parent_values(cs);
+
 	number_of_cpusets++;
 	return &cs->css ;
 }
#1 lore raw
Quoted 1 line · Dhaval Giani

Where cpusets goes wrong is to have a *no* default values.

It has a default, empty is still a valid value.

#2 lore raw
Quoted 5 lines · Peter Zijlstra
Quoted 1 line · Dhaval Giani

Where cpusets goes wrong is to have a *no* default values.

It has a default, empty is still a valid value.

Well, it is still not sane. And in the part you snipped, I did mention,

Quoted 2 lines
Quoted 2 lines

do we enforce a policy to have sane defaults
for subsystems if they prevent attaching "regular" tasks by default.

And to add to it, a sane default can be defined as one, where a task can be attached to a cgroup on creation without changing any other parameter.

Dhaval

#3 lore raw
In reply to Dhaval Giani

What about the case where some subset of the parent's mems/cpus are given to a child with the exclusive flag set?

Paul

Quoted 116 lines · Dhaval Giani

Hi folks,

This is a patch (against a somewhat older kernel) which proposes to set a default value for a cpuset cgroup that is created. At this point in time, this is just half done since I would prefer some comments, and see if it is acceptable, and how.

First the description of the patch.

This patch basically sets up default values for the a cpuset that is created. By default right now, cpuset.cpus_allowed and cpuset.mems_allowed is empty. This does not allow a task to be attached to the cpuset. This patch sets the default value of the cpus_allowed and mems_allowed as the same as that of the parent.

TODO:
1. Set the value depending on the exclusive flags set in other cpusets.

This does not break ABI since applications which were explicitly setting up the cpusets will still be setting them up anyway. And if someone was checking if a cpuset was setup or not by checking the state of cpuset.cpus_allowed, then it was broken and should be fixed.

Now the motivation.

Looking from an application programmer's point of view, when using cgroups, he does not want to care about unrelated subsystem and would only manipulate the subsystem which he is concerned with. But this is a decision that is not just limited to the application programmer. It is a decision that is very strongly dependent on the underlying system as well. Cgroups allows multiple subsystems to be mounted together, which then implies they have a common hierarchy.

Now to take an example, consider a system where cpu and memory are mounted together, since the user wants to have the same hierarchy for both cpu and memory. Since the application cares only about memory, it manipulates all those values. But since they are mounted together, every time it creates a cgroup for a task, that task will also be moved to the corresponding cpu cgroup. The solution to this is (and the one we recommend is) to mount all cgroups separately, but this is not always going to happen, because it is quite painful to do this. If you use libcgroup, you need to add additional parameters to your configuration file. If you mount it manually, you have to specify multiple mount commands.

Anyway, coming back to the original issue. Consider that the usecase that the user has is a valid use case, and just mix in cpuset into this case. Now, if the application creates a cgroup, for memory, but not knowing that the user has mounted cpusets together, it is unable to attach a task to its newly created cgroup because cpusets is not setup. Now the programmer is forced to know about cpusets as well.

In order to handle this situation, libcgroup has an API which takes the parameters from the parent cgroup. But that is also broken. Consider this same example. If there is a cgroup, that has its cpu.rt_runtime_us parameter setup in the another child, then the create from parent API will fail since we tried to assign too much rt bandwidth to that cgroup. So you can neither create a cgroup nor can you assign parameters from its parents.

Now rt-cgroups handles this situation quite well. Since real-time is obviously a special case, the default is to have no rt bandwidth for that cgroup. Where cpusets goes wrong is to have a *no* default values. So the question now is, do we expect to have this non uniform policy in implementing subsystems, or do we enforce a policy to have sane defaults for subsystems if they prevent attaching "regular" tasks by default.

Solving it in userspace is just adding another layer, and asking either libcgroup to have a lot of code for just one subsystem, or expecting the programmer to know about every subsystem, just in order to handle every corner case.

Comments?

Thanks!
Dhaval

---
 kernel/cpuset.c |   13 +++++++++++++

 1 file changed, 13 insertions(+)

Index: linux-2.6/kernel/cpuset.c
===================================================================

kernel/cpuset.c +13 -0
@@ -1824,6 +1824,17 @@ static void cpuset_post_clone(struct cgr
  }
 
  /*
+ * Inherit the parent's cpus/mems values. Do not inhert the
+ * exclusivity flag
+ *
+ */
+static void cpuset_inherit_parent_values(struct cpuset *child)
+{
+       cpumask_copy(child->cpus_allowed, child->parent->cpus_allowed);
+       child->mems_allowed = child->parent->mems_allowed;
+}
+
+/*
  *     cpuset_create - create a cpuset
  *     ss:     cpuset cgroup subsystem
  *     cont:   control group that the new cpuset will be part of
@@ -1860,6 +1871,8 @@ static struct cgroup_subsys_state *cpuse
        cs->relax_domain_level = -1;
 
        cs->parent = parent;
+       cpuset_inherit_parent_values(cs);
+
        number_of_cpusets++;
        return &cs->css ;
  }
#4 lore raw
Quoted 3 lines · Paul Menage

What about the case where some subset of the parent's mems/cpus are given to a child with the exclusive flag set?

As I mentioned in the TODO, it is still to be handled. But it should simply exclude those mems/cpus which are exclusive. It was a bit more involved than the effort I wanted to put in before gauging the reactions.

Thanks!
Dhaval

#5 lore raw
Quoted 6 lines · Dhaval Giani
Quoted 3 lines · Paul Menage

What about the case where some subset of the parent's mems/cpus are given to a child with the exclusive flag set?

As I mentioned in the TODO, it is still to be handled.

Oops, sorry, just read the patch :-)

Quoted 4 lines

But it should
simply exclude those mems/cpus which are exclusive. It was a bit more involved than the effort I wanted to put in before gauging the reactions.

I think the idea is reasonable - the only way that I could see it breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

The attempts to set the mem_exclusive flags would fail, since A and B would both have all of the parent's mems.

Paul

#6 lore raw
Quoted 25 lines · Paul Menage
Quoted 6 lines · Dhaval Giani
Quoted 3 lines · Paul Menage

What about the case where some subset of the parent's mems/cpus are given to a child with the exclusive flag set?

As I mentioned in the TODO, it is still to be handled.

Oops, sorry, just read the patch :-)

Quoted 4 lines

But it should
simply exclude those mems/cpus which are exclusive. It was a bit more involved than the effort I wanted to put in before gauging the reactions.

I think the idea is reasonable - the only way that I could see it breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

Quoted 3 lines

The attempts to set the mem_exclusive flags would fail, since A and B would both have all of the parent's mems.

But would this not fail otherwise?

Dhaval

#7 lore raw
Quoted 10 lines · Dhaval Giani
Quoted 10 lines

I think the idea is reasonable - the only way that I could see it breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

Quoted 6 lines
Quoted 3 lines

The attempts to set the mem_exclusive flags would fail, since A and B would both have all of the parent's mems.

But would this not fail otherwise?

Assuming that mems_for_a and mems_for_b were disjoint, it would be fine currently.

Paul

#8 lore raw
Quoted 15 lines · * [email protected] <[email protected]> [2010-05-12 12:36:57]:
Quoted 14 lines · Dhaval Giani
Quoted 13 lines

I think the idea is reasonable - the only way that I could see it breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

The attempts to set the mem_exclusive flags would fail, since A and B would both have all of the parent's mems.

Quoted 6 lines
Quoted 2 lines

But would this not fail otherwise?

Assuming that mems_for_a and mems_for_b were disjoint, it would be fine currently.

Yep, that does seem like breakage.

Signature · 2 lines
	Three Cheers,
	Balbir
#9 lore raw
In reply to Paul Menage
Quoted 15 lines · Paul Menage
Quoted 14 lines · Dhaval Giani
Quoted 13 lines

I think the idea is reasonable - the only way that I could see it breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

The attempts to set the mem_exclusive flags would fail, since A and B would both have all of the parent's mems.

Quoted 6 lines
Quoted 2 lines

But would this not fail otherwise?

Assuming that mems_for_a and mems_for_b were disjoint, it would be fine currently.

Ah my bad. I misread mems_for_a as taking the value from the parent. You are right, that was a case I missed.

Hmm, so how do we fix this? Any solutions? Not fixing the kernel pushes the problem to the userspace, making it hard for tons of more applications to use cgroups without jumping through a lot of hoops.

Dhaval

#10 lore raw
Quoted 23 lines · Dhaval Giani
Quoted 15 lines · Paul Menage
Quoted 14 lines · Dhaval Giani
Quoted 13 lines

I think the idea is reasonable - the only way that I could see it breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

The attempts to set the mem_exclusive flags would fail, since A and B would both have all of the parent's mems.

Quoted 6 lines
Quoted 2 lines

But would this not fail otherwise?

Assuming that mems_for_a and mems_for_b were disjoint, it would be fine currently.

Quoted 7 lines

Ah my bad. I misread mems_for_a as taking the value from the parent. You are right, that was a case I missed.

Hmm, so how do we fix this? Any solutions? Not fixing the kernel pushes the problem to the userspace, making it hard for tons of more applications to use cgroups without jumping through a lot of hoops.

OK, how about this. Introduce a new option, nodefaults (or some such name) which would retain the existing behaviour while the default mount options would moutn cpuset with the defaults. Also, make

mount -t cpuset cpuset /cpuset

equivalent to

mount -t cgroup -onoprefix,nodefaults,cpuset cpuset /cpuset

Would that work?

Thanks,
Dhaval

#11 lore raw
Quoted 36 lines · Dhaval Giani
Quoted 23 lines · Dhaval Giani
Quoted 15 lines · Paul Menage
Quoted 14 lines · Dhaval Giani
Quoted 13 lines

I think the idea is reasonable - the only way that I could see it breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

The attempts to set the mem_exclusive flags would fail, since A and B would both have all of the parent's mems.

Quoted 6 lines
Quoted 2 lines

But would this not fail otherwise?

Assuming that mems_for_a and mems_for_b were disjoint, it would be fine currently.

Quoted 7 lines

Ah my bad. I misread mems_for_a as taking the value from the parent. You are right, that was a case I missed.

Hmm, so how do we fix this? Any solutions? Not fixing the kernel pushes the problem to the userspace, making it hard for tons of more applications to use cgroups without jumping through a lot of hoops.

OK, how about this. Introduce a new option, nodefaults (or some such name) which would retain the existing behaviour while the default mount options would moutn cpuset with the defaults. Also, make

Quoted 6 lines

mount -t cpuset cpuset /cpuset

equivalent to

mount -t cgroup -onoprefix,nodefaults,cpuset cpuset /cpuset

Does something like cpusetinherits make more sense.

Signature · 2 lines
	Three Cheers,
	Balbir
#12 lore raw
In reply to Dhaval Giani
Quoted 23 lines · Dhaval Giani
Quoted 15 lines · Paul Menage
Quoted 14 lines · Dhaval Giani
Quoted 13 lines

I think the idea is reasonable - the only way that I could see it breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

The attempts to set the mem_exclusive flags would fail, since A and B would both have all of the parent's mems.

Quoted 6 lines
Quoted 2 lines

But would this not fail otherwise?

Assuming that mems_for_a and mems_for_b were disjoint, it would be fine currently.

Quoted 7 lines

Ah my bad. I misread mems_for_a as taking the value from the parent. You are right, that was a case I missed.

Hmm, so how do we fix this? Any solutions? Not fixing the kernel pushes the problem to the userspace, making it hard for tons of more applications to use cgroups without jumping through a lot of hoops.

Well, it's not clear to me whether the case I outlined is actually one that would bite people - it's likely a rare case.

Balbir's point that some apps might get upset by finding non-empty mems/cpus in a newly-created cgroup is more reasonable.

How about a per-cgroup cpuset.inherit_defaults file that defaults to false and is inherited from the parent. If the parent's file is set to true, then the mems/cpus are also inherited?

Then the sysadmin who's giving out user-controllable cpuset-based cgroups can just set it to true and the users don't need to worry about setting up the defaults.

Paul

#13 lore raw
Quoted 38 lines · Paul Menage
Quoted 23 lines · Dhaval Giani
Quoted 15 lines · Paul Menage
Quoted 14 lines · Dhaval Giani
Quoted 13 lines

I think the idea is reasonable - the only way that I could see it breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

The attempts to set the mem_exclusive flags would fail, since A and B would both have all of the parent's mems.

Quoted 6 lines
Quoted 2 lines

But would this not fail otherwise?

Assuming that mems_for_a and mems_for_b were disjoint, it would be fine currently.

Quoted 7 lines

Ah my bad. I misread mems_for_a as taking the value from the parent. You are right, that was a case I missed.

Hmm, so how do we fix this? Any solutions? Not fixing the kernel pushes the problem to the userspace, making it hard for tons of more applications to use cgroups without jumping through a lot of hoops.

Well, it's not clear to me whether the case I outlined is actually one that would bite people - it's likely a rare case.

Balbir's point that some apps might get upset by finding non-empty mems/cpus in a newly-created cgroup is more reasonable.

Quoted 8 lines

How about a per-cgroup cpuset.inherit_defaults file that defaults to false and is inherited from the parent. If the parent's file is set to true, then the mems/cpus are also inherited?

Then the sysadmin who's giving out user-controllable cpuset-based cgroups can just set it to true and the users don't need to worry about setting up the defaults.

Default for root being "true" should work. Anything else, and you still require the programmer to know about cpuset and setting that flag to true.

Dhaval

#14 lore raw
Quoted 4 lines

Default for root being "true" should work. Anything else, and you still require the programmer to know about cpuset and setting that flag to true.

To be honest, I don't see an easy way out of this. I think it is more reasonable to have the change since a lot more users are benefited by it. And in most cases, where they are broken, those applications do require some specialist work, and so it should be reasonable to expect the administrator to set the variable to false at mount time.

Dhaval

#15 lore raw
In reply to Dhaval Giani
Quoted 7 lines · Dhaval Giani

Hi folks,

This does not break ABI since applications which were explicitly setting up the cpusets will still be setting them up anyway. And if someone was checking if a cpuset was setup or not by checking the state of cpuset.cpus_allowed, then it was broken and should be fixed.

The only ABI breakage I would consider is

A user space application reading those values and expecting them to be empty before taking any action

Signature · 2 lines
	Three Cheers,
	Balbir