Skip to contentSkip to conversation

SUBCPUSETS: a resource control functionality using CPUSETS

PATCH In review KUROSAWA Takahiro 12 participants 56 messages Sep 8, 2005 – Oct 4, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover SUBCPUSETS: a resource control functionality using CPUSETS

The following patchset adds resource control functionality to the CPUSETS code by subdividing a cpuset into smaller pieces (SUBCPUSETS). It is made up from two parts, one for the resource control framework and another for the specific resource controller. Resource controllers can be added by using interfaces provided by the SUBCPUSETS.

For now, it enables us to guarantee the CPU time percentage that tasks attached to the subcpuset consume. The memory resource controller code is coming soon.

The patchset is for linux-2.6.13 and consists of 5 patches:
[1/5] Fixes minor problem in the current CPUSETS.
[2/5] Implements resource control framework functionality to the CPUSETS
(SUBCPUSETS).
[3/5] Adds document for SUBCPUSETS.
[4/5] Implements the CPU time resource controller.
[5/5] Puts the CPU time resource controller into SUBCPUSETS.

The following step might be to useful see how this patchset works:

# mount -t cpuset none /dev/cpuset
	(mount the cpuset filesystem)
# /bin/echo 1 > /dev/cpuset/subcpuset_top
	(declare that the cpuset is the toplevel directory of subcpusets;
	you can declare the toplevel directory of subcpusets other than
	the root cpuset of course)
# mkdir /dev/cpuset/sub-a
# mkdir /dev/cpuset/sub-b
	(create subcpuset directories)
# /bin/echo 20 > /dev/cpuset/sub-a/cpu_guar
	(guarantee 20% of the CPU time usage to tasks in the "sub-a" subcpuset)
# /bin/echo 80 > /dev/cpuset/sub-b/cpu_guar
	(guarantee 80% of the CPU time usage to tasks in the "sub-b" subcpuset)
# echo $$ > /dev/cpuset/sub-a/tasks
	(change the cpuset of the shell to the "sub-a" subcpuset)
# (while true; do true; done)&
	(start a CPU consuming process in sub-a subcpuset)
# echo $$ > /dev/cpuset/sub-b/tasks
	(change the cpuset of the shell to the "sub-b" subcpuset)
# (while true; do true; done)&
	(start a CPU consuming process in sub-b subcpuset)
# top
	(see how subcpusets works)

Thanks,

KUROSAWA, Takahiro

#1 lore raw

[ Adding Dinakar to explicit cc list, since I mention his work.
Hopefully he can correct any mispresentations of his work I
might have presented. - pj ]

I've just started reading this - it seems well presented and I think you have put much effort into it. Thank-you for posting it.

I have not yet taken the time to understand it properly, but a couple of questions come to mind offhand. I hope these questions will not be too silly.

1) What is the relation of this patch set to CKRM?

2) Would a structure similar to Dinakar's patches to connect cpusets and dynamic sched domains (posted to linux-mm) work here as well?

Let me describe what I see so far, and hope you will be patient with my confusions. Then perhaps I can better explain my question (2) above.

My initial understanding is that subcpusets provide a way to partial out the proportion of cpu and memory used by various tasks. A leaf node cpuset can partition the tasks attached to it into subsets, called subcpusets, where each subcpuset gets a proportion of cpu and memory available to the original leaf node cpuset.

These original leaf node cpusets (the parents of the 'subcpusets') form a non-overlapping and partial covering of the cpus and memory nodes in a system. It is a _partial_ covering because not every leaf node cpuset need be subdivided into 'subcpusets', nor does every cpu or memory node need to be in such a cpuset.

I'm guessing you do not want such cpusets (the parents of subcpusets) to overlap, because if they did, it would seem to confuse the meaning of getting a fixed proportion of available cpu and memory resources. I was a little surprised not to see any additional checks that cpu_exclusive and mem_exclusive must be set true in these cpusets, to insure non- overlapping cpusets.

Dinakar's patches used cpu_exclusive cpusets to form a complete non-overlapping covering of the cpus in a system (the cpus not in cpu_exclusive cpusets got their own separate covers). Then each such cover defined a different dynamic sched domain (a separately scheduled set of tasks).

Your cover elements (your subcpusets) seem to be a somewhat new kind of cpuset, that can only occur as children of original leaf node cpusets.

Instead of that, I might prefer doing as Dinakar did, and use a single boolean flag per existing cpuset, to mark a subset of the cpusets that form the covering. Dinakar got lucky, and was able to use an existing cpuset boolean flag - cpu_exclusive, which had not been very useful until then. You will have to add a new flag.

On the other hand, Dinakar had more work to do than you might, because he needed a complete covering (so had to round up cpus in non exclusive cpusets to form more covering elements). From what I can tell, you don't need a complete covering - it seems fine if some cpus are not managed by this resource control function.

I hope my above remarks are not to far off the mark and do not misrepresent your work too painfully. And I hope they make at least a little sense. Sometimes I have a strange way of describing my thoughts.

Thank-you again for this good work. I look forward to your further considerations. I will make the time soon to read this patch more closely.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#2 lore raw
Quoted 2 lines · On Thu, 8 Sep 2005 00:23:23 -0700 Paul Jackson <[email protected]> wrote:

I've just started reading this - it seems well presented and I think you have put much effort into it. Thank-you for posting it.

Thank you for reading my patches!

Quoted 5 lines

I have not yet taken the time to understand it properly, but a couple of questions come to mind offhand. I hope these questions will not be too silly.

1) What is the relation of this patch set to CKRM?

My patches consist of two parts, one for subcpuset framework and another for the cpu resource controller. Subcpuset framework code doesn't have the relation to CKRM. CKRM could probably use the cpu resource controller code as a resource controller of it with some modification.

Quoted 3 lines

2) Would a structure similar to Dinakar's patches to connect cpusets and dynamic sched domains (posted to linux-mm) work here as well?

Yes, subcpusets could work with the dynamic sched domains. The cpu resource controller divides cpu resources by scaling time_slice of the tasks, so multiple subcpusets can be created under the cpuset that has one cpu only. So, uniprocessor systems also could get the benefit of subcpusets-patched cpusets.

Quoted 5 lines

My initial understanding is that subcpusets provide a way to partial out the proportion of cpu and memory used by various tasks. A leaf node cpuset can partition the tasks attached to it into subsets, called subcpusets, where each subcpuset gets a proportion of cpu and memory available to the original leaf node cpuset.

That's right.

Quoted 6 lines

I'm guessing you do not want such cpusets (the parents of subcpusets) to overlap, because if they did, it would seem to confuse the meaning of getting a fixed proportion of available cpu and memory resources. I was a little surprised not to see any additional checks that cpu_exclusive and mem_exclusive must be set true in these cpusets, to insure non- overlapping cpusets.

Right - I don't want the parents of the subcpusets to overlap and I should add such checks, but wanted to know first what people would think of subcpusets.

Regards,

KUROSAWA, Takahiro

#3 lore raw

Takahiro-san wrote, in reply to Paul:

Quoted 5 lines
Quoted 3 lines

2) Would a structure similar to Dinakar's patches to connect cpusets and dynamic sched domains (posted to linux-mm) work here as well?

Yes, subcpusets could work with the dynamic sched domains.

Ah - I see I was quite unclear about what I was thinking.

I intended to ask not simply would Dinakar's patches work with subcpusets, but something more radical.

These subcpusets, if I understand correctly, are a bit different from ordinary cpusets. For instance, it seems one cannot make child cpusets of them, and one cannot change most of their properties, such as cpus, mems, cpu_exclusive, mem_exclusive, or notify_on_release.

Also the mems and cpus of each subcpuset are required to be exactly equal to the corresponding values of its parent.

One of my passions is to avoid special cases across API boundaries.

I am proposing that you don't do subcpusets like this.

Consider the following alternative I will call 'cpuset meters'.

For each resource named 'R' (cpu and mem, for instance):
* Add a boolean flag 'meter_R' to each cpuset. If set, that R is metered, for the tasks in that cpuset or any descendent cpuset.
* If a cpuset is metered, then files named meter_R_guar, meter_R_lim and meter_R_cur appear in that cpuset to manage R's usage by tasks in that cpuset and descendents.
* There are no additional rules that restrict the ability to change various other cpuset properties such as cpus, mems, cpu_exclusive, mem_exclusive, or notify_on_release, when a cpuset is metered.
* It might be that some (or by design all) resource controllers do not allow nesting metered cpusets. I don't know. But one should (if one has permission) be able to make child cpusets of a metered cpuset, just like one can of any other cpuset.
* A metered cpuset might well have cpus or mems that are not the same as its parent, just like an unmetered cpuset ordinarly does.

If I understand correctly, one could place a job that managed its own child cpusets into a metered cpuset, but not into a subcpuset. Even if you wanted to allow it, it seems jobs in subcpusets cannot make child cpusets or modify the properties of their current cpuset. Is that correct?

The above is just a fragment of an idea. I do not know if it is a good idea or not. And I left off critical details such as just what the resource guarantees and limits mean.

Most likely, the way I stated it, there is some good reason that is very obvious to you why metered cpusets don't work. Perhaps some variation would work and be worthy of consideration as an alternative to subcpusets.

I hope to reduce to a minimum the special limitations on these cpusets, whether subcpusets or metered cpusets.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#4 lore raw
Quoted 7 lines · On Thu, 8 Sep 2005 05:02:32 -0700 Paul Jackson <[email protected]> wrote:

These subcpusets, if I understand correctly, are a bit different from ordinary cpusets. For instance, it seems one cannot make child cpusets of them, and one cannot change most of their properties, such as cpus, mems, cpu_exclusive, mem_exclusive, or notify_on_release.

Also the mems and cpus of each subcpuset are required to be exactly equal to the corresponding values of its parent.

That's right.

Quoted 6 lines

One of my passions is to avoid special cases across API boundaries.

I am proposing that you don't do subcpusets like this.

Consider the following alternative I will call 'cpuset meters'.

Quoted 15 lines

For each resource named 'R' (cpu and mem, for instance):
* Add a boolean flag 'meter_R' to each cpuset. If set, that R is metered, for the tasks in that cpuset or any descendent cpuset.
* If a cpuset is metered, then files named meter_R_guar, meter_R_lim and meter_R_cur appear in that cpuset to manage R's usage by tasks in that cpuset and descendents.
* There are no additional rules that restrict the ability to change various other cpuset properties such as cpus, mems, cpu_exclusive, mem_exclusive, or notify_on_release, when a cpuset is metered.
* It might be that some (or by design all) resource controllers do not allow nesting metered cpusets. I don't know. But one should (if one has permission) be able to make child cpusets of a metered cpuset, just like one can of any other cpuset.
* A metered cpuset might well have cpus or mems that are not the same as its parent, just like an unmetered cpuset ordinarly does.

Jackson-san's idea looks good for me because users don't need to create special cpusets (parents of subcpusets or subcpusets).

Quoted 1 line

From the point of users, maybe they wouldn't like to create

special cpusets.

As for the resource controller that I've posted, it assumes that there are groups of tasks that share the same cpumasks/nodemasks, and that there are no hierarchy in that groups in order to make things easier. I'll investigate how I can attach the resource controller to the cpuset meters.

Quoted 5 lines

If I understand correctly, one could place a job that managed its own child cpusets into a metered cpuset, but not into a subcpuset. Even if you wanted to allow it, it seems jobs in subcpusets cannot make child cpusets or modify the properties of their current cpuset. Is that correct?

Correct. Subcpusets can't make child cpusets, and they don't have cpumask/nodemask properties actually. Properties like cpumasks/nodemask of subcpusets are the same as their parent and subcpusets can't modify those properties.

Thanks,

KUROSAWA, Takahiro

#5 lore raw
Quoted 24 lines · KUROSAWA Takahiro
Quoted 6 lines · On Thu, 8 Sep 2005 05:02:32 -0700 Paul Jackson <[email protected]> wrote:

One of my passions is to avoid special cases across API boundaries.

I am proposing that you don't do subcpusets like this.

Consider the following alternative I will call 'cpuset meters'.

Quoted 15 lines

For each resource named 'R' (cpu and mem, for instance):
* Add a boolean flag 'meter_R' to each cpuset. If set, that R is metered, for the tasks in that cpuset or any descendent cpuset.
* If a cpuset is metered, then files named meter_R_guar, meter_R_lim and meter_R_cur appear in that cpuset to manage R's usage by tasks in that cpuset and descendents.
* There are no additional rules that restrict the ability to change various other cpuset properties such as cpus, mems, cpu_exclusive, mem_exclusive, or notify_on_release, when a cpuset is metered.
* It might be that some (or by design all) resource controllers do not allow nesting metered cpusets. I don't know. But one should (if one has permission) be able to make child cpusets of a metered cpuset, just like one can of any other cpuset.
* A metered cpuset might well have cpus or mems that are not the same as its parent, just like an unmetered cpuset ordinarly does.

Quoted 4 lines

Jackson-san's idea looks good for me because users don't need to create special cpusets (parents of subcpusets or subcpusets). From the point of users, maybe they wouldn't like to create special cpusets.

Yes, from the user POV it must be good to keep the hierarchical model. Ckrm and cpusets both provide a tree with descendents, children and parents. This hierarchical model is very nice IMO and provides a powerful API for the user.

Quoted 5 lines

As for the resource controller that I've posted, it assumes that there are groups of tasks that share the same cpumasks/nodemasks, and that there are no hierarchy in that groups in order to make things easier. I'll investigate how I can attach the resource controller to the cpuset meters.

Subcpusets, compared to cpusets and ckrm, gives the user a flat model. No hierarchy. Limited functionality compared to the hierachical model.

But what I think is important to keep in mind here is that cpusets and subcpusets do very different things. If I understand cpusets correctly, each cpuset may share processors or memory nodes with other cpusets. One task running on a shared processor may starve other cpusets using the same processor. This design works well with cpusets, but for resource controllers that must provide some kind of guarantee, this starvation is unsuitable.

And we already have an hierarchical alternative: ckrm. But look at the complexity and the amount of code. I believe that the complexity in ckrm mainly comes from the hierarchical model.

Maybe it is possible to have an hierarchical model and keep the framework simple and easy to understand while providing guarantees, I'm not sure. But until that happens, I'm quite happy with a simple, limited flat model.

/ magnus

#6 lore raw
Quoted 2 lines · magnus

Maybe it is possible to have an hierarchical model and keep the framework simple and easy to understand while providing guarantees,

Dinakar's patches to use cpu_exclusive cpusets to define dynamic sched domains accomplish something like this.

What scheduler domains and resource control domains both need are non-overlapping subsets of the CPUs and/or Memory Nodes.

In the case of sched domains, you normally want the subsets to cover all the CPUs. You want every CPU to have exactly one scheduler that is responsible for its scheduling.

In the case of resource control domains, you perhaps don't care if some CPUs or Memory Nodes have no particular resources constraints defined for them. In that case, every CPU and every Memory Node maps to _either_ zero or one resource control domain.

Either way, a 'flat model' non-overlapping partitioning of the CPUs and/or Memory Nodes can be obtained from a hierarchical model (nested sets of subsets) by selecting some of the subsets that don't overlap ;). In /dev/cpuset, this selection is normally made by specifying another boolean file (contains '0' or '1') that controls whether that cpuset is one of the selected subsets.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#7 lore raw
Quoted 6 lines · Paul Jackson
Quoted 2 lines · magnus

Maybe it is possible to have an hierarchical model and keep the framework simple and easy to understand while providing guarantees,

Dinakar's patches to use cpu_exclusive cpusets to define dynamic sched domains accomplish something like this.

Ah. I'm not familiar with Dinakar's patches and how these domains work. I'm actually more interested in this from the memory resource control point of view, but these areas are of course somehow related.

Quoted 2 lines

What scheduler domains and resource control domains both need are non-overlapping subsets of the CPUs and/or Memory Nodes.

Yes, that sounds like a good idea. But because English is not my native language, I just want to reassure that we mean the same thing. Non-overlapping subsets of cpu or memory nodes basically mean that children of a cpuset only clear bits in the bitmap, never sets them. Please correct me if I'm wrong.

Quoted 3 lines

In the case of sched domains, you normally want the subsets to cover all the CPUs. You want every CPU to have exactly one scheduler that is responsible for its scheduling.

Hm, I'm not sure how this relates to memory management, but you probably need a place to store per-bit (node/cpu) guarantee count regardless of memory or cpu resource control.

I think one major problem is how the guarantee should be divided between all subcpusets that share one bit. And using one percent value (or page count) per cpuset if more than 1 bit is set raises a similar guarantee question.

One solution to the guarantee dividing problem could be to have a percent value per bit set in the bitmap, maybe something like this:

# echo 0=50% > /dev/cpuset/foo/cpus
Quoted 5 lines

In the case of resource control domains, you perhaps don't care if some CPUs or Memory Nodes have no particular resources constraints defined for them. In that case, every CPU and every Memory Node maps to _either_ zero or one resource control domain.

These "resource control domains", are they similar to your "meter" suggestion earlier, or are they something else?

Quoted 6 lines

Either way, a 'flat model' non-overlapping partitioning of the CPUs and/or Memory Nodes can be obtained from a hierarchical model (nested sets of subsets) by selecting some of the subsets that don't overlap ;). In /dev/cpuset, this selection is normally made by specifying another boolean file (contains '0' or '1') that controls whether that cpuset is one of the selected subsets.

This boolean file, do you mean one of your "meter" files or something else?

Thank you for your comments!

/ magnus - will have a nice weekend now... =)

#8 lore raw
Quoted 2 lines · Magnus

Non-overlapping subsets of cpu or memory nodes basically mean that children of a cpuset only clear bits in the bitmap, never sets them.

X is a subset of Y if every element of X is also in Y.

My phrase "a subset of the CPUs" really just meant "some set of CPUs" on the system. A subset of all CPUs is a set of some CPUs, perhaps empty, perhaps including them all, perhaps in between.

The phrase "non-overlapping" mean that the several subsets had no overlap - no CPU was in more than one of the subsets.

Ditto ... for Memory Nodes.

To be honest, I don't think in terms of bits. The bitmaps, cpumasks and nodemasks are just ways of representing sets. I also uses various lists and arrays to represent sets of CPUs and sets of Memory Nodes, depending on what's convenient.

My math background, before getting into computers, was in Set Theory.

Quoted 1 line

a place to store per-bit (node/cpu) guarantee count

I do not understand this snippet.

Quoted 2 lines

I think one major problem is how the guarantee should be divided between all subcpusets that share one bit

That's why I said 'non-overlapping'. That means no shared bits (no shared CPUs or Memory Nodes).

Quoted 1 line

"resource control domains"

These were whatever sets of CPUs or Memory Nodes had resource control functions. Both Takahiro-san's subcpusets and my cpusets with meter attributes were examples of such. No doubt CKRM has such too, though I've forgotten too much of CKRM to say what.

Quoted 2 lines

This boolean file, do you mean one of your "meter" files or something else?

Look under /dev/cpuset (after mounting it: mount -t cpuset cpuset /dev/cpuset). You will seem that some of the files, such as the files named:
mem_exclusive
cpu_exclusive
notify_on_release

always contain either the string "0\n" or "1\n". These are boolean files. They are file representations of boolean values (True or False).

A cpuset with resource metering attributes needed:
1) one boolean file, to mark that it had such metering attributes.
2) several meter specific files, holding various parameter settings, such as what percentage of the CPUs allowed in this cpuset may be used by the tasks in this cpuset.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#9 lore raw
In reply to Paul Jackson

Hi,

Quoted 20 lines
Quoted 2 lines · magnus

Maybe it is possible to have an hierarchical model and keep the framework simple and easy to understand while providing guarantees,

Dinakar's patches to use cpu_exclusive cpusets to define dynamic sched domains accomplish something like this.

What scheduler domains and resource control domains both need are non-overlapping subsets of the CPUs and/or Memory Nodes.

In the case of sched domains, you normally want the subsets to cover all the CPUs. You want every CPU to have exactly one scheduler that is responsible for its scheduling.

In the case of resource control domains, you perhaps don't care if some CPUs or Memory Nodes have no particular resources constraints defined for them. In that case, every CPU and every Memory Node maps to _either_ zero or one resource control domain.

Quoted 6 lines

Either way, a 'flat model' non-overlapping partitioning of the CPUs and/or Memory Nodes can be obtained from a hierarchical model (nested sets of subsets) by selecting some of the subsets that don't overlap ;). In /dev/cpuset, this selection is normally made by specifying another boolean file (contains '0' or '1') that controls whether that cpuset is one of the selected subsets.

What do you think if you make cpusets for sched domain be able to have their siblings, which have the same attribute and share their resources between them.

I guess it would be simple.

Thanks,
Hirokazu Takahashi.

#10 lore raw
Quoted 3 lines · Takahashi-san

What do you think if you make cpusets for sched domain be able to have their siblings, which have the same attribute and share their resources between them.

I do not understand this question. I guess "cpusets for sched domains" means "cpusets whose 'cpu_exclusive' attribute is marked true, but which have no child cpusets so marked."

But even that guess I am unsure of, and the rest of the sentence "which have the same ..." I don't even have a guess what means.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#11 lore raw

Hi,

Quoted 7 lines
Quoted 3 lines

What do you think if you make cpusets for sched domain be able to have their siblings, which have the same attribute and share their resources between them.

I do not understand this question. I guess "cpusets for sched domains" means "cpusets whose 'cpu_exclusive' attribute is marked true, but which have no child cpusets so marked."

Yes.

Quoted 2 lines

But even that guess I am unsure of, and the rest of the sentence "which have the same ..." I don't even have a guess what means.

Sorry for the poor explanation.
I just thought that it wouldn't be bad to allow "each cpuset whose cpu_exclusive attribute is mark true" to have its clones like the figure below. In this case, cpu-2 and cpu-3 will be used exclusively for the clones --- CPUSET 1, 2, and 3 ---.

I guess it seems very similar to one of your ideas except for reusing cpu_exclusive flag. Do you think reusing the flag is good idea?

     +-------------------+----------------+----------------+
     |                   |                |                |
  CPUSET 0            CPUSET 1         CPUSET 2         CPUSET 3
  sched domain A      sched domain B   sched domain B   sched domain B
  cpus: 0, 1          cpus: 2, 3       cpus: 2, 3       cpus: 2, 3
  cpu_exclusive       cpu_exclusive    cpu_exclusive    cpu_exclusive 
                      meter_cpu        meter_cpu        meter_cpu
                      <------should we call it resouce domain?------>

Thanks,
Hirokazu Takahashi.

#12 lore raw

Well, I suspect I don't understand yet.

Nice picture though - that gives me some idea what you mean.

Do notice that the basic rule of cpu_exclusive cpusets is that their CPUs don't overlap their siblings. Your Cpusets 1, 2, and 3 seem to be marked cpu_exclusive in your picture, but all contain the same CPUs 2 and 3, overlapping each other.

I'm guessing what you are trying to draw is:

Tasks on CPUs 0 and 1 have no resource control limits.

Tasks on CPUs 2 and 3 have resource control limits specifying
what percentage of the CPUs 2 and 3 is available to them.

I might draw my solution to that as:

     +-----------------------------------+
     |                                   |
  CPUSET 0                            CPUSET 1         
  sched domain A                      sched domain B   
  cpus: 0, 1                          cpus: 2, 3       
  cpu_exclusive=1                     cpu_exclusive=1
  meter_cpu=0                         meter_cpu=0
                                         |
                        +----------------+----------------+
                        |                |                |
                     CPUSET 1a        CPUSET 1b        CPUSET 1c
                     cpus: 2, 3       cpus: 2, 3       cpus: 2, 3
		     cpu_exclusive=0  cpu_exclusive=0  cpu_exclusive=0
                     meter_cpu=1      meter_cpu=1      meter_cpu=1
                     meter_cpu_*      meter_cpu_*      meter_cpu_*

The meter_cpu_* files in each of Cpusets 1a, 1b, and 1c control what proportion of the CPU resources in that Cpuset can be used by the tasks in that Cpuset.

If meter_cpu is false (0) then the meter_cpu_* files do not appear, which is equivalent to allowing 100% of the CPUs in that Cpuset to be used by the tasks in that Cpuset (and descendents, of course.)

Don't forget - this all seems like it has significant mission overlap with CKRM. I hate to repeat this, but the relation of your work to CKRM needs to be understood before I am likely to agree to accepting your work into the kernel (not that my acceptance is required; you really just need Linus to agree, though he of course considers the positions of others to some inscrutable degree.)

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#13 lore raw

Hi,

Quoted 8 lines

Well, I suspect I don't understand yet.

Nice picture though - that gives me some idea what you mean.

Do notice that the basic rule of cpu_exclusive cpusets is that their CPUs don't overlap their siblings. Your Cpusets 1, 2, and 3 seem to be marked cpu_exclusive in your picture, but all contain the same CPUs 2 and 3, overlapping each other.

Yes, I know the current design of cpu_exclusive.
I just thought if I could enhance cpu_exclusive to cover a group of cpusets to make the hierarchy as flat as possible.

Quoted 3 lines

I'm guessing what you are trying to draw is:

Tasks on CPUs 0 and 1 have no resource control limits.

I thought CPU 1 could have resource control limits.

Quoted 4 lines

Tasks on CPUs 2 and 3 have resource control limits specifying
what percentage of the CPUs 2 and 3 is available to them.

I might draw my solution to that as:

I understand your idea clearly, which is essentially the same as Korosawa's design. Your design looks very straight and I have no objection to it.

My only concern is that it would become harder to control resources between CPUSET 1a,1b and 1c if some processes are assigned to CPUSET 1 directly. But I just get an idea that it would be OK if CPUSET 1 can have meter_cpu=1 to share the resources.

Quoted 24 lines
     +-----------------------------------+
     |                                   |
  CPUSET 0                            CPUSET 1         
  sched domain A                      sched domain B   
  cpus: 0, 1                          cpus: 2, 3       
  cpu_exclusive=1                     cpu_exclusive=1
  meter_cpu=0                         meter_cpu=0
                                         |
                        +----------------+----------------+
                        |                |                |
                     CPUSET 1a        CPUSET 1b        CPUSET 1c
                     cpus: 2, 3       cpus: 2, 3       cpus: 2, 3
		     cpu_exclusive=0  cpu_exclusive=0  cpu_exclusive=0
                     meter_cpu=1      meter_cpu=1      meter_cpu=1
                     meter_cpu_*      meter_cpu_*      meter_cpu_*

The meter_cpu_* files in each of Cpusets 1a, 1b, and 1c control what proportion of the CPU resources in that Cpuset can be used by the tasks in that Cpuset.

If meter_cpu is false (0) then the meter_cpu_* files do not appear, which is equivalent to allowing 100% of the CPUs in that Cpuset to be used by the tasks in that Cpuset (and descendents, of course.)

Quoted 6 lines

Don't forget - this all seems like it has significant mission overlap with CKRM. I hate to repeat this, but the relation of your work to CKRM needs to be understood before I am likely to agree to accepting your work into the kernel (not that my acceptance is required; you really just need Linus to agree, though he of course considers the positions of others to some inscrutable degree.)

OK.

Thanks,
Hirokazu Takahashi.

#14 lore raw

cover CPUMETER (Re: [PATCH 0/5] SUBCPUSETS: a resource control functionality using CPUSETS)

In reply to Paul Jackson

Jackson-san,

Sorry for the late reply.

I've implemented the cpumeter based on your idea with small modification on handling meter_cpu. To make grouping the metered cpus and nodes easier, I modified your idea so that the meter_cpu file is also used for marking the toplevel of the metered CPUSET (CPUSET 1). The toplevel CPUSET (CPUSET 1) is also metered, that is different from the subcpuset_top file of SUBCPUSETS.

      +-----------------------------------+
      |                                   |
   CPUSET 0                            CPUSET 1
   sched domain A                      sched domain B
   cpus: 0, 1                          cpus: 2, 3
   cpu_exclusive=1                     cpu_exclusive=1
   meter_cpu=0                         meter_cpu=1
                                       meter_cpu_*
                                          |
                         +----------------+----------------+
                         |                |                |
                      CPUSET 1a        CPUSET 1b        CPUSET 1c
                      cpus: 2, 3       cpus: 2, 3       cpus: 2, 3
 		     cpu_exclusive=0  cpu_exclusive=0  cpu_exclusive=0
                      meter_cpu=1      meter_cpu=1      meter_cpu=1
                      meter_cpu_*      meter_cpu_*      meter_cpu_*

|
+------------+------------+
| |
CPUSET 2a CPUSET 2b
cpus: 2, 3 cpus: 2, 3
cpu_exclusive=0 cpu_exclusive=0
meter_cpu=1 meter_cpu=1
meter_cpu_* meter_cpu_*

Here are other rules around meters:
- If meter_cpu is 1, meter_cpu_* files appear.
- The children (CPUSET 1a, 1b, 1c) inherit CPUSET 1's value of cpus/mems/meter_cpu/... and do not have their specific values.
- The metered CPUSETS can have their children
(this is not allowed in SUBCPUSETS).
- meter_cpu in the children of metered CPUSETS can not be modified (can not create normal CPUSETS under metered CPUSETS).

I'll send patches right after this mail.

Comments appreciated,
KUROSAWA, Takahiro

#15 lore raw

Re: [PATCH 0/3] CPUMETER (Re: [PATCH 0/5] SUBCPUSETS: a resource control functionality using CPUSETS)

Takahiro-san,

I spent a little more reading the cpuset side of your cpumeter patches.

I am hopeful that some substantial restructuring of the code would integrate it better with the existing cpuset structure, reducing the size of new code substantially.

I noticed not one, but two, nested CONFIG options added for CPUMETER:
CONFIG_CPUMETER and CONFIG_CPU_RC. If this CPUMETER patch only affected the cpuset code, I'd be tempted to have no additional CONFIG_* options, however since the affect on the scheduler might be more interesting (I am not a scheduler guru), it might make sense to have one new such option - say the CONFIG_CPUMETER one, that covers it all. Two seems at least one too many.

I am surprised to see fixed length arrays of size CPUMETER_CTRLS_MAX, rather than having the cpumeter struct be dynamically allocated,
reference counted and locked. I hope we do not have to have a small
fixed upper limit on the number of distinct cpu controllers.

I suspect that we should have a single additional pointer to cpu resource controller (rc) domain structure, referenced from both cpusets and tasks, where that structure is reference counted and has its own lock. All tasks and all cpuses in a given cpu rc domain would point to the same structure. This would remove any need in the critical scheduler code to reference a tasks cpuset or the parent of that cpuset, and it would allow the cpusets in such a domain to have child cpusets, which would inherit another reference to the same cpumeter rc domain, as I have been hoping would be possible.

So the code now added to struct cpuset:

	#ifdef CONFIG_CPUMETER
		/*
		 * rcdomains: used for the recource control domains
		 *            to keep track of total ammount of resources.
		 * meters:    used for metering resources assigned for
		 *            the cpuset.
		 */
		void *rcdomains[CPUMETER_CTLRS_MAX];
		struct cpumeter meters[CPUMETER_CTLRS_MAX];
	#endif /* CONFIG_CPUMETER */

might collapse to simply:

	struct cpumeter *cpumeter;

The use of the last CPUMETER_CTLRS_MAX (16) bits, starting at offset CS_METER_OFFSET, of the typedef enum cpuset_flagbigs_t for one bit per controller instance is creative, but hopefully unnecessary. Rather, if the struct cpumeters are dynamically allocated, then whether the cpuset->cpumeter pointer is non-NULL will determine if that cpuset is metered, and the pointer will reference the information about that metered domain, shared by all cpusets in that domain, and hence by all tasks in those cpusets. Be sure to update which cpumeter rc domain a task is in, if that task is moved to another cpuset, by attach_task().

The cpumeter_destroy_meters() routine should not be called from cpuset_diput(). Rather the reference count on the meter structure should be incremented and decremented each time a cpuset or task attaches to or detached from it, and it should be free'd when the count goes to zero.

There are 736 lines of code added to the end of kernel/cpuset.c, with many snippets closely resembling existing code from the rest
of kernel/cpuset.c. This is too much duplicated, parallel code,
and too much duplicate, parallel data structures. I am hoping that this can be collapsed into the existing cpuset.c code, adding perhaps 100 or 200 lines instead of 736 lines. But I have not thought about it too hard - this hope might be wishful thinking. Seeing routines such as "cpumeter_file_read_common()", along side the existing "cpuset_common_file_read()", worries me. Way too much duplication of code. And I hope we don't need a whole different set of file_operations for the guar and lim files.

This code provides a basis for other forms of metering besides just cpu metering. I wonder if perhaps it would be better to just do cpu metering here, and remove a bit of the generality that supports other types of controllers. If the addition of this cpuset based controller were sufficiently integrated with the existing cpuset code, and kept simple enough, then adding another controller type, such as for memory, could just do the same sort of thing easy enough. It would be easier to read the code if lines such as the following:

		sprintf(name, "%s%s%s", cpumeter_meter_prefix, c->name,
			cpumeter_guar_suffix);

were instead:

		name = "meter_cpu_guar";

You have been most gracious in your consideration of my suggestions so far. I hope the above thoughts will benefit your work.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#16 lore raw

Re: [PATCH 0/3] CPUMETER (Re: [PATCH 0/5] SUBCPUSETS: a resource control functionality using CPUSETS)

Quoted 5 lines · On Sat, 1 Oct 2005 21:20:26 -0700 Jackson-san wrote:

I spent a little more reading the cpuset side of your cpumeter patches.

I am hopeful that some substantial restructuring of the code would integrate it better with the existing cpuset structure, reducing the size of new code substantially.

Thanks for the suggestion.
I'll update my patch and send it when I finish updating.

#17 lore raw

PATCH 1/3 CPUMETER: add cpumeter framework to the CPUSETS

In reply to Paul Jackson

This patch adds CPUMETER framework code to the CPUSETS. CPUMETER is meant for subdividing cpuset resources.
A few files are added in order to control the guarantee and the limit of the resource amount to the cpuset filesystem. Also, interfaces for the specific resource controller like CPU and memory.

include/linux/cpuset.h +40 -0
@@ -14,6 +14,46 @@
 
 #ifdef CONFIG_CPUSETS
 
+#ifdef CONFIG_CPUMETER
+struct cpumeter_ctlr {
+	char *name;		/* controller name */
+	int idx;		/* used by cpumeter core */
+	void *(*create_rcdomain)(struct cpuset *cs, cpumask_t cpus,
+				 nodemask_t mems);
+	void (*destroy_rcdomain)(void *rcd);
+	void *(*create)(void *rcd, struct cpuset *cs);
+	void (*destroy)(void *ctldata);
+	int (*set_lim)(void *ctldata, unsigned long val);
+	int (*set_guar)(void *ctldata, unsigned long val);
+	int (*get_cur)(void *ctldata, unsigned long *valp);
+};
+
+extern int cpumeter_register_controller(struct cpumeter_ctlr *ctlr);
+extern void *cpumeter_get_controller_data(struct cpuset *cs,
+					  struct cpumeter_ctlr *ctlr);
+extern void *cpumeter_get_rcdomain(struct cpuset *cs,
+				   struct cpumeter_ctlr *ctlr);
+#else /* CONFIG_CPUMETER */
+struct cpumeter_ctlr;
+
+static inline int cpumeter_register_controller(struct cpumeter_ctlr *ctlr)
+{
+	return -EINVAL;
+}
+
+static inline void *cpumeter_get_controller_data(struct cpuset *cs,
+						 struct cpumeter_ctlr *ctlr)
+{
+	return NULL;
+}
+
+static inline void *cpumeter_get_rcdomain(struct cpuset *cs,
+					  struct cpumeter_ctlr *ctlr)
+{
+	return NULL;
+}
+#endif /* CONFIG_CPUMETER */
+
 extern int cpuset_init(void);
 extern void cpuset_init_smp(void);
 extern void cpuset_fork(struct task_struct *p);
init/Kconfig +9 -0
@@ -238,6 +238,15 @@ config CPUSETS
 
 	  Say N if unsure.
 
+config CPUMETER
+	bool "Cpumeter support"
+	depends on CPUSETS
+	help
+	  This option enables the resource control of CPUs and memory
+	  via the CPUSETS interface.
+
+	  Say N if unsure.
+
 menuconfig EMBEDDED
 	bool "Configure standard kernel features (for small systems)"
 	help
kernel/cpuset.c +826 -8

914 lines not on this page · raw message

#18 lore raw

Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

[[ Added [email protected] to the cc list ]]

Welcome Takahiro-san. I see you have been very busy.

First I will read through your patch set, and comment on little details that I happen to notice. Then if I have the energy, I will step back and try to see a larger view of your fine work.

First minor detail ... I think that the following snippet of code:

+ ssize_t retval;

+	size_t n;
	...

+ /* Do nothing if *ppos is at the eof or beyond the eof. */
+ if (s - page <= *ppos)
+ return 0;
+
+ start = page + *ppos;

+	n = s - start;
	...

+ free_page((unsigned long)page);
+ return retval;

could be slightly clearer, and avoid a leak, as something like:

	ssize_t retval = 0;
	size_t n;			/* how much not yet read */
	...
	start = page + *ppos;
	n = s - start;
	if (n <= 0)
		goto done;
	...
done:
	free_page((unsigned long)page);
	return retval;

I find that inequalities involving expressions (such as 's - page') cause my brain to stumble momentarily. And this avoids leaking a memory page if n <= 0.

The above reminds me of a bug fix that you provided in the previous patch set, for the case *ppos >= eof. I wonder if we have duplicated code here.

I see more duplicated code in cpumeter_file_get_written_data(), where you are reading what is I guess a single boolean '0' or '1', but validating that the user provided length is sane by checking it:

+ /* Crude upper limit on largest legitimate cpulist user might write. */
+ if (nbytes > 100 + 6 * NR_CPUS)
+ return -E2BIG;

This is an appropriate check when reading (write system call) an entire cpu list, not a single number. For a single number, you could even consider using a local "char buf[16]" array or some such size, instead of malloc'ing a whole page.

No doubt this "Crude upper limit" check was taken from the existing routine cpuset_common_file_write() routine, which has to handle a greater variety of write calls.

When I see code that is copied from another routine, I ask myself if there is a clean way to avoid the duplication of code. Though I will confess to being too lazy at the moment to see if there is a good way to do that here.

+#ifdef CONFIG_CPUMETER
+config CPUMETER
+ bool "Cpumeter support"
+ depends on CPUSETS

It might not be worth having CPUMETER as a separate option on top of CPUSETS. I guess that depends in part on how much CPUMETER grows the compiled kernel text size, and what performance impact it has if any. You might want to look for a way to measure both those (text size increase and performance penalty on some appropriate benchmark.) If CPUMETER is not too big an extra burden above and beyond CPUSETS, then I would probably prefer avoiding the extra ifdefs and config option, and make it one with CPUSETS.

+ s += snprintf(s, PAGE_SIZE, "%lu", val);
+ *s++ = '\n';
+ *s = '\0';

Can the above be collapsed to:

	s += snprintf(s, PAGE_SIZE, "%lu\n", val);

I see that the cpu controller patch, as it must, has hooks in the critical scheduler paths. How much does this slow down a system if CONFIG_CPUMETER is enabled, but the system is running in the default cpuset and cpumeter configuration, such as it would be after a reboot, before any intentional administration to setup particular cpusets or meters is attempted?

... hmmm ... I am ready to retire for the night. I will have to continue my review another day.

You will need to encourage someone else, with scheduler expertise, to review that portion of the patch. The kernel/sched.c file is too hard for me; I stick to easier files such as kernel/cpuset.c.

I continue to be quite suspicious that perhaps there should be a tighter relation between your work and CKRM. For one thing, I suspect that CKRM has a cpu controller that serves essentially the same purpose as yours. If that is so, I cannot imagine that we would ever put both cpu controllers in the kernel. They touch on code that is changing too rapidly, and too critical for performance.

My wild guess would be that the right answer would be to take the CKRM cpu controller instead of yours, and connect it to cpusets in the manner that you have done here. But I have no expertise in cpu controllers, so am quite unfit to judge which one or the other, or perhaps some combination of the two cpu controllers, is the best one.

I hesitate to keep asking about CKRM, as I recall how much I disliked it when Andrew kept asking me much the same questions, about the relation between cpusets and CKRM. Such is life; one must do ones duty.

Looking back at your nice opening picture, I see you write:

Quoted 5 lines

cpus/mems/meter_cpu/... and do not have their specific values.
- The metered CPUSETS can have their children
(this is not allowed in SUBCPUSETS).
- meter_cpu in the children of metered CPUSETS can not be modified (can not create normal CPUSETS under metered CPUSETS).

This seems more restrictive than necessary. Indeed, it reminds me of some of the concerns I had with the previous SUBCPUSET proposal. I think we should only need the following:

* Some cpuset C in the hierarchy is marked cpu_exclusive (hence its ancestors are also so marked.)
* None of C's descendents are cpu_exclusive. This will make cpuset C define a sched domain.
* Each of the -immediate- children of C are marked meter_cpu.
* But C is not marked meter_cpu, none of the ancestors of C are marked meter_cpu, and none of the descendents of C's children are marked meter_cpu. Just C's children as so marked.
* C's immediate children must have the same CPU's as C. Children of these children can have any CPU's (that are a subset of C's, of course.)
* Each of C's immediate children gets a certain portion of the CPU time, as specified in its meter_cpu_* files, to be shared amongst all tasks running in that cpuset, or any descendent of that cpuset.
* This should allow for creating normal cpusets under metered cpusets.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#19 lore raw

Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Quoted 11 lines · Paul Jackson

You will need to encourage someone else, with scheduler expertise, to review that portion of the patch. The kernel/sched.c file is too hard for me; I stick to easier files such as kernel/cpuset.c.

I continue to be quite suspicious that perhaps there should be a tighter relation between your work and CKRM. For one thing, I suspect that CKRM has a cpu controller that serves essentially the same purpose as yours. If that is so, I cannot imagine that we would ever put both cpu controllers in the kernel. They touch on code that is changing too rapidly, and too critical for performance.

Quoted 6 lines

My wild guess would be that the right answer would be to take the CKRM cpu controller instead of yours, and connect it to cpusets in the manner that you have done here. But I have no expertise in cpu controllers, so am quite unfit to judge which one or the other, or perhaps some combination of the two cpu controllers, is the best one.

Last time I looked at the CKRM cpu controller code I found it was quite horrible, with a great deal of duplication and very intrusive large and complex.

It could have come a long way since then, but this code looks much neater than the code I reviewed.

I guess the question of the resource controller stuff is going to come up again sooner or later. I would hope to have just a single CPU resource controller (presumably based on cpusets), the simpler the better ;)

Nick

Signature · 5 lines
SUSE Labs, Novell Inc.



Send instant messages to your online friends http://au.messenger.yahoo.com 
#20 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Quoted 1 line · Nick

this code looks much neater than the code I reviewed.

This does not surprise me. Takahiro-san writes nice code.

I will abstain from picking one cpu controller over another, as that is not an area of my expertise. But I definitely agree with your hope:

Quoted 1 line

to have just a single CPU resource controller

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#21 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

In reply to Nick Piggin
Quoted 4 lines · Nick Piggin

Last time I looked at the CKRM cpu controller code I found it was quite horrible, with a great deal of duplication and very intrusive large and complex.

I admit it :)... and that was the reason why we did not post that to lkml.

Quoted 1 line

It could have come a long way since then, but this code looks

Since we were not planning to use it there isn't much change in the code :(

Quoted 6 lines

much neater than the code I reviewed.

I guess the question of the resource controller stuff is going to come up again sooner or later. I would hope to have just a single CPU resource controller (presumably based on cpusets), the simpler the better ;)

We were planning to start on a simplified CPU controller that can provide the functionalities CKRM is expected to provide.

As I stated in an earlier email cpusubsets looks promising for CKRM, but we are not able to spend more time on it as of now as the team is very busy trimming down CKRM.

Quoted 3 lines

Nick

Signature · 4 lines
----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - [email protected]   |      .......you may get it.
----------------------------------------------------------------------
#22 lore raw

Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

In reply to Nick Piggin
Quoted 2 lines · On Tue, 27 Sep 2005 19:22:17 +1000 Nick Piggin <[email protected]> wrote:

It could have come a long way since then, but this code looks much neater than the code I reviewed.

I'm glad to hear that! But the cpu resource controller has some problems, for example:

- The controller only controls the time_slice value and doesn't care the balance of cpus (the controller leaves balancing to the existing balancing code in the scheduler). So far I don't have any good idea to solve this.

- The controller holds a spinlock once per 1 second. I don't know this is permissive or not, but the current scheduler doesn't hold any spinlocks normally, so...

Signature · 1 line
KUROSAWA, Takahiro
#23 lore raw

Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Hello,

Quoted 5 lines
Quoted 2 lines

It could have come a long way since then, but this code looks much neater than the code I reviewed.

I'm glad to hear that! But the cpu resource controller has some problems, for example:

These are issues to be improved.

Quoted 4 lines

- The controller only controls the time_slice value and doesn't care the balance of cpus (the controller leaves balancing to the existing balancing code in the scheduler). So far I don't have any good idea to solve this.

Is there any problem to enhance load_balance()? Applying some weight to each process, it would be used for calculating load of each runqueue.

Quoted 3 lines

- The controller holds a spinlock once per 1 second. I don't know this is permissive or not, but the current scheduler doesn't hold any spinlocks normally, so...

I think each runqueue has its own spinlock, which is held quite often.

Thanks,
Hirokazu Takahashi.

#24 lore raw

Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Quoted 6 lines · On Wed, 28 Sep 2005 19:08:22 +0900 (JST) Hirokazu Takahashi…
Quoted 4 lines

- The controller only controls the time_slice value and doesn't care the balance of cpus (the controller leaves balancing to the existing balancing code in the scheduler). So far I don't have any good idea to solve this.

Is there any problem to enhance load_balance()? Applying some weight to each process, it would be used for calculating load of each runqueue.

Probably we can solve this issue by applying some weight to each tasks and calculating the load of runqueues based on that weight. But I can not get to the implementation out of your idea so far.

Quoted 4 lines
Quoted 3 lines

- The controller holds a spinlock once per 1 second. I don't know this is permissive or not, but the current scheduler doesn't hold any spinlocks normally, so...

I think each runqueue has its own spinlock, which is held quite often.

Ah, thanks for pointing out my mistake. Maybe I confused that with something else.
Then it should be more permissive than I had thought, at least :)

Signature · 1 line
KUROSAWA, Takahiro
#25 lore raw

Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

In reply to Paul Jackson

Jackson-san,

Thank you for the review. I'll fix the bugs that you have found.

Quoted 3 lines · On Tue, 27 Sep 2005 01:37:51 -0700 Paul Jackson <[email protected]> wrote:

The above reminds me of a bug fix that you provided in the previous patch set, for the case *ppos >= eof. I wonder if we have duplicated code here.

Ah, yes, we need to fix the bug in the cpuset code introduced by me...

Quoted 6 lines

I see that the cpu controller patch, as it must, has hooks in the critical scheduler paths. How much does this slow down a system if CONFIG_CPUMETER is enabled, but the system is running in the default cpuset and cpumeter configuration, such as it would be after a reboot, before any intentional administration to setup particular cpusets or meters is attempted?

I don't have any benchmark numbers yet, but the cpu controller code will add some if statements and will not hold any spinlocks to the scheduler in the default configuration.

Quoted 10 lines

Looking back at your nice opening picture, I see you write:

Quoted 5 lines

cpus/mems/meter_cpu/... and do not have their specific values.
- The metered CPUSETS can have their children
(this is not allowed in SUBCPUSETS).
- meter_cpu in the children of metered CPUSETS can not be modified (can not create normal CPUSETS under metered CPUSETS).

This seems more restrictive than necessary. Indeed, it reminds me of some of the concerns I had with the previous SUBCPUSET proposal. I think we should only need the following:

I have a few questions for your idea to make the design in my mind clearer.

Quoted 5 lines

* Some cpuset C in the hierarchy is marked cpu_exclusive (hence its ancestors are also so marked.)
* None of C's descendents are cpu_exclusive. This will make cpuset C define a sched domain.
* Each of the -immediate- children of C are marked meter_cpu.

Can the cpuset C have immediate children with meter_cpu=0? Or should it be prohibited to set meter_cpu=0 for the immediate children of C?

If it is prohibited to set meter_cpu=0 for the immediate children of C, cpuset_create() needs a check whether the siblings are metered or not. If the siblings are metered, the newly created cpuset should also be metered. And probably it is not allowed to set meter_cpu=1 if the sibling cpusets have meter_cpu=0.

Quoted 3 lines

* But C is not marked meter_cpu, none of the ancestors of C are marked meter_cpu, and none of the descendents of C's children are marked meter_cpu. Just C's children as so marked.

Good idea, but I'd like to make sure...
Is it prohibited for any decendant of C's children to set meter_cpu=1 ?

Quoted 9 lines

* C's immediate children must have the same CPU's as C. Children of these children can have any CPU's (that are a subset of C's, of course.)
* Each of C's immediate children gets a certain portion of the CPU time, as specified in its meter_cpu_* files, to be shared amongst all tasks running in that cpuset, or any descendent of that cpuset.
* This should allow for creating normal cpusets under metered cpusets.

Best regards,

Signature · 1 line
KUROSAWA, Takahiro
#26 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Takahiro-san asks perceptive question:

Quoted 3 lines

If it is prohibited to set meter_cpu=0 for the immediate children of C, cpuset_create() needs a check whether the siblings are metered or not.

Very good question. It exposes an impossibility in my proposal, as stated. The rule I had in mind was that either all the children of C had meter_cpu set, or none of them. But since one can only mark one cpuset at a time, this is impossible to setup if there is more than one child already.

Allow me to try to fix my proposal.

Instead of doing the impossible and trying to mark all the children of C as meter_cpu all at same instant in time, I should just mark the parent cpuset C, one time. Then if C is so marked, its -children- now have the meter_cpu_* files, which can default at that instant in time to providing (1/N) of the cpu to each of the N children of C. I would say that C can only be marked meter_cpu if:
* C is already marked cpu_exclusive.
* All of its children have the same 'cpus' setting as C.
* Any new child of C created after C is marked meter_cpu will automatically start with the same 'cpus' setting as C, and with the meter_cpu_* files.
* It is prohibited to change the 'cpus' of any cpuset whose parent is marked meter_cpu.
* Changing the 'cpus' of a cpuset such as C that is itself marked meter_cpu will instantly change the 'cpus' of each of its children.
* It is prohibited to turn on the cpu_exclusive flag of a cpuset whose parent is marked meter_cpu, or to turn off the cpu_exclusive flag on a cpuset that is itself marked meter_cpu.

Similar rules would apply for mem_exclusive and meter_mem.

The metered children of C may have their own children in turn, which may have cpus any subset of the cpus in C, but which cannot be marked cpu_exclusive or meter_cpu.

Borrowing your fine art work, and modifying it slightly, this looks like:

      +-----------------------------------+
      |                                   |
   CPUSET 0                            CPUSET 1 (aka 'C')
   sched domain A                      sched domain B
   cpus: 0, 1                          cpus: 2, 3
   cpu_exclusive=1                     cpu_exclusive=1
   meter_cpu=0                         meter_cpu=1
                                          |
                         +----------------+----------------+
                         |                |                |
                      CPUSET 1a        CPUSET 1b        CPUSET 1c
                      cpus: 2, 3       cpus: 2, 3       cpus: 2, 3
                      cpu_exclusive=0  cpu_exclusive=0  cpu_exclusive=0
                      meter_cpu=0      meter_cpu=0      meter_cpu=0
                      meter_cpu_*      meter_cpu_*      meter_cpu_*
                         |
            +------------+------------+
            |                         |
         CPUSET 2a                CPUSET 2b
         cpus: 2                  cpus: 3
         meter_cpu=0              meter_cpu=0
         cpu_exclusive=0          cpu_exclusive=0

Note here that marking C (CPUSET 1) as meter_cpu exposes the meter_cpu_* files in the children of C.

Quoted 1 line

Is it prohibited for any decendant of C's children to set meter_cpu=1 ?

Yes, I presume so, and made up my new rules above assuming that. It is definitely worth an effort in my opinion to allow creating nested ordinary (not metered) cpusets below the children of C, but I am guessing it would be too hard to try to allow nesting of metered cpusets below metered cpusets. If you have a mind to try that however, I am more than willing to listen to your proposal.

The above proposal makes it more obvious than ever that I am starting to overload the meaning of cpu_exclusive and mem_exclusive perhaps a bit too much.

One or the other of the two *_exclusive flags should be required preconditions for some of these special properties (sched domains, GFP_KERNEL memory allocation confinement, oom killer confinement, cpu metering and memory metering), but perhaps actually enabling any of these special properties should be an additional and distinct choice.

Therefore I propose some new cpuset flags:
* 'sched_domain' to mark sched domains (now done by the cpu_exclusive flag),
* 'kernel_memory' to mark the constraints on GFP_KERNEL allocations,
* 'oom_killer' to mark the constraints on oom killing,
* your 'meter_cpu' flag to mark a set of metered cpus, and
* your 'meter_mem' flag to mark a set of metered mems.

Each of these new flags would require the appropriate cpu_exclusive or mem_exclusive flag on the same cpuset to already be set, but just setting the *_exclusive flags by themselves would not be enough to get you the special behaviour. You would also have to set the appropriate one of these new flags.

So, for example, the condition to define a sched domain would change, from just being the lowest level cpuset marked cpu_exclusive (or the left over CPUs not marked exclusive), to being both that -and- having its "sched_domain" flag marked True (or being the left over CPUs, again).

At first writing, I like the sound of this. But then I often think my suggestions are good, when I first write them <grin>.

Without these new flags, the interface has an odd assymmetry to it. Just setting cpu_exclusive could get you a sched domain for instance, but you had to have both cpu_exclusive and meter_cpu to get the cpu metering code. The only reason for this was that Dinakar got his sched domain patch in before you got your cpu meter patch in, which is a poor reason if I do say so.

These extra flags have an additional benefit. They make explicit to the user level what additional semantics are switched on, rather than hiding them as implicit side affects of the cpu_exclusive configuration.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#27 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Quoted 3 lines · On Tue, 27 Sep 2005 08:49:05 -0700 Paul Jackson <[email protected]> wrote:

Instead of doing the impossible and trying to mark all the children of C as meter_cpu all at same instant in time, I should just mark the parent cpuset C, one time.

I agree with your idea.
Marking the parent cpuset C instead of marking each child seems to make things quite easier.

Quoted 3 lines

Then if C is so marked, its -children- now
have the meter_cpu_* files, which can default at that instant in time to providing (1/N) of the cpu to each of the N children of C.

Maybe the default value of meter_cpu_guar (guarantee) could be 0 in this design, since guarantee 0 means no guarantee. Resources can even be allocated without guarantee.

Quoted 6 lines

I would say that C can only be marked meter_cpu if:
* C is already marked cpu_exclusive.
* All of its children have the same 'cpus' setting as C.
* Any new child of C created after C is marked meter_cpu will automatically start with the same 'cpus' setting as C, and with the meter_cpu_* files.

Quoted 8 lines

* It is prohibited to change the 'cpus' of any cpuset whose parent is marked meter_cpu.
* Changing the 'cpus' of a cpuset such as C that is itself marked meter_cpu will instantly change the 'cpus' of each of its children.
* It is prohibited to turn on the cpu_exclusive flag of a cpuset whose parent is marked meter_cpu, or to turn off the cpu_exclusive flag on a cpuset that is itself marked meter_cpu.

These seem good for me.

Quoted 30 lines

The metered children of C may have their own children in turn, which may have cpus any subset of the cpus in C, but which cannot be marked cpu_exclusive or meter_cpu.

Borrowing your fine art work, and modifying it slightly, this looks like:

      +-----------------------------------+
      |                                   |
   CPUSET 0                            CPUSET 1 (aka 'C')
   sched domain A                      sched domain B
   cpus: 0, 1                          cpus: 2, 3
   cpu_exclusive=1                     cpu_exclusive=1
   meter_cpu=0                         meter_cpu=1
                                          |
                         +----------------+----------------+
                         |                |                |
                      CPUSET 1a        CPUSET 1b        CPUSET 1c
                      cpus: 2, 3       cpus: 2, 3       cpus: 2, 3
                      cpu_exclusive=0  cpu_exclusive=0  cpu_exclusive=0
                      meter_cpu=0      meter_cpu=0      meter_cpu=0
                      meter_cpu_*      meter_cpu_*      meter_cpu_*
                         |
            +------------+------------+
            |                         |
         CPUSET 2a                CPUSET 2b
         cpus: 2                  cpus: 3
         meter_cpu=0              meter_cpu=0
         cpu_exclusive=0          cpu_exclusive=0
Quoted 2 lines

Note here that marking C (CPUSET 1) as meter_cpu exposes the meter_cpu_* files in the children of C.

If we split the cpus {2, 3} into {2} and {3} by creating CPUSET 2a and CPUSET 2b, the guarantee assigned to CPUSET 1a might not be satisfied. For example, the maximum cpu resource usage of tasks in CPUSET 2a should essentially be 50% because tasks in CPUSET 2a can only use the half number of cpus. If we set meter_cpu_guar=60% on CPUSET 1a, the problem might occur. Maybe (or maybe not), we can leave this problem as it is.

Quoted 8 lines
Quoted 1 line

Is it prohibited for any decendant of C's children to set meter_cpu=1 ?

Yes, I presume so, and made up my new rules above assuming that. It is definitely worth an effort in my opinion to allow creating nested ordinary (not metered) cpusets below the children of C, but I am guessing it would be too hard to try to allow nesting of metered cpusets below metered cpusets. If you have a mind to try that however, I am more than willing to listen to your proposal.

Probably allowing nested metered cpusets is too hard both to implement and to use. So far, I woundn't like to implement it.

Quoted 7 lines

Therefore I propose some new cpuset flags:
* 'sched_domain' to mark sched domains (now done by the cpu_exclusive flag),
* 'kernel_memory' to mark the constraints on GFP_KERNEL allocations,
* 'oom_killer' to mark the constraints on oom killing,
* your 'meter_cpu' flag to mark a set of metered cpus, and
* your 'meter_mem' flag to mark a set of metered mems.

This seems very interesting. Tasks enclosed by a certain cpuset may want to change the behavior of oom_killer.

Signature · 1 line
KUROSAWA, Takahiro
#28 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Quoted 2 lines · Takahiro-san

Maybe the default value of meter_cpu_guar (guarantee) could be 0 in this design, since guarantee 0 means no guarantee.

Ah yes - that is a better default. Excellent.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#29 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

In reply to KUROSAWA Takahiro

Takahiro-san, replying to pj:

Quoted 14 lines
Quoted 14 lines
                         |
                      CPUSET 1a
                      cpus: 2, 3
                      cpu_exclusive=0
                      meter_cpu=0
                      meter_cpu_*
                         |
            +------------+------------+
            |                         |
         CPUSET 2a                CPUSET 2b
         cpus: 2                  cpus: 3
         meter_cpu=0              meter_cpu=0
         cpu_exclusive=0          cpu_exclusive=0
Quoted 8 lines
Quoted 2 lines

Note here that marking C (CPUSET 1) as meter_cpu exposes the meter_cpu_* files in the children of C.

If we split the cpus {2, 3} into {2} and {3} by creating CPUSET 2a and CPUSET 2b, the guarantee assigned to CPUSET 1a might not be satisfied. For example, the maximum cpu resource usage of tasks in CPUSET 2a should essentially be 50% because tasks in CPUSET 2a can only use the half number of cpus.

Ah, yes, this could be difficult and not worth doing.

It might help if I stated more of what I mean, which I didn't before.

I intended that all tasks in the combination of cpusets 1a, 2a, and 2b would collectively be allowed what ever percentage of cpu cycles the meter_cpu_* files in cpuset 1a prescribed. I did not intend to suggest any particular balance between these tasks in 1a, 2a and 2b would be enforced. In particular, I did not expect for anything like a 50%
split between the tasks in 2a and 2b to be enforced. For the purposes
of your cpu controller, just treat the entire set of tasks in all three of these cpusets as one pool, governed by one meter_cpu_* setting, just as if all these tasks were in cpuset 1a, and as if cpusets 2a and 2b didn't exist.

This might be easier to do - I don't know. If this is still hard, then I guess my dream of allowing metered cpusets to have child cpusets with a "proper subset" of CPUs (strictly fewer CPUs) is too hard to do now. In the abstract, this seems like a serious limitation to me. But practically speaking, it might not be a big problem. I don't know.

If it is still hard, even this way, it may just have to wait, until someone needs it bad enough to find a way. If we have layed a good foundation and allowed for this possibility in our architecture, then perhaps we will have done all we can do for now.

Quoted 2 lines

Probably allowing nested metered cpusets is too hard both to implement and to use. So far, I woundn't like to implement it.

A wise choice.

Quoted 2 lines

This seems very interesting. Tasks enclosed by a certain cpuset may want to change the behavior of oom_killer.

Yes - I think so too.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#30 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Quoted 10 lines · On Wed, 28 Sep 2005 00:08:39 -0700 Jackson-san wrote:
Quoted 5 lines

If we split the cpus {2, 3} into {2} and {3} by creating CPUSET 2a and CPUSET 2b, the guarantee assigned to CPUSET 1a might not be satisfied. For example, the maximum cpu resource usage of tasks in CPUSET 2a should essentially be 50% because tasks in CPUSET 2a can only use the half number of cpus.

Ah, yes, this could be difficult and not worth doing.

It might help if I stated more of what I mean, which I didn't before.

Quoted 10 lines

I intended that all tasks in the combination of cpusets 1a, 2a, and 2b would collectively be allowed what ever percentage of cpu cycles the meter_cpu_* files in cpuset 1a prescribed. I did not intend to suggest any particular balance between these tasks in 1a, 2a and 2b would be enforced. In particular, I did not expect for anything like a 50%
split between the tasks in 2a and 2b to be enforced. For the purposes
of your cpu controller, just treat the entire set of tasks in all three of these cpusets as one pool, governed by one meter_cpu_* setting, just as if all these tasks were in cpuset 1a, and as if cpusets 2a and 2b didn't exist.

This seems good for me.
I'd like to make sure that tasks in CPUSET 2a and CPUSET 2b actually have the cpumask of CPUSET 1a. Is this correct?

If it's correct, a small hack to cpuset_cpus_allowed() would only be needed to do this.

Quoted 1 line

This might be easier to do - I don't know.

This is quite easier. Thanks for your clarifying the idea.

Signature · 1 line
KUROSAWA, Takahiro
#31 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Quoted 3 lines · Takahiro-san

This seems good for me.
I'd like to make sure that tasks in CPUSET 2a and CPUSET 2b actually have the cpumask of CPUSET 1a. Is this correct?

Oh I think not. My primary motivation in pushing on this point of the design was to allow CPUSET 2a and 2b to have a smaller cpumask than CPUSET 1a. This is used for example to allow a job that is running in 1a to setup two child cpusets, 2a and 2b, to run two subtasks that are constrained to smaller portions of the CPUs allowed to the job in 1a.

How would hacking cpuset_cpus_allowed() help here?

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#32 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Quoted 10 lines · On Wed, 28 Sep 2005 09:49:32 -0700 Jackson-san wrote:
Quoted 3 lines

This seems good for me.
I'd like to make sure that tasks in CPUSET 2a and CPUSET 2b actually have the cpumask of CPUSET 1a. Is this correct?

Oh I think not. My primary motivation in pushing on this point of the design was to allow CPUSET 2a and 2b to have a smaller cpumask than CPUSET 1a. This is used for example to allow a job that is running in 1a to setup two child cpusets, 2a and 2b, to run two subtasks that are constrained to smaller portions of the CPUs allowed to the job in 1a.

Maybe I still misunderstand your idea.
The guarantee assigned to CPUSET 1a might not be satisfied if tasks are attached to CPUSET 2a only and no tasks are attached to CPUSET 1a nor CPUSET 2b. Does your idea leave as it is because the user sets up CPUSETs like that?

Leaving that situation as it is seems enough feasible as one option. Another option is to prohibit CPUSET 2a/2b from having smaller portions of CPUs that is assigned to CPUSET 1a.

Quoted 1 line

How would hacking cpuset_cpus_allowed() help here?

I was going to modify cpuset_cpus_allowed(CPUSET 2a/2b) as returning cpumask of CPUSET 1a, if tasks in CPUSET 2a and CPUSET 2b have the cpumask of CPUSET 1a.

BTW, thank you for fixing the [PATCH] mail that I had sent in wrong format.

Signature · 1 line
KUROSAWA, Takahiro
#33 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Quoted 1 line · Takahiro-san

Maybe I still misunderstand your idea.

I will have to wait a day before responding - sorry. My boss has me doing something more urgent.

Quoted 1 line

thank you for fixing the [PATCH]

You are very welcome.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#34 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

In reply to KUROSAWA Takahiro
Quoted 7 lines · KUROSAWA Takahiro
Quoted 6 lines

Oh I think not. My primary motivation in pushing on this point of the design was to allow CPUSET 2a and 2b to have a smaller cpumask than CPUSET 1a. This is used for example to allow a job that is running in 1a to setup two child cpusets, 2a and 2b, to run two subtasks that are constrained to smaller portions of the CPUs allowed to the job in 1a.

Quoted 5 lines

Maybe I still misunderstand your idea.
The guarantee assigned to CPUSET 1a might not be satisfied if tasks are attached to CPUSET 2a only and no tasks are attached to CPUSET 1a nor CPUSET 2b. Does your idea leave as it is because the user sets up CPUSETs like that?

Hi Takahiro-san

It seems to me that this "guarantee" can only be guaranteed if the owner of cpuset 1a:
-runs enough tasks to use all the cpus of cpuset 1a
-does not tamper with the scheduler decisions by creating other cpusets inside cpuset 1a, or by using sched_setaffinity().

Outside of this, he accepts to get less cpu than "guaranteed". Sounds acceptable to me.

Simon.

#35 lore raw

Re: [ckrm-tech] Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

Good reply, Simon. Thank-you.

My apologies for the delay in responding, Takahiro-san.

As I think Simon was saying, we can only guarantee to make available to an application a certain amount of CPU cycles. We cannot guarantee that the application will use them. Essentially, we can only guarantee that some no -other- application will use them.

If a job of several tasks is running on a cpuset containing multiple CPUs, and is guaranteed a certain percentage of those CPUs and limited to some other percentage, then you've got a set of tasks that altogether cannot use more than so much of those CPUs (the meter_cpu_limit) and that will always have at least a certain amount of those CPUs free and available for its use (the meter_cpu_guarantee).

If that job uses child cpusets to force some of its tasks on one of those CPUs, and some other of its tasks on another of those CPUs, that makes no difference whatsoever to the guarantees and limits in the previous paragraph. Those child cpusets should not have any settings for m->guar or m->lim. Your code must not just look in the current tasks cpuset for guarantees and limits, because there might not be any meter settings there. Rather it must look up the cpuset hierarchy, to find the nearest ancestor cpuset that has meter data.

Indeed, this might be the best reason to NOT do what I am suggesting.

With the current simple locking of cpusets (one global cpuset_sem), and even with the improvements suggested by Roman that I hope soon to writeup (two global locks - a semaphore and a spinlock), it might be too expensive to do this now. I am no scheduler guru, but I am pretty sure that we do not want to take multiple global semaphores for each task we examine on the hot path of the scheduler.

We probably need someone with more locking expertise than I have (so far at least) to refine the cpuset locking enough to make this practical.

So I think I have just talked myself out of asking you to allow a metered cpuset, such as 1a in our example, to be split into child cpusets. Even if we could reach a shared understanding of what that meant, it would be too slow.

So - metered cpusets may not have child cpusets, until someone wants it enough to figure out how to make the locking efficient.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#36 lore raw

Ok to change cpuset flags for sched domains? (was [PATCH 1/3] CPUMETER ...)

In reply to Paul Jackson

Dinikar,

How much grief will it cause you if I make the following incompatible change to the special boolean files in each cpuset directory?

I think I goofed in encouraging you to overload "cpu_exclusive" with defining dynamic scheduler domains. I should have asked for a separate flag to be added for that, say "sched_domain", which would require "cpu_exclusive=1" as a precondition. Other attributes that require cpu_exclusive or mem_exclusive are showing up, and it makes more sense for each of them to get their own boolean, and leave the "*_exclusive" flags to specify just the exclusive (no overlap with sibling) attribute.

I don't have much code built on top of this yet, so it won't matter much to me. But I worry about the affect that such an incompatible change would have on your user code.

Here is the proposed change, as I first described it in Takahiro-san's thread on cpumeter:

=====================================================================

The above proposal makes it more obvious than ever that I am starting to overload the meaning of cpu_exclusive and mem_exclusive perhaps a bit too much.

One or the other of the two *_exclusive flags should be required preconditions for some of these special properties (sched domains, GFP_KERNEL memory allocation confinement, oom killer confinement, cpu metering and memory metering), but perhaps actually enabling any of these special properties should be an additional and distinct choice.

Therefore I propose some new cpuset flags:
* 'sched_domain' to mark sched domains (now done by the cpu_exclusive flag),
* 'kernel_memory' to mark the constraints on GFP_KERNEL allocations,
* 'oom_killer' to mark the constraints on oom killing,
* your 'meter_cpu' flag to mark a set of metered cpus, and
* your 'meter_mem' flag to mark a set of metered mems.

Each of these new flags would require the appropriate cpu_exclusive or mem_exclusive flag on the same cpuset to already be set, but just setting the *_exclusive flags by themselves would not be enough to get you the special behaviour. You would also have to set the appropriate one of these new flags.

So, for example, the condition to define a sched domain would change, from just being the lowest level cpuset marked cpu_exclusive (or the left over CPUs not marked exclusive), to being both that -and- having its "sched_domain" flag marked True (or being the left over CPUs, again).

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#37 lore raw

Re: Ok to change cpuset flags for sched domains? (was [PATCH 1/3] CPUMETER ...)

I have been wanting to follow the cpumeter discussion more closely, but currently am tied up. I hope to have more time towards the end of this week.

I had a few queries below, though

Quoted 5 lines · Paul Jackson

Dinikar,

How much grief will it cause you if I make the following incompatible change to the special boolean files in each cpuset directory?

Quoted 8 lines

I think I goofed in encouraging you to overload "cpu_exclusive" with defining dynamic scheduler domains. I should have asked for a separate flag to be added for that, say "sched_domain", which would require "cpu_exclusive=1" as a precondition. Other attributes that require cpu_exclusive or mem_exclusive are showing up, and it makes more sense for each of them to get their own boolean, and leave the "*_exclusive" flags to specify just the exclusive (no overlap with sibling) attribute.

One of the reasons for overloading the cpu_exclusive flag was to ensure that the rebalance code does not try to pull tasks unnecessarily

With the scheme that you are proposing that is a possibility if you turn on the cpu_exclusive and meter_cpu for example and not turn on sched_domain. Is there a reason why we would want to have exclusive cpusets not attached to sched domains at all?

I am not entirely convinced that we can compare sched_domains and meter_cpus.

However I am still open if there is a convincing reason to have exclusive cpusets that dont form sched domains.

-Dinakar

#38 lore raw

Re: [ckrm-tech] Re: Ok to change cpuset flags for sched domains? (was [PATCH 1/3] CPUMETER ...)

Dinakar, responding to pj:

Quoted 4 lines
Quoted 2 lines

I think I goofed in encouraging you to overload "cpu_exclusive" with defining dynamic scheduler domains.

I am not entirely convinced ...

You're right. It's correct the way it is.

The placement of dynamic sched domains does not cause some user visible change in behaviour that it makes sense for the user to want to turn on for some cpusets and off for others.

Rather it is a performance optimization for scheduler scalability, and the resulting domains must cover all the CPUs in the system.

So the placement of sched domains should not be a user option, and should be done for all cpu_exclusive domains as it is done now.

Good.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#39 lore raw

PATCH 1/1 fix memory leak on reading cpuset files after seeking beyond eof

In reply to KUROSAWA Takahiro
Quoted 7 lines · On Tue, 27 Sep 2005 20:39:02 +0900 KUROSAWA Takahiro <[email protected]>…
Quoted 3 lines · On Tue, 27 Sep 2005 01:37:51 -0700 Paul Jackson <[email protected]> wrote:

The above reminds me of a bug fix that you provided in the previous patch set, for the case *ppos >= eof. I wonder if we have duplicated code here.

Ah, yes, we need to fix the bug in the cpuset code introduced by me...

I fixed the bug. Sorry for my previous incomplete patch. The following patch is against linux-2.6.14-rc2.

kernel/cpuset.c +6 -5
@@ -969,7 +969,7 @@ static ssize_t cpuset_common_file_read(s
 	ssize_t retval = 0;
 	char *s;
 	char *start;
-	size_t n;
+	ssize_t n;
 
 	if (!(page = (char *)__get_free_page(GFP_KERNEL)))
 		return -ENOMEM;
@@ -999,12 +999,13 @@ static ssize_t cpuset_common_file_read(s
 	*s++ = '\n';
 	*s = '\0';
 
-	/* Do nothing if *ppos is at the eof or beyond the eof. */
-	if (s - page <= *ppos)
-		return 0;
-
 	start = page + *ppos;
 	n = s - start;
+
+	/* Do nothing if *ppos is at the eof or beyond the eof. */
+	if (n <= 0)
+		goto out;
+
 	retval = n - copy_to_user(buf, start, min(n, nbytes));
 	*ppos += retval;
 out:
#40 lore raw

Re: [PATCH][BUG] fix memory leak on reading cpuset files after seeking beyond eof

Quoted 2 lines · Takahiro-san

I fixed the bug. Sorry for my previous incomplete patch. The following patch is against linux-2.6.14-rc2.

The patch looks perfect. But it needs a good opening comment and Subject.

The header comment of the patch needs to be just the words that should go in the source control (git) log comment, for someone to read to understand this patch, years from now, when they have long since forgotten our little email thread here. Andrew Morton's and Linus Torvalds's automatic patch handling tools just pull that comment out, unedited, and put it into the log of changes for the kernel/cpuset.c file, when they accept a patch.

The Subject needs to be a good name for the patch. I always use a short phrase starting with the word "cpuset" if it is a cpuset patch. If it is a fix, Andrew seems to enjoy having the word 'fix' as the last word of the Subject. The tools of Andrew and Linus will take that exact Subject line, trim off whatever is inside the '[...]', replace spaces with hyphens '-', and use that for the name of the patch file.

Directly include Andrew or Linus in the Cc list, if you want them to apply the patch. They might notice the patch even if you don't Cc them, but it is less certain in that case.

Ah - one cute trick - I hand edited the pathname of the file that shows in the first two lines of the actual patch, to show the release of Linux "linux-2.6.14-rc2" you worked against. I hope I didn't break the patch by hand editing it. If I were good, I would send the patch just to myself first, and make sure it still applied. But I am lazy and over confident.

See also the following three documents, on how to submit patches to the Linux kernel:

1) Documentation/SubmittingPatches, a file in the kernel source
2) Andrew Morton's "The Perfect Patch", available at:
http://www.zip.com.au/~akpm/linux/patches/stuff/tpp.txt
3) Jeff Garzik's "Linux kernel patch submission format", at:
http://linux.yyz.us/patch-format.html

I will resend this patch with a suitable header comment and Subject in a few minutes. You will easily see what I mean when you see my next message.

The header comment on what I will resend now will be quite short, as this is an easy patch to explain. The header comments on most of the patches I send are much longer -- probably too long, but Andrew seems reluctant to complain about comments that are too long. He is much more likely to complain if they are too short.

Excellent work. Thank-you.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#41 lore raw

PATCH 1/1 cpuset read past eof memory leak fix

In reply to KUROSAWA Takahiro

Don't leak a page of memory if user reads a cpuset file past eof.

kernel/cpuset.c +6 -5
@@ -969,7 +969,7 @@ static ssize_t cpuset_common_file_read(s
 	ssize_t retval = 0;
 	char *s;
 	char *start;
-	size_t n;
+	ssize_t n;
 
 	if (!(page = (char *)__get_free_page(GFP_KERNEL)))
 		return -ENOMEM;
@@ -999,12 +999,13 @@ static ssize_t cpuset_common_file_read(s
 	*s++ = '\n';
 	*s = '\0';
 
-	/* Do nothing if *ppos is at the eof or beyond the eof. */
-	if (s - page <= *ppos)
-		return 0;
-
 	start = page + *ppos;
 	n = s - start;
+
+	/* Do nothing if *ppos is at the eof or beyond the eof. */
+	if (n <= 0)
+		goto out;
+
 	retval = n - copy_to_user(buf, start, min(n, nbytes));
 	*ppos += retval;
 out:
Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#42 lore raw

Re: [PATCH] cpuset read past eof memory leak fix

Applied.

Just a non-technical note: the sign-offs makes me suspect the patch (or an earlier version of it) may have been written by Kurosawa-san. However, it wasn't clear, so I didn't make that change, and so it now ends up being in git with Paul as the author.

If that was wrong, please for the future keep authorship intact by having a "From: A U Thor <[email protected]>" at the head of the description, which will make authorship clear and allow the tools to pick that up.

Linus

Quoted 9 lines · Paul Jackson

Don't leak a page of memory if user reads a cpuset file past eof.

kernel/cpuset.c +0 -0
@@ -969,7 +969,7 @@ static ssize_t cpuset_common_file_read(s

...

#43 lore raw

Re: [PATCH] cpuset read past eof memory leak fix

Quoted 2 lines · Linus

please for the future keep authorship intact by having a "From: ...

You guessed right. Your non-technical note was applicable.

Andrew - why doesn't your "The perfect patch" mention this?
http://www.zip.com.au/~akpm/linux/patches/stuff/tpp.txt

Linus - would you like a patch to Documentation/SubmittingPatches
that mentions this?

Hmmm ... I don't even see the "Acked-by" in these patch guides either. Probably I should include that in SubmittingPatches as well.

Too bad that first line doesn't start "Author:" instead of "From:". Oh well - I see Andrew already suggested that, and you declined. (You should'a listened to him ;).

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#44 lore raw

Re: [PATCH] cpuset read past eof memory leak fix

Quoted 12 lines · Paul Jackson
Quoted 2 lines · Linus

please for the future keep authorship intact by having a "From: ...

You guessed right. Your non-technical note was applicable.

Andrew - why doesn't your "The perfect patch" mention this?
http://www.zip.com.au/~akpm/linux/patches/stuff/tpp.txt

Linus - would you like a patch to Documentation/SubmittingPatches
that mentions this?

Quoted 6 lines

Hmmm ... I don't even see the "Acked-by" in these patch guides either. Probably I should include that in SubmittingPatches as well.

Too bad that first line doesn't start "Author:" instead of "From:". Oh well - I see Andrew already suggested that, and you declined. (You should'a listened to him ;).

Paul, did you see Linus's email on the canonical patch format? (I can understand if you missed it. :)

http://www.ussg.iu.edu/hypermail/linux/kernel/0504.0/1865.html

Signature · 1 line
~Randy
#45 lore raw

Re: [ckrm-tech] Re: [PATCH] cpuset read past eof memory leak fix

Quoted 1 line

Paul, did you see Linus's email on the canonical patch format?

I had just found it in my archives, a few minutes earlier. Thanks!

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#46 lore raw

Re: [PATCH] cpuset read past eof memory leak fix

In reply to Paul Jackson
Quoted 4 lines · Paul Jackson

Too bad that first line doesn't start "Author:" instead of "From:". Oh well - I see Andrew already suggested that, and you declined. (You should'a listened to him ;).

The "From:" rule has been implicit in my tools for a _loong_ time, and switching to "Author:" would just break the tools for no actual technical gain. Not just the tools, either, since Andrew isn't the only one who follows that From: rule - it would break "people".

So you'd have to make the tools accept _both_ "From:" and "Author:", and I personally prefer an _unambiguously_ slightly misnamed thing over some "either X or Y" where X is slightly misnamed but accepted because it's the one more commonly used.

It's like the unix "creat()" system call. Sure, it would make more sense to add the "e", but it wouldn't actually really _help_ anybody.

As to documenting the "From:" thing - yes, we probably should. It's quite commonly used.

Linus

#47 lore raw

Re: [PATCH 1/3] CPUMETER: add cpumeter framework to the CPUSETS

In reply to KUROSAWA Takahiro
Quoted 3 lines · Takahiro-san

but the cpu controller code will add some if statements

From what I read, I suspect that the more interesting question

will be how many additional cache lines and memory pages are touched, on the most common paths through the scheduler.

However, guessing about performance is a risky business. Benchmarks will be most helpful for a significant scheduler patch such as this one.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#48 lore raw

PATCH 2/3 CPUMETER: CPU resource controller

In reply to Paul Jackson

This patch adds CPU resource controller. It enables us to control CPU time percentage of tasks grouped by the cpu_rc structure. It controls time_slice of tasks based on the feedback of difference between the target value and the current usage in order to control the percentage of the CPU usage to the target value.

include/linux/cpu_rc.h new file +65 -0
@@ -0,0 +1,65 @@
+#ifndef _LINUX_CPU_RC_H_
+#define _LINUX_CPU_RC_H_
+/*
+ *  CPU resource controller interface
+ *
+ *  Copyright 2005 FUJITSU LIMITED
+ *
+ *  This file is subject to the terms and conditions of the GNU General Public
+ *  License.  See the file COPYING in the main directory of the Linux
+ *  distribution for more details.
+ */
+
+#include <linux/config.h>
+#include <linux/sched.h>
+#include <linux/cpuset.h>
+
+#ifdef CONFIG_CPU_RC
+
+#ifdef __KERNEL__
+void cpu_rc_init(void);
+unsigned int cpu_rc_scale_timeslice(task_t *tsk, unsigned int slice);
+void cpu_rc_account(task_t *tsk, unsigned long now);
+void cpu_rc_collect_hunger(task_t *tsk);
+
+static inline void cpu_rc_record_activated(task_t *tsk, unsigned long now)
+{
+	tsk->last_activated = now;
+}
+
+static inline void cpu_rc_record_allocation(task_t *tsk,
+					    unsigned int slice,
+					    unsigned long now)
+{
+	if (slice == 0) {
+		/* minimal allocated time_slice is 1 (see sched_fork()). */
+		slice = 1;
+	}
+
+	tsk->last_slice = slice;
+	tsk->ts_alloced = now;
+}
+#endif /* __KERNEL__ */
+
+#else /* CONFIG_CPU_RC */
+
+#ifdef __KERNEL__
+static inline void cpu_rc_init(void) {}
+static inline void cpu_rc_account(task_t *tsk, unsigned long now) {}
+static inline void cpu_rc_collect_hunger(task_t *tsk) {}
+static inline void cpu_rc_record_activated(task_t *tsk, unsigned long now) {}
+static inline void cpu_rc_record_allocation(task_t *tsk,
+					    unsigned int slice,
+					    unsigned long now) {}
+
+static inline unsigned int cpu_rc_scale_timeslice(task_t *tsk,
+						  unsigned int slice)
+{
+	return slice;
+}
+#endif /* __KERNEL__ */
+
+#endif /* CONFIG_CPU_RC */
+
+#endif /* _LINUX_CPU_RC_H_ */
+
include/linux/sched.h +5 -0
@@ -769,6 +769,11 @@ struct task_struct {
 	nodemask_t mems_allowed;
 	int cpuset_mems_generation;
 #endif
+#ifdef CONFIG_CPU_RC
+	unsigned int last_slice;
+	unsigned long ts_alloced;
+	unsigned long last_activated;
+#endif
 	atomic_t fs_excl;	/* holding fs exclusive resources */
 };
 
init/Kconfig +8 -0
@@ -247,6 +247,14 @@ config CPUMETER
 
 	  Say N if unsure.
 
+config CPU_RC
+	bool "CPU resource controller"
+	help
+	  This options will let you control the CPU resource by scaling 
+	  the timeslice allocated for each tasks.
+
+	  Say N if unsure.
+
 menuconfig EMBEDDED
 	bool "Configure standard kernel features (for small systems)"
 	help
init/main.c +2 -1
@@ -42,6 +42,7 @@
 #include <linux/writeback.h>
 #include <linux/cpu.h>
 #include <linux/cpuset.h>
+#include <linux/cpu_rc.h>
 #include <linux/efi.h>
 #include <linux/unistd.h>
 #include <linux/rmap.h>
@@ -524,7 +525,7 @@ asmlinkage void __init start_kernel(void
 	proc_root_init();
 #endif
 	cpuset_init();
-
+	cpu_rc_init();
 	check_bugs();
 
 	acpi_early_init(); /* before LAPIC and SMP init */
kernel/Makefile +1 -0
@@ -20,6 +20,7 @@ obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
 obj-$(CONFIG_KEXEC) += kexec.o
 obj-$(CONFIG_COMPAT) += compat.o
 obj-$(CONFIG_CPUSETS) += cpuset.o
+obj-$(CONFIG_CPU_RC) += cpu_rc.o
 obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_IKCONFIG_PROC) += configs.o
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
kernel/cpu_rc.c new file +233 -0

233 lines not on this page · raw message

kernel/sched.c +29 -2
@@ -41,6 +41,7 @@
 #include <linux/rcupdate.h>
 #include <linux/cpu.h>
 #include <linux/cpuset.h>
+#include <linux/cpu_rc.h>
 #include <linux/percpu.h>
 #include <linux/kthread.h>
 #include <linux/seq_file.h>
@@ -168,10 +169,17 @@
 
 static unsigned int task_timeslice(task_t *p)
 {
+	unsigned int timeslice;
+
 	if (p->static_prio < NICE_TO_PRIO(0))
-		return SCALE_PRIO(DEF_TIMESLICE*4, p->static_prio);
+		timeslice = SCALE_PRIO(DEF_TIMESLICE*4, p->static_prio);
 	else
-		return SCALE_PRIO(DEF_TIMESLICE, p->static_prio);
+		timeslice = SCALE_PRIO(DEF_TIMESLICE, p->static_prio);
+
+	if (!TASK_INTERACTIVE(p))
+		timeslice = cpu_rc_scale_timeslice(p, timeslice);
+
+	return timeslice;
 }
 #define task_hot(p, now, sd) ((long long) ((now) - (p)->last_ran)	\
 				< (long long) (sd)->cache_hot_time)
@@ -660,6 +668,7 @@ static int effective_prio(task_t *p)
  */
 static inline void __activate_task(task_t *p, runqueue_t *rq)
 {
+	cpu_rc_record_activated(p, jiffies);
 	enqueue_task(p, rq->active);
 	rq->nr_running++;
 }
@@ -1294,6 +1303,7 @@ int fastcall wake_up_state(task_t *p, un
 void fastcall sched_fork(task_t *p, int clone_flags)
 {
 	int cpu = get_cpu();
+	unsigned long now = jiffies;
 
 #ifdef CONFIG_SMP
 	cpu = sched_balance_self(cpu, SD_BALANCE_FORK);
@@ -1330,9 +1340,12 @@ void fastcall sched_fork(task_t *p, int 
 	 * The remainder of the first timeslice might be recovered by
 	 * the parent if the child exits early enough.
 	 */
+	cpu_rc_account(current, now);
 	p->first_time_slice = 1;
 	current->time_slice >>= 1;
 	p->timestamp = sched_clock();
+	cpu_rc_record_allocation(current, current->time_slice, now);
+	cpu_rc_record_allocation(p, p->time_slice, now);
 	if (unlikely(!current->time_slice)) {
 		/*
 		 * This case is rare, it happens when the parent has only
@@ -1390,6 +1403,7 @@ void fastcall wake_up_new_task(task_t * 
 				p->array = current->array;
 				p->array->nr_active++;
 				rq->nr_running++;
+				cpu_rc_record_activated(p, jiffies);
 			}
 			set_need_resched();
 		} else
@@ -1440,16 +1454,21 @@ void fastcall sched_exit(task_t * p)
 {
 	unsigned long flags;
 	runqueue_t *rq;
+	unsigned long now = jiffies;
 
 	/*
 	 * If the child was a (relative-) CPU hog then decrease
 	 * the sleep_avg of the parent as well.
 	 */
 	rq = task_rq_lock(p->parent, &flags);
+	cpu_rc_account(p, now);
 	if (p->first_time_slice) {
+		cpu_rc_account(p->parent, now);
 		p->parent->time_slice += p->time_slice;
 		if (unlikely(p->parent->time_slice > task_timeslice(p)))
 			p->parent->time_slice = task_timeslice(p);
+		cpu_rc_record_allocation(p->parent,
+					 p->parent->time_slice, now);
 	}
 	if (p->sleep_avg < p->parent->sleep_avg)
 		p->parent->sleep_avg = p->parent->sleep_avg /
@@ -2487,6 +2506,7 @@ void scheduler_tick(void)
 	runqueue_t *rq = this_rq();
 	task_t *p = current;
 	unsigned long long now = sched_clock();
+	unsigned long jnow = jiffies;
 
 	update_cpu_clock(p, rq, now);
 
@@ -2521,6 +2541,9 @@ void scheduler_tick(void)
 			p->time_slice = task_timeslice(p);
 			p->first_time_slice = 0;
 			set_tsk_need_resched(p);
+#ifdef CONFIG_CPU_RC
+			/* XXX  need accounting even for rt_task? */
+#endif
 
 			/* put it at the end of the queue: */
 			requeue_task(p, rq->active);
@@ -2530,9 +2553,12 @@ void scheduler_tick(void)
 	if (!--p->time_slice) {
 		dequeue_task(p, rq->active);
 		set_tsk_need_resched(p);
+		cpu_rc_account(p, jnow);
 		p->prio = effective_prio(p);
 		p->time_slice = task_timeslice(p);
 		p->first_time_slice = 0;
+		cpu_rc_record_allocation(p, p->time_slice, jnow);
+		cpu_rc_record_activated(p, jnow);
 
 		if (!rq->expired_timestamp)
 			rq->expired_timestamp = jiffies;
@@ -2891,6 +2917,7 @@ switch_tasks:
 	rcu_qsctr_inc(task_cpu(prev));
 
 	update_cpu_clock(prev, rq, now);
+	cpu_rc_collect_hunger(next);
 
 	prev->sleep_avg -= run_time;
 	if ((long)prev->sleep_avg <= 0)
#49 lore raw

PATCH 3/3 CPUMETER: connect the CPU resource controller to CPUMETER

In reply to Paul Jackson

This patch puts the CPU resource controller into CPUMETER framework. With this patch, we can control the CPU resource from the cpuset filesystem.

init/Kconfig +1 -0
@@ -249,6 +249,7 @@ config CPUMETER
 
 config CPU_RC
 	bool "CPU resource controller"
+	depends on CPUMETER
 	help
 	  This options will let you control the CPU resource by scaling 
 	  the timeslice allocated for each tasks.
kernel/cpu_rc.c +118 -0

124 lines not on this page · raw message

#50 lore raw
In reply to Magnus Damm
Quoted 52 lines · Magnus Damm
Quoted 24 lines · KUROSAWA Takahiro
Quoted 6 lines · On Thu, 8 Sep 2005 05:02:32 -0700 Paul Jackson <[email protected]> wrote:

One of my passions is to avoid special cases across API boundaries.

I am proposing that you don't do subcpusets like this.

Consider the following alternative I will call 'cpuset meters'.

Quoted 15 lines

For each resource named 'R' (cpu and mem, for instance):
* Add a boolean flag 'meter_R' to each cpuset. If set, that R is metered, for the tasks in that cpuset or any descendent cpuset.
* If a cpuset is metered, then files named meter_R_guar, meter_R_lim and meter_R_cur appear in that cpuset to manage R's usage by tasks in that cpuset and descendents.
* There are no additional rules that restrict the ability to change various other cpuset properties such as cpus, mems, cpu_exclusive, mem_exclusive, or notify_on_release, when a cpuset is metered.
* It might be that some (or by design all) resource controllers do not allow nesting metered cpusets. I don't know. But one should (if one has permission) be able to make child cpusets of a metered cpuset, just like one can of any other cpuset.
* A metered cpuset might well have cpus or mems that are not the same as its parent, just like an unmetered cpuset ordinarly does.

Quoted 4 lines

Jackson-san's idea looks good for me because users don't need to create special cpusets (parents of subcpusets or subcpusets). From the point of users, maybe they wouldn't like to create special cpusets.

Yes, from the user POV it must be good to keep the hierarchical model. Ckrm and cpusets both provide a tree with descendents, children and parents. This hierarchical model is very nice IMO and provides a powerful API for the user.

Quoted 5 lines

As for the resource controller that I've posted, it assumes that there are groups of tasks that share the same cpumasks/nodemasks, and that there are no hierarchy in that groups in order to make things easier. I'll investigate how I can attach the resource controller to the cpuset meters.

Subcpusets, compared to cpusets and ckrm, gives the user a flat model. No hierarchy. Limited functionality compared to the hierachical model.

But what I think is important to keep in mind here is that cpusets and subcpusets do very different things. If I understand cpusets correctly, each cpuset may share processors or memory nodes with other cpusets. One task running on a shared processor may starve other cpusets using the same processor. This design works well with cpusets, but for resource controllers that must provide some kind of guarantee, this starvation is unsuitable.

Quoted 3 lines

And we already have an hierarchical alternative: ckrm. But look at the complexity and the amount of code. I believe that the complexity in ckrm mainly comes from the hierarchical model.

I've been trying to find ways to significantly reduce CKRM's kernel
footprint. I recently posted an RFC patch to CKRM-Tech describing a 5000-line reduction:
http://sourceforge.net/mailarchive/forum.php?thread_id=8132624&forum_id=35191
Feedback on the approach presented in the RFC patch would be appreciated.

Quoted 6 lines

Maybe it is possible to have an hierarchical model and keep the framework simple and easy to understand while providing guarantees, I'm not sure. But until that happens, I'm quite happy with a simple, limited flat model.

/ magnus

#51 lore raw
In reply to Paul Jackson

Interesting implementation of resource controls. Cross posting this to ckrm-tech as well. I am sure CKRM folks have something to say...

Any thoughts about how you want to add more resource control features on top of/in addition to this setup. (Such as memory etc)

Quoted 6 lines · Paul Jackson

I'm guessing you do not want such cpusets (the parents of subcpusets) to overlap, because if they did, it would seem to confuse the meaning of getting a fixed proportion of available cpu and memory resources. I was a little surprised not to see any additional checks that cpu_exclusive and mem_exclusive must be set true in these cpusets, to insure non- overlapping cpusets.

I agree with Paul here. You would want to build your controllers on top of exclusive cpusets to keep things sane.

Quoted 5 lines

On the other hand, Dinakar had more work to do than you might, because he needed a complete covering (so had to round up cpus in non exclusive cpusets to form more covering elements). From what I can tell, you don't need a complete covering - it seems fine if some cpus are not managed by this resource control function.

I think it makes more sense to add this functionality directly as part of the existing cpusets instead of creating further leaf cpusets (or subcpusets as you call it) where we can specify resource control parameters. I think that approach would be much more intuitive and simple to work with rather than what you have currently.

-Dinakar

#52 lore raw
Quoted 9 lines · Dinakar Guniguntala
Quoted 7 lines

On the other hand, Dinakar had more work to do than you might, because he needed a complete covering (so had to round up cpus in non exclusive cpusets to form more covering elements). From what I can tell, you don't need a complete covering - it seems fine if some cpus are not managed by this resource control function.

Quoted 5 lines

I think it makes more sense to add this functionality directly as part of the existing cpusets instead of creating further leaf cpusets (or subcpusets as you call it) where we can specify resource control parameters. I think that approach would be much more intuitive and simple to work with rather than what you have currently.

If what subcpusets is doing is slicing cpusets resources, then wouldn't it be more intusive to call them slice0, slice1 etc. under the respective cpuset ?

Thanks
Dipankar

#53 lore raw
Quoted 3 lines · Dipankar

If what subcpusets is doing is slicing cpusets resources, then wouldn't it be more intusive to call them slice0, slice1 etc. under the respective cpuset ?

If we continue with Takahiro-san's design, then I agree that the name 'subcpusets' doesn't have quite the right connotations, and would join you in exploring alternative names.

First it looks to be worth exploring alternatives that make resource control more an attribute of existing cpusets than a new (sub) type of cpuset, as Dinakar and I have both contemplated in earlier posts today.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#54 lore raw
In reply to Dinakar Guniguntala
Quoted 1 line · Dinakar

Cross posting this to ckrm-tech as well.

Good idea - thanks.

Hopefully Takahiro-san and the CKRM folks can reach an understanding on how their two proposals relate.

Signature · 3 lines
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <[email protected]> 1.925.600.0401
#55 lore raw
In reply to Dinakar Guniguntala
Quoted 1 line · Dinakar Guniguntala

Interesting implementation of resource controls. Cross posting this

I second this :)

Browsed a little through the docs/patches... seems to fit very well into a resource management solution (hint CKRM :) than CPUSET (resource isolation).

I can see the usefulness of resource management inside CPUSET. We have had discussions earlier(in lkml and lse-tech) about how CKRM can play inside a CPUSET, and this plays directly into that, providing resource management inside a CPUSET.

The parameters used, guarantee and limit, fits very well into CKRM's shares usage model.

Takahiro-san, How much effort you think will be needed to make this work under CKRM. thanks.

Quoted 42 lines

to ckrm-tech as well. I am sure CKRM folks have something to say...

Any thoughts about how you want to add more resource control features on top of/in addition to this setup. (Such as memory etc)

Quoted 6 lines · Paul Jackson

I'm guessing you do not want such cpusets (the parents of subcpusets) to overlap, because if they did, it would seem to confuse the meaning of getting a fixed proportion of available cpu and memory resources. I was a little surprised not to see any additional checks that cpu_exclusive and mem_exclusive must be set true in these cpusets, to insure non- overlapping cpusets.

I agree with Paul here. You would want to build your controllers on top of exclusive cpusets to keep things sane.

Quoted 5 lines

On the other hand, Dinakar had more work to do than you might, because he needed a complete covering (so had to round up cpus in non exclusive cpusets to form more covering elements). From what I can tell, you don't need a complete covering - it seems fine if some cpus are not managed by this resource control function.

I think it makes more sense to add this functionality directly as part of the existing cpusets instead of creating further leaf cpusets (or subcpusets as you call it) where we can specify resource control parameters. I think that approach would be much more intuitive and simple to work with rather than what you have currently.

-Dinakar

-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

Signature · 4 lines
----------------------------------------------------------------------
    Chandra Seetharaman               | Be careful what you choose....
              - [email protected]   |      .......you may get it.
----------------------------------------------------------------------