CPUSET Proposal
Looks good, but you aren't likely to get much acceptance or testing if it only works on ia64. You need to make a version for i386 as well.
Also, don't send your patch as base64 encode attachment, it makes working with text tools harder.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Stephen Hemminger
Stephen> Looks good, but you aren't likely to get much acceptance or Stephen> testing if it only works on ia64. You need to make a Stephen> version for i386 as well.
Is this true for >8-way machines?
--david
This might be worth comparing notes on with the CKRM folks (cc:'d above).
gerrit
Quoted 56 lines · Simon Derr
Hi,
We have developped a new feature in the Linux kernel, controlling CPU placements, which are useful on large SMP machines, especially NUMA ones. We call it CPUSETS, and we would highly appreciate to know about anyone who would be interested in such a feature. This has been somewhat inspired by the pset or cpumemset patches existing for Linux 2.4.
CPUSETs are lightweight objects in the linux kernel that enable users to partition their multiprocessor machine by creating execution areas. A virtualization layer has been added so it becomes possible to split a machine in terms of CPUs.
Furthermore, HPC applications often need to bind their processes to a specific CPU, and can achieve this by calling
sched_setaffinity()in the recent Linux kernels. But running several HPC applications on a large system will result in several processes running on the same processor. This problem is addressed by the CPUSET mechanism.CPUSETS allow to:
----------------
1/ create sets of CPUs on the system, and bind applications to them2/ translate the masks of CPUs given to
sched_setaffinity()so they stay
inside the set of CPUs. With this mechanism, processors are virtualized, for the use ofsched_setaffinity()and /proc information. Thus, any former application using this syscall to bind processes to processors will work with virtual CPUs without any change.3/ provide a way to create sets of cpus *inside* a set of cpus : hence a
system administrator can partition a system among users, and users can partition their partition among their applications.4/ Change on the fly the execution area of a whole set of processes (to
give more resources to a critical application, for example)....
5/ In the future, probably associate a memory allocation policy (such as local node, or round robin) to a set of cpus.These features have been implemented as a kernel patch for Linux 2.6 and a suite of userland tools.
You can find the associated manpages and a slightly more detailed explanation here: http://www.bullopensource.org/cpuset/
Any feedback, comment or opinion is welcome:
[email protected],
[email protected]Thanks,
Simon and Sylvain.
Interesting ...
I'm still digesting it. However, one of the documents, at:
http://www.bullopensource.org/cpuset/cpuset.html
was painful to read in a web browser, because it was just one big <pre>...</pre> block of text, with rather long lines (over 400 characters in one line) requiring much horizontal scrolling.
So I have reformatted it, using more common html markup. You are welcome to steal my reformatting - it's visible at:
http://www.speakeasy.org/~pj99/cpuset_formatted.html
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373Where's the user level cpuset.h file? I wasn't able to find it so far -- just a few links in the man pages to it, but only on some local file system to which I lack access.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373On the pchange man page, an example starts with:
# pcreate -np 4 --strict
new area created with id 2How does the 'pcreate' invoker know that the new area had an id of "2" ?
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373On Wed, 24 Sep 2003 09:30:44 -0700, Stephen Hemminger <[email protected]> said:
Stephen> Looks good, but you aren't likely to get much acceptance or Stephen> testing if it only works on ia64. You need to make a Stephen> version for i386 as well.
Quoted 1 line · David Mosberger
Is this true for >8-way machines?
x86's architectural limitations are 64x for serial APIC -based machines (e.g. NUMA-Q) and 255x for xAPIC -based machines (no known extant > 32x machines, apparently some kind of non-architectural regression), where the non-power-of-two number of cpus is due to the broadcast ID reserved from an 8-bit interrupt controller ID space. A likely explanation for the current xAPIC limitations is the recommended (publicly documented) physical APIC ID enumeration scheme breaking down for > 32x.
Custom interrupt controllers may exceed these limits, but I don't know of any that have actually been made use of to do so. Though it sucks and very, very badly, x86 is not limited to anything like 8x.
-- wli
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · William Lee Irwin III
Bill> On Wed, 24 Sep 2003 09:30:44 -0700, Stephen Hemminger Bill> <[email protected]> said:
Stephen> Looks good, but you aren't likely to get much acceptance or Stephen> testing if it only works on ia64. You need to make a Stephen> version for i386 as well.
Quoted 1 line · David Mosberger
Quoted 1 line
Is this true for >8-way machines?
Bill> x86's architectural limitations are 64x for serial APIC -based machines Bill> (e.g. NUMA-Q) and 255x for xAPIC -based machines (no known extant > 32x Bill> machines, apparently some kind of non-architectural regression), where Bill> the non-power-of-two number of cpus is due to the broadcast ID reserved Bill> from an 8-bit interrupt controller ID space. A likely explanation for Bill> the current xAPIC limitations is the recommended (publicly documented) Bill> physical APIC ID enumeration scheme breaking down for > 32x.
Bill> Custom interrupt controllers may exceed these limits, but I don't know Bill> of any that have actually been made use of to do so. Though it sucks Bill> and very, very badly, x86 is not limited to anything like 8x.
I wasn't suggesting that x86 is limited to 8-way, I was wondering how many > 8-way x86 Linux machines are actually out there. I wasn't even being facetious---just curious.
--david
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · David Mosberger
Bill> x86's architectural limitations are 64x for serial APIC -based Bill> machines (e.g. NUMA-Q) and 255x for xAPIC -based machines (no Bill> known extant > 32x machines, apparently some kind of Bill> non-architectural regression), where the non-power-of-two Bill> number of cpus is due to the broadcast ID reserved from an Bill> 8-bit interrupt controller ID space. A likely explanation for Bill> the current xAPIC limitations is the recommended (publicly Bill> documented) physical APIC ID enumeration scheme breaking down Bill> for > 32x.
Bill> Custom interrupt controllers may exceed these limits, but I Bill> don't know of any that have actually been made use of to do Bill> so. Though it sucks and very, very badly, x86 is not limited Bill> to anything like 8x.
David> I wasn't suggesting that x86 is limited to 8-way, I was David> wondering how many > 8-way x86 Linux machines are actually David> out there. I wasn't even being facetious---just curious.
Incidentally, the first "big" SMP machine I had access to was some sort of Sequent (S81/10?), with ~12 80386 CPUs (yes, that was a long time ago... ;-).
--david
On Wed, 24 Sep 2003 23:02:34 -0700, William Lee Irwin III <[email protected]> said:
Bill> Custom interrupt controllers may exceed these limits, but I don't know Bill> of any that have actually been made use of to do so. Though it sucks Bill> and very, very badly, x86 is not limited to anything like 8x.
Quoted 3 lines · David Mosberger
I wasn't suggesting that x86 is limited to 8-way, I was wondering how many > 8-way x86 Linux machines are actually out there. I wasn't even being facetious---just curious.
I am not able to get any kind of useful numerical estimate of how mahy machines of that kind are manufactured or sold. I suspect the numbers are meaningful to and kept secret by someone, e.g. marketing ppl. Not to say that I've tried very hard.
-- wli
On Wed, 24 Sep 2003 23:57:10 -0700, David Mosberger <[email protected]> said:
David> I wasn't suggesting that x86 is limited to 8-way, I was
David> wondering how many > 8-way x86 Linux machines are actually David> out there. I wasn't even being facetious---just curious.
Quoted 3 lines · David Mosberger
Incidentally, the first "big" SMP machine I had access to was some sort of Sequent (S81/10?), with ~12 80386 CPUs (yes, that was a long time ago... ;-).
Aha, I've been on S-81's myself. Those definitely predated APIC's. I think they used the SLIC or whatever the precursor to the CSLIC was for an interrupt controller, and I'm sure there's a Sequent historian around somewhere to correct me if my memory's (which wouldn't have been of kernel hacking back then) failed me. =)
-- wli
Quoted 3 lines · David Mosberger
I wasn't suggesting that x86 is limited to 8-way, I was wondering how many > 8-way x86 Linux machines are actually out there. I wasn't even being facetious---just curious.
Well, besides the NUMA-Q, which went up to 60x and is dead now, there are at least the IBM Summit chipset machines. They're sold as 32-ways today on the x445 (that's physical, without hyperthreading). I've personally booted Linux on a 16-way, but I'm know others have booted on the 32-way configuration. Patches for this were posted in the last week by James Cleverdon.
There's also the bigsmp code in the kernel for other P4-based systems that are >8x. I haven't seen any of them yet, but I wouldn't imagine that people would put support in the kernel for hardware that wasn't at least *close* to production.
Signature · 2 lines
Dave Hansen [email protected]
Quoted 3 lines · Gerrit Huizenga
This might be worth comparing notes on with the CKRM folks (cc:'d above).
I went through the cpusets proposal trying to see the commonalities and differences with what CKRM is trying to achieve.
Speaking of CPUs alone:
The way I see it (corrections welcome), CPUSETS has two objectives
- performance isolation for arbitrary groups of processes by constraining them to run on arbitrary sets of CPUs
- gaining performance benefits for a CPUSET by specifying *which* CPUs it can use. This could be helpful in NUMA and Hyperthreaded CPU cases as well as for regular CPUs simply by increasing the chance of cache warmth etc. i.e. the same benefits as seen in sched_affinity, but generalized for groups of processes.
CKRM shares some aspects of the first objective i.e. it also seeks performance isolation for arbitrary groups of processes (classes) too but differs in that it
- uses scheduler modifications to achieve isolation
- has a quantitative measures for how much resource is used
CKRM does not share the second objective i.e. it does not try to control which CPUs are allocated by the kernel schedulers. If such constraints are placed on allocation, whether by sched_affinity, NUMA sched changes or CPUSETs, CKRM will cooperate and operate within that constraint.
For this second objective, CKRM is orthogonal to CPUSETs. For the first one, it is another way of achieving isolation, one that broadly sacrifices stricter guarantees (which are possible by physically isolating CPUs as in CPUSETS) for better load balancing and utilization.
Coming to memory, it wasn't very clear what CPUSETs future plans/objectives are. Does it intend to control which address ranges a CPUSET can allocate from (for NUMA reasons) AND limit how much memory it can consume ? The latter intersects with CKRM objectives, the former doesn't.
Finally, CKRM is dealing with I/O and planning to incorporate inbound and possibly outbound networking into its framework which would appear to be outside the scope of something like CPUSETs.
The user interfaces used by CPUSETs like pexec are quite good. We'd been wanting to develop something similar for CKRM usage as well - allow a user to use a single commandline to specify a job/program and the constraints under which it should operate. The command will then create a class with the specified cpu/mem/io/net constraints, run the job/program within that class and autoclean it when done.
-- Shailabh
Quoted 9 lines · Simon Derr
Quoted 9 lines
Hi,
We have developped a new feature in the Linux kernel, controlling CPU placements, which are useful on large SMP machines, especially NUMA ones. We call it CPUSETS, and we would highly appreciate to know about anyone who would be interested in such a feature. This has been somewhat inspired by the pset or cpumemset patches existing for Linux 2.4.........
<snip>
Quoted 12 lines
Quoted 12 lines
You can find the associated manpages and a slightly more detailed explanation here: http://www.bullopensource.org/cpuset/
Any feedback, comment or opinion is welcome:
[email protected],
[email protected]Thanks,
Simon and Sylvain.
Quoted 3 lines · David Mosberger
Quoted 3 lines
I wasn't suggesting that x86 is limited to 8-way, I was wondering how many > 8-way x86 Linux machines are actually out there. I wasn't even being facetious---just curious.
Quoted 10 lines · Dave Hansen
Well, besides the NUMA-Q, which went up to 60x and is dead now, there are at least the IBM Summit chipset machines. They're sold as 32-ways today on the x445 (that's physical, without hyperthreading). I've personally booted Linux on a 16-way, but I'm know others have booted on the 32-way configuration. Patches for this were posted in the last week by James Cleverdon.
There's also the bigsmp code in the kernel for other P4-based systems that are >8x. I haven't seen any of them yet, but I wouldn't imagine that people would put support in the kernel for hardware that wasn't at least *close* to production.
I figured the ES7000 and x440/x445 were recent enough they'd be fresh in people's mind. Small correction to the NUMA-Q: it was 64x.
http://www-3.ibm.com/software/data/db2/benchmarks/050300.html
-- wli
Quoted 6 lines · Shailabh Nagar
CKRM shares some aspects of the first objective i.e. it also seeks performance isolation for arbitrary groups of processes (classes) too but differs in that it
- uses scheduler modifications to achieve isolation
- has a quantitative measures for how much resource is used
Let me modify that since CPUSETs also provides a coarse-grain (#cpus) quantitative measure.
CKRM allows % of the total resource (cpu ticks, mem pages etc.) to be specified and is consequently at a finer granularity.
Quoted 5 lines · Shailabh Nagar
Coming to memory, it wasn't very clear what CPUSETs future plans/objectives are. Does it intend to control which address ranges a CPUSET can allocate from (for NUMA reasons) AND limit how much memory it can consume ? The latter intersects with CKRM objectives, the former doesn't.
The future plans are still to be decided. The control on memory provided by future versions of CPUSET might be to:
-associate one node on wich to allocate memory to each cpuset -associate one allocation policy (local node first, round robin) to each cpuset.
At first glance, limiting the amount of memory used will probably not be in our objectives. The initial goal of CPUSET was obtaining good performance on NUMA architectures. But this is an open question.
Quoted 3 lines
Finally, CKRM is dealing with I/O and planning to incorporate inbound and possibly outbound networking into its framework which would appear to be outside the scope of something like CPUSETs.
That's right.
Simon.
Quoted 1 line · Simon
The control on memory provided by future versions of CPUSET might be ...
I anticipate that we end up with a kernel "mems_allowed" attribute of a task (or perhaps of a vma), that is quite similar to cpus_allowed.
The mems_allowed field is a multiword (well, one word, until someone ventures above 64 memory nodes) bit vector of node numbers, and indicates on which one _or_more_ nodes memory may be allocated. It controls allocation in mm/page_alloc.c:__alloc_pages(), and related such places. Allocation by the kernel for user address space is distinguished from allocation for the kernel's own needs.
Actually, the mems_allowed field should not be directly in the task (or vma?) struct, but in the shared cpuset struct, referenced from the task
or vma. Simon and Sylvain's cpuset proposal already does this for thecpus_allowed bit vector, moving it from the task struct to the shared cpuset struct referenced from the task struct.
On the larger scale mems_allowed is set administratively (via convenient tools such as batch and job managers) for an entire job or related set of jobs. On the smaller scale, it is set for various tasks (or vmas) of a job, relative to the cpuset that job is executing on. It is usually set from user level code, sometimes by admin utilities, sometimes by system services, sometimes by a numa aware application itself.
There may be a need for kernel or low level library assistance in setting it, for situations such as when a multithreaded scientific program that is tuned for running on an SMP system knows enough to get the number of threads right, but doesn't know enough to place them across nodes optimally. In such cases, one sometimes has to intercede at a relatively low level, behind the applications back, to get the proper distribution of tasks and vmas across cpus and memory nodes for optimal performance.
Additional work is required to handle cases where one runs out of memory on allowed nodes - one might want to steal memory from other nodes, or swap or kill or sleep or grant more nodes to the greedy application, depending on administrative policies. Eventually, the kernel should provide minimal mechanisms in support of each such policy.
The essential attribute of NUMA systems is "non-uniform memory access" (not surprisingly). Getting memory placed is just as essential for optimum performance as getting cpu usage placed. The two must work together.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373