Tiny cpusets -- cpusets for small systems?
A couple of proposals have been made recently by people working Linux on smaller systems, for improving realtime isolation and memory pressure handling:
(1) cpu isolation for hard(er) realtime
http://lkml.org/lkml/2008/2/21/517
Max Krasnyanskiy <[email protected]> [PATCH sched-devel 0/7] CPU isolation extensions
(2) notify user space of tight memory
http://lkml.org/lkml/2008/2/9/144 KOSAKI Motohiro <[email protected]> [PATCH 0/8][for -mm] mem_notify v6
In both cases, some of us have responded "why not use cpusets", and the original submitters have replied "cpusets are too fat" (well, they were more diplomatic than that, but I guess I can say that ;)
I wonder if there might be room for a "tiny cpusets" configuration option:
* provide the same hooks to the rest of the kernel, and
* provide the same syntactic interface to user space, but
* with more limited semantics.
The primary semantic limit I'd suggest would be supporting exactly one layer depth of cpusets, not a full hierarchy. So one could still successfully issue from user space 'mkdir /dev/cpuset/foo', but trying to do 'mkdir /dev/cpuset/foo/bar' would fail. This reminds me of very early FAT file systems, which had just a single, fixed size root directory ;). There might even be a configurable fixed upper limit on how many /dev/cpuset/* directories were allowed, further simplifying the locking and dynamic memory behavior of this apparatus.
Some other features that aren't so easy to implement, and which have less value on small systems, such as notify_on_release, could also be stubbed out and always disabled, simply returning error if requested to be enabled from user space. The recent, chunky piece of code needed to compute dynamic sched domains from the cpuset hierarchy probably admits of a simpler variant in the tiny cpuset configuration.
I suppose it would still be a vfs-based pseudo file system (even embedded Linux still has that infrastructure), except that the vfs operator functions could be simpler, as this would really be just a flat set of cpumask_t's and nodemask_t's at the core of the implementation, not an arbitrarily nested hierarchy of them. See further my comments on cgroups, below.
The rest of the kernel would see no difference ... except that some of the cpuset_*() hooks would return more quickly. This tiny cpuset option would provide the same kernel hooks as are now provided by the defines and inline stubs, in the "#else" to "#endif" half of the "#ifdef CONFIG_CPUSETS" code lines in linux/cpuset.h.
User space would see the same API, except that some valid operations on full cpusets, such as a nested mkdir, would fail on tiny cpusets.
How this extends to cgroups I don't know; for now I suspect that most cgroup module development is motivated by the needs of larger systems, not smaller systems. However, cpusets is now a module client of cgroups, and it is cgroups that now provides cpusets with its interface to the vfs infrastructure. It would seem unfortunate if this relation was not continued with tiny cpusets. Perhaps someone can imagine a tiny cgroups? This might be the most difficult part of this proposal.
Looking at some IA64 sn2 config builds I have laying about, I see the following text sizes for a couple of versions, showing the growth of the cpuset/cgroup apparatus over time:
25933 2.6.18-rc3-mm1/kernel/cpuset.o (Aug 2006) vs. 37823 2.6.25-rc2-mm1/kernel/cgroup.o (Feb 2008) 19558 2.6.25-rc2-mm1/kernel/cpuset.o
So the total has grown from 25933 to 57381 text bytes (note that this is IA64 arch; most arch's will have proportionately smaller text sizes.)
Unfortunately, ideas without code are usually met with the sound of silence, as well they should be. Furthermore, I can promise that I have no time to design or develop this myself; my good employer is quite focused on the other end of things - the big honkin NUMA and cluster systems.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.940.382.4214Quoted 9 lines · Paul Jackson
A couple of proposals have been made recently by people working Linux
on smaller systems, for improving realtime isolation and memory pressure handling:(1) cpu isolation for hard(er) realtime
http://lkml.org/lkml/2008/2/21/517
Max Krasnyanskiy <[email protected]> [PATCH sched-devel 0/7] CPU isolation extensions
Quoted 8 lines
(2) notify user space of tight memory
http://lkml.org/lkml/2008/2/9/144 KOSAKI Motohiro <[email protected]> [PATCH 0/8][for -mm] mem_notify v6In both cases, some of us have responded "why not use cpusets", and the original submitters have replied "cpusets are too fat" (well, they were more diplomatic than that, but I guess I can say that ;)
Having read those threads, it looks to me as though:
- the parts of Max's problem that would be solved by cpusets can be mostly accomplished just via sched_setaffinity()
- Motohiro wants to add a new system-wide API that you would also like to have available on a per-cpuset basis. (Why not just add two access points for the same feature?)
I'm don't think that either of these would be enough to justify big changes to cpusets or cgroups, although eliminating bloat is always a good thing.
Quoted 8 lines
The primary semantic limit I'd suggest would be supporting exactly one layer depth of cpusets, not a full hierarchy. So one could still successfully issue from user space 'mkdir
/dev/cpuset/foo', but trying to do 'mkdir/dev/cpuset/foo/bar' would fail. This reminds me of very early FAT file systems, which had just a single, fixed size root directory ;). There might even be a configurable fixed upper limit on how many/dev/cpuset/*directories were allowed, further simplifying the locking and dynamic memory behavior of this apparatus.
I'm not sure that either of these would make much difference to the overall footprint.
A single layer of cpusets would allow you to simplify validate_change() but not much else.
I don't see how a fixed upper limit on the number of cpusets makes the locking sufficiently simpler to save much code.
Quoted 8 lines
How this extends to cgroups I don't know; for now I suspect that most cgroup module development is motivated by the needs of larger systems, not smaller systems. However, cpusets is now a module client of cgroups, and it is cgroups that now provides cpusets with its interface to the vfs infrastructure. It would seem unfortunate if this relation was not continued with tiny cpusets. Perhaps someone can imagine a tiny cgroups? This might be the most difficult part of this proposal.
If we wanted to go this way, I can imagine a cgroups config option that forces just a single hierarchy, which would allow a bunch of simplifications that would save plenty of text.
Quoted 5 lines
Looking at some IA64 sn2 config builds I have laying about, I see the following text sizes for a couple of versions, showing the growth of the cpuset/cgroup apparatus over time:
Quoted 8 lines
25933 2.6.18-rc3-mm1/kernel/cpuset.o (Aug 2006) vs. 37823 2.6.25-rc2-mm1/kernel/cgroup.o (Feb 2008) 19558 2.6.25-rc2-mm1/kernel/cpuset.oSo the total has grown from 25933 to 57381 text bytes (note that this is IA64 arch; most arch's will have proportionately smaller text sizes.)
On x86_64 they're:
cgroup.o: 17348
cpuset.o: 8533
Paul
Quoted 3 lines · Paul M
I'm don't think that either of these would be enough to justify big changes to cpusets or cgroups, although eliminating bloat is always a good thing.
My "tiny cpuset" idea doesn't so much eliminate bloat, as provide a thin alternative, along side of the existing fat alternative. So far as kernel source goes, it would get bigger, not smaller, with now two CONFIG choices for cpusets, fat or tiny.
The odds are, however, given that one of us has just promised not to code this, and the other of us doesn't figure it's worth it, this idea will not live long. Someone would have to step up from the embedded side with a coded version that saved a nice chunk of memory (from their perspective) to get this off the ground, and no telling whether even that would meet with a warm reception.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.940.382.4214Quoted 10 lines · Paul Jackson
Quoted 3 lines · Paul M
I'm don't think that either of these would be enough to justify big changes to cpusets or cgroups, although eliminating bloat is always a good thing.
My "tiny cpuset" idea doesn't so much eliminate bloat, as provide a thin alternative, along side of the existing fat alternative. So far as kernel source goes, it would get bigger, not smaller, with now two CONFIG choices for cpusets, fat or tiny.
Quoted 7 lines
The odds are, however, given that one of us has just promised not to code this, and the other of us doesn't figure it's worth it, this idea will not live long. Someone would have to step up from the embedded side with a coded version that saved a nice chunk of memory (from their perspective) to get this off the ground, and no telling whether even that would meet with a warm reception.
This has actually been on my TODO list for awhile, though not quite in the way that you outlined in your initial post. A good reason for why cpusets are fat at the moment is largely because there's no isolation or configurability of individual features supported and exposed, leaving one with an all-or-nothing scenario.
Both the SMP and NUMA bits are fairly orthogonal, and I think isolating those and making each configurable would already help trim things down quite a bit (ie, nodemask handling, scheduler domains, etc.). The filesystem bits are not really that heavy comparatively, so rather than working on a tiny cpuset implementation, simply splitting up the existing implementation seems like a much saner approach, and one that can be done incrementally.
While we're on the topic of cpuset reform, co-processors are another thing that cpusets is in pretty good shape to handle (particularly in terms of carving up large grid and dataflow processors and things of that nature, which we do have customer use cases for in embedded space today).
I'll try to follow up to this thread with an initial patch series in the not-too-distant future.
Hi Paul,
Quoted 9 lines
A couple of proposals have been made recently by people working Linux on smaller systems, for improving realtime isolation and memory pressure handling:
(1) cpu isolation for hard(er) realtime
http://lkml.org/lkml/2008/2/21/517
Max Krasnyanskiy <[email protected]> [PATCH sched-devel 0/7] CPU isolation extensions
Quoted 8 lines
(2) notify user space of tight memory
http://lkml.org/lkml/2008/2/9/144 KOSAKI Motohiro <[email protected]> [PATCH 0/8][for -mm] mem_notify v6In both cases, some of us have responded "why not use cpusets", and the original submitters have replied "cpusets are too fat" (well, they were more diplomatic than that, but I guess I can say that ;)
My primary issue with cpusets (from CPU isolation perspective that is) was not the fatness. I did make a couple of comments like "On dual-cpu box I do not need cpusets to manage the CPUs" but that's not directly related to the CPU isolation.
For the CPU isolation in particular I need code like this
int select_irq_affinity(unsigned int irq)
{
cpumask_t usable_cpus;
cpus_andnot(usable_cpus, cpu_online_map, cpu_isolated_map);
irq_desc[irq].affinity = usable_cpus;
irq_desc[irq].chip->set_affinity(irq, usable_cpus);
return 0;
}How would you implement that with cpusets ?
I haven't seen you patches but I'd imagine that they will still need locks and iterators for "Is CPU N isolated" functionality.
So. I see cpusets as a higher level API/mechanism and cpu_isolated_map as lower level mechanism that actually makes kernel aware of what's isolated what's not. Kind of like sched domain/cpuset relationship. ie cpusets affect sched domains but scheduler does not use cpusets directly.
Quoted 5 lines
I wonder if there might be room for a "tiny cpusets" configuration option:
* provide the same hooks to the rest of the kernel, and
* provide the same syntactic interface to user space, but
* with more limited semantics.
Quoted 8 lines
The primary semantic limit I'd suggest would be supporting exactly one layer depth of cpusets, not a full hierarchy. So one could still successfully issue from user space 'mkdir
/dev/cpuset/foo', but trying to do 'mkdir/dev/cpuset/foo/bar' would fail. This reminds me of very early FAT file systems, which had just a single, fixed size root directory ;). There might even be a configurable fixed upper limit on how many/dev/cpuset/*directories were allowed, further simplifying the locking and dynamic memory behavior of this apparatus.
In a foreseeable future 2-8 cores will be most common configuration. Do you think that cpusets are needed/useful for those machines ? The reason I'm asking is because given the restrictions you mentioned above it seems that you might as well just do
taskset -c 1,2,3 app1 taskset -c 3,4,5 app2
Yes it's not quite the same of course but imo covers most cases. That's what we do on 2-4 cores these days, and are quite happy with that. ie We either let the specialized apps manage their thread affinities themselves or use "taskset" to manage the apps.
Quoted 2 lines
User space would see the same API, except that some valid operations on full cpusets, such as a nested mkdir, would fail on tiny cpusets.
Speaking of user-space API. I guess it's not directly related to the tiny-cpusets proposal but rather to the cpusets in general.
Stuff that I'm working on this days (wireless basestations) is designed with the following model:
cpuN - runs soft-RT networking and management code
cpuN+1 to cpuN+x - are used as dedicated engines
ie Simplest example would be
cpu0 - runs IP, L2 and control plane
cpu1 - runs hard-RT MAC
So if CPU isolation is implemented on top of the cpusets what kind of API do you envision for such an app ? I mean currently cpusets seems to be mostly dealing with entire processes, whereas in this case we're really dealing with the threads. ie Different threads of the same process require different policies, some must run on isolated cpus some must not. I guess one could write a thread's pid into cpusets fs but that's not very convenient. pthread_set_affinity() is exactly what's needed. Personally I do not see much use for cpusets for those kinds of designs. But maybe I missing something. I got really excited when cpusets where first merged into mainline but after looking closer I could not really find a use for them, at least for not for our apps.
Max
Quoted 4 lines
So. I see cpusets as a higher level API/mechanism and
cpu_isolated_mapas lower level mechanism that actually makes kernel aware of what's isolated what's not. Kind of like sched domain/cpuset relationship. ie cpusets affect sched domains but scheduler does not use cpusets directly.
One could use cpusets to control the setting of cpu_isolated_map, separate from the code such as your select_irq_affinity() that uses it.
Quoted 6 lines
In a foreseeable future 2-8 cores will be most common configuration. Do you think that cpusets are needed/useful for those machines ? The reason I'm asking is because given the restrictions you mentioned above it seems that you might as well just do
taskset -c 1,2,3 app1 taskset -c 3,4,5 app2
People tend to manage the CPU and memory placement of the threads and processes within a single co-operating job using taskset (sched_setaffinity) and numactl (mbind, set_mempolicy.)
They tend to manage the placement of multiple unrelated jobs onto a single system, whether on separate or shared CPUs and nodes, using cpusets.
Something like cpu_isolated_map looks to me like a system-wide mechanism, which should, like sched_domains, be managed system-wide. Managing it with a mechanism that encourages each thread to update it directly, as if that thread owned the system, will break down, resulting in conflicting updates, as multiple, insufficiently co-operating threads issue conflicting settings.
Quoted 10 lines
Stuff that I'm working on this days (wireless basestations) is designed with the following model:
cpuN - runs soft-RT networking and management code cpuN+1 to cpuN+x - are used as dedicated engines
ie Simplest example would be
cpu0 - runs IP, L2 and control plane
cpu1 - runs hard-RT MACSo if CPU isolation is implemented on top of the cpusets what kind of API do you envision for such an app ?
That depends on what more API is needed. Do we need to place irqs better ... cpusets might not be a natural for that use. Aren't irqs directed to specific CPUs, not to hierarchically nested subsets of CPUs.
Separate question:
Is it desired that the dedicated CPUs cpuN+1 ... cpuN+x even appear as general purpose systems running a Linux kernel in your systems? These dedicated engines seem more like intelligent devices to me, such as disk controllers, which the kernel controls via device drivers, not by loading itself on them too.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.940.382.4214Quoted 8 lines · Paul Jackson
Quoted 4 lines
So. I see cpusets as a higher level API/mechanism and
cpu_isolated_mapas lower level mechanism that actually makes kernel aware of what's isolated what's not. Kind of like sched domain/cpuset relationship. ie cpusets affect sched domains but scheduler does not use cpusets directly.One could use cpusets to control the setting of
cpu_isolated_map, separate from the code such as yourselect_irq_affinity()that uses it.
Yes. That's what I proposed too. In one of the CPU isolation threads with Peter. The only issue is that you need to simulate CPU_DOWN hotplug even in order to cleanup what's already running on those CPUs.
Quoted 15 lines
Quoted 6 lines
In a foreseeable future 2-8 cores will be most common configuration. Do you think that cpusets are needed/useful for those machines ? The reason I'm asking is because given the restrictions you mentioned above it seems that you might as well just do
taskset -c 1,2,3 app1 taskset -c 3,4,5 app2People tend to manage the CPU and memory placement of the threads and processes within a single co-operating job using taskset (
sched_setaffinity) and numactl (mbind,set_mempolicy.)They tend to manage the placement of multiple unrelated jobs onto a single system, whether on separate or shared CPUs and nodes, using cpusets.
Quoted 6 lines
Something like
cpu_isolated_maplooks to me like a system-wide mechanism, which should, likesched_domains, be managed system-wide. Managing it with a mechanism that encourages each thread to update it directly, as if that thread owned the system, will break down, resulting in conflicting updates, as multiple, insufficiently co-operating threads issue conflicting settings.
I'm not sure how to interpret that. I think you might have mixed a couple of things I asked about in one reply ;-).
The question was that given the restrictions you talked about when you explained tiny-cpusets functionality I asked how much one gains from using them compared to the taskset/numactl. ie On the machines with 2-8 cores it's fairly easy to manage cpus with simple affinity masks.
The second part of your reply seems to imply that I somehow made you think that I suggested that cpu_isolated_map is managed per thread. That is of course not the case. It's definitely a system-wide mechanism and individual threads have nothing to do with it. btw I just re-read my prev reply. I definitely did not say anything about threads managing cpu_isolated_map :).
Quoted 8 lines
Quoted 8 lines
Stuff that I'm working on this days (wireless basestations) is designed with the following model:
cpuN - runs soft-RT networking and management code cpuN+1 to cpuN+x - are used as dedicated engines
ie Simplest example would be
cpu0 - runs IP, L2 and control plane
cpu1 - runs hard-RT MAC
Quoted 6 lines
Quoted 1 line
So if CPU isolation is implemented on top of the cpusets what kind of API do you envision for such an app ?
That depends on what more API is needed. Do we need to place irqs better ... cpusets might not be a natural for that use. Aren't irqs directed to specific CPUs, not to hierarchically nested subsets of CPUs.
You clipped the part where I elaborated. Which was:
Quoted 4 lines
Quoted 4 lines
So if CPU isolation is implemented on top of the cpusets what kind of API do you envision for such an app ? I mean currently cpusets seems to be mostly dealing with entire processes, whereas in this case we're really dealing with the threads. ie Different threads of the same process require different policies, some must run on isolated cpus some must not. I guess one could write a thread's pid into cpusets
fs but that's not very convenient.pthread_set_affinity()is exactly what's needed.
In other words how would an app place its individual threads into the different cpusets.
IRQ stuff is separate, like we said above cpusets could simply update cpu_isolated_map which would take care of IRQs. I was talking specifically about the thread management.
Quoted 6 lines
Separate question:
Is it desired that the dedicated CPUs cpuN+1 ... cpuN+x even appear as general purpose systems running a Linux kernel in your systems? These dedicated engines seem more like intelligent devices to me, such as disk controllers, which the kernel controls via device drivers, not by loading itself on them too.
We still want to be able to run normal threads on them. Which means IPI, memory management, etc is still needed. So yes they better show up as normal CPUs :) Also with dynamic isolation you can for example un-isolate a cpu when you're compiling stuff on the machine and then isolate it when you're running special app(s).
Max
Hi Pual
Quoted 4 lines
Looking at some IA64 sn2 config builds I have laying about, I see the following text sizes for a couple of versions, showing the growth of the cpuset/cgroup apparatus over time:
Quoted 8 lines
25933 2.6.18-rc3-mm1/kernel/cpuset.o (Aug 2006) vs. 37823 2.6.25-rc2-mm1/kernel/cgroup.o (Feb 2008) 19558 2.6.25-rc2-mm1/kernel/cpuset.oSo the total has grown from 25933 to 57381 text bytes (note that this is IA64 arch; most arch's will have proportionately smaller text sizes.)
hm, interesting.
but unfortunately the cpuset have more than depend.(i.e. CONFIG_SMP)
To more bad thing, some embedded cpu have poor or no atomic instruction support.
at that, turn on CONFIG_SMP become large performace regression ;)
I am not already embedded engineer.
thus, I might have made a mistake.
(BTW: I am large server engineer now)
but no thinking dependency is wrong, may be.
Pavel, what do you think it?
- kosaki