getting processor numbers
More and more code depends on knowing the number of processors in the system to efficiently scale the code. E.g., in OpenMP it is used by default to determine how many threads to create. Creating more threads than there are processors/cores doesn't make sense.
glibc for a long time provides functionality to retrieve the number through sysconf() and this is what fortunately most programs use. The problem is that we are currently using /proc/cpuinfo since this is all there was available at that time. Creating /proc/cpuinfo takes the kernel quite a long time, unfortunately (I think Jakub said it is mainly the interrupt information).
The alternative today is to use /sys/devices/system/cpu and count the number of cpu* directories in it. This is somewhat faster. But there would be another possibility: simply stat /sys/devices/system/cpu and use st_nlink - 2.
This last step unfortunately it made impossible by recent changes:
http://article.gmane.org/gmane.linux.kernel/413178
I would like to propose changing that patch, move the sched_* pseudo-files in some other directly and permanently ban putting any new file into /sys/devices/system/cpu.
To get some numbers, you can try
http://people.redhat.com/drepper/nproc-timing.c
The numbers I see on x86-64:
cpuinfo 10145810 cycles for 100 accesses
readdir /sys 3113870 cycles for 100 accesses
stat /sys 741070 cycles for 100 accesses
Note that for the first two methods I skipped the actual parsing part. This means in the real solution the gap between those two and the simple stat() call is even bigger.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 3 lines · Ulrich Drepper
More and more code depends on knowing the number of processors in the system to efficiently scale the code. E.g., in OpenMP it is used by default to determine how many threads to create.
There are more uses for it.
Quoted 2 lines
Creating more threads
than there are processors/cores doesn't make sense.
There was a proposal some time ago to put that into the ELF aux vector Unfortunately there was disagreement on what information to put there exactly (full topology, only limited numbers etc.)
My proposal was number of CPUs, number of cores, number of nodes as three 16 bit numbers.
-Andi
Quoted 3 lines · Andi Kleen
There was a proposal some time ago to put that into the ELF aux vector Unfortunately there was disagreement on what information to put there exactly (full topology, only limited numbers etc.)
Topology, yes, I'm likely in favor of it.
Processor number: no. Unless you want to rip out hotpluging. I'm certainly in favor of that, it creates huge problems for no real benefit for the common use cases. But as it is, the number of processors is not necessarily constant over the lifetime of a process. The machine architecture is.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 3 lines
benefit for the common use cases. But as it is, the number of processors is not necessarily constant over the lifetime of a process. The machine architecture is.
Not once you have migration capable virtualisation it isnt.
Quoted 6 lines · Ulrich Drepper
Quoted 3 lines · Andi Kleen
There was a proposal some time ago to put that into the ELF aux vector Unfortunately there was disagreement on what information to put there exactly (full topology, only limited numbers etc.)
Topology, yes, I'm likely in favor of it.
What topology and what use case?
Quoted 1 line
Processor number: no. Unless you want to rip out hotpluging. I'm
Topology is dependent on the number of CPUs.
Hot plugging is a completely orthogonal problem. Even your original proposal wouldn't address it. Mine doesn't neither, because i suspect most programs won't care. If it's addressed it could work on top of it. The aux vector to get the information quickly at program startup and later updates can get it from /sys.
If some program starts caring we would need to implement some notification mechanism (that would be possible), but it might be hard to fit into glibc because you don't have a event loop.
-Andi
Quoted 5 lines · Alan Cox
Quoted 3 lines
benefit for the common use cases. But as it is, the number of processors is not necessarily constant over the lifetime of a process. The machine architecture is.
Not once you have migration capable virtualisation it isnt.
Migration is fundamentally incompatible with many CPU optimizations. But that's not a reason to not optimize anymore.
But I guess luckily most migration users will be able to live with a little decreased performance after it.
-Andi
Quoted 31 lines · Ulrich Drepper
More and more code depends on knowing the number of processors in the system to efficiently scale the code. E.g., in OpenMP it is used by default to determine how many threads to create. Creating more threads than there are processors/cores doesn't make sense.
glibc for a long time provides functionality to retrieve the number through
sysconf()and this is what fortunately most programs use. The problem is that we are currently using/proc/cpuinfosince this is all there was available at that time. Creating/proc/cpuinfotakes the kernel quite a long time, unfortunately (I think Jakub said it is mainly the interrupt information).The alternative today is to use
/sys/devices/system/cpuand count the number of cpu* directories in it. This is somewhat faster. But there would be another possibility: simply stat/sys/devices/system/cpuand usest_nlink- 2.This last step unfortunately it made impossible by recent changes:
http://article.gmane.org/gmane.linux.kernel/413178
I would like to propose changing that patch, move the sched_* pseudo-files in some other directly and permanently ban putting any new file into
/sys/devices/system/cpu.To get some numbers, you can try
http://people.redhat.com/drepper/nproc-timing.c
The numbers I see on x86-64:
Quoted 12 lines
cpuinfo 10145810 cycles for 100 accesses
readdir /sys 3113870 cycles for 100 accesses
stat /sys 741070 cycles for 100 accessesNote that for the first two methods I skipped the actual parsing part. This means in the real solution the gap between those two and the simple
stat()call is even bigger.Signature · 1 line
ÿÿ Ulrich Drepper ÿÿ Red Hat, Inc. ÿÿ 444 Castro St ÿÿ Mountain View, CA ÿÿ
Shouldn't it just be another system call? 223 is currently unused. You could fill that up with __NR_nr_cpus. The value already exists in the kernel.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.65 BogoMips). New book: http://www.AbominableFirebug.com/
_
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.
Quoted 1 line · Andi Kleen
Topology is dependent on the number of CPUs.
Not all of it.
Quoted 2 lines
Hot plugging is a completely orthogonal problem. Even your original proposal wouldn't address it.
Nonsense. Reading /proc/cpuinfo or /sys/devices/system/cpu reflects the current CPU count. The information read would (and is not) cached, it's re-read every time. We might add very limited caching (for a few seconds) but that's as much as we can go.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 4 lines · Ulrich Drepper
Quoted 1 line · Andi Kleen
Topology is dependent on the number of CPUs.
Not all of it.
What is not?
Quoted 2 lines
We might add very limited caching (for a few
seconds) but that's as much as we can go.
Hmm, e.g. in OpenMP you would have another thread that just reads /proc/cpuinfo in a loop and starts new threads on new CPUs?
That sounds ...... "expensive"
The other use case in glibc I know of is the Opteron optimized memcpy which can use different functions depending on the number of cores. But having a separate thread regularly rereading cpuinfo for memcpy also sounds quite crazy.
-Andi
Quoted 3 lines · linux-os (Dick Johnson)
Shouldn't it just be another system call? 223 is currently unused. You could fill that up with
__NR_nr_cpus. The value already exists in the kernel.
You forget about Linus' credo "there shall be no sysconf-like syscall".
I'd be all for sys_sysconf or even the limited sys_nr_cpus although
ideally then we'd have two syscalls (probed CPUs, active CPUs, in which case sys_sysconf is the better choice).
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 4 lines · Andi Kleen
Quoted 2 lines
Quoted 1 line
Topology is dependent on the number of CPUs.
Not all of it.
What is not?
Memory banks can exist without a CPU present. The places where you can plug in memory don't change and so the memory hierarchy can be described.
Quoted 4 lines
Hmm, e.g. in OpenMP you would have another thread that just reads
/proc/cpuinfoin a loop and starts new threads on new CPUs?That sounds ...... "expensive"
That's the cost of doing business.
There is an inexpensive solution: finally make the vdso concept a bit more flexible. You could add a vdso call to get the processor count. The vdso code itself can use a data page mapped in from the kernel. This page (read-only at userlevel) would contain global information such as processor count and topology.
But we're getting IMO off topic here. That's a separate and far more complicated issue.
Here we now have the concrete issue that determining the CPU count is terribly expensive and there is a simple proposal to make it faster by keeping /sys/devices/system/cpu/ free from anything but cpu* directories.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 2 lines · Ulrich Drepper
Reading
/proc/cpuinfoor/sys/devices/system/cpureflects the current CPU count.
Not all of the cpu* directories in /sys/devices/system/cpu may be online.
thanks,
suresh
Quoted 8 lines · Ulrich Drepper
Quoted 4 lines · Andi Kleen
Quoted 2 lines
Quoted 1 line
Topology is dependent on the number of CPUs.
Not all of it.
What is not?
Memory banks can exist without a CPU present. The places where you can plug in memory don't change and so the memory hierarchy can be described.
There are systems that support node hotplug, like Altix or the larger IBM or Unisys x86 systems. Basically they are a bunch of smaller systems connected with cables running a cache coherent network protocol. With that memory can appear (and disappear but we don't handle that yet) unexpectedly.
That said we might have some idea of that in advance -- it can be described in ACPI SRAT -- but the trouble is that many quite ordinary x86 server systems always describe hotplug zones even though it is unlikely they will ever get any. Trusting that can be quite inefficient.
Quoted 3 lines
There is an inexpensive solution: finally make the vdso concept a bit more flexible. You could add a vdso call to get the processor count. The vdso code itself can use a data page mapped in from the kernel.
The ELF aux vector is exactly that already.
Quoted 2 lines
This page (read-only at userlevel) would contain global information such as processor count and topology.
You would still need an event notification mechanism, won't you?
Quoted 4 lines
But we're getting IMO off topic here. That's a separate and far more complicated issue.
Yes I agree. Hotplug is best ignored for now
Quoted 3 lines
Here we now have the concrete issue that determining the CPU count is terribly expensive and there is a simple proposal to make it faster by keeping
/sys/devices/system/cpu/free from anything but cpu* directories.
The cost will be still large. Accessing sysfs will be never cheap. For once anything going through the VFS tens to take a two sometimes three digit number of locks.
If you want it cheap look for some other way.
-Andi
Quoted 2 lines · Suresh B
Not all of the cpu* directories in
/sys/devices/system/cpumay be online.
Brilliant. You people really know how to create user interfaces. So now in any case per CPU another stat/access syscall is needed to check the 'online' pseudo-file? With this the readdir solution is only marginally faster than parsing /proc/cpuinfo which means, it's unacceptably slow.
I cannot believe all these big system people are allowed to screw everybody else up with their nonsense.
So, anybody else has a proposal? This is a pressing issue and cannot wait until someday in the distant future NUMA topology information is easily and speedily accessible.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 5 lines · Andi Kleen
Quoted 3 lines
There is an inexpensive solution: finally make the vdso concept a bit more flexible. You could add a vdso call to get the processor count. The vdso code itself can use a data page mapped in from the kernel.
The ELF aux vector is exactly that already.
No. The aux vector cannot be changed after the process is changed. The memory belong to the process and not the kernel. It must be possible at any time to get the correct information even if the system changed.
Quoted 4 lines
Quoted 2 lines
This page (read-only at userlevel) would contain global information such as processor count and topology.
You would still need an event notification mechanism, won't you?
No, why? The vdso call would be so inexpensive (just a simple function call) that it can be done whenever a topology-based decision has to be made. Use cookies to determine whether nothing has been changed since the last call etc.
Quoted 3 lines
The cost will be still large. Accessing sysfs will be never cheap. For once anything going through the VFS tens to take a two sometimes three digit number of locks.
That stat solution actually ain't that bad. It takes ~7400 cycles on my machine.
Quoted 1 line
If you want it cheap look for some other way.
Well, who's brace enough to submit sys_sysconf() again?
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 10 lines · Ulrich Drepper
Quoted 5 lines · Andi Kleen
Quoted 3 lines
There is an inexpensive solution: finally make the vdso concept a bit more flexible. You could add a vdso call to get the processor count. The vdso code itself can use a data page mapped in from the kernel.
The ELF aux vector is exactly that already.
No. The aux vector cannot be changed after the process is changed. The memory belong to the process and not the kernel. It must be possible at any time to get the correct information even if the system changed.
That's probably debatable, but ok.
I would be opposed for adding another page per process at least because the per process memory footprint in Linux is imho already too large.
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
This page (read-only at userlevel) would contain global information such as processor count and topology.
You would still need an event notification mechanism, won't you?
No, why? The vdso call would be so inexpensive (just a simple function call) that it can be done whenever a topology-based decision has to be made. Use cookies to determine whether nothing has been changed since the last call etc.
But how would that mix with the OpenMP use case where you have thread pools that normally don't make decisions afer startup, but just stay around? I think for those you would need events of some sort to start or remove threads as needed.
Asking the kernel every time you submit some work to the threads would probably not fly.
Quoted 3 lines
Quoted 1 line
If you want it cheap look for some other way.
Well, who's brace enough to submit
sys_sysconf()again?
If there's a good use case fine for me. However I suspect it's either "slow is ok" or "want it very fast" where even a syscall would hurt.
-Andi
Quoted 3 lines · Andi Kleen
I would be opposed for adding another page per process at least because the per process memory footprint in Linux is imho already too large.
That's a single page shared by all threads on the system. Or make this a page per NUMA node. And if the number of threads is larger than the share counter for page, create a few more.
But in general the extra overhead would be minimal from the memory consumption POV.
Quoted 8 lines
Quoted 4 lines
No, why? The vdso call would be so inexpensive (just a simple function call) that it can be done whenever a topology-based decision has to be made. Use cookies to determine whether nothing has been changed since the last call etc.
But how would that mix with the OpenMP use case where you have thread pools that normally don't make decisions afer startup, but just stay around?
There is a different between having threads in the thread pool and actually using them. For every #omp loop the number of processors is checked again and this is the number of threads from the pool which is used.
Quoted 2 lines
I think for those you would need events of some sort to start or remove threads as needed.
We need no events if determining the number of processors is cheap. There is really no reason why it shouldn't. Restarting all the threads is not the cheapest operation so a single syscall (sys_sysconf, etc) does not increase the cost a lot.
Quoted 3 lines
If there's a good use case fine for me. However I suspect it's either "slow is ok" or "want it very fast" where even a syscall would hurt.
Ideally, as I said, an optimized vdso call is best. But a syscall is OK. The nice thing about the vdso is that for now one could simply implement it using a syscall and in future add optimizations to avoid the kernel entry if possible.
A single syscall is two order of magnitude better than the best solution available today. This is my main concern right now.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 6 lines · * Ulrich Drepper ([email protected]) wrote:
glibc for a long time provides functionality to retrieve the number through
sysconf()and this is what fortunately most programs use. The problem is that we are currently using/proc/cpuinfosince this is all there was available at that time. Creating/proc/cpuinfotakes the kernel quite a long time, unfortunately (I think Jakub said it is mainly the interrupt information).
It's not only expensive to create, it's expensive and annoying to parse; I don't think it is vaguely consistent accross different architectures not to mention kernel versions.
Dave
Signature · 4 lines
-----Open up your eyes, open up your mind, open up your code ------- / Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \ \ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex / \ _________________________|_____ http://www.treblig.org |_______/
Quoted 27 lines · Ulrich Drepper
More and more code depends on knowing the number of processors in the system to efficiently scale the code. E.g., in OpenMP it is used by default to determine how many threads to create. Creating more threads than there are processors/cores doesn't make sense.
glibc for a long time provides functionality to retrieve the number through
sysconf()and this is what fortunately most programs use. The problem is that we are currently using/proc/cpuinfosince this is all there was available at that time. Creating/proc/cpuinfotakes the kernel quite a long time, unfortunately (I think Jakub said it is mainly the interrupt information).The alternative today is to use
/sys/devices/system/cpuand count the number of cpu* directories in it. This is somewhat faster. But there would be another possibility: simply stat/sys/devices/system/cpuand usest_nlink- 2.This last step unfortunately it made impossible by recent changes:
http://article.gmane.org/gmane.linux.kernel/413178
I would like to propose changing that patch, move the sched_* pseudo-files in some other directly and permanently ban putting any new file into
/sys/devices/system/cpu.To get some numbers, you can try
Quoted 7 lines
http://people.redhat.com/drepper/nproc-timing.c
The numbers I see on x86-64:
cpuinfo 10145810 cycles for 100 accesses
readdir /sys 3113870 cycles for 100 accesses
stat /sys 741070 cycles for 100 accesses
It sucks when seen from a micro-bench POV, but does it really matter overall? The vast majority of software usually calls sysconf(_SC_NPROCESSORS_*) with very little frequency (mostly once at initialization time) anyway. That's what 50us / call?
- Davide
Quoted 4 lines · Davide Libenzi
It sucks when seen from a micro-bench POV, but does it really matter overall? The vast majority of software usually calls
sysconf(_SC_NPROCESSORS_*)with very little frequency (mostly once at initialization time) anyway. That's what 50us / call?
This is not today's situation. Yes, 10 years ago when I added the support to glibc it wasn't much of a problem. But times change. As I said before in this thread, OpenMP by default scales the number of threads used for a parallel loops depending on the number of available processors/cores and therefore the number must be retrieved every time (with perhaps minimal caching of a few secs, but this requires gettimeofday calls...). All of a sudden this is not micro benchmark anymore. It's a real issue which we only became aware of because it is noticeable in real life.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 9 lines · Ulrich Drepper
Quoted 2 lines · Suresh B
Not all of the cpu* directories in
/sys/devices/system/cpumay be online.Brilliant. You people really know how to create user interfaces. So now in any case per CPU another stat/access syscall is needed to check the 'online' pseudo-file? With this the readdir solution is only marginally faster than parsing
/proc/cpuinfowhich means, it's unacceptably slow.
Note that glibc actually parses /proc/stat in preference of /proc/cpuinfo ATM, because /proc/stat is at least uniform while parsing /proc/cpuinfo needs a special parser for each architecture. And /proc/stat reading is even slower than /proc/cpuinfo, on x86_64 reading/parsing /proc/stat takes about 450usec, while e.g. stat64 on /sys/devices/system/cpu is just 2.5usec.
But if that can't be trusted as the number of online CPUs, can somebody please add a short file to proc or sysfs which will contain the number of online and number of configured CPUs?
See e.g. http://openmp.org/pipermail/omp/2007/000714.html
where the first time after second g++ invocation is with omp_set_dynamic (1) and ought to be about as fast as omp_set_dynamic (0) case with the same number of threads, but it is far slower due to slow
sysconf (_SC_NPROCESSORS_ONLN).
Jakub
Siddha, Suresh B wrote:a
Quoted 2 lines
Not all of the cpu* directories in
/sys/devices/system/cpumay be online.
Apparently this information isn't needed. It's very easy to verify:
$ ls /sys/devices/system/cpu/*/online /sys/devices/system/cpu/cpu1/online /sys/devices/system/cpu/cpu2/online /sys/devices/system/cpu/cpu3/online
This is a quad core machine and cpu0 doesn't have the 'online' file (2.6.19 kernel). So, if nobody noticed this it's not needed and we can just remove CPUs from /sys/devices/system/cpu when they are brought offline, right?
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Hi Ulrich,
Quoted 3 lines · Ulrich Drepper
So, anybody else has a proposal? This is a pressing issue and cannot wait until someday in the distant future NUMA topology information is easily and speedily accessible.
Since for now you just need a fast and dirty hack, which will be replaced with better interfaces, I suggest creating a directory with some files in it. These should just contain, what you need to handle your most pressing cases.
I propose /sys/devices/system/topology_counters/ for that. These can contain "online_cpu", "proped_cpu", "max_cpu"
and maybe the same for nodes. All that as a simple file with an integer value.
Since sysfs-attribute files are pollable (if the owners notifies sysfs on changes), you also have the notification system you need (select, poll, epoll etc.).
If you promise to just keep the slow code around, than one day when the shiny NUMA topology stuff is ready, this directory can be completely removed and glibc (plus all their users) keeps working. It will then even work better with a new glibc version, which supports the shiny new NUMA topology stuff.
The kernel can create these counters quiete easy, since most of them are the hamming weight (or population count) of some bitmaps.
Does this sound like a proper hacky solution? :-)
Regards
Ingo Oeser
Quoted 4 lines · Ulrich Drepper
Siddha, Suresh B wrote:a
Quoted 2 lines
Not all of the cpu* directories in
/sys/devices/system/cpumay be online.
Quoted 8 lines
Apparently this information isn't needed. It's very easy to verify:
$ ls /sys/devices/system/cpu/*/online /sys/devices/system/cpu/cpu1/online /sys/devices/system/cpu/cpu2/online /sys/devices/system/cpu/cpu3/onlineThis is a quad core machine and cpu0 doesn't have the 'online' file (2.6.19 kernel).
I think that is expected and intentional, as the current cpu hotplug code doesn't support offlining cpu0.
Quoted 3 lines
So, if nobody noticed this it's not needed and we can
just remove CPUs from/sys/devices/system/cpuwhen they are brought offline, right?
No. Logical cpu hotplug uses these interfaces to make a cpu go offline and online.
thanks,
suresh
Quoted 4 lines · On Tue, 03 Apr 2007 09:54:46 -0700 Ulrich Drepper <[email protected]> wrote:
More and more code depends on knowing the number of processors in the system to efficiently scale the code. E.g., in OpenMP it is used by default to determine how many threads to create. Creating more threads than there are processors/cores doesn't make sense.
but... It would be a mistake for an application to assume that it is allowed to _use_ all the present CPUs. People can and do run applications within cpusets, and under sched_setaffinity().
So I'd have thought that in general an application should be querying its present affinity mask - something like sched_getaffinity()? That fixes the CPU hotplug issues too, of course.
But we discussed this all a couple years back and it was decided that sched_getaffinity() was unsuitable. I remember at the time not really understanding why?
Quoted 6 lines · Ulrich Drepper
Siddha, Suresh B wrote:a
Quoted 2 lines
Not all of the cpu* directories in
/sys/devices/system/cpumay be online.Apparently this information isn't needed. It's very easy to verify:
Quoted 8 lines
$ ls /sys/devices/system/cpu/*/online /sys/devices/system/cpu/cpu1/online /sys/devices/system/cpu/cpu2/online /sys/devices/system/cpu/cpu3/onlineThis is a quad core machine and cpu0 doesn't have the 'online' file (2.6.19 kernel). So, if nobody noticed this it's not needed and we can just remove CPUs from
/sys/devices/system/cpuwhen they are brought offline, right?
No... the online sysfs files are used to show and change cpus' online/offline state. You wouldn't be able to bring an offlined cpu back online again.
cpu0 doesn't have an online file on machines which don't support offlining of the boot cpu.
Quoted 2 lines · Suresh B
No. Logical cpu hotplug uses these interfaces to make a cpu go offline and online.
You missed my sarcasms, email is bad for conveying it. The point is nobody really cares about that hotplug nonsense to have noticed the bug.
And still does this nonsense prevent real problems from being addressed.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 3 lines · Andi Kleen
Migration is fundamentally incompatible with many CPU optimizations. But that's not a reason to not optimize anymore.
I've been thinking about ways in which Xen could provide the current vcpu->cpu map to guest domains. Obviously this would change over time, but it could remain current enough to be useful.
Quoted 3 lines
But I guess luckily most migration users will be able to live with a little decreased performance after it.
At least in the Xen case, the source and target machines need to be fairly similar in architecture (can't deal with vastly different CPU types, for example).
J
Quoted 8 lines · Ulrich Drepper
Quoted 2 lines · Suresh B
No. Logical cpu hotplug uses these interfaces to make a cpu go offline and online.
You missed my sarcasms, email is bad for conveying it. The point is nobody really cares about that hotplug nonsense to have noticed the bug.
And still does this nonsense prevent real problems from being addressed.
Please dont focus on /sys being your holy gral.
1) AFAIK /sys/devices/system/cpu was not designed to meet glibc needs.
2) Many production machines dont mount /sys at all
$ uname -r 2.6.20 $ ls -al /sys/devices/system/cpu ls: /sys/devices/system/cpu: No such file or directory $ grep processor /proc/cpuinfo processor : 0 processor : 1
Quoted 1 line · Andrew Morton
Did we mean to go off-list?
Oops, no, pressed the wrong button.
Quoted 21 lines
Quoted 8 lines
Quoted 3 lines · Andrew Morton
So I'd have thought that in general an application should be querying its present affinity mask - something like
sched_getaffinity()? That fixes the CPU hotplug issues too, of course.Does it really?
My recollection is that the affinity masks of running processes is not updated on hotplugging. Is this addressed?
ah, yes, you're correct.
Inside a cpuset:
sched_setaffinity()is constrained to those CPUs which are in the cpuset.If a cpu if on/offlined we update each cpuset's cpu mask appropriately but we do not update all the tasks presently running in the cpuset.
Outside a cpuset:
Quoted 7 lines
sched_setaffinity()is constrained to all possible cpusWe don't update each task's
cpus_allowedwhen a CPU is removed.I think we trivially _could_ update each tasks's
cpus_allowedmask when a CPU is removed, actually.
I think it has to be done. But that's not so trivial. What happens if all the CPUs a process was supposed to be runnable on vanish. Shouldn't, if no affinity mask is defined, new processors be added? I agree that if the process has a defined affinity mask no new processors should be added _automatically_.
Quoted 7 lines
Quoted 4 lines
If yes,
sched_getaffinityis a solution until the NUMA topology framework can provide something better. Even without a popcnt instruction in the CPU (64-bit albeit) it's twice as fast as the thestat()method proposed.I'm surprised - I'd have expected
sched_getaffinity()to be vastly quicker that doing fileystem operations.
You mean because it's only a factor of two? Well, it's not once you count the whole overhead.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 31 lines · On Tue, 03 Apr 2007 15:13:09 -0700 Ulrich Drepper <[email protected]> wrote:
Quoted 1 line · Andrew Morton
Did we mean to go off-list?
Oops, no, pressed the wrong button.
Quoted 26 lines
Quoted 8 lines
Quoted 3 lines · Andrew Morton
So I'd have thought that in general an application should be querying its present affinity mask - something like
sched_getaffinity()? That fixes the CPU hotplug issues too, of course.Does it really?
My recollection is that the affinity masks of running processes is not updated on hotplugging. Is this addressed?
ah, yes, you're correct.
Inside a cpuset:
sched_setaffinity()is constrained to those CPUs which are in the cpuset.If a cpu if on/offlined we update each cpuset's cpu mask appropriately but we do not update all the tasks presently running in the cpuset.
Outside a cpuset:
sched_setaffinity()is constrained to all possible cpusWe don't update each task's
cpus_allowedwhen a CPU is removed.
Quoted 9 lines
Quoted 2 lines
I think we trivially _could_ update each tasks's
cpus_allowedmask when a CPU is removed, actually.I think it has to be done. But that's not so trivial. What happens if all the CPUs a process was supposed to be runnable on vanish. Shouldn't, if no affinity mask is defined, new processors be added? I agree that if the process has a defined affinity mask no new processors should be added _automatically_.
Yes, some policy decision needs to be made there.
But whatever we decide to do, the implementation will be relatively straightforward, because hot-unplug uses stop_machine_run() and later, we hope, will use the process freezer. This setting of the whole machine into a known state means (I think) that we can avoid a whole lot of fuss which happens when affinity is altered.
Anyway. It's not really clear who maintains CPU hotplug nowadays. <adds a few cc's>. But yes, I do thing we should do <something sane> with process affinity when CPU hot[un]plug happens.
Now it could be argued that the current behaviour is that sane thing: we allow the process to "pin" itself to not-present CPUs and just handle it in the CPU scheduler.
Paul, could you please describe what cpusets' policy is in the presence of CPU additional and removal?
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
If yes,
sched_getaffinityis a solution until the NUMA topology framework can provide something better. Even without a popcnt instruction in the CPU (64-bit albeit) it's twice as fast as the thestat()method proposed.I'm surprised - I'd have expected
sched_getaffinity()to be vastly quicker that doing fileystem operations.You mean because it's only a factor of two? Well, it's not once you count the whole overhead.
Is it kernel overhead, or userspace? The overhead of counting the bits?
Because sched_getaffinity() could be easily sped up in the case where it is operating on the current process.
Anyway, where do we stand? Assuming we can address the CPU hotplug issues, does sched_getaffinity() look like it will be suitable?
Quoted 3 lines · Andrew Morton
Now it could be argued that the current behaviour is that sane thing: we allow the process to "pin" itself to not-present CPUs and just handle it in the CPU scheduler.
As a stop-gap solution Jakub will likely implement the sched_getaffinity hack. So, it would realy be best to get the masks updated.
But all this of course does not solve the issue sysconf() has. In sysconf we cannot use sched_getaffinity since all the systems CPUs must be reported.
Quoted 1 line
Is it kernel overhead, or userspace? The overhead of counting the bits?
The overhead I meant is userland.
Quoted 2 lines
Because
sched_getaffinity()could be easily sped up in the case where it is operating on the current process.
If there is possibility to treat this case special and make it faster, please do so. It would be best to allow pid==0 as a special case so that callers don't have to find out the TID (which they shouldn't have to know).
Quoted 2 lines
Anyway, where do we stand? Assuming we can address the CPU hotplug issues, does
sched_getaffinity()look like it will be suitable?
It's only usable for the special case on the OpenMP code where the number of threads is used to determine the number of worker threads. For sysconf() we still need better support. Maybe now somebody will step up and say they need faster sysconf as well.
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 5 lines · On Tue, 03 Apr 2007 16:00:50 -0700 Ulrich Drepper <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
Now it could be argued that the current behaviour is that sane thing: we allow the process to "pin" itself to not-present CPUs and just handle it in the CPU scheduler.
Quoted 7 lines
As a stop-gap solution Jakub will likely implement the
sched_getaffinityhack. So, it would realy be best to get the masks updated.But all this of course does not solve the issue
sysconf()has. In sysconf we cannot usesched_getaffinitysince all the systems CPUs must be reported.
OK.
This is excecptionally gruesome, but one could run sched_getaffinity() against pid 1 (init). Which will break nicely in the OS-virtualised future when the system has multiple pid-1-inits running in containers...
Quoted 5 lines
Quoted 2 lines
Is it kernel overhead, or userspace? The overhead of counting the bits?
The overhead I meant is userland.
OK. Your cost of counting those bits is proportional to CONFIG_NR_CPUS.
It's a bit sad that sys_sched_get_get_affinity() returns sizeof(cpumask_t), because that means that userspace must handle 256 or whatever CPUs on a machine which only has two CPUs.
Does anyone see a reason why sys_sched_getaffinity() cannot be altered to return maximum-possible-cpu-id-on-this-machine? That way, your hweight operation will be much faster on sane-sized machines.
Quoted 9 lines
Quoted 3 lines
Because
sched_getaffinity()could be easily sped up in the case where it is operating on the current process.If there is possibility to treat this case special and make it faster, please do so. It would be best to allow pid==0 as a special case so that callers don't have to find out the TID (which they shouldn't have to know).
OK.
Does anyone see a reason why we cannot do this?
kernel/sched.c renamed from kernel/sched.c~sched_getaffinity-speedup +13 -8
@@ -4381,8 +4381,12 @@ long sched_getaffinity(pid_t pid, cpumasstruct task_struct *p;int retval; -lock_cpu_hotplug (); -read_lock (&tasklist_lock); +if (pid) { +lock_cpu_hotplug (); +read_lock (&tasklist_lock); + }else { +preempt_disable ();/* Prevent CPU hotplugging */ + } retval = -ESRCH ; p =find_process_by_pid (pid);
@@ -4396,12 +4400,13 @@ long sched_getaffinity(pid_t pid, cpumascpus_and (*mask, p->cpus_allowed, cpu_online_map); out_unlock: -read_unlock (&tasklist_lock); -unlock_cpu_hotplug (); -if (retval) -return retval; - -return 0 ; +if (pid) { +read_unlock (&tasklist_lock); +unlock_cpu_hotplug (); + }else { +preempt_enable (); + } +return retval; }/**
_
Quoted 8 lines
Quoted 3 lines
Anyway, where do we stand? Assuming we can address the CPU hotplug issues, does
sched_getaffinity()look like it will be suitable?It's only usable for the special case on the OpenMP code where the number of threads is used to determine the number of worker threads. For
sysconf()we still need better support. Maybe now somebody will step up and say they need faster sysconf as well.
I guess we could add a simple sys_get_nr_cpus(). If we want more than that (ie: topology, SMT/MC/NUMA/numa-distance etc) then it gets much more complex and sysfs is more appropriate for that.
Quoted 25 lines · Ingo Oeser
Hi Ulrich,
Quoted 3 lines · Ulrich Drepper
So, anybody else has a proposal? This is a pressing issue and cannot wait until someday in the distant future NUMA topology information is easily and speedily accessible.
Since for now you just need a fast and dirty hack, which will be replaced with better interfaces, I suggest creating a directory with some files in it. These should just contain, what you need to handle your most pressing cases.
I propose
/sys/devices/system/topology_counters/for that. These can contain "online_cpu", "proped_cpu", "max_cpu"
and maybe the same for nodes. All that as a simple file with an integer value.Since sysfs-attribute files are pollable (if the owners notifies sysfs on changes), you also have the notification system you need (select, poll, epoll etc.).
If you promise to just keep the slow code around, than one day when the shiny NUMA topology stuff is ready, this directory can be completely removed and glibc (plus all their users) keeps working. It will then even work better with a new glibc version, which supports the shiny new NUMA topology stuff.
Quoted 5 lines
The kernel can create these counters quiete easy, since most of them are the hamming weight (or population count) of some bitmaps.
Does this sound like a proper hacky solution? :-)
Just a point of view from someone who has to parse /proc/cpuinfo. That sort of file tree thing is useful to work from the command line but its a kick in the a** to use from a program.
This makes you just to re-parse the tree each time you have to get the info (open, read, atoi, close...) to fill your internal variables.
I don't know if its possible, but I would like something like:
__packt_it_tight_please struct cpumap_t {
u16 ncpus;
u16 ncpus_onln;
u16 ncpus_inmyset; // for procsets
// Here possibly more info about topology, pack-core-thread structure...
// in simple arrays...
};struct cpumap_t *cpumap = mmap("/proc/sys/hw/cpumap",sizeof(struct cpumap_t));
for (...cpumap->ncpus_inmyset ....) //
As I said, I don't know if its possible. I vaguely remember some comments against binary info in /proc...
Even it could be simplified if you realize some things:
- Usually people dont worry about if cpus are all the online ones or I'm running in a proc set. Just want to know how many can I use.
- Don't care if they are hyper-threaded, cores os independent processors. To adjust processing for hyper-threaded cpus, one needs to tie processes to processors, and you need to be root for that.
Really, anything dependent on topology is not usable for normal programs, because you need to be root to control that.
So topology is not so important.
Some (probably stupid) ideas...
--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2007.1 (Cooker) for i586
Linux 2.6.20-jam08 (gcc 4.1.2 20070302 (prerelease) (4.1.2-1mdv2007.1)) #1 SMP PREEMPT
Quoted 1 line · Andrew Morton
Does anyone see a reason why we cannot do this?
Shouldn't sched_setaffinity get the same treatment for symmetry reasons?
Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
Quoted 12 lines · Ulrich Drepper
Quoted 1 line · Davide Libenzi
It sucks when seen from a micro-bench POV, but does it really matter overall? The vast majority of software usually calls
sysconf(_SC_NPROCESSORS_*)with very little frequency (mostly once at initialization time) anyway. That's what 50us / call?This is not today's situation. Yes, 10 years ago when I added the support to glibc it wasn't much of a problem. But times change. As I said before in this thread, OpenMP by default scales the number of threads used for a parallel loops depending on the number of available processors/cores and therefore the number must be retrieved every time (with perhaps minimal caching of a few secs, but this requires gettimeofday calls...). All of a sudden this is not micro benchmark anymore. It's a real issue which we only became aware of because it is noticeable in real life.
Sounds like it would need a device which can be waited upon for changes.
-hpa
Quoted 1 line · H. Peter Anvin
Sounds like it would need a device which can be waited upon for changes.
A vdso-like shared page could have a futex in it.
J
Quoted 4 lines · Jeremy Fitzhardinge
Quoted 1 line · H. Peter Anvin
Sounds like it would need a device which can be waited upon for changes.
A vdso-like shared page could have a futex in it.
Yes, but a futex couldn't be waited upon with a bunch of other things as part of a poll or a select. The cost of reading the information is minimal.
-hpa
Quoted 3 lines · Andrew
I'd have thought that in general an application should be querying its present affinity mask - something like
sched_getaffinity()? That fixes the CPU hotplug issues too, of course.
The sched_getaffinity call is quick, too, and it nicely reflects any cpuset constraints, while still working on kernels which don't have CPUSETs configured.
There are really at least four "number of CPUs" answers here, and we should be aware of which we are providing. There are, in order of decreasing size:
1) the size of the kernels cpumask_t (NR_CPUS),
2) the maximum number of CPUs that might ever be hotplugged into a booted system,
3) the current number of CPUs online in that system, and
4) the number of CPUs that the current task is allowed to use.
I would suggest that (4) is what we should typically return. Certainly it would seem that the use that Ulrich is concerned with, by OpenMP, wants (4).
Currently, the sysconf(_SC_NPROCESSORS_CONF) returns (3), by counting the CPUs in /proc/stat, which is rather bogus on cpuset, or even sched_setaffinity, constrained systems.
Quoted 3 lines
But we discussed this all a couple years back and it was decided that
sched_getaffinity()was unsuitable. I remember at the time not really understanding why?
Perhaps it was because a robust invocation of sched_getaffinity takes a page of code to write, as Andi Kleen noticed in his libnuma coding of "static int number_of_cpus(void)". One has to size the mask passed in, in case the kernel was compiled with a larger cpumask_t size than you guessed up front. In other words, to get (4) using sched_getaffinity, one first needs an upper bound on (1), above, the kernels configured NR_CPUS.
One can either size it by repeatedly invoking sched_getaffinity with larger masks until it stops failing EINVAL, or one can examine the length of the "Cpus_allowed" mask displayed in /proc/self/status (it takes 9 ascii chars, if you include the commas and trailing newline, to display each 32 bits of cpumask_t.) There may be other ways as well; those seem to be the most common.
At least the kernel cpumask_t size can be cached for the life of the process and any descendents, so the cost of obtaining it should be less critical.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 2 lines · Andrew
Paul, could you please describe what cpusets' policy is in the presence of CPU additional and removal?
Currently, if we remove the last CPU in a cpuset, we give that cpuset the CPUs of its parent cpuset, in order to ensure that every cpuset with tasks attached actually has some CPUs they can run on. See the routine kernel/cpuset.c:guarantee_online_cpus_mems_in_subtree(), and kernel/cpuset.c:guarantee_online_cpus(), and close your eyes to the recursion ... yeah that has to get fixed someday ;).
But Cliff Wickman <[email protected]> (added to CC) figured out that this was broken, as it could easily violate the cpu_exclusive property of cpusets. He is working on a patch that will move the tasks in the CPU-deficient cpuset up to their parent cpuset.
In general, the idea is to ensure that every task has at least one CPU on which it can run. If the sysadmin intends to unplug all the CPUs required by some task, he "should" first move those tasks somewhere else where they can continue to run. If he doesn't, then the kernel "makes do", by either (currently) adding CPUs to the CPU-deficient cpuset, or (future) moving the CPU-deprived tasks somewhere else.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 9 lines · Andrew
Quoted 3 lines
But all this of course does not solve the issue
sysconf()has. In sysconf we cannot usesched_getaffinitysince all the systems CPUs must be reported.OK.
This is excecptionally gruesome, but one could run
sched_getaffinity()against pid 1 (init). Which will break nicely in the OS-virtualised future when the system has multiple pid-1-inits running in containers...
That nicely breaks on typical cpuset managed systems as well, which frequently put init into a small cpuset (with the classic Unix daemon and login load), leaving the bulk of the system to be managed by a batch scheduler.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 3 lines · Ulrich
But all this of course does not solve the issue
sysconf()has. In sysconf we cannot usesched_getaffinitysince all the systems CPUs must be reported.
Good point. Yes, sysconf(_SC_NPROCESSORS_CONF) really needs to continue returning the number of CPUs online, to maintain compatibility with the current implementation, and because with a name like that, just about anything else would be "surprising".
And OpenMPI shouldn't be calling sysconf(_SC_NPROCESSORS_CONF), as it does not want to know how much hardware is there, but rather how much hardware it is allowed to use. Something based on sched_getaffinity() would seem to be ideal for its purposes.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 2 lines · Ulrich
For
sysconf()we still need better support. Maybe now somebody will step up and say they need faster sysconf as well.
That won't be me ;).
For any kernel compiled with CONFIG_CPUSETS (which includes the major distros I am aware of), one can just count the bits in the top cpusets 'cpus' value, which is always the online CPU mask. Even if your user level software is making no conscious use of cpusets, this works, for kernels so configured.
If there is someone who needs both a faster sysconf (_SC_NPROCESSORS_CONF) and a kernel without CPUSETS configured, then they will have to speak up for themselves.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 8 lines · H. Peter Anvin
Quoted 4 lines · Jeremy Fitzhardinge
Quoted 1 line · H. Peter Anvin
Sounds like it would need a device which can be waited upon for changes.
A vdso-like shared page could have a futex in it.
Yes, but a futex couldn't be waited upon with a bunch of other things as part of a poll or a select. The cost of reading the information is minimal.
There is one thing that always worried me.
Intel & AMD manuals make clear that mixing data and program in the same page is bad for performance.
In particular, x86_64 vsyscall put jiffies and other vsyscall_gtod_data_t right in the midle of code. That is certainly not wise.
A probably sane implementation should use two pages, one for code, one for data.
Quoted 9 lines · Eric Dumazet
There is one thing that always worried me.
Intel & AMD manuals make clear that mixing data and program in the same page is bad for performance.
In particular,
x86_64vsyscall put jiffies and othervsyscall_gtod_data_tright in the midle of code. That is certainly not wise.A probably sane implementation should use two pages, one for code, one for data.
Mutable data should be separated from code. I think any current CPU will do fine as long as they are in separate 128-byte chunks, but they need at least that much separation.
Readonly data does not need to be separated from code.
-hpa
Quoted 3 lines · H. Peter Anvin
Mutable data should be separated from code. I think any current CPU will do fine as long as they are in separate 128-byte chunks, but they need at least that much separation.
P4 manual says that if one processor modifies data within 2k of another processor executing code, it will trash the entire trace cache.
J
Quoted 6 lines · H. Peter Anvin
Quoted 5 lines · Eric Dumazet
There is one thing that always worried me.
Intel & AMD manuals make clear that mixing data and program in the same page is bad for performance.
Quoted 9 lines
Quoted 4 lines
In particular,
x86_64vsyscall put jiffies and othervsyscall_gtod_data_tright in the midle of code. That is certainly not wise.A probably sane implementation should use two pages, one for code, one for data.
Mutable data should be separated from code. I think any current CPU will do fine as long as they are in separate 128-byte chunks, but they need at least that much separation.
Readonly data does not need to be separated from code.
Yes... jiffies & vsyscall_gtod_data_t is writen HZ times per second.
Not really readonly I'm afraid.
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 3 lines · H. Peter Anvin
Mutable data should be separated from code. I think any current CPU will do fine as long as they are in separate 128-byte chunks, but they need at least that much separation.
P4 manual says that if one processor modifies data within 2k of another processor executing code, it will trash the entire trace cache.
Yuck. Didn't realize the P4 was that sensitive. OK, so at the least we need a half-page of separation.
-hpa
Quoted 9 lines · H. Peter Anvin
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 3 lines · H. Peter Anvin
Mutable data should be separated from code. I think any current CPU will do fine as long as they are in separate 128-byte chunks, but they need at least that much separation.
P4 manual says that if one processor modifies data within 2k of another processor executing code, it will trash the entire trace cache.
Yuck. Didn't realize the P4 was that sensitive. OK, so at the least we need a half-page of separation.
Yes but vsyscall API currently defines 4 entry points :
vsyscall0 (vgettimeofday) = ADDR
vsyscall1 (vtime) = ADDR+1024
vsyscall2 (vgetcpu) = ADDR+2048
vsyscall3 (vxxxxx) = ADDR+3072
Quoted 18 lines · Paul Jackson
Quoted 3 lines · Andrew
I'd have thought that in general an application should be querying its present affinity mask - something like
sched_getaffinity()? That fixes the CPU hotplug issues too, of course.The
sched_getaffinitycall is quick, too, and it nicely reflects any cpuset constraints, while still working on kernels which don't have CPUSETs configured.There are really at least four "number of CPUs" answers here, and we should be aware of which we are providing. There are, in order of decreasing size:
1) the size of the kernelscpumask_t(NR_CPUS),
2) the maximum number of CPUs that might ever be hotplugged into a booted system,
3) the current number of CPUs online in that system, and
4) the number of CPUs that the current task is allowed to use.
Quoted 7 lines
I would suggest that (4) is what we should typically return. Certainly it would seem that the use that Ulrich is concerned with, by OpenMP, wants (4).
Currently, the
sysconf(_SC_NPROCESSORS_CONF)returns (3), by counting the CPUs in/proc/stat, which is rather bogus on cpuset, or evensched_setaffinity, constrained systems.
OpenMP wants (4) and I'll change it that way.
sysconf(_SC_NPROCESSORS_ONLN) must return (3) (this currently scans /proc/stat) andsysconf(_SC_NPROCESSORS_CONF) should IMHO return (2) (this currently scans /proc/cpuinfo on alpha and sparc{,64} for ((ncpus|CPUs) probed|cpus detected) and for the rest just returns sysconf(_SC_NPROCESSORS_ONLN)).
Neither of the sysconf returned values should be affected by affinity.
Jakub
Quoted 1 line · Jakub
OpenMP wants (4) and I'll change it that way.
Good. Are you referring to a change in glibc or in OpenMP?
Quoted 1 line
sysconf(_SC_NPROCESSORS_ONLN)must return (3) (this currently scans/proc/stat)
Ok
Quoted 3 lines
sysconf(_SC_NPROCESSORS_CONF)should IMHO return (2) (this currently scans/proc/cpuinfoon alpha and sparc{,64} for ((ncpus|CPUs) probed|cpus detected) and for the rest just returnssysconf(_SC_NPROCESSORS_ONLN)).
Not quite what I would have guessed, but seems ok.
Thanks for spelling this out.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401Quoted 18 lines · Andrew Morton
Quoted 5 lines · On Tue, 03 Apr 2007 16:00:50 -0700 Ulrich Drepper <[email protected]> wrote:
If there is possibility to treat this case special and make it faster, please do so. It would be best to allow pid==0 as a special case so that callers don't have to find out the TID (which they shouldn't have to know).
OK.
Does anyone see a reason why we cannot do this?
kernel/sched.c renamed from kernel/sched.c~sched_getaffinity-speedup +0
-0@@ -4381,8 +4381,12 @@ long sched_getaffinity(pid_t pid, cpumasstruct task_struct *p;
kernel/sched.c long sched_getaffinity(pid_t pid, cpumas · line 4389
Quoted 10 lines
kernel/sched.c renamed from kernel/sched.c~sched_getaffinity-speedup +6
-2int retval; -lock_cpu_hotplug (); -read_lock (&tasklist_lock); +if (pid) { +lock_cpu_hotplug (); +read_lock (&tasklist_lock); + }else { +preempt_disable ();/* Prevent CPU hotplugging */ + }
But we don't need tasklist_lock at all, we can use rcu_read_lock/unlock.
Q: don't we need task_rq_lock() to read ->cpus_allowed "atomically" ?
UNTESTED.
kernel/sched.c renamed from kernel/sched.c~ +5 -10
@@ -4433,22 +4433,17 @@ long sched_setaffinity(pid_t pid, cpumasint retval;mutex_lock (&sched_hotcpu_mutex); -read_lock (&tasklist_lock); +rcu_read_lock (); p =find_process_by_pid (pid);if (!p) { -read_unlock (&tasklist_lock); +rcu_read_unlock ();mutex_unlock (&sched_hotcpu_mutex);return -ESRCH ; } -/* -* It is not safe to call set_cpus_allowed with the -* tasklist_lock held. We will bump the task_struct's -* usage count and then drop tasklist_lock. -*/ get_task_struct (p); -read_unlock (&tasklist_lock); +rcu_read_unlock (); retval = -EPERM ; if ((current->euid != p->euid) && (current->euid != p->uid) &&
@@ -4523,7 +4518,7 @@ long sched_getaffinity(pid_t pid, cpumasint retval;mutex_lock (&sched_hotcpu_mutex); -read_lock (&tasklist_lock); +rcu_read_lock (); retval = -ESRCH ; p =find_process_by_pid (pid);
@@ -4537,7 +4532,7 @@ long sched_getaffinity(pid_t pid, cpumascpus_and (*mask, p->cpus_allowed, cpu_online_map); out_unlock: -read_unlock (&tasklist_lock); +rcu_read_unlock ();mutex_unlock (&sched_hotcpu_mutex);if (retval)return retval;
Quoted 3 lines · Oleg Nesterov
But we don't need
tasklist_lockat all, we can use rcu_read_lock/unlock. Q: don't we needtask_rq_lock()to read ->cpus_allowed "atomically" ?
right now ->cpus_allowed is protected by tasklist_lock. We cannot do RCU here because ->cpus_allowed modifications are not RCUified.
Ingo
Quoted 9 lines · Ingo Molnar
Quoted 3 lines · Oleg Nesterov
But we don't need
tasklist_lockat all, we can use rcu_read_lock/unlock. Q: don't we needtask_rq_lock()to read ->cpus_allowed "atomically" ?right now ->cpus_allowed is protected by
tasklist_lock. We cannot do RCU here because ->cpus_allowed modifications are not RCUified.
Is it so? that was my question. Afaics, set_cpus_allowed() does p->cpus_allowed = new_mask under rq->lock, so I don't understand how tasklist_lock can help.
Could you clarify?
Oleg.
Quoted 8 lines · Oleg Nesterov
Quoted 7 lines · Ingo Molnar
Quoted 3 lines · Oleg Nesterov
But we don't need
tasklist_lockat all, we can use rcu_read_lock/unlock. Q: don't we needtask_rq_lock()to read ->cpus_allowed "atomically" ?
Quoted 6 lines
Quoted 2 lines
right now ->cpus_allowed is protected by
tasklist_lock. We cannot do RCU here because ->cpus_allowed modifications are not RCUified.Is it so? that was my question. Afaics,
set_cpus_allowed()doesp->cpus_allowed=new_maskunderrq->lock, so I don't understand howtasklist_lockcan help.
you are right, we could (and should) make this depend on rq_lock only - i.e. just take away the tasklist_lock like your patch does. It's not like the user could expect to observe any ordering between PID lookup and affinity-mask changes. And my RCU comment is bogus: it's not like we allocate ->cpus_allowed :-/
Ingo
Quoted 10 lines · Jakub Jelinek
Quoted 8 lines · Paul Jackson
There are really at least four "number of CPUs" answers here, and we should be aware of which we are providing. There are, in order of decreasing size:
1) the size of the kernelscpumask_t(NR_CPUS),
2) the maximum number of CPUs that might ever be hotplugged into a booted system,
3) the current number of CPUs online in that system, and
4) the number of CPUs that the current task is allowed to use.
Quoted 4 lines
sysconf(_SC_NPROCESSORS_CONF)should IMHO return (2) (this currently scans/proc/cpuinfoon alpha and sparc{,64} for ((ncpus|CPUs) probed|cpus detected) and for the rest just returnssysconf(_SC_NPROCESSORS_ONLN)).
Neither of the sysconf returned values should be affected by affinity.
I'm looking at an ia64 system, and when a cpu is hot-unplugged it is removed from /proc/cpuinfo. Wouldn't /sys/devices/system/cpu/ be a better source for 2) ?
Signature · 4 lines
Cliff Wickman Silicon Graphics, Inc. [email protected] (651) 683-3824