Hotplug CPU and setaffinity?
Hi,
I got to wondering how Hotplug CPU and sched_setaffinity interact;
if I have a process that has its affinity set to one CPU and some nasty person comes along and unplugs it what happens to that process- does it get scheduled onto another cpu, just not get any time or die ?
In particular I was thinking of the cases where a thread has a
functional reason for remaining on one particular CPU (e.g. if you
had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.
Dave
-----Open up your eyes, open up your mind, open up your code -------
/ Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \ \ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex / \ _________________________|_____ http://www.treblig.org |_______/
Quoted 6 lines · Dr. David Alan Gilbert
I got to wondering how Hotplug CPU and
sched_setaffinityinteract;
if I have a process that has its affinity set to one CPU and some nasty person comes along and unplugs it what happens to that process- does it get scheduled onto another cpu, just not get any time or die ?
The affinity of the process is reset to the default and it is migrated to another cpu, for better or worse. The kernel assumes the admin know what he/she is doing.
Quoted 8 lines
In particular I was thinking of the cases where a thread has a
functional reason for remaining on one particular CPU (e.g. if you
had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.
Yeah. But I don't think this is an issue to be solved in the kernel. Applications that are this sensitive to cpu hotplugging need to arrange to be notified before the hotplug occurs, which I think would be best done with dbus or some other IPC.
Nathan
* Nathan Lynch ([email protected]) wrote:
Hi Nathan,
Thanks for the reply.
Quoted 11 lines
Quoted 6 lines · Dr. David Alan Gilbert
I got to wondering how Hotplug CPU and
sched_setaffinityinteract;
if I have a process that has its affinity set to one CPU and some nasty person comes along and unplugs it what happens to that process- does it get scheduled onto another cpu, just not get any time or die ?The affinity of the process is reset to the default and it is migrated to another cpu, for better or worse. The kernel assumes the admin know what he/she is doing.
Yeh that's ok - is there anything that would hotplug a cpu automatically; say on receiving some MCEs ; and thus not give the admin a look in.
Quoted 8 lines
Quoted 7 lines
In particular I was thinking of the cases where a thread has a
functional reason for remaining on one particular CPU (e.g. if you
had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.
Quoted 4 lines
Yeah. But I don't think this is an issue to be solved in the kernel. Applications that are this sensitive to cpu hotplugging need to arrange to be notified before the hotplug occurs, which I think would be best done with dbus or some other IPC.
Agreed; since the kernel will reset it to the default affinity then this involvement must happen before the hotplug, if the kernel were to stop scheduling them anywhere then this is something that could be fixed up externally to the app by a hotplug type of thing after the hot un-plug happened.
Dave
-----Open up your eyes, open up your mind, open up your code -------
/ Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \ \ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex / \ _________________________|_____ http://www.treblig.org |_______/
Quoted 9 lines
Quoted 4 lines
The affinity of the process is reset to the default and it is migrated to another cpu, for better or worse. The kernel assumes the admin know what he/she is doing.
Yeh that's ok - is there anything that would hotplug a cpu automatically; say on receiving some MCEs ; and thus not give the admin a look in.
On ppc64 we have CPU guard, which would remove a processor if it is failing. Of course, the implications of not removing such a CPU are pretty terrible.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
In particular I was thinking of the cases where a thread has a functional reason for remaining on one particular CPU (e.g. if you had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.
This sounds like a theoretical problem. Can you think of any real examples? The only cases I can think of cause performance hits, but not functional problems.
Quoted 1 line · Joel Schopp
On ppc64 we have CPU guard, which would remove a processor if it is failing. Of course, the implications of not removing such a CPU are pretty terrible.
Indeed.
Quoted 10 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
In particular I was thinking of the cases where a thread has a functional reason for remaining on one particular CPU (e.g. if you had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.
This sounds like a theoretical problem. Can you think of any real examples? The only cases I can think of cause performance hits, but not functional problems.
Well I'm not aware of anything that currently would break with it; but I was gently thinking of whether it would be possible to read cycle counters (as a faster gettimeofday) even on systems which had unsynchronised counters if you could lock the thread that did it to a particular physical cpu. But this behaviour currently makes that a bad idea; in this case it would be nicer if the kernel either just killed my process or just unscheduled it or sent it a signal.
Dave