Skip to contentSkip to conversation

Hotplug CPU and setaffinity?

DISCUSSION Dr. David Alan Gilbert 4 participants 5 messages Apr 23, 2005 – Apr 25, 2005

lore
Share & tools
lore
#0 lore raw

Hi,
I got to wondering how Hotplug CPU and sched_setaffinity interact;
if I have a process that has its affinity set to one CPU and some nasty person comes along and unplugs it what happens to that process- does it get scheduled onto another cpu, just not get any time or die ?

In particular I was thinking of the cases where a thread has a
functional reason for remaining on one particular CPU (e.g. if you
had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.

Dave

-----Open up your eyes, open up your mind, open up your code -------

/ Dr. David Alan Gilbert    | Running GNU/Linux on Alpha,68K| Happy  \ 
\ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/
#1 lore raw
Quoted 6 lines · Dr. David Alan Gilbert

I got to wondering how Hotplug CPU and sched_setaffinity interact;
if I have a process that has its affinity set to one CPU and some nasty person comes along and unplugs it what happens to that process- does it get scheduled onto another cpu, just not get any time or die ?

The affinity of the process is reset to the default and it is migrated to another cpu, for better or worse. The kernel assumes the admin know what he/she is doing.

Quoted 8 lines

In particular I was thinking of the cases where a thread has a
functional reason for remaining on one particular CPU (e.g. if you
had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.

Yeah. But I don't think this is an issue to be solved in the kernel. Applications that are this sensitive to cpu hotplugging need to arrange to be notified before the hotplug occurs, which I think would be best done with dbus or some other IPC.

Nathan

#2 lore raw

* Nathan Lynch ([email protected]) wrote:

Hi Nathan,
Thanks for the reply.

Quoted 11 lines
Quoted 6 lines · Dr. David Alan Gilbert

I got to wondering how Hotplug CPU and sched_setaffinity interact;
if I have a process that has its affinity set to one CPU and some nasty person comes along and unplugs it what happens to that process- does it get scheduled onto another cpu, just not get any time or die ?

The affinity of the process is reset to the default and it is migrated to another cpu, for better or worse. The kernel assumes the admin know what he/she is doing.

Yeh that's ok - is there anything that would hotplug a cpu automatically; say on receiving some MCEs ; and thus not give the admin a look in.

Quoted 8 lines
Quoted 7 lines

In particular I was thinking of the cases where a thread has a
functional reason for remaining on one particular CPU (e.g. if you
had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.

Quoted 4 lines

Yeah. But I don't think this is an issue to be solved in the kernel. Applications that are this sensitive to cpu hotplugging need to arrange to be notified before the hotplug occurs, which I think would be best done with dbus or some other IPC.

Agreed; since the kernel will reset it to the default affinity then this involvement must happen before the hotplug, if the kernel were to stop scheduling them anywhere then this is something that could be fixed up externally to the app by a hotplug type of thing after the hot un-plug happened.

Dave
-----Open up your eyes, open up your mind, open up your code -------

/ Dr. David Alan Gilbert    | Running GNU/Linux on Alpha,68K| Happy  \ 
\ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/
#3 lore raw
Quoted 9 lines
Quoted 4 lines

The affinity of the process is reset to the default and it is migrated to another cpu, for better or worse. The kernel assumes the admin know what he/she is doing.

Yeh that's ok - is there anything that would hotplug a cpu automatically; say on receiving some MCEs ; and thus not give the admin a look in.

On ppc64 we have CPU guard, which would remove a processor if it is failing. Of course, the implications of not removing such a CPU are pretty terrible.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

In particular I was thinking of the cases where a thread has a functional reason for remaining on one particular CPU (e.g. if you had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.

This sounds like a theoretical problem. Can you think of any real examples? The only cases I can think of cause performance hits, but not functional problems.

#4 lore raw
Quoted 1 line · Joel Schopp

On ppc64 we have CPU guard, which would remove a processor if it is failing. Of course, the implications of not removing such a CPU are pretty terrible.

Indeed.

Quoted 10 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

In particular I was thinking of the cases where a thread has a functional reason for remaining on one particular CPU (e.g. if you had calibrated for some feature of that CPU say its time stamp counter skew/speed). Another case would be a set of threads which had set their affinity to the same CPU and then made memory consistency or locking assumptions that wouldn't be valid if they got rescheduled onto different CPUs.

This sounds like a theoretical problem. Can you think of any real examples? The only cases I can think of cause performance hits, but not functional problems.

Well I'm not aware of anything that currently would break with it; but I was gently thinking of whether it would be possible to read cycle counters (as a faster gettimeofday) even on systems which had unsynchronised counters if you could lock the thread that did it to a particular physical cpu. But this behaviour currently makes that a bad idea; in this case it would be nicer if the kernel either just killed my process or just unscheduled it or sent it a signal.

Dave