bulk cpu removal support
cover bulk cpu removal support
CPU hotremove will migrate tasks and redirect interrupts off dead cpu. To remove multiple CPUs, we should iteratively do single cpu removal. If tasks and interrupts are migrated to a cpu which will be soon removed, then we will trash tasks and interrupts again. The following patches allow remove several cpus one time. It's fast and avoids unnecessary repeated trash tasks and interrupts. This will help NUMA style hardware removal and SMP suspend/resume. Comments and suggestions are appreciated.
Thanks,
Shaohua
Quoted 8 lines · Shaohua Li
CPU hotremove will migrate tasks and redirect interrupts off dead cpu. To remove multiple CPUs, we should iteratively do single cpu removal. If tasks and interrupts are migrated to a cpu which will be soon removed, then we will trash tasks and interrupts again. The following patches allow remove several cpus one time. It's fast and avoids unnecessary repeated trash tasks and interrupts. This will help NUMA style hardware removal and SMP suspend/resume. Comments and suggestions are appreciated.
Some quantification of the benefits of adding this complexity would be appreciated. Like, how long does it take to offline all the cpus in a node serially, and how much faster is the bulk method?
Quoted 10 lines · Nathan Lynch
Quoted 8 lines · Shaohua Li
CPU hotremove will migrate tasks and redirect interrupts off dead cpu. To remove multiple CPUs, we should iteratively do single cpu removal. If tasks and interrupts are migrated to a cpu which will be soon removed, then we will trash tasks and interrupts again. The following patches allow remove several cpus one time. It's fast and avoids unnecessary repeated trash tasks and interrupts. This will help NUMA style hardware removal and SMP suspend/resume. Comments and suggestions are appreciated.
Quoted 3 lines
Some quantification of the benefits of adding this complexity would be appreciated. Like, how long does it take to offline all the cpus in a node serially, and how much faster is the bulk method?
This depends on the workload and I haven't a data at hand, sorry. Fast (less tasks/timers/softirqs migration) is one benefit. The method also means less time in stop_machine_run and less interrupt redirection, so smaller impact to the whole system.
Thanks,
Shaohua
Quoted 9 lines · Shaohua Li
CPU hotremove will migrate tasks and redirect interrupts off dead cpu. To remove multiple CPUs, we should iteratively do single cpu removal. If tasks and interrupts are migrated to a cpu which will be soon removed, then we will trash tasks and interrupts again. The following patches allow remove several cpus one time. It's fast and avoids unnecessary repeated trash tasks and interrupts. This will help NUMA style hardware removal and SMP suspend/resume. Comments and suggestions are appreciated.
Thanks shaohua, since the initial code has been collecting dust for some several months now, and I never got time to finish it.
Think you forgot to cc the ia64 crowd. We tried to do it a little bit more generic way, but some required arch assist due to some differences.
Andrew: please help stage in -mm since we would need more extensive testing and help feedback to fix any before becomming prime time.
It would also help to check if other archs have any compile issues etc..
Signature · 3 lines
Cheers, Ashok Raj - Open Source Technology Center
Quoted 9 lines · Shaohua Li
CPU hotremove will migrate tasks and redirect interrupts off dead cpu. To remove multiple CPUs, we should iteratively do single cpu removal. If tasks and interrupts are migrated to a cpu which will be soon removed, then we will trash tasks and interrupts again. The following patches allow remove several cpus one time. It's fast and avoids unnecessary repeated trash tasks and interrupts. This will help NUMA style hardware removal and SMP suspend/resume. Comments and suggestions are appreciated.
This seems an awful lot of code for something which happens so infrequently.
How big is the problem you're fixing here, and what are the user-observeable effects of these changes?
On Wed, May 10, 2006 at 11:06:06PM -0700, Andrew Morton wrote:
Hi Andrew,
Quoted 8 lines
Quoted 2 lines · Shaohua Li
CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
This seems an awful lot of code for something which happens so infrequently.
How big is the problem you're fixing here, and what are the user-observeable effects of these changes?
This is useful when say a NUMA node is being removed. With new multi-core CPUs comming up, considering a 2 core with HT, we could have up to 4 logical per socket. On NUMA node with 4 sockets, a node removal will mean we do 16 single cpu offlines. Each time the process and interrupts could end up on a CPU that might be removed just immediatly.
The same is also useful for SMP Suspend/resume cases since the logical offline is same here as well.
Even thought the code changes seem a lot, most of it is just preparation of functions ready to accept a cpumask_t instead of a single cpu like earlier. The reason we split them to smaller chunks so the scope of change is well understood with each patch.
The major changes are
- stop machine to run cpu offline functions on each cpu going offline
- prepare offline functions in offline path to take cpumask_t
- Some task migrate dead lock removal consideration that we ran into during stress test.
I know Shaohua ran tests for more than 20+ hrs with the patch, both on i386 and x86_64.
once we get some time deltas on a bigger machine it will help a lot. Iam also trying to check with some OEM';s who have such large machines for some data.. keep posted.
ashokr
Signature · 3 lines
Cheers, Ashok Raj - Open Source Technology Center
Quoted 34 lines · Ashok Raj
On Wed, May 10, 2006 at 11:06:06PM -0700, Andrew Morton wrote:
Hi Andrew,Quoted 8 lines
Quoted 2 lines · Shaohua Li
CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
This seems an awful lot of code for something which happens so infrequently.
How big is the problem you're fixing here, and what are the user-observeable effects of these changes?
This is useful when say a NUMA node is being removed. With new multi-core CPUs comming up, considering a 2 core with HT, we could have up to 4 logical per socket. On NUMA node with 4 sockets, a node removal will mean we do 16 single cpu offlines. Each time the process and interrupts could end up on a CPU that might be removed just immediatly.
The same is also useful for SMP Suspend/resume cases since the logical offline is same here as well.
Even thought the code changes seem a lot, most of it is just preparation of functions ready to accept a
cpumask_tinstead of a single cpu like earlier. The reason we split them to smaller chunks so the scope of change is well understood with each patch.The major changes are
- stop machine to run cpu offline functions on each cpu going offline
- prepare offline functions in offline path to takecpumask_t
- Some task migrate dead lock removal consideration that we ran into during stress test.
Quoted 7 lines
I know Shaohua ran tests for more than 20+ hrs with the patch, both on i386 and
x86_64.once we get some time deltas on a bigger machine it will help a lot. Iam also trying to check with some OEM';s who have such large machines for some data.. keep posted.
OK, thanks. I'm a little surprised that this patch wasn't accompanied by a problem description, really. I mean, if a single CPU offlining takes three milliseconds then why bother?
I assume it must take much longer, else you wouldn't have written the code. Have you any ballpark numbers for how long it _does_ take?
Quoted 4 lines · Andrew Morton
OK, thanks. I'm a little surprised that this patch wasn't accompanied by a problem description, really. I mean, if a single CPU offlining takes three milliseconds then why bother?
It depends on whats running at the time... with some light load, i measured wall clock time, i remember seeing 2 secs at times, but its been a long time i did that.. so take that with a pinch :-)_
i will try to get those idle and load times worked out again... the best i have is a 16 way, if i get help from big system oems i will send the numbers out
Quoted 3 lines
I assume it must take much longer, else you wouldn't have written the code. Have you any ballpark numbers for how long it _does_ take?
Signature · 3 lines
Cheers, Ashok Raj - Open Source Technology Center
Quoted 7 lines · Ashok Raj
Quoted 3 lines · Andrew Morton
OK, thanks. I'm a little surprised that this patch wasn't accompanied by a problem description, really. I mean, if a single CPU offlining takes three milliseconds then why bother?
Quoted 5 lines
It depends on whats running at the time... with some light load, i measured wall clock time, i remember seeing 2 secs at times, but its been a long time i did that.. so take that with a pinch :-)_
i will try to get those idle and load times worked out again... the best i have is a 16 way, if i get help from big system oems i will send the numbers out
Why is taking 30s to offline CPUs a problem?
M.
Quoted 7 lines · Martin Bligh
Quoted 6 lines · Ashok Raj
It depends on whats running at the time... with some light load, i measured wall clock time, i remember seeing 2 secs at times, but its been a long time i did that.. so take that with a pinch :-)_
Quoted 6 lines
Quoted 3 lines
i will try to get those idle and load times worked out again... the best i have is a 16 way, if i get help from big system oems i will send the numbers out
Why is taking 30s to offline CPUs a problem?
Well, the real problem is that for each cpu offline we schedule a RT thread kstopmachine() on each cpu, then turn off interrupts until this one cpu has removed. stop_machine_run() is a big enough sledge hammer during cpu offline and doing this repeatedly... say on a 4 socket system, where each socket=16 logical cpus.
the system would tend to get hick ups 64 times, since we do the stopmachine thread once for each logical cpu. When we want to replace a node for reliability reasons, its not clear if this hick ups is a good thing.
Doing kstopmachine() on a single system is in itself noticable, what we heard from some OEM's is this would have other app level impact as well.
With the bulk removal, we do stop machine just once, but all the 16 cpus get removed once hence there is just one hickup, instead of 64.
Less time to offline, avoid process and interrupt bouncing on and off a cpu which is just about to be offlined are almost extra fringe benefits you get with the bulk removal approach.
Signature · 3 lines
Cheers, Ashok Raj - Open Source Technology Center
Quoted 21 lines · Ashok Raj
Quoted 7 lines · Martin Bligh
Quoted 6 lines · Ashok Raj
It depends on whats running at the time... with some light load, i measured wall clock time, i remember seeing 2 secs at times, but its been a long time i did that.. so take that with a pinch :-)_
Quoted 6 lines
Quoted 3 lines
i will try to get those idle and load times worked out again... the best i have is a 16 way, if i get help from big system oems i will send the numbers out
Why is taking 30s to offline CPUs a problem?
Well, the real problem is that for each cpu offline we schedule a RT thread
kstopmachine()on each cpu, then turn off interrupts until this one cpu has removed.stop_machine_run()is a big enough sledge hammer during cpu offline and doing this repeatedly... say on a 4 socket system, where each socket=16 logical cpus.
Quoted 3 lines
the system would tend to get hick ups 64 times, since we do the stopmachine thread once for each logical cpu. When we want to replace a node for reliability reasons, its not clear if this hick ups is a good thing.
Can you provide more detail on what you mean by hiccups?
Quoted 2 lines
Doing
kstopmachine()on a single system is in itself noticable, what we heard from some OEM's is this would have other app level impact as well.
What "other app level impact"?
Quoted 2 lines
With the bulk removal, we do stop machine just once, but all the 16 cpus get removed once hence there is just one hickup, instead of 64.
Have you done any profiling or other instrumentation that identifies stopmachine as the real culprit here? I mean, it's a reasonable assumption to make, but are you sure there's not something else causing the hiccups? Perhaps contention on the cpu hotplug lock, or something wrong in the architecture cpu_disable code?
Module unload also uses stop_machine_run, iirc. Do you see hiccups with that, too?
Quoted 3 lines
Less time to offline, avoid process and interrupt bouncing on and off a cpu which is just about to be offlined are almost extra fringe benefits you get with the bulk removal approach.
Ok, so that's not the primary motivation for these patches?
Quoted 9 lines · Ashok Raj
On Wed, May 10, 2006 at 11:06:06PM -0700, Andrew Morton wrote:
Hi Andrew,Quoted 6 lines
Quoted 2 lines · Shaohua Li
CPU hotremove will migrate tasks and redirect interrupts off dead cpu.
This seems an awful lot of code for something which happens so infrequently.
Quoted 8 lines
Quoted 2 lines
How big is the problem you're fixing here, and what are the user-observeable effects of these changes?
This is useful when say a NUMA node is being removed. With new multi-core CPUs comming up, considering a 2 core with HT, we could have up to 4 logical per socket. On NUMA node with 4 sockets, a node removal will mean we do 16 single cpu offlines. Each time the process and interrupts could end up on a CPU that might be removed just immediatly.
But offlining all the cpus in a node is already something that just works. If the user is all that concerned about not thrashing the tasks running on that node, they would have a workload manager that migrates the tasks off the node before shooting down cpus. Similar argument applies to interrupt affinity.
I really haven't seen a compelling argument for why this is needed, just a bunch of handwaving so far, sorry.
Quoted 9 lines · Nathan Lynch
But offlining all the cpus in a node is already something that just works. If the user is all that concerned about not thrashing the tasks running on that node, they would have a workload manager that migrates the tasks off the node before shooting down cpus. Similar argument applies to interrupt affinity.
I really haven't seen a compelling argument for why this is needed, just a bunch of handwaving so far, sorry.
Hand waving? Dont think that was intensional though.. i think we are trying to address a real problem, if there is a reasonable alternate already that we are not aware of, no problemo...
1. Regarding process migration, someone needs to make sure they run something like a taskset away from all the cpus that are planned to be removed upfront. This needs to be done on all processes on the system.
[nick, is there an easier way to do this in today's sched infrastructure or otherwise]
2. For interrrupt migration, today when we take a cpu offline, we pick a random online cpu today. So if you have a cpu going offline, and the next logical cpu is also part of the same package, or node, we have no smarts today to keep migration away from those "to be offlined" cpus.
If we have a solution to these already, or a simpler alternative, we are open to those... and iam getting help on large system validation.. it might not be easy right away, but its comming.
Cheers,
ashok
Signature · 3 lines
Cheers, Ashok Raj - Open Source Technology Center
Quoted 8 lines · Ashok Raj
Quoted 7 lines · Nathan Lynch
But offlining all the cpus in a node is already something that just works. If the user is all that concerned about not thrashing the tasks running on that node, they would have a workload manager that migrates the tasks off the node before shooting down cpus. Similar argument applies to interrupt affinity.
Quoted 6 lines
Quoted 2 lines
I really haven't seen a compelling argument for why this is needed, just a bunch of handwaving so far, sorry.
Hand waving? Dont think that was intensional though.. i think we are trying to address a real problem, if there is a reasonable alternate already that we are not aware of, no problemo...
If the motivation for these patches is to minimize disruption of the workload when offlining a group of cpus, then I think the reasonable alternative is for the admin (or a script) to migrate sensitive tasks and interrupts to cpus that are not going to be offlined -- before offlining any cpus.
On the other hand, I'm getting the feeling that the problem you're really trying to address is that offlining lots of cpus takes a long time (on the order of a couple seconds per cpu) on your architectures.
So is it one of these two things, or a combination of both, or what?
Quoted 3 lines
1. Regarding process migration, someone needs to make sure they run something like a taskset away from all the cpus that are planned to be removed upfront. This needs to be done on all processes on the system.
I guess I don't understand what you're getting at here. Are you agreeing with me? Or are you saying that doing this is a problem?
Quoted 2 lines
2. For interrrupt migration, today when we take a cpu offline, we pick a random online cpu today.
That's an architecture-specific behavior. On powerpc, interrupts bound to a dying cpu have their affinity reset to the default (all cpus).
Quoted 4 lines
So if you have a cpu going offline, and the
next logical cpu is also part of the same package, or node, we have no smarts today to keep migration away from those "to be offlined" cpus.
Yes, and the kernel really shouldn't have to be smart about this. This is a matter of policy. The admin knows which cpus are going to be online. The admin is free to reassign the irqs before offlining cpus for optimal behavior. (Of course, the kernel should still have a sane fallback behavior, and it does.)
So, I still don't see the benefit of adding this change which works around the kernel's scheduling behavior etc. when the admin easily has the ability to mitigate the disruption by taking some preliminary measures.
Quoted 6 lines · Nathan Lynch
Quoted 5 lines
Quoted 1 line
just a bunch of handwaving so far, sorry.
Hand waving? Dont think that was intensional though.. i think we are trying to address a real problem, if there is a reasonable alternate already that we are not aware of, no problemo...
Quoted 8 lines
If the motivation for these patches is to minimize disruption of the workload when offlining a group of cpus, then I think the reasonable alternative is for the admin (or a script) to migrate sensitive tasks and interrupts to cpus that are not going to be offlined -- before offlining any cpus.
On the other hand, I'm getting the feeling that the problem you're really trying to address is that offlining lots of cpus takes a long
Think i relied to the other thread... we can invent lots of goodies to keep process and interrupt bouncing... but nothing is going to stop the kstopmachine() hammer for each logical cpu offline in any architecture:-(