CPU hot swap for 2.4.3 + s390 support
PATCH 1/1 CPU hot swap for 2.4.3 + s390 support
Hi,
You can find a new version of the hot swap cpu patch at:
http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch
The version for s390 (you need to first apply the 2.4.3 kernel patch available on the IBM s390 Linux website) is at:
http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch-s390
Many thanks to Heiko Carstens <[email protected]> for adding s390 support and fixing a few bugs in the initial implementation. You should be able to attach and detach CPUs depending on workload in your s390 Linux guest images :)
One of the advantages of this patch is that it removes cpu_logical_map() and cpu_number_map() which people had a tendency to get wrong.
It should also be easy to support more than BITS_PER_LONG cpus as there is no concept of online_cpu_map any more.
Anton
Has anyone looked into memory hot swap/hot add support? Especially with systems with Chipkill coming out, this would be great to support...
- Pete
Quoted 28 lines · Anton Blanchard
Hi,
You can find a new version of the hot swap cpu patch at:
http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch
The version for s390 (you need to first apply the 2.4.3 kernel patch available on the IBM s390 Linux website) is at:
http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch-s390
Many thanks to Heiko Carstens <[email protected]> for adding s390 support and fixing a few bugs in the initial implementation. You should be able to attach and detach CPUs depending on workload in your s390 Linux guest images :)
One of the advantages of this patch is that it removes
cpu_logical_map()andcpu_number_map()which people had a tendency to get wrong.It should also be easy to support more than
BITS_PER_LONGcpus as there is no concept ofonline_cpu_mapany more.Anton
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:
Has anyone looked into memory hot swap/hot add support?
Adding memory probably isn't going to be too hard... but taking existing memory off line is tricky. You have to find some way of finding all the pages that are in use and then dealing with them appropriately, and when some are locked or contain kernel data this would be extremely difficult I should think.
Especially with systems with Chipkill coming out, this would be great to support...
Chipkill?
--cw
Quoted 5 lines
Adding memory probably isn't going to be too hard... but taking existing memory off line is tricky. You have to find some way of finding all the pages that are in use and then dealing with them appropriately, and when some are locked or contain kernel data this would be extremely difficult I should think.
Wouldn't that be lot of the same issues as a "swapoff" with some portion of that in use? (except for the kernel data case of course...)
M
Quoted 3 lines · Mitch Adair
Wouldn't that be lot of the same issues as a "swapoff" with some portion of that in use? (except for the kernel data case of course...)
Basically, yes.
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/ http://distro.conectiva.com/
Send all your spam to [email protected] (spam digging piggy)
On Sat, May 05, 2001 at 11:34:16AM -0500, Mitch Adair wrote:
Wouldn't that be lot of the same issues as a "swapoff" with some portion of that in use? (except for the kernel data case of course...)
No. Swapoff makes pages allocated to userland applications in swap move back into main memory -- this is much easier because:
- the pages are on disk, we _know_ the aren't locked my mlock or pinned for IO (kiobufs, whatever)
- there are no kernel pages/buffers in this area, even harder than the above to deal with
--cw
Quoted 6 lines · Chris Wedgwood
On Sat, May 05, 2001 at 11:34:16AM -0500, Mitch Adair wrote:
Wouldn't that be lot of the same issues as a "swapoff" with some portion of that in use? (except for the kernel data case of course...)
Quoted 8 lines
No. Swapoff makes pages allocated to userland applications in swap move back into main memory -- this is much easier because:
- the pages are on disk, we _know_ the aren't locked my mlock or pinned for IO (kiobufs, whatever)
- there are no kernel pages/buffers in this area, even harder than the above to deal with
Details, details ... ;)
For the "pinned for IO" case, we can simply wait until they're unpinned, this shouldn't take too long. Mlock isn't an issue either, just remove the page from the page tables, relocate the page and put it back. This may in theory violate some mlock behaviour, but in practice it'll be no worse than normal scheduling latency.
Kernel pages/buffers are no big deal since they're accessed through the kernel's page tables (for which the same rule applies as what we do with the mlock()ed pages).
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/ http://distro.conectiva.com/
Send all your spam to [email protected] (spam digging piggy)
Quoted 9 lines · Chris Wedgwood
On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:
Has anyone looked into memory hot swap/hot add support?
Adding memory probably isn't going to be too hard... but taking existing memory off line is tricky. You have to find some way of finding all the pages that are in use and then dealing with them appropriately, and when some are locked or contain kernel data this would be extremely difficult I should think.
Actually:
1. the kernel uses virtual memory itself and accesses its data structures through page tables
2. reverse mapping stuff is easy (though it costs 8 bytes of overhead per mapped pte, probably doubling page table overhead)
This only leaves two issues, the first is device drivers and the second is the question whether we'd want the overhead needed to implement the (fairly easy) memory relocation.
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/ http://distro.conectiva.com/
Send all your spam to [email protected] (spam digging piggy)
On Sat, May 05, 2001 at 11:19:09PM -0300, Rik van Riel wrote:
This only leaves two issues, the first is device drivers and the second is the question whether we'd want the overhead needed to implement the (fairly easy) memory relocation.
How do you relocate
-- pages which are mlocked without violating RT contraints?
-- pages which contain kernel pointers and might be accessed from
interrupt context?
--cw
Quoted 9 lines · Chris Wedgwood
On Sat, May 05, 2001 at 11:19:09PM -0300, Rik van Riel wrote:
This only leaves two issues, the first is device drivers and the second is the question whether we'd want the overhead needed to implement the (fairly easy) memory relocation.
How do you relocate
-- pages which are mlocked without violating RT contraints?
Fuck RT constraints. Linux doesn't have infinitely small scheduling latencies, it's easy to copy a page without increasing the scheduling latencies much.
Quoted 2 lines
-- pages which contain kernel pointers and might be accessed from
interrupt context?
Block interrupts while copying a kernel page?
Remember that this is an emergency situation, so it's ok to use "big hammer" solutions.
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/ http://distro.conectiva.com/
Send all your spam to [email protected] (spam digging piggy)
Quoted 4 lines · [email protected] said:
How do you relocate
-- pages which are mlocked without violating RT contraints? -- pages which contain kernel pointers and might be accessed from
interrupt context?
Those two are the same problem, essentially. You have to copy the page, then map it into the same virtual address (be that userspace or kernelspace) as the old one. Mark the page readonly when you start to copy it, and have a fault handler which immediately marks it writable and returns. If the source is writable by the time you've finished the copy, repeat.
If you have to repeat yourself more than $n times, you're probably experiencing livelock. At that point, do what Rik said - to hell with the RT constraints, disable interrupts and do the copy. At least your cache is warm :)
--
dwmw2
Quoted 10 lines · Chris Wedgwood
On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:
Has anyone looked into memory hot swap/hot add support?
Adding memory probably isn't going to be too hard... but taking existing memory off line is tricky. You have to find some way of finding all the pages that are in use and then dealing with them appropriately, and when some are locked or contain kernel data this would be extremely difficult I should think.
Hrmm... I agree this is a hard problem. I know people smarter than I have been thinking about this type of problem at Compaq. While I haven't talked to them directly, my only guess would be that we'd have to hand-rewrite some page tables after copying the page contents to a new area. It's late Saturday and I really haven't thought this through fully, so I'm not even sure that would work, but it's something like how we support replicated text segments on our GS series...don't know why it wouldn't work here. *shrug*
Quoted 5 lines
Especially with systems with Chipkill coming out, this would be great to support...
Chipkill?
It's the IBM technology that works around bad memory by detecting single-bit errors and removing the chip that caused it from use. I'd think of this as a big hammer version of that in software. Besides, eventually you'll want to replace the DIMM that has the bad chip, and what better way then while the system is still running (as long as it's stable, of course ;) I'm just thinking out loud, so someone can correct me if I'm being loopy...
- Pete
On Sat, May 05, 2001 at 08:08:59PM -0400, Peter Rival wrote:
Hrmm... I agree this is a hard problem. I know people smarter than I have been thinking about this type of problem at Compaq.
It's hard with current memory allocation and management paradigms, if we wanted to abstract things more and make (break) certain rules, I'm sure it can me made to work -- the only thing is, we would loose _MUCH_ speed and efficiency (and waste much more space), so much so I doubt anyone would serious want to know about it.
We would also have to violate certain assumptions of RT applications.
While I haven't talked to them directly, my only guess would be that we'd have to hand-rewrite some page tables after copying the page contents to a new area.
That in itself isn't too bad, except if the pages are mlocked this is nasty, you have to block all access to the page during copy, very bad for RT stuff.
Not only that, what if the pages themselves have kernel allocations in them? We cannot find these (at present) let alone have _any_ idea how to move them. I guess it could be fidged to work using the MMU if we were allowed to _COMPLETELY_ lock the system duing the removal phase from all interrupts and such like... seems pretty horrible to me.
It's late Saturday and I really haven't thought this through fully, so I'm not even sure that would work, but it's something like how we support replicated text segments on our GS series...don't know why it wouldn't work here. *shrug*
There have been demonstrations in the past of this sort of thing. I think HP may have done one. Not with a commodity OS though.
Actually, I just thought of a kill, what about platforms that have physically mapped page-tables? This makes like even harder as you have to move them :(
It's the IBM technology that works around bad memory by detecting single-bit errors and removing the chip that caused it from use.
I think Solaris claims to do this right now... no idea if it works, I know of at least one Solaris 7 machine with a dicky memory bit and it keeps moaning about parity corrections so I guess it doesn't lock it out. Maybe later versions (8) do?
I'd think of this as a big hammer version of that in software.
It's much easier to do in hardware with current OS design I should think.
Besides, eventually you'll want to replace the DIMM that has the bad chip, and what better way then while the system is still running (as long as it's stable, of course ;) I'm just thinking out loud, so someone can correct me if I'm being loopy...
Again, you could do this is hardware... have the hardware route writes to the memory elsewhere and only take reads from the old memory until the 'refresh' cycles have copied all the data over. Hmm, when I think about this, doing this in the memory controller chipset seems much easier I wonder if someone hasn't actually done it...
--cw
Hey, this is cool.
How far away is the capability to "teleport" processes from one machine to another over the network? Think of the uptime!
Signature · 1 line
Dwayne C. Litzenberger - [email protected]
Quoted 1 line · Peter Rival
Has anyone looked into memory hot swap/hot add support?
How do you hotswap RAM? What happens to the data that was on the removed memory module?
Quoted 5 lines · Aaron Lehmann
Quoted 1 line · Peter Rival
Has anyone looked into memory hot swap/hot add support?
How do you hotswap RAM? What happens to the data that was on the removed memory module?
Dont know about the s390 - but on some hardware you get notification when someone presses the hotswap request button. You would then be requiried to move all the data on the memory elsewhere. Then you tell the hardware it is ok to hotswap. On many systems this will make the little LED light up to say the hotswap is safe :)
Stephen
Quoted 4 lines · Dwayne C. Litzenberger
Hey, this is cool.
How far away is the capability to "teleport" processes from one machine to another over the network? Think of the uptime!
Quoted 5 lines · Dwayne C. Litzenberger
Hey, this is cool.
How far away is the capability to "teleport" processes from one machine to another over the network? Think of the uptime!
It is here. Look at Mosix.
Signature · 5 lines
I'd rather listen to Newton than to Mundie [MS flunkie who made a speech on the evil-ness of open source]. He may have been dead for almost three hundred years, but despite that he stinks up the room less. Linus
Quoted 9 lines · Ben Ford
Quoted 5 lines · Dwayne C. Litzenberger
Hey, this is cool.
How far away is the capability to "teleport" processes from one machine to another over the network? Think of the uptime!
It is here. Look at Mosix.
No. Not for uptime.
The "responsibility" for process completion does not get delegated. A process will always be bound to it's home-node (in mosix terms), no matter how far
it's "teleported". If the home-node fails, the process won't know what hit
it.
There are good reasons why mosix let's processes depend on their home nodes.
This is not meant as backstabbing mosix, it's a great environment for a lot of things.
But it's not the universal silver bullet.
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
Quoted 13 lines
Quoted 5 lines
Quoted 3 lines
How far away is the capability to "teleport" processes from one machine to another over the network? Think of the uptime!
It is here. Look at Mosix.
No. Not for uptime.
The "responsibility" for process completion does not get delegated. A process will always be bound to it's home-node (in mosix terms), no matter how far
it's "teleported". If the home-node fails, the process won't know what hit
it.
Quoted 6 lines
There are good reasons why mosix let's processes depend on their home nodes.
This is not meant as backstabbing mosix, it's a great environment for a lot of things.
But it's not the universal silver bullet.
Take a look at
http://citeseer.nj.nec.com/299905.html
for something along the lines of what you want, I think (transparent process migration between nodes). As a bonus, it's also architecture-independent.
Bruce