Skip to contentSkip to conversation

CPU hot swap for 2.4.3 + s390 support

PATCH In review Anton Blanchard 12 participants 20 messages May 5, 2001 – May 7, 2001

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 CPU hot swap for 2.4.3 + s390 support

Hi,

You can find a new version of the hot swap cpu patch at:

http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch

The version for s390 (you need to first apply the 2.4.3 kernel patch available on the IBM s390 Linux website) is at:

http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch-s390

Many thanks to Heiko Carstens <[email protected]> for adding s390 support and fixing a few bugs in the initial implementation. You should be able to attach and detach CPUs depending on workload in your s390 Linux guest images :)

One of the advantages of this patch is that it removes cpu_logical_map() and cpu_number_map() which people had a tendency to get wrong.

It should also be easy to support more than BITS_PER_LONG cpus as there is no concept of online_cpu_map any more.

Anton

#1 lore raw

Has anyone looked into memory hot swap/hot add support? Especially with systems with Chipkill coming out, this would be great to support...

- Pete

Quoted 28 lines · Anton Blanchard

Hi,

You can find a new version of the hot swap cpu patch at:

http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch

The version for s390 (you need to first apply the 2.4.3 kernel patch available on the IBM s390 Linux website) is at:

http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch-s390

Many thanks to Heiko Carstens <[email protected]> for adding s390 support and fixing a few bugs in the initial implementation. You should be able to attach and detach CPUs depending on workload in your s390 Linux guest images :)

One of the advantages of this patch is that it removes cpu_logical_map() and cpu_number_map() which people had a tendency to get wrong.

It should also be easy to support more than BITS_PER_LONG cpus as there is no concept of online_cpu_map any more.

Anton
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
In reply to Anton Blanchard

On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:

Has anyone looked into memory hot swap/hot add support?

Adding memory probably isn't going to be too hard... but taking existing memory off line is tricky. You have to find some way of finding all the pages that are in use and then dealing with them appropriately, and when some are locked or contain kernel data this would be extremely difficult I should think.

Especially with systems with Chipkill coming out, this would be great to support...

Chipkill?

--cw

#3 lore raw
Quoted 5 lines

Adding memory probably isn't going to be too hard... but taking existing memory off line is tricky. You have to find some way of finding all the pages that are in use and then dealing with them appropriately, and when some are locked or contain kernel data this would be extremely difficult I should think.

Wouldn't that be lot of the same issues as a "swapoff" with some portion of that in use? (except for the kernel data case of course...)

M

#4 lore raw
Quoted 3 lines · Mitch Adair

Wouldn't that be lot of the same issues as a "swapoff" with some portion of that in use? (except for the kernel data case of course...)

Basically, yes.

regards,

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

#5 lore raw
In reply to Chris Wedgwood

On Sat, May 05, 2001 at 11:34:16AM -0500, Mitch Adair wrote:

Wouldn't that be lot of the same issues as a "swapoff" with some portion of that in use? (except for the kernel data case of course...)

No. Swapoff makes pages allocated to userland applications in swap move back into main memory -- this is much easier because:

- the pages are on disk, we _know_ the aren't locked my mlock or pinned for IO (kiobufs, whatever)

- there are no kernel pages/buffers in this area, even harder than the above to deal with

--cw

#6 lore raw
Quoted 6 lines · Chris Wedgwood

On Sat, May 05, 2001 at 11:34:16AM -0500, Mitch Adair wrote:

Wouldn't that be lot of the same issues as a "swapoff" with some portion of that in use? (except for the kernel data case of course...)

Quoted 8 lines

No. Swapoff makes pages allocated to userland applications in swap move back into main memory -- this is much easier because:

- the pages are on disk, we _know_ the aren't locked my mlock or pinned for IO (kiobufs, whatever)

- there are no kernel pages/buffers in this area, even harder than the above to deal with

Details, details ... ;)

For the "pinned for IO" case, we can simply wait until they're unpinned, this shouldn't take too long. Mlock isn't an issue either, just remove the page from the page tables, relocate the page and put it back. This may in theory violate some mlock behaviour, but in practice it'll be no worse than normal scheduling latency.

Kernel pages/buffers are no big deal since they're accessed through the kernel's page tables (for which the same rule applies as what we do with the mlock()ed pages).

regards,

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

#7 lore raw
In reply to Chris Wedgwood
Quoted 9 lines · Chris Wedgwood

On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:

Has anyone looked into memory hot swap/hot add support?

Adding memory probably isn't going to be too hard... but taking existing memory off line is tricky. You have to find some way of finding all the pages that are in use and then dealing with them appropriately, and when some are locked or contain kernel data this would be extremely difficult I should think.

Actually:

1. the kernel uses virtual memory itself and accesses its data structures through page tables
2. reverse mapping stuff is easy (though it costs 8 bytes of overhead per mapped pte, probably doubling page table overhead)

This only leaves two issues, the first is device drivers and the second is the question whether we'd want the overhead needed to implement the (fairly easy) memory relocation.

regards,

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

#8 lore raw
In reply to Chris Wedgwood

On Sat, May 05, 2001 at 11:19:09PM -0300, Rik van Riel wrote:

This only leaves two issues, the first is device drivers and the second is the question whether we'd want the overhead needed to implement the (fairly easy) memory relocation.

How do you relocate

-- pages which are mlocked without violating RT contraints?

-- pages which contain kernel pointers and might be accessed from
interrupt context?

--cw

#9 lore raw
Quoted 9 lines · Chris Wedgwood

On Sat, May 05, 2001 at 11:19:09PM -0300, Rik van Riel wrote:

This only leaves two issues, the first is device drivers and the second is the question whether we'd want the overhead needed to implement the (fairly easy) memory relocation.

How do you relocate

-- pages which are mlocked without violating RT contraints?

Fuck RT constraints. Linux doesn't have infinitely small scheduling latencies, it's easy to copy a page without increasing the scheduling latencies much.

Quoted 2 lines

-- pages which contain kernel pointers and might be accessed from
interrupt context?

Block interrupts while copying a kernel page?

Remember that this is an emergency situation, so it's ok to use "big hammer" solutions.

regards,

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

#10 lore raw
In reply to Chris Wedgwood
Quoted 4 lines · [email protected] said:

How do you relocate
-- pages which are mlocked without violating RT contraints? -- pages which contain kernel pointers and might be accessed from
interrupt context?

Those two are the same problem, essentially. You have to copy the page, then map it into the same virtual address (be that userspace or kernelspace) as the old one. Mark the page readonly when you start to copy it, and have a fault handler which immediately marks it writable and returns. If the source is writable by the time you've finished the copy, repeat.

If you have to repeat yourself more than $n times, you're probably experiencing livelock. At that point, do what Rik said - to hell with the RT constraints, disable interrupts and do the copy. At least your cache is warm :)

--
dwmw2

#11 lore raw
In reply to Anton Blanchard
Quoted 10 lines · Chris Wedgwood

On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:

Has anyone looked into memory hot swap/hot add support?

Adding memory probably isn't going to be too hard... but taking existing memory off line is tricky. You have to find some way of finding all the pages that are in use and then dealing with them appropriately, and when some are locked or contain kernel data this would be extremely difficult I should think.

Hrmm... I agree this is a hard problem. I know people smarter than I have been thinking about this type of problem at Compaq. While I haven't talked to them directly, my only guess would be that we'd have to hand-rewrite some page tables after copying the page contents to a new area. It's late Saturday and I really haven't thought this through fully, so I'm not even sure that would work, but it's something like how we support replicated text segments on our GS series...don't know why it wouldn't work here. *shrug*

Quoted 5 lines

Especially with systems with Chipkill coming out, this would be great to support...

Chipkill?

It's the IBM technology that works around bad memory by detecting single-bit errors and removing the chip that caused it from use. I'd think of this as a big hammer version of that in software. Besides, eventually you'll want to replace the DIMM that has the bad chip, and what better way then while the system is still running (as long as it's stable, of course ;) I'm just thinking out loud, so someone can correct me if I'm being loopy...

- Pete

#12 lore raw
In reply to Anton Blanchard

On Sat, May 05, 2001 at 08:08:59PM -0400, Peter Rival wrote:

Hrmm... I agree this is a hard problem. I know people smarter than I have been thinking about this type of problem at Compaq.

It's hard with current memory allocation and management paradigms, if we wanted to abstract things more and make (break) certain rules, I'm sure it can me made to work -- the only thing is, we would loose _MUCH_ speed and efficiency (and waste much more space), so much so I doubt anyone would serious want to know about it.

We would also have to violate certain assumptions of RT applications.

While I haven't talked to them directly, my only guess would be that we'd have to hand-rewrite some page tables after copying the page contents to a new area.

That in itself isn't too bad, except if the pages are mlocked this is nasty, you have to block all access to the page during copy, very bad for RT stuff.

Not only that, what if the pages themselves have kernel allocations in them? We cannot find these (at present) let alone have _any_ idea how to move them. I guess it could be fidged to work using the MMU if we were allowed to _COMPLETELY_ lock the system duing the removal phase from all interrupts and such like... seems pretty horrible to me.

It's late Saturday and I really haven't thought this through fully, so I'm not even sure that would work, but it's something like how we support replicated text segments on our GS series...don't know why it wouldn't work here. *shrug*

There have been demonstrations in the past of this sort of thing. I think HP may have done one. Not with a commodity OS though.

Actually, I just thought of a kill, what about platforms that have physically mapped page-tables? This makes like even harder as you have to move them :(

It's the IBM technology that works around bad memory by detecting single-bit errors and removing the chip that caused it from use.

I think Solaris claims to do this right now... no idea if it works, I know of at least one Solaris 7 machine with a dicky memory bit and it keeps moaning about parity corrections so I guess it doesn't lock it out. Maybe later versions (8) do?

I'd think of this as a big hammer version of that in software.

It's much easier to do in hardware with current OS design I should think.

Besides, eventually you'll want to replace the DIMM that has the bad chip, and what better way then while the system is still running (as long as it's stable, of course ;) I'm just thinking out loud, so someone can correct me if I'm being loopy...

Again, you could do this is hardware... have the hardware route writes to the memory elsewhere and only take reads from the old memory until the 'refresh' cycles have copied all the data over. Hmm, when I think about this, doing this in the memory controller chipset seems much easier I wonder if someone hasn't actually done it...

--cw

#15 lore raw
Quoted 5 lines · Aaron Lehmann
Quoted 1 line · Peter Rival

Has anyone looked into memory hot swap/hot add support?

How do you hotswap RAM? What happens to the data that was on the removed memory module?

Dont know about the s390 - but on some hardware you get notification when someone presses the hotswap request button. You would then be requiried to move all the data on the memory elsewhere. Then you tell the hardware it is ok to hotswap. On many systems this will make the little LED light up to say the hotswap is safe :)

Stephen

#17 lore raw
In reply to Anton Blanchard
Quoted 5 lines · Dwayne C. Litzenberger

Hey, this is cool.

How far away is the capability to "teleport" processes from one machine to another over the network? Think of the uptime!

It is here. Look at Mosix.

Signature · 5 lines
I'd rather listen to Newton than to Mundie [MS flunkie who made a speech on
the evil-ness of open source]. He may have been dead for almost three
hundred years, but despite that he stinks up the room less.

Linus
#18 lore raw
In reply to Anton Blanchard
Quoted 9 lines · Ben Ford
Quoted 5 lines · Dwayne C. Litzenberger

Hey, this is cool.

How far away is the capability to "teleport" processes from one machine to another over the network? Think of the uptime!

It is here. Look at Mosix.

No. Not for uptime.

The "responsibility" for process completion does not get delegated. A process will always be bound to it's home-node (in mosix terms), no matter how far
it's "teleported". If the home-node fails, the process won't know what hit
it.

There are good reasons why mosix let's processes depend on their home nodes.

This is not meant as backstabbing mosix, it's a great environment for a lot of things.

But it's not the universal silver bullet.

Signature · 6 lines
................................................................
:   [email protected]   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
#19 lore raw
Quoted 13 lines
Quoted 5 lines
Quoted 3 lines

How far away is the capability to "teleport" processes from one machine to another over the network? Think of the uptime!

It is here. Look at Mosix.

No. Not for uptime.

The "responsibility" for process completion does not get delegated. A process will always be bound to it's home-node (in mosix terms), no matter how far
it's "teleported". If the home-node fails, the process won't know what hit
it.

Quoted 6 lines

There are good reasons why mosix let's processes depend on their home nodes.

This is not meant as backstabbing mosix, it's a great environment for a lot of things.

But it's not the universal silver bullet.

Take a look at

http://citeseer.nj.nec.com/299905.html

for something along the lines of what you want, I think (transparent process migration between nodes). As a bonus, it's also architecture-independent.

Bruce