Memory Paging and fork copy-on-write semantics
Hi,
I have been making use of copy-on-write semantics of linux fork to duplicate a process around 100+ times to generate client load against a server. The copy-on-write allows me to run many more processes without swap thrashing than I'd otherwise be able to. The client code is in perl, so the process sizes are in the MBs. Using this technique I only need about 2 MB per user, with around 5.5 MB shared.
What I've found is that everything works fine, so long as I don't run so many clients that the kernel pages out part of a process. When this happens, it seems (from looking at top output) that the shared memory is copied when it is paged-out. What's worse is it seems that it is copied for each process that is sharing it. The net effect is that one page is gained, but many more pages are created in the other processes that were sharing the memory. I typically see shared memory in each perl process drop down to less than 2 MB when this occurs, so each process now consumes about 6 MB of unshared memory (or so top tells me).
The upshot is the VM system thrashes really badly, because now it has to swap more data out to disk. The sytem becomes unresponsive and I have to wait minutes for keystrokes to be recognised (think CTRL-C ;) ). The good news is that when the scripts complete, the system seems completely normal.
My sytem is a PIII-550 intel BX chipset 7200 RPM IDE drive with 384 MB RAM if anyone's interested.
Is this expected and reasonable behaviour? Is it possible for pages to remain shared, even when they are swapped to disk? Does that already happen anyway, meaning my analysis of the situation is off base?
I am currently running a vanilla 2.4.12, but have observed the same kind of behaviour in all linux 2.4.x series kernel with the client test application. I haven't tried it with 2.2.x or Alan Cox's latest kernel series.
Thanks for listening and thanks to all the kernel hackers for all their hard work.
Cheers,
Steve Butler
Quoted 6 lines · Steven Butler
I have been making use of copy-on-write semantics of linux fork to duplicate a process around 100+ times to generate client load against a server. The copy-on-write allows me to run many more processes without swap thrashing than I'd otherwise be able to. The client code is in perl, so the process sizes are in the MBs. Using this technique I only need about 2 MB per user, with around 5.5 MB shared.
[snip COW undone on swapout, leading to thrashing]
Quoted 1 line
Is this expected and reasonable behaviour?
Absolutely not, this is not supposed to happen.
Quoted 2 lines
Is it possible for pages to remain shared, even when they are swapped
to disk?
I think this already happens in the -ac kernels, probably in -linus too (though I'm not 100% sure).
Quoted 2 lines
Does that already happen anyway, meaning my analysis of the situation is off base?
Possible, but it's also possible you ran into a real bug, it would be interesting to debug this further...
I wouldn't rule out a bug with the remove-from-swapcache logic on swapin, either.
regards,
Rik
Signature · 3 lines
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/ http://www.surriel.com/ http://distro.conectiva.com/
Quoted 8 lines · Steven Butler
I have been making use of copy-on-write semantics of linux fork to duplicate a process around 100+ times to generate client load against a server. The copy-on-write allows me to run many more processes without swap thrashing than I'd otherwise be able to. The client code is in perl, so the process sizes are in the MBs. Using this technique I only need about 2 MB per user, with around 5.5 MB shared.
Quoted 9 lines
What I've found is that everything works fine, so long as I don't run so many clients that the kernel pages out part of a process. When this happens, it seems (from looking at top output) that the shared memory is copied when it is paged-out. What's worse is it seems that it is copied for each process that is sharing it. The net effect is that one page is gained, but many more pages are created in the other processes that were sharing the memory. I typically see shared memory in each perl process drop down to less than 2 MB when this occurs, so each process now consumes about 6 MB of unshared memory (or so top tells me).
It's to be expected that "top" (or 3rd field of /proc/pid/statm) will tell you that under heavy swapout. It counts a page as "shared" if it's currently mapped into more than one process (approx. description), and swapout is doing its best to remove the mappings from processes, to free up pages later; the usage count of the corresponding data, which would include the swap usage count, is not being counted here (nor should it be). If you were to swapoff, ideally you would find those shared counts going back up to what they were; however, those processes do need that swap, so in practice swapoff will probably fail or hang or cause OOM kills instead.
Hugh