Odd swapping issue
It happened twice recently(within one week), therefore I'm posting here.
32bit Fedora7, 2.6.23.17-88.fc7PAE, 16G ram, 12G swap, 2*dualXeon
My machine went sluggish. Tried to debug it, and observed, that free ram(the unused one) was increasing while swap usage was increasing as well. It stopped after whole swap partition was used. Then it came back normal. Required processes were brought back to ram and server become responsible again.
Now the most strange thing is that there was 5G free(unused) ram, during those few minutes it growed to 12G! In the very same time swap usage increased from ~4G to 12G. This is irrational to me. Can anybody explain? How to prevent?
vm.swappiness was decreased to 30, to avoid excessive swap usage, but no avail.
It's application and nfs server/client if it matters. During that "memory sweep" I observed many nfsd processes(kernel threads?) in D state.
Quoted 3 lines · Rafal Wijata
It happened twice recently(within one week), therefore I'm posting here.
32bit Fedora7, 2.6.23.17-88.fc7PAE, 16G ram, 12G swap, 2*dualXeon
16g ram on a 32bit kernel is stretching it... may I suggest running a 64bit kernel?
Quoted 3 lines · Peter Zijlstra
16g ram on a 32bit kernel is stretching it... may I suggest running a 64bit kernel?
Is it? Do we expect that behaviour with so much RAM?
Quoted 6 lines · David Newall
Quoted 3 lines · Peter Zijlstra
16g ram on a 32bit kernel is stretching it... may I suggest running a 64bit kernel?
Is it? Do we expect that behaviour with so much RAM?
The memory must be mapped into the 4GB (32-bit) virtual address space to be used. Having this much memory on a system with 32 bit pointers leads to a lot of VM overhead and strange behavior.
Cheers,
Dan
Signature · 3 lines
/--------------- - - - - - -
| Dan Noé
| http://isomerica.net/~dpn/Quoted 6 lines · David Newall
Quoted 3 lines · Peter Zijlstra
16g ram on a 32bit kernel is stretching it... may I suggest running a 64bit kernel?
Is it? Do we expect that behaviour with so much RAM?
I expect your 32bit machine to go wonky after say 8gb,.. maybe even earlier for some workloads.
The thing is, 32bit machines have less than 1gb of kernel memory - the more memory you have the less of that remains for system duties since such things as the page frame array grow with the number of physical pages present.
Also, all such things as buffer cache reside in that tiny space. At some point the kernel itself is just going to run out of memory trying to manage your machine.
Hi Rafal
Quoted 4 lines
vm.swappiness was decreased to 30, to avoid excessive swap usage, but no avail.
It's application and nfs server/client if it matters. During that "memory sweep" I observed many nfsd processes(kernel threads?) in D state.
I think no related.
nfsd always remain D state.
because if nfsd waiting to new packet, ps (or other tool) display "D".
OK, things are little more clear now I suppose.
I tried (just for fun) echo 8 > /proc/sys/vm/nr_hugepages and it happened again, and no hugepages were allocated of course.
But this brought the following to my mind.
Since it's nfs server/client, and the nfs is rather heavily utilized(at least sometimes), maybe nfsd tasks were unable to find long enough lowmem chunk. How can we get such ram? By sweeping pages out to swap, isn't it?
Now the question is, if Linux actually has such feature like blind swapping out if lowmem is too fragmented to find requested size? Can anybody confirm or negate?
Next, I was thinking then I could run 64bit system, but I don't want to reinstall it. Anybody knows if it's possible to have installed 32bit OS, but run 64bit kernel alone(no 64bit glibc or anything else)?
Thanx in advance.
Quoted 7 lines · Rafal Wijata (NULL)
OK, things are little more clear now I suppose.
I tried (just for fun) echo 8 >
/proc/sys/vm/nr_hugepagesand it happened again, and no hugepages were allocated of course.But this brought the following to my mind.
Quoted 7 lines
Since it's nfs server/client, and the nfs is rather heavily utilized(at least sometimes), maybe nfsd tasks were unable to find long enough lowmem chunk. How can we get such ram? By sweeping pages out to swap, isn't it?
Now the question is, if Linux actually has such feature like blind swapping out if lowmem is too fragmented to find requested size?
Can anybody confirm or negate?
Well, not blind - we have lumpy reclaim that tries to focus on the right spot for large order allocations. However there are a myrad of conditions that can make it impossible to reclaim certain pages, therefore defeating this strategy.
Quoted 3 lines
Next, I was thinking then I could run 64bit system, but I don't want to reinstall it. Anybody knows if it's possible to have installed 32bit OS, but run 64bit kernel alone(no 64bit glibc or anything else)?
Yes running as I type ;-)