Skip to contentSkip to conversation

Odd swapping issue

DISCUSSION Rafal Wijata 5 participants 8 messages Jul 23, 2008 – Jul 24, 2008

lore
Share & tools
lore
#0 lore raw

It happened twice recently(within one week), therefore I'm posting here.

32bit Fedora7, 2.6.23.17-88.fc7PAE, 16G ram, 12G swap, 2*dualXeon

My machine went sluggish. Tried to debug it, and observed, that free ram(the unused one) was increasing while swap usage was increasing as well. It stopped after whole swap partition was used. Then it came back normal. Required processes were brought back to ram and server become responsible again.

Now the most strange thing is that there was 5G free(unused) ram, during those few minutes it growed to 12G! In the very same time swap usage increased from ~4G to 12G. This is irrational to me. Can anybody explain? How to prevent?

vm.swappiness was decreased to 30, to avoid excessive swap usage, but no avail.

It's application and nfs server/client if it matters. During that "memory sweep" I observed many nfsd processes(kernel threads?) in D state.

#1 lore raw
Quoted 3 lines · Rafal Wijata

It happened twice recently(within one week), therefore I'm posting here.

32bit Fedora7, 2.6.23.17-88.fc7PAE, 16G ram, 12G swap, 2*dualXeon

16g ram on a 32bit kernel is stretching it... may I suggest running a 64bit kernel?

#2 lore raw
Quoted 3 lines · Peter Zijlstra

16g ram on a 32bit kernel is stretching it... may I suggest running a 64bit kernel?

Is it? Do we expect that behaviour with so much RAM?

#3 lore raw
Quoted 6 lines · David Newall
Quoted 3 lines · Peter Zijlstra

16g ram on a 32bit kernel is stretching it... may I suggest running a 64bit kernel?

Is it? Do we expect that behaviour with so much RAM?

The memory must be mapped into the 4GB (32-bit) virtual address space to be used. Having this much memory on a system with 32 bit pointers leads to a lot of VM overhead and strange behavior.

Cheers,
Dan

Signature · 3 lines
                    /--------------- - -  -  -   -   -
                    |  Dan Noé
                    |  http://isomerica.net/~dpn/
#4 lore raw
In reply to David Newall
Quoted 6 lines · David Newall
Quoted 3 lines · Peter Zijlstra

16g ram on a 32bit kernel is stretching it... may I suggest running a 64bit kernel?

Is it? Do we expect that behaviour with so much RAM?

I expect your 32bit machine to go wonky after say 8gb,.. maybe even earlier for some workloads.

The thing is, 32bit machines have less than 1gb of kernel memory - the more memory you have the less of that remains for system duties since such things as the page frame array grow with the number of physical pages present.

Also, all such things as buffer cache reside in that tiny space. At some point the kernel itself is just going to run out of memory trying to manage your machine.

#5 lore raw
In reply to Rafal Wijata

Hi Rafal

Quoted 4 lines

vm.swappiness was decreased to 30, to avoid excessive swap usage, but no avail.
It's application and nfs server/client if it matters. During that "memory sweep" I observed many nfsd processes(kernel threads?) in D state.

I think no related.

nfsd always remain D state.
because if nfsd waiting to new packet, ps (or other tool) display "D".

#6 lore raw
In reply to Rafal Wijata

OK, things are little more clear now I suppose.

I tried (just for fun) echo 8 > /proc/sys/vm/nr_hugepages and it happened again, and no hugepages were allocated of course.

But this brought the following to my mind.

Since it's nfs server/client, and the nfs is rather heavily utilized(at least sometimes), maybe nfsd tasks were unable to find long enough lowmem chunk. How can we get such ram? By sweeping pages out to swap, isn't it?

Now the question is, if Linux actually has such feature like blind swapping out if lowmem is too fragmented to find requested size? Can anybody confirm or negate?

Next, I was thinking then I could run 64bit system, but I don't want to reinstall it. Anybody knows if it's possible to have installed 32bit OS, but run 64bit kernel alone(no 64bit glibc or anything else)?

Thanx in advance.

#7 lore raw
Quoted 7 lines · Rafal Wijata (NULL)

OK, things are little more clear now I suppose.

I tried (just for fun) echo 8 > /proc/sys/vm/nr_hugepages and it happened again, and no hugepages were allocated of course.

But this brought the following to my mind.

Quoted 7 lines

Since it's nfs server/client, and the nfs is rather heavily utilized(at least sometimes), maybe nfsd tasks were unable to find long enough lowmem chunk. How can we get such ram? By sweeping pages out to swap, isn't it?

Now the question is, if Linux actually has such feature like blind swapping out if lowmem is too fragmented to find requested size?
Can anybody confirm or negate?

Well, not blind - we have lumpy reclaim that tries to focus on the right spot for large order allocations. However there are a myrad of conditions that can make it impossible to reclaim certain pages, therefore defeating this strategy.

Quoted 3 lines

Next, I was thinking then I could run 64bit system, but I don't want to reinstall it. Anybody knows if it's possible to have installed 32bit OS, but run 64bit kernel alone(no 64bit glibc or anything else)?

Yes running as I type ;-)