buffer/memory strangeness in 2.4.16
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
I am experiencing a bit of strange system behaviour in a vanilla 2.4.16 kernel (2.95.3, very stable machine etc.)
I noticed, that after running for a while (day) I had significantly less memory available for my simulation program than right after booting. Looking at the problem using 'xosview' (or 'free'), I noticed that there was a large number of MBs filled with 'buffers' that did not get wiped when other programs need the memory. The system seems to rather kill an 'offender' than clean out buffers.
Right after booting, I can allocate about 650MBs memory using the little program attached below. After a day (or after running updatedb), under the same conditions, even in single user mode with only a shell running (!) this is not possible anymore and the program (below), trying to allocate only 300-400MBs, gets killed by the system after making it unresponsive for many seconds.
Apparently this problem occurs after running 'updatedb', which fills 'free memory' and generates lots of filled cache and buffers on my system.
This sort of behaviour must have been introduced after 2.4.13, which does not show these problems.
Please tell me if somebody needs more information to debug this, or if this behaviour is normal or expected. Please cc: me as I am only on lkml-digest.
Cheers
Jan
P.S. All RAM slots are full, so please don't suggest buying more memory as a solution :^)
-------------------%<-----------------------
#include <stdio.h>
#define ONE_MEG 1024 * 1024
main ()
{
long mem_avail = ONE_MEG;
char *buf;
char userchar = '@';
int howmany;
while (1)
{
printf ("Number of MBs to allocate? ");
scanf ("%d", &howmany);
printf ("Trying to allocate %ld bytes: ", mem_avail*howmany);
getchar ();
if ((buf = (char *) malloc ((size_t) mem_avail*howmany))){
printf (" success!\n");
printf ("Now filling it up...\n");
memset (buf, userchar, mem_avail * howmany);
printf ("Hit ENTER to free the memory.\n");
getchar ();
free (buf);
} else {
printf (" failed :(\n");
}
}
}-------------------%<-----------------------
Quoted 6 lines · [email protected] wrote:
Hi,
I am experiencing a bit of strange system behaviour in a vanilla 2.4.16 kernel (2.95.3, very stable machine etc.)
Quoted 8 lines
I noticed, that after running for a while (day) I had significantly less memory available for my simulation program than right after booting. Looking at the problem using 'xosview' (or 'free'), I noticed that there was a large number of MBs filled with 'buffers' that did not get wiped when other programs need the memory. The system seems to rather kill an 'offender' than clean out buffers.
Seconded. After an updatedb run, my 768 megabyte 2.5.1-pre4 machine
shows:
total used free shared buffers cached
Mem: 770668 384460 386208 0 138548 17744 -/+ buffers/cache: 228168 542500
and, after malloc/memset of 700 megs:
total used free shared buffers cached
Mem: 770668 73340 697328 0 41160 5960 -/+ buffers/cache: 26220 744448 Swap: 499928 18628 481300
I repeated the malloc/memset a few times, wrote a gigabyte file and was unable to make the 40 megabytes of buffermem go away.
MemTotal: 770668 kB MemFree: 698008 kB MemShared: 0 kB Buffers: 42092 kB Cached: 6088 kB SwapCached: 9808 kB Active: 48064 kB Inactive: 10112 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 770668 kB LowFree: 698008 kB SwapTotal: 499928 kB SwapFree: 484512 kB
After running an extremely memory-intensive test program for two minutes, buffermem fell to 38 megabytes.
Seems broken to me.
-
Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2
In reply to a message not in the archiveIt is still the same in 2.4.17pre2 :-(
Quoted 49 lines · Andrew Morton
Quoted 11 lines · [email protected] wrote:
Hi,
I am experiencing a bit of strange system behaviour in a vanilla 2.4.16 kernel (2.95.3, very stable machine etc.)
I noticed, that after running for a while (day) I had significantly less memory available for my simulation program than right after booting. Looking at the problem using 'xosview' (or 'free'), I noticed that there was a large number of MBs filled with 'buffers' that did not get wiped when other programs need the memory. The system seems to rather kill an 'offender' than clean out buffers.
Seconded. After an updatedb run, my 768 megabyte 2.5.1-pre4 machine
shows:total used free shared buffers cached
Mem: 770668 384460 386208 0 138548 17744 -/+ buffers/cache: 228168 542500and, after malloc/memset of 700 megs:
total used free shared buffers cached
Mem: 770668 73340 697328 0 41160 5960 -/+ buffers/cache: 26220 744448 Swap: 499928 18628 481300I repeated the malloc/memset a few times, wrote a gigabyte file and was unable to make the 40 megabytes of buffermem go away.
MemTotal: 770668 kB MemFree: 698008 kB MemShared: 0 kB Buffers: 42092 kB Cached: 6088 kB SwapCached: 9808 kB Active: 48064 kB Inactive: 10112 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 770668 kB LowFree: 698008 kB SwapTotal: 499928 kB SwapFree: 484512 kBAfter running an extremely memory-intensive test program for two minutes, buffermem fell to 38 megabytes.
Seems broken to me.
Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2
In reply to a message not in the archiveQuoted 3 lines · [email protected] wrote:
It is still the same in 2.4.17pre2 :-(
Yes, well, it would be.
It seems that what's happening is that all your buffercache memory is on the active list, and all your anonymous memory is on the
inactive list. Consequently the logic in shrink_caches() which
is supposed to move pages onto the inactive list doesn't do anything - it just sits there calling refill_inactive() with a zero argument all the time.
You'll find that if you push the machine really hard - allocate 1.5x physical memory and touch it all then the VM will, eventually, with great reluctance and much swapping, relinquish the 30 megabytes of buffercache memory. But it's out of whack.
I tested the pre7aa1 patches and it seemed possibly a little better, but not right.
If we put anon pages on the active list instead, then shrink_caches() and refill_inactive() start to do something, and they move that stale old buffercache memory onto the inactive list where it can be freed.
This is just a random hack. I don't understand what's going on in the VM, let alone what's *supposed* to be going on. And given the state of documentation available to us, I never will.
mm/memory.c +1 -0
@@ -1174,6 +1174,7 @@ static int do_anonymous_page(struct mm_sflush_page_to_ram (page); entry =pte_mkwrite (pte_mkdirty (mk_pte (page, vma->vm_page_prot)));lru_cache_add (page); +activate_page (page); }set_pte (page_table, entry);
-
Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2
Quoted 4 lines · Andrew Morton
You'll find that if you push the machine really hard - allocate 1.5x physical memory and touch it all then the VM will, eventually, with great reluctance and much swapping, relinquish the 30 megabytes of buffercache memory. But it's out of whack.
This is an expected (and very bad) side effect of use-once.
Quoted 3 lines
If we put anon pages on the active list instead, then
shrink_caches()andrefill_inactive()start to do something, and they move that stale old buffercache memory onto the inactive list where it can be freed.
This would fix the problem of not being able to evict stale active pages, but I have no idea if it would unbalance something else.
Quoted 3 lines
This is just a random hack. I don't understand what's going on in the VM, let alone what's *supposed* to be going on. And given the state of documentation available to us, I never will.
The balancing in Andrea's VM is just too subtle to understand without docs, that is, if there is any particular idea behind it and it isn't just experimentation.
regards,
Rik
Signature · 3 lines
Shortwave goes a long way: irc.starchat.net #swl http://www.surriel.com/ http://distro.conectiva.com/
Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2
I verified this by throwing in 1GB of swap. It does lose the buffers eventually, but it does so under great pains. However, since I do have *plenty* memory and slow disks, I prefer running without swap. These settings appear to be broken (2.4.13 worked!).
Without swap, I would have to reboot every day to make my simulation happy.- But it needs to allocate and use *only* about 300 of 768MB which *should* be available, only because the night before a cron job kicked off 'updatdb' filling the buffers.
Without swap, it really is *this* bad in 2.4.16 and the latest pre-releases. RAM is full of buffers that just won't disappear to make room for more important stuff. My simulation gets killed *every time* until I reboot to free 'buffers' or add swap. (This makes everything slower in total.)
:-(
Jan
Quoted 27 lines · Rik
Quoted 4 lines · Andrew Morton
You'll find that if you push the machine really hard - allocate 1.5x physical memory and touch it all then the VM will, eventually, with great reluctance and much swapping, relinquish the 30 megabytes of buffercache memory. But it's out of whack.
This is an expected (and very bad) side effect of use-once.
Quoted 3 lines
If we put anon pages on the active list instead, then
shrink_caches()andrefill_inactive()start to do something, and they move that stale old buffercache memory onto the inactive list where it can be freed.This would fix the problem of not being able to evict stale active pages, but I have no idea if it would unbalance something else.
Quoted 3 lines
This is just a random hack. I don't understand what's going on in the VM, let alone what's *supposed* to be going on. And given the state of documentation available to us, I never will.
The balancing in Andrea's VM is just too subtle to understand without docs, that is, if there is any particular idea behind it and it isn't just experimentation.
regards,
Rik
Re: buffer/memory strangeness in 2.4.16; fixed in 2.4.17-rc1
In reply to a message not in the archiveHi,
just to let you know the problem described below seems to have been fixed in 2.4.17-rc1. It behaves as it should. Great work!
Cheers,
Jan
Quoted 24 lines · you
Hi,
I am experiencing a bit of strange system behaviour in a vanilla 2.4.16 kernel (2.95.3, very stable machine etc.)
I noticed, that after running for a while (day) I had significantly less memory available for my simulation program than right after booting. Looking at the problem using 'xosview' (or 'free'), I noticed that there was a large number of MBs filled with 'buffers' that did not get wiped when other programs need the memory. The system seems to rather kill an 'offender' than clean out buffers.
Right after booting, I can allocate about 650MBs memory using the little program attached below. After a day (or after running updatedb), under the same conditions, even in single user mode with only a shell running (!) this is not possible anymore and the program (below), trying to allocate only 300-400MBs, gets killed by the system after making it unresponsive for many seconds.
Apparently this problem occurs after running 'updatedb', which fills 'free memory' and generates lots of filled cache and buffers on my system.
This sort of behaviour must have been introduced after 2.4.13, which does not show these problems.