Skip to contentSkip to conversation

buffer/memory strangeness in 2.4.16

DISCUSSION [email protected] 3 participants 7 messages Nov 29, 2001 – Dec 14, 2001 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hi,

I am experiencing a bit of strange system behaviour in a vanilla 2.4.16 kernel (2.95.3, very stable machine etc.)

I noticed, that after running for a while (day) I had significantly less memory available for my simulation program than right after booting. Looking at the problem using 'xosview' (or 'free'), I noticed that there was a large number of MBs filled with 'buffers' that did not get wiped when other programs need the memory. The system seems to rather kill an 'offender' than clean out buffers.

Right after booting, I can allocate about 650MBs memory using the little program attached below. After a day (or after running updatedb), under the same conditions, even in single user mode with only a shell running (!) this is not possible anymore and the program (below), trying to allocate only 300-400MBs, gets killed by the system after making it unresponsive for many seconds.

Apparently this problem occurs after running 'updatedb', which fills 'free memory' and generates lots of filled cache and buffers on my system.

This sort of behaviour must have been introduced after 2.4.13, which does not show these problems.

Please tell me if somebody needs more information to debug this, or if this behaviour is normal or expected. Please cc: me as I am only on lkml-digest.

Cheers
Jan

P.S. All RAM slots are full, so please don't suggest buying more memory as a solution :^)

-------------------%<-----------------------

#include <stdio.h>
#define ONE_MEG 1024 * 1024

main ()

{
        long mem_avail = ONE_MEG;
        char *buf;
        char userchar = '@';
        int howmany;

        while (1)
        {
                printf ("Number of MBs to allocate? ");
                scanf ("%d", &howmany);
                printf ("Trying to allocate %ld bytes: ", mem_avail*howmany);

                getchar ();
                if ((buf = (char *) malloc ((size_t) mem_avail*howmany))){
                        printf (" success!\n");
                        printf ("Now filling it up...\n");
                        memset (buf, userchar, mem_avail * howmany);
                        printf ("Hit ENTER to free the memory.\n");
                        getchar ();
                        free (buf);
                } else {
                        printf (" failed :(\n");
                }
        }
}

-------------------%<-----------------------

#2 lore raw
In reply to a message not in the archive
Quoted 6 lines · [email protected] wrote:

Hi,

I am experiencing a bit of strange system behaviour in a vanilla 2.4.16 kernel (2.95.3, very stable machine etc.)

Quoted 8 lines

I noticed, that after running for a while (day) I had significantly less memory available for my simulation program than right after booting. Looking at the problem using 'xosview' (or 'free'), I noticed that there was a large number of MBs filled with 'buffers' that did not get wiped when other programs need the memory. The system seems to rather kill an 'offender' than clean out buffers.

Seconded. After an updatedb run, my 768 megabyte 2.5.1-pre4 machine
shows:

total used free shared buffers cached

Mem:        770668     384460     386208          0     138548      17744
-/+ buffers/cache:     228168     542500

and, after malloc/memset of 700 megs:

total used free shared buffers cached

Mem:        770668      73340     697328          0      41160       5960
-/+ buffers/cache:      26220     744448
Swap:       499928      18628     481300

I repeated the malloc/memset a few times, wrote a gigabyte file and was unable to make the 40 megabytes of buffermem go away.

MemTotal:       770668 kB
MemFree:        698008 kB
MemShared:           0 kB
Buffers:         42092 kB
Cached:           6088 kB
SwapCached:       9808 kB
Active:          48064 kB
Inactive:        10112 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       770668 kB
LowFree:        698008 kB
SwapTotal:      499928 kB
SwapFree:       484512 kB

After running an extremely memory-intensive test program for two minutes, buffermem fell to 38 megabytes.

Seems broken to me.

-

#3 lore raw

Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2

In reply to a message not in the archive

It is still the same in 2.4.17pre2 :-(

Quoted 49 lines · Andrew Morton
Quoted 11 lines · [email protected] wrote:

Hi,

I am experiencing a bit of strange system behaviour in a vanilla 2.4.16 kernel (2.95.3, very stable machine etc.)

I noticed, that after running for a while (day) I had significantly less memory available for my simulation program than right after booting. Looking at the problem using 'xosview' (or 'free'), I noticed that there was a large number of MBs filled with 'buffers' that did not get wiped when other programs need the memory. The system seems to rather kill an 'offender' than clean out buffers.

Seconded. After an updatedb run, my 768 megabyte 2.5.1-pre4 machine
shows:

total used free shared buffers cached

Mem:        770668     384460     386208          0     138548      17744
-/+ buffers/cache:     228168     542500

and, after malloc/memset of 700 megs:

total used free shared buffers cached

Mem:        770668      73340     697328          0      41160       5960
-/+ buffers/cache:      26220     744448
Swap:       499928      18628     481300

I repeated the malloc/memset a few times, wrote a gigabyte file and was unable to make the 40 megabytes of buffermem go away.

MemTotal:       770668 kB
MemFree:        698008 kB
MemShared:           0 kB
Buffers:         42092 kB
Cached:           6088 kB
SwapCached:       9808 kB
Active:          48064 kB
Inactive:        10112 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       770668 kB
LowFree:        698008 kB
SwapTotal:      499928 kB
SwapFree:       484512 kB

After running an extremely memory-intensive test program for two minutes, buffermem fell to 38 megabytes.

Seems broken to me.

#4 lore raw

Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2

In reply to a message not in the archive
Quoted 3 lines · [email protected] wrote:

It is still the same in 2.4.17pre2 :-(

Yes, well, it would be.

It seems that what's happening is that all your buffercache memory is on the active list, and all your anonymous memory is on the
inactive list. Consequently the logic in shrink_caches() which
is supposed to move pages onto the inactive list doesn't do anything - it just sits there calling refill_inactive() with a zero argument all the time.

You'll find that if you push the machine really hard - allocate 1.5x physical memory and touch it all then the VM will, eventually, with great reluctance and much swapping, relinquish the 30 megabytes of buffercache memory. But it's out of whack.

I tested the pre7aa1 patches and it seemed possibly a little better, but not right.

If we put anon pages on the active list instead, then shrink_caches() and refill_inactive() start to do something, and they move that stale old buffercache memory onto the inactive list where it can be freed.

This is just a random hack. I don't understand what's going on in the VM, let alone what's *supposed* to be going on. And given the state of documentation available to us, I never will.

mm/memory.c +1 -0
@@ -1174,6 +1174,7 @@ static int do_anonymous_page(struct mm_s
 		flush_page_to_ram(page);
 		entry = pte_mkwrite(pte_mkdirty(mk_pte(page, vma->vm_page_prot)));
 		lru_cache_add(page);
+		activate_page(page);
 	}
 
 	set_pte(page_table, entry);

-

#5 lore raw

Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2

Quoted 4 lines · Andrew Morton

You'll find that if you push the machine really hard - allocate 1.5x physical memory and touch it all then the VM will, eventually, with great reluctance and much swapping, relinquish the 30 megabytes of buffercache memory. But it's out of whack.

This is an expected (and very bad) side effect of use-once.

Quoted 3 lines

If we put anon pages on the active list instead, then shrink_caches() and refill_inactive() start to do something, and they move that stale old buffercache memory onto the inactive list where it can be freed.

This would fix the problem of not being able to evict stale active pages, but I have no idea if it would unbalance something else.

Quoted 3 lines

This is just a random hack. I don't understand what's going on in the VM, let alone what's *supposed* to be going on. And given the state of documentation available to us, I never will.

The balancing in Andrea's VM is just too subtle to understand without docs, that is, if there is any particular idea behind it and it isn't just experimentation.

regards,

Rik

Signature · 3 lines
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/
#6 lore raw

Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2

I verified this by throwing in 1GB of swap. It does lose the buffers eventually, but it does so under great pains. However, since I do have *plenty* memory and slow disks, I prefer running without swap. These settings appear to be broken (2.4.13 worked!).

Without swap, I would have to reboot every day to make my simulation happy.- But it needs to allocate and use *only* about 300 of 768MB which *should* be available, only because the night before a cron job kicked off 'updatdb' filling the buffers.

Without swap, it really is *this* bad in 2.4.16 and the latest pre-releases. RAM is full of buffers that just won't disappear to make room for more important stuff. My simulation gets killed *every time* until I reboot to free 'buffers' or add swap. (This makes everything slower in total.)

:-(

Jan

Quoted 27 lines · Rik
Quoted 4 lines · Andrew Morton

You'll find that if you push the machine really hard - allocate 1.5x physical memory and touch it all then the VM will, eventually, with great reluctance and much swapping, relinquish the 30 megabytes of buffercache memory. But it's out of whack.

This is an expected (and very bad) side effect of use-once.

Quoted 3 lines

If we put anon pages on the active list instead, then shrink_caches() and refill_inactive() start to do something, and they move that stale old buffercache memory onto the inactive list where it can be freed.

This would fix the problem of not being able to evict stale active pages, but I have no idea if it would unbalance something else.

Quoted 3 lines

This is just a random hack. I don't understand what's going on in the VM, let alone what's *supposed* to be going on. And given the state of documentation available to us, I never will.

The balancing in Andrea's VM is just too subtle to understand without docs, that is, if there is any particular idea behind it and it isn't just experimentation.

regards,

Rik

#7 lore raw

Re: buffer/memory strangeness in 2.4.16; fixed in 2.4.17-rc1

In reply to a message not in the archive

Hi,

just to let you know the problem described below seems to have been fixed in 2.4.17-rc1. It behaves as it should. Great work!

Cheers,
Jan

Quoted 24 lines · you

Hi,

I am experiencing a bit of strange system behaviour in a vanilla 2.4.16 kernel (2.95.3, very stable machine etc.)

I noticed, that after running for a while (day) I had significantly less memory available for my simulation program than right after booting. Looking at the problem using 'xosview' (or 'free'), I noticed that there was a large number of MBs filled with 'buffers' that did not get wiped when other programs need the memory. The system seems to rather kill an 'offender' than clean out buffers.

Right after booting, I can allocate about 650MBs memory using the little program attached below. After a day (or after running updatedb), under the same conditions, even in single user mode with only a shell running (!) this is not possible anymore and the program (below), trying to allocate only 300-400MBs, gets killed by the system after making it unresponsive for many seconds.

Apparently this problem occurs after running 'updatedb', which fills 'free memory' and generates lots of filled cache and buffers on my system.

This sort of behaviour must have been introduced after 2.4.13, which does not show these problems.