Memory Measurements and Lots of Files and Inodes

Hello All,

I've got a box with 750MB of RAM.. for the sake of this test, I've turned off swap. I also have a little program that creates and deletes lots of little files on a RAMDISK with ext2 on it. The ramdisk is only 4MB max and the files are about 3K and there are never more than four of them at a time - but the program is constantly creating and deleting hundreds of thousands of them. I have no reason to think this issue is unique to the ramdisk - it just makes files faster.

Just sitting back and watching vmstat while this runs my 'free' memory drops from ~600MB to about ~16MB.. the buffers and cache remain roughly constant.. at 16MB some sort of garbage collection kicks in - there is a notable system pause and ~70MB moves from the used to the 'free' column... this process repeats more or less in a steady state.

If, while this is going on, I run another little app that does {x= malloc(300MB), memset (x,0,300MB), free (x)}.. suddenly I can move 300MB from the used to the 'free' state...

I assume there is some VFS structure that is growing (perhaps related to the tremendous # of inodes I'm going through) and is only cleansed when the number of free pages gets too low.. I'd be interested in any details someone could provide.

My real question is about the commonly (?) used metric of free+buffers+cached representing the size of an allocation that could succeed.. my 300MB allocation above proves that doesn't apply here.. although, if I turn on strict accounting, that 300MB does fail even though there is only perhaps 150MB of userspace allocations outstanding on the box.

Can anybody provide a better metric for "ram free for userspace allocations"?

Thanks,
-Patrick

Probably `negative dentries' - cached directory entries which say "this file isn't there" so we don't need to go into the fs to find that out.

If you could share your test apps that would help a lot.

The slab cache (general kernel memory allocator+cache) has mechanisms for freeing cached objects when memory gets tight. That will recycle all those negative dentries. If that's what you're seeing.

On your machine it'll be "all of swap plus all of physical memory minus whatever malloc'ed memory you're using now minus 8-12 megabytes". There isn't much memory which cannot be reclaimed unless you have a huge machine or you're doing odd things.

[Andrew Morton: Dec 11 16:09]

sure! this is the "lots of files" program.

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

#define BSZ 2000
unsigned char buf[BSZ+256];

main ()

{
    int fd,i,j;
    unsigned int ctr=0;
    unsigned char r=0;
    char f1[32],f2[32],f3[32],f4[32];
    
    while (1)
    {
        sprintf (f1,"fxa_%06d",ctr++);  sprintf (f2,"fxa_%06d",ctr++);
        sprintf (f3,"fxa_%06d",ctr++);  sprintf (f4,"fxa_%06d",ctr++);

        fd = open (f1,O_CREAT | O_RDWR);
        memset (buf,r++,BSZ+256);  write (fd,buf,BSZ+r);    close (fd);
        
        fd = open (f2,O_CREAT | O_RDWR);
        memset (buf,r++,BSZ+256);   write (fd,buf,BSZ+r);    close (fd);
        
        fd = open (f3,O_CREAT | O_RDWR);
        memset (buf,r++,BSZ+256);   write (fd,buf,BSZ+r);     close (fd);
        
        fd = open (f4,O_CREAT | O_RDWR);
        memset (buf,r++,BSZ+256);  write (fd,buf,BSZ+r);      close (fd);
        
        unlink (f1);   unlink (f2);    unlink (f3);   unlink (f4);
        if (!r)
          fprintf (stdout,".");fflush (stdout);
    }
}

and this is the "gimme 300MB even though free sez I hardly have anything" program:

#define BS (30*1024*1024)
main()

{
    char *x;
    x = malloc (BS);
    if (!x)
        printf ("heh - x is NULL\n");
    else
    {
        printf ("allocd\n");
        memset (x,0x31,BS);
        printf ("set\n");
        free (x);
        printf ("freed\n");
    }
}

this is useful advice, thanks. Basically what the new procps does?

Yep, negative dentries:

      dentry_cache:   149092KB   149092KB  100.0 
  ext3_inode_cache:     3425KB    10102KB   33.90
   radix_tree_node:      594KB     1612KB   36.88
       buffer_head:      543KB     1264KB   42.99

You can monitor these via /proc/slabinfo, or using Bill's bloatmeter script from http://www.zip.com.au/~akpm/linux/patches/stuff/

I don't know what procps does.

But your fill-up-all-memory program will certainly do the trick. Run it, create a huge swapstorm (or get oom-killed if there's no swap) and then see what `free' says. That's as much memory as the kernel will ever give you.

This is a constant battle for programmers trying to develop sophisticated
applications for sophisticated operating systems.

If all of the applications are cooperating, you can do it fairly easily.
First, ask the kernel how much physical memory there is. Then fudge a reduction based on kernel usage. Then subtract the amount of RAM each application is 'really using' by asking it.

If you write all the applications, this isn't hard to do. If you have to
cooperate with other applications you didn't write, things get trickier. If you're the only application, things are really easy.

There are any number of heuristics and guesstimates. Ultimately, I recommend
mostly leaving your memory usage user-configurable.

DS