repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

We've been experiencing a problem where an errant process would run in a tight loop trying to create files in a directory where it did not have access. While this errant process was running, we'd notice all of the available memory shift from buffers/cache (or free) to used and stay that way while the process was running. vmstat also reports heavy in/out traffic on the swap, but swap consumption does not grow past a few dozen megabytes. The memory used by the process itself does not grow.

Note that we increase the default values for certain FS parameters:

echo '16384' >/proc/sys/fs/super-max
echo '32768' >/proc/sys/fs/file-max
echo '65535' > /proc/sys/fs/inode-max

We've created the following test program, which excercises the problem on both 2.2.19 and 2.4.7. All of the machines have at least 512mb of memory. I'd appreciate any feedback about reproducing the problem and potential causes/fixes.

Thanks.

-------- Original Message --------
Subject: memory consumption
Date: Wed, 1 Aug 2001 16:33:18 -0400 (EDT)
From: Dave Johnson <[email protected]>
Reply-To: [email protected]
To: [email protected]

$ mkdir testdir
$ chmod a-w testdir
$ ./open_test testdir/test

strace shows:

open("testdir/test.10001051", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES (Permission denied)
open("testdir/test.10001052", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES (Permission denied)
open("testdir/test.10001053", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES (Permission denied)
open("testdir/test.10001054", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES (Permission denied)

memory starts getting used until the system begins to swap like crazy....

interestingly.. the filename needs to change, and it wont happen for a while if you start with 0.

Happens with both 2.4.7 and 2.2.19

-----

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int main (int argc, char *argv[])
{
int i;
int fd;
char name[100];

if (argc != 2)

    return 1;

  for (i=10000000; i>=0; i++)
    {

      snprintf(name,99,"%s.%d",argv[1],i);

      fd = open(name, O_WRONLY|O_CREAT|O_EXCL, 0666);
      if (fd >= 0) close(fd);

    }

return 0;
}

--------

David Johnson                       Starent Networks, Corp.
978-851-1173                        30 International Place
[email protected]        Tewksbury, MA 01876

You write:

You are probably creating negative dentries. Check /proc/slabinfo for the number of dentries, and it will confirm this. I'm not sure why that would cause swapping, but then again I haven't checked the policy for shrinking the dentry cache recently, and there have been a number of changes in that area lately.

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Yow! Right on. On 2.2.19 and 2.4.7, the line for dentry_cache in /proc/slabinfo skyrockets while the test program is running. Also, on 2.2.19 but not 2.4.7 the line for size-32 climbs steadily at around the same pace as dentry_cache when the test program is running. After I stop the test program, the number slowly declines as other processes allocate memory.

Brian Ristuccia

So it is identified, but still probably needs to be fixed in some way. Otherwise, you could potentially have DOS from someone trying to read or create files, even if they don't have write permission _anywhere_ on the system.

Looking at the code, it appears that if we call shrink_dcache_memory() from while trying to allocate memoty for a filesystem, it returns without doing anything, to avoid a deadlock. Al Viro and/or Marcelo Tosatti probably know how to fix this.

Cheers, Andreas

Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

In article <[email protected]> you write:

Doesn't matter.

The thing that you end up hitting looks very much like just tons of negative dentries lying around.

If you are even moderately confident about playing around with the kernel, I would suggest testing the following approach (if you aren't comforable with playing with the kernel, I hope somebody else is willing to try this out, or I could try to cook up a patch to test later this week)

- add a new dentry list in addition to the current 'dentry_unused' list in linux/fs/dentry.c:

	static LIST_HEAD(dentry_unused_negative);

- when 'dput()' adds a dentry to the old 'dentry_unused' list it would instead check whether the dentry is negative, and if so add it to the negative list instead:

	list = &dentry_unused;
	if (!dentry->d_inode)
		list = &dentry_unused_negative;
	list_add(&dentry->d_lru, list);

- add a new "shrink_negative_dentries()" function that just does something like

	struct list_head *tmp;

	spin_lock(&dcache_lock);
	for (;;) {
		struct list_head * tmp = dentry_unused_negative.next;
		if (tmp == &dentry_unused_negative)
			break;
		struct dentry *dentry = list_entry(tmp, struct dentry, d_lru);
		tmp = tmp->next;
		if (atomic_read(&dentry->d_count))
			BUG();
		if (dentry->d_inode)
			BUG();
		prune_one_dentry(dentry);
		/* prune_one_dentry() releases the lock */
		spin_lock(&dcache_lock);
	}
	spin_unlock(&dcache_lock);

- make all the things that shrink dentries (notably the shrink_dcache_memory() function) call the above function first.

Does that fix the behaviour for you?

Linus

That will kill _all_ negative dentries whenever we get any amount of memory pressure. For stuff a-la $PATH it will get very ugly - currently we have a lot of negative dentries in /usr/local/bin that prevent tons of bogus lookups there,

I agree. I'm a big fan of negative dentries myself.

However, I'd like to see what the patch does for the bad case first, and then we can see whether there are less drastic methods (like only killing half of the negative dentries or something).

The current behaviour obviously has some problems, and negative dentries _are_ different from positive ones. Trond (or somebody - maybe it was Neil) also mentioned that negative dentries tend to hurt some NFS benchmarks by virtue of filling up memory a bit too much.

Btw, my fix description wasn't perfect - you need to get things like the unused dentry counts right too, etc. The basic approach stands, though.

Linus

In article <[email protected]> you write:

Note that negative dentries are different, and can be happily thrown away even from a filesystem context - so the suggested fix with special handling for negative dentries (another email) should work fine.

Linus

Removing the "second chance" logics for negative dentries might be a good start...

Both the "second chance" logic and pure fifo are bad.

Something like Daniel Phillips' "use once" logic would be fine for dentries, possibly even simpler.

Dentries could start their live at the head of the "active" list so they are the first to be moved to the "reclaim me" list.

If they get referenced while on the second list, we move them to the tail of the active list.

As a balancing rule, we could tune the system to always keep half of the dentries in the "reclaim me" list.

regards,

Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"

		http://www.surriel.com/

http://www.conectiva.com/ http://distro.conectiva.com/