memory not released after using cdrecord/cdrdao (was: audio cd writing causes massive swap and crash)

#include <hallo.h>

Just FYI: we have a similar bug description in the Debian BTS, where the user reports that kernel does not release memory assigned to userspace after cdrdao or cdrecord have used it (writting in DAO mode), though he could not find what allocated this memory. For details:
http://bugs.debian.org/256871 (dump attached).

Regards,
Eduard.

-!- Znarck [[email protected]] has joined #debian.de
<Znarck> hallo!
<Znarck> join #debian
<Znarck> ls
<Znarck> hello
<Znarck> quit
-!- Znarck [[email protected]] has quit [Client Quit]

Debian Bug report logs - #256871
cdrecord triggers memory leak in kernel space

Package: cdrecord; Reported by: Jeff King <[email protected]>; Date: Tue, 29 Jun 2004 15:48:02 UTC;
Maintainer for cdrecord is Joerg Jaspert <[email protected]>; Source for cdrecord is cdrtools.

View this report as an mbox folder.

-------------------------------------------------------------------------------
Report forwarded to [email protected], Joerg Jaspert <[email protected]>:
Bug#256871; Package cdrecord. Full text available.
-------------------------------------------------------------------------------
Acknowledgement sent to Jeff King <[email protected]>:
New Bug report received and forwarded. Copy sent to Joerg Jaspert <[email protected]>. Full text available.
-------------------------------------------------------------------------------

Message received at [email protected]:

Received: (at submit) by bugs.debian.org; 29 Jun 2004 15:44:37 +0000

Return-path: <[email protected]>
Received: from 66-23-211-5.clients.speedfactory.net (peff.net) [66.23.211.5]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1BfKmj-0003gN-00; Tue, 29 Jun 2004 08:44:37 -0700
Received: (qmail 88814 invoked from network); 29 Jun 2004 15:44:40 -0000
Received: from unknown (HELO coredump.intra.peff.net) (10.0.0.2)
by peff.net with SMTP; 29 Jun 2004 15:44:40 -0000
Received: by coredump.intra.peff.net (sSMTP sendmail emulation); Tue, 29 Jun 2004 11:44:36 -0400
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: Jeff King <[email protected]>
To: Debian Bug Tracking System <[email protected]>
Subject: cdrecord triggers memory leak in kernel space
Bcc: Jeff King <[email protected]>
X-Mailer: reportbug 2.62
Date: Tue, 29 Jun 2004 11:44:36 -0400
Message-Id: <[email protected]>
Delivered-To: [email protected]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_03_25

        (1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-8.0 required=4.0 tests=BAYES_00,HAS_PACKAGE
        autolearn=no version=2.60-bugs.debian.org_2004_03_25

X-Spam-Level:

Package: cdrecord
Version: 4:2.0+a30.pre1-1
Severity: normal

I am running cdrecord as root with the following options:
cdrecord dev=/dev/hdc -v -dao -pad -audio *.wav

One CD burns OK. When I try to burn another, about mid-way through the burn I notice random apps being killed. Sure enough, dmesg reports this:
Out of Memory: Killed process 6700 (firefox-bin).
Out of Memory: Killed process 6587 (xterm).
Out of Memory: Killed process 2806 (netbiff).
Out of Memory: Killed process 2861 (xterm).
I'm fairly sure it's triggered by the cdrecord command, as I'm not doing anything else out of the ordinary and it happens consistently. Looking at the output of "free", something is using all of my memory:

total used free shared buffers cached
Mem: 1036972 1004756 32216 0 3000 72616
-/+ buffers/cache: 929140 107832
Swap: 0 0 0

Running "top" or "ps" shows that userland processes aren't responsible for this:

  $ sum=0
  $ ps -AF | awk '{print $6}' | tail +2 | while read f
    sum=$(($sum + $f))
    echo $sum
    done
  ...
  75288

I've used cdrecord many times in the past and not run into this problem. Here are the things I'm doing differently recently:
- move to 2.6.6 kernel; this is the first burn I've done with it, but I have done others with the 2.6 kernel series
- use the /dev/hdc device syntax; I believe I have done this before and it worked OK, but I'm not sure
- used -dao mode. This is DEFINITELY the first time I have tried using -dao mode.

Because of the massive amount of memory being gobbled, I can only guess that the buffer isn't being freed by the kernel for whatever reason. I suspect this is probably actually a kernel issue, but I thought I'd start here in case there have been other reports.

-- System Information:
Debian Release: testing/unstable
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: i386 (i686)
Kernel: Linux 2.6.6-1-k7-smp
Locale: LANG=C, LC_CTYPE=C

Versions of packages cdrecord depends on:

ii  debconf                     1.4.29       Debian configuration management sy
ii  libc6                       2.3.2.ds1-13 GNU C Library: Shared libraries an
ii  makedev                     2.3.1-70     Creates device files in /dev

-- debconf information:
* cdrecord/SUID_bit: false
cdrecord/MAKEDEV:
cdrecord/MAKEDEVNEW: true
cdrecord/do_it_yourself:

-------------------------------------------------------------------------------
Information forwarded to [email protected], Joerg Jaspert <[email protected]>:
Bug#256871; Package cdrecord. Full text available.
-------------------------------------------------------------------------------
Acknowledgement sent to Andreas Metzler <[email protected]>:
Extra info received and forwarded to list. Copy sent to Joerg Jaspert <[email protected]>. Full text available.
-------------------------------------------------------------------------------

Message received at [email protected]:

Received: (at 256871) by bugs.debian.org; 29 Jun 2004 17:11:28 +0000

Return-path: <[email protected]>
Received: from server.logic.univie.ac.at [131.130.190.41] ([fSfjV4L/RPRbY9sU7VqRJ9D2UhTd4nZv])
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1BfM8m-00017v-00; Tue, 29 Jun 2004 10:11:28 -0700
Received: from m-134-246.adsl.univie.ac.at ([131.130.134.246])
by server.logic.univie.ac.at with asmtp (Exim 4.34)
id 1BfM8j-0007qs-I1; Tue, 29 Jun 2004 19:11:26 +0200
Received: from ametzler by downhill.univie.ac.at with local (Exim 4.34)

        id 1BfM8j-0005Ya-RP; Tue, 29 Jun 2004 19:11:25 +0200

Date: Tue, 29 Jun 2004 19:11:25 +0200
From: Andreas Metzler <[email protected]>
To: Jeff King <[email protected]>, [email protected]
Subject: Re: Bug#256871: cdrecord triggers memory leak in kernel space
Message-ID: <[email protected]>
References: <[email protected]>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <[email protected]>
X-GPG-Fingerprint: BCF7 1345 BE42 B5B8 1A57 EE09 1D33 9C65 8B8D 7663
User-Agent: Mutt/1.5.6+20040523i
Delivered-To: [email protected]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_03_25

        (1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-6.0 required=4.0 tests=BAYES_00,HAS_BUG_NUMBER
        autolearn=no version=2.60-bugs.debian.org_2004_03_25

X-Spam-Level:

On 2004-06-29 Jeff King <[email protected]> wrote:
[...]

[...]

Don't. Use dev=ATA:x,y,z.
cu andreas

-------------------------------------------------------------------------------
Information forwarded to [email protected], Joerg Jaspert <[email protected]>:
Bug#256871; Package cdrecord. Full text available.
-------------------------------------------------------------------------------
Acknowledgement sent to Eduard Bloch <[email protected]>:
Extra info received and forwarded to list. Copy sent to Joerg Jaspert <[email protected]>. Full text available.
-------------------------------------------------------------------------------

Message received at [email protected]:

Received: (at 256871) by bugs.debian.org; 29 Jun 2004 17:26:25 +0000

Return-path: <[email protected]>
Received: from mailgate1.uni-kl.de [131.246.120.5]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian)) id 1BfMNE-0001sP-00; Tue, 29 Jun 2004 10:26:25 -0700
Received: from aixs1.rhrk.uni-kl.de (aixs1.rhrk.uni-kl.de [131.246.137.3])

        by mailgate1.uni-kl.de (8.12.10/8.12.10) with ESMTP id i5THQMJY006306;
        Tue, 29 Jun 2004 19:26:22 +0200
Received: from rotes255.wohnheim.uni-kl.de ([131.246.178.65] helo=debian.zombie.inka.de)
        by aixs1.rhrk.uni-kl.de with esmtp (TLSv1:RC4-SHA:128)
        (Exim 4.20)
        id 1BfMNB-000Afw-RE; Tue, 29 Jun 2004 19:26:21 +0200
Received: from inet by debian.zombie.inka.de with local (Exim 4.34)
        id 1BfMN9-0001o3-Iw; Tue, 29 Jun 2004 19:26:19 +0200

Date: Tue, 29 Jun 2004 19:26:19 +0200
From: Eduard Bloch <[email protected]>
To: Jeff King <[email protected]>, [email protected]
Subject: Re: Bug#256871: cdrecord triggers memory leak in kernel space
Message-ID: <[email protected]>
References: <[email protected]>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <[email protected]>
User-Agent: Mutt/1.5.6+20040523i
Sender: <[email protected]>
Delivered-To: [email protected]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_03_25

        (1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-6.0 required=4.0 tests=BAYES_00,HAS_BUG_NUMBER
        autolearn=no version=2.60-bugs.debian.org_2004_03_25

X-Spam-Level:

#include <hallo.h>

Okay, this sounds more likely like a kernel problem than a cdrecord problem. If you have some time, please try with combinations of following factors to find the reason of the trouble:

- enable swap (does not fight the root of the problems but gives you more time to experiment)
- use a non-SMP kernel
- try kernel 2.6.7
- run cdrecord.shm instead of cdrecord or cdrecord.mmap (note that there are some limits with .shm, see README.Debian)

AFAICS there were no such reports and that's what it is confusing. I do not count on much help from the upstream author there.

Thanks for your efforts,
Eduard.

-------------------------------------------------------------------------------
Information forwarded to [email protected], Joerg Jaspert <[email protected]>:
Bug#256871; Package cdrecord. Full text available.
-------------------------------------------------------------------------------
Acknowledgement sent to Joerg Schilling <[email protected]>:
Extra info received and forwarded to list. Copy sent to Joerg Jaspert <[email protected]>. Full text available.
-------------------------------------------------------------------------------

Message received at [email protected]:

Received: (at 256871) by bugs.debian.org; 4 Jul 2004 13:08:19 +0000

Return-path: <[email protected]>
Received: from mailhub.fokus.fraunhofer.de [193.174.154.14]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian)) id 1Bh6jC-0003Wl-00; Sun, 04 Jul 2004 06:08:19 -0700
Received: from burner.fokus.fraunhofer.de (burner [193.175.133.116])

        by mailhub.fokus.fraunhofer.de (8.11.6p2/8.11.6) with ESMTP id i64D8FU15504;
        Sun, 4 Jul 2004 15:08:15 +0200 (MEST)
Received: (from jes@localhost)
        by burner.fokus.fraunhofer.de (8.12.9+Sun/8.12.9/Submit) id i64D6fPT013802;
        Sun, 4 Jul 2004 15:06:41 +0200 (CEST)

Date: Sun, 4 Jul 2004 15:06:41 +0200 (CEST)
From: Joerg Schilling <[email protected]>
Message-Id: <[email protected]>
To: [email protected]
Cc: [email protected]
Subject: Re: Bug#256871: cdrecord triggers memory leak in kernel space
Delivered-To: [email protected]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_03_25

        (1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-3.0 required=4.0 tests=HAS_BUG_NUMBER autolearn=no
        version=2.60-bugs.debian.org_2004_03_25

X-Spam-Level:

A sad truth :-(

In the past, the Linux Kernel maintainers have not been very helpful with Linux Kernel bugs.

I did stop sending bug reports to the Linux Kernel team two years ago for this reason. For me it makes not sense to waste my time with unwilling people.

A possible reson for this Linux kernel bug is Kernel design bug that is known for a long time:

Linux does not keep track of the size of the processes. If e.g. a process forks, Linux creates copy on write pages for the data segment of the fork. This idea has been taken from SunOS ideas in the mid 1980s. Unlike SunOS, Linux does not compute the needed virtual memory and overcommits the available size.

When a process later modifies the parts of it's address space that is just a copy on write marked page, the kernel needs to create a second page before the modification may take place. The space for this page may not be available and the Linux kernel management is unable to tell which process is the cause for the missing virtual memory space. Linux for this reason kills random processes to regain virtual memory space. If you have a multi processor machine, it is even
worse: Linux in this case becomes catatonic and may only recover by a "reset" induced reboot. This is caused by the way a dual or multi processor Linux works.

BTW: As cdrecord touches all it's memory at process start, cdrercord is not one of the processes where Linux does not know the correct process size.

J?rg

--
EMail:[email protected] (home) J?rg Schilling D-13353 Berlin
[email protected] (uni) If you don't have iso-8859-1
[email protected] (work) chars I am J"org Schilling
URL: http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

-------------------------------------------------------------------------------
Information forwarded to [email protected], Joerg Jaspert <[email protected]>:
Bug#256871; Package cdrecord. Full text available.
-------------------------------------------------------------------------------
Acknowledgement sent to Jeff King <[email protected]>:
Extra info received and forwarded to list. Copy sent to Joerg Jaspert <[email protected]>. Full text available.
-------------------------------------------------------------------------------

Message received at [email protected]:

Received: (at 256871) by bugs.debian.org; 5 Jul 2004 05:56:13 +0000

Return-path: <[email protected]>
Received: from 66-23-211-5.clients.speedfactory.net (peff.net) [66.23.211.5]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1BhMSb-00019j-00; Sun, 04 Jul 2004 22:56:13 -0700
Received: (qmail 58673 invoked from network); 5 Jul 2004 05:56:16 -0000
Received: from unknown (HELO localhost) (127.0.0.1)
by peff.net with AES256-SHA encrypted SMTP; 5 Jul 2004 05:56:16 -0000
Date: Mon, 5 Jul 2004 01:56:15 -0400 (EDT)
From: Jeff King <[email protected]>
To: [email protected]
Subject: Re: Bug#256871: cdrecord triggers memory leak in kernel space
In-Reply-To: <[email protected]>
Message-ID: <[email protected]>
References: <[email protected]>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Delivered-To: [email protected]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_03_25

        (1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-3.1 required=4.0 tests=BAYES_44,HAS_BUG_NUMBER
        autolearn=no version=2.60-bugs.debian.org_2004_03_25

X-Spam-Level:

I'm not sure if this is my problem. I do get the OOM killer killing off random processes, but it's because the kernel thinks that it is legitimately out of memory. Using "free" reports that all memory is in use (and not by buffers/cache). I believe the real problem is a leak somewhere in the kernel.

Sorry for the slowness of response on this...I've been out of town all week and will be for a few more days. I will try to do some tests and see if I can pin down the offending behavior (hopefully in a way that doesn't involve frying a CD-R each time!) and will report back.

-Peff

-------------------------------------------------------------------------------
Information forwarded to [email protected], Joerg Jaspert <[email protected]>:
Bug#256871; Package cdrecord. Full text available.
-------------------------------------------------------------------------------
Acknowledgement sent to Joerg Schilling <[email protected]>:
Extra info received and forwarded to list. Copy sent to Joerg Jaspert <[email protected]>. Full text available.
-------------------------------------------------------------------------------

Message received at [email protected]:

Received: (at 256871) by bugs.debian.org; 6 Jul 2004 09:56:24 +0000

Return-path: <[email protected]>
Received: from mailhub.fokus.fraunhofer.de [193.174.154.14]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian)) id 1BhmfS-0008Pi-00; Tue, 06 Jul 2004 02:56:24 -0700
Received: from burner.fokus.fraunhofer.de (burner [193.175.133.116])

        by mailhub.fokus.fraunhofer.de (8.11.6p2/8.11.6) with ESMTP id i669tCU15351
        for <[email protected]>; Tue, 6 Jul 2004 11:55:12 +0200 (MEST)
Received: (from jes@localhost)
        by burner.fokus.fraunhofer.de (8.12.9+Sun/8.12.9/Submit) id i669qFwe024525;
        Tue, 6 Jul 2004 11:52:15 +0200 (CEST)

Date: Tue, 6 Jul 2004 11:52:15 +0200 (CEST)
From: Joerg Schilling <[email protected]>
Message-Id: <[email protected]>
To: [email protected], [email protected]
Subject: Re: Bug#256871: cdrecord triggers memory leak in kernel space
Delivered-To: [email protected]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_03_25

        (1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-3.1 required=4.0 tests=BAYES_44,HAS_BUG_NUMBER
        autolearn=no version=2.60-bugs.debian.org_2004_03_25

X-Spam-Level:

Hi,

I would not believe that there is a big memory leak in Linux.

However, you seem to missunderstand me:

The problem is not that Linux incorrectly believes that there is no memory. The problem is that Linux _before_ believes that there is _plenty_ of memory although there is none left over.

Try this:

char buf[16*1024*1024];
main()

{
        int     i;
        int     ret;

        for (i = 0; i < 1000; i++) {
                if (ret == 0)
                        sleep(1000);
                if (ret < 0) {
                        perror("fork");
                        printf("i: %d\n", i);
                        exit(1);
                }
        }
}

On Solaris, you will see something like:

fork: Not enough space
i: 53

On Linux, I asume that you will see no error message.....

If you modify the loop to:

        for (i = 0; i < 1000; i++) {
                if (ret == 0) {
                        sleep(10);
                        memset(buf, 0, sizeof(buf));
                        sleep(1000);
                }

...

You will see the same problems as you did describe.

J?rg

--
EMail:[email protected] (home) J?rg Schilling D-13353 Berlin
[email protected] (uni) If you don't have iso-8859-1
[email protected] (work) chars I am J"org Schilling
URL: http://www.fokus.fraunhofer.de/usr/schilling ftp://ftp.berlios.de/pub/schily

-------------------------------------------------------------------------------
Information forwarded to [email protected], Joerg Jaspert <[email protected]>:
Bug#256871; Package cdrecord. Full text available.
-------------------------------------------------------------------------------
Acknowledgement sent to Jeff King <[email protected]>:
Extra info received and forwarded to list. Copy sent to Joerg Jaspert <[email protected]>. Full text available.
-------------------------------------------------------------------------------

Message received at [email protected]:

Received: (at 256871) by bugs.debian.org; 6 Jul 2004 21:02:11 +0000

Return-path: <[email protected]>
Received: from 66-23-211-5.clients.speedfactory.net (peff.net) [66.23.211.5]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1Bhx4t-00023B-00; Tue, 06 Jul 2004 14:02:11 -0700
Received: (qmail 1592 invoked from network); 6 Jul 2004 21:02:15 -0000
Received: from unknown (HELO localhost) (127.0.0.1)
by peff.net with AES256-SHA encrypted SMTP; 6 Jul 2004 21:02:15 -0000
Date: Tue, 6 Jul 2004 17:02:15 -0400 (EDT)
From: Jeff King <[email protected]>
To: [email protected]
Subject: Re: Bug#256871: cdrecord triggers memory leak in kernel space
In-Reply-To: <[email protected]>
Message-ID: <[email protected]>
References: <[email protected]>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Delivered-To: [email protected]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_03_25

        (1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-3.1 required=4.0 tests=BAYES_44,HAS_BUG_NUMBER
        autolearn=no version=2.60-bugs.debian.org_2004_03_25

X-Spam-Level:

I understand your complaints about memory overcommits. However, in my case, the OOM killing is not the problem, but rather a symptom that comes from not realizing that memory is available. The problem is that Linux incorrectly believes there is no memory. The sequence is something like this:

1. I have 1G of memory. There is about 850M free (as reported by free, not counting buffers). Userland is using 100M or so of that (as reported by ps).

2. I run cdrecord. It burns the CD correctly, then exits.

3. There is now only 100M free. Userland is still using 100M of memory. Running cdrecord again causes OOM killing because of memory overcommits.

Where did all of the memory go? If it's in use by the kernel, on whose behalf? There are no programs running after the burn that were not running before the burn. I believe that the kernel has marked some memory as used on behalf of cdrecord and failed to free it when the program exited.

At this point it's just a theory. I will attempt to gather evidence later this week when I return home to either justify or disprove my theory.

-Peff

-------------------------------------------------------------------------------
Information forwarded to [email protected], Joerg Jaspert <[email protected]>:
Bug#256871; Package cdrecord. Full text available.
-------------------------------------------------------------------------------
Acknowledgement sent to Jeff King <[email protected]>:
Extra info received and forwarded to list. Copy sent to Joerg Jaspert <[email protected]>. Full text available.
-------------------------------------------------------------------------------

Message received at [email protected]:

Received: (at 256871) by bugs.debian.org; 9 Jul 2004 07:37:43 +0000

Return-path: <[email protected]>
Received: from 66-23-211-5.clients.speedfactory.net (peff.net) [66.23.211.5]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1Bipx1-0001GI-00; Fri, 09 Jul 2004 00:37:43 -0700
Received: (qmail 73572 invoked from network); 9 Jul 2004 07:37:48 -0000
Received: from unknown (HELO coredump.intra.peff.net) (10.0.0.2)
by peff.net with AES256-SHA encrypted SMTP; 9 Jul 2004 07:37:48 -0000
Date: Fri, 9 Jul 2004 03:37:42 -0400 (EDT)
From: Jeff King <[email protected]>
To: [email protected]
Subject: Re: Bug#256871: cdrecord triggers memory leak in kernel space
In-Reply-To: <[email protected]>
Message-ID: <[email protected]>
References: <[email protected]>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Delivered-To: [email protected]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_03_25

        (1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-3.1 required=4.0 tests=BAYES_44,HAS_BUG_NUMBER
        autolearn=no version=2.60-bugs.debian.org_2004_03_25

X-Spam-Level:

OK, I tested a few different configurations. My procedure was more or less to run cdrecord -dummy and keep an eye on total memory free (using 'free'). I considered the problem to occur if memory usage (minus buffers) increased steadily as cdrecord ran, and then stayed high when the program exited. The rate of memory gobbling generally seemed to be several hundred kilobytes per second (one would guess if it is a leak of a data buffer, that it would be dropping about 176400 * 4 (the speed of the writer), which is close to what I saw). The machine was otherwise idle.

Here's what I found:
- cdrecord.mmap versus cdrecord.shm made no difference
- problem occurred on kernel 2.6.7-1-k7-smp
- problem also occurred on kernel 2.6.7-1-k7 (non-smp)
- problem did not occur with -tao, but did occur with -dao; no other command line options seemed to effect whether it occurred

Other fun facts:
- I tried using cdrdao; problem also occurred there
- all tests were done using ide-cd kernel module (dev=ATA:1,0,0)

I think it's probably not related to the mmap'ing of the buffer memory (because it happens with cdrecord.shm), but rather to some problem within the kernel CD driver. Seeing what functional differences -dao and -tao have (which should hopefully lead us in the right direction) is kind of hard. The strace output is just a bunch of ioctls. :) I'm hoping somebody more familiar with the code can comment on how the behavior differs, and we can hopefully produce a very short program that triggers the kernel behavior.

-Peff

-------------------------------------------------------------------------------

Debian bug tracking system administrator <[email protected]>. Last
modified: Sun, 25 Jul 2004 09:45:23 UTC

Debian Bug tracking system
copyright 1999 Darren O. Benham, 1997, 2003 nCipher Corporation Ltd, 1994-7 Ian Jackson.

Indeed, i burned a smaller cd and got very similar results. I actually got to see the oom messages and all, of course nothing in userspace was hogging memory. The kernel is most definitely not releasing the memory it is using. I further would have to say that the kernel isn't being given the correct memory addresses that the data is actually at ( and could be the reason why it's not freed) because every single audio cd i've burned partially or completely with 2.6.6+ has sounded garbled or been completly unreadable. This is the 20'th cd i've tested using different modes, settings, speeds, kernels and the results have been the same every time. I haven't tested kernels prior to 2.6.6 though. If there were any patches to the block layer or ide-cd code from 2.6.5 to 2.6.6 i'll go ahead and test 2.6.5... I'll go and check the changelog now but it's likely that a whole lot of code was changed between those versions if any was changed at all.

btw, the bug seems to be much more pronounced if you burn at 48x (some speed that you'd have to use burnfree or it would underrun)

My dmesg output.

Same here.. After upgrading to 2.6.8-rc2 the OOM-killer is going crazy. It's particularly angry at the backup client 'dsmc' (from Tivoli Storage Manager). I'm monitoring its usage with 'top', and 'dsmc' is not using more than ~150MB in either size or RSS when the OOM-killer takes it down.

The 'dsmc'-process is reporting that it's processed 2,719,000 files, and transfered 164.34 MB when it gets killed. i.e. it's traversed a lot of files, but only read about 164 MB data, so it shouldn't have filled up any buffer cache...

The system still has lots of free memory (~900 MB), and also 2 GB of unused swap. Actually there's 0K used swap..??

I've tried turning on vm.overcommit_memory, but it had no effect. Also tried changing the swappiness both up to 90% and down to 10%, but it never uses any swap.. ???

BTW: I had no OOM-killer problems on 2.6.7.

-jf

Can you just check you CONFIG_SWAP is on and /proc/sys/vm/laptop_mode is 0, and that you have some swap enabled.

If the problem persists, can you send a copy each of /proc/sys/fs/dentry-state, /proc/slabinfo and /proc/vmstat before and after you run dsmc until it goes OOM please?

Thanks.

# grep CONFIG_SWAP .config
CONFIG_SWAP=y
# cat /proc/sys/vm/laptop_mode
0
# free
             total       used       free     shared    buffers     cached
Mem:       2074708    1223324     851384          0        296     258376
-/+ buffers/cache:     964652    1110056
Swap:      2040244          0    2040244

I turned of a option (MEMORYEFFICIENTBACKUP) in 'dsmc', and then it uses a bit more memory, and crashes quicker.

# cat /proc/sys/fs/dentry-state /proc/slabinfo /proc/vmstat  ; dsmc incremental ; cat /proc/sys/fs/dentry-state /proc/slabinfo /proc/vmstat
644923  572300  45      0       0       0
slabinfo - version: 2.0
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
xfrm6_tunnel_spi       0      0     64   61    1 : tunables  120   60    8 : slabdata      0      0      0
fib6_nodes             7    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
ip6_dst_cache          9     18    224   18    1 : tunables  120   60    8 : slabdata      1      1      0
ndisc_cache            1     25    160   25    1 : tunables  120   60    8 : slabdata      1      1      0
raw6_sock              0      0    672    6    1 : tunables   54   27    8 : slabdata      0      0      0
udp6_sock              0      0    640    6    1 : tunables   54   27    8 : slabdata      0      0      0
tcp6_sock             14     18   1184    6    2 : tunables   24   12    8 : slabdata      3      3      0
ip_fib_hash           18    226     16  226    1 : tunables  120   60    8 : slabdata      1      1      0
ip_conntrack         494    516    320   12    1 : tunables   54   27    8 : slabdata     43     43      0
xfs_dqtrx              0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_dquots            32     36    336   12    1 : tunables   54   27    8 : slabdata      3      3      0
dm_tio                 0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
dm_io                  0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
rpc_buffers            8      8   2048    2    1 : tunables   24   12    8 : slabdata      4      4      0
rpc_tasks              8     25    160   25    1 : tunables  120   60    8 : slabdata      1      1      0
rpc_inode_cache        6      9    448    9    1 : tunables   54   27    8 : slabdata      1      1      0
unix_sock             36     36    416    9    1 : tunables   54   27    8 : slabdata      4      4      0
ip_mrt_cache           0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
tcp_tw_bucket          3     31    128   31    1 : tunables  120   60    8 : slabdata      1      1      0
tcp_bind_bucket       21    226     16  226    1 : tunables  120   60    8 : slabdata      1      1      0
tcp_open_request       2     41     96   41    1 : tunables  120   60    8 : slabdata      1      1      0
inet_peer_cache        8     61     64   61    1 : tunables  120   60    8 : slabdata      1      1      0
secpath_cache          0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
ip_dst_cache          74    165    256   15    1 : tunables  120   60    8 : slabdata     11     11      0
arp_cache             75     75    160   25    1 : tunables  120   60    8 : slabdata      3      3      0
raw4_sock              0      0    512    7    1 : tunables   54   27    8 : slabdata      0      0      0
udp_sock              21     21    544    7    1 : tunables   54   27    8 : slabdata      3      3      0
tcp_sock              32     42   1056    7    2 : tunables   24   12    8 : slabdata      6      6      0
flow_cache             0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
qla2xxx_srbs         136    186    128   31    1 : tunables  120   60    8 : slabdata      5      6      0
scsi_cmd_cache         6     22    352   11    1 : tunables   54   27    8 : slabdata      2      2      0
mqueue_inode_cache      1      7    512    7    1 : tunables   54   27    8 : slabdata      1      1      0
xfs_acl                0      0    304   13    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_chashlist      39376  45695     20  185    1 : tunables  120   60    8 : slabdata    247    247      0
xfs_ili              219    476    140   28    1 : tunables  120   60    8 : slabdata     17     17      0
xfs_ifork              0      0     56   70    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_efi_item           0      0    260   15    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_efd_item           0      0    260   15    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_buf_item           0      0    148   27    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_dabuf              0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_da_state           0      0    336   12    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_trans              2     13    596   13    2 : tunables   54   27    8 : slabdata      1      1      0
xfs_inode         927591 980848    368   11    1 : tunables   54   27    8 : slabdata  89168  89168      0
xfs_btree_cur          0      0    140   28    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_bmap_free_item      0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_buf_t             40     60    256   15    1 : tunables  120   60    8 : slabdata      4      4      0
linvfs_icache     927591 980810    384   10    1 : tunables   54   27    8 : slabdata  98081  98081      0
nfs_write_data        36     36    448    9    1 : tunables   54   27    8 : slabdata      4      4      0
nfs_read_data         32     36    416    9    1 : tunables   54   27    8 : slabdata      4      4      0
nfs_inode_cache        0      0    608    6    1 : tunables   54   27    8 : slabdata      0      0      0
nfs_page               0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
isofs_inode_cache      0      0    352   11    1 : tunables   54   27    8 : slabdata      0      0      0
hugetlbfs_inode_cache      1     12    324   12    1 : tunables   54   27    8 : slabdata      1      1      0
ext2_inode_cache       0      0    480    8    1 : tunables   54   27    8 : slabdata      0      0      0
ext2_xattr             0      0     48   81    1 : tunables  120   60    8 : slabdata      0      0      0
journal_handle        24    135     28  135    1 : tunables  120   60    8 : slabdata      1      1      0
journal_head          32    324     48   81    1 : tunables  120   60    8 : slabdata      4      4      0
revoke_table           4    290     12  290    1 : tunables  120   60    8 : slabdata      1      1      0
revoke_record          0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
ext3_inode_cache     818   1096    480    8    1 : tunables   54   27    8 : slabdata    137    137      0
ext3_xattr             0      0     48   81    1 : tunables  120   60    8 : slabdata      0      0      0
dquot                  0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
eventpoll_pwq          0      0     36  107    1 : tunables  120   60    8 : slabdata      0      0      0
eventpoll_epi          0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
kioctx                 0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
kiocb                  0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
dnotify_cache          0      0     20  185    1 : tunables  120   60    8 : slabdata      0      0      0
file_lock_cache       69     82     96   41    1 : tunables  120   60    8 : slabdata      2      2      0
fasync_cache           0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
shmem_inode_cache     21     36    448    9    1 : tunables   54   27    8 : slabdata      4      4      0
posix_timers_cache      0      0    104   38    1 : tunables  120   60    8 : slabdata      0      0      0
uid_cache              4    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
sgpool-128            32     32   2048    2    1 : tunables   24   12    8 : slabdata     16     16      0
sgpool-64             32     32   1024    4    1 : tunables   54   27    8 : slabdata      8      8      0
sgpool-32             32     32    512    8    1 : tunables   54   27    8 : slabdata      4      4      0
sgpool-16             32     45    256   15    1 : tunables  120   60    8 : slabdata      3      3      0
sgpool-8              48     62    128   31    1 : tunables  120   60    8 : slabdata      2      2      0
cfq_pool              64    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
crq_pool               0      0     40   96    1 : tunables  120   60    8 : slabdata      0      0      0
deadline_drq           0      0     52   75    1 : tunables  120   60    8 : slabdata      0      0      0
as_arq                37    122     64   61    1 : tunables  120   60    8 : slabdata      2      2      0
blkdev_ioc            71    185     20  185    1 : tunables  120   60    8 : slabdata      1      1      0
blkdev_queue          32     32    464    8    1 : tunables   54   27    8 : slabdata      4      4      0
blkdev_requests       50    100    160   25    1 : tunables  120   60    8 : slabdata      4      4      0
biovec-(256)         256    256   3072    2    2 : tunables   24   12    8 : slabdata    128    128      0
biovec-128           256    260   1536    5    2 : tunables   24   12    8 : slabdata     52     52      0
biovec-64            258    260    768    5    1 : tunables   54   27    8 : slabdata     52     52      0
biovec-16            256    260    192   20    1 : tunables  120   60    8 : slabdata     13     13      0
biovec-4             256    305     64   61    1 : tunables  120   60    8 : slabdata      5      5      0
biovec-1             277    452     16  226    1 : tunables  120   60    8 : slabdata      2      2      0
bio                  266    369     96   41    1 : tunables  120   60    8 : slabdata      8      9      0
sock_inode_cache      90     90    384   10    1 : tunables   54   27    8 : slabdata      9      9      0
skbuff_head_cache    457    660    192   20    1 : tunables  120   60    8 : slabdata     33     33      0
sock                   3     11    352   11    1 : tunables   54   27    8 : slabdata      1      1      0
proc_inode_cache     576    616    352   11    1 : tunables   54   27    8 : slabdata     56     56      0
sigqueue              16     27    148   27    1 : tunables  120   60    8 : slabdata      1      1      0
radix_tree_node     6769   6818    276   14    1 : tunables   54   27    8 : slabdata    487    487      0
bdev_cache            14     27    448    9    1 : tunables   54   27    8 : slabdata      3      3      0
mnt_cache             30     41     96   41    1 : tunables  120   60    8 : slabdata      1      1      0
inode_cache         1481   1485    352   11    1 : tunables   54   27    8 : slabdata    135    135      0
dentry_cache      645063 703566    144   27    1 : tunables  120   60    8 : slabdata  26058  26058      0
filp                 857    900    160   25    1 : tunables  120   60    8 : slabdata     36     36      0
names_cache           21     21   4096    1    1 : tunables   24   12    8 : slabdata     21     21      0
idr_layer_cache       69     87    136   29    1 : tunables  120   60    8 : slabdata      3      3      0
buffer_head         5110   9600     52   75    1 : tunables  120   60    8 : slabdata    128    128      0
mm_struct             72     77    544    7    1 : tunables   54   27    8 : slabdata     11     11      0
vm_area_struct      1937   1974     84   47    1 : tunables  120   60    8 : slabdata     42     42      0
fs_cache              89    183     64   61    1 : tunables  120   60    8 : slabdata      3      3      0
files_cache           73     81    416    9    1 : tunables   54   27    8 : slabdata      9      9      0
signal_cache         123    164     96   41    1 : tunables  120   60    8 : slabdata      4      4      0
sighand_cache        108    108   1312    3    1 : tunables   24   12    8 : slabdata     36     36      0
task_struct          123    130   1440    5    2 : tunables   24   12    8 : slabdata     26     26      0
anon_vma             812   1160     12  290    1 : tunables  120   60    8 : slabdata      4      4      0
pgd                   57     57   4096    1    1 : tunables   24   12    8 : slabdata     57     57      0
size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-131072            0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-65536             0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-32768             0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
size-16384             2      2  16384    1    4 : tunables    8    4    0 : slabdata      2      2      0
size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0
size-8192              7      7   8192    1    2 : tunables    8    4    0 : slabdata      7      7      0
size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    8 : slabdata      0      0      0
size-4096            458    458   4096    1    1 : tunables   24   12    8 : slabdata    458    458      0
size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    8 : slabdata      0      0      0
size-2048            205    212   2048    2    1 : tunables   24   12    8 : slabdata    106    106      0
size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
size-1024            188    188   1024    4    1 : tunables   54   27    8 : slabdata     47     47      0
size-512(DMA)          0      0    512    8    1 : tunables   54   27    8 : slabdata      0      0      0
size-512             363   1336    512    8    1 : tunables   54   27    8 : slabdata    167    167      0
size-256(DMA)          0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
size-256             343   1230    256   15    1 : tunables  120   60    8 : slabdata     82     82      0
size-192(DMA)          0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
size-192           13208  14100    192   20    1 : tunables  120   60    8 : slabdata    705    705      0
size-128(DMA)          0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
size-128           10775  12276    128   31    1 : tunables  120   60    8 : slabdata    396    396      0
size-96(DMA)           0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
size-96            18193  20910     96   41    1 : tunables  120   60    8 : slabdata    510    510      0
size-64(DMA)           0      0     64   61    1 : tunables  120   60    8 : slabdata      0      0      0
size-64            71339  95770     64   61    1 : tunables  120   60    8 : slabdata   1570   1570      0
size-32(DMA)           0      0     32  119    1 : tunables  120   60    8 : slabdata      0      0      0
size-32             3469   4403     32  119    1 : tunables  120   60    8 : slabdata     37     37      0
kmem_cache           175    175    160   25    1 : tunables  120   60    8 : slabdata      7      7      0
nr_dirty 7
nr_writeback 0
nr_unstable 0
nr_page_table_pages 184
nr_mapped 6676
nr_slab 219569
pgpgin 10457106
pgpgout 4609136
pswpin 0
pswpout 0
pgalloc_high 3300613
pgalloc_normal 122918151
pgalloc_dma 2598782
pgfree 129043476
pgactivate 1736104
pgdeactivate 1494127
pgfault 3833745
pgmajfault 829
pgrefill_high 0
pgrefill_normal 1481727
pgrefill_dma 12400
pgsteal_high 0
pgsteal_normal 2209866
pgsteal_dma 29734
pgscan_kswapd_high 0
pgscan_kswapd_normal 2076562
pgscan_kswapd_dma 31042
pgscan_direct_high 0
pgscan_direct_normal 636887
pgscan_direct_dma 10854
pginodesteal 36508
slabs_scanned 56099472
kswapd_steal 1783428
kswapd_inodesteal 317433
pageoutrun 12166
allocstall 12962
pgrotated 14968
Tivoli Storage Manager
Command Line Backup/Archive Client Interface - Version 5, Release 1, Level 6.0
(C) Copyright IBM Corporation 1990, 2003 All Rights Reserved.

Node Name: BCCSFS
Session established with server TSM: Solaris 7/8
Server Version 5, Release 1, Level 6.5
Server date/time: 07/26/2004 13:02:47 Last access: 07/26/2004 13:01:56

Incremental backup of volume '/etc/'

Incremental backup of volume '/export/homebccs/'

Incremental backup of volume '/export/homebccs1/'

Incremental backup of volume '/export/netbccs/'

Incremental backup of volume '/export/local/'
ANS1898I ***** Processed 500 files *****
ANS1898I ***** Processed 1,000 files *****
Terminated
570734 495922 45 0 0 0
slabinfo - version: 2.0

# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
xfrm6_tunnel_spi       0      0     64   61    1 : tunables  120   60    8 : slabdata      0      0      0
fib6_nodes             7    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
ip6_dst_cache          9     18    224   18    1 : tunables  120   60    8 : slabdata      1      1      0
ndisc_cache            1     25    160   25    1 : tunables  120   60    8 : slabdata      1      1      0
raw6_sock              0      0    672    6    1 : tunables   54   27    8 : slabdata      0      0      0
udp6_sock              0      0    640    6    1 : tunables   54   27    8 : slabdata      0      0      0
tcp6_sock             14     18   1184    6    2 : tunables   24   12    8 : slabdata      3      3      0
ip_fib_hash           18    226     16  226    1 : tunables  120   60    8 : slabdata      1      1      0
ip_conntrack         499    516    320   12    1 : tunables   54   27    8 : slabdata     43     43      0
xfs_dqtrx              0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_dquots            32     36    336   12    1 : tunables   54   27    8 : slabdata      3      3      0
dm_tio                 0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
dm_io                  0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
rpc_buffers            8      8   2048    2    1 : tunables   24   12    8 : slabdata      4      4      0
rpc_tasks              8     25    160   25    1 : tunables  120   60    8 : slabdata      1      1      0
rpc_inode_cache        6      9    448    9    1 : tunables   54   27    8 : slabdata      1      1      0
unix_sock             18     36    416    9    1 : tunables   54   27    8 : slabdata      4      4      0
ip_mrt_cache           0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
tcp_tw_bucket          3     31    128   31    1 : tunables  120   60    8 : slabdata      1      1      0
tcp_bind_bucket       25    226     16  226    1 : tunables  120   60    8 : slabdata      1      1      0
tcp_open_request       0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
inet_peer_cache        8     61     64   61    1 : tunables  120   60    8 : slabdata      1      1      0
secpath_cache          0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
ip_dst_cache          69    165    256   15    1 : tunables  120   60    8 : slabdata     11     11      0
arp_cache             65     75    160   25    1 : tunables  120   60    8 : slabdata      3      3      0
raw4_sock              0      0    512    7    1 : tunables   54   27    8 : slabdata      0      0      0
udp_sock              14     21    544    7    1 : tunables   54   27    8 : slabdata      3      3      0
tcp_sock              35     42   1056    7    2 : tunables   24   12    8 : slabdata      6      6      0
flow_cache             0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
qla2xxx_srbs         145    155    128   31    1 : tunables  120   60    8 : slabdata      5      5      0
scsi_cmd_cache        22     22    352   11    1 : tunables   54   27    8 : slabdata      2      2      0
mqueue_inode_cache      1      7    512    7    1 : tunables   54   27    8 : slabdata      1      1      0
xfs_acl                0      0    304   13    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_chashlist      37540  45695     20  185    1 : tunables  120   60    8 : slabdata    247    247    480
xfs_ili              219    476    140   28    1 : tunables  120   60    8 : slabdata     17     17      0
xfs_ifork              0      0     56   70    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_efi_item           0      0    260   15    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_efd_item           0      0    260   15    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_buf_item           0      0    148   27    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_dabuf              0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_da_state           0      0    336   12    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_trans             13     13    596   13    2 : tunables   54   27    8 : slabdata      1      1      0
xfs_inode         828633 980507    368   11    1 : tunables   54   27    8 : slabdata  89137  89137    216
xfs_btree_cur          0      0    140   28    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_bmap_free_item      0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_buf_t             57     60    256   15    1 : tunables  120   60    8 : slabdata      4      4      0
linvfs_icache     828629 980220    384   10    1 : tunables   54   27    8 : slabdata  98022  98022    216
nfs_write_data        36     36    448    9    1 : tunables   54   27    8 : slabdata      4      4      0
nfs_read_data         32     36    416    9    1 : tunables   54   27    8 : slabdata      4      4      0
nfs_inode_cache        0      0    608    6    1 : tunables   54   27    8 : slabdata      0      0      0
nfs_page               0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
isofs_inode_cache      0      0    352   11    1 : tunables   54   27    8 : slabdata      0      0      0
hugetlbfs_inode_cache      1     12    324   12    1 : tunables   54   27    8 : slabdata      1      1      0
ext2_inode_cache       0      0    480    8    1 : tunables   54   27    8 : slabdata      0      0      0
ext2_xattr             0      0     48   81    1 : tunables  120   60    8 : slabdata      0      0      0
journal_handle        24    135     28  135    1 : tunables  120   60    8 : slabdata      1      1      0
journal_head         242    324     48   81    1 : tunables  120   60    8 : slabdata      4      4      0
revoke_table           4    290     12  290    1 : tunables  120   60    8 : slabdata      1      1      0
revoke_record          0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
ext3_inode_cache    1488   1488    480    8    1 : tunables   54   27    8 : slabdata    186    186      0
ext3_xattr             0      0     48   81    1 : tunables  120   60    8 : slabdata      0      0      0
dquot                  0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
eventpoll_pwq          0      0     36  107    1 : tunables  120   60    8 : slabdata      0      0      0
eventpoll_epi          0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
kioctx                 0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
kiocb                  0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
dnotify_cache          0      0     20  185    1 : tunables  120   60    8 : slabdata      0      0      0
file_lock_cache       21     82     96   41    1 : tunables  120   60    8 : slabdata      2      2      0
fasync_cache           0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
shmem_inode_cache     20     36    448    9    1 : tunables   54   27    8 : slabdata      4      4      0
posix_timers_cache      0      0    104   38    1 : tunables  120   60    8 : slabdata      0      0      0
uid_cache              3    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
sgpool-128            33     34   2048    2    1 : tunables   24   12    8 : slabdata     17     17      0
sgpool-64             32     32   1024    4    1 : tunables   54   27    8 : slabdata      8      8      0
sgpool-32             32     32    512    8    1 : tunables   54   27    8 : slabdata      4      4      0
sgpool-16             32     45    256   15    1 : tunables  120   60    8 : slabdata      3      3      0
sgpool-8              62     62    128   31    1 : tunables  120   60    8 : slabdata      2      2      0
cfq_pool              64    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
crq_pool               0      0     40   96    1 : tunables  120   60    8 : slabdata      0      0      0
deadline_drq           0      0     52   75    1 : tunables  120   60    8 : slabdata      0      0      0
as_arq               152    183     64   61    1 : tunables  120   60    8 : slabdata      3      3     30
blkdev_ioc            83    185     20  185    1 : tunables  120   60    8 : slabdata      1      1      0
blkdev_queue          32     32    464    8    1 : tunables   54   27    8 : slabdata      4      4      0
blkdev_requests      150    150    160   25    1 : tunables  120   60    8 : slabdata      6      6      0
biovec-(256)         256    256   3072    2    2 : tunables   24   12    8 : slabdata    128    128      0
biovec-128           256    260   1536    5    2 : tunables   24   12    8 : slabdata     52     52      0
biovec-64            260    260    768    5    1 : tunables   54   27    8 : slabdata     52     52      0
biovec-16            256    260    192   20    1 : tunables  120   60    8 : slabdata     13     13      0
biovec-4             273    305     64   61    1 : tunables  120   60    8 : slabdata      5      5      0
biovec-1             448    452     16  226    1 : tunables  120   60    8 : slabdata      2      2     60
bio                  410    410     96   41    1 : tunables  120   60    8 : slabdata     10     10     60
sock_inode_cache      78     90    384   10    1 : tunables   54   27    8 : slabdata      9      9      0
skbuff_head_cache    457    660    192   20    1 : tunables  120   60    8 : slabdata     33     33      0
sock                   3     11    352   11    1 : tunables   54   27    8 : slabdata      1      1      0
proc_inode_cache     581    616    352   11    1 : tunables   54   27    8 : slabdata     56     56      0
sigqueue               6     27    148   27    1 : tunables  120   60    8 : slabdata      1      1      0
radix_tree_node     6888   6888    276   14    1 : tunables   54   27    8 : slabdata    492    492     27
bdev_cache            14     27    448    9    1 : tunables   54   27    8 : slabdata      3      3      0
mnt_cache             30     41     96   41    1 : tunables  120   60    8 : slabdata      1      1      0
inode_cache         1482   1485    352   11    1 : tunables   54   27    8 : slabdata    135    135      0
dentry_cache      571383 703458    144   27    1 : tunables  120   60    8 : slabdata  26054  26054    480
filp                 857    900    160   25    1 : tunables  120   60    8 : slabdata     36     36      0
names_cache           23     23   4096    1    1 : tunables   24   12    8 : slabdata     23     23      0
idr_layer_cache       69     87    136   29    1 : tunables  120   60    8 : slabdata      3      3      0
buffer_head         5402   9600     52   75    1 : tunables  120   60    8 : slabdata    128    128    120
mm_struct             77     77    544    7    1 : tunables   54   27    8 : slabdata     11     11      0
vm_area_struct      1913   1974     84   47    1 : tunables  120   60    8 : slabdata     42     42      0
fs_cache              75    183     64   61    1 : tunables  120   60    8 : slabdata      3      3      0
files_cache           59     81    416    9    1 : tunables   54   27    8 : slabdata      9      9      0
signal_cache         110    164     96   41    1 : tunables  120   60    8 : slabdata      4      4      0
sighand_cache        103    108   1312    3    1 : tunables   24   12    8 : slabdata     36     36      0
task_struct          117    130   1440    5    2 : tunables   24   12    8 : slabdata     26     26      0
anon_vma             788   1160     12  290    1 : tunables  120   60    8 : slabdata      4      4      0
pgd                   57     57   4096    1    1 : tunables   24   12    8 : slabdata     57     57      0
size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-131072            0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-65536             0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-32768             0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
size-16384             2      2  16384    1    4 : tunables    8    4    0 : slabdata      2      2      0
size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0
size-8192              7      7   8192    1    2 : tunables    8    4    0 : slabdata      7      7      0
size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    8 : slabdata      0      0      0
size-4096            482    482   4096    1    1 : tunables   24   12    8 : slabdata    482    482     24
size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    8 : slabdata      0      0      0
size-2048            206    212   2048    2    1 : tunables   24   12    8 : slabdata    106    106      6
size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
size-1024            188    188   1024    4    1 : tunables   54   27    8 : slabdata     47     47      0
size-512(DMA)          0      0    512    8    1 : tunables   54   27    8 : slabdata      0      0      0
size-512             363   1336    512    8    1 : tunables   54   27    8 : slabdata    167    167      0
size-256(DMA)          0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
size-256             358   1230    256   15    1 : tunables  120   60    8 : slabdata     82     82      0
size-192(DMA)          0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
size-192           12878  14100    192   20    1 : tunables  120   60    8 : slabdata    705    705    480
size-128(DMA)          0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
size-128           10671  12276    128   31    1 : tunables  120   60    8 : slabdata    396    396    330
size-96(DMA)           0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
size-96            17860  20910     96   41    1 : tunables  120   60    8 : slabdata    510    510    480
size-64(DMA)           0      0     64   61    1 : tunables  120   60    8 : slabdata      0      0      0
size-64            66057  95770     64   61    1 : tunables  120   60    8 : slabdata   1570   1570    480
size-32(DMA)           0      0     32  119    1 : tunables  120   60    8 : slabdata      0      0      0
size-32             3469   4403     32  119    1 : tunables  120   60    8 : slabdata     37     37      0
kmem_cache           175    175    160   25    1 : tunables  120   60    8 : slabdata      7      7      0
nr_dirty 2
nr_writeback 0
nr_unstable 0
nr_page_table_pages 184
nr_mapped 6676
nr_slab 219558
pgpgin 10459226
pgpgout 4610264
pswpin 0
pswpout 0
pgalloc_high 3302227
pgalloc_normal 122926982
pgalloc_dma 2598793
pgfree 129053912
pgactivate 1774683
pgdeactivate 1532657
pgfault 3836188
pgmajfault 829
pgrefill_high 0
pgrefill_normal 1520257
pgrefill_dma 12400
pgsteal_high 0
pgsteal_normal 2210379
pgsteal_dma 29734
pgscan_kswapd_high 0
pgscan_kswapd_normal 2084649
pgscan_kswapd_dma 31042
pgscan_direct_high 0
pgscan_direct_normal 671488
pgscan_direct_dma 10854
pginodesteal 36536
slabs_scanned 56443602
kswapd_steal 1783794
kswapd_inodesteal 317433
pageoutrun 12197
allocstall 13108
pgrotated 15029

-jf

Good. Just making sure.

Thanks. Let's see.

dentry-state before

after

slabinfo before

after

So you're basically drowning in mostly reclaimable slab here. These three entries are consuming over 800MB of zone_normal alone.

vmstat before

after

The things are being slowly scanned and freed, but it is being pretty lethargic.

Can you try echo 10000 > /proc/sys/vm/vfs_cache_pressure, and see how that goes?

Yes! Now it works. So is vfs_cache_pressure=10000 a sensible value to use?

# cat /proc/sys/fs/dentry-state /proc/slabinfo /proc/vmstat  ; dsmc incremental ; cat /proc/sys/fs/dentry-state /proc/slabinfo /proc/vmstat
354005  299163  45      0       0       0
slabinfo - version: 2.0
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
xfrm6_tunnel_spi       0      0     64   61    1 : tunables  120   60    8 : slabdata      0      0      0
fib6_nodes             7    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
ip6_dst_cache          9     18    224   18    1 : tunables  120   60    8 : slabdata      1      1      0
ndisc_cache            1     25    160   25    1 : tunables  120   60    8 : slabdata      1      1      0
raw6_sock              0      0    672    6    1 : tunables   54   27    8 : slabdata      0      0      0
udp6_sock              0      0    640    6    1 : tunables   54   27    8 : slabdata      0      0      0
tcp6_sock             14     18   1184    6    2 : tunables   24   12    8 : slabdata      3      3      0
ip_fib_hash           18    226     16  226    1 : tunables  120   60    8 : slabdata      1      1      0
ip_conntrack         480    516    320   12    1 : tunables   54   27    8 : slabdata     43     43      0
xfs_dqtrx              0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_dquots            34     36    336   12    1 : tunables   54   27    8 : slabdata      3      3      0
dm_tio                 0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
dm_io                  0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
rpc_buffers            8      8   2048    2    1 : tunables   24   12    8 : slabdata      4      4      0
rpc_tasks              8     25    160   25    1 : tunables  120   60    8 : slabdata      1      1      0
rpc_inode_cache        6      9    448    9    1 : tunables   54   27    8 : slabdata      1      1      0
unix_sock             36     36    416    9    1 : tunables   54   27    8 : slabdata      4      4      0
ip_mrt_cache           0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
tcp_tw_bucket          3     31    128   31    1 : tunables  120   60    8 : slabdata      1      1      0
tcp_bind_bucket       19    226     16  226    1 : tunables  120   60    8 : slabdata      1      1      0
tcp_open_request       0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
inet_peer_cache        5     61     64   61    1 : tunables  120   60    8 : slabdata      1      1      0
secpath_cache          0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
ip_dst_cache         127    150    256   15    1 : tunables  120   60    8 : slabdata     10     10      0
arp_cache             68    100    160   25    1 : tunables  120   60    8 : slabdata      4      4      0
raw4_sock              0      0    512    7    1 : tunables   54   27    8 : slabdata      0      0      0
udp_sock              21     21    544    7    1 : tunables   54   27    8 : slabdata      3      3      0
tcp_sock              31     42   1056    7    2 : tunables   24   12    8 : slabdata      6      6      0
flow_cache             0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
qla2xxx_srbs         128    155    128   31    1 : tunables  120   60    8 : slabdata      5      5      0
scsi_cmd_cache        11     22    352   11    1 : tunables   54   27    8 : slabdata      2      2      0
mqueue_inode_cache      1      7    512    7    1 : tunables   54   27    8 : slabdata      1      1      0
xfs_acl                0      0    304   13    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_chashlist      31401  45695     20  185    1 : tunables  120   60    8 : slabdata    247    247      0
xfs_ili              237    476    140   28    1 : tunables  120   60    8 : slabdata     17     17      0
xfs_ifork              0      0     56   70    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_efi_item           0      0    260   15    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_efd_item           0      0    260   15    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_buf_item           0      0    148   27    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_dabuf              0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_da_state           0      0    336   12    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_trans              0      0    596   13    2 : tunables   54   27    8 : slabdata      0      0      0
xfs_inode         614056 974083    368   11    1 : tunables   54   27    8 : slabdata  88553  88553     10
xfs_btree_cur          0      0    140   28    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_bmap_free_item      0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_buf_t             40     60    256   15    1 : tunables  120   60    8 : slabdata      4      4      0
linvfs_icache     614056 971420    384   10    1 : tunables   54   27    8 : slabdata  97142  97142     10
nfs_write_data        36     36    448    9    1 : tunables   54   27    8 : slabdata      4      4      0
nfs_read_data         32     36    416    9    1 : tunables   54   27    8 : slabdata      4      4      0
nfs_inode_cache        0      0    608    6    1 : tunables   54   27    8 : slabdata      0      0      0
nfs_page               0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
isofs_inode_cache      0      0    352   11    1 : tunables   54   27    8 : slabdata      0      0      0
hugetlbfs_inode_cache      1     12    324   12    1 : tunables   54   27    8 : slabdata      1      1      0
ext2_inode_cache       0      0    480    8    1 : tunables   54   27    8 : slabdata      0      0      0
ext2_xattr             0      0     48   81    1 : tunables  120   60    8 : slabdata      0      0      0
journal_handle        20    135     28  135    1 : tunables  120   60    8 : slabdata      1      1      0
journal_head         313    648     48   81    1 : tunables  120   60    8 : slabdata      8      8      0
revoke_table           4    290     12  290    1 : tunables  120   60    8 : slabdata      1      1      0
revoke_record          0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
ext3_inode_cache    3689   3696    480    8    1 : tunables   54   27    8 : slabdata    462    462      0
ext3_xattr             0      0     48   81    1 : tunables  120   60    8 : slabdata      0      0      0
dquot                  0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
eventpoll_pwq          0      0     36  107    1 : tunables  120   60    8 : slabdata      0      0      0
eventpoll_epi          0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
kioctx                 0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
kiocb                  0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
dnotify_cache          0      0     20  185    1 : tunables  120   60    8 : slabdata      0      0      0
file_lock_cache       27     82     96   41    1 : tunables  120   60    8 : slabdata      2      2      0
fasync_cache           0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
shmem_inode_cache     20     36    448    9    1 : tunables   54   27    8 : slabdata      4      4      0
posix_timers_cache      0      0    104   38    1 : tunables  120   60    8 : slabdata      0      0      0
uid_cache              3    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
sgpool-128            32     32   2048    2    1 : tunables   24   12    8 : slabdata     16     16      0
sgpool-64             32     32   1024    4    1 : tunables   54   27    8 : slabdata      8      8      0
sgpool-32             32     32    512    8    1 : tunables   54   27    8 : slabdata      4      4      0
sgpool-16             32     45    256   15    1 : tunables  120   60    8 : slabdata      3      3      0
sgpool-8              69     93    128   31    1 : tunables  120   60    8 : slabdata      3      3      0
cfq_pool              64    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
crq_pool               0      0     40   96    1 : tunables  120   60    8 : slabdata      0      0      0
deadline_drq           0      0     52   75    1 : tunables  120   60    8 : slabdata      0      0      0
as_arq                71    122     64   61    1 : tunables  120   60    8 : slabdata      2      2      0
blkdev_ioc            87    185     20  185    1 : tunables  120   60    8 : slabdata      1      1      0
blkdev_queue          32     32    464    8    1 : tunables   54   27    8 : slabdata      4      4      0
blkdev_requests       71    125    160   25    1 : tunables  120   60    8 : slabdata      5      5      0
biovec-(256)         256    256   3072    2    2 : tunables   24   12    8 : slabdata    128    128      0
biovec-128           256    260   1536    5    2 : tunables   24   12    8 : slabdata     52     52      0
biovec-64            256    260    768    5    1 : tunables   54   27    8 : slabdata     52     52      0
biovec-16            256    260    192   20    1 : tunables  120   60    8 : slabdata     13     13      0
biovec-4             256    305     64   61    1 : tunables  120   60    8 : slabdata      5      5      0
biovec-1             367    678     16  226    1 : tunables  120   60    8 : slabdata      2      3     60
bio                  342    369     96   41    1 : tunables  120   60    8 : slabdata      9      9      0
sock_inode_cache      70    100    384   10    1 : tunables   54   27    8 : slabdata     10     10      0
skbuff_head_cache    483    660    192   20    1 : tunables  120   60    8 : slabdata     33     33     60
sock                   3     11    352   11    1 : tunables   54   27    8 : slabdata      1      1      0
proc_inode_cache     564    616    352   11    1 : tunables   54   27    8 : slabdata     56     56      0
sigqueue               8     27    148   27    1 : tunables  120   60    8 : slabdata      1      1      0
radix_tree_node     7385   7392    276   14    1 : tunables   54   27    8 : slabdata    528    528      0
bdev_cache            14     27    448    9    1 : tunables   54   27    8 : slabdata      3      3      0
mnt_cache             30     41     96   41    1 : tunables  120   60    8 : slabdata      1      1      0
inode_cache         1480   1485    352   11    1 : tunables   54   27    8 : slabdata    135    135      0
dentry_cache      354127 697977    144   27    1 : tunables  120   60    8 : slabdata  25851  25851     24
filp                 825    850    160   25    1 : tunables  120   60    8 : slabdata     34     34      0
names_cache            6      6   4096    1    1 : tunables   24   12    8 : slabdata      6      6      0
idr_layer_cache       69     87    136   29    1 : tunables  120   60    8 : slabdata      3      3      0
buffer_head         6529   9900     52   75    1 : tunables  120   60    8 : slabdata    132    132      0
mm_struct             71     77    544    7    1 : tunables   54   27    8 : slabdata     11     11      0
vm_area_struct      1861   1974     84   47    1 : tunables  120   60    8 : slabdata     42     42      0
fs_cache              87    183     64   61    1 : tunables  120   60    8 : slabdata      3      3      0
files_cache           72     81    416    9    1 : tunables   54   27    8 : slabdata      9      9      0
signal_cache         122    164     96   41    1 : tunables  120   60    8 : slabdata      4      4      0
sighand_cache        108    108   1312    3    1 : tunables   24   12    8 : slabdata     36     36      0
task_struct          123    130   1440    5    2 : tunables   24   12    8 : slabdata     26     26      0
anon_vma             750   1160     12  290    1 : tunables  120   60    8 : slabdata      4      4      0
pgd                   57     57   4096    1    1 : tunables   24   12    8 : slabdata     57     57      0
size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-131072            0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-65536             0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-32768             0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
size-16384             2      2  16384    1    4 : tunables    8    4    0 : slabdata      2      2      0
size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0
size-8192              7      7   8192    1    2 : tunables    8    4    0 : slabdata      7      7      0
size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    8 : slabdata      0      0      0
size-4096            457    457   4096    1    1 : tunables   24   12    8 : slabdata    457    457     12
size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    8 : slabdata      0      0      0
size-2048            212    212   2048    2    1 : tunables   24   12    8 : slabdata    106    106      0
size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
size-1024            186    200   1024    4    1 : tunables   54   27    8 : slabdata     50     50      0
size-512(DMA)          0      0    512    8    1 : tunables   54   27    8 : slabdata      0      0      0
size-512             406   1336    512    8    1 : tunables   54   27    8 : slabdata    167    167     27
size-256(DMA)          0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
size-256             388   1230    256   15    1 : tunables  120   60    8 : slabdata     82     82      0
size-192(DMA)          0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
size-192            8415  14060    192   20    1 : tunables  120   60    8 : slabdata    703    703      0
size-128(DMA)          0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
size-128            8732  12276    128   31    1 : tunables  120   60    8 : slabdata    396    396      0
size-96(DMA)           0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
size-96            14336  20910     96   41    1 : tunables  120   60    8 : slabdata    510    510     40
size-64(DMA)           0      0     64   61    1 : tunables  120   60    8 : slabdata      0      0      0
size-64            46020  95770     64   61    1 : tunables  120   60    8 : slabdata   1570   1570      0
size-32(DMA)           0      0     32  119    1 : tunables  120   60    8 : slabdata      0      0      0
size-32             3562   4403     32  119    1 : tunables  120   60    8 : slabdata     37     37      0
kmem_cache           175    175    160   25    1 : tunables  120   60    8 : slabdata      7      7      0
nr_dirty 228
nr_writeback 0
nr_unstable 0
nr_page_table_pages 184
nr_mapped 6682
nr_slab 218165
pgpgin 10514058
pgpgout 4921463
pswpin 0
pswpout 0
pgalloc_high 3382039
pgalloc_normal 123996797
pgalloc_dma 2598824
pgfree 130194109
pgactivate 1779298
pgdeactivate 1535398
pgfault 3914892
pgmajfault 841
pgrefill_high 0
pgrefill_normal 1522984
pgrefill_dma 12414
pgsteal_high 0
pgsteal_normal 2214669
pgsteal_dma 29742
pgscan_kswapd_high 0
pgscan_kswapd_normal 2088411
pgscan_kswapd_dma 31054
pgscan_direct_high 0
pgscan_direct_normal 673264
pgscan_direct_dma 10866
pginodesteal 36536
slabs_scanned 57079454
kswapd_steal 1787022
kswapd_inodesteal 318737
pageoutrun 12219
allocstall 13132
pgrotated 15048
Tivoli Storage Manager
Command Line Backup/Archive Client Interface - Version 5, Release 1, Level 6.0
(C) Copyright IBM Corporation 1990, 2003 All Rights Reserved.

Node Name: BCCSFS
Session established with server TSM: Solaris 7/8
Server Version 5, Release 1, Level 6.5
Server date/time: 07/26/2004 13:56:02 Last access: 07/26/2004 13:55:37

Incremental backup of volume '/etc/'

Incremental backup of volume '/export/homebccs/'

Incremental backup of volume '/export/homebccs1/'

Incremental backup of volume '/export/netbccs/'

Incremental backup of volume '/export/local/'
ANS1898I ***** Processed 500 files *****
ANS1898I ***** Processed 3,000 files *****

<snip>

Total number of objects inspected: 4,719,746
Total number of objects backed up: 538

Total number of objects updated:          0
Total number of objects rebound:          0
Total number of objects deleted:          0
Total number of objects expired:         22
Total number of objects failed:           1
Total number of bytes transferred:    154.57 MB
Data transfer time:                    8.20 sec
Network data transfer rate:        19,295.48 KB/sec
Aggregate data transfer rate:         60.29 KB/sec
Objects compressed by:                    0%
Elapsed processing time:           00:43:44

54530 49612 45 0 0 0
slabinfo - version: 2.0

# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
xfrm6_tunnel_spi       0      0     64   61    1 : tunables  120   60    8 : slabdata      0      0      0
fib6_nodes             7    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
ip6_dst_cache          9     18    224   18    1 : tunables  120   60    8 : slabdata      1      1      0
ndisc_cache            1     25    160   25    1 : tunables  120   60    8 : slabdata      1      1      0
raw6_sock              0      0    672    6    1 : tunables   54   27    8 : slabdata      0      0      0
udp6_sock              0      0    640    6    1 : tunables   54   27    8 : slabdata      0      0      0
tcp6_sock             14     18   1184    6    2 : tunables   24   12    8 : slabdata      3      3      0
ip_fib_hash           18    226     16  226    1 : tunables  120   60    8 : slabdata      1      1      0
ip_conntrack         465    480    320   12    1 : tunables   54   27    8 : slabdata     40     40      0
xfs_dqtrx              0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_dquots            10     36    336   12    1 : tunables   54   27    8 : slabdata      3      3      0
dm_tio                 0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
dm_io                  0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
rpc_buffers            8      8   2048    2    1 : tunables   24   12    8 : slabdata      4      4      0
rpc_tasks              8     25    160   25    1 : tunables  120   60    8 : slabdata      1      1      0
rpc_inode_cache        6      9    448    9    1 : tunables   54   27    8 : slabdata      1      1      0
unix_sock             11     36    416    9    1 : tunables   54   27    8 : slabdata      4      4      0
ip_mrt_cache           0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
tcp_tw_bucket         16     31    128   31    1 : tunables  120   60    8 : slabdata      1      1      0
tcp_bind_bucket       20    226     16  226    1 : tunables  120   60    8 : slabdata      1      1      0
tcp_open_request       0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
inet_peer_cache       24     61     64   61    1 : tunables  120   60    8 : slabdata      1      1      0
secpath_cache          0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
xfrm_dst_cache         0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
ip_dst_cache         149    195    256   15    1 : tunables  120   60    8 : slabdata     13     13      0
arp_cache             31     75    160   25    1 : tunables  120   60    8 : slabdata      3      3      0
raw4_sock              0      0    512    7    1 : tunables   54   27    8 : slabdata      0      0      0
udp_sock              10     21    544    7    1 : tunables   54   27    8 : slabdata      3      3      0
tcp_sock              33     49   1056    7    2 : tunables   24   12    8 : slabdata      7      7      0
flow_cache             0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
qla2xxx_srbs         279    279    128   31    1 : tunables  120   60    8 : slabdata      9      9      0
scsi_cmd_cache        52     66    352   11    1 : tunables   54   27    8 : slabdata      6      6      0
mqueue_inode_cache      1      7    512    7    1 : tunables   54   27    8 : slabdata      1      1      0
xfs_acl                0      0    304   13    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_chashlist       3780  45510     20  185    1 : tunables  120   60    8 : slabdata    246    246    180
xfs_ili               38    392    140   28    1 : tunables  120   60    8 : slabdata     14     14      0
xfs_ifork              0      0     56   70    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_efi_item           0      0    260   15    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_efd_item           0      0    260   15    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_buf_item           0      0    148   27    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_dabuf             75    226     16  226    1 : tunables  120   60    8 : slabdata      1      1      0
xfs_da_state           0      0    336   12    1 : tunables   54   27    8 : slabdata      0      0      0
xfs_trans             13     13    596   13    2 : tunables   54   27    8 : slabdata      1      1      0
xfs_inode          78837  86878    368   11    1 : tunables   54   27    8 : slabdata   7898   7898      0
xfs_btree_cur          0      0    140   28    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_bmap_free_item      0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
xfs_buf_t            117    165    256   15    1 : tunables  120   60    8 : slabdata     10     11      0
linvfs_icache      78836  86880    384   10    1 : tunables   54   27    8 : slabdata   8688   8688      0
nfs_write_data        36     36    448    9    1 : tunables   54   27    8 : slabdata      4      4      0
nfs_read_data         32     36    416    9    1 : tunables   54   27    8 : slabdata      4      4      0
nfs_inode_cache        0      0    608    6    1 : tunables   54   27    8 : slabdata      0      0      0
nfs_page               0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
isofs_inode_cache      0      0    352   11    1 : tunables   54   27    8 : slabdata      0      0      0
hugetlbfs_inode_cache      1     12    324   12    1 : tunables   54   27    8 : slabdata      1      1      0
ext2_inode_cache       0      0    480    8    1 : tunables   54   27    8 : slabdata      0      0      0
ext2_xattr             0      0     48   81    1 : tunables  120   60    8 : slabdata      0      0      0
journal_handle        17    135     28  135    1 : tunables  120   60    8 : slabdata      1      1      0
journal_head          93    162     48   81    1 : tunables  120   60    8 : slabdata      2      2      0
revoke_table           4    290     12  290    1 : tunables  120   60    8 : slabdata      1      1      0
revoke_record          0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
ext3_inode_cache     257    648    480    8    1 : tunables   54   27    8 : slabdata     81     81      0
ext3_xattr             0      0     48   81    1 : tunables  120   60    8 : slabdata      0      0      0
dquot                  0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
eventpoll_pwq          0      0     36  107    1 : tunables  120   60    8 : slabdata      0      0      0
eventpoll_epi          0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
kioctx                 0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
kiocb                  0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
dnotify_cache          0      0     20  185    1 : tunables  120   60    8 : slabdata      0      0      0
file_lock_cache       18     82     96   41    1 : tunables  120   60    8 : slabdata      2      2      0
fasync_cache           0      0     16  226    1 : tunables  120   60    8 : slabdata      0      0      0
shmem_inode_cache     20     36    448    9    1 : tunables   54   27    8 : slabdata      4      4      0
posix_timers_cache      0      0    104   38    1 : tunables  120   60    8 : slabdata      0      0      0
uid_cache              3    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
sgpool-128            32     32   2048    2    1 : tunables   24   12    8 : slabdata     16     16      0
sgpool-64             32     32   1024    4    1 : tunables   54   27    8 : slabdata      8      8      0
sgpool-32             40     40    512    8    1 : tunables   54   27    8 : slabdata      5      5      0
sgpool-16             38     45    256   15    1 : tunables  120   60    8 : slabdata      3      3      0
sgpool-8             107    155    128   31    1 : tunables  120   60    8 : slabdata      5      5      0
cfq_pool              64    119     32  119    1 : tunables  120   60    8 : slabdata      1      1      0
crq_pool               0      0     40   96    1 : tunables  120   60    8 : slabdata      0      0      0
deadline_drq           0      0     52   75    1 : tunables  120   60    8 : slabdata      0      0      0
as_arq               145    183     64   61    1 : tunables  120   60    8 : slabdata      3      3      0
blkdev_ioc            73    185     20  185    1 : tunables  120   60    8 : slabdata      1      1      0
blkdev_queue          32     32    464    8    1 : tunables   54   27    8 : slabdata      4      4      0
blkdev_requests      151    175    160   25    1 : tunables  120   60    8 : slabdata      7      7      0
biovec-(256)         256    256   3072    2    2 : tunables   24   12    8 : slabdata    128    128      0
biovec-128           256    260   1536    5    2 : tunables   24   12    8 : slabdata     52     52      0
biovec-64            260    260    768    5    1 : tunables   54   27    8 : slabdata     52     52      0
biovec-16            258    260    192   20    1 : tunables  120   60    8 : slabdata     13     13      0
biovec-4             302    427     64   61    1 : tunables  120   60    8 : slabdata      6      7      0
biovec-1             354    452     16  226    1 : tunables  120   60    8 : slabdata      2      2      0
bio                  356    410     96   41    1 : tunables  120   60    8 : slabdata     10     10      0
sock_inode_cache      79     90    384   10    1 : tunables   54   27    8 : slabdata      9      9      0
skbuff_head_cache    542    640    192   20    1 : tunables  120   60    8 : slabdata     32     32     60
sock                   3     11    352   11    1 : tunables   54   27    8 : slabdata      1      1      0
proc_inode_cache     411    572    352   11    1 : tunables   54   27    8 : slabdata     52     52    172
sigqueue              18     27    148   27    1 : tunables  120   60    8 : slabdata      1      1      0
radix_tree_node    48011  48104    276   14    1 : tunables   54   27    8 : slabdata   3436   3436    156
bdev_cache            14     27    448    9    1 : tunables   54   27    8 : slabdata      3      3      0
mnt_cache             30     41     96   41    1 : tunables  120   60    8 : slabdata      1      1      0
inode_cache         1480   1485    352   11    1 : tunables   54   27    8 : slabdata    135    135      0
dentry_cache       54561  65205    144   27    1 : tunables  120   60    8 : slabdata   2415   2415      0
filp                 851    875    160   25    1 : tunables  120   60    8 : slabdata     35     35      0
names_cache           15     20   4096    1    1 : tunables   24   12    8 : slabdata     15     20      0
idr_layer_cache       69     87    136   29    1 : tunables  120   60    8 : slabdata      3      3      0
buffer_head         1582   9525     52   75    1 : tunables  120   60    8 : slabdata    127    127      0
mm_struct             77     77    544    7    1 : tunables   54   27    8 : slabdata     11     11      0
vm_area_struct      1906   1974     84   47    1 : tunables  120   60    8 : slabdata     42     42      0
fs_cache              73    183     64   61    1 : tunables  120   60    8 : slabdata      3      3      0
files_cache           58     81    416    9    1 : tunables   54   27    8 : slabdata      9      9      0
signal_cache         108    164     96   41    1 : tunables  120   60    8 : slabdata      4      4      0
sighand_cache        104    108   1312    3    1 : tunables   24   12    8 : slabdata     36     36      0
task_struct          116    130   1440    5    2 : tunables   24   12    8 : slabdata     26     26      0
anon_vma             856   1160     12  290    1 : tunables  120   60    8 : slabdata      4      4      0
pgd                   58     58   4096    1    1 : tunables   24   12    8 : slabdata     58     58      0
size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-131072            0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-65536             0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-32768             0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
size-16384             2      2  16384    1    4 : tunables    8    4    0 : slabdata      2      2      0
size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0
size-8192              6      6   8192    1    2 : tunables    8    4    0 : slabdata      6      6      0
size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    8 : slabdata      0      0      0
size-4096            459    464   4096    1    1 : tunables   24   12    8 : slabdata    459    464      0
size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    8 : slabdata      0      0      0
size-2048            258    258   2048    2    1 : tunables   24   12    8 : slabdata    129    129     60
size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
size-1024            174    192   1024    4    1 : tunables   54   27    8 : slabdata     48     48      0
size-512(DMA)          0      0    512    8    1 : tunables   54   27    8 : slabdata      0      0      0
size-512             412   1336    512    8    1 : tunables   54   27    8 : slabdata    167    167     27
size-256(DMA)          0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
size-256             554   1230    256   15    1 : tunables  120   60    8 : slabdata     82     82     60
size-192(DMA)          0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
size-192            1327   6240    192   20    1 : tunables  120   60    8 : slabdata    312    312      0
size-128(DMA)          0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
size-128            1691   9641    128   31    1 : tunables  120   60    8 : slabdata    311    311      0
size-96(DMA)           0      0     96   41    1 : tunables  120   60    8 : slabdata      0      0      0
size-96             2387   4838     96   41    1 : tunables  120   60    8 : slabdata    118    118      0
size-64(DMA)           0      0     64   61    1 : tunables  120   60    8 : slabdata      0      0      0
size-64             6278  13237     64   61    1 : tunables  120   60    8 : slabdata    217    217    120
size-32(DMA)           0      0     32  119    1 : tunables  120   60    8 : slabdata      0      0      0
size-32             3710   4403     32  119    1 : tunables  120   60    8 : slabdata     37     37      0
kmem_cache           175    175    160   25    1 : tunables  120   60    8 : slabdata      7      7      0
nr_dirty 10
nr_writeback 0
nr_unstable 0
nr_page_table_pages 189
nr_mapped 6811
nr_slab 25974
pgpgin 12321778
pgpgout 5971354
pswpin 0
pswpout 0
pgalloc_high 3640914
pgalloc_normal 130923538
pgalloc_dma 2717458
pgfree 137566296
pgactivate 2013396
pgdeactivate 1719566
pgfault 4145489
pgmajfault 950
pgrefill_high 0
pgrefill_normal 1694966
pgrefill_dma 24600
pgsteal_high 0
pgsteal_normal 2415816
pgsteal_dma 52874
pgscan_kswapd_high 0
pgscan_kswapd_normal 2259582
pgscan_kswapd_dma 52964
pgscan_direct_high 0
pgscan_direct_normal 709564
pgscan_direct_dma 12812
pginodesteal 36838
slabs_scanned 73218791
kswapd_steal 1974234
kswapd_inodesteal 425845
pageoutrun 13329
allocstall 14250
pgrotated 15049

-jf

Well if it keeps you going then yes. For now. I'll get back to you with something to try though.

Thanks.

This is not the same problem as I and other are describing. There is no free memory when the OOM killer activates in our situation. The kernel has allocated all available ram and as such, the OOM killer can't kill the memory hog because it's the kernel, itself. So the OOM killer kills all the big apps running ...but it's to no use because the kernel just keeps trying to use more until the cd is completed. After which the memory is still never released.

Your thread has nothing to do with mine.

I believe it could be the same problem. Jan-Frode's system has all ZONE_NORMAL memory used up. The free memory would be highmem which would be unsuable for those allocations that are causing OOM.

The vfs_cache_pressure change could possibly be responsible for the problem... I don't have the code in front of me, but I think it divides by 100 first, then multiplies by vfs_cache_pressure. I wouldn't have thought this would have such a large impact though.

Ed,

Can you please try Nicks suggestion? A vm problem makes sense to me, what doesn't make sense is that we leak memory on some hardware while burning and don't on others.

Jens Axboe

I tried it, i dont slow down or crash when burning the cd the first time. It's a small cd that doesn't take up my entire ram size, but the memory is still not freed. If i tried it again i would be rebooting right now. I only have 70MB out of 650MB free after burning the cd. Cache only takes up 122MB, and buf takes up 1MB. and i'm using 100MB of swap. I will run vmstat when i do it when i get home later today.

It's not so much that the kernel is leaking memory, I think it thinks it's handling a pointer to data it's supposed to write to disk, but it's writing the wrong data, either a slightly misaligned offset or mangled pointer because the audio cd did write but the audio it wrote is unintelligable. It almost sort of sounds like it should but it's completely fubared. And i've done this with swab on and off before thinking the drive automatically wrote audio with SWAB on and cdrecord's swab was countering it or something but that was not the case. The audio source files were ripped from a cd using the same drive and they sound good on the harddrive. The drive seems to have no real problem ripping audio. Just writing it. Normal cds show no problem as i've previously mentioned.

If this is a vfs problem then i'd like to know what audio writing has to do with filesystems since it's raw data. Even ignoring the mem leak problem that appears to manifest in different ways on different computers, this OOM situation only happens to me when burning audio cds, not data.

OK so it does sound like a different problem.

I didn't follow your other thread closely... does /proc/slabinfo show any evidence of a leak?

Surprisingly no. You'd think that since the kernel is responsible for saying what memory can't be touched or swapped out it would have some sort of tag on the huge 600MB of ram I currently can't do anything with since i burned that audio cd but slabinfo doesn't seem to show anything about it. Maybe i'm reading it wrong.

# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext2_inode_cache      11     18    416    9    1 : tunables   54   27    0 : slabdata      2      2      0
ext2_xattr             0      0     44   88    1 : tunables  120   60    0 : slabdata      0      0      0
smb_request            0      0    256   15    1 : tunables  120   60    0 : slabdata      0      0      0
smb_inode_cache        0      0    320   12    1 : tunables   54   27    0 : slabdata      0      0      0
unix_sock             47     50    384   10    1 : tunables   54   27    0 : slabdata      5      5      0
tcp_tw_bucket          0      0     96   41    1 : tunables  120   60    0 : slabdata      0      0      0
tcp_bind_bucket       16    226     16  226    1 : tunables  120   60    0 : slabdata      1      1      0
tcp_open_request       0      0     64   61    1 : tunables  120   60    0 : slabdata      0      0      0
inet_peer_cache        0      0     64   61    1 : tunables  120   60    0 : slabdata      0      0      0
ip_fib_hash           10    226     16  226    1 : tunables  120   60    0 : slabdata      1      1      0
ip_dst_cache          21     30    256   15    1 : tunables  120   60    0 : slabdata      2      2      0
arp_cache              2     31    128   31    1 : tunables  120   60    0 : slabdata      1      1      0
raw4_sock              0      0    480    8    1 : tunables   54   27    0 : slabdata      0      0      0
udp_sock               1      8    480    8    1 : tunables   54   27    0 : slabdata      1      1      0
tcp_sock              28     28    992    4    1 : tunables   54   27    0 : slabdata      7      7      0
flow_cache             0      0     96   41    1 : tunables  120   60    0 : slabdata      0      0      0
uhci_urb_priv          1     88     44   88    1 : tunables  120   60    0 : slabdata      1      1      0
mqueue_inode_cache      1      8    480    8    1 : tunables   54   27    0 : slabdata      1      1      0
journal_handle         2    135     28  135    1 : tunables  120   60    0 : slabdata      1      1      0
journal_head          29    162     48   81    1 : tunables  120   60    0 : slabdata      2      2      0
revoke_table          12    290     12  290    1 : tunables  120   60    0 : slabdata      1      1      0
revoke_record          0      0     16  226    1 : tunables  120   60    0 : slabdata      0      0      0
ext3_inode_cache     829   1752    480    8    1 : tunables   54   27    0 : slabdata    219    219      0
ext3_xattr             0      0     44   88    1 : tunables  120   60    0 : slabdata      0      0      0
eventpoll_pwq          0      0     36  107    1 : tunables  120   60    0 : slabdata      0      0      0
eventpoll_epi          0      0     96   41    1 : tunables  120   60    0 : slabdata      0      0      0
kioctx                 0      0    160   25    1 : tunables  120   60    0 : slabdata      0      0      0
kiocb                  0      0     96   41    1 : tunables  120   60    0 : slabdata      0      0      0
dnotify_cache          0      0     20  185    1 : tunables  120   60    0 : slabdata      0      0      0
file_lock_cache       41     41     96   41    1 : tunables  120   60    0 : slabdata      1      1      0
fasync_cache           2    226     16  226    1 : tunables  120   60    0 : slabdata      1      1      0
shmem_inode_cache     11     18    416    9    1 : tunables   54   27    0 : slabdata      2      2      0
posix_timers_cache      0      0     96   41    1 : tunables  120   60    0 : slabdata      0      0      0
uid_cache              4    119     32  119    1 : tunables  120   60    0 : slabdata      1      1      0
cfq_pool              64    119     32  119    1 : tunables  120   60    0 : slabdata      1      1      0
crq_pool               0      0     36  107    1 : tunables  120   60    0 : slabdata      0      0      0
deadline_drq           0      0     48   81    1 : tunables  120   60    0 : slabdata      0      0      0
as_arq                20     65     60   65    1 : tunables  120   60    0 : slabdata      1      1      0
blkdev_ioc            67    185     20  185    1 : tunables  120   60    0 : slabdata      1      1      0
blkdev_queue          19     24    452    8    1 : tunables   54   27    0 : slabdata      3      3      0
blkdev_requests       19     26    152   26    1 : tunables  120   60    0 : slabdata      1      1      0
biovec-(256)         256    256   3072    2    2 : tunables   24   12    0 : slabdata    128    128      0
biovec-128           256    260   1536    5    2 : tunables   24   12    0 : slabdata     52     52      0
biovec-64            256    260    768    5    1 : tunables   54   27    0 : slabdata     52     52      0
biovec-16            256    260    192   20    1 : tunables  120   60    0 : slabdata     13     13      0
biovec-4             256    305     64   61    1 : tunables  120   60    0 : slabdata      5      5      0
biovec-1             264    452     16  226    1 : tunables  120   60    0 : slabdata      2      2      0
bio                  264    305     64   61    1 : tunables  120   60    0 : slabdata      5      5      0
sock_inode_cache      94     99    352   11    1 : tunables   54   27    0 : slabdata      9      9      0
skbuff_head_cache    230    350    160   25    1 : tunables  120   60    0 : slabdata     14     14      0
sock                   2     12    320   12    1 : tunables   54   27    0 : slabdata      1      1      0
proc_inode_cache      51    156    320   12    1 : tunables   54   27    0 : slabdata     13     13      0
sigqueue              16     27    148   27    1 : tunables  120   60    0 : slabdata      1      1      0
radix_tree_node     2148   2254    276   14    1 : tunables   54   27    0 : slabdata    161    161      0
bdev_cache            12     18    416    9    1 : tunables   54   27    0 : slabdata      2      2      0
mnt_cache             23     41     96   41    1 : tunables  120   60    0 : slabdata      1      1      0
inode_cache          522    528    320   12    1 : tunables   54   27    0 : slabdata     44     44      0
dentry_cache        1642   4592    140   28    1 : tunables  120   60    0 : slabdata    164    164      0
filp                1430   1450    160   25    1 : tunables  120   60    0 : slabdata     58     58      0
names_cache            6      6   4096    1    1 : tunables   24   12    0 : slabdata      6      6      0
idr_layer_cache       86     87    136   29    1 : tunables  120   60    0 : slabdata      3      3      0
buffer_head        26173  28593     48   81    1 : tunables  120   60    0 : slabdata    353    353      0
mm_struct             70     70    512    7    1 : tunables   54   27    0 : slabdata     10     10      0
vm_area_struct      3212   3525     84   47    1 : tunables  120   60    0 : slabdata     75     75      0
fs_cache              72    119     32  119    1 : tunables  120   60    0 : slabdata      1      1      0
files_cache           63     63    416    9    1 : tunables   54   27    0 : slabdata      7      7      0
signal_cache          82     82     96   41    1 : tunables  120   60    0 : slabdata      2      2      0
sighand_cache         75     75   1312    3    1 : tunables   24   12    0 : slabdata     25     25      0
task_struct          100    100   1440    5    2 : tunables   24   12    0 : slabdata     20     20      0
anon_vma            1602   2035      8  407    1 : tunables  120   60    0 : slabdata      5      5      0
pgd                   59     59   4096    1    1 : tunables   24   12    0 : slabdata     59     59      0
size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-131072            0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
size-65536             1      1  65536    1   16 : tunables    8    4    0 : slabdata      1      1      0
size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-32768             0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
size-16384             3      3  16384    1    4 : tunables    8    4    0 : slabdata      3      3      0
size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0
size-8192            103    103   8192    1    2 : tunables    8    4    0 : slabdata    103    103      0
size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    0 : slabdata      0      0      0
size-4096             62     62   4096    1    1 : tunables   24   12    0 : slabdata     62     62      0
size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    0 : slabdata      0      0      0
size-2048            160    160   2048    2    1 : tunables   24   12    0 : slabdata     80     80      0
size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    0 : slabdata      0      0      0
size-1024             89     92   1024    4    1 : tunables   54   27    0 : slabdata     23     23      0
size-512(DMA)          0      0    512    8    1 : tunables   54   27    0 : slabdata      0      0      0
size-512             262    456    512    8    1 : tunables   54   27    0 : slabdata     57     57      0
size-256(DMA)          0      0    256   15    1 : tunables  120   60    0 : slabdata      0      0      0
size-256             172    345    256   15    1 : tunables  120   60    0 : slabdata     23     23      0
size-192(DMA)          0      0    192   20    1 : tunables  120   60    0 : slabdata      0      0      0
size-192             160    160    192   20    1 : tunables  120   60    0 : slabdata      8      8      0
size-128(DMA)          0      0    128   31    1 : tunables  120   60    0 : slabdsize-128            1744   1829    128   31    1 : tunables  120   60    0 : slabdata     59     59      0
size-64(DMA)           0      0     64   61    1 : tunables  120   60    0 : slabdata      0      0      0
size-64             2673   3416     64   61    1 : tunables  120   60    0 : slabdata     56     56      0
size-32(DMA)           0      0     32  119    1 : tunables  120   60    0 : slabdata      0      0      0
size-32             2321   2380     32  119    1 : tunables  120   60    0 : slabdata     20     20      0
kmem_cache           124    124    128   31    1 : tunables  120   60    0 : slabdata      4      4      0
ata      0      0      0

It could be memory coming straight out of the page allocator that isn't being freed.

Jens, any ideas?

Con Kolivas' 2.6.8-rc1-ck6 snapshot patch seems fix the problem. Not only is my audio not corrupted when i write a disk but I get no mem leak situation and thus no OOM. I did 5 dummy burns with no swap being used and stable vm statistics, final real burn resulted in successful disc.

2.6.8-rc1 2.6.8-rc1-mm both flipped out. ck touches all relevent files so something the patch does fixed whatever was wrong.

This makes about zero sense to me (a leak I can understand, corrupted audio is more weird). Can you point me at the specific patch used?

Jens Axboe

the corruption may have been a combination of not burning a cd without -pad and without -swab at the same time as using -audio. It appears my drive burns audio just fine with just -audio as the argument for cdrecord.

http://ck.kolivas.org/patches/2.6/2.6.7/2.6.8-rc1/snapshot-2.6.8-rc1-ck6-0407151120.bz2

The corruption issue _must_ be a user error.

Did you use cfq with the other kernels as well? If your problem isn't one of the vm, I still don't see how this patch can make any difference whatsoever. There are no changes to the paths used writing the data, once you exit the page allocation.

Jens Axboe

Is there any way we could get a characterization of the kind of memory that's proliferating here? e.g. could you snapshot /proc/meminfo, /proc/slabinfo, and /proc/vmstat at regular intervals during the run?

-- wli