Skip to contentSkip to conversation

Tracking down a memory leak

DISCUSSION Marco Colombo 2 participants 4 messages Jun 20, 2005 – Jun 28, 2005 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi,
today I've found a server in a OOM condition, the funny thing is that after some investigation I've found no process that has mem allocated to. I even switched to single user, here's what I've found:

sh-2.05b# ps axu; free

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  2708  432 ?        S    May16   0:10 init [S]
root         2  0.0  0.0     0    0 ?        SWN  May16   0:03 [ksoftirqd/0]
root         3  0.0  0.0     0    0 ?        SW<  May16   1:49 [events/0]
root         4  0.0  0.0     0    0 ?        SW<  May16   0:00 [khelper]
root        16  0.0  0.0     0    0 ?        SW<  May16   0:00 [kacpid]
root        87  0.0  0.0     0    0 ?        SW<  May16   0:33 [kblockd/0]
root        95  0.0  0.0     0    0 ?        SW   May16   0:00 [khubd]
root       152  0.0  0.0     0    0 ?        SW<  May16   0:00 [aio/0]
root       151  0.0  0.0     0    0 ?        SW   May16  45:44 [kswapd0]
root       245  0.0  0.0     0    0 ?        SW   May16   0:00 [kseriod]
root       473  0.0  0.0     0    0 ?        SW   May16   0:00 [md7_raid1]
root       475  0.0  0.0     0    0 ?        SW   May16   0:11 [md6_raid1]
root       477  0.0  0.0     0    0 ?        SW   May16   0:00 [md5_raid1]
root       479  0.0  0.0     0    0 ?        SW   May16   0:00 [md4_raid1]
root       481  0.0  0.0     0    0 ?        SW   May16   0:00 [md3_raid1]
root       483  0.0  0.0     0    0 ?        SW   May16   0:08 [md2_raid1]
root       485  0.0  0.0     0    0 ?        SW   May16   0:00 [md1_raid1]
root       486  0.0  0.0     0    0 ?        SW   May16   0:02 [md0_raid1]
root       487  0.0  0.0     0    0 ?        SW   May16   0:38 [kjournald]
root       690  0.0  0.0     0    0 ?        SW   May16   0:03 [kjournald]
root       691  0.0  0.0     0    0 ?        SW   May16   4:05 [kjournald]
root       692  0.0  0.0     0    0 ?        SW   May16   0:03 [kjournald]
root       693  0.0  0.0     0    0 ?        SW   May16   0:03 [kjournald]
root       694  0.0  0.0     0    0 ?        SW   May16   0:03 [kjournald]
root       695  0.0  0.0     0    0 ?        SW   May16   6:14 [kjournald]
root       696  0.0  0.0     0    0 ?        SW   May16   0:00 [kjournald]
root     16545  0.0  0.0     0    0 ?        SW   11:47   0:00 [pdflush]
root     16551  0.0  0.0     0    0 ?        SW   11:47   0:00 [pdflush]
root     16889  0.0  0.0  3072  744 ?        S    11:59   0:00 minilogd
root     17235  0.0  0.0  2708  444 ttyS0    S    12:00   0:00 init [S]
root     17236  0.0  0.1  3176 1180 ttyS0    S    12:00   0:00 /bin/sh
root     17248  0.0  0.0  2732  748 ttyS0    R    12:07   0:00 ps axu
             total       used       free     shared    buffers     cached
Mem:       1035812     898524     137288          0       3588      16732
-/+ buffers/cache:     878204     157608

Swap: 1049248 788 1048460
sh-2.05b# uptime
12:13:28 up 35 days, 1:48, 0 users, load average: 0.00, 0.59, 16.13
sh-2.05b# uname -a
Linux xxxx.example.org 2.6.10-1.12_FC2.marco #1 Mon Feb 7 14:53:42 CET 2005 i686 athlon i386 GNU/Linux

I know this is an old Fedora Core 2 kernel, eventually I'll bring the issue on thier lists. An upgrade has already been scheduled for this host, so I'm not really pressed in tracking this specific bug (unless it occurs on the new system, of course).

Anyway, I just wonder if generally there's a way to find out where those 850+ MBs are allocated. Since there are no big user processes, I'm assuming it's a memory leak in kernel space. I'm curious, this is the first time I see something like this. Any suggestion what to look at besides 'ps' and 'free'?

The server has been mainly running PostgreSQL at a fairly high load for the last 35 days, BTW.

TIA,
.TM.

Signature · 5 lines
      ____/  ____/   /
     /      /       /                   Marco Colombo
    ___/  ___  /   /                  Technical Manager
   /          /   /                      ESI s.r.l.
 _____/ _____/  _/                      [email protected]
#1 lore raw
Quoted 4 lines · Marco Colombo

Hi,
today I've found a server in a OOM condition, the funny thing is that after some investigation I've found no process that has mem allocated to. I even switched to single user, here's what I've found:

[...]

Quoted 21 lines

total used free shared buffers cached

Mem:       1035812     898524     137288          0       3588      16732
-/+ buffers/cache:     878204     157608

Swap: 1049248 788 1048460
sh-2.05b# uptime
12:13:28 up 35 days, 1:48, 0 users, load average: 0.00, 0.59, 16.13
sh-2.05b# uname -a
Linux xxxx.example.org 2.6.10-1.12_FC2.marco #1 Mon Feb 7 14:53:42 CET 2005 i686 athlon i386 GNU/Linux

I know this is an old Fedora Core 2 kernel, eventually I'll bring the issue on thier lists. An upgrade has already been scheduled for this host, so I'm not really pressed in tracking this specific bug (unless it occurs on the new system, of course).

Anyway, I just wonder if generally there's a way to find out where those 850+ MBs are allocated. Since there are no big user processes, I'm assuming it's a memory leak in kernel space. I'm curious, this is the first time I see something like this. Any suggestion what to look at besides 'ps' and 'free'?

Quoted 5 lines

The server has been mainly running PostgreSQL at a fairly high load for the last 35 days, BTW.

TIA,
.TM.

Thanks to everybody who replied to me. Here's more data:

sh-2.05b# sort -rn +1 /proc/slabinfo | head -5

biovec-1          7502216 7502296     16  226    1 : tunables  120   60    0 : slabdata  33196  33196      0
bio               7502216 7502262     96   41    1 : tunables  120   60    0 : slabdata 182982 182982      0
size-64             4948   5307     64   61    1 : tunables  120   60    0 : slabdata     87     87      0
buffer_head         3691   3750     52   75    1 : tunables  120   60    0 : slabdata     50     50      0
dentry_cache        2712   2712    164   24    1 : tunables  120   60    0 : slabdata    113    113      0

I've found no way do free that memory, so decided to reboot it. In the following days I had been monitoring the system after upgrading to kernel-2.6.10-1.770_FC2. Here are the results I got, day by day:

bio               115333 115333     96   41    1 : tunables  120   60    0 : slabdata   2813   2813      0
biovec-1          115322 115486     16  226    1 : tunables  120   60    0 : slabdata    511    511      0
biovec-1          325006 325440     16  226    1 : tunables  120   60    0 : slabdata   1440   1440      0
bio               324987 325212     96   41    1 : tunables  120   60    0 : slabdata   7930   7932      0
bio               538535 538535     96   41    1 : tunables  120   60    0 : slabdata  13135  13135      0
biovec-1          538528 538784     16  226    1 : tunables  120   60    0 : slabdata   2384   2384      0
bio               749870 750218     96   41    1 : tunables  120   60    0 : slabdata  18296  18298      0
biovec-1          749886 750772     16  226    1 : tunables  120   60    0 : slabdata   3322   3322      0
bio               960630 960630     96   41    1 : tunables  120   60    0 : slabdata  23430  23430      0
biovec-1          960642 960726     16  226    1 : tunables  120   60    0 : slabdata   4251   4251      0
bio               1170079 1170345     96   41    1 : tunables  120   60    0 : slabdata  28543  28545      0
biovec-1          1170066 1170906     16  226    1 : tunables  120   60    0 : slabdata   5181   5181      0
bio               1379857 1380019     96   41    1 : tunables  120   60    0 : slabdata  33658  33659      0
biovec-1          1379854 1380408     16  226    1 : tunables  120   60    0 : slabdata   6108   6108      0

Clearly, something was going on. So I decided to run a vanilla kernel instead.

I'm running 2.6.12.1 right now, and after about one day of uptime:

bio                  345    369     96   41    1 : tunables  120   60    0 : slabdata      9      9      0
biovec-1             376    678     16  226    1 : tunables  120   60    0 : slabdata      3      3      0

which seem to me sane values (and they stay like that as far as I can see). No more daily increase of more than 200,000.

I'll keep an eye on it in the next days, but I think 2.6.12.1 is not affected.

.TM.

Signature · 5 lines
      ____/  ____/   /
     /      /       /                   Marco Colombo
    ___/  ___  /   /                  Technical Manager
   /          /   /                      ESI s.r.l.
 _____/ _____/  _/                      [email protected]
#2 lore raw
Quoted 6 lines · Marco Colombo

Thanks to everybody who replied to me. Here's more data:

sh-2.05b# sort -rn +1 /proc/slabinfo | head -5

 biovec-1          7502216 7502296     16  226    1 : tunables  120   60    0 : slabdata  33196  33196      0
 bio               7502216 7502262     96   41    1 : tunables  120   60    0 : slabdata 182982 182982      0

Are you using RAID?

From: Neil Brown <[email protected]>

insert a missing bio_put when writing the md superblock.

Without this we have a steady growth in the "bio" slab.

---

 drivers/md/md.c |    1 +
 1 files changed, 1 insertion(+)
drivers/md/md.c renamed from drivers/md/md.c~md-bio-leak-fix +1 -0
diff -puN drivers/md/md.c~md-bio-leak-fix drivers/md/md.c
@@ -338,6 +338,7 @@ static int super_written(struct bio *bio
 
 	if (atomic_dec_and_test(&rdev->mddev->pending_writes))
 		wake_up(&rdev->mddev->sb_wait);
+	bio_put(bio);
 	return 0;
 }
 

_

#3 lore raw
Quoted 9 lines · Andrew Morton
Quoted 6 lines · Marco Colombo

Thanks to everybody who replied to me. Here's more data:

sh-2.05b# sort -rn +1 /proc/slabinfo | head -5

 biovec-1          7502216 7502296     16  226    1 : tunables  120   60    0 : slabdata  33196  33196      0
 bio               7502216 7502262     96   41    1 : tunables  120   60    0 : slabdata 182982 182982      0

Are you using RAID?

Thanks Andrew,
yes I'm using RAID:

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [multipath]
md1 : active raid1 hdc7[1] hda7[0]
      524544 blocks [2/2] [UU]

md2 : active raid1 hdc8[1] hda8[0]

      524544 blocks [2/2] [UU]

md3 : active raid1 hdc9[1] hda9[0]

      3145856 blocks [2/2] [UU]

md4 : active raid1 hdc10[1] hda10[0]

      1048192 blocks [2/2] [UU]

md5 : active raid1 hdc11[1] hda11[0]

      2097024 blocks [2/2] [UU]

md6 : active raid1 hdc12[1] hda12[0]

      25961408 blocks [2/2] [UU]

md7 : active raid1 hdc13[1] hda13[0]

      25961408 blocks [2/2] [UU]

md0 : active raid1 hdc5[1] hda5[0]

      262400 blocks [2/2] [UU]

unused devices: <none>

This is a pretty standard configuration for me here, I use it almost everywhere as a mean to protect data from simple disk failures (that are way to common these days). I have at least 5-6 servers at hand, all with disk mirroring. Some of them match the version exactly, and the hardware is very similar, but there was only one host showing the leak. Two things are different on the affected server:
- it uses the dc395x driver (only a DAT device is attached);
- it's a database (PostgreSQL) server (that is, with some shm usage).

The dc395x sometimes spits some error messages:

Jun 28 05:37:47 xxxx kernel: dc395x: Interrupt from DMA engine: 0x63!
Jun 28 05:37:47 xxxx kernel: dc395x: Ignoring DMA error (probably a bad

thing) ...

I have to live with those for now. They seem to be harmless. The same messages occur with 2.6.12.1, but the bio leak is gone (or so it seems after 1 day of uptime), I guess they may be unrelated (the dc395x driver used to be my suspect #1).

Quoted 17 lines

From: Neil Brown <[email protected]>

insert a missing bio_put when writing the md superblock.

Without this we have a steady growth in the "bio" slab.

---

 drivers/md/md.c |    1 +
 1 files changed, 1 insertion(+)

diff -puN drivers/md/md.c~md-bio-leak-fix drivers/md/md.c

drivers/md/md.c static int super_written(struct bio *bio · line 344

Quoted 7 lines
drivers/md/md.c renamed from drivers/md/md.c~md-bio-leak-fix +1 -0
@@ -338,6 +338,7 @@ static int super_written(struct bio *bio
 
 	if (atomic_dec_and_test(&rdev->mddev->pending_writes))
 		wake_up(&rdev->mddev->sb_wait);
+	bio_put(bio);
 	return 0;
 }
 

I'm sorry, but I'm missing something: I'm not able to locate that piece of code in neither the (FC2) linux-2.6.10 tree nor the (unpatched) linux-2.6.12.1 tree. There's no super_written() function at all, BTW. The code looks completely different.

[...google digging...]

Is it related to this patch?

http://marc.theaimsgroup.com/?l=linux-raid&m=111155701929593&w=2

If so, I don't think I'm using it.
But the symptoms looks like the same.

Anyway, it seems solved in 2.6.12.1, so I'm not that worried. I'll wait and see if after the upgrade to FC4 the problem is still there (with the official FC4 kernel). I'll bring the issue to the Fedora lists, in case. As far as I can see, the problem is already solved in mainstream. Thanks again.

.TM.

Signature · 5 lines
      ____/  ____/   /
     /      /       /                   Marco Colombo
    ___/  ___  /   /                  Technical Manager
   /          /   /                      ESI s.r.l.
 _____/ _____/  _/                      [email protected]