Skip to contentSkip to conversation

Size-128 slab leak

DISCUSSION Kevin O'Connor 7 participants 9 messages Jan 31, 2006 – Feb 14, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I have an annoying slab leak on my kernel. Every day, I lose about 50Megs of memory to the leak. It seems to be related to disk accesses, because the count only goes up noticeable around 4:00am when the system locate utility runs.

I can tell there is a leak because /proc/slabinfo shows "size-128" growing continuously. For example, it currently reads:

size-128 4086041 4106550 128 30 1 : tunables 120 60 8 : slabdata 136885 136885 0

The machine is a vanilla lkml kernel:

Linux double 2.6.15 #1 SMP Wed Jan 4 23:13:51 EST 2006 x86_64 x86_64 x86_64 GNU/Linux

I've noticed this bug on a 2.6.14 kernel also. This machine is using libata (sata_uli) along with reiserfs, ext3, and lvm. I'm interested in finding ways of diagnosing this problem. I can provide more information on demand. Please CC me on any replies.

Thanks,
-Kevin

00:00.0 Host bridge: ALi Corporation M1695 K8 Northbridge [PCI Express and HyperTransport] 00:01.0 PCI bridge: ALi Corporation: Unknown device 524b
00:02.0 PCI bridge: ALi Corporation: Unknown device 524c
00:04.0 Host bridge: ALi Corporation M1689 K8 Northbridge [Super K8 Single Chip]
00:05.0 PCI bridge: ALi Corporation AGP8X Controller
00:06.0 PCI bridge: ALi Corporation M5249 HTT to PCI Bridge
00:07.0 ISA bridge: ALi Corporation M1563 HyperTransport South Bridge (rev 70)
00:07.1 Bridge: ALi Corporation M7101 Power Management Controller [PMU]
00:08.0 Multimedia audio controller: ALi Corporation M5455 PCI AC-Link Controller Audio Device (rev 20) 00:11.0 Ethernet controller: ALi Corporation M5263 Ethernet Controller (rev 40)
00:12.0 IDE interface: ALi Corporation M5229 IDE (rev c7)
00:12.1 IDE interface: ALi Corporation ULi 5289 SATA (rev 10)
00:13.0 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.1 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.2 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.3 USB Controller: ALi Corporation USB 2.0 Controller (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 03:00.0 VGA compatible controller: ATI Technologies Inc RV280 [Radeon 9200 PRO] (rev 01) 03:00.1 Display controller: ATI Technologies Inc: Unknown device 5940 (rev 01)

processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 43
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
stepping : 1
cpu MHz : 1000.051
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 2002.42
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 43
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
stepping : 1
cpu MHz : 1000.051
cache size : 512 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 2002.42
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

Module                  Size  Used by
cramfs                 46144  1 
loop                   19472  2 
ppp_deflate             8192  0 
zlib_deflate           26016  1 ppp_deflate
ppp_async              15488  0 
crc_ccitt               3200  1 ppp_async
ppp_generic            36384  2 ppp_deflate,ppp_async
slhc                    8960  1 ppp_generic
vfat                   17152  0 
fat                    60976  1 vfat
usb_storage            86592  0 
snd_rtctimer            4888  0 
udf                    92064  1 
nls_utf8                3328  0 
ipaq                   40880  0 
usbserial              38612  1 ipaq
radeon                119584  1 
drm                   100776  2 radeon
ipv6                  311680  14 
parport_pc             33900  0 
lp                     16960  0 
parport                46604  2 parport_pc,lp
autofs4                25608  1 
w83627hf               35344  0 
hwmon_vid               3712  1 w83627hf
hwmon                   4616  1 w83627hf
eeprom                  9744  0 
i2c_isa                 7552  1 w83627hf
i2c_dev                14208  0 
i2c_core               27904  4 w83627hf,eeprom,i2c_isa,i2c_dev
sunrpc                184504  1 
pcmcia                 48816  0 
yenta_socket           31628  0 
rsrc_nonstatic         15872  1 yenta_socket
pcmcia_core            50228  3 pcmcia,yenta_socket,rsrc_nonstatic
reiserfs              285688  2 
video                  20488  0 
button                  8992  0 
battery                12168  0 
ac                      6792  0 
ohci_hcd               24708  0 
ehci_hcd               38920  0 
shpchp                 53888  0 
snd_intel8x0           39592  0 
snd_ac97_codec        117180  1 snd_intel8x0
snd_ac97_bus            3840  1 snd_ac97_codec
snd_seq_dummy           5380  0 
snd_seq_oss            41700  0 
snd_seq_midi_event     10368  1 snd_seq_oss
snd_seq                70616  5 snd_seq_dummy,snd_seq_oss,snd_seq_midi_event
snd_seq_device         12048  3 snd_seq_dummy,snd_seq_oss,snd_seq
snd_pcm_oss            63264  0 
snd_mixer_oss          21632  1 snd_pcm_oss
snd_pcm               111624  3 snd_intel8x0,snd_ac97_codec,snd_pcm_oss
snd_timer              30600  3 snd_rtctimer,snd_seq,snd_pcm
snd                    73696  9 snd_intel8x0,snd_ac97_codec,snd_seq_oss,snd_seq,snd_seq_device,snd_pcm_oss,snd_mixer_oss,snd_pcm,snd_timer
soundcore              13088  1 snd
snd_page_alloc         13840  2 snd_intel8x0,snd_pcm
uli526x                21268  0 
dm_snapshot            18768  0 
dm_zero                 2816  0 
dm_mirror              25320  0 
ext3                  152848  2 
jbd                    68904  1 ext3
dm_mod                 69064  8 dm_snapshot,dm_zero,dm_mirror
sata_uli                8964  1 
libata                 67224  1 sata_uli
sd_mod                 21632  1 
scsi_mod              166712  3 usb_storage,libata,sd_mod
#1 lore raw
Quoted 11 lines · Kevin O'Connor

I have an annoying slab leak on my kernel. Every day, I lose about
50Megs of memory to the leak. It seems to be related to disk accesses, because the count only goes up noticeable around 4:00am when the system locate utility runs.

I can tell there is a leak because /proc/slabinfo shows "size-128" growing continuously. For example, it currently reads:

size-128 4086041 4106550 128 30 1 : tunables 120 60 8 : slabdata 136885 136885 0

Quoted 7 lines

The machine is a vanilla lkml kernel:

Linux double 2.6.15 #1 SMP Wed Jan 4 23:13:51 EST 2006 x86_64 x86_64 x86_64 GNU/Linux

I've noticed this bug on a 2.6.14 kernel also. This machine is using libata (sata_uli) along with reiserfs, ext3, and lvm. I'm interested in finding ways of diagnosing this problem.

-mm kernels have a patch (slab-leak-detector.patch) which will help. Here's a version for 2.6.16-rc1. It requires CONFIG_DEBUG_SLAB. Thanks.

From: Manfred Spraul <[email protected]>

Maintenance work from Alexander Nyberg <[email protected]>

With the patch applied,

	echo "size-4096 0 0 0" > /proc/slabinfo

walks the objects in the size-4096 slab, printing out the calling address of whoever allocated that object.

It is for leak detection.

---

 mm/slab.c |   46 +++++++++++++++++++++++++++++++++++++++++++---
 1 files changed, 43 insertions(+), 3 deletions(-)
mm/slab.c renamed from mm/slab.c~slab-leak-detector +43 -3
diff -puN mm/slab.c~slab-leak-detector mm/slab.c
@@ -198,7 +198,7 @@
  * is less than 512 (PAGE_SIZE<<3), but greater than 256.
  */
 
-typedef unsigned int kmem_bufctl_t;
+typedef unsigned long kmem_bufctl_t;
 #define BUFCTL_END	(((kmem_bufctl_t)(~0U))-0)
 #define BUFCTL_FREE	(((kmem_bufctl_t)(~0U))-1)
 #define	SLAB_LIMIT	(((kmem_bufctl_t)(~0U))-2)
@@ -2393,7 +2393,7 @@ static void check_slabp(kmem_cache_t *ca
 		     i < sizeof(slabp) + cachep->num * sizeof(kmem_bufctl_t);
 		     i++) {
 			if ((i % 16) == 0)
-				printk("\n%03x:", i);
+				printk("\n%04lx:", i);
 			printk(" %02x", ((unsigned char *)slabp)[i]);
 		}
 		printk("\n");
@@ -2550,6 +2550,15 @@ static void *cache_alloc_debugcheck_afte
 		*dbg_redzone1(cachep, objp) = RED_ACTIVE;
 		*dbg_redzone2(cachep, objp) = RED_ACTIVE;
 	}
+	{
+		int objnr;
+		struct slab *slabp;
+
+		slabp = page_get_slab(virt_to_page(objp));
+
+		objnr = (objp - slabp->s_mem) / cachep->objsize;
+		slab_bufctl(slabp)[objnr] = (unsigned long)caller;
+	}
 	objp += obj_dbghead(cachep);
 	if (cachep->ctor && cachep->flags & SLAB_POISON) {
 		unsigned long ctor_flags = SLAB_CTOR_CONSTRUCTOR;
@@ -2691,7 +2700,7 @@ static void free_block(kmem_cache_t *cac
 		check_spinlock_acquired_node(cachep, node);
 		check_slabp(cachep, slabp);
 
-#if DEBUG
+#if 0
 		/* Verify that the slab belongs to the intended node */
 		WARN_ON(slabp->nodeid != node);
 
@@ -3573,6 +3582,36 @@ struct seq_operations slabinfo_op = {
 	.show = s_show,
 };
 
+static void do_dump_slabp(kmem_cache_t *cachep)
+{
+#if DEBUG
+	struct list_head *q;
+	int node;
+
+	check_irq_on();
+	spin_lock_irq(&cachep->spinlock);
+	for_each_online_node(node) {
+		struct kmem_list3 *rl3 = cachep->nodelists[node];
+		spin_lock(&rl3->list_lock);
+
+		list_for_each(q, &rl3->slabs_full) {
+			int i;
+			struct slab *slabp = list_entry(q, struct slab, list);
+
+			for (i = 0; i < cachep->num; i++) {
+				unsigned long sym = slab_bufctl(slabp)[i];
+
+				printk("obj %p/%d: %p", slabp, i, (void *)sym);
+				print_symbol(" <%s>", sym);
+				printk("\n");
+			}
+		}
+		spin_unlock(&rl3->list_lock);
+	}
+	spin_unlock_irq(&cachep->spinlock);
+#endif
+}
+
 #define MAX_SLABINFO_WRITE 128
 /**
  * slabinfo_write - Tuning for the slab allocator
@@ -3612,6 +3651,7 @@ ssize_t slabinfo_write(struct file *file
 			if (limit < 1 ||
 			    batchcount < 1 ||
 			    batchcount > limit || shared < 0) {
+				do_dump_slabp(cachep);
 				res = 0;
 			} else {
 				res = do_tune_cpucache(cachep, limit,

_

#2 lore raw

Hi,

mm/slab.c static void *cache_alloc_debugcheck_afte · line 2561

Quoted 13 lines · Andrew Morton
mm/slab.c renamed from mm/slab.c~slab-leak-detector +9 -0
@@ -2550,6 +2550,15 @@ static void *cache_alloc_debugcheck_afte
                *dbg_redzone1(cachep, objp) = RED_ACTIVE;
                *dbg_redzone2(cachep, objp) = RED_ACTIVE;
        }
+       {
+               int objnr;
+               struct slab *slabp;
+
+               slabp = page_get_slab(virt_to_page(objp));
+
+               objnr = (objp - slabp->s_mem) / cachep->objsize;
+               slab_bufctl(slabp)[objnr] = (unsigned long)caller;
+       }

We already have last caller in dbg_userword. Manfred, is there a reason we're not using it?

Pekka

#3 lore raw
Quoted 5 lines · Pekka Enberg

We already have last caller in dbg_userword. Manfred, is there a reason we're not using it?

IIRC only due to historical reasons: dbg_userword is newer than this patch.

--
Manfred

#4 lore raw
In reply to Andrew Morton
Quoted 7 lines · Andrew Morton
Quoted 6 lines · Kevin O'Connor

I have an annoying slab leak on my kernel. Every day, I lose about
50Megs of memory to the leak. It seems to be related to disk accesses, because the count only goes up noticeable around 4:00am when the system locate utility runs.

Quoted 5 lines
Quoted 2 lines

I can tell there is a leak because /proc/slabinfo shows "size-128" growing continuously. For example, it currently reads:

-mm kernels have a patch (slab-leak-detector.patch) which will help. Here's a version for 2.6.16-rc1. It requires CONFIG_DEBUG_SLAB. Thanks.

Thanks Andrew.

I've applied the patch and found the leak. It's in kzalloc. :-)

With kzalloc inlined, however, it appears that selinux is the likely culprit. I would not have expected that.

After running updatedb I got 23530 occurrences of:

kernel: obj ffff81003f04f000/12: ffffffff801ed7b7 <selinux_inode_alloc_security+0x37/0x100>

I'm not sure how to debug selinux issues, but at least I can disable it.

-Kevin

#5 lore raw
Quoted 7 lines · Kevin O'Connor

Thanks Andrew.

I've applied the patch and found the leak. It's in kzalloc. :-)

With kzalloc inlined, however, it appears that selinux is the likely culprit. I would not have expected that.

Quoted 6 lines

After running updatedb I got 23530 occurrences of:

kernel: obj ffff81003f04f000/12: ffffffff801ed7b7 <selinux_inode_alloc_security+0x37/0x100>

I'm not sure how to debug selinux issues, but at least I can disable it.

Hmm...that allocation call occurs upon alloc_inode() via security_inode_alloc, and the associated free call occurs upon destroy_inode() via security_inode_free. However, when Jeff Mahoney introduced the support for "private inodes" (S_PRIVATE flag) to support reiserfs xattrs-as-files, he added the IS_PRIVATE guards to both security_inode_alloc and security_inode_free. I think that this ends up causing SELinux to allocate a security structure for every reiserfs inode including private inodes since they are not marked until later by reiserfs, while preventing SELinux from ever freeing the security structure for the private inodes. Note that selinux_inode_free_security() should be safe even for the private inodes, as it doesn't assume any other initialization beyond the allocation-time initialization. Patch below.

BTW, Jeff - I assume you know about the unrelated breakage in SELinux/reiserfs support that was introduced by the changes for atomic security labeling of inodes in 2.6.14. If you care, you might want to use the same workaround upstreamed for xfs for 2.6.16. But I understand that SELinux is not a priority in SuSE presently.

BTW, Chris - are you still serving as LSM maintainer? MAINTAINERS still lists your osdl address.

---

Remove private inode tests from security_inode_alloc and security_inode_free, as we otherwise end up leaking inode security structures for private inodes.

---

 include/linux/security.h |    4 ----
 1 file changed, 4 deletions(-)

Index: linux-2.6/include/linux/security.h
===================================================================
RCS file: /nfshome/pal/CVS/linux-2.6/include/linux/security.h,v retrieving revision 1.50

include/linux/security.h +0 -4
diff -u -p -r1.50 security.h
@@ -1437,15 +1437,11 @@ static inline void security_sb_post_pivo
 
 static inline int security_inode_alloc (struct inode *inode)
 {
-	if (unlikely (IS_PRIVATE (inode)))
-		return 0;
 	return security_ops->inode_alloc_security (inode);
 }
 
 static inline void security_inode_free (struct inode *inode)
 {
-	if (unlikely (IS_PRIVATE (inode)))
-		return;
 	security_ops->inode_free_security (inode);
 }
 
Signature · 2 lines
Stephen Smalley
National Security Agency
#7 lore raw
In reply to Stephen Smalley
Quoted 5 lines · Stephen Smalley
Quoted 4 lines · Kevin O'Connor

After running updatedb I got 23530 occurrences of:

kernel: obj ffff81003f04f000/12: ffffffff801ed7b7 <selinux_inode_alloc_security+0x37/0x100>

Quoted 13 lines

Hmm...that allocation call occurs upon alloc_inode() via security_inode_alloc, and the associated free call occurs upon destroy_inode() via security_inode_free. However, when Jeff Mahoney introduced the support for "private inodes" (S_PRIVATE flag) to support reiserfs xattrs-as-files, he added the IS_PRIVATE guards to both security_inode_alloc and security_inode_free. I think that this ends up causing SELinux to allocate a security structure for every reiserfs inode including private inodes since they are not marked until later by reiserfs, while preventing SELinux from ever freeing the security structure for the private inodes. Note that selinux_inode_free_security() should be safe even for the private inodes, as it doesn't assume any other initialization beyond the allocation-time initialization. Patch below.

Hi Stephen,

I've applied your patch. It seems to be working. (Multiple runs of updatedb no longer grow the size-128 slab.)

Thanks,
-Kevin

#8 lore raw
In reply to Stephen Smalley
Quoted 5 lines · Stephen Smalley

BTW, Jeff - I assume you know about the unrelated breakage in SELinux/reiserfs support that was introduced by the changes for atomic security labeling of inodes in 2.6.14. If you care, you might want to use the same workaround upstreamed for xfs for 2.6.16. But I understand that SELinux is not a priority in SuSE presently.

Hi Stephen -

Thanks for the reminder on this one. I have a series of patches that rework the reiserfs xattr implementation. One of the patches adds journalling for xattrs, including putting the inherited ACLs in the same transaction as the object creation. Adding the SELinux attribute to the same transaction wouldn't be too much of a problem.

I should have a patch tomorrow, though it will depend on the xattr patchset.

-Jeff

--
Jeff Mahoney
SUSE Labs