Size-128 slab leak
Subsystems (2)
Hi,
I have an annoying slab leak on my kernel. Every day, I lose about 50Megs of memory to the leak. It seems to be related to disk accesses, because the count only goes up noticeable around 4:00am when the system locate utility runs.
I can tell there is a leak because /proc/slabinfo shows "size-128" growing continuously. For example, it currently reads:
size-128 4086041 4106550 128 30 1 : tunables 120 60 8 : slabdata 136885 136885 0
The machine is a vanilla lkml kernel:
Linux double 2.6.15 #1 SMP Wed Jan 4 23:13:51 EST 2006 x86_64 x86_64 x86_64 GNU/Linux
I've noticed this bug on a 2.6.14 kernel also. This machine is using libata (sata_uli) along with reiserfs, ext3, and lvm. I'm interested in finding ways of diagnosing this problem. I can provide more information on demand. Please CC me on any replies.
Thanks,
-Kevin
00:00.0 Host bridge: ALi Corporation M1695 K8 Northbridge [PCI Express and HyperTransport] 00:01.0 PCI bridge: ALi Corporation: Unknown device 524b
00:02.0 PCI bridge: ALi Corporation: Unknown device 524c
00:04.0 Host bridge: ALi Corporation M1689 K8 Northbridge [Super K8 Single Chip]
00:05.0 PCI bridge: ALi Corporation AGP8X Controller
00:06.0 PCI bridge: ALi Corporation M5249 HTT to PCI Bridge
00:07.0 ISA bridge: ALi Corporation M1563 HyperTransport South Bridge (rev 70)
00:07.1 Bridge: ALi Corporation M7101 Power Management Controller [PMU]
00:08.0 Multimedia audio controller: ALi Corporation M5455 PCI AC-Link Controller Audio Device (rev 20) 00:11.0 Ethernet controller: ALi Corporation M5263 Ethernet Controller (rev 40)
00:12.0 IDE interface: ALi Corporation M5229 IDE (rev c7)
00:12.1 IDE interface: ALi Corporation ULi 5289 SATA (rev 10)
00:13.0 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.1 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.2 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.3 USB Controller: ALi Corporation USB 2.0 Controller (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 03:00.0 VGA compatible controller: ATI Technologies Inc RV280 [Radeon 9200 PRO] (rev 01) 03:00.1 Display controller: ATI Technologies Inc: Unknown device 5940 (rev 01)
processor : 0vendor_id : AuthenticAMD
cpu family : 15
model : 43
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
stepping : 1
cpu MHz : 1000.051
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fpu : yesfpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 2002.42
TLB size : 1024 4K pages
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp
processor : 1vendor_id : AuthenticAMD
cpu family : 15
model : 43
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
stepping : 1
cpu MHz : 1000.051
cache size : 512 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fpu : yesfpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 2002.42
TLB size : 1024 4K pages
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp
Module Size Used by cramfs 46144 1 loop 19472 2 ppp_deflate 8192 0 zlib_deflate 26016 1 ppp_deflate ppp_async 15488 0 crc_ccitt 3200 1 ppp_async ppp_generic 36384 2 ppp_deflate,ppp_async slhc 8960 1 ppp_generic vfat 17152 0 fat 60976 1 vfat usb_storage 86592 0 snd_rtctimer 4888 0 udf 92064 1 nls_utf8 3328 0 ipaq 40880 0 usbserial 38612 1 ipaq radeon 119584 1 drm 100776 2 radeon ipv6 311680 14 parport_pc 33900 0 lp 16960 0 parport 46604 2 parport_pc,lp autofs4 25608 1 w83627hf 35344 0 hwmon_vid 3712 1 w83627hf hwmon 4616 1 w83627hf eeprom 9744 0 i2c_isa 7552 1 w83627hf i2c_dev 14208 0 i2c_core 27904 4 w83627hf,eeprom,i2c_isa,i2c_dev sunrpc 184504 1 pcmcia 48816 0 yenta_socket 31628 0 rsrc_nonstatic 15872 1 yenta_socket pcmcia_core 50228 3 pcmcia,yenta_socket,rsrc_nonstatic reiserfs 285688 2 video 20488 0 button 8992 0 battery 12168 0 ac 6792 0 ohci_hcd 24708 0 ehci_hcd 38920 0 shpchp 53888 0 snd_intel8x0 39592 0 snd_ac97_codec 117180 1 snd_intel8x0 snd_ac97_bus 3840 1 snd_ac97_codec snd_seq_dummy 5380 0 snd_seq_oss 41700 0 snd_seq_midi_event 10368 1 snd_seq_oss snd_seq 70616 5 snd_seq_dummy,snd_seq_oss,snd_seq_midi_event snd_seq_device 12048 3 snd_seq_dummy,snd_seq_oss,snd_seq snd_pcm_oss 63264 0 snd_mixer_oss 21632 1 snd_pcm_oss snd_pcm 111624 3 snd_intel8x0,snd_ac97_codec,snd_pcm_oss snd_timer 30600 3 snd_rtctimer,snd_seq,snd_pcm snd 73696 9 snd_intel8x0,snd_ac97_codec,snd_seq_oss,snd_seq,snd_seq_device,snd_pcm_oss,snd_mixer_oss,snd_pcm,snd_timer soundcore 13088 1 snd snd_page_alloc 13840 2 snd_intel8x0,snd_pcm uli526x 21268 0 dm_snapshot 18768 0 dm_zero 2816 0 dm_mirror 25320 0 ext3 152848 2 jbd 68904 1 ext3 dm_mod 69064 8 dm_snapshot,dm_zero,dm_mirror sata_uli 8964 1 libata 67224 1 sata_uli sd_mod 21632 1 scsi_mod 166712 3 usb_storage,libata,sd_mod
Quoted 11 lines · Kevin O'Connor
I have an annoying slab leak on my kernel. Every day, I lose about
50Megs of memory to the leak. It seems to be related to disk accesses, because the count only goes up noticeable around 4:00am when the system locate utility runs.I can tell there is a leak because
/proc/slabinfoshows "size-128" growing continuously. For example, it currently reads:size-128 4086041 4106550 128 30 1 : tunables 120 60 8 : slabdata 136885 136885 0
Quoted 7 lines
The machine is a vanilla lkml kernel:
Linux double 2.6.15 #1 SMP Wed Jan 4 23:13:51 EST 2006
x86_64x86_64x86_64GNU/LinuxI've noticed this bug on a 2.6.14 kernel also. This machine is using libata (
sata_uli) along with reiserfs, ext3, and lvm. I'm interested in finding ways of diagnosing this problem.
-mm kernels have a patch (slab-leak-detector.patch) which will help. Here's a version for 2.6.16-rc1. It requires CONFIG_DEBUG_SLAB. Thanks.
From: Manfred Spraul <[email protected]>
Maintenance work from Alexander Nyberg <[email protected]>
With the patch applied,
echo "size-4096 0 0 0" > /proc/slabinfo
walks the objects in the size-4096 slab, printing out the calling address of whoever allocated that object.
It is for leak detection.
---
mm/slab.c | 46 +++++++++++++++++++++++++++++++++++++++++++--- 1 files changed, 43 insertions(+), 3 deletions(-)
mm/slab.c renamed from mm/slab.c~slab-leak-detector +43 -3
@@ -198,7 +198,7 @@* is less than 512 (PAGE_SIZE<<3), but greater than 256. */ -typedef unsigned int kmem_bufctl_t ; +typedef unsigned long kmem_bufctl_t ;#define BUFCTL_END (((kmem_bufctl_t)(~0U))-0)#define BUFCTL_FREE (((kmem_bufctl_t)(~0U))-1)#define SLAB_LIMIT (((kmem_bufctl_t)(~0U))-2)
@@ -2393,7 +2393,7 @@ static void check_slabp(kmem_cache_t *ca i <sizeof (slabp) + cachep->num *sizeof (kmem_bufctl_t); i++) {if ((i %16 ) ==0 ) -printk ("\n%03x:" , i); +printk ("\n%04lx:" , i);printk (" %02x" , ((unsigned char *)slabp)[i]); }printk ("\n" );
@@ -2550,6 +2550,15 @@ static void *cache_alloc_debugcheck_afte *dbg_redzone1 (cachep, objp) =RED_ACTIVE ; *dbg_redzone2 (cachep, objp) =RED_ACTIVE ; } + { +int objnr; +struct slab *slabp; + + slabp =page_get_slab (virt_to_page (objp)); + + objnr = (objp - slabp->s_mem) / cachep->objsize; +slab_bufctl (slabp)[objnr] = (unsigned long )caller; + } objp +=obj_dbghead (cachep);if (cachep->ctor && cachep->flags &SLAB_POISON ) {unsigned long ctor_flags =SLAB_CTOR_CONSTRUCTOR ;
@@ -2691,7 +2700,7 @@ static void free_block(kmem_cache_t *caccheck_spinlock_acquired_node (cachep, node);check_slabp (cachep, slabp); -#if DEBUG +#if 0 /* Verify that the slab belongs to the intended node */ WARN_ON (slabp->nodeid != node);
@@ -3573,6 +3582,36 @@ struct seq_operations slabinfo_op = { .show = s_show, }; +static void do_dump_slabp (kmem_cache_t *cachep) +{ +#if DEBUG +struct list_head *q; +int node; + +check_irq_on (); +spin_lock_irq (&cachep->spinlock); +for_each_online_node (node) { +struct kmem_list3 *rl3 = cachep->nodelists[node]; +spin_lock (&rl3->list_lock); + +list_for_each (q, &rl3->slabs_full) { +int i; + struct slab *slabp = list_entry(q, struct slab, list); + +for (i =0 ; i < cachep->num; i++) { +unsigned long sym =slab_bufctl (slabp)[i]; + +printk ("obj %p/%d: %p" , slabp, i, (void *)sym); +print_symbol (" <%s>" , sym); +printk ("\n" ); + } + } +spin_unlock (&rl3->list_lock); + } +spin_unlock_irq (&cachep->spinlock); +#endif +} +#define MAX_SLABINFO_WRITE 128/** * slabinfo_write - Tuning for the slab allocator
@@ -3612,6 +3651,7 @@ ssize_t slabinfo_write(struct file *fileif (limit <1 || batchcount <1 || batchcount > limit || shared <0 ) { +do_dump_slabp (cachep); res =0 ; }else { res = do_tune_cpucache(cachep, limit,
_
Hi,
mm/slab.c static void *cache_alloc_debugcheck_afte · line 2561
Quoted 13 lines · Andrew Morton
mm/slab.c renamed from mm/slab.c~slab-leak-detector +9
-0@@ -2550,6 +2550,15 @@ static void *cache_alloc_debugcheck_afte *dbg_redzone1 (cachep, objp) =RED_ACTIVE ; *dbg_redzone2 (cachep, objp) =RED_ACTIVE ; } + { +int objnr; +struct slab *slabp; + + slabp =page_get_slab (virt_to_page (objp)); + + objnr = (objp - slabp->s_mem) / cachep->objsize; +slab_bufctl (slabp)[objnr] = (unsigned long )caller; + }
We already have last caller in dbg_userword. Manfred, is there a reason we're not using it?
Pekka
Quoted 7 lines · Andrew Morton
Quoted 6 lines · Kevin O'Connor
I have an annoying slab leak on my kernel. Every day, I lose about
50Megs of memory to the leak. It seems to be related to disk accesses, because the count only goes up noticeable around 4:00am when the system locate utility runs.
Quoted 5 lines
Quoted 2 lines
I can tell there is a leak because
/proc/slabinfoshows "size-128" growing continuously. For example, it currently reads:-mm kernels have a patch (slab-leak-detector.patch) which will help. Here's a version for 2.6.16-rc1. It requires
CONFIG_DEBUG_SLAB. Thanks.
Thanks Andrew.
I've applied the patch and found the leak. It's in kzalloc. :-)
With kzalloc inlined, however, it appears that selinux is the likely culprit. I would not have expected that.
After running updatedb I got 23530 occurrences of:
kernel: obj ffff81003f04f000/12: ffffffff801ed7b7 <selinux_inode_alloc_security+0x37/0x100>
I'm not sure how to debug selinux issues, but at least I can disable it.
-Kevin
Quoted 5 lines · Pekka Enberg
We already have last caller in
dbg_userword. Manfred, is there a reason we're not using it?
IIRC only due to historical reasons: dbg_userword is newer than this patch.
--
Manfred
Quoted 7 lines · Kevin O'Connor
Thanks Andrew.
I've applied the patch and found the leak. It's in kzalloc. :-)
With kzalloc inlined, however, it appears that selinux is the likely culprit. I would not have expected that.
Quoted 6 lines
After running updatedb I got 23530 occurrences of:
kernel: obj ffff81003f04f000/12:
ffffffff801ed7b7<selinux_inode_alloc_security+0x37/0x100>I'm not sure how to debug selinux issues, but at least I can disable it.
Hmm...that allocation call occurs upon alloc_inode() via security_inode_alloc, and the associated free call occurs upon destroy_inode() via security_inode_free. However, when Jeff Mahoney introduced the support for "private inodes" (S_PRIVATE flag) to support reiserfs xattrs-as-files, he added the IS_PRIVATE guards to both security_inode_alloc and security_inode_free. I think that this ends up causing SELinux to allocate a security structure for every reiserfs inode including private inodes since they are not marked until later by reiserfs, while preventing SELinux from ever freeing the security structure for the private inodes. Note that selinux_inode_free_security() should be safe even for the private inodes, as it doesn't assume any other initialization beyond the allocation-time initialization. Patch below.
BTW, Jeff - I assume you know about the unrelated breakage in SELinux/reiserfs support that was introduced by the changes for atomic security labeling of inodes in 2.6.14. If you care, you might want to use the same workaround upstreamed for xfs for 2.6.16. But I understand that SELinux is not a priority in SuSE presently.
BTW, Chris - are you still serving as LSM maintainer? MAINTAINERS still lists your osdl address.
---
Remove private inode tests from security_inode_alloc and security_inode_free, as we otherwise end up leaking inode security structures for private inodes.
---
include/linux/security.h | 4 ---- 1 file changed, 4 deletions(-)
Index: linux-2.6/include/linux/security.h
===================================================================
RCS file: /nfshome/pal/CVS/linux-2.6/include/linux/security.h,v retrieving revision 1.50
include/linux/security.h +0 -4
@@ -1437,15 +1437,11 @@ static inline void security_sb_post_pivostatic inline int security_inode_alloc (struct inode *inode) { -if (unlikely (IS_PRIVATE (inode))) -return 0 ;return security_ops->inode_alloc_security (inode); }static inline void security_inode_free (struct inode *inode) { -if (unlikely (IS_PRIVATE (inode))) -return ; security_ops->inode_free_security (inode); }
Signature · 2 lines
Stephen Smalley National Security Agency
Quoted 1 line · Stephen Smalley
Signed-off-by Stephen Smalley <[email protected]>
Signature · 2 lines
James Morris <[email protected]>
Quoted 5 lines · Stephen Smalley
Quoted 4 lines · Kevin O'Connor
After running updatedb I got 23530 occurrences of:
kernel: obj ffff81003f04f000/12:
ffffffff801ed7b7<selinux_inode_alloc_security+0x37/0x100>
Quoted 13 lines
Hmm...that allocation call occurs upon
alloc_inode()viasecurity_inode_alloc, and the associated free call occurs upondestroy_inode()viasecurity_inode_free. However, when Jeff Mahoney introduced the support for "private inodes" (S_PRIVATEflag) to support reiserfs xattrs-as-files, he added theIS_PRIVATEguards to bothsecurity_inode_allocandsecurity_inode_free. I think that this ends up causing SELinux to allocate a security structure for every reiserfs inode including private inodes since they are not marked until later by reiserfs, while preventing SELinux from ever freeing the security structure for the private inodes. Note thatselinux_inode_free_security()should be safe even for the private inodes, as it doesn't assume any other initialization beyond the allocation-time initialization. Patch below.
Hi Stephen,
I've applied your patch. It seems to be working. (Multiple runs of updatedb no longer grow the size-128 slab.)
Thanks,
-Kevin
Quoted 5 lines · Stephen Smalley
BTW, Jeff - I assume you know about the unrelated breakage in SELinux/reiserfs support that was introduced by the changes for atomic security labeling of inodes in 2.6.14. If you care, you might want to use the same workaround upstreamed for xfs for 2.6.16. But I understand that SELinux is not a priority in SuSE presently.
Hi Stephen -
Thanks for the reminder on this one. I have a series of patches that rework the reiserfs xattr implementation. One of the patches adds journalling for xattrs, including putting the inherited ACLs in the same transaction as the object creation. Adding the SELinux attribute to the same transaction wouldn't be too much of a problem.
I should have a patch tomorrow, though it will depend on the xattr patchset.
-Jeff
--
Jeff Mahoney
SUSE Labs