2.6.0-test5/6 (and probably 7 too) size-4096 memory leak
Hi there!
I want to report a memory leak for 2.6.0-test5 that I've noticed today on a mail server after 32 days of uptime.
As I'm upgrading it tomorrow to test7 I wasn't going to report this until verifying if the behaviour continued, but I saw on kernelnewbies that others were having this issue with test7 too, so I decided to post a report with the information before I reboot the server.
The attached files are gzipped for space reasons, and were taken at night when the server isn't very loaded.
The workload is a simple sendmail with ipop3d and imapd, nothing much, for about 6500 users; the machine is a dual Pentium III with 1gb of RAM and a couple of SCSI disks.
Slabinfo reports that size-4096 has 104341 active objects and growing.
On another box at home I see the same issue with test6, but "only" with 11612 objects; I'm not posting info on this box as I guess the mailserver is much more important because the leak is really noticeable.
Please let me know if I can help with anything.
Thanks,
Alberto
Quoted 16 lines · Alberto Bertogli
I want to report a memory leak for 2.6.0-test5 that I've noticed today on
a mail server after 32 days of uptime.As I'm upgrading it tomorrow to test7 I wasn't going to report this until verifying if the behaviour continued, but I saw on kernelnewbies that others were having this issue with test7 too, so I decided to post a report with the information before I reboot the server.
The attached files are gzipped for space reasons, and were taken at night when the server isn't very loaded.
The workload is a simple sendmail with ipop3d and imapd, nothing much, for about 6500 users; the machine is a dual Pentium III with 1gb of RAM and a couple of SCSI disks.
Quoted 5 lines
Slabinfo reports that size-4096 has 104341 active objects and growing.
On another box at home I see the same issue with test6, but "only" with 11612 objects; I'm not posting info on this box as I guess the mailserver is much more important because the leak is really noticeable.
At least I'm not the only one; my main desktop machine does the same thing. It leaks two megabytes a day into size-4096, like clockwork. It's up to 43 megs now.
I was ignoring it and hoping it would go away. Ho hum. Tricky.
Quoted 4 lines · Alberto Bertogli
Quoted 4 lines
Slabinfo reports that size-4096 has 104341 active objects and growing. On another box at home I see the same issue with test6, but "only" with 11612 objects; I'm not posting info on this box as I guess the mailserver is much more important because the leak is really noticeable.
Quoted 4 lines · Andrew Morton
At least I'm not the only one; my main desktop machine does the same thing. It leaks two megabytes a day into size-4096, like clockwork. It's up to 43 megs now.
I was ignoring it and hoping it would go away. Ho hum. Tricky.
I immediately thought of bundling this in with the do_exit() BUG() and /proc/ oopsen, but we would see a task_t leak also in that case. I still say the /proc/ change is swiss cheese (well, in concept there's nothing wrong with what it wants to do, but there's something definitely wrong with the implementation since backing it out stops things from oopsing), but this looks unrelated therefore (which is actually depressing, since we can't kill all three in one shot and/or get anywhere by correlating). I should try using a dedicated stack slab to see if they're stacks even though task_t's aren't leaking.
-- wli
Quoted 13 lines · William Lee Irwin III
Quoted 4 lines · Alberto Bertogli
Quoted 4 lines
Slabinfo reports that size-4096 has 104341 active objects and growing. On another box at home I see the same issue with test6, but "only" with 11612 objects; I'm not posting info on this box as I guess the mailserver is much more important because the leak is really noticeable.
Quoted 4 lines · Andrew Morton
At least I'm not the only one; my main desktop machine does the same thing. It leaks two megabytes a day into size-4096, like clockwork. It's up to 43 megs now.
I was ignoring it and hoping it would go away. Ho hum. Tricky.
Quoted 10 lines
I immediately thought of bundling this in with the
do_exit()BUG()and/proc/oopsen, but we would see atask_tleak also in that case. I still say the/proc/change is swiss cheese (well, in concept there's nothing wrong with what it wants to do, but there's something definitely wrong with the implementation since backing it out stops things from oopsing), but this looks unrelated therefore (which is actually depressing, since we can't kill all three in one shot and/or get anywhere by correlating). I should try using a dedicated stack slab to see if they're stacks even though task_t's aren't leaking.
This leak is at least a couple of months old.
The recent (test7) /proc oops was fixed when Linus reverted the job-control-in-signal-struct patch.
This leak is of size-4096: it isn't kernel stacks.
I did a quicky audit of all kmalloc(PAGE_SIZE) instances and everything looked OK: one suspect in the NFS server but I wasn't able to force speedier bloat by exercising the NFS server in my normal usage pattern.
I'm thinking we need to stuff builtin_return_address(0) into the object and write a dumper, but I haven't looked into that. Maybe I can persuade Manfred to cook up a custom patch to do that? Just for size-4096? Something really crude will be fine.
Quoted 6 lines · Andrew Morton
I'm thinking we need to stuff
builtin_return_address(0)into the object and write a dumper, but I haven't looked into that. Maybe I can persuade Manfred to cook up a custom patch to do that? Just for size-4096? Something really crude will be fine.
I've attached something: with the patch applied, echo "size-4096 0 0 0" > /proc/slabinfo dumps all caller addresses.
It works fine and dumps all 25 outstanding objects of my bochs setup - you might have to limit the dumps if you have 100k objects.
--
Manfred
mm/slab.c +28 -0
@@ -1891,6 +1891,15 @@ *dbg_redzone1 (cachep, objp) =RED_ACTIVE ; *dbg_redzone2 (cachep, objp) =RED_ACTIVE ; } + { +int objnr; +struct slab *slabp; + + slabp =GET_PAGE_SLAB (virt_to_page (objp)); + + objnr = (objp - slabp->s_mem) / cachep->objsize; +slab_bufctl (slabp)[objnr] = (int )caller; + } objp +=obj_dbghead (cachep);if (cachep->ctor && cachep->flags &SLAB_POISON ) {unsigned long ctor_flags =SLAB_CTOR_CONSTRUCTOR ;
@@ -1952,12 +1961,14 @@ objnr = (objp - slabp->s_mem) / cachep->objsize;check_slabp (cachep, slabp);#if DEBUG +#if 0 if (slab_bufctl (slabp)[objnr] !=BUFCTL_FREE ) {printk (KERN_ERR "slab: double free detected in cache '%s', objp %p.\n" , cachep->name, objp);BUG (); }#endif +#endif slab_bufctl (slabp)[objnr] = slabp->free; slabp->free = objnr;STATS_DEC_ACTIVE (cachep);
@@ -2694,6 +2705,22 @@ .show = s_show, }; +static void do_dump_slabp (kmem_cache_t *cachep) +{ +struct list_head *q; + +check_irq_on (); +spin_lock_irq (&cachep->spinlock); +list_for_each (q,&cachep->lists.slabs_full) { +struct slab *slabp; +int i; + slabp = list_entry(q, struct slab, list); +for (i=0 ;i<cachep->num;i++) +printk (KERN_DEBUG "obj %p/%d: %p\n" , slabp, i, (void *)(slab_bufctl (slabp)[i])); + } +spin_unlock_irq (&cachep->spinlock); +} +#define MAX_SLABINFO_WRITE 128/** * slabinfo_write - Tuning for the slab allocator
@@ -2734,6 +2761,7 @@ batchcount < 1 || batchcount > limit || shared < 0) { + do_dump_slabp(cachep); res = -EINVAL; } else { res =do_tune_cpucache (cachep, limit, batchcount, shared);
Quoted 3 lines · Manfred Spraul
I've attached something: with the patch applied, `echo "size-4096 0 0 0"
Quoted 1 line
/proc/slabinfo` dumps all caller addresses.
Awesome, thanks.
I added some tweaks (why was it returning -EINVAL?).
Is there any reason why we shouldn't merge this up?
mm/slab.c | 17 +++++++++++++---- 1 files changed, 13 insertions(+), 4 deletions(-)
mm/slab.c renamed from mm/slab.c~slab-leak-detector-tweaks +13 -4
@@ -2708,6 +2708,7 @@ struct seq_operations slabinfo_op = {static void do_dump_slabp (kmem_cache_t *cachep) { +#if DEBUG struct list_head *q;check_irq_on ();
@@ -2716,10 +2717,17 @@ static void do_dump_slabp(kmem_cache_t *struct slab *slabp;int i; slabp = list_entry(q, struct slab, list); -for (i=0 ;i<cachep->num;i++) -printk (KERN_DEBUG "obj %p/%d: %p\n" , slabp, i, (void *)(slab_bufctl (slabp)[i])); +for (i =0 ; i < cachep->num; i++) { +unsigned long sym =slab_bufctl (slabp)[i]; + +printk (KERN_DEBUG "obj %p/%d: %p" , + slabp, i, (void *)sym); +print_symbol (" <%s>" , sym); +printk ("\n" ); + } }spin_unlock_irq (&cachep->spinlock); +#endif }#define MAX_SLABINFO_WRITE 128
@@ -2763,9 +2771,10 @@ ssize_t slabinfo_write(struct file *file batchcount > limit || shared < 0) { do_dump_slabp(cachep); - res = -EINVAL; + res = 0; } else { - res =do_tune_cpucache (cachep, limit, batchcount, shared); + res =do_tune_cpucache (cachep, limit, + batchcount, shared); }break ; }
_
Quoted 8 lines · Andrew Morton
Quoted 4 lines · Manfred Spraul
I've attached something: with the patch applied, `echo "size-4096 0 0 0"
Quoted 1 line
/proc/slabinfo` dumps all caller addresses.
Quoted 7 lines
Awesome, thanks.
I added some tweaks (why was it returning
-EINVAL?).Is there any reason why we shouldn't merge this up?
It works only on 32-archs, and I had to disable the double-free detection - the bufctl integers were already in use (the hunk with the #if 0) for that.
--
Manfred
Quoted 3 lines · Alberto Bertogli
I want to report a memory leak for 2.6.0-test5 that I've noticed today on
a mail server after 32 days of uptime.
'twas in ext3.
fs/jbd/commit.c | 8 ++++++++ fs/jbd/journal.c | 2 ++ 2 files changed, 10 insertions(+)
fs/jbd/commit.c renamed from fs/jbd/commit.c~jbd-leak-fix +8 -0
@@ -172,6 +172,14 @@ void journal_commit_transaction(journal_while (commit_transaction->t_reserved_list) { jh = commit_transaction->t_reserved_list;JBUFFER_TRACE (jh,"reserved, unused: refile" ); +/* +* A journal_get_undo_access()+journal_release_buffer() may +* leave undo-committed data. +*/ +if (jh->b_committed_data) { +kfree (jh->b_committed_data); + jh->b_committed_data =NULL ; + }journal_refile_buffer (journal, jh); }
fs/jbd/journal.c renamed from fs/jbd/journal.c~jbd-leak-fix +2 -0
@@ -1729,6 +1729,8 @@ static void __journal_remove_journal_heaJ_ASSERT_BH (bh,buffer_jbd (bh));J_ASSERT_BH (bh,jh2bh (jh) == bh);BUFFER_TRACE (bh,"remove journal_head" ); +J_ASSERT_BH (bh, !jh->b_frozen_data); +J_ASSERT_BH (bh, !jh->b_committed_data); bh->b_private =NULL ; jh->b_bh =NULL ;/* debug, really */ clear_buffer_jbd (bh);
_