Skip to contentSkip to conversation

2.6.0-test5/6 (and probably 7 too) size-4096 memory leak

DISCUSSION Alberto Bertogli 4 participants 8 messages Oct 16, 2003 – Oct 17, 2003 Memory management

lore
Share & tools
lore
#0 lore raw

Hi there!

I want to report a memory leak for 2.6.0-test5 that I've noticed today on a mail server after 32 days of uptime.

As I'm upgrading it tomorrow to test7 I wasn't going to report this until verifying if the behaviour continued, but I saw on kernelnewbies that others were having this issue with test7 too, so I decided to post a report with the information before I reboot the server.

The attached files are gzipped for space reasons, and were taken at night when the server isn't very loaded.

The workload is a simple sendmail with ipop3d and imapd, nothing much, for about 6500 users; the machine is a dual Pentium III with 1gb of RAM and a couple of SCSI disks.

Slabinfo reports that size-4096 has 104341 active objects and growing.

On another box at home I see the same issue with test6, but "only" with 11612 objects; I'm not posting info on this box as I guess the mailserver is much more important because the leak is really noticeable.

Please let me know if I can help with anything.

Thanks,
Alberto

#1 lore raw
Quoted 16 lines · Alberto Bertogli

I want to report a memory leak for 2.6.0-test5 that I've noticed today on
a mail server after 32 days of uptime.

As I'm upgrading it tomorrow to test7 I wasn't going to report this until verifying if the behaviour continued, but I saw on kernelnewbies that others were having this issue with test7 too, so I decided to post a report with the information before I reboot the server.

The attached files are gzipped for space reasons, and were taken at night when the server isn't very loaded.

The workload is a simple sendmail with ipop3d and imapd, nothing much, for about 6500 users; the machine is a dual Pentium III with 1gb of RAM and a couple of SCSI disks.

Quoted 5 lines

Slabinfo reports that size-4096 has 104341 active objects and growing.

On another box at home I see the same issue with test6, but "only" with 11612 objects; I'm not posting info on this box as I guess the mailserver is much more important because the leak is really noticeable.

At least I'm not the only one; my main desktop machine does the same thing. It leaks two megabytes a day into size-4096, like clockwork. It's up to 43 megs now.

I was ignoring it and hoping it would go away. Ho hum. Tricky.

#2 lore raw
Quoted 4 lines · Alberto Bertogli
Quoted 4 lines

Slabinfo reports that size-4096 has 104341 active objects and growing. On another box at home I see the same issue with test6, but "only" with 11612 objects; I'm not posting info on this box as I guess the mailserver is much more important because the leak is really noticeable.

Quoted 4 lines · Andrew Morton

At least I'm not the only one; my main desktop machine does the same thing. It leaks two megabytes a day into size-4096, like clockwork. It's up to 43 megs now.
I was ignoring it and hoping it would go away. Ho hum. Tricky.

I immediately thought of bundling this in with the do_exit() BUG() and /proc/ oopsen, but we would see a task_t leak also in that case. I still say the /proc/ change is swiss cheese (well, in concept there's nothing wrong with what it wants to do, but there's something definitely wrong with the implementation since backing it out stops things from oopsing), but this looks unrelated therefore (which is actually depressing, since we can't kill all three in one shot and/or get anywhere by correlating). I should try using a dedicated stack slab to see if they're stacks even though task_t's aren't leaking.

-- wli

#3 lore raw
Quoted 13 lines · William Lee Irwin III
Quoted 4 lines · Alberto Bertogli
Quoted 4 lines

Slabinfo reports that size-4096 has 104341 active objects and growing. On another box at home I see the same issue with test6, but "only" with 11612 objects; I'm not posting info on this box as I guess the mailserver is much more important because the leak is really noticeable.

Quoted 4 lines · Andrew Morton

At least I'm not the only one; my main desktop machine does the same thing. It leaks two megabytes a day into size-4096, like clockwork. It's up to 43 megs now.
I was ignoring it and hoping it would go away. Ho hum. Tricky.

Quoted 10 lines

I immediately thought of bundling this in with the do_exit() BUG() and /proc/ oopsen, but we would see a task_t leak also in that case. I still say the /proc/ change is swiss cheese (well, in concept there's nothing wrong with what it wants to do, but there's something definitely wrong with the implementation since backing it out stops things from oopsing), but this looks unrelated therefore (which is actually depressing, since we can't kill all three in one shot and/or get anywhere by correlating). I should try using a dedicated stack slab to see if they're stacks even though task_t's aren't leaking.

This leak is at least a couple of months old.

The recent (test7) /proc oops was fixed when Linus reverted the job-control-in-signal-struct patch.

This leak is of size-4096: it isn't kernel stacks.

I did a quicky audit of all kmalloc(PAGE_SIZE) instances and everything looked OK: one suspect in the NFS server but I wasn't able to force speedier bloat by exercising the NFS server in my normal usage pattern.

I'm thinking we need to stuff builtin_return_address(0) into the object and write a dumper, but I haven't looked into that. Maybe I can persuade Manfred to cook up a custom patch to do that? Just for size-4096? Something really crude will be fine.

#4 lore raw
Quoted 6 lines · Andrew Morton

I'm thinking we need to stuff builtin_return_address(0) into the object and write a dumper, but I haven't looked into that. Maybe I can persuade Manfred to cook up a custom patch to do that? Just for size-4096? Something really crude will be fine.

I've attached something: with the patch applied, echo "size-4096 0 0 0" > /proc/slabinfo dumps all caller addresses.

It works fine and dumps all 25 outstanding objects of my bochs setup - you might have to limit the dumps if you have 100k objects.

--
Manfred

mm/slab.c +28 -0
@@ -1891,6 +1891,15 @@
 		*dbg_redzone1(cachep, objp) = RED_ACTIVE;
 		*dbg_redzone2(cachep, objp) = RED_ACTIVE;
 	}
+	{
+		int objnr;
+		struct slab *slabp;
+
+		slabp = GET_PAGE_SLAB(virt_to_page(objp));
+
+		objnr = (objp - slabp->s_mem) / cachep->objsize;
+		slab_bufctl(slabp)[objnr] = (int)caller;
+	}
 	objp += obj_dbghead(cachep);
 	if (cachep->ctor && cachep->flags & SLAB_POISON) {
 		unsigned long	ctor_flags = SLAB_CTOR_CONSTRUCTOR;
@@ -1952,12 +1961,14 @@
 		objnr = (objp - slabp->s_mem) / cachep->objsize;
 		check_slabp(cachep, slabp);
 #if DEBUG
+#if 0
 		if (slab_bufctl(slabp)[objnr] != BUFCTL_FREE) {
 			printk(KERN_ERR "slab: double free detected in cache '%s', objp %p.\n",
 						cachep->name, objp);
 			BUG();
 		}
 #endif
+#endif
 		slab_bufctl(slabp)[objnr] = slabp->free;
 		slabp->free = objnr;
 		STATS_DEC_ACTIVE(cachep);
@@ -2694,6 +2705,22 @@
 	.show	= s_show,
 };
 
+static void do_dump_slabp(kmem_cache_t *cachep)
+{
+	struct list_head *q;
+
+	check_irq_on();
+	spin_lock_irq(&cachep->spinlock);
+	list_for_each(q,&cachep->lists.slabs_full) {
+		struct slab *slabp;
+		int i;
+		slabp = list_entry(q, struct slab, list);
+		for (i=0;i<cachep->num;i++)
+			printk(KERN_DEBUG "obj %p/%d: %p\n", slabp, i, (void*)(slab_bufctl(slabp)[i]));
+	}
+	spin_unlock_irq(&cachep->spinlock);
+}
+
 #define MAX_SLABINFO_WRITE 128
 /**
  * slabinfo_write - Tuning for the slab allocator
@@ -2734,6 +2761,7 @@
 			    batchcount < 1 ||
 			    batchcount > limit ||
 			    shared < 0) {
+				do_dump_slabp(cachep);
 				res = -EINVAL;
 			} else {
 				res = do_tune_cpucache(cachep, limit, batchcount, shared);
#5 lore raw
Quoted 3 lines · Manfred Spraul

I've attached something: with the patch applied, `echo "size-4096 0 0 0"

Quoted 1 line

/proc/slabinfo` dumps all caller addresses.

Awesome, thanks.

I added some tweaks (why was it returning -EINVAL?).

Is there any reason why we shouldn't merge this up?

 mm/slab.c |   17 +++++++++++++----
 1 files changed, 13 insertions(+), 4 deletions(-)
mm/slab.c renamed from mm/slab.c~slab-leak-detector-tweaks +13 -4
diff -puN mm/slab.c~slab-leak-detector-tweaks mm/slab.c
@@ -2708,6 +2708,7 @@ struct seq_operations slabinfo_op = {
 
 static void do_dump_slabp(kmem_cache_t *cachep)
 {
+#if DEBUG
 	struct list_head *q;
 
 	check_irq_on();
@@ -2716,10 +2717,17 @@ static void do_dump_slabp(kmem_cache_t *
 		struct slab *slabp;
 		int i;
 		slabp = list_entry(q, struct slab, list);
-		for (i=0;i<cachep->num;i++)
-			printk(KERN_DEBUG "obj %p/%d: %p\n", slabp, i, (void*)(slab_bufctl(slabp)[i]));
+		for (i = 0; i < cachep->num; i++) {
+			unsigned long sym = slab_bufctl(slabp)[i];
+
+			printk(KERN_DEBUG "obj %p/%d: %p",
+					slabp, i, (void *)sym);
+			print_symbol(" <%s>", sym);
+			printk("\n");
+		}
 	}
 	spin_unlock_irq(&cachep->spinlock);
+#endif
 }
 
 #define MAX_SLABINFO_WRITE 128
@@ -2763,9 +2771,10 @@ ssize_t slabinfo_write(struct file *file
 			    batchcount > limit ||
 			    shared < 0) {
 				do_dump_slabp(cachep);
-				res = -EINVAL;
+				res = 0;
 			} else {
-				res = do_tune_cpucache(cachep, limit, batchcount, shared);
+				res = do_tune_cpucache(cachep, limit,
+							batchcount, shared);
 			}
 			break;
 		}

_

#6 lore raw
Quoted 8 lines · Andrew Morton
Quoted 4 lines · Manfred Spraul

I've attached something: with the patch applied, `echo "size-4096 0 0 0"

Quoted 1 line

/proc/slabinfo` dumps all caller addresses.

Quoted 7 lines

Awesome, thanks.

I added some tweaks (why was it returning -EINVAL?).

Is there any reason why we shouldn't merge this up?

It works only on 32-archs, and I had to disable the double-free detection - the bufctl integers were already in use (the hunk with the #if 0) for that.

--
Manfred

#7 lore raw
In reply to Alberto Bertogli
Quoted 3 lines · Alberto Bertogli

I want to report a memory leak for 2.6.0-test5 that I've noticed today on
a mail server after 32 days of uptime.

'twas in ext3.

 fs/jbd/commit.c  |    8 ++++++++
 fs/jbd/journal.c |    2 ++
 2 files changed, 10 insertions(+)
fs/jbd/commit.c renamed from fs/jbd/commit.c~jbd-leak-fix +8 -0
diff -puN fs/jbd/commit.c~jbd-leak-fix fs/jbd/commit.c
@@ -172,6 +172,14 @@ void journal_commit_transaction(journal_
 	while (commit_transaction->t_reserved_list) {
 		jh = commit_transaction->t_reserved_list;
 		JBUFFER_TRACE(jh, "reserved, unused: refile");
+		/*
+		 * A journal_get_undo_access()+journal_release_buffer() may
+		 * leave undo-committed data.
+		 */
+		if (jh->b_committed_data) {
+			kfree(jh->b_committed_data);
+			jh->b_committed_data = NULL;
+		}
 		journal_refile_buffer(journal, jh);
 	}
 
fs/jbd/journal.c renamed from fs/jbd/journal.c~jbd-leak-fix +2 -0
diff -puN fs/jbd/journal.c~jbd-leak-fix fs/jbd/journal.c
@@ -1729,6 +1729,8 @@ static void __journal_remove_journal_hea
 			J_ASSERT_BH(bh, buffer_jbd(bh));
 			J_ASSERT_BH(bh, jh2bh(jh) == bh);
 			BUFFER_TRACE(bh, "remove journal_head");
+			J_ASSERT_BH(bh, !jh->b_frozen_data);
+			J_ASSERT_BH(bh, !jh->b_committed_data);
 			bh->b_private = NULL;
 			jh->b_bh = NULL;	/* debug, really */
 			clear_buffer_jbd(bh);

_