Skip to contentSkip to conversation

How to diagnose a kernel memory leak

DISCUSSION Bruce Guenter 4 participants 10 messages May 9, 2005 – May 19, 2005

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

Greetings.

I am trying to diagnose a slow kernel memory leak, and am having no luck in pining it down.

I am currently running unpatched 2.6.12-rc3 (x86 on Gentoo, I saw the same symptoms with gentoo-sources 2.6.11-r6 and 2.6.11-r4. Over the course of several days, the server in question has the amount of available memory (free minus buffers+cache) gradually decrease. If I leave it go, it does eventually thrash itself to death after about a week (give or take). The rate is about 150MB per day (the system has 2GB of RAM total so it takes several days). The working set of processes remains the same through the whole period at between 50-150MB (depending on if you count VSZ or RSS). Nothing shows up in dmesg except for a couple of one-time lockd and nfs messages (the system uses two remote filesystems). The local filesystems are ReiserFS on a 3Ware 7500-4 controller, and the NIC is an Intel E100.

# free
             total       used       free     shared    buffers     cached
Mem:       2076180    2024068      52112          0     166760      93200
-/+ buffers/cache:    1764108     312072
Swap:      1028152         56    1028096
# cat /proc/meminfo
MemTotal:      2076180 kB
MemFree:         63080 kB
Buffers:        158776 kB
Cached:          91664 kB
SwapCached:          4 kB
Active:        1055244 kB
Inactive:       874660 kB
HighTotal:     1179072 kB
HighFree:          640 kB
LowTotal:       897108 kB
LowFree:         62440 kB
SwapTotal:     1028152 kB
SwapFree:      1028096 kB
Dirty:             768 kB
Writeback:           0 kB
Mapped:          12648 kB
Slab:            69872 kB
CommitLimit:   2066240 kB
Committed_AS:    26316 kB
PageTables:       1492 kB
VmallocTotal:   114680 kB
VmallocUsed:      4700 kB
VmallocChunk:   109784 kB

I would be happy to provide any additional information. As it stands, I have to reboot about once a week to clear the RAM or else it thrashes itself to death.

Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8
#1 lore raw
Quoted 5 lines · Bruce Guenter

Greetings.

I am trying to diagnose a slow kernel memory leak, and am having no luck in pining it down.

Quoted 12 lines

I am currently running unpatched 2.6.12-rc3 (x86 on Gentoo, I saw the same symptoms with gentoo-sources 2.6.11-r6 and 2.6.11-r4. Over the course of several days, the server in question has the amount of available memory (free minus buffers+cache) gradually decrease. If I leave it go, it does eventually thrash itself to death after about a week (give or take). The rate is about 150MB per day (the system has 2GB of RAM total so it takes several days). The working set of processes remains the same through the whole period at between 50-150MB (depending on if you count VSZ or RSS). Nothing shows up in dmesg except for a couple of one-time lockd and nfs messages (the system uses two remote filesystems). The local filesystems are ReiserFS on a 3Ware 7500-4 controller, and the NIC is an Intel E100.

Try looking at slabtop(1) output after a few days.

#2 lore raw
Quoted 1 line · Zwane Mwaikambo

Try looking at slabtop(1) output after a few days.

Well, this is interesting and all, and reiser_inode_cache is taking up a lot of memory, but according to the /proc/meminfo I had posted, slab is only using about 60MB. It doesn't appear to be the cause of any leaks.

Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8
#3 lore raw
In reply to Bruce Guenter
Quoted 15 lines

I am trying to diagnose a slow kernel memory leak, and am having no luck in pining it down.

I am currently running unpatched 2.6.12-rc3 (x86 on Gentoo, I saw the same symptoms with gentoo-sources 2.6.11-r6 and 2.6.11-r4. Over the course of several days, the server in question has the amount of available memory (free minus buffers+cache) gradually decrease. If I leave it go, it does eventually thrash itself to death after about a week (give or take). The rate is about 150MB per day (the system has 2GB of RAM total so it takes several days). The working set of processes remains the same through the whole period at between 50-150MB (depending on if you count VSZ or RSS). Nothing shows up in dmesg except for a couple of one-time lockd and nfs messages (the system uses two remote filesystems). The local filesystems are ReiserFS on a 3Ware 7500-4 controller, and the NIC is an Intel E100.

You should keep an eye on /proc/meminfo but if there is memory that is not accounted for then the patch below might help as it works on a lower level. It accounts for bare pages in the system available from /proc/page_owner. So a cat /proc/page_owner > tmpfile would be good when the system starts to go low. There's a sorting program in Documentation/page_owner.c used to sort the rather large output. Also the meminfo you posted, how long had the box been alive when you took it?

Select Track page owner under kernel hacking.

Documentation/page_owner.c new file +140 -0
Index: linux-2.6/Documentation/page_owner.c
===================================================================
@@ -0,0 +1,140 @@
+/*
+ * User-space helper to sort the output of /proc/page_owner
+ *
+ * Example use:
+ * cat /proc/page_owner > page_owner.txt
+ * ./sort page_owner.txt sorted_page_owner.txt
+*/
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <string.h>
+
+struct block_list {
+	char *txt;
+	int len;
+	int num;
+};
+
+
+static struct block_list *list;
+static int list_size;
+static int max_size;
+
+struct block_list *block_head;
+
+int read_block(char *buf, FILE *fin)
+{
+	int ret = 0;
+	int hit = 0;
+	char *curr = buf;
+
+	for (;;) {
+		*curr = getc(fin);
+		if (*curr == EOF) return -1;
+
+		ret++;
+		if (*curr == '\n' && hit == 1)
+			return ret - 1;
+		else if (*curr == '\n')
+			hit = 1;
+		else
+			hit = 0;
+		curr++;
+	}
+}
+
+static int compare_txt(struct block_list *l1, struct block_list *l2)
+{
+	return strcmp(l1->txt, l2->txt);
+}
+
+static int compare_num(struct block_list *l1, struct block_list *l2)
+{
+	return l2->num - l1->num;
+}
+
+static void add_list(char *buf, int len)
+{
+	if (list_size != 0 &&
+	    len == list[list_size-1].len &&
+	    memcmp(buf, list[list_size-1].txt, len) == 0) {
+		list[list_size-1].num++;
+		return;
+	}
+	if (list_size == max_size) {
+		printf("max_size too small??\n");
+		exit(1);
+	}
+	list[list_size].txt = malloc(len+1);
+	list[list_size].len = len;
+	list[list_size].num = 1;
+	memcpy(list[list_size].txt, buf, len);
+	list[list_size].txt[len] = 0;
+	list_size++;
+	if (list_size % 1000 == 0) {
+		printf("loaded %d\r", list_size);
+		fflush(stdout);
+	}
+}
+
+int main(int argc, char **argv)
+{
+	FILE *fin, *fout;
+	char buf[1024];
+	int ret, i, count;
+	struct block_list *list2;
+	struct stat st;
+
+	fin = fopen(argv[1], "r");
+	fout = fopen(argv[2], "w");
+	if (!fin || !fout) {
+		printf("Usage: ./program <input> <output>\n");
+		perror("open: ");
+		exit(2);
+	}
+
+	fstat(fileno(fin), &st);
+	max_size = st.st_size / 100; /* hack ... */
+
+	list = malloc(max_size * sizeof(*list));
+
+	for(;;) {
+		ret = read_block(buf, fin);
+		if (ret < 0)
+			break;
+
+		buf[ret] = '\0';
+		add_list(buf, ret);
+	}
+
+	printf("loaded %d\n", list_size);
+
+	printf("sorting ....\n");
+
+	qsort(list, list_size, sizeof(list[0]), compare_txt);
+
+	list2 = malloc(sizeof(*list) * list_size);
+
+	printf("culling\n");
+
+	for (i=count=0;i<list_size;i++) {
+		if (count == 0 ||
+		    strcmp(list2[count-1].txt, list[i].txt) != 0) {
+			list2[count++] = list[i];
+		} else {
+			list2[count-1].num += list[i].num;
+		}
+	}
+
+	qsort(list2, count, sizeof(list[0]), compare_num);
+
+	for (i=0;i<count;i++) {
+		fprintf(fout, "%d times:\n%s\n", list2[i].num, list2[i].txt);
+	}
+	return 0;
+}
fs/proc/proc_misc.c +68 -0

78 lines not on this page · raw message

include/linux/mm.h +5 -0
Index: linux-2.6/include/linux/mm.h
===================================================================
@@ -257,6 +257,11 @@
 	void *virtual;			/* Kernel virtual address (NULL if
 					   not kmapped, ie. highmem) */
 #endif /* WANT_PAGE_VIRTUAL */
+#ifdef CONFIG_PAGE_OWNER
+	int order;
+	unsigned int gfp_mask;
+	unsigned long trace[8];
+#endif
 };
 
 /*
lib/Kconfig.debug +10 -0
Index: linux-2.6/lib/Kconfig.debug
===================================================================
@@ -139,6 +139,16 @@
 	  automatically, but we'd like to make it more efficient by not
 	  having to do that.
 
+config PAGE_OWNER
+	bool "Track page owner"
+	depends on DEBUG_KERNEL && X86
+	help
+	  This keeps track of what call chain is the owner of a page, may
+	  help to find bare alloc_page(s) leaks. Eats a fair amount of memory.
+	  See Documentation/page_owner.c for user-space helper.
+
+	  If unsure, say N.
+
 config DEBUG_FS
 	bool "Debug Filesystem"
 	depends on DEBUG_KERNEL
mm/page_alloc.c +57 -0

81 lines not on this page · raw message

#4 lore raw
Quoted 5 lines · Alexander Nyberg

You should keep an eye on /proc/meminfo but if there is memory that is not accounted for then the patch below might help as it works on a lower level. It accounts for bare pages in the system available from /proc/page_owner.
Select Track page owner under kernel hacking.

I will try that the next time I have to reboot. As this is a server, I cannot arbitrarily take it down unfortunately.

Quoted 2 lines

Also the meminfo you posted, how long had the box been alive when you took it?

Almost exactly 2 days.

Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8
#5 lore raw
In reply to Alexander Nyberg
Quoted 5 lines · Alexander Nyberg

the patch below might help as it works on a lower
level. It accounts for bare pages in the system available from /proc/page_owner. So a cat /proc/page_owner > tmpfile would be good when the system starts to go low. There's a sorting program in Documentation/page_owner.c used to sort the rather large output.

I've been running this for a day and a half now, and a few hundred megs of memory is now missing:

# free
             total       used       free     shared    buffers     cached
Mem:       2055648    2001884      53764          0     259024     868484
-/+ buffers/cache:     874376    1181272
Swap:      1028152         56    1028096

I've put the output from the sorting program up at
http://untroubled.org/misc/page_owner_sorted

Is this useful information yet, or is there still too much in cached pages to really identify the source?

Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8
#6 lore raw

(Please always do reply-to-all)

Quoted 17 lines · Bruce Guenter
Quoted 5 lines · Alexander Nyberg

the patch below might help as it works on a lower
level. It accounts for bare pages in the system available from /proc/page_owner. So a cat /proc/page_owner > tmpfile would be good when the system starts to go low. There's a sorting program in Documentation/page_owner.c used to sort the rather large output.

I've been running this for a day and a half now, and a few hundred megs of memory is now missing:

# free
             total       used       free     shared    buffers     cached
Mem:       2055648    2001884      53764          0     259024     868484
-/+ buffers/cache:     874376    1181272
Swap:      1028152         56    1028096
Quoted 5 lines

I've put the output from the sorting program up at
http://untroubled.org/misc/page_owner_sorted

Is this useful information yet, or is there still too much in cached pages to really identify the source?

It all looks pretty innocent. Please send the contents of /proc/meminfo rather than the `free' output. /proc/meminfo has much more info. Sometimes /proc/vmstat is also useful.

If the /proc/meminfo output indicates that there are a lot of slab pages then /proc/slabinfo should be looked at.

#7 lore raw
Quoted 2 lines · Andrew Morton

It all looks pretty innocent. Please send the contents of /proc/meminfo rather than the `free' output. /proc/meminfo has much more info.

Here are the current meminfo numbers:

MemTotal:      2055648 kB
MemFree:         56512 kB
Buffers:        236880 kB
Cached:         869616 kB
SwapCached:          0 kB
Active:        1004124 kB
Inactive:       729732 kB
HighTotal:     1179072 kB
HighFree:         3584 kB
LowTotal:       876576 kB
LowFree:         52928 kB
SwapTotal:     1028152 kB
SwapFree:      1028096 kB
Dirty:            1036 kB
Writeback:           0 kB
Mapped:          13100 kB
Slab:           252444 kB
CommitLimit:   2055976 kB
Committed_AS:    25704 kB
PageTables:       1060 kB
VmallocTotal:   114680 kB
VmallocUsed:      4700 kB
VmallocChunk:   109836 kB

If I am counting right, free+buffers+cached+slab comes to 1415452 kB. Of course, at this point, it is far from out of memory like it has been in the past. I am continuing to monitor, and will post numbers when it gets closer to what I have previously observed.

Quoted 2 lines

If the /proc/meminfo output indicates that there are a lot of slab pages then /proc/slabinfo should be looked at.

That was my first thought, yes. However, when it has run out of memory, even the slab totals were low (my first post showed only about 60 MB in slab).

Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8
#8 lore raw

If you don't do reply-to-all there's a chance people will miss out on your mails...

Quoted 5 lines
Quoted 2 lines

It all looks pretty innocent. Please send the contents of /proc/meminfo rather than the `free' output. /proc/meminfo has much more info.

Here are the current meminfo numbers:

What's happening with this? It's been a week now so I'm curious.

What you can do is run the attached program, it's a simple memory eater that will eat the amount of memory you specify, ie. "./a.out 2000" will simply eat 2G of memory. This is because all caches get reaped to a minimum leavel and distinguishing trouble makes is easier this way.

If you think the machine has lost memory at this time please do:
gcc memeat.c
./a.out 2000
wait until program is done
save /proc/meminfo
save /proc/page_owner
sort page_owner output

#include <stdio.h>
#include <stdlib.h>

#define page_size 4096

int main(int argc, char *argv[])

{
	long size = strtoul(argv[1], NULL, 0);
	long curr = 0;

	printf("allocating %ldmb\n", size);
	
	size *= 1024 * 1024;

	while (curr <= size) {
		char *ptr = malloc(page_size);

		if (!ptr) {
			printf("Couldn't allocate after %ld\n", curr);
			sleep(1);
			continue;
		}
			
		memset(ptr, 0, page_size);
		curr += page_size;
	}

	return 0;
}
#9 lore raw
Quoted 6 lines · Alexander Nyberg
Quoted 4 lines
Quoted 2 lines

It all looks pretty innocent. Please send the contents of /proc/meminfo rather than the `free' output. /proc/meminfo has much more info.

Here are the current meminfo numbers:

What's happening with this? It's been a week now so I'm curious.

It appears the memory consumption I thought I was seeing is now gone, and only conclusively appeared with the Gentoo kernels. I will take this back to their bug tracker. Sorry for the false alarm.

Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8