How to diagnose a kernel memory leak
Subsystems (6)
Greetings.
I am trying to diagnose a slow kernel memory leak, and am having no luck in pining it down.
I am currently running unpatched 2.6.12-rc3 (x86 on Gentoo, I saw the same symptoms with gentoo-sources 2.6.11-r6 and 2.6.11-r4. Over the course of several days, the server in question has the amount of available memory (free minus buffers+cache) gradually decrease. If I leave it go, it does eventually thrash itself to death after about a week (give or take). The rate is about 150MB per day (the system has 2GB of RAM total so it takes several days). The working set of processes remains the same through the whole period at between 50-150MB (depending on if you count VSZ or RSS). Nothing shows up in dmesg except for a couple of one-time lockd and nfs messages (the system uses two remote filesystems). The local filesystems are ReiserFS on a 3Ware 7500-4 controller, and the NIC is an Intel E100.
# free
total used free shared buffers cached
Mem: 2076180 2024068 52112 0 166760 93200
-/+ buffers/cache: 1764108 312072
Swap: 1028152 56 1028096# cat /proc/meminfo MemTotal: 2076180 kB MemFree: 63080 kB Buffers: 158776 kB Cached: 91664 kB SwapCached: 4 kB Active: 1055244 kB Inactive: 874660 kB HighTotal: 1179072 kB HighFree: 640 kB LowTotal: 897108 kB LowFree: 62440 kB SwapTotal: 1028152 kB SwapFree: 1028096 kB Dirty: 768 kB Writeback: 0 kB Mapped: 12648 kB Slab: 69872 kB CommitLimit: 2066240 kB Committed_AS: 26316 kB PageTables: 1492 kB VmallocTotal: 114680 kB VmallocUsed: 4700 kB VmallocChunk: 109784 kB
I would be happy to provide any additional information. As it stands, I have to reboot about once a week to clear the RAM or else it thrashes itself to death.
Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/ OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA 2E2A E96F B2DC 6999 80E8
Quoted 5 lines · Bruce Guenter
Greetings.
I am trying to diagnose a slow kernel memory leak, and am having no luck in pining it down.
Quoted 12 lines
I am currently running unpatched 2.6.12-rc3 (x86 on Gentoo, I saw the same symptoms with gentoo-sources 2.6.11-r6 and 2.6.11-r4. Over the course of several days, the server in question has the amount of available memory (free minus buffers+cache) gradually decrease. If I leave it go, it does eventually thrash itself to death after about a week (give or take). The rate is about 150MB per day (the system has 2GB of RAM total so it takes several days). The working set of processes remains the same through the whole period at between 50-150MB (depending on if you count VSZ or RSS). Nothing shows up in dmesg except for a couple of one-time lockd and nfs messages (the system uses two remote filesystems). The local filesystems are ReiserFS on a 3Ware 7500-4 controller, and the NIC is an Intel E100.
Try looking at slabtop(1) output after a few days.
Quoted 15 lines
I am trying to diagnose a slow kernel memory leak, and am having no luck in pining it down.
I am currently running unpatched 2.6.12-rc3 (x86 on Gentoo, I saw the same symptoms with gentoo-sources 2.6.11-r6 and 2.6.11-r4. Over the course of several days, the server in question has the amount of available memory (free minus buffers+cache) gradually decrease. If I leave it go, it does eventually thrash itself to death after about a week (give or take). The rate is about 150MB per day (the system has 2GB of RAM total so it takes several days). The working set of processes remains the same through the whole period at between 50-150MB (depending on if you count VSZ or RSS). Nothing shows up in dmesg except for a couple of one-time lockd and nfs messages (the system uses two remote filesystems). The local filesystems are ReiserFS on a 3Ware 7500-4 controller, and the NIC is an Intel E100.
You should keep an eye on /proc/meminfo but if there is memory that is not accounted for then the patch below might help as it works on a lower level. It accounts for bare pages in the system available from /proc/page_owner. So a cat /proc/page_owner > tmpfile would be good when the system starts to go low. There's a sorting program in Documentation/page_owner.c used to sort the rather large output. Also the meminfo you posted, how long had the box been alive when you took it?
Select Track page owner under kernel hacking.
Documentation/page_owner.c new file +140 -0
@@ -0,0 +1,140 @@ +/* +* User-space helper to sort the output of /proc/page_owner +* +* Example use: +* cat /proc/page_owner > page_owner.txt +* ./sort page_owner.txt sorted_page_owner.txt +*/ + +#include <stdio.h> +#include <stdlib.h> +#include <sys/types.h> +#include <sys/stat.h> +#include <fcntl.h> +#include <unistd.h> +#include <string.h> + +struct block_list { +char *txt; +int len; +int num; +}; + + +static struct block_list *list; +static int list_size; +static int max_size; + +struct block_list *block_head; + +int read_block (char *buf,FILE *fin) +{ +int ret =0 ; +int hit =0 ; +char *curr = buf; + +for (;;) { + *curr =getc (fin); +if (*curr ==EOF )return -1 ; + + ret++; +if (*curr =='\n' && hit ==1 ) +return ret -1 ; +else if (*curr =='\n' ) + hit =1 ; +else + hit =0 ; + curr++; + } +} + +static int compare_txt (struct block_list *l1,struct block_list *l2) +{ +return strcmp (l1->txt, l2->txt); +} + +static int compare_num (struct block_list *l1,struct block_list *l2) +{ +return l2->num - l1->num; +} + +static void add_list (char *buf,int len) +{ +if (list_size !=0 && + len == list[list_size-1 ].len && +memcmp (buf, list[list_size-1 ].txt, len) ==0 ) { + list[list_size-1 ].num++; +return ; + } +if (list_size == max_size) { +printf ("max_size too small??\n" ); +exit (1 ); + } + list[list_size].txt =malloc (len+1 ); + list[list_size].len = len; + list[list_size].num =1 ; +memcpy (list[list_size].txt, buf, len); + list[list_size].txt[len] =0 ; + list_size++; +if (list_size %1000 ==0 ) { +printf ("loaded %d\r" , list_size); +fflush (stdout); + } +} + +int main (int argc,char **argv) +{ +FILE *fin, *fout; +char buf[1024 ]; +int ret, i, count; +struct block_list *list2; +struct stat st; + + fin =fopen (argv[1 ],"r" ); + fout =fopen (argv[2 ],"w" ); +if (!fin || !fout) { +printf ("Usage: ./program <input> <output>\n" ); +perror ("open: " ); +exit (2 ); + } + +fstat (fileno (fin), &st); + max_size = st.st_size /100 ;/* hack ... */ + + list =malloc (max_size *sizeof (*list)); + +for (;;) { + ret =read_block (buf, fin); +if (ret <0 ) +break ; + + buf[ret] ='\0' ; +add_list (buf, ret); + } + +printf ("loaded %d\n" , list_size); + +printf ("sorting ....\n" ); + +qsort (list, list_size,sizeof (list[0 ]), compare_txt); + + list2 =malloc (sizeof (*list) * list_size); + +printf ("culling\n" ); + +for (i=count=0 ;i<list_size;i++) { +if (count ==0 || +strcmp (list2[count-1 ].txt, list[i].txt) !=0 ) { + list2[count++] = list[i]; + }else { + list2[count-1 ].num += list[i].num; + } + } + +qsort (list2, count,sizeof (list[0 ]), compare_num); + +for (i=0 ;i<count;i++) { +fprintf (fout,"%d times:\n%s\n" , list2[i].num, list2[i].txt); + } +return 0 ; +}
fs/proc/proc_misc.c +68 -0
78 lines not on this page · raw message
include/linux/mm.h +5 -0
@@ -257,6 +257,11 @@void *virtual;/* Kernel virtual address (NULL if not kmapped, ie. highmem) */ #endif /* WANT_PAGE_VIRTUAL */ +#ifdef CONFIG_PAGE_OWNER +int order; +unsigned int gfp_mask; +unsigned long trace[8 ]; +#endif };/*
lib/Kconfig.debug +10 -0
@@ -139,6 +139,16 @@ automatically, but we'd like to make it more efficient by not having to do that. +config PAGE_OWNER + bool "Track page owner" + depends on DEBUG_KERNEL && X86 + help + This keeps track of what call chain is the owner of a page, may + help to find bare alloc_page(s) leaks. Eats a fair amount of memory. + See Documentation/page_owner.c for user-space helper. + + If unsure, say N. + config DEBUG_FS bool "Debug Filesystem" depends on DEBUG_KERNEL
mm/page_alloc.c +57 -0
81 lines not on this page · raw message
Quoted 1 line · Zwane Mwaikambo
Try looking at slabtop(1) output after a few days.
Well, this is interesting and all, and reiser_inode_cache is taking up a lot of memory, but according to the /proc/meminfo I had posted, slab is only using about 60MB. It doesn't appear to be the cause of any leaks.
Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/ OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA 2E2A E96F B2DC 6999 80E8
Quoted 5 lines · Alexander Nyberg
You should keep an eye on
/proc/meminfobut if there is memory that is not accounted for then the patch below might help as it works on a lower level. It accounts for bare pages in the system available from/proc/page_owner.
Select Track page owner under kernel hacking.
I will try that the next time I have to reboot. As this is a server, I cannot arbitrarily take it down unfortunately.
Quoted 2 lines
Also the meminfo you posted, how long had the box been alive when you took it?
Almost exactly 2 days.
Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/ OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA 2E2A E96F B2DC 6999 80E8
Quoted 5 lines · Alexander Nyberg
the patch below might help as it works on a lower
level. It accounts for bare pages in the system available from/proc/page_owner. So a cat/proc/page_owner> tmpfile would be good when the system starts to go low. There's a sorting program inDocumentation/page_owner.cused to sort the rather large output.
I've been running this for a day and a half now, and a few hundred megs of memory is now missing:
# free
total used free shared buffers cached
Mem: 2055648 2001884 53764 0 259024 868484
-/+ buffers/cache: 874376 1181272
Swap: 1028152 56 1028096I've put the output from the sorting program up at
http://untroubled.org/misc/page_owner_sorted
Is this useful information yet, or is there still too much in cached pages to really identify the source?
Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/ OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA 2E2A E96F B2DC 6999 80E8
(Please always do reply-to-all)
Quoted 17 lines · Bruce Guenter
Quoted 5 lines · Alexander Nyberg
the patch below might help as it works on a lower
level. It accounts for bare pages in the system available from/proc/page_owner. So a cat/proc/page_owner> tmpfile would be good when the system starts to go low. There's a sorting program inDocumentation/page_owner.cused to sort the rather large output.I've been running this for a day and a half now, and a few hundred megs of memory is now missing:
# free total used free shared buffers cached Mem: 2055648 2001884 53764 0 259024 868484 -/+ buffers/cache: 874376 1181272 Swap: 1028152 56 1028096
Quoted 5 lines
I've put the output from the sorting program up at
http://untroubled.org/misc/page_owner_sortedIs this useful information yet, or is there still too much in cached pages to really identify the source?
It all looks pretty innocent. Please send the contents of /proc/meminfo rather than the `free' output. /proc/meminfo has much more info. Sometimes /proc/vmstat is also useful.
If the /proc/meminfo output indicates that there are a lot of slab pages then /proc/slabinfo should be looked at.
Quoted 2 lines · Andrew Morton
It all looks pretty innocent. Please send the contents of
/proc/meminforather than the `free' output./proc/meminfohas much more info.
Here are the current meminfo numbers:
MemTotal: 2055648 kB MemFree: 56512 kB Buffers: 236880 kB Cached: 869616 kB SwapCached: 0 kB Active: 1004124 kB Inactive: 729732 kB HighTotal: 1179072 kB HighFree: 3584 kB LowTotal: 876576 kB LowFree: 52928 kB SwapTotal: 1028152 kB SwapFree: 1028096 kB Dirty: 1036 kB Writeback: 0 kB Mapped: 13100 kB Slab: 252444 kB CommitLimit: 2055976 kB Committed_AS: 25704 kB PageTables: 1060 kB VmallocTotal: 114680 kB VmallocUsed: 4700 kB VmallocChunk: 109836 kB
If I am counting right, free+buffers+cached+slab comes to 1415452 kB. Of course, at this point, it is far from out of memory like it has been in the past. I am continuing to monitor, and will post numbers when it gets closer to what I have previously observed.
Quoted 2 lines
If the
/proc/meminfooutput indicates that there are a lot of slab pages then/proc/slabinfoshould be looked at.
That was my first thought, yes. However, when it has run out of memory, even the slab totals were low (my first post showed only about 60 MB in slab).
Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/ OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA 2E2A E96F B2DC 6999 80E8
If you don't do reply-to-all there's a chance people will miss out on your mails...
Quoted 5 lines
Quoted 2 lines
It all looks pretty innocent. Please send the contents of
/proc/meminforather than the `free' output./proc/meminfohas much more info.Here are the current meminfo numbers:
What's happening with this? It's been a week now so I'm curious.
What you can do is run the attached program, it's a simple memory eater that will eat the amount of memory you specify, ie. "./a.out 2000" will simply eat 2G of memory. This is because all caches get reaped to a minimum leavel and distinguishing trouble makes is easier this way.
If you think the machine has lost memory at this time please do:
gcc memeat.c
./a.out 2000
wait until program is done
save /proc/meminfo
save /proc/page_owner
sort page_owner output
#include <stdio.h>
#include <stdlib.h>
#define page_size 4096
int main(int argc, char *argv[])
{
long size = strtoul(argv[1], NULL, 0);
long curr = 0;
printf("allocating %ldmb\n", size);
size *= 1024 * 1024;
while (curr <= size) {
char *ptr = malloc(page_size);
if (!ptr) {
printf("Couldn't allocate after %ld\n", curr);
sleep(1);
continue;
}
memset(ptr, 0, page_size);
curr += page_size;
}
return 0;
}Quoted 6 lines · Alexander Nyberg
Quoted 4 lines
Quoted 2 lines
It all looks pretty innocent. Please send the contents of
/proc/meminforather than the `free' output./proc/meminfohas much more info.Here are the current meminfo numbers:
What's happening with this? It's been a week now so I'm curious.
It appears the memory consumption I thought I was seeing is now gone, and only conclusively appeared with the Gentoo kernels. I will take this back to their bug tracker. Sorry for the false alarm.
Signature · 2 lines
Bruce Guenter <[email protected]> http://em.ca/~bruceg/ http://untroubled.org/ OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA 2E2A E96F B2DC 6999 80E8