New: Memory usage doubles after more then 20 hours of uptime.

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Sounds like a memory leak.

Please get the machine into this state and then send us

- the contents of /proc/meminfo

- the contents of /proc/slabinfo

- the contents of /proc/vmstat

- as root:

dmesg -c
echo m > /proc/sysrq-trigger
dmesg

thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Will do. But it will take a day or two to get there, I rebooted today because of this problem.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Here is the requested content:

free -m: http://pastebin.com/vb878a9Y
cat /proc/meminfo : http://pastebin.com/zUDFcYEW
cat /proc/slabinfo : http://pastebin.com/kswsJ7Hk
cat /proc/vmstat : http://pastebin.com/wUebJqJe

dmesg -c : http://pastebin.com/f7cTu8Wv

echo m > /proc/sysrq-trigger && dmesg : http://pastebin.com/p68DcHUy

And here are also files with that content:
http://ubuntuone.com/5GUVahBTiZRP0QjQdP3gkQ

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

You've lost 2-3GB of ZONE_NORMAL and I see no sign there to indicate where it went.

/proc/slabinfo indicates that it isn't a slab leak, and kmemleak won't tell us about alloc_pages() leaks. I'm stumped. Dave, any progress at your end?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I didn't understood anything you sad, but never mind.

In -rc2 there was a problem with massive iowait when anything was done (starting an app, loading a web page, etc ...), and there was a massive read operation from my /home partition. In -rc3 that stopped, and this started happening. Maybe it is related somehow?

Or maybe, it is just some problem with nvidia blob and 3.7 kernel loosing VM_RELEASE (in a blob's mmap.c it was replaced with VM_DONTEXPAND | VM_DONTDUMP ). - or maybe I'm just saying nonsense here.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I'm using Intel graphics, so it's not nvidia related for me, at least.

I've been recording a bunch of gunk from /proc once a minute for the past 16 hours or so. I've grepped some of it in to a log file (but I've got a *LOT* more than this):

	http://sr71.net/~dave/linux/leak-20121113/log.1353087988.txt.gz

and their sum. That should capture _most_ of the memory use on the system, and if we see that sum going down, it's probably a sign of the leak, especially when we see a trend over a long period. The file is in roughly this format, if anyone cares:

<nr/date> <meminfo fields> sums: <sum fields> <delta>

The system in question is my laptop. What I can tell is that it doesn't leak much when I'm not using it. But, it's leaking pretty steadily since I started using the system today (~6am in the logs). It _averages_ leaking about 400kB/minute when idle and almost 9MB/minute when in active use.

I've tried to provoke the leak doing specific things like large downloads, kernel compiles, watching video, alloc'ing a bunch of transparent huge pages, then exiting... No smoking gun so far.

Anybody have ideas what to try next or want to poke holes in my statistics? :)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

For me, it mostly happens over night, when there is only tvtime and deluge active (and rest of the programs are open but I don't use them - firefox, evolution and pidgin).
Only ones it happened while I was on the PC doing something. It was fresh after reboot, and I was restarting Firefox, 10 or more times (I was experimenting with addons).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Maybe resurrect the below patch? It's probably six years old. It should allow us to find out who allocated those pages.

Then perhaps we should merge the sucker this time.

From: Alexander Nyberg <[email protected]>

Introduces CONFIG_PAGE_OWNER that keeps track of the call chain under which a page was allocated. Includes a user-space helper in Documentation/page_owner.c to sort the enormous amount of output that this may give (thanks tridge).

Information available through /proc/page_owner

x86_64 introduces some stack noise in certain call chains so for exact output use of x86 && CONFIG_FRAME_POINTER is suggested. Tested on x86, x86 && CONFIG_FRAME_POINTER, x86_64

Output looks like:

4819 times:
Page allocated via order 0, mask 0x50
[0xc012b7b9] find_lock_page+25
[0xc012b8c8] find_or_create_page+152
[0xc0147d74] grow_dev_page+36
[0xc0148164] __find_get_block+84
[0xc0147ebc] __getblk_slow+124
[0xc0148164] __find_get_block+84
[0xc01481e7] __getblk+55
[0xc0185d14] do_readahead+100

We use a custom stack unwinder because using __builtin_return_address([0-7]) causes gcc to generate code that might try to unwind the stack looking for function return addresses and "fall off" causing early panics if the call chain is not deep enough. So in that case we could have had a depth of around 3 functions in all traces (I experimented a bit with this).

From: Dave Hansen <[email protected]>

make page_owner handle non-contiguous page ranges

From: Alexander Nyberg <[email protected]>

I've cleaned up the __alloc_pages() part to a simple set_page_owner() call.

DESC
Update page->order at an appropriate time when tracking PAGE_OWNER EDESC
From: [email protected] (Mel Gorman)

PAGE_OWNER tracks free pages by setting page->order to -1. However, it is set during __free_pages() which is not the only free path as __pagevec_free() and free_compound_page() do not go through __free_pages().
This leads to a situation where free pages are visible in /proc/page_owner
which is confusing and might be interpreted as a memory leak.

This patch sets page->owner when PageBuddy is set. It also prints a warning to the kernel log if a free page is found that does not appear free to PAGE_OWNER. This should be considered a fix to page-owner-tracking-leak-detector.patch.

This only applies to -mm as PAGE_OWNER is not in mainline.

DESC
Print out PAGE_OWNER statistics in relation to fragmentation avoidance EDESC
From: Mel Gorman <[email protected]>

When PAGE_OWNER is set, more information is available of relevance to fragmentation avoidance. A second line is added to /proc/page_owner showing the PFN, the pageblock number, the mobility type of the page based on its allocation flags, whether the allocation is improperly placed and the flags. A sample entry looks like

Page allocated via order 0, mask 0x1280d2
PFN 7355 Block 7 type 3 Fallback Flags LA
[0xc01528c6] __handle_mm_fault+598
[0xc0320427] do_page_fault+279
[0xc031ed9a] error_code+114

This information can be used to identify pages that are improperly placed. As the format of PAGE_OWNER data is now different, the comment at the top of Documentation/page_owner.c is updated with new instructions.

As PAGE_OWNER tracks the GFP flags used to allocate the pages, /proc/pagetypeinfo is enhanced to contain how many mixed blocks exist. The additional output looks like

Number of mixed blocks Unmovable Reclaimable Movable Reserve

Node 0, zone      DMA            0            1            2            1
Node 0, zone   Normal            2           11           33            0

DESC
Allow PAGE_OWNER to be set on any architecture
EDESC
From: Mel Gorman <[email protected]>

Currently PAGE_OWNER depends on CONFIG_X86. This appears to be due to pfn_to_page() being called in an inappropriate for many memory models and the presense of memory holes. This patch ensures that pfn_valid() and pfn_valid_within() is called at the appropriate places and the offsets correctly updated so that PAGE_OWNER is safe on any architecture.

In situations where CONFIG_HOLES_IN_ZONES is set (IA64 with VIRTUAL_MEM_MAP), there may be cases where pages allocated within a MAX_ORDER_NR_PAGES block of pages may not be displayed in /proc/page_owner if the hole is at the start of the block. Addressing this would be quite complex, perform slowly and is of no clear benefit.

Once PAGE_OWNER is allowed on all architectures, the statistics for grouping pages by mobility that declare how many pageblocks contain mixed page types becomes optionally available on all arches.

This patch was tested successfully on x86, x86_64, ppc64 and IA64 machines.

DESC allow-page_owner-to-be-set-on-any-architecture-fix EDESC
From: Andrew Morton <[email protected]>

Cc: Andy Whitcroft <[email protected]>
Cc: Mel Gorman <[email protected]>
DESC allow-page_owner-to-be-set-on-any-architecture-fix fix EDESC
From: [email protected] (Mel Gorman)

Page-owner-tracking stores the a backtrace of an allocation in the struct page. How the stack trace is generated depends on whether CONFIG_FRAME_POINTER is set or not. If CONFIG_FRAME_POINTER is set, the frame pointer must be read using some inline assembler which is not available for all architectures.

This patch uses the frame pointer where it is available but has a fallback where it is not.

---

 Documentation/page_owner.c |  141 ++++++++++++++++++++++++++++++++++
 fs/proc/proc_misc.c        |  145 +++++++++++++++++++++++++++++++++++
 include/linux/mm_types.h   |    5 +
 lib/Kconfig.debug          |   10 ++
 mm/page_alloc.c            |   66 +++++++++++++++
 mm/vmstat.c                |   93 ++++++++++++++++++++++
 6 files changed, 460 insertions(+)

diff -puN /dev/null Documentation/page_owner.c
@@ -0,0 +1,141 @@
+/*
+ * User-space helper to sort the output of /proc/page_owner
+ *
+ * Example use:
+ * cat /proc/page_owner > page_owner_full.txt
+ * grep -v ^PFN page_owner_full.txt > page_owner.txt
+ * ./sort page_owner.txt sorted_page_owner.txt
+*/
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <string.h>
+
+struct block_list {
+	char *txt;
+	int len;
+	int num;
+};
+
+
+static struct block_list *list;
+static int list_size;
+static int max_size;
+
+struct block_list *block_head;
+
+int read_block(char *buf, FILE *fin)
+{
+	int ret = 0;
+	int hit = 0;
+	char *curr = buf;
+
+	for (;;) {
+		*curr = getc(fin);
+		if (*curr == EOF) return -1;
+
+		ret++;
+		if (*curr == '\n' && hit == 1)
+			return ret - 1;
+		else if (*curr == '\n')
+			hit = 1;
+		else
+			hit = 0;
+		curr++;
+	}
+}
+
+static int compare_txt(struct block_list *l1, struct block_list *l2)
+{
+	return strcmp(l1->txt, l2->txt);
+}
+
+static int compare_num(struct block_list *l1, struct block_list *l2)
+{
+	return l2->num - l1->num;
+}
+
+static void add_list(char *buf, int len)
+{
+	if (list_size != 0 &&
+	    len == list[list_size-1].len &&
+	    memcmp(buf, list[list_size-1].txt, len) == 0) {
+		list[list_size-1].num++;
+		return;
+	}
+	if (list_size == max_size) {
+		printf("max_size too small??\n");
+		exit(1);
+	}
+	list[list_size].txt = malloc(len+1);
+	list[list_size].len = len;
+	list[list_size].num = 1;
+	memcpy(list[list_size].txt, buf, len);
+	list[list_size].txt[len] = 0;
+	list_size++;
+	if (list_size % 1000 == 0) {
+		printf("loaded %d\r", list_size);
+		fflush(stdout);
+	}
+}
+
+int main(int argc, char **argv)
+{
+	FILE *fin, *fout;
+	char buf[1024];
+	int ret, i, count;
+	struct block_list *list2;
+	struct stat st;
+
+	fin = fopen(argv[1], "r");
+	fout = fopen(argv[2], "w");
+	if (!fin || !fout) {
+		printf("Usage: ./program <input> <output>\n");
+		perror("open: ");
+		exit(2);
+	}
+
+	fstat(fileno(fin), &st);
+	max_size = st.st_size / 100; /* hack ... */
+
+	list = malloc(max_size * sizeof(*list));
+
+	for(;;) {
+		ret = read_block(buf, fin);
+		if (ret < 0)
+			break;
+
+		buf[ret] = '\0';
+		add_list(buf, ret);
+	}
+
+	printf("loaded %d\n", list_size);
+
+	printf("sorting ....\n");
+
+	qsort(list, list_size, sizeof(list[0]), compare_txt);
+
+	list2 = malloc(sizeof(*list) * list_size);
+
+	printf("culling\n");
+
+	for (i=count=0;i<list_size;i++) {
+		if (count == 0 ||
+		    strcmp(list2[count-1].txt, list[i].txt) != 0) {
+			list2[count++] = list[i];
+		} else {
+			list2[count-1].num += list[i].num;
+		}
+	}
+
+	qsort(list2, count, sizeof(list[0]), compare_num);
+
+	for (i=0;i<count;i++) {
+		fprintf(fout, "%d times:\n%s\n", list2[i].num, list2[i].txt);
+	}
+	return 0;
+}

diff -puN include/linux/mm_types.h~page-owner-tracking-leak-detector include/linux/mm_types.h
@@ -101,6 +101,11 @@ struct page {
 #ifdef CONFIG_KMEMCHECK
 	void *shadow;
 #endif
+#ifdef CONFIG_PAGE_OWNER
+	int order;
+	unsigned int gfp_mask;
+	unsigned long trace[8];
+#endif
 };
 
 /*

diff -puN lib/Kconfig.debug~page-owner-tracking-leak-detector lib/Kconfig.debug
@@ -66,6 +66,16 @@ config UNUSED_SYMBOLS
 	  you really need it, and what the merge plan to the mainline kernel for
 	  your module is.
 
+config PAGE_OWNER
+	bool "Track page owner"
+	depends on DEBUG_KERNEL
+	help
+	  This keeps track of what call chain is the owner of a page, may
+	  help to find bare alloc_page(s) leaks. Eats a fair amount of memory.
+	  See Documentation/page_owner.c for user-space helper.
+
+	  If unsure, say N.
+
 config DEBUG_FS
 	bool "Debug Filesystem"
 	depends on SYSFS

_

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I at least got the sucker recompiling. Not my finest work, but here goes:

http://sr71.net/~dave/linux/leak-20121113/pageowner_for_3.7-rc5.patch

It's not pretty, and probably needs to (at least) get moved over to debugfs before getting merged, but it does appear to give some reasonable output. Figured I'd post it in case anyone else wants to give it a spin.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

page_owner didn't help, at least not directly. As the pages "leak", they stop showing up in page_owner, which means they're in the allocator. Check out buddyinfo/meminfo:

...

Just the 398 order-10 zone Normal pages account for ~1.6GB of free memory, yet the MemFree is ~1.3GB, and that's a *SINGLE* bucket in the buddy allocator. Adding them all up, it's fairly close to the amount of memory that I'm missing at the moment.

Rather than being a real leak, it looks like this might just be an accounting problem:

$ cat /proc/zoneinfo  | egrep 'free_pages|Node'
Node 0, zone      DMA
    nr_free_pages 3976
Node 0, zone    DMA32
    nr_free_pages 177041
Node 0, zone   Normal
    nr_free_pages 16148

That 16148 pages for ZONE_NORMAL is obviously bogus compared to what buddyinfo is saying.

Commit d1ce749a0d did mess with NR_FREE_PAGES accounting quite a bit. Guess I'll try a revert and see where I end up.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I managed to reproduce this on a second machine. The new system ran basically all weekend doing kernel compiles: no leak. But, I added some memory pressure, and made it start allocating a bunch of hugetlbfs pages. That made this bug kick in there too. It's somewhat hard to tell, but I _think_ the leaking is correlated with compaction activity.

I'm trying a bisect now.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

resending to linux-mm@...

I turned on kmemleak and was able to reproduce this on a second reboot, but it went most of a workday before I noticed it had leaked a bunch. Unfortunately, kmemleak didn't help at all. It found a few small things that _may_ be leaks, but nothing to account for this _massive_ loss. I'm stumped so far.

My next step is to add some logging to at least see if this is a gradual thing or it happens all at once, and maybe figure out what the heck I'm doing to trigger it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>