Is btrfs related to OOM death problems on my 8GB server with both 3.15.1 and 3.14?

I upgraded my server from 3.14 to 3.15.1 last week, and since then it's been running out of memory and deadlocking (panic= doesn't even work). I downgraded back to 3.14, but I already had the problem once since then.

OOM comes in, even though I have 0 swap used and AFAIK all my RAM isn't gone, it then fails to kill enough stuff and eventually it dies like this:

[80943.542209] Swap cache stats: add 814596, delete 814595, find 2567491/2808869
[80943.565106] Free swap  = 15612448kB
[80943.577607] Total swap = 15616764kB
[80943.589766] 2021665 pages RAM
[80943.600281] 0 pages HighMem/MovableOnly
[80943.613284] 28468 pages reserved
[80943.624330] 0 pages hwpoisoned
[80943.634824] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[80943.659669] [  918]     0   918      855        0       5      236         -1000 udevd
[80943.684789] [ 8022]     0  8022     3074        0       5       89         -1000 auditd
[80943.710154] [ 8253]     0  8253     1813        0       6      123         -1000 sshd
[80943.735024] [12001]     0 12001      854        0       5      241         -1000 udevd
[80943.760152] [18969]     0 18969      854        0       5      223         -1000 udevd
[80943.785293] Kernel panic - not syncing: Out of memory and no killable processes...

Here is my more recent capture on 3.14 when I was able to catch it before the panic and dump a bunch of sysrq data.
http://marc.merlins.org/tmp/btrfs-oom.txt

Things to note in that log:

[90621.895715] 2962 total pagecache pages
[90621.895716] 5 pages in swap cache
[90621.895717] Swap cache stats: add 145004, delete 144999, find 3314901/3316382
[90621.895718] Free swap  = 15230536kB
[90621.895718] Total swap = 15616764kB
[90621.895718] Total swap = 15616764kB
[90621.895719] 2021665 pages RAM
[90621.895720] 0 pages HighMem/MovableOnly
[90621.895720] 28468 pages reserved

I'm not a VM person so I don't know how to read this, but am I out of RAM but not out of swap (since clearly none was used), or am I out of a specific memory region that is causing me problems?

I'm not 100% certain btrfs is to blame, but somehow it's suspect when ugprading to 3.15 and getting btrfs problems then caused my 3 months running fine 3.14.0 kernel also to die with the same OOM problems.
Then again, I understand it could be red herring. Suggestions either way are appreciated :)

I tried raising this:

gargamel:~# echo 100 > /proc/sys/vm/swappiness

But so far I have too much unused RAM for any swap to be touched.

gargamel:~# free
             total       used       free     shared    buffers     cached
Mem:       7894792    4938596    2956196          0       2396    2909204
-/+ buffers/cache:    2026996    5867796
Swap:     15616764          0   15616764

The log is too big to paste here, but you can grep it for:

[90817.715833] SysRq : Show Memory
[90817.715833] SysRq : Show Memory
[90817.715833] SysRq : Show Memory
[90893.571151] SysRq : Show backtrace of all active CPUs
[90921.781599] SysRq : Show Blocked State
[91075.976611] SysRq : Show State
[91406.972046] SysRq : Terminate All Tasks
[91410.771584] SysRq : Emergency Remount R/O
[91413.222483] SysRq : Emergency Sync
[91430.316955] SysRq : Power Off
^^^^^^^^^^^^^^^^^^^^^^^
note the kernel was wedged enough that Power Off didn't work, apparently because it failed
to swap:
[91447.490142] CPU: 3 PID: 48 Comm: kswapd0 Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
[91447.490143] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[91447.490145] task: ffff8802126b6490 ti: ffff8802126e4000 task.ti: ffff8802126e4000
[91447.490146] RIP: 0010:[<ffffffff810898f5>]  [<ffffffff810898f5>] do_raw_spin_lock+0x23/0x27

Right after OOM started kicking in, console showed apparent deadlocks in btrfs. But is it possible that btrfs is then also eating all my memory somehow?

You can find the long details here:
http://marc.merlins.org/tmp/btrfs-oom.txt

[90801.680821] INFO: task btrfs-transacti:3433 blocked for more than 120 seconds.
[90801.712345]       Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
[90801.734394] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[90801.882691] btrfs-transacti D ffff88021387e800     0  3433      2 0x00000000
[90801.904863]  ffff88020b20de10 0000000000000046 ffff88020b20dfd8 ffff88021387e2d0
[90801.928448]  00000000000141c0 ffff88021387e2d0 ffff880211e94800 ffff880029c00dc0
[90801.952015]  ffff8802009d1f28 ffff8802009d1ed0 0000000000000000 ffff88020b20de20
[90801.975701] Call Trace:
[90801.984443]  [<ffffffff8160d2a1>] schedule+0x73/0x75
[90802.000438]  [<ffffffff8122b575>] btrfs_commit_transaction+0x330/0x849
[90802.021140]  [<ffffffff81085116>] ? finish_wait+0x65/0x65
[90802.038438]  [<ffffffff81227c48>] transaction_kthread+0xf8/0x1ab
[90802.057571]  [<ffffffff81227b50>] ? btrfs_cleanup_transaction+0x43f/0x43f
[90802.079092]  [<ffffffff8106bc62>] kthread+0xae/0xb6
[90802.094838]  [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[90802.113485]  [<ffffffff8161637c>] ret_from_fork+0x7c/0xb0
[90802.130821]  [<ffffffff8106bbb4>] ? __kthread_parkme+0x61/0x61
[90802.153881] INFO: task bash:6863 blocked for more than 120 seconds.
[90802.174582]       Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
[90802.195291] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[90802.220747] bash            D ffff88016177e780     0  6863   6862 0x20120086
[90802.268599]  ffff88006f7eb918 0000000000000046 ffff88006f7ebfd8 ffff88016177e250
[90802.292124]  00000000000141c0 ffff88016177e250 ffff88006f7eb980 ffff8801ad7a484c
[90802.315631]  0000000001ddc000 ffff88006f7eb998 ffff8801ad7a4830 ffff88006f7eb928
[90802.339144] Call Trace:
[90802.347518]  [<ffffffff8160d2a1>] schedule+0x73/0x75
[90802.363446]  [<ffffffff81242468>] lock_extent_bits+0x11e/0x195
[90802.382008]  [<ffffffff81085116>] ? finish_wait+0x65/0x65
[90802.399262]  [<ffffffff81238674>] lock_and_cleanup_extent_if_need+0x6c/0x198
[90802.421485]  [<ffffffff81239a21>] __btrfs_buffered_write+0x1e9/0x427
[90802.441622]  [<ffffffff8160dd00>] ? mutex_unlock+0x16/0x18
[90802.459170]  [<ffffffff8123a028>] btrfs_file_aio_write+0x3c9/0x4b7
[90802.478803]  [<ffffffff811552b8>] do_sync_write+0x59/0x78
[90802.496102]  [<ffffffff810b3677>] do_acct_process+0x314/0x39c
[90802.514452]  [<ffffffff810b3ca7>] acct_process+0x77/0x92
[90802.531520]  [<ffffffff810520f4>] do_exit+0x3a0/0x938
[90802.547785]  [<ffffffff812986dd>] ? security_task_wait+0x16/0x18
[90802.566914]  [<ffffffff8105c571>] ? __dequeue_signal+0x1c/0x101
[90802.585790]  [<ffffffff81052731>] do_group_exit+0x6f/0xa2
[90802.603094]  [<ffffffff8105ed02>] get_signal_to_deliver+0x4e6/0x533
[90802.623011]  [<ffffffff8100f38f>] do_signal+0x49/0x5dc
[90802.639553]  [<ffffffff810b6499>] ? C_SYSC_wait4+0x28/0xba
[90802.657105]  [<ffffffff813b6353>] ? tty_ldisc_deref+0x16/0x18
[90802.675433]  [<ffffffff8115d770>] ? path_put+0x1e/0x21
[90802.691916]  [<ffffffff8100f957>] do_notify_resume+0x35/0x72
[90802.709952]  [<ffffffff816166ea>] int_signal+0x12/0x17

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Is there any correlation between such problems and BTRFS operations such as creating snapshots or running a scrub/balance?

Back in ~3.10 days I had serious problems with BTRFS memory use when removing multiple snapshots or balancing. But at about 3.13 they all seemed to get fixed.

I usually didn't have a kernel panic when I had such problems (although I sometimes had a system lock up solid such that I couldn't even determine what it's problem was). Usually the Oom handler started killing big processes such as chromium when it shouldn't have needed to.

Note that I haven't verified that the BTRFS memory use is reasonable in all such situations. Merely that it doesn't use enough to kill my systems.

My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

So far I don't have time data of when it happened, but I'm running new code to do better logging and hopefully give me a hint of when this is happening. However since it takes 6 to 48H to trigger, it'll take a little bit before I can report back.

Thanks
Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Hi,

Were you running scrub, Marc?

http://marc.merlins.org/tmp/btrfs-oom.txt:
===
...

[90621.895922] [ 8034]     0  8034     1315      164       5       46             0 btrfs-scrub

...
===

In this case, you would hit kernel memory leak. However, I can't find who is the root cause from this log.

Marc, do you change

- software and its setting,
- operations,
- hardware configuration,

or any other, just before detecting first OOM?

You have 8GB RAM and there is plenty of swap space.

===============================================================================

[90621.895719] 2021665 pages RAM
...
[90621.895718] Free swap  = 15230536kB

===============================================================================

Here are the avaliable memory of for each OOM-killer.

1st OOM:
===============================================================================

[90622.074758] Out of memory: Kill process 11452 (mh) score 2 or sacrifice child
[90622.074760] Killed process 11452 (mh) total-vm:66208kB, anon-rss:0kB, file-rss:872kB
[90622.425826] rfx-xpl-static invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
                                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~

It failed to acquire order=0 (2^0=1) page. So it's not kernel-memory-fragmentation case. Since __GFP_IO(0x80) and __GFP_FS(0x80) is set in gfp_mask, it can swap out anon/file pages to swap/filesystems to prepare free memories.

[90622.425829] rfx-xpl-static cpuset=/ mems_allowed=0
[90622.425832] CPU: 2 PID: 748 Comm: rfx-xpl-static Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
[90622.425833] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3806 08/20/2012
[90622.425834]  0000000000000000 ffff8801414a79d8 ffffffff8160a06d ffff8801434b2050
[90622.425838]  ffff8801414a7a68 ffffffff81607078 0000000000000000 ffffffff8160dd00
[90622.425841]  ffff8801414a7a08 ffffffff810501b4 ffff8801414a7a48 ffffffff8109cb05
[90622.425844] Call Trace:
[90622.425846]  [<ffffffff8160a06d>] dump_stack+0x4e/0x7a
[90622.425851]  [<ffffffff81607078>] dump_header+0x7f/0x206
[90622.425854]  [<ffffffff8160dd00>] ? mutex_unlock+0x16/0x18
[90622.425857]  [<ffffffff810501b4>] ? put_online_cpus+0x6c/0x6e
[90622.425861]  [<ffffffff8109cb05>] ? rcu_oom_notify+0xb3/0xc6
[90622.425865]  [<ffffffff81101a7f>] oom_kill_process+0x6e/0x30e
[90622.425869]  [<ffffffff811022b6>] out_of_memory+0x42e/0x461
[90622.425872]  [<ffffffff81106dfe>] __alloc_pages_nodemask+0x673/0x854
[90622.425876]  [<ffffffff8113b654>] alloc_pages_vma+0xd1/0x116
[90622.425880]  [<ffffffff81130ab7>] read_swap_cache_async+0x74/0x13b
[90622.425883]  [<ffffffff81130cc1>] swapin_readahead+0x143/0x152
[90622.425886]  [<ffffffff810fede9>] ? find_get_page+0x69/0x75
[90622.425889]  [<ffffffff81122adf>] handle_mm_fault+0x56b/0x9b0
[90622.425892]  [<ffffffff81612de6>] __do_page_fault+0x381/0x3cd
[90622.425895]  [<ffffffff81078cfc>] ? wake_up_state+0x12/0x12
[90622.425899]  [<ffffffff8115d770>] ? path_put+0x1e/0x21
[90622.425903]  [<ffffffff81612e57>] do_page_fault+0x25/0x27
[90622.425906]  [<ffffffff816103f8>] page_fault+0x28/0x30
[90622.425910] Mem-Info:
[90622.425910] Node 0 DMA per-cpu:
[90622.425913] CPU    0: hi:    0, btch:   1 usd:   0
[90622.425914] CPU    1: hi:    0, btch:   1 usd:   0
[90622.425915] CPU    2: hi:    0, btch:   1 usd:   0
[90622.425916] CPU    3: hi:    0, btch:   1 usd:   0
[90622.425916] Node 0 DMA32 per-cpu:
[90622.425919] CPU    0: hi:  186, btch:  31 usd:  24
[90622.425920] CPU    1: hi:  186, btch:  31 usd:   1
[90622.425921] CPU    2: hi:  186, btch:  31 usd:   0
[90622.425922] CPU    3: hi:  186, btch:  31 usd:   0
[90622.425923] Node 0 Normal per-cpu:
[90622.425924] CPU    0: hi:  186, btch:  31 usd:   0
[90622.425925] CPU    1: hi:  186, btch:  31 usd:   0
[90622.425926] CPU    2: hi:  186, btch:  31 usd:   0
[90622.425928] CPU    3: hi:  186, btch:  31 usd:   0
[90622.425932] active_anon:57 inactive_anon:92 isolated_anon:0
[90622.425932]  active_file:987 inactive_file:1232 isolated_file:0
[90622.425932]  unevictable:1389 dirty:590 writeback:1 unstable:0
[90622.425932]  free:25102 slab_reclaimable:9147 slab_unreclaimable:30944

There are few anon/file, in other word, reclaimable pages. The system would be almost full of kernel memory. As I said, kernel memory leak would happen here.

[90622.425932]  mapped:771 shmem:104 pagetables:1487 bounce:0
[90622.425932]  free_cma:0
[90622.425933] Node 0 DMA free:15360kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15360kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

~~~~~~~~~~~~~~~~~~~~~~

"all_unreclaimable? == yes" means "page reclaim work do my best and there is nothing to do any more".

[90622.425940] lowmem_reserve[]: 0 3204 7691 7691
[90622.425943] Node 0 DMA32 free:45816kB min:28100kB low:35124kB high:42148kB active_anon:0kB inactive_anon:88kB active_file:1336kB inactive_file:1624kB unevictable:1708kB isolated(anon):0kB isolated(file):0kB present:3362328kB managed:3284952kB mlocked:1708kB dirty:244kB writeback:0kB mapped:964kB shmem:0kB slab_reclaimable:128kB slab_unreclaimable:4712kB kernel_stack:1824kB pagetables:2096kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4807 all_unreclaimable? yes
[90622.425950] lowmem_reserve[]: 0 0 4486 4486
[90622.425953] Node 0 Normal free:39232kB min:39348kB low:49184kB high:59020kB active_anon:228kB inactive_anon:280kB active_file:2612kB inactive_file:3304kB unevictable:3848kB isolated(anon):0kB isolated(file):0kB present:4708352kB managed:4594480kB mlocked:3848kB dirty:2116kB writeback:4kB mapped:2120kB shmem:416kB slab_reclaimable:36460kB slab_unreclaimable:119064kB kernel_stack:2040kB pagetables:3852kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9683 all_unreclaimable? yes
[90622.425959] lowmem_reserve[]: 0 0 0 0
[90622.425962] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15360kB
[90622.425973] Node 0 DMA32: 10492*4kB (UEM) 2*8kB (U) 0*16kB 0*32kB 1*64kB (R) 1*128kB (R) 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 0*4096kB = 46016kB
[90622.425985] Node 0 Normal: 8763*4kB (UEM) 33*8kB (UE) 2*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 39444kB
[90622.425997] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[90622.425998] 3257 total pagecache pages
[90622.425999] 53 pages in swap cache
[90622.426000] Swap cache stats: add 145114, delete 145061, find 3322456/3324032
[90622.426001] Free swap  = 15277320kB
[90622.426002] Total swap = 15616764kB
[90622.426002] 2021665 pages RAM
[90622.426003] 0 pages HighMem/MovableOnly
[90622.426004] 28468 pages reserved
[90622.426004] 0 pages hwpoisoned
[90622.426005] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[90622.426011] [  917]     0   917      754       96       5      135         -1000 udevd
[90622.426014] [ 1634]     0  1634      592       81       5       50             0 bootlogd
[90622.426016] [ 1635]     0  1635      510       50       4       15             0 startpar
[90622.426020] [ 4336]     0  4336     1257      153       5      260             0 pinggw
[90622.426022] [ 7130]     0  7130      677       99       5       57             0 rpcbind
[90622.426024] [ 7160]   122  7160      746      152       5      100             0 rpc.statd
[90622.426026] [ 7195]     0  7195      757       74       5       44             0 rpc.idmapd
[90622.426028] [ 7604]     0  7604      753       87       5      136         -1000 udevd
[90622.426030] [ 8016]     0  8016      564      144       4       24             0 getty

===============================================================================

All processes above uses a little memory. It's because they are already evicted to swap/filesystem beforehand.

Thanks,
Satoru

Thank you for your answer. I'll put the conclusion and question at the top for easier reading:

So, should I understand that
1) I have enough RAM in my system but all of it disappears, apparently claimed by the kernel and not released

2) this could be a kernel memory leak in btrfs or somewhere else, there is no good way to know

If so, in a case like this, is there additional output I can capture to figure out how the memory is lost and help find out which part of the kernel is eating the memory without releasing it?
While btrfs is likely to blame, for now it's really just a guess and it would be good to confirm.

Yes, I was due to the other problem I was discussing on the list. I wanted to know if scrub would find any problem (it did not). I think I'll now try to read every file of every filesystem to see what btrfs does (this will take a while, that's around 100 million files).

But the last times I had this OOM problem with 3.15.1 it was happening within 6 hours sometimes, and I was not starting scrub every time the system booted, so scrub may be partially responsible but it's not the core problem.
(Also I run scrub on this system every few weeks and it hadn't OOM crashed inthe past)

Those are 3 good questions, I asked myself the same thing.

However this machine has many cronjobs, it does rsyncs to and from remote systems, it has btrfs send/receive going to and from it, and snapshots every hour.
Those are not new, but if any of them changed in a small way, I guess they could trigger bugs.

Correct.

Thanks for explaining.

Understood. I moved my question that was here at the top.

Thank you,
Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

It would be a good idea to run a few scrubs and see if this is a repeatable problem. If it's a repeatable problem then it's something to fix regardless of whether it's the only issue you have.

If a scrub can reliably trigger the problem it would be good to test 3.14 for the same behavior. Knowing whether it's a regression would help the developers.

Even without much swap 8G should be a plenty. My main workstation has 4G of RAM and 6G of swap. I almost never use more than 3G of swap because the system becomes so slow as to be almost unusable when swap gets to 4G (Chromium is to blame). However that is for a 120G non-RAID filesystem. Presumably a RAID array will need some more kernel memory and a larger filesystem will also need a little more, but it still shouldn't be that much.

My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Fair point. Re-running scrub now. That will take 36H or so :)

I'm already back to 3.14. 3.15 was dying about once a day without scrub, enough that this was causing me real problems (it's a server that is supposed to do work :) ).

You're correct. I only have more swap than ram, both as a habit in case I ever want to hibernate a system (I don't with this one) and in case some userland stuff leaks a lot (I've had versions of Xorg leak and that would improve the time between which I had to restart X and lose all my window state :) ).

Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Marc MERLIN posted on Fri, 04 Jul 2014 07:24:16 -0700 as excerpted:

Generally speaking if you're running out of RAM and swap is unused, it's because something is leaking memory that cannot be swapped. The kernel is the most likely suspect as kernel memory doesn't swap, especially so if the OOM-killer kills everything it can and the problem is still there.

So it's almost certainly the kernel, and btrfs is a good candidate, but I'm not a dev, and at least from that vantage point I didn't see enough information to conclusively pin it on btrfs. If the devs can't either, then either turning on additional debugging options or further debugging patches would seem to be in order.

So the above seems correct from here, yes.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

I didn't see any mention of the btrfs utility version in this thread (I may be blind though).

My server was suffering from frequent panics upon scrub / defrag / balance, until I updated the btrfs utility. That resolved all my issues.

My server has two filesystems of ~25 TiB, and 4 GB of RAM (15% used). A hardware upgrade is imminent, just waiting for the backup to complete.

~~
Andrew E. Mileski

Really? The userland tool should only send ioctls to the kernel, I really can't see how it would cause the kernel code to panic or not.

gargamel:~# btrfs --version
Btrfs v3.14.1
which is the latest in debian unstable.

As an update, after 1.7 days of scrubbing, the system has started getting sluggish, I'm getting synchronization problems/crashes in some of my tools that talk to serial ports (likely due to mini deadlocks in the kernel), and I'm now getting a few btrfs hangs.

A lot of my RAM seems gone now (1GB free but only 500MB actually used by user space), but the system is still up for now, just somewhat sluggish. I've been told meminfo, slabinfo and zoneinfo may be useful to debug that, so I'll attach them.
I see an interesting line about nf_conntrack and vmalloc in there. When my scrub is done, I'll reboot and disable nf_conntrack just to see what happens, and I'll also wait a bit to see how stable the RAM is before I start more scrubs to see if scrubs do create memmory leaks.

slabinfo (attached) doesn't seem to show that btrfs owns a lot of memory, but maybe I'm not sure how to read it.

gargamel:~# free
             total       used       free     shared    buffers     cached
Mem:       7894792    7370872     523920          0         28     564272
-/+ buffers/cache:    6806572    1088220
Swap:     15616764     483792   15132972

That one could be interesting, although just a a symptom of the memory pressure:

[144323.739212] nf_conntrack: falling back to vmalloc.
[144323.757139] nf_conntrack: falling back to vmalloc.
[144323.778092] nf_conntrack: falling back to vmalloc.
[144323.797785] nf_conntrack: falling back to vmalloc.

Is it possible for btrfs to be hung on getting memory allocations? Is a 2mn wait even possible for the right memory block to become available?

[149171.560084] INFO: task btrfs:17366 blocked for more than 120 seconds.
[149171.582499]       Not tainted 3.14.0-amd64-i915-preempt-20140216 #2
[149171.604251] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[149171.630523] btrfs           D ffff88006897c980     0 17366  18362 0x00000080
[149171.654267]  ffff880083a51720 0000000000000086 ffff880083a51fd8 ffff88006897c450
[149171.679265]  00000000000141c0 ffff88006897c450 ffff88021f3941c0 ffff88006897c450
[149171.704141]  ffff880083a517c0 ffffffff810fe7a2 0000000000000002 ffff880083a51730
[149171.729093] Call Trace:
[149171.738754]  [<ffffffff810fe7a2>] ? wait_on_page_read+0x3c/0x3c
[149171.759582]  [<ffffffff8160d2a1>] schedule+0x73/0x75
[149171.776663]  [<ffffffff8160d446>] io_schedule+0x60/0x7a
[149171.794400]  [<ffffffff810fe7b0>] sleep_on_page+0xe/0x12
[149171.812292]  [<ffffffff8160d6d8>] __wait_on_bit+0x48/0x7a
[149171.830401]  [<ffffffff810fe750>] wait_on_page_bit+0x7a/0x7c
[149171.849307]  [<ffffffff8108514a>] ? autoremove_wake_function+0x34/0x34
[149171.871229]  [<ffffffff81246320>] read_extent_buffer_pages+0x1bf/0x204
[149171.892813]  [<ffffffff81223dc0>] ? free_root_pointers+0x5b/0x5b
[149171.912642]  [<ffffffff81224ac2>] btree_read_extent_buffer_pages.constprop.45+0x66/0x100
[149171.938737]  [<ffffffff81225a17>] read_tree_block+0x2f/0x47
[149171.957308]  [<ffffffff8120eb66>] read_block_for_search.isra.26+0x24a/0x287
[149171.980128]  [<ffffffff812103a7>] btrfs_search_slot+0x4f4/0x6bb
[149171.999679]  [<ffffffff8127174d>] scrub_stripe+0x43d/0xc9e
[149172.017930]  [<ffffffff81085116>] ? finish_wait+0x65/0x65
[149172.035901]  [<ffffffff81272084>] scrub_chunk.isra.9+0xd6/0x10d
[149172.055614]  [<ffffffff8127232f>] scrub_enumerate_chunks+0x274/0x418
[149172.076546]  [<ffffffff81085100>] ? finish_wait+0x4f/0x65
[149172.094506]  [<ffffffff81272a73>] btrfs_scrub_dev+0x254/0x3cb
[149172.113493]  [<ffffffff8116e31e>] ? __mnt_want_write+0x62/0x78
[149172.132714]  [<ffffffff81256314>] btrfs_ioctl+0x1114/0x24bd
[149172.151111]  [<ffffffff81015f07>] ? paravirt_sched_clock+0x9/0xd
[149172.170794]  [<ffffffff810164ad>] ? sched_clock+0x9/0xb
[149172.188107]  [<ffffffff8107b2fe>] ? sched_clock_cpu+0x1a/0xc5
[149172.207380]  [<ffffffff811409bd>] ? ____cache_alloc+0x1c/0x29b
[149172.226446]  [<ffffffff81140d2b>] ? kmem_cache_alloc_node+0xef/0x179
[149172.247033]  [<ffffffff8160f97b>] ? _raw_spin_unlock+0x17/0x2a
[149172.266018]  [<ffffffff81163ffe>] do_vfs_ioctl+0x3d2/0x41d
[149172.283939]  [<ffffffff8116c1f0>] ? __fget+0x6f/0x79
[149172.300268]  [<ffffffff81164099>] SyS_ioctl+0x50/0x7b
[149172.316818]  [<ffffffff8161642d>] system_call_fastpath+0x1a/0x1f

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Sat Jul 5 07:30:09 PDT 2014

MemTotal:        7894792 kB
MemFree:          489800 kB
MemAvailable:     929864 kB
Buffers:              28 kB
Cached:           581012 kB
SwapCached:        33024 kB
Active:           368028 kB
Inactive:         388080 kB
Active(anon):      81496 kB
Inactive(anon):    97180 kB
Active(file):     286532 kB
Inactive(file):   290900 kB
Unevictable:        5352 kB
Mlocked:            5352 kB
SwapTotal:      15616764 kB
SwapFree:       15136100 kB
Dirty:              1672 kB
Writeback:          3396 kB
AnonPages:        171784 kB
Mapped:            29572 kB
Shmem:               660 kB
Slab:             186660 kB
SReclaimable:      63132 kB
SUnreclaim:       123528 kB
KernelStack:        5184 kB
PageTables:        10040 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    19564160 kB
Committed_AS:    1717256 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      358996 kB
VmallocChunk:   34359281468 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      144920 kB
DirectMap2M:     7942144 kB

slabinfo - version: 2.1

# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
pid_2                  0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
nfsd_drc               0      0    112   35    1 : tunables  120   60    8 : slabdata      0      0      0
nfsd4_delegations      0      0    368   11    1 : tunables   54   27    8 : slabdata      0      0      0
nfsd4_stateids         1     33    120   33    1 : tunables  120   60    8 : slabdata      1      1      0
nfsd4_files            1     31    128   31    1 : tunables  120   60    8 : slabdata      1      1      0
nfsd4_lockowners       0      0    384   10    1 : tunables   54   27    8 : slabdata      0      0      0
nfsd4_openowners       1     10    392   10    1 : tunables   54   27    8 : slabdata      1      1      0
nfs_direct_cache       0      0    208   19    1 : tunables  120   60    8 : slabdata      0      0      0
nfs_commit_data        4     11    704   11    2 : tunables   54   27    8 : slabdata      1      1      0
nfs_write_data        32     32    960    4    1 : tunables   54   27    8 : slabdata      8      8      0
nfs_read_data          0      0    896    4    1 : tunables   54   27    8 : slabdata      0      0      0
nfs_inode_cache        0      0   1016    4    1 : tunables   54   27    8 : slabdata      0      0      0
nfs_page               0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
fscache_cookie_jar      1     48     80   48    1 : tunables  120   60    8 : slabdata      1      1      0
rpc_inode_cache       15     18    640    6    1 : tunables   54   27    8 : slabdata      3      3      0
rpc_buffers            8      8   2048    2    1 : tunables   24   12    8 : slabdata      4      4      0
rpc_tasks              8     15    256   15    1 : tunables  120   60    8 : slabdata      1      1      0
ext4_groupinfo_1k     61     62    128   31    1 : tunables  120   60    8 : slabdata      2      2      0
jbd2_1k                0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
dm_snap_pending_exception      0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
dm_exception           0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
raid5-md8          16384  16384   1936    2    1 : tunables   24   12    8 : slabdata   8192   8192      0
nf_conntrack_ffffffff81cc7380    269    507    304   13    1 : tunables   54   27    8 : slabdata     39     39      0
nf_conntrack_expect      0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
fuse_request           9      9    408    9    1 : tunables   54   27    8 : slabdata      1      1      0
fuse_inode            94    100    768    5    1 : tunables   54   27    8 : slabdata     20     20      0
kvm_async_pf           0      0    136   29    1 : tunables  120   60    8 : slabdata      0      0      0
kvm_vcpu               0      0  16128    1    4 : tunables    8    4    0 : slabdata      0      0      0
kvm_mmu_page_header      0      0    168   23    1 : tunables  120   60    8 : slabdata      0      0      0
pte_list_desc          0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
bio-2                690   1530    256   15    1 : tunables  120   60    8 : slabdata    101    102    360
dm_crypt_io          708   1743    184   21    1 : tunables  120   60    8 : slabdata     82     83    300
kcopyd_job             0      0   3312    2    2 : tunables   24   12    8 : slabdata      0      0      0
io                     0      0     64   60    1 : tunables  120   60    8 : slabdata      0      0      0
dm_uevent              0      0   2608    3    2 : tunables   24   12    8 : slabdata      0      0      0
dm_rq_target_io        0      0    424    9    1 : tunables   54   27    8 : slabdata      0      0      0
dm_io                711   2511     40   93    1 : tunables  120   60    8 : slabdata     27     27    420
raid5-md5          16384  16384   1936    2    1 : tunables   24   12    8 : slabdata   8192   8192      0
btrfs_prelim_ref       4     48     80   48    1 : tunables  120   60    8 : slabdata      1      1      0
btrfs_delayed_extent_op    279    279     40   93    1 : tunables  120   60    8 : slabdata      3      3      0
btrfs_delayed_data_ref     96    200     96   40    1 : tunables  120   60    8 : slabdata      5      5      0
btrfs_delayed_tree_ref    264    264     88   44    1 : tunables  120   60    8 : slabdata      6      6      0
btrfs_delayed_ref_head    312    312    160   24    1 : tunables  120   60    8 : slabdata     13     13     60
btrfs_inode_defrag      0      0     64   60    1 : tunables  120   60    8 : slabdata      0      0      0
btrfs_delayed_node    447    871    304   13    1 : tunables   54   27    8 : slabdata     67     67      0
btrfs_ordered_extent      8     60    376   10    1 : tunables   54   27    8 : slabdata      4      6      0
btrfs_extent_map   46900  61854    152   26    1 : tunables  120   60    8 : slabdata   2379   2379    420
bio-1                333    912    320   12    1 : tunables   54   27    8 : slabdata     75     76    189
btrfs_extent_buffer  14930  16536    320   12    1 : tunables   54   27    8 : slabdata   1378   1378    162
btrfs_extent_state   3716  10400     96   40    1 : tunables  120   60    8 : slabdata    260    260    300
btrfs_delalloc_work      1     29    136   29    1 : tunables  120   60    8 : slabdata      1      1      0
btrfs_free_space   50084  54780     64   60    1 : tunables  120   60    8 : slabdata    913    913      0
btrfs_path           238    297    144   27    1 : tunables  120   60    8 : slabdata     11     11      0
btrfs_transaction     24     48    240   16    1 : tunables  120   60    8 : slabdata      3      3      0
btrfs_trans_handle     48     48    160   24    1 : tunables  120   60    8 : slabdata      2      2      0
btrfs_inode        12932  12932   1008    4    1 : tunables   54   27    8 : slabdata   3233   3233      0
fib6_nodes            13     60     64   60    1 : tunables  120   60    8 : slabdata      1      1      0
ip6_dst_cache         12     30    384   10    1 : tunables   54   27    8 : slabdata      3      3      0
ip6_mrt_cache          0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
PINGv6                 0      0   1088    7    2 : tunables   24   12    8 : slabdata      0      0      0
RAWv6                  7      7   1088    7    2 : tunables   24   12    8 : slabdata      1      1      0
UDPLITEv6              0      0   1088    7    2 : tunables   24   12    8 : slabdata      0      0      0
UDPv6                 12     35   1088    7    2 : tunables   24   12    8 : slabdata      5      5      0
tw_sock_TCPv6          0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
request_sock_TCPv6      0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
TCPv6                 30     30   1920    2    1 : tunables   24   12    8 : slabdata     15     15      0
flow_cache             0      0    104   37    1 : tunables  120   60    8 : slabdata      0      0      0
scsi_sense_cache     341    341    128   31    1 : tunables  120   60    8 : slabdata     11     11    180
scsi_cmd_cache       190    190    384   10    1 : tunables   54   27    8 : slabdata     19     19    108
sd_ext_cdb             2    113     32  113    1 : tunables  120   60    8 : slabdata      1      1      0
i915_gem_object        9     10    384   10    1 : tunables   54   27    8 : slabdata      1      1      0
btree_node             0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
cfq_io_cq            828    924    120   33    1 : tunables  120   60    8 : slabdata     28     28      0
cfq_queue            859    884    232   17    1 : tunables  120   60    8 : slabdata     52     52      0
bsg_cmd                0      0    312   12    1 : tunables   54   27    8 : slabdata      0      0      0
mqueue_inode_cache      1      4    896    4    1 : tunables   54   27    8 : slabdata      1      1      0
hugetlbfs_inode_cache      1      7    584    7    1 : tunables   54   27    8 : slabdata      1      1      0
jbd2_transaction_s      0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
jbd2_inode             0      0     48   78    1 : tunables  120   60    8 : slabdata      0      0      0
jbd2_journal_handle      0      0     48   78    1 : tunables  120   60    8 : slabdata      0      0      0
jbd2_journal_head      0      0    112   35    1 : tunables  120   60    8 : slabdata      0      0      0
jbd2_revoke_table_s      2    204     16  204    1 : tunables  120   60    8 : slabdata      1      1      0
jbd2_revoke_record_s      0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
ext4_inode_cache     349    352    968    4    1 : tunables   54   27    8 : slabdata     88     88      0
ext4_xattr             0      0     88   44    1 : tunables  120   60    8 : slabdata      0      0      0
ext4_free_data         0      0     64   60    1 : tunables  120   60    8 : slabdata      0      0      0
ext4_allocation_context      0      0    136   29    1 : tunables  120   60    8 : slabdata      0      0      0
ext4_prealloc_space      0      0    104   37    1 : tunables  120   60    8 : slabdata      0      0      0
ext4_system_zone       0      0     40   93    1 : tunables  120   60    8 : slabdata      0      0      0
ext4_io_end            0      0     72   53    1 : tunables  120   60    8 : slabdata      0      0      0
ext4_extent_status      8     93     40   93    1 : tunables  120   60    8 : slabdata      1      1      0
configfs_dir_cache      1     44     88   44    1 : tunables  120   60    8 : slabdata      1      1      0
dquot                  0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
kioctx                12     12    640    6    1 : tunables   54   27    8 : slabdata      2      2      0
kiocb                  0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
fanotify_event_info      0      0     64   60    1 : tunables  120   60    8 : slabdata      0      0      0
fanotify_response_event      0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
fsnotify_mark          0      0    112   35    1 : tunables  120   60    8 : slabdata      0      0      0
inotify_inode_mark    122    165    120   33    1 : tunables  120   60    8 : slabdata      5      5      0
dnotify_mark           1     33    120   33    1 : tunables  120   60    8 : slabdata      1      1      0
dnotify_struct         1    113     32  113    1 : tunables  120   60    8 : slabdata      1      1      0
dio                    0      0    640    6    1 : tunables   54   27    8 : slabdata      0      0      0
fasync_cache           0      0     48   78    1 : tunables  120   60    8 : slabdata      0      0      0
pid_namespace          0      0   2192    3    2 : tunables   24   12    8 : slabdata      0      0      0
user_namespace         0      0    264   15    1 : tunables   54   27    8 : slabdata      0      0      0
posix_timers_cache      0      0    248   16    1 : tunables  120   60    8 : slabdata      0      0      0
uid_cache             28     31    128   31    1 : tunables  120   60    8 : slabdata      1      1      0
UNIX                 204    204    896    4    1 : tunables   54   27    8 : slabdata     51     51      0
ip_mrt_cache           0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
UDP-Lite               0      0    896    4    1 : tunables   54   27    8 : slabdata      0      0      0
tcp_bind_bucket      170    300     64   60    1 : tunables  120   60    8 : slabdata      5      5      0
inet_peer_cache      248    740    192   20    1 : tunables  120   60    8 : slabdata     37     37      0
secpath_cache          0      0     64   60    1 : tunables  120   60    8 : slabdata      0      0      0
xfrm_dst_cache         0      0    512    8    1 : tunables   54   27    8 : slabdata      0      0      0
ip_fib_trie           22     68     56   68    1 : tunables  120   60    8 : slabdata      1      1      0
ip_fib_alias          25     78     48   78    1 : tunables  120   60    8 : slabdata      1      1      0
ip_dst_cache          73    140    192   20    1 : tunables  120   60    8 : slabdata      7      7      0
PING                   0      0    896    4    1 : tunables   54   27    8 : slabdata      0      0      0
RAW                    6      8    896    4    1 : tunables   54   27    8 : slabdata      2      2      0
UDP                   75     76    896    4    1 : tunables   54   27    8 : slabdata     19     19      0
tw_sock_TCP           38    120    192   20    1 : tunables  120   60    8 : slabdata      6      6      0
request_sock_TCP      30     30    256   15    1 : tunables  120   60    8 : slabdata      2      2      0
TCP                  108    108   1792    2    1 : tunables   24   12    8 : slabdata     54     54      0
eventpoll_pwq        265    265     72   53    1 : tunables  120   60    8 : slabdata      5      5      0
eventpoll_epi        120    248    128   31    1 : tunables  120   60    8 : slabdata      8      8      0
sgpool-128            49     49   4096    1    1 : tunables   24   12    8 : slabdata     49     49     12
sgpool-64             64     64   2048    2    1 : tunables   24   12    8 : slabdata     32     32     24
sgpool-32            112    112   1024    4    1 : tunables   54   27    8 : slabdata     28     28      0
sgpool-16             96     96    512    8    1 : tunables   54   27    8 : slabdata     12     12     27
sgpool-8             300    300    256   15    1 : tunables  120   60    8 : slabdata     20     20    120
scsi_data_buffer       0      0     24  146    1 : tunables  120   60    8 : slabdata      0      0      0
blkdev_integrity       0      0    112   35    1 : tunables  120   60    8 : slabdata      0      0      0
blkdev_queue          22     24   1896    2    1 : tunables   24   12    8 : slabdata     12     12      0
blkdev_requests      334    490    384   10    1 : tunables   54   27    8 : slabdata     49     49    189
blkdev_ioc           472    518    104   37    1 : tunables  120   60    8 : slabdata     14     14      0
bio-0               2430   3480    256   15    1 : tunables  120   60    8 : slabdata    232    232    420
biovec-256           315    376   4096    1    1 : tunables   24   12    8 : slabdata    315    376     96
biovec-128           109    160   2048    2    1 : tunables   24   12    8 : slabdata     80     80     60
biovec-64            341    352   1024    4    1 : tunables   54   27    8 : slabdata     88     88    189
biovec-16            210    210    256   15    1 : tunables  120   60    8 : slabdata     14     14     60
bio_integrity_payload      4     20    192   20    1 : tunables  120   60    8 : slabdata      1      1      0
khugepaged_mm_slot      0      0     40   93    1 : tunables  120   60    8 : slabdata      0      0      0
ksm_mm_slot            0      0     48   78    1 : tunables  120   60    8 : slabdata      0      0      0
ksm_stable_node        0      0     48   78    1 : tunables  120   60    8 : slabdata      0      0      0
ksm_rmap_item          0      0     64   60    1 : tunables  120   60    8 : slabdata      0      0      0
dmaengine-unmap-256      1      3   2112    3    2 : tunables   24   12    8 : slabdata      1      1      0
dmaengine-unmap-128      1      7   1088    7    2 : tunables   24   12    8 : slabdata      1      1      0
dmaengine-unmap-16      1     20    192   20    1 : tunables  120   60    8 : slabdata      1      1      0
dmaengine-unmap-2      1     60     64   60    1 : tunables  120   60    8 : slabdata      1      1      0
sock_inode_cache     444    444    640    6    1 : tunables   54   27    8 : slabdata     74     74      0
skbuff_fclone_cache     56     56    512    7    1 : tunables   54   27    8 : slabdata      8      8      0
skbuff_head_cache    465    555    256   15    1 : tunables  120   60    8 : slabdata     37     37    180
file_lock_cache      120    120    192   20    1 : tunables  120   60    8 : slabdata      6      6      0
net_namespace          0      0   4352    1    2 : tunables    8    4    0 : slabdata      0      0      0
shmem_inode_cache   1745   1800    656    6    1 : tunables   54   27    8 : slabdata    300    300      0
ftrace_event_file   1197   1232     88   44    1 : tunables  120   60    8 : slabdata     28     28      0
ftrace_event_field   2781   2808     48   78    1 : tunables  120   60    8 : slabdata     36     36      0
pool_workqueue        39     75    256   15    1 : tunables  120   60    8 : slabdata      5      5      0
task_delay_info      880    910    112   35    1 : tunables  120   60    8 : slabdata     26     26      0
taskstats             36     36    328   12    1 : tunables   54   27    8 : slabdata      3      3      0
proc_inode_cache    4038   4038    632    6    1 : tunables   54   27    8 : slabdata    673    673      0
sigqueue             235    288    160   24    1 : tunables  120   60    8 : slabdata     12     12      7
bdev_cache            41     48    832    4    1 : tunables   54   27    8 : slabdata     12     12      0
kernfs_node_cache  29369  29535    120   33    1 : tunables  120   60    8 : slabdata    895    895      0
mnt_cache             58     72    320   12    1 : tunables   54   27    8 : slabdata      6      6      0
filp                5715   6000    256   15    1 : tunables  120   60    8 : slabdata    400    400    480
inode_cache         9527  10283    568    7    1 : tunables   54   27    8 : slabdata   1469   1469      0
dentry             46720  46720    192   20    1 : tunables  120   60    8 : slabdata   2336   2336      0
names_cache           77     77   4096    1    1 : tunables   24   12    8 : slabdata     77     77     24
key_jar                4     20    192   20    1 : tunables  120   60    8 : slabdata      1      1      0
buffer_head           13    333    104   37    1 : tunables  120   60    8 : slabdata      9      9      0
nsproxy                1     78     48   78    1 : tunables  120   60    8 : slabdata      1      1      0
vm_area_struct     16239  17073    184   21    1 : tunables  120   60    8 : slabdata    813    813    360
mm_struct            364    364    960    4    1 : tunables   54   27    8 : slabdata     91     91      0
fs_cache             505    540     64   60    1 : tunables  120   60    8 : slabdata      9      9      0
files_cache          360    360    640    6    1 : tunables   54   27    8 : slabdata     60     60      0
signal_cache         602    602   1088    7    2 : tunables   24   12    8 : slabdata     86     86      0
sighand_cache        576    576   2112    3    2 : tunables   24   12    8 : slabdata    192    192      0
task_xstate          729    729    832    9    2 : tunables   54   27    8 : slabdata     81     81      0
task_struct          658    674   6304    1    2 : tunables    8    4    0 : slabdata    658    674      0
cred_jar            1408   1580    192   20    1 : tunables  120   60    8 : slabdata     79     79     60
Acpi-Operand        3606   3657     72   53    1 : tunables  120   60    8 : slabdata     69     69      0
Acpi-ParseExt         16     53     72   53    1 : tunables  120   60    8 : slabdata      1      1      0
Acpi-Parse            16     78     48   78    1 : tunables  120   60    8 : slabdata      1      1      0
Acpi-State            16     48     80   48    1 : tunables  120   60    8 : slabdata      1      1      0
Acpi-Namespace      1870   1953     40   93    1 : tunables  120   60    8 : slabdata     21     21      0
anon_vma_chain     17601  20220     64   60    1 : tunables  120   60    8 : slabdata    337    337    360
anon_vma           10261  11016     56   68    1 : tunables  120   60    8 : slabdata    162    162    180
pid                  851    961    128   31    1 : tunables  120   60    8 : slabdata     31     31      0
shared_policy_node      0      0     48   78    1 : tunables  120   60    8 : slabdata      0      0      0
numa_policy            3    146     24  146    1 : tunables  120   60    8 : slabdata      1      1      0
radix_tree_node    19286  20566    560    7    1 : tunables   54   27    8 : slabdata   2938   2938    135
idr_layer_cache      184    195   2112    3    2 : tunables   24   12    8 : slabdata     65     65      0
dma-kmalloc-4194304      0      0 4194304    1 1024 : tunables    1    1    0 : slabdata      0      0      0
dma-kmalloc-2097152      0      0 2097152    1  512 : tunables    1    1    0 : slabdata      0      0      0
dma-kmalloc-1048576      0      0 1048576    1  256 : tunables    1    1    0 : slabdata      0      0      0
dma-kmalloc-524288      0      0 524288    1  128 : tunables    1    1    0 : slabdata      0      0      0
dma-kmalloc-262144      0      0 262144    1   64 : tunables    1    1    0 : slabdata      0      0      0
dma-kmalloc-131072      0      0 131072    1   32 : tunables    8    4    0 : slabdata      0      0      0
dma-kmalloc-65536      0      0  65536    1   16 : tunables    8    4    0 : slabdata      0      0      0
dma-kmalloc-32768      0      0  32768    1    8 : tunables    8    4    0 : slabdata      0      0      0
dma-kmalloc-16384      0      0  16384    1    4 : tunables    8    4    0 : slabdata      0      0      0
dma-kmalloc-8192       0      0   8192    1    2 : tunables    8    4    0 : slabdata      0      0      0
dma-kmalloc-4096       0      0   4096    1    1 : tunables   24   12    8 : slabdata      0      0      0
dma-kmalloc-2048       0      0   2048    2    1 : tunables   24   12    8 : slabdata      0      0      0
dma-kmalloc-1024       0      0   1024    4    1 : tunables   54   27    8 : slabdata      0      0      0
dma-kmalloc-512        0      0    512    8    1 : tunables   54   27    8 : slabdata      0      0      0
dma-kmalloc-256        0      0    256   15    1 : tunables  120   60    8 : slabdata      0      0      0
dma-kmalloc-128        0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
dma-kmalloc-64         0      0     64   60    1 : tunables  120   60    8 : slabdata      0      0      0
dma-kmalloc-32         0      0     32  113    1 : tunables  120   60    8 : slabdata      0      0      0
dma-kmalloc-192        0      0    192   20    1 : tunables  120   60    8 : slabdata      0      0      0
dma-kmalloc-96         0      0    128   31    1 : tunables  120   60    8 : slabdata      0      0      0
kmalloc-4194304        0      0 4194304    1 1024 : tunables    1    1    0 : slabdata      0      0      0
kmalloc-2097152        0      0 2097152    1  512 : tunables    1    1    0 : slabdata      0      0      0
kmalloc-1048576        0      0 1048576    1  256 : tunables    1    1    0 : slabdata      0      0      0
kmalloc-524288         0      0 524288    1  128 : tunables    1    1    0 : slabdata      0      0      0
kmalloc-262144         0      0 262144    1   64 : tunables    1    1    0 : slabdata      0      0      0
kmalloc-131072         4      4 131072    1   32 : tunables    8    4    0 : slabdata      4      4      0
kmalloc-65536          7      7  65536    1   16 : tunables    8    4    0 : slabdata      7      7      0
kmalloc-32768          2      2  32768    1    8 : tunables    8    4    0 : slabdata      2      2      0
kmalloc-16384        287    287  16384    1    4 : tunables    8    4    0 : slabdata    287    287      0
kmalloc-8192          37     37   8192    1    2 : tunables    8    4    0 : slabdata     37     37      0
kmalloc-4096        1355   1357   4096    1    1 : tunables   24   12    8 : slabdata   1355   1357     36
kmalloc-2048         947    956   2048    2    1 : tunables   24   12    8 : slabdata    478    478     12
kmalloc-1024        2864   2864   1024    4    1 : tunables   54   27    8 : slabdata    716    716      0
kmalloc-512         2439   2632    512    8    1 : tunables   54   27    8 : slabdata    329    329     27
kmalloc-256        11945  12105    256   15    1 : tunables  120   60    8 : slabdata    807    807      0
kmalloc-192         6880   6940    192   20    1 : tunables  120   60    8 : slabdata    347    347    120
kmalloc-96          3410   5208    128   31    1 : tunables  120   60    8 : slabdata    168    168    480
kmalloc-64         28688  34080     64   60    1 : tunables  120   60    8 : slabdata    568    568    180
kmalloc-128        12130  13795    128   31    1 : tunables  120   60    8 : slabdata    445    445    120
kmalloc-32         20274  25764     32  113    1 : tunables  120   60    8 : slabdata    228    228    480
kmem_cache           233    260    192   20    1 : tunables  120   60    8 : slabdata     13     13      0

Node 0, zone DMA

  pages free     3840
        min      32
        low      40
        high     48
        scanned  0
        spanned  4095
        present  3995
        managed  3840
    nr_free_pages 3840
    nr_alloc_batch 8
    nr_inactive_anon 0
    nr_active_anon 0
    nr_inactive_file 0
    nr_active_file 0
    nr_unevictable 0
    nr_mlock     0
    nr_anon_pages 0
    nr_mapped    0
    nr_file_pages 0
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 0
    nr_slab_unreclaimable 0
    nr_page_table_pages 0
    nr_kernel_stack 0
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 0
    nr_vmscan_immediate_reclaim 0
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     0
    nr_dirtied   0
    nr_written   0
    numa_hit     16
    numa_miss    0
    numa_foreign 0
    numa_interleave 0
    numa_local   16
    numa_other   0
    nr_anon_transparent_hugepages 0
    nr_free_cma  0
        protection: (0, 3204, 7691, 7691)
  pagesets
    cpu: 0
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 2
    cpu: 1
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 2
    cpu: 2
              count: 0
              high:  0
              batch: 1
  vm stats threshold: 2
    cpu: 3
              count: 0
              high:  0
              batch: 1

vm stats threshold: 2
all_unreclaimable: 1
start_pfn: 1
inactive_ratio: 1
Node 0, zone DMA32

  pages free     106302
        min      7025
        low      8781
        high     10537
        scanned  0
        spanned  1044480
        present  840582
        managed  821238
    nr_free_pages 106302
    nr_alloc_batch 460
    nr_inactive_anon 23190
    nr_active_anon 19470
    nr_inactive_file 71852
    nr_active_file 70899
    nr_unevictable 569
    nr_mlock     569
    nr_anon_pages 41807
    nr_mapped    6552
    nr_file_pages 150284
    nr_dirty     328
    nr_writeback 0
    nr_slab_reclaimable 2314
    nr_slab_unreclaimable 2274
    nr_page_table_pages 1068
    nr_kernel_stack 376
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 138697
    nr_vmscan_immediate_reclaim 29869
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     62
    nr_dirtied   22352669
    nr_written   22020249
    numa_hit     1744799762
    numa_miss    0
    numa_foreign 0
    numa_interleave 0
    numa_local   1744799762
    numa_other   0
    nr_anon_transparent_hugepages 0
    nr_free_cma  0
        protection: (0, 0, 4486, 4486)
  pagesets
    cpu: 0
              count: 76
              high:  186
              batch: 31
  vm stats threshold: 36
    cpu: 1
              count: 92
              high:  186
              batch: 31
  vm stats threshold: 36
    cpu: 2
              count: 180
              high:  186
              batch: 31
  vm stats threshold: 36
    cpu: 3
              count: 60
              high:  186
              batch: 31

vm stats threshold: 36
all_unreclaimable: 0
start_pfn: 4096
inactive_ratio: 5
Node 0, zone Normal

  pages free     12279
        min      9837
        low      12296
        high     14755
        scanned  0
        spanned  1177088
        present  1177088
        managed  1148620
    nr_free_pages 12279
    nr_alloc_batch 2448
    nr_inactive_anon 1105
    nr_active_anon 904
    nr_inactive_file 873
    nr_active_file 734
    nr_unevictable 769
    nr_mlock     769
    nr_anon_pages 1139
    nr_mapped    841
    nr_file_pages 3232
    nr_dirty     90
    nr_writeback 849
    nr_slab_reclaimable 13469
    nr_slab_unreclaimable 28608
    nr_page_table_pages 1442
    nr_kernel_stack 272
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 36540130
    nr_vmscan_immediate_reclaim 430109
    nr_writeback_temp 0
    nr_isolated_anon 0
    nr_isolated_file 0
    nr_shmem     103
    nr_dirtied   14670364
    nr_written   45400987
    numa_hit     2124679820
    numa_miss    0
    numa_foreign 0
    numa_interleave 11035
    numa_local   2124679820
    numa_other   0
    nr_anon_transparent_hugepages 0
    nr_free_cma  0
        protection: (0, 0, 0, 0)
  pagesets
    cpu: 0
              count: 69
              high:  186
              batch: 31
  vm stats threshold: 42
    cpu: 1
              count: 166
              high:  186
              batch: 31
  vm stats threshold: 42
    cpu: 2
              count: 122
              high:  186
              batch: 31
  vm stats threshold: 42
    cpu: 3
              count: 157
              high:  186
              batch: 31

vm stats threshold: 42
all_unreclaimable: 0
start_pfn: 1048576
inactive_ratio: 6

It reached the point that I got so frustrated that I built the utilities from git, as an up-to-date package wasn't available at the time. I'm currently using v3.14.2 but v3.14.1 was the version that resolved my server's panics.

I've not looked into the code very deeply yet, so I can't theorize on why it resolved my server's panics. I can only state with complete certainty that it did. I too was somewhat surprised.

However, it seems we've probably now ruled it out in your case.

~~
Andrew E. Mileski

Predictably, it died yesterday afternoon after going into memory death (it was answering pings, but userspace was dead, and even sysrq-o did not respond, I had to power cycle the power outlet).

This happened just before my 3rd scrub finished, so I'm now 2 out of 2:
running scrub on my 3 filesystems kills the system half way through the 3rd scrub.

This is the last memory log that reached the disk:
http://marc.merlins.org/tmp/btrfs-oom2.txt

Do those logs point to any possible culprit, or a kernel memory leak cannot be pointed to its source because the kernel loses track of who requested the memory that leaked?

Excerpt here:
Sat Jul 5 14:25:04 PDT 2014
total used free shared buffers cached

Mem:       7894792    7712384     182408          0         28     227480
-/+ buffers/cache:    7484876     409916

Swap: 15616764 463732 15153032

Userspace is using 345MB according to ps

Sat Jul 5 14:25:04 PDT 2014

MemTotal:        7894792 kB
MemFree:          184556 kB
MemAvailable:     269568 kB
Buffers:              28 kB
Cached:           228164 kB
SwapCached:        18296 kB
Active:           178196 kB
Inactive:         187016 kB
Active(anon):      70068 kB
Inactive(anon):    71100 kB
Active(file):     108128 kB
Inactive(file):   115916 kB
Unevictable:        5624 kB
Mlocked:            5624 kB
SwapTotal:      15616764 kB
SwapFree:       15152768 kB
Dirty:             17716 kB
Writeback:           516 kB
AnonPages:        140588 kB
Mapped:            21940 kB
Shmem:               688 kB
Slab:             181708 kB
SReclaimable:      59808 kB
SUnreclaim:       121900 kB
KernelStack:        4728 kB
PageTables:         8480 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    19564160 kB
Committed_AS:    1633204 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      358996 kB
VmallocChunk:   34359281468 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      144920 kB
DirectMap2M:     7942144 kB

Thanks,
Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Ok, I now changed the subject line to confirm that btrfs is to blame.

I had my system booted 6 days and it held steady at 6.4GB free. I mounted 2 of my 4 btrfs filesystems (one by one) and waited a few days, and no problems with RAM going down.

Then I mounted my 3rd btrfs filesystem, the one that holds many files that has rsync backups running, and I lost 1.4GB of RAM overnight. I've just umounted one of its mountpoints, the one where all the backups happen while leaving its main pool mounted and will see if RAM keeps going down or not (i.e. whether the memory leak is due to rsync activity or some other background btrfs process).

But generally, is there a tool to locate which kernel function allocated all that RAM that seems to get allocated and forgotten?

Is /proc/slabinfo supposed to show anything useful?

This is the filesystem in question:

gargamel:~# btrfs fi df /mnt/btrfs_pool2/
Data, single: total=3.34TiB, used=3.32TiB
System, DUP: total=8.00MiB, used=400.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=77.50GiB, used=59.87GiB
Metadata, single: total=8.00MiB, used=0.00

Thanks,
Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Mmmh, now that I think of it, I do have quota enabled on that filesystem.
Due to the many many files, this may be what's causing the problem. I thought quotas were supposed to work with 3.15, but maybe there is still a leak?

I just turned quotas off and I'm going to let my server run for a while to see if the leak stops.

Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

-------- Original Message --------
Subject: Re: btrfs is related to OOM death problems on my 8GB server with both 3.15.1 and 3.14?
From: Marc MERLIN <[email protected]>
To: Andrew E. Mileski <[email protected]>
Date: 2014年07月13日 22:29

This can be done by kernel memleak detection.
Location:
-> Kernel hacking

    -> Memory Debugging
        -> Kernel memory leak detector

Then you can check /sys/kernel/debug/memleak to see which function call caused the problem.

Thanks,
Qu

Hi

I'd just like to comment that I currently have the same problem, quota are enabled, server can't stay up more than 15 days without hogging all of its memory. Use case is a torrent server which has more read than writes and a desktop. Kmemleak does not detect this leak after 10 days and at least 2 GB used.

On my system, only leaks found were from Nvidia and AppArmor (Ubuntu 14.04 kernel 3.13 with Nvidia proprietary driver).

Hi,

For this same problem I once got into single user after 2 weeks of utilisation and killed all, umounted all FS except root which is ext4, rmmod'ed all modules and see by yourself:
http://i39.tinypic.com/2rrrjtl.jpg

For those who want it textually:
15 days uptime
10 user mode processes (systemd, top and bash)
2 GB memory usage, 4 GB total memory, 17 MB cache, 32 KB buffers.

I wanted to report back on this.
Unfortunately my laptop with 3.15.5 does not stay up more than 30mn, sometimes fewer when kmemleak is on. When I turn it off from the boot command line, the laptop seems to behave ok.

Very interesting, thanks for confirming.
So even by removing the btrfs module, the memory was not reclaimed, correct?

For what it's worth, my server seems ok now that I turned off quotas on it.

Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

I forgot to attach the logs of what I get soon after enabling kmemleak:
http://marc.merlins.org/tmp/btrfs_hang_kmemleak.txt

Highlight, first what got logged as stuff started hanging, and later the top of sysrq-w (look for Sysrq to get to it more quickly)

kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)

INFO: task kworker/u16:4:95 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u16:4   D 0000000000000000     0    95      2 0x00000000
Workqueue: btrfs-flush_delalloc normal_work_helper
 ffff880407297b60 0000000000000046 ffff880407297b30 ffff880407297fd8
 ffff8804075f44d0 00000000000141c0 ffff88041e2941c0 ffff8804075f44d0
 ffff880407297c00 0000000000000002 ffffffff810fdda8 ffff880407297b70
Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff810731c8>] ? ttwu_stat+0x97/0xcc
 [<ffffffff8162223d>] ? _raw_spin_unlock_irqrestore+0x1f/0x32
 [<ffffffff81078164>] ? try_to_wake_up+0x1b3/0x1c5
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810fff50>] filemap_flush+0x1c/0x1e
 [<ffffffff81231945>] btrfs_run_delalloc_work+0x32/0x69
 [<ffffffff81252437>] normal_work_helper+0xfe/0x240
 [<ffffffff81065e29>] process_one_work+0x195/0x2d2
 [<ffffffff810660cb>] worker_thread+0x136/0x205
 [<ffffffff81065f95>] ? process_scheduled_works+0x2f/0x2f
 [<ffffffff8106b564>] kthread+0xae/0xb6
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61
 [<ffffffff81628d7c>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61
INFO: task btrfs-transacti:548 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
btrfs-transacti D 0000000000000000 0 548 2 0x00000000
ffff880404bbfc80 0000000000000046 000000000000e5fc ffff880404bbffd8 ffff880404bb82d0 00000000000141c0 7fffffffffffffff ffff8802099d2888 0000000000000002 ffffffff8161f048 7fffffffffffffff ffff880404bbfc90

Call Trace:
 [<ffffffff8161f048>] ? sock_rps_reset_flow+0x37/0x37
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161f081>] schedule_timeout+0x39/0x129
 [<ffffffff810765ed>] ? get_parent_ip+0xd/0x3c
 [<ffffffff81625a8f>] ? preempt_count_add+0x7a/0x8d
 [<ffffffff81620244>] __wait_for_common+0x11a/0x159
 [<ffffffff810781bf>] ? wake_up_state+0x12/0x12
 [<ffffffff816202a7>] wait_for_completion+0x24/0x26
 [<ffffffff8123981f>] btrfs_wait_and_free_delalloc_work+0x16/0x28
 [<ffffffff812417f8>] btrfs_run_ordered_operations+0x1e7/0x21e
 [<ffffffff8122cbad>] btrfs_commit_transaction+0x27/0x8b0
 [<ffffffff8122966a>] transaction_kthread+0xf8/0x1ab
 [<ffffffff81229572>] ? btrfs_cleanup_transaction+0x44c/0x44c
 [<ffffffff8106b564>] kthread+0xae/0xb6
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61
 [<ffffffff81628d7c>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61
INFO: task pidgin:11402 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pidgin D 0000000000000006 0 11402 5650 0x00000080
ffff8803c4307c30 0000000000000082 ffff8803c4307c00 ffff8803c4307fd8 ffff8803cb684050 00000000000141c0 ffff88041e3941c0 ffff8803cb684050 ffff8803c4307cd0 0000000000000002 ffffffff810fdda8 ffff8803c4307c40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81317ac9>] ? debug_smp_processor_id+0x17/0x19
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff8132254e>] ? __percpu_counter_add+0x8c/0xa6
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c46a>] SyS_fsync+0x10/0x14
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f
INFO: task BrowserBlocking:15468 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
BrowserBlocking D 0000000000000000 0 15468 15395 0x00000080
ffff88038ba6fc30 0000000000000086 ffff88038ba6fc00 ffff88038ba6ffd8 ffff8800b394c310 00000000000141c0 ffff88041e2141c0 ffff8800b394c310 ffff88038ba6fcd0 0000000000000002 ffffffff810fdda8 ffff88038ba6fc40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff81158d94>] ? __sb_end_write+0x2d/0x5b
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c46a>] SyS_fsync+0x10/0x14
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f
INFO: task Chrome_DBThread:15469 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Chrome_DBThread D 0000000000000004 0 15469 15395 0x00000080
ffff8803967afc30 0000000000000086 ffff8803967afc00 ffff8803967affd8 ffff880395892290 00000000000141c0 ffff88041e3141c0 ffff880395892290 ffff8803967afcd0 0000000000000002 ffffffff810fdda8 ffff8803967afc40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff8111d502>] ? copy_page_to_iter+0x163/0x26b
 [<ffffffff810fdf7d>] ? file_accessed+0x13/0x15
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c481>] SyS_fdatasync+0x13/0x17
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f
INFO: task Chrome_FileThre:15470 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Chrome_FileThre D 0000000000000005 0 15470 15395 0x00000080
ffff88038bd7fc30 0000000000000086 ffff88038bd7fc00 ffff88038bd7ffd8 ffff8803a065c250 00000000000141c0 ffff88041e3541c0 ffff8803a065c250 ffff88038bd7fcd0 0000000000000002 ffffffff810fdda8 ffff88038bd7fc40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff816204bc>] ? mutex_unlock+0x16/0x18
 [<ffffffff8123bb5d>] ? btrfs_file_aio_write+0x3e9/0x4b6
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff81158d94>] ? __sb_end_write+0x2d/0x5b
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c46a>] SyS_fsync+0x10/0x14
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f
INFO: task IndexedDB:15475 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
IndexedDB D 0000000000000007 0 15475 15395 0x00000080
ffff88038ba87c30 0000000000000086 ffff88038ba87c00 ffff88038ba87fd8 ffff88038bda6150 00000000000141c0 ffff88041e3d41c0 ffff88038bda6150 ffff88038ba87cd0 0000000000000002 ffffffff810fdda8 ffff88038ba87c40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff816204bc>] ? mutex_unlock+0x16/0x18
 [<ffffffff8123bb5d>] ? btrfs_file_aio_write+0x3e9/0x4b6
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a743>] btrfs_sync_file+0xa8/0x2b3
 [<ffffffff8132254e>] ? __percpu_counter_add+0x8c/0xa6
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c481>] SyS_fdatasync+0x13/0x17
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f
INFO: task BrowserBlocking:15487 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
BrowserBlocking D ffff8804044d4080 0 15487 15395 0x00000080
ffff88038ba0bc30 0000000000000086 ffff88038ba0bc00 ffff88038ba0bfd8 ffff8800b4256350 00000000000141c0 ffff88041e2941c0 ffff8800b4256350 ffff88038ba0bcd0 0000000000000002 ffffffff810fdda8 ffff88038ba0bc40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff8111d38c>] ? pagefault_enable+0xe/0x21
 [<ffffffff8111d502>] ? copy_page_to_iter+0x163/0x26b
 [<ffffffff810fdf7d>] ? file_accessed+0x13/0x15
 [<ffffffff81246a18>] extent_writepages+0xffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c481>] SyS_fdatasync+0x13/0x17
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f
INFO: task BrowserBlocking:15501 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
BrowserBlocking D 0000000000000002 0 15501 15395 0x00000080
ffff8800af733c30 0000000000000086 ffff8800af733c00 ffff8800af733fd8 ffff8800af73c3d0 00000000000141c0 ffff88041e2941c0 ffff8800af73c3d0 ffff8800af733cd0 0000000000000002 ffffffff810fdda8 ffff8800af733c40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff81158d94>] ? __sb_end_write+0x2d/0x5b
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c46a>] SyS_fsync+0x10/0x14
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f
INFO: task Chrome_HistoryT:15507 blocked for more than 120 seconds.
      Tainted: G           O  3.15.5-amd64-i915-preempt-20140714 #2

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Chrome_HistoryT D 0000000000000000 0 15507 15395 0x00000080
ffff88038772fc30 0000000000000086 ffff88038772fc00 ffff88038772ffd8 ffff88038ba0e1d0 00000000000141c0 ffff88041e2141c0 ffff88038ba0e1d0 ffff88038772fcd0 0000000000000002 ffffffff810fdda8 ffff88038772fc40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff813085a7>] ? radix_tree_lookup_slot+0x10/0x1f
 [<ffffffff810fe4bc>] ? find_get_entry+0x69/0x75
 [<ffffffff810fdf7d>] ? file_accessed+0x13/0x15
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c481>] SyS_fdatasync+0x13/0x17
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f

SysRq : Show Blocked State
task PC stack pid father
kworker/u16:4 D 0000000000000000 0 95 2 0x00000000

Workqueue: btrfs-flush_delalloc normal_work_helper
 ffff880407297b60 0000000000000046 ffff880407297b30 ffff880407297fd8
 ffff8804075f44d0 00000000000141c0 ffff88041e2941c0 ffff8804075f44d0
 ffff880407297c00 0000000000000002 ffffffff810fdda8 ffff880407297b70
Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff810731c8>] ? ttwu_stat+0x97/0xcc
 [<ffffffff8162223d>] ? _raw_spin_unlock_irqrestore+0x1f/0x32
 [<ffffffff81078164>] ? try_to_wake_up+0x1b3/0x1c5
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810fff50>] filemap_flush+0x1c/0x1e
 [<ffffffff81231945>] btrfs_run_delalloc_work+0x32/0x69
 [<ffffffff81252437>] normal_work_helper+0xfe/0x240
 [<ffffffff81065e29>] process_one_work+0x195/0x2d2
 [<ffffffff810660cb>] worker_thread+0x136/0x205
 [<ffffffff81065f95>] ? process_scheduled_works+0x2f/0x2f
 [<ffffffff8106b564>] kthread+0xae/0xb6
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61
 [<ffffffff81628d7c>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61

btrfs-transacti D 0000000000000000 0 548 2 0x00000000
ffff880404bbfc80 0000000000000046 000000000000e5fc ffff880404bbffd8 ffff880404bb82d0 00000000000141c0 7fffffffffffffff ffff8802099d2888 0000000000000002 ffffffff8161f048 7fffffffffffffff ffff880404bbfc90

Call Trace:
 [<ffffffff8161f048>] ? sock_rps_reset_flow+0x37/0x37
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161f081>] schedule_timeout+0x39/0x129
 [<ffffffff810765ed>] ? get_parent_ip+0xd/0x3c
 [<ffffffff81625a8f>] ? preempt_count_add+0x7a/0x8d
 [<ffffffff81620244>] __wait_for_common+0x11a/0x159
 [<ffffffff810781bf>] ? wake_up_state+0x12/0x12
 [<ffffffff816202a7>] wait_for_completion+0x24/0x26
 [<ffffffff8123981f>] btrfs_wait_and_free_delalloc_work+0x16/0x28
 [<ffffffff812417f8>] btrfs_run_ordered_operations+0x1e7/0x21e
 [<ffffffff8122cbad>] btrfs_commit_transaction+0x27/0x8b0
 [<ffffffff8122966a>] transaction_kthread+0xf8/0x1ab
 [<ffffffff81229572>] ? btrfs_cleanup_transaction+0x44c/0x44c
 [<ffffffff8106b564>] kthread+0xae/0xb6
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61
 [<ffffffff81628d7c>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61

pidgin D 0000000000000006 0 11402 5650 0x00000080
ffff8803c4307c30 0000000000000082 ffff8803c4307c00 ffff8803c4307fd8 ffff8803cb684050 00000000000141c0 ffff88041e3941c0 ffff8803cb684050 ffff8803c4307cd0 0000000000000002 ffffffff810fdda8 ffff8803c4307c40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81317ac9>] ? debug_smp_processor_id+0x17/0x19
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff8132254e>] ? __percpu_counter_add+0x8c/0xa6
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c46a>] SyS_fsync+0x10/0x14
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f
BrowserBlocking D 0000000000000000     0 15468  15395hedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff8111d502>] ? copy_page_to_iter+0x163/0x26b
 [<ffffffff810fdf7d>] ? file_accessed+0x13/0x15
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c481>] SyS_fdatasync+0x13/0x17
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f

Chrome_FileThre D 0000000000000005 0 15470 15395 0x00000080
ffff88038bd7fc30 0000000000000086 ffff88038bd7fc00 ffff88038bd7ffd8 ffff8803a065c250 00000000000141c0 ffff88041e3541c0 ffff8803a065c250 ffff88038bd7fcd0 0000000000000002 ffffffff810fdda8 ffff88038bd7fc40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff816204bc>] ? mutex_unlock+0x16/0x18
 [<ffffffff8123bb5d>] ? btrfs_file_aio_write+0x3e9/0x4b6
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff81158d94>] ? __sb_end_write+0x2d/0x5b
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c46a>] SyS_fsync+0x10/0x14
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f

IndexedDB D 0000000000000007 0 15475 15395 0x00000080
ffff88038ba87c30 0000000000000086 ffff88038ba87c00 ffff88038ba87fd8 ffff88038bda6150 00000000000141c0 ffff88041e3d41c0 ffff88038bda6150 ffff88038ba87cd0 0000000000000002 ffffffff810fdda8 ffff88038ba87c40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake/0x21
 [<ffffffff8111d502>] ? copy_page_to_iter+0x163/0x26b
 [<ffffffff810fdf7d>] ? file_accessed+0x13/0x15
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a743>] btrfs_sync_file+0xa8/0x2b3
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c481>] SyS_fdatasync+0x13/0x17
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f

BrowserBlocking D 0000000000000002 0 15501 15395 0x00000080
ffff8800af733c30 0000000000000086 ffff8800af733c00 ffff8800af733fd8 ffff8800af73c3d0 00000000000141c0 ffff88041e2941c0 ffff8800af73c3d0 ffff8800af733cd0 0000000000000002 ffffffff810fdda8 ffff8800af733c40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff810ff507>] __filemap_fdatawrite_range+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff81158d94>] ? __sb_end_write+0x2d/0x5b
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c46a>] SyS_fsync+0x10/0x14
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f

Chrome_HistoryT D 0000000000000000 0 15507 15395 0x00000080
ffff88038772fc30 0000000000000086 ffff88038772fc00 ffff88038772ffd8 ffff88038ba0e1d0 00000000000141c0 ffff88041e2141c0 ffff88038ba0e1d0 ffff88038772fcd0 0000000000000002 ffffffff810fdda8 ffff88038772fc40

Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff813085a7>] ? radix_tree_lookup_slot+0x10/0x1f
 [<ffffffff810fe4bc>] ? find_get_entry+0x69/0x75
 [<ffffffff810fdf7d>] ? file_accessed+0x13/0x15
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_wrrange+0x55/0x57
 [<ffffffff810ff57d>] filemap_fdatawrite_range+0x13/0x15
 [<ffffffff8123a726>] btrfs_sync_file+0x8b/0x2b3
 [<ffffffff8117c02f>] vfs_fsync_range+0x18/0x22
 [<ffffffff8117c055>] vfs_fsync+0x1c/0x1e
 [<ffffffff8117c261>] do_fsync+0x2c/0x4c
 [<ffffffff8117c481>] SyS_fdatasync+0x13/0x17
 [<ffffffff81628e2d>] system_call_fastpath+0x1a/0x1f
kworker/u16:5   D 0000000000000000     0 15410      2 0x00000080
Workqueue: btrfs-delalloc normal_work_helper
 ffff88039b863860 0000000000000046 ffff88039b863830 ffff88039b863fd8
 ffff88039f87a0d0 00000000000141c0 ffff88041e3141c0 ffff88039f87a0d0
 ffff88039b863900 0000000000000002 ffffffff810fdda8 ffff88039b863870
Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff810fe757>] lock_page+0x19/0x1c
 [<ffffffff810fe7b3>] find_lock_entry+0x33/0x55
 [<ffffffff810fe7e3>] find_lock_page+0xe/0x1b
 [<ffffffff810feccc>] find_or_create_page+0x31/0x83
 [<ffffffff81260ce7>] io_ctl_prepare_pages+0x49/0x11c
 [<ffffffff81262897>] __load_free_space_cache+0x1be/0x56c
 [<ffffffff81262d2b>] load_free_space_cache+0xe6/0x199
 [<ffffffff810765ed>] ? get_parent_ip+0xd/0x3c
 [<ffffffff81217d49>] cache_block_group+0x1c4/0x343
 [<ffffffff8108489d>] ? finish_wait+0x65/0x65
 [<ffffffff8121c7ed>] find_free_extent+0x391/0x89e
 [<ffffffff8121ce7a>] btrfs_reserve_extent+0x70/0x114
 [<ffffffff81232765>] cow_file_range+0x1b0/0x388
 [<ffffffff81233414>] submit_compressed_extents+0x102/0x40f
 [<ffffffff81152ae3>] ? delete_object_full+0x29/0x2c
 [<ffffffff81231910>] ? async_cow_free+0x24/0x27
 [<ffffffff812337a7>] async_cow_submit+0x86/0x8b
 [<ffffffff812524cd>] normal_work_helper+0x194/0x240
 [<ffffffff81065e29>] process_one_work+0x195/0x2d2
 [<ffffffff810660cb>] worker_thread+0x136/0x205
 [<ffffffff81065f95>] ? process_scheduled_works+0x2f/0x2f
 [<ffffffff8106b564>] kthread+0xae/0xb6
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61
 [<ffffffff81628d7c>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61
kworker/u16:11  D 0000000000000000     0 15419      2 0x00000080
Workqueue: writeback bdi_writeback_workfn (flush-btrfs-1)
 ffff8800b3823a00 0000000000000046 ffff8800b38239d0 ffff8800b3823fd8
 ffff8800b6f60110 00000000000141c0 ffff88041e3541c0 ffff8800b6f60110
 ffff8800b3823aa0 0000000000000002 ffffffff810fdda8 ffff8800b3823a10
Call Trace:
 [<ffffffff810fdda8>] ? wait_on_page_read+0x3c/0x3c
 [<ffffffff8161fa5e>] schedule+0x73/0x75
 [<ffffffff8161fc03>] io_schedule+0x60/0x7a
 [<ffffffff810fddb6>] sleep_on_page+0xe/0x12
 [<ffffffff8161ff93>] __wait_on_bit_lock+0x46/0x8a
 [<ffffffff810fde71>] __lock_page+0x69/0x6b
 [<ffffffff810848d1>] ? autoremove_wake_function+0x34/0x34
 [<ffffffff81242aaf>] lock_page+0x1e/0x21
 [<ffffffff812465ba>] extent_write_cache_pages.isra.16.constprop.32+0x10e/0x2c6
 [<ffffffff81246a18>] extent_writepages+0x4b/0x5c
 [<ffffffff81230d08>] ? btrfs_submit_direct+0x3f4/0x3f4
 [<ffffffff81625a8c>] ? preempt_count_add+0x77/0x8d
 [<ffffffff8122f2e3>] btrfs_writepages+0x28/0x2a
 [<ffffffff8110873d>] do_writepages+0x1e/0x2c
 [<ffffffff81177d7f>] __writeback_single_inode+0x7d/0x238
 [<ffffffff81178ab7>] writeback_sb_inodes+0x1eb/0x339
 [<ffffffff81178c79>] __writeback_inodes_wb+0x74/0xb7
 [<ffffffff81178df4>] wb_writeback+0x138/0x293
 [<ffffffff8117942c>] bdi_writeback_workfn+0x19a/0x329
 [<ffffffff8100d047>] ? load_TLS+0xb/0xf
 [<ffffffff81065e29>] process_one_work+0x195/0x2d2
 [<ffffffff810660cb>] worker_thread+0x136/0x205
 [<ffffffff81065f95>] ? process_scheduled_works+0x2f/0x2f
 [<ffffffff8106b564>] kthread+0xae/0xb6
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61
 [<ffffffff81628d7c>] ret_from_fork+0x7c/0xb0
 [<ffffffff8106b4b6>] ? __kthread_parkme+0x61/0x61

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

I didn't see any mention of the btrfs utility version in this thread (I may be blind though).

My server was suffering from frequent panics upon scrub / defrag / balance, until I updated the btrfs utility. That resolved all my issues.

My server has two BTRFS filesystems of ~25 TiB each, and 4 GB of RAM (15% used). A hardware upgrade is imminent; just waiting for the backup to complete.

~~
Andrew E. Mileski

Mmmh, look what I found in my quota output before I turned it off. Many many unknown subvolumes.

If I were to guess, when I rotate snapshots and delete them, their quota does not, and a lot of crap stays behind. Not sure if that is causing the memory leak, but that can't be good.
Hopefully quota disable will have deleted all that cruft and will stop the leak.

gargamel:/mnt/btrfs_pool2# btrfs-quota.py .
subvol                                          group         total    unshared
-------------------------------------------------------------------------------
(unknown)                                       0/5           0.00G       0.00G
backup                                          0/257        -0.00G       0.00G
Soft                                            0/258        59.27G       0.00G
Win                                             0/259       112.60G       0.00G
backup/debian32                                 0/262      -166.13G      -8.72G
backup/debian64                                 0/263       644.62G       0.00G
backup/ubuntu                                   0/264       326.59G      -0.49G
backup-test                                     0/265         0.00G       0.00G
backup/0Notmachines                             0/266       268.16G      -0.13G
backup/1Appliances                              0/566         2.09G       0.00G
backup/win                                      0/570       461.61G       0.00G
(unknown)                                       0/1039        0.00G       0.00G
(unknown)                                       0/1041        0.00G       0.00G
(unknown)                                       0/1044        0.00G       0.00G
(unknown)                                       0/1045        0.00G       0.00G
(unknown)                                       0/1046        0.00G       0.00G
(unknown)                                       0/1047        0.00G       0.00G
(unknown)                                       0/1048        0.00G       0.00G
(unknown)                                       0/1049        0.00G       0.00G
(unknown)                                       0/1050        0.00G       0.00G
(unknown)                                       0/1051        0.00G       0.00G
(unknown)                                       0/1052        0.00G       0.00G
(unknown)                                       0/1053        0.00G       0.00G
(unknown)                                       0/1054      300.95G       0.00G
(unknown)                                       0/1055      300.95G       0.00G
(unknown)                                       0/1056        2.09G       0.00G
(unknown)                                       0/1057        2.09G       0.00G
(unknown)                                       0/1058      461.61G       0.00G
(unknown)                                       0/1059      461.61G       0.00G
(unknown)                                       0/1518        0.00G      -0.00G
(unknown)                                       0/1519        0.00G       0.00G
(unknown)                                       0/1520       59.26G       0.00G
(unknown)                                       0/1521       59.27G       0.00G
(unknown)                                       0/1522      112.60G       0.00G
(unknown)                                       0/1523      112.60G       0.00G
(unknown)                                       0/1524        0.00G       0.00G
(unknown)                                       0/1525        0.00G       0.00G
(unknown)                                       0/1526      278.62G      -2.52G
(unknown)                                       0/1527      282.31G       0.00G

(1600 lines of unknown snipped)

legolas/tmp_ggm_daily_ro.20140527_10:03:17 0/4125 0.22G 0.00G
legolas/tmp_ggm_daily_ro.20140527_10:03:17_daily_20140528_00:03:01 0/4258 0.22G 0.00G

legolas/var_ggm_daily_ro.20140529_10:19:37      0/4565      297.75G       0.00G
legolas/var_ggm_daily_ro.20140530_10:10:24      0/4599      301.64G       0.01G
legolas/var_ggm_daily_ro.20140603_10:22:18      0/4710      303.93G       0.94G
legolas/var_ggm_daily_ro.20140605_10:20:02      0/4764      304.09G       0.00G
legolas/var_ggm_daily_ro.20140606_10:23:59      0/4792      305.97G       0.41G
legolas/tmp_ggm_daily_ro.20140611_10:40:08      0/4938        0.29G       0.01G
legolas/var_ggm_daily_ro.20140613_10:30:23      0/4997      288.92G       0.00G
legolas/var_ggm_daily_ro.20140614_10:26:37      0/5030      289.02G       0.00G
legolas/var_ggm_daily_rw.20140614_10:26:37      0/5032      289.02G       0.00G
backup_weekly_20140615_00:04:01                 0/5051       -0.00G       0.00G
Soft_weekly_20140615_00:04:01                   0/5052       59.27G       0.00G
Win_weekly_20140615_00:04:01                    0/5053      112.60G       0.00G
backup/debian32_weekly_20140615_00:04:01        0/5054       77.47G       0.44G
backup/debian64_weekly_20140615_00:04:01        0/5055      787.45G       0.00G
backup/ubuntu_weekly_20140615_00:04:01          0/5056      340.51G       0.11G
backup/0Notmachines_weekly_20140615_00:04:01    0/5057      271.30G       0.15G
backup/1Appliances_weekly_20140615_00:04:01     0/5058        2.09G       0.00G
backup/win_weekly_20140615_00:04:01             0/5059      461.61G       0.00G
legolas/var_ggm_daily_ro.20140615_11:17:36      0/5068      288.89G       0.00G
legolas/var_ggm_daily_rw.20140615_11:17:36      0/5069      288.89G       0.00G
legolas/var_ggm_daily_ro.20140616_11:59:11      0/5095      288.28G       0.00G
legolas/var_ggm_daily_rw.20140616_11:59:11      0/5096      288.28G       0.00G
legolas/var_ggm_daily_rw.20140617_10:26:39      0/5124      288.42G       0.00G
legolas/var_ggm_daily_ro.20140618_10:27:06      0/5151      288.97G       1.05G
legolas/var_ggm_daily_ro.20140619_10:18:57      0/5178      289.15G       1.26G
legolas/var_ggm_daily_ro.20140621_11:00:21      0/5229      288.12G       0.30G
backup_weekly_20140622_00:04:01                 0/5248       -0.00G       0.00G
Soft_weekly_20140622_00:04:01                   0/5249       59.27G       0.00G
Win_weekly_20140622_00:04:01                    0/5250      112.60G       0.00G
backup/debian32_weekly_20140622_00:04:01        0/5251       20.62G       0.18G
backup/debian64_weekly_20140622_00:04:01        0/5252      716.66G       0.00G
backup/ubuntu_weekly_20140622_00:04:01          0/5253      336.38G       0.04G
backup/0Notmachines_weekly_20140622_00:04:01    0/5254      270.47G       0.01G
backup/1Appliances_weekly_20140622_00:04:01     0/5255        2.09G       0.00G
backup/win_weekly_20140622_00:04:01             0/5256      461.61G       0.00G
legolas/var_ggm_daily_ro.20140622_10:16:05      0/5265      287.50G       0.20G
legolas/var_ggm_daily_ro.20140624_10:17:10      0/5310      299.05G       0.00G
legolas/var_ggm_daily_rw.20140624_10:17:10      0/5311      299.05G       0.00G
legolas/var_ggm_daily_ro.20140625_10:21:06      0/5343      299.50G       1.46G
legolas/var_ggm_daily_ro.20140627_12:14:51      0/5382      295.13G       3.17G
legolas/var_ggm_daily_ro.20140628_11:16:05      0/5420      299.93G       2.05G
backup_weekly_20140629_00:04:01                 0/5438       -0.00G       0.00G
Soft_weekly_20140629_00:04:01                   0/5440       59.27G       0.00G
Win_weekly_20140629_00:04:01                    0/5442      112.60G       0.00G
backup/debian32_weekly_20140629_00:04:01        0/5445      -33.15G       0.77G
backup/debian64_weekly_20140629_00:04:01        0/5446      709.35G       0.00G
backup/ubuntu_weekly_20140629_00:04:01          0/5447      333.13G       0.26G
backup/0Notmachines_weekly_20140629_00:04:01    0/5448      269.65G       0.00G
backup/1Appliances_weekly_20140629_00:04:01     0/5449        2.09G       0.00G
backup/win_weekly_20140629_00:04:01             0/5450      461.61G       0.00G
legolas/var_ggm_daily_ro.20140629_10:28:56      0/5459      300.07G       2.30G
legolas/home_ggm_daily_ro.20140630_10:02:02     0/5478       77.76G       0.00G
legolas/var_ggm_daily_ro.20140630_10:19:42      0/5486      300.08G       2.31G
legolas/home_ggm_daily_rw.20140701_10:02:05     0/5506       78.30G       0.00G
legolas/var_ggm_daily_ro.20140701_10:49:22      0/5513      299.19G       0.40G
legolas/home_ggm_daily_ro.20140702_10:02:06     0/5532       78.38G       0.00G
legolas/home_ggm_daily_rw.20140702_10:02:06     0/5533       78.38G       0.00G
legolas/root_ggm_daily_ro.20140702_10:20:09     0/5534        1.66G       0.00G
legolas/root_ggm_daily_rw.20140702_10:20:09     0/5535        1.66G       0.00G
legolas/tmp_ggm_daily_ro.20140702_10:20:50      0/5536        0.08G       0.00G
legolas/tmp_ggm_daily_rw.20140702_10:20:50      0/5537        0.08G       0.00G
legolas/usr_ggm_daily_ro.20140702_10:21:11      0/5538       10.36G       0.00G
legolas/usr_ggm_daily_rw.20140702_10:21:11      0/5539       10.36G       0.00G
legolas/var_ggm_daily_ro.20140702_10:21:31      0/5540      299.19G       0.41G
backup_hourly_20140703_00:03:01                 0/5541       -0.00G       0.00G
backup_daily_20140703_00:03:01                  0/5542       -0.00G       0.00G
Soft_hourly_20140703_00:03:01                   0/5543       59.27G       0.00G
Soft_daily_20140703_00:03:01                    0/5544       59.27G       0.00G
Win_hourly_20140703_00:03:01                    0/5545      112.60G       0.00G
Win_daily_20140703_00:03:01                     0/5546      112.60G       0.00G
backup/debian32_hourly_20140703_00:03:01        0/5547     -103.86G       0.00G
backup/debian32_daily_20140703_00:03:01         0/5548     -103.86G       0.00G
backup/debian64_daily_20140703_00:03:01         0/5549      643.32G       0.00G
backup/debian64_hourly_20140703_00:03:01        0/5550      643.32G       0.00G
backup/ubuntu_hourly_20140703_00:03:01          0/5551      328.48G       0.00G
backup/ubuntu_daily_20140703_00:03:01           0/5552      328.48G       0.00G
backup/0Notmachines_hourly_20140703_00:03:01    0/5553      268.91G       0.00G
backup/0Notmachines_daily_20140703_00:03:01     0/5554      268.91G       0.00G
backup/1Appliances_hourly_20140703_00:03:01     0/5555        2.09G       0.00G
backup/1Appliances_daily_20140703_00:03:01      0/5556        2.09G       0.00G
backup/win_hourly_20140703_00:03:01             0/5557      461.61G       0.00G
backup/win_daily_20140703_00:03:01              0/5558      461.61G       0.00G
legolas/home_ggm_daily_ro.20140703_10:02:03     0/5559       78.41G       0.00G
legolas/home_ggm_daily_rw.20140703_10:02:03     0/5560       78.41G       0.00G
legolas/root_ggm_daily_ro.20140703_10:19:21     0/5561        1.66G       0.00G
legolas/root_ggm_daily_rw.20140703_10:19:21     0/5562        1.66G       0.00G
legolas/tmp_ggm_daily_ro.20140703_10:20:12      0/5563        0.08G       0.00G
legolas/tmp_ggm_daily_rw.20140703_10:20:12      0/5564        0.08G       0.00G
legolas/usr_ggm_daily_ro.20140703_10:21:08      0/5565       10.36G       0.00G
legolas/usr_ggm_daily_rw.20140703_10:21:08      0/5566       10.36G       0.00G
legolas/var_ggm_daily_ro.20140703_10:21:56      0/5567      301.30G       4.01G
backup_hourly_20140704_00:03:01                 0/5568       -0.00G       0.00G
backup_daily_20140704_00:03:01                  0/5569       -0.00G       0.00G
Soft_daily_20140704_00:03:01                    0/5570       59.27G       0.00G
Soft_hourly_20140704_00:03:01                   0/5571       59.27G       0.00G
Win_daily_20140704_00:03:01                     0/5572      112.60G       0.00G
Win_hourly_20140704_00:03:01                    0/5573      112.60G       0.00G
backup/debian32_daily_20140704_00:03:01         0/5574     -101.81G       0.00G
backup/debian32_hourly_20140704_00:03:01        0/5575     -101.81G       0.00G
backup/debian64_hourly_20140704_00:03:01        0/5576      643.32G       0.00G
backup/debian64_daily_20140704_00:03:01         0/5577      643.32G       0.00G
backup/ubuntu_daily_20140704_00:03:01           0/5578      328.48G       0.00G
backup/ubuntu_hourly_20140704_00:03:01          0/5579      328.48G       0.00G
backup/0Notmachines_daily_20140704_00:03:01     0/5580      268.90G       0.00G
backup/0Notmachines_hourly_20140704_00:03:01    0/5581      268.90G       0.00G
backup/1Appliances_hourly_20140704_00:03:01     0/5582        2.09G       0.00G
backup/1Appliances_daily_20140704_00:03:01      0/5583        2.09G       0.00G
backup/win_hourly_20140704_00:03:01             0/5584      461.61G       0.00G
backup/win_daily_20140704_00:03:01              0/5585      461.61G       0.00G
legolas/home_ggm_daily_ro.20140704_10:02:03     0/5586       77.52G       0.00G
legolas/home_ggm_daily_rw.20140704_10:02:03     0/5587       77.52G       0.00G
legolas/root_ggm_daily_ro.20140704_10:16:59     0/5588        1.66G       0.00G
legolas/root_ggm_daily_rw.20140704_10:16:59     0/5589        1.66G       0.00G
legolas/tmp_ggm_daily_ro.20140704_10:17:07      0/5590        0.08G       0.00G
legolas/tmp_ggm_daily_rw.20140704_10:17:07      0/5591        0.08G       0.00G
legolas/usr_ggm_daily_ro.20140704_10:17:24      0/5592       10.36G       0.00G
legolas/usr_ggm_daily_rw.20140704_10:17:24      0/5593       10.36G       0.00G
legolas/var_ggm_daily_ro.20140704_10:17:39      0/5594      304.28G      19.49G
backup_hourly_20140705_00:03:00                 0/5595       -0.00G       0.00G
backup_daily_20140705_00:03:00                  0/5596       -0.00G       0.00G
Soft_daily_20140705_00:03:00                    0/5597       59.27G       0.00G
Soft_hourly_20140705_00:03:00                   0/5598       59.27G       0.00G
Win_hourly_20140705_00:03:00                    0/5599      112.60G       0.00G
Win_daily_20140705_00:03:00                     0/5600      112.60G       0.00G
backup/debian32_daily_20140705_00:03:00         0/5601     -118.33G       0.00G
backup/debian32_hourly_20140705_00:03:00        0/5602     -118.33G       0.00G
backup/debian64_daily_20140705_00:03:00         0/5603      643.76G       0.00G
backup/debian64_hourly_20140705_00:03:00        0/5604      643.76G       0.00G
backup/ubuntu_daily_20140705_00:03:00           0/5605      328.12G       0.00G
backup/ubuntu_hourly_20140705_00:03:00          0/5606      328.12G       0.00G
backup/0Notmachines_daily_20140705_00:03:00     0/5607      268.78G       0.00G
backup/0Notmachines_hourly_20140705_00:03:00    0/5608      268.78G       0.00G
backup/1Appliances_daily_20140705_00:03:00      0/5609        2.09G       0.00G
backup/1Appliances_hourly_20140705_00:03:00     0/5610        2.09G       0.00G
backup/win_daily_20140705_00:03:00              0/5611      461.61G       0.00G
backup/win_hourly_20140705_00:03:00             0/5612      461.61G       0.00G
legolas/home_ggm_daily_ro.20140705_10:02:09     0/5613       77.54G       0.10G
legolas/root_ggm_daily_ro.20140705_14:07:48     0/5614        1.66G       0.00G
legolas/root_ggm_daily_rw.20140705_14:07:48     0/5615        1.66G       0.00G
legolas/tmp_ggm_daily_ro.20140705_14:08:41      0/5616        0.08G       0.00G
legolas/tmp_ggm_daily_rw.20140705_14:08:41      0/5617        0.08G       0.00G
legolas/usr_ggm_daily_ro.20140705_14:09:13      0/5618       10.36G       0.00G
legolas/usr_ggm_daily_rw.20140705_14:09:13      0/5619       10.36G       0.00G
legolas/var_ggm_daily_ro.20140705_14:10:09      0/5620      299.50G       0.79G
backup_daily_20140706_00:03:01                  0/5621       -0.00G       0.00G
backup_hourly_20140706_00:03:01                 0/5622       -0.00G       0.00G
Soft_daily_20140706_00:03:01                    0/5623       59.27G       0.00G
Soft_hourly_20140706_00:03:01                   0/5624       59.27G       0.00G
Win_daily_20140706_00:03:01                     0/5625      112.60G       0.00G
Win_hourly_20140706_00:03:01                    0/5626      112.60G       0.00G
backup/debian32_daily_20140706_00:03:01         0/5627     -129.94G       0.00G
backup/debian32_hourly_20140706_00:03:01        0/5628     -129.94G       0.00G
backup/debian64_daily_20140706_00:03:01         0/5629      644.62G       0.00G
backup/debian64_hourly_20140706_00:03:01        0/5630      644.62G       0.00G
backup/ubuntu_hourly_20140706_00:03:01          0/5631      328.12G       0.00G
backup/ubuntu_daily_20140706_00:03:01           0/5632      328.12G       0.00G
backup/0Notmachines_daily_20140706_00:03:01     0/5633      268.65G       0.00G
backup/0Notmachines_hourly_20140706_00:03:01    0/5634      268.65G       0.00G
backup/1Appliances_daily_20140706_00:03:01      0/5635        2.09G       0.00G
backup/1Appliances_hourly_20140706_00:03:01     0/5636        2.09G       0.00G
backup/win_daily_20140706_00:03:01              0/5637      461.61G       0.00G
backup/win_hourly_20140706_00:03:01             0/5638      461.61G       0.00G
backup_weekly_20140706_00:04:01                 0/5639       -0.00G       0.00G
Soft_weekly_20140706_00:04:01                   0/5640       59.27G       0.00G
Win_weekly_20140706_00:04:01                    0/5641      112.60G       0.00G
backup/debian32_weekly_20140706_00:04:01        0/5642     -129.94G       0.00G
backup/debian64_weekly_20140706_00:04:01        0/5643      644.62G       0.00G
backup/ubuntu_weekly_20140706_00:04:01          0/5644      328.12G       0.00G
backup/0Notmachines_weekly_20140706_00:04:01    0/5645      268.65G       0.00G
backup/1Appliances_weekly_20140706_00:04:01     0/5646        2.09G       0.00G
backup/win_weekly_20140706_00:04:01             0/5647      461.61G       0.00G
legolas/home_ggm_daily_ro.20140706_10:02:01     0/5648       77.53G       0.00G
legolas/home_ggm_daily_rw.20140706_10:02:01     0/5649       77.53G       0.00G
legolas/root_ggm_daily_ro.20140706_11:08:26     0/5650        1.67G       0.00G
legolas/root_ggm_daily_rw.20140706_11:08:26     0/5651        1.67G       0.00G
legolas/tmp_ggm_daily_ro.20140706_11:09:31      0/5652        0.08G       0.00G
legolas/tmp_ggm_daily_rw.20140706_11:09:31      0/5653        0.08G       0.00G
legolas/usr_ggm_daily_ro.20140706_11:10:34      0/5654       10.36G       0.00G
legolas/usr_ggm_daily_rw.20140706_11:10:34      0/5655       10.36G       0.00G
legolas/var_ggm_daily_ro.20140706_11:11:30      0/5656      301.21G       4.53G
gargamel:/mnt/btrfs_pool2# btrfs quota disable .

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

-------- Original Message --------
Subject: Re: btrfs quotas related to OOM death problems on my 8GB server with both 3.15.1 and 3.14?
From: Marc MERLIN <[email protected]>
To: Andrew E. Mileski <[email protected]>, Duncan <[email protected]>, Russell Coker <[email protected]>, Satoru Takeuchi <[email protected]>, [email protected]
Date: 2014年07月13日 23:45

This seems to be a known issue.

When you enable quota and create a subvolume, a qgroup(0/<subvol id>) will be created and bind to the newly created subvolume. But on the other hand, when you delete the subvolume, the qgroup will *not* be deleted automatically.
So you need to remove the qgroup manually.

More info can be found in the btrfs wiki:
https://btrfs.wiki.kernel.org/index.php/Quota_support#Known_issues

Thanks,
Qu

This was updated since I last looked at it, thanks for pointing that out.

So there must be a leak that is related to how many qtrees you have. Because even if I had 1600 qtrees, that shouldn't slowly eat up 6GB of RAM, correct?

That said, I'll try the memleak detector you pointed me to, and read up on how it's supposed to work.

Thanks for your help,
Marc

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901