Skip to contentSkip to conversation

Heavy memory leak when using quota groups

DISCUSSION Tomasz Chmielewski 4 participants 7 messages Aug 5, 2013 – Aug 8, 2013 Btrfs file system

lore
Share & tools
lore

Tomasz Chmielewski

author
#0 lore raw
Quoted 6 lines

I am trying to use qgroups
functionality & with a basic random-write workload, it constantly keeps leaking memory & within few minutes of IO, there is either out-of-memory killer trying to kill some tasks or there are page-allocation failures that btrfs or other kernel module experiences.

FYI, I just saw something similar with 3.10 on a server with 32 GB RAM:

The result was a frozen server and a need to hard reset.

Aug  4 21:51:14 bkp010 kernel: [661831.436751] btrfs-endio-wri: page allocation failure: order:0, mode:0x284020
Aug  4 21:51:14 bkp010 kernel: [661831.436805] CPU: 3 PID: 19598 Comm: btrfs-endio-wri Not tainted 3.10.0 #3
Aug  4 21:51:14 bkp010 kernel: [661831.436853] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug  4 21:51:14 bkp010 kernel: [661831.436943]  0000000000284020 ffff88047e3fd518 ffffffff81371348 ffff88047e3fd5a8
Aug  4 21:51:14 bkp010 kernel: [661831.437033]  ffffffff810a5629 0000000000000000 ffffea0002adab20 ffff88081fad4a70
Aug  4 21:51:14 bkp010 kernel: [661831.437123]  ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Aug  4 21:51:14 bkp010 kernel: [661831.437213] Call Trace:
Aug  4 21:51:14 bkp010 kernel: [661831.437263]  [<ffffffff81371348>] dump_stack+0x19/0x1b
Aug  4 21:51:14 bkp010 kernel: [661831.437312]  [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
Aug  4 21:51:14 bkp010 kernel: [661831.437360]  [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
Aug  4 21:51:14 bkp010 kernel: [661831.437419]  [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
Aug  4 21:51:14 bkp010 kernel: [661831.437477]  [<ffffffff810d3fb9>] new_slab+0xb5/0x269
Aug  4 21:51:14 bkp010 kernel: [661831.437525]  [<ffffffff813701b2>] __slab_alloc+0x30e/0x493
Aug  4 21:51:14 bkp010 kernel: [661831.437583]  [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.437637]  [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.437686]  [<ffffffff810d5ddb>] kmem_cache_alloc+0x95/0x110
Aug  4 21:51:14 bkp010 kernel: [661831.437737]  [<ffffffffa071b6a6>] __alloc_extent_buffer+0x25/0xd8 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.437769] swapper/7: page allocation failure: order:0, mode:0x20
Aug  4 21:51:14 bkp010 kernel: [661831.437770] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
Aug  4 21:51:14 bkp010 kernel: [661831.437771] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug  4 21:51:14 bkp010 kernel: [661831.437772]  0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
Aug  4 21:51:14 bkp010 kernel: [661831.437774]  ffffffff810a5629 ffff880400000000 0000000000011780 ffff88081fbc3bd8
Aug  4 21:51:14 bkp010 kernel: [661831.437775]  ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Aug  4 21:51:14 bkp010 kernel: [661831.437775] Call Trace:
Aug  4 21:51:14 bkp010 kernel: [661831.437778]  <IRQ>  [<ffffffff81371348>] dump_stack+0x19/0x1b
Aug  4 21:51:14 bkp010 kernel: [661831.437780]  [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
Aug  4 21:51:14 bkp010 kernel: [661831.437782]  [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
Aug  4 21:51:14 bkp010 kernel: [661831.437785]  [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
Aug  4 21:51:14 bkp010 kernel: [661831.437788]  [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3
Aug  4 21:51:14 bkp010 kernel: [661831.437789]  [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb
Aug  4 21:51:14 bkp010 kernel: [661831.437798]  [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169]
Aug  4 21:51:14 bkp010 kernel: [661831.437800]  [<ffffffff812d47f2>] net_rx_action+0xa3/0x182
Aug  4 21:51:14 bkp010 kernel: [661831.437803]  [<ffffffff81034c55>] __do_softirq+0xb7/0x16d
Aug  4 21:51:14 bkp010 kernel: [661831.437804]  [<ffffffff81034dde>] irq_exit+0x3e/0x83
Aug  4 21:51:14 bkp010 kernel: [661831.437807]  [<ffffffff81003c35>] do_IRQ+0x98/0xaf
Aug  4 21:51:14 bkp010 kernel: [661831.437809]  [<ffffffff813750ea>] common_interrupt+0x6a/0x6a
Aug  4 21:51:14 bkp010 kernel: [661831.437811]  <EOI>  [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac
Aug  4 21:51:14 bkp010 kernel: [661831.437812]  [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac
Aug  4 21:51:14 bkp010 kernel: [661831.437813]  [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f
Aug  4 21:51:14 bkp010 kernel: [661831.437815]  [<ffffffff810097de>] arch_cpu_idle+0x9/0x18
Aug  4 21:51:14 bkp010 kernel: [661831.437816]  [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104
Aug  4 21:51:14 bkp010 kernel: [661831.437819]  [<ffffffff8136581f>] start_secondary+0x1af/0x1b3
Aug  4 21:51:14 bkp010 kernel: [661831.437819] Mem-Info:
Aug  4 21:51:14 bkp010 kernel: [661831.437820] Node 0 DMA per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.437821] CPU    0: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437821] CPU    1: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437822] CPU    2: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437822] CPU    3: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437823] CPU    4: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437823] CPU    5: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437824] CPU    6: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437825] CPU    7: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437825] Node 0 DMA32 per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.437826] CPU    0: hi:  186, btch:  31 usd: 176
Aug  4 21:51:14 bkp010 kernel: [661831.437826] CPU    1: hi:  186, btch:  31 usd:  37
Aug  4 21:51:14 bkp010 kernel: [661831.437827] CPU    2: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437827] CPU    3: hi:  186, btch:  31 usd:  41
Aug  4 21:51:14 bkp010 kernel: [661831.437828] CPU    4: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437828] CPU    5: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437829] CPU    6: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437830] CPU    7: hi:  186, btch:  31 usd: 172
Aug  4 21:51:14 bkp010 kernel: [661831.437830] Node 0 Normal per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.437831] CPU    0: hi:  186, btch:  31 usd: 180
Aug  4 21:51:14 bkp010 kernel: [661831.437832] CPU    1: hi:  186, btch:  31 usd:  64
Aug  4 21:51:14 bkp010 kernel: [661831.437833] CPU    2: hi:  186, btch:  31 usd:  60
Aug  4 21:51:14 bkp010 kernel: [661831.437833] CPU    3: hi:  186, btch:  31 usd:  84
Aug  4 21:51:14 bkp010 kernel: [661831.437834] CPU    4: hi:  186, btch:  31 usd:  90
Aug  4 21:51:14 bkp010 kernel: [661831.437835] CPU    5: hi:  186, btch:  31 usd:  66
Aug  4 21:51:14 bkp010 kernel: [661831.437835] CPU    6: hi:  186, btch:  31 usd:  20
Aug  4 21:51:14 bkp010 kernel: [661831.437836] CPU    7: hi:  186, btch:  31 usd: 209
Aug  4 21:51:14 bkp010 kernel: [661831.437839] active_anon:49535 inactive_anon:56858 isolated_anon:0
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  active_file:12576 inactive_file:14674 isolated_file:0
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  unevictable:0 dirty:4574 writeback:17253 unstable:0
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  free:34679 slab_reclaimable:1154786 slab_unreclaimable:18894
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  mapped:1456 shmem:169 pagetables:3293 bounce:0
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  free_cma:0
Aug  4 21:51:14 bkp010 kernel: [661831.437843] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug  4 21:51:14 bkp010 kernel: [661831.437845] lowmem_reserve[]: 0 3174 31846 31846
Aug  4 21:51:14 bkp010 kernel: [661831.437849] Node 0 DMA32 free:115440kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:68 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.437850] lowmem_reserve[]: 0 0 28672 28672
Aug  4 21:51:14 bkp010 kernel: [661831.437853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69012kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:101 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.437854] lowmem_reserve[]: 0 0 0 0
Aug  4 21:51:14 bkp010 kernel: [661831.437859] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Aug  4 21:51:14 bkp010 kernel: [661831.437863] Node 0 DMA32: 28844*4kB (EM) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115408kB
Aug  4 21:51:14 bkp010 kernel: [661831.437867] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB
Aug  4 21:51:14 bkp010 kernel: [661831.437868] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug  4 21:51:14 bkp010 kernel: [661831.437869] 44196 total pagecache pages
Aug  4 21:51:14 bkp010 kernel: [661831.437870] 16756 pages in swap cache
Aug  4 21:51:14 bkp010 kernel: [661831.437870] Swap cache stats: add 828307, delete 811551, find 615437/665659
Aug  4 21:51:14 bkp010 kernel: [661831.437870] Free swap  = 16634240kB
Aug  4 21:51:14 bkp010 kernel: [661831.437871] Total swap = 16776116kB
Aug  4 21:51:14 bkp010 kernel: [661831.443483]  [<ffffffffa072128d>] btrfs_clone_extent_buffer+0x3a/0xc3 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443574]  [<ffffffffa06ebefb>] btrfs_search_old_slot+0x680/0x78b [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443626]  [<ffffffffa06edd71>] btrfs_next_old_leaf+0xc9/0x3a6 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443680]  [<ffffffffa074da50>] __resolve_indirect_refs+0x331/0x467 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443772]  [<ffffffffa074e205>] find_parent_nodes+0x357/0x5be [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443825]  [<ffffffffa074e4e8>] btrfs_find_all_roots+0x7c/0xd7 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443874]  [<ffffffff810d58b9>] ? kmem_cache_free+0x164/0x17a
Aug  4 21:51:14 bkp010 kernel: [661831.443925]  [<ffffffffa07514bd>] btrfs_qgroup_account_ref+0x10c/0x44f [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444018]  [<ffffffffa071c49f>] ? clear_state_bit+0x106/0x115 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444069]  [<ffffffffa06f316f>] btrfs_delayed_refs_qgroup_accounting+0xab/0xdf [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444163]  [<ffffffffa0706896>] __btrfs_end_transaction+0x49/0x281 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444215]  [<ffffffffa06f4514>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444307]  [<ffffffffa0706af9>] btrfs_end_transaction+0xb/0xd [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444360]  [<ffffffffa0711224>] btrfs_finish_ordered_io+0x683/0x84e [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444448]  [<ffffffff810a3683>] ? mempool_free_slab+0x12/0x14
Aug  4 21:51:14 bkp010 kernel: [661831.444496]  [<ffffffff810a3717>] ? mempool_free+0x72/0x7a
Aug  4 21:51:14 bkp010 kernel: [661831.444547]  [<ffffffffa073c357>] ? end_compressed_bio_write+0xc0/0xc8 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444639]  [<ffffffffa07113ff>] finish_ordered_fn+0x10/0x12 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444692]  [<ffffffffa072ab5c>] worker_loop+0x15e/0x48e [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444744]  [<ffffffffa072a9fe>] ? btrfs_queue_worker+0x267/0x267 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444793]  [<ffffffff81048ab2>] kthread+0xb5/0xbd
Aug  4 21:51:14 bkp010 kernel: [661831.444840]  [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43
Aug  4 21:51:14 bkp010 kernel: [661831.444889]  [<ffffffff8137576c>] ret_from_fork+0x7c/0xb0
Aug  4 21:51:14 bkp010 kernel: [661831.444936]  [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43
Aug  4 21:51:14 bkp010 kernel: [661831.444984] Mem-Info:
Aug  4 21:51:14 bkp010 kernel: [661831.445026] Node 0 DMA per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.445070] CPU    0: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445115] CPU    1: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445161] CPU    2: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445206] CPU    3: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445235] swapper/7: page allocation failure: order:0, mode:0x20
Aug  4 21:51:14 bkp010 kernel: [661831.445237] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
Aug  4 21:51:14 bkp010 kernel: [661831.445237] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug  4 21:51:14 bkp010 kernel: [661831.445238]  0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
Aug  4 21:51:14 bkp010 kernel: [661831.445240]  ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8
Aug  4 21:51:14 bkp010 kernel: [661831.445241]  ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Aug  4 21:51:14 bkp010 kernel: [661831.445241] Call Trace:
Aug  4 21:51:14 bkp010 kernel: [661831.445244]  <IRQ>  [<ffffffff81371348>] dump_stack+0x19/0x1b
Aug  4 21:51:14 bkp010 kernel: [661831.445245]  [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
Aug  4 21:51:14 bkp010 kernel: [661831.445247]  [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
Aug  4 21:51:14 bkp010 kernel: [661831.445250]  [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
Aug  4 21:51:14 bkp010 kernel: [661831.445252]  [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3
Aug  4 21:51:14 bkp010 kernel: [661831.445254]  [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb
Aug  4 21:51:14 bkp010 kernel: [661831.445260]  [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169]
Aug  4 21:51:14 bkp010 kernel: [661831.445261]  [<ffffffff812d47f2>] net_rx_action+0xa3/0x182
Aug  4 21:51:14 bkp010 kernel: [661831.445264]  [<ffffffff81034c55>] __do_softirq+0xb7/0x16d
Aug  4 21:51:14 bkp010 kernel: [661831.445265]  [<ffffffff81034dde>] irq_exit+0x3e/0x83
Aug  4 21:51:14 bkp010 kernel: [661831.445267]  [<ffffffff81003c35>] do_IRQ+0x98/0xaf
Aug  4 21:51:14 bkp010 kernel: [661831.445268]  [<ffffffff813750ea>] common_interrupt+0x6a/0x6a
Aug  4 21:51:14 bkp010 kernel: [661831.445270]  <EOI>  [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac
Aug  4 21:51:14 bkp010 kernel: [661831.445271]  [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac
Aug  4 21:51:14 bkp010 kernel: [661831.445272]  [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f
Aug  4 21:51:14 bkp010 kernel: [661831.445274]  [<ffffffff810097de>] arch_cpu_idle+0x9/0x18
Aug  4 21:51:14 bkp010 kernel: [661831.445275]  [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104
Aug  4 21:51:14 bkp010 kernel: [661831.445277]  [<ffffffff8136581f>] start_secondary+0x1af/0x1b3
Aug  4 21:51:14 bkp010 kernel: [661831.445278] Mem-Info:
Aug  4 21:51:14 bkp010 kernel: [661831.445278] Node 0 DMA per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.445279] CPU    0: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445279] CPU    1: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445280] CPU    2: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445280] CPU    3: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445280] CPU    4: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445281] CPU    5: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445281] CPU    6: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445282] CPU    7: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445282] Node 0 DMA32 per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.445283] CPU    0: hi:  186, btch:  31 usd: 165
Aug  4 21:51:14 bkp010 kernel: [661831.445284] CPU    1: hi:  186, btch:  31 usd:  37
Aug  4 21:51:14 bkp010 kernel: [661831.445284] CPU    2: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445285] CPU    3: hi:  186, btch:  31 usd:  41
Aug  4 21:51:14 bkp010 kernel: [661831.445285] CPU    4: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445286] CPU    5: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445286] CPU    6: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445287] CPU    7: hi:  186, btch:  31 usd: 172
Aug  4 21:51:14 bkp010 kernel: [661831.445287] Node 0 Normal per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.445288] CPU    0: hi:  186, btch:  31 usd: 181
Aug  4 21:51:14 bkp010 kernel: [661831.445288] CPU    1: hi:  186, btch:  31 usd:  64
Aug  4 21:51:14 bkp010 kernel: [661831.445289] CPU    2: hi:  186, btch:  31 usd:  60
Aug  4 21:51:14 bkp010 kernel: [661831.445289] CPU    3: hi:  186, btch:  31 usd:  84
Aug  4 21:51:14 bkp010 kernel: [661831.445290] CPU    4: hi:  186, btch:  31 usd:  90
Aug  4 21:51:14 bkp010 kernel: [661831.445290] CPU    5: hi:  186, btch:  31 usd:  66
Aug  4 21:51:14 bkp010 kernel: [661831.445291] CPU    6: hi:  186, btch:  31 usd:  20
Aug  4 21:51:14 bkp010 kernel: [661831.445291] CPU    7: hi:  186, btch:  31 usd: 209
Aug  4 21:51:14 bkp010 kernel: [661831.445293] active_anon:49535 inactive_anon:56858 isolated_anon:0
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  active_file:12576 inactive_file:14654 isolated_file:0
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  unevictable:0 dirty:4574 writeback:17362 unstable:0
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  mapped:1456 shmem:169 pagetables:3293 bounce:0
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  free_cma:0
Aug  4 21:51:14 bkp010 kernel: [661831.445298] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug  4 21:51:14 bkp010 kernel: [661831.445299] lowmem_reserve[]: 0 3174 31846 31846
Aug  4 21:51:14 bkp010 kernel: [661831.445303] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.445305] lowmem_reserve[]: 0 0 28672 28672
Aug  4 21:51:14 bkp010 kernel: [661831.445309] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.445310] lowmem_reserve[]: 0 0 0 0
Aug  4 21:51:14 bkp010 kernel: [661831.445316] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Aug  4 21:51:14 bkp010 kernel: [661831.445320] Node 0 DMA32: 28841*4kB (UEM) 15*8kB (UEM) 1*16kB (M) 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115532kB
Aug  4 21:51:14 bkp010 kernel: [661831.445324] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB
Aug  4 21:51:14 bkp010 kernel: [661831.445325] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug  4 21:51:14 bkp010 kernel: [661831.445325] 44279 total pagecache pages
Aug  4 21:51:14 bkp010 kernel: [661831.445326] 16820 pages in swap cache
Aug  4 21:51:14 bkp010 kernel: [661831.445326] Swap cache stats: add 828373, delete 811553, find 615437/665659
Aug  4 21:51:14 bkp010 kernel: [661831.445327] Free swap  = 16633976kB
Aug  4 21:51:14 bkp010 kernel: [661831.445327] Total swap = 16776116kB
Aug  4 21:51:14 bkp010 kernel: [661831.450878] CPU    4: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.450924] CPU    5: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.450969] CPU    6: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451014] CPU    7: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451059] Node 0 DMA32 per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.451104] CPU    0: hi:  186, btch:  31 usd: 165
Aug  4 21:51:14 bkp010 kernel: [661831.451149] CPU    1: hi:  186, btch:  31 usd:  37
Aug  4 21:51:14 bkp010 kernel: [661831.451194] CPU    2: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451240] CPU    3: hi:  186, btch:  31 usd:  41
Aug  4 21:51:14 bkp010 kernel: [661831.451285] CPU    4: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451330] CPU    5: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451376] CPU    6: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451421] CPU    7: hi:  186, btch:  31 usd: 172
Aug  4 21:51:14 bkp010 kernel: [661831.451466] Node 0 Normal per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.451510] CPU    0: hi:  186, btch:  31 usd: 181
Aug  4 21:51:14 bkp010 kernel: [661831.451556] CPU    1: hi:  186, btch:  31 usd:  64
Aug  4 21:51:14 bkp010 kernel: [661831.451601] CPU    2: hi:  186, btch:  31 usd:  60
Aug  4 21:51:14 bkp010 kernel: [661831.451647] CPU    3: hi:  186, btch:  31 usd:  84
Aug  4 21:51:14 bkp010 kernel: [661831.451692] CPU    4: hi:  186, btch:  31 usd:  90
Aug  4 21:51:14 bkp010 kernel: [661831.451737] CPU    5: hi:  186, btch:  31 usd:  66
Aug  4 21:51:14 bkp010 kernel: [661831.451783] CPU    6: hi:  186, btch:  31 usd:  20
Aug  4 21:51:14 bkp010 kernel: [661831.451828] CPU    7: hi:  186, btch:  31 usd: 209
Aug  4 21:51:14 bkp010 kernel: [661831.451875] active_anon:49535 inactive_anon:56858 isolated_anon:0
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  active_file:12576 inactive_file:14654 isolated_file:0
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  unevictable:0 dirty:4574 writeback:17362 unstable:0
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  mapped:1456 shmem:169 pagetables:3293 bounce:0
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  free_cma:0
Aug  4 21:51:14 bkp010 kernel: [661831.452149] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug  4 21:51:14 bkp010 kernel: [661831.452432] lowmem_reserve[]: 0 3174 31846 31846
Aug  4 21:51:14 bkp010 kernel: [661831.452479] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.452806] lowmem_reserve[]: 0 0 28672 28672
Aug  4 21:51:14 bkp010 kernel: [661831.452853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.453182] lowmem_reserve[]: 0 0 0 0
Aug  4 21:51:14 bkp010 kernel: [661831.453230] swapper/7: page allocation failure: order:0, mode:0x20
Aug  4 21:51:14 bkp010 kernel: [661831.453231] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
Aug  4 21:51:14 bkp010 kernel: [661831.453231] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug  4 21:51:14 bkp010 kernel: [661831.453233]  0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
Aug  4 21:51:14 bkp010 kernel: [661831.453235]  ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8
Aug  4 21:51:14 bkp010 kernel: [661831.453236]  ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#1 lore raw

Hello.

Quoted 8 lines
Quoted 6 lines

I am trying to use qgroups
functionality & with a basic random-write workload, it constantly keeps leaking memory & within few minutes of IO, there is either out-of-memory killer trying to kill some tasks or there are page-allocation failures that btrfs or other kernel module experiences.

FYI, I just saw something similar with 3.10 on a server with 32 GB RAM:

Would please show how many qgroups are there in the system.

btrfs qgroup show <mnt>

Quoted 247 lines

The result was a frozen server and a need to hard reset.

Aug  4 21:51:14 bkp010 kernel: [661831.436751] btrfs-endio-wri: page allocation failure: order:0, mode:0x284020
Aug  4 21:51:14 bkp010 kernel: [661831.436805] CPU: 3 PID: 19598 Comm: btrfs-endio-wri Not tainted 3.10.0 #3
Aug  4 21:51:14 bkp010 kernel: [661831.436853] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug  4 21:51:14 bkp010 kernel: [661831.436943]  0000000000284020 ffff88047e3fd518 ffffffff81371348 ffff88047e3fd5a8
Aug  4 21:51:14 bkp010 kernel: [661831.437033]  ffffffff810a5629 0000000000000000 ffffea0002adab20 ffff88081fad4a70
Aug  4 21:51:14 bkp010 kernel: [661831.437123]  ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Aug  4 21:51:14 bkp010 kernel: [661831.437213] Call Trace:
Aug  4 21:51:14 bkp010 kernel: [661831.437263]  [<ffffffff81371348>] dump_stack+0x19/0x1b
Aug  4 21:51:14 bkp010 kernel: [661831.437312]  [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
Aug  4 21:51:14 bkp010 kernel: [661831.437360]  [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
Aug  4 21:51:14 bkp010 kernel: [661831.437419]  [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
Aug  4 21:51:14 bkp010 kernel: [661831.437477]  [<ffffffff810d3fb9>] new_slab+0xb5/0x269
Aug  4 21:51:14 bkp010 kernel: [661831.437525]  [<ffffffff813701b2>] __slab_alloc+0x30e/0x493
Aug  4 21:51:14 bkp010 kernel: [661831.437583]  [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.437637]  [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.437686]  [<ffffffff810d5ddb>] kmem_cache_alloc+0x95/0x110
Aug  4 21:51:14 bkp010 kernel: [661831.437737]  [<ffffffffa071b6a6>] __alloc_extent_buffer+0x25/0xd8 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.437769] swapper/7: page allocation failure: order:0, mode:0x20
Aug  4 21:51:14 bkp010 kernel: [661831.437770] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
Aug  4 21:51:14 bkp010 kernel: [661831.437771] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug  4 21:51:14 bkp010 kernel: [661831.437772]  0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
Aug  4 21:51:14 bkp010 kernel: [661831.437774]  ffffffff810a5629 ffff880400000000 0000000000011780 ffff88081fbc3bd8
Aug  4 21:51:14 bkp010 kernel: [661831.437775]  ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Aug  4 21:51:14 bkp010 kernel: [661831.437775] Call Trace:
Aug  4 21:51:14 bkp010 kernel: [661831.437778]  <IRQ>  [<ffffffff81371348>] dump_stack+0x19/0x1b
Aug  4 21:51:14 bkp010 kernel: [661831.437780]  [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
Aug  4 21:51:14 bkp010 kernel: [661831.437782]  [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
Aug  4 21:51:14 bkp010 kernel: [661831.437785]  [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
Aug  4 21:51:14 bkp010 kernel: [661831.437788]  [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3
Aug  4 21:51:14 bkp010 kernel: [661831.437789]  [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb
Aug  4 21:51:14 bkp010 kernel: [661831.437798]  [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169]
Aug  4 21:51:14 bkp010 kernel: [661831.437800]  [<ffffffff812d47f2>] net_rx_action+0xa3/0x182
Aug  4 21:51:14 bkp010 kernel: [661831.437803]  [<ffffffff81034c55>] __do_softirq+0xb7/0x16d
Aug  4 21:51:14 bkp010 kernel: [661831.437804]  [<ffffffff81034dde>] irq_exit+0x3e/0x83
Aug  4 21:51:14 bkp010 kernel: [661831.437807]  [<ffffffff81003c35>] do_IRQ+0x98/0xaf
Aug  4 21:51:14 bkp010 kernel: [661831.437809]  [<ffffffff813750ea>] common_interrupt+0x6a/0x6a
Aug  4 21:51:14 bkp010 kernel: [661831.437811]  <EOI>  [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac
Aug  4 21:51:14 bkp010 kernel: [661831.437812]  [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac
Aug  4 21:51:14 bkp010 kernel: [661831.437813]  [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f
Aug  4 21:51:14 bkp010 kernel: [661831.437815]  [<ffffffff810097de>] arch_cpu_idle+0x9/0x18
Aug  4 21:51:14 bkp010 kernel: [661831.437816]  [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104
Aug  4 21:51:14 bkp010 kernel: [661831.437819]  [<ffffffff8136581f>] start_secondary+0x1af/0x1b3
Aug  4 21:51:14 bkp010 kernel: [661831.437819] Mem-Info:
Aug  4 21:51:14 bkp010 kernel: [661831.437820] Node 0 DMA per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.437821] CPU    0: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437821] CPU    1: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437822] CPU    2: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437822] CPU    3: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437823] CPU    4: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437823] CPU    5: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437824] CPU    6: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437825] CPU    7: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437825] Node 0 DMA32 per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.437826] CPU    0: hi:  186, btch:  31 usd: 176
Aug  4 21:51:14 bkp010 kernel: [661831.437826] CPU    1: hi:  186, btch:  31 usd:  37
Aug  4 21:51:14 bkp010 kernel: [661831.437827] CPU    2: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437827] CPU    3: hi:  186, btch:  31 usd:  41
Aug  4 21:51:14 bkp010 kernel: [661831.437828] CPU    4: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437828] CPU    5: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437829] CPU    6: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.437830] CPU    7: hi:  186, btch:  31 usd: 172
Aug  4 21:51:14 bkp010 kernel: [661831.437830] Node 0 Normal per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.437831] CPU    0: hi:  186, btch:  31 usd: 180
Aug  4 21:51:14 bkp010 kernel: [661831.437832] CPU    1: hi:  186, btch:  31 usd:  64
Aug  4 21:51:14 bkp010 kernel: [661831.437833] CPU    2: hi:  186, btch:  31 usd:  60
Aug  4 21:51:14 bkp010 kernel: [661831.437833] CPU    3: hi:  186, btch:  31 usd:  84
Aug  4 21:51:14 bkp010 kernel: [661831.437834] CPU    4: hi:  186, btch:  31 usd:  90
Aug  4 21:51:14 bkp010 kernel: [661831.437835] CPU    5: hi:  186, btch:  31 usd:  66
Aug  4 21:51:14 bkp010 kernel: [661831.437835] CPU    6: hi:  186, btch:  31 usd:  20
Aug  4 21:51:14 bkp010 kernel: [661831.437836] CPU    7: hi:  186, btch:  31 usd: 209
Aug  4 21:51:14 bkp010 kernel: [661831.437839] active_anon:49535 inactive_anon:56858 isolated_anon:0
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  active_file:12576 inactive_file:14674 isolated_file:0
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  unevictable:0 dirty:4574 writeback:17253 unstable:0
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  free:34679 slab_reclaimable:1154786 slab_unreclaimable:18894
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  mapped:1456 shmem:169 pagetables:3293 bounce:0
Aug  4 21:51:14 bkp010 kernel: [661831.437839]  free_cma:0
Aug  4 21:51:14 bkp010 kernel: [661831.437843] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug  4 21:51:14 bkp010 kernel: [661831.437845] lowmem_reserve[]: 0 3174 31846 31846
Aug  4 21:51:14 bkp010 kernel: [661831.437849] Node 0 DMA32 free:115440kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:68 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.437850] lowmem_reserve[]: 0 0 28672 28672
Aug  4 21:51:14 bkp010 kernel: [661831.437853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69012kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:101 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.437854] lowmem_reserve[]: 0 0 0 0
Aug  4 21:51:14 bkp010 kernel: [661831.437859] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Aug  4 21:51:14 bkp010 kernel: [661831.437863] Node 0 DMA32: 28844*4kB (EM) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115408kB
Aug  4 21:51:14 bkp010 kernel: [661831.437867] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB
Aug  4 21:51:14 bkp010 kernel: [661831.437868] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug  4 21:51:14 bkp010 kernel: [661831.437869] 44196 total pagecache pages
Aug  4 21:51:14 bkp010 kernel: [661831.437870] 16756 pages in swap cache
Aug  4 21:51:14 bkp010 kernel: [661831.437870] Swap cache stats: add 828307, delete 811551, find 615437/665659
Aug  4 21:51:14 bkp010 kernel: [661831.437870] Free swap  = 16634240kB
Aug  4 21:51:14 bkp010 kernel: [661831.437871] Total swap = 16776116kB
Aug  4 21:51:14 bkp010 kernel: [661831.443483]  [<ffffffffa072128d>] btrfs_clone_extent_buffer+0x3a/0xc3 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443574]  [<ffffffffa06ebefb>] btrfs_search_old_slot+0x680/0x78b [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443626]  [<ffffffffa06edd71>] btrfs_next_old_leaf+0xc9/0x3a6 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443680]  [<ffffffffa074da50>] __resolve_indirect_refs+0x331/0x467 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443772]  [<ffffffffa074e205>] find_parent_nodes+0x357/0x5be [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443825]  [<ffffffffa074e4e8>] btrfs_find_all_roots+0x7c/0xd7 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.443874]  [<ffffffff810d58b9>] ? kmem_cache_free+0x164/0x17a
Aug  4 21:51:14 bkp010 kernel: [661831.443925]  [<ffffffffa07514bd>] btrfs_qgroup_account_ref+0x10c/0x44f [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444018]  [<ffffffffa071c49f>] ? clear_state_bit+0x106/0x115 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444069]  [<ffffffffa06f316f>] btrfs_delayed_refs_qgroup_accounting+0xab/0xdf [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444163]  [<ffffffffa0706896>] __btrfs_end_transaction+0x49/0x281 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444215]  [<ffffffffa06f4514>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444307]  [<ffffffffa0706af9>] btrfs_end_transaction+0xb/0xd [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444360]  [<ffffffffa0711224>] btrfs_finish_ordered_io+0x683/0x84e [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444448]  [<ffffffff810a3683>] ? mempool_free_slab+0x12/0x14
Aug  4 21:51:14 bkp010 kernel: [661831.444496]  [<ffffffff810a3717>] ? mempool_free+0x72/0x7a
Aug  4 21:51:14 bkp010 kernel: [661831.444547]  [<ffffffffa073c357>] ? end_compressed_bio_write+0xc0/0xc8 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444639]  [<ffffffffa07113ff>] finish_ordered_fn+0x10/0x12 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444692]  [<ffffffffa072ab5c>] worker_loop+0x15e/0x48e [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444744]  [<ffffffffa072a9fe>] ? btrfs_queue_worker+0x267/0x267 [btrfs]
Aug  4 21:51:14 bkp010 kernel: [661831.444793]  [<ffffffff81048ab2>] kthread+0xb5/0xbd
Aug  4 21:51:14 bkp010 kernel: [661831.444840]  [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43
Aug  4 21:51:14 bkp010 kernel: [661831.444889]  [<ffffffff8137576c>] ret_from_fork+0x7c/0xb0
Aug  4 21:51:14 bkp010 kernel: [661831.444936]  [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43
Aug  4 21:51:14 bkp010 kernel: [661831.444984] Mem-Info:
Aug  4 21:51:14 bkp010 kernel: [661831.445026] Node 0 DMA per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.445070] CPU    0: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445115] CPU    1: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445161] CPU    2: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445206] CPU    3: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445235] swapper/7: page allocation failure: order:0, mode:0x20
Aug  4 21:51:14 bkp010 kernel: [661831.445237] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
Aug  4 21:51:14 bkp010 kernel: [661831.445237] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug  4 21:51:14 bkp010 kernel: [661831.445238]  0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
Aug  4 21:51:14 bkp010 kernel: [661831.445240]  ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8
Aug  4 21:51:14 bkp010 kernel: [661831.445241]  ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Aug  4 21:51:14 bkp010 kernel: [661831.445241] Call Trace:
Aug  4 21:51:14 bkp010 kernel: [661831.445244]  <IRQ>  [<ffffffff81371348>] dump_stack+0x19/0x1b
Aug  4 21:51:14 bkp010 kernel: [661831.445245]  [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124
Aug  4 21:51:14 bkp010 kernel: [661831.445247]  [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793
Aug  4 21:51:14 bkp010 kernel: [661831.445250]  [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5
Aug  4 21:51:14 bkp010 kernel: [661831.445252]  [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3
Aug  4 21:51:14 bkp010 kernel: [661831.445254]  [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb
Aug  4 21:51:14 bkp010 kernel: [661831.445260]  [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169]
Aug  4 21:51:14 bkp010 kernel: [661831.445261]  [<ffffffff812d47f2>] net_rx_action+0xa3/0x182
Aug  4 21:51:14 bkp010 kernel: [661831.445264]  [<ffffffff81034c55>] __do_softirq+0xb7/0x16d
Aug  4 21:51:14 bkp010 kernel: [661831.445265]  [<ffffffff81034dde>] irq_exit+0x3e/0x83
Aug  4 21:51:14 bkp010 kernel: [661831.445267]  [<ffffffff81003c35>] do_IRQ+0x98/0xaf
Aug  4 21:51:14 bkp010 kernel: [661831.445268]  [<ffffffff813750ea>] common_interrupt+0x6a/0x6a
Aug  4 21:51:14 bkp010 kernel: [661831.445270]  <EOI>  [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac
Aug  4 21:51:14 bkp010 kernel: [661831.445271]  [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac
Aug  4 21:51:14 bkp010 kernel: [661831.445272]  [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f
Aug  4 21:51:14 bkp010 kernel: [661831.445274]  [<ffffffff810097de>] arch_cpu_idle+0x9/0x18
Aug  4 21:51:14 bkp010 kernel: [661831.445275]  [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104
Aug  4 21:51:14 bkp010 kernel: [661831.445277]  [<ffffffff8136581f>] start_secondary+0x1af/0x1b3
Aug  4 21:51:14 bkp010 kernel: [661831.445278] Mem-Info:
Aug  4 21:51:14 bkp010 kernel: [661831.445278] Node 0 DMA per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.445279] CPU    0: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445279] CPU    1: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445280] CPU    2: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445280] CPU    3: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445280] CPU    4: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445281] CPU    5: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445281] CPU    6: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445282] CPU    7: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445282] Node 0 DMA32 per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.445283] CPU    0: hi:  186, btch:  31 usd: 165
Aug  4 21:51:14 bkp010 kernel: [661831.445284] CPU    1: hi:  186, btch:  31 usd:  37
Aug  4 21:51:14 bkp010 kernel: [661831.445284] CPU    2: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445285] CPU    3: hi:  186, btch:  31 usd:  41
Aug  4 21:51:14 bkp010 kernel: [661831.445285] CPU    4: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445286] CPU    5: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445286] CPU    6: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.445287] CPU    7: hi:  186, btch:  31 usd: 172
Aug  4 21:51:14 bkp010 kernel: [661831.445287] Node 0 Normal per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.445288] CPU    0: hi:  186, btch:  31 usd: 181
Aug  4 21:51:14 bkp010 kernel: [661831.445288] CPU    1: hi:  186, btch:  31 usd:  64
Aug  4 21:51:14 bkp010 kernel: [661831.445289] CPU    2: hi:  186, btch:  31 usd:  60
Aug  4 21:51:14 bkp010 kernel: [661831.445289] CPU    3: hi:  186, btch:  31 usd:  84
Aug  4 21:51:14 bkp010 kernel: [661831.445290] CPU    4: hi:  186, btch:  31 usd:  90
Aug  4 21:51:14 bkp010 kernel: [661831.445290] CPU    5: hi:  186, btch:  31 usd:  66
Aug  4 21:51:14 bkp010 kernel: [661831.445291] CPU    6: hi:  186, btch:  31 usd:  20
Aug  4 21:51:14 bkp010 kernel: [661831.445291] CPU    7: hi:  186, btch:  31 usd: 209
Aug  4 21:51:14 bkp010 kernel: [661831.445293] active_anon:49535 inactive_anon:56858 isolated_anon:0
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  active_file:12576 inactive_file:14654 isolated_file:0
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  unevictable:0 dirty:4574 writeback:17362 unstable:0
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  mapped:1456 shmem:169 pagetables:3293 bounce:0
Aug  4 21:51:14 bkp010 kernel: [661831.445293]  free_cma:0
Aug  4 21:51:14 bkp010 kernel: [661831.445298] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug  4 21:51:14 bkp010 kernel: [661831.445299] lowmem_reserve[]: 0 3174 31846 31846
Aug  4 21:51:14 bkp010 kernel: [661831.445303] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.445305] lowmem_reserve[]: 0 0 28672 28672
Aug  4 21:51:14 bkp010 kernel: [661831.445309] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.445310] lowmem_reserve[]: 0 0 0 0
Aug  4 21:51:14 bkp010 kernel: [661831.445316] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Aug  4 21:51:14 bkp010 kernel: [661831.445320] Node 0 DMA32: 28841*4kB (UEM) 15*8kB (UEM) 1*16kB (M) 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115532kB
Aug  4 21:51:14 bkp010 kernel: [661831.445324] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB
Aug  4 21:51:14 bkp010 kernel: [661831.445325] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug  4 21:51:14 bkp010 kernel: [661831.445325] 44279 total pagecache pages
Aug  4 21:51:14 bkp010 kernel: [661831.445326] 16820 pages in swap cache
Aug  4 21:51:14 bkp010 kernel: [661831.445326] Swap cache stats: add 828373, delete 811553, find 615437/665659
Aug  4 21:51:14 bkp010 kernel: [661831.445327] Free swap  = 16633976kB
Aug  4 21:51:14 bkp010 kernel: [661831.445327] Total swap = 16776116kB
Aug  4 21:51:14 bkp010 kernel: [661831.450878] CPU    4: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.450924] CPU    5: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.450969] CPU    6: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451014] CPU    7: hi:    0, btch:   1 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451059] Node 0 DMA32 per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.451104] CPU    0: hi:  186, btch:  31 usd: 165
Aug  4 21:51:14 bkp010 kernel: [661831.451149] CPU    1: hi:  186, btch:  31 usd:  37
Aug  4 21:51:14 bkp010 kernel: [661831.451194] CPU    2: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451240] CPU    3: hi:  186, btch:  31 usd:  41
Aug  4 21:51:14 bkp010 kernel: [661831.451285] CPU    4: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451330] CPU    5: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451376] CPU    6: hi:  186, btch:  31 usd:   0
Aug  4 21:51:14 bkp010 kernel: [661831.451421] CPU    7: hi:  186, btch:  31 usd: 172
Aug  4 21:51:14 bkp010 kernel: [661831.451466] Node 0 Normal per-cpu:
Aug  4 21:51:14 bkp010 kernel: [661831.451510] CPU    0: hi:  186, btch:  31 usd: 181
Aug  4 21:51:14 bkp010 kernel: [661831.451556] CPU    1: hi:  186, btch:  31 usd:  64
Aug  4 21:51:14 bkp010 kernel: [661831.451601] CPU    2: hi:  186, btch:  31 usd:  60
Aug  4 21:51:14 bkp010 kernel: [661831.451647] CPU    3: hi:  186, btch:  31 usd:  84
Aug  4 21:51:14 bkp010 kernel: [661831.451692] CPU    4: hi:  186, btch:  31 usd:  90
Aug  4 21:51:14 bkp010 kernel: [661831.451737] CPU    5: hi:  186, btch:  31 usd:  66
Aug  4 21:51:14 bkp010 kernel: [661831.451783] CPU    6: hi:  186, btch:  31 usd:  20
Aug  4 21:51:14 bkp010 kernel: [661831.451828] CPU    7: hi:  186, btch:  31 usd: 209
Aug  4 21:51:14 bkp010 kernel: [661831.451875] active_anon:49535 inactive_anon:56858 isolated_anon:0
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  active_file:12576 inactive_file:14654 isolated_file:0
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  unevictable:0 dirty:4574 writeback:17362 unstable:0
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  mapped:1456 shmem:169 pagetables:3293 bounce:0
Aug  4 21:51:14 bkp010 kernel: [661831.451875]  free_cma:0
Aug  4 21:51:14 bkp010 kernel: [661831.452149] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug  4 21:51:14 bkp010 kernel: [661831.452432] lowmem_reserve[]: 0 3174 31846 31846
Aug  4 21:51:14 bkp010 kernel: [661831.452479] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.452806] lowmem_reserve[]: 0 0 28672 28672
Aug  4 21:51:14 bkp010 kernel: [661831.452853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no
Aug  4 21:51:14 bkp010 kernel: [661831.453182] lowmem_reserve[]: 0 0 0 0
Aug  4 21:51:14 bkp010 kernel: [661831.453230] swapper/7: page allocation failure: order:0, mode:0x20
Aug  4 21:51:14 bkp010 kernel: [661831.453231] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3
Aug  4 21:51:14 bkp010 kernel: [661831.453231] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
Aug  4 21:51:14 bkp010 kernel: [661831.453233]  0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18
Aug  4 21:51:14 bkp010 kernel: [661831.453235]  ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8
Aug  4 21:51:14 bkp010 kernel: [661831.453236]  ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000
Signature · 6 lines
Tomasz Chmielewski
http://wpkg.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
author
#2 lore raw
Quoted 9 lines · On Tue, 6 Aug 2013 07:29:03 +0800 Wang Shilong <[email protected]>…

Hello.

Quoted 7 lines
Quoted 6 lines

I am trying to use qgroups
functionality & with a basic random-write workload, it constantly keeps leaking memory & within few minutes of IO, there is either out-of-memory killer trying to kill some tasks or there are page-allocation failures that btrfs or other kernel module experiences.

Quoted 6 lines
Quoted 2 lines

FYI, I just saw something similar with 3.10 on a server with 32 GB RAM:

Would please show how many qgroups are there in the system.

btrfs qgroup show <mnt>

There are two different btrfs filesystems here; notice there are quite a bit of qgroups with negative numbers.

# btrfs qgroup show /mnt/lxc1
0/258 2068160512 2068160512
0/1601 0 0
0/2359 1361963708416 1361963708416
0/3870 12996984832 4096
0/3871 12936032256 275820544
0/3872 40094982144 -3129344
0/3873 1238417408 -286720
0/3900 40061427712 2592768
0/3901 0 4096
0/3902 -4096 0
0/3903 895385600 4096
0/3904 -8192 -4096
0/3905 40075735040 3133440
0/3906 -4096 0
0/3908 891162624 114688
0/3909 40076042240 372736
0/3910 -4096 0
0/3911 891224064 139264
0/3912 40076103680 708608
0/3913 7718686720 4096
0/3918 7631826944 3203072
0/3922 891256832 278528
0/3923 5142913024 4096
0/3924 0 0
0/3928 4530425856 21856256
0/3973 891744256 4096
0/3974 854753280 4096
0/3975 41452056576 4096
0/3976 72792285184 4096
0/3977 43131879424 4096
0/3978 882757632 0
0/3980 882757632 0
0/3981 852537344 0
0/3983 2410848256 4096
0/3984 852537344 0
0/3985 109482221568 4096
0/3987 -4096 0
0/3989 34457268224 4096
0/3990 -4096 0
0/4005 2454089728 0
0/4009 2454089728 0
0/4014 10458886144 4096
0/4020 1543729152 4096
0/4022 1015640064 4096
0/4039 55234445312 4096
0/4059 1014833152 0
0/4061 1014833152 0
0/4074 1457680384 0
0/4077 1457680384 0
0/4083 24318590976 0
0/4087 24318590976 0
0/4162 10207858688 0
0/4164 10207858688 0
0/4243 1598795776 1347584
0/4245 1598795776 0
0/4277 43304882176 0
0/4280 43304882176 0
0/4328 0 0
0/4339 108909649920 0
0/4341 108909649920 0
0/4342 72641134592 0
0/4343 72641134592 0
0/4346 54204522496 0
0/4347 54204522496 0
0/4348 5137539072 2932736
0/4349 7714144256 729088
0/4350 894935040 618496
0/4351 12996780032 210608128
0/4352 40108421120 8347648
0/4353 41457135616 99024896
0/4354 852701184 17911808
0/4355 888950784 23416832
0/4356 72641462272 45056
0/4357 43208957952 54571008
0/4358 2400776192 41017344
0/4359 109428551680 117022720
0/4360 24396820480 11505664
0/4361 10275590144 12472320
0/4362 1543102464 22564864
0/4363 1014951936 425984
0/4364 54707920896 183185408
0/4365 5142007808 5021696
0/4366 7714271232 774144
0/4367 895250432 733184
0/4368 78995476480 4096
0/4369 12999450624 209784832
0/4370 40116916224 7626752
0/4371 41466302464 11272192
0/4372 853426176 18030592
0/4373 889573376 23425024
0/4374 72641757184 57344
0/4375 79529275392 1884889088
0/4376 43121913856 58900480
0/4377 2397597696 37494784
0/4378 109448671232 149102592
0/4379 24440803328 27103232
0/4380 10342871040 8065024
0/4381 1543413760 557056
0/4382 1015025664 466944
0/4383 54857736192 180170752
0/4384 77708091392 10326016
0/4385 5139664896 4096
0/4386 5139664896 4096
0/4387 7718678528 4096
0/4388 7718678528 4096
0/4389 895197184 4096
0/4390 895197184 4096
0/4391 12951941120 4096
0/4392 12951941120 4096
0/4393 40108793856 4096
0/4394 40108793856 4096
0/4396 41437810688 4096
0/4397 41437810688 4096
0/4398 854634496 4096
0/4399 854634496 4096
0/4400 890630144 4096
0/4401 890630144 4096
0/4402 72791916544 4096
0/4403 72791916544 4096
0/4404 43130384384 4096
0/4405 43130384384 4096
0/4406 2395500544 4096
0/4407 2395500544 4096
0/4408 109446926336 4096
0/4409 109446926336 4096
0/4410 24507891712 4096
0/4411 24507891712 4096
0/4412 10401660928 4096
0/4413 10401660928 4096
0/4414 1543028736 4096
0/4415 1543028736 4096
0/4416 1014980608 4096
0/4417 1014980608 4096
0/4418 54934056960 4096
0/4419 54934056960 4096
0/4420 5142913024 4096
0/4421 7718686720 4096
0/4422 895385600 4096
0/4423 12996984832 4096
0/4424 40112762880 4096
0/4425 41452056576 4096
0/4426 854753280 4096
0/4427 891744256 4096
0/4428 72792285184 4096
0/4429 43131879424 4096
0/4430 2410848256 4096
0/4431 109482221568 4096
0/4432 34457268224 4096
0/4433 10458886144 4096
0/4434 1543729152 4096
0/4435 1015640064 4096
0/4436 55234445312 4096
0/4437 78995476480 4096
# btrfs qgroup show /mnt/lxc2
0/260 161639960576 161639960576
0/261 181149696 181149696
0/262 155557888 1007616
0/377 180408320 25899008
0/378 336346464256 336346464256
0/535 572301312 417751040
0/536 70203207680 70203207680
0/642 247451648 92934144
0/1276 631365632 4096
0/1277 38598176768 0
0/1278 603922432 -12591104
0/1279 38268366848 -21348352
0/1284 38299160576 0
0/1285 38299160576 0
0/1286 607039488 -12697600
0/1287 620396544 241664
0/1293 38301650944 21417984
0/1294 623255552 13103104
0/1303 38449602560 10883072
0/1304 624873472 4096
0/1305 624873472 4096
0/1307 38449627136 106496
0/1319 38450397184 94208
0/1321 628367360 192512
0/1322 0 0
0/1323 0 0
0/1324 0 -1051197440
0/1325 1051197440 0
0/1326 1073741824 0
0/1331 38597541888 61440
0/1332 630042624 118784
0/1335 38598545408 4096
0/1336 38598545408 4096
0/1337 631840768 4096
0/1338 631840768 4096
1/1 375578464256 375578464256
2/1 -4096 22732800
Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#3 lore raw
In reply to Tomasz Chmielewski
Quoted 11 lines · Tomasz Chmielewski
Quoted 6 lines

I am trying to use qgroups
functionality & with a basic random-write workload, it constantly keeps leaking memory & within few minutes of IO, there is either out-of-memory killer trying to kill some tasks or there are page-allocation failures that btrfs or other kernel module experiences.

FYI, I just saw something similar with 3.10 on a server with 32 GB RAM:

The result was a frozen server and a need to hard reset.

What do I have to do to reproduce it here? How do you generate the load? What is the disk setup, what the qgroups setup?

Thanks,
Arne

author
#4 lore raw
Quoted 8 lines · On Tue, 06 Aug 2013 07:24:26 +0200 Arne Jansen <[email protected]> wrote:
Quoted 7 lines · Tomasz Chmielewski
Quoted 6 lines

I am trying to use qgroups
functionality & with a basic random-write workload, it constantly keeps leaking memory & within few minutes of IO, there is either out-of-memory killer trying to kill some tasks or there are page-allocation failures that btrfs or other kernel module experiences.

Quoted 8 lines
Quoted 5 lines

FYI, I just saw something similar with 3.10 on a server with 32 GB RAM:

The result was a frozen server and a need to hard reset.

What do I have to do to reproduce it here? How do you generate the load? What is the disk setup, what the qgroups setup?

Unfortunately I don't have a way to reproduce, as the issue happened to me only once.

Server uptime was about 2 weeks, and there were millions of files with multiple hardlinks on disk (backuppc archive copied from ext4 to btrfs). Then, "rm" command was running for many days, about 5 times in parallel, and many other rsync commands, with snapshots being added and removed.

When I saw the issue I looked up in btrfs list archives that there was a similar report in the past - but I doubt I will be able to find a reliable way to reproduce this.

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
author
#5 lore raw
Quoted 5 lines · On Wed, 7 Aug 2013 00:42:27 +0800 Tomasz Chmielewski <[email protected]> wrote:
Quoted 2 lines

What do I have to do to reproduce it here? How do you generate the load? What is the disk setup, what the qgroups setup?

Unfortunately I don't have a way to reproduce, as the issue happened to me only once.

I keep removing the files, and memory usage has grown to 28 GB (out of 32 GB).

Is there anything I can debug at this point?

# free
             total       used       free     shared    buffers     cached
Mem:      32638512   32476992     161520          0        732    3531152
-/+ buffers/cache:   28945108    3693404
Swap:     16776116     173392   16602724
# cat /proc/meminfo
MemTotal:       32638512 kB
MemFree:          163364 kB
Buffers:             864 kB
Cached:          3531940 kB
SwapCached:        60992 kB
Active:          3599804 kB
Inactive:        2422176 kB
Active(anon):    1860412 kB
Inactive(anon):   629848 kB
Active(file):    1739392 kB
Inactive(file):  1792328 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      16776116 kB
SwapFree:       16602916 kB
Dirty:             85432 kB
Writeback:         10320 kB
AnonPages:       2458792 kB
Mapped:            11068 kB
Shmem:               796 kB
Slab:            3168820 kB
SReclaimable:    3099388 kB
SUnreclaim:        69432 kB
KernelStack:        3408 kB
PageTables:        14764 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    33095372 kB
Committed_AS:    4301988 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      130036 kB
VmallocChunk:   34359595900 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       12520 kB
DirectMap2M:    33220608 kB
Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#6 lore raw

I'm pretty I encountered the same leak over here, 15 days uptime and I had consistent out of memory problems, I got angry and did go in single user mode and noticed that with only systemd->screen->bash(2) I had 2 GB of memory used on 4 GB total. This box mostly has random reads and just before the out of memory condition had 12 GB written to a lot of files.

# btrfs qgroup show /mnt/btrfs
0/337 13953818624 9494970368
0/338 0 0
0/353 20934205440 20934205440
0/373 36420952064 36420952064
0/668 0 19161088
0/669 21279420416 16839733248
0/890 -97652736 -97652736
0/907 65536 65536
0/910 0 0

Screenshot of top/processes/memory: http://i39.tinypic.com/2rrrjtl.jpg

Quoted 71 lines · Tomasz Chmielewski
Quoted 5 lines · On Wed, 7 Aug 2013 00:42:27 +0800 Tomasz Chmielewski <[email protected]> wrote:
Quoted 2 lines

What do I have to do to reproduce it here? How do you generate the load? What is the disk setup, what the qgroups setup?

Unfortunately I don't have a way to reproduce, as the issue happened to me only once.

I keep removing the files, and memory usage has grown to 28 GB (out of 32 GB).

Is there anything I can debug at this point?

# free
             total       used       free     shared    buffers     cached
Mem:      32638512   32476992     161520          0        732 3531152
-/+ buffers/cache:   28945108    3693404
Swap:     16776116     173392   16602724
# cat /proc/meminfo
MemTotal:       32638512 kB
MemFree:          163364 kB
Buffers:             864 kB
Cached:          3531940 kB
SwapCached:        60992 kB
Active:          3599804 kB
Inactive:        2422176 kB
Active(anon):    1860412 kB
Inactive(anon):   629848 kB
Active(file):    1739392 kB
Inactive(file):  1792328 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      16776116 kB
SwapFree:       16602916 kB
Dirty:             85432 kB
Writeback:         10320 kB
AnonPages:       2458792 kB
Mapped:            11068 kB
Shmem:               796 kB
Slab:            3168820 kB
SReclaimable:    3099388 kB
SUnreclaim:        69432 kB
KernelStack:        3408 kB
PageTables:        14764 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    33095372 kB
Committed_AS:    4301988 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      130036 kB
VmallocChunk:   34359595900 kB
HardwareCorrupted:     0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       12520 kB
DirectMap2M:    33220608 kB

--
Tomasz Chmielewski
http://wpkg.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html