memcg with kmem limit doesn't recover after disk i/o causes limit to be hit
Subsystems (2)
Hi all,
I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers.
I feel there are probably two bugs here
- the disk i/o is not successfully managed within the kmem limit
- the cgroup never recovers, despite the untar i/o process exiting
I'm happy to help with any further debugging or try patches.
To replicate on Linux 3.14.0, run the following 6 commands:
# mkdir -p /sys/fs/cgroup/test/ # cat /sys/fs/cgroup/cpuset.cpus > /sys/fs/cgroup/test/cpuset.cpus # cat /sys/fs/cgroup/cpuset.mems > /sys/fs/cgroup/test/cpuset.mems # echo $((1<<26)) >/sys/fs/cgroup/test/memory.kmem.limit_in_bytes # echo $$ > /sys/fs/cgroup/test/tasks # tar xfz linux-3.14.1.tar.gz
Part way through the untar, the tar command exits after many errors:
...linux-3.14.1/include/linux/ima.h: Can't create 'linux-3.14.1/include/linux/ima.h' linux-3.14.1/include/linux/in.h: Can't create 'linux-3.14.1/include/linux/in.h' linux-3.14.1/include/linux/in6.h: Can't create 'linux-3.14.1/include/linux/in6.h'
...
At the same time, many errors are logged in the kernel log, of the form:
14:58:45 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
14:58:45 kernel: cache: radix_tree_node(3:test2), object size: 560, buffer size: 568, default order: 2, min order: 0 14:58:45 kernel: node 0: slabs: 140, objs: 3920, free: 0 14:58:45 kernel: node 1: slabs: 464, objs: 12992, free: 0
and
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0
and
15:31:15 kernel: ENOMEM in journal_alloc_journal_head, retrying. 15:31:20 kernel: __slab_alloc: 6946855 callbacks suppressed 15:31:20 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8050)
15:31:20 kernel: cache: jbd2_journal_head(2:test), object size: 112, buffer size: 112, default order: 0, min order: 0 15:31:20 kernel: node 0: slabs: 80, objs: 2880, free: 0 15:31:20 kernel: node 1: slabs: 0, objs: 0, free: 0
After this, the bash prompt in the cgroup is unusable:
# ls -bash: fork: Cannot allocate memory
But typically the system outside the cgroup continues to work.
All of the above happens every time that I run these 6 commands.
I am happy to help with extra information on kernel configuration, but I hope that the above is sufficient for others to replicate. I'm also happy to try suggestions and patches.
In addition, sometimes these 6 commands trigger more serious issues beyond the cgroup. I have copied a long kernel log of such an incident below.
Thanks in advance for your help,
Richard.
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0
15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f 15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c 15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b 15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77 15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2 15:11:36 kernel: [<ffffffff811017ad>] ? rcu_eqs_enter+0x70/0x83
15:11:36 kernel: [<ffffffff811017ce>] ? rcu_user_enter+0xe/0x10
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d 15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97 15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52 15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 0 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 24 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 54 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 7 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 3 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 80 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 167 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 183 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 46
15:11:36 kernel: active_anon:20251 inactive_anon:3312 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7963301 slab_reclaimable:19802 slab_unreclaimable:7992
15:11:36 kernel: mapped:7165 shmem:1531 pagetables:1575 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3390912kB min:9572kB low:11964kB high:14356kB active_anon:7148kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4836kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12257092kB min:35416kB low:44268kB high:53124kB active_anon:28044kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:12072kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16189300kB min:45076kB low:56344kB high:67612kB active_anon:45812kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 49*4kB (UEM) 62*8kB (U) 30*16kB (U) 12*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 2*2048kB (MR) 826*4096kB (MR) = 3390740kB
15:11:36 kernel: Node 0 Normal: 112*4kB (UE) 57*8kB (UE) 12*16kB (UEM) 32*32kB (UEM) 15*64kB (UEM) 5*128kB (UM) 3*256kB (UEM) 3*512kB (UEM) 2*1024kB (UM) 1*2048kB (M) 2990*4096kB (MR) = 12257160kB
15:11:36 kernel: Node 1 Normal: 95*4kB (UM) 8*8kB (U) 0*16kB 8*32kB (U) 6*64kB (UM) 10*128kB (UM) 2*256kB (EM) 2*512kB (UE) 0*1024kB 1*2048kB (M) 3951*4096kB (MR) = 16189244kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183020 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe 15:11:36 kernel: [ 4353] 104 4353 88747 10609 59 0 0 mysqld 15:11:36 kernel: [ 4354] -2 4354 1023 158 8 0 0 logger 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 627 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 4353 (mysqld) score 0 or sacrifice child 15:11:36 kernel: Killed process 4353 (mysqld) total-vm:354988kB, anon-rss:35704kB, file-rss:6732kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 45 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 57 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 131 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 174 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 113
15:11:36 kernel: active_anon:11418 inactive_anon:3312 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7971790 slab_reclaimable:19802 slab_unreclaimable:7992
15:11:36 kernel: mapped:5661 shmem:1531 pagetables:1478 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:6640kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16219896kB min:45076kB low:56344kB high:67612kB active_anon:15112kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:1780kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB
15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 55*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258116kB
15:11:36 kernel: Node 1 Normal: 245*4kB (UEM) 101*8kB (UEM) 21*16kB (UEM) 23*32kB (UEM) 19*64kB (UEM) 18*128kB (UEM) 9*256kB (M) 7*512kB (UEM) 2*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16220140kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183020 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 3988 (named) score 0 or sacrifice child 15:11:36 kernel: Killed process 3988 (named) total-vm:165728kB, anon-rss:15848kB, file-rss:2172kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 33 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 53 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 151 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 44 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 137
15:11:36 kernel: active_anon:11808 inactive_anon:3312 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7971490 slab_reclaimable:19802 slab_unreclaimable:7992
15:11:36 kernel: mapped:6534 shmem:1531 pagetables:1575 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10132kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16218696kB min:45076kB low:56344kB high:67612kB active_anon:16672kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB
15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 58*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258140kB
15:11:36 kernel: Node 1 Normal: 90*4kB (UEM) 64*8kB (UEM) 7*16kB (UE) 12*32kB (UEM) 15*64kB (UEM) 17*128kB (UEM) 7*256kB (M) 7*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16218776kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183020 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: [ 5197] -2 5197 12302 1440 25 0 0 mysqld 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger
15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child 15:11:36 kernel: Killed process 4394 (postgres) total-vm:99840kB, anon-rss:1844kB, file-rss:1004kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff810c9351>] ? has_capability_noaudit+0x12/0x14
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 30 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 51 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 47 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173
15:11:36 kernel: active_anon:10570 inactive_anon:2937 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7972794 slab_reclaimable:19802 slab_unreclaimable:7927
15:11:36 kernel: mapped:6267 shmem:1531 pagetables:1575 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3395828kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:2592kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:3960kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12270500kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:5904kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10520kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16208948kB min:45076kB low:56344kB high:67612kB active_anon:26032kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 137*4kB (UEM) 92*8kB (UM) 49*16kB (UM) 26*32kB (UEM) 12*64kB (UM) 11*128kB (UEM) 3*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3395796kB
15:11:36 kernel: Node 0 Normal: 337*4kB (UM) 138*8kB (UEM) 47*16kB (UEM) 47*32kB (UEM) 22*64kB (UEM) 10*128kB (UM) 2*256kB (UE) 3*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12270820kB
15:11:36 kernel: Node 1 Normal: 44*4kB (UEM) 70*8kB (UE) 9*16kB (UEM) 9*32kB (U) 6*64kB (U) 10*128kB (UE) 0*256kB 3*512kB (UEM) 1*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16209168kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183117 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 452 36 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: [ 5197] 104 5197 57326 4132 35 0 0 mysqld 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger
15:11:36 kernel: Out of memory: Kill process 5197 (mysqld) score 0 or sacrifice child 15:11:36 kernel: Killed process 5197 (mysqld) total-vm:231548kB, anon-rss:17736kB, file-rss:5256kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 67 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 197 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 82 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 2 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 156 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 164 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 173
15:11:36 kernel: active_anon:7704 inactive_anon:1830 isolated_anon:0
15:11:36 kernel: active_file:29305 inactive_file:151670 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7976547 slab_reclaimable:19802 slab_unreclaimable:7927
15:11:36 kernel: mapped:4180 shmem:220 pagetables:1478 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397200kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:1024kB active_file:15260kB inactive_file:79780kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2208kB shmem:112kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12273064kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:3304kB active_file:63984kB inactive_file:434284kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:3924kB shmem:448kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16220024kB min:45076kB low:56344kB high:67612kB active_anon:14568kB inactive_anon:2992kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:320kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 117*8kB (UEM) 56*16kB (UEM) 29*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397532kB
15:11:36 kernel: Node 0 Normal: 314*4kB (UEM) 156*8kB (UEM) 55*16kB (UEM) 51*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273176kB
15:11:36 kernel: Node 1 Normal: 276*4kB (UEM) 126*8kB (UEM) 31*16kB (UEM) 28*32kB (UEM) 20*64kB (UM) 20*128kB (UEM) 7*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16220592kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181806 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 1271 35 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: [ 5202] -2 5202 2336 77 7 0 0 mysqld_safe
15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child 15:11:36 kernel: Killed process 4357 (postgres) total-vm:99116kB, anon-rss:1232kB, file-rss:3852kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 98 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 196 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 79 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 179 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 156 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:6997 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7976969 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12273056kB min:35416kB low:44268kB high:53124kB active_anon:13500kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16221640kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 357*4kB (UEM) 164*8kB (UEM) 64*16kB (UEM) 52*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273588kB
15:11:36 kernel: Node 1 Normal: 408*4kB (UEM) 141*8kB (UEM) 45*16kB (UEM) 28*32kB (UEM) 19*64kB (UM) 22*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16221912kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 1999 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 1999 (httpd) total-vm:329268kB, anon-rss:3272kB, file-rss:1208kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 75 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 159 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 180 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:6430 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7977670 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275340kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16222160kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 390*4kB (UEM) 183*8kB (UEM) 68*16kB (UEM) 56*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276112kB
15:11:36 kernel: Node 1 Normal: 431*4kB (UEM) 150*8kB (UEM) 46*16kB (UEM) 30*32kB (UEM) 19*64kB (UM) 23*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16222284kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 2001 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 2001 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1212kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 72 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 181 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:5804 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7978396 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275600kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16224804kB min:45076kB low:56344kB high:67612kB active_anon:10240kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 393*4kB (UEM) 180*8kB (UEM) 71*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276340kB
15:11:36 kernel: Node 1 Normal: 506*4kB (UEM) 161*8kB (UEM) 49*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16224960kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 2000 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 2000 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1208kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 68 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 183 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:5143 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10972kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB
15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 182*8kB (UEM) 50*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227520kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 1997 (elastic-poolio) total-vm:19684kB, anon-rss:276kB, file-rss:1212kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel: ffff8804119696a0 ffff8800d56dfbf8 ffffffff8185d77b ffff880411969080 15:11:36 kernel: ffff880411969080 ffff8800d56dfca8 ffffffff8112b820 ffff8800d56dfc28 15:11:36 kernel: ffffffff81133deb ffff880409060590 ffff88040f73c500 ffff8800d56dfc38 15:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>] dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>] dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ? __put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ? put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ? filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ? ___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>] oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>] out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>] pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>] mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>] __do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ? cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ? account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ? vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>] do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>] page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:
15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 159 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 106
15:11:36 kernel: active_anon:5078 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10712kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB
15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 198*8kB (UEM) 55*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227728kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned
15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash
15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 1995 (httpd) total-vm:31888kB, anon-rss:952kB, file-rss:2344kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4551 (apache2) total-vm:299432kB, anon-rss:1896kB, file-rss:760kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4550 (apache2) total-vm:299440kB, anon-rss:1896kB, file-rss:756kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4548 (apache2) total-vm:75736kB, anon-rss:1648kB, file-rss:572kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4540 (apache2) total-vm:76004kB, anon-rss:1680kB, file-rss:1492kB 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child 15:11:37 kernel: Killed process 4386 (bash) total-vm:19872kB, anon-rss:484kB, file-rss:1472kB 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
[CC Vladimir]
Quoted 1,773 lines · Richard Davies
Hi all,
I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers.
I feel there are probably two bugs here
- the disk i/o is not successfully managed within the kmem limit
- the cgroup never recovers, despite the untar i/o process exitingI'm happy to help with any further debugging or try patches.
To replicate on Linux 3.14.0, run the following 6 commands:
# mkdir -p /sys/fs/cgroup/test/ # cat /sys/fs/cgroup/cpuset.cpus > /sys/fs/cgroup/test/cpuset.cpus # cat /sys/fs/cgroup/cpuset.mems > /sys/fs/cgroup/test/cpuset.mems # echo $((1<<26)) >/sys/fs/cgroup/test/memory.kmem.limit_in_bytes # echo $$ > /sys/fs/cgroup/test/tasks # tar xfz linux-3.14.1.tar.gzPart way through the untar, the tar command exits after many errors:
...
linux-3.14.1/include/linux/ima.h: Can't create 'linux-3.14.1/include/linux/ima.h'linux-3.14.1/include/linux/in.h: Can't create 'linux-3.14.1/include/linux/in.h'linux-3.14.1/include/linux/in6.h: Can't create 'linux-3.14.1/include/linux/in6.h'
...At the same time, many errors are logged in the kernel log, of the form:
14:58:45 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
14:58:45 kernel: cache: radix_tree_node(3:test2), object size: 560, buffer size: 568, default order: 2, min order: 0 14:58:45 kernel: node 0: slabs: 140, objs: 3920, free: 0 14:58:45 kernel: node 1: slabs: 464, objs: 12992, free: 0and
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0and
15:31:15 kernel: ENOMEM in
journal_alloc_journal_head, retrying. 15:31:20 kernel:__slab_alloc: 6946855 callbacks suppressed 15:31:20 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8050)15:31:20 kernel: cache: jbd2_journal_head(2:test), object size: 112, buffer size: 112, default order: 0, min order: 0 15:31:20 kernel: node 0: slabs: 80, objs: 2880, free: 0 15:31:20 kernel: node 1: slabs: 0, objs: 0, free: 0After this, the bash prompt in the cgroup is unusable:
# ls -bash: fork: Cannot allocate memoryBut typically the system outside the cgroup continues to work.
All of the above happens every time that I run these 6 commands.
I am happy to help with extra information on kernel configuration, but I hope that the above is sufficient for others to replicate. I'm also happy to try suggestions and patches.
In addition, sometimes these 6 commands trigger more serious issues beyond the cgroup. I have copied a long kernel log of such an incident below.
Thanks in advance for your help,
Richard.
15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50)
15:11:36 kernel: cache: ext4_inode_cache(2:test), object size: 920, buffer size: 928, default order: 3, min order: 0 15:11:36 kernel: node 0: slabs: 946, objs: 33110, free: 0 15:11:36 kernel: node 1: slabs: 78, objs: 2730, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0xd0) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 0 15:11:36 kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x50) 15:11:36 kernel: cache: radix_tree_node(2:test), object size: 560, buffer size: 568, default order: 2, min order: 0 15:11:36 kernel: node 0: slabs: 558, objs: 15624, free: 0 15:11:36 kernel: node 1: slabs: 56, objs: 1568, free: 015:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x7715:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e215:11:36 kernel: [<ffffffff811017ad>] ?rcu_eqs_enter+0x70/0x83
15:11:36 kernel: [<ffffffff811017ce>] ?rcu_user_enter+0xe/0x10
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x9715:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x5215:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 0 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 24 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 54 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 7 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 3 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 80 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 167 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 183 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 4615:11:36 kernel: active_anon:20251 inactive_anon:3312 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7963301 slab_reclaimable:19802 slab_unreclaimable:7992
15:11:36 kernel: mapped:7165 shmem:1531 pagetables:1575 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3390912kB min:9572kB low:11964kB high:14356kB active_anon:7148kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4836kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12257092kB min:35416kB low:44268kB high:53124kB active_anon:28044kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:12072kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16189300kB min:45076kB low:56344kB high:67612kB active_anon:45812kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 49*4kB (UEM) 62*8kB (U) 30*16kB (U) 12*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 2*2048kB (MR) 826*4096kB (MR) = 3390740kB
15:11:36 kernel: Node 0 Normal: 112*4kB (UE) 57*8kB (UE) 12*16kB (UEM) 32*32kB (UEM) 15*64kB (UEM) 5*128kB (UM) 3*256kB (UEM) 3*512kB (UEM) 2*1024kB (UM) 1*2048kB (M) 2990*4096kB (MR) = 12257160kB
15:11:36 kernel: Node 1 Normal: 95*4kB (UM) 8*8kB (U) 0*16kB 8*32kB (U) 6*64kB (UM) 10*128kB (UM) 2*256kB (EM) 2*512kB (UE) 0*1024kB 1*2048kB (M) 3951*4096kB (MR) = 16189244kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183020 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe 15:11:36 kernel: [ 4353] 104 4353 88747 10609 59 0 0 mysqld 15:11:36 kernel: [ 4354] -2 4354 1023 158 8 0 0 logger 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 627 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash15:11:36 kernel: Out of memory: Kill process 4353 (mysqld) score 0 or sacrifice child 15:11:36 kernel: Killed process 4353 (mysqld) total-vm:354988kB, anon-rss:35704kB, file-rss:6732kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff810c9351>] ?has_capability_noaudit+0x12/0x14
15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 45 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 57 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 131 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 174 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 11315:11:36 kernel: active_anon:11418 inactive_anon:3312 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7971790 slab_reclaimable:19802 slab_unreclaimable:7992
15:11:36 kernel: mapped:5661 shmem:1531 pagetables:1478 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:6640kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16219896kB min:45076kB low:56344kB high:67612kB active_anon:15112kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:1780kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB
15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 55*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258116kB
15:11:36 kernel: Node 1 Normal: 245*4kB (UEM) 101*8kB (UEM) 21*16kB (UEM) 23*32kB (UEM) 19*64kB (UEM) 18*128kB (UEM) 9*256kB (M) 7*512kB (UEM) 2*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16220140kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183020 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 3988] 102 3988 41432 4505 55 0 0 named 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2335 365 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash15:11:36 kernel: Out of memory: Kill process 3988 (named) score 0 or sacrifice child 15:11:36 kernel: Killed process 3988 (named) total-vm:165728kB, anon-rss:15848kB, file-rss:2172kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff810c9351>] ?has_capability_noaudit+0x12/0x14
15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 35 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 33 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 53 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 151 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 26 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 65 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 0 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 44 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 13715:11:36 kernel: active_anon:11808 inactive_anon:3312 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7971490 slab_reclaimable:19802 slab_unreclaimable:7992
15:11:36 kernel: mapped:6534 shmem:1531 pagetables:1575 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3393084kB min:9572kB low:11964kB high:14356kB active_anon:4348kB inactive_anon:3572kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:4252kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12258280kB min:35416kB low:44268kB high:53124kB active_anon:26212kB inactive_anon:6424kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10132kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16218696kB min:45076kB low:56344kB high:67612kB active_anon:16672kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:11752kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13940kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 52*4kB (UEM) 66*8kB (UM) 35*16kB (UM) 13*32kB (UEM) 4*64kB (UM) 4*128kB (UE) 2*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3392944kB
15:11:36 kernel: Node 0 Normal: 51*4kB (UEM) 58*8kB (UEM) 14*16kB (UE) 31*32kB (UE) 14*64kB (UE) 5*128kB (UM) 2*256kB (UE) 2*512kB (UE) 2*1024kB (UM) 2*2048kB (M) 2990*4096kB (MR) = 12258140kB
15:11:36 kernel: Node 1 Normal: 90*4kB (UEM) 64*8kB (UEM) 7*16kB (UE) 12*32kB (UEM) 15*64kB (UEM) 17*128kB (UEM) 7*256kB (M) 7*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16218776kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183020 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 406 36 0 0 postgres 15:11:36 kernel: [ 4394] 103 4394 24960 712 39 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: [ 5197] -2 5197 12302 1440 25 0 0 mysqld 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child 15:11:36 kernel: Killed process 4394 (postgres) total-vm:99840kB, anon-rss:1844kB, file-rss:1004kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff810c9351>] ?has_capability_noaudit+0x12/0x14
15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 133 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 55 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 30 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 51 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 12 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 1 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 47 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 155 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 17315:11:36 kernel: active_anon:10570 inactive_anon:2937 isolated_anon:0
15:11:36 kernel: active_file:29061 inactive_file:151914 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7972794 slab_reclaimable:19802 slab_unreclaimable:7927
15:11:36 kernel: mapped:6267 shmem:1531 pagetables:1575 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3395828kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:2592kB active_file:15064kB inactive_file:79976kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:3960kB shmem:1864kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12270500kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:5904kB active_file:63204kB inactive_file:435064kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:10520kB shmem:3164kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2584kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16208948kB min:45076kB low:56344kB high:67612kB active_anon:26032kB inactive_anon:3252kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:1096kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 137*4kB (UEM) 92*8kB (UM) 49*16kB (UM) 26*32kB (UEM) 12*64kB (UM) 11*128kB (UEM) 3*256kB (UM) 1*512kB (U) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3395796kB
15:11:36 kernel: Node 0 Normal: 337*4kB (UM) 138*8kB (UEM) 47*16kB (UEM) 47*32kB (UEM) 22*64kB (UEM) 10*128kB (UM) 2*256kB (UE) 3*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12270820kB
15:11:36 kernel: Node 1 Normal: 44*4kB (UEM) 70*8kB (UE) 9*16kB (UEM) 9*32kB (U) 6*64kB (U) 10*128kB (UE) 0*256kB 3*512kB (UEM) 1*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16209168kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 183117 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 2424 40 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4392] 103 4392 24775 464 38 0 0 postgres 15:11:36 kernel: [ 4393] 103 4393 24775 452 36 0 0 postgres 15:11:36 kernel: [ 4395] 103 4395 16883 418 34 0 0 postgres 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: [ 5197] 104 5197 57326 4132 35 0 0 mysqld 15:11:36 kernel: [ 5198] -2 5198 1023 158 8 0 0 logger15:11:36 kernel: Out of memory: Kill process 5197 (mysqld) score 0 or sacrifice child 15:11:36 kernel: Killed process 5197 (mysqld) total-vm:231548kB, anon-rss:17736kB, file-rss:5256kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 67 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 160 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 197 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 82 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 2 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 104 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 156 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 164 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 17315:11:36 kernel: active_anon:7704 inactive_anon:1830 isolated_anon:0
15:11:36 kernel: active_file:29305 inactive_file:151670 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:354 writeback:38 unstable:0
15:11:36 kernel: free:7976547 slab_reclaimable:19802 slab_unreclaimable:7927
15:11:36 kernel: mapped:4180 shmem:220 pagetables:1478 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397200kB min:9572kB low:11964kB high:14356kB active_anon:1996kB inactive_anon:1024kB active_file:15260kB inactive_file:79780kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2208kB shmem:112kB slab_reclaimable:3504kB slab_unreclaimable:2268kB kernel_stack:880kB pagetables:1548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12273064kB min:35416kB low:44268kB high:53124kB active_anon:14252kB inactive_anon:3304kB active_file:63984kB inactive_file:434284kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:0kB writeback:0kB mapped:3924kB shmem:448kB slab_reclaimable:64444kB slab_unreclaimable:15760kB kernel_stack:1832kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16220024kB min:45076kB low:56344kB high:67612kB active_anon:14568kB inactive_anon:2992kB active_file:37976kB inactive_file:92616kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1412kB writeback:152kB mapped:10588kB shmem:320kB slab_reclaimable:11260kB slab_unreclaimable:13680kB kernel_stack:1824kB pagetables:2168kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 117*8kB (UEM) 56*16kB (UEM) 29*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397532kB
15:11:36 kernel: Node 0 Normal: 314*4kB (UEM) 156*8kB (UEM) 55*16kB (UEM) 51*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273176kB
15:11:36 kernel: Node 1 Normal: 276*4kB (UEM) 126*8kB (UEM) 31*16kB (UEM) 28*32kB (UEM) 20*64kB (UM) 20*128kB (UEM) 7*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16220592kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181806 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4019] -2 4019 2336 366 10 0 0 mysqld_safe 15:11:36 kernel: [ 4357] 103 4357 24779 1271 35 0 0 postgres 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9579 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash 15:11:36 kernel: [ 5202] -2 5202 2336 77 7 0 0 mysqld_safe15:11:36 kernel: Out of memory: Kill process 4357 (postgres) score 0 or sacrifice child 15:11:36 kernel: Killed process 4357 (postgres) total-vm:99116kB, anon-rss:1232kB, file-rss:3852kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 98 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 196 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 79 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 167 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 179 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 156 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 10615:11:36 kernel: active_anon:6997 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7976969 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12273056kB min:35416kB low:44268kB high:53124kB active_anon:13500kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16221640kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 357*4kB (UEM) 164*8kB (UEM) 64*16kB (UEM) 52*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 3*2048kB (M) 2991*4096kB (MR) = 12273588kB
15:11:36 kernel: Node 1 Normal: 408*4kB (UEM) 141*8kB (UEM) 45*16kB (UEM) 28*32kB (UEM) 19*64kB (UM) 22*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16221912kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 1999] -2 1999 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash15:11:36 kernel: Out of memory: Kill process 1999 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 1999 (httpd) total-vm:329268kB, anon-rss:3272kB, file-rss:1208kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 75 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 185 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 159 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 180 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 10615:11:36 kernel: active_anon:6430 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7977670 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1400 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275340kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16222160kB min:45076kB low:56344kB high:67612kB active_anon:12744kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:2004kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 390*4kB (UEM) 183*8kB (UEM) 68*16kB (UEM) 56*32kB (UEM) 24*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276112kB
15:11:36 kernel: Node 1 Normal: 431*4kB (UEM) 150*8kB (UEM) 46*16kB (UEM) 30*32kB (UEM) 19*64kB (UM) 23*128kB (UEM) 8*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3955*4096kB (MR) = 16222284kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2001] -2 2001 82317 1120 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash15:11:36 kernel: Out of memory: Kill process 2001 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 2001 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1212kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 195 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 72 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 168 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 181 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 10615:11:36 kernel: active_anon:5804 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7978396 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275600kB min:35416kB low:44268kB high:53124kB active_anon:11232kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16224804kB min:45076kB low:56344kB high:67612kB active_anon:10240kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 393*4kB (UEM) 180*8kB (UEM) 71*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276340kB
15:11:36 kernel: Node 1 Normal: 506*4kB (UEM) 161*8kB (UEM) 49*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 3*2048kB (M) 3955*4096kB (MR) = 16224960kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 2000] -2 2000 82317 1119 53 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash15:11:36 kernel: Out of memory: Kill process 2000 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 2000 (httpd) total-vm:329268kB, anon-rss:3268kB, file-rss:1208kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1 15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 68 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 183 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 10615:11:36 kernel: active_anon:5143 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10972kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB
15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 182*8kB (UEM) 50*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227520kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 1997] -2 1997 4921 372 15 0 0 elastic-poolio 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 1997 (elastic-poolio) total-vm:19684kB, anon-rss:276kB, file-rss:1212kB 15:11:36 kernel: tar invoked oom-killer: gfp_mask=0x0, order=0, oom_score_adj=0 15:11:36 kernel: tar cpuset=test mems_allowed=0-1
15:11:36 kernel: CPU: 3 PID: 5102 Comm: tar Not tainted 3.14.0-elastic #1
15:11:36 kernel: Hardware name: Supermicro H8DMT-IBX/H8DMT-IBX, BIOS 080014 10/17/2009 15:11:36 kernel:ffff8804119696a0ffff8800d56dfbf8ffffffff8185d77bffff88041196908015:11:36 kernel:ffff880411969080ffff8800d56dfca8ffffffff8112b820ffff8800d56dfc2815:11:36 kernel:ffffffff81133debffff880409060590ffff88040f73c500ffff8800d56dfc3815:11:36 kernel: Call Trace:
15:11:36 kernel: [<ffffffff8185d77b>]dump_stack+0x51/0x77
15:11:36 kernel: [<ffffffff8112b820>]dump_header+0x7a/0x208
15:11:36 kernel: [<ffffffff81133deb>] ?__put_single_page+0x1b/0x1f
15:11:36 kernel: [<ffffffff8113448f>] ?put_page+0x22/0x24
15:11:36 kernel: [<ffffffff81129e38>] ?filemap_fault+0x2c2/0x36c
15:11:36 kernel: [<ffffffff813d146e>] ?___ratelimit+0xe6/0x104
15:11:36 kernel: [<ffffffff8112bc1b>]oom_kill_process+0x6a/0x33b
15:11:36 kernel: [<ffffffff8112c306>]out_of_memory+0x41a/0x44d
15:11:36 kernel: [<ffffffff8112c39e>]pagefault_out_of_memory+0x65/0x77
15:11:36 kernel: [<ffffffff8106939e>]mm_fault_error+0xab/0x176
15:11:36 kernel: [<ffffffff810696f0>]__do_page_fault+0x287/0x3e2
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810f5018>] ?cpuacct_account_field+0x55/0x5d
15:11:36 kernel: [<ffffffff810e7816>] ?account_user_time+0x6e/0x97
15:11:36 kernel: [<ffffffff810e788c>] ?vtime_account_user+0x4d/0x52
15:11:36 kernel: [<ffffffff8106988f>]do_page_fault+0x44/0x61
15:11:36 kernel: [<ffffffff818618f8>]page_fault+0x28/0x30
15:11:36 kernel: Mem-Info:
15:11:36 kernel: Node 0 DMA per-cpu:15:11:36 kernel: CPU 0: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 1: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 2: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 3: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 4: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 5: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 6: hi: 0, btch: 1 usd: 0 15:11:36 kernel: CPU 7: hi: 0, btch: 1 usd: 0 15:11:36 kernel: Node 0 DMA32 per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 150 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 166 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 100 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 162 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 3 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 163 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 18 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 162 15:11:36 kernel: Node 0 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 194 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 159 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 180 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 35 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 176 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 8 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 185 15:11:36 kernel: Node 1 Normal per-cpu: 15:11:36 kernel: CPU 0: hi: 186, btch: 31 usd: 178 15:11:36 kernel: CPU 1: hi: 186, btch: 31 usd: 46 15:11:36 kernel: CPU 2: hi: 186, btch: 31 usd: 173 15:11:36 kernel: CPU 3: hi: 186, btch: 31 usd: 115 15:11:36 kernel: CPU 4: hi: 186, btch: 31 usd: 64 15:11:36 kernel: CPU 5: hi: 186, btch: 31 usd: 182 15:11:36 kernel: CPU 6: hi: 186, btch: 31 usd: 157 15:11:36 kernel: CPU 7: hi: 186, btch: 31 usd: 10615:11:36 kernel: active_anon:5078 inactive_anon:1812 isolated_anon:0
15:11:36 kernel: active_file:29590 inactive_file:151570 isolated_file:0
15:11:36 kernel: unevictable:621 dirty:389 writeback:38 unstable:0
15:11:36 kernel: free:7979097 slab_reclaimable:19834 slab_unreclaimable:7983
15:11:36 kernel: mapped:3222 shmem:164 pagetables:1303 bounce:0
15:11:36 kernel: free_cma:0
15:11:36 kernel: Node 0 DMA free:15900kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes 15:11:36 kernel: lowmem_reserve[]: 0 3425 16097 16097
15:11:36 kernel: Node 0 DMA32 free:3397280kB min:9572kB low:11964kB high:14356kB active_anon:1744kB inactive_anon:1012kB active_file:15288kB inactive_file:79760kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3522184kB managed:3509084kB mlocked:0kB dirty:4kB writeback:0kB mapped:2088kB shmem:88kB slab_reclaimable:3504kB slab_unreclaimable:2404kB kernel_stack:864kB pagetables:1400kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 12671 12671
15:11:36 kernel: Node 0 Normal free:12275860kB min:35416kB low:44268kB high:53124kB active_anon:10712kB inactive_anon:3316kB active_file:64280kB inactive_file:434628kB unevictable:320kB isolated(anon):0kB isolated(file):0kB present:13238272kB managed:12975484kB mlocked:320kB dirty:364kB writeback:0kB mapped:3232kB shmem:256kB slab_reclaimable:64444kB slab_unreclaimable:15780kB kernel_stack:1840kB pagetables:2196kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 1 Normal free:16227348kB min:45076kB low:56344kB high:67612kB active_anon:7856kB inactive_anon:2920kB active_file:38792kB inactive_file:91892kB unevictable:2164kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16514488kB mlocked:2164kB dirty:1188kB writeback:152kB mapped:7568kB shmem:312kB slab_reclaimable:11388kB slab_unreclaimable:13748kB kernel_stack:1712kB pagetables:1616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no 15:11:36 kernel: lowmem_reserve[]: 0 0 0 0
15:11:36 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
15:11:36 kernel: Node 0 DMA32: 229*4kB (UEM) 119*8kB (UEM) 56*16kB (UEM) 23*32kB (UEM) 13*64kB (UM) 12*128kB (UEM) 4*256kB (UM) 2*512kB (UM) 0*1024kB 3*2048kB (MR) 826*4096kB (MR) = 3397356kB
15:11:36 kernel: Node 0 Normal: 433*4kB (UEM) 187*8kB (UEM) 73*16kB (UEM) 56*32kB (UEM) 27*64kB (UEM) 11*128kB (UM) 5*256kB (UEM) 5*512kB (UEM) 4*1024kB (UM) 4*2048kB (M) 2991*4096kB (MR) = 12276588kB
15:11:36 kernel: Node 1 Normal: 588*4kB (UEM) 198*8kB (UEM) 55*16kB (UEM) 30*32kB (UEM) 20*64kB (UM) 22*128kB (UEM) 9*256kB (M) 9*512kB (UEM) 3*1024kB (M) 2*2048kB (M) 3956*4096kB (MR) = 16227728kB
15:11:36 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
15:11:36 kernel: 181749 total pagecache pages
15:11:36 kernel: 0 pages in swap cache
15:11:36 kernel: Swap cache stats: add 0, delete 0, find 0/0
15:11:36 kernel: Free swap = 16507900kB
15:11:36 kernel: Total swap = 16507900kB
15:11:36 kernel: 8388414 pages RAM
15:11:36 kernel: 0 pages HighMem/MovableOnly
15:11:36 kernel: 65682 pages reserved
15:11:36 kernel: 0 pages hwpoisoned15:11:36 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name 15:11:36 kernel: [ 1345] 0 1345 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1346] 0 1346 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1347] 0 1347 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1348] 0 1348 3072 214 12 0 0 agetty 15:11:36 kernel: [ 1349] 0 1349 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1350] 0 1350 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1351] 0 1351 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1352] 0 1352 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1353] 0 1353 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1354] 0 1354 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1355] 0 1355 3072 213 12 0 0 agetty 15:11:36 kernel: [ 1356] 0 1356 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1357] 0 1357 3072 214 11 0 0 agetty 15:11:36 kernel: [ 1358] 0 1358 4928 121 14 0 0 rc.startup 15:11:36 kernel: [ 1359] 0 1359 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1360] 0 1360 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1361] 0 1361 3072 202 12 0 0 agetty 15:11:36 kernel: [ 1362] 0 1362 4928 157 14 0 0 rc.startup 15:11:36 kernel: [ 1430] 0 1430 29365 482 22 0 0 rsyslogd 15:11:36 kernel: [ 1436] 0 1436 7887 300 20 0 -1000 sshd 15:11:36 kernel: [ 1439] 0 1439 9030 343 18 0 0 automount 15:11:36 kernel: [ 1449] 0 1449 1012 113 8 0 0 iscsid 15:11:36 kernel: [ 1450] 0 1450 1135 623 8 0 -1000 iscsid 15:11:36 kernel: [ 1454] 65534 1454 3115 153 10 0 0 dnsmasq 15:11:36 kernel: [ 1456] 0 1456 3359 277 12 0 0 ntpd 15:11:36 kernel: [ 1509] 0 1509 3616 125 8 0 -1000 tgtd 15:11:36 kernel: [ 1511] 0 1511 1567 83 8 0 -1000 tgtd 15:11:36 kernel: [ 1526] 0 1526 7947 397 20 0 0 lighttpd 15:11:36 kernel: [ 1530] 0 1530 5651 423 17 0 0 elastic-sshd 15:11:36 kernel: [ 1968] 0 1968 36379 105 25 0 0 diod 15:11:36 kernel: [ 1995] -2 1995 7972 824 20 0 0 httpd 15:11:36 kernel: [ 2084] 0 2084 4928 219 14 0 0 elastic-floodwa 15:11:36 kernel: [ 2141] 0 2141 5130 446 15 0 0 elastic 15:11:36 kernel: [ 2283] 0 2283 5130 447 14 0 0 elastic 15:11:36 kernel: [ 2371] -2 2371 962 93 7 0 0 contain 15:11:36 kernel: [ 2566] -2 2566 2663 195 11 0 0 init 15:11:36 kernel: [ 2604] -2 2604 962 93 7 0 0 contain 15:11:36 kernel: [ 2706] -2 2706 2663 195 12 0 0 init 15:11:36 kernel: [ 3640] -2 3640 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3727] -2 3727 2489 643 8 0 0 dhclient 15:11:36 kernel: [ 3822] 0 3822 9614 689 24 0 0 sshd 15:11:36 kernel: [ 3834] -2 3834 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3902] -2 3902 4721 208 14 0 0 cron 15:11:36 kernel: [ 3912] -2 3912 12147 384 22 0 0 rsyslogd 15:11:36 kernel: [ 3963] -2 3963 12483 298 28 0 0 sshd 15:11:36 kernel: [ 3986] -2 3986 12483 298 27 0 0 sshd 15:11:36 kernel: [ 4001] -2 4001 3645 229 13 0 0 getty 15:11:36 kernel: [ 4386] 0 4386 4968 489 15 0 0 bash 15:11:36 kernel: [ 4540] -2 4540 19001 793 41 0 0 apache2 15:11:36 kernel: [ 4548] 33 4548 18934 555 39 0 0 apache2 15:11:36 kernel: [ 4550] 33 4550 74860 663 68 0 0 apache2 15:11:36 kernel: [ 4551] 33 4551 74858 664 68 0 0 apache2 15:11:36 kernel: [ 4608] -2 4608 4721 207 15 0 0 cron 15:11:36 kernel: [ 4625] -2 4625 3645 229 13 0 0 getty 15:11:36 kernel: [ 5085] 0 5085 9613 689 24 0 0 sshd 15:11:36 kernel: [ 5088] 0 5088 4968 487 15 0 0 bash 15:11:36 kernel: [ 5098] 0 5098 2033 102 10 0 0 tail 15:11:36 kernel: [ 5102] 0 5102 7554 652 19 0 0 tar 15:11:36 kernel: [ 5163] 0 5163 1994 104 9 0 0 sleep 15:11:36 kernel: [ 5169] 0 5169 9579 690 23 0 0 sshd 15:11:36 kernel: [ 5172] 0 5172 3072 200 12 0 0 agetty 15:11:36 kernel: [ 5173] 0 5173 3072 201 12 0 0 agetty 15:11:36 kernel: [ 5174] 0 5174 4968 488 15 0 0 bash15:11:36 kernel: Out of memory: Kill process 1995 (httpd) score 0 or sacrifice child 15:11:36 kernel: Killed process 1995 (httpd) total-vm:31888kB, anon-rss:952kB, file-rss:2344kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4551 (apache2) total-vm:299432kB, anon-rss:1896kB, file-rss:760kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4550 (apache2) total-vm:299440kB, anon-rss:1896kB, file-rss:756kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4548 (apache2) total-vm:75736kB, anon-rss:1648kB, file-rss:572kB 15:11:36 kernel: Out of memory: Kill process 4540 (apache2) score 0 or sacrifice child 15:11:36 kernel: Killed process 4540 (apache2) total-vm:76004kB, anon-rss:1680kB, file-rss:1492kB 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child 15:11:37 kernel: Killed process 4386 (bash) total-vm:19872kB, anon-rss:484kB, file-rss:1472kB 15:11:37 kernel: Out of memory: Kill process 3822 (sshd) score 0 or sacrifice child
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Richard,
18.04.2014 19:59, Michal Hocko:
Quoted 13 lines
[CC Vladimir]
Quoted 10 lines · Richard Davies
Hi all,
I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers.
I feel there are probably two bugs here
- the disk i/o is not successfully managed within the kmem limit
- the cgroup never recovers, despite the untar i/o process exiting
Unfortunately, work on per cgroup kmem limits is not completed yet. Currently it lacks kmem reclaim on per cgroup memory pressure, which is vital for using kmem limits in real life. Basically that means that if a process inside a memory cgroup reaches its kmem limit, it will be returned ENOMEM on any allocation attempt, and no attempt will be made to reclaim old cached data.
In your case untar consumes all kmem available to the cgroup by allocating memory for storing fs metadata (inodes, dentries). Those metadata are left cached in memory after untar dies, because they can be potentially used by other processes. As a result, any further attempt to allocate kmem (e.g. to create a process) will fail. It should try to reclaim the cached metadata instead, but this functionality is not implemented yet.
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
Thanks.
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
Maybe we should make this explicit in both Documentation/cgroups/memory.txt and config MEMCG_KMEM help text.
Signature · 2 lines
Michal Hocko SUSE Labs
18.04.2014 22:20, Michal Hocko:
Quoted 6 lines · Vladimir Davydov
Quoted 2 lines
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
Maybe we should make this explicit in both
Documentation/cgroups/memory.txtand configMEMCG_KMEMhelp text.
Yeah, definitely. I'll send the patch.
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Vladimir DavydovQuoted 9 lines · Vladimir Davydov
Quoted 4 lines · Richard Davies
I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers.
Unfortunately, work on per cgroup kmem limits is not completed yet. Currently it lacks kmem reclaim on per cgroup memory pressure, which is vital for using kmem limits in real life.
...
Quoted 2 lines
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
Thanks for explaining the strange errors I got.
My motivation is to prevent a fork bomb in a container from affecting other processes outside that container.
kmem limits were the preferred mechanism in several previous discussions about two years ago (I'm copying in participants from those previous discussions and give links below). So I tried kmem first but found bugs.
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROC exists but is per-user, not per-container.
Perhaps there is an up-to-date task counter patchset or similar?
Thank you all,
Richard.
Some references to previous discussions:
Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for memcg: stripped down version
http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372
Re: [PATCH 00/10] cgroups: Task counter subsystem v8
http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310
[RFD] Merge task counter into memcg
http://thread.gmane.org/gmane.linux.kernel/1280302
Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186
[PATCH] new cgroup controller "fork"
http://thread.gmane.org/gmane.linux.kernel/1210878
Re: Process Limit cgroups
http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369
Re: [lxc-devel] process number limit
https://www.mail-archive.com/[email protected]/msg03309.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
I would still be in strong support of a cgroup replacement for NPROC rlimit.
Quoted 62 lines · Richard Davies
Quoted 9 lines · Vladimir Davydov
Quoted 4 lines · Richard Davies
I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers.
Unfortunately, work on per cgroup kmem limits is not completed yet. Currently it lacks kmem reclaim on per cgroup memory pressure, which is vital for using kmem limits in real life.
...
Quoted 2 lines
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
Thanks for explaining the strange errors I got.
My motivation is to prevent a fork bomb in a container from affecting other processes outside that container.
kmem limits were the preferred mechanism in several previous discussions about two years ago (I'm copying in participants from those previous discussions and give links below). So I tried kmem first but found bugs.
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Thank you all,
Richard.
Some references to previous discussions:
Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for
memcg: stripped down version
http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372Re: [PATCH 00/10] cgroups: Task counter subsystem v8
http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310[RFD] Merge task counter into memcg
http://thread.gmane.org/gmane.linux.kernel/1280302Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186[PATCH] new cgroup controller "fork"
http://thread.gmane.org/gmane.linux.kernel/1210878Re: Process Limit cgroups
http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369Re: [lxc-devel] process number limit
https://www.mail-archive.com/[email protected]/msg03309.html
A message that is not in the archive (sent off-list or not yet received): 20140420142830.GC22077-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
I would still be in strong support of a cgroup replacement for NPROC rlimit.
Quoted 62 lines · Richard Davies
Quoted 9 lines · Vladimir Davydov
Quoted 4 lines · Richard Davies
I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers.
Unfortunately, work on per cgroup kmem limits is not completed yet. Currently it lacks kmem reclaim on per cgroup memory pressure, which is vital for using kmem limits in real life.
...
Quoted 2 lines
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
Thanks for explaining the strange errors I got.
My motivation is to prevent a fork bomb in a container from affecting other processes outside that container.
kmem limits were the preferred mechanism in several previous discussions about two years ago (I'm copying in participants from those previous discussions and give links below). So I tried kmem first but found bugs.
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Thank you all,
Richard.
Some references to previous discussions:
Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for
memcg: stripped down version
http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372Re: [PATCH 00/10] cgroups: Task counter subsystem v8
http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310[RFD] Merge task counter into memcg
http://thread.gmane.org/gmane.linux.kernel/1280302Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186[PATCH] new cgroup controller "fork"
http://thread.gmane.org/gmane.linux.kernel/1210878Re: Process Limit cgroups
http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369Re: [lxc-devel] process number limit
https://www.mail-archive.com/[email protected]/msg03309.html
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to a message not in the archiveQuoted 26 lines · On Sun, 20 Apr 2014 15:28:30 +0100 Richard Davies…
Quoted 9 lines · Vladimir Davydov
Quoted 4 lines · Richard Davies
I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers.
Unfortunately, work on per cgroup kmem limits is not completed yet. Currently it lacks kmem reclaim on per cgroup memory pressure, which is vital for using kmem limits in real life.
...
Quoted 2 lines
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
Thanks for explaining the strange errors I got.
My motivation is to prevent a fork bomb in a container from affecting other processes outside that container.
kmem limits were the preferred mechanism in several previous discussions about two years ago (I'm copying in participants from those previous discussions and give links below). So I tried kmem first but found bugs.
Quoted 5 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 33 lines
Thank you all,
Richard.
Some references to previous discussions:
Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for memcg: stripped down version
http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372Re: [PATCH 00/10] cgroups: Task counter subsystem v8
http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310[RFD] Merge task counter into memcg
http://thread.gmane.org/gmane.linux.kernel/1280302Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186[PATCH] new cgroup controller "fork"
http://thread.gmane.org/gmane.linux.kernel/1210878Re: Process Limit cgroups
http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369Re: [lxc-devel] process number limit
https://www.mail-archive.com/[email protected]/msg03309.html
_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Richard DaviesQuoted 26 lines · On Sun, 20 Apr 2014 15:28:30 +0100 Richard Davies <[email protected]> wrote:
Quoted 9 lines · Vladimir Davydov
Quoted 4 lines · Richard Davies
I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers.
Unfortunately, work on per cgroup kmem limits is not completed yet. Currently it lacks kmem reclaim on per cgroup memory pressure, which is vital for using kmem limits in real life.
...
Quoted 2 lines
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
Thanks for explaining the strange errors I got.
My motivation is to prevent a fork bomb in a container from affecting other processes outside that container.
kmem limits were the preferred mechanism in several previous discussions about two years ago (I'm copying in participants from those previous discussions and give links below). So I tried kmem first but found bugs.
Quoted 5 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 27 lines
Thank you all,
Richard.
Some references to previous discussions:
Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for memcg: stripped down version
http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372Re: [PATCH 00/10] cgroups: Task counter subsystem v8
http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310[RFD] Merge task counter into memcg
http://thread.gmane.org/gmane.linux.kernel/1280302Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186[PATCH] new cgroup controller "fork"
http://thread.gmane.org/gmane.linux.kernel/1210878Re: Process Limit cgroups
http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369
Quoted 6 lines
Re: [lxc-devel] process number limit
https://www.mail-archive.com/[email protected]/msg03309.html
_______________________________________________
Containers mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/containers
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 6 lines
Quoted 5 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Thanks,
Richard.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Who in kernel-land still needs to be convinced of the utility of this idea?
Quoted 20 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 6 lines
Quoted 5 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 6 lines
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Thanks,
Richard.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
A message that is not in the archive (sent off-list or not yet received): 20140422200531.GA19334-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Who in kernel-land still needs to be convinced of the utility of this idea?
Quoted 26 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 6 lines
Quoted 5 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Thanks,
Richard.
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to a message not in the archiveQuoted 15 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 9 lines
Quoted 5 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
Quoted 7 lines
Quoted 4 lines
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Richard DaviesQuoted 15 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 9 lines
Quoted 5 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
Quoted 7 lines
Quoted 4 lines
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 22 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov <[email protected]> wrote:
Quoted 21 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 14 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 6 lines
Quoted 2 lines
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.git cpuacct-task-limit-3.13git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
Quoted 13 lines
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 8 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14
Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
-serge
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 33 lines · Serge Hallyn
Quoted 32 lines · Dwight Engen
Quoted 28 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 16 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 10 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
Quoted 7 lines
Quoted 4 lines
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Quoted 8 lines
Quoted 4 lines
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 39 lines · Michal Hocko
Quoted 38 lines · Serge Hallyn
Quoted 36 lines · Dwight Engen
Quoted 22 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 21 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 14 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 6 lines
Quoted 2 lines
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14
Quoted 7 lines
Quoted 3 lines
Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
It has nothing to do with the state of the limit. I simply don't believe that emulating RLIMIT_NPROC by controlling stack size is a good idea.
-serge
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 48 lines · Serge Hallyn
Quoted 39 lines · Michal Hocko
Quoted 38 lines · Serge Hallyn
Quoted 36 lines · Dwight Engen
Quoted 22 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 21 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 14 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 6 lines
Quoted 2 lines
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Quoted 7 lines
Quoted 3 lines
Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
Quoted 5 lines
It has nothing to do with the state of the limit. I simply don't believe that emulating
RLIMIT_NPROCby controlling stack size is a good idea.-serge
I think that having a limit on the number of processes allowed in a cgroup is a lot better then relaying on the kmem limit.
The problem that task-limit tries to solve is degradation of system performance caused by too many processes in a certain cgroup. I'm currently testing the patches with 3.12.16.
-hackman
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Serge HallynQuoted 48 lines · Serge Hallyn
Quoted 39 lines · Michal Hocko
Quoted 38 lines · Serge Hallyn
Quoted 36 lines · Dwight Engen
Quoted 22 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 21 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 14 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 6 lines
Quoted 2 lines
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Quoted 7 lines
Quoted 3 lines
Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
Quoted 5 lines
It has nothing to do with the state of the limit. I simply don't believe that emulating
RLIMIT_NPROCby controlling stack size is a good idea.-serge
I think that having a limit on the number of processes allowed in a cgroup is a lot better then relaying on the kmem limit.
The problem that task-limit tries to solve is degradation of system performance caused by too many processes in a certain cgroup. I'm currently testing the patches with 3.12.16.
-hackman
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Serge HallynThank you. These are two different things. They may have a relationship but they ate not the same, and pretending they are is a bad experience.
Quoted 57 lines · Serge Hallyn
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Serge Hallyn
Quoted 8 lines · Dwight Engen
Quoted 5 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov <[email protected]> wrote:
Quoted 4 lines · Richard Davies
Quoted 3 lines · Dwight Engen
Quoted 2 lines · Richard Davies
Quoted 1 line · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently
broken.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Do not use it. We are working on making it usable though.
...
Quoted 6 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or
similar?
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 11 lines
Quoted 8 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable
substitute.
Quoted 4 lines
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
It has nothing to do with the state of the limit. I simply don't believe that emulating
RLIMIT_NPROCby controlling stack size is a good idea.-serge
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Serge HallynThank you. These are two different things. They may have a relationship but they ate not the same, and pretending they are is a bad experience.
Quoted 57 lines · Serge Hallyn
Quoted 10 lines · Michal Hocko
Quoted 9 lines · Serge Hallyn
Quoted 8 lines · Dwight Engen
Quoted 5 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 4 lines · Richard Davies
Quoted 3 lines · Dwight Engen
Quoted 2 lines · Richard Davies
Quoted 1 line · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently
broken.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Do not use it. We are working on making it usable though.
...
Quoted 6 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or
similar?
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 11 lines
Quoted 8 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable
substitute.
Quoted 4 lines
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
It has nothing to do with the state of the limit. I simply don't believe that emulating
RLIMIT_NPROCby controlling stack size is a good idea.-serge
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Serge HallynQuoted 44 lines · Serge Hallyn
Quoted 43 lines · Michal Hocko
Quoted 41 lines · Serge Hallyn
Quoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 8 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
Quoted 7 lines
Quoted 3 lines
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
It has nothing to do with the state of the limit. I simply don't believe that emulating
RLIMIT_NPROCby controlling stack size is a good idea.
I was not the one who decided that the kmem extension of memory controller should cover also the task number as a side effect but still the decision sounds plausible to me because the kmem approach is more generic.
Btw. if this is a problem them please go ahead and continue the original discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the other people involved.
I do not see any new arguments here, except that the kmem implementation is not ready yet.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
Is there some specific technical reason why you think this is a bad idea? I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).
Quoted 62 lines · Michal Hocko
Quoted 44 lines · Serge Hallyn
Quoted 43 lines · Michal Hocko
Quoted 41 lines · Serge Hallyn
Quoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov <[email protected]> wrote:
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 8 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
Quoted 7 lines
Quoted 3 lines
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
It has nothing to do with the state of the limit. I simply don't believe that emulating
RLIMIT_NPROCby controlling stack size is a good idea.I was not the one who decided that the kmem extension of memory controller should cover also the task number as a side effect but still the decision sounds plausible to me because the kmem approach is more generic.
Btw. if this is a problem them please go ahead and continue the original discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the other people involved.
Quoted 5 lines
I do not see any new arguments here, except that the kmem implementation is not ready yet.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 3 lines · Tim Hockin
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does but it brings other problems that aren't fixable in kmem itself, we can certainly reconsider this cgroup subsystem. But I haven't yet seen argument of this kind yet.
Quoted 6 lines
Is there some specific technical reason why you think this is a bad idea?
I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).
Because it's core code and it adds complications and overhead in the fork/exit path. We just don't add new core code just for the sake of slightly prettier interfaces.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Here's the reason it doesn't work for us: It doesn't work. It was something like 2 YEARS since we first wanted this, and it STILL does not work. You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want. What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 26 lines · Frederic Weisbecker
Quoted 3 lines · Tim Hockin
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does but it brings other problems that aren't fixable in kmem itself, we can certainly reconsider this cgroup subsystem. But I haven't yet seen argument of this kind yet.
Quoted 6 lines
Is there some specific technical reason why you think this is a bad idea?
I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).Because it's core code and it adds complications and overhead in the fork/exit path. We just don't add new core code just for the sake of slightly prettier interfaces.
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Quoted 27 lines · Frederic Weisbecker
Quoted 26 lines · Tim Hockin
Quoted 3 lines
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does but it brings other problems that aren't fixable in kmem itself, we can certainly reconsider this cgroup subsystem. But I haven't yet seen argument of this kind yet.
Quoted 6 lines
Is there some specific technical reason why you think this is a bad idea?
I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).Because it's core code and it adds complications and overhead in the fork/exit path. We just don't add new core code just for the sake of slightly prettier interfaces.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 22 lines · On Tue, 29 Apr 2014 19:06:39 +0200 Michal Hocko <[email protected]> wrote:
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 6 lines
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Quoted 26 lines
Quoted 26 lines
Quoted 24 lines · Frederic Weisbecker
Quoted 4 lines · Tim Hockin
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does but it brings other problems that aren't fixable in kmem itself, we can certainly reconsider this cgroup subsystem. But I haven't yet seen argument of this kind yet.
Quoted 6 lines
Is there some specific technical reason why you think this is a bad idea?
I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Because it's core code and it adds complications and overhead in the fork/exit path. We just don't add new core code just for the sake of slightly prettier interfaces.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 26 lines · Dwight Engen
Quoted 25 lines · Michal Hocko
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 8 lines
Quoted 3 lines
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
I am also somewhat worried that normal software use could legitimately go above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing just by booting a distro with a few daemons in a container.
Richard.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 39 lines · Richard Davies
Quoted 26 lines · Dwight Engen
Quoted 25 lines · Michal Hocko
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 8 lines
Quoted 3 lines
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
Quoted 6 lines
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a userspace accounted memory with the memory cgroup controller. And you do not have to limit that one. Kmem accounting refers to kernel internal allocations - slab memory and per process kernel stack. You can see how much memory is allocated per container by memory.kmem.usage_in_bytes or have a look at /proc/slabinfo to see what kind of memory kernel allocates globally and might be accounted for a container as well.
The primary problem with the kmem accounting right now is that such a memory is not "reclaimed" and so if the kmem limit is reached all the further kmem allocations fail. The biggest user of the kmem allocations on many systems is dentry and inode chache which is reclaimable easily. When this is implemented the kmem limit will be usable to both prevent forkbombs but also other DOS scenarios when the kernel is pushed to allocate a huge amount of memory.
HTH
Quoted 5 lines
I am also somewhat worried that normal software use could legitimately go above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing just by booting a distro with a few daemons in a container.
Richard.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 14 lines · Michal Hocko
Quoted 13 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
Quoted 6 lines
Quoted 2 lines
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a userspace accounted memory with the memory cgroup controller. And you do not have to limit that one.
OK, that's helpful - thanks.
As an aside, with the normal (non-kmem) cgroup controller, is there a way for me to exclude page cache and only limit the equivalent of the rss line in memory.stat?
e.g. say I have a 256GB physical machine, running 200 containers, each with 1GB normal-mem limit (for running software) and 256MB kmem limit (to stop fork-bombs).
The physical disk IO bandwidth is a shared resource between all the containers, so ideally I would like the kernel to used the 56GB of RAM as shared page cache however it best reduces physical IOPs, rather than having a per-container limit.
Thanks,
Richard.
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 22 lines · Richard Davies
Quoted 14 lines · Michal Hocko
Quoted 13 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
Quoted 6 lines
Quoted 2 lines
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a userspace accounted memory with the memory cgroup controller. And you do not have to limit that one.
Quoted 5 lines
OK, that's helpful - thanks.
As an aside, with the normal (non-kmem) cgroup controller, is there a way for me to exclude page cache and only limit the equivalent of the rss line in memory.stat?
No
Quoted 8 lines
e.g. say I have a 256GB physical machine, running 200 containers, each with 1GB normal-mem limit (for running software) and 256MB kmem limit (to stop fork-bombs).
The physical disk IO bandwidth is a shared resource between all the containers, so ideally I would like the kernel to used the 56GB of RAM as shared page cache however it best reduces physical IOPs, rather than having a per-container limit.
Then do not use any memory.limit_in_bytes and if there is a memory pressure then the global reclaim will shrink all the containers proportionally and the page cache will be the #1 target for the reclaim (but we are getting off-topic here I am afraid).
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
A message that is not in the archive (sent off-list or not yet received): 20140429183928.GF29606-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 22 lines · Richard Davies
Quoted 14 lines · Michal Hocko
Quoted 13 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
Quoted 6 lines
Quoted 2 lines
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a userspace accounted memory with the memory cgroup controller. And you do not have to limit that one.
Quoted 5 lines
OK, that's helpful - thanks.
As an aside, with the normal (non-kmem) cgroup controller, is there a way for me to exclude page cache and only limit the equivalent of the rss line in memory.stat?
No
Quoted 8 lines
e.g. say I have a 256GB physical machine, running 200 containers, each with 1GB normal-mem limit (for running software) and 256MB kmem limit (to stop fork-bombs).
The physical disk IO bandwidth is a shared resource between all the containers, so ideally I would like the kernel to used the 56GB of RAM as shared page cache however it best reduces physical IOPs, rather than having a per-container limit.
Then do not use any memory.limit_in_bytes and if there is a memory pressure then the global reclaim will shrink all the containers proportionally and the page cache will be the #1 target for the reclaim (but we are getting off-topic here I am afraid).
Signature · 2 lines
Michal Hocko SUSE Labs
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Michal HockoQuoted 55 lines · Michal Hocko
Quoted 39 lines · Richard Davies
Quoted 26 lines · Dwight Engen
Quoted 25 lines · Michal Hocko
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 8 lines
Quoted 3 lines
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
Quoted 6 lines
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a userspace accounted memory with the memory cgroup controller. And you do not have to limit that one. Kmem accounting refers to kernel internal allocations - slab memory and per process kernel stack. You can see how much memory is allocated per container by memory.kmem.usage_in_bytes or have a look at
/proc/slabinfoto see what kind of memory kernel allocates globally and might be accounted for a container as well.
Quoted 7 lines
The primary problem with the kmem accounting right now is that such a memory is not "reclaimed" and so if the kmem limit is reached all the further kmem allocations fail. The biggest user of the kmem allocations on many systems is dentry and inode chache which is reclaimable easily. When this is implemented the kmem limit will be usable to both prevent forkbombs but also other DOS scenarios when the kernel is pushed to allocate a huge amount of memory.
I would have to disagree here.
If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem. The simple scheduling of so many processes generates have load on the machine.
Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes.
Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure on the kmem, but will would not produce many processes. On the other hand, forking thousands of processes with low memory footprint will hit the scheduler a lot faster then hitting the kmem limit.
Kmem limit is something that we need! But firmly believe that we need a simple NPROC limit for cgroups.
-hackman
Quoted 9 lines
HTH
Quoted 5 lines
I am also somewhat worried that normal software use could legitimately go above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing just by booting a distro with a few daemons in a container.
Richard.
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 64 lines · Marian Marinov
Quoted 55 lines · Michal Hocko
Quoted 39 lines · Richard Davies
Quoted 26 lines · Dwight Engen
Quoted 25 lines · Michal Hocko
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 8 lines
Quoted 3 lines
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
Quoted 6 lines
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a userspace accounted memory with the memory cgroup controller. And you do not have to limit that one. Kmem accounting refers to kernel internal allocations - slab memory and per process kernel stack. You can see how much memory is allocated per container by memory.kmem.usage_in_bytes or have a look at
/proc/slabinfoto see what kind of memory kernel allocates globally and might be accounted for a container as well.Quoted 7 lines
The primary problem with the kmem accounting right now is that such a memory is not "reclaimed" and so if the kmem limit is reached all the further kmem allocations fail. The biggest user of the kmem allocations on many systems is dentry and inode chache which is reclaimable easily. When this is implemented the kmem limit will be usable to both prevent forkbombs but also other DOS scenarios when the kernel is pushed to allocate a huge amount of memory.
Quoted 4 lines
I would have to disagree here.
If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem. The simple scheduling of so many processes generates have load on the machine.
Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes.
What prevents you from setting the kmem limit to NR_PROC * 8K + slab_pillow?
Quoted 2 lines
Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure on the kmem, but will would not produce many processes.
Once we have a proper slab reclaim implementation this shouldn't be a problem.
Quoted 5 lines
On the other hand, forking thousands of processes with low memory footprint will hit the scheduler a lot faster then hitting the kmem limit.
Kmem limit is something that we need! But firmly believe that we need a simple NPROC limit for cgroups.
Once again. If you feel that your usecase is not covered by the kmem limit follow up on the original email thread I have referenced earlier in the thread. Splitting up the discussion doesn't help at all.
Quoted 12 lines
-hackman
Quoted 9 lines
HTH
Quoted 5 lines
I am also somewhat worried that normal software use could legitimately go above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing just by booting a distro with a few daemons in a container.
Richard.
Signature · 2 lines
Michal Hocko SUSE Labs
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 64 lines · Marian Marinov
Quoted 55 lines · Michal Hocko
Quoted 39 lines · Richard Davies
Quoted 26 lines · Dwight Engen
Quoted 25 lines · Michal Hocko
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 8 lines
Quoted 3 lines
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
Quoted 6 lines
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a userspace accounted memory with the memory cgroup controller. And you do not have to limit that one. Kmem accounting refers to kernel internal allocations - slab memory and per process kernel stack. You can see how much memory is allocated per container by memory.kmem.usage_in_bytes or have a look at
/proc/slabinfoto see what kind of memory kernel allocates globally and might be accounted for a container as well.Quoted 7 lines
The primary problem with the kmem accounting right now is that such a memory is not "reclaimed" and so if the kmem limit is reached all the further kmem allocations fail. The biggest user of the kmem allocations on many systems is dentry and inode chache which is reclaimable easily. When this is implemented the kmem limit will be usable to both prevent forkbombs but also other DOS scenarios when the kernel is pushed to allocate a huge amount of memory.
Quoted 4 lines
I would have to disagree here.
If a container starts to create many processes it will use kmem, however my use cases, the memory is not the problem. The simple scheduling of so many processes generates have load on the machine.
Even if I have the memory to handle this... the problem becomes the scheduling of all of these processes.
What prevents you from setting the kmem limit to NR_PROC * 8K + slab_pillow?
Quoted 2 lines
Typical rsync of 2-3TB of small files(1-100k) will generate heavy pressure on the kmem, but will would not produce many processes.
Once we have a proper slab reclaim implementation this shouldn't be a problem.
Quoted 5 lines
On the other hand, forking thousands of processes with low memory footprint will hit the scheduler a lot faster then hitting the kmem limit.
Kmem limit is something that we need! But firmly believe that we need a simple NPROC limit for cgroups.
Once again. If you feel that your usecase is not covered by the kmem limit follow up on the original email thread I have referenced earlier in the thread. Splitting up the discussion doesn't help at all.
Quoted 12 lines
-hackman
Quoted 9 lines
HTH
Quoted 5 lines
I am also somewhat worried that normal software use could legitimately go above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing just by booting a distro with a few daemons in a container.
Richard.
Signature · 2 lines
Michal Hocko SUSE Labs
A message that is not in the archive (sent off-list or not yet received): 20140429180927.GB29606-2oeHp4OYwSjPZs67QiJbJtBPR1lH4CV8@public.gmane.org
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 39 lines · Richard Davies
Quoted 26 lines · Dwight Engen
Quoted 25 lines · Michal Hocko
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 8 lines
Quoted 3 lines
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
Quoted 6 lines
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
Page cache (which is what you mean by disk cache probably) is a userspace accounted memory with the memory cgroup controller. And you do not have to limit that one. Kmem accounting refers to kernel internal allocations - slab memory and per process kernel stack. You can see how much memory is allocated per container by memory.kmem.usage_in_bytes or have a look at /proc/slabinfo to see what kind of memory kernel allocates globally and might be accounted for a container as well.
The primary problem with the kmem accounting right now is that such a memory is not "reclaimed" and so if the kmem limit is reached all the further kmem allocations fail. The biggest user of the kmem allocations on many systems is dentry and inode chache which is reclaimable easily. When this is implemented the kmem limit will be usable to both prevent forkbombs but also other DOS scenarios when the kernel is pushed to allocate a huge amount of memory.
HTH
Quoted 5 lines
I am also somewhat worried that normal software use could legitimately go above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing just by booting a distro with a few daemons in a container.
Richard.
Signature · 2 lines
Michal Hocko SUSE Labs
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 26 lines · Dwight Engen
Quoted 25 lines · Michal Hocko
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 8 lines
Quoted 3 lines
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Certainly I would like to be able to limit container fork-bombs without limiting the amount of disk IO caching for processes in those containers.
In my testing with of kmem limits, I needed a limit of 256MB or lower to catch fork bombs early enough. I would definitely like more than 256MB of disk caching.
So if we go the "working kmem" route, I would like to be able to specify a limit excluding disk cache.
I am also somewhat worried that normal software use could legitimately go above 256MB of kmem (even excluding disk cache) - I got to 50MB in testing just by booting a distro with a few daemons in a container.
Richard.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 22 lines · On Tue, 29 Apr 2014 19:06:39 +0200 Michal Hocko…
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 6 lines
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Is there a plan to separately account/limit stack pages vs kmem in general? Richard would have to verify, but I suspect kmem is not currently viable as a process limiter for him because icache/dcache/stack is all accounted together.
Quoted 30 lines
Quoted 29 lines
Quoted 27 lines · Frederic Weisbecker
Quoted 4 lines · Tim Hockin
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does but it brings other problems that aren't fixable in kmem itself, we can certainly reconsider this cgroup subsystem. But I haven't yet seen argument of this kind yet.
Quoted 6 lines
Is there some specific technical reason why you think this is a bad idea?
I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).Because it's core code and it adds complications and overhead in the fork/exit path. We just don't add new core code just for the sake of slightly prettier interfaces.
A message that is not in the archive (sent off-list or not yet received): CAO_Rewa20dneL8e3T4UPnu2Dkv28KTgFJR9_YSmRBKp-_yqewg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 1 line · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work.
There is a "simple" solution for that. Help us to fix it.
Quoted 2 lines
It was something like 2 YEARS since we first wanted this, and it STILL does not work.
My recollection is that it was primarily Parallels and Google asking for the kmem accounting. The reason why I didn't fight against inclusion although the implementation at the time didn't have a proper slab shrinking implemented was that that would happen later. Well, that later hasn't happened yet and we are slowly getting there.
Quoted 3 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want.
But we cannot simply add a new interface that will have to be maintained for ever just because something else that is supposed to workaround bugs.
Quoted 2 lines
What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
You can use an out-of-tree patchset for the time being or help to get kmem into shape. If there are principal reasons why kmem cannot be used then you better articulate them.
Quoted 27 lines · Frederic Weisbecker
Quoted 26 lines · Tim Hockin
Quoted 3 lines
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does but it brings other problems that aren't fixable in kmem itself, we can certainly reconsider this cgroup subsystem. But I haven't yet seen argument of this kind yet.
Quoted 6 lines
Is there some specific technical reason why you think this is a bad idea?
I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).Because it's core code and it adds complications and overhead in the fork/exit path. We just don't add new core code just for the sake of slightly prettier interfaces.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to a message not in the archiveQuoted 3 lines · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work. It was something like 2 YEARS since we first wanted this, and it STILL does not work.
When I was working on the task counter cgroup subsystem 2 years ago, the patches were actually pushed back by google people, in favour of task stack kmem cgroup subsystem.
The reason was that expressing the forkbomb issue in terms of number of tasks as a resource is awkward and that the real resource in the game comes from kernel memory exhaustion due to task stack being allocated over and over, swap ping-pong and stuffs...
And that was a pretty good argument. I still agree with that. Especially since that could solve others people issues at the same time. kmem cgroup has a quite large domain of application.
Quoted 4 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want. What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
The request is simple but I don't think that adding the task counter cgroup subsystem is simpler than extending the kmem code to apply limits to only task stack. Especially in terms of maintainance.
Also you guys have very good mm kernel developers who are already familiar with this.
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Tim HockinQuoted 3 lines · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work. It was something like 2 YEARS since we first wanted this, and it STILL does not work.
When I was working on the task counter cgroup subsystem 2 years ago, the patches were actually pushed back by google people, in favour of task stack kmem cgroup subsystem.
The reason was that expressing the forkbomb issue in terms of number of tasks as a resource is awkward and that the real resource in the game comes from kernel memory exhaustion due to task stack being allocated over and over, swap ping-pong and stuffs...
And that was a pretty good argument. I still agree with that. Especially since that could solve others people issues at the same time. kmem cgroup has a quite large domain of application.
Quoted 4 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want. What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
The request is simple but I don't think that adding the task counter cgroup subsystem is simpler than extending the kmem code to apply limits to only task stack. Especially in terms of maintainance.
Also you guys have very good mm kernel developers who are already familiar with this.
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 21 lines · Frederic Weisbecker
Quoted 3 lines · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work. It was something like 2 YEARS since we first wanted this, and it STILL does not work.
When I was working on the task counter cgroup subsystem 2 years ago, the patches were actually pushed back by google people, in favour of task stack kmem cgroup subsystem.
The reason was that expressing the forkbomb issue in terms of number of tasks as a resource is awkward and that the real resource in the game comes from kernel memory exhaustion due to task stack being allocated over and over, swap ping-pong and stuffs...
And that was a pretty good argument. I still agree with that. Especially since that could solve others people issues at the same time. kmem cgroup has a quite large domain of application.
Quoted 4 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want. What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 6 lines
The request is simple but I don't think that adding the task counter cgroup subsystem is simpler than extending the kmem code to apply limits to only task stack. Especially in terms of maintainance.
Also you guys have very good mm kernel developers who are already familiar with this.
I would look at this from a Usability point of view. It is a lot easier to understand number of processes then the mount of KMEM those processes will need. Setting something like
ProcessLimit=1000 in a systemd unit file is easy to explain. Now if systemd has the ability to translate this into something that makes sense in terms of kmem cgroup, then my argument goes away.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 27 lines · Daniel J Walsh
Quoted 25 lines · Frederic Weisbecker
Quoted 3 lines · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work. It was something like 2 YEARS since we first wanted this, and it STILL does not work.
When I was working on the task counter cgroup subsystem 2 years ago, the patches were actually pushed back by google people, in favour of task stack kmem cgroup subsystem.
The reason was that expressing the forkbomb issue in terms of number of tasks as a resource is awkward and that the real resource in the game comes from kernel memory exhaustion due to task stack being allocated over and over, swap ping-pong and stuffs...
And that was a pretty good argument. I still agree with that. Especially since that could solve others people issues at the same time. kmem cgroup has a quite large domain of application.
Quoted 4 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want. What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
The request is simple but I don't think that adding the task counter cgroup subsystem is simpler than extending the kmem code to apply limits to only task stack. Especially in terms of maintainance.
Quoted 6 lines
Quoted 2 lines
Also you guys have very good mm kernel developers who are already familiar with this.
I would look at this from a Usability point of view. It is a lot easier to understand number of processes then the mount of KMEM those processes will need. Setting something like
ProcessLimit=1000 in a systemd unit file is easy to explain.
Yeah that's a fair point.
Quoted 2 lines
Now if systemd has the ability to translate this into something that makes sense in terms of kmem cgroup, then my argument goes away.
Yeah if we keep the kmem direction, this can be a place where we do the mapping. Now I just hope the amount of stack memory allocated doesn't differ too much per arch.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 27 lines · Daniel J Walsh
Quoted 25 lines · Frederic Weisbecker
Quoted 3 lines · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work. It was something like 2 YEARS since we first wanted this, and it STILL does not work.
When I was working on the task counter cgroup subsystem 2 years ago, the patches were actually pushed back by google people, in favour of task stack kmem cgroup subsystem.
The reason was that expressing the forkbomb issue in terms of number of tasks as a resource is awkward and that the real resource in the game comes from kernel memory exhaustion due to task stack being allocated over and over, swap ping-pong and stuffs...
And that was a pretty good argument. I still agree with that. Especially since that could solve others people issues at the same time. kmem cgroup has a quite large domain of application.
Quoted 4 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want. What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
The request is simple but I don't think that adding the task counter cgroup subsystem is simpler than extending the kmem code to apply limits to only task stack. Especially in terms of maintainance.
Quoted 6 lines
Quoted 2 lines
Also you guys have very good mm kernel developers who are already familiar with this.
I would look at this from a Usability point of view. It is a lot easier to understand number of processes then the mount of KMEM those processes will need. Setting something like
ProcessLimit=1000 in a systemd unit file is easy to explain.
Yeah that's a fair point.
Quoted 2 lines
Now if systemd has the ability to translate this into something that makes sense in terms of kmem cgroup, then my argument goes away.
Yeah if we keep the kmem direction, this can be a place where we do the mapping. Now I just hope the amount of stack memory allocated doesn't differ too much per arch.
A message that is not in the archive (sent off-list or not yet received): 20140429214454.GF6129-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 21 lines · Frederic Weisbecker
Quoted 3 lines · Tim Hockin
Here's the reason it doesn't work for us: It doesn't work. It was something like 2 YEARS since we first wanted this, and it STILL does not work.
When I was working on the task counter cgroup subsystem 2 years ago, the patches were actually pushed back by google people, in favour of task stack kmem cgroup subsystem.
The reason was that expressing the forkbomb issue in terms of number of tasks as a resource is awkward and that the real resource in the game comes from kernel memory exhaustion due to task stack being allocated over and over, swap ping-pong and stuffs...
And that was a pretty good argument. I still agree with that. Especially since that could solve others people issues at the same time. kmem cgroup has a quite large domain of application.
Quoted 4 lines
You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want. What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 6 lines
The request is simple but I don't think that adding the task counter cgroup subsystem is simpler than extending the kmem code to apply limits to only task stack. Especially in terms of maintainance.
Also you guys have very good mm kernel developers who are already familiar with this.
I would look at this from a Usability point of view. It is a lot easier to understand number of processes then the mount of KMEM those processes will need. Setting something like
ProcessLimit=1000 in a systemd unit file is easy to explain. Now if systemd has the ability to translate this into something that makes sense in terms of kmem cgroup, then my argument goes away.
A message that is not in the archive (sent off-list or not yet received): 20140429165114.GE6129-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Here's the reason it doesn't work for us: It doesn't work. It was something like 2 YEARS since we first wanted this, and it STILL does not work. You're postponing a pretty simple request indefinitely in favor of a much more complex feature, which still doesn't really give me what I want. What I want is an API that works like rlimit but per-cgroup, rather than per-UID.
Quoted 26 lines · Frederic Weisbecker
Quoted 3 lines · Tim Hockin
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does but it brings other problems that aren't fixable in kmem itself, we can certainly reconsider this cgroup subsystem. But I haven't yet seen argument of this kind yet.
Quoted 6 lines
Is there some specific technical reason why you think this is a bad idea?
I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).Because it's core code and it adds complications and overhead in the fork/exit path. We just don't add new core code just for the sake of slightly prettier interfaces.
A message that is not in the archive (sent off-list or not yet received): CAO_RewYZDGLBAKit4CudTbqVk+zfDRX8kP0W6Zz90xJh7abM9Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 3 lines · Tim Hockin
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
It's not a side effect, the kmem task stack control was partly motivated to solve forkbomb issues in containers.
Also in general if we can reuse existing features and code to solve a problem without disturbing side issues, we just do it.
Now if kmem doesn't solve the issue for you for any reason, or it does but it brings other problems that aren't fixable in kmem itself, we can certainly reconsider this cgroup subsystem. But I haven't yet seen argument of this kind yet.
Quoted 6 lines
Is there some specific technical reason why you think this is a bad idea?
I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).
Because it's core code and it adds complications and overhead in the fork/exit path. We just don't add new core code just for the sake of slightly prettier interfaces.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Why the insistence that we manage something that REALLY IS a first-class concept (hey, it has it's own RLIMIT) as a side effect of something that doesn't quite capture what we want to achieve?
Is there some specific technical reason why you think this is a bad idea? I would think, especially in a more unified hierarchy world, that more cgroup controllers with smaller sets of responsibility would make for more manageable code (within limits, obviously).
Quoted 67 lines · Michal Hocko
Quoted 44 lines · Serge Hallyn
Quoted 43 lines · Michal Hocko
Quoted 41 lines · Serge Hallyn
Quoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 8 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
Quoted 7 lines
Quoted 3 lines
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
It has nothing to do with the state of the limit. I simply don't believe that emulating
RLIMIT_NPROCby controlling stack size is a good idea.I was not the one who decided that the kmem extension of memory controller should cover also the task number as a side effect but still the decision sounds plausible to me because the kmem approach is more generic.
Btw. if this is a problem them please go ahead and continue the original discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the other people involved.
I do not see any new arguments here, except that the kmem implementation is not ready yet.
--
Michal Hocko
SUSE Labs
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Serge HallynQuoted 44 lines · Serge Hallyn
Quoted 43 lines · Michal Hocko
Quoted 41 lines · Serge Hallyn
Quoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 8 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
Quoted 7 lines
Quoted 3 lines
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
It has nothing to do with the state of the limit. I simply don't believe that emulating
RLIMIT_NPROCby controlling stack size is a good idea.
I was not the one who decided that the kmem extension of memory controller should cover also the task number as a side effect but still the decision sounds plausible to me because the kmem approach is more generic.
Btw. if this is a problem them please go ahead and continue the original discussion (http://marc.info/?l=linux-kernel&m=133417075309923) with the other people involved.
I do not see any new arguments here, except that the kmem implementation is not ready yet.
Signature · 2 lines
Michal Hocko SUSE Labs
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 39 lines · Michal Hocko
Quoted 38 lines · Serge Hallyn
Quoted 36 lines · Dwight Engen
Quoted 22 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 21 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 14 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 6 lines
Quoted 2 lines
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14
Quoted 7 lines
Quoted 3 lines
Thanks, Dwight. FWIW I'm agreed with Tim, Dwight, Richard, and Marian that a task limit would be a proper cgroup extension, and specifically that approximating that with a kmem limit is not a reasonable substitute.
The current state of the kmem limit, which is improving a lot thanks to Vladimir, is not a reason for a new extension/controller. We are just not yet there.
It has nothing to do with the state of the limit. I simply don't believe that emulating RLIMIT_NPROC by controlling stack size is a good idea.
-serge
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Dwight EngenQuoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov <[email protected]> wrote:
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 9 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14
Guys I tested the patches with 3.12.16. However I see a problem with them.
Trying to set the limit to a cgroup which already have processes in it does not work:
[root@sp2 lxc]# echo 50 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]# echo 0 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]#
I have even tried to remove this check:
+ if (cgroup_task_count(cgrp) || !list_empty(&cgrp->children))
+ return -EBUSY;
But still give me 'Device or resource busy'.
Any pointers of why is this happening ?
Marian
Quoted 14 lines
Quoted 13 lines
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 35 lines · On Tue, 06 May 2014 14:40:55 +0300 Marian Marinov <[email protected]> wrote:
Quoted 34 lines · Dwight Engen
Quoted 22 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov <[email protected]> wrote:
Quoted 21 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 14 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 6 lines
Quoted 2 lines
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
Quoted 8 lines
Quoted 3 lines
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Guys I tested the patches with 3.12.16. However I see a problem with them.
Trying to set the limit to a cgroup which already have processes in it does not work:
This is a similar check/limitation to the one for kmem in memcg, and is done here to keep the res_counters consistent and from going negative. It could probably be relaxed slightly by using res_counter_set_limit() instead, but you would still need to initially set a limit before adding tasks to the group.
Quoted 8 lines
[root@sp2 lxc]# echo 50 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]# echo 0 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]#I have even tried to remove this check:
+ if (cgroup_task_count(cgrp)
Quoted 29 lines
|| !list_empty(&cgrp->children)) + return -EBUSY;But still give me 'Device or resource busy'.
Any pointers of why is this happening ?
Marian
Quoted 20 lines
Quoted 13 lines
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 44 lines · Dwight Engen
Quoted 41 lines · On Tue, 06 May 2014 14:40:55 +0300 Marian Marinov…
Quoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 9 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Guys I tested the patches with 3.12.16. However I see a problem with them.
Quoted 8 lines
Quoted 2 lines
Trying to set the limit to a cgroup which already have processes in it does not work:
This is a similar check/limitation to the one for kmem in memcg, and is done here to keep the
res_countersconsistent and from going negative. It could probably be relaxed slightly by usingres_counter_set_limit()instead, but you would still need to initially set a limit before adding tasks to the group.
I have removed the check entirely and still receive the EBUSY... I just don't understand what is returning it. If you have any pointers, I would be happy to take a look.
I'll look at set_limit(), thanks for pointing that one.
What I'm proposing is the following checks:
if (val > RES_COUNTER_MAX || val < 0)
return -EBUSY;
if (val != 0 && val <= cgroup_task_count(cgrp))
return -EBUSY;
res_counter_write_u64(&ca->task_limit, type, val);This way we ensure that val is within the limits > 0 and < RES_COUNTER_MAX. And also allow only values of 0 or greater then the current task count.
Marian
Quoted 44 lines
Quoted 9 lines
[root@sp2 lxc]# echo 50 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]# echo 0 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]#I have even tried to remove this check:
+ if (cgroup_task_count(cgrp)Quoted 29 lines
|| !list_empty(&cgrp->children)) + return -EBUSY;But still give me 'Device or resource busy'.
Any pointers of why is this happening ?
Marian
Quoted 20 lines
Quoted 13 lines
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
Signature · 5 lines
Marian Marinov Founder & CEO of 1H Ltd. Jabber/GTalk: hackman-/[email protected] ICQ: 7556201 Mobile: +359 886 660 270
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 69 lines · Marian Marinov
Quoted 44 lines · Dwight Engen
Quoted 41 lines · On Tue, 06 May 2014 14:40:55 +0300 Marian Marinov…
Quoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 9 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Guys I tested the patches with 3.12.16. However I see a problem with them.
Quoted 8 lines
Quoted 2 lines
Trying to set the limit to a cgroup which already have processes in it does not work:
This is a similar check/limitation to the one for kmem in memcg, and is done here to keep the
res_countersconsistent and from going negative. It could probably be relaxed slightly by usingres_counter_set_limit()instead, but you would still need to initially set a limit before adding tasks to the group.I have removed the check entirely and still receive the EBUSY... I just don't understand what is returning it. If you have any pointers, I would be happy to take a look.
I'll look at
set_limit(), thanks for pointing that one.What I'm proposing is the following checks:
if (val > RES_COUNTER_MAX || val < 0) return -EBUSY; if (val != 0 && val <= cgroup_task_count(cgrp)) return -EBUSY; res_counter_write_u64(&ca->task_limit, type, val);
Quoted 3 lines
This way we ensure that val is within the limits > 0 and <
RES_COUNTER_MAX. And also allow only values of 0 or greater then the current task count.
I have also noticed that I can't change many different cgroup limits while there are tasks running in the cgroup - not just cpuacct.task_limit, but also kmem and even normal memory.limit_in_bytes
I would like to be able to change all of these limits, as long as the new limit is greater than the actual current use.
Could a method like this be used for all of the others too?
Richard.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
[root@sp2 lxc]# echo 50 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]# echo 0 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]#I have even tried to remove this check:
+ if (cgroup_task_count(cgrp)|| !list_empty(&cgrp->children)) + return -EBUSY;But still give me 'Device or resource busy'.
Any pointers of why is this happening ?
Marian
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 35 lines · On Tue, 06 May 2014 14:40:55 +0300 Marian Marinov…
Quoted 34 lines · Dwight Engen
Quoted 22 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 21 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 14 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 6 lines
Quoted 2 lines
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
Quoted 8 lines
Quoted 3 lines
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14Guys I tested the patches with 3.12.16. However I see a problem with them.
Trying to set the limit to a cgroup which already have processes in it does not work:
This is a similar check/limitation to the one for kmem in memcg, and is done here to keep the res_counters consistent and from going negative. It could probably be relaxed slightly by using res_counter_set_limit() instead, but you would still need to initially set a limit before adding tasks to the group.
Quoted 37 lines
[root@sp2 lxc]# echo 50 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]# echo 0 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]#I have even tried to remove this check:
+ if (cgroup_task_count(cgrp)|| !list_empty(&cgrp->children)) + return -EBUSY;But still give me 'Device or resource busy'.
Any pointers of why is this happening ?
Marian
Quoted 20 lines
Quoted 13 lines
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 9 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14
Guys I tested the patches with 3.12.16. However I see a problem with them.
Trying to set the limit to a cgroup which already have processes in it does not work:
[root@sp2 lxc]# echo 50 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]# echo 0 > cpuacct.task_limit
-bash: echo: write error: Device or resource busy
[root@sp2 lxc]#
I have even tried to remove this check:
+ if (cgroup_task_count(cgrp) || !list_empty(&cgrp->children))
+ return -EBUSY;
But still give me 'Device or resource busy'.
Any pointers of why is this happening ?
Marian
Quoted 20 lines
Quoted 13 lines
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to a message not in the archiveQuoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 8 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14
I did a backport of the patches to 3.12.16 and forward ported them to 3.12.20.
I'm very happy with how they work.
I used the patches on machines with 10-20k processes and it worked perfectly when some of the containers spawned 100s of processes. It really saved us when one of the containers was attacked :)
The only thing that I'm going to add is on the fly change of the limit.
Marian
Quoted 15 lines
Quoted 14 lines
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Dwight EngenQuoted 28 lines · Dwight Engen
Quoted 25 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 23 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 7 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Quoted 8 lines
Quoted 3 lines
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.gitcpuacct-task-limit-3.13git://github.com/dwengen/linux.gitcpuacct-task-limit-3.14
I did a backport of the patches to 3.12.16 and forward ported them to 3.12.20.
I'm very happy with how they work.
I used the patches on machines with 10-20k processes and it worked perfectly when some of the containers spawned 100s of processes. It really saved us when one of the containers was attacked :)
The only thing that I'm going to add is on the fly change of the limit.
Marian
Quoted 21 lines
Quoted 14 lines
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 22 lines · On Wed, 23 Apr 2014 09:07:28 +0300 Marian Marinov…
Quoted 21 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 14 lines
Quoted 6 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Quoted 6 lines
Quoted 2 lines
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Dwight, do you have these patches in any public repo?
I would like to test them also.
Hi Marian, I put the patches against 3.13.11 and 3.14.1 up at:
git://github.com/dwengen/linux.git cpuacct-task-limit-3.13git://github.com/dwengen/linux.git cpuacct-task-limit-3.14
Quoted 13 lines
Marian
Quoted 10 lines
Thanks,
Richard.
--
To unsubscribe from this list: send the line "unsubscribe cgroups" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
In reply to Richard DaviesQuoted 15 lines · Richard Davies
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 9 lines
Quoted 5 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
Quoted 7 lines
Quoted 4 lines
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Any chance for a 3.15 rebase, since the changes from cgroup_fork() makes the operation no longer trivial.
Cheers,
Alin.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 4 lines · Dwight Engen
Quoted 3 lines · Richard Davies
Quoted 2 lines · Vladimir Davydov
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
...
Quoted 6 lines
Quoted 5 lines
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROCexists but is per-user, not per-container.Perhaps there is an up-to-date task counter patchset or similar?
Quoted 7 lines
I updated Frederic's task counter patches and included Max Kellermann's fork limiter here:
http://thread.gmane.org/gmane.linux.kernel.containers/27212
I can send you a more recent patchset (against 3.13.10) if you would find it useful.
Yes please, I would be interested in that. Ideally even against 3.14.1 if you have that too.
Thanks,
Richard.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Protection against container fork bombs [WAS: Re: memcg with kmem limit doesn't recover after disk i/o causes limit to be hit]
Quoted 9 lines · Vladimir Davydov
Quoted 4 lines · Richard Davies
I have a simple reproducible test case in which untar in a memcg with a kmem limit gets into trouble during heavy disk i/o (on ext3) and never properly recovers. This is simplified from real world problems with heavy disk i/o inside containers.
Unfortunately, work on per cgroup kmem limits is not completed yet. Currently it lacks kmem reclaim on per cgroup memory pressure, which is vital for using kmem limits in real life.
...
Quoted 2 lines
In short, kmem limiting for memory cgroups is currently broken. Do not use it. We are working on making it usable though.
Thanks for explaining the strange errors I got.
My motivation is to prevent a fork bomb in a container from affecting other processes outside that container.
kmem limits were the preferred mechanism in several previous discussions about two years ago (I'm copying in participants from those previous discussions and give links below). So I tried kmem first but found bugs.
What is the best mechanism available today, until kmem limits mature?
RLIMIT_NPROC exists but is per-user, not per-container.
Perhaps there is an up-to-date task counter patchset or similar?
Thank you all,
Richard.
Some references to previous discussions:
Fork bomb limitation in memcg WAS: Re: [PATCH 00/11] kmem controller for memcg: stripped down version
http://thread.gmane.org/gmane.linux.kernel/1318266/focus=1319372
Re: [PATCH 00/10] cgroups: Task counter subsystem v8
http://thread.gmane.org/gmane.linux.kernel/1246704/focus=1467310
[RFD] Merge task counter into memcg
http://thread.gmane.org/gmane.linux.kernel/1280302
Re: [PATCH -mm] cgroup: Fix task counter common ancestor logic
http://thread.gmane.org/gmane.linux.kernel/1212650/focus=1220186
[PATCH] new cgroup controller "fork"
http://thread.gmane.org/gmane.linux.kernel/1210878
Re: Process Limit cgroups
http://thread.gmane.org/gmane.linux.kernel.cgroups/9368/focus=9369
Re: [lxc-devel] process number limit
https://www.mail-archive.com/[email protected]/msg03309.html