3.4-rc4 oom killer out of control.
Subsystems (3)
On a test machine that was running my system call fuzzer, I just saw the oom killer take out everything but the process that was doing all the memory exhausting.
Partial logs below. The machine locked up completely (even capslock wouldn't work). The console had logs up to 5041.xxxxxx before the wedge, but they never made it to disk. It was just more of the same below..
Note that the trinity processes have largest RSS, yet seem immune to getting killed.
Dave
Apr 26 13:56:45 dhcp-189-232 kernel: [ 4959.198003] modprobe invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.198349] modprobe cpuset=/ mems_allowed=0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.198705] Pid: 14950, comm: modprobe Not tainted 3.4.0-rc4+ #55 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.199406] Call Trace: Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.200214] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.201084] [<ffffffff816a32a7>] dump_header+0x83/0x2eb Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.201928] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.202814] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.203687] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.204539] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.205416] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.206277] [<ffffffff8119d366>] alloc_pages_current+0xb6/0x120 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.207142] [<ffffffff811569e7>] __page_cache_alloc+0xb7/0xe0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.207999] [<ffffffff81159bbf>] filemap_fault+0x2ff/0x4c0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.208892] [<ffffffff8117df8f>] __do_fault+0x6f/0x540 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.209752] [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.210633] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.211509] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.212384] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.213475] [<ffffffff81335a1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.214137] [<ffffffff816af019>] ? retint_swapgs+0x13/0x1b Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.215018] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.215886] [<ffffffff816af2e5>] page_fault+0x25/0x30 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.216722] Mem-Info: Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.217608] Node 0 DMA per-cpu: Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.218500] CPU 0: hi: 0, btch: 1 usd: 0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.219400] CPU 1: hi: 0, btch: 1 usd: 0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.220292] CPU 2: hi: 0, btch: 1 usd: 0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.221186] CPU 3: hi: 0, btch: 1 usd: 0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.222067] Node 0 DMA32 per-cpu: Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.222937] CPU 0: hi: 186, btch: 31 usd: 16 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.223833] CPU 1: hi: 186, btch: 31 usd: 7 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.224692] CPU 2: hi: 186, btch: 31 usd: 1 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.225554] CPU 3: hi: 186, btch: 31 usd: 9 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.226408] Node 0 Normal per-cpu: Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.227253] CPU 0: hi: 186, btch: 31 usd: 2 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.228095] CPU 1: hi: 186, btch: 31 usd: 65 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.228918] CPU 2: hi: 186, btch: 31 usd: 33 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.229767] CPU 3: hi: 186, btch: 31 usd: 21 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230600] active_anon:1421942 inactive_anon:313560 isolated_anon:0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230604] active_file:88 inactive_file:120 isolated_file:70 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230608] unevictable:3038 dirty:7 writeback:0 unstable:0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230610] free:25574 slab_reclaimable:14238 slab_unreclaimable:101544 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230611] mapped:232 shmem:86 pagetables:89711 bounce:0 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.234745] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.237548] lowmem_reserve[]: 0 3246 8034 8034 Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.238522] Node 0 DMA32 free:46340kB min:27252kB low:34064kB high:40876kB active_anon:2330464kB inactive_anon:582676kB active_file:16kB inactive_file:44kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:112kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132232kB kernel_stack:2272kB pagetables:133664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:42 all_unreclaimable? yes Apr 26 13:56:50 dhcp-189-232 kernel: [ 4959.241635] lowmem_reserve[]: 0 0 4788 4788 Apr 26 13:57:09 dhcp-189-232 kernel: [ 4959.242701] Node 0 Normal free:40080kB min:40196kB low:50244kB high:60292kB active_anon:3357304kB inactive_anon:671564kB active_file:336kB inactive_file:436kB unevictable:7404kB isolated(anon):0kB isolated(file):280kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:816kB shmem:240kB slab_reclaimable:54120kB slab_unreclaimable:273912kB kernel_stack:4528kB pagetables:225180kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:970 all_unreclaimable? yes Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.246216] lowmem_reserve[]: 0 0 0 0 Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.247423] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.248684] Node 0 DMA32: 217*4kB 152*8kB 158*16kB 232*32kB 134*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46340kB Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.249873] Node 0 Normal: 475*4kB 405*8kB 272*16kB 352*32kB 144*64kB 40*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 39956kB Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.251173] 2037 total pagecache pages Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.252403] 1692 pages in swap cache Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.253630] Swap cache stats: add 1828870, delete 1827178, find 493465/493919 Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.254887] Free swap = 0kB Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.256126] Total swap = 1023996kB Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.347525] 2097136 pages RAM Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.348055] 59776 pages reserved Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.348865] 2554149 pages shared Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.349991] 2007673 pages non-shared Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.351168] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.352382] [ 353] 0 353 4425 23 2 -17 -1000 udevd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.353575] [ 355] 0 355 5802 1 2 0 0 systemd-stdout- Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.354796] [ 737] 0 737 4766 1 3 0 0 smartd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.356016] [ 751] 0 751 29591 24 0 0 0 crond Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.357246] [ 755] 0 755 1616 1 1 0 0 acpid Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.358447] [ 756] 0 756 67222 107 0 0 0 NetworkManager Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.359656] [ 770] 0 770 7083 1 0 0 0 systemd-logind Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.360879] [ 773] 70 773 7005 50 3 0 0 avahi-daemon Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.362082] [ 776] 70 776 6972 4 0 0 0 avahi-daemon Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.363301] [ 778] 0 778 1612 1 0 0 0 mcelog Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.364526] [ 783] 994 783 4893 24 0 0 0 chronyd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.365748] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.366862] [ 790] 0 790 29064 39 0 0 0 ksmtuned Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.368085] [ 797] 0 797 46610 31 1 -13 -900 polkitd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.369305] [ 802] 0 802 60782 52 1 0 0 rsyslogd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.370527] [ 815] 0 815 18861 1 0 -17 -1000 sshd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.371756] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.372995] [ 819] 0 819 4792 17 2 0 0 rpcbind Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.374229] [ 824] 29 824 6938 1 0 0 0 rpc.statd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.375462] [ 837] 0 837 27232 1 3 0 0 agetty Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.376695] [ 842] 0 842 21549 1 2 0 0 dhclient Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.377938] [ 1170] 1000 1170 6134 81 0 0 0 tmux Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.379192] [ 1171] 1000 1171 29914 1 2 0 0 bash Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.380455] [ 1262] 1000 1262 4022 125 3 0 0 trinity Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.381709] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.382928] [15065] 0 15065 3725 87 3 0 0 anacron Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.384164] [19925] 0 19925 4424 1 3 -17 -1000 udevd Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.385449] [13786] 0 13786 26704 68 2 0 0 sleep Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.386748] [14941] 0 14941 1629 87 0 0 0 modprobe Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.388042] [14944] 0 14944 1628 63 0 0 0 modprobe Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.389325] [14945] 1000 14945 4418 639 3 0 0 trinity Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.390618] [14950] 0 14950 106 3 2 0 0 modprobe Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.391913] [14952] 1000 14952 4022 203 0 0 0 trinity Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.393252] [14955] 1000 14955 4022 124 0 0 0 trinity Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.394560] Out of memory: Kill process 355 (systemd-stdout-) score 1 or sacrifice child Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.395847] Killed process 355 (systemd-stdout-) total-vm:23208kB, anon-rss:0kB, file-rss:4kB Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.358425] systemd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.358956] systemd cpuset=/ mems_allowed=0 Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.359956] Pid: 1, comm: systemd Not tainted 3.4.0-rc4+ #55 Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.361271] Call Trace: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.362557] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.363827] [<ffffffff816a32a7>] dump_header+0x83/0x2eb Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.365052] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.366208] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.367393] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.368532] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.369673] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.370780] [<ffffffff8119d366>] alloc_pages_current+0xb6/0x120 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.371855] [<ffffffff811569e7>] __page_cache_alloc+0xb7/0xe0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.372910] [<ffffffff81159bbf>] filemap_fault+0x2ff/0x4c0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.373935] [<ffffffff8117df8f>] __do_fault+0x6f/0x540 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.374939] [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.375917] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.376883] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.377826] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.378765] [<ffffffff8120f046>] ? sys_epoll_wait+0x96/0x470 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.379700] [<ffffffff816b6b15>] ? sysret_check+0x22/0x5d Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.380620] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.381549] [<ffffffff816af2e5>] page_fault+0x25/0x30 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.382409] Mem-Info: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.383329] Node 0 DMA per-cpu: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.384259] CPU 0: hi: 0, btch: 1 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.385200] CPU 1: hi: 0, btch: 1 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.386123] CPU 2: hi: 0, btch: 1 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.387024] CPU 3: hi: 0, btch: 1 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.387898] Node 0 DMA32 per-cpu: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.388802] CPU 0: hi: 186, btch: 31 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.389691] CPU 1: hi: 186, btch: 31 usd: 30 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.390587] CPU 2: hi: 186, btch: 31 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.391466] CPU 3: hi: 186, btch: 31 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.392334] Node 0 Normal per-cpu: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.393198] CPU 0: hi: 186, btch: 31 usd: 75 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.394078] CPU 1: hi: 186, btch: 31 usd: 30 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.394934] CPU 2: hi: 186, btch: 31 usd: 22 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.395810] CPU 3: hi: 186, btch: 31 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396663] active_anon:1421870 inactive_anon:313545 isolated_anon:0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396664] active_file:103 inactive_file:50 isolated_file:64 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396665] unevictable:3038 dirty:7 writeback:0 unstable:0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396666] free:25649 slab_reclaimable:14238 slab_unreclaimable:101538 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396668] mapped:237 shmem:86 pagetables:89658 bounce:0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.400859] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.403671] lowmem_reserve[]: 0 3246 8034 8034 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.404653] Node 0 DMA32 free:46192kB min:27252kB low:34064kB high:40876kB active_anon:2330460kB inactive_anon:582672kB active_file:0kB inactive_file:96kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:108kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132232kB kernel_stack:2272kB pagetables:133664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.407764] lowmem_reserve[]: 0 0 4788 4788 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.408824] Node 0 Normal free:40528kB min:40196kB low:50244kB high:60292kB active_anon:3357020kB inactive_anon:671508kB active_file:412kB inactive_file:104kB unevictable:7404kB isolated(anon):0kB isolated(file):256kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:840kB shmem:240kB slab_reclaimable:54120kB slab_unreclaimable:273888kB kernel_stack:4528kB pagetables:224968kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:23 all_unreclaimable? no Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.412250] lowmem_reserve[]: 0 0 0 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.413420] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.414664] Node 0 DMA32: 226*4kB 155*8kB 160*16kB 229*32kB 134*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46336kB Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.415839] Node 0 Normal: 542*4kB 428*8kB 273*16kB 353*32kB 145*64kB 40*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40520kB Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.417112] 2026 total pagecache pages Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.418352] 1685 pages in swap cache Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.419591] Swap cache stats: add 1828918, delete 1827233, find 493469/493923 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.420839] Free swap = 0kB Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.422068] Total swap = 1023996kB Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.502902] 2097136 pages RAM Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.503469] 59776 pages reserved Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.504253] 2553433 pages shared Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.505446] 2007548 pages non-shared Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.506614] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.507848] [ 353] 0 353 4425 23 2 -17 -1000 udevd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.509055] [ 737] 0 737 4766 1 3 0 0 smartd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.510247] [ 751] 0 751 29591 24 0 0 0 crond Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.511462] [ 755] 0 755 1616 1 1 0 0 acpid Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.512665] [ 756] 0 756 67222 107 1 0 0 NetworkManager Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.513879] [ 770] 0 770 7083 1 0 0 0 systemd-logind Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.515077] [ 773] 70 773 7005 50 3 0 0 avahi-daemon Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.516211] [ 776] 70 776 6972 4 0 0 0 avahi-daemon Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.517431] [ 778] 0 778 1612 1 0 0 0 mcelog Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.518630] [ 783] 994 783 4893 24 0 0 0 chronyd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.519848] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.521071] [ 790] 0 790 29064 39 0 0 0 ksmtuned Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.522285] [ 797] 0 797 46610 31 2 -13 -900 polkitd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.523503] [ 802] 0 802 60782 52 1 0 0 rsyslogd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.524745] [ 815] 0 815 18861 1 0 -17 -1000 sshd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.525972] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.527201] [ 819] 0 819 4792 17 2 0 0 rpcbind Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.528428] [ 824] 29 824 6938 1 0 0 0 rpc.statd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.529659] [ 837] 0 837 27232 1 3 0 0 agetty Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.530899] [ 842] 0 842 21549 1 2 0 0 dhclient Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.532097] [ 1170] 1000 1170 6134 78 2 0 0 tmux Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.533306] [ 1171] 1000 1171 29914 1 2 0 0 bash Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.534547] [ 1262] 1000 1262 4022 125 1 0 0 trinity Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.535816] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.537076] [15065] 0 15065 3725 87 3 0 0 anacron Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.538337] [19925] 0 19925 4424 1 3 -17 -1000 udevd Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.539605] [13786] 0 13786 26704 68 2 0 0 sleep Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.540888] [14941] 0 14941 1629 87 1 0 0 modprobe Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.542179] [14944] 0 14944 1628 63 1 0 0 modprobe Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.543473] [14945] 1000 14945 4418 652 2 0 0 trinity Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.544756] [14950] 0 14950 106 7 3 0 0 modprobe Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.546051] [14955] 1000 14955 4022 124 2 0 0 trinity Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.547353] Out of memory: Kill process 737 (smartd) score 1 or sacrifice child Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.548644] Killed process 737 (smartd) total-vm:19064kB, anon-rss:0kB, file-rss:4kB Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.791375] trinity invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.791871] trinity cpuset=/ mems_allowed=0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.792948] Pid: 14972, comm: trinity Not tainted 3.4.0-rc4+ #55 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.794230] Call Trace: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.795559] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.796866] [<ffffffff816a32a7>] dump_header+0x83/0x2eb Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.798155] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.799432] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.800679] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.801892] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.803092] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.804244] [<ffffffff8119fdb3>] alloc_pages_vma+0xb3/0x190 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.805393] [<ffffffff81181300>] handle_pte_fault+0x6a0/0xa10 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.806510] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807590] [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807596] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807602] [<ffffffff811b4f68>] do_huge_pmd_anonymous_page+0xc8/0x380 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807607] [<ffffffff81181976>] handle_mm_fault+0x146/0x2f0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807610] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807614] [<ffffffff811871d6>] ? do_brk+0x246/0x360 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807619] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807623] [<ffffffff816af2e5>] page_fault+0x25/0x30 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807627] Mem-Info: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807630] Node 0 DMA per-cpu: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807633] CPU 0: hi: 0, btch: 1 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807637] CPU 1: hi: 0, btch: 1 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807639] CPU 2: hi: 0, btch: 1 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807641] CPU 3: hi: 0, btch: 1 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807643] Node 0 DMA32 per-cpu: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807646] CPU 0: hi: 186, btch: 31 usd: 1 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807648] CPU 1: hi: 186, btch: 31 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807651] CPU 2: hi: 186, btch: 31 usd: 3 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807653] CPU 3: hi: 186, btch: 31 usd: 1 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807655] Node 0 Normal per-cpu: Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807658] CPU 0: hi: 186, btch: 31 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807660] CPU 1: hi: 186, btch: 31 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807662] CPU 2: hi: 186, btch: 31 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807664] CPU 3: hi: 186, btch: 31 usd: 0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807670] active_anon:1421827 inactive_anon:313613 isolated_anon:0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807672] active_file:63 inactive_file:73 isolated_file:110 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807673] unevictable:3038 dirty:7 writeback:0 unstable:0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807674] free:25681 slab_reclaimable:14174 slab_unreclaimable:101572 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807675] mapped:229 shmem:86 pagetables:89725 bounce:0 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807677] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807687] lowmem_reserve[]: 0 3246 8034 8034 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807692] Node 0 DMA32 free:46508kB min:27252kB low:34064kB high:40876kB active_anon:2330216kB inactive_anon:582636kB active_file:24kB inactive_file:28kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132256kB kernel_stack:2296kB pagetables:133680kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:83 all_unreclaimable? yes Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807702] lowmem_reserve[]: 0 0 4788 4788 Apr 26 13:57:14 dhcp-189-232 systemd[1]: systemd-stdout-syslog-bridge.service: main process exited, code=killed, status=9 Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807706] Node 0 Normal free:40340kB min:40196kB low:50244kB high:60292kB active_anon:3357092kB inactive_anon:671816kB active_file:228kB inactive_file:264kB unevictable:7404kB isolated(anon):0kB isolated(file):440kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:812kB shmem:240kB slab_reclaimable:53864kB slab_unreclaimable:274000kB kernel_stack:4576kB pagetables:225220kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:770 all_unreclaimable? yes Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807717] lowmem_reserve[]: 0 0 0 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807720] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807731] Node 0 DMA32: 219*4kB 156*8kB 166*16kB 230*32kB 135*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46508kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807742] Node 0 Normal: 517*4kB 412*8kB 266*16kB 350*32kB 145*64kB 42*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40340kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807752] 1526 total pagecache pages Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807754] 1249 pages in swap cache Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807757] Swap cache stats: add 1829703, delete 1828454, find 493950/494458 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807759] Free swap = 0kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807760] Total swap = 1023996kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.924517] 2097136 pages RAM Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.925073] 59776 pages reserved Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.925829] 2551230 pages shared Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.927017] 2007729 pages non-shared Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.928103] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.929299] [ 353] 0 353 4425 23 2 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.930485] [ 751] 0 751 29591 24 0 0 0 crond Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.931669] [ 755] 0 755 1616 1 1 0 0 acpid Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.932860] [ 756] 0 756 67222 107 1 0 0 NetworkManager Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.934065] [ 770] 0 770 7083 1 0 0 0 systemd-logind Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.935251] [ 773] 70 773 7005 50 3 0 0 avahi-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.936454] [ 776] 70 776 6972 4 0 0 0 avahi-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.937669] [ 778] 0 778 1612 1 0 0 0 mcelog Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.938838] [ 783] 994 783 4893 24 0 0 0 chronyd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.940037] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.941223] [ 790] 0 790 29064 39 0 0 0 ksmtuned Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.942428] [ 797] 0 797 46610 31 0 -13 -900 polkitd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.943620] [ 802] 0 802 60782 59 1 0 0 rsyslogd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.944740] [ 815] 0 815 18861 1 0 -17 -1000 sshd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.945960] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.947173] [ 819] 0 819 4792 17 3 0 0 rpcbind Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.948373] [ 824] 29 824 6938 1 0 0 0 rpc.statd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.949594] [ 837] 0 837 27232 1 3 0 0 agetty Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.950802] [ 842] 0 842 21549 1 2 0 0 dhclient Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.952021] [ 1170] 1000 1170 6133 77 0 0 0 tmux Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.953241] [ 1171] 1000 1171 29914 1 2 0 0 bash Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.954555] [ 1262] 1000 1262 4022 125 2 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.955715] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.956971] [15065] 0 15065 3725 87 3 0 0 anacron Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.958231] [19925] 0 19925 4424 1 3 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.959476] [14944] 0 14944 1629 86 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.960698] [14950] 0 14950 1595 57 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.961949] [14954] 0 14954 1595 36 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.963230] [14958] 0 14958 2810 19 2 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.964494] [14961] 1000 14961 4286 459 1 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.965773] [14965] 0 14965 1618 15 0 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.967069] [14966] 0 14966 2290 19 1 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.968363] [14967] 0 14967 1621 35 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.969669] [14970] 0 14970 1595 58 0 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.970972] [14972] 1000 14972 4220 456 3 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.972269] [14975] 0 14975 104 1 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.973563] Out of memory: Kill process 751 (crond) score 1 or sacrifice child Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.974863] Killed process 751 (crond) total-vm:118364kB, anon-rss:92kB, file-rss:4kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.036332] systemd-cgroups invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.036811] systemd-cgroups cpuset=/ mems_allowed=0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.037772] Pid: 14958, comm: systemd-cgroups Not tainted 3.4.0-rc4+ #55 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.039032] Call Trace: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.040280] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.041507] [<ffffffff816a32a7>] dump_header+0x83/0x2eb Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.042610] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.043793] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.044947] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.046063] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.047156] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.048226] [<ffffffff8119d366>] alloc_pages_current+0xb6/0x120 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.049259] [<ffffffff811569e7>] __page_cache_alloc+0xb7/0xe0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.050282] [<ffffffff81159bbf>] filemap_fault+0x2ff/0x4c0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.051270] [<ffffffff8117df8f>] __do_fault+0x6f/0x540 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.052234] [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.053175] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.054132] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.055081] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.056015] [<ffffffff811c1453>] ? sys_close+0x43/0x1a0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.056940] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.057895] [<ffffffff816af2e5>] page_fault+0x25/0x30 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.058855] Mem-Info: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.059727] Node 0 DMA per-cpu: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.060674] CPU 0: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.061619] CPU 1: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.062547] CPU 2: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.063456] CPU 3: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.064373] Node 0 DMA32 per-cpu: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.065269] CPU 0: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.066178] CPU 1: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.067069] CPU 2: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.067947] CPU 3: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.068854] Node 0 Normal per-cpu: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.069730] CPU 0: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.070587] CPU 1: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.071458] CPU 2: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.072321] CPU 3: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073165] active_anon:1421858 inactive_anon:313540 isolated_anon:0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073166] active_file:94 inactive_file:65 isolated_file:151 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073167] unevictable:3038 dirty:9 writeback:0 unstable:0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073170] free:25544 slab_reclaimable:14174 slab_unreclaimable:101572 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073172] mapped:229 shmem:86 pagetables:89708 bounce:0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.077353] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.080149] lowmem_reserve[]: 0 3246 8034 8034 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.081123] Node 0 DMA32 free:46240kB min:27252kB low:34064kB high:40876kB active_anon:2330228kB inactive_anon:582636kB active_file:28kB inactive_file:36kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:8kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132256kB kernel_stack:2288kB pagetables:133684kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5 all_unreclaimable? no Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.084201] lowmem_reserve[]: 0 0 4788 4788 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.085253] Node 0 Normal free:40060kB min:40196kB low:50244kB high:60292kB active_anon:3357204kB inactive_anon:671524kB active_file:348kB inactive_file:224kB unevictable:7404kB isolated(anon):0kB isolated(file):604kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:812kB shmem:240kB slab_reclaimable:53864kB slab_unreclaimable:274000kB kernel_stack:4600kB pagetables:225148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:28 all_unreclaimable? no Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.088669] lowmem_reserve[]: 0 0 0 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.089845] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.091098] Node 0 DMA32: 204*4kB 143*8kB 161*16kB 231*32kB 136*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46360kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.092319] Node 0 Normal: 565*4kB 415*8kB 265*16kB 350*32kB 145*64kB 42*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40540kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.093613] 1616 total pagecache pages Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.094829] 1336 pages in swap cache Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.096071] Swap cache stats: add 1829817, delete 1828481, find 493950/494458 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.097332] Free swap = 0kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.098596] Total swap = 1023996kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.205067] 2097136 pages RAM Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.205470] 59776 pages reserved Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206413] 2549841 pages shared Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206415] 2007690 pages non-shared Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206418] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206444] [ 353] 0 353 4425 23 2 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206459] [ 755] 0 755 1616 1 1 0 0 acpid Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206465] [ 756] 0 756 67222 107 1 0 0 NetworkManager Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206470] [ 770] 0 770 7083 1 0 0 0 systemd-logind Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206475] [ 773] 70 773 7005 50 3 0 0 avahi-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206480] [ 776] 70 776 6972 4 0 0 0 avahi-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206485] [ 778] 0 778 1612 1 0 0 0 mcelog Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206490] [ 783] 994 783 4893 24 0 0 0 chronyd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206495] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206500] [ 790] 0 790 29064 39 2 0 0 ksmtuned Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206505] [ 797] 0 797 46610 31 1 -13 -900 polkitd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206510] [ 802] 0 802 60782 59 1 0 0 rsyslogd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206515] [ 815] 0 815 18861 1 0 -17 -1000 sshd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206520] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206524] [ 819] 0 819 4792 17 3 0 0 rpcbind Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206529] [ 824] 29 824 6938 1 0 0 0 rpc.statd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206534] [ 837] 0 837 27232 1 3 0 0 agetty Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206539] [ 842] 0 842 21549 1 2 0 0 dhclient Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206544] [ 1170] 1000 1170 6133 77 2 0 0 tmux Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206549] [ 1171] 1000 1171 29914 1 2 0 0 bash Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206554] [ 1262] 1000 1262 4022 125 3 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206559] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206565] [15065] 0 15065 3725 87 3 0 0 anacron Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206573] [19925] 0 19925 4424 1 3 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206581] [14944] 0 14944 1629 86 3 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206585] [14950] 0 14950 1628 63 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206590] [14954] 0 14954 1595 36 3 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206596] [14958] 0 14958 2810 20 0 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206600] [14961] 1000 14961 4286 459 1 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206606] [14965] 0 14965 1618 15 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206610] [14966] 0 14966 2290 19 3 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206615] [14967] 0 14967 1621 35 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206620] [14970] 0 14970 1628 62 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206625] [14972] 1000 14972 4286 529 3 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206631] [14975] 0 14975 106 8 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206636] Out of memory: Kill process 755 (acpid) score 1 or sacrifice child Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206640] Killed process 755 (acpid) total-vm:6464kB, anon-rss:0kB, file-rss:4kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.882053] trinity invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.882827] trinity cpuset=/ mems_allowed=0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.883513] Pid: 14972, comm: trinity Not tainted 3.4.0-rc4+ #55 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.884768] Call Trace: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.886025] [<ffffffff816ae775>] ? _raw_spin_unlock+0x55/0x60 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.887232] [<ffffffff816a32a7>] dump_header+0x83/0x2eb Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.888433] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.889579] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.890703] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.891787] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.892873] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.893940] [<ffffffff8119fdb3>] alloc_pages_vma+0xb3/0x190 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.894969] [<ffffffff81181300>] handle_pte_fault+0x6a0/0xa10 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.895959] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.896952] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.897917] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.898874] [<ffffffff811871d6>] ? do_brk+0x246/0x360 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.899797] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.900733] [<ffffffff816af2e5>] page_fault+0x25/0x30 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.901645] Mem-Info: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.902542] Node 0 DMA per-cpu: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.903447] CPU 0: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.904352] CPU 1: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.905264] CPU 2: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.906166] CPU 3: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.907061] Node 0 DMA32 per-cpu: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.907881] CPU 0: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.908779] CPU 1: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.909655] CPU 2: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.910525] CPU 3: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.911379] Node 0 Normal per-cpu: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.912233] CPU 0: hi: 186, btch: 31 usd: 1 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.913091] CPU 1: hi: 186, btch: 31 usd: 4 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.913930] CPU 2: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.914792] CPU 3: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915639] active_anon:1421866 inactive_anon:313517 isolated_anon:0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915641] active_file:86 inactive_file:127 isolated_file:114 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915642] unevictable:3038 dirty:9 writeback:0 unstable:0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915643] free:25566 slab_reclaimable:14174 slab_unreclaimable:101572 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915644] mapped:229 shmem:86 pagetables:89704 bounce:0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.919846] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.922609] lowmem_reserve[]: 0 3246 8034 8034 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.923570] Node 0 DMA32 free:46384kB min:27252kB low:34064kB high:40876kB active_anon:2330352kB inactive_anon:582628kB active_file:0kB inactive_file:64kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:8kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132256kB kernel_stack:2280kB pagetables:133684kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:228 all_unreclaimable? yes Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.926571] lowmem_reserve[]: 0 0 4788 4788 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.927617] Node 0 Normal free:40004kB min:40196kB low:50244kB high:60292kB active_anon:3357112kB inactive_anon:671440kB active_file:344kB inactive_file:444kB unevictable:7404kB isolated(anon):0kB isolated(file):456kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:812kB shmem:240kB slab_reclaimable:53864kB slab_unreclaimable:274000kB kernel_stack:4600kB pagetables:225132kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.931017] lowmem_reserve[]: 0 0 0 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.932183] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.933420] Node 0 DMA32: 206*4kB 145*8kB 161*16kB 231*32kB 136*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46384kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.934698] Node 0 Normal: 506*4kB 408*8kB 263*16kB 351*32kB 145*64kB 42*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40248kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.935988] 1630 total pagecache pages Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.937219] 1286 pages in swap cache Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.938456] Swap cache stats: add 1829855, delete 1828569, find 493952/494460 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.939701] Free swap = 0kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.940881] Total swap = 1023996kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.024043] 2097136 pages RAM Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.024445] 59776 pages reserved Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.025432] 2548364 pages shared Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.026651] 2007726 pages non-shared Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.027872] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.029123] [ 353] 0 353 4425 23 2 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.030346] [ 756] 0 756 67222 111 0 0 0 NetworkManager Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.031539] [ 770] 0 770 7083 1 0 0 0 systemd-logind Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.032769] [ 773] 70 773 7005 50 3 0 0 avahi-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.034004] [ 776] 70 776 6972 4 0 0 0 avahi-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.035225] [ 778] 0 778 1612 1 0 0 0 mcelog Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.036452] [ 783] 994 783 4893 24 0 0 0 chronyd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.037654] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.038878] [ 790] 0 790 29064 39 1 0 0 ksmtuned Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.040105] [ 797] 0 797 46610 39 2 -13 -900 polkitd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.041252] [ 802] 0 802 60782 59 1 0 0 rsyslogd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.042478] [ 815] 0 815 18861 1 0 -17 -1000 sshd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.043706] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.044954] [ 819] 0 819 4792 17 3 0 0 rpcbind Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.046187] [ 824] 29 824 6938 1 0 0 0 rpc.statd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.047426] [ 837] 0 837 27232 1 3 0 0 agetty Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.048669] [ 842] 0 842 21549 1 2 0 0 dhclient Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.049899] [ 1170] 1000 1170 6133 77 2 0 0 tmux Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.051132] [ 1171] 1000 1171 29914 1 2 0 0 bash Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.052362] [ 1262] 1000 1262 4022 125 2 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.053596] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.054842] [15065] 0 15065 3725 87 3 0 0 anacron Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.056115] [19925] 0 19925 4424 1 3 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.057359] [14944] 0 14944 1629 86 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.058602] [14950] 0 14950 1628 65 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.059853] [14954] 0 14954 1595 36 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.061136] [14958] 0 14958 2810 20 1 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.062414] [14961] 1000 14961 4286 459 0 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.063716] [14965] 0 14965 1618 16 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.065015] [14966] 0 14966 2290 19 2 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.066295] [14967] 0 14967 1595 37 0 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.067591] [14970] 0 14970 1628 65 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.068897] [14972] 1000 14972 4286 521 3 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.070217] [14975] 0 14975 106 8 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.071527] [14979] 1000 14979 4022 125 1 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.072832] Out of memory: Kill process 756 (NetworkManager) score 1 or sacrifice child Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.074110] Killed process 842 (dhclient) total-vm:86196kB, anon-rss:0kB, file-rss:4kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.242452] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.242915] systemd-cgroups cpuset=/ mems_allowed=0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.243952] Pid: 14958, comm: systemd-cgroups Not tainted 3.4.0-rc4+ #55 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.245244] Call Trace: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.246511] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.247766] [<ffffffff816a32a7>] dump_header+0x83/0x2eb Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.248996] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.250204] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.251350] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.252505] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.253627] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.254720] [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.255793] [<ffffffff8119fdb3>] alloc_pages_vma+0xb3/0x190 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.256780] [<ffffffff8117e248>] __do_fault+0x328/0x540 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.257795] [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.258789] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.259769] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.260734] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.261697] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.262650] [<ffffffff816af2e5>] page_fault+0x25/0x30 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.263586] Mem-Info: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.264522] Node 0 DMA per-cpu: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.265473] CPU 0: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.266423] CPU 1: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.267372] CPU 2: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.268309] CPU 3: hi: 0, btch: 1 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.269234] Node 0 DMA32 per-cpu: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.270149] CPU 0: hi: 186, btch: 31 usd: 2 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.271066] CPU 1: hi: 186, btch: 31 usd: 1 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.271957] CPU 2: hi: 186, btch: 31 usd: 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.272856] CPU 3: hi: 186, btch: 31 usd: 30 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.273681] Node 0 Normal per-cpu: Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.274571] CPU 0: hi: 186, btch: 31 usd: 100 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.275466] CPU 1: hi: 186, btch: 31 usd: 45 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.276351] CPU 2: hi: 186, btch: 31 usd: 87 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.277222] CPU 3: hi: 186, btch: 31 usd: 41 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278087] active_anon:1421461 inactive_anon:313478 isolated_anon:0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278088] active_file:185 inactive_file:290 isolated_file:69 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278090] unevictable:3038 dirty:0 writeback:0 unstable:0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278091] free:25550 slab_reclaimable:14134 slab_unreclaimable:101600 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278092] mapped:305 shmem:86 pagetables:89707 bounce:0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.282445] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.285312] lowmem_reserve[]: 0 3246 8034 8034 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.286303] Node 0 DMA32 free:46272kB min:27252kB low:34064kB high:40876kB active_anon:2330240kB inactive_anon:582708kB active_file:24kB inactive_file:20kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2800kB slab_unreclaimable:132256kB kernel_stack:2296kB pagetables:133744kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:296 all_unreclaimable? yes Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.289453] lowmem_reserve[]: 0 0 4788 4788 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.290460] Node 0 Normal free:40052kB min:40196kB low:50244kB high:60292kB active_anon:3355604kB inactive_anon:671204kB active_file:716kB inactive_file:1140kB unevictable:7404kB isolated(anon):0kB isolated(file):276kB present:4902912kB mlocked:7404kB dirty:0kB writeback:0kB mapped:1116kB shmem:240kB slab_reclaimable:53736kB slab_unreclaimable:274112kB kernel_stack:4616kB pagetables:225084kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4068 all_unreclaimable? yes Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.295125] lowmem_reserve[]: 0 0 0 0 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.296327] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.297596] Node 0 DMA32: 179*4kB 143*8kB 158*16kB 229*32kB 136*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46148kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.298883] Node 0 Normal: 449*4kB 360*8kB 263*16kB 353*32kB 150*64kB 42*128kB 10*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 39764kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.300186] 2587 total pagecache pages Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.301448] 1924 pages in swap cache Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.302721] Swap cache stats: add 1830963, delete 1829039, find 494366/494878 Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.303994] Free swap = 0kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.305280] Total swap = 1023996kB Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.371694] 2097136 pages RAM Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.372170] 59776 pages reserved Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.373070] 2548580 pages shared Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.374252] 2007488 pages non-shared Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.375461] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.376713] [ 353] 0 353 4425 23 2 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.377924] [ 756] 0 756 67222 108 1 0 0 NetworkManager Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.379160] [ 770] 0 770 7083 1 0 0 0 systemd-logind Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.380398] [ 773] 70 773 7005 50 3 0 0 avahi-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.381640] [ 776] 70 776 6972 4 0 0 0 avahi-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.382883] [ 778] 0 778 1612 1 0 0 0 mcelog Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.384116] [ 783] 994 783 4893 24 0 0 0 chronyd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.385370] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.386595] [ 790] 0 790 29064 41 1 0 0 ksmtuned Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.387830] [ 797] 0 797 46610 34 2 -13 -900 polkitd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.389066] [ 802] 0 802 60782 63 1 0 0 rsyslogd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.390244] [ 815] 0 815 18861 1 0 -17 -1000 sshd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.391487] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.392739] [ 819] 0 819 4792 17 3 0 0 rpcbind Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.393980] [ 824] 29 824 6938 1 0 0 0 rpc.statd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.395227] [ 837] 0 837 27232 1 3 0 0 agetty Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.396486] [ 1170] 1000 1170 6133 112 3 0 0 tmux Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.397727] [ 1171] 1000 1171 29914 1 2 0 0 bash Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.398971] [ 1262] 1000 1262 4022 131 1 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.400219] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.401493] [15065] 0 15065 3725 87 3 0 0 anacron Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.402759] [19925] 0 19925 4424 1 3 -17 -1000 udevd Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.404053] [14950] 0 14950 1628 89 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.405370] [14954] 0 14954 1595 56 3 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.406583] [14958] 0 14958 3763 32 2 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.407859] [14965] 0 14965 1621 42 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.409294] [14966] 0 14966 2810 43 3 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.410460] [14967] 0 14967 1595 59 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.411770] [14970] 0 14970 1628 89 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.413072] [14975] 0 14975 1621 55 1 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.414379] [14977] 0 14977 1750 36 2 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.415685] [14979] 1000 14979 4022 271 1 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.417022] [14980] 0 14980 1750 36 0 0 0 systemd-cgroups Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.418337] [14982] 0 14982 666 31 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.419660] [14986] 1000 14986 4319 557 2 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.420971] [14987] 1000 14987 4022 269 0 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.422278] [14988] 1000 14988 4187 402 3 0 0 trinity Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.423520] [14990] 0 14990 666 35 0 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.424793] [14992] 0 14992 104 1 2 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.426033] [14994] 0 14994 104 1 0 0 0 modprobe Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.427247] Out of memory: Kill process 756 (NetworkManager) score 1 or sacrifice child Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.428451] Killed process 756 (NetworkManager) total-vm:268888kB, anon-rss:424kB, file-rss:8kB
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
I rebooted, and reran the test, and within minutes, got it into a state where it was killing things again fairly quickly.
This time however, it seems to have killed almost everything on the box, but is still alive. The problem is that all the memory is eaten up by something, and kswapd/ksmd is eating all the cpu. (Attempting to profile with perf causes perf to be oom-killed).
# free
total used free shared buffers cached
Mem: 8149440 8046560 102880 0 272 3316
-/+ buffers/cache: 8042972 106468
Swap: 1023996 1023996 0Attempting to flush the buffers with drop_caches made no difference.
A lot of VMAs in slab..
Active / Total Objects (% used) : 467327 / 494733 (94.5%) Active / Total Slabs (% used) : 18195 / 18195 (100.0%) Active / Total Caches (% used) : 145 / 207 (70.0%) Active / Total Size (% used) : 241177.72K / 263399.54K (91.6%)
Minimum / Average / Maximum Object : 0.33K / 0.53K / 9.16K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 213216 213167 99% 0.49K 6663 32 106608K vm_area_struct 74718 74674 99% 0.37K 3558 21 28464K anon_vma_chain 37820 37663 99% 0.52K 1220 31 19520K anon_vma 33263 33188 99% 0.51K 1073 31 17168K kmalloc-192 25344 25340 99% 0.48K 768 33 12288K sysfs_dir_cache 12740 12660 99% 0.38K 637 20 5096K btrfs_free_space_cache 18981 9213 48% 0.58K 703 27 11248K dentry 8112 8075 99% 0.33K 338 24 2704K kmalloc-8 9982 6064 60% 1.34K 434 23 13888K inode_cache 5768 5605 97% 0.57K 206 28 3296K kmalloc-256 5640 5201 92% 0.38K 282 20 2256K kmalloc-64 4600 4577 99% 0.34K 200 23 1600K kmalloc-16 6182 4002 64% 0.36K 281 22 2248K debug_objects_cache 3002 2980 99% 1.62K 158 19 5056K mm_struct
ps axf shows there's hardly anything left running after all the oom killing:
http://fpaste.org/Dd9p/raw/
sysrq-t traces..
ksmd D ffff88021b81c940 5760 46 2 0x00000000
ffff88021b847b20 0000000000000046 0000000000000000 0000000000000000 ffff88021b81c940 ffff88021b847fd8 ffff88021b847fd8 ffff88021b847fd8 ffff8802232f24a0 ffff88021b81c940 ffff88021b847b50 ffff88021b847d20
Call Trace: [<ffffffff816ace99>] schedule+0x29/0x70 [<ffffffff816aad75>] schedule_timeout+0x385/0x4f0 [<ffffffff816ac23f>] ? wait_for_common+0x3f/0x160 [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90 [<ffffffff810a591d>] ? try_to_wake_up+0x26d/0x310 [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50 [<ffffffff816ac313>] wait_for_common+0x113/0x160 [<ffffffff810a59c0>] ? try_to_wake_up+0x310/0x310 [<ffffffff81166c30>] ? __pagevec_release+0x40/0x40 [<ffffffff816ac43d>] wait_for_completion+0x1d/0x20 [<ffffffff81086b76>] flush_work+0x46/0x70 [<ffffffff81085740>] ? do_work_for_cpu+0x30/0x30 [<ffffffff8108a37b>] schedule_on_each_cpu+0xeb/0x130 [<ffffffff81166c55>] lru_add_drain_all+0x15/0x20 [<ffffffff811a5e45>] ksm_scan_thread+0x8c5/0xd90 [<ffffffff81090040>] ? __init_waitqueue_head+0x60/0x60 [<ffffffff811a5580>] ? run_store+0x2b0/0x2b0 [<ffffffff8108f397>] kthread+0xb7/0xc0 [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0 [<ffffffff816b7f34>] kernel_thread_helper+0x4/0x10 [<ffffffff8109c08c>] ? finish_task_switch+0x7c/0x120 [<ffffffff816af034>] ? retint_restore_args+0x13/0x13 [<ffffffff8108f2e0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff816b7f30>] ? gs_change+0x13/0x13
kswapd0 R running task 4152 45 2 0x00000008
ffff88021b833c80 0000000000000000 ffff88021b833b60 ffffffff8116d6df ffff880200000000 ffff880200000001 0000000000000000 0000000000000002 0000000200000001 ffff88022ffebc40 ffff880226035598 000000000000001e
Call Trace: [<ffffffff8116d6df>] ? shrink_inactive_list+0x17f/0x590 [<ffffffff8116e2d8>] ? shrink_mem_cgroup_zone+0x448/0x5d0 [<ffffffff8116e4d6>] ? shrink_zone+0x76/0xa0 [<ffffffff8116fd35>] ? balance_pgdat+0x555/0x7a0 [<ffffffff8117011c>] ? kswapd+0x19c/0x5f0 [<ffffffff81090040>] ? __init_waitqueue_head+0x60/0x60 [<ffffffff8116ff80>] ? balance_pgdat+0x7a0/0x7a0 [<ffffffff8108f397>] ? kthread+0xb7/0xc0 [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0 [<ffffffff816b7f34>] ? kernel_thread_helper+0x4/0x10 [<ffffffff8109c08c>] ? finish_task_switch+0x7c/0x120 [<ffffffff816af034>] ? retint_restore_args+0x13/0x13 [<ffffffff8108f2e0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff816b7f30>] ? gs_change+0x13/0x13
khugepaged S 000000010039c99c 5728 47 2 0x00000000
ffff88021ba1bc60 0000000000000046 ffff88021ba1bc98 ffffffff81c28be0 ffff88021ba48000 ffff88021ba1bfd8 ffff88021ba1bfd8 ffff88021ba1bfd8 ffff8802232f24a0 ffff88021ba48000 ffff88021ba1bc50 ffff88021ba1bc98
Call Trace: [<ffffffff816ace99>] schedule+0x29/0x70 [<ffffffff816aaba9>] schedule_timeout+0x1b9/0x4f0 [<ffffffff81076370>] ? lock_timer_base+0x70/0x70 [<ffffffff811b2233>] khugepaged+0x273/0x14a0 [<ffffffff8109c08c>] ? finish_task_switch+0x7c/0x120 [<ffffffff81090040>] ? __init_waitqueue_head+0x60/0x60 [<ffffffff811b1fc0>] ? khugepaged_alloc_sleep+0x160/0x160 [<ffffffff8108f397>] kthread+0xb7/0xc0 [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0 [<ffffffff816b7f34>] kernel_thread_helper+0x4/0x10 [<ffffffff8109c08c>] ? finish_task_switch+0x7c/0x120 [<ffffffff816af034>] ? retint_restore_args+0x13/0x13 [<ffffffff8108f2e0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff816b7f30>] ? gs_change+0x13/0x13
sysrq-M ...
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 27
CPU 1: hi: 186, btch: 31 usd: 0
CPU 2: hi: 186, btch: 31 usd: 24
CPU 3: hi: 186, btch: 31 usd: 1
Node 0 Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 67
CPU 1: hi: 186, btch: 31 usd: 3
CPU 2: hi: 186, btch: 31 usd: 55
CPU 3: hi: 186, btch: 31 usd: 13
active_anon:1565586 inactive_anon:283198 isolated_anon:0
active_file:241 inactive_file:505 isolated_file:0
unevictable:1414 dirty:14 writeback:0 unstable:0
free:25817 slab_reclaimable:10704 slab_unreclaimable:56662
mapped:262 shmem:45 pagetables:45795 bounce:0
Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 3246 8034 8034
Node 0 DMA32 free:47320kB min:27252kB low:34064kB high:40876kB active_anon:2651136kB inactive_anon:530560kB active_file:244kB inactive_file:232kB unevictable:788kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:788kB dirty:0kB writeback:0kB mapped:268kB shmem:68kB slab_reclaimable:1200kB slab_unreclaimable:24680kB kernel_stack:336kB pagetables:29924kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2 all_unreclaimable? no lowmem_reserve[]: 0 0 4788 4788
Node 0 Normal free:40072kB min:40196kB low:50244kB high:60292kB active_anon:3611208kB inactive_anon:602232kB active_file:720kB inactive_file:1404kB unevictable:4868kB isolated(anon):0kB isolated(file):0kB present:4902912kB mlocked:4868kB dirty:56kB writeback:0kB mapped:780kB shmem:112kB slab_reclaimable:41616kB slab_unreclaimable:201936kB kernel_stack:2960kB pagetables:153256kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:123 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Node 0 DMA32: 84*4kB 76*8kB 27*16kB 16*32kB 4*64kB 5*128kB 5*256kB 2*512kB 5*1024kB 2*2048kB 8*4096kB = 47072kB
Node 0 Normal: 64*4kB 171*8kB 279*16kB 384*32kB 135*64kB 54*128kB 18*256kB 3*512kB 0*1024kB 0*2048kB 0*4096kB = 40072kB
1774 total pagecache pages
1040 pages in swap cache
Swap cache stats: add 1966902, delete 1965862, find 483713/484292
Free swap = 0kB
Total swap = 1023996kB
2097136 pages RAM
59776 pages reserved
996409 pages shared
2007653 pages non-shared
sysrq-l traces show ksmd is doing *lots* of scanning..
Process ksmd (pid: 46, threadinfo ffff88021b846000, task ffff88021b81c940)
Stack: ffff88021b847de0 ffffffff811a4881 ffff8800136a5260 ffffea0006236f00 ffff88021b847e90 ffffffff811a5f7d 0000000000000000 ffff88018ef64780 ffff88021b81c940 ffff88021b847fd8 ffff88021b847e50 ffff88018ef64828 Call Trace: [<ffffffff811a4881>] memcmp_pages+0x61/0xc0 [<ffffffff811a5f7d>] ksm_scan_thread+0x9fd/0xd90 [<ffffffff81090040>] ? __init_waitqueue_head+0x60/0x60 [<ffffffff811a5580>] ? run_store+0x2b0/0x2b0 [<ffffffff8108f397>] kthread+0xb7/0xc0 [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0 [<ffffffff816b7f34>] kernel_thread_helper+0x4/0x10 [<ffffffff8109c08c>] ? finish_task_switch+0x7c/0x120 [<ffffffff816af034>] ? retint_restore_args+0x13/0x13 [<ffffffff8108f2e0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff816b7f30>] ? gs_change+0x13/0x13
The machine is still alive, though somewhat crippled due to the oom-killer kicking in a lot, but I can keep it up for additional debug info gathering..
Dave
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerQuoted 7 lines · Dave Jones
I rebooted, and reran the test, and within minutes, got it into a state where it was killing things again fairly quickly.
This time however, it seems to have killed almost everything on the box, but is still alive. The problem is that all the memory is eaten up by something, and kswapd/ksmd is eating all the cpu. (Attempting to profile with perf causes perf to be oom-killed).
That makes sense since the oom killer will go along killing all user processes and will leave kthreads such as kswapd and ksmd alone; them using all of the cpu then, especially kswapd aggressively trying to reclaim memory, would be typical.
Quoted 6 lines
A lot of VMAs in slab..
Active / Total Objects (% used) : 467327 / 494733 (94.5%) Active / Total Slabs (% used) : 18195 / 18195 (100.0%) Active / Total Caches (% used) : 145 / 207 (70.0%) Active / Total Size (% used) : 241177.72K / 263399.54K (91.6%)
Quoted 7 lines
Minimum / Average / Maximum Object : 0.33K / 0.53K / 9.16K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 213216 213167 99% 0.49K 6663 32 106608K vm_area_struct 74718 74674 99% 0.37K 3558 21 28464K anon_vma_chain 37820 37663 99% 0.52K 1220 31 19520K anon_vma 33263 33188 99% 0.51K 1073 31 17168K kmalloc-192
This all depends on the workload, but these numbers don't look particularly surprising.
Quoted 5 lines
active_anon:1565586 inactive_anon:283198 isolated_anon:0
active_file:241 inactive_file:505 isolated_file:0 unevictable:1414 dirty:14 writeback:0 unstable:0 free:25817 slab_reclaimable:10704 slab_unreclaimable:56662 mapped:262 shmem:45 pagetables:45795 bounce:0
You have ~7GB of an 8GB macine consumed by anonymous memory, 100MB isn't allocatable probably because of /proc/sys/vm/lowmem_reserve_ratio and the per-zone min watermarks. 263MB of slab isn't atypical considering the workload and you certainly have a lot of memory allocated by pagetables.
So I think what's going to happen if your merge my patch is that you'll see a memory hog be killed and then something will be killed on a consistent basis anytime you completely exhaust all memory like this but nowhere near the amount you saw before without the patch which turns the oom killer into a serial killer.
I would be interested to know where all this anonymous memory is coming from, though, considering the largest rss size from your first global oom condition posted in the first message of this thread was 639 pages, or 2.5MB.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerQuoted 4 lines · Dave Jones
On a test machine that was running my system call fuzzer, I just saw the oom killer take out everything but the process that was doing all the memory exhausting.
Would it be possible to try the below patch? It should kill the thread using the most memory (which happens to only be a couple more megabytes on your system), but it might just delay the inevitable since the system is still in a pretty bad state.
KOSAKI-san suggested doing this before and I think it's the best direction to go in anyway.
--- fs/proc/base.c | 5 +++-- include/linux/oom.h | 5 +++-- mm/oom_kill.c | 39 ++++++++++++++------------------------- 3 files changed, 20 insertions(+), 29 deletions(-)
fs/proc/base.c +3 -2
@@ -410,12 +410,13 @@ static const struct file_operations proc_lstats_operations = {static int proc_oom_score (struct task_struct *task,char *buffer) { +unsigned long totalpages = totalram_pages + total_swap_pages;unsigned long points =0 ;read_lock (&tasklist_lock);if (pid_alive (task)) - points =oom_badness (task,NULL ,NULL , - totalram_pages + total_swap_pages); + points =oom_badness (task,NULL ,NULL , totalpages) * +1000 / totalpages;read_unlock (&tasklist_lock);return sprintf (buffer,"%lu\n" , points); }
include/linux/oom.h +3 -2
@@ -43,8 +43,9 @@ enum oom_constraint {extern void compare_swap_oom_score_adj (int old_val,int new_val);extern int test_set_oom_score_adj (int new_val); -extern unsigned int oom_badness (struct task_struct *p,struct mem_cgroup *memcg, -const nodemask_t *nodemask,unsigned long totalpages); +extern unsigned long oom_badness (struct task_struct *p, +struct mem_cgroup *memcg,const nodemask_t *nodemask, +unsigned long totalpages);extern int try_set_zonelist_oom (struct zonelist *zonelist,gfp_t gfp_flags);extern void clear_zonelist_oom (struct zonelist *zonelist,gfp_t gfp_flags);
mm/oom_kill.c +14 -25
@@ -180,10 +180,10 @@ static bool oom_unkillable_task(struct task_struct *p,* predictable as possible. The goal is to return the highest value for the * task consuming the most memory to avoid subsequent oom failures. */ -unsigned int oom_badness (struct task_struct *p,struct mem_cgroup *memcg, -const nodemask_t *nodemask,unsigned long totalpages) +unsigned long oom_badness (struct task_struct *p,struct mem_cgroup *memcg, +const nodemask_t *nodemask,unsigned long totalpages) { -long points; +unsigned long points;if (oom_unkillable_task (p, memcg, nodemask))return 0 ;
@@ -198,45 +198,33 @@ unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *memcg, }/* -* The memory controller may have a limit of 0 bytes, so avoid a divide -* by zero, if necessary. -*/ -if (!totalpages) - totalpages =1 ; - -/* * The baseline for the badness score is the proportion of RAM that each * task's rss, pagetable and swap space use. */ - points =get_mm_rss (p->mm) + p->mm->nr_ptes; - points +=get_mm_counter (p->mm,MM_SWAPENTS ); - - points *=1000 ; - points /= totalpages; + points =get_mm_rss (p->mm) + p->mm->nr_ptes + +get_mm_counter (p->mm,MM_SWAPENTS );task_unlock (p);/* * Root processes get 3% bonus, just like the __vm_enough_memory() * implementation used by LSMs. */ -if (has_capability_noaudit (p,CAP_SYS_ADMIN )) - points -=30 ; +if (has_capability_noaudit (p,CAP_SYS_ADMIN ) && totalpages) + points -=30 * totalpages /1000 ;/* * /proc/pid/oom_score_adj ranges from -1000 to +1000 such that it may * either completely disable oom killing or always prefer a certain * task. */ - points += p->signal->oom_score_adj; + points += p->signal->oom_score_adj * totalpages /1000 ;/* * Never return 0 for an eligible task that may be killed since it's * possible that no single user task uses more than 0.1% of memory and * no single admin tasks uses more than 3.0%. */ -if (points <=0 ) -return 1 ; -return (points <1000 ) ? points :1000 ; +return points <=0 ?1 : points; }/*
@@ -314,7 +302,7 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, {struct task_struct *g, *p;struct task_struct *chosen =NULL ; - *ppoints =0 ; +unsigned long chosen_points =0 ; do_each_thread(g, p) {unsigned int points;
@@ -354,7 +342,7 @@ static struct task_struct *select_bad_process(unsigned int *ppoints,*/ if (p == current) { chosen = p; - *ppoints =1000 ; + chosen_points =ULONG_MAX ; }else if (!force_kill) {/* * If this task is not being ptraced on exit,
@@ -367,12 +355,13 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, } points =oom_badness (p, memcg, nodemask, totalpages); -if (points > *ppoints) { +if (points > chosen_points) { chosen = p; - *ppoints = points; + chosen_points = points; } }while_each_thread (g, p); + *ppoints = chosen_points *1000 / totalpages;return chosen; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · David Rientjes
Quoted 4 lines · Dave Jones
On a test machine that was running my system call fuzzer, I just saw the oom killer take out everything but the process that was doing all the memory exhausting.
Quoted 7 lines
Would it be possible to try the below patch? It should kill the thread using the most memory (which happens to only be a couple more megabytes on your system), but it might just delay the inevitable since the system is still in a pretty bad state.
KOSAKI-san suggested doing this before and I think it's the best direction to go in anyway.
Sure, I'll give it a shot when I reboot.
However, see my follow-up message. I think there are two bugs here.
1) The over-aggressive oom-killer, and 2) ksmd going mental.
/sys/kernel/mm/ksm/full_scans is increasing constantly
full_scans: 146370pages_shared: 1pages_sharing: 4pages_to_scan: 1250pages_unshared: 867pages_volatile: 1
run: 1sleep_millisecs: 20
everything in /sys/kernel/mm/hugepages/hugepages-2048kB, is 0.
/sys/kernel/mm/transparent_hugepage/khugepaged:alloc_sleep_millisecs 60000
defrag 1full_scans 15max_ptes_none 511pages_collapsed 6pages_to_scan 4096scan_sleep_millisecs 10000
Dave
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerQuoted 11 lines · Dave Jones
/sys/kernel/mm/ksm/full_scansis increasing constantly
full_scans: 146370pages_shared: 1pages_sharing: 4pages_to_scan: 1250pages_unshared: 867pages_volatile: 1
run: 1sleep_millisecs: 20
full_scans is just a counter of how many times it has scanned mergable memory so it should be increasing constantly. Whether pages_to_scan == 1250 and sleep_millisecs == 20 is good for your system is unknown. You may want to try disabling ksm entirely (echo 0 > /sys/kernel/mm/ksm/run) to see if it significantly increases responsiveness for your workload.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 14 lines · David Rientjes
Quoted 11 lines · Dave Jones
/sys/kernel/mm/ksm/full_scansis increasing constantly
full_scans: 146370pages_shared: 1pages_sharing: 4pages_to_scan: 1250pages_unshared: 867pages_volatile: 1
run: 1sleep_millisecs: 20
Quoted 5 lines
full_scansis just a counter of how many times it has scanned mergable memory so it should be increasing constantly. Whetherpages_to_scan== 1250 andsleep_millisecs== 20 is good for your system is unknown. You may want to try disabling ksm entirely (echo 0 >/sys/kernel/mm/ksm/run) to see if it significantly increases responsiveness for your workload.
Disabling it stops it hogging the cpu obviously, but there's still 8G of RAM and 1G of used swap sitting around doing something.
# free
total used free shared buffers cached
Mem: 8149440 8025716 123724 0 148 7764
-/+ buffers/cache: 8017804 131636
Swap: 1423736 1066112 357624SysRq : Show Memory
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 19
CPU 1: hi: 186, btch: 31 usd: 175
CPU 2: hi: 186, btch: 31 usd: 140
CPU 3: hi: 186, btch: 31 usd: 182
Node 0 Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 167
CPU 1: hi: 186, btch: 31 usd: 176
CPU 2: hi: 186, btch: 31 usd: 102
CPU 3: hi: 186, btch: 31 usd: 94
active_anon:1529737 inactive_anon:306307 isolated_anon:0
active_file:1124 inactive_file:2170 isolated_file:0
unevictable:1414 dirty:1 writeback:0 unstable:0
free:35645 slab_reclaimable:10150 slab_unreclaimable:56678
mapped:404 shmem:48 pagetables:45796 bounce:0
Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 3246 8034 8034
Node 0 DMA32 free:62632kB min:27252kB low:34064kB high:40876kB active_anon:2637356kB inactive_anon:527504kB active_file:72kB inactive_file:84kB unevictable:788kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:788kB dirty:0kB writeback:0kB mapped:212kB shmem:72kB slab_reclaimable:944kB slab_unreclaimable:24736kB kernel_stack:336kB pagetables:30028kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:116 all_unreclaimable? no lowmem_reserve[]: 0 0 4788 4788
Node 0 Normal free:64072kB min:40196kB low:50244kB high:60292kB active_anon:3481592kB inactive_anon:697724kB active_file:4424kB inactive_file:8596kB unevictable:4868kB isolated(anon):0kB isolated(file):0kB present:4902912kB mlocked:4868kB dirty:4kB writeback:0kB mapped:1404kB shmem:120kB slab_reclaimable:39656kB slab_unreclaimable:201944kB kernel_stack:2968kB pagetables:153156kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Node 0 DMA32: 214*4kB 124*8kB 65*16kB 55*32kB 32*64kB 15*128kB 9*256kB 7*512kB 7*1024kB 4*2048kB 8*4096kB = 62632kB
Node 0 Normal: 670*4kB 573*8kB 402*16kB 468*32kB 171*64kB 73*128kB 31*256kB 12*512kB 1*1024kB 0*2048kB 0*4096kB = 64064kB
5683 total pagecache pages
2341 pages in swap cache
Swap cache stats: add 2029253, delete 2026912, find 483987/484568
Free swap = 343568kB
Total swap = 1423736kB
2097136 pages RAM
59776 pages reserved
891838 pages shared
1996710 pages non-shared
All that anon memory seems to be unaccounted for.
[ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
[ 351] 0 351 4372 2 3 -17 -1000 udevd [ 818] 0 818 18861 26 0 -17 -1000 sshd [ 1199] 0 1199 4372 2 3 -17 -1000 udevd [ 1214] 0 1214 4371 2 1 -17 -1000 udevd [28963] 0 28963 30988 271 2 0 0 sshd [28987] 81 28987 5439 150 3 -13 -900 dbus-daemon [28990] 0 28990 7085 136 0 0 0 systemd-logind [28995] 1000 28995 31023 373 1 0 0 sshd [29008] 1000 29008 29864 875 2 0 0 bash [29132] 1000 29132 44732 155 3 0 0 sudo [29135] 0 29135 29870 1094 3 0 0 bash [29521] 0 29521 4877 196 0 0 0 systemd-kmsg-sy [29541] 0 29541 27232 211 3 0 0 agetty [29553] 0 29553 29870 875 2 0 0 bash
Dave
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerQuoted 3 lines · Dave Jones
Disabling it stops it hogging the cpu obviously, but there's still 8G of RAM and 1G of used swap sitting around doing something.
Right, I eluded to this in another email because the rss sizes from your oom log weren't necessarily impressive. Could you post the output of /proc/meminfo?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 9 lines · David Rientjes
Quoted 3 lines · Dave Jones
Disabling it stops it hogging the cpu obviously, but there's still 8G of RAM and 1G of used swap sitting around doing something.
Right, I eluded to this in another email because the rss sizes from your oom log weren't necessarily impressive. Could you post the output of
/proc/meminfo?
MemTotal: 8149440 kB MemFree: 142560 kB Buffers: 1408 kB Cached: 11504 kB SwapCached: 9336 kB Active: 6124932 kB Inactive: 1232176 kB Active(anon): 6119160 kB Inactive(anon): 1225228 kB Active(file): 5772 kB Inactive(file): 6948 kB Unevictable: 5656 kB Mlocked: 5656 kB SwapTotal: 1423736 kB SwapFree: 343596 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 7341364 kB Mapped: 5720 kB Shmem: 192 kB Slab: 267408 kB SReclaimable: 40808 kB SUnreclaim: 226600 kB KernelStack: 3280 kB PageTables: 183104 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 5498456 kB Committed_AS: 111294188 kB VmallocTotal: 34359738367 kB VmallocUsed: 346720 kB VmallocChunk: 34359384056 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 98344 kB DirectMap2M: 8288256 kB
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 15 lines · Dave Jones
Quoted 14 lines · David Rientjes
Quoted 11 lines · Dave Jones
/sys/kernel/mm/ksm/full_scansis increasing constantly
full_scans: 146370pages_shared: 1pages_sharing: 4pages_to_scan: 1250pages_unshared: 867pages_volatile: 1
run: 1sleep_millisecs: 20
Quoted 6 lines
Quoted 5 lines
full_scansis just a counter of how many times it has scanned mergable memory so it should be increasing constantly. Whetherpages_to_scan== 1250 andsleep_millisecs== 20 is good for your system is unknown. You may want to try disabling ksm entirely (echo 0 >/sys/kernel/mm/ksm/run) to see if it significantly increases responsiveness for your workload.
You didn't happen to see any RCU CPU stalls, did you?
-- Steve
Quoted 69 lines
Disabling it stops it hogging the cpu obviously, but there's still 8G of RAM and 1G of used swap sitting around doing something.
# free total used free shared buffers cached Mem: 8149440 8025716 123724 0 148 7764 -/+ buffers/cache: 8017804 131636 Swap: 1423736 1066112 357624SysRq : Show Memory
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 19
CPU 1: hi: 186, btch: 31 usd: 175
CPU 2: hi: 186, btch: 31 usd: 140
CPU 3: hi: 186, btch: 31 usd: 182
Node 0 Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 167
CPU 1: hi: 186, btch: 31 usd: 176
CPU 2: hi: 186, btch: 31 usd: 102
CPU 3: hi: 186, btch: 31 usd: 94
active_anon:1529737 inactive_anon:306307 isolated_anon:0
active_file:1124 inactive_file:2170 isolated_file:0
unevictable:1414 dirty:1 writeback:0 unstable:0
free:35645 slab_reclaimable:10150 slab_unreclaimable:56678
mapped:404 shmem:48 pagetables:45796 bounce:0
Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? yes lowmem_reserve[]: 0 3246 8034 8034
Node 0 DMA32 free:62632kB min:27252kB low:34064kB high:40876kB active_anon:2637356kB inactive_anon:527504kB active_file:72kB inactive_file:84kB unevictable:788kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:788kB dirty:0kB writeback:0kB mapped:212kB shmem:72kB slab_reclaimable:944kB slab_unreclaimable:24736kB kernel_stack:336kB pagetables:30028kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:116all_unreclaimable? no lowmem_reserve[]: 0 0 4788 4788
Node 0 Normal free:64072kB min:40196kB low:50244kB high:60292kB active_anon:3481592kB inactive_anon:697724kB active_file:4424kB inactive_file:8596kB unevictable:4868kB isolated(anon):0kB isolated(file):0kB present:4902912kB mlocked:4868kB dirty:4kB writeback:0kB mapped:1404kB shmem:120kB slab_reclaimable:39656kB slab_unreclaimable:201944kB kernel_stack:2968kB pagetables:153156kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0all_unreclaimable? no lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Node 0 DMA32: 214*4kB 124*8kB 65*16kB 55*32kB 32*64kB 15*128kB 9*256kB 7*512kB 7*1024kB 4*2048kB 8*4096kB = 62632kB
Node 0 Normal: 670*4kB 573*8kB 402*16kB 468*32kB 171*64kB 73*128kB 31*256kB 12*512kB 1*1024kB 0*2048kB 0*4096kB = 64064kB
5683 total pagecache pages
2341 pages in swap cache
Swap cache stats: add 2029253, delete 2026912, find 483987/484568
Free swap = 343568kB
Total swap = 1423736kB
2097136 pages RAM
59776 pages reserved
891838 pages shared
1996710 pages non-sharedAll that anon memory seems to be unaccounted for.
[ pid ] uid tgid
total_vmrss cpuoom_adjoom_score_adjname[ 351] 0 351 4372 2 3 -17 -1000 udevd [ 818] 0 818 18861 26 0 -17 -1000 sshd [ 1199] 0 1199 4372 2 3 -17 -1000 udevd [ 1214] 0 1214 4371 2 1 -17 -1000 udevd [28963] 0 28963 30988 271 2 0 0 sshd [28987] 81 28987 5439 150 3 -13 -900 dbus-daemon [28990] 0 28990 7085 136 0 0 0 systemd-logind [28995] 1000 28995 31023 373 1 0 0 sshd [29008] 1000 29008 29864 875 2 0 0 bash [29132] 1000 29132 44732 155 3 0 0 sudo [29135] 0 29135 29870 1094 3 0 0 bash [29521] 0 29521 4877 196 0 0 0 systemd-kmsg-sy [29541] 0 29541 27232 211 3 0 0 agetty [29553] 0 29553 29870 875 2 0 0 bash
Quoted 7 lines
Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Steven Rostedt
Quoted 6 lines
Quoted 5 lines
full_scansis just a counter of how many times it has scanned mergable memory so it should be increasing constantly. Whetherpages_to_scan== 1250 andsleep_millisecs== 20 is good for your system is unknown. You may want to try disabling ksm entirely (echo 0 >/sys/kernel/mm/ksm/run) to see if it significantly increases responsiveness for your workload.You didn't happen to see any RCU CPU stalls, did you?
nothing got reported in dmesg..
Dave
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerQuoted 129 lines · David Rientjes
fs/proc/base.c +3
-2@@ -410,12 +410,13 @@ static const struct file_operations proc_lstats_operations = {static int proc_oom_score (struct task_struct *task,char *buffer) { +unsigned long totalpages = totalram_pages + total_swap_pages;unsigned long points =0 ;read_lock (&tasklist_lock);if (pid_alive (task)) - points =oom_badness (task,NULL ,NULL , - totalram_pages + total_swap_pages); + points =oom_badness (task,NULL ,NULL , totalpages) * +1000 / totalpages;read_unlock (&tasklist_lock);return sprintf (buffer,"%lu\n" , points); }include/linux/oom.h +3
-2@@ -43,8 +43,9 @@ enum oom_constraint {extern void compare_swap_oom_score_adj (int old_val,int new_val);extern int test_set_oom_score_adj (int new_val); -extern unsigned int oom_badness (struct task_struct *p,struct mem_cgroup *memcg, -const nodemask_t *nodemask,unsigned long totalpages); +extern unsigned long oom_badness (struct task_struct *p, +struct mem_cgroup *memcg,const nodemask_t *nodemask, +unsigned long totalpages);extern int try_set_zonelist_oom (struct zonelist *zonelist,gfp_t gfp_flags);extern void clear_zonelist_oom (struct zonelist *zonelist,gfp_t gfp_flags);mm/oom_kill.c +12
-24@@ -180,10 +180,10 @@ static bool oom_unkillable_task(struct task_struct *p,* predictable as possible. The goal is to return the highest value for the * task consuming the most memory to avoid subsequent oom failures. */ -unsigned int oom_badness (struct task_struct *p,struct mem_cgroup *memcg, -const nodemask_t *nodemask,unsigned long totalpages) +unsigned long oom_badness (struct task_struct *p,struct mem_cgroup *memcg, +const nodemask_t *nodemask,unsigned long totalpages) { -long points; +unsigned long points;if (oom_unkillable_task (p, memcg, nodemask))return 0 ;@@ -198,45 +198,33 @@ unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *memcg, }/* -* The memory controller may have a limit of 0 bytes, so avoid a divide -* by zero, if necessary. -*/ -if (!totalpages) - totalpages =1 ; - -/* * The baseline for the badness score is the proportion of RAM that each * task's rss, pagetable and swap space use. */ - points =get_mm_rss (p->mm) + p->mm->nr_ptes; - points +=get_mm_counter (p->mm,MM_SWAPENTS ); - - points *=1000 ; - points /= totalpages; + points =get_mm_rss (p->mm) + p->mm->nr_ptes + +get_mm_counter (p->mm,MM_SWAPENTS );task_unlock (p);/* * Root processes get 3% bonus, just like the __vm_enough_memory() * implementation used by LSMs. */ -if (has_capability_noaudit (p,CAP_SYS_ADMIN )) - points -=30 ; +if (has_capability_noaudit (p,CAP_SYS_ADMIN ) && totalpages) + points -=30 * totalpages /1000 ;/* * /proc/pid/oom_score_adj ranges from -1000 to +1000 such that it may * either completely disable oom killing or always prefer a certain * task. */ - points += p->signal->oom_score_adj; + points += p->signal->oom_score_adj * totalpages /1000 ;/* * Never return 0 for an eligible task that may be killed since it's * possible that no single user task uses more than 0.1% of memory and * no single admin tasks uses more than 3.0%. */ -if (points <=0 ) -return 1 ; -return (points <1000 ) ? points :1000 ; +return points <=0 ?1 : points; }/* @@ -314,7 +302,7 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, {struct task_struct *g, *p;struct task_struct *chosen =NULL ; - *ppoints =0 ; +unsigned long chosen_points =0 ; do_each_thread(g, p) {unsigned int points;@@ -354,7 +342,7 @@ static struct task_struct *select_bad_process(unsigned int *ppoints,*/ if (p == current) { chosen = p; - *ppoints =1000 ; + chosen_points =ULONG_MAX ; }else if (!force_kill) {/* * If this task is not being ptraced on exit, @@ -367,12 +355,13 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, } points =oom_badness (p, memcg, nodemask, totalpages); -if (points > *ppoints) { +if (points > chosen_points) {
mm/oom_kill.c static struct task_struct *select_bad_process(unsigned int *ppoints, · line 367
Quoted 10 lines
mm/oom_kill.c +2
-1chosen = p; - *ppoints = points; + chosen_points = points; } }while_each_thread (g, p); + *ppoints = chosen_points *1000 / totalpages;return chosen; }
Dave, did you get a chance to test this out? It's something we'll want in the oom killer so if I can add your Tested-by it would be great. Thanks!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 2 lines · David Rientjes
Dave, did you get a chance to test this out? It's something we'll want in the oom killer so if I can add your Tested-by it would be great. Thanks!
Yes, this seems to be an improvement in my case (the fuzzer got killed every time rather than arbitary system processes).
Feel free to add my Tested-by:
thanks,
Dave
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerPATCH 1/1 mm, oom: normalize oom scores to oom_score_adj scale only for userspace
The oom_score_adj scale ranges from -1000 to 1000 and represents the proportion of memory available to the process at allocation time. This means an oom_score_adj value of 300, for example, will bias a process as though it was using an extra 30.0% of available memory and a value of -350 will discount 35.0% of available memory from its usage.
The oom killer badness heuristic also uses this scale to report the oom score for each eligible process in determining the "best" process to kill. Thus, it can only differentiate each process's memory usage by 0.1% of system RAM.
On large systems, this can end up being a large amount of memory: 256MB on 256GB systems, for example.
This can be fixed by having the badness heuristic to use the actual memory usage in scoring threads and then normalizing it to the oom_score_adj scale for userspace. This results in better comparison between eligible threads for kill and no change from the userspace perspective.
--- fs/proc/base.c | 5 +++-- include/linux/oom.h | 5 +++-- mm/oom_kill.c | 39 ++++++++++++++------------------------- 3 files changed, 20 insertions(+), 29 deletions(-)
fs/proc/base.c +3 -2
@@ -410,12 +410,13 @@ static const struct file_operations proc_lstats_operations = {static int proc_oom_score (struct task_struct *task,char *buffer) { +unsigned long totalpages = totalram_pages + total_swap_pages;unsigned long points =0 ;read_lock (&tasklist_lock);if (pid_alive (task)) - points =oom_badness (task,NULL ,NULL , - totalram_pages + total_swap_pages); + points =oom_badness (task,NULL ,NULL , totalpages) * +1000 / totalpages;read_unlock (&tasklist_lock);return sprintf (buffer,"%lu\n" , points); }
include/linux/oom.h +3 -2
@@ -43,8 +43,9 @@ enum oom_constraint {extern void compare_swap_oom_score_adj (int old_val,int new_val);extern int test_set_oom_score_adj (int new_val); -extern unsigned int oom_badness (struct task_struct *p,struct mem_cgroup *memcg, -const nodemask_t *nodemask,unsigned long totalpages); +extern unsigned long oom_badness (struct task_struct *p, +struct mem_cgroup *memcg,const nodemask_t *nodemask, +unsigned long totalpages);extern int try_set_zonelist_oom (struct zonelist *zonelist,gfp_t gfp_flags);extern void clear_zonelist_oom (struct zonelist *zonelist,gfp_t gfp_flags);
mm/oom_kill.c +14 -25
@@ -180,10 +180,10 @@ static bool oom_unkillable_task(struct task_struct *p,* predictable as possible. The goal is to return the highest value for the * task consuming the most memory to avoid subsequent oom failures. */ -unsigned int oom_badness (struct task_struct *p,struct mem_cgroup *memcg, -const nodemask_t *nodemask,unsigned long totalpages) +unsigned long oom_badness (struct task_struct *p,struct mem_cgroup *memcg, +const nodemask_t *nodemask,unsigned long totalpages) { -long points; +unsigned long points;if (oom_unkillable_task (p, memcg, nodemask))return 0 ;
@@ -198,45 +198,33 @@ unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *memcg, }/* -* The memory controller may have a limit of 0 bytes, so avoid a divide -* by zero, if necessary. -*/ -if (!totalpages) - totalpages =1 ; - -/* * The baseline for the badness score is the proportion of RAM that each * task's rss, pagetable and swap space use. */ - points =get_mm_rss (p->mm) + p->mm->nr_ptes; - points +=get_mm_counter (p->mm,MM_SWAPENTS ); - - points *=1000 ; - points /= totalpages; + points =get_mm_rss (p->mm) + p->mm->nr_ptes + +get_mm_counter (p->mm,MM_SWAPENTS );task_unlock (p);/* * Root processes get 3% bonus, just like the __vm_enough_memory() * implementation used by LSMs. */ -if (has_capability_noaudit (p,CAP_SYS_ADMIN )) - points -=30 ; +if (has_capability_noaudit (p,CAP_SYS_ADMIN ) && totalpages) + points -=30 * totalpages /1000 ;/* * /proc/pid/oom_score_adj ranges from -1000 to +1000 such that it may * either completely disable oom killing or always prefer a certain * task. */ - points += p->signal->oom_score_adj; + points += p->signal->oom_score_adj * totalpages /1000 ;/* * Never return 0 for an eligible task that may be killed since it's * possible that no single user task uses more than 0.1% of memory and * no single admin tasks uses more than 3.0%. */ -if (points <=0 ) -return 1 ; -return (points <1000 ) ? points :1000 ; +return points <=0 ?1 : points; }/*
@@ -314,7 +302,7 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, {struct task_struct *g, *p;struct task_struct *chosen =NULL ; - *ppoints =0 ; +unsigned long chosen_points =0 ; do_each_thread(g, p) {unsigned int points;
@@ -354,7 +342,7 @@ static struct task_struct *select_bad_process(unsigned int *ppoints,*/ if (p == current) { chosen = p; - *ppoints =1000 ; + chosen_points =ULONG_MAX ; }else if (!force_kill) {/* * If this task is not being ptraced on exit,
@@ -367,12 +355,13 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, } points =oom_badness (p, memcg, nodemask, totalpages); -if (points > *ppoints) { +if (points > chosen_points) { chosen = p; - *ppoints = points; + chosen_points = points; } }while_each_thread (g, p); + *ppoints = chosen_points *1000 / totalpages;return chosen; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [patch] mm, oom: normalize oom scores to oom_score_adj scale only for userspace
Quoted 42 lines · On Thu, 17 May 2012 14:33:27 -0700 (PDT) David Rientjes <[email protected]>…
The
oom_score_adjscale ranges from -1000 to 1000 and represents the proportion of memory available to the process at allocation time. This means anoom_score_adjvalue of 300, for example, will bias a process as though it was using an extra 30.0% of available memory and a value of -350 will discount 35.0% of available memory from its usage.The oom killer badness heuristic also uses this scale to report the oom score for each eligible process in determining the "best" process to kill. Thus, it can only differentiate each process's memory usage by 0.1% of system RAM.
On large systems, this can end up being a large amount of memory: 256MB on 256GB systems, for example.
This can be fixed by having the badness heuristic to use the actual memory usage in scoring threads and then normalizing it to the
oom_score_adjscale for userspace. This results in better comparison between eligible threads for kill and no change from the userspace perspective....
mm/oom_kill.c +1
-12@@ -198,45 +198,33 @@ unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *memcg, }/* -* The memory controller may have a limit of 0 bytes, so avoid a divide -* by zero, if necessary. -*/ -if (!totalpages) - totalpages =1 ; - -/* * The baseline for the badness score is the proportion of RAM that each * task's rss, pagetable and swap space use. */ - points =get_mm_rss (p->mm) + p->mm->nr_ptes; - points +=get_mm_counter (p->mm,MM_SWAPENTS ); - - points *=1000 ; - points /= totalpages; + points = get_mm_rss(p->mm) + p->mm->nr_ptes +
mm/oom_kill.c unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *memcg, · line 212
Quoted 10 lines
mm/oom_kill.c +2
-2+get_mm_counter (p->mm,MM_SWAPENTS );task_unlock (p);/* * Root processes get 3% bonus, just like the __vm_enough_memory() * implementation used by LSMs. */ -if (has_capability_noaudit (p,CAP_SYS_ADMIN )) - points -=30 ; +if (has_capability_noaudit (p,CAP_SYS_ADMIN ) && totalpages)
There doesn't seem much point in testing totalpages here - it's a micro-optimisation which adds a branch, on a slow path.
mm/oom_kill.c unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *memcg, · line 220
Quoted 9 lines
mm/oom_kill.c +2
-1+ points -=30 * totalpages /1000 ;/* * /proc/pid/oom_score_adj ranges from -1000 to +1000 such that it may * either completely disable oom killing or always prefer a certain * task. */ - points += p->signal->oom_score_adj; + points += p->signal->oom_score_adj * totalpages /1000 ;
And if we *do* want to add that micro-optimisation, we may as well extend it to cover this expression also:
if (totalpages) { /* reason goes here */
if (has_capability_noaudit(...))
points -= 30 * totalpages / 1000;
p->signal->oom_score_adj * totalpages / 1000;
}mm/oom_kill.c unsigned int oom_badness(struct task_struct *p, struct mem_cgroup *memcg, · line 227
Quoted 9 lines
mm/oom_kill.c +1
-3/* * Never return 0 for an eligible task that may be killed since it's * possible that no single user task uses more than 0.1% of memory and * no single admin tasks uses more than 3.0%. */ -if (points <=0 ) -return 1 ; -return (points <1000 ) ? points :1000 ; +return points <=0 ?1 : points;
`points' is unsigned - testing it for negative looks odd.
Quoted 24 lines
mm/oom_kill.c +2
-2}/* @@ -314,7 +302,7 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, {struct task_struct *g, *p;struct task_struct *chosen =NULL ; - *ppoints =0 ; +unsigned long chosen_points =0 ; do_each_thread(g, p) {unsigned int points;@@ -354,7 +342,7 @@ static struct task_struct *select_bad_process(unsigned int *ppoints,*/ if (p == current) { chosen = p; - *ppoints =1000 ; + chosen_points =ULONG_MAX ; }else if (!force_kill) {/* * If this task is not being ptraced on exit, @@ -367,12 +355,13 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, }
mm/oom_kill.c static struct task_struct *select_bad_process(unsigned int *ppoints, · line 364
Quoted 10 lines
mm/oom_kill.c +3
-2points =oom_badness (p, memcg, nodemask, totalpages); -if (points > *ppoints) { +if (points > chosen_points) { chosen = p; - *ppoints = points; + chosen_points = points; } }while_each_thread (g, p); + *ppoints = chosen_points *1000 / totalpages;
So it's up to the select_bad_process() callers to prevent the divide-by-zero. It is unobvious that they actually do this, and this important and unobvious caller requirement is undocumented.
mm/oom_kill.c static struct task_struct *select_bad_process(unsigned int *ppoints, · line 367
Quoted 3 lines
mm/oom_kill.c +0
-0return chosen; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerPATCH v2 1/1 mm, oom: normalize oom scores to oom_score_adj scale only for userspace
The oom_score_adj scale ranges from -1000 to 1000 and represents the proportion of memory available to the process at allocation time. This means an oom_score_adj value of 300, for example, will bias a process as though it was using an extra 30.0% of available memory and a value of -350 will discount 35.0% of available memory from its usage.
The oom killer badness heuristic also uses this scale to report the oom score for each eligible process in determining the "best" process to kill. Thus, it can only differentiate each process's memory usage by 0.1% of system RAM.
On large systems, this can end up being a large amount of memory: 256MB on 256GB systems, for example.
This can be fixed by having the badness heuristic to use the actual memory usage in scoring threads and then normalizing it to the oom_score_adj scale for userspace. This results in better comparison between eligible threads for kill and no change from the userspace perspective.
--- fs/proc/base.c | 5 +++-- include/linux/oom.h | 5 +++-- mm/oom_kill.c | 44 ++++++++++++++++---------------------------- 3 files changed, 22 insertions(+), 32 deletions(-)
fs/proc/base.c +3 -2
@@ -410,12 +410,13 @@ static const struct file_operations proc_lstats_operations = {static int proc_oom_score (struct task_struct *task,char *buffer) { +unsigned long totalpages = totalram_pages + total_swap_pages;unsigned long points =0 ;read_lock (&tasklist_lock);if (pid_alive (task)) - points =oom_badness (task,NULL ,NULL , - totalram_pages + total_swap_pages); + points =oom_badness (task,NULL ,NULL , totalpages) * +1000 / totalpages;read_unlock (&tasklist_lock);return sprintf (buffer,"%lu\n" , points); }
include/linux/oom.h +3 -2
@@ -43,8 +43,9 @@ enum oom_constraint {extern void compare_swap_oom_score_adj (int old_val,int new_val);extern int test_set_oom_score_adj (int new_val); -extern unsigned int oom_badness (struct task_struct *p,struct mem_cgroup *memcg, -const nodemask_t *nodemask,unsigned long totalpages); +extern unsigned long oom_badness (struct task_struct *p, +struct mem_cgroup *memcg,const nodemask_t *nodemask, +unsigned long totalpages);extern int try_set_zonelist_oom (struct zonelist *zonelist,gfp_t gfp_flags);extern void clear_zonelist_oom (struct zonelist *zonelist,gfp_t gfp_flags);
mm/oom_kill.c +16 -28
103 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Andrew Morton
maintainerRe: [patch v2] mm, oom: normalize oom scores to oom_score_adj scale only for userspace
Quoted 28 lines · On Wed, 23 May 2012 00:15:03 -0700 (PDT) David Rientjes <[email protected]>…
The
oom_score_adjscale ranges from -1000 to 1000 and represents the proportion of memory available to the process at allocation time. This means anoom_score_adjvalue of 300, for example, will bias a process as though it was using an extra 30.0% of available memory and a value of -350 will discount 35.0% of available memory from its usage.The oom killer badness heuristic also uses this scale to report the oom score for each eligible process in determining the "best" process to kill. Thus, it can only differentiate each process's memory usage by 0.1% of system RAM.
On large systems, this can end up being a large amount of memory: 256MB on 256GB systems, for example.
This can be fixed by having the badness heuristic to use the actual memory usage in scoring threads and then normalizing it to the
oom_score_adjscale for userspace. This results in better comparison between eligible threads for kill and no change from the userspace perspective....
mm/oom_kill.c +1
-1@@ -367,12 +354,13 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, } points =oom_badness (p, memcg, nodemask, totalpages); -if (points > *ppoints) { +if (points > chosen_points) {
mm/oom_kill.c static struct task_struct *select_bad_process(unsigned int *ppoints, · line 366
Quoted 10 lines
mm/oom_kill.c +2
-1chosen = p; - *ppoints = points; + chosen_points = points; } }while_each_thread (g, p); + *ppoints = chosen_points *1000 / totalpages;return chosen; }
It's still not obvious that we always avoid the divide-by-zero here. If there's some weird way of convincing constrained_alloc() to look at an empty nodemask, or a nodemask which covers only empty nodes then blam.
Now, it's probably the case that this is a can't-happen but that guarantee would be pretty convoluted and fragile?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
David Rientjes
reviewerRe: [patch v2] mm, oom: normalize oom scores to oom_score_adj scale only for userspace
Quoted 22 lines · Andrew Morton
Quoted 6 lines
mm/oom_kill.c +1
-1@@ -367,12 +354,13 @@ static struct task_struct *select_bad_process(unsigned int *ppoints, } points =oom_badness (p, memcg, nodemask, totalpages); -if (points > *ppoints) { +if (points > chosen_points) {mm/oom_kill.c static struct task_struct *select_bad_process(unsigned int *ppoints, · line 366
Quoted 10 lines
mm/oom_kill.c +2
-1chosen = p; - *ppoints = points; + chosen_points = points; } }while_each_thread (g, p); + *ppoints = chosen_points *1000 / totalpages;return chosen; }It's still not obvious that we always avoid the divide-by-zero here. If there's some weird way of convincing
constrained_alloc()to look at an empty nodemask, or a nodemask which covers only empty nodes then blam.
Quoted 3 lines
Now, it's probably the case that this is a can't-happen but that guarantee would be pretty convoluted and fragile?
It can only happen for memcg with a zero limit, something I tried to prevent by not allowing tasks to be attached to the memcgs with such a limit in a different patch but you didn't like that :)
So I fixed it in this patch with this:
@@ -572,7 +560,7 @@ void mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask,
}
check_panic_on_oom(CONSTRAINT_MEMCG, gfp_mask, order, NULL);
- limit = mem_cgroup_get_limit(memcg) >> PAGE_SHIFT;
+ limit = mem_cgroup_get_limit(memcg) >> PAGE_SHIFT ? : 1;
read_lock(&tasklist_lock);
p = select_bad_process(&points, limit, memcg, NULL, false); if (p && PTR_ERR(p) != -1UL)
Cpusets do not allow threads to be attached without a set of mems or the final mem in a cpuset to be removed while tasks are still attached. The page allocator certainly wouldn't be calling the oom killer for a set of zones that span no pages.
Any suggestion on where to put the check for !totalpages so it's easier to understand?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>