sk_page_frag_refill OOM killing spree
Subsystems (2)
Hi Eric,
seems like sk_page_frag_refill() can cause oom-killer invocation:
postgres invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0
Pid: 10551, comm: postgres Tainted: G O 3.8.6-5.g613ca40-smp #1
Call Trace: [<c106dbd5>] ? dump_header+0x60/0x191 [<c1133d3a>] ? ___ratelimit+0xb2/0xc4 [<c106dfd3>] ? oom_kill_process+0x61/0x2d1 [<c1030042>] ? has_capability_noaudit+0x1c/0x23 [<c106df0f>] ? oom_badness+0x8c/0xef [<c106e446>] ? out_of_memory+0x203/0x247 [<c107128a>] ? __alloc_pages_nodemask+0x42b/0x4c3 [<c11fa66a>] ? sk_page_frag_refill+0x6a/0xd2 [<c1233548>] ? tcp_sendmsg+0x3e8/0x7c6 [<c124f34b>] ? inet_sendmsg+0x6b/0x75 [<c11f74d8>] ? sock_sendmsg+0x8d/0xa6 [<c11f7b83>] ? sys_sendto+0x105/0x130 [<c1025927>] ? __kunmap_atomic+0x62/0x8a [<c1025940>] ? __kunmap_atomic+0x7b/0x8a [<c1073d78>] ? __lru_cache_add+0x18/0x47 [<c10812f9>] ? handle_pte_fault+0x745/0x751 [<c1025a2d>] ? kmap_atomic_prot+0xd3/0xf1 [<c10817da>] ? handle_mm_fault+0x112/0x121 [<c11f7be5>] ? sys_send+0x37/0x3b
The system is busy, so, order-3 alloc failure doesn't strike me as odd.
There are no allocation failures with order != 3.
Sometimes this can happen in very short sucession, i.e. and oom-killer did end up zapping 30 processes or so.
My question is, should sk_page_frag_refill use __GFP_NORETRY, at least for order 3 requests?
Quoted 29 lines · Florian Westphal
Hi Eric,
seems like
sk_page_frag_refill()can cause oom-killer invocation:postgres invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0
Pid: 10551, comm: postgres Tainted: G O 3.8.6-5.g613ca40-smp #1Call Trace: [<c106dbd5>] ? dump_header+0x60/0x191 [<c1133d3a>] ? ___ratelimit+0xb2/0xc4 [<c106dfd3>] ? oom_kill_process+0x61/0x2d1 [<c1030042>] ? has_capability_noaudit+0x1c/0x23 [<c106df0f>] ? oom_badness+0x8c/0xef [<c106e446>] ? out_of_memory+0x203/0x247 [<c107128a>] ? __alloc_pages_nodemask+0x42b/0x4c3 [<c11fa66a>] ? sk_page_frag_refill+0x6a/0xd2 [<c1233548>] ? tcp_sendmsg+0x3e8/0x7c6 [<c124f34b>] ? inet_sendmsg+0x6b/0x75 [<c11f74d8>] ? sock_sendmsg+0x8d/0xa6 [<c11f7b83>] ? sys_sendto+0x105/0x130 [<c1025927>] ? __kunmap_atomic+0x62/0x8a [<c1025940>] ? __kunmap_atomic+0x7b/0x8a [<c1073d78>] ? __lru_cache_add+0x18/0x47 [<c10812f9>] ? handle_pte_fault+0x745/0x751 [<c1025a2d>] ? kmap_atomic_prot+0xd3/0xf1 [<c10817da>] ? handle_mm_fault+0x112/0x121 [<c11f7be5>] ? sys_send+0x37/0x3bThe system is busy, so, order-3 alloc failure doesn't strike me as odd.
Quoted 7 lines
There are no allocation failures with order != 3.
Sometimes this can happen in very short sucession, i.e. and oom-killer did end up zapping 30 processes or so.
My question is, should
sk_page_frag_refilluse__GFP_NORETRY, at least for order 3 requests?
Yes, you may be right, but I prefer some mm expertise here, so I CC lkml
Quoted 28 lines · Eric Dumazet
Quoted 27 lines · Florian Westphal
Hi Eric,
seems like
sk_page_frag_refill()can cause oom-killer invocation:postgres invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0
Pid: 10551, comm: postgres Tainted: G O 3.8.6-5.g613ca40-smp #1Call Trace: [<c106dbd5>] ? dump_header+0x60/0x191 [<c1133d3a>] ? ___ratelimit+0xb2/0xc4 [<c106dfd3>] ? oom_kill_process+0x61/0x2d1 [<c1030042>] ? has_capability_noaudit+0x1c/0x23 [<c106df0f>] ? oom_badness+0x8c/0xef [<c106e446>] ? out_of_memory+0x203/0x247 [<c107128a>] ? __alloc_pages_nodemask+0x42b/0x4c3 [<c11fa66a>] ? sk_page_frag_refill+0x6a/0xd2 [<c1233548>] ? tcp_sendmsg+0x3e8/0x7c6 [<c124f34b>] ? inet_sendmsg+0x6b/0x75 [<c11f74d8>] ? sock_sendmsg+0x8d/0xa6 [<c11f7b83>] ? sys_sendto+0x105/0x130 [<c1025927>] ? __kunmap_atomic+0x62/0x8a [<c1025940>] ? __kunmap_atomic+0x7b/0x8a [<c1073d78>] ? __lru_cache_add+0x18/0x47 [<c10812f9>] ? handle_pte_fault+0x745/0x751 [<c1025a2d>] ? kmap_atomic_prot+0xd3/0xf1 [<c10817da>] ? handle_mm_fault+0x112/0x121 [<c11f7be5>] ? sys_send+0x37/0x3b
Quoted 7 lines
Quoted 7 lines
The system is busy, so, order-3 alloc failure doesn't strike me as odd.
There are no allocation failures with order != 3.
Sometimes this can happen in very short sucession, i.e. and oom-killer did end up zapping 30 processes or so.
Aside from the __GFP_NORETRY issue, could you post the full oom killer log where it kills more than one process with /proc/sys/vm/oom_dump_tasks enabled? That shouldn't happen unless you have a memory leak.
Quoted 28 lines · David Rientjes
Quoted 28 lines · Florian Westphal
Quoted 27 lines
seems like
sk_page_frag_refill()can cause oom-killer invocation:postgres invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0
Pid: 10551, comm: postgres Tainted: G O 3.8.6-5.g613ca40-smp #1Call Trace: [<c106dbd5>] ? dump_header+0x60/0x191 [<c1133d3a>] ? ___ratelimit+0xb2/0xc4 [<c106dfd3>] ? oom_kill_process+0x61/0x2d1 [<c1030042>] ? has_capability_noaudit+0x1c/0x23 [<c106df0f>] ? oom_badness+0x8c/0xef [<c106e446>] ? out_of_memory+0x203/0x247 [<c107128a>] ? __alloc_pages_nodemask+0x42b/0x4c3 [<c11fa66a>] ? sk_page_frag_refill+0x6a/0xd2 [<c1233548>] ? tcp_sendmsg+0x3e8/0x7c6 [<c124f34b>] ? inet_sendmsg+0x6b/0x75 [<c11f74d8>] ? sock_sendmsg+0x8d/0xa6 [<c11f7b83>] ? sys_sendto+0x105/0x130 [<c1025927>] ? __kunmap_atomic+0x62/0x8a [<c1025940>] ? __kunmap_atomic+0x7b/0x8a [<c1073d78>] ? __lru_cache_add+0x18/0x47 [<c10812f9>] ? handle_pte_fault+0x745/0x751 [<c1025a2d>] ? kmap_atomic_prot+0xd3/0xf1 [<c10817da>] ? handle_mm_fault+0x112/0x121 [<c11f7be5>] ? sys_send+0x37/0x3bThe system is busy, so, order-3 alloc failure doesn't strike me as odd.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
There are no allocation failures with order != 3.
Sometimes this can happen in very short sucession, i.e. and oom-killer did end up zapping 30 processes or so.
Aside from the
__GFP_NORETRYissue, could you post the full oom killer log where it kills more than one process with/proc/sys/vm/oom_dump_tasksenabled? That shouldn't happen unless you have a memory leak.
HOWEVER, before you spend time on this:
I don't think there is an issue with oom killer, I only noticed that after kernel update oom killer invocations are
a) more frequent than before
b) often show above backtrace (sk_page_frag_refill).
I'm not even saying "sk_page_frag_refill is broken", since I don't know if adding GFP_NORETRY might add silent performance degradation, etc.
Its very well possible that everything is working as intended :-)