Skip to contentSkip to conversation

sk_page_frag_refill OOM killing spree

DISCUSSION Florian Westphal 3 participants 4 messages May 21, 2013 – May 22, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi Eric,

seems like sk_page_frag_refill() can cause oom-killer invocation:

postgres invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0
Pid: 10551, comm: postgres Tainted: G O 3.8.6-5.g613ca40-smp #1

Call Trace:
 [<c106dbd5>] ? dump_header+0x60/0x191
 [<c1133d3a>] ? ___ratelimit+0xb2/0xc4
 [<c106dfd3>] ? oom_kill_process+0x61/0x2d1
 [<c1030042>] ? has_capability_noaudit+0x1c/0x23
 [<c106df0f>] ? oom_badness+0x8c/0xef
 [<c106e446>] ? out_of_memory+0x203/0x247
 [<c107128a>] ? __alloc_pages_nodemask+0x42b/0x4c3
 [<c11fa66a>] ? sk_page_frag_refill+0x6a/0xd2
 [<c1233548>] ? tcp_sendmsg+0x3e8/0x7c6
 [<c124f34b>] ? inet_sendmsg+0x6b/0x75
 [<c11f74d8>] ? sock_sendmsg+0x8d/0xa6
 [<c11f7b83>] ? sys_sendto+0x105/0x130
 [<c1025927>] ? __kunmap_atomic+0x62/0x8a
 [<c1025940>] ? __kunmap_atomic+0x7b/0x8a
 [<c1073d78>] ? __lru_cache_add+0x18/0x47
 [<c10812f9>] ? handle_pte_fault+0x745/0x751
 [<c1025a2d>] ? kmap_atomic_prot+0xd3/0xf1
 [<c10817da>] ? handle_mm_fault+0x112/0x121
 [<c11f7be5>] ? sys_send+0x37/0x3b

The system is busy, so, order-3 alloc failure doesn't strike me as odd.

There are no allocation failures with order != 3.

Sometimes this can happen in very short sucession, i.e. and oom-killer did end up zapping 30 processes or so.

My question is, should sk_page_frag_refill use __GFP_NORETRY, at least for order 3 requests?

#1 lore raw
Quoted 29 lines · Florian Westphal

Hi Eric,

seems like sk_page_frag_refill() can cause oom-killer invocation:

postgres invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0
Pid: 10551, comm: postgres Tainted: G O 3.8.6-5.g613ca40-smp #1

Call Trace:
 [<c106dbd5>] ? dump_header+0x60/0x191
 [<c1133d3a>] ? ___ratelimit+0xb2/0xc4
 [<c106dfd3>] ? oom_kill_process+0x61/0x2d1
 [<c1030042>] ? has_capability_noaudit+0x1c/0x23
 [<c106df0f>] ? oom_badness+0x8c/0xef
 [<c106e446>] ? out_of_memory+0x203/0x247
 [<c107128a>] ? __alloc_pages_nodemask+0x42b/0x4c3
 [<c11fa66a>] ? sk_page_frag_refill+0x6a/0xd2
 [<c1233548>] ? tcp_sendmsg+0x3e8/0x7c6
 [<c124f34b>] ? inet_sendmsg+0x6b/0x75
 [<c11f74d8>] ? sock_sendmsg+0x8d/0xa6
 [<c11f7b83>] ? sys_sendto+0x105/0x130
 [<c1025927>] ? __kunmap_atomic+0x62/0x8a
 [<c1025940>] ? __kunmap_atomic+0x7b/0x8a
 [<c1073d78>] ? __lru_cache_add+0x18/0x47
 [<c10812f9>] ? handle_pte_fault+0x745/0x751
 [<c1025a2d>] ? kmap_atomic_prot+0xd3/0xf1
 [<c10817da>] ? handle_mm_fault+0x112/0x121
 [<c11f7be5>] ? sys_send+0x37/0x3b

The system is busy, so, order-3 alloc failure doesn't strike me as odd.

Quoted 7 lines

There are no allocation failures with order != 3.

Sometimes this can happen in very short sucession, i.e. and oom-killer did end up zapping 30 processes or so.

My question is, should sk_page_frag_refill use __GFP_NORETRY, at least for order 3 requests?

Yes, you may be right, but I prefer some mm expertise here, so I CC lkml

#2 lore raw
Quoted 28 lines · Eric Dumazet
Quoted 27 lines · Florian Westphal

Hi Eric,

seems like sk_page_frag_refill() can cause oom-killer invocation:

postgres invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0
Pid: 10551, comm: postgres Tainted: G O 3.8.6-5.g613ca40-smp #1

Call Trace:
 [<c106dbd5>] ? dump_header+0x60/0x191
 [<c1133d3a>] ? ___ratelimit+0xb2/0xc4
 [<c106dfd3>] ? oom_kill_process+0x61/0x2d1
 [<c1030042>] ? has_capability_noaudit+0x1c/0x23
 [<c106df0f>] ? oom_badness+0x8c/0xef
 [<c106e446>] ? out_of_memory+0x203/0x247
 [<c107128a>] ? __alloc_pages_nodemask+0x42b/0x4c3
 [<c11fa66a>] ? sk_page_frag_refill+0x6a/0xd2
 [<c1233548>] ? tcp_sendmsg+0x3e8/0x7c6
 [<c124f34b>] ? inet_sendmsg+0x6b/0x75
 [<c11f74d8>] ? sock_sendmsg+0x8d/0xa6
 [<c11f7b83>] ? sys_sendto+0x105/0x130
 [<c1025927>] ? __kunmap_atomic+0x62/0x8a
 [<c1025940>] ? __kunmap_atomic+0x7b/0x8a
 [<c1073d78>] ? __lru_cache_add+0x18/0x47
 [<c10812f9>] ? handle_pte_fault+0x745/0x751
 [<c1025a2d>] ? kmap_atomic_prot+0xd3/0xf1
 [<c10817da>] ? handle_mm_fault+0x112/0x121
 [<c11f7be5>] ? sys_send+0x37/0x3b
Quoted 7 lines
Quoted 7 lines

The system is busy, so, order-3 alloc failure doesn't strike me as odd.

There are no allocation failures with order != 3.

Sometimes this can happen in very short sucession, i.e. and oom-killer did end up zapping 30 processes or so.

Aside from the __GFP_NORETRY issue, could you post the full oom killer log where it kills more than one process with /proc/sys/vm/oom_dump_tasks enabled? That shouldn't happen unless you have a memory leak.

#3 lore raw
Quoted 28 lines · David Rientjes
Quoted 28 lines · Florian Westphal
Quoted 27 lines

seems like sk_page_frag_refill() can cause oom-killer invocation:

postgres invoked oom-killer: gfp_mask=0x42d0, order=3, oom_score_adj=0
Pid: 10551, comm: postgres Tainted: G O 3.8.6-5.g613ca40-smp #1

Call Trace:
 [<c106dbd5>] ? dump_header+0x60/0x191
 [<c1133d3a>] ? ___ratelimit+0xb2/0xc4
 [<c106dfd3>] ? oom_kill_process+0x61/0x2d1
 [<c1030042>] ? has_capability_noaudit+0x1c/0x23
 [<c106df0f>] ? oom_badness+0x8c/0xef
 [<c106e446>] ? out_of_memory+0x203/0x247
 [<c107128a>] ? __alloc_pages_nodemask+0x42b/0x4c3
 [<c11fa66a>] ? sk_page_frag_refill+0x6a/0xd2
 [<c1233548>] ? tcp_sendmsg+0x3e8/0x7c6
 [<c124f34b>] ? inet_sendmsg+0x6b/0x75
 [<c11f74d8>] ? sock_sendmsg+0x8d/0xa6
 [<c11f7b83>] ? sys_sendto+0x105/0x130
 [<c1025927>] ? __kunmap_atomic+0x62/0x8a
 [<c1025940>] ? __kunmap_atomic+0x7b/0x8a
 [<c1073d78>] ? __lru_cache_add+0x18/0x47
 [<c10812f9>] ? handle_pte_fault+0x745/0x751
 [<c1025a2d>] ? kmap_atomic_prot+0xd3/0xf1
 [<c10817da>] ? handle_mm_fault+0x112/0x121
 [<c11f7be5>] ? sys_send+0x37/0x3b

The system is busy, so, order-3 alloc failure doesn't strike me as odd.

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

There are no allocation failures with order != 3.

Sometimes this can happen in very short sucession, i.e. and oom-killer did end up zapping 30 processes or so.

Aside from the __GFP_NORETRY issue, could you post the full oom killer log where it kills more than one process with /proc/sys/vm/oom_dump_tasks enabled? That shouldn't happen unless you have a memory leak.

http://strlen.de/fw/oom.txt

HOWEVER, before you spend time on this:
I don't think there is an issue with oom killer, I only noticed that after kernel update oom killer invocations are
a) more frequent than before
b) often show above backtrace (sk_page_frag_refill).

I'm not even saying "sk_page_frag_refill is broken", since I don't know if adding GFP_NORETRY might add silent performance degradation, etc.

Its very well possible that everything is working as intended :-)