net: fix sk_page_frag() recursion from memory reclaim
Subsystems (6)
PATCH 1/1 net: fix sk_page_frag() recursion from memory reclaim
From f0335a5d14d3596d36e3ffddb2fd4fa0dc6ca9c2 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Sat, 19 Oct 2019 09:10:57 -0700
sk_page_frag() optimizes skb_frag allocations by using per-task skb_frag cache when it knows it's the only user. The condition is determined by seeing whether the socket allocation mask allows blocking - if the allocation may block, it obviously owns the task's context and ergo exclusively owns current->task_frag.
Unfortunately, this misses recursion through memory reclaim path. Please take a look at the following backtrace.
[2] RIP: 0010:tcp_sendmsg_locked+0xccf/0xe10
...
tcp_sendmsg+0x27/0x40
sock_sendmsg+0x30/0x40
sock_xmit.isra.24+0xa1/0x170 [nbd]
nbd_send_cmd+0x1d2/0x690 [nbd]
nbd_queue_rq+0x1b5/0x3b0 [nbd]
__blk_mq_try_issue_directly+0x108/0x1b0
blk_mq_request_issue_directly+0xbd/0xe0
blk_mq_try_issue_list_directly+0x41/0xb0
blk_mq_sched_insert_requests+0xa2/0xe0
blk_mq_flush_plug_list+0x205/0x2a0
blk_flush_plug_list+0xc3/0xf0
[1] blk_finish_plug+0x21/0x2e
_xfs_buf_ioapply+0x313/0x460
__xfs_buf_submit+0x67/0x220
xfs_buf_read_map+0x113/0x1a0
xfs_trans_read_buf_map+0xbf/0x330
xfs_btree_read_buf_block.constprop.42+0x95/0xd0
xfs_btree_lookup_get_block+0x95/0x170
xfs_btree_lookup+0xcc/0x470
xfs_bmap_del_extent_real+0x254/0x9a0
__xfs_bunmapi+0x45c/0xab0
xfs_bunmapi+0x15/0x30
xfs_itruncate_extents_flags+0xca/0x250
xfs_free_eofblocks+0x181/0x1e0
xfs_fs_destroy_inode+0xa8/0x1b0
destroy_inode+0x38/0x70
dispose_list+0x35/0x50
prune_icache_sb+0x52/0x70
super_cache_scan+0x120/0x1a0
do_shrink_slab+0x120/0x290
shrink_slab+0x216/0x2b0
shrink_node+0x1b6/0x4a0
do_try_to_free_pages+0xc6/0x370
try_to_free_mem_cgroup_pages+0xe3/0x1e0
try_charge+0x29e/0x790
mem_cgroup_charge_skmem+0x6a/0x100
__sk_mem_raise_allocated+0x18e/0x390
__sk_mem_schedule+0x2a/0x40
[0] tcp_sendmsg_locked+0x8eb/0xe10
tcp_sendmsg+0x27/0x40
sock_sendmsg+0x30/0x40
___sys_sendmsg+0x26d/0x2b0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x42/0x100
entry_SYSCALL_64_after_hwframe+0x44/0xa9In [0], tcp_send_msg_locked() was using current->page_frag when it called sk_wmem_schedule(). It already calculated how many bytes can be fit into current->page_frag. Due to memory pressure, sk_wmem_schedule() called into memory reclaim path which called into xfs and then IO issue path. Because the filesystem in question is backed by nbd, the control goes back into the tcp layer - back into tcp_sendmsg_locked().
nbd sets sk_allocation to (GFP_NOIO | __GFP_MEMALLOC) which makes sense - it's in the process of freeing memory and wants to be able to, e.g., drop clean pages to make forward progress. However, this confused sk_page_frag() called from [2]. Because it only tests whether the allocation allows blocking which it does, it now thinks current->page_frag can be used again although it already was being used in [0].
After [2] used current->page_frag, the offset would be increased by the used amount. When the control returns to [0], current->page_frag's offset is increased and the previously calculated number of bytes now may overrun the end of allocated memory leading to silent memory corruptions.
Fix it by updating sk_page_frag() to test __GFP_MEMALLOC and not use current->task_frag if set.
--- include/net/sock.h | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
include/net/sock.h +12 -3
@@ -2233,12 +2233,21 @@ struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp,* sk_page_frag - return an appropriate page_frag * @sk: socket * -* If socket allocation mode allows current thread to sleep, it means its -* safe to use the per task page_frag instead of the per socket one. +* Use the per task page_frag instead of the per socket one for +* optimization when we know there can be no other users. +* +* 1. The socket allocation mode allows current thread to sleep. This is +* the sleepable context which owns the task page_frag. +* +* 2. The socket allocation mode doesn't indicate that the socket is being +* used to reclaim memory. Memory reclaim may nest inside other socket +* operations and end up recursing into sk_page_frag() while it's +* already in use. */ static inline struct page_frag *sk_page_frag (struct sock *sk) { -if (gfpflags_allow_blocking (sk->sk_allocation)) +if (gfpflags_allow_blocking (sk->sk_allocation) && + !(sk->sk_allocation & __GFP_MEMALLOC))return ¤t->task_frag;return &sk->sk_frag;
Signature · 1 line
2.17.1
Quoted 113 lines · Tejun Heo
From
f0335a5d14d3596d36e3ffddb2fd4fa0dc6ca9c2Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Sat, 19 Oct 2019 09:10:57 -0700
sk_page_frag()optimizesskb_fragallocations by using per-taskskb_fragcache when it knows it's the only user. The condition is determined by seeing whether the socket allocation mask allows blocking - if the allocation may block, it obviously owns the task's context and ergo exclusively ownscurrent->task_frag.Unfortunately, this misses recursion through memory reclaim path. Please take a look at the following backtrace.
[2] RIP: 0010:tcp_sendmsg_locked+0xccf/0xe10
... tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 sock_xmit.isra.24+0xa1/0x170 [nbd] nbd_send_cmd+0x1d2/0x690 [nbd] nbd_queue_rq+0x1b5/0x3b0 [nbd] __blk_mq_try_issue_directly+0x108/0x1b0 blk_mq_request_issue_directly+0xbd/0xe0 blk_mq_try_issue_list_directly+0x41/0xb0 blk_mq_sched_insert_requests+0xa2/0xe0 blk_mq_flush_plug_list+0x205/0x2a0 blk_flush_plug_list+0xc3/0xf0 [1] blk_finish_plug+0x21/0x2e _xfs_buf_ioapply+0x313/0x460 __xfs_buf_submit+0x67/0x220 xfs_buf_read_map+0x113/0x1a0 xfs_trans_read_buf_map+0xbf/0x330 xfs_btree_read_buf_block.constprop.42+0x95/0xd0 xfs_btree_lookup_get_block+0x95/0x170 xfs_btree_lookup+0xcc/0x470 xfs_bmap_del_extent_real+0x254/0x9a0 __xfs_bunmapi+0x45c/0xab0 xfs_bunmapi+0x15/0x30 xfs_itruncate_extents_flags+0xca/0x250 xfs_free_eofblocks+0x181/0x1e0 xfs_fs_destroy_inode+0xa8/0x1b0 destroy_inode+0x38/0x70 dispose_list+0x35/0x50 prune_icache_sb+0x52/0x70 super_cache_scan+0x120/0x1a0 do_shrink_slab+0x120/0x290 shrink_slab+0x216/0x2b0 shrink_node+0x1b6/0x4a0 do_try_to_free_pages+0xc6/0x370 try_to_free_mem_cgroup_pages+0xe3/0x1e0 try_charge+0x29e/0x790 mem_cgroup_charge_skmem+0x6a/0x100 __sk_mem_raise_allocated+0x18e/0x390 __sk_mem_schedule+0x2a/0x40 [0] tcp_sendmsg_locked+0x8eb/0xe10 tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 ___sys_sendmsg+0x26d/0x2b0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x42/0x100 entry_SYSCALL_64_after_hwframe+0x44/0xa9In [0],
tcp_send_msg_locked()was usingcurrent->page_fragwhen it calledsk_wmem_schedule(). It already calculated how many bytes can be fit intocurrent->page_frag. Due to memory pressure,sk_wmem_schedule()called into memory reclaim path which called into xfs and then IO issue path. Because the filesystem in question is backed by nbd, the control goes back into the tcp layer - back intotcp_sendmsg_locked().nbd sets
sk_allocationto (GFP_NOIO|__GFP_MEMALLOC) which makes sense - it's in the process of freeing memory and wants to be able to, e.g., drop clean pages to make forward progress. However, this confusedsk_page_frag()called from [2]. Because it only tests whether the allocation allows blocking which it does, it now thinkscurrent->page_fragcan be used again although it already was being used in [0].After [2] used
current->page_frag, the offset would be increased by the used amount. When the control returns to [0], current->page_frag's offset is increased and the previously calculated number of bytes now may overrun the end of allocated memory leading to silent memory corruptions.Fix it by updating
sk_page_frag()to test__GFP_MEMALLOCand not usecurrent->task_fragif set.--- include/net/sock.h | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)include/net/sock.h +9
-2@@ -2233,12 +2233,21 @@ struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp,* sk_page_frag - return an appropriate page_frag * @sk: socket * -* If socket allocation mode allows current thread to sleep, it means its -* safe to use the per task page_frag instead of the per socket one. +* Use the per task page_frag instead of the per socket one for +* optimization when we know there can be no other users. +* +* 1. The socket allocation mode allows current thread to sleep. This is +* the sleepable context which owns the task page_frag. +* +* 2. The socket allocation mode doesn't indicate that the socket is being +* used to reclaim memory. Memory reclaim may nest inside other socket +* operations and end up recursing into sk_page_frag() while it's
include/net/sock.h struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp, · line 2253
Quoted 10 lines
include/net/sock.h +3
-1+* already in use. */ static inline struct page_frag *sk_page_frag (struct sock *sk) { -if (gfpflags_allow_blocking (sk->sk_allocation)) +if (gfpflags_allow_blocking (sk->sk_allocation) && + !(sk->sk_allocation & __GFP_MEMALLOC))return ¤t->task_frag;return &sk->sk_frag;
It seems compiler generates better code with :
include/net/sock.h +2 -1
@@ -2238,7 +2238,8 @@ struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp,*/ static inline struct page_frag *sk_page_frag (struct sock *sk) { -if (gfpflags_allow_blocking (sk->sk_allocation)) +if (likely ((sk->sk_allocation & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC)) == + __GFP_DIRECT_RECLAIM))return ¤t->task_frag;return &sk->sk_frag;
WDYT ?
Thanks !
Hello,
Quoted 18 lines · Eric Dumazet
It seems compiler generates better code with :
include/net/sock.h +2
-1@@ -2238,7 +2238,8 @@ struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp,*/ static inline struct page_frag *sk_page_frag (struct sock *sk) { -if (gfpflags_allow_blocking (sk->sk_allocation)) +if (likely ((sk->sk_allocation & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC)) == + __GFP_DIRECT_RECLAIM))return ¤t->task_frag;return &sk->sk_frag;
Quoted 1 line
WDYT ?
Whatever works is fine by me. gfpflags_allow_blocking() is clearer than testing __GFP_DIRECT_RECLAIM directly tho. Maybe a better way is introducing a new gfpflags_ helper?
Thanks.
Signature · 1 line
tejun
Quoted 3 lines · Tejun Heo
Whatever works is fine by me.
gfpflags_allow_blocking()is clearer than testing__GFP_DIRECT_RECLAIMdirectly tho. Maybe a better way is introducing a newgfpflags_helper?
Sounds good to me !
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Tejun Heo
Whatever works is fine by me.
gfpflags_allow_blocking()is clearer than testing__GFP_DIRECT_RECLAIMdirectly tho. Maybe a better way is introducing a newgfpflags_helper?Sounds good to me !
IIUC there will be a v2 with a new helper, dropping this from patchwork.
PATCH v2 1/1 net: fix sk_page_frag() recursion from memory reclaim
In reply to Tejun Heosk_page_frag() optimizes skb_frag allocations by using per-task skb_frag cache when it knows it's the only user. The condition is determined by seeing whether the socket allocation mask allows blocking - if the allocation may block, it obviously owns the task's context and ergo exclusively owns current->task_frag.
Unfortunately, this misses recursion through memory reclaim path. Please take a look at the following backtrace.
[2] RIP: 0010:tcp_sendmsg_locked+0xccf/0xe10
...
tcp_sendmsg+0x27/0x40
sock_sendmsg+0x30/0x40
sock_xmit.isra.24+0xa1/0x170 [nbd]
nbd_send_cmd+0x1d2/0x690 [nbd]
nbd_queue_rq+0x1b5/0x3b0 [nbd]
__blk_mq_try_issue_directly+0x108/0x1b0
blk_mq_request_issue_directly+0xbd/0xe0
blk_mq_try_issue_list_directly+0x41/0xb0
blk_mq_sched_insert_requests+0xa2/0xe0
blk_mq_flush_plug_list+0x205/0x2a0
blk_flush_plug_list+0xc3/0xf0
[1] blk_finish_plug+0x21/0x2e
_xfs_buf_ioapply+0x313/0x460
__xfs_buf_submit+0x67/0x220
xfs_buf_read_map+0x113/0x1a0
xfs_trans_read_buf_map+0xbf/0x330
xfs_btree_read_buf_block.constprop.42+0x95/0xd0
xfs_btree_lookup_get_block+0x95/0x170
xfs_btree_lookup+0xcc/0x470
xfs_bmap_del_extent_real+0x254/0x9a0
__xfs_bunmapi+0x45c/0xab0
xfs_bunmapi+0x15/0x30
xfs_itruncate_extents_flags+0xca/0x250
xfs_free_eofblocks+0x181/0x1e0
xfs_fs_destroy_inode+0xa8/0x1b0
destroy_inode+0x38/0x70
dispose_list+0x35/0x50
prune_icache_sb+0x52/0x70
super_cache_scan+0x120/0x1a0
do_shrink_slab+0x120/0x290
shrink_slab+0x216/0x2b0
shrink_node+0x1b6/0x4a0
do_try_to_free_pages+0xc6/0x370
try_to_free_mem_cgroup_pages+0xe3/0x1e0
try_charge+0x29e/0x790
mem_cgroup_charge_skmem+0x6a/0x100
__sk_mem_raise_allocated+0x18e/0x390
__sk_mem_schedule+0x2a/0x40
[0] tcp_sendmsg_locked+0x8eb/0xe10
tcp_sendmsg+0x27/0x40
sock_sendmsg+0x30/0x40
___sys_sendmsg+0x26d/0x2b0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x42/0x100
entry_SYSCALL_64_after_hwframe+0x44/0xa9In [0], tcp_send_msg_locked() was using current->page_frag when it called sk_wmem_schedule(). It already calculated how many bytes can be fit into current->page_frag. Due to memory pressure, sk_wmem_schedule() called into memory reclaim path which called into xfs and then IO issue path. Because the filesystem in question is backed by nbd, the control goes back into the tcp layer - back into tcp_sendmsg_locked().
nbd sets sk_allocation to (GFP_NOIO | __GFP_MEMALLOC) which makes sense - it's in the process of freeing memory and wants to be able to, e.g., drop clean pages to make forward progress. However, this confused sk_page_frag() called from [2]. Because it only tests whether the allocation allows blocking which it does, it now thinks current->page_frag can be used again although it already was being used in [0].
After [2] used current->page_frag, the offset would be increased by the used amount. When the control returns to [0], current->page_frag's offset is increased and the previously calculated number of bytes now may overrun the end of allocated memory leading to silent memory corruptions.
Fix it by adding gfpflags_normal_context() which tests sleepable && !reclaim and use it to determine whether to use current->task_frag.
v2: Eric didn't like gfp flags being tested twice. Introduce a new
helper gfpflags_normal_context() and combine the two tests.
--- include/linux/gfp.h | 23 +++++++++++++++++++++++ include/net/sock.h | 11 ++++++++--- 2 files changed, 31 insertions(+), 3 deletions(-)
include/linux/gfp.h +23 -0
@@ -325,6 +325,29 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags)return !!(gfp_flags & __GFP_DIRECT_RECLAIM); } +/** +* gfpflags_normal_context - is gfp_flags a normal sleepable context? +* @gfp_flags: gfp_flags to test +* +* Test whether @gfp_flags indicates that the allocation is from the +* %current context and allowed to sleep. +* +* An allocation being allowed to block doesn't mean it owns the %current +* context. When direct reclaim path tries to allocate memory, the +* allocation context is nested inside whatever %current was doing at the +* time of the original allocation. The nested allocation may be allowed +* to block but modifying anything %current owns can corrupt the outer +* context's expectations. +* +* %true result from this function indicates that the allocation context +* can sleep and use anything that's associated with %current. +*/ +static inline bool gfpflags_normal_context (const gfp_t gfp_flags) +{ +return (gfp_flags & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC)) == + __GFP_DIRECT_RECLAIM; +} +#ifdef CONFIG_HIGHMEM #define OPT_ZONE_HIGHMEM ZONE_HIGHMEM#else
include/net/sock.h +8 -3
@@ -2242,12 +2242,17 @@ struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp,* sk_page_frag - return an appropriate page_frag * @sk: socket * -* If socket allocation mode allows current thread to sleep, it means its -* safe to use the per task page_frag instead of the per socket one. +* Use the per task page_frag instead of the per socket one for +* optimization when we know that we're in the normal context and owns +* everything that's associated with %current. +* +* gfpflags_allow_blocking() isn't enough here as direct reclaim may nest +* inside other socket operations and end up recursing into sk_page_frag() +* while it's already in use. */ static inline struct page_frag *sk_page_frag (struct sock *sk) { -if (gfpflags_allow_blocking (sk->sk_allocation)) +if (gfpflags_normal_context (sk->sk_allocation))return ¤t->task_frag;return &sk->sk_frag;
David Miller
maintainerFrom: Tejun Heo <[email protected]>
Date: Thu, 24 Oct 2019 13:50:27 -0700
Quoted 8 lines
sk_page_frag()optimizesskb_fragallocations by using per-taskskb_fragcache when it knows it's the only user. The condition is determined by seeing whether the socket allocation mask allows blocking - if the allocation may block, it obviously owns the task's context and ergo exclusively ownscurrent->task_frag.Unfortunately, this misses recursion through memory reclaim path. Please take a look at the following backtrace.
...
Quoted 22 lines
In [0],
tcp_send_msg_locked()was usingcurrent->page_fragwhen it calledsk_wmem_schedule(). It already calculated how many bytes can be fit intocurrent->page_frag. Due to memory pressure,sk_wmem_schedule()called into memory reclaim path which called into xfs and then IO issue path. Because the filesystem in question is backed by nbd, the control goes back into the tcp layer - back intotcp_sendmsg_locked().nbd sets
sk_allocationto (GFP_NOIO|__GFP_MEMALLOC) which makes sense - it's in the process of freeing memory and wants to be able to, e.g., drop clean pages to make forward progress. However, this confusedsk_page_frag()called from [2]. Because it only tests whether the allocation allows blocking which it does, it now thinkscurrent->page_fragcan be used again although it already was being used in [0].After [2] used
current->page_frag, the offset would be increased by the used amount. When the control returns to [0], current->page_frag's offset is increased and the previously calculated number of bytes now may overrun the end of allocated memory leading to silent memory corruptions.
Quoted 7 lines
Fix it by adding
gfpflags_normal_context()which tests sleepable && !reclaim and use it to determine whether to usecurrent->task_frag.v2: Eric didn't like gfp flags being tested twice. Introduce a new
helpergfpflags_normal_context()and combine the two tests.Signed-off-by Tejun Heo <[email protected]>
Applied and queued up for -stable, thanks Tejun.
+Michal Hocko
Quoted 114 lines · Tejun Heo
sk_page_frag()optimizesskb_fragallocations by using per-taskskb_fragcache when it knows it's the only user. The condition is determined by seeing whether the socket allocation mask allows blocking - if the allocation may block, it obviously owns the task's context and ergo exclusively ownscurrent->task_frag.Unfortunately, this misses recursion through memory reclaim path. Please take a look at the following backtrace.
[2] RIP: 0010:tcp_sendmsg_locked+0xccf/0xe10
... tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 sock_xmit.isra.24+0xa1/0x170 [nbd] nbd_send_cmd+0x1d2/0x690 [nbd] nbd_queue_rq+0x1b5/0x3b0 [nbd] __blk_mq_try_issue_directly+0x108/0x1b0 blk_mq_request_issue_directly+0xbd/0xe0 blk_mq_try_issue_list_directly+0x41/0xb0 blk_mq_sched_insert_requests+0xa2/0xe0 blk_mq_flush_plug_list+0x205/0x2a0 blk_flush_plug_list+0xc3/0xf0 [1] blk_finish_plug+0x21/0x2e _xfs_buf_ioapply+0x313/0x460 __xfs_buf_submit+0x67/0x220 xfs_buf_read_map+0x113/0x1a0 xfs_trans_read_buf_map+0xbf/0x330 xfs_btree_read_buf_block.constprop.42+0x95/0xd0 xfs_btree_lookup_get_block+0x95/0x170 xfs_btree_lookup+0xcc/0x470 xfs_bmap_del_extent_real+0x254/0x9a0 __xfs_bunmapi+0x45c/0xab0 xfs_bunmapi+0x15/0x30 xfs_itruncate_extents_flags+0xca/0x250 xfs_free_eofblocks+0x181/0x1e0 xfs_fs_destroy_inode+0xa8/0x1b0 destroy_inode+0x38/0x70 dispose_list+0x35/0x50 prune_icache_sb+0x52/0x70 super_cache_scan+0x120/0x1a0 do_shrink_slab+0x120/0x290 shrink_slab+0x216/0x2b0 shrink_node+0x1b6/0x4a0 do_try_to_free_pages+0xc6/0x370 try_to_free_mem_cgroup_pages+0xe3/0x1e0 try_charge+0x29e/0x790 mem_cgroup_charge_skmem+0x6a/0x100 __sk_mem_raise_allocated+0x18e/0x390 __sk_mem_schedule+0x2a/0x40 [0] tcp_sendmsg_locked+0x8eb/0xe10 tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 ___sys_sendmsg+0x26d/0x2b0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x42/0x100 entry_SYSCALL_64_after_hwframe+0x44/0xa9In [0],
tcp_send_msg_locked()was usingcurrent->page_fragwhen it calledsk_wmem_schedule(). It already calculated how many bytes can be fit intocurrent->page_frag. Due to memory pressure,sk_wmem_schedule()called into memory reclaim path which called into xfs and then IO issue path. Because the filesystem in question is backed by nbd, the control goes back into the tcp layer - back intotcp_sendmsg_locked().nbd sets
sk_allocationto (GFP_NOIO|__GFP_MEMALLOC) which makes sense - it's in the process of freeing memory and wants to be able to, e.g., drop clean pages to make forward progress. However, this confusedsk_page_frag()called from [2]. Because it only tests whether the allocation allows blocking which it does, it now thinkscurrent->page_fragcan be used again although it already was being used in [0].After [2] used
current->page_frag, the offset would be increased by the used amount. When the control returns to [0], current->page_frag's offset is increased and the previously calculated number of bytes now may overrun the end of allocated memory leading to silent memory corruptions.Fix it by adding
gfpflags_normal_context()which tests sleepable && !reclaim and use it to determine whether to usecurrent->task_frag.v2: Eric didn't like gfp flags being tested twice. Introduce a new
helpergfpflags_normal_context()and combine the two tests.Signed-off-by Tejun Heo <[email protected]>Cc Josef Bacik <[email protected]>Cc Eric Dumazet <[email protected]>--- include/linux/gfp.h | 23 +++++++++++++++++++++++ include/net/sock.h | 11 ++++++++--- 2 files changed, 31 insertions(+), 3 deletions(-)include/linux/gfp.h +11
-0@@ -325,6 +325,29 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags)return !!(gfp_flags & __GFP_DIRECT_RECLAIM); } +/** +* gfpflags_normal_context - is gfp_flags a normal sleepable context? +* @gfp_flags: gfp_flags to test +* +* Test whether @gfp_flags indicates that the allocation is from the +* %current context and allowed to sleep. +* +* An allocation being allowed to block doesn't mean it owns the %current +* context. When direct reclaim path tries to allocate memory, the +* allocation context is nested inside whatever %current was doing at the +* time of the original allocation. The nested allocation may be allowed
include/linux/gfp.h static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) · line 348
Quoted 10 lines
include/linux/gfp.h +10
-0+* to block but modifying anything %current owns can corrupt the outer +* context's expectations. +* +* %true result from this function indicates that the allocation context +* can sleep and use anything that's associated with %current. +*/ +static inline bool gfpflags_normal_context (const gfp_t gfp_flags) +{ +return (gfp_flags & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC)) == + __GFP_DIRECT_RECLAIM;
I think we should be checking PF_MEMALLOC here instead. Something like:
return gfpflags_allow_blocking(gfp_flags) && !(current->flags & PF_MEMALLOC);
In my limited understanding, __GFP_MEMALLOC gives access to reserve but we have overloaded PF_MEMALLOC to also define the reclaim context. There are PF_MEMALLOC users which does not use __GFP_MEMALLOC like iscsi_sw_tcp_pdu_xmit() which can call sock_sendmsg().
Quoted 30 lines
include/linux/gfp.h +2
-0+} +#ifdef CONFIG_HIGHMEM #define OPT_ZONE_HIGHMEM ZONE_HIGHMEM#else include/net/sock.h +8
-3@@ -2242,12 +2242,17 @@ struct sk_buff *sk_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp,* sk_page_frag - return an appropriate page_frag * @sk: socket * -* If socket allocation mode allows current thread to sleep, it means its -* safe to use the per task page_frag instead of the per socket one. +* Use the per task page_frag instead of the per socket one for +* optimization when we know that we're in the normal context and owns +* everything that's associated with %current. +* +* gfpflags_allow_blocking() isn't enough here as direct reclaim may nest +* inside other socket operations and end up recursing into sk_page_frag() +* while it's already in use. */ static inline struct page_frag *sk_page_frag (struct sock *sk) { -if (gfpflags_allow_blocking (sk->sk_allocation)) +if (gfpflags_normal_context (sk->sk_allocation))return ¤t->task_frag;return &sk->sk_frag;
Quoted 130 lines · Shakeel Butt
+Michal Hocko
Quoted 114 lines · Tejun Heo
sk_page_frag()optimizesskb_fragallocations by using per-taskskb_fragcache when it knows it's the only user. The condition is determined by seeing whether the socket allocation mask allows blocking - if the allocation may block, it obviously owns the task's context and ergo exclusively ownscurrent->task_frag.Unfortunately, this misses recursion through memory reclaim path. Please take a look at the following backtrace.
[2] RIP: 0010:tcp_sendmsg_locked+0xccf/0xe10
... tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 sock_xmit.isra.24+0xa1/0x170 [nbd] nbd_send_cmd+0x1d2/0x690 [nbd] nbd_queue_rq+0x1b5/0x3b0 [nbd] __blk_mq_try_issue_directly+0x108/0x1b0 blk_mq_request_issue_directly+0xbd/0xe0 blk_mq_try_issue_list_directly+0x41/0xb0 blk_mq_sched_insert_requests+0xa2/0xe0 blk_mq_flush_plug_list+0x205/0x2a0 blk_flush_plug_list+0xc3/0xf0 [1] blk_finish_plug+0x21/0x2e _xfs_buf_ioapply+0x313/0x460 __xfs_buf_submit+0x67/0x220 xfs_buf_read_map+0x113/0x1a0 xfs_trans_read_buf_map+0xbf/0x330 xfs_btree_read_buf_block.constprop.42+0x95/0xd0 xfs_btree_lookup_get_block+0x95/0x170 xfs_btree_lookup+0xcc/0x470 xfs_bmap_del_extent_real+0x254/0x9a0 __xfs_bunmapi+0x45c/0xab0 xfs_bunmapi+0x15/0x30 xfs_itruncate_extents_flags+0xca/0x250 xfs_free_eofblocks+0x181/0x1e0 xfs_fs_destroy_inode+0xa8/0x1b0 destroy_inode+0x38/0x70 dispose_list+0x35/0x50 prune_icache_sb+0x52/0x70 super_cache_scan+0x120/0x1a0 do_shrink_slab+0x120/0x290 shrink_slab+0x216/0x2b0 shrink_node+0x1b6/0x4a0 do_try_to_free_pages+0xc6/0x370 try_to_free_mem_cgroup_pages+0xe3/0x1e0 try_charge+0x29e/0x790 mem_cgroup_charge_skmem+0x6a/0x100 __sk_mem_raise_allocated+0x18e/0x390 __sk_mem_schedule+0x2a/0x40 [0] tcp_sendmsg_locked+0x8eb/0xe10 tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 ___sys_sendmsg+0x26d/0x2b0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x42/0x100 entry_SYSCALL_64_after_hwframe+0x44/0xa9In [0],
tcp_send_msg_locked()was usingcurrent->page_fragwhen it calledsk_wmem_schedule(). It already calculated how many bytes can be fit intocurrent->page_frag. Due to memory pressure,sk_wmem_schedule()called into memory reclaim path which called into xfs and then IO issue path. Because the filesystem in question is backed by nbd, the control goes back into the tcp layer - back intotcp_sendmsg_locked().nbd sets
sk_allocationto (GFP_NOIO|__GFP_MEMALLOC) which makes sense - it's in the process of freeing memory and wants to be able to, e.g., drop clean pages to make forward progress. However, this confusedsk_page_frag()called from [2]. Because it only tests whether the allocation allows blocking which it does, it now thinkscurrent->page_fragcan be used again although it already was being used in [0].After [2] used
current->page_frag, the offset would be increased by the used amount. When the control returns to [0], current->page_frag's offset is increased and the previously calculated number of bytes now may overrun the end of allocated memory leading to silent memory corruptions.Fix it by adding
gfpflags_normal_context()which tests sleepable && !reclaim and use it to determine whether to usecurrent->task_frag.v2: Eric didn't like gfp flags being tested twice. Introduce a new
helpergfpflags_normal_context()and combine the two tests.Signed-off-by Tejun Heo <[email protected]>Cc Josef Bacik <[email protected]>Cc Eric Dumazet <[email protected]>--- include/linux/gfp.h | 23 +++++++++++++++++++++++ include/net/sock.h | 11 ++++++++--- 2 files changed, 31 insertions(+), 3 deletions(-)include/linux/gfp.h +11
-0@@ -325,6 +325,29 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags)return !!(gfp_flags & __GFP_DIRECT_RECLAIM); } +/** +* gfpflags_normal_context - is gfp_flags a normal sleepable context? +* @gfp_flags: gfp_flags to test +* +* Test whether @gfp_flags indicates that the allocation is from the +* %current context and allowed to sleep. +* +* An allocation being allowed to block doesn't mean it owns the %current +* context. When direct reclaim path tries to allocate memory, the +* allocation context is nested inside whatever %current was doing at the +* time of the original allocation. The nested allocation may be allowed include/linux/gfp.h static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) · line 348
Quoted 10 lines
include/linux/gfp.h +10
-0+* to block but modifying anything %current owns can corrupt the outer +* context's expectations. +* +* %true result from this function indicates that the allocation context +* can sleep and use anything that's associated with %current. +*/ +static inline bool gfpflags_normal_context (const gfp_t gfp_flags) +{ +return (gfp_flags & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC)) == + __GFP_DIRECT_RECLAIM;I think we should be checking
PF_MEMALLOChere instead. Something like:
Quoted 6 lines
return
gfpflags_allow_blocking(gfp_flags)&& !(current->flags &PF_MEMALLOC);In my limited understanding,
__GFP_MEMALLOCgives access to reserve but we have overloadedPF_MEMALLOCto also define the reclaim context. There arePF_MEMALLOCusers which does not use__GFP_MEMALLOClikeiscsi_sw_tcp_pdu_xmit()which can callsock_sendmsg().
Why would this layer not set sk->sk_allocation to GFP_ATOMIC ?
And it also might call sk_set_memalloc() too.
Please double check scsi layer, I am pretty sure it did well at some point.
Quoted 141 lines · Eric Dumazet
Quoted 130 lines · Shakeel Butt
+Michal Hocko
Quoted 114 lines · Tejun Heo
sk_page_frag()optimizesskb_fragallocations by using per-taskskb_fragcache when it knows it's the only user. The condition is determined by seeing whether the socket allocation mask allows blocking - if the allocation may block, it obviously owns the task's context and ergo exclusively ownscurrent->task_frag.Unfortunately, this misses recursion through memory reclaim path. Please take a look at the following backtrace.
[2] RIP: 0010:tcp_sendmsg_locked+0xccf/0xe10
... tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 sock_xmit.isra.24+0xa1/0x170 [nbd] nbd_send_cmd+0x1d2/0x690 [nbd] nbd_queue_rq+0x1b5/0x3b0 [nbd] __blk_mq_try_issue_directly+0x108/0x1b0 blk_mq_request_issue_directly+0xbd/0xe0 blk_mq_try_issue_list_directly+0x41/0xb0 blk_mq_sched_insert_requests+0xa2/0xe0 blk_mq_flush_plug_list+0x205/0x2a0 blk_flush_plug_list+0xc3/0xf0 [1] blk_finish_plug+0x21/0x2e _xfs_buf_ioapply+0x313/0x460 __xfs_buf_submit+0x67/0x220 xfs_buf_read_map+0x113/0x1a0 xfs_trans_read_buf_map+0xbf/0x330 xfs_btree_read_buf_block.constprop.42+0x95/0xd0 xfs_btree_lookup_get_block+0x95/0x170 xfs_btree_lookup+0xcc/0x470 xfs_bmap_del_extent_real+0x254/0x9a0 __xfs_bunmapi+0x45c/0xab0 xfs_bunmapi+0x15/0x30 xfs_itruncate_extents_flags+0xca/0x250 xfs_free_eofblocks+0x181/0x1e0 xfs_fs_destroy_inode+0xa8/0x1b0 destroy_inode+0x38/0x70 dispose_list+0x35/0x50 prune_icache_sb+0x52/0x70 super_cache_scan+0x120/0x1a0 do_shrink_slab+0x120/0x290 shrink_slab+0x216/0x2b0 shrink_node+0x1b6/0x4a0 do_try_to_free_pages+0xc6/0x370 try_to_free_mem_cgroup_pages+0xe3/0x1e0 try_charge+0x29e/0x790 mem_cgroup_charge_skmem+0x6a/0x100 __sk_mem_raise_allocated+0x18e/0x390 __sk_mem_schedule+0x2a/0x40 [0] tcp_sendmsg_locked+0x8eb/0xe10 tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 ___sys_sendmsg+0x26d/0x2b0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x42/0x100 entry_SYSCALL_64_after_hwframe+0x44/0xa9In [0],
tcp_send_msg_locked()was usingcurrent->page_fragwhen it calledsk_wmem_schedule(). It already calculated how many bytes can be fit intocurrent->page_frag. Due to memory pressure,sk_wmem_schedule()called into memory reclaim path which called into xfs and then IO issue path. Because the filesystem in question is backed by nbd, the control goes back into the tcp layer - back intotcp_sendmsg_locked().nbd sets
sk_allocationto (GFP_NOIO|__GFP_MEMALLOC) which makes sense - it's in the process of freeing memory and wants to be able to, e.g., drop clean pages to make forward progress. However, this confusedsk_page_frag()called from [2]. Because it only tests whether the allocation allows blocking which it does, it now thinkscurrent->page_fragcan be used again although it already was being used in [0].After [2] used
current->page_frag, the offset would be increased by the used amount. When the control returns to [0], current->page_frag's offset is increased and the previously calculated number of bytes now may overrun the end of allocated memory leading to silent memory corruptions.Fix it by adding
gfpflags_normal_context()which tests sleepable && !reclaim and use it to determine whether to usecurrent->task_frag.v2: Eric didn't like gfp flags being tested twice. Introduce a new
helpergfpflags_normal_context()and combine the two tests.Signed-off-by Tejun Heo <[email protected]>Cc Josef Bacik <[email protected]>Cc Eric Dumazet <[email protected]>--- include/linux/gfp.h | 23 +++++++++++++++++++++++ include/net/sock.h | 11 ++++++++--- 2 files changed, 31 insertions(+), 3 deletions(-)include/linux/gfp.h +11
-0@@ -325,6 +325,29 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags)return !!(gfp_flags & __GFP_DIRECT_RECLAIM); } +/** +* gfpflags_normal_context - is gfp_flags a normal sleepable context? +* @gfp_flags: gfp_flags to test +* +* Test whether @gfp_flags indicates that the allocation is from the +* %current context and allowed to sleep. +* +* An allocation being allowed to block doesn't mean it owns the %current +* context. When direct reclaim path tries to allocate memory, the +* allocation context is nested inside whatever %current was doing at the +* time of the original allocation. The nested allocation may be allowed include/linux/gfp.h static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) · line 348
Quoted 10 lines
include/linux/gfp.h +10
-0+* to block but modifying anything %current owns can corrupt the outer +* context's expectations. +* +* %true result from this function indicates that the allocation context +* can sleep and use anything that's associated with %current. +*/ +static inline bool gfpflags_normal_context (const gfp_t gfp_flags) +{ +return (gfp_flags & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC)) == + __GFP_DIRECT_RECLAIM;I think we should be checking
PF_MEMALLOChere instead. Something like:Quoted 6 lines
return
gfpflags_allow_blocking(gfp_flags)&& !(current->flags &PF_MEMALLOC);In my limited understanding,
__GFP_MEMALLOCgives access to reserve but we have overloadedPF_MEMALLOCto also define the reclaim context. There arePF_MEMALLOCusers which does not use__GFP_MEMALLOClikeiscsi_sw_tcp_pdu_xmit()which can callsock_sendmsg().
Quoted 6 lines
Why would this layer not set
sk->sk_allocationtoGFP_ATOMIC?And it also might call
sk_set_memalloc()too.Please double check scsi layer, I am pretty sure it did well at some point.
Yes, you are right, quoted the wrong example. SCSI is indeed setting sk->sk_allocation to GFP_ATOMIC and sk_set_memalloc() in iscsi_sw_tcp_conn_bind().
Basically what I wanted to say that MM treats PF_MEMALLOC as the reclaim context while __GFP_MEMALLOC just tells to give access to the reserves. As gfpflags_allow_blocking() can be used beyond net subsystem, my only concern is its potential usage under PF_MEMALLOC context but without __GFP_MEMALLOC.
Hello,
Quoted 5 lines · Shakeel Butt
Basically what I wanted to say that MM treats
PF_MEMALLOCas the reclaim context while__GFP_MEMALLOCjust tells to give access to the reserves. Asgfpflags_allow_blocking()can be used beyond net subsystem, my only concern is its potential usage underPF_MEMALLOCcontext but without__GFP_MEMALLOC.
Yeah, PF_MEMALLOC is likely the better condition to check here as we primarily want to know whether %current might be recursing and that should be indicated reliably with PF_MEMALLOC. Wanna prep a patch for it?
Thanks.
Signature · 1 line
tejun
Quoted 10 lines · Tejun Heo
Hello,
Quoted 5 lines · Shakeel Butt
Basically what I wanted to say that MM treats
PF_MEMALLOCas the reclaim context while__GFP_MEMALLOCjust tells to give access to the reserves. Asgfpflags_allow_blocking()can be used beyond net subsystem, my only concern is its potential usage underPF_MEMALLOCcontext but without__GFP_MEMALLOC.
Quoted 4 lines
Yeah,
PF_MEMALLOCis likely the better condition to check here as we primarily want to know whether %current might be recursing and that should be indicated reliably withPF_MEMALLOC. Wanna prep a patch for it?
Sure, I will keep your commit message and authorship (if you are ok with it).
Quoted 5 lines
Thanks.
--
tejun
Quoted 6 lines · Shakeel Butt
Quoted 4 lines
Yeah,
PF_MEMALLOCis likely the better condition to check here as we primarily want to know whether %current might be recursing and that should be indicated reliably withPF_MEMALLOC. Wanna prep a patch for it?Sure, I will keep your commit message and authorship (if you are ok with it).
I think the patch already got merged, so it may be easier to do an incremental one.
Thanks.
Signature · 1 line
tejun
Quoted 11 lines · Tejun Heo
Quoted 6 lines · Shakeel Butt
Quoted 4 lines
Yeah,
PF_MEMALLOCis likely the better condition to check here as we primarily want to know whether %current might be recursing and that should be indicated reliably withPF_MEMALLOC. Wanna prep a patch for it?Sure, I will keep your commit message and authorship (if you are ok with it).
I think the patch already got merged, so it may be easier to do an incremental one.
Oh ok, I will send the incremental one once I have this patch in the mm tree.
Quoted 5 lines
Thanks.
--
tejun
Quoted 1 line · Shakeel Butt
Oh ok, I will send the incremental one once I have this patch in the mm tree.
It's in the networking tree already.
Thanks.
Signature · 1 line
tejun
Andrew Morton
maintainerQuoted 62 lines · On Thu, 31 Oct 2019 10:35:21 -0700 Shakeel Butt <[email protected]> wrote:
+Michal Hocko
Quoted 59 lines · Tejun Heo
sk_page_frag()optimizesskb_fragallocations by using per-taskskb_fragcache when it knows it's the only user. The condition is determined by seeing whether the socket allocation mask allows blocking - if the allocation may block, it obviously owns the task's context and ergo exclusively ownscurrent->task_frag.Unfortunately, this misses recursion through memory reclaim path. Please take a look at the following backtrace.
[2] RIP: 0010:tcp_sendmsg_locked+0xccf/0xe10
... tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 sock_xmit.isra.24+0xa1/0x170 [nbd] nbd_send_cmd+0x1d2/0x690 [nbd] nbd_queue_rq+0x1b5/0x3b0 [nbd] __blk_mq_try_issue_directly+0x108/0x1b0 blk_mq_request_issue_directly+0xbd/0xe0 blk_mq_try_issue_list_directly+0x41/0xb0 blk_mq_sched_insert_requests+0xa2/0xe0 blk_mq_flush_plug_list+0x205/0x2a0 blk_flush_plug_list+0xc3/0xf0 [1] blk_finish_plug+0x21/0x2e _xfs_buf_ioapply+0x313/0x460 __xfs_buf_submit+0x67/0x220 xfs_buf_read_map+0x113/0x1a0 xfs_trans_read_buf_map+0xbf/0x330 xfs_btree_read_buf_block.constprop.42+0x95/0xd0 xfs_btree_lookup_get_block+0x95/0x170 xfs_btree_lookup+0xcc/0x470 xfs_bmap_del_extent_real+0x254/0x9a0 __xfs_bunmapi+0x45c/0xab0 xfs_bunmapi+0x15/0x30 xfs_itruncate_extents_flags+0xca/0x250 xfs_free_eofblocks+0x181/0x1e0 xfs_fs_destroy_inode+0xa8/0x1b0 destroy_inode+0x38/0x70 dispose_list+0x35/0x50 prune_icache_sb+0x52/0x70 super_cache_scan+0x120/0x1a0 do_shrink_slab+0x120/0x290 shrink_slab+0x216/0x2b0 shrink_node+0x1b6/0x4a0 do_try_to_free_pages+0xc6/0x370 try_to_free_mem_cgroup_pages+0xe3/0x1e0 try_charge+0x29e/0x790 mem_cgroup_charge_skmem+0x6a/0x100 __sk_mem_raise_allocated+0x18e/0x390 __sk_mem_schedule+0x2a/0x40 [0] tcp_sendmsg_locked+0x8eb/0xe10 tcp_sendmsg+0x27/0x40 sock_sendmsg+0x30/0x40 ___sys_sendmsg+0x26d/0x2b0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x42/0x100 entry_SYSCALL_64_after_hwframe+0x44/0xa9In [0],
tcp_send_msg_locked()was usingcurrent->page_fragwhen it
"tcp_sendmsg_locked" and "current->task_frag". Stuff like this makes review harder :(
Quoted 21 lines
Quoted 21 lines
called
sk_wmem_schedule(). It already calculated how many bytes can be fit intocurrent->page_frag. Due to memory pressure,sk_wmem_schedule()called into memory reclaim path which called into xfs and then IO issue path. Because the filesystem in question is backed by nbd, the control goes back into the tcp layer - back intotcp_sendmsg_locked().nbd sets
sk_allocationto (GFP_NOIO|__GFP_MEMALLOC) which makes sense - it's in the process of freeing memory and wants to be able to, e.g., drop clean pages to make forward progress. However, this confusedsk_page_frag()called from [2]. Because it only tests whether the allocation allows blocking which it does, it now thinkscurrent->page_fragcan be used again although it already was being used in [0].After [2] used
current->page_frag, the offset would be increased by the used amount. When the control returns to [0], current->page_frag's offset is increased and the previously calculated number of bytes now may overrun the end of allocated memory leading to silent memory corruptions.
Quoted 3 lines
Quoted 3 lines
Fix it by adding
gfpflags_normal_context()which tests sleepable && !reclaim and use it to determine whether to usecurrent->task_frag.
Dumb-but-obvious question. Rather than putzing with allocation modes, is it not feasible to change the net layer to copy the current value of current->task_frag into a local then restore its value when it has finished being used?
Hello,
Quoted 4 lines · Andrew Morton
Quoted 1 line
Quoted 1 line
In [0],
tcp_send_msg_locked()was usingcurrent->page_fragwhen it"
tcp_sendmsg_locked" and "current->task_frag". Stuff like this makes review harder :(
lol, sorry about that.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Fix it by adding
gfpflags_normal_context()which tests sleepable && !reclaim and use it to determine whether to usecurrent->task_frag.Dumb-but-obvious question. Rather than putzing with allocation modes, is it not feasible to change the net layer to copy the current value of
current->task_fraginto a local then restore its value when it has finished being used?
It's being used as an allocation cache, so doing so would lead to the same area getting allocated for two packets at the same time instead overrunning the end of the cache.
Thanks.
Signature · 1 line
tejun