fuse: Remove __GFP_FS flag to avoid allocator recursing
Subsystems (2)
PATCH 1/1 fuse: Remove __GFP_FS flag to avoid allocator recursing
Found a deadlock between kswapd, writeback thread and fuse process Here are the sequence of events with callstacks on the deadlock.
process#1 process#2 process#3__switch_to+0x150 __switch_to+0x150 try_to_free_pages__schedule+0x984 __schedule+0x984
memalloc_noreclaim_restoreschedule+0x70 schedule+0x70 __perform_reclaimbit_wait+0x14 __fuse_request_send+0x154
__alloc_pages_direct_reclaim__wait_on_bit+0x70 fuse_simple_request+0x174inode_wait_for_writeback+0xa0
__alloc_pages_slowpath fuse_flush_times+0x10c evict+0xa4 fuse_write_inode+0x5c __alloc_pages_nodemask iput+0x248 __writeback_single_inode+0x3d4 dentry_unlink_inode+0xd8 __alloc_pages_node writeback_sb_inodes+0x4a0 __dentry_kill+0x160 __writeback_inodes_wb+0xac shrink_dentry_list+0x170 alloc_pages_node wb_writeback+0x26c fuse_copy_fill
prune_dcache_sb+0x54 wb_workfn+0x2c0 fuse_copy_onesuper_cache_scan+0x114 process_one_work+0x278 fuse_read_single_forgetdo_shrink_slab+0x24c worker_thread+0x26c fuse_read_forgetshrink_slab+0xa8 kthread+0x118 fuse_dev_do_readshrink_node+0x118 fuse_dev_splice_readkswapd+0x92c do_splice_to
do_splice
Process#1(kswapd) held an inode lock and initaited a writeback to free the pages, as the inode superblock is fuse, process#2 forms a fuse request. Process#3 (Fuse daemon threads) while serving process#2 request, it requires memory(pages) and as the system is already running in low memory it ends up in calling try_to_ free_pages(), which might now call kswapd again, which is already stuck with an inode lock held. Thus forms a deadlock.
So, remove __GFP_FS flag to avoid allocator recursing into the filesystem that might already held locks.
--- fs/fuse/dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/fuse/dev.c +1 -1
@@ -708,7 +708,7 @@ static int fuse_copy_fill(struct fuse_copy_state *cs)if (cs->nr_segs >= cs->pipe->max_usage)return -EIO ; - page =alloc_page (GFP_HIGHUSER ); + page =alloc_page (GFP_NOFS | __GFP_HARDWALL | __GFP_HIGHMEM);if (!page)return -ENOMEM ;
Signature · 1 line
2.7.4
Quoted 10 lines · Pradeep P V K
Process#1(kswapd) held an inode lock and initaited a writeback to free the pages, as the inode superblock is fuse, process#2 forms a fuse request. Process#3 (Fuse daemon threads) while serving process#2 request, it requires memory(pages) and as the system is already running in low memory it ends up in calling
try_to_free_pages(), which might now call kswapd again, which is already stuck with an inode lock held. Thus forms a deadlock.So, remove
__GFP_FSflag to avoid allocator recursing into the filesystem that might already held locks.
This is the wrong way to fix the problem. The fuse daemon threads should have called memalloc_nofs_save() as this prevents them from inadvertently tripping over other places where they forgot to use GFP_NOFS (or have no way to pass a GFP_NOFS flags argument).
Quoted 9 lines · Matthew Wilcox
Quoted 8 lines · Pradeep P V K
Process#1(kswapd) held an inode lock and initaited a writeback to free the pages, as the inode superblock is fuse, process#2 forms a fuse request. Process#3 (Fuse daemon threads) while serving process#2 request, it requires memory(pages) and as the system is already running in low memory it ends up in calling
try_to_free_pages(), which might now call kswapd again, which is already stuck with an inode lock held. Thus forms a deadlock.
Quoted 7 lines
Quoted 2 lines
So, remove
__GFP_FSflag to avoid allocator recursing into the filesystem that might already held locks.This is the wrong way to fix the problem. The fuse daemon threads should have called
memalloc_nofs_save()as this prevents them from inadvertently tripping over other places where they forgot to useGFP_NOFS(or have no way to pass aGFP_NOFSflags argument).
Thanks Matthew for pointing this. I will address this in my next patch set.
Thanks and Regards,
Pradeep