fuse: fix readahead reclaim deadlock
Subsystems (3)
PATCH 1/1 fuse: fix readahead reclaim deadlock
A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:
Quoted 1 line
Quoted 1 line
Quoted 1 line
stack_trace(1504735)
folio_wait_bit_common (mm/filemap.c:1308:4) folio_lock (./include/linux/pagemap.h:1052:3) truncate_inode_pages_range (mm/truncate.c:336:10) fuse_evict_inode (fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3) dentry_unlink_inode (fs/dcache.c:412:3) __dentry_kill (fs/dcache.c:615:3) shrink_kill (fs/dcache.c:1060:12) shrink_dentry_list (fs/dcache.c:1087:3) prune_dcache_sb (fs/dcache.c:1168:2) super_cache_scan (fs/super.c:221:10) do_shrink_slab (mm/shrinker.c:435:9) shrink_slab (mm/shrinker.c:626:10) shrink_node (mm/vmscan.c:5951:2) shrink_zones (mm/vmscan.c:6195:3) do_try_to_free_pages (mm/vmscan.c:6257:3) do_swap_page (mm/memory.c:4136:11) handle_pte_fault (mm/memory.c:5562:10) handle_mm_fault (mm/memory.c:5870:9) do_user_addr_fault (arch/x86/mm/fault.c:1338:10) handle_page_fault (arch/x86/mm/fault.c:1481:3) exc_page_fault (arch/x86/mm/fault.c:1539:2) asm_exc_page_fault+0x22/0x27
During readahead, the folio is locked. When fuse_evict_inode() is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.
The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.
This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.
--- fs/fuse/file.c | 7 +++++++ 1 file changed, 7 insertions(+)
fs/fuse/file.c +7 -0
@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,if (ia->ff)fuse_file_put (ia->ff, false); +iput (inode);fuse_io_free (ia); }
@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac) ia =fuse_io_alloc (NULL , cur_pages);if (!ia)break ; +/* +* Acquire the inode ref here to prevent reclaim from +* deadlocking. The ref gets dropped in fuse_readpages_end(). +*/ +igrab (inode); + ap = &ia->ap;while (pages < cur_pages) {
Signature · 1 line
2.47.3
Quoted 58 lines · Joanne Koong
A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:
Quoted 1 line
Quoted 1 line
Quoted 1 line
stack_trace(1504735)
folio_wait_bit_common(mm/filemap.c:1308:4)folio_lock(./include/linux/pagemap.h:1052:3)truncate_inode_pages_range(mm/truncate.c:336:10)fuse_evict_inode(fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3)dentry_unlink_inode(fs/dcache.c:412:3)__dentry_kill(fs/dcache.c:615:3)shrink_kill(fs/dcache.c:1060:12)shrink_dentry_list(fs/dcache.c:1087:3)prune_dcache_sb(fs/dcache.c:1168:2)super_cache_scan(fs/super.c:221:10)do_shrink_slab(mm/shrinker.c:435:9)shrink_slab(mm/shrinker.c:626:10)shrink_node(mm/vmscan.c:5951:2)shrink_zones(mm/vmscan.c:6195:3)do_try_to_free_pages(mm/vmscan.c:6257:3)do_swap_page(mm/memory.c:4136:11)handle_pte_fault(mm/memory.c:5562:10)handle_mm_fault(mm/memory.c:5870:9)do_user_addr_fault(arch/x86/mm/fault.c:1338:10)handle_page_fault(arch/x86/mm/fault.c:1481:3)exc_page_fault(arch/x86/mm/fault.c:1539:2)asm_exc_page_fault+0x22/0x27During readahead, the folio is locked. When
fuse_evict_inode()is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.
This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.
Signed-off-by Joanne Koong <[email protected]>Reported-by Omar Sandoval <[email protected]>--- fs/fuse/file.c | 7 +++++++ 1 file changed, 7 insertions(+)fs/fuse/file.c +0
-0
fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 896
Quoted 4 lines
fs/fuse/file.c +1
-0@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,if (ia->ff)fuse_file_put (ia->ff, false); +iput (inode);
It's somewhat odd to use igrab and iput in the read(ahead) context.
I wonder for FUSE, if it's possible to just wait ongoing locked folios when i_count == 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.
Thanks,
Gao Xiang
Quoted 16 lines
fs/fuse/file.c +6
-0fuse_io_free (ia); }@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac) ia =fuse_io_alloc (NULL , cur_pages);if (!ia)break ; +/* +* Acquire the inode ref here to prevent reclaim from +* deadlocking. The ref gets dropped in fuse_readpages_end(). +*/ +igrab (inode); + ap = &ia->ap;while (pages < cur_pages) {
Quoted 68 lines · Gao Xiang
Quoted 58 lines · Joanne Koong
A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:
Quoted 1 line
Quoted 1 line
Quoted 1 line
stack_trace(1504735)
folio_wait_bit_common(mm/filemap.c:1308:4)folio_lock(./include/linux/pagemap.h:1052:3)truncate_inode_pages_range(mm/truncate.c:336:10)fuse_evict_inode(fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3)dentry_unlink_inode(fs/dcache.c:412:3)__dentry_kill(fs/dcache.c:615:3)shrink_kill(fs/dcache.c:1060:12)shrink_dentry_list(fs/dcache.c:1087:3)prune_dcache_sb(fs/dcache.c:1168:2)super_cache_scan(fs/super.c:221:10)do_shrink_slab(mm/shrinker.c:435:9)shrink_slab(mm/shrinker.c:626:10)shrink_node(mm/vmscan.c:5951:2)shrink_zones(mm/vmscan.c:6195:3)do_try_to_free_pages(mm/vmscan.c:6257:3)do_swap_page(mm/memory.c:4136:11)handle_pte_fault(mm/memory.c:5562:10)handle_mm_fault(mm/memory.c:5870:9)do_user_addr_fault(arch/x86/mm/fault.c:1338:10)handle_page_fault(arch/x86/mm/fault.c:1481:3)exc_page_fault(arch/x86/mm/fault.c:1539:2)asm_exc_page_fault+0x22/0x27During readahead, the folio is locked. When
fuse_evict_inode()is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.
This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.
Signed-off-by Joanne Koong <[email protected]>Reported-by Omar Sandoval <[email protected]>---
fs/fuse/file.c| 7 +++++++1 file changed, 7 insertions(+)fs/fuse/file.c +0
-0fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 895
Quoted 3 lines
fs/fuse/file.c +1
-0@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, if (ia->ff) fuse_file_put(ia->ff, false); + iput(inode);It's somewhat odd to use
igrabandiputin the read(ahead) context.
Quoted 5 lines
I wonder for FUSE, if it's possible to just wait ongoing locked folios when
i_count== 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.
Oh, it was in the dentry LRU list instead, I don't think it can work.
Or normally the kernel filesystem uses GFP_NOFS to avoid such deadlock (see if (!(sc->gfp_mask & __GFP_FS)) in super_cache_scan()), I wonder if the daemon should simply use prctl(PR_SET_IO_FLUSHER) so that the user daemon won't be called into the fs reclaim context again.
Thanks,
Gao Xiang
Quoted 20 lines
Thanks,
Gao XiangQuoted 14 lines
fs/fuse/file.c +6
-0fuse_io_free(ia); }@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac) ia = fuse_io_alloc(NULL, cur_pages); if (!ia) break; + /* + * Acquire the inode ref here to prevent reclaim from + * deadlocking. The ref gets dropped in fuse_readpages_end(). + */ + igrab(inode); + ap = &ia->ap; while (pages < cur_pages) {
Quoted 75 lines · Gao Xiang
Quoted 67 lines · Gao Xiang
Quoted 58 lines · Joanne Koong
A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:
Quoted 1 line
Quoted 1 line
Quoted 1 line
stack_trace(1504735)
folio_wait_bit_common(mm/filemap.c:1308:4)folio_lock(./include/linux/pagemap.h:1052:3)truncate_inode_pages_range(mm/truncate.c:336:10)fuse_evict_inode(fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3)dentry_unlink_inode(fs/dcache.c:412:3)__dentry_kill(fs/dcache.c:615:3)shrink_kill(fs/dcache.c:1060:12)shrink_dentry_list(fs/dcache.c:1087:3)prune_dcache_sb(fs/dcache.c:1168:2)super_cache_scan(fs/super.c:221:10)do_shrink_slab(mm/shrinker.c:435:9)shrink_slab(mm/shrinker.c:626:10)shrink_node(mm/vmscan.c:5951:2)shrink_zones(mm/vmscan.c:6195:3)do_try_to_free_pages(mm/vmscan.c:6257:3)do_swap_page(mm/memory.c:4136:11)handle_pte_fault(mm/memory.c:5562:10)handle_mm_fault(mm/memory.c:5870:9)do_user_addr_fault(arch/x86/mm/fault.c:1338:10)handle_page_fault(arch/x86/mm/fault.c:1481:3)exc_page_fault(arch/x86/mm/fault.c:1539:2)asm_exc_page_fault+0x22/0x27During readahead, the folio is locked. When
fuse_evict_inode()is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.
This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.
Signed-off-by Joanne Koong <[email protected]>Reported-by Omar Sandoval <[email protected]>--- fs/fuse/file.c | 7 +++++++ 1 file changed, 7 insertions(+)fs/fuse/file.c +0
-0fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 895
Quoted 3 lines
fs/fuse/file.c +1
-0@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,if (ia->ff)fuse_file_put (ia->ff, false); +iput (inode);It's somewhat odd to use
igrabandiputin the read(ahead) context.Quoted 5 lines
I wonder for FUSE, if it's possible to just wait ongoing locked folios when
i_count== 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.
Quoted 8 lines
Oh, it was in the dentry LRU list instead, I don't think it can work.
Or normally the kernel filesystem uses
GFP_NOFSto avoid such deadlock (seeif (!(sc->gfp_mask & __GFP_FS))insuper_cache_scan()), I wonder if the daemon should simply useprctl(PR_SET_IO_FLUSHER)so that the user daemon won't be called into the fs reclaim context again.
Hi Gao,
We cannot rely on the daemon to set this unfortunately. This can tie up reclaim and kswapd for the entire system so I think this enforcement needs to be guaranteed and at the kernel level. For example, there is the possibility of malicious servers, which we cannot rely on to set FR_SET_IO_FLUSHER.
Thanks,
Joanne
Quoted 25 lines
Thanks,
Gao XiangQuoted 20 lines
Thanks,
Gao XiangQuoted 14 lines
fs/fuse/file.c +6
-0fuse_io_free (ia); }@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac) ia =fuse_io_alloc (NULL , cur_pages);if (!ia)break ; +/* +* Acquire the inode ref here to prevent reclaim from +* deadlocking. The ref gets dropped in fuse_readpages_end(). +*/ +igrab (inode); + ap = &ia->ap;while (pages < cur_pages) {
Quoted 85 lines · Joanne Koong
Quoted 75 lines · Gao Xiang
Quoted 67 lines · Gao Xiang
Quoted 58 lines · Joanne Koong
A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:
Quoted 1 line
Quoted 1 line
Quoted 1 line
stack_trace(1504735)
folio_wait_bit_common(mm/filemap.c:1308:4)folio_lock(./include/linux/pagemap.h:1052:3)truncate_inode_pages_range(mm/truncate.c:336:10)fuse_evict_inode(fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3)dentry_unlink_inode(fs/dcache.c:412:3)__dentry_kill(fs/dcache.c:615:3)shrink_kill(fs/dcache.c:1060:12)shrink_dentry_list(fs/dcache.c:1087:3)prune_dcache_sb(fs/dcache.c:1168:2)super_cache_scan(fs/super.c:221:10)do_shrink_slab(mm/shrinker.c:435:9)shrink_slab(mm/shrinker.c:626:10)shrink_node(mm/vmscan.c:5951:2)shrink_zones(mm/vmscan.c:6195:3)do_try_to_free_pages(mm/vmscan.c:6257:3)do_swap_page(mm/memory.c:4136:11)handle_pte_fault(mm/memory.c:5562:10)handle_mm_fault(mm/memory.c:5870:9)do_user_addr_fault(arch/x86/mm/fault.c:1338:10)handle_page_fault(arch/x86/mm/fault.c:1481:3)exc_page_fault(arch/x86/mm/fault.c:1539:2)asm_exc_page_fault+0x22/0x27During readahead, the folio is locked. When
fuse_evict_inode()is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.
This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.
Signed-off-by Joanne Koong <[email protected]>Reported-by Omar Sandoval <[email protected]>--- fs/fuse/file.c | 7 +++++++ 1 file changed, 7 insertions(+)fs/fuse/file.c +0
-0fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 895
Quoted 3 lines
fs/fuse/file.c +1
-0@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,if (ia->ff)fuse_file_put (ia->ff, false); +iput (inode);It's somewhat odd to use
igrabandiputin the read(ahead) context.Quoted 5 lines
I wonder for FUSE, if it's possible to just wait ongoing locked folios when
i_count== 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.Quoted 8 lines
Oh, it was in the dentry LRU list instead, I don't think it can work.
Or normally the kernel filesystem uses
GFP_NOFSto avoid such deadlock (seeif (!(sc->gfp_mask & __GFP_FS))insuper_cache_scan()), I wonder if the daemon should simply useprctl(PR_SET_IO_FLUSHER)so that the user daemon won't be called into the fs reclaim context again.
Quoted 7 lines
Hi Gao,
We cannot rely on the daemon to set this unfortunately. This can tie up reclaim and kswapd for the entire system so I think this enforcement needs to be guaranteed and at the kernel level. For example, there is the possibility of malicious servers, which we cannot rely on to set
FR_SET_IO_FLUSHER.
Hi Joanne,
Yes, currently I don't have a saner way in my mind but iput() in such nested context sounds a new entry (e.g. I thought kernel page fault path should have nothing tangled with evict() directly but I may be wrong.)
In principle, typical the kernel filesystem holds a valid file during the entire buffered read (file)/mmap (vma->vm_file) submission path (and of course they won't upcall to userspace and then do random behavior in the userspace for I/O processing).
So for the kernel filesystems I think the GFP_NOFS allocation isn't needed since file.f_path always takes a valid dentry ref during the submission so that such dentry/inode reclaim above is impossible IMO.)
Thanks,
Gao Xiang
Quoted 4 lines
Thanks,
Joanne
Quoted 96 lines · Gao Xiang
Quoted 85 lines · Joanne Koong
Quoted 75 lines · Gao Xiang
Quoted 67 lines · Gao Xiang
Quoted 58 lines · Joanne Koong
A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:
Quoted 1 line
Quoted 1 line
Quoted 1 line
stack_trace(1504735)
folio_wait_bit_common(mm/filemap.c:1308:4)folio_lock(./include/linux/pagemap.h:1052:3)truncate_inode_pages_range(mm/truncate.c:336:10)fuse_evict_inode(fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3)dentry_unlink_inode(fs/dcache.c:412:3)__dentry_kill(fs/dcache.c:615:3)shrink_kill(fs/dcache.c:1060:12)shrink_dentry_list(fs/dcache.c:1087:3)prune_dcache_sb(fs/dcache.c:1168:2)super_cache_scan(fs/super.c:221:10)do_shrink_slab(mm/shrinker.c:435:9)shrink_slab(mm/shrinker.c:626:10)shrink_node(mm/vmscan.c:5951:2)shrink_zones(mm/vmscan.c:6195:3)do_try_to_free_pages(mm/vmscan.c:6257:3)do_swap_page(mm/memory.c:4136:11)handle_pte_fault(mm/memory.c:5562:10)handle_mm_fault(mm/memory.c:5870:9)do_user_addr_fault(arch/x86/mm/fault.c:1338:10)handle_page_fault(arch/x86/mm/fault.c:1481:3)exc_page_fault(arch/x86/mm/fault.c:1539:2)asm_exc_page_fault+0x22/0x27During readahead, the folio is locked. When
fuse_evict_inode()is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.
This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.
Signed-off-by Joanne Koong <[email protected]>Reported-by Omar Sandoval <[email protected]>---
fs/fuse/file.c| 7 +++++++1 file changed, 7 insertions(+)fs/fuse/file.c +0
-0fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 895
Quoted 3 lines
fs/fuse/file.c +1
-0@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, if (ia->ff) fuse_file_put(ia->ff, false); + iput(inode);It's somewhat odd to use
igrabandiputin the read(ahead) context.Quoted 5 lines
I wonder for FUSE, if it's possible to just wait ongoing locked folios when
i_count== 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.Quoted 8 lines
Oh, it was in the dentry LRU list instead, I don't think it can work.
Or normally the kernel filesystem uses
GFP_NOFSto avoid such deadlock (seeif (!(sc->gfp_mask & __GFP_FS))insuper_cache_scan()), I wonder if the daemon should simply useprctl(PR_SET_IO_FLUSHER)so that the user daemon won't be called into the fs reclaim context again.Quoted 7 lines
Hi Gao,
We cannot rely on the daemon to set this unfortunately. This can tie up reclaim and kswapd for the entire system so I think this enforcement needs to be guaranteed and at the kernel level. For example, there is the possibility of malicious servers, which we cannot rely on to set
FR_SET_IO_FLUSHER.
Quoted 5 lines
Hi Joanne,
Yes, currently I don't have a saner way in my mind but
iput()in such nested context sounds a new entry (e.g. I thought kernel page fault path should have nothing tangled with
To clarify:
^ kernel file read page fault path tangled with this particular
inode in progress (I doesn't mean random inode reclaimation).
Quoted 19 lines
evict()directly but I may be wrong.)In principle, typical the kernel filesystem holds a valid
fileduring the entire buffered read (file)/mmap (vma->vm_file) submission path (and of course they won't upcall to userspace and then do random behavior in the userspace for I/O processing).So for the kernel filesystems I think the
GFP_NOFSallocation isn't needed sincefile.f_pathalways takes a valid dentry ref during the submission so that such dentry/inode reclaim above is impossible IMO.)Thanks,
Gao XiangQuoted 4 lines
Thanks,
Joanne
Miklos Szeredi
maintainerQuoted 2 lines · Gao Xiang
In principle, typical the kernel filesystem holds a valid
fileduring the entire buffered read (file)/mmap (vma->vm_file)
Actually, fuse does hold a ref to fuse_file, which should make sure that the inode is not released while the readahead is ongoing.
See igrab() in fuse_prepare_release() and iput() in fuse_release_end().
So I don't understand what's going on.
Joanne, do you have a reproducer?
Thanks,
Miklos
Quoted 10 lines · Miklos Szeredi
Quoted 2 lines · Gao Xiang
In principle, typical the kernel filesystem holds a valid
fileduring the entire buffered read (file)/mmap (vma->vm_file)Actually, fuse does hold a ref to
fuse_file, which should make sure that the inode is not released while the readahead is ongoing.See
igrab()infuse_prepare_release()andiput()infuse_release_end().
If the file is mmaped, couldn't the release happen before the page fault?
This is the chain of events I'm thinking of:
file is opened, file is mmapped, file is closed (which triggers the igrab() and iput() you mentioned in fuse_prepare_release()/fuse_release_end()), then client tries to read from the mmapped address which triggers a page fault which triggers readahead.
Or am I missing something here? I'm not super familiar with the mmap code but as far as I can tell from the logic in vm_mmap_pgoff() -> do_mmap() -> __mmap_region(), it doesn't grab a refcount on the inode or the file descriptor.
Quoted 4 lines
So I don't understand what's going on.
Joanne, do you have a reproducer?
I don't have a reproducer but we saw the stack trace in the commit message on our servers a couple of times and used drgn to pinpoint that there was a readahead request submitted.
Thanks,
Joanne
Quoted 3 lines
Thanks,
Miklos
Miklos Szeredi
maintainerQuoted 4 lines · Joanne Koong
Or am I missing something here? I'm not super familiar with the mmap code but as far as I can tell from the logic in
vm_mmap_pgoff()->do_mmap()->__mmap_region(), it doesn't grab a refcount on the inode or the file descriptor.
mmap keeps a ref on the file (vma->vm_file), so only munmap() will trigger ->release().
Thanks,
Miklos
Quoted 10 lines · Miklos Szeredi
Quoted 4 lines · Joanne Koong
Or am I missing something here? I'm not super familiar with the mmap code but as far as I can tell from the logic in
vm_mmap_pgoff()->do_mmap()->__mmap_region(), it doesn't grab a refcount on the inode or the file descriptor.mmap keeps a ref on the file (
vma->vm_file), so onlymunmap()will trigger ->release().
Ah okay I see, I thought the .release callback is for when the last fd for a file gets closed but clearly that's not how it works. Thanks for clarifying that.
I'm confused too then how we can have a fuse inode with no refcount on it but with one of its folios indefinitely locked. The paths where a folio can be indefinitely locked afaict are from reads/readahead, buffered writes, and notify retrieves. But each of those paths hold a reference to the inode, so I'm confused how we end up in this situation.
Thanks,
Joanne
Quoted 3 lines
Thanks,
Miklos
Quoted 6 lines · Miklos Szeredi
Quoted 2 lines · Gao Xiang
In principle, typical the kernel filesystem holds a valid
fileduring the entire buffered read (file)/mmap (vma->vm_file)
Quoted 8 lines
Actually, fuse does hold a ref to
fuse_file, which should make sure that the inode is not released while the readahead is ongoing.See
igrab()infuse_prepare_release()andiput()infuse_release_end().So I don't understand what's going on.
Joanne, do you have a reproducer?
Omar figured out that the servers where we ran into this had fc->no_open set.
The igrab() in fuse_prepare_release() you mentioned above happens only when fc->no_open is not set. commit e26ee4efbc79 "fuse: allocate ff->release_args only if release is needed" is what changed this behavior.
If we revert this commit, I think this fixes the issue. Unless you prefer another solution?
Thanks,
Joanne
Quoted 3 lines
Thanks,
Miklos
Miklos Szeredi
maintainerQuoted 42 lines · Joanne Koong
A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:
Quoted 1 line
Quoted 1 line
Quoted 1 line
stack_trace(1504735)
folio_wait_bit_common(mm/filemap.c:1308:4)folio_lock(./include/linux/pagemap.h:1052:3)truncate_inode_pages_range(mm/truncate.c:336:10)fuse_evict_inode(fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3)dentry_unlink_inode(fs/dcache.c:412:3)__dentry_kill(fs/dcache.c:615:3)shrink_kill(fs/dcache.c:1060:12)shrink_dentry_list(fs/dcache.c:1087:3)prune_dcache_sb(fs/dcache.c:1168:2)super_cache_scan(fs/super.c:221:10)do_shrink_slab(mm/shrinker.c:435:9)shrink_slab(mm/shrinker.c:626:10)shrink_node(mm/vmscan.c:5951:2)shrink_zones(mm/vmscan.c:6195:3)do_try_to_free_pages(mm/vmscan.c:6257:3)do_swap_page(mm/memory.c:4136:11)handle_pte_fault(mm/memory.c:5562:10)handle_mm_fault(mm/memory.c:5870:9)do_user_addr_fault(arch/x86/mm/fault.c:1338:10)handle_page_fault(arch/x86/mm/fault.c:1481:3)exc_page_fault(arch/x86/mm/fault.c:1539:2)asm_exc_page_fault+0x22/0x27During readahead, the folio is locked. When
fuse_evict_inode()is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.
Quoted 5 lines
This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.
I don't see a better way to fix this, but Cc-ing Willy as the readahead/mm expert.
Quoted 2 lines
Signed-off-by Joanne Koong <[email protected]>Reported-by Omar Sandoval <[email protected]>
This is not a new bug, right? So at least add a
Thanks,
Miklos
Quoted 32 lines
--- fs/fuse/file.c | 7 +++++++ 1 file changed, 7 insertions(+)fs/fuse/file.c +7
-1@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,if (ia->ff)fuse_file_put (ia->ff, false); +iput (inode);fuse_io_free (ia); }@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac) ia =fuse_io_alloc (NULL , cur_pages);if (!ia)break ; +/* +* Acquire the inode ref here to prevent reclaim from +* deadlocking. The ref gets dropped in fuse_readpages_end(). +*/ +igrab (inode); + ap = &ia->ap;while (pages < cur_pages) { --2.47.3