Skip to contentSkip to conversation

fuse: fix readahead reclaim deadlock

PATCH Superseded Joanne Koong 3 participants 12 messages Sep 25, 2025 – Oct 7, 2025

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fuse: fix readahead reclaim deadlock

A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:

Quoted 1 line
Quoted 1 line
Quoted 1 line

stack_trace(1504735)

folio_wait_bit_common (mm/filemap.c:1308:4) folio_lock (./include/linux/pagemap.h:1052:3) truncate_inode_pages_range (mm/truncate.c:336:10) fuse_evict_inode (fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3) dentry_unlink_inode (fs/dcache.c:412:3) __dentry_kill (fs/dcache.c:615:3) shrink_kill (fs/dcache.c:1060:12) shrink_dentry_list (fs/dcache.c:1087:3) prune_dcache_sb (fs/dcache.c:1168:2) super_cache_scan (fs/super.c:221:10) do_shrink_slab (mm/shrinker.c:435:9) shrink_slab (mm/shrinker.c:626:10) shrink_node (mm/vmscan.c:5951:2) shrink_zones (mm/vmscan.c:6195:3) do_try_to_free_pages (mm/vmscan.c:6257:3) do_swap_page (mm/memory.c:4136:11) handle_pte_fault (mm/memory.c:5562:10) handle_mm_fault (mm/memory.c:5870:9) do_user_addr_fault (arch/x86/mm/fault.c:1338:10) handle_page_fault (arch/x86/mm/fault.c:1481:3) exc_page_fault (arch/x86/mm/fault.c:1539:2) asm_exc_page_fault+0x22/0x27

During readahead, the folio is locked. When fuse_evict_inode() is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.

The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.

This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.

---
 fs/fuse/file.c | 7 +++++++
 1 file changed, 7 insertions(+)
fs/fuse/file.c +7 -0
index f1ef77a0be05..8e759061b843 100644
@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,
 	if (ia->ff)
 		fuse_file_put(ia->ff, false);
 
+	iput(inode);
 	fuse_io_free(ia);
 }
 
@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac)
 		ia = fuse_io_alloc(NULL, cur_pages);
 		if (!ia)
 			break;
+		/*
+		 *  Acquire the inode ref here to prevent reclaim from
+		 *  deadlocking. The ref gets dropped in fuse_readpages_end().
+		 */
+		igrab(inode);
+
 		ap = &ia->ap;
 
 		while (pages < cur_pages) {
Signature · 1 line
2.47.3
#1 lore raw
Quoted 58 lines · Joanne Koong

A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:

Quoted 1 line
Quoted 1 line
Quoted 1 line

stack_trace(1504735)

folio_wait_bit_common (mm/filemap.c:1308:4) folio_lock (./include/linux/pagemap.h:1052:3) truncate_inode_pages_range (mm/truncate.c:336:10) fuse_evict_inode (fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3) dentry_unlink_inode (fs/dcache.c:412:3) __dentry_kill (fs/dcache.c:615:3) shrink_kill (fs/dcache.c:1060:12) shrink_dentry_list (fs/dcache.c:1087:3) prune_dcache_sb (fs/dcache.c:1168:2) super_cache_scan (fs/super.c:221:10) do_shrink_slab (mm/shrinker.c:435:9) shrink_slab (mm/shrinker.c:626:10) shrink_node (mm/vmscan.c:5951:2) shrink_zones (mm/vmscan.c:6195:3) do_try_to_free_pages (mm/vmscan.c:6257:3) do_swap_page (mm/memory.c:4136:11) handle_pte_fault (mm/memory.c:5562:10) handle_mm_fault (mm/memory.c:5870:9) do_user_addr_fault (arch/x86/mm/fault.c:1338:10) handle_page_fault (arch/x86/mm/fault.c:1481:3) exc_page_fault (arch/x86/mm/fault.c:1539:2) asm_exc_page_fault+0x22/0x27

During readahead, the folio is locked. When fuse_evict_inode() is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.

The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.

This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.

---
  fs/fuse/file.c | 7 +++++++
  1 file changed, 7 insertions(+)
fs/fuse/file.c +0 -0
index f1ef77a0be05..8e759061b843 100644

fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 896

Quoted 4 lines
fs/fuse/file.c +1 -0
@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,
  	if (ia->ff)
  		fuse_file_put(ia->ff, false);
 
+	iput(inode);

It's somewhat odd to use igrab and iput in the read(ahead) context.

I wonder for FUSE, if it's possible to just wait ongoing locked folios when i_count == 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.

Thanks,
Gao Xiang

Quoted 16 lines
fs/fuse/file.c +6 -0
  	fuse_io_free(ia);
  }
 
@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac)
  		ia = fuse_io_alloc(NULL, cur_pages);
  		if (!ia)
  			break;
+		/*
+		 *  Acquire the inode ref here to prevent reclaim from
+		 *  deadlocking. The ref gets dropped in fuse_readpages_end().
+		 */
+		igrab(inode);
+
  		ap = &ia->ap;
 
  		while (pages < cur_pages) {
#2 lore raw
Quoted 68 lines · Gao Xiang
Quoted 58 lines · Joanne Koong

A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:

Quoted 1 line
Quoted 1 line
Quoted 1 line

stack_trace(1504735)

  folio_wait_bit_common (mm/filemap.c:1308:4) folio_lock (./include/linux/pagemap.h:1052:3) truncate_inode_pages_range (mm/truncate.c:336:10) fuse_evict_inode (fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3) dentry_unlink_inode (fs/dcache.c:412:3) __dentry_kill (fs/dcache.c:615:3) shrink_kill (fs/dcache.c:1060:12) shrink_dentry_list (fs/dcache.c:1087:3) prune_dcache_sb (fs/dcache.c:1168:2) super_cache_scan (fs/super.c:221:10) do_shrink_slab (mm/shrinker.c:435:9) shrink_slab (mm/shrinker.c:626:10) shrink_node (mm/vmscan.c:5951:2) shrink_zones (mm/vmscan.c:6195:3) do_try_to_free_pages (mm/vmscan.c:6257:3) do_swap_page (mm/memory.c:4136:11) handle_pte_fault (mm/memory.c:5562:10) handle_mm_fault (mm/memory.c:5870:9) do_user_addr_fault (arch/x86/mm/fault.c:1338:10) handle_page_fault (arch/x86/mm/fault.c:1481:3) exc_page_fault (arch/x86/mm/fault.c:1539:2) asm_exc_page_fault+0x22/0x27

During readahead, the folio is locked. When fuse_evict_inode() is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.

The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.

This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.

---
  fs/fuse/file.c | 7 +++++++

  1 file changed, 7 insertions(+)
fs/fuse/file.c +0 -0
index f1ef77a0be05..8e759061b843 100644

fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 895

Quoted 3 lines
fs/fuse/file.c +1 -0
@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,
       if (ia->ff)
           fuse_file_put(ia->ff, false);
+    iput(inode);

It's somewhat odd to use igrab and iput in the read(ahead) context.

Quoted 5 lines

I wonder for FUSE, if it's possible to just wait ongoing locked folios when i_count == 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.

Oh, it was in the dentry LRU list instead, I don't think it can work.

Or normally the kernel filesystem uses GFP_NOFS to avoid such deadlock (see if (!(sc->gfp_mask & __GFP_FS)) in super_cache_scan()), I wonder if the daemon should simply use prctl(PR_SET_IO_FLUSHER) so that the user daemon won't be called into the fs reclaim context again.

Thanks,
Gao Xiang

Quoted 20 lines

Thanks,
Gao Xiang

Quoted 14 lines
fs/fuse/file.c +6 -0
       fuse_io_free(ia);
   }
@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac)
           ia = fuse_io_alloc(NULL, cur_pages);
           if (!ia)
               break;
+        /*
+         *  Acquire the inode ref here to prevent reclaim from
+         *  deadlocking. The ref gets dropped in fuse_readpages_end().
+         */
+        igrab(inode);
+
           ap = &ia->ap;
           while (pages < cur_pages) {
#3 lore raw
Quoted 75 lines · Gao Xiang
Quoted 67 lines · Gao Xiang
Quoted 58 lines · Joanne Koong

A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:

Quoted 1 line
Quoted 1 line
Quoted 1 line

stack_trace(1504735)

folio_wait_bit_common (mm/filemap.c:1308:4) folio_lock (./include/linux/pagemap.h:1052:3) truncate_inode_pages_range (mm/truncate.c:336:10) fuse_evict_inode (fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3) dentry_unlink_inode (fs/dcache.c:412:3) __dentry_kill (fs/dcache.c:615:3) shrink_kill (fs/dcache.c:1060:12) shrink_dentry_list (fs/dcache.c:1087:3) prune_dcache_sb (fs/dcache.c:1168:2) super_cache_scan (fs/super.c:221:10) do_shrink_slab (mm/shrinker.c:435:9) shrink_slab (mm/shrinker.c:626:10) shrink_node (mm/vmscan.c:5951:2) shrink_zones (mm/vmscan.c:6195:3) do_try_to_free_pages (mm/vmscan.c:6257:3) do_swap_page (mm/memory.c:4136:11) handle_pte_fault (mm/memory.c:5562:10) handle_mm_fault (mm/memory.c:5870:9) do_user_addr_fault (arch/x86/mm/fault.c:1338:10) handle_page_fault (arch/x86/mm/fault.c:1481:3) exc_page_fault (arch/x86/mm/fault.c:1539:2) asm_exc_page_fault+0x22/0x27

During readahead, the folio is locked. When fuse_evict_inode() is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.

The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.

This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.

---
  fs/fuse/file.c | 7 +++++++
  1 file changed, 7 insertions(+)
fs/fuse/file.c +0 -0
index f1ef77a0be05..8e759061b843 100644

fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 895

Quoted 3 lines
fs/fuse/file.c +1 -0
@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,
      if (ia->ff)
          fuse_file_put(ia->ff, false);
+    iput(inode);

It's somewhat odd to use igrab and iput in the read(ahead) context.

Quoted 5 lines

I wonder for FUSE, if it's possible to just wait ongoing locked folios when i_count == 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.

Quoted 8 lines

Oh, it was in the dentry LRU list instead, I don't think it can work.

Or normally the kernel filesystem uses GFP_NOFS to avoid such deadlock (see if (!(sc->gfp_mask & __GFP_FS)) in super_cache_scan()), I wonder if the daemon should simply use prctl(PR_SET_IO_FLUSHER) so that the user daemon won't be called into the fs reclaim context again.

Hi Gao,

We cannot rely on the daemon to set this unfortunately. This can tie up reclaim and kswapd for the entire system so I think this enforcement needs to be guaranteed and at the kernel level. For example, there is the possibility of malicious servers, which we cannot rely on to set FR_SET_IO_FLUSHER.

Thanks,
Joanne

Quoted 25 lines

Thanks,
Gao Xiang

Quoted 20 lines

Thanks,
Gao Xiang

Quoted 14 lines
fs/fuse/file.c +6 -0
      fuse_io_free(ia);
  }
@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac)
          ia = fuse_io_alloc(NULL, cur_pages);
          if (!ia)
              break;
+        /*
+         *  Acquire the inode ref here to prevent reclaim from
+         *  deadlocking. The ref gets dropped in fuse_readpages_end().
+         */
+        igrab(inode);
+
          ap = &ia->ap;
          while (pages < cur_pages) {
#4 lore raw
Quoted 85 lines · Joanne Koong
Quoted 75 lines · Gao Xiang
Quoted 67 lines · Gao Xiang
Quoted 58 lines · Joanne Koong

A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:

Quoted 1 line
Quoted 1 line
Quoted 1 line

stack_trace(1504735)

folio_wait_bit_common (mm/filemap.c:1308:4) folio_lock (./include/linux/pagemap.h:1052:3) truncate_inode_pages_range (mm/truncate.c:336:10) fuse_evict_inode (fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3) dentry_unlink_inode (fs/dcache.c:412:3) __dentry_kill (fs/dcache.c:615:3) shrink_kill (fs/dcache.c:1060:12) shrink_dentry_list (fs/dcache.c:1087:3) prune_dcache_sb (fs/dcache.c:1168:2) super_cache_scan (fs/super.c:221:10) do_shrink_slab (mm/shrinker.c:435:9) shrink_slab (mm/shrinker.c:626:10) shrink_node (mm/vmscan.c:5951:2) shrink_zones (mm/vmscan.c:6195:3) do_try_to_free_pages (mm/vmscan.c:6257:3) do_swap_page (mm/memory.c:4136:11) handle_pte_fault (mm/memory.c:5562:10) handle_mm_fault (mm/memory.c:5870:9) do_user_addr_fault (arch/x86/mm/fault.c:1338:10) handle_page_fault (arch/x86/mm/fault.c:1481:3) exc_page_fault (arch/x86/mm/fault.c:1539:2) asm_exc_page_fault+0x22/0x27

During readahead, the folio is locked. When fuse_evict_inode() is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.

The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.

This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.

---
   fs/fuse/file.c | 7 +++++++
   1 file changed, 7 insertions(+)
fs/fuse/file.c +0 -0
index f1ef77a0be05..8e759061b843 100644

fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 895

Quoted 3 lines
fs/fuse/file.c +1 -0
@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,
       if (ia->ff)
           fuse_file_put(ia->ff, false);
+    iput(inode);

It's somewhat odd to use igrab and iput in the read(ahead) context.

Quoted 5 lines

I wonder for FUSE, if it's possible to just wait ongoing locked folios when i_count == 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.

Quoted 8 lines

Oh, it was in the dentry LRU list instead, I don't think it can work.

Or normally the kernel filesystem uses GFP_NOFS to avoid such deadlock (see if (!(sc->gfp_mask & __GFP_FS)) in super_cache_scan()), I wonder if the daemon should simply use prctl(PR_SET_IO_FLUSHER) so that the user daemon won't be called into the fs reclaim context again.

Quoted 7 lines

Hi Gao,

We cannot rely on the daemon to set this unfortunately. This can tie up reclaim and kswapd for the entire system so I think this enforcement needs to be guaranteed and at the kernel level. For example, there is the possibility of malicious servers, which we cannot rely on to set FR_SET_IO_FLUSHER.

Hi Joanne,

Yes, currently I don't have a saner way in my mind but iput() in such nested context sounds a new entry (e.g. I thought kernel page fault path should have nothing tangled with evict() directly but I may be wrong.)

In principle, typical the kernel filesystem holds a valid file during the entire buffered read (file)/mmap (vma->vm_file) submission path (and of course they won't upcall to userspace and then do random behavior in the userspace for I/O processing).

So for the kernel filesystems I think the GFP_NOFS allocation isn't needed since file.f_path always takes a valid dentry ref during the submission so that such dentry/inode reclaim above is impossible IMO.)

Thanks,
Gao Xiang

Quoted 4 lines

Thanks,
Joanne

#5 lore raw
Quoted 96 lines · Gao Xiang
Quoted 85 lines · Joanne Koong
Quoted 75 lines · Gao Xiang
Quoted 67 lines · Gao Xiang
Quoted 58 lines · Joanne Koong

A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:

Quoted 1 line
Quoted 1 line
Quoted 1 line

stack_trace(1504735)

   folio_wait_bit_common (mm/filemap.c:1308:4) folio_lock (./include/linux/pagemap.h:1052:3) truncate_inode_pages_range (mm/truncate.c:336:10) fuse_evict_inode (fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3) dentry_unlink_inode (fs/dcache.c:412:3) __dentry_kill (fs/dcache.c:615:3) shrink_kill (fs/dcache.c:1060:12) shrink_dentry_list (fs/dcache.c:1087:3) prune_dcache_sb (fs/dcache.c:1168:2) super_cache_scan (fs/super.c:221:10) do_shrink_slab (mm/shrinker.c:435:9) shrink_slab (mm/shrinker.c:626:10) shrink_node (mm/vmscan.c:5951:2) shrink_zones (mm/vmscan.c:6195:3) do_try_to_free_pages (mm/vmscan.c:6257:3) do_swap_page (mm/memory.c:4136:11) handle_pte_fault (mm/memory.c:5562:10) handle_mm_fault (mm/memory.c:5870:9) do_user_addr_fault (arch/x86/mm/fault.c:1338:10) handle_page_fault (arch/x86/mm/fault.c:1481:3) exc_page_fault (arch/x86/mm/fault.c:1539:2) asm_exc_page_fault+0x22/0x27

During readahead, the folio is locked. When fuse_evict_inode() is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.

The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.

This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.

---
   fs/fuse/file.c | 7 +++++++

   1 file changed, 7 insertions(+)
fs/fuse/file.c +0 -0
index f1ef77a0be05..8e759061b843 100644

fs/fuse/file.c static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args, · line 895

Quoted 3 lines
fs/fuse/file.c +1 -0
@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,
        if (ia->ff)
            fuse_file_put(ia->ff, false);
+    iput(inode);

It's somewhat odd to use igrab and iput in the read(ahead) context.

Quoted 5 lines

I wonder for FUSE, if it's possible to just wait ongoing locked folios when i_count == 0 (e.g. in .drop_inode) before adding into lru so that the final inode eviction won't wait its readahead requests itself so that deadlock like this can be avoided.

Quoted 8 lines

Oh, it was in the dentry LRU list instead, I don't think it can work.

Or normally the kernel filesystem uses GFP_NOFS to avoid such deadlock (see if (!(sc->gfp_mask & __GFP_FS)) in super_cache_scan()), I wonder if the daemon should simply use prctl(PR_SET_IO_FLUSHER) so that the user daemon won't be called into the fs reclaim context again.

Quoted 7 lines

Hi Gao,

We cannot rely on the daemon to set this unfortunately. This can tie up reclaim and kswapd for the entire system so I think this enforcement needs to be guaranteed and at the kernel level. For example, there is the possibility of malicious servers, which we cannot rely on to set FR_SET_IO_FLUSHER.

Quoted 5 lines

Hi Joanne,

Yes, currently I don't have a saner way in my mind but iput() in such nested context sounds a new entry (e.g. I thought kernel page fault path should have nothing tangled with

To clarify:
^ kernel file read page fault path tangled with this particular
inode in progress (I doesn't mean random inode reclaimation).

Quoted 19 lines

evict() directly but I may be wrong.)

In principle, typical the kernel filesystem holds a valid file during the entire buffered read (file)/mmap (vma->vm_file) submission path (and of course they won't upcall to userspace and then do random behavior in the userspace for I/O processing).

So for the kernel filesystems I think the GFP_NOFS allocation isn't needed since file.f_path always takes a valid dentry ref during the submission so that such dentry/inode reclaim above is impossible IMO.)

Thanks,
Gao Xiang

Quoted 4 lines

Thanks,
Joanne

maintainer
#6 lore raw
In reply to Gao Xiang
Quoted 2 lines · Gao Xiang

In principle, typical the kernel filesystem holds a valid file during the entire buffered read (file)/mmap (vma->vm_file)

Actually, fuse does hold a ref to fuse_file, which should make sure that the inode is not released while the readahead is ongoing.

See igrab() in fuse_prepare_release() and iput() in fuse_release_end().

So I don't understand what's going on.

Joanne, do you have a reproducer?

Thanks,
Miklos

#7 lore raw
Quoted 10 lines · Miklos Szeredi
Quoted 2 lines · Gao Xiang

In principle, typical the kernel filesystem holds a valid file during the entire buffered read (file)/mmap (vma->vm_file)

Actually, fuse does hold a ref to fuse_file, which should make sure that the inode is not released while the readahead is ongoing.

See igrab() in fuse_prepare_release() and iput() in fuse_release_end().

If the file is mmaped, couldn't the release happen before the page fault?

This is the chain of events I'm thinking of:
file is opened, file is mmapped, file is closed (which triggers the igrab() and iput() you mentioned in fuse_prepare_release()/fuse_release_end()), then client tries to read from the mmapped address which triggers a page fault which triggers readahead.

Or am I missing something here? I'm not super familiar with the mmap code but as far as I can tell from the logic in vm_mmap_pgoff() -> do_mmap() -> __mmap_region(), it doesn't grab a refcount on the inode or the file descriptor.

Quoted 4 lines

So I don't understand what's going on.

Joanne, do you have a reproducer?

I don't have a reproducer but we saw the stack trace in the commit message on our servers a couple of times and used drgn to pinpoint that there was a readahead request submitted.

Thanks,
Joanne

Quoted 3 lines

Thanks,
Miklos

maintainer
#8 lore raw
Quoted 4 lines · Joanne Koong

Or am I missing something here? I'm not super familiar with the mmap code but as far as I can tell from the logic in vm_mmap_pgoff() -> do_mmap() -> __mmap_region(), it doesn't grab a refcount on the inode or the file descriptor.

mmap keeps a ref on the file (vma->vm_file), so only munmap() will trigger ->release().

Thanks,
Miklos

#9 lore raw
Quoted 10 lines · Miklos Szeredi
Quoted 4 lines · Joanne Koong

Or am I missing something here? I'm not super familiar with the mmap code but as far as I can tell from the logic in vm_mmap_pgoff() -> do_mmap() -> __mmap_region(), it doesn't grab a refcount on the inode or the file descriptor.

mmap keeps a ref on the file (vma->vm_file), so only munmap() will trigger ->release().

Ah okay I see, I thought the .release callback is for when the last fd for a file gets closed but clearly that's not how it works. Thanks for clarifying that.

I'm confused too then how we can have a fuse inode with no refcount on it but with one of its folios indefinitely locked. The paths where a folio can be indefinitely locked afaict are from reads/readahead, buffered writes, and notify retrieves. But each of those paths hold a reference to the inode, so I'm confused how we end up in this situation.

Thanks,
Joanne

Quoted 3 lines

Thanks,
Miklos

#10 lore raw
In reply to Miklos Szeredi
Quoted 6 lines · Miklos Szeredi
Quoted 2 lines · Gao Xiang

In principle, typical the kernel filesystem holds a valid file during the entire buffered read (file)/mmap (vma->vm_file)

Quoted 8 lines

Actually, fuse does hold a ref to fuse_file, which should make sure that the inode is not released while the readahead is ongoing.

See igrab() in fuse_prepare_release() and iput() in fuse_release_end().

So I don't understand what's going on.

Joanne, do you have a reproducer?

Omar figured out that the servers where we ran into this had fc->no_open set.

The igrab() in fuse_prepare_release() you mentioned above happens only when fc->no_open is not set. commit e26ee4efbc79 "fuse: allocate ff->release_args only if release is needed" is what changed this behavior.

If we revert this commit, I think this fixes the issue. Unless you prefer another solution?

Thanks,
Joanne

Quoted 3 lines

Thanks,
Miklos

maintainer
#11 lore raw
In reply to Joanne Koong
Quoted 42 lines · Joanne Koong

A deadlock can occur if the server triggers reclaim while servicing a readahead request, and reclaim attempts to evict the inode of the file being read ahead:

Quoted 1 line
Quoted 1 line
Quoted 1 line

stack_trace(1504735)

folio_wait_bit_common (mm/filemap.c:1308:4) folio_lock (./include/linux/pagemap.h:1052:3) truncate_inode_pages_range (mm/truncate.c:336:10) fuse_evict_inode (fs/fuse/inode.c:161:2) evict (fs/inode.c:704:3) dentry_unlink_inode (fs/dcache.c:412:3) __dentry_kill (fs/dcache.c:615:3) shrink_kill (fs/dcache.c:1060:12) shrink_dentry_list (fs/dcache.c:1087:3) prune_dcache_sb (fs/dcache.c:1168:2) super_cache_scan (fs/super.c:221:10) do_shrink_slab (mm/shrinker.c:435:9) shrink_slab (mm/shrinker.c:626:10) shrink_node (mm/vmscan.c:5951:2) shrink_zones (mm/vmscan.c:6195:3) do_try_to_free_pages (mm/vmscan.c:6257:3) do_swap_page (mm/memory.c:4136:11) handle_pte_fault (mm/memory.c:5562:10) handle_mm_fault (mm/memory.c:5870:9) do_user_addr_fault (arch/x86/mm/fault.c:1338:10) handle_page_fault (arch/x86/mm/fault.c:1481:3) exc_page_fault (arch/x86/mm/fault.c:1539:2) asm_exc_page_fault+0x22/0x27

During readahead, the folio is locked. When fuse_evict_inode() is called, it attempts to remove all folios associated with the inode from the page cache (truncate_inode_pages_range()), which requires acquiring the folio lock. If the server triggers reclaim while servicing a readahead request, reclaim will block indefinitely waiting for the folio lock, while readahead cannot relinquish the lock because it is itself blocked in reclaim, resulting in a deadlock.

The inode is only evicted if it has no remaining references after its dentry is unlinked. Since readahead is asynchronous, it is not guaranteed that the inode will have any references at this point.

Quoted 5 lines

This fixes the deadlock by holding a reference on the inode while readahead is in progress, which prevents the inode from being evicted until readahead completes. Additionally, this also prevents a malicious or buggy server from indefinitely blocking kswapd if it never fulfills a readahead request.

I don't see a better way to fix this, but Cc-ing Willy as the readahead/mm expert.

Quoted 2 lines

This is not a new bug, right? So at least add a

Thanks,
Miklos

Quoted 32 lines
---
 fs/fuse/file.c | 7 +++++++
 1 file changed, 7 insertions(+)
fs/fuse/file.c +7 -1
index f1ef77a0be05..8e759061b843 100644
@@ -893,6 +893,7 @@ static void fuse_readpages_end(struct fuse_mount *fm, struct fuse_args *args,
        if (ia->ff)
                fuse_file_put(ia->ff, false);
 
+       iput(inode);
        fuse_io_free(ia);
 }
 
@@ -973,6 +974,12 @@ static void fuse_readahead(struct readahead_control *rac)
                ia = fuse_io_alloc(NULL, cur_pages);
                if (!ia)
                        break;
+               /*
+                *  Acquire the inode ref here to prevent reclaim from
+                *  deadlocking. The ref gets dropped in fuse_readpages_end().
+                */
+               igrab(inode);
+
                ap = &ia->ap;
 
                while (pages < cur_pages) {
--

2.47.3