Skip to contentSkip to conversation

btrfs: fix the delayed iputs ASSERT() when the fs

PATCH In review Qu Wenruo 3 participants 6 messages Mar 7, 2025 Btrfs file system

lore
Share & tools
lore

Qu Wenruo

author
#0 lore raw

cover btrfs: fix the delayed iputs ASSERT() when the fs

Although Filipe fixes a lot of cases where the delayed iputs ASSERT() can be triggered, that's all focusing on the common cases where the fs is still in a good shape.

However in my experimental 2K block size runs, the test case generic/475 has a very high chance that the emulated error makes the fs to flips into an error status.

That error status will trigger a completely new path inside close_ctree(), calling btrfs_error_commit_super() which flush and wait for all ordered extents.

That btrfs_error_commit_super() will generate new delayed iputs asynchronously, thus trigger the later ASSERT().

The first patch is to fix the error, at least I can no longer trigger the ASSERT() in generic/475.

The second patch is to add an extra WARN_ON_ONCE() inside btrfs_add_delayed_iput(), triggered if close_ctree() has processed after btrfs_error_commit_super() calls.

Qu Wenruo (2):
btrfs: run btrfs_error_commit_super() early
btrfs: add extra warning if delayed iput is added when it's not
allowed

 fs/btrfs/disk-io.c | 14 +++++++++++---
 fs/btrfs/fs.h      |  4 ++++
 fs/btrfs/inode.c   |  4 ++++
 3 files changed, 19 insertions(+), 3 deletions(-)
Signature · 1 line
2.48.1
author
#1 lore raw

PATCH 1/2 btrfs: run btrfs_error_commit_super() early

[BUG]
Even after all the error fixes related the "ASSERT(list_empty(&fs_info->delayed_iputs));" in close_ctree(), I can still hit it reliably with my experimental 2K block size.

[CAUSE]
In my case, all the error is triggered after the fs is already in error status.

I find the following call trace to be the cause of race:

Main thread | endio_write_workers
---------------------------------------+------------------------

close_ctree()                          |
|- btrfs_error_commit_super()          |
|  |- btrfs_cleanup_tranasction()      |
|  |  |- btrfs_wait_ordered_roots()    |
|  |- btrfs_run_delayed_iputs()        |
|                                      | btrfs_finish_ordered_io()
|                                      | |- btrfs_put_ordered_extent()
|                                      |    |- btrfs_add_delayed_iput()
|- ASSERT(list_empty(delayed_iputs))   |

!!! Triggered !!!

The root cause is that, btrfs_wait_ordered_roots() only wait for ordered extents to finish their IOs, not to wait for them to finish and removed.

[FIX]
Since btrfs_error_commit_super() will flush and wait for all ordered extents, it should be executed early, before we start flushing the workqueues.

---
 fs/btrfs/disk-io.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
fs/btrfs/disk-io.c +8 -3
index b0f125d8efa0..320136a59db2 100644
@@ -4320,6 +4320,14 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
 	/* clear out the rbtree of defraggable inodes */
 	btrfs_cleanup_defrag_inodes(fs_info);
 
+	/*
+	 * Handle the error fs first, as it will flush and wait for
+	 * all ordred extents.
+	 * This will generate delayed iputs, thus we want to handle it first.
+	 */
+	if (unlikely(BTRFS_FS_ERROR(fs_info)))
+		btrfs_error_commit_super(fs_info);
+
 	/*
 	 * Wait for any fixup workers to complete.
 	 * If we don't wait for them here and they are still running by the time
@@ -4410,9 +4418,6 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
 			btrfs_err(fs_info, "commit super ret %d", ret);
 	}
 
-	if (BTRFS_FS_ERROR(fs_info))
-		btrfs_error_commit_super(fs_info);
-
 	kthread_stop(fs_info->transaction_kthread);
 	kthread_stop(fs_info->cleaner_kthread);
 
Signature · 1 line
2.48.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] btrfs: run btrfs_error_commit_super() early

Quoted 10 lines · Qu Wenruo

[BUG]
Even after all the error fixes related the "ASSERT(list_empty(&fs_info->delayed_iputs));" in close_ctree(), I can still hit it reliably with my experimental 2K block size.

[CAUSE]
In my case, all the error is triggered after the fs is already in error status.

Quoted 7 lines

I find the following call trace to be the cause of race:

Main thread | endio_write_workers
---------------------------------------+------------------------

close_ctree()                          |
|- btrfs_error_commit_super()          |
|  |- btrfs_cleanup_tranasction()      |

btrfs_cleanup_tranasction() ->btrfs_cleanup_transaction()

Quoted 1 line

| | |- btrfs_wait_ordered_roots() |

It took me a bit to see where this call came from, which is from btrfs_destroy_all_ordered_extents() which is called by btrfs_cleanup_transaction().
Could be added here.

Quoted 26 lines
|  |- btrfs_run_delayed_iputs()        |
|                                      | btrfs_finish_ordered_io()
|                                      | |- btrfs_put_ordered_extent()
|                                      |    |- btrfs_add_delayed_iput()
|- ASSERT(list_empty(delayed_iputs))   |

!!! Triggered !!!

The root cause is that, btrfs_wait_ordered_roots() only wait for ordered extents to finish their IOs, not to wait for them to finish and removed.

[FIX]
Since btrfs_error_commit_super() will flush and wait for all ordered extents, it should be executed early, before we start flushing the workqueues.

---
 fs/btrfs/disk-io.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
fs/btrfs/disk-io.c +0 -0
index b0f125d8efa0..320136a59db2 100644

fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4325

Quoted 6 lines
fs/btrfs/disk-io.c +3 -0
@@ -4320,6 +4320,14 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
        /* clear out the rbtree of defraggable inodes */
        btrfs_cleanup_defrag_inodes(fs_info);
 
+       /*
+        * Handle the error fs first, as it will flush and wait for
+        * all ordred extents.

ordred -> ordered

Those can be fixed when pushing to for-next.

Looks good:

Thanks.

Quoted 21 lines
fs/btrfs/disk-io.c +5 -4
+        * This will generate delayed iputs, thus we want to handle it first.
+        */
+       if (unlikely(BTRFS_FS_ERROR(fs_info)))
+               btrfs_error_commit_super(fs_info);
+
        /*
         * Wait for any fixup workers to complete.
         * If we don't wait for them here and they are still running by the time
@@ -4410,9 +4418,6 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
                        btrfs_err(fs_info, "commit super ret %d", ret);
        }
 
-       if (BTRFS_FS_ERROR(fs_info))
-               btrfs_error_commit_super(fs_info);
-
        kthread_stop(fs_info->transaction_kthread);
        kthread_stop(fs_info->cleaner_kthread);
 
--

2.48.1

author
#3 lore raw

PATCH 2/2 btrfs: add extra warning if delayed iput is added when it's not allowed

In reply to Qu Wenruo

Since I have triggered the ASSERT() on the delayed iput too many times, now is the time to add some extra debug warnings for delayed iput.

After all btrfs_commit_super() calls, we should no longer allow any new delayed iput being added.

So introduce a new BTRFS_FS_STATE_NO_DELAYED_IPUT for debug builds, set after above mentioned timing.
And all btrfs_add_delayed_iput() will check that flag and give a WARN_ON_ONCE().

I really hope this warning will never be triggered.

---
 fs/btrfs/disk-io.c | 3 +++
 fs/btrfs/fs.h      | 4 ++++
 fs/btrfs/inode.c   | 4 ++++
 3 files changed, 11 insertions(+)
fs/btrfs/disk-io.c +3 -0
index 320136a59db2..bb20c015b779 100644
@@ -4417,6 +4417,9 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
 		if (ret)
 			btrfs_err(fs_info, "commit super ret %d", ret);
 	}
+#ifdef CONFIG_BTRFS_DEBUG
+	set_bit(BTRFS_FS_STATE_NO_DELAYED_IPUT, &fs_info->fs_state);
+#endif
 
 	kthread_stop(fs_info->transaction_kthread);
 	kthread_stop(fs_info->cleaner_kthread);
fs/btrfs/fs.h +4 -0
index b8c2e59ffc43..ee298dd0f568 100644
@@ -117,6 +117,10 @@ enum {
 	/* Indicates there was an error cleaning up a log tree. */
 	BTRFS_FS_STATE_LOG_CLEANUP_ERROR,
 
+#ifdef CONFIG_BTRFS_DEBUG
+	/* No more delayed iput can be queued. */
+	BTRFS_FS_STATE_NO_DELAYED_IPUT,
+#endif
 	BTRFS_FS_STATE_COUNT
 };
 
fs/btrfs/inode.c +4 -0
index 8ac4858b70e7..d2bf81c08f13 100644
@@ -3435,6 +3435,10 @@ void btrfs_add_delayed_iput(struct btrfs_inode *inode)
 	if (atomic_add_unless(&inode->vfs_inode.i_count, -1, 1))
 		return;
 
+#ifdef CONFIG_BTRFS_DEBUG
+	WARN_ON_ONCE(test_bit(BTRFS_FS_STATE_NO_DELAYED_IPUT, &fs_info->fs_state));
+#endif
+
 	atomic_inc(&fs_info->nr_delayed_iputs);
 	/*
 	 * Need to be irq safe here because we can be called from either an irq
Signature · 1 line
2.48.1
#4 lore raw

Re: [PATCH 2/2] btrfs: add extra warning if delayed iput is added when it's not allowed

Quoted 26 lines · Qu Wenruo

Since I have triggered the ASSERT() on the delayed iput too many times, now is the time to add some extra debug warnings for delayed iput.

After all btrfs_commit_super() calls, we should no longer allow any new delayed iput being added.

So introduce a new BTRFS_FS_STATE_NO_DELAYED_IPUT for debug builds, set after above mentioned timing.
And all btrfs_add_delayed_iput() will check that flag and give a WARN_ON_ONCE().

I really hope this warning will never be triggered.

---
 fs/btrfs/disk-io.c | 3 +++
 fs/btrfs/fs.h      | 4 ++++
 fs/btrfs/inode.c   | 4 ++++
 3 files changed, 11 insertions(+)
fs/btrfs/disk-io.c +0 -0
index 320136a59db2..bb20c015b779 100644

fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4422

Quoted 6 lines
fs/btrfs/disk-io.c +3 -0
@@ -4417,6 +4417,9 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
                if (ret)
                        btrfs_err(fs_info, "commit super ret %d", ret);
        }
+#ifdef CONFIG_BTRFS_DEBUG
+       set_bit(BTRFS_FS_STATE_NO_DELAYED_IPUT, &fs_info->fs_state);
+#endif

We'll want this earlier, right after we call btrfs_run_delayed_iputs(), so that we don't miss the case where for example we are forgetting to flush some queue and then the delayed iput is added after running iputs and before calling btfs_commit_super(), or right after calling btrfs_commit_super() and before setting the bit.

Quoted 23 lines
fs/btrfs/disk-io.c +0 -0
        kthread_stop(fs_info->transaction_kthread);
        kthread_stop(fs_info->cleaner_kthread);
fs/btrfs/fs.h +4 -0
index b8c2e59ffc43..ee298dd0f568 100644
@@ -117,6 +117,10 @@ enum {
        /* Indicates there was an error cleaning up a log tree. */
        BTRFS_FS_STATE_LOG_CLEANUP_ERROR,
 
+#ifdef CONFIG_BTRFS_DEBUG
+       /* No more delayed iput can be queued. */
+       BTRFS_FS_STATE_NO_DELAYED_IPUT,
+#endif
        BTRFS_FS_STATE_COUNT
 };
 
fs/btrfs/inode.c +0 -0
index 8ac4858b70e7..d2bf81c08f13 100644

fs/btrfs/inode.c void btrfs_add_delayed_iput(struct btrfs_inode *inode) · line 3441

Quoted 7 lines
fs/btrfs/inode.c +4 -0
@@ -3435,6 +3435,10 @@ void btrfs_add_delayed_iput(struct btrfs_inode *inode)
        if (atomic_add_unless(&inode->vfs_inode.i_count, -1, 1))
                return;
 
+#ifdef CONFIG_BTRFS_DEBUG
+       WARN_ON_ONCE(test_bit(BTRFS_FS_STATE_NO_DELAYED_IPUT, &fs_info->fs_state));
+#endif
+

I would get rid of the #ifdef CONFIG_BTRFS_DEBUG everywhere. It's very cheap code, negligible.

Thanks.

Quoted 6 lines
fs/btrfs/inode.c +0 -1
        atomic_inc(&fs_info->nr_delayed_iputs);
        /*
         * Need to be irq safe here because we can be called from either an irq
--

2.48.1

#5 lore raw

Re: [PATCH 2/2] btrfs: add extra warning if delayed iput is added when it's not allowed

Quoted 32 lines · Filipe Manana
Quoted 31 lines · Qu Wenruo

Since I have triggered the ASSERT() on the delayed iput too many times, now is the time to add some extra debug warnings for delayed iput.

After all btrfs_commit_super() calls, we should no longer allow any new delayed iput being added.

So introduce a new BTRFS_FS_STATE_NO_DELAYED_IPUT for debug builds, set after above mentioned timing.
And all btrfs_add_delayed_iput() will check that flag and give a WARN_ON_ONCE().

I really hope this warning will never be triggered.

---
  fs/btrfs/disk-io.c | 3 +++
  fs/btrfs/fs.h      | 4 ++++
  fs/btrfs/inode.c   | 4 ++++
  3 files changed, 11 insertions(+)
fs/btrfs/disk-io.c +2 -0
index 320136a59db2..bb20c015b779 100644
@@ -4417,6 +4417,9 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
                 if (ret)
                         btrfs_err(fs_info, "commit super ret %d", ret);
         }
+#ifdef CONFIG_BTRFS_DEBUG
+       set_bit(BTRFS_FS_STATE_NO_DELAYED_IPUT, &fs_info->fs_state);
Quoted 8 lines

fs/btrfs/disk-io.c void __cold close_ctree(struct btrfs_fs_info *fs_info) · line 4422

Quoted 1 line
fs/btrfs/disk-io.c +1 -0
+#endif

We'll want this earlier, right after we call btrfs_run_delayed_iputs(), so that we don't miss the case where for example we are forgetting to flush some queue and then the delayed iput is added after running iputs and before calling btfs_commit_super(), or right after calling btrfs_commit_super() and before setting the bit.

I guess this means the call of btrfs_run_delayed_iputs() inside btrfs_commit_super() should also be removed in this case?

Thanks,
Qu

Quoted 46 lines
Quoted 24 lines
fs/btrfs/disk-io.c +0 -0
         kthread_stop(fs_info->transaction_kthread);
         kthread_stop(fs_info->cleaner_kthread);
fs/btrfs/fs.h +4 -0
index b8c2e59ffc43..ee298dd0f568 100644
@@ -117,6 +117,10 @@ enum {
         /* Indicates there was an error cleaning up a log tree. */
         BTRFS_FS_STATE_LOG_CLEANUP_ERROR,
 
+#ifdef CONFIG_BTRFS_DEBUG
+       /* No more delayed iput can be queued. */
+       BTRFS_FS_STATE_NO_DELAYED_IPUT,
+#endif
         BTRFS_FS_STATE_COUNT
  };
 
fs/btrfs/inode.c +0 -0
index 8ac4858b70e7..d2bf81c08f13 100644

fs/btrfs/inode.c void btrfs_add_delayed_iput(struct btrfs_inode *inode) · line 3441

Quoted 7 lines
fs/btrfs/inode.c +4 -0
@@ -3435,6 +3435,10 @@ void btrfs_add_delayed_iput(struct btrfs_inode *inode)
         if (atomic_add_unless(&inode->vfs_inode.i_count, -1, 1))
                 return;
 
+#ifdef CONFIG_BTRFS_DEBUG
+       WARN_ON_ONCE(test_bit(BTRFS_FS_STATE_NO_DELAYED_IPUT, &fs_info->fs_state));
+#endif
+

I would get rid of the #ifdef CONFIG_BTRFS_DEBUG everywhere. It's very cheap code, negligible.

Thanks.

Quoted 6 lines
fs/btrfs/inode.c +0 -1
         atomic_inc(&fs_info->nr_delayed_iputs);
         /*
          * Need to be irq safe here because we can be called from either an irq
--

2.48.1