btrfs: scrub: wip, pause on fs freeze
Subsystems (2)
David Sterba
maintainerPATCH 1/1 btrfs: scrub: wip, pause on fs freeze
Implement sb->freeze_super that can instruct our threads to pause themselves. In case of (read-write) scrub this means to undo mnt_want_write, implemented as sb_start_write()/sb_end_write(). The freeze_super callback is necessary otherwise the call sb_want_write() inside the generic implementation hangs.
This works with concurrent scrub running and 'fsfreeze --freeze', not with process freezing (like with suspend).
References: https://lwn.net/Articles/1018341/
--- fs/btrfs/fs.h | 2 ++ fs/btrfs/scrub.c | 21 +++++++++++++++++++++ fs/btrfs/super.c | 36 ++++++++++++++++++++++++++++++++---- 3 files changed, 55 insertions(+), 4 deletions(-)
fs/btrfs/fs.h +2 -0
@@ -137,6 +137,8 @@ enum {BTRFS_FS_QUOTA_OVERRIDE ,/* Used to record internally whether fs has been frozen */ BTRFS_FS_FROZEN , +/* Started freezing, pause your progress. */ + BTRFS_FS_FREEZING,/* * Indicate that balance has been set up from the ioctl and is in the * main phase. The fs_info::balance_ctl is initialized.
fs/btrfs/scrub.c +21 -0
@@ -2250,6 +2250,27 @@ static int scrub_simple_mirror(struct scrub_ctx *sctx, ret = -ECANCELED ;break ; } + +/* Freezing? */ +if (test_bit (BTRFS_FS_FREEZING , &fs_info->flags)) { +atomic_inc (&fs_info->scrubs_paused); +smp_mb (); +wake_up (&fs_info->scrub_pause_wait); + +if (!sctx->readonly) +sb_end_write (fs_info->sb); + +try_to_freeze (); +wait_on_bit (&fs_info->flags,BTRFS_FS_FREEZING ,TASK_UNINTERRUPTIBLE ); + +if (!sctx->readonly) +sb_start_write (fs_info->sb); + +atomic_dec (&fs_info->scrubs_paused); +smp_mb (); +wake_up (&fs_info->scrub_pause_wait); + } +/* Paused? */ if (atomic_read (&fs_info->scrub_pause_req)) {/* Push queued extents */
fs/btrfs/super.c +32 -4
@@ -2279,7 +2279,33 @@ static long btrfs_control_ioctl(struct file *file, unsigned int cmd,return ret; } -static int btrfs_freeze (struct super_block *sb) +static int btrfs_freeze_super (struct super_block *sb,enum freeze_holder who, +const void *freeze_owner) +{ +struct btrfs_fs_info *fs_info =btrfs_sb (sb); +int ret; + +set_bit (BTRFS_FS_FREEZING , &fs_info->flags); + ret =freeze_super (sb, who, freeze_owner); +if (ret <0 ) +clear_bit (BTRFS_FS_FREEZING , &fs_info->flags); +return ret; +} + +static int btrfs_thaw_super (struct super_block *sb,enum freeze_holder who, +const void *freeze_owner) +{ +struct btrfs_fs_info *fs_info =btrfs_sb (sb); +int ret; + + ret =thaw_super (sb, who, freeze_owner); +clear_bit (BTRFS_FS_FREEZING , &fs_info->flags); +smp_mb (); +wake_up_bit (&fs_info->flags,BTRFS_FS_FREEZING ); +return ret; +} + +static int btrfs_freeze_fs (struct super_block *sb) {struct btrfs_fs_info *fs_info =btrfs_sb (sb);
@@ -2345,7 +2371,7 @@ static int check_dev_super(struct btrfs_device *dev)return ret; } -static int btrfs_unfreeze (struct super_block *sb) +static int btrfs_unfreeze_fs (struct super_block *sb) {struct btrfs_fs_info *fs_info =btrfs_sb (sb);struct btrfs_device *device;
@@ -2426,8 +2452,10 @@ static const struct super_operations btrfs_super_ops = { .destroy_inode = btrfs_destroy_inode, .free_inode = btrfs_free_inode, .statfs = btrfs_statfs, - .freeze_fs = btrfs_freeze, - .unfreeze_fs = btrfs_unfreeze, + .freeze_super = btrfs_freeze_super, + .thaw_super = btrfs_thaw_super, + .freeze_fs = btrfs_freeze_fs, + .unfreeze_fs = btrfs_unfreeze_fs, .nr_cached_objects = btrfs_nr_cached_objects, .free_cached_objects = btrfs_free_cached_objects, };
Signature · 1 line
2.49.0
Quoted 37 lines · David Sterba
Implement
sb->freeze_superthat can instruct our threads to pause themselves. In case of (read-write) scrub this means to undomnt_want_write, implemented as sb_start_write()/sb_end_write(). Thefreeze_supercallback is necessary otherwise the callsb_want_write()inside the generic implementation hangs.This works with concurrent scrub running and 'fsfreeze --freeze', not with process freezing (like with suspend).
References: https://lwn.net/Articles/1018341/
Signed-off-by David Sterba <[email protected]>---
fs/btrfs/fs.h | 2 ++ fs/btrfs/scrub.c | 21 +++++++++++++++++++++ fs/btrfs/super.c | 36 ++++++++++++++++++++++++++++++++---- 3 files changed, 55 insertions(+), 4 deletions(-)fs/btrfs/fs.h +2
-0@@ -137,6 +137,8 @@ enum {BTRFS_FS_QUOTA_OVERRIDE ,/* Used to record internally whether fs has been frozen */ BTRFS_FS_FROZEN , +/* Started freezing, pause your progress. */ + BTRFS_FS_FREEZING,/* * Indicate that balance has been set up from the ioctl and is in the * main phase. The fs_info::balance_ctl is initialized. fs/btrfs/scrub.c +0
-0
fs/btrfs/scrub.c static int scrub_simple_mirror(struct scrub_ctx *sctx, · line 2257
Quoted 8 lines
fs/btrfs/scrub.c +5
-0@@ -2250,6 +2250,27 @@ static int scrub_simple_mirror(struct scrub_ctx *sctx, ret = -ECANCELED ;break ; } + +/* Freezing? */ +if (test_bit (BTRFS_FS_FREEZING , &fs_info->flags)) { +atomic_inc (&fs_info->scrubs_paused); +smp_mb ();
The memory barrier before wake_up seems not needed
Quoted 84 lines
fs/btrfs/scrub.c +16
-0+wake_up (&fs_info->scrub_pause_wait); + +if (!sctx->readonly) +sb_end_write (fs_info->sb); + +try_to_freeze (); +wait_on_bit (&fs_info->flags,BTRFS_FS_FREEZING ,TASK_UNINTERRUPTIBLE ); + +if (!sctx->readonly) +sb_start_write (fs_info->sb); + +atomic_dec (&fs_info->scrubs_paused); +smp_mb (); +wake_up (&fs_info->scrub_pause_wait); + } +/* Paused? */ if (atomic_read (&fs_info->scrub_pause_req)) {/* Push queued extents */ fs/btrfs/super.c +32
-4@@ -2279,7 +2279,33 @@ static long btrfs_control_ioctl(struct file *file, unsigned int cmd,return ret; } -static int btrfs_freeze (struct super_block *sb) +static int btrfs_freeze_super (struct super_block *sb,enum freeze_holder who, +const void *freeze_owner) +{ +struct btrfs_fs_info *fs_info =btrfs_sb (sb); +int ret; + +set_bit (BTRFS_FS_FREEZING , &fs_info->flags); + ret =freeze_super (sb, who, freeze_owner); +if (ret <0 ) +clear_bit (BTRFS_FS_FREEZING , &fs_info->flags); +return ret; +} + +static int btrfs_thaw_super (struct super_block *sb,enum freeze_holder who, +const void *freeze_owner) +{ +struct btrfs_fs_info *fs_info =btrfs_sb (sb); +int ret; + + ret =thaw_super (sb, who, freeze_owner); +clear_bit (BTRFS_FS_FREEZING , &fs_info->flags); +smp_mb (); +wake_up_bit (&fs_info->flags,BTRFS_FS_FREEZING ); +return ret; +} + +static int btrfs_freeze_fs (struct super_block *sb) {struct btrfs_fs_info *fs_info =btrfs_sb (sb);@@ -2345,7 +2371,7 @@ static int check_dev_super(struct btrfs_device *dev)return ret; } -static int btrfs_unfreeze (struct super_block *sb) +static int btrfs_unfreeze_fs (struct super_block *sb) {struct btrfs_fs_info *fs_info =btrfs_sb (sb);struct btrfs_device *device;@@ -2426,8 +2452,10 @@ static const struct super_operations btrfs_super_ops = { .destroy_inode = btrfs_destroy_inode, .free_inode = btrfs_free_inode, .statfs = btrfs_statfs, - .freeze_fs = btrfs_freeze, - .unfreeze_fs = btrfs_unfreeze, + .freeze_super = btrfs_freeze_super, + .thaw_super = btrfs_thaw_super, + .freeze_fs = btrfs_freeze_fs, + .unfreeze_fs = btrfs_unfreeze_fs, .nr_cached_objects = btrfs_nr_cached_objects, .free_cached_objects = btrfs_free_cached_objects, };Signature · 1 line
2.49.0
Quoted 40 lines · Alan Huang
Quoted 39 lines · David Sterba
Implement
sb->freeze_superthat can instruct our threads to pause themselves. In case of (read-write) scrub this means to undomnt_want_write, implemented as sb_start_write()/sb_end_write(). Thefreeze_supercallback is necessary otherwise the callsb_want_write()inside the generic implementation hangs.This works with concurrent scrub running and 'fsfreeze --freeze', not with process freezing (like with suspend).
References: https://lwn.net/Articles/1018341/
Signed-off-by David Sterba <[email protected]>---
fs/btrfs/fs.h | 2 ++ fs/btrfs/scrub.c | 21 +++++++++++++++++++++ fs/btrfs/super.c | 36 ++++++++++++++++++++++++++++++++---- 3 files changed, 55 insertions(+), 4 deletions(-)fs/btrfs/fs.h +2
-0@@ -137,6 +137,8 @@ enum {BTRFS_FS_QUOTA_OVERRIDE ,/* Used to record internally whether fs has been frozen */ BTRFS_FS_FROZEN , +/* Started freezing, pause your progress. */ + BTRFS_FS_FREEZING,/* * Indicate that balance has been set up from the ioctl and is in the * main phase. The fs_info::balance_ctl is initialized. fs/btrfs/scrub.c +0
-0@@ -2250,6 +2250,27 @@ static int scrub_simple_mirror(struct scrub_ctx *sctx, ret = -ECANCELED ;break ;
Quoted 8 lines
fs/btrfs/scrub.c static int scrub_simple_mirror(struct scrub_ctx *sctx, · line 2257
Quoted 6 lines
fs/btrfs/scrub.c +5
-0} + +/* Freezing? */ +if (test_bit (BTRFS_FS_FREEZING , &fs_info->flags)) { +atomic_inc (&fs_info->scrubs_paused); +smp_mb ();The memory barrier before
wake_upseems not needed
Yeah, possibly smp_mb__after_atomic(), I took the simplest approach not to get distracted by reasoning about barriers.
Quoted 51 lines · David Sterba
Quoted 40 lines · Alan Huang
Quoted 39 lines · David Sterba
Implement
sb->freeze_superthat can instruct our threads to pause themselves. In case of (read-write) scrub this means to undomnt_want_write, implemented as sb_start_write()/sb_end_write(). Thefreeze_supercallback is necessary otherwise the callsb_want_write()inside the generic implementation hangs.This works with concurrent scrub running and 'fsfreeze --freeze', not with process freezing (like with suspend).
References: https://lwn.net/Articles/1018341/
Signed-off-by David Sterba <[email protected]>---
fs/btrfs/fs.h | 2 ++ fs/btrfs/scrub.c | 21 +++++++++++++++++++++ fs/btrfs/super.c | 36 ++++++++++++++++++++++++++++++++---- 3 files changed, 55 insertions(+), 4 deletions(-)fs/btrfs/fs.h +2
-0@@ -137,6 +137,8 @@ enum {BTRFS_FS_QUOTA_OVERRIDE ,/* Used to record internally whether fs has been frozen */ BTRFS_FS_FROZEN , +/* Started freezing, pause your progress. */ + BTRFS_FS_FREEZING,/* * Indicate that balance has been set up from the ioctl and is in the * main phase. The fs_info::balance_ctl is initialized. fs/btrfs/scrub.c +0
-0@@ -2250,6 +2250,27 @@ static int scrub_simple_mirror(struct scrub_ctx *sctx, ret = -ECANCELED ;break ;Quoted 8 lines
fs/btrfs/scrub.c static int scrub_simple_mirror(struct scrub_ctx *sctx, · line 2257
Quoted 6 lines
fs/btrfs/scrub.c +5
-0} + +/* Freezing? */ +if (test_bit (BTRFS_FS_FREEZING , &fs_info->flags)) { +atomic_inc (&fs_info->scrubs_paused); +smp_mb ();The memory barrier before
wake_upseems not needed
Quoted 2 lines
Yeah, possibly
smp_mb__after_atomic(), I took the simplest approach not to get distracted by reasoning about barriers.
smp_mb__after_atomic() is only needed before wake_up_bit().
No memory barrier is needed before wake_up()
Quoted 5 lines · David Sterba
Implement
sb->freeze_superthat can instruct our threads to pause themselves. In case of (read-write) scrub this means to undomnt_want_write, implemented as sb_start_write()/sb_end_write(). Thefreeze_supercallback is necessary otherwise the callsb_want_write()inside the generic implementation hangs.
I don't this this is really going to work.
The main problem is out of btrfs, it's all about the s_writer.rw_sem.
If we have a running scrub, it holds the mnt_want_write_file(), which is a read lock on the rw_sem.
Then we start freezing, which will call sb_wait_write(), which will do a write lock on the rw_sem, waiting for the scrub to finish.
However the ->freeze() callback is only called when freeze_super() got the write lock on the rw_sem.
At that stage, it's already too late.
Thanks,
Qu>
Quoted 118 lines
This works with concurrent scrub running and 'fsfreeze --freeze', not with process freezing (like with suspend).
References: https://lwn.net/Articles/1018341/
Signed-off-by David Sterba <[email protected]>--- fs/btrfs/fs.h | 2 ++ fs/btrfs/scrub.c | 21 +++++++++++++++++++++ fs/btrfs/super.c | 36 ++++++++++++++++++++++++++++++++---- 3 files changed, 55 insertions(+), 4 deletions(-)fs/btrfs/fs.h +2
-0@@ -137,6 +137,8 @@ enum {BTRFS_FS_QUOTA_OVERRIDE ,/* Used to record internally whether fs has been frozen */ BTRFS_FS_FROZEN , +/* Started freezing, pause your progress. */ + BTRFS_FS_FREEZING,/* * Indicate that balance has been set up from the ioctl and is in the * main phase. The fs_info::balance_ctl is initialized. fs/btrfs/scrub.c +21
-0@@ -2250,6 +2250,27 @@ static int scrub_simple_mirror(struct scrub_ctx *sctx, ret = -ECANCELED ;break ; } + +/* Freezing? */ +if (test_bit (BTRFS_FS_FREEZING , &fs_info->flags)) { +atomic_inc (&fs_info->scrubs_paused); +smp_mb (); +wake_up (&fs_info->scrub_pause_wait); + +if (!sctx->readonly) +sb_end_write (fs_info->sb); + +try_to_freeze (); +wait_on_bit (&fs_info->flags,BTRFS_FS_FREEZING ,TASK_UNINTERRUPTIBLE ); + +if (!sctx->readonly) +sb_start_write (fs_info->sb); + +atomic_dec (&fs_info->scrubs_paused); +smp_mb (); +wake_up (&fs_info->scrub_pause_wait); + } +/* Paused? */ if (atomic_read (&fs_info->scrub_pause_req)) {/* Push queued extents */ fs/btrfs/super.c +32
-4@@ -2279,7 +2279,33 @@ static long btrfs_control_ioctl(struct file *file, unsigned int cmd,return ret; } -static int btrfs_freeze (struct super_block *sb) +static int btrfs_freeze_super (struct super_block *sb,enum freeze_holder who, +const void *freeze_owner) +{ +struct btrfs_fs_info *fs_info =btrfs_sb (sb); +int ret; + +set_bit (BTRFS_FS_FREEZING , &fs_info->flags); + ret =freeze_super (sb, who, freeze_owner); +if (ret <0 ) +clear_bit (BTRFS_FS_FREEZING , &fs_info->flags); +return ret; +} + +static int btrfs_thaw_super (struct super_block *sb,enum freeze_holder who, +const void *freeze_owner) +{ +struct btrfs_fs_info *fs_info =btrfs_sb (sb); +int ret; + + ret =thaw_super (sb, who, freeze_owner); +clear_bit (BTRFS_FS_FREEZING , &fs_info->flags); +smp_mb (); +wake_up_bit (&fs_info->flags,BTRFS_FS_FREEZING ); +return ret; +} + +static int btrfs_freeze_fs (struct super_block *sb) {struct btrfs_fs_info *fs_info =btrfs_sb (sb);@@ -2345,7 +2371,7 @@ static int check_dev_super(struct btrfs_device *dev)return ret; } -static int btrfs_unfreeze (struct super_block *sb) +static int btrfs_unfreeze_fs (struct super_block *sb) {struct btrfs_fs_info *fs_info =btrfs_sb (sb);struct btrfs_device *device;@@ -2426,8 +2452,10 @@ static const struct super_operations btrfs_super_ops = { .destroy_inode = btrfs_destroy_inode, .free_inode = btrfs_free_inode, .statfs = btrfs_statfs, - .freeze_fs = btrfs_freeze, - .unfreeze_fs = btrfs_unfreeze, + .freeze_super = btrfs_freeze_super, + .thaw_super = btrfs_thaw_super, + .freeze_fs = btrfs_freeze_fs, + .unfreeze_fs = btrfs_unfreeze_fs, .nr_cached_objects = btrfs_nr_cached_objects, .free_cached_objects = btrfs_free_cached_objects, };
Quoted 13 lines · Qu Wenruo
Quoted 5 lines · David Sterba
Implement
sb->freeze_superthat can instruct our threads to pause themselves. In case of (read-write) scrub this means to undomnt_want_write, implemented as sb_start_write()/sb_end_write(). Thefreeze_supercallback is necessary otherwise the callsb_want_write()inside the generic implementation hangs.I don't this this is really going to work.
The main problem is out of btrfs, it's all about the s_writer.rw_sem.
Quoted 8 lines
If we have a running scrub, it holds the
mnt_want_write_file(), which is a read lock on therw_sem.Then we start freezing, which will call
sb_wait_write(), which will do a write lock on therw_sem, waiting for the scrub to finish.However the ->freeze() callback is only called when
freeze_super()got the write lock on therw_sem.
Note there are 2 callbacks for freeze, sb::freeze_super and sb::freeze_fs.
ioctl_fsfreeze
if fs->freeze_super
call fs_freeze_super()
else
freeze_super()
sb_wait_write()
if (sb->freeze_fs)
call sb->freeze_fs()What you describe matches the 'else' branch as this unconditionally blocks on the sb_wait_write and never gets to our callback. This is what I observed too.
To fix that I've added the sb::freeze_super so it can do something to avoid the lockup in sb_wait_write(), because we must at some point call freeze_super().
Schematically it goes like this:
Freeze thread Scrub
btrfs_ioctl_scrub
mnt_want_write_file (effectively sb_want_write())ioctl_fsfreeze
fs->freeze_super
set fs->flags FREEZING
freeze_super
sb_wait_freeze()
(BLOCKS)
(chunk loop)
scrub_simple_mirror()
if fs->flags
sb_end_write()
wait_on_bit fs->flags FREEZING
(BLOCKS)
(UNBLOCKS)
...Unfreezing is basically the same in reverse:
ioctl_fsthaw
fs->thaw_super
thaw_super()
unset fs->flags FREEZING
(UNBLOCKS)
sb_want_write()
...So if you see a problem in that please let me know, I've tested the patch and it worked but I might me missing something.
Quoted 23 lines · David Sterba
Quoted 13 lines · Qu Wenruo
Quoted 5 lines · David Sterba
Implement
sb->freeze_superthat can instruct our threads to pause themselves. In case of (read-write) scrub this means to undomnt_want_write, implemented as sb_start_write()/sb_end_write(). Thefreeze_supercallback is necessary otherwise the callsb_want_write()inside the generic implementation hangs.I don't this this is really going to work.
The main problem is out of btrfs, it's all about the s_writer.rw_sem.
Quoted 8 lines
If we have a running scrub, it holds the
mnt_want_write_file(), which is a read lock on therw_sem.Then we start freezing, which will call
sb_wait_write(), which will do a write lock on therw_sem, waiting for the scrub to finish.However the ->freeze() callback is only called when
freeze_super()got the write lock on therw_sem.
Quoted 6 lines
Note there are 2 callbacks for freeze,
sb::freeze_superandsb::freeze_fs.
ioctl_fsfreeze
iffs->freeze_super
callfs_freeze_super()
Oh I forgot you implemented a new callback, ->freeze_super() to do the extra btrfs specific handling.
But this means, we're exposing ourselves to all the extra VFS handling,
I'm not sure if this is the preferred way, nor if the extra btrfs handling is good enough to cover what's done inside freeze_super() function (I guess not though).
The only user of ->freeze_super() callback is GFS2, and I didn't think we have enough locking/protection compared to either GFS2 nor VFS.
In that case, I still don't believe it's the correct way to go.
Add fsdevel list to see if it's even recommended to use ->freeze_super() callback.
Thanks,
Qu
Quoted 48 lines
else freeze_super() sb_wait_write() if (sb->freeze_fs) call sb->freeze_fs()What you describe matches the 'else' branch as this unconditionally blocks on the
sb_wait_writeand never gets to our callback. This is what I observed too.To fix that I've added the
sb::freeze_superso it can do something to avoid the lockup insb_wait_write(), because we must at some point callfreeze_super().Schematically it goes like this:
Freeze thread Scrub
btrfs_ioctl_scrub
mnt_want_write_file(effectivelysb_want_write())ioctl_fsfreeze
fs->freeze_superset fs->flags FREEZING freeze_super sb_wait_freeze() (BLOCKS) (chunk loop) scrub_simple_mirror() if fs->flags sb_end_write() wait_on_bit fs->flags FREEZING (BLOCKS) (UNBLOCKS) ...Unfreezing is basically the same in reverse:
ioctl_fsthaw
fs->thaw_superthaw_super() unset fs->flags FREEZING (UNBLOCKS) sb_want_write() ...So if you see a problem in that please let me know, I've tested the patch and it worked but I might me missing something.
Quoted 29 lines · Qu Wenruo
Quoted 26 lines · David Sterba
Quoted 13 lines · Qu Wenruo
Quoted 5 lines · David Sterba
Implement
sb->freeze_superthat can instruct our threads to pause themselves. In case of (read-write) scrub this means to undomnt_want_write, implemented as sb_start_write()/sb_end_write(). Thefreeze_supercallback is necessary otherwise the callsb_want_write()inside the generic implementation hangs.I don't this this is really going to work.
The main problem is out of btrfs, it's all about the s_writer.rw_sem.
Quoted 8 lines
If we have a running scrub, it holds the
mnt_want_write_file(), which is a read lock on therw_sem.Then we start freezing, which will call
sb_wait_write(), which will do a write lock on therw_sem, waiting for the scrub to finish.However the ->freeze() callback is only called when
freeze_super()got the write lock on therw_sem.Note there are 2 callbacks for freeze,
sb::freeze_superandsb::freeze_fs.
Quoted 8 lines
Quoted 3 lines
ioctl_fsfreeze
iffs->freeze_super
callfs_freeze_super()Oh I forgot you implemented a new callback, ->freeze_super() to do the extra btrfs specific handling.
But this means, we're exposing ourselves to all the extra VFS handling,
I'm not sure if this is the preferred way, nor if the extra btrfs handling is good enough to cover what's done insidefreeze_super()function (I guess not though).
My bad, it's calling back into freeze_super() function, so it's fine.
The remaining concern is just should we do this by using ->freeze_super() callback, or do the f2fs' way by poking into s_writer.frozen.
I'd prefer the later one, as that's already exposed to filesystems and we do not need extra callback functions especially when it's not widely used.
Thanks,
Qu
Quoted 61 lines
The only user of ->freeze_super() callback is GFS2, and I didn't think we have enough locking/protection compared to either GFS2 nor VFS.
In that case, I still don't believe it's the correct way to go.
Add fsdevel list to see if it's even recommended to use ->freeze_super() callback.
Thanks,
QuQuoted 48 lines
else
freeze_super()
sb_wait_write()
if (sb->freeze_fs)
callsb->freeze_fs()What you describe matches the 'else' branch as this unconditionally blocks on the
sb_wait_writeand never gets to our callback. This is what I observed too.To fix that I've added the
sb::freeze_superso it can do something to avoid the lockup insb_wait_write(), because we must at some point callfreeze_super().Schematically it goes like this:
Freeze thread Scrub
btrfs_ioctl_scrubmnt_want_write_file(effectivelysb_want_write())ioctl_fsfreeze
fs->freeze_super
setfs->flagsFREEZING
freeze_super
sb_wait_freeze()
(BLOCKS)
(chunk loop)scrub_simple_mirror()iffs->flagssb_end_write()
wait_on_bitfs->flagsFREEZING (BLOCKS)
(UNBLOCKS)
...Unfreezing is basically the same in reverse:
ioctl_fsthaw
fs->thaw_super
thaw_super()
unsetfs->flagsFREEZING
(UNBLOCKS)sb_want_write()
...So if you see a problem in that please let me know, I've tested the patch and it worked but I might me missing something.
Christian Brauner
maintainerQuoted 39 lines · Qu Wenruo
Quoted 36 lines · Qu Wenruo
Quoted 23 lines · David Sterba
Quoted 13 lines · Qu Wenruo
Quoted 5 lines · David Sterba
Implement
sb->freeze_superthat can instruct our threads to pause themselves. In case of (read-write) scrub this means to undomnt_want_write, implemented as sb_start_write()/sb_end_write(). Thefreeze_supercallback is necessary otherwise the callsb_want_write()inside the generic implementation hangs.I don't this this is really going to work.
The main problem is out of btrfs, it's all about the s_writer.rw_sem.
Quoted 8 lines
If we have a running scrub, it holds the
mnt_want_write_file(), which is a read lock on therw_sem.Then we start freezing, which will call
sb_wait_write(), which will do a write lock on therw_sem, waiting for the scrub to finish.However the ->freeze() callback is only called when
freeze_super()got the write lock on therw_sem.Quoted 6 lines
Note there are 2 callbacks for freeze,
sb::freeze_superandsb::freeze_fs.
ioctl_fsfreeze
iffs->freeze_super
callfs_freeze_super()Oh I forgot you implemented a new callback, ->freeze_super() to do the extra btrfs specific handling.
Quoted 6 lines
Quoted 4 lines
But this means, we're exposing ourselves to all the extra VFS handling, I'm not sure if this is the preferred way, nor if the extra btrfs handling is good enough to cover what's done inside
freeze_super()function (I guess not though).My bad, it's calling back into
freeze_super()function, so it's fine.
You have to expose yourself to all the VFS handling and you must call freeze_super() at some point. You can't just do freeze on your own. You have roughly the following classes of freeze calls:
(1) initiated from userspace
(1.1) initiated from the block layer => upcall into the filesystem (1.2) initiated from the filesystem
(2) initiated from the kernel
(2.1) initiated from the filesystem
(2.2) suspend/hibernation
All of that requires synchronization and you cannot do this without going through the VFS layer's locking safely. At least not without causing royal pain for yourself and the block and VFS layer.
(Fwiw, I would still really like to see support for fs_holder_ops so btrfs freezing works correctly when called from the block layer.)
Quoted 7 lines
The remaining concern is just should we do this by using ->freeze_super() callback, or do the f2fs' way by poking into s_writer.frozen.
I'd prefer the later one, as that's already exposed to filesystems and we do not need extra callback functions especially when it's not widely used.
This would be simpler. You should probably add a helper that you can share with gfs2 to sample the state instead of raw accesses to s_writers->frozen. I find the latter a bit ugly and error prone if we ever change this. I'd rather have it encapsulated in the VFS layer as best as we can.
Quoted 13 lines · Christian Brauner
Quoted 1 line
My bad, it's calling back into
freeze_super()function, so it's fine.You have to expose yourself to all the VFS handling and you must call
freeze_super()at some point. You can't just do freeze on your own. You have roughly the following classes of freeze calls:(1) initiated from userspace
(1.1) initiated from the block layer => upcall into the filesystem (1.2) initiated from the filesystem
(2) initiated from the kernel
(2.1) initiated from the filesystem
(2.2) suspend/hibernation
Quoted 3 lines
All of that requires synchronization and you cannot do this without going through the VFS layer's locking safely. At least not without causing royal pain for yourself and the block and VFS layer.
You weren't CCed on the initial patch
https://lore.kernel.org/linux-btrfs/[email protected]/
so you can see we're not avoiding VFS, freeze_super() is called.
The problem I am solving is that we need to pass the information about freezing being started before freeze_super() is called, or alternatively by something that is set there before sb_wait_write() call.
Quoted 4 lines
Quoted 4 lines
The remaining concern is just should we do this by using ->freeze_super() callback, or do the f2fs' way by poking into s_writer.frozen.
Quoted 8 lines
Quoted 2 lines
I'd prefer the later one, as that's already exposed to filesystems and we do not need extra callback functions especially when it's not widely used.
This would be simpler. You should probably add a helper that you can share with gfs2 to sample the state instead of raw accesses to
s_writers->frozen. I find the latter a bit ugly and error prone if we ever change this. I'd rather have it encapsulated in the VFS layer as best as we can.
I chose to use separate bit rather than using s_writers->frozen, which is what Qu prefers. I disgree with that on the API level as we probably should not do such direct check. A helper would be best, semantically teh same as the added filesystem bit.
Quoted 1 line · David Sterba
On Mon, Jul 14, 2025 at 12:27:37PM +0200, Christian Brauner wrote:
[...]
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
The remaining concern is just should we do this by using ->freeze_super() callback, or do the f2fs' way by poking into s_writer.frozen.
Quoted 8 lines
Quoted 2 lines
I'd prefer the later one, as that's already exposed to filesystems and we do not need extra callback functions especially when it's not widely used.
This would be simpler. You should probably add a helper that you can share with gfs2 to sample the state instead of raw accesses to
s_writers->frozen. I find the latter a bit ugly and error prone if we ever change this. I'd rather have it encapsulated in the VFS layer as best as we can.
Quoted 4 lines
I chose to use separate bit rather than using
s_writers->frozen, which is what Qu prefers. I disgree with that on the API level as we probably should not do such direct check. A helper would be best, semantically teh same as the added filesystem bit.
My biggest complain about the new ->freeze_super() callback is, it's very confusing to have both ->freeze_super() and ->freeze_fs().
Currently only GFS2 is utilizing that, and I'm not sure if it can be removed or not, as it's some gfs specific lock handling.
In our btrfs case, it's really just a single bit that can also be done by checking s_writer.frozen, thus I don't think doing the extra callbacks just for that bit is really worthy.
Talking about the extra helpers to handle the access, it's valid, although we may have several different helpers, as each fs has different checks. This makes the helper almost the same to the raw access to s_writers->frozen.
E.g. f2fs' gc and quota check if the fs is being/already frozen, thus it's the same as the btrfs's one.
But there are other call sites checking if the freezing is fully completed, and xfs even checks on the specific frozen stage.
So if one day we really change the s_writer->frozen stages, it will touch quite some fs codes anyway.
Thanks,
Qu
I just tested this patch on real hardware (laptop), and it didn't work.
This is what I did:
- I have btrfs on my root filesystem
- It is btrfs-raid with 2 copies of everything
- I started scrub, and in the same time did suspend
Suspend didn't work.
I did experiment twice: with /sys/power/freeze_filesystems set to 0 and to 1. In both cases suspend didn't work.
( /sys/power/freeze_filesystems was introduced in
https://lore.kernel.org/linux-fsdevel/[email protected]/
)
--
Askar Safin