fs: Prevent syncing frozen file system
RFC PATCH 1/1 fs: Prevent syncing frozen file system
Currently we can end up in a deadlock because of broken sb_start_write -> s_umount ordering.
The race goes like this:
- write the file
- unlink the file - final_iput will not be calles as file is opened
- freeze the file system
- Now simultaneously close the file and call sync (or syncfs on that particular file system). Sync will get to wait_sb_inodes() where it will grab the referece to the inode (__iget()) and later to call iput(). If we manage to close the file and drop the reference in between those calls sync will attempt to do a iput_final() because the inode is now unlinked and we're holding the last reference to it. This will however block on a frozen file system (ext4_delete_inode for example).
Note that I've not been able to reproduce the issue, I've only seen this happen once. However with some instrumentation (like msleep() in the wait_sb_inodes() it can be achieved.
Fix this by properly doing sb_start_write/sb_end_write to prevent us from fsfreeze.
Note that with this patch syncfs will block on the frozen file system which is probably ok, but sync will block if any file system happens to be frozen - not sure if that's a problem, but it's certainly different from what we've been used to.
--- fs/super.c | 7 +++++++ fs/sync.c | 8 ++++++++ 2 files changed, 15 insertions(+)
fs/super.c +7 -0
@@ -514,10 +514,17 @@ void iterate_supers(void (*f)(struct super_block *, void *), void *arg) sb->s_count++;spin_unlock (&sb_lock); +/* +* Whatever we're going to do to the file system we have to +* make sure that we'll not end up blocking on frozen file +* system. +*/ +sb_start_write (sb);down_read (&sb->s_umount);if (sb->s_root && (sb->s_flags &MS_BORN ))f (sb, arg);up_read (&sb->s_umount); +sb_end_write (sb);spin_lock (&sb_lock);if (p)
fs/sync.c +8 -0
@@ -156,9 +156,17 @@ SYSCALL_DEFINE1(syncfs, int, fd)return -EBADF ; sb = f.file->f_path.dentry->d_sb; +/* +* If the file system is frozen we can't proceed because we +* could potentially block on frozen file system. This would +* lead to a deadlock, because we'll be holding s_umount which +* has to be taken in order to thaw the file system as well +*/ +sb_start_write (sb);down_read (&sb->s_umount); ret =sync_filesystem (sb);up_read (&sb->s_umount); +sb_end_write (sb);fdput (f);return ret;
Signature · 1 line
1.8.3.1
Quoted 11 lines · Lukas Czerner
Currently we can end up in a deadlock because of broken
sb_start_write->s_umountordering.The race goes like this:
- write the file
- unlink the file -final_iputwill not be calles as file is opened
- freeze the file system
- Now simultaneously close the file and call sync (or syncfs on that particular file system). Sync will get towait_sb_inodes()where it will grab the referece to the inode (__iget()) and later to calliput().
This problem goes away with the sync scalability patchset that josef has been trying to get merged:
git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git superblock-scaling
That patchset removes the full sb inodes list walk in wait_sb_inodes() and replaces it with a walk of inodes cleaned during the sync, which will be an empty list in the case of sync running on an empty filesystem. This commit does the work:
<As a separate note, can we *please* get that patchset merged given that there are now several outstanding issues that it fixes in one go?>
Quoted 10 lines
If we manage to close the file and drop the reference in between those calls sync will attempt to do a
iput_final()because the inode is now unlinked and we're holding the last reference to it. This will however block on a frozen file system (ext4_delete_inodefor example).Note that I've not been able to reproduce the issue, I've only seen this happen once. However with some instrumentation (like
msleep()in thewait_sb_inodes()it can be achieved.
Quoted 7 lines
Fix this by properly doing sb_start_write/sb_end_write to prevent us from fsfreeze.
Note that with this patch syncfs will block on the frozen file system which is probably ok, but sync will block if any file system happens to be frozen - not sure if that's a problem, but it's certainly different from what we've been used to.
sync should not block on frozen fileystems. By definition, a frozen filesystem is a clean filesystem, and so sync should really just be skipping over them.
Quoted 9 lines
+++
b/fs/super.cfs/super.c +4
-0@@ -514,10 +514,17 @@ void iterate_supers(void (*f)(struct super_block *, void *), void *arg) sb->s_count++;spin_unlock (&sb_lock); +/* +* Whatever we're going to do to the file system we have to +* make sure that we'll not end up blocking on frozen file +* system.
fs/super.c void iterate_supers(void (*f)(struct super_block *, void *), void *arg) · line 530
Quoted 10 lines
fs/super.c +3
-0+*/ +sb_start_write (sb);down_read (&sb->s_umount);if (sb->s_root && (sb->s_flags &MS_BORN ))f (sb, arg);up_read (&sb->s_umount); +sb_end_write (sb);spin_lock (&sb_lock);if (p)
That deadlocks sysrq-j (emergency thaw)...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 16 lines · Dave Chinner
Quoted 11 lines · Lukas Czerner
Currently we can end up in a deadlock because of broken
sb_start_write->s_umountordering.The race goes like this:
- write the file
- unlink the file -final_iputwill not be calles as file is opened
- freeze the file system
- Now simultaneously close the file and call sync (or syncfs on that particular file system). Sync will get towait_sb_inodes()where it will grab the referece to the inode (__iget()) and later to calliput().This problem goes away with the sync scalability patchset that josef has been trying to get merged:
Quoted 6 lines
git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.gitsuperblock-scalingThat patchset removes the full sb inodes list walk in
wait_sb_inodes()and replaces it with a walk of inodes cleaned during the sync, which will be an empty list in the case of sync running on an empty filesystem. This commit does the work:
s/empty/frozen
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Jan Kara
reviewerQuoted 23 lines · Dave Chinner
Quoted 11 lines · Lukas Czerner
Currently we can end up in a deadlock because of broken
sb_start_write->s_umountordering.The race goes like this:
- write the file
- unlink the file -final_iputwill not be calles as file is opened
- freeze the file system
- Now simultaneously close the file and call sync (or syncfs on that particular file system). Sync will get towait_sb_inodes()where it will grab the referece to the inode (__iget()) and later to calliput().This problem goes away with the sync scalability patchset that josef has been trying to get merged:
git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.gitsuperblock-scalingThat patchset removes the full sb inodes list walk in
wait_sb_inodes()and replaces it with a walk of inodes cleaned during the sync, which will be an empty list in the case of sync running on an empty filesystem. This commit does the work:
Quoted 5 lines
<As a separate note, can we *please* get that patchset merged given that there are now several outstanding issues that it fixes in one go?>
Not sure where that got stuck - oh, maybe on Tejun's memcg writeback series which was clashing with it. Josef?
Quoted 13 lines
Quoted 13 lines
If we manage to close the file and drop the reference in between those calls sync will attempt to do a
iput_final()because the inode is now unlinked and we're holding the last reference to it. This will however block on a frozen file system (ext4_delete_inodefor example).Note that I've not been able to reproduce the issue, I've only seen this happen once. However with some instrumentation (like
msleep()in thewait_sb_inodes()it can be achieved.Fix this by properly doing sb_start_write/sb_end_write to prevent us from fsfreeze.
Quoted 8 lines
Quoted 4 lines
Note that with this patch syncfs will block on the frozen file system which is probably ok, but sync will block if any file system happens to be frozen - not sure if that's a problem, but it's certainly different from what we've been used to.
sync should not block on frozen fileystems. By definition, a frozen filesystem is a clean filesystem, and so sync should really just be skipping over them.
Just for record I agree with Dave. Sync on frozen fs should just return. And freeze protection in iterate_supers() looks just wrong.
Honza
Quoted 32 lines
Quoted 9 lines
+++
b/fs/super.cfs/super.c +4
-0@@ -514,10 +514,17 @@ void iterate_supers(void (*f)(struct super_block *, void *), void *arg) sb->s_count++;spin_unlock (&sb_lock); +/* +* Whatever we're going to do to the file system we have to +* make sure that we'll not end up blocking on frozen file +* system. fs/super.c void iterate_supers(void (*f)(struct super_block *, void *), void *arg) · line 530
Quoted 10 lines
fs/super.c +3
-0+*/ +sb_start_write (sb);down_read (&sb->s_umount);if (sb->s_root && (sb->s_flags &MS_BORN ))f (sb, arg);up_read (&sb->s_umount); +sb_end_write (sb);spin_lock (&sb_lock);if (p)That deadlocks sysrq-j (emergency thaw)...
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 59 lines · Jan Kara
Date: Fri, 10 Jul 2015 16:25:25 +0200
From: Jan Kara <[email protected]>
To: Dave Chinner <[email protected]>
Cc: Lukas Czerner <[email protected]>, [email protected],
[email protected], [email protected], [email protected]
Subject: Re: [RFC][PATCH] fs: Prevent syncing frozen file systemQuoted 23 lines · Dave Chinner
Quoted 11 lines · Lukas Czerner
Currently we can end up in a deadlock because of broken
sb_start_write->s_umountordering.The race goes like this:
- write the file
- unlink the file -final_iputwill not be calles as file is opened
- freeze the file system
- Now simultaneously close the file and call sync (or syncfs on that particular file system). Sync will get towait_sb_inodes()where it will grab the referece to the inode (__iget()) and later to calliput().This problem goes away with the sync scalability patchset that josef has been trying to get merged:
git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.gitsuperblock-scalingThat patchset removes the full sb inodes list walk in
wait_sb_inodes()and replaces it with a walk of inodes cleaned during the sync, which will be an empty list in the case of sync running on an empty filesystem. This commit does the work:Quoted 5 lines
<As a separate note, can we *please* get that patchset merged given that there are now several outstanding issues that it fixes in one go?>
Not sure where that got stuck - oh, maybe on Tejun's memcg writeback series which was clashing with it. Josef?
Quoted 18 lines
Quoted 17 lines
If we manage to close the file and drop the reference in between those calls sync will attempt to do a
iput_final()because the inode is now unlinked and we're holding the last reference to it. This will however block on a frozen file system (ext4_delete_inodefor example).Note that I've not been able to reproduce the issue, I've only seen this happen once. However with some instrumentation (like
msleep()in thewait_sb_inodes()it can be achieved.Fix this by properly doing sb_start_write/sb_end_write to prevent us from fsfreeze.
Note that with this patch syncfs will block on the frozen file system which is probably ok, but sync will block if any file system happens to be frozen - not sure if that's a problem, but it's certainly different from what we've been used to.
Quoted 6 lines
Quoted 3 lines
sync should not block on frozen fileystems. By definition, a frozen filesystem is a clean filesystem, and so sync should really just be skipping over them.
Just for record I agree with Dave. Sync on frozen fs should just return. And freeze protection in
iterate_supers()looks just wrong.
Sure, that's why it's rfc. Anyway with the change Dave mentioned the deadlock should not be possible anymore. However anywhere where we take s_umount before sb_start_write we could deadlock, so it might be worth adding a warning into sb_start_write() maybe ?
-Lukas
Quoted 36 lines
Honza
Quoted 32 lines
Quoted 9 lines
+++
b/fs/super.cfs/super.c +4
-0@@ -514,10 +514,17 @@ void iterate_supers(void (*f)(struct super_block *, void *), void *arg) sb->s_count++;spin_unlock (&sb_lock); +/* +* Whatever we're going to do to the file system we have to +* make sure that we'll not end up blocking on frozen file +* system. fs/super.c void iterate_supers(void (*f)(struct super_block *, void *), void *arg) · line 530
Quoted 10 lines
fs/super.c +3
-0+*/ +sb_start_write (sb);down_read (&sb->s_umount);if (sb->s_root && (sb->s_flags &MS_BORN ))f (sb, arg);up_read (&sb->s_umount); +sb_end_write (sb);spin_lock (&sb_lock);if (p)That deadlocks sysrq-j (emergency thaw)...
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Jan Kara
reviewerQuoted 65 lines · Lukáš Czerner
Quoted 63 lines · Jan Kara
Date: Fri, 10 Jul 2015 16:25:25 +0200
From: Jan Kara <[email protected]>
To: Dave Chinner <[email protected]>
Cc: Lukas Czerner <[email protected]>, [email protected],
[email protected], [email protected], [email protected]
Subject: Re: [RFC][PATCH] fs: Prevent syncing frozen file systemQuoted 23 lines · Dave Chinner
Quoted 11 lines · Lukas Czerner
Currently we can end up in a deadlock because of broken
sb_start_write->s_umountordering.The race goes like this:
- write the file
- unlink the file -final_iputwill not be calles as file is opened
- freeze the file system
- Now simultaneously close the file and call sync (or syncfs on that particular file system). Sync will get towait_sb_inodes()where it will grab the referece to the inode (__iget()) and later to calliput().This problem goes away with the sync scalability patchset that josef has been trying to get merged:
git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.gitsuperblock-scalingThat patchset removes the full sb inodes list walk in
wait_sb_inodes()and replaces it with a walk of inodes cleaned during the sync, which will be an empty list in the case of sync running on an empty filesystem. This commit does the work:Quoted 5 lines
<As a separate note, can we *please* get that patchset merged given that there are now several outstanding issues that it fixes in one go?>
Not sure where that got stuck - oh, maybe on Tejun's memcg writeback series which was clashing with it. Josef?
Quoted 21 lines
Quoted 10 lines
If we manage to close the file and drop the reference in between those calls sync will attempt to do a
iput_final()because the inode is now unlinked and we're holding the last reference to it. This will however block on a frozen file system (ext4_delete_inodefor example).Note that I've not been able to reproduce the issue, I've only seen this happen once. However with some instrumentation (like
msleep()in thewait_sb_inodes()it can be achieved.Quoted 7 lines
Fix this by properly doing sb_start_write/sb_end_write to prevent us from fsfreeze.
Note that with this patch syncfs will block on the frozen file system which is probably ok, but sync will block if any file system happens to be frozen - not sure if that's a problem, but it's certainly different from what we've been used to.
sync should not block on frozen fileystems. By definition, a frozen filesystem is a clean filesystem, and so sync should really just be skipping over them.
Quoted 7 lines
Quoted 2 lines
Just for record I agree with Dave. Sync on frozen fs should just return. And freeze protection in
iterate_supers()looks just wrong.Sure, that's why it's rfc. Anyway with the change Dave mentioned the deadlock should not be possible anymore. However anywhere where we take
s_umountbeforesb_start_writewe could deadlock, so it might be worth adding a warning intosb_start_write()maybe ?
Lockdep should warn you about that since fs freezing is properly hooked into it...
Honza
Quoted 33 lines
Quoted 33 lines
Quoted 32 lines
Quoted 9 lines
+++
b/fs/super.cfs/super.c +4
-0@@ -514,10 +514,17 @@ void iterate_supers(void (*f)(struct super_block *, void *), void *arg) sb->s_count++;spin_unlock (&sb_lock); +/* +* Whatever we're going to do to the file system we have to +* make sure that we'll not end up blocking on frozen file +* system. fs/super.c void iterate_supers(void (*f)(struct super_block *, void *), void *arg) · line 530
Quoted 10 lines
fs/super.c +3
-0+*/ +sb_start_write (sb);down_read (&sb->s_umount);if (sb->s_root && (sb->s_flags &MS_BORN ))f (sb, arg);up_read (&sb->s_umount); +sb_end_write (sb);spin_lock (&sb_lock);if (p)That deadlocks sysrq-j (emergency thaw)...
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR