vfs: have syncfs() return error when there are writeback errors
Subsystems (4)
v5 cover vfs: have syncfs() return error when there are writeback errors
v5:
- use RCU to ensure that bd_super doesn't go away while we're using it
This is the fifth iteration of this patchset. The main difference from v4 is that this one uses RCU to ensure validity of the bd_super pointer when we're marking it after a buffer_head writeback error.
Jeff Layton (2):
vfs: track per-sb writeback errors and report them to syncfs
buffer: record blockdev write errors in super_block that it backs
drivers/dax/device.c | 1 + fs/buffer.c | 7 +++++++ fs/file_table.c | 1 + fs/open.c | 3 +-- fs/sync.c | 6 ++++-- include/linux/fs.h | 16 ++++++++++++++++ include/linux/pagemap.h | 5 ++++- 7 files changed, 34 insertions(+), 5 deletions(-)
Signature · 1 line
2.25.2
PATCH v5 1/2 vfs: track per-sb writeback errors and report them to syncfs
From: Jeff Layton <[email protected]>
Usually we suggest that applications call fsync when they want to ensure that all data written to the file has made it to the backing store, but that can be inefficient when there are a lot of open files.
Calling syncfs on the filesystem can be more efficient in some situations, but the error reporting doesn't currently work the way most people expect. If a single inode on a filesystem reports a writeback error, syncfs won't necessarily return an error. syncfs only returns an error if __sync_blockdev fails, and on some filesystems that's a no-op.
It would be better if syncfs reported an error if there were any writeback failures. Then applications could call syncfs to see if there are any errors on any open files, and could then call fsync on all of the other descriptors to figure out which one failed.
This patch adds a new errseq_t to struct super_block, and has mapping_set_error also record writeback errors there.
To report those errors, we also need to keep an errseq_t in struct file to act as a cursor. This patch adds a dedicated field for that purpose, which slots nicely into 4 bytes of padding at the end of struct file on x86_64.
An earlier version of this patch used an O_PATH file descriptor to cue the kernel that the open file should track the superblock error and not the inode's writeback error.
I think that API is just too weird though. This is simpler and should make syncfs error reporting "just work" even if someone is multiplexing fsync and syncfs on the same fds.
--- drivers/dax/device.c | 1 + fs/file_table.c | 1 + fs/open.c | 3 +-- fs/sync.c | 6 ++++-- include/linux/fs.h | 16 ++++++++++++++++ include/linux/pagemap.h | 5 ++++- 6 files changed, 27 insertions(+), 5 deletions(-)
drivers/dax/device.c +1 -0
@@ -377,6 +377,7 @@ static int dax_open(struct inode *inode, struct file *filp) inode->i_mapping->a_ops = &dev_dax_aops; filp->f_mapping = inode->i_mapping; filp->f_wb_err =filemap_sample_wb_err (filp->f_mapping); + filp->f_sb_err =file_sample_sb_err (filp); filp->private_data = dev_dax; inode->i_flags =S_DAX ;
fs/file_table.c +1 -0
@@ -198,6 +198,7 @@ static struct file *alloc_file(const struct path *path, int flags, file->f_inode = path->dentry->d_inode; file->f_mapping = path->dentry->d_inode->i_mapping; file->f_wb_err =filemap_sample_wb_err (file->f_mapping); + file->f_sb_err =file_sample_sb_err (file);if ((file->f_mode &FMODE_READ ) &&likely (fop->read || fop->read_iter)) file->f_mode |=FMODE_CAN_READ ;
fs/open.c +1 -2
@@ -743,9 +743,8 @@ static int do_dentry_open(struct file *f,path_get (&f->f_path); f->f_inode = inode; f->f_mapping = inode->i_mapping; - -/* Ensure that we skip any errors that predate opening of the file */ f->f_wb_err =filemap_sample_wb_err (f->f_mapping); + f->f_sb_err =file_sample_sb_err (f);if (unlikely (f->f_flags &O_PATH )) { f->f_mode =FMODE_PATH |FMODE_OPENED ;
fs/sync.c +4 -2
@@ -161,7 +161,7 @@ SYSCALL_DEFINE1(syncfs, int, fd) {struct fd f =fdget (fd);struct super_block *sb; -int ret; +int ret, ret2;if (!f.file)return -EBADF ;
@@ -171,8 +171,10 @@ SYSCALL_DEFINE1(syncfs, int, fd) ret =sync_filesystem (sb);up_read (&sb->s_umount); + ret2 =errseq_check_and_advance (&sb->s_wb_err, &f.file->f_sb_err); +fdput (f); -return ret; +return ret ? ret : ret2; }/**
include/linux/fs.h +16 -0
@@ -976,6 +976,7 @@ struct file {#endif /* #ifdef CONFIG_EPOLL */ struct address_space *f_mapping;errseq_t f_wb_err; +errseq_t f_sb_err;/* for syncfs */ }__randomize_layout __attribute__((aligned(4))); /* lest something weird decides that 2 is OK */
@@ -1520,6 +1521,9 @@ struct super_block {/* Being remounted read-only */ int s_readonly_remount; +/* per-sb errseq_t for reporting writeback errors via syncfs */ +errseq_t s_wb_err; +/* AIO completions deferred from interrupt context */ struct workqueue_struct *s_dio_done_wq;struct hlist_head s_pins;
@@ -2827,6 +2831,18 @@ static inline errseq_t filemap_sample_wb_err(struct address_space *mapping)return errseq_sample (&mapping->wb_err); } +/** +* file_sample_sb_err - sample the current errseq_t to test for later errors +* @mapping: mapping to be sampled +* +* Grab the most current superblock-level errseq_t value for the given +* struct file. +*/ +static inline errseq_t file_sample_sb_err (struct file *file) +{ +return errseq_sample (&file->f_path.dentry->d_sb->s_wb_err); +} +static inline int filemap_nr_thps (struct address_space *mapping) {#ifdef CONFIG_READ_ONLY_THP_FOR_FS
include/linux/pagemap.h +4 -1
@@ -51,7 +51,10 @@ static inline void mapping_set_error(struct address_space *mapping, int error)return ;/* Record in wb_err for checkers using errseq_t based tracking */ -filemap_set_wb_err (mapping, error); +__filemap_set_wb_err (mapping, error); + +/* Record it in superblock */ +errseq_set (&mapping->host->i_sb->s_wb_err, error);/* Record it in flags for now, for legacy callers */ if (error == -ENOSPC )
Signature · 1 line
2.25.2
PATCH v5 2/2 buffer: record blockdev write errors in super_block that it backs
In reply to Jeff LaytonFrom: Jeff Layton <[email protected]>
When syncing out a block device (a'la __sync_blockdev), any error encountered will only be recorded in the bd_inode's mapping. When the blockdev contains a filesystem however, we'd like to also record the error in the super_block that's stored there.
Make mark_buffer_write_io_error also record the error in the corresponding super_block when a writeback error occurs and the block device contains a mounted superblock.
Since superblocks are RCU freed, hold the rcu_read_lock to ensure that the superblock doesn't go away while we're marking it.
--- fs/buffer.c | 7 +++++++ 1 file changed, 7 insertions(+)
fs/buffer.c +7 -0
@@ -1154,12 +1154,19 @@ EXPORT_SYMBOL(mark_buffer_dirty);void mark_buffer_write_io_error (struct buffer_head *bh) { +struct super_block *sb; +set_buffer_write_io_error (bh);/* FIXME: do we need to set this in both places? */ if (bh->b_page && bh->b_page->mapping)mapping_set_error (bh->b_page->mapping, -EIO );if (bh->b_assoc_map)mapping_set_error (bh->b_assoc_map, -EIO ); +rcu_read_lock (); + sb = bh->b_bdev->bd_super; +if (sb) +errseq_set (&sb->s_wb_err, -EIO ); +rcu_read_unlock (); }EXPORT_SYMBOL (mark_buffer_write_io_error);
Signature · 1 line
2.25.2
Jan Kara
maintainerRe: [PATCH v5 2/2] buffer: record blockdev write errors in super_block that it backs
Quoted 27 lines · Jeff Layton
From: Jeff Layton <[email protected]>
When syncing out a block device (a'la
__sync_blockdev), any error encountered will only be recorded in the bd_inode's mapping. When the blockdev contains a filesystem however, we'd like to also record the error in thesuper_blockthat's stored there.Make
mark_buffer_write_io_erroralso record the error in the correspondingsuper_blockwhen a writeback error occurs and the block device contains a mounted superblock.Since superblocks are RCU freed, hold the
rcu_read_lockto ensure that the superblock doesn't go away while we're marking it.Signed-off-by Jeff Layton <[email protected]>--- fs/buffer.c | 7 +++++++ 1 file changed, 7 insertions(+)fs/buffer.c +0
-0@@ -1154,12 +1154,19 @@ EXPORT_SYMBOL(mark_buffer_dirty);void mark_buffer_write_io_error (struct buffer_head *bh) {
fs/buffer.c EXPORT_SYMBOL(mark_buffer_dirty); · line 1166
Quoted 10 lines
fs/buffer.c +4
-0+struct super_block *sb; +set_buffer_write_io_error (bh);/* FIXME: do we need to set this in both places? */ if (bh->b_page && bh->b_page->mapping)mapping_set_error (bh->b_page->mapping, -EIO );if (bh->b_assoc_map)mapping_set_error (bh->b_assoc_map, -EIO ); +rcu_read_lock (); + sb = bh->b_bdev->bd_super;
You still need READ_ONCE() here. Otherwise the dereference below can still result in refetch and NULL ptr deref.
Honza
Quoted 9 lines
fs/buffer.c +3
-0+if (sb) +errseq_set (&sb->s_wb_err, -EIO ); +rcu_read_unlock (); }EXPORT_SYMBOL (mark_buffer_write_io_error);Signature · 1 line
2.25.2
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH v5 2/2] buffer: record blockdev write errors in super_block that it backs
Quoted 42 lines · Jan Kara
Quoted 27 lines · Jeff Layton
From: Jeff Layton <[email protected]>
When syncing out a block device (a'la
__sync_blockdev), any error encountered will only be recorded in the bd_inode's mapping. When the blockdev contains a filesystem however, we'd like to also record the error in thesuper_blockthat's stored there.Make
mark_buffer_write_io_erroralso record the error in the correspondingsuper_blockwhen a writeback error occurs and the block device contains a mounted superblock.Since superblocks are RCU freed, hold the
rcu_read_lockto ensure that the superblock doesn't go away while we're marking it.Signed-off-by Jeff Layton <[email protected]>--- fs/buffer.c | 7 +++++++ 1 file changed, 7 insertions(+)fs/buffer.c +0
-0@@ -1154,12 +1154,19 @@ EXPORT_SYMBOL(mark_buffer_dirty);void mark_buffer_write_io_error (struct buffer_head *bh) {fs/buffer.c EXPORT_SYMBOL(mark_buffer_dirty); · line 1166
Quoted 10 lines
fs/buffer.c +4
-0+struct super_block *sb; +set_buffer_write_io_error (bh);/* FIXME: do we need to set this in both places? */ if (bh->b_page && bh->b_page->mapping)mapping_set_error (bh->b_page->mapping, -EIO );if (bh->b_assoc_map)mapping_set_error (bh->b_assoc_map, -EIO ); +rcu_read_lock (); + sb = bh->b_bdev->bd_super;You still need
READ_ONCE()here. Otherwise the dereference below can still result in refetch and NULL ptr deref.
Quoted 2 lines
Honza
Huh? That seems like a really suspicious thing for the compiler/arch to do. We are checking that sb isn't NULL before we dereference it. Doesn't that imply a data dependency? How could the value of "sb" change after that?
I'm also not sure I understand how using READ_ONCE really helps there if we can't count on the value of a local variable not changing.
Quoted 9 lines
Quoted 9 lines
fs/buffer.c +3
-0+if (sb) +errseq_set (&sb->s_wb_err, -EIO ); +rcu_read_unlock (); }EXPORT_SYMBOL (mark_buffer_write_io_error);Signature · 1 line
2.25.2
Signature · 1 line
Jeff Layton <[email protected]>
Jan Kara
maintainerRe: [PATCH v5 2/2] buffer: record blockdev write errors in super_block that it backs
Quoted 43 lines · Jeff Layton
Quoted 42 lines · Jan Kara
Quoted 27 lines · Jeff Layton
From: Jeff Layton <[email protected]>
When syncing out a block device (a'la
__sync_blockdev), any error encountered will only be recorded in the bd_inode's mapping. When the blockdev contains a filesystem however, we'd like to also record the error in thesuper_blockthat's stored there.Make
mark_buffer_write_io_erroralso record the error in the correspondingsuper_blockwhen a writeback error occurs and the block device contains a mounted superblock.Since superblocks are RCU freed, hold the
rcu_read_lockto ensure that the superblock doesn't go away while we're marking it.Signed-off-by Jeff Layton <[email protected]>--- fs/buffer.c | 7 +++++++ 1 file changed, 7 insertions(+)fs/buffer.c +0
-0@@ -1154,12 +1154,19 @@ EXPORT_SYMBOL(mark_buffer_dirty);void mark_buffer_write_io_error (struct buffer_head *bh) {fs/buffer.c EXPORT_SYMBOL(mark_buffer_dirty); · line 1166
Quoted 10 lines
fs/buffer.c +4
-0+struct super_block *sb; +set_buffer_write_io_error (bh);/* FIXME: do we need to set this in both places? */ if (bh->b_page && bh->b_page->mapping)mapping_set_error (bh->b_page->mapping, -EIO );if (bh->b_assoc_map)mapping_set_error (bh->b_assoc_map, -EIO ); +rcu_read_lock (); + sb = bh->b_bdev->bd_super;You still need
READ_ONCE()here. Otherwise the dereference below can still result in refetch and NULL ptr deref.
Quoted 7 lines
Quoted 2 lines
Honza
Huh? That seems like a really suspicious thing for the compiler/arch to do. We are checking that sb isn't NULL before we dereference it. Doesn't that imply a data dependency? How could the value of "sb" change after that?
Because the compiler is free to optimize the local variable away and actually compile the dereference below as bh->b_bdev->bd_super->s_wb_err (from C11 standard POV such code is equivalent since in C11 memory model it is assumed there are no concurrent accesses). And READ_ONCE() is a way to forbid compiler from doing such optimization - through 'volatile' keyword it tells the compiler there may be concurrent accesses happening and makes sure the value is really fetched into the local variable and used from there. There are good articles about this on LWN - I'd give you a link but LWN seems to be down today. But the latest article is about KCSAN and from there are links to older articles about compiler optimizations.
Quoted 2 lines
I'm also not sure I understand how using
READ_ONCEreally helps there if we can't count on the value of a local variable not changing.
I hope I've explained this above.
Honza
Quoted 13 lines
Quoted 9 lines
Quoted 9 lines
fs/buffer.c +3
-0+if (sb) +errseq_set (&sb->s_wb_err, -EIO ); +rcu_read_unlock (); }EXPORT_SYMBOL (mark_buffer_write_io_error);Signature · 1 line
2.25.2Signature · 1 line
Jeff Layton <[email protected]>
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH v5 2/2] buffer: record blockdev write errors in super_block that it backs
Quoted 63 lines · Jan Kara
Quoted 43 lines · Jeff Layton
Quoted 42 lines · Jan Kara
Quoted 27 lines · Jeff Layton
From: Jeff Layton <[email protected]>
When syncing out a block device (a'la
__sync_blockdev), any error encountered will only be recorded in the bd_inode's mapping. When the blockdev contains a filesystem however, we'd like to also record the error in thesuper_blockthat's stored there.Make
mark_buffer_write_io_erroralso record the error in the correspondingsuper_blockwhen a writeback error occurs and the block device contains a mounted superblock.Since superblocks are RCU freed, hold the
rcu_read_lockto ensure that the superblock doesn't go away while we're marking it.Signed-off-by Jeff Layton <[email protected]>--- fs/buffer.c | 7 +++++++ 1 file changed, 7 insertions(+)fs/buffer.c +0
-0@@ -1154,12 +1154,19 @@ EXPORT_SYMBOL(mark_buffer_dirty);void mark_buffer_write_io_error (struct buffer_head *bh) {fs/buffer.c EXPORT_SYMBOL(mark_buffer_dirty); · line 1166
Quoted 10 lines
fs/buffer.c +4
-0+struct super_block *sb; +set_buffer_write_io_error (bh);/* FIXME: do we need to set this in both places? */ if (bh->b_page && bh->b_page->mapping)mapping_set_error (bh->b_page->mapping, -EIO );if (bh->b_assoc_map)mapping_set_error (bh->b_assoc_map, -EIO ); +rcu_read_lock (); + sb = bh->b_bdev->bd_super;You still need
READ_ONCE()here. Otherwise the dereference below can still result in refetch and NULL ptr deref.Quoted 7 lines
Quoted 2 lines
Honza
Huh? That seems like a really suspicious thing for the compiler/arch to do. We are checking that sb isn't NULL before we dereference it. Doesn't that imply a data dependency? How could the value of "sb" change after that?
Because the compiler is free to optimize the local variable away and actually compile the dereference below as
bh->b_bdev->bd_super->s_wb_err(from C11 standard POV such code is equivalent since in C11 memory model it is assumed there are no concurrent accesses). AndREAD_ONCE()is a way to forbid compiler from doing such optimization - through 'volatile' keyword it tells the compiler there may be concurrent accesses happening and makes sure the value is really fetched into the local variable and used from there. There are good articles about this on LWN - I'd give you a link but LWN seems to be down today. But the latest article is about KCSAN and from there are links to older articles about compiler optimizations.
Quoted 5 lines
Quoted 2 lines
I'm also not sure I understand how using
READ_ONCEreally helps there if we can't count on the value of a local variable not changing.I hope I've explained this above.
Got it. Thanks for the explanation. Now I'll have nightmares about all of the race conditions I've created in the past by making this assumption!
I'll send a v6 set in a few mins.
Quoted 13 lines
Quoted 13 lines
Quoted 9 lines
Quoted 9 lines
fs/buffer.c +3
-0+if (sb) +errseq_set (&sb->s_wb_err, -EIO ); +rcu_read_unlock (); }EXPORT_SYMBOL (mark_buffer_write_io_error);Signature · 1 line
2.25.2Signature · 1 line
Jeff Layton <[email protected]>
Signature · 1 line
Jeff Layton <[email protected]>