f2fs: use flush command instead of FUA for zoned device
Jaegeuk Kim
maintainerPATCH 1/1 f2fs: use flush command instead of FUA for zoned device
The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.
--- fs/f2fs/file.c | 4 +++- fs/f2fs/node.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)
fs/f2fs/file.c +3 -1
@@ -372,7 +372,9 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,f2fs_remove_ino_entry (sbi, ino,APPEND_INO );clear_inode_flag (inode,FI_APPEND_WRITE ); flush_out: -if (!atomic &&F2FS_OPTION (sbi).fsync_mode !=FSYNC_MODE_NOBARRIER ) +if ((!atomic &&F2FS_OPTION (sbi).fsync_mode !=FSYNC_MODE_NOBARRIER ) || + (atomic && !test_opt (sbi,NOBARRIER ) && +f2fs_sb_has_blkzoned (sbi))) ret =f2fs_issue_flush (sbi, inode->i_ino);if (!ret) {f2fs_remove_ino_entry (sbi, ino,UPDATE_INO );
fs/f2fs/node.c +1 -1
@@ -1633,7 +1633,7 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted, goto redirty_out; } -if (atomic && !test_opt (sbi,NOBARRIER )) +if (atomic && !test_opt (sbi,NOBARRIER ) && !f2fs_sb_has_blkzoned (sbi)) fio.op_flags |=REQ_PREFLUSH |REQ_FUA ;/* should add to global list before clearing PAGECACHE status */
Signature · 1 line
2.36.0.rc0.470.gd361397f0d-goog
Quoted 2 lines · Jaegeuk Kim
The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.
The block layer can reorder all commands. Given that FUA only affects the command itself that should not matter. What is this trying to fix?
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Jaegeuk Kim
maintainerQuoted 6 lines · Christoph Hellwig
Quoted 2 lines · Jaegeuk Kim
The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.
The block layer can reorder all commands. Given that FUA only affects the command itself that should not matter. What is this trying to fix?
Not for zoned disk, as I mentioned. The mq-deadline serializes write commands but I got an reordered command, if there's a FUA|PREFLUSH command, which emits EIO by the disk.
Quoted 2 lines · Jaegeuk Kim
The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.
Stricktly speaking, for a request that has data, the problem is triggered by REQ_PREFLUSH since in this case the request does not go through the scheduler and is processed through the blk-flush machinery. REQ_FUA on its own should not matter if the device supports it. If the device does not support FUA, then the same problem can happen due to POSTFLUSH (again no scheduler).
Bypassing the scheduler leads to the write not write-locking the zone, which leads to reordering... Completely overlooked that case when the zone write locking was implemented.
Ideally, the FS should not have to care about this. blk-flush machinery should be a little more intelligent and process the write phase of the request using the scheduler. Need to look into that.
Quoted 12 lines
Signed-off-by Jaegeuk Kim <[email protected]>--- fs/f2fs/file.c | 4 +++- fs/f2fs/node.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)fs/f2fs/file.c +0
-0
fs/f2fs/file.c static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, · line 377
Quoted 7 lines
fs/f2fs/file.c +3
-1@@ -372,7 +372,9 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,f2fs_remove_ino_entry (sbi, ino,APPEND_INO );clear_inode_flag (inode,FI_APPEND_WRITE ); flush_out: -if (!atomic &&F2FS_OPTION (sbi).fsync_mode !=FSYNC_MODE_NOBARRIER ) +if ((!atomic &&F2FS_OPTION (sbi).fsync_mode !=FSYNC_MODE_NOBARRIER ) || + (atomic && !test_opt (sbi,NOBARRIER ) && +f2fs_sb_has_blkzoned (sbi)))
Aligning the conditions and not breaking the second line would make this a lot easier to read...
Quoted 8 lines
fs/f2fs/file.c +0
-0ret =f2fs_issue_flush (sbi, inode->i_ino);if (!ret) {f2fs_remove_ino_entry (sbi, ino,UPDATE_INO );fs/f2fs/node.c +0
-0
fs/f2fs/node.c static int __write_node_page(struct page *page, bool atomic, bool *submitted, · line 1637
Quoted 6 lines
fs/f2fs/node.c +1
-1@@ -1633,7 +1633,7 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted, goto redirty_out; } -if (atomic && !test_opt (sbi,NOBARRIER )) +if (atomic && !test_opt (sbi,NOBARRIER ) && !f2fs_sb_has_blkzoned (sbi)) fio.op_flags |=REQ_PREFLUSH |REQ_FUA ;
Is this really OK to do ? flush + write as different operations may not lead to the same result as a preflush+fua write.
Until the block layer is fixed to properly handle this, a simpler fix for f2fs would be to force enable the NOBARRIER option for zoned drives ? That would avoid these changes no ?
Also, with all the testing we do on SMR disks and f2fs (smaller, older SMR disks due to the 16TB limit), we never have triggered this problem. How did you trigger it ?
Quoted 2 lines
fs/f2fs/node.c +0
-0/* should add to global list before clearing PAGECACHE status */
Signature · 2 lines
Damien Le Moal Western Digital Research
Jaegeuk Kim
maintainerQuoted 10 lines · Damien Le Moal
Quoted 2 lines · Jaegeuk Kim
The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.
Stricktly speaking, for a request that has data, the problem is triggered by
REQ_PREFLUSHsince in this case the request does not go through the scheduler and is processed through the blk-flush machinery.REQ_FUAon its own should not matter if the device supports it. If the device does not support FUA, then the same problem can happen due to POSTFLUSH (again no scheduler).
I think the problem is a piggy-backed data along with flush or fua whatever, but this made me use a separate flush command.
Quoted 8 lines
Bypassing the scheduler leads to the write not write-locking the zone, which leads to reordering... Completely overlooked that case when the zone write locking was implemented.
Ideally, the FS should not have to care about this. blk-flush machinery should be a little more intelligent and process the write phase of the request using the scheduler. Need to look into that.
Please. I'm okay to revert this, once the block layer supports.
Quoted 8 lines
Quoted 8 lines
Signed-off-by Jaegeuk Kim <[email protected]>--- fs/f2fs/file.c | 4 +++- fs/f2fs/node.c | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-)
Quoted 15 lines
fs/f2fs/file.c static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, · line 377
Quoted 12 lines
fs/f2fs/file.c +3
-1@@ -372,7 +372,9 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,f2fs_remove_ino_entry (sbi, ino,APPEND_INO );clear_inode_flag (inode,FI_APPEND_WRITE ); flush_out: -if (!atomic &&F2FS_OPTION (sbi).fsync_mode !=FSYNC_MODE_NOBARRIER ) +if ((!atomic &&F2FS_OPTION (sbi).fsync_mode !=FSYNC_MODE_NOBARRIER ) || + (atomic && !test_opt (sbi,NOBARRIER ) && +f2fs_sb_has_blkzoned (sbi)))Aligning the conditions and not breaking the second line would make this a lot easier to read...
Sure.
Quoted 19 lines
Quoted 9 lines
fs/f2fs/file.c +0
-0ret =f2fs_issue_flush (sbi, inode->i_ino);if (!ret) {f2fs_remove_ino_entry (sbi, ino,UPDATE_INO );fs/f2fs/node.c +0
-0fs/f2fs/node.c static int __write_node_page(struct page *page, bool atomic, bool *submitted, · line 1637
Quoted 6 lines
fs/f2fs/node.c +1
-1@@ -1633,7 +1633,7 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted, goto redirty_out; } -if (atomic && !test_opt (sbi,NOBARRIER )) +if (atomic && !test_opt (sbi,NOBARRIER ) && !f2fs_sb_has_blkzoned (sbi)) fio.op_flags |=REQ_PREFLUSH |REQ_FUA ;Is this really OK to do ? flush + write as different operations may not lead to the same result as a preflush+fua write.
Quoted 3 lines
Until the block layer is fixed to properly handle this, a simpler fix for f2fs would be to force enable the NOBARRIER option for zoned drives ? That would avoid these changes no ?
No, it will hurt the stability of FS metadata consistency.
Quoted 4 lines
Also, with all the testing we do on SMR disks and f2fs (smaller, older SMR disks due to the 16TB limit), we never have triggered this problem. How did you trigger it ?
This happens in Android only, since atomic_write for sqlite is taking this path.
Quoted 8 lines
Quoted 3 lines
fs/f2fs/node.c +0
-0/* should add to global list before clearing PAGECACHE status */ Signature · 2 lines
Damien Le Moal Western Digital Research
Jaegeuk Kim
maintainerThe block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.
---
change log from v1:
- minor clean up
fs/f2fs/file.c | 3 ++- fs/f2fs/node.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
fs/f2fs/file.c +2 -1
@@ -372,7 +372,8 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,f2fs_remove_ino_entry (sbi, ino,APPEND_INO );clear_inode_flag (inode,FI_APPEND_WRITE ); flush_out: -if (!atomic &&F2FS_OPTION (sbi).fsync_mode !=FSYNC_MODE_NOBARRIER ) +if ((!atomic &&F2FS_OPTION (sbi).fsync_mode !=FSYNC_MODE_NOBARRIER ) || + (atomic && !test_opt (sbi,NOBARRIER ) &&f2fs_sb_has_blkzoned (sbi))) ret =f2fs_issue_flush (sbi, inode->i_ino);if (!ret) {f2fs_remove_ino_entry (sbi, ino,UPDATE_INO );
fs/f2fs/node.c +1 -1
@@ -1631,7 +1631,7 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted, goto redirty_out; } -if (atomic && !test_opt (sbi,NOBARRIER )) +if (atomic && !test_opt (sbi,NOBARRIER ) && !f2fs_sb_has_blkzoned (sbi)) fio.op_flags |=REQ_PREFLUSH |REQ_FUA ;/* should add to global list before clearing PAGECACHE status */
Signature · 1 line
2.36.0.rc2.479.g8af0fa9b8e-goog