Skip to contentSkip to conversation

f2fs: use flush command instead of FUA for zoned device

PATCH In review Jaegeuk Kim 3 participants 6 messages Apr 19, 2022 – Apr 21, 2022 F2FS file system

lore
Share & tools
lore

Jaegeuk Kim

author maintainer
#0 lore raw

PATCH 1/1 f2fs: use flush command instead of FUA for zoned device

The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.

---
 fs/f2fs/file.c | 4 +++-
 fs/f2fs/node.c | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)
fs/f2fs/file.c +3 -1
index f08e6208e183..2aef0632f35b 100644
@@ -372,7 +372,9 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
 	f2fs_remove_ino_entry(sbi, ino, APPEND_INO);
 	clear_inode_flag(inode, FI_APPEND_WRITE);
 flush_out:
-	if (!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER)
+	if ((!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER) ||
+			(atomic && !test_opt(sbi, NOBARRIER) &&
+					f2fs_sb_has_blkzoned(sbi)))
 		ret = f2fs_issue_flush(sbi, inode->i_ino);
 	if (!ret) {
 		f2fs_remove_ino_entry(sbi, ino, UPDATE_INO);
fs/f2fs/node.c +1 -1
index c280f482c741..7224a980056f 100644
@@ -1633,7 +1633,7 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
 		goto redirty_out;
 	}
 
-	if (atomic && !test_opt(sbi, NOBARRIER))
+	if (atomic && !test_opt(sbi, NOBARRIER) && !f2fs_sb_has_blkzoned(sbi))
 		fio.op_flags |= REQ_PREFLUSH | REQ_FUA;
 
 	/* should add to global list before clearing PAGECACHE status */
Signature · 1 line
2.36.0.rc0.470.gd361397f0d-goog
author maintainer
#2 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 2 lines · Jaegeuk Kim

The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.

The block layer can reorder all commands. Given that FUA only affects the command itself that should not matter. What is this trying to fix?

Not for zoned disk, as I mentioned. The mq-deadline serializes write commands but I got an reordered command, if there's a FUA|PREFLUSH command, which emits EIO by the disk.

#3 lore raw
In reply to Jaegeuk Kim
Quoted 2 lines · Jaegeuk Kim

The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.

Stricktly speaking, for a request that has data, the problem is triggered by REQ_PREFLUSH since in this case the request does not go through the scheduler and is processed through the blk-flush machinery. REQ_FUA on its own should not matter if the device supports it. If the device does not support FUA, then the same problem can happen due to POSTFLUSH (again no scheduler).

Bypassing the scheduler leads to the write not write-locking the zone, which leads to reordering... Completely overlooked that case when the zone write locking was implemented.

Ideally, the FS should not have to care about this. blk-flush machinery should be a little more intelligent and process the write phase of the request using the scheduler. Need to look into that.

Quoted 12 lines
---
 fs/f2fs/file.c | 4 +++-
 fs/f2fs/node.c | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)
fs/f2fs/file.c +0 -0
index f08e6208e183..2aef0632f35b 100644

fs/f2fs/file.c static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, · line 377

Quoted 7 lines
fs/f2fs/file.c +3 -1
@@ -372,7 +372,9 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
 	f2fs_remove_ino_entry(sbi, ino, APPEND_INO);
 	clear_inode_flag(inode, FI_APPEND_WRITE);
 flush_out:
-	if (!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER)
+	if ((!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER) ||
+			(atomic && !test_opt(sbi, NOBARRIER) &&
+					f2fs_sb_has_blkzoned(sbi)))

Aligning the conditions and not breaking the second line would make this a lot easier to read...

Quoted 8 lines
fs/f2fs/file.c +0 -0
 		ret = f2fs_issue_flush(sbi, inode->i_ino);
 	if (!ret) {
 		f2fs_remove_ino_entry(sbi, ino, UPDATE_INO);
fs/f2fs/node.c +0 -0
index c280f482c741..7224a980056f 100644

fs/f2fs/node.c static int __write_node_page(struct page *page, bool atomic, bool *submitted, · line 1637

Quoted 6 lines
fs/f2fs/node.c +1 -1
@@ -1633,7 +1633,7 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
 		goto redirty_out;
 	}
 
-	if (atomic && !test_opt(sbi, NOBARRIER))
+	if (atomic && !test_opt(sbi, NOBARRIER) && !f2fs_sb_has_blkzoned(sbi))
 		fio.op_flags |= REQ_PREFLUSH | REQ_FUA;

Is this really OK to do ? flush + write as different operations may not lead to the same result as a preflush+fua write.

Until the block layer is fixed to properly handle this, a simpler fix for f2fs would be to force enable the NOBARRIER option for zoned drives ? That would avoid these changes no ?

Also, with all the testing we do on SMR disks and f2fs (smaller, older SMR disks due to the 16TB limit), we never have triggered this problem. How did you trigger it ?

Quoted 2 lines
fs/f2fs/node.c +0 -0
 	/* should add to global list before clearing PAGECACHE status */
Signature · 2 lines
Damien Le Moal
Western Digital Research
author maintainer
#4 lore raw
Quoted 10 lines · Damien Le Moal
Quoted 2 lines · Jaegeuk Kim

The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.

Stricktly speaking, for a request that has data, the problem is triggered by REQ_PREFLUSH since in this case the request does not go through the scheduler and is processed through the blk-flush machinery. REQ_FUA on its own should not matter if the device supports it. If the device does not support FUA, then the same problem can happen due to POSTFLUSH (again no scheduler).

I think the problem is a piggy-backed data along with flush or fua whatever, but this made me use a separate flush command.

Quoted 8 lines

Bypassing the scheduler leads to the write not write-locking the zone, which leads to reordering... Completely overlooked that case when the zone write locking was implemented.

Ideally, the FS should not have to care about this. blk-flush machinery should be a little more intelligent and process the write phase of the request using the scheduler. Need to look into that.

Please. I'm okay to revert this, once the block layer supports.

Quoted 8 lines
Quoted 8 lines
---
 fs/f2fs/file.c | 4 +++-
 fs/f2fs/node.c | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)
Quoted 15 lines

fs/f2fs/file.c static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, · line 377

Quoted 12 lines
fs/f2fs/file.c +3 -1
index f08e6208e183..2aef0632f35b 100644
@@ -372,7 +372,9 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
 	f2fs_remove_ino_entry(sbi, ino, APPEND_INO);
 	clear_inode_flag(inode, FI_APPEND_WRITE);
 flush_out:
-	if (!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER)
+	if ((!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER) ||
+			(atomic && !test_opt(sbi, NOBARRIER) &&
+					f2fs_sb_has_blkzoned(sbi)))

Aligning the conditions and not breaking the second line would make this a lot easier to read...

Sure.

Quoted 19 lines
Quoted 9 lines
fs/f2fs/file.c +0 -0
 		ret = f2fs_issue_flush(sbi, inode->i_ino);
 	if (!ret) {
 		f2fs_remove_ino_entry(sbi, ino, UPDATE_INO);
fs/f2fs/node.c +0 -0
index c280f482c741..7224a980056f 100644

fs/f2fs/node.c static int __write_node_page(struct page *page, bool atomic, bool *submitted, · line 1637

Quoted 6 lines
fs/f2fs/node.c +1 -1
@@ -1633,7 +1633,7 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
 		goto redirty_out;
 	}
 
-	if (atomic && !test_opt(sbi, NOBARRIER))
+	if (atomic && !test_opt(sbi, NOBARRIER) && !f2fs_sb_has_blkzoned(sbi))
 		fio.op_flags |= REQ_PREFLUSH | REQ_FUA;

Is this really OK to do ? flush + write as different operations may not lead to the same result as a preflush+fua write.

Quoted 3 lines

Until the block layer is fixed to properly handle this, a simpler fix for f2fs would be to force enable the NOBARRIER option for zoned drives ? That would avoid these changes no ?

No, it will hurt the stability of FS metadata consistency.

Quoted 4 lines

Also, with all the testing we do on SMR disks and f2fs (smaller, older SMR disks due to the 16TB limit), we never have triggered this problem. How did you trigger it ?

This happens in Android only, since atomic_write for sqlite is taking this path.

Quoted 8 lines
Quoted 3 lines
fs/f2fs/node.c +0 -0
 	/* should add to global list before clearing PAGECACHE status */
Signature · 2 lines
Damien Le Moal
Western Digital Research
author maintainer
#5 lore raw
In reply to Jaegeuk Kim

The block layer for zoned disk can reorder the FUA'ed IOs. Let's use flush command to keep the write order.

---

change log from v1:
- minor clean up

 fs/f2fs/file.c | 3 ++-
 fs/f2fs/node.c | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
fs/f2fs/file.c +2 -1
index f08e6208e183..eae2e7908072 100644
@@ -372,7 +372,8 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
 	f2fs_remove_ino_entry(sbi, ino, APPEND_INO);
 	clear_inode_flag(inode, FI_APPEND_WRITE);
 flush_out:
-	if (!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER)
+	if ((!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER) ||
+	    (atomic && !test_opt(sbi, NOBARRIER) && f2fs_sb_has_blkzoned(sbi)))
 		ret = f2fs_issue_flush(sbi, inode->i_ino);
 	if (!ret) {
 		f2fs_remove_ino_entry(sbi, ino, UPDATE_INO);
fs/f2fs/node.c +1 -1
index c45d341dcf6e..144f9f966690 100644
@@ -1631,7 +1631,7 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
 		goto redirty_out;
 	}
 
-	if (atomic && !test_opt(sbi, NOBARRIER))
+	if (atomic && !test_opt(sbi, NOBARRIER) && !f2fs_sb_has_blkzoned(sbi))
 		fio.op_flags |= REQ_PREFLUSH | REQ_FUA;
 
 	/* should add to global list before clearing PAGECACHE status */
Signature · 1 line
2.36.0.rc2.479.g8af0fa9b8e-goog