Skip to contentSkip to conversation

bcachefs: fix incorrect usage of REQ_OP_FLUSH

PATCH In review Christoph Hellwig 3 participants 11 messages Jan 11, 2024 – Jan 23, 2024 VFS

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bcachefs: fix incorrect usage of REQ_OP_FLUSH

REQ_OP_FLUSH is only for internal use in the blk-mq and request based drivers. File systems and other block layer consumers must use REQ_OP_WRITE | REQ_PREFLUSH as documented in Documentation/block/writeback_cache_control.rst.

While REQ_OP_FLUSH appears to work for blk-mq drivers it does not get the proper flush state machine handling, and completely fails for any bio based drivers, including all the stacking drivers. The block layer will also get a check in 6.8 to reject this use case entirely.

[Note: completely untested, but as this never got fixed since the original bug report in November:

https://bugzilla.kernel.org/show_bug.cgi?id=218184

and the the discussion in December:

    https://lore.kernel.org/all/[email protected]/T/

this seems to be best way to force it]

---
 fs/bcachefs/fs-io.c      | 2 +-
 fs/bcachefs/journal_io.c | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)
fs/bcachefs/fs-io.c +1 -1
index b0e8144ec5500c..a8500af6c7438f 100644
@@ -79,7 +79,7 @@ void bch2_inode_flush_nocow_writes_async(struct bch_fs *c,
 			continue;
 
 		bio = container_of(bio_alloc_bioset(ca->disk_sb.bdev, 0,
-						    REQ_OP_FLUSH,
+						    REQ_OP_WRITE | REQ_PREFLUSH,
 						    GFP_KERNEL,
 						    &c->nocow_flush_bioset),
 				   struct nocow_flush, bio);
fs/bcachefs/journal_io.c +2 -1
index 3eb6c3f62a811b..43c76c9ad9a316 100644
@@ -1948,7 +1948,8 @@ CLOSURE_CALLBACK(bch2_journal_write)
 			percpu_ref_get(&ca->io_ref);
 
 			bio = ca->journal.bio;
-			bio_reset(bio, ca->disk_sb.bdev, REQ_OP_FLUSH);
+			bio_reset(bio, ca->disk_sb.bdev,
+					REQ_OP_WRITE | REQ_PREFLUSH),
 			bio->bi_end_io		= journal_write_endio;
 			bio->bi_private		= ca;
 			closure_bio_submit(bio, cl);
Signature · 1 line
2.39.2
#1 lore raw
Quoted 16 lines · Christoph Hellwig

REQ_OP_FLUSH is only for internal use in the blk-mq and request based drivers. File systems and other block layer consumers must use REQ_OP_WRITE | REQ_PREFLUSH as documented in Documentation/block/writeback_cache_control.rst.

While REQ_OP_FLUSH appears to work for blk-mq drivers it does not get the proper flush state machine handling, and completely fails for any bio based drivers, including all the stacking drivers. The block layer will also get a check in 6.8 to reject this use case entirely.

[Note: completely untested, but as this never got fixed since the original bug report in November:

   https://bugzilla.kernel.org/show_bug.cgi?id=218184
Quoted 7 lines

and the the discussion in December:

    https://lore.kernel.org/all/[email protected]/T/

this seems to be best way to force it]

Hey Christoph, thanks - applied.

#2 lore raw
In reply to Christoph Hellwig
Quoted 16 lines · Christoph Hellwig

REQ_OP_FLUSH is only for internal use in the blk-mq and request based drivers. File systems and other block layer consumers must use REQ_OP_WRITE | REQ_PREFLUSH as documented in Documentation/block/writeback_cache_control.rst.

While REQ_OP_FLUSH appears to work for blk-mq drivers it does not get the proper flush state machine handling, and completely fails for any bio based drivers, including all the stacking drivers. The block layer will also get a check in 6.8 to reject this use case entirely.

[Note: completely untested, but as this never got fixed since the original bug report in November:

   https://bugzilla.kernel.org/show_bug.cgi?id=218184
Quoted 7 lines

and the the discussion in December:

    https://lore.kernel.org/all/[email protected]/T/

this seems to be best way to force it]

I've got a new bug report with this patch:
https://www.reddit.com/r/bcachefs/comments/19a2u3c/error_writing_journal_entry/

We seem to be geting -EOPNOTSUPP back from REQ_OP_FLUSH?

#3 lore raw
Quoted 5 lines · Kent Overstreet

I've got a new bug report with this patch:
https://www.reddit.com/r/bcachefs/comments/19a2u3c/error_writing_journal_entry/

We seem to be geting -EOPNOTSUPP back from REQ_OP_FLUSH?

With the patch you are replying to you will not get -EOPNOTSUPP for REQ_OP_FLUSH?, because bcachefs won't send it as it's supposed to.

Without this patch as in current mainline you will get -EOPNOTSUPP because sending REQ_OP_FLUSH and finally check for that to catch bugs like the one fixed with this patch.

#4 lore raw
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Kent Overstreet

I've got a new bug report with this patch:
https://www.reddit.com/r/bcachefs/comments/19a2u3c/error_writing_journal_entry/

Quoted 8 lines
Quoted 1 line

We seem to be geting -EOPNOTSUPP back from REQ_OP_FLUSH?

With the patch you are replying to you will not get -EOPNOTSUPP for REQ_OP_FLUSH?, because bcachefs won't send it as it's supposed to.

Without this patch as in current mainline you will get -EOPNOTSUPP because sending REQ_OP_FLUSH and finally check for that to catch bugs like the one fixed with this patch.

Then why did the user report -EOPNOTSUPP with the patch, which went away when reverted?

#5 lore raw
Quoted 6 lines · Kent Overstreet
Quoted 3 lines

Without this patch as in current mainline you will get -EOPNOTSUPP because sending REQ_OP_FLUSH and finally check for that to catch bugs like the one fixed with this patch.

Then why did the user report -EOPNOTSUPP with the patch, which went away when reverted?

I have no idea, as we never return -EOPNOTSUPP for REQ_OP_WRITE | REQ_PREFLUSH this would be odd. According to your report the users runs a Fedora kernel and this commit never went upstream or even into linux-next, which doesn't quite add a up to me either.

But in the end just try it yourself. On current mainline with the buggy REQ_OP_FLUSH you should be seeing -EOPNOTSUPP every single time you try a data integrity operation, without it you should not. Even the most basic test should be catching this.

#6 lore raw
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Kent Overstreet
Quoted 3 lines

Without this patch as in current mainline you will get -EOPNOTSUPP because sending REQ_OP_FLUSH and finally check for that to catch bugs like the one fixed with this patch.

Quoted 8 lines
Quoted 2 lines

Then why did the user report -EOPNOTSUPP with the patch, which went away when reverted?

I have no idea, as we never return -EOPNOTSUPP for REQ_OP_WRITE | REQ_PREFLUSH this would be odd. According to your report the users runs a Fedora kernel and this commit never went upstream or even into linux-next, which doesn't quite add a up to me either.

Ahh - I misread the bug report (fedora puts out kernels before rc1!?). Thanks, your patch is back in :)

#7 lore raw
Quoted 2 lines · Kent Overstreet

Ahh - I misread the bug report (fedora puts out kernels before rc1!?). Thanks, your patch is back in :)

Please throw it in your test setup and watch the results carefully. I'm pretty sure it is correct, but I do not actually have a way to verify it right now.

#8 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 2 lines · Kent Overstreet

Ahh - I misread the bug report (fedora puts out kernels before rc1!?). Thanks, your patch is back in :)

Please throw it in your test setup and watch the results carefully. I'm pretty sure it is correct, but I do not actually have a way to verify it right now.

updating my tests for the MD_FAULTY removal, then will do. Is there anything you want me to look for?

considering most tests won't break or won't clearly break if flush/fua is being dropped (even generic/388 was passing reliably, of course, since virtual block devices aren't going to reorder writes...) maybe we could do some print statement sanity checking...

#9 lore raw
Quoted 2 lines · Kent Overstreet

updating my tests for the MD_FAULTY removal, then will do. Is there anything you want me to look for?

Nothing fancy. Just that you do data integrity operations and do not see an error.

Quoted 4 lines

considering most tests won't break or won't clearly break if flush/fua is being dropped (even generic/388 was passing reliably, of course, since virtual block devices aren't going to reorder writes...) maybe we could do some print statement sanity checking...

Well, xfstests is not very good about power fail testing, because it can't inject a power fail.. Which is a problem in it's own, but would need hardware power failing or at least some pretty good VM emulation of the same outside the scope of the actual xfstests runner.

But basically what this patch needs is just seeing that you don't get I/O errors on anything that flushes, which on currently mainline without the patch you should be seeing.

#10 lore raw
Quoted 12 lines · Christoph Hellwig
Quoted 2 lines · Kent Overstreet

updating my tests for the MD_FAULTY removal, then will do. Is there anything you want me to look for?

Nothing fancy. Just that you do data integrity operations and do not see an error.

Quoted 4 lines

considering most tests won't break or won't clearly break if flush/fua is being dropped (even generic/388 was passing reliably, of course, since virtual block devices aren't going to reorder writes...) maybe we could do some print statement sanity checking...

Quoted 4 lines

Well, xfstests is not very good about power fail testing, because it can't inject a power fail.. Which is a problem in it's own, but would need hardware power failing or at least some pretty good VM emulation of the same outside the scope of the actual xfstests runner.

We do actually have stuff in fstests that checks write vs flush ordering as issued by the filesystem. We use dm-logwrites for tracking the writes and flushes and to be able to replay arbitrary groups of writes between flushes. generic/482 is one of those tests.

These are the ordering problems that power failures expose, so we do actually have tests that cover the same conditions that pulling the power from a device would exercise.

I even wrote a debugging script (tools/dm-logwrite-replay) to iterate write+fua groups in the test corpse to isolate the write groups where the consistency failure occurs when doing work to optimise flushes being issued by the XFS journal checkpoint writes.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]