bcachefs: fix incorrect usage of REQ_OP_FLUSH
PATCH 1/1 bcachefs: fix incorrect usage of REQ_OP_FLUSH
REQ_OP_FLUSH is only for internal use in the blk-mq and request based drivers. File systems and other block layer consumers must use REQ_OP_WRITE | REQ_PREFLUSH as documented in Documentation/block/writeback_cache_control.rst.
While REQ_OP_FLUSH appears to work for blk-mq drivers it does not get the proper flush state machine handling, and completely fails for any bio based drivers, including all the stacking drivers. The block layer will also get a check in 6.8 to reject this use case entirely.
[Note: completely untested, but as this never got fixed since the original bug report in November:
https://bugzilla.kernel.org/show_bug.cgi?id=218184
and the the discussion in December:
https://lore.kernel.org/all/[email protected]/T/
this seems to be best way to force it]
--- fs/bcachefs/fs-io.c | 2 +- fs/bcachefs/journal_io.c | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-)
fs/bcachefs/fs-io.c +1 -1
@@ -79,7 +79,7 @@ void bch2_inode_flush_nocow_writes_async(struct bch_fs *c,continue ; bio =container_of (bio_alloc_bioset (ca->disk_sb.bdev,0 , -REQ_OP_FLUSH , +REQ_OP_WRITE |REQ_PREFLUSH ,GFP_KERNEL , &c->nocow_flush_bioset), struct nocow_flush, bio);
fs/bcachefs/journal_io.c +2 -1
@@ -1948,7 +1948,8 @@ CLOSURE_CALLBACK(bch2_journal_write)percpu_ref_get (&ca->io_ref); bio = ca->journal.bio; -bio_reset (bio, ca->disk_sb.bdev,REQ_OP_FLUSH ); +bio_reset (bio, ca->disk_sb.bdev, +REQ_OP_WRITE |REQ_PREFLUSH ), bio->bi_end_io = journal_write_endio; bio->bi_private = ca;closure_bio_submit (bio, cl);
Signature · 1 line
2.39.2
Quoted 16 lines · Christoph Hellwig
REQ_OP_FLUSHis only for internal use in the blk-mq and request based drivers. File systems and other block layer consumers must useREQ_OP_WRITE|REQ_PREFLUSHas documented inDocumentation/block/writeback_cache_control.rst.While
REQ_OP_FLUSHappears to work for blk-mq drivers it does not get the proper flush state machine handling, and completely fails for any bio based drivers, including all the stacking drivers. The block layer will also get a check in 6.8 to reject this use case entirely.[Note: completely untested, but as this never got fixed since the original bug report in November:
https://bugzilla.kernel.org/show_bug.cgi?id=218184
Quoted 7 lines
and the the discussion in December:
https://lore.kernel.org/all/[email protected]/T/this seems to be best way to force it]
Signed-off-by Christoph Hellwig <[email protected]>
Hey Christoph, thanks - applied.
Quoted 16 lines · Christoph Hellwig
REQ_OP_FLUSHis only for internal use in the blk-mq and request based drivers. File systems and other block layer consumers must useREQ_OP_WRITE|REQ_PREFLUSHas documented inDocumentation/block/writeback_cache_control.rst.While
REQ_OP_FLUSHappears to work for blk-mq drivers it does not get the proper flush state machine handling, and completely fails for any bio based drivers, including all the stacking drivers. The block layer will also get a check in 6.8 to reject this use case entirely.[Note: completely untested, but as this never got fixed since the original bug report in November:
https://bugzilla.kernel.org/show_bug.cgi?id=218184
Quoted 7 lines
and the the discussion in December:
https://lore.kernel.org/all/[email protected]/T/this seems to be best way to force it]
Signed-off-by Christoph Hellwig <[email protected]>
I've got a new bug report with this patch:
https://www.reddit.com/r/bcachefs/comments/19a2u3c/error_writing_journal_entry/
We seem to be geting -EOPNOTSUPP back from REQ_OP_FLUSH?
Quoted 5 lines · Kent Overstreet
I've got a new bug report with this patch:
https://www.reddit.com/r/bcachefs/comments/19a2u3c/error_writing_journal_entry/We seem to be geting
-EOPNOTSUPPback fromREQ_OP_FLUSH?
With the patch you are replying to you will not get -EOPNOTSUPP for REQ_OP_FLUSH?, because bcachefs won't send it as it's supposed to.
Without this patch as in current mainline you will get -EOPNOTSUPP because sending REQ_OP_FLUSH and finally check for that to catch bugs like the one fixed with this patch.
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Kent Overstreet
I've got a new bug report with this patch:
https://www.reddit.com/r/bcachefs/comments/19a2u3c/error_writing_journal_entry/
Quoted 8 lines
Quoted 1 line
We seem to be geting
-EOPNOTSUPPback fromREQ_OP_FLUSH?With the patch you are replying to you will not get
-EOPNOTSUPPforREQ_OP_FLUSH?, because bcachefs won't send it as it's supposed to.Without this patch as in current mainline you will get
-EOPNOTSUPPbecause sendingREQ_OP_FLUSHand finally check for that to catch bugs like the one fixed with this patch.
Then why did the user report -EOPNOTSUPP with the patch, which went away when reverted?
Quoted 6 lines · Kent Overstreet
Quoted 3 lines
Without this patch as in current mainline you will get
-EOPNOTSUPPbecause sendingREQ_OP_FLUSHand finally check for that to catch bugs like the one fixed with this patch.Then why did the user report
-EOPNOTSUPPwith the patch, which went away when reverted?
I have no idea, as we never return -EOPNOTSUPP for REQ_OP_WRITE | REQ_PREFLUSH this would be odd. According to your report the users runs a Fedora kernel and this commit never went upstream or even into linux-next, which doesn't quite add a up to me either.
But in the end just try it yourself. On current mainline with the buggy REQ_OP_FLUSH you should be seeing -EOPNOTSUPP every single time you try a data integrity operation, without it you should not. Even the most basic test should be catching this.
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines · Kent Overstreet
Quoted 3 lines
Without this patch as in current mainline you will get
-EOPNOTSUPPbecause sendingREQ_OP_FLUSHand finally check for that to catch bugs like the one fixed with this patch.
Quoted 8 lines
Quoted 2 lines
Then why did the user report
-EOPNOTSUPPwith the patch, which went away when reverted?I have no idea, as we never return
-EOPNOTSUPPforREQ_OP_WRITE|REQ_PREFLUSHthis would be odd. According to your report the users runs a Fedora kernel and this commit never went upstream or even into linux-next, which doesn't quite add a up to me either.
Ahh - I misread the bug report (fedora puts out kernels before rc1!?). Thanks, your patch is back in :)
Quoted 2 lines · Kent Overstreet
Ahh - I misread the bug report (fedora puts out kernels before rc1!?). Thanks, your patch is back in :)
Please throw it in your test setup and watch the results carefully. I'm pretty sure it is correct, but I do not actually have a way to verify it right now.
Quoted 7 lines · Christoph Hellwig
Quoted 2 lines · Kent Overstreet
Ahh - I misread the bug report (fedora puts out kernels before rc1!?). Thanks, your patch is back in :)
Please throw it in your test setup and watch the results carefully. I'm pretty sure it is correct, but I do not actually have a way to verify it right now.
updating my tests for the MD_FAULTY removal, then will do. Is there anything you want me to look for?
considering most tests won't break or won't clearly break if flush/fua is being dropped (even generic/388 was passing reliably, of course, since virtual block devices aren't going to reorder writes...) maybe we could do some print statement sanity checking...
Quoted 2 lines · Kent Overstreet
updating my tests for the
MD_FAULTYremoval, then will do. Is there anything you want me to look for?
Nothing fancy. Just that you do data integrity operations and do not see an error.
Quoted 4 lines
considering most tests won't break or won't clearly break if flush/fua is being dropped (even generic/388 was passing reliably, of course, since virtual block devices aren't going to reorder writes...) maybe we could do some print statement sanity checking...
Well, xfstests is not very good about power fail testing, because it can't inject a power fail.. Which is a problem in it's own, but would need hardware power failing or at least some pretty good VM emulation of the same outside the scope of the actual xfstests runner.
But basically what this patch needs is just seeing that you don't get I/O errors on anything that flushes, which on currently mainline without the patch you should be seeing.
Quoted 12 lines · Christoph Hellwig
Quoted 2 lines · Kent Overstreet
updating my tests for the
MD_FAULTYremoval, then will do. Is there anything you want me to look for?Nothing fancy. Just that you do data integrity operations and do not see an error.
Quoted 4 lines
considering most tests won't break or won't clearly break if flush/fua is being dropped (even generic/388 was passing reliably, of course, since virtual block devices aren't going to reorder writes...) maybe we could do some print statement sanity checking...
Quoted 4 lines
Well, xfstests is not very good about power fail testing, because it can't inject a power fail.. Which is a problem in it's own, but would need hardware power failing or at least some pretty good VM emulation of the same outside the scope of the actual xfstests runner.
We do actually have stuff in fstests that checks write vs flush ordering as issued by the filesystem. We use dm-logwrites for tracking the writes and flushes and to be able to replay arbitrary groups of writes between flushes. generic/482 is one of those tests.
These are the ordering problems that power failures expose, so we do actually have tests that cover the same conditions that pulling the power from a device would exercise.
I even wrote a debugging script (tools/dm-logwrite-replay) to iterate write+fua groups in the test corpse to isolate the write groups where the consistency failure occurs when doing work to optimise flushes being issued by the XFS journal checkpoint writes.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]