btrfs: make read repair work in synchronous mode
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest13 patches 19 repliesIn review
Changes in v1 from the cover letter
Most updates are to reduce the memory deadlock in endio function context.
- Allocate a new mempool for read_repair_bio
This is to avoid allocating the same btrfs_bio while we're still holding one btrfs_bio in its endio function.
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/13 | btrfs: introduce a pure data checksum checking helperfs/ fs/ fs/ | +38 | 1 reply | |
| 2/13 | btrfs: quit early if the fs has no RAID56 support for raid56 related checksfs/ | +6 | 0 replies | |
| 3/13 | btrfs: save the original bi_iter into btrfs_bio for buffered readfs/ fs/ | +14 | 0 replies | |
| 4/13 | btrfs: remove duplicated parameters from submit_data_read_repair()fs/ | +8 | 0 replies | |
| 5/13 | btrfs: add btrfs_read_repair_ctrl to record corrupted sectorsfs/ fs/ fs/ +3 more | +170 | 11 replies | |
| 6/13 | btrfs: add a helper to queue a corrupted sector for read repairfs/ fs/ fs/ | +157 | 4 replies | |
| 7/13 | btrfs: introduce a helper to repair from one mirrorfs/ fs/ fs/ | +163 | 0 replies | |
| 8/13 | btrfs: allow btrfs read repair to submit writes in asynchronous modefs/ | +32 | 0 replies | |
| 9/13 | btrfs: handle RAID56 read repair differentlyfs/ fs/ | +28 | 0 replies | |
| 10/13 | btrfs: switch buffered read to the new read repair routinefs/ fs/ | +82 | 0 replies | |
| 11/13 | btrfs: switch direct IO routine to use btrfs_read_repair_ctrlfs/ fs/ fs/ +1 more | +51 | 0 replies | |
| 12/13 | btrfs: remove io_failure_record infrastructure completelyfs/ fs/ fs/ +1 more | +4 | 0 replies | |
| 13/13 | btrfs: remove btrfs_inode::io_failure_treefs/ fs/ fs/ +1 more | +0 | 1 reply |
Reviews trailers per patch
2 of 13 patches have a Reviewed-by or Acked-by.
| Patch | Christoph Hellwig |
|---|---|
| 1/13 | R-b |
| 2/13 | |
| 3/13 | |
| 4/13 | R-b |
| 5/13 | |
| 6/13 | |
| 7/13 | |
| 8/13 | |
| 9/13 | |
| 10/13 | |
| 11/13 | |
| 12/13 | |
| 13/13 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH 00/13] btrfs: make read repair work in synchronous mode
[CHANGELOG]
RFC v2 -> v1:
Most updates are to reduce the memory deadlock in endio function context.
- Allocate a new mempool for read_repair_bio
This is to avoid allocating the same btrfs_bio while we're still holding one btrfs_bio in its endio function.
The problem is there for a long time in the existing code, only recently Christoph mentioned the possible deadlock scenario.
Furthermore, our new read_repair_bio is much smaller than btrfs_bio, avoid wasting memory on unused members.
- Submit the assembled bio immediate if the next sector is not mergeable Instead of holding them in a bio list, this gives us higher chance to reclaim the space allocated for the read_repair_bio.
- Pre-allocate needed two bitmaps inside btrfs_submit_data_bio() If we failed to allocated the memory, we just fail the bio, and VFS layer will re-try with much smaller range, and we will have a much higher chance to allocate the needed memory in the next try.
- Fix the btrfs/157 failure by introducing RAID56 specific repair The old repair_io_failure() can handle it pretty well, although in that case we will lose the async bio submission, but that should still be acceptable just for RAID56.
RFC v1 -> RFC v2:
- Assemble a bio list for read/write bios and submit them in one go This allows less submit bio hooks, while still allow us to wait for them all to finish.
- Completely remove io_failure_tree infrastructure
Now we don't need to remember which mirror we hit error. At end_bio_extent_readpage() we either get good data and done the repair already, or we there aren't enough mirrors for us to recover all data.
This is mostly trading on-stack memory of end_bio_extent_readpage() with btrfs_inode::io_failure_tree.
The latter tree has a much longer lifespan, thus I think it's still a win overall
[RFC POINTS]
- How to improve read_repair_get_sector()?
Currently we always iterate the whole bio to grab the target page/pgoff.
Is there any better cached method to avoid such iteration?
- Is this new code logically more reader-friendly?
It's more for sure straight-forward, but I doubt if it's any easier to read compared to the old code.
- btrfs/157 failure
Need extra check to find out why btrfs/157 failed.
In theory, we should just iterate through all mirrors, I guess it's we have no way to exhaust all combinations, thus the extra 2 "mirrors" can gave us wrong result for RAID6.
[BEFORE]
For corrupted sectors, we just record the logical bytenr and mirror number into io_failure_tree, then re-queue the same block with different mirror number and call it a day.
The re-queued read will trigger enter the same endio function, with extra failrec handling to either continue re-queue (csum mismatch/read failure), or clear the current failrec and submit a write to fix the corrupted mirror (read succeeded and csum match/no csum).
This is harder to read, as we need to enter the same river twice or even more.
[AFTER]
Before submitting a data read bio, we will pre-allocate the bitmaps used by read repair first.
If we have no memory, we just fail and let VFS layer to retry with smaller range, and we will have a larger chance to get the memory in next try.
For corrupted sectors, we record the following things into an on-stack structure in end_bio_extent_readpage():
- The original bio
- The original file offset of the bio
This is for direct IO case, as we can not grab file offset just using page_offset()
- Offset inside the bio of the corrupted sector
- Corrupted mirror
Then in the new btrfs_read_repair_ctrl structure, we hold those info like:
Original bio logical = X, file_offset = Y, inode=(R/I)
Offset inside bio: 0 4k 8K 12K 16Kcur_bad_bitmap | X| X| | X|
Each set bit will indicate we have a corrupted sector inside the original bio.
After we have iterated all sectors of the original bio, then we call btrfs_read_repair_finish() to do the real repair by:
- Assemble and submit read bios
For above case, bio offset [0, 8K) will be inside one bio, while another bio for bio offset [12K, 16K).
And the page/pgoff will be extracted from the original bio.
This is a little different from the old behavior, as old behavior will submit a new bio for each sector.
The new behavior will save some btrfs_map_bio() calls.
- Submit the last read bio and wait them to finish
- Re-verify the read result
- Submit write for the corrupted mirror
We do the same behavior just like read bios, assemble and submit them.
And for repaired sectors, remove them from @cur_bad_bitmap.
- Do the same loop until either 1) we tried all mirrors, or 2) no more corrupted sectors
- Handle the remaining corrupted sectors
Either mark them error for buffered read, or just return an error for direct IO.
By this we can:
- Remove the re-entry behavior of endio function Now everything is handled inside end_bio_extent_readpage().
- Remove the io_failure_tree completely
As we don't need to record which mirror has failed.
- Slightly reduced overhead on read repair
Now we won't call btrfs_map_bio() for each corrupted sector, as we can merge the sectors into a much larger bio.
Qu Wenruo (13):
btrfs: introduce a pure data checksum checking helper
btrfs: quit early if the fs has no RAID56 support for raid56 related
checks
btrfs: save the original bi_iter into btrfs_bio for buffered read
btrfs: remove duplicated parameters from submit_data_read_repair()
btrfs: add btrfs_read_repair_ctrl to record corrupted sectors
btrfs: add a helper to queue a corrupted sector for read repair
btrfs: introduce a helper to repair from one mirror
btrfs: allow btrfs read repair to submit writes in asynchronous mode
btrfs: handle RAID56 read repair differently
btrfs: switch buffered read to the new read repair routine
btrfs: switch direct IO routine to use btrfs_read_repair_ctrl
btrfs: remove io_failure_record infrastructure completely
btrfs: remove btrfs_inode::io_failure_tree
fs/btrfs/Makefile | 2 +- fs/btrfs/btrfs_inode.h | 5 - fs/btrfs/compression.c | 10 +- fs/btrfs/ctree.h | 2 + fs/btrfs/extent-io-tree.h | 15 -- fs/btrfs/extent_io.c | 490 +++++------------------------------ fs/btrfs/extent_io.h | 28 +- fs/btrfs/inode.c | 121 +++++---- fs/btrfs/read-repair.c | 459 ++++++++++++++++++++++++++++++++ fs/btrfs/read-repair.h | 84 ++++++ fs/btrfs/super.c | 9 +- fs/btrfs/volumes.c | 6 + fs/btrfs/volumes.h | 4 + include/trace/events/btrfs.h | 1 - 14 files changed, 695 insertions(+), 541 deletions(-) create mode 100644 fs/btrfs/read-repair.c create mode 100644 fs/btrfs/read-repair.h
Signature · 1 line
2.36.0