Massive data corruption on replace + fail + rebuild

Hello all,
I wrote some days ago to the ML with subject "MD RAID hot-replace wants to rewrite to the source! (and fails, and kicks)" The problem went much worse than that and ended with massive data corruption on the replacement drive sdm.

In this report I will use the same letters as in previous email: sdl for the failing drive, sdm for the replacement drive. This report is for raid5 on kernel 3.4.34 .
Bad blocks list is not enabled.
Bitmap is enabled
See previous post for details, mdstat and dmesg log.

It seems that the following series of events deeply corrupts the replacement drive:
1) drive sdl is flaky, so the user initiates replacement process (want_replacement) to the spare drive sdm
2) the disk sdl has read errors on some sectors. MD performs reconstruct read and then rewrite for those sectors. Currently MD wants to rewrite the source drive sdl, instead of just to the replacement drive sdm which I would much prefer.
3) sdl unfortunately is too flaky to receive rewrites, so it fails on the rewrites and is kicked by MD. The array is now degraded
4) At this point, MD apparently continues the rebuild process transforming the replacement into a full rebuild, but continuing from the point where sdl was kicked, and not from the start. This seems smart, however I guess there is a bug in doing this, maybe an off-by-N error. In the previous post you can see the dmesg line "[865031.586650] md: resuming recovery of md54 from checkpoint."

At the end of the rebuild, when MD starts using sdm as a member drive, an enormous amount of errors appear on the filesystems located on that array.

In fact, I performed a check afterwards, and this was the mismatch_cnt:

  root@server:/sys/block/md54/md# cat mismatch_cnt
  5296438776

This is on a 5x3TB array so about 90.4% of it has mismatches, if my math is correct.

The drive sdl was kicked at about 1% of the replacement process, so this 90.4% would not match, should have been 99%, but considering that many stripes could be zeroes, an 8.5% could be parities on zeroes which match just by chance.

So I suppose there is a problem in the handover between the replacement and the rebuild. I would bet on an off-by-N problem, i.e. a shifting of the data. Maybe the sources start a reconstruct-read from the beginning of the array while the destination continues writing from the point of the handover, or vice versa.

Currently I have "solved" the problem already, by artificially failing the drive sdm, and introducing another disk as spare to perform a clean rebuild from scratch. After failing sdm and dropping the caches, before inserting the new spare, the filesystems were readable again, so I was optimistic, and in fact at the end of the clean rebuild this appears to have recovered our data and mismatch_cnt is now zero. However I was lucky to have immediately understood what happened, otherwise we would have probably lost all our data, so please look into this.

Thanks for your work
PS: I would appreciate if you can also make MD not rewrite to the source during replacement :-) JJ

Bug was fixed in 3.4.56

0761d079bbc2 ("md/raid5: fix interaction of 'replace' and 'recovery'.")

and 3.11

f94c0b6658c7 ("md/raid5: fix interaction of 'replace' and 'recovery'.")

It probably makes sense to avoid repairing read errors of a drive being replaced. I've put it on my todo list, but I'm afraid I haven't looked at that much lately.

However any writes to the array would have to be written to both drives. If we only ever wrote to the replacement drive, it would be very hard to keep track of where to read from.
So if you have a device that cannot survive being written to, then you would need to avoid writing to the array completely.

NeilBrown