Skip to contentSkip to conversation

data corruption after rebuild

DISCUSSION Pavel Herrmann 3 participants 11 messages Jul 19, 2011 – Jul 20, 2011 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi,

I have a big problem with mdadm, I removed a drive from my raid6, after replacing it the raid started an online resync. I accidentally pushed the computer and it shut down (power cord moved), and after booting it again the online resync continued.

the problem is that the rebuilt array is corrupted. most of the data is fine, but every several MB there is an error (which doesn't look like being caused by a crash), effectively invalidating all data on the drive (about 7TB, mainly HD video)

I do monthly scans, so the redundancy syndromes should have been up-to-date, the array is made of 8 disks, the setup is ext4 on lvm on mdraid

is there anything to solve this? or at least ideas what happened?

#1 lore raw

Hello,

Quoted 4 lines · On Tue, 19 Jul 2011 15:55:35 +0200 Pavel Herrmann <[email protected]>…

the problem is that the rebuilt array is corrupted. most of the data is fine, but every several MB there is an error (which doesn't look like being caused by a crash), effectively invalidating all data on the drive (about 7TB, mainly HD video)

Which model of SATA controller/HBA do you use?

Kernel version, mdadm version?

Anything unusual in SMART reports of any of the drives (e.g. a nonzero UDMA CRC Error count)?

Quoted 3 lines

I do monthly scans, so the redundancy syndromes should have been up-to-date, the array is made of 8 disks, the setup is ext4 on lvm on mdraid

Did you notice any nonzero mismatch_cnt during those scans?

Signature · 2 lines
With respect,
Roman
#2 lore raw
Quoted 11 lines · On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:

Hello,

On Tue, 19 Jul 2011 15:55:35 +0200

Quoted 4 lines · Pavel Herrmann

the problem is that the rebuilt array is corrupted. most of the data is fine, but every several MB there is an error (which doesn't look like being caused by a crash), effectively invalidating all data on the drive (about 7TB, mainly HD video)

Which model of SATA controller/HBA do you use?

4 drives on ahci (ICH10R), 4 drives on sata_mv (adaptec 1430SA)

Quoted 1 line

Kernel version, mdadm version?

2.6.33-gentoo-r1, mdadm - v3.1.5 - 23rd March 2011

Quoted 2 lines

Anything unusual in SMART reports of any of the drives (e.g. a nonzero UDMA CRC Error count)?

one current-pending-sector on the drive that was removed, and one on one more

Quoted 6 lines
Quoted 4 lines

I do monthly scans, so the redundancy syndromes should have been up-to-date, the array is made of 8 disks, the setup is ext4 on lvm on mdraid

Did you notice any nonzero mismatch_cnt during those scans?

where would I find this?

syslog for last scan is just:

Jul  2 08:40:01 Bloomfield kernel: [83795.157876] md: data-check of RAID array md0
Jul  2 08:40:01 Bloomfield mdadm[2613]: RebuildStarted event detected on md device /dev/md0
Jul  2 09:46:41 Bloomfield mdadm[2613]: Rebuild21 event detected on md device /dev/md0
Jul  2 10:53:21 Bloomfield mdadm[2613]: Rebuild42 event detected on md device /dev/md0
Jul  2 12:00:02 Bloomfield mdadm[2613]: Rebuild61 event detected on md device /dev/md0
Jul  2 13:40:02 Bloomfield mdadm[2613]: Rebuild82 event detected on md device /dev/md0
Jul  2 16:02:46 Bloomfield kernel: [110348.161984] md: md0: data-check done.
Jul  2 16:02:47 Bloomfield mdadm[2613]: RebuildFinished event detected on md device /dev/md0, component device  mismatches found: 72

I presume the nonzero "mismatches found" is a bad thing?

just to mention, all files were fine two dayss ago (I do keep md5sums of all files to check for bit rot)

#3 lore raw
Quoted 6 lines · On Tue, 19 Jul 2011 18:18:56 +0200 Pavel Herrmann <[email protected]>…
Quoted 5 lines · On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:

Hello,

On Tue, 19 Jul 2011 15:55:35 +0200

Pavel Herrmann <[email protected]> wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

the problem is that the rebuilt array is corrupted. most of the data is fine, but every several MB there is an error (which doesn't look like being caused by a crash), effectively invalidating all data on the drive (about 7TB, mainly HD video)

Which model of SATA controller/HBA do you use?

4 drives on ahci (ICH10R), 4 drives on sata_mv (adaptec 1430SA)

OK, one more guess - any Samsung drives, F4EG? http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks

Signature · 2 lines
With respect,
Roman
#4 lore raw
Quoted 10 lines · On Tuesday 19 of July 2011 23:38:56 Roman Mamedov wrote:

On Tue, 19 Jul 2011 18:18:56 +0200

Quoted 4 lines · Pavel Herrmann
Quoted 1 line · On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:

Which model of SATA controller/HBA do you use?

4 drives on ahci (ICH10R), 4 drives on sata_mv (adaptec 1430SA)

OK, one more guess - any Samsung drives, F4EG?
http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks

nope, all disks are WDC_WD20EARS-00MVWB0,
4 were bought about 10 months ago, other 4 about 6 months ago

#5 lore raw
In reply to Roman Mamedov
Quoted 5 lines · On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:
Quoted 3 lines

I do monthly scans, so the redundancy syndromes should have been up-to-date, the array is made of 8 disks, the setup is ext4 on lvm on mdraid

Did you notice any nonzero mismatch_cnt during those scans?

oh crap, the rebuild finished with

Jul 19 09:41:24 Bloomfield mdadm[3996]: RebuildFinished event detected on md device /dev/md0, component device  mismatches found: 3184

this is really bad i presume?

#6 lore raw
Quoted 5 lines · On Tue, 19 Jul 2011 18:35:26 +0200 Pavel Herrmann <[email protected]>…
Quoted 4 lines · On Tuesday 19 of July 2011 21:12:40 Roman Mamedov wrote:
Quoted 3 lines

I do monthly scans, so the redundancy syndromes should have been up-to-date, the array is made of 8 disks, the setup is ext4 on lvm on mdraid

Quoted 7 lines
Quoted 1 line

Did you notice any nonzero mismatch_cnt during those scans?

oh crap, the rebuild finished with

Jul 19 09:41:24 Bloomfield mdadm[3996]: RebuildFinished event detected on md device /dev/md0, component device  mismatches found: 3184

this is really bad i presume?

Well, this basically tells you what you already know - a part of the data you have was corrupted. In this case I think it's 3184 512-byte sectors, which is about 1.6MB total. How it got there and how to prevent that from happening in the future - that's a whole different question.

Signature · 2 lines
With respect,
Roman
#7 lore raw
Quoted 4 lines · On Tuesday 19 of July 2011 22:48:19 Roman Mamedov wrote:

On Tue, 19 Jul 2011 18:35:26 +0200
Well, this basically tells you what you already know - a part of the data you have was corrupted. In this case I think it's 3184 512-byte sectors, which is about 1.6MB total.

the number seems too low to me, I have about 2000 video files larger than 1GB on that array, and every one i tried has been corrupted enough to create almost unstoppable visual artifacts in hi-def video.

Quoted 2 lines

How it got there and how to prevent that from happening in the future - that's a whole different question.

would ZFS in raidz2 mode be much better than raid6+ext4? I understand its not the topic of this list, but file-level checksummed rebuild looks like a nice feature

#8 lore raw
Quoted 6 lines · On Tue, 19 Jul 2011 19:05:39 +0200 Pavel Herrmann <[email protected]>…
Quoted 2 lines

How it got there and how to prevent that from happening in the future - that's a whole different question.

would ZFS in raidz2 mode be much better than raid6+ext4? I understand its not the topic of this list, but file-level checksummed rebuild looks like a nice feature

Personally I prefer to not bother with ZFS, it brings way too many complications into software choice, and I just want to use my favorite GNU/Linux distro and not Solaris, and also not trusting 12 TB of data to a third-party kernel module or a FUSE driver which are barely tested and have uncertain future. I'd put more hope in BTRFS RAID5, but that one is a long way ahead from becoming a viable option too.

Regarding mdadm+raid6, AFAIK it currently does not try to heal itself from silent corruption inside a single chunk, even though that should be possible with RAID6. On a repair, if the data chunks are readable with no I/O error, they are considered to be the golden standard and all parity chunks are simply recalculated from data and overwritten (also incrementing mismatch_cnt, if they changed). So maybe implementing a more advanced repair feature could give protection against silent corruption not much weaker than what is offered by per-file checksumming RAID implementations.

Signature · 2 lines
With respect,
Roman
#9 lore raw
In reply to Pavel Herrmann
Quoted 12 lines · On Tue, 19 Jul 2011 15:55:35 +0200 Pavel Herrmann <[email protected]>…

Hi,

I have a big problem with mdadm, I removed a drive from my raid6, after replacing it the raid started an online resync. I accidentally pushed the computer and it shut down (power cord moved), and after booting it again the online resync continued.

the problem is that the rebuilt array is corrupted. most of the data is fine, but every several MB there is an error (which doesn't look like being caused by a crash), effectively invalidating all data on the drive (about 7TB, mainly HD video)

Quoted 4 lines

I do monthly scans, so the redundancy syndromes should have been up-to-date, the array is made of 8 disks, the setup is ext4 on lvm on mdraid

is there anything to solve this? or at least ideas what happened?

My suggestion would be to remove the drive you recently added and then see if the data is still corrupted. It may not help but is probably worth a try.

There was a bug prior to 2.6.32 where RAID6 could sometimes write the wrong data when recovering to a spare. It would only happen if you were accessing that data at the same time as it was recovery it, and if you were unlucky.

However you are running a newer kernel so that shouldn't affect you, but you never know.

BTW the monthly scans that you do are primarily for finding sleeping bad blocks - blocks that you cannot read. They do check for inconsistencies in the parity but only report them, it doesn't correct them. This is because automatically correcting can cause more problems than it solves.

When the monthly check reported inconsistencies you "should" have confirmed that all the drives seem to be functioning correctly and then run a 'repair' pass to fix the parity blocks up.

As you didn't that bad parity would have created bad data when you recovered.

NeilBrown

#10 lore raw
Quoted 3 lines · On Wednesday 20 of July 2011 16:24:31 you wrote:

My suggestion would be to remove the drive you recently added and then see if the data is still corrupted. It may not help but is probably worth a try.

tried that, did not help, probably due to the finished rebuild that "repaired" all the parity data

Quoted 3 lines

There was a bug prior to 2.6.32 where RAID6 could sometimes write the wrong data when recovering to a spare. It would only happen if you were accessing that data at the same time as it was recovery it, and if you were unlucky.

I was accessing some data (read-mostly), but now the largest undamaged file on the filesystem has just under 3MB - that looks a bit suspicious, as the stripe- width is 6x 512K = 3M

Quoted 11 lines

BTW the monthly scans that you do are primarily for finding sleeping bad blocks - blocks that you cannot read. They do check for inconsistencies in the parity but only report them, it doesn't correct them. This is because automatically correcting can cause more problems than it solves.

When the monthly check reported inconsistencies you "should" have confirmed that all the drives seem to be functioning correctly and then run a 'repair' pass to fix the parity blocks up.

As you didn't that bad parity would have created bad data when you recovered.

see the last part, at this point I would be perfectly OK with 72 damaged blocks, as per the last scan (or even a few hundred, for that matter)

PS: forgot to include maillist