Skip to contentSkip to conversation

Please, add optional RAID1 feature (= chunk checksums) to make it more robust

RFE Stan Hoeppner 6 participants 12 messages Jul 23, 2012 – Jul 27, 2012 1 message not in the archive MD RAID

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

Please keep discussion on list. This is probably an MUA issue. Happens to me on occasion when I hit "reply to list" instead of "reply to all".
vger doesn't provide a List-Post: header so "reply to list" doesn't
work and you end up replying to the sender.

Quoted 53 lines · Jaromir Capik
Quoted 4 lines
Quoted 3 lines

I admit, that the problem could lie elsewhere ... but that doesn't change anything on the fact, that the data became corrupted without me noticing that.

Quoted 11 lines

The key here I think is "without me noticing that". Drives normally cry
out in the night, spitting errors to logs, when they encounter problems.
You may not receive an immediate error in your application, especially
when the drive is a RAID member and the data can be shipped regardless
of the drive error. If you never check your logs, or simply don't see
these disk errors, how will you know there's a problem?

Hello Stan.

I used to periodically check logs as well as S.M.A.R.T. attributes. And I believe I've already mentioned two of the cases and how I finally discovered the issues. Moreover I switched from manual checking to receiving emails from monitoring daemons. And even if you receive such email, it usually takes some time to replace the failing drive. That time window might be fatal for your data if junk is read from one of the drives and when it's followed by a write. Such write would destroy the second correct copy ...

Quoted 5 lines

Likewise, if the checksumming you request is implemented in md/RAID1, and your application never sees a problem when a drive heads South, and
you never check your logs and thus don't see the checksum errors...

You wouldn't have to ... because the corrupted chunks would be immediately resynced with good data and you'll REALLY get some errors in the logs if the harddrive or controller or it's driver doesn't produce them for whatever reason.

Quoted 4 lines

How is this new checksumming any better than the current situation?
The
drive is still failing and you're still unaware of it.

Do you believe, that other reasons of silent data corruptions simply do not exist? Try to imagine a case, when the correct data aren't written at all to one of the drives due to a bug in the drive's firmware or due to a bug in the controller design or due to a bug in the controller driver or due to other reasons. Such bug could be tiggered by anything ... it could be a delay in the read operation when the sector is not well readable or any race condition, etc. Especially new devices and their very first versions are expected to be buggy. Checksuming would prevent them all and would make the whole I/O really bulletproof.

#2 lore raw

Hello Stan.

I received your reply without having the Linux RAID list in Cc and thus I was unsure if you wanna discuss that privately or not. I always choose reply to all unless I really want to remove some of the recipients :]

Cheers,
Jaromir.

Quoted 75 lines

Please keep discussion on list. This is probably an MUA issue.
Happens
to me on occasion when I hit "reply to list" instead of "reply to all".
vger doesn't provide a List-Post: header so "reply to list" doesn't
work and you end up replying to the sender.

Quoted 64 lines · Jaromir Capik
Quoted 6 lines
Quoted 5 lines

I admit, that the problem could lie elsewhere ... but that doesn't
change anything on the fact, that the data became corrupted without
me noticing that.

Quoted 12 lines

The key here I think is "without me noticing that". Drives normally
cry
out in the night, spitting errors to logs, when they encounter problems.
You may not receive an immediate error in your application, especially
when the drive is a RAID member and the data can be shipped regardless
of the drive error. If you never check your logs, or simply don't see
these disk errors, how will you know there's a problem?

Hello Stan.

I used to periodically check logs as well as S.M.A.R.T. attributes. And I believe I've already mentioned two of the cases and how I finally discovered the issues. Moreover I switched from manual checking to receiving emails from monitoring daemons. And even if you receive such email, it usually takes some time to replace the failing drive. That time window might be fatal for your data if junk is read from one of the drives and when it's followed by a write. Such write would destroy the second correct copy ...

Quoted 8 lines

Likewise, if the checksumming you request is implemented in md/RAID1,
and your application never sees a problem when a drive heads South,
and
you never check your logs and thus don't see the checksum errors...

You wouldn't have to ... because the corrupted chunks would be immediately resynced with good data and you'll REALLY get some errors
in the logs if the harddrive or controller or it's driver doesn't produce them for whatever reason.

Quoted 5 lines

How is this new checksumming any better than the current situation?
The
drive is still failing and you're still unaware of it.

Do you believe, that other reasons of silent data corruptions simply
do not exist? Try to imagine a case, when the correct data aren't written at all to one of the drives due to a bug in the drive's firmware
or due to a bug in the controller design or due to a bug in the controller driver or due to other reasons. Such bug could be tiggered
by anything ... it could be a delay in the read operation when the sector is not well readable or any race condition, etc. Especially new devices and their very first versions are expected to be buggy. Checksuming would prevent them all and would make the whole I/O really bulletproof.

#3 lore raw
Quoted 6 lines · Jaromir Capik

Hello Stan.

I received your reply without having the Linux RAID list in Cc and thus I was unsure if you wanna discuss that privately or not. I always choose reply to all unless I really want to remove some of the recipients :]

When you saw the same message also arrive via the list, that wasn't a clue. ;)

Signature · 1 line
Stan
#4 lore raw
In reply to Jaromir Capik

Hi all,

actually, what you would like to do is already possible, albeit it will kill the performance of a rotating, mechanical, HDD.
With SSD might work better.

If you take an HDD and partition it, let's say with 100 partitions (GPT will be required), then you can build a RAID-6 using this 100 partitions, having a redundancy of 2%. Taking two, or more, of such configured RAID-6, it will be possible to build (with them) a RAID-1 (or else).

If a check of this RAID-1 returns mismatches, it will be possible to check the single devices and find out which is not OK.
With RAID-6 (per device), and a bit of luck, it will be possible to fix it directly.

Of course a lot of variables are tunable here. For example the number of partitions, the chunk size, or even the fact that with X partitions it could be possible to build more than one RAID-6, increasing the effective redundancy.

All with the performance price I mentioned at the beginning.

bye,

Signature · 1 line
piergiorgio
#5 lore raw

yeah, i think this too, but IMO Jamiro exposed a specific scenario, let´s get back to it and after check a generic scenario,
he is using small computers (i don´t know if it´s ARM or X86) with space to only 2 disks (i told him to use raid5 or raid6 because the checksums but he don´t have space for >=3 disks in computer case, maybe if we could run raid5 with 2 disks could help... or 1 disk... just a idiot idea, but this could help...)
i don´t know the real scenario, i think he will not use it in 100partitions, maybe 4 or 5 partitions, and performance to be a second option, security is priority here
in the implementation of this new layer (maybe like LINEAR, MULTIPATH or another not raid level) we could focus on security and after performace

just some ideas...

2012/7/23 Piergiorgio Sartor <[email protected]>

Quoted 33 lines

Hi all,

actually, what you would like to do is already possible, albeit it will kill the performance of a rotating, mechanical, HDD.
With SSD might work better.

If you take an HDD and partition it, let's say with 100 partitions (GPT will be required), then you can build a RAID-6 using this 100 partitions, having a redundancy of 2%. Taking two, or more, of such configured RAID-6, it will be possible to build (with them) a RAID-1 (or else).

If a check of this RAID-1 returns mismatches, it will be possible to check the single devices and find out which is not OK.
With RAID-6 (per device), and a bit of luck, it will be possible to fix it directly.

Of course a lot of variables are tunable here. For example the number of partitions, the chunk size, or even the fact that with X partitions it could be possible to build more than one RAID-6, increasing the effective redundancy.

All with the performance price I mentioned at the beginning.

bye,

Quoted 7 lines

--

piergiorgio
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

Been mulling this problem over and I keep getting hung up on one problem with ECC on a two disk RAID1 setup.

In the event of silent corruption of one disk, which one is the good copy?

It works fine if the ECC code is identical across both mirrors. Just checksum both chunks and discard the incorrect one.

It also works fine if the ECC codes are corrupted but the data chunks are identical. Discard the bad checksum.

What if the corruption goes across several sectors and both data & ECC chuncks are corrupted? Now you're back to square one.

Signature · 4 lines
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
#7 lore raw

that´s the point...
in few words....

2012/7/23 Drew <[email protected]>:

Quoted 7 lines

Been mulling this problem over and I keep getting hung up on one problem with ECC on a two disk RAID1 setup.

In the event of silent corruption of one disk, which one is the good copy?

It works fine if the ECC code is identical across both mirrors. Just checksum both chunks and discard the incorrect one.

nice we can recover data =)

Quoted 3 lines

It also works fine if the ECC codes are corrupted but the data chunks are identical. Discard the bad checksum.

nice we can recover data too =)

Quoted 3 lines

What if the corruption goes across several sectors and both data & ECC chuncks are corrupted? Now you're back to square one.

report badblock to upper layer (file system or a mdraid or lvm or anyother process)
the same should occur on harddisk with known corrupted data but in this case we know that´s wrong and we can report it! =) that´s the nice part!
very different from "silent data corruption" where no alert or warning or error is reported, that´s the today bad part...
ok... you will tell about NEC report? they have this 'software security' in firmware (i think we could make something similar)

Quoted 9 lines

--
Drew

"Nothing in life is to be feared. It is only to be understood." --Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 6 lines
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#8 lore raw
In reply to Drew
Quoted 2 lines · Drew

What if the corruption goes across several sectors and both data & ECC chuncks are corrupted?

What if the 'silent' corruption spans 10 million sectors (~5GB)?

Signature · 1 line
Stan
#9 lore raw

10 milion bad blocks (5gb of lost information)
check that we are talking about one device (it can be a disk, a partition, a raid1 a raid0, nbd, drbd, or anything else)

2012/7/24 Stan Hoeppner <[email protected]>:

Quoted 14 lines · Drew
Quoted 3 lines

What if the corruption goes across several sectors and both data & ECC chuncks are corrupted?

What if the 'silent' corruption spans 10 million sectors (~5GB)?

--
Stan

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Roberto Spadim
Spadim Technology / SPAEmpresarial
#12 lore raw
In reply to Roberto Spadim
Quoted 7 lines

yeah, i think this too, but IMO Jamiro exposed a specific scenario, let´s get back to it and after check a generic scenario, he is using small computers (i don´t know if it´s ARM or X86) with space to only 2 disks (i told him to use raid5 or raid6 because the checksums but he don´t have space for >=3 disks in computer case, maybe if we could run raid5 with 2 disks could help... or 1 disk... just a idiot idea, but this could help...)

I believe, that Piergiorgio meant something else. It was about creation of a high number of small partitions on two physical drives and then build a RAID6 array on top of them. But that's really a bit overkill :]

Quoted 56 lines

i don´t know the real scenario, i think he will not use it in 100partitions, maybe 4 or 5 partitions, and performance to be a second
option, security is priority here
in the implementation of this new layer (maybe like LINEAR, MULTIPATH or another not raid level) we could focus on security and after performace

just some ideas...

2012/7/23 Piergiorgio Sartor <[email protected]>

Quoted 33 lines

Hi all,

actually, what you would like to do is already possible, albeit it will kill the performance of a rotating, mechanical, HDD.
With SSD might work better.

If you take an HDD and partition it, let's say with 100 partitions (GPT will be required), then you can build a RAID-6 using this 100 partitions, having a redundancy of 2%. Taking two, or more, of such configured RAID-6, it will be possible to build (with them) a RAID-1 (or else).

If a check of this RAID-1 returns mismatches, it will be possible to check the single devices and find out which is not OK.
With RAID-6 (per device), and a bit of luck, it will be possible to fix it directly.

Of course a lot of variables are tunable here. For example the number of partitions, the chunk size, or even the fact that with X partitions it could be possible to build more than one RAID-6, increasing the effective redundancy.

All with the performance price I mentioned at the beginning.

bye,

Quoted 8 lines

--

piergiorgio
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

--
Roberto Spadim
Spadim Technology / SPAEmpresarial

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw
In reply to Drew
Quoted 6 lines · Drew

Been mulling this problem over and I keep getting hung up on one problem with ECC on a two disk RAID1 setup.

In the event of silent corruption of one disk, which one is the good copy?

Quoted 8 lines

It works fine if the ECC code is identical across both mirrors. Just checksum both chunks and discard the incorrect one.

It also works fine if the ECC codes are corrupted but the data chunks are identical. Discard the bad checksum.

What if the corruption goes across several sectors and both data & ECC chuncks are corrupted? Now you're back to square one.

I know I'm a bit late to this discussion, and I know very little about the code level/etc... however, I thought the whole point of the checksum is to determine that the data + checksum do not match, therefore the data is wrong and should be discarded. You would re-write the data and checksum from another source (ie, the other drive in RAID1, or other drives in RAID5/6 etc...).

ie, it should be treated the same as a bad block / non-readable sector (or lots of unreadable sectors....)

Regards,
Adam

Signature · 3 lines
Adam Goryachev
Website Managers
www.websitemanagers.com.au
#11 lore raw

IMO
the first idea was put this only in md_raid1,
the second idea was a new md device (maybe a md_security or md_redundancy or md_conformity or any another beautiful name...) in this case the device will do a checksum and report 'badblock' (maybe the right word could be badchecksum), that the option that i agree, since we could do it to any device, doesn´t matter if it´s a raid1 or raid4 or raidXYZ

just to explain words:
badchecksum -> we can read data but we know that it doesn´t match checksum, or checksum doesn´t match data
badblock -> we can´t read, because 'physical block' reported as bad

for mirror layers we could do more than just know if we have a badchecksum (this is not good, check...)
in the case of all mirrors reporting badchecksum, we could read data (doesn´t matter the badchecksum information) and vote to the data that have more repeated values and resync data from this new 'primary information', for example:

/dev/md0 -> disks: /dev/sda /dev/sdb /dev/sdc
original data: block= "ABCDEF", checksum=5

for /dev/sda: block="ABCDEH", checksum=5 (badchecksum) for /dev/sdb: block="ABCDEG", checksum=5 (badchecksum) for /dev/sdc: block="ABCDEG", checksum=5 (badchecksum)

in this case, we could elect "ABCEG" (2 repeats) as the 'new data' recalcule the checksum and sync data to all devices (check that we coudl have a a 1 repeat for each device and couln´t elect the new primary information source...)

well this ideal could be good and bad... for application level that´s bad, since we done a silent data corruption..., but maybe for a recovery tool this could be good since we corrected the checksum... maybe this could be a tool of the new device level... (CHECKS and REPAIRS like mdadm do today with echo "check"> /sys/block/md0/md/sync_action, or echo "repair" > /sys/block/md0/md/sync_action )

i don´t like the idea of put the 'recovery' inside md_raid1, i prefer a badblock per device (doesn´t matter if it´s a badblock or badchecksum..), and don´t do any 'silent recover' of information at raid level, to do a checksum correction or data correction, maybe leave this problem to a external tool, like harddisks have badblocks tools, we could have a badblock tool too

going back to our new device,
check that a data corruption (silent or not) is a data corruption, and in any case (checksum corruption or data corruption) we have a bad device, and we should report that we have a badblock in that read operation
the best we could do when we have a badchecksum is reread many times and recalculate the checksum, if the good matches are bigger than X% (maybe 80%) we could send a write to device (to ensure that disk wrote the good value to disk again) and do a new read if that match (only with 1 read) that´s nice we done a good 'silent' repair with a 'good' (80% of probabilty of good) data, this could be an option of the new device to the new device ("silent recover")

i think that´s all we could do of interesting =)
maybe in some future... we could do a realoc?! like ssd do... mark the badchecksum block as badblock (inside a badblock list) and sync the data inside current badblock, to a new never used block (we could alloc 1% of device to use as never used blocks), this could be good for data security, but administrator should read logs to ensure that system don´t run with badblocks....

that´s are the ideas of the 'new' security device level that i could imagine... thanks guys :)

2012/7/27 Adam Goryachev <[email protected]>:

Quoted 40 lines · Drew
Quoted 6 lines

Been mulling this problem over and I keep getting hung up on one problem with ECC on a two disk RAID1 setup.

In the event of silent corruption of one disk, which one is the good copy?

Quoted 8 lines

It works fine if the ECC code is identical across both mirrors. Just checksum both chunks and discard the incorrect one.

It also works fine if the ECC codes are corrupted but the data chunks are identical. Discard the bad checksum.

What if the corruption goes across several sectors and both data & ECC chuncks are corrupted? Now you're back to square one.

I know I'm a bit late to this discussion, and I know very little about the code level/etc... however, I thought the whole point of the checksum is to determine that the data + checksum do not match, therefore the data is wrong and should be discarded. You would re-write the data and checksum from another source (ie, the other drive in RAID1, or other drives in RAID5/6 etc...).

ie, it should be treated the same as a bad block / non-readable sector (or lots of unreadable sectors....)

Regards,
Adam

--
Adam Goryachev
Website Managers
www.websitemanagers.com.au

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 6 lines
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html