OT: silent data corruption reading from hard drives

Quick intro: Last year I was having problems with an md array continuously having a mismatch_cnt in the tens of thousands, inexplicably. After a week or two of hardware swapping and such, I narrowed it down to bad reads of the hard drive block devices. I used scripts that would repetitively do something like this on all my drives:

      dd if=/dev/sdk1 bs=1024 count=50000000 |md5sum -b

Some devices would intermittently get different results. I ended up resolving (?) it by replacing the cheapo (Syba) SATA controller cards with other cheapo (Rosewill) ones. I've been fine for about a year since then.

But now it's just started happening again. Although this isn't an md question per se, I'm hoping some of you raid/kernel/storage gurus can give me some tips on how to trace this in a better way than my haphazard method last year. Is there any way to detect these bad reads when they
happen? (Apparently not?) What about finding out if the cause is the
motherboard, the controller card, the device driver, or the kernel? (Besides swapping hardware?) Can the md layer help out in this regard?
Are there known bugs or hardware nuances that relate to this? Is
silent data corruption like this simply to be expected when using cheap commodity hardware?

Thanks for reading...

matt

Syba was mentioned in a bad context last time we discussed this here, somewhere down the thread: http://comments.gmane.org/gmane.linux.raid/33346
I don't remember if you have participated in that discussion, if not, have an entertaining read. Is your new Rosewill card also Silicon Image based?

I'd say no. There are known good and known bad hardware, where both good and bad is most easily measured by the amount (or lack of) problem reports you hear or can find. One method to do that is asking a searching engine about "<chip name> data corruption".

With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

When you pay less than 1/4th the price of one drive for the controller card directing all your drives, is it really necessary to ask this question?

Two large pizzas cost more than a Syba/Rosewill/Koutech/etc SATA card. The pizza is consumed in one night, maybe some for breakfast. Would you trust a day worth of pizza with your RAID? With your data?

One tank of gas for the average car today costs $50, the same as two of these SATA cards. The gas is gone in a week or two. You want the cards to run for 2-4 years.

Your drives cost anywhere from $400-$1000, yet your controller maybe $50. Do I really need to say any more? Spend $150-250 on a decent SAS/SATA controller such as an LSI or Adaptec and you won't have to worry about this kind of thing.

Stan

well this will work on systems that have pci-express or pci slots for systems that don´t have slots like arm systems, i don´t know if this could work... maybe in some future we will have arm devices running NAS solutions, and a more secure system is nice...

2012/8/1 Stan Hoeppner <[email protected]>:

Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

just an example...
this arm device have USB3.0 and could plug external drivers with a fast speed

http://gizmodo.com/5646025/marvells-15ghz-triple-core-arm-processor-can-play-140-hours-of-music-on-one-charge

it´s a nice tool to implement since we don´t know what´s the quality of USB external drives, and we could use it as a home NAS server

2012/8/1 Roberto Spadim <[email protected]>:

Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

more? Spend $150-250 on a decent SAS/SATA controller such as an LSI or Adaptec and you won't have to worry about this kind of thing. On a related note, is there a specific model card you would suggest that fits the following:
a) 4 or 8 ports
b) PCI (for older systems)
c) affordable (note, not cheap, but not expensive)
d) does not need to support RAID since we will use software raid anyway

I am always unsure, and see very cheap SATA cards (around $50) and worry as you suggest, and very expensive (around $1000), and feel that is a waste of money for what should be a fairly simple card.

Also, would you use on board controller, or prefer to always use an external controller? Or is it worthwhile using a combination (in case of controller failure)?

Thanks for your opinions.

Regards,
Adam

Adam Goryachev
Website Managers
www.websitemanagers.com.au

check this link at oracle
https://oss.oracle.com/projects/data-integrity/

i don´t know if this is a good solution, but i think it´s a start point...

2012/8/1 Adam Goryachev <[email protected]>:

Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

If you're stuck with PCI I'd stick with a more recognized vendor such as Promise Tech. They still sell a 4 port PCI card. $70 USD at Newegg.

A "very cheap" SATA card is a 2-port SiI 3512 based card--Syba, Rosewill, Koutech, etc-- for $15 on sale, $20 regular price, all prices USD. A $50 card with the same port count *should* hopefully be much higher quality.

Motherboard-down SATA controllers are usually fine for md/RAID as they're part of an Intel or AMD chipset, and of high quality. But this also depends on the quality of the board as well--there have been plenty of boards with good chipsets but also a buggy BIOS. Overall they're usually much better than a $20 card.

In 20+ years of computing I've never had an HBA or RAID card fail on me.
I've had plenty of mobos go toes up and plenty of NICs and VGA cards
burn out. I have a 128MB 3-channel AMI MegaRAID 428 UW-SCSI built in 1998 that still works and an Adaptec 1542 ISA SCSI card from 1993 that still works. I've tossed piles of dead mobos over the same period.

Stan

A good solid card I've found is the Promise SATA300TX4. 4port SATA2 controller that fits in a standard PCI slot.

Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

While you're at it also don't forget to get yourself some quality Ethernet cabling, you don't want any data corruption on your LAN, do you
http://www.amazon.com/Denon-AKDL1-Dedicated-Link-Cable/dp/tech-data/B000I1X6PM/ref=de_a_smtd

It's not a matter of "just pay more and all your problems will be solved".

There are some really crappy controllers even in the "enterprise" range (e.g. those using the mv_sas driver). And there's a lot of $15 models giving zero problems whatsoever (e.g. the ahci-compliant Marvell 88SE912x, JMicron JMB36x).

Also, if you only use the mdadm software RAID, getting an enterprise hardware RAID controller is truly a waste of money, unless you really need the extra port density.

With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

The hyperbole isn't necessary Roman, and detracts from your argument.

I didn't say, nor imply such a thing. You missed my point entirely about dollar amount. It had two aspects:

1. Some people are willing to spend hundreds or thousands on drives, then hook them to $15 controllers without giving that ratio a thought, no reflection upon the sanity of it.

2. $150-250 is the price range of about a dozen high quality HBAs on the market. I don't have the time to go out and identify and then list them all here. Mentioning the price range simply makes sure people will be looking at the right cards.

First, there are not "enterprise" controllers that use the Marvell SAS chip, period. It's a low/mid cost, entry level 8 port SAS ASIC. The cards based on it a not crappy, but actually pretty decent. They run very well with Windows, *BSD, SCO, NetWare, and other OSes. It just happens that the Linux driver, yes mv_sas, totally sucks, despite continuous development. In fact, hop on the linux-scsi mailing list and you can chat with who I believe is the primary developer of that driver, Xiangliang Yu. He works for Marvell.

You're confusing SATA controller ASICs, which may be of high quality and have quality drivers, with HBAs, which are the finished products that incorporate these ASICs. The HBAs also incorporate many other ICs and analog parts, which are often of questionable quality in the $15-30 range of HBAs. You're buying an HBA, not just the SATA ASIC. QC/QA in the manufacture of HBAs that retail for these prices is very low compared to more expensive models. And you can't get support from the manufacturer. Call Syba, Rosewill, Koutech, etc, and see how helpful they are--if you can get a hold of anyone that is. Then call LSI.

That logic is flawed. If you have an enterprise hardware RAID controller you're not going to use md/RAID, unless you're stitching LUNs together with linear or RAID0, as I often do. In fact that's my most frequent use of md/RAID--concatenation of hardware RAID or SAN LUNs.

If you're talking about enterprise class HBAs, your logic is still flawed. Not only are they higher quality and performance, but you can daisy chain multiple expanders to them. The $175 LSI 9211-4i is a 4-port (single 8087) enterprise HBA that can attach up to 256 SAS/SATA drives using fanned or daisy chained expanders. You simply can't do this with SATA port multipliers, which still tend to have compat issues quite often.

The most you can get is 5 downstream SATA links per PMP. And the PMPs, the SiI based models, are almost $100 a pop. So a 4 port SiI HBA with 4 SiI PMPs gets you 20 drives connected for ~$450 at PCIe 1.0 x1, for 250MB/s one way. Or you can buy the LSI 9211-4i and the Intel 24-port SAS expander for ~$400 at PCIe 2.0 x4, for 2GB/s one way.

In this case not only is the "enterprise" HBA solution cheaper, it's also without compatibility issues or bugs, has excellent support, and has an 8x faster bus, and a much faster ASIC able to handle over 20x the IOPS. Sure, there are Marvell based PCIe x4 cheapo HBAs, Rosewill makes a 4 port model, but the Marvell SATA chips don't work with the SiI PMPs very well, if at all, and AFAIK nobody offers a standalone Marvell PMP, nor any other PMP. SiI pretty much has the PMP market to itself.

Stan

Why should there be any reflection? There is no no automatic "Hmm, this thing costs $XXX, so the other thing should also cost $XXX" rule.

$15 is simply the right price for a perfectly working 2 port SATA controller (based on some of those chip models I named) on the market.

The rule is not "buy expensive", it's "avoid buying (known-)broken". Maybe it's more difficult to come across a broken card in the expensive segment. Maybe it's not. In any case my suggestion is to just do some research before you buy, identify broken chips and cards, avoid those, and save yourself those $185 or whatever.

The end result for the user is the same, they accidentally buy that card, they use GNU/Linux, they have to use that sucky driver. The best choice is to avoid buying the card in the first place, but how do you know you should, if you didn't do the research (see above) and just looked at the price tag.

I do not recommend using hardware RAID. It locks you into one card/vendor, usually is much less flexible than mdadm, and often even provides lower performance. See http://linux.yyz.us/why-software-raid.html

With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

You're Russian. In the U.S. there has been an axiom for generations that says "You get what you pay for". This isn't always true, but more often than not.

If it works out of the box, or after a month, yes. But many of these $15 cards are DOA or fail shortly after entering service, due to poor QC. This is one of the many problems one avoids with higher quality boards, which are usually more expensive. A twist on the above axiom, "pay more, get more".

Of course not.

This is not knowable but for compatibility. All the $15 cards have a higher DOA/failure rate than the high quality cards. By that logic one should always avoid the cheap cards. Coincidentally, this is one of the reasons I recommend against them. One is gambling whether his new Syba is going to be DOA. You can be pretty damn sure your new LSI won't be DOA or fail after one week.

With hundreds of SAS/SATA HBAs on the market, as with just about any product, most people do not have the time to research every small purchase. Again, the axiom comes into play: "You get what you pay for".

You ask around, which smart people do, on various lists. Or they suffer.

You stated the ASIC itself was junk. I was simply correcting that point. Note what I recently stated on the linux-ide list:

I've made the same statement dozens of times on the various LKML sub lists.

Of course not. You don't work in or support an environment that benefits from it. Tell me how well md/RAID works for you the next time you need to setup 4 IBM Bladecenters each w/14 dual socket blades booting ESX from FC SAN, and hosting hundreds of Linux guests.

I can have a Nexsan E60+E60x w/120 SAS drives configured into 6x 20 drive RAID10 volumes, carve up all my LUNs and export them, booting the nodes within a few hours, including racking the two chassis and inserting all the drives.

How many days (or weeks) would it take you part spec and order, build, test, verify, and put into production a roughly equivalent 120 SAS drive self built FC LUN serving SAN array, using md/RAID with DAS controllers/drives on the back end? A long damn time, if you could even do it. There are no guides out there for this scenario, though there are a few for iSCSI.

This is precisely why proprietary hardware RAID sells, and precisely why md/RAID isn't the solution for everything, but quite the opposite. In fact, md/RAID sees almost zero use in the business/corporate world in the U.S., probably the same in other countries. md/RAID is great for what it does, but there is so much it simply cannot do, from the storage network standpoint, or simply can't do without a boat load of custom development.

Stan

While that *can* be the case, I haven't ever seen md beating a HW RAID adapter with battery-backed write-cache in certain metadata-intensive workloads, so as usual it depends on what kind of HW raid you have, and what performance you measure. Broad descriptions are not very helpful.

regards,
iustin

Thanks for the replies, everyone. I didn't expect the debate on controller price ranges, though. :)

My main concern was not really the price of the hardware ("OMG why doesn't my $15 card work?!?"), but that the entire setup, hardware + software, allows silent data corruption to occur. It's not hard to imagine that something similar might happen with an $800 controller card, for whatever reason. And how would the user know, unless they noticed bad data, or happened to look at the mismatch_cnt?

(And in my case, I think that, once the weekly raid-check runs and updates mismatch_cnt, it's already too late, as the check process has potentially rewritten bad data to the drives based on the bogus reads.
Is that correct? My array is RAID-6, btw.)

Anyway, I'm going give up on SI-based controllers for now and try a StarTech PEXSAT34, as the Marvell 88SE9128 is supported natively by the kernel's AHCI driver.

Thanks,
matt

well i think the integrity is know, but it´s not fully needed since the security isn´t a problem we can buy secure sata/sas controlers/disks
the main problem will be in some days when we are using SoC systems and we only have USB to connect a harddrive... maybe when this become more popular we will see a development of a module to have data integrity (silient corruption detection and maybe repair)

2012/8/2 <[email protected]>:

Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

The only ways I know of to currently detect/repair silent data corruption are via the use of T10-DIF on SAS drives with 520-byte sectors and embedded per block CRCs (bytes 513-520) or via a patented algorithm used in a commercial Linux software RAID product (www.streamscale.com).

Neither approach is cost effective for small or personal use RAID applications.

------------------------------
Jeff Johnson
Manager
Aeon Computing

[email protected]
www.aeoncomputing.com
t: 858-412-3810 x101   f: 858-412-3845
m: 619-204-9061

/* New Address */
4170 Morena Boulevard, Suite D - San Diego, CA 92117

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Well, you can simply use BTRFS RAID1/RAID10 (RAID5/6 to come some time soon). It has per block checksums and auto-healing from other drive(s) if data on some drive turns out not to match the checksums anymore.

With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

btrfs and jfs have silent data corruption protection, but it could be nice to have this at block layer, since some databases (innodb at mysql) can use block device to allocate information (as a performace boost), and since i know, it don´t have silent data corruption protection...
a solution at block device is better than filesystem? yes and no, but at block device we can use with any filesystem, even with fat16 or other old filesystem
we could add more features in future (like jornaling) but i think just data integrity layer is nice... there´s others features like reallocation (when write to a silent write detected block, realloc it to another block like good SSD do)

2012/8/2 Roman Mamedov <[email protected]>:

Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

yes, that´s what i know too
the point is develop a algorithm to make this in block level like a md device it write to disk and check if it was write ok
it read from disk and check parity, if it´s wrong read again, after X times reading wrong report it as a badblock (now a silent corruption becomes a well know badblock disk corruption)

i think that´s all... just need someone to implement it, some ideas:

// create:
mdadm --create /dev/md0 --level=integrity /dev/sda1

// check:
echo "check" > /sys/block/md0/md/sync_action

// repair: (like harddisks badblock recovery, mark the block as badblock, and it will never be used)
echo "repair" > /sys/block/md0/md/sync_action

when using it with raid1 raid10 or any other mirror mdadm device, the device with silent corruption can be reported (checksum don´t match data) as badblock (mdadm have news features to deal with badblocks...) the disk with badblock will be ignored and read will be done in next good disk...

we could use a md5 checksum (32bits?! - 4 bytes) and data with less information, example today block is 512bytes.. change it to 508bytes of data + 4 bytes of checksum

i think that´s all...

2012/8/2 Jeff Johnson <[email protected]>:

Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

With Fedora, I get an email Monday morning, after the raid-check, which warns of a non-zero mismatch_cnt.

Ah, okay, good. I thought I'd read here a while back that "check" & "repair" do the same thing.

Good point. It does not. Might be time for me to upgrade to a mobo with ECC support.

klogd is not running, but I think the latest rsyslog handles the kernel messages. There was nothing in the logs related to my corruption issues, however.

Thanks, this is good info, and kind of gets at my thinking when I posted my initial question. In a typical consumer hardware setup, with a current linux kernel, do I have to take any steps to enable these kinds of checks? Can the kernel log any failed checks at the levels you mention? I guess my confusion with my silent data corruption issues stems from my naive assumption that all the various data transfers happening would have some way of detecting or flagging the bad reads as they happened.

But maybe as you suggest, my issue is related to memory, and ECC might help in the future?

Thanks,
matt

Hi Matt,

I now see that I hit the wrong "reply" button--my apologies to the list.
You've quoted the important stuff, though, so I won't resend.

Good to know. I'm on gentoo, and I use my own script in logwatch, so I'm not familiar with the various distros practice on this.

In my opinion, any corruption noticed in a non-ECC system is most likely due to the RAM. You really need to run memtest86 on your system, preferably for 24 hours or more.

I meant logging of ECC RAM correction events (warnings) and uncorrectable errors. Your kernel has to support that. I would be shocked if Fedora didn't support it. You also need the user space "mcelog" package. "mce" ==> "Machine Check Exception"

You won't get ram corruption error reports if you don't have ECC ram. Data transfer errors between CPU and chipset might generate machine check exceptions, but if not recoverable, the machine just dies. Errors on PCIe lanes and SATA/SAS connections cause retransmissions until success or the driver times out. That would show up in dmesg.

You don't have to guess. Boot into memtest86 and see. And yes, any machine handling data you really care about should have ECC ram.

Phil

[ ... ]

That's pretty common, but many disk drive models also have bugs, and most hw RAID host adapters have many (terrible) bugs.

Even that is not conclusive. Some "memory" errors are due to activity/noise spikes on the PCI/PCIe bus due to hw bugs or poorly electrically designed cards.

A warning to the masses: USB mass storage is weak as to this and in particular as to error recovery, and most USB chipsets (especially USB-drive ones, but also motherboard ones) are massively buggy.

... should have end-to-end verification, that is the data itself should be checksummed at least to detect corruption. For example by putting it into checksummed containers (even just ZIP without compression).

Oh yes, and any machine should have ECC RAM as the cost is really modest. Unfortunately the usual evil marketers like to segment artificially the market into cheap stuff without ECC and premium stuff with ECC, and will not put ECC into cheap stuff to avoid tempting business customers to buy it instead of the premium stuff.

While I agree that all machine's should have ECC Ram (there are still some people think its not worth it), last time I checked on newegg, I found ECC prices not that much higher. My servers both run happily with ECC ram.

As for data corruption, I've also been there and know it simply just happens. Yes I had shitty IDE drives on a shitty 'rocketraid 404' controller, but that's no excuse to simply assume all data will always be right. Maybe in a few years from now, we'll have some 'open cores' for properly designed almost bug free hardware :)

well this discussion is very nice...
but let´s get back to our problem?

we can have data integrity with high cost (for some guys it´s high for others it doesn´t matter), using 'especial' hardware...
what we can do to get integrity higher with a low cost hardware? any software solution could gime more data integrity with more CRC or other method of error check? could we develop it at block device level (md device)?
i think that´s the point
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html