single bit errors on files stored on USB-HDDs via USB2/usb_storage
Hi
I'm using a Bunch auf HDDs in USB-Enclosures for storing files. (currently 38 HDD, with a total capacity of 9,5 TB of which 8,5 TB is used)
After i realised about a year(!) ago that the files copied to the HDDs sometimes aren't identical to the "original"-files i changed my procedured so that each file is MD5 before and after and deleted/copied again if an error is detected.
My averate file size is about 1GB with files from about 400MB to 5000MB I estimate the average error-rate at about one damaged file in about 10GB of data.
I'm not sure and haven't checked if the files are wrongly written or "only" wrongly read back as i delete the defective files and copy them again.
Today i copied a few files back and checked them against the stored MD5 sums and 5 files of 86 (each about 700 MB) had errors. So i copied the 5 files again. 4 of the files were OK after that and coping the last file the third time also resulted in the correct MD5.
This time i kept the defective files and used "vbindiff" to show me the difference. Strangly in EVERY case the difference is a single bit in a sequence of "0xff"-Bytes inside a block of varing bit-values that changed a "0xff" into a "0xf7".
Also interesting is that each error is at a 0xXXXXXXX5-Position
Attached is a file with 5 of the 6 differences named 1-5. Of each of the 5 2x3 lines-blocks the first 3 lines are the original the following 3 lines contain the error in the middle row 6th value.
NEVER did i see any messages in syslog regarding erros or an aborting program due to errors passed down from the kernel or something like that.
Data for the computer/software:
Hardware:
Computer is a Dual P3-933Mhz with 3GB (ECC) SD-RAM, Severworks HE-SL-Chipset Source-HDD is a 200GB S-ATA device connected to a Promise TX-4 using libata.
Destination-HDDs: Several different models in several different enclosures and different chipsets, mostly Genesys Logic)
USB-controller: Currently i use a EHCI/OHCI-NEC-Chipset add-on card but since about 4-5 month ago i used a EHCI/UHCI-VIA-Chipset add-on card with same results.
Software:
Kernel: <What was current 1 year ago> up to 2.6.18, self compiled vanilla kernels.
I haven't tried 2.6.19 and i don't expect any changes from it.
Distribution: Debian SID
I you need any other information i will provide them as good as i can.
Bis denn
Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as bad a concept in Text Editors as it is in women. No, the Real Programmer wants a "you asked for it, you got it" text editor -- complicated, cryptic, powerful, unforgiving, dangerous.
1:
245A E2F0: 0E D9 35 01 00 F4 7B F8 00 00 01 E0 09 00 80 00 ..5...{. ........
245A E300: 0D FF FF FF FF FF FF FF FF FF FF FF FF FF DF FC ........ ........
245A E310: 20 92 50 90 DC F4 0C 1A 1A 18 DB 80 4E 61 25 80 .P..... ....Na%.245A E2F0: 0E D9 35 01 00 F4 7B F8 00 00 01 E0 09 00 80 00 ..5...{. ........
245A E300: 0D FF FF FF FF F7 FF FF FF FF FF FF FF FF DF FC ........ ........
245A E310: 20 92 50 90 DC F4 0C 1A 1A 18 DB 80 4E 61 25 80 .P..... ....Na%.2:
24F9 F770: 00 00 01 E0 09 00 80 00 0D FF FF FF FF FF FF FF ........ ........ 24F9 F780: FF FF FF FF FF FF FC 13 64 0B 38 68 EA A2 11 86 ........ d.8h.... 24F9 F790: 61 7A EE EC ED 1D 6F 31 32 6E 4D D9 B5 31 37 66 az....o1 2nM..17f
24F9 F770: 00 00 01 E0 09 00 80 00 0D FF FF FF FF FF FF FF ........ ........ 24F9 F780: FF FF FF FF FF F7 FC 13 64 0B 38 68 EA A2 11 86 ........ d.8h.... 24F9 F790: 61 7A EE EC ED 1D 6F 31 32 6E 4D D9 B5 31 37 66 az....o1 2nM..17f
3:
20CB C6B0: 00 FB 3F F8 00 00 01 E0 09 00 80 80 0D 21 2A 1B ..?..... .....!*.
20CB C6C0: 65 F1 FF FF FF FF FF FF FF FF 3E C4 BC 2B 39 A4 e....... ..>..+9.
20CB C6D0: 8E 85 50 EB 7B 02 7B 93 79 77 50 EF 60 32 8C 03 ..P.{.{. ywP.`2..20CB C6B0: 00 FB 3F F8 00 00 01 E0 09 00 80 80 0D 21 2A 1B ..?..... .....!*.
20CB C6C0: 65 F1 FF FF FF F7 FF FF FF FF 3E C4 BC 2B 39 A4 e....... ..>..+9.
20CB C6D0: 8E 85 50 EB 7B 02 7B 93 79 77 50 EF 60 32 8C 03 ..P.{.{. ywP.`2..4:
1F13 06B0: 00 00 01 E0 09 00 80 00 0D FF FF FF FF FF FF FF ........ ........ 1F13 06C0: FF FF FF FF FF FF 7F 5C 14 05 F2 9E 90 0F 6F A4 .......\ ......o. 1F13 06D0: B8 10 BF E9 6A 78 A3 00 13 00 FD 9C 00 A5 5B EB ....jx.. ......[.
1F13 06B0: 00 00 01 E0 09 00 80 00 0D FF FF FF FF FF FF FF ........ ........ 1F13 06C0: FF FF FF FF FF F7 7F 5C 14 05 F2 9E 90 0F 6F A4 .......\ ......o. 1F13 06D0: B8 10 BF E9 6A 78 A3 00 13 00 FD 9C 00 A5 5B EB ....jx.. ......[.
5:
1F13 06B0: 00 00 01 E0 09 00 80 00 0D FF FF FF FF FF FF FF ........ ........ 1F13 06C0: FF FF FF FF FF FF 7F 5C 14 05 F2 9E 90 0F 6F A4 .......\ ......o. 1F13 06D0: B8 10 BF E9 6A 78 A3 00 13 00 FD 9C 00 A5 5B EB ....jx.. ......[.
1F13 06B0: 00 00 01 E0 09 00 80 00 0D FF FF FF FF FF FF FF ........ ........ 1F13 06C0: FF FF FF FF FF F7 7F 5C 14 05 F2 9E 90 0F 6F A4 .......\ ......o. 1F13 06D0: B8 10 BF E9 6A 78 A3 00 13 00 FD 9C 00 A5 5B EB ....jx.. ......[.
Quoted 30 lines · Matthias Schniedermeyer
Hi
I'm using a Bunch auf HDDs in USB-Enclosures for storing files. (currently 38 HDD, with a total capacity of 9,5 TB of which 8,5 TB is used)
After i realised about a year(!) ago that the files copied to the HDDs sometimes aren't identical to the "original"-files i changed my procedured so that each file is MD5 before and after and deleted/copied again if an error is detected.
My averate file size is about 1GB with files from about 400MB to 5000MB I estimate the average error-rate at about one damaged file in about 10GB of data.
I'm not sure and haven't checked if the files are wrongly written or "only" wrongly read back as i delete the defective files and copy them again.
Today i copied a few files back and checked them against the stored MD5 sums and 5 files of 86 (each about 700 MB) had errors. So i copied the 5 files again. 4 of the files were OK after that and coping the last file the third time also resulted in the correct MD5.
This time i kept the defective files and used "vbindiff" to show me the difference. Strangly in EVERY case the difference is a single bit in a sequence of "0xff"-Bytes inside a block of varing bit-values that changed a "
0xff" into a "0xf7".
Also interesting is that each error is at a 0xXXXXXXX5-Position
Quoted 6 lines
Attached is a file with 5 of the 6 differences named 1-5. Of each of the 5 2x3 lines-blocks the first 3 lines are the original the following 3 lines contain the error in the middle row 6th value.
NEVER did i see any messages in syslog regarding erros or an aborting program due to errors passed down from the kernel or something like that.
This was almost certainly caused by hardware flaws in the USB interface chips of the enclosures. There's nothing the kernel can do about it because the errors aren't reported; all that happens is that incorrect data is sent to or from the drive.
Alan Stern
Quoted 37 lines · Alan Stern
Quoted 34 lines · Matthias Schniedermeyer
Hi
I'm using a Bunch auf HDDs in USB-Enclosures for storing files. (currently 38 HDD, with a total capacity of 9,5 TB of which 8,5 TB is used)
After i realised about a year(!) ago that the files copied to the HDDs sometimes aren't identical to the "original"-files i changed my procedured so that each file is MD5 before and after and deleted/copied again if an error is detected.
My averate file size is about 1GB with files from about 400MB to 5000MB I estimate the average error-rate at about one damaged file in about 10GB of data.
I'm not sure and haven't checked if the files are wrongly written or "only" wrongly read back as i delete the defective files and copy them again.
Today i copied a few files back and checked them against the stored MD5 sums and 5 files of 86 (each about 700 MB) had errors. So i copied the 5 files again. 4 of the files were OK after that and coping the last file the third time also resulted in the correct MD5.
This time i kept the defective files and used "vbindiff" to show me the difference. Strangly in EVERY case the difference is a single bit in a sequence of "0xff"-Bytes inside a block of varing bit-values that changed a "
0xff" into a "0xf7".
Also interesting is that each error is at a 0xXXXXXXX5-PositionAttached is a file with 5 of the 6 differences named 1-5. Of each of the 5 2x3 lines-blocks the first 3 lines are the original the following 3 lines contain the error in the middle row 6th value.
Quoted 8 lines
Quoted 2 lines
NEVER did i see any messages in syslog regarding erros or an aborting program due to errors passed down from the kernel or something like that.
This was almost certainly caused by hardware flaws in the USB interface chips of the enclosures. There's nothing the kernel can do about it because the errors aren't reported; all that happens is that incorrect data is sent to or from the drive.
So pretty much all ich can do is to pray that the errors don't corrupt the Filesystem-Metadata (XFS).
So i should definetly consider writing me a "NO-FS" where the "filesystem"-part is stored elsewhere and the HDD contains 100% content (Minus a Dummy-MBR-Block for sector 0). On the plus side such a filesystem won't have any overhead at all, but on the flipside you loose pretty much the whole content if you lose the metadata. But i guess in my case it would considerably lower the risk of loosing data.
Bis denn
Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as bad a concept in Text Editors as it is in women. No, the Real Programmer wants a "you asked for it, you got it" text editor -- complicated, cryptic, powerful, unforgiving, dangerous.
Hi Matthias :)
* Matthias Schniedermeyer <[email protected]> dixit:
Quoted 4 lines
My averate file size is about 1GB with files from about 400MB to 5000MB I estimate the average error-rate at about one damaged file in about 10GB of data.
Quoted 8 lines
I'm not sure and haven't checked if the files are wrongly written or "only" wrongly read back as i delete the defective files and copy them again.
Today i copied a few files back and checked them against the stored MD5 sums and 5 files of 86 (each about 700 MB) had errors. So i copied the 5 files again. 4 of the files were OK after that and coping the last file the third time also resulted in the correct MD5.
I had more or less the same issue a week or two ago. I performed
lots of tests and only by replacing the USB2.0 PCI card, the USB cable and the power supply of the usb-hdd adapter got the problem solved.
I'm not sure if the problem is really gone, but the system works now
reliably. I don't know if sooner or later I'll get the issue again, because I didn't really identify a culprit: looks like the card+adapter+cable combination was just "ugly", and errors from the adapter were not reported correctly.
Quoted 3 lines
NEVER did i see any messages in syslog regarding erros or an aborting program due to errors passed down from the kernel or something like that.
The same here! Looks like USB-HDD adapters don't report any errors
to the kernel :?????
The best advice I can give you, from my limited experience with the
problem, is: replace the cable. This minimizes the chance of corrupted data getting into the adapter. If that doesn't solve the problem, try removing any unconnected cable that is plugged into the USB card. Believe it or not, a long but unconnected cable (put there just to be able to plug my USB card-reader without having to look for the cable in a drawer) was causing errors *even in a Kingston USB key that worked flawlessly otherwise*!!!
If you have any other question, feel free to drop me a note. I'm
sorry I cannot give a much more technical or scientific answer, but unfortunately I have none :((
Raúl Núñez de Arenas Coronado
Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net It's my PC and I'll cry if I want to... RAmen!
Quoted 22 lines · DervishD
Hi Matthias :)
* Matthias Schniedermeyer <[email protected]> dixit:
Quoted 4 lines
My averate file size is about 1GB with files from about 400MB to 5000MB I estimate the average error-rate at about one damaged file in about 10GB of data.
Quoted 8 lines
I'm not sure and haven't checked if the files are wrongly written or "only" wrongly read back as i delete the defective files and copy them again.
Today i copied a few files back and checked them against the stored MD5 sums and 5 files of 86 (each about 700 MB) had errors. So i copied the 5 files again. 4 of the files were OK after that and coping the last file the third time also resulted in the correct MD5.
I had more or less the same issue a week or two ago. I performed
lots of tests and only by replacing the USB2.0 PCI card, the USB cable and the power supply of the usb-hdd adapter got the problem solved.
Quoted 5 lines
I'm not sure if the problem is really gone, but the system works now
reliably. I don't know if sooner or later I'll get the issue again, because I didn't really identify a culprit: looks like the card+adapter+cable combination was just "ugly", and errors from the adapter were not reported correctly.
The 38 HDDs are in 38 enclosures, so each has it's own power supply. I have used different cables and i replaced the USB-Controller once.
So it can't be a single faulty component. Except when the computer itself would be the culprit.
Quoted 7 lines
Quoted 3 lines
NEVER did i see any messages in syslog regarding erros or an aborting program due to errors passed down from the kernel or something like that.
The same here! Looks like USB-HDD adapters don't report any errors
to the kernel :?????
Quoted 8 lines
The best advice I can give you, from my limited experience with the
problem, is: replace the cable. This minimizes the chance of corrupted data getting into the adapter. If that doesn't solve the problem, try removing any unconnected cable that is plugged into the USB card. Believe it or not, a long but unconnected cable (put there just to be able to plug my USB card-reader without having to look for the cable in a drawer) was causing errors *even in a Kingston USB key that worked flawlessly otherwise*!!!
Hmmm. That's the only thing that i currently may be doing wrong. I have a 1,5 Meter and a 4,5 Meter cable connected to the USB-Controller and i only use of them depending on where the HDD is placed in my room, the other one is dangling unconnected.
Then i will unconnect the short cable and use the long cable exclusivly and see if it gets better(tm).
Quoted 3 lines
If you have any other question, feel free to drop me a note. I'm
sorry I cannot give a much more technical or scientific answer, but unfortunately I have none :((
Thank you anyway.
Bis denn
Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as bad a concept in Text Editors as it is in women. No, the Real Programmer wants a "you asked for it, you got it" text editor -- complicated, cryptic, powerful, unforgiving, dangerous.
Quoted 7 lines · Matthias Schniedermeyer
DervishD wrote:
The 38 HDDs are in 38 enclosures, so each has it's own power supply. I have used different cables and i replaced the USB-Controller once.
So it can't be a single faulty component. Except when the computer itself would be the culprit.
Or a production failure.
-`J'
Quoted 8 lines · Matthias Schniedermeyer
Quoted 7 lines
Quoted 7 lines
I'm using a Bunch auf HDDs in USB-Enclosures for storing files. (currently 38 HDD, with a total capacity of 9,5 TB of which 8,5 TB is used) [....]
This time i kept the defective files and used "vbindiff" to show me the difference. Strangly in EVERY case the difference is a single bit in a sequence of "0xff"-Bytes inside a block of varing bit-values that changed a "0xff" into a "0xf7".
Quoted 7 lines
Quoted 4 lines
This was almost certainly caused by hardware flaws in the USB interface chips of the enclosures. There's nothing the kernel can do about it because the errors aren't reported; all that happens is that incorrect data is sent to or from the drive.
So pretty much all ich can do is to pray that the errors don't corrupt the Filesystem-Metadata (XFS).
No, this is not all. You should buy a variety of different enclosures with different chipsets (e.g. find a Freecom if you can), and also use decent cables.
-- Pete
Quoted 17 lines · Pete Zaitcev
Quoted 14 lines · Matthias Schniedermeyer
Quoted 13 lines
Quoted 7 lines
I'm using a Bunch auf HDDs in USB-Enclosures for storing files. (currently 38 HDD, with a total capacity of 9,5 TB of which 8,5 TB is used) [....]
This time i kept the defective files and used "vbindiff" to show me the difference. Strangly in EVERY case the difference is a single bit in a sequence of "0xff"-Bytes inside a block of varing bit-values that changed a "0xff" into a "0xf7".This was almost certainly caused by hardware flaws in the USB interface chips of the enclosures. There's nothing the kernel can do about it because the errors aren't reported; all that happens is that incorrect data is sent to or from the drive.
Quoted 6 lines
Quoted 2 lines
So pretty much all ich can do is to pray that the errors don't corrupt the Filesystem-Metadata (XFS).
No, this is not all. You should buy a variety of different enclosures with different chipsets (e.g. find a Freecom if you can),
That would definetly cost way to much money and time to be in any way "efficient".
Quoted 1 line
and also use decent cables.
I replaced all cables with "High Quality"-cables.
But as a "Joe user" it is practically impossible to really know if the cables are good.
All i can say is that the "original" cables that came with the enclosures appear a bit thin and the ones i bought appear much more thick, have gold plated contacts and have a massive plaited shielding IOW appear much more trustworthy. But, as i said, in the end i can't really know if they are better than the original ones.
Bis denn
Signature · 4 lines
Real Programmers consider "what you see is what you get" to be just as bad a concept in Text Editors as it is in women. No, the Real Programmer wants a "you asked for it, you got it" text editor -- complicated, cryptic, powerful, unforgiving, dangerous.
Hi Matthias :)
* Matthias Schniedermeyer <[email protected]> dixit:
Quoted 13 lines
Quoted 8 lines
* Matthias Schniedermeyer <[email protected]> dixit:
Quoted 5 lines
Today i copied a few files back and checked them against the stored MD5 sums and 5 files of 86 (each about 700 MB) had errors. So i copied the 5 files again. 4 of the files were OK after that and coping the last file the third time also resulted in the correct MD5.
Quoted 5 lines
I had more or less the same issue a week or two ago. I performed lots of tests and only by replacing the USB2.0 PCI card, the USB cable and the power supply of the usb-hdd adapter got the problem solved.
Quoted 5 lines
The 38 HDDs are in 38 enclosures, so each has it's own power supply. I have used different cables and i replaced the USB-Controller once.
So it can't be a single faulty component. Except when the computer itself would be the culprit.
In my case, the same applied: the problem didn't seem to be a single
faulty component, but a combination. Since I finally wasn't able to check in another computer where I could carry tests, or check under windows in the same machine, I don't know if the problem was a faulty device driver, a faulty motherboard, RAM problems (although memtest said my RAM was OK), etc.
Quoted 9 lines
Quoted 4 lines
The best advice I can give you, from my limited experience with the
problem, is: replace the cable. This minimizes the chance of corrupted data getting into the adapter. If that doesn't solve the problem, try removing any unconnected cable that is plugged into the USB card.Hmmm. That's the only thing that i currently may be doing wrong. I have a 1,5 Meter and a 4,5 Meter cable connected to the USB-Controller and i only use of them depending on where the HDD is placed in my room, the other one is dangling unconnected.
I don't know why the heck this was causing problems, since any
electric noise that could be picked from the dangling cable shouldn't affect neither the card nor the other USB ports, but...
Raúl Núñez de Arenas Coronado
Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net It's my PC and I'll cry if I want to... RAmen!
Quoted 1 line · Matthias Schniedermeyer
Pete Zaitcev wrote:
...
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I'm using a Bunch auf HDDs in USB-Enclosures for storing files. (currently 38 HDD, with a total capacity of 9,5 TB of which 8,5 TB is used)
...
Quoted 5 lines
Quoted 2 lines
You should buy a variety of different enclosures with different chipsets (e.g. find a Freecom if you can),
That would definetly cost way to much money and time to be in any way "efficient".
Search for firmware updates from the manufacturer of the enclosure, of the bridge board, or of the bridge chip... if you didn't do so already. Some chips support firmware upload to an EEPROM, usually via a Windows utility.
Signature · 3 lines
Stefan Richter -=====-=-==- ==-- -=--- http://arcgraph.de/sr/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Stefan" == Stefan Richter <[email protected]> writes:
Stefan> Search for firmware updates from the manufacturer of the Stefan> enclosure, of the bridge board, or of the bridge chip... if Stefan> you didn't do so already. Some chips support firmware upload Stefan> to an EEPROM, usually via a Windows utility.
I did this with my USB2.0/Firewire external enclosure and it fixed most of the problems, but enough remained that I've basically given up on those enclosures, they're just not reliable in my book.
John