Good news / bad news - The joys of RAID

The bad news is I lost another disk tonight. Remind me *never* to buy Maxtor drives again.

The good news is that my RAID5 array was configured as 5 + 1 spare. I powered down the server, used the Maxtor PowerMax utility to identify the bad disk, pulled it out and re-booted. My array is currently re-syncing.

[root@dude root]# mdadm --detail /dev/md5
/dev/md5:

        Version : 00.90.01
  Creation Time : Thu Jul 29 21:41:38 2004
     Raid Level : raid5
     Array Size : 974566400 (929.42 GiB 997.96 GB)
    Device Size : 243641600 (232.35 GiB 249.49 GB)

Raid Devices : 5
Total Devices : 5
Preferred Minor : 5
Persistence : Superblock is persistent

Update Time : Fri Nov 19 20:52:58 2004
State : dirty, resyncing
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 128K

Rebuild Status : 0% complete

UUID : a4bbcd09:5e178c5b:3bf8bd45:8c31d2a1
Events : 0.1765551

Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 8 18 1 active sync /dev/sdb2
2 8 34 2 active sync /dev/sdc2
3 8 50 3 active sync /dev/sdd2
4 8 66 4 active sync /dev/sde2

Thinking about what happened, I would have expected that the bad drive would just be removed from the array and spare activated and re-syncing started automatically.

What actually happened was that I rebooted to activate a new kernel and the box didn't come back up. As the machine runs headless, I had to power it off and take it to a monitor/keyboard to check it. In the new location it came up fine so I shut it down again and put it back in my "server room" (read: cellar). I still couldn't see it from the network so I dragged an old 14" CRT out of the shed and connected it up. The login prompt was there but there was an "ata2 timeout" error message and the console was dead. I power-cycled to reboot and as it booted I saw a message something like "postponing resync of md0 as it uses the same device as md5. waiting for md5 to resync. I then got a further ata timeout error. I had to physically disconnect the bad drive and reboot in order to re-start the re-sync.

Further md information:

[root@dude log]# mdadm --detail --scan
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=11caa547:1ba8d185:1f1f771f:d66368c9
devices=/dev/sdc1
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=be8ad31a:f13b6f4b:c39732fc:c84f32a8
devices=/dev/sdb1,/dev/sde1
ARRAY /dev/md5 level=raid5 num-devices=5 UUID=a4bbcd09:5e178c5b:3bf8bd45:8c31d2a1
devices=/dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2,/dev/sde2
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=4b28338c:bf08d0bc:bb2899fc:e7f35eae
devices=/dev/sda1,/dev/sdd1

It was /dev/sdf that failed which contained two partitions, one of them part of md2 (now running un-mirrored but still showing two devices) and the other part of md5 (now re-syncing but only showing five devices).

Is this normal behaviour?

R.

http://robinbowes.com

Reminder....
Never buy Maxtor drives again!

Guy

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Robin Bowes
Sent: Friday, November 19, 2004 4:07 PM
To: [email protected]
Subject: Good news / bad news - The joys of RAID

The bad news is I lost another disk tonight. Remind me *never* to buy Maxtor drives again.

The good news is that my RAID5 array was configured as 5 + 1 spare. I powered down the server, used the Maxtor PowerMax utility to identify the bad disk, pulled it out and re-booted. My array is currently re-syncing.

[root@dude root]# mdadm --detail /dev/md5
/dev/md5:

         Version : 00.90.01

Creation Time : Thu Jul 29 21:41:38 2004
Raid Level : raid5
Array Size : 974566400 (929.42 GiB 997.96 GB)
Device Size : 243641600 (232.35 GiB 249.49 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 5
Persistence : Superblock is persistent

Update Time : Fri Nov 19 20:52:58 2004
State : dirty, resyncing
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 128K

Rebuild Status : 0% complete

UUID : a4bbcd09:5e178c5b:3bf8bd45:8c31d2a1
Events : 0.1765551

Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 8 18 1 active sync /dev/sdb2
2 8 34 2 active sync /dev/sdc2
3 8 50 3 active sync /dev/sdd2
4 8 66 4 active sync /dev/sde2

Thinking about what happened, I would have expected that the bad drive would just be removed from the array and spare activated and re-syncing started automatically.

What actually happened was that I rebooted to activate a new kernel and the box didn't come back up. As the machine runs headless, I had to power it off and take it to a monitor/keyboard to check it. In the new location it came up fine so I shut it down again and put it back in my "server room" (read: cellar). I still couldn't see it from the network so I dragged an old 14" CRT out of the shed and connected it up. The login prompt was there but there was an "ata2 timeout" error message and the console was dead. I power-cycled to reboot and as it booted I saw a message something like "postponing resync of md0 as it uses the same device as md5. waiting for md5 to resync. I then got a further ata timeout error. I had to physically disconnect the bad drive and reboot in order to re-start the re-sync.

Further md information:

[root@dude log]# mdadm --detail --scan
ARRAY /dev/md2 level=raid1 num-devices=2
UUID=11caa547:1ba8d185:1f1f771f:d66368c9
devices=/dev/sdc1
ARRAY /dev/md1 level=raid1 num-devices=2
UUID=be8ad31a:f13b6f4b:c39732fc:c84f32a8

    devices=/dev/sdb1,/dev/sde1
ARRAY /dev/md5 level=raid5 num-devices=5 
UUID=a4bbcd09:5e178c5b:3bf8bd45:8c31d2a1
    devices=/dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2,/dev/sde2
ARRAY /dev/md0 level=raid1 num-devices=2 
UUID=4b28338c:bf08d0bc:bb2899fc:e7f35eae
    devices=/dev/sda1,/dev/sdd1

It was /dev/sdf that failed which contained two partitions, one of them part of md2 (now running un-mirrored but still showing two devices) and the other part of md5 (now re-syncing but only showing five devices).

Is this normal behaviour?

R.

http://robinbowes.com
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

you imply that Maxtor drives are somehow inherently flawed. can you explain why you think millions of people/companies are naive idiots for continuing to buy Maxtor disks?

this sort of thing is just not plausible: Maxtor competes with the other top-tier disk vendors with similar products and prices and reliability. yes, if you buy a 1-year disk, you can expect it to have been less carefully tested, possibly be of lower-end design and reliability, and to have been handle more poorly by the supply chain. thankfully, you don't have to buy 1-year disks any more.

read the specs. make sure your supply chain knows how to handle disks. make sure your disks are mounted correctly, both mechanically and with enough airflow. use raid and some form of archiving/backups. don't get hung up on which of the 4-5 top-tier vendors makes your disk.

I have had far more failures of Maxtor drives than any other. I have also had problems with WD drives. I know someone that had 4-6 IBM disks, most of which have failed.

I am talking about disks with 3 year warranties! Based on the spec. But OEM disks have none. You must return them to the PC manufacture. Most of my failures were within 3 years, but beyond the warranty period of the system. So the OEM issue has occurred too often.

I have had good luck with Seagate.

I use RAID, it is a must with the failure rate!
I do backup also, but RAID tends to save me.

Most people have a PC with 1 disk. I don't understand RAID, and they don't understand that everything will be lost if the disk breaks! They think "Dell will just fix it". But wrong, Dell will just replace it! Big difference.

Today's disks claim a MTBF of about 1,000,000 hours! That's about 114 years. So, if I had 10 disks I should expect 1 failure every 11.4 years. That would be so cool! But not in the real world.

Can you explain how the disks have a MTBF of 1,000,000 hours? But fail more often than that? Maybe I just don't understand some aspect of MTBF.

Guy

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Mark Hahn
Sent: Saturday, November 20, 2004 1:43 PM
To: [email protected]
Subject: RE: Good news / bad news - The joys of RAID

you imply that Maxtor drives are somehow inherently flawed. can you explain why you think millions of people/companies are naive idiots for continuing to buy Maxtor disks?

this sort of thing is just not plausible: Maxtor competes with the other top-tier disk vendors with similar products and prices and reliability. yes, if you buy a 1-year disk, you can expect it to have been less carefully tested, possibly be of lower-end design and reliability, and to have been handle more poorly by the supply chain. thankfully, you don't have to buy 1-year disks any more.

read the specs. make sure your supply chain knows how to handle disks. make sure your disks are mounted correctly, both mechanically and with enough airflow. use raid and some form of archiving/backups. don't get hung up on which of the 4-5 top-tier vendors makes your disk.

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

MTBF is statistic based upon the expected 'use' of the drive and the replacement of the drive after its end of life (3-5 years)...

It's extremely complex and boring but the figure is only relative if the drive is being used within an environment that matches those of the calculations.

SATA / IDE drives have an MTBF similar to that of SCSI / Fibre. But this is based upon their expected use... i.e. SCSI used to be [power on hours = 24hr] [use = 8 hours].. whilst SATA used to be [power on = 8 hours] and [use = 20 mins].

Regardless of what some people clam (usually those that only sell sata based raids), the drives are not constructed the same in any way.

SATA's fail more within a raid environment (probably around 10:1) because of the heavy use and also because they are not as intelligent... therefore when they do not respond we have no way of interrogating them or resetting them, whilst with scsi we do both. This means that a raid controller / driver has no option to but simply fail the drive.

Maxtor lead the way in capacity and also reliability... I personal had to recall countless earlier IBMs and replace them with maxtor. But the new generation of IBM's (Hitachi) have got it together.

So - I guess you are all right :)

the vendors I talk to always quote SCSI/FC at 100% power 100% duty, and PATA/SATA at 100% power 20% duty.

obviously, there *have* been pairs of SCSI/ATA disks which had identical mech/analog sections. but the mech/analog fall into just two kinds:

- optimized for IOPS: 10-15K rpm for minimal rotational latency, narrow recording area for low seek distance, quite low bit and track density to avoid long waits for the head to stabilize after a seek.

- optimized for density/bandwidth: high bit/track density, wide recording area, modest seeks/rotation speed.

the first is SCSI/FC and the second ATA, mainly for historic reasons.

what connection are you drawing between raid and "heavy use"? how does being in a raid increase the IO load per disk?

you've never seen a SCSI reset that looks just like an ATA reset? sorry, but SCSI has no magic.

no.

afaikt, the deathstar incident was actually bad firmware (didn't correctly flush data when hard powered off, resulting in blocks on disk with bogus ECC, which had to be considered bad from then on, even if the media was perfect.)

You got any links related to this?
"the deathstar incident was actually bad firmware"

Can a user download and update the firmware?

If so, I know someone that may have some bad disks that are not so bad.

If he can repair his disks, I will report the status back on this list.

Previously I thought IBM made very good disks, until my friend had more than a 75% failure rate. And within the warranty period.

I personally have an IBM SCSI disk that is running 100% of the time, and the cooling is real bad. The drive is much too hot to touch. Been like that for 5+ years. Never had any issues. The system also has a Seagate that is too hot to touch, but only been running 3+ years. Both are 18 Gig. The disks are in a system my wife uses! Don't tell her. :) I got to fix that someday.

Guy

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Mark Hahn
Sent: Saturday, November 20, 2004 5:18 PM
To: [email protected]
Subject: RE: Good news / bad news - The joys of RAID

the vendors I talk to always quote SCSI/FC at 100% power 100% duty, and PATA/SATA at 100% power 20% duty.

obviously, there *have* been pairs of SCSI/ATA disks which had identical mech/analog sections. but the mech/analog fall into just two kinds:

- optimized for IOPS: 10-15K rpm for minimal rotational latency, narrow recording area for low seek distance, quite low bit and track density to avoid long waits for the head to stabilize after a seek.

- optimized for density/bandwidth: high bit/track density, wide recording area, modest seeks/rotation speed.

the first is SCSI/FC and the second ATA, mainly for historic reasons.

what connection are you drawing between raid and "heavy use"? how does being in a raid increase the IO load per disk?

you've never seen a SCSI reset that looks just like an ATA reset? sorry, but SCSI has no magic.

no.

afaikt, the deathstar incident was actually bad firmware (didn't correctly flush data when hard powered off, resulting in blocks on disk with bogus ECC, which had to be considered bad from then on, even if the media was perfect.)

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I do not think the deathstar incident was due to a firmware problem as you describe at all. I had a lot of these drives fail, and I read as much as I could find on the subject. The problem was most likely caused by the fact that these drives used IBM's new glass substrate technology. This substrate had heat expansion issues which caused the heads to misalign on tracks and eventually cross write over tracks, corrupting data. The classic "click of death" was the sound of the drive searching for a track repetitively. In some cases a format would allow the drive to be used again, in many cases it would not. It is my belief that formatting was inneffective at fixing the drive because the cross writing probably hit some of the low level data, which the drive cannot repair on a format.

Perhaps servo/timing data? Also I recall some Kennedy Winchester
drives back in the early 80s that if you had a power outage would get header CRC errors at pairs of blocks that were arranged in a spiral as the head
headed for the landing zone. I recall writing a standalone program
that would read the entire drive and then 'correct' the CRC errors as it
found them. Since much of the drive was unused I finally figurered out
that the data was fine it was the header CRC that got clobbered, apparently there was a bug in the powerdown hardware so it would enable the write head when it was in the the interblock zone as it was flying to land....

Ahh for the days of poking into device registers (in Memory) to get I/O to
happen (from the console).
<scw>

-----
Stephen C. Woods; UCLA SEASnet; 2567 Boelter hall; LA CA 90095; (310)-825-8614
Unless otherwise noted these statements are my own, Not those of the 
University of California.                      Internet mail:[email protected]

the ecc *is* the low-level data. without performing a controlled experiment that recreates the power-off scenario, there's no way to distinguish a block whose media is actually bad from one whose ecc fails because the ecc is bad.

the firmware theory is supported by the fact that many deathstars performed perfectly well for many years. I have at least one that lasted for 4+ years, and was powered off only a few times, and all of those cleanly.

The ECC is not the low level data. The servo tracks are. I bet there are start of track/sector header marks also. I believe a low level format will not re-write the servo tracks. Some drives reserve 1 side of 1 platter for servo data. Others mix the servo data with user data. I don't know the full details, just tidbit I have read over the years.

If your drives were cooled better than most, that may explain why you did not have the "substrate had heat expansion issues". Just a guess.

If the problem was a firmware issue, why didn't IBM release a firmware update?

You said:
"the firmware theory is supported by the fact that many deathstars performed perfectly well for many years"

Are you saying some drives had good firmware, while others had bad firmware? Otherwise, I don't understand your logic, since a drive not failing does not prove a firmware bug.

Guy

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Mark Hahn
Sent: Thursday, December 02, 2004 10:37 PM
To: TJ
Cc: [email protected]
Subject: Re: Good news / bad news - The joys of RAID

the

the ecc *is* the low-level data. without performing a controlled experiment that recreates the power-off scenario, there's no way to distinguish a block whose media is actually bad from one whose ecc fails because the ecc is bad.

the firmware theory is supported by the fact that many deathstars performed perfectly well for many years. I have at least one that lasted for 4+ years, and was powered off only a few times, and all of those cleanly.

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

ecc is in the disk controller with the phaselock loop and other analog circuit to convert the analog signal from the head back into 1's and 0's for the ecc code in firmware to correct any obvious head read errors

track/sector info is written to the disk with low level format
( usually at the manufacturer
- you can also do lowlevel format with superformat

- it contains sector and track info and other header info along with gaps and timing/spacing between each field

- disks are now soft sectored .. ( no servo info ) ( 512bytes or 1K or 2K or 4K(?) bytes per sector )

- there is just one "index" mark to indicate one full platter rotation

- you can change any/all of the data ... as long as the apps can read the data its lower-level drivers did to the disk
- firmware level is the lowest changes ( on the disk controller )

- some brave soles put "raid" in firmware .. ( risky in my book )

we use mke2fs, mkreiserfs etc to write file system data to make the platter useful

we use software and other utilities to do more ecc checking on the data we expect to get back

- if the system memory is bad ..
we overwrite good disk data with bad data from bad memory

- if the disk read/write is bad ...
we can sometimes compensate for it by keeping the disk cooler ( <= 30C for disk temp is good )

if ecc on the disk controller cannot fix it .. the disk is basically worthless

c ya
alvin

I believe they did. I recall downloading something similar to this:

http://support.dell.com/support/downloads/format.aspx?releaseid=r37239&c=us&l=en&s=biz&cs=555

at the time, to fix one of my drives.

Regards,

Richard

The updated IBM firmware helped. Before, every power outage would produce disks with unreadable sectors. Now, all our IBM disks have the "new" firmware and they hardly ever develop unreadable sectors.

This makes me suspect that there are *two* unrelated problems:
1) the "scribble at power down" problem, fixed by the firmware update;
2) the "overheated disks lose data due to platter thermal expansion" problem, probably unfixable, other than by keeping the disks cool.

Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Followup to:  <[email protected]>
By author:    TJ <[email protected]>
In newsgroup: linux.dev.raid

It's also worth noting that there was extremely high correlation between which factory built the drives and the failure rates. Apparently some factories had virtually zero instances of this problem.

-hpa

simple: the MTBF applies to very large sets of disks. if you had millions of disks, you'd expect to average mtbf/ndisks between failures. with statistically trivial sample sizes (10 disks), you can't really say much. of course, a proper model of the failure rate would have a lot more than 1 parameter...

for instance, my organization will be buying about .5 PB of storage soon. here are some options:

disk n mtbf hours $/disk $K total

250GB SATA 1920 1e6 500 399 766
600GB SATA 800 1e6 1250 600? 480

73GB SCSI/FC 6575 1.3e6 198 389 2558
146GB SCSI/FC 3288 1.3e6 395 600 1973
300GB SCSI/FC 1600 1.3e6 813 1200 1920

these mtbf's are basically made up, since disk vendors aren't really very helpful in publishing their true reliability distributions. these disk counts are starting to be big enough to give some meaning to the hours=mtbf/n calculation - I'd WAG that "hours" is within a factor of two. (I looked at only three lines of SCSI disks to get 1.3e6 - two quoted 1.2 and the newer was 1.4.) vendors seem to be switching to quoting "annualized failure rates", which are probably easier to understand - 1.2e6 MTBF or 0.73% AFR, for instance. the latter makes it more clear that we're talking about gambling ;)

but the message is clear: for a fixed, large capacity, your main concern should be bigger disks. since our money is also fixed, you can see that SCSI/FC prices are a big problem (these are real list prices from a tier-1 vendor who marks up their SATA by an embarassing amount...) further, there's absolutely no chance we could ever keep .5 PB of disks busy at 100% duty cycle, so that's not a reason to buy SCSI/FC either...

regards, mark hahn.

Yeah, you're right.
Of course - the fact that 2 of *my* 6 Maxtor 250Gb SATA drives (3 year warranty) date stamped at various times in 2004 have failed is coincidence and should, of course, be expected with a MTBF of millions of hours.

Oh, please note I'm not Robin - that must be a coincidence too :)

Personally I'm waiting for the revelation that they are recycled IBM Deskstar 70's ;)

I take your point about supply chain though - anything that's shipped by courier is suspect.

David

You said:
"anything that's shipped by courier is suspect."

Humm, the way the drives are packed you would have a hard time exceeding 300Gs. Even UPS can't do that I bet. But I must admit, I have no idea what force a drive would "feel" in a 4 foot drop. Remember, the drive is packed very well! Also, they only refer to 2 ms. So, no idea if that is equal to 150 Gs for 4ms. Or 75 Gs for 8ms.

From a 300G Maxtor drive.

Reliability
- Shock Tolerance: 60Gs @ 2 ms half-sine pulse (Operating), 300Gs @ 2 ms half-sine pulse (Non-operating)
- Data Error Rate: < 1 /10E15 bits read (Non-recoverable)
- MTBF: 1000000 Hours

Guy

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of David Greaves
Sent: Saturday, November 20, 2004 2:41 PM
To: Mark Hahn
Cc: [email protected]
Subject: Re: Good news / bad news - The joys of RAID

Yeah, you're right.
Of course - the fact that 2 of *my* 6 Maxtor 250Gb SATA drives (3 year warranty) date stamped at various times in 2004 have failed is coincidence and should, of course, be expected with a MTBF of millions of hours.

Oh, please note I'm not Robin - that must be a coincidence too :)

Personally I'm waiting for the revelation that they are recycled IBM Deskstar 70's ;)

I take your point about supply chain though - anything that's shipped by courier is suspect.

David

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

ALL of the Maxtor junk that I have sitting next to me were in factory packaging, and not likely to have been affected by either physical or electrical shock.

HE might have implied, I am saying it! Why ask someone as you did in sentence #2? Ask them - or yourself.

Of course, he probably missed the warranty statement to not run Linux.

Followup to:  <Pine.LNX.4.44.0411201238320.19120-100000@coffee.psychology.mcmaster.ca>
By author:    Mark Hahn <[email protected]>
In newsgroup: linux.dev.raid

In my experience, that is bullshit. Maxtor competes on price using inferior products. I bought two Maxtor drives, both of them failed within 13 months. That was my first attempt at trying Maxtor again after taking them off my sh*tlist from last time.

-hpa

When will you learn? :)

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of H. Peter Anvin
Sent: Tuesday, November 23, 2004 2:11 PM
To: [email protected]
Subject: Re: Good news / bad news - The joys of RAID

Followup to:
<Pine.LNX.4.44.0411201238320.19120-100000@coffee.psychology.mcmaster.ca>

By author:    Mark Hahn <[email protected]>
In newsgroup: linux.dev.raid

In my experience, that is bullshit. Maxtor competes on price using inferior products. I bought two Maxtor drives, both of them failed within 13 months. That was my first attempt at trying Maxtor again after taking them off my sh*tlist from last time.

-hpa
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

exactly - you can conclude absolutely nothing from two samples.

I read that mail as "I stopped buying Maxtor (for whatever reason) then tried them again and had an 100% failure rate (albeit with a small sample size) so have stopped buying them again" rather than "I bought two Maxtor drives that failed so Maxtor drives are shit".

My own personal experience (I'm the OP in this thread) is that the 250GB SATA Maxtor Maxline II drives I have purchased have an unacceptable failure rate (something like 40% in 5 months)

R.

http://robinbowes.com

I understood! I was poking fun that you tried them again, and again lost! I hope you understood me. "When will you learn? :)"

Also, I thought of this about 4 years ago. Describes many managers! "Sure you saved money, but at what cost?" - Guy Watkins

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Robin Bowes
Sent: Tuesday, November 23, 2004 6:03 PM
To: [email protected]
Subject: Re: Good news / bad news - The joys of RAID

I read that mail as "I stopped buying Maxtor (for whatever reason) then tried them again and had an 100% failure rate (albeit with a small sample size) so have stopped buying them again" rather than "I bought two Maxtor drives that failed so Maxtor drives are shit".

My own personal experience (I'm the OP in this thread) is that the 250GB SATA Maxtor Maxline II drives I have purchased have an unacceptable failure rate (something like 40% in 5 months)

R.

http://robinbowes.com
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

I think I have 4 1/2 out of 6. Better?

Actually, you can. Having two fail in short order should be an extremely rare event.

-hpa

About 2 years ago I had a disk fail, not 100%, but intermittent problems. So I replaced it. The replacement started acting up about 6-12 months ago. Read errors about every 1-2 months, finally it went off-line. But, intermittently. I did think it was odd that the drive in the same position was failing, and with similar problems, but figured it was just a quincidence. Today I replaced it, after replacing it, I had some problems. It is in a case with 6 other disks, so I could tell by the LEDs that the replacement drive was acting wrong, intermittently. I determined that the Molex power plug going to the drive was causing the problems. What a pain! So, the 2 drives that I replaced may have been good. The first drive I took apart. I have the magnets to prove it! But it may have been a good drive!

To make a long story short, check the cables for failures, including the power cables.

The drives are Seagate, and I have at least 26 in service, so 2 failures out of 26 in 3 years is not so bad. However, if the Molex connector was at fault, then 0 failures out of 26 in 3 years, is just fine.

The drive is model ST118282LC, MTBF 1,000,000. I think with 26 drives I should have 1 failure in about 4.4 years. The drives have a 5 year warranty, but they are OEM, so I get nothing. I am not the first owner, but they were unused. And I bet they are about 5 years old now.

Too much info? Sorry. Maybe I need a blog? :)

Can anyone spell "quincidence"?

Guy

http://dictionary.reference.com/search?q=coincidence

R.

http://robinbowes.com

The re-sync to the spare should have been automatic, without a re-boot. Your errors related to ata timeout is not a Linux issue. My guess is the bios could see the drive, but the drive was not responding correctly. I think this is life with ata. I have had similar problems with SCSI. 1 drive failed in a way that it caused problems with other drives on the same SCSI bus.

It could be that your array was re-building, but did not finish. In that case it would start over from the beginning. Which may look like it did not attempt to re-build until the re-boot. Did you check the status before you shut it down? I use mdadm's monitor mode to send me email when events occur. By the time I read my emails, a drive has failed and the re-sync to the spare is done. No need to check logs.

Yes, it is normal that md will not re-sync 2 arrays that share a common device. One will be delayed until the other finishes.

Second reminder....
Never buy Maxtor drives again!

This quote seems to fit real well!
"Sure you saved money, but at what cost?" - Guy Watkins

Guy

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Robin Bowes
Sent: Friday, November 19, 2004 4:07 PM
To: [email protected]
Subject: Good news / bad news - The joys of RAID

The bad news is I lost another disk tonight. Remind me *never* to buy Maxtor drives again.

The good news is that my RAID5 array was configured as 5 + 1 spare. I powered down the server, used the Maxtor PowerMax utility to identify the bad disk, pulled it out and re-booted. My array is currently re-syncing.

[root@dude root]# mdadm --detail /dev/md5
/dev/md5:

         Version : 00.90.01

Creation Time : Thu Jul 29 21:41:38 2004
Raid Level : raid5
Array Size : 974566400 (929.42 GiB 997.96 GB)
Device Size : 243641600 (232.35 GiB 249.49 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 5
Persistence : Superblock is persistent

Update Time : Fri Nov 19 20:52:58 2004
State : dirty, resyncing
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 128K

Rebuild Status : 0% complete

UUID : a4bbcd09:5e178c5b:3bf8bd45:8c31d2a1
Events : 0.1765551

Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 8 18 1 active sync /dev/sdb2
2 8 34 2 active sync /dev/sdc2
3 8 50 3 active sync /dev/sdd2
4 8 66 4 active sync /dev/sde2

Thinking about what happened, I would have expected that the bad drive would just be removed from the array and spare activated and re-syncing started automatically.

What actually happened was that I rebooted to activate a new kernel and the box didn't come back up. As the machine runs headless, I had to power it off and take it to a monitor/keyboard to check it. In the new location it came up fine so I shut it down again and put it back in my "server room" (read: cellar). I still couldn't see it from the network so I dragged an old 14" CRT out of the shed and connected it up. The login prompt was there but there was an "ata2 timeout" error message and the console was dead. I power-cycled to reboot and as it booted I saw a message something like "postponing resync of md0 as it uses the same device as md5. waiting for md5 to resync. I then got a further ata timeout error. I had to physically disconnect the bad drive and reboot in order to re-start the re-sync.

Further md information:

[root@dude log]# mdadm --detail --scan
ARRAY /dev/md2 level=raid1 num-devices=2
UUID=11caa547:1ba8d185:1f1f771f:d66368c9
devices=/dev/sdc1
ARRAY /dev/md1 level=raid1 num-devices=2
UUID=be8ad31a:f13b6f4b:c39732fc:c84f32a8

    devices=/dev/sdb1,/dev/sde1
ARRAY /dev/md5 level=raid5 num-devices=5 
UUID=a4bbcd09:5e178c5b:3bf8bd45:8c31d2a1
    devices=/dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2,/dev/sde2
ARRAY /dev/md0 level=raid1 num-devices=2 
UUID=4b28338c:bf08d0bc:bb2899fc:e7f35eae
    devices=/dev/sda1,/dev/sdd1

It was /dev/sdf that failed which contained two partitions, one of them part of md2 (now running un-mirrored but still showing two devices) and the other part of md5 (now re-syncing but only showing five devices).

Is this normal behaviour?

R.

http://robinbowes.com
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Guy,

I've been meaning to write this for a while...

I tried monitoring once but had a problem when shutting down as the arrays were reported as "busy" because mdadm --monitor was running on them. I guess it needs to be killed earlier in the shutdown process.

So, can you share with me how you start/stop mdadm to run in monitor mode?

Thanks,

R.

http://robinbowes.com

That bug was fixed in mdadm 1.6.0

NeilBrown

From the ChangeLog:
Changes Prior to 1.6.0 release
...
- Fix bug in --monitor where an array could be held open and so
could not be stopped without killing mdadm.
...

If I recall correctly, this fixes the primary symptom, but not the whole problem. When in --monitor mode, mdadm will reopen each device every 15 seconds to scan its status. As such, a shutdown could still fail if mdadm is still running and the timing is right. In that instance, retrying the shutdown on failure would likely be enough to solve the problem, but that sounds icky to me. Would be much better if mdadm could open a control device of some sort and query about running arrays instead of opening the arrays themselves.

  Doug Ledford <[email protected]>
         Red Hat, Inc.
         1801 Varsity Dr.
         Raleigh, NC 27606

Wouldn't simply killing the "mdadm --monitor" process early on in the shutdown process achieve the same result?

R.

http://robinbowes.com

Not directly related to your RAID issue, but I've been running headless servers with console on serial ports as of late. LILO has an option to put output on a serial line, and there's a kernel compile flag and an append instruction to make it all work.

That combined with a power cycler makes me feel more at ease about the remote servers I run.

Just don't connect 2 PCs back to back and run a getty on each serial line...

Gordon