RAID down, dont know why!
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
What kind of logs do you need, I really need help!
This is an automatically generated mail message from mdadm running on ALEXANDRIA
A Fail event had been detected on md device /dev/md0.
It could be related to component device /dev/sdl1.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid6 sdl1[9](F) sdm1[8] sdi1[10](F) sdj1[11](F) sdk1[12](F) sdh1[3] sdf1[1] sdg1[2] sde1[0]
6837318656 blocks level 6, 1024k chunk, algorithm 2 [9/5] [UUUU____U]
md1 : active raid0 sdc1[1] sdb1[0]
586067072 blocks 64k chunks
unused devices: <none>
Signature · 2 lines
Andrew Dunn http://agdunn.net
Quoted 2 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
It looks like you have 4 failed drives. sdl,sdi,sdj,sdk
Is it possible you lost power or connectivity to those drives?
If you have lsscsi installed, what does lsscsi tell you about this?
lsscsi | grep sd[ijkl]
Given the proximity of the drives in ordering, I'd suspect a power loss, or cable seating, or similar to those drives.
Reseat power/signal cables on the drive bays, and see if this helps.
Joe
Signature · 9 lines
Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: [email protected] web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615
storrgie@ALEXANDRIA:~$ lsscsi | grep sd[ijkl] [11:0:0:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdi [11:0:1:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdj [11:0:2:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdk [11:0:3:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdl
Quoted 20 lines · Joe Landman
Quoted 2 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
It looks like you have 4 failed drives. sdl,sdi,sdj,sdk
Is it possible you lost power or connectivity to those drives?
If you have lsscsi installed, what does lsscsi tell you about this?
lsscsi | grep sd[ijkl]
Given the proximity of the drives in ordering, I'd suspect a power loss, or cable seating, or similar to those drives.
Reseat power/signal cables on the drive bays, and see if this helps.
Joe
Signature · 2 lines
Andrew Dunn http://agdunn.net
Quoted 6 lines · Andrew Dunn
storrgie@ALEXANDRIA:~$ lsscsi | grep sd[ijkl] [11:0:0:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdi [11:0:1:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdj [11:0:2:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdk [11:0:3:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdl
Does smartctl report drive failure?
smartctl -a /dev/sdi | grep "SMART overall-health" smartctl -a /dev/sdj | grep "SMART overall-health" smartctl -a /dev/sdk | grep "SMART overall-health" smartctl -a /dev/sdl | grep "SMART overall-health"
Quoted 22 lines
Quoted 19 lines · Joe Landman
Quoted 2 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
It looks like you have 4 failed drives. sdl,sdi,sdj,sdk
Is it possible you lost power or connectivity to those drives?
If you have lsscsi installed, what does lsscsi tell you about this?
lsscsi | grep sd[ijkl]
Given the proximity of the drives in ordering, I'd suspect a power loss, or cable seating, or similar to those drives.
Reseat power/signal cables on the drive bays, and see if this helps.
Joe
Signature · 9 lines
Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: [email protected] web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615
storrgie@ALEXANDRIA:~$ sudo mdadm -D /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Fri Nov 6 07:06:34 2009
Raid Level : raid6
Array Size : 6837318656 (6520.58 GiB 7001.41 GB)
Used Dev Size : 976759808 (931.51 GiB 1000.20 GB)
Raid Devices : 9
Total Devices : 9
Preferred Minor : 0
Persistence : Superblock is persistent Update Time : Sun Nov 8 09:17:55 2009
State : clean, degraded, recovering
Active Devices : 8
Working Devices : 9
Failed Devices : 0
Spare Devices : 1
Chunk Size : 1024K
Rebuild Status : 0% complete
UUID : 397e0b3f:34cbe4cc:613e2239:070da8c8 (local to host
ALEXANDRIA)
Events : 0.56
Number Major Minor RaidDevice State
0 8 65 0 active sync /dev/sde1
1 8 81 1 active sync /dev/sdf1
2 8 97 2 active sync /dev/sdg1
3 8 113 3 active sync /dev/sdh1
4 8 129 4 active sync /dev/sdi1
5 8 145 5 active sync /dev/sdj1
9 8 161 6 spare rebuilding /dev/sdk1
7 8 177 7 active sync /dev/sdl1
8 8 193 8 active sync /dev/sdm1
Did a:
sudo mdadm --assemble --force /dev/md0 /dev/sd[efghijklm]1
Now its rebuilding? Why did it go down in the first place?
Power and connections are fine and smart reports:
storrgie@ALEXANDRIA:~$ sudo smartctl -a /dev/sde | grep "SMART overall-health" SMART overall-health self-assessment test result: PASSED storrgie@ALEXANDRIA:~$ sudo smartctl -a /dev/sdf | grep "SMART overall-health" SMART overall-health self-assessment test result: PASSED storrgie@ALEXANDRIA:~$ sudo smartctl -a /dev/sdg | grep "SMART overall-health" SMART overall-health self-assessment test result: PASSED storrgie@ALEXANDRIA:~$ sudo smartctl -a /dev/sdh | grep "SMART overall-health" SMART overall-health self-assessment test result: PASSED storrgie@ALEXANDRIA:~$ sudo smartctl -a /dev/sdi | grep "SMART overall-health" SMART overall-health self-assessment test result: PASSED storrgie@ALEXANDRIA:~$ sudo smartctl -a /dev/sdj | grep "SMART overall-health" SMART overall-health self-assessment test result: PASSED storrgie@ALEXANDRIA:~$ sudo smartctl -a /dev/sdk | grep "SMART overall-health" SMART overall-health self-assessment test result: PASSED storrgie@ALEXANDRIA:~$ sudo smartctl -a /dev/sdl | grep "SMART overall-health" SMART overall-health self-assessment test result: PASSED storrgie@ALEXANDRIA:~$ sudo smartctl -a /dev/sdm | grep "SMART overall-health" SMART overall-health self-assessment test result: PASSED
Quoted 39 lines · Joe Landman
Quoted 6 lines · Andrew Dunn
storrgie@ALEXANDRIA:~$ lsscsi | grep sd[ijkl] [11:0:0:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdi [11:0:1:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdj [11:0:2:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdk [11:0:3:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdlDoes smartctl report drive failure?
smartctl -a /dev/sdi | grep "SMART overall-health" smartctl -a /dev/sdj | grep "SMART overall-health" smartctl -a /dev/sdk | grep "SMART overall-health" smartctl -a /dev/sdl | grep "SMART overall-health"Quoted 22 lines
Quoted 19 lines · Joe Landman
Quoted 2 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
It looks like you have 4 failed drives. sdl,sdi,sdj,sdk
Is it possible you lost power or connectivity to those drives?
If you have lsscsi installed, what does lsscsi tell you about this?
lsscsi | grep sd[ijkl]
Given the proximity of the drives in ordering, I'd suspect a power loss, or cable seating, or similar to those drives.
Reseat power/signal cables on the drive bays, and see if this helps.
Joe
Signature · 2 lines
Andrew Dunn http://agdunn.net
A message that is not in the archive (sent off-list or not yet received): [email protected]
/dev/sdm did not drop out, and it is on the same cable as sd[kl]
Quoted 22 lines · Maurice Hilarius
Quoted 6 lines · Andrew Dunn
storrgie@ALEXANDRIA:~$ lsscsi | grep sd[ijkl] [11:0:0:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdi [11:0:1:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdj [11:0:2:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdk [11:0:3:0] disk ATA WDC WD1001FALS-0 0K05 /dev/sdlLooks like cable got "jostled" and the 4 drives "dropped out" of the RAID,
then it re-connected and now they show up as devs again.Signature · 7 lines
With our best regards, /Maurice W. Hilarius Telephone: 01-780-456-9771 Hard Data Ltd. FAX: 01-780-456-9772 11060 - 166 Avenue email:[email protected] <mailto:email:[email protected]> Edmonton, AB, Canada T5X 1Y3/
Signature · 2 lines
Andrew Dunn http://agdunn.net
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Its something different than hardware, I am getting random failures now, as in:
Normal (X marks no drive even present, U marks a drop):
A B X X
C D X X
E F X X
G H I X
First loss:
A B X X
C D X X
U U X X
U U I X
Second Loss:
First loss:
U U X X
C D X X
E U X X
G H U X
Just dropping randomly.
I had the system running under ubuntu 9.04 server for 1 month without ever seeing this, it wont run an hour without this issue now. Currently trying to find my 9.04 disk so I can just go back, I need this thing online by wednesday.
I wish I could have easily gotten more information for everyone, this is a huge problem for people with my setup. I think its possibly a controller driver thing.
Quoted 22 lines · Joe Landman
Quoted 8 lines · Maurice Hilarius
Quoted 7 lines · Joe Landman
Quoted 2 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
It looks like you have 4 failed drives. sdl,sdi,sdj,sdk
Quoted 6 lines
Exactly.
Looks like one multilane cable is disconnected.
Each "feeds" 4 drives.
a b c d
e f g h
i j k lYes. Tha is what I was thinking. I asked Andrew to reseat cables, and check to make sure that power is going to the block on the backplane or mobile storage cannister.
Signature · 2 lines
Andrew Dunn http://agdunn.net
Quoted 5 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
What kind of logs do you need, I really need help!
From the message you've posted, it looks like something has triggered the (simultaneous) removal of four drives from the array. I'd check the dmesg output - it should provide some information. I'd guess these four drives are all attached to the same controller (are they external or internal?), so possibly the controller reset (or for external drives, it could be a cable issue).
You should be able to force an assembly anyway (using the --force flag) but I'd make sure you know exactly what the issue is first, otherwise this is likely to happen again.
Cheers,
Robin
Signature · 4 lines
___
( ' } | Robin Hill <[email protected]> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |What would I be looking for on this? Its a lot to sift through.
Currently just line-by-lining it.
Quoted 23 lines · Robin Hill
Quoted 6 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
What kind of logs do you need, I really need help!
From the message you've posted, it looks like something has triggered the (simultaneous) removal of four drives from the array. I'd check the dmesg output - it should provide some information. I'd guess these four drives are all attached to the same controller (are they external or internal?), so possibly the controller reset (or for external drives, it could be a cable issue).
You should be able to force an assembly anyway (using the --force flag) but I'd make sure you know exactly what the issue is first, otherwise this is likely to happen again.
Cheers,
Robin
Signature · 2 lines
Andrew Dunn http://agdunn.net
Quoted 17 lines · Andrew Dunn
Quoted 9 lines · Robin Hill
Quoted 6 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
What kind of logs do you need, I really need help!
Quoted 7 lines
From the message you've posted, it looks like something has triggered the (simultaneous) removal of four drives from the array. I'd check the dmesg output - it should provide some information. I'd guess these four drives are all attached to the same controller (are they external or internal?), so possibly the controller reset (or for external drives, it could be a cable issue).
Quoted 4 lines
What would I be looking for on this? Its a lot to sift through.
Currently just line-by-lining it.
Look for where the drives are being kicked out of the array (should be towards the bottom). Just above that should be some error messages (often including bus resets).
Cheers,
Robin
Signature · 4 lines
___
( ' } | Robin Hill <[email protected]> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |Is this the box on your blog at http://blog.agdunn.net/?p=391 ? If so those cards are to be used in the Supermicro UIO slot, which is basically just an inverted PCI Express slot. However since there is only one UIO slot per board they might have not tested compatibility with multiple in the same system.
I do have one of these boards installed on an Intel board without issue. I have had the 7 drives connected in mdadm RAID for almost 2 years now with no dropouts. You might try a port multiplier since the card supports it and one drive isn't going to use the full bandwidth of a single SAS cable.
Ryan
Quoted 26 lines · Robin Hill
Quoted 17 lines · Andrew Dunn
Quoted 9 lines · Robin Hill
Quoted 6 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
What kind of logs do you need, I really need help!
Quoted 7 lines
From the message you've posted, it looks like something has triggered the (simultaneous) removal of four drives from the array. I'd check the dmesg output - it should provide some information. I'd guess these four drives are all attached to the same controller (are they external or internal?), so possibly the controller reset (or for external drives, it could be a cable issue).
Quoted 4 lines
What would I be looking for on this? Its a lot to sift through.
Currently just line-by-lining it.
Look for where the drives are being kicked out of the array (should be towards the bottom). Just above that should be some error messages (often including bus resets).
Quoted 8 lines
Cheers,
Robin
--
___( ' } | Robin Hill <[email protected]> | / / ) | Little Jim says .... | // !! | "He fallen in de water !!" |
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
I had this setup running under ubuntu 9.04 with raid6, all nine drives for about a month without any of these issues.
I installed the new ubuntu fresh and tried to rebuild the array with a larger chunk size.
I also created the file system with the proper stride and width parameters.
I am wondering if TLER, or a larger chunk, ext options, or OS change might have caused this. I do not think that it is a backpane issue... I will however re-seat all of the drives here in a little bit.
Quoted 62 lines · Ryan Wagoner
Is this the box on your blog at http://blog.agdunn.net/?p=391 ? If so those cards are to be used in the Supermicro UIO slot, which is basically just an inverted PCI Express slot. However since there is only one UIO slot per board they might have not tested compatibility with multiple in the same system.
I do have one of these boards installed on an Intel board without issue. I have had the 7 drives connected in mdadm RAID for almost 2 years now with no dropouts. You might try a port multiplier since the card supports it and one drive isn't going to use the full bandwidth of a single SAS cable.
Ryan
Quoted 32 lines · Robin Hill
Quoted 21 lines · Andrew Dunn
Quoted 11 lines · Robin Hill
Quoted 7 lines · Andrew Dunn
I just copied 4+ TiB of information to this array, restarted 5 times and tried to access it.... What is going on?
What kind of logs do you need, I really need help!
Quoted 8 lines
From the message you've posted, it looks like something has triggered the (simultaneous) removal of four drives from the array. I'd check the dmesg output - it should provide some information. I'd guess these four drives are all attached to the same controller (are they external or internal?), so possibly the controller reset (or for external drives, it could be a cable issue).
Quoted 5 lines
What would I be looking for on this? Its a lot to sift through.
Currently just line-by-lining it.
Look for where the drives are being kicked out of the array (should be towards the bottom). Just above that should be some error messages (often including bus resets).
Quoted 9 lines
Cheers,
Robin
--___ ( ' } | Robin Hill <[email protected]> | / / ) | Little Jim says .... | // !! | "He fallen in de water !!" |--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Andrew Dunn http://agdunn.net