Raid 5 Problem
Hi. I hope someone can tell me what I have done wrong. I have a 4 disk Raid 5 array running on Fedora9. I've run this array for 2.5 years with no issues. I recently rebooted after upgrading to Kernel 2.6.27.7. When I did this I found that only 3 of my disks were in the array. When I examine the three active elements of the array (/dev/sdd1, /dev/sde1, /dev/sdc1) they all show that the array has 3 drives and one missing. When I examine the missing drive it shows that all members of the array are present, which I don't understand! When I try to add the missing drive back is says the device is busy. Please see below and let me know what I need to do to get this working again. Thanks Nigel:
==================================================================
[root@homepc ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[0] sdc1[3] sde1[1]
735334656 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U]
md_d0 : inactive sdb[2](S)
245117312 blocks
unused devices: <none>
[root@homepc ~]#
==================================================================
[root@homepc ~]# cat /etc/mdadm.conf
# mdadm.conf written out by anaconda # DEVICE partitions MAILADDR [email protected] ARRAY /dev/md0 level=raid5 num-devices=4 devices=/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sde1
[root@homepc ~]#
==================================================================
[root@homepc ~]# mdadm --detail /dev/md0/dev/md0:
Version : 0.90
Creation Time : Tue Apr 18 17:44:34 2006
Raid Level : raid5
Array Size : 735334656 (701.27 GiB 752.98 GB)
Used Dev Size : 245111552 (233.76 GiB 250.99 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Fri Dec 12 17:48:12 2008
State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 128K
UUID : 50e3173e:b5d2bdb6:7db3576b:644409bb
Events : 0.6485812
Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 65 1 active sync /dev/sde1
2 0 0 2 removed
3 8 33 3 active sync /dev/sdc1
[root@homepc ~]# mdadm --examine /dev/sdd1
/dev/sdd1:
Magic : a92b4efc
Version : 0.90.03
UUID : 50e3173e:b5d2bdb6:7db3576b:644409bb
Creation Time : Tue Apr 18 17:44:34 2006
Raid Level : raid5
Used Dev Size : 245111552 (233.76 GiB 250.99 GB)
Array Size : 735334656 (701.27 GiB 752.98 GB) Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Fri Dec 12 17:48:22 2008
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 2ec4d801 - correct
Events : 6485814
Layout : left-symmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 0 8 49 0 active sync /dev/sdd1
0 0 8 49 0 active sync /dev/sdd1
1 1 8 65 1 active sync /dev/sde1
2 2 0 0 2 faulty removed
3 3 8 33 3 active sync /dev/sdc1
[root@homepc ~]# mdadm --examine /dev/sde1
/dev/sde1:
Magic : a92b4efc
Version : 0.90.03
UUID : 50e3173e:b5d2bdb6:7db3576b:644409bb
Creation Time : Tue Apr 18 17:44:34 2006
Raid Level : raid5
Used Dev Size : 245111552 (233.76 GiB 250.99 GB)
Array Size : 735334656 (701.27 GiB 752.98 GB) Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Fri Dec 12 17:48:22 2008
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 2ec4d813 - correct
Events : 6485814
Layout : left-symmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 1 8 65 1 active sync /dev/sde1
0 0 8 49 0 active sync /dev/sdd1
1 1 8 65 1 active sync /dev/sde1
2 2 0 0 2 faulty removed
3 3 8 33 3 active sync /dev/sdc1
[root@homepc ~]# mdadm --examine /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 0.90.03
UUID : 50e3173e:b5d2bdb6:7db3576b:644409bb
Creation Time : Tue Apr 18 17:44:34 2006
Raid Level : raid5
Used Dev Size : 245111552 (233.76 GiB 250.99 GB)
Array Size : 735334656 (701.27 GiB 752.98 GB) Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Fri Dec 12 17:48:22 2008
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 2ec4d7f7 - correct
Events : 6485814
Layout : left-symmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 3 8 33 3 active sync /dev/sdc1
0 0 8 49 0 active sync /dev/sdd1
1 1 8 65 1 active sync /dev/sde1
2 2 0 0 2 faulty removed
3 3 8 33 3 active sync /dev/sdc1
[root@homepc ~]# mdadm --examine /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 0.90.03
UUID : 50e3173e:b5d2bdb6:7db3576b:644409bb
Creation Time : Tue Apr 18 17:44:34 2006
Raid Level : raid5
Used Dev Size : 245111552 (233.76 GiB 250.99 GB)
Array Size : 735334656 (701.27 GiB 752.98 GB) Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Fri Dec 12 17:29:15 2008
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : 2ec4d1d6 - correct
Events : 6485600
Layout : left-symmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 2 8 17 2 active sync /dev/sdb1
0 0 8 49 0 active sync /dev/sdd1 1 1 8 65 1 active sync /dev/sde1 2 2 8 17 2 active sync /dev/sdb1 3 3 8 33 3 active sync /dev/sdc1
[root@homepc ~]# mdadm /dev/md0 --add /dev/sdb1
mdadm: Cannot open /dev/sdb1: Device or resource busy [root@homepc ~]#
Quoted 11 lines · nterry
Hi. I hope someone can tell me what I have done wrong. I have a 4 disk Raid 5 array running on Fedora9. I've run this array for 2.5 years with no issues. I recently rebooted after upgrading to Kernel 2.6.27.7. When I did this I found that only 3 of my disks were in the array. When I examine the three active elements of the array (
/dev/sdd1,/dev/sde1,/dev/sdc1) they all show that the array has 3 drives and one missing. When I examine the missing drive it shows that all members of the array are present, which I don't understand! When I try to add the missing drive back is says the device is busy. Please see below and let me know what I need to do to get this working again. Thanks Nigel:==================================================================
[root@homepc ~]# cat/proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[0] sdc1[3] sde1[1]735334656 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U] md_d0 : inactive sdb[2](S) 245117312 blocks unused devices: <none>[root@homepc ~]#
For some reason, it looks like you have 2 raid arrays visible - md0 and md_d0. The latter took sdb (not sdb1) as its component.
sd{c,d,e}1 is in assembeld array (with appropriately updated superblocks), thus mdadm --examine calls show one device as removed, but sdb is part of another inactive array, and the superblock is untouched and shows "old" situation. Note that 0.9 superblock is stored at the end
of the device (see md(4) for details), so its position could be valid
for both sdb and sdb1.
This might be an effect of --incremental assembly mode. Hard to tell more without seeing startup scripts, mdadm.conf, udev rules, partition layout... Did upgrade involve anything more besides kernel ?
Stop both arrays, check mdadm.conf, assemble md0 manually (mdadm -A /dev/md0 /dev/sd{c,d,e}1 ), verify situation with mdadm -D. If everything looks sane, add /dev/sdb1 to the array. Still, w/o checking out startup stuff, it might happen again after reboot. Adding DEVICE /dev/sd[bcde]1 to mdadm.conf might help though.
Wait a bit for other suggestions as well.
Quoted 15 lines · Michal Soltys
Quoted 11 lines · nterry
Hi. I hope someone can tell me what I have done wrong. I have a 4 disk Raid 5 array running on Fedora9. I've run this array for 2.5 years with no issues. I recently rebooted after upgrading to Kernel 2.6.27.7. When I did this I found that only 3 of my disks were in the array. When I examine the three active elements of the array (
/dev/sdd1,/dev/sde1,/dev/sdc1) they all show that the array has 3 drives and one missing. When I examine the missing drive it shows that all members of the array are present, which I don't understand! When I try to add the missing drive back is says the device is busy. Please see below and let me know what I need to do to get this working again. Thanks Nigel:==================================================================
[root@homepc ~]# cat/proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[0] sdc1[3] sde1[1]735334656 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U] md_d0 : inactive sdb[2](S) 245117312 blocks unused devices: <none>[root@homepc ~]#
For some reason, it looks like you have 2 raid arrays visible - md0 and
md_d0. The latter took sdb (not sdb1) as its component.
Quoted 13 lines
sd{c,d,e}1 is in assembeld array (with appropriately updated superblocks), thus mdadm --examine calls show one device as removed, but sdb is part of another inactive array, and the superblock is untouched and shows "old" situation. Note that 0.9 superblock is stored at the end of the device (see md(4) for details), so its position could be valid for both sdb and sdb1.
This might be an effect of --incremental assembly mode. Hard to tell more without seeing startup scripts, mdadm.conf, udev rules, partition layout... Did upgrade involve anything more besides kernel ?
Stop both arrays, check mdadm.conf, assemble md0 manually (mdadm -A
/dev/md0/dev/sd{c,d,e}1), verify situation with mdadm -D. If everything looks sane, add/dev/sdb1to the array. Still, w/o checking out startup stuff, it might happen again after reboot. Adding DEVICE /dev/sd[bcde]1 to mdadm.conf might help though.Wait a bit for other suggestions as well.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
I don't think the Kernel upgrade actually caused the problem. I tried booting up on an older (2.6.27.5) kernel and that made no difference. I checked the logs for anything else that might have made a difference, but couldn't see anything that made any sense to me. I did note that on an earlier update mdadm was upgraded:
Nov 26 17:08:32 Updated: mdadm-2.6.7.1-1.fc9.x86_64
and I did not reboot after that upgrade
I included my mdadm.conf with the last email and it includes ARRAY /dev/md0 level=raid5 num-devices=4 devices=/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sde1 My configuration is just vanilla Fedora9 with the mdadm.conf I sent
I've never had a /dev/md_d0 array, so that must have been automatically created. I may have had other devices and partitions in /dev/md0 as I know I had several attempts at getting it working 2.5 years ago, and I had other issues when Fedora changed device naming, I think at FC7. There is only one partition on /dev/sdb, see below:
(parted) select /dev/sdb Using /dev/sdb
(parted) print Model: ATA Maxtor 6L250R0 (scsi)
Disk /dev/sdb: 251GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags 1 32.3kB 251GB 251GB primary boot, raid
So it looks like something is creating the /dev/md_d0 and adding /dev/sdb to it before /dev/md0 gets started.
So I tried:
[root@homepc ~]# mdadm --stop /dev/md_d0
mdadm: stopped /dev/md_d0
[root@homepc ~]# mdadm --add /dev/md0 /dev/sdb1
mdadm: re-added /dev/sdb1
[root@homepc ~]# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sdb1[4] sdd1[0] sdc1[3] sde1[1]
735334656 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U]
[>....................] recovery = 0.1% (299936/245111552) finish=81.6min speed=49989K/sec
unused devices: <none>
[root@homepc ~]#
Great - All working. Then I rebooted and was back to square one with only 3 drives in /dev/md0 and /dev/sdb in /dev/md_d0
So I am still not understanding where /dev/md_d0 is coming from and although I know how to get things working after a reboot, clearly this is not a long term solution...
Quoted 58 lines · nterry
Quoted 15 lines · Michal Soltys
Quoted 11 lines · nterry
Hi. I hope someone can tell me what I have done wrong. I have a 4 disk Raid 5 array running on Fedora9. I've run this array for 2.5 years with no issues. I recently rebooted after upgrading to Kernel 2.6.27.7. When I did this I found that only 3 of my disks were in the array. When I examine the three active elements of the array (
/dev/sdd1,/dev/sde1,/dev/sdc1) they all show that the array has 3 drives and one missing. When I examine the missing drive it shows that all members of the array are present, which I don't understand! When I try to add the missing drive back is says the device is busy. Please see below and let me know what I need to do to get this working again. Thanks Nigel:==================================================================
[root@homepc ~]# cat/proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[0] sdc1[3] sde1[1]735334656 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U] md_d0 : inactive sdb[2](S) 245117312 blocks unused devices: <none>[root@homepc ~]#
For some reason, it looks like you have 2 raid arrays visible - md0 and
md_d0. The latter took sdb (not sdb1) as its component.Quoted 13 lines
sd{c,d,e}1 is in assembeld array (with appropriately updated superblocks), thus mdadm --examine calls show one device as removed, but sdb is part of another inactive array, and the superblock is untouched and shows "old" situation. Note that 0.9 superblock is stored at the end of the device (see md(4) for details), so its position could be valid for both sdb and sdb1.
This might be an effect of --incremental assembly mode. Hard to tell more without seeing startup scripts, mdadm.conf, udev rules, partition layout... Did upgrade involve anything more besides kernel ?
Stop both arrays, check mdadm.conf, assemble md0 manually (mdadm -A
/dev/md0/dev/sd{c,d,e}1), verify situation with mdadm -D. If everything looks sane, add/dev/sdb1to the array. Still, w/o checking out startup stuff, it might happen again after reboot. Adding DEVICE /dev/sd[bcde]1 to mdadm.conf might help though.Wait a bit for other suggestions as well.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlI don't think the Kernel upgrade actually caused the problem. I tried booting up on an older (2.6.27.5) kernel and that made no difference. I checked the logs for anything else that might have made a difference, but couldn't see anything that made any sense to me. I did note that on an earlier update mdadm was upgraded:
Nov 26 17:08:32 Updated: mdadm-2.6.7.1-1.fc9.x86_64and I did not reboot after that upgrade
I included my mdadm.conf with the last email and it includes ARRAY
/dev/md0level=raid5 num-devices=4 devices=/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sde1 My configuration is just vanilla Fedora9 with the mdadm.conf I sentI've never had a
/dev/md_d0array, so that must have been automatically created. I may have had other devices and partitions in/dev/md0as I know I had several attempts at getting it working 2.5 years ago, and I had other issues when Fedora changed device naming, I think at FC7. There is only one partition on/dev/sdb, see below:(parted) select
/dev/sdbUsing/dev/sdb
(parted) print Model: ATA Maxtor 6L250R0 (scsi)
Disk/dev/sdb: 251GB
Sector size (logical/physical): 512B/512B
Partition Table: msdosNumber Start End Size Type File system Flags 1 32.3kB 251GB 251GB primary boot, raid
So it looks like something is creating the
/dev/md_d0and adding/dev/sdbto it before/dev/md0gets started.So I tried:
[root@homepc ~]# mdadm --stop/dev/md_d0
mdadm: stopped/dev/md_d0
[root@homepc ~]# mdadm --add/dev/md0/dev/sdb1
mdadm: re-added/dev/sdb1
[root@homepc ~]# cat/proc/mdstatPersonalities : [raid6] [raid5] [raid4] md0 : active raid5 sdb1[4] sdd1[0] sdc1[3] sde1[1]735334656 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U] [>....................] recovery = 0.1% (299936/245111552)
Quoted 7 lines
finish=81.6min speed=49989K/sec
unused devices: <none>
[root@homepc ~]#Great - All working. Then I rebooted and was back to square one with only 3 drives in
/dev/md0and/dev/sdbin/dev/md_d0
So I am still not understanding where/dev/md_d0is coming from and although I know how to get things working after a reboot, clearly this is not a long term solution...
What does:
mdadm --examine --scan
Say?
Are you using a kernel with an initrd+modules or is everything compiled in?
Justin.
Quoted 72 lines · Justin Piszcz
Quoted 58 lines · nterry
Quoted 15 lines · Michal Soltys
Quoted 11 lines · nterry
Hi. I hope someone can tell me what I have done wrong. I have a 4 disk Raid 5 array running on Fedora9. I've run this array for 2.5 years with no issues. I recently rebooted after upgrading to Kernel 2.6.27.7. When I did this I found that only 3 of my disks were in the array. When I examine the three active elements of the array (
/dev/sdd1,/dev/sde1,/dev/sdc1) they all show that the array has 3 drives and one missing. When I examine the missing drive it shows that all members of the array are present, which I don't understand! When I try to add the missing drive back is says the device is busy. Please see below and let me know what I need to do to get this working again. Thanks Nigel:==================================================================
[root@homepc ~]# cat/proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[0] sdc1[3] sde1[1]735334656 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U] md_d0 : inactive sdb[2](S) 245117312 blocks unused devices: <none>[root@homepc ~]#
For some reason, it looks like you have 2 raid arrays visible - md0 and
md_d0. The latter took sdb (not sdb1) as its component.Quoted 13 lines
sd{c,d,e}1 is in assembeld array (with appropriately updated superblocks), thus mdadm --examine calls show one device as removed, but sdb is part of another inactive array, and the superblock is untouched and shows "old" situation. Note that 0.9 superblock is stored at the end of the device (see md(4) for details), so its position could be valid for both sdb and sdb1.
This might be an effect of --incremental assembly mode. Hard to tell more without seeing startup scripts, mdadm.conf, udev rules, partition layout... Did upgrade involve anything more besides kernel ?
Stop both arrays, check mdadm.conf, assemble md0 manually (mdadm -A
/dev/md0/dev/sd{c,d,e}1), verify situation with mdadm -D. If everything looks sane, add/dev/sdb1to the array. Still, w/o checking out startup stuff, it might happen again after reboot. Adding DEVICE /dev/sd[bcde]1 to mdadm.conf might help though.Wait a bit for other suggestions as well.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlI don't think the Kernel upgrade actually caused the problem. I tried booting up on an older (2.6.27.5) kernel and that made no difference. I checked the logs for anything else that might have made a difference, but couldn't see anything that made any sense to me. I did note that on an earlier update mdadm was upgraded:
Nov 26 17:08:32 Updated: mdadm-2.6.7.1-1.fc9.x86_64and I did not reboot after that upgrade
I included my mdadm.conf with the last email and it includes ARRAY
/dev/md0level=raid5 num-devices=4 devices=/dev/sdb1,/dev/sdc1,/dev/sdd1,/dev/sde1 My configuration is just vanilla Fedora9 with the mdadm.conf I sentI've never had a
/dev/md_d0array, so that must have been automatically created. I may have had other devices and partitions in/dev/md0as I know I had several attempts at getting it working 2.5 years ago, and I had other issues when Fedora changed device naming, I think at FC7. There is only one partition on/dev/sdb, see below:(parted) select
/dev/sdbUsing/dev/sdb
(parted) print Model: ATA Maxtor 6L250R0 (scsi)
Disk/dev/sdb: 251GB
Sector size (logical/physical): 512B/512B
Partition Table: msdosNumber Start End Size Type File system Flags 1 32.3kB 251GB 251GB primary boot, raid
So it looks like something is creating the
/dev/md_d0and adding/dev/sdbto it before/dev/md0gets started.So I tried:
[root@homepc ~]# mdadm --stop/dev/md_d0
mdadm: stopped/dev/md_d0
[root@homepc ~]# mdadm --add/dev/md0/dev/sdb1
mdadm: re-added/dev/sdb1
[root@homepc ~]# cat/proc/mdstatPersonalities : [raid6] [raid5] [raid4] md0 : active raid5 sdb1[4] sdd1[0] sdc1[3] sde1[1]735334656 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U] [>....................] recovery = 0.1% (299936/245111552)Quoted 7 lines
finish=81.6min speed=49989K/sec
unused devices: <none>
[root@homepc ~]#Great - All working. Then I rebooted and was back to square one with only 3 drives in
/dev/md0and/dev/sdbin/dev/md_d0
So I am still not understanding
where/dev/md_d0is coming from and although I know how to get things working after a reboot, clearly this is not a long term solution...What does:
Quoted 12 lines
mdadm --examine --scan
Say?
Are you using a kernel with an initrd+modules or is everything compiled in?
Justin.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
[root@homepc ~]# mdadm --examine --scan
ARRAY /dev/md0 level=raid5 num-devices=2 UUID=c57d50aa:1b3bcabd:ab04d342:6049b3f1
spares=1
ARRAY /dev/md0 level=raid5 num-devices=4 UUID=50e3173e:b5d2bdb6:7db3576b:644409bb
spares=1
ARRAY /dev/md0 level=raid5 num-devices=4 UUID=50e3173e:b5d2bdb6:7db3576b:644409bb
spares=1
[root@homepc ~]#
I'm not sure I really know the answer to your second question. I'm using a regular Fedora9 kernel, so I think that is initrd+modules [root@homepc ~]# uname -a
Linux homepc.nigelterry.net 2.6.27.7-53.fc9.x86_64 #1 SMP Thu Nov 27 02:05:02 EST 2008 x86_64 x86_64 x86_64 GNU/Linux [root@homepc ~]#
Quoted 9 lines · nterry
Quoted 7 lines · Justin Piszcz
Quoted 3 lines · nterry
Quoted 2 lines · Michal Soltys
Quoted 1 line · nterry
Hi. I hope someone can tell me what I have done wrong. I have a 4 disk Raid 5 array running on Fedora9. I've run this array for 2.5 years with no issues. I recently rebooted after upgrading to Kernel 2.6.27.7.
Quoted 8 lines
[root@homepc ~]# mdadm --examine --scan
ARRAY/dev/md0level=raid5 num-devices=2 UUID=c57d50aa:1b3bcabd:ab04d342:6049b3f1
spares=1
ARRAY/dev/md0level=raid5 num-devices=4 UUID=50e3173e:b5d2bdb6:7db3576b:644409bb
spares=1
ARRAY/dev/md0level=raid5 num-devices=4 UUID=50e3173e:b5d2bdb6:7db3576b:644409bb
spares=1
[root@homepc ~]#
I saw Debian do something like this to one of my raids once and it was because /etc/mdadm/mdadm.conf had been changed through an upgrade or some such to use md0_X, I changed it back to /dev/md0 and the problem went away.
You have another issue here though, it looks like your "few" attempts have lead to multiple RAID superblocks. I have always wondered how one can clean this up without dd if=/dev/zero of=/dev/dsk & (for each disk, wipe it) to get rid of them all, you should only have [1] /dev/md0 for your raid 5, not 3.
Neil?
Justin.
Quoted 26 lines · Justin Piszcz
Quoted 9 lines · nterry
Quoted 7 lines · Justin Piszcz
Quoted 3 lines · nterry
Quoted 2 lines · Michal Soltys
Quoted 1 line · nterry
Hi. I hope someone can tell me what I have done wrong. I have a 4 disk Raid 5 array running on Fedora9. I've run this array for 2.5 years with no issues. I recently rebooted after upgrading to Kernel 2.6.27.7.
Quoted 8 lines
[root@homepc ~]# mdadm --examine --scan
ARRAY/dev/md0level=raid5 num-devices=2 UUID=c57d50aa:1b3bcabd:ab04d342:6049b3f1
spares=1
ARRAY/dev/md0level=raid5 num-devices=4 UUID=50e3173e:b5d2bdb6:7db3576b:644409bb
spares=1
ARRAY/dev/md0level=raid5 num-devices=4 UUID=50e3173e:b5d2bdb6:7db3576b:644409bb
spares=1
[root@homepc ~]#I saw Debian do something like this to one of my raids once and it was because
/etc/mdadm/mdadm.confhad been changed through an upgrade or some such to usemd0_X, I changed it back to/dev/md0and the problem went away.
Quoted 9 lines
You have another issue here though, it looks like your "few" attempts have lead to multiple RAID superblocks. I have always wondered how one can clean this up without dd if=/dev/zero of=/dev/dsk & (for each disk, wipe it) to get rid of them all, you should only have [1]
/dev/md0for your raid 5, not 3.Neil?
Justin.
The difference in my case is that I don't have /dev/md_d0 in /etc/mdadm.conf and have never had that. It seems that something is automatically creating it at boot and that has changed in the last few days.
Wait for Neil I guess...
Quoted 5 lines · Justin Piszcz
You have another issue here though, it looks like your "few" attempts have lead to multiple RAID superblocks. I have always wondered how one can clean this up without dd if=/dev/zero of=/dev/dsk & (for each disk, wipe it) to get rid of them all, you should only have [1]
/dev/md0for your raid 5, not 3.
Well - 0.9 superblock is 4K at 64K boundary, at least 64K from the end of the device, but less 128K (man 4 md). If a filesystem on some partition taking up that space *didn't* overwrite this space with its [meta]data (check with debugfs, xfs_db, etc.), just clean it up *carefully* with dd or hexeditor.
Quoted 5 lines · nterry
Great - All working. Then I rebooted and was back to square one with only 3 drives in
/dev/md0and/dev/sdbin/dev/md_d0
So I am still not understanding where/dev/md_d0is coming from and although I know how to get things working after a reboot, clearly this is not a long term solution...
My blind shot is that udev rules of your distro are doing mdadm --incremental assembly and picking sdb as a part of nonexisting array from the long ago (leftover after old experimentations ?). Or something else is doing so.
What does mdadm -Esvv /dev/sdb show ?
Add
DEVICE /dev/sd[bcde]1
on top of your mdadm.conf - it should stop --incremental from picking up sdb. Assuming that's the cause of the problem.
Also note, that FC9 might be trying to assemble the array during initramfs stage (assuming it uses one) and having problems there. I've never used Fedora so hard to tell for me - but definitely peek there, particulary at udev and mdadm part of things.
Quoted 13 lines · Michal Soltys
Quoted 5 lines · nterry
Great - All working. Then I rebooted and was back to square one with only 3 drives in
/dev/md0and/dev/sdbin/dev/md_d0
So I am still not understanding
where/dev/md_d0is coming from and although I know how to get things working after a reboot, clearly this is not a long term solution...My blind shot is that udev rules of your distro are doing mdadm --incremental assembly and picking sdb as a part of nonexisting array
from the long ago (leftover after old experimentations ?). Or something else is doing so.What does mdadm -Esvv
/dev/sdbshow ?
Quoted 13 lines
Add
DEVICE /dev/sd[bcde]1
on top of your mdadm.conf - it should stop --incremental from picking up sdb. Assuming that's the cause of the problem.
Also note, that FC9 might be trying to assemble the array during initramfs stage (assuming it uses one) and having problems there. I've never used Fedora so hard to tell for me - but definitely peek there, particulary at udev and mdadm part of things.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
[root@homepc ~]# mdadm -Esvv /dev/sdb/dev/sdb:
Magic : a92b4efc
Version : 0.90.02
UUID : c57d50aa:1b3bcabd:ab04d342:6049b3f1
Creation Time : Thu Dec 15 15:29:36 2005
Raid Level : raid5
Used Dev Size : 245111552 (233.76 GiB 250.99 GB)
Array Size : 245111552 (233.76 GiB 250.99 GB) Raid Devices : 2
Total Devices : 3
Preferred Minor : 0
Update Time : Wed Apr 5 13:43:20 2006
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
Checksum : 2bd59790 - correct
Events : 1530654
Layout : left-symmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 2 22 0 2 spare
0 0 8 1 0 active sync /dev/sda1 1 1 8 17 1 active sync /dev/sdb1 2 2 22 0 2 spare
[root@homepc ~]#
I added the DEVICE /dev/sd[bcde]1 to mdadm.conf and that apepars to have fixed the problem. 2 reboots and it worked both times.
I also note now that:
[root@homepc ~]# mdadm --examine --scan
ARRAY /dev/md0 level=raid5 num-devices=4 UUID=50e3173e:b5d2bdb6:7db3576b:644409bb
spares=1
[root@homepc ~]#
Frankly I don't know enough about the workings of udev and the boot process to be able to get into that. However these two files might mean something to you:
[root@homepc ~]# cat /etc/udev/rules.d/64-md-raid.rules
# do not edit this file, it will be overwritten on update
SUBSYSTEM!="block", GOTO="md_end"
ACTION!="add|change", GOTO="md_end"
# import data from a raid member and activate it
#ENV{ID_FS_TYPE}=="linux_raid_member", IMPORT{program}="/sbin/mdadm --examine --export $tempnode", RUN+="/sbin/mdadm --incremental $env{DEVNAME}"
# import data from a raid set
KERNEL!="md*", GOTO="md_end"ATTR{md/array_state}=="|clear|inactive", GOTO="md_end"
IMPORT{program}="/sbin/mdadm --detail --export $tempnode" ENV{MD_NAME}=="?*", SYMLINK+="disk/by-id/md-name-$env{MD_NAME}" ENV{MD_UUID}=="?*", SYMLINK+="disk/by-id/md-uuid-$env{MD_UUID}"
IMPORT{program}="vol_id --export $tempnode"
OPTIONS="link_priority=100"
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}" ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_LABEL_ENC}=="?*", SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"
LABEL="md_end"
[root@homepc ~]#
AND...
[root@homepc ~]# cat /etc/udev/rules.d/70-mdadm.rules
# This file causes block devices with Linux RAID (mdadm) signatures to
# automatically cause mdadm to be run. # See udev(8) for syntax
SUBSYSTEM=="block", ACTION=="add|change", ENV{ID_FS_TYPE}=="linux_raid*", \
RUN+="/sbin/mdadm -I --auto=yes $root/%k"
[root@homepc ~]#
Thanks for getting me working
Nigel
Quoted 7 lines · nterry
[root@homepc ~]# mdadm -Esvv
/dev/sdb/dev/sdb:Magic : a92b4efc Version : 0.90.02 UUID : c57d50aa:1b3bcabd:ab04d342:6049b3f1Creation Time : Thu Dec 15 15:29:36 2005
Raid Level : raid5
Yup - that's leftover block from one of your earlier arrays/attempts. It just happens to sit in unused area of /dev/sdb1, but is picked as valid /dev/sdb superblock
Quoted 8 lines
I added the DEVICE /dev/sd[bcde]1 to mdadm.conf and that apepars to have fixed the problem. 2 reboots and it worked both times.
I also note now that:
[root@homepc ~]# mdadm --examine --scan
ARRAY/dev/md0level=raid5 num-devices=4 UUID=50e3173e:b5d2bdb6:7db3576b:644409bb
spares=1
[root@homepc ~]#
That DEVICE line limits mdadm to only look at sd[bcde]1 as potential array members, so examine shows only current array. If you plan to add more raid arrays later, be sure to change it. Also read below.
Quoted 6 lines
Frankly I don't know enough about the workings of udev and the boot process to be able to get into that. However these two files might mean something to you:
[...]
AND...
Quoted 9 lines
[root@homepc ~]# cat
/etc/udev/rules.d/70-mdadm.rules
# This file causes block devices with Linux RAID (mdadm) signatures to# automatically cause mdadm to be run. # See udev(8) for syntaxSUBSYSTEM=="block", ACTION=="add|change", ENV{ID_FS_TYPE}=="linux_raid*", \ RUN+="/sbin/mdadm -I --auto=yes $root/%k"[root@homepc ~]#
That's the part of the problem. In human language it means, that during udev event related to device add or change, any device detected as being part of linux raid, will be incrementally assembled. ID_FS_TYPE is exported in one of the earlier rules by vol_id helper tool. So during initial udev run, /dev/sdb is picked up, and forces mdadm later to drop /dev/sdb1 from array during every boot.
You can comment it out safely (or just remove whole 70-mdadm.rules file). Then you can remove DEVICE line from mdadm.conf
Quoted 2 lines
Thanks for getting me working
np :)
Quoted 3 lines · On Sunday December 14, [email protected] wrote:
I added the DEVICE /dev/sd[bcde]1 to mdadm.conf and that apepars to have fixed the problem. 2 reboots and it worked both times.
An alternate fix in this case would be
mdadm --zero-superblock /dev/sdb
to remove the old superblock that is confusing things.
NeilBrown
Quoted 9 lines · Neil Brown
Quoted 2 lines · On Sunday December 14, [email protected] wrote:
I added the DEVICE /dev/sd[bcde]1 to mdadm.conf and that apepars to have fixed the problem. 2 reboots and it worked both times.
An alternate fix in this case would be
mdadm --zero-superblock /dev/sdb
Quoted 10 lines
to remove the old superblock that is confusing things.
NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
That fails as:
[root@homepc ~]# mdadm --zero-superblock /dev/sdb
mdadm: Couldn't open /dev/sdb for write - not zeroing [root@homepc ~]#
I also discovered that /dev/sdc appears to have a superblock which maybe explains why # mdadm --examine --scan throws up three arrays. Trying to zero the superblock on /dev/sdc gives the same error.
Quoted 16 lines · nterry
Quoted 15 lines · Neil Brown
An alternate fix in this case would be
mdadm --zero-superblock /dev/sdbto remove the old superblock that is confusing things.
NeilBrown
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines
That fails as:
[root@homepc ~]# mdadm --zero-superblock
/dev/sdb
mdadm: Couldn't open/dev/sdbfor write - not zeroing [root@homepc ~]#I also discovered that
/dev/sdcappears to have a superblock which maybe explains why # mdadm --examine --scan throws up three arrays. Trying to zero the superblock on/dev/sdcgives the same error.
OK, I solved it, but not in a clean manner. I had to remove /dev/sdb1 from the array before I could zero the superblock on /dev/sdb as below:
[root@homepc ~]# mdadm /dev/md0 --fail /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0 [root@homepc ~]# mdadm /dev/md0 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1
[root@homepc ~]# mdadm --zero-superblock --verbose /dev/sdb [root@homepc ~]# mdadm /dev/md0 --re-add /dev/sdb1
mdadm: re-added /dev/sdb1
[root@homepc ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[4] sdd1[0] sdc1[3] sde1[1]
735334656 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U]
[>....................] recovery = 0.0% (204800/245111552) finish=79.6min speed=51200K/sec
unused devices: <none>
I did this on both sdb and sdc and now I only have the one array when I mdadm --examine --scan --verbose. However is there a better way to do this that doesn't involve a full recovery? I thought --re-add would handle that?