md RAID5: Disk wrongly marked "spare", need to force re-add it
I have a RAID5 with 8 disks. It worked fine.
I made an update from ubuntu 10.04 to 12.04 using do-release-upgrade, and rebooted.
before: kernel 2.6.32-41, mdadm v2.6.7.1, after: kernel 3.2.0-41, mdadm 3.2.5 - and I made a copy of the root partition before I updated, so I can still boot both OSs
After I rebooted, one drive dropped from the array. I don't know why: it was fine hardware-wise, and the same happened with my other RAID5 array, where the dropped drive was fine, too. This seems like an MD or Ubuntu bug to me.
So, I readded it and resynced. When the resync was around or over 80% done, another harddrive failed (maybe because the resync is now using all sectors the first time since a long time), this one with a real hardware failure, so I had to permanently removed it. Now, the first drive, which was dropped although it was fine, is marked as spare, although it should have mostly good data on it. mdadm refuses to re-add it, no matter what I try. mdadm 3.2.5 says "not possible" (see [1]), while mdadm 2.5.7.1 re-adds it, but as spare, not as real member, so the array still won't restart.
I need to forcibly re-add the drive that's marked as spare, because it should have good data on it. I understand that some blocks may be out of sync or corrupted, but the array has many TB and I want to get to the rest of all the data that's still good. Even if I get 80% recovered, that's still better than 0%.
NB 1:
Please do NOT respond with
* restore backup - the backup doesn't have the new data, which I really need
* re-"create" the array - unless you can give me the exact --create command that would recover it with data - other people tried this based on suggestions in forums and they lost all data
Appendix 1:
dmesg during resync:
...
[45345.341865] XFS (dm-13): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
[45345.341909] XFS (dm-13): metadata I/O error: block 0x45986fc0 ("xfs_trans_read_buf") error 5 buf count 8192(many times, about the same handful of blocks)
[45345.610858] XFS (dm-13): metadata I/O error: block 0x7a434f20 ("xfs_trans_read_buf") error 5 buf count 4096(many times, about the same handful of blocks)
(And then probably MD shut down the disk, and the RAID, and so all the other filesystems went in the trash, too:)
[45353.184049] XFS (dm-10): xfs_log_force: error 5 returned.
[45472.694283] XFS (dm-8): metadata I/O error: block 0x29e89be8 ("xfs_trans_read_buf") error 5 buf count 4096
[45473.504044] XFS (dm-10): xfs_log_force: error 5 returned.
[45485.246757] XFS (dm-8): metadata I/O error: block 0x19010c29 ("xlog_iodone") error 5 buf count 2560
[45485.248966] XFS (dm-8): xfs_do_force_shutdown(0x2) called from line 1007 of file /build/buildd/linux-3.2.0/fs/xfs/xfs_log.c. Return address = 0xffffffffa031ede1
[45485.249011] XFS (dm-8): Log I/O Error Detected. Shutting down filesystem
[45485.251126] XFS (dm-8): Please umount the filesystem and rectify the problem(s)
[45503.584037] XFS (dm-10): xfs_log_force: error 5 returned.
[45514.848040] XFS (dm-8): xfs_log_force: error 5 returned.Appendix 2:
1. During resync, i.e. after sdl was wrongly dropped, and before sdk failed:
# cat /proc/mdstat
md0 : active raid5 sdl[8] sdp[5] sdq[7] sdk[1] sdj[0] sdo[4] sdn[6] sdm[3]
6837337472 blocks level 5, 64k chunk, algorithm 2 [8/7] [UU_UUUUU]
[>....................] recovery = 0.1% (1075328/976762496)
finish=468.6min speed=34700K/sec# mdadm --detail /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Persistence : Superblock is persistent Update Time : Thu Apr 11 16:59:55 2013
State : clean, degraded, recovering
Active Devices : 7
Working Devices : 8
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Rebuild Status : 0% complete
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Events : 0.13240512
Number Major Minor RaidDevice State
0 8 144 0 active sync /dev/sdj
1 8 160 1 active sync /dev/sdk
8 8 176 2 spare rebuilding /dev/sdl
3 8 192 3 active sync /dev/sdm
4 8 224 4 active sync /dev/sdo
5 8 240 5 active sync /dev/sdp
6 8 208 6 active sync /dev/sdn
7 65 0 7 active sync /dev/sdq
2. after sdk had a hardware fault during resync:
after resync, towards the end (between 78 and 100%), again:
md0 : active raid5 sdj[0] sdl[8](S) sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] sdk[9](F)
6837337472 blocks level 5, 64k chunk, algorithm 2 [8/6] [U__UUUUU]
*-disk:1
description: ATA Disk
product: WDC WD10EACS-00D
vendor: Western Digital
physical id: 0.1.0
bus info: scsi@7:0.1.0
logical name: /dev/sdk
version: 1A01
serial: WD-...8520
size: 931GiB (1TB)
capacity: 931GiB (1TB)
capabilities: 15000rpm
configuration: ansiversion=5*-disk:2
description: ATA Disk
product: WDC WD10EACS-00D
vendor: Western Digital
physical id: 0.2.0
bus info: scsi@7:0.2.0
logical name: /dev/sdl
version: 1A01
serial: WD-WCAU45964913
size: 931GiB (1TB)
capacity: 931GiB (1TB)
capabilities: 15000rpm
configuration: ansiversion=53. Current state, after fix attempts:
(sdk has hardware failure
sdl is probably good, but marked spare)
# cat /proc/mdstat
md0 : inactive sdk[9](S) sdl[8](S) sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3]
7814099968 blocks# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Persistence : Superblock is persistent Update Time : Fri Apr 12 17:09:30 2013
State : active, degraded, Not Started
Active Devices : 6
Working Devices : 8
Failed Devices : 0
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 64K
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Events : 0.13274865
Number Major Minor RaidDevice State
0 8 144 0 active sync /dev/sdj
1 0 0 1 removed
2 0 0 2 removed
3 8 192 3 active sync /dev/sdm
4 8 224 4 active sync /dev/sdo
5 8 240 5 active sync /dev/sdp
6 8 208 6 active sync /dev/sdn
7 65 0 7 active sync /dev/sdq
8 8 176 - spare /dev/sdl
9 8 160 - spare /dev/sdk
# mdadm -E /dev/sd[jlmnopqk]
(sdl is the one I need to add:)/dev/sdl:
Magic : a92b4efc
Version : 00.90.00
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB) Raid Devices : 8
Total Devices : 7
Preferred Minor : 0
Update Time : Fri Apr 12 15:00:38 2013
State : clean
Active Devices : 6
Working Devices : 7
Failed Devices : 2
Spare Devices : 1
Checksum : ca6e81a9 - correct
Events : 13274863
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 8 176 8 spare /dev/sdl
0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq 8 8 8 176 8 spare /dev/sdl
/dev/sdj:
Magic : a92b4efc
Version : 00.90.00
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB) Raid Devices : 8
Total Devices : 6
Preferred Minor : 0
Update Time : Fri Apr 12 17:09:30 2013
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : c9a40ffb - correct
Events : 13274865
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 144 0 active sync /dev/sdj
0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdm:
Magic : a92b4efc
Version : 00.90.00
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB) Raid Devices : 8
Total Devices : 6
Preferred Minor : 0
Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : c9a41030 - correct
Events : 13274865
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 192 3 active sync /dev/sdm
0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdn:
Magic : a92b4efc
Version : 00.90.00
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB) Raid Devices : 8
Total Devices : 6
Preferred Minor : 0
Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : c9a41046 - correct
Events : 13274865
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 208 6 active sync /dev/sdn
0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdo:
Magic : a92b4efc
Version : 00.90.00
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB) Raid Devices : 8
Total Devices : 6
Preferred Minor : 0
Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : c9a41052 - correct
Events : 13274865
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 224 4 active sync /dev/sdo
0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdp:
Magic : a92b4efc
Version : 00.90.00
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB) Raid Devices : 8
Total Devices : 6
Preferred Minor : 0
Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : c9a41064 - correct
Events : 13274865
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 240 5 active sync /dev/sdp
0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdq:
Magic : a92b4efc
Version : 00.90.00
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB) Raid Devices : 8
Total Devices : 6
Preferred Minor : 0
Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : c9a40fb1 - correct
Events : 13274865
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 65 0 7 active sync /dev/sdq
0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
(sdk with serial ...8520 had the hardware fault:)
/dev/sdk:
Magic : a92b4efc
Version : 00.90.00
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Creation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB) Raid Devices : 8
Total Devices : 6
Preferred Minor : 0
Update Time : Fri Apr 12 17:09:30 2013
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum : c9a410bf - correct
Events : 13274865
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 9 8 160 -1 spare /dev/sdk
0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
NB 2:
The original problem is that md dropped a perfectly good drive from the array, just because I upgraded the OS. It seems to me that Linux MD is all too happy and quick to kick out drives from the array, and then refuses to readd them without a resync. This might be fine approach on paper, but not in reality, where the resync is probably when another drive fails, and then you have no parity left and you're told that your data is gone.
1. It shouldn't drop drives so quickly
2. It should allow me to re-add them, if I think the data is good
3. There must be a recovery mechanism, to at least partially recover data. Arrays can easily have 10+ TB, and just because a few blocks/sectors in one filesystem are bad doesn't mean that I need to throw away all filesystems that are on that LVM, and all data in the broken filesystem.
NB 3: Seems like other people have the exact same problem:
http://www.linuxquestions.org/questions/linux-server-73/mdadm-re-added-disk-treated-as-spare-750739/
http://forums.gentoo.org/viewtopic-t-716757.html
https://raid.wiki.kernel.org/index.php/RAID_Recovery#Recreating_an_array
NB 4: Last time I upgraded the OS on the RAID server, I ended up with a similar mess, due to another md bug: https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/136252 )
Quoted 1 line
I have a RAID5 with 8 disks. It worked fine.
It's a thead a week or so about people who run lots of drives in RAID-5 and lose a drive, and then anothoer during rebuild, or in other ways have double disk failure. With eight drives in a single RAID-8 and no good backup, it's like asking for trouble. Use RAID-6.
Vennlige hilsener / Best regards
roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Ben Bucksch
I have a RAID5 with 8 disks. It worked fine.
Raid5 on 8 disks, EEP. Raid6! Really, raid6.
Quoted 11 lines
I made an update from ubuntu 10.04 to 12.04 using do-release-upgrade, and rebooted.
before: kernel 2.6.32-41, mdadm v2.6.7.1, after: kernel 3.2.0-41, mdadm 3.2.5 - and I made a copy of the root partition before I updated, so I can still boot both OSsAfter I rebooted, one drive dropped from the array. I don't know why: it was fine hardware-wise, and the same happened with my other RAID5 array, where the dropped drive was fine, too. This seems like an MD or Ubuntu bug to me.
I've seen that happen to, but with all disks marked as spare. This apparently could be race condition with udev etc.
You should have tried mdadm -A /dev/md0 and see if it would reassemble it.
Quoted 11 lines
So, I readded it and resynced. When the resync was around or over 80% done, another harddrive failed (maybe because the resync is now using all sectors the first time since a long time), this one with a real hardware failure, so I had to permanently removed it. Now, the first drive, which was dropped although it was fine, is marked as spare, although it should have mostly good data on it. mdadm refuses to re-add it, no matter what I try. mdadm 3.2.5 says "not possible" (see [1]), while mdadm 2.5.7.1 re-adds it, but as spare, not as real member, so the array still won't restart.
Quoted 5 lines
I need to forcibly re-add the drive that's marked as spare, because it should have good data on it. I understand that some blocks may be out of sync or corrupted, but the array has many TB and I want to get to the rest of all the data that's still good. Even if I get 80% recovered, that's still better than 0%.
Firstly, have you written anything TOO the array while resyncing? If not, chances are your array is in a reasonable shape still.
The 'spare' drive, I don't know what its status is. Theoretically, I would assume that the resync the data written to the disk is exactly the same as it was before, so keep that in mind as a last resort. But basically, you should ignore this drive, its data is not to be trusted.
Now the broken drive. Check your cables!! and run smartctl on it to give smart a chance to 'fix' the drive somewhat and check its status/health.
Now check the event count for all your drivers and compare. If the 'broken' drive is only a few off (1 or 2 I think i spotted below, try the following)
mdadm --run --force -A /dev/md0 /dev/sd[1-7] (leave out the earlier 'spare')
You should have your array back up and running. mount -o ro /md0 /mnt and copy anything you need off.
IF can't recover any files (due to not having enough free space) and if smartctl said your broken drive was somewhat sane, you can try re-adding it again and hope it'll work this time. If it won't let you, mdadm --zero-superblock /dev/brokendisk.
But do try to copy the most important stuff off, continuing may make things worse.
If it fails again (at 80% because of hardware failure) you can't re-use the broken disk. It really is broken :p Re-force the assembly as above and copy the rest, it's all you can do.
That said, if all else fails, your very last hope, is to not use the broken drive, and 'force' the above using the earlier marked spare. Maybe you can get more data off the array then.
After recovering your data and replacing your broken disk, make it an 8 disk raid6 instead! (or if you need the space, 10disk raid6. Raid5 while awesome, on big big arrays it's asking for trouble still.
Quoted 511 lines
NB 1:
Please do NOT respond with* restore backup - the backup doesn't have the new data, which I really need
* re-"create" the array - unless you can give me the exact --create command that would recover it with data - other people tried this based on suggestions in forums and they lost all dataAppendix 1:
dmesg during resync:
...
[45345.341865] XFS (dm-13): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5. [45345.341909] XFS (dm-13): metadata I/O error: block 0x45986fc0("
xfs_trans_read_buf") error 5 buf count 8192(many times, about the same handful of blocks)
[45345.610858] XFS (dm-13): metadata I/O error: block 0x7a434f20("
xfs_trans_read_buf") error 5 buf count 4096(many times, about the same handful of blocks)
(And then probably MD shut down the disk, and the RAID, and so all the other filesystems went in the trash, too:)
[45353.184049] XFS (dm-10): xfs_log_force: error 5 returned. [45472.694283] XFS (dm-8): metadata I/O error: block 0x29e89be8 ("xfs_trans_read_buf") error 5 buf count 4096 [45473.504044] XFS (dm-10): xfs_log_force: error 5 returned. [45485.246757] XFS (dm-8): metadata I/O error: block 0x19010c29 ("xlog_iodone") error 5 buf count 2560 [45485.248966] XFS (dm-8): xfs_do_force_shutdown(0x2) called from line 1007 of file /build/buildd/linux-3.2.0/fs/xfs/xfs_log.c. Return address = 0xffffffffa031ede1 [45485.249011] XFS (dm-8): Log I/O Error Detected. Shutting down filesystem [45485.251126] XFS (dm-8): Please umount the filesystem and rectify the problem(s) [45503.584037] XFS (dm-10): xfs_log_force: error 5 returned. [45514.848040] XFS (dm-8): xfs_log_force: error 5 returned.Appendix 2:
1. During resync, i.e. after sdl was wrongly dropped, and before sdk failed:
# cat /proc/mdstat md0 : active raid5 sdl[8] sdp[5] sdq[7] sdk[1] sdj[0] sdo[4] sdn[6] sdm[3] 6837337472 blocks level 5, 64k chunk, algorithm 2 [8/7] [UU_UUUUU] [>....................] recovery = 0.1% (1075328/976762496) finish=468.6min speed=34700K/sec# mdadm --detail /dev/md0 /dev/md0: Version : 0.90 Creation Time : Sun Mar 22 15:51:17 2009 Raid Level : raid5 Array Size : 6837337472 (6520.59 GiB 7001.43 GB) Used Dev Size : 976762496 (931.51 GiB 1000.20 GB) Raid Devices : 8 Total Devices : 8 Preferred Minor : 0 Persistence : Superblock is persistentUpdate Time : Thu Apr 11 16:59:55 2013
State : clean, degraded, recovering
Active Devices : 7
Working Devices : 8
Failed Devices : 0
Spare Devices : 1Layout : left-symmetric
Chunk Size : 64KRebuild Status : 0% complete
UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Events : 0.13240512Number Major Minor RaidDevice State
0 8 144 0 active sync/dev/sdj
1 8 160 1 active sync/dev/sdk
8 8 176 2 spare rebuilding/dev/sdl
3 8 192 3 active sync/dev/sdm
4 8 224 4 active sync/dev/sdo
5 8 240 5 active sync/dev/sdp
6 8 208 6 active sync/dev/sdn
7 65 0 7 active sync/dev/sdq2. after sdk had a hardware fault during resync:
after resync, towards the end (between 78 and 100%), again:
md0 : active raid5 sdj[0] sdl[8](S) sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] sdk[9](F)
6837337472 blocks level 5, 64k chunk, algorithm 2 [8/6] [U__UUUUU]*-disk:1 description: ATA Disk product: WDC WD10EACS-00D vendor: Western Digital physical id: 0.1.0 bus info: scsi@7:0.1.0 logical name: /dev/sdk version: 1A01 serial: WD-...8520 size: 931GiB (1TB) capacity: 931GiB (1TB) capabilities: 15000rpm configuration: ansiversion=5 *-disk:2 description: ATA Disk product: WDC WD10EACS-00D vendor: Western Digital physical id: 0.2.0 bus info: scsi@7:0.2.0 logical name: /dev/sdl version: 1A01 serial: WD-WCAU45964913 size: 931GiB (1TB) capacity: 931GiB (1TB) capabilities: 15000rpm configuration: ansiversion=53. Current state, after fix attempts:
(sdk has hardware failure
sdl is probably good, but marked spare)# cat /proc/mdstat md0 : inactive sdk[9](S) sdl[8](S) sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 7814099968 blocks# mdadm --detail /dev/md0 /dev/md0: Version : 00.90 Creation Time : Sun Mar 22 15:51:17 2009 Raid Level : raid5 Used Dev Size : 976762496 (931.51 GiB 1000.20 GB) Raid Devices : 8 Total Devices : 8 Preferred Minor : 0 Persistence : Superblock is persistentUpdate Time : Fri Apr 12 17:09:30 2013
State : active, degraded, Not Started
Active Devices : 6
Working Devices : 8
Failed Devices : 0
Spare Devices : 2Layout : left-symmetric
Chunk Size : 64KUUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41c
Events : 0.13274865Number Major Minor RaidDevice State
0 8 144 0 active sync/dev/sdj
1 0 0 1 removed
2 0 0 2 removed
3 8 192 3 active sync/dev/sdm
4 8 224 4 active sync/dev/sdo
5 8 240 5 active sync/dev/sdp
6 8 208 6 active sync/dev/sdn
7 65 0 7 active sync/dev/sdq8 8 176 - spare
/dev/sdl
9 8 160 - spare/dev/sdk# mdadm -E /dev/sd[jlmnopqk](sdl is the one I need to add:)
/dev/sdl:Magic : a92b4efc Version : 00.90.00 UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41cCreation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Raid Devices : 8
Total Devices : 7
Preferred Minor : 0Update Time : Fri Apr 12 15:00:38 2013
State : clean
Active Devices : 6
Working Devices : 7
Failed Devices : 2
Spare Devices : 1
Checksum :ca6e81a9- correct
Events : 13274863Layout : left-symmetric
Chunk Size : 64KNumber Major Minor RaidDevice State
this 8 8 176 8 spare/dev/sdl0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq 8 8 8 176 8 spare /dev/sdl
/dev/sdj:Magic : a92b4efc Version : 00.90.00 UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41cCreation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 0Update Time : Fri Apr 12 17:09:30 2013
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum :c9a40ffb- correct
Events : 13274865Layout : left-symmetric
Chunk Size : 64KNumber Major Minor RaidDevice State
this 0 8 144 0 active sync/dev/sdj0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdm:Magic : a92b4efc Version : 00.90.00 UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41cCreation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 0Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum :c9a41030- correct
Events : 13274865Layout : left-symmetric
Chunk Size : 64KNumber Major Minor RaidDevice State
this 3 8 192 3 active sync/dev/sdm0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdn:Magic : a92b4efc Version : 00.90.00 UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41cCreation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 0Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum :c9a41046- correct
Events : 13274865Layout : left-symmetric
Chunk Size : 64KNumber Major Minor RaidDevice State
this 6 8 208 6 active sync/dev/sdn0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdo:Magic : a92b4efc Version : 00.90.00 UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41cCreation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 0Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum :c9a41052- correct
Events : 13274865Layout : left-symmetric
Chunk Size : 64KNumber Major Minor RaidDevice State
this 4 8 224 4 active sync/dev/sdo0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdp:Magic : a92b4efc Version : 00.90.00 UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41cCreation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 0Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum :c9a41064- correct
Events : 13274865Layout : left-symmetric
Chunk Size : 64KNumber Major Minor RaidDevice State
this 5 8 240 5 active sync/dev/sdp0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq
/dev/sdq:Magic : a92b4efc Version : 00.90.00 UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41cCreation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 0Update Time : Fri Apr 12 17:09:30 2013
State : active
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum :c9a40fb1- correct
Events : 13274865Layout : left-symmetric
Chunk Size : 64KNumber Major Minor RaidDevice State
this 7 65 0 7 active sync/dev/sdq0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdq(sdk with serial ...8520 had the hardware fault:)
/dev/sdk:Magic : a92b4efc Version : 00.90.00 UUID : c71c4168:de3a9b44:5ac2d0d1:4a2cd41cCreation Time : Sun Mar 22 15:51:17 2009
Raid Level : raid5
Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
Array Size : 6837337472 (6520.59 GiB 7001.43 GB)
Raid Devices : 8
Total Devices : 6
Preferred Minor : 0Update Time : Fri Apr 12 17:09:30 2013
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 2
Spare Devices : 0
Checksum :c9a410bf- correct
Events : 13274865Layout : left-symmetric
Chunk Size : 64KNumber Major Minor RaidDevice State
this 9 8 160 -1 spare/dev/sdk0 0 8 144 0 active sync /dev/sdj 1 1 0 0 1 faulty removed 2 2 0 0 2 faulty removed 3 3 8 192 3 active sync /dev/sdm 4 4 8 224 4 active sync /dev/sdo 5 5 8 240 5 active sync /dev/sdp 6 6 8 208 6 active sync /dev/sdn 7 7 65 0 7 active sync /dev/sdqNB 2:
The original problem is that md dropped a perfectly good drive from the array, just because I upgraded the OS. It seems to me that Linux MD is all too happy and quick to kick out drives from the array, and then refuses to readd them without a resync. This might be fine approach on paper, but not in reality, where the resync is probably when another drive fails, and then you have no parity left and you're told that your data is gone.
1. It shouldn't drop drives so quickly
2. It should allow me to re-add them, if I think the data is good
3. There must be a recovery mechanism, to at least partially recover data. Arrays can easily have 10+ TB, and just because a few blocks/sectors in one filesystem are bad doesn't mean that I need to throw away all filesystems that are on that LVM, and all data in the broken filesystem.NB 3: Seems like other people have the exact same problem:
http://www.linuxquestions.org/questions/linux-server-73/mdadm-re-added-disk-treated-as-spare-750739/http://forums.gentoo.org/viewtopic-t-716757.html
https://raid.wiki.kernel.org/index.php/RAID_Recovery#Recreating_an_arrayNB 4: Last time I upgraded the OS on the RAID server, I ended up with a similar mess, due to another md bug:
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/136252 )--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hey Oliver,
first off: thanks for trying to help me.
Oliver Schinagl wrote, On 15.04.2013 00:40:
Quoted 1 line
Firstly, have you written anything TOO the array while resyncing? If not, chances are your array is in a reasonable shape still.
I did write to the array (in fact, I did a bonnie++, which in retrospective is very stupid, and I'm upset I did it, but hindsight is 20/20 - I assumed the array was fine at that time), BUT if you look at the "event count" of each drive, the sdl marked "spare" has an event count just 2 lower then all the others, so they are very close.
Quoted 1 line
Now check the event count for all your drivers and compare. If the 'broken' drive is only a few off (1 or 2 I think i spotted below, try the following)
Exactly.
Quoted 1 line
The 'spare' drive, I don't know what its status is.
According to SMART, it's just fine. Its event status is very close to the others.
Quoted 1 line
Theoretically, I would assume that the resync the data written to the disk is exactly the same as it was before, so keep that in mind as a last resort.
Yes, that's my plan. My question is: HOW can I tell mdadm to use it?
Quoted 1 line
mdadm --run --force -A
/dev/md0/dev/sd...
I've tried that, and it tells me the array can't be started, because I have RAID 5 with 8 drives (in normal situation), 6 good drives, and 2 spares (1 working fine, 1 with hardware failure). So, after this command, I end up in "inactive" operation mode.
Quoted 2 lines
Now the broken drive. Check your cables!! and run smartctl on it to give smart a chance to 'fix' the drive somewhat and check its status/health. ... If it fails again (at 80% because of hardware failure) you can't re-use the broken disk. It really is broken :p
It failed twice during resync, at around the same point, and smartctl tells me it's broken, so I assume it's gone for good. (Also, the failed drive is also marked as "spare" currently.)
Quoted 1 line
your very last hope, is to not use the broken drive, and 'force' the above using the earlier marked spare.
How? I haven't managed to do that, that's my whole question.
Quoted 17 lines · Ben Bucksch
Hey Oliver,
first off: thanks for trying to help me.
Oliver Schinagl wrote, On 15.04.2013 00:40:
Quoted 1 line
Firstly, have you written anything TOO the array while resyncing? If not, chances are your array is in a reasonable shape still.
I did write to the array (in fact, I did a bonnie++, which in retrospective is very stupid, and I'm upset I did it, but hindsight is 20/20 - I assumed the array was fine at that time), BUT if you look at the "event count" of each drive, the sdl marked "spare" has an event count just 2 lower then all the others, so they are very close.
Quoted 1 line
Now check the event count for all your drivers and compare. If the 'broken' drive is only a few off (1 or 2 I think i spotted below, try the following)
Exactly.
Quoted 1 line
The 'spare' drive, I don't know what its status is.
According to SMART, it's just fine. Its event status is very close to the others.
Quoted 7 lines
Quoted 1 line
Theoretically, I would assume that the resync the data written to the disk is exactly the same as it was before, so keep that in mind as a last resort.
Yes, that's my plan. My question is: HOW can I tell mdadm to use it?
Quoted 1 line
mdadm --run --force -A
/dev/md0/dev/sd...I've tried that, and it tells me the array can't be started, because I have RAID 5 with 8 drives (in normal situation), 6 good drives, and 2 spares (1 working fine, 1 with hardware failure). So, after this command, I end up in "inactive" operation mode.
Make sure to list all known 'good' devices (don't list the really broken device). --run --force should make it come up.
I recently (see previous thread) had an issue aswel and I found the order of commands mattered. I may have put the wrong ones up here. Doing history | grep mdadm the last used command, and thus probably the right one was:
mdadm --assemble --run --force /dev/md0 /dev/sd[1-7].
Make sure to mdadm --stop /dev/md0 before trying to assemble it.
Quoted 14 lines
Quoted 3 lines
Now the broken drive. Check your cables!! and run smartctl on it to give smart a chance to 'fix' the drive somewhat and check its status/health. ... If it fails again (at 80% because of hardware failure) you can't re-use the broken disk. It really is broken :p
It failed twice during resync, at around the same point, and smartctl tells me it's broken, so I assume it's gone for good. (Also, the failed drive is also marked as "spare" currently.)
Quoted 1 line
your very last hope, is to not use the broken drive, and 'force' the above using the earlier marked spare.
How? I haven't managed to do that, that's my whole question.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Oliver Schinagl wrote, On 14.04.2013 19:30:
Quoted 2 lines
mdadm --assemble --run --force
/dev/md0/dev/sd[1-7]. Make sure to mdadm --stop/dev/md0before trying to assemble it.
# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq]
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: Not enough devices to start the array.
# cat /proc/mdstat
md0 : inactive sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3]
5860574976 blocks
(Note that sdl is not even listed)
# mdadm --re-add /dev/md0 /dev/sdl
mdadm: re-added /dev/sdl
# cat /proc/mdstat
md0 : inactive sdl[8](S) sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3]
6837337472 blocksNow, sdl is listed, but as spare. I need it to be treated not as spare, but as good drive with correct data (well, almost, 2 events off only). How do I do that?
Quoted 14 lines · Ben Bucksch
Oliver Schinagl wrote, On 14.04.2013 19:30:
Quoted 2 lines
mdadm --assemble --run --force
/dev/md0/dev/sd[1-7]. Make sure to mdadm --stop/dev/md0before trying to assemble it.# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq] mdadm: failed to RUN_ARRAY /dev/md0: Input/output error mdadm: Not enough devices to start the array. # cat /proc/mdstat md0 : inactive sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 5860574976 blocks (Note that sdl is not even listed) # mdadm --re-add /dev/md0 /dev/sdl mdadm: re-added /dev/sdl
That can't work I don't think. You want to create a degraded raid5 array, e.g. 7 disks. It tried (and failed) to create a 6 disk array. Re-adding sdl will make it won't to resync. How you can force that however I don't know. I hoped for you that the above command would actually do that.
Quoted 11 lines
# cat /proc/mdstat md0 : inactive sdl[8](S) sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 6837337472 blocksNow, sdl is listed, but as spare. I need it to be treated not as spare, but as good drive with correct data (well, almost, 2 events off only). How do I do that?
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
To re-summarize (for full info, see first post of thread):
* There are 2 RAID5 arrays in the machine, each have 8 disks.
* I upgraded Ubuntu 10.04 to 12.04.
* After reboot, both arrays had each ejected one disk.
The ejected disks are working fine (at least now).
* During the resync mandated by above ejection,
one other drive failed, this one fatally with a real hardware failure.
* The second array resynced fine, further proving that the
disks ejected during upgrade were working.
* Now I am left with: originally 8-disk RAID5, 6 disks are healthy, 1 disk with hardware failure, and 1 disk that was ejected, but is
working.
* The latter is currently marked "spare" by md and has an event count (only) 2 events lower than the other 6 disks.
* My task is to get the latter disk back online *with* its data, without resync.
I desperately need help, please.
Based on suggestions here by Oliver and on forums, I did (and the result is):
Quoted 17 lines
# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq] mdadm: failed to RUN_ARRAY /dev/md0: Input/output error mdadm: Not enough devices to start the array. # cat /proc/mdstat md0 : inactive sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 5860574976 blocks (Note that sdl is not even listed) # mdadm --re-add /dev/md0 /dev/sdl mdadm: re-added /dev/sdl # cat /proc/mdstat md0 : inactive sdl[8](S) sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 6837337472 blocksNow, sdl is listed, but as spare. I need it to be treated not as spare, but as good drive with correct data (well, almost, 2 events off only). How do I do that?
Quoted 17 lines · Ben Bucksch
To re-summarize (for full info, see first post of thread):
* There are 2 RAID5 arrays in the machine, each have 8 disks.
* I upgraded Ubuntu 10.04 to 12.04.
* After reboot, both arrays had each ejected one disk.
The ejected disks are working fine (at least now).
* During the resync mandated by above ejection,
one other drive failed, this one fatally with a real hardware failure.
* The second array resynced fine, further proving that the disks ejected during upgrade were working.
* Now I am left with: originally 8-disk RAID5, 6 disks are healthy, 1 disk with hardware failure, and 1 disk that was ejected, but is
working.
* The latter is currently marked "spare" by md and has an event count (only) 2 events lower than the other 6 disks.
* My task is to get the latter disk back online *with* its data, without resync.
Quoted 10 lines
I desperately need help, please.
Based on suggestions here by Oliver and on forums, I did (and the result is):
Quoted 5 lines
# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq] mdadm: failed to RUN_ARRAY /dev/md0: mdadm: Not enough devices to start the array.
At this point, does dmesg show anything pointing to that input/output error ? The procedure is correct, I've used that myself on several occasions when confronted with a two-disk failure.
What you ought to do before anything, is ascertain all those seven drives are fully readable without any problems occurring. I use dd_rescue for that; dd_rescue /dev/sd{x} /dev/null. You can run parallel dd_rescue sessions. When finished, verify that all dd_rescue sessions reported zero errors. If not, clone that drive using dd_rescue to a fresh new drive, as retry assembly with that new one instead.
I have no (further) idea why mdadm insists there are not enough devices, but I'm willing to bet it is that input/output error that is at the root of that. So do the dd_rescue procedure as described.
Oh and, make sure you get that drive out of the array as spare, as that is definitely NOT what you want. When in doubt how to do that safely, clone it first, attempt removal later. If you value your data.
Good luck!
Maarten
Quoted 19 lines
Quoted 14 lines
# cat /proc/mdstat md0 : inactive sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 5860574976 blocks (Note that sdl is not even listed) # mdadm --re-add /dev/md0 /dev/sdl mdadm: re-added /dev/sdl # cat /proc/mdstat md0 : inactive sdl[8](S) sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 6837337472 blocksNow, sdl is listed, but as spare. I need it to be treated not as spare, but as good drive with correct data (well, almost, 2 events off only). How do I do that?
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Maarten wrote, On 18.04.2013 15:58:
Quoted 23 lines · Ben Bucksch
Quoted 22 lines
To re-summarize (for full info, see first post of thread):
* There are 2 RAID5 arrays in the machine, each have 8 disks.
* I upgraded Ubuntu 10.04 to 12.04.
* After reboot, both arrays had each ejected one disk.
The ejected disks are working fine (at least now).
* During the resync mandated by above ejection,
one other drive failed, this one fatally with a real hardware failure.
* The second array resynced fine, further proving that the disks ejected during upgrade were working.
* Now I am left with: originally 8-disk RAID5, 6 disks are healthy, 1 disk with hardware failure, and 1 disk that was ejected, but is
working.
* The latter is currently marked "spare" by md and has an event count (only) 2 events lower than the other 6 disks.
* My task is to get the latter disk back online *with* its data, without resync.I desperately need help, please.
Based on suggestions here by Oliver and on forums, I did (and the result is):
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq] mdadm: failed to RUN_ARRAY /dev/md0: mdadm: Not enough devices to start the array.At this point, does dmesg show anything pointing to that input/output error ? The procedure is correct
[630786.513314] md: md0 stopped. [630786.513341] md: unbind<sdl> [630786.590662] md: export_rdev(sdl) [630786.590744] md: unbind<sdj> [630786.670652] md: export_rdev(sdj) [630786.670887] md: unbind<sdq> [630786.750650] md: export_rdev(sdq) [630786.750707] md: unbind<sdn> [630786.830649] md: export_rdev(sdn) [630786.830712] md: unbind<sdp> [630786.910651] md: export_rdev(sdp) [630786.910710] md: unbind<sdo> [630786.990649] md: export_rdev(sdo) [630786.990700] md: unbind<sdm> [630787.070649] md: export_rdev(sdm) [630793.315121] md: md0 stopped. [630794.785328] md: bind<sdm> [630794.785512] md: bind<sdo> [630794.785695] md: bind<sdp> [630794.785891] md: bind<sdn> [630794.786643] md: bind<sdq> [630794.787009] md: bind<sdl> [630794.788164] md: bind<sdj> [630794.788236] md: kicking non-fresh sdl from array! [630794.788250] md: unbind<sdl> [630794.810082] md: export_rdev(sdl) [630794.812725] raid5: device sdj operational as raid disk 0 [630794.812734] raid5: device sdq operational as raid disk 7 [630794.812740] raid5: device sdn operational as raid disk 6 [630794.812745] raid5: device sdp operational as raid disk 5 [630794.812750] raid5: device sdo operational as raid disk 4 [630794.812755] raid5: device sdm operational as raid disk 3 [630794.813895] raid5: allocated 8490kB for md0 [630794.813966] 0: w=1 pa=0 pr=8 m=1 a=2 r=8 op1=0 op2=0 [630794.813974] 7: w=2 pa=0 pr=8 m=1 a=2 r=8 op1=0 op2=0 [630794.813980] 6: w=3 pa=0 pr=8 m=1 a=2 r=8 op1=0 op2=0 [630794.813986] 5: w=4 pa=0 pr=8 m=1 a=2 r=8 op1=0 op2=0 [630794.813993] 4: w=5 pa=0 pr=8 m=1 a=2 r=8 op1=0 op2=0 [630794.813999] 3: w=6 pa=0 pr=8 m=1 a=2 r=8 op1=0 op2=0 [630794.814005] raid5: not enough operational devices for md0 (2/8 failed) [630794.820671] RAID5 conf printout: [630794.820675] --- rd:8 wd:6 [630794.820680] disk 0, o:1, dev:sdj [630794.820685] disk 3, o:1, dev:sdm [630794.820689] disk 4, o:1, dev:sdo [630794.820693] disk 5, o:1, dev:sdp [630794.820697] disk 6, o:1, dev:sdn [630794.820701] disk 7, o:1, dev:sdq [630794.820945] raid5: failed to run raid set md0 [630794.826530] md: pers->run() failed ... [630794.834455] md: export_rdev(sdl) [630794.834463] md: export_rdev(sdl)
The problem is:
md: kicking non-fresh sdl from array!
thus:
raid5: not enough operational devices for md0 (2/8 failed)
# mdadm -E /dev/sdl Checksum : ca6e81a9 - correct Events : 13274863 # mdadm -E /dev/sdn Checksum : c9a41046 - correct Events : 13274865
So, the question is: How do I convince md not to be so anal retentive and prevent me from accessing any of my data? The drive ***is fine***, has practically all the data (I don't care about these 2 events), just use it already. Nobody seems to know the magic shell commands to do that.
The lack of a proper shell command for that effectively constitutes a dataloss bug. I've been patient, but I'm getting more and more upset at md. Thanks, Maarten, for your help. I hope 1) you or anybody else can help me, and I hope 2) these kinds of problems will be fixed once and for good by the devs.
Quoted 1 line
Good luck!
Thanks.
Ben
[email protected] wrote, On 20.04.2013 00:56:
Quoted 33 lines
Maarten wrote, On 18.04.2013 15:58:
Quoted 24 lines · Ben Bucksch
Quoted 23 lines
To re-summarize (for full info, see first post of thread):
* There are 2 RAID5 arrays in the machine, each have 8 disks.
* I upgraded Ubuntu 10.04 to 12.04.
* After reboot, both arrays had each ejected one disk.
The ejected disks are working fine (at least now).
* During the resync mandated by above ejection,
one other drive failed, this one fatally with a real hardware
failure.
* The second array resynced fine, further proving that the disks ejected during upgrade were working.
* Now I am left with: originally 8-disk RAID5, 6 disks are healthy, 1 disk with hardware failure, and 1 disk that was ejected, but is
working.
* The latter is currently marked "spare" by md and has an event count (only) 2 events lower than the other 6 disks.
* My task is to get the latter disk back online *with* its data, without resync.I desperately need help, please.
Based on suggestions here by Oliver and on forums, I did (and the result is):
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq] mdadm: failed to RUN_ARRAY /dev/md0: mdadm: Not enough devices to start the array.At this point, does dmesg show anything pointing to that input/output error ? The procedure is correct
Quoted 7 lines
[dmesg]
The problem is:
md: kicking non-fresh sdl from array!
thus:
raid5: not enough operational devices for md0 (2/8 failed)So, the question is: How do I convince md not to be so anal retentive and prevent me from accessing any of my data? The drive ***is fine***, has practically all the data (I don't care about these 2 events), just use it already. Nobody seems to know the magic shell commands to do that.
Good news:
In my desperation, I now ran the following dangerous command:
mdadm --create /dev/md0 --assume-clean --level=raid5 -n 8 --chunk=64 --layout=left-symmetric --metadata=0.90 /dev/sdj missing /dev/sdl /dev/sd[mopnq] and that worked. I can read my files again, without problem, all is happy.
Before doing that, I saved the superblock, using (no warranty!):
1. mdadm -E /dev/sdj
2. "Used Dev Size" (in KB) * 1024 / 64 - 1 (use this as <skip blocks>)
3. dd if=/dev/sdl of=/root/sdj.mdsuperblock ibs=64 skip=<skip blocks>
---
Thanks, Maarten and Oliver, for your help and moral support.
---
I still maintain that all of this represents 2 design bugs in the md implementation:
1. ejecting devices out that are working
1.1. individual sectors not readable/writable, but rest of device working
(This is very common these days with large drives)
1.2. temporary errors, e.g. disk not connected, loose cable, bad controller etc. 1.3. Linux distro upgrade, no disk problem at all (my case)
2. not allowing me to re-add ejected disks, with data, without resync
The result of this is:
1. a device is ejected for no good reasons
2. a resync is triggered
3. the resync discovers a disk that is *really* broken
I am left with 2 disks marked "failed", but only 1 actually failed, so normally I should be able to recover, yet I cannot read anything. This fails the very definition of RAID5, therefore is a bug. I have to do risky operations like re-create that can easily destroy all data. Effectively, md achieves the opposite that is intended: It actively risks and destroys my data.
I am BEGGING you md raid devs to fix these.
Ben Bucksch
Ben Bucksch wrote, On 20.04.2013 03:26:
Quoted 1 line
I can read my files again, without problem, all is happy.
Actually, no. XFS filesystem structure is not sane. I must have done something wrong. (If possible, please let me know what, all data should be posted.)
At first, it looked OK, as if only one recently written directory was broken. I unmounted one of the FS, did xfs_repair, and after re-mounting, almost all directories are gone. Almost 100% dataloss. I can't describe how upset I am against md.
Oh, and in case you're wondering about my backup: That's gone, too, due to bugs in btrfs that trashed the FS and also stopped the dedicated backup machine from booting automatically, so I don't have any current backup either.
Ben
Quoted 11 lines · Ben Bucksch
Ben Bucksch wrote, On 20.04.2013 03:26:
Quoted 1 line
I can read my files again, without problem, all is happy.
Actually, no. XFS filesystem structure is not sane. I must have done something wrong. (If possible, please let me know what, all data should be posted.)
At first, it looked OK, as if only one recently written directory was broken. I unmounted one of the FS, did
xfs_repair, and after re-mounting, almost all directories are gone. Almost 100% dataloss. I can't describe how upset I am against md.
As others have already told you, md does not go randomly kicking drives from arrays. Your system had a failure of some kind which caused the loss of two drives. You tried to recover it and managed to get a drive into the spare state. After much troubleshooting, you used the canon of last resort "assume-clean" after which (without properly verifying your drives were in the correct order) you ran a terribly destructive write to the disks and have almost certainly ruined any chance you had at recovering your data.
I fail to see where the fault lies with md.
Had you searched or asked a little more, you would have found a number of people who have written permutation scripts which would have iterated every possible arrangement of drives to allow you to run a read-only fsck on each one, which would have positively identified the correct order of your disks.
Your best bet now is to post on the xfs list to find out if there is _any_ way of undoing what you just did, or working around it (backup superblocks or whatever) and then running a permutation on your drives to see if any combination shows you any valid data.
Brad Campbell wrote, On 21.04.2013 09:23:
Quoted 1 line
As others have already told you, md does not go randomly kicking drives from arrays. Your system had a failure of some kind which caused the loss of two drives.
You ignore the facts and do "mi mi mi" in face of bugs reports. 2 different arrays lost 1 drive, both at the same time at reboot after the OS upgrade, and both drives are working fine. Facts.
And even *if* they had a temporary error, my case shows why it's a *bug* to kick them out of the array. And it's a *bug* to not let me put them back in with data. Tons of other people have suffered dataloss because of various temporary, easily recoverable problems and these 2 bugs.
People like you are the reason why people like me suffer dataloss.
Quoted 8 lines · Ben Bucksch
Brad Campbell wrote, On 21.04.2013 09:23:
Quoted 3 lines
As others have already told you, md does not go randomly kicking drives from arrays. Your system had a failure of some kind which caused the loss of two drives.
You ignore the facts and do "mi mi mi" in face of bugs reports. 2 different arrays lost 1 drive, both at the same time at reboot after the OS upgrade, and both drives are working fine. Facts.
Those are not facts, they are uninformed guesses at what happened. You have no facts other than something bad happened and two drives were ejected from the array. If you had actual facts then we'd have been able to assist you in determining what actually happened and how it might have been rectified.
Quoted 4 lines
And even *if* they had a temporary error, my case shows why it's a *bug* to kick them out of the array. And it's a *bug* to not let me put them back in with data. Tons of other people have suffered dataloss because of various temporary, easily recoverable problems and these 2 bugs.
It's not a bug. It is working as intended. That it is not working the way _you_ would like it to work is not a bug at all.
When you have an array, you don't get "temporary errors". It's either good or its not. An error is an error is an error. You had an error, which means something in your storage stack is broken. That you can't figure out what it is is even more insidious and needs to be fixed before you can continue.
May I point you at the source of both the kernel and md and suggest if you'd like it to "work" differently you might attempt to make it do so.
Question. Have you ever worked with hardware arrays? What do you think would happen in the same set of circumstances with a hardware array (hint, precisely the same thing). The bonus with md is (if you know what you are doing and with the right assistance) you can do things like --create --assume-clean and get access to your data. You can't do that with any hardware array I've ever used.
Quoted 1 line
People like you are the reason why people like me suffer dataloss.
Riiiight.
Remember, and I quote "I have to do risky operations like re-create that can easily destroy all data. Effectively, md achieves the opposite that is intended: It actively risks and destroys my data."
So you knew the operation was risky, yet you went ahead without enough information to do it safely and blitzed all your data. I'm sorry, but that's not my fault.
Again : "Good news: In my desperation, I now ran the following dangerous command: mdadm --create /dev/md0 --assume-clean --level=raid5 -n 8 --chunk=64 --layout=left-symmetric --metadata=0.90 /dev/sdj missing /dev/sdl /dev/sd[mopnq]"
How did you verify you had your disks in the correct order? Where did that command line come from?
This will be my last post on the subject. I pointed you at a path of action in my last post.
Hi Brad,
I'm sorry you've been sucked into exchanges with this troll. (I spotted the attitude in the original post.) A couple comments below:
Quoted 4 lines · Brad Campbell
Again : "Good news: In my desperation, I now ran the following dangerous
command: mdadm --create/dev/md0--assume-clean --level=raid5 -n 8 --chunk=64 --layout=left-symmetric --metadata=0.90/dev/sdjmissing/dev/sdl/dev/sd[mopnq]"
I did find it interesting that Ben tried this on his own, given that's the very advice he demanded not be given in his OP.
Quoted 2 lines
How did you verify you had your disks in the correct order? Where did that command line come from?
Ben's shell skills seem to match his interpersonal communication skills (weak). The construct /dev/sd[mopnq] expands as if it was specified /dev/sd[mnopq]. His misunderstanding of bracket syntax has wrecked his array. If he had used braces, or spelled out all the devices, he'd probably be fine right now.
If he tries again, with this in mind, he might still be fine. (I haven't checked if his attempted device order was correct, though.) If I was inclined to respond directly to him, I'd further suggest he review the list archives for "error recovery", "timeouts", and "scrubbing". He might learn enough to not suffer so much next time.
Quoted 2 lines
This will be my last post on the subject. I pointed you at a path of action in my last post.
If Ben's attitude had moderated in following posts, I might have set aside my first impression and cut him some slack. That's a significant consideration when non-native English is involved. But it's clearly not the case here.
My first *and* last post on this topic.
Phil
Phil Turmel wrote, On 21.04.2013 20:17:
Quoted 2 lines
I'm sorry you've been sucked into exchanges with this troll.
[referring to me]
Thank you. I came here for help. I *desperately* needed help with an actual, real problem. My goal was to rescue my data, and make sure this never happens again to anybody else. The goal of a troll is to enrage people and cause useless debate.
Before I came here, I had already extensively searched on Google, and found lots of unhelpful comments. I wanted to avoid these. Unfortunately, I found the same kind of responses here. Responses like "You should have used ABC" are not helpful to the problem at hand at all, and in fact only serve to enrage the person so "advised". Such advise is fair enough when offered *after* actual practical help.
A few people had tried to help, but could not, because apparently there is no safe command to do what I need.
Quoted 6 lines
Quoted 4 lines
Again : "Good news: In my desperation, I now ran the following dangerous
command: mdadm --create/dev/md0--assume-clean --level=raid5 -n 8 --chunk=64 --layout=left-symmetric --metadata=0.90/dev/sdjmissing/dev/sdl/dev/sd[mopnq]"I did find it interesting that Ben tried this on his own, given that's the very advice he demanded not be given in his OP.
That's not correct. I wrote:
"Please do NOT respond with re-"create" the array - unless you can give me the exact --create command that would recover it with data - other people tried this based on suggestions in forums and they lost all data"
Note the "unless". Unfortunately, nonewithstanding my disclaimer, several people suggested 1) to use the disk that I already wrote is (really, actually) dead 2) hexediting superblocks (without any info on how) 3) using --create (ditto), but *nobody* offered the exact command to run. (I had posted all relevant device info for that purpose.)
I didn't feel comfortable with --create, but when nobody offered a real alternative, I saw no other option than to try that. Yet, I was wrong to do that, because:
Quoted 6 lines
The construct /dev/sd[mopnq] expands as if it was specified /dev/sd[mnopq]. His misunderstanding of bracket syntax has wrecked his array. If he had used braces, or spelled out all the devices, he'd probably be fine right now.
If he tries again, with this in mind, he might still be fine.
Ah, thanks. This is what I consider practical help. Thank you. Indeed, I had no idea (not even thought of the mere possibility) that the shell would reorder my [] device list. And in fact:
# cat /proc/mdstat md0 : active raid5 sdq[7] sdp[6] sdo[5] sdn[4] sdm[3] sdl[2] sdj[0]
So, the [] was indeed what killed me and caused my dataloss. Unfortunately, the most important FS on the array is now totally corrupted.
FWIW, this is exactly why I had asked for a concrete --create command for my case.
Quoted 1 line
That's a significant consideration when non-native English is involved. But it's clearly not the case here.
FWIW, I am not a native English speaker.
Ben
Quoted 10 lines
Brad Campbell wrote, On 21.04.2013 09:23:
Quoted 3 lines
As others have already told you, md does not go randomly kicking drives from arrays. Your system had a failure of some kind which caused the loss of two drives.
You ignore the facts and do "mi mi mi" in face of bugs reports. 2 different arrays lost 1 drive, both at the same time at reboot after the
OS upgrade, and both drives are working fine. Facts.
Quoted 7 lines
And even *if* they had a temporary error, my case shows why it's a *bug*
to kick them out of the array. And it's a *bug* to not let me put them back in with data. Tons of other people have suffered dataloss because of various temporary, easily recoverable problems and these 2 bugs.People like you are the reason why people like me suffer dataloss.
Well, just restore from backup. Shouldn't be too hard.
Vennlige hilsener / Best regards
roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 11 lines · On Sat, 20 Apr 2013 03:53:49 +0200 Ben Bucksch <[email protected]> wrote:
Ben Bucksch wrote, On 20.04.2013 03:26:
Quoted 1 line
I can read my files again, without problem, all is happy.
Actually, no. XFS filesystem structure is not sane. I must have done something wrong. (If possible, please let me know what, all data should be posted.)
At first, it looked OK, as if only one recently written directory was broken. I unmounted one of the FS, did
xfs_repair, and after re-mounting, almost all directories are gone. Almost 100% dataloss. I can't describe how upset I am against md.
So data was accessible before "xfs_repair", is not accessible after 'xfs_repair', yet you blame md rather than xfs_repair? Interesting.
You've clearly had a bad experience - I'm sorry about that.
I doubt there is anything you can do to unrepair whatever xfs_repair did, but the place to ask would be on the xfs list.
NeilBrown
Quoted 11 lines
Oh, and in case you're wondering about my backup: That's gone, too, due to bugs in btrfs that trashed the FS and also stopped the dedicated backup machine from booting automatically, so I don't have any current backup either.
Ben
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 61 lines · On Sat, 20 Apr 2013 03:26:43 +0200 Ben Bucksch <[email protected]> wrote:
[email protected] wrote, On 20.04.2013 00:56:
Quoted 41 lines
Maarten wrote, On 18.04.2013 15:58:
Quoted 26 lines · Ben Bucksch
Quoted 25 lines
To re-summarize (for full info, see first post of thread):
* There are 2 RAID5 arrays in the machine, each have 8 disks.
* I upgraded Ubuntu 10.04 to 12.04.
* After reboot, both arrays had each ejected one disk.
The ejected disks are working fine (at least now).
* During the resync mandated by above ejection,
one other drive failed, this one fatally with a real hardware
failure.
* The second array resynced fine, further proving that the disks ejected during upgrade were working.
* Now I am left with: originally 8-disk RAID5, 6 disks are healthy, 1 disk with hardware failure, and 1 disk that was ejected, but is
working.
* The latter is currently marked "spare" by md and has an event count (only) 2 events lower than the other 6 disks.
* My task is to get the latter disk back online *with* its data, without
resync.I desperately need help, please.
Based on suggestions here by Oliver and on forums, I did (and the result
is):Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq] mdadm: failed to RUN_ARRAY /dev/md0: mdadm: Not enough devices to start the array.At this point, does dmesg show anything pointing to that input/output error ? The procedure is correct
[dmesg]
The problem is:
md: kicking non-fresh sdl from array!
thus:
raid5: not enough operational devices for md0 (2/8 failed)Quoted 4 lines
So, the question is: How do I convince md not to be so anal retentive and prevent me from accessing any of my data? The drive ***is fine***, has practically all the data (I don't care about these 2 events), just use it already. Nobody seems to know the magic shell commands to do that.
Good news:
In my desperation, I now ran the following dangerous command:
mdadm --create/dev/md0--assume-clean --level=raid5 -n 8 --chunk=64 --layout=left-symmetric --metadata=0.90/dev/sdjmissing/dev/sdl/dev/sd[mopnq]
and that worked. I can read my files again, without problem, all is happy.Before doing that, I saved the superblock, using (no warranty!):
1. mdadm -E/dev/sdj
2. "Used Dev Size" (in KB) * 1024 / 64 - 1 (use this as <skip blocks>)
3. dd if=/dev/sdl of=/root/sdj.mdsuperblock ibs=64 skip=<skip blocks>---
Quoted 7 lines
Thanks, Maarten and Oliver, for your help and moral support.
---
I still maintain that all of this represents 2 design bugs in the md implementation:
1. ejecting devices out that are working
Without being able to examine the full sequence of events I cannot be sure what happened here, but my best guess is that the working device wasn't "ejected" so much as it simply wasn't included.
The modern approach to booting involves devices appearing asynchronously, with filesystems being mounted as the relevant devices appear. This is slightly awkward for md/raid. If you have a 5-disk RAID5 and only 4 disks have appeared, do you start the array degraded, or do you wait for the 5th disk to appear.
What if the 5th disk has been physically removed? That would mean waiting forever.
mdadm doesn't impose a policy but allows the boot scripts to choose one. Some boot scripts might get this wrong.
If you have a write-intent-bitmap on your array, then getting it wrong isn't too bad: when the 5th disk does appear it can easily be re-added. Without the bitmap, it cannot.
My guess is that you got bitten by something going wrong in the init scripts.
Quoted 2 lines
1.1. individual sectors not readable/writable, but rest of device working
(This is very common these days with large drives)
Yes, this is a problem. There is code to handle it better by recording bad
blocks. It isn't quite production read yet. And it'll never work on 0.90
metadata.
Quoted 3 lines
1.2. temporary errors, e.g. disk not connected, loose cable, bad controller etc.
1.3. Linux distro upgrade, no disk problem at all (my case)
unless there are bugs in the distro scripts.
Quoted 1 line
2. not allowing me to re-add ejected disks, with data, without resync
It *must* be hard to do this, because it *will* cause data loss. Maybe it shouldn't be quite as hard as it is. But then there are lots of improvements that could be made, but not very many developers working on it.
NeilBrown
Quoted 20 lines
The result of this is:
1. a device is ejected for no good reasons
2. a resync is triggered
3. the resync discovers a disk that is *really* brokenI am left with 2 disks marked "failed", but only 1 actually failed, so normally I should be able to recover, yet I cannot read anything. This fails the very definition of RAID5, therefore is a bug. I have to do risky operations like re-create that can easily destroy all data. Effectively, md achieves the opposite that is intended: It actively risks and destroys my data.
I am BEGGING you md raid devs to fix these.
Ben Bucksch
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
----- Opprinnelig melding -----
Quoted 2 lines
To re-summarize (for full info, see first post of thread):
* There are 2 RAID5 arrays in the machine, each have 8 disks.
Once more: 8 drives in RAID-5 isn't very safe. The chances are rather high to get a double disk failure with that amount of drives in R5.
See the list for examples.
Just my 2c
Signature · 15 lines
Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 98013356 [email protected] http://blogg.karlsbakk.net/ GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 17 lines · Ben Bucksch
To re-summarize (for full info, see first post of thread):
* There are 2 RAID5 arrays in the machine, each have 8 disks.
* I upgraded Ubuntu 10.04 to 12.04.
* After reboot, both arrays had each ejected one disk.
The ejected disks are working fine (at least now).
* During the resync mandated by above ejection,
one other drive failed, this one fatally with a real hardware failure.
* The second array resynced fine, further proving that the disks ejected during upgrade were working.
* Now I am left with: originally 8-disk RAID5, 6 disks are healthy, 1 disk with hardware failure, and 1 disk that was ejected, but is
working.
* The latter is currently marked "spare" by md and has an event count (only) 2 events lower than the other 6 disks.
* My task is to get the latter disk back online *with* its data, without resync.
Quoted 20 lines
I desperately need help, please.
Based on suggestions here by Oliver and on forums, I did (and the result is):
Quoted 15 lines
# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq] mdadm: failed to RUN_ARRAY /dev/md0: Input/output error mdadm: Not enough devices to start the array. # cat /proc/mdstat md0 : inactive sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 5860574976 blocks (Note that sdl is not even listed) # mdadm --re-add /dev/md0 /dev/sdl mdadm: re-added /dev/sdl # cat /proc/mdstat md0 : inactive sdl[8](S) sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 6837337472 blocks
That won't work here as sdl was being rebuilt at the time of the failure. md therefore 'knows' that it doesn't have the correct data on and can't be used to assemble the array (I think the array position of the disk is only written to the metadata when recovery completes).
Quoted 4 lines
Quoted 4 lines
Now, sdl is listed, but as spare. I need it to be treated not as spare, but as good drive with correct data (well, almost, 2 events off only). How do I do that?
I can see two options here:
- Image the known faulty disk to a new one and then use that to force assemble the array (with the possibility of some data loss, depending on how much can be read from the faulty disk).
- Modify the metadata on sdl so that it shows as being a valid member of the array. This will require either manual editing of the superblock, or rerunning the "mdadm --create" command with the correct mdadm version (so data offsets match), disk order and parameters (chunk size, etc). If done correctly then there should be no data loss (providing that sdl, when re-added to the array, used the same data offset as it originally had), but get anything wrong any you'll be even further up the creek.
Personally I'd look into option one first. Despite the probability of some data loss, I think it's a lower risk option overall.
Cheers,
Robin
Signature · 4 lines
___
( ' } | Robin Hill <[email protected]> |
/ / ) | Little Jim says .... |
// !! | "He fallen in de water !!" |Quoted 48 lines · Robin Hill
Quoted 17 lines · Ben Bucksch
To re-summarize (for full info, see first post of thread):
* There are 2 RAID5 arrays in the machine, each have 8 disks.
* I upgraded Ubuntu 10.04 to 12.04.
* After reboot, both arrays had each ejected one disk.
The ejected disks are working fine (at least now).
* During the resync mandated by above ejection,
one other drive failed, this one fatally with a real hardware failure.
* The second array resynced fine, further proving that the disks ejected during upgrade were working.
* Now I am left with: originally 8-disk RAID5, 6 disks are healthy, 1 disk with hardware failure, and 1 disk that was ejected, but is
working.
* The latter is currently marked "spare" by md and has an event count (only) 2 events lower than the other 6 disks.
* My task is to get the latter disk back online *with* its data, without resync.Quoted 20 lines
I desperately need help, please.
Based on suggestions here by Oliver and on forums, I did (and the result is):
Quoted 15 lines
# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble --run --force /dev/md0 /dev/sd[jlmnopq] mdadm: failed to RUN_ARRAY /dev/md0: Input/output error mdadm: Not enough devices to start the array. # cat /proc/mdstat md0 : inactive sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 5860574976 blocks (Note that sdl is not even listed) # mdadm --re-add /dev/md0 /dev/sdl mdadm: re-added /dev/sdl # cat /proc/mdstat md0 : inactive sdl[8](S) sdj[0] sdq[7] sdn[6] sdp[5] sdo[4] sdm[3] 6837337472 blocksThat won't work here as sdl was being rebuilt at the time of the failure. md therefore 'knows' that it doesn't have the correct data on and can't be used to assemble the array (I think the array position of the disk is only written to the metadata when recovery completes).
Quoted 4 lines
Quoted 4 lines
Now, sdl is listed, but as spare. I need it to be treated not as spare, but as good drive with correct data (well, almost, 2 events off only). How do I do that?
Quoted 5 lines
I can see two options here:
- Image the known faulty disk to a new one and then use that to force assemble the array (with the possibility of some data loss, depending on how much can be read from the faulty disk).
IF you have an extra spare disk, this probably is the best idea then. The event count might not even be off at all.
Use ddrescue though, check the options and possibilities, going start -> end and end -> start Let is try very often. You get the idea.
You could use the spare disk for this, but that removes any option in trying to recover using that. (Hex editing the superblock, possible but not easy). IF you have enough room on the other array, you can always dd if=/dev/spare of=file_on_array
Also, make a backup of your superblock!! dd if=/dev/sd* of=sd*.superblock size=<superblocksize> count=1
Quoted 11 lines
- Modify the metadata on sdl so that it shows as being a valid member of the array. This will require either manual editing of the superblock, or rerunning the "mdadm --create" command with the correct mdadm version (so data offsets match), disk order and parameters (chunk size, etc). If done correctly then there should be no data loss (providing that sdl, when re-added to the array, used the same data offset as it originally had), but get anything wrong any you'll be even further up the creek.
Personally I'd look into option one first. Despite the probability of some data loss, I think it's a lower risk option overall.
I agree, best option would be that
oliver
Quoted 4 lines
Cheers,
Robin