Spare disk not becoming active
Hallo,
I'm having some trouble with an array I have that has become degraded.
I have an array with this array state:
md101 : active raid1 sdf1[0] sdb1[2](S)
1953511936 blocks [2/1] [U_]
mdadm --detail says:
/dev/md101:
Version : 0.90
Creation Time : Thu Jan 13 14:34:27 2011
Raid Level : raid1
Array Size : 1953511936 (1863.01 GiB 2000.40 GB)
Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 101
Persistence : Superblock is persistent
Update Time : Fri Nov 23 03:23:04 2012
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1
UUID : 43e92a79:90295495:0a76e71e:56c99031 (local to host
barney)
Events : 0.2127
Number Major Minor RaidDevice State
0 8 81 0 active sync /dev/sdf1
1 0 0 1 removed
2 8 17 - spare /dev/sdb1
If I attempt to force the spare to become active it begins to recover:
$ sudo mdadm -S /dev/md101
mdadm: stopped /dev/md101
$ sudo mdadm --assemble --force --no-degraded /dev/md101 /dev/sdf1 /dev/sdb1
mdadm: /dev/md101 has been started with 1 drive (out of 2) and 1 spare.
$ cat /proc/mdstat
md101 : active raid1 sdf1[0] sdb1[2]
1953511936 blocks [2/1] [U_]
[>....................] recovery = 0.0% (541440/1953511936)
finish=420.8min speed=77348K/secThis runs for the allotted time but returns to the state of spare.
Neither disk partition report errors:
$ cat /sys/block/md101/md/dev-sdf1/errors 0 $ cat /sys/block/md101/md/dev-sdb1/errors 0
Are there mdadm logs to find out why this is not recovering properly? How otherwise do I debug this?
Cheers,
Tudor.
I don't mean to be rude, but it's been two weeks and my system is still in this state. Bump, anyone?
A thorough search of the web (before I originally posted this to the list) revealed nothing. No explanation as to why this occurs seemed apparent, only that it's happened a number of times. Most reports indicate that a complete stop of the array and reassemble fixes it, but I tried that and it still returned to spare. Some reports indicated my position but no response that seems complete.
Eventually the discussions runs to wiping the disks and starting again. That seems a bit drastic and I'm concerned that *one* of the disks is faulty but not being reported as such, and I don't want to pick the wrong one to wipe off the superblock. mdadm reports no errors, but SMART indicates there may be a problem with the *active* disk, which is even more worrying because without making the spare active I can't remove it to test it properly.
Any ideas?
Cheers,
Tudor.
Quoted 69 lines · Tudor Holton
Hallo,
I'm having some trouble with an array I have that has become degraded.
I have an array with this array state:
md101 : active raid1 sdf1[0] sdb1[2](S)
1953511936 blocks [2/1] [U_]mdadm --detail says:
/dev/md101:Version : 0.90Creation Time : Thu Jan 13 14:34:27 2011
Raid Level : raid1
Array Size : 1953511936 (1863.01 GiB 2000.40 GB)
Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 101
Persistence : Superblock is persistentUpdate Time : Fri Nov 23 03:23:04 2012
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1UUID : 43e92a79:90295495:0a76e71e:56c99031 (local to hostbarney)
Events : 0.2127Number Major Minor RaidDevice State
0 8 81 0 active sync/dev/sdf1
1 0 0 1 removed2 8 17 - spare
/dev/sdb1If I attempt to force the spare to become active it begins to recover:
$ sudo mdadm -S /dev/md101 mdadm: stopped /dev/md101 $ sudo mdadm --assemble --force --no-degraded /dev/md101 /dev/sdf1 /dev/sdb1 mdadm: /dev/md101 has been started with 1 drive (out of 2) and 1 spare. $ cat /proc/mdstat md101 : active raid1 sdf1[0] sdb1[2] 1953511936 blocks [2/1] [U_] [>....................] recovery = 0.0% (541440/1953511936) finish=420.8min speed=77348K/secThis runs for the allotted time but returns to the state of spare.
Neither disk partition report errors:
$ cat /sys/block/md101/md/dev-sdf1/errors 0 $ cat /sys/block/md101/md/dev-sdb1/errors 0Are there mdadm logs to find out why this is not recovering properly? How otherwise do I debug this?
Cheers,
Tudor.Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 59 lines · Tudor Holton
Hallo,
I'm having some trouble with an array I have that has become degraded.
I have an array with this array state:
md101 : active raid1 sdf1[0] sdb1[2](S)
1953511936 blocks [2/1] [U_]mdadm --detail says:
/dev/md101:Version : 0.90Creation Time : Thu Jan 13 14:34:27 2011
Raid Level : raid1
Array Size : 1953511936 (1863.01 GiB 2000.40 GB)
Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 101
Persistence : Superblock is persistentUpdate Time : Fri Nov 23 03:23:04 2012
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1UUID : 43e92a79:90295495:0a76e71e:56c99031 (local to host barney)
Events : 0.2127Number Major Minor RaidDevice State
0 8 81 0 active sync/dev/sdf1
1 0 0 1 removed2 8 17 - spare
/dev/sdb1If I attempt to force the spare to become active it begins to recover:
$ sudo mdadm -S /dev/md101 mdadm: stopped /dev/md101 $ sudo mdadm --assemble --force --no-degraded /dev/md101 /dev/sdf1 /dev/sdb1 mdadm: /dev/md101 has been started with 1 drive (out of 2) and 1 spare. $ cat /proc/mdstat md101 : active raid1 sdf1[0] sdb1[2] 1953511936 blocks [2/1] [U_] [>....................] recovery = 0.0% (541440/1953511936) finish=420.8min speed=77348K/secThis runs for the allotted time but returns to the state of spare.
Neither disk partition report errors:
$ cat /sys/block/md101/md/dev-sdf1/errors 0 $ cat /sys/block/md101/md/dev-sdb1/errors 0
Quoted 5 lines
Are there mdadm logs to find out why this is not recovering properly? How otherwise do I debug this?
Cheers,
Tudor.
Did you look in the various /var/log/messages (current and previous ones) to see what it indicated happened the about the time it completed?
There is almost certainly something in there indicating what went wrong.
Quoted 69 lines · Roger Heflin
Quoted 59 lines · Tudor Holton
Hallo,
I'm having some trouble with an array I have that has become degraded.
I have an array with this array state:
md101 : active raid1 sdf1[0] sdb1[2](S)
1953511936 blocks [2/1] [U_]mdadm --detail says:
/dev/md101:Version : 0.90Creation Time : Thu Jan 13 14:34:27 2011
Raid Level : raid1
Array Size : 1953511936 (1863.01 GiB 2000.40 GB)
Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 101
Persistence : Superblock is persistentUpdate Time : Fri Nov 23 03:23:04 2012
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1UUID : 43e92a79:90295495:0a76e71e:56c99031 (local to host barney)
Events : 0.2127Number Major Minor RaidDevice State
0 8 81 0 active sync/dev/sdf1
1 0 0 1 removed2 8 17 - spare
/dev/sdb1If I attempt to force the spare to become active it begins to recover:
$ sudo mdadm -S /dev/md101 mdadm: stopped /dev/md101 $ sudo mdadm --assemble --force --no-degraded /dev/md101 /dev/sdf1 /dev/sdb1 mdadm: /dev/md101 has been started with 1 drive (out of 2) and 1 spare. $ cat /proc/mdstat md101 : active raid1 sdf1[0] sdb1[2] 1953511936 blocks [2/1] [U_] [>....................] recovery = 0.0% (541440/1953511936) finish=420.8min speed=77348K/secThis runs for the allotted time but returns to the state of spare.
Neither disk partition report errors:
$ cat /sys/block/md101/md/dev-sdf1/errors 0 $ cat /sys/block/md101/md/dev-sdb1/errors 0Quoted 5 lines
Are there mdadm logs to find out why this is not recovering properly? How otherwise do I debug this?
Cheers,
Tudor.Did you look in the various
/var/log/messages(current and previous ones) to see what it indicated happened the about the time it completed?
Quoted 5 lines
There is almost certainly something in there indicating what went wrong. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Thanks. I watched the logs messages during the recovery. During the last 0.1% (at 99.9%) messages like this appeared:
Dec 24 18:20:32 barney kernel: [2796835.703313] sd 2:0:0:0: [sdf] Unhandled sense code Dec 24 18:20:32 barney kernel: [2796835.703316] sd 2:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 24 18:20:32 barney kernel: [2796835.703320] sd 2:0:0:0: [sdf] Sense Key : Medium Error [current] [descriptor] Dec 24 18:20:32 barney kernel: [2796835.703325] Descriptor sense data with sense descriptors (in hex): Dec 24 18:20:32 barney kernel: [2796835.703327] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 Dec 24 18:20:32 barney kernel: [2796835.703335] e8 e0 5f 86 Dec 24 18:20:32 barney kernel: [2796835.703339] sd 2:0:0:0: [sdf] Add. Sense: Unrecovered read error - auto reallocate failed Dec 24 18:20:32 barney kernel: [2796835.703345] sd 2:0:0:0: [sdf] CDB: Read(10): 28 00 e8 e0 5f 7f 00 00 08 00 Dec 24 18:20:32 barney kernel: [2796835.703353] end_request: I/O error, dev sdf, sector 3907018630 Dec 24 18:20:32 barney kernel: [2796835.703366] ata3: EH complete Dec 24 18:20:32 barney kernel: [2796835.703383] md/raid1:md101: sdf: unrecoverable I/O read error for block 3907018496
Unfortunately, sdf is the active disk in this case. So I guess my only option left is to create a new array and copy as much over as it will let me?
Cheers,
Tudor.
Quoted 104 lines · Tudor Holton
Quoted 76 lines · Roger Heflin
Quoted 62 lines · Tudor Holton
Hallo,
I'm having some trouble with an array I have that has become degraded.
I have an array with this array state:
md101 : active raid1 sdf1[0] sdb1[2](S)
1953511936 blocks [2/1] [U_]mdadm --detail says:
/dev/md101:Version : 0.90Creation Time : Thu Jan 13 14:34:27 2011
Raid Level : raid1
Array Size : 1953511936 (1863.01 GiB 2000.40 GB)
Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 101
Persistence : Superblock is persistentUpdate Time : Fri Nov 23 03:23:04 2012
State : clean, degraded
Active Devices : 1
Working Devices : 2
Failed Devices : 0
Spare Devices : 1UUID : 43e92a79:90295495:0a76e71e:56c99031 (local to hostbarney)
Events : 0.2127Number Major Minor RaidDevice State
0 8 81 0 active sync/dev/sdf1
1 0 0 1 removed2 8 17 - spare
/dev/sdb1If I attempt to force the spare to become active it begins to recover:
$ sudo mdadm -S /dev/md101 mdadm: stopped /dev/md101 $ sudo mdadm --assemble --force --no-degraded /dev/md101 /dev/sdf1 /dev/sdb1 mdadm: /dev/md101 has been started with 1 drive (out of 2) and 1 spare. $ cat /proc/mdstat md101 : active raid1 sdf1[0] sdb1[2] 1953511936 blocks [2/1] [U_] [>....................] recovery = 0.0% (541440/1953511936) finish=420.8min speed=77348K/secThis runs for the allotted time but returns to the state of spare.
Neither disk partition report errors:
$ cat /sys/block/md101/md/dev-sdf1/errors 0 $ cat /sys/block/md101/md/dev-sdb1/errors 0Quoted 6 lines
Are there mdadm logs to find out why this is not recovering properly? How
otherwise do I debug this?Cheers,
Tudor.Did you look in the various
/var/log/messages(current and previous ones) to see what it indicated happened the about the time it completed?Quoted 5 lines
There is almost certainly something in there indicating what went wrong. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlThanks. I watched the logs messages during the recovery. During the last 0.1% (at 99.9%) messages like this appeared:
Dec 24 18:20:32 barney kernel: [2796835.703313] sd 2:0:0:0: [sdf] Unhandled sense code Dec 24 18:20:32 barney kernel: [2796835.703316] sd 2:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 24 18:20:32 barney kernel: [2796835.703320] sd 2:0:0:0: [sdf] Sense Key : Medium Error [current] [descriptor] Dec 24 18:20:32 barney kernel: [2796835.703325] Descriptor sense data with sense descriptors (in hex): Dec 24 18:20:32 barney kernel: [2796835.703327] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 Dec 24 18:20:32 barney kernel: [2796835.703335] e8 e0 5f 86 Dec 24 18:20:32 barney kernel: [2796835.703339] sd 2:0:0:0: [sdf] Add. Sense: Unrecovered read error - auto reallocate failed Dec 24 18:20:32 barney kernel: [2796835.703345] sd 2:0:0:0: [sdf] CDB: Read(10): 28 00 e8 e0 5f 7f 00 00 08 00 Dec 24 18:20:32 barney kernel: [2796835.703353] end_request: I/O error, dev sdf, sector 3907018630 Dec 24 18:20:32 barney kernel: [2796835.703366] ata3: EH complete Dec 24 18:20:32 barney kernel: [2796835.703383] md/raid1:md101: sdf:
Quoted 5 lines
unrecoverable I/O read error for block 3907018496
Unfortunately, sdf is the active disk in this case. So I guess my only option left is to create a new array and copy as much over as it will let me?
If you are lucky that may be an unused area of the fs and you may not lose
any data at all. Worse case is you will probably lose a couple of
files, the way
to tell which you lost is when you read the files with the bad block you will get an
io error.
I don't know if there is a better way, but the process you mentioned is probably reasonable (build a new one and copy all of the data over), then add the bad disk
back to the new mirror and it will build back in, since it is writing it will write to
that block and either correct the error or the disk will relocate the sector if the sector
is too bad.
One you have the array rebuild, make sure on the new array to do check either 1x
per month or 1x per week (man md - see section on scrubbing), if you were not doing
this before then a "sector" can slowly go bad and is never read so the disk (if the
disk is not doing test reads) won't know the sector is going bad until something
like a rebuild happens and that is too late, if the disk is forced to read it or is scanning
itself then the disk will find and relocate (or rewrite) the sector before it is completely bad.