Help on first dangerous scrub / suggestions
Hi all
we have a server with a 12 disks raid-6.
It has been up for 1 year now but I have never scrubbed it because at the time I did not know about this good practice (a note on man mdadm would help). The array is currently not degraded and has spares.
Now I am scared about initiating the first scrub because if it turns out that 3 areas in different disks have bad sectors I think am gonna lose the whole array.
Doing backups now it's also scary because if I hit a bad (uncorrectable) area in anyone of the disks while reading, a rebuild will start on the spare and that's like initiating the scrub with all associated risks.
About this point, I would like to suggest a new "mode" of the array, let's call it "nodegrade" in which no degradation can occur, and I/O in unreadable areas simply fails with I/O error. By temporarily putting the array in that mode, at least one could backup without anxiety. I understand it would not be possible to add a spare / rebuild in this mode but that's ok.
BTW I would like to ask an info on "readonly" mode mentioned here:
http://www.mjmwired.net/kernel/Documentation/md.txt
upon read error, will it initiate a rebuild / degrade the array or not?
Anyway the "nodegrade" mode I suggest above would be still more useful because you do not need to put the array in readonly mode, which is important for doing backups during normal operation.
Coming back to my problem, I have thought that the best approach would probably be to first collect information on how good are my 12 drives, and I probably can do that by reading each device like
dd if=/dev/sda of=/dev/null
and see how many of them read with errors. I just hope my 3ware disk controllers won't disconnect the whole drive upon read error.
(anyone has a better strategy?)
But then if it turns out that 3 of them indeed have unreadable areas I am screwed anyway. Even with dd_rescue there's no strategy that can save my data, even if the unreadable areas have different placement in the 3 disks (and that's a case where it should instead be possible to get data back).
This brings to my second suggestion:
I would like to see 12 (in my case) devices like:
/dev/md0_fromparity/{sda1,sdb1,...} (all readonly)
that behave like this: when reading from /dev/md0_fromparity/sda1 , what comes out is the bytes that should be in sda1, but computed from the other disks. Reading from these devices should never degrade an array, at most give read error.
Why is this useful?
Because one could recover sda1 from a disastered array with multiple unreadable areas (unless too many are overlapping) in this way:
With the array in "nodegrade" mode and blockdevice marked as readonly:
1- dd_rescue if=/dev/sda1 of=/dev/sdz1 [sdz is a good drive to
eventually take sda place]
take note of failed sectors
2- dd_rescue from /dev/md0_fromparity/sda1 to /dev/sdz1 only for the
sectors that were unreadable from above
3- stop array, take out sda1, and reassemble the array with sdz1 in
place of sda1
... repeat for all the other drives to get a good array back.
What do you think?
I have another question on scrubbing: I am not sure about the exact behaviour of "check" and "repair":
- will "check" degrade an array if it finds an uncorrectable read-error? The manual only mentions what happens if the checksums of the parity disks don't match with data, but that's not what I'm interested in right now.
- will "repair" .... (same question as above)
Thanks for your comments
Quoted 45 lines · Asdo
Hi all
we have a server with a 12 disks raid-6.
It has been up for 1 year now but I have never scrubbed it because at the time I did not know about this good practice (a note on man mdadm would help). The array is currently not degraded and has spares.Now I am scared about initiating the first scrub because if it turns out that 3 areas in different disks have bad sectors I think am gonna lose the whole array.
Doing backups now it's also scary because if I hit a bad (uncorrectable) area in anyone of the disks while reading, a rebuild will start on the spare and that's like initiating the scrub with all associated risks.
About this point, I would like to suggest a new "mode" of the array, let's call it "nodegrade" in which no degradation can occur, and I/O in unreadable areas simply fails with I/O error. By temporarily putting the array in that mode, at least one could backup without anxiety. I understand it would not be possible to add a spare / rebuild in this mode but that's ok.
BTW I would like to ask an info on "readonly" mode mentioned here:
http://www.mjmwired.net/kernel/Documentation/md.txt
upon read error, will it initiate a rebuild / degrade the array or not?Anyway the "nodegrade" mode I suggest above would be still more useful because you do not need to put the array in readonly mode, which is important for doing backups during normal operation.
Coming back to my problem, I have thought that the best approach would probably be to first collect information on how good are my 12 drives, and I probably can do that by reading each device like dd if=/dev/sda of=/dev/null
and see how many of them read with errors. I just hope my 3ware disk controllers won't disconnect the whole drive upon read error.
(anyone has a better strategy?)But then if it turns out that 3 of them indeed have unreadable areas I am screwed anyway. Even with
dd_rescuethere's no strategy that can save my data, even if the unreadable areas have different placement in the 3 disks (and that's a case where it should instead be possible to get data back).This brings to my second suggestion:
I would like to see 12 (in my case) devices like:/dev/md0_fromparity/{sda1,sdb1,...}(all readonly)
that behave like this: when reading from/dev/md0_fromparity/sda1, what comes out is the bytes that should be in sda1, but computed from the other disks. Reading from these devices should never degrade an array, at most give read error.Why is this useful?
Because one could recover sda1 from a disastered array with multiple unreadable areas (unless too many are overlapping) in this way:
With the array in "nodegrade" mode and blockdevice marked as readonly:
1-dd_rescueif=/dev/sda1 of=/dev/sdz1 [sdz is a good drive to eventually take sda place]
take note of failed sectors
2-dd_rescuefrom/dev/md0_fromparity/sda1to/dev/sdz1only for the sectors that were unreadable from above 3- stop array, take out sda1, and reassemble the array with sdz1 in place of sda1
... repeat for all the other drives to get a good array back.What do you think?
I have another question on scrubbing: I am not sure about the exact behaviour of "check" and "repair":
- will "check" degrade an array if it finds an uncorrectable read-error? The
manual only mentions what happens if the checksums of the parity disks don't match with data, but that's not what I'm interested in right now.
- will "repair" .... (same question as above)
Quoted 6 lines
Thanks for your comments
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Have you gotten any filesystem errors thus far?
How bad are the disks?
Can you show the smartctl -a output of each of the 12 drives? Can you rsync all of the data to another host?
What filesystem is being used?
If your disks are failing I'd recommend an rsync ASAP over trying to read/write/test the disks with dd or other tests.
Justin.
Quoted 54 lines · Justin Piszcz
Quoted 38 lines · Asdo
Hi all
we have a server with a 12 disks raid-6.
It has been up for 1 year now but I have never scrubbed it because at the time I did not know about this good practice (a note on man mdadm would help). The array is currently not degraded and has spares.Now I am scared about initiating the first scrub because if it turns out that 3 areas in different disks have bad sectors I think am gonna lose the whole array.
Doing backups now it's also scary because if I hit a bad (uncorrectable) area in anyone of the disks while reading, a rebuild will start on the spare and that's like initiating the scrub with all associated risks.
About this point, I would like to suggest a new "mode" of the array, let's call it "nodegrade" in which no degradation can occur, and I/O in unreadable areas simply fails with I/O error. By temporarily putting the array in that mode, at least one could backup without anxiety. I understand it would not be possible to add a spare / rebuild in this mode but that's ok.
BTW I would like to ask an info on "readonly" mode mentioned here:
http://www.mjmwired.net/kernel/Documentation/md.txt
upon read error, will it initiate a rebuild / degrade the array or not?Anyway the "nodegrade" mode I suggest above would be still more useful because you do not need to put the array in readonly mode, which is important for doing backups during normal operation.
Coming back to my problem, I have thought that the best approach would probably be to first collect information on how good are my 12 drives, and I probably can do that by reading each device like dd if=/dev/sda of=/dev/null
and see how many of them read with errors. I just hope my 3ware disk controllers won't disconnect the whole drive upon read error.
(anyone has a better strategy?)But then if it turns out that 3 of them indeed have unreadable areas I am screwed anyway. Even with
dd_rescuethere's no strategy that can save my data, even if the unreadable areas have different placement in the 3 disks (and that's a case where it should instead be possible to get data back).This brings to my second suggestion:
I would like to see 12 (in my case) devices like:/dev/md0_fromparity/{sda1,sdb1,...}(all readonly)
that behave like this: when reading from/dev/md0_fromparity/sda1, what comes out is the bytes that should be in sda1, but computed from the other disks. Reading from these devices should never degrade an array, at most give read error.Why is this useful?
Because one could recover sda1 from a disastered array with multiple unreadable areas (unless too many are overlapping) in this way:
With the array in "nodegrade" mode and blockdevice marked as readonly:
1-dd_rescueif=/dev/sda1 of=/dev/sdz1 [sdz is a good drive to eventually take sda place]
take note of failed sectors
2-dd_rescuefrom/dev/md0_fromparity/sda1to/dev/sdz1only for the sectors that were unreadable from above 3- stop array, take out sda1, and reassemble the array with sdz1 in place of sda1
... repeat for all the other drives to get a good array back.Quoted 13 lines
What do you think?
I have another question on scrubbing: I am not sure about the exact behaviour of "check" and "repair":
- will "check" degrade an array if it finds an uncorrectable
read-error? The manual only mentions what happens if the checksums of the parity disks don't match with data, but that's not what I'm interested in right now.
- will "repair" .... (same question as above)Thanks for your comments
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines
Have you gotten any filesystem errors thus far?
How bad are the disks?
Only one disk gave correctable read errors in dmesg twice (no filesystem errors), 64 sectors in sequence each time. Smartctl -a reports indeed those errors on that disk, and no errors on all the other disks.
(
on the partially-bad disk:
SMART overall-health self-assessment test result: PASSED
...
1 Raw_Read_Error_Rate 0x000f 200 200 051 Pre-fail Always - 138
...
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
the other disks have values: PASSED, 0, 0
)
However I never ran smartctl tests, so the only errors smartctl is aware of are indeed those I also got from md.
Quoted 5 lines
Can you show the smartctl -a output of each of the 12 drives? Can you rsync all of the data to another host?
What filesystem is being used?If your disks are failing I'd recommend an rsync ASAP over trying to read/write/test the disks with dd or other tests.
Filesystem is ext3
For the rsync I am worried, have you read my original post? If rsync hits an area with uncorrectable read errors the rebuild will start and then if turns out there are other 2 partially-unreadable disks I will lose the array. And I will lose it *right now* and without knowing for sure before. What are the drawbacks you see against the dd test I proposed? It's just to probe to have an idea of how bad is the situation, without changing the situation yet...
Quoted 4 lines · Asdo
Quoted 4 lines
Quoted 4 lines
BTW I would like to ask an info on "readonly" mode mentioned here:
http://www.mjmwired.net/kernel/Documentation/md.txt
upon read error, will it initiate a rebuild / degrade the array or not?
This is a good question but it is difficult to test as each use case is different. That would be a question for Neil.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Anyway the "nodegrade" mode I suggest above would be still more useful because you do not need to put the array in readonly mode, which is important for doing backups during normal operation.
Coming back to my problem, I have thought that the best approach would probably be to first collect information on how good are my 12 drives, and I probably can do that by reading each device like dd if=/dev/sda of=/dev/null
and see how many of them read with errors. I just hope my 3ware disk controllers won't disconnect the whole drive upon read error.
(anyone has a better strategy?)
I see where you're going here. Read below but if you go this route I assume you would first stop the array (?) mdadm -S /dev/mdX and then test each individual disk one at a time?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
But then if it turns out that 3 of them indeed have unreadable areas I am screwed anyway. Even with
dd_rescuethere's no strategy that can save my data, even if the unreadable areas have different placement in the 3 disks (and that's a case where it should instead be possible to get data back).
So wouldn't your priority to copy/rsync the *MOST* important data off the machine first before resorting to more invasive methods?
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
This brings to my second suggestion:
I would like to see 12 (in my case) devices like:/dev/md0_fromparity/{sda1,sdb1,...}(all readonly)
that behave like this: when reading from/dev/md0_fromparity/sda1, what comes out is the bytes that should be in sda1, but computed from the other disks. Reading from these devices should never degrade an array, at most give read error.Why is this useful?
Because one could recover sda1 from a disastered array with multiple unreadable areas (unless too many are overlapping) in this way:
With the array in "nodegrade" mode and blockdevice marked as readonly:
1-dd_rescueif=/dev/sda1 of=/dev/sdz1 [sdz is a good drive to eventually take sda place]
take note of failed sectors
2-dd_rescuefrom/dev/md0_fromparity/sda1to/dev/sdz1only for the sectors that were unreadable from above 3- stop array, take out sda1, and reassemble the array with sdz1 in place of sda1
... repeat for all the other drives to get a good array back.
Quoted 1 line
Quoted 1 line
Quoted 1 line
What do you think?
While this may be possibly, has anyone on this list done something like this and had it work successfully?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
I have another question on scrubbing: I am not sure about the exact behaviour of "check" and "repair":
- will "check" degrade an array if it finds an uncorrectable read-error?
From README.checkarray:
'check' is a read-only operation, even though the kernel logs may suggest otherwise (e.g. /proc/mdstat and several kernel messages will mention "resync"). Please also see question 21 of the FAQ.
If, however, while reading, a read error occurs, the check will trigger the normal response to read errors which is to generate the 'correct' data and try to write that out - so it is possible that a 'check' will trigger a write. However in the absence of read errors it is read-only.
Per md.txt:
resync - redundancy is being recalculated after unclean
shutdown or creation
repair - A full check and repair is happening. This is
similar to 'resync', but was requested by the user, and the write-intent bitmap is NOT used to optimise the process.
check - A full check of redundancy was requested and is
happening. This reads all block and checks them. A repair may also happen for some raid levels.
Quoted 12 lines
Quoted 10 lines
Quoted 9 lines
The manual only mentions what happens if the checksums of the parity disks don't match with data, but that's not what I'm interested in right now.
- will "repair" .... (same question as above)Thanks for your comments
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 2 lines
Have you gotten any filesystem errors thus far?
How bad are the disks?
Quoted 14 lines
Only one disk gave correctable read errors in dmesg twice (no filesystem errors), 64 sectors in sequence each time. Smartctl -a reports indeed those errors on that disk, and no errors on all the other disks.
(
on the partially-bad disk:
SMART overall-health self-assessment test result: PASSED
...
1Raw_Read_Error_Rate0x000f200 200 051 Pre-fail Always
- 138
...
5Reallocated_Sector_Ct0x0033200 200 140 Pre-fail Always
- 0
the other disks have values: PASSED, 0, 0
)
However I never ran smartctl tests, so the only errors smartctl is aware of are indeed those I also got from md.
Ouch, in addition if you do not run the 'offline' test mentioned in the smartctl manpage, (all of the offline-test related statistics) will NOT be updated, so there is no way to tell how bad the disk really is, the smartctl statistics for those disks are unknown because they have not been updated. I had a REAL weird issue once with a mdadm raid-1 where one disk kept dropping out of the array (two raptor 150s) and I had not run the offline test and got fed up with it all and put them on a 3ware controller. Shortly thereafter, I built a new RAID-1 with the same disks and I saw many re-allocated sectors, the drive was on its way out. However, since I had not run an offline test before, the disk looked completely FINE, all smart tests had passed (short,long) && the output from smartctl -a looked good too!
Quoted 8 lines
Quoted 6 lines
Can you show the smartctl -a output of each of the 12 drives? Can you rsync all of the data to another host?
What filesystem is being used?If your disks are failing I'd recommend an rsync ASAP over trying to read/write/test the disks with dd or other tests.
Filesystem is ext3
For the rsync I am worried, have you read my original post? If rsync hits an area with uncorrectable read errors the rebuild will start and then if turns out there are other 2 partially-unreadable disks I will lose the array. And I will lose it *right now* and without knowing for sure before.
Per your other reply, it is plausible that what you are saying may occur. I have to ask though if you have 12 disks on a 3ware controller, why are you not using HW RAID-6? Whenever there are read errors on a 3ware controller it simply remaps the bad sector and marks it as bad, for each sector, and it does not drop out of the array until there are > 100-300 reallocated sectors (if these are enterprise drives) (and depending on how the drive fails of course)..
Aside from that, if your array is say, 50% full and you rsync, you only need to read what is on the disks and not the entire array (as you would need to do with the dd). In addition, this would also allow you to rsync your most important data off at your choosing. If you go ahead with the dd test and through it you find 3 disks fail during the process, what have you gained?
There is a risk you take either way, your method may bear less risk as long as no drives completely fail during your read tests. Whereas if you copy or rsync the data, you may be successful, or not; however in the second scenario you (hopefully) end up with the data in a second location, to which you can then run all of the tests you want thereafter.
Quoted 1 line
What are the drawbacks you see against the dd test I proposed? It's just to probe to have an idea of how bad is the situation, without changing the situation yet...
Maybe.. As long as the dd test does not brick the drives (unlikely) but it could happen.
Quoted 12 lines · Justin Piszcz
Quoted 4 lines · Asdo
Quoted 4 lines
Quoted 4 lines
BTW I would like to ask an info on "readonly" mode mentioned here:
http://www.mjmwired.net/kernel/Documentation/md.txt
upon read error, will it initiate a rebuild / degrade the array or not?This is a good question but it is difficult to test as each use case is different. That would be a question for Neil.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Anyway the "nodegrade" mode I suggest above would be still more useful because you do not need to put the array in readonly mode, which is important for doing backups during normal operation.
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Coming back to my problem, I have thought that the best approach would probably be to first collect information on how good are my 12 drives, and I probably can do that by reading each device like dd if=/dev/sda of=/dev/null
and see how many of them read with errors. I just hope my 3ware disk controllers won't disconnect the whole drive upon read error.
(anyone has a better strategy?)I see where you're going here. Read below but if you go this route I assume you would first stop the array (?) mdadm -S
/dev/mdXand then test each individual disk one at a time?
I don't plan to stop the array prior to reading the drives. Reads should not be harmful... You think otherwise?
I think I did read the drives in the past on a mounted array and it was no problem.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
But then if it turns out that 3 of them indeed have unreadable areas I am screwed anyway. Even with
dd_rescuethere's no strategy that can save my data, even if the unreadable areas have different placement in the 3 disks (and that's a case where it should instead be possible to get data back).So wouldn't your priority to copy/rsync the *MOST* important data off the machine first before resorting to more invasive methods?
Yeah I will eventually do that if I find more than 2 drives with read errors.
(dd read to the individual drives is less invasive than rsync imho)
So You are saying that even if I find less than 2 disks with read errors (which might even be correctable) with dd reads, you would anyway proceed with a backup before the scrub?
(Actually I would need to also test the spares for write functionality, heck... Oh well... I have many spares...)
I miss so much a "nodegrade" mode as described in my original post :-/ ("undegradeable" would probably be more correct btw)
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
This brings to my second suggestion:
I would like to see 12 (in my case) devices like:/dev/md0_fromparity/{sda1,sdb1,...}(all readonly)
that behave like this: when reading from/dev/md0_fromparity/sda1, what comes out is the bytes that should be in sda1, but computed from the other disks. Reading from these devices should never degrade an array, at most give read error.Why is this useful?
Because one could recover sda1 from a disastered array with multiple unreadable areas (unless too many are overlapping) in this way:
With the array in "nodegrade" mode and blockdevice marked as readonly:
1-dd_rescueif=/dev/sda1 of=/dev/sdz1 [sdz is a good drive to eventually take sda place]
take note of failed sectors
2-dd_rescuefrom/dev/md0_fromparity/sda1to/dev/sdz1only for the sectors that were unreadable from above 3- stop array, take out sda1, and reassemble the array with sdz1 in place of sda1
... repeat for all the other drives to get a good array back.
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
What do you think?
While this may be possibly, has anyone on this list done something like this and had it work successfully?
Nobody could try this way because the /dev/md0_fromparity/{sda1,sdb1,...} do not exist. This is a feature request...
Quoted 25 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
I have another question on scrubbing: I am not sure about the exact behaviour of "check" and "repair":
- will "check" degrade an array if it finds an uncorrectable
read-error?From README.checkarray:
'check' is a read-only operation, even though the kernel logs may suggest otherwise (e.g.
/proc/mdstatand several kernel messages will mention "resync"). Please also see question 21 of the FAQ.If, however, while reading, a read error occurs, the check will trigger the normal response to read errors which is to generate the 'correct' data and try to write that out - so it is possible that a 'check' will trigger a write. However in the absence of read errors it is read-only.
Per md.txt:
resync - redundancy is being recalculated after unclean
shutdown or creationrepair - A full check and repair is happening. This is
similar to 'resync', but was requested by the user, and the write-intent bitmap is NOT used to optimise the process.
Quoted 4 lines
check - A full check of redundancy was requested and is
happening. This reads all block and checks them. A repair may also happen for some raid levels.
Unfortunately this does not specifically answer the question, even though the sentence
"If, however, while reading, a read error occurs, the check will trigger the normal response to read errors..." seems to suggest that in case of uncorrectable read error the drive will be kicked.
Quoted 17 lines · Asdo
Quoted 16 lines · Justin Piszcz
Quoted 4 lines · Asdo
Quoted 4 lines
Quoted 4 lines
BTW I would like to ask an info on "readonly" mode mentioned here:
http://www.mjmwired.net/kernel/Documentation/md.txt
upon read error, will it initiate a rebuild / degrade the array or not?This is a good question but it is difficult to test as each use case is different. That would be a question for Neil.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Anyway the "nodegrade" mode I suggest above would be still more useful because you do not need to put the array in readonly mode, which is important for doing backups during normal operation.
Coming back to my problem, I have thought that the best approach would probably be to first collect information on how good are my 12 drives, and I probably can do that by reading each device like dd if=/dev/sda of=/dev/null
and see how many of them read with errors. I just hope my 3ware disk controllers won't disconnect the whole drive upon read error.
(anyone has a better strategy?)
Quoted 6 lines
Quoted 3 lines
I see where you're going here. Read below but if you go this route I assume you would first stop the array (?) mdadm -S
/dev/mdXand then test each individual disk one at a time?I don't plan to stop the array prior to reading the drives. Reads should not be harmful... You think otherwise?
Depends on the drives, ever try to copy a file from a failing drive? Some drives will start to click/reset/ATA errors, etc. Depends on how it is failing and what is wrong with it.
Quoted 1 line
I think I did read the drives in the past on a mounted array and it was no problem.
Good to hear, let us know which option you choose & what the outcome is!
Justin.
Quoted 9 lines · Justin Piszcz
Quoted 6 lines
Quoted 3 lines
I see where you're going here. Read below but if you go this route I assume you would first stop the array (?) mdadm -S
/dev/mdXand then test each individual disk one at a time?I don't plan to stop the array prior to reading the drives. Reads should not be harmful... You think otherwise?
Depends on the drives, ever try to copy a file from a failing drive? Some drives will start to click/reset/ATA errors, etc. Depends on how it is failing and what is wrong with it.
You are right, good thinking...
The drives are WD RE2, so they do have TLER, however the default retry time is I think 7 seconds. So if I read a drive with dd and it hits an unreadable area it will retry for 7 second and during that time it would not be responsive and any request coming from MD, and I think 7secs is probably enough for MD to kick the drive out of the array. If the MD requests are queued in the elevator maybe MD will wait... not sure. I might help them to queue in elevator by disabling NCQ.
TLER is configurable in theory (so to lower the time to e.g. 1 second) but I have never done it and it seems I would need to reboot into MSDOS / Freedos (wdtler is DOS utility as it seems). The problem is that the computer is in heavy use and the array also.
This would be another question for Neil (if 7 secs is enough for MD to kick out the drive, and if the drive will still be kicked if MD requests to it are queued in the linux elevator).
Without knowing this I will probably opt for your way: rsync data out, starting from the smallest and most important stuff...
Quoted 1 line
Quoted 1 line
I think I did read the drives in the past on a mounted array and it was no problem.
I forgot to mention that in that case there were no read errors... :-D
Quoted 3 lines
Good to hear, let us know which option you choose & what the outcome is!
Justin.
Thank you
Quoted 3 lines · Asdo
....
Without knowing this I will probably opt for your way: rsync data out, starting from the smallest and most important stuff...
...
I had another thought:
If I take the computer offline, boot with a livecd (one of the worst messes here is that the root filesystem is also in that array), run the raid6 array in READONLY MODE and maybe without spares, then I start a check (scrub) ...
If drives are kicked I can probably reasseble --force the array and it's like nothing happened, right? Since it was mounted readonly I think it would be clean...
Only problem would be if 1 or more drives definitively die during the procedure, but I hope this is unlikely...
If less than 3 drives die I can still reassemble --force, and take the data out (at least SOME data, then if it degrades, reassemble again and try to get out data from another location...)
Do you agree?
I am starting to think that during the procedure for taking the data out and/or attempt first scrubbing the main problem are write accesses to the array, because if rebuild starts on a spare and then fails again and then there were writes in the middle... I think I end up doomed. Probably even reassemble --force would refuse to work on me. What do you think?
Thank you
Quoted 11 lines · Asdo
Quoted 3 lines · Asdo
....
Without knowing this I will probably opt for your way: rsync data out, starting from the smallest and most important stuff...
...I had another thought:
If I take the computer offline, boot with a livecd (one of the worst messes here is that the root filesystem is also in that array), run the raid6 array in READONLY MODE and maybe without spares, then I start a check (scrub) ...
If drives are kicked I can probably reasseble --force the array and it's like nothing happened, right?
If *more* than 2 drives fail you would need to --force. Also, when you do that (I have done it before), you need to fsck the filesystem and often many of your files will end up in /lost+found, depending on how bad it is. But in all of my tests, the FS was R/W, not R/O, so I am unsure of the outcome, it sounds like a possibility. With SW/RAID-6 I have lost two disks before and suffered no problems at all (you can't use Western Digital Velociraptors in RAID), I was able to copy the data that I needed off of the array without any issues but my server was not being heavily utilized as yours is.
I think you need to run a smartctl on each of the drives, e.g.:
for disk in /dev/sd[a-z]
do
echo "disk $disk" >> /tmp/a
smartctl -a $disk >> /tmp/a
done
Inspect each disk, is there really a failing disk?..
Quoted 6 lines
Since it was mounted readonly I think it would be clean...
Only problem would be if 1 or more drives definitively die during the procedure, but I hope this is unlikely...
If less than 3 drives die I can still reassemble --force, and take the data out (at least SOME data, then if it degrades, reassemble again and try to get out data from another location...)Do you agree?
I agree, but would you not want to just rsync the data off first before going through all of this?
Quoted 2 lines
I am starting to think that during the procedure for taking the data out and/or attempt first scrubbing the main problem are write accesses to the array, because if rebuild starts on a spare and then fails again and then there were writes in the middle... I think I end up doomed. Probably even reassemble --force would refuse to work on me. What do you think?
I think you have a point there.. In my opinion:
1. Check each disk, how bad is it, really? (it seems like your array and disks are fine, one disk may have a re-allocated sector or two, nothing to worry about) in all seriousness. Do any of the attributes say *FAILING NOW*?
2. If everything looks OK, copy all of the data off while the system is ONLINE and WORKING, it will be *MUCH* more difficult trying to extract pieces of data using dd_rescue and friends vs. just rsyncing the entire array to another host.
3. Do you not have another host to rsync to? If that is the case then we may need to approach this problem from a different angle. E.g., making it read-only after booting from a LiveCD may not be a bad idea, but doing that BEFORE you rsync'd all the data off is still risking all of the data on the array, whereas since it is currently up and running you could at least make a point-in-time copy of all of the data that lives there right now.
Justin.
Quoted 6 lines · On Fri, 27 Nov 2009 19:11:31 +0100 Asdo <[email protected]> wrote:
Quoted 4 lines · Asdo
....
Without knowing this I will probably opt for your way: rsync data out, starting from the smallest and most important stuff...
...
Quoted 6 lines
I had another thought:
If I take the computer offline, boot with a livecd (one of the worst messes here is that the root filesystem is also in that array), run the raid6 array in READONLY MODE and maybe without spares, then I start a check (scrub) ...
If the array is marked read-only, it wont do a scrub.
However if you simply don't have any filesystem mounted, then the array will remain 'clean' and any failures are less likely cause further failures.
So doing it off line is a good idea, but setting the array to read-only won't work.
NeilBrown
Quoted 9 lines · Neil Brown
If the array is marked read-only, it wont do a scrub.
However if you simply don't have any filesystem mounted, then the array will remain 'clean' and any failures are less likely cause further failures.
So doing it off line is a good idea, but setting the array to read-only won't work.
Thanks for the info Neil,
would mounting the filesystems readonly be safe in the same way? (I might try to rsync some data out during the scrub) Thank you
Asdo
Quoted 1 line · Asdo
Now I am scared about initiating the first scrub because if it turns out that 3 areas in different disks have bad sectors I think am gonna lose the whole array.
What kernel are you using?
As of 2.6.15 or so, sending "repair" (or "resync", I don't remember exactly) to the md will read all data and if there is bad data, parity will be used to write to the bad sector (it shouldn't kick the disk).
<http://linux-raid.osdl.org/index.php/RAID_Administration>
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 10 lines · Mikael Abrahamsson
Quoted 1 line · Asdo
Now I am scared about initiating the first scrub because if it turns out that 3 areas in different disks have bad sectors I think am gonna lose the whole array.
What kernel are you using?
As of 2.6.15 or so, sending "repair" (or "resync", I don't remember exactly) to the md will read all data and if there is bad data, parity will be used to write to the bad sector (it shouldn't kick the disk).
Kernel is ubuntu kernel 2.6.24 .
In the page you are linking I don't see mention of the fact that drives won't be kicked with a "repair" or "check". In fact regarding "check" this is written:
'check' just reads everything and doesn't trigger any writes unless a
read error is detected, in which case the normal read-error handing kicks in.
"normal error handling" seems to suggest that if the read error is uncorrectable the drive will be kicked. You don't think so?
Thank you