problem killing raid 5
Hello,
I had a raid 5 array on three disks. Because of a hardware problem two disks dissapeared one after the other. I have since been trying to create a new array with them.
Between the degradation of the two disks I tryied removing one of the failed disks and re-adding it to the array. When the second disk failed I noticed the drive numbers on the broken array, and misteriously a fourth drive appeared on it. Now I have numbers 0,1 and 3, but no number 2. mdadm tells me that number 3 is a spare.
Now I want to start all over again, but even after zeroing the superblocks on all three disks, and creation of a new array, /proc/mdstat shows the same drive numbers, while reconstructing the third drive. What should I do ?
Daniel Santos
I retried rebuilding the array once again from scratch, and this time checked the syslog messages. The reconstructions process is getting stuck at a disk block that it can't read. I double checked the block number by repeating the array creation, and did a bad block scan. No bad blocks were found. How could the md driver be stuck if the block is fine ?
Supposing that the disk has bad blocks, can I have a raid device on disks that have badblocks ? Each one of the disks is 400 GB.
Probably not a good idea because if a drive has bad blocks it probably will have more in the future. But anyway, can I ? The bad blocks would have to be known to the md driver.
Quoted 16 lines · Daniel Santos
Hello,
I had a raid 5 array on three disks. Because of a hardware problem two disks dissapeared one after the other. I have since been trying to create a new array with them.
Between the degradation of the two disks I tryied removing one of the failed disks and re-adding it to the array. When the second disk failed I noticed the drive numbers on the broken array, and misteriously a fourth drive appeared on it. Now I have numbers 0,1 and 3, but no number 2. mdadm tells me that number 3 is a spare.
Now I want to start all over again, but even after zeroing the superblocks on all three disks, and creation of a new array,
/proc/mdstatshows the same drive numbers, while reconstructing the third drive. What should I do ?Daniel Santos
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Daniel Santos
I retried rebuilding the array once again from scratch, and this time checked the syslog messages. The reconstructions process is getting stuck at a disk block that it can't read. I double checked the block number by repeating the array creation, and did a bad block scan. No bad blocks were found. How could the md driver be stuck if the block is fine ?
Quoted 6 lines
Supposing that the disk has bad blocks, can I have a raid device on disks that have badblocks ? Each one of the disks is 400 GB.
Probably not a good idea because if a drive has bad blocks it probably will have more in the future. But anyway, can I ?
The bad blocks would have to be known to the md driver.
Well, almost all modern drives can remap bad blocks (at least I know no drive that can't). Most of the time it happens on write - becaue if such a bad block is found during read operation and the drive really can't read the content of that block, it can't remap it either without losing data. From my expirience (about 20 years, many 100s of drives, mostly (old) SCSI but (old) IDE too), it's pretty normal for a drive to develop several bad blocks, especially during first year of usage. Sometimes however, number of bad blocks grows quite rapidly and such a drive definietely should be replaced - at least Seagate drives are covered by warranty in this case.
SCSI drives has 2 so-called "defect lists", stored somewhere inside the drive - factory-preset list (bad blocks found during internal testing when producing a drive), and grown list (bad blocks found by drive during normal usage). Factory-preset list can contain from 0 to about 1000 entries or even more (depending on the size too), grown list can be as large as 500 blocks or more, whenever it's fatal or not depends on whenever new bad blocks continues to be found or not. We have several drives which developed that many bad blocks in first few months of usage, the list stopped growing, and they're still working just fine for >5 years. Both defect lists can be shown by scsitools programs.
I don't know how one can see defect lists on a IDE or SATA drive.
Note that md layer (raid1, 4, 5, 6, 10 - but obviously not raid0 and linear) are now able to repair bad blocks automatically, by forcing write to the same place of the drive where a read error occured - this usually forces drive to automatically reallocate that block and continue.
But in any case, md should not stall - be it during reconstruction or not. For this, I can't comment - to me it smells like a bug somewhere (md layer? error handling in driver? something else?) which should be found and fixed. And for this, some more details are needed I guess -- kernel version is a start.
/mjt
Quoted 45 lines · Michael Tokarev
Quoted 6 lines · Daniel Santos
I retried rebuilding the array once again from scratch, and this time checked the syslog messages. The reconstructions process is getting stuck at a disk block that it can't read. I double checked the block number by repeating the array creation, and did a bad block scan. No bad blocks were found. How could the md driver be stuck if the block is fine ?
Quoted 6 lines
Supposing that the disk has bad blocks, can I have a raid device on disks that have badblocks ? Each one of the disks is 400 GB.
Probably not a good idea because if a drive has bad blocks it probably will have more in the future. But anyway, can I ?
The bad blocks would have to be known to the md driver.Well, almost all modern drives can remap bad blocks (at least I know no drive that can't). Most of the time it happens on write - becaue if such a bad block is found during read operation and the drive really can't read the content of that block, it can't remap it either without losing data. From my expirience (about 20 years, many 100s of drives, mostly (old) SCSI but (old) IDE too), it's pretty normal for a drive to develop several bad blocks, especially during first year of usage. Sometimes however, number of bad blocks grows quite rapidly and such a drive definietely should be replaced - at least Seagate drives are covered by warranty in this case.
SCSI drives has 2 so-called "defect lists", stored somewhere inside the drive - factory-preset list (bad blocks found during internal testing when producing a drive), and grown list (bad blocks found by drive during normal usage). Factory-preset list can contain from 0 to about 1000 entries or even more (depending on the size too), grown list can be as large as 500 blocks or more, whenever it's fatal or not depends on whenever new bad blocks continues to be found or not. We have several drives which developed that many bad blocks in first few months of usage, the list stopped growing, and they're still working just fine for >5 years. Both defect lists can be shown by scsitools programs.
I don't know how one can see defect lists on a IDE or SATA drive.
Note that md layer (raid1, 4, 5, 6, 10 - but obviously not raid0 and linear) are now able to repair bad blocks automatically, by forcing write to the same place of the drive where a read error occured - this usually forces drive to automatically reallocate that block and continue.
Quoted 5 lines
But in any case, md should not stall - be it during reconstruction or not. For this, I can't comment - to me it smells like a bug somewhere (md layer? error handling in driver? something else?) which should be found and fixed. And for this, some more details are needed I guess -- kernel version is a start.
Really? It's my understanding that if md finds an unreadable block during raid5 reconstruction, it has no option but to fail since the information can't be reconstructed. When this happened to me, I had to wipe the bad block, which should allow reconstruction to proceed at the cost of losing the chunk that's on the unreadable block. The bad block howto and messages in this list ~2 years ago explain how to figure out which file(s) is affected.
This is why it's important to run a weekly check so md can repair blocks *before* a drive fails.
cheers,
/Patrik
Quoted 1 line · Patrik Jonsson
Michael Tokarev wrote:
[]
Quoted 9 lines
Quoted 5 lines
But in any case, md should not stall - be it during reconstruction or not. For this, I can't comment - to me it smells like a bug somewhere (md layer? error handling in driver? something else?) which should be found and fixed. And for this, some more details are needed I guess -- kernel version is a start.
Really? It's my understanding that if md finds an unreadable block during raid5 reconstruction, it has no option but to fail since the information can't be reconstructed. When this happened to me, I had to
Yes indeed, it should fail, but not stuck as Daniel reported. Ie, it should either complete the work or fail, but not sleep somewhere in between.
[]
Quoted 2 lines
This is why it's important to run a weekly check so md can repair blocks *before* a drive fails.
*nod*.
/mjt
It stopped the reconstruction process and the output of /proc/mdstat was :
oraculo:/home/dlsa# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [linear]
md0 : active raid5 sdc1[3](S) sdb1[4](F) sdd1[0]
781417472 blocks level 5, 256k chunk, algorithm 2 [3/1] [U__]I then stopped the array and tried to assemble it with a scan :
oraculo:/home/dlsa# mdadm --assemble --scan
mdadm: /dev/md0 assembled from 1 drive and 1 spare - not enough to start the array.
oraculo:/home/dlsa# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [linear]
md0 : inactive sdd1[0](S) sdc1[3](S) sdb1[1](S)
1172126208 blocksThe fourth drive I had to put in mdadm.conf as missing.
The result was that because of the read error, the reconstruction process for the new array aborted, and the assemble came up with an array that seems like the one that failed before I created the new one.
I am running debian with a 2.6.22 kernel.
Quoted 32 lines · Michael Tokarev
Quoted 2 lines · Patrik Jonsson
Michael Tokarev wrote:
[]
Quoted 10 lines
Quoted 6 lines
But in any case, md should not stall - be it during reconstruction or not. For this, I can't comment - to me it smells like a bug somewhere (md layer? error handling in driver? something else?) which should be found and fixed. And for this, some more details are needed I guess -- kernel version is a start.
Really? It's my understanding that if md finds an unreadable block during raid5 reconstruction, it has no option but to fail since the information can't be reconstructed. When this happened to me, I had to
Yes indeed, it should fail, but not stuck as Daniel reported. Ie, it should either complete the work or fail, but not sleep somewhere in between.
[]
Quoted 3 lines
This is why it's important to run a weekly check so md can repair blocks *before* a drive fails.
*nod*.
/mjt
Quoted 53 lines · Daniel Santos
It stopped the reconstruction process and the output of
/proc/mdstatwas :oraculo:/home/dlsa# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [linear] md0 : active raid5 sdc1[3](S) sdb1[4](F) sdd1[0] 781417472 blocks level 5, 256k chunk, algorithm 2 [3/1] [U__]I then stopped the array and tried to assemble it with a scan :
oraculo:/home/dlsa# mdadm --assemble --scan mdadm: /dev/md0 assembled from 1 drive and 1 spare - not enough to start the array. oraculo:/home/dlsa# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [linear] md0 : inactive sdd1[0](S) sdc1[3](S) sdb1[1](S) 1172126208 blocksThe fourth drive I had to put in mdadm.conf as missing.
The result was that because of the read error, the reconstruction process for the new array aborted, and the assemble came up with an array that seems like the one that failed before I created the new one.
I am running debian with a 2.6.22 kernel.
Quoted 29 lines · Michael Tokarev
Quoted 2 lines · Patrik Jonsson
Michael Tokarev wrote:
[]
Quoted 10 lines
Quoted 6 lines
But in any case, md should not stall - be it during reconstruction or not. For this, I can't comment - to me it smells like a bug somewhere (md layer? error handling in driver? something else?) which should be found and fixed. And for this, some more details are needed I guess -- kernel version is a start.
Really? It's my understanding that if md finds an unreadable block during raid5 reconstruction, it has no option but to fail since the information can't be reconstructed. When this happened to me, I had to
Yes indeed, it should fail, but not stuck as Daniel reported. Ie, it should either complete the work or fail, but not sleep somewhere in between.
[]
Quoted 3 lines
This is why it's important to run a weekly check so md can repair blocks *before* a drive fails.
*nod*.
Quoted 9 lines
Quoted 3 lines
/mjt
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Yikes. By the way are all those drives on the same chipset? What type of drives did you use?
Justin.
All the drives are identical, and they are on identical usb enclosures. I am starting to suspect USB. It frequently resets the enclosures. I'll have to look at that first. Anyway I had it working before for some time.
Quoted 71 lines · Justin Piszcz
Quoted 53 lines · Daniel Santos
It stopped the reconstruction process and the output of
/proc/mdstatwas :oraculo:/home/dlsa# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [linear] md0 : active raid5 sdc1[3](S) sdb1[4](F) sdd1[0] 781417472 blocks level 5, 256k chunk, algorithm 2 [3/1] [U__]I then stopped the array and tried to assemble it with a scan :
oraculo:/home/dlsa# mdadm --assemble --scan mdadm: /dev/md0 assembled from 1 drive and 1 spare - not enough to start the array. oraculo:/home/dlsa# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [linear] md0 : inactive sdd1[0](S) sdc1[3](S) sdb1[1](S) 1172126208 blocksThe fourth drive I had to put in mdadm.conf as missing.
The result was that because of the read error, the reconstruction process for the new array aborted, and the assemble came up with an array that seems like the one that failed before I created the new one.
I am running debian with a 2.6.22 kernel.
Quoted 29 lines · Michael Tokarev
Quoted 2 lines · Patrik Jonsson
Michael Tokarev wrote:
[]
Quoted 10 lines
Quoted 6 lines
But in any case, md should not stall - be it during reconstruction or not. For this, I can't comment - to me it smells like a bug somewhere (md layer? error handling in driver? something else?) which should be found and fixed. And for this, some more details are needed I guess -- kernel version is a start.
Really? It's my understanding that if md finds an unreadable block during raid5 reconstruction, it has no option but to fail since the information can't be reconstructed. When this happened to me, I had to
Yes indeed, it should fail, but not stuck as Daniel reported. Ie, it should either complete the work or fail, but not sleep somewhere in between.
[]
Quoted 3 lines
This is why it's important to run a weekly check so md can repair blocks *before* a drive fails.
*nod*.
Quoted 9 lines
Quoted 3 lines
/mjt
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlYikes. By the way are all those drives on the same chipset? What type of drives did you use?
Justin.