When does a disk get flagged as bad?
OK, lets see if I can understand how a disk gets flagged as bad and removed from an array. I was under the impression that any read or write operation failure flags the drive as bad and it gets removed automatically from the array.
However, as I indicated in a prior post I am having problems where the array is never degraded. Does an error of type:end_request: I/O error, dev sdb, sector ....
not count as a read/write error?
Thanks,
Alberto
Signature · 3 lines
Alberto Alonso Global Gate Systems LLC. (512) 351-7233 http://www.ggsys.net Hardware, consulting, sysadmin, monitoring and remote backups
Quoted 9 lines · Alberto Alonso
OK, lets see if I can understand how a disk gets flagged as bad and removed from an array. I was under the impression that any read or write operation failure flags the drive as bad and it gets removed automatically from the array.
However, as I indicated in a prior post I am having problems where the array is never degraded. Does an error of type:
end_request: I/O error, dev sdb, sector ....
not count as a read/write error?
I was also under the impression that any read or write error would fail the drive out of the array but some recent experiments with error injecting seem to indicate otherwise at least for raid1. My working hypothesis is that only write errors fail the drive. Read errors appear to just redirect the sector to a different mirror.
I actually ran across what looks like a bug in the raid1 recovery/check/repair read error logic that I posted about last week but which hasn't generated any response yet (cf.
http://article.gmane.org/gmane.linux.raid/15354). This bug results in sending a zero length write request down to the underlying device driver. A consequence of issuing a zero length write is that it fails at the device level, which raid1 sees as a write failure, which then fails the array. The fix I proposed actually has the effect of *not* failing the array in this case since the spurious failing write is never generated. I'm not sure what is actually supposed to happen in this case. Hopefully, someone more knowledgeable will comment soon.
--
Mike Accetta
ECI Telecom Ltd.
Data Networking Division (previously Laurel Networks)
Quoted 17 lines · Mike Accetta
Quoted 9 lines · Alberto Alonso
OK, lets see if I can understand how a disk gets flagged as bad and removed from an array. I was under the impression that any read or write operation failure flags the drive as bad and it gets removed automatically from the array.
However, as I indicated in a prior post I am having problems where the array is never degraded. Does an error of type:
end_request: I/O error, dev sdb, sector ....
not count as a read/write error?I was also under the impression that any read or write error would fail the drive out of the array but some recent experiments with error injecting seem to indicate otherwise at least for raid1. My working hypothesis is that only write errors fail the drive. Read errors appear to just redirect the sector to a different mirror.
Quoted 13 lines
I actually ran across what looks like a bug in the raid1 recovery/check/repair read error logic that I posted about last week but which hasn't generated any response yet (cf.
http://article.gmane.org/gmane.linux.raid/15354). This bug results in sending a zero length write request down to the underlying device driver. A consequence of issuing a zero length write is that it fails at the device level, which raid1 sees as a write failure, which then fails the array. The fix I proposed actually has the effect of *not* failing the array in this case since the spurious failing write is never generated. I'm not sure what is actually supposed to happen in this case. Hopefully, someone more knowledgeable will comment soon.
--
Mike Accetta
I was starting to think that nobody got my posts, I know there are plenty of people that understand raid and didn't get any answers to any of my related posts.
After thinking about your post, I guess I can see some logic behind not failing on the read, although I would say that after x amount of read failures a drive should be kicked out no matter what.
In my case I believe the errors are during writes, which is still confusing.
Unfortunately I've never done any kind of disk I/O code so I am afraid of looking at the code and getting completely lost.
Alberto
Quoted 4 lines · On Wednesday May 30, [email protected] wrote:
After thinking about your post, I guess I can see some logic behind not failing on the read, although I would say that after x amount of read failures a drive should be kicked out no matter what.
When md gets a read error, it collects the correct data from elsewhere and tries to write it to the drive that apparently failed. If that succeeds, it tries to read it back again. If that succeeds as well, it assumes that the problem has been fixed. Otherwise it fails the drive.
NeilBrown
Quoted 7 lines · Neil Brown
Quoted 4 lines · On Wednesday May 30, [email protected] wrote:
After thinking about your post, I guess I can see some logic behind not failing on the read, although I would say that after x amount of read failures a drive should be kicked out no matter what.
Quoted 6 lines
When md gets a read error, it collects the correct data from elsewhere and tries to write it to the drive that apparently failed. If that succeeds, it tries to read it back again. If that succeeds as well, it assumes that the problem has been fixed. Otherwise it fails the drive.
That's the way it should work, but hopefully the error is logged somewhere. Other than as a SMART relocate?
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979
So, what kind of error is:
end_request: I/O error, dev sdb, sector 42644555end_request: I/O error, dev sdb, sector 124365763
...
I am still trying to figure out why that just make one of my servers unresponsive.
Is there a way to have the md code kick that drive out of the array?
The datacenter people are starting to get impatient having to reboot it every other day.
Thanks,
Alberto
Quoted 18 lines · Neil Brown
Quoted 4 lines · On Wednesday May 30, [email protected] wrote:
After thinking about your post, I guess I can see some logic behind not failing on the read, although I would say that after x amount of read failures a drive should be kicked out no matter what.
When md gets a read error, it collects the correct data from elsewhere and tries to write it to the drive that apparently failed. If that succeeds, it tries to read it back again. If that succeeds as well, it assumes that the problem has been fixed. Otherwise it fails the drive.
NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
Alberto Alonso Global Gate Systems LLC. (512) 351-7233 http://www.ggsys.net Hardware, consulting, sysadmin, monitoring and remote backups