Raid 6 - TLER/CCTL/ERC
A message that is not in the archive (sent off-list or not yet received): 8469417.401286406060921.JavaMail.SYSTEM@ninja
Quoted 3 lines · ----- "Stefan /*St0fF*/ Hübner" <[email protected]> wrote:
Hi,
it has been discussed many times before on the list ...
My apologies. I browsed a little into the past, but obviously not far enough.
Quoted 9 lines
Am 06.10.2010 16:12, schrieb Lemur Kryptering:
Quoted 3 lines
I'll definitely give that a shot when I rebuild this thing.
In the meantime, is there anything that I can do to convince md not
to kick the last disk (running on 6 out of 8 disks) when reading a bad spot? I've tried setting the array to read-only, but this didn't seem to help.
Quoted 13 lines
You can set the ERC values of your drives. Then they'll stop processing
their internal error recovery procedure after the timeout and continue
to react. Without ERC-timeout, the drive tries to correct the error on
its own (not reacting on any requests), mdraid assumes an error after a
while and tries to rewrite the "missing" sector (assembled from the other disks). But the drive will still not react to the write request
as it is still doing its internal recovery procedure. Now mdraid assumes the disk to be bad and kicks it.
That sounds exactly like what I'm seeing in the logs -- the sector initially reported as bad is indeed unreadable via dd. All of the subsequent problems reported in other sectors aren't actually problems when I check on them at a later point. Couldn't this be worked around by exposing whatever timeouts there are in mdraid to something that could be adjusted in /sys?
Quoted 5 lines
There's nothing you can do about this viscious circle except either enabling ERC or using Raid-Edition disk (which have ERC enabled by default).
I tried connecting the drives directly to my motherboard (my controller didn't seem to want to let me pass the smart commands ERC commands to the drives). The ERC commands took, in so far as I was able to read them back with what I set them to. This didn't seem to help much with the issues I was having, however.
Lesson-learned on the non-raid edition disks. I would have spent the extra to avoid all this headache, but am now stuck with these things. I realize that not fixing the problem at the core (the drives themselves), essentially puts the burden on mdraid (which would be forced to block for a ridiculous amount of time waiting for the drive instead of just kicking it), however, in my particular case, this sort of delay would not be a cause for concern.
Would someone be able to nudge me in the right direction as far as where the logic that handles this is located?
Quoted 70 lines
Stefan
Quoted 2 lines
All I'm really trying to do is dd data off of it using
"conv=sync,noerror". When it hits the unreadable spot, it simply kicks the drive from the array, leaving 4/8 disks active, taking down the array.
Quoted 2 lines
Again, I don't understand why md would take this action. It would
make a lot more sense if it simply reported an IO error to whatever made the request.
Quoted 9 lines
Peter Zieba
312-285-3794----- Original Message -----
From: "Phil Turmel" <[email protected]>
To: "Peter Zieba" <[email protected]>
Cc: [email protected]
Sent: Wednesday, October 6, 2010 6:57:58 AM GMT -06:00 US/CanadaCentral
Quoted 15 lines
Subject: Re: Raid 6 - TLER/CCTL/ERC
Quoted 12 lines · Peter Zieba
Hey all,
I have a question regarding Linux raid and degraded arrays.
My configuration involves:
- 8x Samsung HD103UJ 1TB drives (terrible consumer-grade)
- AOC-USAS-L8i Controller
- CentOS 5.5 2.6.18-194.11.1.el5xen (64-bit)
- Each drive has one maximum-sized partition.
- 8-drives are configured in a raid 6.My understanding is that with a raid 6, if a disk cannot return a
given sector, it should still be possible to get what should have been returned from the first disk, from two other disks. My understanding is also that if this is successful, this should be written back to the disk that originally failed to read the given sector. I'm assuming that's what a message such as this indicates:
Quoted 1 line
Quoted 1 line
Sep 17 04:01:12 doorstop kernel: raid5:md0: read error corrected (8sectors at 1647989048 on sde1)
Quoted 2 lines
Quoted 2 lines
I was hoping to confirm my suspicion on the meaning of that
message.
Quoted 3 lines
Quoted 3 lines
On occasion, I'll also see this:
Oct 1 01:50:53 doorstop kernel: raid5:md0: read error notcorrectable (sector 1647369400 on sdh1).
Quoted 2 lines
Quoted 2 lines
This seems to involved the drive being kicked from the array, even
though the drive is still readable for the most part (save for a few sectors).
Quoted 6 lines
[snip /]
Hi Peter,
For read errors that aren't permanent (gone after writing to the
affected sectors), a "repair" action is your friend. I used to deal with occasional kicked-out drives in my arrays until I started running the following script in a weekly cron job:
Quoted 8 lines
#!/bin/bash
#
for x in/sys/block/md*/md/sync_action; do
echo repair >$x
done
Quoted 8 lines
Quoted 5 lines
HTH,
Phil
--
To unsubscribe from this list: send the line "unsubscribelinux-raid" in
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Am 07.10.2010 01:11, schrieb Lemur Kryptering:
Quoted 9 lines
[...]
That sounds exactly like what I'm seeing in the logs -- the sector initially reported as bad is indeed unreadable via dd. All of the subsequent problems reported in other sectors aren't actually problems when I check on them at a later point. Couldn't this be worked around by exposing whatever timeouts there are in mdraid to something that could be adjusted in /sys?
Quoted 4 lines
There's nothing you can do about this viscious circle except either enabling ERC or using Raid-Edition disk (which have ERC enabled by default).
I must say yesterday we had our first Hitachi UltraStar Drives - which are supposed to be Raid-Edition. They didn't have ERC enabled. I'll inquire Hitachi about that today.
Quoted 3 lines
Quoted 1 line
I tried connecting the drives directly to my motherboard (my controller didn't seem to want to let me pass the smart commands ERC commands to the drives). The ERC commands took, in so far as I was able to read them back with what I set them to. This didn't seem to help much with the issues I was having, however.
Which wouldn't work, as the SCT ERC settings are volatile. I.e.:
they're gone after a power cycle.
Quoted 12 lines
Lesson-learned on the non-raid edition disks. I would have spent the extra to avoid all this headache, but am now stuck with these things. I realize that not fixing the problem at the core (the drives themselves), essentially puts the burden on mdraid (which would be forced to block for a ridiculous amount of time waiting for the drive instead of just kicking it), however, in my particular case, this sort of delay would not be a cause for concern.
Would someone be able to nudge me in the right direction as far as where the logic that handles this is located?
Quoted 7 lines
[...]
Quoted 6 lines
#!/bin/bash
#
for x in/sys/block/md*/md/sync_action; do
echo repair >$x
done
[...]
That is probably the only thing you can try. As this does indeed try to reconstruct the sector from the redundancy. But I'd try it with ERC enabled. Maybe you find a way where this works. (i.e. move the whole Raid to the other computer...)
Stefan