fsck, raid reconstruction & bad bad 2.4.3
Quoted 9 lines · linas
First problem: In kernel-2.4.2 and earlier, if the machine is not cleanly shut down, then upon reboot, RAID reconstruction is automatically started. (For RAID-1, this more-or-less ammounts to copying the entire contents
of one disk partition on one disk to another). The reconstruction
code seems to be clever: it will try to use the full bandwidth when the system is idle, and it will throttle back when busy. It will only throttle back so far: it tries to maintain at least a minimum amount of work going, in order to gaurentee forward progress even on a busy system.
And it works great!
Quoted 5 lines
The problem: this dramatically slows fsck after an unclean shut-down. You can hear the drives machine-gunning. I haven't stop-watch timed it, but its on the order of 5x slower to fsck a raid partition when there's reconstruction going on, then when the raid thinks its clean. This makes unclean reboots quite painful.
Since the alternative is to sit there and do NOTHING until the reconstruction is complete, ala Solaris 2.5, it's WONDERFUL the way it is. This change was extensively discussed on the raid mailing list a couple of years ago. You can look it up for review.
Quoted 2 lines
(There is no config file to disable/alter this .. no work-around that I know of ..)
You can't be serious. Go sit down and think about what's going on.
Quoted 7 lines
--------
The second problem: oparallelizing fsck doesn't realize that different/dev/mdraid volumes are on the same physical disks, and thus tries
to parallelize .... again slowing things down. There is a work-around,
modify/etc/fstabto set the rder of fsck's. However, I doubt the HOWTO really gets into this .... it would be nice to get fsck to 'do the right thing'.
You probably have your fstab incorrectly setup.
<snip>
#> In particular, how does fsck deal with md devices? It parallelizes #> itself for multiple disks, but if the volumes are all actually striped #> over the same disks, fsck will perform better if it's serial. #
#The "pass" field in /etc/fstab is for exactly this: fsck -a will #serialise devices with different pass numbers. Pass==1 is for root, #pass==2 is for normal devices which fsck knows how to serialise. If you #want to force serialisation on md devices, use larger pass numbers. </snip>
Do a little work, it won't hurt you. Fsck should not (and may not be able to) decode metadevice structures.
Your third part was ignored, given the above.
Quoted 4 lines · In article <[email protected]> you wrote:
Quoted 2 lines
(There is no config file to disable/alter this .. no work-around that I know of ..)
You can't be serious. Go sit down and think about what's going on.
Well, there are two potential solutions:
a) stop rebuild until fsck is fixed
b) wait with fsck until rebuild is fixed
Both of them are valid. The first one is valid in a scenario where you want to decrease downtimes in favor of insecure operation/or multiple redundancy
The second one is good if you prefer data consitency over small down times. It might actually speed up the bootup process, one has to measure this.
Greetings
Bernd
Quoted 9 lines · Bernd Eckenfels
Quoted 4 lines · In article <[email protected]> you wrote:
Quoted 2 lines
(There is no config file to disable/alter this .. no work-around that I know of ..)
You can't be serious. Go sit down and think about what's going on.
Well, there are two potential solutions:
a) stop rebuild until fsck is fixed
And let fsck read bad data because the raid doesn't yet recognize the correct one....
There is nothing to fix in fsck. It should NOT know about the low level block storage devices. If it does, then fsck for EACH filesystem will have to know about ALL different raid hardware/software implementations.
Quoted 1 line
b) wait with fsck until rebuild is fixed
Depends on your definition of "fixed". The most I can see to fix is reduce the amount of continued update in favor of updating those blocks being read (by fsck or anything else). This really ought to be a runtime configuration option. If it is set to 0, then no automatic repair would be done.
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]
Any opinions expressed are solely my own.
Quoted 7 lines · Jesse Pollard
Quoted 1 line
b) wait with fsck until rebuild is fixed
Depends on your definition of "fixed". The most I can see to fix is reduce the amount of continued update in favor of updating those blocks being read (by fsck or anything else). This really ought to be a runtime configuration option. If it is set to 0, then no automatic repair would be done.
The original post was referring to RAID 1; there's no repair necessary at the RAID level to give fsck the correct data. Seems to me the basic problem here is that the RAID re-sync is supposed to be throttling back to allow other activity to run, but that in the case of fsck the other activity is still slower by a large factor (compared to no RAID re-sync).
Is this a pathological case because of the way fsck does business, or does the RAID re-sync affect any disk-bound process that severely?
Signature · 1 line
/Jonathan Lundell.
Quoted 9 lines · Jonathan Lundell
Quoted 7 lines · Jesse Pollard
Quoted 1 line
b) wait with fsck until rebuild is fixed
Depends on your definition of "fixed". The most I can see to fix is reduce the amount of continued update in favor of updating those blocks being read (by fsck or anything else). This really ought to be a runtime configuration option. If it is set to 0, then no automatic repair would be done.
Quoted 5 lines
The original post was referring to RAID 1; there's no repair necessary at the RAID level to give fsck the correct data. Seems to me the basic problem here is that the RAID re-sync is supposed to be throttling back to allow other activity to run, but that in the case of fsck the other activity is still slower by a large factor (compared to no RAID re-sync).
If I've got the numbering right;
0 - concatenated stripes => no sync required
1 - mirrored => resync required
a: which drive has the correct info?
b: having determined that, read the correct block,
it must now be written to the mirror.
all others => resync required (rebuild possible bad block)
Quoted 2 lines
Is this a pathological case because of the way fsck does business, or does the RAID re-sync affect any disk-bound process that severely?
My experience has been with hardware raid, and even then there has been a 1-5% decrease in I/O during resync (not accurately measured - fsck took longer, and then only when the channel is maxed out -- otherwise the 1-5% is not visible; filesystem was 3 IRIX efs, spread across two raid luns).
fsck is particularly bad, since nearly every read instigates a write to the mirror drive. Fsck can then modify the block and write it back, causing two more writes; for a total of 3 writes for a read (worst case).
It does mean that when fsck finishes, MOST of the re-sync will be finished. All of the metadata will be synced, and only file data blocks will remain.
Signature · 5 lines
------------------------------------------------------------------------- Jesse I Pollard, II Email: [email protected] Any opinions expressed are solely my own.
Quoted 4 lines · In article <01041521302600.15046@tabby> you wrote:
Quoted 1 line
a) stop rebuild until fsck is fixed
And let fsck read bad data because the raid doesn't yet recognize the correct one....
a degraded raid will not deliver broken data. and even if it does, one more reason not to check a degraded raid.
Quoted 3 lines
There is nothing to fix in fsck. It should NOT know about the low level block storage devices. If it does, then fsck for EACH filesystem will have to know about ALL different raid hardware/software implementations.
fsck does not neet to be changed, yoi can have a shell script loop and check the raid state before caling the fsck.
Quoted 3 lines
Quoted 1 line
b) wait with fsck until rebuild is fixed
Depends on your definition of "fixed"
fixed as in rebuild, thats what we where tlking about, no?
. The most I can see to fix is
Quoted 4 lines
reduce the amount of continued update in favor of updating those blocks being read (by fsck or anything else). This really ought to be a runtime configuration option. If it is set to 0, then no automatic repair would be done.
yes would be a nice feature if rebuild can be made to only to io which is required by the kernel anyway. since fsck will reach a lot of meta data this is a fairly good start for a slow rebuild.
Greetings
Bernd
On Sun, Apr 15, 2001 at 09:51:52PM -0500, Jesse Pollard wrote:
...
Quoted 5 lines
If I've got the numbering right;
0 - concatenated stripes => no sync required 1 - mirrored => resync required
a: which drive has the correct info?
a: there are timestamps in the superblocks. one disk is simply chosen as
the "newest", and the other disks are updateed from that one.
Quoted 3 lines
b: having determined that, read the correct block,
it must now be written to the mirror.
all others => resync required (rebuild possible bad block)
If a resync is required, it is because we can derive the correct information *without* having the array in sync (resync puts correct information in places
where we know it's not currently correct). fsck can never see "old" information,
no matter the state of the resync. If it could, RAID would be broken.
Levels 1, 4 and 5 need a resync - but they all give you the correct information even *before* resync.
Levels 0 and linear cannot be synced (and will fail if you pull a disk, all because
they do not have redundant information). Talking about sync in these cases doesn't
make sense.
Quoted 8 lines
Quoted 3 lines
Is this a pathological case because of the way fsck does business, or does the RAID re-sync affect any disk-bound process that severely?
My experience has been with hardware raid, and even then there has been a 1-5% decrease in I/O during resync (not accurately measured - fsck took longer, and then only when the channel is maxed out -- otherwise the 1-5% is not visible; filesystem was 3 IRIX efs, spread across two raid luns).
I think the problem is that RAID throttles based on bandwidth, not based on requests. If fsck needs a lot of seeking, the RAID code won't notice that the array is being used much, and thus won't throttle a lot.
Also, even if fsck does large sequential reads, the RAID code may throttle, but it will then introduce small frequent seeks to when it updates a few blocks every now and then. Seeks have a huge impact on otherwise sequential reads.
Quoted 4 lines
fsck is particularly bad, since nearly every read instigates a write to the mirror drive. Fsck can then modify the block and write it back, causing two more writes; for a total of 3 writes for a read (worst case).
Oh, do we resync blocks that are read ? I didn't know that actually...
This should give little overhead on RAID-1, since most reads would be done on one disk and the "older disk" (the one being synced with information from the newer one) would only be written to.
Quoted 4 lines
It does mean that when fsck finishes, MOST of the re-sync will be finished. All of the metadata will be synced, and only file data blocks will remain.
If we resync RAID-1 blocks as they are read, I don't understand the claimed performance impact of resync.
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: