Mirror resync direction
I'm running kernel 2.6.17 using the lvm2 package currently in debian testing (2.02.06-3). I use it to set up a VG containing two PVs, /dev/sda2 and /dev/sdb2. I create a mirror LV using corelog:
lvcreate --mirror 1 --corelog ...
I create a filesystem on it, and populate it with files. I then try testing how it copes with drive failures, by simulating a replaced disk.
To do this, I zero out one of the sides using
dd if=/dev/zero of=/dev/sda2
Having done that, I do the usual restore method
pvcreate --uuid 'whatever' /dev/sda2 vgcfgrestore vg vgchange -a y
And I note that it starts to resync the mirror. But it goes in the "wrong" direction - sda2's PV is the master, but that's been zeroed out. sdb2's is the slave but contains the real data. The block copying goes in the wrong direction here, copying the master's zeroes over the good data in the slave.
Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?
Signature · 5 lines
Paul "LeoNerd" Evans [email protected] ICQ# 4135350 | Registered Linux# 179460 http://www.leonerd.org.uk/
Quoted 5 lines · Paul LeoNerd Evans
testing (2.02.06-3). I use it to set up a VG containing two
Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?
Firstly, this is work in progress, so you should be using newer tools. Secondly, with newer tools you should follow a different
procedure: use vgreduce --removemissing (carefully) to remove the lost device, add the replacement disk to the VG in the normal way, then use lvconvert to turn the LV back into a mirror.
Alasdair
Signature · 1 line
[email protected]
Quoted 5 lines · Alasdair G Kergon
Firstly, this is work in progress, so you should be using newer tools. Secondly, with newer tools you should follow a different
procedure: use vgreduce --removemissing (carefully) to remove the lost device, add the replacement disk to the VG in the normal way, then use lvconvert to turn the LV back into a mirror.
And I should add, dmeventd will do these steps automatically on failure if you configure it into the lvm2 and device-mapper packages and apply the right kernel patches (they're nearly ready for upstream submission).
Alasdair
Signature · 1 line
[email protected]
Quoted 7 lines · On Thu, 26 Oct 2006 21:27:02 +0100 Alasdair G Kergon <[email protected]> wrote:
Quoted 5 lines · Paul LeoNerd Evans
testing (2.02.06-3). I use it to set up a VG containing two
Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?
Quoted 5 lines
Firstly, this is work in progress, so you should be using newer tools. Secondly, with newer tools you should follow a different
procedure: use vgreduce --removemissing (carefully) to remove the lost device, add the replacement disk to the VG in the normal way, then use lvconvert to turn the LV back into a mirror.
Thanks for the quick reply :)
I've looked at my copy of vgreduce, and that has the option you named. I tried that and it did as I suspected, namely converted the mirror back to a plain linear LV, that I can mount as normal and use. Then running pvcreate / vgextend / lvconvert --mirror 1, and now that works fine.
Thanks for the help.
Quoted 3 lines
And I should add, dmeventd will do these steps automatically on failure if you configure it into the lvm2 and device-mapper packages and apply the right kernel patches (they're nearly ready for upstream submission).
Will that be a separate userland program? Presumably it'll appear in a debian package of the same name at some point.
Signature · 5 lines
Paul "LeoNerd" Evans [email protected] ICQ# 4135350 | Registered Linux# 179460 http://www.leonerd.org.uk/
Quoted 8 lines · Paul LeoNerd Evans
Quoted 4 lines
And I should add, dmeventd will do these steps automatically on failure if you configure it into the lvm2 and device-mapper packages and apply the right kernel patches (they're nearly ready for upstream submission).
Will that be a separate userland program? Presumably it'll appear in a debian package of the same name at some point.
dmeventd will be a separate userland binary, but will be part of the device-mapper package.
brassow
Quoted 7 lines · On Thu, 26 Oct 2006 21:27:02 +0100 Alasdair G Kergon <[email protected]> wrote:
Quoted 5 lines · Paul LeoNerd Evans
testing (2.02.06-3). I use it to set up a VG containing two
Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?
Quoted 5 lines
Firstly, this is work in progress, so you should be using newer tools. Secondly, with newer tools you should follow a different
procedure: use vgreduce --removemissing (carefully) to remove the lost device, add the replacement disk to the VG in the normal way, then use lvconvert to turn the LV back into a mirror.
Thanks for the quick reply :)
I've looked at my copy of vgreduce, and that has the option you named. I tried that and it did as I suspected, namely converted the mirror back to a plain linear LV, that I can mount as normal and use. Then running pvcreate / vgextend / lvconvert --mirror 1, and now that works fine.
Thanks for the help.
Quoted 3 lines
And I should add, dmeventd will do these steps automatically on failure if you configure it into the lvm2 and device-mapper packages and apply the right kernel patches (they're nearly ready for upstream submission).
Will that be a separate userland program? Presumably it'll appear in a debian package of the same name at some point.
Signature · 13 lines
Paul "LeoNerd" Evans [email protected] ICQ# 4135350 | Registered Linux# 179460 http://www.leonerd.org.uk/ -- Paul "LeoNerd" Evans [email protected] ICQ# 4135350 | Registered Linux# 179460 http://www.leonerd.org.uk/
Quoted 4 lines · Paul LeoNerd Evans
I'm running kernel 2.6.17 using the lvm2 package currently in debian testing (2.02.06-3). I use it to set up a VG containing two PVs,
/dev/sda2and/dev/sdb2. I create a mirror LV using corelog:
Quoted 8 lines
lvcreate --mirror 1 --corelog ...
I create a filesystem on it, and populate it with files. I then try testing how it copes with drive failures, by simulating a replaced disk.
To do this, I zero out one of the sides using
dd if=/dev/zero of=/dev/sda2
This doesn't really reflect what would happen if you replaced a disk, I don't think. With this, you are simply going around mirror's back to write data to a device which it controls.
In real life, I think you would have a disk failure (which would force you to run 'vgreduce --removemissing vg'), then you would insert a new block device, pvcreate/vgextend/lvconvert.
(When dmeventd is properly running, it should take care of the steps that are currently manual.)
Quoted 11 lines
Having done that, I do the usual restore method
pvcreate --uuid 'whatever' /dev/sda2 vgcfgrestore vg vgchange -a yAnd I note that it starts to resync the mirror. But it goes in the "wrong" direction - sda2's PV is the master, but that's been zeroed out. sdb2's is the slave but contains the real data. The block copying goes in the wrong direction here, copying the master's zeroes over the good data in the slave.
Again, mirror has no idea that you did this. The act of dd'ing to an underlying device is like dd'ing to a device under a file system - it doesn't know you are doing it and causes corruption.
Quoted 4 lines
Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?
LVM2 mirroring is still a work in progress (trying to get the right pieces upstream). Depending on the version of LVM2 and the kernel patches you have, it may not work.
The use of --corelog simply means that the mirror device will be resynchronized every time the device is activated. The persistent log (disk log) tracks what resynchronization has been done and can avoid all complete resyncs.
brassow
Quoted 3 lines · On Thu, 26 Oct 2006 16:14:34 -0500 Jonathan E Brassow <[email protected]>…
This doesn't really reflect what would happen if you replaced a disk, I don't think. With this, you are simply going around mirror's back to write data to a device which it controls.
Well, I did it with the VG turned off; namely
vgchange -a n
Quoted 3 lines
In real life, I think you would have a disk failure (which would force you to run 'vgreduce --removemissing vg'), then you would insert a new block device, pvcreate/vgextend/lvconvert.
Yes, I wasn't aware of that procedure before, but Alasdair Kergon's mail has pointed that out to me - I've tried with that and it seems to work fine.
Quoted 3 lines
LVM2 mirroring is still a work in progress (trying to get the right pieces upstream). Depending on the version of LVM2 and the kernel patches you have, it may not work.
Seems happy once I follow the correct procedure - see above :)
Quoted 4 lines
The use of --corelog simply means that the mirror device will be resynchronized every time the device is activated. The persistent log (disk log) tracks what resynchronization has been done and can avoid all complete resyncs.
Yes, I'm aware of that bit, but I couldn't see any other way to achieve it with only two PVs. The "--alloc anywhere" as suggested in the error message doesn't have any effect for me.
Signature · 5 lines
Paul "LeoNerd" Evans [email protected] ICQ# 4135350 | Registered Linux# 179460 http://www.leonerd.org.uk/
Quoted 9 lines · Paul LeoNerd Evans
Quoted 4 lines
The use of --corelog simply means that the mirror device will be resynchronized every time the device is activated. The persistent log (disk log) tracks what resynchronization has been done and can avoid all complete resyncs.
Yes, I'm aware of that bit, but I couldn't see any other way to achieve it with only two PVs. The "--alloc anywhere" as suggested in the error message doesn't have any effect for me.
Jun'ichi Nomura from NEC is working on the allocation issues - it's a bit buggy WRT mirrors.
However, you could use a known bug to get around having to use --corelog when you only have two devices.
1) lvcreate -m1 --corelog -L <don't use all the space> -n mirror vg
2) lvconvert -m1 vg/mirror
The convert doesn't currently honor the allocation schemes very well right now either. It will simply put the disk log on one of the two devices.
brassow