Skip to contentSkip to conversation

Mirror resync direction

linux-lvm Paul LeoNerd Evans 4 participants 9 messages Oct 26, 2006 – Nov 1, 2006

lore
Share & tools
lore
#0 lore raw

I'm running kernel 2.6.17 using the lvm2 package currently in debian testing (2.02.06-3). I use it to set up a VG containing two PVs, /dev/sda2 and /dev/sdb2. I create a mirror LV using corelog:

lvcreate --mirror 1 --corelog ...

I create a filesystem on it, and populate it with files. I then try testing how it copes with drive failures, by simulating a replaced disk.

To do this, I zero out one of the sides using

dd if=/dev/zero of=/dev/sda2

Having done that, I do the usual restore method

  pvcreate --uuid 'whatever' /dev/sda2
  vgcfgrestore vg
  vgchange -a y

And I note that it starts to resync the mirror. But it goes in the "wrong" direction - sda2's PV is the master, but that's been zeroed out. sdb2's is the slave but contains the real data. The block copying goes in the wrong direction here, copying the master's zeroes over the good data in the slave.

Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?

Signature · 5 lines
Paul "LeoNerd" Evans

[email protected]
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/
#1 lore raw
Quoted 5 lines · Paul LeoNerd Evans

testing (2.02.06-3). I use it to set up a VG containing two

Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?

Firstly, this is work in progress, so you should be using newer tools. Secondly, with newer tools you should follow a different
procedure: use vgreduce --removemissing (carefully) to remove the lost device, add the replacement disk to the VG in the normal way, then use lvconvert to turn the LV back into a mirror.

Alasdair

Signature · 1 line
[email protected]
#2 lore raw
Quoted 5 lines · Alasdair G Kergon

Firstly, this is work in progress, so you should be using newer tools. Secondly, with newer tools you should follow a different
procedure: use vgreduce --removemissing (carefully) to remove the lost device, add the replacement disk to the VG in the normal way, then use lvconvert to turn the LV back into a mirror.

And I should add, dmeventd will do these steps automatically on failure if you configure it into the lvm2 and device-mapper packages and apply the right kernel patches (they're nearly ready for upstream submission).

Alasdair

Signature · 1 line
[email protected]
#3 lore raw
In reply to Alasdair G Kergon
Quoted 7 lines · On Thu, 26 Oct 2006 21:27:02 +0100 Alasdair G Kergon <[email protected]> wrote:
Quoted 5 lines · Paul LeoNerd Evans

testing (2.02.06-3). I use it to set up a VG containing two

Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?

Quoted 5 lines

Firstly, this is work in progress, so you should be using newer tools. Secondly, with newer tools you should follow a different
procedure: use vgreduce --removemissing (carefully) to remove the lost device, add the replacement disk to the VG in the normal way, then use lvconvert to turn the LV back into a mirror.

Thanks for the quick reply :)

I've looked at my copy of vgreduce, and that has the option you named. I tried that and it did as I suspected, namely converted the mirror back to a plain linear LV, that I can mount as normal and use. Then running pvcreate / vgextend / lvconvert --mirror 1, and now that works fine.

Thanks for the help.

Quoted 3 lines

And I should add, dmeventd will do these steps automatically on failure if you configure it into the lvm2 and device-mapper packages and apply the right kernel patches (they're nearly ready for upstream submission).

Will that be a separate userland program? Presumably it'll appear in a debian package of the same name at some point.

Signature · 5 lines
Paul "LeoNerd" Evans

[email protected]
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/
#4 lore raw
Quoted 8 lines · Paul LeoNerd Evans
Quoted 4 lines

And I should add, dmeventd will do these steps automatically on failure if you configure it into the lvm2 and device-mapper packages and apply the right kernel patches (they're nearly ready for upstream submission).

Will that be a separate userland program? Presumably it'll appear in a debian package of the same name at some point.

dmeventd will be a separate userland binary, but will be part of the device-mapper package.

brassow

#5 lore raw
In reply to Alasdair G Kergon
Quoted 7 lines · On Thu, 26 Oct 2006 21:27:02 +0100 Alasdair G Kergon <[email protected]> wrote:
Quoted 5 lines · Paul LeoNerd Evans

testing (2.02.06-3). I use it to set up a VG containing two

Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?

Quoted 5 lines

Firstly, this is work in progress, so you should be using newer tools. Secondly, with newer tools you should follow a different
procedure: use vgreduce --removemissing (carefully) to remove the lost device, add the replacement disk to the VG in the normal way, then use lvconvert to turn the LV back into a mirror.

Thanks for the quick reply :)

I've looked at my copy of vgreduce, and that has the option you named. I tried that and it did as I suspected, namely converted the mirror back to a plain linear LV, that I can mount as normal and use. Then running pvcreate / vgextend / lvconvert --mirror 1, and now that works fine.

Thanks for the help.

Quoted 3 lines

And I should add, dmeventd will do these steps automatically on failure if you configure it into the lvm2 and device-mapper packages and apply the right kernel patches (they're nearly ready for upstream submission).

Will that be a separate userland program? Presumably it'll appear in a debian package of the same name at some point.

Signature · 13 lines
Paul "LeoNerd" Evans

[email protected]
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/


-- 
Paul "LeoNerd" Evans

[email protected]
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/
#6 lore raw
In reply to Paul LeoNerd Evans
Quoted 4 lines · Paul LeoNerd Evans

I'm running kernel 2.6.17 using the lvm2 package currently in debian testing (2.02.06-3). I use it to set up a VG containing two PVs, /dev/sda2 and /dev/sdb2. I create a mirror LV using corelog:

Quoted 8 lines

lvcreate --mirror 1 --corelog ...

I create a filesystem on it, and populate it with files. I then try testing how it copes with drive failures, by simulating a replaced disk.

To do this, I zero out one of the sides using

dd if=/dev/zero of=/dev/sda2

This doesn't really reflect what would happen if you replaced a disk, I don't think. With this, you are simply going around mirror's back to write data to a device which it controls.

In real life, I think you would have a disk failure (which would force you to run 'vgreduce --removemissing vg'), then you would insert a new block device, pvcreate/vgextend/lvconvert.

(When dmeventd is properly running, it should take care of the steps that are currently manual.)

Quoted 11 lines

Having done that, I do the usual restore method

  pvcreate --uuid 'whatever' /dev/sda2
  vgcfgrestore vg
  vgchange -a y

And I note that it starts to resync the mirror. But it goes in the "wrong" direction - sda2's PV is the master, but that's been zeroed out. sdb2's is the slave but contains the real data. The block copying goes in the wrong direction here, copying the master's zeroes over the good data in the slave.

Again, mirror has no idea that you did this. The act of dd'ing to an underlying device is like dd'ing to a device under a file system - it doesn't know you are doing it and causes corruption.

Quoted 4 lines

Is this a bug in the LVM2 tools, or the dm-mirror target, or what? Is it perhaps a limitation of --corelog; in which case, how can I protect against that?

LVM2 mirroring is still a work in progress (trying to get the right pieces upstream). Depending on the version of LVM2 and the kernel patches you have, it may not work.

The use of --corelog simply means that the mirror device will be resynchronized every time the device is activated. The persistent log (disk log) tracks what resynchronization has been done and can avoid all complete resyncs.

brassow

#7 lore raw
Quoted 3 lines · On Thu, 26 Oct 2006 16:14:34 -0500 Jonathan E Brassow <[email protected]>…

This doesn't really reflect what would happen if you replaced a disk, I don't think. With this, you are simply going around mirror's back to write data to a device which it controls.

Well, I did it with the VG turned off; namely

vgchange -a n

Quoted 3 lines

In real life, I think you would have a disk failure (which would force you to run 'vgreduce --removemissing vg'), then you would insert a new block device, pvcreate/vgextend/lvconvert.

Yes, I wasn't aware of that procedure before, but Alasdair Kergon's mail has pointed that out to me - I've tried with that and it seems to work fine.

Quoted 3 lines

LVM2 mirroring is still a work in progress (trying to get the right pieces upstream). Depending on the version of LVM2 and the kernel patches you have, it may not work.

Seems happy once I follow the correct procedure - see above :)

Quoted 4 lines

The use of --corelog simply means that the mirror device will be resynchronized every time the device is activated. The persistent log (disk log) tracks what resynchronization has been done and can avoid all complete resyncs.

Yes, I'm aware of that bit, but I couldn't see any other way to achieve it with only two PVs. The "--alloc anywhere" as suggested in the error message doesn't have any effect for me.

Signature · 5 lines
Paul "LeoNerd" Evans

[email protected]
ICQ# 4135350       |  Registered Linux# 179460
http://www.leonerd.org.uk/
#8 lore raw
Quoted 9 lines · Paul LeoNerd Evans
Quoted 4 lines

The use of --corelog simply means that the mirror device will be resynchronized every time the device is activated. The persistent log (disk log) tracks what resynchronization has been done and can avoid all complete resyncs.

Yes, I'm aware of that bit, but I couldn't see any other way to achieve it with only two PVs. The "--alloc anywhere" as suggested in the error message doesn't have any effect for me.

Jun'ichi Nomura from NEC is working on the allocation issues - it's a bit buggy WRT mirrors.

However, you could use a known bug to get around having to use --corelog when you only have two devices.

1) lvcreate -m1 --corelog -L <don't use all the space> -n mirror vg
2) lvconvert -m1 vg/mirror

The convert doesn't currently honor the allocation schemes very well right now either. It will simply put the disk log on one of the two devices.

brassow