Skip to contentSkip to conversation

Simulating LVM Mirror Failure and recovery

linux-lvm yogeen honnavar 10 participants 13 messages Apr 16, 2009

lore
Share & tools
lore
#0 lore raw

Dear users,

We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.

now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1

dd if=/dev/zero of=/dev/vg1/lv1 count=10

as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

thanks and regards
-yogeen

Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/

#1 lore raw
Quoted 6 lines · yogeen honnavar

Dear users,

We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.

now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1

Quoted 8 lines

dd if=/dev/zero of=/dev/vg1/lv1 count=10

as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

thanks and regards
-yogeen

Hi yogeen,

i guess you wanted to do a 'dd if=/dev/zero of=/dev/sdb3 count=10', with your command, you filled the logical volume with zeros and that is mirrored perfectly on both underlying devices - as it should ;-).

Signature · 8 lines
Mit freundlichen Grüssen / best regards

Klaus Strebel, Dipl.-Inform. (FH), mailto:[email protected]

/"\
\ /     ASCII RIBBON CAMPAIGN
 X        AGAINST HTML MAIL
/ \
#2 lore raw
In reply to yogeen honnavar
Quoted 6 lines · yogeen honnavar

Dear users,

We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.

now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1

Quoted 7 lines

dd if=/dev/zero of=/dev/vg1/lv1 count=10

as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

thanks and regards

I think to simulate error failure you have to write to the underlying devices, not the mirrored device. When you write to the mirrored device, lvl1 in this case, data is mirrored so there is no failure at all. So I think you have to dd to either /dev/sda3 or /dev/sdb3.

Vu

#3 lore raw
Quoted 4 lines · vu pham
Quoted 3 lines · yogeen honnavar

Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any

Can you provide the reference - where does it say this?

Alasdair

Signature · 1 line
[email protected]
#4 lore raw
Quoted 4 lines · Alasdair G Kergon
Quoted 2 lines · vu pham

yogeen honnavar wrote:

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any

Can you provide the reference - where does it say this?

Alasdair

I believe this is the reference.

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html

Their example may be in error.

#5 lore raw
Quoted 9 lines · Eric Brunson
Quoted 8 lines · Alasdair G Kergon
Quoted 4 lines · vu pham
Quoted 3 lines · yogeen honnavar

Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any

Can you provide the reference - where does it say this?

Quoted 10 lines
Quoted 1 line

Alasdair

I believe this is the reference.

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html

Their example may be in error.

Upon review, I think it's correct, but worded awkwardly:

In this example, the primary leg of the mirror |/dev/sda1| fails. Any write activity to the mirrored volume causes LVM to detect the failed mirror. When this occurs, LVM converts the mirror into a single linear volume. In this case, to trigger the conversion, we execute a |dd| command

[root@link-08 ~]#*|dd if=/dev/zero of=/dev/vg/groupfs count=10|* 10+0 records in
10+0 records out

I believe the dd is not to cause the failure, it's simply to generate write activity to the volume group after the mirror leg has been caused to fail through other means not documented.

Anyone else think that's the case?

e.

Quoted 5 lines

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#6 lore raw
Quoted 1 line · Eric Brunson

Upon review, I think it's correct, but worded awkwardly:

Maybe it's because I'm not a native speaker, but I find it difficult to understand this differently.

Quoted 1 line

In this example, the primary leg of the mirror |/dev/sda1| fails.

It fails. Not "we make it fail". No indication what makes it fail.

Quoted 4 lines

Any write activity to the mirrored volume causes LVM to detect the failed mirror. When this occurs, LVM converts the mirror into a single linear volume. In this case, to trigger the conversion, we execute a |dd| command

"we execute a |dd| command" "to trigger the *conversion*". Not to make it fail.

Quoted 3 lines

I believe the dd is not to cause the failure, it's simply to generate write activity to the volume group after the mirror leg has been caused to fail through other means not documented.

... as they said ...

So the dd command the OP sent was correct after all. He just misunderstood its purpose.

Now back to our original topic - how do you make a mirror copy fail? ;-)

Lupe Christoph

Signature · 2 lines
| There is no substitute for bad design except worse design.                   |
| /me                                                                          |
#7 lore raw
In reply to Eric Brunson

Hey Eric

Nice to see your still around. How's it all going?

best,
Don B

Quoted 49 lines · Eric Brunson
Quoted 10 lines · Eric Brunson
Quoted 9 lines · Alasdair G Kergon
Quoted 5 lines · vu pham
Quoted 4 lines · yogeen honnavar

Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any

Can you provide the reference - where does it say this?

Quoted 10 lines
Quoted 1 line

Alasdair

I believe this is the reference.

http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Cluster_Logical_Volume_Manager/mirrorrecover.html

Their example may be in error.

Upon review, I think it's correct, but worded awkwardly:

In this example, the primary leg of the mirror |/dev/sda1| fails. Any write activity to the mirrored volume causes LVM to detect the failed mirror. When this occurs, LVM converts the mirror into a single linear volume. In this case, to trigger the conversion, we execute a |dd| command

[root@link-08 ~]#*|dd if=/dev/zero of=/dev/vg/groupfs count=10|* 10+0 records in
10+0 records out

I believe the dd is not to cause the failure, it's simply to generate write activity to the volume group after the mirror leg has been caused to fail through other means not documented.

Anyone else think that's the case?

e.

Quoted 5 lines

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Quoted 7 lines

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

CONFIDENTIALITY NOTICE
This electronic mail transmission and any accompanying documents contain information belonging to the sender that may be confidential and legally privileged. This information is intended only for the use of the individual or entity to whom this electronic mail transmission was sent as indicated above. If you are not the intended recipient, any disclosure, copying, distribution, or action taken in reliance on the contents of the information contained in this transmissions is strictly prohibited. If you have received this transmission in error, please reply by e-mail to let me know and then permanently delete the message. Thank you.

#8 lore raw
In reply to vu pham
Quoted 7 lines · vu pham
Quoted 5 lines

as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

thanks and regards

I think to simulate error failure you have to write to the underlying devices, not the mirrored device. When you write to the mirrored device, lvl1 in this case, data is mirrored so there is no failure at all. So I think you have to dd to either /dev/sda3 or /dev/sdb3.

How will LVM distinguish between a valid and invalid copy in that case?

Are you sure that on reads, LVM will return data from the "good" copy, not the one you just overwrote with "dd if=/dev/zero of=/dev/sdb seek=xxx bs=1M count=10"?

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#9 lore raw
Quoted 6 lines · Tomasz Chmielewski
Quoted 4 lines · vu pham
Quoted 4 lines

as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

thanks and regards

I think to simulate error failure you have to write to the underlying devices, not the mirrored device. When you write to the mirrored device, lvl1 in this case, data is mirrored so there is no failure at all. So I think you have to dd to either /dev/sda3 or /dev/sdb3.

How will LVM distinguish between a valid and invalid copy in that case?

Are you sure that on reads, LVM will return data from the "good" copy, not the one you just overwrote with "dd if=/dev/zero of=/dev/sdb seek=xxx bs=1M count=10"?

Good question. I thought that dd will destroy the lvm information at the beginning of the partition which makes the leg broken. I made the following test and the result is odd:

- created mirrored logical volumn mrlv1 on /dev/sdc1 and /dev/sdd1, having /dev/sdd2 as log device
- mkfs.ext3 /dev/vg1/mrlv1, mount it and copy files onto it.
- dd if=/dev/zero of=/dev/sdd1
- read / write onto the mounted mirror lvm.

The output of lvs shows:
- lvm information on /dev/sdd2 is destroyed
- the 2nd mirror, which is /dev/sdd2, is *unknown device*
- the mirror is still ok

And no lvm log in /var/log/messages.

[root@xen3 ~]# lvs -a -o +devices
Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
LV VG Attr LSize Origin Snap% Move Log
[... deleted unrelated devices ...]

mrlv1 vg1 mwi-ao 200.00M mrlv1_mlog
100.00 mrlv1_mimage_0(0),mrlv1_mimage_1(0)
[mrlv1_mimage_0] vg1 iwi-ao 200.00M

             /dev/sdc1(0)
  [mrlv1_mimage_1] vg1  iwi-ao 200.00M 
             unknown device(0)
  [mrlv1_mlog]     vg1  lwi-ao   4.00M 
             /dev/sdd2(0)

Vu

#10 lore raw
In reply to yogeen honnavar
Quoted 3 lines · yogeen honnavar

dd if=/dev/zero of=/dev/vg1/lv1 count=10

as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

Apart from the problem with the of= parameter that was already pointed out - why do you assume writing *anything* *anywhere* would simulate a mirror failure? The most you do simulate with that is mirror *corruption*.

To simulate a failure, you have to somehow disable a drive. Unplug a cable, force it to spin down (assuming that it is not automatically spun up again, I dunno), change the zoning, whatever is needed to prevent access to the device.

Quoted 1 line

can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

Depends a lot on your hardware. E.g. the system I use for testing are about 400 km away, so I'd rather not unplug anything...

HTH,
Lupe Christoph

Signature · 2 lines
| There is no substitute for bad design except worse design.                   |
| /me                                                                          |
#11 lore raw
In reply to yogeen honnavar
Quoted 5 lines · Lupe Christoph [[email protected]] wrote:
Quoted 3 lines · yogeen honnavar

dd if=/dev/zero of=/dev/vg1/lv1 count=10

as per the redhat guide this should result in mirror leg failure and any

subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.

Quoted 5 lines

Apart from the problem with the of= parameter that was already pointed out - why do you assume writing *anything* *anywhere* would simulate a mirror failure? The most you do simulate with that is mirror *corruption*.

You are absolutely right.

Quoted 5 lines
Quoted 2 lines

can anyone please let us know the correct way to simulate lvm mirror failure and recovery.

Depends a lot on your hardware. E.g. the system I use for testing are about 400 km away, so I'd rather not unplug anything...

I usually run "echo 1 > /sys/block/sda/device/delete" to make /dev/sda disappear and fail for LVM tests. You can fail the disk in a lot of ways depending on your hardware, but this trick works with almost every device!

#12 lore raw
Quoted 4 lines · [email protected] wrote:

I usually run "echo 1 > /sys/block/sda/device/delete" to make /dev/sda disappear and fail for LVM tests. You can fail the disk in a lot of ways depending on your hardware, but this trick works with almost every device!

# echo offline > /sys/block/sda/device/state

might work as well. In this case, you can recover the failure by

# echo running > /sys/block/sda/device/state

Thanks,
Taka