Simulating LVM Mirror Failure and recovery
Dear users,
We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks /dev/sda and /dev/sdb. output of lvs command shows that mirror image_0 is stored on /dev/sda3 and image_1 on /dev/sdb3 and mirror log on /dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.
now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
thanks and regards
-yogeen
Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
Quoted 6 lines · yogeen honnavar
Dear users,
We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks
/dev/sdaand/dev/sdb. output of lvs command shows that mirrorimage_0is stored on/dev/sda3andimage_1on/dev/sdb3and mirror log on/dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1
Quoted 8 lines
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
thanks and regards
-yogeen
Hi yogeen,
i guess you wanted to do a 'dd if=/dev/zero of=/dev/sdb3 count=10', with your command, you filled the logical volume with zeros and that is mirrored perfectly on both underlying devices - as it should ;-).
Signature · 8 lines
Mit freundlichen Grüssen / best regards Klaus Strebel, Dipl.-Inform. (FH), mailto:[email protected] /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Quoted 6 lines · yogeen honnavar
Dear users,
We have created a mirrored logical volume(lv1) in a volume group vg1. we have 2 disks
/dev/sdaand/dev/sdb. output of lvs command shows that mirrorimage_0is stored on/dev/sda3andimage_1on/dev/sdb3and mirror log on/dev/sdb2. Copy % shows 100%. so mirroring is setup correctly.now we wish to simulate mirror leg failure and recovery. as per the redhat lvm administrators guide we used the following command to trigger the failure of mirror leg1
Quoted 7 lines
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
thanks and regards
I think to simulate error failure you have to write to the underlying devices, not the mirrored device. When you write to the mirrored device, lvl1 in this case, data is mirrored so there is no failure at all. So I think you have to dd to either /dev/sda3 or /dev/sdb3.
Vu
Quoted 4 lines · vu pham
Quoted 3 lines · yogeen honnavar
Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any
Can you provide the reference - where does it say this?
Alasdair
Signature · 1 line
[email protected]
Quoted 4 lines · Alasdair G Kergon
Quoted 2 lines · vu pham
yogeen honnavar wrote:
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and anyCan you provide the reference - where does it say this?
Alasdair
I believe this is the reference.
Their example may be in error.
Quoted 9 lines · Eric Brunson
Quoted 8 lines · Alasdair G Kergon
Quoted 4 lines · vu pham
Quoted 3 lines · yogeen honnavar
Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and anyCan you provide the reference - where does it say this?
Quoted 10 lines
Quoted 1 line
Alasdair
I believe this is the reference.
Their example may be in error.
Upon review, I think it's correct, but worded awkwardly:
In this example, the primary leg of the mirror |/dev/sda1| fails. Any write activity to the mirrored volume causes LVM to detect the failed mirror. When this occurs, LVM converts the mirror into a single linear volume. In this case, to trigger the conversion, we execute a |dd| command
[root@link-08 ~]#*|dd if=/dev/zero of=/dev/vg/groupfs count=10|* 10+0 records in
10+0 records out
I believe the dd is not to cause the failure, it's simply to generate write activity to the volume group after the mirror leg has been caused to fail through other means not documented.
Anyone else think that's the case?
e.
Quoted 5 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 1 line · Eric Brunson
Upon review, I think it's correct, but worded awkwardly:
Maybe it's because I'm not a native speaker, but I find it difficult to understand this differently.
Quoted 1 line
In this example, the primary leg of the mirror |/dev/sda1| fails.
It fails. Not "we make it fail". No indication what makes it fail.
Quoted 4 lines
Any write activity to the mirrored volume causes LVM to detect the failed mirror. When this occurs, LVM converts the mirror into a single linear volume. In this case, to trigger the conversion, we execute a |dd| command
"we execute a |dd| command" "to trigger the *conversion*". Not to make it fail.
Quoted 3 lines
I believe the dd is not to cause the failure, it's simply to generate write activity to the volume group after the mirror leg has been caused to fail through other means not documented.
... as they said ...
So the dd command the OP sent was correct after all. He just misunderstood its purpose.
Now back to our original topic - how do you make a mirror copy fail? ;-)
Lupe Christoph
Signature · 2 lines
| There is no substitute for bad design except worse design. | | /me |
Hey Eric
Nice to see your still around. How's it all going?
best,
Don B
Quoted 49 lines · Eric Brunson
Quoted 10 lines · Eric Brunson
Quoted 9 lines · Alasdair G Kergon
Quoted 5 lines · vu pham
Quoted 4 lines · yogeen honnavar
Dear users,
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and anyCan you provide the reference - where does it say this?
Quoted 10 lines
Quoted 1 line
Alasdair
I believe this is the reference.
Their example may be in error.
Upon review, I think it's correct, but worded awkwardly:
In this example, the primary leg of the mirror |/dev/sda1| fails. Any write activity to the mirrored volume causes LVM to detect the failed mirror. When this occurs, LVM converts the mirror into a single linear volume. In this case, to trigger the conversion, we execute a |dd| command
[root@link-08 ~]#*|dd if=/dev/zero of=/dev/vg/groupfs count=10|* 10+0 records in
10+0 records outI believe the dd is not to cause the failure, it's simply to generate write activity to the volume group after the mirror leg has been caused to fail through other means not documented.
Anyone else think that's the case?
e.
Quoted 5 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 7 lines
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
CONFIDENTIALITY NOTICE
This electronic mail transmission and any accompanying documents contain information belonging to the sender that may be confidential and legally privileged. This information is intended only for the use of the individual or entity to whom this electronic mail transmission was sent as indicated above. If you are not the intended recipient, any disclosure, copying, distribution, or action taken in reliance on the contents of the information contained in this transmissions is strictly prohibited. If you have received this transmission in error, please reply by e-mail to let me know and then permanently delete the message. Thank you.
Quoted 7 lines · vu pham
Quoted 5 lines
as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
thanks and regards
I think to simulate error failure you have to write to the underlying devices, not the mirrored device. When you write to the mirrored device, lvl1 in this case, data is mirrored so there is no failure at all. So I think you have to dd to either
/dev/sda3or/dev/sdb3.
How will LVM distinguish between a valid and invalid copy in that case?
Are you sure that on reads, LVM will return data from the "good" copy, not the one you just overwrote with "dd if=/dev/zero of=/dev/sdb seek=xxx bs=1M count=10"?
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 6 lines · Tomasz Chmielewski
Quoted 4 lines · vu pham
Quoted 4 lines
as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
thanks and regards
I think to simulate error failure you have to write to the underlying devices, not the mirrored device. When you write to the mirrored device, lvl1 in this case, data is mirrored so there is no failure at all. So I think you have to dd to either
/dev/sda3or/dev/sdb3.How will LVM distinguish between a valid and invalid copy in that case?
Are you sure that on reads, LVM will return data from the "good" copy, not the one you just overwrote with "dd if=/dev/zero of=/dev/sdb seek=xxx bs=1M count=10"?
Good question. I thought that dd will destroy the lvm information at the beginning of the partition which makes the leg broken. I made the following test and the result is odd:
- created mirrored logical volumn mrlv1 on /dev/sdc1 and /dev/sdd1, having /dev/sdd2 as log device
- mkfs.ext3 /dev/vg1/mrlv1, mount it and copy files onto it.
- dd if=/dev/zero of=/dev/sdd1
- read / write onto the mounted mirror lvm.
The output of lvs shows:
- lvm information on /dev/sdd2 is destroyed
- the 2nd mirror, which is /dev/sdd2, is *unknown device*
- the mirror is still ok
And no lvm log in /var/log/messages.
[root@xen3 ~]# lvs -a -o +devices
Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'. Couldn't find device with uuid 'ZUOCeZ-Ssb2-0fP6-n5YY-sz5n-TtXe-b7eL3a'.
LV VG Attr LSize Origin Snap% Move Log
[... deleted unrelated devices ...]
mrlv1 vg1 mwi-ao 200.00M mrlv1_mlog
100.00 mrlv1_mimage_0(0),mrlv1_mimage_1(0)
[mrlv1_mimage_0] vg1 iwi-ao 200.00M
/dev/sdc1(0)
[mrlv1_mimage_1] vg1 iwi-ao 200.00M
unknown device(0)
[mrlv1_mlog] vg1 lwi-ao 4.00M
/dev/sdd2(0)Vu
Quoted 3 lines · yogeen honnavar
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
Apart from the problem with the of= parameter that was already pointed out - why do you assume writing *anything* *anywhere* would simulate a mirror failure? The most you do simulate with that is mirror *corruption*.
To simulate a failure, you have to somehow disable a drive. Unplug a cable, force it to spin down (assuming that it is not automatically spun up again, I dunno), change the zoning, whatever is needed to prevent access to the device.
Quoted 1 line
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
Depends a lot on your hardware. E.g. the system I use for testing are about 400 km away, so I'd rather not unplug anything...
HTH,
Lupe Christoph
Signature · 2 lines
| There is no substitute for bad design except worse design. | | /me |
Quoted 5 lines · Lupe Christoph [[email protected]] wrote:
Quoted 3 lines · yogeen honnavar
dd if=/dev/zero of=/dev/vg1/lv1 count=10
as per the redhat guide this should result in mirror leg failure and any
subsequent write activity to the mirror should convert the mirror into linear volume. but the output of lvs command still shows copy % as 100%. also data in filesystem of lv1 is erased and it is converted into read only filesystem.
Quoted 5 lines
Apart from the problem with the of= parameter that was already pointed out - why do you assume writing *anything* *anywhere* would simulate a mirror failure? The most you do simulate with that is mirror *corruption*.
You are absolutely right.
Quoted 5 lines
Quoted 2 lines
can anyone please let us know the correct way to simulate lvm mirror failure and recovery.
Depends a lot on your hardware. E.g. the system I use for testing are about 400 km away, so I'd rather not unplug anything...
I usually run "echo 1 > /sys/block/sda/device/delete" to make /dev/sda disappear and fail for LVM tests. You can fail the disk in a lot of ways depending on your hardware, but this trick works with almost every device!
Quoted 4 lines · [email protected] wrote:
I usually run "echo 1 >
/sys/block/sda/device/delete" to make/dev/sdadisappear and fail for LVM tests. You can fail the disk in a lot of ways depending on your hardware, but this trick works with almost every device!
# echo offline > /sys/block/sda/device/state
might work as well. In this case, you can recover the failure by
# echo running > /sys/block/sda/device/state
Thanks,
Taka