Removing snapshot often fails
Hello,
I have an OpenSuSE 11.4 based Xen server. I use LVM as the 'store' for VM 's and I'm trying to make use of snapshots for backup.
However I discovered that often removing a snapshot fails - unfortunately it's quite unpredictable, as sometimes it works on next try but sometimes it fails 20 times in a row - making it very unpleasant for scripting...
Even when it succeeds it give's me a message 'give up on open_count' - and I couldn't quite find what it means.
Here's what I've been trying by hand:
Quoted 6 lines
# lvcreate -s /dev/vm_store/test_vm1 -n "backup_test_vm1" -l 100%ORIGIN Logical volume "backup_test_vm1" created # lvremove -f /dev/vm_store/backup_test_vm1 give up on open_count Logical volume "backup_test_vm1" successfully removed # lvcreate -s /dev/vm_store/test_vm1 -n "backup_test_vm1" -l 100%ORIGIN
Quoted 8 lines
Logical volume "backup_test_vm1" created # lvremove -f /dev/vm_store/backup_test_vm1 give up on open_count Logical volume "backup_test_vm1" successfully removed # lvcreate -s /dev/vm_store/test_vm1 -n "backup_test_vm1" -l 100%ORIGIN Logical volume "backup_test_vm1" created # lvremove -f /dev/vm_store/backup_test_vm1 Can't remove open logical volume "backup_test_vm1"
Am I doing something wrong?
Is this 'expected' and should I just keep re-trying in a loop until it succeeds?
Thanks, Danilo
Signature · 11 lines
Danilo Godec, sistemska podpora / system administration Predlog! Obiscite prenovljeno spletno stran www.agenda.si ODPRTA KODA IN LINUX STORITVE : POSLOVNE RESITVE : UPRAVLJANJE IT : INFRASTRUKTURA IT : IZOBRAZEVANJE : PROGRAMSKA OPREMA Visit our updated web page at www.agenda.si OPEN SOURCE AND LINUX SERVICES : BUSINESS SOLUTIONS : IT MANAGEMENT : IT INFRASTRUCTURE : TRAINING : SOFTWARE
Quoted 1 line · Danilo Godec
However I discovered that often removing a snapshot fails - unfortunately it's quite unpredictable, as sometimes it works on next try but sometimes it fails 20 times in a row - making it very unpleasant for scripting...
What's the lvm2 version you're using?
With a very high probability, this is caused by the "watch" udev rule. Do you have "udisks" installed? This one sets in its '/lib/udev/rules.d/80-udisks.rules' the 'KERNEL=="dm-*", OPTIONS+="watch"' which causes the udev event to be generated and processed while trying to close the device.
See also https://bugzilla.redhat.com/show_bug.cgi?id=577798 for more information about the problem.
Recently, we've added a retry loop when trying to remove a device-mapper device. This will try to remove the device several times before it fails completely (libdevmapper v1.02.68, lvm2 v2.02.89 released 26th Jan. 2012). Though you still get an error saying "remove ioctl failed" on each failed retry...
Peter
Quoted 3 lines · Peter Rajnoha
Quoted 1 line · Danilo Godec
However I discovered that often removing a snapshot fails - unfortunately it's quite unpredictable, as sometimes it works on next try but sometimes it fails 20 times in a row - making it very unpleasant for scripting...
What's the lvm2 version you're using?
The version is 2.02.67 from 'official' OpenSuSE 11.4 updates.
Quoted 5 lines
With a very high probability, this is caused by the "watch" udev rule. Do you have "udisks" installed? This one sets in its '
/lib/udev/rules.d/80-udisks.rules' the 'KERNEL=="dm-*", OPTIONS+="watch"' which causes the udev event to be generated and processed while trying to close the device.
Yes, udisks package is installed and above mentioned rule file is in place. Not sure if it's really needed, though.
Quoted 2 lines
See also https://bugzilla.redhat.com/show_bug.cgi?id=577798 for more information about the problem.
It seems that 'udevadm control --stop-exec-queue' before removing the snapshot could be a viable workaround.
Quoted 4 lines
Recently, we've added a retry loop when trying to remove a device-mapper device. This will try to remove the device several times before it fails completely (libdevmapper v1.02.68, lvm2 v2.02.89 released 26th Jan. 2012). Though you still get an error saying "remove ioctl failed" on each failed retry...
Thank you,
Danilo
Quoted 7 lines · Danilo Godec
Quoted 3 lines · Peter Rajnoha
Quoted 1 line · Danilo Godec
However I discovered that often removing a snapshot fails - unfortunately it's quite unpredictable, as sometimes it works on next try but sometimes it fails 20 times in a row - making it very unpleasant for scripting...
What's the lvm2 version you're using?
The version is 2.02.67 from 'official' OpenSuSE 11.4 updates.
Quoted 8 lines
Quoted 5 lines
With a very high probability, this is caused by the "watch" udev rule. Do you have "udisks" installed? This one sets in its '
/lib/udev/rules.d/80-udisks.rules' the 'KERNEL=="dm-*", OPTIONS+="watch"' which causes the udev event to be generated and processed while trying to close the device.Yes, udisks package is installed and above mentioned rule file is in place. Not sure if it's really needed, though.
Udisks keeps the information about storage devices and provides an interface for disk management. It's targeted for desktop integration mostly (the "palimpsest - gnome-disk-utility" makes use of it mainly, I think)
Quoted 5 lines
Quoted 2 lines
See also https://bugzilla.redhat.com/show_bug.cgi?id=577798 for more information about the problem.
It seems that 'udevadm control --stop-exec-queue' before removing the snapshot could be a viable workaround.
Better workaround is to delete/comment out the 'KERNEL=="dm-*", OPTIONS+="watch"' line in the 80-udisks.rules. The only thing you'd lose is the immediate update when something changes metadata (filesystem label - the consequent update of /dev/disk symlinks, changes in lvm metadata and update of the udisks state etc.). Though after a normal "CHANGE" udev event comes, this info is updated. This happens after restarting the system or generating the event directly by "echo change > /sys/block/dm-X/uevent" (dm-X being the actual device-mapper device).
The watch udev rule is even removed in latest versions of udisks. The problem here is that the proper logic should be that all the utilities changing any metadata on that device (e.g. the filesystem label, creating a filesystem...) should generate the event themselves which is still not done today. Instead, we're "watching" all closes of the device after being open for "write" (which is what the WATCH rule is about - it's the inotify for "close after open for write" and this is not 100% correct since not all such writes change metadata (writing normal data) and causes useless resource consumption while processing the events, also creating races ending up with problems like you're hitting now.
Quoted 4 lines
Quoted 4 lines
Recently, we've added a retry loop when trying to remove a device-mapper device. This will try to remove the device several times before it fails completely (libdevmapper v1.02.68, lvm2 v2.02.89 released 26th Jan. 2012). Though you still get an error saying "remove ioctl failed" on each failed retry...
This is more or less an "official" workaround we've added for now, but the real solution is the one described above with sending the event by all the utilities changing metadata. Maybe one day this will be changed. I hope...
Peter