lvm Bug? - bad reaction to snapshot creation
Creating a snapshot throws an error "Attempted to decrement suspended device counter below zero." but succeeds. Removing the snapshot fails, attempting a 2nd time succeeds. Somewhere in this process the original LV gets locked and the system needs to be restarted to unlock it. As explained below this happens intermittently but regularly.
Any additional info please let me know directly, I am not a subscriber, Leeman
----
log file: http://www.enlj.com/lvm.txt
lvm config: http://www.enlj.com/lvmconfig.txt
Arch Linux, lvm2 2.02.106-2
line 1: lvcreate -L1G -s -n srvrootsnap /dev/ssd.vg/server-root
Attempted to decrement suspended device counter below zero. Logical volume "srvrootsnap" created
line 148+: why? globalfilter = [ "a|/dev/md|", "r|.*|" ] line 921: after this point I do udevadm settle
no output from udevadm settle
line 923: lvremove -f /dev/ssd.vg/srvrootsnap - fails
Unable to deactivate open ssd.vg-srvrootsnap-cow (253:3) Failed to activate srvrootsnap.
Releasing activation in critical section.
libdevmapper exiting with 1 device(s) still suspended.
line 1567: lvremove -f /dev/ssd.vg/srvrootsnap - again, works this time
Logical volume "srvrootsnap" successfully removed
However, /dev/ssd.vg/server-root is locked up, the VM seizes as no data can be written to the volume. I have to restart the entire system to unlock the volume.
This doesn't happen all the time. It happens every time I do a backup. 1 of the 7 LV snapshots created for the job, not always the same one.
hardware: Supermicro X9DR3-F,
onboard SATA controller:
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06) /dev/md0 consists of 2 Seagate 600 240GB SSDs on that controller/dev/ssd.vg consists of /dev/md0
Dne 7.6.2014 01:26, Leeman Strout napsal(a):
Quoted 9 lines
Creating a snapshot throws an error "Attempted to decrement suspended device counter below zero." but succeeds. Removing the snapshot fails, attempting a 2nd time succeeds. Somewhere in this process the original LV gets locked and the system needs to be restarted to unlock it. As explained below this happens intermittently but regularly.
Any additional info please let me know directly, I am not a subscriber,
Hi
Unsure if this relates to all your problem (since I'm not sure how arch linux is in sync with udev rules & systemd version)
At certain moment systemd added a new 'feature' about locking devices while
updating internal udev state - this locking ignores any udev rule flags and opens internal lvm2 devices - so while for now it's been again disabled for 'dm' devices - you might have installed version of system which has the 'lock everything' feature in?
Thought this doesn't explain your 'counter below zero' error - this looks like some incorrect udev rules are running in the field ?
(Or maybe multiple systemd-udevd are running ?)
Zdenek
Quoted 29 lines · Zdenek Kabelac
Dne 7.6.2014 01:26, Leeman Strout napsal(a):
Quoted 11 lines
Creating a snapshot throws an error "Attempted to decrement suspended device
counter below zero." but succeeds. Removing the snapshot fails, attempting a
2nd time succeeds. Somewhere in this process the original LV gets locked and
the system needs to be restarted to unlock it. As explained below this happens intermittently but regularly.Quoted 1 line
Any additional info please let me know directly, I am not a subscriber,
Hi
Unsure if this relates to all your problem (since I'm not sure how arch linux
is in sync with udev rules & systemd version)At certain moment systemd added a new 'feature' about locking devices while
updating internal udev state - this locking ignores any udev rule flags and opens internal lvm2 devices - so while for now it's been again disabled for 'dm' devices - you might have installed version of system which has the 'lock everything' feature in?
Quoted 4 lines
Thought this doesn't explain your 'counter below zero' error - this looks like
some incorrect udev rules are running in the field ?
(Or maybe multiple systemd-udevd are running ?)
There is also a recent BZ against RHEL7:
https://bugzilla.redhat.com/show_bug.cgi?id=1105732
What are the devices used in the stack?
-- Martian
Quoted 8 lines
Zdenek
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 11 lines · Zdenek Kabelac
Dne 7.6.2014 01:26, Leeman Strout napsal(a):
HiUnsure if this relates to all your problem (since I'm not sure how arch linux is in sync with udev rules & systemd version)
At certain moment systemd added a new 'feature' about locking devices while updating internal udev state - this locking ignores any udev rule flags and opens internal lvm2 devices - so while for now it's been again disabled for 'dm' devices - you might have installed version of system which has the 'lock everything' feature in?
systemd 213-6 is what Arch reports, as far as I can tell from the PKGBUILD it's vanilla 213 plus 2 patches:
- backport fix for faily MACAddress matching (FS#40675)
- backport fix for fsck/udev mess (FS#40706)
Do you have links to discussion of this feature or bugs pertaining to it?
Quoted 3 lines
Thought this doesn't explain your 'counter below zero' error - this looks like some incorrect udev rules are running in the field ? (Or maybe multiple systemd-udevd are running ?)
After a clean restart of everything twice without doing snapshots, the initial lvcreate does not have the 'counter below zero' error. And with special attention to make sure the snapshots are cleaned up prior to attempting a new snapshot I am not getting this initial decrement on lvcreate.
Thanks,
Leeman
Quoted 6 lines · Marian Csontos
There is also a recent BZ against RHEL7:
https://bugzilla.redhat.com/show_bug.cgi?id=1105732
What are the devices used in the stack?
It's all there in the original message,
Quoted 5 lines
hardware: Supermicro X9DR3-F,
onboard SATA controller:
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)/dev/md0consists of 2 Seagate 600 240GB SSDs on that controller/dev/ssd.vgconsists of/dev/md0
If that is not sufficient, what exactly are you looking for?
Thanks,
Leeman
Dne 9.6.2014 16:53, Leeman Strout napsal(a):
Quoted 12 lines · Zdenek Kabelac
Quoted 11 lines
Dne 7.6.2014 01:26, Leeman Strout napsal(a):
HiUnsure if this relates to all your problem (since I'm not sure how arch linux is in sync with udev rules & systemd version)
At certain moment systemd added a new 'feature' about locking devices while updating internal udev state - this locking ignores any udev rule flags and opens internal lvm2 devices - so while for now it's been again disabled for 'dm' devices - you might have installed version of system which has the 'lock everything' feature in?
Quoted 4 lines
systemd 213-6 is what Arch reports, as far as I can tell from the PKGBUILD it's vanilla 213 plus 2 patches:
- backport fix for faily MACAddress matching (FS#40675)
- backport fix for fsck/udev mess (FS#40706)
this commit should be fixing problems for lvm2 & latest systemd:
e918a1b5a94f270186dca59156354acd2a596494
and this is systemd commit which has introduced problem:
3d06f4183470d42361303086ed9dedd29c0ffc1b
Unsure what do you have in your arch build.
Quoted 7 lines
Quoted 3 lines
Thought this doesn't explain your 'counter below zero' error - this looks like some incorrect udev rules are running in the field ? (Or maybe multiple systemd-udevd are running ?)
After a clean restart of everything twice without doing snapshots, the initial lvcreate does not have the 'counter below zero' error. And with special attention to make sure the snapshots are cleaned up prior to attempting a new snapshot I am not getting this initial decrement on lvcreate.
So could we consider this 'counter' case is solved ?
Zdenek
Dne 9.6.2014 16:53, Leeman Strout napsal(a):
Quoted 12 lines · Zdenek Kabelac
Quoted 11 lines
Dne 7.6.2014 01:26, Leeman Strout napsal(a):
HiUnsure if this relates to all your problem (since I'm not sure how arch linux is in sync with udev rules & systemd version)
At certain moment systemd added a new 'feature' about locking devices while updating internal udev state - this locking ignores any udev rule flags and opens internal lvm2 devices - so while for now it's been again disabled for 'dm' devices - you might have installed version of system which has the 'lock everything' feature in?
Quoted 4 lines
systemd 213-6 is what Arch reports, as far as I can tell from the PKGBUILD it's vanilla 213 plus 2 patches:
- backport fix for faily MACAddress matching (FS#40675)
- backport fix for fsck/udev mess (FS#40706)
this commit should be fixing problems for lvm2 & latest systemd:
e918a1b5a94f270186dca59156354acd2a596494
and this is systemd commit which has introduced problem:
3d06f4183470d42361303086ed9dedd29c0ffc1b
Unsure what do you have in your arch build.
Quoted 7 lines
Quoted 3 lines
Thought this doesn't explain your 'counter below zero' error - this looks like some incorrect udev rules are running in the field ? (Or maybe multiple systemd-udevd are running ?)
After a clean restart of everything twice without doing snapshots, the initial lvcreate does not have the 'counter below zero' error. And with special attention to make sure the snapshots are cleaned up prior to attempting a new snapshot I am not getting this initial decrement on lvcreate.
So could we consider this 'counter' case is solved ?
Zdenek
Quoted 40 lines · Zdenek Kabelac
Dne 9.6.2014 16:53, Leeman Strout napsal(a):
Quoted 12 lines · Zdenek Kabelac
Quoted 11 lines
Dne 7.6.2014 01:26, Leeman Strout napsal(a):
HiUnsure if this relates to all your problem (since I'm not sure how arch linux is in sync with udev rules & systemd version)
At certain moment systemd added a new 'feature' about locking devices while updating internal udev state - this locking ignores any udev rule flags and opens internal lvm2 devices - so while for now it's been again disabled for 'dm' devices - you might have installed version of system which has the 'lock everything' feature in?
Quoted 5 lines
systemd 213-6 is what Arch reports, as far as I can tell from the PKGBUILD
it's vanilla 213 plus 2 patches:
- backport fix for faily MACAddress matching (FS#40675)
- backport fix for fsck/udev mess (FS#40706)this commit should be fixing problems for lvm2 & latest systemd:
e918a1b5a94f270186dca59156354acd2a596494and this is systemd commit which has introduced problem:
3d06f4183470d42361303086ed9dedd29c0ffc1bUnsure what do you have in your arch build.
Quoted 9 lines
Quoted 3 lines
Thought this doesn't explain your 'counter below zero' error - this looks like some incorrect udev rules are running in the field ? (Or maybe multiple systemd-udevd are running ?)
After a clean restart of everything twice without doing snapshots, the initial
lvcreate does not have the 'counter below zero' error. And with special attention to make sure the snapshots are cleaned up prior to attempting a new
snapshot I am not getting this initial decrement on lvcreate.
Quoted 4 lines
So could we consider this 'counter' case is solved ?
Zdenek
Am testing the rev Arch dev pushed w/ that change : https://projects.archlinux.org/svntogit/packages.git/diff/trunk/0001-udev-exclude-device-mapper-from-block-device-ownersh.patch?h=packages/systemd&id=331c26905843338b30b4cd240c64953501cd879c
It does not resolve the issue.
Thanks,
Leeman
Quoted 22 lines · Zdenek Kabelac
Dne 9.6.2014 16:53, Leeman Strout napsal(a):
Quoted 12 lines · Zdenek Kabelac
Quoted 11 lines
Dne 7.6.2014 01:26, Leeman Strout napsal(a):
HiUnsure if this relates to all your problem (since I'm not sure how arch linux is in sync with udev rules & systemd version)
At certain moment systemd added a new 'feature' about locking devices while updating internal udev state - this locking ignores any udev rule flags and opens internal lvm2 devices - so while for now it's been again disabled for 'dm' devices - you might have installed version of system which has the 'lock everything' feature in?
Quoted 5 lines
systemd 213-6 is what Arch reports, as far as I can tell from the PKGBUILD
it's vanilla 213 plus 2 patches:
- backport fix for faily MACAddress matching (FS#40675)
- backport fix for fsck/udev mess (FS#40706)this commit should be fixing problems for lvm2 & latest systemd:
Quoted 7 lines
e918a1b5a94f270186dca59156354acd2a596494and this is systemd commit which has introduced problem:
3d06f4183470d42361303086ed9dedd29c0ffc1bUnsure what do you have in your arch build.
This patch: http://lists.freedesktop.org/archives/systemd-devel/2014-June/019863.html applied and tested in Arch solves the issue.
Thanks,
Leeman
Leeman Strout <[email protected]> on Tue, 2014/06/10 11:22:
Quoted 27 lines · Zdenek Kabelac
Quoted 26 lines
Dne 9.6.2014 16:53, Leeman Strout napsal(a):
Quoted 12 lines · Zdenek Kabelac
Quoted 11 lines
Dne 7.6.2014 01:26, Leeman Strout napsal(a):
HiUnsure if this relates to all your problem (since I'm not sure how arch linux is in sync with udev rules & systemd version)
At certain moment systemd added a new 'feature' about locking devices while updating internal udev state - this locking ignores any udev rule flags and opens internal lvm2 devices - so while for now it's been again disabled for 'dm' devices - you might have installed version of system which has the 'lock everything' feature in?
Quoted 5 lines
systemd 213-6 is what Arch reports, as far as I can tell from the PKGBUILD
it's vanilla 213 plus 2 patches:
- backport fix for faily MACAddress matching (FS#40675)
- backport fix for fsck/udev mess (FS#40706)this commit should be fixing problems for lvm2 & latest systemd:
e918a1b5a94f270186dca59156354acd2a596494and this is systemd commit which has introduced problem:
Quoted 7 lines
Quoted 3 lines
3d06f4183470d42361303086ed9dedd29c0ffc1bUnsure what do you have in your arch build.
This patch:
http://lists.freedesktop.org/archives/systemd-devel/2014-June/019863.html
applied and tested in Arch solves the issue.
This has been applied to systemd-213-9 (and systemd upstream) already. ;)
Signature · 4 lines
Schoene Gruesse
Chris
O< ascii ribbon campaign
stop html mail - www.asciiribbon.org