what creates the symlinks in /dev/<volgroup> ?
Hi,
Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?
I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.
Thanks,
Chris
Command vgmknodes
Quoted 17 lines · Chris Friesen
Hi,
Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?
I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.
Thanks,
Chris_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
It appears that "vgscan --mknodes" also creates this symlink, and creates it pointing to /dev/mapper/<name> instead of to /dev/dm-X like udev does.
controller-1:/dev# vgscan -v --mknodes
Wiping cache of LVM-capable devices
Wiping internal VG cache
Reading all physical volumes. This may take a while...
Using volume group(s) on command line.
Found volume group "nova-local" using metadata type lvm2
Found volume group "cgts-vg" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2
Using logical volume(s) on command line.
The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.controller-1:/dev# ls -l /dev/cinder-volumes/ total 0 lrwxrwxrwx 1 root root 7 Jun 20 22:43 anchor-lv -> ../dm-9 lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool -> /dev/mapper/cinder--volumes-cinder--volumes--pool
This is a bad thing, because running the above or "vgmknodes" and then running "vgchange -an cinder-volumes" will leave the /dev/cinder-volumes directory with a dangling symlink in it.
This in turn breaks /usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps erroneously) uses the existence of a non-empty /dev/<volume_group> directory as a test to see if the volume group is active or not.
Chris
Quoted 89 lines · Chris Friesen
Found it.
/usr/lib/udev/rules.d/11-dm-lvm.rulesis what makes the /dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin volume in it it will exit without making a symlink because DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.Given that both vgmknodes and the udev rules come from the lvm2 package, it'd be nice if they agreed on whether or not a symlink should be created and if so where it should link to.
Chris
Quoted 75 lines · Chris Friesen
If I run it I get this:
controller-1:/dev# vgmknodes -v Using logical volume(s) on command line. Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o The link /dev/cinder-volumes/cinder-volumes-pool should have been created by udev but it was not found. Falling back to direct link creation.So it thinks that udev should have already made it. Also, the symlink it generates looks different than the symlinks already in
/dev/cinder-volumes, in that it points to /dev/mapper/<name> rather than pointing to the "real"/dev/dm-Xdevice like the others.controller-1:/dev# ls -l /dev/cinder-volumes/ total 0 lrwxrwxrwx 1 root root 7 Jun 20 17:57 anchor-lv -> ../dm-9 lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool -> /dev/mapper/cinder--volumes-cinder--volumes--pool lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18Given the above, I think that something other than vgmknodes must be involved.
Chris
Quoted 28 lines · Ilya Boka
Command vgmknodes
Quoted 9 lines · Chris Friesen
Hi,
Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?
I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.
Quoted 8 lines
Thanks,
Chris_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Found it. /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the /dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin volume in it it will exit without making a symlink because DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.
Given that both vgmknodes and the udev rules come from the lvm2 package, it'd be nice if they agreed on whether or not a symlink should be created and if so where it should link to.
Chris
Quoted 75 lines · Chris Friesen
If I run it I get this:
controller-1:/dev# vgmknodes -v Using logical volume(s) on command line. Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o The link /dev/cinder-volumes/cinder-volumes-pool should have been created by udev but it was not found. Falling back to direct link creation.So it thinks that udev should have already made it. Also, the symlink it generates looks different than the symlinks already in
/dev/cinder-volumes, in that it points to /dev/mapper/<name> rather than pointing to the "real"/dev/dm-Xdevice like the others.controller-1:/dev# ls -l /dev/cinder-volumes/ total 0 lrwxrwxrwx 1 root root 7 Jun 20 17:57 anchor-lv -> ../dm-9 lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool -> /dev/mapper/cinder--volumes-cinder--volumes--pool lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18Given the above, I think that something other than vgmknodes must be involved.
Chris
Quoted 28 lines · Ilya Boka
Command vgmknodes
Quoted 9 lines · Chris Friesen
Hi,
Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?
I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.
Quoted 8 lines
Thanks,
Chris_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
If I run it I get this:
controller-1:/dev# vgmknodes -v
Using logical volume(s) on command line.
Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR
Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.So it thinks that udev should have already made it. Also, the symlink it generates looks different than the symlinks already in /dev/cinder-volumes, in that it points to /dev/mapper/<name> rather than pointing to the "real" /dev/dm-X device like the others.
controller-1:/dev# ls -l /dev/cinder-volumes/ total 0 lrwxrwxrwx 1 root root 7 Jun 20 17:57 anchor-lv -> ../dm-9 lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool -> /dev/mapper/cinder--volumes-cinder--volumes--pool lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14 lrwxrwxrwx 1 root root 8 Jun 20 17:57 volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18
Given the above, I think that something other than vgmknodes must be involved.
Chris
Quoted 28 lines · Ilya Boka
Command vgmknodes
Quoted 9 lines · Chris Friesen
Hi,
Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?
I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.
Quoted 8 lines
Thanks,
Chris_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Dne 21.6.2016 v 01:13 Chris Friesen napsal(a):
Quoted 22 lines
It appears that "vgscan --mknodes" also creates this symlink, and creates it pointing to /dev/mapper/<name> instead of to
/dev/dm-Xlike udev does.controller-1:/dev# vgscan -v --mknodes Wiping cache of LVM-capable devices Wiping internal VG cache Reading all physical volumes. This may take a while... Using volume group(s) on command line. Found volume group "nova-local" using metadata type lvm2 Found volume group "cgts-vg" using metadata type lvm2 Found volume group "cinder-volumes" using metadata type lvm2 Using logical volume(s) on command line. The link /dev/cinder-volumes/cinder-volumes-pool should have been created by udev but it was not found. Falling back to direct link creation.controller-1:/dev# ls -l /dev/cinder-volumes/ total 0 lrwxrwxrwx 1 root root 7 Jun 20 22:43 anchor-lv -> ../dm-9 lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool -> /dev/mapper/cinder--volumes-cinder--volumes--pool
Quoted 8 lines
This is a bad thing, because running the above or "vgmknodes" and then running "vgchange -an cinder-volumes" will leave the
/dev/cinder-volumesdirectory with a dangling symlink in it.This in turn breaks
/usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps erroneously) uses the existence of a non-empty /dev/<volume_group> directory as a test to see if the volume group is active or not.
This whole thread is presentint one continues sets of bugs and misunderstandings.
On modern Linux system - it should be ONLY udev to create any links in /dev dir. So lvm2 does not create ANY links unless forced to do.
So the clear answer here is 'it is udev rule that creates links'
Now the second part - you system is likely misconfigured.
It's admin responsibility to filter out devices in a way lvm2 does not get any duplicates. If the same device (same UUID) is seen multiple times, lvm2 has no way to know which of them is the right one to use.
Since you have not even shown the version of lvm2 in use and whather lvmetad is in use - there is no way to give you any more hints...
Regards
Zdenek
I'm using the stock CentOS7 version, I think.
LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0
So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?
Personally, I think that running that command should use the same logic as the udev rules to decide whether or not to create a symlink, and should create the symlink pointing to the same place as the udev rules.
I'm not sure what you're talking about as far as duplicates, I'm not seeing any duplicate devices. The problem I see is that "vgscan --mknodes" or "vgmknodes" will both create an additional symlink when compared to the udev rules, and the additional symlink is not deleted when I deactivate the volume group.
For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and
global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]
Chris
Quoted 59 lines · Zdenek Kabelac
Dne 21.6.2016 v 01:13 Chris Friesen napsal(a):
Quoted 22 lines
It appears that "vgscan --mknodes" also creates this symlink, and creates it pointing to /dev/mapper/<name> instead of to
/dev/dm-Xlike udev does.controller-1:/dev# vgscan -v --mknodes Wiping cache of LVM-capable devices Wiping internal VG cache Reading all physical volumes. This may take a while... Using volume group(s) on command line. Found volume group "nova-local" using metadata type lvm2 Found volume group "cgts-vg" using metadata type lvm2 Found volume group "cinder-volumes" using metadata type lvm2 Using logical volume(s) on command line. The link /dev/cinder-volumes/cinder-volumes-pool should have been created by udev but it was not found. Falling back to direct link creation.controller-1:/dev# ls -l /dev/cinder-volumes/ total 0 lrwxrwxrwx 1 root root 7 Jun 20 22:43 anchor-lv -> ../dm-9 lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool -> /dev/mapper/cinder--volumes-cinder--volumes--poolQuoted 8 lines
This is a bad thing, because running the above or "vgmknodes" and then running "vgchange -an cinder-volumes" will leave the
/dev/cinder-volumesdirectory with a dangling symlink in it.This in turn breaks
/usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps erroneously) uses the existence of a non-empty /dev/<volume_group> directory as a test to see if the volume group is active or not.This whole thread is presentint one continues sets of bugs and misunderstandings.
On modern Linux system - it should be ONLY udev to create any links in /dev dir. So lvm2 does not create ANY links unless forced to do.
So the clear answer here is 'it is udev rule that creates links'
Now the second part - you system is likely misconfigured.
It's admin responsibility to filter out devices in a way lvm2 does not get any duplicates. If the same device (same UUID) is seen multiple times, lvm2 has no way to know which of them is the right one to use.Since you have not even shown the version of lvm2 in use and whather lvmetad is in use - there is no way to give you any more hints...
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
Quoted 8 lines
I'm using the stock CentOS7 version, I think.
LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?
Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.
The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.
Quoted 9 lines
I'm not sure what you're talking about as far as duplicates, I'm not seeing any duplicate devices. The problem I see is that "vgscan --mknodes" or "vgmknodes" will both create an additional symlink when compared to the udev rules, and the additional symlink is not deleted when I deactivate the volume group.
For what it's worth, in my lvm.conf I have "
use_lvmetad= 0" and
global_filter= [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]
And now we are getting to the point on your problem:
"a|/dev/sda|" will also match "a|/dev/sda6|"
(and /dev/sda5...)
If you would like to get only 'exact' '/dev/sda' you would need to use
^/dev/sda$
otherwise '/dev/sda' may appear anywhere as substring of your device path.
Regards
Zdenek
Quoted 7 lines · Zdenek Kabelac
Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
Quoted 6 lines
I'm using the stock CentOS7 version, I think.
LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0
Quoted 8 lines
Quoted 2 lines
So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?
Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.
The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.
Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't it make sense to have vgscan use the same logic as udev in terms of what symlinks to make and where to make them to?
Quoted 12 lines
Quoted 9 lines
I'm not sure what you're talking about as far as duplicates, I'm not seeing any duplicate devices. The problem I see is that "vgscan --mknodes" or "vgmknodes" will both create an additional symlink when compared to the udev rules, and the additional symlink is not deleted when I deactivate the volume group.
For what it's worth, in my lvm.conf I have "
use_lvmetad= 0" and
global_filter= [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]And now we are getting to the point on your problem:
Quoted 8 lines
"a|/dev/sda|" will also match "a|/dev/sda6|"
(and/dev/sda5...)If you would like to get only 'exact' '
/dev/sda' you would need to use^/dev/sda$
otherwise '
/dev/sda' may appear anywhere as substring of your device path.
Yes, the "/dev/sda" is intentional since we have multiple /dev/sda* devices that form part of various separate volume groups. Really it's the /dev/sda6 that is extra here since it's already matched.
The problematic issue I'm seeing with udev/vgscan is actually with the volume group on /dev/drbd4.
I really don't think I'm having any problems with duplicates...just with vgscan creating extra symlinks compared to udev, that udev doesn't clean up.
Chris
Dne 22.6.2016 v 16:52 Chris Friesen napsal(a):
Quoted 14 lines · Zdenek Kabelac
Quoted 13 lines
Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
Quoted 8 lines
I'm using the stock CentOS7 version, I think.
LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?
Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.
Quoted 6 lines
Quoted 2 lines
The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.
Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't it make sense to have vgscan use the same logic as udev in terms of what symlinks to make and where to make them to?
It *IS* using same logic.
If the link is not there - the bug is in your udev rules.
When udev is properly configured, vgscan should not show missing link.
It's worth to note - 'dm' subsystem is in general 'ignored' by udev and 'dm' user has to provide proper rules. If your drbd device is missing, then something wrong is with rules...
Likely 'drbd' devices are not properly scanned by udev (and I assume they would not even work on system with use_lvmetad=1).
Quoted 26 lines
Quoted 12 lines
Quoted 9 lines
I'm not sure what you're talking about as far as duplicates, I'm not seeing any duplicate devices. The problem I see is that "vgscan --mknodes" or "vgmknodes" will both create an additional symlink when compared to the udev rules, and the additional symlink is not deleted when I deactivate the volume group.
For what it's worth, in my lvm.conf I have "
use_lvmetad= 0" and
global_filter= [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]And now we are getting to the point on your problem:
Quoted 8 lines
"a|/dev/sda|" will also match "a|/dev/sda6|"
(and/dev/sda5...)If you would like to get only 'exact' '
/dev/sda' you would need to use^/dev/sda$
otherwise '
/dev/sda' may appear anywhere as substring of your device path.Yes, the "
/dev/sda" is intentional since we have multiple/dev/sda*devices that form part of various separate volume groups. Really it's the/dev/sda6that is extra here since it's already matched.
Quoted 6 lines
The problematic issue I'm seeing with udev/vgscan is actually with the volume group on
/dev/drbd4.I really don't think I'm having any problems with duplicates...just with vgscan creating extra symlinks compared to udev, that udev doesn't clean up.
Ahh - ok then - I thought you wanted to see only sda6 on your machine.
Regards
Zdenek
Quoted 22 lines · Zdenek Kabelac
Dne 22.6.2016 v 16:52 Chris Friesen napsal(a):
Quoted 14 lines · Zdenek Kabelac
Quoted 13 lines
Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
Quoted 8 lines
I'm using the stock CentOS7 version, I think.
LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?
Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.
Quoted 6 lines
Quoted 2 lines
The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.
Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't it make sense to have vgscan use the same logic as udev in terms of what symlinks to make and where to make them to?
Quoted 5 lines
It *IS* using same logic.
If the link is not there - the bug is in your udev rules.
When udev is properly configured, vgscan should not show missing link.
It doesn't seem to work this way in practice on a stock CentOS system. Here's the sequence:
1) create a volume group:
"vgcreate chris-volumes /dev/loop2"
At this point there is no /dev/chris-volumes directory.
2) Create a thin pool in the volume group:
"lvcreate -L 1.8GB -T chris-volumes/chris-volumes-pool"
Now udev creates a /dev/chris-volumes directory with a link for the thin pool:
[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 7 Jun 23 12:22 chris-volumes-pool -> ../dm-9
3) Create a thin volume in the thin pool:
"lvcreate -V1G -T chris-volumes/chris-volumes-pool -n thinvolume"
Now the link for the thin pool itself has disappeared:
[root@centos7 centos]# ls -l /dev/chris-volumes total 0
lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11
(At this point /dev/mapper/chris--volumes-chris--volumes--pool-tpool points to dm-9 and /dev/mapper/chris--volumes-chris--volumes--pool points to dm-10.)
4) If I run "vgscan --mknodes", it re-creates the thin pool link, but pointing to the /dev/mapper name instead of directly to the /dev/dm-*. Also, it's indirectly pointing to /dev/dm-10 where before it was pointing to /dev/dm-9:
[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part of
any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.
Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2
Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2
The link /dev/chris-volumes/chris-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.
[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 47 Jun 23 12:25 chris-volumes-pool -> /dev/mapper/chris--volumes-chris--volumes--pool lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11
5) If I run "vgchange -an chris-volumes", I'm left with a /dev/chris-volumes directory containing a broken symlink (broken because /dev/mapper/chris--volumes-chris--volumes--pool doesn't exist anymore):
[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 47 Jun 23 12:25 chris-volumes-pool -> /dev/mapper/chris--volumes-chris--volumes--pool
It looks like /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the /dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin volume in it it will exit without making a symlink because DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.
[root@centos7 centos]# udevadm info /dev/chris-volumes/chris-volumes-pool
P: /devices/virtual/block/dm-10
N: dm-10
E: DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1
Chris
Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):
Quoted 28 lines · Zdenek Kabelac
Quoted 27 lines
Dne 22.6.2016 v 16:52 Chris Friesen napsal(a):
Quoted 15 lines · Zdenek Kabelac
Quoted 14 lines
Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):
Quoted 9 lines
I'm using the stock CentOS7 version, I think.
LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0So are you saying that nobody should run "vgscan --mknodes" on a system where
udev is managing the symlinks?Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.
Quoted 6 lines
Quoted 2 lines
The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.
Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't it make sense to have vgscan use the same logic as udev in terms of what symlinks to make and where to make them to?
It *IS* using same logic.
If the link is not there - the bug is in your udev rules.
Quoted 9 lines
Quoted 1 line
When udev is properly configured, vgscan should not show missing link.
It doesn't seem to work this way in practice on a stock CentOS system. Here's the sequence:
1) create a volume group:
"vgcreate chris-volumes/dev/loop2"
At this point there is no/dev/chris-volumesdirectory.
ONLY active volume do have links
Never ever there is supposed to be directory entry for VG without any active LV.
There is not a term 'active VG' as such - it always is in connection with active LV - thus directory without any active LV inside is pure bug - if you see it - report it as regular BZ..
Quoted 11 lines
2) Create a thin pool in the volume group:
"lvcreate -L 1.8GB -T chris-volumes/chris-volumes-pool"Now udev creates a
/dev/chris-volumesdirectory with a link for the thin pool:
[root@centos7 centos]# ls -l/dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 7 Jun 23 12:22 chris-volumes-pool -> ../dm-93) Create a thin volume in the thin pool:
"lvcreate -V1G -T chris-volumes/chris-volumes-pool -n thinvolume"
Quoted 7 lines
Now the link for the thin pool itself has disappeared:
[root@centos7 centos]# ls -l/dev/chris-volumestotal 0
lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11(At this point
/dev/mapper/chris--volumes-chris--volumes--pool-tpoolpoints to dm-9 and/dev/mapper/chris--volumes-chris--volumes--poolpoints to dm-10.)
correct
Quoted 3 lines
4) If I run "vgscan --mknodes", it re-creates the thin pool link, but pointing to the
/dev/mappername instead of directly to the/dev/dm-*. Also, it's indirectly pointing to/dev/dm-10where before it was pointing to/dev/dm-9:
Only 'final' resolving device does matter.
if link is /dev/vg/lv -> /dev/mapper/vg-lv -> /dev/dm-XXX
or /dev/vg/lv -> /dev/dm-XXX - it does not matter.
There are some 'tricks' related to thin-pool maintainance.
Unused thin-pool is 'public' LV - has /dev/vg/lv link.
Used thin-pool by lvm2 is 'private' LV - doesn't have /dev/vg/lv link.
All device should always have entry in /dev/mapper/ - either links to real devices or direct nodes (on older systems)
lvm2 users are always supposed to use ONLY /dev/vg/lv devices for access.
Quoted 6 lines
[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part
of any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.
fix your lvm.conf (uncomment sections)
Quoted 5 lines
Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2 Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2 The link/dev/chris-volumes/chris-volumes-poolshould have been created by
Ok - there seems to be internal bug in lvm2 - which incorrectly hints link creation for this case.
There should not have been /dev/vg/pool link - this is correctly marked for udev - but incorrectly for udev validation.
However the bug is actually not so much important - it just links
to 'wrapper' device - and eventually we will resolve the problem even without this extra device in table.
Regards
Zdenek
Quoted 19 lines · Zdenek Kabelac
Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):
Quoted 5 lines
[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part
of any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.fix your lvm.conf (uncomment sections)
Quoted 5 lines
Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2 Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2 The link/dev/chris-volumes/chris-volumes-poolshould have been created byOk - there seems to be internal bug in lvm2 - which incorrectly hints link creation for this case.
Quoted 6 lines
There should not have been
/dev/vg/poollink - this is correctly marked for udev - but incorrectly for udev validation.However the bug is actually not so much important - it just links to 'wrapper' device - and eventually we will resolve the problem even without this extra device in table.
The problem that it causes for me is that when I run "vgchange -an chris-volumes" it leaves the /dev/chris-volumes with a broken symlink in it because udev doesn't remove the symlink added by vgscan.
This causes the LVM OCF script in the "resource-agents" package to break, because it is using the existance of the /dev/vg directory as a proxy for whether the volume group is active (or really as you said earlier, whether there are active volumes within the volume group).
I reported this as a bug to the "resource-agents" package developers, and they said that they can't actually call lvm commands in their "status" routines because there have been cases where clustered LVM hung when querying status, causing the OCF script to hang and monitoring to fail.
Ultimately I'll see if I can work around it by not calling "vgscan --mknodes". Originally it was added in to fix some problems, but that was a while back so things may behave properly now.
Thanks for your help,
Chris
Dne 23.6.2016 v 20:02 Chris Friesen napsal(a):
Quoted 23 lines · Zdenek Kabelac
Quoted 22 lines
Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):
Quoted 5 lines
[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part
of any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.fix your lvm.conf (uncomment sections)
Quoted 5 lines
Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2 Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2 The link/dev/chris-volumes/chris-volumes-poolshould have been created byOk - there seems to be internal bug in lvm2 - which incorrectly hints link creation for this case.
There should not have been
/dev/vg/poollink - this is correctly marked for udev - but incorrectly for udev validation.
Quoted 7 lines
Quoted 3 lines
However the bug is actually not so much important - it just links to 'wrapper' device - and eventually we will resolve the problem even without this extra device in table.
The problem that it causes for me is that when I run "vgchange -an chris-volumes" it leaves the
/dev/chris-volumeswith a broken symlink in it because udev doesn't remove the symlink added by vgscan.
Yep - as said - in normal circumstance use should NOT run 'vgmknodes' as the created links will not be known/visible to udev - so this behavior is correct.
Quoted 11 lines
This causes the LVM OCF script in the "resource-agents" package to break, because it is using the existance of the
/dev/vgdirectory as a proxy for whether the volume group is active (or really as you said earlier, whether there are active volumes within the volume group).I reported this as a bug to the "resource-agents" package developers, and they said that they can't actually call lvm commands in their "status" routines because there have been cases where clustered LVM hung when querying status, causing the OCF script to hang and monitoring to fail.
Ultimately I'll see if I can work around it by not calling "vgscan --mknodes".
yes please, start with this one...
vgmknodes are really supposed to be used on some unique urgent problem - not executed in script every hour...
But yes - lvm2 will need to fix link creation of pool-in-use...
Quoted 2 lines
Originally it was added in to fix some problems, but that was a while back so things may behave properly now.
Regards
Zdenek
Dne 23.6.2016 v 20:02 Chris Friesen napsal(a):
Quoted 36 lines · Zdenek Kabelac
Quoted 19 lines
Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):
Quoted 5 lines
[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part
of any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.fix your lvm.conf (uncomment sections)
Quoted 5 lines
Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2 Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2 The link/dev/chris-volumes/chris-volumes-poolshould have been created byOk - there seems to be internal bug in lvm2 - which incorrectly hints link creation for this case.
Quoted 6 lines
There should not have been
/dev/vg/poollink - this is correctly marked for udev - but incorrectly for udev validation.However the bug is actually not so much important - it just links to 'wrapper' device - and eventually we will resolve the problem even without this extra device in table.
The problem that it causes for me is that when I run "vgchange -an chris-volumes" it leaves the
/dev/chris-volumeswith a broken symlink in it because udev doesn't remove the symlink added by vgscan.This causes the LVM OCF script in the "resource-agents" package to break, because it is using the existance of the
/dev/vgdirectory as a proxy for whether the volume group is active (or really as you said earlier, whether there are active volumes within the volume group).
Quoted 8 lines
I reported this as a bug to the "resource-agents" package developers, and they said that they can't actually call lvm commands in their "status" routines because there have been cases where clustered LVM hung when querying status, causing the OCF script to hang and monitoring to fail.
Ultimately I'll see if I can work around it by not calling "vgscan --mknodes". Originally it was added in to fix some problems, but that was a while back so things may behave properly now.
Should be resolved with this upstream commit:
https://www.redhat.com/archives/lvm-devel/2016-June/msg00187.html
Regards
Zdenek