Skip to contentSkip to conversation

what creates the symlinks in /dev/<volgroup> ?

linux-lvm Chris Friesen 4 participants 15 messages Jun 20, 2016 – Jul 1, 2016

lore
Share & tools
lore
#0 lore raw

Hi,

Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?

I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.

Thanks,
Chris

#1 lore raw

Command vgmknodes

Quoted 17 lines · Chris Friesen

Hi,

Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?

I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.

Thanks,
Chris

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#4 lore raw
In reply to Chris Friesen

It appears that "vgscan --mknodes" also creates this symlink, and creates it pointing to /dev/mapper/<name> instead of to /dev/dm-X like udev does.

controller-1:/dev# vgscan -v --mknodes
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Reading all physical volumes.  This may take a while...
    Using volume group(s) on command line.
  Found volume group "nova-local" using metadata type lvm2
  Found volume group "cgts-vg" using metadata type lvm2
  Found volume group "cinder-volumes" using metadata type lvm2
    Using logical volume(s) on command line.
  The link /dev/cinder-volumes/cinder-volumes-pool should have been created by 
udev but it was not found. Falling back to direct link creation.
controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root  7 Jun 20 22:43 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool -> /dev/mapper/cinder--volumes-cinder--volumes--pool

This is a bad thing, because running the above or "vgmknodes" and then running "vgchange -an cinder-volumes" will leave the /dev/cinder-volumes directory with a dangling symlink in it.

This in turn breaks /usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps erroneously) uses the existence of a non-empty /dev/<volume_group> directory as a test to see if the volume group is active or not.

Chris

Quoted 89 lines · Chris Friesen

Found it. /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the /dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin volume in it it will exit without making a symlink because DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.

Given that both vgmknodes and the udev rules come from the lvm2 package, it'd be nice if they agreed on whether or not a symlink should be created and if so where it should link to.

Chris

Quoted 75 lines · Chris Friesen

If I run it I get this:

controller-1:/dev# vgmknodes -v
     Using logical volume(s) on command line.
     Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
     Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
     Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
     Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR
     Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
   The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.

So it thinks that udev should have already made it. Also, the symlink it generates looks different than the symlinks already in /dev/cinder-volumes, in that it points to /dev/mapper/<name> rather than pointing to the "real" /dev/dm-X device like the others.

controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root  7 Jun 20 17:57 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool ->
/dev/mapper/cinder--volumes-cinder--volumes--pool
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18

Given the above, I think that something other than vgmknodes must be involved.

Chris

Quoted 28 lines · Ilya Boka

Command vgmknodes

Quoted 9 lines · Chris Friesen

Hi,

Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?

I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.

Quoted 8 lines

Thanks,
Chris

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#3 lore raw
In reply to Chris Friesen

Found it. /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the /dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin volume in it it will exit without making a symlink because DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.

Given that both vgmknodes and the udev rules come from the lvm2 package, it'd be nice if they agreed on whether or not a symlink should be created and if so where it should link to.

Chris

Quoted 75 lines · Chris Friesen

If I run it I get this:

controller-1:/dev# vgmknodes -v
     Using logical volume(s) on command line.
     Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
     Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
     Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
     Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR
     Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
   The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.

So it thinks that udev should have already made it. Also, the symlink it generates looks different than the symlinks already in /dev/cinder-volumes, in that it points to /dev/mapper/<name> rather than pointing to the "real" /dev/dm-X device like the others.

controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root  7 Jun 20 17:57 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool ->
/dev/mapper/cinder--volumes-cinder--volumes--pool
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14
lrwxrwxrwx 1 root root  8 Jun 20 17:57
volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18

Given the above, I think that something other than vgmknodes must be involved.

Chris

Quoted 28 lines · Ilya Boka

Command vgmknodes

Quoted 9 lines · Chris Friesen

Hi,

Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?

I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.

Quoted 8 lines

Thanks,
Chris

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
In reply to Ilya Boka

If I run it I get this:

controller-1:/dev# vgmknodes -v
    Using logical volume(s) on command line.
    Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
    Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
    Found same device /dev/sda6 with same pvid Fw6C1IZABbnspIT23RbOGf2DXuB4zMhS
    Found same device /dev/drbd4 with same pvid 6yk0HVTSbU9Amo6mwPKt3nZPnBIXBlhR
    Found same device /dev/sda5 with same pvid 1s1dPDodojAS0kqToIRy4hiXFjCp2t2o
  The link /dev/cinder-volumes/cinder-volumes-pool should have been created by 
udev but it was not found. Falling back to direct link creation.

So it thinks that udev should have already made it. Also, the symlink it generates looks different than the symlinks already in /dev/cinder-volumes, in that it points to /dev/mapper/<name> rather than pointing to the "real" /dev/dm-X device like the others.

controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root  7 Jun 20 17:57 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 21:48 cinder-volumes-pool -> /dev/mapper/cinder--volumes-cinder--volumes--pool
lrwxrwxrwx 1 root root  8 Jun 20 17:57 volume-0bc1df18-45d0-4477-9c57-36876d3f82d4 -> ../dm-19
lrwxrwxrwx 1 root root  8 Jun 20 17:57 volume-2fff261f-8860-4b86-8b2e-49bddcf47e9b -> ../dm-17
lrwxrwxrwx 1 root root  8 Jun 20 17:57 volume-48744604-6b02-4f11-ba02-3f692d109953 -> ../dm-15
lrwxrwxrwx 1 root root  8 Jun 20 17:57 volume-8dabc793-e46d-4849-a2fb-dd3d4bc2c988 -> ../dm-20
lrwxrwxrwx 1 root root  8 Jun 20 17:57 volume-be3c9ddb-a6eb-43ca-ac37-5554756a4c13 -> ../dm-16
lrwxrwxrwx 1 root root  8 Jun 20 17:57 volume-eef89318-fa8e-4ca2-a8a7-fe8e143d8792 -> ../dm-14
lrwxrwxrwx 1 root root  8 Jun 20 17:57 volume-f11a7d88-1ad3-4e89-a594-e824019725bb -> ../dm-18

Given the above, I think that something other than vgmknodes must be involved.

Chris

Quoted 28 lines · Ilya Boka

Command vgmknodes

Quoted 9 lines · Chris Friesen

Hi,

Can someone tell me what creates the /dev/<volgroup>/<volume symlinks? Is this LVM or udev (and if udev, do you know which rule)?

I'm seeing some interesting behaviour where if I create a thin pool it creates a symlink for the pool, but once I create a thin volume within the pool then the pool symlink disappears.

Quoted 8 lines

Thanks,
Chris

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#5 lore raw
In reply to Chris Friesen

Dne 21.6.2016 v 01:13 Chris Friesen napsal(a):

Quoted 22 lines

It appears that "vgscan --mknodes" also creates this symlink, and creates it pointing to /dev/mapper/<name> instead of to /dev/dm-X like udev does.

controller-1:/dev# vgscan -v --mknodes
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Reading all physical volumes.  This may take a while...
    Using volume group(s) on command line.
  Found volume group "nova-local" using metadata type lvm2
  Found volume group "cgts-vg" using metadata type lvm2
  Found volume group "cinder-volumes" using metadata type lvm2
    Using logical volume(s) on command line.
  The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.
controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root  7 Jun 20 22:43 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool ->
/dev/mapper/cinder--volumes-cinder--volumes--pool
Quoted 8 lines

This is a bad thing, because running the above or "vgmknodes" and then running "vgchange -an cinder-volumes" will leave the /dev/cinder-volumes directory with a dangling symlink in it.

This in turn breaks /usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps erroneously) uses the existence of a non-empty /dev/<volume_group> directory as a test to see if the volume group is active or not.

This whole thread is presentint one continues sets of bugs and misunderstandings.

On modern Linux system - it should be ONLY udev to create any links in /dev dir. So lvm2 does not create ANY links unless forced to do.

So the clear answer here is 'it is udev rule that creates links'

Now the second part - you system is likely misconfigured.
It's admin responsibility to filter out devices in a way lvm2 does not get any duplicates. If the same device (same UUID) is seen multiple times, lvm2 has no way to know which of them is the right one to use.

Since you have not even shown the version of lvm2 in use and whather lvmetad is in use - there is no way to give you any more hints...

Regards

Zdenek

#6 lore raw

I'm using the stock CentOS7 version, I think.

LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0

So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?

Personally, I think that running that command should use the same logic as the udev rules to decide whether or not to create a symlink, and should create the symlink pointing to the same place as the udev rules.

I'm not sure what you're talking about as far as duplicates, I'm not seeing any duplicate devices. The problem I see is that "vgscan --mknodes" or "vgmknodes" will both create an additional symlink when compared to the udev rules, and the additional symlink is not deleted when I deactivate the volume group.

For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and

global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]

Chris

Quoted 59 lines · Zdenek Kabelac

Dne 21.6.2016 v 01:13 Chris Friesen napsal(a):

Quoted 22 lines

It appears that "vgscan --mknodes" also creates this symlink, and creates it pointing to /dev/mapper/<name> instead of to /dev/dm-X like udev does.

controller-1:/dev# vgscan -v --mknodes
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Reading all physical volumes.  This may take a while...
    Using volume group(s) on command line.
  Found volume group "nova-local" using metadata type lvm2
  Found volume group "cgts-vg" using metadata type lvm2
  Found volume group "cinder-volumes" using metadata type lvm2
    Using logical volume(s) on command line.
  The link /dev/cinder-volumes/cinder-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.
controller-1:/dev# ls -l /dev/cinder-volumes/
total 0
lrwxrwxrwx 1 root root  7 Jun 20 22:43 anchor-lv -> ../dm-9
lrwxrwxrwx 1 root root 49 Jun 20 23:09 cinder-volumes-pool ->
/dev/mapper/cinder--volumes-cinder--volumes--pool
Quoted 8 lines

This is a bad thing, because running the above or "vgmknodes" and then running "vgchange -an cinder-volumes" will leave the /dev/cinder-volumes directory with a dangling symlink in it.

This in turn breaks /usr/lib/ocf/resource.d/heartbeat/LVM, which (perhaps erroneously) uses the existence of a non-empty /dev/<volume_group> directory as a test to see if the volume group is active or not.

This whole thread is presentint one continues sets of bugs and misunderstandings.

On modern Linux system - it should be ONLY udev to create any links in /dev dir. So lvm2 does not create ANY links unless forced to do.

So the clear answer here is 'it is udev rule that creates links'

Now the second part - you system is likely misconfigured.
It's admin responsibility to filter out devices in a way lvm2 does not get any duplicates. If the same device (same UUID) is seen multiple times, lvm2 has no way to know which of them is the right one to use.

Since you have not even shown the version of lvm2 in use and whather lvmetad is in use - there is no way to give you any more hints...

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#7 lore raw

Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):

Quoted 8 lines

I'm using the stock CentOS7 version, I think.

LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0

So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?

Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.

The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.

Quoted 9 lines

I'm not sure what you're talking about as far as duplicates, I'm not seeing any duplicate devices. The problem I see is that "vgscan --mknodes" or "vgmknodes" will both create an additional symlink when compared to the udev rules, and the additional symlink is not deleted when I deactivate the volume group.

For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and

global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]

And now we are getting to the point on your problem:

"a|/dev/sda|" will also match "a|/dev/sda6|"
(and /dev/sda5...)

If you would like to get only 'exact' '/dev/sda' you would need to use

^/dev/sda$

otherwise '/dev/sda' may appear anywhere as substring of your device path.

Regards

Zdenek

#8 lore raw
Quoted 7 lines · Zdenek Kabelac

Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):

Quoted 6 lines

I'm using the stock CentOS7 version, I think.

LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0

Quoted 8 lines
Quoted 2 lines

So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?

Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.

The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.

Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't it make sense to have vgscan use the same logic as udev in terms of what symlinks to make and where to make them to?

Quoted 12 lines
Quoted 9 lines

I'm not sure what you're talking about as far as duplicates, I'm not seeing any duplicate devices. The problem I see is that "vgscan --mknodes" or "vgmknodes" will both create an additional symlink when compared to the udev rules, and the additional symlink is not deleted when I deactivate the volume group.

For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and

global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]

And now we are getting to the point on your problem:

Quoted 8 lines

"a|/dev/sda|" will also match "a|/dev/sda6|"
(and /dev/sda5...)

If you would like to get only 'exact' '/dev/sda' you would need to use

^/dev/sda$

otherwise '/dev/sda' may appear anywhere as substring of your device path.

Yes, the "/dev/sda" is intentional since we have multiple /dev/sda* devices that form part of various separate volume groups. Really it's the /dev/sda6 that is extra here since it's already matched.

The problematic issue I'm seeing with udev/vgscan is actually with the volume group on /dev/drbd4.

I really don't think I'm having any problems with duplicates...just with vgscan creating extra symlinks compared to udev, that udev doesn't clean up.

Chris

#9 lore raw

Dne 22.6.2016 v 16:52 Chris Friesen napsal(a):

Quoted 14 lines · Zdenek Kabelac
Quoted 13 lines

Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):

Quoted 8 lines

I'm using the stock CentOS7 version, I think.

LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0

So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?

Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.

Quoted 6 lines
Quoted 2 lines

The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.

Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't it make sense to have vgscan use the same logic as udev in terms of what symlinks to make and where to make them to?

It *IS* using same logic.

If the link is not there - the bug is in your udev rules.

When udev is properly configured, vgscan should not show missing link.

It's worth to note - 'dm' subsystem is in general 'ignored' by udev and 'dm' user has to provide proper rules. If your drbd device is missing, then something wrong is with rules...

Likely 'drbd' devices are not properly scanned by udev (and I assume they would not even work on system with use_lvmetad=1).

Quoted 26 lines
Quoted 12 lines
Quoted 9 lines

I'm not sure what you're talking about as far as duplicates, I'm not seeing any duplicate devices. The problem I see is that "vgscan --mknodes" or "vgmknodes" will both create an additional symlink when compared to the udev rules, and the additional symlink is not deleted when I deactivate the volume group.

For what it's worth, in my lvm.conf I have "use_lvmetad = 0" and

global_filter = [ "a|/dev/sda|", "a|/dev/drbd4|", "a|/dev/sda6|", "r|.*|" ]

And now we are getting to the point on your problem:

Quoted 8 lines

"a|/dev/sda|" will also match "a|/dev/sda6|"
(and /dev/sda5...)

If you would like to get only 'exact' '/dev/sda' you would need to use

^/dev/sda$

otherwise '/dev/sda' may appear anywhere as substring of your device path.

Yes, the "/dev/sda" is intentional since we have multiple /dev/sda* devices that form part of various separate volume groups. Really it's the /dev/sda6 that is extra here since it's already matched.

Quoted 6 lines

The problematic issue I'm seeing with udev/vgscan is actually with the volume group on /dev/drbd4.

I really don't think I'm having any problems with duplicates...just with vgscan creating extra symlinks compared to udev, that udev doesn't clean up.

Ahh - ok then - I thought you wanted to see only sda6 on your machine.

Regards

Zdenek

#10 lore raw
Quoted 22 lines · Zdenek Kabelac

Dne 22.6.2016 v 16:52 Chris Friesen napsal(a):

Quoted 14 lines · Zdenek Kabelac
Quoted 13 lines

Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):

Quoted 8 lines

I'm using the stock CentOS7 version, I think.

LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0

So are you saying that nobody should run "vgscan --mknodes" on a system where udev is managing the symlinks?

Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.

Quoted 6 lines
Quoted 2 lines

The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.

Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't it make sense to have vgscan use the same logic as udev in terms of what symlinks to make and where to make them to?

Quoted 5 lines

It *IS* using same logic.

If the link is not there - the bug is in your udev rules.

When udev is properly configured, vgscan should not show missing link.

It doesn't seem to work this way in practice on a stock CentOS system. Here's the sequence:

1) create a volume group:
"vgcreate chris-volumes /dev/loop2"
At this point there is no /dev/chris-volumes directory.

2) Create a thin pool in the volume group:
"lvcreate -L 1.8GB -T chris-volumes/chris-volumes-pool"

Now udev creates a /dev/chris-volumes directory with a link for the thin pool:
[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 7 Jun 23 12:22 chris-volumes-pool -> ../dm-9

3) Create a thin volume in the thin pool:
"lvcreate -V1G -T chris-volumes/chris-volumes-pool -n thinvolume"

Now the link for the thin pool itself has disappeared:
[root@centos7 centos]# ls -l /dev/chris-volumes total 0
lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11

(At this point /dev/mapper/chris--volumes-chris--volumes--pool-tpool points to dm-9 and /dev/mapper/chris--volumes-chris--volumes--pool points to dm-10.)

4) If I run "vgscan --mknodes", it re-creates the thin pool link, but pointing to the /dev/mapper name instead of directly to the /dev/dm-*. Also, it's indirectly pointing to /dev/dm-10 where before it was pointing to /dev/dm-9:

[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part of
any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.
Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2
Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2
The link /dev/chris-volumes/chris-volumes-pool should have been created by
udev but it was not found. Falling back to direct link creation.
[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 47 Jun 23 12:25 chris-volumes-pool -> /dev/mapper/chris--volumes-chris--volumes--pool lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11

5) If I run "vgchange -an chris-volumes", I'm left with a /dev/chris-volumes directory containing a broken symlink (broken because /dev/mapper/chris--volumes-chris--volumes--pool doesn't exist anymore):

[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 47 Jun 23 12:25 chris-volumes-pool -> /dev/mapper/chris--volumes-chris--volumes--pool

It looks like /usr/lib/udev/rules.d/11-dm-lvm.rules is what makes the /dev/<VG>/<LV> symlink for normal devices, but for a thin pool with a thin volume in it it will exit without making a symlink because DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1 is set.

[root@centos7 centos]# udevadm info /dev/chris-volumes/chris-volumes-pool
P: /devices/virtual/block/dm-10
N: dm-10
E: DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1

Chris

#11 lore raw

Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):

Quoted 28 lines · Zdenek Kabelac
Quoted 27 lines

Dne 22.6.2016 v 16:52 Chris Friesen napsal(a):

Quoted 15 lines · Zdenek Kabelac
Quoted 14 lines

Dne 21.6.2016 v 17:22 Chris Friesen napsal(a):

Quoted 9 lines

I'm using the stock CentOS7 version, I think.

LVM version: 2.02.130(2)-RHEL7 (2015-12-01)
Library version: 1.02.107-RHEL7 (2015-12-01)
Driver version: 4.33.0

So are you saying that nobody should run "vgscan --mknodes" on a system where
udev is managing the symlinks?

Yes - on such system this command should be used only in case of 'emergency', udev doesn't work properly and you need links.

Quoted 6 lines
Quoted 2 lines

The links however will not be known to udev and likely whole system is going to be crashing soon or is misconfigured in major way.

Okay, I'll see if I can get the call to vgscan removed. But even so wouldn't it make sense to have vgscan use the same logic as udev in terms of what symlinks to make and where to make them to?

It *IS* using same logic.

If the link is not there - the bug is in your udev rules.

Quoted 9 lines
Quoted 1 line

When udev is properly configured, vgscan should not show missing link.

It doesn't seem to work this way in practice on a stock CentOS system. Here's the sequence:

1) create a volume group:
"vgcreate chris-volumes /dev/loop2"
At this point there is no /dev/chris-volumes directory.

ONLY active volume do have links

Never ever there is supposed to be directory entry for VG without any active LV.

There is not a term 'active VG' as such - it always is in connection with active LV - thus directory without any active LV inside is pure bug - if you see it - report it as regular BZ..

Quoted 11 lines

2) Create a thin pool in the volume group:
"lvcreate -L 1.8GB -T chris-volumes/chris-volumes-pool"

Now udev creates a /dev/chris-volumes directory with a link for the thin pool:
[root@centos7 centos]# ls -l /dev/chris-volumes
total 0
lrwxrwxrwx. 1 root root 7 Jun 23 12:22 chris-volumes-pool -> ../dm-9

3) Create a thin volume in the thin pool:
"lvcreate -V1G -T chris-volumes/chris-volumes-pool -n thinvolume"

Quoted 7 lines

Now the link for the thin pool itself has disappeared:
[root@centos7 centos]# ls -l /dev/chris-volumes total 0
lrwxrwxrwx. 1 root root 8 Jun 23 12:23 thinvolume -> ../dm-11

(At this point /dev/mapper/chris--volumes-chris--volumes--pool-tpool points to dm-9 and /dev/mapper/chris--volumes-chris--volumes--pool points to dm-10.)

correct

Quoted 3 lines

4) If I run "vgscan --mknodes", it re-creates the thin pool link, but pointing to the /dev/mapper name instead of directly to the /dev/dm-*. Also, it's indirectly pointing to /dev/dm-10 where before it was pointing to /dev/dm-9:

Only 'final' resolving device does matter.

if link is /dev/vg/lv -> /dev/mapper/vg-lv -> /dev/dm-XXX
or /dev/vg/lv -> /dev/dm-XXX - it does not matter.

There are some 'tricks' related to thin-pool maintainance.

Unused thin-pool is 'public' LV - has /dev/vg/lv link.
Used thin-pool by lvm2 is 'private' LV - doesn't have /dev/vg/lv link.

All device should always have entry in /dev/mapper/ - either links to real devices or direct nodes (on older systems)

lvm2 users are always supposed to use ONLY /dev/vg/lv devices for access.

Quoted 6 lines

[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part
of any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.

fix your lvm.conf (uncomment sections)

Quoted 5 lines

Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2 Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2 The link /dev/chris-volumes/chris-volumes-pool should have been created by

Ok - there seems to be internal bug in lvm2 - which incorrectly hints link creation for this case.

There should not have been /dev/vg/pool link - this is correctly marked for udev - but incorrectly for udev validation.

However the bug is actually not so much important - it just links
to 'wrapper' device - and eventually we will resolve the problem even without this extra device in table.

Regards

Zdenek

#12 lore raw
Quoted 19 lines · Zdenek Kabelac

Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):

Quoted 5 lines

[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part
of any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.

fix your lvm.conf (uncomment sections)

Quoted 5 lines

Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2 Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2 The link /dev/chris-volumes/chris-volumes-pool should have been created by

Ok - there seems to be internal bug in lvm2 - which incorrectly hints link creation for this case.

Quoted 6 lines

There should not have been /dev/vg/pool link - this is correctly marked for udev - but incorrectly for udev validation.

However the bug is actually not so much important - it just links to 'wrapper' device - and eventually we will resolve the problem even without this extra device in table.

The problem that it causes for me is that when I run "vgchange -an chris-volumes" it leaves the /dev/chris-volumes with a broken symlink in it because udev doesn't remove the symlink added by vgscan.

This causes the LVM OCF script in the "resource-agents" package to break, because it is using the existance of the /dev/vg directory as a proxy for whether the volume group is active (or really as you said earlier, whether there are active volumes within the volume group).

I reported this as a bug to the "resource-agents" package developers, and they said that they can't actually call lvm commands in their "status" routines because there have been cases where clustered LVM hung when querying status, causing the OCF script to hang and monitoring to fail.

Ultimately I'll see if I can work around it by not calling "vgscan --mknodes". Originally it was added in to fix some problems, but that was a while back so things may behave properly now.

Thanks for your help,
Chris

#13 lore raw

Dne 23.6.2016 v 20:02 Chris Friesen napsal(a):

Quoted 23 lines · Zdenek Kabelac
Quoted 22 lines

Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):

Quoted 5 lines

[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part
of any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.

fix your lvm.conf (uncomment sections)

Quoted 5 lines

Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2 Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2 The link /dev/chris-volumes/chris-volumes-pool should have been created by

Ok - there seems to be internal bug in lvm2 - which incorrectly hints link creation for this case.

There should not have been /dev/vg/pool link - this is correctly marked for udev - but incorrectly for udev validation.

Quoted 7 lines
Quoted 3 lines

However the bug is actually not so much important - it just links to 'wrapper' device - and eventually we will resolve the problem even without this extra device in table.

The problem that it causes for me is that when I run "vgchange -an chris-volumes" it leaves the /dev/chris-volumes with a broken symlink in it because udev doesn't remove the symlink added by vgscan.

Yep - as said - in normal circumstance use should NOT run 'vgmknodes' as the created links will not be known/visible to udev - so this behavior is correct.

Quoted 11 lines

This causes the LVM OCF script in the "resource-agents" package to break, because it is using the existance of the /dev/vg directory as a proxy for whether the volume group is active (or really as you said earlier, whether there are active volumes within the volume group).

I reported this as a bug to the "resource-agents" package developers, and they said that they can't actually call lvm commands in their "status" routines because there have been cases where clustered LVM hung when querying status, causing the OCF script to hang and monitoring to fail.

Ultimately I'll see if I can work around it by not calling "vgscan --mknodes".

yes please, start with this one...

vgmknodes are really supposed to be used on some unique urgent problem - not executed in script every hour...

But yes - lvm2 will need to fix link creation of pool-in-use...

Quoted 2 lines

Originally it was added in to fix some problems, but that was a while back so things may behave properly now.

Regards

Zdenek

#14 lore raw
In reply to Chris Friesen

Dne 23.6.2016 v 20:02 Chris Friesen napsal(a):

Quoted 36 lines · Zdenek Kabelac
Quoted 19 lines

Dne 23.6.2016 v 18:35 Chris Friesen napsal(a):

Quoted 5 lines

[root@centos7 centos]# vgscan --mknodes
Configuration setting "snapshot_autoextend_percent" invalid. It's not part
of any section.
Configuration setting "snapshot_autoextend_threshold" invalid. It's not part
of any section.

fix your lvm.conf (uncomment sections)

Quoted 5 lines

Reading all physical volumes. This may take a while...
Found volume group "chris-volumes" using metadata type lvm2 Found volume group "centos" using metadata type lvm2
Found volume group "cinder-volumes" using metadata type lvm2 The link /dev/chris-volumes/chris-volumes-pool should have been created by

Ok - there seems to be internal bug in lvm2 - which incorrectly hints link creation for this case.

Quoted 6 lines

There should not have been /dev/vg/pool link - this is correctly marked for udev - but incorrectly for udev validation.

However the bug is actually not so much important - it just links to 'wrapper' device - and eventually we will resolve the problem even without this extra device in table.

The problem that it causes for me is that when I run "vgchange -an chris-volumes" it leaves the /dev/chris-volumes with a broken symlink in it because udev doesn't remove the symlink added by vgscan.

This causes the LVM OCF script in the "resource-agents" package to break, because it is using the existance of the /dev/vg directory as a proxy for whether the volume group is active (or really as you said earlier, whether there are active volumes within the volume group).

Quoted 8 lines

I reported this as a bug to the "resource-agents" package developers, and they said that they can't actually call lvm commands in their "status" routines because there have been cases where clustered LVM hung when querying status, causing the OCF script to hang and monitoring to fail.

Ultimately I'll see if I can work around it by not calling "vgscan --mknodes". Originally it was added in to fix some problems, but that was a while back so things may behave properly now.

Should be resolved with this upstream commit:

https://www.redhat.com/archives/lvm-devel/2016-June/msg00187.html

Regards

Zdenek