Why does every lvm command insist on touching every pv?
Every time I run any lvm command, it goes out and touches every pv, causing all the disks to wake up from standby. Why is this? Shouldn't a simple lvs command be able to look in the /etc/lvm/cache for what it needs instead of touching every pv, even ones with no lvs on them?
Quoted 4 lines · Phillip Susi
Every time I run any lvm command, it goes out and touches every pv, causing all the disks to wake up from standby. Why is this? Shouldn't a simple lvs command be able to look in the
/etc/lvm/cachefor what it needs instead of touching every pv, even ones with no lvs on them?
It shouldn't normally scan unless it detects something might have changed, but if you know the volume group name(s) already, put them on the command line:
lvs vg1 vg2
(As always -vvvv may give you clues as to what is triggering the scan.)
Alasdair
Quoted 5 lines · Alasdair G Kergon
It shouldn't normally scan unless it detects something might have changed, but if you know the volume group name(s) already, put them on the command line:
lvs vg1 vg2(As always -vvvv may give you clues as to what is triggering the scan.)
I can see from the output with -vvvv that it is indeed trying to find a pv label on every block device in /dev. I thought that I had figured out this was because lvm.conf was setting scan = [ "/dev" ] in the devices section, but I tried removing /dev, and just setting it to "", as well as "none" and lvs -vvvv STILL is opening every block device in /dev.
Shouldn't this not happen if you leave the value of scan empty, and shouldn't it be left empty when you have udev running lvscan automatically on block devices that contain physical volumes?
Quoted 5 lines · Phillip Susi
Quoted 4 lines · Alasdair G Kergon
It shouldn't normally scan unless it detects something might have changed, but if you know the volume group name(s) already, put them on the command line:
lvs vg1 vg2
Quoted 7 lines
Quoted 1 line
(As always -vvvv may give you clues as to what is triggering the scan.)
I can see from the output with -vvvv that it is indeed trying to find a pv label on every block device in /dev. I thought that I had figured out this was because lvm.conf was setting scan = [ "/dev" ] in the devices section, but I tried removing /dev, and just setting it to "", as well as "none" and lvs -vvvv STILL is opening every block device in /dev.
Everytime lvm command is executed, all PVs are accessed.
This is a known issue and what we should fix. If there are a lot of PVs, execution of lvm command could take a long time. The scan also affects error recovery time of lvm mirror.
This is an old reference.
Introduce metadata cache feature
https://www.redhat.com/archives/lvm-devel/2009-April/msg00014.html
Thanks,
Taka
Quoted 10 lines · Takahiro Yasui
Everytime lvm command is executed, all PVs are accessed.
This is a known issue and what we should fix. If there are a lot of PVs, execution of lvm command could take a long time. The scan also affects error recovery time of lvm mirror.
This is an old reference.
Introduce metadata cache feature
https://www.redhat.com/archives/lvm-devel/2009-April/msg00014.html
This seems to limit the number of metadata copies on pvs that are accessed to one, but I am seeing even block devices that are not pvs accessed, like it is scanning for new pvs. This patch seems like it would help some though, but it is over a year old. Did it never get applied?
Quoted 9 lines · Phillip Susi
Quoted 8 lines · Takahiro Yasui
Everytime lvm command is executed, all PVs are accessed.
This is a known issue and what we should fix. If there are a lot of PVs, execution of lvm command could take a long time. The scan also affects error recovery time of lvm mirror.
This is an old reference.
Quoted 8 lines
Quoted 2 lines
Introduce metadata cache feature
https://www.redhat.com/archives/lvm-devel/2009-April/msg00014.htmlThis seems to limit the number of metadata copies on pvs that are accessed to one, but I am seeing even block devices that are not pvs accessed, like it is scanning for new pvs. This patch seems like it would help some though, but it is over a year old. Did it never get applied?
As for accessing block devices, you can limit accesses to those devices by filter option in lvm.conf. This is an example which allows accesses only to /dev/sd*
filter = [ "a|/dev/sd|", "r|.*|" ]
You can specify more strict rule like
filter = [ "a|/dev/sd[a-z]|", "r|.*|" ]
For my patch, I posted it one year ago but I need to have more discussion how to solve this issue on lvm-devel.
Thanks,
Taka
Quoted 9 lines · Takahiro Yasui
As for accessing block devices, you can limit accesses to those devices by filter option in lvm.conf. This is an example which allows accesses only to
/dev/sd*filter = [ "a|/dev/sd|", "r|.*|" ]You can specify more strict rule like
filter = [ "a|/dev/sd[a-z]|", "r|.*|" ]
What if I don't want ANY devices to be scanned every time an lvm command is run? Shouldn't they be scanned once when udev first detects they have been attached, and no more? I thought removing /dev from the scan= line would do that, but it didn't.
Dne 16.6.2010 02:34, Phillip Susi napsal(a):
Quoted 7 lines · Takahiro Yasui
Quoted 6 lines
As for accessing block devices, you can limit accesses to those devices by filter option in lvm.conf. This is an example which allows accesses only to
/dev/sd*filter = [ "a|/dev/sd|", "r|.*|" ]
Quoted 9 lines
Quoted 3 lines
You can specify more strict rule like
filter = [ "a|/dev/sd[a-z]|", "r|.*|" ]What if I don't want ANY devices to be scanned every time an lvm command is run? Shouldn't they be scanned once when udev first detects they have been attached, and no more? I thought removing /dev from the scan= line would do that, but it didn't.
I'd have say - this is work-in-progress for now - yes, udev should handle all this for lvm - but we are not yet there...
Meanwhile - you may write few scripts which will pass exact filters for your lvm commands if you know you do not need to touch those devices.
Zdenek
Quoted 2 lines · Zdenek Kabelac
Dne 16.6.2010 02:34, Phillip Susi napsal(a):
Quoted 1 line · Takahiro Yasui
...
Quoted 8 lines
Quoted 5 lines
What if I don't want ANY devices to be scanned every time an lvm command is run? Shouldn't they be scanned once when udev first detects they have been attached, and no more? I thought removing /dev from the scan= line would do that, but it didn't.
I'd have say - this is work-in-progress for now - yes, udev should handle all this for lvm - but we are not yet there...
Hi Zdenek,
Sorry I passed over your post.
May I ask you how you are going to handle this issue by udev? The problem is *not* a scan of devices to update dev-cache but a scan of devices to update lvm metadata cache in memory by lvm command. Currently lvm does not have a kind of metadata cache on a device (e.g. /etc/lvm) and lvm command has to get metadata information stored on each PV whenever it is executed.
It is helpful if udev can handle this issue, but I'm wondering how it can do it.
Thanks,
Taka
Dne 16.6.2010 21:27, Takahiro Yasui napsal(a):
Quoted 24 lines · Zdenek Kabelac
Quoted 2 lines
Dne 16.6.2010 02:34, Phillip Susi napsal(a):
Quoted 1 line · Takahiro Yasui
...
Quoted 8 lines
Quoted 5 lines
What if I don't want ANY devices to be scanned every time an lvm command is run? Shouldn't they be scanned once when udev first detects they have been attached, and no more? I thought removing /dev from the scan= line would do that, but it didn't.
I'd have say - this is work-in-progress for now - yes, udev should handle all this for lvm - but we are not yet there...
Hi Zdenek,
Sorry I passed over your post.
May I ask you how you are going to handle this issue by udev? The problem is *not* a scan of devices to update dev-cache but a scan of devices to update lvm metadata cache in memory by lvm command. Currently lvm does not have a kind of metadata cache on a device (e.g.
/etc/lvm) and lvm command has to get metadata information stored on each PV whenever it is executed.
Quoted 2 lines
It is helpful if udev can handle this issue, but I'm wondering how it can do it.
I'm not working on this part, but AFAIK, once we could start 'trust' udev, we can keep persistent cache aware of any changes that might have happened to devices listed in metadata. Implementation details are still 'moving topic'.
Obviously you can not skip write/update access to metadata areas, but it should be possible to avoid scanning for 'read-only' data access.
Also there is another thing in progress - metadata-balance code - where you essentially do not need to read/write metadata from/to every PV in VG - but just on reasonable safe amount of them - i.e. 5 from 100 of PVs - the rest of them is marked invisible (different from pvcreate --metadatasize 0)
Zdenek
Quoted 10 lines · Zdenek Kabelac
Dne 16.6.2010 21:27, Takahiro Yasui napsal(a):
Quoted 9 lines · Zdenek Kabelac
Quoted 2 lines
Dne 16.6.2010 02:34, Phillip Susi napsal(a):
Quoted 1 line · Takahiro Yasui
...
Quoted 5 lines
Quoted 5 lines
What if I don't want ANY devices to be scanned every time an lvm command is run? Shouldn't they be scanned once when udev first detects they have been attached, and no more? I thought removing /dev from the scan= line would do that, but it didn't.
...
Quoted 9 lines
Quoted 2 lines
It is helpful if udev can handle this issue, but I'm wondering how it can do it.
I'm not working on this part, but AFAIK, once we could start 'trust' udev, we can keep persistent cache aware of any changes that might have happened to devices listed in metadata. Implementation details are still 'moving topic'.
Obviously you can not skip write/update access to metadata areas, but it should be possible to avoid scanning for 'read-only' data access.
Thank you for your explanation. Yes, I agree that it is possible to avoid scanning for 'read-only' data access, but I also believe it is possible for 'write' adata access.
Quoted 4 lines
Also there is another thing in progress - metadata-balance code - where you essentially do not need to read/write metadata from/to every PV in VG - but just on reasonable safe amount of them - i.e. 5 from 100 of PVs - the rest of them is marked invisible (different from pvcreate --metadatasize 0)
AFAIK, metadata-balance feature would reduce the number of disk accesses, but I believe that the goal is to access PVs related to the VG which lvm command is going to manipulate. Introducing metadata cache feature on disk or a kind of daemon managing all metadatas, or using /etc/lvm/backup could be solution.
I hope we could continue discussing this topic on lvm-devel?
Thanks,
Taka
Dne 17.6.2010 15:53, Takahiro Yasui napsal(a):
Quoted 19 lines · Zdenek Kabelac
Quoted 10 lines
Dne 16.6.2010 21:27, Takahiro Yasui napsal(a):
Quoted 9 lines · Zdenek Kabelac
Quoted 2 lines
Dne 16.6.2010 02:34, Phillip Susi napsal(a):
Quoted 1 line · Takahiro Yasui
...
Quoted 5 lines
Quoted 5 lines
What if I don't want ANY devices to be scanned every time an lvm command is run? Shouldn't they be scanned once when udev first detects they have been attached, and no more? I thought removing /dev from the scan= line would do that, but it didn't.
...
Quoted 7 lines
Quoted 2 lines
It is helpful if udev can handle this issue, but I'm wondering how it can do it.
I'm not working on this part, but AFAIK, once we could start 'trust' udev, we can keep persistent cache aware of any changes that might have happened to devices listed in metadata. Implementation details are still 'moving topic'.
Quoted 6 lines
Quoted 2 lines
Obviously you can not skip write/update access to metadata areas, but it should be possible to avoid scanning for 'read-only' data access.
Thank you for your explanation. Yes, I agree that it is possible to avoid scanning for 'read-only' data access, but I also believe it is possible for 'write' adata access.
With current LVM logic - you can't proceed with usage of LVM metadata unless they are properly committed to PVs. i.e. there is no chance you could use partially stored metadata to just some cached devices. Either you update all metadata or you fail - there is nothing between these 2 states.
Quoted 6 lines
Quoted 5 lines
Also there is another thing in progress - metadata-balance code - where you essentially do not need to read/write metadata from/to every PV in VG - but just on reasonable safe amount of them - i.e. 5 from 100 of PVs - the rest of them is marked invisible (different from pvcreate --metadatasize 0)
Quoted 7 lines
AFAIK, metadata-balance feature would reduce the number of disk accesses, but I believe that the goal is to access PVs related to the VG which lvm command is going to manipulate. Introducing metadata cache feature on disk or a kind of daemon managing all metadatas, or using
/etc/lvm/backupcould be solution.I hope we could continue discussing this topic on lvm-devel?
Sure. Daemon is also planned, but for reduction of write access metadata balancing should greatly help. Another step here could be to parallelize all disk operations on different devices. Also udev handling has some perfomance optimalization still.
With properly working udev we shouldn't need to do any device scanning as we will have all 'interesting' devices stored in some cache storage - it could be file, daemon, udev DB entry....
Zdenek
Quoted 28 lines · Zdenek Kabelac
Dne 17.6.2010 15:53, Takahiro Yasui napsal(a):
Quoted 19 lines · Zdenek Kabelac
Quoted 10 lines
Dne 16.6.2010 21:27, Takahiro Yasui napsal(a):
Quoted 9 lines · Zdenek Kabelac
Quoted 2 lines
Dne 16.6.2010 02:34, Phillip Susi napsal(a):
Quoted 1 line · Takahiro Yasui
...
Quoted 5 lines
Quoted 5 lines
What if I don't want ANY devices to be scanned every time an lvm command is run? Shouldn't they be scanned once when udev first detects they have been attached, and no more? I thought removing /dev from the scan= line would do that, but it didn't.
...
Quoted 7 lines
Quoted 2 lines
It is helpful if udev can handle this issue, but I'm wondering how it can do it.
I'm not working on this part, but AFAIK, once we could start 'trust' udev, we can keep persistent cache aware of any changes that might have happened to devices listed in metadata. Implementation details are still 'moving topic'.
Quoted 6 lines
Quoted 2 lines
Obviously you can not skip write/update access to metadata areas, but it should be possible to avoid scanning for 'read-only' data access.
Thank you for your explanation. Yes, I agree that it is possible to avoid scanning for 'read-only' data access, but I also believe it is possible for 'write' adata access.
Quoted 4 lines
With current LVM logic - you can't proceed with usage of LVM metadata unless they are properly committed to PVs. i.e. there is no chance you could use partially stored metadata to just some cached devices. Either you update all metadata or you fail - there is nothing between these 2 states.
I agree with you comment in terms that every metadata stored on PVs in the same VG which are being manipulated should be committed, but I don't think metadata stored on PVs which belong to *different* VGs need to be committed.
For example, there are six PVs and two VGs as below:
VG1: PV1, PV2, PV3
VG2: PV4, PV5, PV6
If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3 need to be updated but not for PV4, PV5, PV6.
Quoted 13 lines
Quoted 5 lines
Quoted 4 lines
Also there is another thing in progress - metadata-balance code - where you essentially do not need to read/write metadata from/to every PV in VG - but just on reasonable safe amount of them - i.e. 5 from 100 of PVs - the rest of them is marked invisible (different from pvcreate --metadatasize 0)
Quoted 7 lines
AFAIK, metadata-balance feature would reduce the number of disk accesses, but I believe that the goal is to access PVs related to the VG which lvm command is going to manipulate. Introducing metadata cache feature on disk or a kind of daemon managing all metadatas, or using
/etc/lvm/backupcould be solution.I hope we could continue discussing this topic on lvm-devel?
Quoted 8 lines
Sure. Daemon is also planned, but for reduction of write access metadata balancing should greatly help. Another step here could be to parallelize all disk operations on different devices. Also udev handling has some perfomance optimalization still.
With properly working udev we shouldn't need to do any device scanning as we will have all 'interesting' devices stored in some cache storage - it could be file, daemon, udev DB entry....
I agree that these approach improve performance, but I don't think accesses to devices which don't belong to a VG being handled are not welcomed.
Thanks,
Taka
Quoted 7 lines · Takahiro Yasui
For example, there are six PVs and two VGs as below:
VG1: PV1, PV2, PV3
VG2: PV4, PV5, PV6If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3 need to be updated but not for PV4, PV5, PV6.
Correct - and the tools have always behaved like that!
(I don't think this thread is going anywhere useful.)
Alasdair
Quoted 10 lines · Alasdair G Kergon
Quoted 7 lines · Takahiro Yasui
For example, there are six PVs and two VGs as below:
VG1: PV1, PV2, PV3
VG2: PV4, PV5, PV6If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3 need to be updated but not for PV4, PV5, PV6.
Correct - and the tools have always behaved like that!
Did I miss something? Or can we configure lvm.conf to make lvm commands behave like that? That's the first topic we were talking about. As far as I use lvm commands, they don't behave as I expected.
Thanks,
Taka
Quoted 7 lines · Takahiro Yasui
Quoted 6 lines · Alasdair G Kergon
Quoted 5 lines · Takahiro Yasui
For example, there are six PVs and two VGs as below:
VG1: PV1, PV2, PV3
VG2: PV4, PV5, PV6
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3 need to be updated but not for PV4, PV5, PV6.
Correct - and the tools have always behaved like that!
Did I miss something? Or can we configure lvm.conf to make lvm commands behave like that? That's the first topic we were talking about. As far as I use lvm commands, they don't behave as I expected.
I'm sorry. My explanation was not good. Only PV1-3 are updated and PV4-6 are not. LVM command have always behaved like this.
I should have added an more sentence like
If we create a new LV or delete VG1, then metadatas on only PV1, PV2, PV3 need to be updated but not for PV4, PV5, PV6. So PV4-6 don't necessarily need to be accessed.
Sorry for making you confused.
Thanks,
Taka
Quoted 7 lines · Phillip Susi
Quoted 6 lines · Takahiro Yasui
As for accessing block devices, you can limit accesses to those devices by filter option in lvm.conf. This is an example which allows accesses only to
/dev/sd*filter = [ "a|/dev/sd|", "r|.*|" ]
Quoted 8 lines
Quoted 3 lines
You can specify more strict rule like
filter = [ "a|/dev/sd[a-z]|", "r|.*|" ]What if I don't want ANY devices to be scanned every time an lvm command is run? Shouldn't they be scanned once when udev first detects they have been attached, and no more? I thought removing /dev from the scan= line would do that, but it didn't.
I understand your requests and I have the same opinion as yours. My suggestion of 'filter' option just reduces the number of accesses to disks which are not related to PVs. Hopefully I would like to find a good solution and fix for this issue.
I think that this kind of options are very helpful to give increased priority to this issue among LVM community.
Thanks,
Taka