Removing VG mappings using dmsetup tool
Hi,
I'm working on storage part of oVirt project [1]. Besides other options, we provide iSCSI storage for VMs. We create PV(s) on attached LUN(s), create volume group from this PV(s) and then create LVs on this VG as needed (each disk on its LV, similar there's dedicated LV for our metadata etc.). VG is what we call (block) storage domain.
When we remove storage domain we deactivate all LVs and remove LVs and VG from all hosts. However, when this fails (e.g. storage is for some reason unavailable), hosts are left with stale LVs and VG which can cause various issues for us. What we are going to do is to remove these stale VG mappings using dmsetup remove command (first we try to remove LVs using lvm, but if there are still some mappings, we will try to remove it using dmsetup).
I'd like to head your opinion on this approach. Is it wise to use dmsetup in such case? Or is there any better way how to handle such situations or remove stale mappings?
Thanks
Vojta
Dne 23. 06. 20 v 16:26 Vojtech Juranek napsal(a):
Quoted 14 lines
Hi,
I'm working on storage part of oVirt project [1]. Besides other options, we provide iSCSI storage for VMs. We create PV(s) on attached LUN(s), create volume group from this PV(s) and then create LVs on this VG as needed (each disk on its LV, similar there's dedicated LV for our metadata etc.). VG is what we call (block) storage domain.When we remove storage domain we deactivate all LVs and remove LVs and VG from all hosts. However, when this fails (e.g. storage is for some reason unavailable), hosts are left with stale LVs and VG which can cause various issues for us. What we are going to do is to remove these stale VG mappings using dmsetup remove command (first we try to remove LVs using lvm, but if there are still some mappings, we will try to remove it using dmsetup).
Quoted 3 lines
I'd like to head your opinion on this approach. Is it wise to use dmsetup in such case? Or is there any better way how to handle such situations or remove stale mappings?
Hi
Of course you can remove 'stale' mapping with 'dmsetup remove' - (after all lvm2 is doing basically very similar job like 'dmsetup create/remove/load/suspend/resume'....
BUT I'd probably focus rather on fixing your oVirt workflow - you should not be 'losing' devices while you have activate mappings on them - that's IMHO major desing flaw - you can very easily cause very SERIOUS damages to your on disk data consistency - so disk would require serious fsck before reuse...
Note - you cannot 'remove' mappings 'in-use' (aka open count of a device is higher then 0 - see 'dmsetup info -c' output for this). However you can replace such mapping with 'error' target - so the underlaying device is relaxed - although we do not support this in lvm2 - you would need to use 'dmsetup' for this (and open lvm2 RFE if there would be some serious justifaction).
Regards
Zdenek
Il 2020-06-23 22:28 Zdenek Kabelac ha scritto:
Quoted 6 lines
Note - you cannot 'remove' mappings 'in-use' (aka open count of a device is higher then 0 - see 'dmsetup info -c' output for this). However you can replace such mapping with 'error' target - so the underlaying device is relaxed - although we do not support this in lvm2 - you would need to use 'dmsetup' for this (and open lvm2 RFE if there would be some serious justifaction).
Not related to oVirt, but I would find a means to set an error target with LVM quite useful - think about immediately stopping IO to/from device in case some serious error happended. Using lvm rather than dmsetup would be easier and less error prone.
Thanks.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it [1] email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Dne 23. 06. 20 v 22:37 Gionatan Danti napsal(a):
Quoted 9 lines
Il 2020-06-23 22:28 Zdenek Kabelac ha scritto:
Quoted 6 lines
Note - you cannot 'remove' mappings 'in-use'� (aka open count of a device is higher then 0� - see 'dmsetup info -c' output for this). However you can replace such mapping with 'error' target - so the underlaying device is relaxed - although we do not support this in lvm2 - you would need to use 'dmsetup' for this (and open lvm2 RFE if there would be some serious justifaction).
Not related to oVirt, but I would find a means to set an error target with LVM quite useful - think about immediately stopping IO to/from device in case some serious error happended. Using lvm rather than dmsetup would be easier and less error prone.
Hi
ATM skilled admin can always easily enforce:
'dmsetup remove --force vg-lv'
for i.e. linear devices to achieve this goal - however resolving this at lvm2 is actually way more complex task when you start to consider the situation should be at least 'somehow' recoverable - it's quite complicated and
not really highly demanded functionality.
It's more simple if you have constrained world of known types of devices
and known use-case you are targeting to solve.
Regards
Zdenek
Il 2020-06-23 23:02 Zdenek Kabelac ha scritto:
Quoted 5 lines
Hi
ATM skilled admin can always easily enforce:
'dmsetup remove --force vg-lv'
Hi Zdenek,
sure, but I find messing with dmsetup more error prone than using an LVM command.
Quoted 7 lines
for i.e. linear devices to achieve this goal - however resolving this at lvm2 is actually way more complex task when you start to consider the situation
should be at least 'somehow' recoverable - it's quite complicated and not really highly demanded functionality.
It's more simple if you have constrained world of known types of devices and known use-case you are targeting to solve.
Would be an initial minimal support userful? For example, something as "lvchange --error vg/lvm" which will only work on simple setup, while on more complex ones would simply print a warning and return with exit code 1.
Thanks.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it [1] email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Dne 23. 06. 20 v 23:34 Gionatan Danti napsal(a):
Quoted 18 lines
Il 2020-06-23 23:02 Zdenek Kabelac ha scritto:
Quoted 5 lines
Hi
ATM skilled admin can always easily enforce:
'dmsetup remove --force�� vg-lv'
Hi Zdenek,
sure, but I find messing with dmsetup more error prone than using an LVM command.Quoted 7 lines
for i.e. linear devices to achieve this goal - however resolving this at lvm2 is actually way more complex task when you start to consider the situation
should be at least 'somehow' recoverable - it's quite complicated and not really highly demanded functionality.
It's more simple if you have constrained world of known types of devices and known use-case you are targeting to solve.
Quoted 2 lines
Would be an initial minimal support userful? For example, something as "lvchange --error vg/lvm" which will only work on simple setup, while on more complex ones would simply print a warning and return with exit code 1.
There are 'couple' issue -
a) it would be nice to have this 'change' permanent -> metadata update ->
but likely this user runs when some devices got missing - so lots of related trouble with 'MISSING' devices and metadata update.
b) when this would be only '--replacewitherror y|n' lvchange modifier - then
with next i.e. lvchange --refresh (or any other 'transient' table update) you get very same problematic reference of original PV.
c) it's unclear if we want to actually provide this as 'lvremove' feature - as
IMHO when 'error' replacement is required - the most cleanest proceed is to never use this device again - and ATM you can't remove 'device' in use - but with this kind of support we could probably migrate such device into
just 'error' placeholders (removable once devices are unused) and space would
be relased in VG (this would be probably my favorite path to add support
for this kind of forcible removal of used LV).
Regards
Zdenek
Quoted 5 lines
BUT I'd probably focus rather on fixing your oVirt workflow - you should not be 'losing' devices while you have activate mappings on them - that's IMHO major desing flaw - you can very easily cause very SERIOUS damages to your on disk data consistency - so disk would require serious fsck before reuse...
Thanks for your answer. Just to clarify - we are not going to loose devices intentionally, we are considering this option only for recovery from situation when storage is suddenly lost (e.g. broken network, storage sever crashed etc.). In these situations disk data consistency would be an issue anyway.
Thanks
Vojta