Why do lvcreate with clvmd insist on VG being available on all nodes?
Hello,
I am building a system where I use clustered LVM on a DRBD to provide shared block devices in a cluster. And I don't quite like and quite not understand some behaviour.
Currently I have two nodes in the cluster, running: Corosync, DLM, clvmd, DRBD, Pacemaker and my service.
Everything works fine when both nodes are up. When I put one to standby with 'crm node node1 standby' (which, among others, stops the DRBD) the other note is not fully functional.
If I leave DLM and CLVMD running on the inactive node, then:
lvchange -aey shared_vg/vol_name
lvchange -aen shared_vg/vol_name
work properly, as I would expect (make the volume available/unavailable on the node). But an attempt to create a new volume:
lvcreate -n new_volume -L 1M shared_vg
fails with:
Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
Indeed, the VG is not available at the standby node at that moment. But, as it is not available there, I see no point in locking it there.
Is there some real, important reason to block lvcreate in such case?
I have also tried stopping (cleanly) dlm_controld and clvmd on the standby node, hoping LVM will then work like in a single-node cluster, but then even volume activation fails with:
cluster request failed: Host is down
…until I restart clvmd on the active host with 'clvmd -S'.
When clvmd is stopped on the inactive node and 'clvmd -S' has been run on the active node, then both 'lvchange' and 'lvcreate' work as expected, but that doesn't look like a graceful switch-over. And another 'clvmd -S' stopped clvmd all together (this seems like a bug to me)
And one more thing bothers me… my system would be very scalable to many nodes, where only two share active storage (when using DRBD). But this won't work if LVM would refuse some operations when any VG is not available on all nodes.
Greets,
Jacek
Dne 14.11.2012 16:16, Jacek Konieczny napsal(a):
Quoted 18 lines
Hello,
I am building a system where I use clustered LVM on a DRBD to provide shared block devices in a cluster. And I don't quite like and quite not understand some behaviour.
Currently I have two nodes in the cluster, running: Corosync, DLM, clvmd, DRBD, Pacemaker and my service.
Everything works fine when both nodes are up. When I put one to standby with 'crm node node1 standby' (which, among others, stops the DRBD) the other note is not fully functional.
If I leave DLM and CLVMD running on the inactive node, then:
lvchange -aey shared_vg/vol_name
lvchange -aen shared_vg/vol_name
Quoted 9 lines
work properly, as I would expect (make the volume available/unavailable on the node). But an attempt to create a new volume:
lvcreate -n
new_volume-L 1Mshared_vgfails with:
Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
Haven't really tried to understand what are you trying to achieve, but if you want to have node being activated only on one cluster node,
you may easily use lvcreate -aey option.
If you are using default clustered operation - it's not surprising, the operation is refused if other nodes are not responding.
Quoted 2 lines
Indeed, the VG is not available at the standby node at that moment. But, as it is not available there, I see no point in locking it there.
Well - you would need to write your own type of locking with support of 'standby' currently clvmd doesn't work with such state (and it's not quite clear to me how it actually should even work).
So far either node is in cluster or is fenced.
Quoted 2 lines
Is there some real, important reason to block lvcreate in such case?
As long as you would use exclusive activation for lvcreate it should work. (or maybe 'local' - just test - but since you are trying to use unsupported operational mode - you need to take responsibility for the results)
Quoted 10 lines
When clvmd is stopped on the inactive node and 'clvmd -S' has been run on the active node, then both 'lvchange' and 'lvcreate' work as expected, but that doesn't look like a graceful switch-over. And another 'clvmd -S' stopped clvmd all together (this seems like a bug to me)
And one more thing bothers me… my system would be very scalable to many nodes, where only two share active storage (when using DRBD). But this won't work if LVM would refuse some operations when any VG is not available on all nodes.
Obviously using clustered VG in non-clustered environment isn't smart plan.
What you could do - is to disable clustering support on VG
vgchange -cn <vg> --config 'global {locking_type = 0}'
Note - you may always go around any locking problem with the above config option - just do not report then problems with broken disk content and badly activated volumes on cluster nodes.
Zdenek
Quoted 11 lines · Zdenek Kabelac
Quoted 9 lines
work properly, as I would expect (make the volume available/unavailable on the node). But an attempt to create a new volume:
lvcreate -n
new_volume-L 1Mshared_vgfails with:
Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
Quoted 6 lines
Haven't really tried to understand what are you trying to achieve, but if you want to have node being activated only on one cluster node,
you may easily use lvcreate -aey option.If you are using default clustered operation - it's not surprising, the operation is refused if other nodes are not responding.
Hmmm didn't though about the initial activation. In fact, I don't need that volume activated as soon as it is created. You are right, I should try 'lvcreate -aey' or 'lvcreate -an'.
My stupid mistake, indeed.
'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.
Though, LVM have some problems with tracking the exclusive activations later…
Quoted 7 lines
Quoted 2 lines
Indeed, the VG is not available at the standby node at that moment. But, as it is not available there, I see no point in locking it there.
Well - you would need to write your own type of locking with support of 'standby' currently clvmd doesn't work with such state (and it's not quite clear to me how it actually should even work).
So far either node is in cluster or is fenced.
I will try to make this work somehow… I don't think a node in standby (or when one of the VGs is not available there) should be quite different that one fenced.
Quoted 5 lines
Quoted 5 lines
When clvmd is stopped on the inactive node and 'clvmd -S' has been run on the active node, then both 'lvchange' and 'lvcreate' work as expected, but that doesn't look like a graceful switch-over. And another 'clvmd -S' stopped clvmd all together (this seems like a bug to me)
Quoted 7 lines
Quoted 4 lines
And one more thing bothers me… my system would be very scalable to many nodes, where only two share active storage (when using DRBD). But this won't work if LVM would refuse some operations when any VG is not available on all nodes.
Obviously using clustered VG in non-clustered environment isn't smart plan.
What you could do - is to disable clustering support on VG
Clusters do not have to be symmetrical. Cluster when different nodes have a bit different set of resources available are still clusters.
I do need clustering locking – in case a volume group is available on a few nodes it must not be possible to more than one node use any of the logical volumes there.
Quoted 3 lines
Note - you may always go around any locking problem with the above config option - just do not report then problems with broken disk content and badly activated volumes on cluster nodes.
I am aware of the risks. That is why I do use clvmd. I don't quite understand some of the clvmd behaviour and I wrote here to know if there are some other risks taken into account by clvmd that I am not aware of.
It seem I still need some work and learning to make it work properly.
Greets,
Jacek
Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
Quoted 23 lines · Zdenek Kabelac
Quoted 11 lines
Quoted 9 lines
work properly, as I would expect (make the volume available/unavailable on the node). But an attempt to create a new volume:
lvcreate -n
new_volume-L 1Mshared_vgfails with:
Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
Quoted 6 lines
Haven't really tried to understand what are you trying to achieve, but if you want to have node being activated only on one cluster node,
you may easily use lvcreate -aey option.If you are using default clustered operation - it's not surprising, the operation is refused if other nodes are not responding.
Hmmm didn't though about the initial activation. In fact, I don't need that volume activated as soon as it is created. You are right, I should try 'lvcreate -aey' or 'lvcreate -an'.
Quoted 6 lines
My stupid mistake, indeed.
'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.
Though, LVM have some problems with tracking the exclusive activations later…
If you know about any such bug - just open rhbz with full description of such erroneous case.
Quoted 22 lines
Quoted 7 lines
Quoted 2 lines
Indeed, the VG is not available at the standby node at that moment. But, as it is not available there, I see no point in locking it there.
Well - you would need to write your own type of locking with support of 'standby' currently clvmd doesn't work with such state (and it's not quite clear to me how it actually should even work).
So far either node is in cluster or is fenced.I will try to make this work somehow… I don't think a node in standby (or when one of the VGs is not available there) should be quite different that one fenced.
Quoted 10 lines
Quoted 9 lines
When clvmd is stopped on the inactive node and 'clvmd -S' has been run on the active node, then both 'lvchange' and 'lvcreate' work as expected, but that doesn't look like a graceful switch-over. And another 'clvmd -S' stopped clvmd all together (this seems like a bug to me)
And one more thing bothers me… my system would be very scalable to many nodes, where only two share active storage (when using DRBD). But this won't work if LVM would refuse some operations when any VG is not available on all nodes.
Quoted 5 lines
Quoted 2 lines
Obviously using clustered VG in non-clustered environment isn't smart plan.
What you could do - is to disable clustering support on VGClusters do not have to be symmetrical. Cluster when different nodes have a bit different set of resources available are still clusters.
You want to support different scheme - thus you need to probably write your own clvmd-like daemon to cover all new cases you bring in with non-symmetrical cases.
Quoted 3 lines
I do need clustering locking – in case a volume group is available on a few nodes it must not be possible to more than one node use any of the logical volumes there.
clvmd typical use case is 'vg' used on couple cluster nodes.
While you are probably trying to use N:M mapping of vg and clustered nodes.
Quoted 10 lines
Quoted 4 lines
Note - you may always go around any locking problem with the above config option - just do not report then problems with broken disk content and badly activated volumes on cluster nodes.
I am aware of the risks. That is why I do use clvmd. I don't quite understand some of the clvmd behaviour and I wrote here to know if there are some other risks taken into account by clvmd that I am not aware of.
It seem I still need some work and learning to make it work properly.
You could surely create something to support your specific use case,
however clvmd needs to have some predictable behaviour in many error-case scenarios - and this is when you would need to start to think hard, how to resolve them in your non-symmetrical case.
Zdenek
Quoted 18 lines · Zdenek Kabelac
Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
Quoted 17 lines · Zdenek Kabelac
Quoted 5 lines
Quoted 5 lines
work properly, as I would expect (make the volume available/unavailable on the node). But an attempt to create a new volume:
lvcreate -n
new_volume-L 1Mshared_vgQuoted 9 lines
Quoted 4 lines
fails with:
Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
Haven't really tried to understand what are you trying to achieve, but if you want to have node being activated only on one cluster node,
you may easily use lvcreate -aey option.My stupid mistake, indeed.
Quoted 8 lines
Quoted 5 lines
'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.
Though, LVM have some problems with tracking the exclusive activations later…
If you know about any such bug - just open rhbz with full description of such erroneous case.
It was just another mistake of mine. LVM properly tracks the exclusive locks – the volumes were being deactivated by something else.
Quoted 6 lines
Quoted 2 lines
Clusters do not have to be symmetrical. Cluster when different nodes have a bit different set of resources available are still clusters.
You want to support different scheme - thus you need to probably write your own clvmd-like daemon to cover all new cases you bring in with non-symmetrical cases.
I think this will not be needed.
Quoted 1 line
clvmd typical use case is 'vg' used on couple cluster nodes.
I see.
Quoted 1 line
While you are probably trying to use N:M mapping of vg and clustered nodes.
Exactly. But it seems it should not be a problem in my case.
After knowing my mistake I can see LVM already provides the
Quoted 18 lines · Zdenek Kabelac
Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
Quoted 17 lines · Zdenek Kabelac
Quoted 5 lines
Quoted 5 lines
work properly, as I would expect (make the volume available/unavailable on the node). But an attempt to create a new volume:
lvcreate -n
new_volume-L 1Mshared_vgQuoted 9 lines
Quoted 4 lines
fails with:
Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
Haven't really tried to understand what are you trying to achieve, but if you want to have node being activated only on one cluster node,
you may easily use lvcreate -aey option.My stupid mistake, indeed.
Quoted 8 lines
Quoted 5 lines
'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.
Though, LVM have some problems with tracking the exclusive activations later…
If you know about any such bug - just open rhbz with full description of such erroneous case.
It was just another mistake of mine. LVM properly tracks the exclusive locks – the volumes were being deactivated by something else.
Quoted 6 lines
Quoted 2 lines
Clusters do not have to be symmetrical. Cluster when different nodes have a bit different set of resources available are still clusters.
You want to support different scheme - thus you need to probably write your own clvmd-like daemon to cover all new cases you bring in with non-symmetrical cases.
I think this will not be needed.
Quoted 1 line
clvmd typical use case is 'vg' used on couple cluster nodes.
I see.
Quoted 1 line
While you are probably trying to use N:M mapping of vg and clustered nodes.
Exactly. But it seems it should not be a problem in my case.
After knowing my mistake I can see LVM already provides the functionality I need.
To summarize:
- The default LV activation mode is '-ay', which means, for clustered volume groups, that the volume is to be active on every node in the cluster.
– The activation is not always explicit. 'lvcreate' uses '-ay' when no other '-a' option is given.
– The '-ay' activation won't work if any node in the cluster cannot access the volume group (e.g. when the DRBD device holding it is Secondary or not configured at that node).
– However, my use case doesn't need more than one node using any of the volumes at any time. In fact, it is very important that only a single machine uses each LV at a time.
– For this scenario '-aey' should be always used.
– Explicitly using '-aey' with 'lvcreate' fixes the 'Volume group for uuid not found' for me.
– Other tests made showed that the locking works as expected when a volume group becomes available and unavailable on various cluster nodes. A node having access to a VG can create and/or activate LVs there in exclusive node and all other nodes will comply with that lock whenever they gain access to this VG.
So, it seems that clvmd is not that bound to the 'symmetrical cluster' scenario, provided no more than one node needs to access a volume at a time.
Does this make sense?
Greets,
Jacek
Dne 15.11.2012 14:30, Jacek Konieczny napsal(a):
Quoted 129 lines · Zdenek Kabelac
Quoted 18 lines
Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
Quoted 17 lines · Zdenek Kabelac
Quoted 5 lines
Quoted 5 lines
work properly, as I would expect (make the volume available/unavailable on the node). But an attempt to create a new volume:
lvcreate -n
new_volume-L 1Mshared_vgQuoted 9 lines
Quoted 4 lines
fails with:
Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
Haven't really tried to understand what are you trying to achieve, but if you want to have node being activated only on one cluster node,
you may easily use lvcreate -aey option.My stupid mistake, indeed.
Quoted 8 lines
Quoted 5 lines
'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.
Though, LVM have some problems with tracking the exclusive activations later…
If you know about any such bug - just open rhbz with full description of such erroneous case.
It was just another mistake of mine. LVM properly tracks the exclusive locks – the volumes were being deactivated by something else.
Quoted 6 lines
Quoted 2 lines
Clusters do not have to be symmetrical. Cluster when different nodes have a bit different set of resources available are still clusters.
You want to support different scheme - thus you need to probably write your own clvmd-like daemon to cover all new cases you bring in with non-symmetrical cases.
I think this will not be needed.
Quoted 1 line
clvmd typical use case is 'vg' used on couple cluster nodes.
I see.
Quoted 1 line
While you are probably trying to use N:M mapping of vg and clustered nodes.
Exactly. But it seems it should not be a problem in my case.
After knowing my mistake I can see LVM already provides the
Quoted 18 lines · Zdenek Kabelac
Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
Quoted 17 lines · Zdenek Kabelac
Quoted 5 lines
Quoted 5 lines
work properly, as I would expect (make the volume available/unavailable on the node). But an attempt to create a new volume:
lvcreate -n
new_volume-L 1Mshared_vgQuoted 9 lines
Quoted 4 lines
fails with:
Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
Haven't really tried to understand what are you trying to achieve, but if you want to have node being activated only on one cluster node,
you may easily use lvcreate -aey option.My stupid mistake, indeed.
Quoted 8 lines
Quoted 5 lines
'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.
Though, LVM have some problems with tracking the exclusive activations later…
If you know about any such bug - just open rhbz with full description of such erroneous case.
It was just another mistake of mine. LVM properly tracks the exclusive locks – the volumes were being deactivated by something else.
Quoted 6 lines
Quoted 2 lines
Clusters do not have to be symmetrical. Cluster when different nodes have a bit different set of resources available are still clusters.
You want to support different scheme - thus you need to probably write your own clvmd-like daemon to cover all new cases you bring in with non-symmetrical cases.
I think this will not be needed.
Quoted 1 line
clvmd typical use case is 'vg' used on couple cluster nodes.
I see.
Quoted 1 line
While you are probably trying to use N:M mapping of vg and clustered nodes.
Exactly. But it seems it should not be a problem in my case.
After knowing my mistake I can see LVM already provides the functionality I need.
To summarize:
- The default LV activation mode is '-ay', which means, for clustered volume groups, that the volume is to be active on every node in the cluster.
– The activation is not always explicit. 'lvcreate' uses '-ay' when no other '-a' option is given.
– The '-ay' activation won't work if any node in the cluster cannot access the volume group (e.g. when the DRBD device holding it is Secondary or not configured at that node).
– However, my use case doesn't need more than one node using any of the volumes at any time. In fact, it is very important that only a single machine uses each LV at a time.
– For this scenario '-aey' should be always used.
– Explicitly using '-aey' with 'lvcreate' fixes the 'Volume group for uuid not found' for me.
– Other tests made showed that the locking works as expected when a volume group becomes available and unavailable on various cluster nodes. A node having access to a VG can create and/or activate LVs there in exclusive node and all other nodes will comply with that lock whenever they gain access to this VG.
Quoted 6 lines
So, it seems that clvmd is not that bound to the 'symmetrical cluster' scenario, provided no more than one node needs to access a volume at a time.
Does this make sense?
I'm still somewhat confused with your term 'stand-by' cluster node. But it looks like -aey mostly fixes your problems.
Zdenek