LVM snapshot with Clustered VG
Hi Vladislav,
I tried the patch you mentioned in https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , but apparently it doesn't work for me. On a locally active LV, I still get
./lvm lvchange -aey -vvvv vg/locktest
...
Activating logical volume "locktest" exclusively
#activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1)
#locking/cluster_locking.c:568 Lock held for oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 7400a8c0 : CR
#activate/activate.c:1067 vg/locktest is active
#locking/cluster_locking.c:508 Locking LV oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL) (0xdd)
#locking/cluster_locking.c:395 Error locking on node 7400a8c0: Device or resource busy
Can you give me a hint what to try further?
Regards
Andreas
Sorry, forgot to reply to list from the first attempt.
Quoted 26 lines · Andreas Pflug
Hi Vladislav,
I tried the patch you mentioned in
https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , but apparently it doesn't work for me. On a locally active LV, I still get./lvm lvchange -aey -vvvv vg/locktest
...
Activating logical volume "locktest" exclusively
#activate/dev_manager.c:284 Getting device info for vg-locktest[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
NF [16384] (*1)
#locking/cluster_locking.c:568 Lock held for
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node7400a8c0: CR
#activate/activate.c:1067 vg/locktest is active
#locking/cluster_locking.c:508 Locking LV
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL) (0xdd)
#locking/cluster_locking.c:395 Error locking on node7400a8c0: Device
Quoted 6 lines
or resource busy
Can you give me a hint what to try further?
Regards
Andreas
Hi Andreas,
Lock convertion is only enabled if you pass --force flag.
Also, to upgrade local lock to exclusive one, you need to ensure IIRC that no more node holds local lock.
Vladislav
Am 01.03.13 16:41, schrieb Vladislav Bogdanov:
Quoted 32 lines
Sorry, forgot to reply to list from the first attempt.
Quoted 28 lines · Andreas Pflug
Hi Vladislav,
I tried the patch you mentioned in
https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , but apparently it doesn't work for me. On a locally active LV, I still get./lvm lvchange -aey -vvvv vg/locktest
...
Activating logical volume "locktest" exclusively
#activate/dev_manager.c:284 Getting device info for vg-locktest[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
NF [16384] (*1)
#locking/cluster_locking.c:568 Lock held for
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node7400a8c0: CR
#activate/activate.c:1067 vg/locktest is active
#locking/cluster_locking.c:508 Locking LV
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL) (0xdd)
#locking/cluster_locking.c:395 Error locking on node7400a8c0: Deviceor resource busy
Quoted 7 lines
Quoted 4 lines
Can you give me a hint what to try further?
Regards
AndreasHi Andreas,
Lock convertion is only enabled if you pass --force flag.
Also, to upgrade local lock to exclusive one, you need to ensure IIRC that no more node holds local lock.
Hm, tried that as well:
tools/lvm lvchange --force -aey -vvvv vg/locktest
--force changes the error from "resource busy" to "invalid argument":
#lvchange.c:243 Activating logical volume "locktest" exclusively (forced)
#activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1)
#locking/cluster_locking.c:568 Lock held for oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 7400a8c0 : CR
#activate/activate.c:1067 vg/locktest is active
#activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1)
#activate/activate.c:1067 vg/locktest is active
#activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1)
#locking/cluster_locking.c:568 Lock held for oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node 7400a8c0 : CR
#activate/activate.c:1067 vg/locktest is active
#locking/cluster_locking.c:508 Locking LV oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd)
#locking/cluster_locking.c:395 Error locking on node 7400a8c0: Invalid argument
Regards
Andreas
Quoted 42 lines · Andreas Pflug
Am 01.03.13 16:41, schrieb Vladislav Bogdanov:
Quoted 32 lines
Sorry, forgot to reply to list from the first attempt.
Quoted 28 lines · Andreas Pflug
Hi Vladislav,
I tried the patch you mentioned in
https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html , but apparently it doesn't work for me. On a locally active LV, I still get./lvm lvchange -aey -vvvv vg/locktest
...
Activating logical volume "locktest" exclusively
#activate/dev_manager.c:284 Getting device info for vg-locktest[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
NF [16384] (*1)
#locking/cluster_locking.c:568 Lock held for
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node7400a8c0: CR
#activate/activate.c:1067 vg/locktest is active
#locking/cluster_locking.c:508 Locking LV
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL) (0xdd)
#locking/cluster_locking.c:395 Error locking on node7400a8c0: Deviceor resource busy
Quoted 8 lines
Quoted 4 lines
Can you give me a hint what to try further?
Regards
AndreasHi Andreas,
Lock convertion is only enabled if you pass --force flag. Also, to upgrade local lock to exclusive one, you need to ensure IIRC that no more node holds local lock.
Quoted 5 lines
Hm, tried that as well:
tools/lvmlvchange --force -aey -vvvv vg/locktest--force changes the error from "resource busy" to "invalid argument":
Is volume active on other nodes at that time? And do you run clvmd from that build tree as well?
Also, can you please try attached patch (on top of that one you have)? I polished conversion a bit more, denying -an if volume is ex-locked somewhere and other fixes to logic.
This patch also allows locking (activation) to be performed on remote nodes. I only tested this with corosync2 (which is set up in a way latest pacemaker - post-1.1.8 git master - needs, nodes has additional 'name' value in nodelist, please see
http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html).
I did not have a time to split this all into git commits suitable for sending upstream, so it all is a little bit messy yet.
Quoted 34 lines
#lvchange.c:243 Activating logical volume "locktest" exclusively
(forced)
#activate/dev_manager.c:284 Getting device info for vg-locktest
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
NF [16384] (*1)
#locking/cluster_locking.c:568 Lock held for
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node7400a8c0: CR
#activate/activate.c:1067 vg/locktest is active
#activate/dev_manager.c:284 Getting device info for vg-locktest
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
NF [16384] (*1)
#activate/activate.c:1067 vg/locktest is active
#activate/dev_manager.c:284 Getting device info for vg-locktest
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
NF [16384] (*1)
#locking/cluster_locking.c:568 Lock held for
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN, node7400a8c0: CR
#activate/activate.c:1067 vg/locktest is active
#locking/cluster_locking.c:508 Locking LV
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd)
#locking/cluster_locking.c:395 Error locking on node7400a8c0: Invalid
argument
Quoted 2 lines
Regards
Andreas
daemons/clvmd/clvmd.c +5 -3
@@ -1293,7 +1293,7 @@ static int read_from_local_sock(struct l }/* Check the node name for validity */ -if (inheader->node[0 ] && clops->csid_from_name (csid, inheader->node)) { +if (inheader->node[0 ] && clops->csid_from_name (csid, inheader->node) <=0 ) {/* Error, node is not in the cluster */ struct clvm_header reply = { .cmd =CLVMD_CMD_REPLY ,
@@ -1436,7 +1436,7 @@ static int distribute_command(struct loc/* Do it on a single node */ char csid[MAX_CSID_LEN ]; -if (clops->csid_from_name (csid, inheader->node)) { +if (clops->csid_from_name (csid, inheader->node) <=0 ) {/* This has already been checked so should not happen */ return 0 ; }else {
@@ -1448,6 +1448,8 @@ static int distribute_command(struct loc/* Are we the requested node ?? */ if (memcmp (csid, our_csid, max_csid_len) ==0 ) {DEBUGLOG ("Doing command on local node only\n" ); + inheader->flags &= ~CLVMD_FLAG_REMOTE ; + inheader->flags |=CLVMD_FLAG_LOCAL ;add_to_lvmqueue (thisfd, inheader, len,NULL ); }else { DEBUGLOG("Sending message to single node: %s\n",
@@ -1497,7 +1499,7 @@ static void process_remote_command(strucclvmd-command.c */ if (msg->cmd ==CLVMD_CMD_VERSION ) {int version_nums[3 ]; -char node[256 ]; +char node[max_cluster_member_name_len];memcpy (version_nums, msg->args,sizeof (version_nums));
daemons/clvmd/clvmd-corosync.c +131 -14
235 lines not on this page · raw message
lib/activate/activate.c +15 -9
@@ -270,7 +270,7 @@ int lv_is_active_exclusive_locally(const {return 0 ; } -int lv_is_active_exclusive_remotely (const struct logical_volume *lv) +int lv_is_active_exclusive_remotely (const struct logical_volume *lv,const char *node) {return 0 ; }
@@ -1014,7 +1014,7 @@ int lvs_in_vg_opened(const struct volume* Returns: 0 or 1 */ static int _lv_is_active (const struct logical_volume *lv, -int *locally,int *exclusive) +int *locally,int *exclusive,const char *node) {int r, l, e;/* remote, local, and exclusive */
@@ -1033,7 +1033,7 @@ static int _lv_is_active(const struct loif (l && !exclusive) goto out; -if ((r =remote_lock_held (lv->lvid.s, &e)) >=0 ) +if ((r =remote_lock_held (lv->lvid.s, &e, node)) >=0 ) goto out;/*
@@ -1071,34 +1071,40 @@ out:int lv_is_active (const struct logical_volume *lv) { -return _lv_is_active (lv,NULL ,NULL ); +return _lv_is_active (lv,NULL ,NULL ,NULL ); +} + +int lv_is_active_remotely (const struct logical_volume *lv,const char *node) +{ +int l; +return _lv_is_active (lv, &l,NULL , node) && !l; }int lv_is_active_but_not_locally (const struct logical_volume *lv) {int l; -return _lv_is_active (lv, &l,NULL ) && !l; +return _lv_is_active (lv, &l,NULL ,NULL ) && !l; }int lv_is_active_exclusive (const struct logical_volume *lv) {int e; -return _lv_is_active (lv,NULL , &e) && e; +return _lv_is_active (lv,NULL , &e,NULL ) && e; }int lv_is_active_exclusive_locally (const struct logical_volume *lv) {int l, e; -return _lv_is_active (lv, &l, &e) && l && e; +return _lv_is_active (lv, &l, &e,NULL ) && l && e; } -int lv_is_active_exclusive_remotely (const struct logical_volume *lv) +int lv_is_active_exclusive_remotely (const struct logical_volume *lv,const char *node) {int l, e; -return _lv_is_active (lv, &l, &e) && !l && e; +return _lv_is_active (lv, &l, &e, node) && !l && e; }#ifdef DMEVENTD
lib/activate/activate.h +2 -1
@@ -130,10 +130,11 @@ int lvs_in_vg_activated(const struct volint lvs_in_vg_opened (const struct volume_group *vg);int lv_is_active (const struct logical_volume *lv); +int lv_is_active_remotely (const struct logical_volume *lv,const char *node);int lv_is_active_but_not_locally (const struct logical_volume *lv);int lv_is_active_exclusive (const struct logical_volume *lv);int lv_is_active_exclusive_locally (const struct logical_volume *lv); -int lv_is_active_exclusive_remotely (const struct logical_volume *lv); +int lv_is_active_exclusive_remotely (const struct logical_volume *lv,const char *node);int lv_has_target_type (struct dm_pool *mem,struct logical_volume *lv,const char *layer,const char *target_type);
lib/commands/toolcontext.h +1 -0
@@ -112,6 +112,7 @@ struct cmd_context {char dev_dir[PATH_MAX ];char proc_dir[PATH_MAX ];char sysfs_dir[PATH_MAX ];/* FIXME Use global value instead. */ +const char *node; };/*
lib/locking/cluster_locking.c +13 -6
@@ -190,8 +190,10 @@ static void _build_header(struct clvm_he }else if (!strcmp (node,NODE_REMOTE )) { head->node[0 ] ='\0' ; head->flags =CLVMD_FLAG_REMOTE ; - }else -strcpy (head->node, node); + }else { +memcpy (head->node, node,strlen (node) +1 ); + head->flags =CLVMD_FLAG_REMOTE ; + } }/*
@@ -315,6 +317,9 @@ static int _lock_for_cluster(struct cmd_assert (name); +if (node ==NULL ) { + node ="" ; + } len =strlen (name) +3 ; args =alloca (len);strcpy (args +2 , name);
@@ -374,7 +379,7 @@ static int _lock_for_cluster(struct cmd_ !(flags & LCK_CLUSTER_VG))) node =NODE_LOCAL ; else if (flags & LCK_REMOTE) - node =NODE_REMOTE ; + node = cmd->node ? cmd->node :NODE_REMOTE ; } status = _cluster_request(clvmd_cmd, node, args, len,
@@ -527,16 +532,18 @@ static int decode_lock_type(const char * }#ifdef CLUSTER_LOCKING_INTERNAL -static int _query_resource(const char *resource, int *mode) +static int _query_resource(const char *resource, int *mode, const char *node)#else -int query_resource(const char *resource, int *mode) +int query_resource(const char *resource, int *mode, const char *node)#endif {int i, status, len, num_responses, saved_errno; -const char *node ="" ;char *args;lvm_response_t *response =NULL ; +if (node ==NULL ) { + node ="" ; + } saved_errno = errno; len =strlen (resource) +3 ; args =alloca (len);
lib/locking/external_locking.c +1 -1
@@ -28,7 +28,7 @@ static int (*_lock_fn) (struct cmd_conte uint32_t flags) = NULL;static int (*_init_fn) (int type,struct dm_config_tree * cft,uint32_t *flags) =NULL ; -static int (*_lock_query_fn) (const char *resource,int *mode) =NULL ; +static int (*_lock_query_fn) (const char *resource,int *mode,const char *node) =NULL ;static int _lock_resource (struct cmd_context *cmd,const char *resource,uint32_t flags)
lib/locking/locking.c +42 -22
106 lines not on this page · raw message
lib/locking/locking.h +8 -2
@@ -25,7 +25,7 @@ void reset_locking(void);int vg_write_lock_held (void );int locking_is_clustered (void ); -int remote_lock_held (const char *vol,int *exclusive); +int remote_lock_held (const char *vol,int *exclusive,const char *node);/* * LCK_VG:
@@ -186,7 +186,7 @@ int check_lvm1_vg_inactive(struct cmd_co#define activate_lv_excl_local_force (cmd, lv) \ lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_LOCAL | (lv_is_active(lv) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0))#define activate_lv_excl_remote_force (cmd, lv) \ - lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_REMOTE | (lv_is_active(lv) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0)) + lock_lv_vol(cmd, lv, LCK_LV_EXCLUSIVE | LCK_HOLD | LCK_REMOTE | (lv_is_active_remotely(lv, cmd->node) && ! lv_is_active_exclusive(lv) ? LCK_TRY_CONVERT : 0))struct logical_volume ;int activate_lv_excl (struct cmd_context *cmd,struct logical_volume *lv);
@@ -197,8 +197,14 @@ int deactivate_lv(struct cmd_context *cm lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_LOCAL)#define activate_lv_local_force (cmd, lv) \ lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_LOCAL | (lv_is_active_exclusive_locally(lv) ? LCK_TRY_CONVERT : 0)) +#define activate_lv_remote (cmd, lv) \ + lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_REMOTE) +#define activate_lv_remote_force (cmd, lv) \ + lock_lv_vol(cmd, lv, LCK_LV_ACTIVATE | LCK_HOLD | LCK_REMOTE | (lv_is_active_exclusive_remotely(lv, cmd->node) ? LCK_TRY_CONVERT : 0))#define deactivate_lv_local (cmd, lv) \ lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE | LCK_LOCAL) +#define deactivate_lv_remote (cmd, lv) \ + lock_lv_vol(cmd, lv, LCK_LV_DEACTIVATE | LCK_REMOTE)#define drop_cached_metadata (vg) \ lock_vol((vg)->cmd, (vg)->name, LCK_VG_DROP_CACHE)#define remote_commit_cached_metadata (vg) \
lib/locking/locking_types.h +1 -1
@@ -18,7 +18,7 @@typedef int (*lock_resource_fn ) (struct cmd_context * cmd,const char *resource,uint32_t flags); -typedef int (*query_resource_fn ) (const char *resource,int *mode); +typedef int (*query_resource_fn ) (const char *resource,int *mode,const char *node);typedef void (*fin_lock_fn ) (void );typedef void (*reset_lock_fn ) (void );
tools/args.h +1 -0
@@ -76,6 +76,7 @@ arg(discards_ARG, '\0', "discards", disc arg(stripes_long_ARG, '\0', "stripes", int_arg, 0) arg(sysinit_ARG, '\0', "sysinit", NULL, 0) arg(thinpool_ARG, '\0', "thinpool", string_arg, 0) +arg (node_ARG,'\0' ,"node" , string_arg,0 ) /* Allow some variations */arg (resizable_ARG,'\0' ,"resizable" , yes_no_arg,0 )
tools/commands.h +2 -1
@@ -87,6 +87,7 @@ xx(lvchange,"\t[-y|--yes]\n" "\t[--version]\n" "\t[-Z|--zero {y|n}]\n" +"\t[--node Node ]\n" "\tLogicalVolume[Path] [LogicalVolume[Path]...]\n" , alloc_ARG, autobackup_ARG, activate_ARG, available_ARG, contiguous_ARG,
@@ -94,7 +95,7 @@ xx(lvchange, major_ARG, minor_ARG, monitor_ARG, noudevsync_ARG, partial_ARG, permission_ARG, persistent_ARG, poll_ARG, readahead_ARG, resync_ARG, refresh_ARG, addtag_ARG, deltag_ARG, sysinit_ARG, test_ARG, yes_ARG, - zero_ARG) + zero_ARG, node_ARG) xx(lvconvert, "Change logical volume layout",
tools/lvchange.c +73 -17
119 lines not on this page · raw message
tools/pvmove.c +1 -1
@@ -240,7 +240,7 @@ static struct logical_volume *_set_up_pv }if (vg_is_clustered (vg) && -lv_is_active_exclusive_remotely (lv)) { +lv_is_active_exclusive_remotely (lv,NULL )) { lv_skipped =1 ;log_print_unless_silent ("Skipping LV %s which is activated " "exclusively on remote node." , lv->name);
tools/vgchange.c +1 -1
@@ -125,7 +125,7 @@ static int _activate_lvs_in_vg(struct cm* If the LV is active exclusive remotely, * then ignore it here */ -if (lv_is_active_exclusive_remotely (lv)) { +if (lv_is_active_exclusive_remotely (lv,NULL )) {log_verbose ("%s/%s is exclusively active on" " a remote node" , vg->name, lv->name);continue ;
Am 06.03.13 08:58, schrieb Vladislav Bogdanov:
Quoted 7 lines
Quoted 6 lines · Andreas Pflug
Am 01.03.13 16:41, schrieb Vladislav Bogdanovr?
Quoted 4 lines
Hi Andreas,
Lock convertion is only enabled if you pass --force flag. Also, to upgrade local lock to exclusive one, you need to ensure IIRC that no more node holds local lock.
Quoted 6 lines
Quoted 5 lines
Hm, tried that as well:
tools/lvmlvchange --force -aey -vvvv vg/locktest--force changes the error from "resource busy" to "invalid argument":
Is volume active on other nodes at that time?
I made sure it's not active on other nodes: lvchange -an vg/locktest ; lvchange -aly vg/locktest
Quoted 5 lines
And do you run clvmd from that build tree as well?
Also, can you please try attached patch (on top of that one you have)? I polished conversion a bit more, denying -an if volume is ex-locked somewhere and other fixes to logic.
I tried that additional patch. I'm running this test versions on my test node only (including clvmd), the other nodes are still running clvmd 2.2.95 (I guess this shouldn't matter since all are inactive). Same result:
#lvchange.c:258 Activating logical volume "locktest" exclusively (forced)
#activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1)
#activate/activate.c:1067 vg/locktest is active
#activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1)
#activate/activate.c:1067 vg/locktest is active
#activate/dev_manager.c:284 Getting device info for vg-locktest [LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN NF [16384] (*1)
#activate/activate.c:1067 vg/locktest is active
#locking/cluster_locking.c:513 Locking LV oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd)
#locking/cluster_locking.c:400 Error locking on node 7400a8c0: Invalid argument
Quoted 5 lines
This patch also allows locking (activation) to be performed on remote nodes. I only tested this with corosync2 (which is set up in a way latest pacemaker - post-1.1.8 git master - needs, nodes has additional 'name' value in nodelist, please see
http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html).
I'm running corosync 1.4.2 (debian wheezy).
Regards,
Andreas
Quoted 17 lines · Andreas Pflug
Am 06.03.13 08:58, schrieb Vladislav Bogdanov:
Quoted 7 lines
Quoted 6 lines · Andreas Pflug
Am 01.03.13 16:41, schrieb Vladislav Bogdanovr?
Quoted 4 lines
Hi Andreas,
Lock convertion is only enabled if you pass --force flag. Also, to upgrade local lock to exclusive one, you need to ensure IIRC that no more node holds local lock.Quoted 6 lines
Quoted 5 lines
Hm, tried that as well:
tools/lvmlvchange --force -aey -vvvv vg/locktest--force changes the error from "resource busy" to "invalid argument":
Is volume active on other nodes at that time?
I made sure it's not active on other nodes: lvchange -an vg/locktest ; lvchange -aly vg/locktest
Quoted 8 lines
Quoted 5 lines
And do you run clvmd from that build tree as well?
Also, can you please try attached patch (on top of that one you have)? I polished conversion a bit more, denying -an if volume is ex-locked somewhere and other fixes to logic.
I tried that additional patch. I'm running this test versions on my test node only (including clvmd), the other nodes are still running clvmd 2.2.95 (I guess this shouldn't matter since all are inactive). Same result:
I believe this matters, because error you see is received from a remote node. Is node with ID 7400a8c0 local?
Quoted 28 lines
#lvchange.c:258 Activating logical volume "locktest" exclusively
(forced)
#activate/dev_manager.c:284 Getting device info for vg-locktest
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
NF [16384] (*1)
#activate/activate.c:1067 vg/locktest is active
#activate/dev_manager.c:284 Getting device info for vg-locktest
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
NF [16384] (*1)
#activate/activate.c:1067 vg/locktest is active
#activate/dev_manager.c:284 Getting device info for vg-locktest
[LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN]
#ioctl/libdm-iface.c:1724 dm info
LVM-oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN
NF [16384] (*1)
#activate/activate.c:1067 vg/locktest is active
#locking/cluster_locking.c:513 Locking LV
oW2W7O2cgWRLUhoVR8qqqQY7wlcYexmWU8y83bGQz9IcnXh3GfXslBN6ziZrC3BN EX (LV|NONBLOCK|CLUSTER|LOCAL|CONVERT) (0x40dd)
#locking/cluster_locking.c:400 Error locking on node7400a8c0: Invalid
argument
Quoted 8 lines
Quoted 6 lines
This patch also allows locking (activation) to be performed on remote nodes. I only tested this with corosync2 (which is set up in a way latest pacemaker - post-1.1.8 git master - needs, nodes has additional 'name' value in nodelist, please see
http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html).I'm running corosync 1.4.2 (debian wheezy).
Which cluster manager interface does clvmd detect? corosync or openais? You should use former, openais one is(was) using LCK service which is very unstable.
Quoted 3 lines
Regards,
Andreas
Am 06.03.13 10:35, schrieb Vladislav Bogdanov:
Quoted 5 lines
Quoted 4 lines · Andreas Pflug
I made sure it's not active on other nodes: lvchange -an vg/locktest ; lvchange -aly vg/locktest
Quoted 2 lines
And do you run clvmd from that build tree as well?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Also, can you please try attached patch (on top of that one you have)? I polished conversion a bit more, denying -an if volume is ex-locked somewhere and other fixes to logic.
I tried that additional patch. I'm running this test versions on my test node only (including clvmd), the other nodes are still running clvmd 2.2.95 (I guess this shouldn't matter since all are inactive). Same result:
I believe this matters, because error you see is received from a remote node. Is node with ID
7400a8c0local?
Yes, that's the test node.
Hm, not funny if I have to upgrade all nodes on the production system... I'm a little surprised that remote inactive nodes need to be aware of that force-exclusive stuff.
Quoted 5 lines
I'm running corosync 1.4.2 (debian wheezy).
Which cluster manager interface does clvmd detect? corosync or openais? You should use former, openais one is(was) using LCK service which is very unstable.
It's using openais. I'm not too happy about the stability, so maybe I'd switch to corosync now. Could this be a reason for the x-lock failure as well?
Regards,
Andreas
Quoted 10 lines · Andreas Pflug
Am 06.03.13 10:35, schrieb Vladislav Bogdanov:
Quoted 9 lines
Quoted 8 lines · Andreas Pflug
I made sure it's not active on other nodes: lvchange -an vg/locktest ; lvchange -aly vg/locktest
Quoted 6 lines
And do you run clvmd from that build tree as well?
Also, can you please try attached patch (on top of that one you have)? I
polished conversion a bit more, denying -an if volume is ex-locked somewhere and other fixes to logic.
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
I tried that additional patch. I'm running this test versions on my test node only (including clvmd), the other nodes are still running clvmd 2.2.95 (I guess this shouldn't matter since all are inactive). Same result:
I believe this matters, because error you see is received from a remote node. Is node with ID
7400a8c0local?Yes, that's the test node.
But you receive "Invalid argument" error from the cluster layer, so is should go from a local clvmd if you are correct about node IDs. You may run clvmd in a debugging mode as well to catch what goes wrong.
Quoted 3 lines
Hm, not funny if I have to upgrade all nodes on the production system... I'm a little surprised that remote inactive nodes need to be aware of that force-exclusive stuff.
You probably should try to test that on a test cluster first (as I do) although I'm almost sure it works correctly in setup I described.
Quoted 7 lines
Quoted 5 lines
I'm running corosync 1.4.2 (debian wheezy).
Which cluster manager interface does clvmd detect? corosync or openais? You should use former, openais one is(was) using LCK service which is very unstable.It's using openais. I'm not too happy about the stability, so maybe I'd switch to corosync now.
That could be the problem btw. I did neither test nor look at openais module implementation in clvmd, because I had plenty problems with it (actually with LCK under it) in the past, so I even forced to use corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older systems (look at -I switch of clvmd).
And, openais is deprecated upstream, so I do not see any reason to use it. Even gfs_controld (which is probably the only well-known user of openais services) actually does not strictly require it, at least I was able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is not compatible with openais plugins.
Also you may need quorum patch found in this list, it does its job well.
Quoted 1 line
Could this be a reason for the x-lock failure as well?
Can you please reword?
Vladislav
Am 06.03.13 12:20, schrieb Vladislav Bogdanov:
Quoted 12 lines
Quoted 6 lines
Quoted 4 lines
I'm running corosync 1.4.2 (debian wheezy).
Which cluster manager interface does clvmd detect? corosync or openais? You should use former, openais one is(was) using LCK service which is very unstable.It's using openais. I'm not too happy about the stability, so maybe I'd switch to corosync now.
That could be the problem btw. I did neither test nor look at openais module implementation in clvmd, because I had plenty problems with it (actually with LCK under it) in the past, so I even forced to use corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older systems (look at -I switch of clvmd).
Quoted 9 lines
And, openais is deprecated upstream, so I do not see any reason to use it. Even
gfs_controld(which is probably the only well-known user of openais services) actually does not strictly require it, at least I was able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is not compatible with openais plugins.Also you may need quorum patch found in this list, it does its job well.
Quoted 1 line
Could this be a reason for the x-lock failure as well?
You just answered the quirky question :-)
Unfortunately, corosync/dlm don't work for me as expected. When starting clvmd -I corosync (with dlm kernel module loaded), creating the dlm lockspace "clvmd" fails, with
dlm: no local IP address has been set
dlm: cannot start dlm lowcomms -107
I haven't found any hint what might go wrong on the machine (checked already hostname resolves correctly to its ip address via /etc/hosts; corosync uses that network too).
Regards
Andreas
Quoted 22 lines · Andreas Pflug
Am 06.03.13 12:20, schrieb Vladislav Bogdanov:
Quoted 12 lines
Quoted 6 lines
Quoted 4 lines
I'm running corosync 1.4.2 (debian wheezy).
Which cluster manager interface does clvmd detect? corosync or openais? You should use former, openais one is(was) using LCK service which is very unstable.It's using openais. I'm not too happy about the stability, so maybe I'd switch to corosync now.
That could be the problem btw. I did neither test nor look at openais module implementation in clvmd, because I had plenty problems with it (actually with LCK under it) in the past, so I even forced to use corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older systems (look at -I switch of clvmd).
Quoted 9 lines
And, openais is deprecated upstream, so I do not see any reason to use it. Even
gfs_controld(which is probably the only well-known user of openais services) actually does not strictly require it, at least I was able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is not compatible with openais plugins.Also you may need quorum patch found in this list, it does its job well.
Quoted 1 line
Could this be a reason for the x-lock failure as well?
Quoted 9 lines
You just answered the quirky question :-)
Unfortunately, corosync/dlm don't work for me as expected. When starting clvmd -I corosync (with dlm kernel module loaded), creating the dlm lockspace "clvmd" fails, with
dlm: no local IP address has been set
dlm: cannot start dlm lowcomms -107
You need to have dlm_controld running on all nodes.
And that is not trivial with corosync1. You need to either use cman or use deprecated dlm_controld.pcmk which was removed from a cluster (cman) package after 3.0.17. Latter does not work well without heavy patching
(http://www.mail-archive.com/[email protected]/msg09959.html
,
http://www.mail-archive.com/[email protected]/msg11123.html
and
http://oss.clusterlabs.org/pipermail/pacemaker/2012-February/013073.html).
Even after that there are some problems with it.
Latest versions (4.x) of dlm (it was split from cman) work fine with corosync2 and pacemaker 1.1.6+. But I doubt it can ever be compiled with corosync1. Yes, it cannot, because it requires quorum which appeared in corosync2.
You may look at thread
http://oss.clusterlabs.org/pipermail/pacemaker/2012-October/015826.html
for some info.
That's all why I moved to corosync2. You can search through pacemaker list archives for dlm-related messages from me to get full picture.
You may also look at slides I presented at LVEE winter 2013:
http://lvee.org/uploads/image_upload/file/267/Linux_Clusters_at_LVEE_2013.pdf
Quoted 6 lines
I haven't found any hint what might go wrong on the machine (checked already hostname resolves correctly to its ip address via
/etc/hosts; corosync uses that network too).Regards
Andreas
Am 06.03.13 14:28, schrieb Vladislav Bogdanov:
Quoted 52 lines
Quoted 22 lines · Andreas Pflug
Am 06.03.13 12:20, schrieb Vladislav Bogdanov:
Quoted 12 lines
Quoted 6 lines
Quoted 4 lines
I'm running corosync 1.4.2 (debian wheezy).
Which cluster manager interface does clvmd detect? corosync or openais? You should use former, openais one is(was) using LCK service which is very unstable.It's using openais. I'm not too happy about the stability, so maybe I'd switch to corosync now.
That could be the problem btw. I did neither test nor look at openais module implementation in clvmd, because I had plenty problems with it (actually with LCK under it) in the past, so I even forced to use corosync (CPG) + dlm instead of detected openais (CPG+LCK) for older systems (look at -I switch of clvmd).
Quoted 9 lines
And, openais is deprecated upstream, so I do not see any reason to use it. Even
gfs_controld(which is probably the only well-known user of openais services) actually does not strictly require it, at least I was able to port it to pure CPG+DLM with dlm4 on top of corosync2, which is not compatible with openais plugins.Also you may need quorum patch found in this list, it does its job well.
Quoted 1 line
Could this be a reason for the x-lock failure as well?
Quoted 9 lines
You just answered the quirky question :-)
Unfortunately, corosync/dlm don't work for me as expected. When starting clvmd -I corosync (with dlm kernel module loaded), creating the dlm lockspace "clvmd" fails, with
dlm: no local IP address has been set
dlm: cannot start dlm lowcomms -107You need to have
dlm_controldrunning on all nodes.
And that is not trivial with corosync1. You need to either use cman or use deprecated dlm_controld.pcmk which was removed from a cluster (cman) package after 3.0.17. Latter does not work well without heavy patching
(http://www.mail-archive.com/[email protected]/msg09959.html
,
http://www.mail-archive.com/[email protected]/msg11123.html
and
http://oss.clusterlabs.org/pipermail/pacemaker/2012-February/013073.html).
Even after that there are some problems with it.Latest versions (4.x) of dlm (it was split from cman) work fine with corosync2 and pacemaker 1.1.6+. But I doubt it can ever be compiled with corosync1. Yes, it cannot, because it requires quorum which appeared in corosync2.
You may look at thread
http://oss.clusterlabs.org/pipermail/pacemaker/2012-October/015826.html
for some info.
Quoted 5 lines
That's all why I moved to corosync2. You can search through pacemaker list archives for dlm-related messages from me to get full picture.
You may also look at slides I presented at LVEE winter 2013:
http://lvee.org/uploads/image_upload/file/267/Linux_Clusters_at_LVEE_2013.pdf
Thanks for the pointers, seems there's no way around reorganizing all clustering stuff.
Regards,
Andreas
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
In reply to Vladislav BogdanovHi Vladislav,
I finally succeeded making all working as expected!
What I did:
- compile corosync 2.3.0 (http://corosync.org/download/corosync-2.3.0.tar.gz)
- compile libqb 0.14.4 (https://fedorahosted.org/releases/q/u/quarterback/libqb-0.14.4.tar.gz)
- compile dlm_4.0.1 (http://git.fedorahosted.org/cgit/dlm.git/snapshot/dlm-4.0.1.tar.gz)
- Compiled LVM 2.2.98 (http://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_98.tar.gz)
with your PREVIOUS patch (
https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html )
I made a 2-node test cluster, both accessing a test iscsi target, all daemons load fine.
Everything works as expected! Snapshotting is possible after -aey --force explicit exclusive locking, lvresize works as well.
Previously, I tried your latest patch. I had a plethiora of errors, starting with lvcreate failing:
#metadata/lv_manip.c:4137 Clearing start of logical volume "testvol2"
#device/dev-cache.c:599 /dev/san2/testvol2: stat failed: No such file or directory
#metadata/lv_manip.c:4140 /dev/san2/testvol2: not found: device not cleared
#metadata/lv_manip.c:4620 Aborting. Failed to wipe start of new LV.
I need to add -Z n to have it created. After that, most activations failed, with error code 5. I went back to vanilla 2.2.98, all ok except exclusive locking, applied the initial locking patch and voila! All done.
Thanks for your support!
Andreas
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Quoted 24 lines · Andreas Pflug
Hi Vladislav,
I finally succeeded making all working as expected!
What I did:
- compile corosync 2.3.0
(http://corosync.org/download/corosync-2.3.0.tar.gz)
- compile libqb 0.14.4
(https://fedorahosted.org/releases/q/u/quarterback/libqb-0.14.4.tar.gz)
- compile dlm_4.0.1
(http://git.fedorahosted.org/cgit/dlm.git/snapshot/dlm-4.0.1.tar.gz)
- Compiled LVM 2.2.98
(http://git.fedorahosted.org/cgit/lvm2.git/snapshot/lvm2-2_02_98.tar.gz)
with your PREVIOUS patch (
https://www.redhat.com/archives/linux-lvm/2013-January/msg00006.html )I made a 2-node test cluster, both accessing a test iscsi target, all daemons load fine.
Everything works as expected! Snapshotting is possible after -aey --force explicit exclusive locking, lvresize works as well.Previously, I tried your latest patch. I had a plethiora of errors, starting with lvcreate failing:
Quoted 6 lines
#metadata/lv_manip.c:4137 Clearing start of logical volume "testvol2"
#device/dev-cache.c:599/dev/san2/testvol2: stat failed: No such
file or directory
#metadata/lv_manip.c:4140/dev/san2/testvol2: not found: device not
cleared
#metadata/lv_manip.c:4620 Aborting. Failed to wipe start of new LV.
Yep, I know about this, I reworked that patch a lot, and now it does every thing I wanted without regressions.
I will try to post updated version in the next few days when I manage to apply it to a git tree.
Quoted 9 lines
I need to add -Z n to have it created. After that, most activations failed, with error code 5. I went back to vanilla 2.2.98, all ok except exclusive locking, applied the initial locking patch and voila! All done.
Thanks for your support!
Andreas
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Am 13.03.13 17:53, schrieb Vladislav Bogdanov:
Quoted 11 lines
#metadata/lv_manip.c:4137 Clearing start of logical volume "testvol2"
#device/dev-cache.c:599/dev/san2/testvol2: stat failed: No such
file or directory
#metadata/lv_manip.c:4140/dev/san2/testvol2: not found: device not
cleared
#metadata/lv_manip.c:4620 Aborting. Failed to wipe start of new LV.
Yep, I know about this, I reworked that patch a lot, and now it does every thing I wanted without regressions.
I will try to post updated version in the next few days when I manage to apply it to a git tree.
Is there a way to find out if a LV is locked exclusively? lvs displaying -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about exclusive locking.
Regards,
Andreas
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Quoted 13 lines · Andreas Pflug
Am 13.03.13 17:53, schrieb Vladislav Bogdanov:
Quoted 11 lines
#metadata/lv_manip.c:4137 Clearing start of logical volume "testvol2"
#device/dev-cache.c:599/dev/san2/testvol2: stat failed: No such
file or directory
#metadata/lv_manip.c:4140/dev/san2/testvol2: not found: device not
cleared
#metadata/lv_manip.c:4620 Aborting. Failed to wipe start of new LV.
Yep, I know about this, I reworked that patch a lot, and now it does every thing I wanted without regressions.
I will try to post updated version in the next few days when I manage to apply it to a git tree.
Quoted 3 lines
Is there a way to find out if a LV is locked exclusively? lvs displaying -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about exclusive locking.
That would break other tools which rely on their output. F.e. cluster resource agents of libvirt (yes, it runs lvm tools rather then using API, which is not yet complete btw). As I also need to obtain this information, I think about writing simple tool (f.e. clvm_tool) which would display needed info.
As a workaround you can run lvchange -aly without force parameter. If it succeeds, the volume is locked in a shared mode, otherwise it is locked exclusively.
Quoted 3 lines
Regards,
Andreas
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Quoted 10 lines · Vladislav Bogdanov
Quoted 4 lines
Is there a way to find out if a LV is locked exclusively? lvs displaying -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about exclusive locking.
That would break other tools which rely on their output. F.e. cluster resource agents of libvirt (yes, it runs lvm tools rather then using API, which is not yet complete btw). As I also need to obtain this information, I think about writing simple tool (f.e.
clvm_tool) which would display needed info.
Quoted 3 lines
As a workaround you can run lvchange -aly without force parameter. If it succeeds, the volume is locked in a shared mode, otherwise it is locked exclusively.
Hm, thats one ugly workaround...
How about a clvmd option, something like -l to list all locks and exit.
Regards,
Andreas
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 11 lines · Vladislav Bogdanov
Quoted 10 lines
Quoted 4 lines
Is there a way to find out if a LV is locked exclusively? lvs displaying -e-- instead of -a-- would be nice. Seems not even lvdisplay knows about exclusive locking.
That would break other tools which rely on their output. F.e. cluster resource agents of libvirt (yes, it runs lvm tools rather then using API, which is not yet complete btw). As I also need to obtain this information, I think about writing simple tool (f.e.
clvm_tool) which would display needed info.
Quoted 7 lines
Quoted 3 lines
As a workaround you can run lvchange -aly without force parameter. If it succeeds, the volume is locked in a shared mode, otherwise it is locked exclusively.
Hm, thats one ugly workaround...
How about a clvmd option, something like -l to list all locks and exit.
I think - the extension to 'lvs' command could be relatively simple (adding a new column)
You may query for exclusive/local activation on the node. (So you cannot just tell on which other node is the device active, but you could print about these states:
active exclusive local
active exclusive
active local
active
Zdenek
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Quoted 18 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 17 lines · Vladislav Bogdanov
Quoted 15 lines
Quoted 6 lines
Is there a way to find out if a LV is locked exclusively? lvs displaying
-e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
exclusive locking.That would break other tools which rely on their output. F.e. cluster resource agents of libvirt (yes, it runs lvm tools rather then using API, which is not yet complete btw). As I also need to obtain this information, I think about writing simple tool (f.e.
clvm_tool) which would display needed info.As a workaround you can run lvchange -aly without force parameter. If it succeeds, the volume is locked in a shared mode, otherwise it is locked exclusively.
Quoted 7 lines
Quoted 3 lines
Hm, thats one ugly workaround...
How about a clvmd option, something like -l to list all locks and exit.I think - the extension to 'lvs' command could be relatively simple (adding a new column)
Yes, that's correct.
Quoted 9 lines
You may query for exclusive/local activation on the node. (So you cannot just tell on which other node is the device active, but you could print about these states:
active exclusive local
active exclusive
active local
active
You also may poll all know nodes, but that is a hack too.
That's why I prefer to have this as a separate tool (with dlm_tool-like params and output) which lists node IDs and lock mode. Unfortunately do not have power to write it now.
Are core LVM devels interested in these two features: lock conversion and managing remote node locks? If yes, then I can (hopefully) prepare git patches next week.
Vladislav
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
Quoted 41 lines
Quoted 18 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 17 lines · Vladislav Bogdanov
Quoted 15 lines
Quoted 6 lines
Is there a way to find out if a LV is locked exclusively? lvs displaying
-e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
exclusive locking.That would break other tools which rely on their output. F.e. cluster resource agents of libvirt (yes, it runs lvm tools rather then using API, which is not yet complete btw). As I also need to obtain this information, I think about writing simple tool (f.e.
clvm_tool) which would display needed info.As a workaround you can run lvchange -aly without force parameter. If it succeeds, the volume is locked in a shared mode, otherwise it is locked exclusively.
Quoted 7 lines
Quoted 3 lines
Hm, thats one ugly workaround...
How about a clvmd option, something like -l to list all locks and exit.I think - the extension to 'lvs' command could be relatively simple (adding a new column)
Yes, that's correct.
Quoted 9 lines
You may query for exclusive/local activation on the node. (So you cannot just tell on which other node is the device active, but you could print about these states:
active exclusive local
active exclusive
active local
activeYou also may poll all know nodes, but that is a hack too.
Quoted 7 lines
That's why I prefer to have this as a separate tool (with dlm_tool-like params and output) which lists node IDs and lock mode. Unfortunately do not have power to write it now.
Are core LVM devels interested in these two features: lock conversion and managing remote node locks? If yes, then I can (hopefully) prepare git patches next week.
I'm not quite sure what do you mean by 'managing remote node locks' ?
Current login behind lvm command is -
You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote node in via some configuration tags)
And you want to 'upgrade' remote locks to something else ?
What would be the use-case you could not resolve with current command line args?
Is that supported by dlm (since lvm locks are mapped to dlm)? How would you resolve error path fallbacks ?
Also I believe the clvmd protocol is out of free bits for extension, so how the protocol would look like ?
Zdenek
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Quoted 48 lines · Zdenek Kabelac
Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
Quoted 47 lines
Quoted 20 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 19 lines · Vladislav Bogdanov
Quoted 17 lines
Quoted 6 lines
Is there a way to find out if a LV is locked exclusively? lvs displaying
-e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
exclusive locking.That would break other tools which rely on their output. F.e. cluster resource agents of libvirt (yes, it runs lvm tools rather then using API, which is not yet complete btw). As I also need to obtain this information, I think about writing simple tool (f.e.
clvm_tool) which would display needed info.As a workaround you can run lvchange -aly without force parameter. If it
succeeds, the volume is locked in a shared mode, otherwise it is locked
exclusively.Quoted 7 lines
Quoted 3 lines
Hm, thats one ugly workaround...
How about a clvmd option, something like -l to list all locks and exit.I think - the extension to 'lvs' command could be relatively simple (adding a new column)
Yes, that's correct.
Quoted 9 lines
You may query for exclusive/local activation on the node. (So you cannot just tell on which other node is the device active, but you could print about these states:
active exclusive local
active exclusive
active local
activeYou also may poll all know nodes, but that is a hack too.
That's why I prefer to have this as a separate tool (with dlm_tool-like params and output) which lists node IDs and lock mode. Unfortunately do not have power to write it now.
Quoted 6 lines
Quoted 3 lines
Are core LVM devels interested in these two features: lock conversion and managing remote node locks? If yes, then I can (hopefully) prepare git patches next week.
I'm not quite sure what do you mean by 'managing remote node locks' ?
Activation/deactivation of LVs on a different corosync cluster node, specified by its node name (with pacemaker-like method to determine that name).
Also conversion of locks on that node.
Quoted 6 lines
Current login behind lvm command is -
You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote node in via some configuration tags)
Could you please explain this - I do not see anything relevant in man pages.
Quoted 2 lines
And you want to 'upgrade' remote locks to something else ?
Yes, shared-to-exclusive end vice verse.
Quoted 3 lines
What would be the use-case you could not resolve with current command line args?
I need to convert lock on a remote node during last stage of ver3 migration in libvirt/qemu. That is a "confirm" stage, which runs on a "source" node, during which "old" vm is killed and disk is released. So, I first ("begin" stage) convert lock from exclusive to shared on a source node, then obtain shared lock on a target node (during "prepare" stage, when receiving qemu instance is started), then migrate vm between two processes which have LV opened, and then release shared lock on a source node ("confirm" stage, after source qemu is killed).
There is no other events on a destination node in ver3 migration protocol, so I'm unable to convert lock to exclusive there after migration is finished. So I do that from a source node, after it released lock.
Quoted 2 lines
Is that supported by dlm (since lvm locks are mapped to dlm)?
Command just sent to a specific clvm instance and performed there.
Quoted 1 line
How would you resolve error path fallbacks ?
Could you please tell what exactly do you mean?
If dlm on a remote node is unable to perform requested operation, then error is returned to a initiator.
Quoted 2 lines
Also I believe the clvmd protocol is out of free bits for extension, so how the protocol would look like ?
It contains 'node' field (I assume it was never actually used before) and with some fixes that works.
Vladislav
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Quoted 99 lines · Vladislav Bogdanov
Quoted 48 lines · Zdenek Kabelac
Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
Quoted 47 lines
Quoted 20 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 19 lines · Vladislav Bogdanov
Quoted 17 lines
Quoted 6 lines
Is there a way to find out if a LV is locked exclusively? lvs displaying
-e-- instead of -a-- would be nice. Seems not even lvdisplay knows about
exclusive locking.That would break other tools which rely on their output. F.e. cluster resource agents of libvirt (yes, it runs lvm tools rather then using API, which is not yet complete btw). As I also need to obtain this information, I think about writing simple tool (f.e.
clvm_tool) which would display needed info.As a workaround you can run lvchange -aly without force parameter. If it
succeeds, the volume is locked in a shared mode, otherwise it is locked
exclusively.Quoted 7 lines
Quoted 3 lines
Hm, thats one ugly workaround...
How about a clvmd option, something like -l to list all locks and exit.I think - the extension to 'lvs' command could be relatively simple (adding a new column)
Yes, that's correct.
Quoted 9 lines
You may query for exclusive/local activation on the node. (So you cannot just tell on which other node is the device active, but you could print about these states:
active exclusive local
active exclusive
active local
activeYou also may poll all know nodes, but that is a hack too.
That's why I prefer to have this as a separate tool (with dlm_tool-like params and output) which lists node IDs and lock mode. Unfortunately do not have power to write it now.
Quoted 6 lines
Quoted 3 lines
Are core LVM devels interested in these two features: lock conversion and managing remote node locks? If yes, then I can (hopefully) prepare git patches next week.
I'm not quite sure what do you mean by 'managing remote node locks' ?
Activation/deactivation of LVs on a different corosync cluster node, specified by its node name (with pacemaker-like method to determine that name).
Also conversion of locks on that node.Quoted 6 lines
Current login behind lvm command is -
You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote node in via some configuration tags)Could you please explain this - I do not see anything relevant in man pages.
Quoted 2 lines
And you want to 'upgrade' remote locks to something else ?
Yes, shared-to-exclusive end vice verse.
Quoted 3 lines
What would be the use-case you could not resolve with current command line args?
I need to convert lock on a remote node during last stage of ver3 migration in libvirt/qemu. That is a "confirm" stage, which runs on a "source" node, during which "old" vm is killed and disk is released. So, I first ("begin" stage) convert lock from exclusive to shared on a source node, then obtain shared lock on a target node (during "prepare" stage, when receiving qemu instance is started), then migrate vm between two processes which have LV opened, and then release shared lock on a source node ("confirm" stage, after source qemu is killed).
There is no other events on a destination node in ver3 migration protocol, so I'm unable to convert lock to exclusive there after migration is finished. So I do that from a source node, after it released lock.
Quoted 2 lines
Is that supported by dlm (since lvm locks are mapped to dlm)?
Command just sent to a specific clvm instance and performed there.
Quoted 1 line
How would you resolve error path fallbacks ?
Quoted 8 lines
Could you please tell what exactly do you mean?
If dlm on a remote node is unable to perform requested operation, then error is returned to a initiator.Quoted 2 lines
Also I believe the clvmd protocol is out of free bits for extension, so how the protocol would look like ?
It contains 'node' field (I assume it was never actually used before) and with some fixes that works.
Fix to myself. clvm client API has that field. And there is no need for changes in on-wire protocol - corosync module already allows to send a message to a specific csid (node).
Quoted 9 lines
Vladislav
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
In reply to Vladislav BogdanovDne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
Quoted 11 lines
Quoted 8 lines · Zdenek Kabelac
Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
Quoted 5 lines
Quoted 3 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 2 lines · Vladislav Bogdanov
Quoted 1 line
You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote node in via some configuration tags)
Could you please explain this - I do not see anything relevant in man pages.
Let's say - you have 3 nodes A, B, C - each have a TAG_A, TAG_B, TAG_C, then on node A you may exclusively activate LV which has TAG_B - this will try to exclusively active LV on the node which has it configured in lvm.conf (see the volume_list= [])
Quoted 5 lines
Quoted 3 lines
And you want to 'upgrade' remote locks to something else ?
Yes, shared-to-exclusive end vice verse.
So how do you convert the lock from shared to exclusive without unlock
(if I get it right - you keep the ConcurrentRead lock - and you want to take Exlusive - to make change state from 'active' to 'active exlusive')
https://en.wikipedia.org/wiki/Distributed_lock_manager
Clvmd 'communicates' via these locks.
So the proper algorithm needs to be there for ending with some proper state after locks changes (and sorry I'm not a dlm expert here)
Quoted 9 lines
Quoted 3 lines
What would be the use-case you could not resolve with current command line args?
I need to convert lock on a remote node during last stage of ver3 migration in libvirt/qemu. That is a "confirm" stage, which runs on a "source" node, during which "old" vm is killed and disk is released. So, I first ("begin" stage) convert lock from exclusive to shared on a source node, then obtain shared lock on a target node (during "prepare"
Which most probably works only your environment - since you do not try to 'break' the logic - but it's probably not a generic concept - i.e. in this controlled environment you may live probably happily even with local activation of LVs - since you always know what the tool is doing.
Quoted 8 lines
There is no other events on a destination node in ver3 migration protocol, so I'm unable to convert lock to exclusive there after migration is finished. So I do that from a source node, after it released lock.
Quoted 2 lines
Is that supported by dlm (since lvm locks are mapped to dlm)?
Command just sent to a specific clvm instance and performed there.
As said - the 'lock' is the thing which controls the activation state.
So faking it on the software level may possible lead to inconsistency between the dlm and clvmd view of the lock state.
Zdenek
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Quoted 11 lines · Zdenek Kabelac
Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
Quoted 10 lines
Quoted 8 lines · Zdenek Kabelac
Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
Quoted 5 lines
Quoted 3 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 2 lines · Vladislav Bogdanov
Quoted 1 line
You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote node in via some configuration tags)
Quoted 7 lines
Quoted 2 lines
Could you please explain this - I do not see anything relevant in man pages.
Let's say - you have 3 nodes A, B, C - each have a
TAG_A,TAG_B,TAG_C, then on node A you may exclusively activate LV which hasTAG_B- this will try to exclusively active LV on the node which has it configured in lvm.conf (see the volume_list= [])
Aha, if I understand correctly this is absolutely not what I need. I want all this to be fully dynamic without any "config-editing voodoo".
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
And you want to 'upgrade' remote locks to something else ?
Yes, shared-to-exclusive end vice verse.
So how do you convert the lock from shared to exclusive without unlock
(if I get it right - you keep the ConcurrentRead lock - and you want to take Exlusive - to make change state from 'active' to 'active exlusive')
https://en.wikipedia.org/wiki/Distributed_lock_manager
I just pass LCKF_CONVERT to dlm_controld if requested and needed. And that is dlm's task to either satisfy conversion or to refuse it.
Quoted 2 lines
Clvmd 'communicates' via these locks.
Not exactly true.
clvmd does cluster communications with corosync, which implements virtual synchrony, so all cluster nodes receive messages in the same order. At the bottom, clvmd uses libdlm to communicate with dlm_controld and request it to lock/unlock.dlm_controld instances use corosync for membership and locally manages in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to communicate.
So request from one clvmd instance goes to another and goes to kernel from there, and then it is distributed to other nodes. Actually that does not matter where does it hits kernel space if it supports delegation of locks to remote nodes, but I suspect it doesn't. But if it doesn't support such thing, then the only option to manage lock on a remote node is to request that's node dlm instance to do the locking job.
Quoted 2 lines
So the proper algorithm needs to be there for ending with some proper state after locks changes (and sorry I'm not a dlm expert here)
That is what actually happens.
There is just no difference between running (to upgrade local lock to exclusive on node <node>.
ssh <node> lvchange -aey --force VG/LV
or
lvchange -aey --node <node> --force VG/LV
The same command, it is just sent via different channels.
Again, I just send locking request to a remote clvmd instance through corosync.
It asks its local dlm to convert (acquire, release) lock and returns its answer back. After dlm answers, operation is either performed, and then OK is send back to a initiator, or refused, and the error is sent back.
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
What would be the use-case you could not resolve with current command line args?
I need to convert lock on a remote node during last stage of ver3 migration in libvirt/qemu. That is a "confirm" stage, which runs on a "source" node, during which "old" vm is killed and disk is released. So, I first ("begin" stage) convert lock from exclusive to shared on a source node, then obtain shared lock on a target node (during "prepare"
Quoted 4 lines
Which most probably works only your environment - since you do not try to 'break' the logic - but it's probably not a generic concept - i.e. in this controlled environment you may live probably happily even with local activation of LVs - since you always know what the tool is doing.
I prefer to do not allow anybody else to open an LV at all (and to be able to take snapshots of it when needed on a node where it is exclusively locked). That's why I implemented a lock-manager driver for libvirt, which controls inter-node locks with clvm exclusive activation, and then controls inner open attempts with local locking mechanism.
While libvirt already has two lok drivers, neither of fully satisfy me because or shared storage requirement. And, I want to take snapshots, that's why I need exclusive activation.
I plan to send libvirt patches to the libvirt list if lvm part is accepted (otherwise it makes no sense to do that and I will need to maintain everything out-of-the-tree).
Quoted 7 lines
Quoted 7 lines
There is no other events on a destination node in ver3 migration protocol, so I'm unable to convert lock to exclusive there after migration is finished. So I do that from a source node, after it released lock.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
Is that supported by dlm (since lvm locks are mapped to dlm)?
Command just sent to a specific clvm instance and performed there.
As said - the 'lock' is the thing which controls the activation state. So faking it on the software level may possible lead to inconsistency between the dlm and clvmd view of the lock state.
No faking. Just a remote management of the same lock.
Vladislav
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
Quoted 30 lines
Quoted 11 lines · Zdenek Kabelac
Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
Quoted 10 lines
Quoted 8 lines · Zdenek Kabelac
Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
Quoted 5 lines
Quoted 3 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 2 lines · Vladislav Bogdanov
Quoted 1 line
You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote node in via some configuration tags)
Quoted 7 lines
Quoted 2 lines
Could you please explain this - I do not see anything relevant in man pages.
Let's say - you have 3 nodes A, B, C - each have a
TAG_A,TAG_B,TAG_C, then on node A you may exclusively activate LV which hasTAG_B- this will try to exclusively active LV on the node which has it configured in lvm.conf (see the volume_list= [])Aha, if I understand correctly this is absolutely not what I need. I want all this to be fully dynamic without any "config-editing voodoo".
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
And you want to 'upgrade' remote locks to something else ?
Yes, shared-to-exclusive end vice verse.
Quoted 8 lines
Quoted 4 lines
So how do you convert the lock from shared to exclusive without unlock
(if I get it right - you keep the ConcurrentRead lock - and you want to take Exlusive - to make change state from 'active' to 'active exlusive')
https://en.wikipedia.org/wiki/Distributed_lock_managerI just pass
LCKF_CONVERTtodlm_controldif requested and needed. And that is dlm's task to either satisfy conversion or to refuse it.
So to understand myself better this thing -
the dlm sends 'unlock' requests to all other nodes except the one which should be converted to exclusive mode and send exclusive lock to the prefered node?
Quoted 26 lines
Quoted 2 lines
Clvmd 'communicates' via these locks.
Not exactly true.
clvmd does cluster communications with corosync, which implements virtual synchrony, so all cluster nodes receive messages in the same order. At the bottom, clvmd uses libdlm to communicate with
dlm_controldand request it to lock/unlock.dlm_controldinstances use corosync for membership and locally manages in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to communicate.
So request from one clvmd instance goes to another and goes to kernel from there, and then it is distributed to other nodes. Actually that does not matter where does it hits kernel space if it supports delegation of locks to remote nodes, but I suspect it doesn't. But if it doesn't support such thing, then the only option to manage lock on a remote node is to request that's node dlm instance to do the locking job.Quoted 2 lines
So the proper algorithm needs to be there for ending with some proper state after locks changes (and sorry I'm not a dlm expert here)
That is what actually happens.
There is just no difference between running (to upgrade local lock to exclusive on node <node>.
Quoted 5 lines
ssh <node> lvchange -aey --force VG/LV
or
lvchange -aey --node <node> --force VG/LV
--node is exactly what the tag is for - each node may have it's tag. lvm doesn't work with cluster nodes.
The question is - could be the code transformed to use this logic ? I guess you need to know dlm node name here right ?
Quoted 15 lines
The same command, it is just sent via different channels.
Again, I just send locking request to a remote clvmd instance through corosync.
It asks its local dlm to convert (acquire, release) lock and returns its answer back. After dlm answers, operation is either performed, and then OK is send back to a initiator, or refused, and the error is sent back.Quoted 6 lines
Quoted 6 lines
There is no other events on a destination node in ver3 migration protocol, so I'm unable to convert lock to exclusive there after migration is finished. So I do that from a source node, after it released lock.
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
Is that supported by dlm (since lvm locks are mapped to dlm)?
Command just sent to a specific clvm instance and performed there.
As said - the 'lock' is the thing which controls the activation state. So faking it on the software level may possible lead to inconsistency between the dlm and clvmd view of the lock state.
No faking. Just a remote management of the same lock.
Could you repost patches against git ?
With some usage examples ?
Zdenek
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Quoted 42 lines · Zdenek Kabelac
Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
Quoted 31 lines
Quoted 12 lines · Zdenek Kabelac
Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
Quoted 11 lines
Quoted 9 lines · Zdenek Kabelac
Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
Quoted 5 lines
Quoted 3 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 2 lines · Vladislav Bogdanov
Quoted 1 line
You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote
node in via some configuration tags)Quoted 7 lines
Quoted 2 lines
Could you please explain this - I do not see anything relevant in man pages.
Let's say - you have 3 nodes A, B, C - each have a
TAG_A,TAG_B,TAG_C, then on node A you may exclusively activate LV which hasTAG_B- this will try to exclusively active LV on the node which has it configured in lvm.conf (see the volume_list= [])Aha, if I understand correctly this is absolutely not what I need. I want all this to be fully dynamic without any "config-editing voodoo".
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
And you want to 'upgrade' remote locks to something else ?
Yes, shared-to-exclusive end vice verse.
Quoted 9 lines
Quoted 5 lines
So how do you convert the lock from shared to exclusive without unlock
(if I get it right - you keep the ConcurrentRead lock - and you want to take Exlusive - to make change state from 'active' to 'active exlusive')
https://en.wikipedia.org/wiki/Distributed_lock_managerI just pass
LCKF_CONVERTtodlm_controldif requested and needed. And that is dlm's task to either satisfy conversion or to refuse it.
Quoted 5 lines
So to understand myself better this thing -
the dlm sends 'unlock' requests to all other nodes except the one which should be converted to exclusive mode and send exclusive lock to the prefered node?
No.
clvmd sends request to a remote clvmd to upgrade or acquire or release the lock.
That remote instance asks local dlm to do the job. dlm either says OK or says ERROR.
It does not do anything except that.
It LV is locked on a remote node, be it shared or exclusive lock, dlm says ERROR if exclusive lock (or conversion to it) is requested.
My patches also allow "-an --force" to release shared locks on other nodes. Exclusive lock may be released or downgraded only on node which holds it (or with --node <node>).
Quoted 30 lines
Quoted 30 lines
Quoted 3 lines
Clvmd 'communicates' via these locks.
Not exactly true.
clvmd does cluster communications with corosync, which implements virtual synchrony, so all cluster nodes receive messages in the same order.
At the bottom, clvmd uses libdlm to communicate withdlm_controldand request it to lock/unlock.dlm_controldinstances use corosync for membership and locally manages in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to communicate.
So request from one clvmd instance goes to another and goes to kernel from there, and then it is distributed to other nodes. Actually that does not matter where does it hits kernel space if it supports delegation of locks to remote nodes, but I suspect it doesn't. But if it doesn't support such thing, then the only option to manage lock on a remote node is to request that's node dlm instance to do the locking job.Quoted 2 lines
So the proper algorithm needs to be there for ending with some proper state after locks changes (and sorry I'm not a dlm expert here)
That is what actually happens.
There is just no difference between running (to upgrade local lock to exclusive on node <node>.ssh <node> lvchange -aey --force VG/LV
Quoted 7 lines
Quoted 3 lines
or
lvchange -aey --node <node> --force VG/LV
--node is exactly what the tag is for - each node may have it's tag. lvm doesn't work with cluster nodes.
But corosync and dlm operate node IDs, and pacemaker operates node names and IDs. None of them use tags.
Quoted 3 lines
The question is - could be the code transformed to use this logic ? I guess you need to know dlm node name here right ?
Node IDs are obtained from corosync membership list, and may be used for that. If corosync is configured with nodelist in a way pacemaker wants it
(http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html),
then node names may be used too.
Quoted 20 lines
Quoted 20 lines
The same command, it is just sent via different channels.
Again, I just send locking request to a remote clvmd instance through corosync.
It asks its local dlm to convert (acquire, release) lock and returns its answer back. After dlm answers, operation is either performed, and then OK is send back to a initiator, or refused, and the error is sent back.Quoted 9 lines
Quoted 8 lines
There is no other events on a destination node in ver3 migration protocol, so I'm unable to convert lock to exclusive there after migration is finished. So I do that from a source node, after it released lock.
Quoted 2 lines
Is that supported by dlm (since lvm locks are mapped to dlm)?
Command just sent to a specific clvm instance and performed there.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
As said - the 'lock' is the thing which controls the activation state. So faking it on the software level may possible lead to inconsistency between the dlm and clvmd view of the lock state.
No faking. Just a remote management of the same lock.
Could you repost patches against git ?
I plan that for the next week.
Quoted 1 line
With some usage examples ?
Yes, if you give me an "example of example" ;)
Vladislav
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
Dne 15.3.2013 16:36, Vladislav Bogdanov napsal(a):
Quoted 104 lines
Quoted 42 lines · Zdenek Kabelac
Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
Quoted 31 lines
Quoted 12 lines · Zdenek Kabelac
Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
Quoted 11 lines
Quoted 9 lines · Zdenek Kabelac
Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
Quoted 5 lines
Quoted 3 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 2 lines · Vladislav Bogdanov
Quoted 1 line
You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote
node in via some configuration tags)Quoted 7 lines
Quoted 2 lines
Could you please explain this - I do not see anything relevant in man pages.
Let's say - you have 3 nodes A, B, C - each have a
TAG_A,TAG_B,TAG_C, then on node A you may exclusively activate LV which hasTAG_B- this will try to exclusively active LV on the node which has it configured in lvm.conf (see the volume_list= [])Aha, if I understand correctly this is absolutely not what I need. I want all this to be fully dynamic without any "config-editing voodoo".
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
And you want to 'upgrade' remote locks to something else ?
Yes, shared-to-exclusive end vice verse.
Quoted 9 lines
Quoted 5 lines
So how do you convert the lock from shared to exclusive without unlock
(if I get it right - you keep the ConcurrentRead lock - and you want to take Exlusive - to make change state from 'active' to 'active exlusive')
https://en.wikipedia.org/wiki/Distributed_lock_managerI just pass
LCKF_CONVERTtodlm_controldif requested and needed. And that is dlm's task to either satisfy conversion or to refuse it.Quoted 5 lines
So to understand myself better this thing -
the dlm sends 'unlock' requests to all other nodes except the one which should be converted to exclusive mode and send exclusive lock to the prefered node?
No.
clvmd sends request to a remote clvmd to upgrade or acquire or release the lock.
That remote instance asks local dlm to do the job. dlm either says OK or says ERROR.
It does not do anything except that.
It LV is locked on a remote node, be it shared or exclusive lock, dlm says ERROR if exclusive lock (or conversion to it) is requested.My patches also allow "-an --force" to release shared locks on other nodes. Exclusive lock may be released or downgraded only on node which holds it (or with --node <node>).
Quoted 30 lines
Quoted 30 lines
Quoted 3 lines
Clvmd 'communicates' via these locks.
Not exactly true.
clvmd does cluster communications with corosync, which implements virtual synchrony, so all cluster nodes receive messages in the same order.
At the bottom, clvmd uses libdlm to communicate withdlm_controldand request it to lock/unlock.dlm_controldinstances use corosync for membership and locally manages in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to communicate.
So request from one clvmd instance goes to another and goes to kernel from there, and then it is distributed to other nodes. Actually that does not matter where does it hits kernel space if it supports delegation of locks to remote nodes, but I suspect it doesn't. But if it doesn't support such thing, then the only option to manage lock on a remote node is to request that's node dlm instance to do the locking job.Quoted 2 lines
So the proper algorithm needs to be there for ending with some proper state after locks changes (and sorry I'm not a dlm expert here)
That is what actually happens.
There is just no difference between running (to upgrade local lock to exclusive on node <node>.ssh <node> lvchange -aey --force VG/LV
Quoted 7 lines
Quoted 3 lines
or
lvchange -aey --node <node> --force VG/LV
--node is exactly what the tag is for - each node may have it's tag. lvm doesn't work with cluster nodes.
But corosync and dlm operate node IDs, and pacemaker operates node names and IDs. None of them use tags.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
The question is - could be the code transformed to use this logic ? I guess you need to know dlm node name here right ?
Node IDs are obtained from corosync membership list, and may be used for that. If corosync is configured with nodelist in a way pacemaker wants it
(http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html),
then node names may be used too.
clvmd knows the dlm node name - but lvm command should reference things via tags. There will be probably no way to add '--node' option into lvm command.
Can you think about using 'tags'?
So your machines will have configured tags (be it machine name) and instead of the node - you would use 'tag' ?
Zdenek
Re: [linux-lvm] LVM snapshot with Clustered VG [SOLVED]
15.03.2013 18:55, Zdenek Kabelac пишет:
Quoted 122 lines
Dne 15.3.2013 16:36, Vladislav Bogdanov napsal(a):
Quoted 114 lines
Quoted 46 lines · Zdenek Kabelac
Dne 15.3.2013 15:51, Vladislav Bogdanov napsal(a):
Quoted 41 lines
Quoted 12 lines · Zdenek Kabelac
Dne 15.3.2013 13:53, Vladislav Bogdanov napsal(a):
Quoted 11 lines
Quoted 9 lines · Zdenek Kabelac
Dne 15.3.2013 10:29, Vladislav Bogdanov napsal(a):
Quoted 5 lines
Quoted 3 lines · Zdenek Kabelac
Dne 14.3.2013 22:57, Andreas Pflug napsal(a):
Quoted 2 lines · Vladislav Bogdanov
Quoted 1 line
You could activate LVs with the above syntax [ael]
(there is a tag support - so you could exclusively activate LV on remote
node in via some configuration tags)Quoted 8 lines
Quoted 2 lines
Could you please explain this - I do not see anything relevant in man pages.
Let's say - you have 3 nodes A, B, C - each have a
TAG_A,TAG_B,TAG_C,
then on node A you may exclusively activate LV which hasTAG_B- this will try to exclusively active LV on the node which has it configured in lvm.conf (see the volume_list= [])Aha, if I understand correctly this is absolutely not what I need. I want all this to be fully dynamic without any "config-editing voodoo".
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
And you want to 'upgrade' remote locks to something else ?
Yes, shared-to-exclusive end vice verse.
Quoted 7 lines
So how do you convert the lock from shared to exclusive without
unlock
(if I get it right - you keep the ConcurrentRead lock - and you want to
take Exlusive - to make change state from 'active' to 'active exlusive')
https://en.wikipedia.org/wiki/Distributed_lock_managerQuoted 3 lines
I just pass
LCKF_CONVERTtodlm_controldif requested and needed. And that is dlm's task to either satisfy conversion or to refuse it.Quoted 5 lines
So to understand myself better this thing -
the dlm sends 'unlock' requests to all other nodes except the one which should be converted to exclusive mode and send exclusive lock to the prefered node?
No.
clvmd sends request to a remote clvmd to upgrade or acquire or release the lock.
That remote instance asks local dlm to do the job. dlm either says OK or says ERROR.
It does not do anything except that.
It LV is locked on a remote node, be it shared or exclusive lock, dlm says ERROR if exclusive lock (or conversion to it) is requested.My patches also allow "-an --force" to release shared locks on other nodes. Exclusive lock may be released or downgraded only on node which holds it (or with --node <node>).
Quoted 33 lines
Quoted 33 lines
Quoted 3 lines
Clvmd 'communicates' via these locks.
Not exactly true.
clvmd does cluster communications with corosync, which implements virtual synchrony, so all cluster nodes receive messages in the same order.
At the bottom, clvmd uses libdlm to communicate withdlm_controldand request it to lock/unlock.dlm_controldinstances use corosync for membership and locally manages in-kernel dlm counter-part, which uses tcp/sctp mesh-like connections to
communicate.
So request from one clvmd instance goes to another and goes to kernel from there, and then it is distributed to other nodes. Actually that does not matter where does it hits kernel space if it supports delegation of locks to remote nodes, but I suspect it doesn't. But if it
doesn't support such thing, then the only option to manage lock on a remote node is to request that's node dlm instance to do the locking job.Quoted 2 lines
So the proper algorithm needs to be there for ending with some proper state after locks changes (and sorry I'm not a dlm expert here)
That is what actually happens.
There is just no difference between running (to upgrade local lock to exclusive on node <node>.ssh <node> lvchange -aey --force VG/LV
Quoted 7 lines
Quoted 3 lines
or
lvchange -aey --node <node> --force VG/LV
--node is exactly what the tag is for - each node may have it's tag. lvm doesn't work with cluster nodes.
But corosync and dlm operate node IDs, and pacemaker operates node names and IDs. None of them use tags.
Quoted 3 lines
The question is - could be the code transformed to use this logic ? I guess you need to know dlm node name here right ?
Quoted 6 lines
Node IDs are obtained from corosync membership list, and may be used for that. If corosync is configured with nodelist in a way pacemaker wants it
(http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html),then node names may be used too.
Quoted 8 lines
clvmd knows the dlm node name - but lvm command should reference things via tags. There will be probably no way to add '--node' option into lvm command.
Can you think about using 'tags'?
So your machines will have configured tags (be it machine name) and instead of the node - you would use 'tag' ?
So, --tag should be used instead on --node?
I'd better make --node a alias for --tag for exactly this case if that is possible...
But... For those with cluster background --node is a native term while --tag is... alien one... So, I'd better leave it as is.
Quoted 3 lines
Zdenek