dlm_controld.pcmk: Fix membership change judging issue
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
Hi,
This is a fix to the membership judging issue in dlm_controld.pcmk. Now, dlm_controld.pcmk gets the membership change information from Pacemaker. Pacemaker get that information from Corosync, which is good. But when Pacemaker itself gets the membership change info, it does some internal processing like aligning the node membership as well as some other node info in the cluster. Before Pacemaker finished, it won't take the node in question as _active_ member. Just at that moment, dlm_controld.pcmk also knows the membership change and goes to read the membership info from Pacemaker. It is a race condition, because Pacemaker hasn't finished all the jobs in one membership change, which means not having finished updating all the info in crm_peer_id_cache, dlm_controld.pcmk read it! So if the node in question is a joining node, it should be regarded as "Added" node, but according to current logic, it is not!
Because all the components get the membership info eventually from Corosync, IMO, for dlm_controld.pcmk, there is no need to wait Pacemaker/crmd to finish all the information processing related to membership change.
Patched attached below, any review and comments are highly appreciated!
Thanks,
Jiaju
--- group/dlm_controld/pacemaker.c | 16 ++++++++++++---- 1 files changed, 12 insertions(+), 4 deletions(-)
group/dlm_controld/pacemaker.c +12 -4
@@ -81,6 +81,7 @@ int setup_cluster(void)void update_cluster (void ) {static uint64_t last_membership =0 ; +ais_dispatch (ais_fd_async,NULL ); cluster_quorate = crm_have_quorum;if (last_membership < crm_peer_seq) {log_debug ("Processing membership %llu" , crm_peer_seq);
@@ -91,7 +92,6 @@ void update_cluster(void)void process_cluster (int ci) { -ais_dispatch (ais_fd_async,NULL );update_cluster (); }
@@ -102,6 +102,14 @@ void close_cluster(void) {#include <arpa/inet.h> #include <corosync/totem/totemip.h> +static gboolean is_member (const crm_node_t *node) +{ +if (node &&safe_str_eq (node->state,CRM_NODE_MEMBER )) +return TRUE; + +return FALSE; +} +void dlm_process_node (gpointer key,gpointer value,gpointer user_data) {int rc =0 ;
@@ -119,7 +127,7 @@ void dlm_process_node(gpointer key, gpointer value, gpointer user_data)snprintf (path,PATH_MAX ,"%s/%d" ,COMMS_DIR , node->id); rc =stat (path, &tmp); - is_active =crm_is_member_active (node); + is_active =is_member (node);if (rc ==0 && is_active) {/* nothing to do?
@@ -212,7 +220,7 @@ void dlm_process_node(gpointer key, gpointer value, gpointer user_data) }log_debug ("%s %sctive node %u '%s': born-on=%llu, last-seen=%llu, this-event=%llu, last-event=%llu" , - action,crm_is_member_active (value)?"a" :"ina" , + action,is_member (value)?"a" :"ina" , node->id, node->uname, node->born, node->last_seen, crm_peer_seq, (unsigned long long )*last); }
@@ -220,7 +228,7 @@ void dlm_process_node(gpointer key, gpointer value, gpointer user_data)int is_cluster_member (uint32_t nodeid) {crm_node_t *node =crm_get_peer (nodeid,NULL ); -return crm_is_member_active (node); +return is_member (node); }char *nodeid2name (int nodeid) {
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
Quoted 24 lines · Lars Marowsky-Bree
On 2010-05-13T16:49:27, Jiaju Zhang <[email protected]> wrote:
Hi Jiaju,
thanks for the patch and the technical explanation!
Quoted 7 lines
Because all the components get the membership info eventually from Corosync, IMO, for dlm_controld.pcmk, there is no need to wait Pacemaker/crmd to finish all the information processing related to membership change.
Patched attached below, any review and comments are highly appreciated!
I'd only like to add the user-visible impact of this - namely, when a previously unknown node joins a pcmk cluster (with running DLM etc), nodes will hang.
So it is fairly important to get fixed.
So, with my limited knowledge of the code, I think the fix is sound. I have just one clerical comment:
Quoted 8 lines
Quoted 5 lines
?void
process_cluster(int ci)
?{
- ? ?ais_dispatch(ais_fd_async, NULL);
? ? ?update_cluster();
?}Can this function be removed? It doesn't do anything any more. Or is there something that should be done with the "ci" argument?
There is something that should be done with the "ci" argument. In fact, the "ci" means one slot which holds a socket(most cases) and a related processing function which is monitored by "poll" in dlm_controld.update_cluster is in different calling path to process_cluster.
Many thanks for the review ;)
Jiaju
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
In reply to a message not in the archiveQuoted 24 lines · Lars Marowsky-Bree
On 2010-05-13T16:49:27, Jiaju Zhang <[email protected]> wrote:
Hi Jiaju,
thanks for the patch and the technical explanation!
Quoted 7 lines
Because all the components get the membership info eventually from Corosync, IMO, for dlm_controld.pcmk, there is no need to wait Pacemaker/crmd to finish all the information processing related to membership change.
Patched attached below, any review and comments are highly appreciated!
I'd only like to add the user-visible impact of this - namely, when a previously unknown node joins a pcmk cluster (with running DLM etc), nodes will hang.
So it is fairly important to get fixed.
So, with my limited knowledge of the code, I think the fix is sound. I have just one clerical comment:
Quoted 7 lines
Quoted 5 lines
?void
process_cluster(int ci)
?{
- ? ?ais_dispatch(ais_fd_async, NULL);
? ? ?update_cluster();
?}Can this function be removed?
No, it can't.
Remove that and the membership (and crm_peer_id_cache) stops being updated.
Which makes me very suspicious of the whole patch because it clearly wasn't tested very well.
Quoted 13 lines
It doesn't do anything any more. Or is
there something that should be done with the "ci" argument?Regards,
? ?Lars--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG N?rnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
Quoted 10 lines · Lars Marowsky-Bree
Quoted 8 lines · Andrew Beekhof
Quoted 7 lines
Quoted 5 lines
?void
process_cluster(int ci)
?{
- ? ?ais_dispatch(ais_fd_async, NULL);
? ? ?update_cluster();
?}Can this function be removed?
Quoted 7 lines
Quoted 5 lines
No, it can't.
Remove that and the membership (andcrm_peer_id_cache) stops being updated.Which makes me very suspicious of the whole patch because it clearly wasn't tested very well.
ais_dispatch()was moved toupdate_cluster(). That's safe, it seems.
But pointless and unrelated to the problem, so why include it?
Quoted 3 lines
My question related to the detail that now
process_cluster()is identical toupdate_cluster(), at initial reading suggesting the function might be redundant now.
process_cluster() and update_cluster() are both API entry points.
Does the behavior still occur with pacemaker 1.1.2?
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
Quoted 3 lines · Andrew Beekhof
Does the behavior still occur with pacemaker 1.1.2?
Yes.
For the record, the most minimal testcase I've managed for this so far is as follows (substitute "/etc/init.d/corosync start" or whatever for "rcopenais start" if you're not on something SUSE-based):
1) Configure corosync/openais on two nodes.
Do not start the cluster yet.
2) On one node:
# rm /var/lib/heartbeat/crm/*
# rcopenais start
# while ! crm_mon -1 | grep -qi online; do \
echo -n "." ; sleep 5 ; done3) Now we have one node online, configure Pacemaker:
# cat <<CONF | crm configure
primitive dlm ocf:pacemaker:controld
primitive clvm ocf:lvm2:clvmd
group g dlm clvm
clone c g meta interleave="true"
property stonith-enabled="false"
property no-quorum-policy="ignore"
commit
CONF Watch "crm_mon -r" until that clone comes online. Should only take a few seconds.
4) On the other node:
# rm /var/lib/heartbeat/crm/*
# rcopenais startThe first node will now either wedge up spectacularly, and/or dlm_recoverd and clvmd will be stuck in D state on both nodes.
Regards,
Tim
Signature · 2 lines
Tim Serong <[email protected]> Senior Clustering Engineer, OPS Engineering, Novell Inc.
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
Quoted 36 lines · Tim Serong
Quoted 3 lines · Andrew Beekhof
Does the behavior still occur with pacemaker 1.1.2?
Yes.
For the record, the most minimal testcase I've managed for this so far is as follows (substitute "
/etc/init.d/corosyncstart" or whatever for "rcopenais start" if you're not on something SUSE-based):1) Configure corosync/openais on two nodes.
? Do not start the cluster yet.2) On one node:
? ? # rm
/var/lib/heartbeat/crm/*
? ? # rcopenais start
? ? # while !crm_mon-1 | grep -qi online; do \
? ? ? ? echo -n "." ; sleep 5 ; done3) Now we have one node online, configure Pacemaker:
? ? # cat <<CONF | crm configure
? ? primitive dlm ocf:pacemaker:controld
? ? primitive clvm ocf:lvm2:clvmd
? ? group g dlm clvm
? ? clone c g meta interleave="true"
? ? property stonith-enabled="false"
? ? property no-quorum-policy="ignore"
? ? commit
? ? CONF? Watch "
crm_mon-r" until that clone comes online. ? Should only take a few seconds.
Quoted 7 lines
4) On the other node:
? ? # rm
/var/lib/heartbeat/crm/*
? ? # rcopenais startThe first node will now either wedge up spectacularly, and/or
dlm_recoverdand clvmd will be stuck in D state on both nodes.
Presumably each thinks the other node isn't a member? Perhaps something like this will help:
lib/ais/plugin.c +2 -2
@@ -498,9 +498,8 @@ static void *pcmk_wait_dispatch (void *aais_notice ("Respawning failed child process: %s" , pcmk_children[lpc].name);spawn_child (&(pcmk_children[lpc])); - }else { -send_cluster_id (); } +send_cluster_id (); } }sched_yield ();
@@ -661,6 +660,7 @@ int pcmk_startup(struct corosync_api_v1 } } } +send_cluster_id ();return 0 ; }
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
I'll test the patch later on. But I may not finished this testing by today because
I have some problem to access the hardwre at this moment.
Thanks a lot ;-)
Jiaju
Quoted 72 lines · Andrew Beekhof
Quoted 36 lines · Tim Serong
Quoted 3 lines · Andrew Beekhof
Does the behavior still occur with pacemaker 1.1.2?
Yes.
For the record, the most minimal testcase I've managed for this so far is as follows (substitute "
/etc/init.d/corosyncstart" or whatever for "rcopenais start" if you're not on something SUSE-based):1) Configure corosync/openais on two nodes.
? Do not start the cluster yet.2) On one node:
? ? # rm
/var/lib/heartbeat/crm/*
? ? # rcopenais start
? ? # while !crm_mon-1 | grep -qi online; do \
? ? ? ? echo -n "." ; sleep 5 ; done3) Now we have one node online, configure Pacemaker:
? ? # cat <<CONF | crm configure
? ? primitive dlm ocf:pacemaker:controld
? ? primitive clvm ocf:lvm2:clvmd
? ? group g dlm clvm
? ? clone c g meta interleave="true"
? ? property stonith-enabled="false"
? ? property no-quorum-policy="ignore"
? ? commit
? ? CONF? Watch "
crm_mon-r" until that clone comes online. ? Should only take a few seconds.Quoted 7 lines
4) On the other node:
? ? # rm
/var/lib/heartbeat/crm/*
? ? # rcopenais startThe first node will now either wedge up spectacularly, and/or
dlm_recoverdand clvmd will be stuck in D state on both nodes.Presumably each thinks the other node isn't a member? Perhaps something like this will help:
diff -r
b59c27dc114alib/ais/plugin.clib/ais/plugin.c ?Fri May 14 12:12:33 2010 +0200 renamed from lib/ais/plugin.c ?Wed May 12 10:51:56 2010 +0200 +2
-2@@ -498,9 +498,8 @@ static void *pcmk_wait_dispatch (void *a ? ? ? ? ? ? ? ? ? ?ais_notice("Respawning failed child process: %s", ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? pcmk_children[lpc].name); ? ? ? ? ? ? ? ? ? ?spawn_child(&(pcmk_children[lpc])); - ? ? ? ? ? ? ? } else { - ? ? ? ? ? ? ? ? ? send_cluster_id(); ? ? ? ? ? ? ? ?} + ? ? ? ? ? ? ? send_cluster_id(); ? ? ? ? ? ?} ? ? ? ?} ? ? ? ?sched_yield ();@@ -661,6 +660,7 @@ int pcmk_startup(struct corosync_api_v1 ? ? ? ? ? ?} ? ? ? ?} ? ? } + ? ?send_cluster_id(); ? ? return 0; ?}
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
Quoted 6 lines · Lars Marowsky-Bree
Quoted 2 lines · Andrew Beekhof
Quoted 1 line
ais_dispatch()was moved toupdate_cluster(). That's safe, it seems.But pointless and unrelated to the problem, so why include it?
I took it that JJ thought this was part of the solution.
ci == ais_async_fd, if there was something to read we should have already had a callback.
Quoted 4 lines
Quoted 2 lines
Does the behavior still occur with pacemaker 1.1.2?
Yes. Which fix do you think would change this?
These two that I mentioned last week:
http://hg.clusterlabs.org/pacemaker/1.1/rev/34ae9f7b4675
http://hg.clusterlabs.org/pacemaker/1.1/rev/19fdc0a3885a
I really doubt the patch is the right fix, conceptually, the node can't be a member until the crmd process is running and done a whole heap of processing.
So CRM_NODE_MEMBER without crm_proc_crmd makes no sense.
More likely, as-in the patches above, the process list wasn't being updated correctly and thats what needs to get fixed.
It just never mattered before.
Quoted 14 lines
JJ's patch has been tested and seems to work in general as well as address the problem.
Regards,
? ?Lars--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG N?rnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
Quoted 12 lines · Andrew Beekhof
Quoted 6 lines · Lars Marowsky-Bree
Quoted 2 lines · Andrew Beekhof
Quoted 1 line
ais_dispatch()was moved toupdate_cluster(). That's safe, it seems.But pointless and unrelated to the problem, so why include it?
I took it that JJ thought this was part of the solution.
ci ==
ais_async_fd, ?if there was something to read we should have already had a callback.Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Does the behavior still occur with pacemaker 1.1.2?
Yes. Which fix do you think would change this?
These two that I mentioned last week:
?http://hg.clusterlabs.org/pacemaker/1.1/rev/34ae9f7b4675 ?http://hg.clusterlabs.org/pacemaker/1.1/rev/19fdc0a3885a
I've confirmed that my pacemaker has the above patch and it doesn't work.
Quoted 9 lines
I really doubt the patch is the right fix, conceptually, the node can't be a member until the crmd process is running and done a whole heap of processing.
SoCRM_NODE_MEMBERwithoutcrm_proc_crmdmakes no sense.More likely, as-in the patches above, the process list wasn't being updated correctly and thats what needs to get fixed.
It just never mattered before.
This is the where the race condition happens. CRM_NODE_MEMBER was set from the local node, but crm_proc_crmd was updated only when the peer node sending the local node this info. When CRM_NODE_MEMBER was updated but crm_proc_crmd was not updated yet, dlm_controld went to read the crm_peer_id_cache, this bug happened.
Yes, Updating CRM_NODE_MEMBER and updating crm_proc_crmd should be an _atomic_ operation, before that has finished, dlm_controld shoud not read the membership info. So in fact, I have been thinking about three solutions to this problem:
1) After updating (CRM_NODE_MEMBER && crm_proc_crmd) finished, notify dlm_controld. But I'm not sure if we can achieve this goal by doing some work around ais_async_fd. Maybe need to dig into the code further.
2) maybe add another flag to do some syncing between Pacemaker and dlm_controld, it might like what dlm_controld and fs_controld has done, but it may need much more work and a little complicated.
3) As the patch has done, only see CRM_NODE_MEMBER but not see crm_proc_crmd. It seems have risk?(open disscussion). But after some consideration, I think this should be OK. Because we eventually get the membership info from Corosync, even we haven't recevied the crm_proc_crmd info this moment, we can receive that info at once. If some bad thing happened and the peer node was unfortunately down, we can still received that info from Corosync. Also, the time we set the membership info to DLM is just the node membership info, we don't need to care if crmd is really started at this moment. The membership info also is udpated at many other places when we do some work about lockspace. So it should be safe as my think.
Thanks,
Jiaju
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
In reply to Andrew BeekhofQuoted 31 lines · Andrew Beekhof
Quoted 30 lines · Lars Marowsky-Bree
On 2010-05-13T16:49:27, Jiaju Zhang <[email protected]> wrote:
Hi Jiaju,
thanks for the patch and the technical explanation!
Quoted 7 lines
Because all the components get the membership info eventually from Corosync, IMO, for dlm_controld.pcmk, there is no need to wait Pacemaker/crmd to finish all the information processing related to membership change.
Patched attached below, any review and comments are highly appreciated!
I'd only like to add the user-visible impact of this - namely, when a previously unknown node joins a pcmk cluster (with running DLM etc), nodes will hang.
So it is fairly important to get fixed.
So, with my limited knowledge of the code, I think the fix is sound. I have just one clerical comment:
Quoted 5 lines
?void
process_cluster(int ci)
?{
- ? ?ais_dispatch(ais_fd_async, NULL);
? ? ?update_cluster();
?}
Quoted 4 lines
Quoted 1 line
Can this function be removed?
No, it can't.
Remove that and the membership (andcrm_peer_id_cache) stops being updated.
Agreed.
Both update_cluster and process_cluster can't be removed. Yes, moving the place of ais_dispatch should have nothing to do with this issue. Oh, this is because I added it in the debugging stage and forgot to remove it in the final version, just sending the working version to the list, sorry, I'll posted an updated patch soon ;-)
Quoted 2 lines
Which makes me very suspicious of the whole patch because it clearly wasn't tested very well.
I have tested in my environment and it works for me.
Thanks,
Jiaju
PATCH 1/1 dlm_controld.pcmk: Fix membership change judging issue
Quoted 6 lines · Jiaju Zhang
Both
update_clusterandprocess_clustercan't be removed. Yes, moving the place ofais_dispatchshould have nothing to do with this issue. Oh, this is because I added it in the debugging stage and forgot to remove it in the final version, just sending the working version to the list, sorry, I'll posted an updated patch soon ;-)
This is the updated version of this patch, thanks a lot for your review and comments ;-)
Thanks,
Jiaju
--- group/dlm_controld/pacemaker.c | 14 +++++++++++--- 1 files changed, 11 insertions(+), 3 deletions(-)
group/dlm_controld/pacemaker.c +11 -3
@@ -102,6 +102,14 @@ void close_cluster(void) {#include <arpa/inet.h> #include <corosync/totem/totemip.h> +static gboolean is_member (const crm_node_t *node) +{ +if (node &&safe_str_eq (node->state,CRM_NODE_MEMBER )) +return TRUE; + +return FALSE; +} +void dlm_process_node (gpointer key,gpointer value,gpointer user_data) {int rc =0 ;
@@ -119,7 +127,7 @@ void dlm_process_node(gpointer key, gpointer value, gpointer user_data)snprintf (path,PATH_MAX ,"%s/%d" ,COMMS_DIR , node->id); rc =stat (path, &tmp); - is_active =crm_is_member_active (node); + is_active =is_member (node);if (rc ==0 && is_active) {/* nothing to do?
@@ -212,7 +220,7 @@ void dlm_process_node(gpointer key, gpointer value, gpointer user_data) }log_debug ("%s %sctive node %u '%s': born-on=%llu, last-seen=%llu, this-event=%llu, last-event=%llu" , - action,crm_is_member_active (value)?"a" :"ina" , + action,is_member (value)?"a" :"ina" , node->id, node->uname, node->born, node->last_seen, crm_peer_seq, (unsigned long long )*last); }
@@ -220,7 +228,7 @@ void dlm_process_node(gpointer key, gpointer value, gpointer user_data)int is_cluster_member (uint32_t nodeid) {crm_node_t *node =crm_get_peer (nodeid,NULL ); -return crm_is_member_active (node); +return is_member (node); }char *nodeid2name (int nodeid) {