dlm_controld: outputs explicit info about stateful

Hi David,
This is just a draft patch for you to review;-) There's an issue I'm not sure: where should we clear "stateful_merge_wait"?

And I need more communications with pacemaker guys and more time for testing. I will send you the formal patch if things get done;-)

Thanks,
Eric

When there are 3 or more partitions that merge, none may see enough clean nodes. Therefore, DLM would be stuck there forever unitl administrator manually reset/restart enough nodes to produce sufficient clean nodes. Therefore, output explicit information for higher code (e.g. pcmk) about the stateful merging state. Now, higher code can use dlm status -v to get "stateful_merge_wait". If it equals "1", we know dlm is waiting manual intervention. Then, higher code can choose one of nodes to fence. DLM will continue to work if "clean nodes >= stateful merged nodes" becomes true.

David advised me to do the right thing;-) Thanks a lot!

---
 dlm_controld/daemon_cpg.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

index 356e80d..8f6434f 100644
@@ -118,6 +118,7 @@ static int zombie_count;
 
 static int fence_result_pid;
 static unsigned int fence_result_try;
+static int stateful_merge_wait; /* cluster is stuck in waiting for manual intervention */
 
 static void send_fence_result(int nodeid, int result, uint32_t flags, uint64_t walltime);
 static void send_fence_clear(int nodeid, int result, uint32_t flags, uint64_t walltime);

@@ -847,10 +848,13 @@ static void daemon_fence_work(void)
 
 		if ((clean_count >= merge_count) && !part_count && (low == our_nodeid))
 			kick_stateful_merge_members();
+		if ((clean_count < merge_count) && !part_count)
+			stateful_merge_wait = 1;
 
 		retry = 1;
 		goto out;
 	}
+	stateful_merge_wait = 0; /* where should this line go? */
 
 	/*
 	 * startup fencing

@@ -2382,7 +2386,8 @@ static int print_state_daemon(char *str)
 		 "fence_pid=%d "
 		 "fence_in_progress_unknown=%d "
 		 "zombie_count=%d "
-		 "monotime=%llu ",
+		 "monotime=%llu "
+		 "stateful_merge_wait=%d ",
 		 daemon_member_count,
 		 daemon_joined_count,
 		 daemon_remove_count,

@@ -2392,7 +2397,8 @@ static int print_state_daemon(char *str)
 		 daemon_fence_pid,
 		 fence_in_progress_unknown,
 		 zombie_count,
-		 (unsigned long long)monotime());
+		 (unsigned long long)monotime(),
+		 stateful_merge_wait);
 
 	return strlen(str) + 1;
 }

2.6.6

Hi David,

Ken Gaillot got me with this question:
Since corosync/pcmk can be healed from such a case, why not DLM? Please look at detailed discussion here:

        [1] https://github.com/ClusterLabs/pacemaker/pull/839

Here is my thoughts, but I'm not sure, CMIIW please:
time: T; cluster:A, B, C; and if we have a lockspace named after $uuid for a shared disk volume, and a CPG for lockspace $uuid; $uuid CPG has members of A, B and C when things are OK, but:

T: quorum lost; cluster partitions into 3 parts; lockspace $uuid cannot perform any lockspace operations because cluster is not quorate;

T+1: quorum regained; dlm_controld daemon CPG has not done its merging/fencing stuff; so here are 2 questions:
Q1: what's stateful merged node?
I've seen the comments within code;-) It means a lockspace has been on the node before it sends protocol message?

Q2: what if we add the stateful merged nodes to dlm_controld daemon cpg instead of fencing them?

if so, CPG $uuid now, e.g. from the perspective of A, may has only one memeber - A itself, it can perform lockspace now because cluster is quorate now (and if we skip fencing); B and C do likewise; then for each node, it looks like every node own this volume; so corruption may happen?

Thanks a lot,
Eric

The details here are fundamental to the way dlm works because the dlm depends on the properties of Virtual Synchrony. Partitions obviously violate VS. ("Extended" forms of virtual synchrony deal with partitions, but they are not very practical. Unfortunately, corosync implements one of these extended forms of VS, which means any application that requires strict VS has to implement an equivalent of this "stateful merging" detection that's in the dlm.)

With VS, message/membership events change the state being kept consistent among nodes. When a partition occurs, nodes have divergent events and inconsistent state. The partition is simple to understand, because partitioned nodes are indistinguishable from failed nodes and are treated as such. But, if partitioned nodes merge, the inconsistent state has to be made consistent. This must be done in the same way a new node is added to an existing node, which means doing "state transfer" from the existing node to the new node to make the state consistent between them.

If the "new" node previously had state because of partition/merge, it must drop that old state and replace it with the state being transferred to it. After this, they will be consistent and can continue. With a simple process, you might just kill it, restart it and add the transferred state. But the dlm isn't a process that can simply be restarted, the state is spread through applications using it, and through the kernel. The only mechanism for resetting the dlm state is resetting the kernel, which is resetting/rebooting the machine.

When the nodes are partitioned, the situation is fairly straight forward -- each node thinks the others are failed, and normal operation is blocked until recovery happens for the failed nodes.

The harder problem is what to do when they merge. The dlm effectively ignores the invalid addition of the merged nodes and calls it a "stateful merge". The merged nodes continue to be considered failed (from the partition) and require a full restart before being added.

Hi David,

Thanks a lot for elaborating this valuable knowledge to me! I've also shard with pacemaker guys. They'll make corresponding changes on pcmk side once the patch of dlm_controld is merged. I've sent the patch to you. Please take a look at;-)

With best regards,
Eric