Chunk: retry initial CLD session open

This was an error in the conversion to ncld. In the cldc code, we kick the state machine and the natural retries do the rest. Any failures occure there. But in ncld the original kick can fail too.

Five retries give CLD server time to reboot. If it's down, then clients refuse to start. This may be a bad idea, or may be not. We may yet change the retries to be infinite, but for now it's better if builds terminate somehow in case of unexpected problems.

---
 server/cldu.c |   12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

commit 44cdb98d2cceb2f4e081db2ee38ec60f1c1a8d8d

Author: Master <[email protected]>
Date:   Sat Apr 17 19:50:06 2010 -0600

Retry the initial connection to the CLD server.

index fafcc3b..58edf4b 100644
@@ -471,6 +471,8 @@ int cld_begin(const char *thishost, uint32_t nid, char *infopath,
 {
 	static struct cld_session *cs = &ses;
 	struct server_poll *sp;
+	int retry_cnt;
+	int newactive;
 
 	if (!nid)
 		return 0;

@@ -540,9 +542,15 @@ int cld_begin(const char *thishost, uint32_t nid, char *infopath,
 	 * -- Actually, it only works when recovering from CLD failure.
 	 *    Thereafter, any slave CLD redirects us to the master.
 	 */
-	if (cldu_set_cldc(cs, 0)) {
+	newactive = 0;
+	retry_cnt = 0;
+	for (;;) {
+		if (!cldu_set_cldc(cs, newactive))
+			break;
 		/* Already logged error */
-		goto err_net;
+		if (++retry_cnt == 5)
+			goto err_net;
+		newactive = cldu_nextactive(cs);
 	}
 
 	return 0;

In the short term, this is acceptable.

In the medium term, this is a protocol detail that should be handled somewhere in libcldc. We want all applications to behave the same way, including the method by which they attempt to contact a master.

Because there could be multiple CLD servers, you cannot think of retries in the context of a single server. This is crucial WRT work on #replica branch, but it is also somewhat relevant to #master, because we might have multiple servers listed in SRV records as fallbacks from which to choose.

You don't want each application implementing this logic, because we want to enforce some level of predictability in master-seeking behavior, and in making decisions about when contacts attempts for -all- servers should cease, as opposed to contact attempts for a -single- server. You don't want it to take 30 minutes to try all servers in a cluster, retrying a number of times on server A, then moving on to server B, etc.

Jeff