Skip to contentSkip to conversation

tabled: Add replication daemon

PATCH Applied Pete Zaitcev 2 participants 5 messages Nov 14, 2009 – Nov 15, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 5/7 tabled: Add replication daemon

This patch adds what amounts to a background process that maintains redundancy for object data. It is far from the complete solution. For one thing, it does not verify checksums. But it's a start.

There's no way to turn this off, by intention. The whole thing must work very reliably, not steal too much from benchmarks (but if it does, it's only honest to take the hit). It is indispensible. However, there's a plan to add useful monitoring of jobs and other state, such as available nodes.

---
 server/Makefile.am |    4 
 server/replica.c   |  753 +++++++++++++++++++++++++++++++++++++++++++
 server/server.c    |   25 +
 server/tabled.h    |   12 
 4 files changed, 789 insertions(+), 5 deletions(-)
server/Makefile.am +2 -2
index f994b36..8faa95a 100644
@@ -4,8 +4,8 @@ INCLUDES	= -I$(top_srcdir)/include @GLIB_CFLAGS@ @CHUNKDC_CFLAGS@ @CLDC_CFLAGS@
 sbin_PROGRAMS	= tabled tdbadm
 
 tabled_SOURCES	= tabled.h		\
-		  bucket.c object.c server.c storage.c storparse.c cldu.c \
-		  config.c util.c
+		  bucket.c object.c server.c storage.c storparse.c replica.c \
+		  cldu.c config.c util.c
 tabled_LDADD	= ../lib/libhttputil.a ../lib/libtdb.a		\
 		  @CHUNKDC_LIBS@ @CLDC_LIBS@ @PCRE_LIBS@ @GLIB_LIBS@ \
 		  @CRYPTO_LIBS@ @DB4_LIBS@ @EVENT_LIBS@ @ARGP_LIBS@ @SSL_LIBS@
server/replica.c new file +753 -0

753 lines not on this page · raw message

server/server.c +24 -1
index 2a504e2..673c151 100644
@@ -1318,6 +1318,20 @@ static void tdb_checkpoint(int fd, short events, void *userdata)
 	add_chkpt_timer();
 }
 
+static void add_kscan_timer(void)
+{
+	static const struct timeval tv = { TABLED_RESCAN_SEC, 0 };
+
+	if (evtimer_add(&tabled_srv.kscan_timer, &tv) < 0)
+		applog(LOG_WARNING, "unable to add key scan timer");
+}
+
+static void tdb_keyscan(int fd, short events, void *userdata)
+{
+	rep_scan();
+	add_kscan_timer();
+}
+
 static void tdb_state_cb(enum db_event event)
 {
 
@@ -1365,7 +1379,7 @@ static void tdb_state_cb(enum db_event event)
  *
  * We don't even bother with registering this callback, just call it by name. 
  *
- * The return value is used to re-arm rescan mechanism.
+ * The return value is used to re-arm storage rescan mechanism.
  */
 int stor_update_cb(void)
 {
@@ -1383,8 +1397,15 @@ int stor_update_cb(void)
 				applog(LOG_DEBUG, " NID %u is up", stn->id);
 			num_up++;
 			stn->up = true;
+			stn->last_up = time(NULL);
+		} else {
+			if (stn->last_up != 0 &&
+			    time(NULL) >= stn->last_up + CHUNK_REBOOT_TIME) {
+				applog(LOG_INFO, " NID %u went down", stn->id);
+			}
 		}
 	}
+
 	if (num_up < 1) {
 		applog(LOG_INFO, "No active storage node(s), waiting");
 		return num_up;
@@ -1695,6 +1716,7 @@ static void tdb_state_process(enum st_tdb new_state)
 			return;
 		}
 		add_chkpt_timer();
+		add_kscan_timer();
 		net_listen();
 	}
 }
@@ -1772,6 +1794,7 @@ int main (int argc, char *argv[])
 
 	event_init();
 	evtimer_set(&tabled_srv.chkpt_timer, tdb_checkpoint, NULL);
+	evtimer_set(&tabled_srv.kscan_timer, tdb_keyscan, NULL);
 
 	/* set up server networking */
 	rc = net_open();
server/tabled.h +10 -2
index fd6142e..31cead2 100644
@@ -19,9 +19,9 @@
  *
  */
 
-
 #include <stdbool.h>
 #include <stdlib.h>
+#include <time.h>
 #include <netinet/in.h>
 #include <openssl/md5.h>
 #include <glib.h>
@@ -43,6 +43,9 @@ enum {
 	TABLED_PGSZ_LOCK	= 4096,
 
 	TABLED_CHKPT_SEC	= 60 * 5,	/* secs between db4 chkpt */
+	TABLED_RESCAN_SEC	= 60*3 + 7,	/* secs btw key rescans */
+
+	CHUNK_REBOOT_TIME	= 3*60,		/* secs to declare chunk dead */
 
 	CLI_REQ_BUF_SZ		= 8192,		/* buffer for req + hdrs */
 	CLI_DATA_BUF_SZ		= 8192,
@@ -89,6 +92,7 @@ struct storage_node {
 	struct list_head	all_link;
 	uint32_t		id;
 	bool			up;
+	time_t			last_up;
 
 	unsigned		alen;
 	int			addr_af;
@@ -115,7 +119,7 @@ struct open_chunk {
 	struct st_client	*stc;
 	struct storage_node	*node;
 	struct list_head	link;
-	struct client		*cli;
+	void			*cli;	/* usually struct client * */
 
 	uint64_t		wtogo;
 	uint64_t		wkey;
@@ -241,6 +245,7 @@ struct server {
 	enum st_net		state_net;
 
 	struct event		chkpt_timer;	/* db4 checkpoint timer */
+	struct event		kscan_timer;	/* db4 key rescan timer */
 
 	struct server_stats	stats;		/* global statistics */
 };
@@ -340,4 +345,7 @@ extern int stor_node_check(struct storage_node *stn);
 /* storparse.c */
 extern void stor_parse(char *fname, const char *text, size_t len);
 
+/* replica.c */
+extern void rep_scan(void);
+
 #endif /* __TABLED_H__ */
#1 lore raw
Quoted 11 lines · Pete Zaitcev

This patch adds what amounts to a background process that maintains redundancy for object data. It is far from the complete solution. For one thing, it does not verify checksums. But it's a start.

There's no way to turn this off, by intention. The whole thing must work very reliably, not steal too much from benchmarks (but if it does, it's only honest to take the hit). It is indispensible. However, there's a plan to add useful monitoring of jobs and other state, such as available nodes.

Just so you're not left hanging, regarding this patch, I'm holding off in-depth review. The other patches were easy, quick to apply without much post-3am thinking... ;)

Jeff

#2 lore raw
In reply to Pete Zaitcev
Quoted 40 lines · Pete Zaitcev
server/replica.c new file +40 -0
+void rep_scan(void)
+{
+	struct cursor cur;
+	struct db_obj_ent *obj;
+	unsigned long kcnt;
+	time_t start_time, t;
+
+	rep_retire();
+
+	start_time = time(NULL);
+	if (debugging)
+		applog(LOG_DEBUG, "key scan start time %lu", (long)start_time);
+
+	memset(&cur, 0, sizeof(struct cursor));	/* enough to construct */
+	cur.db_env = tdb.env;
+	cur.db_objs = tdb.objs;
+
+	kcnt = 0;
+	for (;;) {
+		if ((t = time(NULL))>= start_time + 2) {
+			if (debugging)
+				applog(LOG_DEBUG,
+				       "db release at keys %lu seconds %lu",
+				       kcnt, (long)t);
+			rep_scan_close(&cur);
+		}
+
+		if (rep_scan_get(&cur,&obj) != 0)
+			break;
+
+		/* not needed for db4 with DB_NEXT, but eases our logic */
+		if (rep_scan_parse(&cur, obj) != 0) {
+			free(obj);
+			continue;
+		}
+
+		if (!GUINT32_FROM_LE(obj->flags)&  DB_OBJ_INLINE)
+			rep_scan_verify(&cur, obj);
+
+		free(obj);

server/replica.c line 751

Quoted 10 lines
server/replica.c new file +10 -0
+		kcnt++;
+	}
+
+	rep_scan_close(&cur);
+	free(cur.key);
+	cur.key = NULL;
+
+	if (debugging)
+		applog(LOG_DEBUG, "key scan done keys %lu", kcnt);
+	return;

Major comments:

1) What is the point of db->del() in rep_add_nid() ? You are in the middle of a transaction, and you immediately overwrite that record in the same transaction. That is clearly unnecessary work, when db->put() will simply overwrite an existing record, if requested.

2) rep_scan(): I would rather not make the entire daemon non-responsive for multiple seconds.

The database is already in multi-threaded mode, and db4 is free-threaded, so it would seem to make a lot more sense to simply g_thread_create() a thread to do this work.

Also, there should be no need to scan a chunkd's keys at all, as long as the chunkd instance is still communicating with us.

Jeff

#3 lore raw
Quoted 4 lines · Jeff Garzik

1) What is the point of db->del() in rep_add_nid() ? You are in the middle of a transaction, and you immediately overwrite that record in the same transaction. That is clearly unnecessary work, when db->put() will simply overwrite an existing record, if requested.

That didn't work last I tested, but I forgot what the issue was. The code is copied from object.c. I'll review it.

Quoted 6 lines

2) rep_scan(): I would rather not make the entire daemon non-responsive for multiple seconds.

The database is already in multi-threaded mode, and db4 is free-threaded, so it would seem to make a lot more sense to simply g_thread_create() a thread to do this work.

True, but multi-threading in tabled is a much bigger undertaking than just "simply" creating a thread. There's a lot of common state. I think the 2s problem is less urgent than processing massive replication more efficiently (from a Chunk down).

Quoted 2 lines

Also, there should be no need to scan a chunkd's keys at all, as long as the chunkd instance is still communicating with us.

No, there's a need, although there's no code for it: the scan should probe keys by requesting their metadata, which forturously includes checksums. This is how the results of Chunk's self-test will be communicated to tabled.

-- Pete

#4 lore raw
Quoted 9 lines · Pete Zaitcev
Quoted 4 lines · Jeff Garzik

1) What is the point of db->del() in rep_add_nid() ? You are in the middle of a transaction, and you immediately overwrite that record in the same transaction. That is clearly unnecessary work, when db->put() will simply overwrite an existing record, if requested.

That didn't work last I tested, but I forgot what the issue was. The code is copied from object.c. I'll review it.

The only code matching that pattern is object_put_end(), which also performs unnecessary work of needlessly calling db->del().

This should be recognized as nothing more than programmer laziness, enabled by the sharing of __object_del() between DEL and PUT, and the necessity of deleting associated records (ACLs) in conjunction with the main [deletion | update].

That does not apply to rep_add_nid(), which has neither object data nor related records to delete.

(patches accepted to remove ->del from PUT path, too, if you wish)

Quoted 11 lines
Quoted 6 lines

2) rep_scan(): I would rather not make the entire daemon non-responsive for multiple seconds.

The database is already in multi-threaded mode, and db4 is free-threaded, so it would seem to make a lot more sense to simply g_thread_create() a thread to do this work.

True, but multi-threading in tabled is a much bigger undertaking than just "simply" creating a thread. There's a lot of common state. I think the 2s problem is less urgent than processing massive replication more efficiently (from a Chunk down).

I only see one global variable reference in the whole nicely-designed, nicely self-contained replica.c file. "a lot of common state" seems like quite an exaggeration.

This patch resuscitates the worst application model from the late 1980's, cooperative multi-tasking. I do not want to walk down that road: such programs have weird pauses at weird times, and behave in a very non-deterministic manner.

Furthermore, I think you will find that threading gives you a lot more freedom to work in the background.

Quoted 7 lines
Quoted 2 lines

Also, there should be no need to scan a chunkd's keys at all, as long as the chunkd instance is still communicating with us.

No, there's a need, although there's no code for it: the scan should probe keys by requesting their metadata, which forturously includes checksums. This is how the results of Chunk's self-test will be communicated to tabled.

What is the need? This description provides "what" and "how" but does not answer "why?"

Jeff