tabled: Add replication daemon
PATCH 5/7 tabled: Add replication daemon
This patch adds what amounts to a background process that maintains redundancy for object data. It is far from the complete solution. For one thing, it does not verify checksums. But it's a start.
There's no way to turn this off, by intention. The whole thing must work very reliably, not steal too much from benchmarks (but if it does, it's only honest to take the hit). It is indispensible. However, there's a plan to add useful monitoring of jobs and other state, such as available nodes.
--- server/Makefile.am | 4 server/replica.c | 753 +++++++++++++++++++++++++++++++++++++++++++ server/server.c | 25 + server/tabled.h | 12 4 files changed, 789 insertions(+), 5 deletions(-)
server/Makefile.am +2 -2
@@ -4,8 +4,8 @@ INCLUDES = -I$(top_srcdir)/include @GLIB_CFLAGS@ @CHUNKDC_CFLAGS@ @CLDC_CFLAGS@ sbin_PROGRAMS = tabled tdbadm tabled_SOURCES = tabled.h \ - bucket.c object.c server.c storage.c storparse.c cldu.c \ - config.c util.c + bucket.c object.c server.c storage.c storparse.c replica.c \ + cldu.c config.c util.c tabled_LDADD = ../lib/libhttputil.a ../lib/libtdb.a \ @CHUNKDC_LIBS@ @CLDC_LIBS@ @PCRE_LIBS@ @GLIB_LIBS@ \ @CRYPTO_LIBS@ @DB4_LIBS@ @EVENT_LIBS@ @ARGP_LIBS@ @SSL_LIBS@
server/replica.c new file +753 -0
753 lines not on this page · raw message
server/server.c +24 -1
@@ -1318,6 +1318,20 @@ static void tdb_checkpoint(int fd, short events, void *userdata)add_chkpt_timer (); } +static void add_kscan_timer (void ) +{ +static const struct timeval tv = {TABLED_RESCAN_SEC ,0 }; + +if (evtimer_add (&tabled_srv.kscan_timer, &tv) <0 ) +applog (LOG_WARNING ,"unable to add key scan timer" ); +} + +static void tdb_keyscan (int fd,short events,void *userdata) +{ +rep_scan (); +add_kscan_timer (); +} +static void tdb_state_cb (enum db_event event) {
@@ -1365,7 +1379,7 @@ static void tdb_state_cb(enum db_event event)* * We don't even bother with registering this callback, just call it by name. * -* The return value is used to re-arm rescan mechanism. +* The return value is used to re-arm storage rescan mechanism. */ int stor_update_cb (void ) {
@@ -1383,8 +1397,15 @@ int stor_update_cb(void)applog (LOG_DEBUG ," NID %u is up" , stn->id); num_up++; stn->up = true; + stn->last_up =time (NULL ); + }else { +if (stn->last_up !=0 && +time (NULL ) >= stn->last_up +CHUNK_REBOOT_TIME ) { +applog (LOG_INFO ," NID %u went down" , stn->id); + } } } +if (num_up <1 ) {applog (LOG_INFO ,"No active storage node(s), waiting" );return num_up;
@@ -1695,6 +1716,7 @@ static void tdb_state_process(enum st_tdb new_state)return ; }add_chkpt_timer (); +add_kscan_timer ();net_listen (); } }
@@ -1772,6 +1794,7 @@ int main (int argc, char *argv[])event_init ();evtimer_set (&tabled_srv.chkpt_timer, tdb_checkpoint,NULL ); +evtimer_set (&tabled_srv.kscan_timer, tdb_keyscan,NULL );/* set up server networking */ rc =net_open ();
server/tabled.h +10 -2
@@ -19,9 +19,9 @@* */ -#include <stdbool.h> #include <stdlib.h> +#include <time.h> #include <netinet/in.h> #include <openssl/md5.h> #include <glib.h>
@@ -43,6 +43,9 @@ enum {TABLED_PGSZ_LOCK =4096 ,TABLED_CHKPT_SEC =60 *5 ,/* secs between db4 chkpt */ +TABLED_RESCAN_SEC =60 *3 +7 ,/* secs btw key rescans */ + +CHUNK_REBOOT_TIME =3 *60 ,/* secs to declare chunk dead */ CLI_REQ_BUF_SZ =8192 ,/* buffer for req + hdrs */ CLI_DATA_BUF_SZ = 8192,
@@ -89,6 +92,7 @@ struct storage_node {struct list_head all_link;uint32_t id;bool up; +time_t last_up;unsigned alen;int addr_af;
@@ -115,7 +119,7 @@ struct open_chunk {struct st_client *stc;struct storage_node *node;struct list_head link; -struct client *cli; +void *cli;/* usually struct client * */ uint64_t wtogo;uint64_t wkey;
@@ -241,6 +245,7 @@ struct server {enum st_net state_net;struct event chkpt_timer;/* db4 checkpoint timer */ +struct event kscan_timer;/* db4 key rescan timer */ struct server_stats stats;/* global statistics */ };
@@ -340,4 +345,7 @@ extern int stor_node_check(struct storage_node *stn);/* storparse.c */ extern void stor_parse (char *fname,const char *text,size_t len); +/* replica.c */ +extern void rep_scan (void ); +#endif /* __TABLED_H__ */
Quoted 11 lines · Pete Zaitcev
This patch adds what amounts to a background process that maintains redundancy for object data. It is far from the complete solution. For one thing, it does not verify checksums. But it's a start.
There's no way to turn this off, by intention. The whole thing must work very reliably, not steal too much from benchmarks (but if it does, it's only honest to take the hit). It is indispensible. However, there's a plan to add useful monitoring of jobs and other state, such as available nodes.
Signed-off-by Pete Zaitcev<[email protected]>
Just so you're not left hanging, regarding this patch, I'm holding off in-depth review. The other patches were easy, quick to apply without much post-3am thinking... ;)
Jeff
Quoted 40 lines · Pete Zaitcev
server/replica.c new file +40
-0+void rep_scan (void ) +{ +struct cursor cur; +struct db_obj_ent *obj; +unsigned long kcnt; +time_t start_time, t; + +rep_retire (); + + start_time =time (NULL ); +if (debugging) +applog (LOG_DEBUG ,"key scan start time %lu" , (long )start_time); + +memset (&cur,0 ,sizeof (struct cursor ));/* enough to construct */ + cur.db_env = tdb.env; + cur.db_objs = tdb.objs; + + kcnt =0 ; +for (;;) { +if ((t =time (NULL ))>= start_time +2 ) { +if (debugging) +applog (LOG_DEBUG , +"db release at keys %lu seconds %lu" , + kcnt, (long )t); +rep_scan_close (&cur); + } + +if (rep_scan_get (&cur,&obj) !=0 ) +break ; + +/* not needed for db4 with DB_NEXT, but eases our logic */ +if (rep_scan_parse (&cur, obj) !=0 ) { +free (obj); +continue ; + } + +if (!GUINT32_FROM_LE (obj->flags)&DB_OBJ_INLINE ) +rep_scan_verify (&cur, obj); + +free (obj);
server/replica.c line 751
Quoted 10 lines
server/replica.c new file +10
-0+ kcnt++; + } + +rep_scan_close (&cur); +free (cur.key); + cur.key =NULL ; + +if (debugging) +applog (LOG_DEBUG ,"key scan done keys %lu" , kcnt); +return ;
Major comments:
1) What is the point of db->del() in rep_add_nid() ? You are in the middle of a transaction, and you immediately overwrite that record in the same transaction. That is clearly unnecessary work, when db->put() will simply overwrite an existing record, if requested.
2) rep_scan(): I would rather not make the entire daemon non-responsive for multiple seconds.
The database is already in multi-threaded mode, and db4 is free-threaded, so it would seem to make a lot more sense to simply g_thread_create() a thread to do this work.
Also, there should be no need to scan a chunkd's keys at all, as long as the chunkd instance is still communicating with us.
Jeff
Quoted 4 lines · Jeff Garzik
1) What is the point of
db->del()inrep_add_nid()? You are in the middle of a transaction, and you immediately overwrite that record in the same transaction. That is clearly unnecessary work, whendb->put()will simply overwrite an existing record, if requested.
That didn't work last I tested, but I forgot what the issue was. The code is copied from object.c. I'll review it.
Quoted 6 lines
2)
rep_scan(): I would rather not make the entire daemon non-responsive for multiple seconds.The database is already in multi-threaded mode, and db4 is free-threaded, so it would seem to make a lot more sense to simply
g_thread_create()a thread to do this work.
True, but multi-threading in tabled is a much bigger undertaking than just "simply" creating a thread. There's a lot of common state. I think the 2s problem is less urgent than processing massive replication more efficiently (from a Chunk down).
Quoted 2 lines
Also, there should be no need to scan a chunkd's keys at all, as long as the chunkd instance is still communicating with us.
No, there's a need, although there's no code for it: the scan should probe keys by requesting their metadata, which forturously includes checksums. This is how the results of Chunk's self-test will be communicated to tabled.
-- Pete
Quoted 9 lines · Pete Zaitcev
Quoted 4 lines · Jeff Garzik
1) What is the point of
db->del()inrep_add_nid()? You are in the middle of a transaction, and you immediately overwrite that record in the same transaction. That is clearly unnecessary work, whendb->put()will simply overwrite an existing record, if requested.That didn't work last I tested, but I forgot what the issue was. The code is copied from
object.c. I'll review it.
The only code matching that pattern is object_put_end(), which also performs unnecessary work of needlessly calling db->del().
This should be recognized as nothing more than programmer laziness, enabled by the sharing of __object_del() between DEL and PUT, and the necessity of deleting associated records (ACLs) in conjunction with the main [deletion | update].
That does not apply to rep_add_nid(), which has neither object data nor related records to delete.
(patches accepted to remove ->del from PUT path, too, if you wish)
Quoted 11 lines
Quoted 6 lines
2)
rep_scan(): I would rather not make the entire daemon non-responsive for multiple seconds.The database is already in multi-threaded mode, and db4 is free-threaded, so it would seem to make a lot more sense to simply
g_thread_create()a thread to do this work.True, but multi-threading in tabled is a much bigger undertaking than just "simply" creating a thread. There's a lot of common state. I think the 2s problem is less urgent than processing massive replication more efficiently (from a Chunk down).
I only see one global variable reference in the whole nicely-designed, nicely self-contained replica.c file. "a lot of common state" seems like quite an exaggeration.
This patch resuscitates the worst application model from the late 1980's, cooperative multi-tasking. I do not want to walk down that road: such programs have weird pauses at weird times, and behave in a very non-deterministic manner.
Furthermore, I think you will find that threading gives you a lot more freedom to work in the background.
Quoted 7 lines
Quoted 2 lines
Also, there should be no need to scan a chunkd's keys at all, as long as the chunkd instance is still communicating with us.
No, there's a need, although there's no code for it: the scan should probe keys by requesting their metadata, which forturously includes checksums. This is how the results of Chunk's self-test will be communicated to tabled.
What is the need? This description provides "what" and "how" but does not answer "why?"
Jeff