chunkd: add self-checking
PATCH 4/4 chunkd: add self-checking
This patch adds a background process that periodically verifies the integrity of all known objects. Objects that are found found faulty are made invisible to applications. This way, only known good objects remain visible and applications may implement integrity checking that is run asynchronously with the self-checking. The intent is to let the performance of checking to scale with the number of objects and amount of data stored in the cell.
--- server/Makefile.am | 2 server/be-fs.c | 126 +++++++++++++++- server/chunkd.h | 8 + server/config.c | 28 +++ server/selfcheck.c | 208 ++++++++++++++++++++++++++ server/server.c | 2 test/.gitignore | 1 test/Makefile.am | 4 test/selfcheck-unit.c | 313 ++++++++++++++++++++++++++++++++++++++++ test/server-test.cfg | 2 test/test.h | 2 11 files changed, 693 insertions(+), 3 deletions(-)
commit ded36ccc14d7356ceaa523c43e6d91efc06c51cd
Author: Master <[email protected]> Date: Sun Dec 27 16:36:17 2009 -0700
Selfcheck (with a test).
server/Makefile.am +1 -1
@@ -6,7 +6,7 @@ sbin_PROGRAMS = chunkd chunkd_SOURCES = chunkd.h \ ../lib/chunksrv.c \ - be-fs.c object.c server.c config.c cldu.c util.c \ + be-fs.c object.c server.c selfcheck.c config.c cldu.c util.c \ objcache.c chunkd_LDADD = \ @CLDC_LIBS@ @GLIB_LIBS@ @CRYPTO_LIBS@ \
server/be-fs.c +125 -1
148 lines not on this page · raw message
server/chunkd.h +8 -0
@@ -210,6 +210,7 @@ struct server {char *cell;uint32_t nid;struct geo loc; +time_t chk_period;TCHDB *tbl_master;struct objcache actives;
@@ -217,6 +218,8 @@ struct server {struct server_stats stats;/* global statistics */ }; +#define MDB_TABLE_ID "__chunkd_table_id" +extern struct hail_log cldu_hail_log;/* be-fs.c */
@@ -245,6 +248,7 @@ extern bool fs_obj_write_commit(struct backend_obj *bo, const char *user,extern bool fs_obj_delete (uint32_t table_id,const char *user,const void *kbuf,size_t klen,enum chunk_errcode *err_code); +extern int fs_obj_disable (const char *fn);extern ssize_t fs_obj_sendfile (struct backend_obj *bo,int out_fd,size_t len);extern int fs_list_objs_open (struct fs_obj_lister *t,const char *root_path,uint32_t table_id);
@@ -257,6 +261,7 @@ extern GList *fs_list_objs(uint32_t table_id, const char *user);extern bool fs_table_open (const char *user,const void *kbuf,size_t klen,bool tbl_creat,bool excl_creat,uint32_t *table_id,enum chunk_errcode *err_code); +extern int fs_obj_do_sum (const char *fn,unsigned int klen,char **csump);/* object.c */ extern bool object_del (struct client *cli);
@@ -320,6 +325,9 @@ extern void resp_init_req(struct chunksrv_resp *resp,/* config.c */ extern void read_config (void ); +/* selfcheck.c */ +extern void chk_init (void ); +static inline bool use_sendfile (struct client *cli) {#if defined(HAVE_SENDFILE ) && defined(HAVE_SYS_SENDFILE_H )
server/config.c +28 -0
@@ -442,6 +442,24 @@ static void cfg_elm_end (GMarkupParseContext *context, cc->text =NULL ; } +else if (!strcmp (element_name,"SelfCheckPeriod" )) { +if (!cc->text) { +applog (LOG_WARNING ,"SelfCheckPeriod element empty" ); +return ; + } + n =strtol (cc->text,NULL ,10 ); +if (n <0 || n >=LONG_MAX ) { +applog (LOG_ERR ,"SelfCheckPeriod '%s' is invalid" , + cc->text); +free (cc->text); + cc->text =NULL ; +return ; + } + chunkd_srv.chk_period = n; +free (cc->text); + cc->text =NULL ; + } +else {applog (LOG_WARNING ,"Unknown element \"%s\"" , element_name); }
@@ -527,6 +545,16 @@ void read_config(void)#endif } +/* +* The check period determines how quickly the self-check notices +* bad keys when there are few keys or they are small (otherwise the +* time to rescan determines the observation latency). To save +* power, the default is several hours. Notice that it's randomized. +* Zero means no self-check. +*/ +if (!chunkd_srv.chk_period) + chunkd_srv.chk_period =3 *60 *60 ; +free (ctx.geo_area);free (ctx.geo_zone);free (ctx.geo_rack);
server/selfcheck.c new file +208 -0
208 lines not on this page · raw message
server/server.c +2 -0
@@ -1678,6 +1678,8 @@ int main (int argc, char *argv[]) goto err_out_cld; } +chk_init (); +/* set up server networking */ for (tmpl = chunkd_srv.listeners; tmpl; tmpl = tmpl->next) { rc =net_open (tmpl->data);
test/.gitignore +1 -0
@@ -5,6 +5,7 @@ large-object lotsa-objects nop objcache-unit +selfcheck-unit .libs libtest.a
test/Makefile.am +3 -1
@@ -26,11 +26,12 @@ TESTS = \ auth \ large-object \ lotsa-objects \ + selfcheck-unit \ stop-daemon \ clean-db check_PROGRAMS = auth basic-object it-works large-object \ - lotsa-objects nop objcache-unit + lotsa-objects nop objcache-unit selfcheck-unit TESTLDADD = ../lib/libchunkdc.la \ libtest.a \
@@ -42,6 +43,7 @@ it_works_LDADD = $(TESTLDADD) large_object_LDADD = $(TESTLDADD) lotsa_objects_LDADD = $(TESTLDADD) nop_LDADD = $(TESTLDADD) +selfcheck_unit_LDADD = $(TESTLDADD) objcache_unit_LDADD = @GLIB_LIBS@
test/selfcheck-unit.c new file +313 -0
313 lines not on this page · raw message
test/server-test.cfg +2 -0
@@ -27,3 +27,5 @@ <Host>localhost</Host> </CLD> +<SelfCheckPeriod>35</SelfCheckPeriod> +
test/test.h +2 -0
@@ -38,6 +38,8 @@#define TEST_PORTFILE "chunkd.port"#define TEST_PORTFILE_SSL "chunkd-ssl.port" +#define TEST_CHUNKD_CFG "server-test.cfg" +#define OK (expr) \ do { \ if (!(expr)) { \
Quoted 10 lines · Pete Zaitcev
This patch adds a background process that periodically verifies the integrity of all known objects. Objects that are found found faulty are made invisible to applications. This way, only known good objects remain visible and applications may implement integrity checking that is run asynchronously with the self-checking. The intent is to let the performance of checking to scale with the number of objects and amount of data stored in the cell.
Signed-off-by Pete Zaitcev<[email protected]>
Quoted 13 lines
--- server/Makefile.am | 2 server/be-fs.c | 126 +++++++++++++++- server/chunkd.h | 8 + server/config.c | 28 +++ server/selfcheck.c | 208 ++++++++++++++++++++++++++ server/server.c | 2 test/.gitignore | 1 test/Makefile.am | 4 test/selfcheck-unit.c | 313 ++++++++++++++++++++++++++++++++++++++++ test/server-test.cfg | 2 test/test.h | 2 11 files changed, 693 insertions(+), 3 deletions(-)
OK. The objcache (patches 3 & 5) look good. I was thinking that the "table of open files" was better suited for server/be-fs.c, thus keeping the backend / frontend separation a bit more distinct. However, this is a matter of taste, and I can also see advantages in how you implemented it. So, no changes necessary, we'll how that works out.
The reason why I think about backend / frontend separation, when we only have one storage backend, "the filesystem", is that I see a lot of merit in a second type of backend: single-file (aka raw block device mode). I don't want to stray too far from being able to implement a second type of storage backend. On the flip side, of course, I don't want imaginary future ideas to constraint us in the present.
On to this patch, self-checking. Regarding this patch's implementation, the code appears to be correct. However, I wanted to discuss the design a little bit.
I am definitely concerned that the current implementation does not really scale at all with the datastore. With this change, the admin would have to guess at the self-check period, with not much feedback from the system about how long a scan takes, when a scan begins, or when a scan ends. I also think a
sleep(n)
self_check()
algorithm seems less useful to the average admin than a slightly more complex one that solves the problem defined as "guarantee an object is checked at least every N days." Because, as the dataset grows beyond a test database measures in megabytes, the dataset scan time will dwarf the self-check sleep period. The behavior becomes one of constant scanning, with a tiny period of rest in between.
It seems like either (a) tracking the total dataset size and total dataset scan time, or (b) tracking and modifying per-object last-self-check times, would be needed.
Also of relevance to admins is scan start time. This patch would have a scan begin at essentially random times throughout the day or night, in particular occurring during the most heavily-trafficked portion of the day. An algorithm that occurs on hour N every day (or hour N, day M of each week) is much more predictable, regardless of dataset size. That would be pretty easy:
while (1)
calc time until next desired interval
(example: day 0, hour 5 (every sunday at 5am))
timer_init()
timer_add(scan_timer)
scan_timer()
if (scan thread exists)
log ("still scanning")
return
start scan thread
scan_thread()
dbscan()
end thread (do not loop)
Another option is to add an administrative command "START SCAN", and permit an external utility, scheduled by cron or executed by "make check", to be the entity that starts the scan thread in the background.
That would permit maximum flexibility for both the admin and our
testsuite.
Comments on all this? I will happily apply the objcache stuff once all the self-checking details are settled.
Jeff
Quoted 9 lines · On Tue, 12 Jan 2010 09:21:24 -0500 Jeff Garzik <[email protected]> wrote:
sleep(n)
self_check()algorithm seems less useful to the average admin than a slightly more complex one that solves the problem defined as "guarantee an object is checked at least every N days." Because, as the dataset grows beyond a test database measures in megabytes, the dataset scan time will dwarf the self-check sleep period. The behavior becomes one of constant scanning, with a tiny period of rest in between.
That's obvious. You also forgot to recall that your "fat" node exhacerbates the problem, but conversely, splitting them up helps. The single-threaded design is on purpose: it provides a crude method of load control. But that aside, two things about your scheme:
- how do you select N? It is no different from n in its arbitrariness.
- "as the data set grows beyond a test database", it is going to take more work to satisfy N.
What you are proposing is actually no different. It is not more adaptive.
Quoted 3 lines
It seems like either (a) tracking the total dataset size and total dataset scan time, or (b) tracking and modifying per-object last-self-check times, would be needed.
Well, that would be nice.
Still, we are powerless to keep the scan time down when the dataset grows. At best, we can sacrificy the mean detection time and constrain the power that the scan consumes.
I'm thinking about doing just that actually. But first I'm going to implement a reporting scheme in tabled.
Quoted 5 lines
Also of relevance to admins is scan start time. This patch would have a scan begin at essentially random times throughout the day or night, in particular occurring during the most heavily-trafficked portion of the day. An algorithm that occurs on hour N every day (or hour N, day M of each week) is much more predictable, regardless of dataset size.
So, you want the mean time to detect to be at least 12 hours. I can do that if you insist.
Quoted 5 lines
Another option is to add an administrative command "START SCAN", and permit an external utility, scheduled by cron or executed by "make check", to be the entity that starts the scan thread in the background.
That would permit maximum flexibility for both the admin and our
testsuite.
We can do that too, I guess. Just don't know when.
-- Pete