ulogd2 db rework
cover ulogd2 db rework
Hello,
Here's a patchset for ulogd which is aiming at improving the way databases are handled. It features two different parts:
- a backlog system (patch 06) which allow queries to be store in memory when the database is not available. Data loss is prevented in this case (in the limit of the defined memcap)
- a multithreaded ulogd db output (patch 09) with one thread getting messages from kernel and an other one dedicated to the SQL queries execution.
The backlog system prevent data loss and only cost memory when the database is down. The multithreaded db output uses a ring buffer (which cost memory) for communication between the two threads.
The idea behind the multithreaded db output is to almost fix the time needed to read a kernel message. Doing this, ulogd should be more resistant to burst of kernel messages as kernel-user buffer will not be filled due to the high treatment time needed to execute SQL query.
Both features are modifying the generic db system inside ulogd. So the modification is available in pgsql, mysql and dbi output. It has for now only been tested on pgsql.
Comments, feedbacks and test results are more than welcome on both these features.
Patchset statistics:
configure.ac | 32 ++- include/ulogd/db.h | 67 +++++- output/mysql/ulogd_output_MYSQL.c | 4 +- output/pgsql/ulogd_output_PGSQL.c | 4 +- output/sqlite3/ulogd_output_SQLITE3.c | 2 + src/Makefile.am | 2 +- src/ulogd.c | 2 +- ulogd.conf.in | 13 ++ util/db.c | 371 +++++++++++++++++++++++++++------ 9 files changed, 417 insertions(+), 80 deletions(-)
BR,
--
Eric Leblond <[email protected]>
PATCH 1/11 ulogd: display stack during configuration
--- src/ulogd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
src/ulogd.c +1 -1
@@ -830,7 +830,7 @@ static int create_stack(const char *option) }INIT_LLIST_HEAD (&stack->list); -ulogd_log (ULOGD_DEBUG ,"building new pluginstance stack (%s):\n" , +ulogd_log (ULOGD_NOTICE ,"building new pluginstance stack: '%s'\n" , option);/* PASS 1: find and instanciate plugins of stack, link them together */
Signature · 1 line
1.7.10.4
PATCH 2/11 Fix automagic support of dbi, pcap and sqlite3
In reply to Eric LeblondFrom: Ilya Tumaykin <[email protected]>
ulogd has automagic deps for several output plugins right now, namely dbi, pcap and sqlite3. These plugins are built if the appropriate libs are present on user's system. While this situation is fine with binary distros it is not OK on source-based ones such as Gentoo.
The problem arises when such a program links against libs without user's request and libs are later removed from system which leaves program in a broken state.
This patch is modifying configure.ac which we apply in our package and which fixes mentioned issue. It adds 3 new configure options: -- without-{dbi,pcap.sqlite}. I would like to emphasize that this patch doesn't change default behaviour of configure script at all, so all other distros won't suffer. We simply add options to explicitly disable any attempts to try and detect libs for automagic deps, which is enough to avoid unnecessary linkage.
--- configure.ac | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-)
configure.ac +20 -10
@@ -20,14 +20,6 @@ AC_PROG_LIBTOOL dnl Checks for libraries. AC_SEARCH_LIBS([dlopen], [dl], [libdl_LIBS="$LIBS"; LIBS=""]) AC_SUBST([libdl_LIBS]) -AC_SEARCH_LIBS([pcap_close], [pcap], [libpcap_LIBS="-lpcap"; LIBS=""]) -AC_SUBST([libpcap_LIBS]) -AM_CONDITIONAL([HAVE_PCAP], [test -n "$libpcap_LIBS"]) -if test "x$libpcap_LIBS" != "x"; then - enable_pcap="yes" -else - enable_pcap="no" -fi dnl Checks for header files. AC_HEADER_DIRENT
@@ -88,7 +80,10 @@ else enable_mysql="no" fi -PKG_CHECK_MODULES([libsqlite3], [sqlite3], [], [:]) +AC_ARG_WITH([sqlite], AS_HELP_STRING([--without-sqlite], [Build without SQLITE3 output plugin [default=test]])) +AS_IF([test "x$with_sqlite" != "xno"], [ + PKG_CHECK_MODULES([libsqlite3], [sqlite3], [], [:]) +]) AM_CONDITIONAL([HAVE_SQLITE3], [test -n "$libsqlite3_LIBS"]) if test "x$libsqlite3_LIBS" != "x"; then enable_sqlite3="yes"
@@ -96,7 +91,10 @@ else enable_sqlite3="no" fi -CT_CHECK_DBI() +AC_ARG_WITH([dbi], AS_HELP_STRING([--without-dbi], [Build without DBI output plugin [default=test]])) +AS_IF([test "x$with_dbi" != "xno"], [ + CT_CHECK_DBI() +]) AM_CONDITIONAL(HAVE_DBI, test "x$DBI_LIB" != "x") if test "x$DBI_LIB" != "x"; then enable_dbi="yes"
@@ -104,6 +102,18 @@ else enable_dbi="no" fi +AC_ARG_WITH([pcap], AS_HELP_STRING([--without-pcap], [Build without PCAP output plugin [default=test]])) +AS_IF([test "x$with_pcap" != "xno"], [ + AC_SEARCH_LIBS([pcap_close], [pcap], [libpcap_LIBS="-lpcap"; LIBS=""]) + AC_SUBST([libpcap_LIBS]) +]) +AM_CONDITIONAL([HAVE_PCAP], [test -n "$libpcap_LIBS"]) +if test "x$libpcap_LIBS" != "x"; then + enable_pcap="yes" +else + enable_pcap="no" +fi + dnl AC_SUBST(DATABASE_DIR) dnl AC_SUBST(DATABASE_LIB) dnl AC_SUBST(DATABASE_LIB_DIR)
Signature · 1 line
1.7.10.4
PATCH 3/11 postgresql: add sanity checking
In reply to Eric LeblondClean postgresql handler at deinit.
--- output/pgsql/ulogd_output_PGSQL.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
output/pgsql/ulogd_output_PGSQL.c +3 -1
@@ -214,7 +214,9 @@ static int close_db_pgsql(struct ulogd_pluginstance *upi) {struct pgsql_instance *pi = (struct pgsql_instance *) upi->private; -PQfinish (pi->dbh); +if (pi->dbh) +PQfinish (pi->dbh); + pi->dbh =NULL ;return 0 ; }
Signature · 1 line
1.7.10.4
PATCH 4/11 mysql: add sanity checking
In reply to Eric LeblondNullify mysql handler at deinit.
--- output/mysql/ulogd_output_MYSQL.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
output/mysql/ulogd_output_MYSQL.c +3 -1
@@ -162,7 +162,9 @@ static int get_columns_mysql(struct ulogd_pluginstance *upi)static int close_db_mysql (struct ulogd_pluginstance *upi) {struct mysql_instance *mi = (struct mysql_instance *) upi->private; -mysql_close (mi->dbh); +if (mi->dbh) +mysql_close (mi->dbh); + mi->dbh =NULL ;return 0 ; }
Signature · 1 line
1.7.10.4
PATCH 5/11 sqlite3: add sanity checking
In reply to Eric LeblondNullify sqlite3 handler at deinit.
--- output/sqlite3/ulogd_output_SQLITE3.c | 2 ++ 1 file changed, 2 insertions(+)
output/sqlite3/ulogd_output_SQLITE3.c +2 -0
@@ -431,6 +431,8 @@ sqlite3_stop(struct ulogd_pluginstance *pi)sqlite3_close (priv->dbh); + priv->dbh =NULL ; +return 0 ; }
Signature · 1 line
1.7.10.4
PATCH 6/11 db: store data in memory during database downtime
In reply to Eric LeblondThis patch is adding a mechanism to store query in a backlog build in memory. This allow to store events during downtime in memory and realize the effective insertion when the database comes back. A memory cap is used to avoid any memory flooding.
--- include/ulogd/db.h | 34 +++++++++-- ulogd.conf.in | 9 +++ util/db.c | 170 +++++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 180 insertions(+), 33 deletions(-)
include/ulogd/db.h +29 -5
@@ -20,6 +20,12 @@ struct db_driver { const char *stmt, unsigned int len); }; +struct db_stmt { +char *stmt; +int len; +struct llist_head list; +}; +struct db_instance {char *stmt;/* buffer for our insert statement */ char *stmt_val;/* pointer to the beginning of the "VALUES" part */
@@ -28,9 +34,15 @@ struct db_instance {time_t reconnect;int (*interp)(struct ulogd_pluginstance *upi);struct db_driver *driver; +unsigned int backlog_memcap; +unsigned int backlog_memusage; +unsigned int backlog_oneshot; +unsigned char backlog_full; +struct llist_head backlog; };#define TIME_ERR ((time_t)-1)/* Be paranoid */ #define RECONNECT_DEFAULT 2 +#define MAX_ONESHOT_REQUEST 10#define DB_CES \ { \
@@ -51,13 +63,25 @@ struct db_instance { .key = "procedure", \ .type = CONFIG_TYPE_STRING, \ .options = CONFIG_OPT_MANDATORY, \ + }, \ + { \ + .key = "backlog_memcap", \ + .type = CONFIG_TYPE_INT, \ + .u.value = 0, \ + }, \ + { \ + .key = "backlog_oneshot_requests", \ + .type = CONFIG_TYPE_INT, \ + .u.value = MAX_ONESHOT_REQUEST, \ } -#define DB_CE_NUM 4 -#define table_ce (x) (x->ces[0]) -#define reconnect_ce (x) (x->ces[1]) -#define timeout_ce (x) (x->ces[2]) -#define procedure_ce (x) (x->ces[3]) +#define DB_CE_NUM 6 +#define table_ce (x) (x->ces[0]) +#define reconnect_ce (x) (x->ces[1]) +#define timeout_ce (x) (x->ces[2]) +#define procedure_ce (x) (x->ces[3]) +#define backlog_memcap_ce (x) (x->ces[4]) +#define backlog_oneshot_ce (x) (x->ces[5])void ulogd_db_signal (struct ulogd_pluginstance *upi,int signal);int ulogd_db_start (struct ulogd_pluginstance *upi);
ulogd.conf.in +9 -0
@@ -207,6 +207,13 @@ user="nupik" table="ulog" pass="changeme" procedure="INSERT_PACKET_FULL" +# backlog configuration: +# set backlog_memcap to the size of memory that will be +# allocated to store events in memory if data is temporary down +# and insert them when the database came back. +#backlog_memcap=1000000 +# number of events to insert at once when backlog is not empty +#backlog_oneshot_requests=10 [mysql2] db="nulog"
@@ -224,6 +231,8 @@ table="ulog" #schema="public" pass="changeme" procedure="INSERT_PACKET_FULL" +#backlog_memcap=1000000 +#backlog_oneshot_requests=10 [pgsql2] db="nulog"
util/db.c +142 -28
205 lines not on this page · raw message
Signature · 1 line
1.7.10.4
PATCH 7/11 db: suppress field in db structure
In reply to Eric LeblondThe field is currently only used in a single function as a string pointer and can thus be removed from the db instance structure.
--- include/ulogd/db.h | 2 +- util/db.c | 36 ++++++++++++++++++------------------ 2 files changed, 19 insertions(+), 19 deletions(-)
include/ulogd/db.h +1 -1
@@ -1,6 +1,7 @@/* DB handling functions * * (C) 2000-2005 by Harald Welte <[email protected]> +* (C) 2013 by Eric Leblond <[email protected]> * * This code is distributed under the terms of GNU GPL version 2 */
@@ -29,7 +30,6 @@ struct db_stmt {struct db_instance {char *stmt;/* buffer for our insert statement */ char *stmt_val;/* pointer to the beginning of the "VALUES" part */ -char *stmt_ins;/* pointer to current inser position in statement */ char *schema;time_t reconnect;int (*interp)(struct ulogd_pluginstance *upi);
util/db.c +18 -18
@@ -266,7 +266,7 @@ static void __format_query_db(struct ulogd_pluginstance *upi)unsigned int i; - di->stmt_ins = di->stmt_val; +char * stmt_ins = di->stmt_val;for (i =0 ; i < upi->input.num_keys; i++) {struct ulogd_key *res = upi->input.keys[i].u.source;
@@ -280,52 +280,52 @@ static void __format_query_db(struct ulogd_pluginstance *upi)if (!res || !IS_VALID (*res)) {/* no result, we have to fake something */ - di->stmt_ins +=sprintf (di->stmt_ins,"NULL," ); + stmt_ins +=sprintf (stmt_ins,"NULL," );continue ; }switch (res->type) {case ULOGD_RET_INT8 : -sprintf (di->stmt_ins,"%d," , res->u.value.i8); +sprintf (stmt_ins,"%d," , res->u.value.i8);break ;case ULOGD_RET_INT16 : -sprintf (di->stmt_ins,"%d," , res->u.value.i16); +sprintf (stmt_ins,"%d," , res->u.value.i16);break ;case ULOGD_RET_INT32 : -sprintf (di->stmt_ins,"%d," , res->u.value.i32); +sprintf (stmt_ins,"%d," , res->u.value.i32);break ;case ULOGD_RET_INT64 : -sprintf (di->stmt_ins,"%" PRId64"," , res->u.value.i64); +sprintf (stmt_ins,"%" PRId64"," , res->u.value.i64);break ;case ULOGD_RET_UINT8 : -sprintf (di->stmt_ins,"%u," , res->u.value.ui8); +sprintf (stmt_ins,"%u," , res->u.value.ui8);break ;case ULOGD_RET_UINT16 : -sprintf (di->stmt_ins,"%u," , res->u.value.ui16); +sprintf (stmt_ins,"%u," , res->u.value.ui16);break ;case ULOGD_RET_IPADDR :/* fallthrough when logging IP as u_int32_t */ case ULOGD_RET_UINT32 : -sprintf (di->stmt_ins,"%u," , res->u.value.ui32); +sprintf (stmt_ins,"%u," , res->u.value.ui32);break ;case ULOGD_RET_UINT64 : -sprintf (di->stmt_ins,"%" PRIu64"," , res->u.value.ui64); +sprintf (stmt_ins,"%" PRIu64"," , res->u.value.ui64);break ;case ULOGD_RET_BOOL : -sprintf (di->stmt_ins,"'%d'," , res->u.value.b); +sprintf (stmt_ins,"'%d'," , res->u.value.b);break ;case ULOGD_RET_STRING : - *(di->stmt_ins++) ='\'' ; + *(stmt_ins++) ='\'' ;if (res->u.value.ptr) { - di->stmt_ins += - di->driver->escape_string (upi, di->stmt_ins, + stmt_ins += + di->driver->escape_string (upi, stmt_ins, res->u.value.ptr,strlen (res->u.value.ptr)); } -sprintf (di->stmt_ins,"'," ); +sprintf (stmt_ins,"'," );break ;case ULOGD_RET_RAWSTR : -sprintf (di->stmt_ins,"%s," , (char *) res->u.value.ptr); +sprintf (stmt_ins,"%s," , (char *) res->u.value.ptr);break ;case ULOGD_RET_RAW : ulogd_log(ULOGD_NOTICE,
@@ -336,9 +336,9 @@ static void __format_query_db(struct ulogd_pluginstance *upi) res->type, upi->input.keys[i].name);break ; } - di->stmt_ins = di->stmt +strlen (di->stmt); + stmt_ins = di->stmt +strlen (di->stmt); } - *(di->stmt_ins -1 ) =')' ; + *(stmt_ins -1 ) =')' ; }static int __add_to_backlog (struct ulogd_pluginstance *upi,const char *stmt,unsigned int len)
Signature · 1 line
1.7.10.4
PATCH 8/11 db: use offset instead of direct pointer.
In reply to Eric LeblondUse an offset approach to get the start of values printing area. It is more generic and will be use soon.
--- include/ulogd/db.h | 2 +- util/db.c | 16 +++++++++------- 2 files changed, 10 insertions(+), 8 deletions(-)
include/ulogd/db.h +1 -1
@@ -29,7 +29,7 @@ struct db_stmt {struct db_instance {char *stmt;/* buffer for our insert statement */ -char *stmt_val;/* pointer to the beginning of the "VALUES" part */ +int stmt_offset;/* offset to the beginning of the "VALUES" part */ char *schema;time_t reconnect;int (*interp)(struct ulogd_pluginstance *upi);
util/db.c +9 -7
@@ -66,6 +66,7 @@ static int sql_createstmt(struct ulogd_pluginstance *upi)unsigned int i;char *table =table_ce (upi->config_kset).u.string;char *procedure =procedure_ce (upi->config_kset).u.string; +char *stmt_val =NULL ;if (mi->stmt)free (mi->stmt);
@@ -106,7 +107,7 @@ static int sql_createstmt(struct ulogd_pluginstance *upi)else sprintf (mi->stmt,"%s (" , procedure); - mi->stmt_val = mi->stmt +strlen (mi->stmt); + stmt_val = mi->stmt +strlen (mi->stmt);for (i =0 ; i < upi->input.num_keys; i++) {if (upi->input.keys[i].flags &ULOGD_KEYF_INACTIVE )
@@ -115,19 +116,20 @@ static int sql_createstmt(struct ulogd_pluginstance *upi)strncpy (buf, upi->input.keys[i].name,ULOGD_MAX_KEYLEN );while ((underscore =strchr (buf,'.' ))) *underscore ='_' ; -sprintf (mi->stmt_val,"%s," , buf); - mi->stmt_val = mi->stmt +strlen (mi->stmt); +sprintf (stmt_val,"%s," , buf); + stmt_val = mi->stmt +strlen (mi->stmt); } - *(mi->stmt_val -1 ) =')' ; + *(stmt_val -1 ) =')' ; -sprintf (mi->stmt_val," values (" ); +sprintf (stmt_val," values (" ); }else if (strncasecmp (procedure,"CALL" ,strlen ("CALL" )) ==0 ) {sprintf (mi->stmt,"CALL %s(" , procedure); }else {sprintf (mi->stmt,"SELECT %s(" , procedure); } - mi->stmt_val = mi->stmt +strlen (mi->stmt); + + mi->stmt_offset =strlen (mi->stmt);ulogd_log (ULOGD_DEBUG ,"stmt='%s'\n" , mi->stmt);
@@ -266,7 +268,7 @@ static void __format_query_db(struct ulogd_pluginstance *upi)unsigned int i; -char * stmt_ins = di->stmt_val; +char * stmt_ins = di->stmt + di->stmt_offset;for (i =0 ; i < upi->input.num_keys; i++) {struct ulogd_key *res = upi->input.keys[i].u.source;
Signature · 1 line
1.7.10.4
PATCH 9/11 db: add ring buffer for DB query
In reply to Eric LeblondThis patch adds an optional ring buffer option which modify the way database queries are made. The main thread is only handling kernel message reading and query formatting. The SQL request is made in a separate dedicated thread.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side.
--- configure.ac | 2 + include/ulogd/db.h | 31 ++++++++++- src/Makefile.am | 2 +- ulogd.conf.in | 4 ++ util/db.c | 152 ++++++++++++++++++++++++++++++++++++++++++++++++---- 5 files changed, 178 insertions(+), 13 deletions(-)
configure.ac +2 -0
@@ -39,6 +39,8 @@ AC_CHECK_FUNCS(socket strerror) regular_CFLAGS="-Wall -Wextra -Wno-unused-parameter" AC_SUBST([regular_CFLAGS]) +AC_CHECK_LIB(pthread, pthread_create) + dnl Check for the right nfnetlink version PKG_CHECK_MODULES([LIBNFNETLINK], [libnfnetlink >= 1.0.1]) PKG_CHECK_MODULES([LIBMNL], [libmnl >= 1.0.3])
include/ulogd/db.h +30 -1
@@ -21,6 +21,24 @@ struct db_driver { const char *stmt, unsigned int len); }; +enum { +RING_NO_QUERY , +RING_QUERY_READY , +}; + +struct db_stmt_ring { +/* Ring buffer: 1 status byte + string */ +char * ring;/* pointer to the ring */ +uint32_t size;/* size of ring buffer in element */ +int length;/* length of one ring buffer element */ +uint32_t wr_item;/* write item in ring buffer */ +uint32_t rd_item;/* read item in ring buffer */ +char *wr_place; +pthread_cond_t cond; +pthread_mutex_t mutex; +int full; +}; +struct db_stmt {char *stmt;int len;
@@ -34,6 +52,10 @@ struct db_instance {time_t reconnect;int (*interp)(struct ulogd_pluginstance *upi);struct db_driver *driver; +/* DB ring buffer */ +struct db_stmt_ring ring; +pthread_t db_thread_id; +/* Backlog system */ unsigned int backlog_memcap;unsigned int backlog_memusage;unsigned int backlog_oneshot;
@@ -43,6 +65,7 @@ struct db_instance {#define TIME_ERR ((time_t)-1)/* Be paranoid */ #define RECONNECT_DEFAULT 2#define MAX_ONESHOT_REQUEST 10 +#define RING_BUFFER_DEFAULT_SIZE 10#define DB_CES \ { \
@@ -73,15 +96,21 @@ struct db_instance { .key = "backlog_oneshot_requests", \ .type = CONFIG_TYPE_INT, \ .u.value = MAX_ONESHOT_REQUEST, \ + }, \ + { \ + .key = "ring_buffer_size", \ + .type = CONFIG_TYPE_INT, \ + .u.value = RING_BUFFER_DEFAULT_SIZE, \ } -#define DB_CE_NUM 6 +#define DB_CE_NUM 7#define table_ce (x) (x->ces[0])#define reconnect_ce (x) (x->ces[1])#define timeout_ce (x) (x->ces[2])#define procedure_ce (x) (x->ces[3])#define backlog_memcap_ce (x) (x->ces[4])#define backlog_oneshot_ce (x) (x->ces[5]) +#define ringsize_ce (x) (x->ces[6])void ulogd_db_signal (struct ulogd_pluginstance *upi,int signal);int ulogd_db_start (struct ulogd_pluginstance *upi);
src/Makefile.am +1 -1
@@ -8,4 +8,4 @@ sbin_PROGRAMS = ulogd ulogd_SOURCES = ulogd.c select.c timer.c rbtree.c conffile.c hash.c addr.c ulogd_LDADD = ${libdl_LIBS} -ulogd_LDFLAGS = -export-dynamic +ulogd_LDFLAGS = -export-dynamic -lpthread
ulogd.conf.in +4 -0
@@ -233,6 +233,10 @@ pass="changeme" procedure="INSERT_PACKET_FULL" #backlog_memcap=1000000 #backlog_oneshot_requests=10 +# If superior to 1 a thread dedicated to SQL request execution +# is created. The value stores the number of SQL request to keep +# in the ring buffer +#ring_buffer_size=1000 [pgsql2] db="nulog"
util/db.c +141 -11
262 lines not on this page · raw message
Signature · 1 line
1.7.10.4
Pablo Neira Ayuso
maintainerRe: [PATCH 09/11] db: add ring buffer for DB query
Hi Eric,
Quoted 8 lines · Eric Leblond
This patch adds an optional ring buffer option which modify the way database queries are made. The main thread is only handling kernel message reading and query formatting. The SQL request is made in a separate dedicated thread.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side.
Would be feasible to make asynchronous SQL requests instead, so you can skip the use of pthread?
Regards.
Re: [PATCH 09/11] db: add ring buffer for DB query
Hi,
Quoted 12 lines · Pablo Neira Ayuso
Hi Eric,
Quoted 8 lines · Eric Leblond
This patch adds an optional ring buffer option which modify the way database queries are made. The main thread is only handling kernel message reading and query formatting. The SQL request is made in a separate dedicated thread.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side.
Quoted 2 lines
Would be feasible to make asynchronous SQL requests instead, so you can skip the use of pthread?
That is an excellent question :)
Here's a list of points. From the small to big one.
From a performance point of view, ulogd will still be slow down by the request time due to the network communication with the db . I think the effect is really negligible but may have an impact in the case of distant db.
From a lazy point of view, this would require to update all the database backends instead of simply updating the db abstraction layer as it is done with the proposed patch.
Regarding the use of asynchronous request, I've always stop trying using them after reading the doc. So I've got no practical experience here and correct someone correct me if I'm wrong. What I've understood is that there is no "take my query and just do your work" function. Application has to get the result to the query and free them.
For example, let's have a look at postgresql case. The doc on asynchronous API is here:
http://www.postgresql.org/docs/6.4/static/libpq-chapter17044.htm
Here's an interesting part of the documentation:
PQsendQuery: Submit a query to Postgres without waiting for the result(s). TRUE is returned if the query was successfully dispatched, FALSE if not (in which case, use PQerrorMessage to get more information about the failure).
After successfully calling PQsendQuery, call PQgetResult one or more times to obtain the query results. PQsendQuery may not be called again (on the same connection) until PQgetResult has returned NULL, indicating that the query is done.
On other interesting point regarding PQgetResult:
Don't forget to free each result object with PQclear when done with it.
So, the modification would involve doing query with PQsendQuery and repeatedly call the PQgetResult function to get the result and be able to free it with PQclear.
I think this system does not really fit with ulogd task which consists in multiple short queries. It is far more adapted for long queries and for GUI using them as ulogd will continuously be calling PQgetResult to free result it will not use. So, it is complicated and does not seem really adapted.
Here's the points that make me consider using thread instead of asynchronous call.
BR,
--
Eric
Pablo Neira Ayuso
maintainerRe: [PATCH 09/11] db: add ring buffer for DB query
Hi Eric,
Quoted 13 lines · Eric Leblond
Quoted 12 lines · Pablo Neira Ayuso
Quoted 8 lines · Eric Leblond
This patch adds an optional ring buffer option which modify the way database queries are made. The main thread is only handling kernel message reading and query formatting. The SQL request is made in a separate dedicated thread.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side.Would be feasible to make asynchronous SQL requests instead, so you can skip the use of pthread?
[...]
Quoted 3 lines
From a lazy point of view, this would require to update all the database backends instead of simply updating the db abstraction layer as it is done with the proposed patch.
Fair enough, it provides a generic way to support this and the main thread should be really taking care of netlink messages coming from the kernel to avoid the overrun.
If the target is to overcome small bursts, this should help to relief those peaks for some time, but at some point the intermediate queue (the ring) will also get full if the situation lasts long.
Regards.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 10/11 db: disable SIGHUP if ring buffer is used.
In reply to Eric LeblondThe handling of signal when using threads can be complicated. When ring buffer is used for query, this means ulogd will have to follow some sort of mutex. Thus, it is easier and better performance wise to disable the reload via SIGHUP when the ring buffer is used.
--- util/db.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
util/db.c +8 -3
@@ -607,11 +607,16 @@ static void *__inject_thread(void *gdi)void ulogd_db_signal (struct ulogd_pluginstance *upi,int signal) { +struct db_instance *di = (struct db_instance *) &upi->private;switch (signal) {case SIGHUP : -/* reopen database connection */ -ulogd_db_instance_stop (upi); -ulogd_db_start (upi); +if (!di->ring.size) { +/* reopen database connection */ +ulogd_db_instance_stop (upi); +ulogd_db_start (upi); + }else +ulogd_log (ULOGD_ERROR , +"No SIGHUP handling if ring buffer is used\n" );break ;default :break ;
Signature · 1 line
1.7.10.4
PATCH 11/11 db: db ring has precedence over backlog.
In reply to Eric Leblond--- util/db.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
util/db.c +7 -5
@@ -177,9 +177,14 @@ int ulogd_db_configure(struct ulogd_pluginstance *upi,INIT_LLIST_HEAD (&di->backlog); di->backlog_memusage =0 ; - + + di->ring.size =ringsize_ce (upi->config_kset).u.value; di->backlog_memcap =backlog_memcap_ce (upi->config_kset).u.value; -if (di->backlog_memcap >0 ) { + +if (di->ring.size && di->backlog_memcap) { +ulogd_log (ULOGD_ERROR ,"Ring buffer has precedence over backlog\n" ); + di->backlog_memcap =0 ; + }else if (di->backlog_memcap >0 ) { di->backlog_oneshot =backlog_oneshot_ce (upi->config_kset).u.value;if (di->backlog_oneshot <=2 ) { ulogd_log(ULOGD_ERROR,
@@ -190,9 +195,6 @@ int ulogd_db_configure(struct ulogd_pluginstance *upi, di->backlog_full =0 ; } -/* check ring option */ - di->ring.size =ringsize_ce (upi->config_kset).u.value; -return ret; }
Signature · 1 line
1.7.10.4