Skip to contentSkip to conversation

ulogd2 db rework

RFC PATCH In review Eric Leblond 2 participants 15 messages May 10, 2013 – May 21, 2013 netfilter

lore
Share & tools
lore
#0 lore raw

cover ulogd2 db rework

Hello,

Here's a patchset for ulogd which is aiming at improving the way databases are handled. It features two different parts:
- a backlog system (patch 06) which allow queries to be store in memory when the database is not available. Data loss is prevented in this case (in the limit of the defined memcap)
- a multithreaded ulogd db output (patch 09) with one thread getting messages from kernel and an other one dedicated to the SQL queries execution.

The backlog system prevent data loss and only cost memory when the database is down. The multithreaded db output uses a ring buffer (which cost memory) for communication between the two threads.

The idea behind the multithreaded db output is to almost fix the time needed to read a kernel message. Doing this, ulogd should be more resistant to burst of kernel messages as kernel-user buffer will not be filled due to the high treatment time needed to execute SQL query.

Both features are modifying the generic db system inside ulogd. So the modification is available in pgsql, mysql and dbi output. It has for now only been tested on pgsql.

Comments, feedbacks and test results are more than welcome on both these features.

Patchset statistics:

 configure.ac                          |   32 ++-
 include/ulogd/db.h                    |   67 +++++-
 output/mysql/ulogd_output_MYSQL.c     |    4 +-
 output/pgsql/ulogd_output_PGSQL.c     |    4 +-
 output/sqlite3/ulogd_output_SQLITE3.c |    2 +
 src/Makefile.am                       |    2 +-
 src/ulogd.c                           |    2 +-
 ulogd.conf.in                         |   13 ++
 util/db.c                             |  371 +++++++++++++++++++++++++++------
 9 files changed, 417 insertions(+), 80 deletions(-)

BR,
--
Eric Leblond <[email protected]>

#1 lore raw

PATCH 1/11 ulogd: display stack during configuration

---
 src/ulogd.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
src/ulogd.c +1 -1
index b28d0f8..8f21cc2 100644
@@ -830,7 +830,7 @@ static int create_stack(const char *option)
 	}
 	INIT_LLIST_HEAD(&stack->list);
 
-	ulogd_log(ULOGD_DEBUG, "building new pluginstance stack (%s):\n",
+	ulogd_log(ULOGD_NOTICE, "building new pluginstance stack: '%s'\n",
 		  option);
 
 	/* PASS 1: find and instanciate plugins of stack, link them together */
Signature · 1 line
1.7.10.4
#2 lore raw

PATCH 2/11 Fix automagic support of dbi, pcap and sqlite3

In reply to Eric Leblond

From: Ilya Tumaykin <[email protected]>

ulogd has automagic deps for several output plugins right now, namely dbi, pcap and sqlite3. These plugins are built if the appropriate libs are present on user's system. While this situation is fine with binary distros it is not OK on source-based ones such as Gentoo.

The problem arises when such a program links against libs without user's request and libs are later removed from system which leaves program in a broken state.

This patch is modifying configure.ac which we apply in our package and which fixes mentioned issue. It adds 3 new configure options: -- without-{dbi,pcap.sqlite}. I would like to emphasize that this patch doesn't change default behaviour of configure script at all, so all other distros won't suffer. We simply add options to explicitly disable any attempts to try and detect libs for automagic deps, which is enough to avoid unnecessary linkage.

---
 configure.ac |   30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
configure.ac +20 -10
index c94704b..7e04531 100644
@@ -20,14 +20,6 @@ AC_PROG_LIBTOOL
 dnl Checks for libraries.
 AC_SEARCH_LIBS([dlopen], [dl], [libdl_LIBS="$LIBS"; LIBS=""])
 AC_SUBST([libdl_LIBS])
-AC_SEARCH_LIBS([pcap_close], [pcap], [libpcap_LIBS="-lpcap"; LIBS=""])
-AC_SUBST([libpcap_LIBS])
-AM_CONDITIONAL([HAVE_PCAP], [test -n "$libpcap_LIBS"])
-if test "x$libpcap_LIBS" != "x"; then
-	enable_pcap="yes"
-else
-	enable_pcap="no"
-fi
 
 dnl Checks for header files.
 AC_HEADER_DIRENT
@@ -88,7 +80,10 @@ else
 	enable_mysql="no"
 fi
 
-PKG_CHECK_MODULES([libsqlite3], [sqlite3], [], [:])
+AC_ARG_WITH([sqlite], AS_HELP_STRING([--without-sqlite], [Build without SQLITE3 output plugin [default=test]]))
+AS_IF([test "x$with_sqlite" != "xno"], [
+    PKG_CHECK_MODULES([libsqlite3], [sqlite3], [], [:])
+])
 AM_CONDITIONAL([HAVE_SQLITE3], [test -n "$libsqlite3_LIBS"])
 if test "x$libsqlite3_LIBS" != "x"; then
 	enable_sqlite3="yes"
@@ -96,7 +91,10 @@ else
 	enable_sqlite3="no"
 fi
 
-CT_CHECK_DBI()
+AC_ARG_WITH([dbi], AS_HELP_STRING([--without-dbi], [Build without DBI output plugin [default=test]]))
+AS_IF([test "x$with_dbi" != "xno"], [
+    CT_CHECK_DBI()
+])
 AM_CONDITIONAL(HAVE_DBI, test "x$DBI_LIB" != "x")
 if test "x$DBI_LIB" != "x"; then
 	enable_dbi="yes"
@@ -104,6 +102,18 @@ else
 	enable_dbi="no"
 fi
 
+AC_ARG_WITH([pcap], AS_HELP_STRING([--without-pcap], [Build without PCAP output plugin [default=test]]))
+AS_IF([test "x$with_pcap" != "xno"], [
+    AC_SEARCH_LIBS([pcap_close], [pcap], [libpcap_LIBS="-lpcap"; LIBS=""])
+    AC_SUBST([libpcap_LIBS])
+])
+AM_CONDITIONAL([HAVE_PCAP], [test -n "$libpcap_LIBS"])
+if test "x$libpcap_LIBS" != "x"; then
+	enable_pcap="yes"
+else
+	enable_pcap="no"
+fi
+
 dnl AC_SUBST(DATABASE_DIR)
 dnl AC_SUBST(DATABASE_LIB)
 dnl AC_SUBST(DATABASE_LIB_DIR)
Signature · 1 line
1.7.10.4
#3 lore raw

PATCH 3/11 postgresql: add sanity checking

In reply to Eric Leblond

Clean postgresql handler at deinit.

---
 output/pgsql/ulogd_output_PGSQL.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
output/pgsql/ulogd_output_PGSQL.c +3 -1
index f246153..88fb765 100644
@@ -214,7 +214,9 @@ static int close_db_pgsql(struct ulogd_pluginstance *upi)
 {
 	struct pgsql_instance *pi = (struct pgsql_instance *) upi->private;
 
-	PQfinish(pi->dbh);
+	if (pi->dbh)
+		PQfinish(pi->dbh);
+	pi->dbh = NULL;
 
 	return 0;
 }
Signature · 1 line
1.7.10.4
#4 lore raw

PATCH 4/11 mysql: add sanity checking

In reply to Eric Leblond

Nullify mysql handler at deinit.

---
 output/mysql/ulogd_output_MYSQL.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
output/mysql/ulogd_output_MYSQL.c +3 -1
index 72c080e..0a1ebfc 100644
@@ -162,7 +162,9 @@ static int get_columns_mysql(struct ulogd_pluginstance *upi)
 static int close_db_mysql(struct ulogd_pluginstance *upi)
 {
 	struct mysql_instance *mi = (struct mysql_instance *) upi->private;
-	mysql_close(mi->dbh);
+	if (mi->dbh)
+		mysql_close(mi->dbh);
+	mi->dbh = NULL;
 	return 0;
 }
 
Signature · 1 line
1.7.10.4
#5 lore raw

PATCH 5/11 sqlite3: add sanity checking

In reply to Eric Leblond

Nullify sqlite3 handler at deinit.

---
 output/sqlite3/ulogd_output_SQLITE3.c |    2 ++
 1 file changed, 2 insertions(+)
output/sqlite3/ulogd_output_SQLITE3.c +2 -0
index f9f2462..5c49055 100644
@@ -431,6 +431,8 @@ sqlite3_stop(struct ulogd_pluginstance *pi)
 
 	sqlite3_close(priv->dbh);
 
+	priv->dbh = NULL;
+
 	return 0;
 }
 
Signature · 1 line
1.7.10.4
#6 lore raw

PATCH 6/11 db: store data in memory during database downtime

In reply to Eric Leblond

This patch is adding a mechanism to store query in a backlog build in memory. This allow to store events during downtime in memory and realize the effective insertion when the database comes back. A memory cap is used to avoid any memory flooding.

---
 include/ulogd/db.h |   34 +++++++++--
 ulogd.conf.in      |    9 +++
 util/db.c          |  170 +++++++++++++++++++++++++++++++++++++++++++---------
 3 files changed, 180 insertions(+), 33 deletions(-)
include/ulogd/db.h +29 -5
index 1c910ff..a533902 100644
@@ -20,6 +20,12 @@ struct db_driver {
 			const char *stmt, unsigned int len);
 };
 
+struct db_stmt {
+	char *stmt;
+	int len;
+	struct llist_head list;
+};
+
 struct db_instance {
 	char *stmt; /* buffer for our insert statement */
 	char *stmt_val; /* pointer to the beginning of the "VALUES" part */
@@ -28,9 +34,15 @@ struct db_instance {
 	time_t reconnect;
 	int (*interp)(struct ulogd_pluginstance *upi);
 	struct db_driver *driver;
+	unsigned int backlog_memcap;
+	unsigned int backlog_memusage;
+	unsigned int backlog_oneshot;
+	unsigned char backlog_full;
+	struct llist_head backlog;
 };
 #define TIME_ERR		((time_t)-1)	/* Be paranoid */
 #define RECONNECT_DEFAULT	2
+#define MAX_ONESHOT_REQUEST	10
 
 #define DB_CES							\
 		{						\
@@ -51,13 +63,25 @@ struct db_instance {
 			.key = "procedure",			\
 			.type = CONFIG_TYPE_STRING,		\
 			.options = CONFIG_OPT_MANDATORY,	\
+		},						\
+		{						\
+			.key = "backlog_memcap",		\
+			.type = CONFIG_TYPE_INT,		\
+			.u.value = 0,				\
+		},						\
+		{						\
+			.key = "backlog_oneshot_requests",	\
+			.type = CONFIG_TYPE_INT,		\
+			.u.value = MAX_ONESHOT_REQUEST,		\
 		}
 
-#define DB_CE_NUM	4
-#define table_ce(x)	(x->ces[0])
-#define reconnect_ce(x)	(x->ces[1])
-#define timeout_ce(x)	(x->ces[2])
-#define procedure_ce(x)	(x->ces[3])
+#define DB_CE_NUM		6
+#define table_ce(x)		(x->ces[0])
+#define reconnect_ce(x)		(x->ces[1])
+#define timeout_ce(x)		(x->ces[2])
+#define procedure_ce(x)		(x->ces[3])
+#define backlog_memcap_ce(x)	(x->ces[4])
+#define backlog_oneshot_ce(x)	(x->ces[5])
 
 void ulogd_db_signal(struct ulogd_pluginstance *upi, int signal);
 int ulogd_db_start(struct ulogd_pluginstance *upi);
ulogd.conf.in +9 -0
index f4f63d9..3e5e648 100644
@@ -207,6 +207,13 @@ user="nupik"
 table="ulog"
 pass="changeme"
 procedure="INSERT_PACKET_FULL"
+# backlog configuration:
+# set backlog_memcap to the size of memory that will be
+# allocated to store events in memory if data is temporary down
+# and insert them when the database came back.
+#backlog_memcap=1000000
+# number of events to insert at once when backlog is not empty
+#backlog_oneshot_requests=10
 
 [mysql2]
 db="nulog"
@@ -224,6 +231,8 @@ table="ulog"
 #schema="public"
 pass="changeme"
 procedure="INSERT_PACKET_FULL"
+#backlog_memcap=1000000
+#backlog_oneshot_requests=10
 
 [pgsql2]
 db="nulog"
util/db.c +142 -28

205 lines not on this page · raw message

Signature · 1 line
1.7.10.4
#7 lore raw

PATCH 7/11 db: suppress field in db structure

In reply to Eric Leblond

The field is currently only used in a single function as a string pointer and can thus be removed from the db instance structure.

---
 include/ulogd/db.h |    2 +-
 util/db.c          |   36 ++++++++++++++++++------------------
 2 files changed, 19 insertions(+), 19 deletions(-)
include/ulogd/db.h +1 -1
index a533902..a02afb5 100644
@@ -1,6 +1,7 @@
 /* DB handling functions
  *
  * (C) 2000-2005 by Harald Welte <[email protected]>
+ * (C) 2013 by Eric Leblond <[email protected]>
  *
  * This code is distributed under the terms of GNU GPL version 2 */
 
@@ -29,7 +30,6 @@ struct db_stmt {
 struct db_instance {
 	char *stmt; /* buffer for our insert statement */
 	char *stmt_val; /* pointer to the beginning of the "VALUES" part */
-	char *stmt_ins; /* pointer to current inser position in statement */
 	char *schema;
 	time_t reconnect;
 	int (*interp)(struct ulogd_pluginstance *upi);
util/db.c +18 -18
index d125e21..ec0c045 100644
@@ -266,7 +266,7 @@ static void __format_query_db(struct ulogd_pluginstance *upi)
 
 	unsigned int i;
 
-	di->stmt_ins = di->stmt_val;
+	char * stmt_ins = di->stmt_val;
 
 	for (i = 0; i < upi->input.num_keys; i++) {
 		struct ulogd_key *res = upi->input.keys[i].u.source;
@@ -280,52 +280,52 @@ static void __format_query_db(struct ulogd_pluginstance *upi)
 			
 		if (!res || !IS_VALID(*res)) {
 			/* no result, we have to fake something */
-			di->stmt_ins += sprintf(di->stmt_ins, "NULL,");
+			stmt_ins += sprintf(stmt_ins, "NULL,");
 			continue;
 		}
 		
 		switch (res->type) {
 		case ULOGD_RET_INT8:
-			sprintf(di->stmt_ins, "%d,", res->u.value.i8);
+			sprintf(stmt_ins, "%d,", res->u.value.i8);
 			break;
 		case ULOGD_RET_INT16:
-			sprintf(di->stmt_ins, "%d,", res->u.value.i16);
+			sprintf(stmt_ins, "%d,", res->u.value.i16);
 			break;
 		case ULOGD_RET_INT32:
-			sprintf(di->stmt_ins, "%d,", res->u.value.i32);
+			sprintf(stmt_ins, "%d,", res->u.value.i32);
 			break;
 		case ULOGD_RET_INT64:
-			sprintf(di->stmt_ins, "%" PRId64 ",", res->u.value.i64);
+			sprintf(stmt_ins, "%" PRId64 ",", res->u.value.i64);
 			break;
 		case ULOGD_RET_UINT8:
-			sprintf(di->stmt_ins, "%u,", res->u.value.ui8);
+			sprintf(stmt_ins, "%u,", res->u.value.ui8);
 			break;
 		case ULOGD_RET_UINT16:
-			sprintf(di->stmt_ins, "%u,", res->u.value.ui16);
+			sprintf(stmt_ins, "%u,", res->u.value.ui16);
 			break;
 		case ULOGD_RET_IPADDR:
 			/* fallthrough when logging IP as u_int32_t */
 		case ULOGD_RET_UINT32:
-			sprintf(di->stmt_ins, "%u,", res->u.value.ui32);
+			sprintf(stmt_ins, "%u,", res->u.value.ui32);
 			break;
 		case ULOGD_RET_UINT64:
-			sprintf(di->stmt_ins, "%" PRIu64 ",", res->u.value.ui64);
+			sprintf(stmt_ins, "%" PRIu64 ",", res->u.value.ui64);
 			break;
 		case ULOGD_RET_BOOL:
-			sprintf(di->stmt_ins, "'%d',", res->u.value.b);
+			sprintf(stmt_ins, "'%d',", res->u.value.b);
 			break;
 		case ULOGD_RET_STRING:
-			*(di->stmt_ins++) = '\'';
+			*(stmt_ins++) = '\'';
 			if (res->u.value.ptr) {
-				di->stmt_ins +=
-				di->driver->escape_string(upi, di->stmt_ins,
+				stmt_ins +=
+				di->driver->escape_string(upi, stmt_ins,
 							  res->u.value.ptr,
 							strlen(res->u.value.ptr));
 			}
-			sprintf(di->stmt_ins, "',");
+			sprintf(stmt_ins, "',");
 			break;
 		case ULOGD_RET_RAWSTR:
-			sprintf(di->stmt_ins, "%s,", (char *) res->u.value.ptr);
+			sprintf(stmt_ins, "%s,", (char *) res->u.value.ptr);
 			break;
 		case ULOGD_RET_RAW:
 			ulogd_log(ULOGD_NOTICE,
@@ -336,9 +336,9 @@ static void __format_query_db(struct ulogd_pluginstance *upi)
 				res->type, upi->input.keys[i].name);
 			break;
 		}
-		di->stmt_ins = di->stmt + strlen(di->stmt);
+		stmt_ins = di->stmt + strlen(di->stmt);
 	}
-	*(di->stmt_ins - 1) = ')';
+	*(stmt_ins - 1) = ')';
 }
 
 static int __add_to_backlog(struct ulogd_pluginstance *upi, const char *stmt, unsigned int len)
Signature · 1 line
1.7.10.4
#8 lore raw

PATCH 8/11 db: use offset instead of direct pointer.

In reply to Eric Leblond

Use an offset approach to get the start of values printing area. It is more generic and will be use soon.

---
 include/ulogd/db.h |    2 +-
 util/db.c          |   16 +++++++++-------
 2 files changed, 10 insertions(+), 8 deletions(-)
include/ulogd/db.h +1 -1
index a02afb5..82f37b9 100644
@@ -29,7 +29,7 @@ struct db_stmt {
 
 struct db_instance {
 	char *stmt; /* buffer for our insert statement */
-	char *stmt_val; /* pointer to the beginning of the "VALUES" part */
+	int stmt_offset; /* offset to the beginning of the "VALUES" part */
 	char *schema;
 	time_t reconnect;
 	int (*interp)(struct ulogd_pluginstance *upi);
util/db.c +9 -7
index ec0c045..1a11173 100644
@@ -66,6 +66,7 @@ static int sql_createstmt(struct ulogd_pluginstance *upi)
 	unsigned int i;
 	char *table = table_ce(upi->config_kset).u.string;
 	char *procedure = procedure_ce(upi->config_kset).u.string;
+	char *stmt_val = NULL;
 
 	if (mi->stmt)
 		free(mi->stmt);
@@ -106,7 +107,7 @@ static int sql_createstmt(struct ulogd_pluginstance *upi)
 		else
 			sprintf(mi->stmt, "%s (", procedure);
 
-		mi->stmt_val = mi->stmt + strlen(mi->stmt);
+		stmt_val = mi->stmt + strlen(mi->stmt);
 
 		for (i = 0; i < upi->input.num_keys; i++) {
 			if (upi->input.keys[i].flags & ULOGD_KEYF_INACTIVE)
@@ -115,19 +116,20 @@ static int sql_createstmt(struct ulogd_pluginstance *upi)
 			strncpy(buf, upi->input.keys[i].name, ULOGD_MAX_KEYLEN);	
 			while ((underscore = strchr(buf, '.')))
 				*underscore = '_';
-			sprintf(mi->stmt_val, "%s,", buf);
-			mi->stmt_val = mi->stmt + strlen(mi->stmt);
+			sprintf(stmt_val, "%s,", buf);
+			stmt_val = mi->stmt + strlen(mi->stmt);
 		}
-		*(mi->stmt_val - 1) = ')';
+		*(stmt_val - 1) = ')';
 
-		sprintf(mi->stmt_val, " values (");
+		sprintf(stmt_val, " values (");
 	} else if (strncasecmp(procedure,"CALL", strlen("CALL")) == 0) {
 		sprintf(mi->stmt, "CALL %s(", procedure);
 	} else {
 		sprintf(mi->stmt, "SELECT %s(", procedure);
 
 	}
-	mi->stmt_val = mi->stmt + strlen(mi->stmt);
+
+	mi->stmt_offset = strlen(mi->stmt);
 
 	ulogd_log(ULOGD_DEBUG, "stmt='%s'\n", mi->stmt);
 
@@ -266,7 +268,7 @@ static void __format_query_db(struct ulogd_pluginstance *upi)
 
 	unsigned int i;
 
-	char * stmt_ins = di->stmt_val;
+	char * stmt_ins = di->stmt + di->stmt_offset;
 
 	for (i = 0; i < upi->input.num_keys; i++) {
 		struct ulogd_key *res = upi->input.keys[i].u.source;
Signature · 1 line
1.7.10.4
#9 lore raw

PATCH 9/11 db: add ring buffer for DB query

In reply to Eric Leblond

This patch adds an optional ring buffer option which modify the way database queries are made. The main thread is only handling kernel message reading and query formatting. The SQL request is made in a separate dedicated thread.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side.

---
 configure.ac       |    2 +
 include/ulogd/db.h |   31 ++++++++++-
 src/Makefile.am    |    2 +-
 ulogd.conf.in      |    4 ++
 util/db.c          |  152 ++++++++++++++++++++++++++++++++++++++++++++++++----
 5 files changed, 178 insertions(+), 13 deletions(-)
configure.ac +2 -0
index 7e04531..7351749 100644
@@ -39,6 +39,8 @@ AC_CHECK_FUNCS(socket strerror)
 regular_CFLAGS="-Wall -Wextra -Wno-unused-parameter"
 AC_SUBST([regular_CFLAGS])
 
+AC_CHECK_LIB(pthread, pthread_create)
+
 dnl Check for the right nfnetlink version
 PKG_CHECK_MODULES([LIBNFNETLINK], [libnfnetlink >= 1.0.1])
 PKG_CHECK_MODULES([LIBMNL], [libmnl >= 1.0.3])
include/ulogd/db.h +30 -1
index 82f37b9..823f462 100644
@@ -21,6 +21,24 @@ struct db_driver {
 			const char *stmt, unsigned int len);
 };
 
+enum {
+	RING_NO_QUERY,
+	RING_QUERY_READY,
+};
+
+struct db_stmt_ring {
+	/* Ring buffer: 1 status byte + string */
+	char* ring; /* pointer to the ring */
+	uint32_t size; /* size of ring buffer in element */
+	int length; /* length of one ring buffer element */
+	uint32_t wr_item; /* write item in ring buffer */
+	uint32_t rd_item; /* read item in ring buffer */
+	char *wr_place;
+	pthread_cond_t cond;
+	pthread_mutex_t mutex;
+	int full;
+};
+
 struct db_stmt {
 	char *stmt;
 	int len;
@@ -34,6 +52,10 @@ struct db_instance {
 	time_t reconnect;
 	int (*interp)(struct ulogd_pluginstance *upi);
 	struct db_driver *driver;
+	/* DB ring buffer */
+	struct db_stmt_ring ring;
+	pthread_t db_thread_id;
+	/* Backlog system */
 	unsigned int backlog_memcap;
 	unsigned int backlog_memusage;
 	unsigned int backlog_oneshot;
@@ -43,6 +65,7 @@ struct db_instance {
 #define TIME_ERR		((time_t)-1)	/* Be paranoid */
 #define RECONNECT_DEFAULT	2
 #define MAX_ONESHOT_REQUEST	10
+#define RING_BUFFER_DEFAULT_SIZE	10
 
 #define DB_CES							\
 		{						\
@@ -73,15 +96,21 @@ struct db_instance {
 			.key = "backlog_oneshot_requests",	\
 			.type = CONFIG_TYPE_INT,		\
 			.u.value = MAX_ONESHOT_REQUEST,		\
+		},						\
+		{						\
+			.key = "ring_buffer_size",		\
+			.type = CONFIG_TYPE_INT,		\
+			.u.value = RING_BUFFER_DEFAULT_SIZE,	\
 		}
 
-#define DB_CE_NUM		6
+#define DB_CE_NUM		7
 #define table_ce(x)		(x->ces[0])
 #define reconnect_ce(x)		(x->ces[1])
 #define timeout_ce(x)		(x->ces[2])
 #define procedure_ce(x)		(x->ces[3])
 #define backlog_memcap_ce(x)	(x->ces[4])
 #define backlog_oneshot_ce(x)	(x->ces[5])
+#define ringsize_ce(x)		(x->ces[6])
 
 void ulogd_db_signal(struct ulogd_pluginstance *upi, int signal);
 int ulogd_db_start(struct ulogd_pluginstance *upi);
src/Makefile.am +1 -1
index e462cb2..1097468 100644
@@ -8,4 +8,4 @@ sbin_PROGRAMS = ulogd
 
 ulogd_SOURCES = ulogd.c select.c timer.c rbtree.c conffile.c hash.c addr.c
 ulogd_LDADD   = ${libdl_LIBS}
-ulogd_LDFLAGS = -export-dynamic
+ulogd_LDFLAGS = -export-dynamic -lpthread
ulogd.conf.in +4 -0
index 3e5e648..11a56d6 100644
@@ -233,6 +233,10 @@ pass="changeme"
 procedure="INSERT_PACKET_FULL"
 #backlog_memcap=1000000
 #backlog_oneshot_requests=10
+# If superior to 1 a thread dedicated to SQL request execution
+# is created. The value stores the number of SQL request to keep
+# in the ring buffer
+#ring_buffer_size=1000
 
 [pgsql2]
 db="nulog"
util/db.c +141 -11

262 lines not on this page · raw message

Signature · 1 line
1.7.10.4
#10 lore raw

Re: [PATCH 09/11] db: add ring buffer for DB query

Hi Eric,

Quoted 8 lines · Eric Leblond

This patch adds an optional ring buffer option which modify the way database queries are made. The main thread is only handling kernel message reading and query formatting. The SQL request is made in a separate dedicated thread.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side.

Would be feasible to make asynchronous SQL requests instead, so you can skip the use of pthread?

Regards.

#11 lore raw

Re: [PATCH 09/11] db: add ring buffer for DB query

Hi,

Quoted 12 lines · Pablo Neira Ayuso

Hi Eric,

Quoted 8 lines · Eric Leblond

This patch adds an optional ring buffer option which modify the way database queries are made. The main thread is only handling kernel message reading and query formatting. The SQL request is made in a separate dedicated thread.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side.

Quoted 2 lines

Would be feasible to make asynchronous SQL requests instead, so you can skip the use of pthread?

That is an excellent question :)

Here's a list of points. From the small to big one.

From a performance point of view, ulogd will still be slow down by the request time due to the network communication with the db . I think the effect is really negligible but may have an impact in the case of distant db.

From a lazy point of view, this would require to update all the database backends instead of simply updating the db abstraction layer as it is done with the proposed patch.

Regarding the use of asynchronous request, I've always stop trying using them after reading the doc. So I've got no practical experience here and correct someone correct me if I'm wrong. What I've understood is that there is no "take my query and just do your work" function. Application has to get the result to the query and free them.

For example, let's have a look at postgresql case. The doc on asynchronous API is here:
http://www.postgresql.org/docs/6.4/static/libpq-chapter17044.htm

Here's an interesting part of the documentation:

PQsendQuery: Submit a query to Postgres without waiting for the result(s). TRUE is returned if the query was successfully dispatched, FALSE if not (in which case, use PQerrorMessage to get more information about the failure).
After successfully calling PQsendQuery, call PQgetResult one or more times to obtain the query results. PQsendQuery may not be called again (on the same connection) until PQgetResult has returned NULL, indicating that the query is done.

On other interesting point regarding PQgetResult:

Don't forget to free each result object with PQclear when done with it.

So, the modification would involve doing query with PQsendQuery and repeatedly call the PQgetResult function to get the result and be able to free it with PQclear.

I think this system does not really fit with ulogd task which consists in multiple short queries. It is far more adapted for long queries and for GUI using them as ulogd will continuously be calling PQgetResult to free result it will not use. So, it is complicated and does not seem really adapted.

Here's the points that make me consider using thread instead of asynchronous call.

BR,
--
Eric

#12 lore raw

Re: [PATCH 09/11] db: add ring buffer for DB query

Hi Eric,

Quoted 13 lines · Eric Leblond
Quoted 12 lines · Pablo Neira Ayuso
Quoted 8 lines · Eric Leblond

This patch adds an optional ring buffer option which modify the way database queries are made. The main thread is only handling kernel message reading and query formatting. The SQL request is made in a separate dedicated thread.
The idea is to try to avoid buffer overrun by minimizing the time requested to treat kernel message. Doing synchronous SQL request, as it was made before was causing a delay which could cause some messages to be lost in case of burst from kernel side.

Would be feasible to make asynchronous SQL requests instead, so you can skip the use of pthread?

[...]

Quoted 3 lines

From a lazy point of view, this would require to update all the database backends instead of simply updating the db abstraction layer as it is done with the proposed patch.

Fair enough, it provides a generic way to support this and the main thread should be really taking care of netlink messages coming from the kernel to avoid the overrun.

If the target is to overcome small bursts, this should help to relief those peaks for some time, but at some point the intermediate queue (the ring) will also get full if the situation lasts long.

Regards.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#13 lore raw

PATCH 10/11 db: disable SIGHUP if ring buffer is used.

In reply to Eric Leblond

The handling of signal when using threads can be complicated. When ring buffer is used for query, this means ulogd will have to follow some sort of mutex. Thus, it is easier and better performance wise to disable the reload via SIGHUP when the ring buffer is used.

---
 util/db.c |   11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
util/db.c +8 -3
index 4050f0f..c23c362 100644
@@ -607,11 +607,16 @@ static void *__inject_thread(void *gdi)
 
 void ulogd_db_signal(struct ulogd_pluginstance *upi, int signal)
 {
+	struct db_instance *di = (struct db_instance *) &upi->private;
 	switch (signal) {
 	case SIGHUP:
-		/* reopen database connection */
-		ulogd_db_instance_stop(upi);
-		ulogd_db_start(upi);
+		if (!di->ring.size) {
+			/* reopen database connection */
+			ulogd_db_instance_stop(upi);
+			ulogd_db_start(upi);
+		} else
+			ulogd_log(ULOGD_ERROR,
+				  "No SIGHUP handling if ring buffer is used\n");
 		break;
 	default:
 		break;
Signature · 1 line
1.7.10.4
#14 lore raw

PATCH 11/11 db: db ring has precedence over backlog.

In reply to Eric Leblond
---
 util/db.c |   12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
util/db.c +7 -5
index c23c362..6be06e6 100644
@@ -177,9 +177,14 @@ int ulogd_db_configure(struct ulogd_pluginstance *upi,
 
 	INIT_LLIST_HEAD(&di->backlog);
 	di->backlog_memusage = 0;
-	
+
+	di->ring.size = ringsize_ce(upi->config_kset).u.value;
 	di->backlog_memcap = backlog_memcap_ce(upi->config_kset).u.value;
-	if (di->backlog_memcap > 0) {
+
+	if (di->ring.size && di->backlog_memcap) {
+		ulogd_log(ULOGD_ERROR, "Ring buffer has precedence over backlog\n");
+		di->backlog_memcap = 0;
+	} else if (di->backlog_memcap > 0) {
 		di->backlog_oneshot = backlog_oneshot_ce(upi->config_kset).u.value;
 		if (di->backlog_oneshot <= 2) {
 			ulogd_log(ULOGD_ERROR,
@@ -190,9 +195,6 @@ int ulogd_db_configure(struct ulogd_pluginstance *upi,
 		di->backlog_full = 0;
 	}
 
-	/* check ring option */
-	di->ring.size = ringsize_ce(upi->config_kset).u.value;
-
 	return ret;
 }
 
Signature · 1 line
1.7.10.4