Skip to content

RFC ulogd2 db rework

Eric Leblond netfilter-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest11 patches 3 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11ulogd: display stack during configurationsrc/ulogd.c+1 -10 replies
2/11Fix automagic support of dbi, pcap and sqlite3configure.ac+20 -100 replies
3/11postgresql: add sanity checkingoutput/pgsql/ulogd_output_PGSQL.c+3 -10 replies
4/11mysql: add sanity checkingoutput/mysql/ulogd_output_MYSQL.c+3 -10 replies
5/11sqlite3: add sanity checkingoutput/sqlite3/ulogd_output_SQLITE3.c+2 -00 replies
6/11db: store data in memory during database downtimeinclude/ulogd/db.h ulogd.conf.in util/db.c+180 -330 replies
7/11db: suppress field in db structureinclude/ulogd/db.h util/db.c+19 -190 replies
8/11db: use offset instead of direct pointer.include/ulogd/db.h util/db.c+10 -80 replies
9/11db: add ring buffer for DB queryconfigure.ac include/ulogd/db.h src/Makefile.am +2 more+178 -133 replies
10/11db: disable SIGHUP if ring buffer is used.util/db.c+8 -30 replies
11/11db: db ring has precedence over backlog.util/db.c+7 -50 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH RFC 0/11] ulogd2 db rework

Hello,

Here's a patchset for ulogd which is aiming at improving the way databases are handled. It features two different parts:
- a backlog system (patch 06) which allow queries to be store in memory when the database is not available. Data loss is prevented in this case (in the limit of the defined memcap)
- a multithreaded ulogd db output (patch 09) with one thread getting messages from kernel and an other one dedicated to the SQL queries execution.

The backlog system prevent data loss and only cost memory when the database is down. The multithreaded db output uses a ring buffer (which cost memory) for communication between the two threads.

The idea behind the multithreaded db output is to almost fix the time needed to read a kernel message. Doing this, ulogd should be more resistant to burst of kernel messages as kernel-user buffer will not be filled due to the high treatment time needed to execute SQL query.

Both features are modifying the generic db system inside ulogd. So the modification is available in pgsql, mysql and dbi output. It has for now only been tested on pgsql.

Comments, feedbacks and test results are more than welcome on both these features.

Patchset statistics:

 configure.ac                          |   32 ++-
 include/ulogd/db.h                    |   67 +++++-
 output/mysql/ulogd_output_MYSQL.c     |    4 +-
 output/pgsql/ulogd_output_PGSQL.c     |    4 +-
 output/sqlite3/ulogd_output_SQLITE3.c |    2 +
 src/Makefile.am                       |    2 +-
 src/ulogd.c                           |    2 +-
 ulogd.conf.in                         |   13 ++
 util/db.c                             |  371 +++++++++++++++++++++++++++------
 9 files changed, 417 insertions(+), 80 deletions(-)

BR,
--
Eric Leblond <[email protected]>