RFC ulogd2 db rework
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest11 patches 3 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/11 | ulogd: display stack during configurationsrc/ | +1 | 0 replies | |
| 2/11 | Fix automagic support of dbi, pcap and sqlite3configure.ac | +20 | 0 replies | |
| 3/11 | postgresql: add sanity checkingoutput/ | +3 | 0 replies | |
| 4/11 | mysql: add sanity checkingoutput/ | +3 | 0 replies | |
| 5/11 | sqlite3: add sanity checkingoutput/ | +2 | 0 replies | |
| 6/11 | db: store data in memory during database downtimeinclude/ ulogd.conf.in util/ | +180 | 0 replies | |
| 7/11 | db: suppress field in db structureinclude/ util/ | +19 | 0 replies | |
| 8/11 | db: use offset instead of direct pointer.include/ util/ | +10 | 0 replies | |
| 9/11 | db: add ring buffer for DB queryconfigure.ac include/ src/ +2 more | +178 | 3 replies | |
| 10/11 | db: disable SIGHUP if ring buffer is used.util/ | +8 | 0 replies | |
| 11/11 | db: db ring has precedence over backlog.util/ | +7 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [PATCH RFC 0/11] ulogd2 db rework
Hello,
Here's a patchset for ulogd which is aiming at improving the way databases are handled. It features two different parts:
- a backlog system (patch 06) which allow queries to be store in memory when the database is not available. Data loss is prevented in this case (in the limit of the defined memcap)
- a multithreaded ulogd db output (patch 09) with one thread getting messages from kernel and an other one dedicated to the SQL queries execution.
The backlog system prevent data loss and only cost memory when the database is down. The multithreaded db output uses a ring buffer (which cost memory) for communication between the two threads.
The idea behind the multithreaded db output is to almost fix the time needed to read a kernel message. Doing this, ulogd should be more resistant to burst of kernel messages as kernel-user buffer will not be filled due to the high treatment time needed to execute SQL query.
Both features are modifying the generic db system inside ulogd. So the modification is available in pgsql, mysql and dbi output. It has for now only been tested on pgsql.
Comments, feedbacks and test results are more than welcome on both these features.
Patchset statistics:
configure.ac | 32 ++- include/ulogd/db.h | 67 +++++- output/mysql/ulogd_output_MYSQL.c | 4 +- output/pgsql/ulogd_output_PGSQL.c | 4 +- output/sqlite3/ulogd_output_SQLITE3.c | 2 + src/Makefile.am | 2 +- src/ulogd.c | 2 +- ulogd.conf.in | 13 ++ util/db.c | 371 +++++++++++++++++++++++++++------ 9 files changed, 417 insertions(+), 80 deletions(-)
BR,
--
Eric Leblond <[email protected]>