Skip to content

Implement event loss prevention for db output

Eric Leblond netfilter-devel 1 version New

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest4 patches 0 repliesNew

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4postgresql: add sanity checkingoutput/pgsql/ulogd_output_PGSQL.c+3 -10 replies
2/4mysql: add sanity checkingoutput/mysql/ulogd_output_MYSQL.c+3 -10 replies
3/4sqlite3: add sanity checkingoutput/sqlite3/ulogd_output_SQLITE3.c+2 -00 replies
4/4db: store data in memory during database downtimeinclude/ulogd/db.h ulogd.conf.in util/db.c+180 -330 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [Ulogd PATCH 0/4] Implement event loss prevention for db output

Hello,

The first three patches are cleaning that could be necessary to avoid issue with the fourth one who does the real job.

The idea is to store the queries in memory if the database goes down and to proceed to the insertion when it gets back up. To do so, a chained list of queries is attached to the database instance and queries are added to it till a memory cap is not reached. When the database came back, the queries are played in order so it should not cause any issue for any potential triggers.

This new feature is inactive by default and can be activated by setting "backlog_memcap" to a non null value in the database instance configuration.

To avoid to cause overun on the event capture by inserting too many things at once at recovery (and thus blocking capture for a certain time), a maximum of "backlog_oneshot_requests" " are inserted at a time.

Patchset statistics:

 include/ulogd/db.h                    |   34 ++++++++++--
 output/mysql/ulogd_output_MYSQL.c     |    4 +-
 output/pgsql/ulogd_output_PGSQL.c     |    4 +-
 output/sqlite3/ulogd_output_SQLITE3.c |    2 +
 ulogd.conf.in                         |    9 +++
 util/db.c                             |  170 +++++++++++++++++++++++++++++++++++++++++++++++----------
 6 files changed, 188 insertions(+), 35 deletions(-)

BR,
--
Eric Leblond <[email protected]>