Skip to contentSkip to conversation

[ULOGD PATCH 0/4]

PATCH Applied Eric Leblond 2 participants 12 messages Mar 31, 2008 – Apr 13, 2008 1 message not in the archive netfilter

lore
Share & tools
lore

Eric Leblond

author
#0 lore raw

cover [ULOGD PATCH 0/4]

Hello,

This patchset features some SQL improvement, a partial rewrite of NACCT output plugin and an sync of ulogd.conf.in with current state of the code.

First two patches are SQL related. They simply synchronise MySQL code with current NFCT code:
* Fix SQL field naming.
* Convert SQL procedure to function in MySQL plugins.

Third patch is
* IPv6 compliance of NACCT output plugin.
NACCT code was heavily dependant of the order of keys provided by input plugins. As change in other parts of the code is quiet important, NACCT should'nt have been in a working state. I've rewritten it using current coding style and added IPv6 support.

Last patch is a sync of example configuration file with current ulogd capabilities:
* Add some example to ulogd.conf.in.

BR,
--
Eric Leblond
INL: http://www.inl.fr/
NuFW: http://www.nufw.org/

author
#1 lore raw

PATCH 1/4 Fix SQL field naming.

SQL fields were not correctly named from ulogd key.

Signed-off-by Eric Leblond <[email protected]>
---
 doc/mysql-ulogd2.sql |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
doc/mysql-ulogd2.sql +2 -2
index 304173b..6368dc7 100644
@@ -297,8 +297,8 @@ DROP VIEW IF EXISTS `conntrack`;
 CREATE SQL SECURITY INVOKER VIEW `conntrack` AS
 	SELECT _ct_id,
 	       oob_family,
-	       orig_ip_saddr AS orig_ip_saddr_raw,
-	       orig_ip_daddr AS orig_ip_daddr_raw,
+	       orig_ip_saddr AS orig_ip_saddr_bin,
+	       orig_ip_daddr AS orig_ip_daddr_bin,
 	       orig_ip_protocol,
 	       orig_l4_sport,
 	       orig_l4_dport,
Signature · 1 line
1.5.2.5
maintainer
#2 lore raw

Re: [ULOGD PATCH 1/4] Fix SQL field naming.

Quoted 5 lines · Eric Leblond
doc/mysql-ulogd2.sql +0 -0
index 304173b..6368dc7 100644

doc/mysql-ulogd2.sql DROP VIEW IF EXISTS `conntrack`; · line 304

Quoted 10 lines
doc/mysql-ulogd2.sql +2 -2
@@ -297,8 +297,8 @@ DROP VIEW IF EXISTS `conntrack`;
 CREATE SQL SECURITY INVOKER VIEW `conntrack` AS
 	SELECT _ct_id,
 	       oob_family,
-	       orig_ip_saddr AS orig_ip_saddr_raw,
-	       orig_ip_daddr AS orig_ip_daddr_raw,
+	       orig_ip_saddr AS orig_ip_saddr_bin,
+	       orig_ip_daddr AS orig_ip_daddr_bin,
 	       orig_ip_protocol,
 	       orig_l4_sport,
 	       orig_l4_dport,

What do we gain with this renaming?

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
author
#3 lore raw

Re: [ULOGD PATCH 1/4] Fix SQL field naming.

Hi,

Quoted 10 lines
Quoted 9 lines · Eric Leblond
doc/mysql-ulogd2.sql +0 -1
index 304173b..6368dc7 100644
@@ -297,8 +297,8 @@ DROP VIEW IF EXISTS `conntrack`;
 CREATE SQL SECURITY INVOKER VIEW `conntrack` AS
 	SELECT _ct_id,
 	       oob_family,
-	       orig_ip_saddr AS orig_ip_saddr_raw,
Quoted 8 lines

doc/mysql-ulogd2.sql DROP VIEW IF EXISTS `conntrack`; · line 304

Quoted 6 lines
doc/mysql-ulogd2.sql +2 -1
-	       orig_ip_daddr AS orig_ip_daddr_raw,
+	       orig_ip_saddr AS orig_ip_saddr_bin,
+	       orig_ip_daddr AS orig_ip_daddr_bin,
 	       orig_ip_protocol,
 	       orig_l4_sport,
 	       orig_l4_dport,

What do we gain with this renaming?

This fix an inconsistency of field naming among the different tables and accross the stack NFCT IP2BIN MYSQL.In fact IP2BIN out .bin suffixed fields.

BR,
--
Eric Leblond
INL : http://www.inl.fr
NuFW, Now User Filtering Works (http://www.nufw.org)

maintainer
#4 lore raw

Re: [ULOGD PATCH 1/4] Fix SQL field naming.

Quoted 20 lines · Eric Leblond

Hi,

Quoted 17 lines
Quoted 5 lines · Eric Leblond
doc/mysql-ulogd2.sql +0 -0
index 304173b..6368dc7 100644

doc/mysql-ulogd2.sql DROP VIEW IF EXISTS `conntrack`; · line 304

Quoted 10 lines
doc/mysql-ulogd2.sql +2 -2
@@ -297,8 +297,8 @@ DROP VIEW IF EXISTS `conntrack`;
 CREATE SQL SECURITY INVOKER VIEW `conntrack` AS
 	SELECT _ct_id,
 	       oob_family,
-	       orig_ip_saddr AS orig_ip_saddr_raw,
-	       orig_ip_daddr AS orig_ip_daddr_raw,
+	       orig_ip_saddr AS orig_ip_saddr_bin,
+	       orig_ip_daddr AS orig_ip_daddr_bin,
 	       orig_ip_protocol,
 	       orig_l4_sport,
 	       orig_l4_dport,

What do we gain with this renaming?

Quoted 3 lines

This fix an inconsistency of field naming among the different tables and accross the stack NFCT IP2BIN MYSQL.In fact IP2BIN out .bin suffixed fields.

Makes sense. Applied. Thanks Eric.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
author
#5 lore raw

PATCH 2/4 Convert SQL procedure to function in MySQL plugins.

In reply to Eric Leblond

DB output plugins have been modified to use function instead of procedure. This patch convert the conntrack insertion procedure to function.

Signed-off-by Eric Leblond <[email protected]>
---
 doc/mysql-ulogd2.sql |   56 +++++++++++++++++++++++++------------------------
 1 files changed, 29 insertions(+), 27 deletions(-)
doc/mysql-ulogd2.sql +29 -27
index 6368dc7..d6c6d46 100644
@@ -689,46 +689,48 @@ END
 $$
 
 delimiter $$
-DROP PROCEDURE IF EXISTS INSERT_CT;
-CREATE PROCEDURE INSERT_CT(
-		IN `_orig_ip_saddr` binary(16),
-		IN `_orig_ip_daddr` binary(16),
-		IN `_orig_ip_protocol` tinyint(3) unsigned,
-		IN `_orig_l4_sport` int(5),
-		IN `_orig_l4_dport` int(5),
-		IN `_orig_bytes` bigint,
-		IN `_orig_packets` bigint,
-		IN `_reply_ip_saddr` binary(16),
-		IN `_reply_ip_daddr` binary(16),
-		IN `_reply_ip_protocol` tinyint(3) unsigned,
-		IN `_reply_l4_sport` int(5),
-		IN `_reply_l4_dport` int(5),
-		IN `_reply_bytes` bigint,
-		IN `_reply_packets` bigint,
-		IN `_icmp_code` tinyint(3),
-		IN `_icmp_type` tinyint(3),
-		IN `_ct_mark` bigint,
-		IN `_flow_start_sec` int(10),
-		IN `_flow_start_usec` int(10),
-		IN `_flow_end_sec` int(10),
-		IN `_flow_end_usec` int(10)
-		)
+DROP FUNCTION IF EXISTS INSERT_CT;
+CREATE FUNCTION INSERT_CT(
+		`_oob_family` bigint,
+		`_orig_ip_saddr` binary(16),
+		`_orig_ip_daddr` binary(16),
+		`_orig_ip_protocol` tinyint(3) unsigned,
+		`_orig_l4_sport` int(5),
+		`_orig_l4_dport` int(5),
+		`_orig_bytes` bigint,
+		`_orig_packets` bigint,
+		`_reply_ip_saddr` binary(16),
+		`_reply_ip_daddr` binary(16),
+		`_reply_ip_protocol` tinyint(3) unsigned,
+		`_reply_l4_sport` int(5),
+		`_reply_l4_dport` int(5),
+		`_reply_bytes` bigint,
+		`_reply_packets` bigint,
+		`_icmp_code` tinyint(3),
+		`_icmp_type` tinyint(3),
+		`_ct_mark` bigint,
+		`_flow_start_sec` int(10),
+		`_flow_start_usec` int(10),
+		`_flow_end_sec` int(10),
+		`_flow_end_usec` int(10)
+		) RETURNS bigint unsigned
+READS SQL DATA
 BEGIN
-	INSERT INTO ulog2_ct (orig_ip_saddr, orig_ip_daddr, orig_ip_protocol,
+	INSERT INTO ulog2_ct (oob_family, orig_ip_saddr, orig_ip_daddr, orig_ip_protocol,
 		orig_l4_sport, orig_l4_dport, orig_bytes, orig_packets,
 		reply_ip_saddr, reply_ip_daddr, reply_ip_protocol,
 		reply_l4_sport, reply_l4_dport, reply_bytes, reply_packets,
 		icmp_code, icmp_type, ct_mark, 
 		flow_start_sec, flow_start_usec,
 		flow_end_sec, flow_end_usec)
- 	VALUES (_orig_ip_saddr, _orig_ip_daddr, _orig_ip_protocol,
+ 	VALUES (_oob_family, _orig_ip_saddr, _orig_ip_daddr, _orig_ip_protocol,
 		_orig_l4_sport, _orig_l4_dport, _orig_bytes, _orig_packets,
 		_reply_ip_saddr, _reply_ip_daddr, _reply_ip_protocol,
 		_reply_l4_sport, _reply_l4_dport, _reply_bytes, _reply_packets,
 		_icmp_code, _icmp_type, _ct_mark,
 		_flow_start_sec, _flow_start_usec,
 		_flow_end_sec, _flow_end_usec);
-
+	RETURN LAST_INSERT_ID();
 END
 $$
Signature · 1 line
1.5.2.5
maintainer
#6 lore raw

Re: [ULOGD PATCH 2/4] Convert SQL procedure to function in MySQL plugins.

Quoted 2 lines · Eric Leblond

DB output plugins have been modified to use function instead of procedure. This patch convert the conntrack insertion procedure to function.

Where is the client code that uses the returned value of the INSERT_CT function?

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#8 lore raw

Re: [ULOGD PATCH 2/4] Convert SQL procedure to function in MySQL plugins.

Hello,

Quoted 7 lines
Quoted 3 lines · Eric Leblond

DB output plugins have been modified to use function instead of procedure.
This patch convert the conntrack insertion procedure to function.

Where is the client code that uses the returned value of the INSERT_CT function?

It's not yet developed ;)

In fact, this change has been done to be in sync with a modification in the packet logging infrastructure which uses a function. If we have a function and a procedure we could not use the same SQL request prefix. We should have add an option to be able to set the correct prefix. This is quiet painfull if you consider that having a function can be useful :
as it returns the ID of the packet if could be used by a third party application which use the same database. For example, it could use it to log packet :
- Call to the function INSERT_CT
- Do other treatment

Furthermore, it seems that PROCEDURE is not trully a SQL thing only FUNCTION are defined.

BR,
--
Eric Leblond
INL : http://www.inl.fr
NuFW, Now User Filtering Works (http://www.nufw.org)

author
#9 lore raw

PATCH 3/4 IPv6 compliance of NACCT output plugin.

In reply to Eric Leblond

NACCT was IPv4 only and was heavily dependant of the order of NFCT keys. This patch introduces a explicit list of input keys and obtains IPv6 compliance by using IP2STR output as input for IP address.

Signed-off-by Eric Leblond <[email protected]>
---
 output/ulogd_output_NACCT.c |  137 ++++++++++++++++++++++++++++++-------------
 1 files changed, 96 insertions(+), 41 deletions(-)
output/ulogd_output_NACCT.c +96 -41
index 2da73e7..278df36 100644
@@ -31,30 +31,85 @@
 
 #define NACCT_FILE_DEFAULT	"/var/log/nacctdata.log"
 
-#define HIPQUAD(addr) \
-        ((unsigned char *)&addr)[3], \
-        ((unsigned char *)&addr)[2], \
-        ((unsigned char *)&addr)[1], \
-        ((unsigned char *)&addr)[0]
-
 /* config accessors (lazy me...) */
 #define NACCT_CFG_FILE(pi)	((pi)->config_kset->ces[0].u.string)
 #define NACCT_CFG_SYNC(pi)	((pi)->config_kset->ces[1].u.value)
 
-#define KEY(pi,idx)		((pi)->input.keys[(idx)].u.source)
+enum input_keys {
+	KEY_IP_SADDR,
+	KEY_IP_DADDR,
+	KEY_IP_PROTO,
+	KEY_L4_SPORT,
+	KEY_L4_DPORT,
+	KEY_RAW_PKTLEN,
+	KEY_RAW_PKTCNT,
+	KEY_ICMP_CODE,
+	KEY_ICMP_TYPE,
+	KEY_FLOW_START,
+	KEY_FLOW_END,
+};
 
 /* input keys */
-#define KEY_IP_SADDR(pi)		KEY(pi, 0)
-#define KEY_IP_DADDR(pi)		KEY(pi, 1)
-#define KEY_IP_PROTO(pi)		KEY(pi, 2)
-#define KEY_L4_SPORT(pi)		KEY(pi, 3)
-#define KEY_L4_DPORT(pi)		KEY(pi, 4)
-#define KEY_RAW_PKTLEN(pi)		KEY(pi, 5)
-#define KEY_RAW_PKTCNT(pi)		KEY(pi, 6)
-#define KEY_ICMP_CODE(pi)		KEY(pi, 7)
-#define KEY_ICMP_TYPE(pi)		KEY(pi, 8)
-#define KEY_FLOW_START(pi)		KEY(pi, 11)
-#define KEY_FLOW_END(pi)		KEY(pi, 13)
+static struct ulogd_key nacct_inp[] = {
+	[KEY_IP_SADDR] = {
+		.type = ULOGD_RET_STRING,
+		.flags = ULOGD_RETF_NONE,
+		.name = "orig.ip.saddr.str",
+	},
+	[KEY_IP_DADDR] = {
+		.type = ULOGD_RET_STRING,
+		.flags = ULOGD_RETF_NONE,
+		.name = "orig.ip.daddr.str",
+	},
+	[KEY_IP_PROTO] = {
+			.type	= ULOGD_RET_UINT8,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "orig.ip.protocol",
+
+	},
+	[KEY_L4_SPORT] = {
+	.type	= ULOGD_RET_UINT16,
+		.flags 	= ULOGD_RETF_NONE,
+		.name	= "orig.l4.sport",
+
+	},
+	[KEY_L4_DPORT] = {
+		.type	= ULOGD_RET_UINT16,
+		.flags 	= ULOGD_RETF_NONE,
+		.name	= "orig.l4.dport",
+	},
+	/* Assume we're interested more in download than upload */
+	[KEY_RAW_PKTLEN] = {
+		.type	= ULOGD_RET_UINT32,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "reply.raw.pktlen",
+	},
+	[KEY_RAW_PKTCNT] = {
+		.type	= ULOGD_RET_UINT32,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "reply.raw.pktcount",
+	},
+	[KEY_ICMP_CODE] = {
+		.type	= ULOGD_RET_UINT8,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "icmp.code",
+	},
+	[KEY_ICMP_TYPE] = { 
+		.type	= ULOGD_RET_UINT8,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "icmp.type",
+	},
+	[KEY_FLOW_START] = {
+		.type 	= ULOGD_RET_UINT32,
+		.flags 	= ULOGD_RETF_NONE,
+		.name	= "flow.start.sec",
+	},
+	[KEY_FLOW_END] = {
+		.type	= ULOGD_RET_UINT32,
+		.flags	= ULOGD_RETF_NONE,
+		.name	= "flow.end.sec",
+	},
+};
 
 struct nacct_priv {
 	FILE *of;

@@ -65,32 +120,33 @@ static int
 nacct_interp(struct ulogd_pluginstance *pi)
 {
 	struct nacct_priv *priv = (struct nacct_priv *)&pi->private;
-	static char buf[80];
+	struct ulogd_key *inp = pi->input.keys;
+	static char buf[256];
 
 	/* try to be as close to nacct as possible.  Instead of nacct's
 	   'timestamp' value use 'flow.end.sec' */
-	if (KEY_IP_PROTO(pi)->u.value.ui8 == IPPROTO_ICMP) {
+	if (GET_VALUE(inp, KEY_IP_PROTO).ui8 == IPPROTO_ICMP) {
 		snprintf(buf, sizeof(buf),
-				 "%u\t%u\t%u.%u.%u.%u\t%u\t%u.%u.%u.%u\t%u\t%u\t%u",
-				 KEY_FLOW_END(pi)->u.value.ui32,
-				 KEY_IP_PROTO(pi)->u.value.ui8,
-				 HIPQUAD(KEY_IP_SADDR(pi)->u.value.ui32),
-				 KEY_ICMP_TYPE(pi)->u.value.ui8,
-				 HIPQUAD(KEY_IP_DADDR(pi)->u.value.ui32),
-				 KEY_ICMP_CODE(pi)->u.value.ui8,
-				 KEY_RAW_PKTCNT(pi)->u.value.ui32,
-				 KEY_RAW_PKTLEN(pi)->u.value.ui32);
+				 "%u\t%u\t%s\t%u\t%s\t%u\t%u\t%u",
+				 GET_VALUE(inp, KEY_FLOW_END).ui32,
+				 GET_VALUE(inp, KEY_IP_PROTO).ui8,
+				 GET_VALUE(inp, KEY_IP_SADDR).ptr,
+				 GET_VALUE(inp, KEY_ICMP_TYPE).ui8,
+				 GET_VALUE(inp, KEY_IP_DADDR).ptr,
+				 GET_VALUE(inp, KEY_ICMP_CODE).ui8,
+				 GET_VALUE(inp, KEY_RAW_PKTCNT).ui32,
+				 GET_VALUE(inp, KEY_RAW_PKTLEN).ui32);
 	} else {
 		snprintf(buf, sizeof(buf),
-				 "%u\t%u\t%u.%u.%u.%u\t%u\t%u.%u.%u.%u\t%u\t%u\t%u",
-				 KEY_FLOW_END(pi)->u.value.ui32,
-				 KEY_IP_PROTO(pi)->u.value.ui8,
-				 HIPQUAD(KEY_IP_SADDR(pi)->u.value.ui32),
-				 KEY_L4_SPORT(pi)->u.value.ui8,
-				 HIPQUAD(KEY_IP_DADDR(pi)->u.value.ui32),
-				 KEY_L4_DPORT(pi)->u.value.ui8,
-				 KEY_RAW_PKTCNT(pi)->u.value.ui32,
-				 KEY_RAW_PKTLEN(pi)->u.value.ui32);
+				 "%u\t%u\t%s\t%u\t%s\t%u\t%u\t%u",
+				 GET_VALUE(inp, KEY_FLOW_END).ui32,
+				 GET_VALUE(inp, KEY_IP_PROTO).ui8,
+				 GET_VALUE(inp, KEY_IP_SADDR).ptr,
+				 GET_VALUE(inp, KEY_L4_SPORT).ui16,
+				 GET_VALUE(inp, KEY_IP_DADDR).ptr,
+				 GET_VALUE(inp, KEY_L4_DPORT).ui16,
+				 GET_VALUE(inp, KEY_RAW_PKTCNT).ui32,
+				 GET_VALUE(inp, KEY_RAW_PKTLEN).ui32);
 	}
 
 	fprintf(priv->of, "%s\n", buf);

@@ -147,9 +203,6 @@ nacct_conf(struct ulogd_pluginstance *pi,
 {
 	int ret;
 
-	if ((ret = ulogd_wildcard_inputkeys(pi)) < 0)
-		return ret;
-
 	if ((ret = config_parse_file(pi->id, pi->config_kset)) < 0)
 		return ret;
 

@@ -183,6 +236,8 @@ nacct_fini(struct ulogd_pluginstance *pi)
 static struct ulogd_plugin nacct_plugin = {
 	.name = "NACCT", 
 	.input = {
+		.keys = nacct_inp,
+		.num_keys = ARRAY_SIZE(nacct_inp),
 		.type = ULOGD_DTYPE_PACKET | ULOGD_DTYPE_FLOW,
 	},
 	.output = {
Signature · 1 line
1.5.2.5
maintainer Applied
#10 lore raw

Re: [ULOGD PATCH 3/4] IPv6 compliance of NACCT output plugin.

Quoted 3 lines · Eric Leblond

NACCT was IPv4 only and was heavily dependant of the order of NFCT keys. This patch introduces a explicit list of input keys and obtains IPv6 compliance by using IP2STR output as input for IP address.

Applied. Thanks.

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers
author
#11 lore raw

PATCH 4/4 Add some example to ulogd.conf.in.

In reply to Eric Leblond

This patch adds some example to the default configuration file. It modify some stack to take my latest patches into account.

Signed-off-by Eric Leblond <[email protected]>
---
 ulogd.conf.in |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)
ulogd.conf.in +9 -2
index d18ed5d..fde2e4e 100644
@@ -43,7 +43,8 @@ plugin="@libdir@/ulogd/ulogd_filter_PRINTPKT.so"
 plugin="@libdir@/ulogd/ulogd_filter_PRINTFLOW.so"
 plugin="@libdir@/ulogd/ulogd_output_LOGEMU.so"
 plugin="@libdir@/ulogd/ulogd_output_SYSLOG.so"
-plugin="@libdir@/ulogd/ulogd_output_OPRINT.so"
+#plugin="@libdir@/ulogd/ulogd_output_OPRINT.so"
+#plugin="@libdir@/ulogd/ulogd_output_NACCT.so"
 #plugin="@libdir@/ulogd/ulogd_output_PCAP.so"
 #plugin="@libdir@/ulogd/ulogd_output_PGSQL.so"
 #plugin="@libdir@/ulogd/ulogd_output_MYSQL.so"

@@ -62,7 +63,7 @@ plugin="@libdir@/ulogd/ulogd_raw2packet_BASE.so"
 #stack=ulog1:ULOG,base1:BASE,ip2str1:IP2STR,print1:PRINTPKT,emu1:LOGEMU
 
 # this is a stack for flow-based logging via LOGEMU
-#stack=ct1:NFCT,print1:PRINTFLOW,emu1:LOGEMU
+#stack=ct1:NFCT,ip2str1:IP2STR,print1:PRINTFLOW,emu1:LOGEMU
 
 # this is a stack for flow-based logging via OPRINT
 #stack=ct1:NFCT,op1:OPRINT

@@ -85,6 +86,9 @@ plugin="@libdir@/ulogd/ulogd_raw2packet_BASE.so"
 # this is a stack for flow-based logging to MySQL
 #stack=ct1:NFCT,ip2bin1:IP2BIN,mysql2:MYSQL
 
+# this is a stack for flow-based logging in NACCT compatible format
+#stack=ct1:NFCT,ip2str1:IP2STR,nacct1:NACCT
+
 [ct1]
 
 # IPv4 logging through NFLOG

@@ -137,3 +141,6 @@ procedure="INSERT_PACKET_FULL"
 
 [sys2]
 facility=LOG_LOCAL2
+
+[nacct1]
+sync = 1
Signature · 1 line
1.5.2.5
maintainer Applied
#12 lore raw

Re: [ULOGD PATCH 4/4] Add some example to ulogd.conf.in.

Quoted 2 lines · Eric Leblond

This patch adds some example to the default configuration file. It modify some stack to take my latest patches into account.

Applied. We're in sync, OK? :)

Signature · 1 line
"Los honestos son inadaptados sociales" -- Les Luthiers