a software based on ulogd

RFC New Ken-ichirou MATSUZAWA 2 participants 10 messages Feb 6, 2016 – Feb 10, 2016 netfilter

lore

Share & tools

lore

Ken-ichirou MATSUZAWA

author

2016-02-06 10:42 UTC #0 lore raw

Hi,

I need Netflow data based on conntrack, I am trying to create a software which is based on ulogd.

https://github.com/chamaken/nurs/

I copied COPYING and README from ulogd source code to distribute, and added comment:

based on ulogd which was almost entirely written by Harald Welte, with contributions from fellow hackers such as Pablo Neira Ayuso, Eric Leblond and Pierre Chifflier.

to each source file. Would you tell me other things to reuse ulogd? And I would appreciate it if I could get your advice to this software.

Thanks,

Eric Leblond

2016-02-07 10:52 UTC #1 lore raw

Hello Ken-ichirou,

Le 6 févr. 2016 11:42 AM, Ken-ichirou MATSUZAWA <[email protected]> a écrit :

Quoted 7 lines

Hi,
I need Netflow data based on conntrack, I am trying to create a software which is based on ulogd.
https://github.com/chamaken/nurs/

I feel sorry not being able to merge your work on ulogd due to lack of time on my side. Do you think you could resubmit a patchset adding netflow support to ulogd ? I will make the necessary to get it included.

Best regards,

Quoted 18 lines

I copied COPYING and README from ulogd source code to distribute, and added comment:
based on ulogd which was almost entirely written by Harald Welte, with contributions from fellow hackers such as Pablo Neira Ayuso, Eric Leblond and Pierre Chifflier.
to each source file. Would you tell me other things to reuse ulogd? And I would appreciate it if I could get your advice to this software.
Thanks,
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in 
the body of a message to [email protected] 
More majordomo info at  http://vger.kernel.org/majordomo-info.html 

Ken-ichirou MATSUZAWA

author

2016-02-10 01:54 UTC #2 lore raw

Quoted 1 line · Eric Leblond

Do you think you could resubmit a patchset adding netflow support to ulogd ?

I can send patches, but I have not used this and I don't know if I will use it or not.

I think NFCT plugin also needs to update to propagate delta counter in practical use. Following patch series can be used with NFCT destory event as I wrote in comment, and tested it a little bit with nfdump.

Thanks,

Ken-ichirou MATSUZAWA

author

2016-02-10 01:56 UTC #3 lore raw

PATCH 1/7 ipfix: add flowDirection IE

Signed-off-by Ken-ichirou MATSUZAWA <[email protected]>

---
 include/ulogd/ipfix_protocol.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

include/ulogd/ipfix_protocol.h +1 -1

index aef47f0..bff0b5c 100644
@@ -100,7 +100,7 @@ enum {
 	IPFIX_vlanId			= 58,
 	IPFIX_postVlanId		= 59,
 	IPFIX_ipVersion			= 60,
-	/* reserved */
+	IPFIX_flowDirection		= 61,
 	IPFIX_ipNextHopIPv6Address	= 62,
 	IPFIX_bgpNexthopIPv6Address	= 63,
 	IPFIX_ipv6ExtensionHeaders	= 64,

Signature · 1 line

2.1.4

Ken-ichirou MATSUZAWA

author

2016-02-10 01:58 UTC #4 lore raw

PATCH 2/7 nfct/ipfix: introduce new vendor id

In reply to Ken-ichirou MATSUZAWA

IPFIX_VENDOR_REVERSE, defined in RFC 5103 6.1 Reverse Information Element Private Enterprise Number. And use it at counter in nfct.

Signed-off-by Ken-ichirou MATSUZAWA <[email protected]>

---
 include/ulogd/ipfix_protocol.h  | 3 +++
 input/flow/ulogd_inpflow_NFCT.c | 4 ++--
 2 files changed, 5 insertions(+), 2 deletions(-)

include/ulogd/ipfix_protocol.h +3 -0

index bff0b5c..330f0ea 100644
@@ -11,6 +11,9 @@
 /* defined in http://www.iana.org/assignments/enterprise-numbers */
 #define IPFIX_VENDOR_NETFILTER	21373	/* FIXME: htonl? */
 
+/* defined in RFC 5103 IPFIX Biflow Export */
+#define IPFIX_VENDOR_REVERSE	29305
+
 /* Section 3.1 */
 struct ipfix_msg_hdr {
 	uint16_t	version;

input/flow/ulogd_inpflow_NFCT.c +2 -2

index 899b7e3..0b3b339 100644
@@ -309,7 +309,7 @@ static struct ulogd_key nfct_okeys[] = {
 		.flags	= ULOGD_RETF_NONE,
 		.name	= "reply.raw.pktlen",
 		.ipfix	= {
-			.vendor 	= IPFIX_VENDOR_IETF,
+			.vendor 	= IPFIX_VENDOR_REVERSE,
 			.field_id 	= IPFIX_octetTotalCount,
 			/* FIXME: this could also be octetDeltaCount */
 		},

@@ -319,7 +319,7 @@ static struct ulogd_key nfct_okeys[] = {
 		.flags	= ULOGD_RETF_NONE,
 		.name	= "reply.raw.pktcount",
 		.ipfix	= {
-			.vendor 	= IPFIX_VENDOR_IETF,
+			.vendor 	= IPFIX_VENDOR_REVERSE,
 			.field_id 	= IPFIX_packetTotalCount,
 			/* FIXME: this could also be packetDeltaCount */
 		},

Signature · 1 line

2.1.4

Ken-ichirou MATSUZAWA

author

2016-02-10 02:01 UTC #5 lore raw

PATCH 3/7 nfct/ipfix: introduce NAT entries

In reply to Ken-ichirou MATSUZAWA

Signed-off-by Ken-ichirou MATSUZAWA <[email protected]>

---
 include/ulogd/ipfix_protocol.h  | 10 ++++++++++
 input/flow/ulogd_inpflow_NFCT.c | 17 ++++++++---------
 2 files changed, 18 insertions(+), 9 deletions(-)

include/ulogd/ipfix_protocol.h +10 -0

index 330f0ea..23fa440 100644
@@ -219,6 +219,16 @@ enum {
 	/* reserved */
 	IPFIX_headerLengthIPv4		= 213,
 	IPFIX_mplsPayloadLength		= 214,
+
+	/* select usefuls from:
+	 * http://www.iana.org/assignments/ipfix/ipfix.txt */
+	IPFIX_postNATSourceIPv4Address		= 225,
+	IPFIX_postNATDestinationIPv4Address	= 226,
+	IPFIX_postNAPTSourceTransportPort	= 227,
+	IPFIX_postNAPTDestinationTransportPort	= 228,
+	IPFIX_firewallEvent			= 233,
+	IPFIX_postNATSourceIPv6Address		= 281,
+	IPFIX_postNATDestinationIPv6Address	= 282,
 };
 
 /* Information elements of the netfilter vendor id */

input/flow/ulogd_inpflow_NFCT.c +4 -4

index 0b3b339..8f9492a 100644
@@ -265,7 +265,7 @@ static struct ulogd_key nfct_okeys[] = {
 		.name	= "reply.ip.saddr",
 		.ipfix	= {
 			.vendor = IPFIX_VENDOR_IETF,
-			.field_id = IPFIX_sourceIPv4Address,
+			.field_id = IPFIX_postNATSourceIPv4Address,
 		},
 	},
 	{

@@ -274,7 +274,7 @@ static struct ulogd_key nfct_okeys[] = {
 		.name	= "reply.ip.daddr",
 		.ipfix	= {
 			.vendor = IPFIX_VENDOR_IETF,
-			.field_id = IPFIX_destinationIPv4Address,
+			.field_id = IPFIX_postNATDestinationIPv4Address,
 		},
 	},
 	{

@@ -292,7 +292,7 @@ static struct ulogd_key nfct_okeys[] = {
 		.name	= "reply.l4.sport",
 		.ipfix	= {
 			.vendor 	= IPFIX_VENDOR_IETF,
-			.field_id 	= IPFIX_sourceTransportPort,
+			.field_id 	= IPFIX_postNAPTSourceTransportPort,
 		},
 	},
 	{

@@ -301,7 +301,7 @@ static struct ulogd_key nfct_okeys[] = {
 		.name	= "reply.l4.dport",
 		.ipfix	= {
 			.vendor 	= IPFIX_VENDOR_IETF,
-			.field_id 	= IPFIX_destinationTransportPort,
+			.field_id 	= IPFIX_postNAPTDestinationTransportPort,
 		},
 	},
 	{

Signature · 1 line

2.1.4

Ken-ichirou MATSUZAWA

author

2016-02-10 02:02 UTC #6 lore raw

PATCH 4/7 filter: add new filter for Netflow ICMP_TYPE

In reply to Ken-ichirou MATSUZAWA

This filter creates ICMP_TYPE Netflow v9 from IPFIX icmpTypeIPv4 and icmpCodeIPv4.

Signed-off-by Ken-ichirou MATSUZAWA <[email protected]>

---
 filter/Makefile.am             |   6 ++-
 filter/ulogd_filter_PACKICMP.c | 101 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 106 insertions(+), 1 deletion(-)
 create mode 100644 filter/ulogd_filter_PACKICMP.c

filter/Makefile.am +5 -1

index 875850b..70a2bcc 100644
@@ -7,7 +7,8 @@ pkglib_LTLIBRARIES = ulogd_filter_IFINDEX.la ulogd_filter_PWSNIFF.la \
 			 ulogd_filter_PRINTPKT.la ulogd_filter_PRINTFLOW.la \
 			 ulogd_filter_IP2STR.la ulogd_filter_IP2BIN.la \
 			 ulogd_filter_HWHDR.la ulogd_filter_MARK.la \
-			 ulogd_filter_IP2HBIN.la
+			 ulogd_filter_IP2HBIN.la \
+			 ulogd_filter_PACKICMP.la
 
 ulogd_filter_IFINDEX_la_SOURCES = ulogd_filter_IFINDEX.c
 ulogd_filter_IFINDEX_la_LDFLAGS = -avoid-version -module

@@ -36,3 +37,6 @@ ulogd_filter_PRINTPKT_la_LDFLAGS = -avoid-version -module
 
 ulogd_filter_PRINTFLOW_la_SOURCES = ulogd_filter_PRINTFLOW.c ../util/printflow.c
 ulogd_filter_PRINTFLOW_la_LDFLAGS = -avoid-version -module
+
+ulogd_filter_PACKICMP_la_SOURCES = ulogd_filter_PACKICMP.c
+ulogd_filter_PACKICMP_la_LDFLAGS = -avoid-version -module

filter/ulogd_filter_PACKICMP.c new file +101 -0

101 lines not on this page · raw message

Signature · 1 line

2.1.4

Ken-ichirou MATSUZAWA

author

2016-02-10 02:03 UTC #7 lore raw

PATCH 5/7 filter: add new filter for IPFIX time

In reply to Ken-ichirou MATSUZAWA

This filter creates IPFIX_flow(Start|End)MicroSeconds and IPFIX_flow(Start|End)SysUpTime from "flow.(start|end).sec" and "flow.(start|end).usec".

Signed-off-by Ken-ichirou MATSUZAWA <[email protected]>

---
 filter/Makefile.am             |   5 +-
 filter/ulogd_filter_TIMECONV.c | 316 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 320 insertions(+), 1 deletion(-)
 create mode 100644 filter/ulogd_filter_TIMECONV.c

filter/Makefile.am +4 -1

index 70a2bcc..c1943dc 100644
@@ -8,7 +8,7 @@ pkglib_LTLIBRARIES = ulogd_filter_IFINDEX.la ulogd_filter_PWSNIFF.la \
 			 ulogd_filter_IP2STR.la ulogd_filter_IP2BIN.la \
 			 ulogd_filter_HWHDR.la ulogd_filter_MARK.la \
 			 ulogd_filter_IP2HBIN.la \
-			 ulogd_filter_PACKICMP.la
+			 ulogd_filter_PACKICMP.la ulogd_filter_TIMECONV.la
 
 ulogd_filter_IFINDEX_la_SOURCES = ulogd_filter_IFINDEX.c
 ulogd_filter_IFINDEX_la_LDFLAGS = -avoid-version -module

@@ -40,3 +40,6 @@ ulogd_filter_PRINTFLOW_la_LDFLAGS = -avoid-version -module
 
 ulogd_filter_PACKICMP_la_SOURCES = ulogd_filter_PACKICMP.c
 ulogd_filter_PACKICMP_la_LDFLAGS = -avoid-version -module
+
+ulogd_filter_TIMECONV_la_SOURCES = ulogd_filter_TIMECONV.c
+ulogd_filter_TIMECONV_la_LDFLAGS = -avoid-version -module

filter/ulogd_filter_TIMECONV.c new file +316 -0

316 lines not on this page · raw message

Signature · 1 line

2.1.4

Ken-ichirou MATSUZAWA

author

2016-02-10 02:04 UTC #8 lore raw

PATCH 6/7 ulogd: update calling stop callback condition

In reply to Ken-ichirou MATSUZAWA

TIMECONV filter in previous patch has private data but does not have stop callback, then segfault occured.

Signed-off-by Ken-ichirou MATSUZAWA <[email protected]>

---
 src/ulogd.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

src/ulogd.c +3 -3

index 5b9a586..7e1a42f 100644
@@ -1288,13 +1288,13 @@ static void stop_pluginstances()
 
 	llist_for_each_entry(stack, &ulogd_pi_stacks, stack_list) {
 		llist_for_each_entry_safe(pi, npi, &stack->list, list) {
-			if ((pi->plugin->priv_size > 0 || *pi->plugin->stop) &&
-			    pluginstance_stop(pi)) {
+			if (*pi->plugin->stop && pluginstance_stop(pi)) {
 				ulogd_log(ULOGD_DEBUG, "calling stop for %s\n",
 					  pi->plugin->name);
 				(*pi->plugin->stop)(pi);
-				pi->private[0] = 0;
 			}
+			if (pi->plugin->priv_size > 0)
+				pi->private[0] = 0;
 			free(pi);
 		}
 	}

Signature · 1 line

2.1.4

Ken-ichirou MATSUZAWA

author

2016-02-10 02:05 UTC #9 lore raw

PATCH 7/7 nflow9: introduce new NetFlow v9 output plugin

In reply to Ken-ichirou MATSUZAWA

This patch introduces a NFLOW9 output plugin which sends Netflow v9 encoded NFCT destroy events.

Signed-off-by Ken-ichirou MATSUZAWA <[email protected]>

---
 output/Makefile.am           |   10 +
 output/ulogd_output_NFLOW9.c | 1696 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 1706 insertions(+)
 create mode 100644 output/ulogd_output_NFLOW9.c

output/Makefile.am +10 -0

index ff851ad..6f62b48 100644
@@ -13,6 +13,10 @@ if HAVE_JANSSON
 pkglib_LTLIBRARIES += ulogd_output_JSON.la
 endif
 
+if BUILD_NFCT
+pkglib_LTLIBRARIES += ulogd_output_NFLOW9.la
+endif
+
 ulogd_output_GPRINT_la_SOURCES = ulogd_output_GPRINT.c
 ulogd_output_GPRINT_la_LDFLAGS = -avoid-version -module

@@ -42,3 +46,9 @@ ulogd_output_JSON_la_SOURCES = ulogd_output_JSON.c
 ulogd_output_JSON_la_LIBADD  = ${libjansson_LIBS}
 ulogd_output_JSON_la_LDFLAGS = -avoid-version -module
 endif
+
+if BUILD_NFCT
+ulogd_output_NFLOW9_la_SOURCES = ulogd_output_NFLOW9.c
+ulogd_output_NFLOW9_la_LIBADD  = $(LIBNETFILTER_CONNTRACK_LIBS)
+ulogd_output_NFLOW9_la_LDFLAGS = -avoid-version -module
+endif

output/ulogd_output_NFLOW9.c new file +1696 -0

1,696 lines not on this page · raw message

Signature · 1 line

2.1.4