Skip to contentSkip to conversation

ulogd2 question - meaning of flow.start.sec when hash_mode == 0 (NCFT input, JSON output)

PATCH Applied Michal Soltys 4 participants 6 messages Apr 16, 2019 – Apr 22, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

When using hash_mode == 0 with default event_mask, it looks like destruction entry has uninitialized (or having some specific meaning) flow.start.sec and flow.start.usec - as the former converts to early 1970.

Peeking at the code in event_handler_no_hashtable() it looks like it should be set correctly, but alas something seems amiss.

creation event:

{"timestamp": "2019-04-16T12:02:30+0200", "dvc": "testdev", "orig.ip.protocol": 6, "orig.l4.sport": 43562, "orig.l4.dport": 22, "orig.raw.pktlen": 0, "orig.raw.pktcount": 0, "reply.ip.protocol": 6, "reply.l4.sport": 22, "reply.l4.dport": 43562, "reply.raw.pktlen": 0, "reply.raw.pktcount": 0, "ct.mark": 0, "ct.id": 784049088, "ct.event": 1, "flow.start.sec": 1555408950, "flow.start.usec": 517694, "oob.family": 2, "oob.protocol": 0, "src_ip": "192.168.0.254", "dest_ip": "10.151.151.22", "reply.ip.saddr.str": "10.151.151.22", "reply.ip.daddr.str": "192.168.0.254"}

destruction event:

{"timestamp": "2019-04-16T12:05:00+0200", "dvc": "testdev", "orig.ip.protocol": 6, "orig.l4.sport": 43562, "orig.l4.dport": 22, "orig.raw.pktlen": 115789, "orig.raw.pktcount": 2011, "reply.ip.protocol": 6, "reply.l4.sport": 22, "reply.l4.dport": 43562, "reply.raw.pktlen": 28701079, "reply.raw.pktcount": 13119, "ct.mark": 0, "ct.id": 784049088, "ct.event": 4, "flow.start.sec": 4084, "flow.start.usec": 805056, "flow.end.sec": 1555409100, "flow.end.usec": 321544, "oob.family": 2, "oob.protocol": 0, "src_ip": "192.168.0.254", "dest_ip": "10.151.151.22", "reply.ip.saddr.str": "10.151.151.22", "reply.ip.daddr.str": "192.168.0.254"}

flow.start.sec is 4084 == Thu, 01 Jan 1970 02:08:04 +0100

Any ideas ?

#1 lore raw

Re: ulogd2 question - meaning of flow.start.sec when hash_mode == 0 (NFCT input, JSON output)

Quoted 32 lines · Michal Soltys

When using hash_mode == 0 with default event_mask, it looks like destruction entry has uninitialized (or having some specific meaning) flow.start.sec and flow.start.usec - as the former converts to early 1970.

Peeking at the code in event_handler_no_hashtable() it looks like it should be set correctly, but alas something seems amiss.

creation event:

{"timestamp": "2019-04-16T12:02:30+0200", "dvc": "testdev", "orig.ip.protocol": 6, "orig.l4.sport": 43562, "orig.l4.dport": 22, "orig.raw.pktlen": 0, "orig.raw.pktcount": 0, "reply.ip.protocol": 6, "reply.l4.sport": 22, "reply.l4.dport": 43562, "reply.raw.pktlen": 0, "reply.raw.pktcount": 0, "ct.mark": 0, "ct.id": 784049088, "ct.event":
1, "flow.start.sec": 1555408950, "flow.start.usec": 517694, "oob.family": 2, "oob.protocol": 0, "src_ip": "192.168.0.254", "dest_ip": "10.151.151.22", "reply.ip.saddr.str": "10.151.151.22", "reply.ip.daddr.str": "192.168.0.254"}

destruction event:

{"timestamp": "2019-04-16T12:05:00+0200", "dvc": "testdev", "orig.ip.protocol": 6, "orig.l4.sport": 43562, "orig.l4.dport": 22, "orig.raw.pktlen": 115789, "orig.raw.pktcount": 2011, "reply.ip.protocol": 6, "reply.l4.sport": 22, "reply.l4.dport": 43562, "reply.raw.pktlen": 28701079, "reply.raw.pktcount": 13119, "ct.mark": 0, "ct.id": 784049088, "ct.event": 4, "flow.start.sec": 4084, "flow.start.usec": 805056, "flow.end.sec": 1555409100, "flow.end.usec":
321544, "oob.family": 2, "oob.protocol": 0, "src_ip": "192.168.0.254", "dest_ip": "10.151.151.22", "reply.ip.saddr.str": "10.151.151.22", "reply.ip.daddr.str": "192.168.0.254"}

Quoted 4 lines

flow.start.sec is 4084 == Thu, 01 Jan 1970 02:08:04 +0100

Any ideas ?

For the record, the same unusual timestamp is shown when monitoring with conntrack:

    [NEW] tcp      6 120 SYN_SENT src=192.168.0.254 dst=10.151.151.22 sport=44572 dport=22 [UNREPLIED] src=10.151.151.22 dst=192.168.0.254 sport=22 dport=44572
 [UPDATE] tcp      6 60 SYN_RECV src=192.168.0.254 dst=10.151.151.22 sport=44572 dport=22 src=10.151.151.22 dst=192.168.0.254 sport=22 dport=44572
 [UPDATE] tcp      6 432000 ESTABLISHED src=192.168.0.254 dst=10.151.151.22 sport=44572 dport=22 src=10.151.151.22 dst=192.168.0.254 sport=22 dport=44572 [ASSURED]
 [UPDATE] tcp      6 120 FIN_WAIT src=192.168.0.254 dst=10.151.151.22 sport=44572 dport=22 src=10.151.151.22 dst=192.168.0.254 sport=22 dport=44572 [ASSURED]
 [UPDATE] tcp      6 60 CLOSE_WAIT src=192.168.0.254 dst=10.151.151.22 sport=44572 dport=22 src=10.151.151.22 dst=192.168.0.254 sport=22 dport=44572 [ASSURED]
 [UPDATE] tcp      6 30 LAST_ACK src=192.168.0.254 dst=10.151.151.22 sport=44572 dport=22 src=10.151.151.22 dst=192.168.0.254 sport=22 dport=44572 [ASSURED]
 [UPDATE] tcp      6 120 TIME_WAIT src=192.168.0.254 dst=10.151.151.22 sport=44572 dport=22 src=10.151.151.22 dst=192.168.0.254 sport=22 dport=44572 [ASSURED]
[DESTROY] tcp      6 src=192.168.0.254 dst=10.151.151.22 sport=44572 dport=22 packets=55 bytes=6577 src=10.151.151.22 dst=192.168.0.254 sport=22 dport=44572 packets=51 bytes=8919 [ASSURED] delta-time=1555405003 [start=Thu Jan  1 04:58:06 1970] [stop=Tue Apr 16 14:54:49 2019]
maintainer
#2 lore raw

PATCH 1/1 netfilter: never get/set skb->tstamp

In reply to Michal Soltys

setting net.netfilter.nf_conntrack_timestamp=1 breaks xmit with fq scheduler. skb->tstamp might be "refreshed" using ktime_get_real(), but fq expects CLOCK_MONOTONIC.

This patch removes all places in netfilter that check/set skb->tstamp:

1. To fix the bogus "start" time seen with conntrack timestamping for outgoing packets, never use skb->tstamp and always use current time.
2. In nfqueue and nflog, only use skb->tstamp for incoming packets, as determined by current hook (prerouting, input, forward).
3. xt_time has to use system clock as well rather than skb->tstamp. We could still use skb->tstamp for prerouting/input/foward, but I see no advantage to make this conditional.

Fixes fb420d5d91c1 ("tcp/fq: move back to CLOCK_MONOTONIC")
Cc Eric Dumazet <[email protected]>
---
 net/netfilter/nf_conntrack_core.c |  7 ++-----
 net/netfilter/nfnetlink_log.c     |  2 +-
 net/netfilter/nfnetlink_queue.c   |  2 +-
 net/netfilter/xt_time.c           | 23 ++++++++++++++---------
 4 files changed, 18 insertions(+), 16 deletions(-)
net/netfilter/nf_conntrack_core.c +2 -5
index 3c48d44d6fff..2a714527cde1 100644
@@ -1017,12 +1017,9 @@ __nf_conntrack_confirm(struct sk_buff *skb)
 
 	/* set conntrack timestamp, if enabled. */
 	tstamp = nf_conn_tstamp_find(ct);
-	if (tstamp) {
-		if (skb->tstamp == 0)
-			__net_timestamp(skb);
+	if (tstamp)
+		tstamp->start = ktime_get_real_ns();
 
-		tstamp->start = ktime_to_ns(skb->tstamp);
-	}
 	/* Since the lookup is lockless, hash insertion must be done after
 	 * starting the timer and setting the CONFIRMED bit. The RCU barriers
 	 * guarantee that no other CPU can find the conntrack before the above
net/netfilter/nfnetlink_log.c +1 -1
index b1f9c5303f02..7d69a02bf016 100644
@@ -540,7 +540,7 @@ __build_packet_message(struct nfnl_log_net *log,
 			goto nla_put_failure;
 	}
 
-	if (skb->tstamp) {
+	if (hooknum <= NF_INET_FORWARD && skb->tstamp) {
 		struct nfulnl_msg_packet_timestamp ts;
 		struct timespec64 kts = ktime_to_timespec64(skb->tstamp);
 		ts.sec = cpu_to_be64(kts.tv_sec);
net/netfilter/nfnetlink_queue.c +1 -1
index 0dcc3592d053..e057b2961d31 100644
@@ -582,7 +582,7 @@ nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue,
 	if (nfqnl_put_bridge(entry, skb) < 0)
 		goto nla_put_failure;
 
-	if (entskb->tstamp) {
+	if (entry->state.hook <= NF_INET_FORWARD && entskb->tstamp) {
 		struct nfqnl_msg_packet_timestamp ts;
 		struct timespec64 kts = ktime_to_timespec64(entskb->tstamp);
 
net/netfilter/xt_time.c +14 -9
index c13bcd0ab491..8dbb4d48f2ed 100644
@@ -163,19 +163,24 @@ time_mt(const struct sk_buff *skb, struct xt_action_param *par)
 	s64 stamp;
 
 	/*
-	 * We cannot use get_seconds() instead of __net_timestamp() here.
+	 * We need real time here, but we can neither use skb->tstamp
+	 * nor __net_timestamp().
+	 *
+	 * skb->tstamp and skb->skb_mstamp_ns overlap, however, they
+	 * use different clock types (real vs monotonic).
+	 *
 	 * Suppose you have two rules:
-	 * 	1. match before 13:00
-	 * 	2. match after 13:00
+	 *	1. match before 13:00
+	 *	2. match after 13:00
+	 *
 	 * If you match against processing time (get_seconds) it
 	 * may happen that the same packet matches both rules if
-	 * it arrived at the right moment before 13:00.
+	 * it arrived at the right moment before 13:00, so it would be
+	 * better to check skb->tstamp and set it via __net_timestamp()
+	 * if needed.  This however breaks outgoing packets tx timestamp,
+	 * and causes them to get delayed forever by fq packet scheduler.
 	 */
-	if (skb->tstamp == 0)
-		__net_timestamp((struct sk_buff *)skb);
-
-	stamp = ktime_to_ns(skb->tstamp);
-	stamp = div_s64(stamp, NSEC_PER_SEC);
+	stamp = get_seconds();
 
 	if (info->flags & XT_TIME_LOCAL_TZ)
 		/* Adjust for local timezone */
Signature · 1 line
2.21.0
Acked-by on v1 1/1
#3 lore raw

Re: [PATCH nf] netfilter: never get/set skb->tstamp

Quoted 15 lines · Florian Westphal

setting net.netfilter.nf_conntrack_timestamp=1 breaks xmit with fq scheduler. skb->tstamp might be "refreshed" using ktime_get_real(), but fq expects CLOCK_MONOTONIC.

This patch removes all places in netfilter that check/set skb->tstamp:

1. To fix the bogus "start" time seen with conntrack timestamping for outgoing packets, never use skb->tstamp and always use current time.
2. In nfqueue and nflog, only use skb->tstamp for incoming packets, as determined by current hook (prerouting, input, forward).
3. xt_time has to use system clock as well rather than skb->tstamp. We could still use skb->tstamp for prerouting/input/foward, but I see no advantage to make this conditional.

Quoted 5 lines
Fixes fb420d5d91c1 ("tcp/fq: move back to CLOCK_MONOTONIC")
Cc Eric Dumazet <[email protected]>

---

SGTM, thanks for taking care of this Florian

#4 lore raw

Re: [PATCH nf] netfilter: never get/set skb->tstamp

In reply to Florian Westphal
Quoted 14 lines · Florian Westphal

setting net.netfilter.nf_conntrack_timestamp=1 breaks xmit with fq scheduler. skb->tstamp might be "refreshed" using ktime_get_real(), but fq expects CLOCK_MONOTONIC.

This patch removes all places in netfilter that check/set skb->tstamp:

1. To fix the bogus "start" time seen with conntrack timestamping for

    outgoing packets, never use skb->tstamp and always use current time.

2. In nfqueue and nflog, only use skb->tstamp for incoming packets, as determined by current hook (prerouting, input, forward).
3. xt_time has to use system clock as well rather than skb->tstamp. We could still use skb->tstamp for prerouting/input/foward, but I see no advantage to make this conditional.

Quoted 4 lines
Fixes fb420d5d91c1 ("tcp/fq: move back to CLOCK_MONOTONIC")
Cc Eric Dumazet <[email protected]>

Thanks for quick patch.

Applied it on top of 5.0.8 - wanted to confirm everything seems to be working fine (and timestamps are correct).

maintainer Applied
#5 lore raw

Re: [PATCH nf] netfilter: never get/set skb->tstamp

In reply to Florian Westphal
Quoted 6 lines · Florian Westphal

setting net.netfilter.nf_conntrack_timestamp=1 breaks xmit with fq scheduler. skb->tstamp might be "refreshed" using ktime_get_real(), but fq expects CLOCK_MONOTONIC.

This patch removes all places in netfilter that check/set skb->tstamp:

Quoted 7 lines

1. To fix the bogus "start" time seen with conntrack timestamping for outgoing packets, never use skb->tstamp and always use current time.
2. In nfqueue and nflog, only use skb->tstamp for incoming packets, as determined by current hook (prerouting, input, forward).
3. xt_time has to use system clock as well rather than skb->tstamp. We could still use skb->tstamp for prerouting/input/foward, but I see no advantage to make this conditional.

Applied, thanks Florian.