Skip to contentSkip to conversation

netfilter: revisit conntrack statistics

PATCH In review Florian Westphal 2 participants 6 messages Aug 25, 2020 – Aug 28, 2020

Subsystems (2)
lore
Share & tools
lore

Florian Westphal

author maintainer
#0 lore raw

cover netfilter: revisit conntrack statistics

With recent addition of clash resolution the 'insert_failed' counter has become confusing. Depending on wheter clash resolution is successful, insert_failed increments or both insert_failed and drop increment.

Example (conntrack -S):
[..] insert_failed=15 drop=0 [..] search_restart=268

This means clash resolution worked and the insert_failed increase is harmeless. In case drop is non-zero, things become murky.

It would be better to have a dedicated counter that only increments when clash resolution is successful.

This series revisits conntrack statistics. Counters that do not indicate an error or reside in fast-paths are removed.

With patched kernel and conntrack tool, output looks similar to this during a 'clash resolve' stress test:

[..] insert_failed=9 drop=9 [..] search_restart=123 clash_resolve=3675

Florian Westphal (4):
netfilter: conntrack: do not increment two error counters at same time
netfilter: conntrack: remove ignore stats
netfilter: conntrack: add clash resolution stat counter
netfilter: conntrack: remove unneeded nf_ct_put

 include/linux/netfilter/nf_conntrack_common.h      |    2 -
 include/uapi/linux/netfilter/nfnetlink_conntrack.h |    3 +-
 net/netfilter/nf_conntrack_core.c                  |   25 ++++++++-------------
 net/netfilter/nf_conntrack_netlink.c               |    5 ++--
 net/netfilter/nf_conntrack_standalone.c            |    4 +--
 5 files changed, 18 insertions(+), 21 deletions(-)
author maintainer
#1 lore raw

PATCH 1/4 netfilter: conntrack: do not increment two error counters at same time

The /proc interface for nf_conntrack displays the "error" counter as "icmp_error".

It makes sense to not increment "invalid" when failing to handle an icmp packet since those are special.

For example, its possible for conntrack to see partial and/or fragmented packets inside icmp errors. This should be a separate event and not get mixed with the "invalid" counter.

Likewise, remove the "error" increment for errors from get_l4proto(). After this, the error counter will only increment for errors coming from icmp(v6) packet handling.

---
 net/netfilter/nf_conntrack_core.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
net/netfilter/nf_conntrack_core.c +1 -4
index 5b97d233f89b..3cfbafdff941 100644
@@ -1725,10 +1725,8 @@ nf_conntrack_handle_icmp(struct nf_conn *tmpl,
 	else
 		return NF_ACCEPT;
 
-	if (ret <= 0) {
+	if (ret <= 0)
 		NF_CT_STAT_INC_ATOMIC(state->net, error);
-		NF_CT_STAT_INC_ATOMIC(state->net, invalid);
-	}
 
 	return ret;
 }
@@ -1813,7 +1811,6 @@ nf_conntrack_in(struct sk_buff *skb, const struct nf_hook_state *state)
 	dataoff = get_l4proto(skb, skb_network_offset(skb), state->pf, &protonum);
 	if (dataoff <= 0) {
 		pr_debug("not prepared to track yet or error occurred\n");
-		NF_CT_STAT_INC_ATOMIC(state->net, error);
 		NF_CT_STAT_INC_ATOMIC(state->net, invalid);
 		ret = NF_ACCEPT;
 		goto out;
Signature · 1 line
2.26.2
author maintainer
#2 lore raw

PATCH 2/4 netfilter: conntrack: remove ignore stats

In reply to Florian Westphal

This counter increments when nf_conntrack_in sees a packet that already has a conntrack attached or when the packet is marked as UNTRACKED. Neither is an error.

The former is normal for loopback traffic. The second happens for certain ICMPv6 packets or when nftables/ip(6)tables rules are in place.

In case someone needs to count UNTRACKED packets, or packets that are marked as untracked before conntrack_in this can be done with both nftables and ip(6)tables rules.

---
 include/linux/netfilter/nf_conntrack_common.h      | 1 -
 include/uapi/linux/netfilter/nfnetlink_conntrack.h | 2 +-
 net/netfilter/nf_conntrack_core.c                  | 4 +---
 net/netfilter/nf_conntrack_netlink.c               | 1 -
 net/netfilter/nf_conntrack_standalone.c            | 2 +-
 5 files changed, 3 insertions(+), 7 deletions(-)
include/linux/netfilter/nf_conntrack_common.h +0 -1
index 1db83c931d9c..96b90d7e361f 100644
@@ -8,7 +8,6 @@
 struct ip_conntrack_stat {
 	unsigned int found;
 	unsigned int invalid;
-	unsigned int ignore;
 	unsigned int insert;
 	unsigned int insert_failed;
 	unsigned int drop;
include/uapi/linux/netfilter/nfnetlink_conntrack.h +1 -1
index 262881792671..3e471558da82 100644
@@ -247,7 +247,7 @@ enum ctattr_stats_cpu {
 	CTA_STATS_FOUND,
 	CTA_STATS_NEW,		/* no longer used */
 	CTA_STATS_INVALID,
-	CTA_STATS_IGNORE,
+	CTA_STATS_IGNORE,	/* no longer used */
 	CTA_STATS_DELETE,	/* no longer used */
 	CTA_STATS_DELETE_LIST,	/* no longer used */
 	CTA_STATS_INSERT,
net/netfilter/nf_conntrack_core.c +1 -3
index 3cfbafdff941..a111bcf1b93c 100644
@@ -1800,10 +1800,8 @@ nf_conntrack_in(struct sk_buff *skb, const struct nf_hook_state *state)
 	if (tmpl || ctinfo == IP_CT_UNTRACKED) {
 		/* Previously seen (loopback or untracked)?  Ignore. */
 		if ((tmpl && !nf_ct_is_template(tmpl)) ||
-		     ctinfo == IP_CT_UNTRACKED) {
-			NF_CT_STAT_INC_ATOMIC(state->net, ignore);
+		     ctinfo == IP_CT_UNTRACKED)
 			return NF_ACCEPT;
-		}
 		skb->_nfct = 0;
 	}
 
net/netfilter/nf_conntrack_netlink.c +0 -1
index 832eabecfbdd..c64f23a8f373 100644
@@ -2509,7 +2509,6 @@ ctnetlink_ct_stat_cpu_fill_info(struct sk_buff *skb, u32 portid, u32 seq,
 
 	if (nla_put_be32(skb, CTA_STATS_FOUND, htonl(st->found)) ||
 	    nla_put_be32(skb, CTA_STATS_INVALID, htonl(st->invalid)) ||
-	    nla_put_be32(skb, CTA_STATS_IGNORE, htonl(st->ignore)) ||
 	    nla_put_be32(skb, CTA_STATS_INSERT, htonl(st->insert)) ||
 	    nla_put_be32(skb, CTA_STATS_INSERT_FAILED,
 				htonl(st->insert_failed)) ||
net/netfilter/nf_conntrack_standalone.c +1 -1
index a604f43e3e6b..b673a03624d2 100644
@@ -439,7 +439,7 @@ static int ct_cpu_seq_show(struct seq_file *seq, void *v)
 		   st->found,
 		   0,
 		   st->invalid,
-		   st->ignore,
+		   0,
 		   0,
 		   0,
 		   st->insert,
Signature · 1 line
2.26.2
author maintainer
#3 lore raw

PATCH 3/4 netfilter: conntrack: add clash resolution stat counter

In reply to Florian Westphal

There is a misconception about what "insert_failed" means.

We increment this even when a clash got resolved, so it might not indicate a problem.

Add a dedicated counter for clash resolution and only increment insert_failed if a clash cannot be resolved.

For the old /proc interface, export this in place of an older stat that got removed a while back.
For ctnetlink, export this with a new attribute.

Also correct an outdated comment that implies we add a duplicate tuple -- we only add the (unique) reply direction.

---
 include/linux/netfilter/nf_conntrack_common.h      | 1 +
 include/uapi/linux/netfilter/nfnetlink_conntrack.h | 1 +
 net/netfilter/nf_conntrack_core.c                  | 9 +++++----
 net/netfilter/nf_conntrack_netlink.c               | 4 +++-
 net/netfilter/nf_conntrack_standalone.c            | 2 +-
 5 files changed, 11 insertions(+), 6 deletions(-)
include/linux/netfilter/nf_conntrack_common.h +1 -0
index 96b90d7e361f..0c7d8d1e945d 100644
@@ -10,6 +10,7 @@ struct ip_conntrack_stat {
 	unsigned int invalid;
 	unsigned int insert;
 	unsigned int insert_failed;
+	unsigned int clash_resolve;
 	unsigned int drop;
 	unsigned int early_drop;
 	unsigned int error;
include/uapi/linux/netfilter/nfnetlink_conntrack.h +1 -0
index 3e471558da82..d8484be72fdc 100644
@@ -256,6 +256,7 @@ enum ctattr_stats_cpu {
 	CTA_STATS_EARLY_DROP,
 	CTA_STATS_ERROR,
 	CTA_STATS_SEARCH_RESTART,
+	CTA_STATS_CLASH_RESOLVE,
 	__CTA_STATS_MAX,
 };
 #define CTA_STATS_MAX (__CTA_STATS_MAX - 1)
net/netfilter/nf_conntrack_core.c +5 -4
index a111bcf1b93c..93e77ca0efad 100644
@@ -859,7 +859,6 @@ nf_conntrack_hash_check_insert(struct nf_conn *ct)
 
 out:
 	nf_conntrack_double_unlock(hash, reply_hash);
-	NF_CT_STAT_INC(net, insert_failed);
 	local_bh_enable();
 	return -EEXIST;
 }
@@ -934,7 +933,7 @@ static int __nf_ct_resolve_clash(struct sk_buff *skb,
 		nf_conntrack_put(&loser_ct->ct_general);
 		nf_ct_set(skb, ct, ctinfo);
 
-		NF_CT_STAT_INC(net, insert_failed);
+		NF_CT_STAT_INC(net, clash_resolve);
 		return NF_ACCEPT;
 	}
 
@@ -998,6 +997,8 @@ static int nf_ct_resolve_clash_harder(struct sk_buff *skb, u32 repl_idx)
 
 	hlist_nulls_add_head_rcu(&loser_ct->tuplehash[IP_CT_DIR_REPLY].hnnode,
 				 &nf_conntrack_hash[repl_idx]);
+
+	NF_CT_STAT_INC(net, clash_resolve);
 	return NF_ACCEPT;
 }
 
@@ -1027,10 +1028,10 @@ static int nf_ct_resolve_clash_harder(struct sk_buff *skb, u32 repl_idx)
  *
  * Failing that, the new, unconfirmed conntrack is still added to the table
  * provided that the collision only occurs in the ORIGINAL direction.
- * The new entry will be added after the existing one in the hash list,
+ * The new entry will be added only in the non-clashing REPLY direction,
  * so packets in the ORIGINAL direction will continue to match the existing
  * entry.  The new entry will also have a fixed timeout so it expires --
- * due to the collision, it will not see bidirectional traffic.
+ * due to the collision, it will only see reply traffic.
  *
  * Returns NF_DROP if the clash could not be resolved.
  */
net/netfilter/nf_conntrack_netlink.c +3 -1
index c64f23a8f373..89d99f6dfd0a 100644
@@ -2516,7 +2516,9 @@ ctnetlink_ct_stat_cpu_fill_info(struct sk_buff *skb, u32 portid, u32 seq,
 	    nla_put_be32(skb, CTA_STATS_EARLY_DROP, htonl(st->early_drop)) ||
 	    nla_put_be32(skb, CTA_STATS_ERROR, htonl(st->error)) ||
 	    nla_put_be32(skb, CTA_STATS_SEARCH_RESTART,
-				htonl(st->search_restart)))
+				htonl(st->search_restart)) ||
+	    nla_put_be32(skb, CTA_STATS_CLASH_RESOLVE,
+				htonl(st->clash_resolve)))
 		goto nla_put_failure;
 
 	nlmsg_end(skb, nlh);
net/netfilter/nf_conntrack_standalone.c +1 -1
index b673a03624d2..0ff39740797d 100644
@@ -435,7 +435,7 @@ static int ct_cpu_seq_show(struct seq_file *seq, void *v)
 	seq_printf(seq, "%08x  %08x %08x %08x %08x %08x %08x %08x "
 			"%08x %08x %08x %08x %08x  %08x %08x %08x %08x\n",
 		   nr_conntracks,
-		   0,
+		   st->clash_resolve, /* was: searched */
 		   st->found,
 		   0,
 		   st->invalid,
Signature · 1 line
2.26.2
author maintainer
#4 lore raw

PATCH 4/4 netfilter: conntrack: remove unneeded nf_ct_put

In reply to Florian Westphal

We can delay refcount increment until we reassign the existing entry to the current skb.

A 0 refcount can't happen while the nf_conn object is still in the hash table and parallel mutations are impossible because we hold the bucket lock.

---
 net/netfilter/nf_conntrack_core.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
net/netfilter/nf_conntrack_core.c +3 -4
index 93e77ca0efad..234b7cab37c3 100644
@@ -908,6 +908,7 @@ static void __nf_conntrack_insert_prepare(struct nf_conn *ct)
 		tstamp->start = ktime_get_real_ns();
 }
 
+/* caller must hold locks to prevent concurrent changes */
 static int __nf_ct_resolve_clash(struct sk_buff *skb,
 				 struct nf_conntrack_tuple_hash *h)
 {
@@ -921,13 +922,12 @@ static int __nf_ct_resolve_clash(struct sk_buff *skb,
 	if (nf_ct_is_dying(ct))
 		return NF_DROP;
 
-	if (!atomic_inc_not_zero(&ct->ct_general.use))
-		return NF_DROP;
-
 	if (((ct->status & IPS_NAT_DONE_MASK) == 0) ||
 	    nf_ct_match(ct, loser_ct)) {
 		struct net *net = nf_ct_net(ct);
 
+		nf_conntrack_get(&ct->ct_general);
+
 		nf_ct_acct_merge(ct, ctinfo, loser_ct);
 		nf_ct_add_to_dying_list(loser_ct);
 		nf_conntrack_put(&loser_ct->ct_general);
@@ -937,7 +937,6 @@ static int __nf_ct_resolve_clash(struct sk_buff *skb,
 		return NF_ACCEPT;
 	}
 
-	nf_ct_put(ct);
 	return NF_DROP;
 }
 
Signature · 1 line
2.26.2
#5 lore raw
In reply to Florian Westphal
Quoted 13 lines · Florian Westphal

With recent addition of clash resolution the 'insert_failed' counter has become confusing. Depending on wheter clash resolution is successful, insert_failed increments or both insert_failed and drop increment.

Example (conntrack -S):
[..] insert_failed=15 drop=0 [..] search_restart=268

This means clash resolution worked and the insert_failed increase is harmeless. In case drop is non-zero, things become murky.

It would be better to have a dedicated counter that only increments when clash resolution is successful.

Quoted 7 lines

This series revisits conntrack statistics. Counters that do not indicate an error or reside in fast-paths are removed.

With patched kernel and conntrack tool, output looks similar to this during a 'clash resolve' stress test:

[..] insert_failed=9 drop=9 [..] search_restart=123 clash_resolve=3675

Series applied, thanks.