Skip to contentSkip to conversation

netfilter: conntrack: fix obscure confirmed race

PATCH In review Florian Westphal 2 participants 12 messages Jun 27, 2025 – Jul 16, 2025

Subsystems (3)
lore
Share & tools
lore

Florian Westphal

author maintainer
#0 lore raw

cover netfilter: conntrack: fix obscure confirmed race

We go a crash report pointing at __nf_ct_delete_from_lists. While I've been unable to reproduce this, there appears to be a race, IPS_CONFIRMED bit is set too early and can cause datapath or gc worker to unlink an entry that hasn't been fully initialised.

The last patch is the actual fix, the first 3 patches extend and add a few more conntrack tests to exercise clash resolution for udp.

Florian Westphal (4):
selftests: netfilter: conntrack_resize.sh: extend resize test
selftests: netfilter: add conntrack clash resolution test case
selftests: netfilter: conntrack_resize.sh: also use udpclash tool
netfilter: nf_conntrack: fix crash due to removal of uninitialised
entry

 include/net/netfilter/nf_conntrack.h          |  15 +-
 net/netfilter/nf_conntrack_core.c             |  18 +-
 .../selftests/net/netfilter/.gitignore        |   1 +
 .../testing/selftests/net/netfilter/Makefile  |   3 +
 .../net/netfilter/conntrack_clash.sh          | 175 ++++++++++++++++++
 .../net/netfilter/conntrack_resize.sh         |  97 +++++++++-
 .../selftests/net/netfilter/udpclash.c        | 158 ++++++++++++++++
 7 files changed, 454 insertions(+), 13 deletions(-)
 create mode 100755 tools/testing/selftests/net/netfilter/conntrack_clash.sh
 create mode 100644 tools/testing/selftests/net/netfilter/udpclash.c
Signature · 1 line
2.49.0
author maintainer
#1 lore raw

PATCH 1/4 selftests: netfilter: conntrack_resize.sh: extend resize test

Extend the resize test:
- continuously dump table both via /proc and ctnetlink interfaces while table is resized in a loop.
- if socat is available, send udp packets in additon to ping requests.
- increase/decrease the icmp and udp timeouts while resizes are happening. This makes sure we also exercise the 'ct has expired' check that happens on conntrack lookup.

---
 .../net/netfilter/conntrack_resize.sh         | 80 +++++++++++++++++--
 1 file changed, 75 insertions(+), 5 deletions(-)
tools/testing/selftests/net/netfilter/conntrack_resize.sh +75 -5
index 9e033e80219e..aa1ba07eaf50 100755
@@ -12,6 +12,9 @@ tmpfile=""
 tmpfile_proc=""
 tmpfile_uniq=""
 ret=0
+have_socat=0
+
+socat -h > /dev/null && have_socat=1
 
 insert_count=2000
 [ "$KSFT_MACHINE_SLOW" = "yes" ] && insert_count=400
@@ -123,7 +126,7 @@ ctflush() {
         done
 }
 
-ctflood()
+ct_pingflood()
 {
 	local ns="$1"
 	local duration="$2"
@@ -152,6 +155,28 @@ ctflood()
 	wait
 }
 
+ct_udpflood()
+{
+	local ns="$1"
+	local duration="$2"
+	local now=$(date +%s)
+	local end=$((now + duration))
+
+	[ $have_socat -ne "1" ] && return
+
+        while [ $now -lt $end ]; do
+ip netns exec "$ns" bash<<"EOF"
+	for i in $(seq 1 100);do
+		dport=$(((RANDOM%65536)+1))
+
+		echo bar | socat -u STDIN UDP:"127.0.0.1:$dport" &
+	done > /dev/null 2>&1
+	wait
+EOF
+		now=$(date +%s)
+	done
+}
+
 # dump to /dev/null.  We don't want dumps to cause infinite loops
 # or use-after-free even when conntrack table is altered while dumps
 # are in progress.
@@ -169,6 +194,48 @@ ct_nulldump()
 	wait
 }
 
+ct_nulldump_loop()
+{
+	local ns="$1"
+	local duration="$2"
+	local now=$(date +%s)
+	local end=$((now + duration))
+
+        while [ $now -lt $end ]; do
+		ct_nulldump "$ns"
+		sleep $((RANDOM%2))
+		now=$(date +%s)
+	done
+}
+
+change_timeouts()
+{
+	local ns="$1"
+	local r1=$((RANDOM%2))
+	local r2=$((RANDOM%2))
+
+	[ "$r1" -eq 1 ] && ip netns exec "$ns" sysctl -q net.netfilter.nf_conntrack_icmp_timeout=$((RANDOM%5))
+	[ "$r2" -eq 1 ] && ip netns exec "$ns" sysctl -q net.netfilter.nf_conntrack_udp_timeout=$((RANDOM%5))
+}
+
+ct_change_timeouts_loop()
+{
+	local ns="$1"
+	local duration="$2"
+	local now=$(date +%s)
+	local end=$((now + duration))
+
+        while [ $now -lt $end ]; do
+		change_timeouts "$ns"
+		sleep $((RANDOM%2))
+		now=$(date +%s)
+	done
+
+	# restore defaults
+	ip netns exec "$ns" sysctl -q net.netfilter.nf_conntrack_icmp_timeout=30
+	ip netns exec "$ns" sysctl -q net.netfilter.nf_conntrack_udp_timeout=30
+}
+
 check_taint()
 {
 	local tainted_then="$1"
@@ -198,10 +265,13 @@ insert_flood()
 
 	r=$((RANDOM%$insert_count))
 
-	ctflood "$n" "$timeout" "floodresize" &
+	ct_pingflood "$n" "$timeout" "floodresize" &
+	ct_udpflood "$n" "$timeout" &
+
 	insert_ctnetlink "$n" "$r" &
 	ctflush "$n" "$timeout" &
-	ct_nulldump "$n" &
+	ct_nulldump_loop "$n" "$timeout" &
+	ct_change_timeouts_loop "$n" "$timeout" &
 
 	wait
 }
@@ -306,7 +376,7 @@ test_dump_all()
 
 	ip netns exec "$nsclient1" sysctl -q net.netfilter.nf_conntrack_icmp_timeout=3600
 
-	ctflood "$nsclient1" $timeout "dumpall" &
+	ct_pingflood "$nsclient1" $timeout "dumpall" &
 	insert_ctnetlink "$nsclient2" $insert_count
 
 	wait
@@ -368,7 +438,7 @@ test_conntrack_disable()
 	ct_flush_once "$nsclient1"
 	ct_flush_once "$nsclient2"
 
-	ctflood "$nsclient1" "$timeout" "conntrack disable"
+	ct_pingflood "$nsclient1" "$timeout" "conntrack disable"
 	ip netns exec "$nsclient2" ping -q -c 1 127.0.0.1 >/dev/null 2>&1
 
 	# Disabled, should not have picked up any connection.
Signature · 1 line
2.49.0
author maintainer
#2 lore raw

PATCH 2/4 selftests: netfilter: add conntrack clash resolution test case

In reply to Florian Westphal

Add a dedicated test to exercise conntrack clash resolution path. Test program emits 128 identical udp packets in parallel, then reads back replies from socat echo server.

Also check (via conntrack -S) that the clash path was hit at least once. Due to the racy nature of the test its possible that despite the threaded program all packets were processed in-order or on same cpu, emit a SKIP warning in this case.

Two tests are added:
- one to test the simpler, non-nat case
- one to exercise clash resolution where packets might have different nat transformations attached to them.

---
 .../selftests/net/netfilter/.gitignore        |   1 +
 .../testing/selftests/net/netfilter/Makefile  |   3 +
 .../net/netfilter/conntrack_clash.sh          | 175 ++++++++++++++++++
 .../selftests/net/netfilter/udpclash.c        | 158 ++++++++++++++++
 4 files changed, 337 insertions(+)
 create mode 100755 tools/testing/selftests/net/netfilter/conntrack_clash.sh
 create mode 100644 tools/testing/selftests/net/netfilter/udpclash.c
tools/testing/selftests/net/netfilter/.gitignore +1 -0
index 64c4f8d9aa6c..5d2be9a00627 100644
@@ -5,3 +5,4 @@ conntrack_dump_flush
 conntrack_reverse_clash
 sctp_collision
 nf_queue
+udpclash
tools/testing/selftests/net/netfilter/Makefile +3 -0
index e9b2f553588d..a98ed892f55f 100644
@@ -15,6 +15,7 @@ TEST_PROGS += conntrack_tcp_unreplied.sh
 TEST_PROGS += conntrack_resize.sh
 TEST_PROGS += conntrack_sctp_collision.sh
 TEST_PROGS += conntrack_vrf.sh
+TEST_PROGS += conntrack_clash.sh
 TEST_PROGS += conntrack_reverse_clash.sh
 TEST_PROGS += ipvs.sh
 TEST_PROGS += nf_conntrack_packetdrill.sh
@@ -44,6 +45,7 @@ TEST_GEN_FILES += connect_close nf_queue
 TEST_GEN_FILES += conntrack_dump_flush
 TEST_GEN_FILES += conntrack_reverse_clash
 TEST_GEN_FILES += sctp_collision
+TEST_GEN_FILES += udpclash
 
 include ../../lib.mk
 
@@ -52,6 +54,7 @@ $(OUTPUT)/nf_queue: LDLIBS += $(MNL_LDLIBS)
 
 $(OUTPUT)/conntrack_dump_flush: CFLAGS += $(MNL_CFLAGS)
 $(OUTPUT)/conntrack_dump_flush: LDLIBS += $(MNL_LDLIBS)
+$(OUTPUT)/udpclash: LDLIBS += -lpthread
 
 TEST_FILES := lib.sh
 TEST_FILES += packetdrill
tools/testing/selftests/net/netfilter/conntrack_clash.sh new file +175 -0

175 lines not on this page · raw message

tools/testing/selftests/net/netfilter/udpclash.c new file +158 -0

158 lines not on this page · raw message

Signature · 1 line
2.49.0
author maintainer
#3 lore raw

PATCH 3/4 selftests: netfilter: conntrack_resize.sh: also use udpclash tool

In reply to Florian Westphal

Previous patch added a new clash resolution test case. Also use this during conntrack resize stress test in addition to icmp ping flood.

---
 .../selftests/net/netfilter/conntrack_resize.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
tools/testing/selftests/net/netfilter/conntrack_resize.sh +17 -0
index aa1ba07eaf50..788cd56ea4a0 100755
@@ -177,6 +177,22 @@ EOF
 	done
 }
 
+ct_udpclash()
+{
+	local ns="$1"
+	local duration="$2"
+	local now=$(date +%s)
+	local end=$((now + duration))
+
+	[ -x udpclash ] || return
+
+        while [ $now -lt $end ]; do
+		ip netns exec "$ns" ./udpclash 127.0.0.1 $((RANDOM%65536)) > /dev/null 2>&1
+
+		now=$(date +%s)
+	done
+}
+
 # dump to /dev/null.  We don't want dumps to cause infinite loops
 # or use-after-free even when conntrack table is altered while dumps
 # are in progress.
@@ -267,6 +283,7 @@ insert_flood()
 
 	ct_pingflood "$n" "$timeout" "floodresize" &
 	ct_udpflood "$n" "$timeout" &
+	ct_udpclash "$n" "$timeout" &
 
 	insert_ctnetlink "$n" "$r" &
 	ctflush "$n" "$timeout" &
Signature · 1 line
2.49.0
author maintainer
#4 lore raw

PATCH 4/4 netfilter: nf_conntrack: fix crash due to removal of uninitialised entry

In reply to Florian Westphal

A crash in conntrack was reported while trying to unlink the conntrack entry from the hash bucket list:

    [exception RIP: __nf_ct_delete_from_lists+172]
    [..]
 #7 [ff539b5a2b043aa0] nf_ct_delete at ffffffffc124d421 [nf_conntrack]
 #8 [ff539b5a2b043ad0] nf_ct_gc_expired at ffffffffc124d999 [nf_conntrack]
 #9 [ff539b5a2b043ae0] __nf_conntrack_find_get at ffffffffc124efbc [nf_conntrack]
    [..]

The nf_conn struct is marked as allocated from slab but appears to be in a partially initialised state:

ct hlist pointer is garbage; looks like the ct hash value (hence crash).
ct->status is equal to IPS_CONFIRMED|IPS_DYING, which is expected ct->timeout is 30000 (=30s), which is unexpected.

Everything else looks like normal udp conntrack entry. If we ignore ct->status and pretend its 0, the entry matches those that are newly allocated but not yet inserted into the hash:
- ct hlist pointers are overloaded and store/cache the raw tuple hash
- ct->timeout matches the relative time expected for a new udp flow rather than the absolute 'jiffies' value.

If it were not for the presence of IPS_CONFIRMED, __nf_conntrack_find_get() would have skipped the entry.

Theory is that we did hit following race:

cpu x cpu y cpu z
found entry E found entry E
E is expired <preemption>
nf_ct_delete()
return E to rcu slab
init_conntrack
E is re-inited, ct->status set to 0 reply tuplehash hnnode.pprev stores hash value.

cpu y found E right before it was deleted on cpu x. E is now re-inited on cpu z. cpu y was preempted before checking for expiry and/or confirm bit.

					->refcnt set to 1
					E now owned by skb
					->timeout set to 30000

If cpu y were to resume now, it would observe E as expired but would skip E due to missing CONFIRMED bit.

nf_conntrack_confirm gets called
sets: ct->status |= CONFIRMED This is wrong: E is not yet added to hashtable.

cpu y resumes, it observes E as expired but CONFIRMED:

			<resumes>
			nf_ct_expired()
			 -> yes (ct->timeout is 30s)
			confirmed bit set.

cpu y will try to delete E from the hashtable:

			nf_ct_delete() -> set DYING bit
			__nf_ct_delete_from_lists

Even this scenario doesn't guarantee a crash:
cpu z still holds the table bucket lock(s) so y blocks:

wait for spinlock held by z

CONFIRMED is set but there is no guarantee ct will be added to hash:
"chaintoolong" or "clash resolution" logic both skip the insert step. reply hnnode.pprev still stores the hash value.

unlocks spinlock return NF_DROP
<unblocks, then
crashes on hlist_nulls_del_rcu pprev>

In case CPU z does insert the entry into the hashtable, cpu y will unlink E again right away but no crash occurs.

Without 'cpu y' race, 'garbage' hlist is of no consequence:
ct refcnt remains at 1, eventually skb will be free'd and E gets destroyed via: nf_conntrack_put -> nf_conntrack_destroy -> nf_ct_destroy.

To resolve this, move the IPS_CONFIRMED assignment after the table insertion but before the unlock.

It doesn't matter if other CPUs can observe a newly inserted entry right before the CONFIRMED bit was set:

Such event cannot be distinguished from above "E is the old incarnation"
case: the entry will be skipped.

Also change nf_ct_should_gc() to first check the confirmed bit.

The gc sequence is:
1. Check if entry has expired, if not skip to next entry
2. Obtain a reference to the expired entry.
3. Call nf_ct_should_gc() to double-check step 1.

nf_ct_should_gc() is thus called only for entries that already failed an expiry check. After this patch, once the confirmed bit check passes ct->timeout has been altered to reflect the absolute 'best before' date instead of a relative time. Step 3 will therefore not remove the entry.

Without this change to nf_ct_should_gc() we could still get this sequence:

1. Check if entry has expired.
2. Obtain a reference.
3. Call nf_ct_should_gc() to double-check step 1:
4 - entry is still observed as expired
5 - meanwhile, ct->timeout is corrected to absolute value on other CPU
and confirm bit gets set
6 - confirm bit is seen
7 - valid entry is removed again

First do check 6), then 4) so the gc expiry check always picks up either confirmed bit unset (entry gets skipped) or expiry re-check failure for re-inited conntrack objects.

This change cannot be backported to releases before 5.19. Without commit 8a75a2c17410 ("netfilter: conntrack: remove unconfirmed list") |= IPS_CONFIRMED line cannot be moved without further changes.

Fixes 1397af5bfd7d ("netfilter: conntrack: remove the percpu dying list")
---
 include/net/netfilter/nf_conntrack.h | 15 +++++++++++++--
 net/netfilter/nf_conntrack_core.c    | 18 ++++++++++++------
 2 files changed, 25 insertions(+), 8 deletions(-)
include/net/netfilter/nf_conntrack.h +13 -2
index 3f02a45773e8..ca26274196b9 100644
@@ -306,8 +306,19 @@ static inline bool nf_ct_is_expired(const struct nf_conn *ct)
 /* use after obtaining a reference count */
 static inline bool nf_ct_should_gc(const struct nf_conn *ct)
 {
-	return nf_ct_is_expired(ct) && nf_ct_is_confirmed(ct) &&
-	       !nf_ct_is_dying(ct);
+	if (!nf_ct_is_confirmed(ct))
+		return false;
+
+	/* load ct->timeout after is_confirmed() test.
+	 * Pairs with __nf_conntrack_confirm() which:
+	 * 1. Increases ct->timeout value
+	 * 2. Inserts ct into rcu hlist
+	 * 3. Sets the confirmed bit
+	 * 4. Unlocks the hlist lock
+	 */
+	smp_acquire__after_ctrl_dep();
+
+	return nf_ct_is_expired(ct) && !nf_ct_is_dying(ct);
 }
 
 #define	NF_CT_DAY	(86400 * HZ)
net/netfilter/nf_conntrack_core.c +12 -6
index 201d3c4ec623..442a972a03d7 100644
@@ -1124,6 +1124,7 @@ static int nf_ct_resolve_clash_harder(struct sk_buff *skb, u32 repl_idx)
 
 	hlist_nulls_add_head_rcu(&loser_ct->tuplehash[IP_CT_DIR_REPLY].hnnode,
 				 &nf_conntrack_hash[repl_idx]);
+	loser_ct->status |= IPS_CONFIRMED;
 
 	NF_CT_STAT_INC(net, clash_resolve);
 	return NF_ACCEPT;
@@ -1260,8 +1261,6 @@ __nf_conntrack_confirm(struct sk_buff *skb)
 	 * user context, else we insert an already 'dead' hash, blocking
 	 * further use of that particular connection -JM.
 	 */
-	ct->status |= IPS_CONFIRMED;
-
 	if (unlikely(nf_ct_is_dying(ct))) {
 		NF_CT_STAT_INC(net, insert_failed);
 		goto dying;
@@ -1293,7 +1292,7 @@ __nf_conntrack_confirm(struct sk_buff *skb)
 		}
 	}
 
-	/* Timer relative to confirmation time, not original
+	/* Timeout is relative to confirmation time, not original
 	   setting time, otherwise we'd get timer wrap in
 	   weird delay cases. */
 	ct->timeout += nfct_time_stamp;
@@ -1301,11 +1300,18 @@ __nf_conntrack_confirm(struct sk_buff *skb)
 	__nf_conntrack_insert_prepare(ct);
 
 	/* Since the lookup is lockless, hash insertion must be done after
-	 * starting the timer and setting the CONFIRMED bit. The RCU barriers
-	 * guarantee that no other CPU can find the conntrack before the above
-	 * stores are visible.
+	 * setting ct->timeout. The RCU barriers guarantee that no other CPU
+	 * can find the conntrack before the above stores are visible.
 	 */
 	__nf_conntrack_hash_insert(ct, hash, reply_hash);
+
+	/* IPS_CONFIRMED unset means 'ct not (yet) in hash', conntrack lookups
+	 * skip entries that lack this bit.  This happens when a CPU is looking
+	 * at a stale entry that is being recycled due to SLAB_TYPESAFE_BY_RCU
+	 * or when another CPU encounters this entry right after the insertion
+	 * but before the set-confirm-bit below.
+	 */
+	ct->status |= IPS_CONFIRMED;
 	nf_conntrack_double_unlock(hash, reply_hash);
 	local_bh_enable();
 
Signature · 1 line
2.49.0
#5 lore raw

Re: [PATCH nf 4/4] netfilter: nf_conntrack: fix crash due to removal of uninitialised entry

Hi FLorian,

Thanks for the description, this scenario is esoteric.

Is this bug fully reproducible?

More questions below.

Quoted 202 lines · Florian Westphal

A crash in conntrack was reported while trying to unlink the conntrack entry from the hash bucket list:

    [exception RIP: __nf_ct_delete_from_lists+172]
    [..]
 #7 [ff539b5a2b043aa0] nf_ct_delete at ffffffffc124d421 [nf_conntrack]
 #8 [ff539b5a2b043ad0] nf_ct_gc_expired at ffffffffc124d999 [nf_conntrack]
 #9 [ff539b5a2b043ae0] __nf_conntrack_find_get at ffffffffc124efbc [nf_conntrack]
    [..]

The nf_conn struct is marked as allocated from slab but appears to be in a partially initialised state:

ct hlist pointer is garbage; looks like the ct hash value (hence crash).
ct->status is equal to IPS_CONFIRMED|IPS_DYING, which is expected ct->timeout is 30000 (=30s), which is unexpected.

Everything else looks like normal udp conntrack entry. If we ignore ct->status and pretend its 0, the entry matches those that are newly allocated but not yet inserted into the hash:
- ct hlist pointers are overloaded and store/cache the raw tuple hash
- ct->timeout matches the relative time expected for a new udp flow rather than the absolute 'jiffies' value.

If it were not for the presence of IPS_CONFIRMED, __nf_conntrack_find_get() would have skipped the entry.

Theory is that we did hit following race:

cpu x cpu y cpu z
found entry E found entry E
E is expired <preemption>
nf_ct_delete()
return E to rcu slab
init_conntrack
E is re-inited, ct->status set to 0 reply tuplehash hnnode.pprev stores hash value.

cpu y found E right before it was deleted on cpu x. E is now re-inited on cpu z. cpu y was preempted before checking for expiry and/or confirm bit.

					->refcnt set to 1
					E now owned by skb
					->timeout set to 30000

If cpu y were to resume now, it would observe E as expired but would skip E due to missing CONFIRMED bit.

nf_conntrack_confirm gets called
sets: ct->status |= CONFIRMED This is wrong: E is not yet added to hashtable.

cpu y resumes, it observes E as expired but CONFIRMED:

			<resumes>
			nf_ct_expired()
			 -> yes (ct->timeout is 30s)
			confirmed bit set.

cpu y will try to delete E from the hashtable:

			nf_ct_delete() -> set DYING bit
			__nf_ct_delete_from_lists

Even this scenario doesn't guarantee a crash:
cpu z still holds the table bucket lock(s) so y blocks:

wait for spinlock held by z

CONFIRMED is set but there is no guarantee ct will be added to hash:
"chaintoolong" or "clash resolution" logic both skip the insert step. reply hnnode.pprev still stores the hash value.

unlocks spinlock return NF_DROP
<unblocks, then
crashes on hlist_nulls_del_rcu pprev>

In case CPU z does insert the entry into the hashtable, cpu y will unlink E again right away but no crash occurs.

Without 'cpu y' race, 'garbage' hlist is of no consequence:
ct refcnt remains at 1, eventually skb will be free'd and E gets destroyed via: nf_conntrack_put -> nf_conntrack_destroy -> nf_ct_destroy.

To resolve this, move the IPS_CONFIRMED assignment after the table insertion but before the unlock.

It doesn't matter if other CPUs can observe a newly inserted entry right before the CONFIRMED bit was set:

Such event cannot be distinguished from above "E is the old incarnation"
case: the entry will be skipped.

Also change nf_ct_should_gc() to first check the confirmed bit.

The gc sequence is:
1. Check if entry has expired, if not skip to next entry
2. Obtain a reference to the expired entry.
3. Call nf_ct_should_gc() to double-check step 1.

nf_ct_should_gc() is thus called only for entries that already failed an expiry check. After this patch, once the confirmed bit check passes ct->timeout has been altered to reflect the absolute 'best before' date instead of a relative time. Step 3 will therefore not remove the entry.

Without this change to nf_ct_should_gc() we could still get this sequence:

1. Check if entry has expired.
2. Obtain a reference.
3. Call nf_ct_should_gc() to double-check step 1:
4 - entry is still observed as expired
5 - meanwhile, ct->timeout is corrected to absolute value on other CPU
and confirm bit gets set
6 - confirm bit is seen
7 - valid entry is removed again

First do check 6), then 4) so the gc expiry check always picks up either confirmed bit unset (entry gets skipped) or expiry re-check failure for re-inited conntrack objects.

This change cannot be backported to releases before 5.19. Without commit 8a75a2c17410 ("netfilter: conntrack: remove unconfirmed list") |= IPS_CONFIRMED line cannot be moved without further changes.

Fixes 1397af5bfd7d ("netfilter: conntrack: remove the percpu dying list")
---
 include/net/netfilter/nf_conntrack.h | 15 +++++++++++++--
 net/netfilter/nf_conntrack_core.c    | 18 ++++++++++++------
 2 files changed, 25 insertions(+), 8 deletions(-)
include/net/netfilter/nf_conntrack.h +13 -2
index 3f02a45773e8..ca26274196b9 100644
@@ -306,8 +306,19 @@ static inline bool nf_ct_is_expired(const struct nf_conn *ct)
 /* use after obtaining a reference count */
 static inline bool nf_ct_should_gc(const struct nf_conn *ct)
 {
-	return nf_ct_is_expired(ct) && nf_ct_is_confirmed(ct) &&
-	       !nf_ct_is_dying(ct);
+	if (!nf_ct_is_confirmed(ct))
+		return false;
+
+	/* load ct->timeout after is_confirmed() test.
+	 * Pairs with __nf_conntrack_confirm() which:
+	 * 1. Increases ct->timeout value
+	 * 2. Inserts ct into rcu hlist
+	 * 3. Sets the confirmed bit
+	 * 4. Unlocks the hlist lock
+	 */
+	smp_acquire__after_ctrl_dep();
+
+	return nf_ct_is_expired(ct) && !nf_ct_is_dying(ct);
 }
 
 #define	NF_CT_DAY	(86400 * HZ)
net/netfilter/nf_conntrack_core.c +4 -6
index 201d3c4ec623..442a972a03d7 100644
@@ -1124,6 +1124,7 @@ static int nf_ct_resolve_clash_harder(struct sk_buff *skb, u32 repl_idx)
 
 	hlist_nulls_add_head_rcu(&loser_ct->tuplehash[IP_CT_DIR_REPLY].hnnode,
 				 &nf_conntrack_hash[repl_idx]);
+	loser_ct->status |= IPS_CONFIRMED;
 
 	NF_CT_STAT_INC(net, clash_resolve);
 	return NF_ACCEPT;
@@ -1260,8 +1261,6 @@ __nf_conntrack_confirm(struct sk_buff *skb)
 	 * user context, else we insert an already 'dead' hash, blocking
 	 * further use of that particular connection -JM.
 	 */
-	ct->status |= IPS_CONFIRMED;
-
 	if (unlikely(nf_ct_is_dying(ct))) {
 		NF_CT_STAT_INC(net, insert_failed);
 		goto dying;
@@ -1293,7 +1292,7 @@ __nf_conntrack_confirm(struct sk_buff *skb)
 		}
 	}
 
-	/* Timer relative to confirmation time, not original
+	/* Timeout is relative to confirmation time, not original
 	   setting time, otherwise we'd get timer wrap in
 	   weird delay cases. */
 	ct->timeout += nfct_time_stamp;
@@ -1301,11 +1300,18 @@ __nf_conntrack_confirm(struct sk_buff *skb)
 	__nf_conntrack_insert_prepare(ct);
 
 	/* Since the lookup is lockless, hash insertion must be done after
-	 * starting the timer and setting the CONFIRMED bit. The RCU barriers
-	 * guarantee that no other CPU can find the conntrack before the above
-	 * stores are visible.
+	 * setting ct->timeout. The RCU barriers guarantee that no other CPU
+	 * can find the conntrack before the above stores are visible.

net/netfilter/nf_conntrack_core.c __nf_conntrack_confirm(struct sk_buff *skb) · line 1314

Quoted 10 lines
net/netfilter/nf_conntrack_core.c +8 -0
 	 */
 	__nf_conntrack_hash_insert(ct, hash, reply_hash);
+
+	/* IPS_CONFIRMED unset means 'ct not (yet) in hash', conntrack lookups
+	 * skip entries that lack this bit.  This happens when a CPU is looking
+	 * at a stale entry that is being recycled due to SLAB_TYPESAFE_BY_RCU
+	 * or when another CPU encounters this entry right after the insertion
+	 * but before the set-confirm-bit below.
+	 */
+	ct->status |= IPS_CONFIRMED;

My understanding is that this bit setting can still be reordered.

Quoted 7 lines
net/netfilter/nf_conntrack_core.c +0 -0
 	nf_conntrack_double_unlock(hash, reply_hash);
 	local_bh_enable();
 
Signature · 1 line
2.49.0
author maintainer
#6 lore raw

Re: [PATCH nf 4/4] netfilter: nf_conntrack: fix crash due to removal of uninitialised entry

Quoted 3 lines · Pablo Neira Ayuso

Thanks for the description, this scenario is esoteric.

Is this bug fully reproducible?

No. Unicorn. Only happened once.
Everything is based off reading the backtrace and vmcore.

Quoted 9 lines

net/netfilter/nf_conntrack_core.c __nf_conntrack_confirm(struct sk_buff *skb) · line 1314

Quoted 7 lines
net/netfilter/nf_conntrack_core.c +7 -0
+	/* IPS_CONFIRMED unset means 'ct not (yet) in hash', conntrack lookups
+	 * skip entries that lack this bit.  This happens when a CPU is looking
+	 * at a stale entry that is being recycled due to SLAB_TYPESAFE_BY_RCU
+	 * or when another CPU encounters this entry right after the insertion
+	 * but before the set-confirm-bit below.
+	 */
+	ct->status |= IPS_CONFIRMED;

My understanding is that this bit setting can still be reordered.

You mean it could be moved before hlist_add? So this is needed?

- ct->status |= IPS_CONFIRMED;
+ smp_mb__before_atomic();
+ set_bit(IPS_CONFIRMED_BIT, &ct->status) ?

I can send a v2 with this change.

#7 lore raw

Re: [PATCH nf 4/4] netfilter: nf_conntrack: fix crash due to removal of uninitialised entry

Quoted 7 lines · Florian Westphal
Quoted 3 lines · Pablo Neira Ayuso

Thanks for the description, this scenario is esoteric.

Is this bug fully reproducible?

No. Unicorn. Only happened once.
Everything is based off reading the backtrace and vmcore.

I guess this needs a chaos money to trigger this bug. Else, can we try to catch this unicorn again?

I would push 1/4 and 3/4 to nf.git to start with. Unless you are 100% sure this fix is needed.

Quoted 17 lines
Quoted 9 lines

net/netfilter/nf_conntrack_core.c __nf_conntrack_confirm(struct sk_buff *skb) · line 1314

Quoted 7 lines
net/netfilter/nf_conntrack_core.c +7 -0
+	/* IPS_CONFIRMED unset means 'ct not (yet) in hash', conntrack lookups
+	 * skip entries that lack this bit.  This happens when a CPU is looking
+	 * at a stale entry that is being recycled due to SLAB_TYPESAFE_BY_RCU
+	 * or when another CPU encounters this entry right after the insertion
+	 * but before the set-confirm-bit below.
+	 */
+	ct->status |= IPS_CONFIRMED;

My understanding is that this bit setting can still be reordered.

You mean it could be moved before hlist_add? So this is needed?

- ct->status |= IPS_CONFIRMED;
+ smp_mb__before_atomic();
+ set_bit(IPS_CONFIRMED_BIT, &ct->status) ?

I can send a v2 with this change.

author maintainer
#8 lore raw

Re: [PATCH nf 4/4] netfilter: nf_conntrack: fix crash due to removal of uninitialised entry

Quoted 10 lines · Pablo Neira Ayuso
Quoted 7 lines · Florian Westphal
Quoted 3 lines · Pablo Neira Ayuso

Thanks for the description, this scenario is esoteric.

Is this bug fully reproducible?

No. Unicorn. Only happened once.
Everything is based off reading the backtrace and vmcore.

I guess this needs a chaos money to trigger this bug. Else, can we try to catch this unicorn again?

I would not hold my breath. But I don't see anything that prevents the race described in 4/4, and all the things match in the vmcore, including increment of clash resolution counter. If you think its too perfect then ok, we can keep 4/4 back until someone else reports this problem again.

Quoted 1 line

I would push 1/4 and 3/4 to nf.git to start with. Unless you are 100% sure this fix is needed.

3/4 needs 2/4 present as well. I can then resend 4/4 then with the

Quoted 3 lines
Quoted 3 lines

- ct->status |= IPS_CONFIRMED;
+ smp_mb__before_atomic();
+ set_bit(IPS_CONFIRMED_BIT, &ct->status) ?

change.

#9 lore raw

Re: [PATCH nf 4/4] netfilter: nf_conntrack: fix crash due to removal of uninitialised entry

Quoted 10 lines · Florian Westphal
Quoted 9 lines · Pablo Neira Ayuso
Quoted 7 lines · Florian Westphal
Quoted 3 lines · Pablo Neira Ayuso

Thanks for the description, this scenario is esoteric.

Is this bug fully reproducible?

No. Unicorn. Only happened once.
Everything is based off reading the backtrace and vmcore.

Quoted 7 lines
Quoted 1 line

I guess this needs a chaos money to trigger this bug. Else, can we try to catch this unicorn again?

I would not hold my breath. But I don't see anything that prevents the race described in 4/4, and all the things match in the vmcore, including increment of clash resolution counter. If you think its too perfect then ok, we can keep 4/4 back until someone else reports this problem again.

Hm, I think your sequence is possible, it is the SLAB_TYPESAFE_BY_RCU rule that allows for this to occur.

Could this rare sequence still happen?

cpu x                   cpu y                   cpu z
 found entry E          found entry E
 E is expired           <preemption>
 nf_ct_delete()
 return E to rcu slab
                                        init_conntrack
                                        <preemption>     NOTE: ct->status not yet set to zero

cpu y resumes, it observes E as expired but CONFIRMED:

                        <resumes>
                        nf_ct_expired()
                         -> yes (ct->timeout is 30s)
                        confirmed bit set.
Quoted 3 lines
Quoted 1 line

I would push 1/4 and 3/4 to nf.git to start with. Unless you are 100% sure this fix is needed.

3/4 needs 2/4 present as well. I can then resend 4/4 then with the

Right, I accidentally skipped that test, it should be also included.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

- ct->status |= IPS_CONFIRMED;
+ smp_mb__before_atomic();
+ set_bit(IPS_CONFIRMED_BIT, &ct->status) ?

change.

If the status bit is used to synchronize the different threads, I agree this needs to be set_bit(). But I am not sure yet this is sufficient yet.

Thanks.

author maintainer
#10 lore raw

Re: [PATCH nf 4/4] netfilter: nf_conntrack: fix crash due to removal of uninitialised entry

Quoted 32 lines · Pablo Neira Ayuso
Quoted 10 lines · Florian Westphal
Quoted 9 lines · Pablo Neira Ayuso
Quoted 7 lines · Florian Westphal
Quoted 3 lines · Pablo Neira Ayuso

Thanks for the description, this scenario is esoteric.

Is this bug fully reproducible?

No. Unicorn. Only happened once.
Everything is based off reading the backtrace and vmcore.

Quoted 7 lines
Quoted 1 line

I guess this needs a chaos money to trigger this bug. Else, can we try to catch this unicorn again?

I would not hold my breath. But I don't see anything that prevents the race described in 4/4, and all the things match in the vmcore, including increment of clash resolution counter. If you think its too perfect then ok, we can keep 4/4 back until someone else reports this problem again.

Hm, I think your sequence is possible, it is the SLAB_TYPESAFE_BY_RCU rule that allows for this to occur.

Could this rare sequence still happen?

cpu x                   cpu y                   cpu z
 found entry E          found entry E
 E is expired           <preemption>
 nf_ct_delete()
 return E to rcu slab
                                        init_conntrack
                                        <preemption>     NOTE: ct->status not yet set to zero
Quoted 5 lines

cpu y resumes, it observes E as expired but CONFIRMED:

                        <resumes>
                        nf_ct_expired()
                         -> yes (ct->timeout is 30s)
                        confirmed bit set.

Yes, that can happen, but then the refcount can't be incremented as its 0 (-> entry is skipped). If its nonzero but the object was returned by the kmem cache we have a different kind of bug (free with refcount > 0), or use-after-free.

#11 lore raw

Re: [PATCH nf 4/4] netfilter: nf_conntrack: fix crash due to removal of uninitialised entry

Quoted 33 lines · Florian Westphal
Quoted 32 lines · Pablo Neira Ayuso
Quoted 10 lines · Florian Westphal
Quoted 9 lines · Pablo Neira Ayuso
Quoted 7 lines · Florian Westphal
Quoted 3 lines · Pablo Neira Ayuso

Thanks for the description, this scenario is esoteric.

Is this bug fully reproducible?

No. Unicorn. Only happened once.
Everything is based off reading the backtrace and vmcore.

Quoted 7 lines
Quoted 1 line

I guess this needs a chaos money to trigger this bug. Else, can we try to catch this unicorn again?

I would not hold my breath. But I don't see anything that prevents the race described in 4/4, and all the things match in the vmcore, including increment of clash resolution counter. If you think its too perfect then ok, we can keep 4/4 back until someone else reports this problem again.

Hm, I think your sequence is possible, it is the SLAB_TYPESAFE_BY_RCU rule that allows for this to occur.

Could this rare sequence still happen?

cpu x                   cpu y                   cpu z
 found entry E          found entry E
 E is expired           <preemption>
 nf_ct_delete()
 return E to rcu slab
                                        init_conntrack
                                        <preemption>     NOTE: ct->status not yet set to zero
Quoted 8 lines
Quoted 5 lines

cpu y resumes, it observes E as expired but CONFIRMED:

                        <resumes>
                        nf_ct_expired()
                         -> yes (ct->timeout is 30s)
                        confirmed bit set.

Yes, that can happen, but then the refcount can't be incremented as its 0 (-> entry is skipped).

Right, refcount zero prevents it.

static void nf_ct_gc_expired(struct nf_conn *ct)

{
        if (!refcount_inc_not_zero(&ct->ct_general.use))
                return;
Quoted 3 lines

If its nonzero but the object was returned
by the kmem cache we have a different kind of bug (free with refcount > 0), or use-after-free.

OK, thanks for explaining, use set_bit() and post v2.