Skip to contentSkip to conversation

tcp: tcp_metrics: use the socket netns, not the dst device netns

PATCH Applied Eric Dumazet 5 participants 7 messages Oct 16 – Oct 20

Subsystems (3)
lore
Share & tools
lore

Eric Dumazet

author maintainer
#0 lore raw

PATCH 1/1 tcp: tcp_metrics: use the socket netns, not the dst device netns

tcp_get_metrics(), __tcp_get_metrics_req() and tcpm_new() look up and create metrics entries in the netns of the cached dst device rather than in the netns of the socket.

These usually match, but not after the device has been moved to another netns or unregistered: a socket can keep a stale dst in sk_dst_cache until it is checked again, and tcp_update_metrics() reads it with __sk_dst_get() without any dst_check(). Once dst_dev_put() has run, the dst device is blackhole_netdev and the entry is created in init_net. Before that, for instance for routes through nexthop objects whose cached dsts are released from a nested RCU callback, it is the netns the device has just been moved to.

A TCP socket of netns A that enters TIME_WAIT in that window then creates or updates an entry carrying A's addresses and RTT in another netns, where TCP_METRICS_CMD_GET can dump it and where new connections to the same peer inherit those metrics.

The metrics belong to the netns of the socket, so use sock_net() for full sockets and request sockets, and pass it to tcpm_new().

Kernel sockets that do not hold a netns reference can still reach tcp_time_wait() after tcp_net_metrics_exit_batch() has flushed the entries of their dismantling netns, and an entry created then would never be flushed and would keep a dangling tcpm_net. Make tcpm_new() refuse to create entries when check_net() fails. Its callers are in an RCU read-side section, and cleanup_net() waits for a grace period between the refcount reaching zero and the flush, so an entry added after a successful check is always seen by the flush.

Fixes 51c5d0c4b169 ("tcp: Maintain dynamic metrics in local cache.")
Cc Qiliang Yuan <[email protected]>
---
 net/ipv4/tcp_metrics.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)
net/ipv4/tcp_metrics.c +15 -5
index dc0c081fc1f33f735a38aaae8a7b4ab3d633b148..d58f99c67d4dc0877d0e853c9e9faddd80343c22 100644
@@ -161,16 +161,26 @@ static void tcpm_check_stamp(struct tcp_metrics_block *tm,
 	rcu_dereference_protected(p, lockdep_is_held(&tcp_metrics_lock))
 
 static struct tcp_metrics_block *tcpm_new(struct dst_entry *dst,
+					  struct net *net,
 					  struct inetpeer_addr *saddr,
 					  struct inetpeer_addr *daddr,
 					  unsigned int hash)
 {
 	struct tcp_metrics_block *tm;
 	bool reclaim = false;
-	struct net *net;
 
 	spin_lock_bh(&tcp_metrics_lock);
-	net = dst_dev_net_rcu(dst);
+
+	/* Do not add entries to a dismantling netns: they would not be
+	 * flushed again. Callers are in an RCU read-side section, and
+	 * cleanup_net() waits for a grace period after the netns refcount
+	 * reached zero and before tcp_metrics_flush_all(NULL) runs, so an
+	 * entry added after a successful check_net() is seen by the flush.
+	 */
+	if (unlikely(!check_net(net))) {
+		tm = NULL;
+		goto out_unlock;
+	}
 
 	/* While waiting for the spin-lock the cache might have been populated
 	 * with this entry and so we have to check again.
@@ -273,7 +283,7 @@ static struct tcp_metrics_block *__tcp_get_metrics_req(struct request_sock *req,
 		return NULL;
 	}
 
-	net = dst_dev_net_rcu(dst);
+	net = sock_net(req_to_sk(req));
 	hash ^= net_hash_mix(net);
 	hash = hash_32(hash, tcp_metrics_hash_log);
 
@@ -318,7 +328,7 @@ static struct tcp_metrics_block *tcp_get_metrics(struct sock *sk,
 	else
 		return NULL;
 
-	net = dst_dev_net_rcu(dst);
+	net = sock_net(sk);
 	hash ^= net_hash_mix(net);
 	hash = hash_32(hash, tcp_metrics_hash_log);
 
@@ -326,7 +336,7 @@ static struct tcp_metrics_block *tcp_get_metrics(struct sock *sk,
 	if (tm == TCP_METRICS_RECLAIM_PTR)
 		tm = NULL;
 	if (!tm && create)
-		tm = tcpm_new(dst, &saddr, &daddr, hash);
+		tm = tcpm_new(dst, net, &saddr, &daddr, hash);
 	else
 		tcpm_check_stamp(tm, dst);
 
Signature · 1 line
2.53.0
#1 lore raw
This is an automated message. This series looks like a fix, but its commit messages seem to be missing some information:

Hi!

This is an automated message. This series looks like a fix, but its commit messages seem to be missing some information:

- How the issue was discovered, e.g. hit in production, hit during development, syzbot report, manual code inspection, LLM or static analysis tool scan.

- Whether the issue was actually triggered, or is only theoretical (e.g. found by code inspection). If it was triggered please include the symptoms, like the stack trace or error messages.

Please do not repost the series just to address the above. Instead, reply to this email with the missing information, so that reviewers can take it into account. If the series needs another revision for other reasons, please include the information in the commit messages then.

The evaluation is done by an LLM so it may be wrong, if you think that is the case please reply and explain.

author maintainer
#2 lore raw
Quoted 10 lines · Le ven. 16 oct. 2026 à 10:53, <[email protected]> a écrit :

Hi!

This is an automated message. This series looks like a fix, but its commit messages seem to be missing some information:

- How the issue was discovered, e.g. hit in production, hit during development, syzbot report, manual code inspection, LLM or static analysis tool scan.

Discovered while reviewing
https://lore.kernel.org/netdev/20261016-bug-net-netns-move-rcu-barrier-v1-3-4157d798658f@gmail.com/T/#u

Another patch is also needed in fib_rt_cache updates, sent later today.

Quoted 12 lines

- Whether the issue was actually triggered, or is only theoretical (e.g. found by code inspection). If it was triggered please include the symptoms, like the stack trace or error messages.

Please do not repost the series just to address the above. Instead, reply to this email with the missing information, so that reviewers can take it into account. If the series needs another revision for other reasons, please include the information in the commit messages then.

The evaluation is done by an LLM so it may be wrong, if you think that is the case please reply and explain.

maintainer Reviewed-by Tested-by on v1 1/1
#3 lore raw
In reply to Eric Dumazet
Quoted 31 lines · Eric Dumazet

tcp_get_metrics(), __tcp_get_metrics_req() and tcpm_new() look up and create metrics entries in the netns of the cached dst device rather than in the netns of the socket.

These usually match, but not after the device has been moved to another netns or unregistered: a socket can keep a stale dst in sk_dst_cache until it is checked again, and tcp_update_metrics() reads it with __sk_dst_get() without any dst_check(). Once dst_dev_put() has run, the dst device is blackhole_netdev and the entry is created in init_net. Before that, for instance for routes through nexthop objects whose cached dsts are released from a nested RCU callback, it is the netns the device has just been moved to.

A TCP socket of netns A that enters TIME_WAIT in that window then creates or updates an entry carrying A's addresses and RTT in another netns, where TCP_METRICS_CMD_GET can dump it and where new connections to the same peer inherit those metrics.

The metrics belong to the netns of the socket, so use sock_net() for full sockets and request sockets, and pass it to tcpm_new().

Kernel sockets that do not hold a netns reference can still reach tcp_time_wait() after tcp_net_metrics_exit_batch() has flushed the entries of their dismantling netns, and an entry created then would never be flushed and would keep a dangling tcpm_net. Make tcpm_new() refuse to create entries when check_net() fails. Its callers are in an RCU read-side section, and cleanup_net() waits for a grace period between the refcount reaching zero and the flush, so an entry added after a successful check is always seen by the flush.

Quoted 4 lines
Fixes 51c5d0c4b169 ("tcp: Maintain dynamic metrics in local cache.")
Cc Qiliang Yuan <[email protected]>

---

Thanks, Eric!

I got an LLM to write and run a repro that verifies the bug and validates the fix. The script and its outputs are attached. In the control case, both kernels put the entry in netns A. In the three trigger cases, the base kernel puts the entry in init_net and the patched kernel puts it in netns A.

neal

Output:

# Base: unpatched net/main (191aa41548b2)
) time vng --run /tmp/bzImage.base --disable-kvm --cpus 2 --memory 2G --user root --exec "python3 $REPRO"
kernel: 7.3.0-rc7+
=== v4 none (control): "true" (client state before close: FIN-WAIT-2)

  init_net          : (none)
  netns A (socket)  : 10.9.0.2 age 0.470sec cwnd 10 rtt 5900us rttvar 11200us source 10.9.0.1
  netns B           : (none)
=== v4 move-dev-to-netns-B: "ip -n A link set vA netns B" (client state before close: FIN-WAIT-2)
  init_net          : 10.9.0.2 age 0.384sec cwnd 10 rtt 5164us rttvar 10188us source 10.9.0.1
  netns A (socket)  : (none)
  netns B           : (none)
=== v4 delete-dev: "ip -n A link del vA" (client state before close: FIN-WAIT-2)
  init_net          : 10.9.0.2 age 0.379sec cwnd 10 rtt 5164us rttvar 10161us source 10.9.0.1
  netns A (socket)  : (none)
  netns B           : (none)
=== v4 delete-addr: "ip -n A addr del 10.9.0.1/24 dev vA" (client state before close: FIN-WAIT-2)
  init_net          : 10.9.0.2 age 0.379sec cwnd 10 rtt 5207us rttvar 10268us source 10.9.0.1
  netns A (socket)  : (none)
  netns B           : (none)
=== v6 none (control): "true" (client state before close: FIN-WAIT-2)
  init_net          : (none)
  netns A (socket)  : fd00:9::2 age 0.457sec cwnd 10 rtt 69289us rttvar 113120us source fd00:9::1
  netns B           : (none)
=== v6 move-dev-to-netns-B: "ip -n A link set vA netns B" (client state before close: FIN-WAIT-2)
  init_net          : fd00:9::2 age 0.382sec cwnd 10 rtt 69134us rttvar 112948us source fd00:9::1
  netns A (socket)  : (none)
  netns B           : (none)
=== v6 delete-dev: "ip -n A link del vA" (client state before close: FIN-WAIT-2)
  init_net          : fd00:9::2 age 0.379sec cwnd 10 rtt 66492us rttvar 107579us source fd00:9::1
  netns A (socket)  : (none)
  netns B           : (none)
=== v6 delete-addr: "ip -n A -6 addr del fd00:9::1/64 dev vA" (client state before close: FIN-WAIT-2)
  init_net          : fd00:9::2 age 0.376sec cwnd 10 rtt 69172us rttvar 112945us source fd00:9::1
  netns A (socket)  : (none)
  netns B           : (none)
real    1m12.862s
user    0m56.941s
sys     0m19.470s

# Patched: net/main + Eric's patch
) time vng --run /tmp/bzImage.patched --disable-kvm --cpus 2 --memory 2G --user root --exec "python3 $REPRO" ; xterm
kernel: 7.3.0-rc7+
=== v4 none (control): "true" (client state before close: FIN-WAIT-2)

  init_net          : (none)
  netns A (socket)  : 10.9.0.2 age 0.477sec cwnd 10 rtt 5895us rttvar 11174us source 10.9.0.1
  netns B           : (none)
=== v4 move-dev-to-netns-B: "ip -n A link set vA netns B" (client state before close: FIN-WAIT-2)
  init_net          : (none)
  netns A (socket)  : 10.9.0.2 age 0.489sec cwnd 10 rtt 5153us rttvar 10152us source 10.9.0.1
  netns B           : (none)
=== v4 delete-dev: "ip -n A link del vA" (client state before close: FIN-WAIT-2)
  init_net          : (none)
  netns A (socket)  : 10.9.0.2 age 0.486sec cwnd 10 rtt 5147us rttvar 10150us source 10.9.0.1
  netns B           : (none)
=== v4 delete-addr: "ip -n A addr del 10.9.0.1/24 dev vA" (client state before close: FIN-WAIT-2)
  init_net          : (none)
  netns A (socket)  : 10.9.0.2 age 0.499sec cwnd 10 rtt 5148us rttvar 10144us source 10.9.0.1
  netns B           : (none)
=== v6 none (control): "true" (client state before close: FIN-WAIT-2)
  init_net          : (none)
  netns A (socket)  : fd00:9::2 age 0.490sec cwnd 10 rtt 67433us rttvar 109274us source fd00:9::1
  netns B           : (none)
=== v6 move-dev-to-netns-B: "ip -n A link set vA netns B" (client state before close: FIN-WAIT-2)
  init_net          : (none)
  netns A (socket)  : fd00:9::2 age 0.486sec cwnd 10 rtt 68010us rttvar 110627us source fd00:9::1
  netns B           : (none)
=== v6 delete-dev: "ip -n A link del vA" (client state before close: FIN-WAIT-2)
  init_net          : (none)
  netns A (socket)  : fd00:9::2 age 0.498sec cwnd 10 rtt 67152us rttvar 109100us source fd00:9::1
  netns B           : (none)
=== v6 delete-addr: "ip -n A -6 addr del fd00:9::1/64 dev vA" (client state before close: FIN-WAIT-2)
  init_net          : (none)
  netns A (socket)  : fd00:9::2 age 0.476sec cwnd 10 rtt 69384us rttvar 113220us source fd00:9::1
  netns B           : (none)
real    1m17.848s
user    0m53.937s
sys     0m30.023s

#!/usr/bin/env python3
# Reproducer for: "tcp: tcp_metrics: use the socket netns, not the dst device netns" #
# A TCP client socket in netns A goes to FIN_WAIT2 (shutdown(SHUT_WR), FIN ACKed),

# then its route/device goes away (device moved to netns B, device deleted, or
# address/route deleted) while the socket keeps the now-stale dst in
# sk_dst_cache. close() then runs tcp_time_wait() -> tcp_update_metrics()
# synchronously, which uses __sk_dst_get() without dst_check().
#
# We then report in which netns the tcp_metrics entry for the peer shows up.
# Expected without the patch: in init_net (dst->dev == blackhole_netdev).
# Expected with the patch:    in netns A (the socket's netns).

import os
import socket
import subprocess
import sys
import time

ROOT_NS = os.open('/proc/self/ns/net', os.O_RDONLY)

def sh(cmd, check=True):
r = subprocess.run(cmd, shell=True, capture_output=True, text=True) if check and r.returncode:
print(f'CMD FAILED: {cmd}\n{r.stdout}{r.stderr}', flush=True) sys.exit(1)
return r.stdout

def enter_ns(name):

    fd = os.open(f'/run/netns/{name}', os.O_RDONLY)
    os.setns(fd, os.CLONE_NEWNET)
    os.close(fd)

def enter_root():
os.setns(ROOT_NS, os.CLONE_NEWNET)

def metrics(ns):
pfx = f'ip -n {ns} ' if ns else 'ip '
out = sh(pfx + 'tcp_metrics show', check=False).strip() return out if out else '(none)'

def recv_exact(s, n):
buf = b''
while len(buf) < n:
chunk = s.recv(n - len(buf))
if not chunk:
raise RuntimeError('unexpected EOF')
buf += chunk
return buf

def setup():
for ns in ('A', 'B', 'S'):
sh(f'ip netns del {ns}', check=False)
sh('ip tcp_metrics flush all', check=False)
for ns in ('A', 'B', 'S'):
sh(f'ip netns add {ns}')
sh(f'ip -n {ns} link set lo up')
sh('ip link add vA netns A type veth peer name vS netns S') sh('ip -n A addr add 10.9.0.1/24 dev vA')
sh('ip -n S addr add 10.9.0.2/24 dev vS')
sh('ip -n A -6 addr add fd00:9::1/64 dev vA nodad') sh('ip -n S -6 addr add fd00:9::2/64 dev vS nodad') sh('ip -n A link set vA up')
sh('ip -n S link set vS up')
time.sleep(0.3)

TRIGGERS = {
'none (control)': 'true', 'move-dev-to-netns-B': 'ip -n A link set vA netns B', 'delete-dev': 'ip -n A link del vA', 'delete-addr': None, # filled per family
}

def run_case(family, trigger):

    setup()
    if family == socket.AF_INET:
        peer, fam = '10.9.0.2', 'v4'
        addrdel = 'ip -n A addr del 10.9.0.1/24 dev vA'
    else:
        peer, fam = 'fd00:9::2', 'v6'
        addrdel = 'ip -n A -6 addr del fd00:9::1/64 dev vA'
    cmd = TRIGGERS[trigger] or addrdel

    enter_ns('S')
    srv = socket.socket(family, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind((peer, 5000))
    srv.listen(1)
    enter_ns('A')
    cli = socket.socket(family, socket.SOCK_STREAM)
    enter_root()

    cli.connect((peer, 5000))
    conn, _ = srv.accept()
    for _ in range(20):  # get an RTT sample
        cli.sendall(b'x' * 1000)
        recv_exact(conn, 1000)
        conn.sendall(b'y' * 1000)
        recv_exact(cli, 1000)
    cli.shutdown(socket.SHUT_WR)  # FIN -> ACKed -> FIN_WAIT2
    time.sleep(0.5)
    state = [ln.split()[0] for ln in
             sh('ip netns exec A ss -Htan', check=False).splitlines()
             if ':5000' in ln]
    sh(cmd)
    time.sleep(1.0)  # let RCU callbacks (dst_dev_put()) run
    cli.close()  # FIN_WAIT2 -> tcp_time_wait() -> tcp_update_metrics()
    time.sleep(0.3)

    print(f'=== {fam} {trigger}: "{cmd}" (client state before close: '
          f'{state[0] if state else "?"})', flush=True)
    for ns, label in ((None, 'init_net'), ('A', 'netns A (socket)'),
                      ('B', 'netns B')):
        print(f'  {label:18s}: {metrics(ns)}', flush=True)
    conn.close()
    srv.close()

def main():
print('kernel:', os.uname().release, flush=True) for family in (socket.AF_INET, socket.AF_INET6):
for trigger in TRIGGERS:
run_case(family, trigger)

if __name__ == '__main__':

    main()
author maintainer
#4 lore raw
Quoted 10 lines · Neal Cardwell

Thanks, Eric!

I got an LLM to write and run a repro that verifies the bug and validates the fix. The script and its outputs are attached. In the control case, both kernels put the entry in netns A. In the three trigger cases, the base kernel puts the entry in init_net and the patched kernel puts it in netns A.

Nice, thanks a lot Neal.

Quoted 3 lines

neal

reviewer Reviewed-by on v1 1/1
#5 lore raw
In reply to Eric Dumazet
Quoted 30 lines · Eric Dumazet

tcp_get_metrics(), __tcp_get_metrics_req() and tcpm_new() look up and create metrics entries in the netns of the cached dst device rather than in the netns of the socket.

These usually match, but not after the device has been moved to another netns or unregistered: a socket can keep a stale dst in sk_dst_cache until it is checked again, and tcp_update_metrics() reads it with __sk_dst_get() without any dst_check(). Once dst_dev_put() has run, the dst device is blackhole_netdev and the entry is created in init_net. Before that, for instance for routes through nexthop objects whose cached dsts are released from a nested RCU callback, it is the netns the device has just been moved to.

A TCP socket of netns A that enters TIME_WAIT in that window then creates or updates an entry carrying A's addresses and RTT in another netns, where TCP_METRICS_CMD_GET can dump it and where new connections to the same peer inherit those metrics.

The metrics belong to the netns of the socket, so use sock_net() for full sockets and request sockets, and pass it to tcpm_new().

Kernel sockets that do not hold a netns reference can still reach tcp_time_wait() after tcp_net_metrics_exit_batch() has flushed the entries of their dismantling netns, and an entry created then would never be flushed and would keep a dangling tcpm_net. Make tcpm_new() refuse to create entries when check_net() fails. Its callers are in an RCU read-side section, and cleanup_net() waits for a grace period between the refcount reaching zero and the flush, so an entry added after a successful check is always seen by the flush.

Quoted 2 lines
Fixes 51c5d0c4b169 ("tcp: Maintain dynamic metrics in local cache.")
#6 lore raw
In reply to Eric Dumazet
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 13 lines · On Fri, 16 Oct 2026 10:48:02 +0200 you wrote:

tcp_get_metrics(), __tcp_get_metrics_req() and tcpm_new() look up and create metrics entries in the netns of the cached dst device rather than in the netns of the socket.

These usually match, but not after the device has been moved to another netns or unregistered: a socket can keep a stale dst in sk_dst_cache until it is checked again, and tcp_update_metrics() reads it with __sk_dst_get() without any dst_check(). Once dst_dev_put() has run, the dst device is blackhole_netdev and the entry is created in init_net. Before that, for instance for routes through nexthop objects whose cached dsts are released from a nested RCU callback, it is the netns the device has just been moved to.

Quoted 1 line

[...]

Here is the summary with links:
- [net] tcp: tcp_metrics: use the socket netns, not the dst device netns

    https://git.kernel.org/netdev/net/c/504bdcb8c156

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html