ss command not showing raw sockets? (regression)
Subsystems (2)
Hi there,
I've got following installation:
* ping 32 bit version
* Linux 4.4.0 x86_64 (yes, somewhat ancient)
* iproute2 4.9.0 or 4.20.0 or 5.10.0
With one ping command active, there are two raw sockets on my system: one for IPv4 and one for IPv6 (just one of those is used).
My problem is that
ss -awp
shows
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)
So is this a bug or is this wont-fix (then, if it is related to kernel version, package maintainers may be interested)?
Best regards
Jonas Bechtel
Jakub Kicinski
maintainerQuoted 8 lines · On Sun, 15 Aug 2021 23:17:38 +0200 Jonas Bechtel wrote:
I've got following installation:
* ping 32 bit version
* Linux 4.4.0x86_64(yes, somewhat ancient)
* iproute2 4.9.0 or 4.20.0 or 5.10.0With one ping command active, there are two raw sockets on my system:
one for IPv4 and one for IPv6 (just one of those is used).
Quoted 8 lines
My problem is that
ss -awp
shows
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)
Could you clarify how the bug manifests itself? Does ss crash?
Quoted 2 lines
So is this a bug or is this wont-fix (then, if it is related to kernel version, package maintainers may be interested)?
Quoted 11 lines · Jakub Kicinski
Quoted 10 lines · On Sun, 15 Aug 2021 23:17:38 +0200 Jonas Bechtel wrote:
I've got following installation:
* ping 32 bit version
* Linux 4.4.0x86_64(yes, somewhat ancient)
* iproute2 4.9.0 or 4.20.0 or 5.10.0With one ping command active, there are two raw sockets on my system:
one for IPv4 and one for IPv6 (just one of those is used).My problem is that
Quoted 8 lines
Quoted 6 lines
ss -awp
shows
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)Could you clarify how the bug manifests itself? Does ss crash?
I take it kernel version is constant and iproute2 version changes, correct? Can you download the source and do a git bisect?
Quoted 3 lines
Quoted 3 lines
So is this a bug or is this wont-fix (then, if it is related to kernel version, package maintainers may be interested)?
Jakub Kicinski
maintainerQuoted 11 lines · On Mon, 16 Aug 2021 15:08:00 -0700 Jakub Kicinski wrote:
Quoted 10 lines · On Sun, 15 Aug 2021 23:17:38 +0200 Jonas Bechtel wrote:
I've got following installation:
* ping 32 bit version
* Linux 4.4.0x86_64(yes, somewhat ancient)
* iproute2 4.9.0 or 4.20.0 or 5.10.0With one ping command active, there are two raw sockets on my system:
one for IPv4 and one for IPv6 (just one of those is used).My problem is that
Quoted 11 lines
Quoted 6 lines
ss -awp
shows
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)Could you clarify how the bug manifests itself? Does ss crash?
Quoted 2 lines
So is this a bug or is this wont-fix (then, if it is related to kernel version, package maintainers may be interested)?
I had a look, I don't see anything out of the ordinary. I checked with v4.6, I don't have a 4.4 box handy. It seems ss got support for dumping over netlink in the 4.9. On a 4.4 kernel it should fall back to using procfs tho, raw_show() calls inet_show_netlink() which should fails and therefore the code should fall through to the old procfs stuff.
No idea why that doesn't happen for you. Is this vanilla 4.4 or does it have backports? Is there a /sys/module/raw_diag/ directory on your system after you run those commands?
Does setting PROC_NET_RAW make the newer iproute version work for you?
$ PROC_NET_RAW=/proc/net/raw ss -awp
On Tue, 17 Aug 2021 08:04:51 -0700
Jakub Kicinski <[email protected]> wrote with subject "Re: ss command not showing raw sockets? (regression)":
Quoted 31 lines · On Mon, 16 Aug 2021 15:08:00 -0700 Jakub Kicinski wrote:
Quoted 11 lines · On Sun, 15 Aug 2021 23:17:38 +0200 Jonas Bechtel wrote:
Quoted 10 lines
I've got following installation:
* ping 32 bit version
* Linux 4.4.0x86_64(yes, somewhat ancient)
* iproute2 4.9.0 or 4.20.0 or 5.10.0With one ping command active, there are two raw sockets on my
system: one for IPv4 and one for IPv6 (just one of those is used).My problem is that
Quoted 11 lines
Quoted 6 lines
ss -awp
shows
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)Could you clarify how the bug manifests itself? Does ss crash?
Quoted 2 lines
So is this a bug or is this wont-fix (then, if it is related to kernel version, package maintainers may be interested)?
I had a look, I don't see anything out of the ordinary. I checked with v4.6, I don't have a 4.4 box handy. It seems ss got support for dumping over netlink in the 4.9. On a 4.4 kernel it should fall back to using procfs tho,
raw_show()callsinet_show_netlink()which should fails and therefore the code should fall through to the old procfs stuff.
Quoted 3 lines
No idea why that doesn't happen for you. Is this vanilla 4.4 or does it have backports? Is there a
/sys/module/raw_diag/directory on your system after you run those commands?
It's was Knoppix distributed package. I don't know about the exact contents, there's also no hint in package description. I just know that it works without initrd, as it directly mounts the root disk.
No, there's /sys/module but no /sys/module/raw_diag/ neither before nor after running those commands.
Quoted 4 lines
Does setting
PROC_NET_RAWmake the newer iproute version work for you?$ PROC_NET_RAW=/proc/net/raw ss -awp
Yes, this did the trick. (And again I was thinking programs were doing something "magical", but in the end it's just a file they access)
Furthermore I checked with Linux 4.19.0 amd64 RT (Debian package; from package description: "This kernel includes the PREEMPT_RT realtime patch set."). With this kernel there was no need for PROC_NET_RAW. All iproute versions worked out of the box and showed even command name, pid and fd number (that's why ss traverses all /proc/[pids]/fd/ directories?).
See attached log file, with kernel versions and iproute2 versions printed.
@kuba With PROC_NET_RAW I consider the problem is found, isn't it? So I will not download/bisect<->build or otherwise investigate the problem until one of you explicitely asks me to do so.
I have now redirected invocation of command with set PROC_NET_RAW on my system, and may (try to) update to Linux 4.19.
Thank you!
Best Regards
jbechtel
user@machine:~$ uname -a; for file in ./ss_{4.9.0,4.20.0,5.10.0}; do echo $file; $file -awp; PROC_NET_RAW=/proc/net/raw $file -awp; done
Linux machine 4.4.0-64 #2 SMP PREEMPT Mon Jan 25 01:54:18 CET 2016 x86_64 GNU/Linux
./ss_4.9.0
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 *:icmp *:*
UNCONN 0 0 :::ipv6-icmp :::*
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 *:icmp *:*
UNCONN 0 0 :::ipv6-icmp :::*
./ss_4.20.0
State Recv-Q Send-Q Local Address:Port Peer Address:Port
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 0.0.0.0:icmp 0.0.0.0:*
UNCONN 0 0 *:ipv6-icmp *:*
./ss_5.10.0
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
UNCONN 0 0 0.0.0.0:icmp 0.0.0.0:*
UNCONN 0 0 *:ipv6-icmp *:*
user@machine:~$ # [kernel change, reboot, etc.]
user@machine:~$ uname -a; for file in ./ss_{4.9.0,4.20.0,5.10.0}; do echo $file; $file -awp; PROC_NET_RAW=/proc/net/raw $file -awp; done
Linux (none) 4.19.0-16-rt-amd64 #1 SMP PREEMPT RT Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
./ss_4.9.0
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 *:icmp *:* users:(("ping",pid=272,fd=3))
UNCONN 0 0 :::ipv6-icmp :::* users:(("ping",pid=272,fd=4))
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 *:icmp *:* users:(("ping",pid=272,fd=3))
UNCONN 0 0 :::ipv6-icmp :::* users:(("ping",pid=272,fd=4))
./ss_4.20.0
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 0.0.0.0:icmp 0.0.0.0:* users:(("ping",pid=272,fd=3))
UNCONN 0 0 *:ipv6-icmp *:* users:(("ping",pid=272,fd=4))
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 0.0.0.0:icmp 0.0.0.0:* users:(("ping",pid=272,fd=3))
UNCONN 0 0 *:ipv6-icmp *:* users:(("ping",pid=272,fd=4))
./ss_5.10.0
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
UNCONN 0 0 0.0.0.0:icmp 0.0.0.0:* users:(("ping",pid=272,fd=3))
UNCONN 0 0 *:ipv6-icmp *:* users:(("ping",pid=272,fd=4))
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
UNCONN 0 0 0.0.0.0:icmp 0.0.0.0:* users:(("ping",pid=272,fd=3))
UNCONN 0 0 *:ipv6-icmp *:* users:(("ping",pid=272,fd=4))
user@machine:~$ Jakub Kicinski
maintainerQuoted 13 lines · On Tue, 17 Aug 2021 20:21:35 +0200 Jonas Bechtel wrote:
On Tue, 17 Aug 2021 08:04:51 -0700
Jakub Kicinski <[email protected]> wrote with subjectQuoted 10 lines
I had a look, I don't see anything out of the ordinary. I checked with v4.6, I don't have a 4.4 box handy. It seems ss got support for dumping over netlink in the 4.9. On a 4.4 kernel it should fall back to using procfs tho,
raw_show()callsinet_show_netlink()which should fails and therefore the code should fall through to the old procfs stuff.No idea why that doesn't happen for you. Is this vanilla 4.4 or does it have backports? Is there a
/sys/module/raw_diag/directory on your system after you run those commands?
Quoted 6 lines
It's was Knoppix distributed package. I don't know about the exact contents, there's also no hint in package description. I just know that it works without initrd, as it directly mounts the root disk.
No, there's
/sys/modulebut no/sys/module/raw_diag/neither before nor after running those commands.
Hm. Could you share the config for that kernel? I don't know Knoppix but there should be a /boot/config-* or /proc/config.gz, hopefully.
Quoted 22 lines
Quoted 4 lines
Does setting
PROC_NET_RAWmake the newer iproute version work for you?$ PROC_NET_RAW=/proc/net/raw ss -awpYes, this did the trick. (And again I was thinking programs were doing something "magical", but in the end it's just a file they access)
Furthermore I checked with Linux 4.19.0 amd64 RT (Debian package; from package description: "This kernel includes the
PREEMPT_RTrealtime patch set."). With this kernel there was no need forPROC_NET_RAW. All iproute versions worked out of the box and showed even command name, pid and fd number (that's why ss traverses all /proc/[pids]/fd/ directories?).See attached log file, with kernel versions and iproute2 versions printed.
Quoted 6 lines
@kuba With
PROC_NET_RAWI consider the problem is found, isn't it? So I will not download/bisect<->build or otherwise investigate the problem until one of you explicitely asks me to do so.I have now redirected invocation of command with set
PROC_NET_RAWon my system, and may (try to) update to Linux 4.19.
I suspect the bisection would end up at the commit which added the netlink dump support, so you can hold off for now, yes.
My best guess right now is that Knoppix has a cut-down kernel config and we don't handle that case correctly.
Quoted 9 lines · Jakub Kicinski
Quoted 6 lines
@kuba With
PROC_NET_RAWI consider the problem is found, isn't it? So I will not download/bisect<->build or otherwise investigate the problem until one of you explicitely asks me to do so.I have now redirected invocation of command with set
PROC_NET_RAWon my system, and may (try to) update to Linux 4.19.I suspect the bisection would end up at the commit which added the netlink dump support, so you can hold off for now, yes.
agreed.
Quoted 4 lines
My best guess right now is that Knoppix has a cut-down kernel config and we don't handle that case correctly.
CONFIG_INET_RAW_DIAG (or INET_DIAG) is probably disabled. surprised the netlink dump does not return an error and it falls back to the proc file:
if (!getenv("PROC_NET_RAW") && !getenv("PROC_ROOT") &&
inet_show_netlink(f, NULL, IPPROTO_RAW) == 0)
return 0;can you strace it?
Jakub Kicinski
maintainerQuoted 17 lines · On Tue, 17 Aug 2021 13:54:53 -0600 David Ahern wrote:
Quoted 9 lines · Jakub Kicinski
Quoted 6 lines
@kuba With
PROC_NET_RAWI consider the problem is found, isn't it? So I will not download/bisect<->build or otherwise investigate the problem until one of you explicitely asks me to do so.I have now redirected invocation of command with set
PROC_NET_RAWon my system, and may (try to) update to Linux 4.19.I suspect the bisection would end up at the commit which added the netlink dump support, so you can hold off for now, yes.
agreed.
Quoted 4 lines
My best guess right now is that Knoppix has a cut-down kernel config and we don't handle that case correctly.
Quoted 8 lines
CONFIG_INET_RAW_DIAG(orINET_DIAG) is probably disabled. surprised the netlink dump does not return an error and it falls back to the proc file:if (!getenv("PROC_NET_RAW") && !getenv("PROC_ROOT") && inet_show_netlink(f, NULL, IPPROTO_RAW) == 0) return 0;can you strace it?
Ah, good point, strace will show it.
/me goes off to look at the strace Jonas sent off list.
Well this is unexpected:
sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=76, type=DCCPDIAG_GETSOCK, ...
--->8----------------
From: Jakub Kicinski <[email protected]>
Subject: ss: fix fallback to procfs for raw and sctp sockets
sockdiag_send() diverts to tcpdiag_send() to try the older netlink interface. tcpdiag_send() works for TCP and DCCP but not other protocols. Instead of rejecting unsupported protocols (and missing RAW and SCTP) match on supported ones.
--- misc/ss.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
misc/ss.c +4 -4
@@ -3404,13 +3404,13 @@ static int tcpdiag_send(int fd, int protocol, struct filter *f)struct iovec iov[3 ];int iovlen =1 ; -if (protocol ==IPPROTO_UDP || protocol ==IPPROTO_MPTCP ) -return -1 ; -if (protocol ==IPPROTO_TCP ) req.nlh.nlmsg_type =TCPDIAG_GETSOCK ; -else +else if (protocol ==IPPROTO_DCCP ) req.nlh.nlmsg_type =DCCPDIAG_GETSOCK ; +else +return -1 ; +if (show_mem) { req.r.idiag_ext |= (1 <<(INET_DIAG_MEMINFO -1 )); req.r.idiag_ext |= (1 <<(INET_DIAG_SKMEMINFO -1 ));
Signature · 1 line
2.31.1
Quoted 64 lines · Jakub Kicinski
Quoted 17 lines · On Tue, 17 Aug 2021 13:54:53 -0600 David Ahern wrote:
Quoted 9 lines · Jakub Kicinski
Quoted 6 lines
@kuba With
PROC_NET_RAWI consider the problem is found, isn't it? So I will not download/bisect<->build or otherwise investigate the problem until one of you explicitely asks me to do so.I have now redirected invocation of command with set
PROC_NET_RAWon my system, and may (try to) update to Linux 4.19.I suspect the bisection would end up at the commit which added the netlink dump support, so you can hold off for now, yes.
agreed.
Quoted 4 lines
My best guess right now is that Knoppix has a cut-down kernel config and we don't handle that case correctly.
Quoted 8 lines
CONFIG_INET_RAW_DIAG(orINET_DIAG) is probably disabled. surprised the netlink dump does not return an error and it falls back to the proc file:if (!getenv("PROC_NET_RAW") && !getenv("PROC_ROOT") && inet_show_netlink(f, NULL, IPPROTO_RAW) == 0) return 0;can you strace it?
Ah, good point, strace will show it.
/me goes off to look at the strace Jonas sent off list.
Well this is unexpected:
sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=76, type=DCCPDIAG_GETSOCK, ...
--->8----------------
From: Jakub Kicinski <[email protected]>
Subject: ss: fix fallback to procfs for raw and sctp sockets
sockdiag_send()diverts totcpdiag_send()to try the older netlink interface.tcpdiag_send()works for TCP and DCCP but not other protocols. Instead of rejecting unsupported protocols (and missing RAW and SCTP) match on supported ones.Fixes 41fe6c34de50 ("ss: Add inet raw sockets information gathering via netlink diag interface")Signed-off-by Jakub Kicinski <[email protected]>--- misc/ss.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)misc/ss.c +0
-3@@ -3404,13 +3404,13 @@ static int tcpdiag_send(int fd, int protocol, struct filter *f)struct iovec iov[3 ];int iovlen =1 ; -if (protocol ==IPPROTO_UDP || protocol ==IPPROTO_MPTCP ) -return -1 ; -if (protocol ==IPPROTO_TCP )
misc/ss.c static int tcpdiag_send(int fd, int protocol, struct filter *f) · line 3416
Quoted 10 lines
misc/ss.c +4
-1req.nlh.nlmsg_type =TCPDIAG_GETSOCK ; - else + else if (protocol == IPPROTO_DCCP) req.nlh.nlmsg_type = DCCPDIAG_GETSOCK; + else + return -1; +if (show_mem) { req.r.idiag_ext |= (1 <<(INET_DIAG_MEMINFO -1 )); req.r.idiag_ext |= (1 <<(INET_DIAG_SKMEMINFO -1 ));
That looks correct to me.
Jonas: can you build iproute2 and test?
On Tue, 17 Aug 2021 18:47:06 -0600
David Ahern <[email protected]> wrote with subject "Re: ss command not showing raw sockets? (regression)":
Quoted 52 lines · Jakub Kicinski
Quoted 38 lines
Ah, good point, strace will show it.
/me goes off to look at the strace Jonas sent off list.
Well this is unexpected:
sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=76, type=DCCPDIAG_GETSOCK, ... --->8----------------
From: Jakub Kicinski <[email protected]>
Subject: ss: fix fallback to procfs for raw and sctp sockets
sockdiag_send()diverts totcpdiag_send()to try the older netlink interface.tcpdiag_send()works for TCP and DCCP but not other protocols. Instead of rejecting unsupported protocols (and missing RAW and SCTP) match on supported ones.Fixes 41fe6c34de50 ("ss: Add inet raw sockets informationgathering via netlink diag interface") Signed-off-by: Jakub Kicinski <[email protected]> ---
misc/ss.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)misc/ss.c +0
-3@@ -3404,13 +3404,13 @@ static int tcpdiag_send(int fd, int protocol, struct filter *f) struct iovec iov[3];int iovlen =1 ; -if (protocol ==IPPROTO_UDP || protocol ==IPPROTO_MPTCP ) -return -1 ; -if (protocol ==IPPROTO_TCP )misc/ss.c static int tcpdiag_send(int fd, int · line 3416
Quoted 10 lines
misc/ss.c +4
-1req.nlh.nlmsg_type =TCPDIAG_GETSOCK ; - else + else if (protocol == IPPROTO_DCCP) req.nlh.nlmsg_type = DCCPDIAG_GETSOCK; + else + return -1; +if (show_mem) { req.r.idiag_ext |= (1 <<(INET_DIAG_MEMINFO -1 )); req.r.idiag_ext |= (1 <<(INET_DIAG_SKMEMINFO -1 ));That looks correct to me.
Quoted 1 line
Jonas: can you build iproute2 and test?
I've cloned branch main as instructed in https://wiki.linuxfoundation.org/networking/iproute2. Most recent commit is 9b7ea92b9e3f. After building, no socket was listed in table.
Then I [manually] applied the patch and rebuilt. The patched version works well, I do see the two sockets right now.
Command was in both cases misc/ss -awp
Quoted 59 lines · Jonas Bechtel
On Tue, 17 Aug 2021 18:47:06 -0600
David Ahern <[email protected]> wrote with subject "Re: ss command not showing raw sockets? (regression)":Quoted 52 lines · Jakub Kicinski
Quoted 38 lines
Ah, good point, strace will show it.
/me goes off to look at the strace Jonas sent off list.
Well this is unexpected:
sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=76, type=DCCPDIAG_GETSOCK, ... --->8----------------
From: Jakub Kicinski <[email protected]>
Subject: ss: fix fallback to procfs for raw and sctp sockets
sockdiag_send()diverts totcpdiag_send()to try the older netlink interface.tcpdiag_send()works for TCP and DCCP but not other protocols. Instead of rejecting unsupported protocols (and missing RAW and SCTP) match on supported ones.Fixes 41fe6c34de50 ("ss: Add inet raw sockets informationgathering via netlink diag interface") Signed-off-by: Jakub Kicinski <[email protected]> ---
misc/ss.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)misc/ss.c +0
-3@@ -3404,13 +3404,13 @@ static int tcpdiag_send(int fd, int protocol, struct filter *f) struct iovec iov[3];int iovlen =1 ; -if (protocol ==IPPROTO_UDP || protocol ==IPPROTO_MPTCP ) -return -1 ; -if (protocol ==IPPROTO_TCP )misc/ss.c static int tcpdiag_send(int fd, int · line 3416
Quoted 10 lines
misc/ss.c +4
-1req.nlh.nlmsg_type =TCPDIAG_GETSOCK ; - else + else if (protocol == IPPROTO_DCCP) req.nlh.nlmsg_type = DCCPDIAG_GETSOCK; + else + return -1; +if (show_mem) { req.r.idiag_ext |= (1 <<(INET_DIAG_MEMINFO -1 )); req.r.idiag_ext |= (1 <<(INET_DIAG_SKMEMINFO -1 ));That looks correct to me.
Quoted 8 lines
Quoted 1 line
Jonas: can you build iproute2 and test?
I've cloned branch main as instructed in https://wiki.linuxfoundation.org/networking/iproute2. Most recent commit is
9b7ea92b9e3f. After building, no socket was listed in table.Then I [manually] applied the patch and rebuilt. The patched version works well, I do see the two sockets right now.
Command was in both cases misc/ss -awp
Awesome, thanks for doing that. Jakub: want to send a formal patch with his Tested-by?