net: introduce ip_local_unbindable_ports sysctl
Subsystems (5)
PATCH 1/1 net: introduce ip_local_unbindable_ports sysctl
From: Maciej Żenczykowski <[email protected]>
and associated inet_is_local_unbindable_port() helper function:
use it to make explicitly binding to an unbindable port return -EPERM 'Operation not permitted'.
Autobind doesn't honour this new sysctl since:
(a) you can simply set both if that's the behaviour you desire
(b) there could be a use for preventing explicit while allowing auto
(c) it's faster in the relatively critical path of doing port selection during connect() to only check one bitmap instead of both
Various ports may have special use cases which are not suitable for use by general userspace applications. Currently, ports specified in ip_local_reserved_ports sysctl will not be returned only in case of automatic port assignment, but nothing prevents you from explicitly binding to them - even from an entirely unprivileged process.
In certain cases it is desirable to prevent the host from assigning the ports even in case of explicit binds, even from superuser processes.
Example use cases might be:
- a port being stolen by the nic for remote serial console, remote power management or some other sort of debugging functionality (crash collection, gdb, direct access to some other microcontroller on the nic or motherboard, remote management of the nic itself).
- a transparent proxy where packets are being redirected: in case a socket matches this connection, packets from this application would be incorrectly sent to one of the endpoints.
Initially I wanted to solve this problem via the simple one line:
static inline bool inet_port_requires_bind_service(struct net *net, unsigned short port) {
- return port < net->ipv4.sysctl_ip_prot_sock;
+ return port < net->ipv4.sysctl_ip_prot_sock || inet_is_local_reserved_port(net, port);
}
However, this doesn't work for two reasons:
(a) it changes userspace visible behaviour of the existing local reserved ports sysctl, and there appears to be enough documentation on the internet talking about setting it to make this a bad idea
(b) it doesn't prevent privileged apps from using these ports, CAP_BIND_SERVICE is relatively likely to be available to, for example, a recursive DNS server so it can listed on port 53, which also needs to do src port randomization for outgoing queries due to security reasons (and it thus does manual port binding).
If we *know* that certain ports are simply unusable, then it's better nothing even gets the opportunity to try to use them. This way we at least get a quick failure, instead of some sort of timeout (or possibly even corruption of the data stream of the non-kernel based use case).
Test:
vm:~# cat /proc/sys/net/ipv4/ip_local_unbindable_ports
vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0); s.bind(("::", 3967))'
vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0); s.bind(("::", 3967))'
vm:~# echo 3967 > /proc/sys/net/ipv4/ip_local_unbindable_ports
vm:~# cat /proc/sys/net/ipv4/ip_local_unbindable_ports
3967
vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0); s.bind(("::", 3967))'
socket.error: (1, 'Operation not permitted')
vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0); s.bind(("::", 3967))'
socket.error: (1, 'Operation not permitted')--- Documentation/networking/ip-sysctl.txt | 13 +++++++++++++ include/net/ip.h | 12 ++++++++++++ include/net/netns/ipv4.h | 1 + net/ipv4/af_inet.c | 4 ++++ net/ipv4/sysctl_net_ipv4.c | 18 ++++++++++++++++-- net/ipv6/af_inet6.c | 2 ++ net/sctp/socket.c | 5 +++++ 7 files changed, 53 insertions(+), 2 deletions(-)
Documentation/networking/ip-sysctl.txt +13 -0
@@ -940,6 +940,19 @@ ip_local_reserved_ports - list of comma separated ranges Default: Empty +ip_local_unbindable_ports - list of comma separated ranges + Specify the ports which are not directly bind()able. + + Usually you would use this to block the use of ports which + are invalid due to something outside of the control of the + kernel. For example a port stolen by the nic for serial + console, remote power management or debugging. + + There's a relatively high chance you will also want to list + these ports in 'ip_local_reserved_ports' to prevent autobinding. + + Default: Empty + ip_unprivileged_port_start - INTEGER This is a per-namespace sysctl. It defines the first unprivileged port in the network namespace. Privileged ports
include/net/ip.h +12 -0
@@ -346,6 +346,13 @@ static inline bool inet_is_local_reserved_port(struct net *net, unsigned short preturn test_bit (port, net->ipv4.sysctl_local_reserved_ports); } +static inline bool inet_is_local_unbindable_port (struct net *net,unsigned short port) +{ +if (!net->ipv4.sysctl_local_unbindable_ports) +return false; +return test_bit (port, net->ipv4.sysctl_local_unbindable_ports); +} +static inline bool sysctl_dev_name_is_allowed (const char *name) {return strcmp (name,"default" ) !=0 &&strcmp (name,"all" ) !=0 ;
@@ -362,6 +369,11 @@ static inline bool inet_is_local_reserved_port(struct net *net, unsigned short preturn false; } +static inline bool inet_is_local_unbindable_port (struct net *net,unsigned short port) +{ +return false; +} +static inline bool inet_port_requires_bind_service (struct net *net,unsigned short port) {return port <PROT_SOCK ;
include/net/netns/ipv4.h +1 -0
@@ -197,6 +197,7 @@ struct netns_ipv4 {#ifdef CONFIG_SYSCTL unsigned long *sysctl_local_reserved_ports; +unsigned long *sysctl_local_unbindable_ports;int sysctl_ip_prot_sock;#endif
net/ipv4/af_inet.c +4 -0
@@ -494,6 +494,10 @@ int __inet_bind(struct sock *sk, struct sockaddr *uaddr, int addr_len, goto out; snum =ntohs (addr->sin_port); + err = -EPERM ; +if (snum &&inet_is_local_unbindable_port (net, snum)) + goto out; + err = -EACCES ;if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))
net/ipv4/sysctl_net_ipv4.c +16 -2
@@ -745,6 +745,13 @@ static struct ctl_table ipv4_net_table[] = { .mode = 0644, .proc_handler = proc_do_large_bitmap, }, + { + .procname = "ip_local_unbindable_ports", + .data = &init_net.ipv4.sysctl_local_unbindable_ports, + .maxlen = 65536, + .mode = 0644, + .proc_handler = proc_do_large_bitmap, + }, { .procname = "ip_no_pmtu_disc", .data = &init_net.ipv4.sysctl_ip_no_pmtu_disc,
@@ -1353,11 +1360,17 @@ static __net_init int ipv4_sysctl_init_net(struct net *net) net->ipv4.sysctl_local_reserved_ports =kzalloc (65536 /8 ,GFP_KERNEL );if (!net->ipv4.sysctl_local_reserved_ports) - goto err_ports; + goto err_reserved_ports; + + net->ipv4.sysctl_local_unbindable_ports =kzalloc (65536 /8 ,GFP_KERNEL ); +if (!net->ipv4.sysctl_local_unbindable_ports) + goto err_unbindable_ports;return 0 ; -err_ports: +err_unbindable_ports: +kfree (net->ipv4.sysctl_local_reserved_ports); +err_reserved_ports:unregister_net_sysctl_table (net->ipv4.ipv4_hdr); err_reg:if (!net_eq (net, &init_net))
@@ -1370,6 +1383,7 @@ static __net_exit void ipv4_sysctl_exit_net(struct net *net) {struct ctl_table *table; +kfree (net->ipv4.sysctl_local_unbindable_ports);kfree (net->ipv4.sysctl_local_reserved_ports); table = net->ipv4.ipv4_hdr->ctl_table_arg;unregister_net_sysctl_table (net->ipv4.ipv4_hdr);
net/ipv6/af_inet6.c +2 -0
@@ -292,6 +292,8 @@ static int __inet6_bind(struct sock *sk, struct sockaddr *uaddr, int addr_len,return -EINVAL ; snum =ntohs (addr->sin6_port); +if (snum &&inet_is_local_unbindable_port (net, snum)) +return -EPERM ;if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))return -EACCES ;
net/sctp/socket.c +5 -0
@@ -384,6 +384,9 @@ static int sctp_do_bind(struct sock *sk, union sctp_addr *addr, int len) } } +if (snum &&inet_is_local_unbindable_port (net, snum)) +return -EPERM ; +if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))return -EACCES ;
@@ -1061,6 +1064,8 @@ static int sctp_connect_new_asoc(struct sctp_endpoint *ep,if (sctp_autobind (sk))return -EAGAIN ; }else { +if (inet_is_local_unbindable_port (net, ep->base.bind_addr.port)) +return -EPERM ;if (inet_port_requires_bind_service (net, ep->base.bind_addr.port) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))return -EACCES ;
Signature · 1 line
2.24.0.432.g9d3f5f5b63-goog
Quoted 267 lines · Maciej Żenczykowski
From: Maciej Żenczykowski <[email protected]>
and associated
inet_is_local_unbindable_port()helper function:
use it to make explicitly binding to an unbindable port return-EPERM'Operation not permitted'.Autobind doesn't honour this new sysctl since:
(a) you can simply set both if that's the behaviour you desire
(b) there could be a use for preventing explicit while allowing auto
(c) it's faster in the relatively critical path of doing port
selection
duringconnect()to only check one bitmap instead of bothVarious ports may have special use cases which are not suitable for use by general userspace applications. Currently, ports specified in
ip_local_reserved_portssysctl will not be returned only in case of automatic port assignment, but nothing prevents you from explicitly binding to them - even from an entirely unprivileged process.In certain cases it is desirable to prevent the host from assigning the ports even in case of explicit binds, even from superuser processes.
Example use cases might be:
- a port being stolen by the nic for remote serial console, remote power management or some other sort of debugging functionality (crash collection, gdb, direct access to some other microcontroller on the nic or motherboard, remote management of the nic itself).
- a transparent proxy where packets are being redirected: in case a socket matches this connection, packets from this application would be incorrectly sent to one of the endpoints.Initially I wanted to solve this problem via the simple one line:
static inline bool
inet_port_requires_bind_service(struct net *net, unsigned short port){(quoted diff) +1
-1- return port < net->ipv4.sysctl_ip_prot_sock; + return port < net->ipv4.sysctl_ip_prot_sock ||
inet_is_local_reserved_port(net, port);
}However, this doesn't work for two reasons:
(a) it changes userspace visible behaviour of the existing local reserved ports sysctl, and there appears to be enough
documentation
on the internet talking about setting it to make this a bad idea
(b) it doesn't prevent privileged apps from using these ports,CAP_BIND_SERVICEis relatively likely to be available to, for
example,
a recursive DNS server so it can listed on port 53, which also
needs
to do src port randomization for outgoing queries due to security reasons (and it thus does manual port binding).If we *know* that certain ports are simply unusable, then it's better nothing even gets the opportunity to try to use them. This way we at least get a quick failure, instead of some sort of timeout (or possibly even corruption of the data stream of the non-kernel based use case).
Test:
vm:~# cat /proc/sys/net/ipv4/ip_local_unbindable_portsvm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0); s.bind(("::", 3967))' vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0); s.bind(("::", 3967))' vm:~# echo 3967 > /proc/sys/net/ipv4/ip_local_unbindable_ports vm:~# cat /proc/sys/net/ipv4/ip_local_unbindable_ports 3967 vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0); s.bind(("::", 3967))' socket.error: (1, 'Operation not permitted') vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0); s.bind(("::", 3967))' socket.error: (1, 'Operation not permitted')Cc Sean Tranchetti <[email protected]>Cc Subash Abhinov Kasiviswanathan <[email protected]>Cc Eric Dumazet <[email protected]>Cc Linux SCTP <[email protected]>Signed-off-by Maciej Żenczykowski <[email protected]>--- Documentation/networking/ip-sysctl.txt | 13 +++++++++++++ include/net/ip.h | 12 ++++++++++++ include/net/netns/ipv4.h | 1 + net/ipv4/af_inet.c | 4 ++++ net/ipv4/sysctl_net_ipv4.c | 18 ++++++++++++++++-- net/ipv6/af_inet6.c | 2 ++ net/sctp/socket.c | 5 +++++ 7 files changed, 53 insertions(+), 2 deletions(-)Documentation/networking/ip-sysctl.txt +0
-0@@ -940,6 +940,19 @@ ip_local_reserved_ports - list of comma separatedranges
Documentation/networking/ip-sysctl.txt +13
-0Default: Empty +ip_local_unbindable_ports - list of comma separated ranges + Specify the ports which are not directly bind()able. + + Usually you would use this to block the use of ports which + are invalid due to something outside of the control of the + kernel. For example a port stolen by the nic for serial + console, remote power management or debugging. + + There's a relatively high chance you will also want to list + these ports in 'ip_local_reserved_ports' to prevent autobinding. + + Default: Empty + ip_unprivileged_port_start - INTEGER This is a per-namespace sysctl. It defines the first unprivileged port in the network namespace. Privileged portsinclude/net/ip.h +0
-0@@ -346,6 +346,13 @@ static inline boolinet_is_local_reserved_port(struct net *net, unsigned short p
include/net/ip.h +1
-0return test_bit (port, net->ipv4.sysctl_local_reserved_ports); } +static inline bool inet_is_local_unbindable_port(struct net *net,unsigned short port)
include/net/ip.h +6
-0+{ +if (!net->ipv4.sysctl_local_unbindable_ports) +return false; +return test_bit (port, net->ipv4.sysctl_local_unbindable_ports); +} +static inline bool sysctl_dev_name_is_allowed (const char *name) {return strcmp (name,"default" ) !=0 &&strcmp (name,"all" ) !=0 ;@@ -362,6 +369,11 @@ static inline boolinet_is_local_reserved_port(struct net *net, unsigned short p
include/net/ip.h +1
-0return false; } +static inline bool inet_is_local_unbindable_port(struct net *net,unsigned short port)
include/net/ip.h +4
-0+{ +return false; +} + static inline bool inet_port_requires_bind_service(struct net *net,unsigned short port)
{include/net/ip.h +0
-0return port <PROT_SOCK ;include/net/netns/ipv4.h +1
-0@@ -197,6 +197,7 @@ struct netns_ipv4 {#ifdef CONFIG_SYSCTL unsigned long *sysctl_local_reserved_ports; +unsigned long *sysctl_local_unbindable_ports;int sysctl_ip_prot_sock;#endif net/ipv4/af_inet.c +4
-0@@ -494,6 +494,10 @@ int __inet_bind(struct sock *sk, struct sockaddr *uaddr, int addr_len, goto out; snum =ntohs (addr->sin_port); + err = -EPERM ; +if (snum &&inet_is_local_unbindable_port (net, snum)) + goto out; + err = -EACCES ; if (snum && inet_port_requires_bind_service(net, snum) &&net/ipv4/af_inet.c +0
-0!ns_capable(net->user_ns, CAP_NET_BIND_SERVICE))net/ipv4/sysctl_net_ipv4.c +15
-2@@ -745,6 +745,13 @@ static struct ctl_table ipv4_net_table[] = { .mode = 0644, .proc_handler = proc_do_large_bitmap, }, + { + .procname = "ip_local_unbindable_ports", + .data = &init_net.ipv4.sysctl_local_unbindable_ports, + .maxlen = 65536, + .mode = 0644, + .proc_handler = proc_do_large_bitmap, + }, { .procname = "ip_no_pmtu_disc", .data = &init_net.ipv4.sysctl_ip_no_pmtu_disc,@@ -1353,11 +1360,17 @@ static __net_init int ipv4_sysctl_init_net(struct net *net) net->ipv4.sysctl_local_reserved_ports = kzalloc(65536 / 8,GFP_KERNEL );if (!net->ipv4.sysctl_local_reserved_ports) - goto err_ports; + goto err_reserved_ports; + + net->ipv4.sysctl_local_unbindable_ports =kzalloc (65536 /8 ,GFP_KERNEL ); +if (!net->ipv4.sysctl_local_unbindable_ports) + goto err_unbindable_ports;return 0 ; -err_ports: +err_unbindable_ports: +kfree (net->ipv4.sysctl_local_reserved_ports); +err_reserved_ports:net/ipv4/sysctl_net_ipv4.c +1
-0unregister_net_sysctl_table (net->ipv4.ipv4_hdr); err_reg:if (!net_eq (net, &init_net))@@ -1370,6 +1383,7 @@ static __net_exit void ipv4_sysctl_exit_net(struct net *net) {struct ctl_table *table; +kfree (net->ipv4.sysctl_local_unbindable_ports);kfree (net->ipv4.sysctl_local_reserved_ports); table = net->ipv4.ipv4_hdr->ctl_table_arg;net/ipv4/sysctl_net_ipv4.c +0
-0unregister_net_sysctl_table (net->ipv4.ipv4_hdr);net/ipv6/af_inet6.c +2
-0@@ -292,6 +292,8 @@ static int __inet6_bind(struct sock *sk, struct sockaddr *uaddr, int addr_len, return -EINVAL; snum =ntohs (addr->sin6_port); +if (snum &&inet_is_local_unbindable_port (net, snum)) +return -EPERM ;if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))net/ipv6/af_inet6.c +0
-0return -EACCES ;net/sctp/socket.c +3
-0@@ -384,6 +384,9 @@ static int sctp_do_bind(struct sock *sk, union sctp_addr *addr, int len) } } +if (snum &&inet_is_local_unbindable_port (net, snum)) +return -EPERM ; +if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))net/sctp/socket.c +2
-0return -EACCES ;@@ -1061,6 +1064,8 @@ static int sctp_connect_new_asoc(struct sctp_endpoint *ep, if (sctp_autobind(sk)) return -EAGAIN; }else { +if (inet_is_local_unbindable_port (net, ep->base.bind_addr.port)) +return -EPERM ;if (inet_port_requires_bind_service (net, ep->base.bind_addr.port) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))
Quoted 1 line
return
-EACCES;
Thanks Maciej.
This works fine for me (seeing some minor merge conflicts on net-next but applies
fine on net).
Marcelo Ricardo Leitner
maintainerQuoted 11 lines · Maciej Żenczykowski
From: Maciej Żenczykowski <[email protected]>
and associated
inet_is_local_unbindable_port()helper function:
use it to make explicitly binding to an unbindable port return-EPERM'Operation not permitted'.Autobind doesn't honour this new sysctl since:
(a) you can simply set both if that's the behaviour you desire
(b) there could be a use for preventing explicit while allowing auto
(c) it's faster in the relatively critical path of doing port selection duringconnect()to only check one bitmap instead of both
...
Quoted 4 lines
If we *know* that certain ports are simply unusable, then it's better nothing even gets the opportunity to try to use them. This way we at least get a quick failure, instead of some sort of timeout (or possibly even corruption of the data stream of the non-kernel based use case).
This is doable with SELinux today, no?
Quoted 13 lines · Marcelo Ricardo Leitner
Quoted 11 lines · Maciej Żenczykowski
From: Maciej Żenczykowski <[email protected]>
and associated
inet_is_local_unbindable_port()helper function:
use it to make explicitly binding to an unbindable port return-EPERM'Operation not permitted'.Autobind doesn't honour this new sysctl since:
(a) you can simply set both if that's the behaviour you desire
(b) there could be a use for preventing explicit while allowing auto
(c) it's faster in the relatively critical path of doing port selection duringconnect()to only check one bitmap instead of both
Quoted 7 lines
...
Quoted 4 lines
If we *know* that certain ports are simply unusable, then it's better nothing even gets the opportunity to try to use them. This way we at least get a quick failure, instead of some sort of timeout (or possibly even corruption of the data stream of the non-kernel based use case).
This is doable with SELinux today, no?
Perhaps, but SELinux isn't used by many distros, including the servers where I have nics that steal some ports. It's also much much more difficult, requiring a policy, compilers, etc... and it gets even more complex if you need to dynamically modify the set of ports, which requires extra tools and runtime permissions.
David Miller
maintainerFrom: Maciej ¯enczykowski <[email protected]>
Date: Wed, 27 Nov 2019 12:50:39 -0800
Quoted 21 lines
Quoted 19 lines · Marcelo Ricardo Leitner
Quoted 11 lines · Maciej ¯enczykowski
From: Maciej ¯enczykowski <[email protected]>
and associated
inet_is_local_unbindable_port()helper function:
use it to make explicitly binding to an unbindable port return-EPERM'Operation not permitted'.Autobind doesn't honour this new sysctl since:
(a) you can simply set both if that's the behaviour you desire
(b) there could be a use for preventing explicit while allowing auto
(c) it's faster in the relatively critical path of doing port selection duringconnect()to only check one bitmap instead of both...
Quoted 4 lines
If we *know* that certain ports are simply unusable, then it's better nothing even gets the opportunity to try to use them. This way we at least get a quick failure, instead of some sort of timeout (or possibly even corruption of the data stream of the non-kernel based use case).
Quoted 7 lines
Quoted 1 line
This is doable with SELinux today, no?
Perhaps, but SELinux isn't used by many distros, including the servers where I have nics that steal some ports. It's also much much more difficult, requiring a policy, compilers, etc... and it gets even more complex if you need to dynamically modify the set of ports, which requires extra tools and runtime permissions.
I can see both sides of this argument, but anyways this is a new features and thus net-next material. It's nice to keep this discussion going, of course, but if this trends in the positive you still need to resubmit this when net-next opens back up.
Thanks.
Marcelo Ricardo Leitner
maintainerQuoted 21 lines · Maciej Żenczykowski
Quoted 19 lines · Marcelo Ricardo Leitner
Quoted 11 lines · Maciej Żenczykowski
From: Maciej Żenczykowski <[email protected]>
and associated
inet_is_local_unbindable_port()helper function:
use it to make explicitly binding to an unbindable port return-EPERM'Operation not permitted'.Autobind doesn't honour this new sysctl since:
(a) you can simply set both if that's the behaviour you desire
(b) there could be a use for preventing explicit while allowing auto
(c) it's faster in the relatively critical path of doing port selection duringconnect()to only check one bitmap instead of both...
Quoted 4 lines
If we *know* that certain ports are simply unusable, then it's better nothing even gets the opportunity to try to use them. This way we at least get a quick failure, instead of some sort of timeout (or possibly even corruption of the data stream of the non-kernel based use case).
Quoted 7 lines
Quoted 1 line
This is doable with SELinux today, no?
Perhaps, but SELinux isn't used by many distros, including the servers where I have nics that steal some ports. It's also much much more difficult, requiring a policy, compilers, etc... and it gets even more complex if you need to dynamically modify the set of ports, which requires extra tools and runtime permissions.
I'm no SELinux expert, but my /etc/ssh/sshd_config has this nice handy comment:
# If you want to change the port on a SELinux system, you have to tell # SELinux about this change.
# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
The kernel has no specific knowledge of 'ssh_port_t' and all I need to do to allow such port, is run the command above. No compiler, etc. The distribution would have to have a policy, say, 'unbindable_ports_t', and it could work similarly, I suppose, but I have no knowledge on this part.
As a reference only,
# semanage port -l gives a great list of ports that daemons are supposed to be using, and it supports ranges and so, like: amqp_port_t tcp 15672, 5671-5672 gluster_port_t tcp 38465-38469, 24007-24027
On not having SELinux enabled, you got me there. I not really willing to enter a "to do SELinux or not" discussion. :-)
Quoted 11 lines
Quoted 5 lines
Perhaps, but SELinux isn't used by many distros, including the servers where I have nics that steal some ports. It's also much much more difficult, requiring a policy, compilers, etc... and it gets even more complex if you need to dynamically modify the set of ports, which requires extra tools and runtime permissions.
I'm no SELinux expert, but my
/etc/ssh/sshd_confighas this nice handy comment:
# If you want to change the port on a SELinux system, you have to tell # SELinux about this change.# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
Right, so I'm also not at all an SELinux expert.
But: I run Fedora as my preferred distro of choice and it is of course SELinux enabled (and has been for many many years now) and I'm aware of that ssh port magic, precisely because I sometimes run ssh on a port different than 22.
I'm also working full time on Android, where I bash my head against SELinux quite regularly ;-)
Quoted 2 lines
The kernel has no specific knowledge of '
ssh_port_t' and all I need to do to allow such port, is run the command above.
I don't think that's actually true. I believe somewhere in the SELinux policy there is a reference to a set named ssh_port_t, some binary number assigned to it, and a default set of ports (ie. 22).
semanage, looks up 'ssh_port_t' maps it to some number X and tells the kernel that port P needs to be added to set X, and it needs super privileges to do so, because obviously managing the SELinux policy is about as privileged as you can get. And then it stores that P should be in X somewhere on disk so it survives reboot. I don't know if along the way it reloaded the entire policy or just a subset.
strace'ing semanage seems to roughly confirm this (although there's tons and tons of output...)
Quoted 1 line
No compiler, etc.
The compiler is present on fedora at least. I don't know if semanage invokes it or not, obviously at some point it was invoked.
Quoted 3 lines
The distribution would have to have a policy, say, '
unbindable_ports_t', and it could work similarly, I suppose, but I have no knowledge on this part.
Yes, the OS image has to include 'semanage' and all the selinux tooling, and has to ship a policy which has the unbindable_ports_t already defined in it...
But here's I think the fundamental problem with an SELinux approach. By default SELinux is deny all. All you can do is grant extra privs. I'm not aware of a way to actually say system wide disallow X. 'neverallow' is a compile time policy enforcement hack, and isn't (afaik) translated into anything that's actually given to the kernel.
So really instead you'd need to create a port set for every app - including a default , and then make sure that whatever port you want to block is excluded from each of those sets.
ie.
[root@gaia ~]# semanage port -l | egrep '32768|65535'
ephemeral_port_t tcp 32768-60999 ephemeral_port_t udp 32768-60999 unreserved_port_t sctp 1024-65535 unreserved_port_t tcp 61001-65535, 1024-32767 unreserved_port_t udp 61001-65535, 1024-32767
You need to iterate through all port sets, and remove port P from each of them in turn.
(can port sets overlap? not sure... do ports need to be in some set? perhaps also need to add the port to the unbindable set or something)
It's certainly doable, but it's a *lot* of work.
If you're on Fedora... then this isn't a huge problem, because someone did most of the work for you already... But, on any other distro?
And let's not even get started wrt. interactions with network namespaces.
I also don't know of a way to allow changes to selinux for NET_ADMIN (correct me if I'm wrong),
I also don't immediately know how to do this with /proc settings, but it seems like it should be achievable either via some additional patches, or via some clever boot-time network namespace creation hackery (on Android I imagine the answer would involve some sort of selinux tagging of the appropriate sysctl file in order to allow netd to change it).
Quoted 2 lines
On not having SELinux enabled, you got me there. I not really willing to enter a "to do SELinux or not" discussion. :-)
:-)
I'm of the opinion that SELinux and other security policy modules should be reserved for things related to system wide security policy. Not for things that are more along the lines of 'functionality'.
Also selinux has 'permissive' mode which causes the system to ignore all selinux access controls (in favour of just logging) and this is what is commonly used during development (because it's such a pain to work with).
This could also technically be probably done via bpf syscall filters, or bpf cgroup hooks...
But those approaches have performance implications and require a huge amount of machinery and complexity to manage.
Marcelo Ricardo Leitner
maintainerOn Fri, Nov 29, 2019 at 09:00:19PM +0100, Maciej Żenczykowski wrote:
...
Quoted 3 lines
I'm of the opinion that SELinux and other security policy modules should be reserved for things related to system wide security policy. Not for things that are more along the lines of 'functionality'.
Makes sense.
Quoted 5 lines
Also selinux has 'permissive' mode which causes the system to ignore all selinux access controls (in favour of just logging) and this is what is commonly used during development (because it's such a pain to work with).
Agree, this would be a big problem.
IOW, "you don't have permission to access to this" != "you just can't use this, no matter what"
FWIW, I rest my case :-)
Thanks,
Marcelo
Since net-next is open, I'm resending, with just Cc/reviewed-by updates - code is the same.
PATCH v2 1/1 net: introduce ip_local_unbindable_ports sysctl
From: Maciej Żenczykowski <[email protected]>
and associated inet_is_local_unbindable_port() helper function:
use it to make explicitly binding to an unbindable port return -EPERM 'Operation not permitted'.
Autobind doesn't honour this new sysctl since:
(a) you can simply set both if that's the behaviour you desire
(b) there could be a use for preventing explicit while allowing auto
(c) it's faster in the relatively critical path of doing port selection during connect() to only check one bitmap instead of both
Various ports may have special use cases which are not suitable for use by general userspace applications. Currently, ports specified in ip_local_reserved_ports sysctl will not be returned only in case of automatic port assignment, but nothing prevents you from explicitly binding to them - even from an entirely unprivileged process.
In certain cases it is desirable to prevent the host from assigning the ports even in case of explicit binds, even from superuser processes.
Example use cases might be:
- a port being stolen by the nic for remote serial console, remote power management or some other sort of debugging functionality (crash collection, gdb, direct access to some other microcontroller on the nic or motherboard, remote management of the nic itself).
- a transparent proxy where packets are being redirected: in case a socket matches this connection, packets from this application would be incorrectly sent to one of the endpoints.
Initially I wanted to solve this problem via the simple one line:
static inline bool inet_port_requires_bind_service(struct net *net, unsigned short port) {
- return port < net->ipv4.sysctl_ip_prot_sock;
+ return port < net->ipv4.sysctl_ip_prot_sock || inet_is_local_reserved_port(net, port);
}
However, this doesn't work for two reasons:
(a) it changes userspace visible behaviour of the existing local reserved ports sysctl, and there appears to be enough documentation on the internet talking about setting it to make this a bad idea
(b) it doesn't prevent privileged apps from using these ports, CAP_BIND_SERVICE is relatively likely to be available to, for example, a recursive DNS server so it can listed on port 53, which also needs to do src port randomization for outgoing queries due to security reasons (and it thus does manual port binding).
If we *know* that certain ports are simply unusable, then it's better nothing even gets the opportunity to try to use them. This way we at least get a quick failure, instead of some sort of timeout (or possibly even corruption of the data stream of the non-kernel based use case).
Test:
vm:~# cat /proc/sys/net/ipv4/ip_local_unbindable_ports
vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0); s.bind(("::", 3967))'
vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0); s.bind(("::", 3967))'
vm:~# echo 3967 > /proc/sys/net/ipv4/ip_local_unbindable_ports
vm:~# cat /proc/sys/net/ipv4/ip_local_unbindable_ports
3967
vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0); s.bind(("::", 3967))'
socket.error: (1, 'Operation not permitted')
vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0); s.bind(("::", 3967))'
socket.error: (1, 'Operation not permitted')--- Documentation/networking/ip-sysctl.txt | 13 +++++++++++++ include/net/ip.h | 12 ++++++++++++ include/net/netns/ipv4.h | 1 + net/ipv4/af_inet.c | 4 ++++ net/ipv4/sysctl_net_ipv4.c | 18 ++++++++++++++++-- net/ipv6/af_inet6.c | 2 ++ net/sctp/socket.c | 5 +++++ 7 files changed, 53 insertions(+), 2 deletions(-)
Documentation/networking/ip-sysctl.txt +13 -0
@@ -940,6 +940,19 @@ ip_local_reserved_ports - list of comma separated ranges Default: Empty +ip_local_unbindable_ports - list of comma separated ranges + Specify the ports which are not directly bind()able. + + Usually you would use this to block the use of ports which + are invalid due to something outside of the control of the + kernel. For example a port stolen by the nic for serial + console, remote power management or debugging. + + There's a relatively high chance you will also want to list + these ports in 'ip_local_reserved_ports' to prevent autobinding. + + Default: Empty + ip_unprivileged_port_start - INTEGER This is a per-namespace sysctl. It defines the first unprivileged port in the network namespace. Privileged ports
include/net/ip.h +12 -0
@@ -346,6 +346,13 @@ static inline bool inet_is_local_reserved_port(struct net *net, unsigned short preturn test_bit (port, net->ipv4.sysctl_local_reserved_ports); } +static inline bool inet_is_local_unbindable_port (struct net *net,unsigned short port) +{ +if (!net->ipv4.sysctl_local_unbindable_ports) +return false; +return test_bit (port, net->ipv4.sysctl_local_unbindable_ports); +} +static inline bool sysctl_dev_name_is_allowed (const char *name) {return strcmp (name,"default" ) !=0 &&strcmp (name,"all" ) !=0 ;
@@ -362,6 +369,11 @@ static inline bool inet_is_local_reserved_port(struct net *net, unsigned short preturn false; } +static inline bool inet_is_local_unbindable_port (struct net *net,unsigned short port) +{ +return false; +} +static inline bool inet_port_requires_bind_service (struct net *net,unsigned short port) {return port <PROT_SOCK ;
include/net/netns/ipv4.h +1 -0
@@ -197,6 +197,7 @@ struct netns_ipv4 {#ifdef CONFIG_SYSCTL unsigned long *sysctl_local_reserved_ports; +unsigned long *sysctl_local_unbindable_ports;int sysctl_ip_prot_sock;#endif
net/ipv4/af_inet.c +4 -0
@@ -494,6 +494,10 @@ int __inet_bind(struct sock *sk, struct sockaddr *uaddr, int addr_len, goto out; snum =ntohs (addr->sin_port); + err = -EPERM ; +if (snum &&inet_is_local_unbindable_port (net, snum)) + goto out; + err = -EACCES ;if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))
net/ipv4/sysctl_net_ipv4.c +16 -2
@@ -745,6 +745,13 @@ static struct ctl_table ipv4_net_table[] = { .mode = 0644, .proc_handler = proc_do_large_bitmap, }, + { + .procname = "ip_local_unbindable_ports", + .data = &init_net.ipv4.sysctl_local_unbindable_ports, + .maxlen = 65536, + .mode = 0644, + .proc_handler = proc_do_large_bitmap, + }, { .procname = "ip_no_pmtu_disc", .data = &init_net.ipv4.sysctl_ip_no_pmtu_disc,
@@ -1353,11 +1360,17 @@ static __net_init int ipv4_sysctl_init_net(struct net *net) net->ipv4.sysctl_local_reserved_ports =kzalloc (65536 /8 ,GFP_KERNEL );if (!net->ipv4.sysctl_local_reserved_ports) - goto err_ports; + goto err_reserved_ports; + + net->ipv4.sysctl_local_unbindable_ports =kzalloc (65536 /8 ,GFP_KERNEL ); +if (!net->ipv4.sysctl_local_unbindable_ports) + goto err_unbindable_ports;return 0 ; -err_ports: +err_unbindable_ports: +kfree (net->ipv4.sysctl_local_reserved_ports); +err_reserved_ports:unregister_net_sysctl_table (net->ipv4.ipv4_hdr); err_reg:if (!net_eq (net, &init_net))
@@ -1370,6 +1383,7 @@ static __net_exit void ipv4_sysctl_exit_net(struct net *net) {struct ctl_table *table; +kfree (net->ipv4.sysctl_local_unbindable_ports);kfree (net->ipv4.sysctl_local_reserved_ports); table = net->ipv4.ipv4_hdr->ctl_table_arg;unregister_net_sysctl_table (net->ipv4.ipv4_hdr);
net/ipv6/af_inet6.c +2 -0
@@ -292,6 +292,8 @@ static int __inet6_bind(struct sock *sk, struct sockaddr *uaddr, int addr_len,return -EINVAL ; snum =ntohs (addr->sin6_port); +if (snum &&inet_is_local_unbindable_port (net, snum)) +return -EPERM ;if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))return -EACCES ;
net/sctp/socket.c +5 -0
@@ -384,6 +384,9 @@ static int sctp_do_bind(struct sock *sk, union sctp_addr *addr, int len) } } +if (snum &&inet_is_local_unbindable_port (net, snum)) +return -EPERM ; +if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))return -EACCES ;
@@ -1061,6 +1064,8 @@ static int sctp_connect_new_asoc(struct sctp_endpoint *ep,if (sctp_autobind (sk))return -EAGAIN ; }else { +if (inet_is_local_unbindable_port (net, ep->base.bind_addr.port)) +return -EPERM ;if (inet_port_requires_bind_service (net, ep->base.bind_addr.port) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))return -EACCES ;
Signature · 1 line
2.24.0.393.g34dc348eaf-goog
Quoted 5 lines · Maciej Å»enczykowski
Example use cases might be:
- a port being stolen by the nic for remote serial console, remote power management or some other sort of debugging functionality (crash collection, gdb, direct access to some other microcontroller on the nic or motherboard, remote management of the nic itself).
This use case makes me a little uncomfortable.
Could you elaborate what protocols and products are in need of this functionality?
Why can't the NIC just get its own IP like it usually does with NCSI?
Quoted 2 lines
Could you elaborate what protocols and products are in need of this functionality?
The ones I'm aware of are:
(a) Google's servers
(b) Android on at least some chipsets (Qualcomm at the bare minimum, but I think it's pretty standard a solution) where there's a complex port sharing scheme between the Linux kernel on the Application Processor and the Firmware running on the modem (for ipv4 we only get one ip address from the cellular carrier). It's basically required for things like wifi calling to work.
Quoted 1 line
Why can't the NIC just get its own IP like it usually does with NCSI?
Because often these nics are deployed as in place upgrades in environments where there's a limited number of IPs.
Say a rack with a /27 ipv4 subnet (2**5 = 32 -> 29 usable ips, since network/broadcast/gateway are burned) and 15+ pre-existing machines. This means there's not enough IPs to assign separate ones for the nics. Renumbering the rack, would imply renumbering the datacenter, etc... And ipv4 - even RFC1918 - has long run out - so even in new deployments there's not enough IPv4 ips to give to nics, and IPv6 isn't yet deployed *everywhere*.
Quoted 11 lines · Maciej Å»enczykowski
Quoted 2 lines
Could you elaborate what protocols and products are in need of this functionality?
The ones I'm aware of are:
(a) Google's servers
(b) Android on at least some chipsets (Qualcomm at the bare minimum, but I think it's pretty standard a solution) where there's a complex port sharing scheme between the Linux kernel on the Application Processor and the Firmware running on the modem (for ipv4 we only get one ip address from the cellular carrier). It's basically required for things like wifi calling to work.
Okay, that's what I was suspecting. It'd be great if the real motivation for a patch was spelled out in the commit message :/
So some SoCs which run non-vanilla kernels require hacks to steal ports from the networking stack for use by proprietary firmware.
I don't see how merging this patch benefits the community.
Quoted 11 lines
Quoted 1 line
Why can't the NIC just get its own IP like it usually does with NCSI?
Because often these nics are deployed as in place upgrades in environments where there's a limited number of IPs.
Say a rack with a /27 ipv4 subnet (2**5 = 32 -> 29 usable ips, since network/broadcast/gateway are burned) and 15+ pre-existing machines. This means there's not enough IPs to assign separate ones for the nics. Renumbering the rack, would imply renumbering the datacenter, etc... And ipv4 - even RFC1918 - has long run out - so even in new deployments there's not enough IPv4 ips to give to nics, and IPv6 isn't yet deployed *everywhere*.
So the conditions for this are:
- in-place upgrade of an existing rack
- IPv4 only
- the existing servers didn't have NCSI or otherwise IPs for OOB control
Unlike the AP one this sounds like a very rare scenario..
Quoted 8 lines
Okay, that's what I was suspecting. It'd be great if the real motivation for a patch was spelled out in the commit message :/
So some SoCs which run non-vanilla kernels require hacks to steal ports from the networking stack for use by proprietary firmware.
I don't see how merging this patch benefits the community.
This is just a transparent proxy scenario though.
We block the specific ports so that there is no unrelated traffic belonging to host proxied here incorrectly.
Quoted 2 lines
Okay, that's what I was suspecting. It'd be great if the real motivation for a patch was spelled out in the commit message :/
It is, but the commit message is already extremely long. At some point essays and discussions belong in email and not in the commit message.
Here's another use case:
A network where firewall policy or network behaviour blocks all traffic using specific ports.
I've seen generic firewalls that unconditionally drop all BGP or SMTP port traffic, or all traffic on ports 5060/5061 (regardless of direction) or on 25/53/80/123/443/853/3128/8000/8080/8088/8888 (usually due to some ill guided security policies against sip or open proxies or xxx). If you happen to use port XXXX as your source port your connection just hangs (packets are blackholed).
Sure you can argue the network is broken, but in the real world you often can't fix it... Go try and convince your ISP that they should only drop inbound connections to port 8000, but not outgoing connections from port 8000 - you'll go crazy before you find someone who even understands what you're talking about - and even if you find such a person, they'll probably be too busy to change things - and even though it might be a 1 letter change (port -> dport) - it still might take months of testing and rollout before it's fully deployed.
I've seen networks where specific ports are automatically classified as super high priority (network control) so you don't want anything using these ports without very good reason (common for BGP for example, or for encap schemes).
Or a specific port number being reserved by GUE or other udp encap schemes and thus unsafe to use for generic traffic (because the network or even the kernel itself might for example auto decapsulate it [via tc ebpf for example], or parse the interior of the packet for flowhashing purposes...).
[I'll take this opportunity to point out that due to poor flow hashing behaviour GRE is basically unusable at scale (not to mention poorly extensible), and thus GUE and other UDP encap schemes are taking over]
Or you might want to forward udp port 4500 from your external IP to a dedicated ipsec box or some hardware offload engine... etc.
Quoted 3 lines
So some SoCs which run non-vanilla kernels require hacks to steal ports from the networking stack for use by proprietary firmware. I don't see how merging this patch benefits the community.
I think you're failing to account for the fact that the majority of Linux users are Android users - there's around 2.5 billion Android phones in the wild... - but perhaps you don't consider your users (or Android?) to be part of your community?
btw. Chrome OS is also Linux based (and if a quick google search is to be believed, about 1/7th of the linux desktop/laptop share), but since it supports running Android apps, it needs to have all Android specific generic kernel changes...
The reason Android runs non-vanilla kernels is *because* patches like this - that make Linux work in the real world - are missing from vanilla Linux
(I can think of a few other networking patches off the top of my head where we've been unable to upstream them for no particularly good reason).
Quoted 2 lines
So the conditions for this are:
- in-place upgrade of an existing rack
No that's just an example. That said in place upgrades aren't particularly rare.
Quoted 1 line
- IPv4 only
Believe it or not most embedded gear is still very much ipv4 only, as much as I hate that - I've been working on ipv6 deployment for over a decade now, and the amount of stuff that's IPv4-only is still staggering.
Quoted 1 line
Unlike the AP one this sounds like a very rare scenario..
And yet I have 2 ports for 2 different pieces of hardware that I need to block this way (and a third one for GUE and a fourth one for some ipsec-like crypto transport).
Quoted 11 lines · On Tue, 10 Dec 2019 07:00:29 +0000, [email protected] wrote:
Quoted 7 lines
Okay, that's what I was suspecting. It'd be great if the real motivation for a patch was spelled out in the commit message :/
So some SoCs which run non-vanilla kernels require hacks to steal ports from the networking stack for use by proprietary firmware.
I don't see how merging this patch benefits the community.
This is just a transparent proxy scenario though.
We block the specific ports so that there is no unrelated traffic belonging to host proxied here incorrectly.
It's a form of one, agreed, although let's be honest - someone reading the transparent proxy use case in the commit message will not think of a complex AP scenario, but rather of a locally configured transparent proxy with IPtables or sockets or such.
Quoted 4 lines · Maciej Å»enczykowski
Quoted 2 lines
Okay, that's what I was suspecting. It'd be great if the real motivation for a patch was spelled out in the commit message :/
It is, but the commit message is already extremely long.
Long, yet it doesn't mention the _real_ reason for the patch.
Quoted 2 lines
At some point essays and discussions belong in email and not in the commit message.
Ugh just admit you didn't mention the primary use case in the commit log, and we can move on.
Quoted 32 lines
Here's another use case:
A network where firewall policy or network behaviour blocks all traffic using specific ports.
I've seen generic firewalls that unconditionally drop all BGP or SMTP port traffic, or all traffic on ports 5060/5061 (regardless of direction) or on 25/53/80/123/443/853/3128/8000/8080/8088/8888 (usually due to some ill guided security policies against sip or open proxies or xxx). If you happen to use port XXXX as your source port your connection just hangs (packets are blackholed).
Sure you can argue the network is broken, but in the real world you often can't fix it... Go try and convince your ISP that they should only drop inbound connections to port 8000, but not outgoing connections from port 8000 - you'll go crazy before you find someone who even understands what you're talking about - and even if you find such a person, they'll probably be too busy to change things - and even though it might be a 1 letter change (port -> dport) - it still might take months of testing and rollout before it's fully deployed.
I've seen networks where specific ports are automatically classified as super high priority (network control) so you don't want anything using these ports without very good reason (common for BGP for example, or for encap schemes).
Or a specific port number being reserved by GUE or other udp encap schemes and thus unsafe to use for generic traffic (because the network or even the kernel itself might for example auto decapsulate it [via tc ebpf for example], or parse the interior of the packet for flowhashing purposes...).
Quoted 6 lines
[I'll take this opportunity to point out that due to poor flow hashing behaviour GRE is basically unusable at scale (not to mention poorly extensible), and thus GUE and other UDP encap schemes are taking over]
Or you might want to forward udp port 4500 from your external IP to a dedicated ipsec box or some hardware offload engine... etc.
It's networking you can concoct a scenario to justify anything.
Quoted 8 lines
Quoted 3 lines
So some SoCs which run non-vanilla kernels require hacks to steal ports from the networking stack for use by proprietary firmware. I don't see how merging this patch benefits the community.
I think you're failing to account for the fact that the majority of Linux users are Android users - there's around 2.5 billion Android phones in the wild... - but perhaps you don't consider your users (or Android?) to be part of your community?
I don't consider users of non-vanilla kernels to necessarily be a reason to merge patches upstream, no. They carry literally millions of lines of patches out of tree, let them carry this patch, too. If I can't boot a vanilla kernel on those devices, and clearly there is no intent by the device manufacturers for me to ever will, why would I care? Some companies care about upstream, and those should be rewarded by us taking some of the maintenance off their hands. Some don't:
https://www.youtube.com/watch?v=_36yNWw_07g (link to Linus+nVidia video) even tho they sell majority of SoCs for 2.5 billion devices.
Quoted 11 lines
btw. Chrome OS is also Linux based (and if a quick google search is to be believed, about 1/7th of the linux desktop/laptop share), but since it supports running Android apps, it needs to have all Android specific generic kernel changes...
The reason Android runs non-vanilla kernels is *because* patches like this - that make Linux work in the real world - are missing from vanilla Linux
(I can think of a few other networking patches off the top of my head where we've been unable to upstream them for no particularly good reason).
The way to get those patches upstream is to have a honest discussion about the use case so people can validate the design. Not by sending a patch with a 5 page commit message which fails to clearly state the motivation for the feature :/
Quoted 4 lines
It's a form of one, agreed, although let's be honest - someone reading the transparent proxy use case in the commit message will not think of a complex AP scenario, but rather of a locally configured transparent proxy with IPtables or sockets or such.
Transparent proxy could be implemented using eBPF + XDP and those don't need sockets. However, in that case we do need to block those specific ports
to avoid messing with unrelated traffic.
David Miller
maintainerFrom: Maciej ¯enczykowski <[email protected]>
Date: Tue, 10 Dec 2019 12:46:29 +0100
Quoted 2 lines
At some point essays and discussions belong in email and not in the commit message.
Wrong, full details on the context and impetus matter, no matter how voluminous.
If you put what you ate for breakfast in the commit log message I wouldn't complain.
Quoted 6 lines · Jakub Kicinski
I don't consider users of non-vanilla kernels to necessarily be a reason to merge patches upstream, no. They carry literally millions of lines of patches out of tree, let them carry this patch, too. If I can't boot a vanilla kernel on those devices, and clearly there is no intent by the device manufacturers for me to ever will, why would I care?
That's *not* the intent.
https://arstechnica.com/gadgets/2019/11/google-outlines-plans-for-mainline-linux-kernel-support-in-android/
Quoted 3 lines
Quoted 3 lines
The reason Android runs non-vanilla kernels is *because* patches like this - that make Linux work in the real world - are missing from vanilla Linux
That's exactly the point here. Saying, "Android will never use mainline, so why should mainline take their patches" is a self-fulfilling prophecy. Obviously, if mainline never takes Android patches, then yes, Android will never be able to use mainline. We do have an Android tree we can take this patch into. But we don't want to take it without at least attempting to get it into mainline first.
The use case here is pretty simple. There are many CPUs in a mobile phone. The baseband processor ("modem") implements much of the functionality required by cellular networks, so if you want cellular voice or data, it needs to be able to talk to the network. For many reasons (architectural, power conservation, security), the modem needs to be able to talk directly to the cellular network. This includes, for example, SIP/RTP media streams that go directly to the audio hardware, IKE traffic that is sent directly by the modem because only the modem has the keys, etc. Normally this happens directly on the cellular interface and Linux/Android is unaware of it. But, when using wifi calling (which is an IPsec tunnel over wifi to an endpoint inside the cellular network), the device only has one IPv4 address, and the baseband processor and the application processor (the CPU that runs Linux/Android) have to share it. This means that some ports have to be reserved so that the baseband processor can depend on using them. NAT cannot be used because the 3GPP standards require protocols that are not very NAT-friendly, and because the modem needs to be able to accept unsolicited inbound traffic.
Other than "commit message doesn't have a use case", are there technical concerns with this patch?
Quoted 11 lines · Marcelo Ricardo Leitner
I'm no SELinux expert, but my
/etc/ssh/sshd_confighas this nice handy comment:
# If you want to change the port on a SELinux system, you have to tell # SELinux about this change.# semanage port -a -t ssh_port_t -p tcp #PORTNUMBERThe kernel has no specific knowledge of '
ssh_port_t' and all I need to do to allow such port, is run the command above. No compiler, etc. The distribution would have to have a policy, say, 'unbindable_ports_t', and it could work similarly, I suppose, but I have no knowledge on this part.
For security reasons, Android does not allow reloading selinux policy after boot. I'm not a selinux expert either, but semanage-port(8) has:
-N, --noreload
Do not reload policy after commit
which suggests that it works by reloading the policy.
Quoted 41 lines · Lorenzo Colitti
Quoted 6 lines · Jakub Kicinski
I don't consider users of non-vanilla kernels to necessarily be a reason to merge patches upstream, no. They carry literally millions of lines of patches out of tree, let them carry this patch, too. If I can't boot a vanilla kernel on those devices, and clearly there is no intent by the device manufacturers for me to ever will, why would I care?
That's *not* the intent.
https://arstechnica.com/gadgets/2019/11/google-outlines-plans-for-mainline-linux-kernel-support-in-android/Quoted 3 lines
Quoted 3 lines
The reason Android runs non-vanilla kernels is *because* patches like this - that make Linux work in the real world - are missing from vanilla Linux
That's exactly the point here. Saying, "Android will never use mainline, so why should mainline take their patches" is a self-fulfilling prophecy. Obviously, if mainline never takes Android patches, then yes, Android will never be able to use mainline. We do have an Android tree we can take this patch into. But we don't want to take it without at least attempting to get it into mainline first.
The use case here is pretty simple. There are many CPUs in a mobile phone. The baseband processor ("modem") implements much of the functionality required by cellular networks, so if you want cellular voice or data, it needs to be able to talk to the network. For many reasons (architectural, power conservation, security), the modem needs to be able to talk directly to the cellular network. This includes, for example, SIP/RTP media streams that go directly to the audio hardware, IKE traffic that is sent directly by the modem because only the modem has the keys, etc. Normally this happens directly on the cellular interface and Linux/Android is unaware of it. But, when using wifi calling (which is an IPsec tunnel over wifi to an endpoint inside the cellular network), the device only has one IPv4 address, and the baseband processor and the application processor (the CPU that runs Linux/Android) have to share it. This means that some ports have to be reserved so that the baseband processor can depend on using them. NAT cannot be used because the 3GPP standards require protocols that are not very NAT-friendly, and because the modem needs to be able to accept unsolicited inbound traffic.
Quoted 2 lines
Other than "commit message doesn't have a use case", are there technical concerns with this patch?
Maybe a minor question or two, but the main complaint is the commit message.
How are the ports which get reserved communicated between the baseband and the AP? Is this part of the standard? Is the driver that talks to the base band in the user space and it knows which ports to reserve statically? Or does the modem dynamically request ports to reserve/inform the host of ports in use?
Should the sysfs interface make sure there are not existing sockets using requested ports which would stop working? If we may need it one day better add it now..
Quoted 5 lines · Jakub Kicinski
How are the ports which get reserved communicated between the baseband and the AP? Is this part of the standard? Is the driver that talks to the base band in the user space and it knows which ports to reserve statically? Or does the modem dynamically request ports to reserve/inform the host of ports in use?
I'm not an expert in that part of the system, but my understanding is that the primary way this is used is to pre-allocate a block of ports to be used by the modem on boot, before other applications can bind to ports. Subash, do you have more details?
Quoted 8 lines · Lorenzo Colitti
Quoted 5 lines · Jakub Kicinski
How are the ports which get reserved communicated between the baseband and the AP? Is this part of the standard? Is the driver that talks to the base band in the user space and it knows which ports to reserve statically? Or does the modem dynamically request ports to reserve/inform the host of ports in use?
Quoted 4 lines
I'm not an expert in that part of the system, but my understanding is that the primary way this is used is to pre-allocate a block of ports to be used by the modem on boot, before other applications can bind to ports. Subash, do you have more details?
AFAIK these ports are randomly picked and not from a standard. Userspace gets this information through qrtr during boot.
Atleast in our case, there cannot be any existing user of these ports since these ports are blocked prior to mobile connection establishment. We could call SOCK_DIAG_DESTROY on these ports from userspace as a precaution as applications would gracefully handle the socket errors.
Quoted 13 lines · On Thu, 12 Dec 2019 18:53:19 -0700, [email protected] wrote:
Quoted 11 lines · Lorenzo Colitti
Quoted 5 lines · Jakub Kicinski
How are the ports which get reserved communicated between the baseband and the AP? Is this part of the standard? Is the driver that talks to the base band in the user space and it knows which ports to reserve statically? Or does the modem dynamically request ports to reserve/inform the host of ports in use?
I'm not an expert in that part of the system, but my understanding is that the primary way this is used is to pre-allocate a block of ports to be used by the modem on boot, before other applications can bind to ports. Subash, do you have more details?
Quoted 5 lines
AFAIK these ports are randomly picked and not from a standard. Userspace gets this information through qrtr during boot.
Atleast in our case, there cannot be any existing user of these ports since these ports are blocked prior to mobile connection establishment.
Not even a listening socket?
Quoted 2 lines
We could call
SOCK_DIAG_DESTROYon these ports from userspace as a precaution as applications would gracefully handle the socket errors.
Right, or kernel could walk them, since presumably every application using this functionality should do it, anyway? But no strong feeling on this if nobody else feels this is needed.
Quoted 247 lines · Maciej Żenczykowski
From: Maciej Żenczykowski <[email protected]>
and associated
inet_is_local_unbindable_port()helper function:
use it to make explicitly binding to an unbindable port return-EPERM'Operation not permitted'.Autobind doesn't honour this new sysctl since:
(a) you can simply set both if that's the behaviour you desire
(b) there could be a use for preventing explicit while allowing auto
(c) it's faster in the relatively critical path of doing port selection duringconnect()to only check one bitmap instead of bothVarious ports may have special use cases which are not suitable for use by general userspace applications. Currently, ports specified in
ip_local_reserved_portssysctl will not be returned only in case of automatic port assignment, but nothing prevents you from explicitly binding to them - even from an entirely unprivileged process.In certain cases it is desirable to prevent the host from assigning the ports even in case of explicit binds, even from superuser processes.
Example use cases might be:
- a port being stolen by the nic for remote serial console, remote power management or some other sort of debugging functionality (crash collection, gdb, direct access to some other microcontroller on the nic or motherboard, remote management of the nic itself).
- a transparent proxy where packets are being redirected: in case a socket matches this connection, packets from this application would be incorrectly sent to one of the endpoints.Initially I wanted to solve this problem via the simple one line:
static inline bool
inet_port_requires_bind_service(struct net *net, unsigned short port){(quoted diff) +1
-1- return port < net->ipv4.sysctl_ip_prot_sock; + return port < net->ipv4.sysctl_ip_prot_sock || inet_is_local_reserved_port(net, port);}
However, this doesn't work for two reasons:
(a) it changes userspace visible behaviour of the existing local reserved ports sysctl, and there appears to be enough documentation on the internet talking about setting it to make this a bad idea
(b) it doesn't prevent privileged apps from using these ports,CAP_BIND_SERVICEis relatively likely to be available to, for example, a recursive DNS server so it can listed on port 53, which also needs to do src port randomization for outgoing queries due to security reasons (and it thus does manual port binding).If we *know* that certain ports are simply unusable, then it's better nothing even gets the opportunity to try to use them. This way we at least get a quick failure, instead of some sort of timeout (or possibly even corruption of the data stream of the non-kernel based use case).
Test:
vm:~# cat /proc/sys/net/ipv4/ip_local_unbindable_portsvm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0); s.bind(("::", 3967))' vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0); s.bind(("::", 3967))' vm:~# echo 3967 > /proc/sys/net/ipv4/ip_local_unbindable_ports vm:~# cat /proc/sys/net/ipv4/ip_local_unbindable_ports 3967 vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, 0); s.bind(("::", 3967))' socket.error: (1, 'Operation not permitted') vm:~# python -c 'import socket; s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM, 0); s.bind(("::", 3967))' socket.error: (1, 'Operation not permitted')Cc Sean Tranchetti <[email protected]>Cc Subash Abhinov Kasiviswanathan <[email protected]>Cc Eric Dumazet <[email protected]>Cc Linux SCTP <[email protected]>Signed-off-by Maciej Żenczykowski <[email protected]>--- Documentation/networking/ip-sysctl.txt | 13 +++++++++++++ include/net/ip.h | 12 ++++++++++++ include/net/netns/ipv4.h | 1 + net/ipv4/af_inet.c | 4 ++++ net/ipv4/sysctl_net_ipv4.c | 18 ++++++++++++++++-- net/ipv6/af_inet6.c | 2 ++ net/sctp/socket.c | 5 +++++ 7 files changed, 53 insertions(+), 2 deletions(-)Documentation/networking/ip-sysctl.txt +13
-0@@ -940,6 +940,19 @@ ip_local_reserved_ports - list of comma separated ranges Default: Empty +ip_local_unbindable_ports - list of comma separated ranges + Specify the ports which are not directly bind()able. + + Usually you would use this to block the use of ports which + are invalid due to something outside of the control of the + kernel. For example a port stolen by the nic for serial + console, remote power management or debugging. + + There's a relatively high chance you will also want to list + these ports in 'ip_local_reserved_ports' to prevent autobinding. + + Default: Empty + ip_unprivileged_port_start - INTEGER This is a per-namespace sysctl. It defines the first unprivileged port in the network namespace. Privileged portsinclude/net/ip.h +12
-0@@ -346,6 +346,13 @@ static inline bool inet_is_local_reserved_port(struct net *net, unsigned short preturn test_bit (port, net->ipv4.sysctl_local_reserved_ports); } +static inline bool inet_is_local_unbindable_port (struct net *net,unsigned short port) +{ +if (!net->ipv4.sysctl_local_unbindable_ports) +return false; +return test_bit (port, net->ipv4.sysctl_local_unbindable_ports); +} +static inline bool sysctl_dev_name_is_allowed (const char *name) {return strcmp (name,"default" ) !=0 &&strcmp (name,"all" ) !=0 ;@@ -362,6 +369,11 @@ static inline bool inet_is_local_reserved_port(struct net *net, unsigned short preturn false; } +static inline bool inet_is_local_unbindable_port (struct net *net,unsigned short port) +{ +return false; +} +static inline bool inet_port_requires_bind_service (struct net *net,unsigned short port) {return port <PROT_SOCK ;include/net/netns/ipv4.h +1
-0@@ -197,6 +197,7 @@ struct netns_ipv4 {#ifdef CONFIG_SYSCTL unsigned long *sysctl_local_reserved_ports; +unsigned long *sysctl_local_unbindable_ports;int sysctl_ip_prot_sock;#endif net/ipv4/af_inet.c +4
-0@@ -494,6 +494,10 @@ int __inet_bind(struct sock *sk, struct sockaddr *uaddr, int addr_len, goto out; snum =ntohs (addr->sin_port); + err = -EPERM ; +if (snum &&inet_is_local_unbindable_port (net, snum)) + goto out; + err = -EACCES ;if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))net/ipv4/sysctl_net_ipv4.c +16
-2@@ -745,6 +745,13 @@ static struct ctl_table ipv4_net_table[] = { .mode = 0644, .proc_handler = proc_do_large_bitmap, }, + { + .procname = "ip_local_unbindable_ports", + .data = &init_net.ipv4.sysctl_local_unbindable_ports, + .maxlen = 65536, + .mode = 0644, + .proc_handler = proc_do_large_bitmap, + }, { .procname = "ip_no_pmtu_disc", .data = &init_net.ipv4.sysctl_ip_no_pmtu_disc,@@ -1353,11 +1360,17 @@ static __net_init int ipv4_sysctl_init_net(struct net *net) net->ipv4.sysctl_local_reserved_ports =kzalloc (65536 /8 ,GFP_KERNEL );if (!net->ipv4.sysctl_local_reserved_ports) - goto err_ports; + goto err_reserved_ports; + + net->ipv4.sysctl_local_unbindable_ports =kzalloc (65536 /8 ,GFP_KERNEL ); +if (!net->ipv4.sysctl_local_unbindable_ports) + goto err_unbindable_ports;return 0 ; -err_ports: +err_unbindable_ports: +kfree (net->ipv4.sysctl_local_reserved_ports); +err_reserved_ports:unregister_net_sysctl_table (net->ipv4.ipv4_hdr); err_reg:if (!net_eq (net, &init_net))@@ -1370,6 +1383,7 @@ static __net_exit void ipv4_sysctl_exit_net(struct net *net) {struct ctl_table *table; +kfree (net->ipv4.sysctl_local_unbindable_ports);kfree (net->ipv4.sysctl_local_reserved_ports); table = net->ipv4.ipv4_hdr->ctl_table_arg;unregister_net_sysctl_table (net->ipv4.ipv4_hdr);net/ipv6/af_inet6.c +2
-0@@ -292,6 +292,8 @@ static int __inet6_bind(struct sock *sk, struct sockaddr *uaddr, int addr_len,return -EINVAL ; snum =ntohs (addr->sin6_port); +if (snum &&inet_is_local_unbindable_port (net, snum)) +return -EPERM ;if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))return -EACCES ;net/sctp/socket.c +5
-0@@ -384,6 +384,9 @@ static int sctp_do_bind(struct sock *sk, union sctp_addr *addr, int len) } } +if (snum &&inet_is_local_unbindable_port (net, snum)) +return -EPERM ; +if (snum &&inet_port_requires_bind_service (net, snum) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))return -EACCES ;@@ -1061,6 +1064,8 @@ static int sctp_connect_new_asoc(struct sctp_endpoint *ep,if (sctp_autobind (sk))return -EAGAIN ; }else { +if (inet_is_local_unbindable_port (net, ep->base.bind_addr.port)) +return -EPERM ;if (inet_port_requires_bind_service (net, ep->base.bind_addr.port) && !ns_capable (net->user_ns,CAP_NET_BIND_SERVICE ))return -EACCES ;Signature · 1 line
2.24.0.432.g9d3f5f5b63-goog
Just out of curiosity, why are the portreserve and portrelease utilities not a solution to this use case?
Neil
Quoted 2 lines · On Fri, 13 Dec 2019, 20:49 Neil Horman, <[email protected]> wrote:
Just out of curiosity, why are the portreserve and portrelease utilities not a solution to this use case?
As I understand it, those utilities keep the ports reserved by binding to them so that no other process can. This doesn't work for Android because there are conformance tests that probe the device from the network and check that there are no open ports.
Quoted 9 lines · Lorenzo Colitti
Quoted 2 lines · On Fri, 13 Dec 2019, 20:49 Neil Horman, <[email protected]> wrote:
Just out of curiosity, why are the portreserve and portrelease utilities not a solution to this use case?
As I understand it, those utilities keep the ports reserved by binding to them so that no other process can. This doesn't work for Android because there are conformance tests that probe the device from the network and check that there are no open ports.
But you can address that with some augmentation to portreserve (i.e. just have it add an iptables rule to drop frames on that port, or respond with a port unreachable icmp message)
Neil
Quoted 8 lines · Neil Horman
Quoted 5 lines
As I understand it, those utilities keep the ports reserved by binding to them so that no other process can. This doesn't work for Android because there are conformance tests that probe the device from the network and check that there are no open ports.
But you can address that with some augmentation to portreserve (i.e. just have it add an iptables rule to drop frames on that port, or respond with a port unreachable icmp message)
There are also tests that run on device by inspecting /proc/net/{tcp,udp} to check that there are no open sockets. We'd have to change them as well.
But sure. It's not impossible to do this in userspace. We wouldn't use portreserve itself because the work to package it and make it work on Android (which has no /etc/services file), would likely be greater than just adding the code to an existing Android daemon (and because the reaction of the portreserve maintainers might be similar to yours:
"you don't need to add code to portreserve for this, just use a script that shells out to iptables").
But in any case, the result would be more complicated to use and maintain, and it would likely also be less realistic, such that a sophisticated conformance test might still find that the port was actually bound. Other users of the kernel wouldn't get to use this sysctl, and the userspace code can't be easily reused in other open-source projects, so the community gets nothing useful. That doesn't seem great.
Or, we could take this patch and maintain it in the Android kernel tree. Android kernels get a tiny bit further from mainline. Other uses of the kernel wouldn't get to use this sysctl, and again the community gets nothing useful. That doesn't seem great either.
Quoted 6 lines · Lorenzo Colitti
Quoted 5 lines · Neil Horman
Quoted 5 lines
As I understand it, those utilities keep the ports reserved by binding to them so that no other process can. This doesn't work for Android because there are conformance tests that probe the device from the network and check that there are no open ports.
Quoted 8 lines
Quoted 3 lines
But you can address that with some augmentation to portreserve (i.e. just have it add an iptables rule to drop frames on that port, or respond with a port unreachable icmp message)
There are also tests that run on device by inspecting
/proc/net/{tcp,udp}to check that there are no open sockets. We'd have to change them as well.
Ok, that seems reasonable.
Quoted 8 lines
But sure. It's not impossible to do this in userspace. We wouldn't use portreserve itself because the work to package it and make it work on Android (which has no
/etc/servicesfile), would likely be greater than just adding the code to an existing Android daemon (and because the reaction of the portreserve maintainers might be similar to yours:
"you don't need to add code to portreserve for this, just use a script that shells out to iptables").
Possibly, but sure, you could add the same functionality to some other existing daemon.
Quoted 4 lines
But in any case, the result would be more complicated to use and maintain, and it would likely also be less realistic, such that a sophisticated conformance test might still find that the port was actually bound.
One would think that a sufficiently sophisticated script could understand that a port was bound not for the purposes of use, but rather for the purposes of prevention of use by other processes. But I take your meaning, the fanout here starts to get large.
Quoted 5 lines
Other users of the kernel wouldn't get to use this sysctl, and the userspace code can't be easily reused in other open-source projects, so the community gets nothing useful. That doesn't seem great.
Well, that assumes you implement this in a non-open daemon, but thats your perogative.
Quoted 5 lines
Or, we could take this patch and maintain it in the Android kernel tree. Android kernels get a tiny bit further from mainline. Other uses of the kernel wouldn't get to use this sysctl, and again the community gets nothing useful. That doesn't seem great either.
That seems....agressive. I'm not saying this is a bad feature, I'm really just trying to think through how else this might be accomplished without the need to implement and maintain another sysctl.
FWIW, bpf offers hooks in both inet6_bind and inet_bind. Another option would be to implement a bfp program at each of those hooks that filtered on the set of blacklisted ports you want to prevent the use of. I'm not sure how wide the scope of this feature is for use, but if its limited to your use case, perhaps thats an alternative solution.
Neil
David Miller
maintainerFrom: Lorenzo Colitti <[email protected]>
Date: Thu, 19 Dec 2019 23:02:32 +0900
Quoted 7 lines
But in any case, the result would be more complicated to use and maintain, and it would likely also be less realistic, such that a sophisticated conformance test might still find that the port was actually bound. Other users of the kernel wouldn't get to use this sysctl, and the userspace code can't be easily reused in other open-source projects, so the community gets nothing useful. That doesn't seem great.
The same argument can be made about kernel changes that are only needed by Android because they refuse to use a userspace solution that frankly can do the job.
Can you see why these Android special case discussions are so frustrating for kernel devs?
And using the "we'll just have a local kernel change in the Android kernel" threat as leverage in the discussion... yeah very unpleasant indeed.