Bug with IPv6-UDP address binding

Hi NetDev

I think I have found a problem/bug with IPv6-UDP address binding.

I found this problem while playing with IPVS and IPv6-UDP, but its also present in more basic/normal situations.

If you have two IPv6 addresses, within the same IPv6 subnet, then one of the IPv6 addrs takes precedence over the other (for UDP only).

Meaning that, if connecting to the "secondary" IPv6 via UDP, will result in userspace see/bind the connection as being created to the "primary" IP, even-though tcpdump shows that the IPv6-UDP packets are dest the "secondary".

The result is; that only the first IPv6-UDP packet is delivered to userspace, and the next packets are denied by the kernel as the UDP socket is "established" with the "primary" IPv6 addr.

I would appreciate some hints to where in the IPv6 code I should look for this bug. If any one else wants to fix it, I'm also fine with that ;-)

Its quite easy to reproduce, using netcat (nc).

Add two addresses to the "server" e.g.:
ip addr add fee0:cafe::102/64 dev eth0
ip addr add fee0:cafe::bad/64 dev eth0

Run a netcat listener on "server":
nc -6 -u -l 2000
(Notice restart the listener between runs, due to limitation in nc)

On the client add an IPv6 addr e.g.:
ip addr add fee0:cafe::101/64 dev eth0

Run a netcat UDP-IPv6 producer on "client":
nc -6 -u fee0:cafe::bad 2000

Notice that first packet, will get through, but second packets will not (nc: Write error: Connection refused). Running a tcpdump shows that the kernel is sending back ICMP6, destination unreachable, unreachable port.

Its also possible to see the problem, simply running "netstat -uan" on "server", which will show that the "established" UDP connection, is bound to the wrong "Local Address".

(Tested on both latest net-next kernel at commit 79cda75a1, and also on RHEL6 approx 2.6.32)

Quoted 45 lines · Jesper Dangaard Brouer

Hi NetDev

I think I have found a problem/bug with IPv6-UDP address binding.

I found this problem while playing with IPVS and IPv6-UDP, but its also present in more basic/normal situations.

If you have two IPv6 addresses, within the same IPv6 subnet, then one of the IPv6 addrs takes precedence over the other (for UDP only).

Meaning that, if connecting to the "secondary" IPv6 via UDP, will result in userspace see/bind the connection as being created to the "primary" IP, even-though tcpdump shows that the IPv6-UDP packets are dest the "secondary".

The result is; that only the first IPv6-UDP packet is delivered to userspace, and the next packets are denied by the kernel as the UDP socket is "established" with the "primary" IPv6 addr.

I would appreciate some hints to where in the IPv6 code I should look for this bug. If any one else wants to fix it, I'm also fine with that ;-)

Its quite easy to reproduce, using netcat (nc).

Add two addresses to the "server" e.g.:
ip addr add fee0:cafe::102/64 dev eth0
ip addr add fee0:cafe::bad/64 dev eth0

Run a netcat listener on "server":
nc -6 -u -l 2000
(Notice restart the listener between runs, due to limitation in nc)

On the client add an IPv6 addr e.g.:
ip addr add fee0:cafe::101/64 dev eth0

Run a netcat UDP-IPv6 producer on "client":
nc -6 -u fee0:cafe::bad 2000

Notice that first packet, will get through, but second packets will not (nc: Write error: Connection refused). Running a tcpdump shows that the kernel is sending back ICMP6, destination unreachable, unreachable port.

Quoted 7 lines

Its also possible to see the problem, simply running "netstat -uan" on "server", which will show that the "established" UDP connection, is bound to the wrong "Local Address".

(Tested on both latest net-next kernel at commit 79cda75a1, and also on RHEL6 approx 2.6.32)

Hi Jesper

Thats because the "nc -6 -u -l 2000" on server does :

bind(3, {sa_family=AF_INET6, sin6_port=htons(2000), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0

recvfrom(3, "\n", 1024, MSG_PEEK, {sa_family=AF_INET6, sin6_port=htons(53696), inet_pton(AF_INET6, "fee0:cafe::101", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 1

connect(3, {sa_family=AF_INET6, sin6_port=htons(53696), inet_pton(AF_INET6, "fee0:cafe::101", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0

And the kernel automatically chooses a SOURCE address (fee0:cafe::102) that is not what you expected (fee0:cafe::bad)

So its a bug in the application.

UDP connect() is tricky : In this case, nc should learn on what IP address the client sent the frame. (using recvmsg() and appropriate ancillary message)

Then nc should bind a new socket on this address, then do the connect()

Yes, I guess its an application bug, because Berkely-derived implementations don't handle multihomeing well for UDP.

Why are we keeping this, counter-intuitive behavior?

What about changing the implementation to act like Solaris, which IMHO makes much more sense?

(BTW, iperf also have this "bug")

Quoted 3 lines · Eric Dumazet

On Wed, 2012-08-08 at 22:37 +0200, Jesper Dangaard Brouer wrote:

Then nc should bind a new socket on this address, then do the connect()

Yes, after the difficult extraction of the dest IP of the UDP packet.

Now I better understand, why the DNS server named/bind is so annoying, that is requires a restart after adding IPs. I guess they didn't implement this recvmsg(), and instead chooses to bind to all avail IPs on init/start.

Quoted 7 lines · Jesper Dangaard Brouer

Quoted 3 lines · Eric Dumazet

On Wed, 2012-08-08 at 22:37 +0200, Jesper Dangaard Brouer wrote:

Then nc should bind a new socket on this address, then do the connect()

Yes, after the difficult extraction of the dest IP of the UDP packet.

Thats 10 lines of code.

The hard part is in kernel actually ;)

Quoted 4 lines

Now I better understand, why the DNS server named/bind is so annoying, that is requires a restart after adding IPs. I guess they didn't implement this recvmsg(), and instead chooses to bind to all avail IPs on init/start.

Thats an implementation choice, no more no less.

Here is an IPv4 sample UDP application, able to echo packets with the IP source set to original DST address of the ping packet.

Doing the same on IPv6 is probably trivial as well

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <linux/udp.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <arpa/inet.h>

#define PORT 4040

int pktinfo_get(struct msghdr *my_hdr, struct in_pktinfo *pktinfo)

{
	int res = -1;

	if (my_hdr->msg_controllen > 0) {
		struct cmsghdr *get_cmsg;
		for (get_cmsg = CMSG_FIRSTHDR(my_hdr); get_cmsg;
			get_cmsg = CMSG_NXTHDR(my_hdr, get_cmsg)) {
			if (get_cmsg->cmsg_type == IP_PKTINFO) {
				struct in_pktinfo *get_pktinfo = (struct in_pktinfo *)CMSG_DATA(get_cmsg);
				memcpy(pktinfo, get_pktinfo, sizeof(*pktinfo));
				res = 0;
			}
		}
	}
	return res;
}

int main(int argc, char *argv[])

{
	int fd = socket(AF_INET, SOCK_DGRAM, 0);
	struct sockaddr_in addr, rem_addr;
	int res, on = 1;
	struct msghdr msghdr;
	struct iovec vec[1];
	char cbuf[512];
	char frame[4096];
	struct in_pktinfo pktinfo;
	int c, count = 1000000;

	while ((c = getopt(argc, argv, "c:")) != -1) {
		if (c == 'c') count = atoi(optarg);
		}
	memset(&addr, 0, sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_port = htons(PORT);
	if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) == -1) {
		perror("bind");
		return 1;
	}
	setsockopt(fd, SOL_IP, IP_PKTINFO, &on, sizeof(on));

	while (1) {
		memset(&msghdr, 0, sizeof(msghdr));
		msghdr.msg_control = cbuf;
		msghdr.msg_controllen = sizeof(cbuf);
		msghdr.msg_iov = vec;
		msghdr.msg_iovlen = 1;
		vec[0].iov_base = frame;
		vec[0].iov_len = sizeof(frame);
		msghdr.msg_name = &rem_addr;
		msghdr.msg_namelen = sizeof(rem_addr);
		res = recvmsg(fd, &msghdr, 0);
		if (res == -1)
			break;
		if (pktinfo_get(&msghdr, &pktinfo) == 0)
			printf("Got IP_PKTINFO dst addr=%s\n", inet_ntoa(pktinfo.ipi_spec_dst));

		/* ok, just echo reply this frame.
		 * Using sendmsg() will provide IP_PKTINFO back to kernel
		 * to let it use the 'right' source address
		 * (destination address of the incoming packet)

		 */
		vec[0].iov_len = res;
		sendmsg(fd, &msghdr, 0);
		if (--count == 0)
			break;
	}
	return 0;
}

Here is an IPv4 sample UDP application, able to echo packets with the IP source set to original DST address of the ping packet.

Doing the same on IPv6 is probably trivial as well

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <linux/udp.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
#include <arpa/inet.h>

#define PORT 4040

int pktinfo_get(struct msghdr *my_hdr, struct in_pktinfo *pktinfo)

{
	int res = -1;

	if (my_hdr->msg_controllen > 0) {
		struct cmsghdr *get_cmsg;
		for (get_cmsg = CMSG_FIRSTHDR(my_hdr); get_cmsg;
			get_cmsg = CMSG_NXTHDR(my_hdr, get_cmsg)) {
			if (get_cmsg->cmsg_type == IP_PKTINFO) {
				struct in_pktinfo *get_pktinfo = (struct in_pktinfo *)CMSG_DATA(get_cmsg);
				memcpy(pktinfo, get_pktinfo, sizeof(*pktinfo));
				res = 0;
			}
		}
	}
	return res;
}

int main(int argc, char *argv[])

{
	int fd = socket(AF_INET, SOCK_DGRAM, 0);
	struct sockaddr_in addr, rem_addr;
	int res, on = 1;
	struct msghdr msghdr;
	struct iovec vec[1];
	char cbuf[512];
	char frame[4096];
	struct in_pktinfo pktinfo;
	int c, count = 1000000;

	while ((c = getopt(argc, argv, "c:")) != -1) {
		if (c == 'c') count = atoi(optarg);
		}
	memset(&addr, 0, sizeof(addr));
	addr.sin_family = AF_INET;
	addr.sin_port = htons(PORT);
	if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) == -1) {
		perror("bind");
		return 1;
	}
	setsockopt(fd, SOL_IP, IP_PKTINFO, &on, sizeof(on));

	while (1) {
		memset(&msghdr, 0, sizeof(msghdr));
		msghdr.msg_control = cbuf;
		msghdr.msg_controllen = sizeof(cbuf);
		msghdr.msg_iov = vec;
		msghdr.msg_iovlen = 1;
		vec[0].iov_base = frame;
		vec[0].iov_len = sizeof(frame);
		msghdr.msg_name = &rem_addr;
		msghdr.msg_namelen = sizeof(rem_addr);
		res = recvmsg(fd, &msghdr, 0);
		if (res == -1)
			break;
		if (pktinfo_get(&msghdr, &pktinfo) == 0)
			printf("Got IP_PKTINFO dst addr=%s\n", inet_ntoa(pktinfo.ipi_spec_dst));

		/* ok, just echo reply this frame.
		 * Using sendmsg() will provide IP_PKTINFO back to kernel
		 * to let it use the 'right' source address
		 * (destination address of the incoming packet)
		 */
		vec[0].iov_len = res;
		sendmsg(fd, &msghdr, 0);
		if (--count == 0)
			break;
	}
	return 0;
}

Guess I should have read Stephens[1] first, as this problem with multihomed hosts is described (on page 219). He also states, that this is a problem/feature related to Berkely-derived implementations. E.g. Solaris handle this, the way I expected. That is, the source IP address for the server's reply is the dest IP of the client's request.