Skip to contentSkip to conversation

qemu: Fix race condition when binding ports

PATCH Superseded Andrew Cooper 5 participants 9 messages Dec 9, 2013 – Dec 10, 2013

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 qemu: Fix race condition when binding ports

Two Qemus can race to bind the same VNC port. It is valid for multiple listen()s on the same socket to succeed, but only the first bind() will succeed.

In the case that two Qemus are racing, the second one will fail with an EADDRINUSE, and bail with a fatal error which renders the domain functionally useless.

In the case that bind() fails with EADDRINUSE, rebind the socket again and try for the next port.

CC Ian Campbell <[email protected]>
CC Ian Jackson <[email protected]>
CC Stefano Stabellini <[email protected]>
CC George Dunlap <[email protected]>

---

This fix has been in XenServer for two and a half years

---
 qemu-sockets.c |   23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
qemu-sockets.c +16 -7
index 9b9fa77..e54f6ad 100644
@@ -163,6 +163,7 @@ int inet_listen(const char *str, char *ostr, int olen,
 
     /* create socket + bind */
     for (e = res; e != NULL; e = e->ai_next) {
+    rebind:
 	getnameinfo((struct sockaddr*)e->ai_addr,e->ai_addrlen,
 		    uaddr,INET6_ADDRSTRLEN,uport,32,
 		    NI_NUMERICHOST | NI_NUMERICSERV);
@@ -186,7 +187,20 @@ int inet_listen(const char *str, char *ostr, int olen,
                 if (sockets_debug)
                     fprintf(stderr,"%s: bind(%s,%s,%d): OK\n", __FUNCTION__,
                             inet_strfamily(e->ai_family), uaddr, inet_getport(e));
-                goto listen;
+                if (listen(slisten,1) == 0) {
+                    goto listened;
+                } else {
+                    int err = errno;
+
+                    perror("listen");
+                    closesocket(slisten);
+
+                    if (err == EADDRINUSE)
+                        goto rebind;
+                    freeaddrinfo(res);
+                    return -1;
+                }
+
             }
             try_next = to && (inet_getport(e) <= to + port_offset);
             if (!try_next || sockets_debug)
@@ -205,12 +219,7 @@ int inet_listen(const char *str, char *ostr, int olen,
     freeaddrinfo(res);
     return -1;
 
-listen:
-    if (listen(slisten,1) != 0) {
-        perror("listen");
-        closesocket(slisten);
-        return -1;
-    }
+listened:
     if (ostr) {
         if (e->ai_family == PF_INET6) {
             snprintf(ostr, olen, "[%s]:%d%s", uaddr,
Signature · 1 line
1.7.10.4
#1 lore raw
Quoted 10 lines · Andrew Cooper

Two Qemus can race to bind the same VNC port. It is valid for multiple listen()s on the same socket to succeed, but only the first bind() will succeed.

In the case that two Qemus are racing, the second one will fail with an EADDRINUSE, and bail with a fatal error which renders the domain functionally useless.

In the case that bind() fails with EADDRINUSE, rebind the socket again and try for the next port.

Sorry if I'm being a bit dense here, but it looks like you have those mixed up: if listen() fails with EADDRINUSE, you're going back to call bind() again; which would suggest that you can have multible bind()s but only a single listen()? Or am I confused?

In any case, this is clearly a bug fix, and a very well tested one at that, so:

Quoted 62 lines
CC Ian Campbell <[email protected]>
CC Ian Jackson <[email protected]>
CC Stefano Stabellini <[email protected]>
CC George Dunlap <[email protected]>

---

This fix has been in XenServer for two and a half years

---
  qemu-sockets.c |   23 ++++++++++++++++-------
  1 file changed, 16 insertions(+), 7 deletions(-)
qemu-sockets.c +16 -7
index 9b9fa77..e54f6ad 100644
@@ -163,6 +163,7 @@ int inet_listen(const char *str, char *ostr, int olen,
 
      /* create socket + bind */
      for (e = res; e != NULL; e = e->ai_next) {
+    rebind:
  	getnameinfo((struct sockaddr*)e->ai_addr,e->ai_addrlen,
  		    uaddr,INET6_ADDRSTRLEN,uport,32,
  		    NI_NUMERICHOST | NI_NUMERICSERV);
@@ -186,7 +187,20 @@ int inet_listen(const char *str, char *ostr, int olen,
                  if (sockets_debug)
                      fprintf(stderr,"%s: bind(%s,%s,%d): OK\n", __FUNCTION__,
                              inet_strfamily(e->ai_family), uaddr, inet_getport(e));
-                goto listen;
+                if (listen(slisten,1) == 0) {
+                    goto listened;
+                } else {
+                    int err = errno;
+
+                    perror("listen");
+                    closesocket(slisten);
+
+                    if (err == EADDRINUSE)
+                        goto rebind;
+                    freeaddrinfo(res);
+                    return -1;
+                }
+
              }
              try_next = to && (inet_getport(e) <= to + port_offset);
              if (!try_next || sockets_debug)
@@ -205,12 +219,7 @@ int inet_listen(const char *str, char *ostr, int olen,
      freeaddrinfo(res);
      return -1;
 
-listen:
-    if (listen(slisten,1) != 0) {
-        perror("listen");
-        closesocket(slisten);
-        return -1;
-    }
+listened:
      if (ostr) {
          if (e->ai_family == PF_INET6) {
              snprintf(ostr, olen, "[%s]:%d%s", uaddr,
#2 lore raw
Quoted 10 lines · George Dunlap
Quoted 9 lines · Andrew Cooper

Two Qemus can race to bind the same VNC port. It is valid for multiple listen()s on the same socket to succeed, but only the first bind() will succeed.

In the case that two Qemus are racing, the second one will fail with an EADDRINUSE, and bail with a fatal error which renders the domain functionally
useless.

Quoted 8 lines
Quoted 3 lines

In the case that bind() fails with EADDRINUSE, rebind the socket again and try
for the next port.

Sorry if I'm being a bit dense here, but it looks like you have those mixed up: if listen() fails with EADDRINUSE, you're going back to call bind() again; which would suggest that you can have multible bind()s but only a single listen()? Or am I confused?

Nope - very definitely me who messed up the message. v2 on its way.

~Andrew

#3 lore raw

PATCH v2 1/1 qemu: Fix race condition when opening ports

In reply to George Dunlap

Two Qemus can race to bind the same VNC port. It is valid for multiple bind()s on the same socket to succeed, but only the first listen() will succeed.

In the case that two Qemus are starting at the same time, and both trying to grab the next free VNC port, the second one will fail with an EADDRINUSE, and bail with a fatal error which renders the domain functionally useless.

In the case that listen() fails with EADDRINUSE, rebind the socket again and try for the next port.

CC Ian Campbell <[email protected]>
CC Ian Jackson <[email protected]>
CC Stefano Stabellini <[email protected]>

---

Changes in v2:
* Fix commit message.

This fix has been in XenServer for two and a half years

---
 qemu-sockets.c |   23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
qemu-sockets.c +16 -7
index 9b9fa77..e54f6ad 100644
@@ -163,6 +163,7 @@ int inet_listen(const char *str, char *ostr, int olen,
 
     /* create socket + bind */
     for (e = res; e != NULL; e = e->ai_next) {
+    rebind:
 	getnameinfo((struct sockaddr*)e->ai_addr,e->ai_addrlen,
 		    uaddr,INET6_ADDRSTRLEN,uport,32,
 		    NI_NUMERICHOST | NI_NUMERICSERV);
@@ -186,7 +187,20 @@ int inet_listen(const char *str, char *ostr, int olen,
                 if (sockets_debug)
                     fprintf(stderr,"%s: bind(%s,%s,%d): OK\n", __FUNCTION__,
                             inet_strfamily(e->ai_family), uaddr, inet_getport(e));
-                goto listen;
+                if (listen(slisten,1) == 0) {
+                    goto listened;
+                } else {
+                    int err = errno;
+
+                    perror("listen");
+                    closesocket(slisten);
+
+                    if (err == EADDRINUSE)
+                        goto rebind;
+                    freeaddrinfo(res);
+                    return -1;
+                }
+
             }
             try_next = to && (inet_getport(e) <= to + port_offset);
             if (!try_next || sockets_debug)
@@ -205,12 +219,7 @@ int inet_listen(const char *str, char *ostr, int olen,
     freeaddrinfo(res);
     return -1;
 
-listen:
-    if (listen(slisten,1) != 0) {
-        perror("listen");
-        closesocket(slisten);
-        return -1;
-    }
+listened:
     if (ostr) {
         if (e->ai_family == PF_INET6) {
             snprintf(ostr, olen, "[%s]:%d%s", uaddr,
Signature · 1 line
1.7.10.4
#4 lore raw

Re: [PATCH v2 qemu-traditional] qemu: Fix race condition when opening ports

Quoted 10 lines · Andrew Cooper

Two Qemus can race to bind the same VNC port. It is valid for multiple bind()s on the same socket to succeed, but only the first listen() will succeed.

In the case that two Qemus are starting at the same time, and both trying to grab the next free VNC port, the second one will fail with an EADDRINUSE, and bail with a fatal error which renders the domain functionally useless.

In the case that listen() fails with EADDRINUSE, rebind the socket again and try for the next port.

Does this do the right thing if vncunused=0? In that case the user has given a specific port which we want to bind to, so going to the next port is not desired.

Quoted 33 lines
CC Ian Campbell <[email protected]>
CC Ian Jackson <[email protected]>
CC Stefano Stabellini <[email protected]>

---

Changes in v2:
* Fix commit message.

This fix has been in XenServer for two and a half years

---
 qemu-sockets.c |   23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
qemu-sockets.c +1 -1
index 9b9fa77..e54f6ad 100644
@@ -163,6 +163,7 @@ int inet_listen(const char *str, char *ostr, int olen,
 
     /* create socket + bind */
     for (e = res; e != NULL; e = e->ai_next) {
+    rebind:
 	getnameinfo((struct sockaddr*)e->ai_addr,e->ai_addrlen,
 		    uaddr,INET6_ADDRSTRLEN,uport,32,
 		    NI_NUMERICHOST | NI_NUMERICSERV);
@@ -186,7 +187,20 @@ int inet_listen(const char *str, char *ostr, int olen,
                 if (sockets_debug)
                     fprintf(stderr,"%s: bind(%s,%s,%d): OK\n", __FUNCTION__,
                             inet_strfamily(e->ai_family), uaddr, inet_getport(e));
-                goto listen;

qemu-sockets.c int inet_listen(const char *str, char *ostr, int olen, · line 199

Quoted 10 lines
qemu-sockets.c +10 -0
+                if (listen(slisten,1) == 0) {
+                    goto listened;
+                } else {
+                    int err = errno;
+
+                    perror("listen");
+                    closesocket(slisten);
+
+                    if (err == EADDRINUSE)
+                        goto rebind;

Does this code actually try for the next port? The try_next check is after the goto here (you can see it in the context below). Won't it just try to rebind the same port again and again?

Quoted 21 lines
qemu-sockets.c +5 -6
+                    freeaddrinfo(res);
+                    return -1;
+                }
+
             }
             try_next = to && (inet_getport(e) <= to + port_offset);
             if (!try_next || sockets_debug)
@@ -205,12 +219,7 @@ int inet_listen(const char *str, char *ostr, int olen,
     freeaddrinfo(res);
     return -1;
 
-listen:
-    if (listen(slisten,1) != 0) {
-        perror("listen");
-        closesocket(slisten);
-        return -1;
-    }
+listened:
     if (ostr) {
         if (e->ai_family == PF_INET6) {
             snprintf(ostr, olen, "[%s]:%d%s", uaddr,
#5 lore raw

Re: [PATCH v2 qemu-traditional] qemu: Fix race condition when opening ports

In reply to Andrew Cooper
Quoted 17 lines

-----Original Message-----
From: [email protected] [mailto:xen-devel- [email protected]] On Behalf Of Andrew Cooper
Sent: 09 December 2013 1:27 PM
To: Xen-devel
Cc: Andrew Cooper; Ian Jackson; Ian Campbell; Stefano Stabellini
Subject: [Xen-devel] [PATCH v2 qemu-traditional] qemu: Fix race condition when opening ports

Two Qemus can race to bind the same VNC port. It is valid for multiple bind()s on the same socket to succeed, but only the first listen() will succeed.

In the case that two Qemus are starting at the same time, and both trying to grab the next free VNC port, the second one will fail with an EADDRINUSE, and bail with a fatal error which renders the domain functionally useless.

Quoted 8 lines

In the case that listen() fails with EADDRINUSE, rebind the socket again and try for the next port.

CC Ian Campbell <[email protected]>
CC Ian Jackson <[email protected]>
CC Stefano Stabellini <[email protected]>

Andrew, if you are upstream other people stuff you should at least give credit to the original author.

Frediano

#6 lore raw

Re: [PATCH v2 qemu-traditional] qemu: Fix race condition when opening ports

Quoted 20 lines · Frediano Ziglio
Quoted 20 lines

-----Original Message-----
From: [email protected] [mailto:xen-devel- [email protected]] On Behalf Of Andrew Cooper
Sent: 09 December 2013 1:27 PM
To: Xen-devel
Cc: Andrew Cooper; Ian Jackson; Ian Campbell; Stefano Stabellini
Subject: [Xen-devel] [PATCH v2 qemu-traditional] qemu: Fix race condition when opening ports

Two Qemus can race to bind the same VNC port. It is valid for multiple bind()s on the same socket to succeed, but only the first listen() will succeed.

In the case that two Qemus are starting at the same time, and both trying to grab the next free VNC port, the second one will fail with an EADDRINUSE, and bail with a fatal error which renders the domain functionally useless.

In the case that listen() fails with EADDRINUSE, rebind the socket again and try for the next port.

Quoted 9 lines
Quoted 5 lines
CC Ian Campbell <[email protected]>
CC Ian Jackson <[email protected]>
CC Stefano Stabellini <[email protected]>

Andrew, if you are upstream other people stuff you should at least give credit to the original author.

Frediano

The original author was "xenhg" according to the history. It is not even clear from the original bug ticket who the author could have been.

In this case, my Signed-off-by line falls into category b) of the Developer’s Certificate of Origin.

This is an unfortunate problem with attempting to weed through a neglected patch queue where proper authorship was not considered at the time of commit.

~Andrew

#7 lore raw

Re: [PATCH v2 qemu-traditional] qemu: Fix race condition when opening ports

In reply to Andrew Cooper

Andrew Cooper writes ("[PATCH v2 qemu-traditional] qemu: Fix race condition when opening ports"):

Quoted 3 lines

Two Qemus can race to bind the same VNC port. It is valid for multiple bind()s on the same socket to succeed, but only the first listen() will succeed.

I don't think this is correct. The SuS page for listen(2)
http://pubs.opengroup.org/onlinepubs/9699919799/functions/listen.html
doesn't list EADDRINUSE as a possible error condition.

Although wheezy's listen(2) manpage does.

Quoted 6 lines

In the case that two Qemus are starting at the same time, and both trying to grab the next free VNC port, the second one will fail with an EADDRINUSE, and bail with a fatal error which renders the domain functionally useless.

In the case that listen() fails with EADDRINUSE, rebind the socket again and try for the next port.

My preliminary view is that this is a kernel bug.

Ian.

#8 lore raw

Re: [PATCH v2 qemu-traditional] qemu: Fix race condition when opening ports

Quoted 16 lines

Andrew Cooper writes ("[PATCH v2 qemu-traditional] qemu: Fix race condition when opening ports"):

Quoted 3 lines

Two Qemus can race to bind the same VNC port. It is valid for multiple bind()s on the same socket to succeed, but only the first listen() will succeed.

I don't think this is correct. The SuS page for listen(2)
http://pubs.opengroup.org/onlinepubs/9699919799/functions/listen.html
doesn't list EADDRINUSE as a possible error condition.

Although wheezy's listen(2) manpage does.

Quoted 3 lines

In the case that two Qemus are starting at the same time, and both trying to grab the next free VNC port, the second one will fail with an EADDRINUSE, and bail with a fatal error which renders the domain

Quoted 9 lines

functionally useless.

Quoted 3 lines

In the case that listen() fails with EADDRINUSE, rebind the socket again and try for the next port.

My preliminary view is that this is a kernel bug.

Ian.

My Ubuntu "man 2 listen" reports

EADDRINUSE
Another socket is already listening on the same port.

(same from http://man7.org/linux/man-pages/man2/listen.2.html)

Frediano