Skip to contentSkip to conversation

any way to let host act as TCP server OR client on same IP/port?

DISCUSSION Chris Friesen 3 participants 9 messages Jul 13, 2011 – Jul 14, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I've been asked an interesting question about TCP. We have some people that want to set up a TCP socket that can listen for connections on a given IP/port, but also initiate connections from that same IP/port. (Only one at a time, of course.)

The TCP state machine seems to allow this (moving from LISTEN to SYN_SENT) but it's not a normal transition.

Is there any way to do this using the socket API?

I thought up a hack whereby we could use NFQUEUE to detect an incoming SYN and delay it while we call listen() on the socket. Is there any better way to do this?

Thanks,

Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#1 lore raw
Quoted 5 lines · Chris Friesen

I've been asked an interesting question about TCP. We have some people that want to set up a TCP socket that can listen for connections on a given IP/port, but also initiate connections from that same IP/port. (Only one at a time, of course.)

Quoted 8 lines

The TCP state machine seems to allow this (moving from LISTEN to SYN_SENT) but it's not a normal transition.

Is there any way to do this using the socket API?

I thought up a hack whereby we could use NFQUEUE to detect an incoming SYN and delay it while we call listen() on the socket. Is there any better way to do this?

Could you try SO_REUSEADDR, on both listener and connect attempt ?

#2 lore raw
Quoted 9 lines · Eric Dumazet
Quoted 8 lines · Chris Friesen

I've been asked an interesting question about TCP. We have some people that want to set up a TCP socket that can listen for connections on a given IP/port, but also initiate connections from that same IP/port. (Only one at a time, of course.)

The TCP state machine seems to allow this (moving from LISTEN to SYN_SENT) but it's not a normal transition.

Quoted 7 lines
Quoted 5 lines

Is there any way to do this using the socket API?

I thought up a hack whereby we could use NFQUEUE to detect an incoming SYN and delay it while we call listen() on the socket. Is there any better way to do this?

Could you try SO_REUSEADDR, on both listener and connect attempt ?

I was thinking the same thing, but it appears to not work under:

raj@tardy:~$ uname -a
Linux tardy 2.6.35-30-generic #54-Ubuntu SMP Tue Jun 7 18:41:54 UTC 2011 x86_64 GNU/Linux
raj@tardy:~$ ./both
bind active: Address already in use

(nor HP-UX 11.31)

unless I botched the test code somehow:

raj@tardy:~$ cat both.c
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <string.h>
#include <arpa/inet.h>
#include <stdlib.h>

int
main(int argc, char *argv[]) {

  struct sockaddr_in me,him;
  int on = 1;
  socklen_t  len;
  int  listener, active, client;


  if ((listener = socket(AF_INET,SOCK_STREAM,0)) < 0) {
    perror("socket listener");
    exit(-1);
  }

  if ((active = socket(AF_INET,SOCK_STREAM,0)) < 0) {
    perror("socket active");
    exit(-1);
  }

  if ((client = socket(AF_INET,SOCK_STREAM,0)) < 0) {
    perror("socket client");
    exit(-1);
  }

  /* setup the listen endpoint */
  memset(&me,0,sizeof(me));

  if (!inet_pton(AF_INET,"127.0.0.1",&(me.sin_addr.s_addr))) {
    perror("inet_pton me");
    exit(-1);
  }
  me.sin_port = htons(23456);
  me.sin_family = AF_INET;

if (setsockopt(listener,

		 SOL_SOCKET,
		 SO_REUSEADDR,
		 (char *)&on ,
		 sizeof(on)) < 0) {
    perror("setsockopt SO_REUSEADDR listener");
    exit(-1);

}

if (bind(listener,

	   (struct sockaddr *)&me,
	   sizeof(me)) < 0) {
    perror("bind listener");
    exit(-1);

}

  if (listen(listener,128) < 0) {
    perror("listen listener");
    exit(-1);
  }

  /* connect something to it */
  if (connect(client,(struct sockaddr *)&me,sizeof(me)) < 0) {
    perror("connect client");
    exit(-1);
  }

/* now try to make an active connection from "me" */ memset(&him,0,sizeof(him));

  if (!inet_pton(AF_INET,"127.0.0.1",&(him.sin_addr.s_addr))) {
    perror("inet_pton him");
    exit(-1);
  }

him.sin_port = htons(9); /* assume someone is listening on

					"discard"*/

him.sin_family = AF_INET;

if (setsockopt(active,

		 SOL_SOCKET,
		 SO_REUSEADDR,
		 (char *)&on ,
		 sizeof(on)) < 0) {
    perror("setsockopt SO_REUSEADDR active");
    exit(-1);

}

if (bind(active,

	   (struct sockaddr *)&me,
	   sizeof(me)) < 0) {
    perror("bind active");
    exit(-1);

}

  if (connect(active,(struct sockaddr *)&him,sizeof(him)) < 0) {
    perror("connect active");
    exit(-1);
  }

printf("Successfully created a listen endpoint and an active
connection from the same IP/port pair\n");

}

rick jones

#3 lore raw
Quoted 19 lines · Rick Jones
Quoted 9 lines · Eric Dumazet
Quoted 8 lines · Chris Friesen

I've been asked an interesting question about TCP. We have some people that want to set up a TCP socket that can listen for connections on a given IP/port, but also initiate connections from that same IP/port. (Only one at a time, of course.)

The TCP state machine seems to allow this (moving from LISTEN to SYN_SENT) but it's not a normal transition.

Quoted 7 lines
Quoted 5 lines

Is there any way to do this using the socket API?

I thought up a hack whereby we could use NFQUEUE to detect an incoming SYN and delay it while we call listen() on the socket. Is there any better way to do this?

Could you try SO_REUSEADDR, on both listener and connect attempt ?

I was thinking the same thing, but it appears to not work under:

<snip>

Quoted 12 lines

if (bind(listener,
(struct sockaddr *)&me,
sizeof(me)) < 0) {
perror("bind listener");
exit(-1);
}

if (listen(listener,128) < 0) {
perror("listen listener");
exit(-1);
}

Quoted 4 lines

/* connect something to it */
if (connect(client,(struct sockaddr *)&me,sizeof(me)) < 0) { perror("connect client");
exit(-1);

In our case we don't need to actually be connected, just be listening and ready to either accept() a connection or connect() to someone else.

However, even after removing the connect() call I get:
"bind active: Address already in use"

The TCP state machine shows a single connection going from LISTEN to SYN_SENT via a "send" operation in the application. Presumably this would logically map to a sendto/sendmsg but according to the man page those don't support specifying addresses for connection-oriented sockets. I tried it anyways and got no errors but the following trace shows that it's dying with SIGPIPE:

bind(3, {sa_family=AF_INET, sin_port=htons(23456), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
listen(3, 128) = 0
sendto(3, "\1", 1, 0, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EPIPE (Broken pipe) --- {si_signo=SIGPIPE, si_code=SI_USER, si_pid=20609, si_uid=8382, si_value={int=2722689790, ptr=0x3ca248f2fe}} (Broken pipe) --- +++ killed by SIGPIPE +++

Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#4 lore raw
Quoted 17 lines
Quoted 2 lines

I was thinking the same thing, but it appears to not work under:

<snip>

Quoted 12 lines

if (bind(listener,
(struct sockaddr *)&me,
sizeof(me))< 0) {
perror("bind listener");
exit(-1);
}

if (listen(listener,128)< 0) {
perror("listen listener");
exit(-1);
}

Quoted 7 lines
Quoted 4 lines

/* connect something to it */
if (connect(client,(struct sockaddr *)&me,sizeof(me))< 0) { perror("connect client");
exit(-1);

In our case we don't need to actually be connected, just be listening and ready to either accept() a connection or connect() to someone else.

If one calls listen() against a socket, TCP connections can be established at any time, before the accept() calls are made. As for what might happen when one calls connect() on a listen socket when there are queued connections awaiting accept() I've no idea. Presumably they would follow the same path as if close() were called against the listen endpoint before they were accept()ed. And if the listen endpoint is transitioned to CONNECTED then any subsequently arriving connection requests will likely elicit RSTs back to their senders.

If your application's sematics are OK with pending connections being dumped, and there is no issue with something else binding to IP,port and putting it into the LISTEN state, and if I've read between the lines of what you've written correctly, you could simply close() the listen socket and create a fresh socket with which to do the connect()?

I think it is only if you need both accepting new connections and calling connect() all in parallel that you need enhancements to the stack.

Quoted 2 lines

However, even after removing the connect() call I get:
"bind active: Address already in use"

Some of this behaviour may come from the "standards" (either formal or informal), and what they call for for error returns on the different calls under what conditions.

Historically, and not necessarily specific to any one stack, SO_REUSEADDR allows one to kill a server application, and restart it and get its LISTEN endpoint restarted even while there are old connections with IP,port as half the four-tuple (esp TIME_WAITs). But SO_REUSEADDR won't allow the bind(IP,port) to succeed if there is another listen() endpoint bound to that IP,port. That would be an ambiguity. But, if the second endpoint to be bound to IP,port doesn't go listen(), there isn't an ambiguity. This may be where the standards or historical expectations come-in if they "require" the bind() fail even if it isn't clear there will be a subsequent listen() call against that endpoint.

Presumably, as far as the TCP state machine goes, having one endpoint bound only to IP1,Port1 and in the LISTEN state and one simply bound to IP,port but not in LISTEN (presumably on its way to a connection establishment) is ok. A arriving segment will hit the four-tuple if it exists, and only go to the two-tuple if is in LISTEN and the segment is a SYN and won't go to the non-listen two-tuple at all. But at the API level the listen() and connect() calls need to be permitted/able to return EADDRINUSE (or something akin) when they then try to create a two/four-tuple (listen/connect) that is already in use.

FWIW, the listen() manpage on my Maverick system does list:

ERRORS
EADDRINUSE
Another socket is already listening on the same port.

suggesting that the "delay the failure until listen() on the same IP,port" is possible, but I don't know if that is vestigial manpage cruft or "real." The manpage for connect() shows EADDRINUSE as well - again though I not sure if that is vestigial, so it may not accurately convey current stack intent.

killed by SIGPIPE +++

Quoted 12 lines

The TCP state machine shows a single connection going from LISTEN to SYN_SENT via a "send" operation in the application. Presumably this would logically map to a sendto/sendmsg but according to the man page those don't support specifying addresses for connection-oriented sockets. I tried it anyways and got no errors but the following trace shows that it's dying with SIGPIPE:

bind(3, {sa_family=AF_INET, sin_port=htons(23456), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
listen(3, 128) = 0
sendto(3, "\1", 1, 0, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EPIPE (Broken pipe)

killed by SIGPIPE +++ renamed from {si_signo=SIGPIPE, si_code=SI_USER, si_pid=20609, si_uid=8382, si_value={int=2722689790, ptr=0x3ca248f2fe}} (Broken pipe) --- +0 -0

You might try a connect() call - since T/TCP (vs the even older than netperf, ttcp benchmark :) never caught-on, sendto() against a TCP endpoint is probably a no-go.

If I'm thinking of the same state machine diagram, the "send" is rather, well, generic and does not map directly to any API.

(I tried it very quick and dirty, with no joy)

If your application's sematics are OK with pending connections being dumped, and there is no issue with something else rick

#5 lore raw
Quoted 22 lines · Rick Jones
Quoted 22 lines
Quoted 2 lines

I was thinking the same thing, but it appears to not work under:

<snip>

Quoted 16 lines

if (bind(listener,
(struct sockaddr *)&me,
sizeof(me))< 0) {
perror("bind listener");
exit(-1);
}

if (listen(listener,128)< 0) {
perror("listen listener");
exit(-1);
}

/* connect something to it */
if (connect(client,(struct sockaddr *)&me,sizeof(me))< 0) { perror("connect client");
exit(-1);

Quoted 7 lines
Quoted 2 lines

In our case we don't need to actually be connected, just be listening and ready to either accept() a connection or connect() to someone else.

If one calls listen() against a socket, TCP connections can be established at any time, before the accept() calls are made. As for what might happen when one calls connect() on a listen socket when there are queued connections awaiting accept() I've no idea.

In Linux at least it appears that you aren't allowed to call listen() and then connect(). See below, however.

Quoted 5 lines

If your application's sematics are OK with pending connections being dumped, and there is no issue with something else binding to IP,port and putting it into the LISTEN state, and if I've read between the lines of what you've written correctly, you could simply close() the listen socket and create a fresh socket with which to do the connect()?

Ah, of course. It actually looks like a shutdown() might be sufficient to reset the state of the socket enough to allow me to then call connect() without needing to create a new socket and bind it.

I'll see what the application guys say.

Thanks,
Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#6 lore raw
In reply to Chris Friesen
Quoted 17 lines · Chris Friesen
Quoted 16 lines · Rick Jones
Quoted 15 lines · Eric Dumazet
Quoted 13 lines · Chris Friesen

I've been asked an interesting question about TCP. We have some people that want to set up a TCP socket that can listen for connections on a given IP/port, but also initiate connections from that same IP/port. (Only one at a time, of course.)

The TCP state machine seems to allow this (moving from LISTEN to SYN_SENT) but it's not a normal transition.

Is there any way to do this using the socket API?

I thought up a hack whereby we could use NFQUEUE to detect an incoming SYN and delay it while we call listen() on the socket. Is there any better way to do this?

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Could you try SO_REUSEADDR, on both listener and connect attempt ?

I was thinking the same thing, but it appears to not work under:

In our case we don't need to actually be connected, just be listening and ready to either accept() a connection or connect() to someone else.

It turns out that the application people really do want the server side to be able to listen() at the same time as calling connect() from the same address/port, so Rick's testcase was accurate.

It would be interesting if linux were to allow this behaviour. It doesn't seem to violate any specs, since we could still return an error if we try to connect() to an address that is already connected to us.

Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com
#7 lore raw
Quoted 6 lines
Quoted 2 lines

In our case we don't need to actually be connected, just be listening and ready to either accept() a connection or connect() to someone else.

It turns out that the application people really do want the server side to be able to listen() at the same time as calling connect() from the same address/port, so Rick's testcase was accurate.

So, we are left asking "Why do the application people want that?" Does the server connect() to pseudo-random places, or does it only ever connect back to clients which have already connected to it?

If the desire is to have the same well-known port number involved in all connections to/from the server (perhaps for simplistic firewall rules involving serverip.magicport?) then if the connections are back to the clients, the clients could simply open listen endpoints bound to the magic port number and the firewall rule become "server IP is source or destination) AND (magic port is source or destination)" - assuming the clients aren't bind()ing to the magic port number before connect()ing to the server.

rick jones

#8 lore raw
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

In our case we don't need to actually be connected, just be listening and ready to either accept() a connection or connect() to someone else.

It turns out that the application people really do want the server side to be able to listen() at the same time as calling connect() from the same address/port, so Rick's testcase was accurate.

So, we are left asking "Why do the application people want that?" Does the server connect() to pseudo-random places, or does it only ever connect back to clients which have already connected to it?

I'm not an expert in the application domain, but here's what I know. You can think of the situation as having a server, a gateway, and many clients. The goal is to map each client to a single unique port on the server-facing side of the gateway, such that all traffic for that client goes through that port.

The complicating factor is that traffic may be initiated by either the client or the server. With UDP this isn't a problem, but for TCP it's tricky.

For IPv4 the app team originally set it up so that the gateway listens on a single port on the server side of the gateway, and when the server opens a connection to a client's unique port (with an ephemeral port as the source port) it gets NAT'd to that single port. (The unique ports on the server side ensure that there are no 4-tuple collisions.) For traffic originating from the client, the gateway binds a socket to the client's unique port and uses it to open a connection to the server.

IPv6 doesn't support NAT (at least not in the stock kernel), so they're looking for alternatives.

Quoted 3 lines

If the desire is to have the same well-known port number involved in all connections to/from the server (perhaps for simplistic firewall rules involving serverip.magicport?)

I suspect it's something like that.

Quoted 6 lines

then if the connections are back to the
clients, the clients could simply open listen endpoints bound to the magic port number and the firewall rule become "server IP is source or destination) AND (magic port is source or destination)" - assuming the clients aren't bind()ing to the magic port number before connect()ing to the server.

We don't control the server, only the gateway. Ephemeral source ports are used for connections opened by the server, so a known port is needed on the gateway side.

Chris

Signature · 5 lines
Chris Friesen
Software Developer
GENBAND
[email protected]
www.genband.com