Skip to contentSkip to conversation

What's the status of TCP friends?

DISCUSSION Zheng Liu 5 participants 8 messages Mar 16, 2014 – Mar 18, 2014

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Zheng Liu

author
#0 lore raw

Hi all,

Until now the TCP friends patch set doesn't be applied. Now what's the status of TCP friends? Is it applicable to be merged into upstream kernel? Any problem that needs to be fixed? Please let me know if I can help.

Thanks,
- Zheng

#1 lore raw
Quoted 6 lines · Zheng Liu

Hi all,

Until now the TCP friends patch set doesn't be applied. Now what's the status of TCP friends? Is it applicable to be merged into upstream kernel? Any problem that needs to be fixed? Please let me know if I can help.

Well, last attempts showed that while the idea sounded cool, implementation opened many races and added quite a lot of complexity in fast path.

We have AF_UNIX with a lot of problems in it, do we really want to bring these AF_UNIX problems to AF_INET ?

I would rather spend time on AF_UNIX if it doesn't fit your needs right now, or consider jumping to KDBUS modern design.

Using AF_INET for IPC is poor choice.

author
#2 lore raw
Quoted 11 lines · Eric Dumazet
Quoted 6 lines · Zheng Liu

Hi all,

Until now the TCP friends patch set doesn't be applied. Now what's the status of TCP friends? Is it applicable to be merged into upstream kernel? Any problem that needs to be fixed? Please let me know if I can help.

Well, last attempts showed that while the idea sounded cool, implementation opened many races and added quite a lot of complexity in fast path.

Thanks for letting me know the current status of this patch set.

Quoted 3 lines

We have AF_UNIX with a lot of problems in it, do we really want to bring these AF_UNIX problems to AF_INET ?

Please bear with me because I am a newbie. Out of curiosity, what's the problem in AF_UNIX?

Quoted 5 lines

I would rather spend time on AF_UNIX if it doesn't fit your needs right now, or consider jumping to KDBUS modern design.

Using AF_INET for IPC is poor choice.

The reason why we use AF_INET for IPC rather than use AF_UNIX is that we have two applications that need to communicate with each other. They could be deployed on the same server or different servers. So obviously if we use AF_INET, we just need to indicate a IP address in config file. That sounds rational and maintainable for our operation team.

Regards,
- Zheng

#3 lore raw
Quoted 13 lines · Zheng Liu
Quoted 11 lines · Eric Dumazet
Quoted 6 lines · Zheng Liu

Hi all,

Until now the TCP friends patch set doesn't be applied. Now what's the status of TCP friends? Is it applicable to be merged into upstream kernel? Any problem that needs to be fixed? Please let me know if I can help.

Well, last attempts showed that while the idea sounded cool, implementation opened many races and added quite a lot of complexity in fast path.

Quoted 8 lines

Thanks for letting me know the current status of this patch set.

Quoted 3 lines

We have AF_UNIX with a lot of problems in it, do we really want to bring these AF_UNIX problems to AF_INET ?

Please bear with me because I am a newbie. Out of curiosity, what's the problem in AF_UNIX?

AFAIK AF_UNIX/SOCK_STREAM is fine.

I currently know about two problems with AF_UNIX datagram modes:

1) Not correclty handling socket receive buffer limits. SOCK_DGRAM simply relies on the sending window only and socket receive queue is only limited by the number of packets enqueued. A DoS against another UNIX/DGRAM server is possible by not freeing its sending window if the client doesn't simply pick up its packets.

2) POLLOUT handling seems a bit messed up. This problem was reported by Tamas Morvai. We simply trigger POLLOUT too often thus waking up the sending side unnecessarily.

Patch for 1) was rightfully rejected because it could easily break backwards compatibility.

There were some ideas floating around which I discussed with Tamas but nothing definite for 2), as I got stuck to work on 1) and I am still unsure what side-effects could have the needed removal of the per-packet socket-receive limit, which seems to be needed for solving 2). The total amount of memory in use by AF_UNIX/DGRAM sockets would be limited by the sending buffers and rlimits, but it still seems unwise to do so.

Also solving 2) could make problems regarding backwards-compatibility.

Quoted 10 lines
Quoted 4 lines

I would rather spend time on AF_UNIX if it doesn't fit your needs right now, or consider jumping to KDBUS modern design.

Using AF_INET for IPC is poor choice.

The reason why we use AF_INET for IPC rather than use AF_UNIX is that we have two applications that need to communicate with each other. They could be deployed on the same server or different servers. So obviously if we use AF_INET, we just need to indicate a IP address in config file. That sounds rational and maintainable for our operation team.

As soon as we are allowed to silently drop packets in AF_UNIX/DGRAM some problems would vanish, too. ;)

Greetings,

Hannes

author
#4 lore raw

Hi Hannes,

Quoted 40 lines · Hannes Frederic Sowa
Quoted 13 lines · Zheng Liu
Quoted 11 lines · Eric Dumazet
Quoted 6 lines · Zheng Liu

Hi all,

Until now the TCP friends patch set doesn't be applied. Now what's the status of TCP friends? Is it applicable to be merged into upstream kernel? Any problem that needs to be fixed? Please let me know if I can help.

Well, last attempts showed that while the idea sounded cool, implementation opened many races and added quite a lot of complexity in fast path.

Quoted 8 lines

Thanks for letting me know the current status of this patch set.

Quoted 3 lines

We have AF_UNIX with a lot of problems in it, do we really want to bring these AF_UNIX problems to AF_INET ?

Please bear with me because I am a newbie. Out of curiosity, what's the problem in AF_UNIX?

AFAIK AF_UNIX/SOCK_STREAM is fine.

I currently know about two problems with AF_UNIX datagram modes:

1) Not correclty handling socket receive buffer limits. SOCK_DGRAM simply relies on the sending window only and socket receive queue is only limited by the number of packets enqueued. A DoS against another UNIX/DGRAM server is possible by not freeing its sending window if the client doesn't simply pick up its packets.

2) POLLOUT handling seems a bit messed up. This problem was reported by Tamas Morvai. We simply trigger POLLOUT too often thus waking up the sending side unnecessarily.

Patch for 1) was rightfully rejected because it could easily break backwards compatibility.

Quoted 8 lines

There were some ideas floating around which I discussed with Tamas but nothing definite for 2), as I got stuck to work on 1) and I am still unsure what side-effects could have the needed removal of the per-packet socket-receive limit, which seems to be needed for solving 2). The total amount of memory in use by AF_UNIX/DGRAM sockets would be limited by the sending buffers and rlimits, but it still seems unwise to do so.

Also solving 2) could make problems regarding backwards-compatibility.

Thanks for your explanation.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

I would rather spend time on AF_UNIX if it doesn't fit your needs right now, or consider jumping to KDBUS modern design.

Using AF_INET for IPC is poor choice.

Quoted 8 lines
Quoted 5 lines

The reason why we use AF_INET for IPC rather than use AF_UNIX is that we have two applications that need to communicate with each other. They could be deployed on the same server or different servers. So obviously if we use AF_INET, we just need to indicate a IP address in config file. That sounds rational and maintainable for our operation team.

As soon as we are allowed to silently drop packets in AF_UNIX/DGRAM some problems would vanish, too. ;)

That sounds great! But it might not satisfy our requirement. If we use AF_UNIX our program will not be deployed on two servers. Meanwhile AF_INET has been applied in our program to communicate with other clients. So DGRAM seems that it is not a good idea. Now our program needs a IPC mechansim that can commnucate between two servers and provide a high performance when two processes are run on the same server. That is the reason why I am interested in TCP friends. :)

Regards,
- Zheng

maintainer
#5 lore raw

From: Zheng Liu <[email protected]>
Date: Tue, 18 Mar 2014 11:13:46 +0800

Quoted 3 lines

Now our program needs a IPC mechansim that can commnucate between two servers and provide a high performance when two processes are run on the same server.

Using both TCP and AF_UNIX, as needed, and dynamically, is a technique which has been in use for more than two decades by X.

Nothing stops you from adopting such a scheme too.

Usually when I someone says "I can't" it sounds to me more like "I've decided to stop trying to think of a way to make it work."

#6 lore raw
In reply to Zheng Liu
Quoted 7 lines · Zheng Liu

That sounds great! But it might not satisfy our requirement. If we use AF_UNIX our program will not be deployed on two servers. Meanwhile AF_INET has been applied in our program to communicate with other clients. So DGRAM seems that it is not a good idea. Now our program needs a IPC mechansim that can commnucate between two servers and provide a high performance when two processes are run on the same server. That is the reason why I am interested in TCP friends. :)

TCP friends is another layer added into TCP stack, for what ?

Improving performance for lazy applications ?

Really, if you cared about performance, you would have added a way to use fast IPC if available.

TCP friends will still be slower than the available IPC mechanisms, by an order of magnitude.

So instead of spending time on this TCP friends dream, I think you should focus on existing and supported mechanisms.

I for example could add zerocopy support to AF_UNIX, if you think its worth it.

#7 lore raw
Quoted 17 lines · On Mon, 17 Mar 2014 21:03:20 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 7 lines · Zheng Liu

That sounds great! But it might not satisfy our requirement. If we use AF_UNIX our program will not be deployed on two servers. Meanwhile AF_INET has been applied in our program to communicate with other clients. So DGRAM seems that it is not a good idea. Now our program needs a IPC mechansim that can commnucate between two servers and provide a high performance when two processes are run on the same server. That is the reason why I am interested in TCP friends. :)

TCP friends is another layer added into TCP stack, for what ?

Improving performance for lazy applications ?

Really, if you cared about performance, you would have added a way to use fast IPC if available.

Quoted 8 lines

TCP friends will still be slower than the available IPC mechanisms, by an order of magnitude.

So instead of spending time on this TCP friends dream, I think you should focus on existing and supported mechanisms.

I for example could add zerocopy support to AF_UNIX, if you think its worth it.

I find ZMQ to be a useful API for this. It allows abstracting away the choice of transport. You can do inprocess, unix and tcp by specifying different connection in URL like syntax.