Using struct socket for subflows
Christoph Paasch
Hello,
(first, sorry for the long e-mail)
I now started working on cleaning up the input path to prepare it for input-processing without holding the MPTCP-level lock.
To do this, I go through all the places where we access data-structures from the meta-socket and see if I can move it to mptcp_data_ready or mptcp_write_space (the callbacks that are called from sk_data_ready/sk_write_space).
In tcp_check_space() we have the following:
if (mptcp(tcp_sk(sk)) ||
(sk->sk_socket &&
test_bit(SOCK_NOSPACE, &sk->sk_socket->flags))) {
tcp_new_space(sk);
if (sk->sk_socket && !test_bit(SOCK_NOSPACE, &sk->sk_socket->flags))
tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED);
}We do the if (mptcp(...)), because currently subflow's sk_socket is pointing to application's struct socket. Thus, we need to avoid the check for SOCK_NOSPACE, as otherwise we might end up not calling sk_write_space.
Other kernel-modules that create TCP-connections rather have an in-kernel struct socket. And modules like RDS even force SOCK_NOSPACE to be set, such that the TCP-stack keeps on up-calling. I thought that this was a good thing to do in MPTCP as well.
So, my goal became to have a fully functional struct socket for subflows. The benefit is also that we can end up using kernel_sendmsg, kernel_recvmsg,... in the future. It also allows to do kernel_accept() on the MPTCP-level socket to receive new subflows (a problem I mentioned in an earlier mail).
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the subflows. An idea that came up in the past when we were thinking on how to expose an MPTCP API that allows apps to control certain things on the subflows.
To get there, there are a few places where things would need to change:
* mptcp_init4_subsockets - Here, this works perfectly. It also allows to
avoid "faking" the struct socket, as we are currently doing.
* mptcp_alloc_mpcb for the active opener - This is the first problem. mptcp_alloc_mpcb() can be called with bh disabled. But sock_create_lite() assumes that bh is enabled
as it ends up doing an alloc with GFP_KERNEL.
A few ways this could be solved:
- Schedule a work-queue item in mptcp_alloc_mpcb that creates the struct socket. This looks a bit racy to me. Not sure what side-effects this
might have.
- Change things entirely, such that the master-sock is being allocated
when the connection is created. That way, we allocate all the necessary struct socket's right away.
In the past, we decided to allocate the master-sk only when receiving
the SYN/ACK. We did that so as to minimize the impact on regular TCP
when the server does not support MPTCP. But, as we are moving towards explicitly exposing MPTCP at the socket-layer, we can rethink that
decision.
Any thoughts? Is it ok to pay the cost of allocating a master-sk before
we know whether the server supports MPTCP?
I think, we should do this, and transition to that model.
* mptcp_alloc_mpcb for the passive opener - same problem as above but on the other side. We could allocate the master's struct socket upon the accept() call from the application. This again sounds a bit racy to me. The struct socket will be there for the subflow potentially much later than it has
been established. What happens if the peer sends data or an
MP_FASTCLOSE,... ?
* New subflows on the passive opener side - again, we are receiving those subflows while bh is disabled. So, we have to schedule a work-queue to
do a kerne_accept() on the MPTCP-socket.
Again something that can potentially be racy.
In general, subflow-establishment on the passive-opener side again seems to be a major pain-point. I think, we really need to redesign that.
Any thoughts, feedback, suggestions?
Or maybe, using real struct socket for subflows is not worth it? :)
Thanks,
Christoph
Lorenzo Colitti
Quoted 7 lines · Christoph Paasch
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the
subflows. An idea that came up in the past when we were thinking on how to expose an MPTCP API that allows apps to control certain things on the subflows.
+1 to this since it will allow client platforms to use setsockopt to, for example, bind subflows to interfaces (e.g., with SO_BINDTODEVICE, or with SO_MARK as used by Android).
Christoph Paasch
Quoted 11 lines · Lorenzo Colitti
Quoted 7 lines · Christoph Paasch
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the
subflows. An idea that came up in the past when we were thinking on how to expose an MPTCP API that allows apps to control certain things on the subflows.
Quoted 3 lines
+1 to this since it will allow client platforms to use setsockopt to, for example, bind subflows to interfaces (e.g., with
SO_BINDTODEVICE, or withSO_MARKas used by Android).
Thanks, Lorenzo!
Have you seen the netlink path-manager patches that have been sent to the mptcp-dev list?
(https://sympa-2.sipr.ucl.ac.be/sympa/arc/mptcp-dev/2018-03/msg00057.html)
The goal of them is to allow a daemon to control the creation and destruction of subflows through netlink.
Would that be something that fits your model?
Christoph
Mat Martineau
Hi Christoph,
Quoted 33 lines · Christoph Paasch
Hello,
(first, sorry for the long e-mail)
I now started working on cleaning up the input path to prepare it for input-processing without holding the MPTCP-level lock.
To do this, I go through all the places where we access data-structures from the meta-socket and see if I can move it to
mptcp_data_readyormptcp_write_space(the callbacks that are called from sk_data_ready/sk_write_space).In
tcp_check_space()we have the following:if (mptcp(tcp_sk(sk)) || (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags))) { tcp_new_space(sk); if (sk->sk_socket && !test_bit(SOCK_NOSPACE, &sk->sk_socket->flags)) tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); }We do the if (mptcp(...)), because currently subflow's
sk_socketis pointing to application's struct socket. Thus, we need to avoid the check forSOCK_NOSPACE, as otherwise we might end up not callingsk_write_space.Other kernel-modules that create TCP-connections rather have an in-kernel struct socket. And modules like RDS even force
SOCK_NOSPACEto be set, such that the TCP-stack keeps on up-calling. I thought that this was a good thing to do in MPTCP as well.
Quoted 5 lines
So, my goal became to have a fully functional struct socket for subflows. The benefit is also that we can end up using
kernel_sendmsg,kernel_recvmsg,... in the future. It also allows to dokernel_accept()on the MPTCP-level socket to receive new subflows (a problem I mentioned in an earlier mail).
I think we'll have a patch set illustrating this approach ready to post tomorrow. Peter and I have been working on this net-next-based code with in-kernel sockets for many months now. I missed the SOCK_NOSPACE detail though - I'll take a look at that.
Rather than using kernel_sendmsg and kernel_recvmsg, I use do_tcp_sendpages (to reduce copying and let the MPTCP connection layer share data across skbs and subflows as it needs to) and tcp_read_sock (to read intact skbs out of the rx queue). The kernel_* functions are used for most everything else though.
Quoted 5 lines
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the subflows. An idea that came up in the past when we were thinking on how to expose an MPTCP API that allows apps to control certain things on the subflows.
I like that idea too. We hadn't come up with a good design idea for propagating many of the getsockopt/setsockopt operations from the connection-level socket to all of the subflows.
Quoted 15 lines
To get there, there are a few places where things would need to change:
*
mptcp_init4_subsockets- Here, this works perfectly. It also allows to
avoid "faking" the struct socket, as we are currently doing.
*mptcp_alloc_mpcbfor the active opener - This is the first problem.mptcp_alloc_mpcb()can be called with bh disabled. Butsock_create_lite()assumes that bh is enabled
as it ends up doing an alloc withGFP_KERNEL.
A few ways this could be solved:
- Schedule a work-queue item inmptcp_alloc_mpcbthat creates the struct socket. This looks a bit racy to me. Not sure what side-effects this
might have.
- Change things entirely, such that the master-sock is being allocated
when the connection is created. That way, we allocate all the necessary struct socket's right away.
Quoted 8 lines
In the past, we decided to allocate the master-sk only when receiving the SYN/ACK. We did that so as to minimize the impact on regular TCP when the server does not support MPTCP. But, as we are moving towards explicitly exposing MPTCP at the socket-layer, we can rethink that decision.
Any thoughts? Is it ok to pay the cost of allocating a master-sk before we know whether the server supports MPTCP?
I think, we should do this, and transition to that model.
I prefer the latter. If MPTCP is exposed at the socket layer, it's only MPTCP sockets that experience the overhead (relative to a regular TCP socket) when regular TCP is used.
Quoted 11 lines
*
mptcp_alloc_mpcbfor the passive opener - same problem as above but on the other side. We could allocate the master's struct socket upon theaccept()call from the application. This again sounds a bit racy to me. The struct socket will be there for the subflow potentially much later than it has been established. What happens if the peer sends data or anMP_FASTCLOSE,... ?
* New subflows on the passive opener side - again, we are receiving those subflows while bh is disabled. So, we have to schedule a work-queue to do akerne_accept()on the MPTCP-socket.
Again something that can potentially be racy.
Quoted 6 lines
In general, subflow-establishment on the passive-opener side again seems to be a major pain-point. I think, we really need to redesign that.
Any thoughts, feedback, suggestions?
Or maybe, using real struct socket for subflows is not worth it? :)
Using the struct socket for subflows will take some work, but I think it's an important part of shaping the code for upstream. If the subflows look more like regular TCP sockets I think that will help keep the MPTCP code well partitioned from TCP.
--
Mat Martineau
Intel OTC
Rao Shoaib
Quoted 12 lines · Lorenzo Colitti
On Tue, Mar 27, 2018 at 11:18 AM, Christoph Paasch <cpaasch(a)apple.com <mailto:cpaasch(a)apple.com>> wrote:
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the
subflows. An idea that came up in the past when we were thinking on how to
expose an MPTCP API that allows apps to control certain things on the subflows.
Quoted 3 lines
+1 to this since it will allow client platforms to use setsockopt to, for example, bind subflows to interfaces (e.g., with
SO_BINDTODEVICE, or withSO_MARKas used by Android).
May I ask why it can not be done -- Only flow id should be needed. Plus for policy enforcement the meta socket would have to be consulted. I am new to Linux but based on what I have read SO_BINDTODEVICE does not guarantee much for Tx if the routing says otherwise, quick check of ip_queue_xmit() shows it first consults the routing table. I understand policy routing can be used to steer the packets. Perhaps my understanding is incorrect.
IMHO this whole idea of exposing individual flows to the application seems to go against the basic design of MPTCP. I am not sure what the use case is as TCP is streams based and MPTCP will spread out data on multiple flows. I am not a involved in the cellular space so I don' know, feel free to enlighten me.
Shoaib
Quoted 6 lines
_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp
Christoph Paasch
Quoted 38 lines · Mat Martineau
Hi Christoph,
Quoted 33 lines · Christoph Paasch
Hello,
(first, sorry for the long e-mail)
I now started working on cleaning up the input path to prepare it for input-processing without holding the MPTCP-level lock.
To do this, I go through all the places where we access data-structures from the meta-socket and see if I can move it to
mptcp_data_readyormptcp_write_space(the callbacks that are called from sk_data_ready/sk_write_space).In
tcp_check_space()we have the following:if (mptcp(tcp_sk(sk)) || (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags))) { tcp_new_space(sk); if (sk->sk_socket && !test_bit(SOCK_NOSPACE, &sk->sk_socket->flags)) tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); }We do the if (mptcp(...)), because currently subflow's
sk_socketis pointing to application's struct socket. Thus, we need to avoid the check forSOCK_NOSPACE, as otherwise we might end up not callingsk_write_space.Other kernel-modules that create TCP-connections rather have an in-kernel struct socket. And modules like RDS even force
SOCK_NOSPACEto be set, such that the TCP-stack keeps on up-calling. I thought that this was a good thing to do in MPTCP as well.
Quoted 8 lines
Quoted 5 lines
So, my goal became to have a fully functional struct socket for subflows. The benefit is also that we can end up using
kernel_sendmsg,kernel_recvmsg,... in the future. It also allows to dokernel_accept()on the MPTCP-level socket to receive new subflows (a problem I mentioned in an earlier mail).I think we'll have a patch set illustrating this approach ready to post tomorrow.
Awesome! I am waiting for your patch :)
Quoted 9 lines
Peter and I have been working on this net-next-based code with in-kernel sockets for many months now. I missed the
SOCK_NOSPACEdetail though - I'll take a look at that.Rather than using
kernel_sendmsgandkernel_recvmsg, I usedo_tcp_sendpages(to reduce copying and let the MPTCP connection layer share data across skbs and subflows as it needs to) andtcp_read_sock(to read intact skbs out of the rx queue). The kernel_* functions are used for most everything else though.
Do you also handle kernel_accept ?
If yes, how do you do it? Because, it's quite complex to avoid races.
Quoted 34 lines
Quoted 5 lines
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the subflows. An idea that came up in the past when we were thinking on how to expose an MPTCP API that allows apps to control certain things on the subflows.
I like that idea too. We hadn't come up with a good design idea for propagating many of the getsockopt/setsockopt operations from the connection-level socket to all of the subflows.
Quoted 15 lines
To get there, there are a few places where things would need to change:
*
mptcp_init4_subsockets- Here, this works perfectly. It also allows to
avoid "faking" the struct socket, as we are currently doing.
*mptcp_alloc_mpcbfor the active opener - This is the first problem.mptcp_alloc_mpcb()can be called with bh disabled. Butsock_create_lite()assumes that bh is enabled
as it ends up doing an alloc withGFP_KERNEL.
A few ways this could be solved:
- Schedule a work-queue item inmptcp_alloc_mpcbthat creates the struct socket. This looks a bit racy to me. Not sure what side-effects this
might have.
- Change things entirely, such that the master-sock is being allocated
when the connection is created. That way, we allocate all the necessary struct socket's right away.Quoted 8 lines
In the past, we decided to allocate the master-sk only when receiving the SYN/ACK. We did that so as to minimize the impact on regular TCP when the server does not support MPTCP. But, as we are moving towards explicitly exposing MPTCP at the socket-layer, we can rethink that decision.
Any thoughts? Is it ok to pay the cost of allocating a master-sk before we know whether the server supports MPTCP?
I think, we should do this, and transition to that model.
Quoted 3 lines
I prefer the latter. If MPTCP is exposed at the socket layer, it's only MPTCP sockets that experience the overhead (relative to a regular TCP socket) when regular TCP is used.
Are you doing the latter in your implementation?
Christoph
Quoted 28 lines
Quoted 12 lines
*
mptcp_alloc_mpcbfor the passive opener - same problem as above but on the other side. We could allocate the master's struct socket upon theaccept()call from the application. This again sounds a bit racy to me. The struct socket will be there for the subflow potentially much later than it has been established. What happens if the peer sends data or anMP_FASTCLOSE,... ?
* New subflows on the passive opener side - again, we are receiving those subflows while bh is disabled. So, we have to schedule a work-queue to do akerne_accept()on the MPTCP-socket.
Again something that can potentially be racy.Quoted 6 lines
In general, subflow-establishment on the passive-opener side again seems to be a major pain-point. I think, we really need to redesign that.
Any thoughts, feedback, suggestions?
Or maybe, using real struct socket for subflows is not worth it? :)Using the struct socket for subflows will take some work, but I think it's an important part of shaping the code for upstream. If the subflows look more like regular TCP sockets I think that will help keep the MPTCP code well partitioned from TCP.
--
Mat Martineau
Intel OTC
Mat Martineau
Quoted 54 lines · Christoph Paasch
Quoted 38 lines · Mat Martineau
Hi Christoph,
Quoted 33 lines · Christoph Paasch
Hello,
(first, sorry for the long e-mail)
I now started working on cleaning up the input path to prepare it for input-processing without holding the MPTCP-level lock.
To do this, I go through all the places where we access data-structures from the meta-socket and see if I can move it to
mptcp_data_readyormptcp_write_space(the callbacks that are called from sk_data_ready/sk_write_space).In
tcp_check_space()we have the following:if (mptcp(tcp_sk(sk)) || (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags))) { tcp_new_space(sk); if (sk->sk_socket && !test_bit(SOCK_NOSPACE, &sk->sk_socket->flags)) tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); }We do the if (mptcp(...)), because currently subflow's
sk_socketis pointing to application's struct socket. Thus, we need to avoid the check forSOCK_NOSPACE, as otherwise we might end up not callingsk_write_space.Other kernel-modules that create TCP-connections rather have an in-kernel struct socket. And modules like RDS even force
SOCK_NOSPACEto be set, such that the TCP-stack keeps on up-calling. I thought that this was a good thing to do in MPTCP as well.Quoted 8 lines
Quoted 5 lines
So, my goal became to have a fully functional struct socket for subflows. The benefit is also that we can end up using
kernel_sendmsg,kernel_recvmsg,... in the future. It also allows to dokernel_accept()on the MPTCP-level socket to receive new subflows (a problem I mentioned in an earlier mail).I think we'll have a patch set illustrating this approach ready to post tomorrow.
Awesome! I am waiting for your patch :)
Quoted 4 lines
Peter and I have been working on this net-next-based code with in-kernel sockets for many months now. I missed the
SOCK_NOSPACEdetail though - I'll take a look at that.
Quoted 8 lines
Quoted 5 lines
Rather than using
kernel_sendmsgandkernel_recvmsg, I usedo_tcp_sendpages(to reduce copying and let the MPTCP connection layer share data across skbs and subflows as it needs to) andtcp_read_sock(to read intact skbs out of the rx queue). The kernel_* functions are used for most everything else though.Do you also handle
kernel_accept?
If yes, how do you do it? Because, it's quite complex to avoid races.
Yes, but we don't support joins yet so it's maybe only the trivial case. The connection level socket owns a subflow listening socket (which it created at init time), and when the application calls accept() it's immediately passed through to kernel_accept() for the subflow.
Quoted 35 lines
Quoted 35 lines
Quoted 6 lines
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the subflows. An idea that came up in the past when we were thinking on how to expose an MPTCP API that allows apps to control certain things on the subflows.
I like that idea too. We hadn't come up with a good design idea for propagating many of the getsockopt/setsockopt operations from the connection-level socket to all of the subflows.
Quoted 23 lines
To get there, there are a few places where things would need to change:
*
mptcp_init4_subsockets- Here, this works perfectly. It also allows to
avoid "faking" the struct socket, as we are currently doing.
*mptcp_alloc_mpcbfor the active opener - This is the first problem.mptcp_alloc_mpcb()can be called with bh disabled. Butsock_create_lite()assumes that bh is enabled
as it ends up doing an alloc withGFP_KERNEL.
A few ways this could be solved:
- Schedule a work-queue item inmptcp_alloc_mpcbthat creates the struct socket. This looks a bit racy to me. Not sure what side-effects this
might have.
- Change things entirely, such that the master-sock is being allocated
when the connection is created. That way, we allocate all the necessary struct socket's right away.
In the past, we decided to allocate the master-sk only when receiving
the SYN/ACK. We did that so as to minimize the impact on regular TCP
when the server does not support MPTCP. But, as we are moving towards explicitly exposing MPTCP at the socket-layer, we can rethink that
decision.
Any thoughts? Is it ok to pay the cost of allocating a master-sk before
we know whether the server supports MPTCP?
I think, we should do this, and transition to that model.
Quoted 5 lines
Quoted 3 lines
I prefer the latter. If MPTCP is exposed at the socket layer, it's only MPTCP sockets that experience the overhead (relative to a regular TCP socket) when regular TCP is used.
Are you doing the latter in your implementation?
Yes. There's always a connection level socket and a subflow socket if an IPPROTO_MPTCP socket was requested, even if only regular TCP ends up being used.
Mat
Quoted 26 lines
Quoted 26 lines
Quoted 13 lines
*
mptcp_alloc_mpcbfor the passive opener - same problem as above but on the other side. We could allocate the master's struct socket upon theaccept()call from the application. This again sounds a bit racy to me. The struct socket will be there for the subflow potentially much later than it has been established. What happens if the peer sends data or anMP_FASTCLOSE,... ?
* New subflows on the passive opener side - again, we are receiving those subflows while bh is disabled. So, we have to schedule a work-queue to do akerne_accept()on the MPTCP-socket.
Again something that can potentially be racy.Quoted 6 lines
In general, subflow-establishment on the passive-opener side again seems to be a major pain-point. I think, we really need to redesign that.
Any thoughts, feedback, suggestions?
Or maybe, using real struct socket for subflows is not worth it? :)Using the struct socket for subflows will take some work, but I think it's an important part of shaping the code for upstream. If the subflows look more like regular TCP sockets I think that will help keep the MPTCP code well partitioned from TCP.
Quoted 4 lines
Quoted 3 lines
--
Mat Martineau
Intel OTC
--
Mat Martineau
Intel OTC
Stephen Brennan
Hi Rao et al.,
Quoted 15 lines · Rao Shoaib
Quoted 12 lines · Lorenzo Colitti
On Tue, Mar 27, 2018 at 11:18 AM, Christoph Paasch <cpaasch(a)apple.com <mailto:cpaasch(a)apple.com>> wrote:
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the
subflows. An idea that came up in the past when we were thinking on how to
expose an MPTCP API that allows apps to control certain things on the subflows.
Quoted 4 lines
Quoted 3 lines
+1 to this since it will allow client platforms to use setsockopt to, for example, bind subflows to interfaces (e.g., with
SO_BINDTODEVICE, or withSO_MARKas used by Android).May I ask why it can not be done -- Only flow id should be needed.
It sounds like you're referring to the proposed MPTCP_SUB_GETSOCKOPT and MPTCP_SUB_SETSOCKOPT operations in this IETF draft [1], correct?
[1]: https://tools.ietf.org/html/draft-hesmans-mptcp-socket-00
Quoted 7 lines
Plus for
policy enforcement the meta socket would have to be consulted. I am new to Linux but based on what I have readSO_BINDTODEVICEdoes not guarantee much for Tx if the routing says otherwise, quick check ofip_queue_xmit()shows it first consults the routing table. I understand policy routing can be used to steer the packets. Perhaps my understanding is incorrect.
Quoted 5 lines
IMHO this whole idea of exposing individual flows to the application seems to go against the basic design of MPTCP. I am not sure what the use case is as TCP is streams based and MPTCP will spread out data on multiple flows. I am not a involved in the cellular space so I don' know, feel free to enlighten me.
I'm also a bit confused at this. I've read the Socket API drafts from IETF, which seem to be motivated by the idea that mobile apps will want to directly control which subflows and paths are created (similar to the iOS APIs released recently). It sounds like we have two approaches: (1) MPTCP as a drop-in replacement for TCP, so unmodified applications may use it, and (2) MPTCP as an explicitly requested protocol which user-space applications request, and then manipulate via socket options. Given the amount of drafts and other work put in so far, it seems that use case (2) has enough demand that it ought to be supported.
Going slightly off topic from this discussion, it seems that a lot of what is desired in use case (2) is allowing the application to manage its own paths. We now have a Netlink API for userspace path managers. Have we thought about the possibility of a userspace application creating an MPTCP socket, and also a netlink socket, so that it could manage its own paths? Just an off-the-wall idea. If such a thing were possible, it could give the userspace app much more control over its own path management, while reducing duplication of functionality in-kernel (e.g. wouldn't have both socket option for opening a subflow, plus a netlink command for opening a subflow). A userspace library could hide some of this complexity for users of use-case (2), while still allowing use-case (1) to exist through the normal socket API.
Stephen
Christoph Paasch
Quoted 63 lines · Mat Martineau
Quoted 60 lines · Christoph Paasch
Quoted 38 lines · Mat Martineau
Hi Christoph,
Quoted 33 lines · Christoph Paasch
Hello,
(first, sorry for the long e-mail)
I now started working on cleaning up the input path to prepare it for input-processing without holding the MPTCP-level lock.
To do this, I go through all the places where we access data-structures from the meta-socket and see if I can move it to
mptcp_data_readyormptcp_write_space(the callbacks that are called from sk_data_ready/sk_write_space).In
tcp_check_space()we have the following:if (mptcp(tcp_sk(sk)) || (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags))) { tcp_new_space(sk); if (sk->sk_socket && !test_bit(SOCK_NOSPACE, &sk->sk_socket->flags)) tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); }We do the if (mptcp(...)), because currently subflow's
sk_socketis pointing to application's struct socket. Thus, we need to avoid the check forSOCK_NOSPACE, as otherwise we might end up not callingsk_write_space.Other kernel-modules that create TCP-connections rather have an in-kernel struct socket. And modules like RDS even force
SOCK_NOSPACEto be set, such that the TCP-stack keeps on up-calling. I thought that this was a good thing to do in MPTCP as well.Quoted 8 lines
Quoted 5 lines
So, my goal became to have a fully functional struct socket for subflows. The benefit is also that we can end up using
kernel_sendmsg,kernel_recvmsg,... in the future. It also allows to dokernel_accept()on the MPTCP-level socket to receive new subflows (a problem I mentioned in an earlier mail).I think we'll have a patch set illustrating this approach ready to post tomorrow.
Awesome! I am waiting for your patch :)
Quoted 9 lines
Peter and I have been working on this net-next-based code with in-kernel sockets for many months now. I missed the
SOCK_NOSPACEdetail though - I'll take a look at that.Rather than using
kernel_sendmsgandkernel_recvmsg, I usedo_tcp_sendpages(to reduce copying and let the MPTCP connection layer share data across skbs and subflows as it needs to) andtcp_read_sock(to read intact skbs out of the rx queue). The kernel_* functions are used for most everything else though.
Quoted 7 lines
Quoted 2 lines
Do you also handle
kernel_accept?
If yes, how do you do it? Because, it's quite complex to avoid races.Yes, but we don't support joins yet so it's maybe only the trivial case. The connection level socket owns a subflow listening socket (which it created at init time), and when the application calls
accept()it's immediately passed through tokernel_accept()for the subflow.
I see you posted your patches. I will try to go over them before our meeting tonight.
Christoph
Christoph Paasch
Hello,
Quoted 41 lines · Stephen Brennan
Hi Rao et al.,
Quoted 15 lines · Rao Shoaib
Quoted 12 lines · Lorenzo Colitti
On Tue, Mar 27, 2018 at 11:18 AM, Christoph Paasch <cpaasch(a)apple.com <mailto:cpaasch(a)apple.com>> wrote:
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the
subflows. An idea that came up in the past when we were thinking on how to
expose an MPTCP API that allows apps to control certain things on the subflows.Quoted 4 lines
Quoted 3 lines
+1 to this since it will allow client platforms to use setsockopt to, for example, bind subflows to interfaces (e.g., with
SO_BINDTODEVICE, or withSO_MARKas used by Android).May I ask why it can not be done -- Only flow id should be needed.
It sounds like you're referring to the proposed
MPTCP_SUB_GETSOCKOPTandMPTCP_SUB_SETSOCKOPToperations in this IETF draft [1], correct?[1]: https://tools.ietf.org/html/draft-hesmans-mptcp-socket-00
Quoted 7 lines
Plus for
policy enforcement the meta socket would have to be consulted. I am new to Linux but based on what I have readSO_BINDTODEVICEdoes not guarantee much for Tx if the routing says otherwise, quick check ofip_queue_xmit()shows it first consults the routing table. I understand policy routing can be used to steer the packets. Perhaps my understanding is incorrect.Quoted 5 lines
IMHO this whole idea of exposing individual flows to the application seems to go against the basic design of MPTCP. I am not sure what the use case is as TCP is streams based and MPTCP will spread out data on multiple flows. I am not a involved in the cellular space so I don' know, feel free to enlighten me.
Quoted 4 lines
I'm also a bit confused at this. I've read the Socket API drafts from IETF, which seem to be motivated by the idea that mobile apps will want to directly control which subflows and paths are created (similar to the iOS APIs released recently).
The iOS APIs only allow an app to communicate its intention (handover, interactive, aggregate). At the end of the day, the app is not able to trigger the creation of a subflow on cellular.
Just nit-picking ;-)
Quoted 6 lines
It sounds like we have two approaches: (1) MPTCP
as a drop-in replacement for TCP, so unmodified applications may use it, and (2) MPTCP as an explicitly requested protocol which user-space applications request, and then manipulate via socket options. Given the amount of drafts and other work put in so far, it seems that use case (2) has enough demand that it ought to be supported.
Yes, the drop-in replacement does not work IMO, because the MPTCP-stack needs more context to make decisions.
Quoted 10 lines
Going slightly off topic from this discussion, it seems that a lot of what is desired in use case (2) is allowing the application to manage its own paths. We now have a Netlink API for userspace path managers. Have we thought about the possibility of a userspace application creating an MPTCP socket, and also a netlink socket, so that it could manage its own paths? Just an off-the-wall idea. If such a thing were possible, it could give the userspace app much more control over its own path management, while reducing duplication of functionality in-kernel (e.g. wouldn't have both socket option for opening a subflow, plus a netlink command for opening a subflow).
Yes, this is definitely a viable solution. The app could use a library that is able to speak netlink-API and manage subflows.
Your library? ;-)
Quoted 3 lines
A userspace library could hide some of this complexity for users of use-case (2), while still allowing use-case (1) to exist through the normal socket API.
The idea behind exposing subflows as FDs came from the need that apps would want to control socket-options on a per-subflow basis. E.g., TCP-keepalive, SO_MARK (for Android),...
The app would want to get stats like TCP_INFO,...
We don't have a good way at the moment to do this. And, using file-descriptors seemed like a viable solution.
That doesn't mean that we will do this right away :)
But - as from my original mail - having fully functional struct socket attached to the subflows allows us to move that way in the future.
Christoph
Rao Shoaib
Quoted 45 lines · Christoph Paasch
Hello,
Quoted 38 lines · Stephen Brennan
Hi Rao et al.,
Quoted 14 lines · Rao Shoaib
Quoted 12 lines · Lorenzo Colitti
On Tue, Mar 27, 2018 at 11:18 AM, Christoph Paasch <cpaasch(a)apple.com <mailto:cpaasch(a)apple.com>> wrote:
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the
subflows. An idea that came up in the past when we were thinking on how to
expose an MPTCP API that allows apps to control certain things on the subflows.Quoted 4 lines
Quoted 3 lines
+1 to this since it will allow client platforms to use setsockopt to, for example, bind subflows to interfaces (e.g., with
SO_BINDTODEVICE, or withSO_MARKas used by Android).May I ask why it can not be done -- Only flow id should be needed.
It sounds like you're referring to the proposed
MPTCP_SUB_GETSOCKOPTandMPTCP_SUB_SETSOCKOPToperations in this IETF draft [1], correct?[1]: https://tools.ietf.org/html/draft-hesmans-mptcp-socket-00
Quoted 7 lines
Plus for
policy enforcement the meta socket would have to be consulted. I am new to Linux but based on what I have readSO_BINDTODEVICEdoes not guarantee much for Tx if the routing says otherwise, quick check ofip_queue_xmit()shows it first consults the routing table. I understand policy routing can be used to steer the packets. Perhaps my understanding is incorrect.Quoted 5 lines
IMHO this whole idea of exposing individual flows to the application seems to go against the basic design of MPTCP. I am not sure what the use case is as TCP is streams based and MPTCP will spread out data on multiple flows. I am not a involved in the cellular space so I don' know, feel free to enlighten me.
Quoted 4 lines
I'm also a bit confused at this. I've read the Socket API drafts from IETF, which seem to be motivated by the idea that mobile apps will want to directly control which subflows and paths are created (similar to the iOS APIs released recently).
Quoted 5 lines
The iOS APIs only allow an app to communicate its intention (handover, interactive, aggregate). At the end of the day, the app is not able to trigger the creation of a subflow on cellular.
Just nit-picking ;-)
Creating subflows and managing them can be done today.
Quoted 9 lines
Quoted 7 lines
It sounds like we have two approaches: (1) MPTCP
as a drop-in replacement for TCP, so unmodified applications may use it, and (2) MPTCP as an explicitly requested protocol which user-space applications request, and then manipulate via socket options. Given the amount of drafts and other work put in so far, it seems that use case (2) has enough demand that it ought to be supported.Yes, the drop-in replacement does not work IMO, because the MPTCP-stack needs more context to make decisions.
#1 does not exclude #2. You can have both options, system wide or just on demand. The current implementation hands over the full packet to MPCTP to do whatever it wants.
Provide an example that can not be handled by the current design that I have posted.
Quoted 11 lines
Quoted 11 lines
Going slightly off topic from this discussion, it seems that a lot of what is desired in use case (2) is allowing the application to manage its own paths. We now have a Netlink API for userspace path managers. Have we thought about the possibility of a userspace application creating an MPTCP socket, and also a netlink socket, so that it could manage its own paths? Just an off-the-wall idea. If such a thing were possible, it could give the userspace app much more control over its own path management, while reducing duplication of functionality in-kernel (e.g. wouldn't have both socket option for opening a subflow, plus a netlink command for opening a subflow).
Quoted 4 lines
Yes, this is definitely a viable solution. The app could use a library that is able to speak netlink-API and manage subflows.
Your library? ;-)
Yes that is what I have in mind also but this does not require any change in the current design. I have not read the netlink design but I thought that it would already have that.
Quoted 11 lines
Quoted 4 lines
A userspace library could hide some of this complexity for users of use-case (2), while still allowing use-case (1) to exist through the normal socket API.
The idea behind exposing subflows as FDs came from the need that apps would want to control socket-options on a per-subflow basis. E.g., TCP-keepalive,
SO_MARK(for Android),...
The app would want to get stats likeTCP_INFO,...We don't have a good way at the moment to do this. And, using file-descriptors seemed like a viable solution.
No it does not. Application needs to handle only one fd. All the above can be done but is not in scope for the initial implementation.
Quoted 5 lines
That doesn't mean that we will do this right away :)
But - as from my original mail - having fully functional struct socket attached to the subflows allows us to move that way in the future.
I do not see any technical argument. In fact I argue that if for example fd are needed in the future it would be simple to expose the current subflows without requiring any change to MPTCP or TCP. The current design does not exclude anything.
Shoaib
Rao Shoaib
Hi Steve,
Sorry I did not respond to your email, too many issues to worry about :-)
Quoted 17 lines · Stephen Brennan
Hi Rao et al.,
Quoted 14 lines · Rao Shoaib
Quoted 12 lines · Lorenzo Colitti
On Tue, Mar 27, 2018 at 11:18 AM, Christoph Paasch <cpaasch(a)apple.com <mailto:cpaasch(a)apple.com>> wrote:
It also would allow us to expose subflows as file-descriptors to the user-space. That way the user-space can do setsockopt, getsockopt,... on the
subflows. An idea that came up in the past when we were thinking on how to
expose an MPTCP API that allows apps to control certain things on the subflows.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
+1 to this since it will allow client platforms to use setsockopt to, for example, bind subflows to interfaces (e.g., with
SO_BINDTODEVICE, or withSO_MARKas used by Android).May I ask why it can not be done -- Only flow id should be needed.
It sounds like you're referring to the proposed
MPTCP_SUB_GETSOCKOPTandMPTCP_SUB_SETSOCKOPToperations in this IETF draft [1], correct?[1]: https://tools.ietf.org/html/draft-hesmans-mptcp-socket-00
Actually I have not read the draft, my comment was based on general understanding of the kernel. I am coming of Solaris and we know how to create custom socket, special setopt/getopt , special fd's to expose something etc etc. It is not rocket science. If someone has already thought of it then we have less work to do.
Any change of this kind has to be discussed in a standards body so vendors and users can comment.
Quoted 13 lines
Quoted 8 lines
Plus for
policy enforcement the meta socket would have to be consulted. I am new to Linux but based on what I have readSO_BINDTODEVICEdoes not guarantee much for Tx if the routing says otherwise, quick check ofip_queue_xmit()shows it first consults the routing table. I understand policy routing can be used to steer the packets. Perhaps my understanding is incorrect.Quoted 5 lines
IMHO this whole idea of exposing individual flows to the application seems to go against the basic design of MPTCP. I am not sure what the use case is as TCP is streams based and MPTCP will spread out data on multiple flows. I am not a involved in the cellular space so I don' know, feel free to enlighten me.
Quoted 9 lines
I'm also a bit confused at this. I've read the Socket API drafts from IETF, which seem to be motivated by the idea that mobile apps will want to directly control which subflows and paths are created (similar to the iOS APIs released recently). It sounds like we have two approaches: (1) MPTCP as a drop-in replacement for TCP, so unmodified applications may use it, and (2) MPTCP as an explicitly requested protocol which user-space applications request, and then manipulate via socket options. Given the amount of drafts and other work put in so far, it seems that use case (2) has enough demand that it ought to be supported.
Can you point me to the drafts or requests. As I said with the current design everything is possible. No change is needed.
Shoaib
Quoted 17 lines
Going slightly off topic from this discussion, it seems that a lot of what is desired in use case (2) is allowing the application to manage its own paths. We now have a Netlink API for userspace path managers. Have we thought about the possibility of a userspace application creating an MPTCP socket, and also a netlink socket, so that it could manage its own paths? Just an off-the-wall idea. If such a thing were possible, it could give the userspace app much more control over its own path management, while reducing duplication of functionality in-kernel (e.g. wouldn't have both socket option for opening a subflow, plus a netlink command for opening a subflow). A userspace library could hide some of this complexity for users of use-case (2), while still allowing use-case (1) to exist through the normal socket API.
Stephen
Rao Shoaib
Quoted 75 lines · Rao Shoaib
Hi Steve,
Sorry I did not respond to your email, too many issues to worry about :-)
Quoted 23 lines · Stephen Brennan
Hi Rao et al.,
Quoted 20 lines · Rao Shoaib
Quoted 18 lines · Lorenzo Colitti
On Tue, Mar 27, 2018 at 11:18 AM, Christoph Paasch <cpaasch(a)apple.com <mailto:cpaasch(a)apple.com>> wrote:
It also would allow us to expose subflows as file-descriptors to the
user-space. That way the user-space can do setsockopt, getsockopt,... on the
subflows. An idea that came up in the past when we were thinking on how to
expose an MPTCP API that allows apps to control certain things on the
subflows.+1 to this since it will allow client platforms to use setsockopt to, for example, bind subflows to interfaces (e.g., with
SO_BINDTODEVICE, or
withSO_MARKas used by Android).Quoted 5 lines
Quoted 1 line
May I ask why it can not be done -- Only flow id should be needed.
It sounds like you're referring to the proposed
MPTCP_SUB_GETSOCKOPTandMPTCP_SUB_SETSOCKOPToperations in this IETF draft [1], correct?[1]: https://tools.ietf.org/html/draft-hesmans-mptcp-socket-00
Actually I have not read the draft, my comment was based on general understanding of the kernel. I am coming of Solaris and we know how to create custom socket, special setopt/getopt , special fd's to expose something etc etc. It is not rocket science. If someone has already thought of it then we have less work to do.
Any change of this kind has to be discussed in a standards body so vendors and users can comment.
Quoted 20 lines
Quoted 12 lines
Plus for
policy enforcement the meta socket would have to be consulted. I am new to
Linux but based on what I have readSO_BINDTODEVICEdoes not guarantee much
for Tx if the routing says otherwise, quick check ofip_queue_xmit()shows
it first consults the routing table. I understand policy routing can be used
to steer the packets. Perhaps my understanding is incorrect.Quoted 8 lines
IMHO this whole idea of exposing individual flows to the application seems
to go against the basic design of MPTCP. I am not sure what the use case is
as TCP is streams based and MPTCP will spread out data on multiple flows. I
am not a involved in the cellular space so I don' know, feel free to enlighten me.Quoted 13 lines
I'm also a bit confused at this. I've read the Socket API drafts from IETF,
which seem to be motivated by the idea that mobile apps will want to directly control which subflows and paths are created (similar to the iOS
APIs released recently). It sounds like we have two approaches: (1) MPTCP
as a drop-in replacement for TCP, so unmodified applications may use it, and (2) MPTCP as an explicitly requested protocol which user-space applications request, and then manipulate via socket options. Given the amount of drafts and other work put in so far, it seems that use case
(2)
has enough demand that it ought to be supported.
Quoted 4 lines
Can you point me to the drafts or requests. As I said with the current design everything is possible. No change is needed.
Shoaib
Steve take a look at the following -- The possibilities are endless.
https://arxiv.org/pdf/1707.03585.pdf
The best approach I think is to enhance netlink. I have a pie in the sky idea, we can have meta/master socket register with netlink, identified by src and destination addresses for direct communication.
Shoaib