Skip to contentSkip to conversation

Separating MPTCP packet processing from TCP

MPTCP Rao Shoaib 3 participants 6 messages May 16, 2017 Networking MPTCP

lore
Share & tools
lore

Rao Shoaib

author
#0 lore raw

Hi,

I am thinking the cleanest implementation would be to separate MPTCP processing from TCP. On the receive side, when a packet arrives TCP only does TCP processing (That leaves the code untouched), the packet is than passed upto MPTCP which does MPTCP processing and can either process the packet, drop it or send a reset.

The current implementation is doing receive processing in TCP because it wants to validate the packet and accept it in TCP or not -- but why ? that seems to be an implementation choice.

In the case where the receiver drops the packet in MPTCP, no data ack will be sent and MPTCP will re-transmit, It can retransmit even on the same flow. To achieve this the code requires some change as the DSS option has to be saved. I think this is doable and is a much cleaner solution.

Similarly we need to think about the Tx side -- That is not so straight forward but we need to think harder.

I can work on making the Rx changes but want to discuss it in case I am missing potential issues and if this is not a got option.

Comments ?

Rao

#1 lore raw

Hello Rao,

Quoted 9 lines · Rao Shoaib

I am thinking the cleanest implementation would be to separate MPTCP processing from TCP. On the receive side, when a packet arrives TCP only does TCP processing (That leaves the code untouched), the packet is than passed upto MPTCP which does MPTCP processing and can either process the packet, drop it or send a reset.

The current implementation is doing receive processing in TCP because it wants to validate the packet and accept it in TCP or not -- but why ? that seems to be an implementation choice.

you can look at mptcp_handle_options for the conditions upon which we stop processing the segment, when coming from tcp_validate_incoming. For example, when a MP_FASTCLOSE is received, the subflow gets killed with a TCP-RST.

The same holds when receiving a DSS-option without DSS-checksum although the DSS-checksum has been negotiated.

The question would be, whether we can delay the killing of the subflow & sending of the RST until later. This would mean that the packet gets completely handled (acks processed, incoming data acknowledged,...) by the TCP-stack and later on the subflow gets killed.
This *might* be ok from a protocol-perspective, but I'm not sure.

Quoted 4 lines

In the case where the receiver drops the packet in MPTCP, no data ack will be sent and MPTCP will re-transmit, It can retransmit even on the same flow. To achieve this the code requires some change as the DSS option has to be saved. I think this is doable and is a much cleaner solution.

For incoming data, that's already how it is handled. We pass the segment to the MPTCP-stack through the sk_data_ready callback, where we go over the segments, check whether their DSS-checksum is correct, check if it is in-window,... And if all is good, queue it at the MPTCP-level and send a DATA_ACK.

Cheers,
Christoph

Quoted 55 lines

Similarly we need to think about the Tx side -- That is not so straight forward but we need to think harder.

I can work on making the Rx changes but want to discuss it in case I am missing potential issues and if this is not a got option.

Comments ?

Rao

From: Rao Shoaib <rao.shoaib(a)oracle.com>
To: "multipathtcp(a)ietf.org" <multipathtcp(a)ietf.org>
Date: Mon, 15 May 2017 17:39:12 -0700
Subject: [multipathtcp] Question on Data Sequence Mapping

Hi,

Section 3.3.1 of RFC 6824 says

The data sequence mapping specifies a mapping from subflow sequence space to data sequence space. This is expressed in terms of starting sequence numbers for the subflow and the data level, and a length of bytes for which this mapping is valid.

<...>

A mapping is fixed, in that the subflow sequence number is bound to the data sequence number after the mapping has been processed. A sender MUST NOT change this mapping after it has been declared;

Does it mean that I can not map the data sequence to another (higher) TCP sequence number on the same flow.

The reason I am asking this is that if this is allowed TCP and MPTCP processing can be separated on the recv side. For example, TCP could accept the packet but MPTCP could reject it. Since data level ack would not be sent the peer would retransmit, possibly using the same flow but with a different mapping so that TCP would accept the packet.

Is this legal to do and if not than why ?

Regards,

Rao

_______________________________________________
multipathtcp mailing list
multipathtcp(a)ietf.org
https://www.ietf.org/mailman/listinfo/multipathtcp

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

author
#2 lore raw
Quoted 20 lines · Christoph Paasch

Hello Rao,

Quoted 9 lines · Rao Shoaib

I am thinking the cleanest implementation would be to separate MPTCP processing from TCP. On the receive side, when a packet arrives TCP only does TCP processing (That leaves the code untouched), the packet is than passed upto MPTCP which does MPTCP processing and can either process the packet, drop it or send a reset.

The current implementation is doing receive processing in TCP because it wants to validate the packet and accept it in TCP or not -- but why ? that seems to be an implementation choice.

you can look at mptcp_handle_options for the conditions upon which we stop processing the segment, when coming from tcp_validate_incoming. For example, when a MP_FASTCLOSE is received, the subflow gets killed with a TCP-RST.

The same holds when receiving a DSS-option without DSS-checksum although the DSS-checksum has been negotiated.

Quoted 5 lines

The question would be, whether we can delay the killing of the subflow & sending of the RST until later. This would mean that the packet gets completely handled (acks processed, incoming data acknowledged,...) by the TCP-stack and later on the subflow gets killed.
This *might* be ok from a protocol-perspective, but I'm not sure.

Hi Christoph,

That is exactly what I am suggesting. I know this is fine from the protocol perspective and I don't see why it can not be implemented. That will leave the TCP code untouched on the receive side and make the upstream guys happy because that is the fast path. Last night I was looking at the control path, that needs to be cleaned up also but that is the slow path.

Quoted 10 lines
Quoted 5 lines

In the case where the receiver drops the packet in MPTCP, no data ack will be sent and MPTCP will re-transmit, It can retransmit even on the same flow. To achieve this the code requires some change as the DSS option has to be saved. I think this is doable and is a much cleaner solution.

For incoming data, that's already how it is handled. We pass the segment to the MPTCP-stack through the sk_data_ready callback, where we go over the segments, check whether their DSS-checksum is correct, check if it is in-window,... And if all is good, queue it at the MPTCP-level and send a DATA_ACK.

Yes. However the MPTCP options are parsed and mptcp_flags and dss offset are extracted, no need to do that. Let MPTCP handle it. It's not a lot of overhead and leaves the TCP code clean. This also removes the requirement of trying to find space in tcp_skb_cb to pass any information to MPTCP.

Regards,

Rao.

Quoted 58 lines

Cheers,
Christoph

Quoted 53 lines

Similarly we need to think about the Tx side -- That is not so straight forward but we need to think harder.

I can work on making the Rx changes but want to discuss it in case I am missing potential issues and if this is not a got option.

Comments ?

Rao

From: Rao Shoaib <rao.shoaib(a)oracle.com>
To: "multipathtcp(a)ietf.org" <multipathtcp(a)ietf.org>
Date: Mon, 15 May 2017 17:39:12 -0700
Subject: [multipathtcp] Question on Data Sequence Mapping

Hi,

Section 3.3.1 of RFC 6824 says

The data sequence mapping specifies a mapping from subflow sequence space to data sequence space. This is expressed in terms of starting sequence numbers for the subflow and the data level, and a length of bytes for which this mapping is valid.

<...>

A mapping is fixed, in that the subflow sequence number is bound to the data sequence number after the mapping has been processed. A sender MUST NOT change this mapping after it has been declared;

Does it mean that I can not map the data sequence to another (higher) TCP sequence number on the same flow.

The reason I am asking this is that if this is allowed TCP and MPTCP processing can be separated on the recv side. For example, TCP could accept the packet but MPTCP could reject it. Since data level ack would not be sent the peer would retransmit, possibly using the same flow but with a different mapping so that TCP would accept the packet.

Is this legal to do and if not than why ?

Regards,

Rao

_______________________________________________
multipathtcp mailing list
multipathtcp(a)ietf.org
https://www.ietf.org/mailman/listinfo/multipathtcp
_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

#3 lore raw
Quoted 30 lines · Rao Shoaib
Quoted 22 lines · Christoph Paasch

Hello Rao,

Quoted 10 lines · Rao Shoaib

I am thinking the cleanest implementation would be to separate MPTCP processing from TCP. On the receive side, when a packet arrives TCP only does TCP processing (That leaves the code untouched), the packet is than passed upto MPTCP which does MPTCP processing and can either process the packet, drop it or send a reset.

The current implementation is doing receive processing in TCP because it wants to validate the packet and accept it in TCP or not -- but why ? that
seems to be an implementation choice.

you can look at mptcp_handle_options for the conditions upon which we stop processing the segment, when coming from tcp_validate_incoming. For example, when a MP_FASTCLOSE is received, the subflow gets killed with a TCP-RST.

The same holds when receiving a DSS-option without DSS-checksum although the DSS-checksum has been negotiated.

Quoted 5 lines

The question would be, whether we can delay the killing of the subflow & sending of the RST until later. This would mean that the packet gets completely handled (acks processed, incoming data acknowledged,...) by the TCP-stack and later on the subflow gets killed.
This *might* be ok from a protocol-perspective, but I'm not sure.

Quoted 7 lines

Hi Christoph,

That is exactly what I am suggesting. I know this is fine from the protocol perspective and I don't see why it can not be implemented. That will leave the TCP code untouched on the receive side and make the upstream guys happy because that is the fast path. Last night I was looking at the control path, that needs to be cleaned up also but that is the slow path.

This looks like a good direction to me, with the disclaimer that I don't have the depth of MPTCP protocol expertise that Christoph does. I searched through the RFC for all of the "MUSTs", and didn't turn up anything that would require earlier processing of the received MPTCP options. Maybe the TCP layer will acknowledge the packet, but that should be fine for the MPTCP layer.

The cost of parsing the TCP options twice seems pretty low, since the first pass would skip over MPTCP and the second pass would skip over "regular" options.

Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

In the case where the receiver drops the packet in MPTCP, no data ack will be sent and MPTCP will re-transmit, It can retransmit even on the same flow.
To achieve this the code requires some change as the DSS option has to be saved. I think this is doable and is a much cleaner solution.

For incoming data, that's already how it is handled. We pass the segment to the MPTCP-stack through the sk_data_ready callback, where we go over the segments, check whether their DSS-checksum is correct, check if it is in-window,... And if all is good, queue it at the MPTCP-level and send a DATA_ACK.

Quoted 8 lines

Yes. However the MPTCP options are parsed and mptcp_flags and dss offset are extracted, no need to do that. Let MPTCP handle it. It's not a lot of overhead and leaves the TCP code clean. This also removes the requirement of trying to find space in tcp_skb_cb to pass any information to MPTCP.

Quoted 3 lines
Quoted 3 lines

Similarly we need to think about the Tx side -- That is not so straight forward but we need to think harder.

Agreed. One thing I'm thinking over is whether we can pre-populate some of the TCP header. We could set a bit in the control block to indicate that there is option data there already, and when the rest of the header is written the information could be read, moved, or replaced. One tricky part of this is making sure everything works right with TSO/GSO, etc.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I can work on making the Rx changes but want to discuss it in case I am missing potential issues and if this is not a got option.

Are you approaching this as a refactor of the multipath-tcp.org kernel, or by building up from net-next?

Thanks,

--
Mat Martineau
Intel OTC

author
#4 lore raw
Quoted 54 lines · Mat Martineau
Quoted 35 lines · Rao Shoaib
Quoted 26 lines · Christoph Paasch

Hello Rao,

Quoted 9 lines · Rao Shoaib

I am thinking the cleanest implementation would be to separate MPTCP processing from TCP. On the receive side, when a packet arrives TCP only
does TCP processing (That leaves the code untouched), the packet is than
passed upto MPTCP which does MPTCP processing and can either process the
packet, drop it or send a reset.

Quoted 5 lines

The current implementation is doing receive processing in TCP because it
wants to validate the packet and accept it in TCP or not -- but why ? that
seems to be an implementation choice.

you can look at mptcp_handle_options for the conditions upon which we stop processing the segment, when coming from tcp_validate_incoming. For example, when a MP_FASTCLOSE is received, the subflow gets killed with a TCP-RST.

The same holds when receiving a DSS-option without DSS-checksum although the DSS-checksum has been negotiated.

Quoted 6 lines

The question would be, whether we can delay the killing of the subflow &
sending of the RST until later. This would mean that the packet gets completely handled (acks processed, incoming data acknowledged,...) by the TCP-stack and later on the subflow gets killed. This *might* be ok from a protocol-perspective, but I'm not sure.

Quoted 8 lines

Hi Christoph,

That is exactly what I am suggesting. I know this is fine from the protocol perspective and I don't see why it can not be implemented. That will leave the TCP code untouched on the receive side and make the upstream guys happy because that is the fast path. Last night I was looking at the control path, that needs to be cleaned up also but that is the slow path.

This looks like a good direction to me, with the disclaimer that I don't have the depth of MPTCP protocol expertise that Christoph does. I searched through the RFC for all of the "MUSTs", and didn't turn up anything that would require earlier processing of the received MPTCP options. Maybe the TCP layer will acknowledge the packet, but that should be fine for the MPTCP layer.

Quoted 3 lines

The cost of parsing the TCP options twice seems pretty low, since the first pass would skip over MPTCP and the second pass would skip over "regular" options.

I did confirm my suggestion by posting on the IETF list.

Quoted 24 lines
Quoted 24 lines
Quoted 9 lines
Quoted 9 lines

In the case where the receiver drops the packet in MPTCP, no data ack will
be sent and MPTCP will re-transmit, It can retransmit even on the same flow.
To achieve this the code requires some change as the DSS option has to be
saved. I think this is doable and is a much cleaner solution.

Quoted 8 lines

For incoming data, that's already how it is handled. We pass the segment to
the MPTCP-stack through the sk_data_ready callback, where we go over the
segments, check whether their DSS-checksum is correct, check if it is in-window,... And if all is good, queue it at the MPTCP-level and send a
DATA_ACK.

Yes. However the MPTCP options are parsed and mptcp_flags and dss offset are extracted, no need to do that. Let MPTCP handle it. It's not a lot of overhead and leaves the TCP code clean. This also removes the requirement of trying to find space in tcp_skb_cb to pass any information to MPTCP.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Similarly we need to think about the Tx side -- That is not so straight
forward but we need to think harder.

Agreed. One thing I'm thinking over is whether we can pre-populate some of the TCP header. We could set a bit in the control block to indicate that there is option data there already, and when the rest of the header is written the information could be read, moved, or replaced.

This is exactly what I was thinking of doing. There is nothing that says that TCP options have to be in front. MPTCP can fill in the options and update the data offset. TCP can work with it.

Quoted 2 lines

One tricky part of this is making sure everything works right with TSO/GSO, etc.

I have not looked into that.

This approach will not require TCP fiddling with any options except what it knows about right now and we can leave the TCP code mostly untouched.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I can work on making the Rx changes but want to discuss it in case I am
missing potential issues and if this is not a got option.

Are you approaching this as a refactor of the multipath-tcp.org kernel, or by building up from net-next?

I have been working with the code from multipath-tcp.org kernel. I am about to finish a prototype for allocating external memory to pass the options. That is when this idea came to me. I will post that may be today or tomorrow.

Rao.

Quoted 6 lines

Thanks,

Signature · 2 lines
Mat Martineau
Intel OTC
author
#5 lore raw
Quoted 57 lines · Rao Shoaib
Quoted 54 lines · Mat Martineau
Quoted 35 lines · Rao Shoaib
Quoted 26 lines · Christoph Paasch

Hello Rao,

Quoted 9 lines · Rao Shoaib

I am thinking the cleanest implementation would be to separate MPTCP processing from TCP. On the receive side, when a packet arrives TCP only
does TCP processing (That leaves the code untouched), the packet is than
passed upto MPTCP which does MPTCP processing and can either process the
packet, drop it or send a reset.

Quoted 5 lines

The current implementation is doing receive processing in TCP because it
wants to validate the packet and accept it in TCP or not -- but why ? that
seems to be an implementation choice.

you can look at mptcp_handle_options for the conditions upon which we stop processing the segment, when coming from tcp_validate_incoming. For example, when a MP_FASTCLOSE is received, the subflow gets killed with a TCP-RST.

The same holds when receiving a DSS-option without DSS-checksum although the DSS-checksum has been negotiated.

Quoted 6 lines

The question would be, whether we can delay the killing of the subflow &
sending of the RST until later. This would mean that the packet gets completely handled (acks processed, incoming data acknowledged,...) by the TCP-stack and later on the subflow gets killed. This *might* be ok from a protocol-perspective, but I'm not sure.

Quoted 8 lines

Hi Christoph,

That is exactly what I am suggesting. I know this is fine from the protocol perspective and I don't see why it can not be implemented. That will leave the TCP code untouched on the receive side and make the upstream guys happy because that is the fast path. Last night I was looking at the control path, that needs to be cleaned up also but that is the slow path.

This looks like a good direction to me, with the disclaimer that I don't have the depth of MPTCP protocol expertise that Christoph does. I searched through the RFC for all of the "MUSTs", and didn't turn up anything that would require earlier processing of the received MPTCP options. Maybe the TCP layer will acknowledge the packet, but that should be fine for the MPTCP layer.

Quoted 4 lines
Quoted 3 lines

The cost of parsing the TCP options twice seems pretty low, since the first pass would skip over MPTCP and the second pass would skip over "regular" options.

I did confirm my suggestion by posting on the IETF list.

Christoph if you see any potential issues than please speak up. I rather discuss the issues now instead of later.

Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
Quoted 10 lines
Quoted 10 lines

In the case where the receiver drops the packet in MPTCP, no data ack will
be sent and MPTCP will re-transmit, It can retransmit even on the same flow.
To achieve this the code requires some change as the DSS option has to be
saved. I think this is doable and is a much cleaner solution.

Quoted 8 lines

For incoming data, that's already how it is handled. We pass the segment to
the MPTCP-stack through the sk_data_ready callback, where we go over the
segments, check whether their DSS-checksum is correct, check if it is in-window,... And if all is good, queue it at the MPTCP-level and send a
DATA_ACK.

Yes. However the MPTCP options are parsed and mptcp_flags and dss offset are extracted, no need to do that. Let MPTCP handle it. It's not a lot of overhead and leaves the TCP code clean. This also removes the requirement of trying to find space in tcp_skb_cb to pass any information to MPTCP.

Quoted 4 lines
Quoted 4 lines

Similarly we need to think about the Tx side -- That is not so straight
forward but we need to think harder.

Quoted 8 lines
Quoted 5 lines

Agreed. One thing I'm thinking over is whether we can pre-populate some of the TCP header. We could set a bit in the control block to indicate that there is option data there already, and when the rest of the header is written the information could be read, moved, or replaced.

This is exactly what I was thinking of doing. There is nothing that says that TCP options have to be in front. MPTCP can fill in the options and update the data offset. TCP can work with it.

Actually MPTCP options would have to be at the bottom. So that will just work out.
There would be some issues to deal with re-transmission and coalescing and fragmentation.

Rao.

Quoted 28 lines
Quoted 2 lines

One tricky part of this is making sure everything works right with TSO/GSO, etc.

I have not looked into that.

This approach will not require TCP fiddling with any options except what it knows about right now and we can leave the TCP code mostly untouched.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I can work on making the Rx changes but want to discuss it in case I am
missing potential issues and if this is not a got option.

Are you approaching this as a refactor of the multipath-tcp.org kernel, or by building up from net-next?

I have been working with the code from multipath-tcp.org kernel. I am about to finish a prototype for allocating external memory to pass the options. That is when this idea came to me. I will post that may be today or tomorrow.

Rao.

Quoted 6 lines

Thanks,

Signature · 2 lines
Mat Martineau
Intel OTC