Skip to contentSkip to conversation

mptcp: add ooo prune support

RFC In review Florian Westphal 2 participants 4 messages Oct 2, 2020 – Oct 9, 2020

Subsystems (2)
lore
Share & tools
lore

Florian Westphal

author
#0 lore raw

RFC 1/1 mptcp: add ooo prune support

It might be possible that entire receive buffer is occupied by skbs in the OOO queue.

In this case we can't pull more skbs from subflows and the holes will never be filled.

If this happens, schedule the work queue and prune ~12% of skbs to make space available. Also add a MIB counter for this.

Signed-off-by Florian Westphal <fw(a)strlen.de>

---
Paolo, this does relate a bit to our discussion wrt. oow tracking. I thought we might need to add some sort of cushion to account for window discrepancies, but that might then get us in a state where wmem might be full...

What do you think?

I did NOT see such a problem in practice, this is a theoretical "fix". TCP has similar code to deal with corner cases of small-oow packets.

 net/mptcp/mib.c      |  1 +
 net/mptcp/mib.h      |  1 +
 net/mptcp/protocol.c | 48 ++++++++++++++++++++++++++++++++++++++++++--
 net/mptcp/protocol.h |  1 +
 4 files changed, 49 insertions(+), 2 deletions(-)
net/mptcp/mib.c +1 -0
index 84d119436b22..65c575e3af60 100644
@@ -25,6 +25,7 @@ static const struct snmp_mib mptcp_snmp_list[] = {
 	SNMP_MIB_ITEM("OFOQueueTail", MPTCP_MIB_OFOQUEUETAIL),
 	SNMP_MIB_ITEM("OFOQueue", MPTCP_MIB_OFOQUEUE),
 	SNMP_MIB_ITEM("OFOMerge", MPTCP_MIB_OFOMERGE),
+	SNMP_MIB_ITEM("OFOPrune", MPTCP_MIB_OFOPRUNE),
 	SNMP_MIB_ITEM("NoDSSInWindow", MPTCP_MIB_NODSSWINDOW),
 	SNMP_MIB_ITEM("DuplicateData", MPTCP_MIB_DUPDATA),
 	SNMP_MIB_ITEM("AddAddr", MPTCP_MIB_ADDADDR),
net/mptcp/mib.h +1 -0
index 47bcecce1106..75a7fb3a87db 100644
@@ -18,6 +18,7 @@ enum linux_mptcp_mib_field {
 	MPTCP_MIB_OFOQUEUETAIL,	/* Segments inserted into OoO queue tail */
 	MPTCP_MIB_OFOQUEUE,		/* Segments inserted into OoO queue */
 	MPTCP_MIB_OFOMERGE,		/* Segments merged in OoO queue */
+	MPTCP_MIB_OFOPRUNE,		/* Segments pruned from OoO queue */
 	MPTCP_MIB_NODSSWINDOW,		/* Segments not in MPTCP windows */
 	MPTCP_MIB_DUPDATA,		/* Segments discarded due to duplicate DSS */
 	MPTCP_MIB_ADDADDR,		/* Received ADD_ADDR with echo-flag=0 */
net/mptcp/protocol.c +46 -2

66 lines not on this page · raw message

net/mptcp/protocol.h +1 -0
index d33e9676a1a3..360441fdaa93 100644
@@ -91,6 +91,7 @@
 #define MPTCP_WORK_EOF		3
 #define MPTCP_FALLBACK_DONE	4
 #define MPTCP_WORKER_RUNNING	5
+#define MPTCP_WORK_PRUNE_OFO	6
 
 static inline bool before64(__u64 seq1, __u64 seq2)
 {
Signature · 1 line
2.26.2
#1 lore raw
Quoted 16 lines · Florian Westphal

It might be possible that entire receive buffer is occupied by skbs in the OOO queue.

In this case we can't pull more skbs from subflows and the holes will never be filled.

If this happens, schedule the work queue and prune ~12% of skbs to make space available. Also add a MIB counter for this.

Signed-off-by Florian Westphal <fw(a)strlen.de>

---
Paolo, this does relate a bit to our discussion wrt. oow tracking. I thought we might need to add some sort of cushion to account for window discrepancies, but that might then get us in a state where wmem might be full...

Quoted 5 lines

What do you think?

I did NOT see such a problem in practice, this is a theoretical "fix". TCP has similar code to deal with corner cases of small-oow packets.

Is there a benefit to relying on the workqueue to discard skbs from the ooo queue rather than handling that as data is moved from the subflows?

The cause of "holes" in MPTCP reassembly is a little different from TCP, since TCP takes care of the packet loss problem it seems to me the main issue is recovering from path loss (assuming the window and rcv buffer sizes are sane). We need to keep pulling from the subflows to potentially fill the holes, and the most valuable data to make progress would be earlier in the sequence space. We potentially throw less away if discarding later-sequence ooo queued skbs as subflow data arrives rather than doing the whole 12.5% chunk, and don't have the latency of waiting for the workqueue to get scheduled.

(I should look at the multipath-tcp.org kernel to understand how it's done there but I haven't had a chance to do that yet)

--
Mat Martineau
Intel

author
#2 lore raw
Quoted 18 lines · Mat Martineau
Quoted 16 lines · Florian Westphal

It might be possible that entire receive buffer is occupied by skbs in the OOO queue.

In this case we can't pull more skbs from subflows and the holes will never be filled.

If this happens, schedule the work queue and prune ~12% of skbs to make space available. Also add a MIB counter for this.

Signed-off-by Florian Westphal <fw(a)strlen.de>

---
Paolo, this does relate a bit to our discussion wrt. oow tracking. I thought we might need to add some sort of cushion to account for window discrepancies, but that might then get us in a state where wmem might be full...

Quoted 8 lines
Quoted 5 lines

What do you think?

I did NOT see such a problem in practice, this is a theoretical "fix". TCP has similar code to deal with corner cases of small-oow packets.

Is there a benefit to relying on the workqueue to discard skbs from the ooo queue rather than handling that as data is moved from the subflows?

It seemed simpler to do it this way. When data_ready callback is running its the first spot where we make checks wrt. buffer occupancy.

move_skbs_to_msk() (which is called right after) may not be able to acquire the msk lock. I did not consider this performance critical to deal with both 'can do it inline' and 'need to defer to worker' cases.

Quoted 9 lines

The cause of "holes" in MPTCP reassembly is a little different from TCP, since TCP takes care of the packet loss problem it seems to me the main issue is recovering from path loss (assuming the window and rcv buffer sizes are sane). We need to keep pulling from the subflows to potentially fill the holes, and the most valuable data to make progress would be earlier in the sequence space. We potentially throw less away if discarding later-sequence ooo queued skbs as subflow data arrives rather than doing the whole 12.5% chunk, and don't have the latency of waiting for the workqueue to get scheduled.

Hmmm, thats true. OTOH I never obsevered this from happening, so I just went for a 'compact' patch. Not sure it makes sense to optimize this.

#3 lore raw
Quoted 28 lines · Florian Westphal
Quoted 18 lines · Mat Martineau
Quoted 16 lines · Florian Westphal

It might be possible that entire receive buffer is occupied by skbs in the OOO queue.

In this case we can't pull more skbs from subflows and the holes will never be filled.

If this happens, schedule the work queue and prune ~12% of skbs to make space available. Also add a MIB counter for this.

Signed-off-by Florian Westphal <fw(a)strlen.de>

---
Paolo, this does relate a bit to our discussion wrt. oow tracking. I thought we might need to add some sort of cushion to account for window discrepancies, but that might then get us in a state where wmem might be full...

Quoted 8 lines
Quoted 5 lines

What do you think?

I did NOT see such a problem in practice, this is a theoretical "fix". TCP has similar code to deal with corner cases of small-oow packets.

Is there a benefit to relying on the workqueue to discard skbs from the ooo queue rather than handling that as data is moved from the subflows?

Quoted 7 lines

It seemed simpler to do it this way. When data_ready callback is running its the first spot where we make checks wrt. buffer occupancy.

move_skbs_to_msk() (which is called right after) may not be able to acquire the msk lock. I did not consider this performance critical to deal with both 'can do it inline' and 'need to defer to worker' cases.

Ok, makes sense.

Quoted 10 lines
Quoted 9 lines

The cause of "holes" in MPTCP reassembly is a little different from TCP, since TCP takes care of the packet loss problem it seems to me the main issue is recovering from path loss (assuming the window and rcv buffer sizes are sane). We need to keep pulling from the subflows to potentially fill the holes, and the most valuable data to make progress would be earlier in the sequence space. We potentially throw less away if discarding later-sequence ooo queued skbs as subflow data arrives rather than doing the whole 12.5% chunk, and don't have the latency of waiting for the workqueue to get scheduled.

Quoted 3 lines

Hmmm, thats true. OTOH I never obsevered this from happening, so I just went for a 'compact' patch. Not sure it makes sense to optimize this.

I do wonder if different packet schedulers might run in to this more, but agree that there isn't evidence to justify optimizing now. If we merge this (or similar) we will have the MIB counter to help understand how much it does happen.

--
Mat Martineau
Intel