Skip to contentSkip to conversation

Gap not retransmitted after switchover

DISCUSSION Vlad Yasevich 2 participants 7 messages May 11, 2010 – May 13, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw
Quoted 12 lines · Georgios Cheimonidis

Hi Vlad!

I have repeated the test with the net-next kernel tree. It seems that the problem persists. Below, I summarize what I observed from the capture at the server side (the client's capture agrees with these observations). Although the timing differs somewhat from the previous test, the basic observation is still the same. After the server switches primary address and removes the previous primary from the association, some unacknowledged DATA packets that were transmitted to the previous primary (after it became unreachable) are never retransmitted to the new one.

Thanks for testing. I am looking to see what can be happening.

-vlad

Quoted 143 lines

Observations from the capture at the server side:
------------------------------------------------------
- Initially (before the client's ethernet cable is removed), the server sends (and receives acknowledgements for) the segments with TSNs up to #034.
- Suddenly the address (X) that used to be the primary destination becomes unavailable – unreachable.
- Server sends data segments with TSNs #035 to #039 to address X (which are never received by the client because this address is no longer usable – reachable).
- Server retransmits data segments with TSNs #035, #036, #038 to address Y.
- Server receives SACK from client (Cumulative: #036, Selective: -)
- Server retransmits data segment #037
- Server receives SACK from client (Cumulative: #036, Selective: From #38 to #38)
- Server sends #040 to address X (which is never received by client because X is unreachable)
- Server retransmits #039 to address Y
- Server sends #041 to address X (which is never received by client...)
- Server retransmits #039 to address Y
- Server receives SACK from client (Cumulative: #039 Selective: -) ----------- at this point of time there is no gap, but #040 to #041 have not been acknowledged -----------
- Server receives an ASCONF from the client to set its primary address to Y.
- Server sends an ASCONF_ACK and bundles DATA #042 to address Y.
- Server sends #043 to #047 to address Y.
- Server receives an ASCONF from the client, that says to remove address X from the association.
- Server sends an ASCONF_ACK.
- Server receives SACKs from the client that acknowledge the received packets and also indicate that there is a gap (from #040 to #041). The server continues to send new packets but does not retransmit the gap. The exchange of DATA and SACKs between the server and the client continues until server sends #084 and client acknowledges it. The SACKs sent from the client always have a cumulative TSN of #039 and indicate that there is a gap (from #040 to #041). After that, the client and the server exchange only HEARTBEATs. No data transmission take place. The client application is blocked in recv() and the server application is blocked in send(). The last receive window reported by the client is 3136 bytes. The application messages are 4928 bytes.

I also attach the kernel log of the server host. The relation between the addresses mentioned above and the addresses shown in the kernel log is:
X: 213.XXX.XXX.XXX (client's eth)
Y: 192.YYY.YYY.YYY (client's wlan)
Z: 213.ZZZ.ZZZ.ZZZ (server's)

Hope this helps!
/George

Quoted 91 lines · Vlad Yasevich

Hi George

Can you try this against net-next-2.6 tree. There were a few patches that went in recently that might fix what you are seeing.

Also, lksctp-developers list will drop attachments. You are better of using the [email protected].

-vlad

Quoted 79 lines · Georgios Cheimonidis

Hi!

I have observed a problem while doing some tests with dynamic address reconfiguration. Let me first describe my setup and application.

Setup: I have two hosts, one that acts as a client and another that acts as a server. The client has two IPv4 addresses (one on eth, let's call it X, and another on wlan, let's call it Y). The server is single homed. Both hosts are running 2.6.34-rc5 kernel, downloaded from David Miller's net-2.6 source tree on 05-May-2010). I have enabled SCTP debugging messages.

Application: In my simple application, only the server transmits messages to the client, of 4928 bytes payload each (probably this doesn't matter). The server uses blocking send() and the client uses blocking recv(). My client application has a simple policy: When the ethernet cable is removed, a monitoring process reports this event to my application. This monitoring process at the same time removes the ethernet's IP address and relevant routes in the routing table. When my application receives this event notification, it takes two consecutive actions. First it calls setsockopt(SET_PEER_PRIMARY_ADDR) to change the peer's (server's) primary destination to address Y. Immediately after that, it calls sctp_bindx() to remove IP address X from the association. So, when I remove the ethernet cable from the client, the server should change its primary (destination) address to Y and then remove address X from its list of destination addresses.

In the following experiment, I start the association with the client having both IP addresses (address X is used for the initial handshake) and after some seconds I remove the ethernet cable. In short, this is what happens (from the server's point of view according to the capture from wireshark; the capture on the client agrees with these observations):

- Initially (before the client's ethernet cable is removed), the server sends (and receives acknowledgements) for the segments with TSNs up to #717.
- Suddenly the client's address (X) that used to be the server's primary destination becomes unavailable
- Server sends data segments with TSNs #718 to #722 to X (which are never received by the client because this address is no longer usable – reachable).
- Server receives an ASCONF from the client and acknowledges it, to set its primary address to Y.
- Server sends data segments with TSNs #723 to #727 to Y.
- Server receives an ASCONF from the client and acknowledges it, to delete address X.
- Server sends data segments with TSNs #728 to #762 to Y. The server receives SACKs from the client that indicate that there is a gap. Actually, the client always includes a cumulative acknowledgment TSN of #717 and also acknowledges all the TSNs after the gap. However, the server does NOT retransmit the gap (TSNs from #718 to #722).
- After that, the server doesn't send any new TSNs (the server application blocks in send()), and the last reported receive window by the client is 7040 bytes. The server and client continue exchanging HEARTBEATs, but no data transfer takes place. The client also blocks in recv() because it cannot deliver the data to the upper layer due to the missing TSNs.

I don't know why the server doesn't retransmit the gap even though the client reports it (in 40 consecutive SACKs). I am also attaching the kernel log of the server host. Any help is highly appreciated!

Best regards,
George

------------------------------------------------------------------------

------------------------------------------------------------------------------

------------------------------------------------------------------------

_______________________________________________
Lksctp-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers

#1 lore raw
Quoted 15 lines · Vlad Yasevich
Quoted 12 lines · Georgios Cheimonidis

Hi Vlad!

I have repeated the test with the net-next kernel tree. It seems that the problem persists. Below, I summarize what I observed from the capture at the server side (the client's capture agrees with these observations). Although the timing differs somewhat from the previous test, the basic observation is still the same. After the server switches primary address and removes the previous primary from the association, some unacknowledged DATA packets that were transmitted to the previous primary (after it became unreachable) are never retransmitted to the new one.

Quoted 4 lines

Thanks for testing. I am looking to see what can be happening.

-vlad

Hi George.

I figured out why there were no retransmits. Because you changed primary path, you kicked in the SFR-CACC algorithm, and our implementation didn't deal properly with the fact that some chunks may have moved from the old primary to the new one without going though a retransmit.

There are really 2 ways to deal with this:
1). If we are deleting a transport that had outstanding data, automatically retransmit the data on the new transport.

or.

2) Under the same condition as above, move the data to the new primary destination and let fast-recovery take care of the issue.

Linux implemented (2) from above, and thus this bug surfaced.

Try the attached patch, and let me know if it fixes it for you.

-vlad

From 7634892e75811970f501aebf88c7c97a86e77066 Mon Sep 17 00:00:00 2001
From: Vlad Yasevich <[email protected]>
Date: Tue, 11 May 2010 11:16:29 -0400
Subject: [PATCH] sctp: teach CACC algorithm about removed transports

When we have have to remove a transport due to ASCONF, we move the data to a new active path. This can trigger CACC algorithm to not mark that data as missing when SACKs arrive. This is because the transport passed to the CACC algorithm is the one this data is sitting on, not the one it was sent on (that one may be gone). So, by sending the original transport (even if it's NULL), we may start marking data as missing.

---
 net/sctp/outqueue.c |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)
net/sctp/outqueue.c +8 -3
index 5d05717..dd55f63 100644
@@ -131,7 +131,8 @@ static inline int sctp_cacc_skip_3_1_d(struct sctp_transport *primary,
 static inline int sctp_cacc_skip_3_1_f(struct sctp_transport *transport,
 				       int count_of_newacks)
 {
-	if (count_of_newacks < 2 && !transport->cacc.cacc_saw_newack)
+	if (count_of_newacks < 2 &&
+			(transport && !transport->cacc.cacc_saw_newack))
 		return 1;
 	return 0;
 }
@@ -620,9 +621,12 @@ redo:
 
 			/* If we are retransmitting, we should only
 			 * send a single packet.
+			 * Otherwise, try appending this chunk again.
 			 */
 			if (rtx_timeout || fast_rtx)
 				done = 1;
+			else
+				goto redo;
 
 			/* Bundle next chunk in the next round.  */
 			break;
@@ -1685,8 +1689,9 @@ static void sctp_mark_missing(struct sctp_outq *q,
 			/* SFR-CACC may require us to skip marking
 			 * this chunk as missing.
 			 */
-			if (!transport || !sctp_cacc_skip(primary, transport,
-					    count_of_newacks, tsn)) {
+			if (!transport || !sctp_cacc_skip(primary,
+						chunk->transport,
+					    	count_of_newacks, tsn)) {
 				chunk->tsn_missing_report++;
 
 				SCTP_DEBUG_PRINTK(
Signature · 1 line
1.6.0.4
#2 lore raw
In reply to Vlad Yasevich

Hi Vlad!

I will compile it now and test it tomorrow morning. I will let you know as soon as I have the results.

Thanks!
/George

Quoted 44 lines · Vlad Yasevich
Quoted 15 lines · Vlad Yasevich
Quoted 12 lines · Georgios Cheimonidis

Hi Vlad!

I have repeated the test with the net-next kernel tree. It seems that the problem persists. Below, I summarize what I observed from the capture at the server side (the client's capture agrees with these observations). Although the timing differs somewhat from the previous test, the basic observation is still the same. After the server switches primary address and removes the previous primary from the association, some unacknowledged DATA packets that were transmitted to the previous primary (after it became unreachable) are never retransmitted to the new one.

Quoted 4 lines

Thanks for testing. I am looking to see what can be happening.

-vlad

Hi George.

I figured out why there were no retransmits. Because you changed primary path, you kicked in the SFR-CACC algorithm, and our implementation didn't deal properly with the fact that some chunks may have moved from the old primary to the new one without going though a retransmit.

There are really 2 ways to deal with this:
1). If we are deleting a transport that had outstanding data, automatically retransmit the data on the new transport.

or.

2) Under the same condition as above, move the data to the new primary destination and let fast-recovery take care of the issue.

Linux implemented (2) from above, and thus this bug surfaced.

Try the attached patch, and let me know if it fixes it for you.

-vlad

#3 lore raw
In reply to Vlad Yasevich

Hi Vlad!

I made quite a lot of tests today. Here are my results.

When I repeated my previous test (IPv4 addresses only) I did not experience any problems. So, it seems that the patch worked! The server, after receiving three consecutive SACKs with the reported gap (three miss indications), it retransmitted the missing TSNs and the data flow continued normally. I repeated it many times and the result was always the same.

However, I experienced the same problem (not always but some times) when I had the following setup.
- Server having both IPv4 and IPv6 addresses on ethernet interface.
- Client having IPv6 on ethernet (X) and IPv4 on wlan (Y).
- Association established with all the above addresses belonging to the association. The client uses its IPv6 address to contact the IPv6 address of the server (initially), so the initial handshake is done using the IPv6 addresses. The client sends an ASCONF just after association establishment to tell the server to set its primary to the X.
- Whenever the ethernet cable is removed at the client, the client calls setsockopt(SET_PEER_PRIMARY_ADDR) to tell the server to set Y as its primary and then calles sctp_bindx() to remove X from the association. In this scenario, sometimes the server does not retransmit the gap (after changing primary from X to Y and deleting Y from association).

Another observation that I have made, is that sometimes, after the ethernet cable is removed and I call setsockopt(SET_PEER_PRIMARY_ADDR) on the client to set the peer's primary to Y, the actual transmission of the ASCONF chunk is observed after many seconds (sometimes I observed the transmission 30 seconds after the call to setsockopt). I don't know if this is normal. Even with IPv4 only test I observed a small delay between calling setsockopt() and observing the ASCONF chunk, but it was about 1-2 seconds. With the IPv4/IPv6 test, this delay varied more.

Looking forward to your comments! Let me know if you want me to test something more.

Best regards,
George

Quoted 44 lines · Vlad Yasevich
Quoted 15 lines · Vlad Yasevich
Quoted 12 lines · Georgios Cheimonidis

Hi Vlad!

I have repeated the test with the net-next kernel tree. It seems that the problem persists. Below, I summarize what I observed from the capture at the server side (the client's capture agrees with these observations). Although the timing differs somewhat from the previous test, the basic observation is still the same. After the server switches primary address and removes the previous primary from the association, some unacknowledged DATA packets that were transmitted to the previous primary (after it became unreachable) are never retransmitted to the new one.

Quoted 4 lines

Thanks for testing. I am looking to see what can be happening.

-vlad

Hi George.

I figured out why there were no retransmits. Because you changed primary path, you kicked in the SFR-CACC algorithm, and our implementation didn't deal properly with the fact that some chunks may have moved from the old primary to the new one without going though a retransmit.

There are really 2 ways to deal with this:
1). If we are deleting a transport that had outstanding data, automatically retransmit the data on the new transport.

or.

2) Under the same condition as above, move the data to the new primary destination and let fast-recovery take care of the issue.

Linux implemented (2) from above, and thus this bug surfaced.

Try the attached patch, and let me know if it fixes it for you.

-vlad

#4 lore raw
In reply to Vlad Yasevich
Quoted 26 lines · Georgios Cheimonidis

Hi Vlad!

I made quite a lot of tests today. Here are my results.

When I repeated my previous test (IPv4 addresses only) I did not experience any problems. So, it seems that the patch worked! The server, after receiving three consecutive SACKs with the reported gap (three miss indications), it retransmitted the missing TSNs and the data flow continued normally. I repeated it many times and the result was always the same.

However, I experienced the same problem (not always but some times) when I had the following setup.
- Server having both IPv4 and IPv6 addresses on ethernet interface.
- Client having IPv6 on ethernet (X) and IPv4 on wlan (Y).
- Association established with all the above addresses belonging to the association. The client uses its IPv6 address to contact the IPv6 address of the server (initially), so the initial handshake is done using the IPv6 addresses. The client sends an ASCONF just after association establishment to tell the server to set its primary to the X.
- Whenever the ethernet cable is removed at the client, the client calls setsockopt(SET_PEER_PRIMARY_ADDR) to tell the server to set Y as its primary and then calles sctp_bindx() to remove X from the association. In this scenario, sometimes the server does not retransmit the gap (after changing primary from X to Y and deleting Y from association).

Quoted 9 lines

Another observation that I have made, is that sometimes, after the ethernet cable is removed and I call setsockopt(SET_PEER_PRIMARY_ADDR) on the client to set the peer's primary to Y, the actual transmission of the ASCONF chunk is observed after many seconds (sometimes I observed the transmission 30 seconds after the call to setsockopt). I don't know if this is normal. Even with IPv4 only test I observed a small delay between calling setsockopt() and observing the ASCONF chunk, but it was about 1-2 seconds. With the IPv4/IPv6 test, this delay varied more.

Interesting. Looks like what happens is that we continue to try and use the current primary destination, which uses the interface that lost the link. So, that most likely triggers retransmissions. Depending on the rto.max, you might see a delay...

The DEL_IP ends up being delayed untill the first one succeeds.

What happens if you reverse your two calls? Call bindx() first to remove the address, and then call SET_PEER_PRIMARY. BTW, with only 2 paths, you don't really need to change the primary since there will only be 1 path and it will automatically become primary.

Additionally, IPv6 routing is not always correct right now. Thus, you may end up with IPv6 route even though it should not be used any more. The switch in the call order above might help with that. I am working on fixing the v6 routing right now.

-vlad

Quoted 57 lines

Looking forward to your comments! Let me know if you want me to test something more.

Best regards,
George

Quoted 47 lines · Vlad Yasevich
Quoted 17 lines · Vlad Yasevich
Quoted 14 lines · Georgios Cheimonidis

Hi Vlad!

I have repeated the test with the net-next kernel tree. It seems that the problem persists. Below, I summarize what I observed from the capture at the server side (the client's capture agrees with these observations). Although the timing differs somewhat from the previous test, the basic observation is still the same. After the server switches
primary address and removes the previous primary from the association, some unacknowledged DATA packets that were transmitted to the previous primary (after it became unreachable) are never retransmitted to the new
one.

Quoted 4 lines

Thanks for testing. I am looking to see what can be happening.

-vlad

Hi George.

I figured out why there were no retransmits. Because you changed primary path, you kicked in the SFR-CACC algorithm, and our implementation didn't deal properly with the fact that some chunks may have moved from the old primary to the new one without going though a retransmit.

There are really 2 ways to deal with this:
1). If we are deleting a transport that had outstanding data, automatically retransmit the data on the new transport.

or.

2) Under the same condition as above, move the data to the new
primary
destination and let fast-recovery take care of the issue.

Linux implemented (2) from above, and thus this bug surfaced.

Try the attached patch, and let me know if it fixes it for you.

-vlad

#5 lore raw
In reply to Vlad Yasevich

Hi Vlad!

I made some tests after reversing the order of the two calls. So, on the client. whenever the ethernet cable is removed, I first call sctp_bindx() to remove the IPv6 address and then setsockopt() to set the peer's primary to the IPv4 of client's wlan. (Note: The reason I am also trying to set the peer's primary is because I will generally have more than 2 IP addresses on the client and I want to be able to affect the incoming interface and not just let the peer pick whichever it wants).

So, even though I reversed the calls, sometimes I observe a large delay between the actual call to sctp_bindx(DEL_ADDR) and the transmission of the ASCONF chunk on the wlan interface. Once I observed it 12 seconds after the call, and another time I observed it 16 seconds after the call. Many times it was 2-3 seconds after the call.

In addition, sometimes the second ASCONF (for setting peer's primary) is transmitted some seconds after the first ASCONF_ACK received for the first ASCONF. Sometimes it was transmitted 2 seconds after, some times 6 seconds after, 8 seconds after and once I observed it 30 seconds after! I understand that the second ASCONF gets delayed until the first one succeeds, but why does it have to wait more to get transmitted? Could it be that the host also tries to send the second ASCONF using the unusable interface (eth) and then retransmits it to the usable one (wlan)?

Best regards,
George

Quoted 115 lines · Vlad Yasevich
Quoted 26 lines · Georgios Cheimonidis

Hi Vlad!

I made quite a lot of tests today. Here are my results.

When I repeated my previous test (IPv4 addresses only) I did not experience any problems. So, it seems that the patch worked! The server, after receiving three consecutive SACKs with the reported gap (three miss indications), it retransmitted the missing TSNs and the data flow continued normally. I repeated it many times and the result was always the same.

However, I experienced the same problem (not always but some times) when I had the following setup.
- Server having both IPv4 and IPv6 addresses on ethernet interface.
- Client having IPv6 on ethernet (X) and IPv4 on wlan (Y).
- Association established with all the above addresses belonging to the association. The client uses its IPv6 address to contact the IPv6 address of the server (initially), so the initial handshake is done using the IPv6 addresses. The client sends an ASCONF just after association establishment to tell the server to set its primary to the X.
- Whenever the ethernet cable is removed at the client, the client calls setsockopt(SET_PEER_PRIMARY_ADDR) to tell the server to set Y as its primary and then calles sctp_bindx() to remove X from the association. In this scenario, sometimes the server does not retransmit the gap (after changing primary from X to Y and deleting Y from association).

Quoted 9 lines

Another observation that I have made, is that sometimes, after the ethernet cable is removed and I call setsockopt(SET_PEER_PRIMARY_ADDR) on the client to set the peer's primary to Y, the actual transmission of the ASCONF chunk is observed after many seconds (sometimes I observed the transmission 30 seconds after the call to setsockopt). I don't know if this is normal. Even with IPv4 only test I observed a small delay between calling setsockopt() and observing the ASCONF chunk, but it was about 1-2 seconds. With the IPv4/IPv6 test, this delay varied more.

Interesting. Looks like what happens is that we continue to try and use the current primary destination, which uses the interface that lost the link. So, that most likely triggers retransmissions. Depending on the rto.max, you might see a delay...

The DEL_IP ends up being delayed untill the first one succeeds.

What happens if you reverse your two calls? Call bindx() first to remove the address, and then call SET_PEER_PRIMARY. BTW, with only 2 paths, you don't really need to change the primary since there will only be 1 path and it will automatically become primary.

Additionally, IPv6 routing is not always correct right now. Thus, you may end up with IPv6 route even though it should not be used any more. The switch in the call order above might help with that. I am working on fixing the v6 routing right now.

-vlad

Quoted 57 lines

Looking forward to your comments! Let me know if you want me to test something more.

Best regards,
George

Quoted 47 lines · Vlad Yasevich
Quoted 17 lines · Vlad Yasevich
Quoted 14 lines · Georgios Cheimonidis

Hi Vlad!

I have repeated the test with the net-next kernel tree. It seems that the problem persists. Below, I summarize what I observed from the capture at the server side (the client's capture agrees with these observations). Although the timing differs somewhat from the previous test, the basic observation is still the same. After the server switches
primary address and removes the previous primary from the association, some unacknowledged DATA packets that were transmitted to the previous primary (after it became unreachable) are never retransmitted to the new
one.

Quoted 4 lines

Thanks for testing. I am looking to see what can be happening.

-vlad

Hi George.

I figured out why there were no retransmits. Because you changed primary path, you kicked in the SFR-CACC algorithm, and our implementation didn't deal properly with the fact that some chunks may have moved from the old primary to the new one without going though a retransmit.

There are really 2 ways to deal with this:
1). If we are deleting a transport that had outstanding data, automatically retransmit the data on the new transport.

or.

2) Under the same condition as above, move the data to the new
primary
destination and let fast-recovery take care of the issue.

Linux implemented (2) from above, and thus this bug surfaced.

Try the attached patch, and let me know if it fixes it for you.

-vlad

#6 lore raw
In reply to Vlad Yasevich
Quoted 16 lines · Georgios Cheimonidis

Hi Vlad!

I made some tests after reversing the order of the two calls. So, on the client. whenever the ethernet cable is removed, I first call sctp_bindx() to remove the IPv6 address and then setsockopt() to set the peer's primary to the IPv4 of client's wlan. (Note: The reason I am also trying to set the peer's primary is because I will generally have more than 2 IP addresses on the client and I want to be able to affect the incoming interface and not just let the peer pick whichever it wants).

So, even though I reversed the calls, sometimes I observe a large delay between the actual call to sctp_bindx(DEL_ADDR) and the transmission of the ASCONF chunk on the wlan interface. Once I observed it 12 seconds after the call, and another time I observed it 16 seconds after the call. Many times it was 2-3 seconds after the call.

Quoted 9 lines

In addition, sometimes the second ASCONF (for setting peer's primary) is transmitted some seconds after the first ASCONF_ACK received for the first ASCONF. Sometimes it was transmitted 2 seconds after, some times 6 seconds after, 8 seconds after and once I observed it 30 seconds after! I understand that the second ASCONF gets delayed until the first one succeeds, but why does it have to wait more to get transmitted? Could it be that the host also tries to send the second ASCONF using the unusable interface (eth) and then retransmits it to the usable one (wlan)?

It's possible. Like I said, the IPv6 routing is rather broken in sctp. Not sure it has ever been tested with address removal. Let me see if can work up a patch for you try.

-vlad

Quoted 136 lines

Best regards,
George

Quoted 128 lines · Vlad Yasevich
Quoted 27 lines · Georgios Cheimonidis

Hi Vlad!

I made quite a lot of tests today. Here are my results.

When I repeated my previous test (IPv4 addresses only) I did not experience any problems. So, it seems that the patch worked! The server, after receiving three consecutive SACKs with the reported gap (three miss indications), it retransmitted the missing TSNs and the data flow continued normally. I repeated it many times and the result was always the same.

However, I experienced the same problem (not always but some times) when I had the following setup.
- Server having both IPv4 and IPv6 addresses on ethernet interface.
- Client having IPv6 on ethernet (X) and IPv4 on wlan (Y).
- Association established with all the above addresses belonging to the association. The client uses its IPv6 address to contact the IPv6 address of the server (initially), so the initial handshake is done using the IPv6 addresses. The client sends an ASCONF just after association establishment to tell the server to set its primary to the X.
- Whenever the ethernet cable is removed at the client, the client calls setsockopt(SET_PEER_PRIMARY_ADDR) to tell the server to set Y as its primary and then calles sctp_bindx() to remove X from the association. In this scenario, sometimes the server does not retransmit the gap (after changing primary from X to Y and deleting Y from association).

Quoted 9 lines

Another observation that I have made, is that sometimes, after the ethernet cable is removed and I call setsockopt(SET_PEER_PRIMARY_ADDR) on the client to set the peer's primary to Y, the actual transmission of the ASCONF chunk is observed after many seconds (sometimes I observed the transmission 30 seconds after the call to setsockopt). I don't know if this is normal. Even with IPv4 only test I observed a small delay between calling setsockopt() and observing the ASCONF chunk, but it was about 1-2 seconds. With the IPv4/IPv6 test, this delay varied more.

Interesting. Looks like what happens is that we continue to try and use the current primary destination, which uses the interface that lost the link.
So, that most likely triggers retransmissions. Depending on the rto.max, you might see a delay...

The DEL_IP ends up being delayed untill the first one succeeds.

What happens if you reverse your two calls? Call bindx() first to remove the
address, and then call SET_PEER_PRIMARY. BTW, with only 2 paths, you don't
really need to change the primary since there will only be 1 path and it will
automatically become primary.

Additionally, IPv6 routing is not always correct right now. Thus, you may
end up with IPv6 route even though it should not be used any more. The switch
in the call order above might help with that. I am working on fixing the v6
routing right now.

-vlad

Quoted 62 lines

Looking forward to your comments! Let me know if you want me to test something more.

Best regards,
George

Quoted 52 lines · Vlad Yasevich
Quoted 19 lines · Vlad Yasevich
Quoted 16 lines · Georgios Cheimonidis

Hi Vlad!

I have repeated the test with the net-next kernel tree. It seems that the problem persists. Below, I summarize what I observed from the capture at the server side (the client's capture agrees with these observations). Although the timing differs somewhat from the previous test, the basic observation is still the same. After the server switches
primary address and removes the previous primary from the association,
some unacknowledged DATA packets that were transmitted to the previous
primary (after it became unreachable) are never retransmitted to the new
one.

Quoted 4 lines

Thanks for testing. I am looking to see what can be happening.

-vlad

Hi George.

I figured out why there were no retransmits. Because you changed primary
path, you kicked in the SFR-CACC algorithm, and our implementation didn't
deal properly with the fact that some chunks may have moved from the old
primary to the new one without going though a retransmit.

There are really 2 ways to deal with this:
1). If we are deleting a transport that had outstanding data, automatically retransmit the data on the new transport.

or.

2) Under the same condition as above, move the data to the new
primary
destination and let fast-recovery take care of the issue.

Linux implemented (2) from above, and thus this bug surfaced.

Try the attached patch, and let me know if it fixes it for you.

-vlad