SCTP Multihoming Heartbeat ACK Behavior
Hello everyone!
We are having some issues regarding SCTP multihoming and we would like to ask your opinion on this matter. We have two RHEL6.4 (2.6.32-358.el6.x86_64, lksctp-tools -1.0.10-5(64 bit)) machines connected by two L2 switch and a L3 switch (please see "Environment Setup" below). When we execute SCTP connection (using multihoming) between the two machines, the following behavior occurred:
CLIENT L2 and L3 SERVER
Secondary Primary | Primary Secondary
| | | | |
| | | | |
| |INIT-INIT_ACK | | |
| |<-------------|--------->| |
| |COOKIE_ECHO-COOKIE_ACK | |
| | | | |
| |<-------------|--------->| |
| |HB/HB_ACK | | |
| | | | |
|<-------|--------------|----------| |
| | | HB | |
| |--------------|--------->| |
| |HB_ACK | | |
| | : | | |
| | : | | | INIT/INIT_ACK handshake occurred in the primary
path of both machines which is expected. When a
Primary path sends HEARTBEAT to another Primary,HEARTBEAT_ACK was returned to the sender. But
when a Primary path sends HEARTBEAT to a
Secondary path, the HEARTBEAT_ACK chunk was sent
by the Primary path. We expect that theHEARTBEAT_ACK would come from the Secondary.
[Questions]
1. Is this a normal behavior with regards to SCTP multihoming?
2. Is the SCTP kernel module has something to do with this behavior?
3. Is there a solution to force Client to use its Secondary path in sending the HEARTBEAT_ACK chunk to Server's primary?
Environment Setup:
CLIENT SERVER
172.168.39.91 172.168.40.93 +-----------+ +------+ +-----------+ | eth0|----| L2 |----|eth0 | | | +------+ | | | | | | | | | +----------+ | | | | | L3 | | | | | +----------+ | | | | | | | | | +------+ | | | eth1|----| L2 |----|eth1 | +-----------+ +------+ +-----------+ 172.168.39.92 172.168.40.94
Route Setup:
---------- CLIENT ----------
# ip rule show 0: from all lookup local 197: from all to 172.168.39.91 lookup rt2 198: from 172.168.39.91 lookup rt2 199: from all to 172.168.39.92 lookup rt3 200: from 172.168.39.92 lookup rt3 32766: from all lookup main 32767: from all lookup default
# ip route show table rt2 172.168.40.0/24 dev eth0 scope link src 172.168.39.91 172.168.39.0/24 dev eth0 scope link src 172.168.39.91
# ip route show table rt3 172.168.40.0/24 dev eth1 scope link src 172.168.39.92 172.168.39.0/24 dev eth1 scope link src 172.168.39.92
# ip route show table main 192.168.40.0/24 dev eth1 proto kernel scope link src 192.168.40.212
# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.40.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
---------- SERVER ----------
# ip rule show 0: from all lookup local 197: from all to 172.168.40.93 lookup rt2 198: from 172.168.40.93 lookup rt2 199: from all to 172.168.40.94 lookup rt3 200: from 172.168.40.94 lookup rt3 32766: from all lookup main 32767: from all lookup default
# ip route show table rt2 172.168.40.0/24 dev eth0 scope link src 172.168.40.93 172.168.39.0/24 dev eth0 scope link src 172.168.40.93
# ip route show table rt3 172.168.40.0/24 dev eth1 scope link src 172.168.40.94 172.168.39.0/24 dev eth1 scope link src 172.168.40.94
# ip route show table main 192.168.40.0/24 dev eth1 proto kernel scope link src 192.168.40.127
# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.40.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
We are hoping for your kind response and thank you in advance!
Wins
Quoted 37 lines · Winston V. Tizon
Hello everyone!
We are having some issues regarding SCTP multihoming and we would like to ask your opinion on this matter. We have two RHEL6.4 (2.6.32-358.el6.x86_64, lksctp-tools -1.0.10-5(64 bit)) machines connected by two L2 switch and a L3 switch (please see "Environment Setup" below). When we execute SCTP connection (using multihoming) between the two machines, the following behavior occurred:
CLIENT L2 and L3 SERVER
Secondary Primary | Primary Secondary| | | | | | | | | | | |INIT-INIT_ACK | | | | |<-------------|--------->| | | |COOKIE_ECHO-COOKIE_ACK | | | | | | | | |<-------------|--------->| | | |HB/HB_ACK | | | | | | | | |<-------|--------------|----------| | | | | HB | | | |--------------|--------->| | | |HB_ACK | | | | | : | | | | | : | | |INIT/INIT_ACK handshake occurred in the primary path of both machines which is expected. When a Primary path sends HEARTBEAT to another Primary,
HEARTBEAT_ACKwas returned to the sender. But when a Primary path sends HEARTBEAT to a Secondary path, theHEARTBEAT_ACKchunk was sent by the Primary path. We expect that theHEARTBEAT_ACKwould come from the Secondary.
Quoted 2 lines
[Questions]
1. Is this a normal behavior with regards to SCTP multihoming?
So looking at the RFC, it says (RFC4960, 3.3.6. + 6.4.) ...
An endpoint should send this chunk to its peer endpoint as a response to a HEARTBEAT chunk (see Section 8.3). A HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ack is responding.
[...]
An endpoint SHOULD transmit reply chunks (e.g., SACK, HEARTBEAT ACK, etc.) to the same destination transport address from which it received the DATA or control chunk to which it is replying. This rule should also be followed if the endpoint is bundling DATA chunks together with the reply chunk.
... it would be more correct to reply via the same transport, imho. I just checked upstream kernel with multihoming on 2 machines with 5 interfaces each, and HB, HB-ACK replies seem to be fine there, that is, HB-ACKs go via same transports.
On 06/30/2014 10:58 AM, Winston V. Tizon wrote:
...
Quoted 1 line
Thank you very much for your quick reply.
[ Please don't top-post. ]
Quoted 3 lines
I need to confirm if your environment there with 2 machines with 5 interfaces each have the same RHEL and LKSCTP versions with the ones we are using?
Note, I said, I was using latest upstream kernel, didn't check yet with RHEL.
Quoted 10 lines · Winston V. Tizon
TO: Mr. Borkmann
Good day!
Thank you very much for your quick reply.
I need to confirm if your environment there with 2 machines with 5 interfaces each have the same RHEL and LKSCTP versions with the ones we are using?
Quoted 9 lines
If you are using the latest RHEL and LKSCTP versions and HB-ACKS are doing fine there, we might think that our older LKSCTP version has something to do with the abnormal HB-ACK behavior.
Thanks,
W. Tizon
How are you determining which transport is getting used to send the HEARTBEAT ACK? Are you looking at the chunks destination IP address? Looking at sctp_outq_flush in RHEL6 it appears we should always use the inbound transport to send the response, which is the correct thing to do. Sometimes however, while the destination ip address is correct, funny routing tables can lead to a single source address getting selected.
Neil
Quoted 74 lines
Quoted 56 lines · On Mon, 30 Jun 2014 10:24:09 +0200 Daniel Borkmann <[email protected]> wrote:
Quoted 37 lines · Winston V. Tizon
Hello everyone!
We are having some issues regarding SCTP multihoming and we would like to ask your opinion on this matter. We have two RHEL6.4 (2.6.32-358.el6.x86_64, lksctp-tools -1.0.10-5(64 bit)) machines connected by two L2 switch and a L3 switch (please see "Environment Setup" below). When we execute SCTP connection (using multihoming) between the two machines, the following behavior occurred:
CLIENT L2 and L3 SERVER
Secondary Primary | Primary Secondary| | | | | | | | | | | |INIT-INIT_ACK | | | | |<-------------|--------->| | | |COOKIE_ECHO-COOKIE_ACK | | | | | | | | |<-------------|--------->| | | |HB/HB_ACK | | | | | | | | |<-------|--------------|----------| | | | | HB | | | |--------------|--------->| | | |HB_ACK | | | | | : | | | | | : | | |INIT/INIT_ACK handshake occurred in the primary path of both machines which is expected. When a Primary path sends HEARTBEAT to another Primary,
HEARTBEAT_ACKwas returned to the sender. But when a Primary path sends HEARTBEAT to a Secondary path, theHEARTBEAT_ACKchunk was sent by the Primary path. We expect that theHEARTBEAT_ACKwould come from the Secondary.Quoted 2 lines
[Questions]
1. Is this a normal behavior with regards to SCTP multihoming?So looking at the RFC, it says (RFC4960, 3.3.6. + 6.4.) ...
An endpoint should send this chunk to its peer endpoint as a response to a HEARTBEAT chunk (see Section 8.3). A HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ack is responding.
[...]
An endpoint SHOULD transmit reply chunks (e.g., SACK, HEARTBEAT ACK, etc.) to the same destination transport address from which it received the DATA or control chunk to which it is replying. This rule should also be followed if the endpoint is bundling DATA chunks together with the reply chunk.
Quoted 8 lines
... it would be more correct to reply via the same transport, imho. I just checked upstream kernel with multihoming on 2 machines with 5 interfaces each, and HB, HB-ACK replies seem to be fine there, that is, HB-ACKs go via same transports.
--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 62 lines · Winston V. Tizon
Quoted 8 lines · On Mon, 30 Jun 2014 08:30:20 -0400 Neil Horman <[email protected]> wrote:
How are you determining which transport is getting used to send the HEARTBEAT ACK? Are you looking at the chunks destination IP address? Looking at
sctp_outq_flushin RHEL6 it appears we should always use the inbound transport to send the response, which is the correct thing to do. Sometimes however, while the destination ip address is correct, funny routing tables can lead to a single source address getting selected.Neil
Thanks for the reply Neil.
We've just performed test to check if Multihoming Heartbeat ACK Behavior I've mentioned is related to routing table configuration or
SCTP Primary IP address setting. Please see below test information and results."Test case1" (Set the primary in "172.168.39.2")
[client]
# sctp_darn -H 172.168.39.2 -B 172.168.39.3 -P 9099 -h 172.168.39.4 -p 9099 -s[server]
# sctp_darn -H 172.168.39.4 -B 172.168.39.5 -P 9099 -l[result]
172.168.39.2(client) 172.168.39.4(server) INIT 172.168.39.4(server) 172.168.39.2(client)INIT_ACK172.168.39.2(client) 172.168.39.4(server)COOKIE_ECHO172.168.39.4(server) 172.168.39.2(client)COOKIE_ACK172.168.39.4(server) 172.168.39.2(client) HB 172.168.39.2(client) 172.168.39.4(server)HB_ACK172.168.39.4(server) 172.168.39.3(client) HB
172.168.39.2(client) 172.168.39.4(server)HB_ACK(***)
172.168.39.2(client) 172.168.39.4(server) DATA 172.168.39.4(server) 172.168.39.2(client) SACK"Test case2" (Set the primary in "172.168.39.3")
[client]
# sctp_darn -H 172.168.39.3 -B 172.168.39.2 -P 9099 -h 172.168.39.4 -p 9099 -s[server]
# sctp_darn -H 172.168.39.4 -B 172.168.39.5 -P 9099 -l[result]
172.168.39.3(client) 172.168.39.4(server) INIT 172.168.39.4(server) 172.168.39.3(client)INIT_ACK172.168.39.3(client) 172.168.39.4(server)COOKIE_ECHO172.168.39.4(server) 172.168.39.3(client)COOKIE_ACK172.168.39.4(server) 172.168.39.3(client) HB 172.168.39.3(client) 172.168.39.4(server)HB_ACK172.168.39.4(server) 172.168.39.2(client) HB
172.168.39.3(client) 172.168.39.4(server)HB_ACK(***)
172.168.39.3(client) 172.168.39.4(server) DATA 172.168.39.4(server) 172.168.39.3(client) SACK
Quoted 6 lines
Based on results, SCTP kernel always choose Primary IP address so I think this is not related to routing table configuration problems.
(***) -> Based on SCTP RFC4960, expected behavior is secondary IP address
should be used as path in sending theHB_ACK.
Actually, its quite the opposite, this confirms that the sctp protocol is functioning normally. RFC 4960 says this about HB_ACK's:
3.3.6. Heartbeat Acknowledgement (HEARTBEAT ACK) (5)
An endpoint should send this chunk to its peer endpoint as a response to a HEARTBEAT chunk (see Section 8.3). A HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ack is responding.
The only thing that a peer has to do regarding a HB frame is sent an HB_ACK to the source ip address of the corresponding HB frame (in this case172.168.39.4), which we do my recording the inbound transport that the HB frame arrived on. The source address selection is made during L3 packet routing, which, according to sctp_transport_route is made by querying the route tables.
The only thing thats a bit odd is the fact your not taking what is likely the selected source address from the default route. That usually indicates that you're using path mtu features, and you're routing based on a locally selected source and destination address.
The point however is, that a transport is only defined by a destination address within an association, not by a source and a destination, the source address selection is made by the local peer, and should not be relevant to the peer receiving the data.
Neil
Quoted 1 line · Neil Horman
On Tue, Jul 01, 2014 at 03:48:32PM +0800, Winston V. Tizon wrote:
...
Quoted 8 lines
Quoted 3 lines
(***) -> Based on SCTP RFC4960, expected behavior is secondary IP address
should be used as path in sending theHB_ACK.Actually, its quite the opposite, this confirms that the sctp protocol is functioning normally. RFC 4960 says this about HB_ACK's:
3.3.6. Heartbeat Acknowledgement (HEARTBEAT ACK) (5)
Quoted 8 lines
An endpoint should send this chunk to its peer endpoint as a response to a HEARTBEAT chunk (see Section 8.3). A HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ack is responding.
The only thing that a peer has to do regarding a HB frame is sent an
HB_ACKto the source ip address of the corresponding HB frame (in this case172.168.39.4), which we do my recording the inbound transport that the HB frame arrived on.
I agree with you, Neil, the RFC only mentions that we need to "sent to the source IP address", which was what I've quoted earlier on as well, so above statement to use "secondary IP address should be used as path in sending the HB_ACK" is not a MUST.
Quoted 16 lines · Daniel Borkmann
Quoted 1 line · Neil Horman
On Tue, Jul 01, 2014 at 03:48:32PM +0800, Winston V. Tizon wrote:
...
Quoted 13 lines
Quoted 3 lines
(***) -> Based on SCTP RFC4960, expected behavior is secondary IP address
should be used as path in sending theHB_ACK.Actually, its quite the opposite, this confirms that the sctp protocol is functioning normally. RFC 4960 says this about HB_ACK's:
3.3.6. Heartbeat Acknowledgement (HEARTBEAT ACK) (5)
An endpoint should send this chunk to its peer endpoint as a response to a HEARTBEAT chunk (see Section 8.3). A HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ack is responding.
Quoted 8 lines
Quoted 3 lines
The only thing that a peer has to do regarding a HB frame is sent an
HB_ACKto the source ip address of the corresponding HB frame (in this case172.168.39.4), which we do my recording the inbound transport that the HB frame arrived on.I agree with you, Neil, the RFC only mentions that we need to "sent to the source IP address", which was what I've quoted earlier on as well, so above statement to use "secondary IP address should be used as path in sending the
HB_ACK" is not a MUST.
RFC 4960 does not make explicit statements about source address selection in general...
Best regards
Michael
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Also notice that the RFC states this:
" The Sender-Specific Heartbeat Info field should normally include
information about the sender's current time when this HEARTBEAT chunk is sent and the destination transport address to which this HEARTBEAT is sent (see Section 8.3)."
So that is perfectly well expected that the HEARTBEAT ACK source address won't be the destination address of the HEARTBEAT.
Jeff Carter
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Michael Tuexen
Sent: Tuesday, July 01, 2014 7:58 AM
To: Daniel Borkmann
Cc: Neil Horman; Winston V. Tizon; [email protected]
Subject: Re: SCTP Multihoming Heartbeat ACK Behavior
Quoted 16 lines · Daniel Borkmann
Quoted 1 line · Neil Horman
On Tue, Jul 01, 2014 at 03:48:32PM +0800, Winston V. Tizon wrote:
...
Quoted 13 lines
Quoted 3 lines
(***) -> Based on SCTP RFC4960, expected behavior is secondary IP address
should be used as path in sending theHB_ACK.Actually, its quite the opposite, this confirms that the sctp protocol is functioning normally. RFC 4960 says this about HB_ACK's:
3.3.6. Heartbeat Acknowledgement (HEARTBEAT ACK) (5)
An endpoint should send this chunk to its peer endpoint as a response to a HEARTBEAT chunk (see Section 8.3). A HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ack is responding.
Quoted 8 lines
Quoted 3 lines
The only thing that a peer has to do regarding a HB frame is sent an
HB_ACKto the source ip address of the corresponding HB frame (in this case172.168.39.4), which we do my recording the inbound transport that the HB frame arrived on.I agree with you, Neil, the RFC only mentions that we need to "sent to the source IP address", which was what I've quoted earlier on as well, so above statement to use "secondary IP address should be used as path in sending the
HB_ACK" is not a MUST.
RFC 4960 does not make explicit statements about source address selection in general...
Best regards
Michael
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 104 lines · Winston V. Tizon
Quoted 21 lines · On Tue, 1 Jul 2014 07:20:01 -0400 Neil Horman <[email protected]> wrote:
Actually, its quite the opposite, this confirms that the sctp protocol is functioning normally. RFC 4960 says this about HB_ACK's:
3.3.6. Heartbeat Acknowledgement (HEARTBEAT ACK) (5)
An endpoint should send this chunk to its peer endpoint as a response to a HEARTBEAT chunk (see Section 8.3). A HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ack is responding.
The only thing that a peer has to do regarding a HB frame is sent an
HB_ACKto the source ip address of the corresponding HB frame (in this case172.168.39.4), which we do my recording the inbound transport that the HB frame arrived on. The source address selection is made during L3 packet routing, which, according tosctp_transport_routeis made by querying the route tables.The only thing thats a bit odd is the fact your not taking what is likely the selected source address from the default route. That usually indicates that you're using path mtu features, and you're routing based on a locally selected source and destination address.
Quoted 7 lines
The point however is, that a transport is only defined by a destination address within an association, not by a source and a destination, the source address selection is made by the local peer, and should not be relevant to the peer receiving the data.
Neil
TO: Neil, Daniel, Michael
Thank you very much for your answers. I misunderstood the SCTP RFC statement regarding reply Chunks in a multi-homed association. I thought reply chunks should be sent always to the same path from where DATA or Control chunks were received.
I'm sorry if I haven't explained yet the other details of our problem using the following "Environment Setup" below.
CLIENT SERVER
172.168.39.91 172.168.40.93 +-----------+ +------+ +-----------+ | eth0|----| L2 |----|eth0 | | | +------+ | | | | | | | | | +----------+ | | | | | L3 | | | | | +----------+ | | | | | | | | | +------+ | | | eth1|----| L2 |----|eth1 | +-----------+ +------+ +-----------+ 172.168.39.92 172.168.40.94Our problem goes like this. Normal sequence is shown in "[NORMAL]" sequence. While our problem started when we've performed "[ABNORMAL]"sequence below.
[NORMAL]
SERVER L2 and L3 CLIENT
Secondary Primary | Primary Secondary| | | | | | | | | | | |INIT-INIT_ACK | | | | |<-------------|--------->| | | |COOKIE_ECHO-COOKIE_ACK | | | | | | | | |<-------------|--------->| | | |HB/HB_ACK | | | | | | | | |<-------|--------------|----------| | | | | HB | | | |--------------|--------->| | | |HB_ACK | | | | | : | | | | | : | | |[ABNORMAL]
SERVER L2 and L3 CLIENT
Secondary Primary | Primary Secondary| | | | | | | | | | | |INIT-INIT_ACK | | | | |<-------------|--------->| | | |COOKIE_ECHO-COOKIE_ACK | | | | | | | | |<-------------|--------->| | | |HB/HB_ACK | | | | | | | | |<-------|--------------|----------| | | | | HB | | | |---->X | | | | |HB_ACK | | | | | : | | | |<-------|--------------|----------| |ABORT
X -> LAN cable unplug, cannot send
HB_ACKto ClientBased on what happened in our environment, the Client sent ABORT chunk to Secondary IP address of Server causing this path to be closed.
Quoted 8 lines
We don't want this to happen in our environment.
It would be of great help to us if you have any idea on how we could solve question #3 below.
[Question]
3. Is there a solution to force Server to use its Secondary path in sending theHEARTBEAT_ACKchunk to Client's primary?
It is always difficult to have multiple addresses in the same network configured on a host. I guess you problems go away if you use
CLIENT SERVER
172.168.39.91 172.168.39.93 +-----------+ +------+ +-----------+ | eth0|----| L2 |----|eth0 | | | +------+ | | | | | | | | | +----------+ | | | | | L3 | | | | | +----------+ | | | | | | | | | +------+ | | | eth1|----| L2 |----|eth1 | +-----------+ +------+ +-----------+ 172.168.40.92 172.168.40.94
assuming all masks being 255.255.255.0
Best regards
Michael
Quoted 8 lines
Wins
--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 32 lines · Winston V. Tizon
Quoted 21 lines · On Tue, 1 Jul 2014 07:20:01 -0400 Neil Horman <[email protected]> wrote:
Actually, its quite the opposite, this confirms that the sctp protocol is functioning normally. RFC 4960 says this about HB_ACK's:
3.3.6. Heartbeat Acknowledgement (HEARTBEAT ACK) (5)
An endpoint should send this chunk to its peer endpoint as a response to a HEARTBEAT chunk (see Section 8.3). A HEARTBEAT ACK is always sent to the source IP address of the IP datagram containing the HEARTBEAT chunk to which this ack is responding.
The only thing that a peer has to do regarding a HB frame is sent an
HB_ACKto the source ip address of the corresponding HB frame (in this case172.168.39.4), which we do my recording the inbound transport that the HB frame arrived on. The source address selection is made during L3 packet routing, which, according tosctp_transport_routeis made by querying the route tables.The only thing thats a bit odd is the fact your not taking what is likely the selected source address from the default route. That usually indicates that you're using path mtu features, and you're routing based on a locally selected source and destination address.
Quoted 7 lines
The point however is, that a transport is only defined by a destination address within an association, not by a source and a destination, the source address selection is made by the local peer, and should not be relevant to the peer receiving the data.
Neil
TO: Neil, Daniel, Michael
Quoted 8 lines
Thank you very much for your answers. I misunderstood the SCTP RFC statement regarding reply Chunks in a multi-homed association. I thought reply chunks should be sent always to the same path from where DATA or Control chunks were received.
I'm sorry if I haven't explained yet the other details of our problem using the following "Environment Setup" below.
This is a bit of a strange setup. Multiple subnets on the same L2 switch without vlans can lead to strange behaviors. Its also a bit strange that you have a single interface with what must be multiple ip addresses on each of two router interfaces.
Quoted 57 lines
CLIENT SERVER
172.168.39.91 172.168.40.93 +-----------+ +------+ +-----------+ | eth0|----| L2 |----|eth0 | | | +------+ | | | | | | | | | +----------+ | | | | | L3 | | | | | +----------+ | | | | | | | | | +------+ | | | eth1|----| L2 |----|eth1 | +-----------+ +------+ +-----------+ 172.168.39.92 172.168.40.94Our problem goes like this. Normal sequence is shown in "[NORMAL]" sequence. While our problem started when we've performed "[ABNORMAL]"sequence below.
[NORMAL]
SERVER L2 and L3 CLIENT
Secondary Primary | Primary Secondary | | | | | | | | | | | |INIT-INIT_ACK | | | | |<-------------|--------->| | | |COOKIE_ECHO-COOKIE_ACK | | | | | | | | |<-------------|--------->| | | |HB/HB_ACK | | | | | | | | |<-------|--------------|----------| | | | | HB | | | |--------------|--------->| | | |HB_ACK | | | | | : | | | | | : | | |[ABNORMAL]
SERVER L2 and L3 CLIENT
Secondary Primary | Primary Secondary | | | | | | | | | | | |INIT-INIT_ACK | | | | |<-------------|--------->| | | |COOKIE_ECHO-COOKIE_ACK | | | | | | | | |<-------------|--------->| | | |HB/HB_ACK | | | | | | | | |<-------|--------------|----------| | | | | HB | | | |---->X | | | | |HB_ACK | | | | | : | | | |<-------|--------------|----------| |
Quoted 4 lines
ABORT
X -> LAN cable unplug, cannot send
HB_ACKto Client
Exctly what cable was it that got unplugged here? In your configuration above, if the primary ip cable got pulled, then the server system would have updated its route table and the secondary ip address would get used. If the link to the router of the client got pulled though, theres no real hope here. Also, what inspired the ABORT here? I presume the timer ran out on the client and it sent an ABORT because it didn't receive the HBACK in a timely fashion?
Quoted 12 lines
Based on what happened in our environment, the Client sent ABORT chunk to Secondary IP address of Server causing this path to be closed.
We don't want this to happen in our environment.
It would be of great help to us if you have any idea on how we could solve question #3 below.
[Question]
3. Is there a solution to force Server to use its Secondary path in sending theHEARTBEAT_ACKchunk to Client's primary?
Yes, but its the same solution that all SCTP traffic uses, which is to say that it relies on the routing tables. If the primary interface on the sending host goes down, then the route table will expire all routes associated with that path, and sctp will pick a new one (the mechanics are that sctp_transport_dst_check will fail, another routing table lookup will occur, and a route will be cloned that contains the secondary address as the saddr variable). That said, if you're not pulling the interface cable (i.e. if you're pulling the cable to the router, or the client system), then the server has no way to know that the route using the primary source is unreachable, and will continue to use it.
SCTP is fault tolerant over multiple network paths, but that comes with the limitation that SCTP needs to be aware of path failures in a timely fashion. You're setup doesn't really allow for that.
I would suggest one of two things:
1) Convert your setup to have each client and server interface be on a separate subnet. That way the client and server primary addresses are reachable via L2 only, as are the secondary addresses. This will make path failure easier to detect.
2) Use the bonding driver to do failover between your interfacs. That gives you a single path, but the failover detection is handled at L2 rather than L3, which is faster and more reliable for local failures like link drops (and can be made more robust with ip monitoring). If you want L3 failover capabilities, then I sugggest looking into the possibility of vrrp on your router.
Neil