route/dst_entry caching issue

Lksctp-developers Matija Glavinic Pecotic 2 participants 4 messages Sep 5, 2012 – Nov 14, 2012

Subsystems (2)

Networking SCTP protocol

lore

Share & tools

lore

Matija Glavinic Pecotic

author

2012-09-05 11:27 UTC #0 lore raw

Hello lksctp developers,

a while back we noticed below described issue.

In order to quick-fix this, we implemented following workaround:

1) added interface to sctp for invalidating all dst entries on all associations (0002-sctp-export-dst-entries-invalidation-if.patch) - this is on top of below mentioned patch (0001-sctp-check-cached-dst-before-using-it.patch)

2) in a separate module route and ipsec events are monitored and upon addition and deletion of routes, along with xfrm events interface is called. This is done by netlink socket opened within the kernel. If interested, I can share that one as well.

Could you please give some comments on both workaround and mentioned issue?

Thanks and regards,

Matija

Quoted 129 lines · ext Matija Glavinic Pecotic

Hello,
We're facing an issue regarding route change and startup of ipsec tunnel. Here is setup and test scenarios:
1) Setup
Simple sctp client and server. Client is bound to the loopback address 10.0.0.1 and it is connected to the server via eth3 (192.168.2.1) which is in LAN with eth2 (192.168.2.2) and eth3 (192.168.2.3) on the server and server is bound to the 20.0.0.1 which is loopback.
Client:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 10.0.0.1/8 scope global lo
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
13: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
      link/ether 00:40:43:30:81:80 brd ff:ff:ff:ff:ff:ff
      inet 192.168.2.1/24 brd 192.168.2.255 scope global eth3
      inet6 fe80::240:43ff:fe30:8180/64 scope link
         valid_lft forever preferred_lft forever
192.168.2.0/24 dev eth3 proto kernel scope link src 192.168.2.1
192.168.253.0/24 dev rio0  proto kernel  scope link  src 192.168.253.16
192.168.254.0/23 dev eth0  proto kernel  scope link  src 192.168.255.1
20.0.0.0/8 via 192.168.2.1 dev eth3 scope link
Server:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 20.0.0.1/8 scope global lo
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
12: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
      link/ether 00:40:43:b1:f8:ae brd ff:ff:ff:ff:ff:ff
      inet 192.168.2.2/24 brd 192.168.2.255 scope global eth2
      inet6 fe80::240:43ff:feb1:f8ae/64 scope link
         valid_lft forever preferred_lft forever
13: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
      link/ether 00:40:43:b1:f8:af brd ff:ff:ff:ff:ff:ff
      inet 192.168.2.3/24 brd 192.168.2.255 scope global eth3
      inet6 fe80::240:43ff:feb1:f8af/64 scope link
         valid_lft forever preferred_lft forever
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.2
192.168.2.0/24 dev eth3  proto kernel  scope link  src 192.168.2.3
192.168.253.0/24 dev rio0  proto kernel  scope link  src 192.168.253.16
192.168.255.0/24 dev eth0  proto kernel  scope link  src 192.168.255.2
10.0.0.0/8 via 192.168.2.1 dev eth2
10.0.0.0/8 via 192.168.2.1 dev eth3 metric 10
Server listens on 20.0.0.1 and when client is connected it sends message indefinitely
Kernel is 2.6.32.58 and sctp is applied with patch from this thread:
http://sourceforge.net/mailarchive/message.php?msg_id=25786006
(0001-sctp-check-cached-dst-before-using-it.patch)
2) Test scenario 1
Start sctp_server
Start sctp_client -> clients receives messages from the server Observe traffic flow -> as expected it goes over eth2 on the server Delete route 10.0.0.0/8 via 192.168.2.1 dev eth2
Traffic still goes over eth2, on server restart it goes over eth3
For reference, I've run also TCP server/client which does the same thing, after deletion of mentioned route traffic continued over eth3. Other kinds of traffic are also rerouted on route deletion (icmp)
3) Test scenario 2
Setup same as before, eth3 on server isn't used.
Start sctp_server
Start sctp_client -> traffic ok
Start ipsec tunnel:
      left=192.168.2.2
      leftsubnet=20.0.0.0/24
      right=192.168.2.1
      rightsubnet=10.0.0.0/24
After start of ipsec tunnel, traffic went over same dst unencrypted for a short period of time.
For reference, again same scenario with TCP server/client -> traffic started to be encrypted immediately
------
In my opinion there is problem with cached dst_entry. I've tried following:
[sctp.h]
/* The cookie is always 0 since this is how it's used in the
* pmtu code.
*/
static inline struct dst_entry *sctp_transport_dst_check(struct sctp_transport *t)
{
	if (t->dst) {
		dst_release(t->dst);
		t->dst = NULL;
	}

	return t->dst;
}
With this I just wanted to force sctp to route every time it wants to send packet, and after this change above mentioned scenarios passed. Here is my short analysis, ignore it if I got something wrong: I see potential problem with SCTP's dst_entry: struct dst_entry *dst which is member of sctp_transport. For example, ipv4&6 implement operations on dst_entry (ipv4_dst_ops ) which are to be done when event occurs, such as unreachability, route deleteion, etc. SCTP doesn't, and it also doesn't use dst_entries from ipv*. In this way, only events which impact SCTPs dst_entries are default ones, NETDEV_DOWN and NETDEV_UNREGISTER and these are only which will be noted with dst_check().
Could you comment please? I'll be glad to make tests/verifications or provide any sort of help on this.
Thanks & regards,
Matija

Quoted 11 lines

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Lksctp-developers mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers

include/net/sctp/sctp.h +6 -0

diff -purN linux.unpatched/include/net/sctp/sctp.h linux/include/net/sctp/sctp.h
@@ -704,4 +704,10 @@ static inline struct dst_entry *sctp_tra
 	return t->dst;
 }
 
+/** 
+ * invalidate all dst_cache entries, goes through all associations and 
+ * invalidates associated dst_entries. 
+ */
+void __sctp_invalidate_all_assocs_dst(void);
+
 #endif /* __net_sctp_h__ */

net/sctp/protocol.c +39 -1

diff -purN linux.unpatched/net/sctp/protocol.c linux/net/sctp/protocol.c
@@ -1058,6 +1058,42 @@ static void sctp_v4_del_protocol(void)
 	unregister_inetaddr_notifier(&sctp_inetaddr_notifier);
 }
 
+void __sctp_invalidate_all_assocs_dst(void)
+{
+	struct sctp_hashbucket *head;
+	struct sctp_ep_common *epb;
+	struct sctp_association *assoc;
+	struct hlist_node *node;
+	int    hash = 0;
+	struct sctp_transport *transport;
+	union sctp_addr *primary;
+	
+	printk(KERN_INFO "[%s:%d]: Invalidating all dst_entries\n", __FUNCTION__, __LINE__);
+
+	for(hash = 0; hash < sctp_assoc_hashsize; hash++)
+	{	
+		head = &sctp_assoc_hashtable[hash];
+		sctp_local_bh_disable();
+		write_lock(&head->lock);
+	
+		sctp_for_each_hentry(epb, node, &head->chain) {
+			assoc = sctp_assoc(epb);
+			
+			primary = &assoc->peer.primary_addr;
+	
+			list_for_each_entry(transport, &assoc->peer.transport_addr_list, transports) 
+			{
+				printk(KERN_INFO "[%s:%d]: Invalidating: transport@0x%p, dst@0x%p, dst->obsolete=%d\n", __FUNCTION__, __LINE__, transport, (*transport).dst, (*transport).dst->obsolete);
+				(*transport).dst->obsolete = 2;				
+			}
+			
+		}
+		write_unlock(&head->lock);
+		sctp_local_bh_enable();
+	}
+}
+EXPORT_SYMBOL(__sctp_invalidate_all_assocs_dst);
+
 /* Initialize the universe into something sensible.  */
 SCTP_STATIC __init int sctp_init(void)
 {

@@ -1069,6 +1105,8 @@ SCTP_STATIC __init int sctp_init(void)
 	int max_share;
 	int order;
 
+	printk(KERN_WARNING "[%s:%d]: Initializing SCTP stack patched for routes and ipsec operations\n", __FUNCTION__, __LINE__);
+	
 	/* SCTP_DEBUG sanity check. */
 	if (!sctp_sanity_check())
 		goto out;

@@ -1297,7 +1335,7 @@ SCTP_STATIC __init int sctp_init(void)
 	status = sctp_v6_add_protocol();
 	if (status)
 		goto err_v6_add_protocol;
-
+		
 	status = 0;
 out:
 	return status;

Vlad Yasevich

2012-09-05 14:55 UTC #1 lore raw

Quoted 11 lines · Matija Glavinic Pecotic

Hello lksctp developers,
a while back we noticed below described issue.
In order to quick-fix this, we implemented following workaround:
1) added interface to sctp for invalidating all dst entries on all associations (0002-sctp-export-dst-entries-invalidation-if.patch) - this is on top of below mentioned patch (0001-sctp-check-cached-dst-before-using-it.patch)

Quoted 6 lines

2) in a separate module route and ipsec events are monitored and upon addition and deletion of routes, along with xfrm events interface is called. This is done by netlink socket opened within the kernel. If interested, I can share that one as well.
Could you please give some comments on both workaround and mentioned issue?

These workarounds should not be needed. The code should function correctly. We just need to get to the root cause...

Quoted 36 lines

Thanks and regards,
Matija
Quoted 30 lines · ext Matija Glavinic Pecotic
Hello,
We're facing an issue regarding route change and startup of ipsec tunnel. Here is setup and test scenarios:
1) Setup
Simple sctp client and server. Client is bound to the loopback address 10.0.0.1 and it is connected to the server via eth3 (192.168.2.1) which is in LAN with eth2 (192.168.2.2) and eth3 (192.168.2.3) on the server and server is bound to the 20.0.0.1 which is loopback.
Client:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 10.0.0.1/8 scope global lo
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
13: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
      link/ether 00:40:43:30:81:80 brd ff:ff:ff:ff:ff:ff
      inet 192.168.2.1/24 brd 192.168.2.255 scope global eth3
      inet6 fe80::240:43ff:fe30:8180/64 scope link
         valid_lft forever preferred_lft forever
192.168.2.0/24 dev eth3 proto kernel scope link src 192.168.2.1
192.168.253.0/24 dev rio0  proto kernel  scope link  src 192.168.253.16
192.168.254.0/23 dev eth0  proto kernel  scope link  src 192.168.255.1
20.0.0.0/8 via 192.168.2.1 dev eth3 scope link

Should the above route be via 192.168.2.2 (typo), or is this correct?

Quoted 37 lines

Quoted 37 lines
Server:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 20.0.0.1/8 scope global lo
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
12: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
      link/ether 00:40:43:b1:f8:ae brd ff:ff:ff:ff:ff:ff
      inet 192.168.2.2/24 brd 192.168.2.255 scope global eth2
      inet6 fe80::240:43ff:feb1:f8ae/64 scope link
         valid_lft forever preferred_lft forever
13: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
      link/ether 00:40:43:b1:f8:af brd ff:ff:ff:ff:ff:ff
      inet 192.168.2.3/24 brd 192.168.2.255 scope global eth3
      inet6 fe80::240:43ff:feb1:f8af/64 scope link
         valid_lft forever preferred_lft forever
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.2
192.168.2.0/24 dev eth3  proto kernel  scope link  src 192.168.2.3
192.168.253.0/24 dev rio0  proto kernel  scope link  src 192.168.253.16
192.168.255.0/24 dev eth0  proto kernel  scope link  src 192.168.255.2
10.0.0.0/8 via 192.168.2.1 dev eth2
10.0.0.0/8 via 192.168.2.1 dev eth3 metric 10
Server listens on 20.0.0.1 and when client is connected it sends message indefinitely
Kernel is 2.6.32.58 and sctp is applied with patch from this thread:
http://sourceforge.net/mailarchive/message.php?msg_id%786006
(0001-sctp-check-cached-dst-before-using-it.patch)

Quoted 7 lines

Quoted 7 lines
2) Test scenario 1
Start sctp_server
Start sctp_client -> clients receives messages from the server Observe traffic flow -> as expected it goes over eth2 on the server Delete route 10.0.0.0/8 via 192.168.2.1 dev eth2
Traffic still goes over eth2, on server restart it goes over eth3

Very interesting... Upon route deletion the cache should be invalidated and sctp should look-up a new route.

Can you turn on debug output in net/sctp/protocol.c and see what the route printed is after the delete call?

You can turn it on by adding #define CONFIG_SCTP_DBG_MSG to the top of protocol.c

-vlad

Quoted 88 lines

Quoted 61 lines
For reference, I've run also TCP server/client which does the same thing, after deletion of mentioned route traffic continued over eth3. Other kinds of traffic are also rerouted on route deletion (icmp)
3) Test scenario 2
Setup same as before, eth3 on server isn't used.
Start sctp_server
Start sctp_client -> traffic ok
Start ipsec tunnel:
      left2.168.2.2
      leftsubnet .0.0.0/24
      right2.168.2.1
      rightsubnet.0.0.0/24
After start of ipsec tunnel, traffic went over same dst unencrypted for a short period of time.
For reference, again same scenario with TCP server/client -> traffic started to be encrypted immediately
------
In my opinion there is problem with cached dst_entry. I've tried following:
[sctp.h]
/* The cookie is always 0 since this is how it's used in the
* pmtu code.
*/
static inline struct dst_entry *sctp_transport_dst_check(struct sctp_transport *t)
{
    if (t->dst) {
        dst_release(t->dst);
        t->dst = NULL;
    }

    return t->dst;
}
With this I just wanted to force sctp to route every time it wants to send packet, and after this change above mentioned scenarios passed. Here is my short analysis, ignore it if I got something wrong: I see potential problem with SCTP's dst_entry: struct dst_entry *dst which is member of sctp_transport. For example, ipv4&6 implement operations on dst_entry (ipv4_dst_ops ) which are to be done when event occurs, such as unreachability, route deleteion, etc. SCTP doesn't, and it also doesn't use dst_entries from ipv*. In this way, only events which impact SCTPs dst_entries are default ones, NETDEV_DOWN and NETDEV_UNREGISTER and these are only which will be noted with dst_check().
Could you comment please? I'll be glad to make tests/verifications or provide any sort of help on this.
Thanks & regards,
Matija
------------------------------------------------------------------------------
Quoted 10 lines
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Lksctp-developers mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Lksctp-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers

Matija Glavinic Pecotic

author

2012-09-10 06:33 UTC #2 lore raw

In reply to Matija Glavinic Pecotic

Hello,

re-sending after 4 days, I believe something went wrong.

Regards,

Matija

Quoted 256 lines · Matija Glavinic Pecotic

Hello,
Quoted 56 lines · ext Vlad Yasevich
Quoted 11 lines · Matija Glavinic Pecotic
Hello lksctp developers,
a while back we noticed below described issue.
In order to quick-fix this, we implemented following workaround:
1) added interface to sctp for invalidating all dst entries on all associations (0002-sctp-export-dst-entries-invalidation-if.patch) - this is on top of below mentioned patch (0001-sctp-check-cached-dst-before-using-it.patch)
Quoted 7 lines
2) in a separate module route and ipsec events are monitored and upon addition and deletion of routes, along with xfrm events interface is called. This is done by netlink socket opened within the kernel. If interested, I can share that one as well.
Could you please give some comments on both workaround and mentioned issue?
These workarounds should not be needed. The code should function correctly. We just need to get to the root cause...
Quoted 32 lines
Thanks and regards,
Matija
Quoted 26 lines · ext Matija Glavinic Pecotic
Hello,
We're facing an issue regarding route change and startup of ipsec tunnel. Here is setup and test scenarios:
1) Setup
Simple sctp client and server. Client is bound to the loopback address 10.0.0.1 and it is connected to the server via eth3 (192.168.2.1) which is in LAN with eth2 (192.168.2.2) and eth3 (192.168.2.3) on the server and server is bound to the 20.0.0.1 which is loopback.
Client:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 10.0.0.1/8 scope global lo
       inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
13: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
       link/ether 00:40:43:30:81:80 brd ff:ff:ff:ff:ff:ff
       inet 192.168.2.1/24 brd 192.168.2.255 scope global eth3
       inet6 fe80::240:43ff:fe30:8180/64 scope link
          valid_lft forever preferred_lft forever
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
192.168.2.0/24 dev eth3 proto kernel scope link src 192.168.2.1
192.168.253.0/24 dev rio0  proto kernel  scope link  src 192.168.253.16
192.168.254.0/23 dev eth0  proto kernel  scope link  src 192.168.255.1
20.0.0.0/8 via 192.168.2.1 dev eth3 scope link
Should the above route be via 192.168.2.2 (typo), or is this correct?
No, I want to say that traffic on the lo is to go over eth3 interface. I dont want to tie it directly to 192.168.2.2.
Quoted 46 lines
Quoted 38 lines
Quoted 38 lines
Server:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 20.0.0.1/8 scope global lo
       inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
12: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
       link/ether 00:40:43:b1:f8:ae brd ff:ff:ff:ff:ff:ff
       inet 192.168.2.2/24 brd 192.168.2.255 scope global eth2
       inet6 fe80::240:43ff:feb1:f8ae/64 scope link
          valid_lft forever preferred_lft forever
13: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
       link/ether 00:40:43:b1:f8:af brd ff:ff:ff:ff:ff:ff
       inet 192.168.2.3/24 brd 192.168.2.255 scope global eth3
       inet6 fe80::240:43ff:feb1:f8af/64 scope link
          valid_lft forever preferred_lft forever
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.2
192.168.2.0/24 dev eth3  proto kernel  scope link  src 192.168.2.3
192.168.253.0/24 dev rio0  proto kernel  scope link  src 192.168.253.16
192.168.255.0/24 dev eth0  proto kernel  scope link  src 192.168.255.2
10.0.0.0/8 via 192.168.2.1 dev eth2
10.0.0.0/8 via 192.168.2.1 dev eth3 metric 10
Server listens on 20.0.0.1 and when client is connected it sends message
indefinitely
Kernel is 2.6.32.58 and sctp is applied with patch from this thread:
http://sourceforge.net/mailarchive/message.php?msg_id=25786006
(0001-sctp-check-cached-dst-before-using-it.patch)
Quoted 7 lines
Quoted 7 lines
2) Test scenario 1
Start sctp_server
Start sctp_client -> clients receives messages from the server Observe traffic flow -> as expected it goes over eth2 on the server Delete route 10.0.0.0/8 via 192.168.2.1 dev eth2
Traffic still goes over eth2, on server restart it goes over eth3
Quoted 8 lines
Very interesting... Upon route deletion the cache should be invalidated and sctp should look-up a new route.
Can you turn on debug output in net/sctp/protocol.c and see what the route printed is after the delete call?
You can turn it on by adding #define CONFIG_SCTP_DBG_MSG to the top of protocol.c
attached two logs, in one, cable was plugged out, we can see that stack copes with it, especially since we have new dst (I added print of address)
Quoted 3 lines
    if (dst)
        SCTP_DEBUG_PRINTK("rt_dst:%pI4, rt_src:%pI4, rt@0x%p\n",
                  &rt->rt_dst, &rt->rt_src, rt);
other log is as scenario mentioned, additional info inside regarding routes. You can look for:
2004-01-01T00:01:48.329137+00:00 (none) [notice]      root:  Deleting route
Scenario went as described previously, traffic continued over eth2
Thanks and regards,
Matija
Quoted 112 lines
-vlad
Quoted 90 lines
Quoted 63 lines
For reference, I've run also TCP server/client which does the same thing, after deletion of mentioned route traffic continued over eth3. Other kinds of traffic are also rerouted on route deletion (icmp)
3) Test scenario 2
Setup same as before, eth3 on server isn't used.
Start sctp_server
Start sctp_client -> traffic ok
Start ipsec tunnel:
       left=192.168.2.2
       leftsubnet=20.0.0.0/24
       right=192.168.2.1
       rightsubnet=10.0.0.0/24
After start of ipsec tunnel, traffic went over same dst unencrypted for a short period of time.
For reference, again same scenario with TCP server/client -> traffic started to be encrypted immediately
------
In my opinion there is problem with cached dst_entry. I've tried following:
[sctp.h]
/* The cookie is always 0 since this is how it's used in the
* pmtu code.
*/
static inline struct dst_entry *sctp_transport_dst_check(struct sctp_transport *t)
{
     if (t->dst) {
         dst_release(t->dst);
         t->dst = NULL;
     }

     return t->dst;
}
With this I just wanted to force sctp to route every time it wants to send packet, and after this change above mentioned scenarios passed. Here is my short analysis, ignore it if I got something wrong: I see potential problem with SCTP's dst_entry: struct dst_entry *dst which is member of sctp_transport. For example, ipv4&6 implement operations on dst_entry (ipv4_dst_ops ) which are to be done when event occurs, such as unreachability, route deleteion, etc. SCTP doesn't, and it also doesn't use dst_entries from ipv*. In this way, only events which impact
SCTPs dst_entries are default ones, NETDEV_DOWN and NETDEV_UNREGISTER and these are only which will be noted with dst_check().
Could you comment please? I'll be glad to make tests/verifications or provide any sort of help on this.
Thanks & regards,
Matija
------------------------------------------------------------------------------
Quoted 12 lines
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Lksctp-developers mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Quoted 5 lines
_______________________________________________
Lksctp-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Lksctp-developers mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers

Matija Glavinic Pecotic

author

2012-11-14 15:44 UTC #3 lore raw

In reply to Matija Glavinic Pecotic

Hello,

sorry for resending, but did someone maybe had a look on this? Any feedback would be appreciated.

Thanks!

Matija

Quoted 279 lines · Matija Glavinic Pecotic

Hello,
re-sending after 4 days, I believe something went wrong.
Regards,
Matija
Quoted 269 lines · Matija Glavinic Pecotic
Hello,
Quoted 58 lines · ext Vlad Yasevich
Quoted 12 lines · Matija Glavinic Pecotic
Hello lksctp developers,
a while back we noticed below described issue.
In order to quick-fix this, we implemented following workaround:
1) added interface to sctp for invalidating all dst entries on all associations (0002-sctp-export-dst-entries-invalidation-if.patch) - this
is on top of below mentioned patch (0001-sctp-check-cached-dst-before-using-it.patch)
Quoted 7 lines
2) in a separate module route and ipsec events are monitored and upon addition and deletion of routes, along with xfrm events interface is called. This is done by netlink socket opened within the kernel. If interested, I can share that one as well.
Could you please give some comments on both workaround and mentioned issue?
These workarounds should not be needed. The code should function correctly. We just need to get to the root cause...
Quoted 33 lines
Thanks and regards,
Matija
Quoted 27 lines · ext Matija Glavinic Pecotic
Hello,
We're facing an issue regarding route change and startup of ipsec tunnel. Here is setup and test scenarios:
1) Setup
Simple sctp client and server. Client is bound to the loopback address 10.0.0.1 and it is connected to the server via eth3 (192.168.2.1) which
is in LAN with eth2 (192.168.2.2) and eth3 (192.168.2.3) on the server and server is bound to the 20.0.0.1 which is loopback.
Client:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 10.0.0.1/8 scope global lo
       inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
13: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
       link/ether 00:40:43:30:81:80 brd ff:ff:ff:ff:ff:ff
       inet 192.168.2.1/24 brd 192.168.2.255 scope global eth3
       inet6 fe80::240:43ff:fe30:8180/64 scope link
          valid_lft forever preferred_lft forever
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
192.168.2.0/24 dev eth3 proto kernel scope link src 192.168.2.1 192.168.253.0/24 dev rio0 proto kernel scope link src 192.168.253.16
192.168.254.0/23 dev eth0 proto kernel scope link src 192.168.255.1 20.0.0.0/8 via 192.168.2.1 dev eth3 scope link
Should the above route be via 192.168.2.2 (typo), or is this correct?
No, I want to say that traffic on the lo is to go over eth3 interface. I dont want to tie it directly to 192.168.2.2.
Quoted 47 lines
Quoted 39 lines
Quoted 39 lines
Server:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 20.0.0.1/8 scope global lo
       inet6 ::1/128 scope host
          valid_lft forever preferred_lft forever
12: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
       link/ether 00:40:43:b1:f8:ae brd ff:ff:ff:ff:ff:ff
       inet 192.168.2.2/24 brd 192.168.2.255 scope global eth2
       inet6 fe80::240:43ff:feb1:f8ae/64 scope link
          valid_lft forever preferred_lft forever
13: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
       link/ether 00:40:43:b1:f8:af brd ff:ff:ff:ff:ff:ff
       inet 192.168.2.3/24 brd 192.168.2.255 scope global eth3
       inet6 fe80::240:43ff:feb1:f8af/64 scope link
          valid_lft forever preferred_lft forever
192.168.2.0/24 dev eth2  proto kernel  scope link  src 192.168.2.2
192.168.2.0/24 dev eth3  proto kernel  scope link  src 192.168.2.3
192.168.253.0/24 dev rio0 proto kernel scope link src 192.168.253.16
192.168.255.0/24 dev eth0 proto kernel scope link src 192.168.255.2 10.0.0.0/8 via 192.168.2.1 dev eth2
10.0.0.0/8 via 192.168.2.1 dev eth3 metric 10
Server listens on 20.0.0.1 and when client is connected it sends message
indefinitely
Kernel is 2.6.32.58 and sctp is applied with patch from this thread:
http://sourceforge.net/mailarchive/message.php?msg_id%786006
(0001-sctp-check-cached-dst-before-using-it.patch)
Quoted 7 lines
Quoted 7 lines
2) Test scenario 1
Start sctp_server
Start sctp_client -> clients receives messages from the server Observe traffic flow -> as expected it goes over eth2 on the server Delete route 10.0.0.0/8 via 192.168.2.1 dev eth2
Traffic still goes over eth2, on server restart it goes over eth3
Quoted 8 lines
Very interesting... Upon route deletion the cache should be invalidated and sctp should look-up a new route.
Can you turn on debug output in net/sctp/protocol.c and see what the route printed is after the delete call?
You can turn it on by adding #define CONFIG_SCTP_DBG_MSG to the top of protocol.c
attached two logs, in one, cable was plugged out, we can see that stack copes with it, especially since we have new dst (I added print of address)
Quoted 3 lines
    if (dst)
        SCTP_DEBUG_PRINTK("rt_dst:%pI4, rt_src:%pI4, rt@0x%p\n",
                  &rt->rt_dst, &rt->rt_src, rt);
other log is as scenario mentioned, additional info inside regarding routes. You can look for:
2004-01-01T00:01:48.329137+00:00 (none) [notice]      root:  Deleting
route
Scenario went as described previously, traffic continued over eth2
Thanks and regards,
Matija
Quoted 119 lines
-vlad
Quoted 94 lines
Quoted 66 lines
For reference, I've run also TCP server/client which does the same thing, after deletion of mentioned route traffic continued over eth3. Other kinds of traffic are also rerouted on route deletion (icmp)
3) Test scenario 2
Setup same as before, eth3 on server isn't used.
Start sctp_server
Start sctp_client -> traffic ok
Start ipsec tunnel:
       left2.168.2.2
       leftsubnet .0.0.0/24
       right2.168.2.1
       rightsubnet.0.0.0/24
After start of ipsec tunnel, traffic went over same dst unencrypted for
a short period of time.
For reference, again same scenario with TCP server/client -> traffic started to be encrypted immediately
------
In my opinion there is problem with cached dst_entry. I've tried following:
[sctp.h]
/* The cookie is always 0 since this is how it's used in the
* pmtu code.
*/
static inline struct dst_entry *sctp_transport_dst_check(struct sctp_transport *t)
{
     if (t->dst) {
         dst_release(t->dst);
         t->dst = NULL;
     }

     return t->dst;
}
With this I just wanted to force sctp to route every time it wants to send packet, and after this change above mentioned scenarios passed. Here is my short analysis, ignore it if I got something wrong: I see potential problem with SCTP's dst_entry: struct dst_entry *dst which is
member of sctp_transport. For example, ipv4&6 implement operations on dst_entry (ipv4_dst_ops ) which are to be done when event occurs, such as unreachability, route deleteion, etc. SCTP doesn't, and it also doesn't use dst_entries from ipv*. In this way, only events which impact
SCTPs dst_entries are default ones, NETDEV_DOWN and NETDEV_UNREGISTER and these are only which will be noted with dst_check().
Could you comment please? I'll be glad to make tests/verifications or provide any sort of help on this.
Thanks & regards,
Matija
------------------------------------------------------------------------------
Quoted 12 lines
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Lksctp-developers mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Quoted 5 lines
_______________________________________________
Lksctp-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Lksctp-developers mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/lksctp-developers