Skip to contentSkip to conversation

Multicast routing stops functioning after 4G multicast packets recived.

DISCUSSION Bob Falken 5 participants 9 messages Dec 19, 2013 – Jan 5, 2014

Subsystems (3)
lore
Share & tools
lore

Bob Falken

author
#0 lore raw

Hello, I have an issue after kernel 2.6.37 and above.
If i roll back to kernel 2.6.36.4 everything is fine.
if recive more than 4294967295 multicast packets, the kernel does not register the multicast packets. and multicast routing does not functioning. (Tested bouth FIB_HASH and FIB_TRIE)
Tested with xorp and pimd.
I have abount 24 multicast groups, and it takes me about 17hours to reproduce the issue after a reboot.
Reboot is reqired to fix the issue. (Tested to stop/start pimd/xorp, reload network module for the network interface "e1000e", Used birdge adapter and remove bridge adapter and readd bridge adapter to clear counters. none of thouse solves the issue.)

When the packet count for the input interface goes over 4294967295 packets. br0 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:XX
inet addr:10.255.255.1 Bcast:10.255.255.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4295483655 errors:0 dropped:0 overruns:0 frame:0
TX packets:4210 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:3350477250900 (3.0 TiB) TX bytes:240828 (235.1 KiB)

"/proc/net/ip_mr_cache" goes blank.

This is the state of "/proc/net/ip_mr_cache" a couple of seconds before the stop:
Group Origin Iif Pkts Bytes Wrong Oifs
0A0202EF 0BFFFF0A 1 181610713 4217402668 0
090202EF 0BFFFF0A 1 181610715 4217404228 0
080202EF 0BFFFF0A 1 181610716 4217405008 0
070202EF 0BFFFF0A 1 181610716 4217405008 0
180202EF 0DFFFF0A 1 181611351 4217900308 0
170202EF 0DFFFF0A 1 181611353 4217901868 0
120202EF 0CFFFF0A 1 170885816 146950304 0
020202EF 0AFFFF0A 1 181609636 4216562608 0
160202EF 0DFFFF0A 1 181611358 4217905768 0
150202EF 0DFFFF0A 1 181611359 4217906548 0
060202EF 0AFFFF0A 1 181609641 4216566508 0
140202EF 0DFFFF0A 1 181611364 4217910448 0
050202EF 0AFFFF0A 1 181609644 4216568848 0
130202EF 0DFFFF0A 1 181611366 4217912008 0
040202EF 0AFFFF0A 1 181609649 4216572748 0
030202EF 0AFFFF0A 1 181609651 4216574308 0 0:1
110202EF 0CFFFF0A 1 170885832 146962784 0
010202EF 0AFFFF0A 1 181609652 4216575088 0 0:1
100202EF 0CFFFF0A 1 170885837 146966684 0
0F0202EF 0CFFFF0A 1 170885838 146967464 0
0E0202EF 0CFFFF0A 1 170885841 146969804 0
0D0202EF 0CFFFF0A 1 170885842 146970584 0
0C0202EF 0BFFFF0A 1 181610767 4217444788 0
0B0202EF 0BFFFF0A 1 181610772 4217448688 0
i have also tested the following kernels:linux-2.6.38.8 linux-2.6.39.4
linux-3.2.53
linux-3.10.24
linux-3.11.9
I can reproduce the issue on all of them.

dmesg does provide any messages when this issue occure.

My arcutecture is "amd64".

I think there might be a variable in the kernel that get overflown. though i cannot be sure as im not a programmer.

Let me know if you need more debug information.

Please help.

#1 lore raw
Quoted 8 lines · Bob Falken

Hello, I have an issue after kernel 2.6.37 and above.
If i roll back to kernel 2.6.36.4 everything is fine.
if recive more than 4294967295 multicast packets, the kernel does not register the multicast packets. and multicast routing does not functioning. (Tested bouth FIB_HASH and FIB_TRIE)
Tested with xorp and pimd.
I have abount 24 multicast groups, and it takes me about 17hours to reproduce the issue after a reboot.
Reboot is reqired to fix the issue. (Tested to stop/start pimd/xorp, reload network module for the network interface "e1000e", Used birdge adapter and remove bridge adapter and readd bridge adapter to clear counters. none of thouse solves the issue.)

Please test this with a recent kernel. 2.6.37 is really old and you normally won't get good support here with such old kernels.

Greetings,

Hannes

#2 lore raw
Quoted 10 lines · Hannes Frederic Sowa
Quoted 8 lines · Bob Falken

Hello, I have an issue after kernel 2.6.37 and above.
If i roll back to kernel 2.6.36.4 everything is fine.
if recive more than 4294967295 multicast packets, the kernel does not register the multicast packets. and multicast routing does not functioning. (Tested bouth FIB_HASH and FIB_TRIE)
Tested with xorp and pimd.
I have abount 24 multicast groups, and it takes me about 17hours to reproduce the issue after a reboot.
Reboot is reqired to fix the issue. (Tested to stop/start pimd/xorp, reload network module for the network interface "e1000e", Used birdge adapter and remove bridge adapter and readd bridge adapter to clear counters. none of thouse solves the issue.)

Quoted 2 lines

Please test this with a recent kernel. 2.6.37 is really old and you normally won't get good support here with such old kernels.

Note that he did test up to 3.11.9 and it still showed failures.

Thanks,
Ben

Quoted 10 lines

Greetings,

Hannes

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#3 lore raw
Quoted 16 lines · Ben Greear
Quoted 15 lines · Hannes Frederic Sowa
Quoted 13 lines · Bob Falken

Hello, I have an issue after kernel 2.6.37 and above.
If i roll back to kernel 2.6.36.4 everything is fine.
if recive more than 4294967295 multicast packets, the kernel does not register the multicast packets. and multicast routing does not functioning.
(Tested bouth FIB_HASH and FIB_TRIE)
Tested with xorp and pimd.
I have abount 24 multicast groups, and it takes me about 17hours to reproduce the issue after a reboot.
Reboot is reqired to fix the issue. (Tested to stop/start pimd/xorp, reload network module for the network interface "e1000e", Used birdge adapter and remove bridge adapter and readd bridge adapter to clear counters. none of thouse solves the issue.)

Quoted 5 lines
Quoted 3 lines

Please test this with a recent kernel. 2.6.37 is really old and you normally
won't get good support here with such old kernels.

Note that he did test up to 3.11.9 and it still showed failures.

Oh sorry, I did not read to the end. ;)

An interesting hint could be to use dropwatch or perf script net_dropmonitor to check where the fragments get dropped. Also nstat could give additional hints where something might get wrong. Please use a recent kernel while debugging this issue. Maybe a patch can get backported later.

Thanks,

Hannes

#4 lore raw
In reply to Bob Falken

Hello,

Quoted 3 lines · Bob Falken

Hello, I have an issue after kernel 2.6.37 and above.
If i roll back to kernel 2.6.36.4 everything is fine.
if recive more than 4294967295 multicast packets, the kernel does not register the multicast packets. and multicast routing does not functioning.

...

Quoted 5 lines

I think there might be a variable in the kernel that get overflown. though i cannot be sure as im not a programmer.

Let me know if you need more debug information.

Please help.

As Hannes guessed, may be it is really the missing
flags = FIB_LOOKUP_NOREF in ipmr_fib_lookup() when calling fib_rules_lookup(), we have an atomic_inc_not_zero() there that can stop to work after 4G lookups when rule's refcnt reaches 0. As result, fib_rules_lookup() can start to return -ESRCH.

Regards

--
Julian Anastasov <[email protected]>

#5 lore raw
Quoted 15 lines · Julian Anastasov

Hello,

Quoted 3 lines · Bob Falken

Hello, I have an issue after kernel 2.6.37 and above.
If i roll back to kernel 2.6.36.4 everything is fine.
if recive more than 4294967295 multicast packets, the kernel does not register the multicast packets. and multicast routing does not functioning.

...

Quoted 4 lines

I think there might be a variable in the kernel that get overflown. though i cannot be sure as im not a programmer.

Let me know if you need more debug information.

Quoted 7 lines
Quoted 1 line

Please help.

As Hannes guessed, may be it is really the missing
flags = FIB_LOOKUP_NOREF in ipmr_fib_lookup() when calling fib_rules_lookup(), we have an atomic_inc_not_zero() there that can stop to work after 4G lookups when rule's refcnt reaches 0. As result, fib_rules_lookup() can start to return -ESRCH.

I guess we should just try. I somehow forgot to look after that. Thanks for reminding, Julian.

Bob, can you try with this patch again?

net/ipv4/ipmr.c +5 -2
index 421a249..b9b3472 100644
@@ -157,9 +157,12 @@ static struct mr_table *ipmr_get_table(struct net *net, u32 id)
 static int ipmr_fib_lookup(struct net *net, struct flowi4 *flp4,
 			   struct mr_table **mrt)
 {
-	struct ipmr_result res;
-	struct fib_lookup_arg arg = { .result = &res, };
 	int err;
+	struct ipmr_result res;
+	struct fib_lookup_arg arg = {
+		.result = &res,
+		.flags = FIB_LOOKUP_NOREF,
+	};
 
 	err = fib_rules_lookup(net->ipv4.mr_rules_ops,
 			       flowi4_to_flowi(flp4), 0, &arg);
#6 lore raw

Hello,

Quoted 2 lines · Hannes Frederic Sowa

I guess we should just try. I somehow forgot to look after that. Thanks for reminding, Julian.

If that patch solves the problem, do not forget
to include same fix for ip6mr_fib_lookup(). As for net/decnet/, it looks ok, it uses fib_rule_put() in dn_fib_res_put().

Quoted 11 lines

Bob, can you try with this patch again?

net/ipv4/ipmr.c +0 -1
index 421a249..b9b3472 100644
@@ -157,9 +157,12 @@ static struct mr_table *ipmr_get_table(struct net *net, u32 id)
 static int ipmr_fib_lookup(struct net *net, struct flowi4 *flp4,
 			   struct mr_table **mrt)
 {
-	struct ipmr_result res;

net/ipv4/ipmr.c static struct mr_table *ipmr_get_table(struct net *net, u32 id) · line 168

Quoted 10 lines
net/ipv4/ipmr.c +5 -1
-	struct fib_lookup_arg arg = { .result = &res, };
 	int err;
+	struct ipmr_result res;
+	struct fib_lookup_arg arg = {
+		.result = &res,
+		.flags = FIB_LOOKUP_NOREF,
+	};
 
 	err = fib_rules_lookup(net->ipv4.mr_rules_ops,
 			       flowi4_to_flowi(flp4), 0, &arg);

Regards

--
Julian Anastasov <[email protected]>

#7 lore raw
Quoted 11 lines · Julian Anastasov

Hello,

Quoted 2 lines · Hannes Frederic Sowa

I guess we should just try. I somehow forgot to look after that. Thanks for reminding, Julian.

If that patch solves the problem, do not forget
to include same fix for ip6mr_fib_lookup(). As for net/decnet/, it looks ok, it uses fib_rule_put() in dn_fib_res_put().

Thanks, I had IPv6 on my radar but did not check for other callers.

Greetings,

Hannes

#8 lore raw
Quoted 6 lines · Hannes Frederic Sowa
Quoted 5 lines · Julian Anastasov

Hello,

On Sun, 5 Jan 2014, Hannes Frederic Sowa wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I guess we should just try. I somehow forgot to look after that. Thanks for reminding, Julian.

If that patch solves the problem, do not forget
to include same fix for ip6mr_fib_lookup(). As for net/decnet/, it looks ok, it uses fib_rule_put() in dn_fib_res_put().

Thanks, I had IPv6 on my radar but did not check for other callers.

BTW, I wonder if the old kernels (2.6.32 & 2.6.34) need something as well ?

Before commit ebc0ffae5dfb444 and commit 7fa7cb7109d0, it looks like we had an overflow in fib_rule_get() and refcnt was wrapping.

So maybe we should add a fib_rule_put() for stable kernels, and use FIB_LOOKUP_NOREF on next kernels ...