Multicast routing stops functioning after 4G multicast packets recived.
Subsystems (2)
OK, so at the exact time that the incoming interface for multicast packet count reaches 2^32, the /proc/net/ip_mr_cache stops updating.
after a while, one by one the multicast groups in ip_mr_cache disapperes, and after 227sec all of them are gone.
perf script net_dropmonitor:
-----------
# ========
# captured on: Sat Dec 21 23:27:37 2013 # ======== # Starting trace (Ctrl-C to dump results) Warning: Processed 788648 events and lost 118 chunks!
Check IO/CPU overload!
Gathering kallsyms data
35200/35200
LOCATION OFFSET COUNT _stext 18446744071578845580 6 _stext 18446744071578843536 785790 _stext 18446744071578843530 1
-------------
netstat -s:
Ip:
622406 total packets received
2 with invalid addresses
0 forwarded
0 incoming packets discarded
599574 incoming packets delivered
520762 requests sent out
8 dropped because of missing route
Icmp:
19361 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
echo requests: 8415
echo replies: 10946
19361 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
echo request: 10946
echo replies: 8415
IcmpMsg:
InType0: 10946
InType8: 8415
OutType0: 8415
OutType8: 10946
Tcp:
15 active connections openings
15 passive connection openings
0 failed connection attempts
0 connection resets received
29 connections established
477938 segments received
482321 segments send out
4 segments retransmited
0 bad segments received.
0 resets sent
Udp:
586 packets received
0 packets to unknown port received.
0 packet receive errors
649 packets sent
UdpLite:
TcpExt:
15862 delayed acks sent
Quick ack mode was activated 1 times
1 packets directly queued to recvmsg prequeue.
390374 packet headers predicted
1767 acknowledgments not containing data payload received 58169 predicted acknowledgments
4 congestion windows recovered without slow start after partial ack 4 other TCP timeouts
1 DSACKs sent for old packets
4 DSACKs received
TCPSackShiftFallback: 3
IpExt:
InNoRoutes: 1
InMcastPkts: 40015
OutMcastPkts: 18427
InBcastPkts: 80035
InOctets: 1116615859
OutOctets: 33742922
InMcastOctets: 1046924948
OutMcastOctets: 734556
InBcastOctets: 7255577
---------------------
----- Original Message -----
From: Hannes Frederic Sowa
Sent: 12/19/13 06:32 PM
To: Eric Dumazet
Subject: Re: Multicast routing stops functioning after 4G multicast packets recived.
Quoted 14 lines · Eric Dumazet
Quoted 13 lines · Bob Falken
The only reason why i give information about 2.6.36.4 is that its the only latest kernel that was functioning properly.
i.e kernel >= 2.6.37 is not woking. so its a bisecting of the kernel versions to help a coder see when/where the isse was implemented in the kernel.I do not need a backport patch for an old kernel, I generally only need the issue looked into and get fixed so that I dont have to use an old kernel. :)
I have no issue reproducing the issue on the recent kernels. however i have not tried the GIT kernel.
Quoted 8 lines
Quoted 2 lines
I restarted the server just a moment ago. i will install and run dropwatch and provide feedback in about 17hours.
You said that "cat
/proc/net/ip_mr_cache" gives nothing at all after 2^32 packets ?Thats a bit scary ... maybe a 32bit refcnt overflow, because of some imbalance...
That's my thought, too. :/
The ipmr.c rcu conversion happend in 2.6.37.
Quoted 21 lines · Bob Falken
Quoted 20 lines · Eric Dumazet
Quoted 10 lines · Bob Falken
The only reason why i give information about 2.6.36.4 is that its the only latest kernel that was functioning properly.
i.e kernel >= 2.6.37 is not woking. so its a bisecting of the kernel versions to help a coder see when/where the isse was implemented in the kernel.I do not need a backport patch for an old kernel, I generally only need the issue looked into and get fixed so that I dont have to use an old kernel. :)
Quoted 5 lines
I have no issue reproducing the issue on the recent kernels. however i have not tried the GIT kernel.
I restarted the server just a moment ago. i will install and run dropwatch and provide feedback in about 17hours.
You said that "cat
/proc/net/ip_mr_cache" gives nothing at all after 2^32 packets ?
Quoted 7 lines
Quoted 2 lines
Thats a bit scary ... maybe a 32bit refcnt overflow, because of some imbalance...
That's my thought, too. :/
The
ipmr.crcu conversion happend in 2.6.37.
Just as a follow-up. You could verify if we actually have a refcount overflow if you try a grsec kernel with PAX_REFCOUNT enabled. If a refcoount overflow happens, dmesg should show then.
Greetings,
Hannes