Skip to contentSkip to conversation

Multicast routing stops functioning after 4G multicast packets recived.

DISCUSSION Bob Falken 2 participants 2 messages Dec 21, 2013 – Jan 3, 2014

Subsystems (2)
lore
Share & tools
lore

Bob Falken

author
#0 lore raw

OK, so at the exact time that the incoming interface for multicast packet count reaches 2^32, the /proc/net/ip_mr_cache stops updating.
after a while, one by one the multicast groups in ip_mr_cache disapperes, and after 227sec all of them are gone.

perf script net_dropmonitor:
-----------
# ========

# captured on: Sat Dec 21 23:27:37 2013
# ========
#
Starting trace (Ctrl-C to dump results)
Warning:
Processed 788648 events and lost 118 chunks!

Check IO/CPU overload!

Gathering kallsyms data
35200/35200
                 LOCATION                    OFFSET                     COUNT _stext      18446744071578845580                         6 _stext      18446744071578843536                    785790 _stext      18446744071578843530                         1

-------------

netstat -s:
Ip:
    622406 total packets received
    2 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    599574 incoming packets delivered
    520762 requests sent out
    8 dropped because of missing route
Icmp:
    19361 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        echo requests: 8415
        echo replies: 10946
    19361 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        echo request: 10946
        echo replies: 8415
IcmpMsg:
        InType0: 10946
        InType8: 8415
        OutType0: 8415
        OutType8: 10946
Tcp:
    15 active connections openings
    15 passive connection openings
    0 failed connection attempts
    0 connection resets received
    29 connections established
    477938 segments received
    482321 segments send out
    4 segments retransmited
    0 bad segments received.
    0 resets sent
Udp:
    586 packets received
    0 packets to unknown port received.
    0 packet receive errors
    649 packets sent
UdpLite:
TcpExt:
    15862 delayed acks sent
    Quick ack mode was activated 1 times
    1 packets directly queued to recvmsg prequeue.
    390374 packet headers predicted
    1767 acknowledgments not containing data payload received 58169 predicted acknowledgments
    4 congestion windows recovered without slow start after partial ack 4 other TCP timeouts
    1 DSACKs sent for old packets
    4 DSACKs received
    TCPSackShiftFallback: 3
IpExt:
    InNoRoutes: 1
    InMcastPkts: 40015
    OutMcastPkts: 18427
    InBcastPkts: 80035
    InOctets: 1116615859
    OutOctets: 33742922
    InMcastOctets: 1046924948
    OutMcastOctets: 734556
    InBcastOctets: 7255577

---------------------
----- Original Message -----
From: Hannes Frederic Sowa
Sent: 12/19/13 06:32 PM
To: Eric Dumazet
Subject: Re: Multicast routing stops functioning after 4G multicast packets recived.

Quoted 14 lines · Eric Dumazet
Quoted 13 lines · Bob Falken

The only reason why i give information about 2.6.36.4 is that its the only latest kernel that was functioning properly.
i.e kernel >= 2.6.37 is not woking. so its a bisecting of the kernel versions to help a coder see when/where the isse was implemented in the kernel.

I do not need a backport patch for an old kernel, I generally only need the issue looked into and get fixed so that I dont have to use an old kernel. :)

I have no issue reproducing the issue on the recent kernels. however i have not tried the GIT kernel.

Quoted 8 lines
Quoted 2 lines

I restarted the server just a moment ago. i will install and run dropwatch and provide feedback in about 17hours.

You said that "cat /proc/net/ip_mr_cache" gives nothing at all after 2^32 packets ?

Thats a bit scary ... maybe a 32bit refcnt overflow, because of some imbalance...

That's my thought, too. :/

The ipmr.c rcu conversion happend in 2.6.37.

#1 lore raw
Quoted 21 lines · Bob Falken
Quoted 20 lines · Eric Dumazet
Quoted 10 lines · Bob Falken

The only reason why i give information about 2.6.36.4 is that its the only latest kernel that was functioning properly.
i.e kernel >= 2.6.37 is not woking. so its a bisecting of the kernel versions to help a coder see when/where the isse was implemented in the kernel.

I do not need a backport patch for an old kernel, I generally only need the issue looked into and get fixed so that I dont have to use an old kernel. :)

Quoted 5 lines

I have no issue reproducing the issue on the recent kernels. however i have not tried the GIT kernel.

I restarted the server just a moment ago. i will install and run dropwatch and provide feedback in about 17hours.

You said that "cat /proc/net/ip_mr_cache" gives nothing at all after 2^32 packets ?

Quoted 7 lines
Quoted 2 lines

Thats a bit scary ... maybe a 32bit refcnt overflow, because of some imbalance...

That's my thought, too. :/

The ipmr.c rcu conversion happend in 2.6.37.

Just as a follow-up. You could verify if we actually have a refcount overflow if you try a grsec kernel with PAX_REFCOUNT enabled. If a refcoount overflow happens, dmesg should show then.

Greetings,

Hannes