Skip to contentSkip to conversation

3.3-rc snmp6 panic

DISCUSSION Dave Jones 3 participants 5 messages Mar 15, 2012 – Mar 16, 2012

Subsystems (3)
lore
Share & tools
lore

Dave Jones

author
#0 lore raw

I've been seeing an occasional panic when I shut down my router since I put 3.3 on there. It happens about once a week, always during shutdown. It wedges before I can get a good capture of the trace. This is the best I've captured so far.. https://twitpic.com/8wh5l5
(apologies in advance for blurriness)

Quoted 1 line

From comparing the Code: line, and the objdump output, the code it's

choking on in mld_sendpack seems to be a skb_dst macro in the NF_HOOK..

        err = NF_HOOK(NFPROTO_IPV6, NF_INET_LOCAL_OUT, skb, NULL, skb->dev,
    2dd9:       4c 8b 43 20             mov    0x20(%rbx),%r8
    2ddd:       e9 00 00 00 00          jmpq   2de2 <mld_sendpack+0x1b2>
static inline struct dst_entry *skb_dst(const struct sk_buff *skb)
{
        /* If refdst was not refcounted, check we still are in a 
         * rcu_read_lock section
         */
        WARN_ON((skb->_skb_refdst & SKB_DST_NOREF) &&
    2de2:       48 8b 43 58             mov    0x58(%rbx),%rax
    2de6:       a8 01                   test   $0x1,%al
    2de8:       0f 85 d2 01 00 00       jne    2fc0 <mld_sendpack+0x390>
                !rcu_read_lock_held() &&
                !rcu_read_lock_bh_held());
        return (struct dst_entry *)(skb->_skb_refdst & SKB_DST_PTRMASK);
    2dee:       48 83 e0 fe             and    $0xfffffffffffffffe,%rax
    2df2:       48 89 df                mov    %rbx,%rdi
    2df5:       ff 50 58                callq  *0x58(%rax)           <-----  BOOM

This machine is running an snmpd, for my mrtg setup, so the teardown of that service is probably what's triggering it. But I can start/stop it in a loop as much as I want without it happening, so maybe the kernel needs to accumulate some state from it for a while first ?

Anyone have any ideas what's happening here ?

Dave

#1 lore raw
Quoted 30 lines · Dave Jones

I've been seeing an occasional panic when I shut down my router since I put 3.3 on there. It happens about once a week, always during shutdown. It wedges before I can get a good capture of the trace. This is the best I've captured so far.. https://twitpic.com/8wh5l5
(apologies in advance for blurriness)

From comparing the Code: line, and the objdump output, the code it's choking on in mld_sendpack seems to be a skb_dst macro in the NF_HOOK..

        err = NF_HOOK(NFPROTO_IPV6, NF_INET_LOCAL_OUT, skb, NULL, skb->dev,
    2dd9:       4c 8b 43 20             mov    0x20(%rbx),%r8
    2ddd:       e9 00 00 00 00          jmpq   2de2 <mld_sendpack+0x1b2>
static inline struct dst_entry *skb_dst(const struct sk_buff *skb)
{
        /* If refdst was not refcounted, check we still are in a 
         * rcu_read_lock section
         */
        WARN_ON((skb->_skb_refdst & SKB_DST_NOREF) &&
    2de2:       48 8b 43 58             mov    0x58(%rbx),%rax
    2de6:       a8 01                   test   $0x1,%al
    2de8:       0f 85 d2 01 00 00       jne    2fc0 <mld_sendpack+0x390>
                !rcu_read_lock_held() &&
                !rcu_read_lock_bh_held());
        return (struct dst_entry *)(skb->_skb_refdst & SKB_DST_PTRMASK);
    2dee:       48 83 e0 fe             and    $0xfffffffffffffffe,%rax
    2df2:       48 89 df                mov    %rbx,%rdi
    2df5:       ff 50 58                callq  *0x58(%rax)           <-----  BOOM
Quoted 7 lines

This machine is running an snmpd, for my mrtg setup, so the teardown of that service is probably what's triggering it. But I can start/stop it in a loop as much as I want without it happening, so maybe the kernel needs to accumulate some state from it for a while first ?

Anyone have any ideas what's happening here ?

Hmm, please try :

net/ipv6/route.c +1 -1
index 8c2e3ab..22b7664 100644
@@ -1077,7 +1077,7 @@ struct dst_entry *icmp6_dst_alloc(struct net_device *dev,
 	struct net *net = dev_net(dev);
 
 	if (unlikely(!idev))
-		return NULL;
+		return ERR_PTR(-ENODEV);
 
 	rt = ip6_dst_alloc(&net->ipv6.ip6_dst_ops, dev, 0);
 	if (unlikely(!rt)) {
#2 lore raw
Quoted 36 lines · Eric Dumazet
Quoted 35 lines · Dave Jones

I've been seeing an occasional panic when I shut down my router since I put 3.3 on there. It happens about once a week, always during shutdown. It wedges before I can get a good capture of the trace. This is the best I've captured so far.. https://twitpic.com/8wh5l5
(apologies in advance for blurriness)

From comparing the Code: line, and the objdump output, the code it's choking on in mld_sendpack seems to be a skb_dst macro in the NF_HOOK..

        err = NF_HOOK(NFPROTO_IPV6, NF_INET_LOCAL_OUT, skb, NULL, skb->dev,
    2dd9:       4c 8b 43 20             mov    0x20(%rbx),%r8
    2ddd:       e9 00 00 00 00          jmpq   2de2 <mld_sendpack+0x1b2>
static inline struct dst_entry *skb_dst(const struct sk_buff *skb)
{
        /* If refdst was not refcounted, check we still are in a 
         * rcu_read_lock section
         */
        WARN_ON((skb->_skb_refdst & SKB_DST_NOREF) &&
    2de2:       48 8b 43 58             mov    0x58(%rbx),%rax
    2de6:       a8 01                   test   $0x1,%al
    2de8:       0f 85 d2 01 00 00       jne    2fc0 <mld_sendpack+0x390>
                !rcu_read_lock_held() &&
                !rcu_read_lock_bh_held());
        return (struct dst_entry *)(skb->_skb_refdst & SKB_DST_PTRMASK);
    2dee:       48 83 e0 fe             and    $0xfffffffffffffffe,%rax
    2df2:       48 89 df                mov    %rbx,%rdi
    2df5:       ff 50 58                callq  *0x58(%rax)           <-----  BOOM

This machine is running an snmpd, for my mrtg setup, so the teardown of that service is probably what's triggering it. But I can start/stop it in a loop as much as I want without it happening, so maybe the kernel needs to accumulate some state from it for a while first ?

Quoted 4 lines
Quoted 2 lines

Anyone have any ideas what's happening here ?

Hmm, please try :

Since patch is trivial, here is the official submission

Thanks !

[PATCH] ipv6: fix icmp6_dst_alloc()

commit 87a115783 ( ipv6: Move xfrm_lookup() call down into icmp6_dst_alloc().) forgot to convert one error path, leading to crashes in mld_sendpack()

Many thanks to Dave Jones for providing a very complete bug report.

---
 net/ipv6/route.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/ipv6/route.c +1 -1
index 8c2e3ab..22b7664 100644
@@ -1077,7 +1077,7 @@ struct dst_entry *icmp6_dst_alloc(struct net_device *dev,
 	struct net *net = dev_net(dev);
 
 	if (unlikely(!idev))
-		return NULL;
+		return ERR_PTR(-ENODEV);
 
 	rt = ip6_dst_alloc(&net->ipv6.ip6_dst_ops, dev, 0);
 	if (unlikely(!rt)) {
author
#4 lore raw
In reply to Eric Dumazet
Quoted 8 lines · Eric Dumazet
Quoted 6 lines

This machine is running an snmpd, for my mrtg setup, so the teardown of that service is probably what's triggering it. But I can start/stop it in a loop as much as I want without it happening, so maybe the kernel needs to accumulate some state from it for a while first ?

Anyone have any ideas what's happening here ?

Hmm, please try :

Will do. Given I can't reproduce it quickly though, it might be a while before I can provide feedback on whether it does the trick :)

thanks,

Dave

maintainer
#3 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Thu, 15 Mar 2012 00:13:11 -0700

Quoted 10 lines

[PATCH] ipv6: fix icmp6_dst_alloc()

commit 87a115783 ( ipv6: Move xfrm_lookup() call down into icmp6_dst_alloc().) forgot to convert one error path, leading to crashes in mld_sendpack()

Many thanks to Dave Jones for providing a very complete bug report.

Applied and queued up for -stable, thanks.