OOPS with nf_conntrack_ipv6, probably fragmented UDPv6
Subsystems (3)
Hi,
I've hit another kernel oops with 2.6.20-rc3 on i386 platform. It is reproducible, as soon as I load nf_conntrack_ipv6 and try to send something large (scp or so) inside an OpenVPN tunnel on my client (patched with UDPv6 transport) the router (another box) OOPSes.
tcpdump suggests the problem appears as soon as my client sends fragmented UDPv6 packets towards the destination. It does not happen when nf_conntrack_ipv6 is not loaded. This is the OOPS as dumped from the serial console:
heimdall login: Oops: 0000 [#1]
Modules linked in: sit sch_red sch_htb pppoe pppox ppp_generic slhc xt_CLASSIFY ipt_TOS xt_length ipt_tos ipt_TCPMSS xt_tcpudp ipt_MASQUERADE xt_state iptable_mangle iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 ip_tables x_tables nf_conntrack_ipv6 nf_conntrack nfnetlink CPU: 0 EIP: 0060:[<00000001>] Not tainted VLI EFLAGS: 00010246 (2.6.20-rc3 #2)
EIP is at 0x1
eax: cd215bc0 ebx: cd1f3160 ecx: cc59002a edx: cd215bc0
esi: cd215bc0 edi: cd215bc0 ebp: 00000000 esp: c030bd3c
ds: 007b es: 007b ss: 0068
Process swapper (pid: 0, ti=c030a000 task=c02e93a0 task.ti=c030a000)
Stack: c0212cc4 00000004 cc83f160 cd2130c0 cd215bc0 cd2130c0 cd215bc0 c021734b
c030bdb4 c0307a60 0000000a cceee800 cceee800 cd215bc0 cd1f3160
00000000
c021896b c0307a60 cd215bc0 cd215bc0 cceee800 cd1f3160 c025f1c6
00000000
Call Trace:
[<c0212cc4>] __kfree_skb+0x84/0xe0
[<c021734b>] dev_hard_start_xmit+0x1bb/0x1d0
[<c021896b>] dev_queue_xmit+0x11b/0x1b0
[<c025f1c6>] ip6_output2+0x276/0x2b0
[<c025ed30>] ip6_output_finish+0x0/0xf0
[<c025fc0a>] ip6_output+0x90a/0x940
[<c013e9e5>] cache_alloc_refill+0x2c5/0x3f0
[<c0212eed>] pskb_expand_head+0xdd/0x130
[<c02608d5>] ip6_forward+0x465/0x4b0
[<c02618c6>] ip6_rcv_finish+0x16/0x30
[<ce81a056>] nf_ct_frag6_output+0x86/0xb0 [nf_conntrack_ipv6]
[<c02618b0>] ip6_rcv_finish+0x0/0x30
[<ce81911b>] ipv6_defrag+0x3b/0x50 [nf_conntrack_ipv6]
[<c02618b0>] ip6_rcv_finish+0x0/0x30
[<c022c618>] nf_iterate+0x38/0x70
[<c02618b0>] ip6_rcv_finish+0x0/0x30
[<c022c75d>] nf_hook_slow+0x4d/0xc0
[<c02618b0>] ip6_rcv_finish+0x0/0x30
[<c0261ac0>] ipv6_rcv+0x1e0/0x250
[<c02618b0>] ip6_rcv_finish+0x0/0x30
[<c0217068>] netif_receive_skb+0x1a8/0x200
[<c021868e>] process_backlog+0x6e/0xe0
[<c0218752>] net_rx_action+0x52/0xd0
[<c0113885>] __do_softirq+0x35/0x80
[<c01138f2>] do_softirq+0x22/0x30
[<c010453e>] do_IRQ+0x5e/0x70
[<c0102b33>] common_interrupt+0x23/0x30
[<c0101820>] default_idle+0x0/0x40
[<c0101847>] default_idle+0x27/0x40
[<c0101017>] cpu_idle+0x37/0x50
[<c030c676>] start_kernel+0x266/0x270
[<c030c200>] unknown_bootoption+0x0/0x210
=======================
Code: Bad EIP value.EIP: [<00000001>] 0x1 SS:ESP 0068:c030bd3c
<0>Kernel panic - not syncing: Fatal exception in interrupt <0>Rebooting in 20 seconds..<4>atkbd.c: Spurious ACK on
isa0060/serio0. Some program might be trying access hardware directly.
Do you need more information?
Regards,
Bernhard
Quoted 67 lines · On Thu, 04 Jan 2007 17:58:23 +0100 Bernhard Schmidt <[email protected]> wrote:
Hi,
I've hit another kernel oops with 2.6.20-rc3 on i386 platform. It is reproducible, as soon as I load
nf_conntrack_ipv6and try to send something large (scp or so) inside an OpenVPN tunnel on my client (patched with UDPv6 transport) the router (another box) OOPSes.tcpdump suggests the problem appears as soon as my client sends fragmented UDPv6 packets towards the destination. It does not happen when
nf_conntrack_ipv6is not loaded. This is the OOPS as dumped from the serial console:heimdall login: Oops: 0000 [#1]
Modules linked in: sit sch_red sch_htb pppoe pppox ppp_generic slhc
xt_CLASSIFYipt_TOSxt_lengthipt_tosipt_TCPMSSxt_tcpudpipt_MASQUERADExt_stateiptable_mangleiptable_filter
iptable_natnf_natnf_conntrack_ipv4ip_tablesx_tablesnf_conntrack_ipv6nf_conntracknfnetlinkCPU: 0 EIP: 0060:[<00000001>] Not tainted VLI EFLAGS: 00010246 (2.6.20-rc3 #2)EIP is at
0x1
eax:cd215bc0ebx:cd1f3160ecx:cc59002aedx:cd215bc0
esi:cd215bc0edi:cd215bc0ebp: 00000000 esp:c030bd3c
ds: 007b es: 007b ss: 0068
Process swapper (pid: 0, ti=c030a000 task=c02e93a0 task.ti=c030a000)Stack: c0212cc4 00000004 cc83f160 cd2130c0 cd215bc0 cd2130c0 cd215bc0 c021734b c030bdb4 c0307a60 0000000a cceee800 cceee800 cd215bc0 cd1f3160 00000000 c021896b c0307a60 cd215bc0 cd215bc0 cceee800 cd1f3160 c025f1c6 00000000 Call Trace: [<c0212cc4>] __kfree_skb+0x84/0xe0 [<c021734b>] dev_hard_start_xmit+0x1bb/0x1d0 [<c021896b>] dev_queue_xmit+0x11b/0x1b0 [<c025f1c6>] ip6_output2+0x276/0x2b0 [<c025ed30>] ip6_output_finish+0x0/0xf0 [<c025fc0a>] ip6_output+0x90a/0x940 [<c013e9e5>] cache_alloc_refill+0x2c5/0x3f0 [<c0212eed>] pskb_expand_head+0xdd/0x130 [<c02608d5>] ip6_forward+0x465/0x4b0 [<c02618c6>] ip6_rcv_finish+0x16/0x30 [<ce81a056>] nf_ct_frag6_output+0x86/0xb0 [nf_conntrack_ipv6] [<c02618b0>] ip6_rcv_finish+0x0/0x30 [<ce81911b>] ipv6_defrag+0x3b/0x50 [nf_conntrack_ipv6] [<c02618b0>] ip6_rcv_finish+0x0/0x30 [<c022c618>] nf_iterate+0x38/0x70 [<c02618b0>] ip6_rcv_finish+0x0/0x30 [<c022c75d>] nf_hook_slow+0x4d/0xc0 [<c02618b0>] ip6_rcv_finish+0x0/0x30 [<c0261ac0>] ipv6_rcv+0x1e0/0x250 [<c02618b0>] ip6_rcv_finish+0x0/0x30 [<c0217068>] netif_receive_skb+0x1a8/0x200 [<c021868e>] process_backlog+0x6e/0xe0 [<c0218752>] net_rx_action+0x52/0xd0 [<c0113885>] __do_softirq+0x35/0x80 [<c01138f2>] do_softirq+0x22/0x30 [<c010453e>] do_IRQ+0x5e/0x70 [<c0102b33>] common_interrupt+0x23/0x30 [<c0101820>] default_idle+0x0/0x40 [<c0101847>] default_idle+0x27/0x40 [<c0101017>] cpu_idle+0x37/0x50 [<c030c676>] start_kernel+0x266/0x270 [<c030c200>] unknown_bootoption+0x0/0x210 ======================= Code: Bad EIP value.
Quoted 5 lines
EIP: [<00000001>]
0x1SS:ESP 0068:c030bd3c
<0>Kernel panic - not syncing: Fatal exception in interrupt <0>Rebooting in 20 seconds..<4>atkbd.c: Spurious ACK on
isa0060/serio0. Some program might be trying access hardware directly.
At a guess I'd say that skb->nfct->destroy has value 0x00000001. Not a good function address.
Presumably it is suppsoed to be zero...
Quoted 34 lines · Bernhard Schmidt
I've hit another kernel oops with 2.6.20-rc3 on i386 platform. It is reproducible, as soon as I load
nf_conntrack_ipv6and try to send something large (scp or so) inside an OpenVPN tunnel on my client (patched with UDPv6 transport) the router (another box) OOPSes.tcpdump suggests the problem appears as soon as my client sends fragmented UDPv6 packets towards the destination. It does not happen when
nf_conntrack_ipv6is not loaded. This is the OOPS as dumped from the serial console:heimdall login: Oops: 0000 [#1]
Modules linked in: sit sch_red sch_htb pppoe pppox ppp_generic slhc
xt_CLASSIFYipt_TOSxt_lengthipt_tosipt_TCPMSSxt_tcpudpipt_MASQUERADExt_stateiptable_mangleiptable_filter
iptable_natnf_natnf_conntrack_ipv4ip_tablesx_tablesnf_conntrack_ipv6nf_conntracknfnetlinkCPU: 0 EIP: 0060:[<00000001>] Not tainted VLI EFLAGS: 00010246 (2.6.20-rc3 #2)EIP is at
0x1
eax:cd215bc0ebx:cd1f3160ecx:cc59002aedx:cd215bc0
esi:cd215bc0edi:cd215bc0ebp: 00000000 esp:c030bd3c
ds: 007b es: 007b ss: 0068
Process swapper (pid: 0, ti=c030a000 task=c02e93a0 task.ti=c030a000)Stack: c0212cc4 00000004 cc83f160 cd2130c0 cd215bc0 cd2130c0 cd215bc0 c021734b c030bdb4 c0307a60 0000000a cceee800 cceee800 cd215bc0 cd1f3160 00000000 c021896b c0307a60 cd215bc0 cd215bc0 cceee800 cd1f3160 c025f1c6 00000000 Call Trace: [<c0212cc4>] __kfree_skb+0x84/0xe0 [<c021734b>] dev_hard_start_xmit+0x1bb/0x1d0 [<c021896b>] dev_queue_xmit+0x11b/0x1b0
Quoted 8 lines
[<c025f1c6>] ip6_output2+0x276/0x2b0 [<c025ed30>] ip6_output_finish+0x0/0xf0 [<c025fc0a>] ip6_output+0x90a/0x940 [<c013e9e5>] cache_alloc_refill+0x2c5/0x3f0 [<c0212eed>] pskb_expand_head+0xdd/0x130 [<c02608d5>] ip6_forward+0x465/0x4b0 [<c02618c6>] ip6_rcv_finish+0x16/0x30 [<ce81a056>] nf_ct_frag6_output+0x86/0xb0 [nf_conntrack_ipv6]
Does this patch help?
[NETFILTER]: nf_conntrack_ipv6: fix crash when handling fragments
When IPv6 connection tracking splits up a defragmented packet into its original fragments, the packets are taken from a list and are passed to the network stack with skb->next still set. This causes dev_hard_start_xmit to treat them as GSO fragments, resulting in a use after free when connection tracking handles the next fragment.
---
commit f7c932bb23afe7873586fb9bad82718e3f16a3af
tree c2e18743b831f43fa7859d29ea9b2a622c58da99
parent fe6df90eb909a84593b6902e6e4f802687bc4564
author Patrick McHardy <[email protected]> Tue, 09 Jan 2007 10:35:28 +0100 committer Patrick McHardy <[email protected]> Tue, 09 Jan 2007 10:35:28 +0100
net/ipv6/netfilter/nf_conntrack_reasm.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
net/ipv6/netfilter/nf_conntrack_reasm.c +2 -0
@@ -835,6 +835,8 @@ void nf_ct_frag6_output(unsigned int hoo s->nfct_reasm = skb; s2 = s->next; + s->next =NULL ; +NF_HOOK_THRESH (PF_INET6 , hooknum, s, in, out, okfn,NF_IP6_PRI_CONNTRACK_DEFRAG +1 ); s = s2;
Quoted 34 lines · Bernhard Schmidt
I've hit another kernel oops with 2.6.20-rc3 on i386 platform. It is reproducible, as soon as I load
nf_conntrack_ipv6and try to send something large (scp or so) inside an OpenVPN tunnel on my client (patched with UDPv6 transport) the router (another box) OOPSes.tcpdump suggests the problem appears as soon as my client sends fragmented UDPv6 packets towards the destination. It does not happen when
nf_conntrack_ipv6is not loaded. This is the OOPS as dumped from the serial console:heimdall login: Oops: 0000 [#1]
Modules linked in: sit sch_red sch_htb pppoe pppox ppp_generic slhc
xt_CLASSIFYipt_TOSxt_lengthipt_tosipt_TCPMSSxt_tcpudpipt_MASQUERADExt_stateiptable_mangleiptable_filter
iptable_natnf_natnf_conntrack_ipv4ip_tablesx_tablesnf_conntrack_ipv6nf_conntracknfnetlinkCPU: 0 EIP: 0060:[<00000001>] Not tainted VLI EFLAGS: 00010246 (2.6.20-rc3 #2)EIP is at
0x1
eax:cd215bc0ebx:cd1f3160ecx:cc59002aedx:cd215bc0
esi:cd215bc0edi:cd215bc0ebp: 00000000 esp:c030bd3c
ds: 007b es: 007b ss: 0068
Process swapper (pid: 0, ti=c030a000 task=c02e93a0 task.ti=c030a000)Stack: c0212cc4 00000004 cc83f160 cd2130c0 cd215bc0 cd2130c0 cd215bc0 c021734b c030bdb4 c0307a60 0000000a cceee800 cceee800 cd215bc0 cd1f3160 00000000 c021896b c0307a60 cd215bc0 cd215bc0 cceee800 cd1f3160 c025f1c6 00000000 Call Trace: [<c0212cc4>] __kfree_skb+0x84/0xe0 [<c021734b>] dev_hard_start_xmit+0x1bb/0x1d0 [<c021896b>] dev_queue_xmit+0x11b/0x1b0
Quoted 8 lines
[<c025f1c6>] ip6_output2+0x276/0x2b0 [<c025ed30>] ip6_output_finish+0x0/0xf0 [<c025fc0a>] ip6_output+0x90a/0x940 [<c013e9e5>] cache_alloc_refill+0x2c5/0x3f0 [<c0212eed>] pskb_expand_head+0xdd/0x130 [<c02608d5>] ip6_forward+0x465/0x4b0 [<c02618c6>] ip6_rcv_finish+0x16/0x30 [<ce81a056>] nf_ct_frag6_output+0x86/0xb0 [nf_conntrack_ipv6]
Does this patch help?
[NETFILTER]: nf_conntrack_ipv6: fix crash when handling fragments
When IPv6 connection tracking splits up a defragmented packet into its original fragments, the packets are taken from a list and are passed to the network stack with skb->next still set. This causes dev_hard_start_xmit to treat them as GSO fragments, resulting in a use after free when connection tracking handles the next fragment.
---
commit f7c932bb23afe7873586fb9bad82718e3f16a3af
tree c2e18743b831f43fa7859d29ea9b2a622c58da99
parent fe6df90eb909a84593b6902e6e4f802687bc4564
author Patrick McHardy <[email protected]> Tue, 09 Jan 2007 10:35:28 +0100 committer Patrick McHardy <[email protected]> Tue, 09 Jan 2007 10:35:28 +0100
net/ipv6/netfilter/nf_conntrack_reasm.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
net/ipv6/netfilter/nf_conntrack_reasm.c +2 -0
@@ -835,6 +835,8 @@ void nf_ct_frag6_output(unsigned int hoo s->nfct_reasm = skb; s2 = s->next; + s->next =NULL ; +NF_HOOK_THRESH (PF_INET6 , hooknum, s, in, out, okfn,NF_IP6_PRI_CONNTRACK_DEFRAG +1 ); s = s2;
Quoted 10 lines · Patrick McHardy
Quoted 9 lines
I've hit another kernel oops with 2.6.20-rc3 on i386 platform. It is reproducible, as soon as I load
nf_conntrack_ipv6and try to send something large (scp or so) inside an OpenVPN tunnel on my client (patched with UDPv6 transport) the router (another box) OOPSes.tcpdump suggests the problem appears as soon as my client sends fragmented UDPv6 packets towards the destination. It does not happen when
nf_conntrack_ipv6is not loaded. This is the OOPS as dumped from the serial console:Does this patch help?
Yes, seems to be working fine.
Can you tell since when this bug is in the kernel?
Regards,
Bernhard
Quoted 12 lines · Bernhard Schmidt
Quoted 10 lines · Patrick McHardy
Quoted 9 lines
I've hit another kernel oops with 2.6.20-rc3 on i386 platform. It is reproducible, as soon as I load
nf_conntrack_ipv6and try to send something large (scp or so) inside an OpenVPN tunnel on my client (patched with UDPv6 transport) the router (another box) OOPSes.tcpdump suggests the problem appears as soon as my client sends fragmented UDPv6 packets towards the destination. It does not happen when
nf_conntrack_ipv6is not loaded. This is the OOPS as dumped from the serial console:
Quoted 4 lines
Quoted 1 line
Does this patch help?
Yes, seems to be working fine.
Thanks, I'll send it upstream tonight.
Quoted 1 line
Can you tell since when this bug is in the kernel?
The real bug is in nf_conntrack and probably has been there since the beginning (which I think is about 1.5 years ago). It blows up in combination with the GSO code, which I believe was added in 2.6.18-rc, but it might have caused troubles somewhere else before.