Skip to contentSkip to conversation

gre/gro oops in skb_gro_receive+0x118/0x453

bisected In review Timo Teras 3 participants 13 messages May 16, 2014

Subsystems (2)
lore
Share & tools
lore

Timo Teras

author
#0 lore raw

The oops happens when forwarding traffic between ethX <-> gre. Where the GRE tunnel is an NBMA tunnel and the GRE traffic is IPsec'ed in transport mode. It seems that locally originating traffic to gre is not affected.

The oops also goes away if GRO is turned off for gre tunnel device.

I have bisected this always reproducible oops to commit:

8a29111c7ca68d928dfab58636f3f6acf0ac04f7 is the first bad commit

commit 8a29111c7ca68d928dfab58636f3f6acf0ac04f7
Author: Eric Dumazet <[email protected]>
Date: Oct 8 09:02:23 2013 -0700
net: gro: allow to build full sized skb

This oops backtrace is from vanilla 3.14.4 kernel, but it is identical up to the offending commit.

[  286.927713] BUG: unable to handle kernel paging request at 2b90bdc8
[  286.930813] IP: [<c120fc88>] skb_gro_receive+0x118/0x453
[  286.930813] *pde = 00000000 
[  286.930813] Oops: 0000 [#1] SMP 
[  286.930813] Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport

deflate ctr twofish_generic twofish_i586 twofish_common camellia_generic serpent_sse2_i586 xts lrw gf128mul serpent_generic glue_helper ablk_helper cryptd blowfish_generic blowfish_common cast5_generic cast_common des_generic cbc cmac xcbc rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre ip_tunnel nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_raw ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT xt_helper nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables nf_conntrack_ftp nf_conntrack_sip xt_CT ip6table_raw xt_LOG xt_limit xt_policy xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_recent xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ipv6 af_packet mousedev via_rng rng_core via_cputemp hwmon hwmon_vid padlock_aes padlock_sha serio_raw psmouse pcspkr shpchp i2c_viapro i2c_core via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169 firmware_class mii fan evdev parport_pc parport thermal button acpi_cpufreq processor nls_utf8 nls_cp437 vfat fat sata_via ehci_pci ehci_hcd uhci_hcd pata_via pata_acpi ata_generic libata usb_storage usbcore usb_common sd_mod scsi_mod crc_t10dif crct10dif_common squashfs loop

[  286.930813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.4 #1-fragbisect
[  286.930813] Hardware name:    /CN700-8237, BIOS 6.00 PG 08/06/2008
[  286.930813] task: c13e8930 ti: f6408000 task.ti: c13de000
[  286.930813] EIP: 0060:[<c120fc88>] EFLAGS: 00210202 CPU: 0
[  286.930813] EIP is at skb_gro_receive+0x118/0x453
[  286.930813] EAX: 2b90bdc8 EBX: e9f669c0 ECX: 00000034 EDX: 00000596
[  286.930813] ESI: e9f669c0 EDI: f648f134 EBP: f6409eb4 ESP: f6409e7c
[  286.930813]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  286.930813] CR0: 8005003b CR2: 2b90bdc8 CR3: 2a731000 CR4: 00000690
[  286.930813] Stack:
[  286.930813]  c120df4b e9f4f796 00000046 00000000 f648f134 c013a600 0000007a e9f4fd40
[  286.930813]  2b90bdc8 00000034 e9f663c0 e9f663c0 e9f669c0 f648f134 f6409ee0 c125cc6f
[  286.930813]  00000000 00000020 00000046 2e001080 e9f4ed96 00000562 e9f663c0 e9f4f782
[  286.930813] Call Trace:
[  286.930813]  [<c120df4b>] ? csum_partial_ext+0x16/0x18
[  286.930813]  [<c125cc6f>] tcp_gro_receive+0x1a8/0x218
[  286.930813]  [<c125cd9f>] tcp4_gro_receive+0xc0/0xc8
[  286.930813]  [<c1268439>] inet_gro_receive+0x1c5/0x1df
[  286.930813]  [<c1219a15>] dev_gro_receive+0x231/0x393
[  286.930813]  [<f80e006b>] ? rh_timer_func+0x8/0xa [usbcore]
[  286.930813]  [<c1219c1c>] napi_gro_receive+0xb/0x5e
[  286.930813]  [<f8650056>] gro_cell_poll+0x56/0x73 [ip_tunnel]
[  286.930813]  [<c121a2da>] net_rx_action+0xb0/0x14d
[  286.930813]  [<c1030d43>] __do_softirq+0xb8/0x1a5
[  286.930813]  [<c1030c8b>] ? cpu_callback+0xec/0xec
[  286.930813]  <IRQ> 
[  286.930813]  [<c1030fa5>] ? irq_exit+0x44/0x81
[  286.930813]  [<c1002da0>] ? do_IRQ+0x9f/0xb3
[  286.930813]  [<c106e642>] ? clockevents_notify+0x10f/0x116
[  286.930813]  [<c1298d33>] ? common_interrupt+0x33/0x40
[  286.930813]  [<c11f0292>] ? cpuidle_enter_state+0x39/0xa3
[  286.930813]  [<c11f03a5>] ? cpuidle_idle_call+0xa9/0xe6
[  286.930813]  [<c1007f8f>] ? arch_cpu_idle+0x8/0x1c
[  286.930813]  [<c1060542>] ? cpu_startup_entry+0xf3/0x159
[  286.930813]  [<c128b36b>] ? rest_init+0x5d/0x5f
[  286.930813]  [<c1415a17>] ? start_kernel+0x3b2/0x3b8
[  286.930813]  [<c141549b>] ? repair_env_string+0x51/0x51
[  286.930813]  [<c14152d0>] ? i386_start_kernel+0x7a/0x7e
[  286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29 d1 89 8e b4 00 00 00
e9 09 03 00 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00
<8a> 00 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05
[  286.930813] EIP: [<c120fc88>] skb_gro_receive+0x118/0x453 SS:ESP 0068:f6409e7c
[  286.930813] CR2: 000000002b90bdc8
[  286.930813] ---[ end trace ac04411e60d3534c ]---
[  286.930813] Kernel panic - not syncing: Fatal exception in interrupt
[  286.930813] Kernel Offset: 0x0 from 0xc1000000 (relocation range: 0xc0000000-0xf7ffdfff)
#1 lore raw
Quoted 88 lines · Timo Teras

The oops happens when forwarding traffic between ethX <-> gre. Where the GRE tunnel is an NBMA tunnel and the GRE traffic is IPsec'ed in transport mode. It seems that locally originating traffic to gre is not affected.

The oops also goes away if GRO is turned off for gre tunnel device.

I have bisected this always reproducible oops to commit:

8a29111c7ca68d928dfab58636f3f6acf0ac04f7 is the first bad commit

commit 8a29111c7ca68d928dfab58636f3f6acf0ac04f7
Author: Eric Dumazet <[email protected]>
Date: Oct 8 09:02:23 2013 -0700
net: gro: allow to build full sized skb

This oops backtrace is from vanilla 3.14.4 kernel, but it is identical up to the offending commit.

[  286.927713] BUG: unable to handle kernel paging request at 2b90bdc8
[  286.930813] IP: [<c120fc88>] skb_gro_receive+0x118/0x453
[  286.930813] *pde = 00000000 
[  286.930813] Oops: 0000 [#1] SMP 
[  286.930813] Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport

deflate ctr twofish_generic twofish_i586 twofish_common camellia_generic serpent_sse2_i586 xts lrw gf128mul serpent_generic glue_helper ablk_helper cryptd blowfish_generic blowfish_common cast5_generic cast_common des_generic cbc cmac xcbc rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre ip_tunnel nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_raw ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT xt_helper nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables nf_conntrack_ftp nf_conntrack_sip xt_CT ip6table_raw xt_LOG xt_limit xt_policy xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_recent xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ipv6 af_packet mousedev via_rng rng_core via_cputemp hwmon hwmon_vid padlock_aes padlock_sha serio_raw psmouse pcspkr shpchp i2c_viapro i2c_core via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169 firmware_class mii fan evdev parport_pc parport thermal button acpi_cpufreq processor nls_utf8 nls_cp437 vfat fat sata_via ehci_pci ehci_hcd uhci_hcd pata_via pata_acpi ata_generic libata usb_storage usbcore usb_common sd_mod scsi_mod crc_t10dif crct10dif_common squashfs loop

[  286.930813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.4 #1-fragbisect
[  286.930813] Hardware name:    /CN700-8237, BIOS 6.00 PG 08/06/2008
[  286.930813] task: c13e8930 ti: f6408000 task.ti: c13de000
[  286.930813] EIP: 0060:[<c120fc88>] EFLAGS: 00210202 CPU: 0
[  286.930813] EIP is at skb_gro_receive+0x118/0x453
[  286.930813] EAX: 2b90bdc8 EBX: e9f669c0 ECX: 00000034 EDX: 00000596
[  286.930813] ESI: e9f669c0 EDI: f648f134 EBP: f6409eb4 ESP: f6409e7c
[  286.930813]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  286.930813] CR0: 8005003b CR2: 2b90bdc8 CR3: 2a731000 CR4: 00000690
[  286.930813] Stack:
[  286.930813]  c120df4b e9f4f796 00000046 00000000 f648f134 c013a600 0000007a e9f4fd40
[  286.930813]  2b90bdc8 00000034 e9f663c0 e9f663c0 e9f669c0 f648f134 f6409ee0 c125cc6f
[  286.930813]  00000000 00000020 00000046 2e001080 e9f4ed96 00000562 e9f663c0 e9f4f782
[  286.930813] Call Trace:
[  286.930813]  [<c120df4b>] ? csum_partial_ext+0x16/0x18
[  286.930813]  [<c125cc6f>] tcp_gro_receive+0x1a8/0x218
[  286.930813]  [<c125cd9f>] tcp4_gro_receive+0xc0/0xc8
[  286.930813]  [<c1268439>] inet_gro_receive+0x1c5/0x1df
[  286.930813]  [<c1219a15>] dev_gro_receive+0x231/0x393
[  286.930813]  [<f80e006b>] ? rh_timer_func+0x8/0xa [usbcore]
[  286.930813]  [<c1219c1c>] napi_gro_receive+0xb/0x5e
[  286.930813]  [<f8650056>] gro_cell_poll+0x56/0x73 [ip_tunnel]
[  286.930813]  [<c121a2da>] net_rx_action+0xb0/0x14d
[  286.930813]  [<c1030d43>] __do_softirq+0xb8/0x1a5
[  286.930813]  [<c1030c8b>] ? cpu_callback+0xec/0xec
[  286.930813]  <IRQ> 
[  286.930813]  [<c1030fa5>] ? irq_exit+0x44/0x81
[  286.930813]  [<c1002da0>] ? do_IRQ+0x9f/0xb3
[  286.930813]  [<c106e642>] ? clockevents_notify+0x10f/0x116
[  286.930813]  [<c1298d33>] ? common_interrupt+0x33/0x40
[  286.930813]  [<c11f0292>] ? cpuidle_enter_state+0x39/0xa3
[  286.930813]  [<c11f03a5>] ? cpuidle_idle_call+0xa9/0xe6
[  286.930813]  [<c1007f8f>] ? arch_cpu_idle+0x8/0x1c
[  286.930813]  [<c1060542>] ? cpu_startup_entry+0xf3/0x159
[  286.930813]  [<c128b36b>] ? rest_init+0x5d/0x5f
[  286.930813]  [<c1415a17>] ? start_kernel+0x3b2/0x3b8
[  286.930813]  [<c141549b>] ? repair_env_string+0x51/0x51
[  286.930813]  [<c14152d0>] ? i386_start_kernel+0x7a/0x7e
[  286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29 d1 89 8e b4 00 00 00
e9 09 03 00 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00
<8a> 00 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05
[  286.930813] EIP: [<c120fc88>] skb_gro_receive+0x118/0x453 SS:ESP 0068:f6409e7c
[  286.930813] CR2: 000000002b90bdc8
[  286.930813] ---[ end trace ac04411e60d3534c ]---
[  286.930813] Kernel panic - not syncing: Fatal exception in interrupt
[  286.930813] Kernel Offset: 0x0 from 0xc1000000 (relocation range: 0xc0000000-0xf7ffdfff)
Quoted 1 line

--

Unffortunately bisecting wont help a lot, as we knew the commit was faulty.

Please check 9d8506cc2d7ea1f911c72c100193a3677f6668c3

author
#2 lore raw
Quoted 94 lines · On Fri, 16 May 2014 05:59:17 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 89 lines · Timo Teras

The oops happens when forwarding traffic between ethX <-> gre. Where the GRE tunnel is an NBMA tunnel and the GRE traffic is IPsec'ed in transport mode. It seems that locally originating traffic to gre is not affected.

The oops also goes away if GRO is turned off for gre tunnel device.

I have bisected this always reproducible oops to commit:

8a29111c7ca68d928dfab58636f3f6acf0ac04f7 is the first bad commit

commit 8a29111c7ca68d928dfab58636f3f6acf0ac04f7
Author: Eric Dumazet <[email protected]>
Date: Oct 8 09:02:23 2013 -0700
net: gro: allow to build full sized skb

This oops backtrace is from vanilla 3.14.4 kernel, but it is identical up to the offending commit.

[  286.927713] BUG: unable to handle kernel paging request at
2b90bdc8 [  286.930813] IP: [<c120fc88>] skb_gro_receive+0x118/0x453
[  286.930813] *pde = 00000000 
[  286.930813] Oops: 0000 [#1] SMP 
[  286.930813] Modules linked in: sha1_generic authenc esp4

xfrm4_mode_transport deflate ctr twofish_generic twofish_i586 twofish_common camellia_generic serpent_sse2_i586 xts lrw gf128mul serpent_generic glue_helper ablk_helper cryptd blowfish_generic blowfish_common cast5_generic cast_common des_generic cbc cmac xcbc rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre ip_tunnel nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_raw ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT xt_helper nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables nf_conntrack_ftp nf_conntrack_sip xt_CT ip6table_raw xt_LOG xt_limit xt_policy xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_recent xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ipv6 af_packet mousedev via_rng rng_core via_cputemp hwmon hwmon_vid padlock_aes padlock_sha serio_raw psmouse pcspkr shpchp i2c_viapro i2c_core via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169 firmware_class mii fan evdev parport_pc parport thermal button acpi_cpufreq processor nls_utf8 nls_cp437 vfat fat sata_via ehci_pci ehci_hcd uhci_hcd pata_via pata_acpi ata_generic libata usb_storage usbcore usb_common sd_mod scsi_mod crc_t10dif crct10dif_common squashfs loop [ 286.930813] CPU: 0 PID: 0 Comm:
swapper/0 Not tainted 3.14.4 #1-fragbisect [ 286.930813] Hardware
name: /CN700-8237, BIOS 6.00 PG 08/06/2008 [ 286.930813] task:
c13e8930 ti: f6408000 task.ti: c13de000 [ 286.930813] EIP:
0060:[<c120fc88>] EFLAGS: 00210202 CPU: 0 [ 286.930813] EIP is at

skb_gro_receive+0x118/0x453 [  286.930813] EAX: 2b90bdc8 EBX:
e9f669c0 ECX: 00000034 EDX: 00000596 [  286.930813] ESI: e9f669c0
EDI: f648f134 EBP: f6409eb4 ESP: f6409e7c [  286.930813]  DS: 007b
ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [  286.930813] CR0: 8005003b
CR2: 2b90bdc8 CR3: 2a731000 CR4: 00000690 [  286.930813] Stack:
[  286.930813]  c120df4b e9f4f796 00000046 00000000 f648f134
c013a600 0000007a e9f4fd40 [  286.930813]  2b90bdc8 00000034
e9f663c0 e9f663c0 e9f669c0 f648f134 f6409ee0 c125cc6f
[  286.930813]  00000000 00000020 00000046 2e001080 e9f4ed96
00000562 e9f663c0 e9f4f782 [  286.930813] Call Trace:
[  286.930813]  [<c120df4b>] ? csum_partial_ext+0x16/0x18
[  286.930813]  [<c125cc6f>] tcp_gro_receive+0x1a8/0x218
[  286.930813]  [<c125cd9f>] tcp4_gro_receive+0xc0/0xc8
[  286.930813]  [<c1268439>] inet_gro_receive+0x1c5/0x1df
[  286.930813]  [<c1219a15>] dev_gro_receive+0x231/0x393
[  286.930813]  [<f80e006b>] ? rh_timer_func+0x8/0xa [usbcore]
[  286.930813]  [<c1219c1c>] napi_gro_receive+0xb/0x5e
[  286.930813]  [<f8650056>] gro_cell_poll+0x56/0x73 [ip_tunnel]
[  286.930813]  [<c121a2da>] net_rx_action+0xb0/0x14d
[  286.930813]  [<c1030d43>] __do_softirq+0xb8/0x1a5
[  286.930813]  [<c1030c8b>] ? cpu_callback+0xec/0xec
[  286.930813]  <IRQ> [  286.930813]  [<c1030fa5>] ?
irq_exit+0x44/0x81 [  286.930813]  [<c1002da0>] ? do_IRQ+0x9f/0xb3
[  286.930813]  [<c106e642>] ? clockevents_notify+0x10f/0x116
[  286.930813]  [<c1298d33>] ? common_interrupt+0x33/0x40
[  286.930813]  [<c11f0292>] ? cpuidle_enter_state+0x39/0xa3
[  286.930813]  [<c11f03a5>] ? cpuidle_idle_call+0xa9/0xe6
[  286.930813]  [<c1007f8f>] ? arch_cpu_idle+0x8/0x1c
[  286.930813]  [<c1060542>] ? cpu_startup_entry+0xf3/0x159
[  286.930813]  [<c128b36b>] ? rest_init+0x5d/0x5f [  286.930813]
[<c1415a17>] ? start_kernel+0x3b2/0x3b8 [  286.930813]
[<c141549b>] ? repair_env_string+0x51/0x51 [  286.930813]
[<c14152d0>] ? i386_start_kernel+0x7a/0x7e [  286.930813] Code: 54
29 56 50 c7 46 54 00 00 00 00 29 d1 89 8e b4 00 00 00 e9 09 03 00
00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00 <8a> 00
88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05
[  286.930813] EIP: [<c120fc88>] skb_gro_receive+0x118/0x453 SS:ESP
0068:f6409e7c [  286.930813] CR2: 000000002b90bdc8 [  286.930813]
---[ end trace ac04411e60d3534c ]--- [  286.930813] Kernel panic -
Quoted 3 lines

not syncing: Fatal exception in interrupt [ 286.930813] Kernel
Offset: 0x0 from 0xc1000000 (relocation range:
0xc0000000-0xf7ffdfff) --

Quoted 4 lines

Unffortunately bisecting wont help a lot, as we knew the commit was faulty.

Please check 9d8506cc2d7ea1f911c72c100193a3677f6668c3

Ah, did not notice that one. So I'm adding Herbert as explicit cc.

$ git describe --contains 9d8506cc2d7ea1f911c72c100193a3677f6668c3
v3.13-rc1~7^2

But as you may have noticed, the attached oops is from 3.14.4 that includes the fix commit. While bisecting, I tested multiple versions between 3.12 and 3.14. Bisecting clearly points this is the commit that introduces this oops, and it is not fixed in any version up until 3.14.4.

Also the forward path from gre->ethX goes to eth that does not have TSO/GSO support. So I'd assume that the fix commit is not that relevant in my specific test case.

The faulty commit must be introducing additional bug, and the commit you refer did not fix it.

#3 lore raw
Quoted 11 lines · Timo Teras

Ah, did not notice that one. So I'm adding Herbert as explicit cc.

$ git describe --contains 9d8506cc2d7ea1f911c72c100193a3677f6668c3
v3.13-rc1~7^2

But as you may have noticed, the attached oops is from 3.14.4 that includes the fix commit. While bisecting, I tested multiple versions between 3.12 and 3.14. Bisecting clearly points this is the commit that introduces this oops, and it is not fixed in any version up until 3.14.4.

Quoted 6 lines

Also the forward path from gre->ethX goes to eth that does not have TSO/GSO support. So I'd assume that the fix commit is not that relevant in my specific test case.

The faulty commit must be introducing additional bug, and the commit you refer did not fix it.

Right. The commit could expose a dormant bug in a driver.

For example, some drivers might forget to use skb_put()

Please give us more details.

What is the driver receiving ethernet frames ?

Is it using GRO ?

author
#4 lore raw
Quoted 22 lines · On Fri, 16 May 2014 09:29:29 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 11 lines · Timo Teras

Ah, did not notice that one. So I'm adding Herbert as explicit cc.

$ git describe --contains 9d8506cc2d7ea1f911c72c100193a3677f6668c3
v3.13-rc1~7^2

But as you may have noticed, the attached oops is from 3.14.4 that includes the fix commit. While bisecting, I tested multiple versions between 3.12 and 3.14. Bisecting clearly points this is the commit that introduces this oops, and it is not fixed in any version up until 3.14.4.

Quoted 6 lines

Also the forward path from gre->ethX goes to eth that does not have TSO/GSO support. So I'd assume that the fix commit is not that relevant in my specific test case.

The faulty commit must be introducing additional bug, and the commit you refer did not fix it.

Right. The commit could expose a dormant bug in a driver.

Quoted 7 lines

For example, some drivers might forget to use skb_put()

Please give us more details.

What is the driver receiving ethernet frames ?

Is it using GRO ?

Both the gre bound interface, and the inside interface are identical.

And you are right. It might be driver / hw / gro capability specific issue. I had also report from a user using similar gre/ipsec/forwarding setup, that 3.13-stable kernels works on his hardware.

dmesg tells:

[   23.946984] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   23.978145] r8169 0000:00:09.0 (unregistered net_device): not PCI Express
[   23.997147] r8169 0000:00:09.0 eth0: RTL8169sc/8110sc at 0xf8358000, 00:30:18:a8:14:ac, XID 18000000 IRQ 18
[   24.007012] r8169 0000:00:09.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]
[   24.032389] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   24.039530] r8169 0000:00:0b.0 (unregistered net_device): not PCI Express
[   24.057092] r8169 0000:00:0b.0 eth1: RTL8169sc/8110sc at 0xf835a000, 00:30:18:ab:69:4b, XID 18000000 IRQ 19
[   24.066955] r8169 0000:00:0b.0 eth1: jumbo features [frames: 7152 bytes, tx checksumming: ok]
[   24.092330] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   24.098745] r8169 0000:00:0c.0 (unregistered net_device): not PCI Express
[   24.117118] r8169 0000:00:0c.0 eth2: RTL8169sc/8110sc at 0xf835c000, 00:30:18:a8:14:ad, XID 18000000 IRQ 16
[   24.126983] r8169 0000:00:0c.0 eth2: jumbo features [frames: 7152 bytes, tx checksumming: ok]

And, ethtool -k eth0

Features for eth0:
rx-checksumming: on
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]

Using unmodified in-tree r8169 module.

Please let me know if additional information is needed.

Thanks.
Timo

#5 lore raw
Quoted 10 lines · Timo Teras

Both the gre bound interface, and the inside interface are identical.

And you are right. It might be driver / hw / gro capability specific issue. I had also report from a user using similar gre/ipsec/forwarding setup, that 3.13-stable kernels works on his hardware.

Using unmodified in-tree r8169 module.

Please let me know if additional information is needed.

What happens if you disable gro on eth0 ?

author
#6 lore raw
Quoted 11 lines · On Fri, 16 May 2014 09:50:27 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 8 lines · Timo Teras

Both the gre bound interface, and the inside interface are identical.

And you are right. It might be driver / hw / gro capability specific issue. I had also report from a user using similar gre/ipsec/forwarding setup, that 3.13-stable kernels works on his hardware.

Quoted 5 lines
Quoted 3 lines

Using unmodified in-tree r8169 module.

Please let me know if additional information is needed.

What happens if you disable gro on eth0 ?

I believe it crashes, but cannot say with 100% certainty. I can test later on if needed. Based on earlier experiment, the only way to avoid the crash was to turn off gro on gre1.

In any case I don't think it should make any effect, since the GRE packets arrive IPseced. eth0 is receiving ESP packets - and I believe there's no ESP GRO support. Only after decryption they go to gre1, so gre1 gro is basically the place where received packets are coalesced.

#7 lore raw

On Fri, 2014-05-16 at 20:01 +0300, Timo Teras wrote:
Documentation/oops-tracing.txtscripts/decodecode

Quoted 8 lines

I believe it crashes, but cannot say with 100% certainty. I can test later on if needed. Based on earlier experiment, the only way to avoid the crash was to turn off gro on gre1.

In any case I don't think it should make any effect, since the GRE packets arrive IPseced. eth0 is receiving ESP packets - and I believe there's no ESP GRO support. Only after decryption they go to gre1, so gre1 gro is basically the place where received packets are coalesced.

Oh this is definitely useful.

I thought we might have a problem now with have GRE enabled GRO in native GRO layer. I was wondering if the double attempt to GRO packets a second time was a problem.

So it looks like ESP provides packets that are not very gentle for skb_gro_receive()...

Could you provide scripts/decodecode output ?

Thanks

author
#8 lore raw
Quoted 14 lines · On Fri, 16 May 2014 10:13:50 -0700 Eric Dumazet <[email protected]> wrote:

On Fri, 2014-05-16 at 20:01 +0300, Timo Teras wrote:
Documentation/oops-tracing.txtscripts/decodecode

Quoted 9 lines

I believe it crashes, but cannot say with 100% certainty. I can test later on if needed. Based on earlier experiment, the only way to avoid the crash was to turn off gro on gre1.

In any case I don't think it should make any effect, since the GRE packets arrive IPseced. eth0 is receiving ESP packets - and I believe there's no ESP GRO support. Only after decryption they go to gre1, so gre1 gro is basically the place where received packets are coalesced.

Oh this is definitely useful.

Quoted 8 lines

I thought we might have a problem now with have GRE enabled GRO in native GRO layer. I was wondering if the double attempt to GRO packets a second time was a problem.

So it looks like ESP provides packets that are not very gentle for skb_gro_receive()...

Could you provide scripts/decodecode output ?

[ 286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29

d1 89 8e b4 00 00 00 e9 09 03 00 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00 <8a> 00 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05

All code
========

   0:	54                   	push   %esp
   1:	29 56 50             	sub    %edx,0x50(%esi)
   4:	c7 46 54 00 00 00 00 	movl   $0x0,0x54(%esi)
   b:	29 d1                	sub    %edx,%ecx
   d:	89 8e b4 00 00 00    	mov    %ecx,0xb4(%esi)
  13:	e9 09 03 00 00       	jmp    0x321
  18:	8b 45 f0             	mov    -0x10(%ebp),%eax
  1b:	f6 80 87 00 00 00 01 	testb  $0x1,0x87(%eax)
  22:	8b 45 e8             	mov    -0x18(%ebp),%eax
  25:	0f 84 cf 00 00 00    	je     0xfa
  2b:*	8a 00                	mov    (%eax),%al		<-- trapping instruction
  2d:	88 45 d8             	mov    %al,-0x28(%ebp)
  30:	0f b6 d0             	movzbl %al,%edx
  33:	8b 45 f0             	mov    -0x10(%ebp),%eax
  36:	8b 80 ac 00 00 00    	mov    0xac(%eax),%eax
  3c:	89 45 d4             	mov    %eax,-0x2c(%ebp)
  3f:	05                   	.byte 0x5

Code starting with the faulting instruction
===========================================

   0:	8a 00                	mov    (%eax),%al
   2:	88 45 d8             	mov    %al,-0x28(%ebp)
   5:	0f b6 d0             	movzbl %al,%edx
   8:	8b 45 f0             	mov    -0x10(%ebp),%eax
   b:	8b 80 ac 00 00 00    	mov    0xac(%eax),%eax
  11:	89 45 d4             	mov    %eax,-0x2c(%ebp)
  14:	05                   	.byte 0x5

Unfortunately, I do not have fully matching identical .s, or debug built kernel. I can do that on Monday if this does not help.

Additionally, the .s file from separate, but similar build, that seems to have matching assembly is as follows:

	movl	168(%esi), %edx	# MEM[(const struct sk_buff *)skb_19(D)].end, D.55920
	subl	172(%esi), %edx	# MEM[(const struct sk_buff *)skb_19(D)].head, D.55920
	movb	$1, 43(%esi)	#, MEM[(struct napi_gro_cb *)skb_19(D) + 24B].free
	leal	-384(%ecx), %eax	#, delta_truesize
	subl	%edx, %eax	# D.55920, delta_truesize
	movl	84(%esi), %edx	# skb_19(D)->data_len, D.55922
	subl	%edx, 80(%esi)	# D.55922, skb_19(D)->len
	movl	$0, 84(%esi)	#, skb_19(D)->data_len
	subl	%edx, %ecx	# D.55922, tmp300
	movl	%ecx, 180(%esi)	# tmp300, skb_19(D)->truesize
	jmp	.L572	#
.L568:
	movl	-16(%ebp), %eax	# %sfp, skb
	testb	$1, 135(%eax)	#, *skb_19(D)
	movl	-24(%ebp), %eax	# %sfp, D.55921
	je	.L573	#,
## Crash on the next opcode
	movb	(%eax), %al	# MEM[(struct skb_shared_info *)_29].nr_frags, D.55923
	movb	%al, -40(%ebp)	# D.55923, %sfp
	movzbl	%al, %edx	# D.55923, nr_frags
	movl	-16(%ebp), %eax	# %sfp, skb
	movl	172(%eax), %eax	# skb_19(D)->head, skb_19(D)->head
	movl	%eax, -44(%ebp)	# skb_19(D)->head, %sfp
	addl	$1073741824, %eax	#, page
	shrl	$12, %eax	#, page
	sall	$5, %eax	#, page
	addl	mem_map, %eax	# mem_map, page
	movl	(%eax), %ecx	# MEM[(const long unsigned int *)page_213], D.55925
	movl	%eax, %edi	# page, D.55929
	andb	$128, %ch	#, D.55925
	je	.L574	#,
	movl	28(%eax), %esi	# page_213->D.14510.first_page, head
	movl	(%eax), %ecx	# MEM[(const long unsigned int *)page_213], D.55925
	andb	$128, %ch	#, D.55925
	je	.L587	#,
	movl	%esi, %edi	# head, D.55929
	jmp	.L574	#
#9 lore raw
Quoted 5 lines · Timo Teras

Unfortunately, I do not have fully matching identical .s, or debug built kernel. I can do that on Monday if this does not help.

Additionally, the .s file from separate, but similar build, that seems to have matching assembly is as follows:

Thanks a lot, I found the bug, its yet another case of skb->cb[] being not clean ;)

I am testing a patch atm.

#10 lore raw

PATCH 1/1 net: gro: make sure skb->cb[] initial content has not to be zero

From: Eric Dumazet <[email protected]>

Starting from linux-3.13, GRO attempts to build full size skbs.

Problem is the commit assumed one particular field in skb->cb[] was clean, but it is not the case on some stacked devices.

Timo reported a crash in case traffic is decrypted before reaching a GRE device.

Fix this by initializing NAPI_GRO_CB(skb)->last at the right place, this also removes one conditional.

Thanks a lot to Timo for providing full reports and bisecting this.

Fixes 8a29111c7ca6 ("net: gro: allow to build full sized skb")

---
Please Timo test this patch so that we confirm it fixes the bug for you.

 net/core/dev.c    |    1 +
 net/core/skbuff.c |    4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
net/core/dev.c +1 -0
index 6da649bde4f7..ed928e846559 100644
@@ -3951,6 +3951,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 	}
 	NAPI_GRO_CB(skb)->count = 1;
 	NAPI_GRO_CB(skb)->age = jiffies;
+	NAPI_GRO_CB(skb)->last = skb;
 	skb_shinfo(skb)->gso_size = skb_gro_len(skb);
 	skb->next = napi->gro_list;
 	napi->gro_list = skb;
net/core/skbuff.c +2 -2
index 1b62343f5837..8383b2bddeb9 100644
@@ -3076,7 +3076,7 @@ int skb_gro_receive(struct sk_buff **head, struct sk_buff *skb)
 	if (unlikely(p->len + len >= 65536))
 		return -E2BIG;
 
-	lp = NAPI_GRO_CB(p)->last ?: p;
+	lp = NAPI_GRO_CB(p)->last;
 	pinfo = skb_shinfo(lp);
 
 	if (headlen <= offset) {
@@ -3192,7 +3192,7 @@ merge:
 
 	__skb_pull(skb, offset);
 
-	if (!NAPI_GRO_CB(p)->last)
+	if (NAPI_GRO_CB(p)->last == p)
 		skb_shinfo(p)->frag_list = skb;
 	else
 		NAPI_GRO_CB(p)->last->next = skb;
author Tested-by on v1 1/1
#11 lore raw

Re: [PATCH] net: gro: make sure skb->cb[] initial content has not to be zero

Quoted 13 lines · On Fri, 16 May 2014 11:34:37 -0700 Eric Dumazet <[email protected]> wrote:

From: Eric Dumazet <[email protected]>

Starting from linux-3.13, GRO attempts to build full size skbs.

Problem is the commit assumed one particular field in skb->cb[] was clean, but it is not the case on some stacked devices.

Timo reported a crash in case traffic is decrypted before reaching a GRE device.

Fix this by initializing NAPI_GRO_CB(skb)->last at the right place, this also removes one conditional.

Quoted 8 lines

Thanks a lot to Timo for providing full reports and bisecting this.

Fixes 8a29111c7ca6 ("net: gro: allow to build full sized skb")

---
Please Timo test this patch so that we confirm it fixes the bug for you.

Yes, everything looks green now! (The test case reproduced the crash reliably and immediately. Now it's been up and working for more than tens minutes already :)

Needless to say, this should go to 3.14-stable as well.

Thanks a lot Eric!

maintainer
#12 lore raw

Re: [PATCH] net: gro: make sure skb->cb[] initial content has not to be zero

In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Fri, 16 May 2014 11:34:37 -0700

Quoted 13 lines

From: Eric Dumazet <[email protected]>

Starting from linux-3.13, GRO attempts to build full size skbs.

Problem is the commit assumed one particular field in skb->cb[] was clean, but it is not the case on some stacked devices.

Timo reported a crash in case traffic is decrypted before reaching a GRE device.

Fix this by initializing NAPI_GRO_CB(skb)->last at the right place, this also removes one conditional.

Quoted 7 lines

Thanks a lot to Timo for providing full reports and bisecting this.

Fixes 8a29111c7ca6 ("net: gro: allow to build full sized skb")

---
Please Timo test this patch so that we confirm it fixes the bug for you.

Applied and queued up for -stable, thanks everyone.