gre/gro oops in skb_gro_receive+0x118/0x453
Subsystems (2)
The oops happens when forwarding traffic between ethX <-> gre. Where the GRE tunnel is an NBMA tunnel and the GRE traffic is IPsec'ed in transport mode. It seems that locally originating traffic to gre is not affected.
The oops also goes away if GRO is turned off for gre tunnel device.
I have bisected this always reproducible oops to commit:
8a29111c7ca68d928dfab58636f3f6acf0ac04f7 is the first bad commit
commit 8a29111c7ca68d928dfab58636f3f6acf0ac04f7
Author: Eric Dumazet <[email protected]>
Date: Oct 8 09:02:23 2013 -0700
net: gro: allow to build full sized skb
This oops backtrace is from vanilla 3.14.4 kernel, but it is identical up to the offending commit.
[ 286.927713] BUG: unable to handle kernel paging request at 2b90bdc8 [ 286.930813] IP: [<c120fc88>] skb_gro_receive+0x118/0x453 [ 286.930813] *pde = 00000000 [ 286.930813] Oops: 0000 [#1] SMP [ 286.930813] Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport
deflate ctr twofish_generic twofish_i586 twofish_common camellia_generic serpent_sse2_i586 xts lrw gf128mul serpent_generic glue_helper ablk_helper cryptd blowfish_generic blowfish_common cast5_generic cast_common des_generic cbc cmac xcbc rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre ip_tunnel nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_raw ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT xt_helper nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables nf_conntrack_ftp nf_conntrack_sip xt_CT ip6table_raw xt_LOG xt_limit xt_policy xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_recent xt_multiport xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ipv6 af_packet mousedev via_rng rng_core via_cputemp hwmon hwmon_vid padlock_aes padlock_sha serio_raw psmouse pcspkr shpchp i2c_viapro i2c_core via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169 firmware_class mii fan evdev parport_pc parport thermal button acpi_cpufreq processor nls_utf8 nls_cp437 vfat fat sata_via ehci_pci ehci_hcd uhci_hcd pata_via pata_acpi ata_generic libata usb_storage usbcore usb_common sd_mod scsi_mod crc_t10dif crct10dif_common squashfs loop
[ 286.930813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.4 #1-fragbisect [ 286.930813] Hardware name: /CN700-8237, BIOS 6.00 PG 08/06/2008 [ 286.930813] task: c13e8930 ti: f6408000 task.ti: c13de000 [ 286.930813] EIP: 0060:[<c120fc88>] EFLAGS: 00210202 CPU: 0 [ 286.930813] EIP is at skb_gro_receive+0x118/0x453 [ 286.930813] EAX: 2b90bdc8 EBX: e9f669c0 ECX: 00000034 EDX: 00000596 [ 286.930813] ESI: e9f669c0 EDI: f648f134 EBP: f6409eb4 ESP: f6409e7c [ 286.930813] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 286.930813] CR0: 8005003b CR2: 2b90bdc8 CR3: 2a731000 CR4: 00000690 [ 286.930813] Stack: [ 286.930813] c120df4b e9f4f796 00000046 00000000 f648f134 c013a600 0000007a e9f4fd40 [ 286.930813] 2b90bdc8 00000034 e9f663c0 e9f663c0 e9f669c0 f648f134 f6409ee0 c125cc6f [ 286.930813] 00000000 00000020 00000046 2e001080 e9f4ed96 00000562 e9f663c0 e9f4f782 [ 286.930813] Call Trace: [ 286.930813] [<c120df4b>] ? csum_partial_ext+0x16/0x18 [ 286.930813] [<c125cc6f>] tcp_gro_receive+0x1a8/0x218 [ 286.930813] [<c125cd9f>] tcp4_gro_receive+0xc0/0xc8 [ 286.930813] [<c1268439>] inet_gro_receive+0x1c5/0x1df [ 286.930813] [<c1219a15>] dev_gro_receive+0x231/0x393 [ 286.930813] [<f80e006b>] ? rh_timer_func+0x8/0xa [usbcore] [ 286.930813] [<c1219c1c>] napi_gro_receive+0xb/0x5e [ 286.930813] [<f8650056>] gro_cell_poll+0x56/0x73 [ip_tunnel] [ 286.930813] [<c121a2da>] net_rx_action+0xb0/0x14d [ 286.930813] [<c1030d43>] __do_softirq+0xb8/0x1a5 [ 286.930813] [<c1030c8b>] ? cpu_callback+0xec/0xec [ 286.930813] <IRQ> [ 286.930813] [<c1030fa5>] ? irq_exit+0x44/0x81 [ 286.930813] [<c1002da0>] ? do_IRQ+0x9f/0xb3 [ 286.930813] [<c106e642>] ? clockevents_notify+0x10f/0x116 [ 286.930813] [<c1298d33>] ? common_interrupt+0x33/0x40 [ 286.930813] [<c11f0292>] ? cpuidle_enter_state+0x39/0xa3 [ 286.930813] [<c11f03a5>] ? cpuidle_idle_call+0xa9/0xe6 [ 286.930813] [<c1007f8f>] ? arch_cpu_idle+0x8/0x1c [ 286.930813] [<c1060542>] ? cpu_startup_entry+0xf3/0x159 [ 286.930813] [<c128b36b>] ? rest_init+0x5d/0x5f [ 286.930813] [<c1415a17>] ? start_kernel+0x3b2/0x3b8 [ 286.930813] [<c141549b>] ? repair_env_string+0x51/0x51 [ 286.930813] [<c14152d0>] ? i386_start_kernel+0x7a/0x7e [ 286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29 d1 89 8e b4 00 00 00 e9 09 03 00 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00 <8a> 00 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05 [ 286.930813] EIP: [<c120fc88>] skb_gro_receive+0x118/0x453 SS:ESP 0068:f6409e7c [ 286.930813] CR2: 000000002b90bdc8 [ 286.930813] ---[ end trace ac04411e60d3534c ]--- [ 286.930813] Kernel panic - not syncing: Fatal exception in interrupt [ 286.930813] Kernel Offset: 0x0 from 0xc1000000 (relocation range: 0xc0000000-0xf7ffdfff)
Quoted 88 lines · Timo Teras
The oops happens when forwarding traffic between ethX <-> gre. Where the GRE tunnel is an NBMA tunnel and the GRE traffic is IPsec'ed in transport mode. It seems that locally originating traffic to gre is not affected.
The oops also goes away if GRO is turned off for gre tunnel device.
I have bisected this always reproducible oops to commit:
8a29111c7ca68d928dfab58636f3f6acf0ac04f7is the first bad commitcommit
8a29111c7ca68d928dfab58636f3f6acf0ac04f7
Author: Eric Dumazet <[email protected]>
Date: Oct 8 09:02:23 2013 -0700
net: gro: allow to build full sized skbThis oops backtrace is from vanilla 3.14.4 kernel, but it is identical up to the offending commit.
[ 286.927713] BUG: unable to handle kernel paging request at 2b90bdc8 [ 286.930813] IP: [<c120fc88>] skb_gro_receive+0x118/0x453 [ 286.930813] *pde = 00000000 [ 286.930813] Oops: 0000 [#1] SMP [ 286.930813] Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transportdeflate ctr
twofish_generictwofish_i586twofish_commoncamellia_genericserpent_sse2_i586xts lrw gf128mulserpent_genericglue_helperablk_helpercryptdblowfish_genericblowfish_commoncast5_genericcast_commondes_genericcbc cmac xcbc rmd160sha512_generichmaccrypto_nullaf_keyxfrm_algoip_greip_tunnelnf_conntrack_netbios_nsnf_conntrack_broadcastiptable_rawipt_MASQUERADEiptable_natnf_nat_ipv4nf_natipt_REJECTxt_helpernf_conntrack_ipv4nf_defrag_ipv4iptable_filterip_tablesnf_conntrack_ftpnf_conntrack_sipxt_CTip6table_rawxt_LOGxt_limitxt_policyxt_tcpudpnf_conntrack_ipv6nf_defrag_ipv6xt_recentxt_multiportxt_conntracknf_conntrackip6table_filterip6_tablesx_tablesipv6af_packetmousedevvia_rngrng_corevia_cputemphwmonhwmon_vidpadlock_aespadlock_shaserio_rawpsmouse pcspkr shpchpi2c_viaproi2c_corevia_rhinesnd_via82xxsnd_ac97_codecsnd_pcmsnd_timerac97_bussnd_mpu401_uartsnd_rawmidisnd_seq_devicesnd soundcorefirewire_ohcifirewire_corecrc_itu_tvia_agpagpgart r8169firmware_classmii fan evdevparport_pcparport thermal buttonacpi_cpufreqprocessornls_utf8nls_cp437vfat fatsata_viaehci_pciehci_hcduhci_hcdpata_viapata_acpiata_genericlibatausb_storageusbcoreusb_commonsd_modscsi_modcrc_t10difcrct10dif_commonsquashfs loop[ 286.930813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.4 #1-fragbisect [ 286.930813] Hardware name: /CN700-8237, BIOS 6.00 PG 08/06/2008 [ 286.930813] task: c13e8930 ti: f6408000 task.ti: c13de000 [ 286.930813] EIP: 0060:[<c120fc88>] EFLAGS: 00210202 CPU: 0 [ 286.930813] EIP is at skb_gro_receive+0x118/0x453 [ 286.930813] EAX: 2b90bdc8 EBX: e9f669c0 ECX: 00000034 EDX: 00000596 [ 286.930813] ESI: e9f669c0 EDI: f648f134 EBP: f6409eb4 ESP: f6409e7c [ 286.930813] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 286.930813] CR0: 8005003b CR2: 2b90bdc8 CR3: 2a731000 CR4: 00000690 [ 286.930813] Stack: [ 286.930813] c120df4b e9f4f796 00000046 00000000 f648f134 c013a600 0000007a e9f4fd40 [ 286.930813] 2b90bdc8 00000034 e9f663c0 e9f663c0 e9f669c0 f648f134 f6409ee0 c125cc6f [ 286.930813] 00000000 00000020 00000046 2e001080 e9f4ed96 00000562 e9f663c0 e9f4f782 [ 286.930813] Call Trace: [ 286.930813] [<c120df4b>] ? csum_partial_ext+0x16/0x18 [ 286.930813] [<c125cc6f>] tcp_gro_receive+0x1a8/0x218 [ 286.930813] [<c125cd9f>] tcp4_gro_receive+0xc0/0xc8 [ 286.930813] [<c1268439>] inet_gro_receive+0x1c5/0x1df [ 286.930813] [<c1219a15>] dev_gro_receive+0x231/0x393 [ 286.930813] [<f80e006b>] ? rh_timer_func+0x8/0xa [usbcore] [ 286.930813] [<c1219c1c>] napi_gro_receive+0xb/0x5e [ 286.930813] [<f8650056>] gro_cell_poll+0x56/0x73 [ip_tunnel] [ 286.930813] [<c121a2da>] net_rx_action+0xb0/0x14d [ 286.930813] [<c1030d43>] __do_softirq+0xb8/0x1a5 [ 286.930813] [<c1030c8b>] ? cpu_callback+0xec/0xec [ 286.930813] <IRQ> [ 286.930813] [<c1030fa5>] ? irq_exit+0x44/0x81 [ 286.930813] [<c1002da0>] ? do_IRQ+0x9f/0xb3 [ 286.930813] [<c106e642>] ? clockevents_notify+0x10f/0x116 [ 286.930813] [<c1298d33>] ? common_interrupt+0x33/0x40 [ 286.930813] [<c11f0292>] ? cpuidle_enter_state+0x39/0xa3 [ 286.930813] [<c11f03a5>] ? cpuidle_idle_call+0xa9/0xe6 [ 286.930813] [<c1007f8f>] ? arch_cpu_idle+0x8/0x1c [ 286.930813] [<c1060542>] ? cpu_startup_entry+0xf3/0x159 [ 286.930813] [<c128b36b>] ? rest_init+0x5d/0x5f [ 286.930813] [<c1415a17>] ? start_kernel+0x3b2/0x3b8 [ 286.930813] [<c141549b>] ? repair_env_string+0x51/0x51 [ 286.930813] [<c14152d0>] ? i386_start_kernel+0x7a/0x7e [ 286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29 d1 89 8e b4 00 00 00 e9 09 03 00 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00 <8a> 00 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05 [ 286.930813] EIP: [<c120fc88>] skb_gro_receive+0x118/0x453 SS:ESP 0068:f6409e7c [ 286.930813] CR2: 000000002b90bdc8 [ 286.930813] ---[ end trace ac04411e60d3534c ]--- [ 286.930813] Kernel panic - not syncing: Fatal exception in interrupt [ 286.930813] Kernel Offset: 0x0 from 0xc1000000 (relocation range: 0xc0000000-0xf7ffdfff)
Quoted 1 line
--
Unffortunately bisecting wont help a lot, as we knew the commit was faulty.
Please check 9d8506cc2d7ea1f911c72c100193a3677f6668c3
Quoted 94 lines · On Fri, 16 May 2014 05:59:17 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 89 lines · Timo Teras
The oops happens when forwarding traffic between ethX <-> gre. Where the GRE tunnel is an NBMA tunnel and the GRE traffic is IPsec'ed in transport mode. It seems that locally originating traffic to gre is not affected.
The oops also goes away if GRO is turned off for gre tunnel device.
I have bisected this always reproducible oops to commit:
8a29111c7ca68d928dfab58636f3f6acf0ac04f7is the first bad commitcommit
8a29111c7ca68d928dfab58636f3f6acf0ac04f7
Author: Eric Dumazet <[email protected]>
Date: Oct 8 09:02:23 2013 -0700
net: gro: allow to build full sized skbThis oops backtrace is from vanilla 3.14.4 kernel, but it is identical up to the offending commit.
[ 286.927713] BUG: unable to handle kernel paging request at 2b90bdc8 [ 286.930813] IP: [<c120fc88>] skb_gro_receive+0x118/0x453 [ 286.930813] *pde = 00000000 [ 286.930813] Oops: 0000 [#1] SMP [ 286.930813] Modules linked in: sha1_generic authenc esp4
xfrm4_mode_transportdeflate ctrtwofish_generictwofish_i586twofish_commoncamellia_genericserpent_sse2_i586xts lrw gf128mulserpent_genericglue_helperablk_helpercryptdblowfish_genericblowfish_commoncast5_genericcast_commondes_genericcbc cmac xcbc rmd160sha512_generichmaccrypto_nullaf_keyxfrm_algoip_greip_tunnelnf_conntrack_netbios_nsnf_conntrack_broadcastiptable_rawipt_MASQUERADEiptable_natnf_nat_ipv4nf_natipt_REJECTxt_helpernf_conntrack_ipv4nf_defrag_ipv4iptable_filterip_tablesnf_conntrack_ftpnf_conntrack_sipxt_CTip6table_rawxt_LOGxt_limitxt_policyxt_tcpudpnf_conntrack_ipv6nf_defrag_ipv6xt_recentxt_multiportxt_conntracknf_conntrackip6table_filterip6_tablesx_tablesipv6af_packetmousedevvia_rngrng_corevia_cputemphwmonhwmon_vidpadlock_aespadlock_shaserio_rawpsmouse pcspkr shpchpi2c_viaproi2c_corevia_rhinesnd_via82xxsnd_ac97_codecsnd_pcmsnd_timerac97_bussnd_mpu401_uartsnd_rawmidisnd_seq_devicesnd soundcorefirewire_ohcifirewire_corecrc_itu_tvia_agpagpgart r8169firmware_classmii fan evdevparport_pcparport thermal buttonacpi_cpufreqprocessornls_utf8nls_cp437vfat fatsata_viaehci_pciehci_hcduhci_hcdpata_viapata_acpiata_genericlibatausb_storageusbcoreusb_commonsd_modscsi_modcrc_t10difcrct10dif_commonsquashfs loop [ 286.930813] CPU: 0 PID: 0 Comm:
swapper/0 Not tainted 3.14.4 #1-fragbisect [ 286.930813] Hardware
name: /CN700-8237, BIOS 6.00 PG 08/06/2008 [ 286.930813] task:c13e8930ti:f6408000task.ti:c13de000[ 286.930813] EIP:
0060:[<c120fc88>] EFLAGS: 00210202 CPU: 0 [ 286.930813] EIP is atskb_gro_receive+0x118/0x453 [ 286.930813] EAX: 2b90bdc8 EBX: e9f669c0 ECX: 00000034 EDX: 00000596 [ 286.930813] ESI: e9f669c0 EDI: f648f134 EBP: f6409eb4 ESP: f6409e7c [ 286.930813] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 286.930813] CR0: 8005003b CR2: 2b90bdc8 CR3: 2a731000 CR4: 00000690 [ 286.930813] Stack: [ 286.930813] c120df4b e9f4f796 00000046 00000000 f648f134 c013a600 0000007a e9f4fd40 [ 286.930813] 2b90bdc8 00000034 e9f663c0 e9f663c0 e9f669c0 f648f134 f6409ee0 c125cc6f [ 286.930813] 00000000 00000020 00000046 2e001080 e9f4ed96 00000562 e9f663c0 e9f4f782 [ 286.930813] Call Trace: [ 286.930813] [<c120df4b>] ? csum_partial_ext+0x16/0x18 [ 286.930813] [<c125cc6f>] tcp_gro_receive+0x1a8/0x218 [ 286.930813] [<c125cd9f>] tcp4_gro_receive+0xc0/0xc8 [ 286.930813] [<c1268439>] inet_gro_receive+0x1c5/0x1df [ 286.930813] [<c1219a15>] dev_gro_receive+0x231/0x393 [ 286.930813] [<f80e006b>] ? rh_timer_func+0x8/0xa [usbcore] [ 286.930813] [<c1219c1c>] napi_gro_receive+0xb/0x5e [ 286.930813] [<f8650056>] gro_cell_poll+0x56/0x73 [ip_tunnel] [ 286.930813] [<c121a2da>] net_rx_action+0xb0/0x14d [ 286.930813] [<c1030d43>] __do_softirq+0xb8/0x1a5 [ 286.930813] [<c1030c8b>] ? cpu_callback+0xec/0xec [ 286.930813] <IRQ> [ 286.930813] [<c1030fa5>] ? irq_exit+0x44/0x81 [ 286.930813] [<c1002da0>] ? do_IRQ+0x9f/0xb3 [ 286.930813] [<c106e642>] ? clockevents_notify+0x10f/0x116 [ 286.930813] [<c1298d33>] ? common_interrupt+0x33/0x40 [ 286.930813] [<c11f0292>] ? cpuidle_enter_state+0x39/0xa3 [ 286.930813] [<c11f03a5>] ? cpuidle_idle_call+0xa9/0xe6 [ 286.930813] [<c1007f8f>] ? arch_cpu_idle+0x8/0x1c [ 286.930813] [<c1060542>] ? cpu_startup_entry+0xf3/0x159 [ 286.930813] [<c128b36b>] ? rest_init+0x5d/0x5f [ 286.930813] [<c1415a17>] ? start_kernel+0x3b2/0x3b8 [ 286.930813] [<c141549b>] ? repair_env_string+0x51/0x51 [ 286.930813] [<c14152d0>] ? i386_start_kernel+0x7a/0x7e [ 286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29 d1 89 8e b4 00 00 00 e9 09 03 00 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00 <8a> 00 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05 [ 286.930813] EIP: [<c120fc88>] skb_gro_receive+0x118/0x453 SS:ESP 0068:f6409e7c [ 286.930813] CR2: 000000002b90bdc8 [ 286.930813] ---[ end trace ac04411e60d3534c ]--- [ 286.930813] Kernel panic -Quoted 3 lines
not syncing: Fatal exception in interrupt [ 286.930813] Kernel
Offset:0x0from0xc1000000(relocation range:
0xc0000000-0xf7ffdfff) --
Quoted 4 lines
Unffortunately bisecting wont help a lot, as we knew the commit was faulty.
Please check
9d8506cc2d7ea1f911c72c100193a3677f6668c3
Ah, did not notice that one. So I'm adding Herbert as explicit cc.
$ git describe --contains 9d8506cc2d7ea1f911c72c100193a3677f6668c3 v3.13-rc1~7^2
But as you may have noticed, the attached oops is from 3.14.4 that includes the fix commit. While bisecting, I tested multiple versions between 3.12 and 3.14. Bisecting clearly points this is the commit that introduces this oops, and it is not fixed in any version up until 3.14.4.
Also the forward path from gre->ethX goes to eth that does not have TSO/GSO support. So I'd assume that the fix commit is not that relevant in my specific test case.
The faulty commit must be introducing additional bug, and the commit you refer did not fix it.
Quoted 11 lines · Timo Teras
Ah, did not notice that one. So I'm adding Herbert as explicit cc.
$ git describe --contains 9d8506cc2d7ea1f911c72c100193a3677f6668c3 v3.13-rc1~7^2But as you may have noticed, the attached oops is from 3.14.4 that includes the fix commit. While bisecting, I tested multiple versions between 3.12 and 3.14. Bisecting clearly points this is the commit that introduces this oops, and it is not fixed in any version up until 3.14.4.
Quoted 6 lines
Also the forward path from
gre->ethXgoes to eth that does not have TSO/GSO support. So I'd assume that the fix commit is not that relevant in my specific test case.The faulty commit must be introducing additional bug, and the commit you refer did not fix it.
Right. The commit could expose a dormant bug in a driver.
For example, some drivers might forget to use skb_put()
Please give us more details.
What is the driver receiving ethernet frames ?
Is it using GRO ?
Quoted 22 lines · On Fri, 16 May 2014 09:29:29 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 11 lines · Timo Teras
Ah, did not notice that one. So I'm adding Herbert as explicit cc.
$ git describe --contains 9d8506cc2d7ea1f911c72c100193a3677f6668c3 v3.13-rc1~7^2But as you may have noticed, the attached oops is from 3.14.4 that includes the fix commit. While bisecting, I tested multiple versions between 3.12 and 3.14. Bisecting clearly points this is the commit that introduces this oops, and it is not fixed in any version up until 3.14.4.
Quoted 6 lines
Also the forward path from
gre->ethXgoes to eth that does not have TSO/GSO support. So I'd assume that the fix commit is not that relevant in my specific test case.The faulty commit must be introducing additional bug, and the commit you refer did not fix it.
Right. The commit could expose a dormant bug in a driver.
Quoted 7 lines
For example, some drivers might forget to use
skb_put()Please give us more details.
What is the driver receiving ethernet frames ?
Is it using GRO ?
Both the gre bound interface, and the inside interface are identical.
And you are right. It might be driver / hw / gro capability specific issue. I had also report from a user using similar gre/ipsec/forwarding setup, that 3.13-stable kernels works on his hardware.
dmesg tells:
[ 23.946984] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 23.978145] r8169 0000:00:09.0 (unregistered net_device): not PCI Express [ 23.997147] r8169 0000:00:09.0 eth0: RTL8169sc/8110sc at 0xf8358000, 00:30:18:a8:14:ac, XID 18000000 IRQ 18 [ 24.007012] r8169 0000:00:09.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok] [ 24.032389] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 24.039530] r8169 0000:00:0b.0 (unregistered net_device): not PCI Express [ 24.057092] r8169 0000:00:0b.0 eth1: RTL8169sc/8110sc at 0xf835a000, 00:30:18:ab:69:4b, XID 18000000 IRQ 19 [ 24.066955] r8169 0000:00:0b.0 eth1: jumbo features [frames: 7152 bytes, tx checksumming: ok] [ 24.092330] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 24.098745] r8169 0000:00:0c.0 (unregistered net_device): not PCI Express [ 24.117118] r8169 0000:00:0c.0 eth2: RTL8169sc/8110sc at 0xf835c000, 00:30:18:a8:14:ad, XID 18000000 IRQ 16 [ 24.126983] r8169 0000:00:0c.0 eth2: jumbo features [frames: 7152 bytes, tx checksumming: ok]
And, ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
Using unmodified in-tree r8169 module.
Please let me know if additional information is needed.
Thanks.
Timo
Quoted 10 lines · Timo Teras
Both the gre bound interface, and the inside interface are identical.
And you are right. It might be driver / hw / gro capability specific issue. I had also report from a user using similar gre/ipsec/forwarding setup, that 3.13-stable kernels works on his hardware.
Using unmodified in-tree r8169 module.
Please let me know if additional information is needed.
What happens if you disable gro on eth0 ?
Quoted 11 lines · On Fri, 16 May 2014 09:50:27 -0700 Eric Dumazet <[email protected]> wrote:
Quoted 8 lines · Timo Teras
Both the gre bound interface, and the inside interface are identical.
And you are right. It might be driver / hw / gro capability specific issue. I had also report from a user using similar gre/ipsec/forwarding setup, that 3.13-stable kernels works on his hardware.
Quoted 5 lines
Quoted 3 lines
Using unmodified in-tree r8169 module.
Please let me know if additional information is needed.
What happens if you disable gro on eth0 ?
I believe it crashes, but cannot say with 100% certainty. I can test later on if needed. Based on earlier experiment, the only way to avoid the crash was to turn off gro on gre1.
In any case I don't think it should make any effect, since the GRE packets arrive IPseced. eth0 is receiving ESP packets - and I believe there's no ESP GRO support. Only after decryption they go to gre1, so gre1 gro is basically the place where received packets are coalesced.
On Fri, 2014-05-16 at 20:01 +0300, Timo Teras wrote:Documentation/oops-tracing.txtscripts/decodecode
Quoted 8 lines
I believe it crashes, but cannot say with 100% certainty. I can test later on if needed. Based on earlier experiment, the only way to avoid the crash was to turn off gro on gre1.
In any case I don't think it should make any effect, since the GRE packets arrive IPseced. eth0 is receiving ESP packets - and I believe there's no ESP GRO support. Only after decryption they go to gre1, so gre1 gro is basically the place where received packets are coalesced.
Oh this is definitely useful.
I thought we might have a problem now with have GRE enabled GRO in native GRO layer. I was wondering if the double attempt to GRO packets a second time was a problem.
So it looks like ESP provides packets that are not very gentle for skb_gro_receive()...
Could you provide scripts/decodecode output ?
Thanks
Quoted 14 lines · On Fri, 16 May 2014 10:13:50 -0700 Eric Dumazet <[email protected]> wrote:
On Fri, 2014-05-16 at 20:01 +0300, Timo Teras wrote:
Documentation/oops-tracing.txtscripts/decodecodeQuoted 9 lines
I believe it crashes, but cannot say with 100% certainty. I can test later on if needed. Based on earlier experiment, the only way to avoid the crash was to turn off gro on gre1.
In any case I don't think it should make any effect, since the GRE packets arrive IPseced. eth0 is receiving ESP packets - and I believe there's no ESP GRO support. Only after decryption they go to gre1, so gre1 gro is basically the place where received packets are coalesced.
Oh this is definitely useful.
Quoted 8 lines
I thought we might have a problem now with have GRE enabled GRO in native GRO layer. I was wondering if the double attempt to GRO packets a second time was a problem.
So it looks like ESP provides packets that are not very gentle for
skb_gro_receive()...Could you provide
scripts/decodecodeoutput ?
[ 286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29
d1 89 8e b4 00 00 00 e9 09 03 00 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00 <8a> 00 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05
All code
========
0: 54 push %esp 1: 29 56 50 sub %edx,0x50(%esi) 4: c7 46 54 00 00 00 00 movl $0x0,0x54(%esi) b: 29 d1 sub %edx,%ecx d: 89 8e b4 00 00 00 mov %ecx,0xb4(%esi) 13: e9 09 03 00 00 jmp 0x321 18: 8b 45 f0 mov -0x10(%ebp),%eax 1b: f6 80 87 00 00 00 01 testb $0x1,0x87(%eax) 22: 8b 45 e8 mov -0x18(%ebp),%eax 25: 0f 84 cf 00 00 00 je 0xfa 2b:* 8a 00 mov (%eax),%al <-- trapping instruction 2d: 88 45 d8 mov %al,-0x28(%ebp) 30: 0f b6 d0 movzbl %al,%edx 33: 8b 45 f0 mov -0x10(%ebp),%eax 36: 8b 80 ac 00 00 00 mov 0xac(%eax),%eax 3c: 89 45 d4 mov %eax,-0x2c(%ebp) 3f: 05 .byte 0x5
Code starting with the faulting instruction
===========================================
0: 8a 00 mov (%eax),%al 2: 88 45 d8 mov %al,-0x28(%ebp) 5: 0f b6 d0 movzbl %al,%edx 8: 8b 45 f0 mov -0x10(%ebp),%eax b: 8b 80 ac 00 00 00 mov 0xac(%eax),%eax 11: 89 45 d4 mov %eax,-0x2c(%ebp) 14: 05 .byte 0x5
Unfortunately, I do not have fully matching identical .s, or debug built kernel. I can do that on Monday if this does not help.
Additionally, the .s file from separate, but similar build, that seems to have matching assembly is as follows:
movl 168(%esi), %edx # MEM[(const struct sk_buff *)skb_19(D)].end, D.55920 subl 172(%esi), %edx # MEM[(const struct sk_buff *)skb_19(D)].head, D.55920 movb $1, 43(%esi) #, MEM[(struct napi_gro_cb *)skb_19(D) + 24B].free leal -384(%ecx), %eax #, delta_truesize subl %edx, %eax # D.55920, delta_truesize movl 84(%esi), %edx # skb_19(D)->data_len, D.55922 subl %edx, 80(%esi) # D.55922, skb_19(D)->len movl $0, 84(%esi) #, skb_19(D)->data_len subl %edx, %ecx # D.55922, tmp300 movl %ecx, 180(%esi) # tmp300, skb_19(D)->truesize jmp .L572 # .L568: movl -16(%ebp), %eax # %sfp, skb testb $1, 135(%eax) #, *skb_19(D) movl -24(%ebp), %eax # %sfp, D.55921 je .L573 #, ## Crash on the next opcode movb (%eax), %al # MEM[(struct skb_shared_info *)_29].nr_frags, D.55923 movb %al, -40(%ebp) # D.55923, %sfp movzbl %al, %edx # D.55923, nr_frags movl -16(%ebp), %eax # %sfp, skb movl 172(%eax), %eax # skb_19(D)->head, skb_19(D)->head movl %eax, -44(%ebp) # skb_19(D)->head, %sfp addl $1073741824, %eax #, page shrl $12, %eax #, page sall $5, %eax #, page addl mem_map, %eax # mem_map, page movl (%eax), %ecx # MEM[(const long unsigned int *)page_213], D.55925 movl %eax, %edi # page, D.55929 andb $128, %ch #, D.55925 je .L574 #, movl 28(%eax), %esi # page_213->D.14510.first_page, head movl (%eax), %ecx # MEM[(const long unsigned int *)page_213], D.55925 andb $128, %ch #, D.55925 je .L587 #, movl %esi, %edi # head, D.55929 jmp .L574 #
Quoted 5 lines · Timo Teras
Unfortunately, I do not have fully matching identical .s, or debug built kernel. I can do that on Monday if this does not help.
Additionally, the .s file from separate, but similar build, that seems to have matching assembly is as follows:
Thanks a lot, I found the bug, its yet another case of skb->cb[] being not clean ;)
I am testing a patch atm.
PATCH 1/1 net: gro: make sure skb->cb[] initial content has not to be zero
From: Eric Dumazet <[email protected]>
Starting from linux-3.13, GRO attempts to build full size skbs.
Problem is the commit assumed one particular field in skb->cb[] was clean, but it is not the case on some stacked devices.
Timo reported a crash in case traffic is decrypted before reaching a GRE device.
Fix this by initializing NAPI_GRO_CB(skb)->last at the right place, this also removes one conditional.
Thanks a lot to Timo for providing full reports and bisecting this.
---
Please Timo test this patch so that we confirm it fixes the bug for you.
net/core/dev.c | 1 + net/core/skbuff.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)
net/core/dev.c +1 -0
@@ -3951,6 +3951,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff }NAPI_GRO_CB (skb)->count =1 ;NAPI_GRO_CB (skb)->age = jiffies; +NAPI_GRO_CB (skb)->last = skb;skb_shinfo (skb)->gso_size =skb_gro_len (skb); skb->next = napi->gro_list; napi->gro_list = skb;
net/core/skbuff.c +2 -2
@@ -3076,7 +3076,7 @@ int skb_gro_receive(struct sk_buff **head, struct sk_buff *skb)if (unlikely (p->len + len >=65536 ))return -E2BIG ; - lp =NAPI_GRO_CB (p)->last ?: p; + lp =NAPI_GRO_CB (p)->last; pinfo =skb_shinfo (lp);if (headlen <= offset) {
@@ -3192,7 +3192,7 @@ merge:__skb_pull (skb, offset); -if (!NAPI_GRO_CB (p)->last) +if (NAPI_GRO_CB (p)->last == p)skb_shinfo (p)->frag_list = skb;else NAPI_GRO_CB (p)->last->next = skb;
Re: [PATCH] net: gro: make sure skb->cb[] initial content has not to be zero
Quoted 13 lines · On Fri, 16 May 2014 11:34:37 -0700 Eric Dumazet <[email protected]> wrote:
From: Eric Dumazet <[email protected]>
Starting from linux-3.13, GRO attempts to build full size skbs.
Problem is the commit assumed one particular field in skb->cb[] was clean, but it is not the case on some stacked devices.
Timo reported a crash in case traffic is decrypted before reaching a GRE device.
Fix this by initializing NAPI_GRO_CB(skb)->last at the right place, this also removes one conditional.
Quoted 8 lines
Thanks a lot to Timo for providing full reports and bisecting this.
Fixes 8a29111c7ca6 ("net: gro: allow to build full sized skb")Bisected-by Timo Teras <[email protected]>Signed-off-by Eric Dumazet <[email protected]>---
Please Timo test this patch so that we confirm it fixes the bug for you.
Yes, everything looks green now! (The test case reproduced the crash reliably and immediately. Now it's been up and working for more than tens minutes already :)
Needless to say, this should go to 3.14-stable as well.
Thanks a lot Eric!
David Miller
maintainerRe: [PATCH] net: gro: make sure skb->cb[] initial content has not to be zero
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
Date: Fri, 16 May 2014 11:34:37 -0700
Quoted 13 lines
From: Eric Dumazet <[email protected]>
Starting from linux-3.13, GRO attempts to build full size skbs.
Problem is the commit assumed one particular field in skb->cb[] was clean, but it is not the case on some stacked devices.
Timo reported a crash in case traffic is decrypted before reaching a GRE device.
Fix this by initializing NAPI_GRO_CB(skb)->last at the right place, this also removes one conditional.
Quoted 7 lines
Thanks a lot to Timo for providing full reports and bisecting this.
Fixes 8a29111c7ca6 ("net: gro: allow to build full sized skb")Bisected-by Timo Teras <[email protected]>Signed-off-by Eric Dumazet <[email protected]>---
Please Timo test this patch so that we confirm it fixes the bug for you.
Applied and queued up for -stable, thanks everyone.