BUG: unable to handle kernel NULL pointer dereference in ipv6_select_ident
Subsystems (5)
Hi folks,
I'm working on getting a 2-node VM cluster up and running, with DRBD and Corosync/Pacemaker, running KVM VMs.
I can trigger a kernel panic in either _host_ system when running an rsync on a _guest_ VM. The rsync is simply SSH over IPv6 from a remote mail store (containing maildirs) to a local filesystem. I'm basically working on migrating a physical IMAP server to one inside a VM.
After a few seconds of fairly heavy IPv6 traffic, I get the panic below. You'll notice the panic refers to vhost_net, but I tried without that and the kernel panics at exactly the same call point.
Panic:
[ 461.232932] BUG: unable to handle kernel NULL pointer dereference at 0000000000000298 [ 461.240790] IP: [<ffffffff812dde61>] ipv6_select_ident+0x31/0xa7 [ 461.246916] PGD 42d3e8067 PUD 41facf067 PMD 0 [ 461.251537] Oops: 0000 [#1] SMP [ 461.254795] CPU 4 [ 461.256648] Modules linked in: sha1_generic hmac sha256_generic dlm configfs ebtable_nat ebtables acpi_cpufreq mperf cpufreq_stats cpufreq_conservative cpufreq_powersave cpufreq_userspace microcode xt_NOTRACK ip_set_hash_net act_police cls_basic cls_flow cls_fw cls_u32 sch_
tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq xt_realm xt_connlimit xt_addrtype ip_set_hash_ip iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ip6_queue ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn nf_conntrack_proto_udplite ipt_CLUSTERIP ipt_ah xt_time xt_s et xt_TCPMSS ip_set xt_sctp xt_policy nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ip6t_LOG ts_kmp ip6t_REJECT nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf _conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlin k_log xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_connmark xt_CLASSIFY xt_AUDIT ipt_LOG xt_tcpudp xt_state nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat xt_conntrack nf_conntrack_ipv4 nf_defrag_ipv4 ip6table _raw nf_conntrack ip6table_mangle iptable_mangle nfnetlink iptable_filter ip_tables ip6table_filter ip6_tables x_tables bridge stp bonding w83627ehf hwmon_vid coretemp crc32c_intel aesni_intel cryptd aes_x86_64 aes_generic ipmi_poweroff ipmi_devintf ipmi_si ipmi_msghandler vho st_net macvtap macvlan tun drbd lru_cache cn loop kvm_intel kvm snd_pcm snd_timer snd soundcore snd_page_alloc psmouse i2c_i801 processor iTCO_wdt iTCO_vendor_support i2c_core evdev thermal_sys joydev serio_raw pcspkr button ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod sd_mod c rc_t10dif usb_storage uas usbhid hid ahci libahci libata igb ehci_hcd scsi_mod usbcore e1000e dca [last unloaded: scsi_wait_scan]
[ 461.446246] [ 461.447812] Pid: 5756, comm: vhost-5753 Not tainted 3.1.0-1-amd64 #1 Supermicro X9SCL/X9SCM/X9SCL/X9SCM [ 461.457562] RIP: 0010:[<ffffffff812dde61>] [<ffffffff812dde61>] ipv6_select_ident+0x31/0xa7 [ 461.466271] RSP: 0018:ffff88043fd03758 EFLAGS: 00010202 [ 461.471708] RAX: ffff8803f7159038 RBX: 00000000000001f4 RCX: 0000000000000000 [ 461.478979] RDX: 0000000000000016 RSI: ffff88042dadcf30 RDI: ffff8803f715906e [ 461.486207] RBP: ffff8803f715906e R08: 0ea8c9feff005256 R09: 00000000000080fe [ 461.493542] R10: 5256000000000000 R11: 80fe4e6fdcfeff00 R12: 0000000040115ba9 [ 461.500851] R13: ffffffff8168ed11 R14: 0000000000000011 R15: ffff8803f7159880 [ 461.508167] FS: 0000000000000000(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000 [ 461.516444] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 461.522317] CR2: 0000000000000298 CR3: 000000042d87e000 CR4: 00000000000426e0 [ 461.529607] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 461.536916] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 461.544267] Process vhost-5753 (pid: 5756, threadinfo ffff88042cb4a000, task ffff88042d13c970) [ 461.553108] Stack: [ 461.555202] ffff8803f7159038 ffff88042be0e280 0000000000000028 ffffffff812efa7f [ 461.562840] 0000000000000246 ffff8803f7159054 ffff88042be0e318 ffff88042be0e280 [ 461.570544] ffffffffffffffa3 0000000040115ba9 ffffffff8168edc0 ffffffff812dbfe0 [ 461.578229] Call Trace: [ 461.580742] <IRQ> [ 461.582870] [<ffffffff812efa7f>] ? udp6_ufo_fragment+0x124/0x1a2 [ 461.589054] [<ffffffff812dbfe0>] ? ipv6_gso_segment+0xc0/0x155 [ 461.595140] [<ffffffff812700c6>] ? skb_gso_segment+0x208/0x28b [ 461.601198] [<ffffffffa03f236b>] ? ipv6_confirm+0x146/0x15e [nf_conntrack_ipv6] [ 461.608786] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.614227] [<ffffffff81271d64>] ? dev_hard_start_xmit+0x357/0x543 [ 461.620659] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.626440] [<ffffffffa0379745>] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 461.633581] [<ffffffff812722ff>] ? dev_queue_xmit+0x3af/0x459 [ 461.639577] [<ffffffffa03747d2>] ? br_dev_queue_push_xmit+0x72/0x76 [bridge] [ 461.646887] [<ffffffffa03791e3>] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 461.653997] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.659473] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.665485] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.671234] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.677299] [<ffffffffa0379215>] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 461.684891] [<ffffffffa03bb0e5>] ? nf_ct_zone+0xa/0x17 [nf_conntrack] [ 461.691520] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.697572] [<ffffffffa0374812>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 461.704616] [<ffffffffa0379031>] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 461.712329] [<ffffffffa037929f>] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 461.719490] [<ffffffffa037900a>] ? nf_bridge_pull_encap_header+0x1c/0x27 [bridge] [ 461.727223] [<ffffffffa0379974>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 461.734292] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.739758] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.746203] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.751950] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.758378] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.765454] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.771881] [<ffffffffa0374812>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 461.778908] [<ffffffffa03749a6>] ? br_forward+0x16/0x5a [bridge] [ 461.785041] [<ffffffffa03754db>] ? br_handle_frame_finish+0x1a1/0x20f [bridge] [ 461.792604] [<ffffffffa0379333>] ? br_nf_pre_routing_finish_ipv6+0x89/0x92 [bridge] [ 461.800513] [<ffffffffa0378e7b>] ? setup_pre_routing+0x38/0x5d [bridge] [ 461.807440] [<ffffffffa0379e65>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge] [ 461.814463] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.819908] [<ffffffff8103f89d>] ? select_task_rq_fair+0x369/0x610 [ 461.826347] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.833463] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.840532] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.846134] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.853222] [<ffffffff81036010>] ? test_tsk_need_resched+0xa/0x13 [ 461.859601] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.866650] [<ffffffffa0375320>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge] [ 461.873686] [<ffffffffa03756ed>] ? br_handle_frame+0x1a4/0x1bb [bridge] [ 461.880569] [<ffffffffa0375549>] ? br_handle_frame_finish+0x20f/0x20f [bridge] [ 461.888042] [<ffffffff8126f132>] ? __netif_receive_skb+0x2d6/0x415 [ 461.894467] [<ffffffff8126f2dd>] ? process_backlog+0x6c/0x123 [ 461.900414] [<ffffffff81026b37>] ? native_apic_msr_write+0x2c/0x2f [ 461.906790] [<ffffffff81271034>] ? net_rx_action+0xa1/0x1af [ 461.912625] [<ffffffff81036010>] ? test_tsk_need_resched+0xa/0x13 [ 461.919010] [<ffffffff8104ad04>] ? __do_softirq+0xb9/0x177 [ 461.924796] [<ffffffff8133452c>] ? call_softirq+0x1c/0x30 [ 461.930474] <EOI> [ 461.932684] [<ffffffff8100f845>] ? do_softirq+0x3c/0x7b [ 461.938106] [<ffffffff81271324>] ? netif_rx_ni+0x1e/0x27 [ 461.943610] [<ffffffffa0298721>] ? tun_get_user+0x39a/0x3c2 [tun] [ 461.949923] [<ffffffffa0298766>] ? tun_sendmsg+0x1d/0x1f [tun] [ 461.955938] [<ffffffffa02a6b50>] ? handle_tx+0x340/0x3de [vhost_net] [ 461.962609] [<ffffffffa02a46cb>] ? vhost_worker+0x10b/0x121 [vhost_net] [ 461.969447] [<ffffffffa02a45c0>] ? vhost_attach_cgroups_work+0x1b/0x1b [vhost_net] [ 461.977274] [<ffffffff8105e5d1>] ? kthread+0x76/0x7e [ 461.982440] [<ffffffff81334434>] ? kernel_thread_helper+0x4/0x10 [ 461.988667] [<ffffffff8105e55b>] ? kthread_worker_fn+0x139/0x139 [ 461.994940] [<ffffffff81334430>] ? gs_change+0x13/0x13 [ 462.000283] Code: fd 53 48 89 f3 50 74 70 48 83 be 10 01 00 00 00 75 0d be 01 00 00 00 48 89 df e8 be 9d 00 00 48 8b 9b 10 01 00 00 48 85 db 74 4d <8b> 83 a4 00 00 00 85 c0 7f 21 80 3d 2e e5 51 00 01 74 18 be 68 [ 462.021671] RIP [<ffffffff812dde61>] ipv6_select_ident+0x31/0xa7 [ 462.027977] RSP <ffff88043fd03758> [ 462.031554] CR2: 0000000000000298 [ 462.034892] ---[ end trace d158e8d429419372 ]--- [ 462.039878] Kernel panic - not syncing: Fatal exception in interrupt
Any insight will be gratefully received.
Thanks,
Chris
Signature · 2 lines
Chris Boot [email protected]
Quoted 185 lines · Chris Boot
Hi folks,
I'm working on getting a 2-node VM cluster up and running, with DRBD and Corosync/Pacemaker, running KVM VMs.
I can trigger a kernel panic in either _host_ system when running an rsync on a _guest_ VM. The rsync is simply SSH over IPv6 from a remote mail store (containing maildirs) to a local filesystem. I'm basically working on migrating a physical IMAP server to one inside a VM.
After a few seconds of fairly heavy IPv6 traffic, I get the panic below. You'll notice the panic refers to
vhost_net, but I tried without that and the kernel panics at exactly the same call point.Panic:
[ 461.232932] BUG: unable to handle kernel NULL pointer dereference at 0000000000000298 [ 461.240790] IP: [<ffffffff812dde61>] ipv6_select_ident+0x31/0xa7 [ 461.246916] PGD 42d3e8067 PUD 41facf067 PMD 0 [ 461.251537] Oops: 0000 [#1] SMP [ 461.254795] CPU 4 [ 461.256648] Modules linked in: sha1_generic hmac sha256_generic dlmconfigfs
ebtable_natebtablesacpi_cpufreqmperfcpufreq_statscpufreq_conservativecpufreq_powersavecpufreq_userspacemicrocodext_NOTRACKip_set_hash_netact_policecls_basiccls_flowcls_fwcls_u32sch_tbfsch_priosch_htbsch_hfscsch_ingresssch_sfqxt_realmxt_connlimitxt_addrtypeip_set_hash_ipiptable_rawxt_commentxt_recentipt_ULOGipt_REJECTipt_REDIRECTip6_queueipt_NETMAPipt_MASQUERADEipt_ECNipt_ecnnf_conntrack_proto_udpliteipt_CLUSTERIPipt_ahxt_timext_setxt_TCPMSSip_setxt_sctpxt_policynf_nat_tftpnf_nat_snmp_basicnf_conntrack_snmpnf_nat_sipnf_nat_pptpnf_nat_proto_grenf_nat_ircnf_nat_h323nf_nat_ftpnf_nat_amandaip6t_LOGts_kmpip6t_REJECTnf_conntrack_amandanf_conntrack_sanenf_conntrack_tftpnf_conntrack_sipnf_conntrack_proto_sctpnf_conntrack_pptpnf_conntrack_proto_grenf_conntrack_netlinknf_conntrack_netbios_nsnf_conntrack_broadcastnf_conntrack_ircnf_conntrack_h323nf_conntrack_ftpxt_TPROXYnf_tproxy_corext_tcpmssxt_pkttypext_physdevxt_ownerxt_NFQUEUExt_NFLOGnfnetlink_logxt_multiportxt_markxt_macxt_limitxt_lengthxt_iprangext_helperxt_hashlimitxt_DSCPxt_dscpxt_dccpxt_connmarkxt_CLASSIFYxt_AUDITipt_LOGxt_tcpudpxt_statenf_conntrack_ipv6nf_defrag_ipv6iptable_natnf_natxt_conntracknf_conntrack_ipv4nf_defrag_ipv4ip6table_rawnf_conntrackip6table_mangleiptable_manglenfnetlinkiptable_filterip_tablesip6table_filterip6_tablesx_tablesbridge stp bonding w83627ehfhwmon_vidcoretempcrc32c_intelaesni_intelcryptdaes_x86_64aes_genericipmi_poweroffipmi_devintfipmi_siipmi_msghandlervhost_netmacvtap macvlan tun drbdlru_cachecn loopkvm_intelkvmsnd_pcmsnd_timersnd soundcoresnd_page_allocpsmousei2c_i801processoriTCO_wdtiTCO_vendor_supporti2c_coreevdevthermal_sysjoydevserio_rawpcspkr button ext4 mbcache jbd2 crc16dm_modraid1md_modsd_modcrc_t10difusb_storageuas usbhid hid ahci libahci libata igbehci_hcdscsi_modusbcore e1000e dca [last unloaded:scsi_wait_scan][ 461.446246] [ 461.447812] Pid: 5756, comm: vhost-5753 Not tainted 3.1.0-1-amd64 #1 Supermicro X9SCL/X9SCM/X9SCL/X9SCM [ 461.457562] RIP: 0010:[<ffffffff812dde61>] [<ffffffff812dde61>] ipv6_select_ident+0x31/0xa7 [ 461.466271] RSP: 0018:ffff88043fd03758 EFLAGS: 00010202 [ 461.471708] RAX: ffff8803f7159038 RBX: 00000000000001f4 RCX: 0000000000000000 [ 461.478979] RDX: 0000000000000016 RSI: ffff88042dadcf30 RDI: ffff8803f715906e [ 461.486207] RBP: ffff8803f715906e R08: 0ea8c9feff005256 R09: 00000000000080fe [ 461.493542] R10: 5256000000000000 R11: 80fe4e6fdcfeff00 R12: 0000000040115ba9 [ 461.500851] R13: ffffffff8168ed11 R14: 0000000000000011 R15: ffff8803f7159880 [ 461.508167] FS: 0000000000000000(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000 [ 461.516444] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 461.522317] CR2: 0000000000000298 CR3: 000000042d87e000 CR4: 00000000000426e0 [ 461.529607] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 461.536916] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 461.544267] Process vhost-5753 (pid: 5756, threadinfo ffff88042cb4a000, task ffff88042d13c970) [ 461.553108] Stack: [ 461.555202] ffff8803f7159038 ffff88042be0e280 0000000000000028 ffffffff812efa7f [ 461.562840] 0000000000000246 ffff8803f7159054 ffff88042be0e318 ffff88042be0e280 [ 461.570544] ffffffffffffffa3 0000000040115ba9 ffffffff8168edc0 ffffffff812dbfe0 [ 461.578229] Call Trace: [ 461.580742] <IRQ> [ 461.582870] [<ffffffff812efa7f>] ? udp6_ufo_fragment+0x124/0x1a2 [ 461.589054] [<ffffffff812dbfe0>] ? ipv6_gso_segment+0xc0/0x155 [ 461.595140] [<ffffffff812700c6>] ? skb_gso_segment+0x208/0x28b [ 461.601198] [<ffffffffa03f236b>] ? ipv6_confirm+0x146/0x15e [nf_conntrack_ipv6] [ 461.608786] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.614227] [<ffffffff81271d64>] ? dev_hard_start_xmit+0x357/0x543 [ 461.620659] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.626440] [<ffffffffa0379745>] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 461.633581] [<ffffffff812722ff>] ? dev_queue_xmit+0x3af/0x459 [ 461.639577] [<ffffffffa03747d2>] ? br_dev_queue_push_xmit+0x72/0x76 [bridge] [ 461.646887] [<ffffffffa03791e3>] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 461.653997] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.659473] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.665485] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.671234] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.677299] [<ffffffffa0379215>] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 461.684891] [<ffffffffa03bb0e5>] ? nf_ct_zone+0xa/0x17 [nf_conntrack] [ 461.691520] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.697572] [<ffffffffa0374812>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 461.704616] [<ffffffffa0379031>] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 461.712329] [<ffffffffa037929f>] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 461.719490] [<ffffffffa037900a>] ? nf_bridge_pull_encap_header+0x1c/0x27 [bridge] [ 461.727223] [<ffffffffa0379974>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 461.734292] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.739758] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.746203] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.751950] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.758378] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.765454] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.771881] [<ffffffffa0374812>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 461.778908] [<ffffffffa03749a6>] ? br_forward+0x16/0x5a [bridge] [ 461.785041] [<ffffffffa03754db>] ? br_handle_frame_finish+0x1a1/0x20f [bridge] [ 461.792604] [<ffffffffa0379333>] ? br_nf_pre_routing_finish_ipv6+0x89/0x92 [bridge] [ 461.800513] [<ffffffffa0378e7b>] ? setup_pre_routing+0x38/0x5d [bridge] [ 461.807440] [<ffffffffa0379e65>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge] [ 461.814463] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.819908] [<ffffffff8103f89d>] ? select_task_rq_fair+0x369/0x610 [ 461.826347] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.833463] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.840532] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.846134] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.853222] [<ffffffff81036010>] ? test_tsk_need_resched+0xa/0x13 [ 461.859601] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 461.866650] [<ffffffffa0375320>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge] [ 461.873686] [<ffffffffa03756ed>] ? br_handle_frame+0x1a4/0x1bb [bridge] [ 461.880569] [<ffffffffa0375549>] ? br_handle_frame_finish+0x20f/0x20f [bridge] [ 461.888042] [<ffffffff8126f132>] ? __netif_receive_skb+0x2d6/0x415 [ 461.894467] [<ffffffff8126f2dd>] ? process_backlog+0x6c/0x123 [ 461.900414] [<ffffffff81026b37>] ? native_apic_msr_write+0x2c/0x2f [ 461.906790] [<ffffffff81271034>] ? net_rx_action+0xa1/0x1af [ 461.912625] [<ffffffff81036010>] ? test_tsk_need_resched+0xa/0x13 [ 461.919010] [<ffffffff8104ad04>] ? __do_softirq+0xb9/0x177 [ 461.924796] [<ffffffff8133452c>] ? call_softirq+0x1c/0x30 [ 461.930474] <EOI> [ 461.932684] [<ffffffff8100f845>] ? do_softirq+0x3c/0x7b [ 461.938106] [<ffffffff81271324>] ? netif_rx_ni+0x1e/0x27 [ 461.943610] [<ffffffffa0298721>] ? tun_get_user+0x39a/0x3c2 [tun] [ 461.949923] [<ffffffffa0298766>] ? tun_sendmsg+0x1d/0x1f [tun] [ 461.955938] [<ffffffffa02a6b50>] ? handle_tx+0x340/0x3de [vhost_net] [ 461.962609] [<ffffffffa02a46cb>] ? vhost_worker+0x10b/0x121 [vhost_net] [ 461.969447] [<ffffffffa02a45c0>] ? vhost_attach_cgroups_work+0x1b/0x1b [vhost_net] [ 461.977274] [<ffffffff8105e5d1>] ? kthread+0x76/0x7e [ 461.982440] [<ffffffff81334434>] ? kernel_thread_helper+0x4/0x10 [ 461.988667] [<ffffffff8105e55b>] ? kthread_worker_fn+0x139/0x139 [ 461.994940] [<ffffffff81334430>] ? gs_change+0x13/0x13 [ 462.000283] Code: fd 53 48 89 f3 50 74 70 48 83 be 10 01 00 00 00 75 0d be 01 00 00 00 48 89 df e8 be 9d 00 00 48 8b 9b 10 01 00 00 48 85 db 74 4d <8b> 83 a4 00 00 00 85 c0 7f 21 80 3d 2e e5 51 00 01 74 18 be 68 [ 462.021671] RIP [<ffffffff812dde61>] ipv6_select_ident+0x31/0xa7 [ 462.027977] RSP <ffff88043fd03758> [ 462.031554] CR2: 0000000000000298 [ 462.034892] ---[ end trace d158e8d429419372 ]--- [ 462.039878] Kernel panic - not syncing: Fatal exception in interrupt
Quoted 5 lines
Any insight will be gratefully received.
Thanks,
Chris
Is it a debian kernel ?
You need : https://lkml.org/lkml/2011/10/11/291
Quoted 19 lines · Eric Dumazet
Quoted 18 lines · Chris Boot
Hi folks,
I'm working on getting a 2-node VM cluster up and running, with DRBD and Corosync/Pacemaker, running KVM VMs.
I can trigger a kernel panic in either _host_ system when running an rsync on a _guest_ VM. The rsync is simply SSH over IPv6 from a remote mail store (containing maildirs) to a local filesystem. I'm basically working on migrating a physical IMAP server to one inside a VM.
After a few seconds of fairly heavy IPv6 traffic, I get the panic below. You'll notice the panic refers to
vhost_net, but I tried without that and the kernel panics at exactly the same call point.Panic:
[snip]
Quoted 8 lines
Quoted 5 lines
Any insight will be gratefully received.
Thanks,
ChrisIs it a debian kernel ?
You need : https://lkml.org/lkml/2011/10/11/291
Eric,
Aha, that sounds like exactly the culprit, thanks. However I can't find any reference to it in the 3.1 to 3.1.5 changelogs. Is it fixed in any of those kernels or would I have to attempt to forward-port the fix myself?
Cheers,
Chris
Signature · 2 lines
Chris Boot [email protected]
Quoted 27 lines · Chris Boot
Quoted 26 lines · Eric Dumazet
Quoted 14 lines · Chris Boot
Hi folks,
I'm working on getting a 2-node VM cluster up and running, with DRBD and Corosync/Pacemaker, running KVM VMs.
I can trigger a kernel panic in either _host_ system when running an rsync on a _guest_ VM. The rsync is simply SSH over IPv6 from a remote mail store (containing maildirs) to a local filesystem. I'm basically working on migrating a physical IMAP server to one inside a VM.
After a few seconds of fairly heavy IPv6 traffic, I get the panic below. You'll notice the panic refers to
vhost_net, but I tried without that and the kernel panics at exactly the same call point.Quoted 9 lines
Panic:
[snip]
Any insight will be gratefully received.
Thanks,
ChrisIs it a debian kernel ?
Quoted 7 lines
Quoted 1 line
You need : https://lkml.org/lkml/2011/10/11/291
Eric,
Aha, that sounds like exactly the culprit, thanks. However I can't find any reference to it in the 3.1 to 3.1.5 changelogs. Is it fixed in any of those kernels or would I have to attempt to forward-port the fix myself?
Good point, thats a different problem then, since 3.1 is not supposed to have this bug.
It seems rt->rt6i_peer points to invalid memory in your crash.
(RBX=00000000000001f4)
8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax p->refcnt
1f4+a4 -> CR2=0000000000000298
Quoted 10 lines · Eric Dumazet
Good point, thats a different problem then, since 3.1 is not supposed to have this bug.
It seems
rt->rt6i_peerpoints to invalid memory in your crash.(RBX=00000000000001f4)
8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax
p->refcnt
1f4+a4 -> CR2=0000000000000298
It would help if you can confirm latest linux tree can reproduce the bug.
Thanks !
Quoted 7 lines · Eric Dumazet
Quoted 5 lines · Eric Dumazet
Good point, thats a different problem then, since 3.1 is not supposed to have this bug.
It seems
rt->rt6i_peerpoints to invalid memory in your crash.
Quoted 7 lines
Quoted 5 lines
(RBX=00000000000001f4)
8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax
p->refcnt
1f4+a4 -> CR2=0000000000000298It would help if you can confirm latest linux tree can reproduce the bug.
Hi Eric,
I just built a v3.2-rc6-140-gb9e26df with the same config as the Debian 3.1.0 kernel. I can reproduce the bug just as easily with this kernel as with the Debian kernel. Unfortunately I wasn't able to get an entire trace, for some reason it didn't appear to be printed to the serial port and hung after the (long) list of loaded kernel modules. The crash happens at the same offset:
[ 356.683420] BUG: unable to handle kernel NULL pointer dereference at 0000000000000298 [ 356.691438] IP: [<ffffffff812ea0da>] ipv6_select_ident+0x31/0xa7 [ 356.697633] PGD 425060067 PUD 41de8d067 PMD 0 [ 356.702123] Oops: 0000 [#1] SMP [ 356.705451] CPU 4 [ 356.707366] Modules linked in: tun sha1_ssse3 sha1_generic hmac sha256_generic dlm configfs ebtable_nat ebtables acpi_cpufreq mperf cpufreq_stats cpufreq_conservative cpufreq_powersave cpufreq_userspace microcode xt_NOTRACK ip_set_hash_net act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq xt_realm xt_addrtype xt_connlimit iptable_raw ip_set_hash_ip xt_comment ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp xt_recent nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ip6_queue xt_set ip_set nf_conntrack_tftp nf_conntrack_sip nf_conntrack_sane nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp ts_kmp xt_NFLOG nfnetlink_log nf_conntrack_amanda xt_TPROXY nf_tproxy_core xt_time xt_TCPMSS xt_tcpmss xt_sctp xt_policy xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_connmark ip6t_LOG xt_CLASSIFY ip6t_REJECT xt_AUDIT nf_conntrack_ipv6
I loaded the kernel in gdb to see where it thought that was:
(gdb) list *ipv6_select_ident+0x31/0xa70xffffffff812ea0a9 is in ipv6_select_ident (net/ipv6/ip6_output.c:602). 597
598 return offset;
599 }
600
601 void ipv6_select_ident(struct frag_hdr *fhdr, struct rt6_info *rt)
602 {
603 static atomic_t ipv6_fragmentation_id;
604 int old, new;
605
606 if (rt) {
HTH,
Chris
Signature · 2 lines
Chris Boot [email protected]
Quoted 16 lines · Chris Boot
Quoted 7 lines · Eric Dumazet
Quoted 5 lines · Eric Dumazet
Good point, thats a different problem then, since 3.1 is not supposed to have this bug.
It seems
rt->rt6i_peerpoints to invalid memory in your crash.Quoted 7 lines
Quoted 5 lines
(RBX=00000000000001f4)
8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax
p->refcnt
1f4+a4 -> CR2=0000000000000298It would help if you can confirm latest linux tree can reproduce the bug.
Quoted 9 lines
Hi Eric,
I just built a v3.2-rc6-140-gb9e26df with the same config as the Debian 3.1.0 kernel. I can reproduce the bug just as easily with this kernel as with the Debian kernel. Unfortunately I wasn't able to get an entire trace, for some reason it didn't appear to be printed to the serial port and hung after the (long) list of loaded kernel modules. The crash happens at the same offset:
Thanks !
Oh well, br_netfilter fake_rtable strikes again.
I'll cook a patch in a couple of minutes...
Quoted 27 lines · Eric Dumazet
Quoted 16 lines · Chris Boot
Quoted 7 lines · Eric Dumazet
Quoted 5 lines · Eric Dumazet
Good point, thats a different problem then, since 3.1 is not supposed to have this bug.
It seems
rt->rt6i_peerpoints to invalid memory in your crash.Quoted 7 lines
Quoted 5 lines
(RBX=00000000000001f4)
8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax
p->refcnt
1f4+a4 -> CR2=0000000000000298It would help if you can confirm latest linux tree can reproduce the bug.
Quoted 9 lines
Hi Eric,
I just built a v3.2-rc6-140-gb9e26df with the same config as the Debian 3.1.0 kernel. I can reproduce the bug just as easily with this kernel as with the Debian kernel. Unfortunately I wasn't able to get an entire trace, for some reason it didn't appear to be printed to the serial port and hung after the (long) list of loaded kernel modules. The crash happens at the same offset:
Quoted 6 lines
Thanks !
Oh well,
br_netfilterfake_rtablestrikes again.I'll cook a patch in a couple of minutes...
Could you try following patch ?
Thanks !
include/net/dst.h | 1 + net/bridge/br_netfilter.c | 2 +- net/ipv4/route.c | 4 ++-- net/ipv6/ip6_output.c | 2 +- 4 files changed, 5 insertions(+), 4 deletions(-)
include/net/dst.h +1 -0
@@ -53,6 +53,7 @@ struct dst_entry {#define DST_NOHASH 0x0008#define DST_NOCACHE 0x0010#define DST_NOCOUNT 0x0020 +#define DST_NOPEER 0x0040short error;short obsolete;
net/bridge/br_netfilter.c +1 -1
@@ -141,7 +141,7 @@ void br_netfilter_rtable_init(struct net_bridge *br) rt->dst.dev = br->dev; rt->dst.path = &rt->dst;dst_init_metrics (&rt->dst, br_dst_default_metrics, true); - rt->dst.flags =DST_NOXFRM ; + rt->dst.flags =DST_NOXFRM |DST_NOPEER ; rt->dst.ops = &fake_dst_ops; }
net/ipv4/route.c +2 -2
@@ -1271,7 +1271,7 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more) {struct rtable *rt = (struct rtable *) dst; -if (rt) { +if (rt && !(rt->dst.flags &DST_NOPEER )) {if (rt->peer ==NULL )rt_bind_peer (rt, rt->rt_dst,1 );
@@ -1282,7 +1282,7 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more) iph->id =htons (inet_getid (rt->peer, more));return ; } - }else + }else if (!rt)printk (KERN_DEBUG "rt_bind_peer(0) @%p\n" ,__builtin_return_address (0 ));
net/ipv6/ip6_output.c +1 -1
@@ -603,7 +603,7 @@ void ipv6_select_ident(struct frag_hdr *fhdr, struct rt6_info *rt)static atomic_t ipv6_fragmentation_id;int old, new; -if (rt) { +if (rt && !(rt->dst.flags &DST_NOPEER )) {struct inet_peer *peer;if (!rt->rt6i_peer)
Quoted 28 lines · Eric Dumazet
Quoted 27 lines · Eric Dumazet
Quoted 15 lines · Chris Boot
Quoted 7 lines · Eric Dumazet
Quoted 5 lines · Eric Dumazet
Good point, thats a different problem then, since 3.1 is not supposed to have this bug.
It seems
rt->rt6i_peerpoints to invalid memory in your crash.Quoted 7 lines
Quoted 5 lines
(RBX=00000000000001f4)
8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax
p->refcnt
1f4+a4 -> CR2=0000000000000298It would help if you can confirm latest linux tree can reproduce the bug.
Quoted 9 lines
Hi Eric,
I just built a v3.2-rc6-140-gb9e26df with the same config as the Debian 3.1.0 kernel. I can reproduce the bug just as easily with this kernel as with the Debian kernel. Unfortunately I wasn't able to get an entire trace, for some reason it didn't appear to be printed to the serial port and hung after the (long) list of loaded kernel modules. The crash happens at the same offset:
Thanks !
Quoted 7 lines
Quoted 4 lines
Oh well,
br_netfilterfake_rtablestrikes again.I'll cook a patch in a couple of minutes...
Could you try following patch ?
[snip]
Eric,
It looks good! The rsync that caused the crash real quick hasn't done it at all with the patch applied. I'll keep testing it of course, but I think that's done it.
Many thanks indeed!
Chris
Signature · 2 lines
Chris Boot [email protected]
Quoted 33 lines · Chris Boot
Quoted 32 lines · Eric Dumazet
Quoted 31 lines · Eric Dumazet
Quoted 15 lines · Chris Boot
Quoted 7 lines · Eric Dumazet
Quoted 5 lines · Eric Dumazet
Good point, thats a different problem then, since 3.1 is not supposed to have this bug.
It seems
rt->rt6i_peerpoints to invalid memory in your crash.Quoted 7 lines
Quoted 5 lines
(RBX=00000000000001f4)
8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax
p->refcnt
1f4+a4 -> CR2=0000000000000298It would help if you can confirm latest linux tree can reproduce the bug.
Quoted 9 lines
Hi Eric,
I just built a v3.2-rc6-140-gb9e26df with the same config as the Debian 3.1.0 kernel. I can reproduce the bug just as easily with this kernel as with the Debian kernel. Unfortunately I wasn't able to get an entire trace, for some reason it didn't appear to be printed to the serial port and hung after the (long) list of loaded kernel modules. The crash happens at the same offset:
Thanks !
Oh well,
br_netfilterfake_rtablestrikes again.I'll cook a patch in a couple of minutes...
Quoted 7 lines
Quoted 3 lines
Could you try following patch ?
[snip]
Eric,
It looks good! The rsync that caused the crash real quick hasn't done it at all with the patch applied. I'll keep testing it of course, but I think that's done it.
No, sorry, false hope. The following does look rather different however:
[ 1944.056075] BUG: unable to handle kernel NULL pointer dereference at (null) [ 1944.063987] IP: [< (null)>] (null) [ 1944.069156] PGD 412c75067 PUD 413722067 PMD 0 [ 1944.073783] Oops: 0010 [#1] SMP [ 1944.077201] CPU 0 [ 1944.079136] Modules linked in: sha1_ssse3 sha1_generic hmac sha256_generic dlm configfs ebtable_nat ebtables acpi_cpufreq mperf cpufreq_stats cpufreq_conservative cpufreq_powersave cpufreq_userspace microcode xt_NOTRACK ip_set_hash_net act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_hfsc sch_htb sch_ingress sch_sfq xt_realm xt_addrtype xt_connlimit ip_set_hash_ip iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ip6_queue xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_conntrack_proto_udplite nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_sctp nf_conntrack_pptp ts_kmp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_amanda nf_conntrack_h323 nf_conntrack_ftp xt_time xt_TCPMSS xt_TPROXY xt_sctp xt_policy nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mark xt_mac xt_limit ip6t_LOG ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp ip6table_mangle xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT ipt_LOG xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables ip6table_filter ip6_tables x_tables bridge stp bonding w83627ehf hwmon_vid coretemp crc32c_intel aesni_intel cryptd aes_x86_64 aes_generic ipmi_poweroff ipmi_devintf ipmi_si ipmi_msghandler vhost_net macvtap macvlan tun drbd lru_cache cn loop kvm_intel kvm snd_pcm snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core processor psmouse iTCO_wdt thermal_sys evdev button pcspkr joydev serio_raw iTCO_vendor_support ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod usb_storage usbhid uas hid sd_mod crc_t10dif ahci libahci libata ehci_hcd igb scsi_mod dca usbcore e1000e usb_common [last unloaded: scsi_wait_scan] [ 1944.270355] [ 1944.271898] Pid: 6833, comm: vhost-6831 Not tainted 3.2.0-rc6+ #2 Supermicro X9SCL/X9SCM/X9SCL/X9SCM [ 1944.281431] RIP: 0010:[<0000000000000000>] [< (null)>] (null) [ 1944.288974] RSP: 0018:ffff88043fc03960 EFLAGS: 00010206 [ 1944.294422] RAX: ffffffffa0394940 RBX: ffff8804249ab580 RCX: 0000000000000000 [ 1944.301690] RDX: ffff880425f36f70 RSI: ffffffffa03897ac RDI: ffff880425f36f70 [ 1944.308975] RBP: ffff8803fb9a904e R08: ffff880425f36000 R09: ffff88043fc03970 [ 1944.316295] R10: ffff8804249ab580 R11: ffff8804249ab580 R12: 0000000000000014 [ 1944.323669] R13: ffff880427373200 R14: ffff880425f36000 R15: ffff88042315c000 [ 1944.330984] FS: 0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000 [ 1944.339247] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1944.345153] CR2: 0000000000000000 CR3: 0000000412e64000 CR4: 00000000000426e0 [ 1944.352471] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1944.359696] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1944.366948] Process vhost-6831 (pid: 6833, threadinfo ffff8804134b2000, task ffff880415e93080) [ 1944.375836] Stack: [ 1944.377893] ffffffff812a62e1 ffff880425f36f70 ffffffff8168f810 ffff880400000014 [ 1944.385566] ffffffffa038e82d ffff8804249ab580 ffffffffa03897ac ffff880427373200 [ 1944.393283] 0000000000000008 ffff880425f36000 ffff8804249ab580 ffff880425f36000 [ 1944.400926] Call Trace: [ 1944.403442] <IRQ> [ 1944.405668] [<ffffffff812a62e1>] ? ip_fragment+0x9d/0x62a [ 1944.411264] [<ffffffffa038e82d>] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 1944.418403] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.424500] [<ffffffffa038e2cb>] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 1944.431661] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1944.437075] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.443168] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.449299] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1944.455055] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.461162] [<ffffffffa038e2fd>] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 1944.468782] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.474866] [<ffffffffa038985e>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1944.481904] [<ffffffffa038e119>] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 1944.489657] [<ffffffffa038e387>] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 1944.496793] [<ffffffffa038e6d0>] ? br_parse_ip_options+0x3d/0x19a [bridge] [ 1944.503956] [<ffffffffa038ea5c>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 1944.510928] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1944.516355] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.522803] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.529241] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1944.534941] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.541337] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.548501] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.554921] [<ffffffffa038985e>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1944.561963] [<ffffffffa03899f2>] ? br_forward+0x16/0x5a [bridge] [ 1944.568219] [<ffffffffa038a51b>] ? br_handle_frame_finish+0x1a1/0x20f [bridge] [ 1944.575725] [<ffffffffa038e5f4>] ? br_nf_pre_routing_finish+0x1d0/0x1dd [bridge] [ 1944.583459] [<ffffffffa038dfe5>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge] [ 1944.589926] [<ffffffffa038ef4d>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge] [ 1944.597036] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1944.602482] [<ffffffffa0071d51>] ? scsi_request_fn+0x33f/0x404 [scsi_mod] [ 1944.609613] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.616701] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1944.622438] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.629566] [<ffffffffa00714dd>] ? scsi_run_queue+0x212/0x221 [scsi_mod] [ 1944.636506] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.643533] [<ffffffffa038a360>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge] [ 1944.650579] [<ffffffffa038a73c>] ? br_handle_frame+0x1b3/0x1cb [bridge] [ 1944.657440] [<ffffffffa038a589>] ? br_handle_frame_finish+0x20f/0x20f [bridge] [ 1944.664924] [<ffffffff8127ae5d>] ? __netif_receive_skb+0x324/0x41f [ 1944.671318] [<ffffffff8127afc4>] ? process_backlog+0x6c/0x123 [ 1944.677277] [<ffffffff81194394>] ? blk_done_softirq+0x65/0x74 [ 1944.683235] [<ffffffff8127ce3e>] ? net_rx_action+0xa1/0x1af [ 1944.688995] [<ffffffff81036f83>] ? test_tsk_need_resched+0xa/0x13 [ 1944.695229] [<ffffffff8104bdc8>] ? __do_softirq+0xb9/0x177 [ 1944.700887] [<ffffffff8134122c>] ? call_softirq+0x1c/0x30 [ 1944.706513] <EOI> [ 1944.708704] [<ffffffff8100f841>] ? do_softirq+0x3c/0x7b [ 1944.714116] [<ffffffff8127d12e>] ? netif_rx_ni+0x1e/0x27 [ 1944.719617] [<ffffffffa0264721>] ? tun_get_user+0x39a/0x3c2 [tun] [ 1944.725927] [<ffffffffa0264766>] ? tun_sendmsg+0x1d/0x1f [tun] [ 1944.731950] [<ffffffffa026cb50>] ? handle_tx+0x340/0x3de [vhost_net] [ 1944.738506] [<ffffffffa026a6cb>] ? vhost_worker+0x10b/0x121 [vhost_net] [ 1944.745315] [<ffffffffa026a5c0>] ? vhost_attach_cgroups_work+0x1b/0x1b [vhost_net] [ 1944.753141] [<ffffffff8105ee8d>] ? kthread+0x76/0x7e [ 1944.758332] [<ffffffff81341134>] ? kernel_thread_helper+0x4/0x10 [ 1944.764533] [<ffffffff8105ee17>] ? kthread_worker_fn+0x139/0x139 [ 1944.770759] [<ffffffff81341130>] ? gs_change+0x13/0x13 [ 1944.776109] Code: Bad RIP value. [ 1944.779528] RIP [< (null)>] (null) [ 1944.784777] RSP <ffff88043fc03960> [ 1944.788327] CR2: 0000000000000000 [ 1944.860298] ---[ end trace 7b7c89d77e0af47a ]--- [ 1944.864943] Kernel panic - not syncing: Fatal exception in interrupt [ 1944.871420] Pid: 6833, comm: vhost-6831 Tainted: G D 3.2.0-rc6+ #2 [ 1944.878608] Call Trace: [ 1944.881070] <IRQ> [<ffffffff81333a17>] ? panic+0x95/0x1a5 [ 1944.886655] [<ffffffff8133ae46>] ? oops_end+0xa9/0xb6 [ 1944.891815] [<ffffffff8133336e>] ? no_context+0x1ff/0x20e [ 1944.897343] [<ffffffff8133ce59>] ? do_page_fault+0x1a8/0x337 [ 1944.903091] [<ffffffff81293e3c>] ? sch_direct_xmit+0x85/0x12f [ 1944.908942] [<ffffffff8133a5b5>] ? page_fault+0x25/0x30 [ 1944.914291] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.920282] [<ffffffff812a62e1>] ? ip_fragment+0x9d/0x62a [ 1944.925826] [<ffffffffa038e82d>] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 1944.932948] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.938939] [<ffffffffa038e2cb>] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 1944.945967] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1944.951296] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.957272] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.963246] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1944.968857] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.974810] [<ffffffffa038e2fd>] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 1944.982277] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.988264] [<ffffffffa038985e>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1944.995213] [<ffffffffa038e119>] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 1945.002868] [<ffffffffa038e387>] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 1945.009908] [<ffffffffa038e6d0>] ? br_parse_ip_options+0x3d/0x19a [bridge] [ 1945.016924] [<ffffffffa038ea5c>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 1945.023815] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1945.029154] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.035466] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.041807] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1945.047397] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.053711] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.060641] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.066982] [<ffffffffa038985e>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1945.073908] [<ffffffffa03899f2>] ? br_forward+0x16/0x5a [bridge] [ 1945.080083] [<ffffffffa038a51b>] ? br_handle_frame_finish+0x1a1/0x20f [bridge] [ 1945.087497] [<ffffffffa038e5f4>] ? br_nf_pre_routing_finish+0x1d0/0x1dd [bridge] [ 1945.095719] [<ffffffffa038dfe5>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge] [ 1945.102187] [<ffffffffa038ef4d>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge] [ 1945.109095] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1945.114455] [<ffffffffa0071d51>] ? scsi_request_fn+0x33f/0x404 [scsi_mod] [ 1945.121420] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.128399] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1945.134054] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.141043] [<ffffffffa00714dd>] ? scsi_run_queue+0x212/0x221 [scsi_mod] [ 1945.147903] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.154875] [<ffffffffa038a360>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge] [ 1945.161830] [<ffffffffa038a73c>] ? br_handle_frame+0x1b3/0x1cb [bridge] [ 1945.168580] [<ffffffffa038a589>] ? br_handle_frame_finish+0x20f/0x20f [bridge] [ 1945.175945] [<ffffffff8127ae5d>] ? __netif_receive_skb+0x324/0x41f [ 1945.182243] [<ffffffff8127afc4>] ? process_backlog+0x6c/0x123 [ 1945.188118] [<ffffffff81194394>] ? blk_done_softirq+0x65/0x74 [ 1945.193958] [<ffffffff8127ce3e>] ? net_rx_action+0xa1/0x1af [ 1945.199638] [<ffffffff81036f83>] ? test_tsk_need_resched+0xa/0x13 [ 1945.205887] [<ffffffff8104bdc8>] ? __do_softirq+0xb9/0x177 [ 1945.211498] [<ffffffff8134122c>] ? call_softirq+0x1c/0x30 [ 1945.217018] <EOI> [<ffffffff8100f841>] ? do_softirq+0x3c/0x7b [ 1945.223149] [<ffffffff8127d12e>] ? netif_rx_ni+0x1e/0x27 [ 1945.228602] [<ffffffffa0264721>] ? tun_get_user+0x39a/0x3c2 [tun] [ 1945.234841] [<ffffffffa0264766>] ? tun_sendmsg+0x1d/0x1f [tun] [ 1945.240805] [<ffffffffa026cb50>] ? handle_tx+0x340/0x3de [vhost_net] [ 1945.247269] [<ffffffffa026a6cb>] ? vhost_worker+0x10b/0x121 [vhost_net] [ 1945.254020] [<ffffffffa026a5c0>] ? vhost_attach_cgroups_work+0x1b/0x1b [vhost_net] [ 1945.261773] [<ffffffff8105ee8d>] ? kthread+0x76/0x7e [ 1945.266844] [<ffffffff81341134>] ? kernel_thread_helper+0x4/0x10 [ 1945.272963] [<ffffffff8105ee17>] ? kthread_worker_fn+0x139/0x139 [ 1945.279100] [<ffffffff81341130>] ? gs_change+0x13/0x13 [ 1945.284417] Rebooting in 120 seconds..
(gdb) list *ip_fragment+0x9d0xffffffff812a62e1 is in ip_fragment (include/net/dst.h:209).
204 return dst_metric(dst, RTAX_FEATURES) & feature;
205 }
206
207 static inline u32 dst_mtu(const struct dst_entry *dst)
208 {
209 return dst->ops->mtu(dst);
210 }
211
212 /* RTT metrics are stored in milliseconds for user ABI, but used as jiffies */
213 static inline unsigned long dst_metric_rtt(const struct dst_entry *dst, int metric)
(gdb) list *br_flood+0xfa0x17ac is in br_dev_queue_push_xmit (net/bridge/br_forward.c:43).
38 {
39 return skb->len - (skb->protocol == htons(ETH_P_8021Q) ? VLAN_HLEN : 0);
40 }
41
42 int br_dev_queue_push_xmit(struct sk_buff *skb)
43 {
44 /* ip_fragment doesn't copy the MAC header */
45 if (nf_bridge_maybe_copy_header(skb) ||
46 (packet_length(skb) > skb->dev->mtu && !skb_is_gso(skb))) {
47 kfree_skb(skb);
Thanks,
Chris
Signature · 2 lines
Chris Boot [email protected]
Quoted 204 lines · Chris Boot
Quoted 33 lines · Chris Boot
Quoted 32 lines · Eric Dumazet
Quoted 31 lines · Eric Dumazet
Quoted 15 lines · Chris Boot
Quoted 7 lines · Eric Dumazet
Quoted 5 lines · Eric Dumazet
Good point, thats a different problem then, since 3.1 is not supposed to have this bug.
It seems
rt->rt6i_peerpoints to invalid memory in your crash.Quoted 7 lines
Quoted 5 lines
(RBX=00000000000001f4)
8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax
p->refcnt
1f4+a4 -> CR2=0000000000000298It would help if you can confirm latest linux tree can reproduce the bug.
Quoted 9 lines
Hi Eric,
I just built a v3.2-rc6-140-gb9e26df with the same config as the Debian 3.1.0 kernel. I can reproduce the bug just as easily with this kernel as with the Debian kernel. Unfortunately I wasn't able to get an entire trace, for some reason it didn't appear to be printed to the serial port and hung after the (long) list of loaded kernel modules. The crash happens at the same offset:
Thanks !
Oh well,
br_netfilterfake_rtablestrikes again.I'll cook a patch in a couple of minutes...
Quoted 7 lines
Quoted 3 lines
Could you try following patch ?
[snip]
Eric,
It looks good! The rsync that caused the crash real quick hasn't done it at all with the patch applied. I'll keep testing it of course, but I think that's done it.
No, sorry, false hope. The following does look rather different however:
[ 1944.056075] BUG: unable to handle kernel NULL pointer dereference at (null) [ 1944.063987] IP: [< (null)>] (null) [ 1944.069156] PGD 412c75067 PUD 413722067 PMD 0 [ 1944.073783] Oops: 0010 [#1] SMP [ 1944.077201] CPU 0 [ 1944.079136] Modules linked in: sha1_ssse3 sha1_generic hmac sha256_generic dlm configfs ebtable_nat ebtables acpi_cpufreq mperf cpufreq_stats cpufreq_conservative cpufreq_powersave cpufreq_userspace microcode xt_NOTRACK ip_set_hash_net act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_hfsc sch_htb sch_ingress sch_sfq xt_realm xt_addrtype xt_connlimit ip_set_hash_ip iptable_raw xt_comment xt_recent ipt_ULOG ipt_REJECT ipt_REDIRECT ipt_NETMAP ipt_MASQUERADE ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ip6_queue xt_set ip_set nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_conntrack_proto_udplite nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_sctp nf_conntrack_pptp ts_kmp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_amanda nf_conntrack_h323 nf_conntrack_ftp xt_time xt_TCPMSS xt_TPROXY xt_sctp xt_policy nf_tproxy_core xt_tcpmss xt_pkttype xt_physdev xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mark xt_mac xt_limit ip6t_LOG ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp ip6table_mangle xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT ipt_LOG xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables ip6table_filter ip6_tables x_tables bridge stp bonding w83627ehf hwmon_vid coretemp crc32c_intel aesni_intel cryptd aes_x86_64 aes_generic ipmi_poweroff ipmi_devintf ipmi_si ipmi_msghandler vhost_net macvtap macvlan tun drbd lru_cache cn loop kvm_intel kvm snd_pcm snd_timer snd soundcore snd_page_alloc i2c_i801 i2c_core processor psmouse iTCO_wdt thermal_sys evdev button pcspkr joydev serio_raw iTCO_vendor_support ext4 mbcache jbd2 crc16 dm_mod raid1 md_mod usb_storage usbhid uas hid sd_mod crc_t10dif ahci libahci libata ehci_hcd igb scsi_mod dca usbcore e1000e usb_common [last unloaded: scsi_wait_scan] [ 1944.270355] [ 1944.271898] Pid: 6833, comm: vhost-6831 Not tainted 3.2.0-rc6+ #2 Supermicro X9SCL/X9SCM/X9SCL/X9SCM [ 1944.281431] RIP: 0010:[<0000000000000000>] [< (null)>] (null) [ 1944.288974] RSP: 0018:ffff88043fc03960 EFLAGS: 00010206 [ 1944.294422] RAX: ffffffffa0394940 RBX: ffff8804249ab580 RCX: 0000000000000000 [ 1944.301690] RDX: ffff880425f36f70 RSI: ffffffffa03897ac RDI: ffff880425f36f70 [ 1944.308975] RBP: ffff8803fb9a904e R08: ffff880425f36000 R09: ffff88043fc03970 [ 1944.316295] R10: ffff8804249ab580 R11: ffff8804249ab580 R12: 0000000000000014 [ 1944.323669] R13: ffff880427373200 R14: ffff880425f36000 R15: ffff88042315c000 [ 1944.330984] FS: 0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000 [ 1944.339247] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1944.345153] CR2: 0000000000000000 CR3: 0000000412e64000 CR4: 00000000000426e0 [ 1944.352471] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1944.359696] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1944.366948] Process vhost-6831 (pid: 6833, threadinfo ffff8804134b2000, task ffff880415e93080) [ 1944.375836] Stack: [ 1944.377893] ffffffff812a62e1 ffff880425f36f70 ffffffff8168f810 ffff880400000014 [ 1944.385566] ffffffffa038e82d ffff8804249ab580 ffffffffa03897ac ffff880427373200 [ 1944.393283] 0000000000000008 ffff880425f36000 ffff8804249ab580 ffff880425f36000 [ 1944.400926] Call Trace: [ 1944.403442] <IRQ> [ 1944.405668] [<ffffffff812a62e1>] ? ip_fragment+0x9d/0x62a [ 1944.411264] [<ffffffffa038e82d>] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 1944.418403] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.424500] [<ffffffffa038e2cb>] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 1944.431661] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1944.437075] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.443168] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.449299] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1944.455055] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.461162] [<ffffffffa038e2fd>] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 1944.468782] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.474866] [<ffffffffa038985e>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1944.481904] [<ffffffffa038e119>] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 1944.489657] [<ffffffffa038e387>] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 1944.496793] [<ffffffffa038e6d0>] ? br_parse_ip_options+0x3d/0x19a [bridge] [ 1944.503956] [<ffffffffa038ea5c>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 1944.510928] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1944.516355] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.522803] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.529241] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1944.534941] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.541337] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.548501] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1944.554921] [<ffffffffa038985e>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1944.561963] [<ffffffffa03899f2>] ? br_forward+0x16/0x5a [bridge] [ 1944.568219] [<ffffffffa038a51b>] ? br_handle_frame_finish+0x1a1/0x20f [bridge] [ 1944.575725] [<ffffffffa038e5f4>] ? br_nf_pre_routing_finish+0x1d0/0x1dd [bridge] [ 1944.583459] [<ffffffffa038dfe5>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge] [ 1944.589926] [<ffffffffa038ef4d>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge] [ 1944.597036] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1944.602482] [<ffffffffa0071d51>] ? scsi_request_fn+0x33f/0x404 [scsi_mod] [ 1944.609613] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.616701] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1944.622438] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.629566] [<ffffffffa00714dd>] ? scsi_run_queue+0x212/0x221 [scsi_mod] [ 1944.636506] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1944.643533] [<ffffffffa038a360>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge] [ 1944.650579] [<ffffffffa038a73c>] ? br_handle_frame+0x1b3/0x1cb [bridge] [ 1944.657440] [<ffffffffa038a589>] ? br_handle_frame_finish+0x20f/0x20f [bridge] [ 1944.664924] [<ffffffff8127ae5d>] ? __netif_receive_skb+0x324/0x41f [ 1944.671318] [<ffffffff8127afc4>] ? process_backlog+0x6c/0x123 [ 1944.677277] [<ffffffff81194394>] ? blk_done_softirq+0x65/0x74 [ 1944.683235] [<ffffffff8127ce3e>] ? net_rx_action+0xa1/0x1af [ 1944.688995] [<ffffffff81036f83>] ? test_tsk_need_resched+0xa/0x13 [ 1944.695229] [<ffffffff8104bdc8>] ? __do_softirq+0xb9/0x177 [ 1944.700887] [<ffffffff8134122c>] ? call_softirq+0x1c/0x30 [ 1944.706513] <EOI> [ 1944.708704] [<ffffffff8100f841>] ? do_softirq+0x3c/0x7b [ 1944.714116] [<ffffffff8127d12e>] ? netif_rx_ni+0x1e/0x27 [ 1944.719617] [<ffffffffa0264721>] ? tun_get_user+0x39a/0x3c2 [tun] [ 1944.725927] [<ffffffffa0264766>] ? tun_sendmsg+0x1d/0x1f [tun] [ 1944.731950] [<ffffffffa026cb50>] ? handle_tx+0x340/0x3de [vhost_net] [ 1944.738506] [<ffffffffa026a6cb>] ? vhost_worker+0x10b/0x121 [vhost_net] [ 1944.745315] [<ffffffffa026a5c0>] ? vhost_attach_cgroups_work+0x1b/0x1b [vhost_net] [ 1944.753141] [<ffffffff8105ee8d>] ? kthread+0x76/0x7e [ 1944.758332] [<ffffffff81341134>] ? kernel_thread_helper+0x4/0x10 [ 1944.764533] [<ffffffff8105ee17>] ? kthread_worker_fn+0x139/0x139 [ 1944.770759] [<ffffffff81341130>] ? gs_change+0x13/0x13 [ 1944.776109] Code: Bad RIP value. [ 1944.779528] RIP [< (null)>] (null) [ 1944.784777] RSP <ffff88043fc03960> [ 1944.788327] CR2: 0000000000000000 [ 1944.860298] ---[ end trace 7b7c89d77e0af47a ]--- [ 1944.864943] Kernel panic - not syncing: Fatal exception in interrupt [ 1944.871420] Pid: 6833, comm: vhost-6831 Tainted: G D 3.2.0-rc6+ #2 [ 1944.878608] Call Trace: [ 1944.881070] <IRQ> [<ffffffff81333a17>] ? panic+0x95/0x1a5 [ 1944.886655] [<ffffffff8133ae46>] ? oops_end+0xa9/0xb6 [ 1944.891815] [<ffffffff8133336e>] ? no_context+0x1ff/0x20e [ 1944.897343] [<ffffffff8133ce59>] ? do_page_fault+0x1a8/0x337 [ 1944.903091] [<ffffffff81293e3c>] ? sch_direct_xmit+0x85/0x12f [ 1944.908942] [<ffffffff8133a5b5>] ? page_fault+0x25/0x30 [ 1944.914291] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.920282] [<ffffffff812a62e1>] ? ip_fragment+0x9d/0x62a [ 1944.925826] [<ffffffffa038e82d>] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 1944.932948] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.938939] [<ffffffffa038e2cb>] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 1944.945967] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1944.951296] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.957272] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.963246] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1944.968857] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.974810] [<ffffffffa038e2fd>] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 1944.982277] [<ffffffffa03897ac>] ? br_flood+0xfa/0xfa [bridge] [ 1944.988264] [<ffffffffa038985e>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1944.995213] [<ffffffffa038e119>] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 1945.002868] [<ffffffffa038e387>] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 1945.009908] [<ffffffffa038e6d0>] ? br_parse_ip_options+0x3d/0x19a [bridge] [ 1945.016924] [<ffffffffa038ea5c>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 1945.023815] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1945.029154] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.035466] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.041807] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1945.047397] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.053711] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.060641] [<ffffffffa0389918>] ? __br_deliver+0xa0/0xa0 [bridge] [ 1945.066982] [<ffffffffa038985e>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 1945.073908] [<ffffffffa03899f2>] ? br_forward+0x16/0x5a [bridge] [ 1945.080083] [<ffffffffa038a51b>] ? br_handle_frame_finish+0x1a1/0x20f [bridge] [ 1945.087497] [<ffffffffa038e5f4>] ? br_nf_pre_routing_finish+0x1d0/0x1dd [bridge] [ 1945.095719] [<ffffffffa038dfe5>] ? NF_HOOK_THRESH+0x3b/0x55 [bridge] [ 1945.102187] [<ffffffffa038ef4d>] ? br_nf_pre_routing+0x3e8/0x3f5 [bridge] [ 1945.109095] [<ffffffff8129db6d>] ? nf_iterate+0x41/0x77 [ 1945.114455] [<ffffffffa0071d51>] ? scsi_request_fn+0x33f/0x404 [scsi_mod] [ 1945.121420] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.128399] [<ffffffff8129dc0b>] ? nf_hook_slow+0x68/0x101 [ 1945.134054] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.141043] [<ffffffffa00714dd>] ? scsi_run_queue+0x212/0x221 [scsi_mod] [ 1945.147903] [<ffffffffa038a37a>] ? NF_HOOK.constprop.4+0x56/0x56 [bridge] [ 1945.154875] [<ffffffffa038a360>] ? NF_HOOK.constprop.4+0x3c/0x56 [bridge] [ 1945.161830] [<ffffffffa038a73c>] ? br_handle_frame+0x1b3/0x1cb [bridge] [ 1945.168580] [<ffffffffa038a589>] ? br_handle_frame_finish+0x20f/0x20f [bridge] [ 1945.175945] [<ffffffff8127ae5d>] ? __netif_receive_skb+0x324/0x41f [ 1945.182243] [<ffffffff8127afc4>] ? process_backlog+0x6c/0x123 [ 1945.188118] [<ffffffff81194394>] ? blk_done_softirq+0x65/0x74 [ 1945.193958] [<ffffffff8127ce3e>] ? net_rx_action+0xa1/0x1af [ 1945.199638] [<ffffffff81036f83>] ? test_tsk_need_resched+0xa/0x13 [ 1945.205887] [<ffffffff8104bdc8>] ? __do_softirq+0xb9/0x177 [ 1945.211498] [<ffffffff8134122c>] ? call_softirq+0x1c/0x30 [ 1945.217018] <EOI> [<ffffffff8100f841>] ? do_softirq+0x3c/0x7b [ 1945.223149] [<ffffffff8127d12e>] ? netif_rx_ni+0x1e/0x27 [ 1945.228602] [<ffffffffa0264721>] ? tun_get_user+0x39a/0x3c2 [tun] [ 1945.234841] [<ffffffffa0264766>] ? tun_sendmsg+0x1d/0x1f [tun] [ 1945.240805] [<ffffffffa026cb50>] ? handle_tx+0x340/0x3de [vhost_net] [ 1945.247269] [<ffffffffa026a6cb>] ? vhost_worker+0x10b/0x121 [vhost_net] [ 1945.254020] [<ffffffffa026a5c0>] ? vhost_attach_cgroups_work+0x1b/0x1b [vhost_net] [ 1945.261773] [<ffffffff8105ee8d>] ? kthread+0x76/0x7e [ 1945.266844] [<ffffffff81341134>] ? kernel_thread_helper+0x4/0x10 [ 1945.272963] [<ffffffff8105ee17>] ? kthread_worker_fn+0x139/0x139 [ 1945.279100] [<ffffffff81341130>] ? gs_change+0x13/0x13 [ 1945.284417] Rebooting in 120 seconds..
Quoted 11 lines
(gdb) list *ip_fragment+0x9d
0xffffffff812a62e1is inip_fragment(include/net/dst.h:209).
204 returndst_metric(dst, RTAX_FEATURES)& feature;
205 }
206
207 static inline u32dst_mtu(const struct dst_entry *dst)
208 {
209 returndst->ops->mtu(dst);
210 }
211
212 /* RTT metrics are stored in milliseconds for user ABI, but used as jiffies */
This one is different, its not IPv6 related but IPv4 :
fake_dst_ops lacks a .mtu() field
Bug added in commit 618f9bc74a039da76 (net: Move mtu handling down to the protocol depended handlers)
Here is an updated patch, thanks again !
include/net/dst.h | 1 + net/bridge/br_netfilter.c | 8 +++++++- net/ipv4/route.c | 4 ++-- net/ipv6/ip6_output.c | 2 +- 4 files changed, 11 insertions(+), 4 deletions(-)
include/net/dst.h +1 -0
@@ -53,6 +53,7 @@ struct dst_entry {#define DST_NOHASH 0x0008#define DST_NOCACHE 0x0010#define DST_NOCOUNT 0x0020 +#define DST_NOPEER 0x0040short error;short obsolete;
net/bridge/br_netfilter.c +7 -1
@@ -114,12 +114,18 @@ static struct neighbour *fake_neigh_lookup(const struct dst_entry *dst, const voreturn NULL ; } +static unsigned int fake_mtu (const struct dst_entry *dst) +{ +return dst->dev->mtu; +} +static struct dst_ops fake_dst_ops = { .family =AF_INET , .protocol =cpu_to_be16 (ETH_P_IP ), .update_pmtu = fake_update_pmtu, .cow_metrics = fake_cow_metrics, .neigh_lookup = fake_neigh_lookup, + .mtu = fake_mtu, };/*
@@ -141,7 +147,7 @@ void br_netfilter_rtable_init(struct net_bridge *br) rt->dst.dev = br->dev; rt->dst.path = &rt->dst;dst_init_metrics (&rt->dst, br_dst_default_metrics, true); - rt->dst.flags =DST_NOXFRM ; + rt->dst.flags =DST_NOXFRM |DST_NOPEER ; rt->dst.ops = &fake_dst_ops; }
net/ipv4/route.c +2 -2
@@ -1366,7 +1366,7 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more) {struct rtable *rt = (struct rtable *) dst; -if (rt) { +if (rt && !(rt->dst.flags &DST_NOPEER )) {if (rt->peer ==NULL )rt_bind_peer (rt, rt->rt_dst,1 );
@@ -1377,7 +1377,7 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more) iph->id =htons (inet_getid (rt->peer, more));return ; } - }else + }else if (!rt)printk (KERN_DEBUG "rt_bind_peer(0) @%p\n" ,__builtin_return_address (0 ));
net/ipv6/ip6_output.c +1 -1
@@ -603,7 +603,7 @@ void ipv6_select_ident(struct frag_hdr *fhdr, struct rt6_info *rt)static atomic_t ipv6_fragmentation_id;int old, new; -if (rt) { +if (rt && !(rt->dst.flags &DST_NOPEER )) {struct inet_peer *peer;if (!rt->rt6i_peer)
PATCH 1/1 bridge: provide a mtu() method for fake_dst_ops
Commit 618f9bc74a039da76 (net: Move mtu handling down to the protocol depended handlers) forgot the bridge netfilter case, adding a NULL dereference in ip_fragment().
--- net/bridge/br_netfilter.c | 6 ++++++ 1 file changed, 6 insertions(+)
net/bridge/br_netfilter.c +6 -0
@@ -114,12 +114,18 @@ static struct neighbour *fake_neigh_lookup(const struct dst_entry *dst, const voreturn NULL ; } +static unsigned int fake_mtu (const struct dst_entry *dst) +{ +return dst->dev->mtu; +} +static struct dst_ops fake_dst_ops = { .family =AF_INET , .protocol =cpu_to_be16 (ETH_P_IP ), .update_pmtu = fake_update_pmtu, .cow_metrics = fake_cow_metrics, .neigh_lookup = fake_neigh_lookup, + .mtu = fake_mtu, };/*
Re: [PATCH] bridge: provide a mtu() method for fake_dst_ops
Quoted 7 lines · Eric Dumazet
Commit
618f9bc74a039da76(net: Move mtu handling down to the protocol depended handlers) forgot the bridge netfilter case, adding a NULL dereference inip_fragment().Reported-by Chris Boot <[email protected]>CC Steffen Klassert <[email protected]>Signed-off-by Eric Dumazet <[email protected]>
As I said in the other mail, the bug is probaply arround for longer. Anyway,
Thanks!
Quoted 14 lines · Eric Dumazet
Quoted 12 lines · Chris Boot
(gdb) list *ip_fragment+0x9d
0xffffffff812a62e1is inip_fragment(include/net/dst.h:209).
204 returndst_metric(dst, RTAX_FEATURES)& feature;
205 }
206
207 static inline u32dst_mtu(const struct dst_entry *dst)
208 {
209 returndst->ops->mtu(dst);
210 }
211
212 /* RTT metrics are stored in milliseconds for user ABI, but used as jiffies */
Quoted 7 lines
This one is different, its not IPv6 related but IPv4 :
fake_dst_opslacks a .mtu() fieldBug added in commit
618f9bc74a039da76(net: Move mtu handling down to the protocol depended handlers)
Before I did this patch, I changed the .default_mtu field of "struct dst_ops" to .mtu and converted all users of default_mtu to mtu. fake_dst_ops never had a default_mtu field, so I did not touch it. I guess this bug is arround since fake_dst_ops exists. It's now just much easier to trigger as commit 618f9bc74a039da76 changed dst_mtu() to call dst->ops->mtu() unconditionally.
Quoted 23 lines · Steffen Klassert
Quoted 14 lines · Eric Dumazet
Quoted 12 lines · Chris Boot
(gdb) list *ip_fragment+0x9d
0xffffffff812a62e1is inip_fragment(include/net/dst.h:209).
204 returndst_metric(dst, RTAX_FEATURES)& feature;
205 }
206
207 static inline u32dst_mtu(const struct dst_entry *dst)
208 {
209 returndst->ops->mtu(dst);
210 }
211
212 /* RTT metrics are stored in milliseconds for user ABI, but used as jiffies */Quoted 7 lines
This one is different, its not IPv6 related but IPv4 :
fake_dst_opslacks a .mtu() fieldBug added in commit
618f9bc74a039da76(net: Move mtu handling down to the protocol depended handlers)
Quoted 5 lines
Before I did this patch, I changed the .default_mtu field of "struct
dst_ops" to .mtu and converted all users ofdefault_mtuto mtu.fake_dst_opsnever had adefault_mtufield, so I did not touch it. I guess this bug is arround sincefake_dst_opsexists. It's now just much easier to trigger as commit618f9bc74a039da76changeddst_mtu()to calldst->ops->mtu()unconditionally.
I dont think so.
Prior to 618f9bc74a0, we were calling
static inline u32 dst_mtu(const struct dst_entry *dst)
{
u32 mtu = dst_metric_raw(dst, RTAX_MTU);
if (!mtu)
mtu = dst->ops->mtu(dst);
return mtu;
}with dst = fake_rtable
and we did :
dst_init_metrics(&rt->dst, br_dst_default_metrics, true);
so dst_metric_raw(dst, RTAX_MTU) was returning 1500
So bug is new, we dont need to backport this fix.
Quoted 15 lines · Eric Dumazet
Prior to
618f9bc74a0, we were callingstatic inline u32
dst_mtu(const struct dst_entry *dst){ u32 mtu = dst_metric_raw(dst, RTAX_MTU); if (!mtu) mtu = dst->ops->mtu(dst); return mtu; }with dst =
fake_rtable
Quoted 8 lines
and we did :
dst_init_metrics(&rt->dst, br_dst_default_metrics, true);so
dst_metric_raw(dst, RTAX_MTU)was returning 1500So bug is new, we dont need to backport this fix.
Ok, I see. Thanks for pointing it out!
Quoted 18 lines · Steffen Klassert
Quoted 17 lines · Eric Dumazet
Prior to
618f9bc74a0, we were callingstatic inline u32
dst_mtu(const struct dst_entry *dst){ u32 mtu = dst_metric_raw(dst, RTAX_MTU); if (!mtu) mtu = dst->ops->mtu(dst); return mtu; }with dst =
fake_rtableand we did :
Quoted 8 lines
Quoted 6 lines
dst_init_metrics(&rt->dst, br_dst_default_metrics, true);so
dst_metric_raw(dst, RTAX_MTU)was returning 1500So bug is new, we dont need to backport this fix.
Ok, I see. Thanks for pointing it out!
You're welcome.
By the way we no longer have this 1500 hard coded, not sure it it matters or not, but it looks better anyway.
Quoted 4 lines · Eric Dumazet
By the way we no longer have this 1500 hard coded, not sure it it matters or not, but it looks better anyway.
It looks like the bridge code sets the mtu value on this metric to the mtu of the bridge device in br_add_if(). So seems to be not really fixed to 1500. Not sure why netfilter really needs to have this 1500 in the metric before the mtu is initialized from the bridge code.
We could make fake_mtu() look like dn_dst_mtu(). The mtu of the bridge device should be reflected in the metric once the bridge code initialized it. Also learned pmtu values would be taken into account then.
Quoted 47 lines · Eric Dumazet
Quoted 37 lines · Chris Boot
Quoted 35 lines · Chris Boot
Quoted 34 lines · Eric Dumazet
Quoted 25 lines · Eric Dumazet
Quoted 15 lines · Chris Boot
Quoted 7 lines · Eric Dumazet
Quoted 5 lines · Eric Dumazet
Good point, thats a different problem then, since 3.1 is not supposed to have this bug.
It seems
rt->rt6i_peerpoints to invalid memory in your crash.Quoted 7 lines
Quoted 5 lines
(RBX=00000000000001f4)
8b 83 a4 00 00 00 mov 0xa4(%rbx),%eax
p->refcnt
1f4+a4 -> CR2=0000000000000298It would help if you can confirm latest linux tree can reproduce the bug.
Quoted 9 lines
Hi Eric,
I just built a v3.2-rc6-140-gb9e26df with the same config as the Debian 3.1.0 kernel. I can reproduce the bug just as easily with this kernel as with the Debian kernel. Unfortunately I wasn't able to get an entire trace, for some reason it didn't appear to be printed to the serial port and hung after the (long) list of loaded kernel modules. The crash happens at the same offset:
Quoted 6 lines
Thanks !
Oh well,
br_netfilterfake_rtablestrikes again.I'll cook a patch in a couple of minutes...
Could you try following patch ?
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
[snip]
Eric,
It looks good! The rsync that caused the crash real quick hasn't done it at all with the patch applied. I'll keep testing it of course, but I think that's done it.
No, sorry, false hope. The following does look rather different however:
[snip]
This one is different, its not IPv6 related but IPv4 :
Quoted 8 lines
fake_dst_opslacks a .mtu() fieldBug added in commit
618f9bc74a039da76(net: Move mtu handling down to the protocol depended handlers)Here is an updated patch, thanks again !
[snip]
Eric,
So far so good. I've had this running for several hours this morning with more of the prodding that would normally have crashed it, both IPv4 and IPv6, and it's holding up well.
Thanks again,
Chris
Signature · 2 lines
Chris Boot [email protected]
Eric Dumazet
Tested-by Chris BootQuoted 6 lines · Chris Boot
Eric,
So far so good. I've had this running for several hours this morning with more of the prodding that would normally have crashed it, both IPv4 and IPv6, and it's holding up well.
Thanks for testing.
Here is the official patch then (the .mtu() bit belongs to a separate patch)
[PATCH] net: introduce DST_NOPEER dst flag
Chris Boot reported crashes occurring in ipv6_select_ident().
[ 461.457562] RIP: 0010:[<ffffffff812dde61>] [<ffffffff812dde61>]
ipv6_select_ident+0x31/0xa7
[ 461.578229] Call Trace: [ 461.580742] <IRQ> [ 461.582870] [<ffffffff812efa7f>] ? udp6_ufo_fragment+0x124/0x1a2 [ 461.589054] [<ffffffff812dbfe0>] ? ipv6_gso_segment+0xc0/0x155 [ 461.595140] [<ffffffff812700c6>] ? skb_gso_segment+0x208/0x28b [ 461.601198] [<ffffffffa03f236b>] ? ipv6_confirm+0x146/0x15e [nf_conntrack_ipv6] [ 461.608786] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.614227] [<ffffffff81271d64>] ? dev_hard_start_xmit+0x357/0x543 [ 461.620659] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.626440] [<ffffffffa0379745>] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 461.633581] [<ffffffff812722ff>] ? dev_queue_xmit+0x3af/0x459 [ 461.639577] [<ffffffffa03747d2>] ? br_dev_queue_push_xmit+0x72/0x76 [bridge] [ 461.646887] [<ffffffffa03791e3>] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 461.653997] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.659473] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.665485] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.671234] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.677299] [<ffffffffa0379215>] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 461.684891] [<ffffffffa03bb0e5>] ? nf_ct_zone+0xa/0x17 [nf_conntrack] [ 461.691520] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.697572] [<ffffffffa0374812>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 461.704616] [<ffffffffa0379031>] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 461.712329] [<ffffffffa037929f>] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 461.719490] [<ffffffffa037900a>] ? nf_bridge_pull_encap_header+0x1c/0x27 [bridge] [ 461.727223] [<ffffffffa0379974>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 461.734292] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.739758] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.746203] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.751950] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.758378] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56
[bridge]
This is caused by bridge netfilter special dst_entry (fake_rtable), a special shared entry, where attaching an inetpeer makes no sense.
Problem is present since commit 87c48fa3b46 (ipv6: make fragment identifications less predictable)
Introduce DST_NOPEER dst flag and make sure ipv6_select_ident() and __ip_select_ident() fallback to the 'no peer attached' handling.
--- include/net/dst.h | 1 + net/bridge/br_netfilter.c | 2 +- net/ipv4/route.c | 4 ++-- net/ipv6/ip6_output.c | 2 +- 4 files changed, 5 insertions(+), 4 deletions(-)
include/net/dst.h +1 -0
@@ -53,6 +53,7 @@ struct dst_entry {#define DST_NOHASH 0x0008#define DST_NOCACHE 0x0010#define DST_NOCOUNT 0x0020 +#define DST_NOPEER 0x0040short error;short obsolete;
net/bridge/br_netfilter.c +1 -1
@@ -141,7 +141,7 @@ void br_netfilter_rtable_init(struct net_bridge *br) rt->dst.dev = br->dev; rt->dst.path = &rt->dst;dst_init_metrics (&rt->dst, br_dst_default_metrics, true); - rt->dst.flags =DST_NOXFRM ; + rt->dst.flags =DST_NOXFRM |DST_NOPEER ; rt->dst.ops = &fake_dst_ops; }
net/ipv4/route.c +2 -2
@@ -1366,7 +1366,7 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more) {struct rtable *rt = (struct rtable *) dst; -if (rt) { +if (rt && !(rt->dst.flags &DST_NOPEER )) {if (rt->peer ==NULL )rt_bind_peer (rt, rt->rt_dst,1 );
@@ -1377,7 +1377,7 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more) iph->id =htons (inet_getid (rt->peer, more));return ; } - }else + }else if (!rt)printk (KERN_DEBUG "rt_bind_peer(0) @%p\n" ,__builtin_return_address (0 ));
net/ipv6/ip6_output.c +1 -1
@@ -603,7 +603,7 @@ void ipv6_select_ident(struct frag_hdr *fhdr, struct rt6_info *rt)static atomic_t ipv6_fragmentation_id;int old, new; -if (rt) { +if (rt && !(rt->dst.flags &DST_NOPEER )) {struct inet_peer *peer;if (!rt->rt6i_peer)
Quoted 134 lines · Eric Dumazet
Quoted 6 lines · Chris Boot
Eric,
So far so good. I've had this running for several hours this morning with more of the prodding that would normally have crashed it, both IPv4 and IPv6, and it's holding up well.
Thanks for testing.
Here is the official patch then (the .mtu() bit belongs to a separate patch)
[PATCH] net: introduce
DST_NOPEERdst flagChris Boot reported crashes occurring in
ipv6_select_ident().[ 461.457562] RIP: 0010:[<ffffffff812dde61>] [<ffffffff812dde61>]
ipv6_select_ident+0x31/0xa7[ 461.578229] Call Trace: [ 461.580742]<IRQ> [ 461.582870] [<ffffffff812efa7f>] ? udp6_ufo_fragment+0x124/0x1a2 [ 461.589054] [<ffffffff812dbfe0>] ? ipv6_gso_segment+0xc0/0x155 [ 461.595140] [<ffffffff812700c6>] ? skb_gso_segment+0x208/0x28b [ 461.601198] [<ffffffffa03f236b>] ? ipv6_confirm+0x146/0x15e [nf_conntrack_ipv6] [ 461.608786] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.614227] [<ffffffff81271d64>] ? dev_hard_start_xmit+0x357/0x543 [ 461.620659] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.626440] [<ffffffffa0379745>] ? br_parse_ip_options+0x19a/0x19a [bridge] [ 461.633581] [<ffffffff812722ff>] ? dev_queue_xmit+0x3af/0x459 [ 461.639577] [<ffffffffa03747d2>] ? br_dev_queue_push_xmit+0x72/0x76 [bridge] [ 461.646887] [<ffffffffa03791e3>] ? br_nf_post_routing+0x17d/0x18f [bridge] [ 461.653997] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.659473] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.665485] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.671234] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.677299] [<ffffffffa0379215>] ? nf_bridge_update_protocol+0x20/0x20 [bridge] [ 461.684891] [<ffffffffa03bb0e5>] ? nf_ct_zone+0xa/0x17 [nf_conntrack] [ 461.691520] [<ffffffffa0374760>] ? br_flood+0xfa/0xfa [bridge] [ 461.697572] [<ffffffffa0374812>] ? NF_HOOK.constprop.8+0x3c/0x56 [bridge] [ 461.704616] [<ffffffffa0379031>] ? nf_bridge_push_encap_header+0x1c/0x26 [bridge] [ 461.712329] [<ffffffffa037929f>] ? br_nf_forward_finish+0x8a/0x95 [bridge] [ 461.719490] [<ffffffffa037900a>] ? nf_bridge_pull_encap_header+0x1c/0x27 [bridge] [ 461.727223] [<ffffffffa0379974>] ? br_nf_forward_ip+0x1c0/0x1d4 [bridge] [ 461.734292] [<ffffffff81291c4d>] ? nf_iterate+0x41/0x77 [ 461.739758] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.746203] [<ffffffff81291cf6>] ? nf_hook_slow+0x73/0x111 [ 461.751950] [<ffffffffa03748cc>] ? __br_deliver+0xa0/0xa0 [bridge] [ 461.758378] [<ffffffffa037533a>] ? NF_HOOK.constprop.4+0x56/0x56[bridge]
This is caused by bridge netfilter special
dst_entry(fake_rtable), a special shared entry, where attaching an inetpeer makes no sense.Problem is present since commit
87c48fa3b46(ipv6: make fragment identifications less predictable)Introduce
DST_NOPEERdst flag and make sureipv6_select_ident()and__ip_select_ident()fallback to the 'no peer attached' handling.Reported-by Chris Boot<[email protected]>Tested-by Chris Boot<[email protected]>Signed-off-by Eric Dumazet<[email protected]>--- include/net/dst.h | 1 + net/bridge/br_netfilter.c | 2 +- net/ipv4/route.c | 4 ++-- net/ipv6/ip6_output.c | 2 +- 4 files changed, 5 insertions(+), 4 deletions(-)include/net/dst.h +1
-0@@ -53,6 +53,7 @@ struct dst_entry {#define DST_NOHASH 0x0008#define DST_NOCACHE 0x0010#define DST_NOCOUNT 0x0020 +#define DST_NOPEER 0x0040short error;short obsolete;net/bridge/br_netfilter.c +1
-1@@ -141,7 +141,7 @@ void br_netfilter_rtable_init(struct net_bridge *br) rt->dst.dev = br->dev; rt->dst.path =&rt->dst;dst_init_metrics (&rt->dst, br_dst_default_metrics, true); - rt->dst.flags =DST_NOXFRM ; + rt->dst.flags =DST_NOXFRM |DST_NOPEER ; rt->dst.ops =&fake_dst_ops; }net/ipv4/route.c +2
-2@@ -1366,7 +1366,7 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more) {struct rtable *rt = (struct rtable *) dst; -if (rt) { +if (rt&& !(rt->dst.flags&DST_NOPEER )) {if (rt->peer ==NULL )rt_bind_peer (rt, rt->rt_dst,1 );@@ -1377,7 +1377,7 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more) iph->id =htons (inet_getid (rt->peer, more));return ; } - }else + }else if (!rt)printk (KERN_DEBUG "rt_bind_peer(0) @%p\n" ,__builtin_return_address (0 ));net/ipv6/ip6_output.c +0
-0
net/ipv6/ip6_output.c void ipv6_select_ident(struct frag_hdr *fhdr, struct rt6_info *rt) · line 609
Quoted 8 lines
net/ipv6/ip6_output.c +1
-1@@ -603,7 +603,7 @@ void ipv6_select_ident(struct frag_hdr *fhdr, struct rt6_info *rt)static atomic_t ipv6_fragmentation_id;int old, new; -if (rt) { +if (rt&& !(rt->dst.flags&DST_NOPEER )) {struct inet_peer *peer;if (!rt->rt6i_peer)
Eric,
I'm seeing a new problem now with IPv6 on my bridge, I don't know if it's related to any of the patches you gave me.
Basically, I have eth0 and eth1 in a balance-rr bond (bond0), and this is added as a port in a bridge. When the bond and bridge are freshly brought up, IPv4 works fine but it appears to be deaf to IPv6 traffic. It stays this way until I run 'tcpdump -i eth0' and 'tcpdump -i eth1'. If I run tcpdump with the -p flag to not enable promiscuous mode, the interfaces remain deaf. Only if they enter promiscuous mode do they start to listen to each other. It also doesn't help to run tcpdump against bond0 or br0 at all.
I'm not sure if once I've killed tcpdump they become deaf again, but I have enough IPv6 traffic to keep the neighbour entry alive.
Cheers,
Chris
Signature · 2 lines
Chris Boot [email protected]
Quoted 14 lines · Chris Boot
Eric,
I'm seeing a new problem now with IPv6 on my bridge, I don't know if it's related to any of the patches you gave me.
Basically, I have eth0 and eth1 in a balance-rr bond (bond0), and this is added as a port in a bridge. When the bond and bridge are freshly brought up, IPv4 works fine but it appears to be deaf to IPv6 traffic. It stays this way until I run 'tcpdump -i eth0' and 'tcpdump -i eth1'. If I run tcpdump with the -p flag to not enable promiscuous mode, the interfaces remain deaf. Only if they enter promiscuous mode do they start to listen to each other. It also doesn't help to run tcpdump against bond0 or br0 at all.
Quoted 2 lines
I'm not sure if once I've killed tcpdump they become deaf again, but I have enough IPv6 traffic to keep the neighbour entry alive.
Thats probably a separate issue. It would be better to open a new thread on netdev, instead of continuing this one.
IPv6 needs multicast capabilities, so maybe balance-rr bonding mode is not propagating multicat filters properly on both nics.
It might be a problem on one of your NIC.
(tcpdump automatically installs promiscous mode, so all multicast frames are received, regardless of previous multicast filters in place)
David Miller
maintainerFrom: Eric Dumazet <[email protected]>
Date: Thu, 22 Dec 2011 18:41:32 +0100
Quoted 4 lines
IPv6 needs multicast capabilities, so maybe balance-rr bonding mode is not propagating multicat filters properly on both nics.
It might be a problem on one of your NIC.
This is my impression too, definitely some kind of multicast problem.
From: Eric Dumazet <[email protected]>
Date: Thu, 22 Dec 2011 15:15:53 +0100
Quoted 3 lines
[PATCH] net: introduce
DST_NOPEERdst flagChris Boot reported crashes occurring in
ipv6_select_ident().
...
Quoted 6 lines
This is caused by bridge netfilter special
dst_entry(fake_rtable), a special shared entry, where attaching an inetpeer makes no sense.Problem is present since commit
87c48fa3b46(ipv6: make fragment identifications less predictable)
Quoted 6 lines
Introduce
DST_NOPEERdst flag and make sureipv6_select_ident()and__ip_select_ident()fallback to the 'no peer attached' handling.Reported-by Chris Boot <[email protected]>Tested-by Chris Boot <[email protected]>Signed-off-by Eric Dumazet <[email protected]>
Applied.