Skip to contentSkip to conversation

igb: add more checks for disconnected adapter

PATCH In review Jarod Wilson 6 participants 14 messages Sep 21, 2015 – Nov 20, 2015

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 igb: add more checks for disconnected adapter

Some pci changes upcoming in 4.3 seem to cause additional disconnects, which can happen at unfortuitous times for igb, leading to issues such as this, where the disconnect happened just before igb_configure_tx_ring():

[  414.440115] igb 0000:15:00.0: enabling device (0000 -> 0002)
[  414.474934] pps pps0: new PPS source ptp1
[  414.474937] igb 0000:15:00.0: added PHC on eth0
[  414.474938] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[  414.474940] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[  414.475072] igb 0000:15:00.0: eth0: PBA No: 000200-000
[  414.475073] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[  414.478453] igb 0000:15:00.0 enp21s0: renamed from eth0
[  414.497747] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[  414.536745] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[  414.854808] BUG: unable to handle kernel paging request at 0000000000003818
[  414.854827] IP: [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.854846] PGD 0
[  414.854849] Oops: 0002 [#1] SMP
[  414.854856] Modules linked in: firewire_ohci firewire_core crc_itu_t igb dca ctr ccm arc4 iwlmvm mac80211 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter bnep dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_hdmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel iTCO_wdt ppdev kvm iTCO_vendor_support hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
[  414.855073]  drbg ansi_cprng snd_hda_codec_realtek snd_hda_codec_generic aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_codec microcode snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm iwlwifi uvcvideo btusb cfg80211 videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel bluetooth v4l2_common snd_timer videodev snd parport_pc rtsx_pci_ms joydev pcspkr input_leds i2c_i801 media sg memstick rfkill soundcore lpc_ich 8250_fintek parport mei_me hp_accel ie31200_edac shpchp lis3lv02d mei edac_core input_polldev hp_wireless tpm_infineon sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom rtsx_pci_sdmmc mmc_core crc32c_intel serio_raw rtsx_pci nouveau mxm_wmi ahci hwmon libahci e1000e drm_kms_helper
[  414.855309]  ptp xhci_pci pps_core ttm xhci_hcd wmi video ipv6 autofs4
[  414.855331] CPU: 2 PID: 875 Comm: NetworkManager Not tainted 4.2.0-5.el7_UNSUPPORTED.x86_64 #1
[  414.855348] Hardware name: Hewlett-Packard HP ZBook 15 G2/2253, BIOS M70 Ver. 01.07 02/26/2015
[  414.855365] task: ffff880484698c00 ti: ffff88005859c000 task.ti: ffff88005859c000
[  414.855380] RIP: 0010:[<ffffffffa0b95a9c>]  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.855401] RSP: 0018:ffff88005859f608  EFLAGS: 00010246
[  414.855410] RAX: 0000000000003818 RBX: 0000000000000000 RCX: 0000000000003818
[  414.855424] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00000000002a9fe6
[  414.855437] RBP: ffff88005859f638 R08: 0000000003030300 R09: 00000000ffffffe7
[  414.855451] R10: ffffffff81fa91b4 R11: 00000000000007e3 R12: 0000000000000000
[  414.855464] R13: ffff880471c98840 R14: ffff8804670a1180 R15: 0000000483cce000
[  414.855478] FS:  00007f389c6fb8c0(0000) GS:ffff88049dc80000(0000) knlGS:0000000000000000
[  414.855493] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  414.855504] CR2: 0000000000003818 CR3: 00000004875da000 CR4: 00000000001406e0
[  414.855518] Stack:
[  414.855520]  ffff88005859f638 ffff880471c98840 ffff880471c98df8 0000000000000001
[  414.855538]  ffff880471c98848 0000000000000001 ffff88005859f698 ffffffffa0b99cb0
[  414.855555]  ffff88005859f678 59ab02179a7fe4d0 f3ce6b27ad46225f f5454218094e72d1
[  414.855572] Call Trace:
[  414.855577]  [<ffffffffa0b99cb0>] igb_configure+0x240/0x400 [igb]
[  414.855590]  [<ffffffffa0b99f32>] __igb_open+0xc2/0x560 [igb]
[  414.855602]  [<ffffffff8108f43d>] ? notifier_call_chain+0x4d/0x80
[  414.855614]  [<ffffffffa0b9a540>] igb_open+0x10/0x20 [igb]
[  414.855625]  [<ffffffff81581b81>] __dev_open+0xb1/0x130
[  414.855636]  [<ffffffff81581e91>] __dev_change_flags+0xa1/0x160
[  414.855647]  [<ffffffff81581f79>] dev_change_flags+0x29/0x60
[  414.855658]  [<ffffffff8158efc3>] do_setlink+0x5d3/0xaa0
[  414.855679]  [<ffffffff81308073>] ? nla_parse+0xa3/0x100
[  414.855689]  [<ffffffff815905f0>] rtnl_newlink+0x4f0/0x880
[  414.855700]  [<ffffffff815901f3>] ? rtnl_newlink+0xf3/0x880
[  414.855721]  [<ffffffff815ae23e>] ? netlink_unicast+0x1ae/0x220
[  414.855734]  [<ffffffff81266648>] ? security_capable+0x48/0x60
[  414.855746]  [<ffffffff810796bd>] ? ns_capable+0x2d/0x60
[  414.855756]  [<ffffffff8158db35>] rtnetlink_rcv_msg+0x95/0x240
[  414.855768]  [<ffffffff8126adc0>] ? sock_has_perm+0x70/0x90
[  414.855779]  [<ffffffff8158daa0>] ? rtnetlink_rcv+0x40/0x40
[  414.855789]  [<ffffffff815ae7ff>] netlink_rcv_skb+0xaf/0xc0
[  414.855800]  [<ffffffff8158da8c>] rtnetlink_rcv+0x2c/0x40
[  414.855810]  [<ffffffff815ae1de>] netlink_unicast+0x14e/0x220
[  414.855821]  [<ffffffff815ae5ca>] netlink_sendmsg+0x31a/0x390
[  414.855833]  [<ffffffff81563208>] sock_sendmsg+0x38/0x50
[  414.855843]  [<ffffffff81563b4e>] ___sys_sendmsg+0x27e/0x2a0
[  414.855855]  [<ffffffff8123d82f>] ? sysctl_head_finish+0x3f/0x50
[  414.855866]  [<ffffffff81077c10>] ? proc_put_long+0xb0/0xb0
[  414.855877]  [<ffffffff8123d9d9>] ? proc_sys_call_handler+0x79/0xc0
[  414.855890]  [<ffffffff812ec2fc>] ? lockref_put_or_lock+0x4c/0x80
[  414.855902]  [<ffffffff81564432>] __sys_sendmsg+0x42/0x80
[  414.855913]  [<ffffffff81564482>] SyS_sendmsg+0x12/0x20
[  414.855924]  [<ffffffff8165042e>] entry_SYSCALL_64_fastpath+0x12/0x71
[  414.855935] Code: c1 49 89 4e 30 49 8b 85 c0 05 00 00 48 85 c0 0f 84 0e 01 00 00 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b 46 30 31 d2 <89> 10 41 8b 95 44 06 00 00 b8 14 01 10 02 83
fa 05 74 0b 83 fa
[  414.856037] RIP  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.856052]  RSP <ffff88005859f608>
[  414.856057] CR2: 0000000000003818
[  414.872327] ---[ end trace e97522c0c584ea70 ]---

This can at least be reduced to a harmless initialization failure with some additional checking of device presence, similar to ixgbe. With this patch in place, instead we get:

[ 8010.562550] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 8010.597402] pps pps0: new PPS source ptp1
[ 8010.597406] igb 0000:15:00.0: added PHC on eth0
[ 8010.597407] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 8010.597409] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[ 8010.597543] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 8010.597545] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 8010.600468] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 8010.619354] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 8010.663999] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[ 8011.012427] igb 0000:15:00.0: Unable to allocate memory for vectors
CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org

---
Note: this is a follow-up patch in addition to the previously submitted
"igb: don't unmap NULL hw_addr"

 drivers/net/ethernet/intel/igb/igb_main.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +13 -2
index 6369f9e..7060edf 100644
@@ -952,6 +952,11 @@ static int igb_request_msix(struct igb_adapter *adapter)
 	if (err)
 		goto err_out;
 
+	if (E1000_REMOVED(hw->hw_addr)) {
+		err = -EIO;
+		goto err_free;
+	}
+
 	for (i = 0; i < adapter->num_q_vectors; i++) {
 		struct igb_q_vector *q_vector = adapter->q_vector[i];
 
@@ -1199,6 +1204,9 @@ static int igb_alloc_q_vector(struct igb_adapter *adapter,
 	if (txr_count > 1 || rxr_count > 1)
 		return -ENOMEM;
 
+	if (E1000_REMOVED(adapter->hw.hw_addr))
+		return -EIO;
+
 	ring_count = txr_count + rxr_count;
 	size = sizeof(struct igb_q_vector) +
 	       (sizeof(struct igb_ring) * ring_count);
@@ -3248,6 +3256,9 @@ void igb_configure_tx_ring(struct igb_adapter *adapter,
 	u64 tdba = ring->dma;
 	int reg_idx = ring->reg_idx;
 
+	if (E1000_REMOVED(adapter->io_addr))
+		return;
+
 	/* disable the queue */
 	wr32(E1000_TXDCTL(reg_idx), 0);
 	wrfl();
@@ -3259,7 +3270,7 @@ void igb_configure_tx_ring(struct igb_adapter *adapter,
 	     tdba & 0x00000000ffffffffULL);
 	wr32(E1000_TDBAH(reg_idx), tdba >> 32);
 
-	ring->tail = hw->hw_addr + E1000_TDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_TDT(reg_idx);
 	wr32(E1000_TDH(reg_idx), 0);
 	writel(0, ring->tail);
 
@@ -3615,7 +3626,7 @@ void igb_configure_rx_ring(struct igb_adapter *adapter,
 	     ring->count * sizeof(union e1000_adv_rx_desc));
 
 	/* initialize head and tail */
-	ring->tail = hw->hw_addr + E1000_RDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_RDT(reg_idx);
 	wr32(E1000_RDH(reg_idx), 0);
 	writel(0, ring->tail);
 
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 110 lines · Jarod Wilson

Some pci changes upcoming in 4.3 seem to cause additional disconnects, which can happen at unfortuitous times for igb, leading to issues such as this, where the disconnect happened just before igb_configure_tx_ring():

[  414.440115] igb 0000:15:00.0: enabling device (0000 -> 0002)
[  414.474934] pps pps0: new PPS source ptp1
[  414.474937] igb 0000:15:00.0: added PHC on eth0
[  414.474938] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[  414.474940] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[  414.475072] igb 0000:15:00.0: eth0: PBA No: 000200-000
[  414.475073] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[  414.478453] igb 0000:15:00.0 enp21s0: renamed from eth0
[  414.497747] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[  414.536745] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[  414.854808] BUG: unable to handle kernel paging request at 0000000000003818
[  414.854827] IP: [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.854846] PGD 0
[  414.854849] Oops: 0002 [#1] SMP
[  414.854856] Modules linked in: firewire_ohci firewire_core crc_itu_t igb dca ctr ccm arc4 iwlmvm mac80211 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter bnep dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_hdmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel iTCO_wdt ppdev kvm iTCO_vendor_support hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
[  414.855073]  drbg ansi_cprng snd_hda_codec_realtek snd_hda_codec_generic aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_codec microcode snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm iwlwifi uvcvideo btusb cfg80211 videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel bluetooth v4l2_common snd_timer videodev snd parport_pc rtsx_pci_ms joydev pcspkr input_leds i2c_i801 media sg memstick rfkill soundcore lpc_ich 8250_fintek parport mei_me hp_accel ie31200_edac shpchp lis3lv02d mei edac_core input_polldev hp_wireless tpm_infineon sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom rtsx_pci_sdmmc mmc_core crc32c_intel serio_raw rtsx_pci nouveau mxm_wmi ahci hwmon libahci e1000e drm_kms_helper
[  414.855309]  ptp xhci_pci pps_core ttm xhci_hcd wmi video ipv6 autofs4
[  414.855331] CPU: 2 PID: 875 Comm: NetworkManager Not tainted 4.2.0-5.el7_UNSUPPORTED.x86_64 #1
[  414.855348] Hardware name: Hewlett-Packard HP ZBook 15 G2/2253, BIOS M70 Ver. 01.07 02/26/2015
[  414.855365] task: ffff880484698c00 ti: ffff88005859c000 task.ti: ffff88005859c000
[  414.855380] RIP: 0010:[<ffffffffa0b95a9c>]  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.855401] RSP: 0018:ffff88005859f608  EFLAGS: 00010246
[  414.855410] RAX: 0000000000003818 RBX: 0000000000000000 RCX: 0000000000003818
[  414.855424] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00000000002a9fe6
[  414.855437] RBP: ffff88005859f638 R08: 0000000003030300 R09: 00000000ffffffe7
[  414.855451] R10: ffffffff81fa91b4 R11: 00000000000007e3 R12: 0000000000000000
[  414.855464] R13: ffff880471c98840 R14: ffff8804670a1180 R15: 0000000483cce000
[  414.855478] FS:  00007f389c6fb8c0(0000) GS:ffff88049dc80000(0000) knlGS:0000000000000000
[  414.855493] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  414.855504] CR2: 0000000000003818 CR3: 00000004875da000 CR4: 00000000001406e0
[  414.855518] Stack:
[  414.855520]  ffff88005859f638 ffff880471c98840 ffff880471c98df8 0000000000000001
[  414.855538]  ffff880471c98848 0000000000000001 ffff88005859f698 ffffffffa0b99cb0
[  414.855555]  ffff88005859f678 59ab02179a7fe4d0 f3ce6b27ad46225f f5454218094e72d1
[  414.855572] Call Trace:
[  414.855577]  [<ffffffffa0b99cb0>] igb_configure+0x240/0x400 [igb]
[  414.855590]  [<ffffffffa0b99f32>] __igb_open+0xc2/0x560 [igb]
[  414.855602]  [<ffffffff8108f43d>] ? notifier_call_chain+0x4d/0x80
[  414.855614]  [<ffffffffa0b9a540>] igb_open+0x10/0x20 [igb]
[  414.855625]  [<ffffffff81581b81>] __dev_open+0xb1/0x130
[  414.855636]  [<ffffffff81581e91>] __dev_change_flags+0xa1/0x160
[  414.855647]  [<ffffffff81581f79>] dev_change_flags+0x29/0x60
[  414.855658]  [<ffffffff8158efc3>] do_setlink+0x5d3/0xaa0
[  414.855679]  [<ffffffff81308073>] ? nla_parse+0xa3/0x100
[  414.855689]  [<ffffffff815905f0>] rtnl_newlink+0x4f0/0x880
[  414.855700]  [<ffffffff815901f3>] ? rtnl_newlink+0xf3/0x880
[  414.855721]  [<ffffffff815ae23e>] ? netlink_unicast+0x1ae/0x220
[  414.855734]  [<ffffffff81266648>] ? security_capable+0x48/0x60
[  414.855746]  [<ffffffff810796bd>] ? ns_capable+0x2d/0x60
[  414.855756]  [<ffffffff8158db35>] rtnetlink_rcv_msg+0x95/0x240
[  414.855768]  [<ffffffff8126adc0>] ? sock_has_perm+0x70/0x90
[  414.855779]  [<ffffffff8158daa0>] ? rtnetlink_rcv+0x40/0x40
[  414.855789]  [<ffffffff815ae7ff>] netlink_rcv_skb+0xaf/0xc0
[  414.855800]  [<ffffffff8158da8c>] rtnetlink_rcv+0x2c/0x40
[  414.855810]  [<ffffffff815ae1de>] netlink_unicast+0x14e/0x220
[  414.855821]  [<ffffffff815ae5ca>] netlink_sendmsg+0x31a/0x390
[  414.855833]  [<ffffffff81563208>] sock_sendmsg+0x38/0x50
[  414.855843]  [<ffffffff81563b4e>] ___sys_sendmsg+0x27e/0x2a0
[  414.855855]  [<ffffffff8123d82f>] ? sysctl_head_finish+0x3f/0x50
[  414.855866]  [<ffffffff81077c10>] ? proc_put_long+0xb0/0xb0
[  414.855877]  [<ffffffff8123d9d9>] ? proc_sys_call_handler+0x79/0xc0
[  414.855890]  [<ffffffff812ec2fc>] ? lockref_put_or_lock+0x4c/0x80
[  414.855902]  [<ffffffff81564432>] __sys_sendmsg+0x42/0x80
[  414.855913]  [<ffffffff81564482>] SyS_sendmsg+0x12/0x20
[  414.855924]  [<ffffffff8165042e>] entry_SYSCALL_64_fastpath+0x12/0x71
[  414.855935] Code: c1 49 89 4e 30 49 8b 85 c0 05 00 00 48 85 c0 0f 84 0e 01 00 00 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b 46 30 31 d2 <89> 10 41 8b 95 44 06 00 00 b8 14 01 10 02 83
fa 05 74 0b 83 fa
[  414.856037] RIP  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.856052]  RSP <ffff88005859f608>
[  414.856057] CR2: 0000000000003818
[  414.872327] ---[ end trace e97522c0c584ea70 ]---

This can at least be reduced to a harmless initialization failure with some additional checking of device presence, similar to ixgbe. With this patch in place, instead we get:

[ 8010.562550] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 8010.597402] pps pps0: new PPS source ptp1
[ 8010.597406] igb 0000:15:00.0: added PHC on eth0
[ 8010.597407] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 8010.597409] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[ 8010.597543] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 8010.597545] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 8010.600468] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 8010.619354] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 8010.663999] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[ 8011.012427] igb 0000:15:00.0: Unable to allocate memory for vectors
CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org

---
Note: this is a follow-up patch in addition to the previously submitted
"igb: don't unmap NULL hw_addr"

  drivers/net/ethernet/intel/igb/igb_main.c | 15 +++++++++++++--
  1 file changed, 13 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +0 -0
index 6369f9e..7060edf 100644
@@ -952,6 +952,11 @@ static int igb_request_msix(struct igb_adapter *adapter)
  	if (err)

drivers/net/ethernet/intel/igb/igb_main.c static int igb_request_msix(struct igb_adapter *adapter) · line 962

Quoted 10 lines
drivers/net/ethernet/intel/igb/igb_main.c +5 -0
  		goto err_out;
 
+	if (E1000_REMOVED(hw->hw_addr)) {
+		err = -EIO;
+		goto err_free;
+	}
+
  	for (i = 0; i < adapter->num_q_vectors; i++) {
  		struct igb_q_vector *q_vector = adapter->q_vector[i];
 

Instead of using E1000_REMOVED we should just replace the adapter->hw.hw_addr in the setup of the itr_register with adapter->io_addr like you did for Rx/Tx below.

drivers/net/ethernet/intel/igb/igb_main.c static int igb_alloc_q_vector(struct igb_adapter *adapter, · line 1212

Quoted 10 lines
drivers/net/ethernet/intel/igb/igb_main.c +3 -0
@@ -1199,6 +1204,9 @@ static int igb_alloc_q_vector(struct igb_adapter *adapter,
  	if (txr_count > 1 || rxr_count > 1)
  		return -ENOMEM;
 
+	if (E1000_REMOVED(adapter->hw.hw_addr))
+		return -EIO;
+
  	ring_count = txr_count + rxr_count;
  	size = sizeof(struct igb_q_vector) +
  	       (sizeof(struct igb_ring) * ring_count);

Same here.

drivers/net/ethernet/intel/igb/igb_main.c void igb_configure_tx_ring(struct igb_adapter *adapter, · line 3264

Quoted 10 lines
drivers/net/ethernet/intel/igb/igb_main.c +3 -0
@@ -3248,6 +3256,9 @@ void igb_configure_tx_ring(struct igb_adapter *adapter,
  	u64 tdba = ring->dma;
  	int reg_idx = ring->reg_idx;
 
+	if (E1000_REMOVED(adapter->io_addr))
+		return;
+
  	/* disable the queue */
  	wr32(E1000_TXDCTL(reg_idx), 0);
  	wrfl();

This bit provides no value. The value in adapter->io_addr should never be NULL unless the driver failed to load back in probe.

Quoted 11 lines
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
@@ -3259,7 +3270,7 @@ void igb_configure_tx_ring(struct igb_adapter *adapter,
  	     tdba & 0x00000000ffffffffULL);
  	wr32(E1000_TDBAH(reg_idx), tdba >> 32);
 
-	ring->tail = hw->hw_addr + E1000_TDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_TDT(reg_idx);
  	wr32(E1000_TDH(reg_idx), 0);
  	writel(0, ring->tail);
 

drivers/net/ethernet/intel/igb/igb_main.c void igb_configure_rx_ring(struct igb_adapter *adapter, · line 3632

Quoted 8 lines
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
@@ -3615,7 +3626,7 @@ void igb_configure_rx_ring(struct igb_adapter *adapter,
  	     ring->count * sizeof(union e1000_adv_rx_desc));
 
  	/* initialize head and tail */
-	ring->tail = hw->hw_addr + E1000_RDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_RDT(reg_idx);
  	wr32(E1000_RDH(reg_idx), 0);
  	writel(0, ring->tail);
 

These two fixes are correct. We just need to apply it to any other spots where we are storing register offsets for writing.

#2 lore raw
Quoted 167 lines · Alexander Duyck
Quoted 166 lines · Jarod Wilson

Some pci changes upcoming in 4.3 seem to cause additional disconnects, which can happen at unfortuitous times for igb, leading to issues such as this, where the disconnect happened just before igb_configure_tx_ring():

[ 414.440115] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 414.474934] pps pps0: new PPS source ptp1
[ 414.474937] igb 0000:15:00.0: added PHC on eth0
[ 414.474938] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network
Connection
[ 414.474940] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1)
e8:ea:6a:00:1b:2a
[ 414.475072] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 414.475073] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s),
4 tx queue(s)
[ 414.478453] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 414.497747] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 414.536745] igb 0000:15:00.0 enp21s0: PCIe link lost, device now
detached
[ 414.854808] BUG: unable to handle kernel paging request at
0000000000003818
[ 414.854827] IP: [<ffffffffa0b95a9c>]
igb_configure_tx_ring+0x14c/0x250 [igb]
[ 414.854846] PGD 0
[ 414.854849] Oops: 0002 [#1] SMP
[ 414.854856] Modules linked in: firewire_ohci firewire_core crc_itu_t

igb dca ctr ccm arc4 iwlmvm mac80211 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter bnep dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_hdmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel iTCO_wdt ppdev kvm iTCO_vendor_support hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel

[ 414.855073] drbg ansi_cprng snd_hda_codec_realtek

snd_hda_codec_generic aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_codec microcode snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm iwlwifi uvcvideo btusb cfg80211 videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel bluetooth v4l2_common snd_timer videodev snd parport_pc rtsx_pci_ms joydev pcspkr input_leds i2c_i801 media sg memstick rfkill soundcore lpc_ich 8250_fintek parport mei_me hp_accel ie31200_edac shpchp lis3lv02d mei edac_core input_polldev hp_wireless tpm_infineon sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom rtsx_pci_sdmmc mmc_core crc32c_intel serio_raw rtsx_pci nouveau mxm_wmi ahci hwmon libahci e1000e drm_kms_helper

[ 414.855309] ptp xhci_pci pps_core ttm xhci_hcd wmi video ipv6 autofs4
[ 414.855331] CPU: 2 PID: 875 Comm: NetworkManager Not tainted
4.2.0-5.el7_UNSUPPORTED.x86_64 #1
[ 414.855348] Hardware name: Hewlett-Packard HP ZBook 15 G2/2253, BIOS
M70 Ver. 01.07 02/26/2015
[ 414.855365] task: ffff880484698c00 ti: ffff88005859c000 task.ti:
ffff88005859c000
[ 414.855380] RIP: 0010:[<ffffffffa0b95a9c>] [<ffffffffa0b95a9c>]
igb_configure_tx_ring+0x14c/0x250 [igb]
[ 414.855401] RSP: 0018:ffff88005859f608 EFLAGS: 00010246
[ 414.855410] RAX: 0000000000003818 RBX: 0000000000000000 RCX:
0000000000003818
[ 414.855424] RDX: 0000000000000000 RSI: 0000000000000008 RDI:
00000000002a9fe6
[ 414.855437] RBP: ffff88005859f638 R08: 0000000003030300 R09:
00000000ffffffe7
[ 414.855451] R10: ffffffff81fa91b4 R11: 00000000000007e3 R12:
0000000000000000
[ 414.855464] R13: ffff880471c98840 R14: ffff8804670a1180 R15:
0000000483cce000
[ 414.855478] FS: 00007f389c6fb8c0(0000) GS:ffff88049dc80000(0000)
knlGS:0000000000000000
[ 414.855493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 414.855504] CR2: 0000000000003818 CR3: 00000004875da000 CR4:
00000000001406e0
[ 414.855518] Stack:
[ 414.855520] ffff88005859f638 ffff880471c98840 ffff880471c98df8
0000000000000001
[ 414.855538] ffff880471c98848 0000000000000001 ffff88005859f698
ffffffffa0b99cb0
[ 414.855555] ffff88005859f678 59ab02179a7fe4d0 f3ce6b27ad46225f
f5454218094e72d1
[ 414.855572] Call Trace:
[ 414.855577] [<ffffffffa0b99cb0>] igb_configure+0x240/0x400 [igb]
[ 414.855590] [<ffffffffa0b99f32>] __igb_open+0xc2/0x560 [igb]
[ 414.855602] [<ffffffff8108f43d>] ? notifier_call_chain+0x4d/0x80
[ 414.855614] [<ffffffffa0b9a540>] igb_open+0x10/0x20 [igb]
[ 414.855625] [<ffffffff81581b81>] __dev_open+0xb1/0x130
[ 414.855636] [<ffffffff81581e91>] __dev_change_flags+0xa1/0x160
[ 414.855647] [<ffffffff81581f79>] dev_change_flags+0x29/0x60
[ 414.855658] [<ffffffff8158efc3>] do_setlink+0x5d3/0xaa0
[ 414.855679] [<ffffffff81308073>] ? nla_parse+0xa3/0x100
[ 414.855689] [<ffffffff815905f0>] rtnl_newlink+0x4f0/0x880
[ 414.855700] [<ffffffff815901f3>] ? rtnl_newlink+0xf3/0x880
[ 414.855721] [<ffffffff815ae23e>] ? netlink_unicast+0x1ae/0x220
[ 414.855734] [<ffffffff81266648>] ? security_capable+0x48/0x60
[ 414.855746] [<ffffffff810796bd>] ? ns_capable+0x2d/0x60
[ 414.855756] [<ffffffff8158db35>] rtnetlink_rcv_msg+0x95/0x240
[ 414.855768] [<ffffffff8126adc0>] ? sock_has_perm+0x70/0x90
[ 414.855779] [<ffffffff8158daa0>] ? rtnetlink_rcv+0x40/0x40
[ 414.855789] [<ffffffff815ae7ff>] netlink_rcv_skb+0xaf/0xc0
[ 414.855800] [<ffffffff8158da8c>] rtnetlink_rcv+0x2c/0x40
[ 414.855810] [<ffffffff815ae1de>] netlink_unicast+0x14e/0x220
[ 414.855821] [<ffffffff815ae5ca>] netlink_sendmsg+0x31a/0x390
[ 414.855833] [<ffffffff81563208>] sock_sendmsg+0x38/0x50
[ 414.855843] [<ffffffff81563b4e>] ___sys_sendmsg+0x27e/0x2a0
[ 414.855855] [<ffffffff8123d82f>] ? sysctl_head_finish+0x3f/0x50
[ 414.855866] [<ffffffff81077c10>] ? proc_put_long+0xb0/0xb0
[ 414.855877] [<ffffffff8123d9d9>] ? proc_sys_call_handler+0x79/0xc0
[ 414.855890] [<ffffffff812ec2fc>] ? lockref_put_or_lock+0x4c/0x80
[ 414.855902] [<ffffffff81564432>] __sys_sendmsg+0x42/0x80
[ 414.855913] [<ffffffff81564482>] SyS_sendmsg+0x12/0x20
[ 414.855924] [<ffffffff8165042e>] entry_SYSCALL_64_fastpath+0x12/0x71
[ 414.855935] Code: c1 49 89 4e 30 49 8b 85 c0 05 00 00 48 85 c0 0f 84
0e 01 00 00 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b 46
30 31 d2 <89> 10 41 8b 95 44 06 00 00 b8 14 01 10 02 83
fa 05 74 0b 83 fa
[ 414.856037] RIP [<ffffffffa0b95a9c>]
igb_configure_tx_ring+0x14c/0x250 [igb]
[ 414.856052] RSP <ffff88005859f608>
[ 414.856057] CR2: 0000000000003818
[ 414.872327] ---[ end trace e97522c0c584ea70 ]---

This can at least be reduced to a harmless initialization failure with some
additional checking of device presence, similar to ixgbe. With this patch in place, instead we get:

[ 8010.562550] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 8010.597402] pps pps0: new PPS source ptp1
[ 8010.597406] igb 0000:15:00.0: added PHC on eth0
[ 8010.597407] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network
Connection
[ 8010.597409] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1)
e8:ea:6a:00:1b:2a
[ 8010.597543] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 8010.597545] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx
queue(s), 4 tx queue(s)
[ 8010.600468] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 8010.619354] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 8010.663999] igb 0000:15:00.0 enp21s0: PCIe link lost, device now
detached
[ 8011.012427] igb 0000:15:00.0: Unable to allocate memory for vectors
CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org

---
Note: this is a follow-up patch in addition to the previously submitted "igb: don't unmap NULL hw_addr"

drivers/net/ethernet/intel/igb/igb_main.c | 15 +++++++++++++--

1 file changed, 13 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +0 -0
index 6369f9e..7060edf 100644
@@ -952,6 +952,11 @@ static int igb_request_msix(struct igb_adapter
 *adapter)
 if (err)
 goto err_out;
 
Quoted 13 lines

drivers/net/ethernet/intel/igb/igb_main.c static int igb_request_msix(struct igb_adapter *adapter) · line 961

Quoted 8 lines
drivers/net/ethernet/intel/igb/igb_main.c +5 -0
+ if (E1000_REMOVED(hw->hw_addr)) {
+ err = -EIO;
+ goto err_free;
+ }
+
 for (i = 0; i < adapter->num_q_vectors; i++) {
 struct igb_q_vector *q_vector = adapter->q_vector[i];

Instead of using E1000_REMOVED we should just replace the adapter->hw.hw_addr in the setup of the itr_register with adapter->io_addr like you did for Rx/Tx below.

I just tried that, and it reliably blows up horrifically, wedging the machine to the point where all I could get was a screen shot with my phone thus far, when we jump from igb_request_msix() to igb_configure_msix() to igb_assign_vector() and finally to igb_write_ivar(), at least as best as I can tell from what I was able to see in the trace remnants still on-screen.

Quoted 19 lines
Quoted 11 lines
drivers/net/ethernet/intel/igb/igb_main.c +3 -0
@@ -1199,6 +1204,9 @@ static int igb_alloc_q_vector(struct igb_adapter
 *adapter,
 if (txr_count > 1 || rxr_count > 1)
 return -ENOMEM;
 
+ if (E1000_REMOVED(adapter->hw.hw_addr))
+ return -EIO;
+
 ring_count = txr_count + rxr_count;

size = sizeof(struct igb_q_vector) +
(sizeof(struct igb_ring) * ring_count);

Same here.

Quoted 5 lines
drivers/net/ethernet/intel/igb/igb_main.c +0 -0
@@ -3248,6 +3256,9 @@ void igb_configure_tx_ring(struct igb_adapter
 *adapter,
 u64 tdba = ring->dma;
 int reg_idx = ring->reg_idx;
 
Quoted 9 lines
Quoted 6 lines
drivers/net/ethernet/intel/igb/igb_main.c +3 -0
+ if (E1000_REMOVED(adapter->io_addr))
+ return;
+
 /* disable the queue */
 wr32(E1000_TXDCTL(reg_idx), 0);

wrfl();

This bit provides no value. The value in adapter->io_addr should never be NULL unless the driver failed to load back in probe.

Ah, yeah, you're right. I was looking at the ixgbe "is hardware gone" check, which is a bit more sophisticated, and failed to notice/recall that E1000_REMOVED() is simply an "is it NULL?" check. Using adapter->io_addr here seems to avoid the crash I was seeing in igb_configure_tx_ring(), though if the hardware really got disconnected, maybe checking for NULL hw->hw_addr really is the right thing here?

Quoted 14 lines
Quoted 14 lines
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
@@ -3259,7 +3270,7 @@ void igb_configure_tx_ring(struct igb_adapter
 *adapter,
 tdba & 0x00000000ffffffffULL);
 wr32(E1000_TDBAH(reg_idx), tdba >> 32);
 
- ring->tail = hw->hw_addr + E1000_TDT(reg_idx);
+ ring->tail = adapter->io_addr + E1000_TDT(reg_idx);
 wr32(E1000_TDH(reg_idx), 0);
 writel(0, ring->tail);
@@ -3615,7 +3626,7 @@ void igb_configure_rx_ring(struct igb_adapter
 *adapter,
 ring->count * sizeof(union e1000_adv_rx_desc));
 
Quoted 10 lines

drivers/net/ethernet/intel/igb/igb_main.c void igb_configure_rx_ring(struct igb_adapter *adapter, · line 3631

Quoted 7 lines
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
 /* initialize head and tail */
- ring->tail = hw->hw_addr + E1000_RDT(reg_idx);
+ ring->tail = adapter->io_addr + E1000_RDT(reg_idx);
 wr32(E1000_RDH(reg_idx), 0);
 writel(0, ring->tail);

These two fixes are correct. We just need to apply it to any other spots where we are storing register offsets for writing.

Just switching to adapter->io_addr everywhere seems to not work as noted above. :\ Note that I'm also chasing this from the other end with the author of the pci patches that seem to have triggered this, so the real bug might be over in pci-land, but hardening against explosions in igb still seems like a worthwhile effort here.

Signature · 2 lines
Jarod Wilson
jarod at redhat.com
#3 lore raw
Quoted 5 lines · Jarod Wilson

Just switching to adapter->io_addr everywhere seems to not work as noted above. :\ Note that I'm also chasing this from the other end with the author of the pci patches that seem to have triggered this, so the real bug might be over in pci-land, but hardening against explosions in igb still seems like a worthwhile effort here.

My understanding is that there can be problems if too many writes to a removed device happen. That is why ixgbe avoids doing that by testing for removal in some places. The io_addr does get used in the transmit path simply to avoid adding a test to that hot path. That approach seems to be working well for ixgbe.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 867 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20150921/cb90f123/attachment.asc>

#4 lore raw
In reply to Jarod Wilson
Quoted 184 lines · Jarod Wilson
Quoted 169 lines · Alexander Duyck
Quoted 168 lines · Jarod Wilson

Some pci changes upcoming in 4.3 seem to cause additional disconnects, which can happen at unfortuitous times for igb, leading to issues such as
this, where the disconnect happened just before igb_configure_tx_ring():

[ 414.440115] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 414.474934] pps pps0: new PPS source ptp1
[ 414.474937] igb 0000:15:00.0: added PHC on eth0
[ 414.474938] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network
Connection
[ 414.474940] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1)
e8:ea:6a:00:1b:2a
[ 414.475072] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 414.475073] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s),
4 tx queue(s)
[ 414.478453] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 414.497747] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 414.536745] igb 0000:15:00.0 enp21s0: PCIe link lost, device now
detached
[ 414.854808] BUG: unable to handle kernel paging request at
0000000000003818
[ 414.854827] IP: [<ffffffffa0b95a9c>]
igb_configure_tx_ring+0x14c/0x250 [igb]
[ 414.854846] PGD 0
[ 414.854849] Oops: 0002 [#1] SMP
[ 414.854856] Modules linked in: firewire_ohci firewire_core crc_itu_t

igb dca ctr ccm arc4 iwlmvm mac80211 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter bnep dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_hdmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel iTCO_wdt ppdev kvm iTCO_vendor_support hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel

[ 414.855073] drbg ansi_cprng snd_hda_codec_realtek

snd_hda_codec_generic aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_codec microcode snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm iwlwifi uvcvideo btusb cfg80211 videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel bluetooth v4l2_common snd_timer videodev snd parport_pc rtsx_pci_ms joydev pcspkr input_leds i2c_i801 media sg memstick rfkill soundcore lpc_ich 8250_fintek parport mei_me hp_accel ie31200_edac shpchp lis3lv02d mei edac_core input_polldev hp_wireless tpm_infineon sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom rtsx_pci_sdmmc mmc_core crc32c_intel serio_raw rtsx_pci nouveau mxm_wmi ahci hwmon libahci e1000e drm_kms_helper

[ 414.855309] ptp xhci_pci pps_core ttm xhci_hcd wmi video ipv6 autofs4
[ 414.855331] CPU: 2 PID: 875 Comm: NetworkManager Not tainted
4.2.0-5.el7_UNSUPPORTED.x86_64 #1
[ 414.855348] Hardware name: Hewlett-Packard HP ZBook 15 G2/2253, BIOS
M70 Ver. 01.07 02/26/2015
[ 414.855365] task: ffff880484698c00 ti: ffff88005859c000 task.ti:
ffff88005859c000
[ 414.855380] RIP: 0010:[<ffffffffa0b95a9c>] [<ffffffffa0b95a9c>]
igb_configure_tx_ring+0x14c/0x250 [igb]
[ 414.855401] RSP: 0018:ffff88005859f608 EFLAGS: 00010246
[ 414.855410] RAX: 0000000000003818 RBX: 0000000000000000 RCX:
0000000000003818
[ 414.855424] RDX: 0000000000000000 RSI: 0000000000000008 RDI:
00000000002a9fe6
[ 414.855437] RBP: ffff88005859f638 R08: 0000000003030300 R09:
00000000ffffffe7
[ 414.855451] R10: ffffffff81fa91b4 R11: 00000000000007e3 R12:
0000000000000000
[ 414.855464] R13: ffff880471c98840 R14: ffff8804670a1180 R15:
0000000483cce000
[ 414.855478] FS: 00007f389c6fb8c0(0000) GS:ffff88049dc80000(0000)
knlGS:0000000000000000
[ 414.855493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 414.855504] CR2: 0000000000003818 CR3: 00000004875da000 CR4:
00000000001406e0
[ 414.855518] Stack:
[ 414.855520] ffff88005859f638 ffff880471c98840 ffff880471c98df8
0000000000000001
[ 414.855538] ffff880471c98848 0000000000000001 ffff88005859f698
ffffffffa0b99cb0
[ 414.855555] ffff88005859f678 59ab02179a7fe4d0 f3ce6b27ad46225f
f5454218094e72d1
[ 414.855572] Call Trace:
[ 414.855577] [<ffffffffa0b99cb0>] igb_configure+0x240/0x400 [igb]
[ 414.855590] [<ffffffffa0b99f32>] __igb_open+0xc2/0x560 [igb]
[ 414.855602] [<ffffffff8108f43d>] ? notifier_call_chain+0x4d/0x80
[ 414.855614] [<ffffffffa0b9a540>] igb_open+0x10/0x20 [igb]
[ 414.855625] [<ffffffff81581b81>] __dev_open+0xb1/0x130
[ 414.855636] [<ffffffff81581e91>] __dev_change_flags+0xa1/0x160
[ 414.855647] [<ffffffff81581f79>] dev_change_flags+0x29/0x60
[ 414.855658] [<ffffffff8158efc3>] do_setlink+0x5d3/0xaa0
[ 414.855679] [<ffffffff81308073>] ? nla_parse+0xa3/0x100
[ 414.855689] [<ffffffff815905f0>] rtnl_newlink+0x4f0/0x880
[ 414.855700] [<ffffffff815901f3>] ? rtnl_newlink+0xf3/0x880
[ 414.855721] [<ffffffff815ae23e>] ? netlink_unicast+0x1ae/0x220
[ 414.855734] [<ffffffff81266648>] ? security_capable+0x48/0x60
[ 414.855746] [<ffffffff810796bd>] ? ns_capable+0x2d/0x60
[ 414.855756] [<ffffffff8158db35>] rtnetlink_rcv_msg+0x95/0x240
[ 414.855768] [<ffffffff8126adc0>] ? sock_has_perm+0x70/0x90
[ 414.855779] [<ffffffff8158daa0>] ? rtnetlink_rcv+0x40/0x40
[ 414.855789] [<ffffffff815ae7ff>] netlink_rcv_skb+0xaf/0xc0
[ 414.855800] [<ffffffff8158da8c>] rtnetlink_rcv+0x2c/0x40
[ 414.855810] [<ffffffff815ae1de>] netlink_unicast+0x14e/0x220
[ 414.855821] [<ffffffff815ae5ca>] netlink_sendmsg+0x31a/0x390
[ 414.855833] [<ffffffff81563208>] sock_sendmsg+0x38/0x50
[ 414.855843] [<ffffffff81563b4e>] ___sys_sendmsg+0x27e/0x2a0
[ 414.855855] [<ffffffff8123d82f>] ? sysctl_head_finish+0x3f/0x50
[ 414.855866] [<ffffffff81077c10>] ? proc_put_long+0xb0/0xb0
[ 414.855877] [<ffffffff8123d9d9>] ? proc_sys_call_handler+0x79/0xc0
[ 414.855890] [<ffffffff812ec2fc>] ? lockref_put_or_lock+0x4c/0x80
[ 414.855902] [<ffffffff81564432>] __sys_sendmsg+0x42/0x80
[ 414.855913] [<ffffffff81564482>] SyS_sendmsg+0x12/0x20
[ 414.855924] [<ffffffff8165042e>] entry_SYSCALL_64_fastpath+0x12/0x71
[ 414.855935] Code: c1 49 89 4e 30 49 8b 85 c0 05 00 00 48 85 c0 0f 84
0e 01 00 00 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b 46
30 31 d2 <89> 10 41 8b 95 44 06 00 00 b8 14 01 10 02 83
fa 05 74 0b 83 fa
[ 414.856037] RIP [<ffffffffa0b95a9c>]
igb_configure_tx_ring+0x14c/0x250 [igb]
[ 414.856052] RSP <ffff88005859f608>
[ 414.856057] CR2: 0000000000003818
[ 414.872327] ---[ end trace e97522c0c584ea70 ]---

This can at least be reduced to a harmless initialization failure with some
additional checking of device presence, similar to ixgbe. With this patch
in place, instead we get:

[ 8010.562550] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 8010.597402] pps pps0: new PPS source ptp1
[ 8010.597406] igb 0000:15:00.0: added PHC on eth0
[ 8010.597407] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network
Connection
[ 8010.597409] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1)
e8:ea:6a:00:1b:2a
[ 8010.597543] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 8010.597545] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx
queue(s), 4 tx queue(s)
[ 8010.600468] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 8010.619354] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 8010.663999] igb 0000:15:00.0 enp21s0: PCIe link lost, device now
detached
[ 8011.012427] igb 0000:15:00.0: Unable to allocate memory for vectors
CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org

---
Note: this is a follow-up patch in addition to the previously submitted "igb: don't unmap NULL hw_addr"

drivers/net/ethernet/intel/igb/igb_main.c | 15 +++++++++++++--

1 file changed, 13 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +0 -0
index 6369f9e..7060edf 100644
@@ -952,6 +952,11 @@ static int igb_request_msix(struct igb_adapter
 *adapter)
 if (err)
 goto err_out;
 
Quoted 13 lines

drivers/net/ethernet/intel/igb/igb_main.c static int igb_request_msix(struct igb_adapter *adapter) · line 961

Quoted 8 lines
drivers/net/ethernet/intel/igb/igb_main.c +5 -0
+ if (E1000_REMOVED(hw->hw_addr)) {
+ err = -EIO;
+ goto err_free;
+ }
+
 for (i = 0; i < adapter->num_q_vectors; i++) {
 struct igb_q_vector *q_vector = adapter->q_vector[i];

Instead of using E1000_REMOVED we should just replace the adapter->hw.hw_addr in the setup of the itr_register with adapter->io_addr like you did for Rx/Tx below.

Quoted 6 lines

I just tried that, and it reliably blows up horrifically, wedging the machine to the point where all I could get was a screen shot with my phone thus far, when we jump from igb_request_msix() to igb_configure_msix() to igb_assign_vector() and finally to igb_write_ivar(), at least as best as I can tell from what I was able to see in the trace remnants still on-screen.

Take a look at array_rd32, that is buggy and doesn't match up with the rd32 implementation. If you fix that then the blow-up should go away.

You shouldn't need to worry about itr_register since it will only get triggered if the interrupt can be enabled and that shouldn't be able to happen if the device is not present.

Quoted 25 lines
Quoted 14 lines
Quoted 14 lines
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
@@ -3259,7 +3270,7 @@ void igb_configure_tx_ring(struct igb_adapter
 *adapter,
 tdba & 0x00000000ffffffffULL);
 wr32(E1000_TDBAH(reg_idx), tdba >> 32);
 
- ring->tail = hw->hw_addr + E1000_TDT(reg_idx);
+ ring->tail = adapter->io_addr + E1000_TDT(reg_idx);
 wr32(E1000_TDH(reg_idx), 0);
 writel(0, ring->tail);
@@ -3615,7 +3626,7 @@ void igb_configure_rx_ring(struct igb_adapter
 *adapter,
 ring->count * sizeof(union e1000_adv_rx_desc));
 
Quoted 10 lines

drivers/net/ethernet/intel/igb/igb_main.c void igb_configure_rx_ring(struct igb_adapter *adapter, · line 3631

Quoted 7 lines
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
 /* initialize head and tail */
- ring->tail = hw->hw_addr + E1000_RDT(reg_idx);
+ ring->tail = adapter->io_addr + E1000_RDT(reg_idx);
 wr32(E1000_RDH(reg_idx), 0);
 writel(0, ring->tail);

These two fixes are correct. We just need to apply it to any other spots where we are storing register offsets for writing.

Quoted 5 lines

Just switching to adapter->io_addr everywhere seems to not work as noted above. :\ Note that I'm also chasing this from the other end with the author of the pci patches that seem to have triggered this, so the real bug might be over in pci-land, but hardening against explosions in igb still seems like a worthwhile effort here.

I am pretty sure array_rd32 is the problem. If you fix that then I suspect you should quit seeing any further issues.

- Alex

#5 lore raw
Quoted 9 lines · Alexander Duyck
Quoted 8 lines · Jarod Wilson
Quoted 7 lines · Alexander Duyck
Quoted 6 lines · Jarod Wilson

Some pci changes upcoming in 4.3 seem to cause additional disconnects, which can happen at unfortuitous times for igb, leading to issues such as
this, where the disconnect happened just before igb_configure_tx_ring():

...

Quoted 43 lines
Quoted 30 lines
Quoted 16 lines
Quoted 16 lines

Note: this is a follow-up patch in addition to the previously submitted "igb: don't unmap NULL hw_addr"

drivers/net/ethernet/intel/igb/igb_main.c | 15 +++++++++++++--

1 file changed, 13 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/igb/igb_main.c +0 -0
index 6369f9e..7060edf 100644
@@ -952,6 +952,11 @@ static int igb_request_msix(struct igb_adapter
 *adapter)
 if (err)
 goto err_out;
 
Quoted 13 lines

drivers/net/ethernet/intel/igb/igb_main.c static int igb_request_msix(struct igb_adapter *adapter) · line 961

Quoted 8 lines
drivers/net/ethernet/intel/igb/igb_main.c +5 -0
+ if (E1000_REMOVED(hw->hw_addr)) {
+ err = -EIO;
+ goto err_free;
+ }
+
 for (i = 0; i < adapter->num_q_vectors; i++) {
 struct igb_q_vector *q_vector = adapter->q_vector[i];

Instead of using E1000_REMOVED we should just replace the adapter->hw.hw_addr in the setup of the itr_register with adapter->io_addr like you did for Rx/Tx below.

Quoted 6 lines

I just tried that, and it reliably blows up horrifically, wedging the machine to the point where all I could get was a screen shot with my phone thus far, when we jump from igb_request_msix() to igb_configure_msix() to igb_assign_vector() and finally to igb_write_ivar(), at least as best as I can tell from what I was able to see in the trace remnants still on-screen.

Take a look at array_rd32, that is buggy and doesn't match up with the rd32 implementation. If you fix that then the blow-up should go away.

You shouldn't need to worry about itr_register since it will only get triggered if the interrupt can be enabled and that shouldn't be able to happen if the device is not present.

...

Quoted 8 lines
Quoted 5 lines

Just switching to adapter->io_addr everywhere seems to not work as noted above. :\ Note that I'm also chasing this from the other end with the author of the pci patches that seem to have triggered this, so the real bug might be over in pci-land, but hardening against explosions in igb still seems like a worthwhile effort here.

I am pretty sure array_rd32 is the problem. If you fix that then I suspect you should quit seeing any further issues.

Okay, I've reworked array_rd32 a bit to call igb_rd32 as well, and with that and some other minor tweaks, nothing blowing up. The hardware still doesn't work when hotplugged, due to disappearing off the pci bus temporarily, but I think that's a pci hotplug issue, not igb's fault. Things seem to behave just fine if the device is connected at boot. Updated patch forthcoming for dissection...

Signature · 2 lines
Jarod Wilson
jarod at redhat.com
#6 lore raw

PATCH 1/1 igb: improve handling of disconnected adapters

Some pci changes upcoming in 4.3 seem to cause additional disconnects, which can happen at unfortuitous times for igb, leading to issues such as this:

[  414.440115] igb 0000:15:00.0: enabling device (0000 -> 0002)
[  414.474934] pps pps0: new PPS source ptp1
[  414.474937] igb 0000:15:00.0: added PHC on eth0
[  414.474938] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[  414.474940] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[  414.475072] igb 0000:15:00.0: eth0: PBA No: 000200-000
[  414.475073] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[  414.478453] igb 0000:15:00.0 enp21s0: renamed from eth0
[  414.497747] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[  414.536745] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[  414.854808] BUG: unable to handle kernel paging request at 0000000000003818
[  414.854827] IP: [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.854846] PGD 0
[  414.854849] Oops: 0002 [#1] SMP
[  414.854856] Modules linked in: firewire_ohci firewire_core crc_itu_t igb dca ctr ccm arc4 iwlmvm mac80211 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter bnep dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_hdmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel iTCO_wdt ppdev kvm iTCO_vendor_support hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
[  414.855073]  drbg ansi_cprng snd_hda_codec_realtek snd_hda_codec_generic aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_codec microcode snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm iwlwifi uvcvideo btusb cfg80211 videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel bluetooth v4l2_common snd_timer videodev snd parport_pc rtsx_pci_ms joydev pcspkr input_leds i2c_i801 media sg memstick rfkill soundcore lpc_ich 8250_fintek parport mei_me hp_accel ie31200_edac shpchp lis3lv02d mei edac_core input_polldev hp_wireless tpm_infineon sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom rtsx_pci_sdmmc mmc_core crc32c_intel serio_raw rtsx_pci nouveau mxm_wmi ahci hwmon libahci e1000e drm_kms_helper
[  414.855309]  ptp xhci_pci pps_core ttm xhci_hcd wmi video ipv6 autofs4
[  414.855331] CPU: 2 PID: 875 Comm: NetworkManager Not tainted 4.2.0-5.el7_UNSUPPORTED.x86_64 #1
[  414.855348] Hardware name: Hewlett-Packard HP ZBook 15 G2/2253, BIOS M70 Ver. 01.07 02/26/2015
[  414.855365] task: ffff880484698c00 ti: ffff88005859c000 task.ti: ffff88005859c000
[  414.855380] RIP: 0010:[<ffffffffa0b95a9c>]  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.855401] RSP: 0018:ffff88005859f608  EFLAGS: 00010246
[  414.855410] RAX: 0000000000003818 RBX: 0000000000000000 RCX: 0000000000003818
[  414.855424] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00000000002a9fe6
[  414.855437] RBP: ffff88005859f638 R08: 0000000003030300 R09: 00000000ffffffe7
[  414.855451] R10: ffffffff81fa91b4 R11: 00000000000007e3 R12: 0000000000000000
[  414.855464] R13: ffff880471c98840 R14: ffff8804670a1180 R15: 0000000483cce000
[  414.855478] FS:  00007f389c6fb8c0(0000) GS:ffff88049dc80000(0000) knlGS:0000000000000000
[  414.855493] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  414.855504] CR2: 0000000000003818 CR3: 00000004875da000 CR4: 00000000001406e0
[  414.855518] Stack:
[  414.855520]  ffff88005859f638 ffff880471c98840 ffff880471c98df8 0000000000000001
[  414.855538]  ffff880471c98848 0000000000000001 ffff88005859f698 ffffffffa0b99cb0
[  414.855555]  ffff88005859f678 59ab02179a7fe4d0 f3ce6b27ad46225f f5454218094e72d1
[  414.855572] Call Trace:
[  414.855577]  [<ffffffffa0b99cb0>] igb_configure+0x240/0x400 [igb]
[  414.855590]  [<ffffffffa0b99f32>] __igb_open+0xc2/0x560 [igb]
[  414.855602]  [<ffffffff8108f43d>] ? notifier_call_chain+0x4d/0x80
[  414.855614]  [<ffffffffa0b9a540>] igb_open+0x10/0x20 [igb]
[  414.855625]  [<ffffffff81581b81>] __dev_open+0xb1/0x130
[  414.855636]  [<ffffffff81581e91>] __dev_change_flags+0xa1/0x160
[  414.855647]  [<ffffffff81581f79>] dev_change_flags+0x29/0x60
[  414.855658]  [<ffffffff8158efc3>] do_setlink+0x5d3/0xaa0
[  414.855679]  [<ffffffff81308073>] ? nla_parse+0xa3/0x100
[  414.855689]  [<ffffffff815905f0>] rtnl_newlink+0x4f0/0x880
[  414.855700]  [<ffffffff815901f3>] ? rtnl_newlink+0xf3/0x880
[  414.855721]  [<ffffffff815ae23e>] ? netlink_unicast+0x1ae/0x220
[  414.855734]  [<ffffffff81266648>] ? security_capable+0x48/0x60
[  414.855746]  [<ffffffff810796bd>] ? ns_capable+0x2d/0x60
[  414.855756]  [<ffffffff8158db35>] rtnetlink_rcv_msg+0x95/0x240
[  414.855768]  [<ffffffff8126adc0>] ? sock_has_perm+0x70/0x90
[  414.855779]  [<ffffffff8158daa0>] ? rtnetlink_rcv+0x40/0x40
[  414.855789]  [<ffffffff815ae7ff>] netlink_rcv_skb+0xaf/0xc0
[  414.855800]  [<ffffffff8158da8c>] rtnetlink_rcv+0x2c/0x40
[  414.855810]  [<ffffffff815ae1de>] netlink_unicast+0x14e/0x220
[  414.855821]  [<ffffffff815ae5ca>] netlink_sendmsg+0x31a/0x390
[  414.855833]  [<ffffffff81563208>] sock_sendmsg+0x38/0x50
[  414.855843]  [<ffffffff81563b4e>] ___sys_sendmsg+0x27e/0x2a0
[  414.855855]  [<ffffffff8123d82f>] ? sysctl_head_finish+0x3f/0x50
[  414.855866]  [<ffffffff81077c10>] ? proc_put_long+0xb0/0xb0
[  414.855877]  [<ffffffff8123d9d9>] ? proc_sys_call_handler+0x79/0xc0
[  414.855890]  [<ffffffff812ec2fc>] ? lockref_put_or_lock+0x4c/0x80
[  414.855902]  [<ffffffff81564432>] __sys_sendmsg+0x42/0x80
[  414.855913]  [<ffffffff81564482>] SyS_sendmsg+0x12/0x20
[  414.855924]  [<ffffffff8165042e>] entry_SYSCALL_64_fastpath+0x12/0x71
[  414.855935] Code: c1 49 89 4e 30 49 8b 85 c0 05 00 00 48 85 c0 0f 84 0e 01 00 00 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b 46 30 31 d2 <89> 10 41 8b 95 44 06 00 00 b8 14 01 10 02 83
fa 05 74 0b 83 fa
[  414.856037] RIP  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.856052]  RSP <ffff88005859f608>
[  414.856057] CR2: 0000000000003818
[  414.872327] ---[ end trace e97522c0c584ea70 ]---

This can at least be reduced to a harmless disconnect if we do some additional checking of device presence, similar to ixgbe, and use a saved io_addr instead of hw_addr, which we may have made NULL.

[ 8010.562550] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 8010.597402] pps pps0: new PPS source ptp1
[ 8010.597406] igb 0000:15:00.0: added PHC on eth0
[ 8010.597407] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 8010.597409] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[ 8010.597543] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 8010.597545] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 8010.600468] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 8010.619354] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 8010.663999] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[ 8011.012427] igb 0000:15:00.0: Unable to allocate memory for vectors
CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC Alexand Duyck <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org
---
 drivers/net/ethernet/intel/igb/e1000_regs.h |  7 +++----
 drivers/net/ethernet/intel/igb/igb_main.c   | 14 +++++++-------
 2 files changed, 10 insertions(+), 11 deletions(-)
drivers/net/ethernet/intel/igb/e1000_regs.h +3 -4
index 4af2870..0227f0b 100644
@@ -368,7 +368,7 @@
 
 struct e1000_hw;
 
-u32 igb_rd32(struct e1000_hw *hw, u32 reg);
+u32 igb_rd32(struct e1000_hw *hw, u32 reg, u32 offset);
 
 /* write operations, indexed using DWORDS */
 #define wr32(reg, val) \
@@ -378,15 +378,14 @@ do { \
 		writel((val), &hw_addr[(reg)]); \
 } while (0)
 
-#define rd32(reg) (igb_rd32(hw, reg))
+#define rd32(reg) (igb_rd32(hw, reg, 0))
 
 #define wrfl() ((void)rd32(E1000_STATUS))
 
 #define array_wr32(reg, offset, value) \
 	wr32((reg) + ((offset) << 2), (value))
 
-#define array_rd32(reg, offset) \
-	(readl(hw->hw_addr + reg + ((offset) << 2)))
+#define array_rd32(reg, offset) (igb_rd32(hw, reg, offset << 2))
 
 /* DMA Coalescing registers */
 #define E1000_PCIEMISC	0x05BB8 /* PCIE misc config register */
drivers/net/ethernet/intel/igb/igb_main.c +7 -7
index d645d65..3b21f85 100644
@@ -743,7 +743,7 @@ static void igb_cache_ring_register(struct igb_adapter *adapter)
 	}
 }
 
-u32 igb_rd32(struct e1000_hw *hw, u32 reg)
+u32 igb_rd32(struct e1000_hw *hw, u32 reg, u32 offset)
 {
 	struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
 	u8 __iomem *hw_addr = ACCESS_ONCE(hw->hw_addr);
@@ -752,10 +752,10 @@ u32 igb_rd32(struct e1000_hw *hw, u32 reg)
 	if (E1000_REMOVED(hw_addr))
 		return ~value;
 
-	value = readl(&hw_addr[reg]);
+	value = readl(&hw_addr[reg + offset]);
 
 	/* reads should not return all F's */
-	if (!(~value) && (!reg || !(~readl(hw_addr)))) {
+	if (!(~value) && (!(reg + offset) || !(~readl(hw_addr)))) {
 		struct net_device *netdev = igb->netdev;
 		hw->hw_addr = NULL;
 		netif_device_detach(netdev);
@@ -959,7 +959,7 @@ static int igb_request_msix(struct igb_adapter *adapter)
 
 		vector++;
 
-		q_vector->itr_register = hw->hw_addr + E1000_EITR(vector);
+		q_vector->itr_register = adapter->io_addr + E1000_EITR(vector);
 
 		if (q_vector->rx.ring && q_vector->tx.ring)
 			sprintf(q_vector->name, "%s-TxRx-%u", netdev->name,
@@ -1230,7 +1230,7 @@ static int igb_alloc_q_vector(struct igb_adapter *adapter,
 	q_vector->tx.work_limit = adapter->tx_work_limit;
 
 	/* initialize ITR configuration */
-	q_vector->itr_register = adapter->hw.hw_addr + E1000_EITR(0);
+	q_vector->itr_register = adapter->io_addr + E1000_EITR(0);
 	q_vector->itr_val = IGB_START_ITR;
 
 	/* initialize pointer to rings */
@@ -3284,7 +3284,7 @@ void igb_configure_tx_ring(struct igb_adapter *adapter,
 	     tdba & 0x00000000ffffffffULL);
 	wr32(E1000_TDBAH(reg_idx), tdba >> 32);
 
-	ring->tail = hw->hw_addr + E1000_TDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_TDT(reg_idx);
 	wr32(E1000_TDH(reg_idx), 0);
 	writel(0, ring->tail);
 
@@ -3640,7 +3640,7 @@ void igb_configure_rx_ring(struct igb_adapter *adapter,
 	     ring->count * sizeof(union e1000_adv_rx_desc));
 
 	/* initialize head and tail */
-	ring->tail = hw->hw_addr + E1000_RDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_RDT(reg_idx);
 	wr32(E1000_RDH(reg_idx), 0);
 	writel(0, ring->tail);
 
Signature · 1 line
1.8.3.1
#7 lore raw

Re: [PATCH] igb: improve handling of disconnected adapters

config: i386-randconfig-x009-201540 (attached as .config) reproduce:

Hi Jarod,

[auto build test ERROR on v4.3-rc4 -- if it's inappropriate base, please ignore]

config: i386-randconfig-x009-201540 (attached as .config) reproduce:

        # save the attached .config to linux build tree
        make ARCH=i386 

All error/warnings (new ones prefixed by >>):

drivers/net/ethernet/intel/igb/igb_main.c: In function 'igb_request_msix':

>> drivers/net/ethernet/intel/igb/igb_main.c:962:35: error: 'struct igb_adapter' has no member named 'io_addr'
      q_vector->itr_register = adapter->io_addr + E1000_EITR(vector);
                                      ^
>> drivers/net/ethernet/intel/igb/igb_main.c:949:19: warning: unused variable 'hw' [-Wunused-variable]
     struct e1000_hw *hw = &adapter->hw;
                      ^
   drivers/net/ethernet/intel/igb/igb_main.c: In function 'igb_alloc_q_vector':
   drivers/net/ethernet/intel/igb/igb_main.c:1233:34: error: 'struct igb_adapter' has no member named 'io_addr'
     q_vector->itr_register = adapter->io_addr + E1000_EITR(0);
                                     ^
   drivers/net/ethernet/intel/igb/igb_main.c: In function 'igb_configure_tx_ring':
   drivers/net/ethernet/intel/igb/igb_main.c:3282:22: error: 'struct igb_adapter' has no member named 'io_addr'
     ring->tail = adapter->io_addr + E1000_TDT(reg_idx);
                         ^
   drivers/net/ethernet/intel/igb/igb_main.c: In function 'igb_configure_rx_ring':
   drivers/net/ethernet/intel/igb/igb_main.c:3638:22: error: 'struct igb_adapter' has no member named 'io_addr'
     ring->tail = adapter->io_addr + E1000_RDT(reg_idx);
                         ^

vim +962 drivers/net/ethernet/intel/igb/igb_main.c

943 * igb_request_msix allocates MSI-X vectors and requests interrupts from the
944 * kernel.
945 **/
946 static int igb_request_msix(struct igb_adapter *adapter)
947 {
948 struct net_device *netdev = adapter->netdev;

 > 949		struct e1000_hw *hw = &adapter->hw;
   950		int i, err = 0, vector = 0, free_vector = 0;
   951	
   952		err = request_irq(adapter->msix_entries[vector].vector,
   953				  igb_msix_other, 0, netdev->name, adapter);
   954		if (err)
   955			goto err_out;
   956	
   957		for (i = 0; i < adapter->num_q_vectors; i++) {
   958			struct igb_q_vector *q_vector = adapter->q_vector[i];
   959	
   960			vector++;
   961	
 > 962			q_vector->itr_register = adapter->io_addr + E1000_EITR(vector);

963
964 if (q_vector->rx.ring && q_vector->tx.ring)
965 sprintf(q_vector->name, "%s-TxRx-%u", netdev->name,

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
-------------- next part --------------
A non-text attachment was scrubbed...
Name: .config.gz
Type: application/octet-stream
Size: 24334 bytes
Desc: not available
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20151007/212bf100/attachment-0001.obj>

#8 lore raw

Re: [PATCH] igb: improve handling of disconnected adapters

In reply to Jarod Wilson
Quoted 108 lines · Jarod Wilson

Some pci changes upcoming in 4.3 seem to cause additional disconnects, which can happen at unfortuitous times for igb, leading to issues such as this:

[  414.440115] igb 0000:15:00.0: enabling device (0000 -> 0002)
[  414.474934] pps pps0: new PPS source ptp1
[  414.474937] igb 0000:15:00.0: added PHC on eth0
[  414.474938] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[  414.474940] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[  414.475072] igb 0000:15:00.0: eth0: PBA No: 000200-000
[  414.475073] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[  414.478453] igb 0000:15:00.0 enp21s0: renamed from eth0
[  414.497747] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[  414.536745] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[  414.854808] BUG: unable to handle kernel paging request at 0000000000003818
[  414.854827] IP: [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.854846] PGD 0
[  414.854849] Oops: 0002 [#1] SMP
[  414.854856] Modules linked in: firewire_ohci firewire_core crc_itu_t igb dca ctr ccm arc4 iwlmvm mac80211 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter bnep dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_hdmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel iTCO_wdt ppdev kvm iTCO_vendor_support hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
[  414.855073]  drbg ansi_cprng snd_hda_codec_realtek snd_hda_codec_generic aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_codec microcode snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm iwlwifi uvcvideo btusb cfg80211 videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel bluetooth v4l2_common snd_timer videodev snd parport_pc rtsx_pci_ms joydev pcspkr input_leds i2c_i801 media sg memstick rfkill soundcore lpc_ich 8250_fintek parport mei_me hp_accel ie31200_edac shpchp lis3lv02d mei edac_core input_polldev hp_wireless tpm_infineon sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom rtsx_pci_sdmmc mmc_core crc32c_intel serio_raw rtsx_pci nouveau mxm_wmi ahci hwmon libahci e1000e drm_kms_helper
[  414.855309]  ptp xhci_pci pps_core ttm xhci_hcd wmi video ipv6 autofs4
[  414.855331] CPU: 2 PID: 875 Comm: NetworkManager Not tainted 4.2.0-5.el7_UNSUPPORTED.x86_64 #1
[  414.855348] Hardware name: Hewlett-Packard HP ZBook 15 G2/2253, BIOS M70 Ver. 01.07 02/26/2015
[  414.855365] task: ffff880484698c00 ti: ffff88005859c000 task.ti: ffff88005859c000
[  414.855380] RIP: 0010:[<ffffffffa0b95a9c>]  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.855401] RSP: 0018:ffff88005859f608  EFLAGS: 00010246
[  414.855410] RAX: 0000000000003818 RBX: 0000000000000000 RCX: 0000000000003818
[  414.855424] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00000000002a9fe6
[  414.855437] RBP: ffff88005859f638 R08: 0000000003030300 R09: 00000000ffffffe7
[  414.855451] R10: ffffffff81fa91b4 R11: 00000000000007e3 R12: 0000000000000000
[  414.855464] R13: ffff880471c98840 R14: ffff8804670a1180 R15: 0000000483cce000
[  414.855478] FS:  00007f389c6fb8c0(0000) GS:ffff88049dc80000(0000) knlGS:0000000000000000
[  414.855493] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  414.855504] CR2: 0000000000003818 CR3: 00000004875da000 CR4: 00000000001406e0
[  414.855518] Stack:
[  414.855520]  ffff88005859f638 ffff880471c98840 ffff880471c98df8 0000000000000001
[  414.855538]  ffff880471c98848 0000000000000001 ffff88005859f698 ffffffffa0b99cb0
[  414.855555]  ffff88005859f678 59ab02179a7fe4d0 f3ce6b27ad46225f f5454218094e72d1
[  414.855572] Call Trace:
[  414.855577]  [<ffffffffa0b99cb0>] igb_configure+0x240/0x400 [igb]
[  414.855590]  [<ffffffffa0b99f32>] __igb_open+0xc2/0x560 [igb]
[  414.855602]  [<ffffffff8108f43d>] ? notifier_call_chain+0x4d/0x80
[  414.855614]  [<ffffffffa0b9a540>] igb_open+0x10/0x20 [igb]
[  414.855625]  [<ffffffff81581b81>] __dev_open+0xb1/0x130
[  414.855636]  [<ffffffff81581e91>] __dev_change_flags+0xa1/0x160
[  414.855647]  [<ffffffff81581f79>] dev_change_flags+0x29/0x60
[  414.855658]  [<ffffffff8158efc3>] do_setlink+0x5d3/0xaa0
[  414.855679]  [<ffffffff81308073>] ? nla_parse+0xa3/0x100
[  414.855689]  [<ffffffff815905f0>] rtnl_newlink+0x4f0/0x880
[  414.855700]  [<ffffffff815901f3>] ? rtnl_newlink+0xf3/0x880
[  414.855721]  [<ffffffff815ae23e>] ? netlink_unicast+0x1ae/0x220
[  414.855734]  [<ffffffff81266648>] ? security_capable+0x48/0x60
[  414.855746]  [<ffffffff810796bd>] ? ns_capable+0x2d/0x60
[  414.855756]  [<ffffffff8158db35>] rtnetlink_rcv_msg+0x95/0x240
[  414.855768]  [<ffffffff8126adc0>] ? sock_has_perm+0x70/0x90
[  414.855779]  [<ffffffff8158daa0>] ? rtnetlink_rcv+0x40/0x40
[  414.855789]  [<ffffffff815ae7ff>] netlink_rcv_skb+0xaf/0xc0
[  414.855800]  [<ffffffff8158da8c>] rtnetlink_rcv+0x2c/0x40
[  414.855810]  [<ffffffff815ae1de>] netlink_unicast+0x14e/0x220
[  414.855821]  [<ffffffff815ae5ca>] netlink_sendmsg+0x31a/0x390
[  414.855833]  [<ffffffff81563208>] sock_sendmsg+0x38/0x50
[  414.855843]  [<ffffffff81563b4e>] ___sys_sendmsg+0x27e/0x2a0
[  414.855855]  [<ffffffff8123d82f>] ? sysctl_head_finish+0x3f/0x50
[  414.855866]  [<ffffffff81077c10>] ? proc_put_long+0xb0/0xb0
[  414.855877]  [<ffffffff8123d9d9>] ? proc_sys_call_handler+0x79/0xc0
[  414.855890]  [<ffffffff812ec2fc>] ? lockref_put_or_lock+0x4c/0x80
[  414.855902]  [<ffffffff81564432>] __sys_sendmsg+0x42/0x80
[  414.855913]  [<ffffffff81564482>] SyS_sendmsg+0x12/0x20
[  414.855924]  [<ffffffff8165042e>] entry_SYSCALL_64_fastpath+0x12/0x71
[  414.855935] Code: c1 49 89 4e 30 49 8b 85 c0 05 00 00 48 85 c0 0f 84 0e 01 00 00 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b 46 30 31 d2 <89> 10 41 8b 95 44 06 00 00 b8 14 01 10 02 83
fa 05 74 0b 83 fa
[  414.856037] RIP  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.856052]  RSP <ffff88005859f608>
[  414.856057] CR2: 0000000000003818
[  414.872327] ---[ end trace e97522c0c584ea70 ]---

This can at least be reduced to a harmless disconnect if we do some additional checking of device presence, similar to ixgbe, and use a saved io_addr instead of hw_addr, which we may have made NULL.

[ 8010.562550] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 8010.597402] pps pps0: new PPS source ptp1
[ 8010.597406] igb 0000:15:00.0: added PHC on eth0
[ 8010.597407] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 8010.597409] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[ 8010.597543] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 8010.597545] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 8010.600468] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 8010.619354] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 8010.663999] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[ 8011.012427] igb 0000:15:00.0: Unable to allocate memory for vectors
CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC Alexand Duyck <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org
---
  drivers/net/ethernet/intel/igb/e1000_regs.h |  7 +++----
  drivers/net/ethernet/intel/igb/igb_main.c   | 14 +++++++-------
  2 files changed, 10 insertions(+), 11 deletions(-)
drivers/net/ethernet/intel/igb/e1000_regs.h +0 -0
index 4af2870..0227f0b 100644

drivers/net/ethernet/intel/igb/e1000_regs.h line 372

Quoted 6 lines
drivers/net/ethernet/intel/igb/e1000_regs.h +1 -1
@@ -368,7 +368,7 @@
 
  struct e1000_hw;
 
-u32 igb_rd32(struct e1000_hw *hw, u32 reg);
+u32 igb_rd32(struct e1000_hw *hw, u32 reg, u32 offset);
 

There is no need to modify igb_rd32.

Quoted 8 lines
drivers/net/ethernet/intel/igb/e1000_regs.h +1 -1
  /* write operations, indexed using DWORDS */
  #define wr32(reg, val) \
@@ -378,15 +378,14 @@ do { \
  		writel((val), &hw_addr[(reg)]); \
  } while (0)
 
-#define rd32(reg) (igb_rd32(hw, reg))
+#define rd32(reg) (igb_rd32(hw, reg, 0))

drivers/net/ethernet/intel/igb/e1000_regs.h do { \ · line 389

Quoted 10 lines
drivers/net/ethernet/intel/igb/e1000_regs.h +1 -2
 
  #define wrfl() ((void)rd32(E1000_STATUS))
 
  #define array_wr32(reg, offset, value) \
  	wr32((reg) + ((offset) << 2), (value))
 
-#define array_rd32(reg, offset) \
-	(readl(hw->hw_addr + reg + ((offset) << 2)))
+#define array_rd32(reg, offset) (igb_rd32(hw, reg, offset << 2))
 

What you could do here is just call (igb_rd32(hw, reg + ((offset) << 2))). It is important to keep offset wrapped in an extra set of parenthesis because this is a macro so if there is some expression being passed as the offset you want to make sure the result of the expression is shifted instead of the last part of it.

Quoted 18 lines
drivers/net/ethernet/intel/igb/e1000_regs.h +0 -0
  /* DMA Coalescing registers */
  #define E1000_PCIEMISC	0x05BB8 /* PCIE misc config register */
drivers/net/ethernet/intel/igb/igb_main.c +1 -1
index d645d65..3b21f85 100644
@@ -743,7 +743,7 @@ static void igb_cache_ring_register(struct igb_adapter *adapter)
  	}
  }
 
-u32 igb_rd32(struct e1000_hw *hw, u32 reg)
+u32 igb_rd32(struct e1000_hw *hw, u32 reg, u32 offset)
  {
  	struct igb_adapter *igb = container_of(hw, struct igb_adapter, hw);
  	u8 __iomem *hw_addr = ACCESS_ONCE(hw->hw_addr);
@@ -752,10 +752,10 @@ u32 igb_rd32(struct e1000_hw *hw, u32 reg)
  	if (E1000_REMOVED(hw_addr))
  		return ~value;

drivers/net/ethernet/intel/igb/igb_main.c u32 igb_rd32(struct e1000_hw *hw, u32 reg) · line 761

Quoted 10 lines
drivers/net/ethernet/intel/igb/igb_main.c +2 -2
 
-	value = readl(&hw_addr[reg]);
+	value = readl(&hw_addr[reg + offset]);
 
  	/* reads should not return all F's */
-	if (!(~value) && (!reg || !(~readl(hw_addr)))) {
+	if (!(~value) && (!(reg + offset) || !(~readl(hw_addr)))) {
  		struct net_device *netdev = igb->netdev;
  		hw->hw_addr = NULL;
  		netif_device_detach(netdev);

These changes aren't necessary if you just pass reg + offset instead of reg and offset as two separate variables.

Quoted 36 lines
drivers/net/ethernet/intel/igb/igb_main.c +4 -4
@@ -959,7 +959,7 @@ static int igb_request_msix(struct igb_adapter *adapter)
 
  		vector++;
 
-		q_vector->itr_register = hw->hw_addr + E1000_EITR(vector);
+		q_vector->itr_register = adapter->io_addr + E1000_EITR(vector);
 
  		if (q_vector->rx.ring && q_vector->tx.ring)
  			sprintf(q_vector->name, "%s-TxRx-%u", netdev->name,
@@ -1230,7 +1230,7 @@ static int igb_alloc_q_vector(struct igb_adapter *adapter,
  	q_vector->tx.work_limit = adapter->tx_work_limit;
 
  	/* initialize ITR configuration */
-	q_vector->itr_register = adapter->hw.hw_addr + E1000_EITR(0);
+	q_vector->itr_register = adapter->io_addr + E1000_EITR(0);
  	q_vector->itr_val = IGB_START_ITR;
 
  	/* initialize pointer to rings */
@@ -3284,7 +3284,7 @@ void igb_configure_tx_ring(struct igb_adapter *adapter,
  	     tdba & 0x00000000ffffffffULL);
  	wr32(E1000_TDBAH(reg_idx), tdba >> 32);
 
-	ring->tail = hw->hw_addr + E1000_TDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_TDT(reg_idx);
  	wr32(E1000_TDH(reg_idx), 0);
  	writel(0, ring->tail);
 
@@ -3640,7 +3640,7 @@ void igb_configure_rx_ring(struct igb_adapter *adapter,
  	     ring->count * sizeof(union e1000_adv_rx_desc));
 
  	/* initialize head and tail */
-	ring->tail = hw->hw_addr + E1000_RDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_RDT(reg_idx);
  	wr32(E1000_RDH(reg_idx), 0);
  	writel(0, ring->tail);
 
#9 lore raw

PATCH v3 1/1 igb: improve handling of disconnected adapters

Some pci changes upcoming in 4.3 seem to cause additional disconnects, which can happen at unfortuitous times for igb, leading to issues such as this:

[  414.440115] igb 0000:15:00.0: enabling device (0000 -> 0002)
[  414.474934] pps pps0: new PPS source ptp1
[  414.474937] igb 0000:15:00.0: added PHC on eth0
[  414.474938] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[  414.474940] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[  414.475072] igb 0000:15:00.0: eth0: PBA No: 000200-000
[  414.475073] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[  414.478453] igb 0000:15:00.0 enp21s0: renamed from eth0
[  414.497747] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[  414.536745] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[  414.854808] BUG: unable to handle kernel paging request at 0000000000003818
[  414.854827] IP: [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.854846] PGD 0
[  414.854849] Oops: 0002 [#1] SMP
[  414.854856] Modules linked in: firewire_ohci firewire_core crc_itu_t igb dca ctr ccm arc4 iwlmvm mac80211 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter bnep dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_hdmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel iTCO_wdt ppdev kvm iTCO_vendor_support hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
[  414.855073]  drbg ansi_cprng snd_hda_codec_realtek snd_hda_codec_generic aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_codec microcode snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm iwlwifi uvcvideo btusb cfg80211 videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel bluetooth v4l2_common snd_timer videodev snd parport_pc rtsx_pci_ms joydev pcspkr input_leds i2c_i801 media sg memstick rfkill soundcore lpc_ich 8250_fintek parport mei_me hp_accel ie31200_edac shpchp lis3lv02d mei edac_core input_polldev hp_wireless tpm_infineon sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom rtsx_pci_sdmmc mmc_core crc32c_intel serio_raw rtsx_pci nouveau mxm_wmi ahci hwmon libahci e1000e drm_kms_helper
[  414.855309]  ptp xhci_pci pps_core ttm xhci_hcd wmi video ipv6 autofs4
[  414.855331] CPU: 2 PID: 875 Comm: NetworkManager Not tainted 4.2.0-5.el7_UNSUPPORTED.x86_64 #1
[  414.855348] Hardware name: Hewlett-Packard HP ZBook 15 G2/2253, BIOS M70 Ver. 01.07 02/26/2015
[  414.855365] task: ffff880484698c00 ti: ffff88005859c000 task.ti: ffff88005859c000
[  414.855380] RIP: 0010:[<ffffffffa0b95a9c>]  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.855401] RSP: 0018:ffff88005859f608  EFLAGS: 00010246
[  414.855410] RAX: 0000000000003818 RBX: 0000000000000000 RCX: 0000000000003818
[  414.855424] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00000000002a9fe6
[  414.855437] RBP: ffff88005859f638 R08: 0000000003030300 R09: 00000000ffffffe7
[  414.855451] R10: ffffffff81fa91b4 R11: 00000000000007e3 R12: 0000000000000000
[  414.855464] R13: ffff880471c98840 R14: ffff8804670a1180 R15: 0000000483cce000
[  414.855478] FS:  00007f389c6fb8c0(0000) GS:ffff88049dc80000(0000) knlGS:0000000000000000
[  414.855493] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  414.855504] CR2: 0000000000003818 CR3: 00000004875da000 CR4: 00000000001406e0
[  414.855518] Stack:
[  414.855520]  ffff88005859f638 ffff880471c98840 ffff880471c98df8 0000000000000001
[  414.855538]  ffff880471c98848 0000000000000001 ffff88005859f698 ffffffffa0b99cb0
[  414.855555]  ffff88005859f678 59ab02179a7fe4d0 f3ce6b27ad46225f f5454218094e72d1
[  414.855572] Call Trace:
[  414.855577]  [<ffffffffa0b99cb0>] igb_configure+0x240/0x400 [igb]
[  414.855590]  [<ffffffffa0b99f32>] __igb_open+0xc2/0x560 [igb]
[  414.855602]  [<ffffffff8108f43d>] ? notifier_call_chain+0x4d/0x80
[  414.855614]  [<ffffffffa0b9a540>] igb_open+0x10/0x20 [igb]
[  414.855625]  [<ffffffff81581b81>] __dev_open+0xb1/0x130
[  414.855636]  [<ffffffff81581e91>] __dev_change_flags+0xa1/0x160
[  414.855647]  [<ffffffff81581f79>] dev_change_flags+0x29/0x60
[  414.855658]  [<ffffffff8158efc3>] do_setlink+0x5d3/0xaa0
[  414.855679]  [<ffffffff81308073>] ? nla_parse+0xa3/0x100
[  414.855689]  [<ffffffff815905f0>] rtnl_newlink+0x4f0/0x880
[  414.855700]  [<ffffffff815901f3>] ? rtnl_newlink+0xf3/0x880
[  414.855721]  [<ffffffff815ae23e>] ? netlink_unicast+0x1ae/0x220
[  414.855734]  [<ffffffff81266648>] ? security_capable+0x48/0x60
[  414.855746]  [<ffffffff810796bd>] ? ns_capable+0x2d/0x60
[  414.855756]  [<ffffffff8158db35>] rtnetlink_rcv_msg+0x95/0x240
[  414.855768]  [<ffffffff8126adc0>] ? sock_has_perm+0x70/0x90
[  414.855779]  [<ffffffff8158daa0>] ? rtnetlink_rcv+0x40/0x40
[  414.855789]  [<ffffffff815ae7ff>] netlink_rcv_skb+0xaf/0xc0
[  414.855800]  [<ffffffff8158da8c>] rtnetlink_rcv+0x2c/0x40
[  414.855810]  [<ffffffff815ae1de>] netlink_unicast+0x14e/0x220
[  414.855821]  [<ffffffff815ae5ca>] netlink_sendmsg+0x31a/0x390
[  414.855833]  [<ffffffff81563208>] sock_sendmsg+0x38/0x50
[  414.855843]  [<ffffffff81563b4e>] ___sys_sendmsg+0x27e/0x2a0
[  414.855855]  [<ffffffff8123d82f>] ? sysctl_head_finish+0x3f/0x50
[  414.855866]  [<ffffffff81077c10>] ? proc_put_long+0xb0/0xb0
[  414.855877]  [<ffffffff8123d9d9>] ? proc_sys_call_handler+0x79/0xc0
[  414.855890]  [<ffffffff812ec2fc>] ? lockref_put_or_lock+0x4c/0x80
[  414.855902]  [<ffffffff81564432>] __sys_sendmsg+0x42/0x80
[  414.855913]  [<ffffffff81564482>] SyS_sendmsg+0x12/0x20
[  414.855924]  [<ffffffff8165042e>] entry_SYSCALL_64_fastpath+0x12/0x71
[  414.855935] Code: c1 49 89 4e 30 49 8b 85 c0 05 00 00 48 85 c0 0f 84 0e 01 00 00 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b 46 30 31 d2 <89> 10 41 8b 95 44 06 00 00 b8 14 01 10 02 83
fa 05 74 0b 83 fa
[  414.856037] RIP  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.856052]  RSP <ffff88005859f608>
[  414.856057] CR2: 0000000000003818
[  414.872327] ---[ end trace e97522c0c584ea70 ]---

This can at least be reduced to a harmless disconnect if we do some additional checking of device presence, similar to ixgbe, and use a saved io_addr instead of hw_addr, which we may have made NULL, and clean up array_rd32 so that it uses igb_rd32 like rd32 does, per Alexander Duyck's suggestion.

[ 8010.562550] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 8010.597402] pps pps0: new PPS source ptp1
[ 8010.597406] igb 0000:15:00.0: added PHC on eth0
[ 8010.597407] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 8010.597409] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[ 8010.597543] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 8010.597545] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 8010.600468] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 8010.619354] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 8010.663999] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[ 8011.012427] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready

The device is still not functional at this point, but I believe that's a pci hotplug issue, not an igb issue.

CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC Alexander Duyck <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org

---
note: this patch is dependent on the previously submitted "igb: don't unmap hw_addr
if its NULL", which added io_addr.

v3: don't modify igb_rd32, just pass reg + ((offset) <<2) as suggested by
Alexander Duyck.

 drivers/net/ethernet/intel/igb/e1000_regs.h | 3 +--
 drivers/net/ethernet/intel/igb/igb_main.c   | 8 ++++----
 2 files changed, 5 insertions(+), 6 deletions(-)
drivers/net/ethernet/intel/igb/e1000_regs.h +1 -2
index 4af2870..e7cb23f 100644
@@ -385,8 +385,7 @@ do { \
 #define array_wr32(reg, offset, value) \
 	wr32((reg) + ((offset) << 2), (value))
 
-#define array_rd32(reg, offset) \
-	(readl(hw->hw_addr + reg + ((offset) << 2)))
+#define array_rd32(reg, offset) (igb_rd32(hw, reg + ((offset) << 2)))
 
 /* DMA Coalescing registers */
 #define E1000_PCIEMISC	0x05BB8 /* PCIE misc config register */
drivers/net/ethernet/intel/igb/igb_main.c +4 -4
index d645d65..b05d1cb 100644
@@ -959,7 +959,7 @@ static int igb_request_msix(struct igb_adapter *adapter)
 
 		vector++;
 
-		q_vector->itr_register = hw->hw_addr + E1000_EITR(vector);
+		q_vector->itr_register = adapter->io_addr + E1000_EITR(vector);
 
 		if (q_vector->rx.ring && q_vector->tx.ring)
 			sprintf(q_vector->name, "%s-TxRx-%u", netdev->name,
@@ -1230,7 +1230,7 @@ static int igb_alloc_q_vector(struct igb_adapter *adapter,
 	q_vector->tx.work_limit = adapter->tx_work_limit;
 
 	/* initialize ITR configuration */
-	q_vector->itr_register = adapter->hw.hw_addr + E1000_EITR(0);
+	q_vector->itr_register = adapter->io_addr + E1000_EITR(0);
 	q_vector->itr_val = IGB_START_ITR;
 
 	/* initialize pointer to rings */
@@ -3284,7 +3284,7 @@ void igb_configure_tx_ring(struct igb_adapter *adapter,
 	     tdba & 0x00000000ffffffffULL);
 	wr32(E1000_TDBAH(reg_idx), tdba >> 32);
 
-	ring->tail = hw->hw_addr + E1000_TDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_TDT(reg_idx);
 	wr32(E1000_TDH(reg_idx), 0);
 	writel(0, ring->tail);
 
@@ -3640,7 +3640,7 @@ void igb_configure_rx_ring(struct igb_adapter *adapter,
 	     ring->count * sizeof(union e1000_adv_rx_desc));
 
 	/* initialize head and tail */
-	ring->tail = hw->hw_addr + E1000_RDT(reg_idx);
+	ring->tail = adapter->io_addr + E1000_RDT(reg_idx);
 	wr32(E1000_RDH(reg_idx), 0);
 	writel(0, ring->tail);
 
Signature · 1 line
1.8.3.1
Acked-by on v3 1/1
#10 lore raw

Re: [PATCH v3] igb: improve handling of disconnected adapters

Quoted 107 lines · Jarod Wilson

Some pci changes upcoming in 4.3 seem to cause additional disconnects, which can happen at unfortuitous times for igb, leading to issues such as this:

[  414.440115] igb 0000:15:00.0: enabling device (0000 -> 0002)
[  414.474934] pps pps0: new PPS source ptp1
[  414.474937] igb 0000:15:00.0: added PHC on eth0
[  414.474938] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[  414.474940] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[  414.475072] igb 0000:15:00.0: eth0: PBA No: 000200-000
[  414.475073] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[  414.478453] igb 0000:15:00.0 enp21s0: renamed from eth0
[  414.497747] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[  414.536745] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[  414.854808] BUG: unable to handle kernel paging request at 0000000000003818
[  414.854827] IP: [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.854846] PGD 0
[  414.854849] Oops: 0002 [#1] SMP
[  414.854856] Modules linked in: firewire_ohci firewire_core crc_itu_t igb dca ctr ccm arc4 iwlmvm mac80211 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter bnep dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_hdmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel iTCO_wdt ppdev kvm iTCO_vendor_support hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
[  414.855073]  drbg ansi_cprng snd_hda_codec_realtek snd_hda_codec_generic aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_codec microcode snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm iwlwifi uvcvideo btusb cfg80211 videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel bluetooth v4l2_common snd_timer videodev snd parport_pc rtsx_pci_ms joydev pcspkr input_leds i2c_i801 media sg memstick rfkill soundcore lpc_ich 8250_fintek parport mei_me hp_accel ie31200_edac shpchp lis3lv02d mei edac_core input_polldev hp_wireless tpm_infineon sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom rtsx_pci_sdmmc mmc_core crc32c_intel serio_raw rtsx_pci nouveau mxm_wmi ahci hwmon libahci e1000e drm_kms_helper
[  414.855309]  ptp xhci_pci pps_core ttm xhci_hcd wmi video ipv6 autofs4
[  414.855331] CPU: 2 PID: 875 Comm: NetworkManager Not tainted 4.2.0-5.el7_UNSUPPORTED.x86_64 #1
[  414.855348] Hardware name: Hewlett-Packard HP ZBook 15 G2/2253, BIOS M70 Ver. 01.07 02/26/2015
[  414.855365] task: ffff880484698c00 ti: ffff88005859c000 task.ti: ffff88005859c000
[  414.855380] RIP: 0010:[<ffffffffa0b95a9c>]  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.855401] RSP: 0018:ffff88005859f608  EFLAGS: 00010246
[  414.855410] RAX: 0000000000003818 RBX: 0000000000000000 RCX: 0000000000003818
[  414.855424] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00000000002a9fe6
[  414.855437] RBP: ffff88005859f638 R08: 0000000003030300 R09: 00000000ffffffe7
[  414.855451] R10: ffffffff81fa91b4 R11: 00000000000007e3 R12: 0000000000000000
[  414.855464] R13: ffff880471c98840 R14: ffff8804670a1180 R15: 0000000483cce000
[  414.855478] FS:  00007f389c6fb8c0(0000) GS:ffff88049dc80000(0000) knlGS:0000000000000000
[  414.855493] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  414.855504] CR2: 0000000000003818 CR3: 00000004875da000 CR4: 00000000001406e0
[  414.855518] Stack:
[  414.855520]  ffff88005859f638 ffff880471c98840 ffff880471c98df8 0000000000000001
[  414.855538]  ffff880471c98848 0000000000000001 ffff88005859f698 ffffffffa0b99cb0
[  414.855555]  ffff88005859f678 59ab02179a7fe4d0 f3ce6b27ad46225f f5454218094e72d1
[  414.855572] Call Trace:
[  414.855577]  [<ffffffffa0b99cb0>] igb_configure+0x240/0x400 [igb]
[  414.855590]  [<ffffffffa0b99f32>] __igb_open+0xc2/0x560 [igb]
[  414.855602]  [<ffffffff8108f43d>] ? notifier_call_chain+0x4d/0x80
[  414.855614]  [<ffffffffa0b9a540>] igb_open+0x10/0x20 [igb]
[  414.855625]  [<ffffffff81581b81>] __dev_open+0xb1/0x130
[  414.855636]  [<ffffffff81581e91>] __dev_change_flags+0xa1/0x160
[  414.855647]  [<ffffffff81581f79>] dev_change_flags+0x29/0x60
[  414.855658]  [<ffffffff8158efc3>] do_setlink+0x5d3/0xaa0
[  414.855679]  [<ffffffff81308073>] ? nla_parse+0xa3/0x100
[  414.855689]  [<ffffffff815905f0>] rtnl_newlink+0x4f0/0x880
[  414.855700]  [<ffffffff815901f3>] ? rtnl_newlink+0xf3/0x880
[  414.855721]  [<ffffffff815ae23e>] ? netlink_unicast+0x1ae/0x220
[  414.855734]  [<ffffffff81266648>] ? security_capable+0x48/0x60
[  414.855746]  [<ffffffff810796bd>] ? ns_capable+0x2d/0x60
[  414.855756]  [<ffffffff8158db35>] rtnetlink_rcv_msg+0x95/0x240
[  414.855768]  [<ffffffff8126adc0>] ? sock_has_perm+0x70/0x90
[  414.855779]  [<ffffffff8158daa0>] ? rtnetlink_rcv+0x40/0x40
[  414.855789]  [<ffffffff815ae7ff>] netlink_rcv_skb+0xaf/0xc0
[  414.855800]  [<ffffffff8158da8c>] rtnetlink_rcv+0x2c/0x40
[  414.855810]  [<ffffffff815ae1de>] netlink_unicast+0x14e/0x220
[  414.855821]  [<ffffffff815ae5ca>] netlink_sendmsg+0x31a/0x390
[  414.855833]  [<ffffffff81563208>] sock_sendmsg+0x38/0x50
[  414.855843]  [<ffffffff81563b4e>] ___sys_sendmsg+0x27e/0x2a0
[  414.855855]  [<ffffffff8123d82f>] ? sysctl_head_finish+0x3f/0x50
[  414.855866]  [<ffffffff81077c10>] ? proc_put_long+0xb0/0xb0
[  414.855877]  [<ffffffff8123d9d9>] ? proc_sys_call_handler+0x79/0xc0
[  414.855890]  [<ffffffff812ec2fc>] ? lockref_put_or_lock+0x4c/0x80
[  414.855902]  [<ffffffff81564432>] __sys_sendmsg+0x42/0x80
[  414.855913]  [<ffffffff81564482>] SyS_sendmsg+0x12/0x20
[  414.855924]  [<ffffffff8165042e>] entry_SYSCALL_64_fastpath+0x12/0x71
[  414.855935] Code: c1 49 89 4e 30 49 8b 85 c0 05 00 00 48 85 c0 0f 84 0e 01 00 00 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b 46 30 31 d2 <89> 10 41 8b 95 44 06 00 00 b8 14 01 10 02 83
fa 05 74 0b 83 fa
[  414.856037] RIP  [<ffffffffa0b95a9c>] igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.856052]  RSP <ffff88005859f608>
[  414.856057] CR2: 0000000000003818
[  414.872327] ---[ end trace e97522c0c584ea70 ]---

This can at least be reduced to a harmless disconnect if we do some additional checking of device presence, similar to ixgbe, and use a saved io_addr instead of hw_addr, which we may have made NULL, and clean up array_rd32 so that it uses igb_rd32 like rd32 does, per Alexander Duyck's suggestion.

[ 8010.562550] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 8010.597402] pps pps0: new PPS source ptp1
[ 8010.597406] igb 0000:15:00.0: added PHC on eth0
[ 8010.597407] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 8010.597409] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1) e8:ea:6a:00:1b:2a
[ 8010.597543] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 8010.597545] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 8010.600468] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 8010.619354] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 8010.663999] igb 0000:15:00.0 enp21s0: PCIe link lost, device now detached
[ 8011.012427] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready

The device is still not functional at this point, but I believe that's a pci hotplug issue, not an igb issue.

CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC Alexander Duyck <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org

---
note: this patch is dependent on the previously submitted "igb: don't unmap hw_addr
if its NULL", which added io_addr.

Quoted 7 lines

v3: don't modify igb_rd32, just pass reg + ((offset) <<2) as suggested by
Alexander Duyck.

  drivers/net/ethernet/intel/igb/e1000_regs.h | 3 +--
  drivers/net/ethernet/intel/igb/igb_main.c   | 8 ++++----
  2 files changed, 5 insertions(+), 6 deletions(-)

Looks good.

#11 lore raw

Re: [PATCH v3] igb: improve handling of disconnected adapters

In reply to Jarod Wilson
Quoted 167 lines · Jarod Wilson

Some pci changes upcoming in 4.3 seem to cause additional disconnects,
which can happen at unfortuitous times for igb, leading to issues such as
this:

[  414.440115] igb 0000:15:00.0: enabling device (0000 -> 0002)
[  414.474934] pps pps0: new PPS source ptp1
[  414.474937] igb 0000:15:00.0: added PHC on eth0
[  414.474938] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network
Connection
[  414.474940] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1)
e8:ea:6a:00:1b:2a
[  414.475072] igb 0000:15:00.0: eth0: PBA No: 000200-000
[  414.475073] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx
queue(s), 4 tx queue(s)
[  414.478453] igb 0000:15:00.0 enp21s0: renamed from eth0
[  414.497747] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[  414.536745] igb 0000:15:00.0 enp21s0: PCIe link lost, device now
detached
[  414.854808] BUG: unable to handle kernel paging request at
0000000000003818
[  414.854827] IP: [<ffffffffa0b95a9c>]
igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.854846] PGD 0
[  414.854849] Oops: 0002 [#1] SMP
[  414.854856] Modules linked in: firewire_ohci firewire_core

crc_itu_t igb dca ctr ccm arc4 iwlmvm mac80211 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter bnep dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_hdmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel iTCO_wdt ppdev kvm iTCO_vendor_support hp_wmi sparse_keymap crct10dif_pclmul crc32_pclmul ghash_clmulni_intel

[  414.855073]  drbg ansi_cprng snd_hda_codec_realtek

snd_hda_codec_generic aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel snd_hda_codec microcode snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm iwlwifi uvcvideo btusb cfg80211 videobuf2_vmalloc videobuf2_memops btrtl btbcm videobuf2_core btintel bluetooth v4l2_common snd_timer videodev snd parport_pc rtsx_pci_ms joydev pcspkr input_leds i2c_i801 media sg memstick rfkill soundcore lpc_ich 8250_fintek parport mei_me hp_accel ie31200_edac shpchp lis3lv02d mei edac_core input_polldev hp_wireless tpm_infineon sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom rtsx_pci_sdmmc mmc_core crc32c_intel serio_raw rtsx_pci nouveau mxm_wmi ahci hwmon libahci e1000e drm_kms_helper

[  414.855309]  ptp xhci_pci pps_core ttm xhci_hcd wmi video ipv6
autofs4
[  414.855331] CPU: 2 PID: 875 Comm: NetworkManager Not tainted 4.2.0
-5.el7_UNSUPPORTED.x86_64 #1
[  414.855348] Hardware name: Hewlett-Packard HP ZBook 15 G2/2253,
BIOS M70 Ver. 01.07 02/26/2015
[  414.855365] task: ffff880484698c00 ti: ffff88005859c000 task.ti:
ffff88005859c000
[  414.855380] RIP: 0010:[<ffffffffa0b95a9c>]  [<ffffffffa0b95a9c>]
igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.855401] RSP: 0018:ffff88005859f608  EFLAGS: 00010246
[  414.855410] RAX: 0000000000003818 RBX: 0000000000000000 RCX:
0000000000003818
[  414.855424] RDX: 0000000000000000 RSI: 0000000000000008 RDI:
00000000002a9fe6
[  414.855437] RBP: ffff88005859f638 R08: 0000000003030300 R09:
00000000ffffffe7
[  414.855451] R10: ffffffff81fa91b4 R11: 00000000000007e3 R12:
0000000000000000
[  414.855464] R13: ffff880471c98840 R14: ffff8804670a1180 R15:
0000000483cce000
[  414.855478] FS:  00007f389c6fb8c0(0000) GS:ffff88049dc80000(0000)
knlGS:0000000000000000
[  414.855493] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  414.855504] CR2: 0000000000003818 CR3: 00000004875da000 CR4:
00000000001406e0
[  414.855518] Stack:
[  414.855520]  ffff88005859f638 ffff880471c98840 ffff880471c98df8
0000000000000001
[  414.855538]  ffff880471c98848 0000000000000001 ffff88005859f698
ffffffffa0b99cb0
[  414.855555]  ffff88005859f678 59ab02179a7fe4d0 f3ce6b27ad46225f
f5454218094e72d1
[  414.855572] Call Trace:
[  414.855577]  [<ffffffffa0b99cb0>] igb_configure+0x240/0x400 [igb]
[  414.855590]  [<ffffffffa0b99f32>] __igb_open+0xc2/0x560 [igb]
[  414.855602]  [<ffffffff8108f43d>] ? notifier_call_chain+0x4d/0x80
[  414.855614]  [<ffffffffa0b9a540>] igb_open+0x10/0x20 [igb]
[  414.855625]  [<ffffffff81581b81>] __dev_open+0xb1/0x130
[  414.855636]  [<ffffffff81581e91>] __dev_change_flags+0xa1/0x160
[  414.855647]  [<ffffffff81581f79>] dev_change_flags+0x29/0x60
[  414.855658]  [<ffffffff8158efc3>] do_setlink+0x5d3/0xaa0
[  414.855679]  [<ffffffff81308073>] ? nla_parse+0xa3/0x100
[  414.855689]  [<ffffffff815905f0>] rtnl_newlink+0x4f0/0x880
[  414.855700]  [<ffffffff815901f3>] ? rtnl_newlink+0xf3/0x880
[  414.855721]  [<ffffffff815ae23e>] ? netlink_unicast+0x1ae/0x220
[  414.855734]  [<ffffffff81266648>] ? security_capable+0x48/0x60
[  414.855746]  [<ffffffff810796bd>] ? ns_capable+0x2d/0x60
[  414.855756]  [<ffffffff8158db35>] rtnetlink_rcv_msg+0x95/0x240
[  414.855768]  [<ffffffff8126adc0>] ? sock_has_perm+0x70/0x90
[  414.855779]  [<ffffffff8158daa0>] ? rtnetlink_rcv+0x40/0x40
[  414.855789]  [<ffffffff815ae7ff>] netlink_rcv_skb+0xaf/0xc0
[  414.855800]  [<ffffffff8158da8c>] rtnetlink_rcv+0x2c/0x40
[  414.855810]  [<ffffffff815ae1de>] netlink_unicast+0x14e/0x220
[  414.855821]  [<ffffffff815ae5ca>] netlink_sendmsg+0x31a/0x390
[  414.855833]  [<ffffffff81563208>] sock_sendmsg+0x38/0x50
[  414.855843]  [<ffffffff81563b4e>] ___sys_sendmsg+0x27e/0x2a0
[  414.855855]  [<ffffffff8123d82f>] ? sysctl_head_finish+0x3f/0x50
[  414.855866]  [<ffffffff81077c10>] ? proc_put_long+0xb0/0xb0
[  414.855877]  [<ffffffff8123d9d9>] ?
proc_sys_call_handler+0x79/0xc0
[  414.855890]  [<ffffffff812ec2fc>] ? lockref_put_or_lock+0x4c/0x80
[  414.855902]  [<ffffffff81564432>] __sys_sendmsg+0x42/0x80
[  414.855913]  [<ffffffff81564482>] SyS_sendmsg+0x12/0x20
[  414.855924]  [<ffffffff8165042e>]
entry_SYSCALL_64_fastpath+0x12/0x71
[  414.855935] Code: c1 49 89 4e 30 49 8b 85 c0 05 00 00 48 85 c0 0f
84 0e 01 00 00 81 c2 10 38 00 00 48 63 d2 48 01 d0 31 d2 89 10 49 8b
46 30 31 d2 <89> 10 41 8b 95 44 06 00 00 b8 14 01 10 02 83
fa 05 74 0b 83 fa
[  414.856037] RIP  [<ffffffffa0b95a9c>]
igb_configure_tx_ring+0x14c/0x250 [igb]
[  414.856052]  RSP <ffff88005859f608>
[  414.856057] CR2: 0000000000003818
[  414.872327] ---[ end trace e97522c0c584ea70 ]---

This can at least be reduced to a harmless disconnect if we do some additional checking of device presence, similar to ixgbe, and use a saved
io_addr instead of hw_addr, which we may have made NULL, and clean up array_rd32 so that it uses igb_rd32 like rd32 does, per Alexander Duyck's
suggestion.

[ 8010.562550] igb 0000:15:00.0: enabling device (0000 -> 0002)
[ 8010.597402] pps pps0: new PPS source ptp1
[ 8010.597406] igb 0000:15:00.0: added PHC on eth0
[ 8010.597407] igb 0000:15:00.0: Intel(R) Gigabit Ethernet Network
Connection
[ 8010.597409] igb 0000:15:00.0: eth0: (PCIe:2.5Gb/s:Width x1)
e8:ea:6a:00:1b:2a
[ 8010.597543] igb 0000:15:00.0: eth0: PBA No: 000200-000
[ 8010.597545] igb 0000:15:00.0: Using MSI-X interrupts. 4 rx
queue(s), 4 tx queue(s)
[ 8010.600468] igb 0000:15:00.0 enp21s0: renamed from eth0
[ 8010.619354] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready
[ 8010.663999] igb 0000:15:00.0 enp21s0: PCIe link lost, device now
detached
[ 8011.012427] IPv6: ADDRCONF(NETDEV_UP): enp21s0: link is not ready

The device is still not functional at this point, but I believe that's a
pci hotplug issue, not an igb issue.

CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC Alexander Duyck <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org

---
note: this patch is dependent on the previously submitted "igb: don't unmap hw_addr
if its NULL", which added io_addr.

Quoted 7 lines

v3: don't modify igb_rd32, just pass reg + ((offset) <<2) as suggested by
Alexander Duyck.

 drivers/net/ethernet/intel/igb/e1000_regs.h | 3 +--
 drivers/net/ethernet/intel/igb/igb_main.c   | 8 ++++----
 2 files changed, 5 insertions(+), 6 deletions(-)

Does not apply cleanly to my next-queue tree (dev-queue branch).
Please rebase to my dev-queue branch and re-submit.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20151012/54ff9960/attachment.asc>

#12 lore raw

PATCH v2 1/1 igb: improve handling of disconnected adapters

Clean up array_rd32 so that it uses igb_rd32 the same as rd32, per the suggestion of Alexander Duyck, and use io_addr in more places, so that we don't have the need to call E1000_REMOVED (which simply looks for a null hw_addr) nearly as much.

CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC Alexander Duyck <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org

---
Note: this patch is rebased on Jeff's next-queue/dev-queue branch, which already had an earlier revision of this applied, so I've essentially reverted a2675ab and applied the revised version of this, squashed them together, and here is the end result, which matches the version Alex acked.

 drivers/net/ethernet/intel/igb/e1000_regs.h |  3 +--
 drivers/net/ethernet/intel/igb/igb_main.c   | 15 ++-------------
 2 files changed, 3 insertions(+), 15 deletions(-)
drivers/net/ethernet/intel/igb/e1000_regs.h +1 -2
index 0fdcd4d..21d9d02 100644
@@ -386,8 +386,7 @@ do { \
 #define array_wr32(reg, offset, value) \
 	wr32((reg) + ((offset) << 2), (value))
 
-#define array_rd32(reg, offset) \
-	(readl(hw->hw_addr + reg + ((offset) << 2)))
+#define array_rd32(reg, offset) (igb_rd32(hw, reg + ((offset) << 2)))
 
 /* DMA Coalescing registers */
 #define E1000_PCIEMISC	0x05BB8 /* PCIE misc config register */
drivers/net/ethernet/intel/igb/igb_main.c +2 -13
index 044a23e..68006a5 100644
@@ -954,17 +954,12 @@ static int igb_request_msix(struct igb_adapter *adapter)
 	if (err)
 		goto err_out;
 
-	if (E1000_REMOVED(hw->hw_addr)) {
-		err = -EIO;
-		goto err_free;
-	}
-
 	for (i = 0; i < adapter->num_q_vectors; i++) {
 		struct igb_q_vector *q_vector = adapter->q_vector[i];
 
 		vector++;
 
-		q_vector->itr_register = hw->hw_addr + E1000_EITR(vector);
+		q_vector->itr_register = adapter->io_addr + E1000_EITR(vector);
 
 		if (q_vector->rx.ring && q_vector->tx.ring)
 			sprintf(q_vector->name, "%s-TxRx-%u", netdev->name,
@@ -1206,9 +1201,6 @@ static int igb_alloc_q_vector(struct igb_adapter *adapter,
 	if (txr_count > 1 || rxr_count > 1)
 		return -ENOMEM;
 
-	if (E1000_REMOVED(adapter->hw.hw_addr))
-		return -EIO;
-
 	ring_count = txr_count + rxr_count;
 	size = sizeof(struct igb_q_vector) +
 	       (sizeof(struct igb_ring) * ring_count);
@@ -1238,7 +1230,7 @@ static int igb_alloc_q_vector(struct igb_adapter *adapter,
 	q_vector->tx.work_limit = adapter->tx_work_limit;
 
 	/* initialize ITR configuration */
-	q_vector->itr_register = adapter->hw.hw_addr + E1000_EITR(0);
+	q_vector->itr_register = adapter->io_addr + E1000_EITR(0);
 	q_vector->itr_val = IGB_START_ITR;
 
 	/* initialize pointer to rings */
@@ -3281,9 +3273,6 @@ void igb_configure_tx_ring(struct igb_adapter *adapter,
 	u64 tdba = ring->dma;
 	int reg_idx = ring->reg_idx;
 
-	if (E1000_REMOVED(adapter->io_addr))
-		return;
-
 	/* disable the queue */
 	wr32(E1000_TXDCTL(reg_idx), 0);
 	wrfl();
Signature · 1 line
1.8.3.1
Tested-by on v2 1/1
#13 lore raw

RE: [Intel-wired-lan] [PATCH v2] igb: improve handling of disconnected adapters

Quoted 27 lines

From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jarod Wilson
Sent: Monday, October 19, 2015 8:52 AM
To: linux-kernel at vger.kernel.org
Cc: netdev at vger.kernel.org; Jarod Wilson <[email protected]>; intel-wired- lan at lists.osuosl.org
Subject: [Intel-wired-lan] [PATCH v2] igb: improve handling of disconnected adapters

Clean up array_rd32 so that it uses igb_rd32 the same as rd32, per the suggestion of Alexander Duyck, and use io_addr in more places, so that we don't have the need to call E1000_REMOVED (which simply looks for a null hw_addr) nearly as much.

CC Mark Rustad <[email protected]>
CC Jeff Kirsher <[email protected]>
CC Alexander Duyck <[email protected]>
CC intel-wired-lan at lists.osuosl.org
CC netdev at vger.kernel.org

---
Note: this patch is rebased on Jeff's next-queue/dev-queue branch, which already had an earlier revision of this applied, so I've essentially reverted a2675ab and applied the revised version of this, squashed them together, and here is the end result, which matches the version Alex acked.

Quoted 4 lines
 drivers/net/ethernet/intel/igb/e1000_regs.h |  3 +--
 drivers/net/ethernet/intel/igb/igb_main.c   | 15 ++-------------
 2 files changed, 3 insertions(+), 15 deletions(-)