Skip to contentSkip to conversation

fm10k: prevent null pointer dereference of msix_entries table

PATCH New Jacob Keller 2 participants 11 messages Feb 4, 2016 – Feb 25, 2016

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/6 fm10k: prevent null pointer dereference of msix_entries table

According to the C standard dereferencing a variable before it is checked invokes undefined behavior, and thus compilers are free to assume the check for NULL isn't necessary. Prevent this by re-ordering the NULL check of msix_entries in fm10k_free_mbx_irq.

---
 drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +3 -1
index 6190a81b7c32..8c23fb3df572 100644
@@ -1143,14 +1143,16 @@ static irqreturn_t fm10k_msix_mbx_pf(int __always_unused irq, void *data)
 
 void fm10k_mbx_free_irq(struct fm10k_intfc *interface)
 {
-	struct msix_entry *entry = &interface->msix_entries[FM10K_MBX_VECTOR];
 	struct fm10k_hw *hw = &interface->hw;
+	struct msix_entry *entry;
 	int itr_reg;
 
 	/* no mailbox IRQ to free if MSI-X is not enabled */
 	if (!interface->msix_entries)
 		return;
 
+	entry = &interface->msix_entries[FM10K_MBX_VECTOR];
+
 	/* disconnect the mailbox */
 	hw->mbx.ops.disconnect(hw, &hw->mbx);
 
Signature · 1 line
2.7.0.236.gda096a0.dirty
#1 lore raw

PATCH 2/6 fm10k: don't initialize service task until later in probe

delay initialization of the service timer and service task until late probe. If we don't wait, failures in probe do not properly cleanup the service timer or service task items, which results in the kernel panic below, potentially freezing the whole system. In addition, ensure that the SERVICE_DISABLE bit is set before we request the mbx irq since the mbx interrupt attempts to schedule the service task otherwise. This prevents a similar trace from occurring after this change.

We didn't notice this issue before because probe almost always completes successfully. I discovered it due to a mis-ordered mailbox handler array, which resulted in the following failure when requesting mailbox interrupt.

[  555.325619] ------------[ cut here ]------------
[  555.325628] WARNING: CPU: 0 PID: 4941 at lib/list_debug.c:33 __list_add+0xa0/0xd0()
[  555.325631] list_add corruption. prev->next should be next (ffffffff81f46648), but was           (null). (prev=ffff8807fad5d0e8).
<snip>
[  555.325722] CPU: 0 PID: 4941 Comm: insmod Tainted: G           OE   4.0.4-303.fc22.x86_64 #1
[  555.325725] Hardware name: Intel Corporation S2600CO/S2600CO, BIOS SE5C600.86B.02.03.8x23.060520140825 06/05/2014
[  555.325727]  0000000000000000 00000000b4f161b3 ffff88081a21f8e8 ffffffff81783124
[  555.325734]  0000000000000000 ffff88081a21f940 ffff88081a21f928 ffffffff8109c66a
[  555.325740]  0000000064000000 ffff8807fad5d0e8 ffff8807fad5d0e8 ffffffff81f46648
[  555.325746] Call Trace:
[  555.325752]  [<ffffffff81783124>] dump_stack+0x45/0x57
[  555.325757]  [<ffffffff8109c66a>] warn_slowpath_common+0x8a/0xc0
[  555.325759]  [<ffffffff8109c6f5>] warn_slowpath_fmt+0x55/0x70
[  555.325763]  [<ffffffff813ba270>] __list_add+0xa0/0xd0
[  555.325768]  [<ffffffff81102d1d>] __internal_add_timer+0x9d/0x110
[  555.325771]  [<ffffffff81102dbf>] internal_add_timer+0x2f/0xc0
[  555.325774]  [<ffffffff81104e5a>] mod_timer+0x12a/0x230
[  555.325782]  [<ffffffffa03d54ca>] fm10k_probe+0x69a/0xc80 [fm10k]
[  555.325787]  [<ffffffff813e8355>] local_pci_probe+0x45/0xa0
[  555.325791]  [<ffffffff8129cf42>] ? sysfs_do_create_link_sd.isra.2+0x72/0xc0
[  555.325794]  [<ffffffff813e96b9>] pci_device_probe+0xf9/0x150
[  555.325799]  [<ffffffff814d7e73>] driver_probe_device+0xa3/0x400
[  555.325802]  [<ffffffff814d82ab>] __driver_attach+0x9b/0xa0
[  555.325805]  [<ffffffff814d8210>] ? __device_attach+0x40/0x40
[  555.325808]  [<ffffffff814d5bd3>] bus_for_each_dev+0x73/0xc0
[  555.325811]  [<ffffffff814d78ce>] driver_attach+0x1e/0x20
[  555.325815]  [<ffffffff814d7480>] bus_add_driver+0x180/0x250
[  555.325819]  [<ffffffffa03b2000>] ? 0xffffffffa03b2000
[  555.325823]  [<ffffffff814d8aa4>] driver_register+0x64/0xf0
[  555.325826]  [<ffffffff813e7bec>] __pci_register_driver+0x4c/0x50
[  555.325832]  [<ffffffffa03d6ca3>] fm10k_register_pci_driver+0x23/0x30 [fm10k]
[  555.325838]  [<ffffffffa03b2080>] fm10k_init_module+0x80/0x1000 [fm10k]
[  555.325843]  [<ffffffff81002128>] do_one_initcall+0xb8/0x200
[  555.325848]  [<ffffffff811e10d2>] ? __vunmap+0xa2/0x100
[  555.325852]  [<ffffffff811fe239>] ? kmem_cache_alloc_trace+0x1b9/0x240
[  555.325855]  [<ffffffff8178230e>] ? do_init_module+0x28/0x1cb
[  555.325858]  [<ffffffff81782346>] do_init_module+0x60/0x1cb
[  555.325862]  [<ffffffff8112168e>] load_module+0x205e/0x26b0
[  555.325866]  [<ffffffff8111d110>] ? store_uevent+0x70/0x70
[  555.325870]  [<ffffffff812234b0>] ? kernel_read+0x50/0x80
[  555.325873]  [<ffffffff81121f3e>] SyS_finit_module+0xbe/0xf0
[  555.325878]  [<ffffffff81789749>] system_call_fastpath+0x12/0x17
[  555.325880] ---[ end trace 9e0f58d071eafd2a ]---
---
 drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +16 -9
index 8c23fb3df572..ed1f8cf39508 100644
@@ -1795,15 +1795,6 @@ static int fm10k_sw_init(struct fm10k_intfc *interface,
 	/* initialize DCBNL interface */
 	fm10k_dcbnl_set_ops(netdev);
 
-	/* Initialize service timer and service task */
-	set_bit(__FM10K_SERVICE_DISABLE, &interface->state);
-	setup_timer(&interface->service_timer, &fm10k_service_timer,
-		    (unsigned long)interface);
-	INIT_WORK(&interface->service_task, fm10k_service_task);
-
-	/* kick off service timer now, even when interface is down */
-	mod_timer(&interface->service_timer, (HZ * 2) + jiffies);
-
 	/* Intitialize timestamp data */
 	fm10k_ts_init(interface);
 
@@ -1989,6 +1980,12 @@ static int fm10k_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	if (err)
 		goto err_sw_init;
 
+	/* the mbx interrupt might attempt to schedule the service task, so we
+	 * must ensure it is disabled since we haven't yet requested the timer
+	 * or work item.
+	 */
+	set_bit(__FM10K_SERVICE_DISABLE, &interface->state);
+
 	err = fm10k_mbx_request_irq(interface);
 	if (err)
 		goto err_mbx_interrupt;
@@ -2008,6 +2005,16 @@ static int fm10k_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	/* stop all the transmit queues from transmitting until link is up */
 	netif_tx_stop_all_queues(netdev);
 
+	/* Initialize service timer and service task late in order to avoid
+	 * cleanup issues.
+	 */
+	setup_timer(&interface->service_timer, &fm10k_service_timer,
+		    (unsigned long)interface);
+	INIT_WORK(&interface->service_task, fm10k_service_task);
+
+	/* kick off service timer now, even when interface is down */
+	mod_timer(&interface->service_timer, (HZ * 2) + jiffies);
+
 	/* Register PTP interface */
 	fm10k_ptp_register(interface);
 
Signature · 1 line
2.7.0.236.gda096a0.dirty
#2 lore raw

PATCH 2/6 fm10k: don't initialize service task until later in probe

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 2/6] fm10k: don't initialize service task until later in probe

delay initialization of the service timer and service task until late probe. If we don't wait, failures in probe do not properly cleanup the service timer or service task items, which results in the kernel panic below, potentially freezing the whole system. In addition, ensure that the SERVICE_DISABLE bit is set before we request the mbx irq since the mbx interrupt attempts to schedule the service task otherwise. This prevents a similar trace from occurring after this change.

We didn't notice this issue before because probe almost always completes successfully. I discovered it due to a mis-ordered mailbox handler array, which resulted in the following failure when requesting mailbox interrupt.

[  555.325619] ------------[ cut here ]------------ [  555.325628] WARNING: CPU: 0 PID: 4941 at lib/list_debug.c:33 __list_add+0xa0/0xd0()
[  555.325631] list_add corruption. prev->next should be next (ffffffff81f46648), but was           (null). (prev=ffff8807fad5d0e8).
<snip>
[  555.325722] CPU: 0 PID: 4941 Comm: insmod Tainted: G           OE   4.0.4-303.fc22.x86_64 #1
[  555.325725] Hardware name: Intel Corporation S2600CO/S2600CO, BIOS SE5C600.86B.02.03.8x23.060520140825 06/05/2014 [  555.325727]  0000000000000000 00000000b4f161b3 ffff88081a21f8e8 ffffffff81783124 [  555.325734]  0000000000000000 ffff88081a21f940 ffff88081a21f928 ffffffff8109c66a [  555.325740]  0000000064000000 ffff8807fad5d0e8 ffff8807fad5d0e8 ffffffff81f46648 [  555.325746] Call Trace:
[  555.325752]  [<ffffffff81783124>] dump_stack+0x45/0x57 [  555.325757]  [<ffffffff8109c66a>] warn_slowpath_common+0x8a/0xc0 [  555.325759]  [<ffffffff8109c6f5>] warn_slowpath_fmt+0x55/0x70 [  555.325763]  [<ffffffff813ba270>] __list_add+0xa0/0xd0 [  555.325768]  [<ffffffff81102d1d>] __internal_add_timer+0x9d/0x110 [  555.325771]  [<ffffffff81102dbf>] internal_add_timer+0x2f/0xc0 [  555.325774]  [<ffffffff81104e5a>] mod_timer+0x12a/0x230 [  555.325782]  [<ffffffffa03d54ca>] fm10k_probe+0x69a/0xc80 [fm10k] [  555.325787]  [<ffffffff813e8355>] local_pci_probe+0x45/0xa0 [  555.325791]  [<ffffffff8129cf42>] ? sysfs_do_create_link_sd.isra.2+0x72/0xc0
[  555.325794]  [<ffffffff813e96b9>] pci_device_probe+0xf9/0x150 [  555.325799]  [<ffffffff814d7e73>] driver_probe_device+0xa3/0x400 [  555.325802]  [<ffffffff814d82ab>] __driver_attach+0x9b/0xa0 [  555.325805]  [<ffffffff814d8210>] ? __device_attach+0x40/0x40 [  555.325808]  [<ffffffff814d5bd3>] bus_for_each_dev+0x73/0xc0 [  555.325811]  [<ffffffff814d78ce>] driver_attach+0x1e/0x20 [  555.325815]  [<ffffffff814d7480>] bus_add_driver+0x180/0x250 [  555.325819]  [<ffffffffa03b2000>] ? 0xffffffffa03b2000 [  555.325823]  [<ffffffff814d8aa4>] driver_register+0x64/0xf0 [  555.325826]  [<ffffffff813e7bec>] __pci_register_driver+0x4c/0x50 [  555.325832]  [<ffffffffa03d6ca3>] fm10k_register_pci_driver+0x23/0x30 [fm10k] [  555.325838]  [<ffffffffa03b2080>] fm10k_init_module+0x80/0x1000 [fm10k] [  555.325843]  [<ffffffff81002128>] do_one_initcall+0xb8/0x200 [  555.325848]  [<ffffffff811e10d2>] ? __vunmap+0xa2/0x100 [  555.325852]  [<ffffffff811fe239>] ? kmem_cache_alloc_trace+0x1b9/0x240
[  555.325855]  [<ffffffff8178230e>] ? do_init_module+0x28/0x1cb [  555.325858]  [<ffffffff81782346>] do_init_module+0x60/0x1cb [  555.325862]  [<ffffffff8112168e>] load_module+0x205e/0x26b0 [  555.325866]  [<ffffffff8111d110>] ? store_uevent+0x70/0x70 [  555.325870]  [<ffffffff812234b0>] ? kernel_read+0x50/0x80 [  555.325873]  [<ffffffff81121f3e>] SyS_finit_module+0xbe/0xf0 [  555.325878]  [<ffffffff81789749>] system_call_fastpath+0x12/0x17 [  555.325880] ---[ end trace 9e0f58d071eafd2a ]---

---

#3 lore raw

PATCH 3/6 fm10k: base queue scheme covered by RSS

In reply to Jacob Keller

In fm10k_set_num_queues, we previously assigned the base template. This would always be overwritten by either fm10k_set_qos_queues or fm10k_set_rss_queues. In either case, we don't need the base values, so we can just remove them.

Change-Id I8380f6a836e2409da438af258c7b252e39cf108b

Change-type: Customization
Complexity: Low
Customer-visible: No
Reviewed-on: https://git-amr-1.devtools.intel.com/gerrit/67458

---
 drivers/net/ethernet/intel/fm10k/fm10k_main.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/ethernet/intel/fm10k/fm10k_main.c +2 -4
index 134ce4daa994..bd4e6361ff92 100644
@@ -1572,13 +1572,11 @@ static bool fm10k_set_rss_queues(struct fm10k_intfc *interface)
  **/
 static void fm10k_set_num_queues(struct fm10k_intfc *interface)
 {
-	/* Start with base case */
-	interface->num_rx_queues = 1;
-	interface->num_tx_queues = 1;
-
+	/* Attempt to setup QoS and RSS first */
 	if (fm10k_set_qos_queues(interface))
 		return;
 
+	/* If we don't have QoS, just fallback to only RSS. */
 	fm10k_set_rss_queues(interface);
 }
 
Signature · 1 line
2.7.0.236.gda096a0.dirty
#4 lore raw

PATCH 4/6 fm10k: print error message when stop_hw fails

In reply to Jacob Keller

fm10k_stop_hw_generic calls fm10k_disable_queues_generic, which may return an error code indicating that the queues were not stopped within the time limit. Notify the user by displaying a message in the kernel message ring, in a similar way to how we notify the user when reset_hw fails. There isn't much we can do to recover from this error, so currently nothing else is done.

---
 drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +4 -1
index ed1f8cf39508..3c7c819ac8d9 100644
@@ -1656,6 +1656,7 @@ void fm10k_down(struct fm10k_intfc *interface)
 {
 	struct net_device *netdev = interface->netdev;
 	struct fm10k_hw *hw = &interface->hw;
+	int err;
 
 	/* signal that we are down to the interrupt handler and service task */
 	set_bit(__FM10K_DOWN, &interface->state);
@@ -1680,7 +1681,9 @@ void fm10k_down(struct fm10k_intfc *interface)
 	fm10k_update_stats(interface);
 
 	/* Disable DMA engine for Tx/Rx */
-	hw->mac.ops.stop_hw(hw);
+	err = hw->mac.ops.stop_hw(hw);
+	if (err)
+		dev_err(&interface->pdev->dev, "stop_hw failed: %d\n", err);
 
 	/* free any buffers still on the rings */
 	fm10k_clean_all_tx_rings(interface);
Signature · 1 line
2.7.0.236.gda096a0.dirty
#5 lore raw

PATCH 4/6 fm10k: print error message when stop_hw fails

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 4/6] fm10k: print error message when stop_hw fails

fm10k_stop_hw_generic calls fm10k_disable_queues_generic, which may return an error code indicating that the queues were not stopped within the time limit. Notify the user by displaying a message in the kernel message ring, in a similar way to how we notify the user when reset_hw fails. There isn't much we can do to recover from this error, so currently nothing else is done.

---

#6 lore raw

PATCH 5/6 fm10k: free mbx irq before clearing interrupt scheme

In reply to Jacob Keller

During fm10k_io_error_detected we were clearing the interrupt scheme before we freed the mbx IRQ. This causes a kernel panic because the mbx IRQ are assigned after MSI-X initialization. Clearing the interrupt scheme results in removing the MSI-X entry table. Fix this by freeing the mbx IRQ before we clear the interrupt scheme, as we do elsewhere in the driver.

---
 drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +2 -2
index 3c7c819ac8d9..da38af052519 100644
@@ -2274,11 +2274,11 @@ static pci_ers_result_t fm10k_io_error_detected(struct pci_dev *pdev,
 	if (netif_running(netdev))
 		fm10k_close(netdev);
 
+	fm10k_mbx_free_irq(interface);
+
 	/* free interrupts */
 	fm10k_clear_queueing_scheme(interface);
 
-	fm10k_mbx_free_irq(interface);
-
 	pci_disable_device(pdev);
 
 	/* Request a slot reset. */
Signature · 1 line
2.7.0.236.gda096a0.dirty
#7 lore raw

PATCH 5/6 fm10k: free mbx irq before clearing interrupt scheme

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 5/6] fm10k: free mbx irq before clearing interrupt scheme

During fm10k_io_error_detected we were clearing the interrupt scheme before we freed the mbx IRQ. This causes a kernel panic because the mbx IRQ are assigned after MSI-X initialization. Clearing the interrupt scheme results in removing the MSI-X entry table. Fix this by freeing the mbx IRQ before we clear the interrupt scheme, as we do elsewhere in the driver.

---

#8 lore raw

PATCH 6/6 fm10k: prevent RCU issues during AER events

In reply to Jacob Keller

During an AER action response, we were calling fm10k_close without holding the rtnl_lock() which could lead to possible RCU warnings being produced due to 64bit stat updates among other causes.

---
 drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +4 -0
index da38af052519..a10b98025c69 100644
@@ -2271,6 +2271,8 @@ static pci_ers_result_t fm10k_io_error_detected(struct pci_dev *pdev,
 	if (state == pci_channel_io_perm_failure)
 		return PCI_ERS_RESULT_DISCONNECT;
 
+	rtnl_lock();
+
 	if (netif_running(netdev))
 		fm10k_close(netdev);
 
@@ -2279,6 +2281,8 @@ static pci_ers_result_t fm10k_io_error_detected(struct pci_dev *pdev,
 	/* free interrupts */
 	fm10k_clear_queueing_scheme(interface);
 
+	rtnl_unlock();
+
 	pci_disable_device(pdev);
 
 	/* Request a slot reset. */
Signature · 1 line
2.7.0.236.gda096a0.dirty
#9 lore raw

PATCH 6/6 fm10k: prevent RCU issues during AER events

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 6/6] fm10k: prevent RCU issues during AER events

During an AER action response, we were calling fm10k_close without holding the rtnl_lock() which could lead to possible RCU warnings being produced due to 64bit stat updates among other causes.

---

#10 lore raw

PATCH 1/6 fm10k: prevent null pointer dereference of msix_entries table

In reply to Jacob Keller

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 1/6] fm10k: prevent null pointer dereference of msix_entries table

According to the C standard dereferencing a variable before it is checked invokes undefined behavior, and thus compilers are free to assume the check for NULL isn't necessary. Prevent this by re-ordering the NULL check of msix_entries in fm10k_free_mbx_irq.

---