fm10k: prevent null pointer dereference of msix_entries table
Subsystems (2)
PATCH 1/6 fm10k: prevent null pointer dereference of msix_entries table
According to the C standard dereferencing a variable before it is checked invokes undefined behavior, and thus compilers are free to assume the check for NULL isn't necessary. Prevent this by re-ordering the NULL check of msix_entries in fm10k_free_mbx_irq.
--- drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +3 -1
@@ -1143,14 +1143,16 @@ static irqreturn_t fm10k_msix_mbx_pf(int __always_unused irq, void *data)void fm10k_mbx_free_irq (struct fm10k_intfc *interface) { -struct msix_entry *entry = &interface->msix_entries[FM10K_MBX_VECTOR ];struct fm10k_hw *hw = &interface->hw; +struct msix_entry *entry;int itr_reg;/* no mailbox IRQ to free if MSI-X is not enabled */ if (!interface->msix_entries)return ; + entry = &interface->msix_entries[FM10K_MBX_VECTOR ]; +/* disconnect the mailbox */ hw->mbx.ops.disconnect (hw, &hw->mbx);
Signature · 1 line
2.7.0.236.gda096a0.dirty
PATCH 2/6 fm10k: don't initialize service task until later in probe
delay initialization of the service timer and service task until late probe. If we don't wait, failures in probe do not properly cleanup the service timer or service task items, which results in the kernel panic below, potentially freezing the whole system. In addition, ensure that the SERVICE_DISABLE bit is set before we request the mbx irq since the mbx interrupt attempts to schedule the service task otherwise. This prevents a similar trace from occurring after this change.
We didn't notice this issue before because probe almost always completes successfully. I discovered it due to a mis-ordered mailbox handler array, which resulted in the following failure when requesting mailbox interrupt.
[ 555.325619] ------------[ cut here ]------------ [ 555.325628] WARNING: CPU: 0 PID: 4941 at lib/list_debug.c:33 __list_add+0xa0/0xd0() [ 555.325631] list_add corruption. prev->next should be next (ffffffff81f46648), but was (null). (prev=ffff8807fad5d0e8). <snip> [ 555.325722] CPU: 0 PID: 4941 Comm: insmod Tainted: G OE 4.0.4-303.fc22.x86_64 #1 [ 555.325725] Hardware name: Intel Corporation S2600CO/S2600CO, BIOS SE5C600.86B.02.03.8x23.060520140825 06/05/2014 [ 555.325727] 0000000000000000 00000000b4f161b3 ffff88081a21f8e8 ffffffff81783124 [ 555.325734] 0000000000000000 ffff88081a21f940 ffff88081a21f928 ffffffff8109c66a [ 555.325740] 0000000064000000 ffff8807fad5d0e8 ffff8807fad5d0e8 ffffffff81f46648 [ 555.325746] Call Trace: [ 555.325752] [<ffffffff81783124>] dump_stack+0x45/0x57 [ 555.325757] [<ffffffff8109c66a>] warn_slowpath_common+0x8a/0xc0 [ 555.325759] [<ffffffff8109c6f5>] warn_slowpath_fmt+0x55/0x70 [ 555.325763] [<ffffffff813ba270>] __list_add+0xa0/0xd0 [ 555.325768] [<ffffffff81102d1d>] __internal_add_timer+0x9d/0x110 [ 555.325771] [<ffffffff81102dbf>] internal_add_timer+0x2f/0xc0 [ 555.325774] [<ffffffff81104e5a>] mod_timer+0x12a/0x230 [ 555.325782] [<ffffffffa03d54ca>] fm10k_probe+0x69a/0xc80 [fm10k] [ 555.325787] [<ffffffff813e8355>] local_pci_probe+0x45/0xa0 [ 555.325791] [<ffffffff8129cf42>] ? sysfs_do_create_link_sd.isra.2+0x72/0xc0 [ 555.325794] [<ffffffff813e96b9>] pci_device_probe+0xf9/0x150 [ 555.325799] [<ffffffff814d7e73>] driver_probe_device+0xa3/0x400 [ 555.325802] [<ffffffff814d82ab>] __driver_attach+0x9b/0xa0 [ 555.325805] [<ffffffff814d8210>] ? __device_attach+0x40/0x40 [ 555.325808] [<ffffffff814d5bd3>] bus_for_each_dev+0x73/0xc0 [ 555.325811] [<ffffffff814d78ce>] driver_attach+0x1e/0x20 [ 555.325815] [<ffffffff814d7480>] bus_add_driver+0x180/0x250 [ 555.325819] [<ffffffffa03b2000>] ? 0xffffffffa03b2000 [ 555.325823] [<ffffffff814d8aa4>] driver_register+0x64/0xf0 [ 555.325826] [<ffffffff813e7bec>] __pci_register_driver+0x4c/0x50 [ 555.325832] [<ffffffffa03d6ca3>] fm10k_register_pci_driver+0x23/0x30 [fm10k] [ 555.325838] [<ffffffffa03b2080>] fm10k_init_module+0x80/0x1000 [fm10k] [ 555.325843] [<ffffffff81002128>] do_one_initcall+0xb8/0x200 [ 555.325848] [<ffffffff811e10d2>] ? __vunmap+0xa2/0x100 [ 555.325852] [<ffffffff811fe239>] ? kmem_cache_alloc_trace+0x1b9/0x240 [ 555.325855] [<ffffffff8178230e>] ? do_init_module+0x28/0x1cb [ 555.325858] [<ffffffff81782346>] do_init_module+0x60/0x1cb [ 555.325862] [<ffffffff8112168e>] load_module+0x205e/0x26b0 [ 555.325866] [<ffffffff8111d110>] ? store_uevent+0x70/0x70 [ 555.325870] [<ffffffff812234b0>] ? kernel_read+0x50/0x80 [ 555.325873] [<ffffffff81121f3e>] SyS_finit_module+0xbe/0xf0 [ 555.325878] [<ffffffff81789749>] system_call_fastpath+0x12/0x17 [ 555.325880] ---[ end trace 9e0f58d071eafd2a ]---
--- drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +16 -9
@@ -1795,15 +1795,6 @@ static int fm10k_sw_init(struct fm10k_intfc *interface,/* initialize DCBNL interface */ fm10k_dcbnl_set_ops (netdev); -/* Initialize service timer and service task */ -set_bit (__FM10K_SERVICE_DISABLE, &interface->state); -setup_timer (&interface->service_timer, &fm10k_service_timer, - (unsigned long )interface); -INIT_WORK (&interface->service_task, fm10k_service_task); - -/* kick off service timer now, even when interface is down */ -mod_timer (&interface->service_timer, (HZ *2 ) + jiffies); -/* Intitialize timestamp data */ fm10k_ts_init (interface);
@@ -1989,6 +1980,12 @@ static int fm10k_probe(struct pci_dev *pdev, const struct pci_device_id *ent)if (err) goto err_sw_init; +/* the mbx interrupt might attempt to schedule the service task, so we +* must ensure it is disabled since we haven't yet requested the timer +* or work item. +*/ +set_bit (__FM10K_SERVICE_DISABLE, &interface->state); + err =fm10k_mbx_request_irq (interface);if (err) goto err_mbx_interrupt;
@@ -2008,6 +2005,16 @@ static int fm10k_probe(struct pci_dev *pdev, const struct pci_device_id *ent)/* stop all the transmit queues from transmitting until link is up */ netif_tx_stop_all_queues (netdev); +/* Initialize service timer and service task late in order to avoid +* cleanup issues. +*/ +setup_timer (&interface->service_timer, &fm10k_service_timer, + (unsigned long )interface); +INIT_WORK (&interface->service_task, fm10k_service_task); + +/* kick off service timer now, even when interface is down */ +mod_timer (&interface->service_timer, (HZ *2 ) + jiffies); +/* Register PTP interface */ fm10k_ptp_register (interface);
Signature · 1 line
2.7.0.236.gda096a0.dirty
Singh
PATCH 2/6 fm10k: don't initialize service task until later in probe
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 2/6] fm10k: don't initialize service task until later in probe
delay initialization of the service timer and service task until late probe. If we don't wait, failures in probe do not properly cleanup the service timer or service task items, which results in the kernel panic below, potentially freezing the whole system. In addition, ensure that the SERVICE_DISABLE bit is set before we request the mbx irq since the mbx interrupt attempts to schedule the service task otherwise. This prevents a similar trace from occurring after this change.
We didn't notice this issue before because probe almost always completes successfully. I discovered it due to a mis-ordered mailbox handler array, which resulted in the following failure when requesting mailbox interrupt.
[ 555.325619] ------------[ cut here ]------------ [ 555.325628] WARNING: CPU: 0 PID: 4941 at lib/list_debug.c:33 __list_add+0xa0/0xd0() [ 555.325631] list_add corruption. prev->next should be next (ffffffff81f46648), but was (null). (prev=ffff8807fad5d0e8). <snip> [ 555.325722] CPU: 0 PID: 4941 Comm: insmod Tainted: G OE 4.0.4-303.fc22.x86_64 #1 [ 555.325725] Hardware name: Intel Corporation S2600CO/S2600CO, BIOS SE5C600.86B.02.03.8x23.060520140825 06/05/2014 [ 555.325727] 0000000000000000 00000000b4f161b3 ffff88081a21f8e8 ffffffff81783124 [ 555.325734] 0000000000000000 ffff88081a21f940 ffff88081a21f928 ffffffff8109c66a [ 555.325740] 0000000064000000 ffff8807fad5d0e8 ffff8807fad5d0e8 ffffffff81f46648 [ 555.325746] Call Trace: [ 555.325752] [<ffffffff81783124>] dump_stack+0x45/0x57 [ 555.325757] [<ffffffff8109c66a>] warn_slowpath_common+0x8a/0xc0 [ 555.325759] [<ffffffff8109c6f5>] warn_slowpath_fmt+0x55/0x70 [ 555.325763] [<ffffffff813ba270>] __list_add+0xa0/0xd0 [ 555.325768] [<ffffffff81102d1d>] __internal_add_timer+0x9d/0x110 [ 555.325771] [<ffffffff81102dbf>] internal_add_timer+0x2f/0xc0 [ 555.325774] [<ffffffff81104e5a>] mod_timer+0x12a/0x230 [ 555.325782] [<ffffffffa03d54ca>] fm10k_probe+0x69a/0xc80 [fm10k] [ 555.325787] [<ffffffff813e8355>] local_pci_probe+0x45/0xa0 [ 555.325791] [<ffffffff8129cf42>] ? sysfs_do_create_link_sd.isra.2+0x72/0xc0 [ 555.325794] [<ffffffff813e96b9>] pci_device_probe+0xf9/0x150 [ 555.325799] [<ffffffff814d7e73>] driver_probe_device+0xa3/0x400 [ 555.325802] [<ffffffff814d82ab>] __driver_attach+0x9b/0xa0 [ 555.325805] [<ffffffff814d8210>] ? __device_attach+0x40/0x40 [ 555.325808] [<ffffffff814d5bd3>] bus_for_each_dev+0x73/0xc0 [ 555.325811] [<ffffffff814d78ce>] driver_attach+0x1e/0x20 [ 555.325815] [<ffffffff814d7480>] bus_add_driver+0x180/0x250 [ 555.325819] [<ffffffffa03b2000>] ? 0xffffffffa03b2000 [ 555.325823] [<ffffffff814d8aa4>] driver_register+0x64/0xf0 [ 555.325826] [<ffffffff813e7bec>] __pci_register_driver+0x4c/0x50 [ 555.325832] [<ffffffffa03d6ca3>] fm10k_register_pci_driver+0x23/0x30 [fm10k] [ 555.325838] [<ffffffffa03b2080>] fm10k_init_module+0x80/0x1000 [fm10k] [ 555.325843] [<ffffffff81002128>] do_one_initcall+0xb8/0x200 [ 555.325848] [<ffffffff811e10d2>] ? __vunmap+0xa2/0x100 [ 555.325852] [<ffffffff811fe239>] ? kmem_cache_alloc_trace+0x1b9/0x240 [ 555.325855] [<ffffffff8178230e>] ? do_init_module+0x28/0x1cb [ 555.325858] [<ffffffff81782346>] do_init_module+0x60/0x1cb [ 555.325862] [<ffffffff8112168e>] load_module+0x205e/0x26b0 [ 555.325866] [<ffffffff8111d110>] ? store_uevent+0x70/0x70 [ 555.325870] [<ffffffff812234b0>] ? kernel_read+0x50/0x80 [ 555.325873] [<ffffffff81121f3e>] SyS_finit_module+0xbe/0xf0 [ 555.325878] [<ffffffff81789749>] system_call_fastpath+0x12/0x17 [ 555.325880] ---[ end trace 9e0f58d071eafd2a ]---
---
PATCH 3/6 fm10k: base queue scheme covered by RSS
In reply to Jacob KellerIn fm10k_set_num_queues, we previously assigned the base template. This would always be overwritten by either fm10k_set_qos_queues or fm10k_set_rss_queues. In either case, we don't need the base values, so we can just remove them.
Change-type: Customization
Complexity: Low
Customer-visible: No
Reviewed-on: https://git-amr-1.devtools.intel.com/gerrit/67458
--- drivers/net/ethernet/intel/fm10k/fm10k_main.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/ethernet/intel/fm10k/fm10k_main.c +2 -4
@@ -1572,13 +1572,11 @@ static bool fm10k_set_rss_queues(struct fm10k_intfc *interface)**/ static void fm10k_set_num_queues (struct fm10k_intfc *interface) { -/* Start with base case */ - interface->num_rx_queues =1 ; - interface->num_tx_queues =1 ; - +/* Attempt to setup QoS and RSS first */ if (fm10k_set_qos_queues (interface))return ; +/* If we don't have QoS, just fallback to only RSS. */ fm10k_set_rss_queues (interface); }
Signature · 1 line
2.7.0.236.gda096a0.dirty
PATCH 4/6 fm10k: print error message when stop_hw fails
In reply to Jacob Kellerfm10k_stop_hw_generic calls fm10k_disable_queues_generic, which may return an error code indicating that the queues were not stopped within the time limit. Notify the user by displaying a message in the kernel message ring, in a similar way to how we notify the user when reset_hw fails. There isn't much we can do to recover from this error, so currently nothing else is done.
--- drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +4 -1
@@ -1656,6 +1656,7 @@ void fm10k_down(struct fm10k_intfc *interface) {struct net_device *netdev = interface->netdev;struct fm10k_hw *hw = &interface->hw; +int err;/* signal that we are down to the interrupt handler and service task */ set_bit (__FM10K_DOWN, &interface->state);
@@ -1680,7 +1681,9 @@ void fm10k_down(struct fm10k_intfc *interface)fm10k_update_stats (interface);/* Disable DMA engine for Tx/Rx */ - hw->mac.ops.stop_hw (hw); + err = hw->mac.ops.stop_hw (hw); +if (err) +dev_err (&interface->pdev->dev,"stop_hw failed: %d\n" , err);/* free any buffers still on the rings */ fm10k_clean_all_tx_rings (interface);
Signature · 1 line
2.7.0.236.gda096a0.dirty
Singh
PATCH 4/6 fm10k: print error message when stop_hw fails
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 4/6] fm10k: print error message when stop_hw fails
fm10k_stop_hw_generic calls fm10k_disable_queues_generic, which may return an error code indicating that the queues were not stopped within the time limit. Notify the user by displaying a message in the kernel message ring, in a similar way to how we notify the user when reset_hw fails. There isn't much we can do to recover from this error, so currently nothing else is done.
---
PATCH 5/6 fm10k: free mbx irq before clearing interrupt scheme
In reply to Jacob KellerDuring fm10k_io_error_detected we were clearing the interrupt scheme before we freed the mbx IRQ. This causes a kernel panic because the mbx IRQ are assigned after MSI-X initialization. Clearing the interrupt scheme results in removing the MSI-X entry table. Fix this by freeing the mbx IRQ before we clear the interrupt scheme, as we do elsewhere in the driver.
--- drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +2 -2
@@ -2274,11 +2274,11 @@ static pci_ers_result_t fm10k_io_error_detected(struct pci_dev *pdev,if (netif_running (netdev))fm10k_close (netdev); +fm10k_mbx_free_irq (interface); +/* free interrupts */ fm10k_clear_queueing_scheme (interface); -fm10k_mbx_free_irq (interface); -pci_disable_device (pdev);/* Request a slot reset. */
Signature · 1 line
2.7.0.236.gda096a0.dirty
Singh
PATCH 5/6 fm10k: free mbx irq before clearing interrupt scheme
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 5/6] fm10k: free mbx irq before clearing interrupt scheme
During fm10k_io_error_detected we were clearing the interrupt scheme before we freed the mbx IRQ. This causes a kernel panic because the mbx IRQ are assigned after MSI-X initialization. Clearing the interrupt scheme results in removing the MSI-X entry table. Fix this by freeing the mbx IRQ before we clear the interrupt scheme, as we do elsewhere in the driver.
---
PATCH 6/6 fm10k: prevent RCU issues during AER events
In reply to Jacob KellerDuring an AER action response, we were calling fm10k_close without holding the rtnl_lock() which could lead to possible RCU warnings being produced due to 64bit stat updates among other causes.
--- drivers/net/ethernet/intel/fm10k/fm10k_pci.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/net/ethernet/intel/fm10k/fm10k_pci.c +4 -0
@@ -2271,6 +2271,8 @@ static pci_ers_result_t fm10k_io_error_detected(struct pci_dev *pdev,if (state == pci_channel_io_perm_failure)return PCI_ERS_RESULT_DISCONNECT ; +rtnl_lock (); +if (netif_running (netdev))fm10k_close (netdev);
@@ -2279,6 +2281,8 @@ static pci_ers_result_t fm10k_io_error_detected(struct pci_dev *pdev,/* free interrupts */ fm10k_clear_queueing_scheme (interface); +rtnl_unlock (); +pci_disable_device (pdev);/* Request a slot reset. */
Signature · 1 line
2.7.0.236.gda096a0.dirty
Singh
PATCH 6/6 fm10k: prevent RCU issues during AER events
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 6/6] fm10k: prevent RCU issues during AER events
During an AER action response, we were calling fm10k_close without holding the rtnl_lock() which could lead to possible RCU warnings being produced due to 64bit stat updates among other causes.
---
Singh
PATCH 1/6 fm10k: prevent null pointer dereference of msix_entries table
In reply to Jacob Keller-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jacob Keller
Sent: Thursday, February 4, 2016 10:48 AM
To: Intel Wired LAN <[email protected]>
Subject: [Intel-wired-lan] [PATCH 1/6] fm10k: prevent null pointer dereference of msix_entries table
According to the C standard dereferencing a variable before it is checked invokes undefined behavior, and thus compilers are free to assume the check for NULL isn't necessary. Prevent this by re-ordering the NULL check of msix_entries in fm10k_free_mbx_irq.
---