bnxt_en: Two bug fixes.
Michael Chan
maintainercover bnxt_en: Two bug fixes.
The first patch fixes AER recovery by reducing the time from several minutes to a more reasonable 20 - 30 seconds. The second patch fixes a possible NULL pointer crash during firmware reset.
Please queue for -stable also. Thanks.
Vasundhara Volam (2):
bnxt_en: Avoid sending firmware messages when AER error is detected.
bnxt_en: Fix NULL ptr dereference crash in bnxt_fw_reset_task()
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 13 +++++++------ drivers/net/ethernet/broadcom/bnxt/bnxt.h | 4 ++++ 2 files changed, 11 insertions(+), 6 deletions(-)
Signature · 1 line
1.8.3.1
Michael Chan
maintainerPATCH 1/2 bnxt_en: Avoid sending firmware messages when AER error is detected.
From: Vasundhara Volam <[email protected]>
When the driver goes through PCIe AER reset in error state, all firmware messages will timeout because the PCIe bus is no longer accessible. This can lead to AER reset taking many minutes to complete as each firmware command takes time to timeout.
Define a new macro BNXT_NO_FW_ACCESS() to skip these firmware messages when either firmware is in fatal error state or when pci_channel_offline() is true. It now takes a more reasonable 20 to 30 seconds to complete AER recovery.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 6 +++--- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 4 ++++ 2 files changed, 7 insertions(+), 3 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +3 -3
@@ -4305,7 +4305,7 @@ static int bnxt_hwrm_do_send_msg(struct bnxt *bp, void *msg, u32 msg_len,u32 bar_offset =BNXT_GRCPF_REG_CHIMP_COMM ;u16 dst =BNXT_HWRM_CHNL_CHIMP ; -if (test_bit (BNXT_STATE_FW_FATAL_COND , &bp->state)) +if (BNXT_NO_FW_ACCESS (bp))return -EBUSY ;if (msg_len >BNXT_HWRM_MAX_REQ_LEN ) {
@@ -5723,7 +5723,7 @@ static int hwrm_ring_free_send_msg(struct bnxt *bp,struct hwrm_ring_free_output *resp = bp->hwrm_cmd_resp_addr;u16 error_code; -if (test_bit (BNXT_STATE_FW_FATAL_COND , &bp->state)) +if (BNXT_NO_FW_ACCESS (bp))return 0 ;bnxt_hwrm_cmd_hdr_init (bp, &req,HWRM_RING_FREE , cmpl_ring_id,-1 );
@@ -7817,7 +7817,7 @@ static int bnxt_set_tpa(struct bnxt *bp, bool set_tpa)if (set_tpa) tpa_flags = bp->flags &BNXT_FLAG_TPA ; -else if (test_bit (BNXT_STATE_FW_FATAL_COND , &bp->state)) +else if (BNXT_NO_FW_ACCESS (bp))return 0 ;for (i =0 ; i < bp->nr_vnics; i++) { rc =bnxt_hwrm_vnic_set_tpa (bp, i, tpa_flags);
drivers/net/ethernet/broadcom/bnxt/bnxt.h +4 -0
@@ -1737,6 +1737,10 @@ struct bnxt {#define BNXT_STATE_FW_FATAL_COND 6#define BNXT_STATE_DRV_REGISTERED 7 +#define BNXT_NO_FW_ACCESS (bp) \ + (test_bit(BNXT_STATE_FW_FATAL_COND, &(bp)->state) || \ + pci_channel_offline((bp)->pdev)) +struct bnxt_irq *irq_tbl;int total_irqs;u8 mac_addr[ETH_ALEN ];
Signature · 1 line
1.8.3.1
Michael Chan
maintainerPATCH 2/2 bnxt_en: Fix NULL ptr dereference crash in bnxt_fw_reset_task()
In reply to Michael ChanFrom: Vasundhara Volam <[email protected]>
bnxt_fw_reset_task() which runs from a workqueue can race with bnxt_remove_one(). For example, if firmware reset and VF FLR are happening at about the same time.
bnxt_remove_one() already cancels the workqueue and waits for it to finish, but we need to do this earlier before the devlink reporters are destroyed. This will guarantee that
the devlink reporters will always be valid when bnxt_fw_reset_task() is still running.
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +4 -3
@@ -11779,6 +11779,10 @@ static void bnxt_remove_one(struct pci_dev *pdev)if (BNXT_PF (bp))bnxt_sriov_disable (bp); +clear_bit (BNXT_STATE_IN_FW_RESET , &bp->state); +bnxt_cancel_sp_work (bp); + bp->sp_event =0 ; +bnxt_dl_fw_reporters_destroy (bp, true);if (BNXT_PF (bp))devlink_port_type_clear (&bp->dl_port);
@@ -11786,9 +11790,6 @@ static void bnxt_remove_one(struct pci_dev *pdev)unregister_netdev (dev);bnxt_dl_unregister (bp);bnxt_shutdown_tc (bp); -clear_bit (BNXT_STATE_IN_FW_RESET , &bp->state); -bnxt_cancel_sp_work (bp); - bp->sp_event =0 ;bnxt_clear_int_mode (bp);bnxt_hwrm_func_drv_unrgtr (bp);
Signature · 1 line
1.8.3.1
Jakub Kicinski
maintainerRe: [PATCH net 2/2] bnxt_en: Fix NULL ptr dereference crash in bnxt_fw_reset_task()
Quoted 36 lines · On Sat, 5 Sep 2020 22:55:37 -0400 Michael Chan wrote:
From: Vasundhara Volam <[email protected]>
bnxt_fw_reset_task()which runs from a workqueue can race withbnxt_remove_one(). For example, if firmware reset and VF FLR are happening at about the same time.
bnxt_remove_one()already cancels the workqueue and waits for it to finish, but we need to do this earlier before the devlink reporters are destroyed. This will guarantee that
the devlink reporters will always be valid whenbnxt_fw_reset_task()is still running.Fixes b148bb238c02 ("bnxt_en: Fix possible crash in bnxt_fw_reset_task().")Reviewed-by Edwin Peer <[email protected]>Signed-off-by Vasundhara Volam <[email protected]>Signed-off-by Michael Chan <[email protected]>--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)drivers/net/ethernet/broadcom/bnxt/bnxt.c +4
-0@@ -11779,6 +11779,10 @@ static void bnxt_remove_one(struct pci_dev *pdev)if (BNXT_PF (bp))bnxt_sriov_disable (bp); +clear_bit (BNXT_STATE_IN_FW_RESET , &bp->state); +bnxt_cancel_sp_work (bp); + bp->sp_event =0 ; +bnxt_dl_fw_reporters_destroy (bp, true);if (BNXT_PF (bp))devlink_port_type_clear (&bp->dl_port);
drivers/net/ethernet/broadcom/bnxt/bnxt.c static void bnxt_remove_one(struct pci_dev *pdev) · line 11795
Quoted 9 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0
-3@@ -11786,9 +11790,6 @@ static void bnxt_remove_one(struct pci_dev *pdev)unregister_netdev (dev);bnxt_dl_unregister (bp);bnxt_shutdown_tc (bp); -clear_bit (BNXT_STATE_IN_FW_RESET , &bp->state); -bnxt_cancel_sp_work (bp); - bp->sp_event =0 ;bnxt_clear_int_mode (bp);bnxt_hwrm_func_drv_unrgtr (bp);
devlink can itself scheduler a recovery via:
bnxt_fw_fatal_recover() -> bnxt_fw_reset()
no? Maybe don't make the devlink recovery path need to go via a workqueue?
Michael Chan
maintainerRe: [PATCH net 2/2] bnxt_en: Fix NULL ptr dereference crash in bnxt_fw_reset_task()
Quoted 5 lines · Jakub Kicinski
devlink can itself scheduler a recovery via:
bnxt_fw_fatal_recover()->bnxt_fw_reset()
Yes, this is how it is initiated when we call devlink_health_report() to report the error condition. From bnxt_fw_reset(), we use a workqueue because we have to go through many states, requiring sleeping/polling to transition through the states.
Quoted 2 lines
no? Maybe don't make the devlink recovery path need to go via a workqueue?
Current implementation is going through a work queue.
Jakub Kicinski
maintainerRe: [PATCH net 2/2] bnxt_en: Fix NULL ptr dereference crash in bnxt_fw_reset_task()
Quoted 12 lines · On Sun, 6 Sep 2020 15:07:02 -0700 Michael Chan wrote:
Quoted 5 lines · Jakub Kicinski
devlink can itself scheduler a recovery via:
bnxt_fw_fatal_recover()->bnxt_fw_reset()Yes, this is how it is initiated when we call
devlink_health_report()to report the error condition. Frombnxt_fw_reset(), we use a workqueue because we have to go through many states, requiring sleeping/polling to transition through the states.
Quoted 4 lines
Quoted 2 lines
no? Maybe don't make the devlink recovery path need to go via a workqueue?
Current implementation is going through a work queue.
What I'm saying is the code looks like this after this patch:
+ clear_bit(BNXT_STATE_IN_FW_RESET, &bp->state);
+ bnxt_cancel_sp_work(bp);
+ bp->sp_event = 0;
+
bnxt_dl_fw_reporters_destroy(bp, true);
It cancels the work, _then_ destroys the reporter. But I think the reported can be used to schedule a recovery from command line. So the work may get re-scheduled after it has been canceled.
devlink_nl_cmd_health_reporter_recover_doit() -> bnxt_fw_fatal_recover() ->
bnxt_fw_reset() -> bnxt_queue_fw_reset_work()
What am I missing?
Michael Chan
maintainerRe: [PATCH net 2/2] bnxt_en: Fix NULL ptr dereference crash in bnxt_fw_reset_task()
Quoted 27 lines · Jakub Kicinski
Quoted 12 lines · On Sun, 6 Sep 2020 15:07:02 -0700 Michael Chan wrote:
Quoted 5 lines · Jakub Kicinski
devlink can itself scheduler a recovery via:
bnxt_fw_fatal_recover()->bnxt_fw_reset()Yes, this is how it is initiated when we call
devlink_health_report()to report the error condition. Frombnxt_fw_reset(), we use a workqueue because we have to go through many states, requiring sleeping/polling to transition through the states.Quoted 4 lines
Quoted 2 lines
no? Maybe don't make the devlink recovery path need to go via a workqueue?
Current implementation is going through a work queue.
What I'm saying is the code looks like this after this patch:
drivers/net/ethernet/broadcom/bnxt/bnxt.c +4
-0+clear_bit (BNXT_STATE_IN_FW_RESET , &bp->state); +bnxt_cancel_sp_work (bp); + bp->sp_event =0 ; +bnxt_dl_fw_reporters_destroy (bp, true);
Quoted 3 lines
It cancels the work, _then_ destroys the reporter. But I think the reported can be used to schedule a recovery from command line. So the work may get re-scheduled after it has been canceled.
bnxt_en does not support recovery from the command line. We return -EOPNOTSUPP when it comes from the command line.
Recovery has to be triggered from a firmware reported error or a driver detected error.
Jakub Kicinski
maintainerRe: [PATCH net 2/2] bnxt_en: Fix NULL ptr dereference crash in bnxt_fw_reset_task()
Quoted 32 lines · On Sun, 6 Sep 2020 20:48:04 -0700 Michael Chan wrote:
Quoted 27 lines · Jakub Kicinski
Quoted 12 lines · On Sun, 6 Sep 2020 15:07:02 -0700 Michael Chan wrote:
Quoted 5 lines · Jakub Kicinski
devlink can itself scheduler a recovery via:
bnxt_fw_fatal_recover()->bnxt_fw_reset()Yes, this is how it is initiated when we call
devlink_health_report()to report the error condition. Frombnxt_fw_reset(), we use a workqueue because we have to go through many states, requiring sleeping/polling to transition through the states.Quoted 4 lines
Quoted 2 lines
no? Maybe don't make the devlink recovery path need to go via a workqueue?
Current implementation is going through a work queue.
What I'm saying is the code looks like this after this patch:
drivers/net/ethernet/broadcom/bnxt/bnxt.c +4
-0+clear_bit (BNXT_STATE_IN_FW_RESET , &bp->state); +bnxt_cancel_sp_work (bp); + bp->sp_event =0 ; +bnxt_dl_fw_reporters_destroy (bp, true);Quoted 3 lines
It cancels the work, _then_ destroys the reporter. But I think the reported can be used to schedule a recovery from command line. So the work may get re-scheduled after it has been canceled.
Quoted 5 lines
bnxt_endoes not support recovery from the command line. We return-EOPNOTSUPPwhen it comes from the command line.Recovery has to be triggered from a firmware reported error or a driver detected error.
I see it now, thanks.
Jakub Kicinski
maintainerQuoted 5 lines · On Sat, 5 Sep 2020 22:55:35 -0400 Michael Chan wrote:
The first patch fixes AER recovery by reducing the time from several minutes to a more reasonable 20 - 30 seconds. The second patch fixes a possible NULL pointer crash during firmware reset.
Please queue for -stable also. Thanks.
Applied & queued.