scsi: lpfc: Fix race conditions in ELS retry handling
Subsystems (2)
PATCH v4 1/1 scsi: lpfc: Fix race conditions in ELS retry handling
This patch addresses a race condition in the lpfc driver's ELS retry event handling that can lead to a use-after-free in the lpfc_cancel_retry_delay_tmo(), where the NLP_DELAY_TMO flag is cleared before acquiring the lock to check if the retry event is queued. This creates a window where the timer can be rescheduled and fire, causing both the cancel path and the worker thread to release the same reference, resulting in a double-put and use-after-free.
Moved capture ndlp and clear evt_arg1 under hbalock while dequeuing LPFC_EVT_ELS_RETRY in lpfc_work_list_done().
---
Changes since v3:
- More whitespace cleanup. (Paul Ely)
Changes since v2:
- Capture ndlp and clear evt_arg1 under hbalock while dequeuing LPFC_EVT_ELS_RETRY in lpfc_work_list_done(). A requeue by lpfc_els_retry_delay() could cause a NULL dereference in the worker. (AI review)
- Whitespace and line-wrapping cleanup.
Changes since v1:
- Leave test_and_clear_bit op and timer_delete_sync at the beginning of lpfc_cancel_retry_delay_tmo() outside of lock. (Paul Ely)
--- drivers/scsi/lpfc/lpfc_els.c | 29 +++++++++++++++++++++++------ drivers/scsi/lpfc/lpfc_hbadisc.c | 11 ++++++++++- 2 files changed, 33 insertions(+), 7 deletions(-)
drivers/scsi/lpfc/lpfc_els.c +23 -6
@@ -4477,18 +4477,35 @@ lpfc_issue_els_edc(struct lpfc_vport *vport, uint8_t retry)void lpfc_cancel_retry_delay_tmo (struct lpfc_vport *vport,struct lpfc_nodelist *nlp) { -struct lpfc_work_evt *evtp; +struct lpfc_hba *phba = vport->phba; +struct lpfc_work_evt *evtp = &nlp->els_retry_evt; +struct lpfc_nodelist *arg_ndlp =NULL ; +unsigned long flags;if (!test_and_clear_bit (NLP_DELAY_TMO , &nlp->nlp_flag))return ;timer_delete_sync (&nlp->nlp_delayfunc); nlp->nlp_last_elscmd =0 ; -if (!list_empty (&nlp->els_retry_evt.evt_listp)) { -list_del_init (&nlp->els_retry_evt.evt_listp); -/* Decrement nlp reference count held for the delayed retry */ - evtp = &nlp->els_retry_evt; -lpfc_nlp_put ((struct lpfc_nodelist *)evtp->evt_arg1); + +/* +* If retry event is queued, remove it under lock and consume +* its payload to prevent double-put. The event holds a reference +* to the nodelist that must be released exactly once. +*/ +spin_lock_irqsave (&phba->hbalock, flags); +if (!list_empty (&evtp->evt_listp)) { +list_del_init (&evtp->evt_listp); +/* Take local copy and clear to prevent reuse */ + arg_ndlp = (struct lpfc_nodelist *)evtp->evt_arg1; + evtp->evt_arg1 =NULL ; +smp_wmb ();/* Ensure NULL visible before unlock */ } +spin_unlock_irqrestore (&phba->hbalock, flags); + +/* Drop the event-held reference outside the lock */ +if (arg_ndlp) +lpfc_nlp_put (arg_ndlp); +if (test_and_clear_bit (NLP_NPR_2B_DISC , &nlp->nlp_flag)) {if (vport->num_disc_nodes) {if (vport->port_state <LPFC_VPORT_READY ) {
drivers/scsi/lpfc/lpfc_hbadisc.c +10 -1
@@ -840,12 +840,21 @@ lpfc_work_list_done(struct lpfc_hba *phba)while (!list_empty (&phba->work_list)) {list_remove_head ((&phba->work_list), evtp,typeof (*evtp), evt_listp); +/* +* The ELS retry event can be requeued by lpfc_els_retry_delay() +* as soon as hbalock is dropped. Consume the payload within the +* hbalock to prevent reuse or double-put. +*/ +if (evtp->evt ==LPFC_EVT_ELS_RETRY ) { + ndlp = (struct lpfc_nodelist *)(evtp->evt_arg1); + evtp->evt_arg1 =NULL ; + }spin_unlock_irq (&phba->hbalock); hba_pci_err =test_bit (HBA_PCI_ERR , &phba->bit_flags); free_evt =1 ;switch (evtp->evt) {case LPFC_EVT_ELS_RETRY : - ndlp = (struct lpfc_nodelist *) (evtp->evt_arg1); +/* ndlp was already captured under hbalock */ if (!hba_pci_err) {lpfc_els_retry_delay_handler (ndlp); free_evt =0 ;/* evt is part of ndlp */
--
2.55.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] scsi: lpfc: missing Fixes trailer and stale race description in changelog --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] scsi: lpfc: missing Fixes trailer and stale race description in changelog
--
commit c4e00459dde635d0b5ecc529a001464e7b80a848
Author: Kyle Mahlkuch <[email protected]>
Quoted 10 lines
scsi: lpfc: Fix race conditions in ELS retry handling
This patch addresses a race condition in the lpfc driver's ELS retry event handling that can lead to a use-after-free in the
lpfc_cancel_retry_delay_tmo(), where theNLP_DELAY_TMOflag is cleared before acquiring the lock to check if the retry event is queued. This creates a window where the timer can be rescheduled and fire, causing both the cancel path and the worker thread to release the same reference, resulting in a double-put and use-after-free.
Quoted 4 lines
Moved capture ndlp and clear
evt_arg1under hbalock while dequeuingLPFC_EVT_ELS_RETRYinlpfc_work_list_done().Signed-off-by Kyle Mahlkuch <[email protected]>
[Severity: Low]
This fixes the unlocked els_retry_evt removal and lpfc_nlp_put() in lpfc_cancel_retry_delay_tmo(). Should the commit message include:
Also, is the first paragraph of the commit message still accurate? The code still clears NLP_DELAY_TMO and calls timer_delete_sync() before taking phba->hbalock, and fixes the double-put race by serializing els_retry_evt dequeue and evt_arg1 consumption under phba->hbalock.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 109 lines · Kyle Mahlkuch
This patch addresses a race condition in the lpfc driver's ELS retry event handling that can lead to a use-after-free in the
lpfc_cancel_retry_delay_tmo(), where theNLP_DELAY_TMOflag is cleared before acquiring the lock to check if the retry event is queued. This creates a window where the timer can be rescheduled and fire, causing both the cancel path and the worker thread to release the same reference, resulting in a double-put and use-after-free.Moved capture ndlp and clear
evt_arg1under hbalock while dequeuingLPFC_EVT_ELS_RETRYinlpfc_work_list_done().Signed-off-by Kyle Mahlkuch <[email protected]>---
Changes since v3:
- More whitespace cleanup. (Paul Ely)Changes since v2:
- Capture ndlp and clearevt_arg1under hbalock while dequeuingLPFC_EVT_ELS_RETRYinlpfc_work_list_done(). A requeue bylpfc_els_retry_delay()could cause a NULL dereference in the worker. (AI review)
- Whitespace and line-wrapping cleanup.Changes since v1:
- Leavetest_and_clear_bitop andtimer_delete_syncat the beginning oflpfc_cancel_retry_delay_tmo()outside of lock. (Paul Ely)--- drivers/scsi/lpfc/lpfc_els.c | 29 +++++++++++++++++++++++------ drivers/scsi/lpfc/lpfc_hbadisc.c | 11 ++++++++++- 2 files changed, 33 insertions(+), 7 deletions(-)drivers/scsi/lpfc/lpfc_els.c +23
-6@@ -4477,18 +4477,35 @@ lpfc_issue_els_edc(struct lpfc_vport *vport, uint8_t retry)void lpfc_cancel_retry_delay_tmo (struct lpfc_vport *vport,struct lpfc_nodelist *nlp) { -struct lpfc_work_evt *evtp; +struct lpfc_hba *phba = vport->phba; +struct lpfc_work_evt *evtp = &nlp->els_retry_evt; +struct lpfc_nodelist *arg_ndlp =NULL ; +unsigned long flags;if (!test_and_clear_bit (NLP_DELAY_TMO , &nlp->nlp_flag))return ;timer_delete_sync (&nlp->nlp_delayfunc); nlp->nlp_last_elscmd =0 ; -if (!list_empty (&nlp->els_retry_evt.evt_listp)) { -list_del_init (&nlp->els_retry_evt.evt_listp); -/* Decrement nlp reference count held for the delayed retry */ - evtp = &nlp->els_retry_evt; -lpfc_nlp_put ((struct lpfc_nodelist *)evtp->evt_arg1); + +/* +* If retry event is queued, remove it under lock and consume +* its payload to prevent double-put. The event holds a reference +* to the nodelist that must be released exactly once. +*/ +spin_lock_irqsave (&phba->hbalock, flags); +if (!list_empty (&evtp->evt_listp)) { +list_del_init (&evtp->evt_listp); +/* Take local copy and clear to prevent reuse */ + arg_ndlp = (struct lpfc_nodelist *)evtp->evt_arg1; + evtp->evt_arg1 =NULL ; +smp_wmb ();/* Ensure NULL visible before unlock */ } +spin_unlock_irqrestore (&phba->hbalock, flags); + +/* Drop the event-held reference outside the lock */ +if (arg_ndlp) +lpfc_nlp_put (arg_ndlp); +if (test_and_clear_bit (NLP_NPR_2B_DISC , &nlp->nlp_flag)) {if (vport->num_disc_nodes) {if (vport->port_state <LPFC_VPORT_READY ) {drivers/scsi/lpfc/lpfc_hbadisc.c +10
-2@@ -840,12 +840,21 @@ lpfc_work_list_done(struct lpfc_hba *phba)while (!list_empty (&phba->work_list)) {list_remove_head ((&phba->work_list), evtp,typeof (*evtp), evt_listp); +/* +* The ELS retry event can be requeued by lpfc_els_retry_delay() +* as soon as hbalock is dropped. Consume the payload within the +* hbalock to prevent reuse or double-put. +*/ +if (evtp->evt ==LPFC_EVT_ELS_RETRY ) { + ndlp = (struct lpfc_nodelist *)(evtp->evt_arg1); + evtp->evt_arg1 =NULL ; + }spin_unlock_irq (&phba->hbalock); hba_pci_err =test_bit (HBA_PCI_ERR , &phba->bit_flags); free_evt =1 ;switch (evtp->evt) {case LPFC_EVT_ELS_RETRY : - ndlp = (struct lpfc_nodelist *) (evtp->evt_arg1); +/* ndlp was already captured under hbalock */ if (!hba_pci_err) {lpfc_els_retry_delay_handler (ndlp); free_evt =0 ;/* evt is part of ndlp */ --
Quoted 1 line
2.55.0
I thought I already did so, but here it is again:
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Quoted 9 lines · Kyle Mahlkuch
This patch addresses a race condition in the lpfc driver's ELS retry event handling that can lead to a use-after-free in the
lpfc_cancel_retry_delay_tmo(), where theNLP_DELAY_TMOflag is cleared before acquiring the lock to check if the retry event is queued. This creates a window where the timer can be rescheduled and fire, causing both the cancel path and the worker thread to release the same reference, resulting in a double-put and use-after-free.[...]
Applied to 7.4/scsi-fixes, thanks!
[1/1] scsi: lpfc: Fix race conditions in ELS retry handling
https://git.kernel.org/mkp/scsi/c/7470a1f2a84d
Signature · 1 line
Martin K. Petersen