Skip to contentSkip to conversation

scsi: lpfc: Fix race conditions in ELS retry handling

PATCH Applied Kyle Mahlkuch 4 participants 4 messages Nov 11 – Nov 18

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 scsi: lpfc: Fix race conditions in ELS retry handling

This patch addresses a race condition in the lpfc driver's ELS retry event handling that can lead to a use-after-free in the lpfc_cancel_retry_delay_tmo(), where the NLP_DELAY_TMO flag is cleared before acquiring the lock to check if the retry event is queued. This creates a window where the timer can be rescheduled and fire, causing both the cancel path and the worker thread to release the same reference, resulting in a double-put and use-after-free.

Moved capture ndlp and clear evt_arg1 under hbalock while dequeuing LPFC_EVT_ELS_RETRY in lpfc_work_list_done().

---
Changes since v3:
- More whitespace cleanup. (Paul Ely)

Changes since v2:
- Capture ndlp and clear evt_arg1 under hbalock while dequeuing LPFC_EVT_ELS_RETRY in lpfc_work_list_done(). A requeue by lpfc_els_retry_delay() could cause a NULL dereference in the worker. (AI review)
- Whitespace and line-wrapping cleanup.

Changes since v1:
- Leave test_and_clear_bit op and timer_delete_sync at the beginning of lpfc_cancel_retry_delay_tmo() outside of lock. (Paul Ely)

---
 drivers/scsi/lpfc/lpfc_els.c     | 29 +++++++++++++++++++++++------
 drivers/scsi/lpfc/lpfc_hbadisc.c | 11 ++++++++++-
 2 files changed, 33 insertions(+), 7 deletions(-)
drivers/scsi/lpfc/lpfc_els.c +23 -6
index 6f6394a0047c..98d1824aefe1 100644
@@ -4477,18 +4477,35 @@ lpfc_issue_els_edc(struct lpfc_vport *vport, uint8_t retry)
 void
 lpfc_cancel_retry_delay_tmo(struct lpfc_vport *vport, struct lpfc_nodelist *nlp)
 {
-	struct lpfc_work_evt *evtp;
+	struct lpfc_hba *phba = vport->phba;
+	struct lpfc_work_evt *evtp = &nlp->els_retry_evt;
+	struct lpfc_nodelist *arg_ndlp = NULL;
+	unsigned long flags;
 
 	if (!test_and_clear_bit(NLP_DELAY_TMO, &nlp->nlp_flag))
 		return;
 	timer_delete_sync(&nlp->nlp_delayfunc);
 	nlp->nlp_last_elscmd = 0;
-	if (!list_empty(&nlp->els_retry_evt.evt_listp)) {
-		list_del_init(&nlp->els_retry_evt.evt_listp);
-		/* Decrement nlp reference count held for the delayed retry */
-		evtp = &nlp->els_retry_evt;
-		lpfc_nlp_put((struct lpfc_nodelist *)evtp->evt_arg1);
+
+	/*
+	 * If retry event is queued, remove it under lock and consume
+	 * its payload to prevent double-put. The event holds a reference
+	 * to the nodelist that must be released exactly once.
+	 */
+	spin_lock_irqsave(&phba->hbalock, flags);
+	if (!list_empty(&evtp->evt_listp)) {
+		list_del_init(&evtp->evt_listp);
+		/* Take local copy and clear to prevent reuse */
+		arg_ndlp = (struct lpfc_nodelist *)evtp->evt_arg1;
+		evtp->evt_arg1 = NULL;
+		smp_wmb(); /* Ensure NULL visible before unlock */
 	}
+	spin_unlock_irqrestore(&phba->hbalock, flags);
+
+	/* Drop the event-held reference outside the lock */
+	if (arg_ndlp)
+		lpfc_nlp_put(arg_ndlp);
+
 	if (test_and_clear_bit(NLP_NPR_2B_DISC, &nlp->nlp_flag)) {
 		if (vport->num_disc_nodes) {
 			if (vport->port_state < LPFC_VPORT_READY) {
drivers/scsi/lpfc/lpfc_hbadisc.c +10 -1
index 4c673dffa671..75457f19d4ea 100644
@@ -840,12 +840,21 @@ lpfc_work_list_done(struct lpfc_hba *phba)
 	while (!list_empty(&phba->work_list)) {
 		list_remove_head((&phba->work_list), evtp, typeof(*evtp),
 				 evt_listp);
+		/*
+		 * The ELS retry event can be requeued by lpfc_els_retry_delay()
+		 * as soon as hbalock is dropped. Consume the payload within the
+		 * hbalock to prevent reuse or double-put.
+		 */
+		if (evtp->evt == LPFC_EVT_ELS_RETRY) {
+			ndlp = (struct lpfc_nodelist *)(evtp->evt_arg1);
+			evtp->evt_arg1 = NULL;
+		}
 		spin_unlock_irq(&phba->hbalock);
 		hba_pci_err = test_bit(HBA_PCI_ERR, &phba->bit_flags);
 		free_evt = 1;
 		switch (evtp->evt) {
 		case LPFC_EVT_ELS_RETRY:
-			ndlp = (struct lpfc_nodelist *) (evtp->evt_arg1);
+			/* ndlp was already captured under hbalock */
 			if (!hba_pci_err) {
 				lpfc_els_retry_delay_handler(ndlp);
 				free_evt = 0; /* evt is part of ndlp */

--
2.55.0

#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] scsi: lpfc: missing Fixes trailer and stale race description in changelog --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] scsi: lpfc: missing Fixes trailer and stale race description in changelog
--

commit c4e00459dde635d0b5ecc529a001464e7b80a848
Author: Kyle Mahlkuch <[email protected]>

Quoted 10 lines

scsi: lpfc: Fix race conditions in ELS retry handling

This patch addresses a race condition in the lpfc driver's ELS retry event handling that can lead to a use-after-free in the lpfc_cancel_retry_delay_tmo(), where the NLP_DELAY_TMO flag is cleared before acquiring the lock to check if the retry event is queued. This creates a window where the timer can be rescheduled and fire, causing both the cancel path and the worker thread to release the same reference, resulting in a double-put and use-after-free.

Quoted 4 lines

Moved capture ndlp and clear evt_arg1 under hbalock while dequeuing LPFC_EVT_ELS_RETRY in lpfc_work_list_done().

[Severity: Low]
This fixes the unlocked els_retry_evt removal and lpfc_nlp_put() in lpfc_cancel_retry_delay_tmo(). Should the commit message include:

Fixes e47c9093531d ("[SCSI] lpfc 8.2.5 : Correct ndlp referencing issues")

Also, is the first paragraph of the commit message still accurate? The code still clears NLP_DELAY_TMO and calls timer_delete_sync() before taking phba->hbalock, and fixes the double-put race by serializing els_retry_evt dequeue and evt_arg1 consumption under phba->hbalock.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Reviewed-by on v4 1/1
#2 lore raw
In reply to Kyle Mahlkuch
Quoted 109 lines · Kyle Mahlkuch

This patch addresses a race condition in the lpfc driver's ELS retry event handling that can lead to a use-after-free in the lpfc_cancel_retry_delay_tmo(), where the NLP_DELAY_TMO flag is cleared before acquiring the lock to check if the retry event is queued. This creates a window where the timer can be rescheduled and fire, causing both the cancel path and the worker thread to release the same reference, resulting in a double-put and use-after-free.

Moved capture ndlp and clear evt_arg1 under hbalock while dequeuing LPFC_EVT_ELS_RETRY in lpfc_work_list_done().

---
Changes since v3:
- More whitespace cleanup. (Paul Ely)

Changes since v2:
- Capture ndlp and clear evt_arg1 under hbalock while dequeuing LPFC_EVT_ELS_RETRY in lpfc_work_list_done(). A requeue by lpfc_els_retry_delay() could cause a NULL dereference in the worker. (AI review)
- Whitespace and line-wrapping cleanup.

Changes since v1:
- Leave test_and_clear_bit op and timer_delete_sync at the beginning of lpfc_cancel_retry_delay_tmo() outside of lock. (Paul Ely)

---
  drivers/scsi/lpfc/lpfc_els.c     | 29 +++++++++++++++++++++++------
  drivers/scsi/lpfc/lpfc_hbadisc.c | 11 ++++++++++-
  2 files changed, 33 insertions(+), 7 deletions(-)
drivers/scsi/lpfc/lpfc_els.c +23 -6
index 6f6394a0047c..98d1824aefe1 100644
@@ -4477,18 +4477,35 @@ lpfc_issue_els_edc(struct lpfc_vport *vport, uint8_t retry)
  void
  lpfc_cancel_retry_delay_tmo(struct lpfc_vport *vport, struct lpfc_nodelist *nlp)
  {
-	struct lpfc_work_evt *evtp;
+	struct lpfc_hba *phba = vport->phba;
+	struct lpfc_work_evt *evtp = &nlp->els_retry_evt;
+	struct lpfc_nodelist *arg_ndlp = NULL;
+	unsigned long flags;
 
  	if (!test_and_clear_bit(NLP_DELAY_TMO, &nlp->nlp_flag))
  		return;
  	timer_delete_sync(&nlp->nlp_delayfunc);
  	nlp->nlp_last_elscmd = 0;
-	if (!list_empty(&nlp->els_retry_evt.evt_listp)) {
-		list_del_init(&nlp->els_retry_evt.evt_listp);
-		/* Decrement nlp reference count held for the delayed retry */
-		evtp = &nlp->els_retry_evt;
-		lpfc_nlp_put((struct lpfc_nodelist *)evtp->evt_arg1);
+
+	/*
+	 * If retry event is queued, remove it under lock and consume
+	 * its payload to prevent double-put. The event holds a reference
+	 * to the nodelist that must be released exactly once.
+	 */
+	spin_lock_irqsave(&phba->hbalock, flags);
+	if (!list_empty(&evtp->evt_listp)) {
+		list_del_init(&evtp->evt_listp);
+		/* Take local copy and clear to prevent reuse */
+		arg_ndlp = (struct lpfc_nodelist *)evtp->evt_arg1;
+		evtp->evt_arg1 = NULL;
+		smp_wmb(); /* Ensure NULL visible before unlock */
  	}
+	spin_unlock_irqrestore(&phba->hbalock, flags);
+
+	/* Drop the event-held reference outside the lock */
+	if (arg_ndlp)
+		lpfc_nlp_put(arg_ndlp);
+
  	if (test_and_clear_bit(NLP_NPR_2B_DISC, &nlp->nlp_flag)) {
  		if (vport->num_disc_nodes) {
  			if (vport->port_state < LPFC_VPORT_READY) {
drivers/scsi/lpfc/lpfc_hbadisc.c +10 -2
index 4c673dffa671..75457f19d4ea 100644
@@ -840,12 +840,21 @@ lpfc_work_list_done(struct lpfc_hba *phba)
  	while (!list_empty(&phba->work_list)) {
  		list_remove_head((&phba->work_list), evtp, typeof(*evtp),
  				 evt_listp);
+		/*
+		 * The ELS retry event can be requeued by lpfc_els_retry_delay()
+		 * as soon as hbalock is dropped. Consume the payload within the
+		 * hbalock to prevent reuse or double-put.
+		 */
+		if (evtp->evt == LPFC_EVT_ELS_RETRY) {
+			ndlp = (struct lpfc_nodelist *)(evtp->evt_arg1);
+			evtp->evt_arg1 = NULL;
+		}
  		spin_unlock_irq(&phba->hbalock);
  		hba_pci_err = test_bit(HBA_PCI_ERR, &phba->bit_flags);
  		free_evt = 1;
  		switch (evtp->evt) {
  		case LPFC_EVT_ELS_RETRY:
-			ndlp = (struct lpfc_nodelist *) (evtp->evt_arg1);
+			/* ndlp was already captured under hbalock */
  			if (!hba_pci_err) {
  				lpfc_els_retry_delay_handler(ndlp);
  				free_evt = 0; /* evt is part of ndlp */
--
Quoted 1 line

2.55.0

I thought I already did so, but here it is again:

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
maintainer Applied
#3 lore raw
In reply to Kyle Mahlkuch
Quoted 9 lines · Kyle Mahlkuch

This patch addresses a race condition in the lpfc driver's ELS retry event handling that can lead to a use-after-free in the lpfc_cancel_retry_delay_tmo(), where the NLP_DELAY_TMO flag is cleared before acquiring the lock to check if the retry event is queued. This creates a window where the timer can be rescheduled and fire, causing both the cancel path and the worker thread to release the same reference, resulting in a double-put and use-after-free.

[...]

Applied to 7.4/scsi-fixes, thanks!

[1/1] scsi: lpfc: Fix race conditions in ELS retry handling

      https://git.kernel.org/mkp/scsi/c/7470a1f2a84d
Signature · 1 line
Martin K. Petersen