Skip to contentSkip to conversation

scsi: mpi3mr: Fix start_time and error return in mpi3mr_bring_ioc_ready()

PATCH In review Ian Rogers 2 participants 2 messages Nov 19

Subsystems (2)
lore
Share & tools
lore

Ian Rogers

author
#0 lore raw

PATCH 1/1 scsi: mpi3mr: Fix start_time and error return in mpi3mr_bring_ioc_ready()

mpi3mr_bring_ioc_ready() has three bugs in its failure and retry handling:

1. start_time is only assigned after MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC when retry == 0, after three earlier 'goto out_failed' paths. If soft reset, reset-state polling, or admin queue setup fails on the first attempt, out_failed computes elapsed_time_sec from an uninitialized start_time, and if a retry occurs (retry == 1), start_time remains uninitialized for the ready-polling loop as well.

2. In 'elapsed_time_sec < (mrioc->ready_timeout - 60)', ready_timeout is u32, so values below 60 wrap around in 32-bit unsigned arithmetic before zero-extending to u64, spuriously retrying past the timeout.

3. After mpi3mr_setup_admin_qpair() sets retval = 0 (or on entry where retval is initialized to 0), controller fault/reset-history checks, ready-state timeouts, and early unrecoverable or device-not-present exits return retval == 0 (success) to mpi3mr_init_ioc() and mpi3mr_reinit_ioc() even though controller bringup failed.

Initialize start_time = jiffies at function entry for pre-enable failure exits, reset start_time = jiffies the first time MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC is written so pre-enable waits do not drain ready_timeout, rewrite the retry window check as 'elapsed_time_sec + 60 < mrioc->ready_timeout', and ensure retval = -1 on all failure exits.

Found via Clang -Wconditional-uninitialized on an x86_64 allyesconfig build. Verified by building drivers/scsi/mpi3mr/mpi3mr_fw.o with LLVM=1 W=1 (-Werror) and -Wconditional-uninitialized, and with a standalone C logic reproducer under ASan/UBSan; not runtime-tested on physical MPI3 hardware.

Fixes 59bd9cfe3fa0 ("scsi: mpi3mr: Code refactor of IOC init - part2")
Fixes 0a319f162949 ("scsi: mpi3mr: Wait for diagnostic save during controller init")
Fixes 9634bb07083c ("scsi: mpi3mr: Enhance the Enable Controller retry logic")
Fixes 4616a4b3cb8a ("scsi: mpi3mr: Improve wait logic while controller transitions to READY state")
Fixes fb6eb98f3965 ("scsi: mpi3mr: Handling of fault code for insufficient power")
---
 drivers/scsi/mpi3mr/mpi3mr_fw.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/scsi/mpi3mr/mpi3mr_fw.c +11 -3
index f0d3cd398dd0..b92b2467e995 100644
@@ -1582,8 +1582,9 @@ static int mpi3mr_bring_ioc_ready(struct mpi3mr_ioc *mrioc)
 	int retval = 0;
 	enum mpi3mr_iocstate ioc_state;
 	u64 base_info;
+	bool enable_started = false;
 	u8 retry = 0;
-	u64 start_time, elapsed_time_sec;
+	u64 start_time = jiffies, elapsed_time_sec;
 
 retry_bring_ioc_ready:
 
@@ -1595,6 +1596,7 @@ static int mpi3mr_bring_ioc_ready(struct mpi3mr_ioc *mrioc)
 
 	if (!mpi3mr_is_fault_recoverable(mrioc)) {
 		mrioc->unrecoverable = 1;
+		retval = -1;
 		goto out_device_not_present;
 	}
 
@@ -1621,6 +1623,7 @@ static int mpi3mr_bring_ioc_ready(struct mpi3mr_ioc *mrioc)
 		if (!pci_device_is_present(mrioc->pdev)) {
 			mrioc->unrecoverable = 1;
 			ioc_err(mrioc, "controller is not present while waiting to reset\n");
+			retval = -1;
 			goto out_device_not_present;
 		}
 
@@ -1654,6 +1657,7 @@ static int mpi3mr_bring_ioc_ready(struct mpi3mr_ioc *mrioc)
 				if (!pci_device_is_present(mrioc->pdev)) {
 					mrioc->unrecoverable = 1;
 					ioc_err(mrioc, "controller is not present at the bringup\n");
+					retval = -1;
 					goto out_device_not_present;
 				}
 				msleep(100);
@@ -1690,8 +1694,10 @@ static int mpi3mr_bring_ioc_ready(struct mpi3mr_ioc *mrioc)
 	ioc_config |= MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC;
 	writel(ioc_config, &mrioc->sysif_regs->ioc_configuration);
 
-	if (retry == 0)
+	if (!enable_started) {
 		start_time = jiffies;
+		enable_started = true;
+	}
 
 	timeout = mrioc->ready_timeout * 10;
 	do {
@@ -1731,7 +1737,7 @@ static int mpi3mr_bring_ioc_ready(struct mpi3mr_ioc *mrioc)
 
 out_failed:
 	elapsed_time_sec = jiffies_to_msecs(jiffies - start_time)/1000;
-	if ((retry < 2) && (elapsed_time_sec < (mrioc->ready_timeout - 60))) {
+	if (retry < 2 && elapsed_time_sec + 60 < mrioc->ready_timeout) {
 		retry++;
 
 		ioc_warn(mrioc, "retrying to bring IOC ready, retry_count:%d\n"
@@ -1739,6 +1745,8 @@ static int mpi3mr_bring_ioc_ready(struct mpi3mr_ioc *mrioc)
 
 		goto retry_bring_ioc_ready;
 	}
+	if (!retval)
+		retval = -1;
 	ioc_state = mpi3mr_get_iocstate(mrioc);
 	ioc_err(mrioc,
 	    "failed to bring to ready state,  current state: %s\n",
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
Reviewed-by on v1 1/1
#1 lore raw
Quoted 138 lines · Ian Rogers

mpi3mr_bring_ioc_ready() has three bugs in its failure and retry handling:

1. start_time is only assigned after MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC when retry == 0, after three earlier 'goto out_failed' paths. If soft reset, reset-state polling, or admin queue setup fails on the first attempt, out_failed computes elapsed_time_sec from an uninitialized start_time, and if a retry occurs (retry == 1), start_time remains uninitialized for the ready-polling loop as well.

2. In 'elapsed_time_sec < (mrioc->ready_timeout - 60)', ready_timeout is u32, so values below 60 wrap around in 32-bit unsigned arithmetic
   before zero-extending to u64, spuriously retrying past the timeout.

3. After mpi3mr_setup_admin_qpair() sets retval = 0 (or on entry where
   retval is initialized to 0), controller fault/reset-history checks,
   ready-state timeouts, and early unrecoverable or device-not- present
   exits return retval == 0 (success) to mpi3mr_init_ioc() and mpi3mr_reinit_ioc() even though controller bringup failed.

Initialize start_time = jiffies at function entry for pre-enable failure
exits, reset start_time = jiffies the first time MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC is written so pre-enable waits do not
drain ready_timeout, rewrite the retry window check as 'elapsed_time_sec + 60 < mrioc->ready_timeout', and ensure retval = - 1
on all failure exits.

Found via Clang -Wconditional-uninitialized on an x86_64 allyesconfig build. Verified by building drivers/scsi/mpi3mr/mpi3mr_fw.o with LLVM=1 W=1 (-Werror) and -Wconditional-uninitialized, and with a standalone C logic reproducer under ASan/UBSan; not runtime-tested on physical MPI3 hardware.

Fixes 59bd9cfe3fa0 ("scsi: mpi3mr: Code refactor of IOC init -

part2")

Fixes 0a319f162949 ("scsi: mpi3mr: Wait for diagnostic save during

controller init")

Fixes 9634bb07083c ("scsi: mpi3mr: Enhance the Enable Controller

retry logic")

Fixes 4616a4b3cb8a ("scsi: mpi3mr: Improve wait logic while

controller transitions to READY state")

Fixes fb6eb98f3965 ("scsi: mpi3mr: Handling of fault code for

insufficient power")

---
 drivers/scsi/mpi3mr/mpi3mr_fw.c | 14 +++++++++++---

 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/scsi/mpi3mr/mpi3mr_fw.c +2 -1
index f0d3cd398dd0..b92b2467e995 100644
@@ -1582,8 +1582,9 @@ static int mpi3mr_bring_ioc_ready(struct
 mpi3mr_ioc *mrioc)
  	int retval = 0;
  	enum mpi3mr_iocstate ioc_state;
  	u64 base_info;
+	bool enable_started = false;
  	u8 retry = 0;
-	u64 start_time, elapsed_time_sec;
+	u64 start_time = jiffies, elapsed_time_sec;


 retry_bring_ioc_ready:

drivers/scsi/mpi3mr/mpi3mr_fw.c +0 -0
@@ -1595,6 +1596,7 @@ static int mpi3mr_bring_ioc_ready(struct
 mpi3mr_ioc *mrioc)


  if (!mpi3mr_is_fault_recoverable(mrioc)) { mrioc->unrecoverable = 1;

drivers/scsi/mpi3mr/mpi3mr_fw.c +1 -0
+		retval = -1;

  goto out_device_not_present;
  }

drivers/scsi/mpi3mr/mpi3mr_fw.c +0 -0
@@ -1621,6 +1623,7 @@ static int mpi3mr_bring_ioc_ready(struct
 mpi3mr_ioc *mrioc)
  		if (!pci_device_is_present(mrioc->pdev)) {
  			mrioc->unrecoverable = 1;

  ioc_err(mrioc, "controller is not present while waiting to reset\n");

drivers/scsi/mpi3mr/mpi3mr_fw.c +1 -0
+			retval = -1;

  goto out_device_not_present; }

drivers/scsi/mpi3mr/mpi3mr_fw.c +0 -0
@@ -1654,6 +1657,7 @@ static int mpi3mr_bring_ioc_ready(struct
 mpi3mr_ioc *mrioc)

  if (!pci_device_is_present(mrioc-

Quoted 1 line

pdev)) {

  mrioc->unrecoverable = 1; ioc_err(mrioc, "controller is not present at the bringup\n");

drivers/scsi/mpi3mr/mpi3mr_fw.c +1 -0
+					retval = -1;

  goto out_device_not_present; }
  msleep(100);

drivers/scsi/mpi3mr/mpi3mr_fw.c +0 -0
@@ -1690,8 +1694,10 @@ static int mpi3mr_bring_ioc_ready(struct
 mpi3mr_ioc *mrioc)
  	ioc_config |= MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC;
  	writel(ioc_config, &mrioc->sysif_regs->ioc_configuration);

drivers/scsi/mpi3mr/mpi3mr_fw.c +1 -1
-	if (retry == 0)
+	if (!enable_started) {

  start_time = jiffies;

drivers/scsi/mpi3mr/mpi3mr_fw.c +2 -0
+		enable_started = true;
+	}


  timeout = mrioc->ready_timeout * 10;
  do {

drivers/scsi/mpi3mr/mpi3mr_fw.c +0 -0
@@ -1731,7 +1737,7 @@ static int mpi3mr_bring_ioc_ready(struct
 mpi3mr_ioc *mrioc)


 out_failed:
  elapsed_time_sec = jiffies_to_msecs(jiffies - start_time)/1000;

drivers/scsi/mpi3mr/mpi3mr_fw.c +1 -2
-	if ((retry < 2) && (elapsed_time_sec < (mrioc->ready_timeout
- 60))) {
+	if (retry < 2 && elapsed_time_sec + 60 < mrioc-
Quoted 1 line

ready_timeout) {

  retry++;

  ioc_warn(mrioc, "retrying to bring IOC ready, retry_count:%d\n"

drivers/scsi/mpi3mr/mpi3mr_fw.c +0 -0
@@ -1739,6 +1745,8 @@ static int mpi3mr_bring_ioc_ready(struct
 mpi3mr_ioc *mrioc)


  goto retry_bring_ioc_ready;
  }

drivers/scsi/mpi3mr/mpi3mr_fw.c +2 -0
+	if (!retval)
+		retval = -1;
Quoted 3 lines

  ioc_state = mpi3mr_get_iocstate(mrioc); ioc_err(mrioc,
      "failed to bring to ready state,  current state: %s\n",

Hello Ian

Thanks for this. I checked it against current mainline and all three issues are real. Some comments below.

Quoted 1 line

3. After mpi3mr_setup_admin_qpair() sets retval = 0 (or on entry

where

Quoted 1 line

retval is initialized to 0), controller fault/reset-history

checks,

Quoted 1 line

ready-state timeouts, and early unrecoverable or device-not-

present

Quoted 2 lines

exits return retval == 0 (success) to mpi3mr_init_ioc() and mpi3mr_reinit_ioc() even though controller bringup failed.

This is the most serious of the three. Both callers only check "if (retval)", so with the !mpi3mr_is_fault_recoverable() exit we can set mrioc->unrecoverable = 1 and still report success, and init then carries on against a dead controller.

Quoted 2 lines

2. In 'elapsed_time_sec < (mrioc->ready_timeout - 60)', ready_timeout is u32, so values below 60 wrap around in 32-bit unsigned

arithmetic

Quoted 1 line

before zero-extending to u64, spuriously retrying past the

timeout.

Agreed that the arithmetic is wrong, but note the fix changes behavior:
today any firmware advertising a ready timeout below 60s always gets its two retries; after this patch it gets none.

Sathya/Sreekanth, can you confirm whether any shipping MPI3 firmware reports a ready timeout under 60 seconds, and if so whether dropping the retries entirely is the intended behavior?

Quoted 1 line

Initialize start_time = jiffies at function entry for pre-enable

failure

Quoted 2 lines

exits, reset start_time = jiffies the first time MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC is written so pre-enable waits do

not

Quoted 2 lines

drain ready_timeout, rewrite the retry window check as 'elapsed_time_sec + 60 < mrioc->ready_timeout', and ensure retval = -

1

Quoted 1 line

on all failure exits.

Would you consider splitting this into three patches, one per issue, each with only its own Fixes: tag? The retval fix stands on its own and is the one stable and distro kernels will want first; the ready_timeout change may need to wait for Broadcom's answer above. Five Fixes: tags across different commits on one patch also makes backporting harder.

Quoted 5 lines

Found via Clang -Wconditional-uninitialized on an x86_64 allyesconfig build. Verified by building drivers/scsi/mpi3mr/mpi3mr_fw.o with LLVM=1 W=1 (-Werror) and -Wconditional-uninitialized, and with a standalone C logic reproducer under ASan/UBSan; not runtime-tested on physical MPI3 hardware.

I have access to mpi3mr hardware and can run normal bring-up, controller reset and host-reset reinit on a v2 to make sure the good path does not regress, and will send a Tested-by.

With the split, for the retval and start_time fixes:

Regards
Laurence