Skip to contentSkip to conversation

scsi: ufs: Fix runtime PM messages never-ending cycle

PATCH Applied Adrian Hunter 2 participants 5 messages Feb 28, 2022 – Mar 9, 2022 SCSI

lore
Share & tools
lore
#0 lore raw

v4 cover scsi: ufs: Fix runtime PM messages never-ending cycle

Hi

Here is V4 to address comments by Martin. See patches for version history.

Summary:

Kernel messages produced during runtime PM can cause a never-ending cycle because user space utilities (e.g. journald or rsyslog) write the messages back to storage, causing runtime resume, more messages, and so on.

Messages that tell of things that are expected to happen, are arguably unnecessary, so make changes to suppress them for the UFS driver.

Adrian Hunter (2):
scsi: Add quiet_suspend flag for SCSI devices to suppress some PM messages
scsi: ufs: Fix runtime PM messages never-ending cycle

 drivers/scsi/scsi_error.c  |  9 +++++++--
 drivers/scsi/sd.c          |  6 ++++--
 drivers/scsi/ufs/ufshcd.c  | 21 +++++++++++++++++++--
 include/scsi/scsi_device.h |  6 ++++++
 4 files changed, 36 insertions(+), 6 deletions(-)

Regards
Adrian

#1 lore raw

PATCH v4 1/2 scsi: Add quiet_suspend flag for SCSI devices to suppress some PM messages

Kernel messages produced during runtime PM can cause a never-ending cycle because user space utilities (e.g. journald or rsyslog) write the messages back to storage, causing runtime resume, more messages, and so on.

Messages that tell of things that are expected to happen, are arguably unnecessary, so add a flag to suppress them. This flag is used by the UFS driver in a subsequent patch.

---

Changes in V4:

New separate patch for SCSI changes only, without UFS driver changes.

Rename no_rst_sense_msg to quiet_suspend and use to make all messages conditional.

 drivers/scsi/scsi_error.c  | 9 +++++++--
 drivers/scsi/sd.c          | 6 ++++--
 include/scsi/scsi_device.h | 6 ++++++
 3 files changed, 17 insertions(+), 4 deletions(-)
drivers/scsi/scsi_error.c +7 -2
index 60a6ae9d1219..602410bd5dd6 100644
@@ -484,8 +484,13 @@ static void scsi_report_sense(struct scsi_device *sdev,
 
 		if (sshdr->asc == 0x29) {
 			evt_type = SDEV_EVT_POWER_ON_RESET_OCCURRED;
-			sdev_printk(KERN_WARNING, sdev,
-				    "Power-on or device reset occurred\n");
+			/*
+			 * Do not print message if it is an expected side-effect
+			 * of runtime PM.
+			 */
+			if (!sdev->quiet_suspend)
+				sdev_printk(KERN_WARNING, sdev,
+					    "Power-on or device reset occurred\n");
 		}
 
 		if (sshdr->asc == 0x2a && sshdr->ascq == 0x01) {
drivers/scsi/sd.c +4 -2
index 62eb9921cc94..fefe8f1da422 100644
@@ -3752,7 +3752,8 @@ static int sd_suspend_common(struct device *dev, bool ignore_stop_errors)
 		return 0;
 
 	if (sdkp->WCE && sdkp->media_present) {
-		sd_printk(KERN_NOTICE, sdkp, "Synchronizing SCSI cache\n");
+		if (!sdkp->device->quiet_suspend)
+			sd_printk(KERN_NOTICE, sdkp, "Synchronizing SCSI cache\n");
 		ret = sd_sync_cache(sdkp, &sshdr);
 
 		if (ret) {
@@ -3774,7 +3775,8 @@ static int sd_suspend_common(struct device *dev, bool ignore_stop_errors)
 	}
 
 	if (sdkp->device->manage_start_stop) {
-		sd_printk(KERN_NOTICE, sdkp, "Stopping disk\n");
+		if (!sdkp->device->quiet_suspend)
+			sd_printk(KERN_NOTICE, sdkp, "Stopping disk\n");
 		/* an error is not worth aborting a system sleep */
 		ret = sd_start_stop_device(sdkp, 0);
 		if (ignore_stop_errors)
include/scsi/scsi_device.h +6 -0
index 647c53b26105..9f5408b2d8d3 100644
@@ -206,6 +206,12 @@ struct scsi_device {
 	unsigned rpm_autosuspend:1;	/* Enable runtime autosuspend at device
 					 * creation time */
 	unsigned ignore_media_change:1; /* Ignore MEDIA CHANGE on resume */
+	/*
+	 * quiet_suspend to suppress messages due to runtime PM to avoid
+	 * never-ending cycles of messages written back to storage by user space
+	 * causing runtime resume, causing more messages and so on.
+	 */
+	unsigned quiet_suspend:1;	/* Do not print runtime PM related messages */
 
 	unsigned int queue_stopped;	/* request queue is quiesced */
 	bool offline_already;		/* Device offline message logged */
Signature · 1 line
2.25.1
#2 lore raw

PATCH v4 2/2 scsi: ufs: Fix runtime PM messages never-ending cycle

In reply to Adrian Hunter

Kernel messages produced during runtime PM can cause a never-ending cycle because user space utilities (e.g. journald or rsyslog) write the messages back to storage, causing runtime resume, more messages, and so on.

Messages that tell of things that are expected to happen, are arguably unnecessary, so suppress them.

UFS driver messages are changes to from dev_err() to dev_dbg() which means they will not display unless activated by dynamic debug of building with -DDEBUG.

sdev->quiet_suspend is set to skip messages from sd_suspend_common() "Synchronizing SCSI cache", "Stopping disk" and scsi_report_sense() "Power-on or device reset occurred" message (Note, that message appears when the LUN is accessed after runtime PM, not during runtime PM)

Example messages from Ubuntu 21.10:

 $ dmesg | tail
 [ 1620.380071] ufshcd 0000:00:12.5: ufshcd_print_pwr_info:[RX, TX]: gear=[1, 1], lane[1, 1], pwr[SLOWAUTO_MODE, SLOWAUTO_MODE], rate = 0
 [ 1620.408825] ufshcd 0000:00:12.5: ufshcd_print_pwr_info:[RX, TX]: gear=[4, 4], lane[2, 2], pwr[FAST MODE, FAST MODE], rate = 2
 [ 1620.409020] ufshcd 0000:00:12.5: ufshcd_find_max_sup_active_icc_level: Regulator capability was not set, actvIccLevel=0
 [ 1620.409524] sd 0:0:0:0: Power-on or device reset occurred
 [ 1622.938794] sd 0:0:0:0: [sda] Synchronizing SCSI cache
 [ 1622.939184] ufs_device_wlun 0:0:0:49488: Power-on or device reset occurred
 [ 1625.183175] ufshcd 0000:00:12.5: ufshcd_print_pwr_info:[RX, TX]: gear=[1, 1], lane[1, 1], pwr[SLOWAUTO_MODE, SLOWAUTO_MODE], rate = 0
 [ 1625.208041] ufshcd 0000:00:12.5: ufshcd_print_pwr_info:[RX, TX]: gear=[4, 4], lane[2, 2], pwr[FAST MODE, FAST MODE], rate = 2
 [ 1625.208311] ufshcd 0000:00:12.5: ufshcd_find_max_sup_active_icc_level: Regulator capability was not set, actvIccLevel=0
 [ 1625.209035] sd 0:0:0:0: Power-on or device reset occurred

Note for stable: depends on patch "scsi: Add quiet_suspend flag for SCSI devices to suppress some PM messages"

---

Changes in V4:

Separate patch for UFS driver changes only.

Changes in V3:

Keep reset sense message for non-UFS devices

Changes in V2:

Remove offending SCSI messages
Use dev_dbg for offending UFSHCD messages

 drivers/scsi/ufs/ufshcd.c | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)
drivers/scsi/ufs/ufshcd.c +19 -2
index 50b12d60dc1b..f057d9ff0c6c 100644
@@ -585,7 +585,12 @@ static void ufshcd_print_pwr_info(struct ufs_hba *hba)
 		"INVALID MODE",
 	};
 
-	dev_err(hba->dev, "%s:[RX, TX]: gear=[%d, %d], lane[%d, %d], pwr[%s, %s], rate = %d\n",
+	/*
+	 * Using dev_dbg to avoid messages during runtime PM to avoid
+	 * never-ending cycles of messages written back to storage by user space
+	 * causing runtime resume, causing more messages and so on.
+	 */
+	dev_dbg(hba->dev, "%s:[RX, TX]: gear=[%d, %d], lane[%d, %d], pwr[%s, %s], rate = %d\n",
 		 __func__,
 		 hba->pwr_info.gear_rx, hba->pwr_info.gear_tx,
 		 hba->pwr_info.lane_rx, hba->pwr_info.lane_tx,
@@ -5024,6 +5029,12 @@ static int ufshcd_slave_configure(struct scsi_device *sdev)
 		pm_runtime_get_noresume(&sdev->sdev_gendev);
 	else if (ufshcd_is_rpm_autosuspend_allowed(hba))
 		sdev->rpm_autosuspend = 1;
+	/*
+	 * Do not print messages during runtime PM to avoid never-ending cycles
+	 * of messages written back to storage by user space causing runtime
+	 * resume, causing more messages and so on.
+	 */
+	sdev->quiet_suspend = 1;
 
 	ufshcd_crypto_register(hba, q);
 
@@ -7339,7 +7350,13 @@ static u32 ufshcd_find_max_sup_active_icc_level(struct ufs_hba *hba,
 
 	if (!hba->vreg_info.vcc || !hba->vreg_info.vccq ||
 						!hba->vreg_info.vccq2) {
-		dev_err(hba->dev,
+		/*
+		 * Using dev_dbg to avoid messages during runtime PM to avoid
+		 * never-ending cycles of messages written back to storage by
+		 * user space causing runtime resume, causing more messages and
+		 * so on.
+		 */
+		dev_dbg(hba->dev,
 			"%s: Regulator capability was not set, actvIccLevel=%d",
 							__func__, icc_level);
 		goto out;
Signature · 1 line
2.25.1
#3 lore raw
In reply to Adrian Hunter

Adrian,

Quoted 7 lines

Kernel messages produced during runtime PM can cause a never-ending cycle because user space utilities (e.g. journald or rsyslog) write the messages back to storage, causing runtime resume, more messages, and so on.

Messages that tell of things that are expected to happen, are arguably unnecessary, so make changes to suppress them for the UFS driver.

Applied to 5.18/scsi-staging, thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#4 lore raw
In reply to Adrian Hunter
Quoted 10 lines · Adrian Hunter

Here is V4 to address comments by Martin. See patches for version history.

Summary:

Kernel messages produced during runtime PM can cause a never-ending cycle because user space utilities (e.g. journald or rsyslog) write the messages back to storage, causing runtime resume, more messages, and so on.

[...]

Applied to 5.18/scsi-queue, thanks!

[1/2] scsi: Add quiet_suspend flag for SCSI devices to suppress some PM messages

      https://git.kernel.org/mkp/scsi/c/af4edb1d50c6
[2/2] scsi: ufs: Fix runtime PM messages never-ending cycle
      https://git.kernel.org/mkp/scsi/c/71bb9ab6e351
Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering