Skip to contentSkip to conversation

scsi: smartpqi_init: Reporting 'logical unit failure'

PATCH Superseded Erwan Velu 6 participants 14 messages Feb 27, 2019 – Mar 11, 2019

Subsystems (2)
lore
Share & tools
lore

Erwan Velu

author
#0 lore raw

PATCH 1/1 scsi: smartpqi_init: Reporting 'logical unit failure'

When this HARDWARE_ERROR/0x3e/0x1 case is triggered, the logical volume is offlined.
When reading the kernel log, the cause why the device got offlined isn't reported to the user.
This situation makes difficult for admins to estimate _why_ the volume got offlined.
Reading this part of the code makes clear this is because driver received a HARDWARE_ERROR/0x3e/0x1 which is a 'logical unit failure'.

This patch is just about reporting that fact to help admins making a relationship between this event and the offlining.

---
 drivers/scsi/smartpqi/smartpqi_init.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/scsi/smartpqi/smartpqi_init.c +6 -0
index f564af8949e8..89f37d76735c 100644
@@ -2764,6 +2764,12 @@ static void pqi_process_raid_io_error(struct pqi_io_request *io_request)
 				sshdr.sense_key == HARDWARE_ERROR &&
 				sshdr.asc == 0x3e &&
 				sshdr.ascq == 0x1) {
+			struct pqi_ctrl_info *ctrl_info = shost_to_hba(scmd->device->host);
+			struct pqi_scsi_dev *device = scmd->device->hostdata;
+
+			dev_err(&ctrl_info->pci_dev->dev, "received 'logical unit failure' from controller for scsi %d:%d:%d:%d\n",
+							ctrl_info->scsi_host->host_no, device->bus,
+							device->target, device->lun);
 			pqi_take_device_offline(scmd->device, "RAID");
 			host_byte = DID_NO_CONNECT;
 		}
Signature · 1 line
2.20.1
#1 lore raw

Hey,

That makes me wonder why the 0x3e / 0x2 isn't handled here aka

3E/02 DZTPROMAEBKVF TIMEOUT ON LOGICAL UNIT Is it possible the controller send to the kernel this kind of message, if so shouldn't we handle it here ? Erwan,

Le 27/02/2019 à 17:31, Erwan Velu a écrit :

Quoted 29 lines

When this HARDWARE_ERROR/0x3e/0x1 case is triggered, the logical volume is offlined.
When reading the kernel log, the cause why the device got offlined isn't reported to the user.
This situation makes difficult for admins to estimate _why_ the volume got offlined.
Reading this part of the code makes clear this is because driver received a HARDWARE_ERROR/0x3e/0x1 which is a 'logical unit failure'.

This patch is just about reporting that fact to help admins making a relationship between this event and the offlining.

---
  drivers/scsi/smartpqi/smartpqi_init.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/scsi/smartpqi/smartpqi_init.c +6 -0
index f564af8949e8..89f37d76735c 100644
@@ -2764,6 +2764,12 @@ static void pqi_process_raid_io_error(struct pqi_io_request *io_request)
  				sshdr.sense_key == HARDWARE_ERROR &&
  				sshdr.asc == 0x3e &&
  				sshdr.ascq == 0x1) {
+			struct pqi_ctrl_info *ctrl_info = shost_to_hba(scmd->device->host);
+			struct pqi_scsi_dev *device = scmd->device->hostdata;
+
+			dev_err(&ctrl_info->pci_dev->dev, "received 'logical unit failure' from controller for scsi %d:%d:%d:%d\n",
+							ctrl_info->scsi_host->host_no, device->bus,
+							device->target, device->lun);
  			pqi_take_device_offline(scmd->device, "RAID");
  			host_byte = DID_NO_CONNECT;
  		}
#2 lore raw
In reply to Erwan Velu
Quoted 39 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Erwan Velu
Sent: Wednesday, February 27, 2019 10:32 AM
Subject: [PATCH] scsi: smartpqi_init: Reporting 'logical unit failure'

When this HARDWARE_ERROR/0x3e/0x1 case is triggered, the logical volume is offlined. When reading the kernel log, the cause why the device got offlined isn't reported to the user. This situation makes difficult for admins to estimate _why_ the volume got offlined. Reading this part of the code makes clear this is because driver received a HARDWARE_ERROR/0x3e/0x1 which is a 'logical unit failure'.

This patch is just about reporting that fact to help admins making a relationship between this event and the offlining.

---
 drivers/scsi/smartpqi/smartpqi_init.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/scsi/smartpqi/smartpqi_init.c +6 -0
index f564af8949e8..89f37d76735c 100644
@@ -2764,6 +2764,12 @@ static void pqi_process_raid_io_error(struct pqi_io_request *io_request)
 				sshdr.sense_key == HARDWARE_ERROR &&
 				sshdr.asc == 0x3e &&
 				sshdr.ascq == 0x1) {
+			struct pqi_ctrl_info *ctrl_info = shost_to_hba(scmd->device->host);
+			struct pqi_scsi_dev *device = scmd->device->hostdata;
+
+			dev_err(&ctrl_info->pci_dev->dev, "received 'logical unit failure' from controller
 for scsi %d:%d:%d:%d\n",
+							ctrl_info->scsi_host->host_no, device->bus,
+							device->target, device->lun);
 			pqi_take_device_offline(scmd->device, "RAID");
 			host_byte = DID_NO_CONNECT;
Quoted 1 line

}

Be careful printing errors per-IO; you could get thousands of them if things go bad. The block layer print_req_error() uses printk_ratelimited(KERN_ERR) for that reason, and the SCSI layer scsi_io_completion_action() maintains a ratelimit on its own.

The dev_err_ratelimited() macro might be a good fit here.

---
Robert Elliott, HPE Persistent Memory

#3 lore raw

[...]

Quoted 5 lines

Be careful printing errors per-IO; you could get thousands of them if things go bad. The block layer print_req_error() uses printk_ratelimited(KERN_ERR) for that reason, and the SCSI layer scsi_io_completion_action() maintains a ratelimit on its own.

The dev_err_ratelimited() macro might be a good fit here.

Thanks for the tip. I updated the patch and send a V2 for that.

I adjusted also to commit message to make it more explicit.

Thanks for the review.

Erwan,

author
#4 lore raw

PATCH v2 1/1 scsi: smartpqi_init: Reporting 'logical unit failure'

In reply to Erwan Velu

When the HARDWARE_ERROR/0x3e/0x1 case is triggered, the logical volume is offlined. When reading the kernel log, the reason why the device got offlined isn't reported to the user. This situation makes difficult for admins to estimate the root cause of the issue they analize.

Reading this part of the code makes clear this is because driver received a HARDWARE_ERROR/0x3e/0x1 which is a 'logical unit failure'. This patch is just about reporting the reason behind the offlining to ease the analyse.

---
 drivers/scsi/smartpqi/smartpqi_init.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/scsi/smartpqi/smartpqi_init.c +6 -0
index f564af8949e8..dfc4a6813440 100644
@@ -2764,6 +2764,12 @@ static void pqi_process_raid_io_error(struct pqi_io_request *io_request)
 				sshdr.sense_key == HARDWARE_ERROR &&
 				sshdr.asc == 0x3e &&
 				sshdr.ascq == 0x1) {
+			struct pqi_ctrl_info *ctrl_info = shost_to_hba(scmd->device->host);
+			struct pqi_scsi_dev *device = scmd->device->hostdata;
+
+			dev_err_ratelimited(&ctrl_info->pci_dev->dev, "received 'logical unit failure' from controller for scsi %d:%d:%d:%d\n",
+							ctrl_info->scsi_host->host_no, device->bus,
+							device->target, device->lun);
 			pqi_take_device_offline(scmd->device, "RAID");
 			host_byte = DID_NO_CONNECT;
 		}
Signature · 1 line
2.20.1
#5 lore raw
Quoted 7 lines · Erwan Velu
(quoted diff) +1 -0
+			dev_err_ratelimited(&ctrl_info->pci_dev-
Quoted 1 line

dev, "received 'logical unit failure' from controller for scsi

%d:%d:%d:%d\n",

(quoted diff) +1 -0
+							ctrl_info-
Quoted 1 line

scsi_host->host_no, device->bus,

(quoted diff) +1 -0
+							device-
Quoted 1 line

target, device->lun);

Shouldn't this be a variant of sdev/scmd_printk? Otherwise it tells you what disk in the array terms is the problem but not what device in your actual system is affected.

James

#6 lore raw

Le 01/03/2019 à 16:26, James Bottomley a écrit :

Quoted 4 lines

[...]
Shouldn't this be a variant of sdev/scmd_printk? Otherwise it tells you what disk in the array terms is the problem but not what device in your actual system is affected.

Hey James,

My initial take on that was that pqi_take_device_offline(), which is called just after, will print the "re-scanning " message with the same format.

As they will be both printed in the same error context and one after the other, I though that would make sense to represent the same information to ease the reading like cause -> consequence.

As the message is about the LUN itself, which is reported faulty, I though it would worth reporting the info that way.

Shall I consider printing also the disk name in addition ?

Erwan,

#7 lore raw
Quoted 21 lines · Erwan Velu
Quoted 6 lines · James Bottomley

[...]
Shouldn't this be a variant of sdev/scmd_printk? Otherwise it tells
you what disk in the array terms is the problem but not what device in
your actual system is affected.

Hey James,

My initial take on that was that pqi_take_device_offline(), which is called just after, will print the "re-scanning " message with the same
format.

As they will be both printed in the same error context and one after the
other, I though that would make sense to represent the same information
to ease the reading like cause -> consequence.

Quoted 4 lines

As the message is about the LUN itself, which is reported faulty, I though it would worth reporting the info that way.

Shall I consider printing also the disk name in addition ?

I was thinking just

if (printk_ratelimit())
scmd_printk(KERN_ERR, scmd, "received 'logical unit failure' from controller for scsi %d:%d:%d:%d\n", ...

That will give all the necessary information

James

#8 lore raw

Le 01/03/2019 à 16:56, James Bottomley a écrit :

Quoted 7 lines

[...]
I was thinking just

if (printk_ratelimit())
scmd_printk(KERN_ERR, scmd, "received 'logical unit failure' from controller for scsi %d:%d:%d:%d\n", ...

That will give all the necessary information

I'm pretty new to this area, learning from you  ;o)

I'll update the v3 this way.

Thanks for the review.

Erwan,

author
#9 lore raw

PATCH v3 1/1 scsi: smartpqi_init: Reporting 'logical unit failure'

In reply to James Bottomley

When the HARDWARE_ERROR/0x3e/0x1 case is triggered, the logical volume is offlined. When reading the kernel log, the reason why the device got offlined isn't reported to the user. This situation makes difficult for admins to estimate the root cause of the issue they analize.

Reading this part of the code makes clear this is because driver received a HARDWARE_ERROR/0x3e/0x1 which is a 'logical unit failure'. This patch is just about reporting the reason behind the offlining to ease the analyse.

---
 drivers/scsi/smartpqi/smartpqi_init.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/scsi/smartpqi/smartpqi_init.c +6 -0
index f564af8949e8..adebafe56b5b 100644
@@ -2764,6 +2764,12 @@ static void pqi_process_raid_io_error(struct pqi_io_request *io_request)
 				sshdr.sense_key == HARDWARE_ERROR &&
 				sshdr.asc == 0x3e &&
 				sshdr.ascq == 0x1) {
+			struct pqi_ctrl_info *ctrl_info = shost_to_hba(scmd->device->host);
+			struct pqi_scsi_dev *device = scmd->device->hostdata;
+
+			if (printk_ratelimit())
+				scmd_printk(KERN_ERR, scmd, "received 'logical unit failure' from controller for scsi %d:%d:%d:%d\n",
+					ctrl_info->scsi_host->host_no, device->bus, device->target, device->lun);
 			pqi_take_device_offline(scmd->device, "RAID");
 			host_byte = DID_NO_CONNECT;
 		}
Signature · 1 line
2.20.1
maintainer Acked-by Don Brace on v3 1/1
#10 lore raw

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Erwan Velu
Sent: Friday, March 1, 2019 10:08 AM
Cc: Erwan Velu <[email protected]>; Don Brace <[email protected]>; James E.J. Bottomley <[email protected]>; Martin K. Petersen <[email protected]>; open list:MICROSEMI SMART ARRAY SMARTPQI DRIVER (smartpqi) <[email protected]>; open list:MICROSEMI SMART ARRAY SMARTPQI DRIVER (smartpqi) <[email protected]>; open list <[email protected]>
Subject: [PATCH v3] scsi: smartpqi_init: Reporting 'logical unit failure'

When the HARDWARE_ERROR/0x3e/0x1 case is triggered, the logical volume is offlined. When reading the kernel log, the reason why the device got offlined isn't reported to the user. This situation makes difficult for admins to estimate the root cause of the issue they analize.

Reading this part of the code makes clear this is because driver received a HARDWARE_ERROR/0x3e/0x1 which is a 'logical unit failure'. This patch is just about reporting the reason behind the offlining to ease the analyse.

---
 drivers/scsi/smartpqi/smartpqi_init.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/scsi/smartpqi/smartpqi_init.c +6 -0
index f564af8949e8..adebafe56b5b 100644
@@ -2764,6 +2764,12 @@ static void pqi_process_raid_io_error(struct pqi_io_request *io_request)
 				sshdr.sense_key == HARDWARE_ERROR &&
 				sshdr.asc == 0x3e &&
 				sshdr.ascq == 0x1) {
+			struct pqi_ctrl_info *ctrl_info = shost_to_hba(scmd->device->host);
+			struct pqi_scsi_dev *device = scmd->device->hostdata;
+
+			if (printk_ratelimit())
+				scmd_printk(KERN_ERR, scmd, "received 'logical unit failure' from controller for scsi %d:%d:%d:%d\n",
+					ctrl_info->scsi_host->host_no, device->bus, device->target, device->lun);
 			pqi_take_device_offline(scmd->device, "RAID");
 			host_byte = DID_NO_CONNECT;
 		}
Signature · 1 line
2.20.1
#11 lore raw
In reply to Erwan Velu

Erwan,

Quoted 5 lines

When the HARDWARE_ERROR/0x3e/0x1 case is triggered, the logical volume is offlined. When reading the kernel log, the reason why the device got offlined isn't reported to the user. This situation makes difficult for admins to estimate the root cause of the issue they analize.

Applied to 5.1/scsi-queue, thanks.

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
#12 lore raw

Le 06/03/2019 à 18:34, Martin K. Petersen a écrit :

Quoted 9 lines

Erwan,

Quoted 5 lines

When the HARDWARE_ERROR/0x3e/0x1 case is triggered, the logical volume is offlined. When reading the kernel log, the reason why the device got offlined isn't reported to the user. This situation makes difficult for admins to estimate the root cause of the issue they analize.

Applied to 5.1/scsi-queue, thanks.

Thanks Martin !

#13 lore raw
In reply to Martin K. Petersen

Le 06/03/2019 à 18:34, Martin K. Petersen a écrit :

Quoted 7 lines

Erwan,

Quoted 5 lines

When the HARDWARE_ERROR/0x3e/0x1 case is triggered, the logical volume is offlined. When reading the kernel log, the reason why the device got offlined isn't reported to the user. This situation makes difficult for admins to estimate the root cause of the issue they analize.

While I was debugging this scenario, I was wondering if some other cases were possible.

The current code is considering  (sshdr.asc == 0x3e && sshdr.ascq == 0x1), but what if ascq have a different value here ?

The specification (http://www.t10.org/lists/asc-num.htm#ASC_3E) reports other sub-values like ASCQ==02 which means a timeout on the lun.

So, does the raid controllers supported by smartpqi can generates these other values ? If so, how/where are they handled ?

I was considering at least, to a switch statement on sshdr.ascq with a 0x1 case on the current code and a a default one that prints at least a message saying that a message got received but not handled.

Thanks !

Erwan,