Skip to contentSkip to conversation

libsas and drivers: NCQ error handling

PATCH Superseded John Garry 3 participants 24 messages Sep 27, 2022 – Oct 24, 2022

Subsystems (5)

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

John Garry

author
#0 lore raw

v5 cover libsas and drivers: NCQ error handling

As reported in [0], the pm8001 driver NCQ error handling more or less duplicates what libata does in link error handling, as follows:
- abort all commands
- do autopsy with read log ext 10 command
- reset the target to recover, if necessary

Indeed for the hisi_sas driver we want to add similar handling for NCQ errors.

This series add a new libsas API - sas_ata_device_link_abort() - to handle host NCQ errors, and fixes up pm8001 and hisi_sas drivers to use it.

A difference in the pm8001 driver NCQ error handling is that we send SATA_ABORT per-task prior to read log ext10, but I feel that this should not make a difference to the error handling.

Damien kindly tested previous the series for pm8001, but any further pm8001 testing would be appreciated as I have since tweaked pm8001 handling again. This is because the pm8001 driver hangs on my arm64 machine read log ext10 command.

Finally with these changes we can make the libsas task alloc/free APIs private, which they should always have been.

Based on mkp-scsi @ 6.1/scsi-staging 57569c37f0ad ("scsi: iscsi:
iscsi_tcp: Fix null-ptr-deref while calling getpeername()")

[0] https://lore.kernel.org/linux-scsi/[email protected]/

Changes since v4:
- Add Jason's tags (thanks)
- Rebase

Changes since v3:
- Add Damien's tags (thanks)
- Modify hisi_sas processing as follows:
- use sas_task_abort() for rejected IO
- Modify abort task processing to issue softreset in certain circumstances
- rebase

Changes since v2:
- Stop sending SATA_ABORT all for pm8001 handling
- Make "reset" optional in sas_ata_device_link_abort()
- Drop Jack's ACK

John Garry (5):
scsi: libsas: Add sas_ata_device_link_abort()
scsi: hisi_sas: Move slot variable definition in hisi_sas_abort_task()
scsi: pm8001: Modify task abort handling for SATA task
scsi: pm8001: Use sas_ata_device_link_abort() to handle NCQ errors
scsi: libsas: Make sas_{alloc, alloc_slow, free}_task() private

Xingui Yang (2):
scsi: hisi_sas: Add SATA_DISK_ERR bit handling for v3 hw
scsi: hisi_sas: Modify v3 HW SATA disk error state completion
processing

 drivers/scsi/hisi_sas/hisi_sas.h       |   1 +
 drivers/scsi/hisi_sas/hisi_sas_main.c  |  26 +++-
 drivers/scsi/hisi_sas/hisi_sas_v3_hw.c |  53 ++++++-
 drivers/scsi/libsas/sas_ata.c          |  12 ++
 drivers/scsi/libsas/sas_init.c         |   3 -
 drivers/scsi/libsas/sas_internal.h     |   4 +
 drivers/scsi/pm8001/pm8001_hwi.c       | 186 ++++---------------------
 drivers/scsi/pm8001/pm8001_sas.c       |   8 ++
 drivers/scsi/pm8001/pm8001_sas.h       |   4 -
 drivers/scsi/pm8001/pm80xx_hwi.c       | 177 +++--------------------
 include/scsi/libsas.h                  |   4 -
 include/scsi/sas_ata.h                 |   6 +
 12 files changed, 143 insertions(+), 341 deletions(-)
Signature · 1 line
2.35.3
author
#1 lore raw

PATCH v5 1/7 scsi: libsas: Add sas_ata_device_link_abort()

Similar to how AHCI handles NCQ errors in ahci_error_intr() -> ata_port_abort() -> ata_do_link_abort(), add an NCQ error handler for LLDDs to call to initiate a link abort.

This will mark all outstanding QCs as failed and kick-off EH.

Note:
A "force reset" argument is added for drivers which require the ATA error handling to always reset the device.

A driver may require this feature for when SATA device per-SCSI cmnd resources are only released during reset for ATA EH. As such, we need an option to force reset to be done, regardless of what any EH autopsy decides.

---
 drivers/scsi/libsas/sas_ata.c | 12 ++++++++++++
 include/scsi/sas_ata.h        |  6 ++++++
 2 files changed, 18 insertions(+)
drivers/scsi/libsas/sas_ata.c +12 -0
index d35c9296f738..bdffb6852dcf 100644
@@ -861,6 +861,18 @@ void sas_ata_wait_eh(struct domain_device *dev)
 	ata_port_wait_eh(ap);
 }
 
+void sas_ata_device_link_abort(struct domain_device *device, bool force_reset)
+{
+	struct ata_port *ap = device->sata_dev.ap;
+	struct ata_link *link = &ap->link;
+
+	link->eh_info.err_mask |= AC_ERR_DEV;
+	if (force_reset)
+		link->eh_info.action |= ATA_EH_RESET;
+	ata_link_abort(link);
+}
+EXPORT_SYMBOL_GPL(sas_ata_device_link_abort);
+
 int sas_execute_ata_cmd(struct domain_device *device, u8 *fis, int force_phy_id)
 {
 	struct sas_tmf_task tmf_task = {};
include/scsi/sas_ata.h +6 -0
index a1df4f9d57a3..e47f0aec0722 100644
@@ -32,6 +32,7 @@ void sas_probe_sata(struct asd_sas_port *port);
 void sas_suspend_sata(struct asd_sas_port *port);
 void sas_resume_sata(struct asd_sas_port *port);
 void sas_ata_end_eh(struct ata_port *ap);
+void sas_ata_device_link_abort(struct domain_device *dev, bool force_reset);
 int sas_execute_ata_cmd(struct domain_device *device, u8 *fis,
 			int force_phy_id);
 int sas_ata_wait_after_reset(struct domain_device *dev, unsigned long deadline);
@@ -87,6 +88,11 @@ static inline void sas_ata_end_eh(struct ata_port *ap)
 {
 }
 
+static inline void sas_ata_device_link_abort(struct domain_device *dev,
+					     bool force_reset)
+{
+}
+
 static inline int sas_execute_ata_cmd(struct domain_device *device, u8 *fis,
 				      int force_phy_id)
 {
Signature · 1 line
2.35.3
author
#2 lore raw

PATCH v5 2/7 scsi: hisi_sas: Move slot variable definition in hisi_sas_abort_task()

In reply to John Garry

Each branch currently defines a slot variable independently, and it is neater to move it to the function head.

---
 drivers/scsi/hisi_sas/hisi_sas_main.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
drivers/scsi/hisi_sas/hisi_sas_main.c +3 -5
index 699b07abb6b0..8303aa5eaf25 100644
@@ -1547,6 +1547,7 @@ static int hisi_sas_abort_task(struct sas_task *task)
 	struct hisi_sas_internal_abort_data internal_abort_data = { false };
 	struct domain_device *device = task->dev;
 	struct hisi_sas_device *sas_dev = device->lldd_dev;
+	struct hisi_sas_slot *slot = task->lldd_task;
 	struct hisi_hba *hisi_hba;
 	struct device *dev;
 	int rc = TMF_RESP_FUNC_FAILED;
@@ -1560,7 +1561,6 @@ static int hisi_sas_abort_task(struct sas_task *task)
 
 	spin_lock_irqsave(&task->task_state_lock, flags);
 	if (task->task_state_flags & SAS_TASK_STATE_DONE) {
-		struct hisi_sas_slot *slot = task->lldd_task;
 		struct hisi_sas_cq *cq;
 
 		if (slot) {
@@ -1578,8 +1578,7 @@ static int hisi_sas_abort_task(struct sas_task *task)
 	task->task_state_flags |= SAS_TASK_STATE_ABORTED;
 	spin_unlock_irqrestore(&task->task_state_lock, flags);
 
-	if (task->lldd_task && task->task_proto & SAS_PROTOCOL_SSP) {
-		struct hisi_sas_slot *slot = task->lldd_task;
+	if (slot && task->task_proto & SAS_PROTOCOL_SSP) {
 		u16 tag = slot->idx;
 		int rc2;
 
@@ -1613,9 +1612,8 @@ static int hisi_sas_abort_task(struct sas_task *task)
 			hisi_sas_dereg_device(hisi_hba, device);
 			rc = hisi_sas_softreset_ata_disk(device);
 		}
-	} else if (task->lldd_task && task->task_proto & SAS_PROTOCOL_SMP) {
+	} else if (slot && task->task_proto & SAS_PROTOCOL_SMP) {
 		/* SMP */
-		struct hisi_sas_slot *slot = task->lldd_task;
 		u32 tag = slot->idx;
 		struct hisi_sas_cq *cq = &hisi_hba->cq[slot->dlvry_queue];
 
Signature · 1 line
2.35.3
author
#3 lore raw

PATCH v5 3/7 scsi: hisi_sas: Add SATA_DISK_ERR bit handling for v3 hw

In reply to John Garry

From: Xingui Yang <[email protected]>

When CQ header dw3 SATA_DISK_ERR is set it means this SATA disk is in error state and the current IPTT is invalid. An invalid IPTT does not correspond to any slot.

In this scenario, new I/Os that delivered to disk will be rejected by the controller and all I/Os remaining in the disk should be aborted, which we add here with the sas_ata_device_link_abort() call.

In hisi_sas_abort_task() we don't want to issue a softreset as it may cause info to be lost in the target disk for the ATA EH autopsy. In this case, just release resources - the disk won't return other I/Os normally after NCQ Error, so this is safe.

---
 drivers/scsi/hisi_sas/hisi_sas.h       |  1 +
 drivers/scsi/hisi_sas/hisi_sas_main.c  | 18 +++++++++-
 drivers/scsi/hisi_sas/hisi_sas_v3_hw.c | 48 ++++++++++++++++++++++++--
 3 files changed, 64 insertions(+), 3 deletions(-)
drivers/scsi/hisi_sas/hisi_sas.h +1 -0
index 9aebf4a26b13..6f8a52a1b808 100644
@@ -104,6 +104,7 @@ enum {
 enum dev_status {
 	HISI_SAS_DEV_INIT,
 	HISI_SAS_DEV_NORMAL,
+	HISI_SAS_DEV_NCQ_ERR,
 };
 
 enum {
drivers/scsi/hisi_sas/hisi_sas_main.c +17 -1
index 8303aa5eaf25..4c37ae9eb6b6 100644
@@ -1604,13 +1604,26 @@ static int hisi_sas_abort_task(struct sas_task *task)
 	} else if (task->task_proto & SAS_PROTOCOL_SATA ||
 		task->task_proto & SAS_PROTOCOL_STP) {
 		if (task->dev->dev_type == SAS_SATA_DEV) {
+			struct ata_queued_cmd *qc = task->uldd_task;
+
 			rc = hisi_sas_internal_task_abort_dev(sas_dev, false);
 			if (rc < 0) {
 				dev_err(dev, "abort task: internal abort failed\n");
 				goto out;
 			}
 			hisi_sas_dereg_device(hisi_hba, device);
-			rc = hisi_sas_softreset_ata_disk(device);
+
+			/*
+			 * If an ATA internal command times out in ATA EH, it
+			 * need to execute soft reset, so check the scsicmd
+			 */
+			if ((sas_dev->dev_status == HISI_SAS_DEV_NCQ_ERR) &&
+			    qc && qc->scsicmd) {
+				hisi_sas_do_release_task(hisi_hba, task, slot);
+				rc = TMF_RESP_FUNC_COMPLETE;
+			} else {
+				rc = hisi_sas_softreset_ata_disk(device);
+			}
 		}
 	} else if (slot && task->task_proto & SAS_PROTOCOL_SMP) {
 		/* SMP */
@@ -1727,6 +1740,9 @@ static int hisi_sas_I_T_nexus_reset(struct domain_device *device)
 	struct device *dev = hisi_hba->dev;
 	int rc;
 
+	if (sas_dev->dev_status == HISI_SAS_DEV_NCQ_ERR)
+		sas_dev->dev_status = HISI_SAS_DEV_NORMAL;
+
 	rc = hisi_sas_internal_task_abort_dev(sas_dev, false);
 	if (rc < 0) {
 		dev_err(dev, "I_T nexus reset: internal abort (%d)\n", rc);
drivers/scsi/hisi_sas/hisi_sas_v3_hw.c +46 -2

80 lines not on this page · raw message

Signature · 1 line
2.35.3
author
#4 lore raw

PATCH v5 4/7 scsi: hisi_sas: Modify v3 HW SATA disk error state completion processing

In reply to John Garry

From: Xingui Yang <[email protected]>

When an NCQ error occurs, the controller will abnormally complete the I/Os that are newly delivered to disk, and bit8 in CQ dw3 will be set which indicates that the SATA disk is in error state. The current processing flow is to set ts->stat to SAS_OPEN_REJECT and then sas_ata_task_done() will set FIS stat to ATA_ERR. After analyzing the IO by ata_eh_analyze_tf(), err_mask will set to AC_ERR_HSM. If media error occurs for four times within 10 minutes and the chip rejects new I/Os for four times, NCQ will be disabled due to excessive errors, which is undesirable.

Therefore, use sas_task_abort() to handle abnormally completed I/Os when SATA disk is in error state, as these abnormally completed I/Os are already processed by sas_ata_device_link_abort() and qc->flag are set to ATA_QCFLAG_FAILED. If sas_task_abort() is used, qc->err_mask will not be modified in EH. Unlike the current process flow, it will not increase the count of ECAT_TOUT_HSM and not turn off NCQ. Like other I/Os on the disk that do not have an error but do not return after the NCQ error, they are retried after the EH.

---
 drivers/scsi/hisi_sas/hisi_sas_v3_hw.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/scsi/hisi_sas/hisi_sas_v3_hw.c +4 -1
index 4278d3482ebb..9d0a54043883 100644
@@ -428,6 +428,8 @@
 #define CMPLT_HDR_DEV_ID_OFF		16
 #define CMPLT_HDR_DEV_ID_MSK		(0xffff << CMPLT_HDR_DEV_ID_OFF)
 /* dw3 */
+#define SATA_DISK_IN_ERROR_STATUS_OFF	8
+#define SATA_DISK_IN_ERROR_STATUS_MSK	(0x1 << SATA_DISK_IN_ERROR_STATUS_OFF)
 #define CMPLT_HDR_SATA_DISK_ERR_OFF	16
 #define CMPLT_HDR_SATA_DISK_ERR_MSK	(0x1 << CMPLT_HDR_SATA_DISK_ERR_OFF)
 #define CMPLT_HDR_IO_IN_TARGET_OFF	17
@@ -2219,7 +2221,8 @@ slot_err_v3_hw(struct hisi_hba *hisi_hba, struct sas_task *task,
 		} else if (dma_rx_err_type & RX_DATA_LEN_UNDERFLOW_MSK) {
 			ts->residual = trans_tx_fail_type;
 			ts->stat = SAS_DATA_UNDERRUN;
-		} else if (dw3 & CMPLT_HDR_IO_IN_TARGET_MSK) {
+		} else if ((dw3 & CMPLT_HDR_IO_IN_TARGET_MSK) ||
+			   (dw3 & SATA_DISK_IN_ERROR_STATUS_MSK)) {
 			ts->stat = SAS_PHY_DOWN;
 			slot->abort = 1;
 		} else {
Signature · 1 line
2.35.3
author
#5 lore raw

PATCH v5 5/7 scsi: pm8001: Modify task abort handling for SATA task

In reply to John Garry

When we try to abort a SATA task, the CCB of the task which we are trying to avoid may still complete. In this case, we should not touch the task associated with that CCB as we can race with libsas freeing the last later in sas_eh_handle_sas_errors() -> sas_eh_finish_cmd() for when TASK_IS_ABORTED is returned from sas_scsi_find_task()

---
 drivers/scsi/pm8001/pm8001_hwi.c | 15 +++++++++++++--
 drivers/scsi/pm8001/pm8001_sas.c |  8 ++++++++
 drivers/scsi/pm8001/pm80xx_hwi.c | 14 ++++++++++----
 3 files changed, 31 insertions(+), 6 deletions(-)
drivers/scsi/pm8001/pm8001_hwi.c +13 -2
index 628b08ba6770..c0adc3a9d196 100644
@@ -2295,7 +2295,9 @@ mpi_sata_completion(struct pm8001_hba_info *pm8001_ha, void *piomb)
 		if (t->dev && (t->dev->lldd_dev))
 			pm8001_dev = t->dev->lldd_dev;
 	} else {
-		pm8001_dbg(pm8001_ha, FAIL, "task null\n");
+		pm8001_dbg(pm8001_ha, FAIL, "task null, freeing CCB tag %d\n",
+			   ccb->ccb_tag);
+		pm8001_ccb_free(pm8001_ha, ccb);
 		return;
 	}
 
@@ -2675,8 +2677,17 @@ static void mpi_sata_event(struct pm8001_hba_info *pm8001_ha, void *piomb)
 	pm8001_dev = ccb->device;
 	if (event)
 		pm8001_dbg(pm8001_ha, FAIL, "sata IO status 0x%x\n", event);
-	if (unlikely(!t || !t->lldd_task || !t->dev))
+
+	if (unlikely(!t)) {
+		pm8001_dbg(pm8001_ha, FAIL, "task null, freeing CCB tag %d\n",
+			   ccb->ccb_tag);
+		pm8001_ccb_free(pm8001_ha, ccb);
 		return;
+	}
+
+	if (unlikely(!t->lldd_task || !t->dev))
+		return;
+
 	ts = &t->task_status;
 	pm8001_dbg(pm8001_ha, DEVIO,
 		   "port_id:0x%x, device_id:0x%x, tag:0x%x, event:0x%x\n",
drivers/scsi/pm8001/pm8001_sas.c +8 -0
index 8e3f2f9ddaac..d5ec29f69be3 100644
@@ -983,6 +983,7 @@ int pm8001_query_task(struct sas_task *task)
 /*  mandatory SAM-3, still need free task/ccb info, abort the specified task */
 int pm8001_abort_task(struct sas_task *task)
 {
+	struct pm8001_ccb_info *ccb = task->lldd_task;
 	unsigned long flags;
 	u32 tag;
 	struct domain_device *dev ;
@@ -1113,6 +1114,13 @@ int pm8001_abort_task(struct sas_task *task)
 				pm8001_dev, DS_OPERATIONAL);
 			wait_for_completion(&completion);
 		} else {
+			/*
+			 * Ensure that if we see a completion for the ccb
+			 * associated with the task which we are trying to
+			 * abort then we should not touch the sas_task as it
+			 * may race with libsas freeing it when return here.
+			 */
+			ccb->task = NULL;
 			ret = sas_execute_internal_abort_single(dev, tag, 0, NULL);
 		}
 		rc = TMF_RESP_FUNC_COMPLETE;
drivers/scsi/pm8001/pm80xx_hwi.c +10 -4
index f8b8624458f7..dd0e06983cd3 100644
@@ -2396,7 +2396,9 @@ mpi_sata_completion(struct pm8001_hba_info *pm8001_ha,
 		if (t->dev && (t->dev->lldd_dev))
 			pm8001_dev = t->dev->lldd_dev;
 	} else {
-		pm8001_dbg(pm8001_ha, FAIL, "task null\n");
+		pm8001_dbg(pm8001_ha, FAIL, "task null, freeing CCB tag %d\n",
+			   ccb->ccb_tag);
+		pm8001_ccb_free(pm8001_ha, ccb);
 		return;
 	}
 
@@ -2813,12 +2815,16 @@ static void mpi_sata_event(struct pm8001_hba_info *pm8001_ha,
 	ccb = &pm8001_ha->ccb_info[tag];
 	t = ccb->task;
 	pm8001_dev = ccb->device;
-
-	if (unlikely(!t || !t->lldd_task || !t->dev)) {
-		pm8001_dbg(pm8001_ha, FAIL, "task or dev null\n");
+	if (unlikely(!t)) {
+		pm8001_dbg(pm8001_ha, FAIL, "task null, freeing CCB tag %d\n",
+			   ccb->ccb_tag);
+		pm8001_ccb_free(pm8001_ha, ccb);
 		return;
 	}
 
+	if (unlikely(!t->lldd_task || !t->dev))
+		return;
+
 	ts = &t->task_status;
 	pm8001_dbg(pm8001_ha, IOERR, "port_id:0x%x, tag:0x%x, event:0x%x\n",
 		   port_id, tag, event);
Signature · 1 line
2.35.3
author
#6 lore raw

PATCH v5 6/7 scsi: pm8001: Use sas_ata_device_link_abort() to handle NCQ errors

In reply to John Garry

In commit c6b9ef5779c3 ("[SCSI] pm80xx: NCQ error handling changes") the driver had support added to handle NCQ errors but much of what is done in this handling is duplicated from the libata EH.

In that named commit we handle in 2x main steps:
a. Issue read log ext10 to examine and clear the errors b. Issue SATA_ABORT all command

Indeed, in libata EH, we do similar to above:
a. ata_do_eh() -> ata_eh_autopsy() -> ata_eh_link_autopsy() ->
ata_eh_analyze_ncq_error() -> ata_eh_read_log_10h()
b. ata_do_eh() -> ata_eh_recover() which will issue a device soft reset
or hard reset

Since there is so much duplication, use sas_ata_device_link_abort() which will abort all pending IOs and kick of ATA EH which will do the steps, above.

However we will not follow the advisory to send the SATA_ABORT all command after the autopsy in read log ext10. Indeed, in libsas EH, we already send a per-task SATA_ABORT command, and this is prior to the ATA EH kicking in and issuing the read log ext10 in the recovery process. I judge that this is ok as the SATA_ABORT command does not actually send any protocol on the link to abort IO on the other side, so would not change any state on the disk (for the read log ext10 command).

---
 drivers/scsi/pm8001/pm8001_hwi.c | 171 +++----------------------------
 drivers/scsi/pm8001/pm8001_sas.h |   4 -
 drivers/scsi/pm8001/pm80xx_hwi.c | 163 ++---------------------------
 3 files changed, 19 insertions(+), 319 deletions(-)
drivers/scsi/pm8001/pm8001_hwi.c +13 -158
index c0adc3a9d196..ec1a9ab61814 100644
@@ -1724,7 +1724,14 @@ void pm8001_work_fn(struct work_struct *work)
 				pm8001_free_dev(pm8001_dev);
 			}
 		}
-	}	break;
+	}
+	break;
+	case IO_XFER_ERROR_ABORTED_NCQ_MODE:
+	{
+		dev = pm8001_dev->sas_device;
+		sas_ata_device_link_abort(dev, false);
+	}
+	break;
 	}
 	kfree(pw);
 }
@@ -1748,110 +1755,6 @@ int pm8001_handle_event(struct pm8001_hba_info *pm8001_ha, void *data,
 	return ret;
 }
 
-static void pm8001_send_abort_all(struct pm8001_hba_info *pm8001_ha,
-		struct pm8001_device *pm8001_ha_dev)
-{
-	struct pm8001_ccb_info *ccb;
-	struct sas_task *task;
-	struct task_abort_req task_abort;
-	u32 opc = OPC_INB_SATA_ABORT;
-	int ret;
-
-	pm8001_ha_dev->id |= NCQ_ABORT_ALL_FLAG;
-	pm8001_ha_dev->id &= ~NCQ_READ_LOG_FLAG;
-
-	task = sas_alloc_slow_task(GFP_ATOMIC);
-	if (!task) {
-		pm8001_dbg(pm8001_ha, FAIL, "cannot allocate task\n");
-		return;
-	}
-
-	task->task_done = pm8001_task_done;
-
-	ccb = pm8001_ccb_alloc(pm8001_ha, pm8001_ha_dev, task);
-	if (!ccb) {
-		sas_free_task(task);
-		return;
-	}
-
-	memset(&task_abort, 0, sizeof(task_abort));
-	task_abort.abort_all = cpu_to_le32(1);
-	task_abort.device_id = cpu_to_le32(pm8001_ha_dev->device_id);
-	task_abort.tag = cpu_to_le32(ccb->ccb_tag);
-
-	ret = pm8001_mpi_build_cmd(pm8001_ha, 0, opc, &task_abort,
-				   sizeof(task_abort), 0);
-	if (ret) {
-		sas_free_task(task);
-		pm8001_ccb_free(pm8001_ha, ccb);
-	}
-}
-
-static void pm8001_send_read_log(struct pm8001_hba_info *pm8001_ha,
-		struct pm8001_device *pm8001_ha_dev)
-{
-	struct sata_start_req sata_cmd;
-	int res;
-	struct pm8001_ccb_info *ccb;
-	struct sas_task *task = NULL;
-	struct host_to_dev_fis fis;
-	struct domain_device *dev;
-	u32 opc = OPC_INB_SATA_HOST_OPSTART;
-
-	task = sas_alloc_slow_task(GFP_ATOMIC);
-	if (!task) {
-		pm8001_dbg(pm8001_ha, FAIL, "cannot allocate task !!!\n");
-		return;
-	}
-	task->task_done = pm8001_task_done;
-
-	/*
-	 * Allocate domain device by ourselves as libsas is not going to
-	 * provide any.
-	 */
-	dev = kzalloc(sizeof(struct domain_device), GFP_ATOMIC);
-	if (!dev) {
-		sas_free_task(task);
-		pm8001_dbg(pm8001_ha, FAIL,
-			   "Domain device cannot be allocated\n");
-		return;
-	}
-	task->dev = dev;
-	task->dev->lldd_dev = pm8001_ha_dev;
-
-	ccb = pm8001_ccb_alloc(pm8001_ha, pm8001_ha_dev, task);
-	if (!ccb) {
-		sas_free_task(task);
-		kfree(dev);
-		return;
-	}
-
-	pm8001_ha_dev->id |= NCQ_READ_LOG_FLAG;
-	pm8001_ha_dev->id |= NCQ_2ND_RLE_FLAG;
-
-	/* construct read log FIS */
-	memset(&fis, 0, sizeof(struct host_to_dev_fis));
-	fis.fis_type = 0x27;
-	fis.flags = 0x80;
-	fis.command = ATA_CMD_READ_LOG_EXT;
-	fis.lbal = 0x10;
-	fis.sector_count = 0x1;
-
-	memset(&sata_cmd, 0, sizeof(sata_cmd));
-	sata_cmd.tag = cpu_to_le32(ccb->ccb_tag);
-	sata_cmd.device_id = cpu_to_le32(pm8001_ha_dev->device_id);
-	sata_cmd.ncqtag_atap_dir_m = cpu_to_le32((0x1 << 7) | (0x5 << 9));
-	memcpy(&sata_cmd.sata_fis, &fis, sizeof(struct host_to_dev_fis));
-
-	res = pm8001_mpi_build_cmd(pm8001_ha, 0, opc, &sata_cmd,
-				   sizeof(sata_cmd), 0);
-	if (res) {
-		sas_free_task(task);
-		pm8001_ccb_free(pm8001_ha, ccb);
-		kfree(dev);
-	}
-}
-
 /**
  * mpi_ssp_completion- process the event that FW response to the SSP request.
  * @pm8001_ha: our hba card information
@@ -2301,8 +2204,7 @@ mpi_sata_completion(struct pm8001_hba_info *pm8001_ha, void *piomb)
 		return;
 	}
 
-	if ((pm8001_dev && !(pm8001_dev->id & NCQ_READ_LOG_FLAG))
-		&& unlikely(!t || !t->lldd_task || !t->dev)) {
+	if (pm8001_dev && unlikely(!t || !t->lldd_task || !t->dev)) {
 		pm8001_dbg(pm8001_ha, FAIL, "task or dev null\n");
 		return;
 	}
@@ -2360,15 +2262,6 @@ mpi_sata_completion(struct pm8001_hba_info *pm8001_ha, void *piomb)
 		if (param == 0) {
 			ts->resp = SAS_TASK_COMPLETE;
 			ts->stat = SAS_SAM_STAT_GOOD;
-			/* check if response is for SEND READ LOG */
-			if (pm8001_dev &&
-			    (pm8001_dev->id & NCQ_READ_LOG_FLAG)) {
-				pm8001_send_abort_all(pm8001_ha, pm8001_dev);
-				/* Free the tag */
-				pm8001_tag_free(pm8001_ha, tag);
-				sas_free_task(t);
-				return;
-			}
 		} else {
 			u8 len;
 			ts->resp = SAS_TASK_COMPLETE;
@@ -2666,9 +2559,10 @@ static void mpi_sata_event(struct pm8001_hba_info *pm8001_ha, void *piomb)
 	if (event == IO_XFER_ERROR_ABORTED_NCQ_MODE) {
 		/* find device using device id */
 		pm8001_dev = pm8001_find_dev(pm8001_ha, dev_id);
-		/* send read log extension */
 		if (pm8001_dev)
-			pm8001_send_read_log(pm8001_ha, pm8001_dev);
+			pm8001_handle_event(pm8001_ha,
+				pm8001_dev,
+				IO_XFER_ERROR_ABORTED_NCQ_MODE);
 		return;
 	}
 
@@ -3649,12 +3543,7 @@ int pm8001_mpi_task_abort_resp(struct pm8001_hba_info *pm8001_ha, void *piomb)
 	pm8001_ccb_task_free(pm8001_ha, ccb);
 	mb();
 
-	if (pm8001_dev->id & NCQ_ABORT_ALL_FLAG) {
-		sas_free_task(t);
-		pm8001_dev->id &= ~NCQ_ABORT_ALL_FLAG;
-	} else {
-		t->task_done(t);
-	}
+	t->task_done(t);
 
 	return 0;
 }
@@ -4206,7 +4095,6 @@ static int pm8001_chip_sata_req(struct pm8001_hba_info *pm8001_ha,
 	u64 phys_addr;
 	u32 ATAP = 0x0;
 	u32 dir;
-	unsigned long flags;
 	u32  opc = OPC_INB_SATA_HOST_OPSTART;
 
 	memset(&sata_cmd, 0, sizeof(sata_cmd));
@@ -4261,39 +4149,6 @@ static int pm8001_chip_sata_req(struct pm8001_hba_info *pm8001_ha,
 		sata_cmd.esgl = 0;
 	}
 
-	/* Check for read log for failed drive and return */
-	if (sata_cmd.sata_fis.command == 0x2f) {
-		if (((pm8001_ha_dev->id & NCQ_READ_LOG_FLAG) ||
-			(pm8001_ha_dev->id & NCQ_ABORT_ALL_FLAG) ||
-			(pm8001_ha_dev->id & NCQ_2ND_RLE_FLAG))) {
-			struct task_status_struct *ts;
-
-			pm8001_ha_dev->id &= 0xDFFFFFFF;
-			ts = &task->task_status;
-
-			spin_lock_irqsave(&task->task_state_lock, flags);
-			ts->resp = SAS_TASK_COMPLETE;
-			ts->stat = SAS_SAM_STAT_GOOD;
-			task->task_state_flags &= ~SAS_TASK_STATE_PENDING;
-			task->task_state_flags |= SAS_TASK_STATE_DONE;
-			if (unlikely((task->task_state_flags &
-					SAS_TASK_STATE_ABORTED))) {
-				spin_unlock_irqrestore(&task->task_state_lock,
-							flags);
-				pm8001_dbg(pm8001_ha, FAIL,
-					   "task 0x%p resp 0x%x  stat 0x%x but aborted by upper layer\n",
-					   task, ts->resp,
-					   ts->stat);
-				pm8001_ccb_task_free(pm8001_ha, ccb);
-			} else {
-				spin_unlock_irqrestore(&task->task_state_lock,
-							flags);
-				pm8001_ccb_task_free_done(pm8001_ha, ccb);
-				return 0;
-			}
-		}
-	}
-
 	return pm8001_mpi_build_cmd(pm8001_ha, 0, opc, &sata_cmd,
 				    sizeof(sata_cmd), 0);
 }
drivers/scsi/pm8001/pm8001_sas.h +0 -4
index b08f52673889..8ab0654327f9 100644
@@ -579,10 +579,6 @@ struct pm8001_fw_image_header {
 #define FLASH_UPDATE_DNLD_NOT_SUPPORTED		0x10
 #define FLASH_UPDATE_DISABLED			0x11
 
-#define	NCQ_READ_LOG_FLAG			0x80000000
-#define	NCQ_ABORT_ALL_FLAG			0x40000000
-#define	NCQ_2ND_RLE_FLAG			0x20000000
-
 /* Device states */
 #define DS_OPERATIONAL				0x01
 #define DS_PORT_IN_RESET			0x02
drivers/scsi/pm8001/pm80xx_hwi.c +6 -157

201 lines not on this page · raw message

Signature · 1 line
2.35.3
author
#7 lore raw

PATCH v5 7/7 scsi: libsas: Make sas_{alloc, alloc_slow, free}_task() private

In reply to John Garry

We have no users outside libsas any longer, so make sas_alloc_task(), sas_alloc_slow_task(), and sas_free_task() private.

---
 drivers/scsi/libsas/sas_init.c     | 3 ---
 drivers/scsi/libsas/sas_internal.h | 4 ++++
 include/scsi/libsas.h              | 4 ----
 3 files changed, 4 insertions(+), 7 deletions(-)
drivers/scsi/libsas/sas_init.c +0 -3
index e4f77072a58d..f2c05ebeb72f 100644
@@ -35,7 +35,6 @@ struct sas_task *sas_alloc_task(gfp_t flags)
 
 	return task;
 }
-EXPORT_SYMBOL_GPL(sas_alloc_task);
 
 struct sas_task *sas_alloc_slow_task(gfp_t flags)
 {
@@ -56,7 +55,6 @@ struct sas_task *sas_alloc_slow_task(gfp_t flags)
 
 	return task;
 }
-EXPORT_SYMBOL_GPL(sas_alloc_slow_task);
 
 void sas_free_task(struct sas_task *task)
 {
@@ -65,7 +63,6 @@ void sas_free_task(struct sas_task *task)
 		kmem_cache_free(sas_task_cache, task);
 	}
 }
-EXPORT_SYMBOL_GPL(sas_free_task);
 
 /*------------ SAS addr hash -----------*/
 void sas_hash_addr(u8 *hashed, const u8 *sas_addr)
drivers/scsi/libsas/sas_internal.h +4 -0
index 8d0ad3abc7b5..b54bcf3c9a9d 100644
@@ -52,6 +52,10 @@ void sas_unregister_phys(struct sas_ha_struct *sas_ha);
 struct asd_sas_event *sas_alloc_event(struct asd_sas_phy *phy, gfp_t gfp_flags);
 void sas_free_event(struct asd_sas_event *event);
 
+struct sas_task *sas_alloc_task(gfp_t flags);
+struct sas_task *sas_alloc_slow_task(gfp_t flags);
+void sas_free_task(struct sas_task *task);
+
 int  sas_register_ports(struct sas_ha_struct *sas_ha);
 void sas_unregister_ports(struct sas_ha_struct *sas_ha);
 
include/scsi/libsas.h +0 -4
index 2dbead74a2af..f86b56bf7833 100644
@@ -639,10 +639,6 @@ struct sas_task_slow {
 #define SAS_TASK_STATE_ABORTED      4
 #define SAS_TASK_NEED_DEV_RESET     8
 
-extern struct sas_task *sas_alloc_task(gfp_t flags);
-extern struct sas_task *sas_alloc_slow_task(gfp_t flags);
-extern void sas_free_task(struct sas_task *task);
-
 static inline bool sas_is_internal_abort(struct sas_task *task)
 {
 	return task->task_proto == SAS_PROTOCOL_INTERNAL_ABORT;
Signature · 1 line
2.35.3
Tested-by on the v5 series
#8 lore raw
In reply to John Garry
Quoted 57 lines · John Garry

As reported in [0], the pm8001 driver NCQ error handling more or less duplicates what libata does in link error handling, as follows:
- abort all commands
- do autopsy with read log ext 10 command
- reset the target to recover, if necessary

Indeed for the hisi_sas driver we want to add similar handling for NCQ errors.

This series add a new libsas API - sas_ata_device_link_abort() - to handle host NCQ errors, and fixes up pm8001 and hisi_sas drivers to use it.

A difference in the pm8001 driver NCQ error handling is that we send SATA_ABORT per-task prior to read log ext10, but I feel that this should not make a difference to the error handling.

Damien kindly tested previous the series for pm8001, but any further pm8001 testing would be appreciated as I have since tweaked pm8001 handling again. This is because the pm8001 driver hangs on my arm64 machine read log ext10 command.

Finally with these changes we can make the libsas task alloc/free APIs private, which they should always have been.

Based on mkp-scsi @ 6.1/scsi-staging 57569c37f0ad ("scsi: iscsi:
iscsi_tcp: Fix null-ptr-deref while calling getpeername()")

[0] https://lore.kernel.org/linux-scsi/[email protected]/

Changes since v4:
- Add Jason's tags (thanks)
- Rebase

Changes since v3:
- Add Damien's tags (thanks)
- Modify hisi_sas processing as follows:
- use sas_task_abort() for rejected IO
- Modify abort task processing to issue softreset in certain circumstances
- rebase

Changes since v2:
- Stop sending SATA_ABORT all for pm8001 handling
- Make "reset" optional in sas_ata_device_link_abort()
- Drop Jack's ACK

John Garry (5):
scsi: libsas: Add sas_ata_device_link_abort()
scsi: hisi_sas: Move slot variable definition in hisi_sas_abort_task()
scsi: pm8001: Modify task abort handling for SATA task
scsi: pm8001: Use sas_ata_device_link_abort() to handle NCQ errors
scsi: libsas: Make sas_{alloc, alloc_slow, free}_task() private

Xingui Yang (2):
scsi: hisi_sas: Add SATA_DISK_ERR bit handling for v3 hw
scsi: hisi_sas: Modify v3 HW SATA disk error state completion
processing

Quoted 17 lines
 drivers/scsi/hisi_sas/hisi_sas.h       |   1 +
 drivers/scsi/hisi_sas/hisi_sas_main.c  |  26 +++-
 drivers/scsi/hisi_sas/hisi_sas_v3_hw.c |  53 ++++++-
 drivers/scsi/libsas/sas_ata.c          |  12 ++
 drivers/scsi/libsas/sas_init.c         |   3 -
 drivers/scsi/libsas/sas_internal.h     |   4 +
 drivers/scsi/pm8001/pm8001_hwi.c       | 186 ++++---------------------
 drivers/scsi/pm8001/pm8001_sas.c       |   8 ++
 drivers/scsi/pm8001/pm8001_sas.h       |   4 -
 drivers/scsi/pm8001/pm80xx_hwi.c       | 177 +++--------------------
 include/scsi/libsas.h                  |   4 -
 include/scsi/sas_ata.h                 |   6 +
 12 files changed, 143 insertions(+), 341 deletions(-)
Signature · 1 line
2.35.3

For pm80xx (pm8001 changes untested):

Notes unrelated to this patch:

Both before and after this series, this driver prints:

[  215.845053] ata21.00: exception Emask 0x0 SAct 0xfc0000 SErr 0x0 action 0x6
[  215.852308] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.857801] ata21.00: cmd 61/00:00:00:3a:d3/01:00:b3:04:00/40 tag 18 ncq dma 131072 out
                        res 43/04:00:ff:3a:d3/00:00:b3:04:00/40 Emask 0x400 (NCQ error) <F>
[  215.874396] ata21.00: status: { DRDY SENSE ERR }
[  215.879192] ata21.00: error: { ABRT }
[  215.882997] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.888479] ata21.00: cmd 61/00:00:00:3b:d3/01:00:b3:04:00/40 tag 19 ncq dma 131072 out
                        res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.904814] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.910311] ata21.00: cmd 61/00:00:00:3c:d3/01:00:b3:04:00/40 tag 20 ncq dma 131072 out
                        res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.932679] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.941203] ata21.00: cmd 61/00:00:00:3d:d3/01:00:b3:04:00/40 tag 21 ncq dma 131072 out
                        res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.963616] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.972150] ata21.00: cmd 61/00:00:00:3e:d3/01:00:b3:04:00/40 tag 22 ncq dma 131072 out
                        res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.994532] ata21.00: failed command: WRITE FPDMA QUEUED
[  216.003124] ata21.00: cmd 61/00:00:00:3f:d3/01:00:b3:04:00/40 tag 23 ncq dma 131072 out
                        res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)

HSM (Host State Machine) violation errors.

For the same SATA drive connected via AHCI this will instead give:

[ 3796.944923] ata14.00: exception Emask 0x0 SAct 0x80800003 SErr 0xc0000 action 0x0
[ 3796.959375] ata14.00: irq_stat 0x40000008
[ 3796.970140] ata14: SError: { CommWake 10B8B }
[ 3796.981231] ata14.00: failed command: WRITE FPDMA QUEUED
[ 3796.993237] ata14.00: cmd 61/00:08:00:7e:73/02:00:8e:08:00/40 tag 1 ncq dma 262144 out
                        res 43/04:01:00:00:00/00:00:00:00:00/40 Emask 0x1 (device error)
[ 3797.017984] ata14.00: status: { DRDY SENSE ERR }
[ 3797.026833] ata14.00: error: { ABRT }
[ 3797.034664] ata14.00: failed command: WRITE FPDMA QUEUED
[ 3797.043015] ata14.00: cmd 61/00:b8:00:60:73/0a:00:8e:08:00/40 tag 23 ncq dma 1310720 out
                        res 43/04:00:df:67:73/00:00:8e:08:00/40 Emask 0x400 (NCQ error) <F>
[ 3797.065224] ata14.00: status: { DRDY SENSE ERR }
[ 3797.072914] ata14.00: error: { ABRT }
[ 3797.079598] ata14.00: failed command: WRITE FPDMA QUEUED
[ 3797.087920] ata14.00: cmd 61/00:f8:00:6a:73/0a:00:8e:08:00/40 tag 31 ncq dma 1310720 out
                        res 43/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device error)
[ 3797.109800] ata14.00: status: { DRDY SENSE ERR }
[ 3797.117451] ata14.00: error: { ABRT }

device error errors.

Except for the I/O that caused the NCQ error, the remaining outstanding I/Os, regardless if they were aborted by the drive, as a side-effect of reading the NCQ error log (see 13.7.4 Queued Error Log (10h) in SATA 3.5a spec), or if they were aborted by the host (by sas_ata_device_link_abort()), I don't think it is correct to report these as HSM violation errors.

HSM violation errors are e.g. when you try to issue a command to a drive that has ATA_BUSY bit set.

Kind regards,
Niklas

author
#9 lore raw
Quoted 6 lines · Niklas Cassel
Quoted 2 lines

2.35.3

For pm80xx (pm8001 changes untested):

Thanks!

Quoted 52 lines

Notes unrelated to this patch:

Both before and after this series, this driver prints:

[  215.845053] ata21.00: exception Emask 0x0 SAct 0xfc0000 SErr 0x0 action 0x6
[  215.852308] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.857801] ata21.00: cmd 61/00:00:00:3a:d3/01:00:b3:04:00/40 tag 18 ncq dma 131072 out
                         res 43/04:00:ff:3a:d3/00:00:b3:04:00/40 Emask 0x400 (NCQ error) <F>
[  215.874396] ata21.00: status: { DRDY SENSE ERR }
[  215.879192] ata21.00: error: { ABRT }
[  215.882997] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.888479] ata21.00: cmd 61/00:00:00:3b:d3/01:00:b3:04:00/40 tag 19 ncq dma 131072 out
                         res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.904814] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.910311] ata21.00: cmd 61/00:00:00:3c:d3/01:00:b3:04:00/40 tag 20 ncq dma 131072 out
                         res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.932679] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.941203] ata21.00: cmd 61/00:00:00:3d:d3/01:00:b3:04:00/40 tag 21 ncq dma 131072 out
                         res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.963616] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.972150] ata21.00: cmd 61/00:00:00:3e:d3/01:00:b3:04:00/40 tag 22 ncq dma 131072 out
                         res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.994532] ata21.00: failed command: WRITE FPDMA QUEUED
[  216.003124] ata21.00: cmd 61/00:00:00:3f:d3/01:00:b3:04:00/40 tag 23 ncq dma 131072 out
                         res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)

HSM (Host State Machine) violation errors.

For the same SATA drive connected via AHCI this will instead give:

[ 3796.944923] ata14.00: exception Emask 0x0 SAct 0x80800003 SErr 0xc0000 action 0x0
[ 3796.959375] ata14.00: irq_stat 0x40000008
[ 3796.970140] ata14: SError: { CommWake 10B8B }
[ 3796.981231] ata14.00: failed command: WRITE FPDMA QUEUED
[ 3796.993237] ata14.00: cmd 61/00:08:00:7e:73/02:00:8e:08:00/40 tag 1 ncq dma 262144 out
                         res 43/04:01:00:00:00/00:00:00:00:00/40 Emask 0x1 (device error)
[ 3797.017984] ata14.00: status: { DRDY SENSE ERR }
[ 3797.026833] ata14.00: error: { ABRT }
[ 3797.034664] ata14.00: failed command: WRITE FPDMA QUEUED
[ 3797.043015] ata14.00: cmd 61/00:b8:00:60:73/0a:00:8e:08:00/40 tag 23 ncq dma 1310720 out
                         res 43/04:00:df:67:73/00:00:8e:08:00/40 Emask 0x400 (NCQ error) <F>
[ 3797.065224] ata14.00: status: { DRDY SENSE ERR }
[ 3797.072914] ata14.00: error: { ABRT }
[ 3797.079598] ata14.00: failed command: WRITE FPDMA QUEUED
[ 3797.087920] ata14.00: cmd 61/00:f8:00:6a:73/0a:00:8e:08:00/40 tag 31 ncq dma 1310720 out
                         res 43/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device error)
[ 3797.109800] ata14.00: status: { DRDY SENSE ERR }
[ 3797.117451] ata14.00: error: { ABRT }

device error errors.

Quoted 8 lines

Except for the I/O that caused the NCQ error, the remaining outstanding I/Os, regardless if they were aborted by the drive, as a side-effect of reading the NCQ error log (see 13.7.4 Queued Error Log (10h) in SATA 3.5a spec), or if they were aborted by the host (by sas_ata_device_link_abort()), I don't think it is correct to report these as HSM violation errors.

HSM violation errors are e.g. when you try to issue a command to a drive that has ATA_BUSY bit set.

We had a similar issue for hisi_sas and solved in patch 4/7: don't set ATA_ERR in the fis for those IO which complete with error, but abort the IO via sas_abort_task().

For pm80xx the IO is either rejected (actually completes with rejection) or is aborted via internal abort command. Maybe we can do similar for pm8001 as we allow the IO to complete in both cases with error. I'll check.

Thanks,
John

author
#10 lore raw
Quoted 59 lines · John Garry
Quoted 51 lines

Notes unrelated to this patch:

Both before and after this series, this driver prints:

[  215.845053] ata21.00: exception Emask 0x0 SAct 0xfc0000 SErr 0x0 action 0x6
[  215.852308] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.857801] ata21.00: cmd 61/00:00:00:3a:d3/01:00:b3:04:00/40 tag 18 ncq dma 131072 out
                          res 43/04:00:ff:3a:d3/00:00:b3:04:00/40 Emask 0x400 (NCQ error) <F>
[  215.874396] ata21.00: status: { DRDY SENSE ERR }
[  215.879192] ata21.00: error: { ABRT }
[  215.882997] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.888479] ata21.00: cmd 61/00:00:00:3b:d3/01:00:b3:04:00/40 tag 19 ncq dma 131072 out
                          res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.904814] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.910311] ata21.00: cmd 61/00:00:00:3c:d3/01:00:b3:04:00/40 tag 20 ncq dma 131072 out
                          res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.932679] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.941203] ata21.00: cmd 61/00:00:00:3d:d3/01:00:b3:04:00/40 tag 21 ncq dma 131072 out
                          res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.963616] ata21.00: failed command: WRITE FPDMA QUEUED
[  215.972150] ata21.00: cmd 61/00:00:00:3e:d3/01:00:b3:04:00/40 tag 22 ncq dma 131072 out
                          res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)
[  215.994532] ata21.00: failed command: WRITE FPDMA QUEUED
[  216.003124] ata21.00: cmd 61/00:00:00:3f:d3/01:00:b3:04:00/40 tag 23 ncq dma 131072 out
                          res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x2 (HSM violation)

HSM (Host State Machine) violation errors.

For the same SATA drive connected via AHCI this will instead give:

[ 3796.944923] ata14.00: exception Emask 0x0 SAct 0x80800003 SErr 0xc0000 action 0x0
[ 3796.959375] ata14.00: irq_stat 0x40000008
[ 3796.970140] ata14: SError: { CommWake 10B8B }
[ 3796.981231] ata14.00: failed command: WRITE FPDMA QUEUED
[ 3796.993237] ata14.00: cmd 61/00:08:00:7e:73/02:00:8e:08:00/40 tag 1 ncq dma 262144 out
                          res 43/04:01:00:00:00/00:00:00:00:00/40 Emask 0x1 (device error)
[ 3797.017984] ata14.00: status: { DRDY SENSE ERR }
[ 3797.026833] ata14.00: error: { ABRT }
[ 3797.034664] ata14.00: failed command: WRITE FPDMA QUEUED
[ 3797.043015] ata14.00: cmd 61/00:b8:00:60:73/0a:00:8e:08:00/40 tag 23 ncq dma 1310720 out
                          res 43/04:00:df:67:73/00:00:8e:08:00/40 Emask 0x400 (NCQ error) <F>
[ 3797.065224] ata14.00: status: { DRDY SENSE ERR }
[ 3797.072914] ata14.00: error: { ABRT }
[ 3797.079598] ata14.00: failed command: WRITE FPDMA QUEUED
[ 3797.087920] ata14.00: cmd 61/00:f8:00:6a:73/0a:00:8e:08:00/40 tag 31 ncq dma 1310720 out
                          res 43/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device error)
[ 3797.109800] ata14.00: status: { DRDY SENSE ERR }
[ 3797.117451] ata14.00: error: { ABRT }

device error errors.

Quoted 8 lines

Except for the I/O that caused the NCQ error, the remaining outstanding I/Os, regardless if they were aborted by the drive, as a side-effect of reading the NCQ error log (see 13.7.4 Queued Error Log (10h) in SATA 3.5a spec), or if they were aborted by the host (by sas_ata_device_link_abort()), I don't think it is correct to report these as HSM violation errors.

HSM violation errors are e.g. when you try to issue a command to a drive that has ATA_BUSY bit set.

Quoted 7 lines

We had a similar issue for hisi_sas and solved in patch 4/7: don't set ATA_ERR in the fis for those IO which complete with error, but abort the IO via sas_abort_task().

For pm80xx the IO is either rejected (actually completes with rejection) or is aborted via internal abort command. Maybe we can do similar for pm8001 as we allow the IO to complete in both cases with error. I'll check.

Hi Niklas,

Could you try a change like this on top:

void sas_ata_device_link_abort(struct domain_device *device, bool force_reset)

{
	struct ata_port *ap = device->sata_dev.ap;
	struct ata_link *link = &ap->link;

+ device->sata_dev.fis[2] = ATA_ERR | ATA_DRDY;
+ device->sata_dev.fis[3] = 0x04;

	link->eh_info.err_mask |= AC_ERR_DEV;
	if (force_reset)
		link->eh_info.action |= ATA_EH_RESET;
	ata_link_abort(link);
}

EXPORT_SYMBOL_GPL(sas_ata_device_link_abort);

I tried it myself and it looked to work ok, except I have a problem with my arm64 system in that the read log ext times-out and all TF show "device error", like:

[  350.257870] ata1.00: qc timeout (cmd 0x47)
[  350.262054] pm80xx0:: mpi_sata_completion 2293: task null, freeing CCB tag 2
[  350.269128] ata1.00: Read log 0x10 page 0x00 failed, Emask 0x40
[  350.281581] ata1: failed to read log page 10h (errno=-5)
[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen
[  350.584836] ata1.00: failed command: READ FPDMA QUEUED
[  350.589970] ata1.00: cmd 60/00:00:80:26:00/01:00:00:00:00/40 tag 0 ncq dma 131072 in
         res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device 
error)
[  350.605533] ata1.00: status: { DRDY ERR }
[  350.609541] ata1.00: error: { ABRT }
[  350.613115] ata1.00: failed command: READ FPDMA QUEUED
[  350.618248] ata1.00: cmd 60/00:00:80:26:00/01:00:00:00:00/40 tag 1 ncq dma 131072 in
         res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device 
error)
[  350.633809] ata1.00: status: { DRDY ERR
[  350.637813] ata1.00: error: { ABRT }
[  350.641384] ata1.00: failed command: READ FPDMA QUEUED
[  350.646515] ata1.00: cmd 60/00:00:80:26:00/01:00:00:00:00/40 tag 2 ncq dma 131072 in
         res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device 
error)
[  350.662076] ata1.00: status: { DRDY ERR
[  350.666080] ata1.00: error: { ABRT }
[  350.669652] ata1.00: failed command: READ FPDMA QUEUED
[  350.674784] ata1.00: cmd 60/00:00:d8:26:00/01:00:00:00:00/40 tag 3 ncq dma 131072 in
         res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device 
error)
[  350.690344] ata1.00: status: { DRDY ERR
[  350.694348] ata1.00: error: { ABRT }
[  350.697919] ata1.00: failed command: READ FPDMA QUEUED
[  350.703051] ata1.00: cmd 60/00:00:e0:26:00/01:00:00:00:00/40 tag 4 ncq dma 131072 in
         res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device 
error)
[  350.718612] ata1.00: status: { DRDY ERR
[  350.722623] ata1.00: error: { ABRT }
[  350.726196] ata1.00: failed command: READ FPDMA QUEUED
[  350.731329] ata1.00: cmd 60/00:00:c8:26:00/01:00:00:00:00/40 tag 5 ncq dma 131072 in
         res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device 

error)

...

Thanks,
John

#11 lore raw
Quoted 20 lines · John Garry

On 04/10/2022 15:04, John Garry wrote:

Hi Niklas,

Could you try a change like this on top:

void sas_ata_device_link_abort(struct domain_device *device, bool force_reset)

{
	struct ata_port *ap = device->sata_dev.ap;
	struct ata_link *link = &ap->link;
(quoted diff) +2 -0
+	device->sata_dev.fis[2] = ATA_ERR | ATA_DRDY;
+	device->sata_dev.fis[3] = 0x04;
	link->eh_info.err_mask |= AC_ERR_DEV;
	if (force_reset)
		link->eh_info.action |= ATA_EH_RESET;
	ata_link_abort(link);
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(sas_ata_device_link_abort);

I tried it myself and it looked to work ok, except I have a problem with my arm64 system in that the read log ext times-out and all TF show "device error", like:

Do you know why it fails to read the log?
Can you read the NCQ Command Error log using ATA16 passthrough commands?

sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc

The first byte is the last NCQ tag (in hex) that failed.

I tried your patch, and it looks good:

[ 6656.228131] ata5.00: exception Emask 0x0 SAct 0x460000 SErr 0x0 action 0x0
[ 6656.252759] ata5.00: failed command: WRITE FPDMA QUEUED
[ 6656.271554] ata5.00: cmd 61/00:00:00:d8:8a/04:00:ce:03:00/40 tag 17 ncq dma 524288 out
                        res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device error)
[ 6656.309308] ata5.00: status: { DRDY ERR }
[ 6656.316403] ata5.00: error: { ABRT }
[ 6656.322300] ata5.00: failed command: WRITE FPDMA QUEUED
[ 6656.330871] ata5.00: cmd 61/00:00:00:dc:8a/04:00:ce:03:00/40 tag 18 ncq dma 524288 out
                        res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device error)
[ 6656.356295] ata5.00: status: { DRDY ERR }
[ 6656.362931] ata5.00: error: { ABRT }
[ 6656.368897] ata5.00: failed command: WRITE FPDMA QUEUED
[ 6656.377471] ata5.00: cmd 61/00:00:00:d4:8a/04:00:ce:03:00/40 tag 22 ncq dma 524288 out
                        res 43/04:00:ff:d7:8a/00:00:ce:03:00/40 Emask 0x400 (NCQ error) <F>
[ 6656.403149] ata5.00: status: { DRDY SENSE ERR }
[ 6656.410624] ata5.00: error: { ABRT }

However, since this is a change from the existing behavior of this driver, this could go as a separate patch, and does not need to delay this series.

I also think that we should do a similar patch for sas_ata_task_done():

drivers/scsi/libsas/sas_ata.c +1 -1
index d35c9296f738..648d0693ceee 100644
@@ -140,7 +140,7 @@ static void sas_ata_task_done(struct sas_task *task)
                        }
 
                        dev->sata_dev.fis[3] = 0x04; /* status err */
-                       dev->sata_dev.fis[2] = ATA_ERR;
+                       dev->sata_dev.fis[2] = ATA_ERR | ATA_DRDY;
                }
        }
 

To avoid all SAS errors from being reported as HSM errors.

Kind regards,
Niklas

#12 lore raw
Quoted 30 lines · Niklas Cassel
Quoted 20 lines · John Garry

On 04/10/2022 15:04, John Garry wrote:

Hi Niklas,

Could you try a change like this on top:

void sas_ata_device_link_abort(struct domain_device *device, bool force_reset)

{
	struct ata_port *ap = device->sata_dev.ap;
	struct ata_link *link = &ap->link;
(quoted diff) +2 -0
+	device->sata_dev.fis[2] = ATA_ERR | ATA_DRDY;
+	device->sata_dev.fis[3] = 0x04;
	link->eh_info.err_mask |= AC_ERR_DEV;
	if (force_reset)
		link->eh_info.action |= ATA_EH_RESET;
	ata_link_abort(link);
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(sas_ata_device_link_abort);

I tried it myself and it looked to work ok, except I have a problem with my arm64 system in that the read log ext times-out and all TF show "device error", like:

Do you know why it fails to read the log?
Can you read the NCQ Command Error log using ATA16 passthrough commands?

Quoted 3 lines

sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc

The first byte is the last NCQ tag (in hex) that failed.

libata issues read log as a non-ncq command under EH. So the NCQ error log will not help.

Quoted 46 lines

I tried your patch, and it looks good:

[ 6656.228131] ata5.00: exception Emask 0x0 SAct 0x460000 SErr 0x0 action 0x0
[ 6656.252759] ata5.00: failed command: WRITE FPDMA QUEUED
[ 6656.271554] ata5.00: cmd 61/00:00:00:d8:8a/04:00:ce:03:00/40 tag 17 ncq dma 524288 out
                        res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device error)
[ 6656.309308] ata5.00: status: { DRDY ERR }
[ 6656.316403] ata5.00: error: { ABRT }
[ 6656.322300] ata5.00: failed command: WRITE FPDMA QUEUED
[ 6656.330871] ata5.00: cmd 61/00:00:00:dc:8a/04:00:ce:03:00/40 tag 18 ncq dma 524288 out
                        res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device error)
[ 6656.356295] ata5.00: status: { DRDY ERR }
[ 6656.362931] ata5.00: error: { ABRT }
[ 6656.368897] ata5.00: failed command: WRITE FPDMA QUEUED
[ 6656.377471] ata5.00: cmd 61/00:00:00:d4:8a/04:00:ce:03:00/40 tag 22 ncq dma 524288 out
                        res 43/04:00:ff:d7:8a/00:00:ce:03:00/40 Emask 0x400 (NCQ error) <F>
[ 6656.403149] ata5.00: status: { DRDY SENSE ERR }
[ 6656.410624] ata5.00: error: { ABRT }

However, since this is a change from the existing behavior of this driver, this could go as a separate patch, and does not need to delay this series.

I also think that we should do a similar patch for sas_ata_task_done():

drivers/scsi/libsas/sas_ata.c +1 -1
index d35c9296f738..648d0693ceee 100644
@@ -140,7 +140,7 @@ static void sas_ata_task_done(struct sas_task *task)
                        }
 
                        dev->sata_dev.fis[3] = 0x04; /* status err */
-                       dev->sata_dev.fis[2] = ATA_ERR;
+                       dev->sata_dev.fis[2] = ATA_ERR | ATA_DRDY;
                }
        }
 

To avoid all SAS errors from being reported as HSM errors.

Kind regards,
Niklas

Signature · 2 lines
Damien Le Moal
Western Digital Research
#13 lore raw
Quoted 31 lines · Damien Le Moal
Quoted 30 lines · Niklas Cassel
Quoted 20 lines · John Garry

On 04/10/2022 15:04, John Garry wrote:

Hi Niklas,

Could you try a change like this on top:

void sas_ata_device_link_abort(struct domain_device *device, bool force_reset)

{
	struct ata_port *ap = device->sata_dev.ap;
	struct ata_link *link = &ap->link;
(quoted diff) +2 -0
+	device->sata_dev.fis[2] = ATA_ERR | ATA_DRDY;
+	device->sata_dev.fis[3] = 0x04;
	link->eh_info.err_mask |= AC_ERR_DEV;
	if (force_reset)
		link->eh_info.action |= ATA_EH_RESET;
	ata_link_abort(link);
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(sas_ata_device_link_abort);

I tried it myself and it looked to work ok, except I have a problem with my arm64 system in that the read log ext times-out and all TF show "device error", like:

Do you know why it fails to read the log?
Can you read the NCQ Command Error log using ATA16 passthrough commands?

Quoted 6 lines
Quoted 3 lines

sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc

The first byte is the last NCQ tag (in hex) that failed.

libata issues read log as a non-ncq command under EH. So the NCQ error log will not help.

Hello Damien,

John explained that he got a timeout from EH when reading the log:

[  350.281581] ata1: failed to read log page 10h (errno=-5)
[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

ata_eh_read_log_10h() uses ata_read_log_page(), which will first try to read the log using READ LOG DMA EXT. If that fails, it will retry using READ LOG EXT.

Therefore, to see if this is a driver specific bug, I suggested to try to read the NCQ Command Error log using ATA16 passthrough commands:

$ sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc
will read the log using READ LOG DMA EXT.
$ sudo sg_sat_read_gplog --log=0x10 /dev/sdc
will read the log using READ LOG EXT.

Neither of these two suggested commands are NCQ commands. (Neither command is encapsulated in a RECEIVE FPDMA QUEUED, so I'm not sure what you mean.)

Garry, I now see that:

[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

Your port is frozen.

ata_read_log_page() calls ata_exec_internal() which calls ata_exec_internal_sg(), which will simply return an error without sending down the command to the drive, if the port is frozen.

Not sure why your port is frozen, mine is obviously not.

ata_do_link_abort() calls ata_eh_set_pending() without activating fast drain:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/ata/libata-eh.c?h=v6.0#n989

So I'm not sure why your port is frozen.
(The fast drain timer does freeze the port, but it shouldn't be enabled.) It might be worthwhile to see who freezes the port in your case.

Kind regards,
Niklas

#14 lore raw
Quoted 70 lines · Niklas Cassel
Quoted 31 lines · Damien Le Moal
Quoted 30 lines · Niklas Cassel
Quoted 20 lines · John Garry

On 04/10/2022 15:04, John Garry wrote:

Hi Niklas,

Could you try a change like this on top:

void sas_ata_device_link_abort(struct domain_device *device, bool force_reset)

{
	struct ata_port *ap = device->sata_dev.ap;
	struct ata_link *link = &ap->link;
(quoted diff) +2 -0
+	device->sata_dev.fis[2] = ATA_ERR | ATA_DRDY;
+	device->sata_dev.fis[3] = 0x04;
	link->eh_info.err_mask |= AC_ERR_DEV;
	if (force_reset)
		link->eh_info.action |= ATA_EH_RESET;
	ata_link_abort(link);
}
Quoted 5 lines

EXPORT_SYMBOL_GPL(sas_ata_device_link_abort);

I tried it myself and it looked to work ok, except I have a problem with my arm64 system in that the read log ext times-out and all TF show "device error", like:

Do you know why it fails to read the log?
Can you read the NCQ Command Error log using ATA16 passthrough commands?

Quoted 6 lines
Quoted 3 lines

sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc

The first byte is the last NCQ tag (in hex) that failed.

libata issues read log as a non-ncq command under EH. So the NCQ error log will not help.

Hello Damien,

John explained that he got a timeout from EH when reading the log:

[  350.281581] ata1: failed to read log page 10h (errno=-5)
[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

ata_eh_read_log_10h() uses ata_read_log_page(), which will first try to read the log using READ LOG DMA EXT. If that fails, it will retry using READ LOG EXT.

Therefore, to see if this is a driver specific bug, I suggested to try to read the NCQ Command Error log using ATA16 passthrough commands:

$ sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc
will read the log using READ LOG DMA EXT.
$ sudo sg_sat_read_gplog --log=0x10 /dev/sdc
will read the log using READ LOG EXT.

Neither of these two suggested commands are NCQ commands. (Neither command is encapsulated in a RECEIVE FPDMA QUEUED, so I'm not sure what you mean.)

Garry, I now see that:

[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

Your port is frozen.

ata_read_log_page() calls ata_exec_internal() which calls ata_exec_internal_sg(), which will simply return an error without sending down the command to the drive, if the port is frozen.

Quoted 8 lines

Not sure why your port is frozen, mine is obviously not.

ata_do_link_abort() calls ata_eh_set_pending() without activating fast drain:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/ata/libata-eh.c?h=v6.0#n989

So I'm not sure why your port is frozen.
(The fast drain timer does freeze the port, but it shouldn't be enabled.) It might be worthwhile to see who freezes the port in your case.

Might come from the command timeout. John has had many problems with the pm80xx HBA in his Arm machine from a while back. Likely not a driver issue but a hw one... No-one seems to be able to recreate the same problem.

We need to try the HBA on our Arm board to see what happens.

Quoted 4 lines

Kind regards,
Niklas

Signature · 2 lines
Damien Le Moal
Western Digital Research
author
#15 lore raw
Quoted 9 lines · Damien Le Moal
Quoted 9 lines

Hello Damien,

John explained that he got a timeout from EH when reading the log:

[  350.281581] ata1: failed to read log page 10h (errno=-5)
[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

ata_eh_read_log_10h() uses ata_read_log_page(), which will first try to read the log using READ LOG DMA EXT. If that fails, it will retry using READ LOG EXT.

Quoted 8 lines
Quoted 8 lines

Therefore, to see if this is a driver specific bug, I suggested to try to read the NCQ Command Error log using ATA16 passthrough commands:

$ sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc
will read the log using READ LOG DMA EXT.
$ sudo sg_sat_read_gplog --log=0x10 /dev/sdc
will read the log using READ LOG EXT.

Note that I can't get a distro to boot on this system from the HDD for the same timeout problem (so no tools easily available).

Quoted 10 lines
Quoted 10 lines

Neither of these two suggested commands are NCQ commands. (Neither command is encapsulated in a RECEIVE FPDMA QUEUED, so I'm not sure what you mean.)

Garry, I now see that:

[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

Your port is frozen.

Quoted 5 lines
Quoted 5 lines

ata_read_log_page() calls ata_exec_internal() which calls ata_exec_internal_sg(), which will simply return an error without sending down the command to the drive, if the port is frozen.

Not sure why your port is frozen, mine is obviously not.

I think that it gets frozen when the internal command for read log ext times out. More below about that timeout.

Quoted 4 lines
Quoted 4 lines

ata_do_link_abort() calls ata_eh_set_pending() without activating fast drain:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/ata/libata-eh.c?h=v6.0#n989

Quoted 9 lines
Quoted 3 lines

So I'm not sure why your port is frozen.
(The fast drain timer does freeze the port, but it shouldn't be enabled.) It might be worthwhile to see who freezes the port in your case.

Might come from the command timeout. John has had many problems with the pm80xx HBA in his Arm machine from a while back. Likely not a driver issue but a hw one... No-one seems to be able to recreate the same problem.

We need to try the HBA on our Arm board to see what happens.

Yeah, it just looks to be the longstanding issue of using this card on my arm64 machine - that is that I get IO timeouts quite regularly. I should have mentioned that yesterday. This just seems to be a driver issue.

Interestingly this read log ext always seems to timeout, so maybe I could see if there is anything specific about this command which could give a clue to the underlying issue. But I have spent much time trying to debug this issue, so not too motivated any more if I’m completely honest ...

Thanks,
John

#16 lore raw
Quoted 32 lines · John Garry
Quoted 9 lines · Damien Le Moal
Quoted 9 lines

Hello Damien,

John explained that he got a timeout from EH when reading the log:

[  350.281581] ata1: failed to read log page 10h (errno=-5)
[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

ata_eh_read_log_10h() uses ata_read_log_page(), which will first try to read the log using READ LOG DMA EXT. If that fails, it will retry using READ LOG EXT.

Quoted 8 lines
Quoted 8 lines

Therefore, to see if this is a driver specific bug, I suggested to try to read the NCQ Command Error log using ATA16 passthrough commands:

$ sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc
will read the log using READ LOG DMA EXT.
$ sudo sg_sat_read_gplog --log=0x10 /dev/sdc
will read the log using READ LOG EXT.

Note that I can't get a distro to boot on this system from the HDD for the same timeout problem (so no tools easily available).

Quoted 10 lines
Quoted 10 lines

Neither of these two suggested commands are NCQ commands. (Neither command is encapsulated in a RECEIVE FPDMA QUEUED, so I'm not sure what you mean.)

Garry, I now see that:

[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

Your port is frozen.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

ata_read_log_page() calls ata_exec_internal() which calls ata_exec_internal_sg(), which will simply return an error without sending down the command to the drive, if the port is frozen.

Not sure why your port is frozen, mine is obviously not.

I think that it gets frozen when the internal command for read log ext times out. More below about that timeout.

ata_read_log_page() will first try to read using READ LOG DMA EXT. If that fails it will retry with READ LOG EXT.

Your log has this:

[  350.257870] ata1.00: qc timeout (cmd 0x47)

So it is definitely ATA_CMD_READ_LOG_DMA_EXT that times out.

On timeout, ata_exec_internal_sg() will freeze the port:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/ata/libata-core.c?h=v6.0#n1577

When ata_read_log_page() retries with the port frozen, READ LOG EXT will obviously fail (since the port is frozen).

Not sure why READ LOG DMA EXT would timeout for you... Perhaps your drive does not implement this command, and incorrectly reports supporting this command via ata_id_has_read_log_dma_ext().

Perhaps you could try boot your kernel with libata.force=nodmalog on the kernel command line, so that ata_read_log_page() will use READ LOG EXT on the first try.

Damien, it seems that there is no use in retrying if the port is frozen/we got a timeout, so perhaps:

drivers/ata/libata-core.c +2 -1
index e74ab6c0f1a0..1aa628332c8e 100644
@@ -2035,7 +2035,8 @@ unsigned int ata_read_log_page(struct ata_device *dev, u8 log,
        if (err_mask) {
                if (dma) {
                        dev->horkage |= ATA_HORKAGE_NO_DMA_LOG;
-                       goto retry;
+                       if (err_mask != AC_ERR_TIMEOUT)
+                               goto retry;
                }
 

or:

drivers/ata/libata-core.c +2 -1
index e74ab6c0f1a0..2fa03b7573ac 100644
@@ -2035,7 +2035,8 @@ unsigned int ata_read_log_page(struct ata_device *dev, u8 log,
        if (err_mask) {
                if (dma) {
                        dev->horkage |= ATA_HORKAGE_NO_DMA_LOG;
-                       goto retry;
+                       if (!(dev->link->ap->pflags & ATA_PFLAG_FROZEN))
+                               goto retry;
                }
 

would be in order, so that we actually print the real error, instead of a bogus AC_ERR_SYSTEM (returned by ata_exec_internal_sg()) when the port is frozen.

Quoted 12 lines
Quoted 12 lines
Quoted 8 lines

ata_do_link_abort() calls ata_eh_set_pending() without activating fast drain:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/ata/libata-eh.c?h=v6.0#n989

So I'm not sure why your port is frozen.
(The fast drain timer does freeze the port, but it shouldn't be enabled.) It might be worthwhile to see who freezes the port in your case.

Might come from the command timeout. John has had many problems with the pm80xx HBA in his Arm machine from a while back. Likely not a driver issue but a hw one... No-one seems to be able to recreate the same problem.

Quoted 6 lines
Quoted 2 lines

We need to try the HBA on our Arm board to see what happens.

Yeah, it just looks to be the longstanding issue of using this card on my arm64 machine - that is that I get IO timeouts quite regularly. I should have mentioned that yesterday. This just seems to be a driver issue.

Out of curiosity, which arm64 SoC is this?

While it is very unlikely that this is your problem, but I've encountered an issue on an ARM board before, where the PCIe controller was incorrectly configured in device tree, causing the controller to miss interrrupts, which presented itself to the user as timeouts in the WiFi driver:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=97131f85c08e024df49480ed499aae8fb754067f

Kind regards,
Niklas

author
#17 lore raw
Quoted 16 lines · Niklas Cassel
Quoted 2 lines

I think that it gets frozen when the internal command for read log ext times out. More below about that timeout.

ata_read_log_page() will first try to read using READ LOG DMA EXT. If that fails it will retry with READ LOG EXT.

Your log has this:

[  350.257870] ata1.00: qc timeout (cmd 0x47)

So it is definitely ATA_CMD_READ_LOG_DMA_EXT that times out.

On timeout, ata_exec_internal_sg() will freeze the port:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/ata/libata-core.c?h=v6.0#n1577

When ata_read_log_page() retries with the port frozen, READ LOG EXT will obviously fail (since the port is frozen).

Quoted 9 lines

Not sure why READ LOG DMA EXT would timeout for you... Perhaps your drive does not implement this command, and incorrectly reports supporting this command via ata_id_has_read_log_dma_ext().

Perhaps you could try boot your kernel with libata.force=nodmalog on the kernel command line, so that ata_read_log_page() will use READ LOG EXT on the first try.

I tried that and unfortunately it does not appear to help.

I get this log, which proves no dmalog

[   15.757617] ata1.00: FORCE: horkage modified (nodmalog)

but then still fails with timeout:

[  123.094430] ata1.00: qc timeout (cmd 0x2f)
[  123.098637] pm80xx0:: mpi_sata_completion 2293: task null, freeing CCB tag 2
[  123.105711] ata1.00: Read log 0x10 page 0x00 failed, Emask 0x5
[  123.118081] ata1: failed to read log page 10h (errno=-5)
Quoted 45 lines

Damien, it seems that there is no use in retrying if the port is frozen/we got a timeout, so perhaps:

drivers/ata/libata-core.c +2 -1
index e74ab6c0f1a0..1aa628332c8e 100644
@@ -2035,7 +2035,8 @@ unsigned int ata_read_log_page(struct ata_device *dev, u8 log,
         if (err_mask) {
                 if (dma) {
                         dev->horkage |= ATA_HORKAGE_NO_DMA_LOG;
-                       goto retry;
+                       if (err_mask != AC_ERR_TIMEOUT)
+                               goto retry;
                 }
 

or:

drivers/ata/libata-core.c +2 -1
index e74ab6c0f1a0..2fa03b7573ac 100644
@@ -2035,7 +2035,8 @@ unsigned int ata_read_log_page(struct ata_device *dev, u8 log,
         if (err_mask) {
                 if (dma) {
                         dev->horkage |= ATA_HORKAGE_NO_DMA_LOG;
-                       goto retry;
+                       if (!(dev->link->ap->pflags & ATA_PFLAG_FROZEN))
+                               goto retry;
                 }
 

would be in order, so that we actually print the real error, instead of a bogus AC_ERR_SYSTEM (returned by ata_exec_internal_sg()) when the port is frozen.

Quoted 10 lines
Quoted 10 lines
Quoted 6 lines

ata_do_link_abort() calls ata_eh_set_pending() without activating fast drain:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/ata/libata-eh.c?h=v6.0#n989

So I'm not sure why your port is frozen.
(The fast drain timer does freeze the port, but it shouldn't be enabled.) It might be worthwhile to see who freezes the port in your case.

Might come from the command timeout. John has had many problems with the pm80xx HBA in his Arm machine from a while back. Likely not a driver issue but a hw one... No-one seems to be able to recreate the same problem.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

We need to try the HBA on our Arm board to see what happens.

Yeah, it just looks to be the longstanding issue of using this card on my arm64 machine - that is that I get IO timeouts quite regularly. I should have mentioned that yesterday. This just seems to be a driver issue.

Out of curiosity, which arm64 SoC is this?

HiSilicon hi1620 which contains a custom arm v8 implementation. Note that others have also seen the issue with this card on other arm implementations.

Quoted 6 lines

While it is very unlikely that this is your problem, but I've encountered an issue on an ARM board before, where the PCIe controller was incorrectly configured in device tree, causing the controller to miss interrrupts, which presented itself to the user as timeouts in the WiFi driver:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=97131f85c08e024df49480ed499aae8fb754067f

Unlikely. Indeed, when I was checking this issue some time go, I found that not only was there no completion interrupt but also no completion when I manually examine the completion ring buffer read and write pointers.

Here's where I discuss this issue earlier a bit:
https://lore.kernel.org/linux-scsi/PH0PR11MB511238B8FF7B44C375DDDFADEC519@PH0PR11MB5112.namprd11.prod.outlook.com/

Thanks,
John

#18 lore raw
Quoted 16 lines · John Garry
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

Yeah, it just looks to be the longstanding issue of using this card on my arm64 machine - that is that I get IO timeouts quite regularly. I should have mentioned that yesterday. This just seems to be a driver issue.

Out of curiosity, which arm64 SoC is this?

HiSilicon hi1620 which contains a custom arm v8 implementation. Note that others have also seen the issue with this card on other arm implementations.

Quoted 6 lines

While it is very unlikely that this is your problem, but I've encountered an issue on an ARM board before, where the PCIe controller was incorrectly configured in device tree, causing the controller to miss interrrupts, which presented itself to the user as timeouts in the WiFi driver:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=97131f85c08e024df49480ed499aae8fb754067f

Quoted 7 lines

Unlikely. Indeed, when I was checking this issue some time go, I found that not only was there no completion interrupt but also no completion when I manually examine the completion ring buffer read and write pointers.

Here's where I discuss this issue earlier a bit:
https://lore.kernel.org/linux-scsi/PH0PR11MB511238B8FF7B44C375DDDFADEC519@PH0PR11MB5112.namprd11.prod.outlook.com/

Hello John,

For the record, I tested the pm80xx driver on a HoneyComb LX2 board (an arm64 board using ACPI).

I tried v6.1-rc1 both with and without your series in $subject.

I couldn't see any issues.

What I tried:
-Running fio:
fio --name=test --filename=/dev/sdc --ioengine=io_uring --rw=randrw --direct=1 --iodepth=32 --bs=1M on three different HDDs simultaneously for 15+ minutes,
without any errors in fio or dmesg.

-Creating and mounting a btrfs volume, doing a huge dd to the filesystem without issues.

-sg_sat_read_gplog -d --log=0x10 /dev/sda
which successfully returned the log.

It is worth mentioning that this arm64 board has reserved memory regions, but does not yet have a firmware that supplies a IORT RMR (reserved memory regions) revision E.d node, which means that in order to get this board to boot successfully, we need to supply:
"arm-smmu.disable_bypass=0 iommu.passthrough=1"
on the kernel command line.

It could be worth trying the same on your arm64 machine, to see that also makes the pm80xx driver play nice for you.

Kind regards,
Niklas

author
#19 lore raw

Hi Niklas,

Quoted 7 lines

For the record, I tested the pm80xx driver on a HoneyComb LX2 board (an arm64 board using ACPI).

I tried v6.1-rc1 both with and without your series in $subject.

I couldn't see any issues.

ok, thanks for the effort.

Quoted 15 lines

What I tried:
-Running fio:
fio --name=test --filename=/dev/sdc --ioengine=io_uring --rw=randrw --direct=1 --iodepth=32 --bs=1M on three different HDDs simultaneously for 15+ minutes,
without any errors in fio or dmesg.

-Creating and mounting a btrfs volume, doing a huge dd to the filesystem without issues.

-sg_sat_read_gplog -d --log=0x10 /dev/sda
which successfully returned the log.

Quoted 6 lines

It is worth mentioning that this arm64 board has reserved memory regions, but does not yet have a firmware that supplies a IORT RMR (reserved memory regions) revision E.d node, which means that in order to get this board to boot successfully, we need to supply:
"arm-smmu.disable_bypass=0 iommu.passthrough=1"
on the kernel command line.

hmmm... that's interesting. I can try again with the IOMMU turned off, but, as I recall, it did not make a difference before. I think that requiring reserved memory regions would totally bust the driver (if not present) with IOMMU enabled. As I recall, sas 3008 card would not work without RMR for us.

It's also interesting that this LX2 board has A72 cores. For my system, we have newer custom arm v8 cores with quite weak memory ordering implementation. With that same system, I have detected a couple of other driver memory ordering bugs which we did not see on our A72-based platforms.

I always suspected that this issue was a memory ordering issue, but since the hang so reliably occurs I ruled it out. Maybe it is...

thanks,
John

#20 lore raw
Quoted 27 lines · John Garry

Hi Niklas,

Quoted 7 lines

For the record, I tested the pm80xx driver on a HoneyComb LX2 board (an arm64 board using ACPI).

I tried v6.1-rc1 both with and without your series in $subject.

I couldn't see any issues.

ok, thanks for the effort.

Quoted 15 lines

What I tried:
-Running fio:
fio --name=test --filename=/dev/sdc --ioengine=io_uring --rw=randrw --direct=1 --iodepth=32 --bs=1M on three different HDDs simultaneously for 15+ minutes,
without any errors in fio or dmesg.

-Creating and mounting a btrfs volume, doing a huge dd to the filesystem without issues.

-sg_sat_read_gplog -d --log=0x10 /dev/sda
which successfully returned the log.

Quoted 11 lines
Quoted 6 lines

It is worth mentioning that this arm64 board has reserved memory regions, but does not yet have a firmware that supplies a IORT RMR (reserved memory regions) revision E.d node, which means that in order to get this board to boot successfully, we need to supply:
"arm-smmu.disable_bypass=0 iommu.passthrough=1"
on the kernel command line.

hmmm... that's interesting. I can try again with the IOMMU turned off, but, as I recall, it did not make a difference before. I think that requiring reserved memory regions would totally bust the driver (if not present) with IOMMU enabled. As I recall, sas 3008 card would not work without RMR for us.

At least on the HoneyComb LX2,
running with "arm-smmu.disable_bypass=0 iommu.passthrough=1" gives a working system (and working pm80xx).

The ACPI IOMMU code that parses the ACPI IORT RMR revision E.d node was first included in kernel v6.0:
https://lore.kernel.org/linux-iommu/[email protected]/

However, the HoneyComb edk2-platforms code has not yet been updated to add a ACPI IORT RMR revision E.d node.

Considering that it works with "arm-smmu.disable_bypass=0 iommu.passthrough=1", I assume that the ACPI IORT RMR node basically just defines a number of memory regions which the IOMMU should treat as "bypass", while all other memory has to be re-mapped via the IOMMU as per usual.

Kind regards,
Niklas

author
#21 lore raw
Quoted 8 lines · Niklas Cassel

At least on the HoneyComb LX2,
running with "arm-smmu.disable_bypass=0 iommu.passthrough=1" gives a working system (and working pm80xx).

The ACPI IOMMU code that parses the ACPI IORT RMR revision E.d node was first included in kernel v6.0:
https://lore.kernel.org/linux-iommu/[email protected]/

Quoted 7 lines

However, the HoneyComb edk2-platforms code has not yet been updated to add a ACPI IORT RMR revision E.d node.

Considering that it works with "arm-smmu.disable_bypass=0 iommu.passthrough=1", I assume that the ACPI IORT RMR node basically just defines a number of memory regions which the IOMMU should treat as "bypass", while all other memory has to be re-mapped via the IOMMU as per usual.

Hi Niklas,

As I expected, unfortunately that did not help. Note that I actually can't use passthrough on my platform due to SMMUv3 implementation bug, so I just disable SMMUv3 entirely via kernel config.

I'm still betting on a memory ordering issue for me.

Thanks again,
John

#22 lore raw
In reply to Niklas Cassel
Quoted 89 lines · Niklas Cassel
Quoted 32 lines · John Garry
Quoted 9 lines · Damien Le Moal
Quoted 9 lines

Hello Damien,

John explained that he got a timeout from EH when reading the log:

[  350.281581] ata1: failed to read log page 10h (errno=-5)
[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

ata_eh_read_log_10h() uses ata_read_log_page(), which will first try to read the log using READ LOG DMA EXT. If that fails, it will retry using READ LOG EXT.

Quoted 8 lines
Quoted 8 lines

Therefore, to see if this is a driver specific bug, I suggested to try to read the NCQ Command Error log using ATA16 passthrough commands:

$ sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc
will read the log using READ LOG DMA EXT.
$ sudo sg_sat_read_gplog --log=0x10 /dev/sdc
will read the log using READ LOG EXT.

Note that I can't get a distro to boot on this system from the HDD for the same timeout problem (so no tools easily available).

Quoted 10 lines
Quoted 10 lines

Neither of these two suggested commands are NCQ commands. (Neither command is encapsulated in a RECEIVE FPDMA QUEUED, so I'm not sure what you mean.)

Garry, I now see that:

[  350.577181] ata1.00: exception Emask 0x1 SAct 0xffffffff SErr 0x0 action 0x6 frozen

Your port is frozen.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

ata_read_log_page() calls ata_exec_internal() which calls ata_exec_internal_sg(), which will simply return an error without sending down the command to the drive, if the port is frozen.

Not sure why your port is frozen, mine is obviously not.

I think that it gets frozen when the internal command for read log ext times out. More below about that timeout.

ata_read_log_page() will first try to read using READ LOG DMA EXT. If that fails it will retry with READ LOG EXT.

Your log has this:

[  350.257870] ata1.00: qc timeout (cmd 0x47)

So it is definitely ATA_CMD_READ_LOG_DMA_EXT that times out.

On timeout, ata_exec_internal_sg() will freeze the port:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/ata/libata-core.c?h=v6.0#n1577

When ata_read_log_page() retries with the port frozen, READ LOG EXT will obviously fail (since the port is frozen).

Not sure why READ LOG DMA EXT would timeout for you... Perhaps your drive does not implement this command, and incorrectly reports supporting this command via ata_id_has_read_log_dma_ext().

Perhaps you could try boot your kernel with libata.force=nodmalog on the kernel command line, so that ata_read_log_page() will use READ LOG EXT on the first try.

Damien, it seems that there is no use in retrying if the port is frozen/we got a timeout, so perhaps:

drivers/ata/libata-core.c +2 -1
index e74ab6c0f1a0..1aa628332c8e 100644
@@ -2035,7 +2035,8 @@ unsigned int ata_read_log_page(struct ata_device *dev, u8 log,
        if (err_mask) {
                if (dma) {
                        dev->horkage |= ATA_HORKAGE_NO_DMA_LOG;
-                       goto retry;
+                       if (err_mask != AC_ERR_TIMEOUT)
+                               goto retry;
                }
 

or:

drivers/ata/libata-core.c +0 -0
index e74ab6c0f1a0..2fa03b7573ac 100644

drivers/ata/libata-core.c unsigned int ata_read_log_page(struct ata_device *dev, u8 log, · line 2039

Quoted 6 lines
drivers/ata/libata-core.c +2 -1
@@ -2035,7 +2035,8 @@ unsigned int ata_read_log_page(struct ata_device *dev, u8 log,
        if (err_mask) {
                if (dma) {
                        dev->horkage |= ATA_HORKAGE_NO_DMA_LOG;
-                       goto retry;
+                       if (!(dev->link->ap->pflags & ATA_PFLAG_FROZEN))
+                               goto retry;

Yes, something like this is needed. Though I would prefer having a little ata_port_frozen() helper for the condition.

Quoted 4 lines
drivers/ata/libata-core.c +0 -0
                }
 

would be in order, so that we actually print the real error, instead of a bogus AC_ERR_SYSTEM (returned by ata_exec_internal_sg()) when the port is frozen.

yep.

Quoted 30 lines
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines

ata_do_link_abort() calls ata_eh_set_pending() without activating fast drain:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/ata/libata-eh.c?h=v6.0#n989

So I'm not sure why your port is frozen.
(The fast drain timer does freeze the port, but it shouldn't be enabled.) It might be worthwhile to see who freezes the port in your case.

Might come from the command timeout. John has had many problems with the pm80xx HBA in his Arm machine from a while back. Likely not a driver issue but a hw one... No-one seems to be able to recreate the same problem.

Quoted 6 lines
Quoted 2 lines

We need to try the HBA on our Arm board to see what happens.

Yeah, it just looks to be the longstanding issue of using this card on my arm64 machine - that is that I get IO timeouts quite regularly. I should have mentioned that yesterday. This just seems to be a driver issue.

Out of curiosity, which arm64 SoC is this?

While it is very unlikely that this is your problem, but I've encountered an issue on an ARM board before, where the PCIe controller was incorrectly configured in device tree, causing the controller to miss interrrupts, which presented itself to the user as timeouts in the WiFi driver:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=97131f85c08e024df49480ed499aae8fb754067f

Kind regards,
Niklas

Signature · 2 lines
Damien Le Moal
Western Digital Research
author
#23 lore raw
In reply to Niklas Cassel
Quoted 6 lines · Niklas Cassel

Do you know why it fails to read the log?
Can you read the NCQ Command Error log using ATA16 passthrough commands?

sudo sg_sat_read_gplog -d --log=0x10 /dev/sdc

The first byte is the last NCQ tag (in hex) that failed.

Please see other reply.

Quoted 3 lines

I tried your patch, and it looks good:

Thanks, good to know.

Quoted 18 lines
[ 6656.228131] ata5.00: exception Emask 0x0 SAct 0x460000 SErr 0x0 action 0x0
[ 6656.252759] ata5.00: failed command: WRITE FPDMA QUEUED
[ 6656.271554] ata5.00: cmd 61/00:00:00:d8:8a/04:00:ce:03:00/40 tag 17 ncq dma 524288 out
                         res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device error)
[ 6656.309308] ata5.00: status: { DRDY ERR }
[ 6656.316403] ata5.00: error: { ABRT }
[ 6656.322300] ata5.00: failed command: WRITE FPDMA QUEUED
[ 6656.330871] ata5.00: cmd 61/00:00:00:dc:8a/04:00:ce:03:00/40 tag 18 ncq dma 524288 out
                         res 41/04:00:00:00:00/00:00:00:00:00/00 Emask 0x1 (device error)
[ 6656.356295] ata5.00: status: { DRDY ERR }
[ 6656.362931] ata5.00: error: { ABRT }
[ 6656.368897] ata5.00: failed command: WRITE FPDMA QUEUED
[ 6656.377471] ata5.00: cmd 61/00:00:00:d4:8a/04:00:ce:03:00/40 tag 22 ncq dma 524288 out
                         res 43/04:00:ff:d7:8a/00:00:ce:03:00/40 Emask 0x400 (NCQ error) <F>
[ 6656.403149] ata5.00: status: { DRDY SENSE ERR }
[ 6656.410624] ata5.00: error: { ABRT }
Quoted 4 lines

However, since this is a change from the existing behavior of this driver, this could go as a separate patch, and does not need to delay this series.

ok, but I am not sure about this series for 6.1 since it's now, so I will just wait.

Quoted 16 lines

I also think that we should do a similar patch for sas_ata_task_done():

drivers/scsi/libsas/sas_ata.c +1 -1
index d35c9296f738..648d0693ceee 100644
@@ -140,7 +140,7 @@ static void sas_ata_task_done(struct sas_task *task)
                         }
 
                         dev->sata_dev.fis[3] = 0x04; /* status err */
-                       dev->sata_dev.fis[2] = ATA_ERR;
+                       dev->sata_dev.fis[2] = ATA_ERR | ATA_DRDY;
                 }
         }
 
Quoted 1 line

To avoid all SAS errors from being reported as HSM errors.

Yeah, I tend to agree. I can put that change in another patch.

Thanks,
John