Skip to contentSkip to conversation

Fix IOMMU page fault on report zones

PATCH Applied Damien Le Moal 4 participants 6 messages Jul 19, 2024 – Jul 23, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Fix IOMMU page fault on report zones

A couple of patches to avoid IOMMU page faults error when executing a report zones command on ATA SMR drives connected to mpt3sas or mpi3mr SAS HBAs. These page faults happen only with AMD hosts and are not triggered with Intel machines.

Damien Le Moal (2):
scsi: mpi3mr: Avoid IOMMU page faults on report zones
scsi: mpt3sas: Avoid IOMMU page faults on report zones

 drivers/scsi/mpi3mr/mpi3mr_os.c     | 11 +++++++++++
 drivers/scsi/mpt3sas/mpt3sas_base.c | 20 ++++++++++++++++++--
 2 files changed, 29 insertions(+), 2 deletions(-)
Signature · 1 line
2.45.2
#1 lore raw

PATCH 1/2 scsi: mpi3mr: Avoid IOMMU page faults on report zones

Some firmware versions of the 9600 series SAS HBA byte-swap the report zone command reply buffer from ATA-ZAC devices by directly accessing the buffer in the host memory. This does not respect the default command DMA direction and causes IOMMU page faults on architectures with an IOMMU enforcing write-only mappings for DMA_FROM_DEVICE DMA direction (e.g. AMD hosts), leading to the device capacity to be dropped to 0:

scsi 18:0:58:0: Direct-Access-ZBC ATA WDC WSH722626AL W930 PQ: 0 ANSI: 7
scsi 18:0:58:0: Power-on or device reset occurred
sd 18:0:58:0: Attached scsi generic sg9 type 20
sd 18:0:58:0: [sdj] Host-managed zoned block device
mpi3mr 0000:c1:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0xfec0c400 flags=0x0050] mpi3mr 0000:c1:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0xfec0c500 flags=0x0050] sd 18:0:58:0: [sdj] REPORT ZONES start lba 0 failed
sd 18:0:58:0: [sdj] REPORT ZONES: Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK sd 18:0:58:0: [sdj] 0 4096-byte logical blocks: (0 B/0 B)
sd 18:0:58:0: [sdj] Write Protect is off
sd 18:0:58:0: [sdj] Mode Sense: 6b 00 10 08
sd 18:0:58:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
sd 18:0:58:0: [sdj] Attached SCSI disk

Avoid this issue by always mapping the buffer of report zones commands using DMA_BIDIRECTIONAL, that is, using a read-write IOMMU mapping.

Fixes 023ab2a9b4ed ("scsi: mpi3mr: Add support for queue command processing")
---
 drivers/scsi/mpi3mr/mpi3mr_os.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/scsi/mpi3mr/mpi3mr_os.c +11 -0
index bce639a6cca1..3a2ec4b87440 100644
@@ -3453,6 +3453,17 @@ static int mpi3mr_prepare_sg_scmd(struct mpi3mr_ioc *mrioc,
 		    scmd->sc_data_direction);
 		priv->meta_sg_valid = 1; /* To unmap meta sg DMA */
 	} else {
+		/*
+		 * Some firmware versions byte-swap the report zone command
+		 * reply from ATA-ZAC devices by directly accessing in the host
+		 * buffer. This does not respect the default command DMA
+		 * direction and causes IOMMU page faults on some architectures
+		 * with an IOMMU enforcing write mappings (e.g. AMD hosts).
+		 * Avoid such issue by making the report zones buffer mapping
+		 * bi-directional.
+		 */
+		if (scmd->cmnd[0] == ZBC_IN && scmd->cmnd[1] == ZI_REPORT_ZONES)
+			scmd->sc_data_direction = DMA_BIDIRECTIONAL;
 		sg_scmd = scsi_sglist(scmd);
 		sges_left = scsi_dma_map(scmd);
 	}
Signature · 1 line
2.45.2
#2 lore raw

PATCH 2/2 scsi: mpt3sas: Avoid IOMMU page faults on report zones

In reply to Damien Le Moal

Some firmware versions of the 9600 series SAS HBA byte-swap the report zone command reply buffer from ATA-ZAC devices by directly accessing the buffer in the host memory. This does not respect the default command DMA direction and causes IOMMU page faults on architectures with an IOMMU enforcing write-only mappings for DMA_FROM_DEVICE DMA driection (e.g. AMD hosts).

scsi 18:0:0:0: Direct-Access-ZBC ATA WDC WSH722020AL W870 PQ: 0 ANSI: 6
scsi 18:0:0:0: SATA: handle(0x0027), sas_addr(0x300062b2083e7c40), phy(0), device_name(0x5000cca29dc35e11) scsi 18:0:0:0: enclosure logical id (0x300062b208097c40), slot(0)
scsi 18:0:0:0: enclosure level(0x0000), connector name( C0.0)
scsi 18:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
scsi 18:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
sd 18:0:0:0: Attached scsi generic sg2 type 20
sd 18:0:0:0: [sdc] Host-managed zoned block device
mpt3sas 0000:41:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0021 address=0xfff9b200 flags=0x0050] mpt3sas 0000:41:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0021 address=0xfff9b300 flags=0x0050]
mpt3sas_cm0: mpt3sas_ctl_pre_reset_handler: Releasing the trace buffer due to adapter reset. mpt3sas_cm0 fault info from func: mpt3sas_base_make_ioc_ready
mpt3sas_cm0: fault_state(0x2666)!
mpt3sas_cm0: sending diag reset !!
mpt3sas_cm0: diag reset: SUCCESS
sd 18:0:0:0: [sdc] REPORT ZONES start lba 0 failed
sd 18:0:0:0: [sdc] REPORT ZONES: Result: hostbyte=DID_RESET driverbyte=DRIVER_OK
sd 18:0:0:0: [sdc] 0 4096-byte logical blocks: (0 B/0 B)

Avoid such issue by always mapping the buffer of report zones commands using DMA_BIDIRECTIONAL (read+write IOMMU mapping). This is done by introducing the helper function _base_scsi_dma_map() and using this helper in _base_build_sg_scmd() and _base_build_sg_scmd_ieee() instead of calling directly scsi_dma_map().

Fixes 471ef9d4e498 ("mpt3sas: Build MPI SGL LIST on GEN2 HBAs and IEEE SGL LIST on GEN3 HBAs")
---
 drivers/scsi/mpt3sas/mpt3sas_base.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_base.c +18 -2
index b2bcf4a27ddc..6e103875d672 100644
@@ -2671,6 +2671,22 @@ _base_build_zero_len_sge_ieee(struct MPT3SAS_ADAPTER *ioc, void *paddr)
 	_base_add_sg_single_ieee(paddr, sgl_flags, 0, 0, -1);
 }
 
+static inline int _base_scsi_dma_map(struct scsi_cmnd *cmd)
+{
+	/*
+	 * Some firmware versions byte-swap the report zone command reply from
+	 * ATA-ZAC devices by directly accessing in the host buffer. This does
+	 * not respect the default command DMA direction and causes IOMMU page
+	 * faults on some architectures with an IOMMU enforcing write mappings
+	 * (e.g. AMD hosts). Avoid such issue by making the report zones buffer
+	 * mapping bi-directional.
+	 */
+	if (cmd->cmnd[0] == ZBC_IN && cmd->cmnd[1] == ZI_REPORT_ZONES)
+		cmd->sc_data_direction = DMA_BIDIRECTIONAL;
+
+	return scsi_dma_map(cmd);
+}
+
 /**
  * _base_build_sg_scmd - main sg creation routine
  *		pcie_device is unused here!
@@ -2717,7 +2733,7 @@ _base_build_sg_scmd(struct MPT3SAS_ADAPTER *ioc,
 	sgl_flags = sgl_flags << MPI2_SGE_FLAGS_SHIFT;
 
 	sg_scmd = scsi_sglist(scmd);
-	sges_left = scsi_dma_map(scmd);
+	sges_left = _base_scsi_dma_map(scmd);
 	if (sges_left < 0)
 		return -ENOMEM;
 
@@ -2861,7 +2877,7 @@ _base_build_sg_scmd_ieee(struct MPT3SAS_ADAPTER *ioc,
 	}
 
 	sg_scmd = scsi_sglist(scmd);
-	sges_left = scsi_dma_map(scmd);
+	sges_left = _base_scsi_dma_map(scmd);
 	if (sges_left < 0)
 		return -ENOMEM;
 
Signature · 1 line
2.45.2
Reviewed-by on the v1 series
#4 lore raw
In reply to Damien Le Moal
Quoted 5 lines · Damien Le Moal

A couple of patches to avoid IOMMU page faults error when executing a report zones command on ATA SMR drives connected to mpt3sas or mpi3mr SAS HBAs. These page faults happen only with AMD hosts and are not triggered with Intel machines.

Quoted 8 lines

Damien Le Moal (2):
scsi: mpi3mr: Avoid IOMMU page faults on report zones
scsi: mpt3sas: Avoid IOMMU page faults on report zones

  drivers/scsi/mpi3mr/mpi3mr_os.c     | 11 +++++++++++
  drivers/scsi/mpt3sas/mpt3sas_base.c | 20 ++++++++++++++++++--
  2 files changed, 29 insertions(+), 2 deletions(-)

Looks good,

#5 lore raw
In reply to Damien Le Moal

Damien,

Quoted 4 lines

A couple of patches to avoid IOMMU page faults error when executing a report zones command on ATA SMR drives connected to mpt3sas or mpi3mr SAS HBAs. These page faults happen only with AMD hosts and are not triggered with Intel machines.

Applied to 6.11/scsi-staging, thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering