Fix IOMMU page fault on report zones
Subsystems (3)
cover Fix IOMMU page fault on report zones
A couple of patches to avoid IOMMU page faults error when executing a report zones command on ATA SMR drives connected to mpt3sas or mpi3mr SAS HBAs. These page faults happen only with AMD hosts and are not triggered with Intel machines.
Damien Le Moal (2):
scsi: mpi3mr: Avoid IOMMU page faults on report zones
scsi: mpt3sas: Avoid IOMMU page faults on report zones
drivers/scsi/mpi3mr/mpi3mr_os.c | 11 +++++++++++ drivers/scsi/mpt3sas/mpt3sas_base.c | 20 ++++++++++++++++++-- 2 files changed, 29 insertions(+), 2 deletions(-)
Signature · 1 line
2.45.2
PATCH 1/2 scsi: mpi3mr: Avoid IOMMU page faults on report zones
Some firmware versions of the 9600 series SAS HBA byte-swap the report zone command reply buffer from ATA-ZAC devices by directly accessing the buffer in the host memory. This does not respect the default command DMA direction and causes IOMMU page faults on architectures with an IOMMU enforcing write-only mappings for DMA_FROM_DEVICE DMA direction (e.g. AMD hosts), leading to the device capacity to be dropped to 0:
scsi 18:0:58:0: Direct-Access-ZBC ATA WDC WSH722626AL W930 PQ: 0 ANSI: 7
scsi 18:0:58:0: Power-on or device reset occurred
sd 18:0:58:0: Attached scsi generic sg9 type 20
sd 18:0:58:0: [sdj] Host-managed zoned block device
mpi3mr 0000:c1:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0xfec0c400 flags=0x0050] mpi3mr 0000:c1:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0001 address=0xfec0c500 flags=0x0050] sd 18:0:58:0: [sdj] REPORT ZONES start lba 0 failed
sd 18:0:58:0: [sdj] REPORT ZONES: Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK sd 18:0:58:0: [sdj] 0 4096-byte logical blocks: (0 B/0 B)
sd 18:0:58:0: [sdj] Write Protect is off
sd 18:0:58:0: [sdj] Mode Sense: 6b 00 10 08
sd 18:0:58:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
sd 18:0:58:0: [sdj] Attached SCSI disk
Avoid this issue by always mapping the buffer of report zones commands using DMA_BIDIRECTIONAL, that is, using a read-write IOMMU mapping.
--- drivers/scsi/mpi3mr/mpi3mr_os.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/scsi/mpi3mr/mpi3mr_os.c +11 -0
@@ -3453,6 +3453,17 @@ static int mpi3mr_prepare_sg_scmd(struct mpi3mr_ioc *mrioc, scmd->sc_data_direction); priv->meta_sg_valid =1 ;/* To unmap meta sg DMA */ }else { +/* +* Some firmware versions byte-swap the report zone command +* reply from ATA-ZAC devices by directly accessing in the host +* buffer. This does not respect the default command DMA +* direction and causes IOMMU page faults on some architectures +* with an IOMMU enforcing write mappings (e.g. AMD hosts). +* Avoid such issue by making the report zones buffer mapping +* bi-directional. +*/ +if (scmd->cmnd[0 ] ==ZBC_IN && scmd->cmnd[1 ] ==ZI_REPORT_ZONES ) + scmd->sc_data_direction =DMA_BIDIRECTIONAL ; sg_scmd =scsi_sglist (scmd); sges_left =scsi_dma_map (scmd); }
Signature · 1 line
2.45.2
PATCH 2/2 scsi: mpt3sas: Avoid IOMMU page faults on report zones
In reply to Damien Le MoalSome firmware versions of the 9600 series SAS HBA byte-swap the report zone command reply buffer from ATA-ZAC devices by directly accessing the buffer in the host memory. This does not respect the default command DMA direction and causes IOMMU page faults on architectures with an IOMMU enforcing write-only mappings for DMA_FROM_DEVICE DMA driection (e.g. AMD hosts).
scsi 18:0:0:0: Direct-Access-ZBC ATA WDC WSH722020AL W870 PQ: 0 ANSI: 6
scsi 18:0:0:0: SATA: handle(0x0027), sas_addr(0x300062b2083e7c40), phy(0), device_name(0x5000cca29dc35e11) scsi 18:0:0:0: enclosure logical id (0x300062b208097c40), slot(0)
scsi 18:0:0:0: enclosure level(0x0000), connector name( C0.0)
scsi 18:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
scsi 18:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
sd 18:0:0:0: Attached scsi generic sg2 type 20
sd 18:0:0:0: [sdc] Host-managed zoned block device
mpt3sas 0000:41:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0021 address=0xfff9b200 flags=0x0050] mpt3sas 0000:41:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0021 address=0xfff9b300 flags=0x0050]mpt3sas_cm0: mpt3sas_ctl_pre_reset_handler: Releasing the trace buffer due to adapter reset. mpt3sas_cm0 fault info from func: mpt3sas_base_make_ioc_readympt3sas_cm0: fault_state(0x2666)!mpt3sas_cm0: sending diag reset !!mpt3sas_cm0: diag reset: SUCCESS
sd 18:0:0:0: [sdc] REPORT ZONES start lba 0 failed
sd 18:0:0:0: [sdc] REPORT ZONES: Result: hostbyte=DID_RESET driverbyte=DRIVER_OK
sd 18:0:0:0: [sdc] 0 4096-byte logical blocks: (0 B/0 B)
Avoid such issue by always mapping the buffer of report zones commands using DMA_BIDIRECTIONAL (read+write IOMMU mapping). This is done by introducing the helper function _base_scsi_dma_map() and using this helper in _base_build_sg_scmd() and _base_build_sg_scmd_ieee() instead of calling directly scsi_dma_map().
--- drivers/scsi/mpt3sas/mpt3sas_base.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
drivers/scsi/mpt3sas/mpt3sas_base.c +18 -2
@@ -2671,6 +2671,22 @@ _base_build_zero_len_sge_ieee(struct MPT3SAS_ADAPTER *ioc, void *paddr)_base_add_sg_single_ieee (paddr, sgl_flags,0 ,0 ,-1 ); } +static inline int _base_scsi_dma_map (struct scsi_cmnd *cmd) +{ +/* +* Some firmware versions byte-swap the report zone command reply from +* ATA-ZAC devices by directly accessing in the host buffer. This does +* not respect the default command DMA direction and causes IOMMU page +* faults on some architectures with an IOMMU enforcing write mappings +* (e.g. AMD hosts). Avoid such issue by making the report zones buffer +* mapping bi-directional. +*/ +if (cmd->cmnd[0 ] ==ZBC_IN && cmd->cmnd[1 ] ==ZI_REPORT_ZONES ) + cmd->sc_data_direction =DMA_BIDIRECTIONAL ; + +return scsi_dma_map (cmd); +} +/** * _base_build_sg_scmd - main sg creation routine * pcie_device is unused here!
@@ -2717,7 +2733,7 @@ _base_build_sg_scmd(struct MPT3SAS_ADAPTER *ioc, sgl_flags = sgl_flags <<MPI2_SGE_FLAGS_SHIFT ; sg_scmd =scsi_sglist (scmd); - sges_left =scsi_dma_map (scmd); + sges_left =_base_scsi_dma_map (scmd);if (sges_left <0 )return -ENOMEM ;
@@ -2861,7 +2877,7 @@ _base_build_sg_scmd_ieee(struct MPT3SAS_ADAPTER *ioc, } sg_scmd =scsi_sglist (scmd); - sges_left =scsi_dma_map (scmd); + sges_left =_base_scsi_dma_map (scmd);if (sges_left <0 )return -ENOMEM ;
Signature · 1 line
2.45.2
Pretty sad that people think they can get away with this, but as we'll have to deal with it:
Quoted 5 lines · Damien Le Moal
A couple of patches to avoid IOMMU page faults error when executing a report zones command on ATA SMR drives connected to mpt3sas or mpi3mr SAS HBAs. These page faults happen only with AMD hosts and are not triggered with Intel machines.
Quoted 8 lines
Damien Le Moal (2):
scsi: mpi3mr: Avoid IOMMU page faults on report zones
scsi: mpt3sas: Avoid IOMMU page faults on report zonesdrivers/scsi/mpi3mr/mpi3mr_os.c | 11 +++++++++++ drivers/scsi/mpt3sas/mpt3sas_base.c | 20 ++++++++++++++++++-- 2 files changed, 29 insertions(+), 2 deletions(-)
Looks good,
Martin K. Petersen
AppliedDamien,
Quoted 4 lines
A couple of patches to avoid IOMMU page faults error when executing a report zones command on ATA SMR drives connected to mpt3sas or mpi3mr SAS HBAs. These page faults happen only with AMD hosts and are not triggered with Intel machines.
Applied to 6.11/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering