Skip to contentSkip to conversation

mpt3sas: Limit NVMe request size to 2 MiB

PATCH Superseded Ranjan Kumar 3 participants 4 messages Apr 9 – Apr 11

Subsystems (3)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Ranjan Kumar

author maintainer
#0 lore raw

PATCH 1/1 mpt3sas: Limit NVMe request size to 2 MiB

Some firmware reports NVMe maximum transfer sizes that follow the drive capability. When those values are very large, the block layer may build I/O that this driver cannot handle, which can cause a kernel oops.

When an NVMe device is set up, cap how large a single transfer may be to the smaller of the firmware-reported limit and roughly two mebibytes with a small margin. If no valid limit is reported, apply the same upper bound.

---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +11 -1
index 6ff788557294..b6abc83d8121 100644
@@ -54,6 +54,7 @@
 #include <linux/interrupt.h>
 #include <linux/raid_class.h>
 #include <linux/unaligned.h>
+#include <linux/sizes.h>
 
 #include "mpt3sas_base.h"
 
@@ -2738,8 +2739,17 @@ scsih_sdev_configure(struct scsi_device *sdev, struct queue_limits *lim)
 				pcie_device->enclosure_level,
 				pcie_device->connector_name);
 
+		/*
+		 * Firmware may report NVMe MDTS from the drive; values above
+		 * what the driver can handle can cause a kernel oops. Cap queue
+		 * I/O in sectors to min(MDTS, 2 MiB - 4096 B).
+		 */
 		if (pcie_device->nvme_mdts)
-			lim->max_hw_sectors = pcie_device->nvme_mdts / 512;
+			lim->max_hw_sectors = min_t(u32,
+					pcie_device->nvme_mdts / 512,
+					(SZ_2M / 512) - 8);
+		else
+			lim->max_hw_sectors = (SZ_2M / 512) - 8;
 
 		pcie_device_put(pcie_device);
 		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
Signature · 1 line
2.47.3
#1 lore raw
Quoted 33 lines · Ranjan Kumar

Some firmware reports NVMe maximum transfer sizes that follow the drive capability. When those values are very large, the block layer may build I/O that this driver cannot handle, which can cause a kernel oops.

When an NVMe device is set up, cap how large a single transfer may be to the smaller of the firmware-reported limit and roughly two mebibytes with a small margin. If no valid limit is reported, apply the same upper bound.

---
 drivers/scsi/mpt3sas/mpt3sas_scsih.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +1 -0
index 6ff788557294..b6abc83d8121 100644
@@ -54,6 +54,7 @@
 #include <linux/interrupt.h>
 #include <linux/raid_class.h>
 #include <linux/unaligned.h>
+#include <linux/sizes.h>
 
 #include "mpt3sas_base.h"
 

drivers/scsi/mpt3sas/mpt3sas_scsih.c scsih_sdev_configure(struct scsi_device *sdev, struct queue_limits *lim) · line 2746

Quoted 8 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +5 -0
@@ -2738,8 +2739,17 @@ scsih_sdev_configure(struct scsi_device *sdev, struct queue_limits *lim)
 				pcie_device->enclosure_level,
 				pcie_device->connector_name);
 
+		/*
+		 * Firmware may report NVMe MDTS from the drive; values above
+		 * what the driver can handle can cause a kernel oops. Cap queue
+		 * I/O in sectors to min(MDTS, 2 MiB - 4096 B).
+		 */

This comment has some grammar issues and is really hard to parse/understand.

drivers/scsi/mpt3sas/mpt3sas_scsih.c scsih_sdev_configure(struct scsi_device *sdev, struct queue_limits *lim) · line 2752

Quoted 7 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +5 -1
 		if (pcie_device->nvme_mdts)
-			lim->max_hw_sectors = pcie_device->nvme_mdts / 512;
+			lim->max_hw_sectors = min_t(u32,
+					pcie_device->nvme_mdts / 512,
+					(SZ_2M / 512) - 8);
+		else
+			lim->max_hw_sectors = (SZ_2M / 512) - 8;

I am very confused here: SZ_2MB assumes that you have an SSD with a minimum page size of 4K, which can fit 4K / 8 = 512 PRP entries, each referencing 4K (one page), so a maximum of 2MiB. However, if I am not mistaken, there is nothing in nvme specs that forces the MPS field to be 0 (which leads to a page size of 4K). So this seems incorrect to me, even though that will probably work for the vast majority of SSDs out there, some exotic ones will not be correctly supported.

Keith ? Am I missing something here ?

Or do we simply do not care about SSDs with a minimum page size > 4K having their maximum command size truncated ?

Quoted 3 lines
drivers/scsi/mpt3sas/mpt3sas_scsih.c +0 -0
 		pcie_device_put(pcie_device);
 		spin_unlock_irqrestore(&ioc->pcie_device_lock, flags);
Signature · 2 lines
Damien Le Moal
Western Digital Research
#2 lore raw
Quoted 15 lines · Damien Le Moal

drivers/scsi/mpt3sas/mpt3sas_scsih.c scsih_sdev_configure(struct scsi_device *sdev, struct queue_limits *lim) · line 2752

Quoted 7 lines · Ranjan Kumar
drivers/scsi/mpt3sas/mpt3sas_scsih.c +5 -1
 		if (pcie_device->nvme_mdts)
-			lim->max_hw_sectors = pcie_device->nvme_mdts / 512;
+			lim->max_hw_sectors = min_t(u32,
+					pcie_device->nvme_mdts / 512,
+					(SZ_2M / 512) - 8);
+		else
+			lim->max_hw_sectors = (SZ_2M / 512) - 8;

I am very confused here: SZ_2MB assumes that you have an SSD with a minimum page size of 4K, which can fit 4K / 8 = 512 PRP entries, each referencing 4K (one page), so a maximum of 2MiB. However, if I am not mistaken, there is nothing in nvme specs that forces the MPS field to be 0 (which leads to a page size of 4K).

Quoted 7 lines

So this seems incorrect to me, even though that will probably work for the vast majority of SSDs out there, some exotic ones will not be correctly supported.

Keith ? Am I missing something here ?

Or do we simply do not care about SSDs with a minimum page size > 4K having their maximum command size truncated ?

Spec doesn't require it, but industry converged on that as always being the minimum supported page size. The nvme driver rejects any device that doesn't support 4k pages because they can't be reliably supported on a lot of archs, even ones with larger page sizes. So it should be a safe assumption that everyone supports 4k since no on is complaining. :)

On the patch, I initially left the "- 8" in the calculation to account for page offsets. But it's not necessary because that gets absorbed in PRP1 within the command, so we'd have at most 512 entries in the PRP list for a 2M transfer.

author maintainer
#3 lore raw

Hi Damien and Keith,

Thank you both for the review and the clarifications.I will submit the v2 patch.

Quoted 35 lines · Keith Busch
Quoted 15 lines · Damien Le Moal

drivers/scsi/mpt3sas/mpt3sas_scsih.c scsih_sdev_configure(struct scsi_device *sdev, struct queue_limits *lim) · line 2752

Quoted 7 lines · Ranjan Kumar
            if (pcie_device->nvme_mdts)
drivers/scsi/mpt3sas/mpt3sas_scsih.c +5 -1
-                   lim->max_hw_sectors = pcie_device->nvme_mdts / 512;
+                   lim->max_hw_sectors = min_t(u32,
+                                   pcie_device->nvme_mdts / 512,
+                                   (SZ_2M / 512) - 8);
+           else
+                   lim->max_hw_sectors = (SZ_2M / 512) - 8;

I am very confused here: SZ_2MB assumes that you have an SSD with a minimum page size of 4K, which can fit 4K / 8 = 512 PRP entries, each referencing 4K (one page), so a maximum of 2MiB. However, if I am not mistaken, there is nothing in nvme specs that forces the MPS field to be 0 (which leads to a page size of 4K).

Quoted 7 lines

So this seems incorrect to me, even though that will probably work for the vast majority of SSDs out there, some exotic ones will not be correctly supported.

Keith ? Am I missing something here ?

Or do we simply do not care about SSDs with a minimum page size > 4K having their maximum command size truncated ?

Spec doesn't require it, but industry converged on that as always being the minimum supported page size. The nvme driver rejects any device that doesn't support 4k pages because they can't be reliably supported on a lot of archs, even ones with larger page sizes. So it should be a safe assumption that everyone supports 4k since no on is complaining. :)

On the patch, I initially left the "- 8" in the calculation to account for page offsets. But it's not necessary because that gets absorbed in PRP1 within the command, so we'd have at most 512 entries in the PRP list for a 2M transfer.