Skip to contentSkip to conversation

Fixes related to the DMA max_segment_size parameter

PATCH Applied Bart Van Assche 2 participants 6 messages Oct 25, 2019 – Oct 28, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover Fixes related to the DMA max_segment_size parameter

Hi Jason,

These four patches are what I came up with while analyzing a kernel warning triggered by the ib_srp kernel driver. Please consider these patches for inclusion in the Linux kernel.

Thanks,

Bart.

Changes compared to v1:
- Restored the dma_set_max_seg_size() call in setup_dma_device(). Reordered this patch series to keep it bisectable.

Bart Van Assche (4):
RDMA/core: Fix ib_dma_max_seg_size()
rdma_rxe: Increase DMA max_segment_size parameter
siw: Increase DMA max_segment_size parameter RDMA/core: Set DMA parameters correctly

 drivers/infiniband/core/device.c      | 16 ++++++++++++++--
 drivers/infiniband/sw/rxe/rxe_verbs.c |  3 +++
 drivers/infiniband/sw/rxe/rxe_verbs.h |  1 +
 drivers/infiniband/sw/siw/siw.h       |  1 +
 drivers/infiniband/sw/siw/siw_main.c  |  3 +++
 include/rdma/ib_verbs.h               |  4 +---
 6 files changed, 23 insertions(+), 5 deletions(-)
#1 lore raw

PATCH v2 1/4 RDMA/core: Fix ib_dma_max_seg_size()

If dev->dma_device->params == NULL then the maximum DMA segment size is 64 KB. See also the dma_get_max_seg_size() implementation. This patch fixes the following kernel warning:

DMA-API: infiniband rxe0: mapping sg segment longer than device claims to support [len=126976] [max=65536]

WARNING: CPU: 4 PID: 4848 at kernel/dma/debug.c:1220 debug_dma_map_sg+0x3d9/0x450
RIP: 0010:debug_dma_map_sg+0x3d9/0x450
Call Trace:
 srp_queuecommand+0x626/0x18d0 [ib_srp]
 scsi_queue_rq+0xd02/0x13e0 [scsi_mod]
 __blk_mq_try_issue_directly+0x2b3/0x3f0
 blk_mq_request_issue_directly+0xac/0xf0
 blk_insert_cloned_request+0xdf/0x170
 dm_mq_queue_rq+0x43d/0x830 [dm_mod]
 __blk_mq_try_issue_directly+0x2b3/0x3f0
 blk_mq_request_issue_directly+0xac/0xf0
 blk_mq_try_issue_list_directly+0xb8/0x170
 blk_mq_sched_insert_requests+0x23c/0x3b0
 blk_mq_flush_plug_list+0x529/0x730
 blk_flush_plug_list+0x21f/0x260
 blk_mq_make_request+0x56b/0xf20
 generic_make_request+0x196/0x660
 submit_bio+0xae/0x290
 blkdev_direct_IO+0x822/0x900
 generic_file_direct_write+0x110/0x200
 __generic_file_write_iter+0x124/0x2a0
 blkdev_write_iter+0x168/0x270
 aio_write+0x1c4/0x310
 io_submit_one+0x971/0x1390
 __x64_sys_io_submit+0x12a/0x390
 do_syscall_64+0x6f/0x2e0
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
Cc Christoph Hellwig <[email protected]>
Fixes 0b5cb3300ae5 ("RDMA/srp: Increase max_segment_size")
---
 include/rdma/ib_verbs.h | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
include/rdma/ib_verbs.h +1 -3
index cca9985b4cbc..0626b62ed107 100644
@@ -4057,9 +4057,7 @@ static inline void ib_dma_unmap_sg_attrs(struct ib_device *dev,
  */
 static inline unsigned int ib_dma_max_seg_size(struct ib_device *dev)
 {
-	struct device_dma_parameters *p = dev->dma_device->dma_parms;
-
-	return p ? p->max_segment_size : UINT_MAX;
+	return dma_get_max_seg_size(dev->dma_device);
 }
 
 /**
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
#2 lore raw

PATCH v2 2/4 rdma_rxe: Increase DMA max_segment_size parameter

In reply to Bart Van Assche

Increase the DMA max_segment_size parameter from 64 KB to 2 GB.

Cc Christoph Hellwig <[email protected]>
---
 drivers/infiniband/sw/rxe/rxe_verbs.c | 3 +++
 drivers/infiniband/sw/rxe/rxe_verbs.h | 1 +
 2 files changed, 4 insertions(+)
drivers/infiniband/sw/rxe/rxe_verbs.c +3 -0
index fa47bdcc7f54..9dd4bd7aea92 100644
@@ -1175,6 +1175,9 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name)
 	addrconf_addr_eui48((unsigned char *)&dev->node_guid,
 			    rxe->ndev->dev_addr);
 	dev->dev.dma_ops = &dma_virt_ops;
+	dev->dev.dma_parms = &rxe->dma_parms;
+	rxe->dma_parms = (struct device_dma_parameters)
+		{ .max_segment_size = SZ_2G };
 	dma_coerce_mask_and_coherent(&dev->dev,
 				     dma_get_required_mask(&dev->dev));
 
drivers/infiniband/sw/rxe/rxe_verbs.h +1 -0
index 5c4b2239129c..95834206c80c 100644
@@ -384,6 +384,7 @@ struct rxe_port {
 struct rxe_dev {
 	struct ib_device	ib_dev;
 	struct ib_device_attr	attr;
+	struct device_dma_parameters dma_parms;
 	int			max_ucontext;
 	int			max_inline_data;
 	struct mutex	usdev_lock;
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
#3 lore raw

PATCH v2 3/4 siw: Increase DMA max_segment_size parameter

In reply to Bart Van Assche

Increase the DMA max_segment_size parameter from 64 KB to 2 GB.

Cc Christoph Hellwig <[email protected]>
Cc Bernard Metzler <[email protected]>
---
 drivers/infiniband/sw/siw/siw.h      | 1 +
 drivers/infiniband/sw/siw/siw_main.c | 3 +++
 2 files changed, 4 insertions(+)
drivers/infiniband/sw/siw/siw.h +1 -0
index dba4535494ab..1ea3ed249e7b 100644
@@ -70,6 +70,7 @@ struct siw_pd {
 
 struct siw_device {
 	struct ib_device base_dev;
+	struct device_dma_parameters dma_parms;
 	struct net_device *netdev;
 	struct siw_dev_cap attrs;
 
drivers/infiniband/sw/siw/siw_main.c +3 -0
index b8fe43074ad6..48e45a852b51 100644
@@ -382,6 +382,9 @@ static struct siw_device *siw_device_create(struct net_device *netdev)
 	base_dev->phys_port_cnt = 1;
 	base_dev->dev.parent = parent;
 	base_dev->dev.dma_ops = &dma_virt_ops;
+	base_dev->dev.dma_parms = &sdev->dma_parms;
+	sdev->dma_parms = (struct device_dma_parameters)
+		{ .max_segment_size = SZ_2G };
 	base_dev->num_comp_vectors = num_possible_cpus();
 
 	ib_set_device_ops(base_dev, &siw_device_ops);
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
#4 lore raw

PATCH v2 4/4 RDMA/core: Set DMA parameters correctly

In reply to Bart Van Assche

The dma_set_max_seg_size() call in setup_dma_device() does not have any effect since device->dev.dma_parms is NULL. Fix this by initializing device->dev.dma_parms first.

Cc Michael J. Ruhl <[email protected]>
Cc Ira Weiny <[email protected]>
Cc Adit Ranadive <[email protected]>
Cc Shiraz Saleem <[email protected]>
Cc Gal Pressman <[email protected]>
Cc Selvin Xavier <[email protected]>
Fixes d10bcf947a3e ("RDMA/umem: Combine contiguous PAGE_SIZE regions in SGEs")
---
 drivers/infiniband/core/device.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
drivers/infiniband/core/device.c +14 -2
index eb35b663a742..a93c23867fb5 100644
@@ -1196,9 +1196,21 @@ static void setup_dma_device(struct ib_device *device)
 		WARN_ON_ONCE(!parent);
 		device->dma_device = parent;
 	}
-	/* Setup default max segment size for all IB devices */
-	dma_set_max_seg_size(device->dma_device, SZ_2G);
 
+	if (!device->dev.dma_parms) {
+		if (parent) {
+			/*
+			 * The caller did not provide DMA parameters, so
+			 * 'parent' probably represents a PCI device. The PCI
+			 * core sets the maximum segment size to 64
+			 * KB. Increase this parameter to 2 GB.
+			 */
+			device->dev.dma_parms = parent->dma_parms;
+			dma_set_max_seg_size(device->dma_device, SZ_2G);
+		} else {
+			WARN_ON_ONCE(true);
+		}
+	}
 }
 
 /*
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
#5 lore raw
In reply to Bart Van Assche
Quoted 14 lines · Bart Van Assche

Hi Jason,

These four patches are what I came up with while analyzing a kernel warning triggered by the ib_srp kernel driver. Please consider these patches for inclusion in the Linux kernel.

Thanks,

Bart.

Changes compared to v1:
- Restored the dma_set_max_seg_size() call in setup_dma_device(). Reordered this patch series to keep it bisectable.

Quoted 5 lines

Bart Van Assche (4):
RDMA/core: Fix ib_dma_max_seg_size()
rdma_rxe: Increase DMA max_segment_size parameter
siw: Increase DMA max_segment_size parameter RDMA/core: Set DMA parameters correctly

Applied to for-next, thanks

Jason