Fixes related to the DMA max_segment_size parameter
Subsystems (4)
v2 cover Fixes related to the DMA max_segment_size parameter
Hi Jason,
These four patches are what I came up with while analyzing a kernel warning triggered by the ib_srp kernel driver. Please consider these patches for inclusion in the Linux kernel.
Thanks,
Bart.
Changes compared to v1:
- Restored the dma_set_max_seg_size() call in setup_dma_device(). Reordered this patch series to keep it bisectable.
Bart Van Assche (4):
RDMA/core: Fix ib_dma_max_seg_size()
rdma_rxe: Increase DMA max_segment_size parameter
siw: Increase DMA max_segment_size parameter RDMA/core: Set DMA parameters correctly
drivers/infiniband/core/device.c | 16 ++++++++++++++-- drivers/infiniband/sw/rxe/rxe_verbs.c | 3 +++ drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + drivers/infiniband/sw/siw/siw.h | 1 + drivers/infiniband/sw/siw/siw_main.c | 3 +++ include/rdma/ib_verbs.h | 4 +--- 6 files changed, 23 insertions(+), 5 deletions(-)
PATCH v2 1/4 RDMA/core: Fix ib_dma_max_seg_size()
If dev->dma_device->params == NULL then the maximum DMA segment size is 64 KB. See also the dma_get_max_seg_size() implementation. This patch fixes the following kernel warning:
DMA-API: infiniband rxe0: mapping sg segment longer than device claims to support [len=126976] [max=65536]
WARNING: CPU: 4 PID: 4848 at kernel/dma/debug.c:1220 debug_dma_map_sg+0x3d9/0x450 RIP: 0010:debug_dma_map_sg+0x3d9/0x450 Call Trace: srp_queuecommand+0x626/0x18d0 [ib_srp] scsi_queue_rq+0xd02/0x13e0 [scsi_mod] __blk_mq_try_issue_directly+0x2b3/0x3f0 blk_mq_request_issue_directly+0xac/0xf0 blk_insert_cloned_request+0xdf/0x170 dm_mq_queue_rq+0x43d/0x830 [dm_mod] __blk_mq_try_issue_directly+0x2b3/0x3f0 blk_mq_request_issue_directly+0xac/0xf0 blk_mq_try_issue_list_directly+0xb8/0x170 blk_mq_sched_insert_requests+0x23c/0x3b0 blk_mq_flush_plug_list+0x529/0x730 blk_flush_plug_list+0x21f/0x260 blk_mq_make_request+0x56b/0xf20 generic_make_request+0x196/0x660 submit_bio+0xae/0x290 blkdev_direct_IO+0x822/0x900 generic_file_direct_write+0x110/0x200 __generic_file_write_iter+0x124/0x2a0 blkdev_write_iter+0x168/0x270 aio_write+0x1c4/0x310 io_submit_one+0x971/0x1390 __x64_sys_io_submit+0x12a/0x390 do_syscall_64+0x6f/0x2e0 entry_SYSCALL_64_after_hwframe+0x49/0xbe
--- include/rdma/ib_verbs.h | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
include/rdma/ib_verbs.h +1 -3
@@ -4057,9 +4057,7 @@ static inline void ib_dma_unmap_sg_attrs(struct ib_device *dev,*/ static inline unsigned int ib_dma_max_seg_size (struct ib_device *dev) { -struct device_dma_parameters *p = dev->dma_device->dma_parms; - -return p ? p->max_segment_size :UINT_MAX ; +return dma_get_max_seg_size (dev->dma_device); }/**
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 2/4 rdma_rxe: Increase DMA max_segment_size parameter
In reply to Bart Van AsscheIncrease the DMA max_segment_size parameter from 64 KB to 2 GB.
--- drivers/infiniband/sw/rxe/rxe_verbs.c | 3 +++ drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 2 files changed, 4 insertions(+)
drivers/infiniband/sw/rxe/rxe_verbs.c +3 -0
@@ -1175,6 +1175,9 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name)addrconf_addr_eui48 ((unsigned char *)&dev->node_guid, rxe->ndev->dev_addr); dev->dev.dma_ops = &dma_virt_ops; + dev->dev.dma_parms = &rxe->dma_parms; + rxe->dma_parms = (struct device_dma_parameters ) + { .max_segment_size =SZ_2G };dma_coerce_mask_and_coherent (&dev->dev,dma_get_required_mask (&dev->dev));
drivers/infiniband/sw/rxe/rxe_verbs.h +1 -0
@@ -384,6 +384,7 @@ struct rxe_port {struct rxe_dev {struct ib_device ib_dev;struct ib_device_attr attr; +struct device_dma_parameters dma_parms;int max_ucontext;int max_inline_data;struct mutex usdev_lock;
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 3/4 siw: Increase DMA max_segment_size parameter
In reply to Bart Van AsscheIncrease the DMA max_segment_size parameter from 64 KB to 2 GB.
--- drivers/infiniband/sw/siw/siw.h | 1 + drivers/infiniband/sw/siw/siw_main.c | 3 +++ 2 files changed, 4 insertions(+)
drivers/infiniband/sw/siw/siw.h +1 -0
@@ -70,6 +70,7 @@ struct siw_pd {struct siw_device {struct ib_device base_dev; +struct device_dma_parameters dma_parms;struct net_device *netdev;struct siw_dev_cap attrs;
drivers/infiniband/sw/siw/siw_main.c +3 -0
@@ -382,6 +382,9 @@ static struct siw_device *siw_device_create(struct net_device *netdev) base_dev->phys_port_cnt =1 ; base_dev->dev.parent = parent; base_dev->dev.dma_ops = &dma_virt_ops; + base_dev->dev.dma_parms = &sdev->dma_parms; + sdev->dma_parms = (struct device_dma_parameters ) + { .max_segment_size =SZ_2G }; base_dev->num_comp_vectors =num_possible_cpus ();ib_set_device_ops (base_dev, &siw_device_ops);
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 4/4 RDMA/core: Set DMA parameters correctly
In reply to Bart Van AsscheThe dma_set_max_seg_size() call in setup_dma_device() does not have any effect since device->dev.dma_parms is NULL. Fix this by initializing device->dev.dma_parms first.
--- drivers/infiniband/core/device.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
drivers/infiniband/core/device.c +14 -2
@@ -1196,9 +1196,21 @@ static void setup_dma_device(struct ib_device *device)WARN_ON_ONCE (!parent); device->dma_device = parent; } -/* Setup default max segment size for all IB devices */ -dma_set_max_seg_size (device->dma_device,SZ_2G ); +if (!device->dev.dma_parms) { +if (parent) { +/* +* The caller did not provide DMA parameters, so +* 'parent' probably represents a PCI device. The PCI +* core sets the maximum segment size to 64 +* KB. Increase this parameter to 2 GB. +*/ + device->dev.dma_parms = parent->dma_parms; +dma_set_max_seg_size (device->dma_device,SZ_2G ); + }else { +WARN_ON_ONCE (true); + } + } }/*
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Jason Gunthorpe
AppliedQuoted 14 lines · Bart Van Assche
Hi Jason,
These four patches are what I came up with while analyzing a kernel warning triggered by the
ib_srpkernel driver. Please consider these patches for inclusion in the Linux kernel.Thanks,
Bart.
Changes compared to v1:
- Restored thedma_set_max_seg_size()call insetup_dma_device(). Reordered this patch series to keep it bisectable.
Quoted 5 lines
Bart Van Assche (4):
RDMA/core: Fixib_dma_max_seg_size()
rdma_rxe: Increase DMAmax_segment_sizeparameter
siw: Increase DMAmax_segment_sizeparameter RDMA/core: Set DMA parameters correctly
Applied to for-next, thanks
Jason