RDMA: Support CQs with user memory
Subsystems (2)
Michael Margolin
maintainercover RDMA: Support CQs with user memory
This series is adding a common way for creating CQs on top of preallocated memory supplied by userspace. The memory buffer can be provided as a virtual address or by dmabuf, in both cases a umem object is created for driver's use.
EFA is the first to support this new interface, new drivers or existing drivers that want/need to get the memory from userspace are expected to use this flow.
It's follow-up of this discussion:
https://lore.kernel.org/all/[email protected]/
Thanks
Michael Margolin (2):
RDMA/uverbs: Add a common way to create CQ with umem RDMA/efa: Add CQ with external memory support
drivers/infiniband/core/device.c | 3 + drivers/infiniband/core/uverbs_std_types_cq.c | 82 ++++++++++++++++++- drivers/infiniband/hw/efa/efa_main.c | 1 + drivers/infiniband/hw/efa/efa_verbs.c | 47 ++++++++--- include/rdma/ib_verbs.h | 6 ++ include/uapi/rdma/efa-abi.h | 3 +- include/uapi/rdma/ib_user_ioctl_cmds.h | 4 + 7 files changed, 129 insertions(+), 17 deletions(-)
Signature · 1 line
2.47.1
Michael Margolin
maintainerPATCH 1/2 RDMA/uverbs: Add a common way to create CQ with umem
Add ioctl command attributes and a common handling for the option to create CQs with memory buffers passed from userspace. When required attributes are supplied, create umem for driver use and store it in CQ context.
The extension enables creation of CQs on top of preallocated CPU virtual or device memory buffers, by supplying VA or dmabuf fd, in a common way. At first stage the command fails for any driver that didn't explicitly state its support by setting a flag in the ops struct.
--- drivers/infiniband/core/device.c | 3 + drivers/infiniband/core/uverbs_std_types_cq.c | 82 ++++++++++++++++++- include/rdma/ib_verbs.h | 6 ++ include/uapi/rdma/ib_user_ioctl_cmds.h | 4 + 4 files changed, 91 insertions(+), 4 deletions(-)
drivers/infiniband/core/device.c +3 -0
@@ -2667,6 +2667,9 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops) dev_ops->uverbs_no_driver_id_binding |= ops->uverbs_no_driver_id_binding; + dev_ops->uverbs_support_cq_with_umem |= + ops->uverbs_support_cq_with_umem; +SET_DEVICE_OP (dev_ops, add_gid);SET_DEVICE_OP (dev_ops, add_sub_dev);SET_DEVICE_OP (dev_ops, advise_mr);
drivers/infiniband/core/uverbs_std_types_cq.c +78 -4
112 lines not on this page · raw message
include/rdma/ib_verbs.h +6 -0
@@ -1625,6 +1625,7 @@ struct ib_cq { };struct workqueue_struct *comp_wq;struct dim *dim; +struct ib_umem *umem;/* updated only by trace points */ ktime_t timestamp;
@@ -2343,6 +2344,11 @@ struct ib_device_ops {enum rdma_driver_id driver_id;u32 uverbs_abi_ver; unsigned int uverbs_no_driver_id_binding:1; +/* +* Driver gets ownership over the umem and is responsible for releasing +* it on CQ destroy or when it's no longer needed. +*/ + unsigned int uverbs_support_cq_with_umem:1;/* * NOTE: New drivers should not make use of device_group; instead new
include/uapi/rdma/ib_user_ioctl_cmds.h +4 -0
@@ -105,6 +105,10 @@ enum uverbs_attrs_create_cq_cmd_attr_ids {UVERBS_ATTR_CREATE_CQ_FLAGS ,UVERBS_ATTR_CREATE_CQ_RESP_CQE ,UVERBS_ATTR_CREATE_CQ_EVENT_FD , +UVERBS_ATTR_CREATE_CQ_BUFFER_VA , +UVERBS_ATTR_CREATE_CQ_BUFFER_LENGTH , +UVERBS_ATTR_CREATE_CQ_BUFFER_FD , + UVERBS_ATTR_CREATE_CQ_BUFFER_OFFSET, }; enum uverbs_attrs_destroy_cq_cmd_attr_ids {
Signature · 1 line
2.47.1
Jason Gunthorpe
maintainerRe: [PATCH for-next 1/2] RDMA/uverbs: Add a common way to create CQ with umem
Quoted 22 lines · Michael Margolin
Add ioctl command attributes and a common handling for the option to create CQs with memory buffers passed from userspace. When required attributes are supplied, create umem for driver use and store it in CQ context.
The extension enables creation of CQs on top of preallocated CPU virtual or device memory buffers, by supplying VA or dmabuf fd, in a common way. At first stage the command fails for any driver that didn't explicitly state its support by setting a flag in the ops struct.Signed-off-by Michael Margolin <[email protected]>--- drivers/infiniband/core/device.c | 3 + drivers/infiniband/core/uverbs_std_types_cq.c | 82 ++++++++++++++++++- include/rdma/ib_verbs.h | 6 ++ include/uapi/rdma/ib_user_ioctl_cmds.h | 4 + 4 files changed, 91 insertions(+), 4 deletions(-)drivers/infiniband/core/device.c +0
-0
drivers/infiniband/core/device.c void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops) · line 2671
Quoted 5 lines
drivers/infiniband/core/device.c +2
-0@@ -2667,6 +2667,9 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops) dev_ops->uverbs_no_driver_id_binding |= ops->uverbs_no_driver_id_binding; + dev_ops->uverbs_support_cq_with_umem |= + ops->uverbs_support_cq_with_umem;
This seems to have turned out quite nice!
I might just suggest a tweak to streamline this flow
Add:
+ int (*create_cq_umem)(struct ib_cq *cq,
+ const struct ib_cq_init_attr *attr, + struct ib_umem *umem, + struct uverbs_attr_bundle *attrs);
Instead of the uverbs_support_cq_with_umem
Then the core code would check the two ops and if only umem is present it will insist on the new attributes, if it is not present it will refuse them otherwise it will call the correct op.
In the driver the create_cq would obtain the umem from the private attrs and then call create_cq_umem() in the same way
So it becomes quite easy to just reorganize the drivers to support this.
Jason
Margolin, Michael
maintainerRe: [PATCH for-next 1/2] RDMA/uverbs: Add a common way to create CQ with umem
Quoted 50 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 22 lines · Michael Margolin
Add ioctl command attributes and a common handling for the option to create CQs with memory buffers passed from userspace. When required attributes are supplied, create umem for driver use and store it in CQ context.
The extension enables creation of CQs on top of preallocated CPU virtual or device memory buffers, by supplying VA or dmabuf fd, in a common way. At first stage the command fails for any driver that didn't explicitly state its support by setting a flag in the ops struct.Signed-off-by Michael Margolin <[email protected]>--- drivers/infiniband/core/device.c | 3 + drivers/infiniband/core/uverbs_std_types_cq.c | 82 ++++++++++++++++++- include/rdma/ib_verbs.h | 6 ++ include/uapi/rdma/ib_user_ioctl_cmds.h | 4 + 4 files changed, 91 insertions(+), 4 deletions(-)drivers/infiniband/core/device.c +0
-0drivers/infiniband/core/device.c void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops) · line 2671
Quoted 5 lines
drivers/infiniband/core/device.c +2
-0@@ -2667,6 +2667,9 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops) dev_ops->uverbs_no_driver_id_binding |= ops->uverbs_no_driver_id_binding; + dev_ops->uverbs_support_cq_with_umem |= + ops->uverbs_support_cq_with_umem;This seems to have turned out quite nice!
I might just suggest a tweak to streamline this flow
Add:
(quoted diff) +4
-0+ int (*create_cq_umem)(struct ib_cq *cq, + const struct ib_cq_init_attr *attr, + struct ib_umem *umem, + struct uverbs_attr_bundle *attrs);Instead of the
uverbs_support_cq_with_umemThen the core code would check the two ops and if only umem is present it will insist on the new attributes, if it is not present it will refuse them otherwise it will call the correct op.
Quoted 7 lines
In the driver the
create_cqwould obtain the umem from the private attrs and then callcreate_cq_umem()in the same waySo it becomes quite easy to just reorganize the drivers to support this.
Jason
Generally I would say that adding a new op for an additional optional param isn't scalable but I implemented your suggestion and I do think it makes things more structured for drivers as well as clarifying umem ownership.
Sending v2 series.
Michael
Michael Margolin
maintainerPATCH 2/2 RDMA/efa: Add CQ with external memory support
In reply to Michael MargolinAdd an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset or a VA. One of the possible usages is creating CQs that reside in accelerator memory, allowing low latency asynchronous direct polling from the accelerator device. Add a capability bit to reflect on the feature support.
--- drivers/infiniband/hw/efa/efa_main.c | 1 + drivers/infiniband/hw/efa/efa_verbs.c | 47 ++++++++++++++++++++------- include/uapi/rdma/efa-abi.h | 3 +- 3 files changed, 38 insertions(+), 13 deletions(-)
drivers/infiniband/hw/efa/efa_main.c +1 -0
@@ -366,6 +366,7 @@ static const struct ib_device_ops efa_dev_ops = { .owner = THIS_MODULE, .driver_id = RDMA_DRIVER_EFA, .uverbs_abi_ver = EFA_UVERBS_ABI_VERSION, + .uverbs_support_cq_with_umem = 1, .alloc_hw_port_stats = efa_alloc_hw_port_stats, .alloc_hw_device_stats = efa_alloc_hw_device_stats,
drivers/infiniband/hw/efa/efa_verbs.c +35 -12
91 lines not on this page · raw message
include/uapi/rdma/efa-abi.h +2 -1
@@ -1,6 +1,6 @@/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) */ /* -* Copyright 2018-2024 Amazon.com, Inc. or its affiliates. All rights reserved. +* Copyright 2018-2025 Amazon.com, Inc. or its affiliates. All rights reserved. */ #ifndef EFA_ABI_USER_H
@@ -131,6 +131,7 @@ enum {EFA_QUERY_DEVICE_CAPS_DATA_POLLING_128 =1 <<4 ,EFA_QUERY_DEVICE_CAPS_RDMA_WRITE =1 <<5 ,EFA_QUERY_DEVICE_CAPS_UNSOLICITED_WRITE_RECV =1 <<6 , + EFA_QUERY_DEVICE_CAPS_CQ_WITH_EXT_MEM = 1 << 7, }; struct efa_ibv_ex_query_device_resp {
Signature · 1 line
2.47.1
Jason Gunthorpe
maintainerRe: [PATCH for-next 2/2] RDMA/efa: Add CQ with external memory support
drivers/infiniband/hw/efa/efa_verbs.c int efa_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, · line 1218
Quoted 7 lines · Michael Margolin
drivers/infiniband/hw/efa/efa_verbs.c +7
-0+if (umem) { + umem_sgl = ibcq->umem->sgt_append.sgt.sgl; +if (sg_dma_len (umem_sgl) <ib_umem_offset (umem) + cq->size) { +ibdev_dbg (&dev->ibdev,"Non contiguous CQ unsupported\n" ); + err = -EINVAL ; + goto err_free_mem; + }
I'd rather see this call ib_umem_find_best_pgsz()
Maybe like:
len = ib_umem_offset(umem) + cq->size
pgsz = roundup_pow_of_two(len);
ib_umem_find_best_pgoff(umem, pgsz, U64_MAX);
rdma_umem_for_each_dma_block(umem, biter, pgsz) {
dma = rdma_block_iter_dma_address(&biter) + ib_umem_dma_offset(umem, pgsz);
break;}
It turns out the scatterlists can be irregular in some cases so this will handle that properly while the little test above cannot.
And maybe the above thing could be a little helper:
bool ib_umem_get_contiguous_dma(umem, &dma_addr)
Also I am trying hard to keep scatterlist APIS out of the drivers.
Jason
Margolin, Michael
maintainerRe: [PATCH for-next 2/2] RDMA/efa: Add CQ with external memory support
Quoted 26 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
drivers/infiniband/hw/efa/efa_verbs.c int efa_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, · line 1218
Quoted 7 lines · Michael Margolin
drivers/infiniband/hw/efa/efa_verbs.c +7
-0+if (umem) { + umem_sgl = ibcq->umem->sgt_append.sgt.sgl; +if (sg_dma_len (umem_sgl) <ib_umem_offset (umem) + cq->size) { +ibdev_dbg (&dev->ibdev,"Non contiguous CQ unsupported\n" ); + err = -EINVAL ; + goto err_free_mem; + }I'd rather see this call
ib_umem_find_best_pgsz()Maybe like:
len = ib_umem_offset(umem) + cq->size pgsz = roundup_pow_of_two(len); ib_umem_find_best_pgoff(umem, pgsz, U64_MAX); rdma_umem_for_each_dma_block(umem, biter, pgsz) { dma = rdma_block_iter_dma_address(&biter) + ib_umem_dma_offset(umem, pgsz); break; }It turns out the scatterlists can be irregular in some cases so this will handle that properly while the little test above cannot.
Quoted 7 lines
And maybe the above thing could be a little helper:
bool
ib_umem_get_contiguous_dma(umem, &dma_addr)Also I am trying hard to keep scatterlist APIS out of the drivers.
Jason
Some disadvantage of this approach is that if the provided memory is contiguous in its first "roundup_pow_of_two(len)" bytes but consists of additional smaller or unaligned sgl entries, we will refuse to use it.
I will try to compile some combination of the two approaches into a helper function.
Michael
Jason Gunthorpe
maintainerRe: [PATCH for-next 2/2] RDMA/efa: Add CQ with external memory support
Quoted 32 lines · Michael
Quoted 30 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
drivers/infiniband/hw/efa/efa_verbs.c int efa_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, · line 1218
Quoted 7 lines · Michael Margolin
drivers/infiniband/hw/efa/efa_verbs.c +7
-0+if (umem) { + umem_sgl = ibcq->umem->sgt_append.sgt.sgl; +if (sg_dma_len (umem_sgl) <ib_umem_offset (umem) + cq->size) { +ibdev_dbg (&dev->ibdev,"Non contiguous CQ unsupported\n" ); + err = -EINVAL ; + goto err_free_mem; + }I'd rather see this call
ib_umem_find_best_pgsz()Maybe like:
len = ib_umem_offset(umem) + cq->size pgsz = roundup_pow_of_two(len); ib_umem_find_best_pgoff(umem, pgsz, U64_MAX); rdma_umem_for_each_dma_block(umem, biter, pgsz) { dma = rdma_block_iter_dma_address(&biter) + ib_umem_dma_offset(umem, pgsz); break; }It turns out the scatterlists can be irregular in some cases so this will handle that properly while the little test above cannot.
And maybe the above thing could be a little helper:
bool
ib_umem_get_contiguous_dma(umem, &dma_addr)
Quoted 8 lines
Quoted 3 lines
Also I am trying hard to keep scatterlist APIS out of the drivers.
Jason
Some disadvantage of this approach is that if the provided memory is contiguous in its first "
roundup_pow_of_two(len)" bytes but consists of additional smaller or unaligned sgl entries, we will refuse to use it.
I intended "len" to be large enough that there was no additional portion that is not covered.
Jason