RDMA/efa: Add CQ with external memory support
Subsystems (2)
Michael Margolin
maintainerPATCH 1/1 RDMA/efa: Add CQ with external memory support
Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset. Add a capability bit to reflect on the feature support.
--- drivers/infiniband/hw/efa/efa.h | 1 + drivers/infiniband/hw/efa/efa_verbs.c | 60 ++++++++++++++++++++++----- include/uapi/rdma/efa-abi.h | 8 +++- 3 files changed, 58 insertions(+), 11 deletions(-)
drivers/infiniband/hw/efa/efa.h +1 -0
@@ -107,6 +107,7 @@ struct efa_cq {u16 cq_idx;/* NULL when no interrupts requested */ struct efa_eq *eq; +struct ib_umem *umem; };struct efa_qp {
drivers/infiniband/hw/efa/efa_verbs.c +50 -10
98 lines not on this page · raw message
include/uapi/rdma/efa-abi.h +7 -1
@@ -1,6 +1,6 @@/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) */ /* -* Copyright 2018-2024 Amazon.com, Inc. or its affiliates. All rights reserved. +* Copyright 2018-2025 Amazon.com, Inc. or its affiliates. All rights reserved. */ #ifndef EFA_ABI_USER_H
@@ -56,6 +56,7 @@ struct efa_ibv_alloc_pd_resp {enum {EFA_CREATE_CQ_WITH_COMPLETION_CHANNEL =1 <<0 ,EFA_CREATE_CQ_WITH_SGID =1 <<1 , +EFA_CREATE_CQ_WITH_EXT_MEM_DMABUF =1 <<2 , };struct efa_ibv_create_cq {
@@ -64,6 +65,10 @@ struct efa_ibv_create_cq {__u16 num_sub_cqs;__u8 flags;__u8 reserved_58[5 ]; +__aligned_u64 ext_mem_offset; +__aligned_u64 ext_mem_length; +__u32 ext_mem_fd; +__u8 reserved_120[4 ]; };enum {
@@ -131,6 +136,7 @@ enum {EFA_QUERY_DEVICE_CAPS_DATA_POLLING_128 =1 <<4 ,EFA_QUERY_DEVICE_CAPS_RDMA_WRITE =1 <<5 ,EFA_QUERY_DEVICE_CAPS_UNSOLICITED_WRITE_RECV =1 <<6 , + EFA_QUERY_DEVICE_CAPS_CQ_WITH_EXT_MEM_DMABUF = 1 << 7, }; struct efa_ibv_ex_query_device_resp {
Signature · 1 line
2.47.1
Quoted 3 lines · Michael Margolin
Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.
EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.
Thanks
Margolin, Michael
maintainerQuoted 4 lines · Leon Romanovsky
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 8 lines
Quoted 3 lines · Michael Margolin
Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.
EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.
Thanks
I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.
Michael
Quoted 13 lines · Michael
Quoted 11 lines · Leon Romanovsky
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 3 lines · Michael Margolin
Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.
EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.
Quoted 8 lines
Quoted 1 line
Thanks
I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.
I think that this functionality is worth to have as general verb and not DV. mlx5 has something similar and now EFA needs it too.
Let's wait for Jason's response before rushing to implement it.
Thanks
Quoted 3 lines
Michael
Margolin, Michael
maintainerQuoted 24 lines · Leon Romanovsky
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 12 lines · Michael
Quoted 11 lines · Leon Romanovsky
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 3 lines · Michael Margolin
Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.
EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.
Quoted 7 lines
Quoted 1 line
Thanks
I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.
Quoted 6 lines
I think that this functionality is worth to have as general verb and not DV. mlx5 has something similar and now EFA needs it too.
Let's wait for Jason's response before rushing to implement it.
Thanks
Jason, any thoughts on this?
We can probably add optional attributes to create CQ command, but most of the handling will remain vendor specific. I'm not convinced it's beneficial enough.
Michael
Jason Gunthorpe
maintainerQuoted 31 lines · Michael
Quoted 29 lines · Leon Romanovsky
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 12 lines · Michael
Quoted 11 lines · Leon Romanovsky
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 3 lines · Michael Margolin
Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.
EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.
Quoted 7 lines
Quoted 1 line
Thanks
I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.
I think that this functionality is worth to have as general verb and not DV. mlx5 has something similar and now EFA needs it too.
Let's wait for Jason's response before rushing to implement it.
Quoted 7 lines
Quoted 1 line
Thanks
Jason, any thoughts on this?
We can probably add optional attributes to create CQ command, but most of the handling will remain vendor specific. I'm not convinced it's beneficial enough.
I don't know how a general verb would work, CQs in non-dv are polled in software in userspace and the userspace will have trouble reaching into a dmabuf. Plus the entire point of this is usually to write the polling code in a GPU language and run it on a GPU processor.
Meaning I think all users of this will want to use a DV interface from verbs.
At that point, is it worth adding more common verbs support?
Though it is becoming a bit messy that drivers are all open coding creating normal or dmabuf umems. That part might be worth generalizing some more, and then doing QP as well.
Jason
Margolin, Michael
maintainerQuoted 49 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 28 lines · Michael
Quoted 27 lines · Leon Romanovsky
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 12 lines · Michael
Quoted 11 lines · Leon Romanovsky
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 3 lines · Michael Margolin
Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.
EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.
Quoted 7 lines
Quoted 1 line
Thanks
I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.
I think that this functionality is worth to have as general verb and not DV. mlx5 has something similar and now EFA needs it too.
Quoted 8 lines
Quoted 3 lines
Let's wait for Jason's response before rushing to implement it.
Thanks
Jason, any thoughts on this?
We can probably add optional attributes to create CQ command, but most of the handling will remain vendor specific. I'm not convinced it's beneficial enough.
I don't know how a general verb would work, CQs in non-dv are polled in software in userspace and the userspace will have trouble reaching into a dmabuf. Plus the entire point of this is usually to write the polling code in a GPU language and run it on a GPU processor.
Meaning I think all users of this will want to use a DV interface from verbs.
Quoted 7 lines
At that point, is it worth adding more common verbs support?
Though it is becoming a bit messy that drivers are all open coding creating normal or dmabuf umems. That part might be worth generalizing some more, and then doing QP as well.
Jason
Are you suggesting turning mlx5dv_devx_umem_reg into a common verb including the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?
Michael
Jason Gunthorpe
maintainerQuoted 3 lines · Michael
Are you suggesting turning
mlx5dv_devx_umem_reginto a common verb including the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?
I think Leon was, but I'm not sure that is so worthwhile.
I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.
But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?
Jason
Margolin, Michael
maintainerQuoted 16 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 3 lines · Michael
Are you suggesting turning
mlx5dv_devx_umem_reginto a common verb including the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?I think Leon was, but I'm not sure that is so worthwhile.
I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.
Quoted 4 lines
But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?
Jason
Yes, EFA is an example for a driver that doesn't need this on the "standard" flow.
Michael
Margolin, Michael
maintainerQuoted 18 lines · Michael
Quoted 16 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 3 lines · Michael
Are you suggesting turning
mlx5dv_devx_umem_reginto a common verb including the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?I think Leon was, but I'm not sure that is so worthwhile.
I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.
Quoted 9 lines
Quoted 4 lines
But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?
Jason
Yes, EFA is an example for a driver that doesn't need this on the "standard" flow.
Michael
How can we move forward with this patch? It's possible to add additional attributes to the common create CQ ioctl and use it for EFA direct verbs but it won't be easy to move existing drivers to use it.
Michael
Jason Gunthorpe
maintainerQuoted 22 lines · Michael
Quoted 20 lines · Michael
Quoted 17 lines · Jason Gunthorpe
Quoted 4 lines · Michael
Are you suggesting turning
mlx5dv_devx_umem_reginto a common verb including
the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?I think Leon was, but I'm not sure that is so worthwhile.
I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.
But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?
Jason
Quoted 8 lines
Quoted 5 lines
Yes, EFA is an example for a driver that doesn't need this on the "standard" flow.
Michael
How can we move forward with this patch? It's possible to add additional attributes to the common create CQ ioctl and use it for EFA direct verbs but it won't be easy to move existing drivers to use it.
Then it becomes hard to make the other drivers reject those new common attributes :\
Maybe what you have here is the best, but it really does seem unsatisfying from a kernel POV..
Jason
Margolin, Michael
maintainerQuoted 32 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 19 lines · Michael
Quoted 18 lines · Michael
Quoted 17 lines · Jason Gunthorpe
Quoted 4 lines · Michael
Are you suggesting turning
mlx5dv_devx_umem_reginto a common verb including
the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?I think Leon was, but I'm not sure that is so worthwhile.
I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.
But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?
Jason
Quoted 8 lines
Quoted 5 lines
Yes, EFA is an example for a driver that doesn't need this on the "standard" flow.
Michael
How can we move forward with this patch? It's possible to add additional attributes to the common create CQ ioctl and use it for EFA direct verbs but it won't be easy to move existing drivers to use it.
Quoted 7 lines
Then it becomes hard to make the other drivers reject those new common attributes :\
Maybe what you have here is the best, but it really does seem unsatisfying from a kernel POV..
Jason
Apologies for a delayed response, I want to make sure we are aligned on the solution.
I'm going to add a common option in libibverbs for providers to pass buffer info for create CQ ioctl and when the new ioctl attributes are set to the kernel, handle it in core code by creating a umem object and storing it in CQ context for drivers use.
Does this make sense?
Michael