Skip to contentSkip to conversation

RDMA/efa: Add CQ with external memory support

PATCH Superseded Michael Margolin 3 participants 12 messages May 15, 2025 – Jun 24, 2025

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Michael Margolin

author maintainer
#0 lore raw

PATCH 1/1 RDMA/efa: Add CQ with external memory support

Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset. Add a capability bit to reflect on the feature support.

---
 drivers/infiniband/hw/efa/efa.h       |  1 +
 drivers/infiniband/hw/efa/efa_verbs.c | 60 ++++++++++++++++++++++-----
 include/uapi/rdma/efa-abi.h           |  8 +++-
 3 files changed, 58 insertions(+), 11 deletions(-)
drivers/infiniband/hw/efa/efa.h +1 -0
index 838182d0409c..fd609a50cea0 100644
@@ -107,6 +107,7 @@ struct efa_cq {
 	u16 cq_idx;
 	/* NULL when no interrupts requested */
 	struct efa_eq *eq;
+	struct ib_umem *umem;
 };
 
 struct efa_qp {
drivers/infiniband/hw/efa/efa_verbs.c +50 -10

98 lines not on this page · raw message

include/uapi/rdma/efa-abi.h +7 -1
index 11b94b0b035b..f2bcef789571 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) */
 /*
- * Copyright 2018-2024 Amazon.com, Inc. or its affiliates. All rights reserved.
+ * Copyright 2018-2025 Amazon.com, Inc. or its affiliates. All rights reserved.
  */
 
 #ifndef EFA_ABI_USER_H
@@ -56,6 +56,7 @@ struct efa_ibv_alloc_pd_resp {
 enum {
 	EFA_CREATE_CQ_WITH_COMPLETION_CHANNEL = 1 << 0,
 	EFA_CREATE_CQ_WITH_SGID               = 1 << 1,
+	EFA_CREATE_CQ_WITH_EXT_MEM_DMABUF     = 1 << 2,
 };
 
 struct efa_ibv_create_cq {
@@ -64,6 +65,10 @@ struct efa_ibv_create_cq {
 	__u16 num_sub_cqs;
 	__u8 flags;
 	__u8 reserved_58[5];
+	__aligned_u64 ext_mem_offset;
+	__aligned_u64 ext_mem_length;
+	__u32 ext_mem_fd;
+	__u8 reserved_120[4];
 };
 
 enum {
@@ -131,6 +136,7 @@ enum {
 	EFA_QUERY_DEVICE_CAPS_DATA_POLLING_128 = 1 << 4,
 	EFA_QUERY_DEVICE_CAPS_RDMA_WRITE = 1 << 5,
 	EFA_QUERY_DEVICE_CAPS_UNSOLICITED_WRITE_RECV = 1 << 6,
+	EFA_QUERY_DEVICE_CAPS_CQ_WITH_EXT_MEM_DMABUF = 1 << 7,
 };
 
 struct efa_ibv_ex_query_device_resp {
Signature · 1 line
2.47.1
#1 lore raw
Quoted 3 lines · Michael Margolin

Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.

EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.

Thanks

author maintainer
#2 lore raw
Quoted 4 lines · Leon Romanovsky

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 8 lines
Quoted 3 lines · Michael Margolin

Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.

EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.

Thanks

I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.

Michael

#3 lore raw
Quoted 13 lines · Michael
Quoted 11 lines · Leon Romanovsky

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 3 lines · Michael Margolin

Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.

EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.

Quoted 8 lines
Quoted 1 line

Thanks

I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.

I think that this functionality is worth to have as general verb and not DV. mlx5 has something similar and now EFA needs it too.

Let's wait for Jason's response before rushing to implement it.

Thanks

Quoted 3 lines

Michael

author maintainer
#4 lore raw
Quoted 24 lines · Leon Romanovsky

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 12 lines · Michael
Quoted 11 lines · Leon Romanovsky

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 3 lines · Michael Margolin

Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.

EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.

Quoted 7 lines
Quoted 1 line

Thanks

I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.

Quoted 6 lines

I think that this functionality is worth to have as general verb and not DV. mlx5 has something similar and now EFA needs it too.

Let's wait for Jason's response before rushing to implement it.

Thanks

Jason, any thoughts on this?

We can probably add optional attributes to create CQ command, but most of the handling will remain vendor specific. I'm not convinced it's beneficial enough.

Michael

maintainer
#5 lore raw
Quoted 31 lines · Michael
Quoted 29 lines · Leon Romanovsky

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 12 lines · Michael
Quoted 11 lines · Leon Romanovsky

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 3 lines · Michael Margolin

Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.

EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.

Quoted 7 lines
Quoted 1 line

Thanks

I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.

I think that this functionality is worth to have as general verb and not DV. mlx5 has something similar and now EFA needs it too.

Let's wait for Jason's response before rushing to implement it.

Quoted 7 lines
Quoted 1 line

Thanks

Jason, any thoughts on this?

We can probably add optional attributes to create CQ command, but most of the handling will remain vendor specific. I'm not convinced it's beneficial enough.

I don't know how a general verb would work, CQs in non-dv are polled in software in userspace and the userspace will have trouble reaching into a dmabuf. Plus the entire point of this is usually to write the polling code in a GPU language and run it on a GPU processor.

Meaning I think all users of this will want to use a DV interface from verbs.

At that point, is it worth adding more common verbs support?

Though it is becoming a bit messy that drivers are all open coding creating normal or dmabuf umems. That part might be worth generalizing some more, and then doing QP as well.

Jason

author maintainer
#6 lore raw
Quoted 49 lines · Jason Gunthorpe

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 28 lines · Michael
Quoted 27 lines · Leon Romanovsky

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 12 lines · Michael
Quoted 11 lines · Leon Romanovsky

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 3 lines · Michael Margolin

Add an option to create CQ using external memory instead of allocating in the driver. The memory can be passed from userspace by dmabuf fd and an offset.

EFA is unique here. This patch is missing description of why it is needed, and why existing solutions if any exist, can't be used.

Quoted 7 lines
Quoted 1 line

Thanks

I probably should have explained more, the purpose is creating CQs that reside in GPU HBM enabling low latency polling directly by the GPU. EFA isn't unique in receiving pre-allocated memory from userspace, the extension here is the use of dmabuf for that purpose as a general mechanism that allows using memory independent of its source. I will add more info in the commit message.

I think that this functionality is worth to have as general verb and not DV. mlx5 has something similar and now EFA needs it too.

Quoted 8 lines
Quoted 3 lines

Let's wait for Jason's response before rushing to implement it.

Thanks

Jason, any thoughts on this?

We can probably add optional attributes to create CQ command, but most of the handling will remain vendor specific. I'm not convinced it's beneficial enough.

I don't know how a general verb would work, CQs in non-dv are polled in software in userspace and the userspace will have trouble reaching into a dmabuf. Plus the entire point of this is usually to write the polling code in a GPU language and run it on a GPU processor.

Meaning I think all users of this will want to use a DV interface from verbs.

Quoted 7 lines

At that point, is it worth adding more common verbs support?

Though it is becoming a bit messy that drivers are all open coding creating normal or dmabuf umems. That part might be worth generalizing some more, and then doing QP as well.

Jason

Are you suggesting turning mlx5dv_devx_umem_reg into a common verb including the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?

Michael

maintainer
#7 lore raw
Quoted 3 lines · Michael

Are you suggesting turning mlx5dv_devx_umem_reg into a common verb including the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?

I think Leon was, but I'm not sure that is so worthwhile.

I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.

But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?

Jason

author maintainer
#8 lore raw
Quoted 16 lines · Jason Gunthorpe

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 3 lines · Michael

Are you suggesting turning mlx5dv_devx_umem_reg into a common verb including the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?

I think Leon was, but I'm not sure that is so worthwhile.

I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.

Quoted 4 lines

But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?

Jason

Yes, EFA is an example for a driver that doesn't need this on the "standard" flow.

Michael

author maintainer
#9 lore raw
Quoted 18 lines · Michael
Quoted 16 lines · Jason Gunthorpe

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 3 lines · Michael

Are you suggesting turning mlx5dv_devx_umem_reg into a common verb including the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?

I think Leon was, but I'm not sure that is so worthwhile.

I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.

Quoted 9 lines
Quoted 4 lines

But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?

Jason

Yes, EFA is an example for a driver that doesn't need this on the "standard" flow.

Michael

How can we move forward with this patch? It's possible to add additional attributes to the common create CQ ioctl and use it for EFA direct verbs but it won't be easy to move existing drivers to use it.

Michael

maintainer
#10 lore raw
Quoted 22 lines · Michael
Quoted 20 lines · Michael
Quoted 17 lines · Jason Gunthorpe
Quoted 4 lines · Michael

Are you suggesting turning mlx5dv_devx_umem_reg into a common verb including
the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?

I think Leon was, but I'm not sure that is so worthwhile.

I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.

But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?

Jason

Quoted 8 lines
Quoted 5 lines

Yes, EFA is an example for a driver that doesn't need this on the "standard" flow.

Michael

How can we move forward with this patch? It's possible to add additional attributes to the common create CQ ioctl and use it for EFA direct verbs but it won't be easy to move existing drivers to use it.

Then it becomes hard to make the other drivers reject those new common attributes :\

Maybe what you have here is the best, but it really does seem unsatisfying from a kernel POV..

Jason

author maintainer
#11 lore raw
Quoted 32 lines · Jason Gunthorpe

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 19 lines · Michael
Quoted 18 lines · Michael
Quoted 17 lines · Jason Gunthorpe
Quoted 4 lines · Michael

Are you suggesting turning mlx5dv_devx_umem_reg into a common verb including
the kernel part or some kind of rdma-core level abstraction for passing dmabuf+offset+length / address+length to a create CQ/QP function?

I think Leon was, but I'm not sure that is so worthwhile.

I was thinking more of having the ioctls for things like QP/CQ/MR accept a more standard common set of attributes to describe the buffer memory and then making it simpler for the driver to get a umem from those common attributes.

But EFA is alread sort of different because it normally uses a kernel allocated buffer, right?

Jason

Quoted 8 lines
Quoted 5 lines

Yes, EFA is an example for a driver that doesn't need this on the "standard" flow.

Michael

How can we move forward with this patch? It's possible to add additional attributes to the common create CQ ioctl and use it for EFA direct verbs but it won't be easy to move existing drivers to use it.

Quoted 7 lines

Then it becomes hard to make the other drivers reject those new common attributes :\

Maybe what you have here is the best, but it really does seem unsatisfying from a kernel POV..

Jason

Apologies for a delayed response, I want to make sure we are aligned on the solution.

I'm going to add a common option in libibverbs for providers to pass buffer info for create CQ ioctl and when the new ioctl attributes are set to the kernel, handle it in core code by creating a umem object and storing it in CQ context for drivers use.

Does this make sense?

Michael