Skip to contentSkip to conversation

RDMA/rxe: Add dma-buf support

RFC PATCH In review Shunsuke Mie 5 participants 18 messages Nov 22, 2021 – Feb 24

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover RDMA/rxe: Add dma-buf support

This patch series add a dma-buf support for rxe driver.

A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and CMA managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.

[1] https://www.spinics.net/lists/linux-rdma/msg98592.html

To enable to use the dma-buf memory in rxe rdma device, add some changes and implementation in this patch series.

This series consists of two patches. The first patch changes the IB core to support for rdma drivers that has not dma device. The secound patch adds the dma-buf support to rxe driver.

Related user space RDMA library changes are provided as a separate patch.

v4:
* Fix warnings, unused variable and casting
v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+)
* Fix to use dma-buf-map helpers
v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+)
* Instead of using a dummy dma_device to attach dma-buf, just store dma-buf to use software RDMA driver
* Use dma-buf vmap() interface
* Check to pass tests of rdma-core
v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
* The initial patch set
* Use ib_device as dma_device.
* Use dma-buf dynamic attach interface
* Add dma-buf support to rxe device

Shunsuke Mie (2):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Add dma-buf support

 drivers/infiniband/core/umem_dmabuf.c |  20 ++++-
 drivers/infiniband/sw/rxe/rxe_loc.h   |   2 +
 drivers/infiniband/sw/rxe/rxe_mr.c    | 113 ++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_verbs.c |  34 ++++++++
 include/rdma/ib_umem.h                |   1 +
 5 files changed, 166 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

RFC PATCH v4 1/2 RDMA/umem: Change for rdma devices has not dma device

Current implementation requires a dma device for RDMA driver to use dma-buf memory space as RDMA buffer. However, software RDMA drivers has not dma device and copy RDMA data using CPU instead of hardware.

This patch changes to be hold a dma-buf on struct ib_umem_dmabuf. This allows the software RDMA driver to map dma-buf memory for CPU memory accessing.

---
 drivers/infiniband/core/umem_dmabuf.c | 20 ++++++++++++++++----
 include/rdma/ib_umem.h                |  1 +
 2 files changed, 17 insertions(+), 4 deletions(-)
drivers/infiniband/core/umem_dmabuf.c +16 -4
index e824baf4640d..ebbb0a259fd4 100644
@@ -117,9 +117,6 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,
 	if (check_add_overflow(offset, (unsigned long)size, &end))
 		return ret;
 
-	if (unlikely(!ops || !ops->move_notify))
-		return ret;
-
 	dmabuf = dma_buf_get(fd);
 	if (IS_ERR(dmabuf))
 		return ERR_CAST(dmabuf);
@@ -133,6 +130,8 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,
 		goto out_release_dmabuf;
 	}
 
+	umem_dmabuf->dmabuf = dmabuf;
+
 	umem = &umem_dmabuf->umem;
 	umem->ibdev = device;
 	umem->length = size;
@@ -143,6 +142,13 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,
 	if (!ib_umem_num_pages(umem))
 		goto out_free_umem;
 
+	/* Software RDMA drivers has not dma device. Just get dmabuf from fd */
+	if (!device->dma_device)
+		goto done;
+
+	if (unlikely(!ops || !ops->move_notify))
+		goto out_free_umem;
+
 	umem_dmabuf->attach = dma_buf_dynamic_attach(
 					dmabuf,
 					device->dma_device,
@@ -152,6 +158,7 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,
 		ret = ERR_CAST(umem_dmabuf->attach);
 		goto out_free_umem;
 	}
+done:
 	return umem_dmabuf;
 
 out_free_umem:
@@ -165,13 +172,18 @@ EXPORT_SYMBOL(ib_umem_dmabuf_get);
 
 void ib_umem_dmabuf_release(struct ib_umem_dmabuf *umem_dmabuf)
 {
-	struct dma_buf *dmabuf = umem_dmabuf->attach->dmabuf;
+	struct dma_buf *dmabuf = umem_dmabuf->dmabuf;
+
+	if (!umem_dmabuf->attach)
+		goto free_dmabuf;
 
 	dma_resv_lock(dmabuf->resv, NULL);
 	ib_umem_dmabuf_unmap_pages(umem_dmabuf);
 	dma_resv_unlock(dmabuf->resv);
 
 	dma_buf_detach(dmabuf, umem_dmabuf->attach);
+
+free_dmabuf:
 	dma_buf_put(dmabuf);
 	kfree(umem_dmabuf);
 }
include/rdma/ib_umem.h +1 -0
index 5ae9dff74dac..11c0cf7e0dd8 100644
@@ -32,6 +32,7 @@ struct ib_umem {
 struct ib_umem_dmabuf {
 	struct ib_umem umem;
 	struct dma_buf_attachment *attach;
+	struct dma_buf *dmabuf;
 	struct sg_table *sgt;
 	struct scatterlist *first_sg;
 	struct scatterlist *last_sg;
Signature · 1 line
2.17.1
#2 lore raw

RFC PATCH v4 2/2 RDMA/rxe: Add dma-buf support

In reply to Shunsuke Mie

Implement a ib device operation ‘reg_user_mr_dmabuf’. Generate a rxe_map from the memory space linked the passed dma-buf.

---
 drivers/infiniband/sw/rxe/rxe_loc.h   |   2 +
 drivers/infiniband/sw/rxe/rxe_mr.c    | 113 ++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_verbs.c |  34 ++++++++
 3 files changed, 149 insertions(+)
drivers/infiniband/sw/rxe/rxe_loc.h +2 -0
index 1ca43b859d80..8bc19ea1a376 100644
@@ -75,6 +75,8 @@ u8 rxe_get_next_key(u32 last_key);
 void rxe_mr_init_dma(struct rxe_pd *pd, int access, struct rxe_mr *mr);
 int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova,
 		     int access, struct rxe_mr *mr);
+int rxe_mr_dmabuf_init_user(struct rxe_pd *pd, int fd, u64 start, u64 length,
+			    u64 iova, int access, struct rxe_mr *mr);
 int rxe_mr_init_fast(struct rxe_pd *pd, int max_pages, struct rxe_mr *mr);
 int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length,
 		enum rxe_mr_copy_dir dir);
drivers/infiniband/sw/rxe/rxe_mr.c +113 -0

131 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_verbs.c +34 -0
index 0aa0d7e52773..dc7d27b3cb90 100644
@@ -940,6 +940,39 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd,
 	return ERR_PTR(err);
 }
 
+static struct ib_mr *rxe_reg_user_mr_dmabuf(struct ib_pd *ibpd, u64 start,
+					    u64 length, u64 iova, int fd,
+					    int access, struct ib_udata *udata)
+{
+	int err;
+	struct rxe_dev *rxe = to_rdev(ibpd->device);
+	struct rxe_pd *pd = to_rpd(ibpd);
+	struct rxe_mr *mr;
+
+	mr = rxe_alloc(&rxe->mr_pool);
+	if (!mr) {
+		err = -ENOMEM;
+		goto err2;
+	}
+
+	rxe_add_index(mr);
+
+	rxe_add_ref(pd);
+
+	err = rxe_mr_dmabuf_init_user(pd, fd, start, length, iova, access, mr);
+	if (err)
+		goto err3;
+
+	return &mr->ibmr;
+
+err3:
+	rxe_drop_ref(pd);
+	rxe_drop_index(mr);
+	rxe_drop_ref(mr);
+err2:
+	return ERR_PTR(err);
+}
+
 static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type,
 				  u32 max_num_sg)
 {
@@ -1105,6 +1138,7 @@ static const struct ib_device_ops rxe_dev_ops = {
 	.query_qp = rxe_query_qp,
 	.query_srq = rxe_query_srq,
 	.reg_user_mr = rxe_reg_user_mr,
+	.reg_user_mr_dmabuf = rxe_reg_user_mr_dmabuf,
 	.req_notify_cq = rxe_req_notify_cq,
 	.resize_cq = rxe_resize_cq,
 
Signature · 1 line
2.17.1
#3 lore raw
In reply to Shunsuke Mie

Hi maintainers,

Could you please review this patch series?

Regards,
Shunsuke Mie

2021年11月22日(月) 20:08 Shunsuke Mie <[email protected]>:

Quoted 51 lines

This patch series add a dma-buf support for rxe driver.

A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and CMA managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.

[1] https://www.spinics.net/lists/linux-rdma/msg98592.html

To enable to use the dma-buf memory in rxe rdma device, add some changes and implementation in this patch series.

This series consists of two patches. The first patch changes the IB core to support for rdma drivers that has not dma device. The secound patch adds the dma-buf support to rxe driver.

Related user space RDMA library changes are provided as a separate patch.

v4:
* Fix warnings, unused variable and casting
v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+)
* Fix to use dma-buf-map helpers
v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+)
* Instead of using a dummy dma_device to attach dma-buf, just store dma-buf to use software RDMA driver
* Use dma-buf vmap() interface
* Check to pass tests of rdma-core
v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
* The initial patch set
* Use ib_device as dma_device.
* Use dma-buf dynamic attach interface
* Add dma-buf support to rxe device

Shunsuke Mie (2):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Add dma-buf support

 drivers/infiniband/core/umem_dmabuf.c |  20 ++++-
 drivers/infiniband/sw/rxe/rxe_loc.h   |   2 +
 drivers/infiniband/sw/rxe/rxe_mr.c    | 113 ++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_verbs.c |  34 ++++++++
 include/rdma/ib_umem.h                |   1 +
 5 files changed, 166 insertions(+), 4 deletions(-)

--
2.17.1

#4 lore raw
Quoted 3 lines · Shunsuke Mie

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

Jason

#5 lore raw

Hi Jason,
Thank you for replying.

2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:

Quoted 12 lines · Shunsuke Mie
Quoted 3 lines

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.

Is it wrong? If this is wrong, there is no advantages this changes..

Quoted 2 lines

Jason

Regards,
Shunsuke

#6 lore raw
Quoted 12 lines · Shunsuke Mie

Hi Jason,
Thank you for replying.

2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:

Quoted 8 lines · Shunsuke Mie
Quoted 3 lines

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

Quoted 8 lines
Quoted 4 lines

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.

Is it wrong? If this is wrong, there is no advantages this changes..

I don't know what the dmabuf folks did, but yes, it is wrong.

IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.

Christian? If the vmap succeeds what rules must the caller use to access the memory?

Jason

#7 lore raw

Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:

Quoted 19 lines · Shunsuke Mie
Quoted 10 lines

Hi Jason,
Thank you for replying.

2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

Quoted 8 lines
Quoted 4 lines

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.

Is it wrong? If this is wrong, there is no advantages this changes..

Quoted 8 lines

I don't know what the dmabuf folks did, but yes, it is wrong.

IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.

Christian? If the vmap succeeds what rules must the caller use to access the memory?

See dma-buf-map.h and especially struct dma_buf_map.

MMIO memory is perfectly supported here and actually the most common case.

Christian.

Quoted 2 lines

Jason

#8 lore raw
Quoted 26 lines · Christian König

Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:

Quoted 25 lines · Shunsuke Mie
Quoted 10 lines

Hi Jason,
Thank you for replying.

2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

Quoted 8 lines
Quoted 4 lines

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.

Is it wrong? If this is wrong, there is no advantages this changes..

I don't know what the dmabuf folks did, but yes, it is wrong.

IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.

Quoted 6 lines
Quoted 2 lines

Christian? If the vmap succeeds what rules must the caller use to access the memory?

See dma-buf-map.h and especially struct dma_buf_map.

MMIO memory is perfectly supported here and actually the most common case.

Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.

??

Jason

#9 lore raw

Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:

Quoted 29 lines · Christian König
Quoted 28 lines

Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:

Quoted 27 lines · Shunsuke Mie
Quoted 10 lines

Hi Jason,
Thank you for replying.

2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

Quoted 8 lines
Quoted 4 lines

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.

Is it wrong? If this is wrong, there is no advantages this changes..

I don't know what the dmabuf folks did, but yes, it is wrong.

IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.

Christian? If the vmap succeeds what rules must the caller use to access the memory?

Quoted 8 lines
Quoted 3 lines

See dma-buf-map.h and especially struct dma_buf_map.

MMIO memory is perfectly supported here and actually the most common case.

Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.

??

Well, yes that can go boom pretty quickly.

Not sure what they want to use this for.

Christian.

Quoted 2 lines

Jason

#10 lore raw

2021年12月10日(金) 22:29 Christian König <[email protected]>:

Quoted 33 lines

Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:

Quoted 31 lines · Christian König
Quoted 30 lines

Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:

Quoted 19 lines · Shunsuke Mie
Quoted 10 lines

Hi Jason,
Thank you for replying.

2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

Quoted 8 lines
Quoted 4 lines

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.

Is it wrong? If this is wrong, there is no advantages this changes..

Quoted 8 lines

I don't know what the dmabuf folks did, but yes, it is wrong.

IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.

Christian? If the vmap succeeds what rules must the caller use to access the memory?

See dma-buf-map.h and especially struct dma_buf_map.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

MMIO memory is perfectly supported here and actually the most common case.

Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.

??

Well, yes that can go boom pretty quickly.

Sorry, I was wrong. The dma_buf_map treats both iomem and vaddr region, but this RFC only supports vaddr. Advantage of the partial support is we can use the vaddr dma-buf in RXE without changing a rxe data copy implementation.

An example of a dma-buf pointing to a vaddr is some gpu drivers use RAM for VRAM and we can get dma-buf for the region that indicates vaddr regions. Specifically, the gpu driver using gpu/drm/drm_gem_cma_helper.c is one such example.

Quoted 1 line

Not sure what they want to use this for.

I'd like to use RDMA with RXE for that memory region.

Best,
Shunsuke

Quoted 5 lines

Christian.

Quoted 2 lines

Jason

#11 lore raw

Am 13.12.21 um 12:18 schrieb Shunsuke Mie:

Quoted 38 lines

2021年12月10日(金) 22:29 Christian König <[email protected]>:

Quoted 37 lines

Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:

Quoted 36 lines · Christian König
Quoted 26 lines

Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:

Quoted 25 lines · Shunsuke Mie
Quoted 10 lines

Hi Jason,
Thank you for replying.

2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

Quoted 8 lines
Quoted 4 lines

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.

Is it wrong? If this is wrong, there is no advantages this changes..

I don't know what the dmabuf folks did, but yes, it is wrong.

IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.

Quoted 5 lines
Quoted 2 lines

Christian? If the vmap succeeds what rules must the caller use to access the memory?

See dma-buf-map.h and especially struct dma_buf_map.

MMIO memory is perfectly supported here and actually the most common case.

Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.

Quoted 5 lines
Quoted 2 lines
Quoted 1 line

??

Well, yes that can go boom pretty quickly.

Sorry, I was wrong. The dma_buf_map treats both iomem and vaddr region, but this RFC only supports vaddr. Advantage of the partial support is we can use the vaddr dma-buf in RXE without changing a rxe data copy implementation.

Well that is most likely not a good idea.

For example buffers for GPU drivers can be placed in both MMIO memory and system memory.

If you don't want to provoke random failures you *MUST* be able to handle both if you want to use this.

Regards,
Christian.

Quoted 14 lines

An example of a dma-buf pointing to a vaddr is some gpu drivers use RAM for VRAM and we can get dma-buf for the region that indicates vaddr regions. Specifically, the gpu driver using gpu/drm/drm_gem_cma_helper.c is one such example.

Quoted 1 line

Not sure what they want to use this for.

I'd like to use RDMA with RXE for that memory region.

Best,
Shunsuke

Quoted 3 lines

Christian.

Quoted 1 line

Jason

#12 lore raw

2021年12月14日(火) 17:54 Christian König <[email protected]>:

Quoted 46 lines

Am 13.12.21 um 12:18 schrieb Shunsuke Mie:

Quoted 38 lines

2021年12月10日(金) 22:29 Christian König <[email protected]>:

Quoted 37 lines

Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:

Quoted 36 lines · Christian König
Quoted 26 lines

Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:

Quoted 25 lines · Shunsuke Mie
Quoted 10 lines

Hi Jason,
Thank you for replying.

2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

Quoted 8 lines
Quoted 4 lines

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.

Is it wrong? If this is wrong, there is no advantages this changes..

I don't know what the dmabuf folks did, but yes, it is wrong.

IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.

Quoted 5 lines
Quoted 2 lines

Christian? If the vmap succeeds what rules must the caller use to access the memory?

See dma-buf-map.h and especially struct dma_buf_map.

MMIO memory is perfectly supported here and actually the most common case.

Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.

Quoted 5 lines
Quoted 2 lines
Quoted 1 line

??

Well, yes that can go boom pretty quickly.

Sorry, I was wrong. The dma_buf_map treats both iomem and vaddr region, but this RFC only supports vaddr. Advantage of the partial support is we can use the vaddr dma-buf in RXE without changing a rxe data copy implementation.

Quoted 7 lines

Well that is most likely not a good idea.

For example buffers for GPU drivers can be placed in both MMIO memory and system memory.

If you don't want to provoke random failures you *MUST* be able to handle both if you want to use this.

I agree with you. I'll add the support and resubmit patch series.

Thanks a lot,
Shunsuke.

Quoted 19 lines

Regards,
Christian.

Quoted 14 lines

An example of a dma-buf pointing to a vaddr is some gpu drivers use RAM for VRAM and we can get dma-buf for the region that indicates vaddr regions. Specifically, the gpu driver using gpu/drm/drm_gem_cma_helper.c is one such example.

Quoted 1 line

Not sure what they want to use this for.

I'd like to use RDMA with RXE for that memory region.

Best,
Shunsuke

Quoted 3 lines

Christian.

Quoted 1 line

Jason

#13 lore raw
In reply to Jason Gunthorpe
Quoted 32 lines · Jason Gunthorpe
Quoted 31 lines · Christian König

Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:

Quoted 19 lines · Shunsuke Mie
Quoted 10 lines

Hi Jason,
Thank you for replying.

2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

Hi maintainers,

Could you please review this patch series?

Why is it RFC?

Quoted 8 lines
Quoted 4 lines

I'm confused why this is useful?

This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?

I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.

Is it wrong? If this is wrong, there is no advantages this changes..

Quoted 8 lines

I don't know what the dmabuf folks did, but yes, it is wrong.

IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.

Christian? If the vmap succeeds what rules must the caller use to access the memory?

See dma-buf-map.h and especially struct dma_buf_map.

Quoted 7 lines
Quoted 1 line

MMIO memory is perfectly supported here and actually the most common case.

Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.

??

dma_buf_map is fairly new and we haven't rolled it out consistently yet. In the past 10 years we simply yolo'd this :-)

Just an explanation, not an excuse for new code to not use dma_buf_map consistently now that we fixed this mistake.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#14 lore raw
In reply to Shunsuke Mie
Quoted 8 lines · Shunsuke Mie

This patch series add a dma-buf support for rxe driver.

A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and CMA managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.

Quoted 9 lines

[1] https://www.spinics.net/lists/linux-rdma/msg98592.html

To enable to use the dma-buf memory in rxe rdma device, add some changes and implementation in this patch series.

This series consists of two patches. The first patch changes the IB core to support for rdma drivers that has not dma device. The secound patch adds the dma-buf support to rxe driver.

Hi, Shunsuke Mie

I was revisiting your 2021 proposal around dma-buf integration with RDMA and the related discussions at the time.

As you know, dma-buf usage in RDMA-related workflows has gained more traction recently, and we are seeing increasing interest in heterogeneous memory and cross-device buffer sharing. Given the changes in the ecosystem since then, I’m wondering whether you think the original direction might be worth reconsidering.

Do you have any interest in continuing that line of work, or updating the design based on today’s context? If not, I’d still appreciate your perspective on what you see as the main blockers from the previous discussions, and whether you think the landscape has changed enough to justify another attempt.

Depending on the direction, we may consider exploring dma-buf support in rxe or at the core level, but I’d prefer to first understand your view before moving forward.

Zhu Yanjun

Quoted 30 lines

Related user space RDMA library changes are provided as a separate patch.

v4:
* Fix warnings, unused variable and casting
v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+)
* Fix to use dma-buf-map helpers
v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+)
* Instead of using a dummy dma_device to attach dma-buf, just store dma-buf to use software RDMA driver
* Use dma-buf vmap() interface
* Check to pass tests of rdma-core
v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
* The initial patch set
* Use ib_device as dma_device.
* Use dma-buf dynamic attach interface
* Add dma-buf support to rxe device

Shunsuke Mie (2):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Add dma-buf support

  drivers/infiniband/core/umem_dmabuf.c |  20 ++++-
  drivers/infiniband/sw/rxe/rxe_loc.h   |   2 +
  drivers/infiniband/sw/rxe/rxe_mr.c    | 113 ++++++++++++++++++++++++++
  drivers/infiniband/sw/rxe/rxe_verbs.c |  34 ++++++++
  include/rdma/ib_umem.h                |   1 +
  5 files changed, 166 insertions(+), 4 deletions(-)
#15 lore raw

Hi Zhu Yanjun,

Thank you for reaching out and revisiting my 2021 proposal.

Regarding the current status, I haven't been able to make much progress recently as other tasks have taken higher priority. However, I still believe this integration is important.

From a technical perspective, as pointed out during the previous reviews, there were indeed issues with how dma_buf_map was being used. To address this in today's context, I believe we should transition to using iosys_map.

I am still interested in this direction. While my current bandwidth is limited, I would welcome any collaboration—especially regarding the implementation of iosys_map support within rxe or the RDMA core.

I'd be happy to discuss the technical details of this transition if you'd like to dive deeper.

Best,
Shunsuke Mie

2026年2月19日(木) 13:43 Zhu Yanjun <[email protected]>:

Quoted 75 lines · Shunsuke Mie
Quoted 4 lines

This patch series add a dma-buf support for rxe driver.

A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and

CMA

Quoted 11 lines

managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.

[1] https://www.spinics.net/lists/linux-rdma/msg98592.html

To enable to use the dma-buf memory in rxe rdma device, add some changes and implementation in this patch series.

This series consists of two patches. The first patch changes the IB core to support for rdma drivers that has not dma device. The secound patch

adds

Quoted 2 lines

the dma-buf support to rxe driver.

Hi, Shunsuke Mie

I was revisiting your 2021 proposal around dma-buf integration with RDMA and the related discussions at the time.

As you know, dma-buf usage in RDMA-related workflows has gained more traction recently, and we are seeing increasing interest in heterogeneous memory and cross-device buffer sharing. Given the changes in the ecosystem since then, I’m wondering whether you think the original direction might be worth reconsidering.

Do you have any interest in continuing that line of work, or updating the design based on today’s context? If not, I’d still appreciate your perspective on what you see as the main blockers from the previous discussions, and whether you think the landscape has changed enough to justify another attempt.

Depending on the direction, we may consider exploring dma-buf support in rxe or at the core level, but I’d prefer to first understand your view before moving forward.

Zhu Yanjun

Quoted 30 lines

Related user space RDMA library changes are provided as a separate patch.

v4:
* Fix warnings, unused variable and casting
v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+)
* Fix to use dma-buf-map helpers
v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+)
* Instead of using a dummy dma_device to attach dma-buf, just store dma-buf to use software RDMA driver
* Use dma-buf vmap() interface
* Check to pass tests of rdma-core
v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
* The initial patch set
* Use ib_device as dma_device.
* Use dma-buf dynamic attach interface
* Add dma-buf support to rxe device

Shunsuke Mie (2):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Add dma-buf support

  drivers/infiniband/core/umem_dmabuf.c |  20 ++++-
  drivers/infiniband/sw/rxe/rxe_loc.h   |   2 +
  drivers/infiniband/sw/rxe/rxe_mr.c    | 113 ++++++++++++++++++++++++++
  drivers/infiniband/sw/rxe/rxe_verbs.c |  34 ++++++++
  include/rdma/ib_umem.h                |   1 +
  5 files changed, 166 insertions(+), 4 deletions(-)
#16 lore raw
Quoted 11 lines · Shunsuke Mie

Hi Zhu Yanjun,

Thank you for reaching out and revisiting my 2021 proposal.

Regarding the current status, I haven't been able to make much progress recently as other tasks have taken higher priority. However, I still believe this integration is important.

From a technical perspective, as pointed out during the previous reviews, there were indeed issues with how dma_buf_map was being used. To address this in today's context, I believe we should transition to using iosys_map.

I am still interested in this direction. While my current bandwidth is limited, I would welcome any collaboration—especially regarding the implementation of iosys_map support within rxe or the RDMA core.

I'd be happy to discuss the technical details of this transition if you'd like to dive deeper.

DMABUF has become increasingly popular and is widely adopted across various subsystems. If we could also implement DMABUF support in rxe, it would significantly enhance rxe’s usability and overall value.

Given your previous involvement in this area, it would be great to move this forward in rxe. I believe your expertise would be particularly helpful in driving the implementation. I’m happy to review and collaborate on the design and patches as needed.

Zhu Yanjun

Quoted 94 lines

Best,
Shunsuke Mie

2026年2月19日(木) 13:43 Zhu Yanjun <[email protected]>:

在 2021/11/22 3:08, Shunsuke Mie 写道:
> This patch series add a dma-buf support for rxe driver. >
> A dma-buf based memory registering has beed introduced to use the memory
> region that lack of associated page structures (e.g. device memory and CMA
> managed memory) [1]. However, to use the dma-buf based memory, each rdma
> device drivers require add some implementation. The rxe driver has not
> support yet.
>
> [1] https://www.spinics.net/lists/linux-rdma/msg98592.html
>
> To enable to use the dma-buf memory in rxe rdma device, add some changes
> and implementation in this patch series.
>
> This series consists of two patches. The first patch changes the IB core
> to support for rdma drivers that has not dma device. The secound patch adds
> the dma-buf support to rxe driver.
>
Hi, Shunsuke Mie

I was revisiting your 2021 proposal around dma-buf integration with RDMA
and the related discussions at the time.

As you know, dma-buf usage in RDMA-related workflows has gained more traction recently, and we are seeing increasing interest in heterogeneous memory and cross-device buffer sharing. Given the changes
in the ecosystem since then, I’m wondering whether you think the original direction might be worth reconsidering.

Do you have any interest in continuing that line of work, or updating the design based on today’s context? If not, I’d still appreciate your
perspective on what you see as the main blockers from the previous discussions, and whether you think the landscape has changed enough to
justify another attempt.

Depending on the direction, we may consider exploring dma-buf support in
rxe or at the core level, but I’d prefer to first understand your view
before moving forward.

Zhu Yanjun

> Related user space RDMA library changes are provided as a separate patch.
>
> v4:
> * Fix warnings, unused variable and casting
> v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
> * Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+) > * Fix to use dma-buf-map helpers
> v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
> * Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+) > * Instead of using a dummy dma_device to attach dma-buf, just store >    dma-buf to use software RDMA driver
> * Use dma-buf vmap() interface
> * Check to pass tests of rdma-core
> v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
> * The initial patch set
> * Use ib_device as dma_device.
> * Use dma-buf dynamic attach interface
> * Add dma-buf support to rxe device
>
> Shunsuke Mie (2):
>    RDMA/umem: Change for rdma devices has not dma device >    RDMA/rxe: Add dma-buf support
>
>   drivers/infiniband/core/umem_dmabuf.c |  20 ++++-
>   drivers/infiniband/sw/rxe/rxe_loc.h   |   2 +
>   drivers/infiniband/sw/rxe/rxe_mr.c    | 113
++++++++++++++++++++++++++
>   drivers/infiniband/sw/rxe/rxe_verbs.c |  34 ++++++++ >   include/rdma/ib_umem.h                |   1 +
>   5 files changed, 166 insertions(+), 4 deletions(-)
>

Signature · 2 lines
Best Regards,
Yanjun.Zhu
#17 lore raw
In reply to Shunsuke Mie
Quoted 9 lines · Shunsuke Mie

Hi Zhu Yanjun,

Thank you for reaching out and revisiting my 2021 proposal.

Regarding the current status, I haven't been able to make much progress recently as other tasks have taken higher priority. However, I still believe this integration is important.

From a technical perspective, as pointed out during the previous reviews, there were indeed issues with how dma_buf_map was being used. To address this in today's context, I believe we should transition to using iosys_map.

I am still interested in this direction. While my current bandwidth is limited, I would welcome any collaboration—especially regarding the implementation of iosys_map support within rxe or the RDMA core.

Sorry. In the previous mail, there is a HTML part. It can not reach. As such, I resend it.

DMABUF has become increasingly popular and is widely adopted across various subsystems.

If we could also implement DMABUF support in rxe, it would significantly enhance rxe’s usability and overall value.

Given your previous involvement in this area, it would be great to move this forward in rxe.

I believe your expertise would be particularly helpful in driving the implementation.

I’m happy to review and collaborate on the design and patches as needed.

Zhu Yanjun

Quoted 96 lines

I'd be happy to discuss the technical details of this transition if you'd like to dive deeper.

Best,
Shunsuke Mie

2026年2月19日(木) 13:43 Zhu Yanjun <[email protected]>:

在 2021/11/22 3:08, Shunsuke Mie 写道:
> This patch series add a dma-buf support for rxe driver. >
> A dma-buf based memory registering has beed introduced to use the memory
> region that lack of associated page structures (e.g. device memory and CMA
> managed memory) [1]. However, to use the dma-buf based memory, each rdma
> device drivers require add some implementation. The rxe driver has not
> support yet.
>
> [1] https://www.spinics.net/lists/linux-rdma/msg98592.html
>
> To enable to use the dma-buf memory in rxe rdma device, add some changes
> and implementation in this patch series.
>
> This series consists of two patches. The first patch changes the IB core
> to support for rdma drivers that has not dma device. The secound patch adds
> the dma-buf support to rxe driver.
>
Hi, Shunsuke Mie

I was revisiting your 2021 proposal around dma-buf integration with RDMA
and the related discussions at the time.

As you know, dma-buf usage in RDMA-related workflows has gained more traction recently, and we are seeing increasing interest in heterogeneous memory and cross-device buffer sharing. Given the changes
in the ecosystem since then, I’m wondering whether you think the original direction might be worth reconsidering.

Do you have any interest in continuing that line of work, or updating the design based on today’s context? If not, I’d still appreciate your
perspective on what you see as the main blockers from the previous discussions, and whether you think the landscape has changed enough to
justify another attempt.

Depending on the direction, we may consider exploring dma-buf support in
rxe or at the core level, but I’d prefer to first understand your view
before moving forward.

Zhu Yanjun

> Related user space RDMA library changes are provided as a separate patch.
>
> v4:
> * Fix warnings, unused variable and casting
> v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
> * Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+) > * Fix to use dma-buf-map helpers
> v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
> * Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+) > * Instead of using a dummy dma_device to attach dma-buf, just store >    dma-buf to use software RDMA driver
> * Use dma-buf vmap() interface
> * Check to pass tests of rdma-core
> v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
> * The initial patch set
> * Use ib_device as dma_device.
> * Use dma-buf dynamic attach interface
> * Add dma-buf support to rxe device
>
> Shunsuke Mie (2):
>    RDMA/umem: Change for rdma devices has not dma device >    RDMA/rxe: Add dma-buf support
>
>   drivers/infiniband/core/umem_dmabuf.c |  20 ++++-
>   drivers/infiniband/sw/rxe/rxe_loc.h   |   2 +
>   drivers/infiniband/sw/rxe/rxe_mr.c    | 113
++++++++++++++++++++++++++
>   drivers/infiniband/sw/rxe/rxe_verbs.c |  34 ++++++++ >   include/rdma/ib_umem.h                |   1 +
>   5 files changed, 166 insertions(+), 4 deletions(-)
>