RDMA/rxe: Add dma-buf support
Subsystems (3)
v4 cover RDMA/rxe: Add dma-buf support
This patch series add a dma-buf support for rxe driver.
A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and CMA managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.
[1] https://www.spinics.net/lists/linux-rdma/msg98592.html
To enable to use the dma-buf memory in rxe rdma device, add some changes and implementation in this patch series.
This series consists of two patches. The first patch changes the IB core to support for rdma drivers that has not dma device. The secound patch adds the dma-buf support to rxe driver.
Related user space RDMA library changes are provided as a separate patch.
v4:
* Fix warnings, unused variable and casting
v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+)
* Fix to use dma-buf-map helpers
v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+)
* Instead of using a dummy dma_device to attach dma-buf, just store dma-buf to use software RDMA driver
* Use dma-buf vmap() interface
* Check to pass tests of rdma-core
v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
* The initial patch set
* Use ib_device as dma_device.
* Use dma-buf dynamic attach interface
* Add dma-buf support to rxe device
Shunsuke Mie (2):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Add dma-buf support
drivers/infiniband/core/umem_dmabuf.c | 20 ++++- drivers/infiniband/sw/rxe/rxe_loc.h | 2 + drivers/infiniband/sw/rxe/rxe_mr.c | 113 ++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_verbs.c | 34 ++++++++ include/rdma/ib_umem.h | 1 + 5 files changed, 166 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.1
RFC PATCH v4 1/2 RDMA/umem: Change for rdma devices has not dma device
Current implementation requires a dma device for RDMA driver to use dma-buf memory space as RDMA buffer. However, software RDMA drivers has not dma device and copy RDMA data using CPU instead of hardware.
This patch changes to be hold a dma-buf on struct ib_umem_dmabuf. This allows the software RDMA driver to map dma-buf memory for CPU memory accessing.
--- drivers/infiniband/core/umem_dmabuf.c | 20 ++++++++++++++++---- include/rdma/ib_umem.h | 1 + 2 files changed, 17 insertions(+), 4 deletions(-)
drivers/infiniband/core/umem_dmabuf.c +16 -4
@@ -117,9 +117,6 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,if (check_add_overflow (offset, (unsigned long )size, &end))return ret; -if (unlikely (!ops || !ops->move_notify)) -return ret; - dmabuf =dma_buf_get (fd);if (IS_ERR (dmabuf))return ERR_CAST (dmabuf);
@@ -133,6 +130,8 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device, goto out_release_dmabuf; } + umem_dmabuf->dmabuf = dmabuf; + umem = &umem_dmabuf->umem; umem->ibdev = device; umem->length = size;
@@ -143,6 +142,13 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,if (!ib_umem_num_pages (umem)) goto out_free_umem; +/* Software RDMA drivers has not dma device. Just get dmabuf from fd */ +if (!device->dma_device) + goto done; + +if (unlikely (!ops || !ops->move_notify)) + goto out_free_umem; + umem_dmabuf->attach = dma_buf_dynamic_attach( dmabuf, device->dma_device,
@@ -152,6 +158,7 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device, ret =ERR_CAST (umem_dmabuf->attach); goto out_free_umem; } +done:return umem_dmabuf; out_free_umem:
@@ -165,13 +172,18 @@ EXPORT_SYMBOL(ib_umem_dmabuf_get);void ib_umem_dmabuf_release (struct ib_umem_dmabuf *umem_dmabuf) { -struct dma_buf *dmabuf = umem_dmabuf->attach->dmabuf; +struct dma_buf *dmabuf = umem_dmabuf->dmabuf; + +if (!umem_dmabuf->attach) + goto free_dmabuf;dma_resv_lock (dmabuf->resv,NULL );ib_umem_dmabuf_unmap_pages (umem_dmabuf);dma_resv_unlock (dmabuf->resv);dma_buf_detach (dmabuf, umem_dmabuf->attach); + +free_dmabuf:dma_buf_put (dmabuf);kfree (umem_dmabuf); }
include/rdma/ib_umem.h +1 -0
@@ -32,6 +32,7 @@ struct ib_umem {struct ib_umem_dmabuf {struct ib_umem umem;struct dma_buf_attachment *attach; +struct dma_buf *dmabuf;struct sg_table *sgt;struct scatterlist *first_sg;struct scatterlist *last_sg;
Signature · 1 line
2.17.1
RFC PATCH v4 2/2 RDMA/rxe: Add dma-buf support
In reply to Shunsuke MieImplement a ib device operation ‘reg_user_mr_dmabuf’. Generate a rxe_map from the memory space linked the passed dma-buf.
--- drivers/infiniband/sw/rxe/rxe_loc.h | 2 + drivers/infiniband/sw/rxe/rxe_mr.c | 113 ++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_verbs.c | 34 ++++++++ 3 files changed, 149 insertions(+)
drivers/infiniband/sw/rxe/rxe_loc.h +2 -0
@@ -75,6 +75,8 @@ u8 rxe_get_next_key(u32 last_key);void rxe_mr_init_dma (struct rxe_pd *pd,int access,struct rxe_mr *mr);int rxe_mr_init_user (struct rxe_pd *pd,u64 start,u64 length,u64 iova,int access,struct rxe_mr *mr); +int rxe_mr_dmabuf_init_user (struct rxe_pd *pd,int fd,u64 start,u64 length, +u64 iova,int access,struct rxe_mr *mr);int rxe_mr_init_fast (struct rxe_pd *pd,int max_pages,struct rxe_mr *mr);int rxe_mr_copy (struct rxe_mr *mr,u64 iova,void *addr,int length,enum rxe_mr_copy_dir dir);
drivers/infiniband/sw/rxe/rxe_mr.c +113 -0
131 lines not on this page · raw message
drivers/infiniband/sw/rxe/rxe_verbs.c +34 -0
@@ -940,6 +940,39 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd,return ERR_PTR (err); } +static struct ib_mr *rxe_reg_user_mr_dmabuf (struct ib_pd *ibpd,u64 start, +u64 length,u64 iova,int fd, +int access,struct ib_udata *udata) +{ +int err; +struct rxe_dev *rxe =to_rdev (ibpd->device); +struct rxe_pd *pd =to_rpd (ibpd); +struct rxe_mr *mr; + + mr =rxe_alloc (&rxe->mr_pool); +if (!mr) { + err = -ENOMEM ; + goto err2; + } + +rxe_add_index (mr); + +rxe_add_ref (pd); + + err =rxe_mr_dmabuf_init_user (pd, fd, start, length, iova, access, mr); +if (err) + goto err3; + +return &mr->ibmr; + +err3: +rxe_drop_ref (pd); +rxe_drop_index (mr); +rxe_drop_ref (mr); +err2: +return ERR_PTR (err); +} +static struct ib_mr *rxe_alloc_mr (struct ib_pd *ibpd,enum ib_mr_type mr_type,u32 max_num_sg) {
@@ -1105,6 +1138,7 @@ static const struct ib_device_ops rxe_dev_ops = { .query_qp = rxe_query_qp, .query_srq = rxe_query_srq, .reg_user_mr = rxe_reg_user_mr, + .reg_user_mr_dmabuf = rxe_reg_user_mr_dmabuf, .req_notify_cq = rxe_req_notify_cq, .resize_cq = rxe_resize_cq,
Signature · 1 line
2.17.1
Hi maintainers,
Could you please review this patch series?
Regards,
Shunsuke Mie
2021年11月22日(月) 20:08 Shunsuke Mie <[email protected]>:
Quoted 51 lines
This patch series add a dma-buf support for rxe driver.
A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and CMA managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.
[1] https://www.spinics.net/lists/linux-rdma/msg98592.html
To enable to use the dma-buf memory in rxe rdma device, add some changes and implementation in this patch series.
This series consists of two patches. The first patch changes the IB core to support for rdma drivers that has not dma device. The secound patch adds the dma-buf support to rxe driver.
Related user space RDMA library changes are provided as a separate patch.
v4:
* Fix warnings, unused variable and casting
v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+)
* Fix to use dma-buf-map helpers
v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+)
* Instead of using a dummydma_deviceto attach dma-buf, just store dma-buf to use software RDMA driver
* Use dma-bufvmap()interface
* Check to pass tests of rdma-core
v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
* The initial patch set
* Useib_deviceasdma_device.
* Use dma-buf dynamic attach interface
* Add dma-buf support to rxe deviceShunsuke Mie (2):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Add dma-buf supportdrivers/infiniband/core/umem_dmabuf.c | 20 ++++- drivers/infiniband/sw/rxe/rxe_loc.h | 2 + drivers/infiniband/sw/rxe/rxe_mr.c | 113 ++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_verbs.c | 34 ++++++++ include/rdma/ib_umem.h | 1 + 5 files changed, 166 insertions(+), 4 deletions(-)--
2.17.1
Quoted 3 lines · Shunsuke Mie
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
Jason
Hi Jason,
Thank you for replying.
2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:
Quoted 12 lines · Shunsuke Mie
Quoted 3 lines
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.
Is it wrong? If this is wrong, there is no advantages this changes..
Quoted 2 lines
Jason
Regards,
Shunsuke
Quoted 12 lines · Shunsuke Mie
Hi Jason,
Thank you for replying.2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:
Quoted 8 lines · Shunsuke Mie
Quoted 3 lines
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
Quoted 8 lines
Quoted 4 lines
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.
Is it wrong? If this is wrong, there is no advantages this changes..
I don't know what the dmabuf folks did, but yes, it is wrong.
IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.
Christian? If the vmap succeeds what rules must the caller use to access the memory?
Jason
Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
Quoted 19 lines · Shunsuke Mie
Quoted 10 lines
Hi Jason,
Thank you for replying.2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:
Quoted 6 lines · Shunsuke Mie
Quoted 3 lines
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
Quoted 8 lines
Quoted 4 lines
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.
Is it wrong? If this is wrong, there is no advantages this changes..
Quoted 8 lines
I don't know what the dmabuf folks did, but yes, it is wrong.
IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.
Christian? If the vmap succeeds what rules must the caller use to access the memory?
See dma-buf-map.h and especially struct dma_buf_map.
MMIO memory is perfectly supported here and actually the most common case.
Christian.
Quoted 2 lines
Jason
Quoted 26 lines · Christian König
Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
Quoted 25 lines · Shunsuke Mie
Quoted 10 lines
Hi Jason,
Thank you for replying.2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:
Quoted 6 lines · Shunsuke Mie
Quoted 3 lines
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
Quoted 8 lines
Quoted 4 lines
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.
Is it wrong? If this is wrong, there is no advantages this changes..
I don't know what the dmabuf folks did, but yes, it is wrong.
IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.
Quoted 6 lines
Quoted 2 lines
Christian? If the vmap succeeds what rules must the caller use to access the memory?
See
dma-buf-map.hand especially structdma_buf_map.MMIO memory is perfectly supported here and actually the most common case.
Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.
??
Jason
Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:
Quoted 29 lines · Christian König
Quoted 28 lines
Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
Quoted 27 lines · Shunsuke Mie
Quoted 10 lines
Hi Jason,
Thank you for replying.2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:
Quoted 6 lines · Shunsuke Mie
Quoted 3 lines
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
Quoted 8 lines
Quoted 4 lines
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.
Is it wrong? If this is wrong, there is no advantages this changes..
I don't know what the dmabuf folks did, but yes, it is wrong.
IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.
Christian? If the vmap succeeds what rules must the caller use to access the memory?
Quoted 8 lines
Quoted 3 lines
See
dma-buf-map.hand especially structdma_buf_map.MMIO memory is perfectly supported here and actually the most common case.
Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.
??
Well, yes that can go boom pretty quickly.
Not sure what they want to use this for.
Christian.
Quoted 2 lines
Jason
2021年12月10日(金) 22:29 Christian König <[email protected]>:
Quoted 33 lines
Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:
Quoted 31 lines · Christian König
Quoted 30 lines
Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
Quoted 19 lines · Shunsuke Mie
Quoted 10 lines
Hi Jason,
Thank you for replying.2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:
Quoted 6 lines · Shunsuke Mie
Quoted 3 lines
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
Quoted 8 lines
Quoted 4 lines
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.
Is it wrong? If this is wrong, there is no advantages this changes..
Quoted 8 lines
I don't know what the dmabuf folks did, but yes, it is wrong.
IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.
Christian? If the vmap succeeds what rules must the caller use to access the memory?
See
dma-buf-map.hand especially structdma_buf_map.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
MMIO memory is perfectly supported here and actually the most common case.
Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.
??
Well, yes that can go boom pretty quickly.
Sorry, I was wrong. The dma_buf_map treats both iomem and vaddr region, but this RFC only supports vaddr. Advantage of the partial support is we can use the vaddr dma-buf in RXE without changing a rxe data copy implementation.
An example of a dma-buf pointing to a vaddr is some gpu drivers use RAM for VRAM and we can get dma-buf for the region that indicates vaddr regions. Specifically, the gpu driver using gpu/drm/drm_gem_cma_helper.c is one such example.
Quoted 1 line
Not sure what they want to use this for.
I'd like to use RDMA with RXE for that memory region.
Best,
Shunsuke
Quoted 5 lines
Christian.
Quoted 2 lines
Jason
Am 13.12.21 um 12:18 schrieb Shunsuke Mie:
Quoted 38 lines
2021年12月10日(金) 22:29 Christian König <[email protected]>:
Quoted 37 lines
Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:
Quoted 36 lines · Christian König
Quoted 26 lines
Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
Quoted 25 lines · Shunsuke Mie
Quoted 10 lines
Hi Jason,
Thank you for replying.2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:
Quoted 6 lines · Shunsuke Mie
Quoted 3 lines
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
Quoted 8 lines
Quoted 4 lines
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.
Is it wrong? If this is wrong, there is no advantages this changes..
I don't know what the dmabuf folks did, but yes, it is wrong.
IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.
Quoted 5 lines
Quoted 2 lines
Christian? If the vmap succeeds what rules must the caller use to access the memory?
See
dma-buf-map.hand especially structdma_buf_map.MMIO memory is perfectly supported here and actually the most common case.
Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
??
Well, yes that can go boom pretty quickly.
Sorry, I was wrong. The
dma_buf_maptreats both iomem and vaddr region, but this RFC only supports vaddr. Advantage of the partial support is we can use the vaddr dma-buf in RXE without changing a rxe data copy implementation.
Well that is most likely not a good idea.
For example buffers for GPU drivers can be placed in both MMIO memory and system memory.
If you don't want to provoke random failures you *MUST* be able to handle both if you want to use this.
Regards,
Christian.
Quoted 14 lines
An example of a dma-buf pointing to a vaddr is some gpu drivers use RAM for VRAM and we can get dma-buf for the region that indicates vaddr regions. Specifically, the gpu driver using
gpu/drm/drm_gem_cma_helper.cis one such example.Quoted 1 line
Not sure what they want to use this for.
I'd like to use RDMA with RXE for that memory region.
Best,
ShunsukeQuoted 3 lines
Christian.
Quoted 1 line
Jason
2021年12月14日(火) 17:54 Christian König <[email protected]>:
Quoted 46 lines
Am 13.12.21 um 12:18 schrieb Shunsuke Mie:
Quoted 38 lines
2021年12月10日(金) 22:29 Christian König <[email protected]>:
Quoted 37 lines
Am 10.12.21 um 14:26 schrieb Jason Gunthorpe:
Quoted 36 lines · Christian König
Quoted 26 lines
Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
Quoted 25 lines · Shunsuke Mie
Quoted 10 lines
Hi Jason,
Thank you for replying.2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:
Quoted 6 lines · Shunsuke Mie
Quoted 3 lines
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
Quoted 8 lines
Quoted 4 lines
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.
Is it wrong? If this is wrong, there is no advantages this changes..
I don't know what the dmabuf folks did, but yes, it is wrong.
IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.
Quoted 5 lines
Quoted 2 lines
Christian? If the vmap succeeds what rules must the caller use to access the memory?
See
dma-buf-map.hand especially structdma_buf_map.MMIO memory is perfectly supported here and actually the most common case.
Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
??
Well, yes that can go boom pretty quickly.
Sorry, I was wrong. The
dma_buf_maptreats both iomem and vaddr region, but this RFC only supports vaddr. Advantage of the partial support is we can use the vaddr dma-buf in RXE without changing a rxe data copy implementation.
Quoted 7 lines
Well that is most likely not a good idea.
For example buffers for GPU drivers can be placed in both MMIO memory and system memory.
If you don't want to provoke random failures you *MUST* be able to handle both if you want to use this.
I agree with you. I'll add the support and resubmit patch series.
Thanks a lot,
Shunsuke.
Quoted 19 lines
Regards,
Christian.Quoted 14 lines
An example of a dma-buf pointing to a vaddr is some gpu drivers use RAM for VRAM and we can get dma-buf for the region that indicates vaddr regions. Specifically, the gpu driver using
gpu/drm/drm_gem_cma_helper.cis one such example.Quoted 1 line
Not sure what they want to use this for.
I'd like to use RDMA with RXE for that memory region.
Best,
ShunsukeQuoted 3 lines
Christian.
Quoted 1 line
Jason
Quoted 32 lines · Jason Gunthorpe
Quoted 31 lines · Christian König
Am 10.12.21 um 13:42 schrieb Jason Gunthorpe:
Quoted 19 lines · Shunsuke Mie
Quoted 10 lines
Hi Jason,
Thank you for replying.2021年12月8日(水) 2:14 Jason Gunthorpe <[email protected]>:
Quoted 6 lines · Shunsuke Mie
Quoted 3 lines
Hi maintainers,
Could you please review this patch series?
Why is it RFC?
Quoted 8 lines
Quoted 4 lines
I'm confused why this is useful?
This can't do copy from MMIO memory, so it shouldn't be compatible with things like Gaudi - does something prevent this?
I think if an export of the dma-buf supports vmap, CPU is able to access the mmio memory.
Is it wrong? If this is wrong, there is no advantages this changes..
Quoted 8 lines
I don't know what the dmabuf folks did, but yes, it is wrong.
IOMEM must be touched using only special accessors, some platforms crash if you don't do this. Even x86 will crash if you touch it with something like an XMM optimized memcpy.
Christian? If the vmap succeeds what rules must the caller use to access the memory?
See
dma-buf-map.hand especially structdma_buf_map.
Quoted 7 lines
Quoted 1 line
MMIO memory is perfectly supported here and actually the most common case.
Okay that looks sane, but this rxe RFC seems to ignore this completely. It stuffs the vaddr directly into a umem which goes to all manner of places in the driver.
??
dma_buf_map is fairly new and we haven't rolled it out consistently yet. In the past 10 years we simply yolo'd this :-)
Just an explanation, not an excuse for new code to not use dma_buf_map consistently now that we fixed this mistake.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Quoted 8 lines · Shunsuke Mie
This patch series add a dma-buf support for rxe driver.
A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and CMA managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.
Quoted 9 lines
[1] https://www.spinics.net/lists/linux-rdma/msg98592.html
To enable to use the dma-buf memory in rxe rdma device, add some changes and implementation in this patch series.
This series consists of two patches. The first patch changes the IB core to support for rdma drivers that has not dma device. The secound patch adds the dma-buf support to rxe driver.
Hi, Shunsuke Mie
I was revisiting your 2021 proposal around dma-buf integration with RDMA and the related discussions at the time.
As you know, dma-buf usage in RDMA-related workflows has gained more traction recently, and we are seeing increasing interest in heterogeneous memory and cross-device buffer sharing. Given the changes in the ecosystem since then, I’m wondering whether you think the original direction might be worth reconsidering.
Do you have any interest in continuing that line of work, or updating the design based on today’s context? If not, I’d still appreciate your perspective on what you see as the main blockers from the previous discussions, and whether you think the landscape has changed enough to justify another attempt.
Depending on the direction, we may consider exploring dma-buf support in rxe or at the core level, but I’d prefer to first understand your view before moving forward.
Zhu Yanjun
Quoted 30 lines
Related user space RDMA library changes are provided as a separate patch.
v4:
* Fix warnings, unused variable and casting
v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+)
* Fix to use dma-buf-map helpers
v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+)
* Instead of using a dummydma_deviceto attach dma-buf, just store dma-buf to use software RDMA driver
* Use dma-bufvmap()interface
* Check to pass tests of rdma-core
v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
* The initial patch set
* Useib_deviceasdma_device.
* Use dma-buf dynamic attach interface
* Add dma-buf support to rxe deviceShunsuke Mie (2):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Add dma-buf supportdrivers/infiniband/core/umem_dmabuf.c | 20 ++++- drivers/infiniband/sw/rxe/rxe_loc.h | 2 + drivers/infiniband/sw/rxe/rxe_mr.c | 113 ++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_verbs.c | 34 ++++++++ include/rdma/ib_umem.h | 1 + 5 files changed, 166 insertions(+), 4 deletions(-)
Hi Zhu Yanjun,
Thank you for reaching out and revisiting my 2021 proposal.
Regarding the current status, I haven't been able to make much progress recently as other tasks have taken higher priority. However, I still believe this integration is important.
From a technical perspective, as pointed out during the previous reviews, there were indeed issues with how dma_buf_map was being used. To address this in today's context, I believe we should transition to using iosys_map.
I am still interested in this direction. While my current bandwidth is limited, I would welcome any collaboration—especially regarding the implementation of iosys_map support within rxe or the RDMA core.
I'd be happy to discuss the technical details of this transition if you'd like to dive deeper.
Best,
Shunsuke Mie
2026年2月19日(木) 13:43 Zhu Yanjun <[email protected]>:
Quoted 75 lines · Shunsuke Mie
Quoted 4 lines
This patch series add a dma-buf support for rxe driver.
A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and
CMA
Quoted 11 lines
managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.
[1] https://www.spinics.net/lists/linux-rdma/msg98592.html
To enable to use the dma-buf memory in rxe rdma device, add some changes and implementation in this patch series.
This series consists of two patches. The first patch changes the IB core to support for rdma drivers that has not dma device. The secound patch
adds
Quoted 2 lines
the dma-buf support to rxe driver.
Hi, Shunsuke Mie
I was revisiting your 2021 proposal around dma-buf integration with RDMA and the related discussions at the time.
As you know, dma-buf usage in RDMA-related workflows has gained more traction recently, and we are seeing increasing interest in heterogeneous memory and cross-device buffer sharing. Given the changes in the ecosystem since then, I’m wondering whether you think the original direction might be worth reconsidering.
Do you have any interest in continuing that line of work, or updating the design based on today’s context? If not, I’d still appreciate your perspective on what you see as the main blockers from the previous discussions, and whether you think the landscape has changed enough to justify another attempt.
Depending on the direction, we may consider exploring dma-buf support in rxe or at the core level, but I’d prefer to first understand your view before moving forward.
Zhu Yanjun
Quoted 30 lines
Related user space RDMA library changes are provided as a separate patch.
v4:
* Fix warnings, unused variable and casting
v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+)
* Fix to use dma-buf-map helpers
v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
* Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+)
* Instead of using a dummydma_deviceto attach dma-buf, just store dma-buf to use software RDMA driver
* Use dma-bufvmap()interface
* Check to pass tests of rdma-core
v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
* The initial patch set
* Useib_deviceasdma_device.
* Use dma-buf dynamic attach interface
* Add dma-buf support to rxe deviceShunsuke Mie (2):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Add dma-buf supportdrivers/infiniband/core/umem_dmabuf.c | 20 ++++- drivers/infiniband/sw/rxe/rxe_loc.h | 2 + drivers/infiniband/sw/rxe/rxe_mr.c | 113 ++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_verbs.c | 34 ++++++++ include/rdma/ib_umem.h | 1 + 5 files changed, 166 insertions(+), 4 deletions(-)
Quoted 11 lines · Shunsuke Mie
Hi Zhu Yanjun,
Thank you for reaching out and revisiting my 2021 proposal.
Regarding the current status, I haven't been able to make much progress recently as other tasks have taken higher priority. However, I still believe this integration is important.
From a technical perspective, as pointed out during the previous reviews, there were indeed issues with how
dma_buf_mapwas being used. To address this in today's context, I believe we should transition to usingiosys_map.I am still interested in this direction. While my current bandwidth is limited, I would welcome any collaboration—especially regarding the implementation of
iosys_mapsupport within rxe or the RDMA core.I'd be happy to discuss the technical details of this transition if you'd like to dive deeper.
DMABUF has become increasingly popular and is widely adopted across various subsystems. If we could also implement DMABUF support in rxe, it would significantly enhance rxe’s usability and overall value.
Given your previous involvement in this area, it would be great to move this forward in rxe. I believe your expertise would be particularly helpful in driving the implementation. I’m happy to review and collaborate on the design and patches as needed.
Zhu Yanjun
Quoted 94 lines
Best,
Shunsuke Mie2026年2月19日(木) 13:43 Zhu Yanjun <[email protected]>:
在 2021/11/22 3:08, Shunsuke Mie 写道:
> This patch series add a dma-buf support for rxe driver. >
> A dma-buf based memory registering has beed introduced to use the memory
> region that lack of associated page structures (e.g. device memory and CMA
> managed memory) [1]. However, to use the dma-buf based memory, each rdma
> device drivers require add some implementation. The rxe driver has not
> support yet.
>
> [1] https://www.spinics.net/lists/linux-rdma/msg98592.html
>
> To enable to use the dma-buf memory in rxe rdma device, add some changes
> and implementation in this patch series.
>
> This series consists of two patches. The first patch changes the IB core
> to support for rdma drivers that has not dma device. The secound patch adds
> the dma-buf support to rxe driver.
>
Hi, Shunsuke MieI was revisiting your 2021 proposal around dma-buf integration with RDMA
and the related discussions at the time.As you know, dma-buf usage in RDMA-related workflows has gained more traction recently, and we are seeing increasing interest in heterogeneous memory and cross-device buffer sharing. Given the changes
in the ecosystem since then, I’m wondering whether you think the original direction might be worth reconsidering.Do you have any interest in continuing that line of work, or updating the design based on today’s context? If not, I’d still appreciate your
perspective on what you see as the main blockers from the previous discussions, and whether you think the landscape has changed enough to
justify another attempt.Depending on the direction, we may consider exploring dma-buf support in
rxe or at the core level, but I’d prefer to first understand your view
before moving forward.Zhu Yanjun
> Related user space RDMA library changes are provided as a separate patch.
>
> v4:
> * Fix warnings, unused variable and casting
> v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
> * Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+) > * Fix to use dma-buf-map helpers
> v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
> * Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+) > * Instead of using a dummydma_deviceto attach dma-buf, just store > dma-buf to use software RDMA driver
> * Use dma-bufvmap()interface
> * Check to pass tests of rdma-core
> v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
> * The initial patch set
> * Useib_deviceasdma_device.
> * Use dma-buf dynamic attach interface
> * Add dma-buf support to rxe device
>
> Shunsuke Mie (2):
> RDMA/umem: Change for rdma devices has not dma device > RDMA/rxe: Add dma-buf support
>
> drivers/infiniband/core/umem_dmabuf.c | 20 ++++-
> drivers/infiniband/sw/rxe/rxe_loc.h | 2 +
> drivers/infiniband/sw/rxe/rxe_mr.c | 113
++++++++++++++++++++++++++
> drivers/infiniband/sw/rxe/rxe_verbs.c | 34 ++++++++ > include/rdma/ib_umem.h | 1 +
> 5 files changed, 166 insertions(+), 4 deletions(-)
>
Signature · 2 lines
Best Regards, Yanjun.Zhu
Quoted 9 lines · Shunsuke Mie
Hi Zhu Yanjun,
Thank you for reaching out and revisiting my 2021 proposal.
Regarding the current status, I haven't been able to make much progress recently as other tasks have taken higher priority. However, I still believe this integration is important.
From a technical perspective, as pointed out during the previous reviews, there were indeed issues with how
dma_buf_mapwas being used. To address this in today's context, I believe we should transition to usingiosys_map.I am still interested in this direction. While my current bandwidth is limited, I would welcome any collaboration—especially regarding the implementation of
iosys_mapsupport within rxe or the RDMA core.
Sorry. In the previous mail, there is a HTML part. It can not reach. As such, I resend it.
DMABUF has become increasingly popular and is widely adopted across various subsystems.
If we could also implement DMABUF support in rxe, it would significantly enhance rxe’s usability and overall value.
Given your previous involvement in this area, it would be great to move this forward in rxe.
I believe your expertise would be particularly helpful in driving the implementation.
I’m happy to review and collaborate on the design and patches as needed.
Zhu Yanjun
Quoted 96 lines
I'd be happy to discuss the technical details of this transition if you'd like to dive deeper.
Best,
Shunsuke Mie2026年2月19日(木) 13:43 Zhu Yanjun <[email protected]>:
在 2021/11/22 3:08, Shunsuke Mie 写道:
> This patch series add a dma-buf support for rxe driver. >
> A dma-buf based memory registering has beed introduced to use the memory
> region that lack of associated page structures (e.g. device memory and CMA
> managed memory) [1]. However, to use the dma-buf based memory, each rdma
> device drivers require add some implementation. The rxe driver has not
> support yet.
>
> [1] https://www.spinics.net/lists/linux-rdma/msg98592.html
>
> To enable to use the dma-buf memory in rxe rdma device, add some changes
> and implementation in this patch series.
>
> This series consists of two patches. The first patch changes the IB core
> to support for rdma drivers that has not dma device. The secound patch adds
> the dma-buf support to rxe driver.
>
Hi, Shunsuke MieI was revisiting your 2021 proposal around dma-buf integration with RDMA
and the related discussions at the time.As you know, dma-buf usage in RDMA-related workflows has gained more traction recently, and we are seeing increasing interest in heterogeneous memory and cross-device buffer sharing. Given the changes
in the ecosystem since then, I’m wondering whether you think the original direction might be worth reconsidering.Do you have any interest in continuing that line of work, or updating the design based on today’s context? If not, I’d still appreciate your
perspective on what you see as the main blockers from the previous discussions, and whether you think the landscape has changed enough to
justify another attempt.Depending on the direction, we may consider exploring dma-buf support in
rxe or at the core level, but I’d prefer to first understand your view
before moving forward.Zhu Yanjun
> Related user space RDMA library changes are provided as a separate patch.
>
> v4:
> * Fix warnings, unused variable and casting
> v3: https://www.spinics.net/lists/linux-rdma/msg106776.html
> * Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc6+) > * Fix to use dma-buf-map helpers
> v2: https://www.spinics.net/lists/linux-rdma/msg105928.html
> * Rebase to the latest linux-rdma 'for-next' branch (5.15.0-rc1+) > * Instead of using a dummydma_deviceto attach dma-buf, just store > dma-buf to use software RDMA driver
> * Use dma-bufvmap()interface
> * Check to pass tests of rdma-core
> v1: https://www.spinics.net/lists/linux-rdma/msg105376.html
> * The initial patch set
> * Useib_deviceasdma_device.
> * Use dma-buf dynamic attach interface
> * Add dma-buf support to rxe device
>
> Shunsuke Mie (2):
> RDMA/umem: Change for rdma devices has not dma device > RDMA/rxe: Add dma-buf support
>
> drivers/infiniband/core/umem_dmabuf.c | 20 ++++-
> drivers/infiniband/sw/rxe/rxe_loc.h | 2 +
> drivers/infiniband/sw/rxe/rxe_mr.c | 113
++++++++++++++++++++++++++
> drivers/infiniband/sw/rxe/rxe_verbs.c | 34 ++++++++ > include/rdma/ib_umem.h | 1 +
> 5 files changed, 166 insertions(+), 4 deletions(-)
>