Skip to contentSkip to conversation

RDMA/rxe: Add dma-buf support

RFC PATCH In review Shunsuke Mie 7 participants 22 messages Sep 8, 2021 – Sep 14, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover RDMA/rxe: Add dma-buf support

This patch series add a support for rxe driver.

A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and CMA managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.

[1] https://www.spinics.net/lists/linux-rdma/msg98592.html

To enable to use the memories in rxe rdma device, add some changes and implementation in this patch series.

This series consists of three patches. The first patch changes the IB core to support for rdma drivers that have not real dma device. The second patch extracts a memory mapping process of rxe as a common function to use a dma-buf support. The third patch adds the dma-buf support to rxe driver.

Related user space RDMA library changes are provided as a separate patch.

Shunsuke Mie (3):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Extract a mapping process into a function RDMA/rxe: Support dma-buf as memory region

 drivers/infiniband/core/umem_dmabuf.c |   2 +-
 drivers/infiniband/sw/rxe/rxe_loc.h   |   3 +
 drivers/infiniband/sw/rxe/rxe_mr.c    | 186 +++++++++++++++++++++-----
 drivers/infiniband/sw/rxe/rxe_verbs.c |  36 +++++
 4 files changed, 193 insertions(+), 34 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

RFC PATCH 1/3 RDMA/umem: Change for rdma devices has not dma device

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

---
 drivers/infiniband/core/umem_dmabuf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/core/umem_dmabuf.c +1 -1
index c6e875619fac..58d6ac9cb51a 100644
@@ -146,7 +146,7 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,
 
 	umem_dmabuf->attach = dma_buf_dynamic_attach(
 					dmabuf,
-					device->dma_device,
+					device->dma_device ? device->dma_device : &device->dev,
 					ops,
 					umem_dmabuf);
 	if (IS_ERR(umem_dmabuf->attach)) {
Signature · 1 line
2.17.1
#2 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Quoted 3 lines · Shunsuke Mie

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

#3 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Thank you for your comment.

Quoted 8 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf exporter to know the device buffer constraints of importer.
[1] https://lwn.net/Articles/489703/

#4 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Quoted 7 lines · Shunsuke Mie

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

Quoted 5 lines
Quoted 2 lines

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf exporter to know the device buffer constraints of importer.
[1] https://lwn.net/Articles/489703/

Which means for rxe you'd also have to pass the one for the underlying net device.

#5 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 9 lines · Shunsuke Mie
Quoted 7 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf exporter to know the device buffer constraints of importer.
[1] https://lwn.net/Articles/489703/

Which means for rxe you'd also have to pass the one for the underlying net device.

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Thanks,

#6 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Quoted 16 lines · Shunsuke Mie

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 15 lines · Shunsuke Mie
Quoted 12 lines

Thank you for your comment.

Quoted 8 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf exporter to know the device buffer constraints of importer.
[1] https://lwn.net/Articles/489703/

Quoted 5 lines
Quoted 2 lines

Which means for rxe you'd also have to pass the one for the underlying net device.

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Jason

#7 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Am 08.09.21 um 13:18 schrieb Jason Gunthorpe:

Quoted 15 lines · Shunsuke Mie
Quoted 14 lines

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 13 lines · Shunsuke Mie
Quoted 10 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf
exporter to know the device buffer constraints of importer.
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F489703%2F&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C4d18470a94df4ed24c8108d972ba5591%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637666967356417448%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ARwQyo%2BCjMohaNbyREofToHIj2bndL5L0HaU9cOrYq4%3D&amp;reserved=0

Which means for rxe you'd also have to pass the one for the underlying net device.

Quoted 8 lines
Quoted 3 lines

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Yeah, agree that doesn't make much sense.

When you want to access the data with the CPU then why do you want to use DMA-buf in the first place?

Please keep in mind that there is work ongoing to replace the sg table with an DMA address array and so make the underlying struct page inaccessible for importers.

Regards,
Christian.

Quoted 2 lines

Jason

#8 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Quoted 25 lines · Christian König

Am 08.09.21 um 13:18 schrieb Jason Gunthorpe:

Quoted 15 lines · Shunsuke Mie
Quoted 14 lines

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 13 lines · Shunsuke Mie
Quoted 10 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf
exporter to know the device buffer constraints of importer.
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F489703%2F&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C4d18470a94df4ed24c8108d972ba5591%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637666967356417448%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ARwQyo%2BCjMohaNbyREofToHIj2bndL5L0HaU9cOrYq4%3D&amp;reserved=0

Which means for rxe you'd also have to pass the one for the underlying net device.

Quoted 8 lines
Quoted 3 lines

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Quoted 8 lines

Yeah, agree that doesn't make much sense.

When you want to access the data with the CPU then why do you want to use DMA-buf in the first place?

Please keep in mind that there is work ongoing to replace the sg table with an DMA address array and so make the underlying struct page inaccessible for importers.

Also if you do have a dma-buf, you can just dma_buf_vmap() the buffer for cpu access. Which intentionally does not require any device. No idea why there's a dma_buf_attach involved. Now not all exporters support this, but that's fixable, and you must call dma_buf_begin/end_cpu_access for cache management if the allocation isn't cpu coherent. But it's all there, no need to apply hacks of allowing a wrong device or other fun things.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#9 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Quoted 35 lines · Daniel Vetter
Quoted 25 lines · Christian König

Am 08.09.21 um 13:18 schrieb Jason Gunthorpe:

Quoted 15 lines · Shunsuke Mie
Quoted 14 lines

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 13 lines · Shunsuke Mie
Quoted 10 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf
exporter to know the device buffer constraints of importer.
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F489703%2F&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C4d18470a94df4ed24c8108d972ba5591%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637666967356417448%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ARwQyo%2BCjMohaNbyREofToHIj2bndL5L0HaU9cOrYq4%3D&amp;reserved=0

Which means for rxe you'd also have to pass the one for the underlying net device.

Quoted 8 lines
Quoted 3 lines

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Quoted 8 lines

Yeah, agree that doesn't make much sense.

When you want to access the data with the CPU then why do you want to use DMA-buf in the first place?

Please keep in mind that there is work ongoing to replace the sg table with an DMA address array and so make the underlying struct page inaccessible for importers.

Quoted 7 lines

Also if you do have a dma-buf, you can just dma_buf_vmap() the buffer for cpu access. Which intentionally does not require any device. No idea why there's a dma_buf_attach involved. Now not all exporters support this, but that's fixable, and you must call dma_buf_begin/end_cpu_access for cache management if the allocation isn't cpu coherent. But it's all there, no need to apply hacks of allowing a wrong device or other fun things.

Can rxe leave the vmap in place potentially forever?

Jason

#10 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Quoted 44 lines · Jason Gunthorpe
Quoted 35 lines · Daniel Vetter
Quoted 25 lines · Christian König

Am 08.09.21 um 13:18 schrieb Jason Gunthorpe:

Quoted 15 lines · Shunsuke Mie
Quoted 14 lines

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 13 lines · Shunsuke Mie
Quoted 10 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf
exporter to know the device buffer constraints of importer.
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F489703%2F&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C4d18470a94df4ed24c8108d972ba5591%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637666967356417448%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ARwQyo%2BCjMohaNbyREofToHIj2bndL5L0HaU9cOrYq4%3D&amp;reserved=0

Which means for rxe you'd also have to pass the one for the underlying net device.

Quoted 8 lines
Quoted 3 lines

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Quoted 8 lines

Yeah, agree that doesn't make much sense.

When you want to access the data with the CPU then why do you want to use DMA-buf in the first place?

Please keep in mind that there is work ongoing to replace the sg table with an DMA address array and so make the underlying struct page inaccessible for importers.

Quoted 7 lines

Also if you do have a dma-buf, you can just dma_buf_vmap() the buffer for cpu access. Which intentionally does not require any device. No idea why there's a dma_buf_attach involved. Now not all exporters support this, but that's fixable, and you must call dma_buf_begin/end_cpu_access for cache management if the allocation isn't cpu coherent. But it's all there, no need to apply hacks of allowing a wrong device or other fun things.

Quoted 1 line

Can rxe leave the vmap in place potentially forever?

Yeah, it's like perma-pinning the buffer into system memory for non-p2p dma-buf sharing. We just squint and pretend that can't be abused too badly :-) On 32bit you'll run out of vmap space rather quickly, but that's not something anyone cares about here either. We have a bunch of more sw modesetting drivers in drm which use dma_buf_vmap() like this, so it's all fine.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#11 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

2021年9月9日(木) 18:26 Daniel Vetter <[email protected]>:

Quoted 48 lines · Jason Gunthorpe
Quoted 44 lines · Daniel Vetter
Quoted 35 lines · Christian König
Quoted 25 lines

Am 08.09.21 um 13:18 schrieb Jason Gunthorpe:

Quoted 15 lines · Shunsuke Mie
Quoted 14 lines

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 13 lines · Shunsuke Mie
Quoted 10 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf
exporter to know the device buffer constraints of importer.
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F489703%2F&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C4d18470a94df4ed24c8108d972ba5591%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637666967356417448%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ARwQyo%2BCjMohaNbyREofToHIj2bndL5L0HaU9cOrYq4%3D&amp;reserved=0

Which means for rxe you'd also have to pass the one for the underlying net device.

Quoted 8 lines
Quoted 3 lines

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Quoted 8 lines

Yeah, agree that doesn't make much sense.

When you want to access the data with the CPU then why do you want to use DMA-buf in the first place?

Please keep in mind that there is work ongoing to replace the sg table with an DMA address array and so make the underlying struct page inaccessible for importers.

Quoted 7 lines

Also if you do have a dma-buf, you can just dma_buf_vmap() the buffer for cpu access. Which intentionally does not require any device. No idea why there's a dma_buf_attach involved. Now not all exporters support this, but that's fixable, and you must call dma_buf_begin/end_cpu_access for cache management if the allocation isn't cpu coherent. But it's all there, no need to apply hacks of allowing a wrong device or other fun things.

Quoted 1 line

Can rxe leave the vmap in place potentially forever?

Quoted 11 lines

Yeah, it's like perma-pinning the buffer into system memory for non-p2p dma-buf sharing. We just squint and pretend that can't be abused too badly :-) On 32bit you'll run out of vmap space rather quickly, but that's not something anyone cares about here either. We have a bunch of more sw modesetting drivers in drm which use dma_buf_vmap() like this, so it's all fine.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Thanks for your comments.

In the first place, the CMA region cannot be used for RDMA because the region has no struct page. In addition, some GPU drivers use CMA and share the region as dma-buf. As a result, RDMA cannot transfer for the region. To solve this problem, rxe dma-buf support is better I thought.

I'll consider and redesign the rxe dma-buf support using the dma_buf_vmap() instead of the dma_buf_dynamic_attach().

Regards,
Shunsuke

#12 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Quoted 64 lines · Shunsuke Mie

2021年9月9日(木) 18:26 Daniel Vetter <[email protected]>:

Quoted 48 lines · Jason Gunthorpe
Quoted 44 lines · Daniel Vetter
Quoted 35 lines · Christian König
Quoted 25 lines

Am 08.09.21 um 13:18 schrieb Jason Gunthorpe:

Quoted 15 lines · Shunsuke Mie
Quoted 14 lines

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 13 lines · Shunsuke Mie
Quoted 10 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf
exporter to know the device buffer constraints of importer.
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F489703%2F&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C4d18470a94df4ed24c8108d972ba5591%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637666967356417448%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ARwQyo%2BCjMohaNbyREofToHIj2bndL5L0HaU9cOrYq4%3D&amp;reserved=0

Which means for rxe you'd also have to pass the one for the underlying net device.

Quoted 8 lines
Quoted 3 lines

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Quoted 8 lines

Yeah, agree that doesn't make much sense.

When you want to access the data with the CPU then why do you want to use DMA-buf in the first place?

Please keep in mind that there is work ongoing to replace the sg table with an DMA address array and so make the underlying struct page inaccessible for importers.

Quoted 7 lines

Also if you do have a dma-buf, you can just dma_buf_vmap() the buffer for cpu access. Which intentionally does not require any device. No idea why there's a dma_buf_attach involved. Now not all exporters support this, but that's fixable, and you must call dma_buf_begin/end_cpu_access for cache management if the allocation isn't cpu coherent. But it's all there, no need to apply hacks of allowing a wrong device or other fun things.

Quoted 1 line

Can rxe leave the vmap in place potentially forever?

Quoted 11 lines

Yeah, it's like perma-pinning the buffer into system memory for non-p2p dma-buf sharing. We just squint and pretend that can't be abused too badly :-) On 32bit you'll run out of vmap space rather quickly, but that's not something anyone cares about here either. We have a bunch of more sw modesetting drivers in drm which use dma_buf_vmap() like this, so it's all fine.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Thanks for your comments.

Quoted 7 lines

In the first place, the CMA region cannot be used for RDMA because the region has no struct page. In addition, some GPU drivers use CMA and share the region as dma-buf. As a result, RDMA cannot transfer for the region. To solve this problem, rxe dma-buf support is better I thought.

I'll consider and redesign the rxe dma-buf support using the dma_buf_vmap() instead of the dma_buf_dynamic_attach().

btw for next version please cc dri-devel. get_maintainers.pl should pick it up for these patches.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#13 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

2021年9月14日(火) 4:23 Daniel Vetter <[email protected]>:

Quoted 71 lines · Shunsuke Mie
Quoted 69 lines

2021年9月9日(木) 18:26 Daniel Vetter <[email protected]>:

Quoted 48 lines · Jason Gunthorpe
Quoted 44 lines · Daniel Vetter
Quoted 35 lines · Christian König
Quoted 25 lines

Am 08.09.21 um 13:18 schrieb Jason Gunthorpe:

Quoted 15 lines · Shunsuke Mie
Quoted 14 lines

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 13 lines · Shunsuke Mie
Quoted 10 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma device, but devices such as rxe do not have a dma device. For those case, change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf
exporter to know the device buffer constraints of importer.
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F489703%2F&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C4d18470a94df4ed24c8108d972ba5591%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637666967356417448%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ARwQyo%2BCjMohaNbyREofToHIj2bndL5L0HaU9cOrYq4%3D&amp;reserved=0

Which means for rxe you'd also have to pass the one for the underlying net device.

Quoted 8 lines
Quoted 3 lines

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Quoted 8 lines

Yeah, agree that doesn't make much sense.

When you want to access the data with the CPU then why do you want to use DMA-buf in the first place?

Please keep in mind that there is work ongoing to replace the sg table with an DMA address array and so make the underlying struct page inaccessible for importers.

Quoted 7 lines

Also if you do have a dma-buf, you can just dma_buf_vmap() the buffer for cpu access. Which intentionally does not require any device. No idea why there's a dma_buf_attach involved. Now not all exporters support this, but that's fixable, and you must call dma_buf_begin/end_cpu_access for cache management if the allocation isn't cpu coherent. But it's all there, no need to apply hacks of allowing a wrong device or other fun things.

Quoted 1 line

Can rxe leave the vmap in place potentially forever?

Quoted 11 lines

Yeah, it's like perma-pinning the buffer into system memory for non-p2p dma-buf sharing. We just squint and pretend that can't be abused too badly :-) On 32bit you'll run out of vmap space rather quickly, but that's not something anyone cares about here either. We have a bunch of more sw modesetting drivers in drm which use dma_buf_vmap() like this, so it's all fine.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Thanks for your comments.

In the first place, the CMA region cannot be used for RDMA because the region has no struct page. In addition, some GPU drivers use CMA and share the region as dma-buf. As a result, RDMA cannot transfer for the region. To solve this problem, rxe dma-buf support is better I thought.

Quoted 5 lines
Quoted 2 lines

I'll consider and redesign the rxe dma-buf support using the dma_buf_vmap() instead of the dma_buf_dynamic_attach().

btw for next version please cc dri-devel. get_maintainers.pl should pick it up for these patches.

A CC list of these patches is generated by get_maintainers.pl but it didn't pick up the dri-devel. Should I add the dri-devel to the cc manually?

Regards,
Shunsuke

#14 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

Quoted 73 lines · Shunsuke Mie

2021年9月14日(火) 4:23 Daniel Vetter <[email protected]>:

Quoted 71 lines · Shunsuke Mie
Quoted 69 lines

2021年9月9日(木) 18:26 Daniel Vetter <[email protected]>:

Quoted 48 lines · Jason Gunthorpe
Quoted 44 lines · Daniel Vetter
Quoted 35 lines · Christian König
Quoted 25 lines

Am 08.09.21 um 13:18 schrieb Jason Gunthorpe:

Quoted 15 lines · Shunsuke Mie
Quoted 14 lines

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 13 lines · Shunsuke Mie
Quoted 10 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie
Quoted 3 lines

To share memory space using dma-buf, a API of the dma-buf requires dma
device, but devices such as rxe do not have a dma device. For those case,
change to specify a device of struct ib instead of the dma device.

So if dma-buf doesn't actually need a device to dma map why do we ever pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf
exporter to know the device buffer constraints of importer.
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F489703%2F&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C4d18470a94df4ed24c8108d972ba5591%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637666967356417448%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ARwQyo%2BCjMohaNbyREofToHIj2bndL5L0HaU9cOrYq4%3D&amp;reserved=0

Which means for rxe you'd also have to pass the one for the underlying net device.

Quoted 8 lines
Quoted 3 lines

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Quoted 8 lines

Yeah, agree that doesn't make much sense.

When you want to access the data with the CPU then why do you want to use DMA-buf in the first place?

Please keep in mind that there is work ongoing to replace the sg table with an DMA address array and so make the underlying struct page inaccessible for importers.

Quoted 7 lines

Also if you do have a dma-buf, you can just dma_buf_vmap() the buffer for cpu access. Which intentionally does not require any device. No idea why there's a dma_buf_attach involved. Now not all exporters support this, but that's fixable, and you must call dma_buf_begin/end_cpu_access for cache management if the allocation isn't cpu coherent. But it's all there, no need to apply hacks of allowing a wrong device or other fun things.

Quoted 1 line

Can rxe leave the vmap in place potentially forever?

Quoted 11 lines

Yeah, it's like perma-pinning the buffer into system memory for non-p2p dma-buf sharing. We just squint and pretend that can't be abused too badly :-) On 32bit you'll run out of vmap space rather quickly, but that's not something anyone cares about here either. We have a bunch of more sw modesetting drivers in drm which use dma_buf_vmap() like this, so it's all fine.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Thanks for your comments.

In the first place, the CMA region cannot be used for RDMA because the region has no struct page. In addition, some GPU drivers use CMA and share the region as dma-buf. As a result, RDMA cannot transfer for the region. To solve this problem, rxe dma-buf support is better I thought.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I'll consider and redesign the rxe dma-buf support using the dma_buf_vmap() instead of the dma_buf_dynamic_attach().

btw for next version please cc dri-devel. get_maintainers.pl should pick it up for these patches.

A CC list of these patches is generated by get_maintainers.pl but it didn't pick up the dri-devel. Should I add the dri-devel to the cc manually?

Hm yes, on rechecking the regex doesn't match since you're not touching any dma-buf code directly. Or not directly enough for get_maintainers.pl to pick it up.

DMA BUFFER SHARING FRAMEWORK

M:    Sumit Semwal <[email protected]>
M:    Christian König <[email protected]>
L:    [email protected]
L:    [email protected]
L:    [email protected] (moderated for non-subscribers)
S:    Maintained
T:    git git://anongit.freedesktop.org/drm/drm-misc
F:    Documentation/driver-api/dma-buf.rst
F:    drivers/dma-buf/
F:    include/linux/*fence.h
F:    include/linux/dma-buf*
F:    include/linux/dma-resv.h
K:    \bdma_(?:buf|fence|resv)\b

Above is the MAINTAINERS entry that's always good to cc for anything related to dma_buf/fence/resv and any of these related things. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#15 lore raw

Re: [RFC PATCH 1/3] RDMA/umem: Change for rdma devices has not dma device

2021年9月14日(火) 18:38 Daniel Vetter <[email protected]>:

Quoted 103 lines · Shunsuke Mie
Quoted 73 lines

2021年9月14日(火) 4:23 Daniel Vetter <[email protected]>:

Quoted 71 lines · Shunsuke Mie
Quoted 69 lines

2021年9月9日(木) 18:26 Daniel Vetter <[email protected]>:

Quoted 48 lines · Jason Gunthorpe
Quoted 44 lines · Daniel Vetter
Quoted 35 lines · Christian König
Quoted 25 lines

Am 08.09.21 um 13:18 schrieb Jason Gunthorpe:

Quoted 15 lines · Shunsuke Mie
Quoted 14 lines

2021年9月8日(水) 16:20 Christoph Hellwig <[email protected]>:

Quoted 13 lines · Shunsuke Mie
Quoted 10 lines

Thank you for your comment.

Quoted 6 lines · Shunsuke Mie

To share memory space using dma-buf, a API of the dma-buf requires dma
device, but devices such as rxe do not have a dma device. For those case,
change to specify a device of struct ib instead of the dma device.
So if dma-buf doesn't actually need a device to dma map why do we ever
pass the dma_device here? Something does not add up.

As described in the dma-buf api guide [1], the dma_device is used by dma-buf
exporter to know the device buffer constraints of importer.
[1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F489703%2F&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C4d18470a94df4ed24c8108d972ba5591%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637666967356417448%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=ARwQyo%2BCjMohaNbyREofToHIj2bndL5L0HaU9cOrYq4%3D&amp;reserved=0

Which means for rxe you'd also have to pass the one for the underlying net device.

Quoted 8 lines
Quoted 3 lines

I thought of that way too. In that case, the memory region is constrained by the net device, but rxe driver copies data using CPU. To avoid the constraints, I decided to use the ib device.

Well, that is the whole problem.

We can't mix the dmabuf stuff people are doing that doesn't fill in the CPU pages in the SGL with RXE - it is simply impossible as things currently are for RXE to acess this non-struct page memory.

Quoted 8 lines

Yeah, agree that doesn't make much sense.

When you want to access the data with the CPU then why do you want to use DMA-buf in the first place?

Please keep in mind that there is work ongoing to replace the sg table with an DMA address array and so make the underlying struct page inaccessible for importers.

Quoted 7 lines

Also if you do have a dma-buf, you can just dma_buf_vmap() the buffer for cpu access. Which intentionally does not require any device. No idea why there's a dma_buf_attach involved. Now not all exporters support this, but that's fixable, and you must call dma_buf_begin/end_cpu_access for cache management if the allocation isn't cpu coherent. But it's all there, no need to apply hacks of allowing a wrong device or other fun things.

Quoted 1 line

Can rxe leave the vmap in place potentially forever?

Quoted 11 lines

Yeah, it's like perma-pinning the buffer into system memory for non-p2p dma-buf sharing. We just squint and pretend that can't be abused too badly :-) On 32bit you'll run out of vmap space rather quickly, but that's not something anyone cares about here either. We have a bunch of more sw modesetting drivers in drm which use dma_buf_vmap() like this, so it's all fine.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Thanks for your comments.

In the first place, the CMA region cannot be used for RDMA because the region has no struct page. In addition, some GPU drivers use CMA and share the region as dma-buf. As a result, RDMA cannot transfer for the region. To solve this problem, rxe dma-buf support is better I thought.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I'll consider and redesign the rxe dma-buf support using the dma_buf_vmap() instead of the dma_buf_dynamic_attach().

btw for next version please cc dri-devel. get_maintainers.pl should pick it up for these patches.

A CC list of these patches is generated by get_maintainers.pl but it didn't pick up the dri-devel. Should I add the dri-devel to the cc manually?

Hm yes, on rechecking the regex doesn't match since you're not touching any dma-buf code directly. Or not directly enough for get_maintainers.pl to pick it up.

DMA BUFFER SHARING FRAMEWORK

M:    Sumit Semwal <[email protected]>
M:    Christian König <[email protected]>
L:    [email protected]
L:    [email protected]
L:    [email protected] (moderated for non-subscribers)
S:    Maintained
T:    git git://anongit.freedesktop.org/drm/drm-misc
F:    Documentation/driver-api/dma-buf.rst
F:    drivers/dma-buf/
F:    include/linux/*fence.h
F:    include/linux/dma-buf*
F:    include/linux/dma-resv.h
K:    \bdma_(?:buf|fence|resv)\b
Quoted 7 lines

Above is the MAINTAINERS entry that's always good to cc for anything related to dma_buf/fence/resv and any of these related things. -Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Yes, the dma-buf was not directly included in my changes. However, this is related to dma-buf. So I'll add the dma-buf related ML and members to cc using
./scripts/get_maintainer.pl -f drivers/infiniband/core/umem_dmabuf.c. I think it is enough to list the email addresses.

Thank you for letting me know that.

Regards,
Shunsuke,

#16 lore raw

RFC PATCH 2/3 RDMA/rxe: Extract a mapping process into a function

In reply to Shunsuke Mie

Functionization of the process to generate the memory map. The function generates maps from scatterlists to a list of page aligned addresses.

---
 drivers/infiniband/sw/rxe/rxe_mr.c | 89 ++++++++++++++++++------------
 1 file changed, 54 insertions(+), 35 deletions(-)
drivers/infiniband/sw/rxe/rxe_mr.c +54 -35
index be4bcb420fab..8b08705ed62a 100644
@@ -103,15 +103,59 @@ void rxe_mr_init_dma(struct rxe_pd *pd, int access, struct rxe_mr *mr)
 	mr->type = RXE_MR_TYPE_DMA;
 }
 
+/* generate a rxe memory map data structure from ib_umem */
+static int rxe_mr_gen_map(struct rxe_mr *mr, struct ib_umem *umem)
+{
+	int err;
+	int num_buf;
+	struct rxe_map **map;
+	struct rxe_phys_buf *buf = NULL;
+	struct sg_page_iter sg_iter;
+	void *vaddr;
+
+	num_buf = 0;
+	map = mr->map;
+	if (mr->length > 0) {
+		buf = map[0]->buf;
+
+		for_each_sg_page(umem->sg_head.sgl, &sg_iter, umem->nmap, 0) {
+			if (num_buf >= RXE_BUF_PER_MAP) {
+				map++;
+				buf = map[0]->buf;
+				num_buf = 0;
+			}
+
+			vaddr = page_address(sg_page_iter_page(&sg_iter));
+			if (!vaddr) {
+				pr_warn("%s: Unable to get virtual address", __func__);
+				err = -ENOMEM;
+				goto err1;
+			}
+
+			buf->addr = (uintptr_t)vaddr;
+			buf->size = PAGE_SIZE;
+			num_buf++;
+			buf++;
+		}
+	}
+
+	mr->umem = umem;
+	mr->offset = ib_umem_offset(umem);
+	mr->state = RXE_MR_STATE_VALID;
+	mr->type = RXE_MR_TYPE_MR;
+
+	return 0;
+
+err1:
+	ib_umem_release(umem);
+	return err;
+}
+
 int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova,
 		     int access, struct rxe_mr *mr)
 {
-	struct rxe_map		**map;
-	struct rxe_phys_buf	*buf = NULL;
 	struct ib_umem		*umem;
-	struct sg_page_iter	sg_iter;
-	int			num_buf;
-	void			*vaddr;
+	int num_buf;
 	int err;
 	int i;
 
@@ -138,43 +182,18 @@ int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova,
 	mr->page_shift = PAGE_SHIFT;
 	mr->page_mask = PAGE_SIZE - 1;
 
-	num_buf			= 0;
-	map = mr->map;
-	if (length > 0) {
-		buf = map[0]->buf;
-
-		for_each_sg_page(umem->sg_head.sgl, &sg_iter, umem->nmap, 0) {
-			if (num_buf >= RXE_BUF_PER_MAP) {
-				map++;
-				buf = map[0]->buf;
-				num_buf = 0;
-			}
-
-			vaddr = page_address(sg_page_iter_page(&sg_iter));
-			if (!vaddr) {
-				pr_warn("%s: Unable to get virtual address\n",
-						__func__);
-				err = -ENOMEM;
-				goto err_cleanup_map;
-			}
-
-			buf->addr = (uintptr_t)vaddr;
-			buf->size = PAGE_SIZE;
-			num_buf++;
-			buf++;
+	mr->length = length;
 
-		}
+	err = rxe_mr_gen_map(mr, umem);
+	if (err) {
+		pr_warn("%s: Failed to map pages", __func__);
+		goto err_cleanup_map;
 	}
 
 	mr->ibmr.pd = &pd->ibpd;
-	mr->umem = umem;
 	mr->access = access;
-	mr->length = length;
 	mr->iova = iova;
 	mr->va = start;
-	mr->offset = ib_umem_offset(umem);
-	mr->state = RXE_MR_STATE_VALID;
-	mr->type = RXE_MR_TYPE_MR;
 
 	return 0;
 
Signature · 1 line
2.17.1
#17 lore raw

RFC PATCH 3/3 RDMA/rxe: Support dma-buf as memory region

In reply to Shunsuke Mie

Implement a ib device operation ‘reg_user_mr_dmabuf’. Import dma-buf using the IB core API and map the memory area linked the dma-buf.

---
 drivers/infiniband/sw/rxe/rxe_loc.h   |   3 +
 drivers/infiniband/sw/rxe/rxe_mr.c    | 101 ++++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_verbs.c |  36 +++++++++
 3 files changed, 140 insertions(+)
drivers/infiniband/sw/rxe/rxe_loc.h +3 -0
index 1ddb20855dee..206d9d5f8bbf 100644
@@ -75,6 +75,9 @@ u8 rxe_get_next_key(u32 last_key);
 void rxe_mr_init_dma(struct rxe_pd *pd, int access, struct rxe_mr *mr);
 int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova,
 		     int access, struct rxe_mr *mr);
+int rxe_mr_dmabuf_init_user(struct rxe_pd *pd, int fd, u64 start, u64 length,
+			    u64 iova, int access, struct rxe_mr *mr);
+
 int rxe_mr_init_fast(struct rxe_pd *pd, int max_pages, struct rxe_mr *mr);
 int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length,
 		enum rxe_mr_copy_dir dir, u32 *crcp);
drivers/infiniband/sw/rxe/rxe_mr.c +101 -0

113 lines not on this page · raw message

drivers/infiniband/sw/rxe/rxe_verbs.c +36 -0
index c223959ac174..4a38c20730b3 100644
@@ -959,6 +959,39 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd,
 	return ERR_PTR(err);
 }
 
+static struct ib_mr *rxe_reg_user_mr_dmabuf(struct ib_pd *ibpd, u64 start,
+					    u64 length, u64 iova, int fd,
+					    int access, struct ib_udata *udata)
+{
+	int err;
+	struct rxe_dev *rxe = to_rdev(ibpd->device);
+	struct rxe_pd *pd = to_rpd(ibpd);
+	struct rxe_mr *mr;
+
+	mr = rxe_alloc(&rxe->mr_pool);
+	if (!mr) {
+		err = -ENOMEM;
+		goto err1;
+	}
+
+	rxe_add_index(mr);
+
+	rxe_add_ref(pd);
+
+	err = rxe_mr_dmabuf_init_user(pd, fd, start, length, iova, access, mr);
+	if (err)
+		goto err2;
+
+	return &mr->ibmr;
+
+err2:
+	rxe_drop_ref(pd);
+	rxe_drop_index(mr);
+	rxe_drop_ref(mr);
+err1:
+	return ERR_PTR(err);
+}
+
 static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type,
 				  u32 max_num_sg)
 {
@@ -1139,6 +1172,7 @@ static const struct ib_device_ops rxe_dev_ops = {
 	.query_qp = rxe_query_qp,
 	.query_srq = rxe_query_srq,
 	.reg_user_mr = rxe_reg_user_mr,
+	.reg_user_mr_dmabuf = rxe_reg_user_mr_dmabuf,
 	.req_notify_cq = rxe_req_notify_cq,
 	.resize_cq = rxe_resize_cq,
 
@@ -1181,6 +1215,8 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name)
 	}
 	rxe->tfm = tfm;
 
+	dma_coerce_mask_and_coherent(&dev->dev, DMA_BIT_MASK(64));
+
 	err = ib_register_device(dev, ibdev_name, NULL);
 	if (err)
 		pr_warn("%s failed with error %d\n", __func__, err);
Signature · 1 line
2.17.1
#18 lore raw

Re: [RFC PATCH 3/3] RDMA/rxe: Support dma-buf as memory region

[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi Shunsuke,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on v5.14]
[cannot apply to rdma/for-next next-20210908]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Shunsuke-Mie/RDMA-rxe-Add-dma-buf-support/20210908-142001
base:    7d2a07b769330c34b4deabeed939325c77a7ec2f
config: ia64-allmodconfig (attached as .config)
compiler: ia64-linux-gcc (GCC) 11.2.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/80fcf7f4bc573f6f833baeb7e390e44a79ddd84f
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Shunsuke-Mie/RDMA-rxe-Add-dma-buf-support/20210908-142001
git checkout 80fcf7f4bc573f6f833baeb7e390e44a79ddd84f
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=ia64 SHELL=/bin/bash drivers/infiniband/sw/rxe/

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

drivers/infiniband/sw/rxe/rxe_mr.c: In function 'rxe_mr_dmabuf_init_user':

>> drivers/infiniband/sw/rxe/rxe_mr.c:304:14: error: 'i' undeclared (first use in this function)
     304 |         for (i = 0; i < mr->num_map; i++)
         |              ^

drivers/infiniband/sw/rxe/rxe_mr.c:304:14: note: each undeclared identifier is reported only once for each function it appears in

vim +/i +304 drivers/infiniband/sw/rxe/rxe_mr.c

254
255 /* initialize a umem and map all the areas of dma-buf. */
256 int rxe_mr_dmabuf_init_user(struct rxe_pd *pd, int fd, u64 start, u64 length,
257 u64 iova, int access, struct rxe_mr *mr)
258 {
259 struct ib_umem_dmabuf *umem_dmabuf;
260 int num_buf;
261 int err;
262
263 umem_dmabuf = ib_umem_dmabuf_get(pd->ibpd.device, start, length, fd,
264 access, &rxe_ib_dmabuf_attach_ops);
265 if (IS_ERR(umem_dmabuf)) {
266 err = PTR_ERR(umem_dmabuf);
267 pr_err("%s: failed to get umem_dmabuf (%d)", __func__, err);
268 goto err1;
269 }
270
271 umem_dmabuf->private = mr;
272
273 mr->umem = &umem_dmabuf->umem;
274 mr->umem->iova = iova;
275 num_buf = ib_umem_num_pages(mr->umem);
276
277 rxe_mr_init(access, mr);
278
279 err = rxe_mr_alloc(mr, num_buf);
280 if (err)
281 goto err1;
282
283 mr->page_shift = PAGE_SHIFT;
284 mr->page_mask = PAGE_SIZE - 1;
285
286 mr->ibmr.pd = &pd->ibpd;
287 mr->access = access;
288 mr->length = length;
289 mr->iova = iova;
290 mr->va = start;
291 mr->offset = ib_umem_offset(mr->umem);
292 mr->state = RXE_MR_STATE_VALID;
293 mr->type = RXE_MR_TYPE_MR;
294
295 err = rxe_map_dmabuf_mr(mr);
296 if (err) {
297 pr_err("%s: failed to map the dma-buf region", __func__);
298 goto err2;
299 }
300
301 return 0;
302
303 err2:

 > 304		for (i = 0; i < mr->num_map; i++)

305 kfree(mr->map[i]);
306 kfree(mr->map);
307 err1:
308 return err;
309 }
310

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#19 lore raw

Re: [RFC PATCH 3/3] RDMA/rxe: Support dma-buf as memory region

In reply to Shunsuke Mie
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi Shunsuke,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on v5.14]
[cannot apply to rdma/for-next next-20210908]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Shunsuke-Mie/RDMA-rxe-Add-dma-buf-support/20210908-142001
base:    7d2a07b769330c34b4deabeed939325c77a7ec2f
config: x86_64-randconfig-a016-20210908 (attached as .config)
compiler: clang version 14.0.0 (https://github.com/llvm/llvm-project 9c476172b93367d2cb88d7d3f4b1b5b456fa6020)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/80fcf7f4bc573f6f833baeb7e390e44a79ddd84f
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Shunsuke-Mie/RDMA-rxe-Add-dma-buf-support/20210908-142001
git checkout 80fcf7f4bc573f6f833baeb7e390e44a79ddd84f
# save the attached .config to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross O=build_dir ARCH=x86_64 SHELL=/bin/bash drivers/infiniband/

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

>> drivers/infiniband/sw/rxe/rxe_mr.c:304:7: error: use of undeclared identifier 'i'
           for (i = 0; i < mr->num_map; i++)
                ^
   drivers/infiniband/sw/rxe/rxe_mr.c:304:14: error: use of undeclared identifier 'i'
           for (i = 0; i < mr->num_map; i++)
                       ^
   drivers/infiniband/sw/rxe/rxe_mr.c:304:31: error: use of undeclared identifier 'i'
           for (i = 0; i < mr->num_map; i++)
                                        ^
   drivers/infiniband/sw/rxe/rxe_mr.c:305:17: error: use of undeclared identifier 'i'
                   kfree(mr->map[i]);
                                 ^

4 errors generated.

vim +/i +304 drivers/infiniband/sw/rxe/rxe_mr.c

254
255 /* initialize a umem and map all the areas of dma-buf. */
256 int rxe_mr_dmabuf_init_user(struct rxe_pd *pd, int fd, u64 start, u64 length,
257 u64 iova, int access, struct rxe_mr *mr)
258 {
259 struct ib_umem_dmabuf *umem_dmabuf;
260 int num_buf;
261 int err;
262
263 umem_dmabuf = ib_umem_dmabuf_get(pd->ibpd.device, start, length, fd,
264 access, &rxe_ib_dmabuf_attach_ops);
265 if (IS_ERR(umem_dmabuf)) {
266 err = PTR_ERR(umem_dmabuf);
267 pr_err("%s: failed to get umem_dmabuf (%d)", __func__, err);
268 goto err1;
269 }
270
271 umem_dmabuf->private = mr;
272
273 mr->umem = &umem_dmabuf->umem;
274 mr->umem->iova = iova;
275 num_buf = ib_umem_num_pages(mr->umem);
276
277 rxe_mr_init(access, mr);
278
279 err = rxe_mr_alloc(mr, num_buf);
280 if (err)
281 goto err1;
282
283 mr->page_shift = PAGE_SHIFT;
284 mr->page_mask = PAGE_SIZE - 1;
285
286 mr->ibmr.pd = &pd->ibpd;
287 mr->access = access;
288 mr->length = length;
289 mr->iova = iova;
290 mr->va = start;
291 mr->offset = ib_umem_offset(mr->umem);
292 mr->state = RXE_MR_STATE_VALID;
293 mr->type = RXE_MR_TYPE_MR;
294
295 err = rxe_map_dmabuf_mr(mr);
296 if (err) {
297 pr_err("%s: failed to map the dma-buf region", __func__);
298 goto err2;
299 }
300
301 return 0;
302
303 err2:

 > 304		for (i = 0; i < mr->num_map; i++)

305 kfree(mr->map[i]);
306 kfree(mr->map);
307 err1:
308 return err;
309 }
310

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

maintainer
#20 lore raw
In reply to Shunsuke Mie
Quoted 2 lines · Shunsuke Mie

This patch series add a support for rxe driver.

After applying the patches, please run rdma-core tests with the patched kernel. Then fix all the problems in rdma-core.

Thanks
Zhu Yanjun

Quoted 34 lines

A dma-buf based memory registering has beed introduced to use the memory region that lack of associated page structures (e.g. device memory and CMA managed memory) [1]. However, to use the dma-buf based memory, each rdma device drivers require add some implementation. The rxe driver has not support yet.

[1] https://www.spinics.net/lists/linux-rdma/msg98592.html

To enable to use the memories in rxe rdma device, add some changes and implementation in this patch series.

This series consists of three patches. The first patch changes the IB core to support for rdma drivers that have not real dma device. The second patch extracts a memory mapping process of rxe as a common function to use a dma-buf support. The third patch adds the dma-buf support to rxe driver.

Related user space RDMA library changes are provided as a separate patch.

Shunsuke Mie (3):
RDMA/umem: Change for rdma devices has not dma device RDMA/rxe: Extract a mapping process into a function RDMA/rxe: Support dma-buf as memory region

 drivers/infiniband/core/umem_dmabuf.c |   2 +-
 drivers/infiniband/sw/rxe/rxe_loc.h   |   3 +
 drivers/infiniband/sw/rxe/rxe_mr.c    | 186 +++++++++++++++++++++-----
 drivers/infiniband/sw/rxe/rxe_verbs.c |  36 +++++
 4 files changed, 193 insertions(+), 34 deletions(-)

--
2.17.1

#21 lore raw

2021年9月9日(木) 14:45 Zhu Yanjun <[email protected]>:

Quoted 2 lines

After applying the patches, please run rdma-core tests with the patched kernel. Then fix all the problems in rdma-core.

I understand. I'd like to do the tests and fix it before posting the next patches

Regards,
Shunsuke