Skip to contentSkip to conversation

Make use of non-dynamic dmabuf in RDMA

RFC In review Gal Pressman 10 participants 33 messages Aug 18, 2021 – Sep 2, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 Make use of non-dynamic dmabuf in RDMA

Hey all,

Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5).

While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation.

The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs.

This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter.

Thanks

---
 drivers/dma-buf/dma-buf.c             | 5 +++--
 drivers/infiniband/core/umem_dmabuf.c | 2 +-
 include/linux/dma-buf.h               | 2 +-
 3 files changed, 5 insertions(+), 4 deletions(-)
drivers/dma-buf/dma-buf.c +3 -2
index 511fe0d217a0..e3faad8f492c 100644
@@ -727,7 +727,8 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev,
 	if (WARN_ON(!dmabuf || !dev))
 		return ERR_PTR(-EINVAL);
 
-	if (WARN_ON(importer_ops && !importer_ops->move_notify))
+	if (WARN_ON(importer_ops && !importer_ops->move_notify &&
+		    dma_buf_is_dynamic(attach->dmabuf)))
 		return ERR_PTR(-EINVAL);
 
 	attach = kzalloc(sizeof(*attach), GFP_KERNEL);
@@ -1048,7 +1049,7 @@ void dma_buf_move_notify(struct dma_buf *dmabuf)
 	dma_resv_assert_held(dmabuf->resv);
 
 	list_for_each_entry(attach, &dmabuf->attachments, node)
-		if (attach->importer_ops)
+		if (attach->importer_ops && attach->importer_ops->move_notify)
 			attach->importer_ops->move_notify(attach);
 }
 EXPORT_SYMBOL_GPL(dma_buf_move_notify);
drivers/infiniband/core/umem_dmabuf.c +1 -1
index c6e875619fac..c502ae828bd3 100644
@@ -118,7 +118,7 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,
 	if (check_add_overflow(offset, (unsigned long)size, &end))
 		return ret;
 
-	if (unlikely(!ops || !ops->move_notify))
+	if (unlikely(!ops))
 		return ret;
 
 	dmabuf = dma_buf_get(fd);
include/linux/dma-buf.h +1 -1
index efdc56b9d95f..4b2e99012cb1 100644
@@ -473,7 +473,7 @@ static inline bool dma_buf_is_dynamic(struct dma_buf *dmabuf)
 static inline bool
 dma_buf_attachment_is_dynamic(struct dma_buf_attachment *attach)
 {
-	return !!attach->importer_ops;
+	return !!attach->importer_ops->move_notify;
 }
 
 struct dma_buf_attachment *dma_buf_attach(struct dma_buf *dmabuf,
Signature · 1 line
2.32.0
maintainer
#1 lore raw

Am 18.08.21 um 09:43 schrieb Gal Pressman:

Quoted 10 lines

Hey all,

Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5).

While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation.

Quoted 7 lines

The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs.

This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter.

Well NAK to the general approach, this can be solved much easier.

If you can't support dynamic moves while using the buffer then just pin all buffers during import/export.

This avoids the move notification and the framework/exporter can still correctly account for pinned buffers.

But please note that at least amdgpu never uses P2P support for pinned buffers since we want to avoid that unmoveable buffers clutter video memory and create conflicts with V4L and scanout.

If you don't have such concerns in habanalabs then you can implement the pinning there while keeping P2P still enabled.

Regards,
Christian.

Quoted 59 lines

Thanks

---
  drivers/dma-buf/dma-buf.c             | 5 +++--
  drivers/infiniband/core/umem_dmabuf.c | 2 +-
  include/linux/dma-buf.h               | 2 +-
  3 files changed, 5 insertions(+), 4 deletions(-)
drivers/dma-buf/dma-buf.c +3 -2
index 511fe0d217a0..e3faad8f492c 100644
@@ -727,7 +727,8 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev,
  	if (WARN_ON(!dmabuf || !dev))
  		return ERR_PTR(-EINVAL);
 
-	if (WARN_ON(importer_ops && !importer_ops->move_notify))
+	if (WARN_ON(importer_ops && !importer_ops->move_notify &&
+		    dma_buf_is_dynamic(attach->dmabuf)))
  		return ERR_PTR(-EINVAL);
 
  	attach = kzalloc(sizeof(*attach), GFP_KERNEL);
@@ -1048,7 +1049,7 @@ void dma_buf_move_notify(struct dma_buf *dmabuf)
  	dma_resv_assert_held(dmabuf->resv);
 
  	list_for_each_entry(attach, &dmabuf->attachments, node)
-		if (attach->importer_ops)
+		if (attach->importer_ops && attach->importer_ops->move_notify)
  			attach->importer_ops->move_notify(attach);
  }
  EXPORT_SYMBOL_GPL(dma_buf_move_notify);
drivers/infiniband/core/umem_dmabuf.c +1 -1
index c6e875619fac..c502ae828bd3 100644
@@ -118,7 +118,7 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,
  	if (check_add_overflow(offset, (unsigned long)size, &end))
  		return ret;
 
-	if (unlikely(!ops || !ops->move_notify))
+	if (unlikely(!ops))
  		return ret;
 
  	dmabuf = dma_buf_get(fd);
include/linux/dma-buf.h +1 -1
index efdc56b9d95f..4b2e99012cb1 100644
@@ -473,7 +473,7 @@ static inline bool dma_buf_is_dynamic(struct dma_buf *dmabuf)
  static inline bool
  dma_buf_attachment_is_dynamic(struct dma_buf_attachment *attach)
  {
-	return !!attach->importer_ops;
+	return !!attach->importer_ops->move_notify;
  }
 
  struct dma_buf_attachment *dma_buf_attach(struct dma_buf *dmabuf,
#2 lore raw
Quoted 27 lines · Christian König

Am 18.08.21 um 09:43 schrieb Gal Pressman:

Quoted 10 lines

Hey all,

Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5).

While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation.

Quoted 7 lines

The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs.

This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter.

Well NAK to the general approach, this can be solved much easier.

If you can't support dynamic moves while using the buffer then just pin all buffers during import/export.

This avoids the move notification and the framework/exporter can still correctly account for pinned buffers.

Quoted 6 lines

But please note that at least amdgpu never uses P2P support for pinned buffers since we want to avoid that unmoveable buffers clutter video memory and create conflicts with V4L and scanout.

If you don't have such concerns in habanalabs then you can implement the pinning there while keeping P2P still enabled.

Thanks Christian!

Are you suggesting to pass an empty move_notify callback instead of passing NULL? Also, doesn't the pin operation move the memory from the device to host memory?

#3 lore raw
In reply to Gal Pressman
Quoted 14 lines · Gal Pressman

Hey all,

Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5).

While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation.

The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs.

Quoted 8 lines

This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter.

Thanks

Given that habanalabs dma-buf support is very firmly in limbo (at least it's not yet in linux-next or anywhere else) I think you want to solve that problem first before we tackle the additional issue of making p2p work without dynamic dma-buf. Without that it just doesn't make a lot of sense really to talk about solutions here. -Daniel

Quoted 58 lines
---
 drivers/dma-buf/dma-buf.c             | 5 +++--
 drivers/infiniband/core/umem_dmabuf.c | 2 +-
 include/linux/dma-buf.h               | 2 +-
 3 files changed, 5 insertions(+), 4 deletions(-)
drivers/dma-buf/dma-buf.c +3 -2
index 511fe0d217a0..e3faad8f492c 100644
@@ -727,7 +727,8 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev,
        if (WARN_ON(!dmabuf || !dev))
                return ERR_PTR(-EINVAL);
 
-       if (WARN_ON(importer_ops && !importer_ops->move_notify))
+       if (WARN_ON(importer_ops && !importer_ops->move_notify &&
+                   dma_buf_is_dynamic(attach->dmabuf)))
                return ERR_PTR(-EINVAL);
 
        attach = kzalloc(sizeof(*attach), GFP_KERNEL);
@@ -1048,7 +1049,7 @@ void dma_buf_move_notify(struct dma_buf *dmabuf)
        dma_resv_assert_held(dmabuf->resv);
 
        list_for_each_entry(attach, &dmabuf->attachments, node)
-               if (attach->importer_ops)
+               if (attach->importer_ops && attach->importer_ops->move_notify)
                        attach->importer_ops->move_notify(attach);
 }
 EXPORT_SYMBOL_GPL(dma_buf_move_notify);
drivers/infiniband/core/umem_dmabuf.c +1 -1
index c6e875619fac..c502ae828bd3 100644
@@ -118,7 +118,7 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device,
        if (check_add_overflow(offset, (unsigned long)size, &end))
                return ret;
 
-       if (unlikely(!ops || !ops->move_notify))
+       if (unlikely(!ops))
                return ret;
 
        dmabuf = dma_buf_get(fd);
include/linux/dma-buf.h +1 -2
index efdc56b9d95f..4b2e99012cb1 100644
@@ -473,7 +473,7 @@ static inline bool dma_buf_is_dynamic(struct dma_buf *dmabuf)
 static inline bool
 dma_buf_attachment_is_dynamic(struct dma_buf_attachment *attach)
 {
-       return !!attach->importer_ops;
+       return !!attach->importer_ops->move_notify;
 }
 
 struct dma_buf_attachment *dma_buf_attach(struct dma_buf *dmabuf,
--

2.32.0

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#4 lore raw
Quoted 22 lines · Daniel Vetter
Quoted 21 lines · Gal Pressman

Hey all,

Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5).

While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation.

The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs.

This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter.

Thanks

Quoted 7 lines
Quoted 1 line

Given that habanalabs dma-buf support is very firmly in limbo (at least it's not yet in linux-next or anywhere else) I think you want to solve that problem first before we tackle the additional issue of making p2p work without dynamic dma-buf. Without that it just doesn't make a lot of sense really to talk about solutions here.

I have been thinking about adding a dmabuf exporter to VFIO, for basically the same reason habana labs wants to do it.

In that situation we'd want to see an approach similar to this as well to have a broad usability.

The GPU drivers also want this for certain sophisticated scenarios with RDMA, the intree drivers just haven't quite got there yet.

So, I think it is worthwhile to start thinking about this regardless of habana labs.

Jason

#5 lore raw
Quoted 34 lines · Jason Gunthorpe
Quoted 22 lines · Daniel Vetter
Quoted 21 lines · Gal Pressman

Hey all,

Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5).

While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation.

The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs.

This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter.

Thanks

Quoted 7 lines
Quoted 1 line

Given that habanalabs dma-buf support is very firmly in limbo (at least it's not yet in linux-next or anywhere else) I think you want to solve that problem first before we tackle the additional issue of making p2p work without dynamic dma-buf. Without that it just doesn't make a lot of sense really to talk about solutions here.

I have been thinking about adding a dmabuf exporter to VFIO, for basically the same reason habana labs wants to do it.

Quoted 8 lines

In that situation we'd want to see an approach similar to this as well to have a broad usability.

The GPU drivers also want this for certain sophisticated scenarios with RDMA, the intree drivers just haven't quite got there yet.

So, I think it is worthwhile to start thinking about this regardless of habana labs.

Oh sure, I've been having these for a while. I think there's two options:
- some kind of soft-pin, where the contract is that we only revoke when absolutely necessary, and it's expected to be catastrophic on the importer's side. The use-case would be single user that fully controls all accelerator local memory, and so kernel driver evicting stuff. I havent implemented it, but the idea is that essentially in eviction we check whom we're evicting for (by checking owners of buffers maybe, atm those are not tracked in generic code but not that hard to add), and if it's the same userspace owner we don't ever pick these buffers as victims for eviction, preferreing -ENOMEM/-ENOSPC. If a new user comes around then we'd still throw these out to avoid abuse, and it would be up to sysadmins to make sure this doesn't happen untimely, maybe with the next thing.

- cgroups for (pinned) buffers. Mostly because cgroups for local memory is somewhere on the plans anyway, but that one could also take forever since there's questions about overlap with memcg and things like that, plus thus far everyone who cares made and incompatible proposal about how it should be done :-/

A variant of the first one would be device-level revoke, which is a concept we already have in drm for the modesetting side and also for like 20 year old gpu drivers. We could brush that off and close some of the gaps (a student is fixing the locking right now, the thing left to do is mmap revoke), and I think that model of exclusive device ownership with the option to revoke fits pretty well for at least some of the accelerators floating around. In that case importers would never get a move_notify (maybe we should call this revoke_notify to make it clear it's a bit different) callback, except when the entire thing has been yanked. I think that would fit pretty well for VFIO, and I think we should be able to make it work for rdma too as some kind of auto-deregister. The locking might be fun with both of these since I expect some inversions compared to the register path, we'll have to figure these out.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#6 lore raw
Quoted 41 lines · Daniel Vetter
Quoted 40 lines · Jason Gunthorpe
Quoted 22 lines · Daniel Vetter
Quoted 21 lines · Gal Pressman

Hey all,

Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5).

While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation.

The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs.

This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter.

Thanks

Quoted 7 lines
Quoted 1 line

Given that habanalabs dma-buf support is very firmly in limbo (at least it's not yet in linux-next or anywhere else) I think you want to solve that problem first before we tackle the additional issue of making p2p work without dynamic dma-buf. Without that it just doesn't make a lot of sense really to talk about solutions here.

I have been thinking about adding a dmabuf exporter to VFIO, for basically the same reason habana labs wants to do it.

In that situation we'd want to see an approach similar to this as well to have a broad usability.

The GPU drivers also want this for certain sophisticated scenarios with RDMA, the intree drivers just haven't quite got there yet.

Quoted 7 lines
Quoted 2 lines

So, I think it is worthwhile to start thinking about this regardless of habana labs.

Oh sure, I've been having these for a while. I think there's two options:
- some kind of soft-pin, where the contract is that we only revoke when absolutely necessary, and it's expected to be catastrophic on the importer's side.

Honestly, I'm not very keen on this. We don't really have HW support in several RDMA scenarios for even catastrophic unpin.

Gal, can EFA even do this for a MR? You basically have to resize the rkey/lkey to zero length (or invalidate it like a FMR) under the catstrophic revoke. The rkey/lkey cannot just be destroyed as that opens a security problem with rkey/lkey re-use.

I think I saw EFA's current out of tree implementations had this bug.

Quoted 10 lines

to do is mmap revoke), and I think that model of exclusive device ownership with the option to revoke fits pretty well for at least some of the accelerators floating around. In that case importers would never get a move_notify (maybe we should call this revoke_notify to make it clear it's a bit different) callback, except when the entire thing has been yanked. I think that would fit pretty well for VFIO, and I think we should be able to make it work for rdma too as some kind of auto-deregister. The locking might be fun with both of these since I expect some inversions compared to the register path, we'll have to figure these out.

It fits semantically nicely, VFIO also has a revoke semantic for BAR mappings.

The challenge is the RDMA side which doesn't have a 'dma disabled error state' for objects as part of the spec.

Some HW, like mlx5, can implement this for MR objects (see revoke_mr), but I don't know if anything else can, and even mlx5 currently can't do a revoke for any other object type.

I don't know how useful it would be, need to check on some of the use cases.

The locking is tricky as we have to issue a device command, but that device command cannot run concurrently with destruction or the tail part of creation.

Jason

#7 lore raw
Quoted 50 lines · Jason Gunthorpe
Quoted 41 lines · Daniel Vetter
Quoted 40 lines · Jason Gunthorpe
Quoted 22 lines · Daniel Vetter
Quoted 21 lines · Gal Pressman

Hey all,

Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5).

While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation.

The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs.

This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter.

Thanks

Quoted 7 lines
Quoted 1 line

Given that habanalabs dma-buf support is very firmly in limbo (at least it's not yet in linux-next or anywhere else) I think you want to solve that problem first before we tackle the additional issue of making p2p work without dynamic dma-buf. Without that it just doesn't make a lot of sense really to talk about solutions here.

I have been thinking about adding a dmabuf exporter to VFIO, for basically the same reason habana labs wants to do it.

In that situation we'd want to see an approach similar to this as well to have a broad usability.

The GPU drivers also want this for certain sophisticated scenarios with RDMA, the intree drivers just haven't quite got there yet.

Quoted 7 lines
Quoted 2 lines

So, I think it is worthwhile to start thinking about this regardless of habana labs.

Oh sure, I've been having these for a while. I think there's two options:
- some kind of soft-pin, where the contract is that we only revoke when absolutely necessary, and it's expected to be catastrophic on the importer's side.

Quoted 7 lines

Honestly, I'm not very keen on this. We don't really have HW support in several RDMA scenarios for even catastrophic unpin.

Gal, can EFA even do this for a MR? You basically have to resize the rkey/lkey to zero length (or invalidate it like a FMR) under the catstrophic revoke. The rkey/lkey cannot just be destroyed as that opens a security problem with rkey/lkey re-use.

I had some discussions with our hardware guys about such functionality in the past, I think it should be doable (not necessarily the same way that FMR does it).

Though it would've been nicer if we could agree on a solution that could work for more than 1-2 RDMA devices, using the existing tools the RDMA subsystem has. That's why I tried to approach this by denying such attachments for non-ODP importers instead of exposing a "limited" dynamic importer.

#8 lore raw
Quoted 2 lines · Gal Pressman

Though it would've been nicer if we could agree on a solution that could work for more than 1-2 RDMA devices, using the existing tools the RDMA subsystem has.

I don't think it can really be done, revoke is necessary, and isn't a primitive we have today.

Revoke is sort of like rereg MR, but with a guaranteed no-change to the lkey/rkey

Then there is the locking complexity of linking the mr creation and destruction to the lifecycle of the pages, which is messy and maybe not general. For instance mlx5 would call its revoke_mr, disconnect the dmabuf then destroy the mkey - but this is only safe because mlx5 HW can handle concurrent revokes.

Quoted 2 lines

That's why I tried to approach this by denying such attachments for non-ODP importers instead of exposing a "limited" dynamic importer.

That is fine if there is no revoke - once revoke exists we must have driver and HW support.

Jason

#9 lore raw
Quoted 8 lines · Jason Gunthorpe
Quoted 2 lines · Gal Pressman

Though it would've been nicer if we could agree on a solution that could work for more than 1-2 RDMA devices, using the existing tools the RDMA subsystem has.

I don't think it can really be done, revoke is necessary, and isn't a primitive we have today.

Quoted 8 lines

Revoke is sort of like rereg MR, but with a guaranteed no-change to the lkey/rkey

Then there is the locking complexity of linking the mr creation and destruction to the lifecycle of the pages, which is messy and maybe not general. For instance mlx5 would call its revoke_mr, disconnect the dmabuf then destroy the mkey - but this is only safe because mlx5 HW can handle concurrent revokes.

Thanks, that makes sense.

Quoted 5 lines
Quoted 2 lines

That's why I tried to approach this by denying such attachments for non-ODP importers instead of exposing a "limited" dynamic importer.

That is fine if there is no revoke - once revoke exists we must have driver and HW support.

Agree.
IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Which changes do we need to make the third one work?

maintainer
#10 lore raw

Am 21.08.21 um 11:16 schrieb Gal Pressman:

Quoted 22 lines · Jason Gunthorpe
Quoted 7 lines · Gal Pressman
Quoted 3 lines

Though it would've been nicer if we could agree on a solution that could work for more than 1-2 RDMA devices, using the existing tools the RDMA subsystem has.

I don't think it can really be done, revoke is necessary, and isn't a primitive we have today.

Quoted 8 lines

Revoke is sort of like rereg MR, but with a guaranteed no-change to the lkey/rkey

Then there is the locking complexity of linking the mr creation and destruction to the lifecycle of the pages, which is messy and maybe not general. For instance mlx5 would call its revoke_mr, disconnect the dmabuf then destroy the mkey - but this is only safe because mlx5 HW can handle concurrent revokes.

Thanks, that makes sense.

Quoted 4 lines
Quoted 2 lines

That's why I tried to approach this by denying such attachments for non-ODP importers instead of exposing a "limited" dynamic importer.

That is fine if there is no revoke - once revoke exists we must have driver and HW support.

Quoted 7 lines

Agree.
IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a buffer (e.g. before you create an attachment) and the dma_buf_unpin() function after you are done with the DMA-buf.

Regards,
Christian.

#11 lore raw
Quoted 30 lines · Christian König

Am 21.08.21 um 11:16 schrieb Gal Pressman:

Quoted 29 lines · Jason Gunthorpe
Quoted 8 lines · Gal Pressman
Quoted 4 lines

Though it would've been nicer if we could agree on a solution that could work for more than 1-2 RDMA devices, using the existing tools the RDMA subsystem has.

I don't think it can really be done, revoke is necessary, and isn't a primitive we have today.

Quoted 8 lines

Revoke is sort of like rereg MR, but with a guaranteed no-change to the lkey/rkey

Then there is the locking complexity of linking the mr creation and destruction to the lifecycle of the pages, which is messy and maybe not general. For instance mlx5 would call its revoke_mr, disconnect the dmabuf then destroy the mkey - but this is only safe because mlx5 HW can handle concurrent revokes.

Thanks, that makes sense.

Quoted 4 lines
Quoted 2 lines

That's why I tried to approach this by denying such attachments for non-ODP importers instead of exposing a "limited" dynamic importer.

That is fine if there is no revoke - once revoke exists we must have driver and HW support.

Agree.
IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Quoted 7 lines
Quoted 1 line

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a buffer (e.g. before you create an attachment) and the dma_buf_unpin() function after you are done with the DMA-buf.

I replied to your previous mail, but I'll ask again. Doesn't the pin operation migrate the memory to host memory?

maintainer
#12 lore raw

Am 24.08.21 um 11:06 schrieb Gal Pressman:

Quoted 31 lines · Christian König
Quoted 30 lines

Am 21.08.21 um 11:16 schrieb Gal Pressman:

Quoted 29 lines · Jason Gunthorpe
Quoted 8 lines · Gal Pressman
Quoted 4 lines

Though it would've been nicer if we could agree on a solution that could work for more than 1-2 RDMA devices, using the existing tools the RDMA subsystem has.

I don't think it can really be done, revoke is necessary, and isn't a primitive we have today.

Quoted 8 lines

Revoke is sort of like rereg MR, but with a guaranteed no-change to the lkey/rkey

Then there is the locking complexity of linking the mr creation and destruction to the lifecycle of the pages, which is messy and maybe not general. For instance mlx5 would call its revoke_mr, disconnect the dmabuf then destroy the mkey - but this is only safe because mlx5 HW can handle concurrent revokes.

Thanks, that makes sense.

Quoted 4 lines
Quoted 2 lines

That's why I tried to approach this by denying such attachments for non-ODP importers instead of exposing a "limited" dynamic importer.

That is fine if there is no revoke - once revoke exists we must have driver and HW support.

Agree.
IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a buffer (e.g. before you create an attachment) and the dma_buf_unpin() function after you are done with the DMA-buf.

I replied to your previous mail, but I'll ask again. Doesn't the pin operation migrate the memory to host memory?

Sorry missed your previous reply.

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

This is intentional since we don't want any P2P to video memory with pinned objects and want to avoid to run into a situation where one device is doing P2P to video memory while another device needs the DMA-buf in host memory.

You can still do P2P with pinned object, it's just up to the exporting driver if it is allowed or not.

The other option is what Daniel suggested that we have some kind of revoke. This is essentially what our KFD is doing as well when doing interop with 3D GFX, but from Jasons responses I have a bit of doubt that this will actually work on the hardware level for RDMA.

Regards,
Christian.

#13 lore raw
Quoted 5 lines · Christian König

Am 24.08.21 um 11:06 schrieb Gal Pressman:

Quoted 4 lines · Christian König
Quoted 3 lines

Am 21.08.21 um 11:16 schrieb Gal Pressman:

Quoted 2 lines · Jason Gunthorpe
Quoted 1 line · Gal Pressman

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a buffer (e.g. before you create an attachment) and the dma_buf_unpin() function after you are done with the DMA-buf.

Quoted 6 lines
Quoted 2 lines

I replied to your previous mail, but I'll ask again. Doesn't the pin operation migrate the memory to host memory?

Sorry missed your previous reply.

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
Quoted 9 lines

This is intentional since we don't want any P2P to video memory with pinned objects and want to avoid to run into a situation where one device is doing P2P to video memory while another device needs the DMA-buf in host memory.

You can still do P2P with pinned object, it's just up to the exporting driver if it is allowed or not.

The other option is what Daniel suggested that we have some kind of revoke. This is essentially what our KFD is doing as well when doing interop with 3D GFX, but from Jasons responses I have a bit of doubt that this will actually work on the hardware level for RDMA.

Regards,
Christian.

#14 lore raw
Quoted 36 lines · John Hubbard
Quoted 5 lines · Christian König

Am 24.08.21 um 11:06 schrieb Gal Pressman:

Quoted 4 lines · Christian König
Quoted 3 lines

Am 21.08.21 um 11:16 schrieb Gal Pressman:

Quoted 2 lines · Jason Gunthorpe
Quoted 1 line · Gal Pressman

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a buffer (e.g. before you create an attachment) and the dma_buf_unpin() function after you are done with the DMA-buf.

Quoted 8 lines
Quoted 2 lines

I replied to your previous mail, but I'll ask again. Doesn't the pin operation migrate the memory to host memory?

Sorry missed your previous reply.

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

Quoted 4 lines

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

Jason

#15 lore raw

On 8/24/21 10:32 AM, Jason Gunthorpe wrote:
...

Quoted 18 lines
Quoted 13 lines
Quoted 3 lines

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

Quoted 4 lines

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

Quoted 7 lines

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

OK. I just want to avoid creating any API-level assumptions that dma_buf_pin() necessarily implies or requires migrating to host memory.

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#16 lore raw
Quoted 29 lines · John Hubbard

On 8/24/21 10:32 AM, Jason Gunthorpe wrote:
...

Quoted 18 lines
Quoted 13 lines
Quoted 3 lines

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

Quoted 4 lines

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

Quoted 7 lines

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

Quoted 2 lines

OK. I just want to avoid creating any API-level assumptions that dma_buf_pin() necessarily implies or requires migrating to host memory.

I'm not sure we should be allowing dma_buf_pin at all on non-migratable memory, what's to stop someone just pinning all the VRAM and making the GPU unuseable?

I understand not considering more than a single user in these situations is enterprise thinking, but I do worry about pinning is always fine type of thinking when things are shared or multi-user.

My impression from this is we've designed hardware that didn't consider the problem, and now to let us use that hardware in horrible ways we should just allow it to pin all the things.

Dave.

#17 lore raw
Quoted 30 lines · Dave Airlie
Quoted 29 lines · John Hubbard

On 8/24/21 10:32 AM, Jason Gunthorpe wrote:
...

Quoted 25 lines
Quoted 13 lines
Quoted 3 lines

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

Quoted 4 lines

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

Quoted 6 lines
Quoted 2 lines

OK. I just want to avoid creating any API-level assumptions that dma_buf_pin() necessarily implies or requires migrating to host memory.

I'm not sure we should be allowing dma_buf_pin at all on non-migratable memory, what's to stop someone just pinning all the VRAM and making the GPU unuseable?

IMHO the same thinking that prevents pining all of system ram and making the system unusable? GPU isn't so special here. The main restriction is the pinned memory ulimit. For most out-of-the-box cases this is set to something like 64k

For the single-user HPC use cases it is made unlimited.

Quoted 3 lines

My impression from this is we've designed hardware that didn't consider the problem, and now to let us use that hardware in horrible ways we should just allow it to pin all the things.

It is more complex than that, HW that can support dynamic memory under *everything* is complicated (and in some cases slow!). As there is only a weak rational to do this, we don't see it in often in the market.

Jason

#18 lore raw
In reply to Dave Airlie
Quoted 31 lines · Dave Airlie
Quoted 29 lines · John Hubbard

On 8/24/21 10:32 AM, Jason Gunthorpe wrote:
...

Quoted 25 lines
Quoted 13 lines
Quoted 3 lines

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

Quoted 4 lines

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

Quoted 6 lines
Quoted 2 lines

OK. I just want to avoid creating any API-level assumptions that dma_buf_pin() necessarily implies or requires migrating to host memory.

I'm not sure we should be allowing dma_buf_pin at all on non-migratable memory, what's to stop someone just pinning all the VRAM and making the GPU unuseable?

In a lot of cases we have GPUs with more VRAM than system memory, but we allow pinning in system memory.

Alex

Quoted 10 lines

I understand not considering more than a single user in these situations is enterprise thinking, but I do worry about pinning is always fine type of thinking when things are shared or multi-user.

My impression from this is we've designed hardware that didn't consider the problem, and now to let us use that hardware in horrible ways we should just allow it to pin all the things.

Dave.

#19 lore raw
Quoted 47 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Tuesday, August 24, 2021 12:44 PM
To: Dave Airlie <[email protected]>
Cc: John Hubbard <[email protected]>; Jason Gunthorpe <[email protected]>; Christian König <[email protected]>; Gal Pressman <[email protected]>; Daniel Vetter <[email protected]>; Sumit Semwal <[email protected]>; Doug Ledford <[email protected]>; open list:DMA BUFFER SHARING FRAMEWORK <[email protected]>; dri-devel <dri- [email protected]>; Linux Kernel Mailing List <[email protected]>; linux-rdma <[email protected]>; Gabbay, Oded (Habana) <[email protected]>; Tayar, Tomer (Habana) <[email protected]>; Yossi Leybovich <[email protected]>; Alexander Matushevsky <[email protected]>; Leon Romanovsky <[email protected]>; Xiong, Jianxin <[email protected]>
Subject: Re: [RFC] Make use of non-dynamic dmabuf in RDMA

Quoted 34 lines · Dave Airlie
Quoted 31 lines · John Hubbard

On 8/24/21 10:32 AM, Jason Gunthorpe wrote:
...

Quoted 18 lines
Quoted 13 lines
Quoted 3 lines

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

Quoted 4 lines

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

Quoted 7 lines

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

OK. I just want to avoid creating any API-level assumptions that dma_buf_pin() necessarily implies or requires migrating to host memory.

Quoted 8 lines
Quoted 3 lines

I'm not sure we should be allowing dma_buf_pin at all on non-migratable memory, what's to stop someone just pinning all the VRAM and making the GPU unuseable?

In a lot of cases we have GPUs with more VRAM than system memory, but we allow pinning in system memory.

Alex

In addition, the dma-buf exporter can be a non-GPU device.

Jianxin

Quoted 10 lines
Quoted 10 lines

I understand not considering more than a single user in these situations is enterprise thinking, but I do worry about pinning is always fine type of thinking when things are shared or multi-user.

My impression from this is we've designed hardware that didn't consider the problem, and now to let us use that hardware in horrible ways we should just allow it to pin all the things.

Dave.

maintainer
#20 lore raw
In reply to Jason Gunthorpe

Am 24.08.21 um 19:32 schrieb Jason Gunthorpe:

Quoted 39 lines · John Hubbard
Quoted 34 lines · Christian König
Quoted 5 lines

Am 24.08.21 um 11:06 schrieb Gal Pressman:

Quoted 4 lines · Christian König
Quoted 3 lines

Am 21.08.21 um 11:16 schrieb Gal Pressman:

Quoted 2 lines · Jason Gunthorpe
Quoted 1 line · Gal Pressman

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a buffer (e.g. before you create an attachment) and the dma_buf_unpin() function after you are done with the DMA-buf.

Quoted 7 lines
Quoted 2 lines

I replied to your previous mail, but I'll ask again. Doesn't the pin operation migrate the memory to host memory?

Sorry missed your previous reply.

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

Quoted 4 lines

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

Quoted 6 lines

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

Yes, exactly that's the point. Especially GPUs have a great variety of setups.

For example we have APUs where the local memory is just stolen system memory and all buffers must be migrate-able because you might need all of this stolen memory for scanout or page tables. In this case P2P only makes sense to avoid the migration overhead in the first place.

Then you got dGPUs where only a fraction of the VRAM is accessible from the PCIe BUS. Here you also absolutely don't want to pin any buffers because that can easily crash when we need to migrate something into the visible window for CPU access.

The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to migrate anything. But even then you want some mechanism like cgroups to take care of limiting this. Otherwise any runaway process can bring down your whole system.

Key question at least for me as GPU maintainer is if we are going to see modern compute boards together with old non-ODP setups. Since those compute boards are usually used with new hardware (like PCIe v4 for example) the answer I think is most likely "no".

Christian.

Quoted 2 lines

Jason

#21 lore raw

On 8/24/21 11:17 PM, Christian König wrote:
...

Quoted 9 lines
Quoted 6 lines

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

Yes, exactly that's the point. Especially GPUs have a great variety of setups.

Quoted 8 lines

For example we have APUs where the local memory is just stolen system memory and all buffers must be migrate-able because you might need all of this stolen memory for scanout or page tables. In this case P2P only makes sense to avoid the migration overhead in the first place.

Then you got dGPUs where only a fraction of the VRAM is accessible from the PCIe BUS. Here you also absolutely don't want to pin any buffers because that can easily crash when we need to migrate something into the visible window for CPU access.

The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to migrate anything. But even then you want some mechanism like cgroups to take care of limiting this. Otherwise any runaway process can bring down your whole system.

Key question at least for me as GPU maintainer is if we are going to see modern compute boards together with old non-ODP setups. Since those compute boards are usually used with new hardware (like PCIe v4 for example) the answer I think is most likely "no".

That is a really good point. Times have changed and I guess ODP is on most (all?) of the new Infiniband products now, and maybe we don't need to worry so much about providing first-class support for non-ODP setups.

I've got to drag my brain into 2021+! :)

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
maintainer
#22 lore raw
In reply to Christian König
Quoted 5 lines · Christian König

The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to migrate anything. But even then you want some mechanism like cgroups to take care of limiting this. Otherwise any runaway process can bring down your whole system.

Why? It is not the pin that is the problem, it was allocating GPU dedicated memory in the first place. pinning it just changes the sequence to free it. No different than CPU memory.

Quoted 2 lines

Key question at least for me as GPU maintainer is if we are going to see modern compute boards together with old non-ODP setups.

You should stop thinking about it as 'old non-ODP setups'. ODP is very much a special case that allows a narrow set of functionality to work in a narrow situation. It has a high performance penalty and narrow HW support.

So yes, compute boards are routinely used in scenarios where ODP is not available, today and for the foreseeable future.

Jason

maintainer
#23 lore raw

Am 25.08.21 um 14:18 schrieb Jason Gunthorpe:

Quoted 11 lines · Christian König
Quoted 6 lines

The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to migrate anything. But even then you want some mechanism like cgroups to take care of limiting this. Otherwise any runaway process can bring down your whole system.

Why? It is not the pin that is the problem, it was allocating GPU dedicated memory in the first place. pinning it just changes the sequence to free it. No different than CPU memory.

Pinning makes the memory un-evictable.

In other words as long as we don't pin anything we can support as many processes as we want until we run out of swap space. Swapping sucks badly because your applications become pretty much unuseable, but you can easily recover from it by killing some process.

With pinning on the other hand somebody sooner or later receives an -ENOMEM or -ENOSPC and there is no guarantee that this goes to the right process.

Quoted 9 lines
Quoted 2 lines

Key question at least for me as GPU maintainer is if we are going to see modern compute boards together with old non-ODP setups.

You should stop thinking about it as 'old non-ODP setups'. ODP is very much a special case that allows a narrow set of functionality to work in a narrow situation. It has a high performance penalty and narrow HW support.

So yes, compute boards are routinely used in scenarios where ODP is not available, today and for the foreseeable future.

Yeah, that's the stuff I'm not sure about.

Christian.

Quoted 2 lines

Jason

maintainer
#24 lore raw
Quoted 14 lines · Christian König

Am 25.08.21 um 14:18 schrieb Jason Gunthorpe:

Quoted 10 lines · Christian König
Quoted 6 lines

The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to migrate anything. But even then you want some mechanism like cgroups to take care of limiting this. Otherwise any runaway process can bring down your whole system.

Why? It is not the pin that is the problem, it was allocating GPU dedicated memory in the first place. pinning it just changes the sequence to free it. No different than CPU memory.

Pinning makes the memory un-evictable.

Quoted 7 lines

In other words as long as we don't pin anything we can support as many processes as we want until we run out of swap space. Swapping sucks badly because your applications become pretty much unuseable, but you can easily recover from it by killing some process.

With pinning on the other hand somebody sooner or later receives an -ENOMEM or -ENOSPC and there is no guarantee that this goes to the right process.

It is not really different - you have the same failure mode once the system runs out of swap.

This is really the kernel side trying to push a policy to the user side that the user side doesn't want..

Dedicated systems are a significant use case here and should be supported, even if the same solution wouldn't be applicable to someone running a desktop.

Jason

maintainer
#25 lore raw

Am 25.08.21 um 14:38 schrieb Jason Gunthorpe:

Quoted 19 lines · Christian König
Quoted 18 lines

Am 25.08.21 um 14:18 schrieb Jason Gunthorpe:

Quoted 10 lines · Christian König
Quoted 6 lines

The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to migrate anything. But even then you want some mechanism like cgroups to take care of limiting this. Otherwise any runaway process can bring down your whole system.

Why? It is not the pin that is the problem, it was allocating GPU dedicated memory in the first place. pinning it just changes the sequence to free it. No different than CPU memory.

Pinning makes the memory un-evictable.

In other words as long as we don't pin anything we can support as many processes as we want until we run out of swap space. Swapping sucks badly because your applications become pretty much unuseable, but you can easily recover from it by killing some process.

Quoted 7 lines
Quoted 2 lines

With pinning on the other hand somebody sooner or later receives an -ENOMEM or -ENOSPC and there is no guarantee that this goes to the right process.

It is not really different - you have the same failure mode once the system runs out of swap.

This is really the kernel side trying to push a policy to the user side that the user side doesn't want..

But which is still the right thing to do as far as I can see. See userspace also doesn't want proper process isolation since it takes extra time.

Kernel development is driven by exposing the hardware functionality in a save and manageable manner to userspace, and not by fulfilling userspace requirements.

This is very important cause you otherwise you create a specialized system and not a general purpose kernel.

Quoted 3 lines

Dedicated systems are a significant use case here and should be supported, even if the same solution wouldn't be applicable to someone running a desktop.

And exactly that approach is not acceptable.

Christian.

Quoted 2 lines

Jason

maintainer
#26 lore raw
Quoted 22 lines · Christian König

Am 25.08.21 um 14:38 schrieb Jason Gunthorpe:

Quoted 21 lines · Christian König
Quoted 20 lines

Am 25.08.21 um 14:18 schrieb Jason Gunthorpe:

Quoted 10 lines · Christian König
Quoted 6 lines

The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to migrate anything. But even then you want some mechanism like cgroups to take care of limiting this. Otherwise any runaway process can bring down your whole system.

Why? It is not the pin that is the problem, it was allocating GPU dedicated memory in the first place. pinning it just changes the sequence to free it. No different than CPU memory.

Pinning makes the memory un-evictable.

In other words as long as we don't pin anything we can support as many processes as we want until we run out of swap space. Swapping sucks badly because your applications become pretty much unuseable, but you can easily recover from it by killing some process.

With pinning on the other hand somebody sooner or later receives an -ENOMEM or -ENOSPC and there is no guarantee that this goes to the right process.

Quoted 8 lines
Quoted 5 lines

It is not really different - you have the same failure mode once the system runs out of swap.

This is really the kernel side trying to push a policy to the user side that the user side doesn't want..

But which is still the right thing to do as far as I can see. See userspace also doesn't want proper process isolation since it takes extra time.

Why? You are pushing a policy of resource allocation/usage which more properly belongs in userspace.

Quoted 3 lines

Kernel development is driven by exposing the hardware functionality in a save and manageable manner to userspace, and not by fulfilling userspace requirements.

I don't agree with this, that is a 1980's view of OS design. So much these days in the kernel is driven entirely by boutique userspace requirements and is very much not about the classical abstract role of an OS.

Quoted 5 lines
Quoted 3 lines

Dedicated systems are a significant use case here and should be supported, even if the same solution wouldn't be applicable to someone running a desktop.

And exactly that approach is not acceptable.

We have knobs and settings all over the place to allow Linux to support a broad set of use cases from Android to servers, to HPC. So long as they can co-exist and the various optional modes do not critically undermine the security of the kernel, it is well in line with how things have been evolving in the last 15 years.

Here you are talking entirely about policy to control memory allocation, which is already well trodden ground for CPU memory.

There are now endless boutique ways to deal with this, it is a very narrow view to say that GPU memory is so special and different that only one way can be the correct/allowed way.

Jason

maintainer
#27 lore raw

Am 25.08.21 um 16:47 schrieb Jason Gunthorpe:

Quoted 33 lines · Christian König
Quoted 22 lines

Am 25.08.21 um 14:38 schrieb Jason Gunthorpe:

Quoted 21 lines · Christian König
Quoted 20 lines

Am 25.08.21 um 14:18 schrieb Jason Gunthorpe:

Quoted 10 lines · Christian König
Quoted 6 lines

The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to migrate anything. But even then you want some mechanism like cgroups to take care of limiting this. Otherwise any runaway process can bring down your whole system.

Why? It is not the pin that is the problem, it was allocating GPU dedicated memory in the first place. pinning it just changes the sequence to free it. No different than CPU memory.

Pinning makes the memory un-evictable.

In other words as long as we don't pin anything we can support as many processes as we want until we run out of swap space. Swapping sucks badly because your applications become pretty much unuseable, but you can easily recover from it by killing some process.

With pinning on the other hand somebody sooner or later receives an -ENOMEM or -ENOSPC and there is no guarantee that this goes to the right process.

Quoted 7 lines
Quoted 5 lines

It is not really different - you have the same failure mode once the system runs out of swap.

This is really the kernel side trying to push a policy to the user side that the user side doesn't want..

But which is still the right thing to do as far as I can see. See userspace also doesn't want proper process isolation since it takes extra time.

Why? You are pushing a policy of resource allocation/usage which more properly belongs in userspace.

Quoted 7 lines
Quoted 3 lines

Kernel development is driven by exposing the hardware functionality in a save and manageable manner to userspace, and not by fulfilling userspace requirements.

I don't agree with this, that is a 1980's view of OS design. So much these days in the kernel is driven entirely by boutique userspace requirements and is very much not about the classical abstract role of an OS.

But it's still true never the less. Otherwise you would have libraries for filesystem accesses and no system security to speak of.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Dedicated systems are a significant use case here and should be supported, even if the same solution wouldn't be applicable to someone running a desktop.

And exactly that approach is not acceptable.

We have knobs and settings all over the place to allow Linux to support a broad set of use cases from Android to servers, to HPC. So long as they can co-exist and the various optional modes do not critically undermine the security of the kernel, it is well in line with how things have been evolving in the last 15 years.

Yeah, that's exactly what I'm talking about by adding cgroup or similar. You need a knob to control this.

Quoted 6 lines

Here you are talking entirely about policy to control memory allocation, which is already well trodden ground for CPU memory.

There are now endless boutique ways to deal with this, it is a very narrow view to say that GPU memory is so special and different that only one way can be the correct/allowed way.

Well I'm not talking about GPU memory in particular here. This is mandatory for any memory or saying more general any resource.

E.g. you are not allowed to pin large amount of system memory on a default installation for exactly those reasons as well.

That you can have a knob to disable this behavior for your HPC system is perfectly fine, but I thing what Dave notes here as well that this is most likely not the default behavior we want.

Christian.

Quoted 2 lines

Jason

maintainer
#28 lore raw
Quoted 2 lines · Christian König

Yeah, that's exactly what I'm talking about by adding cgroup or similar. You need a knob to control this.

We have the pinned memory ulimit today.

A pinned memory cgroup might be interesting, but even containrs are covered under the ulimit (IIRC), so the driver to do this work might not be so strong.

Jason

#29 lore raw
In reply to Christian König
Quoted 66 lines · Christian König

Am 25.08.21 um 16:47 schrieb Jason Gunthorpe:

Quoted 33 lines · Christian König
Quoted 22 lines

Am 25.08.21 um 14:38 schrieb Jason Gunthorpe:

Quoted 21 lines · Christian König
Quoted 20 lines

Am 25.08.21 um 14:18 schrieb Jason Gunthorpe:

Quoted 10 lines · Christian König
Quoted 6 lines

The only real option where you could do P2P with buffer pinning are those compute boards where we know that everything is always accessible to everybody and we will never need to migrate anything. But even then you want some mechanism like cgroups to take care of limiting this. Otherwise any runaway process can bring down your whole system.

Why? It is not the pin that is the problem, it was allocating GPU dedicated memory in the first place. pinning it just changes the sequence to free it. No different than CPU memory.

Pinning makes the memory un-evictable.

In other words as long as we don't pin anything we can support as many processes as we want until we run out of swap space. Swapping sucks badly because your applications become pretty much unuseable, but you can easily recover from it by killing some process.

With pinning on the other hand somebody sooner or later receives an -ENOMEM or -ENOSPC and there is no guarantee that this goes to the right process.

Quoted 7 lines
Quoted 5 lines

It is not really different - you have the same failure mode once the system runs out of swap.

This is really the kernel side trying to push a policy to the user side that the user side doesn't want..

But which is still the right thing to do as far as I can see. See userspace also doesn't want proper process isolation since it takes extra time.

Why? You are pushing a policy of resource allocation/usage which more properly belongs in userspace.

Quoted 7 lines
Quoted 3 lines

Kernel development is driven by exposing the hardware functionality in a save and manageable manner to userspace, and not by fulfilling userspace requirements.

I don't agree with this, that is a 1980's view of OS design. So much these days in the kernel is driven entirely by boutique userspace requirements and is very much not about the classical abstract role of an OS.

But it's still true never the less. Otherwise you would have libraries for filesystem accesses and no system security to speak of.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Dedicated systems are a significant use case here and should be supported, even if the same solution wouldn't be applicable to someone running a desktop.

And exactly that approach is not acceptable.

We have knobs and settings all over the place to allow Linux to support a broad set of use cases from Android to servers, to HPC. So long as they can co-exist and the various optional modes do not critically undermine the security of the kernel, it is well in line with how things have been evolving in the last 15 years.

Yeah, that's exactly what I'm talking about by adding cgroup or similar. You need a knob to control this.

Quoted 6 lines

Here you are talking entirely about policy to control memory allocation, which is already well trodden ground for CPU memory.

There are now endless boutique ways to deal with this, it is a very narrow view to say that GPU memory is so special and different that only one way can be the correct/allowed way.

Quoted 5 lines

Well I'm not talking about GPU memory in particular here. This is mandatory for any memory or saying more general any resource.

E.g. you are not allowed to pin large amount of system memory on a default installation for exactly those reasons as well.

Unless you are working on a h/w architecture that is designed specifically for a single user.
At least in our h/w architecture, we aim for 100% utilization of the h/w when you are running DL training workloads, as opposed to what is happening inside a GPU when you are running training. Therefore, it is counter-productive to serve multiple users from a performance perspective.
In fact, the training problem is so large, that a single ASIC is hardly sufficient, and you need to run on anywhere between 8 ASICs to 64-128 ASICs to get a reasonable time for training.
So there is no point in serving multiple users in that case and that's why our device memory (HBM) is always pinned.

This is totally different from the usual role of a GPU, where you serve the desktop and multiple other applications that draw on the screen.

I wouldn't want some knob in the kernel to control what is the limit of memory I can pin or not. I just don't think it is useful and/or user friendly.

btw, regarding ODP, we don't support it in h/w (yet) because 99% of the use-cases in DL training will suffer from performance issues if we don't pin the host memory.
There are extreme topologies, with 10TB of parameters that may require this, but like I said, it's very rare and not worth the effort.

Thanks,
Oded

Quoted 10 lines

That you can have a knob to disable this behavior for your HPC system is perfectly fine, but I thing what Dave notes here as well that this is most likely not the default behavior we want.

Christian.

Quoted 2 lines

Jason

#30 lore raw
In reply to Jason Gunthorpe
Quoted 42 lines · Jason Gunthorpe
Quoted 36 lines · John Hubbard
Quoted 5 lines · Christian König

Am 24.08.21 um 11:06 schrieb Gal Pressman:

Quoted 4 lines · Christian König
Quoted 3 lines

Am 21.08.21 um 11:16 schrieb Gal Pressman:

Quoted 2 lines · Jason Gunthorpe
Quoted 1 line · Gal Pressman

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a buffer (e.g. before you create an attachment) and the dma_buf_unpin() function after you are done with the DMA-buf.

Quoted 8 lines
Quoted 2 lines

I replied to your previous mail, but I'll ask again. Doesn't the pin operation migrate the memory to host memory?

Sorry missed your previous reply.

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

Quoted 4 lines

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

Quoted 6 lines

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

So are we OK with exporters implementing dma_buf_pin() without migrating the memory? If so, do we still want a move_notify callback for non-dynamic importers? A noop?

maintainer
#31 lore raw

Am 01.09.21 um 13:20 schrieb Gal Pressman:

Quoted 40 lines · Jason Gunthorpe
Quoted 39 lines · John Hubbard
Quoted 38 lines · Christian König
Quoted 5 lines

Am 24.08.21 um 11:06 schrieb Gal Pressman:

Quoted 4 lines · Christian König
Quoted 3 lines

Am 21.08.21 um 11:16 schrieb Gal Pressman:

Quoted 2 lines · Jason Gunthorpe
Quoted 1 line · Gal Pressman

...

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines

IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a buffer (e.g. before you create an attachment) and the dma_buf_unpin() function after you are done with the DMA-buf.

Quoted 7 lines
Quoted 2 lines

I replied to your previous mail, but I'll ask again. Doesn't the pin operation migrate the memory to host memory?

Sorry missed your previous reply.

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

Quoted 7 lines
Quoted 6 lines

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

So are we OK with exporters implementing dma_buf_pin() without migrating the memory?

I think so, yes.

Quoted 1 line

If so, do we still want a move_notify callback for non-dynamic importers? A noop?

Well we could make the move_notify callback optional, e.g. so that you get the new locking approach but still pin the buffers manually with dma_buf_pin().

Regards,
Christian.

#32 lore raw
Quoted 54 lines · Christian König

Am 01.09.21 um 13:20 schrieb Gal Pressman:

Quoted 42 lines · Jason Gunthorpe
Quoted 41 lines · John Hubbard
Quoted 40 lines · Christian König
Quoted 5 lines

Am 24.08.21 um 11:06 schrieb Gal Pressman:

Quoted 4 lines · Christian König
Quoted 3 lines

Am 21.08.21 um 11:16 schrieb Gal Pressman:

Quoted 2 lines · Jason Gunthorpe
Quoted 1 line · Gal Pressman

...

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 6 lines

IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a
buffer (e.g. before you create an attachment) and the dma_buf_unpin() function
after you are done with the DMA-buf.

Quoted 7 lines
Quoted 2 lines

I replied to your previous mail, but I'll ask again. Doesn't the pin operation migrate the memory to host memory?

Sorry missed your previous reply.

And yes at least for the amdgpu driver we migrate the memory to host memory as soon as it is pinned and I would expect that other GPU drivers do something similar.

Well...for many topologies, migrating to host memory will result in a dramatically slower p2p setup. For that reason, some GPU drivers may want to allow pinning of video memory in some situations.

Ideally, you've got modern ODP devices and you don't even need to pin. But if not, and you still hope to do high performance p2p between a GPU and a non-ODP Infiniband device, then you would need to leave the pinned memory in vidmem.

So I think we don't want to rule out that behavior, right? Or is the thinking more like, "you're lucky that this old non-ODP setup works at all, and we'll make it work by routing through host/cpu memory, but it will be slow"?

Quoted 8 lines
Quoted 6 lines

I think it depends on the user, if the user creates memory which is permanently located on the GPU then it should be pinnable in this way without force migration. But if the memory is inherently migratable then it just cannot be pinned in the GPU at all as we can't indefinately block migration from happening eg if the CPU touches it later or something.

So are we OK with exporters implementing dma_buf_pin() without migrating the memory?

Quoted 6 lines

I think so, yes.

Quoted 1 line

If so, do we still want a move_notify callback for non-dynamic importers? A noop?

Well we could make the move_notify callback optional, e.g. so that you get the new locking approach but still pin the buffers manually with dma_buf_pin().

Thanks Christian!
So the end result will look similar to the original patch I posted, where peer2peer can be enabled without providing move_notify, correct?