Skip to contentSkip to conversation

dma-buf changes for ttm and amdgpu

PATCH In review Alex Deucher 8 participants 9 messages Aug 10, 2017 – Aug 11, 2017 5 messages not in the archive DMA buffer sharing framework

lore
Share & tools
lore
#0 lore raw

cover dma-buf changes for ttm and amdgpu

We have some changes in ttm and amdgpu that depend on these patches. Sumit, can you pull these in via dma-buf or should I roll them up through my amdgpu tree?

Christian König (3):
dma-buf: dma_fence_put is NULL safe
dma-buf: add reservation_object_copy_fences
dma-buf: fix reservation_object_wait_timeout_rcu to wait correctly v2

 drivers/dma-buf/reservation.c | 97 +++++++++++++++++++++++++++++++++----------
 include/linux/reservation.h   |  3 ++
 2 files changed, 78 insertions(+), 22 deletions(-)
Signature · 6 lines
2.5.5

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1502384509-10465-1-git-send-email-alexander.deucher-5C7GfCeVMHo@public.gmane.org

#2 lore raw

PATCH 1/2 dma-buf: add reservation_object_copy_fences (v2)

From: Christian König <[email protected]>

Allows us to copy all the fences in a reservation object to another one.

v2: handle NULL src_list

---
 drivers/dma-buf/reservation.c | 60 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/reservation.h   |  3 +++
 2 files changed, 63 insertions(+)
drivers/dma-buf/reservation.c +60 -0
index 87f8f57..d4881f9 100644
@@ -262,6 +262,66 @@ void reservation_object_add_excl_fence(struct reservation_object *obj,
 EXPORT_SYMBOL(reservation_object_add_excl_fence);
 
 /**
+* reservation_object_copy_fences - Copy all fences from src to dst.
+* @dst: the destination reservation object
+* @src: the source reservation object
+*
+* Copy all fences from src to dst. Both src->lock as well as dst-lock must be
+* held.
+*/
+int reservation_object_copy_fences(struct reservation_object *dst,
+				   struct reservation_object *src)
+{
+	struct reservation_object_list *src_list, *dst_list;
+	struct dma_fence *old, *new;
+	size_t size;
+	unsigned i;
+
+	src_list = reservation_object_get_list(src);
+
+	if (src_list) {
+		size = offsetof(typeof(*src_list),
+				shared[src_list->shared_count]);
+		dst_list = kmalloc(size, GFP_KERNEL);
+		if (!dst_list)
+			return -ENOMEM;
+
+		dst_list->shared_count = src_list->shared_count;
+		dst_list->shared_max = src_list->shared_count;
+		for (i = 0; i < src_list->shared_count; ++i)
+			dst_list->shared[i] =
+				dma_fence_get(src_list->shared[i]);
+	} else {
+		dst_list = NULL;
+	}
+
+	kfree(dst->staged);
+	dst->staged = NULL;
+
+	src_list = reservation_object_get_list(dst);
+
+	old = reservation_object_get_excl(dst);
+	new = reservation_object_get_excl(src);
+
+	dma_fence_get(new);
+
+	preempt_disable();
+	write_seqcount_begin(&dst->seq);
+	/* write_seqcount_begin provides the necessary memory barrier */
+	RCU_INIT_POINTER(dst->fence_excl, new);
+	RCU_INIT_POINTER(dst->fence, dst_list);
+	write_seqcount_end(&dst->seq);
+	preempt_enable();
+
+	if (src_list)
+		kfree_rcu(src_list, rcu);
+	dma_fence_put(old);
+
+	return 0;
+}
+EXPORT_SYMBOL(reservation_object_copy_fences);
+
+/**
  * reservation_object_get_fences_rcu - Get an object's shared and exclusive
  * fences without update side lock held
  * @obj: the reservation object
include/linux/reservation.h +3 -0
index 156cfd3..21fc84d 100644
@@ -254,6 +254,9 @@ int reservation_object_get_fences_rcu(struct reservation_object *obj,
 				      unsigned *pshared_count,
 				      struct dma_fence ***pshared);
 
+int reservation_object_copy_fences(struct reservation_object *dst,
+				   struct reservation_object *src);
+
 long reservation_object_wait_timeout_rcu(struct reservation_object *obj,
 					 bool wait_all, bool intr,
 					 unsigned long timeout);
Signature · 6 lines
2.5.5

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Acked-by
#3 lore raw
In reply to a message not in the archive

Hi Alex,

Quoted 9 lines · Alex Deucher

We have some changes in ttm and amdgpu that depend on these patches. Sumit, can you pull these in via dma-buf or should I roll them up through my amdgpu tree?

Christian König (3):
dma-buf: dma_fence_put is NULL safe
dma-buf: add reservation_object_copy_fences
dma-buf: fix reservation_object_wait_timeout_rcu to wait correctly v2

Looks like my reply on your last set of patches didn't reach you?

Anyways, I'm on a short vacation, so please feel free to take this via drm-misc, with my

Quoted 7 lines
 drivers/dma-buf/reservation.c | 97 +++++++++++++++++++++++++++++++++----------
 include/linux/reservation.h   |  3 ++
 2 files changed, 78 insertions(+), 22 deletions(-)

--
2.5.5

Best,
Sumit.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAO_48GHGPEbzmD2LLePvW5jmpjRxC7UgoQV-_uMjL3syoB1knA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#5 lore raw
Quoted 19 lines

-----Original Message-----
From: Sumit Semwal [mailto:[email protected]]
Sent: Friday, August 11, 2017 11:52 AM
To: Alex Deucher
Cc: amd-gfx list; DRI mailing list; Koenig, Christian; Deucher, Alexander
Subject: Re: [PATCH 0/3] dma-buf changes for ttm and amdgpu

Hi Alex,

Quoted 9 lines · Alex Deucher

We have some changes in ttm and amdgpu that depend on these patches. Sumit, can you pull these in via dma-buf or should I roll them up through my amdgpu tree?

Christian König (3):
dma-buf: dma_fence_put is NULL safe
dma-buf: add reservation_object_copy_fences
dma-buf: fix reservation_object_wait_timeout_rcu to wait correctly v2

Quoted 6 lines

Looks like my reply on your last set of patches didn't reach you?

Anyways, I'm on a short vacation, so please feel free to take this via drm-misc, with my

I got your last ack, but I ended up re-sending the last two patches because the second one had a bug.

Thanks,

Alex

Quoted 11 lines
Quoted 1 line
 drivers/dma-buf/reservation.c | 97

+++++++++++++++++++++++++++++++++----------

Quoted 6 lines
 include/linux/reservation.h   |  3 ++
 2 files changed, 78 insertions(+), 22 deletions(-)

--
2.5.5

Best,
Sumit.

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#6 lore raw

PATCH 2/2 dma-buf: fix reservation_object_wait_timeout_rcu to wait correctly v2

In reply to Alex Deucher

From: Christian König <[email protected]>

With hardware resets in mind it is possible that all shared fences are signaled, but the exlusive isn't. Fix waiting for everything in this situation.

v2: make sure we always wait for the exclusive fence

---
 drivers/dma-buf/reservation.c | 33 +++++++++++++++------------------
 1 file changed, 15 insertions(+), 18 deletions(-)
drivers/dma-buf/reservation.c +15 -18
index d4881f9..dec3a81 100644
@@ -431,12 +431,25 @@ long reservation_object_wait_timeout_rcu(struct reservation_object *obj,
 	long ret = timeout ? timeout : 1;
 
 retry:
-	fence = NULL;
 	shared_count = 0;
 	seq = read_seqcount_begin(&obj->seq);
 	rcu_read_lock();
 
-	if (wait_all) {
+	fence = rcu_dereference(obj->fence_excl);
+	if (fence && !test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)) {
+		if (!dma_fence_get_rcu(fence))
+			goto unlock_retry;
+
+		if (dma_fence_is_signaled(fence)) {
+			dma_fence_put(fence);
+			fence = NULL;
+		}
+
+	} else {
+		fence = NULL;
+	}
+
+	if (!fence && wait_all) {
 		struct reservation_object_list *fobj =
 						rcu_dereference(obj->fence);
 
@@ -463,22 +476,6 @@ long reservation_object_wait_timeout_rcu(struct reservation_object *obj,
 		}
 	}
 
-	if (!shared_count) {
-		struct dma_fence *fence_excl = rcu_dereference(obj->fence_excl);
-
-		if (fence_excl &&
-		    !test_bit(DMA_FENCE_FLAG_SIGNALED_BIT,
-			      &fence_excl->flags)) {
-			if (!dma_fence_get_rcu(fence_excl))
-				goto unlock_retry;
-
-			if (dma_fence_is_signaled(fence_excl))
-				dma_fence_put(fence_excl);
-			else
-				fence = fence_excl;
-		}
-	}
-
 	rcu_read_unlock();
 	if (fence) {
 		if (read_seqcount_retry(&obj->seq, seq)) {
Signature · 6 lines
2.5.5

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#7 lore raw

Re: [PATCH 2/2] dma-buf: fix reservation_object_wait_timeout_rcu to wait correctly v2

Quoting Alex Deucher (2017-08-10 18:01:49)

Quoted 4 lines

From: Christian König <[email protected]>

With hardware resets in mind it is possible that all shared fences are signaled, but the exlusive isn't. Fix waiting for everything in this situation.

I'm still puzzling over this one.

Setting an exclusive fence will clear all shared, so we must have added the shared fences after the exclusive fence. But those shared fences must follow the exclusive fence, you should not be able to reorder the shared fences ahead of the exclusive. If you have completed shared fences, but incomplete exclusive, doesn't that imply you have an ordering issue? -Chris
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

A message that is not in the archive (sent off-list or not yet received): 150238508616.26524.9935181193137231431-M6iVdVfohj6unts5RBS2dVaTQe2KTcn/@public.gmane.org

#9 lore raw

Re: [PATCH 2/2] dma-buf: fix reservation_object_wait_timeout_rcu to wait correctly v2

Am 10.08.2017 um 19:11 schrieb Chris Wilson:

Quoted 5 lines

Quoting Alex Deucher (2017-08-10 18:01:49)

Quoted 4 lines

From: Christian König <[email protected]>

With hardware resets in mind it is possible that all shared fences are signaled, but the exlusive isn't. Fix waiting for everything in this situation.

Quoted 7 lines

I'm still puzzling over this one.

Setting an exclusive fence will clear all shared, so we must have added the shared fences after the exclusive fence. But those shared fences must follow the exclusive fence, you should not be able to reorder the shared fences ahead of the exclusive. If you have completed shared fences, but incomplete exclusive, doesn't that imply you have an ordering issue?

No, that is not an ordering issue.

The problem happens when the shared fences are "aborted" because of a GPU reset.

See the exclusive fence is often a DMA moving bytes on behalf of the kernel while the shared fences are the actual command submission from user space.

What can happen is that the userspace process is killed and/or it's scheduled command submission aborted for other reasons.

In this case the shared fences get an error code, are set to the signaled state and the job they represent is never executed.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): dd839052-89b5-d3ee-9347-4590ac8de0fa-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#11 lore raw

Re: [PATCH 2/2] dma-buf: fix reservation_object_wait_timeout_rcu to wait correctly v2

Quoting Christian König (2017-08-10 19:19:52)

Quoted 23 lines

Am 10.08.2017 um 19:11 schrieb Chris Wilson:

Quoted 5 lines

Quoting Alex Deucher (2017-08-10 18:01:49)

Quoted 4 lines

From: Christian König <[email protected]>

With hardware resets in mind it is possible that all shared fences are signaled, but the exlusive isn't. Fix waiting for everything in this situation.

Quoted 7 lines

I'm still puzzling over this one.

Setting an exclusive fence will clear all shared, so we must have added the shared fences after the exclusive fence. But those shared fences must follow the exclusive fence, you should not be able to reorder the shared fences ahead of the exclusive. If you have completed shared fences, but incomplete exclusive, doesn't that imply you have an ordering issue?

No, that is not an ordering issue.

The problem happens when the shared fences are "aborted" because of a GPU reset.

See the exclusive fence is often a DMA moving bytes on behalf of the kernel while the shared fences are the actual command submission from user space.

Quoted 5 lines

What can happen is that the userspace process is killed and/or it's scheduled command submission aborted for other reasons.

In this case the shared fences get an error code, are set to the signaled state and the job they represent is never executed.

I didn't allow the GPU reset to cause fences to appear to complete out of order, just rewrote them so they would be skipped (with the error status flagged) if the error was terminal but still update the hw semaphore and breadcrumbs (i.e. the user portion was cancel, but our bookkeeping was left intact).
-Chris
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 150239103808.28941.14805152055152204286-M6iVdVfohj6unts5RBS2dVaTQe2KTcn/@public.gmane.org

#13 lore raw

Re: [PATCH 2/2] dma-buf: fix reservation_object_wait_timeout_rcu to wait correctly v2

Am 10.08.2017 um 20:50 schrieb Chris Wilson:

Quoted 26 lines

Quoting Christian König (2017-08-10 19:19:52)

Quoted 25 lines

Am 10.08.2017 um 19:11 schrieb Chris Wilson:

Quoted 5 lines

Quoting Alex Deucher (2017-08-10 18:01:49)

Quoted 4 lines

From: Christian König <[email protected]>

With hardware resets in mind it is possible that all shared fences are signaled, but the exlusive isn't. Fix waiting for everything in this situation.

Quoted 7 lines

I'm still puzzling over this one.

Setting an exclusive fence will clear all shared, so we must have added the shared fences after the exclusive fence. But those shared fences must follow the exclusive fence, you should not be able to reorder the shared fences ahead of the exclusive. If you have completed shared fences, but incomplete exclusive, doesn't that imply you have an ordering issue?

No, that is not an ordering issue.

The problem happens when the shared fences are "aborted" because of a GPU reset.

See the exclusive fence is often a DMA moving bytes on behalf of the kernel while the shared fences are the actual command submission from user space.

What can happen is that the userspace process is killed and/or it's scheduled command submission aborted for other reasons.

Quoted 7 lines
Quoted 2 lines

In this case the shared fences get an error code, are set to the signaled state and the job they represent is never executed.

I didn't allow the GPU reset to cause fences to appear to complete out of order, just rewrote them so they would be skipped (with the error status flagged) if the error was terminal but still update the hw semaphore and breadcrumbs (i.e. the user portion was cancel, but our bookkeeping was left intact).

Well again, they don't signal out of order. The exclusive fence and the shared fence which is aborted belong to completely independent contexts (and can even belong to completely independent hardware).

It's just that the job which should trigger the shared fence never gets executed.

Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx