Skip to contentSkip to conversation

drm/msm: move submit fence wait out of struct_mutex

PATCH In review Rob Clark 3 participants 4 messages Feb 25, 2017

Subsystems (3)
lore
Share & tools
lore

Rob Clark

author
#0 lore raw

PATCH 1/1 drm/msm: move submit fence wait out of struct_mutex

Probably a symptom of needing finer grained locking, but if we wait on the incoming fence-fd (which could come from a different context) while holding struct_mutex, that blocks retire_worker so gpu fences cannot get scheduled.

This causes a problem if userspace manages to get more than a frame ahead, leaving the atomic-commit worker blocked waiting on fences that cannot be signaled because submit is blocked waiting for a fence signalled from vblank (after the atomic commit which is blocked).

If we start having multiple fence ctxs for the gpu, submit_fence_sync() would probably need to move outside of struct_mutex as well.

---
 drivers/gpu/drm/msm/msm_gem_submit.c | 39 +++++++++++++++++-------------------
 1 file changed, 18 insertions(+), 21 deletions(-)
drivers/gpu/drm/msm/msm_gem_submit.c +18 -21
index 1172fe7..1c545eb 100644
@@ -404,6 +404,24 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
 	if (MSM_PIPE_FLAGS(args->flags) & ~MSM_SUBMIT_FLAGS)
 		return -EINVAL;
 
+	if (args->flags & MSM_SUBMIT_FENCE_FD_IN) {
+		in_fence = sync_file_get_fence(args->fence_fd);
+
+		if (!in_fence)
+			return -EINVAL;
+
+		/* TODO if we get an array-fence due to userspace merging multiple
+		 * fences, we need a way to determine if all the backing fences
+		 * are from our own context..
+		 */
+
+		if (in_fence->context != gpu->fctx->context) {
+			ret = dma_fence_wait(in_fence, true);
+			if (ret)
+				return ret;
+		}
+	}
+
 	ret = mutex_lock_interruptible(&dev->struct_mutex);
 	if (ret)
 		return ret;
@@ -431,27 +449,6 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
 	if (ret)
 		goto out;
 
-	if (args->flags & MSM_SUBMIT_FENCE_FD_IN) {
-		in_fence = sync_file_get_fence(args->fence_fd);
-
-		if (!in_fence) {
-			ret = -EINVAL;
-			goto out;
-		}
-
-		/* TODO if we get an array-fence due to userspace merging multiple
-		 * fences, we need a way to determine if all the backing fences
-		 * are from our own context..
-		 */
-
-		if (in_fence->context != gpu->fctx->context) {
-			ret = dma_fence_wait(in_fence, true);
-			if (ret)
-				goto out;
-		}
-
-	}
-
 	if (!(args->fence & MSM_SUBMIT_NO_IMPLICIT)) {
 		ret = submit_fence_sync(submit);
 		if (ret)
Signature · 6 lines
2.9.3

_______________________________________________
Freedreno mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/freedreno
#1 lore raw
Quoted 4 lines · Rob Clark

Probably a symptom of needing finer grained locking, but if we wait on the incoming fence-fd (which could come from a different context) while holding struct_mutex, that blocks retire_worker so gpu fences cannot get scheduled.

s/scheduled/signalled/

Quoted 71 lines

This causes a problem if userspace manages to get more than a frame ahead, leaving the atomic-commit worker blocked waiting on fences that cannot be signaled because submit is blocked waiting for a fence signalled from vblank (after the atomic commit which is blocked).

If we start having multiple fence ctxs for the gpu, submit_fence_sync() would probably need to move outside of struct_mutex as well.

---
 drivers/gpu/drm/msm/msm_gem_submit.c | 39 +++++++++++++++++-------------------
 1 file changed, 18 insertions(+), 21 deletions(-)
drivers/gpu/drm/msm/msm_gem_submit.c +18 -22
index 1172fe7..1c545eb 100644
@@ -404,6 +404,24 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
        if (MSM_PIPE_FLAGS(args->flags) & ~MSM_SUBMIT_FLAGS)
                return -EINVAL;
 
+       if (args->flags & MSM_SUBMIT_FENCE_FD_IN) {
+               in_fence = sync_file_get_fence(args->fence_fd);
+
+               if (!in_fence)
+                       return -EINVAL;
+
+               /* TODO if we get an array-fence due to userspace merging multiple
+                * fences, we need a way to determine if all the backing fences
+                * are from our own context..
+                */
+
+               if (in_fence->context != gpu->fctx->context) {
+                       ret = dma_fence_wait(in_fence, true);
+                       if (ret)
+                               return ret;
+               }
+       }
+
        ret = mutex_lock_interruptible(&dev->struct_mutex);
        if (ret)
                return ret;
@@ -431,27 +449,6 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
        if (ret)
                goto out;
 
-       if (args->flags & MSM_SUBMIT_FENCE_FD_IN) {
-               in_fence = sync_file_get_fence(args->fence_fd);
-
-               if (!in_fence) {
-                       ret = -EINVAL;
-                       goto out;
-               }
-
-               /* TODO if we get an array-fence due to userspace merging multiple
-                * fences, we need a way to determine if all the backing fences
-                * are from our own context..
-                */
-
-               if (in_fence->context != gpu->fctx->context) {
-                       ret = dma_fence_wait(in_fence, true);
-                       if (ret)
-                               goto out;
-               }
-
-       }
-
        if (!(args->fence & MSM_SUBMIT_NO_IMPLICIT)) {
                ret = submit_fence_sync(submit);
                if (ret)
--
Quoted 2 lines

2.9.3

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#2 lore raw
In reply to Rob Clark
Quoted 10 lines · Rob Clark

Probably a symptom of needing finer grained locking, but if we wait on the incoming fence-fd (which could come from a different context) while holding struct_mutex, that blocks retire_worker so gpu fences cannot get scheduled.

This causes a problem if userspace manages to get more than a frame ahead, leaving the atomic-commit worker blocked waiting on fences that cannot be signaled because submit is blocked waiting for a fence signalled from vblank (after the atomic commit which is blocked).

Quoted 4 lines

If we start having multiple fence ctxs for the gpu, submit_fence_sync() would probably need to move outside of struct_mutex as well.

It's unfortunately not yet landed, but as soon as it's there I think we should annotate fences using the cross-release stuff that's cooking:

https://lwn.net/Articles/708628/

That should be able to catch deps between waiter/signaller vs. locking. -Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
#3 lore raw
Quoted 16 lines · Daniel Vetter
Quoted 10 lines · Rob Clark

Probably a symptom of needing finer grained locking, but if we wait on the incoming fence-fd (which could come from a different context) while holding struct_mutex, that blocks retire_worker so gpu fences cannot get scheduled.

This causes a problem if userspace manages to get more than a frame ahead, leaving the atomic-commit worker blocked waiting on fences that cannot be signaled because submit is blocked waiting for a fence signalled from vblank (after the atomic commit which is blocked).

Quoted 4 lines

If we start having multiple fence ctxs for the gpu, submit_fence_sync() would probably need to move outside of struct_mutex as well.

Quoted 7 lines

It's unfortunately not yet landed, but as soon as it's there I think we should annotate fences using the cross-release stuff that's cooking:

https://lwn.net/Articles/708628/

That should be able to catch deps between waiter/signaller vs. locking.

ohh, that would be spiffy

BR,
-R
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel